Skip to content

Run a virtual iPhone on an Apple Silicon Mac.

Swift

DeviceHub: encode a screen frame only when the screen drew

Jacky07376b12cd5e
diff --git a/Research/0_binary_patch_comparison.md b/Research/0_binary_patch_comparison.md
index 102608fe..7fd4c3c5 100644
--- a/Research/0_binary_patch_comparison.md
+++ b/Research/0_binary_patch_comparison.md
@@ -1602,7 +1602,7 @@ do NOT execute these).
| 23 | No binary change: `libvcamcaptured.dylib`, which SystemHook already loads into `cameracaptured`, rewrites the `bl PrewarmThreadSafeSBPs` in `__FigCapturePreloadShadersInternal_block_invoke_2` to `nop` in the daemon's own memory. This runs from the dylib's constructor and only when the paravirtual Metal driver is installed — **all bases** | none (guest dylib `VPhoneGuestComponents/VCamCaptured/Prewarm/`, installed at `/usr/lib/libvcamcaptured.dylib`) | **Stop `cameracaptured` crashing at every boot, and the SpringBoard freeze that follows.** At launch the daemon preloads its shaders. `PrewarmThreadSafeSBPs` runs each video processor's `-prewarm`, and NRFV3's reaches `-[ToneMappingCurves initWithWithContext:]`, which fills textures made from a shared `MTLHeap` with `-replaceRegion:…`. The cloudOS 26.4 `AppleParavirtGPUMetalIOGPUFamily` never sets `_dimension` (a texture's CPU layout) in `-initWithHeap:resource:offset:length:descriptor:`, and `-replaceRegion:` loads it unchecked, so the daemon faults with `KERN_INVALID_ADDRESS` at `0xc` (measured on `iPad16,1 26.6.2 (23G90)`, `standard`: launchd at `successive crashes = 6`). While the daemon crash-loops, SpringBoard's main thread blocked in a synchronous XPC to it as soon as an app started recording. The daemon does have a Metal device, so `kernel-exp-paravirt_user_clients` does not help. Prewarming is only an optimisation, and nothing waits on that function: no return value, no global, no signal. The rest of the preload (processor flags, data migration, and the deferred shader cache copy, whose semaphore deferred processing waits on) is kept. The call site is found from the exported `FigCapturePreloadShaders`: its `b` to the internal function, that function's one `adrp/add/pacia x16` block invoke, then the block's one `ldr x0, [xN, #0x20]; bl` into CMCapture's `__text`. That gives `0x1aeb2fd68` on 23G90 and `0x1b0759d10` on 24A435. Any mismatch leaves the prewarm in place and logs the step that failed. Host test: `make -C VPhoneGuestComponents test-vcam-prewarm`. Validation: `vcamcaptured.log` has `gpu prewarm: … skipped`, no new `cameracaptured` crash report, and `successive crashes = 0`. See `Research/Guest/gpu_acceleration.md` ("Heap Textures Have No CPU Layout"). | Y | Y | Y |
| 24 | No binary change: both spawn hooks insert `libbatteryhealthfix.dylib` into `Settings` (`/Applications/Preferences.app/Preferences`) beside SystemHook, where it interposes `IOPSCopyPowerSourcesInfo`, `IOPSCopyPowerSourcesByType`, `IOPSCopyPowerSourcesByTypePrecise`, `notify_register_check` / `notify_get_state` / `notify_cancel` and `MGGetBoolAnswer` — **all bases** | none (new guest dylib `VPhoneGuestComponents/BatteryHealthFix/libbatteryhealthfix.c`, installed at `/usr/lib/libbatteryhealthfix.dylib`) | **Stop Settings → Battery → Battery Health & Charging loading forever.** The VM's battery is Virtualization.framework's synthetic power source (charge and connectivity only), and the health half of a real battery's description comes from AppleSmartBattery, which no VM has. Measured on `iPhone17,3 27.0 (24A435)`: powerd's internal battery description has one `BatteryPacks` entry holding only `PackID`, `Cycle count` and `Transport Type`, no `Battery Service State`/`Flags` or `Maximum Capacity Percent` anywhere, and `com.apple.thermalmonitor.ageAwareMitigationState` stays 0. `-[BatteryHealthUIController specifiers]` asks `-[BatteryHealthUIInformation batteryDataUnavailable]`, which answers YES while `+getManagementState` is 0, while no pack parses (`-[BatteryHealthUIPackInformation initWithPowerSourceInformation:]` drops a pack without service flags and state), or while the overall genuine status is -1 — and it is -1 because `MGGetBoolAnswer("D6/BMDrlb8V3WSiqL8gL+w")` (HasBatteryModuleAuth, `+[PLGestaltUtilities hasBatteryModuleAuth]`) answers 1 on the guest, sending `+genuineBatteryStatusForPackIndex:` to an `AppleBatteryAuth`/`AppleAuthCPRelay` service that does not exist. The controller then shows its global spinner and re-asks on a timer. iPadOS 26.6.2 (23G90) has the same gate in `+[BatteryUIResourceClass batteryDataUnavailable]`, reading the flat keys of `IOPSCopyPowerSourcesByType`'s array, the same notify name and the same MobileGestalt key. The hook answers as a new battery without battery authentication: each internal-battery description is copied with `Battery Service State` 0, `Battery Service Flags` 0, `Maximum Capacity Percent` 100 and `Cycle count` 0 at the top level, and each of powerd's packs (or one new pack) completed with a `BatteryHealth` dictionary of the same three numbers plus `Date of manufacture`/`Date of first use` (the birth time of `/private/var/mobile`, i.e. the restore); keys already present are never replaced, and a description that already has them is passed through. A mitigation state of 0 on a token registered for the age-aware name reads as 2 ("supporting normal peak performance"). HasBatteryModuleAuth reads as no only while `IOServiceGetMatchingService("AppleBatteryAuth")` finds nothing. Interposing reaches the callers because `BatteryUsageUI.bundle` is a standalone Mach-O bound through dyld's interposing table (cache-to-cache calls are not; see `MISFix/MISFixSignature.c`), and inserting it — rather than SystemHook `dlopen`ing it like row 21 — is what puts the table in place before the bundle loads. `vpInsertedLibraryFor` in `Shared/InjectionEnvironment.h` picks it or libmisfix for a path; the two target sets do not overlap. Ships with `system-launchdaemons-boot-environment`; existing guests get it from the vphoned environment update on their next start with the new bundle (offline `cfw update-environment` only replaces libraries a VM already has). launchd starts Settings itself, so it is the launchd hook that inserts it: a guest updated only by that online sync needs one more reboot before launchd runs the new hook, which the update already reports as `reboot_required`; an offline environment update replaces the hook before boot. **Validated (2026-10-04, `bhtest-iphone`, `iPhone17,3 27.0 (24A435)` + cloudOS 26.4):** before, the page showed only the spinner and the two charging switches; after, Maximum Capacity 100%, Peak Performance Capability "Your battery is currently supporting normal peak performance", the two switches, no spinner; the hook's log shows all three answers taken once per process. Re-checked on the final bundle after a reboot of the same guest, with the launchd hook logging `inserted+batteryhealthfix` for each Settings launch. **iPad16,1 26.6.2 (23G90), `bhtest-ipad`, created with the new bundle:** the hook is inserted and stays inert, and Settings runs without a crash; that iPad has no Battery Health page at all, because `-[BatteryUIController setUpBatteryHealthGroup]` offers the row only on an iPhone or, through `+[PLBatteryUIBackendModel shouldShowModifiedHealthController]`, on an iPad that supports the fixed charge limit with the `PerfPowerServices/battery_health_details` feature on — the model decides, as on hardware. The 26.x gate inside the page is from static analysis; no 26.x iPhone image was at hand to run it. | Y | Y | Y |
| 25 | No binary change: `libvcamcaptured.dylib` stops every other thread in `cameracaptured` (`task_threads` + `thread_suspend`, self excluded, one patcher at a time) around each `vcc_patch_word` / `vcc_patch_two_nops` text rewrite — **all bases** | none (guest dylib `VPhoneGuestComponents/VCamCaptured/Image/VCamCodePatch.m`, installed at `/usr/lib/libvcamcaptured.dylib`) | **Stop the hook's own runtime patching from crashing `cameracaptured`, which froze the whole VM when the camera was opened.** The hook rewrites CMCapture shared-cache `__text` through `vm_protect(RW|COPY)` → write → `vm_protect(RX)`; for the whole flip the 16K page carries no execute permission, and the pages it touches (the `-12783` error-code sites, the master-clock gate, the prewarm call) are live XPC server code. Two crash reports caught a thread fetching an instruction in a page mid-flip while the patching thread was in `vcc_patch_word` on the same boot: `ipad-mini-01` `cameracaptured-2026-10-03-045638.ips` died in `__FigFlashlightGetClassID_block_invoke` on `com.apple.coremedia.flashlight(34)-messages`, and `ipad-pro-13` `cameracaptured-2026-10-04-225558.ips` died in `captureSourceServer_handleRegisterNotificationForProprietaryDefaultChangesMessage` on `com.apple.coremedia.capturesource(523)-messages` — the register-for-source-changes XPC that camera enumeration itself sends. Both are `EXC_BAD_ACCESS (SIGBUS)`, `KERN_PROTECTION_FAILURE` on an instruction fetch, in a region snapshotted `r-x/rwx`, `SM=COW` (the rwx max-protection is the flip's fingerprint). The card: with row 23 removing the boot crash, any later `cameracaptured` restart (jetsam, another crash) while a camera client reconnects sends its first XPCs straight into the +3 s reinstall window; one SIGBUS → launchd throttles → every AVCapture client, SpringBoard's main thread included, blocks forever in a synchronous XPC — the "VM 卡死" report of 2026-10-05. The stopped section is exactly two `vm_protect` calls, the word write and `sys_icache_invalidate`: no locks, no allocation, no logging, so a frozen thread holding any userspace lock cannot deadlock the patcher; every `vcc_log` moved after the resume, and a mutex serialises patchers (constructor prewarm patch vs the +3 s install block). All five runtime text-patching call sites (`vcc_scan_and_patch` ×2, the prewarm NOP, the session-hook pair, the source-installation NOPs) funnel through the two fixed functions. Validation: kill-and-relaunch stress with Camera.app auto-reconnecting (the same collision that produced both crashes) produces no new `cameracaptured` crash report, and the boot log's `patch_word OK` lines are unchanged. See `Research/Guest/virtual_camera_transport.md` ("The patch race that froze the whole VM"). | Y | Y | Y |
-| 26 | No binary change: both spawn hooks insert `libdevicehubfix.dylib` into `cryptexd`, the DDI's `dtremotedisplayd` and `dtdeviceinfod`, and `avconferenced`. In `cryptexd` it interposes `fcntl` and reports an `EPERM` from `F_SETPROTECTIONCLASS` as `ENOTSUP`; in `dtremotedisplayd` it interposes `MediaStreamSupportedFeatures.current` (answers `0x184`: a Mac host's `0x8c` without system audio `0x8`, which the VM window already plays, plus video by display ID `0x100` for the iPhone Duo's per-display stream, below) and `CurrentDevice.isCustomerRestricted` (answers false) from the DDI's `CoreDeviceUtilities`; in `dtdeviceinfod` it interposes `MGCopyAnswer` and fills a missing `FramebufferIdentifier` for `phone11` chrome, and stands in for CoreMotion's `CMAngleManager` and `CMDeviceStateManager` (the foldable hinge and pose DeviceHub draws from) and reports the iPhone Duo's inner `CADisplay` (`VPhone Duo Inner`) as integrated at point scale 3; in `avconferenced` it replaces `VCPSideCarMetal`'s two band reads through the Objective-C runtime and runs one synchronous scaler transform before the first read of each frame — **all bases** | none (new guest dylib `VPhoneGuestComponents/DeviceHubFix/libdevicehubfix.c`, installed at `/usr/lib/libdevicehubfix.dylib`) | **Let Xcode 27's DeviceHub show the guest's screen.** On iOS 27 `cryptexd` stages each DDI asset with `open(O_RDWR)` + `fcntl(F_SETPROTECTIONCLASS, 4)` and accepts only `ENOTSUP` as a failure; a guest answers `EPERM` (`set protection class: [1: Operation not permitted]`, `staging failed`), so CoreDevice shows the device as "connected (no DDI)" for good. With the DDI mounted, DeviceHub asks `dtremotedisplayd` for its media stream features and gets 0 from CoreDeviceUtilities' policy: `isCustomerRestricted` (`isProductionFused && !hasInternalOSBuild && hasInternalDDI == false`) refuses every pre-27.0 device, and remote control is limited to iPhone, Apple Watch and Apple TV, which refuses every iPad. `0x8c` is what a Mac host reports for itself (mirrored primary display 0x4, system audio 0x8, display information 0x80). The 27.x iPhone guest reports `phone11` chrome with no `FramebufferIdentifier`, so DeviceHub drew the screen over the bezel corners; the hook answers only that missing key with Xcode's matching mask `4E5532ED-1470-47D1-BDF4-7AA90C26957A` from `/Library/Developer/DeviceKit/chrome_map.plist`. Interposing reaches all of them because `cryptexd`, `dtremotedisplayd`, `dtdeviceinfod` and `CoreDeviceUtilities` are standalone images; the two Swift symbols are weak flat-namespace imports, resolving to nothing outside `dtremotedisplayd`. On 27.x the view also needs row 11's conditional dispatch, or the capture display stays black. A stream that splits each frame into four bands (`tilesPerFrame` 4) deadlocked the host GPU within 16–35 s of screen activity: `VCPSideCarMetal` reads every frame once per band while the paravirtual scaler's write of that frame can still be unregistered on the host, and a band read that registers the first of the frame's two planes before the write and the second after it waits on the write, which waits on it (`Timeout in timestamp wait`, the guest loses its GPU). Waiting for one synchronous transform puts the band reads behind the frame's write. `VideoProcessing` is in the shared cache, so this part is a method replacement, not an interpose. Ships with `system-launchdaemons-boot-environment`; an existing guest gets it from the environment update. Details: `Research/Guest/devicehub_screen_viewing.md`. **Validated (2026-10-06/07):** `iPad16,1 26.6.2 (23G90)`: DeviceHub receives `supportedFeatures: 140`, `MediaStreamStart` succeeds and the view shows the live home screen. `iPhone17,3 27.0.1 (24A446)`: the DDI mounts (`F_SETPROTECTIONCLASS EPERM, reported as ENOTSUP`), and with row 11's conditional dispatch DeviceHub shows the home screen and Settings live, clipped inside the chrome (`Found FramebufferMaskIdentifier from displayInfo`). **Band reads validated (2026-10-07/08, `iPhone17,3 27.0.1 (24A446)`):** a four-band stream with screen activity deadlocked after 18 s without the barrier and ran 60, 180 and 300 s with it without a GPU error; in the host trace of the 180 s run no IOSurface access waited, and every band read of 6258 frames registered after the frame's write was released, a median 0.76 ms after it. Open: latency that grows over a long session and occasional stalls after a cold start, both upstream of the Mac receiver. | Y | Y | Y |
+| 26 | No binary change: both spawn hooks insert `libdevicehubfix.dylib` into `cryptexd`, the DDI's `dtremotedisplayd` and `dtdeviceinfod`, and `avconferenced`. In `cryptexd` it interposes `fcntl` and reports an `EPERM` from `F_SETPROTECTIONCLASS` as `ENOTSUP`; in `dtremotedisplayd` it interposes `MediaStreamSupportedFeatures.current` (answers `0x184`: a Mac host's `0x8c` without system audio `0x8`, which the VM window already plays, plus video by display ID `0x100` for the iPhone Duo's per-display stream, below) and `CurrentDevice.isCustomerRestricted` (answers false) from the DDI's `CoreDeviceUtilities`; in `dtdeviceinfod` it interposes `MGCopyAnswer` and fills a missing `FramebufferIdentifier` for `phone11` chrome, and stands in for CoreMotion's `CMAngleManager` and `CMDeviceStateManager` (the foldable hinge and pose DeviceHub draws from) and reports the iPhone Duo's inner `CADisplay` (`VPhone Duo Inner`) as integrated at point scale 3; in `avconferenced` it replaces `VCPSideCarMetal`'s two band reads through the Objective-C runtime and runs one synchronous scaler transform before the first read of each frame, and replaces `VCScreenCaptureFigCaptureController`'s capture callback so a frame reaches the encoder only when the main display's render-server frame counter moved, or once every 250 ms — **all bases** | none (new guest dylib `VPhoneGuestComponents/DeviceHubFix/libdevicehubfix.c`, installed at `/usr/lib/libdevicehubfix.dylib`) | **Let Xcode 27's DeviceHub show the guest's screen.** On iOS 27 `cryptexd` stages each DDI asset with `open(O_RDWR)` + `fcntl(F_SETPROTECTIONCLASS, 4)` and accepts only `ENOTSUP` as a failure; a guest answers `EPERM` (`set protection class: [1: Operation not permitted]`, `staging failed`), so CoreDevice shows the device as "connected (no DDI)" for good. With the DDI mounted, DeviceHub asks `dtremotedisplayd` for its media stream features and gets 0 from CoreDeviceUtilities' policy: `isCustomerRestricted` (`isProductionFused && !hasInternalOSBuild && hasInternalDDI == false`) refuses every pre-27.0 device, and remote control is limited to iPhone, Apple Watch and Apple TV, which refuses every iPad. `0x8c` is what a Mac host reports for itself (mirrored primary display 0x4, system audio 0x8, display information 0x80). The 27.x iPhone guest reports `phone11` chrome with no `FramebufferIdentifier`, so DeviceHub drew the screen over the bezel corners; the hook answers only that missing key with Xcode's matching mask `4E5532ED-1470-47D1-BDF4-7AA90C26957A` from `/Library/Developer/DeviceKit/chrome_map.plist`. Interposing reaches all of them because `cryptexd`, `dtremotedisplayd`, `dtdeviceinfod` and `CoreDeviceUtilities` are standalone images; the two Swift symbols are weak flat-namespace imports, resolving to nothing outside `dtremotedisplayd`. On 27.x the view also needs row 11's conditional dispatch, or the capture display stays black. A stream that splits each frame into four bands (`tilesPerFrame` 4) deadlocked the host GPU within 16–35 s of screen activity: `VCPSideCarMetal` reads every frame once per band while the paravirtual scaler's write of that frame can still be unregistered on the host, and a band read that registers the first of the frame's two planes before the write and the second after it waits on the write, which waits on it (`Timeout in timestamp wait`, the guest loses its GPU). Waiting for one synchronous transform puts the band reads behind the frame's write. The capture keeps writing frames back at the capture rate for about a second after each draw, so a Home Screen whose live Clock icon draws ten times a second streamed ~42 fps per display, and every frame is encoded on the host (AVE through VideoToolbox paravirtualization, its user-DPB reference dictionaries plist-serialized per frame); the capture callback now passes a frame only when the main display drew (`VPScreenCaptureIdle.c`). `VideoProcessing` is in the shared cache, so this part is a method replacement, not an interpose. Ships with `system-launchdaemons-boot-environment`; an existing guest gets it from the environment update. Details: `Research/Guest/devicehub_screen_viewing.md`. **Validated (2026-10-06/07):** `iPad16,1 26.6.2 (23G90)`: DeviceHub receives `supportedFeatures: 140`, `MediaStreamStart` succeeds and the view shows the live home screen. `iPhone17,3 27.0.1 (24A446)`: the DDI mounts (`F_SETPROTECTIONCLASS EPERM, reported as ENOTSUP`), and with row 11's conditional dispatch DeviceHub shows the home screen and Settings live, clipped inside the chrome (`Found FramebufferMaskIdentifier from displayInfo`). **Band reads validated (2026-10-07/08, `iPhone17,3 27.0.1 (24A446)`):** a four-band stream with screen activity deadlocked after 18 s without the barrier and ran 60, 180 and 300 s with it without a GPU error; in the host trace of the 180 s run no IOSurface access waited, and every band read of 6258 frames registered after the frame's write was released, a median 0.76 ms after it. **Idle frames validated (2026-10-30, `iPhone19,4 27.1 (24A94425)`, macOS 27.0.1 M5 Pro, DeviceHub open):** idle Home Screen 41–45 → 10 fps per display, host VM process 91% → 46–50%, guest `avconferenced` 17% → 7%; scrolling Settings still 54–60 fps. Open: latency that grows over a long session and occasional stalls after a cold start, both upstream of the Mac receiver. | Y | Y | Y |
| 27 | No binary change: both spawn hooks insert `libdevicename.dylib` into `configd` and `lockdownd` (after `libmisfix.dylib`, the first process to take two inserted libraries). In `configd` it interposes `SCDynamicStoreSetMultiple` and `SCDynamicStoreSetValue` and pins `Setup:/System`'s `ComputerName`; in `lockdownd` it interposes `SCPreferencesSetComputerName`, `SCPreferencesSetHostName` and `SCPreferencesSetLocalHostName` and refuses every rename with `kSCStatusAccessError`. Inert unless `/var/db/vphone/devicename.plist` holds a valid `DeviceName` — **all bases** | none (new guest dylib `VPhoneGuestComponents/DeviceName/libdevicename.c`, installed at `/usr/lib/libdevicename.dylib`; vphoned `device.name.get`/`device.name.set`) | **Pin the guest's device name to the VM's name.** The name is `System/System/ComputerName` in `preferences.plist`; configd's PreferencesMonitor (`updateConfiguration`, 24A435 `0x100062098`) flattens it into `Setup:/System` with one `SCDynamicStoreSetMultiple`, and every reader (`SCDynamicStoreCopyComputerName` in lockdownd's `copy_device_name`, MobileGestalt `UserAssignedDeviceName`, `UIDevice.name`) reads the store. The hook rewrites that publication, not the preferences read: the monitor's model-change path (`sub_10006155c`) writes the `kSCPrefSystem` value it reads back to disk, and the publication route leaves `preferences.plist` with the guest's own name. A publication that leaves `Setup:/System` out as unchanged gets the store's value pinned; one that removes it gets the pinned name instead. `vphone-vm` sends the VM's name to vphoned's `device.name.set` after every connect; vphoned writes the file (one rename, 0644) and applies the preferences unchanged, so configd publishes again and the new pin, or the guest's own name when the file is removed, is live within a second. The file outlives a reboot, so configd's first publication at boot carries it. The library re-reads the file when its inode, size or mtime change. lockdownd's `set_device_name` (`0x1000073b0`) is the only caller of the three setters; it logs their failure and still commits, and the `SetValue DeviceName` handler ignores the result, so a host rename reports success and the name stays. The pinned name itself is refused too: lockdownd runs `set_device_name(copy_device_name())` whenever it starts, and a lockdownd restarted after configd published would otherwise write the pinned name and derived host names into `preferences.plist` (seen once). An NVRAM variable was the first channel and never reached the guest: iBoot keeps only the variables it knows (even `vphone-vm`'s `boot-args` write is replaced). Interposing works because both daemons are standalone images. Ships with `system-launchdaemons-boot-environment` and the environment update; takes effect at the next boot. Verified on iOS 27.0 (24A435) on 2026-10-07: pin at boot and live, host rename refused, `vm rename` followed, preferences untouched. CoreDevice (`devicectl`, DeviceHub) reads the name when the device appears and can lag one boot after a rename. Details: `Research/Guest/device_name_pinning.md`. | Y | Y | Y |
| 28 | `cbz`/`cbnz` -> `b` at the four compact-command choices in `-[PGSerializerRenderCommandEncoder drawIndexedPrimitives:…]` (`0x140b0`, `0x1421c`, `0x143b4`, `0x143c4` in 23E5207q) — **cloudOS 26.4** | `AppleParavirtGPUMetalIOGPUFamily` (`/System/Library/Extensions/AppleParavirtGPUMetalIOGPUFamily.bundle`) | **Heap index buffer fix.** The compact command stores `parentResourceOffset + indexBufferOffset` in 16 bits but is chosen by testing `indexBufferOffset` alone, so an index buffer 64 KiB or more into an `MTLHeap` is read from the wrong address and meshes (Unity games) draw with another mesh's indices. Every indexed draw takes the 64-bit command instead. `system-paravirt_gpu-cfw-heap_index_offset`; see "Heap index buffers reach the host whole (2026-10-14)" and `Research/Guest/gpu_acceleration.md` | Y | Y | Y |
| 29 | No binary change: SystemHook calls `setenv("FS_CACHE_REUSE_ON_REBOOT", "1", 0)` first thing in its constructor, in every process it is inserted into, while `/usr/lib/vphone-metal-cache-reuse` exists — **all bases** | none (`VPhoneGuestComponents/SystemHook/SystemHook-vphone.c`; marker written by `cfw install` / `cfw update-environment`) | **Keep the guest's Metal shader caches across a reboot.** `libCoreFSCache` stamps each cache with `kern.bootsessionuuid` and resets it at the next boot unless the variable is set; on the paravirtual GPU CoreAnimation's precompiled pipelines are disabled, so backboardd rebuilt every pipeline after every boot. `system-systemhook-cfw-metal_cache_reuse`; see "Guest Metal caches survive a reboot (2026-10-19)" | Y | Y | Y |
diff --git a/Research/Guest/devicehub_screen_viewing.md b/Research/Guest/devicehub_screen_viewing.md
index ceb5952a..a704e912 100644
--- a/Research/Guest/devicehub_screen_viewing.md
+++ b/Research/Guest/devicehub_screen_viewing.md
@@ -698,6 +698,61 @@ opening out of closed is SpringBoard's fold effect on one shared panel; see
panel". DeviceHub no longer gets the guest's system audio (`0x184`): the VM
window plays it, and DeviceHub played it again.
+## Encoding only when the screen draws
+
+An open DeviceHub viewer cost the host about a core while the guest sat on
+its Home Screen (iPhone19,4, iOS 27.1 24A94425, macOS 27.0.1 on an M5 Pro):
+the VM process at 91%, against 14% with DeviceHub closed. The guest itself
+used about a third of a core (`avconferenced` 17%).
+
+The encoder runs on the host. `avconferenced` calls VideoProcessing's
+`VCPCompressionSessionCreate` directly (AVConference `EncCreateHandle`), and
+that real-time encoder wraps
+`paravirtualized:com.apple.videotoolbox.videoencoder.ave.hevc`, the host's AVE
+hardware. The real-time encoder runs AVE in user-DPB mode
+(`VCPCompressionAllocateDPB`, `kVTEncodeFrameOptionKey_DPB_ReferenceFrames_*`),
+so each encoded frame carries nested reference-frame dictionaries. A host
+`sample` of the VM process showed, per stream, an `AVE_UCRecv` thread at
+about 36% in `VTParavirtualizationHostEncoderSessionEmitEncodedFrame →
+VTParavirtualizationCreateSerializedAtomDataBlockBufferForSampleBuffer →
+CFPropertyListCreateData` and about 22% more in a synchronous wait for the
+guest's reply, plus the `AppleM2ScalerDevice` queue at about 39%. No guest
+setting leaves user-DPB mode: it is how the real-time encoder controls its
+rate. The Mac decodes with `VCPDec`, about 1.2 ms a frame.
+
+The cost scales with the frame rate, and the rate did not follow the screen.
+The Mac's `avconferenced` logs each stream's input rate
+(`[VCPDecStatsMonitor] … Input_fps=`): the idle Home Screen streamed about
+42 fps on each of the Duo's two displays. The capture is FigVirtualDisplay's
+writeback clone of each lit display in backboardd
+(`IOMFBDisplay::clone_update`); after the screen draws it keeps writing frames
+back at the capture rate for about a second, and a still screen sends nothing
+(Settings left alone dropped to 0 fps, with or without the change below). The
+live Clock icon on the Home Screen draws about ten times a second
+(`CARenderServerGetFrameCounterByIndex(0, 0)`, the main display), so that
+second never ends.
+
+`libdevicehubfix` (`VPScreenCaptureIdle.c`) replaces
+`-[VCScreenCaptureFigCaptureController
+screenCaptureController:didReceiveSampleBuffer:transformFlags:]` in
+`avconferenced`: a frame passes when the main display's frame counter moved
+since the stream's last frame, and otherwise once every 250 ms. A frame that
+carries a change can arrive after the counter moved; the 250 ms frame bounds
+that delay and lets the encoder refine a still image. On the Duo the main
+display is the panel the inner display is shown on, so one counter covers both
+streams.
+
+| sbdisplay-duo, DeviceHub open | before | after |
+|---|---|---|
+| idle Home Screen, input rate per display | 41–45 fps | 10 fps |
+| idle Home Screen, host VM process | 91% | 46–50% |
+| idle Home Screen, guest `avconferenced` | 17% | 7% |
+| scrolling Settings, input rate per display | 58–60 fps | 54–60 fps |
+| Settings left alone | 0 fps after ~1 s | 4 fps for ~1 s, then 0 |
+
+A first version also passed eight frames after each change; with the Clock
+icon changing every 100 ms that never ran out (30 fps with two frames).
+
## Other open items
- iPadOS 27 refuses pairing (see Validation), so DeviceHub is untested there.
@@ -707,6 +762,10 @@ window plays it, and DeviceHub played it again.
- The band-read barrier was tested on the 27.0.1 iPhone only. It depends on
two private `VCPSideCarMetal` method names; if they change, the library logs
that they were not found and banded streams can deadlock again.
+- The idle-frame filter depends on the private
+ `VCScreenCaptureFigCaptureController` callback and
+ `CARenderServerGetFrameCounterByIndex`; if either is missing, the library
+ logs it and every captured frame is encoded, as before.
## Measuring
diff --git a/VPhoneGuestComponents/DeviceHubFix/VPScreenCaptureIdle.c b/VPhoneGuestComponents/DeviceHubFix/VPScreenCaptureIdle.c
new file mode 100644
index 00000000..ff8f9768
--- /dev/null
+++ b/VPhoneGuestComponents/DeviceHubFix/VPScreenCaptureIdle.c
@@ -0,0 +1,130 @@
+// VPScreenCaptureIdle.c — avconferenced encodes a screen frame only when the
+// screen drew.
+//
+// DeviceHub's stream is avconferenced screen capture: FigVirtualDisplay
+// clones each lit display in backboardd, the clone writes frames back, and
+// VCScreenCaptureFigCaptureController hands each one to VideoProcessing's
+// real-time encoder. After the screen draws, the writeback keeps coming at the
+// capture rate for about a second before it stops; a still screen sends
+// nothing. The live Clock icon on the Home Screen draws about ten times a
+// second, so it never stops: an idle Home Screen streamed about 42 fps per
+// display, two streams on an iPhone Duo. Measured on iOS 27.1 (24A94425).
+//
+// On a guest the encode is the expensive part, and it runs on the host: the
+// encoder is the host's AVE through VideoToolbox paravirtualization, and in
+// the real-time encoder's user-DPB mode every frame carries reference-frame
+// dictionaries the host serializes as binary property lists, then waits for
+// the guest's reply. With a DeviceHub viewer open on the idle Home Screen the
+// VM process used 91% of a core (macOS 27.0.1, M5 Pro), 14% without a viewer.
+//
+// So a frame reaches the encoder when the render server's frame counter for
+// the main display (CARenderServerGetFrameCounterByIndex) moved since the last
+// one, and otherwise once per keep-alive interval. A frame written back with a
+// change can arrive after the counter it moved; the keep-alive frame, which
+// also lets the encoder refine a still image, bounds that to 250 ms, and the
+// writeback runs for about a second after the last draw. On an iPhone Duo the
+// main display is the panel the inner display is shown on, so the counter
+// covers both streams. Measured on the idle Home Screen: 10 fps per display,
+// VM process 50%; scrolling Settings still streams 54–60 fps.
+
+#include <dlfcn.h>
+#include <fcntl.h>
+#include <objc/runtime.h>
+#include <pthread.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <time.h>
+#include <unistd.h>
+#include <CoreFoundation/CoreFoundation.h>
+
+#define VP_QUARTZ_CORE "/System/Library/Frameworks/QuartzCore.framework/QuartzCore"
+#define VP_AV_CONFERENCE "/System/Library/PrivateFrameworks/AVConference.framework/AVConference"
+
+// The interval of the frames that pass while the counter stands still.
+#define VP_KEEP_ALIVE_NANOSECONDS 250000000ull
+// Capture controllers followed at once: one per streamed display.
+#define VP_CONTROLLERS 4
+
+// MARK: - Log (same file the rest of the dylib writes)
+
+static void vpIdleLog(const char *format, const char *a, const char *b) {
+ int fd = open("/var/mobile/Library/Caches/vphone-devicehubfix.log",
+ O_WRONLY | O_CREAT | O_APPEND | O_CLOEXEC, 0666);
+ if (fd < 0)
+ return;
+ fchmod(fd, 0666);
+ dprintf(fd, "pid=%d ", getpid());
+ dprintf(fd, format, a, b);
+ dprintf(fd, "\n");
+ close(fd);
+}
+
+// MARK: - The capture callback
+
+typedef void (*VPDidReceiveSampleBuffer)(id, SEL, id, CFTypeRef, uint64_t);
+
+static int (*vpFrameCounter)(unsigned int server, unsigned int display);
+static VPDidReceiveSampleBuffer vpOriginalDidReceiveSampleBuffer;
+
+static struct {
+ const void *controller;
+ int counter;
+ uint64_t passed;
+} vpControllers[VP_CONTROLLERS];
+static pthread_mutex_t vpControllersLock = PTHREAD_MUTEX_INITIALIZER;
+
+static bool vpShouldPass(const void *controller) {
+ int counter = vpFrameCounter(0, 0);
+ uint64_t now = clock_gettime_nsec_np(CLOCK_UPTIME_RAW);
+ bool pass = true;
+ pthread_mutex_lock(&vpControllersLock);
+ int slot = -1;
+ for (int index = 0; index < VP_CONTROLLERS && slot < 0; index++)
+ if (vpControllers[index].controller == controller)
+ slot = index;
+ // A new stream takes the slot that passed a frame longest ago (an empty
+ // one never did); a stream that ended leaves its slot behind.
+ if (slot < 0) {
+ slot = 0;
+ for (int index = 1; index < VP_CONTROLLERS; index++)
+ if (vpControllers[index].passed < vpControllers[slot].passed)
+ slot = index;
+ }
+ if (vpControllers[slot].controller != controller || counter != vpControllers[slot].counter) {
+ vpControllers[slot].controller = controller;
+ vpControllers[slot].counter = counter;
+ } else {
+ pass = now - vpControllers[slot].passed >= VP_KEEP_ALIVE_NANOSECONDS;
+ }
+ if (pass)
+ vpControllers[slot].passed = now;
+ pthread_mutex_unlock(&vpControllersLock);
+ return pass;
+}
+
+static void vpDidReceiveSampleBuffer(id self, SEL command, id capture, CFTypeRef sampleBuffer, uint64_t flags) {
+ if (sampleBuffer && !vpShouldPass(self))
+ return;
+ vpOriginalDidReceiveSampleBuffer(self, command, capture, sampleBuffer, flags);
+}
+
+__attribute__((constructor)) static void vpInstallScreenCaptureIdle(void) {
+ if (strcmp(getprogname(), "avconferenced") != 0)
+ return;
+ vpFrameCounter = dlsym(dlopen(VP_QUARTZ_CORE, RTLD_LAZY | RTLD_LOCAL), "CARenderServerGetFrameCounterByIndex");
+ dlopen(VP_AV_CONFERENCE, RTLD_LAZY | RTLD_LOCAL);
+ Class controller = objc_getClass("VCScreenCaptureFigCaptureController");
+ Method receive = controller ? class_getInstanceMethod(controller, sel_registerName(
+ "screenCaptureController:didReceiveSampleBuffer:transformFlags:"))
+ : NULL;
+ if (!vpFrameCounter || !receive) {
+ vpIdleLog("screen capture idle frames %s, %s", "not found", "every captured frame is encoded");
+ return;
+ }
+ vpOriginalDidReceiveSampleBuffer =
+ (VPDidReceiveSampleBuffer)method_setImplementation(receive, (IMP)vpDidReceiveSampleBuffer);
+ vpIdleLog("screen capture idle frames %s, %s", "found", "frames pass when the screen drew");
+}
diff --git a/VPhoneGuestComponents/DeviceHubFix/libdevicehubfix.c b/VPhoneGuestComponents/DeviceHubFix/libdevicehubfix.c
index 2d527a0d..d34f1091 100644
--- a/VPhoneGuestComponents/DeviceHubFix/libdevicehubfix.c
+++ b/VPhoneGuestComponents/DeviceHubFix/libdevicehubfix.c
@@ -4,7 +4,8 @@
// dtremotedisplayd. On a vphone guest it needs four fixes, one per process:
// cryptexd mounts the DDI, dtremotedisplayd reports media stream features,
// dtdeviceinfod supplies the framebuffer mask, and avconferenced keeps a
-// banded stream from deadlocking the host GPU.
+// banded stream from deadlocking the host GPU. avconferenced also encodes a
+// frame only when the screen drew (VPScreenCaptureIdle.c in this folder).
//
// ## cryptexd: the DDI does not mount on iOS 27
//
@@ -117,10 +118,11 @@
// ones: dyld binds their imports through the interposing table.
// CoreDeviceUtilities is on the DDI and not in the SDK, so its two symbols are
// weak flat-namespace imports; outside dtremotedisplayd they resolve to nothing
-// and dyld skips those entries. VideoProcessing is in the shared cache, so the
-// band reads are replaced through the Objective-C runtime instead. The spawn
-// hooks insert this into those four processes only — `vpIsDeviceHubFixTarget`
-// in Shared/InjectionEnvironment.h.
+// and dyld skips those entries. VideoProcessing and AVConference are in the
+// shared cache, so the band reads and the capture callback are replaced
+// through the Objective-C runtime instead. The spawn hooks insert this into
+// those four processes only — `vpIsDeviceHubFixTarget` in
+// Shared/InjectionEnvironment.h.
#include <dlfcn.h>
#include <errno.h>
@@ -359,7 +361,7 @@ __attribute__((constructor)) static void vpInstallScalerBarrier(void) {
vpLog("VCPSideCarMetal band reads %s, %s", "found", "they wait for the scaler");
}
-int vphone_devicehubfix_version(void) { return 6; }
+int vphone_devicehubfix_version(void) { return 7; }
// The CMAngleManager stand-in for dtdeviceinfod lives in VPCMAngleManager.m
// (same dylib): it swaps the manager's whole ObjC surface and feeds the