Skip to content

Run a virtual iPhone on an Apple Silicon Mac.

Swift

Collect Duo fixes and optional input, external display, and host camera support

Steven Troughton-Smithbb8e71c20d10
diff --git a/Research/0_binary_patch_comparison.md b/Research/0_binary_patch_comparison.md
index 7399847c..41cb4f10 100644
--- a/Research/0_binary_patch_comparison.md
+++ b/Research/0_binary_patch_comparison.md
@@ -5612,3 +5612,23 @@ The host VM view now derives its input geometry from the active Duo viewport (`a
> **Duo display: two displays only (2026-10-29):** `system-springboard-cfw-fold_display` (now titled "Foldable inner display") keeps only the two-display path, and it is always on for iPhone19,4: `libvphonefolddisplay.dylib` checks `hw.machine` in backboardd and SpringBoard instead of the `vphone-duo-inner-display` marker. backboardd: the 2853x2007 `CAWindowServerVirtualDisplay` added from `-[CAWindowServer _detectDisplays]` (server-ready fallback), tagged `unknown`; `DeviceSurfaceType` 1 for the primary and 2 for the inner display through `-[CAWindowServerDisplay extendedProperties]`; `BKHIDDirectTouchEventProcessor` digitizer routing and the 270° position turn while the inner display is lit. SpringBoard: the CADisplay type/pointScale/isExternal adapter with the FrontBoard classification scope, `mainLike` in the inner identity's XPC encoding, the inner screen's idiom, the shared scanout (`-[CADisplayStateControl transitionToDisplayState:withCompletion:]`), the compositor, the content mode resolver hand-over (the resolver is now recorded by a pass-through `-[SBFDisplayContentModeResolver _startResolvingContentModeOfDisplay:]`), and the per-screen `STUIStatusBarGetVisualProviderClassForScreen` detour. Removed hooks: the Main configuration transformer, monitor and identity observers, `SBWindowScene surfaceType`/`sbhScreenType`, `SBWindow _shouldResizeWithScene`, the scene geometry callback, the clock and time-item adapters, the scene status policy, the Main status lookup and status profile factories, `SBHScreenTypeFor*`, `_os_feature_enabled_impl` (`TraitsArbiterWallpaperTokyoDrift`), `PF_IS_PAD_DEVICE`, `_SBWallpaperSecureWindow setHidden:`, the quick-action glyph subclass, the `UIWindow` target-window fallback, the `SBSDeviceDeterminedConfiguration`/`CSProminentLayoutController` loggers, and every `com.vphone.display.*` notification. `libvphoneposterprobe.dylib` (the poster orientation follower) is no longer built, staged, listed in the guest environment, or loaded by SystemHook. Details in `Research/Guest/iphone_duo_bringup.md`.
> **Duo display: flicker, capture and rotation (2026-10-29):** in `system-springboard-cfw-fold_display`, the compositor is now a layer at the top of Main's `UIRootSceneWindow` (points) instead of a display-level `CAContext` (pixels), its hosts sync from a main run loop observer before Core Animation's commit and change only when a value changes, and its per-frame damage alternates two clear colors. New SpringBoard hook: `+[CMMagicPoseManager orientationForMagicPose:deviceSurface:]` answers with `+[CMDeviceStateEvent propertyAForConfigType:propertyA:propertyB:propertyC:]` directly, because the guest has no `CMMagicPoseManager` shared manager and every panel read ambiguous, so neither panel rotated. vphoned's `display.rotation` on a foldable sets the pose orientation for the lit panel. Details in `Research/Guest/iphone_duo_bringup.md`.
+
+> **Local Duo inner-screen corner diagnostic (2026-10-29):** The existing
+> FoldDisplay guest component checks UIScreen corner getter ABIs and, only
+> for its recorded inner display identity, fills a zero radius from the main
+> screen's native metric. Nonzero native radii and other displays pass through.
+> This is a source-level experiment in the existing component, with no new
+> firmware patch or preset. See the expanded Now Playing entry in
+> `Research/Guest/iphone_duo_bringup.md` for evidence and deployment outcome.
+
+> **Local optional external monitor prototype (2026-10-29):** Existing Duo
+> FoldDisplay injection gains an explicit, off-at-boot 1080p/4K virtual monitor
+> and a named external-display adapter. Existing inner-only policies now match
+> the recorded inner identity. Guest RPC captures the external display itself
+> for a separate host window. No new binary patch, preset or hardware graphics
+> port is introduced. See `Research/Guest/virtual_external_display.md` for
+> implementation boundaries and runtime results.
+> Final implementation prepares two disabled render slots before SpringBoard
+> discovery and refreshes their existing CADisplay objects after configuration.
+> Native Broadcasts output, live 1080p-to-4K switching, and disconnect/reconnect
+> passed on `2.14.0-local.b0297e92`; boot remains disconnected by default.
diff --git a/Research/Guest/apple_intelligence_eligibility.md b/Research/Guest/apple_intelligence_eligibility.md
new file mode 100644
index 00000000..be539aaa
--- /dev/null
+++ b/Research/Guest/apple_intelligence_eligibility.md
@@ -0,0 +1,103 @@
+# Apple Intelligence eligibility on the Duo VM — 2026-10-29
+
+Read-only investigation on iPhone-Duo, iPhone19,4 / 27.1 (24A94425),
+cloudOS 26.4 (23E5207q), local bundle 2.14.0-local.e1ece920.
+The AI services were restored before probing; core model/runtime daemons run.
+No region, MobileGestalt cache, eligibility file or device-tree value changed.
+
+A temporary Foundation executable called MGCopyAnswer in a separate process:
+
+| Key | Answer |
+| --- | --- |
+| ProductType | iPhone19,4 |
+| DeviceClass | iPhone |
+| HardwarePlatform | vresearch1 |
+| IsVirtualDevice | true |
+| HasAppleNeuralEngine | true |
+| AppleNeuralEngineSubtype | 784 |
+| generative-model-systems | false |
+| I3lidq+FVH2CeOi/ARa0tg (hashed form) | false |
+| DeviceSupportsGenerativeModelSystems | false |
+| SiriMode | 0 |
+
+The key names/hashes were cross-checked against PoomSmart/MGKeys mapping.h.
+IORegistryEntryCreateCFProperty at IODeviceTree:/product returned assistant=1,
+exclaves-enabled=0 and no vm-supports-generative-models property. The same VM
+property was absent at the root. Its absence is a possible upstream cause,
+not a demonstrated causal explanation of MobileGestalt's false answer.
+
+The GREYMATTER eligibility record has overall answer 2 and its
+OS_ELIGIBILITY_INPUT_GENERATIVE_MODEL_SYSTEM input is 2; the device class,
+languages, language match, region-code, shared-iPad and external-boot-drive
+inputs are 3. Billing input is 7 and country-location input is 1. Numeric
+country statuses were not interpreted as proven additional blockers.
+SIRI_MODE also has input DEVICE_SIRI_MODE=2. Siri AI's saved isEnabled=false
+is an outcome/current setting, not proof of the upstream cause.
+
+A second temporary executable used Apple's public FoundationModels API.
+SystemLanguageModel.default.availability and
+PrivateCloudComputeLanguageModel().availability both returned
+unavailable(deviceNotEligible). No prompt or generation request was sent.
+
+Conclusion: local device/model capability eligibility is a confirmed blocker,
+independent of any later online attestation failure. A successful account or
+region change alone has not been shown to fix it. The next isolated experiment
+would change the capability answer at its source and re-query eligibility;
+that is not yet authorized or performed, nor does it establish working model
+execution or Private Cloud Compute access.
+
+Probe sources/results are in .vphone-setup/ai-diagnosis in the working folder.
+Temporary guest probe executables were removed after reading the results.
+
+## Follow-up capability and inference experiments
+
+The user authorized fixing the eligibility gates. Original MobileGestalt cache
+and eligibility output files were saved under .vphone-setup/ai-diagnosis.
+Setting CacheExtra keys A62OafQ85EJAiiqKn4agtg and
+I3lidq+FVH2CeOi/ARa0tg to true made the corresponding MG answers true, survived
+reboot, and changed GREYMATTER's GENERATIVE_MODEL_SYSTEM status from 2 to 3.
+The overall answer then became 1, with country billing=7 and location=1.
+A SiriMode=2 numeric cache override yielded unstable nonsensical values in the
+probe, including after reboot and with an explicit NULL MGCopyAnswer options
+argument; it was removed. Do not reuse that override without understanding its
+encoding on 27.1.
+
+A narrow cached-result test set only GREYMATTER and FOUNDATION_MODELS answers
+to 4 and their billing/location statuses to 3. Both public FoundationModels
+availability APIs then returned available. A local, nonpersonal generation
+request still failed with SensitiveContentAnalysisML 15, ultimately
+com.apple.tokengeneration 5. Capture from HostInferenceProviderService proved:
+`No remote service with name com.apple.modelmanager.remote`.
+Modelmanagerd selected host-inference for instruct_300m assets. The unrelated
+analyticsd bootstrap error in modelmanagerd's trace was not the inference cause.
+
+Read-only inspection of the Mac's stock modelmanagerd LaunchDaemon shows that
+RemoteServices entry gated by ModelManager/virtualization and
+RequireEntitlement=AppleInternal. No host service, flags, entitlements or
+security policy was changed. Disabling only the guest's existing
+ModelManager/virtualization flag and rebooting still selected HostInferencePro
+and failed the same generation request. The user reported that the new UI did
+appear in the last test configuration, but was insufficient to enable AI.
+Thus UI eligibility was partly overcome; a functioning inference backend was
+not demonstrated.
+
+The temporary guest ModelManager flag, MobileGestalt overrides and eligibility
+output overrides were restored from backups and the VM rebooted. AI/Siri/CarKit
+service exceptions remain. Test sources/configurations/results are preserved
+for reproducibility. No firmware downloaded.
+
+## Snapshot issue set aside at user's request
+
+Snapshot metadata was generated, but existing .ktx files had UF_COMPRESSED,
+compression header type 16, and open/read failed with EIO. This is stronger
+evidence of a hybrid-userland/kernel compression incompatibility than a trim
+problem. A snapshot-directory-only exclusion in AppleFSCompression's system
+rules did not demonstrate a fix; the original rules file was restored when the
+user asked to set thumbnail work aside. No snapshot cache was deleted.
+
+Final cleanup verification: a guest reboot did not reconnect the control agent;
+a clean Launchpad stop/start recovered it, with vphoned answering in 5 seconds.
+The guest is running, unlocked, and / is sealed/read-only again. The original
+compression rules and ModelManager virtualization=true were verified; the three
+experimental MobileGestalt keys are absent. Siri, CarKit and the 21 AI-related
+service exceptions remain enabled. See rollback-verified.json for the check.
diff --git a/Research/Guest/carplay-service-allow.json b/Research/Guest/carplay-service-allow.json
new file mode 100644
index 00000000..33603536
--- /dev/null
+++ b/Research/Guest/carplay-service-allow.json
@@ -0,0 +1,13 @@
+[
+ "com.apple.carkitd",
+ "com.apple.SiriTTSTrainingAgent",
+ "com.apple.assistant_cdmd",
+ "com.apple.assistant_service",
+ "com.apple.assistantd",
+ "com.apple.corespeechd",
+ "com.apple.siri.sirireaderd",
+ "com.apple.siriinferenced",
+ "com.apple.siriknowledged",
+ "com.apple.speechmaintenanced",
+ "com.apple.systemvoiceassistant"
+]
diff --git a/Research/Guest/carplay_connection_diagnosis.md b/Research/Guest/carplay_connection_diagnosis.md
new file mode 100644
index 00000000..8f9179b3
--- /dev/null
+++ b/Research/Guest/carplay_connection_diagnosis.md
@@ -0,0 +1,102 @@
+# CarPlay connection investigation — 2026-10-29
+
+## Reproducing the Siri/CarPlay service repair
+
+Restoring CarKit and Siri is an important part of the working configuration,
+separate from the external-digitizer coordinate fix. The observed sequence was:
+restore carkitd, fully restart the Mac clients, then restore Siri jobs to obtain
+working CarPlay UI. A discoverable iPhone alone did not establish a functioning
+session. This does not establish working Siri AI, online account attestation or
+the smallest possible service dependency set.
+
+`carplay-service-allow.json` records carkitd and the ten Siri jobs enabled during
+the successful investigation. Keep it separate from the 21 services restored
+for the later Apple Intelligence experiment. It is a tested configuration set,
+not a proposed unconditional global trimming policy or a proven minimal list.
+
+To prepare an opt-in repair for an existing service profile, from the repository
+root (replace the machine name):
+
+```sh
+machine='YOUR_MACHINE'
+vphone-launchpad-cli guest rpc "$machine" services.profile '{}' > profile-before.json
+jq -e --slurpfile keep Research/Guest/carplay-service-allow.json '
+ .result.record as $record |
+ if $record == null then error("No recorded profile; inspect services.profile first")
+ else {
+ profile: $record.profile,
+ groups: $record.groups,
+ allow: ((($record.allow // []) + $keep[0]) | unique),
+ force: true
+ } end
+' profile-before.json > carplay-profile.json
+```
+
+Inspect the generated request before applying it. It retains the profile, groups
+and all existing exceptions; it adds only the recorded CarPlay/Siri exceptions.
+Applying a service profile changes guest launchd overrides:
+
+```sh
+vphone-launchpad-cli guest rpc "$machine" services.profile.apply "$(cat carplay-profile.json)"
+```
+
+Check the reply's `failed` array, restart the guest cleanly, then fully quit and
+reopen CarPlay Simulator and reconnect. Query `services.status` for carkitd and
+the Siri jobs; jobs may be on demand, so enabled/loaded is distinct from running.
+Retain `profile-before.json` locally for rollback; do not commit guest dumps.
+The profile API only owns its own overrides, so unrelated manually disabled
+jobs need separate diagnosis. No firmware download or host security change is
+part of this repair.
+
+Upstream review should decide whether to keep these jobs by default, split
+CarPlay/Siri into optional trimming groups, or expose an explicit compatibility
+option. This collection documents reproduction and leaves that product-policy
+decision open rather than silently changing defaults for every guest.
+
+## Original investigation
+
+Target: iPhone-Duo, iPhone19,4 / iOS 27.1 24A94425 / cloudOS 26.4
+23E5207q. Local bundle 2.14.0-local.77724db8.
+
+The slim guest profile disabled com.apple.carkitd. The standalone Simulator's
+initial attempt successfully paired through AMDevice but failed its
+AMDeviceSecureStartService connection to port 50100 with ECONNREFUSED /
+0xe8000065. CarKit was restored by adding com.apple.carkitd to the profile's
+allow list, preserving the six Store/account exceptions and all groups.
+Loading its original LaunchDaemon plist and starting the job succeeded;
+PID 523 remained running, and no related crash report was found.
+
+Subsequent attempts were logged by DeviceHub, including the user's retry at
+18:36:21. CreateServiceSocketActionDeclaration failed with CoreDeviceError
+1001, missing com.apple.dt.servicesocket.create while requesting
+com.apple.carkit.remote-iap.service. The standalone process logged no new
+session attempt during this period. This does not establish which window the
+user was operating or that the standalone app forwards to DeviceHub.
+
+A fresh read through devicectl device info details at approximately 18:37
+reported ddiServicesAvailable=true and advertised Create Service Socket
+(com.apple.dt.servicesocket.create). The DDI contents exist at
+/System/Developer. This contradicts the earlier operation's missing-capability
+view and suggests cached discovery state. Full restart of the Mac clients,
+then another connection attempt, remains to be tested. No need for a new IPSW
+or restoration has been established. UI inspection of the standalone app
+repeatedly timed out; no settings were changed in that app.
+
+## Working connection and external touch rotation
+
+The user confirmed CarPlay connected after fully restarting the Mac clients.
+Siri's disabled launch jobs were subsequently allowed in the service profile
+and loaded; the user confirmed CarPlay UI worked.
+
+CarPlay then showed rotated/swapped touch coordinates while the Duo was open.
+The user confirmed folding the Duo closed corrected CarPlay touch. The
+FoldDisplay direct-touch hook applied `(x,y) -> (1-y,x)` to every digitizer
+whenever the inner panel was lit, including external display digitizers.
+The local fix consults the original digitizer-display UUID resolver and only
+rotates an empty UUID, matching the existing built-in digitizer rerouting
+condition. A nonempty external display UUID keeps its coordinate space.
+Deployed as `2.14.0-local.e1ece920`, preserving the inner 120 Hz and
+scaled-window touch fixes. The Release build, bundle admission, installation
+preflight and guest boot passed. Siri/CarKit profile exceptions persisted.
+After restart, the user confirmed CarPlay touch appeared to work with the
+Duo open. The VM was left running in the open posture.
diff --git a/Research/Guest/display_refresh_rate.md b/Research/Guest/display_refresh_rate.md
index 6c0b8703..aa2dd48e 100644
--- a/Research/Guest/display_refresh_rate.md
+++ b/Research/Guest/display_refresh_rate.md
@@ -1,5 +1,32 @@
# Guest Display Refresh Rate
+## Duo inner display: local investigation, 2026-10-29
+
+Historical observation from a release-based local build. The main-based review
+candidate keeps upstream's explicit 120 Hz digital-mode setup instead of this
+update-rate-only experiment. Exceeding 60 in the window counter did not prove
+that ordinary apps saw a 120 Hz inner UIScreen. See the later digital-mode
+investigation in iphone_duo_bringup.md for the stronger measurements.
+
+On the iPhone19,4 27.1 (24A94425) guest with cloudOS 26.4
+(23E5207q), the cover display reached 119 fps while scrolling Settings,
+but the unfolded inner display stayed at 59–60 fps, as counted by the
+VM window's framebuffer callback counter. Low Power Mode was off.
+`UIScreen.mainScreen.maximumFramesPerSecond` returned 120 from a guest
+daemon session, which described the cover and did not establish the inner
+display's frame rate.
+
+`FoldDisplayBackboard.m` creates the separate `CAWindowServerVirtualDisplay`
+for the inner panel with `kCAVirtualDisplayUpdateRate = 60`. The local fix
+sets that option to 120 to match the cover's ProMotion timing. This changes
+a guest component, requiring an environment redeployment and reboot rather
+than a kernel update or firmware restore.
+
+After deployment as `2.14.0-local.77724db8`, the unfolded Settings scroll
+test produced window-counter samples of 82 and 111 fps (previously 59–60).
+The release build and bundle admission passed. The previous installer/PAC
+and scaled touch-coordinate repairs were retained in the same build.
+
A vphone guest renders at 60 Hz whatever the host display does. This note
records where the 60 comes from, why no Virtualization setting changes it, and
the kernel patch `kernel-cfw-display_promotion` (titled ProMotion) that makes the
diff --git a/Research/Guest/duo_session_handoff.md b/Research/Guest/duo_session_handoff.md
new file mode 100644
index 00000000..e823a0fc
--- /dev/null
+++ b/Research/Guest/duo_session_handoff.md
@@ -0,0 +1,106 @@
+# Duo fixes and optional host peripherals: review handoff
+
+This is a collection of changes developed together on an existing iPhone19,4
+guest: iOS 27.1 (24A94425), cloudOS 26.4 (23E5207q). The tested source baseline
+was release 2.14.0, commit c503e1c51be0f2249bf4b5b19628cf8e33c00001.
+The collection is intended for review and separation into suitable upstream
+changes. It is not a claim of support across every product, firmware or host.
+
+The local main-based review candidate uses upstream commit
+0499378c7c46b1bfab5f1d9c3ffdb158e24bb083. It preserves the upstream digital-mode
+120 Hz setup, stale digitizer UUID recovery and tap-to-wake declarations.
+The original tested release-based patch is retained separately. This adapted
+candidate has not been installed into the guest; do not attribute the original
+runtime acceptance results to the adapted combination without retesting.
+The adapted Release VPhone build passed (295.706 seconds), its diff passes
+`git diff --check`, and the native-PAC reference comparison passed again.
+The CarPlay request-generation example was checked with an isolated fixture
+to retain existing groups/exceptions; it was not reapplied to the running VM.
+
+## Source changes and evidence
+
+| Area | Implementation | Observed validation | Limits / review focus |
+| --- | --- | --- | --- |
+| Native X1 detours | MISFixDetour.c relocates a first-instruction PACIBSPPC with the original PC and SP; refuses later occurrences | 4,096 native signing comparisons and 64 guest MIS validations; user subsequently installed their app through Xcode | Audit scratch registers, instruction encoding and relocation coverage on other CPUs |
+| Store downloads | Exact six-job Store/account allow list and profile-preserving reproduction workflow in installd_and_store_download_repair.md | Maps downloaded, registered and launched; Freeform downloaded and registered | Per-guest configuration; third-party FairPlay, account login and Configurator's App Store-list workflow remain separate |
+| Resized-window touch | VPhoneDisplayGeometry and VMView account for the active viewport at the current backing size | 14 DisplayGeometryTests passed; user confirmed touch fix | Retest cover/inner, rotation, fullscreen and Retina scaling together |
+| Inner refresh rate | Historical local experiment changes the virtual display update rate from 60 to 120 | Window counter exceeded 60, with samples up to 111 fps | Upstream main already has an explicit digital-mode implementation; preserve it when integrating |
+| Siri / CarPlay services | Recorded opt-in service allow list and profile-preserving reproduction commands in carplay_connection_diagnosis.md | User confirmed CarPlay connection and UI after restoring CarKit/Siri and restarting Mac clients | Per-guest configuration; default trimming-policy change remains an explicit upstream decision; the list is not proven minimal |
+| CarPlay touch | FoldDisplayBackboard rotates only the built-in digitizer, preserving nonempty external display UUIDs | User confirmed CarPlay works with Duo open | Retain upstream stale-digitizer UUID recovery; retest after backboardd restart |
+| Expanded activity corners | FoldDisplayAdapter fills zero inner UIScreen corner radius from the main screen's original value | SpringBoard native/main/effective values 0/59/59; Broadcasts panel visually fixed | Guarded private-API experiment; validate cover metric suitability and avoid masking future native geometry |
+| iPhone pointer opt-in | Boolean mobile preference advertises pointer buttons/scroll; host capture is explicit and initially off | Guest pointer RPC accepted; user confirmed capture/release | Advertising input does not promise an iOS cursor or establish all inner-display mappings |
+| Optional mouse capture | Device menu; Control–Option–M releases; focus loss/window close also release | User confirmed capture and release; clicking does not enable it | Check relative motion, held buttons, scroll and accessibility interactions |
+| External monitor | Stable disabled 1080p/4K render slots before SpringBoard discovery; tag selected slot as AirPlay; refresh existing CADisplay objects | Broadcasts native external scene at both resolutions, live switch, disconnect/reconnect; default off | Duo only, private APIs, software JPEG preview rather than hardware scanout; app policy controls output |
+| Live host camera | AVFoundation video source feeds existing BGRA/VSOCK transport; Launchpad camera privacy metadata and entitlement | Default Elgato feed visible in Camera.app; stopped frame counter held, resumed counter advanced | No active HDMI picture was available; photo/video capture with this source not revalidated |
+| Camera picker | Native picker discovers camera names, stores unique IDs, supports System Default and preserves streaming state | Picker sheet deployed; user confirmed camera feed works after clicking Start Streaming | Full multiple-camera switching and physical unplug/replug remain to test; selection is per-window |
+
+Detailed findings are in display_refresh_rate.md, trackpad_gesture_mapping.md,
+iphone_duo_bringup.md, carplay_connection_diagnosis.md,
+virtual_external_display.md, virtual_camera_transport.md and
+apple_intelligence_eligibility.md. New external-display RPCs and pointer opt-in
+are documented in ../vphoned_http_api.md. Full Release VPhone builds passed
+through the final camera-picker version. The separate Release Launchpad build
+and development signing checks also passed. These are historical test results
+for the local 2.14.0-based builds, not CI results for a future upstream merge.
+
+## Guest configuration that is not an upstream source change
+
+The guest was booted using a local Standard-derived selection that additionally
+blocked these ARM64E_X1 lowering patches:
+
+- dyld-boot-arm64e_x1_bootstrap
+- dyld-cfw-arm64e_x1_cache
+- system-cibarcode-cfw-arm64e_x1
+- system-dyld-boot-arm64e_x1_bootstrap
+- system-fsck-boot-arm64e_x1_bootstrap
+- system-launchd-boot-arm64e_x1_bootstrap
+- system-mount-boot-arm64e_x1_bootstrap
+
+This is reproduction context, not a proposed default preset. The early recovery
+temporarily disabled MISFix and FoldDisplay hooks; both were restored after
+the native detour fix. The disabled-hook workaround is not the final state.
+Cached firmware was reused. No new firmware image is included in this change.
+
+The existing service profile kept its groups and added exceptions for CarKit,
+Siri and model/intelligence support. CarPlay depended on restoring carkitd and
+Siri jobs. App Store/account service restoration repaired built-in app downloads
+(Maps and Freeform), but did not establish functioning Apple Account
+authentication or FairPlay. These per-machine changes should not
+silently become global trimming defaults. Inspect the profile and preserve
+existing exceptions before reproducing any service changes.
+
+The mobile-user preference com.vphone.vphoned/iPhonePointerEnabled was set to
+boolean true explicitly. The source default remains off on iPhones. Mouse
+capture also remains off for new windows and is not enabled by clicking.
+
+## Investigations with no retained fix
+
+Apple Intelligence eligibility overrides exposed more UI but did not produce
+working inference: the guest selected a missing com.apple.modelmanager.remote
+host backend. MobileGestalt, eligibility and ModelManager experiments were
+rolled back. No host AppleInternal entitlement, inference service, security
+policy or region spoofing change is included.
+
+Multitasking thumbnails remain unresolved. Compressed snapshot files returned
+EIO; an experimental compression-rule exclusion was rolled back. App Store
+installation through Configurator and online attestation also remain unresolved.
+Do not confuse those with the repaired built-in app placeholder downloads.
+
+## Integration priorities
+
+1. Preserve upstream changes since the tested release, especially inner-panel
+ digital modes, stale digitizer UUID recovery and tap-to-wake declarations.
+2. Review the native detour and touch fixes independently from optional features.
+3. Audit private display API ABIs and native capture/transport error handling.
+ The external-display diagnostics RPC is read-only but potentially verbose.
+4. For camera acceptance, choose a camera and explicitly Start Streaming.
+ Verify actual moving imagery, switching while streaming, cancellation,
+ disconnection, stop/release and permission denial. Inspect runtime failure
+ reporting as well as successful preview delivery.
+5. Run the project builds/tests and repeat guest acceptance after integration.
+ No new VM, firmware download or host security change is necessary merely
+ to review this collection.
+
+Only source and research notes are supplied. Local build products, firmware,
+VM disks, certificates, signing team configuration, account data, diagnostic
+dumps, screenshots and recorded camera imagery are excluded.
diff --git a/Research/Guest/duo_session_probes/README.md b/Research/Guest/duo_session_probes/README.md
new file mode 100644
index 00000000..d2bc82b5
--- /dev/null
+++ b/Research/Guest/duo_session_probes/README.md
@@ -0,0 +1,30 @@
+# Native X1 validation probes
+
+These small standalone probes reproduce historical validation. They are not
+added to normal test schemes and do not change host security settings.
+
+On a host that supports PACIBSPPC/PAUTH-LR, from the repository root:
+
+```sh
+xcrun clang -arch arm64 -march=armv9.5-a+pauth-lr -c \
+ Research/Guest/duo_session_probes/native_pac_reference.s \
+ -o /tmp/vphone-pac-reference.o
+xcrun clang -arch arm64 -O0 \
+ Research/Guest/duo_session_probes/native_pac_relocation.c \
+ /tmp/vphone-pac-reference.o -o /tmp/vphone-pac-probe
+/tmp/vphone-pac-probe
+```
+
+The native reference and relocated sequence sign 4,096 input values with the
+same original PC and SP. This command passed again while preparing the local
+review candidate. Keep the C compilation at baseline ARM64 and unoptimized:
+compiling the whole program for ARMv9.5 may introduce unrelated instructions
+the host does not support, and this fixture depends on matched call-stack
+shape. Unsupported hardware may raise an illegal-instruction exception.
+This is a targeted signing equivalence check, not a comprehensive detour test.
+
+`guest_mis_validation.c` is the original in-guest 64-call MIS validation probe.
+It needs the guest libmisfix hook and an executable path supplied explicitly
+as argv[1]; it does not install or launch an app. Build/sign/deploy it using
+the project's existing guest-tool workflow. No signed probe, app bundle,
+certificate or personal test application is included here.
diff --git a/Research/Guest/duo_session_probes/guest_mis_validation.c b/Research/Guest/duo_session_probes/guest_mis_validation.c
new file mode 100644
index 00000000..9a2d7086
--- /dev/null
+++ b/Research/Guest/duo_session_probes/guest_mis_validation.c
@@ -0,0 +1,20 @@
+#include <CoreFoundation/CoreFoundation.h>
+#include <dlfcn.h>
+#include <stdio.h>
+#include <stdint.h>
+extern int MISValidateSignatureAndCopyInfoWithProgress(CFStringRef,CFDictionaryRef,CFDictionaryRef *,void *);
+int main(int argc,char **argv) {
+ if(argc!=2)return 2;
+ void *h=dlopen("/usr/lib/libmisfix.dylib",RTLD_NOW|RTLD_GLOBAL);
+ if(!h){fprintf(stderr,"%s\n",dlerror());return 3;}
+ CFStringRef path=CFStringCreateWithCString(NULL,argv[1],kCFStringEncodingUTF8);
+ for(int i=0;i<64;i++) {
+ CFDictionaryRef info=NULL;
+ int status=MISValidateSignatureAndCopyInfoWithProgress(path,NULL,&info,NULL);
+ if(status){fprintf(stderr,"validation failed 0x%x\n",status);return 4;}
+ if(info)CFRelease(info);
+ }
+ CFRelease(path);
+ puts("PASS: 64 real MIS signature validations with repaired native-X1 detour");
+ return 0;
+}
diff --git a/Research/Guest/duo_session_probes/native_pac_reference.s b/Research/Guest/duo_session_probes/native_pac_reference.s
new file mode 100644
index 00000000..df7d18d8
--- /dev/null
+++ b/Research/Guest/duo_session_probes/native_pac_reference.s
@@ -0,0 +1,12 @@
+.text
+.globl _reference_sign
+.globl _reference_pac_site
+.p2align 2
+_reference_sign:
+ mov x9, x30
+ mov x30, x0
+_reference_pac_site:
+ pacibsppc
+ mov x0, x30
+ mov x30, x9
+ ret
diff --git a/Research/Guest/duo_session_probes/native_pac_relocation.c b/Research/Guest/duo_session_probes/native_pac_relocation.c
new file mode 100644
index 00000000..0fde8135
--- /dev/null
+++ b/Research/Guest/duo_session_probes/native_pac_relocation.c
@@ -0,0 +1,32 @@
+#include <assert.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/mman.h>
+#include <libkern/OSCacheControl.h>
+#include "../../../VPhoneGuestComponents/MISFix/MISFixDetour.c"
+void MISFixNote(const char *format, ...) { (void)format; }
+const char *MISFixCallerImage(const void *address) { (void)address; return "test"; }
+extern uint64_t reference_sign(uint64_t);
+extern char reference_pac_site;
+int main(void) {
+ uint32_t *code = mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0);
+ assert(code != MAP_FAILED);
+ code[0] = 0xAA1E03E9; // mov x9, lr
+ code[1] = 0xAA0003FE; // mov lr, x0
+ unsigned n = vpRelocate(0xDAC1A7FEu, (uint64_t)&reference_pac_site, code+2);
+ assert(n == 8);
+ code[2+n] = 0xAA1E03E0; // mov x0, lr
+ code[3+n] = 0xAA0903FE; // mov lr, x9
+ code[4+n] = 0xD65F03C0; // ret
+ assert(mprotect(code,16384,PROT_READ|PROT_EXEC)==0);
+ sys_icache_invalidate(code,16384);
+ uint64_t (*relocated)(uint64_t) = (void *)code;
+ for (uint64_t i=1;i<=4096;i++) {
+ uint64_t input = 0x100000000ull + i*16;
+ uint64_t expected = reference_sign(input);
+ uint64_t actual = relocated(input);
+ if (expected != actual) { fprintf(stderr,"PAC mismatch at %llu\n",i); return 1; }
+ }
+ puts("PASS: 4096 relocated signatures match native PACIBSPPC at original PC");
+ return 0;
+}
diff --git a/Research/Guest/installd_and_store_download_repair.md b/Research/Guest/installd_and_store_download_repair.md
new file mode 100644
index 00000000..0348c0fc
--- /dev/null
+++ b/Research/Guest/installd_and_store_download_repair.md
@@ -0,0 +1,50 @@
+# installd and Store downloads: retained repair
+
+Two separate failures were repaired on the iPhone19,4 guest with iOS 27.1
+24A94425 and cloudOS 26.4 23E5207q.
+
+## installd: native X1 detour relocation
+
+MISFixDetour.c previously copied PACIBSPPC into its trampoline unchanged.
+That instruction binds the link-register signature to its instruction PC and
+SP, so moving it changes the signing context. The retained fix signs with
+the original PC and SP through PACIB171615, only for the first displaced
+instruction; later occurrences are rejected to avoid clobbering prologue state.
+
+The native comparison matched 4,096 signatures; 64 guest MIS signature
+validations succeeded. The user subsequently installed their own app through
+Xcode, confirming the installd-backed path. The intermediate report saying
+installd installation was not yet verified predates that successful install.
+The local app, provisioning material, signed probes and raw logs are excluded.
+Reproducible probe sources accompany duo_session_handoff.md.
+
+## Store services: built-in app placeholder downloads
+
+The guest's trimmed service profile had disabled the Store and account jobs.
+DNS resolved Apple endpoints and HTTPS transport succeeded; this was not a
+demonstrated loss of internet access. Restoring the six jobs listed in
+store-service-allow.json repaired built-in app downloads:
+
+- Maps downloaded, registered and launched with foreground verification.
+- Freeform downloaded and registered.
+- appstored subsequently reported download progress for additional built-in
+ apps, including App Store itself. Progress alone is not counted as a completed
+ installation of every listed app.
+
+This is a per-guest configuration repair. To reproduce without discarding
+existing profile groups or exceptions, follow the request-generation and
+apply/restart workflow in carplay_connection_diagnosis.md, replacing the
+`--slurpfile keep` argument with
+`Research/Guest/store-service-allow.json`. Inspect the generated request and
+the apply reply's failed array. Keep the original profile locally for rollback.
+The Store and CarPlay/Siri allow lists can be combined by union; applying
+either workflow to the current profile preserves the other's exceptions.
+
+The default service-trimming policy is unchanged by this collection. Upstream
+review should decide how to expose Store/download compatibility for guests
+that intentionally retain system-app placeholders or the App Store.
+
+This evidence establishes built-in Apple app restoration, not successful
+Apple Account sign-in or third-party FairPlay authorization. Configurator's
+App Store-list installation still returned "device is not booted" after client
+restart; that separate client workflow is not claimed fixed.
diff --git a/Research/Guest/iphone_duo_bringup.md b/Research/Guest/iphone_duo_bringup.md
index 1a035d88..dccbda48 100644
--- a/Research/Guest/iphone_duo_bringup.md
+++ b/Research/Guest/iphone_duo_bringup.md
@@ -4836,3 +4836,33 @@ Window captures (ScreenCaptureKit) cannot judge 120 Hz here: the cover, which
is natively 120, measures the same 40-55 changes per second as the inner
display, because the host composites the window at its own rate. Use guest
display links, or the host's `lastSubmittedTime` poller from the vGPU notes.
+### Expanded Now Playing corners (local diagnostic, 2026-10-29)
+
+On iPhone-Duo (27.1 24A94425), the unfolded expanded Now Playing panel
+provided by Broadcasts has a rectangular outer black background while its
+artwork remains rounded. A read-only UIKit probe reports 59 points for the
+cover's `_displayCornerRadius` and `_displayCornerRadiusIgnoringZoom`.
+Constructing a private, unregistered UIScreen from the inner CADisplay's
+FBSDisplayConfiguration reports zero for both getters and `_isEmbeddedScreen`
+false. This daemon-side object does not include SpringBoard's display adapter,
+so it establishes the native virtual-display default, not the exact live
+SpringBoard screen state.
+
+A local FoldDisplayAdapter experiment intercepts those two getter selectors
+(ABI checked as d16@0:8), matches only the recorded inner display identity,
+and substitutes the main screen's original native value only when the inner
+value is zero. A nonzero native inner value and every other display pass
+through. Limited logging records native/main/effective values in SpringBoard.
+No device-tree, MobileGestalt, firmware or host security setting is changed.
+The original source is saved outside the source tree as
+`.vphone-setup/FoldDisplayAdapter.before-corners.m`; previous environment
+bundle 2.14.0-local.e1ece920 remains available for rollback.
+Validation and visual outcome are recorded below after deployment.
+
+**Validated:** Release VPhone build passed without warnings/errors. Installed
+local bundle `2.14.0-local.232df3b7`, updated the existing guest environment,
+and cleanly restarted (vphoned connected in four seconds). Actual SpringBoard
+logs now confirm `_displayCornerRadius native=0.00 main=59.00 effective=59.00`
+for the inner screen, so the earlier daemon probe matches the live symptom.
+The user reproduced the Broadcasts panel and confirmed “that's fixed now.”
+Previous bundle retained for rollback; no additional change is pending.
diff --git a/Research/Guest/store-service-allow.json b/Research/Guest/store-service-allow.json
new file mode 100644
index 00000000..3bd66dc0
--- /dev/null
+++ b/Research/Guest/store-service-allow.json
@@ -0,0 +1,8 @@
+[
+ "com.apple.appstored",
+ "com.apple.itunesstored",
+ "com.apple.appstorecomponentsd",
+ "com.apple.akd",
+ "com.apple.amsaccountsd",
+ "com.apple.appleaccountd"
+]
diff --git a/Research/Guest/trackpad_gesture_mapping.md b/Research/Guest/trackpad_gesture_mapping.md
index 73ba966e..d1c99fc4 100644
--- a/Research/Guest/trackpad_gesture_mapping.md
+++ b/Research/Guest/trackpad_gesture_mapping.md
@@ -233,3 +233,43 @@ Requires a VM with the paired base and a physical trackpad.
6. Photos/Maps: pinch out zooms in, pinch in zooms out.
7. Resting the pointer over text and swiping must not start a text selection.
8. Toggle the menu item off — trackpad gestures stop, mouse drag still works.
+
+## Explicit mouse capture (local extension, 2026-10-29)
+
+Pointer capability advertising and host capture are independent. The local
+`iPhonePointerEnabled` preference makes iPhone guests advertise the same
+transport as iPads, without promising guest-side pointer handling. Each VM
+window starts with `isMouseCaptured = false`. Normal clicks never enable it.
+Device → Capture Mouse explicitly captures; Device → Release Mouse or
+Control–Option–M releases. A local key-event monitor reserves the release
+shortcut before VZ keyboard forwarding. Focus loss and window close also
+release capture.
+
+Capture disassociates hardware movement from the Mac cursor and accumulates
+relative event deltas in view coordinates, clamped to the view bounds. The
+existing interface-point conversion supplies normalized guest coordinates.
+Click, drag, release and native scrolling all use the same accumulated point.
+Release sends buttons=0 if any were held, ends native scrolling, re-associates
+the Mac cursor and restores the arrow. The opt-in preference persists, but
+actual capture is deliberately never persisted. Existing trackpad-to-touch
+behavior applies outside capture.
+
+Hardware keyboard attachment remains a boot configuration: `config.keyboards`
+is either `[VZUSBKeyboardConfiguration()]` or empty. The SDK's runtime USB
+controller API takes `VZUSBDevice`; its keyboard configuration is not that
+protocol. This does not rule out future research on private runtime keyboard
+attachment; no such experiment was performed here.
+
+Deployment: Release VPhone build passed; installed host bundle
+`2.14.0-local.54db0340` and restarted iPhone-Duo (guest connected in four
+seconds). Guest resources remain at `2.14.0-local.cc9b04bc`, which contains
+the identical guest opt-in code; the final capture change is host-only.
+The boolean opt-in was read back true after restart, and an unpressed
+`input.pointer` request had previously succeeded. The native UI-control tool
+encountered stale-bundle checks and then timed out opening the updated menu;
+manual capture/release verification was requested from the user. Do not treat
+build success or pointer RPC success as proof of the release shortcut.
+
+The user then tested Device → Capture Mouse and Control–Option–M and
+confirmed “Capture and release work.” The final build remains installed;
+capture defaults off and the persistent iPhone capability opt-in remains on.
diff --git a/Research/Guest/virtual_camera_transport.md b/Research/Guest/virtual_camera_transport.md
index e6ef6643..84522add 100644
--- a/Research/Guest/virtual_camera_transport.md
+++ b/Research/Guest/virtual_camera_transport.md
@@ -1,5 +1,64 @@
# Virtual camera transport and capture hooks
+## Live host camera source (local extension, 2026-10-29)
+
+Camera → Source: System Camera… selects a host video camera (see picker below).
+The menu requests video permission explicitly; it does not request microphone
+access. Start Streaming opens an AVCaptureSession on its own serial queue;
+Stop Streaming, Source: Off, source replacement and VM shutdown stop capture.
+Selecting the source without streaming does not start the session. Reconnecting
+the guest transport resumes an active stream but never restarts a stopped one.
+
+The capture delegate retains only its latest pixel buffer under a lock. The
+existing sender samples that buffer at its existing 30 Hz transport cadence,
+aspect-fits it onto a black 1280x720 BGRA frame, and uses the unchanged VSOCK
+1338 protocol. This forwards camera imagery through the synthetic camera;
+it does not pass through the physical camera's USB device, depth data or controls.
+
+Launchpad is the responsible app for the VM process tree. Its source now
+declares NSCameraUsageDescription and the hardened-runtime camera entitlement;
+the signing script verifies that entitlement. The core bundle also carries
+the usage description for standalone launches. An updated, appropriately
+signed Launchpad is required before exercising this source through Launchpad.
+The existing notarized installation has not been replaced with an ad hoc app.
+The Release VPhone build and separate Release Launchpad build both passed.
+The separate Launchpad was development-signed with the standard signing script;
+signature, camera entitlement and helper requirement verification passed.
+It remains under `.vphone-setup/camera-launchpad-build/Build/Products/Release/`
+in the workspace.
+
+Live acceptance: the user approved running the development copy. Installed the
+core as `2.14.0-local.ca790397`, retaining unchanged guest environment
+`2.14.0-local.b0297e92`. The separate development Launchpad started Duo without
+replacing the notarized app or installed privileged helper. Camera permission
+completed, System Camera selection enabled Start Streaming, and the guest's
+Camera.app visibly showed the default host capture device's Elgato NO SIGNAL
+image. Thus the live capture-device feed reaches the native guest preview;
+an active HDMI picture was not available to test. Two stopped observations of
+the 64-byte shared frame header were identical (frame index 1147). Restarting
+streaming advanced the frame index to 1325, and the menu returned to Stop
+Streaming. Capture was stopped again at the end of the test. No camera images
+were saved as test artifacts. The initial implementation followed AVFoundation's
+default video device.
+
+## Camera picker (local extension, 2026-10-29)
+
+Features → Camera → Source: System Camera… presents an accessible native camera
+picker with System Default and discovered devices by localized name. Built-in,
+external, Continuity and Desk View device types are enumerated afresh each time
+the picker opens. Both Launchpad and the core bundle declare the Continuity
+Camera device-type opt-in. Device selection uses unique IDs, never names, so
+duplicate camera names cannot select the wrong device.
+
+Cancel preserves the current source. An explicitly selected disconnected camera
+reports an error instead of falling back to another camera. Failed setup
+preserves the current source; successful switching preserves streaming state.
+The old and new capture sessions share a serial queue so the old session stops
+before the new one starts. The picker remembers selection for the current VM
+window only; it does not change the Mac's preferred camera or persist across
+VM launches. Release builds and development Launchpad signing passed; deployed
+core `2.14.0-local.44b76c2a` retains the unchanged guest environment.
+
## Runtime boundary
The host camera server sends BGRA frames over VSOCK port 1338. `vphoned`
diff --git a/Research/Guest/virtual_external_display.md b/Research/Guest/virtual_external_display.md
new file mode 100644
index 00000000..ea976788
--- /dev/null
+++ b/Research/Guest/virtual_external_display.md
@@ -0,0 +1,118 @@
+# Virtual external monitor prototype (2026-10-29)
+
+Scope: one optional external monitor on the existing iPhone-Duo guest. Only
+1920x1080 and 3840x2160 are accepted. No test app, firmware download, host
+security change or additional paravirtual GPU port is involved.
+
+The existing FoldDisplay injection prepares two named, disabled, untagged
+`CAWindowServerVirtualDisplay` objects during backboardd display detection,
+before SpringBoard starts. An explicit request enables only the selected slot;
+disconnect disables both without destroying their stable identities.
+Named CADisplay overrides classify the selected display as
+wireless (type 2), external, cloning-capable, at 1x for 1080p and 2x for 4K.
+The render display receives the runtime-resolved AirPlay connection tag, which
+gives ordinary clients a native `AirPlay` display identity.
+After configuration, SpringBoard refreshes the existing external CADisplay
+objects with their native `update` method, inactive slot first and active slot
+on the next main-queue turn, so existing monitor observers see reconnections.
+SpringBoard and applications choose their native mirroring/external-scene
+behavior; the host does not substitute a copy of the phone's screenshot.
+The existing Duo hooks now match the recorded inner display identity rather
+than every non-Main display, including status-bar provider and idiom handling.
+
+Configuration uses an atomic request plist, Darwin notification, and a matching
+nonce in the backboardd response. A stale request is not applied at boot.
+`display.external.configure {enabled,width?,height?}` waits up to five seconds
+for acknowledgement. `display.external.status` returns component state;
+`display.external.frame` captures only the named external display via the
+existing CARenderServerRenderDisplay/IOSurface mechanism and encodes a JPEG.
+The display component is currently installed only for the Duo, alongside the
+existing FoldDisplay support; other products are outside this prototype.
+
+Device → External Display offers Connect 1080p, Connect 4K, and Disconnect.
+The host preview window requests one frame at a time, targets the monitor's
+60 Hz refresh, and cannot build a queue of stale frames. Actual presentation
+rate depends on guest rendering, JPEG encoding, transfer, and host decoding;
+this is not a native VZ scanout. Closing the preview requests disconnection.
+The preview currently carries output only, not external pointer/touch input.
+
+Runtime validation and any classification changes are recorded below.
+
+First runtime: the render server created both requested resolutions and the
+host preview opened, but output was black. Broadcasts was foreground and is
+known by its author (the user) to support external displays. FBSDisplayMonitor
+showed the monitor's source disconnected with no raw configuration; CADisplay
+tag was zero. This matches the documented FrontBoard source gate in
+`iphone_duo_bringup.md`: non-always-connected sources with tag zero lose their
+raw configuration. A local probe confirmed the initializer's supported keys
+contain geometry/timing/pixel-format options, not a display-type override.
+The correction reuses the existing guest-resolved unknown connection tag
+before enabling the display. Runtime publication results follow below.
+
+The unknown tag restored the raw configuration but left ordinary clients
+with a disconnected Unknown identity. A separate diagnostic process then
+constructed FBSDisplayConfigurations with isolated CADisplay getter overrides:
+for native virtual type 3, tag 4 produced AirPlay[9-1], tag 65536 produced
+Unknown[9-1], and tag 32 produced Car[9-1]. Types 1 and 2 did not change these
+tag-derived roles. The next build therefore resolves the `AirPlay` tag by name
+through NSStringFromBKSDisplayTags, and uses the wireless/external adapter.
+That intermediate implementation removed the display after disabling it;
+the final stable-slot implementation below retains both render objects.
+This changes guest display classification only; no network AirPlay receiver,
+Apple Account, pairing, or host security setting is added.
+
+The AirPlay tag did produce a connected source in newly initialized FBS
+monitors, including one inspected inside Broadcasts through its existing
+Xcode debugger. Broadcasts still had only its normal application scene;
+UIScreen.screens contained the cover and inner panel, with no external screen.
+The installed app's manifest declares ExternalDisplay and its preference is
+at the enabled default. airplayd, replayd and mediaremoted were enabled and
+running; avconferenced was enabled/loaded and idle. None of those jobs was
+trimmed. The standalone unentitled scene-count probe returned zero despite
+Xcode proving one app scene; do not use that count as authoritative evidence.
+
+A read-only SpringBoard snapshot exposed the inventory boundary: its existing
+FBDisplayManager had only sources 1–7, even after the newly created 4K monitor
+was visible to new monitors. No external root scene existed. This matches the
+historic render-display bring-up notes: a newly initialized monitor sees late
+render objects, while the existing SpringBoard monitor misses their discovery.
+
+The next implementation declares two disabled, untagged software render nodes
+(1080p and 4K) during backboardd's display detection, after the inner node and
+before SpringBoard starts. Only the chosen node receives the AirPlay connection
+tag and enabled state. Disconnect clears its tag and disables it; it does not
+destroy the stable render object. This keeps a single connected external
+monitor while allowing both supported resolutions through native connection
+changes. Captures use the selected node's exact name from component status.
+`display.external.diagnostics` snapshots SpringBoard's monitor, root scenes,
+scene manager and relevant policy metadata without changing them. Its initial
+one-second wait was shorter than first-use snapshot generation; the wait is
+now five seconds. Snapshot files remain under the existing guest cache folder.
+
+Stable-slot runtime (d92ee395): the first 4K connection produced a native
+SBUIWindowSceneSessionRoleNonInteractiveDisplay root and Broadcasts' actual
+external artwork/title UI. The user confirmed seeing the external display;
+the captured image is 3840x2160. Native disconnect removed the external root
+and marked both sources disconnected. Unsupported 1280x720 was rejected while
+leaving the active 4K configuration unchanged.
+
+Reconnection exposed a cache boundary: CADisplay in a fresh client showed the
+requested AirPlay tag, but SpringBoard's existing sources stayed disconnected.
+The next build explicitly updates the two existing external CADisplay objects
+in SpringBoard after a configure notification. `-[CADisplay update]` is verified
+as v16@0:8; it refreshes those objects and their native KVO observers. Other
+displays and the cached object array are untouched. The inactive slot is updated
+first and the active slot on the next main-queue turn. Both slots are explicitly
+untagged/disabled before choosing the active one, and initialization explicitly
+sets tag zero. This replaces reliance on hardware-driven display-cache refresh
+that these virtual render objects do not provide.
+
+Final validation (`2.14.0-local.b0297e92`): boot status was disabled. Broadcasts
+produced its native external artwork/title UI at 1920x1080, switched live to
+3840x2160, and produced the same native UI after disconnect/reconnect at 4K.
+SpringBoard diagnostics contained zero external root scenes while disconnected
+and one after reconnect. The 4K native configuration uses 1920x1080 points at
+2x. Captures and diagnostics are retained in the workspace's
+`.vphone-setup/external-display/final-*` files. The user confirmed visible
+external output. No app source changes, test app, or service untrimming was
+needed for this fix.
diff --git a/Research/vphoned_http_api.md b/Research/vphoned_http_api.md
index 5dbaa31a..4d109429 100644
--- a/Research/vphoned_http_api.md
+++ b/Research/vphoned_http_api.md
@@ -1285,3 +1285,51 @@ properties, plus up to4096 bytes per available report descriptor as base64.
No HID event is dispatched. `available=false` reports missing native APIs or
failure to create the read client. Absent properties are omitted; their absence
does not establish a zero range. Service metadata is not a touch-hit receipt.
+
+### iPhone pointer forwarding opt-in (local extension, 2026-10-29)
+
+An iPhone guest can explicitly advertise `pointer_scroll` and `pointer_buttons`
+by setting the mobile-user preference through RPC:
+
+```text
+settings.set {"domain":"com.vphone.vphoned","key":"iPhonePointerEnabled","type":"bool","value":true}
+```
+
+The preference is read by each health response; the VM window picks up the
+capabilities on its next health probe without a restart. It persists across
+boots. False, deletion, or a non-boolean value restores the default iPhone
+behavior. iPads remain enabled by default; other products do not opt in.
+Advertising support does not capture the mouse. Device → Capture Mouse is
+off for every new window, is never enabled by clicking the display, and is
+not persisted. It forwards relative mouse movement, buttons and native scroll
+only after explicit menu activation. Control–Option–M or Device → Release
+Mouse releases it, as does losing window focus or closing the window. The
+Mac cursor is re-associated on release and any held guest pointer buttons are
+released. Trackpad-to-touch replay remains a separate setting.
+This only advertises/forwards input; it does not enable AssistiveTouch or
+promise iOS will show a pointer. The inner Duo display's pointer mapping is
+not established by this opt-in.
+
+### Optional Duo external display (local extension, 2026-10-29)
+
+The existing Duo display component supports one connected external monitor,
+disabled at boot. Only these pixel dimensions are accepted: 1920x1080 and
+3840x2160. Other dimensions fail before changing the active configuration.
+
+| Method | Parameters and result |
+| --- | --- |
+| `display.external.status` | No parameters; returns `enabled` and, when available, the selected display's name and geometry. |
+| `display.external.configure` | `enabled: Bool`; enabling also requires supported integer `width` and `height`. Waits up to five seconds for a matching acknowledgement. |
+| `display.external.frame` | No parameters; returns a JPEG as base64 `data` with `mime_type: image/jpeg`, captured from the selected external display. Requires a connected monitor. |
+| `display.external.diagnostics` | No parameters; read-only SpringBoard display-source, root-scene and policy snapshot, with up to five seconds for generation. |
+
+Example: `display.external.configure {"enabled":true,"width":3840,"height":2160}`.
+Disconnect with `display.external.configure {"enabled":false}`. This component
+is currently installed only on Duo guests. Native guest policy and each app
+control external scene or mirroring behavior; the frame is not a phone screenshot.
+
+Device → External Display offers the same two modes and Disconnect. Closing
+the preview disconnects it. The output-only host window requests one JPEG at a
+time, targeting 60 Hz with actual throughput dependent on rendering, encoding,
+transfer and decoding. Resolution switching and reconnecting work at runtime;
+neither requires a VM reboot.
diff --git a/VPhoneDaemon/Daemon/GuestAPI+Device.swift b/VPhoneDaemon/Daemon/GuestAPI+Device.swift
index 5cb2a3a1..c6529261 100644
--- a/VPhoneDaemon/Daemon/GuestAPI+Device.swift
+++ b/VPhoneDaemon/Daemon/GuestAPI+Device.swift
@@ -67,6 +67,27 @@ extension GuestAPI {
return ["displays": vp_display_probe()]
case "display.modes":
return vp_display_modes()
+ case "display.external.status":
+ return vp_external_display_status()
+ case "display.external.diagnostics":
+ return vp_external_display_diagnostics()
+ case "display.external.configure":
+ guard let enabled = params["enabled"] as? Bool else {
+ throw GuestAPIError.invalidRequest("enabled must be a boolean")
+ }
+ let width = params["width"] as? Int ?? 1920
+ let height = params["height"] as? Int ?? 1080
+ let result = vp_external_display_configure(enabled, width, height)
+ guard result["ok"] as? Bool == true else {
+ throw GuestAPIError.operationFailed(result["error"] as? String ?? "External display configuration failed")
+ }
+ return result
+ case "display.external.frame":
+ var error: NSString?
+ guard let data = vp_external_display_frame(&error) else {
+ throw GuestAPIError.operationFailed(error as String? ?? "External display capture failed")
+ }
+ return ["mime_type": "image/jpeg", "data": data.base64EncodedString()]
case "device.environment":
return try environmentReport()
case "device.basebin":
diff --git a/VPhoneDaemon/Daemon/GuestAPI.swift b/VPhoneDaemon/Daemon/GuestAPI.swift
index 6cf93a04..2cd6b342 100644
--- a/VPhoneDaemon/Daemon/GuestAPI.swift
+++ b/VPhoneDaemon/Daemon/GuestAPI.swift
@@ -157,10 +157,18 @@ enum GuestAPI {
]
}
- /// iPadOS takes a pointing device as it is; iOS on an iPhone only through
- /// AssistiveTouch, so `input.pointer` and `input.scroll` are an iPad's.
+ /// iPadOS accepts a pointing device directly. iPhones normally need
+ /// AssistiveTouch, but may explicitly opt in to host pointer forwarding.
+ /// Advertising the transport does not promise that iOS consumes it.
private static func pointerCapabilities() -> [String] {
- GuestVirtualAudioProduct.machine().hasPrefix("iPad") ? ["pointer_scroll", "pointer_buttons"] : []
+ let machine = GuestVirtualAudioProduct.machine()
+ if machine.hasPrefix("iPad") { return ["pointer_scroll", "pointer_buttons"] }
+ guard machine.hasPrefix("iPhone"),
+ let preference = try? readPreference(domain: "com.vphone.vphoned", key: "iPhonePointerEnabled"),
+ preference["type"] as? String == "bool",
+ preference["value"] as? Bool == true
+ else { return [] }
+ return ["pointer_scroll", "pointer_buttons"]
}
static func execute(method: String, params: [String: Any]) throws -> [String: Any] {
diff --git a/VPhoneDaemon/Native/Include/VphonedNative.h b/VPhoneDaemon/Native/Include/VphonedNative.h
index ddb19570..9ef1720b 100644
--- a/VPhoneDaemon/Native/Include/VphonedNative.h
+++ b/VPhoneDaemon/Native/Include/VphonedNative.h
@@ -205,6 +205,10 @@ NSDictionary * _Nonnull vp_hinge_get(void);
NSArray * _Nonnull vp_display_probe(void);
/// Read-only CoreAnimation display modes and capability surface.
NSDictionary<NSString *, id> * _Nonnull vp_display_modes(void);
+NSDictionary<NSString *, id> * _Nonnull vp_external_display_status(void);
+NSDictionary<NSString *, id> * _Nonnull vp_external_display_diagnostics(void);
+NSDictionary<NSString *, id> * _Nonnull vp_external_display_configure(BOOL enabled, NSInteger width, NSInteger height);
+NSData * _Nullable vp_external_display_frame(NSString * _Nullable * _Nullable error);
NSDictionary<NSString *, id> * _Nonnull vp_device_pose_get(void);
BOOL vp_hinge_set(double angleDegrees);
/// Sweep the hinge to `targetDegrees` the way Apple's Duo tooling does: over
diff --git a/VPhoneDaemon/Native/vphoned_external_display.m b/VPhoneDaemon/Native/vphoned_external_display.m
new file mode 100644
index 00000000..d2820ebf
--- /dev/null
+++ b/VPhoneDaemon/Native/vphoned_external_display.m
@@ -0,0 +1,96 @@
+#import "Include/VphonedNative.h"
+#import <CoreGraphics/CoreGraphics.h>
+#import <ImageIO/ImageIO.h>
+#include <dlfcn.h>
+#include <notify.h>
+#include <unistd.h>
+
+static NSString *const requestPath = @"/var/mobile/Library/Caches/vphone-external-display.request.plist";
+static NSString *const statusPath = @"/var/mobile/Library/Caches/vphone-external-display.status.plist";
+
+NSDictionary<NSString *, id> *vp_external_display_status(void) {
+ return [NSDictionary dictionaryWithContentsOfFile:statusPath] ?: @{@"enabled": @NO, @"ok": @NO, @"error": @"External display component not connected"};
+}
+
+NSDictionary<NSString *, id> *vp_external_display_diagnostics(void) {
+ NSTimeInterval start = NSDate.date.timeIntervalSince1970;
+ notify_post("com.vphone.external-display.inspect");
+ for (int i = 0; i < 100; i++) {
+ NSDictionary *report = [NSDictionary dictionaryWithContentsOfFile:@"/var/mobile/Library/Caches/vphone-external-display.springboard.plist"];
+ if ([report[@"updated"] doubleValue] >= start) return report;
+ usleep(50000);
+ }
+ return @{@"available": @NO, @"error": @"SpringBoard inspection did not respond"};
+}
+
+NSDictionary<NSString *, id> *vp_external_display_configure(BOOL enabled, NSInteger width, NSInteger height) {
+ if (enabled && !((width == 1920 && height == 1080) || (width == 3840 && height == 2160)))
+ return @{@"ok": @NO, @"error": @"Only 1920x1080 and 3840x2160 are supported"};
+ NSString *nonce = NSUUID.UUID.UUIDString;
+ NSDictionary *request = @{@"enabled": @(enabled), @"width": @(width), @"height": @(height), @"nonce": nonce};
+ if (![request writeToFile:requestPath atomically:YES]) return @{@"ok": @NO, @"error": @"Cannot write display request"};
+ notify_post("com.vphone.external-display.configure");
+ for (int i = 0; i < 50; i++) {
+ NSDictionary *status = vp_external_display_status();
+ if ([status[@"nonce"] isEqual:nonce]) return status;
+ usleep(100000);
+ }
+ return @{@"ok": @NO, @"error": @"External display component did not acknowledge request"};
+}
+
+NSData *vp_external_display_frame(NSString **error) {
+ @autoreleasepool {
+ NSDictionary *status = vp_external_display_status();
+ NSInteger width = [status[@"width"] integerValue], height = [status[@"height"] integerValue];
+ if (![status[@"enabled"] boolValue] || ![status[@"ok"] boolValue] ||
+ !((width == 1920 && height == 1080) || (width == 3840 && height == 2160))) {
+ if (error) *error = @"External display is disconnected";
+ return nil;
+ }
+ void *io = dlopen("/System/Library/Frameworks/IOSurface.framework/IOSurface", RTLD_NOW);
+ void *ca = dlopen("/System/Library/Frameworks/QuartzCore.framework/QuartzCore", RTLD_NOW);
+ void *(*create)(CFDictionaryRef) = dlsym(io, "IOSurfaceCreate");
+ void *(*base)(void *) = dlsym(io, "IOSurfaceGetBaseAddress");
+ size_t (*stride)(void *) = dlsym(io, "IOSurfaceGetBytesPerRow");
+ kern_return_t (*lock)(void *, uint32_t, uint32_t *) = dlsym(io, "IOSurfaceLock");
+ kern_return_t (*unlock)(void *, uint32_t, uint32_t *) = dlsym(io, "IOSurfaceUnlock");
+ void (*render)(mach_port_t, CFStringRef, void *, int32_t, int32_t) = dlsym(ca, "CARenderServerRenderDisplay");
+ if (!create || !base || !stride || !lock || !unlock || !render) {
+ if (error) *error = @"Display capture API unavailable";
+ return nil;
+ }
+ void *surface = create((__bridge CFDictionaryRef)@{@"IOSurfaceWidth": @(width), @"IOSurfaceHeight": @(height),
+ @"IOSurfaceBytesPerElement": @4, @"IOSurfacePixelFormat": @((uint32_t)'BGRA')});
+ if (!surface) { if (error) *error = @"Cannot allocate display surface"; return nil; }
+ NSString *name = status[@"name"];
+ if (![name isEqual:@"VPhone External 1080p"] && ![name isEqual:@"VPhone External 4K"]) {
+ CFRelease(surface); if (error) *error = @"Unexpected external display name"; return nil;
+ }
+ render(0, (__bridge CFStringRef)name, surface, 0, 0);
+ if (lock(surface, 1, NULL) != KERN_SUCCESS) {
+ CFRelease(surface); if (error) *error = @"Cannot read display surface"; return nil;
+ }
+ size_t bytesPerRow = stride(surface);
+ NSData *pixels = base(surface) ? [NSData dataWithBytes:base(surface) length:bytesPerRow * height] : nil;
+ unlock(surface, 1, NULL);
+ CFRelease(surface);
+ if (!pixels) { if (error) *error = @"Display surface has no pixels"; return nil; }
+ CGColorSpaceRef space = CGColorSpaceCreateDeviceRGB();
+ CGDataProviderRef provider = CGDataProviderCreateWithCFData((__bridge CFDataRef)pixels);
+ CGImageRef image = CGImageCreate(width, height, 8, 32, bytesPerRow, space,
+ kCGImageAlphaNoneSkipFirst | kCGBitmapByteOrder32Little, provider, NULL, NO, kCGRenderingIntentDefault);
+ NSMutableData *jpeg = [NSMutableData data];
+ CGImageDestinationRef destination = image ? CGImageDestinationCreateWithData((__bridge CFMutableDataRef)jpeg, CFSTR("public.jpeg"), 1, NULL) : NULL;
+ BOOL success = NO;
+ if (destination) {
+ CGImageDestinationAddImage(destination, image, (__bridge CFDictionaryRef)@{(__bridge NSString *)kCGImageDestinationLossyCompressionQuality: @0.8});
+ success = CGImageDestinationFinalize(destination);
+ CFRelease(destination);
+ }
+ if (image) CGImageRelease(image);
+ CGDataProviderRelease(provider);
+ CGColorSpaceRelease(space);
+ if (!success) { if (error) *error = @"Cannot encode display frame"; return nil; }
+ return jpeg;
+ }
+}
diff --git a/VPhoneExecutable/VPhoneVirtualization/Resources/Info.plist b/VPhoneExecutable/VPhoneVirtualization/Resources/Info.plist
index 377396a1..433b0ec3 100644
--- a/VPhoneExecutable/VPhoneVirtualization/Resources/Info.plist
+++ b/VPhoneExecutable/VPhoneVirtualization/Resources/Info.plist
@@ -14,5 +14,9 @@
<string>2.14.0</string>
<key>NSMicrophoneUsageDescription</key>
<string>Shares this Mac's microphone with the virtual iPhone so apps running inside it can record.</string>
+ <key>NSCameraUsageDescription</key>
+ <string>Shares this Mac's camera with the virtual iPhone while System Camera streaming is enabled.</string>
+ <key>NSCameraUseContinuityCameraDeviceType</key>
+ <true/>
</dict>
</plist>
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/HostDevices/VPhoneCameraServer.swift b/VPhoneExecutable/VPhoneVirtualization/UI/HostDevices/VPhoneCameraServer.swift
index aabc9c49..56ffc493 100644
--- a/VPhoneExecutable/VPhoneVirtualization/UI/HostDevices/VPhoneCameraServer.swift
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/HostDevices/VPhoneCameraServer.swift
@@ -24,6 +24,7 @@ final class VPhoneCameraServer {
case off
case testPattern
case videoFile // .mov / .mp4 / .m4v via AVAssetReader
+ case systemCamera
}
nonisolated static let vsockPort: UInt32 = 1338
@@ -33,6 +34,7 @@ final class VPhoneCameraServer {
nonisolated static let pixelFormat: UInt32 = 0x4247_5241 // 'BGRA' — kCMPixelFormat_32BGRA
private(set) var sourceKind: SourceKind = .off
+ private(set) var cameraDeviceID: String?
private(set) var isConnected = false
private var device: VZVirtioSocketDevice?
@@ -72,16 +74,23 @@ final class VPhoneCameraServer {
// MARK: - Source selection
- func setSource(_ kind: SourceKind, videoURL: URL? = nil) {
- if sourceKind == kind, kind != .videoFile {
+ func setSource(_ kind: SourceKind, videoURL: URL? = nil, cameraDeviceID: String? = nil) throws {
+ if sourceKind == kind, kind != .videoFile,
+ kind != .systemCamera || self.cameraDeviceID == cameraDeviceID {
return
}
+ // Construct first so a failed camera setup preserves the old source.
+ let camera = kind == .systemCamera ? try VPhoneSystemCameraProducer(
+ width: Self.defaultWidth, height: Self.defaultHeight, deviceID: cameraDeviceID) : nil
let wasStreaming = (timer != nil)
if wasStreaming {
stopStreaming()
}
sourceKind = kind
+ self.cameraDeviceID = kind == .systemCamera ? cameraDeviceID : nil
switch kind {
+ case .systemCamera:
+ producer = camera
case .off:
producer = nil
case .testPattern:
@@ -121,6 +130,7 @@ final class VPhoneCameraServer {
if timer != nil {
return
}
+ producer?.start()
let interval = 1.0 / Self.defaultFPS
// Timer fires on the main queue so MainActor-isolated state
// (producer, connectionFD) can be read directly without tripping
@@ -167,8 +177,8 @@ final class VPhoneCameraServer {
}
func stopStreaming() {
- guard let t = timer else { return }
- t.cancel()
+ producer?.stop()
+ timer?.cancel()
timer = nil
print("[camera] streaming stopped")
}
@@ -207,9 +217,8 @@ final class VPhoneCameraServer {
self.isConnected = true
print("[camera] connected on vsock port \(Self.vsockPort)")
self.onConnectionStateChange?(true)
- if self.sourceKind != .off {
- self.startStreaming()
- }
+ // An existing streaming timer resumes sending on this
+ // connection. Do not reopen a source the user stopped.
case let .failure(error):
print("[camera] connect failed: \(error). Retrying in 3s.")
DispatchQueue.main.asyncAfter(deadline: .now() + 3.0) { [weak self] in
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/HostDevices/VPhoneFrameProducer.swift b/VPhoneExecutable/VPhoneVirtualization/UI/HostDevices/VPhoneFrameProducer.swift
index a544fbb3..dc6eec78 100644
--- a/VPhoneExecutable/VPhoneVirtualization/UI/HostDevices/VPhoneFrameProducer.swift
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/HostDevices/VPhoneFrameProducer.swift
@@ -19,9 +19,16 @@ struct VPhoneCameraFrame: Sendable {
/// see the queue isolation, so we opt out of the strict-concurrency
/// check.
protocol VPhoneFrameProducer: AnyObject, Sendable {
+ func start()
+ func stop()
func nextFrame() -> VPhoneCameraFrame?
}
+extension VPhoneFrameProducer {
+ func start() {}
+ func stop() {}
+}
+
// MARK: - Test pattern
/// Generates a smoothly-animating BGRA pattern: a moving vertical gradient
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/HostDevices/VPhoneSystemCameraProducer.swift b/VPhoneExecutable/VPhoneVirtualization/UI/HostDevices/VPhoneSystemCameraProducer.swift
new file mode 100644
index 00000000..7c90a8a1
--- /dev/null
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/HostDevices/VPhoneSystemCameraProducer.swift
@@ -0,0 +1,106 @@
+import AVFoundation
+import CoreImage
+import Foundation
+
+/// The capture queue owns the session; a lock protects the latest buffer shared
+/// with the existing frame sender. Keeping one buffer bounds latency and memory.
+final class VPhoneSystemCameraProducer: NSObject, VPhoneFrameProducer,
+ AVCaptureVideoDataOutputSampleBufferDelegate, @unchecked Sendable
+{
+ private let session = AVCaptureSession()
+ // Shared queue orders stop of the old camera before start of its replacement.
+ private static let captureQueue = DispatchQueue(label: "com.vphone.camera.capture")
+ private let queue = VPhoneSystemCameraProducer.captureQueue
+ private let lock = NSLock()
+ private var latest: CVPixelBuffer?
+ private let context = CIContext()
+ private let width: Int
+ private let height: Int
+
+ init(width: Int, height: Int, deviceID: String? = nil) throws {
+ self.width = width
+ self.height = height
+ super.init()
+ // Permission is requested by the explicit menu action, before creating
+ // a device input. Selecting this source alone does not start capture.
+ guard AVCaptureDevice.authorizationStatus(for: .video) == .authorized else {
+ throw Self.error("Camera access has not been granted.")
+ }
+ let selected = deviceID.flatMap { AVCaptureDevice(uniqueID: $0) }
+ guard let device = deviceID == nil ? AVCaptureDevice.default(for: .video) : selected else {
+ throw Self.error(deviceID == nil ? "No system camera is available." :
+ "The selected camera is no longer available. Choose another camera.")
+ }
+ session.beginConfiguration()
+ defer { session.commitConfiguration() }
+ if session.canSetSessionPreset(.hd1280x720) { session.sessionPreset = .hd1280x720 }
+ let input = try AVCaptureDeviceInput(device: device)
+ guard session.canAddInput(input) else {
+ throw Self.error("The system camera could not be opened.")
+ }
+ session.addInput(input)
+ let output = AVCaptureVideoDataOutput()
+ output.alwaysDiscardsLateVideoFrames = true
+ output.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA]
+ output.setSampleBufferDelegate(self, queue: queue)
+ guard session.canAddOutput(output) else {
+ throw Self.error("The system camera cannot provide video frames.")
+ }
+ session.addOutput(output)
+ }
+
+ func start() {
+ queue.async { [self] in
+ if !session.isRunning { session.startRunning() }
+ }
+ }
+
+ func stop() {
+ queue.async { [self] in
+ if session.isRunning { session.stopRunning() }
+ lock.lock()
+ latest = nil
+ lock.unlock()
+ }
+ }
+
+ func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer,
+ from connection: AVCaptureConnection) {
+ guard let buffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
+ lock.lock()
+ latest = buffer
+ lock.unlock()
+ }
+
+ func nextFrame() -> VPhoneCameraFrame? {
+ lock.lock()
+ let buffer = latest
+ lock.unlock()
+ guard let buffer else { return nil }
+ var destination: CVPixelBuffer?
+ guard CVPixelBufferCreate(kCFAllocatorDefault, width, height,
+ kCVPixelFormatType_32BGRA, nil, &destination) == kCVReturnSuccess,
+ let destination else { return nil }
+ let source = CIImage(cvPixelBuffer: buffer)
+ let scale = min(CGFloat(width) / source.extent.width, CGFloat(height) / source.extent.height)
+ let scaled = source.transformed(by: CGAffineTransform(scaleX: scale, y: scale))
+ let centered = scaled.transformed(by: CGAffineTransform(
+ translationX: (CGFloat(width) - scaled.extent.width) / 2 - scaled.extent.minX,
+ y: (CGFloat(height) - scaled.extent.height) / 2 - scaled.extent.minY))
+ let bounds = CGRect(x: 0, y: 0, width: width, height: height)
+ let image = centered.composited(over: CIImage(color: .black).cropped(to: bounds))
+ context.render(image, to: destination, bounds: bounds, colorSpace: CGColorSpaceCreateDeviceRGB())
+ guard CVPixelBufferLockBaseAddress(destination, .readOnly) == kCVReturnSuccess else { return nil }
+ defer { CVPixelBufferUnlockBaseAddress(destination, .readOnly) }
+ guard let base = CVPixelBufferGetBaseAddress(destination) else { return nil }
+ let stride = CVPixelBufferGetBytesPerRow(destination)
+ return VPhoneCameraFrame(width: width, height: height, bytesPerRow: stride,
+ timestampNS: UInt64(ProcessInfo.processInfo.systemUptime * 1e9),
+ pixels: Data(bytes: base, count: stride * height))
+ }
+
+ private static func error(_ message: String) -> NSError {
+ NSError(domain: "VPhoneSystemCameraProducer", code: 1,
+ userInfo: [NSLocalizedDescriptionKey: message])
+ }
+}
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuCamera.swift b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuCamera.swift
index f0e95df8..b97764ff 100644
--- a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuCamera.swift
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuCamera.swift
@@ -1,4 +1,5 @@
import AppKit
+import AVFoundation
import UniformTypeIdentifiers
// MARK: - Camera Menu
@@ -39,6 +40,10 @@ extension VPhoneMenuController {
cameraSourceVideoFileItem = videoFile
menu.addItem(videoFile)
+ let systemCamera = makeItem("Source: System Camera…", action: #selector(setCameraSourceSystemCamera))
+ cameraSourceSystemCameraItem = systemCamera
+ menu.addItem(systemCamera)
+
menu.addItem(NSMenuItem.separator())
let startStop = makeItem(
@@ -68,11 +73,12 @@ extension VPhoneMenuController {
(kind == .testPattern) ? .on : .off
cameraSourceVideoFileItem?.state =
(kind == .videoFile) ? .on : .off
+ cameraSourceSystemCameraItem?.state = (kind == .systemCamera) ? .on : .off
}
@objc func setCameraSourceOff() {
cameraServer?.stopStreaming()
- cameraServer?.setSource(.off)
+ try? cameraServer?.setSource(.off)
refreshCameraSourceCheckmarks()
cameraStartStopItem?.isEnabled = false
cameraStartStopItem?.title = VPhoneLocalization.text("Start Streaming")
@@ -80,7 +86,7 @@ extension VPhoneMenuController {
}
@objc func setCameraSourceTestPattern() {
- cameraServer?.setSource(.testPattern)
+ try? cameraServer?.setSource(.testPattern)
refreshCameraSourceCheckmarks()
cameraStartStopItem?.isEnabled = (cameraServer?.isConnected ?? false)
}
@@ -102,7 +108,7 @@ extension VPhoneMenuController {
panel.title = VPhoneLocalization.text("Choose Video File")
VPhoneAlert.present(panel) { [weak self] response in
guard let self, response == .OK, let url = panel.url else { return }
- cameraServer?.setSource(.videoFile, videoURL: url)
+ try? cameraServer?.setSource(.videoFile, videoURL: url)
refreshCameraSourceCheckmarks()
cameraStartStopItem?.isEnabled =
(cameraServer?.isConnected ?? false) &&
@@ -110,6 +116,75 @@ extension VPhoneMenuController {
}
}
+ @objc func setCameraSourceSystemCamera() {
+ Task { @MainActor [weak self] in
+ guard let self else { return }
+ let status = AVCaptureDevice.authorizationStatus(for: .video)
+ var allowed = status == .authorized
+ if status == .notDetermined {
+ allowed = await AVCaptureDevice.requestAccess(for: .video)
+ }
+ do {
+ guard allowed else {
+ throw NSError(domain: "VPhoneCamera", code: 1, userInfo: [NSLocalizedDescriptionKey:
+ "Allow camera access for vphone-launchpad in System Settings → Privacy & Security → Camera."])
+ }
+ presentSystemCameraPicker()
+ } catch {
+ VPhoneAlert.present(title: "System Camera Unavailable",
+ message: error.localizedDescription, style: .warning)
+ }
+ }
+ }
+
+ private func presentSystemCameraPicker() {
+ // Discover afresh each time so plugging in a camera requires no restart.
+ let discovery = AVCaptureDevice.DiscoverySession(
+ deviceTypes: [.builtInWideAngleCamera, .external, .continuityCamera, .deskViewCamera],
+ mediaType: .video, position: .unspecified)
+ let devices = discovery.devices.sorted {
+ $0.localizedName.localizedStandardCompare($1.localizedName) == .orderedAscending
+ }
+ let picker = NSPopUpButton(frame: NSRect(x: 0, y: 0, width: 340, height: 26), pullsDown: false)
+ picker.setAccessibilityLabel(VPhoneLocalization.text("Camera"))
+ picker.addItem(withTitle: VPhoneLocalization.text("System Default"))
+ for device in devices {
+ // Use identifiers rather than names: two cameras may have the same name.
+ let item = NSMenuItem(title: device.localizedName, action: nil, keyEquivalent: "")
+ item.representedObject = device.uniqueID
+ picker.menu?.addItem(item)
+ }
+ if let selected = cameraServer?.cameraDeviceID {
+ if let item = picker.itemArray.first(where: { $0.representedObject as? String == selected }) {
+ picker.select(item)
+ } else {
+ let missing = NSMenuItem(title: VPhoneLocalization.text("Selected Camera (Disconnected)"),
+ action: nil, keyEquivalent: "")
+ missing.representedObject = selected
+ picker.menu?.addItem(missing)
+ picker.select(missing)
+ }
+ }
+ let alert = NSAlert()
+ alert.messageText = VPhoneLocalization.text("Choose Camera")
+ alert.informativeText = VPhoneLocalization.text("Choose the Mac camera to share with this virtual iPhone.")
+ alert.accessoryView = picker
+ alert.addButton(withTitle: VPhoneLocalization.text("Use Camera"))
+ alert.addButton(withTitle: VPhoneLocalization.text("Cancel"))
+ VPhoneAlert.present(alert) { [weak self] response in
+ guard let self, response == .alertFirstButtonReturn else { return }
+ do {
+ try cameraServer?.setSource(.systemCamera,
+ cameraDeviceID: picker.selectedItem?.representedObject as? String)
+ refreshCameraSourceCheckmarks()
+ cameraStartStopItem?.isEnabled = cameraServer?.isConnected ?? false
+ } catch {
+ VPhoneAlert.present(title: "System Camera Unavailable",
+ message: error.localizedDescription, style: .warning)
+ }
+ }
+ }
+
@objc func toggleCameraStreaming() {
guard let server = cameraServer else { return }
if cameraStartStopItem?.title == VPhoneLocalization.text("Start Streaming") {
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuController.swift b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuController.swift
index 9b7d0a08..39679cbc 100644
--- a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuController.swift
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuController.swift
@@ -88,6 +88,8 @@ class VPhoneMenuController {
var touchIDMenuItem: NSMenuItem?
var trackpadGesturesItem: NSMenuItem?
+ var mouseCaptureItem: NSMenuItem?
+ lazy var externalDisplayWindow = VPhoneExternalDisplayWindowController(control: control)
var locationProvider: VPhoneLocationProvider?
var locationMenuItem: NSMenuItem?
var locationPresetMenuItem: NSMenuItem?
@@ -100,8 +102,15 @@ class VPhoneMenuController {
var cameraSourceOffItem: NSMenuItem?
var cameraSourceTestPatternItem: NSMenuItem?
var cameraSourceVideoFileItem: NSMenuItem?
+ var cameraSourceSystemCameraItem: NSMenuItem?
var cameraStartStopItem: NSMenuItem?
- weak var captureView: VPhoneVirtualMachineView?
+ weak var captureView: VPhoneVirtualMachineView? {
+ didSet {
+ captureView?.onMouseCaptureChange = { [weak self] captured in
+ self?.mouseCaptureItem?.state = captured ? .on : .off
+ }
+ }
+ }
var batterySyncEnabled = false
var batterySyncStatusItem: NSMenuItem?
var batteryLevelMenuItems: [NSMenuItem] = []
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuDevice.swift b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuDevice.swift
index e3a89a64..5ada6c0e 100644
--- a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuDevice.swift
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuDevice.swift
@@ -121,6 +121,21 @@ extension VPhoneMenuController {
trackpadItem.state = VPhoneTrackpadGestures.isEnabled ? .on : .off
trackpadGesturesItem = trackpadItem
menu.addItem(trackpadItem)
+ let captureItem = makeItem("Capture Mouse", action: #selector(toggleMouseCapture), symbol: "computermouse")
+ captureItem.toolTip = "Forward mouse movement and buttons to the guest. Control–Option–M releases capture."
+ captureItem.state = .off
+ mouseCaptureItem = captureItem
+ menu.addItem(captureItem)
+ menu.addItem(makeItem("Release Mouse", action: #selector(releaseMouseCapture),
+ keyEquivalent: "m", modifiers: [.control, .option]))
+ let externalItem = NSMenuItem(title: "External Display", action: nil, keyEquivalent: "")
+ let externalMenu = NSMenu(title: "External Display")
+ externalMenu.autoenablesItems = false
+ externalMenu.addItem(makeItem("Connect 1080p (1920 × 1080)", action: #selector(connectExternal1080p)))
+ externalMenu.addItem(makeItem("Connect 4K (3840 × 2160)", action: #selector(connectExternal4K)))
+ externalMenu.addItem(makeItem("Disconnect", action: #selector(disconnectExternalDisplay)))
+ externalItem.submenu = externalMenu
+ menu.addItem(externalItem)
// Off unless asked for: the window subtitle then carries the frames the
// guest presented in the last second.
let frameRateItem = makeItem("Show Frame Rate", action: #selector(toggleFrameRateDisplay))
@@ -365,6 +380,34 @@ extension VPhoneMenuController {
captureView?.trackpadGesturesEnabled = enabled
}
+ @objc func toggleMouseCapture() {
+ guard let captureView else { return }
+ captureView.setMouseCaptured(!captureView.isMouseCaptured)
+ }
+
+ @objc func releaseMouseCapture() {
+ captureView?.setMouseCaptured(false)
+ }
+
+ @objc func connectExternal1080p() { connectExternalDisplay(width: 1920, height: 1080) }
+ @objc func connectExternal4K() { connectExternalDisplay(width: 3840, height: 2160) }
+ private func connectExternalDisplay(width: Int, height: Int) {
+ guard !externalDisplayWindow.isConfiguring else { return }
+ Task {
+ do { try await externalDisplayWindow.connect(width: width, height: height) }
+ catch {
+ let alert = NSAlert()
+ alert.messageText = "Could Not Connect External Display"
+ alert.informativeText = error.localizedDescription
+ alert.runModal()
+ }
+ }
+ }
+ @objc func disconnectExternalDisplay() {
+ if externalDisplayWindow.window?.isVisible == true { externalDisplayWindow.close() }
+ else { externalDisplayWindow.disconnect() }
+ }
+
/// Saved to this machine's config.plist; it takes effect from the next
/// guest start and does not unlock the guest now.
@objc func toggleUnlockAtStartup() {
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/VPhoneExternalDisplayWindowController.swift b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/VPhoneExternalDisplayWindowController.swift
new file mode 100644
index 00000000..040154c8
--- /dev/null
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/VPhoneExternalDisplayWindowController.swift
@@ -0,0 +1,85 @@
+import AppKit
+import Foundation
+
+/// A preview of the guest's external display, not a copy of the VM window.
+/// iOS chooses the content: its native mirror or an application's external scene.
+@MainActor
+final class VPhoneExternalDisplayWindowController: NSWindowController, NSWindowDelegate {
+ private weak var control: VPhoneGuestControl?
+ private let imageView = NSImageView()
+ private var frames: Task<Void, Never>?
+ private var generation = UUID()
+ private(set) var isConfiguring = false
+ var onDisconnect: (() -> Void)?
+
+ init(control: VPhoneGuestControl) {
+ self.control = control
+ let window = NSWindow(contentRect: NSRect(x: 0, y: 0, width: 960, height: 540),
+ styleMask: [.titled, .closable, .miniaturizable, .resizable], backing: .buffered, defer: false)
+ window.isReleasedWhenClosed = false
+ window.contentAspectRatio = NSSize(width: 16, height: 9)
+ window.backgroundColor = .black
+ imageView.imageScaling = .scaleProportionallyUpOrDown
+ imageView.autoresizingMask = [.width, .height]
+ imageView.frame = window.contentView?.bounds ?? .zero
+ window.contentView?.addSubview(imageView)
+ super.init(window: window)
+ window.delegate = self
+ window.center()
+ }
+
+ required init?(coder: NSCoder) { fatalError("init(coder:) has not been implemented") }
+
+ func connect(width: Int, height: Int) async throws {
+ guard !isConfiguring, let control else { return }
+ isConfiguring = true
+ defer { isConfiguring = false }
+ _ = try await control.call("display.external.configure", params: ["enabled": true, "width": width, "height": height])
+ frames?.cancel()
+ generation = UUID()
+ let current = generation
+ let title = "External Display — \(width) × \(height)"
+ window?.title = title
+ imageView.image = nil
+ showWindow(nil)
+ window?.makeKeyAndOrderFront(nil)
+ frames = Task { [weak self, weak control] in
+ while !Task.isCancelled {
+ guard let self, let control, generation == current, window?.isVisible == true else { return }
+ let started = ContinuousClock.now
+ do {
+ let frame = try await control.call("display.external.frame")
+ guard !Task.isCancelled, generation == current else { return }
+ guard let encoded = frame["data"] as? String,
+ let data = Data(base64Encoded: encoded), let image = NSImage(data: data)
+ else { throw NSError(domain: "VPhoneExternalDisplay", code: 1,
+ userInfo: [NSLocalizedDescriptionKey: "Invalid display frame"]) }
+ imageView.image = image
+ window?.title = title
+ } catch {
+ if Task.isCancelled { return }
+ window?.title = "\(title) — \(error.localizedDescription)"
+ try? await Task.sleep(for: .milliseconds(500))
+ }
+ // Match the 60 Hz monitor without queuing frames. Slow capture
+ // naturally reduces the rate instead of accumulating latency.
+ let remaining = Duration.nanoseconds(16_666_667) - (ContinuousClock.now - started)
+ if remaining > .zero { try? await Task.sleep(for: remaining) }
+ }
+ }
+ }
+
+ func disconnect() {
+ generation = UUID()
+ frames?.cancel()
+ frames = nil
+ imageView.image = nil
+ let control = control
+ Task {
+ _ = try? await control?.call("display.external.configure", params: ["enabled": false])
+ }
+ onDisconnect?()
+ }
+
+ func windowWillClose(_: Notification) { disconnect() }
+}
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/VPhoneVirtualMachineWindowController.swift b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/VPhoneVirtualMachineWindowController.swift
index 608c19f0..f8050741 100644
--- a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/VPhoneVirtualMachineWindowController.swift
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/VPhoneVirtualMachineWindowController.swift
@@ -357,8 +357,11 @@ class VPhoneVirtualMachineWindowController: NSObject, NSWindowDelegate {
}
})
for name in [NSWindow.didResignKeyNotification, NSWindow.willCloseNotification] {
- keyStateObservers.append(center.addObserver(forName: name, object: window, queue: .main) { _ in
- MainActor.assumeIsolated { VPhoneHostHotKeys.shared.resume() }
+ keyStateObservers.append(center.addObserver(forName: name, object: window, queue: .main) { [weak self] _ in
+ MainActor.assumeIsolated {
+ self?.captureView?.setMouseCaptured(false)
+ VPhoneHostHotKeys.shared.resume()
+ }
})
}
if window.isKeyWindow, !homePodGuest, captureView?.hardwareKeyboardEnabled == true {
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swift b/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swift
index 9950b239..f2588212 100644
--- a/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swift
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swift
@@ -16,6 +16,63 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
/// guest's points for a native scroll.
var screenScale: CGFloat = 3
+ private(set) var isMouseCaptured = false
+ var onMouseCaptureChange: ((Bool) -> Void)?
+ private var capturedMousePoint = NSPoint.zero
+ private var mouseReleaseMonitor: Any?
+
+ /// Explicit per-window capture, never persisted or activated by a click.
+ func setMouseCaptured(_ captured: Bool) {
+ guard captured != isMouseCaptured else { return }
+ if captured {
+ guard window?.isKeyWindow == true, control?.supportsPointerScroll == true else { return }
+ finishScrollTouch()
+ releaseSecondaryPress()
+ endNativeScroll()
+ endPinch()
+ guard CGAssociateMouseAndMouseCursorPosition(0) == .success else { return }
+ capturedMousePoint = NSPoint(x: bounds.midX, y: bounds.midY)
+ if let screenPoint = window?.convertPoint(toScreen: convert(capturedMousePoint, to: nil)),
+ let top = NSScreen.screens.first?.frame.maxY {
+ CGWarpMouseCursorPosition(CGPoint(x: screenPoint.x, y: top - screenPoint.y))
+ }
+ isMouseCaptured = true
+ Self.hiddenCursor.set()
+ control?.movePointer(to: interfacePoint(capturedMousePoint))
+ mouseReleaseMonitor = NSEvent.addLocalMonitorForEvents(matching: .keyDown) { [weak self] event in
+ if event.keyCode == 46,
+ event.modifierFlags.intersection(.deviceIndependentFlagsMask) == [.control, .option],
+ self?.isMouseCaptured == true {
+ self?.setMouseCaptured(false)
+ return nil
+ }
+ return event
+ }
+ } else {
+ if pointerButtons != 0 {
+ pointerButtons = 0
+ control?.setPointer(interfacePoint(capturedMousePoint), buttons: 0)
+ }
+ endNativeScroll()
+ isMouseCaptured = false
+ CGAssociateMouseAndMouseCursorPosition(1)
+ NSCursor.arrow.set()
+ if let monitor = mouseReleaseMonitor { NSEvent.removeMonitor(monitor) }
+ mouseReleaseMonitor = nil
+ }
+ onMouseCaptureChange?(isMouseCaptured)
+ }
+
+ private func advanceCapturedMouse(_ event: NSEvent) {
+ capturedMousePoint.x = min(bounds.maxX, max(bounds.minX, capturedMousePoint.x + event.deltaX))
+ capturedMousePoint.y = min(bounds.maxY, max(bounds.minY,
+ capturedMousePoint.y + (isFlipped ? event.deltaY : -event.deltaY)))
+ }
+
+ private func pointerPoint(_ localPoint: NSPoint) -> CGPoint {
+ interfacePoint(isMouseCaptured ? capturedMousePoint : localPoint)
+ }
+
/// Whether trackpad scroll/pinch gestures are replayed as guest touches.
var trackpadGesturesEnabled = VPhoneTrackpadGestures.isEnabled {
didSet {
@@ -179,12 +236,15 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
/// it follows the mouse, so the guest sees hover as an iPad with a
/// trackpad does.
override func mouseMoved(with event: NSEvent) {
- super.mouseMoved(with: event)
- guard guestPointerFollowsMouse, let control else { return }
+ guard guestPointerFollowsMouse, let control else {
+ super.mouseMoved(with: event)
+ return
+ }
+ advanceCapturedMouse(event)
if hidesHostCursor {
Self.hiddenCursor.set()
}
- control.movePointer(to: interfacePoint(convert(event.locationInWindow, from: nil)))
+ control.movePointer(to: interfacePoint(capturedMousePoint))
}
/// The guest's pointer stands in for the Mac's over the display, so the
@@ -198,7 +258,7 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
}
private var guestPointerFollowsMouse: Bool {
- trackpadGesturesEnabled && control?.supportsPointerScroll == true
+ isMouseCaptured && control?.supportsPointerScroll == true
}
/// Only in the key window: over a window in the background the Mac's
@@ -230,6 +290,7 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
}
override func mouseDragged(with event: NSEvent) {
+ if isMouseCaptured { advanceCapturedMouse(event) }
let localPoint = convert(event.locationInWindow, from: nil)
if dragPointer(to: localPoint) || sendTouchEvent(phase: 1, localPoint: localPoint, timestamp: event.timestamp) {
return
@@ -253,7 +314,7 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
if hidesHostCursor {
Self.hiddenCursor.set()
}
- control.movePointer(to: interfacePoint(localPoint))
+ control.movePointer(to: pointerPoint(localPoint))
}
}
@@ -285,6 +346,7 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
}
override func rightMouseDragged(with event: NSEvent) {
+ if isMouseCaptured { advanceCapturedMouse(event) }
if dragPointer(to: convert(event.locationInWindow, from: nil)) {
return
}
@@ -356,14 +418,14 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
}
releaseSecondaryPress()
pointerButtons |= button
- control.setPointer(interfacePoint(localPoint), buttons: pointerButtons)
+ control.setPointer(pointerPoint(localPoint), buttons: pointerButtons)
return true
}
/// Moves the guest's pointer with its buttons held. False when none is.
private func dragPointer(to localPoint: NSPoint) -> Bool {
guard pointerButtons != 0, let control else { return false }
- control.setPointer(interfacePoint(localPoint), buttons: pointerButtons)
+ control.setPointer(pointerPoint(localPoint), buttons: pointerButtons)
return true
}
@@ -372,7 +434,7 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
private func releasePointer(_ button: Int, at localPoint: NSPoint) -> Bool {
guard pointerButtons & button != 0, let control else { return false }
pointerButtons &= ~button
- control.setPointer(interfacePoint(localPoint), buttons: pointerButtons)
+ control.setPointer(pointerPoint(localPoint), buttons: pointerButtons)
return true
}
@@ -403,7 +465,7 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
/// Replay a trackpad scroll as a guest finger that presses under the pointer
/// and follows the physical direction of the gesture.
override func scrollWheel(with event: NSEvent) {
- if trackpadGesturesEnabled, let control, control.supportsPointerScroll {
+ if guestPointerFollowsMouse, let control {
scrollNatively(event, control: control)
return
}
@@ -485,7 +547,7 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
}
if !nativeScrollActive, delta != .zero {
nativeScrollActive = true
- control.sendScroll(phase: "began", pointer: interfacePoint(convert(event.locationInWindow, from: nil)))
+ control.sendScroll(phase: "began", pointer: pointerPoint(convert(event.locationInWindow, from: nil)))
}
if nativeScrollActive, delta != .zero {
control.sendScroll(phase: "changed", delta: delta)
@@ -1208,25 +1270,13 @@ var panel = normalizeCoordinate(localPoint)
/// outer or inner viewport. Input must be measured against that clipped
/// rectangle; using the full backing bounds makes outer-screen touches
/// land in the hidden part of the framebuffer.
- private var touchViewportSize: CGSize {
- if let active = activePanelPixels, active.width > 0, active.height > 0,
- screenScale > 0 {
- return CGSize(width: active.width / screenScale, height: active.height / screenScale)
- }
- return bounds.size
- }
+ private var touchViewportSize: CGSize { touchViewportBounds.size }
private var touchViewportBounds: CGRect {
- let size = touchViewportSize
- guard size.width > 0, size.height > 0 else { return bounds }
- // VPhoneDisplayContainerView bottom-aligns the full backing view inside
- // the clipped viewport, so the active surface occupies the bottom-left
- // portion of the VM view's local coordinates.
- return CGRect(
- x: 0,
- y: max(0, bounds.height - size.height),
- width: min(bounds.width, size.width),
- height: min(bounds.height, size.height),
+ guard let active = activePanelPixels,
+ let backing = recordingGraphicsDisplay?.sizeInPixels else { return bounds }
+ return VPhoneDisplayGeometry.activeViewportBounds(
+ viewBounds: bounds, activePixels: active, backingPixels: backing
)
}
diff --git a/VPhoneGuestComponents/FoldDisplay/FoldDisplayAdapter.m b/VPhoneGuestComponents/FoldDisplay/FoldDisplayAdapter.m
index 0fdbaa1f..6786e251 100644
--- a/VPhoneGuestComponents/FoldDisplay/FoldDisplayAdapter.m
+++ b/VPhoneGuestComponents/FoldDisplay/FoldDisplayAdapter.m
@@ -3,6 +3,12 @@
#import "FoldDisplayPrivate.h"
+static id vpInnerIdentity;
+
+BOOL vpIsInnerDisplayIdentity(id identity) {
+ return identity && vpInnerIdentity && [identity isEqual:vpInnerIdentity];
+}
+
// MARK: - the inner display and its scenes
BOOL vpIsInnerDisplay(id display) {
@@ -19,7 +25,7 @@ BOOL vpIsInnerDisplay(id display) {
BOOL vpSceneIsOnInnerDisplay(id scene) {
id configuration = vpReadObject(vpReadObject(vpReadObject(scene, "_FBSScene"), "settings"), "displayConfiguration");
id identity = vpReadObject(configuration, "identity");
- return identity && ![[identity description] isEqual:@"Main"];
+ return vpIsInnerDisplayIdentity(identity);
}
id vpSpringBoardWindowScene(BOOL inner) {
@@ -41,7 +47,6 @@ id vpSpringBoardWindowScene(BOOL inner) {
// source already exists" assertion).
static IMP vpOriginalDisplayType, vpOriginalPointScale, vpOriginalExternal, vpOriginalConfigurationInit;
static _Thread_local BOOL vpClassifyingConfiguration;
-static id vpInnerIdentity;
static long long vpInnerDisplayType(id self, SEL selector) {
if (!vpIsInnerDisplay(self)) return ((long long (*)(id, SEL))vpOriginalDisplayType)(self, selector);
@@ -109,7 +114,7 @@ static IMP vpOriginalUpdateScreenIdiom;
static void vpInnerUpdateScreenIdiom(id self, SEL selector) {
((void (*)(id, SEL))vpOriginalUpdateScreenIdiom)(self, selector);
id identity = vpReadObject(vpReadObject(self, "displayConfiguration"), "identity");
- if (!identity || [[identity description] isEqual:@"Main"]) return;
+ if (!vpIsInnerDisplayIdentity(identity)) return;
if (((long long (*)(id, SEL))objc_msgSend)(self, sel_registerName("_userInterfaceIdiom")) != -1) return;
long long idiom = ((long long (*)(id, SEL))objc_msgSend)(vpReadObject(objc_getClass("UIDevice"), "currentDevice"),
sel_registerName("userInterfaceIdiom"));
@@ -129,6 +134,45 @@ static void vpInstallInnerScreenIdiom(void) {
vpOriginalUpdateScreenIdiom = method_setImplementation(update, (IMP)vpInnerUpdateScreenIdiom);
}
+// MARK: - inner screen corners
+
+// The synthetic inner display has no native corner metadata. UIKit therefore
+// supplies zero for its screen, which leaves the expanded Now Playing aperture
+// rectangular. Reuse the device's native main-screen metric only for this
+// display and only when UIKit has no radius of its own. Do not change external
+// screens or replace a future native inner-panel value.
+static IMP vpOriginalScreenCornerRadius, vpOriginalScreenCornerRadiusIgnoringZoom;
+static unsigned int vpCornerRadiusLogCount;
+
+static double vpInnerScreenCornerRadius(id self, SEL selector) {
+ IMP original = selector == sel_registerName("_displayCornerRadiusIgnoringZoom") ?
+ vpOriginalScreenCornerRadiusIgnoringZoom : vpOriginalScreenCornerRadius;
+ double radius = ((double (*)(id, SEL))original)(self, selector);
+ id configuration = vpReadObject(self, "displayConfiguration");
+ id identity = vpReadObject(configuration, "identity");
+ if (!identity || !vpInnerIdentity || ![identity isEqual:vpInnerIdentity]) return radius;
+ id main = vpReadObject(objc_getClass("UIScreen"), "mainScreen");
+ double fallback = main && main != self ? ((double (*)(id, SEL))original)(main, selector) : 0;
+ double result = radius > 0 ? radius : fallback;
+ if (vpCornerRadiusLogCount++ < 4)
+ vpFoldDisplayLog([NSString stringWithFormat:@"inner screen corners: %s native=%.2f main=%.2f effective=%.2f",
+ sel_getName(selector), radius, fallback, result]);
+ return result;
+}
+
+static void vpInstallInnerScreenCorners(void) {
+ Class screen = objc_getClass("UIScreen");
+ Method radius = class_getInstanceMethod(screen, sel_registerName("_displayCornerRadius"));
+ Method unzoomed = class_getInstanceMethod(screen, sel_registerName("_displayCornerRadiusIgnoringZoom"));
+ if (!radius || !unzoomed || strcmp(method_getTypeEncoding(radius), "d16@0:8") != 0 ||
+ strcmp(method_getTypeEncoding(unzoomed), "d16@0:8") != 0) {
+ vpFoldDisplayLog(@"inner screen corners: getter ABI unavailable");
+ return;
+ }
+ vpOriginalScreenCornerRadius = method_setImplementation(radius, (IMP)vpInnerScreenCornerRadius);
+ vpOriginalScreenCornerRadiusIgnoringZoom = method_setImplementation(unzoomed, (IMP)vpInnerScreenCornerRadius);
+}
+
// MARK: - content mode resolution
// SpringBoard's display content mode coordinator starts resolving a window
@@ -232,6 +276,7 @@ static void vpInstallPanelOrientation(void) {
// the type.
void vpInstallInnerDisplayAdapter(void) {
vpInstallInnerScreenIdiom();
+ vpInstallInnerScreenCorners();
vpInstallInnerIdentityEncode();
vpInstallPanelOrientation();
// Type 0 without the FrontBoard scope is a second UIKit-main-like
diff --git a/VPhoneGuestComponents/FoldDisplay/FoldDisplayBackboard.m b/VPhoneGuestComponents/FoldDisplay/FoldDisplayBackboard.m
index 2ee2aff9..7ff2b3e8 100644
--- a/VPhoneGuestComponents/FoldDisplay/FoldDisplayBackboard.m
+++ b/VPhoneGuestComponents/FoldDisplay/FoldDisplayBackboard.m
@@ -14,7 +14,7 @@ static id vpInnerRenderDisplay;
// Tags a render display with the BKS bit the guest itself describes as
// `unknown`. FrontBoard drops an untagged source unless it is always
// connected; this tag is what makes it publish one.
-static void vpTagUnknownDisplay(id display) {
+void vpTagRenderDisplay(id display, NSString *tagName) {
NSString *(*describe)(NSUInteger) = dlsym(RTLD_DEFAULT, "NSStringFromBKSDisplayTags");
SEL setter = sel_registerName("setTag:");
Method method = class_getInstanceMethod(object_getClass(display), setter);
@@ -26,7 +26,7 @@ static void vpTagUnknownDisplay(id display) {
for (unsigned int bit = 0; bit < 24; bit++) {
NSUInteger candidate = (NSUInteger)1 << bit;
NSString *label = describe(candidate);
- if (!label || [label rangeOfString:@"unknown" options:NSCaseInsensitiveSearch].location == NSNotFound) continue;
+ if (!label || [label rangeOfString:tagName options:NSCaseInsensitiveSearch].location == NSNotFound) continue;
if (unknown) { vpFoldDisplayLog(@"inner display: ambiguous unknown tag bit"); return; }
unknown = candidate;
}
@@ -81,8 +81,9 @@ static BOOL vpAddInnerRenderDisplayToServer(id server) {
Method enable = class_getInstanceMethod(object_getClass(display), setEnabled);
if (enable && strcmp(method_getTypeEncoding(enable), "v20@0:8B16") == 0)
((void (*)(id, SEL, BOOL))objc_msgSend)(display, setEnabled, YES);
- vpTagUnknownDisplay(display);
+ vpTagRenderDisplay(display, @"unknown");
vpInnerRenderDisplay = display;
+ vpPrepareExternalRenderDisplays(server);
vpFoldDisplayLog([NSString stringWithFormat:@"inner display: render display added %@ uniqueId=%@", display,
[display respondsToSelector:sel_registerName("uniqueId")] ? [display valueForKey:@"uniqueId"] : @"-"]);
return YES;
@@ -217,7 +218,14 @@ static void vpTurnDigitizerPosition(VPHIDEventRef event) {
}
static void vpInnerHandleDirectTouch(id self, SEL selector, void *event, id service) {
- if (event && vpInnerPanelLit()) vpTurnDigitizerPosition((VPHIDEventRef)event);
+ // Only the built-in digitizer is repurposed for the inner panel. CarPlay
+ // and other external displays already supply their own display UUID and
+ // coordinate space. Ask the original resolver, before our routing hook
+ // replaces the built-in digitizer's empty UUID with the inner display's.
+ id originalDisplayUUID = ((id (*)(id, SEL, id))vpOriginalDigitizerDisplayUUID)(self,
+ sel_registerName("_queue_displayUUIDForDigitizerService:"), service);
+ if (event && ![originalDisplayUUID length] && vpInnerPanelLit())
+ vpTurnDigitizerPosition((VPHIDEventRef)event);
((void (*)(id, SEL, void *, id))vpOriginalHandleDirectTouch)(self, selector, event, service);
}
diff --git a/VPhoneGuestComponents/FoldDisplay/FoldDisplayExternal.m b/VPhoneGuestComponents/FoldDisplay/FoldDisplayExternal.m
new file mode 100644
index 00000000..5bc186a7
--- /dev/null
+++ b/VPhoneGuestComponents/FoldDisplay/FoldDisplayExternal.m
@@ -0,0 +1,213 @@
+// Opt-in external monitor. The render server owns the display; FrontBoard
+// receives an AirPlay/external identity and chooses mirroring or app scenes.
+#import "FoldDisplayPrivate.h"
+#include <notify.h>
+
+static NSString *const vpExternalName = @"VPhone External";
+static NSString *const vpExternalRequest = @"/var/mobile/Library/Caches/vphone-external-display.request.plist";
+static NSString *const vpExternalStatus = @"/var/mobile/Library/Caches/vphone-external-display.status.plist";
+static id vpExternalRenderDisplay;
+static id vpExternal1080Display, vpExternal4KDisplay;
+static int vpExternalNotificationToken;
+static int vpExternalInspectionToken;
+static int vpExternalRefreshToken;
+static IMP vpExternalOriginalType, vpExternalOriginalScale, vpExternalOriginalExternal;
+static IMP vpExternalOriginalCloning;
+
+static BOOL vpIsExternalMonitor(id display) {
+ return [vpReadObject(display, "name") hasPrefix:vpExternalName] ||
+ [vpReadObject(display, "deviceName") hasPrefix:vpExternalName];
+}
+static long long vpExternalType(id self, SEL cmd) {
+ return vpIsExternalMonitor(self) ? 2 : ((long long (*)(id, SEL))vpExternalOriginalType)(self, cmd);
+}
+static NSUInteger vpExternalScale(id self, SEL cmd) {
+ if (!vpIsExternalMonitor(self)) return ((NSUInteger (*)(id, SEL))vpExternalOriginalScale)(self, cmd);
+ CGRect bounds = ((CGRect (*)(id, SEL))objc_msgSend)(self, sel_registerName("bounds"));
+ return bounds.size.width >= 3840 ? 2 : 1;
+}
+static BOOL vpExternalIsExternal(id self, SEL cmd) {
+ return vpIsExternalMonitor(self) ? YES : ((BOOL (*)(id, SEL))vpExternalOriginalExternal)(self, cmd);
+}
+static BOOL vpExternalCloning(id self, SEL cmd) {
+ return vpIsExternalMonitor(self) ? YES : ((BOOL (*)(id, SEL))vpExternalOriginalCloning)(self, cmd);
+}
+
+static void vpRefreshExternalDisplaySources(void) {
+ // Virtual render objects do not drive the hardware display observer's
+ // refresh. Update the existing CADisplay objects so their native KVO
+ // observers publish tag/mode changes to SpringBoard's existing sources.
+ // Keep the objects: replacing the cached array would lose those observers.
+ NSArray *displays = vpReadObject(objc_getClass("CADisplay"), "displays");
+ NSString *activeName = [NSDictionary dictionaryWithContentsOfFile:vpExternalStatus][@"name"];
+ id active = nil;
+ for (id display in displays) {
+ if (!vpIsExternalMonitor(display)) continue;
+ if ([vpReadObject(display, "name") isEqual:activeName]) { active = display; continue; }
+ Method update = class_getInstanceMethod(object_getClass(display), sel_registerName("update"));
+ if (update && strcmp(method_getTypeEncoding(update), "v16@0:8") == 0)
+ ((void (*)(id, SEL))objc_msgSend)(display, sel_registerName("update"));
+ }
+ // Publish old-screen disconnection before the newly selected connection.
+ if (active) dispatch_async(dispatch_get_main_queue(), ^{
+ Method update = class_getInstanceMethod(object_getClass(active), sel_registerName("update"));
+ if (update && strcmp(method_getTypeEncoding(update), "v16@0:8") == 0)
+ ((void (*)(id, SEL))objc_msgSend)(active, sel_registerName("update"));
+ });
+}
+
+// Read-only inspection in the process that owns native display/scene policy.
+// An app can see the BKS source without SpringBoard admitting an app scene.
+static void vpInspectExternalScenePolicy(void) {
+ NSMutableDictionary *report = [@{@"updated": @(NSDate.date.timeIntervalSince1970)} mutableCopy];
+ @try {
+ id app = vpReadObject(objc_getClass("UIApplication"), "sharedApplication");
+ id monitor = vpReadObject(objc_getClass("FBDisplayManager"), "sharedInstance");
+ report[@"monitor"] = [monitor description] ?: @"unavailable";
+ report[@"transformer"] = [vpReadObject(monitor, "transformer") description] ?: @"unavailable";
+ NSMutableArray *scenes = [NSMutableArray array];
+ for (id scene in [vpReadObject(app, "connectedScenes") allObjects] ?: @[]) {
+ id screen = vpReadObject(scene, "screen");
+ id fbs = vpReadObject(scene, "_FBSScene");
+ id configuration = vpReadObject(vpReadObject(fbs, "settings"), "displayConfiguration");
+ [scenes addObject:@{@"class": NSStringFromClass([scene class]),
+ @"scene": [scene description] ?: @"", @"screen": [screen description] ?: @"",
+ @"configuration": [configuration description] ?: @"",
+ @"windows": [vpReadObject(scene, "windows") description] ?: @""}];
+ }
+ report[@"scenes"] = scenes;
+ report[@"sceneManager"] = [vpReadObject(vpReadObject(objc_getClass("FBSceneManager"), "sharedInstance"), "debugDescription") description] ?: @"unavailable";
+ NSMutableDictionary *owners = [NSMutableDictionary dictionary];
+ for (Class cls = [app class]; cls; cls = class_getSuperclass(cls)) {
+ unsigned int count = 0;
+ Ivar *ivars = class_copyIvarList(cls, &count);
+ for (unsigned int i = 0; i < count; i++) {
+ const char *name = ivar_getName(ivars[i]);
+ const char *type = ivar_getTypeEncoding(ivars[i]);
+ if (type[0] != '@' || (!strcasestr(name, "display") && !strcasestr(name, "mirror") && !strcasestr(name, "airplay"))) continue;
+ id value = object_getIvar(app, ivars[i]);
+ if (value) owners[@(name)] = [value description] ?: @"";
+ }
+ free(ivars);
+ }
+ report[@"owners"] = owners;
+ unsigned int count = 0;
+ Class *classes = objc_copyClassList(&count);
+ NSMutableDictionary *surface = [NSMutableDictionary dictionary];
+ for (unsigned int i = 0; i < count; i++) {
+ const char *name = class_getName(classes[i]);
+ if (strncmp(name, "SB", 2) || (!strstr(name, "ExternalDisplay") && !strstr(name, "AirPlay") && !strstr(name, "Mirroring"))) continue;
+ NSMutableArray *methods = [NSMutableArray array];
+ for (int meta = 0; meta < 2; meta++) {
+ unsigned int methodCount = 0;
+ Method *list = class_copyMethodList(meta ? object_getClass(classes[i]) : classes[i], &methodCount);
+ for (unsigned int j = 0; j < methodCount; j++)
+ [methods addObject:[NSString stringWithFormat:@"%s%s %s", meta ? "+" : "-", sel_getName(method_getName(list[j])), method_getTypeEncoding(list[j])]];
+ free(list);
+ }
+ surface[@(name)] = methods;
+ }
+ free(classes);
+ report[@"externalPolicyClasses"] = surface;
+ } @catch (NSException *exception) {
+ report[@"error"] = exception.reason ?: exception.name;
+ }
+ [report writeToFile:@"/var/mobile/Library/Caches/vphone-external-display.springboard.plist" atomically:YES];
+}
+
+void vpPrepareExternalRenderDisplays(id server) {
+ if (!server || (vpExternal1080Display && vpExternal4KDisplay)) return;
+ @try {
+ Class cls = objc_getClass("CAWindowServerVirtualDisplay");
+ Method initializer = class_getInstanceMethod(cls, sel_registerName("initWithOptions:"));
+ if (!initializer || strcmp(method_getTypeEncoding(initializer), "@24@0:8@16") != 0) return;
+ for (int index = 0; index < 2; index++) {
+ if (index ? vpExternal4KDisplay != nil : vpExternal1080Display != nil) continue;
+ NSInteger width = index ? 3840 : 1920, height = index ? 2160 : 1080;
+ NSString *name = index ? @"VPhone External 4K" : @"VPhone External 1080p";
+ id display = ((id (*)(id, SEL, id))objc_msgSend)([cls alloc], sel_registerName("initWithOptions:"), @{
+ @"kCAVirtualDisplayWidth": @(width), @"kCAVirtualDisplayHeight": @(height),
+ @"kCAVirtualDisplayUpdateRate": @60, @"kCAVirtualDisplayName": name,
+ });
+ if (!display) continue;
+ ((void (*)(id, SEL, id))objc_msgSend)(server, sel_registerName("addDisplay:"), display);
+ ((void (*)(id, SEL, long long))objc_msgSend)(display, sel_registerName("setTag:"), 0);
+ ((void (*)(id, SEL, BOOL))objc_msgSend)(display, sel_registerName("setEnabled:"), NO);
+ if (index) vpExternal4KDisplay = display; else vpExternal1080Display = display;
+ vpFoldDisplayLog([NSString stringWithFormat:@"external display: prepared disconnected %@", display]);
+ }
+ } @catch (NSException *exception) {
+ vpFoldDisplayLog([NSString stringWithFormat:@"external display: prepare failed %@", exception.reason]);
+ }
+}
+
+static void vpApplyExternalRequest(void) {
+ NSDictionary *request = [NSDictionary dictionaryWithContentsOfFile:vpExternalRequest];
+ if (![request[@"nonce"] isKindOfClass:NSString.class]) return;
+ NSMutableDictionary *status = [request mutableCopy];
+ @try {
+ id server = vpReadObject(objc_getClass("CAWindowServer"), "serverIfRunning");
+ if (!server) @throw [NSException exceptionWithName:@"ExternalDisplay" reason:@"Render server unavailable" userInfo:nil];
+ for (id display in @[vpExternal1080Display ?: NSNull.null, vpExternal4KDisplay ?: NSNull.null]) {
+ if (display == NSNull.null) continue;
+ // Stable render objects are enumerated before SpringBoard starts.
+ // Only their connection tags and enabled state change at runtime.
+ ((void (*)(id, SEL, long long))objc_msgSend)(display, sel_registerName("setTag:"), 0);
+ ((void (*)(id, SEL, BOOL))objc_msgSend)(display, sel_registerName("setEnabled:"), NO);
+ }
+ vpExternalRenderDisplay = nil;
+ if ([request[@"enabled"] boolValue]) {
+ NSInteger width = [request[@"width"] integerValue], height = [request[@"height"] integerValue];
+ if (!((width == 1920 && height == 1080) || (width == 3840 && height == 2160)))
+ @throw [NSException exceptionWithName:@"ExternalDisplay" reason:@"Only 1080p and 4K are supported" userInfo:nil];
+ id display = width == 3840 ? vpExternal4KDisplay : vpExternal1080Display;
+ if (!display) @throw [NSException exceptionWithName:@"ExternalDisplay" reason:@"External render display was not prepared at boot" userInfo:nil];
+ // FrontBoard clears a non-always-connected source with no tag.
+ // Publish the source before native external-role classification.
+ vpTagRenderDisplay(display, @"AirPlay");
+ ((void (*)(id, SEL, BOOL))objc_msgSend)(display, sel_registerName("setEnabled:"), YES);
+ vpExternalRenderDisplay = display;
+ status[@"name"] = width == 3840 ? @"VPhone External 4K" : @"VPhone External 1080p";
+ vpFoldDisplayLog([NSString stringWithFormat:@"external display: connected %ldx%ld %@", (long)width, (long)height, display]);
+ } else {
+ vpFoldDisplayLog(@"external display: disconnected");
+ }
+ status[@"ok"] = @YES;
+ } @catch (NSException *exception) {
+ status[@"ok"] = @NO;
+ status[@"error"] = exception.reason ?: exception.name;
+ status[@"enabled"] = @(vpExternalRenderDisplay != nil);
+ vpFoldDisplayLog([NSString stringWithFormat:@"external display: %@", status[@"error"]]);
+ }
+ [status writeToFile:vpExternalStatus atomically:YES];
+ notify_post("com.vphone.external-display.changed");
+}
+
+void vpInstallExternalDisplay(BOOL serverProcess) {
+ Class cls = objc_getClass("CADisplay");
+ const char *names[] = {"displayType", "pointScale", "isExternal", "isCloningSupported"};
+ const char *types[] = {"q16@0:8", "Q16@0:8", "B16@0:8", "B16@0:8"};
+ IMP replacements[] = {(IMP)vpExternalType, (IMP)vpExternalScale, (IMP)vpExternalIsExternal, (IMP)vpExternalCloning};
+ IMP *originals[] = {&vpExternalOriginalType, &vpExternalOriginalScale, &vpExternalOriginalExternal, &vpExternalOriginalCloning};
+ for (int i = 0; i < 4; i++) {
+ Method method = class_getInstanceMethod(cls, sel_registerName(names[i]));
+ if (!method || strcmp(method_getTypeEncoding(method), types[i]) != 0) {
+ vpFoldDisplayLog(@"external display: CADisplay ABI unavailable");
+ return;
+ }
+ }
+ for (int i = 0; i < 4; i++)
+ *originals[i] = method_setImplementation(class_getInstanceMethod(cls, sel_registerName(names[i])), replacements[i]);
+ if (!serverProcess) {
+ notify_register_dispatch("com.vphone.external-display.changed", &vpExternalRefreshToken,
+ dispatch_get_main_queue(), ^(int token) { vpRefreshExternalDisplaySources(); });
+ notify_register_dispatch("com.vphone.external-display.inspect", &vpExternalInspectionToken,
+ dispatch_get_main_queue(), ^(int token) { vpInspectExternalScenePolicy(); });
+ return;
+ }
+ // Ignore stale requests at boot: only an explicit host connect activates.
+ [@{@"ok": @YES, @"enabled": @NO} writeToFile:vpExternalStatus atomically:YES];
+ uint32_t result = notify_register_dispatch("com.vphone.external-display.configure", &vpExternalNotificationToken,
+ dispatch_get_main_queue(), ^(int token) { vpApplyExternalRequest(); });
+ vpFoldDisplayLog([NSString stringWithFormat:@"external display: request listener %u", result]);
+}
diff --git a/VPhoneGuestComponents/FoldDisplay/FoldDisplayPrivate.h b/VPhoneGuestComponents/FoldDisplay/FoldDisplayPrivate.h
index fff3f000..568ea752 100644
--- a/VPhoneGuestComponents/FoldDisplay/FoldDisplayPrivate.h
+++ b/VPhoneGuestComponents/FoldDisplay/FoldDisplayPrivate.h
@@ -30,10 +30,14 @@ id vpReadObjectIvar(id object, const char *name);
// MARK: - FoldDisplayBackboard.m
void vpInstallInnerRenderDisplay(void);
+void vpTagRenderDisplay(id display, NSString *tagName);
// MARK: - FoldDisplayAdapter.m
BOOL vpIsInnerDisplay(id display);
+BOOL vpIsInnerDisplayIdentity(id identity);
+void vpInstallExternalDisplay(BOOL serverProcess);
+void vpPrepareExternalRenderDisplays(id server);
BOOL vpSceneIsOnInnerDisplay(id scene);
/// The SpringBoard window scene (not an aperture subclass) on the inner
/// display, or on Main.
diff --git a/VPhoneGuestComponents/FoldDisplay/FoldDisplayStatusBar.m b/VPhoneGuestComponents/FoldDisplay/FoldDisplayStatusBar.m
index 469acb3b..94815daf 100644
--- a/VPhoneGuestComponents/FoldDisplay/FoldDisplayStatusBar.m
+++ b/VPhoneGuestComponents/FoldDisplay/FoldDisplayStatusBar.m
@@ -60,7 +60,8 @@ static Class vpDualStatusClassLookup(id screen, id info) {
if (![NSThread isMainThread]) return original(screen, info);
id identity = vpReadObject(vpReadObject(screen, "displayConfiguration"), "identity");
if (!identity) return original(screen, info);
- BOOL inner = ![[identity description] isEqual:@"Main"];
+ BOOL inner = vpIsInnerDisplayIdentity(identity);
+ if (!inner && ![[identity description] isEqual:@"Main"]) return original(screen, info);
Class selected = vpDuoPanelStatusClass(screen, info, inner);
if (vpStatusLookupLogCount++ < 8)
vpFoldDisplayLog([NSString stringWithFormat:@"status bar: %@ screen takes %@", inner ? @"inner" : @"cover",
diff --git a/VPhoneGuestComponents/FoldDisplay/libvphonefolddisplay.m b/VPhoneGuestComponents/FoldDisplay/libvphonefolddisplay.m
index 83b7f2fc..a1a16f4d 100644
--- a/VPhoneGuestComponents/FoldDisplay/libvphonefolddisplay.m
+++ b/VPhoneGuestComponents/FoldDisplay/libvphonefolddisplay.m
@@ -15,8 +15,10 @@ __attribute__((constructor)) static void vpInstallFoldDisplay(void) {
if (sysctlbyname("hw.machine", product, &productLength, NULL, 0) != 0 || strcmp(product, "iPhone19,4") != 0)
return;
if (strcmp(getprogname(), "backboardd") == 0) {
+ vpInstallExternalDisplay(YES);
vpInstallInnerRenderDisplay();
} else if (strcmp(getprogname(), "SpringBoard") == 0) {
+ vpInstallExternalDisplay(NO);
vpInstallSharedPanel();
vpInstallInnerDisplayAdapter();
vpInstallDualStatusLookup();
diff --git a/VPhoneGuestComponents/MISFix/MISFixDetour.c b/VPhoneGuestComponents/MISFix/MISFixDetour.c
index a1adc7e6..625264b9 100644
--- a/VPhoneGuestComponents/MISFix/MISFixDetour.c
+++ b/VPhoneGuestComponents/MISFix/MISFixDetour.c
@@ -37,7 +37,7 @@
#define kDetourBytes (kDetourWords * 4u)
/// Room for one trampoline: four displaced instructions, each of which may
-/// expand to five words when it has to be rewritten, plus the four-word jump
+/// expand to eight words when it has to be rewritten, plus the four-word jump
/// back.
#define kTrampolineBytes 128u
@@ -95,6 +95,19 @@ static int vpIsTerminator(uint32_t insn) {
/// PC-relative in a way this does not handle — in which case the whole detour
/// is abandoned, because half a relocation is a corrupted function.
static unsigned vpRelocate(uint32_t insn, uint64_t pc, uint32_t *out) {
+ // PACIBSPPC binds LR to both SP and this instruction's original PC.
+ // Executing it verbatim in the trampoline would sign with the wrong PC.
+ // x15-x17 are caller-saved scratch at function entry; the installer below
+ // accepts this expansion only as the first displaced instruction.
+ if (insn == 0xDAC1A7FEu) {
+ out[0] = 0xAA1E03F1u; // mov x17, x30
+ out[1] = 0x910003F0u; // mov x16, sp
+ vpEmitAbsolute(out + 2, 15, pc);
+ out[6] = 0xDAC18FFEu; // pacib171615: sign x17 with SP=x16, PC=x15
+ out[7] = 0xAA1103FEu; // mov x30, x17
+ return 8;
+ }
+
// ADR / ADRP. Both put a PC-relative address in a register, and both
// become "materialise that same absolute address".
if ((insn & 0x1F000000u) == 0x10000000u) {
@@ -197,6 +210,9 @@ MISFixDetourResult MISFixDetour(
for (unsigned index = 0; index < kDetourWords; index += 1) {
if (index + 1 < kDetourWords && vpIsTerminator(displaced[index]))
return MISFixDetourTooShort;
+ // Do not clobber scratch values established by an earlier prologue.
+ if (displaced[index] == 0xDAC1A7FEu && index != 0)
+ return MISFixDetourUnrelocatable;
uint64_t start = (uint64_t)(uintptr_t)target;
uint64_t pc = start + index * 4u;
unsigned written = (displaced[index] & 0xFF000010u) == 0x54000000u
diff --git a/VPhoneKit/VPhoneCoreKit/VirtualMachine/VPhoneDisplayGeometry.swift b/VPhoneKit/VPhoneCoreKit/VirtualMachine/VPhoneDisplayGeometry.swift
index f1f3dd04..de1b516c 100644
--- a/VPhoneKit/VPhoneCoreKit/VirtualMachine/VPhoneDisplayGeometry.swift
+++ b/VPhoneKit/VPhoneCoreKit/VirtualMachine/VPhoneDisplayGeometry.swift
@@ -47,6 +47,18 @@ public struct VPhoneDisplayGeometry: Equatable, Sendable {
)
}
+ /// The visible panel inside a scaled backing view. Pixel dimensions are
+ /// ratios: guest points differ from AppKit points after a window resize.
+ public static func activeViewportBounds(
+ viewBounds: CGRect, activePixels: CGSize, backingPixels: CGSize
+ ) -> CGRect {
+ guard activePixels.width > 0, activePixels.height > 0,
+ backingPixels.width > 0, backingPixels.height > 0 else { return viewBounds }
+ let width = viewBounds.width * min(1, activePixels.width / backingPixels.width)
+ let height = viewBounds.height * min(1, activePixels.height / backingPixels.height)
+ return CGRect(x: viewBounds.minX, y: viewBounds.maxY - height, width: width, height: height)
+ }
+
/// A view point as a 0...1 display position with y down. Points on the
/// letterbox bars clamp to the nearest display edge.
///
diff --git a/VPhoneKit/VPhoneCoreKitTests/VirtualMachine/DisplayGeometryTests.swift b/VPhoneKit/VPhoneCoreKitTests/VirtualMachine/DisplayGeometryTests.swift
index 19eca4f2..ac501887 100644
--- a/VPhoneKit/VPhoneCoreKitTests/VirtualMachine/DisplayGeometryTests.swift
+++ b/VPhoneKit/VPhoneCoreKitTests/VirtualMachine/DisplayGeometryTests.swift
@@ -49,6 +49,27 @@ struct DisplayGeometryTests {
#expect(geometry.displayRect == Self.fullScreen)
}
+ @Test func `resized Duo cover maps visible edges and centre`() {
+ let pixels = CGSize(width: 1398, height: 2034)
+ for scale in [0.5, 1.0, 717.0 / 678.0, 1.5] {
+ let view = CGRect(x: 0, y: 0, width: 466 * scale, height: 678 * scale)
+ let viewport = VPhoneDisplayGeometry.activeViewportBounds(
+ viewBounds: view, activePixels: pixels, backingPixels: pixels
+ )
+ let geometry = VPhoneDisplayGeometry(viewBounds: viewport, displaySize: pixels, isFlipped: false)
+ #expect(Self.close(geometry.normalizedPoint(CGPoint(x: view.minX, y: view.maxY)), .zero))
+ #expect(Self.close(geometry.normalizedPoint(CGPoint(x: view.maxX, y: view.minY)), CGPoint(x: 1, y: 1)))
+ #expect(Self.close(geometry.normalizedPoint(CGPoint(x: view.midX, y: view.midY)), CGPoint(x: 0.5, y: 0.5)))
+ }
+ }
+
+ @Test func `cropped viewport scales with the backing window`() {
+ #expect(VPhoneDisplayGeometry.activeViewportBounds(
+ viewBounds: CGRect(x: 0, y: 0, width: 800, height: 1200),
+ activePixels: CGSize(width: 400, height: 600), backingPixels: CGSize(width: 800, height: 1200)
+ ) == CGRect(x: 0, y: 600, width: 400, height: 600))
+ }
+
// MARK: - Normalizing
@Test func `full screen display edges normalize to 0 and 1`() {
diff --git a/VPhoneLaunchpad/Build/SignLaunchpad.sh b/VPhoneLaunchpad/Build/SignLaunchpad.sh
index 4a027820..c153710e 100644
--- a/VPhoneLaunchpad/Build/SignLaunchpad.sh
+++ b/VPhoneLaunchpad/Build/SignLaunchpad.sh
@@ -7,9 +7,9 @@
# zsh VPhoneLaunchpad/Build/SignLaunchpad.sh <path/to/vphone-launchpad.app> "Developer ID Application: …"
#
# The helper, the command line tool and the VM launcher are signed before the
-# app that seals them, without entitlements. The app gets the two hardened
+# app that seals them, without entitlements. The app gets the hardened
# runtime resource entitlements in Resources/VPhoneLaunchpad.entitlements
-# (microphone and location) and no private ones. They are applied only here:
+# (microphone, camera and location) and no private ones. They are applied only here:
# CODE_SIGNING_ALLOWED is NO at build time, so no xcconfig sets
# CODE_SIGN_ENTITLEMENTS. The launcher is the process macOS holds responsible
# for each VM Launchpad starts; as a tool in the app's Contents/MacOS it is
@@ -42,7 +42,7 @@ codesign --force --options runtime --timestamp --entitlements "$entitlements" \
codesign --verify --strict --deep "$app"
signed=$(codesign -d --entitlements - --xml "$app" 2>/dev/null)
-for key in com.apple.security.device.audio-input com.apple.security.personal-information.location; do
+for key in com.apple.security.device.audio-input com.apple.security.device.camera com.apple.security.personal-information.location; do
if [[ $signed != *"<key>$key</key><true/>"* ]]; then
print -u2 "error: $app is not signed with $key."
exit 1
diff --git a/VPhoneLaunchpad/Resources/VPhoneLaunchpad.Info.plist b/VPhoneLaunchpad/Resources/VPhoneLaunchpad.Info.plist
index 251bc7de..e3c5056b 100644
--- a/VPhoneLaunchpad/Resources/VPhoneLaunchpad.Info.plist
+++ b/VPhoneLaunchpad/Resources/VPhoneLaunchpad.Info.plist
@@ -49,6 +49,10 @@
guest records. Same arrangement as the location descriptions above. -->
<key>NSMicrophoneUsageDescription</key>
<string>Shares this Mac's microphone with the virtual iPhone so apps running inside it can record.</string>
+ <key>NSCameraUsageDescription</key>
+ <string>Shares this Mac's camera with the virtual iPhone while System Camera streaming is enabled.</string>
+ <key>NSCameraUseContinuityCameraDeviceType</key>
+ <true/>
<!-- A machine exported by Launchpad: a gnutar of the machine folder,
zstd compressed, the same archive `vphone-cli vm export` writes.
Launchpad owns the type, so opening one in Finder imports it. -->
diff --git a/VPhoneLaunchpad/Resources/VPhoneLaunchpad.entitlements b/VPhoneLaunchpad/Resources/VPhoneLaunchpad.entitlements
index e3f7d746..6dbc64f2 100644
--- a/VPhoneLaunchpad/Resources/VPhoneLaunchpad.entitlements
+++ b/VPhoneLaunchpad/Resources/VPhoneLaunchpad.entitlements
@@ -4,15 +4,18 @@
applied only by Build/SignLaunchpad.sh: the Xcode build does not sign.
Guests are started through Contents/MacOS/vphone-launchpad-launcher, so
macOS attributes their privacy requests to this app. Under the hardened
- runtime it refuses the microphone or location to an app without the
+ runtime it refuses microphone, camera or location access to an app without the
matching entitlement, without asking. The microphone is used by guest
- audio input; location by VPhoneLocation.app, which vphone-vm starts as a
+ audio input; camera by the optional System Camera frame source;
+ location by VPhoneLocation.app, which vphone-vm starts as a
child when Sync Host Location is on. The helper, the command line tool
and the launcher get no entitlements. -->
<plist version="1.0">
<dict>
<key>com.apple.security.device.audio-input</key>
<true/>
+ <key>com.apple.security.device.camera</key>
+ <true/>
<key>com.apple.security.personal-information.location</key>
<true/>
</dict>