Run a virtual iPhone on an Apple Silicon Mac.
Swift
Click with the guest's pointer on iPad guests
Jacky4ca4902a9b19
diff --git a/Research/vphoned_http_api.md b/Research/vphoned_http_api.mdindex 4c734138..d483badf 100644--- a/Research/vphoned_http_api.md+++ b/Research/vphoned_http_api.md@@ -372,7 +372,7 @@ request carries `"force": true`.| --- | --- || Device | `device.snapshot`, `device.info` (snapshot plus network, screen, rotation, brightness, volume, low power, Developer Mode, agent, `device_name` `{name, own_name}`: the pinned name and the guest's own ComputerName), `device.screen`, `device.network`, `device.ioreg {plane}`, `device.environment`, `device.basebin {archive?}` || Display, audio | `display.brightness {value?}`, `display.rotation {orientation?}`, `display.orientation` (`{degrees, source}`: SpringBoard's interface orientation without a screen capture, for polling; `degrees` is null while SpringBoard has no answer; capability `display_orientation`), `display.rotation_lock {locked}`, `display.auto_lock` (`{auto_lock_seconds, never, lock_screen_minimum_seconds}`: Auto-Lock and the Lock Screen timeout vphoned keeps in step with it, see below; capability `display_auto_lock`), `screen.unlock {passcode?, timeout?}` (`{locked, screen_off, was_locked, was_screen_off}`: the display on and the Lock Screen passed, see below; capability `screen_unlock`), `audio.volume {value?, category?}`, `audio.state` |-| Input | `input.touch`, `input.hid`, `input.button {name}`, `input.key {name}`, `input.type {text, delay_ms?}`, `input.paste {text}`, `input.tap`, `input.double_tap`, `input.long_press`, `input.swipe`, `input.drag {points}`, `input.touch_sequence {events}` — gesture coordinates are screen points; `input.pointer {x, y, seq?}`, `input.scroll {phase, dx?, dy?, x?, y?}` (capability `pointer_scroll`, see below) |+| Input | `input.touch`, `input.hid`, `input.button {name}`, `input.key {name}`, `input.type {text, delay_ms?}`, `input.paste {text}`, `input.tap`, `input.double_tap`, `input.long_press`, `input.swipe`, `input.drag {points}`, `input.touch_sequence {events}` — gesture coordinates are screen points; `input.pointer {x, y, seq?, buttons?}`, `input.scroll {phase, dx?, dy?, x?, y?}` (capabilities `pointer_scroll` and `pointer_buttons`, see below) || UI | `ui.tree` (alias `accessibility.tree`), `ui.element_at`, `ui.tap_element`, `ui.wait`, `ui.wait_gone`, `ui.ocr {languages?, min_confidence?}`, `ui.describe`, `screen.screenshot` || Processes | `processes.list {filter?}`, `processes.kill {pid, signal?}` **force**, `memory.jetsam`, `memory.pressure` (only the three kernel memory sysctls, for polling) || launchd | `services.list`, `status`, `print`, `dump`, `disabled`, `start`, `enable`, `load`, `profile` (see below); `services.stop`, `disable`, `remove`, `signal`, `unload`, `profile.apply {profile, groups?, allow?}` **force**; `launchd.getenv`, `setenv`, `unsetenv`. On tvOS, which never enables user sessions, they read the system domain alone; before icli 0.8.6 the foreground-user lookup trapped there and killed vphoned |@@ -464,32 +464,53 @@ offers the last fix again every 10 s until the guest takes it, since the Macsends a new one only when it moves. A preset or a replay pauses sync for thatrun without changing the setting.-`input.pointer` and `input.scroll` (capability `pointer_scroll`, iPad guests-only; REST `POST /v1/input/pointer`, `POST /v1/input/scroll`) scroll the way an-iPad with a trackpad does, instead of with a dragging finger. On first use-vphoned creates a virtual pointing device (a HID user device with an absolute-position, `VPhoneDaemon/Native/vphoned_pointer.m`); iPadOS then shows a pointer,+`input.pointer` and `input.scroll` (capabilities `pointer_scroll` and+`pointer_buttons`, iPad guests only; REST `POST /v1/input/pointer`,+`POST /v1/input/scroll`) point, click and scroll the way an iPad with a+trackpad does, instead of with a finger. They drive IcliKit's virtual+pointing device (`pointerMove`, `pointerScroll`; a HID user device with an+absolute position, `Sources/IcliPrivate/Pointer.m` in icli), which vphoned+creates on first use and keeps while it runs; iPadOS then shows a pointer,which hides itself after a few seconds without movement.-- `input.pointer {x, y, seq?}` puts the pointer at x, y: 0...1 from the- top-left corner of the interface as it is shown, in any orientation.- Requests run concurrently, so a client that sends moves without waiting- for each answer numbers them with an increasing `seq` (a whole number up- to 2^53); a move that lands after a higher one is dropped, and still- answers. Without `seq` every move is applied.+- `input.pointer {x, y, seq?, buttons?}` puts the pointer at x, y: 0...1 from+ the top-left corner of the interface as it is shown, in any orientation.+ `buttons` are the buttons held from that move on, a whole number 0...7: bit+ 0 primary, bit 1 secondary, bit 2 middle; without it none are held. Holding+ one is a pointer click, which iPadOS answers as a trackpad's, not as a+ touch. Requests run concurrently, so a client that sends moves without+ waiting for each answer numbers them with an increasing `seq` (a whole+ number up to 2^53); a move that lands after a higher one is dropped,+ buttons and all, and still answers. Without `seq` every move is applied. A+ client therefore sends button changes in order, and no move numbered past+ one until it has been answered.- `input.scroll {phase, dx?, dy?, x?, y?}` sends one scroll event to the view- under the pointer, after moving the pointer to x, y when they are given.+ under the pointer, after moving the pointer to x, y when they are given; the+ buttons held stay as they are.`phase` is `began`, `changed`, `ended` or `cancelled`: `began` first, a`changed` per delta, then `ended`. `dx` and `dy` are interface points with abrowser wheel event's signs, positive `dy` toward the end of the content,under the guest's natural scrolling setting (on by default); each is at most10000.+A click under a visible pointer has to be the pointer's. iPadOS ignores a+touch on an alert's button while the pointer hovers it, until the pointer has+hidden itself: on an iPadOS 26.6.2 guest a touch within 0.3 s of the pointer+moving onto SpringBoard's Remove App Cancel button failed 6 times out of 6;+with a Mac click sent as `buttons` the VM window closed it on the first+click every time, after resting on the button. The window therefore sends a+Mac click as `buttons` on a `pointer_buttons` guest, and keeps touches for+`vphone.sock`'s `tap` and `swipe` and the back gesture. IcliKit sends a+button change as a button event rather than a report bit: backboardd never+releases a button that a report pressed, and a stuck button loses every+later click of every pointing device.+The view moves by the deltas and by nothing else: iPadOS does not run onafter `ended`, so a client that wants a trackpad's momentum sends it as more`changed` deltas first. The first few points of a scroll start the gesture-and do not move the view (about 15 points on iPadOS 26). Both answer-`invalid_operation` when the device cannot be made or is not ready yet.+and do not move the view (about 15 points on iPadOS 26). Both answer with+IcliKit's error, `unavailable` or `failed`, when the device cannot be made or+is not ready yet.`time.timezone` (capability `timezone`; REST `GET/PUT /v1/timezone`) returns`{identifier, automatic, seconds_from_gmt}`: the Olson namediff --git a/VPhoneDaemon/Daemon/GuestAPI+Pointer.swift b/VPhoneDaemon/Daemon/GuestAPI+Pointer.swiftnew file mode 100644index 00000000..262495e1--- /dev/null+++ b/VPhoneDaemon/Daemon/GuestAPI+Pointer.swift@@ -0,0 +1,43 @@+import Foundation+import IcliKit++// MARK: - Virtual Pointer++/// `input.pointer` and `input.scroll` go to IcliKit's virtual pointing device,+/// which lives as long as vphoned. iPadOS shows its pointer, and a button held+/// in a move is a pointer click, as a trackpad's is.+extension GuestAPI {+ private static let pointerLock = NSLock()+ /// The highest `seq` applied, under `pointerLock`.+ private nonisolated(unsafe) static var lastPointerSequence: UInt64 = 0++ /// Requests run concurrently, so a move sent earlier can land later;+ /// putting the pointer back where it was would make it jump. A move whose+ /// nonzero `sequence` is not above the last one applied is dropped, buttons+ /// and all, and still answers: a client sends button changes in order and+ /// nothing numbered past them until they are answered.+ static func movePointer(_ point: GuestPointerScroll.Point, sequence: UInt64, buttons: Int32) throws {+ pointerLock.lock()+ defer { pointerLock.unlock() }+ if sequence != 0 {+ guard sequence > lastPointerSequence else { return }+ lastPointerSequence = sequence+ }+ _ = try pointerMove(x: point.x, y: point.y, normalized: true, buttons: PointerButtons(rawValue: Int(buttons)))+ }++ /// One scroll event to the view under the pointer, after putting the+ /// pointer at the scroll's position when it has one. The buttons held stay+ /// as they are.+ static func scrollPointer(_ scroll: GuestPointerScroll.Scroll) throws {+ pointerLock.lock()+ defer { pointerLock.unlock() }+ if let point = scroll.pointer {+ _ = try pointerMove(x: point.x, y: point.y, normalized: true)+ }+ guard let phase = ScrollPhase(rawValue: scroll.phase.rawValue) else {+ throw GuestAPIError.invalidRequest("phase must be began, changed, ended or cancelled")+ }+ _ = try pointerScroll(phase, dx: scroll.dx, dy: scroll.dy)+ }+}diff --git a/VPhoneDaemon/Daemon/GuestAPI.swift b/VPhoneDaemon/Daemon/GuestAPI.swiftindex c7afc0db..e7d6e795 100644--- a/VPhoneDaemon/Daemon/GuestAPI.swift+++ b/VPhoneDaemon/Daemon/GuestAPI.swift@@ -155,7 +155,7 @@ enum GuestAPI {/// iPadOS takes a pointing device as it is; iOS on an iPhone only through/// AssistiveTouch, so `input.pointer` and `input.scroll` are an iPad's.private static func pointerCapabilities() -> [String] {- GuestVirtualAudioProduct.machine().hasPrefix("iPad") ? ["pointer_scroll"] : []+ GuestVirtualAudioProduct.machine().hasPrefix("iPad") ? ["pointer_scroll", "pointer_buttons"] : []}static func execute(method: String, params: [String: Any]) throws -> [String: Any] {@@ -337,14 +337,16 @@ enum GuestAPI {case "input.pointer":let point: GuestPointerScroll.Pointlet sequence: UInt64+ let buttons: Int32do {point = try GuestPointerScroll.point(params)sequence = try GuestPointerScroll.sequence(params)+ buttons = try GuestPointerScroll.buttons(params)} catch {throw GuestAPIError.invalidRequest(error.message)}- try pointerResult(vp_pointer_move(point.x, point.y, sequence))- return ["x": point.x, "y": point.y]+ try movePointer(point, sequence: sequence, buttons: buttons)+ return ["x": point.x, "y": point.y, "buttons": Int(buttons)]case "input.scroll":let scroll: GuestPointerScroll.Scrolldo {@@ -352,10 +354,7 @@ enum GuestAPI {} catch {throw GuestAPIError.invalidRequest(error.message)}- if let point = scroll.pointer {- try pointerResult(vp_pointer_move(point.x, point.y, 0))- }- try pointerResult(vp_pointer_scroll(scroll.phase.bits, scroll.dx, scroll.dy))+ try scrollPointer(scroll)return ["phase": scroll.phase.rawValue]case "input.hid":let page = try integer(params, "page")@@ -620,17 +619,6 @@ enum GuestAPI {/// Digitizer phase codes for vphoned's own multi-finger injection path./// icli keeps its equivalent mapping module-internal, and these are the/// values the private digitizer entry points take: 0 down, 1 move, 3 up.- /// A pointer call's result as an error, nothing when it worked.- static func pointerResult(_ code: Int32) throws {- switch Int(code) {- case Int(VP_POINTER_OK): return- case Int(VP_POINTER_UNAVAILABLE): throw GuestAPIError.operationFailed("this base has no virtual pointing device")- case Int(VP_POINTER_REFUSED): throw GuestAPIError.operationFailed("the guest refused to create the virtual pointing device")- case Int(VP_POINTER_NOT_READY): throw GuestAPIError.operationFailed("the virtual pointing device is not ready yet")- default: throw GuestAPIError.operationFailed("the pointer event could not be delivered")- }- }-static func touchPhaseCode(_ phase: TouchPhase) -> Int32 {switch phase {case .down: 0diff --git a/VPhoneDaemon/Daemon/GuestPointerScroll.swift b/VPhoneDaemon/Daemon/GuestPointerScroll.swiftindex c580c368..81084e42 100644--- a/VPhoneDaemon/Daemon/GuestPointerScroll.swift+++ b/VPhoneDaemon/Daemon/GuestPointerScroll.swift@@ -2,7 +2,8 @@ import Foundation// MARK: - Pointer Scroll-/// The parameters of `input.pointer` and `input.scroll`.+/// The parameters of `input.pointer` and `input.scroll`, which IcliKit's+/// virtual pointer carries out (`GuestAPI+Pointer.swift`).////// A scroll goes in phases, the way a trackpad's does: `began` before the/// first delta, `changed` for each delta, then `ended` or `cancelled`. iPadOS@@ -14,21 +15,11 @@ import Foundation/// Nothing here touches the guest, so `VPhoneDaemon/Tests/run-logic-tests.sh`/// builds this file on the Mac.enum GuestPointerScroll {- enum Phase: String {+ enum Phase: String, CaseIterable {case begancase changedcase endedcase cancelled-- /// The `IOHIDEventPhaseBits` value the scroll event carries.- var bits: Int32 {- switch self {- case .began: 1- case .changed: 2- case .ended: 4- case .cancelled: 8- }- }}/// A position in the interface as it is shown, 0...1 from the top-left@@ -79,6 +70,17 @@ enum GuestPointerScroll {return UInt64(sequence)}+ /// `input.pointer`'s optional `buttons`, the buttons held from that move+ /// on: bit 0 primary, bit 1 secondary, bit 2 middle. None held when it is+ /// absent, so a client that only hovers never presses one.+ static func buttons(_ params: [String: Any]) throws(Problem) -> Int32 {+ guard let value = params["buttons"] else { return 0 }+ guard let buttons = number(value), (0 ... 7).contains(buttons), buttons.rounded() == buttons else {+ throw Problem(message: "buttons must be a whole number within 0...7")+ }+ return Int32(buttons)+ }+static func scroll(_ params: [String: Any]) throws(Problem) -> Scroll {guard let name = params["phase"] as? String, let phase = Phase(rawValue: name) else {throw Problem(message: "phase must be began, changed, ended or cancelled")diff --git a/VPhoneDaemon/Native/Include/VphonedNative.h b/VPhoneDaemon/Native/Include/VphonedNative.hindex b07db652..07a07561 100644--- a/VPhoneDaemon/Native/Include/VphonedNative.h+++ b/VPhoneDaemon/Native/Include/VphonedNative.h@@ -27,29 +27,6 @@ bool vp_hid_load(void);/// `touchSequence` can carry two fingers at once, which is why this exists.void vp_hid_touch2(int phase, double x1, double y1, double x2, double y2);-/// What vp_pointer_move and vp_pointer_scroll return.-enum {- VP_POINTER_OK = 0,- /// The base does not have the private IOKit symbols.- VP_POINTER_UNAVAILABLE = 1,- /// IOHIDUserDeviceCreate refused to make the device.- VP_POINTER_REFUSED = 2,- /// The device exists but iOS has not put an event service over it yet.- VP_POINTER_NOT_READY = 3,- /// The report or the event could not be made or delivered.- VP_POINTER_FAILED = 4,-};--/// Put the virtual pointer at x, y: 0...1 from the top-left corner of the-/// interface as it is shown. Creates the pointing device on first use. A-/// nonzero `sequence` not above the last one applied is a move that arrived-/// late: it is dropped and the call still succeeds. 0 always moves.-int vp_pointer_move(double x, double y, uint64_t sequence);--/// One scroll event at the pointer. `phase` is an IOHIDEventPhaseBits value;-/// dx and dy are interface points, with a browser wheel event's signs.-int vp_pointer_scroll(int phase, double dx, double dy);-/// Turn the display on without pressing a button. False when/// SpringBoardServices has no SBSUndimScreen.bool vp_screen_undim(void);diff --git a/VPhoneDaemon/Native/vphoned_pointer.m b/VPhoneDaemon/Native/vphoned_pointer.mdeleted file mode 100644index 55711e14..00000000--- a/VPhoneDaemon/Native/vphoned_pointer.m+++ /dev/null@@ -1,194 +0,0 @@-/*- * vphoned_pointer — a virtual pointing device, for native scrolling.- *- * A scroll made of touches is a finger that drags and lets go, and how far the- * content runs on afterwards is the guest's guess at the finger's speed from- * the times the moves arrive. Over a network those times bunch up, so the- * run-on changes from one swipe to the next. A pointing device scrolls the way- * an iPad with a trackpad does: the view under the pointer moves by exactly- * the deltas given, and nothing runs on that the client did not send.- *- * The device is a HID user device that reports an absolute position, so- * iPadOS shows its pointer and puts it where a report says, in the interface's- * own orientation (measured on iPadOS 26 in portrait and landscape); the- * pointer hides itself when it has been still for a few seconds. Scroll- * events are dispatched with that device's event service as their sender:- * with any other sender (the touchscreen's included) they reach nothing, or- * the view at the top-left corner. The device is created on first use and- * lives as long as vphoned.- *- * Every entry point is resolved with dlsym, like vphoned_touch.m.- */--#import "Include/VphonedNative.h"--#include <dlfcn.h>-#include <mach/mach_time.h>-#include <math.h>-#include <unistd.h>--typedef void *IOHIDEventSystemClientRef;-typedef void *IOHIDEventRef;-typedef void *IOHIDUserDeviceRef;-typedef double IOHIDFloat;-typedef unsigned int vp_io_object_t;--static IOHIDEventSystemClientRef (*pClientCreate)(CFAllocatorRef);-static void (*pDispatch)(IOHIDEventSystemClientRef, IOHIDEventRef);-static void (*pSetSender)(IOHIDEventRef, uint64_t);-static IOHIDEventRef (*pScroll)(CFAllocatorRef, uint64_t, IOHIDFloat, IOHIDFloat, IOHIDFloat, uint32_t);-static void (*pSetPhase)(IOHIDEventRef, uint32_t);-static IOHIDUserDeviceRef (*pDeviceCreate)(CFAllocatorRef, CFDictionaryRef);-static int (*pHandleReport)(IOHIDUserDeviceRef, const uint8_t *, CFIndex);-static CFMutableDictionaryRef (*pMatching)(const char *);-static int (*pMatchingServices)(unsigned int, CFDictionaryRef, vp_io_object_t *);-static vp_io_object_t (*pNext)(vp_io_object_t);-static CFTypeRef (*pSearchProperty)(vp_io_object_t, const char *, CFStringRef, CFAllocatorRef, uint32_t);-static int (*pEntryID)(vp_io_object_t, uint64_t *);-static int (*pRelease)(vp_io_object_t);--static dispatch_queue_t gPointerQueue;-static bool gPointerLoaded;-static IOHIDEventSystemClientRef gPointerClient;-static IOHIDUserDeviceRef gPointerDevice;-static uint64_t gPointerSender;-static uint64_t gPointerLastSequence;--static NSString *const kPointerProduct = @"vphone pointer";--// Three buttons (never pressed), an absolute X and Y within 0...32767, a wheel-// and a horizontal pan (never turned). Without the wheel and the pan iPadOS-// shows the pointer but sends the scroll events nowhere.-static const uint8_t kPointerDescriptor[] = {- 0x05, 0x01, 0x09, 0x02, 0xA1, 0x01, 0x09, 0x01, 0xA1, 0x00,- 0x05, 0x09, 0x19, 0x01, 0x29, 0x03, 0x15, 0x00, 0x25, 0x01, 0x95, 0x03, 0x75, 0x01, 0x81, 0x02,- 0x95, 0x01, 0x75, 0x05, 0x81, 0x03,- 0x05, 0x01, 0x09, 0x30, 0x09, 0x31, 0x15, 0x00, 0x26, 0xFF, 0x7F, 0x75, 0x10, 0x95, 0x02, 0x81, 0x02,- 0x09, 0x38, 0x15, 0x81, 0x25, 0x7F, 0x75, 0x08, 0x95, 0x01, 0x81, 0x06,- 0x05, 0x0C, 0x0A, 0x38, 0x02, 0x15, 0x81, 0x25, 0x7F, 0x75, 0x08, 0x95, 0x01, 0x81, 0x06,- 0xC0, 0xC0,-};-static const double kPointerMaximum = 32767;--// kIORegistryIterateRecursively | kIORegistryIterateParents-#define VP_SEARCH_UP 3u--static bool load_symbols(void) {- if (gPointerLoaded) return true;- void *ioKit = dlopen("/System/Library/Frameworks/IOKit.framework/IOKit", RTLD_NOW);- if (!ioKit) return false;- pClientCreate = dlsym(ioKit, "IOHIDEventSystemClientCreate");- pDispatch = dlsym(ioKit, "IOHIDEventSystemClientDispatchEvent");- pSetSender = dlsym(ioKit, "IOHIDEventSetSenderID");- pScroll = dlsym(ioKit, "IOHIDEventCreateScrollEvent");- pSetPhase = dlsym(ioKit, "IOHIDEventSetPhase");- pDeviceCreate = dlsym(ioKit, "IOHIDUserDeviceCreate");- pHandleReport = dlsym(ioKit, "IOHIDUserDeviceHandleReport");- pMatching = dlsym(ioKit, "IOServiceMatching");- pMatchingServices = dlsym(ioKit, "IOServiceGetMatchingServices");- pNext = dlsym(ioKit, "IOIteratorNext");- pSearchProperty = dlsym(ioKit, "IORegistryEntrySearchCFProperty");- pEntryID = dlsym(ioKit, "IORegistryEntryGetRegistryEntryID");- pRelease = dlsym(ioKit, "IOObjectRelease");- gPointerLoaded = pClientCreate && pDispatch && pSetSender && pScroll && pSetPhase && pDeviceCreate &&- pHandleReport && pMatching && pMatchingServices && pNext && pSearchProperty && pEntryID &&- pRelease;- return gPointerLoaded;-}--// The registry ID of the event service iOS put over the device, which is the-// sender its own events carry.-static uint64_t find_sender(void) {- vp_io_object_t iterator = 0;- if (pMatchingServices(0, pMatching("IOHIDEventService"), &iterator) != 0) return 0;- uint64_t found = 0;- vp_io_object_t service;- while ((service = pNext(iterator))) {- CFTypeRef product = pSearchProperty(service, "IOService", CFSTR("Product"), kCFAllocatorDefault, VP_SEARCH_UP);- if (!found && product && CFGetTypeID(product) == CFStringGetTypeID() &&- CFStringCompare(product, (__bridge CFStringRef)kPointerProduct, 0) == kCFCompareEqualTo) {- pEntryID(service, &found);- }- if (product) CFRelease(product);- pRelease(service);- }- pRelease(iterator);- return found;-}--// On gPointerQueue. The event service appears a moment after the device;-// waits up to a second for it.-static int ensure_device(void) {- if (gPointerSender) return VP_POINTER_OK;- if (!load_symbols()) return VP_POINTER_UNAVAILABLE;- if (!gPointerClient) gPointerClient = pClientCreate(kCFAllocatorDefault);- if (!gPointerClient) return VP_POINTER_UNAVAILABLE;- if (!gPointerDevice) {- NSDictionary *properties = @{- @"ReportDescriptor": [NSData dataWithBytes:kPointerDescriptor length:sizeof(kPointerDescriptor)],- @"VendorID": @0x05AC,- @"ProductID": @0x7650,- @"Product": kPointerProduct,- @"Transport": @"USB",- @"PrimaryUsagePage": @1,- @"PrimaryUsage": @2,- };- gPointerDevice = pDeviceCreate(kCFAllocatorDefault, (__bridge CFDictionaryRef)properties);- if (!gPointerDevice) {- NSLog(@"vphoned: IOHIDUserDeviceCreate refused the pointer device");- return VP_POINTER_REFUSED;- }- }- for (int attempt = 0; attempt < 50 && !gPointerSender; attempt++) {- gPointerSender = find_sender();- if (!gPointerSender) usleep(20000);- }- return gPointerSender ? VP_POINTER_OK : VP_POINTER_NOT_READY;-}--static dispatch_queue_t pointer_queue(void) {- static dispatch_once_t once;- dispatch_once(&once, ^{- dispatch_queue_attr_t attributes = dispatch_queue_attr_make_with_qos_class(- DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INTERACTIVE, 0);- gPointerQueue = dispatch_queue_create("com.vphone.vphoned.pointer", attributes);- });- return gPointerQueue;-}--int vp_pointer_move(double x, double y, uint64_t sequence) {- __block int result;- dispatch_sync(pointer_queue(), ^{- result = ensure_device();- if (result != VP_POINTER_OK) return;- // Requests run concurrently, so a move sent earlier can land later;- // putting the pointer back where it was would make it jump.- if (sequence) {- if (sequence <= gPointerLastSequence) return;- gPointerLastSequence = sequence;- }- uint16_t px = (uint16_t)lround(fmin(fmax(x, 0), 1) * kPointerMaximum);- uint16_t py = (uint16_t)lround(fmin(fmax(y, 0), 1) * kPointerMaximum);- const uint8_t report[7] = {0, px & 0xFF, px >> 8, py & 0xFF, py >> 8, 0, 0};- if (pHandleReport(gPointerDevice, report, sizeof(report)) != 0) result = VP_POINTER_FAILED;- });- return result;-}--int vp_pointer_scroll(int phase, double dx, double dy) {- __block int result;- dispatch_sync(pointer_queue(), ^{- result = ensure_device();- if (result != VP_POINTER_OK) return;- IOHIDEventRef event = pScroll(kCFAllocatorDefault, mach_absolute_time(), dx, dy, 0, 0);- if (!event) {- result = VP_POINTER_FAILED;- return;- }- pSetPhase(event, (uint32_t)phase);- pSetSender(event, gPointerSender);- pDispatch(gPointerClient, event);- CFRelease(event);- });- return result;-}diff --git a/VPhoneDaemon/Tests/LogicTests.swift b/VPhoneDaemon/Tests/LogicTests.swiftindex 83678445..57c9033d 100644--- a/VPhoneDaemon/Tests/LogicTests.swift+++ b/VPhoneDaemon/Tests/LogicTests.swift@@ -365,10 +365,8 @@ enum LogicTests {Scroll(phase: .began, dx: 0, dy: 0, pointer: nil), "deltas default to 0")expectEqual(try? GuestPointerScroll.scroll(["phase": "began", "x": 0, "y": 1]),Scroll(phase: .began, dx: 0, dy: 0, pointer: .init(x: 0, y: 1)), "a pointer position comes with it")- expectEqual(GuestPointerScroll.Phase.began.bits, 1)- expectEqual(GuestPointerScroll.Phase.changed.bits, 2)- expectEqual(GuestPointerScroll.Phase.ended.bits, 4)- expectEqual(GuestPointerScroll.Phase.cancelled.bits, 8)+ expectEqual(GuestPointerScroll.Phase.allCases.map(\.rawValue), ["began", "changed", "ended", "cancelled"],+ "the names IcliKit's ScrollPhase takes")let phase = "phase must be began, changed, ended or cancelled"expectEqual(scrollProblem([:]), phase)expectEqual(scrollProblem(["phase": "move"]), phase)@@ -390,5 +388,11 @@ enum LogicTests {for bad: Any in [-1, 1.5, true, "7", 9_007_199_254_740_994.0] {expect((try? GuestPointerScroll.sequence(["seq": bad])) == nil, "seq \(bad) is refused")}+ expectEqual(try? GuestPointerScroll.buttons([:]), 0, "no buttons held when there are none")+ expectEqual(try? GuestPointerScroll.buttons(["buttons": 1]), 1)+ expectEqual(try? GuestPointerScroll.buttons(["buttons": 7]), 7)+ for bad: Any in [-1, 8, 1.5, true, "1"] {+ expect((try? GuestPointerScroll.buttons(["buttons": bad])) == nil, "buttons \(bad) is refused")+ }}}diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/GuestCommunication/VPhoneGuestControl.swift b/VPhoneExecutable/VPhoneVirtualization/UI/GuestCommunication/VPhoneGuestControl.swiftindex f86fd7d2..19cc98cc 100644--- a/VPhoneExecutable/VPhoneVirtualization/UI/GuestCommunication/VPhoneGuestControl.swift+++ b/VPhoneExecutable/VPhoneVirtualization/UI/GuestCommunication/VPhoneGuestControl.swift@@ -191,6 +191,8 @@ final class VPhoneGuestControl {orderedInput = nilopenScroll = nilpendingPointer = nil+ openPointerPress = nil+ heldPointerButtons = 0orderedLocation?.cancel()orderedLocation = nildevice = nil@@ -348,9 +350,20 @@ final class VPhoneGuestControl {isConnected && guestCapabilities.contains("pointer_scroll")}+ /// Whether a button held in a pointer move is a click (`input.pointer`'s+ /// `buttons`), so a Mac click can be the guest pointer's.+ var supportsPointerButtons: Bool {+ isConnected && guestCapabilities.contains("pointer_buttons")+ }+@ObservationIgnored private var pointersInFlight = 0@ObservationIgnored private var pendingPointer: CGPoint?@ObservationIgnored private var lastPointerSequence: UInt64 = 0+ /// The buttons the last `setPointer` holds down.+ @ObservationIgnored private var heldPointerButtons = 0+ /// `setPointer` requests not yet answered.+ @ObservationIgnored private var pointerPressesQueued = 0+ @ObservationIgnored private var openPointerPress: PendingPointerPress?/// Moves in flight at once. Each request is its own connection; waiting/// for one answer before the next put a round trip behind every move.@@ -361,28 +374,84 @@ final class VPhoneGuestControl {/// on the order of touches, and a mouse that keeps moving would hold every/// click behind it. Moves go out at once, numbered so that vphoned drops/// one that lands after a later one; past `maximumPointersInFlight` the- /// newest position waits for the next free slot.+ /// newest position waits for the next free slot. While a button is down,+ /// or its release has not been answered, the newest position waits too: a+ /// move holds no buttons, and one numbered past a press would release it.func movePointer(to point: CGPoint) {- guard pointersInFlight < Self.maximumPointersInFlight else {+ guard pointersInFlight < Self.maximumPointersInFlight, heldPointerButtons == 0, pointerPressesQueued == 0+ else {pendingPointer = pointreturn}- // Microseconds of uptime: increasing across reconnects and restarts of- // this process, which a guest that keeps running may still remember.- let now = UInt64(ProcessInfo.processInfo.systemUptime * 1_000_000)- lastPointerSequence = max(now, lastPointerSequence + 1)- let params: [String: Any] = ["x": Double(point.x), "y": Double(point.y), "seq": lastPointerSequence]+ let params: [String: Any] = ["x": Double(point.x), "y": Double(point.y), "seq": nextPointerSequence()]pointersInFlight += 1Task {_ = try? await call("input.pointer", params: params)pointersInFlight -= 1- if let next = pendingPointer {- pendingPointer = nil- movePointer(to: next)+ movePendingPointer()+ }+ }++ /// The position a press or a drag still waiting in the input queue sends.+ private final class PendingPointerPress {+ var point: CGPoint+ let buttons: Int++ init(point: CGPoint, buttons: Int) {+ self.point = point+ self.buttons = buttons+ }+ }++ /// Puts the guest's pointer at `point` with `buttons` held, bit 0 primary+ /// and bit 1 secondary: a click that is the pointer's, as a trackpad's is,+ /// and not a touch. In the ordered input queue, after the keys and touches+ /// sent before it, so no press or release is reordered or dropped. A drag+ /// that keeps the same buttons down joins the move still waiting there,+ /// so a guest slower than the mouse gets fewer moves and loses no release.+ func setPointer(_ point: CGPoint, buttons: Int) {+ heldPointerButtons = buttons+ // A hover from before the press would land after it.+ pendingPointer = nil+ if let open = openPointerPress, open.buttons == buttons {+ open.point = point+ return+ }+ let open = PendingPointerPress(point: point, buttons: buttons)+ openPointerPress = open+ pointerPressesQueued += 1+ let previous = orderedInput+ orderedInput = Task {+ await previous?.value+ if openPointerPress === open {+ openPointerPress = nil+ }+ if !Task.isCancelled {+ let params: [String: Any] = [+ "x": Double(open.point.x), "y": Double(open.point.y), "buttons": open.buttons,+ "seq": nextPointerSequence(),+ ]+ do { _ = try await call("input.pointer", params: params) } catch { print("[control] input.pointer: \(error)") }}+ pointerPressesQueued -= 1+ movePendingPointer()}}+ private func movePendingPointer() {+ guard let next = pendingPointer else { return }+ pendingPointer = nil+ movePointer(to: next)+ }++ /// Microseconds of uptime: increasing across reconnects and restarts of+ /// this process, which a guest that keeps running may still remember.+ private func nextPointerSequence() -> UInt64 {+ let now = UInt64(ProcessInfo.processInfo.systemUptime * 1_000_000)+ lastPointerSequence = max(now, lastPointerSequence + 1)+ return lastPointerSequence+ }+/// The `changed` deltas waiting in the input queue, added up until that/// request goes out.private final class PendingScroll {diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swift b/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swiftindex 82da26c1..421438a9 100644--- a/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swift+++ b/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swift@@ -199,21 +199,16 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {override func mouseDown(with event: NSEvent) {// Clicking the VM display should always restore keyboard focus.window?.makeFirstResponder(self)- // A synthetic trackpad finger would fight this touch for the same slot.- finishScrollTouch()- if nativeScrollActive {- endNativeScroll()- nativeScrollMomentumDropped = true+ let localPoint = convert(event.locationInWindow, from: nil)+ if pressPointer(Self.primaryPointerButton, at: localPoint) {+ return}// A click is a touch, and iPadOS hides its pointer under a finger: the// Mac's shows where the drag is until the mouse moves freely again.if guestPointerFollowsMouse {NSCursor.arrow.set()}- releaseSecondaryPress()- let localPoint = convert(event.locationInWindow, from: nil)- currentTouchSwipeAim = hitTestEdge(at: localPoint)- if sendTouchEvent(phase: 0, localPoint: localPoint, timestamp: event.timestamp) {+ if beginTouch(at: localPoint, timestamp: event.timestamp) {return}super.mouseDown(with: event)@@ -221,7 +216,7 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {override func mouseDragged(with event: NSEvent) {let localPoint = convert(event.locationInWindow, from: nil)- if sendTouchEvent(phase: 1, localPoint: localPoint, timestamp: event.timestamp) {+ if dragPointer(to: localPoint) || sendTouchEvent(phase: 1, localPoint: localPoint, timestamp: event.timestamp) {return}super.mouseDragged(with: event)@@ -229,10 +224,15 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {override func mouseUp(with event: NSEvent) {let localPoint = convert(event.locationInWindow, from: nil)- if !sendTouchEvent(phase: 3, localPoint: localPoint, timestamp: event.timestamp) {+ if releasePointer(Self.primaryPointerButton, at: localPoint) {+ if hidesHostCursor {+ Self.hiddenCursor.set()+ }+ return+ }+ if !endTouch(at: localPoint, timestamp: event.timestamp) {super.mouseUp(with: event)}- currentTouchSwipeAim = 0// Bring the guest's pointer back where the finger lifted.if guestPointerFollowsMouse, let control {if hidesHostCursor {@@ -245,15 +245,20 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {// MARK: - Secondary Click/// A right click is iPadOS's secondary click: the context menu under the- /// pointer. The guest has a touchscreen and no pointer, so it is replayed- /// as a touch-and-hold, which is what opens the same menu by finger. A- /// click released before the hold registers is kept down until it has.- /// Holding and dragging carries on as a drag, as a finger would.+ /// pointer. Where the guest's pointer follows the mouse it is that+ /// pointer's secondary button. Elsewhere the guest has a touchscreen and+ /// no pointer, so it is replayed as a touch-and-hold, which is what opens+ /// the same menu by finger. A click released before the hold registers is+ /// kept down until it has. Holding and dragging carries on as a drag, as a+ /// finger would.override func rightMouseDown(with event: NSEvent) {window?.makeFirstResponder(self)+ let localPoint = convert(event.locationInWindow, from: nil)+ if pressPointer(Self.secondaryPointerButton, at: localPoint) {+ return+ }finishScrollTouch()releaseSecondaryPress()- let localPoint = convert(event.locationInWindow, from: nil)// A hold is never an edge swipe, wherever it lands.currentTouchSwipeAim = 0guard sendTouchEvent(phase: 0, localPoint: localPoint, timestamp: event.timestamp) else {@@ -265,6 +270,9 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {}override func rightMouseDragged(with event: NSEvent) {+ if dragPointer(to: convert(event.locationInWindow, from: nil)) {+ return+ }guard secondaryPressPoint != nil, secondaryReleaseWork == nil else {super.rightMouseDragged(with: event)return@@ -275,6 +283,9 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {}override func rightMouseUp(with event: NSEvent) {+ if releasePointer(Self.secondaryPointerButton, at: convert(event.locationInWindow, from: nil)) {+ return+ }guard secondaryPressPoint != nil, secondaryReleaseWork == nil else {super.rightMouseUp(with: event)return@@ -303,6 +314,75 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {sendTouchEvent(phase: 3, localPoint: point, timestamp: ProcessInfo.processInfo.systemUptime)}+ // MARK: - Pointer Clicks++ private static let primaryPointerButton = 1+ private static let secondaryPointerButton = 2++ /// The buttons Mac clicks hold down on the guest's pointer.+ private var pointerButtons = 0++ /// On a guest whose pointer follows the mouse a click is that pointer's,+ /// as on an iPad with a trackpad. A touch is the wrong input there: iPadOS+ /// ignores a touch on an alert's button while the pointer hovers it, until+ /// the pointer hides itself a few seconds after it last moved.+ private var clicksWithGuestPointer: Bool {+ guestPointerFollowsMouse && control?.supportsPointerButtons == true+ }++ /// Presses `button` on the guest's pointer at `localPoint`, ending any+ /// scroll or touch first. False when a click is a touch on this guest.+ private func pressPointer(_ button: Int, at localPoint: NSPoint) -> Bool {+ guard clicksWithGuestPointer, let control else { return false }+ finishScrollTouch()+ if nativeScrollActive {+ endNativeScroll()+ nativeScrollMomentumDropped = true+ }+ releaseSecondaryPress()+ pointerButtons |= button+ control.setPointer(interfacePoint(localPoint), buttons: pointerButtons)+ return true+ }++ /// Moves the guest's pointer with its buttons held. False when none is.+ private func dragPointer(to localPoint: NSPoint) -> Bool {+ guard pointerButtons != 0, let control else { return false }+ control.setPointer(interfacePoint(localPoint), buttons: pointerButtons)+ return true+ }++ /// Releases `button`. False when the guest's pointer is not holding it, so+ /// a release whose press was a touch lifts the finger instead.+ private func releasePointer(_ button: Int, at localPoint: NSPoint) -> Bool {+ guard pointerButtons & button != 0, let control else { return false }+ pointerButtons &= ~button+ control.setPointer(interfacePoint(localPoint), buttons: pointerButtons)+ return true+ }++ // MARK: - Touches++ /// Puts a finger down at `localPoint`, after ending every other input that+ /// would fight it for the same slot. False when there is no touch path.+ @discardableResult+ private func beginTouch(at localPoint: NSPoint, timestamp: TimeInterval) -> Bool {+ finishScrollTouch()+ if nativeScrollActive {+ endNativeScroll()+ nativeScrollMomentumDropped = true+ }+ releaseSecondaryPress()+ currentTouchSwipeAim = hitTestEdge(at: localPoint)+ return sendTouchEvent(phase: 0, localPoint: localPoint, timestamp: timestamp)+ }++ @discardableResult+ private func endTouch(at localPoint: NSPoint, timestamp: TimeInterval) -> Bool {+ defer { currentTouchSwipeAim = 0 }+ return sendTouchEvent(phase: 3, localPoint: localPoint, timestamp: timestamp)+ }+// MARK: - Trackpad Gestures/// Replay a trackpad scroll as a guest finger that presses under the pointer@@ -826,22 +906,9 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {))}- /// Synthesize an NSEvent at a given window point.- private func synthesizeMouseEvent(type: NSEvent.EventType, at windowPoint: NSPoint) -> NSEvent? {- NSEvent.mouseEvent(- with: type,- location: windowPoint,- modifierFlags: [],- timestamp: ProcessInfo.processInfo.systemUptime,- windowNumber: window?.windowNumber ?? 0,- context: nil,- eventNumber: 0,- clickCount: type == .leftMouseUp ? 0 : 1,- pressure: type == .leftMouseUp ? 0.0 : 1.0,- )- }-/// Inject a tap at pixel coordinates (matching screenshot image dimensions).+ /// Always a touch, even where a Mac click is the guest pointer's: a tap+ /// stands for a finger, and the pointer shows only once the mouse moves.func injectTap(pixelX: Double, pixelY: Double, screenWidth: Int, screenHeight: Int) {let localPoint = pixelToLocal(pixelX: pixelX,@@ -849,20 +916,13 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {screenWidth: screenWidth,screenHeight: screenHeight,)- let windowPoint = convert(localPoint, to: nil)-- if let downEvent = synthesizeMouseEvent(type: .leftMouseDown, at: windowPoint) {- mouseDown(with: downEvent)- }+ beginTouch(at: localPoint, timestamp: ProcessInfo.processInfo.systemUptime)DispatchQueue.main.asyncAfter(deadline: .now() + 0.08) { [weak self] in- guard let self else { return }- if let upEvent = synthesizeMouseEvent(type: .leftMouseUp, at: windowPoint) {- mouseUp(with: upEvent)- }+ self?.endTouch(at: localPoint, timestamp: ProcessInfo.processInfo.systemUptime)}}- /// Inject a swipe from one pixel coordinate to another.+ /// Inject a swipe from one pixel coordinate to another, as a finger.func injectSwipe(fromX: Double,fromY: Double,@@ -879,36 +939,25 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {screenHeight: screenHeight,)let endLocal = pixelToLocal(pixelX: toX, pixelY: toY, screenWidth: screenWidth, screenHeight: screenHeight)- let startWindow = convert(startLocal, to: nil)- let endWindow = convert(endLocal, to: nil)let steps = max(10, durationMs / 16)let stepInterval = Double(durationMs) / Double(steps) / 1000.0- if let downEvent = synthesizeMouseEvent(type: .leftMouseDown, at: startWindow) {- mouseDown(with: downEvent)- }+ beginTouch(at: startLocal, timestamp: ProcessInfo.processInfo.systemUptime)for i in 1 ... steps {let t = Double(i) / Double(steps)- let x = startWindow.x + (endWindow.x - startWindow.x) * t- let y = startWindow.y + (endWindow.y - startWindow.y) * t- let pt = NSPoint(x: x, y: y)- let delay = stepInterval * Double(i)-- if i < steps {- DispatchQueue.main.asyncAfter(deadline: .now() + delay) { [weak self] in- guard let self else { return }- if let dragEvent = synthesizeMouseEvent(type: .leftMouseDragged, at: pt) {- mouseDragged(with: dragEvent)- }- }- } else {- DispatchQueue.main.asyncAfter(deadline: .now() + delay) { [weak self] in- guard let self else { return }- if let upEvent = synthesizeMouseEvent(type: .leftMouseUp, at: pt) {- mouseUp(with: upEvent)- }+ let pt = NSPoint(+ x: startLocal.x + (endLocal.x - startLocal.x) * t,+ y: startLocal.y + (endLocal.y - startLocal.y) * t,+ )+ DispatchQueue.main.asyncAfter(deadline: .now() + stepInterval * Double(i)) { [weak self] in+ guard let self else { return }+ let now = ProcessInfo.processInfo.systemUptime+ if i < steps {+ sendTouchEvent(phase: 1, localPoint: pt, timestamp: now)+ } else {+ endTouch(at: pt, timestamp: now)}}}@@ -925,10 +974,10 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {////// The touch starts a hair inside the left edge on purpose: the system only/// reads a drag as the back gesture when the touch *begins* in the- /// screen-edge zone. It goes through `injectSwipe`, which synthesises- /// ordinary mouse events, so the touch follows whichever injection path the- /// running base uses (vphoned on iOS 18, the native VZ multitouch path on- /// 26 and later).+ /// screen-edge zone. It goes through `injectSwipe`, which is always a+ /// touch, even where a Mac click is the guest pointer's, and follows+ /// whichever injection path the running base uses (vphoned on iOS 18, the+ /// native VZ multitouch path on 26 and later)./// True while a back swipe is in flight. A second press during one cannot/// stack another gesture on top of it, which is what made a double press/// look like nothing happened.