Skip to content

Run a virtual iPhone on an Apple Silicon Mac.

Swift

Click with the guest's pointer on iPad guests

Jacky4ca4902a9b19
diff --git a/Research/vphoned_http_api.md b/Research/vphoned_http_api.md
index 4c734138..d483badf 100644
--- a/Research/vphoned_http_api.md
+++ b/Research/vphoned_http_api.md
@@ -372,7 +372,7 @@ request carries `"force": true`.
| --- | --- |
| Device | `device.snapshot`, `device.info` (snapshot plus network, screen, rotation, brightness, volume, low power, Developer Mode, agent, `device_name` `{name, own_name}`: the pinned name and the guest's own ComputerName), `device.screen`, `device.network`, `device.ioreg {plane}`, `device.environment`, `device.basebin {archive?}` |
| Display, audio | `display.brightness {value?}`, `display.rotation {orientation?}`, `display.orientation` (`{degrees, source}`: SpringBoard's interface orientation without a screen capture, for polling; `degrees` is null while SpringBoard has no answer; capability `display_orientation`), `display.rotation_lock {locked}`, `display.auto_lock` (`{auto_lock_seconds, never, lock_screen_minimum_seconds}`: Auto-Lock and the Lock Screen timeout vphoned keeps in step with it, see below; capability `display_auto_lock`), `screen.unlock {passcode?, timeout?}` (`{locked, screen_off, was_locked, was_screen_off}`: the display on and the Lock Screen passed, see below; capability `screen_unlock`), `audio.volume {value?, category?}`, `audio.state` |
-| Input | `input.touch`, `input.hid`, `input.button {name}`, `input.key {name}`, `input.type {text, delay_ms?}`, `input.paste {text}`, `input.tap`, `input.double_tap`, `input.long_press`, `input.swipe`, `input.drag {points}`, `input.touch_sequence {events}` — gesture coordinates are screen points; `input.pointer {x, y, seq?}`, `input.scroll {phase, dx?, dy?, x?, y?}` (capability `pointer_scroll`, see below) |
+| Input | `input.touch`, `input.hid`, `input.button {name}`, `input.key {name}`, `input.type {text, delay_ms?}`, `input.paste {text}`, `input.tap`, `input.double_tap`, `input.long_press`, `input.swipe`, `input.drag {points}`, `input.touch_sequence {events}` — gesture coordinates are screen points; `input.pointer {x, y, seq?, buttons?}`, `input.scroll {phase, dx?, dy?, x?, y?}` (capabilities `pointer_scroll` and `pointer_buttons`, see below) |
| UI | `ui.tree` (alias `accessibility.tree`), `ui.element_at`, `ui.tap_element`, `ui.wait`, `ui.wait_gone`, `ui.ocr {languages?, min_confidence?}`, `ui.describe`, `screen.screenshot` |
| Processes | `processes.list {filter?}`, `processes.kill {pid, signal?}` **force**, `memory.jetsam`, `memory.pressure` (only the three kernel memory sysctls, for polling) |
| launchd | `services.list`, `status`, `print`, `dump`, `disabled`, `start`, `enable`, `load`, `profile` (see below); `services.stop`, `disable`, `remove`, `signal`, `unload`, `profile.apply {profile, groups?, allow?}` **force**; `launchd.getenv`, `setenv`, `unsetenv`. On tvOS, which never enables user sessions, they read the system domain alone; before icli 0.8.6 the foreground-user lookup trapped there and killed vphoned |
@@ -464,32 +464,53 @@ offers the last fix again every 10 s until the guest takes it, since the Mac
sends a new one only when it moves. A preset or a replay pauses sync for that
run without changing the setting.
-`input.pointer` and `input.scroll` (capability `pointer_scroll`, iPad guests
-only; REST `POST /v1/input/pointer`, `POST /v1/input/scroll`) scroll the way an
-iPad with a trackpad does, instead of with a dragging finger. On first use
-vphoned creates a virtual pointing device (a HID user device with an absolute
-position, `VPhoneDaemon/Native/vphoned_pointer.m`); iPadOS then shows a pointer,
+`input.pointer` and `input.scroll` (capabilities `pointer_scroll` and
+`pointer_buttons`, iPad guests only; REST `POST /v1/input/pointer`,
+`POST /v1/input/scroll`) point, click and scroll the way an iPad with a
+trackpad does, instead of with a finger. They drive IcliKit's virtual
+pointing device (`pointerMove`, `pointerScroll`; a HID user device with an
+absolute position, `Sources/IcliPrivate/Pointer.m` in icli), which vphoned
+creates on first use and keeps while it runs; iPadOS then shows a pointer,
which hides itself after a few seconds without movement.
-- `input.pointer {x, y, seq?}` puts the pointer at x, y: 0...1 from the
- top-left corner of the interface as it is shown, in any orientation.
- Requests run concurrently, so a client that sends moves without waiting
- for each answer numbers them with an increasing `seq` (a whole number up
- to 2^53); a move that lands after a higher one is dropped, and still
- answers. Without `seq` every move is applied.
+- `input.pointer {x, y, seq?, buttons?}` puts the pointer at x, y: 0...1 from
+ the top-left corner of the interface as it is shown, in any orientation.
+ `buttons` are the buttons held from that move on, a whole number 0...7: bit
+ 0 primary, bit 1 secondary, bit 2 middle; without it none are held. Holding
+ one is a pointer click, which iPadOS answers as a trackpad's, not as a
+ touch. Requests run concurrently, so a client that sends moves without
+ waiting for each answer numbers them with an increasing `seq` (a whole
+ number up to 2^53); a move that lands after a higher one is dropped,
+ buttons and all, and still answers. Without `seq` every move is applied. A
+ client therefore sends button changes in order, and no move numbered past
+ one until it has been answered.
- `input.scroll {phase, dx?, dy?, x?, y?}` sends one scroll event to the view
- under the pointer, after moving the pointer to x, y when they are given.
+ under the pointer, after moving the pointer to x, y when they are given; the
+ buttons held stay as they are.
`phase` is `began`, `changed`, `ended` or `cancelled`: `began` first, a
`changed` per delta, then `ended`. `dx` and `dy` are interface points with a
browser wheel event's signs, positive `dy` toward the end of the content,
under the guest's natural scrolling setting (on by default); each is at most
10000.
+A click under a visible pointer has to be the pointer's. iPadOS ignores a
+touch on an alert's button while the pointer hovers it, until the pointer has
+hidden itself: on an iPadOS 26.6.2 guest a touch within 0.3 s of the pointer
+moving onto SpringBoard's Remove App Cancel button failed 6 times out of 6;
+with a Mac click sent as `buttons` the VM window closed it on the first
+click every time, after resting on the button. The window therefore sends a
+Mac click as `buttons` on a `pointer_buttons` guest, and keeps touches for
+`vphone.sock`'s `tap` and `swipe` and the back gesture. IcliKit sends a
+button change as a button event rather than a report bit: backboardd never
+releases a button that a report pressed, and a stuck button loses every
+later click of every pointing device.
+
The view moves by the deltas and by nothing else: iPadOS does not run on
after `ended`, so a client that wants a trackpad's momentum sends it as more
`changed` deltas first. The first few points of a scroll start the gesture
-and do not move the view (about 15 points on iPadOS 26). Both answer
-`invalid_operation` when the device cannot be made or is not ready yet.
+and do not move the view (about 15 points on iPadOS 26). Both answer with
+IcliKit's error, `unavailable` or `failed`, when the device cannot be made or
+is not ready yet.
`time.timezone` (capability `timezone`; REST `GET/PUT /v1/timezone`) returns
`{identifier, automatic, seconds_from_gmt}`: the Olson name
diff --git a/VPhoneDaemon/Daemon/GuestAPI+Pointer.swift b/VPhoneDaemon/Daemon/GuestAPI+Pointer.swift
new file mode 100644
index 00000000..262495e1
--- /dev/null
+++ b/VPhoneDaemon/Daemon/GuestAPI+Pointer.swift
@@ -0,0 +1,43 @@
+import Foundation
+import IcliKit
+
+// MARK: - Virtual Pointer
+
+/// `input.pointer` and `input.scroll` go to IcliKit's virtual pointing device,
+/// which lives as long as vphoned. iPadOS shows its pointer, and a button held
+/// in a move is a pointer click, as a trackpad's is.
+extension GuestAPI {
+ private static let pointerLock = NSLock()
+ /// The highest `seq` applied, under `pointerLock`.
+ private nonisolated(unsafe) static var lastPointerSequence: UInt64 = 0
+
+ /// Requests run concurrently, so a move sent earlier can land later;
+ /// putting the pointer back where it was would make it jump. A move whose
+ /// nonzero `sequence` is not above the last one applied is dropped, buttons
+ /// and all, and still answers: a client sends button changes in order and
+ /// nothing numbered past them until they are answered.
+ static func movePointer(_ point: GuestPointerScroll.Point, sequence: UInt64, buttons: Int32) throws {
+ pointerLock.lock()
+ defer { pointerLock.unlock() }
+ if sequence != 0 {
+ guard sequence > lastPointerSequence else { return }
+ lastPointerSequence = sequence
+ }
+ _ = try pointerMove(x: point.x, y: point.y, normalized: true, buttons: PointerButtons(rawValue: Int(buttons)))
+ }
+
+ /// One scroll event to the view under the pointer, after putting the
+ /// pointer at the scroll's position when it has one. The buttons held stay
+ /// as they are.
+ static func scrollPointer(_ scroll: GuestPointerScroll.Scroll) throws {
+ pointerLock.lock()
+ defer { pointerLock.unlock() }
+ if let point = scroll.pointer {
+ _ = try pointerMove(x: point.x, y: point.y, normalized: true)
+ }
+ guard let phase = ScrollPhase(rawValue: scroll.phase.rawValue) else {
+ throw GuestAPIError.invalidRequest("phase must be began, changed, ended or cancelled")
+ }
+ _ = try pointerScroll(phase, dx: scroll.dx, dy: scroll.dy)
+ }
+}
diff --git a/VPhoneDaemon/Daemon/GuestAPI.swift b/VPhoneDaemon/Daemon/GuestAPI.swift
index c7afc0db..e7d6e795 100644
--- a/VPhoneDaemon/Daemon/GuestAPI.swift
+++ b/VPhoneDaemon/Daemon/GuestAPI.swift
@@ -155,7 +155,7 @@ enum GuestAPI {
/// iPadOS takes a pointing device as it is; iOS on an iPhone only through
/// AssistiveTouch, so `input.pointer` and `input.scroll` are an iPad's.
private static func pointerCapabilities() -> [String] {
- GuestVirtualAudioProduct.machine().hasPrefix("iPad") ? ["pointer_scroll"] : []
+ GuestVirtualAudioProduct.machine().hasPrefix("iPad") ? ["pointer_scroll", "pointer_buttons"] : []
}
static func execute(method: String, params: [String: Any]) throws -> [String: Any] {
@@ -337,14 +337,16 @@ enum GuestAPI {
case "input.pointer":
let point: GuestPointerScroll.Point
let sequence: UInt64
+ let buttons: Int32
do {
point = try GuestPointerScroll.point(params)
sequence = try GuestPointerScroll.sequence(params)
+ buttons = try GuestPointerScroll.buttons(params)
} catch {
throw GuestAPIError.invalidRequest(error.message)
}
- try pointerResult(vp_pointer_move(point.x, point.y, sequence))
- return ["x": point.x, "y": point.y]
+ try movePointer(point, sequence: sequence, buttons: buttons)
+ return ["x": point.x, "y": point.y, "buttons": Int(buttons)]
case "input.scroll":
let scroll: GuestPointerScroll.Scroll
do {
@@ -352,10 +354,7 @@ enum GuestAPI {
} catch {
throw GuestAPIError.invalidRequest(error.message)
}
- if let point = scroll.pointer {
- try pointerResult(vp_pointer_move(point.x, point.y, 0))
- }
- try pointerResult(vp_pointer_scroll(scroll.phase.bits, scroll.dx, scroll.dy))
+ try scrollPointer(scroll)
return ["phase": scroll.phase.rawValue]
case "input.hid":
let page = try integer(params, "page")
@@ -620,17 +619,6 @@ enum GuestAPI {
/// Digitizer phase codes for vphoned's own multi-finger injection path.
/// icli keeps its equivalent mapping module-internal, and these are the
/// values the private digitizer entry points take: 0 down, 1 move, 3 up.
- /// A pointer call's result as an error, nothing when it worked.
- static func pointerResult(_ code: Int32) throws {
- switch Int(code) {
- case Int(VP_POINTER_OK): return
- case Int(VP_POINTER_UNAVAILABLE): throw GuestAPIError.operationFailed("this base has no virtual pointing device")
- case Int(VP_POINTER_REFUSED): throw GuestAPIError.operationFailed("the guest refused to create the virtual pointing device")
- case Int(VP_POINTER_NOT_READY): throw GuestAPIError.operationFailed("the virtual pointing device is not ready yet")
- default: throw GuestAPIError.operationFailed("the pointer event could not be delivered")
- }
- }
-
static func touchPhaseCode(_ phase: TouchPhase) -> Int32 {
switch phase {
case .down: 0
diff --git a/VPhoneDaemon/Daemon/GuestPointerScroll.swift b/VPhoneDaemon/Daemon/GuestPointerScroll.swift
index c580c368..81084e42 100644
--- a/VPhoneDaemon/Daemon/GuestPointerScroll.swift
+++ b/VPhoneDaemon/Daemon/GuestPointerScroll.swift
@@ -2,7 +2,8 @@ import Foundation
// MARK: - Pointer Scroll
-/// The parameters of `input.pointer` and `input.scroll`.
+/// The parameters of `input.pointer` and `input.scroll`, which IcliKit's
+/// virtual pointer carries out (`GuestAPI+Pointer.swift`).
///
/// A scroll goes in phases, the way a trackpad's does: `began` before the
/// first delta, `changed` for each delta, then `ended` or `cancelled`. iPadOS
@@ -14,21 +15,11 @@ import Foundation
/// Nothing here touches the guest, so `VPhoneDaemon/Tests/run-logic-tests.sh`
/// builds this file on the Mac.
enum GuestPointerScroll {
- enum Phase: String {
+ enum Phase: String, CaseIterable {
case began
case changed
case ended
case cancelled
-
- /// The `IOHIDEventPhaseBits` value the scroll event carries.
- var bits: Int32 {
- switch self {
- case .began: 1
- case .changed: 2
- case .ended: 4
- case .cancelled: 8
- }
- }
}
/// A position in the interface as it is shown, 0...1 from the top-left
@@ -79,6 +70,17 @@ enum GuestPointerScroll {
return UInt64(sequence)
}
+ /// `input.pointer`'s optional `buttons`, the buttons held from that move
+ /// on: bit 0 primary, bit 1 secondary, bit 2 middle. None held when it is
+ /// absent, so a client that only hovers never presses one.
+ static func buttons(_ params: [String: Any]) throws(Problem) -> Int32 {
+ guard let value = params["buttons"] else { return 0 }
+ guard let buttons = number(value), (0 ... 7).contains(buttons), buttons.rounded() == buttons else {
+ throw Problem(message: "buttons must be a whole number within 0...7")
+ }
+ return Int32(buttons)
+ }
+
static func scroll(_ params: [String: Any]) throws(Problem) -> Scroll {
guard let name = params["phase"] as? String, let phase = Phase(rawValue: name) else {
throw Problem(message: "phase must be began, changed, ended or cancelled")
diff --git a/VPhoneDaemon/Native/Include/VphonedNative.h b/VPhoneDaemon/Native/Include/VphonedNative.h
index b07db652..07a07561 100644
--- a/VPhoneDaemon/Native/Include/VphonedNative.h
+++ b/VPhoneDaemon/Native/Include/VphonedNative.h
@@ -27,29 +27,6 @@ bool vp_hid_load(void);
/// `touchSequence` can carry two fingers at once, which is why this exists.
void vp_hid_touch2(int phase, double x1, double y1, double x2, double y2);
-/// What vp_pointer_move and vp_pointer_scroll return.
-enum {
- VP_POINTER_OK = 0,
- /// The base does not have the private IOKit symbols.
- VP_POINTER_UNAVAILABLE = 1,
- /// IOHIDUserDeviceCreate refused to make the device.
- VP_POINTER_REFUSED = 2,
- /// The device exists but iOS has not put an event service over it yet.
- VP_POINTER_NOT_READY = 3,
- /// The report or the event could not be made or delivered.
- VP_POINTER_FAILED = 4,
-};
-
-/// Put the virtual pointer at x, y: 0...1 from the top-left corner of the
-/// interface as it is shown. Creates the pointing device on first use. A
-/// nonzero `sequence` not above the last one applied is a move that arrived
-/// late: it is dropped and the call still succeeds. 0 always moves.
-int vp_pointer_move(double x, double y, uint64_t sequence);
-
-/// One scroll event at the pointer. `phase` is an IOHIDEventPhaseBits value;
-/// dx and dy are interface points, with a browser wheel event's signs.
-int vp_pointer_scroll(int phase, double dx, double dy);
-
/// Turn the display on without pressing a button. False when
/// SpringBoardServices has no SBSUndimScreen.
bool vp_screen_undim(void);
diff --git a/VPhoneDaemon/Native/vphoned_pointer.m b/VPhoneDaemon/Native/vphoned_pointer.m
deleted file mode 100644
index 55711e14..00000000
--- a/VPhoneDaemon/Native/vphoned_pointer.m
+++ /dev/null
@@ -1,194 +0,0 @@
-/*
- * vphoned_pointer — a virtual pointing device, for native scrolling.
- *
- * A scroll made of touches is a finger that drags and lets go, and how far the
- * content runs on afterwards is the guest's guess at the finger's speed from
- * the times the moves arrive. Over a network those times bunch up, so the
- * run-on changes from one swipe to the next. A pointing device scrolls the way
- * an iPad with a trackpad does: the view under the pointer moves by exactly
- * the deltas given, and nothing runs on that the client did not send.
- *
- * The device is a HID user device that reports an absolute position, so
- * iPadOS shows its pointer and puts it where a report says, in the interface's
- * own orientation (measured on iPadOS 26 in portrait and landscape); the
- * pointer hides itself when it has been still for a few seconds. Scroll
- * events are dispatched with that device's event service as their sender:
- * with any other sender (the touchscreen's included) they reach nothing, or
- * the view at the top-left corner. The device is created on first use and
- * lives as long as vphoned.
- *
- * Every entry point is resolved with dlsym, like vphoned_touch.m.
- */
-
-#import "Include/VphonedNative.h"
-
-#include <dlfcn.h>
-#include <mach/mach_time.h>
-#include <math.h>
-#include <unistd.h>
-
-typedef void *IOHIDEventSystemClientRef;
-typedef void *IOHIDEventRef;
-typedef void *IOHIDUserDeviceRef;
-typedef double IOHIDFloat;
-typedef unsigned int vp_io_object_t;
-
-static IOHIDEventSystemClientRef (*pClientCreate)(CFAllocatorRef);
-static void (*pDispatch)(IOHIDEventSystemClientRef, IOHIDEventRef);
-static void (*pSetSender)(IOHIDEventRef, uint64_t);
-static IOHIDEventRef (*pScroll)(CFAllocatorRef, uint64_t, IOHIDFloat, IOHIDFloat, IOHIDFloat, uint32_t);
-static void (*pSetPhase)(IOHIDEventRef, uint32_t);
-static IOHIDUserDeviceRef (*pDeviceCreate)(CFAllocatorRef, CFDictionaryRef);
-static int (*pHandleReport)(IOHIDUserDeviceRef, const uint8_t *, CFIndex);
-static CFMutableDictionaryRef (*pMatching)(const char *);
-static int (*pMatchingServices)(unsigned int, CFDictionaryRef, vp_io_object_t *);
-static vp_io_object_t (*pNext)(vp_io_object_t);
-static CFTypeRef (*pSearchProperty)(vp_io_object_t, const char *, CFStringRef, CFAllocatorRef, uint32_t);
-static int (*pEntryID)(vp_io_object_t, uint64_t *);
-static int (*pRelease)(vp_io_object_t);
-
-static dispatch_queue_t gPointerQueue;
-static bool gPointerLoaded;
-static IOHIDEventSystemClientRef gPointerClient;
-static IOHIDUserDeviceRef gPointerDevice;
-static uint64_t gPointerSender;
-static uint64_t gPointerLastSequence;
-
-static NSString *const kPointerProduct = @"vphone pointer";
-
-// Three buttons (never pressed), an absolute X and Y within 0...32767, a wheel
-// and a horizontal pan (never turned). Without the wheel and the pan iPadOS
-// shows the pointer but sends the scroll events nowhere.
-static const uint8_t kPointerDescriptor[] = {
- 0x05, 0x01, 0x09, 0x02, 0xA1, 0x01, 0x09, 0x01, 0xA1, 0x00,
- 0x05, 0x09, 0x19, 0x01, 0x29, 0x03, 0x15, 0x00, 0x25, 0x01, 0x95, 0x03, 0x75, 0x01, 0x81, 0x02,
- 0x95, 0x01, 0x75, 0x05, 0x81, 0x03,
- 0x05, 0x01, 0x09, 0x30, 0x09, 0x31, 0x15, 0x00, 0x26, 0xFF, 0x7F, 0x75, 0x10, 0x95, 0x02, 0x81, 0x02,
- 0x09, 0x38, 0x15, 0x81, 0x25, 0x7F, 0x75, 0x08, 0x95, 0x01, 0x81, 0x06,
- 0x05, 0x0C, 0x0A, 0x38, 0x02, 0x15, 0x81, 0x25, 0x7F, 0x75, 0x08, 0x95, 0x01, 0x81, 0x06,
- 0xC0, 0xC0,
-};
-static const double kPointerMaximum = 32767;
-
-// kIORegistryIterateRecursively | kIORegistryIterateParents
-#define VP_SEARCH_UP 3u
-
-static bool load_symbols(void) {
- if (gPointerLoaded) return true;
- void *ioKit = dlopen("/System/Library/Frameworks/IOKit.framework/IOKit", RTLD_NOW);
- if (!ioKit) return false;
- pClientCreate = dlsym(ioKit, "IOHIDEventSystemClientCreate");
- pDispatch = dlsym(ioKit, "IOHIDEventSystemClientDispatchEvent");
- pSetSender = dlsym(ioKit, "IOHIDEventSetSenderID");
- pScroll = dlsym(ioKit, "IOHIDEventCreateScrollEvent");
- pSetPhase = dlsym(ioKit, "IOHIDEventSetPhase");
- pDeviceCreate = dlsym(ioKit, "IOHIDUserDeviceCreate");
- pHandleReport = dlsym(ioKit, "IOHIDUserDeviceHandleReport");
- pMatching = dlsym(ioKit, "IOServiceMatching");
- pMatchingServices = dlsym(ioKit, "IOServiceGetMatchingServices");
- pNext = dlsym(ioKit, "IOIteratorNext");
- pSearchProperty = dlsym(ioKit, "IORegistryEntrySearchCFProperty");
- pEntryID = dlsym(ioKit, "IORegistryEntryGetRegistryEntryID");
- pRelease = dlsym(ioKit, "IOObjectRelease");
- gPointerLoaded = pClientCreate && pDispatch && pSetSender && pScroll && pSetPhase && pDeviceCreate &&
- pHandleReport && pMatching && pMatchingServices && pNext && pSearchProperty && pEntryID &&
- pRelease;
- return gPointerLoaded;
-}
-
-// The registry ID of the event service iOS put over the device, which is the
-// sender its own events carry.
-static uint64_t find_sender(void) {
- vp_io_object_t iterator = 0;
- if (pMatchingServices(0, pMatching("IOHIDEventService"), &iterator) != 0) return 0;
- uint64_t found = 0;
- vp_io_object_t service;
- while ((service = pNext(iterator))) {
- CFTypeRef product = pSearchProperty(service, "IOService", CFSTR("Product"), kCFAllocatorDefault, VP_SEARCH_UP);
- if (!found && product && CFGetTypeID(product) == CFStringGetTypeID() &&
- CFStringCompare(product, (__bridge CFStringRef)kPointerProduct, 0) == kCFCompareEqualTo) {
- pEntryID(service, &found);
- }
- if (product) CFRelease(product);
- pRelease(service);
- }
- pRelease(iterator);
- return found;
-}
-
-// On gPointerQueue. The event service appears a moment after the device;
-// waits up to a second for it.
-static int ensure_device(void) {
- if (gPointerSender) return VP_POINTER_OK;
- if (!load_symbols()) return VP_POINTER_UNAVAILABLE;
- if (!gPointerClient) gPointerClient = pClientCreate(kCFAllocatorDefault);
- if (!gPointerClient) return VP_POINTER_UNAVAILABLE;
- if (!gPointerDevice) {
- NSDictionary *properties = @{
- @"ReportDescriptor": [NSData dataWithBytes:kPointerDescriptor length:sizeof(kPointerDescriptor)],
- @"VendorID": @0x05AC,
- @"ProductID": @0x7650,
- @"Product": kPointerProduct,
- @"Transport": @"USB",
- @"PrimaryUsagePage": @1,
- @"PrimaryUsage": @2,
- };
- gPointerDevice = pDeviceCreate(kCFAllocatorDefault, (__bridge CFDictionaryRef)properties);
- if (!gPointerDevice) {
- NSLog(@"vphoned: IOHIDUserDeviceCreate refused the pointer device");
- return VP_POINTER_REFUSED;
- }
- }
- for (int attempt = 0; attempt < 50 && !gPointerSender; attempt++) {
- gPointerSender = find_sender();
- if (!gPointerSender) usleep(20000);
- }
- return gPointerSender ? VP_POINTER_OK : VP_POINTER_NOT_READY;
-}
-
-static dispatch_queue_t pointer_queue(void) {
- static dispatch_once_t once;
- dispatch_once(&once, ^{
- dispatch_queue_attr_t attributes = dispatch_queue_attr_make_with_qos_class(
- DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INTERACTIVE, 0);
- gPointerQueue = dispatch_queue_create("com.vphone.vphoned.pointer", attributes);
- });
- return gPointerQueue;
-}
-
-int vp_pointer_move(double x, double y, uint64_t sequence) {
- __block int result;
- dispatch_sync(pointer_queue(), ^{
- result = ensure_device();
- if (result != VP_POINTER_OK) return;
- // Requests run concurrently, so a move sent earlier can land later;
- // putting the pointer back where it was would make it jump.
- if (sequence) {
- if (sequence <= gPointerLastSequence) return;
- gPointerLastSequence = sequence;
- }
- uint16_t px = (uint16_t)lround(fmin(fmax(x, 0), 1) * kPointerMaximum);
- uint16_t py = (uint16_t)lround(fmin(fmax(y, 0), 1) * kPointerMaximum);
- const uint8_t report[7] = {0, px & 0xFF, px >> 8, py & 0xFF, py >> 8, 0, 0};
- if (pHandleReport(gPointerDevice, report, sizeof(report)) != 0) result = VP_POINTER_FAILED;
- });
- return result;
-}
-
-int vp_pointer_scroll(int phase, double dx, double dy) {
- __block int result;
- dispatch_sync(pointer_queue(), ^{
- result = ensure_device();
- if (result != VP_POINTER_OK) return;
- IOHIDEventRef event = pScroll(kCFAllocatorDefault, mach_absolute_time(), dx, dy, 0, 0);
- if (!event) {
- result = VP_POINTER_FAILED;
- return;
- }
- pSetPhase(event, (uint32_t)phase);
- pSetSender(event, gPointerSender);
- pDispatch(gPointerClient, event);
- CFRelease(event);
- });
- return result;
-}
diff --git a/VPhoneDaemon/Tests/LogicTests.swift b/VPhoneDaemon/Tests/LogicTests.swift
index 83678445..57c9033d 100644
--- a/VPhoneDaemon/Tests/LogicTests.swift
+++ b/VPhoneDaemon/Tests/LogicTests.swift
@@ -365,10 +365,8 @@ enum LogicTests {
Scroll(phase: .began, dx: 0, dy: 0, pointer: nil), "deltas default to 0")
expectEqual(try? GuestPointerScroll.scroll(["phase": "began", "x": 0, "y": 1]),
Scroll(phase: .began, dx: 0, dy: 0, pointer: .init(x: 0, y: 1)), "a pointer position comes with it")
- expectEqual(GuestPointerScroll.Phase.began.bits, 1)
- expectEqual(GuestPointerScroll.Phase.changed.bits, 2)
- expectEqual(GuestPointerScroll.Phase.ended.bits, 4)
- expectEqual(GuestPointerScroll.Phase.cancelled.bits, 8)
+ expectEqual(GuestPointerScroll.Phase.allCases.map(\.rawValue), ["began", "changed", "ended", "cancelled"],
+ "the names IcliKit's ScrollPhase takes")
let phase = "phase must be began, changed, ended or cancelled"
expectEqual(scrollProblem([:]), phase)
expectEqual(scrollProblem(["phase": "move"]), phase)
@@ -390,5 +388,11 @@ enum LogicTests {
for bad: Any in [-1, 1.5, true, "7", 9_007_199_254_740_994.0] {
expect((try? GuestPointerScroll.sequence(["seq": bad])) == nil, "seq \(bad) is refused")
}
+ expectEqual(try? GuestPointerScroll.buttons([:]), 0, "no buttons held when there are none")
+ expectEqual(try? GuestPointerScroll.buttons(["buttons": 1]), 1)
+ expectEqual(try? GuestPointerScroll.buttons(["buttons": 7]), 7)
+ for bad: Any in [-1, 8, 1.5, true, "1"] {
+ expect((try? GuestPointerScroll.buttons(["buttons": bad])) == nil, "buttons \(bad) is refused")
+ }
}
}
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/GuestCommunication/VPhoneGuestControl.swift b/VPhoneExecutable/VPhoneVirtualization/UI/GuestCommunication/VPhoneGuestControl.swift
index f86fd7d2..19cc98cc 100644
--- a/VPhoneExecutable/VPhoneVirtualization/UI/GuestCommunication/VPhoneGuestControl.swift
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/GuestCommunication/VPhoneGuestControl.swift
@@ -191,6 +191,8 @@ final class VPhoneGuestControl {
orderedInput = nil
openScroll = nil
pendingPointer = nil
+ openPointerPress = nil
+ heldPointerButtons = 0
orderedLocation?.cancel()
orderedLocation = nil
device = nil
@@ -348,9 +350,20 @@ final class VPhoneGuestControl {
isConnected && guestCapabilities.contains("pointer_scroll")
}
+ /// Whether a button held in a pointer move is a click (`input.pointer`'s
+ /// `buttons`), so a Mac click can be the guest pointer's.
+ var supportsPointerButtons: Bool {
+ isConnected && guestCapabilities.contains("pointer_buttons")
+ }
+
@ObservationIgnored private var pointersInFlight = 0
@ObservationIgnored private var pendingPointer: CGPoint?
@ObservationIgnored private var lastPointerSequence: UInt64 = 0
+ /// The buttons the last `setPointer` holds down.
+ @ObservationIgnored private var heldPointerButtons = 0
+ /// `setPointer` requests not yet answered.
+ @ObservationIgnored private var pointerPressesQueued = 0
+ @ObservationIgnored private var openPointerPress: PendingPointerPress?
/// Moves in flight at once. Each request is its own connection; waiting
/// for one answer before the next put a round trip behind every move.
@@ -361,28 +374,84 @@ final class VPhoneGuestControl {
/// on the order of touches, and a mouse that keeps moving would hold every
/// click behind it. Moves go out at once, numbered so that vphoned drops
/// one that lands after a later one; past `maximumPointersInFlight` the
- /// newest position waits for the next free slot.
+ /// newest position waits for the next free slot. While a button is down,
+ /// or its release has not been answered, the newest position waits too: a
+ /// move holds no buttons, and one numbered past a press would release it.
func movePointer(to point: CGPoint) {
- guard pointersInFlight < Self.maximumPointersInFlight else {
+ guard pointersInFlight < Self.maximumPointersInFlight, heldPointerButtons == 0, pointerPressesQueued == 0
+ else {
pendingPointer = point
return
}
- // Microseconds of uptime: increasing across reconnects and restarts of
- // this process, which a guest that keeps running may still remember.
- let now = UInt64(ProcessInfo.processInfo.systemUptime * 1_000_000)
- lastPointerSequence = max(now, lastPointerSequence + 1)
- let params: [String: Any] = ["x": Double(point.x), "y": Double(point.y), "seq": lastPointerSequence]
+ let params: [String: Any] = ["x": Double(point.x), "y": Double(point.y), "seq": nextPointerSequence()]
pointersInFlight += 1
Task {
_ = try? await call("input.pointer", params: params)
pointersInFlight -= 1
- if let next = pendingPointer {
- pendingPointer = nil
- movePointer(to: next)
+ movePendingPointer()
+ }
+ }
+
+ /// The position a press or a drag still waiting in the input queue sends.
+ private final class PendingPointerPress {
+ var point: CGPoint
+ let buttons: Int
+
+ init(point: CGPoint, buttons: Int) {
+ self.point = point
+ self.buttons = buttons
+ }
+ }
+
+ /// Puts the guest's pointer at `point` with `buttons` held, bit 0 primary
+ /// and bit 1 secondary: a click that is the pointer's, as a trackpad's is,
+ /// and not a touch. In the ordered input queue, after the keys and touches
+ /// sent before it, so no press or release is reordered or dropped. A drag
+ /// that keeps the same buttons down joins the move still waiting there,
+ /// so a guest slower than the mouse gets fewer moves and loses no release.
+ func setPointer(_ point: CGPoint, buttons: Int) {
+ heldPointerButtons = buttons
+ // A hover from before the press would land after it.
+ pendingPointer = nil
+ if let open = openPointerPress, open.buttons == buttons {
+ open.point = point
+ return
+ }
+ let open = PendingPointerPress(point: point, buttons: buttons)
+ openPointerPress = open
+ pointerPressesQueued += 1
+ let previous = orderedInput
+ orderedInput = Task {
+ await previous?.value
+ if openPointerPress === open {
+ openPointerPress = nil
+ }
+ if !Task.isCancelled {
+ let params: [String: Any] = [
+ "x": Double(open.point.x), "y": Double(open.point.y), "buttons": open.buttons,
+ "seq": nextPointerSequence(),
+ ]
+ do { _ = try await call("input.pointer", params: params) } catch { print("[control] input.pointer: \(error)") }
}
+ pointerPressesQueued -= 1
+ movePendingPointer()
}
}
+ private func movePendingPointer() {
+ guard let next = pendingPointer else { return }
+ pendingPointer = nil
+ movePointer(to: next)
+ }
+
+ /// Microseconds of uptime: increasing across reconnects and restarts of
+ /// this process, which a guest that keeps running may still remember.
+ private func nextPointerSequence() -> UInt64 {
+ let now = UInt64(ProcessInfo.processInfo.systemUptime * 1_000_000)
+ lastPointerSequence = max(now, lastPointerSequence + 1)
+ return lastPointerSequence
+ }
+
/// The `changed` deltas waiting in the input queue, added up until that
/// request goes out.
private final class PendingScroll {
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swift b/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swift
index 82da26c1..421438a9 100644
--- a/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swift
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swift
@@ -199,21 +199,16 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
override func mouseDown(with event: NSEvent) {
// Clicking the VM display should always restore keyboard focus.
window?.makeFirstResponder(self)
- // A synthetic trackpad finger would fight this touch for the same slot.
- finishScrollTouch()
- if nativeScrollActive {
- endNativeScroll()
- nativeScrollMomentumDropped = true
+ let localPoint = convert(event.locationInWindow, from: nil)
+ if pressPointer(Self.primaryPointerButton, at: localPoint) {
+ return
}
// A click is a touch, and iPadOS hides its pointer under a finger: the
// Mac's shows where the drag is until the mouse moves freely again.
if guestPointerFollowsMouse {
NSCursor.arrow.set()
}
- releaseSecondaryPress()
- let localPoint = convert(event.locationInWindow, from: nil)
- currentTouchSwipeAim = hitTestEdge(at: localPoint)
- if sendTouchEvent(phase: 0, localPoint: localPoint, timestamp: event.timestamp) {
+ if beginTouch(at: localPoint, timestamp: event.timestamp) {
return
}
super.mouseDown(with: event)
@@ -221,7 +216,7 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
override func mouseDragged(with event: NSEvent) {
let localPoint = convert(event.locationInWindow, from: nil)
- if sendTouchEvent(phase: 1, localPoint: localPoint, timestamp: event.timestamp) {
+ if dragPointer(to: localPoint) || sendTouchEvent(phase: 1, localPoint: localPoint, timestamp: event.timestamp) {
return
}
super.mouseDragged(with: event)
@@ -229,10 +224,15 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
override func mouseUp(with event: NSEvent) {
let localPoint = convert(event.locationInWindow, from: nil)
- if !sendTouchEvent(phase: 3, localPoint: localPoint, timestamp: event.timestamp) {
+ if releasePointer(Self.primaryPointerButton, at: localPoint) {
+ if hidesHostCursor {
+ Self.hiddenCursor.set()
+ }
+ return
+ }
+ if !endTouch(at: localPoint, timestamp: event.timestamp) {
super.mouseUp(with: event)
}
- currentTouchSwipeAim = 0
// Bring the guest's pointer back where the finger lifted.
if guestPointerFollowsMouse, let control {
if hidesHostCursor {
@@ -245,15 +245,20 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
// MARK: - Secondary Click
/// A right click is iPadOS's secondary click: the context menu under the
- /// pointer. The guest has a touchscreen and no pointer, so it is replayed
- /// as a touch-and-hold, which is what opens the same menu by finger. A
- /// click released before the hold registers is kept down until it has.
- /// Holding and dragging carries on as a drag, as a finger would.
+ /// pointer. Where the guest's pointer follows the mouse it is that
+ /// pointer's secondary button. Elsewhere the guest has a touchscreen and
+ /// no pointer, so it is replayed as a touch-and-hold, which is what opens
+ /// the same menu by finger. A click released before the hold registers is
+ /// kept down until it has. Holding and dragging carries on as a drag, as a
+ /// finger would.
override func rightMouseDown(with event: NSEvent) {
window?.makeFirstResponder(self)
+ let localPoint = convert(event.locationInWindow, from: nil)
+ if pressPointer(Self.secondaryPointerButton, at: localPoint) {
+ return
+ }
finishScrollTouch()
releaseSecondaryPress()
- let localPoint = convert(event.locationInWindow, from: nil)
// A hold is never an edge swipe, wherever it lands.
currentTouchSwipeAim = 0
guard sendTouchEvent(phase: 0, localPoint: localPoint, timestamp: event.timestamp) else {
@@ -265,6 +270,9 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
}
override func rightMouseDragged(with event: NSEvent) {
+ if dragPointer(to: convert(event.locationInWindow, from: nil)) {
+ return
+ }
guard secondaryPressPoint != nil, secondaryReleaseWork == nil else {
super.rightMouseDragged(with: event)
return
@@ -275,6 +283,9 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
}
override func rightMouseUp(with event: NSEvent) {
+ if releasePointer(Self.secondaryPointerButton, at: convert(event.locationInWindow, from: nil)) {
+ return
+ }
guard secondaryPressPoint != nil, secondaryReleaseWork == nil else {
super.rightMouseUp(with: event)
return
@@ -303,6 +314,75 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
sendTouchEvent(phase: 3, localPoint: point, timestamp: ProcessInfo.processInfo.systemUptime)
}
+ // MARK: - Pointer Clicks
+
+ private static let primaryPointerButton = 1
+ private static let secondaryPointerButton = 2
+
+ /// The buttons Mac clicks hold down on the guest's pointer.
+ private var pointerButtons = 0
+
+ /// On a guest whose pointer follows the mouse a click is that pointer's,
+ /// as on an iPad with a trackpad. A touch is the wrong input there: iPadOS
+ /// ignores a touch on an alert's button while the pointer hovers it, until
+ /// the pointer hides itself a few seconds after it last moved.
+ private var clicksWithGuestPointer: Bool {
+ guestPointerFollowsMouse && control?.supportsPointerButtons == true
+ }
+
+ /// Presses `button` on the guest's pointer at `localPoint`, ending any
+ /// scroll or touch first. False when a click is a touch on this guest.
+ private func pressPointer(_ button: Int, at localPoint: NSPoint) -> Bool {
+ guard clicksWithGuestPointer, let control else { return false }
+ finishScrollTouch()
+ if nativeScrollActive {
+ endNativeScroll()
+ nativeScrollMomentumDropped = true
+ }
+ releaseSecondaryPress()
+ pointerButtons |= button
+ control.setPointer(interfacePoint(localPoint), buttons: pointerButtons)
+ return true
+ }
+
+ /// Moves the guest's pointer with its buttons held. False when none is.
+ private func dragPointer(to localPoint: NSPoint) -> Bool {
+ guard pointerButtons != 0, let control else { return false }
+ control.setPointer(interfacePoint(localPoint), buttons: pointerButtons)
+ return true
+ }
+
+ /// Releases `button`. False when the guest's pointer is not holding it, so
+ /// a release whose press was a touch lifts the finger instead.
+ private func releasePointer(_ button: Int, at localPoint: NSPoint) -> Bool {
+ guard pointerButtons & button != 0, let control else { return false }
+ pointerButtons &= ~button
+ control.setPointer(interfacePoint(localPoint), buttons: pointerButtons)
+ return true
+ }
+
+ // MARK: - Touches
+
+ /// Puts a finger down at `localPoint`, after ending every other input that
+ /// would fight it for the same slot. False when there is no touch path.
+ @discardableResult
+ private func beginTouch(at localPoint: NSPoint, timestamp: TimeInterval) -> Bool {
+ finishScrollTouch()
+ if nativeScrollActive {
+ endNativeScroll()
+ nativeScrollMomentumDropped = true
+ }
+ releaseSecondaryPress()
+ currentTouchSwipeAim = hitTestEdge(at: localPoint)
+ return sendTouchEvent(phase: 0, localPoint: localPoint, timestamp: timestamp)
+ }
+
+ @discardableResult
+ private func endTouch(at localPoint: NSPoint, timestamp: TimeInterval) -> Bool {
+ defer { currentTouchSwipeAim = 0 }
+ return sendTouchEvent(phase: 3, localPoint: localPoint, timestamp: timestamp)
+ }
+
// MARK: - Trackpad Gestures
/// Replay a trackpad scroll as a guest finger that presses under the pointer
@@ -826,22 +906,9 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
))
}
- /// Synthesize an NSEvent at a given window point.
- private func synthesizeMouseEvent(type: NSEvent.EventType, at windowPoint: NSPoint) -> NSEvent? {
- NSEvent.mouseEvent(
- with: type,
- location: windowPoint,
- modifierFlags: [],
- timestamp: ProcessInfo.processInfo.systemUptime,
- windowNumber: window?.windowNumber ?? 0,
- context: nil,
- eventNumber: 0,
- clickCount: type == .leftMouseUp ? 0 : 1,
- pressure: type == .leftMouseUp ? 0.0 : 1.0,
- )
- }
-
/// Inject a tap at pixel coordinates (matching screenshot image dimensions).
+ /// Always a touch, even where a Mac click is the guest pointer's: a tap
+ /// stands for a finger, and the pointer shows only once the mouse moves.
func injectTap(pixelX: Double, pixelY: Double, screenWidth: Int, screenHeight: Int) {
let localPoint = pixelToLocal(
pixelX: pixelX,
@@ -849,20 +916,13 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
screenWidth: screenWidth,
screenHeight: screenHeight,
)
- let windowPoint = convert(localPoint, to: nil)
-
- if let downEvent = synthesizeMouseEvent(type: .leftMouseDown, at: windowPoint) {
- mouseDown(with: downEvent)
- }
+ beginTouch(at: localPoint, timestamp: ProcessInfo.processInfo.systemUptime)
DispatchQueue.main.asyncAfter(deadline: .now() + 0.08) { [weak self] in
- guard let self else { return }
- if let upEvent = synthesizeMouseEvent(type: .leftMouseUp, at: windowPoint) {
- mouseUp(with: upEvent)
- }
+ self?.endTouch(at: localPoint, timestamp: ProcessInfo.processInfo.systemUptime)
}
}
- /// Inject a swipe from one pixel coordinate to another.
+ /// Inject a swipe from one pixel coordinate to another, as a finger.
func injectSwipe(
fromX: Double,
fromY: Double,
@@ -879,36 +939,25 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
screenHeight: screenHeight,
)
let endLocal = pixelToLocal(pixelX: toX, pixelY: toY, screenWidth: screenWidth, screenHeight: screenHeight)
- let startWindow = convert(startLocal, to: nil)
- let endWindow = convert(endLocal, to: nil)
let steps = max(10, durationMs / 16)
let stepInterval = Double(durationMs) / Double(steps) / 1000.0
- if let downEvent = synthesizeMouseEvent(type: .leftMouseDown, at: startWindow) {
- mouseDown(with: downEvent)
- }
+ beginTouch(at: startLocal, timestamp: ProcessInfo.processInfo.systemUptime)
for i in 1 ... steps {
let t = Double(i) / Double(steps)
- let x = startWindow.x + (endWindow.x - startWindow.x) * t
- let y = startWindow.y + (endWindow.y - startWindow.y) * t
- let pt = NSPoint(x: x, y: y)
- let delay = stepInterval * Double(i)
-
- if i < steps {
- DispatchQueue.main.asyncAfter(deadline: .now() + delay) { [weak self] in
- guard let self else { return }
- if let dragEvent = synthesizeMouseEvent(type: .leftMouseDragged, at: pt) {
- mouseDragged(with: dragEvent)
- }
- }
- } else {
- DispatchQueue.main.asyncAfter(deadline: .now() + delay) { [weak self] in
- guard let self else { return }
- if let upEvent = synthesizeMouseEvent(type: .leftMouseUp, at: pt) {
- mouseUp(with: upEvent)
- }
+ let pt = NSPoint(
+ x: startLocal.x + (endLocal.x - startLocal.x) * t,
+ y: startLocal.y + (endLocal.y - startLocal.y) * t,
+ )
+ DispatchQueue.main.asyncAfter(deadline: .now() + stepInterval * Double(i)) { [weak self] in
+ guard let self else { return }
+ let now = ProcessInfo.processInfo.systemUptime
+ if i < steps {
+ sendTouchEvent(phase: 1, localPoint: pt, timestamp: now)
+ } else {
+ endTouch(at: pt, timestamp: now)
}
}
}
@@ -925,10 +974,10 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
///
/// The touch starts a hair inside the left edge on purpose: the system only
/// reads a drag as the back gesture when the touch *begins* in the
- /// screen-edge zone. It goes through `injectSwipe`, which synthesises
- /// ordinary mouse events, so the touch follows whichever injection path the
- /// running base uses (vphoned on iOS 18, the native VZ multitouch path on
- /// 26 and later).
+ /// screen-edge zone. It goes through `injectSwipe`, which is always a
+ /// touch, even where a Mac click is the guest pointer's, and follows
+ /// whichever injection path the running base uses (vphoned on iOS 18, the
+ /// native VZ multitouch path on 26 and later).
/// True while a back swipe is in flight. A second press during one cannot
/// stack another gesture on top of it, which is what made a double press
/// look like nothing happened.