Skip to content

Run a virtual iPhone on an Apple Silicon Mac.

Swift

Replay trackpad gestures and Esc as guest touches (#533)

lgzcoollg5922ef11810b
diff --git a/Research/Guest/trackpad_gesture_mapping.md b/Research/Guest/trackpad_gesture_mapping.md
new file mode 100644
index 00000000..73ba966e
--- /dev/null
+++ b/Research/Guest/trackpad_gesture_mapping.md
@@ -0,0 +1,235 @@
+# Host Trackpad Gestures → Guest Touch
+
+How macOS trackpad gestures are replayed as touches inside the virtual iPhone,
+and the trade-offs that are load-bearing.
+
+---
+
+## Behavior
+
+| Host gesture | Guest gesture |
+| -------------------------- | -------------------------------------------------------------------- |
+| Two-finger scroll | One finger presses under the pointer and follows the physical motion, then lifts |
+| Two-finger pinch (magnify) | Two fingers spreading/closing horizontally around the pointer |
+
+Both are host-side translations of `NSEvent`s in
+`VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swift`
+(`scrollWheel(with:)` / `magnify(with:)`). The guest never sees a scroll or
+magnify event — it sees an ordinary drag or pinch.
+
+A swipe keeps scrolling for as long as the trackpad keeps reporting travel: it
+is **not** bounded by where the pointer sits or by the size of the window (see
+"Edge re-anchoring" below).
+
+Toggle: **Device → Trackpad Scroll & Pinch to Touch** (on by default). The menu
+item lives in `VPhoneMenuDevice.swift` next to the other input switches and
+writes through to the window's view; persisted as `trackpadGesturesEnabled`
+(`UserDefaults` key `trackpadGesturesDisabled`).
+
+---
+
+## Injection paths
+
+`sendTouchEvent(phase:localPoints:swipeAim:allowOutside:timestamp:)` fans out to
+whichever touch channel the running base uses:
+
+| Base | Channel |
+| --------------------------------- | ---------------------------------------------------------------- |
+| iOS 18 (`useGuestTouchInjection`) | vphoned `input.touch` (one finger, icli) / `input.touch2` (two fingers, `vp_hid_touch2` in `VPhoneDaemon/Native/vphoned_touch.m`) |
+| iOS 26 | Native VZ `_VZTouch(view:index:phase:location:swipeAim:timestamp:)` × N inside one `_VZMultiTouchEvent` |
+
+A guest that predates `touch2` advertises no `touch2` capability
+(`VPhoneGuestControl.supportsMultiTouch`), so pinch degrades to a single-finger
+move there; the auto-update push normally brings the new agent in.
+
+**Why the guest has its own injection path at all.** icli — the guest's general
+automation package — carries exactly one digitizer point per event
+(`TouchEvent{phase, x, y, delayMS}` → `icli_hid_touch(phase, x, y)`), so a pinch
+cannot be expressed through it. `VPhoneDaemon` therefore builds two-finger hand
+events itself over the same private digitizer API. Finger 0 keeps
+`index = 1, identity = 2` so the icli-driven single-finger path is unchanged; the
+second finger carries its own identity so the guest tracks it separately.
+
+`allowOutside` forwards coordinates past the view instead of clamping them.
+Keep it: the finger model runs past the edge on purpose during a re-anchor, and
+clamping the reported position would make the release snap back to the edge.
+
+---
+
+## Direction convention
+
+`scrollingDeltaX/Y` carry AppKit's scroll direction, which is inverted by the
+user's natural-scrolling preference. The code compensates with
+`event.isDirectionInvertedFromDevice` and treats the result as the physical
+finger direction, with one extra sign on the x axis:
+
+```swift
+var dx = -event.scrollingDeltaX
+var dy = event.scrollingDeltaY
+if event.isDirectionInvertedFromDevice { dx = -dx; dy = -dy }
+// dy > 0 == finger moved up, dx > 0 == finger moved right
+```
+
+The vertical sign comes from the header contract (`NSEvent.h`: "the event's
+scrolling values are inverted from the device's physical direction …
+compensate by multiplying -1") plus the historical NSScrollView convention
+that the un-inverted delta is positive for an upward gesture.
+
+The horizontal sign is **empirically calibrated**, not derived: with both axes
+compensated the same way, hardware testing showed the guest finger following
+the fingers vertically but mirrored horizontally. The two axes apparently do
+not share a baseline. `scrollDelta(of:)` is the single place this lives — flip
+the sign there if a base or a scroll-preference setting comes out reversed.
+
+Guest coordinates are top-left origin, so "up" means a smaller `y`; the view's
+non-flipped local space is converted by `normalizeCoordinate`.
+
+---
+
+## Gesture state machine
+
+1. **Start threshold.** A finger is only pressed once the gesture has travelled
+ `scrollStartThreshold` (2pt), and the carried travel is applied immediately
+ afterwards. Taps and zero-travel scrolls therefore never reach the guest as
+ a click, and the finger never sits idle after being pressed — which is what
+ turned pointer rests into long presses and text selection.
+2. **Throttled updates.** Trackpads emit 60-120Hz. The finger's position
+ advances on every frame, but events are forwarded at most once per
+ `scrollMinSendInterval` (1/70s); the release carries the final position. At
+ 120Hz that halves the traffic into the guest's HID queue, which was showing
+ up as stutter and as the finger drifting on after the gesture ended.
+3. **Edge re-anchoring.** See below.
+4. **Home strip snap.** An upward swipe that starts inside the bottom
+ `homeEdgeSnapFraction` (10%) of the window is pressed at the very bottom edge
+ instead of at the pointer. See below.
+5. **Pinch exclusion.** A drag and a pinch never overlap: `magnify` is ignored
+ while a synthetic finger is down and vice versa, because interleaved
+ one-finger and two-finger touches leave the guest with no coherent gesture.
+ `mouseDown` also releases any synthetic finger so it cannot steal the slot
+ from a real click.
+
+---
+
+## Starting point and the Home indicator strip
+
+The finger normally presses down wherever the pointer is, which lets the pointer
+position pick the gesture: from the edge it is an edge gesture, in the content
+area it is a scroll.
+
+Unlocking is the one case where that is not enough. iOS only recognises an
+upward unlock / home gesture when the touch **begins** inside the Home indicator
+strip at the very bottom of the screen. A swipe starting a few points higher is
+an ordinary drag, so on hardware the lock screen unlocked with the pointer
+parked on the bar and stopped working as soon as it moved up a little — the
+boundary, not the travel budget, was what changed.
+
+`scrollStartPoint(for:travel:)` covers that: an upward swipe (`travel.y > |travel.x|`)
+starting inside the bottom `homeEdgeSnapFraction` of the window is pressed at
+`y = 0` instead. Everywhere else the pointer is used verbatim, so the rest of
+the window keeps scrolling normally — including the bottom area for downward
+swipes.
+
+Cost: an upward swipe that begins in that bottom strip is a home / unlock
+gesture rather than a scroll, which is the same trade the real device makes. If
+that strip feels too tall (or too short) for unlocking, tune
+`homeEdgeSnapFraction`.
+
+---
+
+## Edge re-anchoring (why the swipe is not limited by the pointer)
+
+iOS scrolls a view by moving the touch: the offset equals the distance the
+finger travels **on screen**. A window is finite, so a finger that simply walks
+in one direction runs out of screen — and once past the edge the guest clamps
+the touch to the border and stops following it, which makes the rest of the
+swipe scroll nothing at all. That is what "the swipe stops early, and how early
+depends on where my pointer was" was: the travel available was the distance
+from the pointer to the edge in the direction of the swipe.
+
+Handing out-of-range coordinates to the guest (`allowOutside`) does not fix
+this on its own — the clamp happens inside the guest. It only keeps the host's
+own model honest about where the finger is.
+
+The fix is to give the swipe a fresh window of travel whenever it runs out:
+
+```swift
+if isPastScrollEdge(target) {
+ scrollOverflow.x += delta.x
+ scrollOverflow.y += delta.y
+ if hypot(scrollOverflow.x, scrollOverflow.y) >= Self.scrollRebaseThreshold {
+ rebaseScrollTouch() // lift here, re-press on the far side
+ }
+}
+```
+
+`rebaseScrollTouch()` lifts the finger at the edge and presses it back down at
+`scrollRebaseLandingInset` (12% of the window) measured from the edge it ran
+into, **towards the opposite side**. Every re-anchor therefore hands back
+roughly a full window of travel in the direction of the swipe, independent of
+where the pointer is — that is what removes the pointer-position dependency.
+
+Two details are load-bearing:
+
+- **The gap.** `scrollRebaseGap` (20ms) separates the lift from the re-press,
+ and the re-press is issued on the next scroll event once the gap has elapsed.
+ Back-to-back events that share a timestamp get folded into a single move by
+ the guest, so the touch appears to teleport from one edge to the other: iOS
+ reads that as one very fast drag in the wrong direction and the content snaps
+ back across the whole window.
+- **Travel banked during the gap is replayed** (`flushScrollPending()`), not
+ dropped, otherwise a fast swipe would lose distance to every re-press and the
+ scroll would lag behind the fingers.
+
+The threshold (12pt) is deliberately small: past the edge the finger is clamped
+by the guest, so every frame spent out there is a frame where the guest sees an
+idle finger — which it escalates into a long press if it goes on long enough.
+
+### Rejected: momentum replay
+
+Following AppKit's inertia stream dragged the guest on after the user had
+stopped ("my hand is off the trackpad and it is still scrolling"). Momentum is
+AppKit's synthetic deceleration; the guest produces its own once the finger
+lifts at `.ended`, so replaying it double-counts. `momentumPhase` events are
+now dropped — they only force a release in case a finger is somehow still down.
+
+---
+
+## Remaining simplifications
+
+- **One swipe still costs a re-anchor every ~88% of the window.** Each one is a
+ 20ms lift. That is the price of unbounded travel; the alternative is a swipe
+ that stops at the window edge.
+- **Pinch radius is clamped** to `[8, min(width,height) * 0.45]` and only
+ changes with `magnification`; the pinch center stays where the gesture began.
+ Near a screen edge the two fingers get clamped independently, which distorts
+ the ratio slightly.
+- **Mouse wheels are untouched** — only events with
+ `hasPreciseScrollingDeltas` are translated. Devices that report deltas without
+ gesture phases (e.g. Magic Mouse) get a full press-drag-release per event so
+ no finger is left pressed.
+- **`scrollToTouchScale`** is 1.5, not 1: a single trackpad swipe produces much
+ less travel than a finger covers on a phone screen, so at 1:1 an ordinary
+ swipe could not carry a gesture that has to cross a third of the screen
+ (unlock, back) to completion. It is the knob for "how much scroll per
+ centimetre" — raise it for longer travel per swipe, lower it to make the
+ finger easier to aim.
+
+---
+
+## Manual verification checklist
+
+Requires a VM with the paired base and a physical trackpad.
+
+1. Home screen: two-finger swipe left/right — pages turn in the same direction
+ as the fingers.
+2. Lock screen: an upward swipe unlocks with the pointer parked on the Home bar
+ **and** with it anywhere in the bottom 10% of the window.
+3. Settings (long list): a single long swipe scrolls the whole list from top to
+ bottom without stopping part-way, **from any pointer position** — including
+ with the pointer parked against the edge you are swiping towards.
+4. An upward swipe starting *outside* the bottom strip must still scroll, not
+ unlock.
+5. Swipe distance and speed track the fingers: no lag, no run-on after release.
+6. Photos/Maps: pinch out zooms in, pinch in zooms out.
+7. Resting the pointer over text and swiping must not start a text selection.
+8. Toggle the menu item off — trackpad gestures stop, mouse drag still works.
diff --git a/VPhoneDaemon/Daemon/GuestAPI.swift b/VPhoneDaemon/Daemon/GuestAPI.swift
index 3b52a540..398b8fb5 100644
--- a/VPhoneDaemon/Daemon/GuestAPI.swift
+++ b/VPhoneDaemon/Daemon/GuestAPI.swift
@@ -63,6 +63,7 @@ enum GuestAPI {
"ip": ip ?? "",
"capabilities": [
"touch",
+ "touch2",
"hid",
"apps",
"url",
@@ -253,6 +254,19 @@ enum GuestAPI {
phase, x: number(params, "x"), y: number(params, "y"),
normalized: params["normalized"] as? Bool ?? true,
)
+ case "input.touch2":
+ // Two fingers in one hand event. icli's `input.touch` carries a
+ // single digitizer point, so a pinch has to go through vphoned's own
+ // injection path (`vp_hid_touch2`) instead.
+ guard let phase = (params["phase"] as? String).flatMap(TouchPhase.init(rawValue:)) else {
+ throw GuestAPIError.invalidRequest("phase must be down, move or up")
+ }
+ vp_hid_touch2(
+ touchPhaseCode(phase),
+ number(params, "x1"), number(params, "y1"),
+ number(params, "x2"), number(params, "y2"),
+ )
+ return ["phase": phase.rawValue, "fingers": 2]
case "input.hid":
let page = try integer(params, "page")
let usage = try integer(params, "usage")
@@ -491,4 +505,15 @@ enum GuestAPI {
throw GuestAPIError.invalidRequest("Pass force: true to \(action)")
}
}
+
+ /// Digitizer phase codes for vphoned's own multi-finger injection path.
+ /// icli keeps its equivalent mapping module-internal, and these are the
+ /// values the private digitizer entry points take: 0 down, 1 move, 3 up.
+ static func touchPhaseCode(_ phase: TouchPhase) -> Int32 {
+ switch phase {
+ case .down: 0
+ case .move: 1
+ case .up: 3
+ }
+ }
}
diff --git a/VPhoneDaemon/Daemon/GuestHyperTextHandler.swift b/VPhoneDaemon/Daemon/GuestHyperTextHandler.swift
index f30bef18..24497cfb 100644
--- a/VPhoneDaemon/Daemon/GuestHyperTextHandler.swift
+++ b/VPhoneDaemon/Daemon/GuestHyperTextHandler.swift
@@ -236,6 +236,7 @@ final class GuestHyperTextHandler: ChannelInboundHandler, RemovableChannelHandle
case (.POST, "/v1/bootstrap/uninstall"): "bootstrap.uninstall"
case (.POST, "/v1/bootstrap/firmware"): "bootstrap.firmware"
case (.POST, "/v1/input/touch"): "input.touch"
+ case (.POST, "/v1/input/touch2"): "input.touch2"
case (.POST, "/v1/input/hid"): "input.hid"
case (.GET, "/v1/location"): "location.current"
case (.PUT, "/v1/location"): "location.set"
diff --git a/VPhoneDaemon/Daemon/main.swift b/VPhoneDaemon/Daemon/main.swift
index c197cfc9..95744ebc 100644
--- a/VPhoneDaemon/Daemon/main.swift
+++ b/VPhoneDaemon/Daemon/main.swift
@@ -14,6 +14,9 @@ if mode != 1 {
guard vp_native_watch_proxy() == 0 else { exit(1) }
vp_vcam_start()
+// Multi-finger injection is resolved here so the first pinch is not the call
+// that pays for `dlopen`; a base without the symbols logs and stays a no-op.
+_ = vp_hid_load()
GuestIrisinInstaller.refreshBootstrapOnStartup()
let group = MultiThreadedEventLoopGroup(numberOfThreads: 2)
diff --git a/VPhoneDaemon/Native/Include/VphonedNative.h b/VPhoneDaemon/Native/Include/VphonedNative.h
index f4bd6bd5..3b0ae225 100644
--- a/VPhoneDaemon/Native/Include/VphonedNative.h
+++ b/VPhoneDaemon/Native/Include/VphonedNative.h
@@ -15,6 +15,17 @@ int vp_native_run_proxy(void);
int vp_native_watch_proxy(void);
void vp_vcam_start(void);
+/// Load the IOKit digitizer symbols used for multi-finger injection. Idempotent,
+/// safe to call more than once; returns false (injection then stays a no-op)
+/// on bases that do not expose the private symbols.
+bool vp_hid_load(void);
+
+/// Inject one two-finger digitizer event, the shape a trackpad pinch needs.
+/// Phase is 0 = down, 1 = move, 3 = up; coordinates are normalized 0..1 with
+/// the origin at the top-left. Neither icli's `input.touch` nor its
+/// `touchSequence` can carry two fingers at once, which is why this exists.
+void vp_hid_touch2(int phase, double x1, double y1, double x2, double y2);
+
typedef struct {
int32_t pid;
int32_t ppid;
diff --git a/VPhoneDaemon/Native/vphoned_touch.m b/VPhoneDaemon/Native/vphoned_touch.m
new file mode 100644
index 00000000..4ea7d49c
--- /dev/null
+++ b/VPhoneDaemon/Native/vphoned_touch.m
@@ -0,0 +1,143 @@
+/*
+ * vphoned_touch — multi-finger digitizer injection.
+ *
+ * icli's `input.touch` carries a single finger, so a pinch (two fingers moving
+ * together) cannot be expressed through it. This builds one hand event holding
+ * several fingers and dispatches it, the shape WebKit's HIDEventGenerator and
+ * TrollVNC's STHIDEventGenerator use.
+ *
+ * Every entry point is resolved with dlsym, so a base without the private
+ * digitizer symbols still starts; injection then logs once and stays a no-op.
+ *
+ * Coordinates are normalized to 0..1 with the origin at the top-left, matching
+ * what `GuestAPI` hands a gesture.
+ */
+
+#import "Include/VphonedNative.h"
+
+#include <dlfcn.h>
+#include <mach/mach_time.h>
+
+typedef void *IOHIDEventSystemClientRef;
+typedef void *IOHIDEventRef;
+typedef double IOHIDFloat;
+
+static IOHIDEventSystemClientRef (*pCreate)(CFAllocatorRef);
+static void (*pSetSender)(IOHIDEventRef, uint64_t);
+static void (*pDispatch)(IOHIDEventSystemClientRef, IOHIDEventRef);
+
+static IOHIDEventRef (*pDigitizer)(CFAllocatorRef, uint64_t, uint32_t, uint32_t,
+ uint32_t, uint32_t, uint32_t, IOHIDFloat,
+ IOHIDFloat, IOHIDFloat, IOHIDFloat, IOHIDFloat,
+ boolean_t, boolean_t, uint32_t);
+static IOHIDEventRef (*pFinger)(CFAllocatorRef, uint64_t, uint32_t, uint32_t,
+ uint32_t, IOHIDFloat, IOHIDFloat, IOHIDFloat,
+ IOHIDFloat, IOHIDFloat, boolean_t, boolean_t, uint32_t);
+static void (*pAppend)(IOHIDEventRef, IOHIDEventRef, uint32_t);
+static void (*pSetInt)(IOHIDEventRef, uint32_t, int);
+
+static IOHIDEventSystemClientRef gTouchClient;
+static dispatch_queue_t gTouchQueue;
+static bool gTouchUnavailableLogged;
+
+// Digitizer event-mask bits and transducer types (IOHIDEventTypes.h).
+#define VP_DIG_RANGE 0x00000001u
+#define VP_DIG_TOUCH 0x00000002u
+#define VP_DIG_POSITION 0x00000004u
+#define VP_DIG_IDENTITY 0x00000020u
+#define VP_TRANSDUCER_HAND 1
+// kIOHIDEventFieldDigitizerIsDisplayIntegrated: (kIOHIDEventTypeDigitizer << 16) | offset.
+#define VP_FIELD_IS_DISPLAY_INTEGRATED ((((uint32_t)11) << 16) | 25)
+
+bool vp_hid_load(void) {
+ if (gTouchClient) return true;
+
+ void *ioKit = dlopen("/System/Library/Frameworks/IOKit.framework/IOKit", RTLD_NOW);
+ if (!ioKit) {
+ NSLog(@"vphoned: dlopen IOKit failed, touch injection disabled");
+ return false;
+ }
+
+ pCreate = dlsym(ioKit, "IOHIDEventSystemClientCreate");
+ pSetSender = dlsym(ioKit, "IOHIDEventSetSenderID");
+ pDispatch = dlsym(ioKit, "IOHIDEventSystemClientDispatchEvent");
+ pDigitizer = dlsym(ioKit, "IOHIDEventCreateDigitizerEvent");
+ pFinger = dlsym(ioKit, "IOHIDEventCreateDigitizerFingerEvent");
+ pAppend = dlsym(ioKit, "IOHIDEventAppendEvent");
+ pSetInt = dlsym(ioKit, "IOHIDEventSetIntegerValue");
+
+ if (!pCreate || !pSetSender || !pDispatch || !pDigitizer || !pFinger || !pAppend || !pSetInt) {
+ NSLog(@"vphoned: digitizer symbols missing, touch injection disabled");
+ return false;
+ }
+
+ gTouchClient = pCreate(kCFAllocatorDefault);
+ if (!gTouchClient) {
+ NSLog(@"vphoned: IOHIDEventSystemClientCreate returned NULL");
+ return false;
+ }
+
+ dispatch_queue_attr_t attributes = dispatch_queue_attr_make_with_qos_class(
+ DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INTERACTIVE, 0);
+ gTouchQueue = dispatch_queue_create("com.vphone.vphoned.touch", attributes);
+ return true;
+}
+
+// One hand event carrying `count` fingers, dispatched as a single gesture.
+static void dispatch_digitizer_points(const double *xs, const double *ys,
+ int count, boolean_t range, boolean_t touch,
+ uint32_t mask) {
+ if (!gTouchClient || !pDigitizer || !pFinger || !pAppend || !pSetInt || count <= 0) return;
+
+ uint64_t timestamp = mach_absolute_time();
+ IOHIDEventRef hand = pDigitizer(kCFAllocatorDefault, timestamp, VP_TRANSDUCER_HAND,
+ 0, 0, mask, 0, xs[0], ys[0], 0, 0, 0, range, touch, 0);
+ if (!hand) return;
+ pSetInt(hand, VP_FIELD_IS_DISPLAY_INTEGRATED, 1);
+
+ for (int index = 0; index < count; index++) {
+ // Finger 0 keeps index 1 / identity 2 so the single-finger path is
+ // unchanged; extra fingers carry their own identity so the guest tracks
+ // them as separate touches instead of collapsing them into one.
+ IOHIDEventRef finger = pFinger(kCFAllocatorDefault, timestamp, (uint32_t)(index + 1),
+ (uint32_t)(index + 2), mask, xs[index], ys[index],
+ 0, 0, 0, range, touch, 0);
+ if (!finger) continue;
+ pSetInt(finger, VP_FIELD_IS_DISPLAY_INTEGRATED, 1);
+ pAppend(hand, finger, 0);
+ CFRelease(finger);
+ }
+
+ IOHIDEventRef strong = (IOHIDEventRef)CFRetain(hand);
+ dispatch_async(gTouchQueue, ^{
+ pSetSender(strong, 0x8000000817319372);
+ pDispatch(gTouchClient, strong);
+ CFRelease(strong);
+ });
+ CFRelease(hand);
+}
+
+void vp_hid_touch2(int phase, double x1, double y1, double x2, double y2) {
+ if (!gTouchClient) {
+ if (!vp_hid_load() && !gTouchUnavailableLogged) {
+ gTouchUnavailableLogged = true;
+ NSLog(@"vphoned: two-finger injection unavailable on this base");
+ }
+ if (!gTouchClient) return;
+ }
+
+ const double xs[2] = {x1, x2};
+ const double ys[2] = {y1, y2};
+ switch (phase) {
+ case 0: // down
+ dispatch_digitizer_points(xs, ys, 2, 1, 1, VP_DIG_TOUCH | VP_DIG_IDENTITY);
+ break;
+ case 1: // move
+ dispatch_digitizer_points(xs, ys, 2, 1, 1, VP_DIG_POSITION);
+ break;
+ case 3: // up
+ default:
+ dispatch_digitizer_points(xs, ys, 2, 0, 0, VP_DIG_TOUCH | VP_DIG_IDENTITY);
+ break;
+ }
+}
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/GuestCommunication/VPhoneGuestControl.swift b/VPhoneExecutable/VPhoneVirtualization/UI/GuestCommunication/VPhoneGuestControl.swift
index 0dfd2df9..83641a8c 100644
--- a/VPhoneExecutable/VPhoneVirtualization/UI/GuestCommunication/VPhoneGuestControl.swift
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/GuestCommunication/VPhoneGuestControl.swift
@@ -222,6 +222,27 @@ final class VPhoneGuestControl {
enqueueInput("input.touch", params: ["phase": name, "x": x, "y": y, "normalized": true])
}
+ /// Whether the guest takes two fingers in one hand event. Guests that only
+ /// advertise `touch` cannot: icli's single-finger `input.touch` is all they
+ /// have, so a pinch has to fall back to moving one finger.
+ var supportsMultiTouch: Bool {
+ isConnected && guestCapabilities.contains("touch2")
+ }
+
+ /// Two fingers in one hand event, for a trackpad pinch. Coordinates are
+ /// normalized 0..1 with the origin at the top-left, like `sendTouch`.
+ func sendTouch2(phase: Int, x1: Double, y1: Double, x2: Double, y2: Double) {
+ let name =
+ switch phase {
+ case 0: "down"
+ case 1: "move"
+ default: "up"
+ }
+ enqueueInput("input.touch2", params: [
+ "phase": name, "x1": x1, "y1": y1, "x2": x2, "y2": y2, "normalized": true,
+ ])
+ }
+
private func enqueueInput(_ method: String, params: [String: Any]) {
let previous = orderedInput
orderedInput = Task {
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuController.swift b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuController.swift
index c76ec3b1..0c71e1a7 100644
--- a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuController.swift
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuController.swift
@@ -41,6 +41,7 @@ class VPhoneMenuController {
}
var touchIDMenuItem: NSMenuItem?
+ var trackpadGesturesItem: NSMenuItem?
var locationProvider: VPhoneLocationProvider?
var locationMenuItem: NSMenuItem?
var locationPresetMenuItem: NSMenuItem?
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuDevice.swift b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuDevice.swift
index fdc880a1..ff539984 100644
--- a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuDevice.swift
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/Menu/VPhoneMenuDevice.swift
@@ -20,6 +20,17 @@ extension VPhoneMenuController {
modifiers: [.command, .shift],
symbol: "house",
))
+ // iOS has no back key, so this item and the Esc key both replay the
+ // system back gesture. The key equivalent is here so the menu can show
+ // it; VPhoneApplication intercepts Esc before the menu is consulted,
+ // because AppKit's matching for a bare Esc missed presses.
+ menu.addItem(makeItem(
+ "Back",
+ action: #selector(sendBack),
+ keyEquivalent: "\u{1b}",
+ modifiers: [],
+ symbol: "arrow.uturn.backward",
+ ))
menu.addItem(makeItem("Power", action: #selector(sendPower), symbol: "power"))
menu.addItem(makeItem("Volume Up", action: #selector(sendVolumeUp), symbol: "speaker.plus"))
menu.addItem(makeItem("Volume Down", action: #selector(sendVolumeDown), symbol: "speaker.minus"))
@@ -50,6 +61,16 @@ extension VPhoneMenuController {
}
menu.addItem(NSMenuItem.separator())
menu.addItem(makeItem("Open Guest Spotlight", action: #selector(sendSpotlight), symbol: "magnifyingglass"))
+ // Trackpad scroll and pinch arrive as ordinary NSEvents; the view turns
+ // them into guest touches. Off hands both back to AppKit untouched.
+ let trackpadItem = makeItem(
+ "Trackpad Scroll & Pinch to Touch",
+ action: #selector(toggleTrackpadGestures),
+ symbol: "hand.draw",
+ )
+ trackpadItem.state = VPhoneTrackpadGestures.isEnabled ? .on : .off
+ trackpadGesturesItem = trackpadItem
+ menu.addItem(trackpadItem)
let tidItem = makeItem("Touch ID Home Forwarding", action: #selector(toggleTouchIDForwarding))
if hasTouchID {
let tidEnabled = !UserDefaults.standard.bool(forKey: "touchIDForwardingDisabled")
@@ -73,6 +94,13 @@ extension VPhoneMenuController {
keySender.sendHome()
}
+ /// iOS has no back key. Esc and this item both replay the system back
+ /// gesture instead of forwarding a keystroke the guest would only read as
+ /// "cancel" (or, in Safari, "stop loading").
+ @objc func sendBack() {
+ captureView?.performBackGesture()
+ }
+
@objc func sendPower() {
keySender.sendPower()
}
@@ -160,6 +188,15 @@ extension VPhoneMenuController {
}
}
+ /// Replays trackpad scroll and pinch inside the guest instead of letting
+ /// AppKit scroll the window. Persisted, so the choice survives relaunches.
+ @objc func toggleTrackpadGestures() {
+ let enabled = !VPhoneTrackpadGestures.isEnabled
+ VPhoneTrackpadGestures.isEnabled = enabled
+ trackpadGesturesItem?.state = enabled ? .on : .off
+ captureView?.trackpadGesturesEnabled = enabled
+ }
+
// MARK: - Restart
func updateRestartAvailability(available: Bool) {
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/VPhoneApplication.swift b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/VPhoneApplication.swift
index 3c7f5e96..4b15039c 100644
--- a/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/VPhoneApplication.swift
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/UserInterface/VPhoneApplication.swift
@@ -3,19 +3,25 @@ import AppKit
// MARK: - Application
/// Offers each key press in the VM window to the main menu before the VM view
-/// can take it.
+/// can take it, and takes Esc for the guest's back gesture.
///
/// `capturesSystemKeys` makes `VZVirtualMachineView` install a local event
/// monitor that sends every key to the guest and swallows it. AppKit runs
/// local monitors in the order they were added, and the view adds its own
/// again each time it becomes first responder or its window becomes key, so
/// a monitor of ours always runs after it. `sendEvent(_:)` runs before any
-/// local monitor, so the menu is asked here instead.
+/// local monitor, so both the menu lookup and the Esc handling live here.
final class VPhoneApplication: NSApplication {
/// Keys whose key-down a menu item took. Their key-up is dropped too, so
/// the guest never sees a release without a press.
private var menuKeyCodes = Set<UInt16>()
+ /// `kVK_Escape`. Intercepted here instead of as a menu key equivalent:
+ /// AppKit's matching for a modifier-less Esc is not dependable, and a missed
+ /// match forwards the key to the guest as a plain Escape — which is what
+ /// made the back gesture take a second press.
+ private static let escapeKeyCode: UInt16 = 53
+
override func sendEvent(_ event: NSEvent) {
guard event.type == .keyDown || event.type == .keyUp,
event.window?.firstResponder is VPhoneVirtualMachineView
@@ -23,6 +29,17 @@ final class VPhoneApplication: NSApplication {
super.sendEvent(event)
return
}
+ // Esc replays the guest's back gesture: iOS has no back key, and a
+ // forwarded Escape only reads as cancel. The press and its release both
+ // stop here, so the guest sees neither.
+ if event.keyCode == Self.escapeKeyCode {
+ if event.type == .keyDown,
+ let view = event.window?.firstResponder as? VPhoneVirtualMachineView
+ {
+ view.performBackGesture()
+ }
+ return
+ }
if event.type == .keyUp {
if menuKeyCodes.remove(event.keyCode) == nil {
super.sendEvent(event)
diff --git a/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swift b/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swift
index 2ea0040d..badfaa7f 100644
--- a/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swift
+++ b/VPhoneExecutable/VPhoneVirtualization/UI/VirtualMachine/VPhoneVirtualMachineView.swift
@@ -8,9 +8,68 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
var keySender: VPhoneVirtualMachineKeySender?
weak var control: VPhoneGuestControl?
+ /// Whether trackpad scroll/pinch gestures are replayed as guest touches.
+ var trackpadGesturesEnabled = VPhoneTrackpadGestures.isEnabled {
+ didSet {
+ guard !trackpadGesturesEnabled else { return }
+ finishScrollTouch()
+ endPinch()
+ }
+ }
+
private var currentTouchSwipeAim: Int = 0
private var isDragHighlightVisible = false
+ /// Synthetic guest finger driven by an in-flight trackpad scroll gesture.
+ private var scrollTouchPoint: NSPoint?
+ /// Travel accumulated before the finger is pressed down, so the large delta
+ /// that trips the start threshold is not thrown away.
+ private var scrollPending = CGPoint.zero
+ /// Travel wasted while the guest was clamping the finger to a screen edge.
+ private var scrollOverflow = CGPoint.zero
+ /// Landing spot waiting for the re-anchor gap to elapse.
+ private var scrollRebasePoint: NSPoint?
+ /// Uptime at which the finger was lifted to start a re-anchor.
+ private var scrollRebaseTime: TimeInterval = 0
+ /// Uptime of the last touch update forwarded to the guest.
+ private var scrollLastSend: TimeInterval = 0
+
+ /// Center and half-gap of the two synthetic fingers used for pinch gestures.
+ private var pinchTouchCenter: NSPoint?
+ private var pinchTouchRadius: CGFloat = 0
+
+ /// Guest points travelled per point of trackpad delta. Above 1 because a
+ /// single trackpad swipe produces far less travel than a finger covers on a
+ /// phone screen: at 1:1 an ordinary swipe could not carry a gesture that has
+ /// to travel a third of the screen (unlock, back) to completion.
+ private static let scrollToTouchScale: CGFloat = 1.5
+ /// Bottom fraction of the window treated as the Home indicator strip. See
+ /// `scrollStartPoint(for:travel:)`.
+ private static let homeEdgeSnapFraction: CGFloat = 0.10
+ /// Travel required before a scroll gesture presses a finger down. Keeps
+ /// taps and zero-travel scrolls from reaching the guest as a click.
+ private static let scrollStartThreshold: CGFloat = 2
+ /// Minimum interval between forwarded touch updates, ~70Hz: every frame of
+ /// a 60Hz trackpad gets through and a 120Hz stream is halved. Forwarding
+ /// every frame outruns the guest's HID queue, which shows up as stutter and
+ /// as the finger still drifting after the gesture ended. The finger's
+ /// position still advances on every frame; only the events in between are
+ /// dropped, and the release carries the final position.
+ private static let scrollMinSendInterval: TimeInterval = 1.0 / 70.0
+ /// Travel the guest is known to have thrown away that triggers a re-anchor.
+ private static let scrollRebaseThreshold: CGFloat = 12
+ /// Gap between lifting and re-pressing the finger during a re-anchor.
+ /// Back-to-back events sharing a timestamp get folded into one move by the
+ /// guest, which drags the swipe backwards across the whole window instead of
+ /// starting a fresh touch.
+ private static let scrollRebaseGap: TimeInterval = 0.02
+ /// Where a re-anchored finger lands, as a fraction of the window measured
+ /// from the edge it ran into towards the opposite side. Fixed rather than
+ /// derived from the pointer, so the travel a swipe gets is never a function
+ /// of where the pointer happens to sit.
+ private static let scrollRebaseLandingInset: CGFloat = 0.12
+ private static let minPinchRadius: CGFloat = 8
+
// MARK: - Private API Accessors
/// https://github.com/wh1te4ever/super-tart-vphone-writeup/blob/main/contents/ScreenSharingVNC.swift
@@ -51,6 +110,8 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
override func mouseDown(with event: NSEvent) {
// Clicking the VM display should always restore keyboard focus.
window?.makeFirstResponder(self)
+ // A synthetic trackpad finger would fight this touch for the same slot.
+ finishScrollTouch()
let localPoint = convert(event.locationInWindow, from: nil)
currentTouchSwipeAim = hitTestEdge(at: localPoint)
if sendTouchEvent(phase: 0, localPoint: localPoint, timestamp: event.timestamp) {
@@ -80,6 +141,302 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
keySender.sendHome()
}
+ // MARK: - Trackpad Gestures
+
+ /// Replay a trackpad scroll as a guest finger that presses under the pointer
+ /// and follows the physical direction of the gesture.
+ override func scrollWheel(with event: NSEvent) {
+ guard trackpadGesturesEnabled, event.hasPreciseScrollingDeltas else {
+ super.scrollWheel(with: event)
+ return
+ }
+ // A pinch in flight owns both fingers; interleaving it with a synthetic
+ // drag leaves the guest with conflicting touches and no gesture at all.
+ guard pinchTouchCenter == nil else { return }
+
+ // Inertia is synthesised by AppKit once the fingers have left the
+ // trackpad. The guest decelerates on its own when the finger lifts, so
+ // replaying the momentum stream just drags on after the user stopped.
+ guard event.momentumPhase.isEmpty else {
+ finishScrollTouch()
+ return
+ }
+
+ // Complete a re-anchor whose gap has elapsed before this event is
+ // applied, so the finger is down again when its travel lands. Travel
+ // banked during the gap is replayed against the fresh touch.
+ if let point = scrollRebasePoint, hasScrollRebaseGapElapsed() {
+ scrollRebasePoint = nil
+ beginScrollTouch(at: point)
+ flushScrollPending()
+ }
+
+ let delta = scrollDelta(of: event)
+ let pointer = convert(event.locationInWindow, from: nil)
+
+ switch event.phase {
+ case .began, .mayBegin:
+ // A new gesture supersedes whatever was still in flight.
+ finishScrollTouch()
+ followScrollDelta(delta, pointer: pointer)
+ case .ended, .cancelled:
+ followScrollDelta(delta, pointer: pointer)
+ finishScrollTouch()
+ default:
+ // Devices without gesture phases (e.g. Magic Mouse) report plain
+ // deltas and never end the gesture, so press, drag and release each
+ // event instead of leaving a finger down between them.
+ followScrollDelta(delta, pointer: pointer)
+ if event.phase.isEmpty {
+ finishScrollTouch()
+ }
+ }
+ }
+
+ /// Replay a trackpad pinch as two guest fingers spreading or closing around
+ /// the pointer position.
+ override func magnify(with event: NSEvent) {
+ guard trackpadGesturesEnabled else {
+ super.magnify(with: event)
+ return
+ }
+ // A synthetic drag owns the touch surface; ignoring the pinch keeps the
+ // guest from seeing one- and two-finger touches interleaved.
+ guard scrollTouchPoint == nil else { return }
+
+ if event.phase == .began || pinchTouchCenter == nil {
+ beginPinch(at: convert(event.locationInWindow, from: nil))
+ }
+ if pinchTouchCenter != nil {
+ applyPinch(magnification: event.magnification)
+ }
+ if event.phase == .ended || event.phase == .cancelled {
+ endPinch()
+ }
+ }
+
+ /// Apply one trackpad delta to the synthetic finger, pressing it down first
+ /// once the gesture has clearly become a drag.
+ private func followScrollDelta(_ delta: CGPoint, pointer: NSPoint) {
+ guard delta != .zero else { return }
+ // A re-anchor is mid-flight: the finger is deliberately up for the gap.
+ // Bank the travel instead of dropping it, otherwise a fast swipe would
+ // silently lose distance to every re-press.
+ guard scrollRebasePoint == nil else {
+ scrollPending.x += delta.x
+ scrollPending.y += delta.y
+ return
+ }
+
+ guard scrollTouchPoint != nil else {
+ scrollPending.x += delta.x
+ scrollPending.y += delta.y
+ guard hypot(scrollPending.x, scrollPending.y) >= Self.scrollStartThreshold else {
+ return
+ }
+ let carry = scrollPending
+ scrollPending = .zero
+ beginScrollTouch(at: scrollStartPoint(for: pointer, travel: carry))
+ if scrollTouchPoint != nil {
+ moveScrollTouch(by: carry)
+ }
+ return
+ }
+
+ moveScrollTouch(by: delta)
+ }
+
+ /// Trackpad deltas expressed in the physical direction of the gesture so the
+ /// synthetic finger follows the user's fingers.
+ ///
+ /// AppKit flips the deltas when natural scrolling is on. The two axes do not
+ /// share a baseline: on hardware the vertical axis came out matching the
+ /// fingers while the horizontal axis was mirrored, hence the extra sign on
+ /// x. This is the one place the convention lives.
+ private func scrollDelta(of event: NSEvent) -> CGPoint {
+ var dx = -event.scrollingDeltaX
+ var dy = event.scrollingDeltaY
+ if event.isDirectionInvertedFromDevice {
+ dx = -dx
+ dy = -dy
+ }
+ return CGPoint(x: dx * Self.scrollToTouchScale, y: dy * Self.scrollToTouchScale)
+ }
+
+ /// Where to press the synthetic finger: the pointer, except for an upward
+ /// swipe that starts inside the Home indicator strip, which is pressed at
+ /// the very bottom edge instead.
+ ///
+ /// The guest only treats an upward gesture as unlock / home when the touch
+ /// *begins* inside that strip; the same swipe starting a few points higher
+ /// is an ordinary drag, which is why unlocking worked with the pointer
+ /// parked on the bar and stopped as soon as it moved up a little. Snapping
+ /// only in that strip keeps the rest of the window usable for scrolling.
+ private func scrollStartPoint(for pointer: NSPoint, travel: CGPoint) -> NSPoint {
+ guard travel.y > abs(travel.x),
+ pointer.y <= bounds.height * Self.homeEdgeSnapFraction
+ else { return pointer }
+ return NSPoint(x: pointer.x, y: 0)
+ }
+
+ /// Press the synthetic finger down at `point`, so the drag starts where the
+ /// user is pointing.
+ private func beginScrollTouch(at point: NSPoint) {
+ let start = clampedTouchPoint(point)
+ scrollTouchPoint = start
+ scrollLastSend = 0
+ if !sendTouchEvent(
+ phase: 0, localPoint: start, timestamp: ProcessInfo.processInfo.systemUptime
+ ) {
+ scrollTouchPoint = nil
+ }
+ }
+
+ private func moveScrollTouch(by delta: CGPoint) {
+ guard let current = scrollTouchPoint else { return }
+ // The finger is allowed to leave the window rather than being pinned to
+ // the edge. `boundedScrollTouch` is only a guard against a stuck gesture
+ // running away.
+ let target = boundedScrollTouch(
+ NSPoint(x: current.x + delta.x, y: current.y + delta.y)
+ )
+ guard target != current else { return }
+
+ scrollTouchPoint = target
+
+ // Past the edge the guest clamps the touch to the screen and stops
+ // following it, so this travel no longer scrolls anything no matter how
+ // much of it accumulates here. Bank it and re-anchor: otherwise a swipe
+ // dies part-way through — how far it gets decided by where the pointer
+ // happened to be — and the pinned finger is read as a long press.
+ if isPastScrollEdge(target) {
+ scrollOverflow.x += delta.x
+ scrollOverflow.y += delta.y
+ if hypot(scrollOverflow.x, scrollOverflow.y) >= Self.scrollRebaseThreshold {
+ rebaseScrollTouch()
+ }
+ return
+ }
+
+ scrollOverflow = .zero
+ // Advance the finger on every frame but forward events at most at
+ // `scrollMinSendInterval`; the release below carries the final position.
+ let now = ProcessInfo.processInfo.systemUptime
+ guard now - scrollLastSend >= Self.scrollMinSendInterval else { return }
+ scrollLastSend = now
+ sendTouchEvent(phase: 1, localPoint: target, allowOutside: true, timestamp: now)
+ }
+
+ /// Lift the finger where it ran out of window and press it back down near
+ /// the opposite side, so the rest of the same swipe keeps scrolling. The
+ /// landing spot is a fraction of the window rather than anything derived
+ /// from the pointer, which is what stops a swipe from being limited by where
+ /// the pointer sits: every re-anchor hands back nearly a full window of
+ /// travel in the direction of the swipe.
+ private func rebaseScrollTouch() {
+ guard let current = scrollTouchPoint else { return }
+ let overflow = scrollOverflow
+ scrollOverflow = .zero
+ guard overflow != .zero else { return }
+
+ let inset = Self.scrollRebaseLandingInset
+ let landing = clampedTouchPoint(
+ NSPoint(
+ x: overflow.x > 0
+ ? bounds.width * inset
+ : overflow.x < 0 ? bounds.width * (1 - inset) : current.x,
+ y: overflow.y > 0
+ ? bounds.height * inset
+ : overflow.y < 0 ? bounds.height * (1 - inset) : current.y
+ )
+ )
+ guard landing != clampedTouchPoint(current) else { return }
+
+ // The gap matters: back-to-back events sharing a timestamp are folded
+ // into a single move by the guest, which drags the swipe backwards
+ // across the window instead of starting a fresh touch. The re-press is
+ // completed on the next scroll event once the gap has elapsed.
+ scrollTouchPoint = nil
+ scrollRebasePoint = landing
+ scrollRebaseTime = ProcessInfo.processInfo.systemUptime
+ sendTouchEvent(
+ phase: 3, localPoint: clampedTouchPoint(current),
+ timestamp: ProcessInfo.processInfo.systemUptime
+ )
+ }
+
+ private func hasScrollRebaseGapElapsed() -> Bool {
+ ProcessInfo.processInfo.systemUptime - scrollRebaseTime >= Self.scrollRebaseGap
+ }
+
+ /// Replay travel that arrived while the finger was up for a re-anchor.
+ private func flushScrollPending() {
+ guard scrollTouchPoint != nil, scrollPending != .zero else { return }
+ let carry = scrollPending
+ scrollPending = .zero
+ moveScrollTouch(by: carry)
+ }
+
+ /// Whether the finger has left the window, where the guest stops following
+ /// its position.
+ private func isPastScrollEdge(_ point: NSPoint) -> Bool {
+ point.x <= 0 || point.y <= 0 || point.x >= bounds.width || point.y >= bounds.height
+ }
+
+ private func finishScrollTouch() {
+ scrollPending = .zero
+ scrollOverflow = .zero
+ scrollRebasePoint = nil
+ guard let point = scrollTouchPoint else { return }
+ scrollTouchPoint = nil
+ // Release where the finger actually is, even if that is off-screen, so
+ // the guest does not see it snap back to the edge on the way up.
+ sendTouchEvent(
+ phase: 3, localPoint: point, allowOutside: true,
+ timestamp: ProcessInfo.processInfo.systemUptime
+ )
+ }
+
+ private func beginPinch(at center: NSPoint) {
+ endPinch()
+ pinchTouchCenter = clampedTouchPoint(center)
+ pinchTouchRadius = Self.initialPinchRadius(in: bounds)
+ sendPinchTouch(phase: 0)
+ }
+
+ private func applyPinch(magnification: CGFloat) {
+ let radius = pinchTouchRadius * (1 + magnification)
+ pinchTouchRadius = min(max(radius, Self.minPinchRadius), Self.maxPinchRadius(in: bounds))
+ sendPinchTouch(phase: 1)
+ }
+
+ private func endPinch() {
+ guard pinchTouchCenter != nil else { return }
+ sendPinchTouch(phase: 3)
+ pinchTouchCenter = nil
+ pinchTouchRadius = 0
+ }
+
+ private func sendPinchTouch(phase: Int) {
+ guard let center = pinchTouchCenter else { return }
+ let points = [
+ clampedTouchPoint(NSPoint(x: center.x - pinchTouchRadius, y: center.y)),
+ clampedTouchPoint(NSPoint(x: center.x + pinchTouchRadius, y: center.y)),
+ ]
+ sendTouchEvent(
+ phase: phase, localPoints: points, swipeAim: 0,
+ timestamp: ProcessInfo.processInfo.systemUptime
+ )
+ }
+
+ private static func initialPinchRadius(in bounds: CGRect) -> CGFloat {
+ max(minPinchRadius, min(bounds.width, bounds.height) * 0.15)
+ }
+
+ private static func maxPinchRadius(in bounds: CGRect) -> CGFloat {
+ max(initialPinchRadius(in: bounds), min(bounds.width, bounds.height) * 0.45)
+ }
+
// MARK: - Clipboard
/// The Edit menu's Copy, Cut and Paste reach the guest through here, so
@@ -286,17 +643,92 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
}
}
+ // MARK: - Back Gesture
+
+ /// Replay the system back gesture: a drag from the left edge toward the
+ /// centre.
+ ///
+ /// iOS has no back key. This is the gesture every app honours, so both the
+ /// Esc key and the Device menu's Back item come through here.
+ ///
+ /// The touch starts a hair inside the left edge on purpose: the system only
+ /// reads a drag as the back gesture when the touch *begins* in the
+ /// screen-edge zone. It goes through `injectSwipe`, which synthesises
+ /// ordinary mouse events, so the touch follows whichever injection path the
+ /// running base uses (vphoned on iOS 18, the native VZ multitouch path on
+ /// 26 and later).
+ /// True while a back swipe is in flight. A second press during one cannot
+ /// stack another gesture on top of it, which is what made a double press
+ /// look like nothing happened.
+ private var backGestureInFlight = false
+
+ func performBackGesture() {
+ guard !backGestureInFlight, let display = recordingGraphicsDisplay else { return }
+ let size = display.sizeInPixels
+ guard size.width > 0, size.height > 0 else { return }
+
+ backGestureInFlight = true
+ let durationMS = 250
+ // The swipe queues its steps on the main queue too, so clearing the flag
+ // from there after the last step keeps the two in order.
+ DispatchQueue.main.asyncAfter(deadline: .now() + Double(durationMS) / 1000 + 0.05) {
+ [weak self] in
+ self?.backGestureInFlight = false
+ }
+
+ injectSwipe(
+ fromX: size.width * 0.004,
+ fromY: size.height * 0.5,
+ toX: size.width * 0.85,
+ toY: size.height * 0.5,
+ screenWidth: Int(size.width),
+ screenHeight: Int(size.height),
+ durationMs: durationMS,
+ )
+ }
+
// MARK: - Legacy Touch Injection (macOS 15)
@discardableResult
- private func sendTouchEvent(phase: Int, localPoint: NSPoint, timestamp: TimeInterval) -> Bool {
- let normalizedPoint = normalizeCoordinate(localPoint)
+ private func sendTouchEvent(
+ phase: Int, localPoint: NSPoint, allowOutside: Bool = false, timestamp: TimeInterval
+ ) -> Bool {
+ sendTouchEvent(
+ phase: phase, localPoints: [localPoint], swipeAim: currentTouchSwipeAim,
+ allowOutside: allowOutside, timestamp: timestamp
+ )
+ }
+
+ /// Inject one or more fingers at `localPoints` (view coordinates) in a single
+ /// touch event. Guest-side injection carries at most two fingers; the native
+ /// VZ multitouch path numbers them by array index.
+ ///
+ /// `allowOutside` forwards coordinates that fall beyond the view instead of
+ /// clamping them, which the scroll gesture uses to keep a swipe alive after
+ /// it has passed the window edge.
+ @discardableResult
+ private func sendTouchEvent(
+ phase: Int, localPoints: [NSPoint], swipeAim: Int, allowOutside: Bool = false,
+ timestamp: TimeInterval
+ ) -> Bool {
+ guard !localPoints.isEmpty else { return false }
+ let normalized = localPoints.map { normalizeCoordinate($0, allowOutside: allowOutside) }
// iOS 18 bases: the VZ USB touchscreen dext emits no digitizer events on
// the 26.x kernel, so route touches through vphoned's guest-side HID
// injection. 26.x bases fall through to the native VZ multitouch path.
if let control, control.useGuestTouchInjection {
- control.sendTouch(phase: phase, x: Double(normalizedPoint.x), y: Double(normalizedPoint.y))
+ if normalized.count >= 2, control.supportsMultiTouch {
+ control.sendTouch2(
+ phase: phase,
+ x1: Double(normalized[0].x), y1: Double(normalized[0].y),
+ x2: Double(normalized[1].x), y2: Double(normalized[1].y)
+ )
+ } else {
+ control.sendTouch(
+ phase: phase, x: Double(normalized[0].x), y: Double(normalized[0].y)
+ )
+ }
return true
}
@@ -304,21 +736,23 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
virtualMachine != nil
else { return false }
- let touch = Dynamic._VZTouch(
- view: self,
- index: 0,
- phase: phase,
- location: normalizedPoint,
- swipeAim: currentTouchSwipeAim,
- timestamp: timestamp,
- )
+ let touches = normalized.enumerated().compactMap { index, point in
+ Dynamic._VZTouch(
+ view: self,
+ index: index,
+ phase: phase,
+ location: point,
+ swipeAim: swipeAim,
+ timestamp: timestamp
+ ).asObject
+ }
- guard let touchObj = touch.asObject else {
+ guard touches.count == normalized.count else {
print("[vphone] Error: Failed to create _VZTouch")
return false
}
- let touchEvent = Dynamic._VZMultiTouchEvent(touches: [touchObj])
+ let touchEvent = Dynamic._VZMultiTouchEvent(touches: touches)
guard let eventObj = touchEvent.asObject else { return false }
Dynamic(device).sendMultiTouchEvents([eventObj] as NSArray)
@@ -327,6 +761,26 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
// MARK: - Coordinate Helpers
+ private func clampedTouchPoint(_ point: NSPoint) -> NSPoint {
+ let inset: CGFloat = 1
+ let maxX = max(inset, bounds.width - inset)
+ let maxY = max(inset, bounds.height - inset)
+ return NSPoint(
+ x: min(max(point.x, inset), maxX),
+ y: min(max(point.y, inset), maxY)
+ )
+ }
+
+ /// Soft bound for a finger that is allowed past the window edge: wide enough
+ /// that no single swipe reaches it, tight enough that a wedged gesture can
+ /// not drift to absurd coordinates.
+ private func boundedScrollTouch(_ point: NSPoint) -> NSPoint {
+ NSPoint(
+ x: min(max(point.x, -bounds.width), bounds.width * 2),
+ y: min(max(point.y, -bounds.height), bounds.height * 2)
+ )
+ }
+
/// The guest display as drawn in this view. Full screen letterboxes it,
/// so touches are measured against it rather than `bounds`.
private var displayGeometry: VPhoneDisplayGeometry {
@@ -337,8 +791,22 @@ class VPhoneVirtualMachineView: VZVirtualMachineView {
)
}
- private func normalizeCoordinate(_ localPoint: NSPoint) -> CGPoint {
- displayGeometry.normalizedPoint(localPoint)
+ /// Map a view-local point into the guest's normalized touch space,
+ /// measured against the drawn display rather than the view (a full-screen
+ /// VM letterboxes it).
+ ///
+ /// `allowOutside` keeps coordinates that fall beyond the display instead of
+ /// clamping them to its edge. The scroll gesture needs it: once the finger
+ /// reaches the edge the guest stops receiving new positions, so a long
+ /// swipe could not keep scrolling.
+ private func normalizeCoordinate(_ localPoint: NSPoint, allowOutside: Bool = false) -> CGPoint {
+ let geometry = displayGeometry
+ guard allowOutside else { return geometry.normalizedPoint(localPoint) }
+ let rect = geometry.displayRect
+ guard rect.width > 0, rect.height > 0 else { return .zero }
+ let x = (localPoint.x - rect.minX) / rect.width
+ let y = (localPoint.y - rect.minY) / rect.height
+ return CGPoint(x: x, y: geometry.isFlipped ? y : 1 - y)
}
private func hitTestEdge(at point: CGPoint) -> Int {
@@ -358,3 +826,16 @@ extension VPhoneVirtualMachineView: NSMenuItemValidation {
}
}
}
+
+// MARK: - Trackpad Gesture Preference
+
+/// User preference for replaying trackpad scroll/pinch gestures as guest
+/// touches. Enabled by default; the Device menu exposes the toggle.
+enum VPhoneTrackpadGestures {
+ private static let disabledKey = "trackpadGesturesDisabled"
+
+ static var isEnabled: Bool {
+ get { !UserDefaults.standard.bool(forKey: disabledKey) }
+ set { UserDefaults.standard.set(!newValue, forKey: disabledKey) }
+ }
+}