Skip to content

Liquid Glass and Apple Music touches for the Spotify iOS app. A GPL-3.0 fork of spoti.pw 0.21.1.

C

feat: Spatial voice keeps the singer in front as your head turns

My-Name-Is-Jeff0ba2d5d3ea5f
diff --git a/docs/tweaks.md b/docs/tweaks.md
index df1e55c6..4dbe10a6 100644
--- a/docs/tweaks.md
+++ b/docs/tweaks.md
@@ -159,9 +159,15 @@ Shared:
and an engine that stands in Speed and pitch's chain between Spotify's mixer and its output
(SGPlayerSetStage), pulls the mixer a few seconds ahead of what plays, separates two-second windows
there on a worker thread and mixes the vocals down on the render thread. Spotify's clock has the lead
- taken off (SPTPlayerState's positionAsOfTimestamp), and a seek, a skip or a stop drops it. The mic is
- on the redesign's lyrics (Redesigned/Lyrics/SGRSingButton.m). Tested on the Mac against harness/sing/,
- the download against its download/
+ taken off (SPTPlayerState's positionAsOfTimestamp), and a seek, a skip or a stop drops it. Spatial
+ voice holds the separated vocals in front as the head turns: HeadGestures' motion gives the yaw off a
+ front that follows the head over 20 s, and the render thread pans the vocals' middle at equal power,
+ narrowed, with the far ear up to 0.65 ms late and low-passed; it stands down for iOS's own spatial
+ audio. The Spatial voice page's preview (SGSpatialPreview.m) turns a disc of dots under the listener by
+ the same front (SGSpatialVoiceAngle, SGSingEngine.h), on Core Animation alone and only while the page
+ shows. The mic is on the redesign's lyrics (Redesigned/Lyrics/SGRSingButton.m). Tested on the Mac
+ against harness/sing/ (spatial voice and its front without the model: `build/sing spatial`), the
+ download against its download/, the pages in the simulator against harness/spatial-page/
Haptics/ Vibrations (Haptics.h lists its files): a tap of UIKit's feedback generators for the player's and the now
playing bar's controls, the scrubber's tenths and ends, cover swipes, gestures and the lyrics page's tap to
seek, at the strength set for them (ControlHaptics.x, SGFeedback.m); and Music Haptics, Core Haptics
@@ -179,7 +185,9 @@ Shared:
Spotify's collection platform (addURL:showUIConfirmation:completion:), and a shake, which skips; a
tone played through Spotify's playback session confirms each, whatever the Ring/Silent switch says.
Learning one's own nod and shake sets the thresholds, each kept only if the detector, set to it,
- finds the gesture in what was recorded. The detector is
+ finds the gesture in what was recorded. The one CMHeadphoneMotionManager is lent to other features
+ (SGHeadMotionListen: Sing's spatial voice), and runs for them with the switch off, the detector then
+ unfed. The detector is
tested on the Mac against harness/head-gestures/, the hook and the page in the simulator against its sim/
LiveActivity/ a Live Activity on the lock screen and in the Dynamic Island in one of three views, the line being
sung with the next one under it, the tracks up next (a tap on one skipping ahead to it), or a control
@@ -328,7 +336,10 @@ lyrics for every track, naming the source in the redesign, the lock screen, and
the redesign also which of the lyrics, their pronunciation and their translation is set largest, and the
translation's language. Sing, under either look, on the main page and as Lyrics' first row, the row reading out On,
Off or how far the voice model's download has come, kept up to date while the page shows: Sing's switch, which turns
-the mic on and off at once, its Status (a tap says more), the Vocals slider (gone, as sung, the vocals alone), the voice model's download, Runs on (GPU, GPU
+the mic on and off at once, its Status (a tap says more), the Vocals slider (gone, as sung, the vocals alone), Spatial voice
+(where the iPhone reads headphone motion; a page of its own, reading out On or Off, with a live preview at its top that
+follows the head through AirPods, or sways gently without them and holds still under Reduce Motion, a line under it
+saying which, then the switch), the voice model's download, Runs on (GPU, GPU
and Neural Engine, Neural Engine) and Ignore heat warnings, all applying straight away. Player: Gestures, AirPods gestures, Blocked
artists (with the count on the row) and Lock screen widget (Moving artwork, Lyrics or Every song, and the
lyrics' style, Still or Animated), which work with either look;
diff --git a/harness/sing/README.md b/harness/sing/README.md
index 5eb7478b..7f37631e 100644
--- a/harness/sing/README.md
+++ b/harness/sing/README.md
@@ -17,15 +17,32 @@ files from https://huggingface.co/My-Name-Is-Jeff/vitrine-sing, a mirror of Dark
with Sing on from the start: the mix plays dry while the lead fills, then exactly the mix less the vocals the
offline pass finds, the lead given back after Sing is switched off and the mix playing on where it was, the
top of the slider playing the vocals alone, a flush dropping the lead, and no allocation on the render thread
- (`malloc_logger`).
+ (`malloc_logger`);
+- spatial voice, without the model (a separator that hands each window back whole as vocals, so what plays is the
+ vocals placed): a 440 Hz tone at level 1 plays exactly as it came straight ahead; set 90 degrees right, the right
+ ear has it 7.75 dB louder (the narrowed pan's 7.66 and the far ear's low-pass) at the same power, and the left ear
+ 31 frames later (the 28.7 frames around a head and the low-pass's own lag); 90 degrees left mirrors it; back ahead
+ it is exact again; no step from one frame to the next is larger than the tone's own at the louder gain, so no
+ turn clicks; and nothing is allocated on the render thread;
+- the front spatial voice holds the voice off (SGSpatialVoiceAngle, shared with the Spatial voice page's preview),
+ fed at 25 Hz: a head turned 60 degrees left has the voice 60 degrees right, and 20 s on 22 (the front's 1/e);
+ yaw wrapping across 180 degrees moves the voice by the 1 degree it turned; a gap of 2 s starts it ahead again.
The voice is macOS's own speech, `say -o speech.aiff "..."` (any file ExtAudioFile reads will do), repeated a
second apart over 30 seconds.
./build.sh && build/sing <separator.mlmodelc> <voice> <out dir> [all|cpu|gpu|ane]
./build.sh thread && build/sing-thread <separator.mlmodelc> <voice> <out dir> gpu (ThreadSanitizer)
+ ./build.sh && build/sing spatial (spatial voice alone)
-The out dir gets `mix.wav`, `vocals.wav`, `accompaniment.wav` and `engine.wav` (what played).
+The out dir gets `mix.wav`, `vocals.wav`, `accompaniment.wav`, `engine.wav` (what played) and `spatial.wav` (the
+tone turned; the temporary folder's with `spatial` alone).
+
+2026-10-05, spatial voice: everything passes, under ThreadSanitizer too. The full run passes with it; one run on a
+Mac loaded far past its cores had the model fall behind (790 ms a window, 3.6 s played dry) and failed the two
+vocals scores; the harness built from before spatial voice passed right after it, and this one on the next run.
+
+2026-10-06, the front: `build/sing spatial` passes with its four checks.
2026-10-05, a MacBook with an M4, macOS 27: everything passes under `gpu` and under ThreadSanitizer.
diff --git a/harness/sing/main.m b/harness/sing/main.m
index ba7d4f08..56dbaf24 100644
--- a/harness/sing/main.m
+++ b/harness/sing/main.m
@@ -9,8 +9,15 @@
// thread that keeps real time, while the engine's worker separates ahead: the lead it builds, every frame
// out in order, the vocals level applied, a flush, the lead given back when Sing is switched off, and
// no allocation on the render thread.
+// - spatial voice, without the model: a separator that hands the whole window back as vocals, so what plays is
+// the vocals placed. A tone straight ahead plays exactly as it came; at 90 degrees right the right ear has it
+// louder by the pan's 7.7 dB at the same power and the left ear late by the delay around the head; at 90
+// degrees left the same mirrored; back ahead it is exact again; and no turn clicks. Then the front the voice
+// is held off: a turned head has it off to the side, the front catching up over 20 s, across +-180 degrees
+// without a jump, and ahead again after a gap in the motion.
//
// ./build.sh && build/sing <separator.mlmodelc> <voice> <out dir> [all|cpu|gpu|ane]
+// ./build.sh && build/sing spatial
#import <Foundation/Foundation.h>
#import <Accelerate/Accelerate.h>
#import <AudioToolbox/AudioToolbox.h>
@@ -356,11 +363,132 @@ static void checkEngine(SGSingSeparator *separator, Audio mix, Audio voice, Audi
SGSingEngineDestroy(engine);
}
+#pragma mark - spatial voice
+
+// Every window is all vocals.
+@interface SGWholeSeparator : SGSingSeparator
+@end
+
+@implementation SGWholeSeparator
+- (BOOL)separateLeft:(const float *)left right:(const float *)right vocalsLeft:(float *)vocalsLeft vocalsRight:(float *)vocalsRight error:(NSError **)error {
+ memcpy(vocalsLeft, left, kSGSingWindowFrames * sizeof(float));
+ memcpy(vocalsRight, right, kSGSingWindowFrames * sizeof(float));
+ return YES;
+}
+@end
+
+static double energy(const float *samples, size_t from, size_t to) {
+ double sum = 0;
+ for (size_t i = from; i < to; i++) sum += (double)samples[i] * samples[i];
+ return sum;
+}
+
+// The lag in frames by which the left channel follows the right, the best of +-45 (the tone's period is 100).
+static int leftLag(Audio audio, size_t from, size_t to) {
+ int best = 0;
+ double bestSum = -INFINITY;
+ for (int lag = -45; lag <= 45; lag++) {
+ double sum = 0;
+ for (size_t i = from; i < to; i++) sum += (double)audio.right[i] * audio.left[i + lag];
+ if (sum > bestSum) bestSum = sum, best = lag;
+ }
+ return best;
+}
+
+static void checkSpatial(void) {
+ size_t frames = 14 * kSGSingRate;
+ Audio tone = makeAudio(frames);
+ for (size_t i = 0; i < frames; i++) tone.left[i] = tone.right[i] = (float)(0.5 * sin(2 * M_PI * 440 * i / kSGSingRate));
+ SGSingEngine *engine = SGSingEngineCreate();
+ SGSingEngineSetSeparator(engine, [[SGWholeSeparator alloc] initWithModel:nil]);
+ SGSingEngineSetLevel(engine, 1);
+ SGSingEngineSetOn(engine, true);
+ Source source = {tone, 0};
+ atomic_store(&sg_renderAllocations, 0);
+ malloc_logger = countAllocation;
+ Audio out = play(engine, tone, &source, 1024, 14, ^(double played) {
+ if (played >= 5 && played < 8) SGSingEngineSetVoiceAngle(engine, (float)M_PI_2);
+ if (played >= 8 && played < 11) SGSingEngineSetVoiceAngle(engine, -(float)M_PI_2);
+ if (played >= 11) SGSingEngineSetVoiceAngle(engine, 0);
+ });
+ malloc_logger = NULL;
+ CHECK(atomic_load(&sg_renderAllocations) == 0, "spatial voice: no allocation on the render thread (%u)", atomic_load(&sg_renderAllocations));
+ SGSingEngineStats stats = SGSingEngineReadStats(engine);
+ CHECK(stats.ready > 0 && stats.dryFrames < 4 * kSGSingRate, "spatial voice: the vocals are in (%.2f s separated ahead, %.2f s played dry)",
+ stats.ready, stats.dryFrames / (double)kSGSingRate);
+
+ double ahead = 0;
+ for (size_t i = 3 * kSGSingRate; i < 5 * kSGSingRate; i++) ahead = fmax(ahead, fmax(fabsf(out.left[i] - tone.left[i]), fabsf(out.right[i] - tone.right[i])));
+ CHECK(ahead == 0, "straight ahead the vocals play exactly as they came (%.2g)", ahead);
+
+ // Expected at 90 degrees: the pan's gains, sqrt 2 times the cosine and sine of (0.5 + 1) * 45 degrees.
+ double panDB = 20 * log10(sin(1.5 * M_PI_4) / cos(1.5 * M_PI_4));
+ size_t rightFrom = (size_t)(6.5 * kSGSingRate), rightTo = (size_t)(7.5 * kSGSingRate);
+ size_t leftFrom = (size_t)(9.5 * kSGSingRate), leftTo = (size_t)(10.5 * kSGSingRate);
+ double toneEnergy = energy(tone.left, rightFrom, rightTo);
+ double rightDB = 10 * log10(energy(out.right, rightFrom, rightTo) / energy(out.left, rightFrom, rightTo));
+ double rightPower = 10 * log10((energy(out.left, rightFrom, rightTo) + energy(out.right, rightFrom, rightTo)) / (2 * toneEnergy));
+ int rightLag = leftLag(out, rightFrom, rightTo);
+ CHECK(fabs(rightDB - panDB) < 0.5 && fabs(rightPower) < 0.5,
+ "at 90 degrees right the right ear is %.2f dB louder (the pan's %.2f), at %+.2f dB of the power ahead", rightDB, panDB, rightPower);
+ // The delay around the head, 28.7 frames, and the low-pass's own lag at 440 Hz, about 2.
+ CHECK(rightLag >= 27 && rightLag <= 33, "and the left ear hears it %d frames later (%.2f ms)", rightLag, rightLag * 1000.0 / kSGSingRate);
+ double leftDB = 10 * log10(energy(out.left, leftFrom, leftTo) / energy(out.right, leftFrom, leftTo));
+ int leftLagFrames = leftLag(out, leftFrom, leftTo);
+ CHECK(fabs(leftDB - panDB) < 0.5 && leftLagFrames <= -27 && leftLagFrames >= -33,
+ "at 90 degrees left the left ear is %.2f dB louder and the right ear %d frames later", leftDB, -leftLagFrames);
+
+ double back = 0;
+ for (size_t i = 12 * kSGSingRate; i < 14 * kSGSingRate - 1024; i++) back = fmax(back, fmax(fabsf(out.left[i] - tone.left[i]), fabsf(out.right[i] - tone.right[i])));
+ CHECK(back == 0, "back straight ahead, exactly as they came again (%.2g)", back);
+
+ // No click: from one frame to the next the output moves no further than the tone at its loudest placement.
+ double toneStep = 0, step = 0;
+ for (size_t i = 3 * kSGSingRate; i < 14 * kSGSingRate - 1024; i++) {
+ toneStep = fmax(toneStep, fabsf(tone.left[i] - tone.left[i - 1]));
+ step = fmax(step, fmax(fabsf(out.left[i] - out.left[i - 1]), fabsf(out.right[i] - out.right[i - 1])));
+ }
+ double loudest = M_SQRT2 * sin(1.5 * M_PI_4);
+ CHECK(step <= toneStep * loudest * 1.02, "no click through the turns: the largest step is %.4f, the tone's %.4f times the louder gain %.3f",
+ step, toneStep, loudest);
+ writeWAV(@"spatial.wav", out);
+ SGSingEngineDestroy(engine);
+}
+
+// The front the voice is held off (SGSpatialVoiceAngle), fed at 25 Hz as AirPods send their motion.
+static void checkFront(void) {
+ SGSpatialFront front = {0};
+ double angle = SGSpatialVoiceAngle(&front, 0.3, 10);
+ CHECK(angle == 0, "the first motion is the front: the voice straight ahead (%.3f)", angle);
+ // The head turned 60 degrees left (yaw grows) at once, then held there.
+ double turned = SGSpatialVoiceAngle(&front, 0.3 + M_PI / 3, 10.04), held = turned;
+ double time = 10.04;
+ for (; time < 30.04 - 1e-9; time += 0.04) held = SGSpatialVoiceAngle(&front, 0.3 + M_PI / 3, time + 0.04);
+ CHECK(fabs(turned - M_PI / 3) < 0.01 && fabs(held - M_PI / 3 * exp(-1)) < 0.01,
+ "a head turned 60 degrees left has the voice %.1f degrees right, and 20 s on %.1f (the front's 1/e: %.1f)",
+ turned * 180 / M_PI, held * 180 / M_PI, 60 * exp(-1));
+ // Across the back: yaw wraps from +179 to -179 degrees, which is 2 degrees further left, not 358 right.
+ SGSpatialFront back = {0};
+ SGSpatialVoiceAngle(&back, M_PI - 0.01, 0);
+ double wrapped = SGSpatialVoiceAngle(&back, -M_PI + 0.01, 0.04);
+ CHECK(fabs(wrapped - 0.02) < 0.001, "across +-180 degrees the voice moves %.2f degrees", wrapped * 180 / M_PI);
+ // Headphones out for 2 s and in again: the front starts over where the head points.
+ double again = SGSpatialVoiceAngle(&front, -1, time + 2);
+ CHECK(again == 0, "after a gap in the motion the voice is ahead again (%.3f)", again);
+}
+
int main(int argc, char **argv) {
setvbuf(stdout, NULL, _IOLBF, 0);
@autoreleasepool {
+ if (argc == 2 && !strcmp(argv[1], "spatial")) {
+ sg_outDir = NSTemporaryDirectory();
+ checkSpatial();
+ checkFront();
+ printf("%s\n", sg_failures ? "FAILED" : "all passed");
+ return sg_failures ? 1 : 0;
+ }
if (argc < 4) {
- printf("usage: sing <separator.mlmodelc> <voice> <out dir> [all|cpu|gpu|ane]\n");
+ printf("usage: sing <separator.mlmodelc> <voice> <out dir> [all|cpu|gpu|ane]\n sing spatial\n");
return 2;
}
NSString *modelPath = @(argv[1]), *units = argc > 4 ? @(argv[4]) : @"all";
@@ -401,6 +529,8 @@ int main(int argc, char **argv) {
writeWAV(@"accompaniment.wav", accompaniment);
checkEngine(separator, mix, padded, vocals);
+ checkSpatial();
+ checkFront();
printf("%s\n", sg_failures ? "FAILED" : "all passed");
return sg_failures ? 1 : 0;
}
diff --git a/harness/sing/sim/main.m b/harness/sing/sim/main.m
index 5e260d58..aeacaea2 100644
--- a/harness/sing/sim/main.m
+++ b/harness/sing/sim/main.m
@@ -42,6 +42,13 @@ id SGPlayerState(void) {
return nil;
}
+// Shared/HeadGestures' head motion, for spatial voice: none in the simulator, so the voice stays ahead.
+void SGHeadMotionListen(NSString *name, void (^handler)(id motion)) {}
+void SGHeadMotionAskPermission(void) {}
+BOOL SGHeadGesturesAvailable(void) {
+ return NO;
+}
+
@interface SPTPlayerState : NSObject
@property (nonatomic) double positionAsOfTimestamp;
@property (nonatomic, strong) NSDate *timestamp;
diff --git a/harness/spatial-page/README.md b/harness/spatial-page/README.md
new file mode 100644
index 00000000..0567080c
--- /dev/null
+++ b/harness/spatial-page/README.md
@@ -0,0 +1,31 @@
+# Spatial voice page harness
+
+The Sing page (`tweak/Sources/Shared/Sing/SingSettings.m`) and the Spatial voice page under it, with its preview
+(`SGSpatialPreview.m`), on the real Settings/ framework. `stubs.m` stands in for Sing's state (a model on the
+phone, Sing off) and for HeadGestures' motion, which it plays from a script of the head's yaw at 25 motions a
+second while the preview listens; it also answers Motion & Fitness's permission as the launch line says.
+
+ ./build.sh
+ xcrun simctl install <udid> build/SpatialPageHarness.app
+ xcrun simctl launch <udid> com.vojta.spatialpageharness on allowed head=60 spatial
+ xcrun simctl io <udid> screenshot shot.png
+
+Other agents use the simulator too: make a device of your own (`xcrun simctl create`) and address it by UDID.
+`main.m` lists the setup words (`on`, `allowed`, `denied`, `head=sweep`, `head=<degrees>`, `width=<points>`,
+`slow`) and the actions, played one every 0.7 s from 1 s in: `spatial` opens the page, `toggle`, `pop`, and `dump`,
+which logs the rows, the header's size and what VoiceOver reads on the preview. The stub logs each listener
+coming and going:
+
+ xcrun simctl launch --console-pty <udid> com.vojta.spatialpageharness on allowed head=sweep spatial dump pop dump
+
+Reduce Motion: `xcrun simctl spawn <udid> defaults write com.apple.Accessibility ReduceMotionEnabled -bool true`,
+then launch again (and `false` after).
+
+2026-10-06, iPhone 17 Pro on iOS 27.0: the Sing page's Spatial voice row reading On and opening the page; the
+preview swaying with no permission, with Motion & Fitness denied (its caption), and following a head turned 60
+degrees left (the voice 60 degrees right, back to about 36 after 10 s as the front follows); the listener taken
+out on pop and back on push; Reduce Motion holding the disc still, the ripples gone, the turn faded to in steps;
+the header growing with its caption at the largest text sizes, and the disc fitting a 320-point window.
+
+What it does not cover: real AirPods (the yaw's sign is not yet heard, see SGSpatialVoiceAngle), how often they
+send motion, and what the preview costs on a phone's battery.
diff --git a/harness/spatial-page/build.sh b/harness/spatial-page/build.sh
new file mode 100755
index 00000000..2f1f7fc4
--- /dev/null
+++ b/harness/spatial-page/build.sh
@@ -0,0 +1,41 @@
+#!/bin/sh
+# Builds the Spatial voice page harness for the simulator: Shared/Sing's settings and preview, the Settings/
+# framework as they are in the tweak, stubs.m for Sing's state and the head's motion.
+set -e
+SRC=$(cd "$(dirname "$0")/../../tweak/Sources" && pwd)
+OUT=$(dirname "$0")/build
+rm -rf "$OUT"; mkdir -p "$OUT/SpatialPageHarness.app"
+
+SDK=$(xcrun --sdk iphonesimulator --show-sdk-path)
+xcrun -sdk iphonesimulator clang -target arm64-apple-ios17.0-simulator -fobjc-arc -g -O0 \
+ -I"$SRC" -isysroot "$SDK" -Wall -Werror -Wno-deprecated-declarations \
+ "$(dirname "$0")/main.m" "$(dirname "$0")/stubs.m" "$(dirname "$0")/../scene.m" \
+ "$SRC"/Shared/Sing/SingSettings.m "$SRC"/Shared/Sing/SGSpatialPreview.m \
+ "$SRC"/Settings/SGPage.m "$SRC"/Settings/SGPageStyle.m "$SRC"/Settings/SGModPage.m "$SRC"/Settings/SGGlowSwitch.m \
+ "$SRC"/Core/SGLog.m "$SRC"/Core/SGPrefs.m "$SRC"/Core/SGViewTree.m "$SRC"/Core/SGFlagForce.m "$SRC"/Core/SGUIMode.m \
+ -framework UIKit -framework QuartzCore -framework CoreGraphics -framework CoreMotion -framework Foundation \
+ -o "$OUT/SpatialPageHarness.app/SpatialPageHarness"
+
+cat > "$OUT/SpatialPageHarness.app/Info.plist" <<'PLIST'
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0"><dict>
+<key>CFBundleExecutable</key><string>SpatialPageHarness</string>
+<key>CFBundleIdentifier</key><string>com.vojta.spatialpageharness</string>
+<key>CFBundleName</key><string>SpatialPageHarness</string>
+<key>CFBundleVersion</key><string>1</string>
+<key>CFBundleShortVersionString</key><string>1.0</string>
+<key>UIUserInterfaceStyle</key><string>Dark</string>
+<key>UILaunchScreen</key><dict/>
+<key>UIApplicationSceneManifest</key><dict>
+ <key>UIApplicationSupportsMultipleScenes</key><false/>
+ <key>UISceneConfigurations</key><dict>
+ <key>UIWindowSceneSessionRoleApplication</key><array><dict>
+ <key>UISceneConfigurationName</key><string>Default</string>
+ <key>UISceneDelegateClassName</key><string>SGRHarnessScene</string>
+ </dict></array>
+ </dict>
+</dict>
+</dict></plist>
+PLIST
+echo "built $OUT/SpatialPageHarness.app"
diff --git a/harness/spatial-page/main.m b/harness/spatial-page/main.m
new file mode 100644
index 00000000..bfc15613
--- /dev/null
+++ b/harness/spatial-page/main.m
@@ -0,0 +1,124 @@
+// The Sing page (Shared/Sing/SingSettings.m) and the Spatial voice page under it, with its preview
+// (SGSpatialPreview.m), on the real Settings/ framework; stubs.m stands in for Sing's state and plays the head's
+// motion. The launch line sets things up and then plays actions, one every 0.7 s from 1 s in; screenshot after.
+//
+// ./build.sh && xcrun simctl install <udid> build/SpatialPageHarness.app
+// xcrun simctl launch <udid> com.vojta.spatialpageharness [setup...] [action...]
+//
+// Setup: keep (the stored Sing keys stay; otherwise they are cleared first), on (spatial voice on), allowed or
+// denied (what Motion & Fitness answers; unasked otherwise, so the preview never listens), head=sweep (the head
+// turning 45 degrees each way every 6 s), head=<degrees> (still, then turned that far left at 2 s and held), width=<points>
+// (the window that narrow, centred), slow (animations at a tenth of their speed).
+// Actions: spatial (the Spatial voice row tapped), toggle=<section>.<row> (that row's switch flipped the way a tap
+// does), pop, dump (the rows, the header's height and what VoiceOver reads on the preview, to the log).
+#import <CoreMotion/CoreMotion.h>
+#import <UIKit/UIKit.h>
+#import "Core/SGCore.h"
+#import "Settings/SGModPage.h"
+#import "Shared/Sing/Sing.h"
+
+extern CMAuthorizationStatus sg_motionAllowed;
+extern double (^sg_headYaw)(double seconds);
+
+static void findViews(UIView *root, Class kind, NSMutableArray *found) {
+ if ([root isKindOfClass:kind]) [found addObject:root];
+ for (UIView *sub in root.subviews) findViews(sub, kind, found);
+}
+
+@interface AppDelegate : UIResponder <UIApplicationDelegate>
+@property (nonatomic, strong) UIWindow *window;
+@property (nonatomic, strong) UINavigationController *nav;
+@end
+
+@implementation AppDelegate
+
+- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)options {
+ NSArray<NSString *> *args = NSProcessInfo.processInfo.arguments;
+ NSUserDefaults *store = NSUserDefaults.standardUserDefaults;
+ if (![args containsObject:@"keep"]) {
+ for (NSString *key in store.dictionaryRepresentation.allKeys) {
+ if ([key hasPrefix:@"spotifyglass.sing"]) [store removeObjectForKey:key];
+ }
+ }
+ CGFloat width = 0;
+ NSMutableArray<NSString *> *actions = [NSMutableArray array];
+ for (NSString *arg in [args subarrayWithRange:NSMakeRange(1, args.count - 1)]) {
+ if ([arg isEqualToString:@"on"]) SGSetEnabled(SGKeySingSpatial, YES);
+ else if ([arg isEqualToString:@"allowed"]) sg_motionAllowed = CMAuthorizationStatusAuthorized;
+ else if ([arg isEqualToString:@"denied"]) sg_motionAllowed = CMAuthorizationStatusDenied;
+ else if ([arg isEqualToString:@"head=sweep"]) sg_headYaw = ^double(double t) { return M_PI_4 * sin(2 * M_PI * t / 6); };
+ else if ([arg hasPrefix:@"head="]) {
+ double turn = [arg substringFromIndex:5].doubleValue * M_PI / 180;
+ sg_headYaw = ^double(double t) {
+ double s = fmin(1, fmax(0, (t - 2) / 0.6));
+ return turn * s * s * (3 - 2 * s);
+ };
+ } else if ([arg hasPrefix:@"width="]) width = [arg substringFromIndex:6].doubleValue;
+ else if (![@[@"keep", @"slow", @"on"] containsObject:arg]) [actions addObject:arg];
+ }
+
+ CGRect screen = UIScreen.mainScreen.bounds;
+ self.window = [[UIWindow alloc] initWithFrame:width > 0 ? CGRectMake((screen.size.width - width) / 2, 0, width, screen.size.height) : screen];
+ self.window.overrideUserInterfaceStyle = UIUserInterfaceStyleDark;
+ self.nav = [[UINavigationController alloc] initWithRootViewController:SGSingSettingsPage()];
+ self.window.rootViewController = self.nav;
+ [self.window makeKeyAndVisible];
+ if ([args containsObject:@"slow"]) self.window.layer.speed = 0.1;
+
+ [actions enumerateObjectsUsingBlock:^(NSString *action, NSUInteger i, BOOL *stop) {
+ dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)((1 + 0.7 * i) * NSEC_PER_SEC)), dispatch_get_main_queue(), ^{
+ NSLog(@"[harness] %@", action);
+ [self run:action];
+ });
+ }];
+ return YES;
+}
+
+- (UITableView *)table {
+ return ((UITableViewController *)self.nav.topViewController).tableView;
+}
+
+- (void)run:(NSString *)action {
+ NSArray<NSString *> *parts = [action componentsSeparatedByString:@"="];
+ NSString *verb = parts.firstObject, *value = parts.count > 1 ? parts[1] : @"";
+ UITableView *table = self.table;
+ if ([verb isEqualToString:@"spatial"]) {
+ [table.delegate tableView:table didSelectRowAtIndexPath:[NSIndexPath indexPathForRow:3 inSection:0]];
+ } else if ([verb isEqualToString:@"toggle"]) {
+ NSArray<NSString *> *at = [value componentsSeparatedByString:@"."];
+ UITableViewCell *cell = [table cellForRowAtIndexPath:[NSIndexPath indexPathForRow:at[1].integerValue inSection:at[0].integerValue]];
+ NSMutableArray<UISwitch *> *switches = [NSMutableArray array];
+ findViews(cell, UISwitch.class, switches);
+ UISwitch *toggle = switches.firstObject;
+ [toggle setOn:!toggle.on animated:YES];
+ [toggle sendActionsForControlEvents:UIControlEventValueChanged];
+ } else if ([verb isEqualToString:@"pop"]) {
+ [self.nav popViewControllerAnimated:YES];
+ } else if ([verb isEqualToString:@"dump"]) {
+ UIView *header = table.tableHeaderView;
+ NSLog(@"[harness] header %@ %.0fx%.0f; VoiceOver reads \"%@\", \"%@\"", NSStringFromClass(header.class), header.bounds.size.width,
+ header.bounds.size.height, header.accessibilityLabel, header.accessibilityValue);
+ for (NSInteger section = 0; section < table.numberOfSections; section++) {
+ NSMutableArray<NSString *> *rows = [NSMutableArray array];
+ for (NSInteger row = 0; row < [table numberOfRowsInSection:section]; row++) {
+ UITableViewCell *cell = [table cellForRowAtIndexPath:[NSIndexPath indexPathForRow:row inSection:section]];
+ NSMutableArray<UILabel *> *labels = [NSMutableArray array];
+ findViews(cell.contentView, UILabel.class, labels);
+ NSMutableArray<NSString *> *texts = [NSMutableArray array];
+ for (UILabel *label in labels) if (label.text.length) [texts addObject:label.text];
+ [rows addObject:[texts componentsJoinedByString:@" / "]];
+ }
+ NSLog(@"[harness] section %ld: %@", (long)section, [rows componentsJoinedByString:@", "]);
+ }
+ }
+}
+
+@end
+
+int main(int argc, char *argv[]) {
+ @autoreleasepool {
+ // The redesign's look: black, the redesign's own accent.
+ [NSUserDefaults.standardUserDefaults setBool:YES forKey:SGKeyRedesign];
+ return UIApplicationMain(argc, argv, nil, NSStringFromClass(AppDelegate.class));
+ }
+}
diff --git a/harness/spatial-page/stubs.m b/harness/spatial-page/stubs.m
new file mode 100644
index 00000000..ce54f13a
--- /dev/null
+++ b/harness/spatial-page/stubs.m
@@ -0,0 +1,102 @@
+// What the Sing page and the Spatial voice page call outside their own files: Sing.x's and SGSingModel.m's state,
+// a model on the phone and Sing off, and Shared/HeadGestures' motion, played from a script of the head's yaw
+// at 25 motions a second, as AirPods send them, while the preview listens.
+#import <CoreMotion/CoreMotion.h>
+#import <objc/runtime.h>
+#import "Core/SGCore.h"
+#import "Shared/Sing/Sing.h"
+
+#pragma mark - Sing
+
+NSString *const SGSingChangedNotification = @"SGSingChangedNotification";
+NSString *SGSingStatusText(void) { return SGSingOn() ? @"Singing" : @"Off"; }
+NSString *SGSingStatusDetail(void) { return nil; }
+NSString *SGSingMissing(void) { return nil; }
+BOOL SGSingOn(void) { return SGHidden(SGKeySing); }
+void SGSetSingOn(BOOL on) { SGSetEnabled(SGKeySing, on); }
+float SGSingLevel(void) { return 0.15f; }
+void SGSetSingLevel(float level) {}
+void SGSetSingIgnoresHeat(BOOL ignores) {}
+BOOL SGSingSpatialAvailable(void) { return YES; }
+BOOL SGSingSpatial(void) { return SGHidden(SGKeySingSpatial); }
+void SGSetSingSpatial(BOOL on) {
+ SGSetEnabled(SGKeySingSpatial, on);
+ NSLog(@"[harness] spatial voice %@", on ? @"on" : @"off");
+}
+void SGSingComputeUnitsChanged(void) {}
+SGSingModelState SGSingModelCurrentState(void) { return SGSingModelReady; }
+double SGSingModelProgress(void) { return 1; }
+BOOL SGSingModelWaitingForNetwork(void) { return NO; }
+NSString *SGSingModelError(void) { return nil; }
+NSString *SGSingModelSizeText(void) { return @"190 MB"; }
+void SGSingDownloadModel(void) {}
+void SGSingCancelModelDownload(void) {}
+void SGSingDeleteModel(void) {}
+NSArray<NSString *> *SGSingComputeUnitNames(void) { return @[@"GPU and Neural Engine", @"GPU", @"Neural Engine"]; }
+BOOL SGSingOSSupported(void) { return YES; }
+BOOL SGSingDeviceSupported(void) { return YES; }
+
+#pragma mark - the head
+
+@interface FakeAttitude : NSObject
+@property (nonatomic) double yaw;
+@end
+@implementation FakeAttitude
+@end
+
+@interface FakeMotion : NSObject
+@property (nonatomic) NSTimeInterval timestamp;
+@property (nonatomic, strong) FakeAttitude *attitude;
+@end
+@implementation FakeMotion
+@end
+
+// From main.m's launch line: what Motion & Fitness answers, and the head's yaw at a time since launch (NAN: no
+// headphones, so no motion).
+CMAuthorizationStatus sg_motionAllowed = CMAuthorizationStatusNotDetermined;
+double (^sg_headYaw)(double seconds);
+
+static CMAuthorizationStatus fakeAuthorization(id self, SEL _cmd) { return sg_motionAllowed; }
+
+__attribute__((constructor)) static void sg_takeOverPermission(void) {
+ method_setImplementation(class_getClassMethod(CMHeadphoneMotionManager.class, @selector(authorizationStatus)), (IMP)fakeAuthorization);
+}
+
+static NSMutableDictionary<NSString *, void (^)(CMDeviceMotion *)> *sg_listeners;
+static dispatch_queue_t sg_queue;
+static dispatch_source_t sg_timer;
+
+void SGHeadMotionListen(NSString *name, void (^handler)(CMDeviceMotion *motion)) {
+ if (!sg_listeners) {
+ sg_listeners = [NSMutableDictionary dictionary];
+ sg_queue = dispatch_queue_create("harness.motion", DISPATCH_QUEUE_SERIAL);
+ }
+ void (^old)(CMDeviceMotion *) = sg_listeners[name];
+ sg_listeners[name] = [handler copy];
+ NSLog(@"[harness] %@ %@ the head", name, handler ? @"listens to" : @"stops listening to");
+ if (old && !handler) dispatch_async(sg_queue, ^{ old(nil); });
+ NSArray *handlers = sg_listeners.allValues;
+ if (handlers.count && !sg_timer && sg_headYaw) {
+ static CFTimeInterval launched;
+ if (!launched) launched = CACurrentMediaTime();
+ sg_timer = dispatch_source_create(DISPATCH_SOURCE_TYPE_TIMER, 0, 0, sg_queue);
+ dispatch_source_set_timer(sg_timer, DISPATCH_TIME_NOW, NSEC_PER_SEC / 25, 0);
+ dispatch_source_set_event_handler(sg_timer, ^{
+ double now = CACurrentMediaTime() - launched, yaw = sg_headYaw(now);
+ if (isnan(yaw)) return;
+ FakeMotion *motion = [FakeMotion new];
+ motion.timestamp = CACurrentMediaTime();
+ motion.attitude = [FakeAttitude new];
+ motion.attitude.yaw = yaw;
+ for (void (^each)(CMDeviceMotion *) in handlers) each((CMDeviceMotion *)motion);
+ });
+ dispatch_resume(sg_timer);
+ } else if (!handlers.count && sg_timer) {
+ dispatch_source_cancel(sg_timer);
+ sg_timer = nil;
+ }
+}
+
+void SGHeadMotionAskPermission(void) {
+ NSLog(@"[harness] Motion & Fitness asked for");
+}
diff --git a/tweak/Sources/Shared/Sing/SGSingEngine.h b/tweak/Sources/Shared/Sing/SGSingEngine.h
index 37bef31b..ac25fa2e 100644
--- a/tweak/Sources/Shared/Sing/SGSingEngine.h
+++ b/tweak/Sources/Shared/Sing/SGSingEngine.h
@@ -9,6 +9,13 @@
// they are not yet (as the lead fills, or when the model falls behind). Switched off, it pulls half of
// what plays until the lead is given back, and is a straight pull again.
//
+// Spatial voice: the vocals' middle (left plus right, halved) can be put at an angle off straight ahead,
+// where Sing.x holds it as the head turns. The ear facing it hears it louder and the other quieter at the same
+// power (an equal-power pan, narrowed so that the far ear never loses it), and the far ear hears it up to
+// 0.65 ms later (the time sound takes around a head) and duller, through a one-pole low-pass for the head's
+// shadow. The vocals' sides (left less right, halved) stay where they are, so straight ahead plays the vocals
+// exactly as separated.
+//
// What plays is the lead behind what Spotify's decoder has handed over, which is what Spotify's clock
// counts: Sing.x takes the lead off the player's position, and asks for a flush when Spotify seeks or
// skips, which drops the sound held from before it.
@@ -16,6 +23,7 @@
// Threading: SGSingEngineRender on the render thread only, allocation and lock free; the worker is the
// engine's own; everything else from any thread.
#import <Foundation/Foundation.h>
+#import <math.h>
@class SGSingSeparator;
@@ -37,6 +45,33 @@ void SGSingEngineSetOn(SGSingEngine *engine, bool on);
bool SGSingEngineOn(SGSingEngine *engine);
// The vocals' level from 0 (gone) through 1 (as the song has them) to 2 (the vocals alone, the rest gone).
void SGSingEngineSetLevel(SGSingEngine *engine, float level);
+// Where the separated voice sounds, in radians to the listener's right of straight ahead (left is negative,
+// behind sounds as the front does): 0, where it starts, leaves the vocals exactly as the song has them. The
+// render glides to a new angle over a few tens of milliseconds, so it can be set as often as the head moves.
+void SGSingEngineSetVoiceAngle(SGSingEngine *engine, float radians);
+
+// Where spatial voice holds the voice, for Sing.x and the Spatial voice page's preview alike: off a front that
+// follows where the head points over 20 s, so the voice drifts back ahead of a head that stays turned and the
+// attitude's own drift never carries it off. A gap in the motion over 1 s (headphones out and in again), or a
+// front cleared to zero, starts it over where the head points.
+typedef struct {
+ bool hasFront;
+ double front, last;
+} SGSpatialFront;
+
+// The voice's angle off the head in radians to its right, from a head motion's yaw and timestamp (seconds).
+static inline double SGSpatialVoiceAngle(SGSpatialFront *f, double yaw, double time) {
+ // CoreMotion's yaw turns counterclockwise seen from above, so it grows as the head turns left, and the voice
+ // held ahead is then off to the head's right. Not yet heard on AirPods; flip it here.
+ const double yawToRight = 1;
+ const double frontSeconds = 20, motionGap = 1;
+ double since = time - f->last;
+ if (!f->hasFront || since < 0 || since > motionGap) f->front = yaw;
+ else f->front = remainder(f->front + remainder(yaw - f->front, 2 * M_PI) * (1 - exp(-since / frontSeconds)), 2 * M_PI);
+ f->hasFront = true;
+ f->last = time;
+ return yawToRight * remainder(yaw - f->front, 2 * M_PI);
+}
// Holds the worker (a hot phone): what plays is dry, and the lead is given back.
void SGSingEngineSetPaused(SGSingEngine *engine, bool paused);
// Drops the sound held ahead, at the next render.
@@ -50,6 +85,7 @@ typedef struct {
double lead, targetLead; // seconds
double ready; // seconds of the held sound separated, ahead of what plays (negative when behind)
double averageMS; // a window's separation, a running average
+ double voiceAngle; // radians, as last set
unsigned long long windows, failures, dryFrames; // dry: frames played unseparated while on
} SGSingEngineStats;
SGSingEngineStats SGSingEngineReadStats(SGSingEngine *engine);
diff --git a/tweak/Sources/Shared/Sing/SGSingEngine.m b/tweak/Sources/Shared/Sing/SGSingEngine.m
index b618dce4..5be82d7a 100644
--- a/tweak/Sources/Shared/Sing/SGSingEngine.m
+++ b/tweak/Sources/Shared/Sing/SGSingEngine.m
@@ -15,6 +15,13 @@ static const double kLeadPerModelTime = 1.5, kLeadSpare = 0.25;
static const float kFadeSeconds = 0.1f;
// How often the worker looks for a new window.
static const useconds_t kPoll = 10000;
+// Spatial voice: at 90 degrees the pan goes this far of the way to one ear, the far ear hears the voice this
+// many frames late (0.65 ms, sound's way around a head) and through a one-pole low-pass of this coefficient
+// (about 3 kHz); the angle glides to where it is set over this long.
+static const float kVoiceWidth = 0.5f;
+static const float kVoiceDelay = 0.00065f * kSGSingRate;
+static const float kVoiceShadow = 0.35f;
+static const float kVoiceGlideSeconds = 0.05f;
struct SGSingEngine {
float *left, *right, *vocalsLeft, *vocalsRight; // rings, a frame at position p at p % kRing
@@ -25,11 +32,13 @@ struct SGSingEngine {
_Atomic uint64_t base; // where the worker starts over, at the latest flush
atomic_uint generation; // counts the flushes
atomic_bool flushAsked, on, paused, hasSeparator;
- atomic_uint levelBits, targetLead;
+ atomic_uint levelBits, targetLead, voiceAngleBits;
// The render thread's own.
bool running; // the rings are in use: Sing is on, or the lead is not given back yet
float amount, vocalsGain, otherGain;
+ float voiceAngle; // where the voice is now, gliding to voiceAngleBits
+ float shadowLeft, shadowRight; // the low-passes' last outputs, an ear each
// The worker's.
pthread_t worker;
@@ -85,6 +94,38 @@ static void gainsFor(float level, float *vocals, float *other) {
*other = level <= 1 ? 1 : 2 - level;
}
+// Where the voice is put at an angle: each ear's gain, delay in frames and low-pass coefficient (1 passes).
+typedef struct {
+ float gain[2], delay[2], shadow[2];
+} Placement;
+
+static Placement placementAt(float angle) {
+ if (angle == 0) return (Placement){{1, 1}, {0, 0}, {1, 1}};
+ // From -1 (the left ear) to 1 (the right): behind sounds as the front does, as it does to two ears.
+ float side = sinf(angle), far = fabsf(side);
+ // Equal power, with straight ahead at 1 each: the gains' squares always add up to 2.
+ float pan = (kVoiceWidth * side + 1) * (float)M_PI_4;
+ float gainLeft = (float)M_SQRT2 * cosf(pan), gainRight = (float)M_SQRT2 * sinf(pan);
+ float delay = far * kVoiceDelay, shadow = 1 - far * (1 - kVoiceShadow);
+ return side > 0 ? (Placement){{gainLeft, gainRight}, {delay, 0}, {shadow, 1}}
+ : (Placement){{gainLeft, gainRight}, {0, delay}, {1, shadow}};
+}
+
+// The vocals' middle at `position`.
+static inline float middleAt(SGSingEngine *engine, uint64_t position) {
+ uint64_t index = position % kRing;
+ return (engine->vocalsLeft[index] + engine->vocalsRight[index]) * 0.5f;
+}
+
+// The vocals' middle `delay` frames before `position`, between two frames; at once where that is before the
+// rings started over at `base`, whose frames are a different moment's.
+static inline float middleBefore(SGSingEngine *engine, uint64_t position, uint64_t base, float delay) {
+ uint64_t whole = (uint64_t)delay;
+ float part = delay - (float)whole;
+ if (position < base + whole + 1) return middleAt(engine, position);
+ return middleAt(engine, position - whole) * (1 - part) + middleAt(engine, position - whole - 1) * part;
+}
+
OSStatus SGSingEngineRender(SGSingEngine *engine, UInt32 frames, float *left, float *right, SGSingPull pull, void *context) {
// Larger than any slice an output asks for, and more than the rings leave room for: passed as it is.
if (frames > kSGSingRate / 4) return pull(context, frames, left, right);
@@ -123,6 +164,17 @@ OSStatus SGSingEngineRender(SGSingEngine *engine, UInt32 frames, float *left, fl
gainsFor(loadFloat(&engine->levelBits), &vocalsTo, &otherTo);
float vocalsStep = (vocalsTo - engine->vocalsGain) / frames, otherStep = (otherTo - engine->otherGain) / frames;
float fade = 1.0f / (kFadeSeconds * kSGSingRate), amountTo = on ? 1 : 0;
+
+ // The voice's angle glides the short way round towards the one set, and lands on it; within the buffer
+ // each ear's gain, delay and low-pass move in a straight line from where the angle was to where it is.
+ float angleTo = loadFloat(&engine->voiceAngleBits), angleFrom = engine->voiceAngle;
+ float turn = remainderf(angleTo - angleFrom, 2 * (float)M_PI);
+ float angle = fabsf(turn) < 1e-4f ? angleTo : remainderf(angleFrom + turn * (1 - expf(-(float)frames / (kVoiceGlideSeconds * kSGSingRate))), 2 * (float)M_PI);
+ engine->voiceAngle = angle;
+ bool placed = angleFrom != 0 || angle != 0;
+ Placement from = placementAt(angleFrom), to = placementAt(angle);
+ uint64_t base = atomic_load_explicit(&engine->base, memory_order_relaxed);
+
uint64_t dry = 0;
for (UInt32 i = 0; i < frames; i++) {
uint64_t position = played + i, index = position % kRing;
@@ -131,10 +183,23 @@ OSStatus SGSingEngineRender(SGSingEngine *engine, UInt32 frames, float *left, fl
engine->otherGain += otherStep;
if (position < ready) {
engine->amount += fmaxf(-fade, fminf(amountTo - engine->amount, fade));
- float a = engine->amount, vocalsGain = engine->vocalsGain - 1, otherGain = engine->otherGain - 1;
+ float a = engine->amount, otherGain = engine->otherGain - 1;
float v = engine->vocalsLeft[index], w = engine->vocalsRight[index];
- left[i] = x + a * (vocalsGain * v + otherGain * (x - v));
- right[i] = y + a * (vocalsGain * w + otherGain * (y - w));
+ // The vocals where they are to sound: as separated straight ahead.
+ float placedLeft = v, placedRight = w;
+ if (placed) {
+ float t = (float)(i + 1) / (float)frames, sides = (v - w) * 0.5f;
+ float leftIn = middleBefore(engine, position, base, from.delay[0] + (to.delay[0] - from.delay[0]) * t);
+ float rightIn = middleBefore(engine, position, base, from.delay[1] + (to.delay[1] - from.delay[1]) * t);
+ engine->shadowLeft += (from.shadow[0] + (to.shadow[0] - from.shadow[0]) * t) * (leftIn - engine->shadowLeft);
+ engine->shadowRight += (from.shadow[1] + (to.shadow[1] - from.shadow[1]) * t) * (rightIn - engine->shadowRight);
+ placedLeft = (from.gain[0] + (to.gain[0] - from.gain[0]) * t) * engine->shadowLeft + sides;
+ placedRight = (from.gain[1] + (to.gain[1] - from.gain[1]) * t) * engine->shadowRight - sides;
+ } else {
+ engine->shadowLeft = engine->shadowRight = (v + w) * 0.5f;
+ }
+ left[i] = x + a * (engine->vocalsGain * placedLeft - v + otherGain * (x - v));
+ right[i] = y + a * (engine->vocalsGain * placedRight - w + otherGain * (y - w));
} else {
engine->amount = 0;
left[i] = x;
@@ -304,6 +369,10 @@ void SGSingEngineSetLevel(SGSingEngine *engine, float level) {
storeFloat(&engine->levelBits, level);
}
+void SGSingEngineSetVoiceAngle(SGSingEngine *engine, float radians) {
+ storeFloat(&engine->voiceAngleBits, isfinite(radians) ? remainderf(radians, 2 * (float)M_PI) : 0);
+}
+
void SGSingEngineSetPaused(SGSingEngine *engine, bool paused) {
atomic_store(&engine->paused, paused);
}
@@ -326,6 +395,7 @@ SGSingEngineStats SGSingEngineReadStats(SGSingEngine *engine) {
.targetLead = atomic_load(&engine->targetLead) / (double)kSGSingRate,
.ready = ((double)ready - (double)played) / kSGSingRate,
.averageMS = atomic_load(&engine->averageMicroseconds) / 1000.0,
+ .voiceAngle = loadFloat(&engine->voiceAngleBits),
.windows = atomic_load(&engine->windows),
.failures = atomic_load(&engine->failures),
.dryFrames = atomic_load(&engine->dryFrames),
diff --git a/tweak/Sources/Shared/Sing/SGSpatialPreview.m b/tweak/Sources/Shared/Sing/SGSpatialPreview.m
new file mode 100644
index 00000000..b9d32980
--- /dev/null
+++ b/tweak/Sources/Shared/Sing/SGSpatialPreview.m
@@ -0,0 +1,405 @@
+// The Spatial voice page's preview (Sing.h): the listener in the middle of a disc of dots seen from just behind
+// and above the head, the voice a point of the accent colour in front with its light on the floor under it.
+// The disc is the room and the camera is the head: as the head turns the disc turns the other way under it, so
+// the voice keeps its place in the room and comes round to the side, by the angle SGSpatialVoiceAngle hands the
+// engine (SGSingEngine.h). Rings go out slowly from the voice, as sound does. With no head to follow, the disc
+// sways slowly to show the same thing.
+//
+// It is drawn once into layers and moved by Core Animation alone: the turn is the disc layer's rotation and the
+// voice's position, each motion animated from where it is on screen to the next over a little more than the
+// motion's interval, so nothing runs per frame on the main thread. It listens to the head only while it is in
+// a window, Spotify is in front and Motion & Fitness is allowed (the page never asks; the switch does). Under
+// Reduce Motion the disc holds still and a turn of the head past a step fades it to the new angle.
+//
+// Threading: main thread, but for the motion handler, which hands its angle to the main queue.
+#import <CoreMotion/CoreMotion.h>
+#import "Core/SGCore.h"
+#import "Settings/SGPageStyle.h"
+#import "Shared/HeadGestures/HeadGestures.h"
+#import "SGSingEngine.h"
+#import "Sing.h"
+
+static NSString *const kListener = @"sing.preview";
+static const CGFloat kStageHeight = 200;
+static const CGFloat kMiddle = 0.47; // the listener, down the stage
+static const CGFloat kMaxRadius = 168; // the disc's, at the widest phones
+static const CGFloat kTilt = 66 * M_PI / 180; // the disc tipped away from the camera
+static const CGFloat kDistance = 560; // the camera's, for the perspective
+static const int kRings = 10;
+static const CGFloat kDotSpacing = 7.5, kDotRadius = 1.15;
+static const CGFloat kVoiceRing = 0.62; // of the radius
+static const CGFloat kVoiceHeight = 12; // over the floor, as the listener's head is
+// The sway while there is no head to follow: far enough to see the voice come round, slow enough to be calm,
+// starting and turning back with no velocity.
+static const double kSwayAngle = 38 * M_PI / 180, kSwaySeconds = 12;
+// Each motion is animated to over this long, a little more than AirPods' interval, so the turn never stops
+// between two of them.
+static const CFTimeInterval kFollowSeconds = 0.12;
+// Under Reduce Motion, the turn the voice must move by before the still picture fades to it.
+static const double kStillStep = 15 * M_PI / 180;
+
+@implementation SGSpatialPreview {
+ CALayer *_stage, *_plane, *_field, *_listener, *_voice;
+ CAReplicatorLayer *_ripples;
+ UILabel *_caption;
+ CGFloat _radius;
+ double _angle; // the disc's turn as last set, unwrapped
+ BOOL _listening, _live, _swaying;
+ CMAuthorizationStatus _allowed;
+}
+
+- (instancetype)initWithFrame:(CGRect)frame {
+ if (!(self = [super initWithFrame:frame])) return nil;
+ self.preservesSuperviewLayoutMargins = YES;
+ self.isAccessibilityElement = YES;
+ self.accessibilityTraits = UIAccessibilityTraitImage | UIAccessibilityTraitUpdatesFrequently;
+ _stage = [CALayer layer];
+ [self.layer addSublayer:_stage];
+ _caption = [UILabel new];
+ _caption.font = SGSubtitleFont();
+ _caption.adjustsFontForContentSizeCategory = YES;
+ _caption.textColor = SGGrey();
+ _caption.numberOfLines = 0;
+ [self addSubview:_caption];
+ for (NSNotificationName name in @[UIApplicationDidBecomeActiveNotification, UIApplicationDidEnterBackgroundNotification,
+ UIAccessibilityReduceMotionStatusDidChangeNotification]) {
+ [NSNotificationCenter.defaultCenter addObserver:self selector:@selector(refresh) name:name object:nil];
+ }
+ return self;
+}
+
+- (void)dealloc {
+ if (_listening) SGHeadMotionListen(kListener, nil);
+}
+
+#pragma mark - the picture
+
+// A point of the floor, in points from the disc's middle (y negative ahead) and `height` over it, where the
+// camera sees it with the disc turned by `turn`, from the stage's middle; `scale` how much nearer it is.
+- (CGPoint)project:(CGPoint)point height:(CGFloat)height turn:(double)turn scale:(CGFloat *)scale {
+ CGFloat x = point.x * cos(turn) - point.y * sin(turn), y = point.x * sin(turn) + point.y * cos(turn);
+ CGFloat down = y * cos(kTilt) - height * sin(kTilt), near = y * sin(kTilt) + height * cos(kTilt);
+ CGFloat w = 1 - near / kDistance;
+ if (scale) *scale = 1 / w;
+ return CGPointMake(x / w, down / w);
+}
+
+- (CGPoint)voiceAt:(double)turn {
+ CGPoint at = [self project:CGPointMake(0, -kVoiceRing * _radius) height:kVoiceHeight turn:turn scale:NULL];
+ return CGPointMake(_stage.bounds.size.width / 2 + at.x, _stage.bounds.size.height * kMiddle + at.y);
+}
+
+static CALayer *glowDot(CGFloat size, UIColor *color) {
+ CALayer *dot = [CALayer layer];
+ dot.bounds = CGRectMake(0, 0, size, size);
+ dot.cornerRadius = size / 2;
+ dot.backgroundColor = color.CGColor;
+ dot.shadowColor = color.CGColor;
+ dot.shadowOpacity = 0.9;
+ dot.shadowRadius = size * 0.9;
+ dot.shadowOffset = CGSizeZero;
+ dot.shadowPath = [UIBezierPath bezierPathWithOvalInRect:dot.bounds].CGPath;
+ return dot;
+}
+
+// A soft round light `size` across, fading out from `alpha` in its middle.
+static CAGradientLayer *floorLight(CGPoint at, CGFloat size, UIColor *color, CGFloat alpha) {
+ CAGradientLayer *light = [CAGradientLayer layer];
+ light.type = kCAGradientLayerRadial;
+ light.bounds = CGRectMake(0, 0, size, size);
+ light.position = at;
+ light.colors = @[(id)[color colorWithAlphaComponent:alpha].CGColor, (id)[color colorWithAlphaComponent:alpha * 0.35].CGColor,
+ (id)[color colorWithAlphaComponent:0].CGColor];
+ light.locations = @[@0, @0.45, @1];
+ light.startPoint = CGPointMake(0.5, 0.5);
+ light.endPoint = CGPointMake(1, 1);
+ return light;
+}
+
+// Everything is drawn again for a new width; the turn carries on from where it was.
+- (void)build {
+ for (CALayer *layer in [_stage.sublayers copy]) [layer removeFromSuperlayer];
+ CGFloat width = self.bounds.size.width, scale = UIScreen.mainScreen.scale;
+ _radius = MIN(kMaxRadius, width / 2 - 12);
+ _stage.frame = CGRectMake(0, 0, width, kStageHeight);
+ CGPoint middle = CGPointMake(width / 2, kStageHeight * kMiddle);
+ UIColor *accent = SGGreen();
+
+ // The floor: tipped away once, the room turning on it.
+ _plane = [CALayer layer];
+ _plane.bounds = CGRectMake(0, 0, 2 * _radius, 2 * _radius);
+ _plane.position = middle;
+ CATransform3D camera = CATransform3DIdentity;
+ camera.m34 = -1 / kDistance;
+ _plane.sublayerTransform = CATransform3DConcat(CATransform3DMakeRotation(kTilt, 1, 0, 0), camera);
+ [_stage addSublayer:_plane];
+
+ // The listener's own faint light on the floor, which never turns, as the head is the camera.
+ [_plane addSublayer:floorLight(CGPointMake(_radius, _radius), 0.4 * _radius, UIColor.whiteColor, 0.16)];
+
+ _field = [CALayer layer];
+ _field.frame = _plane.bounds;
+ [_field setValue:@(_angle) forKeyPath:@"transform.rotation.z"];
+ [_plane addSublayer:_field];
+ // Rings of dots spaced evenly along each, dimmer outwards, one layer a ring.
+ for (int ring = 1; ring <= kRings; ring++) {
+ CGFloat r = _radius * ring / kRings;
+ int count = (int)round(2 * M_PI * r / kDotSpacing);
+ UIBezierPath *dots = [UIBezierPath bezierPath];
+ for (int i = 0; i < count; i++) {
+ double a = 2 * M_PI * (i + 0.5 * (ring % 2)) / count;
+ CGPoint at = CGPointMake(_radius + r * cos(a), _radius + r * sin(a));
+ [dots moveToPoint:CGPointMake(at.x + kDotRadius, at.y)];
+ [dots addArcWithCenter:at radius:kDotRadius startAngle:0 endAngle:2 * M_PI clockwise:YES];
+ }
+ CAShapeLayer *layer = [CAShapeLayer layer];
+ layer.frame = _field.bounds;
+ layer.path = dots.CGPath;
+ layer.fillColor = UIColor.whiteColor.CGColor;
+ layer.opacity = 0.46 * (1 - 0.6 * pow((double)ring / kRings, 1.4));
+ layer.contentsScale = scale;
+ [_field addSublayer:layer];
+ }
+ // The voice's light on the floor, and the sound going out from it.
+ CGPoint spot = CGPointMake(_radius, _radius - kVoiceRing * _radius);
+ CAGradientLayer *pool = floorLight(spot, 0.7 * _radius, accent, 0.6);
+ [_field addSublayer:pool];
+ _ripples = [CAReplicatorLayer layer];
+ _ripples.bounds = pool.bounds;
+ _ripples.position = spot;
+ CAShapeLayer *ring = [CAShapeLayer layer];
+ ring.frame = pool.bounds;
+ ring.path = [UIBezierPath bezierPathWithOvalInRect:pool.bounds].CGPath;
+ ring.fillColor = nil;
+ ring.strokeColor = accent.CGColor;
+ ring.lineWidth = 1.5;
+ ring.opacity = 0;
+ ring.contentsScale = scale;
+ [_ripples addSublayer:ring];
+ [_field addSublayer:_ripples];
+
+ _listener = glowDot(8, [UIColor colorWithWhite:1 alpha:0.95]);
+ _listener.shadowOpacity = 0.5;
+ CGPoint head = [self project:CGPointZero height:kVoiceHeight turn:0 scale:NULL];
+ _listener.position = CGPointMake(middle.x + head.x, middle.y + head.y);
+ [_stage addSublayer:_listener];
+ _voice = glowDot(10, accent);
+ _voice.borderColor = [UIColor colorWithWhite:1 alpha:0.7].CGColor;
+ _voice.borderWidth = 1.5;
+ _voice.position = [self voiceAt:_angle];
+ [_stage addSublayer:_voice];
+ _swaying = NO;
+ [self moveOn];
+}
+
+- (void)layoutSubviews {
+ [super layoutSubviews];
+ CGFloat width = self.bounds.size.width, inset = self.layoutMargins.left;
+ if (width > 0 && fabs(_stage.bounds.size.width - width) > 0.5) [self build];
+ CGFloat height = ceil([_caption sizeThatFits:CGSizeMake(width - 2 * inset, CGFLOAT_MAX)].height);
+ _caption.frame = CGRectMake(inset, kStageHeight + 4, width - 2 * inset, height);
+}
+
+- (CGSize)sizeThatFits:(CGSize)size {
+ CGFloat inset = self.layoutMargins.left;
+ CGFloat caption = ceil([_caption sizeThatFits:CGSizeMake(size.width - 2 * inset, CGFLOAT_MAX)].height);
+ return CGSizeMake(size.width, kStageHeight + 4 + caption + 8);
+}
+
+#pragma mark - the turn
+
+// The disc through `turns` (the voice's angle, to the listener's right) over `seconds`, and the voice round its
+// ring with it rather than across. A repeating one (the sway) starts at `begin`, in media time.
+- (void)animateTurns:(NSArray<NSNumber *> *)turns duration:(CFTimeInterval)seconds repeatFrom:(CFTimeInterval)begin key:(NSString *)key {
+ NSMutableArray *positions = [NSMutableArray array];
+ for (NSNumber *turn in turns) [positions addObject:[NSValue valueWithCGPoint:[self voiceAt:turn.doubleValue]]];
+ CAKeyframeAnimation *rotation = [CAKeyframeAnimation animationWithKeyPath:@"transform.rotation.z"];
+ rotation.values = turns;
+ CAKeyframeAnimation *position = [CAKeyframeAnimation animationWithKeyPath:@"position"];
+ position.values = positions;
+ for (CAKeyframeAnimation *animation in @[rotation, position]) {
+ animation.duration = seconds;
+ if (begin <= 0) continue;
+ animation.repeatCount = HUGE_VALF;
+ animation.beginTime = begin;
+ // A slow turn is smooth at 60 frames a second, so it asks for no more. Its maximum stays at 120: a cap at 60
+ // would hold the player's transitions down with it, should the player open over the page.
+ animation.preferredFrameRateRange = CAFrameRateRangeMake(30, 120, 60);
+ }
+ [_field addAnimation:rotation forKey:key];
+ [_voice addAnimation:position forKey:key];
+}
+
+// The turn on screen now, unwrapped to the one last set.
+- (double)shownTurn {
+ NSNumber *shown = [_field.presentationLayer valueForKeyPath:@"transform.rotation.z"];
+ return shown ? _angle + remainder(shown.doubleValue - _angle, 2 * M_PI) : _angle;
+}
+
+- (void)setTurn:(double)angle {
+ _angle = angle;
+ [CATransaction begin];
+ [CATransaction setDisableActions:YES];
+ [_field setValue:@(angle) forKeyPath:@"transform.rotation.z"];
+ _voice.position = [self voiceAt:angle];
+ [CATransaction commit];
+}
+
+- (void)turnTo:(double)target seconds:(CFTimeInterval)seconds {
+ double from = [self shownTurn], to = from + remainder(target - from, 2 * M_PI);
+ if (UIAccessibilityIsReduceMotionEnabled()) {
+ if (fabs(to - _angle) < kStillStep && seconds <= kFollowSeconds) return;
+ CATransition *fade = [CATransition animation];
+ fade.type = kCATransitionFade;
+ fade.duration = 0.35;
+ [_stage addAnimation:fade forKey:@"fade"];
+ [self setTurn:to];
+ return;
+ }
+ // The voice's ring in steps of about 4 degrees, straight between them.
+ int steps = MAX(1, (int)ceil(fabs(to - from) / 0.07));
+ NSMutableArray<NSNumber *> *turns = [NSMutableArray array];
+ for (int i = 0; i <= steps; i++) {
+ double t = (double)i / steps;
+ // Eased over a long turn, linear over a motion's own short one, so the next carries straight on.
+ if (seconds > kFollowSeconds) t = t * t * (3 - 2 * t);
+ [turns addObject:@(from + (to - from) * t)];
+ }
+ [self setTurn:to];
+ [self animateTurns:turns duration:seconds repeatFrom:0 key:@"turn"];
+}
+
+// The sway: out to one side, then over to the other and back, slowing to a stop at each end. It sets out from
+// wherever the disc is, eased out to the first side.
+- (void)sway {
+ NSMutableArray<NSNumber *> *turns = [NSMutableArray array];
+ for (int i = 0; i <= 96; i++) [turns addObject:@(-kSwayAngle * cos(2 * M_PI * i / 96))];
+ CFTimeInterval lead = 1.2;
+ [self turnTo:-kSwayAngle seconds:lead];
+ [self animateTurns:turns duration:kSwaySeconds repeatFrom:[_field convertTime:CACurrentMediaTime() fromLayer:nil] + lead key:@"sway"];
+}
+
+// Whichever of following, swaying and holding still is wanted now, and the ripples with it.
+- (void)moveOn {
+ if (!_field) return;
+ BOOL still = UIAccessibilityIsReduceMotionEnabled(), moving = self.window && !still;
+ BOOL sway = moving && !_live;
+ if (sway != _swaying) {
+ _swaying = sway;
+ if (sway) [self sway];
+ else {
+ double shown = [self shownTurn];
+ for (CALayer *layer in @[_field, _voice]) [layer removeAnimationForKey:@"sway"];
+ [self setTurn:shown];
+ }
+ }
+ if (still && !_live && fabs(_angle) > 0.001) [self turnTo:0 seconds:0.6];
+ CALayer *ring = _ripples.sublayers.firstObject;
+ if (moving && ![ring animationForKey:@"ripple"]) {
+ // Three rings going out from the voice in turn, slowly, as sound does from where it comes from.
+ CABasicAnimation *grow = [CABasicAnimation animationWithKeyPath:@"transform.scale"];
+ grow.fromValue = @0.18;
+ grow.toValue = @1;
+ CAKeyframeAnimation *fade = [CAKeyframeAnimation animationWithKeyPath:@"opacity"];
+ fade.values = @[@0, @0.55, @0];
+ fade.keyTimes = @[@0, @0.15, @1];
+ CAAnimationGroup *ripple = [CAAnimationGroup animation];
+ ripple.animations = @[grow, fade];
+ ripple.duration = 3.6;
+ ripple.repeatCount = HUGE_VALF;
+ ripple.timingFunction = [CAMediaTimingFunction functionWithControlPoints:0.23 :1 :0.32 :1];
+ ripple.preferredFrameRateRange = CAFrameRateRangeMake(30, 120, 60);
+ [ring addAnimation:ripple forKey:@"ripple"];
+ _ripples.instanceCount = 3;
+ _ripples.instanceDelay = ripple.duration / 3;
+ } else if (!moving) {
+ [ring removeAnimationForKey:@"ripple"];
+ }
+}
+
+#pragma mark - the head
+
+- (void)didMoveToWindow {
+ [super didMoveToWindow];
+ [self refresh];
+}
+
+- (void)listen:(BOOL)listen {
+ if (listen == _listening) return;
+ _listening = listen;
+ if (!listen) {
+ SGHeadMotionListen(kListener, nil);
+ [self followed:NAN];
+ return;
+ }
+ __weak typeof(self) weakSelf = self;
+ // A front of the preview's own, from the first motion it hears: the audio's has followed the head for as long
+ // as the song has played, and the two agree once the head has been still a while.
+ __block SGSpatialFront front = {0};
+ SGHeadMotionListen(kListener, ^(CMDeviceMotion *motion) {
+ double angle = motion ? SGSpatialVoiceAngle(&front, motion.attitude.yaw, motion.timestamp) : NAN;
+ if (!motion) front.hasFront = false;
+ dispatch_async(dispatch_get_main_queue(), ^{ [weakSelf followed:angle]; });
+ });
+}
+
+// The voice's angle from the head's motion, NAN when the motion stopped.
+- (void)followed:(double)angle {
+ BOOL live = _listening && !isnan(angle);
+ if (live != _live) {
+ _live = live;
+ [self updateCaption];
+ [self moveOn];
+ if (live) {
+ [self turnTo:angle seconds:0.5];
+ return;
+ }
+ }
+ if (live) [self turnTo:angle seconds:kFollowSeconds];
+}
+
+- (void)refresh {
+ _allowed = CMHeadphoneMotionManager.authorizationStatus;
+ BOOL front = UIApplication.sharedApplication.applicationState != UIApplicationStateBackground;
+ [self listen:self.window && front && _allowed == CMAuthorizationStatusAuthorized];
+ [self updateCaption];
+ [self moveOn];
+}
+
+- (void)updateCaption {
+ NSString *text;
+ if (_live) {
+ text = SGSingSpatial() ? @"Following your head. Turn it, and the voice keeps its place in the room while the rest of the song turns "
+ @"with you; stay turned, and it comes round in front again."
+ : @"Following your head. Turn Spatial voice on to hear the voice keep its place as you turn.";
+ } else if (_allowed == CMAuthorizationStatusDenied || _allowed == CMAuthorizationStatusRestricted) {
+ text = @"Motion & Fitness is off for Spotify, so the voice stays ahead. Settings > Privacy & Security > Motion & Fitness "
+ @"turns it on.";
+ } else {
+ text = @"Preview. With AirPods that track your head, it follows yours: turn, and the voice keeps its place in the room "
+ @"while the rest of the song turns with you.";
+ }
+ if ([text isEqualToString:_caption.text]) return;
+ if (_caption.text) {
+ [UIView transitionWithView:_caption duration:0.25 options:UIViewAnimationOptionTransitionCrossDissolve
+ animations:^{ self->_caption.text = text; } completion:nil];
+ } else {
+ _caption.text = text;
+ }
+ // A caption of another length makes the header another height, which the page reads as it lays out.
+ [self.superview setNeedsLayout];
+}
+
+- (NSString *)accessibilityLabel {
+ return @"Spatial voice preview: you in the middle of a field of dots, the voice a point in front of you";
+}
+
+- (NSString *)accessibilityValue {
+ if (!_live) return _caption.text;
+ long degrees = lround(fabs(remainder(_angle, 2 * M_PI)) * 180 / M_PI);
+ NSString *where = degrees < 3 ? @"The voice is in front of you" : [NSString stringWithFormat:@"The voice is %ld degrees to your %@",
+ degrees, remainder(_angle, 2 * M_PI) > 0 ? @"right" : @"left"];
+ return [NSString stringWithFormat:@"%@. %@", where, _caption.text];
+}
+
+@end
diff --git a/tweak/Sources/Shared/Sing/Sing.h b/tweak/Sources/Shared/Sing/Sing.h
index 5890a632..fefabc56 100644
--- a/tweak/Sources/Shared/Sing/Sing.h
+++ b/tweak/Sources/Shared/Sing/Sing.h
@@ -4,10 +4,14 @@
//
// SGSingModel.m the voice model: downloaded, each file checked against its size and SHA-256, loaded
// SGSingSeparator.m the STFT around the model: two seconds in, their vocals out
-// SGSingEngine.m Spotify's mixer pulled ahead of what plays, the vocals separated in the lead and mixed down
+// SGSingEngine.m Spotify's mixer pulled ahead of what plays, the vocals separated in the lead, mixed down
+// and put where spatial voice holds them
// Sing.x the engine put between Spotify's mixer and its output (through Shared/Player/SpeedPitch.x's
-// chain), Spotify's clock and seeks kept true to what plays, the heat, and what Sing is doing
-// SingSettings.m the Sing page
+// chain), Spotify's clock and seeks kept true to what plays, the heat, what Sing is doing,
+// and the head's turn for spatial voice (Shared/HeadGestures' motion)
+// SingSettings.m the Sing page, and Spatial voice's under it
+// SGSpatialPreview.m the Spatial voice page's preview: the voice in front of the listener on a field of dots,
+// turned by the head the way the audio is
//
// The voice model is public and MIT licensed: Mel-Band RoFormer (Ju-Chiang Wang, Wei-Tsung Lu, Minz Won) with
// KimberleyJensen's vocal checkpoint, its spectral core exported for Core ML with two-second windows, from
@@ -25,6 +29,9 @@
#define SGKeySingIgnoreHeat @"spotifyglass.sing.ignoreHeat"
// Where the model runs: an index into SGSingComputeUnitNames(), read as it loads.
#define SGKeySingComputeUnits @"spotifyglass.sing.computeUnits"
+// Spatial voice: through headphones that track the head, the vocals stay in front as it turns; off until
+// switched on, and at once.
+#define SGKeySingSpatial @"spotifyglass.sing.spatial"
// Posted on the main thread whenever what Sing is doing, or its model's download, changes.
extern NSString *const SGSingChangedNotification;
@@ -57,6 +64,19 @@ void SGSetSingLevel(float level);
// A level as the slider reads it out: Gone, a percentage, As sung, Backing and a percentage, Vocals only.
NSString *SGSingLevelText(double level);
void SGSetSingIgnoresHeat(BOOL ignores);
+// Whether this iPhone runs Sing and reads headphone motion, which spatial voice needs.
+BOOL SGSingSpatialAvailable(void);
+BOOL SGSingSpatial(void);
+// Stored, the motion's permission asked for when it goes on, and applied at once.
+void SGSetSingSpatial(BOOL on);
+
+// The top of the Spatial voice page: listener and voice on a field of dots seen from behind the head, which
+// follows the head through HeadGestures' motion while it is on screen and the motion is allowed, and idles
+// otherwise, a line under it saying which.
+@interface SGSpatialPreview : UIView
+// Reads the switch and the motion's permission again.
+- (void)refresh;
+@end
// Loads the model again on the compute units now stored, if it is loaded.
void SGSingComputeUnitsChanged(void);
// What the main page's Sing row reads out: On, Off, or how far the model has come.
diff --git a/tweak/Sources/Shared/Sing/Sing.x b/tweak/Sources/Shared/Sing/Sing.x
index 7a75d1b4..560803b8 100644
--- a/tweak/Sources/Shared/Sing/Sing.x
+++ b/tweak/Sources/Shared/Sing/Sing.x
@@ -20,13 +20,22 @@
// The model is held only while the mic is on, and loaded on a queue of its own. From the thermal state
// Serious on, the engine is held and plays dry, unless Ignore heat warnings is on.
//
-// Threading: the stage on the render thread; Spotify starts its output on a thread of its own; the rest main.
+// Spatial voice listens to Shared/HeadGestures' motion (the app's one CMHeadphoneMotionManager) while Sing is
+// on and Spotify plays, and hands the engine the head's yaw off a front that follows where the head points over
+// 20 s, so the voice drifts back ahead of a head that stays turned and the attitude's own drift never carries
+// it off. With no motion (no such headphones, Motion & Fitness not allowed) the voice stays ahead. iOS's own
+// spatial audio on the route already holds the whole song in place, so spatial voice stands down for it.
+//
+// Threading: the stage on the render thread; Spotify starts its output on a thread of its own; the head's
+// motion on HeadGestures' queue; the rest main.
#import <AudioToolbox/AudioToolbox.h>
+#import <AVFoundation/AVFoundation.h>
#import <CoreML/CoreML.h>
#import <pthread.h>
#import <stdatomic.h>
#import "Core/SGCore.h"
#import "Core/SGRebind.h"
+#import "Shared/HeadGestures/HeadGestures.h"
#import "Shared/Player/PlayerState.h"
#import "Shared/Player/SpeedPitch.h"
#import "Sing.h"
@@ -166,6 +175,58 @@ static void unloadModel(void) {
if (engine) SGSingEngineSetSeparator(engine, nil);
}
+#pragma mark - spatial voice
+
+static BOOL sg_spatialListening;
+// On HeadGestures' motion queue only.
+static SGSpatialFront sg_front;
+
+static void headMoved(CMDeviceMotion *motion) {
+ SGSingEngine *engine = atomic_load_explicit(&sg_engine, memory_order_acquire);
+ if (!motion) {
+ sg_front.hasFront = false;
+ if (engine) SGSingEngineSetVoiceAngle(engine, 0);
+ return;
+ }
+ double angle = SGSpatialVoiceAngle(&sg_front, motion.attitude.yaw, motion.timestamp);
+ if (engine) SGSingEngineSetVoiceAngle(engine, (float)angle);
+}
+
+// iOS's own spatial audio (Spatialize Stereo, or a spatial headphone's own) on the route.
+static BOOL systemSpatial(void) {
+ for (AVAudioSessionPortDescription *port in AVAudioSession.sharedInstance.currentRoute.outputs) {
+ if (port.spatialAudioEnabled) return YES;
+ }
+ return NO;
+}
+
+// Listens to the head while Sing is on, spatial voice is on and Spotify plays, and iOS does not hold the song in
+// place itself; otherwise the voice goes back ahead (HeadGestures hands a listener taken out nil).
+static void updateSpatial(void) {
+ SPTPlayerState *player = SGPlayerState();
+ BOOL wanted = SGSingOn() && !SGSingMissing() && SGSingSpatial() && player.isPlaying && !player.isPaused;
+ BOOL listen = wanted && !systemSpatial();
+ if (listen == sg_spatialListening) return;
+ sg_spatialListening = listen;
+ SGLog(@"sing: spatial voice %@", listen ? @"follows the head" : wanted ? @"stands down for iOS's own spatial audio" : @"is ahead");
+ SGHeadMotionListen(@"sing", listen ? ^(CMDeviceMotion *motion) { headMoved(motion); } : nil);
+}
+
+BOOL SGSingSpatialAvailable(void) {
+ return SGSingOSSupported() && SGSingDeviceSupported() && SGHeadGesturesAvailable();
+}
+
+BOOL SGSingSpatial(void) {
+ return SGHidden(SGKeySingSpatial);
+}
+
+void SGSetSingSpatial(BOOL on) {
+ SGSetEnabled(SGKeySingSpatial, on);
+ // Asked while the Sing page is in front, rather than with the next song.
+ if (on) SGHeadMotionAskPermission();
+ updateSpatial();
+}
+
#pragma mark - the state
static void watch(void);
@@ -187,6 +248,7 @@ static void apply(void) {
SGSingEngineSetOn(engine, on);
if (on) loadModel();
else unloadModel();
+ updateSpatial();
watch();
}
@@ -364,8 +426,9 @@ static void tick(void) {
if (engine && SGSingOn() && CFAbsoluteTimeGetCurrent() - summarized >= 30) {
summarized = CFAbsoluteTimeGetCurrent();
SGSingEngineStats stats = SGSingEngineReadStats(engine);
- SGLog(@"sing: %@, %llu windows at %.0f ms each (1500 ms keeps up), lead %.2f s of %.2f s, %.2f s separated ahead, %llu frames dry, %llu failures",
- SGSingStatusText(), stats.windows, stats.averageMS, stats.lead, stats.targetLead, stats.ready, stats.dryFrames, stats.failures);
+ SGLog(@"sing: %@, %llu windows at %.0f ms each (1500 ms keeps up), lead %.2f s of %.2f s, %.2f s separated ahead, %llu frames dry, %llu failures%@",
+ SGSingStatusText(), stats.windows, stats.averageMS, stats.lead, stats.targetLead, stats.ready, stats.dryFrames, stats.failures,
+ sg_spatialListening ? [NSString stringWithFormat:@", the voice %.0f degrees right", stats.voiceAngle * 180 / M_PI] : @"");
}
// Off, with the lead given back: nothing left to watch.
if (!SGSingOn() && (!engine || SGSingEngineLead(engine) <= 0)) {
@@ -386,6 +449,7 @@ static void watch(void) {
@implementation SGSingTrackWatcher
// A new track well before the last one's end was played, not reached: the lead is the old track's.
- (void)playerStateDidChange:(SPTPlayerState *)state {
+ updateSpatial();
NSString *track = SGURIString(state.track.URI);
if (track && sg_lastTrack && ![track isEqualToString:sg_lastTrack] && sg_lastRaw >= 0 && sg_lastDuration > 0
&& sg_lastDuration - sg_lastRaw > kEndSlack + kWatchEvery && CFAbsoluteTimeGetCurrent() - sg_lastFlush > 1) {
@@ -468,6 +532,11 @@ static void readHeat(void) {
SGAddPlayerStateObserver(watcher);
[NSNotificationCenter.defaultCenter addObserverForName:NSProcessInfoThermalStateDidChangeNotification object:nil queue:NSOperationQueue.mainQueue
usingBlock:^(NSNotification *note) { readHeat(); }];
+ // Headphones in or out, or iOS's own spatial audio switched.
+ for (NSNotificationName name in @[AVAudioSessionRouteChangeNotification, AVAudioSessionSpatialPlaybackCapabilitiesChangedNotification]) {
+ [NSNotificationCenter.defaultCenter addObserverForName:name object:nil queue:NSOperationQueue.mainQueue
+ usingBlock:^(NSNotification *note) { updateSpatial(); }];
+ }
[NSNotificationCenter.defaultCenter addObserverForName:SGSingChangedNotification object:nil queue:nil usingBlock:^(NSNotification *note) {
// A download finishing with the mic already on loads the model.
if (SGSingOn() && !sg_separator && !sg_loading && !sg_loadError) apply();
diff --git a/tweak/Sources/Shared/Sing/SingSettings.m b/tweak/Sources/Shared/Sing/SingSettings.m
index 95184060..d2de5c40 100644
--- a/tweak/Sources/Shared/Sing/SingSettings.m
+++ b/tweak/Sources/Shared/Sing/SingSettings.m
@@ -1,5 +1,6 @@
// The Sing page (Sing on Mod Settings' main page, and Lyrics > Karaoke), under either look: Sing's switch, which
-// turns the mic on and off at once, what Sing is doing, the vocals' level, and the voice model.
+// turns the mic on and off at once, what Sing is doing, the vocals' level, spatial voice's page (its preview,
+// SGSpatialPreview.m, and its switch), and the voice model.
#import "Core/SGCore.h"
#import "Settings/SGModPage.h"
#import "Settings/SGPageStyle.h"
@@ -65,6 +66,50 @@ static void modelTapped(UITableViewController *page) {
}
}
+// Spatial voice's page, under Sing's: the preview at its top and the switch.
+@interface SGSpatialVoicePage : SGModPage
+@property (nonatomic, readonly) SGSpatialPreview *preview;
+@end
+
+@implementation SGSpatialVoicePage
+
+- (void)viewDidLoad {
+ [super viewDidLoad];
+ _preview = [[SGSpatialPreview alloc] initWithFrame:CGRectMake(0, 0, self.tableView.bounds.size.width, 0)];
+ self.tableView.tableHeaderView = _preview;
+}
+
+// Its caption follows Dynamic Type and what the preview is doing, so the header is sized to it, and handed back
+// only when that size changes (SGFitNote's way: the table lays out again on every handing back).
+- (void)viewWillLayoutSubviews {
+ [super viewWillLayoutSubviews];
+ UITableView *table = self.tableView;
+ CGSize size = [_preview sizeThatFits:CGSizeMake(table.bounds.size.width, 0)];
+ if (CGSizeEqualToSize(_preview.bounds.size, size)) return;
+ _preview.frame = (CGRect){_preview.frame.origin, size};
+ table.tableHeaderView = _preview;
+}
+
+@end
+
+static UIViewController *spatialPage(void) {
+ SGModRow *spatial = SGOptionRow(@"Spatial voice", nil, SGKeySingSpatial);
+ __block __weak SGSpatialVoicePage *page;
+ spatial.changed = ^(BOOL on) {
+ SGSetSingSpatial(on);
+ [page.preview refresh];
+ };
+ SGSpatialVoicePage *made = [[SGSpatialVoicePage alloc] initWithTitle:@"Spatial voice" intro:nil sections:@[
+ SGNotedSection(nil, @[spatial],
+ @"Sing's voice keeps its place in front of you as you turn your head, while the rest of the song turns with "
+ @"you; stay turned, and it comes round in front again. It follows your head through headphones that "
+ @"track it: AirPods Pro, AirPods 3 or later, AirPods Max, and asks for Motion & Fitness the first time. "
+ @"While iOS's own Spatialize Stereo is on, which already holds the whole song in place, the voice is left to it."),
+ ] footer:nil];
+ page = made;
+ return made;
+}
+
UIViewController *SGSingSettingsPage(void) {
SGModRow *sing = SGOptionRow(@"Sing", @"Turns the vocals down as the song plays, like the mic on the lyrics", SGKeySing);
sing.changed = ^(BOOL on) { SGSetSingOn(on); };
@@ -74,6 +119,11 @@ UIViewController *SGSingSettingsPage(void) {
});
SGModRow *level = SGSliderRow(@"Vocals", @"From gone, through the song as sung, to the vocals alone", 0, 2, 0.05,
^double { return SGSingLevel(); }, ^(double value) { SGSetSingLevel((float)value); }, ^NSString *(double value) { return SGSingLevelText(value); });
+ SGModRow *spatial = SGPageRow(@"Spatial voice", ^UIViewController *{ return spatialPage(); });
+ spatial.subtitle = @"With AirPods, the voice stays in front of you as you turn your head";
+ spatial.value = ^NSString *{ return SGSingSpatial() ? @"On" : @"Off"; };
+ // Facts about the iPhone, which do not change while the page shows.
+ spatial.visible = ^BOOL { return SGSingSpatialAvailable(); };
__block __weak SGModPage *page;
SGModRow *model = SGStatActionRow(@"Voice model", @"Mel-Band RoFormer, run on the iPhone", ^NSString *{ return modelValue(); }, ^{ modelTapped(page); });
SGModRow *units = SGChoiceRow(@"Runs on", @"Read as the model loads", SGKeySingComputeUnits, SGSingComputeUnitNames(), 0);
@@ -81,7 +131,7 @@ UIViewController *SGSingSettingsPage(void) {
SGModRow *heat = SGOptionRow(@"Ignore heat warnings", @"Keeps Sing going on a hot iPhone, which then gets hotter", SGKeySingIgnoreHeat);
heat.changed = ^(BOOL on) { SGSetSingIgnoresHeat(on); };
SGModPage *made = [[SGModPage alloc] initWithTitle:@"Sing" intro:nil sections:@[
- SGSection(nil, @[SGWithSymbol(sing, @"music.mic"), status, level]),
+ SGSection(nil, @[SGWithSymbol(sing, @"music.mic"), status, level, spatial]),
SGNotedSection(@"Voice model", @[model, units, heat],
@"Sing listens a few seconds ahead of what plays and separates the vocals there, so a song takes a few seconds to "
@"turn its vocals down after it starts or after a seek. Everything runs on the iPhone; no audio leaves it."),