Liquid Glass and Apple Music touches for the Spotify iOS app. A GPL-3.0 fork of spoti.pw 0.21.1.
C
feat: the Karaoke page shows the song, its state and live vocal levels
My-Name-Is-Jeff3010e922b7f9
diff --git a/docs/tweaks.md b/docs/tweaks.mdindex 5aa808f7..4df9d942 100644--- a/docs/tweaks.md+++ b/docs/tweaks.md@@ -224,12 +224,23 @@ Shared:Tested on the Mac against harness/audio-effects/ and harness/autoeq/, the hook in the simulatoragainst its sim/, the page against harness/audio-effects-page/Sing/ Sing, a song's vocals turned down while it plays (Sing.h lists its files): the voice model (Mel-Band- RoFormer, MIT, from Hugging Face) downloaded and checked file by file, a stop keeping what came in for the- next download to carry on from, the STFT around it on Accelerate,+ RoFormer, MIT, from Hugging Face) downloaded and checked file by file over Wi-Fi unless cellular is allowed,+ a stop keeping what came in for the next download to carry on from; loaded only while Spotify is active+ (SGSingLoader.m), a CPU copy first and warmed, then a faster one beside it for the foreground, each load+ with a deadline and kept a minute after the mic goes off; the STFT around it on Accelerate,and an engine that stands in Speed and pitch's chain between Spotify's mixer and its output(SGPlayerSetStage), pulls the mixer a few seconds ahead of what plays, separates two-second windowsthere on a worker thread and mixes the vocals down on the render thread. Spotify's clock has the lead- taken off (SPTPlayerState's positionAsOfTimestamp), and a seek, a skip or a stop drops it. Spatial+ taken off (SPTPlayerState's positionAsOfTimestamp): the lead held when that line of the clock began+ (the same position run on at the same speed, however often a state is stamped again), less what was+ dropped since, as the player reports on a change and not as it plays (SGSingLeadOf, read by every+ correction; every 5 s a "sing: clock:" log line sets Spotify's clock against what the engine pulled+ and played, and every lyrics seek logs where it was sent and where it landed), and a seek, a skip or a stop drops it. Not separating (stopped, held, resting at As sung,+ standing aside), the engine plays the lead it holds on as it is, dry, the clock still corrected, so no+ part of the song is skipped; it lets it go at a pause, where Sing.x seeks Spotify back to what was+ heard, or with a seek, a skip or the output stopping, and builds one only when it separates. From the+ thermal state Serious up Sing is held and lets the model go,+ unless Ignore heat warnings is on, and runs again at Fair. Spatialvoice holds the separated vocals in front as the head turns: HeadGestures' motion gives the yaw off afront that follows the head over 20 s, and the render thread pans the vocals' middle at equal power,narrowed, with the far ear up to 0.65 ms late and low-passed; it stands down for iOS's own spatial@@ -514,11 +525,14 @@ lyrics for every track, naming the source in the redesign, the lock screen, andthe redesign also which of the lyrics, their pronunciation and their translation is set largest, Romanised lyrics(applied at once), and the translation's language. Sing, under either look, on the main page and as Lyrics' first row, the row reading out On,Off or how far the voice model's download has come, kept up to date while the page shows: Sing's switch, which turns-the mic on and off at once, its Status (a tap says more), the Vocals slider (gone, as sung, the vocals alone), Spatial voice+the mic on and off at once, a card at its top (SGSingCard.m: the song, what Sing is doing, a tap saying more, the vocals+and the rest traced live from the engine's loudness, still under Reduce Motion, play and pause, and a tall Vocals slider+from gone through as sung to the vocals alone, with Sing along, Original and Vocals only under it), Spatial voice(where the iPhone reads headphone motion; a page of its own, reading out On or Off, with a live preview at its top thatfollows the head through AirPods, or sways gently without them and holds still under Reduce Motion, a line under it-saying which, then the switch; its row is on the main page too, under Sing's), the voice model's download, Runs on (GPU, GPU-and Neural Engine, Neural Engine) and Ignore heat warnings, all applying straight away. Lock screen, on the main+saying which, then the switch; its row is on the main page too, under Sing's), the voice model's download (Paused and+Checking among its states) and its removal, Ignore heat warnings, and under Advanced Runs on (Automatic, CPU only, GPU,+Neural Engine, GPU and Neural Engine: the faster copy's), all applying straight away. Lock screen, on the mainpage under either look, opens the lock screen widget's page, titled Lock screen (Moving artwork, Lyrics or Every song, and the lyrics' style,Still or Animated, and Spotify's like and dislike buttons' flag; its podcast, audiobook and artwork flags stay inAll flags). Player: Gestures and Blocked artists (with the count on the row), which work with either look;diff --git a/harness/player/stubs.m b/harness/player/stubs.mindex 30543610..94f7d7e8 100644--- a/harness/player/stubs.m+++ b/harness/player/stubs.m@@ -199,6 +199,7 @@ float SGSingLevel(void) { return 0; }void SGSetSingLevel(float level) {}NSString *SGSingLevelText(double level) { return @""; }double SGSingModelProgress(void) { return 0; }+BOOL SGSingModelWaitingForNetwork(void) { return NO; }NSString *SGSingModelError(void) { return nil; }NSString *SGSingModelSizeText(void) { return @""; }void SGSingDownloadModel(void) {}diff --git a/harness/sing/README.md b/harness/sing/README.mdindex 7f37631e..e49802e2 100644--- a/harness/sing/README.md+++ b/harness/sing/README.md@@ -4,20 +4,29 @@ Three parts: the separator and the engine on the Mac, Sing end to end in the sim## The separator and the engine on the Mac (`main.m`)-Sing's separator and engine (`tweak/Sources/Shared/Sing/SGSingSeparator.m`, `SGSingEngine.m`) as the tweak-compiles them, on the Mac, over the voice model as the app downloads it (a `separator.mlmodelc` folder of the five+Sing's loader, separator and engine (`tweak/Sources/Shared/Sing/SGSingLoader.m`, `SGSingSeparator.m`,+`SGSingEngine.m`) as the tweak compiles them, their log lines printed through `shim/Core/SGLog.h`, on the Mac, over the voice model as the app downloads it (a `separator.mlmodelc` folder of the fivefiles from https://huggingface.co/My-Name-Is-Jeff/vitrine-sing, a mirror of Darkkos/spoti-sing, MIT). Each check prints a line:- the STFT: a cosine's peak bin at torch.stft's size (the amplitude times the Hann window's sum over two), noisethrough the STFT and back unchanged, its edges included;+- the loader: a load past a deadline of 0.2 s abandoned and Sing Failed, Failed kept until the mic goes off and on,+ then a fresh load beside the abandoned one with a second want joining it, the CPU copy loaded and warmed with a+ window of silence, the abandoned load let go when it comes back (the separator stays the fresh one's), the faster+ copy (the units asked for; `cpu` for none) loaded beside it, the copies kept over a quick off and on and dropped+ after the time kept, and a load purged on its way let go;- the model: its input and output shapes, the plan Core ML makes for the compute units asked for (how manyoperations go to the CPU, the GPU and the Neural Engine), the time a two second window takes, and a voice mixedover synthetic chords taken apart: the vocals found scored against the voice, beside the mix's own score;- the engine: the mix pulled through `SGSingEngineRender` in buffers of 1024 on a thread that keeps real time,with Sing on from the start: the mix plays dry while the lead fills, then exactly the mix less the vocals the- offline pass finds, the lead given back after Sing is switched off and the mix playing on where it was, the+ offline pass finds, the lead kept after Sing is switched off and the mix playing on where it was, nothing skipped, thetop of the slider playing the vocals alone, a flush dropping the lead, and no allocation on the render thread(`malloc_logger`);+- the lead, without the model: the lead reported is the frames pulled and not played, to the frame; held (the heat,+ resting), given up or switched off, it is kept and played on as it is, every frame in order and none built; a flush+ (a pause, a seek) drops it at once and the next frame pulled plays next; it is built again when it separates; and+ the lead to take off a position Spotify counted at a moment is the lead held then, less what was dropped since;- spatial voice, without the model (a separator that hands each window back whole as vocals, so what plays is thevocals placed): a 440 Hz tone at level 1 plays exactly as it came straight ahead; set 90 degrees right, the rightear has it 7.75 dB louder (the narrowed pan's 7.66 and the far ear's low-pass) at the same power, and the left ear@@ -33,7 +42,7 @@ second apart over 30 seconds../build.sh && build/sing <separator.mlmodelc> <voice> <out dir> [all|cpu|gpu|ane]./build.sh thread && build/sing-thread <separator.mlmodelc> <voice> <out dir> gpu (ThreadSanitizer)- ./build.sh && build/sing spatial (spatial voice alone)+ ./build.sh && build/sing spatial (the lead, spatial voice and falling behind)The out dir gets `mix.wav`, `vocals.wav`, `accompaniment.wav`, `engine.wav` (what played) and `spatial.wav` (thetone turned; the temporary folder's with `spatial` alone).@@ -56,11 +65,36 @@ The model card keeps normalization, attention, softmax and the matrix products iEngine does not run, so splitting the graph between it and the GPU costs more than it saves on the Mac; Sing asksfor the CPU and GPU unless the Sing page's Runs on says otherwise (`SGKeySingComputeUnits`). The vocals found score 28.1 dB against the voice where the mix scores7.6 dB, and the mix less them 20.5 dB against the chords where the mix scores -7.6 dB. In the engine the lead-settles at 2.84 s (a window plus 1.5 times the model's time and a quarter second).+settled at 2.84 s (a window plus 1.5 times the model's time and a quarter second); since 2026-10-06 the spare is+0.75 s, for the hysteresis, and the lead settles at 3.4-3.7 s.++2026-10-06, the runtime (a CPU copy first, a faster one beside it): everything passes under `gpu`, and the model-free+checks under ThreadSanitizer, the new checks with them: the loader's cases above; each window on the faster copy in+the foreground and on the CPU's in the background; a window the faster copy fails done again on the CPU's, the faster+copy resting until the app has been in the background; a model that keeps up and then falls behind fading the vocals+out once and giving up after 8 s short of them; a NaN from the model playing as silence; the loudness the Sing page+draws. The Mac was shared with other work and its load average swung between 6 and 830 through the day, so the times+below were each taken when it was low (under 30) and are a guide, not a benchmark:++| copy | load (cold, then from Core ML's cache) | warm-up window of silence | a window, warm |+|---|---|---|---|+| CPU only | 0.5-5.2 s, then 0.0 s | 0.5-2.3 s | 470-480 ms (the first after the copy sat idle: 1.7-3.7 s) |+| GPU (CPU and GPU) | 0.4-3.7 s | 0.6-1.2 s | 450-760 ms |+| GPU and Neural Engine | 0.2-0.7 s | 1.3-1.4 s | 1260-1600 ms |+| Neural Engine (CPU and Neural Engine) | 139.5 s, then 20.5 s | 6.3-18.0 s | 4000-6800 ms |++Footprint, from a probe that loads copies one after the other and keeps them: one copy 1.40-1.50 GB, a second copy+1.07-1.20 GB more, whatever the units. So the loader wants 1.6 GB left to the process before a second copy. A window+of silence took as long as a window of noise.Not measured: the iPhone. Its GPU is slower than the M4's, and a window must take under 1.5 s (the hop) for thevocals to keep up; when they do not, what plays is dry until they catch up.+2026-10-06, the lead kept while not separating and taken off by the report's moment: everything passes under `gpu`,+and `spatial` under ThreadSanitizer. The lead was off by 0 frames from what was pulled and not played; held from 8 s+to 12 s it stayed at 2.75 s with every frame played in order; a flush read 0 at once, and a position counted at 1 s+then had -1.73 s taken off (the 1.02 s held then less the 2.75 s dropped).+## Sing end to end in the simulator (`sim/main.m`)The voice model downloaded from Hugging Face by `SGSingModel.m` and checked file by file, loaded by `Sing.x`, and@@ -69,13 +103,24 @@ MakeConnection) with `SpeedPitch.x` and `Sing.x` compiled in (logos, internal geexecutable, so the rebound imports take as in Spotify and Sing's stage runs in Speed and pitch's chain. Thedecoder hands over the voice over the chords; a mock SPTPlayerState computes -position the way Spotify's doesand a mock SPTEsperantoPlayer moves the decoder on seekTo:. A render notify on RemoteIO records what plays.-The script turns Sing on at the vocals' 0, plays, seeks to 5 s at 20 s, and turns Sing off at 30 s; it checks the-download, the model loading, the position Spotify shows against what is heard, the seek's result handed back and-the lead dropped, the vocals down again after it, the lead given back, and what played against the chords alone.+The mock player keeps one of three clocks, chosen with `-clock` at launch: `report` (the default) reports when playing+starts, on a seek and on a pause, and its last report runs on by the time; `restamp` hands the lyrics that report+stamped again at every read of the player's state, the same line of the clock; `decoded` reports again every half+second with what the decoder has handed over. The lyrics' position is read from the player's state as the lyrics+read it. The thermal state is the script's. The script turns Sing on at the vocals' 0, plays, taps the lyrics line+at 5 s at 20 s (the lyrics' seek, `SGKaraokeSeek(5000)`, with a lead held), rests at As sung at 27 s and separates+again at 29 s, goes to the thermal state Serious at 31 s and back to Fair at 32 s, stops Sing with a memory warning,+pauses and resumes, and turns it off; it checks the download, the model loading, the lead Sing reports against what+the decoder handed over less what was heard, the position the lyrics read against what is heard at each step (as the+lead fills after a report, a second after the tap, after the seek, resting, held, stopped, paused, resumed), the tap+sending Spotify to the line's 5 s and what is heard a second later being 5 s run on, the lead dropped by the seek,+the lead kept resting, at Serious and stopped, Spotify seeked back to what was heard at the pause with the lead let+go, the song resuming there with nothing skipped, and what played against the chords alone. A pass says Sing.x is+right for that mock's clock, not which clock the iPhone's Spotify keeps: the `sing: clock:` log line says that.THEOS=$HOME/theos VOICE=speech.aiff ./build-sim.shxcrun simctl install <udid> build/sim/SingHarness.app- xcrun simctl launch --console-pty <udid> com.vojta.singharness+ xcrun simctl launch --console-pty <udid> com.vojta.singharness [-clock report|restamp|decoded]2026-10-05, iPhone 17 Pro simulator, iOS 27: the four small files came in from Hugging Face and were checked andkept; the weights reached 52% before the run was stopped, and were then put in the staging folder from a copy@@ -86,6 +131,26 @@ the 6 s held ahead; switched off, the decoder is drained at 0.5x until the leadCore ML on the CPU, 15 s a window, so Sing reads Too slow there and the three vocals checks are skipped: the Macharness is where they pass.+2026-10-06, iPhone 17 Pro simulator, iOS 27, with the runtime and Shared/Player's music output: the model put in the+app's Application Support beforehand, the CPU copy loads in 4.9 s and warms in 7.2 s, Sing reads Ready 12.7 s after+the mic, and the GPU copy loads beside it (2.8 s, warmed in 14.7 s: the simulator runs Core ML on the CPU). Sing reads+the music's output through SGPlayerWatchMusicOutput (44100 Hz, 2 channels, takes it). The position, the seek and the+lead given back pass; the vocals checks are skipped as before (16 s a window).++2026-10-06, iPhone 17 Pro simulator, iOS 27, reports only on a change: everything passes. At 14 s the position shows+14.06 s against 14.05 s heard, where the lead taken off as it is now would have shown 9.81 s; Sing reads As sung,+Held, too hot and Stopped with the 4.25 s it held played on, the position matching what is heard within 0.01 s; at+Fair the model loads again in 10.3 s with the lead kept; paused, Spotify is seeked back from 32.61 s to 28.38 s (heard:+28.36 s) and the lead let go, and 2.07 s after resuming 30.45 s is heard. The vocals checks are skipped as before.++2026-10-06, the three clocks, iPhone 17 Pro simulator, iOS 27: everything passes under each (26 checks; the vocals+checks skipped as before). The tap at 20 s with 4.25 s held sends Spotify to 5.00 s and a second later 5.98 s is heard+and read. Under `report`, a run with the Mac's load average near 60 read the position 0.17 s off what was heard at+three steps, past the 0.15 s allowed, and passed again at a load of 30: the harness measures what is heard by the+decoder, whose converter pulls ahead by what the load lets it. A line let run on across Sing's own+flushes read the resume after the pause 4.25 s ahead of what was heard (the seek back runs on from the line before it),+so a report after one of Sing's seeks, skips or seeks back always starts a line.+## The download on the Mac (`download/`)`SGSingModel.m` as the tweak compiles it, built for Mac Catalyst (Sing.h imports UIKit) with `download/shim` standingdiff --git a/harness/sing/build-sim.sh b/harness/sing/build-sim.shindex d1a4922a..b0d5c5bf 100755--- a/harness/sing/build-sim.sh+++ b/harness/sing/build-sim.sh@@ -17,9 +17,9 @@ SDK=$(xcrun --sdk iphonesimulator --show-sdk-path)xcrun -sdk iphonesimulator clang -target arm64-apple-ios18.0-simulator -fobjc-arc -g -O1 -isysroot "$SDK" -Wno-deprecated-declarations \-I"$SRC" -I"$SING" -I"$SRC/Shared/Player" \sim/main.m ../scene.m "$OUT/gen/SpeedPitch.m" "$OUT/gen/Sing.m" "$SRC/Shared/Player/SGTimePitch.m" \- "$SING/SGSingModel.m" "$SING/SGSingSeparator.m" "$SING/SGSingEngine.m" \+ "$SING/SGSingModel.m" "$SING/SGSingLoader.m" "$SING/SGSingSeparator.m" "$SING/SGSingEngine.m" \"$SRC/Core/SGRebind.m" "$SRC/Core/SGLog.m" "$SRC/Core/SGPrefs.m" "$SRC/Core/SGUIMode.m" "$SRC/Core/SGFlagForce.m" \- -framework UIKit -framework QuartzCore -framework AudioToolbox -framework AVFoundation -framework Accelerate -framework CoreML -framework Foundation \+ -framework UIKit -framework QuartzCore -framework MediaPlayer -framework AudioToolbox -framework AVFoundation -framework Accelerate -framework CoreML -framework Foundation \-o "$OUT/SingHarness.app/SingHarness"if [ -n "$VOICE" ]; then cp "$VOICE" "$OUT/SingHarness.app/voice.aiff"; ficat > "$OUT/SingHarness.app/Info.plist" <<'PLIST'diff --git a/harness/sing/build.sh b/harness/sing/build.shindex e9243e35..d5fef260 100755--- a/harness/sing/build.sh+++ b/harness/sing/build.sh@@ -1,6 +1,6 @@#!/bin/sh-# Builds the Sing harness for the Mac: the separator and the engine (tweak/Sources/Shared/Sing) as the tweak-# compiles them, and main.m.+# Builds the Sing harness for the Mac: the loader, the separator and the engine (tweak/Sources/Shared/Sing) as the+# tweak compiles them, and main.m, with shim/Core/SGLog.h printing their log lines.# ./build.sh build/sing# ./build.sh thread build/sing-thread, under ThreadSanitizerset -e@@ -10,6 +10,6 @@ SANITIZE=$1OUT=build/sing${SANITIZE:+-$SANITIZE}mkdir -p buildxcrun clang -fobjc-arc -O2 -g -Wall -Werror -target arm64-apple-macos15.0 ${SANITIZE:+-fsanitize=$SANITIZE -O1 -fno-omit-frame-pointer} \- -I ../../tweak/Sources main.m "$SING"/SGSingSeparator.m "$SING"/SGSingEngine.m \+ -I shim -I ../../tweak/Sources main.m "$SING"/SGSingLoader.m "$SING"/SGSingSeparator.m "$SING"/SGSingEngine.m \-framework Foundation -framework AudioToolbox -framework Accelerate -framework CoreML -o "$OUT"echo "built $OUT"diff --git a/harness/sing/button/main.m b/harness/sing/button/main.mindex c2264ff4..0827a5d9 100644--- a/harness/sing/button/main.m+++ b/harness/sing/button/main.m@@ -3,6 +3,7 @@//// ./build.sh && xcrun simctl install <udid> build/SingButtonHarness.app// xcrun simctl launch <udid> com.vojta.singbuttonharness -state 7 -panel 1 -level 0.4+// xcrun simctl launch <udid> com.vojta.singbuttonharness -state 2 -progress 0.4 -waiting 1// xcrun simctl io <udid> screenshot shot.png#import <UIKit/UIKit.h>#import "Redesigned/Lyrics/SGRSingButton.h"diff --git a/harness/sing/button/sing-stubs.m b/harness/sing/button/sing-stubs.mindex f1fbbda6..73e6cf67 100644--- a/harness/sing/button/sing-stubs.m+++ b/harness/sing/button/sing-stubs.m@@ -1,5 +1,6 @@// Shared/Sing's calls as the mic button makes them, answering from the launch line instead of the model and the-// audio: -state is an SGSingState (Sing.h), -progress the download's fraction. The lyrics harness links it too,+// audio: -state is an SGSingState (Sing.h), -progress the download's fraction, -waiting 1 a download held for the+// network. The lyrics harness links it too,// since SGRKaraokeView puts the mic on the lyrics.#import <UIKit/UIKit.h>#import "Shared/Sing/Sing.h"@@ -45,6 +46,7 @@ NSString *SGSingLevelText(double level) {}double SGSingModelProgress(void) { return [defaults() doubleForKey:@"progress"]; }+BOOL SGSingModelWaitingForNetwork(void) { return [defaults() boolForKey:@"waiting"]; }NSString *SGSingModelSizeText(void) { return @"489 MB"; }NSString *SGSingModelError(void) { return nil; }void SGSingDownloadModel(void) {}diff --git a/harness/sing/main.m b/harness/sing/main.mindex 56dbaf24..ce3cb57c 100644--- a/harness/sing/main.m+++ b/harness/sing/main.m@@ -3,12 +3,21 @@//// - the STFT: a sine's peak bin at the size torch.stft gives it (amplitude times the window's sum over two), and// noise through the STFT and back unchanged;-// - the model: loaded from a separator.mlmodelc folder at the compute units asked for, the time a window takes,-// and a voice mixed over chords taken apart: the vocals it finds against the voice, beside the mix's own score;+// - the loader (SGSingLoader.m): the CPU copy loaded and warmed first, then the faster copy on the compute units+// asked for; a second want joining the load in flight; a load past its deadline abandoned, Sing Failed, and a+// fresh load working while the abandoned one still runs, which is let go when it comes back; a purge during a+// load dropping its result; the copies kept over a quick off and on, and dropped after the time kept; each+// window on the faster copy only in the foreground; a window the faster copy fails done again on the CPU's;+// - the model: its shapes, the time a window takes, and a voice mixed over chords taken apart: the vocals it finds+// against the voice, beside the mix's own score;// - the engine: Spotify's mixer stood in for by the mix, pulled through SGSingEngineRender in IO buffers on a// thread that keeps real time, while the engine's worker separates ahead: the lead it builds, every frame-// out in order, the vocals level applied, a flush, the lead given back when Sing is switched off, and+// out in order, the vocals level applied, a flush, the lead kept and played on when Sing is switched off, and// no allocation on the render thread.+// - the lead, without the model: the lead reported is the frames pulled and not played, to the frame; held (the+// heat, resting), given up or switched off, it is kept and played on as it is, nothing skipped and none built; a+// flush (a pause, a seek) drops it at once; and the lead to take off a position Spotify counted at a moment is the+// lead held then, less what was dropped since.// - spatial voice, without the model: a separator that hands the whole window back as vocals, so what plays is// the vocals placed. A tone straight ahead plays exactly as it came; at 90 degrees right the right ear has it// louder by the pan's 7.7 dB at the same power and the left ear late by the delay around the head; at 90@@ -16,14 +25,15 @@// is held off: a turned head has it off to the side, the front catching up over 20 s, across +-180 degrees// without a jump, and ahead again after a gap in the motion.//-// ./build.sh && build/sing <separator.mlmodelc> <voice> <out dir> [all|cpu|gpu|ane]-// ./build.sh && build/sing spatial+// ./build.sh && build/sing <separator.mlmodelc> <voice> <out dir> [all|cpu|gpu|ane] (the faster copy's units)+// ./build.sh && build/sing spatial (the checks without the model: the lead, spatial voice, falling behind)#import <Foundation/Foundation.h>#import <Accelerate/Accelerate.h>#import <AudioToolbox/AudioToolbox.h>#import <CoreML/CoreML.h>#import <pthread.h>#import <stdatomic.h>+#import "Shared/Sing/SGSingLoader.h"#import "Shared/Sing/SGSingSeparator.h"#import "Shared/Sing/SGSingEngine.h"@@ -193,20 +203,119 @@ static Audio separateOffline(SGSingSeparator *separator, Audio mix, double *slowreturn vocals;}-static MLModel *loadModel(NSString *path, NSString *units) {- MLModelConfiguration *configuration = [MLModelConfiguration new];- configuration.computeUnits = unitsNamed(units);+// The main run loop turned until `done` holds or `seconds` pass; the loader hands back on the main queue.+static bool waitFor(double seconds, bool (^done)(void)) {+ double until = now() + seconds;+ while (!done() && now() < until) CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.05, true);+ return done();+}++// An MLModel whose every prediction fails, standing in for a GPU copy that iOS stops.+@interface SGFailingModel : MLModel+@end++@implementation SGFailingModel+- (id<MLFeatureProvider>)predictionFromFeatures:(id<MLFeatureProvider>)input error:(NSError **)error {+ if (error) *error = [NSError errorWithDomain:@"harness" code:1 userInfo:@{NSLocalizedDescriptionKey: @"refused, as iOS refuses the GPU in the background"}];+ return nil;+}+@end++// The loader through its cases; hands back the separator it leaves Ready, both copies in it.+static SGSingSeparator *checkLoader(NSString *path, NSString *units) {+ NSURL *url = [NSURL fileURLWithPath:path];+ MLComputeUnits fast = unitsNamed(units);+ SGSingLoaderSetForeground(YES);++ // A load past its deadline is abandoned and Sing is Failed; the mic off and on starts a fresh one at once,+ // which works while the first is still out, and the first is let go when it comes back.+ SGSingLoaderCPUDeadline = 0.2;+ SGSingLoaderWant(url, MLComputeUnitsCPUOnly);+ waitFor(5, ^bool { return SGSingLoaderCurrentState() != SGSingLoaderLoading; });+ CHECK(SGSingLoaderCurrentState() == SGSingLoaderFailed && [SGSingLoaderError() containsString:@"did not load in"],+ "a load past its deadline is abandoned, and Sing is Failed: %s", SGSingLoaderError().UTF8String ?: "(no reason)");+ SGSingLoaderCPUDeadline = 120;+ SGSingLoaderWant(url, MLComputeUnitsCPUOnly);+ CHECK(SGSingLoaderCurrentState() == SGSingLoaderFailed && SGSingLoaderAttempts() == 1, "Failed stays Failed until the mic is switched off and on");+ SGSingLoaderRelease();double began = now();+ SGSingLoaderWant(url, fast);+ SGSingLoaderWant(url, fast);+ CHECK(SGSingLoaderAttempts() == 2 && SGSingLoaderOutstanding() == 2, "off and on starts a fresh load beside the abandoned one, and a second want joins it (%u loads, %u out)",+ SGSingLoaderAttempts(), SGSingLoaderOutstanding());+ waitFor(300, ^bool { return SGSingLoaderCurrentState() != SGSingLoaderLoading; });+ double ready = now() - began;+ SGSingSeparator *separator = SGSingLoaderSeparator();+ CHECK(SGSingLoaderCurrentState() == SGSingLoaderReady && separator, "the CPU copy loads and warms in %.1f s, and Sing is Ready", ready);+ waitFor(300, ^bool { return SGSingLoaderOutstanding() == 0 && SGSingLoaderFastState() != SGSingFastLoading; });+ CHECK(SGSingLoaderSeparator() == separator, "the abandoned load came back and was let go: the separator is still the fresh load's");+ if (fast != MLComputeUnitsCPUOnly) {+ CHECK(SGSingLoaderFastState() == SGSingFastReady, "then the %s copy loads and warms beside it, %.1f s after the want (state %ld)",+ SGSingUnitsName(fast).UTF8String, now() - began, (long)SGSingLoaderFastState());+ }++ // Kept over a quick off and on, dropped once the time kept has passed.+ SGSingLoaderKeepSeconds = 1.5;+ SGSingLoaderRelease();+ waitFor(0.5, ^bool { return false; });+ SGSingLoaderWant(url, fast);+ waitFor(2, ^bool { return false; });+ CHECK(SGSingLoaderSeparator() == separator && SGSingLoaderAttempts() == 2 + (fast != MLComputeUnitsCPUOnly),+ "switched off and on within the time kept, the copies are still there and nothing loads (%u loads)", SGSingLoaderAttempts());+ SGSingLoaderRelease();+ waitFor(3, ^bool { return SGSingLoaderCurrentState() == SGSingLoaderIdle; });+ CHECK(SGSingLoaderCurrentState() == SGSingLoaderIdle && !SGSingLoaderSeparator(), "left off past the time kept, the copies are dropped");++ // A purge during a load: what the load brings back is let go.+ SGSingLoaderWant(url, MLComputeUnitsCPUOnly);+ SGSingLoaderPurge(@"the harness purges during the load");+ waitFor(300, ^bool { return SGSingLoaderOutstanding() == 0; });+ waitFor(0.5, ^bool { return false; });+ CHECK(SGSingLoaderCurrentState() == SGSingLoaderIdle && !SGSingLoaderSeparator(), "a load purged on its way is let go when it comes back");++ // The separator for the rest of the run, both copies in it, as Sing.x has it.+ SGSingLoaderKeepSeconds = 60;+ SGSingLoaderWant(url, fast);+ waitFor(300, ^bool { return SGSingLoaderCurrentState() != SGSingLoaderLoading+ && (fast == MLComputeUnitsCPUOnly || SGSingLoaderFastState() != SGSingFastLoading); });+ separator = SGSingLoaderSeparator();+ CHECK(separator != nil, "loaded again for the rest of the run");+ return separator;+}++// Each window on the copy that fits: the faster only in the foreground, a failed one done again on the CPU's, which+// keeps it until the app has been in the background.+static void checkCopies(SGSingSeparator *separator, Audio mix, bool hasFast) {+ float *vl = calloc(kSGSingWindowFrames, sizeof(float)), *vr = calloc(kSGSingWindowFrames, sizeof(float));NSError *error;- MLModel *model = [MLModel modelWithContentsOfURL:[NSURL fileURLWithPath:path] configuration:configuration error:&error];- CHECK(model != nil, "the model loads (%s compute units) in %.1f s%s%s", units.UTF8String, now() - began, error ? ": " : "",- error.localizedDescription.UTF8String ?: "");- if (!model) exit(1);- MLFeatureDescription *input = model.modelDescription.inputDescriptionsByName[@"spectrum"];- MLFeatureDescription *output = model.modelDescription.outputDescriptionsByName[@"vocals_spectrum"];- CHECK([input.multiArrayConstraint.shape isEqualToArray:(@[@1, @2050, @201, @2])] && [output.multiArrayConstraint.shape isEqualToArray:(@[@1, @2050, @201, @2])],- "its spectrum and vocals_spectrum are [1, 2050, 201, 2]");- return model;+ SGSingSeparatorStats before = [separator stats];+ if (hasFast) {+ [separator separateLeft:mix.left right:mix.right vocalsLeft:vl vocalsRight:vr error:&error];+ SGSingSeparatorStats stats = [separator stats];+ CHECK(stats.fast && stats.windows[1] == before.windows[1] + 1, "in the foreground a window goes to the faster copy");+ SGSingSeparatorSetForeground(false);+ [separator separateLeft:mix.left right:mix.right vocalsLeft:vl vocalsRight:vr error:&error];+ stats = [separator stats];+ CHECK(!stats.fast && stats.windows[0] == before.windows[0] + 1, "in the background it goes to the CPU's");+ SGSingSeparatorSetForeground(true);+ before = stats;+ }+ // The fast copy swapped for one that fails, then put back.+ SGFailingModel *failing = [SGFailingModel alloc];+ [separator setFastModel:failing named:@"failing"];+ BOOL ok = [separator separateLeft:mix.left right:mix.right vocalsLeft:vl vocalsRight:vr error:&error];+ SGSingSeparatorStats stats = [separator stats];+ CHECK(ok && stats.fallbacks == before.fallbacks + 1 && stats.windows[0] == before.windows[0] + 1 && !stats.fast,+ "a window the faster copy fails is done again on the CPU's (%s)", ok ? "separated" : error.localizedDescription.UTF8String);+ [separator separateLeft:mix.left right:mix.right vocalsLeft:vl vocalsRight:vr error:&error];+ stats = [separator stats];+ CHECK(stats.fallbacks == before.fallbacks + 1 && stats.windows[0] == before.windows[0] + 2, "and the faster copy rests for the next window");+ SGSingSeparatorSetForeground(false);+ SGSingSeparatorSetForeground(true);+ [separator separateLeft:mix.left right:mix.right vocalsLeft:vl vocalsRight:vr error:&error];+ CHECK([separator stats].fallbacks == before.fallbacks + 2, "until the app has been in the background, when it is tried again");+ free(vl);+ free(vr);}// Which device Core ML means to run each operation on, counted, where the OS can tell.@@ -303,7 +412,8 @@ static void checkEngine(SGSingSeparator *separator, Audio mix, Audio voice, Audiprintf(" info lead at 12 s %.2f s (at most %.2f s, target %.2f s), %llu windows at %.0f ms average, %llu frames played dry while on\n",leadAtOn, leadMax, stats.targetLead, stats.windows, stats.averageMS, stats.dryFrames);CHECK(leadAtOn > 2 && leadAtOn < 7, "with Sing on the engine pulls ahead of what plays (%.2f s)", leadAtOn);- CHECK(leadAtOff < 0.05, "switched off it gives the lead back (%.3f s left)", leadAtOff);+ CHECK(leadAtOff > 2, "switched off it keeps the lead and plays it on (%.2f s)", leadAtOff);+ size_t skipped = (size_t)llround(stats.dropped * kSGSingRate);// What plays is the mixer's frames in order, from 6 s to 16 s the mix less the vocals the offline pass finds,// with the vocals at 0.size_t from = 6 * kSGSingRate, to = 16 * kSGSingRate;@@ -322,13 +432,13 @@ static void checkEngine(SGSingSeparator *separator, Audio mix, Audio voice, Audiprintf(" info the karaoke against the true instrumental: %.1f dB; the mix's own: %.1f dB\n", snr(instrumental, out, from, to), snr(instrumental, mix, from, to));double after = 0;size_t firstOff = 0;- for (size_t i = 26 * kSGSingRate; i < out.frames / 1024 * 1024; i++) {- double d = fabs(out.left[i] - mix.left[i]) + fabs(out.right[i] - mix.right[i]);+ for (size_t i = 26 * kSGSingRate; i < out.frames / 1024 * 1024 && i + skipped < mix.frames; i++) {+ double d = fabs(out.left[i] - mix.left[i + skipped]) + fabs(out.right[i] - mix.right[i + skipped]);if (d > 1e-6 && !firstOff) firstOff = i;after = fmax(after, d);}if (firstOff) printf(" info first difference at %.4f s: out %g, mix %g, mix a frame on %g\n", firstOff / (double)kSGSingRate, out.left[firstOff], mix.left[firstOff], mix.left[firstOff + 1]);- CHECK(after < 1e-6, "off again, the mix plays on where it was (%.2g)", after);+ CHECK(after < 1e-6 && skipped == 0, "off again, the mix plays on where it was, nothing skipped (%.2g, %.2f s dropped)", after, stats.dropped);writeWAV(@"engine.wav", out);// The level: at 2 only the vocals play.@@ -337,7 +447,16 @@ static void checkEngine(SGSingSeparator *separator, Audio mix, Audio voice, AudiSGSingEngineSetLevel(loud, 2);SGSingEngineSetOn(loud, true);Source third = {mix, 0};- Audio alone = play(loud, mix, &third, 1024, 10, ^(double played) {});+ __block double inAt = -1;+ __block int switches = 0;+ __block bool was = false;+ Audio alone = play(loud, mix, &third, 1024, 10, ^(double played) {+ bool mixing = SGSingEngineReadStats(loud).mixing;+ if (mixing && inAt < 0) inAt = played;+ switches += mixing != was;+ was = mixing;+ });+ printf(" info at the top of the slider the vocals came in at %.2f s and switched %d times\n", inAt, switches);double vocalsOnly = snr(offlineVocals, alone, 6 * kSGSingRate, 9 * kSGSingRate);CHECK(vocalsOnly > 30, "at the top of the slider the vocals play alone (%.1f dB against the offline vocals)", vocalsOnly);SGSingEngineDestroy(loud);@@ -377,6 +496,177 @@ static void checkEngine(SGSingSeparator *separator, Audio mix, Audio voice, Audi}@end+// Whole windows back as vocals, as fast as asked: the first `fastWindows` at once, then `slow` seconds each; with a+// NaN in every window when `poison` is set.+@interface SGPacedSeparator : SGWholeSeparator+@property (nonatomic) int fastWindows;+@property double slow; // atomic: changed while the worker runs a window+@property (nonatomic) bool poison;+@end++@implementation SGPacedSeparator {+ int _done;+}+- (BOOL)separateLeft:(const float *)left right:(const float *)right vocalsLeft:(float *)vocalsLeft vocalsRight:(float *)vocalsRight error:(NSError **)error {+ [super separateLeft:left right:right vocalsLeft:vocalsLeft vocalsRight:vocalsRight error:error];+ if (_poison) vocalsLeft[1000] = vocalsRight[2000] = NAN;+ if (_done++ >= _fastWindows) usleep((useconds_t)(self.slow * 1e6));+ return YES;+}+@end++// The lead without the model: what is reported against what was pulled and played, kept and played on when not+// separating, dropped by a flush, and the lead to take off a position counted at a moment.+static void checkHold(void) {+ size_t frames = 40 * kSGSingRate;+ Audio tone = makeAudio(frames);+ for (size_t i = 0; i < frames; i++) tone.left[i] = tone.right[i] = (float)(0.5 * sin(2 * M_PI * 440 * i / kSGSingRate));+ SGSingEngine *engine = SGSingEngineCreate();+ SGPacedSeparator *steady = [[SGPacedSeparator alloc] initWithModel:nil];+ steady.fastWindows = INT_MAX;+ SGSingEngineSetSeparator(engine, steady);+ SGSingEngineSetLevel(engine, 1);+ SGSingEngineSetOn(engine, true);+ Source source = {tone, 0}, *pulled = &source;+ __block double worst = 0, at1 = 0, leadAt1 = 0, leadAt8 = 0, heldLeast = 100, heldMost = 0, takeOffHeld = -1, leadAt12 = 0;+ __block double afterFlush = -1, takeOffFlushed = 0, leadAt19 = 0, leadAt21 = 0, droppedAt12 = -1;+ malloc_logger = countAllocation;+ Audio out = play(engine, tone, &source, 1024, 22, ^(double played) {+ // Between renders: what was pulled less what was played out, less what was dropped.+ double held = ((double)pulled->pulled - played * kSGSingRate) / kSGSingRate - SGSingEngineReadStats(engine).dropped;+ if (!SGSingEngineReadStats(engine).dropped || played > 12.1) worst = fmax(worst, fabs(SGSingEngineLead(engine) - held));+ // While the lead fills.+ if (played >= 1 && !at1) {+ at1 = CFAbsoluteTimeGetCurrent();+ leadAt1 = SGSingEngineLead(engine);+ }+ if (played >= 8 && !leadAt8) leadAt8 = SGSingEngineLead(engine);+ // Held (the heat, resting) from 8 s to 12 s, then separating again; off at 20 s.+ SGSingEngineSetPaused(engine, played >= 8 && played < 12);+ if (played >= 8.5 && played < 12) {+ heldLeast = fmin(heldLeast, SGSingEngineLead(engine));+ heldMost = fmax(heldMost, SGSingEngineLead(engine));+ }+ if (played >= 11 && takeOffHeld < 0) takeOffHeld = SGSingEngineLeadAt(engine, at1);+ // A pause or a seek: the flush drops the lead, read as dropped at once.+ if (played >= 12 && droppedAt12 < 0) {+ leadAt12 = SGSingEngineLead(engine);+ SGSingEngineFlush(engine);+ afterFlush = SGSingEngineLead(engine);+ takeOffFlushed = SGSingEngineLeadAt(engine, at1);+ droppedAt12 = SGSingEngineReadStats(engine).dropped;+ }+ if (played >= 19 && !leadAt19) leadAt19 = SGSingEngineLead(engine);+ if (played >= 20 && SGSingEngineOn(engine)) SGSingEngineSetOn(engine, false);+ if (played >= 21 && !leadAt21) leadAt21 = SGSingEngineLead(engine);+ });+ malloc_logger = NULL;+ double dropped = SGSingEngineReadStats(engine).dropped;+ CHECK(atomic_load(&sg_renderAllocations) == 0, "the lead: no allocation on the render thread (%u)", atomic_load(&sg_renderAllocations));+ CHECK(worst < 0.5 / kSGSingRate, "the lead reported is the frames pulled and not yet played, to the frame (off by %.0f at most)", worst * kSGSingRate);+ CHECK(heldLeast == leadAt8 && heldMost == leadAt8 && leadAt8 > leadAt1 + 1,+ "held at 8 s (the heat, resting), the %.2f s held are kept as they are (%.2f to %.2f s), none dropped or built", leadAt8, heldLeast, heldMost);+ CHECK(fabs(takeOffHeld - leadAt1) < 1100.0 / kSGSingRate,+ "a position counted at 1 s, as the lead filled, has the %.2f s held then taken off (%.2f s), not the %.2f s held later", leadAt1, takeOffHeld, leadAt8);+ double skippedBefore = 0;+ for (size_t i = 0; i < 12 * kSGSingRate; i++) skippedBefore = fmax(skippedBefore, fabsf(out.left[i] - tone.left[i]));+ CHECK(skippedBefore < 1e-6, "until the flush every frame plays in order, held or not: nothing skipped (%.2g)", skippedBefore);+ CHECK(afterFlush == 0 && droppedAt12 >= 0 && fabs(takeOffFlushed - (leadAt1 - leadAt12)) < 1100.0 / kSGSingRate,+ "a flush (a pause, a seek) drops the %.2f s at once: the lead reads %.2f s, and a position counted at 1 s has %.2f s taken off (%.2f less %.2f)",+ leadAt12, afterFlush, takeOffFlushed, leadAt1, leadAt12);+ size_t skipped = (size_t)llround(dropped * kSGSingRate);+ double after = 0;+ for (size_t i = 12.1 * kSGSingRate; i < 13 * kSGSingRate; i++) after = fmax(after, fabsf(out.left[i] - tone.left[i + skipped]));+ CHECK(after < 1e-6, "after it, the tone plays on from the mixer's next frame, %.2f s on (%.2g)", dropped, after);+ CHECK(leadAt19 > 2, "separating again from 12 s, the lead is built again (%.2f s at 19 s)", leadAt19);+ CHECK(fabs(leadAt21 - leadAt19) < 0.6, "switched off at 20 s, the lead is kept and plays on (%.2f s at 21 s)", leadAt21);+ SGSingEngineDestroy(engine);+}++// A model that keeps up for a while and then falls behind: the vocals fade out once, not at every window, and+// after 8 s short of them the engine gives up and gives the lead back. And a NaN from the model never plays.+static void checkFallingBehind(void) {+ size_t frames = 40 * kSGSingRate;+ Audio tone = makeAudio(frames);+ for (size_t i = 0; i < frames; i++) tone.left[i] = tone.right[i] = (float)(0.5 * sin(2 * M_PI * 440 * i / kSGSingRate));+ SGSingEngine *engine = SGSingEngineCreate();+ SGPacedSeparator *separator = [[SGPacedSeparator alloc] initWithModel:nil];+ separator.fastWindows = 6;+ separator.slow = 2.2;+ SGSingEngineSetSeparator(engine, separator);+ SGSingEngineSetLevel(engine, 0);+ SGSingEngineSetOn(engine, true);+ Source source = {tone, 0};+ __block int switches = 0;+ __block bool was = false, everIn = false;+ __block double gaveUpAt = 0;+ Audio out = play(engine, tone, &source, 1024, 40, ^(double played) {+ bool mixing = SGSingEngineReadStats(engine).mixing;+ if (mixing != was) switches++;+ was = mixing;+ everIn |= mixing;+ if (!gaveUpAt && SGSingEngineGaveUp(engine)) gaveUpAt = played;+ });+ CHECK(everIn && switches <= 2, "a model falling behind: the vocals came in and went out %d times (at most once each)", switches);+ CHECK(gaveUpAt > 0 && SGSingEngineLead(engine) > 1, "and after 8 s short of them the engine gave up (at %.1f s) and kept the lead (%.2f s)",+ gaveUpAt, SGSingEngineLead(engine));+ size_t skipped = (size_t)llround(SGSingEngineReadStats(engine).dropped * kSGSingRate);+ double last = 0;+ // Past the tone's end the source hands over silence.+ for (size_t i = 38 * kSGSingRate; i < 40 * kSGSingRate - 1024; i++) last = fmax(last, fabsf(out.left[i] - (i + skipped < frames ? tone.left[i + skipped] : 0)));+ CHECK(last == 0, "then the song plays as it is (%.2g)", last);+ // Sing.x tries again at the next track by switching the engine off and on: a fresh budget, and the vocals back.+ separator.slow = 0;+ SGSingEngineSetOn(engine, false);+ SGSingEngineSetOn(engine, true);+ Source next = {tone, 0};+ __block bool again = false;+ play(engine, tone, &next, 1024, 12, ^(double played) { again |= SGSingEngineReadStats(engine).mixing; });+ CHECK(again && !SGSingEngineGaveUp(engine), "switched off and on after giving up, it tries again and the vocals come back");+ SGSingEngineDestroy(engine);++ // Held for the heat for 10 s and let go: a start again, which the budget does not count.+ SGSingEngine *held = SGSingEngineCreate();+ SGPacedSeparator *steady = [[SGPacedSeparator alloc] initWithModel:nil];+ steady.fastWindows = INT_MAX;+ SGSingEngineSetSeparator(held, steady);+ SGSingEngineSetOn(held, true);+ Source third = {tone, 0};+ __block bool backIn = false;+ __block double spentAfter = -1;+ __block unsigned long long windowsAt8 = 0, windowsAt15 = 0;+ Audio heldOut = play(held, tone, &third, 1024, 30, ^(double played) {+ SGSingEngineSetPaused(held, played >= 6 && played < 16);+ if (played >= 17 && spentAfter < 0) spentAfter = SGSingEngineReadStats(held).budgetSpent;+ if (played >= 20) backIn |= SGSingEngineReadStats(held).mixing;+ if (played < 8) windowsAt8 = SGSingEngineReadStats(held).windows;+ if (played < 15) windowsAt15 = SGSingEngineReadStats(held).windows;+ });+ CHECK(spentAfter == 0 && !SGSingEngineGaveUp(held) && backIn,+ "held for 10 s and let go, the wait for the vocals is a start (%.1f s of the budget spent 1 s on), they come back, and the engine does not give up", spentAfter);+ // Sing.x rests the same way at the vocals as sung: no window runs, and the song plays exactly as it is.+ double heldDiff = 0;+ skipped = (size_t)llround(SGSingEngineReadStats(held).dropped * kSGSingRate);+ for (size_t i = 8 * kSGSingRate; i < 15 * kSGSingRate; i++) heldDiff = fmax(heldDiff, fabsf(heldOut.left[i] - tone.left[i + skipped]));+ CHECK(windowsAt15 == windowsAt8 && heldDiff == 0, "held, the model runs no window (%llu then %llu) and the song plays exactly as it is (%.2g)",+ windowsAt8, windowsAt15, heldDiff);+ SGSingEngineDestroy(held);++ SGSingEngine *poisoned = SGSingEngineCreate();+ SGPacedSeparator *bad = [[SGPacedSeparator alloc] initWithModel:nil];+ bad.poison = true;+ bad.fastWindows = INT_MAX;+ SGSingEngineSetSeparator(poisoned, bad);+ SGSingEngineSetLevel(poisoned, 2);+ SGSingEngineSetOn(poisoned, true);+ Source second = {tone, 0};+ Audio sung = play(poisoned, tone, &second, 1024, 8, ^(double played) {});+ bool finite = true;+ for (size_t i = 0; i < sung.frames; i++) finite &= isfinite(sung.left[i]) && isfinite(sung.right[i]);+ CHECK(finite && SGSingEngineReadStats(poisoned).mixing, "a NaN from the model plays as silence, never as a NaN");+ SGSingEngineDestroy(poisoned);+}+static double energy(const float *samples, size_t from, size_t to) {double sum = 0;for (size_t i = from; i < to; i++) sum += (double)samples[i] * samples[i];@@ -413,6 +703,11 @@ static void checkSpatial(void) {});malloc_logger = NULL;CHECK(atomic_load(&sg_renderAllocations) == 0, "spatial voice: no allocation on the render thread (%u)", atomic_load(&sg_renderAllocations));+ // The loudness the Sing page draws: all of the tone is vocals here, a sine of 0.5 at an RMS of 0.354.+ float vocalsLevel[5], restLevel[5];+ SGSingEngineReadLevels(engine, vocalsLevel, restLevel, 5);+ CHECK(fabsf(vocalsLevel[0] - 0.3536f) < 0.01f && fabsf(vocalsLevel[4] - 0.3536f) < 0.01f && restLevel[0] < 0.001f,+ "the loudness of what plays now: vocals %.3f (a 0.5 sine's 0.354), the rest %.3f", vocalsLevel[4], restLevel[4]);SGSingEngineStats stats = SGSingEngineReadStats(engine);CHECK(stats.ready > 0 && stats.dryFrames < 4 * kSGSingRate, "spatial voice: the vocals are in (%.2f s separated ahead, %.2f s played dry)",stats.ready, stats.dryFrames / (double)kSGSingRate);@@ -482,13 +777,15 @@ int main(int argc, char **argv) {@autoreleasepool {if (argc == 2 && !strcmp(argv[1], "spatial")) {sg_outDir = NSTemporaryDirectory();+ checkHold();checkSpatial();checkFront();+ checkFallingBehind();printf("%s\n", sg_failures ? "FAILED" : "all passed");return sg_failures ? 1 : 0;}if (argc < 4) {- printf("usage: sing <separator.mlmodelc> <voice> <out dir> [all|cpu|gpu|ane]\n sing spatial\n");+ printf("usage: sing <separator.mlmodelc> <voice> <out dir> [all|cpu|gpu|ane] (the faster copy's units; cpu for none)\n sing spatial\n");return 2;}NSString *modelPath = @(argv[1]), *units = argc > 4 ? @(argv[4]) : @"all";@@ -496,9 +793,15 @@ int main(int argc, char **argv) {[NSFileManager.defaultManager createDirectoryAtPath:sg_outDir withIntermediateDirectories:YES attributes:nil error:nil];checkSTFT();- MLModel *model = loadModel(modelPath, units);+ SGSingSeparator *separator = checkLoader(modelPath, units);+ if (!separator) return 1;+ MLModel *shapes = [MLModel modelWithContentsOfURL:[NSURL fileURLWithPath:modelPath] configuration:[MLModelConfiguration new] error:nil];+ MLFeatureDescription *input = shapes.modelDescription.inputDescriptionsByName[@"spectrum"];+ MLFeatureDescription *output = shapes.modelDescription.outputDescriptionsByName[@"vocals_spectrum"];+ CHECK([input.multiArrayConstraint.shape isEqualToArray:(@[@1, @2050, @201, @2])] && [output.multiArrayConstraint.shape isEqualToArray:(@[@1, @2050, @201, @2])],+ "its spectrum and vocals_spectrum are [1, 2050, 201, 2]");+ shapes = nil;describePlan(modelPath, units);- SGSingSeparator *separator = [[SGSingSeparator alloc] initWithModel:model];Audio voice = readAudio(@(argv[2]));size_t frames = (size_t)kSGSingRate * 30;@@ -513,7 +816,7 @@ int main(int argc, char **argv) {}double slowest, average;Audio vocals = separateOffline(separator, mix, &slowest, &average);- printf(" info a two second window takes %.0f ms on average, %.0f ms at most (the first one warms the model up)\n", average * 1000, slowest * 1000);+ printf(" info a two second window takes %.0f ms on average, %.0f ms at most (the copies were warmed with silence)\n", average * 1000, slowest * 1000);size_t to = frames - kSGSingWindowFrames / 2;double found = snr(padded, vocals, kSGSingRate, to), baseline = snr(padded, mix, kSGSingRate, to);CHECK(found > baseline + 10, "the vocals it finds score %.1f dB against the voice, the mix itself %.1f dB", found, baseline);@@ -529,8 +832,12 @@ int main(int argc, char **argv) {writeWAV(@"accompaniment.wav", accompaniment);checkEngine(separator, mix, padded, vocals);+ // Last, as it swaps the faster copy out.+ checkCopies(separator, mix, unitsNamed(units) != MLComputeUnitsCPUOnly);+ checkHold();checkSpatial();checkFront();+ checkFallingBehind();printf("%s\n", sg_failures ? "FAILED" : "all passed");return sg_failures ? 1 : 0;}diff --git a/harness/sing/shim/Core/SGLog.h b/harness/sing/shim/Core/SGLog.hnew file mode 100644index 00000000..a712ac11--- /dev/null+++ b/harness/sing/shim/Core/SGLog.h@@ -0,0 +1,5 @@+// Stands in for tweak/Sources/Core/SGLog.h in the Mac harness: the separator's and the loader's log lines go to stderr,+// timed from the start, so a run shows them beside the checks.+#import <Foundation/Foundation.h>++#define SGLog(fmt, ...) fprintf(stderr, " log %s\n", [NSString stringWithFormat:(fmt), ##__VA_ARGS__].UTF8String)diff --git a/harness/sing/sim/main.m b/harness/sing/sim/main.mindex aeacaea2..7870e5c4 100644--- a/harness/sing/sim/main.m+++ b/harness/sing/sim/main.m@@ -6,9 +6,11 @@//// The converter's callback stands in for Spotify's decoder: a voice (macOS's speech, bundled by build-sim.sh)// over synthetic chords. A mock SPTPlayerState computes -position the way Spotify's does (positionAsOfTimestamp-// messaged, disassembly at 0x1057735ec), and a mock SPTEsperantoPlayer moves the decoder on seekTo:. A render-// notify on the RemoteIO unit records what plays. The script: Sing on at the vocals' 0, play, seek, Sing off;-// each check prints a line, and what played is scored against the chords alone.+// messaged and run on from its timestamp, disassembly at 0x1057735ec), reported as Spotify's player reports: when+// playing starts and on a seek, not as it plays. A mock SPTEsperantoPlayer moves the decoder on seekTo:. A render+// notify on the RemoteIO unit records what plays. The script: Sing on at the vocals' 0, play, seek, rest at As sung,+// separate again, the thermal state Serious and back to Fair, a memory warning's stop, a pause and a resume, Sing+// off; each check prints a line, and what played is scored against the chords alone.//// THEOS=$HOME/theos VOICE=speech.aiff ./build-sim.sh// xcrun simctl install <udid> build/sim/SingHarness.app@@ -16,8 +18,10 @@#import <UIKit/UIKit.h>#import <AudioToolbox/AudioToolbox.h>#import <AVFoundation/AVFoundation.h>+#import <objc/runtime.h>#import <stdatomic.h>#import "Shared/Sing/Sing.h"+#import "Shared/Sing/SGSingLoader.h"static const double kRate = 44100;enum { kMixSeconds = 60, kRecordSeconds = 70 };@@ -31,15 +35,44 @@ static BOOL sg_modelKeepsUp = YES;#pragma mark - Spotify's player, as far as Sing reads it-// Shared/Player/PlayerState.x's, which the harness does without: no state is reported to Sing's track watcher.+// Shared/Player/PlayerState.x's: the harness's last report, told to Sing's track watcher when the script says.NSString *SGURIString(id uri) {return [uri isKindOfClass:NSString.class] ? uri : nil;}-void SGAddPlayerStateObserver(id observer) {}+@protocol SGHarnessWatcher+- (void)playerStateDidChange:(id)state;+@end++static id sg_watcher, sg_reported, sg_player;++// Which clock the mock player keeps (launch argument -clock): Spotify's last report run on by the time, read as it is+// (report); that report stamped again at every read of the player's state, the same line of the clock (restamp); or+// reported again every half second with what the decoder has handed over (decoded).+typedef NS_ENUM(NSInteger, SGHarnessClock) { SGHarnessClockReport, SGHarnessClockRestamp, SGHarnessClockDecoded };+static SGHarnessClock sg_clock;++void SGAddPlayerStateObserver(id observer) {+ sg_watcher = observer;+}id SGPlayerState(void) {- return nil;+ return sg_reported;+}++// Shared/Lyrics/KaraokeSource.x's: the player the lyrics read and seek, the mock below, sent the line's time in seconds.+@class SPTPlayerState;+@interface SPTEsperantoPlayer : NSObject+- (id)seekTo:(double)position;+- (SPTPlayerState *)state;+@end++id SGKaraokePlayer(void) {+ return sg_player;+}++void SGKaraokeSeek(NSInteger ms) {+ [sg_player seekTo:ms / 1000.0];}// Shared/HeadGestures' head motion, for spatial voice: none in the simulator, so the voice stays ahead.@@ -51,7 +84,11 @@ BOOL SGHeadGesturesAvailable(void) {@interface SPTPlayerState : NSObject@property (nonatomic) double positionAsOfTimestamp;+@property (nonatomic) double spotifyAsOf; // the same, out of the hook's reach, for the mock to stamp again@property (nonatomic, strong) NSDate *timestamp;+@property (nonatomic) BOOL isPaused, isPlaying;+@property (nonatomic) double duration;+@property (nonatomic, strong) id track;@end@implementation SPTPlayerState@@ -66,6 +103,18 @@ BOOL SGHeadGesturesAvailable(void) {}@end+// The thermal state Sing reads, set by the script.+static NSProcessInfoThermalState sg_thermal = NSProcessInfoThermalStateNominal;++static NSProcessInfoThermalState thermalState(id self, SEL _cmd) {+ return sg_thermal;+}++static void setThermal(NSProcessInfoThermalState state) {+ sg_thermal = state;+ [NSNotificationCenter.defaultCenter postNotificationName:NSProcessInfoThermalStateDidChangeNotification object:NSProcessInfo.processInfo];+}+#pragma mark - the decoderstatic float *sg_left, *sg_right, *sg_chordsLeft, *sg_chordsRight;@@ -86,21 +135,30 @@ static OSStatus decoder(void *refCon, AudioUnitRenderActionFlags *flags, const Areturn noErr;}-@interface SPTEsperantoPlayer : NSObject-@end-static id sg_seekResult;// Each command returns an object of Spotify's, which the hook must hand back.@implementation SPTEsperantoPlayer+// As Spotify's, into the last seek, which moves the decoder.- (id)seekTo:(double)position {- atomic_store(&sg_readAt, (uint64_t)(position * kRate));- return sg_seekResult;+ return [self seekTo:position relative:0 options:nil creatorTimestampPositionMs:-1];+}+- (SPTPlayerState *)state {+ SPTPlayerState *reported = sg_reported;+ if (sg_clock != SGHarnessClockRestamp || !reported || reported.isPaused) return reported;+ SPTPlayerState *again = [SPTPlayerState new];+ again.timestamp = [NSDate date];+ again.positionAsOfTimestamp = again.spotifyAsOf = reported.spotifyAsOf - reported.timestamp.timeIntervalSinceNow;+ again.isPlaying = YES;+ return again;}- (id)seekTo:(double)position relative:(long long)relative { return sg_seekResult; }- (id)seekTo:(double)position options:(id)options { return [self seekTo:position]; }- (id)seekTo:(double)position relative:(long long)relative options:(id)options { return sg_seekResult; }-- (id)seekTo:(double)position relative:(long long)relative options:(id)options creatorTimestampPositionMs:(double)creator { return sg_seekResult; }+- (id)seekTo:(double)position relative:(long long)relative options:(id)options creatorTimestampPositionMs:(double)creator {+ atomic_store(&sg_readAt, (uint64_t)(position * kRate));+ return sg_seekResult;+}- (id)skipToNextTrackWithOptions:(id)options { return sg_seekResult; }- (id)skipToPreviousTrackWithOptions:(id)options { return sg_seekResult; }- (id)skipToNextTrackWithOptions:(id)options track:(id)track { return sg_seekResult; }@@ -232,6 +290,8 @@ static void check(OSStatus status, const char *what) {uint64_t _lastDecoded;CFTimeInterval _lastAt;SPTPlayerState *_state;+ BOOL _paused;+ CFTimeInterval _tappedAt;double _seekLead;}@@ -274,11 +334,14 @@ static void check(OSStatus status, const char *what) {dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(seconds * NSEC_PER_SEC)), dispatch_get_main_queue(), block);}-// The player reporting what the decoder has handed over, as Spotify's does now and then.+// The player reporting what the decoder has handed over, as Spotify's does on a change (playing, a seek), not as it plays.- (void)playerReports {_state = [SPTPlayerState new];- _state.positionAsOfTimestamp = atomic_load(&sg_readAt) / kRate;+ _state.positionAsOfTimestamp = _state.spotifyAsOf = atomic_load(&sg_readAt) / kRate;_state.timestamp = [NSDate date];+ _state.isPaused = _paused;+ _state.isPlaying = YES;+ sg_reported = _state;}// What is heard now, in seconds of the mix: what the output played, less what a seek skipped.@@ -286,13 +349,17 @@ static void check(OSStatus status, const char *what) {return atomic_load(&sg_heard) / sg_hardwareRate;}+// Where in the mix what is heard now is, by the decoder less what Sing holds (the Mac harness checks the lead to the frame).+- (double)heardInMix {+ return atomic_load(&sg_readAt) / kRate - SGSingHeldLead();+}+- (void)report {CFTimeInterval now = CACurrentMediaTime();uint64_t decoded = atomic_load(&sg_decoded);- [self playerReports];- NSLog(@"[harness] %5.1f s %-22@ decoder %.2fx position %.2f s, Spotify's own %.2f s (%.2f s ahead of what plays)",+ NSLog(@"[harness] %5.1f s %-22@ decoder %.2fx position %.2f s, heard %.2f s, decoded %.2f s, lead %.2f s",now - _startedAt, SGSingStatusText(), (decoded - _lastDecoded) / kRate / (now - _lastAt), _state.position,- atomic_load(&sg_readAt) / kRate, atomic_load(&sg_readAt) / kRate - _state.position);+ [self heardInMix], atomic_load(&sg_readAt) / kRate, SGSingHeldLead());_lastDecoded = decoded;_lastAt = now;}@@ -305,40 +372,125 @@ static void check(OSStatus status, const char *what) {[self after:seconds do:^{ [self waitFor:condition every:seconds then:then]; }];}+// The position Spotify shows against where in the mix what is heard is.+// The state the lyrics read: the player's.+- (SPTPlayerState *)lyricsState {+ return [(SPTEsperantoPlayer *)sg_player state];+}++- (void)checkPosition:(NSString *)when {+ double position = [self lyricsState].position, heard = [self heardInMix];+ CHECK(fabs(position - heard) < 0.15, @"%@ the position Spotify shows, %.2f s, is what is heard, %.2f s (decoded %.2f s, %.2f s held)", when, position, heard,+ atomic_load(&sg_readAt) / kRate, SGSingHeldLead());+}+- (void)play {- // Played straight through until the first seek: the position Spotify shows against what is heard.- for (int s = 1; s <= 46; s++) [self after:s do:^{ [self report]; }];+ // Spotify reports that it plays, once, and its clock runs on by itself from there; or, as the decoded clock,+ // again every half second while it plays.+ [self playerReports];+ if (sg_clock == SGHarnessClockDecoded) {+ [NSTimer scheduledTimerWithTimeInterval:0.5 repeats:YES block:^(NSTimer *timer) {+ if (!self->_paused) [self playerReports];+ }];+ }+ for (int s = 1; s <= 100; s++) [self after:s do:^{ [self report]; }];[self after:14 do:^{- double heard = [self heardContent], position = self->_state.position;- [self playerReports];- position = self->_state.position;+ double heard = [self heardContent], position = [self lyricsState].position, decoded = atomic_load(&sg_decoded) / kRate;sg_modelKeepsUp = SGSingCurrentState() != SGSingStateBehind && SGSingCurrentState() != SGSingStateBuffering;VOCALS_CHECK(SGSingCurrentState() == SGSingStateSinging, @"Sing is on with the vocals down at 14 s (%@)", SGSingStatusText());- CHECK(fabs(position - heard) < 0.15, @"the position Spotify shows, %.2f s, is what is heard, %.2f s, not what was decoded, %.2f s",- position, heard, atomic_load(&sg_readAt) / kRate);+ CHECK(SGSingHeldLead() > 2 && fabs(decoded - heard - SGSingHeldLead()) < 0.1,+ @"the lead Sing reports, %.2f s, is what the decoder handed over less what was heard (%.2f s less %.2f s)", SGSingHeldLead(), decoded, heard);+ CHECK(fabs(position - heard) < 0.15, @"as the lead filled, the position the lyrics read, %.2f s, is what is heard, %.2f s, "+ @"not what was decoded, %.2f s, nor that less the lead now, %.2f s", position, heard, decoded, position - SGSingHeldLead());}];+ // A tap on the lyrics line at 5 s, with the lead held: the lyrics' seek, as the karaoke view sends it.[self after:20 do:^{- double lead = atomic_load(&sg_readAt) / kRate - self->_state.position;- self->_seekLead = lead;- id result = [[SPTEsperantoPlayer new] seekTo:5];- CHECK(result == sg_seekResult, @"a seek hands Spotify back the result of its seek");+ self->_seekLead = SGSingHeldLead();+ CHECK(self->_seekLead > 2, @"before the tap a lead is held (%.2f s)", self->_seekLead);+ SGKaraokeSeek(5000);+ [self playerReports];+ self->_tappedAt = CACurrentMediaTime();+ CHECK(fabs(atomic_load(&sg_readAt) / kRate - 5) < 0.05, @"the tap sends Spotify to the line's 5.00 s (%.2f s)", atomic_load(&sg_readAt) / kRate);}];[self after:20.5 do:^{- [self playerReports];- double ahead = atomic_load(&sg_readAt) / kRate - self->_state.position;+ double ahead = atomic_load(&sg_readAt) / kRate - [self lyricsState].position;CHECK(ahead < 1.2, @"the seek dropped the %.1f s held ahead: %.2f s ahead half a second later", self->_seekLead, ahead);}];- [self after:30 do:^{- VOCALS_CHECK(SGSingCurrentState() == SGSingStateSinging, @"ten seconds after the seek the vocals are down again (%@)", SGSingStatusText());- SGSetSingOn(NO);+ [self after:21 do:^{+ double since = CACurrentMediaTime() - self->_tappedAt, heard = [self heardInMix];+ CHECK(fabs(heard - (5 + since)) < 0.2, @"a second after the tap what is heard is the line's 5 s run on by %.2f s: %.2f s", since, heard);+ [self checkPosition:@"a second after the tap,"];}];- [self after:45 do:^{- [self playerReports];- double ahead = atomic_load(&sg_readAt) / kRate - self->_state.position;- CHECK(ahead < 0.05 && SGSingCurrentState() == SGSingStateOff, @"switched off, the lead is given back (%.2f s) and Sing is %@", ahead, SGSingStatusText());- [self score];- NSLog(@"[harness] %@", sg_failures ? @"FAILED" : @"all passed");- exit(sg_failures ? 1 : 0);+ [self after:27 do:^{+ CHECK(SGSingHeldLead() > 2, @"after the seek the lead is built again (%.2f s)", SGSingHeldLead());+ [self checkPosition:@"seven seconds after the seek, with no report since,"];+ // At As sung with Spatial voice off and no page reading the lines, Sing rests.+ SGSetSingLevel(1);+ }];+ [self after:29 do:^{+ CHECK(SGSingCurrentState() == SGSingStateWaiting && [SGSingStatusText() isEqualToString:@"As sung"] && SGSingHeldLead() > 2,+ @"resting at As sung, Sing reads %@ and plays on the %.2f s it holds as they are", SGSingStatusText(), SGSingHeldLead());+ [self checkPosition:@"resting,"];+ SGSetSingLevel(0);+ }];+ [self after:31 do:^{+ CHECK(SGSingHeldLead() > 2, @"separating again, the lead was kept (%.2f s)", SGSingHeldLead());+ [self checkPosition:@"separating again,"];+ setThermal(NSProcessInfoThermalStateSerious);+ }];+ [self after:32 do:^{+ CHECK(SGSingCurrentState() == SGSingStateHot && SGSingHeldLead() > 2 && !SGSingLoaderSeparator(),+ @"at the thermal state Serious Sing is held (%@), the model let go and the %.2f s held played on", SGSingStatusText(), SGSingHeldLead());+ [self checkPosition:@"held at Serious,"];+ setThermal(NSProcessInfoThermalStateFair);+ CFTimeInterval fair = CACurrentMediaTime();+ [self waitFor:^BOOL { return SGSingLoaderSeparator() || CACurrentMediaTime() - fair > 40; } every:0.25 then:^{+ CHECK(SGSingCurrentState() != SGSingStateHot && SGSingLoaderSeparator() && SGSingHeldLead() > 2,+ @"at Fair it loads the model again in %.1f s (%@), the lead kept (%.2f s)", CACurrentMediaTime() - fair, SGSingStatusText(), SGSingHeldLead());+ [self checkPosition:@"running again,"];+ // A memory warning stops Sing with the mic still on.+ [NSNotificationCenter.defaultCenter postNotificationName:UIApplicationDidReceiveMemoryWarningNotification object:UIApplication.sharedApplication];+ [self after:1 do:^{+ CHECK(SGSingOn() && [SGSingStatusText() isEqualToString:@"Stopped"] && SGSingHeldLead() > 2,+ @"stopped by a memory warning, Sing reads %@ and plays on the %.2f s it holds", SGSingStatusText(), SGSingHeldLead());+ [self checkPosition:@"stopped,"];+ [self pauseThenResume];+ }];+ }];+ }];+}++// Spotify pauses (its output stops and it reports), Sing seeks it back to what was heard, which drops the lead, and+// Spotify resumes from there: nothing heard is lost.+- (void)pauseThenResume {+ double heard = [self heardInMix], lead = SGSingHeldLead();+ AudioOutputUnitStop(_output);+ _paused = YES;+ [self playerReports];+ [sg_watcher playerStateDidChange:_state];+ [self playerReports]; // Spotify's report of the seek+ double seekedTo = atomic_load(&sg_readAt) / kRate;+ CHECK(fabs(seekedTo - heard) < 0.15 && SGSingHeldLead() == 0, @"paused, Spotify is seeked back from %.2f s to %.2f s, what was heard (%.2f s), and the %.2f s held "+ @"are let go (%.2f s held)", heard + lead, seekedTo, heard, lead, SGSingHeldLead());+ [self checkPosition:@"paused,"];+ _paused = NO;+ AudioOutputUnitStart(_output);+ [self playerReports];+ CFTimeInterval resumed = CACurrentMediaTime();+ [self after:2 do:^{+ double since = CACurrentMediaTime() - resumed;+ CHECK(SGSingHeldLead() < 0.001 && fabs([self heardInMix] - (heard + since)) < 0.2,+ @"resumed, the song plays on from %.2f s, where it was heard, nothing skipped: %.2f s heard %.2f s on (%.2f s held)", heard, [self heardInMix], since,+ SGSingHeldLead());+ [self checkPosition:@"resumed,"];+ SGSetSingOn(NO);+ [self after:1 do:^{+ CHECK(SGSingHeldLead() < 0.001 && SGSingCurrentState() == SGSingStateOff, @"switched off, nothing is held (%.3f s) and Sing is %@",+ SGSingHeldLead(), SGSingStatusText());+ [self score];+ NSLog(@"[harness] %@", sg_failures ? @"FAILED" : @"all passed");+ exit(sg_failures ? 1 : 0);+ }];}];}@@ -371,6 +523,12 @@ static void check(OSStatus status, const char *what) {self.window = [[UIWindow alloc] initWithFrame:UIScreen.mainScreen.bounds];self.window.rootViewController = [UIViewController new];sg_seekResult = [NSObject new];+ sg_player = [SPTEsperantoPlayer new];+ NSString *clock = [NSUserDefaults.standardUserDefaults stringForKey:@"clock"];+ sg_clock = [clock isEqualToString:@"restamp"] ? SGHarnessClockRestamp : [clock isEqualToString:@"decoded"] ? SGHarnessClockDecoded : SGHarnessClockReport;+ NSLog(@"[harness] Spotify's clock: %@", @[@"its last report run on", @"its last report stamped again at every read", @"reported again every 0.5 s with what was decoded"][sg_clock]);+ // On the class processInfo really is, which may override NSProcessInfo's.+ class_replaceMethod(object_getClass(NSProcessInfo.processInfo), @selector(thermalState), (IMP)thermalState, "q16@0:8");makeMix();NSLog(@"[harness] Sing: %@ (%@)", SGSingStatusText(), SGSingMissing() ?: @"nothing missing");CFTimeInterval began = CACurrentMediaTime();diff --git a/harness/spatial-page/README.md b/harness/spatial-page/README.mdindex 0567080c..262438bf 100644--- a/harness/spatial-page/README.md+++ b/harness/spatial-page/README.md@@ -13,10 +13,30 @@ second while the preview listens; it also answers Motion & Fitness's permissionOther agents use the simulator too: make a device of your own (`xcrun simctl create`) and address it by UDID.`main.m` lists the setup words (`on`, `allowed`, `denied`, `head=sweep`, `head=<degrees>`, `width=<points>`,`slow`) and the actions, played one every 0.7 s from 1 s in: `spatial` opens the page, `toggle`, `pop`, and `dump`,-which logs the rows, the header's size and what VoiceOver reads on the preview. The stub logs each listener-coming and going:+which logs the rows, the header's size and what VoiceOver reads on the preview, and `lag`, which reads the disc on+screen every frame for 3 s against where the head's script has the voice then and logs how far behind it is. The+stub logs each listener coming and going:xcrun simctl launch --console-pty <udid> com.vojta.spatialpageharness on allowed head=sweep spatial dump pop dump+ xcrun simctl launch --console-pty <udid> com.vojta.spatialpageharness on allowed head=sweep spatial wait wait wait lag++2026-10-06, `lag` with head=sweep (30 degrees a second on average): 4.6 degrees, 155 ms, behind when each motion was+animated to over a fixed 0.12 s; 0.5 degrees, 16 to 18 ms, once each is animated over the motions' interval to+where the head will be by the next. With head=60 (150 degrees a second at its fastest) the gap is 4.4 at most and+0.4 on average.++The Sing page's card: `state=7 playing` has Sing on and a made-up song's levels traced; `chunk=<ms>` moves the+song's tenths on in renders that long, a part of a tenth out of step with the wall clock, as the engine's own do;+`busy` holds the main thread up to 30 ms at a time; `none` is no track and none ever played, `last` no track with+Holocene the last one played, `title=<text>` the playing track's title. `lines` reads the lines every frame for+5 s and logs the frames where they jumped back or ahead:++ xcrun simctl launch --console-pty <udid> com.vojta.spatialpageharness chunk=85 state=7 playing wait wait lines++2026-10-06, `lines` with chunk=85: the card that slid the lines one tenth on every wall-clock tenth jumped 8 times+back and 11 ahead in 5 s (49 of its 200 draws over 20 s slid a tenth when the levels had not, or not when they had);+drawn as the levels move on and glided from where they are on screen, 0 back and 2 ahead (a catch-up past two+tenths). With chunk=0 both are smooth, and so is the old card under `busy`: its timer stayed in step.Reduce Motion: `xcrun simctl spawn <udid> defaults write com.apple.Accessibility ReduceMotionEnabled -bool true`,then launch again (and `false` after).diff --git a/harness/spatial-page/build.sh b/harness/spatial-page/build.shindex 2f1f7fc4..75a7e329 100755--- a/harness/spatial-page/build.sh+++ b/harness/spatial-page/build.sh@@ -10,7 +10,8 @@ SDK=$(xcrun --sdk iphonesimulator --show-sdk-path)xcrun -sdk iphonesimulator clang -target arm64-apple-ios17.0-simulator -fobjc-arc -g -O0 \-I"$SRC" -isysroot "$SDK" -Wall -Werror -Wno-deprecated-declarations \"$(dirname "$0")/main.m" "$(dirname "$0")/stubs.m" "$(dirname "$0")/../scene.m" \- "$SRC"/Shared/Sing/SingSettings.m "$SRC"/Shared/Sing/SGSpatialPreview.m \+ "$SRC"/Shared/Sing/SingSettings.m "$SRC"/Shared/Sing/SGSingCard.m "$SRC"/Shared/Sing/SGSpatialPreview.m "$SRC"/Core/SGGlass.m \+ "$SRC"/Shared/Player/SGLastTrack.m "$SRC"/Settings/SGMarquee.m \"$SRC"/Settings/SGPage.m "$SRC"/Settings/SGPageStyle.m "$SRC"/Settings/SGModPage.m "$SRC"/Settings/SGGlowSwitch.m \"$SRC"/Core/SGLog.m "$SRC"/Core/SGPrefs.m "$SRC"/Core/SGViewTree.m "$SRC"/Core/SGFlagForce.m "$SRC"/Core/SGUIMode.m \-framework UIKit -framework QuartzCore -framework CoreGraphics -framework CoreMotion -framework Foundation \diff --git a/harness/spatial-page/main.m b/harness/spatial-page/main.mindex bfc15613..f006900e 100644--- a/harness/spatial-page/main.m+++ b/harness/spatial-page/main.m@@ -10,21 +10,112 @@// turning 45 degrees each way every 6 s), head=<degrees> (still, then turned that far left at 2 s and held), width=<points>// (the window that narrow, centred), slow (animations at a tenth of their speed).// Actions: spatial (the Spatial voice row tapped), toggle=<section>.<row> (that row's switch flipped the way a tap-// does), pop, dump (the rows, the header's height and what VoiceOver reads on the preview, to the log).+// does), pop, dump (the rows, the header's height and what VoiceOver reads on the preview, to the log), lag (how far+// the disc on screen trails the head over the next 3 s, to the log), lines (how the Sing card's lines move over the+// next 5 s: the frames where they jump back or ahead, to the log).+// The Sing page's card: state=<SGSingState number> (Sing on for the states that are on), level=<0 to 2>, playing,+// paused=<MB> (no model, a stopped download that kept that much), title=<text> (the track's title, _ for a space), none (no track+// and none ever played), last (no track, Holocene the last one played), chunk=<ms> (the engine's tenths moving on in+// renders this long), busy (the main thread held up to 30 ms at a time).#import <CoreMotion/CoreMotion.h>+#import <objc/runtime.h>#import <UIKit/UIKit.h>#import "Core/SGCore.h"#import "Settings/SGModPage.h"+#import "Shared/Player/SGLastTrack.h"#import "Shared/Sing/Sing.h"extern CMAuthorizationStatus sg_motionAllowed;+extern SGSingState sg_singState;+extern long long sg_pausedBytes;+extern BOOL sg_playing;extern double (^sg_headYaw)(double seconds);+extern CFTimeInterval sg_headStarted;+extern double sg_renderChunk;+extern NSString *sg_trackTitle;+extern long sg_newestTenth;+extern BOOL sg_noTrack;+extern double sg_sentYaw;static void findViews(UIView *root, Class kind, NSMutableArray *found) {if ([root isKindOfClass:kind]) [found addObject:root];for (UIView *sub in root.subviews) findViews(sub, kind, found);}+@interface LagProbe : NSObject+@property (nonatomic, weak) UIView *preview;+@end++@implementation LagProbe {+ double _behind, _off, _most, _speed;+ int _frames;+}+// The voice's true angle now is the one in the last motion the preview heard, moved on by as far as the head has+// turned since that motion was sent (the preview's ivars, read through KVC: _heard, or _angle before it had one).+// The drawn one is the disc layer's on screen. Behind counts the gap the way the head is turning.+- (void)tick:(CADisplayLink *)link {+ UIView *preview = self.preview;+ BOOL hears = class_getInstanceVariable(preview.class, "_heard") != NULL;+ double heard = [[preview valueForKey:hears ? @"heard" : @"angle"] doubleValue];+ CALayer *field = [preview valueForKey:@"field"];+ NSNumber *shown = [field.presentationLayer valueForKeyPath:@"transform.rotation.z"];+ double t = CACurrentMediaTime() - sg_headStarted;+ if (!shown || !sg_headStarted) return;+ double truth = heard + sg_headYaw(t) - sg_sentYaw, turning = (sg_headYaw(t + 0.001) - sg_headYaw(t)) / 0.001;+ double gap = remainder(truth - shown.doubleValue, 2 * M_PI) * 180 / M_PI;+ _behind += turning >= 0 ? gap : -gap;+ _off += fabs(gap);+ _most = fmax(_most, fabs(gap));+ _speed += fabs(turning) * 180 / M_PI;+ _frames++;+}+- (void)report {+ double behind = _frames ? _behind / _frames : 0, off = _frames ? _off / _frames : 0, speed = _frames ? _speed / _frames : 0;+ NSLog(@"[harness] lag over %d frames: %.2f deg behind the head on average (%.2f off either way, %.2f at most), the head at "+ @"%.0f deg/s, so %.0f ms behind", _frames, behind, off, _most, speed, speed > 0 ? 1000 * behind / speed : 0);+}+@end++// The Sing card's lines read every frame: where one tenth of the song is on screen (its place in the levels drawn,+// from the newest tenth the stub handed out at the card's last read, and the line's slide on screen), and how it+// moved from the frame before: back to the right, or on by more than twice the pace, is a jump.+@interface LinesProbe : NSObject+@property (nonatomic, weak) UIView *card;+@end++@implementation LinesProbe {+ long _tenth;+ double _lastX, _lastTime, _travelled, _time;+ int _frames, _back, _ahead, _resting;+}+- (void)tick:(CADisplayLink *)link {+ // What the card drew in this turn of the run loop, committed, so the slide on screen goes with the levels read.+ [CATransaction flush];+ UIView *graph = [self.card valueForKey:@"graph"];+ CALayer *line = [self.card valueForKey:@"vocals"];+ NSNumber *slide = [line.presentationLayer valueForKeyPath:@"transform.translation.x"];+ CGFloat step = graph.bounds.size.width / 47;+ if (!_tenth) _tenth = sg_newestTenth - 10;+ double x = (_tenth - sg_newestTenth + 48) * step + slide.doubleValue, now = CACurrentMediaTime();+ if (_lastTime > 0) {+ double dx = x - _lastX, pace = step * (now - _lastTime) / 0.1;+ if (dx > 0.5) _back++;+ else if (-dx > 2 * pace + 0.5) _ahead++;+ else if (fabs(dx) < 0.05) _resting++;+ _travelled -= dx;+ _time += now - _lastTime;+ _frames++;+ }+ _lastX = x;+ _lastTime = now;+}+- (void)report {+ CGFloat step = ((UIView *)[self.card valueForKey:@"graph"]).bounds.size.width / 47;+ NSLog(@"[harness] lines over %d frames: %d jumped back, %d jumped ahead, %d resting, at %.2f tenths a tenth of a second",+ _frames, _back, _ahead, _resting, _time > 0 ? _travelled / step / (_time / 0.1) : 0);+}+@end+@interface AppDelegate : UIResponder <UIApplicationDelegate>@property (nonatomic, strong) UIWindow *window;@property (nonatomic, strong) UINavigationController *nav;@@ -54,7 +145,27 @@ static void findViews(UIView *root, Class kind, NSMutableArray *found) {return turn * s * s * (3 - 2 * s);};} else if ([arg hasPrefix:@"width="]) width = [arg substringFromIndex:6].doubleValue;- else if (![@[@"keep", @"slow", @"on"] containsObject:arg]) [actions addObject:arg];+ else if ([arg hasPrefix:@"state="]) {+ sg_singState = (SGSingState)[arg substringFromIndex:6].integerValue;+ SGSetEnabled(SGKeySing, sg_singState > SGSingStateOff);+ } else if ([arg hasPrefix:@"level="]) SGSetSingLevel([arg substringFromIndex:6].floatValue);+ else if ([arg hasPrefix:@"paused="]) {+ sg_singState = SGSingStateNoModel;+ sg_pausedBytes = [arg substringFromIndex:7].longLongValue * 1000 * 1000;+ } else if ([arg isEqualToString:@"playing"]) sg_playing = YES;+ else if ([arg hasPrefix:@"chunk="]) sg_renderChunk = [arg substringFromIndex:6].doubleValue / 1000;+ else if ([arg hasPrefix:@"title="]) sg_trackTitle = [[arg substringFromIndex:6] stringByReplacingOccurrencesOfString:@"_" withString:@" "];+ else if ([arg isEqualToString:@"none"] || [arg isEqualToString:@"last"]) {+ sg_noTrack = YES;+ if ([arg isEqualToString:@"none"]) [store removeObjectForKey:SGKeyLastTrack];+ else [store setObject:@{@"uri": @"spotify:track:last", @"title": @"Holocene", @"artist": @"Bon Iver"} forKey:SGKeyLastTrack];+ }+ else if (![@[@"keep", @"slow", @"on", @"busy"] containsObject:arg]) [actions addObject:arg];+ }+ // busy: the main thread held up to 30 ms at a time, about every 50 ms, as Spotify's own work holds it.+ if ([args containsObject:@"busy"]) {+ NSTimer *busy = [NSTimer timerWithTimeInterval:0.05 repeats:YES block:^(NSTimer *timer) { usleep(arc4random_uniform(30000)); }];+ [NSRunLoop.mainRunLoop addTimer:busy forMode:NSRunLoopCommonModes];}CGRect screen = UIScreen.mainScreen.bounds;@@ -83,7 +194,7 @@ static void findViews(UIView *root, Class kind, NSMutableArray *found) {NSString *verb = parts.firstObject, *value = parts.count > 1 ? parts[1] : @"";UITableView *table = self.table;if ([verb isEqualToString:@"spatial"]) {- [table.delegate tableView:table didSelectRowAtIndexPath:[NSIndexPath indexPathForRow:3 inSection:0]];+ [table.delegate tableView:table didSelectRowAtIndexPath:[NSIndexPath indexPathForRow:0 inSection:1]];} else if ([verb isEqualToString:@"toggle"]) {NSArray<NSString *> *at = [value componentsSeparatedByString:@"."];UITableViewCell *cell = [table cellForRowAtIndexPath:[NSIndexPath indexPathForRow:at[1].integerValue inSection:at[0].integerValue]];@@ -92,6 +203,33 @@ static void findViews(UIView *root, Class kind, NSMutableArray *found) {UISwitch *toggle = switches.firstObject;[toggle setOn:!toggle.on animated:YES];[toggle sendActionsForControlEvents:UIControlEventValueChanged];+ } else if ([verb isEqualToString:@"lag"]) {+ // How far the disc drawn on screen trails the head's last angle, read every frame for 3 s: the mean and the+ // largest gap in degrees, and the mean as time at the head's speed then.+ NSMutableArray<UIView *> *previews = [NSMutableArray array];+ findViews(self.window, SGSpatialPreview.class, previews);+ if (!previews.count) return;+ LagProbe *probe = [LagProbe new];+ probe.preview = previews.firstObject;+ CADisplayLink *link = [CADisplayLink displayLinkWithTarget:probe selector:@selector(tick:)];+ link.preferredFrameRateRange = CAFrameRateRangeMake(60, 120, 120);+ [link addToRunLoop:NSRunLoop.mainRunLoop forMode:NSRunLoopCommonModes];+ dispatch_after(dispatch_time(DISPATCH_TIME_NOW, 3 * NSEC_PER_SEC), dispatch_get_main_queue(), ^{+ [link invalidate];+ [probe report];+ });+ } else if ([verb isEqualToString:@"lines"]) {+ NSMutableArray<UIView *> *cards = [NSMutableArray array];+ findViews(self.window, NSClassFromString(@"SGSingCard"), cards);+ if (!cards.count) return;+ LinesProbe *probe = [LinesProbe new];+ probe.card = cards.firstObject;+ CADisplayLink *link = [CADisplayLink displayLinkWithTarget:probe selector:@selector(tick:)];+ [link addToRunLoop:NSRunLoop.mainRunLoop forMode:NSRunLoopCommonModes];+ dispatch_after(dispatch_time(DISPATCH_TIME_NOW, 5 * NSEC_PER_SEC), dispatch_get_main_queue(), ^{+ [link invalidate];+ [probe report];+ });} else if ([verb isEqualToString:@"pop"]) {[self.nav popViewControllerAnimated:YES];} else if ([verb isEqualToString:@"dump"]) {diff --git a/harness/spatial-page/stubs.m b/harness/spatial-page/stubs.mindex ce54f13a..1104793c 100644--- a/harness/spatial-page/stubs.m+++ b/harness/spatial-page/stubs.m@@ -8,14 +8,31 @@#pragma mark - Sing+// From main.m's launch line: the state Sing is in, the model's, and whether the song plays.+SGSingState sg_singState = SGSingStateOff;+long long sg_pausedBytes;+BOOL sg_playing;+static float sg_level = 0.15f;+NSString *const SGSingChangedNotification = @"SGSingChangedNotification";-NSString *SGSingStatusText(void) { return SGSingOn() ? @"Singing" : @"Off"; }-NSString *SGSingStatusDetail(void) { return nil; }+SGSingState SGSingCurrentState(void) { return SGSingOn() || sg_singState <= SGSingStateDownloading ? sg_singState : SGSingStateOff; }+NSString *SGSingStatusText(void) {+ return @[@"Unavailable", @"Needs its voice model", @"Downloading 42%", @"Off", @"Loading the voice model, 12 s", @"Ready", @"Listening ahead", @"On",+ @"Too slow", @"Held, too hot", @"Failed"][SGSingCurrentState()];+}+NSString *SGSingStatusDetail(void) {+ SGSingState state = SGSingCurrentState();+ return state == SGSingStateFailed ? @"The voice model did not load in 120 s, so that load was given up. Switch Sing off and on again to try again."+ : state == SGSingStateSinging ? @"Sing runs on the GPU while Spotify is open, and on the CPU in the background." : nil;+}NSString *SGSingMissing(void) { return nil; }BOOL SGSingOn(void) { return SGHidden(SGKeySing); }void SGSetSingOn(BOOL on) { SGSetEnabled(SGKeySing, on); }-float SGSingLevel(void) { return 0.15f; }-void SGSetSingLevel(float level) {}+float SGSingLevel(void) { return sg_level; }+void SGSetSingLevel(float level) {+ sg_level = level;+ NSLog(@"[harness] level %.2f", level);+}void SGSetSingIgnoresHeat(BOOL ignores) {}BOOL SGSingSpatialAvailable(void) { return YES; }BOOL SGSingSpatial(void) { return SGHidden(SGKeySingSpatial); }@@ -23,19 +40,107 @@ void SGSetSingSpatial(BOOL on) {SGSetEnabled(SGKeySingSpatial, on);NSLog(@"[harness] spatial voice %@", on ? @"on" : @"off");}+// From main.m's launch line (chunk=<ms>): what plays moves on in renders this long, as the engine's clock does, a+// part of a tenth out of step with the wall clock; 0 for a clock that moves smoothly.+double sg_renderChunk;+// The newest tenth the last read handed out, for main.m's `lines`.+long sg_newestTenth;+// A made-up song: a voice that comes in phrases over a steadier band, a tenth of a second at a time.+BOOL SGSingReadLevels(float *vocals, float *rest, int count) {+ if (SGSingCurrentState() != SGSingStateSinging) return NO;+ double played = CACurrentMediaTime() + (sg_renderChunk > 0 ? 0.037 : 0);+ if (sg_renderChunk > 0) played = floor(played / sg_renderChunk) * sg_renderChunk;+ long now = (long)(played * 10);+ sg_newestTenth = now;+ float level = sg_level, vocalsGain = fminf(level, 1), restGain = level <= 1 ? 1 : 2 - level;+ for (int k = 0; k < count; k++) {+ double t = (now - count + 1 + k) / 10.0;+ double phrase = fmax(0, sin(t * 0.9)) * (0.7 + 0.3 * sin(t * 7.3));+ vocals[k] = (float)(0.2 * phrase + 0.004) * vocalsGain;+ rest[k] = (float)(0.07 * (1 + 0.35 * sin(t * 2.1) + 0.2 * fabs(sin(t * 12.6)))) * restGain;+ }+ return YES;+}void SGSingComputeUnitsChanged(void) {}-SGSingModelState SGSingModelCurrentState(void) { return SGSingModelReady; }-double SGSingModelProgress(void) { return 1; }+SGSingModelState SGSingModelCurrentState(void) {+ return sg_singState == SGSingStateDownloading ? SGSingModelDownloading : sg_singState == SGSingStateNoModel ? SGSingModelMissing : SGSingModelReady;+}+double SGSingModelProgress(void) { return 0.42; }BOOL SGSingModelWaitingForNetwork(void) { return NO; }+BOOL SGSingModelChecking(void) { return NO; }+BOOL SGSingModelOverCellular(void) { return NO; }+void SGSingDownloadModelOverCellular(void) {}+long long SGSingModelPausedBytes(void) { return SGSingModelCurrentState() == SGSingModelMissing ? sg_pausedBytes : 0; }NSString *SGSingModelError(void) { return nil; }-NSString *SGSingModelSizeText(void) { return @"190 MB"; }+NSString *SGSingModelSizeText(void) { return @"489 MB"; }void SGSingDownloadModel(void) {}void SGSingCancelModelDownload(void) {}void SGSingDeleteModel(void) {}-NSArray<NSString *> *SGSingComputeUnitNames(void) { return @[@"GPU and Neural Engine", @"GPU", @"Neural Engine"]; }+NSArray<NSString *> *SGSingComputeUnitNames(void) { return @[@"Automatic", @"CPU only", @"GPU", @"Neural Engine", @"GPU and Neural Engine"]; }BOOL SGSingOSSupported(void) { return YES; }BOOL SGSingDeviceSupported(void) { return YES; }+#pragma mark - the player++@interface FakeTrack : NSObject+@property (nonatomic, copy) NSString *trackTitle, *artistName;+@property (nonatomic, strong) id URI;+@end+@implementation FakeTrack+@end++// From main.m's launch line (title=<text>): the playing track's title; and (none) no track at all, as before+// anything has played this launch.+NSString *sg_trackTitle = @"Northern Lights";+BOOL sg_noTrack;++NSString *SGURIString(id uri) { return [uri isKindOfClass:NSString.class] ? uri : nil; }+void SGAddPlayerStateObserver(id observer) {}+NSString *SGLocalFileCoverInURL(NSString *url) { return nil; }+NSURL *SGMotionCanvasIn(NSDictionary *metadata) { return nil; }++@interface FakePlayerState : NSObject+@property (nonatomic, strong) FakeTrack *track;+@property (nonatomic) BOOL isPaused;+@end+@implementation FakePlayerState+@end++@interface FakePlayer : NSObject+@end+@implementation FakePlayer+- (id)pause:(id)options {+ sg_playing = NO;+ NSLog(@"[harness] pause");+ return nil;+}+- (id)resume:(id)options {+ sg_playing = YES;+ NSLog(@"[harness] resume");+ return nil;+}+@end++id SGPlayerState(void) {+ static FakePlayerState *state;+ if (sg_noTrack) return nil;+ if (!state) {+ state = [FakePlayerState new];+ state.track = [FakeTrack new];+ state.track.trackTitle = sg_trackTitle;+ state.track.artistName = @"The Paper Kites";+ state.track.URI = @"spotify:track:harness";+ }+ state.isPaused = !sg_playing;+ return state;+}++id SGKaraokePlayer(void) {+ static FakePlayer *player;+ if (!player) player = [FakePlayer new];+ return player;+}+#pragma mark - the head@interface FakeAttitude : NSObject@@ -65,6 +170,9 @@ __attribute__((constructor)) static void sg_takeOverPermission(void) {static NSMutableDictionary<NSString *, void (^)(CMDeviceMotion *)> *sg_listeners;static dispatch_queue_t sg_queue;static dispatch_source_t sg_timer;+// When the head's script started, and the yaw in the last motion sent, for main.m's `lag`.+CFTimeInterval sg_headStarted;+double sg_sentYaw;void SGHeadMotionListen(NSString *name, void (^handler)(CMDeviceMotion *motion)) {if (!sg_listeners) {@@ -77,13 +185,13 @@ void SGHeadMotionListen(NSString *name, void (^handler)(CMDeviceMotion *motion))if (old && !handler) dispatch_async(sg_queue, ^{ old(nil); });NSArray *handlers = sg_listeners.allValues;if (handlers.count && !sg_timer && sg_headYaw) {- static CFTimeInterval launched;- if (!launched) launched = CACurrentMediaTime();+ if (!sg_headStarted) sg_headStarted = CACurrentMediaTime();sg_timer = dispatch_source_create(DISPATCH_SOURCE_TYPE_TIMER, 0, 0, sg_queue);dispatch_source_set_timer(sg_timer, DISPATCH_TIME_NOW, NSEC_PER_SEC / 25, 0);dispatch_source_set_event_handler(sg_timer, ^{- double now = CACurrentMediaTime() - launched, yaw = sg_headYaw(now);+ double now = CACurrentMediaTime() - sg_headStarted, yaw = sg_headYaw(now);if (isnan(yaw)) return;+ sg_sentYaw = yaw;FakeMotion *motion = [FakeMotion new];motion.timestamp = CACurrentMediaTime();motion.attitude = [FakeAttitude new];diff --git a/tweak/Sources/Headers/SPTPlayer.h b/tweak/Sources/Headers/SPTPlayer.hindex 5440080e..58712a1e 100644--- a/tweak/Sources/Headers/SPTPlayer.h+++ b/tweak/Sources/Headers/SPTPlayer.h@@ -27,9 +27,12 @@@property (nonatomic, readonly) BOOL isPaused;@property (nonatomic, readonly) BOOL isPlaying;@property (nonatomic, readonly) BOOL isLoading;-// Seconds. position runs on from positionAsOfTimestamp by the time elapsed since the state was made.+// Seconds. position runs on from positionAsOfTimestamp by the time elapsed since timestamp, when the player reported.@property (nonatomic, readonly) double position;@property (nonatomic, readonly) double positionAsOfTimestamp;+@property (nonatomic, readonly) NSDate *timestamp;+// What position runs on at: 1, or Speed and pitch's speed (Shared/Player/SpeedPitch.x hooks it).+@property (nonatomic, readonly) double playbackSpeed;// The track's length, in the same unit as position.@property (nonatomic, readonly) double duration;// The tracks to come and the ones played, nearest first; SPTPlayerTrack each, read with a type check.diff --git a/tweak/Sources/Redesigned/Lyrics/SGRKaraokeView.m b/tweak/Sources/Redesigned/Lyrics/SGRKaraokeView.mindex deed49ad..2e86a78c 100644--- a/tweak/Sources/Redesigned/Lyrics/SGRKaraokeView.m+++ b/tweak/Sources/Redesigned/Lyrics/SGRKaraokeView.m@@ -1384,6 +1384,7 @@ typedef struct {if (_plain) return; // a line with no time has nowhere to seek tofor (SGRKaraokeLineView *view in _shown.allValues) {if (!CGRectContainsPoint(CGRectInset(view.frame, -_margin, -_lineGap / 2), point)) continue;+ SGLog(@"lyrics: tapped the line at %ld-%ld ms, the clock at %.0f ms", (long)view.line.start, (long)view.line.end, _clock);SGKaraokeSeek(view.line.start);SGPlayFeedback(SGFeedbackSkip);[self glideTo:view.line.start];diff --git a/tweak/Sources/Redesigned/Lyrics/SGRSingButton.h b/tweak/Sources/Redesigned/Lyrics/SGRSingButton.hindex 2a6d975f..36709e20 100644--- a/tweak/Sources/Redesigned/Lyrics/SGRSingButton.h+++ b/tweak/Sources/Redesigned/Lyrics/SGRSingButton.h@@ -1,7 +1,7 @@// Sing's mic on the redesign's lyrics (Shared/Sing/Sing.h), in the bottom trailing corner of SGRKaraokeView// across from the pronunciation and translation button, as Apple Music has it. It shows what Sing is doing-// on itself: the mic struck through where Sing cannot run, a ring filling with the voice model's download, a-// ring turning while the model prepares and while Sing listens ahead, the mic lit in the accent colour while+// on itself: the mic struck through where Sing cannot run, a ring filling with the voice model's download (held+// where it is under a dimmed wifi struck through while the download waits for the network), a ring turning while the model prepares and while Sing listens ahead, the mic lit in the accent colour while// the vocals are down, a thermometer while the heat holds it, a warning when it failed.//// A tap turns Sing on and off, or says what is missing and offers the model's download. Holding it brings updiff --git a/tweak/Sources/Redesigned/Lyrics/SGRSingButton.m b/tweak/Sources/Redesigned/Lyrics/SGRSingButton.mindex 725bbe80..fa1eebee 100644--- a/tweak/Sources/Redesigned/Lyrics/SGRSingButton.m+++ b/tweak/Sources/Redesigned/Lyrics/SGRSingButton.m@@ -110,7 +110,17 @@ static char kButtonGlassKey, kPanelGlassKey;switch (state) {case SGSingStateUnavailable: glyph = @"mic.slash"; alpha = 0.5; break;case SGSingStateNoModel: glyph = @"mic"; alpha = 0.7; break;- case SGSingStateDownloading: glyph = @"arrow.down"; progress = SGSingModelProgress(); break;+ case SGSingStateDownloading:+ // Offline, the ring stays where the download stopped and the glyph says why, dimmed like the other+ // states that wait.+ if (SGSingModelWaitingForNetwork()) {+ glyph = @"wifi.slash";+ alpha = 0.7;+ } else {+ glyph = @"arrow.down";+ }+ progress = SGSingModelProgress();+ break;case SGSingStateOff: glyph = @"mic"; break;case SGSingStatePreparing:case SGSingStateBuffering: progress = 2; break;diff --git a/tweak/Sources/Shared/Lyrics/KaraokeSource.x b/tweak/Sources/Shared/Lyrics/KaraokeSource.xindex a014855e..54bbee85 100644--- a/tweak/Sources/Shared/Lyrics/KaraokeSource.x+++ b/tweak/Sources/Shared/Lyrics/KaraokeSource.x@@ -11,6 +11,7 @@#import "Shared/LockScreenLyrics/LockScreenLyrics.h"#import "Shared/AnimatedArtwork/AnimatedArtwork.h"#import "Shared/LyricsSources/LyricsSources.h"+#import "Shared/Sing/Sing.h"#import "Headers/SPTPlayer.h"#import <mach-o/dyld.h>#import <objc/runtime.h>@@ -400,8 +401,22 @@ NSInteger SGKaraokePositionMs(void) {void SGKaraokeSeek(NSInteger ms) {id player = sg_player;- if (![player respondsToSelector:@selector(seekTo:)]) return;+ if (![player respondsToSelector:@selector(seekTo:)]) {+ SGLog(@"seek: to %ld ms, but no player to seek", (long)ms);+ return;+ }+ // Where it was and where it is half a second on, with Karaoke's lead and the correction it takes off, so the log+ // says whether a seek lands where it was sent.+ SPTPlayerState *state = playerState();+ NSInteger before = SGKaraokePositionMs();+ NSString *lead = [NSString stringWithFormat:@"Karaoke holds %.2f s, takes %.2f s off", SGSingHeldLead(), state ? SGSingLeadOf(state) : 0];[(id<SPTPlayer>)player seekTo:ms / 1000.0];+ SGLog(@"seek: to %ld ms (sent %.3f s to %@) from %ld ms; %@", (long)ms, ms / 1000.0, [player class], (long)before, lead);+ dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(0.5 * NSEC_PER_SEC)), dispatch_get_main_queue(), ^{+ SPTPlayerState *now = playerState();+ SGLog(@"seek: half a second after the seek to %ld ms the position reads %ld ms (%ld ms off); Karaoke holds %.2f s, takes %.2f s off", (long)ms,+ (long)SGKaraokePositionMs(), (long)(SGKaraokePositionMs() - ms - 500), SGSingHeldLead(), now ? SGSingLeadOf(now) : 0);+ });}// A Spotify track by its base62 id, a local file by its URI (Shared/LocalFiles/LocalFiles.h).diff --git a/tweak/Sources/Shared/Sing/SGSingCard.m b/tweak/Sources/Shared/Sing/SGSingCard.mnew file mode 100644index 00000000..d136ac3b--- /dev/null+++ b/tweak/Sources/Shared/Sing/SGSingCard.m@@ -0,0 +1,449 @@+// The top of the Sing page (SingSettings.m): the song Spotify has (else the last one played, else Not Playing,+// Shared/Player/SGLastTrack.h), its title and artist gliding when too long, what Sing is doing in words, two lines+// that trace the loudness of the vocals and of the rest of the song as heard, play and pause, and a tall slider for+// the vocals' level; under the card, three stops for the level.+//+// The lines are read from the engine thirty times a second (SGSingReadLevels: a few atomic reads, measured on its+// worker), in every run loop mode so they move on while the page scrolls. They move as the engine's own tenths+// do, which come with its renders rather than on the wall clock's tenths: each time the tenths move on, the lines+// are drawn again and glide in from where they were on screen at a tenth's width per tenth of a second, on Core+// Animation alone, with no display link; while none come (a pause), they rest. With Sing off, loading or held they+// lie flat. Under Reduce Motion they do not scroll: they are redrawn in place every two seconds.+//+// Threading: main thread.+#import "Core/SGCore.h"+#import "Core/SGGlass.h"+#import "Settings/SGMarquee.h"+#import "Settings/SGPageStyle.h"+#import "Shared/Lyrics/Lyrics.h"+#import "Shared/Player/PlayerState.h"+#import "Shared/Player/SGLastTrack.h"+#import "Sing.h"++enum { kHistory = 48 }; // tenths of a second across the card+static const NSTimeInterval kTick = 0.1;+// How often the lines are read, and the most tenths a glide catches up by before the rest is jumped.+static const NSTimeInterval kPoll = 1.0 / 30;+static const int kMostBehind = 2;+static const float kLevelStep = 0.05f; // the slider snaps to this, as the mic's does+static const CGFloat kCardHeight = 232, kInset = 16, kColumn = 64;++// Loudness as the lines draw it, 0 at -48 dB to 1 at -6 dB.+static CGFloat height(float rms) {+ return rms > 0 ? fmax(0, fmin(1, (20 * log10(rms) + 48) / 42)) : 0;+}++#pragma mark - the level's slider++// From gone at the bottom through as sung at the middle (a mark across it) to the vocals alone at the top.+@interface SGSingLevelColumn : UIControl+@property (nonatomic) float level;+@end++@implementation SGSingLevelColumn {+ UIView *_fill, *_mark;+ UIImageView *_glyph;+ UISelectionFeedbackGenerator *_feedback;+}++- (instancetype)initWithFrame:(CGRect)frame {+ if (!(self = [super initWithFrame:frame])) return nil;+ // Glass over the card's flat grey shows little of itself, so a light wash gives the track its shape.+ UIVisualEffectView *glass = SGGlassFor(self, @selector(initWithFrame:));+ glass.userInteractionEnabled = NO;+ self.backgroundColor = [UIColor colorWithWhite:1 alpha:0.1];+ _fill = [UIView new];+ _fill.backgroundColor = [UIColor colorWithWhite:0.94 alpha:1];+ _fill.userInteractionEnabled = NO;+ [self addSubview:_fill];+ _mark = [UIView new];+ _mark.backgroundColor = [UIColor colorWithWhite:1 alpha:0.35];+ _mark.userInteractionEnabled = NO;+ [self addSubview:_mark];+ _glyph = SGSymbolView(@"music.mic", 20, UIImageSymbolWeightSemibold, 28);+ [self addSubview:_glyph];+ self.clipsToBounds = YES;+ _feedback = [UISelectionFeedbackGenerator new];+ self.isAccessibilityElement = YES;+ self.accessibilityLabel = @"Vocals";+ self.accessibilityTraits = UIAccessibilityTraitAdjustable;+ return self;+}++- (void)layoutSubviews {+ [super layoutSubviews];+ CGSize size = self.bounds.size;+ SGShapeGlass(SGGlassFor(self, @selector(initWithFrame:)), size.width / 2, YES);+ self.layer.cornerRadius = size.width / 2;+ self.layer.cornerCurve = kCACornerCurveContinuous;+ CGFloat filled = size.height * self.level / 2;+ _fill.frame = CGRectMake(0, size.height - filled, size.width, filled);+ _mark.frame = CGRectMake(size.width * 0.3, size.height / 2 - 0.5, size.width * 0.4, 1);+ _mark.backgroundColor = self.level > 1 ? [UIColor colorWithWhite:0 alpha:0.25] : [UIColor colorWithWhite:1 alpha:0.35];+ _glyph.center = CGPointMake(size.width / 2, size.height - size.width / 2);+ // Dark over the fill, light over the glass.+ _glyph.tintColor = filled > size.width / 2 + 12 ? UIColor.blackColor : UIColor.whiteColor;+}++- (void)setLevel:(float)level {+ _level = fmaxf(0, fminf(level, 2));+ self.accessibilityValue = SGSingLevelText(_level);+ [self setNeedsLayout];+}++- (void)setAndSend:(float)level {+ level = roundf(level / kLevelStep) * kLevelStep;+ if (level == _level) return;+ // A tick at each stop the segments below name.+ if (floorf(level) != floorf(_level) || level == 2) [_feedback selectionChanged];+ self.level = level;+ [self sendActionsForControlEvents:UIControlEventValueChanged];+}++- (float)levelAt:(UITouch *)touch {+ CGFloat y = [touch locationInView:self].y;+ return (float)(2 * (1 - y / self.bounds.size.height));+}++- (BOOL)beginTrackingWithTouch:(UITouch *)touch withEvent:(UIEvent *)event {+ [_feedback prepare];+ [self setAndSend:[self levelAt:touch]];+ return YES;+}++- (BOOL)continueTrackingWithTouch:(UITouch *)touch withEvent:(UIEvent *)event {+ [self setAndSend:[self levelAt:touch]];+ return YES;+}++- (void)accessibilityIncrement {+ [self setAndSend:self.level + 2 * kLevelStep];+}++- (void)accessibilityDecrement {+ [self setAndSend:self.level - 2 * kLevelStep];+}++@end++#pragma mark - the card++// The song and Sing's state, read as one; activated, it says more about the state where there is more to say.+@interface SGSingSongElement : UIAccessibilityElement+@property (nonatomic, copy) void (^activate)(void);+@end++@implementation SGSingSongElement+- (BOOL)accessibilityActivate {+ if (self.activate) self.activate();+ return self.activate != nil;+}+@end++@interface SGSingCard : UIView+@end++@implementation SGSingCard {+ UIView *_card;+ SGMarqueeLabel *_title, *_artist;+ UILabel *_state;+ UIView *_graph;+ CAShapeLayer *_rest, *_vocals;+ UIButton *_play;+ SGSingLevelColumn *_column;+ UISegmentedControl *_stops;+ SGSingSongElement *_song;+ NSTimer *_timer;+ float _drawnVocals[kHistory + 1], _drawnRest[kHistory + 1]; // the levels the lines show now+ BOOL _flat;+ int _ticks;+ SPTPlayerTrack *_track; // the player's track the words were last read for+ SGShownTrack *_shown;+}++- (instancetype)initWithFrame:(CGRect)frame {+ if (!(self = [super initWithFrame:frame])) return nil;+ _card = [UIView new];+ _card.backgroundColor = SGCardBackground();+ _card.layer.cornerRadius = 26;+ _card.layer.cornerCurve = kCACornerCurveContinuous;+ _card.clipsToBounds = YES;+ [self addSubview:_card];++ _title = [SGMarqueeLabel new];+ _title.font = [UIFontMetrics.defaultMetrics scaledFontForFont:[UIFont systemFontOfSize:20 weight:UIFontWeightSemibold] maximumPointSize:26];+ _title.textColor = UIColor.whiteColor;+ _artist = [SGMarqueeLabel new];+ _artist.font = [UIFontMetrics.defaultMetrics scaledFontForFont:[UIFont systemFontOfSize:15] maximumPointSize:20];+ _artist.textColor = SGGrey();+ _state = [UILabel new];+ _state.font = [UIFontMetrics.defaultMetrics scaledFontForFont:[UIFont systemFontOfSize:13 weight:UIFontWeightMedium] maximumPointSize:18];+ // The song element reads them out.+ for (UIView *label in @[_title, _artist, _state]) {+ label.isAccessibilityElement = NO;+ [_card addSubview:label];+ }++ _graph = [UIView new];+ _graph.userInteractionEnabled = NO;+ _graph.accessibilityElementsHidden = YES;+ _graph.layer.masksToBounds = YES;+ _rest = [CAShapeLayer layer];+ _rest.strokeColor = [UIColor colorWithWhite:1 alpha:0.38].CGColor;+ _vocals = [CAShapeLayer layer];+ _vocals.strokeColor = SGGreen().CGColor;+ _vocals.lineWidth = 2.5;+ _rest.lineWidth = 1.5;+ for (CAShapeLayer *line in @[_rest, _vocals]) {+ line.fillColor = nil;+ line.lineCap = line.lineJoin = kCALineCapRound;+ [_graph.layer addSublayer:line];+ }+ // The lines come in from nothing at the left and run under the column at the right.+ CAGradientLayer *ends = [CAGradientLayer layer];+ ends.startPoint = CGPointMake(0, 0.5);+ ends.endPoint = CGPointMake(1, 0.5);+ ends.colors = @[(id)UIColor.clearColor.CGColor, (id)UIColor.blackColor.CGColor, (id)UIColor.blackColor.CGColor, (id)UIColor.clearColor.CGColor];+ ends.locations = @[@0, @0.18, @0.92, @1];+ _graph.layer.mask = ends;+ [_card addSubview:_graph];++ _play = [UIButton buttonWithType:UIButtonTypeSystem];+ _play.tintColor = UIColor.whiteColor;+ SGGlassFor(_play, @selector(addSubview:)).userInteractionEnabled = NO;+ _play.backgroundColor = [UIColor colorWithWhite:1 alpha:0.1];+ _play.layer.cornerRadius = 24;+ [_play addTarget:self action:@selector(playTapped) forControlEvents:UIControlEventTouchUpInside];+ [_card addSubview:_play];++ _column = [SGSingLevelColumn new];+ [_column addTarget:self action:@selector(levelMoved) forControlEvents:UIControlEventValueChanged];+ [_card addSubview:_column];++ _stops = [[UISegmentedControl alloc] initWithItems:@[@"Sing along", @"Original", @"Vocals only"]];+ [_stops addTarget:self action:@selector(stopPicked) forControlEvents:UIControlEventValueChanged];+ [self addSubview:_stops];++ _song = [[SGSingSongElement alloc] initWithAccessibilityContainer:_card];+ __weak typeof(self) weakSelf = self;+ _song.activate = ^{ [weakSelf explainState]; };+ _card.accessibilityElements = @[_song, _play, _column];+ UITapGestureRecognizer *tap = [[UITapGestureRecognizer alloc] initWithTarget:self action:@selector(stateTapped:)];+ [_card addGestureRecognizer:tap];+ [self refresh];+ return self;+}++- (CGSize)sizeThatFits:(CGSize)size {+ CGFloat stops = MAX(36, [_stops sizeThatFits:CGSizeMake(size.width, 0)].height);+ CGFloat text = _title.font.lineHeight + _artist.font.lineHeight + _state.font.lineHeight;+ return CGSizeMake(size.width, 12 + MAX(kCardHeight, text + 168) + 12 + stops + 8);+}++- (void)layoutSubviews {+ [super layoutSubviews];+ CGFloat width = self.bounds.size.width, side = 16;+ CGFloat stopsHeight = MAX(36, [_stops sizeThatFits:CGSizeMake(width, 0)].height);+ CGFloat cardHeight = self.bounds.size.height - 12 - 12 - stopsHeight - 8;+ _card.frame = CGRectMake(side, 12, width - 2 * side, cardHeight);+ _stops.frame = CGRectMake(side, CGRectGetMaxY(_card.frame) + 12, width - 2 * side, stopsHeight);+ CGFloat inner = _card.bounds.size.width, textWidth = inner - 2 * kInset - kColumn - 12;+ _column.frame = CGRectMake(inner - kInset - kColumn, kInset, kColumn, cardHeight - 2 * kInset);+ CGFloat y = kInset + 2;+ for (UIView *label in @[_title, _artist, _state]) {+ CGFloat lineHeight = [(UIFont *)[label valueForKey:@"font"] lineHeight];+ label.frame = CGRectMake(kInset + 2, y, textWidth, lineHeight);+ y += lineHeight + (label == _artist ? 6 : 1);+ }+ _play.frame = CGRectMake(kInset, cardHeight - kInset - 48, 48, 48);+ SGShapeGlass(SGGlassFor(_play, @selector(addSubview:)), 24, YES);+ _graph.frame = CGRectMake(0, y + 8, inner - kInset - kColumn - 6, CGRectGetMinY(_play.frame) - y - 16);+ for (CAShapeLayer *line in @[_rest, _vocals]) line.frame = _graph.bounds;+ _graph.layer.mask.frame = _graph.bounds;+ [self drawLines:NO];+}++- (void)didMoveToWindow {+ [super didMoveToWindow];+ [_timer invalidate];+ _timer = nil;+ if (!self.window) return;+ __weak typeof(self) weakSelf = self;+ // In the common modes, so the lines go on while the page is scrolled.+ _timer = [NSTimer timerWithTimeInterval:kPoll repeats:YES block:^(NSTimer *timer) { [weakSelf tick]; }];+ [NSRunLoop.mainRunLoop addTimer:_timer forMode:NSRunLoopCommonModes];+ _track = nil;+ [self refresh];+}++#pragma mark - what it shows++- (void)refresh {+ SPTPlayerState *player = SGPlayerState();+ // The track's words read again only when the player has another track (or none).+ if (!_shown || player.track != _track) {+ _track = player.track;+ _shown = SGShownTrackNow();+ }+ NSString *title = _shown.title ?: @"Not Playing";+ NSString *artist = _shown ? _shown.artist ?: @"" : @"Play a song to hear Sing";+ _title.text = title;+ _artist.text = artist;+ SGSingState state = SGSingCurrentState();+ NSString *words = SGSingStatusText();+ float stored = SGSingLevel();+ if (state == SGSingStateSinging) {+ words = stored > 0.001 && stored < 0.999 ? [NSString stringWithFormat:@"On · vocals at %@", SGSingLevelText(stored)]+ : [NSString stringWithFormat:@"On · %@", SGSingLevelText(stored)];+ }+ if (SGSingStatusDetail()) words = [words stringByAppendingString:@" ⓘ"];+ if (![_state.text isEqualToString:words]) _state.text = words;+ _state.textColor = state == SGSingStateSinging ? SGGreen() : state == SGSingStateFailed || state == SGSingStateHot ? UIColor.systemOrangeColor : SGGrey();++ BOOL playing = player && !player.isPaused;+ UIImage *glyph = [UIImage systemImageNamed:playing ? @"pause.fill" : @"play.fill"+ withConfiguration:[UIImageSymbolConfiguration configurationWithPointSize:19 weight:UIImageSymbolWeightBold]];+ if (![_play imageForState:UIControlStateNormal] || _play.tag != playing) {+ _play.tag = playing;+ [_play setImage:glyph forState:UIControlStateNormal];+ }+ _play.enabled = player != nil;+ _play.accessibilityLabel = playing ? @"Pause" : @"Play";++ if (!_column.tracking) _column.level = SGSingLevel();+ float level = _column.level;+ NSInteger stop = level == 0 ? 0 : level == 1 ? 1 : level == 2 ? 2 : UISegmentedControlNoSegment;+ if (_stops.selectedSegmentIndex != stop) _stops.selectedSegmentIndex = stop;++ _song.accessibilityLabel = _shown.artist.length ? [NSString stringWithFormat:@"%@, %@", title, artist] : title;+ _song.accessibilityValue = [NSString stringWithFormat:@"Sing: %@. Vocals: %@", SGSingStatusText(), SGSingLevelText(level)];+ _song.accessibilityHint = SGSingStatusDetail() ? @"Double-tap for more about Sing's state." : nil;+ _song.accessibilityTraits = SGSingStatusDetail() ? UIAccessibilityTraitButton : UIAccessibilityTraitStaticText;+ _song.accessibilityFrameInContainerSpace = CGRectMake(0, 0, CGRectGetMinX(_column.frame), CGRectGetMinY(_play.frame));+}++- (void)tick {+ // The words ten times a second, the lines at every read.+ if (_ticks++ % 3 == 0) [self refresh];+ [self drawLines:YES];+}++- (UIBezierPath *)pathFor:(const float *)levels count:(int)count step:(CGFloat)step {+ CGSize size = _graph.bounds.size;+ CGFloat low = size.height - 3, span = size.height - 8;+ UIBezierPath *path = [UIBezierPath bezierPath];+ CGPoint last = CGPointZero;+ for (int k = 0; k < count; k++) {+ CGPoint point = CGPointMake(k * step, low - span * height(levels[k]));+ if (k == 0) [path moveToPoint:point];+ // Through the midpoints, each sample a control point: a smooth line that stays within the samples.+ else [path addQuadCurveToPoint:CGPointMake((last.x + point.x) / 2, (last.y + point.y) / 2) controlPoint:last];+ last = point;+ }+ [path addLineToPoint:last];+ return path;+}++// How many tenths the levels read now have moved on from the ones drawn, up to kMostBehind + 1, or -1 when they are+// not the drawn ones moved on (a seek, the level changed, the first read). The engine hands back the very floats it+// stored, so the drawn ones moved on compare equal.+- (int)movedOnVocals:(const float *)vocals rest:(const float *)rest {+ for (int moved = 0; moved <= kMostBehind + 1; moved++) {+ BOOL same = YES;+ for (int k = 0; k + moved <= kHistory && same; k++) same = vocals[k] == _drawnVocals[k + moved] && rest[k] == _drawnRest[k + moved];+ if (same) return moved;+ }+ return -1;+}++// `live`: from the timer, the lines may glide; otherwise (a layout) they are put where they are, at once. At rest+// the lines sit a tenth to the left, so the newest tenth ends at the graph's right edge, under the column.+- (void)drawLines:(BOOL)live {+ if (CGRectIsEmpty(_graph.bounds)) return;+ float vocals[kHistory + 1] = {0}, rest[kHistory + 1] = {0};+ BOOL heard = SGSingReadLevels(vocals, rest, kHistory + 1);+ BOOL still = UIAccessibilityIsReduceMotionEnabled();+ // Flat, and still nothing to trace.+ if (live && !heard && _flat) return;+ int moved = heard && !_flat ? [self movedOnVocals:vocals rest:rest] : -1;+ // Nothing new: the glide runs on, or the lines rest. Under Reduce Motion, in place every two seconds.+ if (live && moved == 0) return;+ if (live && heard && !_flat && still && _ticks % 60) return;+ CGFloat step = _graph.bounds.size.width / (kHistory - 1), atRest = -step;+ // From where the lines are on screen now, moved back by the tenths that came in, at most kMostBehind behind.+ BOOL glide = live && heard && !still && moved > 0;+ NSNumber *shown = [_vocals.presentationLayer valueForKeyPath:@"transform.translation.x"];+ CGFloat from = glide ? fmin((shown ? shown.doubleValue : atRest) + moved * step, atRest + kMostBehind * step) : atRest;+ // Calm: flat lines, eased there once.+ BOOL settle = live && !heard && !_flat;+ _flat = !heard;+ memcpy(_drawnVocals, vocals, sizeof vocals);+ memcpy(_drawnRest, rest, sizeof rest);+ [CATransaction begin];+ [CATransaction setDisableActions:!settle];+ if (settle) [CATransaction setAnimationDuration:0.4];+ // Calm also in colour: the vocals' line dims while there is nothing to trace.+ _vocals.strokeColor = [SGGreen() colorWithAlphaComponent:heard ? 1 : 0.35].CGColor;+ _vocals.path = [self pathFor:vocals count:kHistory + 1 step:step].CGPath;+ _rest.path = [self pathFor:rest count:kHistory + 1 step:step].CGPath;+ for (CAShapeLayer *line in @[_rest, _vocals]) line.transform = CATransform3DMakeTranslation(atRest, 0, 0);+ [CATransaction commit];+ for (CAShapeLayer *line in @[_rest, _vocals]) {+ [line removeAnimationForKey:@"scroll"];+ if (!glide || from <= atRest) continue;+ // A tenth's width in a tenth of a second, whatever the distance, so the pace never changes.+ CABasicAnimation *slide = [CABasicAnimation animationWithKeyPath:@"transform.translation.x"];+ slide.fromValue = @(from);+ slide.toValue = @(atRest);+ slide.duration = (from - atRest) / step * kTick;+ [line addAnimation:slide forKey:@"scroll"];+ }+}++#pragma mark - what it does++- (void)playTapped {+ id<SPTPlayer> player = SGKaraokePlayer();+ SPTPlayerState *state = SGPlayerState();+ if (!player || !state) return;+ if (state.isPaused) [player resume:nil];+ else [player pause:nil];+}++- (void)levelMoved {+ SGSetSingLevel(_column.level);+ [self refresh];+}++- (void)stopPicked {+ if (_stops.selectedSegmentIndex == UISegmentedControlNoSegment) return;+ SGSetSingLevel((float)_stops.selectedSegmentIndex);+ _column.level = (float)_stops.selectedSegmentIndex;+ [self refresh];+}++- (void)stateTapped:(UITapGestureRecognizer *)tap {+ CGPoint at = [tap locationInView:_card];+ if (at.y > CGRectGetMaxY(_state.frame) + 8 || at.x > CGRectGetMinX(_column.frame)) return;+ [self explainState];+}++- (void)explainState {+ NSString *detail = SGSingStatusDetail();+ if (!detail) return;+ UIAlertController *alert = [UIAlertController alertControllerWithTitle:@"Sing" message:detail preferredStyle:UIAlertControllerStyleAlert];+ if (SGSingCurrentState() == SGSingStateFailed) {+ [alert addAction:[UIAlertAction actionWithTitle:@"Try again" style:UIAlertActionStyleDefault handler:^(UIAlertAction *a) {+ SGSetSingOn(NO);+ SGSetSingOn(YES);+ }]];+ }+ [alert addAction:[UIAlertAction actionWithTitle:SGSingCurrentState() == SGSingStateFailed ? @"Cancel" : @"OK" style:UIAlertActionStyleCancel handler:nil]];+ [SGTopController() presentViewController:alert animated:YES completion:nil];+}++@end++UIView *SGSingCardView(void) {+ return [SGSingCard new];+}diff --git a/tweak/Sources/Shared/Sing/SGSingEngine.h b/tweak/Sources/Shared/Sing/SGSingEngine.hindex ac25fa2e..e7f60b0f 100644--- a/tweak/Sources/Shared/Sing/SGSingEngine.h+++ b/tweak/Sources/Shared/Sing/SGSingEngine.h@@ -5,9 +5,10 @@// render asks for, until the sound it holds (the lead) is a window plus the model's time. A worker thread// takes the held sound two seconds at a time, every 1.5 s, runs the separator and writes the vocals into a// second ring beside it, the half second the windows share crossfaded. Each render plays the held sound-// from the oldest frame on, the vocals turned down by the level wherever they are in, and dry wherever-// they are not yet (as the lead fills, or when the model falls behind). Switched off, it pulls half of-// what plays until the lead is given back, and is a straight pull again.+// from the oldest frame on, the vocals turned down by the level once a hop of them is in ahead, and dry once fewer+// than the reserve are (as the lead fills, or when the model falls behind), with a budget for falling behind. Not+// separating (switched off, held, given up, without a model), it plays the lead it holds on as it is, dry, and lets it+// go only with a flush, where nothing heard is lost; once off and let go, it is a straight pull again.//// Spatial voice: the vocals' middle (left plus right, halved) can be put at an angle off straight ahead,// where Sing.x holds it as the head turns. The ear facing it hears it louder and the other quieter at the same@@ -17,8 +18,10 @@// exactly as separated.//// What plays is the lead behind what Spotify's decoder has handed over, which is what Spotify's clock-// counts: Sing.x takes the lead off the player's position, and asks for a flush when Spotify seeks or-// skips, which drops the sound held from before it.+// counts when the player reports. Its clock then runs on by the time alone, so a position reported at a moment+// (Sing.x: when that line of the clock began) has the lead held at that moment taken off, less what was dropped since (SGSingEngineLeadAt): a lead filling+// after a report does not move what is heard, and one let go moves it on. Sing.x asks for a flush when Spotify+// seeks or skips, which drops the sound held from before it.//// Threading: SGSingEngineRender on the render thread only, allocation and lock free; the worker is the// engine's own; everything else from any thread.@@ -43,6 +46,8 @@ void SGSingEngineDestroy(SGSingEngine *engine);void SGSingEngineSetSeparator(SGSingEngine *engine, SGSingSeparator *separator);void SGSingEngineSetOn(SGSingEngine *engine, bool on);bool SGSingEngineOn(SGSingEngine *engine);+// On, not held, with a separator and not given up: the lead is kept up for the vocals.+bool SGSingEngineSeparating(SGSingEngine *engine);// The vocals' level from 0 (gone) through 1 (as the song has them) to 2 (the vocals alone, the rest gone).void SGSingEngineSetLevel(SGSingEngine *engine, float level);// Where the separated voice sounds, in radians to the listener's right of straight ahead (left is negative,@@ -72,12 +77,18 @@ static inline double SGSpatialVoiceAngle(SGSpatialFront *f, double yaw, double tf->last = time;return yawToRight * remainder(yaw - f->front, 2 * M_PI);}-// Holds the worker (a hot phone): what plays is dry, and the lead is given back.+// The vocals fell short for longer than the engine's budget: it plays the song as it is, the lead kept as it is,+// until switched off and on again.+bool SGSingEngineGaveUp(SGSingEngine *engine);+// Holds the worker (a hot phone, or Sing resting at As sung): what plays is dry, the lead kept as it is.void SGSingEngineSetPaused(SGSingEngine *engine, bool paused);-// Drops the sound held ahead, at the next render.+// Drops the sound held ahead, at the next render; the lead reads as dropped from now.void SGSingEngineFlush(SGSingEngine *engine);-// Seconds of Spotify's sound held ahead of what plays.+// Seconds of Spotify's sound held ahead of what plays, as the render thread left it.double SGSingEngineLead(SGSingEngine *engine);+// The seconds to take off a position Spotify's player counted at `when` (CFAbsoluteTime) and has run on by the+// time since: the lead held then, less what was dropped since; negative when more was dropped than was held.+double SGSingEngineLeadAt(SGSingEngine *engine, CFAbsoluteTime when);OSStatus SGSingEngineRender(SGSingEngine *engine, UInt32 frames, float *left, float *right, SGSingPull pull, void *context);@@ -86,8 +97,16 @@ typedef struct {double ready; // seconds of the held sound separated, ahead of what plays (negative when behind)double averageMS; // a window's separation, a running averagedouble voiceAngle; // radians, as last set- unsigned long long windows, failures, dryFrames; // dry: frames played unseparated while on+ unsigned long long windows, failures, dryFrames; // dry: frames played unseparated while separating+ double dropped; // seconds of held sound never played (flushes), since the engine was made+ unsigned long long written, played; // frames pulled from Spotify's mixer and played, since the engine was made+ bool mixing; // the vocals are turned down now (the hysteresis in SGSingEngine.m)+ double budgetSpent; // seconds short of vocals since they were last in, of the 8 s it gives up after} SGSingEngineStats;SGSingEngineStats SGSingEngineReadStats(SGSingEngine *engine);+// The loudness (RMS) of the separated vocals and of the rest of the song, per tenth of a second, for the `count`+// tenths up to what plays now, oldest first; 0 where none was separated. For the Sing page to draw: a few atomic+// reads, the measuring done on the worker.+void SGSingEngineReadLevels(SGSingEngine *engine, float *vocals, float *rest, int count);// The model's last error, nil when its last window went through.NSString *SGSingEngineError(SGSingEngine *engine);diff --git a/tweak/Sources/Shared/Sing/SGSingEngine.m b/tweak/Sources/Shared/Sing/SGSingEngine.mindex 5be82d7a..b91069ac 100644--- a/tweak/Sources/Shared/Sing/SGSingEngine.m+++ b/tweak/Sources/Shared/Sing/SGSingEngine.m@@ -9,10 +9,25 @@ enum {kLongestLead = kSGSingRate * 6,kShortestLead = kSGSingRate * 5 / 2,};-// The lead asked for over a window: this many times the model's average time, and this much more.-static const double kLeadPerModelTime = 1.5, kLeadSpare = 0.25;+// The lead asked for over a window: this many times the model's average time, and this much more. Just before each+// window lands, the vocals separated ahead of what plays are down to half the model's time and the spare, which must+// stay above the reserve with room for a window that comes in late.+static const double kLeadPerModelTime = 1.5, kLeadSpare = 0.75;// The vocals come in and go out over this long, so the switch and the lead filling do not click.static const float kFadeSeconds = 0.1f;+// Changes in what Spotify's clock runs ahead of what plays, kept for SGSingEngineLeadAt: a fill or two's worth, as+// it is read soon after each report; and the moments last asked about, kept for as long as their report stands.+enum { kHistory = 1024, kAsked = 8 };+// Hysteresis, so a model about as fast as the song fades the vocals out once rather than at every window: the mix+// goes back to the song as it is when the vocals separated ahead of what plays fall under the reserve (more than+// the fade), and comes back to them only with a full hop separated ahead, which the lead (a window, 1.5 times the+// model's time and the spare) reaches just as each window lands. Time spent so after the vocals were first in comes+// out of a budget, renewed by that long with the vocals in throughout; spent, the engine gives up (SGSingEngineGaveUp).+static const uint64_t kReserve = kSGSingRate * 12 / 100;+static const uint64_t kReturn = kSGSingEngineHop;+static const uint64_t kBudget = kSGSingRate * 8;+// The loudness the Sing page draws: per tenth of a second of the song, kept for the last kLevelSlots of them.+enum { kLevelBlock = kSGSingRate / 10, kLevelSlots = 128 };// How often the worker looks for a new window.static const useconds_t kPoll = 10000;// Spatial voice: at 90 degrees the pan goes this far of the way to one ear, the far ear hears the voice this@@ -32,23 +47,42 @@ struct SGSingEngine {_Atomic uint64_t base; // where the worker starts over, at the latest flushatomic_uint generation; // counts the flushesatomic_bool flushAsked, on, paused, hasSeparator;+ atomic_bool mixing, gaveUp; // the render's, for the stats and Sing.x+ atomic_bool freshBudget; // switched on again: the render starts the budget over+ _Atomic uint64_t spent; // the render's `recovering`, for the stats+ _Atomic uint64_t dropped; // frames of held sound never played: flushes+ // What Spotify's clock had counted past what plays, at each render that changed it (written less played,+ // plus dropped: a flush leaves it as it was), with when; `historyCount` counts the entries made.+ struct { _Atomic double at; _Atomic uint64_t ahead; } history[kHistory];+ _Atomic uint64_t historyCount;+ struct { double when; uint64_t ahead; } asked[kAsked];+ unsigned askedNext;atomic_uint levelBits, targetLead, voiceAngleBits;// The render thread's own.bool running; // the rings are in use: Sing is on, or the lead is not given back yetfloat amount, vocalsGain, otherGain;+ bool established; // the vocals have been in since the rings last started over or the work resumed+ bool wasWorking; // the last render's: working resumed (after a heat hold, a new model) starts over+ uint64_t recovering, steady; // frames short of vocals since they were in, and frames in with them sincefloat voiceAngle; // where the voice is now, gliding to voiceAngleBitsfloat shadowLeft, shadowRight; // the low-passes' last outputs, an ear each+ uint64_t ahead; // the last entry made in `history`// The worker's.pthread_t worker;atomic_bool alive;- os_unfair_lock lock; // the separator and the error+ os_unfair_lock lock; // the separator, the error and `asked`void *separator; // retainedNSString *error;float *window[2], *vocals[2];_Atomic uint64_t windows, failures, dryFrames, averageMicroseconds;++ // The worker's loudness of the vocals and the rest per block, the block's number written last; and the next+ // block it measures.+ struct { _Atomic uint64_t block; atomic_uint vocals, rest; } levels[kLevelSlots];+ uint64_t nextLevel;};static float loadFloat(atomic_uint *slot) {@@ -69,10 +103,13 @@ static void storeFloat(atomic_uint *slot, float value) {// Starts the rings over where the mixer is now.static void startOver(SGSingEngine *engine) {uint64_t written = atomic_load_explicit(&engine->written, memory_order_relaxed);+ atomic_fetch_add(&engine->dropped, written - atomic_load_explicit(&engine->played, memory_order_relaxed));atomic_store(&engine->played, written);atomic_store(&engine->base, written);atomic_fetch_add(&engine->generation, 1);engine->amount = 0;+ engine->established = false;+ atomic_store(&engine->mixing, false);}// `count` frames from the mixer into the rings at `at`, in up to two runs where the ring wraps.@@ -137,15 +174,19 @@ OSStatus SGSingEngineRender(SGSingEngine *engine, UInt32 frames, float *left, flatomic_store(&engine->flushAsked, false);}if (atomic_exchange_explicit(&engine->flushAsked, false, memory_order_relaxed)) startOver(engine);+ if (atomic_exchange_explicit(&engine->freshBudget, false, memory_order_relaxed)) engine->recovering = engine->steady = 0;uint64_t written = atomic_load_explicit(&engine->written, memory_order_relaxed);uint64_t played = atomic_load_explicit(&engine->played, memory_order_relaxed);uint64_t held = written - played;bool working = on && !atomic_load_explicit(&engine->paused, memory_order_relaxed)- && atomic_load_explicit(&engine->hasSeparator, memory_order_relaxed);- int64_t want = (working ? (int64_t)atomic_load_explicit(&engine->targetLead, memory_order_relaxed) : 0) - (int64_t)held;- // Up to twice what plays while the lead fills, half while it is given back.- int64_t count = (int64_t)frames + MAX(-(int64_t)frames / 2, MIN(want, (int64_t)frames));+ && atomic_load_explicit(&engine->hasSeparator, memory_order_relaxed)+ && !atomic_load_explicit(&engine->gaveUp, memory_order_relaxed);+ // Separating: up to twice what plays while the lead fills, half while a shorter target is reached. Not separating+ // (off, held, given up, no model), what plays: the lead held plays on as it is, nothing skipped, until a flush+ // (Spotify seeks, skips, stops, or Sing.x seeks it back to what is heard at a pause), and none is built.+ int64_t want = (int64_t)atomic_load_explicit(&engine->targetLead, memory_order_relaxed) - (int64_t)held;+ int64_t count = (int64_t)frames + (working ? MAX(-(int64_t)frames / 2, MIN(want, (int64_t)frames)) : 0);count = MAX(count, (int64_t)frames - (int64_t)held);count = MIN(count, (int64_t)(kRing - kSGSingWindowFrames) - (int64_t)held);if (count > 0) {@@ -160,10 +201,33 @@ OSStatus SGSingEngineRender(SGSingEngine *engine, UInt32 frames, float *left, fl}uint64_t ready = atomic_load_explicit(&engine->ready, memory_order_acquire);+ int64_t ahead = (int64_t)ready - (int64_t)played;+ // Work that resumes (cooled down, a model loaded again) has no vocals ahead yet: that is a start, not falling behind.+ if (working && !engine->wasWorking) {+ engine->established = false;+ engine->recovering = engine->steady = 0;+ }+ engine->wasWorking = working;+ bool mixing = atomic_load_explicit(&engine->mixing, memory_order_relaxed);+ if (mixing && (!working || ahead < (int64_t)(kReserve + frames))) {+ mixing = false;+ engine->steady = 0;+ } else if (!mixing && working && ahead >= (int64_t)kReturn) {+ mixing = engine->established = true;+ }+ if (mixing) {+ engine->steady += frames;+ if (engine->steady >= kBudget) engine->recovering = 0;+ } else if (working && engine->established) {+ engine->recovering += frames;+ if (engine->recovering >= kBudget) atomic_store_explicit(&engine->gaveUp, true, memory_order_relaxed);+ }+ atomic_store_explicit(&engine->mixing, mixing, memory_order_relaxed);+ atomic_store_explicit(&engine->spent, engine->recovering, memory_order_relaxed);float vocalsTo, otherTo;gainsFor(loadFloat(&engine->levelBits), &vocalsTo, &otherTo);float vocalsStep = (vocalsTo - engine->vocalsGain) / frames, otherStep = (otherTo - engine->otherGain) / frames;- float fade = 1.0f / (kFadeSeconds * kSGSingRate), amountTo = on ? 1 : 0;+ float fade = 1.0f / (kFadeSeconds * kSGSingRate), amountTo = on && mixing ? 1 : 0;// The voice's angle glides the short way round towards the one set, and lands on it; within the buffer// each ear's gain, delay and low-pass move in a straight line from where the angle was to where it is.@@ -198,13 +262,13 @@ OSStatus SGSingEngineRender(SGSingEngine *engine, UInt32 frames, float *left, fl} else {engine->shadowLeft = engine->shadowRight = (v + w) * 0.5f;}- left[i] = x + a * (engine->vocalsGain * placedLeft - v + otherGain * (x - v));- right[i] = y + a * (engine->vocalsGain * placedRight - w + otherGain * (y - w));+ left[i] = fmaxf(-1, fminf(x + a * (engine->vocalsGain * placedLeft - v + otherGain * (x - v)), 1));+ right[i] = fmaxf(-1, fminf(y + a * (engine->vocalsGain * placedRight - w + otherGain * (y - w)), 1));} else {engine->amount = 0;left[i] = x;right[i] = y;- dry += on;+ dry += working;}}engine->vocalsGain = vocalsTo;@@ -212,6 +276,14 @@ OSStatus SGSingEngineRender(SGSingEngine *engine, UInt32 frames, float *left, flplayed += frames;atomic_store_explicit(&engine->played, played, memory_order_release);if (dry) atomic_fetch_add_explicit(&engine->dryFrames, dry, memory_order_relaxed);+ uint64_t counted = written - played + atomic_load_explicit(&engine->dropped, memory_order_relaxed);+ if (counted != engine->ahead) {+ engine->ahead = counted;+ uint64_t n = atomic_load_explicit(&engine->historyCount, memory_order_relaxed);+ atomic_store_explicit(&engine->history[n % kHistory].at, CFAbsoluteTimeGetCurrent(), memory_order_relaxed);+ atomic_store_explicit(&engine->history[n % kHistory].ahead, counted, memory_order_relaxed);+ atomic_store_explicit(&engine->historyCount, n + 1, memory_order_release);+ }// The lead given back and the vocals faded out: a straight pull from the next render on.if (!on && played == written) engine->running = false;return noErr;@@ -238,6 +310,26 @@ static void updateTarget(SGSingEngine *engine, double modelSeconds) {atomic_store(&engine->targetLead, (unsigned)fmax(kShortestLead, fmin(lead, kLongestLead)));}+// The loudness (RMS of both channels) of the vocals and of the rest, per block not measured yet from `start`, where+// the window just separated begins, to `ready`; read from the worker's own copy of the window and its vocals, as the+// render may have pulled new sound over the rings there while the model ran.+static void measureLevels(SGSingEngine *engine, uint64_t start, uint64_t ready) {+ engine->nextLevel = MAX(engine->nextLevel, (start + kLevelBlock - 1) / kLevelBlock);+ for (; (engine->nextLevel + 1) * kLevelBlock <= ready; engine->nextLevel++) {+ double vocals = 0, rest = 0;+ for (uint64_t p = engine->nextLevel * kLevelBlock; p < (engine->nextLevel + 1) * kLevelBlock; p++) {+ uint64_t i = p - start;+ float v = engine->vocals[0][i], w = engine->vocals[1][i], x = engine->window[0][i] - v, y = engine->window[1][i] - w;+ vocals += v * v + w * w;+ rest += x * x + y * y;+ }+ __typeof__(engine->levels[0]) *slot = &engine->levels[engine->nextLevel % kLevelSlots];+ storeFloat(&slot->vocals, (float)sqrt(vocals / (2 * kLevelBlock)));+ storeFloat(&slot->rest, (float)sqrt(rest / (2 * kLevelBlock)));+ atomic_store_explicit(&slot->block, engine->nextLevel + 1, memory_order_release); // + 1: 0 is never written+ }+}+static void *work(void *argument) {SGSingEngine *engine = argument;pthread_setname_np("spotifyglass.sing");@@ -248,6 +340,7 @@ static void *work(void *argument) {if (now != generation) {generation = now;start = previousEnd = atomic_load(&engine->base);+ engine->nextLevel = (start + kLevelBlock - 1) / kLevelBlock;atomic_store_explicit(&engine->ready, start, memory_order_release);}SGSingSeparator *separator = currentSeparator(engine);@@ -281,6 +374,12 @@ static void *work(void *argument) {continue;}setError(engine, nil);+ // A value the model got wrong is silence, not a click or a channel gone quiet.+ for (int c = 0; c < 2; c++) {+ for (int i = 0; i < kSGSingWindowFrames; i++) {+ if (!isfinite(engine->vocals[c][i])) engine->vocals[c][i] = 0;+ }+ }uint64_t windows = atomic_fetch_add(&engine->windows, 1);uint64_t average = atomic_load(&engine->averageMicroseconds);average = windows ? (uint64_t)(average * 0.8 + took * 1e6 * 0.2) : (uint64_t)(took * 1e6);@@ -295,6 +394,7 @@ static void *work(void *argument) {engine->vocalsRight[index] = engine->vocalsRight[index] * (1 - a) + engine->vocals[1][i] * a;}if (atomic_load(&engine->generation) == generation) atomic_store_explicit(&engine->ready, start + kSGSingEngineHop, memory_order_release);+ measureLevels(engine, start, start + kSGSingEngineHop);previousEnd = start + kSGSingWindowFrames;start += kSGSingEngineHop;}@@ -358,13 +458,26 @@ void SGSingEngineSetSeparator(SGSingEngine *engine, SGSingSeparator *separator)}void SGSingEngineSetOn(SGSingEngine *engine, bool on) {+ // Switched on again, it tries again with a full budget.+ if (on && !atomic_exchange(&engine->on, true)) {+ atomic_store(&engine->freshBudget, true);+ atomic_store(&engine->gaveUp, false);+ }atomic_store(&engine->on, on);}+bool SGSingEngineGaveUp(SGSingEngine *engine) {+ return atomic_load(&engine->gaveUp);+}+bool SGSingEngineOn(SGSingEngine *engine) {return atomic_load(&engine->on);}+bool SGSingEngineSeparating(SGSingEngine *engine) {+ return engine && atomic_load(&engine->on) && !atomic_load(&engine->paused) && atomic_load(&engine->hasSeparator) && !atomic_load(&engine->gaveUp);+}+void SGSingEngineSetLevel(SGSingEngine *engine, float level) {storeFloat(&engine->levelBits, level);}@@ -381,13 +494,50 @@ void SGSingEngineFlush(SGSingEngine *engine) {atomic_store(&engine->flushAsked, true);}+// The frames a flush asked for and not yet done will drop: the render may not run before it is read (Spotify paused).+static uint64_t pendingDrop(SGSingEngine *engine) {+ if (!atomic_load_explicit(&engine->flushAsked, memory_order_acquire)) return 0;+ uint64_t played = atomic_load_explicit(&engine->played, memory_order_acquire);+ uint64_t written = atomic_load_explicit(&engine->written, memory_order_acquire);+ return written > played ? written - played : 0;+}+double SGSingEngineLead(SGSingEngine *engine) {- if (!engine) return 0;- uint64_t played = atomic_load_explicit(&engine->played, memory_order_relaxed);- uint64_t written = atomic_load_explicit(&engine->written, memory_order_relaxed);+ if (!engine || atomic_load_explicit(&engine->flushAsked, memory_order_acquire)) return 0;+ uint64_t played = atomic_load_explicit(&engine->played, memory_order_acquire);+ uint64_t written = atomic_load_explicit(&engine->written, memory_order_acquire);return written > played ? (double)(written - played) / kSGSingRate : 0;}+double SGSingEngineLeadAt(SGSingEngine *engine, CFAbsoluteTime when) {+ if (!engine) return 0;+ os_unfair_lock_lock(&engine->lock);+ uint64_t ahead = 0;+ bool known = false;+ for (unsigned i = 0; i < kAsked && !known; i++) {+ if (engine->asked[i].when == when) ahead = engine->asked[i].ahead, known = true;+ }+ if (!known) {+ // What the clock had counted past what played at `when`: the last entry made by then, none before the first;+ // kept for the next time the same report is read.+ // ponytail: a `when` first asked about more than kHistory changes on reads the oldest kept; Sing.x reads the+ // player's state twice a second while on. The render would have to make kHistory entries while this reads+ // for one to tear.+ uint64_t n = atomic_load_explicit(&engine->historyCount, memory_order_acquire);+ for (uint64_t k = n; k > 0 && n - k < kHistory; k--) {+ __typeof__(engine->history[0]) *entry = &engine->history[(k - 1) % kHistory];+ ahead = atomic_load_explicit(&entry->ahead, memory_order_relaxed);+ if (atomic_load_explicit(&entry->at, memory_order_relaxed) <= when) break;+ if (k == 1) ahead = 0;+ }+ engine->asked[engine->askedNext++ % kAsked] = (__typeof__(engine->asked[0])){when, ahead};+ }+ os_unfair_lock_unlock(&engine->lock);+ // Less what was dropped since, which what plays skipped, a flush asked for with it.+ double dropped = (double)atomic_load_explicit(&engine->dropped, memory_order_relaxed) + (double)pendingDrop(engine);+ return ((double)ahead - dropped) / kSGSingRate;+}+SGSingEngineStats SGSingEngineReadStats(SGSingEngine *engine) {uint64_t played = atomic_load(&engine->played), ready = atomic_load(&engine->ready);return (SGSingEngineStats){@@ -399,6 +549,11 @@ SGSingEngineStats SGSingEngineReadStats(SGSingEngine *engine) {.windows = atomic_load(&engine->windows),.failures = atomic_load(&engine->failures),.dryFrames = atomic_load(&engine->dryFrames),+ .dropped = atomic_load(&engine->dropped) / (double)kSGSingRate,+ .written = atomic_load(&engine->written),+ .played = played,+ .mixing = atomic_load(&engine->mixing),+ .budgetSpent = atomic_load(&engine->spent) / (double)kSGSingRate,};}@@ -408,3 +563,14 @@ NSString *SGSingEngineError(SGSingEngine *engine) {os_unfair_lock_unlock(&engine->lock);return error;}++void SGSingEngineReadLevels(SGSingEngine *engine, float *vocals, float *rest, int count) {+ uint64_t now = atomic_load_explicit(&engine->played, memory_order_relaxed) / kLevelBlock;+ for (int k = 0; k < count; k++) {+ uint64_t block = now + k + 1 - (uint64_t)count;+ __typeof__(engine->levels[0]) *slot = &engine->levels[block % kLevelSlots];+ bool known = now + 1 >= (uint64_t)count - k && atomic_load_explicit(&slot->block, memory_order_acquire) == block + 1;+ vocals[k] = known ? loadFloat(&slot->vocals) : 0;+ rest[k] = known ? loadFloat(&slot->rest) : 0;+ }+}diff --git a/tweak/Sources/Shared/Sing/SGSingLoader.h b/tweak/Sources/Shared/Sing/SGSingLoader.hnew file mode 100644index 00000000..ee08167d--- /dev/null+++ b/tweak/Sources/Shared/Sing/SGSingLoader.h@@ -0,0 +1,71 @@+// Sing's voice model in memory (Sing.h): loaded, warmed, kept a minute after the mic goes off, and dropped.+//+// A copy on the CPU alone comes first, since that is the one Core ML has always loaded: once one prediction of+// silence has warmed it, the separator is Ready and Sing works. Then, while Spotify is in the foreground, a faster+// copy on the units asked for (the GPU unless the Sing page's Runs on says otherwise) loads beside it, warms the+// same way and is handed to the separator, which uses it for the windows while Spotify is in the foreground. A+// second copy loads only when the process has comfortably more memory left than a copy takes.+//+// Core ML cannot cancel a load, and on an iPhone 15 Pro one has never come back. So each load runs on a queue of its+// own, has a deadline, and is abandoned when the deadline passes or the model is dropped: abandoned, it is left to+// finish or hang where it is, and whatever it brings back later is let go. A CPU copy past its deadline makes Sing+// Failed until the mic is switched off and on; a faster copy past its deadline leaves Sing on the CPU's.+//+// Threading: main thread, but for SGSingLoaderSeparator's separator, which the engine's worker runs.+#import <CoreML/CoreML.h>++@class SGSingSeparator;++typedef NS_ENUM(NSInteger, SGSingLoaderState) {+ SGSingLoaderIdle, // nothing loaded or loading+ SGSingLoaderLoading, // the CPU copy loading or warming+ SGSingLoaderReady, // the CPU copy warm: SGSingLoaderSeparator separates+ SGSingLoaderFailed, // the CPU copy did not load (SGSingLoaderError says why); stays so until a retry+};++// What became of the faster copy.+typedef NS_ENUM(NSInteger, SGSingFastState) {+ SGSingFastNone, // none asked for, or not yet started+ SGSingFastLoading,+ SGSingFastReady,+ SGSingFastSkipped, // too little memory for a second copy+ SGSingFastFailed, // Core ML refused it, or its warm-up failed+ SGSingFastTimedOut, // past its deadline; SGSingLoaderFastTimedOutActive says whether Spotify stayed active through it+};++// Run on the main thread whenever the state, the separator or the faster copy changes.+void SGSingLoaderSetChanged(void (^changed)(void));+// Loads the model at `url` or joins the load already running, with `fast` the units of the faster copy+// (MLComputeUnitsCPUOnly for none); a different `fast` than before drops the faster copy for the new one. Does+// nothing while Failed. Call it only while Spotify is in the foreground.+void SGSingLoaderWant(NSURL *url, MLComputeUnits fast);+// The mic is off: the copies are kept for SGSingLoaderKeepSeconds, then dropped unless wanted again by then.+// Failed is cleared, so the next want is a fresh load whatever an abandoned one is still doing.+void SGSingLoaderRelease(void);+// Both copies dropped at once and any load abandoned, `why` logged when there was something to drop.+void SGSingLoaderPurge(NSString *why);+// The faster copy alone dropped (a memory warning), the CPU's kept.+void SGSingLoaderDropFast(NSString *why);+// Spotify became active or stopped being active: the faster copy is used only while it is, and starts loading+// when it becomes so.+void SGSingLoaderSetForeground(BOOL foreground);++SGSingLoaderState SGSingLoaderCurrentState(void);+// Whether the process has the memory left for a copy (always, where the OS does not say).+BOOL SGSingLoaderHasRoom(void);+// Nil until Ready.+SGSingSeparator *SGSingLoaderSeparator(void);+NSString *SGSingLoaderError(void);+// Seconds since the CPU copy began loading, while Loading.+NSTimeInterval SGSingLoaderSeconds(void);+SGSingFastState SGSingLoaderFastState(void);+BOOL SGSingLoaderFastTimedOutActive(void);+MLComputeUnits SGSingLoaderFastUnits(void);+// "GPU", "Neural Engine", "GPU and Neural Engine", "CPU".+NSString *SGSingUnitsName(MLComputeUnits units);+// Loads started since launch, and loads not come back yet (abandoned ones with them), for the harness.+unsigned SGSingLoaderAttempts(void);+unsigned SGSingLoaderOutstanding(void);++// The deadlines and the minute kept, in seconds; changed only by the harness.+extern double SGSingLoaderCPUDeadline, SGSingLoaderFastDeadline, SGSingLoaderKeepSeconds;diff --git a/tweak/Sources/Shared/Sing/SGSingLoader.m b/tweak/Sources/Shared/Sing/SGSingLoader.mnew file mode 100644index 00000000..98cfed5a--- /dev/null+++ b/tweak/Sources/Shared/Sing/SGSingLoader.m@@ -0,0 +1,289 @@+#import <TargetConditionals.h>+#if TARGET_OS_IPHONE+#import <os/proc.h>+#endif+#import "Core/SGLog.h"+#import "SGSingLoader.h"+#import "SGSingSeparator.h"++// The CPU copy loaded in 0.9-5 s on an M4 and the Neural Engine's in 44.6 s on an iPhone 15 Pro; a GPU load there+// never came back in 14 min. The CPU copy is what Sing waits on, the faster one runs beside a working Sing.+double SGSingLoaderCPUDeadline = 120, SGSingLoaderFastDeadline = 180, SGSingLoaderKeepSeconds = 60;+// A second copy took 1.07-1.15 GB more of the process's footprint on an M4, so it loads only with this much left.+static const unsigned long long kFastRoom = 1600ull * 1000 * 1000;+static const int kStillEvery = 30;++// One load of one copy.+@interface SGSingLoad : NSObject+@property (nonatomic) unsigned number;+@property (nonatomic) MLComputeUnits units;+@property (nonatomic) CFAbsoluteTime began;+@property (nonatomic) BOOL finished, abandoned;+@property (nonatomic) BOOL stayedActive; // Spotify was active from its start to its end or deadline+@end++@implementation SGSingLoad+@end++static void (^sg_changed)(void);+static SGSingLoaderState sg_state;+static NSString *sg_error;+static SGSingSeparator *sg_separator;+static SGSingLoad *sg_cpuLoad, *sg_fastLoad; // the loads that count; abandoned ones are let go+static unsigned sg_outstanding; // loads not come back yet, abandoned ones with them+static unsigned sg_attempts;+static NSURL *sg_url;+static MLComputeUnits sg_fastUnits = MLComputeUnitsCPUOnly;+static SGSingFastState sg_fastState;+static BOOL sg_fastTimedOutActive;+static BOOL sg_foreground;+static NSUInteger sg_keepToken; // counts the mic's releases and wants, so a stale minute drops nothing+static BOOL sg_wanted; // the mic is on: a faster copy is worth loading++// Run after the caller returns, so a listener that wants or purges again does not run inside it.+static void changed(void) {+ dispatch_async(dispatch_get_main_queue(), ^{+ if (sg_changed) sg_changed();+ });+}++NSString *SGSingUnitsName(MLComputeUnits units) {+ switch (units) {+ case MLComputeUnitsCPUAndGPU: return @"GPU";+ case MLComputeUnitsCPUAndNeuralEngine: return @"Neural Engine";+ case MLComputeUnitsAll: return @"GPU and Neural Engine";+ default: return @"CPU";+ }+}++// The memory the process has left before iOS closes it; 0 where the OS does not say (the Mac).+static unsigned long long memoryLeft(void) {+#if TARGET_OS_IPHONE+ return os_proc_available_memory();+#else+ return 0;+#endif+}++static NSString *memoryText(void) {+ unsigned long long left = memoryLeft();+ return left ? [NSString stringWithFormat:@"%.2f GB left to the process", left / 1e9] : @"memory left unknown";+}++// Whether a copy can load beside what is already in memory or still loading.+static BOOL roomForAnother(void) {+ unsigned long long left = memoryLeft();+ return !left || left >= kFastRoom;+}++static void still(SGSingLoad *load, int seconds) {+ dispatch_after(dispatch_time(DISPATCH_TIME_NOW, kStillEvery * NSEC_PER_SEC), dispatch_get_main_queue(), ^{+ if (load.finished || load.abandoned) return;+ SGLog(@"sing: still loading the %@ copy (load %u), %d s", SGSingUnitsName(load.units), load.number, seconds + kStillEvery);+ still(load, seconds + kStillEvery);+ });+}++// Loads and warms a copy on a queue of its own; `done` runs on the main thread unless the load was abandoned+// first, and `late` when its deadline passes before it comes back.+static SGSingLoad *startLoad(NSURL *url, MLComputeUnits units, double deadline, void (^done)(MLModel *model, NSError *error),+ void (^late)(SGSingLoad *load)) {+ SGSingLoad *load = [SGSingLoad new];+ load.number = ++sg_attempts;+ load.units = units;+ load.began = CFAbsoluteTimeGetCurrent();+ load.stayedActive = sg_foreground;+ sg_outstanding++;+ SGLog(@"sing: load %u, the %@ copy, starts (Spotify %@, thermal state %s, %@, %u loads out)", load.number, SGSingUnitsName(units),+ sg_foreground ? @"active" : @"not active", SGSingThermalName(), memoryText(), sg_outstanding);+ NSString *name = [NSString stringWithFormat:@"spotifyglass.sing.load.%u", load.number];+ dispatch_queue_t queue = dispatch_queue_create(name.UTF8String, dispatch_queue_attr_make_with_qos_class(DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INITIATED, 0));+ dispatch_async(queue, ^{+ MLModelConfiguration *configuration = [MLModelConfiguration new];+ configuration.computeUnits = units;+ NSError *error;+ MLModel *model = [MLModel modelWithContentsOfURL:url configuration:configuration error:&error];+ double loaded = CFAbsoluteTimeGetCurrent() - load.began;+ double warm = model ? [SGSingSeparator warmUp:model error:&error] : -1;+ dispatch_async(dispatch_get_main_queue(), ^{+ sg_outstanding--;+ load.finished = YES;+ NSString *outcome = !model ? [NSString stringWithFormat:@"did not load after %.1f s: %@", loaded, error.localizedDescription]+ : warm < 0 ? [NSString stringWithFormat:@"loaded in %.1f s, and its warm-up failed: %@", loaded, error.localizedDescription]+ : [NSString stringWithFormat:@"loaded in %.1f s, and its warm-up window of silence took %.1f s", loaded, warm];+ SGLog(@"sing: load %u, the %@ copy, %@%@", load.number, SGSingUnitsName(units), outcome,+ load.abandoned ? @"; it was abandoned before, so it is let go" : @"");+ if (load.abandoned) return;+ done(warm >= 0 ? model : nil, error);+ });+ });+ still(load, 0);+ dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(deadline * NSEC_PER_SEC)), dispatch_get_main_queue(), ^{+ if (load.finished || load.abandoned) return;+ load.abandoned = YES;+ SGLog(@"sing: load %u, the %@ copy, has not come back in %.0f s (Spotify %@ throughout): abandoned", load.number, SGSingUnitsName(units),+ deadline, load.stayedActive ? @"active" : @"not always active");+ late(load);+ });+ return load;+}++static void startFast(void) {+ if (!sg_wanted || sg_state != SGSingLoaderReady || !sg_foreground || sg_fastUnits == MLComputeUnitsCPUOnly || sg_fastState != SGSingFastNone) return;+ if (!roomForAnother()) {+ sg_fastState = SGSingFastSkipped;+ SGLog(@"sing: no %@ copy: %@, a copy wants %.1f GB, so Sing stays on the CPU", SGSingUnitsName(sg_fastUnits), memoryText(), kFastRoom / 1e9);+ changed();+ return;+ }+ sg_fastState = SGSingFastLoading;+ sg_fastTimedOutActive = NO;+ sg_fastLoad = startLoad(sg_url, sg_fastUnits, SGSingLoaderFastDeadline, ^(MLModel *model, NSError *error) {+ sg_fastLoad = nil;+ sg_fastState = model ? SGSingFastReady : SGSingFastFailed;+ if (model) [sg_separator setFastModel:model named:SGSingUnitsName(sg_fastUnits)];+ changed();+ }, ^(SGSingLoad *load) {+ sg_fastLoad = nil;+ sg_fastState = SGSingFastTimedOut;+ sg_fastTimedOutActive = load.stayedActive;+ changed();+ });+ changed();+}++static void startCPU(void) {+ if (sg_outstanding && !roomForAnother()) {+ sg_state = SGSingLoaderFailed;+ sg_error = [NSString stringWithFormat:@"A load abandoned earlier still holds memory, and %@. Close Spotify and open it again to load the voice model.", memoryText()];+ SGLog(@"sing: no new load: %@", sg_error);+ changed();+ return;+ }+ sg_state = SGSingLoaderLoading;+ sg_error = nil;+ sg_cpuLoad = startLoad(sg_url, MLComputeUnitsCPUOnly, SGSingLoaderCPUDeadline, ^(MLModel *model, NSError *error) {+ sg_cpuLoad = nil;+ sg_separator = model ? [[SGSingSeparator alloc] initWithModel:model] : nil;+ if (sg_separator) {+ sg_state = SGSingLoaderReady;+ } else {+ sg_state = SGSingLoaderFailed;+ sg_error = model ? @"Sing could not set aside memory for the voice model."+ : [NSString stringWithFormat:@"The voice model did not load: %@", error.localizedDescription ?: @"Core ML gave no reason."];+ }+ changed();+ startFast();+ }, ^(SGSingLoad *load) {+ sg_cpuLoad = nil;+ sg_state = SGSingLoaderFailed;+ sg_error = [NSString stringWithFormat:@"The voice model did not load in %.0f s, so that load was given up.", SGSingLoaderCPUDeadline];+ changed();+ });+ changed();+}++static void abandon(SGSingLoad *load) {+ if (load && !load.finished) load.abandoned = YES;+}++void SGSingLoaderSetChanged(void (^block)(void)) {+ sg_changed = block;+}++void SGSingLoaderDropFast(NSString *why) {+ BOOL had = sg_fastState == SGSingFastLoading || sg_fastState == SGSingFastReady;+ abandon(sg_fastLoad);+ sg_fastLoad = nil;+ [sg_separator setFastModel:nil named:nil];+ sg_fastState = SGSingFastNone;+ sg_fastTimedOutActive = NO;+ if (!had) return;+ if (why) SGLog(@"sing: the %@ copy is dropped: %@", SGSingUnitsName(sg_fastUnits), why);+ changed();+}++void SGSingLoaderWant(NSURL *url, MLComputeUnits fast) {+ sg_keepToken++;+ sg_wanted = YES;+ sg_url = url;+ if (fast != sg_fastUnits) {+ SGSingLoaderDropFast(@"Runs on changed");+ sg_fastUnits = fast;+ }+ if (sg_state == SGSingLoaderIdle) startCPU();+ else startFast();+}++void SGSingLoaderPurge(NSString *why) {+ sg_keepToken++;+ sg_wanted = NO;+ BOOL had = sg_separator || sg_cpuLoad || sg_fastLoad;+ abandon(sg_cpuLoad);+ sg_cpuLoad = nil;+ SGSingLoaderDropFast(nil);+ sg_separator = nil;+ if (!had) return;+ sg_state = SGSingLoaderIdle;+ SGLog(@"sing: the voice model is dropped: %@", why);+ changed();+}++void SGSingLoaderRelease(void) {+ NSUInteger token = ++sg_keepToken;+ sg_wanted = NO;+ if (sg_state == SGSingLoaderFailed) {+ sg_state = SGSingLoaderIdle;+ sg_error = nil;+ }+ dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(SGSingLoaderKeepSeconds * NSEC_PER_SEC)), dispatch_get_main_queue(), ^{+ if (token == sg_keepToken) SGSingLoaderPurge([NSString stringWithFormat:@"the mic has been off for %.0f s", SGSingLoaderKeepSeconds]);+ });+}++void SGSingLoaderSetForeground(BOOL foreground) {+ sg_foreground = foreground;+ SGSingSeparatorSetForeground(foreground);+ if (!foreground) sg_fastLoad.stayedActive = NO;+ else startFast();+}++BOOL SGSingLoaderHasRoom(void) {+ return roomForAnother();+}++SGSingLoaderState SGSingLoaderCurrentState(void) {+ return sg_state;+}++SGSingSeparator *SGSingLoaderSeparator(void) {+ return sg_state == SGSingLoaderReady ? sg_separator : nil;+}++NSString *SGSingLoaderError(void) {+ return sg_state == SGSingLoaderFailed ? sg_error : nil;+}++NSTimeInterval SGSingLoaderSeconds(void) {+ return sg_cpuLoad ? CFAbsoluteTimeGetCurrent() - sg_cpuLoad.began : 0;+}++SGSingFastState SGSingLoaderFastState(void) {+ return sg_fastState;+}++BOOL SGSingLoaderFastTimedOutActive(void) {+ return sg_fastTimedOutActive;+}++MLComputeUnits SGSingLoaderFastUnits(void) {+ return sg_fastUnits;+}++unsigned SGSingLoaderAttempts(void) {+ return sg_attempts;+}++unsigned SGSingLoaderOutstanding(void) {+ return sg_outstanding;+}diff --git a/tweak/Sources/Shared/Sing/SGSingModel.m b/tweak/Sources/Shared/Sing/SGSingModel.mindex 3a0c864b..482f58cd 100644--- a/tweak/Sources/Shared/Sing/SGSingModel.m+++ b/tweak/Sources/Shared/Sing/SGSingModel.m@@ -12,6 +12,10 @@static NSString *const kRepo = @"https://huggingface.co/My-Name-Is-Jeff/vitrine-sing/resolve/main/";static const int kRetries = 3;+// Bytes a stopped download has kept, for the row that offers to carry on with it or remove it.+static NSString *const kPausedKey = @"spotifyglass.sing.downloadPaused";+// Room the download leaves on the disk besides itself.+static const long long kSpareSpace = 64ll * 1000 * 1000;typedef struct {NSString *__unsafe_unretained path;@@ -108,7 +112,9 @@ static void announce(void) {static SGSingDownloader *sg_downloader; // while a download runsstatic NSString *sg_lastError;static double sg_progress;-static BOOL sg_waitingForNetwork; // offline: the session holds the request until the iPhone is online+static BOOL sg_waitingForNetwork; // offline, or on cellular unless allowed: the session holds the request+static BOOL sg_cellular; // the download may use cellular and Low Data Mode networks+static BOOL sg_checking; // the weights' checksum is being read@implementation SGSingDownloader {NSURLSession *_session;@@ -121,8 +127,11 @@ static BOOL sg_waitingForNetwork; // offline: the session holds the request un- (void)start {NSURLSessionConfiguration *configuration = NSURLSessionConfiguration.defaultSessionConfiguration;configuration.timeoutIntervalForRequest = 60;- // Offline, a request (a retry too) waits for the network rather than failing at once.+ // Offline, a request (a retry too) waits for the network rather than failing at once; and so it does on cellular+ // or a Low Data Mode network, 489 MB being a lot of a plan, until the user says it may use them.configuration.waitsForConnectivity = YES;+ configuration.allowsExpensiveNetworkAccess = sg_cellular;+ configuration.allowsConstrainedNetworkAccess = sg_cellular;NSOperationQueue *queue = [NSOperationQueue new];queue.maxConcurrentOperationCount = 1;_session = [NSURLSession sessionWithConfiguration:configuration delegate:self delegateQueue:queue];@@ -198,6 +207,7 @@ static BOOL sg_waitingForNetwork; // offline: the session holds the request unsg_downloader = nil;sg_waitingForNetwork = NO;sg_lastError = error;+ if (!error) [NSUserDefaults.standardUserDefaults removeObjectForKey:kPausedKey];sg_modelGeneration++;announce();});@@ -247,7 +257,10 @@ static BOOL sg_waitingForNetwork; // offline: the session holds the request unreturn;}long long size = sizeOf(location);+ BOOL big = file.size > 1000 * 1000;+ if (big) dispatch_async(dispatch_get_main_queue(), ^{ sg_checking = YES; announce(); });NSString *hash = size == file.size ? sha256Of(location) : nil;+ if (big) dispatch_async(dispatch_get_main_queue(), ^{ sg_checking = NO; announce(); });SGLog(@"sing: %@ came in, %ld, %lld bytes", file.path, (long)status, size);if (!answered || size != file.size || ![hash isEqualToString:@(file.sha256)]) {self.error = !answered ? [NSString stringWithFormat:@"The server answered %ld for %@", (long)status, file.path]@@ -326,8 +339,30 @@ NSString *SGSingModelSizeText(void) {return [NSByteCountFormatter stringFromByteCount:totalBytes() countStyle:NSByteCountFormatterCountStyleFile];}+// What is still to come: the files not in the staging folder yet.+static long long bytesToCome(void) {+ long long left = 0;+ for (int i = 0; i < kFileCount; i++) {+ if (sizeOf([stagingFolder() URLByAppendingPathComponent:kFiles[i].path]) != kFiles[i].size) left += kFiles[i].size;+ }+ return left;+}+void SGSingDownloadModel(void) {if (sg_downloader || SGSingModelURL()) return;+ // The volume Application Support is on, which the staging folder's parent may not exist on yet.+ NSURL *support = singFolder().URLByDeletingLastPathComponent.URLByDeletingLastPathComponent;+ NSNumber *free = [support resourceValuesForKeys:@[NSURLVolumeAvailableCapacityForImportantUsageKey] error:nil][NSURLVolumeAvailableCapacityForImportantUsageKey];+ long long wanted = bytesToCome() + kSpareSpace;+ if (free && free.longLongValue < wanted) {+ NSByteCountFormatterCountStyle style = NSByteCountFormatterCountStyleFile;+ sg_lastError = [NSString stringWithFormat:@"The iPhone has %@ free, and the voice model needs %@ more.",+ [NSByteCountFormatter stringFromByteCount:free.longLongValue countStyle:style],+ [NSByteCountFormatter stringFromByteCount:wanted - free.longLongValue countStyle:style]];+ SGLog(@"sing: the download does not start: %@", sg_lastError);+ announce();+ return;+ }sg_lastError = nil;sg_progress = 0;sg_waitingForNetwork = NO;@@ -336,7 +371,33 @@ void SGSingDownloadModel(void) {announce();}+void SGSingDownloadModelOverCellular(void) {+ if (!sg_downloader) {+ sg_cellular = YES;+ SGSingDownloadModel();+ return;+ }+ if (sg_cellular) return;+ // The running download's session forbids cellular: it stops keeping what it has, and starts again allowed.+ SGSingCancelModelDownload();+ sg_cellular = YES;+ dispatch_after(dispatch_time(DISPATCH_TIME_NOW, NSEC_PER_SEC), dispatch_get_main_queue(), ^{ SGSingDownloadModel(); });+}++BOOL SGSingModelOverCellular(void) {+ return sg_cellular;+}++BOOL SGSingModelChecking(void) {+ return sg_downloader && sg_checking;+}++long long SGSingModelPausedBytes(void) {+ return sg_downloader || SGSingModelURL() ? 0 : [NSUserDefaults.standardUserDefaults integerForKey:kPausedKey];+}+void SGSingCancelModelDownload(void) {+ if (sg_downloader) [NSUserDefaults.standardUserDefaults setInteger:(NSInteger)(sg_progress * totalBytes()) forKey:kPausedKey];[sg_downloader cancel];sg_downloader = nil;sg_modelGeneration++;@@ -346,12 +407,13 @@ void SGSingCancelModelDownload(void) {void SGSingDeleteModel(void) {SGSingCancelModelDownload();[NSFileManager.defaultManager removeItemAtURL:singFolder() error:nil];+ [NSUserDefaults.standardUserDefaults removeObjectForKey:kPausedKey];sg_modelGeneration++;announce();}NSArray<NSString *> *SGSingComputeUnitNames(void) {- return @[@"GPU", @"GPU and Neural Engine", @"Neural Engine"];+ return @[@"Automatic", @"CPU only", @"GPU", @"Neural Engine", @"GPU and Neural Engine"];}BOOL SGSingOSSupported(void) {diff --git a/tweak/Sources/Shared/Sing/SGSingSeparator.h b/tweak/Sources/Shared/Sing/SGSingSeparator.hindex 512b5f35..9a953cf4 100644--- a/tweak/Sources/Shared/Sing/SGSingSeparator.h+++ b/tweak/Sources/Shared/Sing/SGSingSeparator.h@@ -12,7 +12,14 @@// The vocals come back from the model's complex mask times the spectrum, through the inverse STFT// (torch.istft: overlap-add of the windowed frames divided by the window's squared sum).//-// Threading: one separator is used by one thread at a time (its buffers are its own).+// Two copies of the model can stand behind one separator: the CPU's, which it is made with, and a faster one+// (the GPU's or the Neural Engine's) handed over once it is loaded and warm. A window goes to the faster copy only+// while Spotify is in the foreground (SGSingSeparatorSetForeground), since iOS refuses GPU work from the+// background; a window the faster copy fails is done again on the CPU's, and the faster copy rests until the app+// has left the foreground. The first windows on each copy are logged with their time.+//+// Threading: one separator is used by one thread at a time (its buffers are its own); the faster copy and the+// foreground can be set from any thread.#import <Foundation/Foundation.h>@class MLModel;@@ -27,9 +34,27 @@ enum {kSGSingSpectrumFloats = kSGSingBins * 2 * kSGSingSTFTFrames * 2,};+// The thermal state in a word, for the log.+const char *SGSingThermalName(void);+// Whether Spotify is in the foreground, process-wide; the faster copy is used only then.+void SGSingSeparatorSetForeground(bool foreground);++typedef struct {+ unsigned long long windows[2], fallbacks; // [0] the CPU's copy, [1] the faster one+ double averageMS[2];+ bool fast; // the last window went to the faster copy+} SGSingSeparatorStats;+@interface SGSingSeparator : NSObject// Nil `model` makes one that only does the STFT, for the harness.- (instancetype)initWithModel:(MLModel *)model;+// The faster copy, already warm, and the name the log gives it; nil takes it away.+- (void)setFastModel:(MLModel *)model named:(NSString *)name;+- (SGSingSeparatorStats)stats;+// One prediction of silence on `model`, which Core ML spends allocating and specialising, so the first real window+// does not; its time in seconds, or a negative number and `error` when the model failed or answered in a shape the+// separator does not read.++ (double)warmUp:(MLModel *)model error:(NSError **)error;// The window's spectrum in the model's layout, `spectrum` holding kSGSingSpectrumFloats.- (void)analyzeLeft:(const float *)left right:(const float *)right into:(float *)spectrum;// The inverse: kSGSingWindowFrames frames per channel out of a spectrum in the model's layout.diff --git a/tweak/Sources/Shared/Sing/SGSingSeparator.m b/tweak/Sources/Shared/Sing/SGSingSeparator.mindex 556bb9e1..e8e84f7a 100644--- a/tweak/Sources/Shared/Sing/SGSingSeparator.m+++ b/tweak/Sources/Shared/Sing/SGSingSeparator.m@@ -1,5 +1,8 @@#import <Accelerate/Accelerate.h>#import <CoreML/CoreML.h>+#import <os/lock.h>+#import <stdatomic.h>+#import "Core/SGLog.h"#import "SGSingSeparator.h"enum {@@ -9,6 +12,26 @@ enum {};static NSString *const kInput = @"spectrum", *const kOutput = @"vocals_spectrum";+// This many windows on each copy are logged with their time.+static const unsigned long long kLoggedWindows = 5;++static atomic_bool sg_foreground;+static atomic_uint sg_backgrounds; // counts the times the app left the foreground++void SGSingSeparatorSetForeground(bool foreground) {+ if (!foreground && atomic_exchange(&sg_foreground, false)) atomic_fetch_add(&sg_backgrounds, 1);+ atomic_store(&sg_foreground, foreground);+}++const char *SGSingThermalName(void) {+ switch (NSProcessInfo.processInfo.thermalState) {+ case NSProcessInfoThermalStateNominal: return "nominal";+ case NSProcessInfoThermalStateFair: return "fair";+ case NSProcessInfoThermalStateSerious: return "serious";+ case NSProcessInfoThermalStateCritical: return "critical";+ }+ return "?";+}// Where bin `bin` of channel `channel` at STFT frame `frame` starts in the model's layout (its real part;// the imaginary part follows it).@@ -18,6 +41,12 @@ static inline size_t at(int bin, int channel, int frame) {@implementation SGSingSeparator {MLModel *_model;+ os_unfair_lock _lock; // the faster copy and its name+ MLModel *_fast;+ NSString *_fastName;+ bool _fastResting; // failed a window: rests until the app has left the foreground+ unsigned _restingFrom; // sg_backgrounds when it failed+ SGSingSeparatorStats _stats; // the worker's; read whole under the lockMLMultiArray *_input;vDSP_DFT_Setup _forward, _inverse;float _window[kSGSingFFT];@@ -32,6 +61,7 @@ static inline size_t at(int bin, int channel, int frame) {- (instancetype)initWithModel:(MLModel *)model {if (!(self = [super init])) return nil;_model = model;+ _lock = OS_UNFAIR_LOCK_INIT;_forward = vDSP_DFT_zrop_CreateSetup(NULL, kSGSingFFT, vDSP_DFT_FORWARD);_inverse = vDSP_DFT_zrop_CreateSetup(_forward, kSGSingFFT, vDSP_DFT_INVERSE);_spectrum = calloc(kSGSingSpectrumFloats, sizeof(float));@@ -157,18 +187,104 @@ static BOOL readOutput(MLMultiArray *array, float *into) {return ok;}+- (void)setFastModel:(MLModel *)model named:(NSString *)name {+ os_unfair_lock_lock(&_lock);+ _fast = model;+ _fastName = name;+ _fastResting = false;+ _stats.windows[1] = 0;+ _stats.averageMS[1] = 0;+ os_unfair_lock_unlock(&_lock);+}++- (SGSingSeparatorStats)stats {+ os_unfair_lock_lock(&_lock);+ SGSingSeparatorStats stats = _stats;+ os_unfair_lock_unlock(&_lock);+ return stats;+}++// The vocals' spectrum of `features` from `model` into `into`, NO and `error` when it failed.+static BOOL predict(MLModel *model, MLDictionaryFeatureProvider *features, float *into, NSError **error) {+ id<MLFeatureProvider> result = [model predictionFromFeatures:features error:error];+ MLMultiArray *output = [result featureValueForName:kOutput].multiArrayValue;+ if (output && readOutput(output, into)) return YES;+ if (error && !*error) *error = [NSError errorWithDomain:@"SGSing" code:1 userInfo:@{NSLocalizedDescriptionKey: @"The model answered in a shape Sing does not read"}];+ return NO;+}+++ (double)warmUp:(MLModel *)model error:(NSError **)error {+ float *silence = calloc(kSGSingSpectrumFloats, sizeof(float)), *out = malloc(kSGSingSpectrumFloats * sizeof(float));+ double took = -1;+ MLMultiArray *input = silence && out ? [[MLMultiArray alloc] initWithDataPointer:silence shape:@[@1, @(2 * kSGSingBins), @(kSGSingSTFTFrames), @2]+ dataType:MLMultiArrayDataTypeFloat32+ strides:@[@(2 * kSGSingBins * kSGSingSTFTFrames * 2), @(kSGSingSTFTFrames * 2), @2, @1]+ deallocator:nil error:error] : nil;+ MLDictionaryFeatureProvider *features = input ? [[MLDictionaryFeatureProvider alloc] initWithDictionary:@{kInput: input} error:error] : nil;+ if (features) {+ CFAbsoluteTime began = CFAbsoluteTimeGetCurrent();+ @autoreleasepool {+ if (predict(model, features, out, error)) took = CFAbsoluteTimeGetCurrent() - began;+ }+ // Silence must come back as silence, not as the non-finite numbers a half-precision floor makes of it.+ for (int i = 0; took >= 0 && i < kSGSingSpectrumFloats; i++) {+ if (!isfinite(out[i])) {+ took = -1;+ if (error) *error = [NSError errorWithDomain:@"SGSing" code:2 userInfo:@{NSLocalizedDescriptionKey: @"The model turned silence into numbers that are not finite"}];+ }+ }+ }+ input = nil; // before the memory it points into+ free(silence);+ free(out);+ return took;+}++// The window's time into the copy's running average, the first windows logged.+- (void)count:(int)copy took:(double)took named:(NSString *)name {+ os_unfair_lock_lock(&_lock);+ unsigned long long n = ++_stats.windows[copy];+ double ms = took * 1000;+ _stats.averageMS[copy] = n == 1 ? ms : _stats.averageMS[copy] * 0.8 + ms * 0.2;+ _stats.fast = copy == 1;+ os_unfair_lock_unlock(&_lock);+ if (n <= kLoggedWindows) SGLog(@"sing: window %llu on the %@ copy took %.0f ms (1500 ms keeps up), thermal state %s", n, name, ms, SGSingThermalName());+}+- (BOOL)separateLeft:(const float *)left right:(const float *)right vocalsLeft:(float *)vocalsLeftvocalsRight:(float *)vocalsRight error:(NSError **)error {if (!_model) return NO;[self analyzeLeft:left right:right into:_spectrum];MLDictionaryFeatureProvider *features = [[MLDictionaryFeatureProvider alloc] initWithDictionary:@{kInput: _input} error:error];if (!features) return NO;- id<MLFeatureProvider> result = [_model predictionFromFeatures:features error:error];- MLMultiArray *output = [result featureValueForName:kOutput].multiArrayValue;- if (!output || !readOutput(output, _vocals)) {- if (error && !*error) *error = [NSError errorWithDomain:@"SGSing" code:1 userInfo:@{NSLocalizedDescriptionKey: @"The model answered in a shape Sing does not read"}];- return NO;+ bool foreground = atomic_load(&sg_foreground);+ unsigned backgrounds = atomic_load(&sg_backgrounds);+ os_unfair_lock_lock(&_lock);+ if (_fastResting && backgrounds != _restingFrom) _fastResting = false;+ MLModel *fast = foreground && !_fastResting ? _fast : nil;+ NSString *fastName = _fastName;+ os_unfair_lock_unlock(&_lock);+ if (fast) {+ CFAbsoluteTime began = CFAbsoluteTimeGetCurrent();+ NSError *fastError;+ if (predict(fast, features, _vocals, &fastError)) {+ [self count:1 took:CFAbsoluteTimeGetCurrent() - began named:fastName];+ [self synthesize:_vocals left:vocalsLeft right:vocalsRight];+ return YES;+ }+ os_unfair_lock_lock(&_lock);+ if (_fast == fast) {+ _fastResting = true;+ _restingFrom = backgrounds;+ }+ _stats.fallbacks++;+ os_unfair_lock_unlock(&_lock);+ SGLog(@"sing: the %@ copy failed a window (%@), so it is done on the CPU's, and the %@ copy rests until Spotify has been in the background",+ fastName, fastError.localizedDescription, fastName);}+ CFAbsoluteTime began = CFAbsoluteTimeGetCurrent();+ if (!predict(_model, features, _vocals, error)) return NO;+ [self count:0 took:CFAbsoluteTimeGetCurrent() - began named:@"CPU"];[self synthesize:_vocals left:vocalsLeft right:vocalsRight];return YES;}diff --git a/tweak/Sources/Shared/Sing/SGSpatialPreview.m b/tweak/Sources/Shared/Sing/SGSpatialPreview.mindex b9d32980..d347780a 100644--- a/tweak/Sources/Shared/Sing/SGSpatialPreview.m+++ b/tweak/Sources/Shared/Sing/SGSpatialPreview.m@@ -6,8 +6,9 @@// sways slowly to show the same thing.//// It is drawn once into layers and moved by Core Animation alone: the turn is the disc layer's rotation and the-// voice's position, each motion animated from where it is on screen to the next over a little more than the-// motion's interval, so nothing runs per frame on the main thread. It listens to the head only while it is in+// voice's position, each motion animated from where it is on screen over the motions' own interval to where the+// head will be by the next one, at the speed it is turning, so the disc keeps up with the head rather than a+// motion behind it, and nothing runs per frame on the main thread. It listens to the head only while it is in// a window, Spotify is in front and Motion & Fitness is allowed (the page never asks; the switch does). Under// Reduce Motion the disc holds still and a turn of the head past a step fades it to the new angle.//@@ -32,9 +33,11 @@ static const CGFloat kVoiceHeight = 12; // over the floor, as the listen// The sway while there is no head to follow: far enough to see the voice come round, slow enough to be calm,// starting and turning back with no velocity.static const double kSwayAngle = 38 * M_PI / 180, kSwaySeconds = 12;-// Each motion is animated to over this long, a little more than AirPods' interval, so the turn never stops-// between two of them.-static const CFTimeInterval kFollowSeconds = 0.12;+// A turn this long or shorter follows a motion: straight, not eased. Each motion is animated to over the+// motions' interval as heard (AirPods send about 25 a second), held within kFollowShortest and kFollowLongest,+// and leads by no more than kFollowLead.+static const CFTimeInterval kFollowSeconds = 0.12, kFollowShortest = 1.0 / 120, kFollowLongest = 0.08;+static const double kFollowLead = 12 * M_PI / 180;// Under Reduce Motion, the turn the voice must move by before the still picture fades to it.static const double kStillStep = 15 * M_PI / 180;@@ -44,6 +47,8 @@ static const double kStillStep = 15 * M_PI / 180;UILabel *_caption;CGFloat _radius;double _angle; // the disc's turn as last set, unwrapped+ double _heard; // the voice's angle in the last motion, and when that motion was (its timestamp)+ NSTimeInterval _heardAt, _interval; // the interval between motions, smoothedBOOL _listening, _live, _swaying;CMAuthorizationStatus _allowed;}@@ -219,7 +224,11 @@ static CAGradientLayer *floorLight(CGPoint at, CGFloat size, UIColor *color, CGFposition.values = positions;for (CAKeyframeAnimation *animation in @[rotation, position]) {animation.duration = seconds;- if (begin <= 0) continue;+ // A turn after the head asks for the display's full rate, so it keeps up with a quick turn.+ if (begin <= 0) {+ if (seconds <= kFollowSeconds) animation.preferredFrameRateRange = CAFrameRateRangeMake(60, 120, 120);+ continue;+ }animation.repeatCount = HUGE_VALF;animation.beginTime = begin;// A slow turn is smooth at 60 frames a second, so it asks for no more. Its maximum stays at 120: a cap at 60@@ -329,7 +338,7 @@ static CAGradientLayer *floorLight(CGPoint at, CGFloat size, UIColor *color, CGF_listening = listen;if (!listen) {SGHeadMotionListen(kListener, nil);- [self followed:NAN];+ [self followed:NAN at:0];return;}__weak typeof(self) weakSelf = self;@@ -339,23 +348,35 @@ static CAGradientLayer *floorLight(CGPoint at, CGFloat size, UIColor *color, CGFSGHeadMotionListen(kListener, ^(CMDeviceMotion *motion) {double angle = motion ? SGSpatialVoiceAngle(&front, motion.attitude.yaw, motion.timestamp) : NAN;if (!motion) front.hasFront = false;- dispatch_async(dispatch_get_main_queue(), ^{ [weakSelf followed:angle]; });+ NSTimeInterval at = motion.timestamp;+ dispatch_async(dispatch_get_main_queue(), ^{ [weakSelf followed:angle at:at]; });});}-// The voice's angle from the head's motion, NAN when the motion stopped.-- (void)followed:(double)angle {+// The voice's angle from the head's motion at its timestamp, NAN when the motion stopped.+- (void)followed:(double)angle at:(NSTimeInterval)at {BOOL live = _listening && !isnan(angle);+ double since = at - _heardAt, turned = remainder(angle - _heard, 2 * M_PI);+ _heard = angle;+ _heardAt = at;if (live != _live) {_live = live;+ _interval = 0;[self updateCaption];[self moveOn];- if (live) {- [self turnTo:angle seconds:0.5];- return;- }+ if (live) [self turnTo:angle seconds:0.5];+ return;+ }+ if (!live) return;+ // A gap (the headphones out a moment, Spotify busy) is no interval, and no speed to lead by.+ if (since <= 0 || since > 4 * kFollowLongest) {+ [self turnTo:angle seconds:kFollowLongest];+ return;}- if (live) [self turnTo:angle seconds:kFollowSeconds];+ _interval = _interval > 0 ? 0.8 * _interval + 0.2 * since : since;+ CFTimeInterval seconds = fmin(kFollowLongest, fmax(kFollowShortest, _interval));+ double lead = fmax(-kFollowLead, fmin(kFollowLead, turned / since * seconds));+ [self turnTo:angle + lead seconds:seconds];}- (void)refresh {diff --git a/tweak/Sources/Shared/Sing/Sing.h b/tweak/Sources/Shared/Sing/Sing.hindex c26d147e..48ef26f0 100644--- a/tweak/Sources/Shared/Sing/Sing.h+++ b/tweak/Sources/Shared/Sing/Sing.h@@ -3,13 +3,15 @@// under either look: the work is on Spotify's sound.//// SGSingModel.m the voice model: downloaded, each file checked against its size and SHA-256, loaded-// SGSingSeparator.m the STFT around the model: two seconds in, their vocals out+// SGSingLoader.m the model in memory: a CPU copy, then a faster one beside it, each load with a deadline+// SGSingSeparator.m the STFT around the model: two seconds in, their vocals out, on the copy that fits// SGSingEngine.m Spotify's mixer pulled ahead of what plays, the vocals separated in the lead, mixed down// and put where spatial voice holds them// Sing.x the engine put between Spotify's mixer and its output (through Shared/Player/SpeedPitch.x's// chain), Spotify's clock and seeks kept true to what plays, the heat, what Sing is doing,// and the head's turn for spatial voice (Shared/HeadGestures' motion)// SingSettings.m the Sing page, and Spatial voice's under it+// SGSingCard.m the Sing page's card: the song, Sing's state, the vocals and the rest traced live, the level// SGSpatialPreview.m the Spatial voice page's preview: the voice in front of the listener on a field of dots,// turned by the head the way the audio is//@@ -21,14 +23,18 @@// Threading: main thread, but for what Sing.x's render side and the engine say.#import <UIKit/UIKit.h>+@class SPTPlayerState;+// The mic: on and off at once, from the button or the Sing page; off until switched on.#define SGKeySing @"spotifyglass.sing"// The vocals' level, 0 (gone) through 1 (as the song has them) to 2 (the vocals alone) at the slider's top.#define SGKeySingLevel @"spotifyglass.sing.level"-// Keeps the model running on a hot iPhone, which otherwise holds it from Serious on.+// Keeps the model running on a hot iPhone, which otherwise lets it go from the thermal state Serious up.#define SGKeySingIgnoreHeat @"spotifyglass.sing.ignoreHeat"-// Where the model runs: an index into SGSingComputeUnitNames(), read as it loads.-#define SGKeySingComputeUnits @"spotifyglass.sing.computeUnits"+// Where the model runs beside the CPU: an index into SGSingComputeUnitNames(), Automatic (0) unless chosen. The+// earlier key held the one place the model ran (GPU, GPU and Neural Engine, Neural Engine) and is carried over.+#define SGKeySingComputeUnits @"spotifyglass.sing.runsOn"+#define SGKeySingComputeUnitsBefore @"spotifyglass.sing.computeUnits"// Spatial voice: through headphones that track the head, the vocals stay in front as it turns; off until// switched on, and at once.#define SGKeySingSpatial @"spotifyglass.sing.spatial"@@ -41,12 +47,13 @@ typedef NS_ENUM(NSInteger, SGSingState) {SGSingStateNoModel, // the model is not downloadedSGSingStateDownloading,SGSingStateOff,- SGSingStatePreparing, // the model loading, which Core ML specialises for the phone the first time- SGSingStateWaiting, // on, Spotify not playing through the chain yet+ SGSingStatePreparing, // the model loading (only while Spotify is active), which Core ML specialises the first time+ SGSingStateWaiting, // on, the model ready, Spotify not playing through the chain yet; or resting at As sung+ // (SGSingStatusText says which)SGSingStateBuffering, // on, the lead filling: what plays is drySGSingStateSinging, // on, the vocals downSGSingStateBehind, // on, the model slower than the song: what plays is dry until it catches up- SGSingStateHot, // on, held for the heat+ SGSingStateHot, // on, held for the heat, the model let goSGSingStateFailed, // the model or Spotify's output failed (SGSingStatusDetail says how)};SGSingState SGSingCurrentState(void);@@ -61,9 +68,20 @@ BOOL SGSingOn(void);void SGSetSingOn(BOOL on);float SGSingLevel(void);void SGSetSingLevel(float level);+// The loudness (RMS) of the vocals and of the rest of the song as heard, per tenth of a second, for the `count`+// tenths up to what plays now, oldest first; NO, and nothing written, unless Sing is turning the vocals down now.+// Reading them says a page shows them: with the vocals as sung and Spatial voice off, Sing separates only while+// they are read (at least once a second), and otherwise rests and plays the song as it is.+BOOL SGSingReadLevels(float *vocals, float *rest, int count);// A level as the slider reads it out: Gone, a percentage, As sung, Backing and a percentage, Vocals only.NSString *SGSingLevelText(double level);void SGSetSingIgnoresHeat(BOOL ignores);+// Seconds of Spotify's sound Sing holds ahead of what plays now, as the render thread left it; 0 when none.+double SGSingHeldLead(void);+// The seconds to take off `state`'s position for what is heard: the lead held when the clock's line `state` is on+// began (its first state seen), less what Sing dropped since (SGSingEngineLeadAt). Every correction for the lead reads+// this; 0 while Sing never ran.+double SGSingLeadOf(SPTPlayerState *state);// Whether this iPhone runs Sing and reads headphone motion, which spatial voice needs.BOOL SGSingSpatialAvailable(void);BOOL SGSingSpatial(void);@@ -79,7 +97,7 @@ void SGSetSingSpatial(BOOL on);@end// Spatial voice's page, its preview and its switch: under Sing's page, and on Mod Settings' main page under Sing.UIViewController *SGSpatialVoiceSettingsPage(void);-// Loads the model again on the compute units now stored, if it is loaded.+// Loads the faster copy again on the compute units now stored, the CPU's kept.void SGSingComputeUnitsChanged(void);// What the main page's Sing row reads out: On, Off, or how far the model has come.NSString *SGSingSummary(void);@@ -95,9 +113,17 @@ typedef NS_ENUM(NSInteger, SGSingModelState) {SGSingModelState SGSingModelCurrentState(void);// 0 to 1 while downloading.double SGSingModelProgress(void);-// Whether the download is held until the iPhone is online again.+// Whether the download is held until the iPhone is online again, or on Wi-Fi unless cellular is allowed.BOOL SGSingModelWaitingForNetwork(void);-// The last download's failure, nil when there was none.+// The download over cellular and Low Data Mode networks too, which it otherwise waits out; for this launch.+void SGSingDownloadModelOverCellular(void);+BOOL SGSingModelOverCellular(void);+// The weights' checksum being read, the download's last step.+BOOL SGSingModelChecking(void);+// Bytes a stopped download kept for the next one to carry on from; 0 when there are none.+long long SGSingModelPausedBytes(void);+// The last download's failure, nil when there was none (too little free space among them: the download does not+// start then).NSString *SGSingModelError(void);// The download's size, for the row that offers it.NSString *SGSingModelSizeText(void);diff --git a/tweak/Sources/Shared/Sing/Sing.x b/tweak/Sources/Shared/Sing/Sing.xindex a2cbce13..3130edd7 100644--- a/tweak/Sources/Shared/Sing/Sing.x+++ b/tweak/Sources/Shared/Sing/Sing.x@@ -4,22 +4,37 @@//// The engine sits in the chain Speed and pitch takes over (Shared/Player/SpeedPitch.x): its stage is handed// the mixer's pull, so the engine can pull the mixer ahead of what plays. It takes Spotify's sound as Spotify-// hands it to the RemoteIO unit, which this file reads when Spotify starts the unit (Spotify's import of-// AudioOutputUnitStart rebound, as Music Haptics and the audio effects do): Sing needs it at the model's-// 44.1 kHz, float, a buffer per channel, two channels, and otherwise passes it as it is.+// hands it to the music's output, whose format this file reads each time Shared/Player says that output changed,+// started again or changed format (SGPlayerWatchMusicOutput; voice search's unit or a second chain's is not the+// music's): Sing needs it at the model's 44.1 kHz, float, a buffer per channel, two channels, and otherwise passes+// it as it is. A new output or a new format drops the sound held ahead, which was in the old one.//// The lead is sound Spotify's decoder has handed over and the speaker has not played yet, and Spotify's clock-// counts what the decoder handed over. -[SPTPlayerState position] is [self positionAsOfTimestamp] run on by-// the time since (disassembly at 0x1057735ec, its -1 kept for no position), so positionAsOfTimestamp is-// hooked to take the lead off: the scrubber, the lyrics and the lock screen follow what is heard. A seek, a+// counts what the decoder handed over when the player reports. -[SPTPlayerState position] is [self+// positionAsOfTimestamp] run on by the time since [self timestamp] (disassembly at 0x1057735ec, its -1 kept for no+// position), and the player reports on a change (a seek, a pause, a track), not as it plays. So what is heard is the+// reported position less the lead held when that line of the clock began, less what was dropped since, and not+// less the lead now: a lead filling after a report does not move what is heard, and taking it off lagged the lyrics+// by it. A state stamped again on the same line (the same position run on at the same speed) is the same report, so+// the lead is looked up by the line, not by the state's stamp (SGSingLeadOf). positionAsOfTimestamp is hooked to take+// that off: the scrubber, the lyrics, the Live Activity and the lock screen follow what is heard. A seek, a// skip or a stop drops the lead (-[SPTEsperantoPlayer seekTo:...] and skipTo...TrackWithOptions:track:loggingParams:// in the binary's method list, each returning Spotify's own result for the command; the shorter skips are// trampolines into those two through objc_msgSend, 0x1096da9e4-0x1096daa08), and so does a track changing well// before the last one ended: something new was played.//-// The model is held only while the mic is on, and loaded on a queue of its own. From the thermal state-// At Critical the engine is held and plays dry, unless Ignore heat warnings is on. Serious is where a-// charging iPhone sits for hours, and iOS itself only throttles hard at Critical.+// The model is loaded by SGSingLoader.m, only while Spotify is active and the mic is on, and kept a minute after the+// mic goes off. From the thermal state Serious up the engine is held, plays the song as it is and lets the model go,+// unless Ignore heat warnings is on; it loads again once the iPhone is back at Fair. The model is what heats it, so+// Sing does not keep a hot iPhone hot. A model that falls 8 s behind stops Sing for the rest of the song, which then+// plays as it is; Sing tries again with the next track, when the iPhone cools or when Runs on changes, and stays+// stopped after three in a row with no song kept up.+// With the vocals as sung, Spatial voice off and the Sing page's lines not on screen, what Sing plays is the song+// itself, so it rests: the engine is held as for the heat and the model kept a minute as for a mic switched off.+// Whenever Sing does not separate (stopped, held, resting, standing aside), the engine plays the sound it holds ahead+// on as it is, dry, and the clock stays corrected for it: letting it go mid-song would skip that much of the song.+// It goes where nothing heard is lost: a seek, a skip, Spotify's output stopping, or a pause, where Spotify is+// seeked back to what was heard (SGSingTrackWatcher).//// Spatial voice listens to Shared/HeadGestures' motion (the app's one CMHeadphoneMotionManager) while Sing is// on and Spotify plays, and hands the engine the head's yaw off a front that follows where the head points over@@ -32,15 +47,19 @@#import <AudioToolbox/AudioToolbox.h>#import <AVFoundation/AVFoundation.h>#import <CoreML/CoreML.h>+#import <MediaPlayer/MediaPlayer.h>+#import <objc/runtime.h>+#import <os/lock.h>#import <pthread.h>#import <stdatomic.h>#import "Core/SGCore.h"-#import "Core/SGRebind.h"#import "Shared/HeadGestures/HeadGestures.h"+#import "Shared/Lyrics/Lyrics.h"#import "Shared/Player/PlayerState.h"#import "Shared/Player/SpeedPitch.h"#import "Sing.h"#import "SGSingEngine.h"+#import "SGSingLoader.h"#import "SGSingSeparator.h"// The vocals' level until one is chosen: down to a guide.@@ -54,7 +73,8 @@ static _Atomic(SGSingEngine *) sg_engine; // made the first time the mic is onstatic atomic_bool sg_formatTaken; // Spotify's output is in a format Sing takesstatic atomic_bool sg_staged; // the stage has run: Spotify's sound comes through the chainstatic atomic_bool sg_formatRefused; // the stage was handed buffers it does not take-static atomic_bool sg_outputStarted; // Spotify has started its RemoteIO unit+static atomic_bool sg_outputStarted; // Spotify has started or connected the music's output+static BOOL sg_outputReachable; // Shared/Player watches the music's output for Sing#pragma mark - the render thread@@ -87,104 +107,108 @@ static OSStatus stage(UInt32 frames, AudioBufferList *data, SGPlayerPull pull, v#pragma mark - Spotify's audio thread-static OSStatus (*sg_startOutput)(AudioUnit unit);--static OSStatus startOutput(AudioUnit unit) {- AudioComponentDescription description = {0};- if (unit && AudioComponentGetDescription(AudioComponentInstanceGetComponent(unit), &description) == noErr- && description.componentType == kAudioUnitType_Output && description.componentSubType == kAudioUnitSubType_RemoteIO) {- atomic_store_explicit(&sg_outputStarted, true, memory_order_relaxed);- AudioStreamBasicDescription format = {0};- UInt32 size = sizeof format;- OSStatus status = AudioUnitGetProperty(unit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Input, 0, &format, &size);- BOOL takes = status == noErr && format.mFormatID == kAudioFormatLinearPCM && format.mSampleRate == kSGSingRate- && format.mChannelsPerFrame == 2 && format.mBitsPerChannel == 32 && (format.mFormatFlags & kAudioFormatFlagIsFloat)- && (format.mFormatFlags & kAudioFormatFlagIsNonInterleaved);- if (takes != atomic_exchange(&sg_formatTaken, takes) || !takes) {- SGLog(@"sing: Spotify hands its output %.0f Hz, %u channels, %u bits, flags 0x%x: %@", format.mSampleRate,- (unsigned)format.mChannelsPerFrame, (unsigned)format.mBitsPerChannel, (unsigned)format.mFormatFlags,- takes ? @"Sing takes it" : @"not a format Sing takes, its sound passes as it is");- }+// Under Shared/Player's lock, one call at a time: the last output and format seen are this function's own.+static void musicOutputChanged(AudioUnit output) {+ static AudioUnit last;+ static AudioStreamBasicDescription lastFormat;+ AudioStreamBasicDescription format = {0};+ BOOL known = output && SGPlayerMusicClientFormat(&format);+ atomic_store_explicit(&sg_outputStarted, output != NULL, memory_order_relaxed);+ BOOL takes = known && format.mFormatID == kAudioFormatLinearPCM && format.mSampleRate == kSGSingRate+ && format.mChannelsPerFrame == 2 && format.mBitsPerChannel == 32 && (format.mFormatFlags & kAudioFormatFlagIsFloat)+ && (format.mFormatFlags & kAudioFormatFlagIsNonInterleaved);+ BOOL changed = output != last || memcmp(&format, &lastFormat, sizeof format) != 0;+ last = output;+ lastFormat = format;+ // The sound held ahead is the old output's or the old format's: dropped before the next render plays any of it.+ SGSingEngine *engine = atomic_load_explicit(&sg_engine, memory_order_acquire);+ if (changed && engine) SGSingEngineFlush(engine);+ if (takes != atomic_exchange(&sg_formatTaken, takes) || (changed && !takes)) {+ SGLog(@"sing: Spotify hands the music's output %.0f Hz, %u channels, %u bits, flags 0x%x: %@", format.mSampleRate,+ (unsigned)format.mChannelsPerFrame, (unsigned)format.mBitsPerChannel, (unsigned)format.mFormatFlags,+ takes ? @"Sing takes it" : output ? @"not a format Sing takes, its sound passes as it is" : @"no music output now");}- return sg_startOutput(unit);}#pragma mark - the model-static SGSingSeparator *sg_separator;-static BOOL sg_loading;-static NSString *sg_loadError;-static NSUInteger sg_loadGeneration; // counts the mic's switches, so a load finished after it went off is dropped+static NSString *sg_setupError; // the engine could not be made+static NSString *sg_stopped; // why Sing stopped by itself, with the switch still on+static NSString *sg_refused; // why Sing stands aside for what plays now (AirPlay, not a song); clears by itself++// What stopped it. Falling behind tries again with the next track, when the iPhone cools or when Runs on changes;+// a memory warning with the next track once there is room for the model; this many falls behind in a row, with no+// song kept up between them, stay stopped until the switch goes off and on, so a phone that can never keep up does+// not start and give up all day.+typedef NS_ENUM(NSInteger, SGSingStop) { SGSingStopNone, SGSingStopBehind, SGSingStopMemory, SGSingStopForGood };+static SGSingStop sg_stopKind;+static const int kGiveUpsKept = 3;+static int sg_giveUps; // falls behind since a song was last kept up+static BOOL sg_sangThisTrack; // the vocals were down at some point of the track playing++// AirPlay's delay changes as it plays, and a podcast or an ad has no song to take the vocals from: Sing stands+// aside, the song plays as it is, and the model is kept for when it can run again. Whether that changed.+static BOOL readRefusal(void) {+ NSString *refused = nil;+ for (AVAudioSessionPortDescription *port in AVAudioSession.sharedInstance.currentRoute.outputs) {+ if ([port.portType isEqualToString:AVAudioSessionPortAirPlay]) refused = @"Sing does not run over AirPlay, whose delay changes as it plays, so the song plays as it is.";+ }+ NSString *uri = SGURIString(SGPlayerState().track.URI);+ if (!refused && uri && ![uri hasPrefix:@"spotify:track:"] && ![uri hasPrefix:@"spotify:local:"]) {+ refused = @"Sing turns down the vocals of songs, and what plays now is not one, so it plays as it is.";+ }+ if (refused == sg_refused || [refused isEqualToString:sg_refused]) return NO;+ sg_refused = refused;+ SGLog(@"sing: %@", refused ?: @"runs again: what plays is a song, not over AirPlay");+ return YES;+}static BOOL sg_hot;+static BOOL sg_active; // Spotify is the active app: the only time a load startsstatic void announce(void) {[NSNotificationCenter.defaultCenter postNotificationName:SGSingChangedNotification object:nil];}-static dispatch_queue_t loadQueue(void) {- static dispatch_queue_t queue;- static dispatch_once_t once;- dispatch_once(&once, ^{- queue = dispatch_queue_create("spotifyglass.sing.load", dispatch_queue_attr_make_with_qos_class(DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INITIATED, 0));- });- return queue;-}--// The Neural Engine unless changed: on an iPhone 15 Pro the GPU never finished its first load of this model.-static MLComputeUnits computeUnits(void) {- switch (SGInt(SGKeySingComputeUnits, 2)) {- case 0: return MLComputeUnitsCPUAndGPU;- case 1: return MLComputeUnitsAll;- default: return MLComputeUnitsCPUAndNeuralEngine;- }-}--static void loadModel(void) {- NSURL *url = SGSingModelURL();- if (sg_separator || sg_loading || !url || SGSingMissing()) return;- sg_loading = YES;- sg_loadError = nil;- NSUInteger generation = sg_loadGeneration;- MLComputeUnits units = computeUnits();- SGLog(@"sing: loading the voice model on the %@", SGSingComputeUnitNames()[(NSUInteger)MIN(MAX(SGInt(SGKeySingComputeUnits, 2), 0), 2)]);- // A load that takes long says so, so a stall reads apart from a slow first compile.- __block void (^still)(int) = nil;- still = ^(int seconds) {- dispatch_after(dispatch_time(DISPATCH_TIME_NOW, 30 * NSEC_PER_SEC), dispatch_get_main_queue(), ^{- if (!sg_loading || generation != sg_loadGeneration) { still = nil; return; }- SGLog(@"sing: still loading the voice model, %d s", seconds + 30);- still(seconds + 30);- });- };- still(0);- dispatch_async(loadQueue(), ^{- CFAbsoluteTime began = CFAbsoluteTimeGetCurrent();- MLModelConfiguration *configuration = [MLModelConfiguration new];- configuration.computeUnits = units;- NSError *error;- MLModel *model = [MLModel modelWithContentsOfURL:url configuration:configuration error:&error];- SGSingSeparator *separator = model ? [[SGSingSeparator alloc] initWithModel:model] : nil;- double took = CFAbsoluteTimeGetCurrent() - began;- dispatch_async(dispatch_get_main_queue(), ^{- sg_loading = NO;- if (!separator) {- sg_loadError = error.localizedDescription ?: @"The voice model could not be loaded";- SGLog(@"sing: the voice model did not load: %@", sg_loadError);- } else if (generation == sg_loadGeneration) {- sg_separator = separator;- SGSingEngineSetSeparator(atomic_load(&sg_engine), separator);- SGLog(@"sing: the voice model is loaded in %.1f s", took);- }- announce();- });- });+// Runs on: Automatic, CPU only, GPU, Neural Engine, GPU and Neural Engine. Automatic is the GPU beside the CPU: on an+// M4 the GPU's copy runs a window in 405-525 ms against the CPU's 1.2 s, while the Neural Engine's copy, which+// leaves the model's float32 operations to the CPU, took 6.25 s. On an iPhone 15 Pro the GPU's load never came+// back from a background launch; it now loads only in the foreground, and a GPU load that runs past its deadline+// with Spotify active throughout keeps Automatic on the CPU alone for this iOS, until Runs on is changed.+static NSString *const kGPUStuckOn = @"spotifyglass.sing.gpuStuckOn";++static NSString *osBuild(void) {+ return NSProcessInfo.processInfo.operatingSystemVersionString;+}++static NSInteger runsOn(void) {+ return SGInt(SGKeySingComputeUnits, 0);+}++static BOOL gpuStuck(void) {+ return [[NSUserDefaults.standardUserDefaults stringForKey:kGPUStuckOn] isEqualToString:osBuild()];+}++static MLComputeUnits fastUnits(void) {+ switch (runsOn()) {+ case 1: return MLComputeUnitsCPUOnly;+ case 2: return MLComputeUnitsCPUAndGPU;+ case 3: return MLComputeUnitsCPUAndNeuralEngine;+ case 4: return MLComputeUnitsAll;+ default: return gpuStuck() ? MLComputeUnitsCPUOnly : MLComputeUnitsCPUAndGPU;+ }}-static void unloadModel(void) {- sg_loadGeneration++;- sg_separator = nil;+// The loader's news: the separator into the engine, a GPU load stuck in the foreground remembered.+static void loaderChanged(void) {SGSingEngine *engine = atomic_load(&sg_engine);- if (engine) SGSingEngineSetSeparator(engine, nil);+ if (engine) SGSingEngineSetSeparator(engine, SGSingLoaderSeparator());+ if (SGSingLoaderFastState() == SGSingFastTimedOut && SGSingLoaderFastTimedOutActive() && runsOn() == 0+ && SGSingLoaderFastUnits() == MLComputeUnitsCPUAndGPU && !gpuStuck()+ && NSProcessInfo.processInfo.thermalState < NSProcessInfoThermalStateSerious) {+ [NSUserDefaults.standardUserDefaults setObject:osBuild() forKey:kGPUStuckOn];+ SGLog(@"sing: the GPU did not load on %@ with Spotify active, so Automatic stays on the CPU on this iOS", osBuild());+ }+ announce();}#pragma mark - spatial voice@@ -232,50 +256,120 @@ BOOL SGSingSpatial(void) {return SGHidden(SGKeySingSpatial);}+static void updateRest(void);+void SGSetSingSpatial(BOOL on) {SGSetEnabled(SGKeySingSpatial, on);// Asked while the Sing page is in front, rather than with the next song.if (on) SGHeadMotionAskPermission();updateSpatial();+ updateRest();}#pragma mark - the statestatic void watch(void);+// Resting: nothing needs the model, so the engine is held as for the heat and the song plays straight. The vocals are+// as sung (which plays exactly the song), Spatial voice is off, and no page traces the lines: the Sing page's card+// reads them ten times a second while it shows (SGSingReadLevels), so a second without a read is the page gone.+static const CFTimeInterval kLinesGoneAfter = 1;+static CFAbsoluteTime sg_linesReadAt;+static BOOL sg_resting;++static BOOL restful(void) {+ BOOL held = sg_hot && !SGHidden(SGKeySingIgnoreHeat);+ return SGSingOn() && !SGSingMissing() && !held && !sg_stopped && !sg_refused && fabsf(SGSingLevel() - 1) <= 0.001f && !SGSingSpatial()+ && CFAbsoluteTimeGetCurrent() - sg_linesReadAt > kLinesGoneAfter;+}+static void apply(void) {BOOL on = SGSingOn() && !SGSingMissing();SGSingEngine *engine = atomic_load(&sg_engine);if (on && !engine) {engine = SGSingEngineCreate();if (!engine) {- sg_loadError = @"Sing could not set aside memory for the song";+ sg_setupError = @"Sing could not set aside memory for the song.";return;}SGSingEngineSetLevel(engine, SGSingLevel());atomic_store_explicit(&sg_engine, engine, memory_order_release);}+ BOOL held = sg_hot && !SGHidden(SGKeySingIgnoreHeat);+ // Off, the model is kept a minute for a quick off and on. Held for the heat, it is let go at once, which frees+ // its memory and stops its work; it loads again once the iPhone cools.+ // ponytail: a phone flickering about Serious loads and drops the model each time; add a cooling-off delay if logs show it.+ // Stopped by itself, the song plays as it is and the switch stays on to say why; the model is kept, but for a+ // memory warning's stop, which let it go. Resting, the model is kept as for a mic switched off.+ BOOL rest = restful(), toRest = rest && !sg_resting;+ if (rest != sg_resting) {+ sg_resting = rest;+ if (rest) SGLog(@"sing: rests, the vocals as sung, Spatial voice off and the lines not shown: the song plays as it is, the model kept %.0f s",+ SGSingLoaderKeepSeconds);+ else if (on && !held && !sg_stopped && !sg_refused) SGLog(@"sing: separates again (the vocals at %.2f, Spatial voice %@, the lines %@)", SGSingLevel(),+ SGSingSpatial() ? @"on" : @"off", CFAbsoluteTimeGetCurrent() - sg_linesReadAt > kLinesGoneAfter ? @"not shown" : @"shown");+ }+ if (sg_stopped || sg_refused) on = NO;+ else if (!on) SGSingLoaderRelease();+ else if (held) SGSingLoaderPurge(@"the iPhone is too hot (thermal state serious or above)");+ else if (rest) { if (toRest) SGSingLoaderRelease(); }+ else if (sg_active) SGSingLoaderWant(SGSingModelURL(), fastUnits());if (!engine) return;- SGSingEngineSetPaused(engine, sg_hot && !SGHidden(SGKeySingIgnoreHeat));+ SGSingEngineSetSeparator(engine, SGSingLoaderSeparator());+ SGSingEngineSetPaused(engine, held || rest);SGSingEngineSetOn(engine, on);- if (on) loadModel();- else unloadModel();+ double lead = SGSingEngineLead(engine);+ if ((!on || held || rest) && lead > 0.05) SGLog(@"sing: keeps the %.2f s held ahead, played as it is, until Spotify pauses, seeks or skips", lead);updateSpatial();watch();}+// Called as often as the slider moves: applies only when resting would change.+static void updateRest(void) {+ if (restful() != sg_resting) apply();+}+BOOL SGSingOn(void) {return SGHidden(SGKeySing);}void SGSetSingOn(BOOL on) {SGSetEnabled(SGKeySing, on);- sg_loadError = nil; // switched on again, a failed load is tried again+ sg_stopped = nil;+ sg_stopKind = SGSingStopNone;+ sg_giveUps = 0;SGLog(@"sing: the mic is %@", on ? @"on" : @"off");apply();announce();}+// A stop for falling behind is over (and one for memory too, with `memory` and room for the model): Sing tries again.+static void retry(NSString *why, BOOL memory) {+ BOOL behind = sg_stopKind == SGSingStopBehind, lowMemory = memory && sg_stopKind == SGSingStopMemory && SGSingLoaderHasRoom();+ if (!SGSingOn() || (!behind && !lowMemory)) return;+ sg_stopped = nil;+ sg_stopKind = SGSingStopNone;+ SGLog(@"sing: tries again, %@ (%d fell behind in a row)", why, sg_giveUps);+ apply();+ announce();+}++BOOL SGSingReadLevels(float *vocals, float *rest, int count) {+ // A page tracing the lines is on screen: Sing separates for it.+ sg_linesReadAt = CFAbsoluteTimeGetCurrent();+ if (sg_resting) updateRest();+ SGSingEngine *engine = atomic_load(&sg_engine);+ if (!engine || SGSingCurrentState() != SGSingStateSinging) return NO;+ SGSingEngineReadLevels(engine, vocals, rest, count);+ // As heard: the vocals' gain up to 1, then the rest's down to nothing at 2.+ float level = SGSingLevel(), vocalsGain = fminf(level, 1), restGain = level <= 1 ? 1 : 2 - level;+ for (int k = 0; k < count; k++) {+ vocals[k] *= vocalsGain;+ rest[k] *= restGain;+ }+ return YES;+}+float SGSingLevel(void) {NSNumber *stored = [NSUserDefaults.standardUserDefaults objectForKey:SGKeySingLevel];return stored ? fmaxf(0, fminf(stored.floatValue, 2)) : kDefaultLevel;@@ -286,11 +380,18 @@ void SGSetSingLevel(float level) {[NSUserDefaults.standardUserDefaults setFloat:level forKey:SGKeySingLevel];SGSingEngine *engine = atomic_load(&sg_engine);if (engine) SGSingEngineSetLevel(engine, level);+ updateRest();}void SGSingComputeUnitsChanged(void) {- unloadModel();- apply();+ // A choice made is a new try, the GPU's included.+ [NSUserDefaults.standardUserDefaults removeObjectForKey:kGPUStuckOn];+ SGLog(@"sing: Runs on is now %@", SGSingComputeUnitNames()[(NSUInteger)MIN(MAX(runsOn(), 0), 4)]);+ // Somewhere else to run is a fresh try, however often it fell behind before.+ sg_giveUps = 0;+ if (sg_stopKind == SGSingStopForGood) sg_stopKind = SGSingStopBehind;+ retry(@"Runs on changed", NO);+ if (SGSingOn()) apply();announce();}@@ -313,16 +414,19 @@ SGSingState SGSingCurrentState(void) {if (model == SGSingModelDownloading) return SGSingStateDownloading;if (model != SGSingModelReady) return SGSingStateNoModel;if (!SGSingOn()) return SGSingStateOff;- SGSingEngine *engine = atomic_load(&sg_engine);- if (sg_loadError || !engine || (engine && SGSingEngineError(engine))) return SGSingStateFailed;- if (sg_loading || !sg_separator) return SGSingStatePreparing;+ // Held for the heat, the model is let go: held, not loading.if (sg_hot && !SGHidden(SGKeySingIgnoreHeat)) return SGSingStateHot;+ SGSingEngine *engine = atomic_load(&sg_engine);+ if (sg_stopped || sg_refused || SGSingLoaderError() || !engine || SGSingEngineError(engine)) return SGSingStateFailed;+ // Resting, ready for the level to move, whether the model is still kept or not.+ if (sg_resting) return SGSingStateWaiting;+ if (!SGSingLoaderSeparator()) return SGSingStatePreparing;// Spotify plays and Speed and pitch never took its output over: the stage, and so Sing, never runs.if (!atomic_load(&sg_staged) && atomic_load(&sg_outputStarted) && !SGPlayerSpeedAllowed()) return SGSingStateFailed;- if (!sg_startOutput || (atomic_load(&sg_staged) && atomic_load(&sg_formatRefused) && !atomic_load(&sg_formatTaken))) return SGSingStateFailed;+ if (!sg_outputReachable || (atomic_load(&sg_staged) && atomic_load(&sg_formatRefused) && !atomic_load(&sg_formatTaken))) return SGSingStateFailed;SGSingEngineStats stats = SGSingEngineReadStats(engine);if (!atomic_load(&sg_staged) || stats.lead < 0.05) return SGSingStateWaiting;- if (stats.ready > 0) return SGSingStateSinging;+ if (stats.mixing) return SGSingStateSinging;// The lead is full and the vocals are still not in: the model is slower than the song.if (stats.windows && (stats.averageMS > kSGSingEngineHop * 1000.0 / kSGSingRate || stats.lead >= stats.targetLead - 0.25)) return SGSingStateBehind;return SGSingStateBuffering;@@ -333,20 +437,45 @@ NSString *SGSingStatusText(void) {case SGSingStateUnavailable: return @"Unavailable";case SGSingStateNoModel: return @"Needs its voice model";case SGSingStateDownloading:- if (SGSingModelWaitingForNetwork()) return @"Waiting for the network";+ if (SGSingModelChecking()) return @"Checking the download";+ if (SGSingModelWaitingForNetwork()) return SGSingModelOverCellular() ? @"Waiting for the network" : @"Waiting for Wi-Fi";return [NSString stringWithFormat:@"Downloading %.0f%%", SGSingModelProgress() * 100];case SGSingStateOff: return @"Off";- case SGSingStatePreparing: return @"Preparing";- case SGSingStateWaiting: return @"Waiting for Spotify";+ case SGSingStatePreparing: {+ NSTimeInterval seconds = SGSingLoaderSeconds();+ return seconds >= 1 ? [NSString stringWithFormat:@"Loading the voice model, %.0f s", seconds] : @"Loading the voice model";+ }+ case SGSingStateWaiting: return sg_resting ? @"As sung" : @"Ready";case SGSingStateBuffering: return @"Listening ahead";case SGSingStateSinging: return @"On";case SGSingStateBehind: return @"Too slow";case SGSingStateHot: return @"Held, too hot";- case SGSingStateFailed: return @"Failed";+ case SGSingStateFailed:+ if (!sg_stopped && sg_refused) return [sg_refused containsString:@"AirPlay"] ? @"Off over AirPlay" : @"Songs only";+ return sg_stopped ? @"Stopped" : @"Failed";}return @"";}+// Where the model runs now, for the details of a working Sing.+static NSString *runsOnNote(void) {+ MLComputeUnits units = SGSingLoaderFastUnits();+ NSString *name = SGSingUnitsName(units);+ if (units == MLComputeUnitsCPUOnly) {+ return runsOn() == 0 ? @"The GPU did not load on this iOS before, so Sing runs on the CPU alone. Choosing GPU under Runs on tries it again."+ : @"Sing runs on the CPU alone, as Runs on says.";+ }+ switch (SGSingLoaderFastState()) {+ case SGSingFastNone: return [NSString stringWithFormat:@"Sing runs on the CPU; a copy for the %@ loads once Spotify is open.", name];+ case SGSingFastLoading: return [NSString stringWithFormat:@"Sing runs on the CPU while a copy for the %@ loads.", name];+ case SGSingFastReady: return [NSString stringWithFormat:@"Sing runs on the %@ while Spotify is open, and on the CPU in the background.", name];+ case SGSingFastSkipped: return [NSString stringWithFormat:@"Too little memory is left for a copy on the %@, so Sing runs on the CPU alone.", name];+ case SGSingFastFailed: return [NSString stringWithFormat:@"The %@ could not load the voice model, so Sing runs on the CPU alone.", name];+ case SGSingFastTimedOut: return [NSString stringWithFormat:@"The %@ did not load the voice model in %.0f minutes, so Sing runs on the CPU alone.", name, SGSingLoaderFastDeadline / 60];+ }+ return nil;+}+NSString *SGSingStatusDetail(void) {switch (SGSingCurrentState()) {case SGSingStateUnavailable:@@ -354,28 +483,45 @@ NSString *SGSingStatusDetail(void) {return SGSingMissing();case SGSingStateDownloading:if (SGSingModelWaitingForNetwork()) {+ if (!SGSingModelOverCellular()) {+ return [NSString stringWithFormat:@"The voice model's download waits for Wi-Fi, as %@ is a lot of a cellular plan, and carries on from %.0f%% once "+ @"the iPhone is on Wi-Fi. The Voice model row on the Sing page can let it use cellular.",+ SGSingModelSizeText(), SGSingModelProgress() * 100];+ }return [NSString stringWithFormat:@"The iPhone is offline. The voice model's download carries on from %.0f%% of %@ once it is online again.",SGSingModelProgress() * 100, SGSingModelSizeText()];}return [NSString stringWithFormat:@"The voice model is coming in, %.0f%% of %@. Sing starts once it is checked.", SGSingModelProgress() * 100, SGSingModelSizeText()];case SGSingStateWaiting:- return @"Sing starts once Spotify plays a song.";+ if (sg_resting) {+ return @"The vocals are as sung and Spatial voice is off, so the song plays as it is and the voice model rests. Sing starts again "+ @"when the level moves, Spatial voice goes on, or the Sing page opens.";+ }+ return [NSString stringWithFormat:@"The voice model is ready, and Sing starts once Spotify plays a song. %@", runsOnNote()];case SGSingStatePreparing:- return @"Core ML prepares the voice model for this iPhone. The first time, that can take a minute.";+ if (SGSingLoaderCurrentState() == SGSingLoaderIdle) return @"The voice model loads once Spotify is open.";+ return [NSString stringWithFormat:@"Core ML prepares the voice model for this iPhone's CPU, %.0f s so far, then Sing starts. The first time, that can take a minute.",+ SGSingLoaderSeconds()];case SGSingStateBuffering:return @"Sing listens a few seconds ahead of what plays, so the vocals are separated before you hear them. Until then the song plays as it is.";+ case SGSingStateSinging:+ return runsOnNote();case SGSingStateBehind: {SGSingEngine *engine = atomic_load(&sg_engine);double ms = engine ? SGSingEngineReadStats(engine).averageMS : 0;- return [NSString stringWithFormat:@"The voice model takes %.1f s for every 1.5 s of song on this iPhone, so the vocals stay in until it catches up. Another choice under Runs on may be faster.", ms / 1000];+ return [NSString stringWithFormat:@"The voice model takes %.1f s for every 1.5 s of song on this iPhone, so the vocals stay in until it catches up. %@",+ ms / 1000, runsOnNote()];}case SGSingStateHot:- return @"The iPhone is hot, so Sing is held and the song plays as it is until it cools down. Ignore heat warnings keeps Sing going.";+ return @"The iPhone is hot, so Sing has let go of the voice model and the song plays as it is. Sing loads it again once the iPhone cools. "+ @"Ignore heat warnings keeps Sing going.";case SGSingStateFailed: {SGSingEngine *engine = atomic_load(&sg_engine);- NSString *error = sg_loadError ?: (engine ? SGSingEngineError(engine) : nil);+ if (sg_stopped ?: sg_refused) return sg_stopped ?: sg_refused;+ NSString *error = SGSingLoaderError() ?: (engine ? SGSingEngineError(engine) : sg_setupError);+ if (SGSingLoaderError()) return [error stringByAppendingString:@" Switch Sing off and on again to try again."];if (error) return error;- if (!sg_startOutput) return @"Spotify's output could not be reached.";+ if (!sg_outputReachable) return @"Spotify's output could not be reached.";if (!atomic_load(&sg_staged) && !SGPlayerSpeedAllowed()) return @"Spotify's output could not be taken over, so Sing cannot reach its sound.";return @"Spotify's output is not 44.1 kHz stereo, which the voice model needs, so the song plays as it is.";}@@ -401,7 +547,7 @@ static SGSingState sg_shown = -1;static NSString *sg_shownText;static NSString *sg_lastTrack;static double sg_lastRaw = -1, sg_lastDuration;-static CFAbsoluteTime sg_lastFlush;+static _Atomic CFAbsoluteTime sg_lastFlush; // Sing.x's own flushes: each comes with a report of Spotify'sstatic __thread BOOL sg_rawPosition; // the hook below hands Spotify's own value backstatic double rawPosition(SPTPlayerState *state) {@@ -411,15 +557,138 @@ static double rawPosition(SPTPlayerState *state) {return position;}+double SGSingHeldLead(void) {+ return SGSingEngineLead(atomic_load_explicit(&sg_engine, memory_order_acquire));+}++// Spotify's clock runs on lines: a position at a moment, run on at the playback speed (paused, it holds). A seek, a+// track, a pause or a resume starts a new line; a state that is the same line stamped again later is not a new+// report of what the decoder handed over, so its lead is the one at the line's first state seen, not at its own+// timestamp. Two states are one line when they put the clock within kSameLine of each other at the same speed.+typedef struct {+ double origin; // the position at the reference date, running; the held position, paused+ double speed;+ bool paused;+ CFAbsoluteTime firstAt; // when the line began, by the first state of it seen+} ClockLine;+enum { kLines = 8 };+static const double kSameLine = 0.25;+static ClockLine sg_lines[kLines];+static unsigned sg_lineCount;+static os_unfair_lock sg_linesLock = OS_UNFAIR_LOCK_INIT;++static double rawAsOf(SPTPlayerState *state) {+ BOOL was = sg_rawPosition;+ sg_rawPosition = YES;+ double position = state.positionAsOfTimestamp;+ sg_rawPosition = was;+ return position;+}++// When the line `state` is on began, for the lead held then: the stamp of the first state of it seen. The stamp is this+// iPhone's clock, as -position runs on from it by timeIntervalSinceNow.+// ponytail: a line first seen through a state stamped after it began reads the lead of that later moment; Spotify's+// own UI, the lyrics and tick read the player at least twice a second, so that is a fill's half second at most.+static CFAbsoluteTime lineStart(SPTPlayerState *state, CFAbsoluteTime at, double asOf) {+ double speed = [state respondsToSelector:@selector(playbackSpeed)] ? state.playbackSpeed : 1;+ if (!(speed > 0)) speed = 1;+ bool paused = state.isPaused;+ double origin = paused ? asOf : asOf - at * speed;+ CFAbsoluteTime first = at;+ os_unfair_lock_lock(&sg_linesLock);+ bool found = false;+ for (unsigned k = 0; k < MIN(sg_lineCount, (unsigned)kLines) && !found; k++) {+ ClockLine *line = &sg_lines[(sg_lineCount - 1 - k) % kLines];+ // A state stamped after a seek, a skip or a pause's seek back is that report, a new line even where it runs on+ // from the last (a seek back to what was heard).+ CFAbsoluteTime flushed = atomic_load_explicit(&sg_lastFlush, memory_order_relaxed);+ bool reported = line->firstAt < flushed && flushed <= at;+ if (!reported && line->paused == paused && fabs(line->speed - speed) < 1e-3 && fabs(line->origin - origin) < kSameLine * (paused ? 1 : speed)) {+ first = line->firstAt;+ found = true;+ }+ }+ if (!found) sg_lines[sg_lineCount++ % kLines] = (ClockLine){origin, speed, paused, at};+ os_unfair_lock_unlock(&sg_linesLock);+ return first;+}++double SGSingLeadOf(SPTPlayerState *state) {+ SGSingEngine *engine = atomic_load_explicit(&sg_engine, memory_order_acquire);+ if (!engine || !state) return 0;+ NSDate *at = [state respondsToSelector:@selector(timestamp)] ? state.timestamp : nil;+ double asOf = rawAsOf(state);+ if (![at isKindOfClass:NSDate.class] || asOf < 0) return SGSingEngineLead(engine);+ return SGSingEngineLeadAt(engine, lineStart(state, at.timeIntervalSinceReferenceDate, asOf));+}++// The last absolute seek's target and the frames pulled when it was asked: from then to the next track, what is heard+// is the target run on by the frames played since, whatever Spotify's clock does. For the log only.+static double sg_seekTarget = -1;+static uint64_t sg_seekWritten;++static void seekAnchor(double target) {+ SGSingEngine *engine = atomic_load(&sg_engine);+ if (!engine) return;+ sg_seekTarget = target;+ sg_seekWritten = SGSingEngineReadStats(engine).written;+}++// Every 5 s while a lead is held or was: Spotify's clock against what the engine pulled and played, for the state+// Shared/Player reports and the one the lyrics read, so a phone's log says which clock Spotify keeps.+static void logClock(SGSingEngine *engine) {+ static CFAbsoluteTime last;+ static SGSingEngineStats before;+ static double rawBefore = -1;+ CFAbsoluteTime now = CFAbsoluteTimeGetCurrent();+ if (!engine || now - last < 5) return;+ SGSingEngineStats stats = SGSingEngineReadStats(engine);+ if (stats.lead <= 0 && stats.dropped == before.dropped && last) {+ last = now;+ return;+ }+ NSMutableString *line = [NSMutableString stringWithFormat:@"sing: clock: lead %.2f s held, %.2f s dropped", stats.lead, stats.dropped];+ double elapsed = last ? now - last : 0;+ if (elapsed > 0) {+ [line appendFormat:@"; over %.1f s Spotify's mixer gave %.2f s, %.2f s played", elapsed, (double)(stats.written - before.written) / kSGSingRate,+ (double)(stats.played - before.played) / kSGSingRate];+ }+ SPTPlayerState *reported = SGPlayerState();+ id player = SGKaraokePlayer();+ SPTPlayerState *read = [player respondsToSelector:@selector(state)] ? [(id<SPTPlayer>)player state] : nil;+ NSArray *states = read && read != reported ? @[reported ?: (id)NSNull.null, read] : @[reported ?: (id)NSNull.null];+ for (NSUInteger i = 0; i < states.count; i++) {+ SPTPlayerState *state = states[i];+ if (![state isKindOfClass:objc_getClass("SPTPlayerState")]) continue;+ double raw = rawPosition(state), correction = SGSingLeadOf(state);+ NSDate *at = [state respondsToSelector:@selector(timestamp)] ? state.timestamp : nil;+ [line appendFormat:@"; %@ state: Spotify's position %.2f s (as of %.2f s, stamped %.2f s ago%@), %.2f s taken off", i ? @"the lyrics'" : @"the reported",+ raw, rawAsOf(state), at ? now - at.timeIntervalSinceReferenceDate : -1, state.isPaused ? @", paused" : @"", correction];+ if (i == 0 && rawBefore >= 0 && elapsed > 0) [line appendFormat:@", moved %.2f s", raw - rawBefore];+ if (i == 0) rawBefore = raw;+ if (i == 0 && sg_seekTarget >= 0) {+ double heard = sg_seekTarget + ((double)stats.played - (double)sg_seekWritten) / kSGSingRate;+ [line appendFormat:@", heard by the last seek %.2f s (Spotify's position less that: %.2f s)", heard, raw - heard];+ }+ }+ SGLog(@"%@", line);+ last = now;+ before = stats;+}++static void flush(NSString *why) {SGSingEngine *engine = atomic_load(&sg_engine);- if (!engine || SGSingEngineLead(engine) <= 0) return;+ double lead = SGSingHeldLead();+ if (!engine || lead <= 0) return;sg_lastFlush = CFAbsoluteTimeGetCurrent();SGSingEngineFlush(engine);- SGLog(@"sing: %@, the %.1f s held ahead are dropped", why, SGSingEngineLead(engine));+ SGLog(@"sing: %@, the %.1f s held ahead are dropped", why, lead);}static void tick(void) {+ // The Sing page gone a second ago: Sing may rest.+ updateRest();SGSingState state = SGSingCurrentState();NSString *text = SGSingStatusText();if (state != sg_shown || ![text isEqualToString:sg_shownText]) {@@ -431,19 +700,44 @@ static void tick(void) {if (player) {sg_lastRaw = rawPosition(player);sg_lastDuration = player.duration;+ // Seen soon after it is reported, its lead is kept for as long as it stands (SGSingEngineLeadAt).+ SGSingLeadOf(player);}SGSingEngine *engine = atomic_load(&sg_engine);+ logClock(engine);+ if (state == SGSingStateSinging) sg_sangThisTrack = YES;+ if (engine && SGSingOn() && !sg_stopped && SGSingEngineGaveUp(engine)) {+ BOOL forGood = ++sg_giveUps >= kGiveUpsKept;+ sg_stopKind = forGood ? SGSingStopForGood : SGSingStopBehind;+ sg_stopped = forGood ? [NSString stringWithFormat:@"Sing could not keep up on this iPhone %d times in a row, so it stopped and songs play as they are. "+ @"Switch Sing off and on, or choose another Runs on, to try again.", sg_giveUps]+ : @"Sing could not keep up with this song on this iPhone, so it plays as it is. Sing tries again with the next song.";+ SGLog(@"sing: stopped, the vocals fell short for 8 s, %d in a row%@; %@", sg_giveUps, forGood ? @", so until switched off and on" : @"",+ SGSingStatusDetail());+ apply();+ }// Now and then while on, what the model costs on this iPhone.static CFAbsoluteTime summarized;if (engine && SGSingOn() && CFAbsoluteTimeGetCurrent() - summarized >= 30) {summarized = CFAbsoluteTimeGetCurrent();SGSingEngineStats stats = SGSingEngineReadStats(engine);- SGLog(@"sing: %@, %llu windows at %.0f ms each (1500 ms keeps up), lead %.2f s of %.2f s, %.2f s separated ahead, %llu frames dry, %llu failures%@",- SGSingStatusText(), stats.windows, stats.averageMS, stats.lead, stats.targetLead, stats.ready, stats.dryFrames, stats.failures,+ SGSingSeparator *separator = SGSingLoaderSeparator();+ SGSingSeparatorStats copies = separator ? [separator stats] : (SGSingSeparatorStats){0};+ MLComputeUnits fast = SGSingLoaderFastUnits();+ NSString *fastPart = fast == MLComputeUnitsCPUOnly ? @"no faster copy"+ : [NSString stringWithFormat:@"%@ copy %@, %llu at %.0f ms", SGSingUnitsName(fast),+ @[@"not started", @"loading", @"ready", @"skipped", @"failed", @"timed out"][(NSUInteger)SGSingLoaderFastState()], copies.windows[1], copies.averageMS[1]];+ SGLog(@"sing: %@, %llu windows at %.0f ms each (1500 ms keeps up; CPU copy %llu at %.0f ms, %@, %llu redone on the CPU, the last on the %@), "+ @"lead %.2f s of %.2f s (the clock's %.2f s), %.2f s separated ahead, %llu frames dry, %.1f s dropped, %.1f s of 8 s short, %llu failures, Spotify %@, "+ @"thermal state %s, %@%@",+ SGSingStatusText(), stats.windows, stats.averageMS, copies.windows[0], copies.averageMS[0], fastPart,+ copies.fallbacks, copies.fast ? SGSingUnitsName(fast) : @"CPU", stats.lead, stats.targetLead, player ? SGSingLeadOf(player) : 0, stats.ready,+ stats.dryFrames, stats.dropped, stats.budgetSpent, stats.failures, sg_active ? @"active" : @"not active", SGSingThermalName(),+ [NSString stringWithFormat:@"Runs on %@", SGSingComputeUnitNames()[(NSUInteger)MIN(MAX(runsOn(), 0), 4)]],sg_spatialListening ? [NSString stringWithFormat:@", the voice %.0f degrees right", stats.voiceAngle * 180 / M_PI] : @"");}- // Off, with the lead given back: nothing left to watch.- if (!SGSingOn() && (!engine || SGSingEngineLead(engine) <= 0)) {+ // Off, with the lead let go: nothing left to watch.+ if (!SGSingOn() && SGSingHeldLead() <= 0) {[sg_watch invalidate];sg_watch = nil;}@@ -461,23 +755,54 @@ static void watch(void) {@implementation SGSingTrackWatcher// A new track well before the last one's end was played, not reached: the lead is the old track's.- (void)playerStateDidChange:(SPTPlayerState *)state {+ if (SGSingOn() && readRefusal()) {+ apply();+ announce();+ }updateSpatial();+ // Paused while not separating: Spotify is seeked back to what was heard, which drops the lead with nothing heard+ // lost, so the song resumes in step.+ SGSingEngine *engine = atomic_load(&sg_engine);+ double lead = SGSingHeldLead();+ if (state.isPaused && lead > 0.05 && !SGSingEngineSeparating(engine)) {+ double heard = state.positionAsOfTimestamp;+ if (heard >= 0) {+ flush([NSString stringWithFormat:@"paused while not separating, Spotify seeked back to %.2f s", heard]);+ SGKaraokeSeek((NSInteger)llround(heard * 1000));+ }+ }NSString *track = SGURIString(state.track.URI);- if (track && sg_lastTrack && ![track isEqualToString:sg_lastTrack] && sg_lastRaw >= 0 && sg_lastDuration > 0- && sg_lastDuration - sg_lastRaw > kEndSlack + kWatchEvery && CFAbsoluteTimeGetCurrent() - sg_lastFlush > 1) {+ BOOL another = track && sg_lastTrack && ![track isEqualToString:sg_lastTrack];+ if (another) sg_seekTarget = -1;+ if (another && sg_lastRaw >= 0 && sg_lastDuration > 0 && sg_lastDuration - sg_lastRaw > kEndSlack + kWatchEvery+ && CFAbsoluteTimeGetCurrent() - sg_lastFlush > 1) {flush(@"another track was played");}if (track) sg_lastTrack = track;+ if (another) {+ // A track Sing kept up with to its end clears the count of falls behind.+ if (!sg_stopped && sg_sangThisTrack) sg_giveUps = 0;+ sg_sangThisTrack = NO;+ retry(@"a new track", YES);+ }}@end#pragma mark - the heatstatic void readHeat(void) {- BOOL hot = NSProcessInfo.processInfo.thermalState >= NSProcessInfoThermalStateCritical;+ static NSProcessInfoThermalState last;+ NSProcessInfoThermalState thermal = NSProcessInfo.processInfo.thermalState;+ BOOL cooler = thermal < last;+ last = thermal;+ // A cooler iPhone runs the model faster.+ if (cooler) retry([NSString stringWithFormat:@"the iPhone is cooler (%s)", SGSingThermalName()], NO);+ // Serious and Critical hold; Fair lets it run again.+ BOOL hot = thermal >= NSProcessInfoThermalStateSerious;if (hot == sg_hot) return;sg_hot = hot;- SGLog(@"sing: the iPhone is %@%@", hot ? @"hot" : @"cool again", hot && SGHidden(SGKeySingIgnoreHeat) ? @", and heat warnings are ignored" : @"");+ SGLog(@"sing: the iPhone is %@ (thermal state %s)%@", hot ? @"hot" : @"cool again", SGSingThermalName(),+ hot && SGHidden(SGKeySingIgnoreHeat) ? @", and heat warnings are ignored" : @", the engine held");apply();announce();}@@ -488,13 +813,36 @@ static void readHeat(void) {- (double)positionAsOfTimestamp {double position = %orig;if (sg_rawPosition || position < 0) return position;- SGSingEngine *engine = atomic_load_explicit(&sg_engine, memory_order_acquire);- double lead = engine ? SGSingEngineLead(engine) : 0;- return lead > 0 ? MAX(0, position - lead) : position;+ double lead = SGSingLeadOf(self);+ return lead != 0 ? MAX(0, position - lead) : position;+}+%end++// The lock screen's elapsed time, when Spotify gives it its own count, which runs off what is heard by the state's+// lead. Which of the player's getters Spotify's now playing code reads is not known, so the time is moved only when it+// is nearer Spotify's own count than the heard one: one already taken through the hook above is left as it is.+%hook MPNowPlayingInfoCenter+- (void)setNowPlayingInfo:(NSDictionary *)info {+ SPTPlayerState *state = SGPlayerState();+ double lead = state ? SGSingLeadOf(state) : 0;+ NSNumber *elapsed = info[MPNowPlayingInfoPropertyElapsedPlaybackTime];+ if (fabs(lead) > 0.05 && [elapsed isKindOfClass:NSNumber.class]) {+ double raw = rawPosition(state), at = elapsed.doubleValue;+ if (raw >= 0 && fabs(at - raw) < fabs(at - (raw - lead))) {+ NSMutableDictionary *heard = [info mutableCopy];+ heard[MPNowPlayingInfoPropertyElapsedPlaybackTime] = @(MAX(0, at - lead));+ %orig(heard);+ return;+ }+ }+ %orig;}%end%hook SPTEsperantoPlayer+// Each seek runs on into the last one with the creator's timestamp (seekTo: 0x1096db0c0 into seekTo:options:, that+// and seekTo:relative: into seekTo:relative:options:, that into the last, which sends the position times 1000 as+// milliseconds), so the anchor is taken there, for an absolute seek (relative 0).- (id)seekTo:(double)position {flush(@"Spotify seeks");return %orig;@@ -513,6 +861,7 @@ static void readHeat(void) {}- (id)seekTo:(double)position relative:(long long)relative options:(id)options creatorTimestampPositionMs:(double)creator {flush(@"Spotify seeks");+ if (relative == 0) seekAnchor(position);return %orig;}// Every skip of Spotify's ends up in one of these two.@@ -531,29 +880,74 @@ static void readHeat(void) {%end%ctor {- if (!SGRebindImport("AudioOutputUnitStart", startOutput, (void **)&sg_startOutput) || !sg_startOutput) {- sg_startOutput = NULL;- SGLog(@"sing: Spotify does not import AudioOutputUnitStart, Sing cannot read its output");- }+ sg_outputReachable = SGPlayerWatchMusicOutput(musicOutputChanged);+ if (!sg_outputReachable) SGLog(@"sing: Spotify's output cannot be watched, Sing cannot read its format");SGPlayerSetStage(stage);%init;SGRequireClasses(@[@"SPTPlayerState", @"SPTEsperantoPlayer"]);+ // Runs on was GPU, GPU and Neural Engine, Neural Engine (unset: Neural Engine); a choice made then is the+ // faster copy's now, under its new place in the list, and unset is Automatic.+ NSUserDefaults *defaults = NSUserDefaults.standardUserDefaults;+ NSNumber *before = [defaults objectForKey:SGKeySingComputeUnitsBefore];+ if (before && ![defaults objectForKey:SGKeySingComputeUnits]) {+ NSInteger was = before.integerValue;+ [defaults setInteger:was == 0 ? 2 : was == 1 ? 4 : 3 forKey:SGKeySingComputeUnits];+ [defaults removeObjectForKey:SGKeySingComputeUnitsBefore];+ }dispatch_async(dispatch_get_main_queue(), ^{static SGSingTrackWatcher *watcher;watcher = [SGSingTrackWatcher new];SGAddPlayerStateObserver(watcher);- [NSNotificationCenter.defaultCenter addObserverForName:NSProcessInfoThermalStateDidChangeNotification object:nil queue:NSOperationQueue.mainQueue- usingBlock:^(NSNotification *note) { readHeat(); }];+ SGSingLoaderSetChanged(^{ loaderChanged(); });+ NSNotificationCenter *center = NSNotificationCenter.defaultCenter;+ // Posted on whichever thread saw the change: handed to main without waiting.+ [center addObserverForName:NSProcessInfoThermalStateDidChangeNotification object:nil queue:nil usingBlock:^(NSNotification *note) {+ dispatch_async(dispatch_get_main_queue(), ^{ readHeat(); });+ }];// Headphones in or out, or iOS's own spatial audio switched.for (NSNotificationName name in @[AVAudioSessionRouteChangeNotification, AVAudioSessionSpatialPlaybackCapabilitiesChangedNotification]) {- [NSNotificationCenter.defaultCenter addObserverForName:name object:nil queue:NSOperationQueue.mainQueue- usingBlock:^(NSNotification *note) { updateSpatial(); }];+ [center addObserverForName:name object:nil queue:NSOperationQueue.mainQueue usingBlock:^(NSNotification *note) {+ if (SGSingOn() && readRefusal()) {+ apply();+ announce();+ }+ updateSpatial();+ }];}- [NSNotificationCenter.defaultCenter addObserverForName:SGSingChangedNotification object:nil queue:nil usingBlock:^(NSNotification *note) {- // A download finishing with the mic already on loads the model.- if (SGSingOn() && !sg_separator && !sg_loading && !sg_loadError) apply();+ // A load starts only while Spotify is active, so never in a launch into the background, and the faster copy+ // runs the windows only then.+ [center addObserverForName:UIApplicationDidBecomeActiveNotification object:nil queue:NSOperationQueue.mainQueue usingBlock:^(NSNotification *note) {+ sg_active = YES;+ SGSingLoaderSetForeground(YES);+ if (SGSingOn()) apply();+ }];+ [center addObserverForName:UIApplicationWillResignActiveNotification object:nil queue:NSOperationQueue.mainQueue usingBlock:^(NSNotification *note) {+ sg_active = NO;+ SGSingLoaderSetForeground(NO);+ }];+ // Low on memory, the model goes at once, before iOS closes Spotify for it.+ [center addObserverForName:UIApplicationDidReceiveMemoryWarningNotification object:nil queue:NSOperationQueue.mainQueue usingBlock:^(NSNotification *note) {+ SGSingLoaderPurge(@"iOS warned that memory is low");+ // Over a stop for falling behind, which would load the model again with the next track whatever the memory.+ if (!SGSingOn() || (sg_stopped && sg_stopKind != SGSingStopBehind)) return;+ sg_stopKind = SGSingStopMemory;+ sg_stopped = @"Sing stopped to free memory, so the song plays as it is. It loads the voice model again with the next song if there is room, "+ @"or switch Sing off and on.";+ apply();+ announce();+ }];+ [center addObserverForName:SGSingChangedNotification object:nil queue:nil usingBlock:^(NSNotification *note) {+ if (SGSingModelCurrentState() != SGSingModelReady) {+ SGSingLoaderPurge(@"the voice model is gone from the iPhone");+ } else if (SGSingOn() && sg_active && !sg_resting && SGSingLoaderCurrentState() == SGSingLoaderIdle && !(sg_hot && !SGHidden(SGKeySingIgnoreHeat))) {+ // A download finishing with the mic already on loads the model.+ apply();+ }}];+ sg_active = UIApplication.sharedApplication.applicationState == UIApplicationStateActive;+ SGSingLoaderSetForeground(sg_active);readHeat();+ readRefusal();if (SGSingOn()) apply();});}diff --git a/tweak/Sources/Shared/Sing/SingSettings.m b/tweak/Sources/Shared/Sing/SingSettings.mindex 6208a0d6..2aaadb85 100644--- a/tweak/Sources/Shared/Sing/SingSettings.m+++ b/tweak/Sources/Shared/Sing/SingSettings.m@@ -1,11 +1,15 @@-// The Sing page (Sing on Mod Settings' main page, and Lyrics > Karaoke), under either look: Sing's switch, which-// turns the mic on and off at once, what Sing is doing, the vocals' level, spatial voice's page (its preview,-// SGSpatialPreview.m, and its switch), and the voice model.+// The Sing page (Sing on Mod Settings' main page, and Lyrics > Karaoke), under either look: the card at its top+// (SGSingCard.m: the song, what Sing is doing, the vocals and the rest traced live, play and pause, the vocals'+// level and its three stops), Sing's switch, which turns the mic on and off at once, Ignore heat warnings, the voice+// model and its removal, spatial voice's page (its preview, SGSpatialPreview.m, and its switch), and Runs on under+// Advanced.#import "Core/SGCore.h"#import "Settings/SGModPage.h"#import "Settings/SGPageStyle.h"#import "Sing.h"+UIView *SGSingCardView(void); // SGSingCard.m+static void tell(NSString *title, NSString *message) {UIAlertController *alert = [UIAlertController alertControllerWithTitle:title message:message preferredStyle:UIAlertControllerStyleAlert];[alert addAction:[UIAlertAction actionWithTitle:@"OK" style:UIAlertActionStyleCancel handler:nil]];@@ -28,13 +32,22 @@ NSString *SGSingLevelText(double level) {return [NSString stringWithFormat:@"Backing %.0f%%", (2 - level) * 100];}+static NSString *bytes(long long count) {+ return [NSByteCountFormatter stringFromByteCount:count countStyle:NSByteCountFormatterCountStyleFile];+}+static NSString *modelValue(void) {if (!SGSingOSSupported() || !SGSingDeviceSupported()) return @"Unavailable";switch (SGSingModelCurrentState()) {case SGSingModelDownloading:- return SGSingModelWaitingForNetwork() ? @"Waiting for the network" : [NSString stringWithFormat:@"%.0f%%", SGSingModelProgress() * 100];- case SGSingModelReady: return @"On the iPhone";- case SGSingModelMissing: return SGSingModelError() ? @"Failed, try again" : [NSString stringWithFormat:@"Download, %@", SGSingModelSizeText()];+ if (SGSingModelChecking()) return @"Checking…";+ if (SGSingModelWaitingForNetwork()) return SGSingModelOverCellular() ? @"Waiting for the network" : @"Waiting for Wi-Fi";+ return [NSString stringWithFormat:@"%.0f%%", SGSingModelProgress() * 100];+ case SGSingModelReady: return [NSString stringWithFormat:@"Downloaded · %@", SGSingModelSizeText()];+ case SGSingModelMissing:+ if (SGSingModelError()) return @"Failed, try again";+ if (SGSingModelPausedBytes()) return [NSString stringWithFormat:@"Paused · %@ of %@", bytes(SGSingModelPausedBytes()), SGSingModelSizeText()];+ return [NSString stringWithFormat:@"Download, %@", SGSingModelSizeText()];}return @"";}@@ -46,12 +59,17 @@ static void modelTapped(UITableViewController *page) {}switch (SGSingModelCurrentState()) {case SGSingModelMissing: {- NSString *message = [NSString stringWithFormat:@"%@%@ from Hugging Face, best over Wi-Fi. It stays on this iPhone, and no audio leaves it.",+ NSString *message = [NSString stringWithFormat:@"%@%@ from Hugging Face. It waits for Wi-Fi unless you let it use cellular. It stays on this iPhone, and no audio leaves it.",SGSingModelError() ? [NSString stringWithFormat:@"The last try failed: %@\n\n", SGSingModelError()] : @"", SGSingModelSizeText()];- ask(@"Download the voice model?", message, @"Download", NO, ^{ SGSingDownloadModel(); });+ ask(SGSingModelPausedBytes() ? @"Carry on with the download?" : @"Download the voice model?", message, @"Download", NO, ^{ SGSingDownloadModel(); });break;}case SGSingModelDownloading:+ if (SGSingModelWaitingForNetwork() && !SGSingModelOverCellular()) {+ ask(@"Download over cellular?", [NSString stringWithFormat:@"The download waits for Wi-Fi. Over cellular or a Low Data Mode network it uses up to %@ of your plan.",+ SGSingModelSizeText()], @"Use cellular", NO, ^{ SGSingDownloadModelOverCellular(); });+ break;+ }ask(@"Stop the download?", @"What has come in is kept, so the next download carries on from it.", @"Stop", YES, ^{ SGSingCancelModelDownload(); });break;case SGSingModelReady:@@ -110,31 +128,70 @@ UIViewController *SGSpatialVoiceSettingsPage(void) {return made;}+// The Sing page: the card (SGSingCard.m) as the table's header, sized to Dynamic Type, then its rows.+@interface SGSingPage : SGModPage+@end++@implementation SGSingPage {+ UIView *_card;+}++- (void)viewDidLoad {+ [super viewDidLoad];+ _card = SGSingCardView();+ self.tableView.tableHeaderView = _card;+}++// Handed back only when its size changes (SGFitNote's way: the table lays out again on every handing back).+- (void)viewWillLayoutSubviews {+ [super viewWillLayoutSubviews];+ UITableView *table = self.tableView;+ CGSize size = [_card sizeThatFits:CGSizeMake(table.bounds.size.width, 0)];+ if (CGSizeEqualToSize(_card.bounds.size, size)) return;+ _card.frame = (CGRect){_card.frame.origin, size};+ table.tableHeaderView = _card;+}++@end+UIViewController *SGSingSettingsPage(void) {- SGModRow *sing = SGOptionRow(@"Sing", @"Turns the vocals down as the song plays, like the mic on the lyrics", SGKeySing);+ SGModRow *sing = SGOptionRow(@"Sing", nil, SGKeySing);sing.changed = ^(BOOL on) { SGSetSingOn(on); };- SGModRow *status = SGStatActionRow(@"Status", nil, ^NSString *{ return SGSingStatusText(); }, ^{- NSString *detail = SGSingStatusDetail();- if (detail) tell(@"Sing", detail);+ SGModRow *heat = SGOptionRow(@"Ignore heat warnings", @"Keeps Sing going on a hot iPhone, which then gets hotter", SGKeySingIgnoreHeat);+ heat.changed = ^(BOOL on) { SGSetSingIgnoresHeat(on); };+ __block __weak SGModPage *page;+ SGModRow *model = SGStatActionRow(@"Voice model", nil, ^NSString *{ return modelValue(); }, ^{ modelTapped(page); });+ SGModRow *remove = SGActionRow(@"Remove voice model", nil, ^{+ BOOL partial = SGSingModelCurrentState() != SGSingModelReady;+ ask(partial ? @"Remove the paused download?" : @"Remove the voice model?",+ partial ? [NSString stringWithFormat:@"It frees %@. A new download starts from the beginning.", bytes(SGSingModelPausedBytes())]+ : [NSString stringWithFormat:@"It frees %@. Sing needs it again to turn the vocals down.", SGSingModelSizeText()],+ @"Remove", YES, ^{+ SGSetSingOn(NO);+ SGSingDeleteModel();+ [page.tableView reloadData];+ });});- SGModRow *level = SGSliderRow(@"Vocals", @"From gone, through the song as sung, to the vocals alone", 0, 2, 0.05,- ^double { return SGSingLevel(); }, ^(double value) { SGSetSingLevel((float)value); }, ^NSString *(double value) { return SGSingLevelText(value); });+ remove.color = SGRed();+ remove.visible = ^BOOL { return SGSingModelCurrentState() == SGSingModelReady || SGSingModelPausedBytes() > 0; };SGModRow *spatial = SGPageRow(@"Spatial voice", ^UIViewController *{ return SGSpatialVoiceSettingsPage(); });spatial.subtitle = @"With AirPods, the voice stays in front of you as you turn your head";spatial.value = ^NSString *{ return SGSingSpatial() ? @"On" : @"Off"; };// Facts about the iPhone, which do not change while the page shows.spatial.visible = ^BOOL { return SGSingSpatialAvailable(); };- __block __weak SGModPage *page;- SGModRow *model = SGStatActionRow(@"Voice model", @"Mel-Band RoFormer, run on the iPhone", ^NSString *{ return modelValue(); }, ^{ modelTapped(page); });- SGModRow *units = SGChoiceRow(@"Runs on", @"Read as the model loads", SGKeySingComputeUnits, SGSingComputeUnitNames(), 2);+ SGModRow *units = SGChoiceRow(@"Runs on", nil, SGKeySingComputeUnits, SGSingComputeUnitNames(), 0);+ units.choiceNotes = @[@"The GPU, unless it did not load on this iOS before", @"Slower, and the least memory", @"Tried every time, even after it did not load",+ @"Experimental: slower than the CPU on a Mac", @"Experimental"];+ units.choiceFooter = @"Sing loads the voice model on the CPU first and starts with it. A second copy then loads for the windows "+ @"played while Spotify is open; in the background they run on the CPU.";units.chosen = ^(NSInteger index) { SGSingComputeUnitsChanged(); };- SGModRow *heat = SGOptionRow(@"Ignore heat warnings", @"Keeps Sing going on a hot iPhone, which then gets hotter", SGKeySingIgnoreHeat);- heat.changed = ^(BOOL on) { SGSetSingIgnoresHeat(on); };- SGModPage *made = [[SGModPage alloc] initWithTitle:@"Sing" intro:nil sections:@[- SGSection(nil, @[SGWithSymbol(sing, @"music.mic"), status, level, spatial]),- SGNotedSection(@"Voice model", @[model, units, heat],- @"Sing listens a few seconds ahead of what plays and separates the vocals there, so a song takes a few seconds to "- @"turn its vocals down after it starts or after a seek. Everything runs on the iPhone; no audio leaves it."),+ SGSingPage *made = [[SGSingPage alloc] initWithTitle:@"Sing" intro:nil sections:@[+ SGSection(nil, @[sing, heat, model, remove]),+ SGSection(nil, @[spatial]),+ SGNotedSection(@"Advanced", @[units],+ @"Sing turns a song's vocals down to sing over, or the rest down to hear the vocals alone, with a voice model that "+ @"runs only on this iPhone: no audio leaves it. It listens a few seconds ahead of what plays, so the vocals change a "+ @"few seconds after a song starts or after a seek."),] footer:nil];page = made;return made;