Skip to content

Liquid Glass and Apple Music touches for the Spotify iOS app. A GPL-3.0 fork of spoti.pw 0.21.1.

C

feat: Karaoke uses one voice model on the Neural Engine and CPU

My-Name-Is-Jeffb8a50bb8e3d8
diff --git a/docs/tweaks.md b/docs/tweaks.md
index aff7278c..4100d0d5 100644
--- a/docs/tweaks.md
+++ b/docs/tweaks.md
@@ -229,10 +229,23 @@ Shared:
Tested on the Mac against harness/audio-effects/ and harness/autoeq/, the hook in the simulator
against its sim/, the page against harness/audio-effects-page/
Sing/ Karaoke (Sing in the code and its keys), a song's vocals turned down while it plays (Sing.h lists
- its files): the voice model (Mel-Band RoFormer, MIT, from Hugging Face) downloaded and checked file by file over Wi-Fi unless cellular is allowed,
- a stop keeping what came in for the next download to carry on from; loaded only while Spotify is active
- (SGSingLoader.m), a CPU copy first and warmed, then a faster one beside it for the foreground, each load
- with a deadline and kept a minute after the mic goes off; the STFT around it on Accelerate,
+ its files): one voice model (Mel-Band RoFormer, MIT, exported by Vitrine for the Neural Engine and
+ palettized to 6 bits: separator-ane.mlmodelc, 210 MB, from Hugging Face; on a FLEX build a copy in
+ Sing/dev/ comes first) downloaded and checked file by file over Wi-Fi unless cellular is allowed,
+ a stop keeping what came in for the next download to carry on from; the 489 MB model downloaded
+ before is kept and loaded on the CPU alone until this one is in (the Voice model row then reads
+ Update available), and deleted as its download ends or at a launch that finds both
+ (SGSingRemoveOldModel, which logs the space freed), Sing moving to the new one at its next load;
+ loaded only while Spotify
+ is active (SGSingLoader.m), a CPU copy first and warmed, then on Automatic, where the iPhone has a
+ Neural Engine, a Neural Engine copy of the same model beside it, which takes every window once it is
+ in, in the background too, each load with a deadline and kept a minute after the mic goes off; the
+ Neural Engine copy loads only with 0.5 GB left to the process, its first load after every install
+ compiling it for up to ten minutes while Karaoke runs on the CPU (while it loads, the CPU's falling
+ behind spends none of the 8 s after which Karaoke gives a song up, nor counts toward its three in a
+ row), and it is dropped for the launch, the CPU copy carrying on, once it fails to load, warm up or
+ run a window (one that falls behind keeps its place); Runs on is Automatic or CPU only, the GPU and
+ Neural Engine choices stored before read as Automatic; the STFT around it on Accelerate,
and an engine that stands in Speed and pitch's chain between Spotify's mixer and its output
(SGPlayerSetStage), pulls the mixer a few seconds ahead of what plays, separates two-second windows
there on a worker thread and mixes the vocals down on the render thread. Spotify's clock has the lead
@@ -243,7 +256,13 @@ Shared:
and played, and every lyrics seek logs where it was sent and where it landed), and a seek, a skip or a stop drops it. Not separating (stopped, held, resting at As sung,
standing aside), the engine plays the lead it holds on as it is, dry, the clock still corrected, so no
part of the song is skipped; it lets it go at a pause, where Sing.x seeks Spotify back to what was
- heard, or with a seek, a skip or the output stopping, and builds one only when it separates. From the
+ heard, or with a seek, a skip or the output stopping, and builds one only when it separates. At As
+ sung with Spatial voice off it rests (held as for the heat, the model kept a minute as for a mic
+ switched off, "sing: rests" in the log) unless the Karaoke page's card is on screen with Spotify in
+ front: its reads of its two lines (SGSingReadLevels; the card's traces, not the lyrics, which are no
+ viewer) keep Karaoke separating the playing song, so the lines move and the slider is heard at once
+ ("sing: separates again"), and a second after the page is popped, covered or in the background it
+ rests again. From the
thermal state Serious up Karaoke is held and lets the model go,
unless Ignore heat warnings is on, and runs again at Fair. Spatial
voice holds the separated vocals in front as the head turns: HeadGestures' motion gives the yaw off a
@@ -252,8 +271,9 @@ Shared:
audio. The Spatial voice page's preview (SGSpatialPreview.m) turns a disc of dots under the listener by
the same front (SGSpatialVoiceAngle, SGSingEngine.h), on Core Animation alone and only while the page
shows. The mic is on the redesign's lyrics (Redesigned/Lyrics/SGRSingButton.m). Tested on the Mac
- against harness/sing/ (spatial voice and its front without the model: `build/sing spatial`), the
- download against its download/, the pages in the simulator against harness/spatial-page/
+ against harness/sing/ (spatial voice and its front without the model: `build/sing spatial`; the
+ model as both copies with `ane`, as the CPU's alone with `cpu`), the download against its download/
+ (`update` for the old model kept until it, `dev` for the dev folder), the pages in the simulator against harness/spatial-page/
Haptics/ Vibrations (Haptics.h lists its files): a tap of UIKit's feedback generators for the player's and the now
playing bar's controls, the scrubber's tenths and ends, cover swipes, gestures and the lyrics page's tap to
seek, at the strength set for them (ControlHaptics.x, SGFeedback.m); and Music Haptics, Core Haptics
@@ -586,13 +606,13 @@ the redesign also which of the lyrics, their pronunciation and their translation
(applied at once), and the translation's language. Karaoke, under either look, on the main page and as Lyrics' first row, the row reading out On,
Off or how far the voice model's download has come, kept up to date while the page shows: Karaoke's switch, which turns
the mic on and off at once, a card at its top (SGSingCard.m: the song, what Karaoke is doing, a tap saying more, the vocals
-and the rest traced live from the engine's loudness, still under Reduce Motion, play and pause, and a tall Vocals slider
+and the rest traced live from the engine's loudness, Karaoke running on the playing song while they show even at As sung, still under Reduce Motion, play and pause, and a tall Vocals slider
from gone through as sung to the vocals alone, with Sing along, Original and Vocals only under it), Spatial voice
(where the iPhone reads headphone motion; a page of its own, reading out On or Off, with a live preview at its top that
follows the head through AirPods, or sways gently without them and holds still under Reduce Motion, a line under it
saying which, then the switch; its row is on the main page too, under Karaoke's), the voice model's download (Paused and
-Checking among its states) and its removal, Ignore heat warnings, and under Advanced Runs on (Automatic, CPU only, GPU,
-Neural Engine, GPU and Neural Engine: the faster copy's), all applying straight away. Lock screen, on the main
+Checking among its states) and its removal, Ignore heat warnings, and under Advanced Runs on (Automatic: the Neural Engine
+beside the CPU; CPU only), all applying straight away. Lock screen, on the main
page under either look, opens the lock screen widget's page, titled Lock screen (Moving artwork, Lyrics or Every song, and the lyrics' style,
Still or Animated, and Spotify's like and dislike buttons' flag; its podcast, audiobook and artwork flags stay in
All flags). Player: Gestures and Blocked artists (with the count on the row), which work with either look;
diff --git a/harness/sing/README.md b/harness/sing/README.md
index e49802e2..2a8d1988 100644
--- a/harness/sing/README.md
+++ b/harness/sing/README.md
@@ -5,19 +5,24 @@ Three parts: the separator and the engine on the Mac, Sing end to end in the sim
## The separator and the engine on the Mac (`main.m`)
Sing's loader, separator and engine (`tweak/Sources/Shared/Sing/SGSingLoader.m`, `SGSingSeparator.m`,
-`SGSingEngine.m`) as the tweak compiles them, their log lines printed through `shim/Core/SGLog.h`, on the Mac, over the voice model as the app downloads it (a `separator.mlmodelc` folder of the five
-files from https://huggingface.co/My-Name-Is-Jeff/vitrine-sing, a mirror of Darkkos/spoti-sing, MIT). Each check prints a line:
+`SGSingEngine.m`) as the tweak compiles them, their log lines printed through `shim/Core/SGLog.h`, on the Mac, over the voice model as the app downloads it (the `separator-ane.mlmodelc` folder of
+five files from https://huggingface.co/My-Name-Is-Jeff/vitrine-sing, the 6-bit Neural Engine export, MIT), which is both
+the CPU copy and the Neural Engine copy. Each check prints a line:
- the STFT: a cosine's peak bin at torch.stft's size (the amplitude times the Hann window's sum over two), noise
through the STFT and back unchanged, its edges included;
- the loader: a load past a deadline of 0.2 s abandoned and Sing Failed, Failed kept until the mic goes off and on,
then a fresh load beside the abandoned one with a second want joining it, the CPU copy loaded and warmed with a
- window of silence, the abandoned load let go when it comes back (the separator stays the fresh one's), the faster
- copy (the units asked for; `cpu` for none) loaded beside it, the copies kept over a quick off and on and dropped
- after the time kept, and a load purged on its way let go;
-- the model: its input and output shapes, the plan Core ML makes for the compute units asked for (how many
- operations go to the CPU, the GPU and the Neural Engine), the time a two second window takes, and a voice mixed
- over synthetic chords taken apart: the vocals found scored against the voice, beside the mix's own score;
+ window of silence, the abandoned load let go when it comes back (the separator stays the fresh one's), the Neural
+ Engine copy of the same model loaded beside it (none with `cpu`), the copies kept over a quick off and on and dropped
+ after the time kept, a load purged on its way let go, and another model wanted (as the update replaces the old one)
+ dropping the first's copies and loading; then a window on the Neural Engine copy, one it fails done
+ again on the CPU's and the failed copy not tried again, the Neural Engine copy dropped when none is asked for, and
+ one past its deadline timing out with Sing Ready on the CPU's (`ane` only);
+- the model: its input and output shapes, the plan Core ML makes for the copy that runs the windows (how many
+ operations go to the CPU, the GPU and the Neural Engine), the time a two second window takes, a voice mixed over
+ synthetic chords taken apart (the vocals found scored against the voice, beside the mix's own score), and with
+ `ane` the Neural Engine copy against the CPU copy on the same window;
- the engine: the mix pulled through `SGSingEngineRender` in buffers of 1024 on a thread that keeps real time,
with Sing on from the start: the mix plays dry while the lead fills, then exactly the mix less the vocals the
offline pass finds, the lead kept after Sing is switched off and the mix playing on where it was, nothing skipped, the
@@ -33,6 +38,15 @@ files from https://huggingface.co/My-Name-Is-Jeff/vitrine-sing, a mirror of Dark
31 frames later (the 28.7 frames around a head and the low-pass's own lag); 90 degrees left mirrors it; back ahead
it is exact again; no step from one frame to the next is larger than the tone's own at the louder gain, so no
turn clicks; and nothing is allocated on the render thread;
+- the Neural Engine copy loading, without the model (copies of the model that take a set time a window and hand back
+ silence): a Neural Engine copy keeping up, dropped at 6 s, another loading until 24 s (a first compile) and the
+ CPU's taking 2.2 s a window meanwhile; with the budget held (as Sing.x holds it while the loader reads Loading) the
+ vocals go out and none of the 8 s is spent, the copy in brings them back, and a load that fails instead starts the
+ budget then, full, so the engine gives up 8 s on;
+- what Sing.x drops a Neural Engine copy on, without the model: a copy slowed to 2.2 s a window until the engine
+ gives up reads unfailed (falling behind is the budget's, and the copy keeps its place); one that fails a window has
+ the window done on the CPU's, and it still reads failed after the next window ran on the CPU's; a fresh copy put
+ in its place reads unfailed;
- the front spatial voice holds the voice off (SGSpatialVoiceAngle, shared with the Spatial voice page's preview),
fed at 25 Hz: a head turned 60 degrees left has the voice 60 degrees right, and 20 s on 22 (the front's 1/e);
yaw wrapping across 180 degrees moves the voice by the 1 degree it turned; a gap of 2 s starts it ahead again.
@@ -40,8 +54,8 @@ files from https://huggingface.co/My-Name-Is-Jeff/vitrine-sing, a mirror of Dark
The voice is macOS's own speech, `say -o speech.aiff "..."` (any file ExtAudioFile reads will do), repeated a
second apart over 30 seconds.
- ./build.sh && build/sing <separator.mlmodelc> <voice> <out dir> [all|cpu|gpu|ane]
- ./build.sh thread && build/sing-thread <separator.mlmodelc> <voice> <out dir> gpu (ThreadSanitizer)
+ ./build.sh && build/sing <separator-ane.mlmodelc> <voice> <out dir> [ane|cpu] (both copies, the default, or the CPU's alone)
+ ./build.sh thread && build/sing-thread <separator-ane.mlmodelc> <voice> <out dir> ane (ThreadSanitizer)
./build.sh && build/sing spatial (the lead, spatial voice and falling behind)
The out dir gets `mix.wav`, `vocals.wav`, `accompaniment.wav`, `engine.wav` (what played) and `spatial.wav` (the
@@ -95,6 +109,54 @@ and `spatial` under ThreadSanitizer. The lead was off by 0 frames from what was
to 12 s it stayed at 2.75 s with every frame played in order; a flush read 0 at once, and a position counted at 1 s
then had -1.73 s taken off (the 1.02 s held then less the 2.75 s dropped).
+2026-10-06, the Neural Engine's model (`build/sing <separator.mlmodelc> <voice> <out dir> ane <separator-ane.mlmodelc>`, the 6-bit export
+compiled for iOS 18): everything passes, the new checks with it. The loader takes it as the faster copy on the CPU and
+Neural Engine: Core ML's plan puts all 2815 operations on the Neural Engine; its first load compiled it in 39.5 s (a
+loaded Mac), and the next process loaded it from Core ML's cache in 0.3 s. A window takes 184-292 ms, 215 ms on
+average, where the CPU copy took 0.6-2.3 s; the vocals score 27.9 dB against the voice (the mix 7.1 dB), and on the
+same window it finds the vocals the first model's CPU copy does within 39.1 dB. Missing, cut short (Core ML fails to
+compile it: a file missing altogether crashes Core ML instead, which is why a dev copy needs every file) or past its
+deadline, its copy fails or times out and Sing stays Ready on the CPU's, and the first model's GPU copy then loads in
+its place and takes the windows, as Sing.x falls back. The `gpu` run and `spatial` pass as before.
+
+2026-10-06, one model (the FLEX test `spotifyglass.sing.oneModel`): the Neural Engine's model as every copy, with
+`build/sing <separator-ane.mlmodelc> <voice> <out dir> cpu|gpu|ane [separator-ane.mlmodelc]`; every run passes, and
+`ane` with it in both places has its CPU, Neural Engine and fallback GPU copies all of the one model. The checks now
+expect a Neural Engine copy to keep background windows (dd486a3). Same Mac, same hour, load 3-7; memory is the whole
+harness process (`/usr/bin/time -l`), the loader's lines log footprint and resident size:
+
+| copy | model | a window, warm (5 logged) | average offline | peak footprint | peak resident |
+|---|---|---|---|---|---|
+| CPU | shipped | 450-469 ms | 534 ms (819-977 at load 6-7) | 0.35 GB | 1.03-1.47 GB |
+| CPU | 6-bit | 461-474 ms | 531 ms (638-758 at load 6-7) | 0.22 GB | 0.68-1.02 GB |
+| GPU | shipped | 344-377 ms | 354 ms | 0.72 GB | 1.96 GB |
+| GPU | 6-bit | 371-419 ms | 382 ms | 0.64 GB | 2.93 GB |
+
+The 6-bit model's GPU copy spreads its weights out when it warms (resident 0.7 to 2.6 GB on that load), where the
+shipped one's stays near 1.1 GB. Its vocals score 27.6-27.7 dB against the voice (the shipped model 28.0-28.1).
+2026-10-06, a faster copy loading (`build/sing spatial`): everything passes. Held, 0.0 s of the budget is spent from
+6 s to 24 s and the engine does not give up; the GPU's copy in at 24 s brings the vocals back; the load failed at 24 s,
+the engine gives up at 32.0 s. The same harness against an engine without the hold gives up at 15.9 s.
+The Neural Engine copy slowed to 2.2 s a window: the engine gives up at 15.9 s and the copy reads unfailed; the
+failing one reads failed after a window on the CPU's, which the old rule (the last window on the faster copy) missed.
+
+2026-10-06, one model (the 6-bit `separator-ane.mlmodelc` as the CPU copy and the Neural Engine copy; no GPU copy):
+everything passes under `ane`, under `cpu` and with `spatial`. Under `ane` (load 4-5) the CPU copy loads and warms in
+2.5 s and the Neural Engine copy beside it 0.5 s later (Core ML had it compiled from before); a window on it takes
+168-202 ms, 173 ms on average; the vocals score 27.8 dB against the voice (the mix 6.9 dB), the engine plays the mix
+less them at 27.6 dB, and the Neural Engine copy finds what the CPU copy does on the same window within 49.2 dB; the
+whole process peaks at 1.08 GB resident and a 0.31 GB footprint. Under `cpu` a window takes 570-662 ms (631 ms on
+average), the engine keeps up (27.5 dB), and the process peaks at 0.68 GB resident and a 0.28 GB footprint. Two `cpu`
+runs earlier, with a simulator booting elsewhere (load average 130-490) and the Mac at thermal state fair, took
+1.1-2.9 s a window and failed the two engine scores; `main`'s harness, run between them at load 8, passed on the same
+model at 521 ms a window, and the last run here passed at load 4.5.
+
+2026-10-06, the old model kept until its update is in: the loader drops the copies of one model when another is wanted
+and loads it (the same files through a link stand in for the update); `ane` passes with it (load 9-10, 306 ms a window
+on the Neural Engine). The old `separator.mlmodelc` as the CPU copy (`cpu`, what Karaoke runs until the update is in)
+loads, warms and finds the vocals at 28.2 dB, every loader check passing, but took 1.8 s a window at load 10-15, so the
+engine fell behind and its score failed: as on the iPhone (2-3.5 s a window), the old model on the CPU does not keep up.
+
## Sing end to end in the simulator (`sim/main.m`)
The voice model downloaded from Hugging Face by `SGSingModel.m` and checked file by file, loaded by `Sing.x`, and
@@ -154,12 +216,27 @@ so a report after one of Sing's seeks, skips or seeks back always starts a line.
## The download on the Mac (`download/`)
`SGSingModel.m` as the tweak compiles it, built for Mac Catalyst (Sing.h imports UIKit) with `download/shim` standing
-in for the mod's core so its log goes to stderr, against Hugging Face. The four small files and a fifth of the weights
-come in, the download is stopped (its resume data kept beside the staging folder), and the next download carries the
-weights on from there through the checks and the move into place. `CFFIXED_USER_HOME` keeps the 489 MB out of the
-Mac's own Application Support.
+in for the mod's core so its log goes to stderr, against Hugging Face. First what the old model's download left
+(its staging folder, its resume data and its paused bytes, stood in by sparse files of their sizes) is deleted as a
+launch deletes it (`SGSingRemoveOldModel`). Then the four small files of `separator-ane.mlmodelc` and a fifth of its
+weights come in, the download is stopped (its resume data kept beside the staging folder, and left alone by the
+cleanup), and the next download carries the weights on from there through the checks and the move into place, nothing
+else left in `Sing/`. `CFFIXED_USER_HOME` keeps the 210 MB out of the Mac's own Application Support. `update` starts
+from the old model (`separator.mlmodelc`, stood in by sparse files of its sizes) instead: kept at launch and the model
+(Ready, the download its update) while the update downloads, deleted as soon as it is in, and deleted by a launch that
+finds both. `dev` checks the FLEX build's dev folder.
download/build.sh && CFFIXED_USER_HOME="$(mktemp -d)" build/download 0.2
+ download/build.sh && CFFIXED_USER_HOME="$(mktemp -d)" build/download update
+ download/build.sh && CFFIXED_USER_HOME="$(mktemp -d)" build/download dev
+
+2026-10-06, one model, the old one kept until it is in: every mode passes. The size row reads 210.4 MB
+(NSByteCountFormatter's decimal megabytes; 201 MiB), the old model 489.7 MB. Default: the old download's leftovers go
+(670 KB of stand-ins), the stop is at 21%, the next download's first bytes read 22%, the server answers the resumed
+weights 206 and the whole 209077208 bytes are handed over and checked. `update`: the launch logs that the old model is
+kept, it is Ready throughout the update's download, and as the update is checked it is deleted (489.7 MB freed, logged).
+`dev`: a dev copy with weights/weight.bin missing is not loaded, a whole one not outside a FLEX build, and on one it is
+the model, Ready without a download.
2026-10-05, a MacBook with an M4, macOS 27: everything passes. The stop is at 20%, the next download's first bytes
read 21%, the server answers the resumed weights 206 and the whole 488986336 bytes are handed over and checked. An
@@ -167,6 +244,16 @@ earlier run lost the network for a few seconds and its three retries failed at o
waits for connectivity. Not run: an expired resume (the server's signed address lasts a day), which starts the file
over.
+`download/build.sh && CFFIXED_USER_HOME="$(mktemp -d)" build/download neural`: the Neural Engine's model beside a
+first model stood in by sparse files of its sizes. 2026-10-06, before it is on Hugging Face: its download asks for
+separator-ane.mlmodelc/metadata.json, gets 404 and ends, the first model Ready throughout, and is not tried again that
+launch; a dev copy with weights/weight.bin missing is not loaded, a whole one not outside a FLEX build, and on one
+(a FLEXManager class registered) it is the Neural Engine's model.
+
+2026-10-06, iPhone 17 Pro simulator, iOS 27, with the Neural Engine's model in the code: everything passes. The
+simulator has no Neural Engine, so Automatic loads the GPU's copy of the first model as before. The first launch
+aborted in CoreAudio ("Start: RPC timeout. Apparently deadlocked") with the Mac's load average at 350; the next passed.
+
## The mic button (`button/`)
`SGRSingButton.m` on a field like the lyrics', in the state the launch line asks for, for screenshots.
diff --git a/harness/sing/button/sing-stubs.m b/harness/sing/button/sing-stubs.m
index 27dfbdd1..9a9893bc 100644
--- a/harness/sing/button/sing-stubs.m
+++ b/harness/sing/button/sing-stubs.m
@@ -47,7 +47,7 @@ NSString *SGSingLevelText(double level) {
double SGSingModelProgress(void) { return [defaults() doubleForKey:@"progress"]; }
BOOL SGSingModelWaitingForNetwork(void) { return [defaults() boolForKey:@"waiting"]; }
-NSString *SGSingModelSizeText(void) { return @"489 MB"; }
+NSString *SGSingModelSizeText(void) { return @"210.4 MB"; }
NSString *SGSingModelError(void) { return nil; }
void SGSingDownloadModel(void) {}
void SGSingCancelModelDownload(void) {}
diff --git a/harness/sing/download/main.m b/harness/sing/download/main.m
index 85192428..19e01748 100644
--- a/harness/sing/download/main.m
+++ b/harness/sing/download/main.m
@@ -1,9 +1,21 @@
-// The voice model's download (SGSingModel.m) as the tweak compiles it, run on the Mac against Hugging Face: the
-// four small files and part of the weights come in, the download is stopped, and the next one carries the weights
+// The voice model's download (SGSingModel.m) as the tweak compiles it, run on the Mac against Hugging Face: what the
+// old model's download left, stood in by files of their sizes, is deleted at launch; the four small files of
+// separator-ane.mlmodelc and part of its weights come in, the download is stopped, and the next one carries the weights
// on from where the stop left them (a 206 from the server) through to the move into place.
//
// ./build.sh && CFFIXED_USER_HOME=<empty dir> ../build/download [stop at, 0 to 1 of the weights]
+//
+// `update` starts from the old model (separator.mlmodelc, stood in by files of its sizes) instead: kept and used until
+// the new one is downloaded, then deleted.
+//
+// ./build.sh && CFFIXED_USER_HOME=<empty dir> ../build/download update
+//
+// `dev` checks the dev folder instead: used only with every file in it and only on a FLEX build (a FLEXManager class),
+// the model then Ready without a download.
+//
+// ./build.sh && CFFIXED_USER_HOME=<empty dir> ../build/download dev
#import <Foundation/Foundation.h>
+#import <objc/runtime.h>
#import "Shared/Sing/Sing.h"
static int failures;
@@ -14,6 +26,76 @@ static void runUntil(BOOL (^done)(void), NSTimeInterval limit) {
while (!done() && end.timeIntervalSinceNow > 0) CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.25, false);
}
+// Files of these sizes, sparse, at these paths under `folder`.
+static void standIn(NSURL *folder, NSDictionary<NSString *, NSNumber *> *files) {
+ for (NSString *path in files) {
+ NSURL *file = [folder URLByAppendingPathComponent:path];
+ [NSFileManager.defaultManager createDirectoryAtURL:file.URLByDeletingLastPathComponent withIntermediateDirectories:YES attributes:nil error:nil];
+ [NSFileManager.defaultManager createFileAtPath:file.path contents:nil attributes:nil];
+ truncate(file.fileSystemRepresentation, files[path].longLongValue);
+ }
+}
+
+static BOOL exists(NSURL *url) {
+ return [NSFileManager.defaultManager fileExistsAtPath:url.path];
+}
+
+// The dev folder: not loaded with a file missing (Core ML would crash on it), nor outside a FLEX build; on one it is
+// the model, Ready without a download.
+static int checkDev(NSURL *sing) {
+ NSURL *dev = [[sing URLByAppendingPathComponent:@"dev"] URLByAppendingPathComponent:@"separator-ane.mlmodelc"];
+ standIn(dev, @{@"metadata.json": @2333, @"coremldata.bin": @388, @"analytics/coremldata.bin": @243, @"model.mil": @1346642});
+ CHECK(!SGSingModelURL() && SGSingModelCurrentState() == SGSingModelMissing, @"a dev copy missing weights/weight.bin is not loaded");
+ standIn(dev, @{@"weights/weight.bin": @209077208});
+ SGSingCancelModelDownload(); // nothing to stop: it has the folders looked at again
+ CHECK(!SGSingModelURL(), @"a whole dev copy is not loaded outside a FLEX build");
+ objc_registerClassPair(objc_allocateClassPair(NSObject.class, "FLEXManager", 0));
+ SGSingCancelModelDownload();
+ CHECK(SGSingModelCurrentState() == SGSingModelReady && [SGSingModelURL().path isEqualToString:dev.path],
+ @"on a FLEX build it is the model, Ready without a download (%@)", SGSingModelURL().path);
+ printf("%s\n", failures ? "FAILED" : "all passed");
+ return failures ? 1 : 0;
+}
+
+// The old model on the iPhone, the new one not: the old one is kept, is the model Karaoke loads and reads as an update
+// available; it stays so while the update downloads, and goes as soon as the update is in. A launch that finds both
+// deletes the old one.
+static int checkUpdate(NSURL *sing) {
+ NSURL *old = [sing URLByAppendingPathComponent:@"separator.mlmodelc"];
+ NSDictionary *oldFiles = @{@"metadata.json": @2431, @"coremldata.bin": @507, @"analytics/coremldata.bin": @243, @"model.mil": @669061,
+ @"weights/weight.bin": @488986336};
+ standIn(old, oldFiles);
+ SGSingRemoveOldModel();
+ CHECK(exists(old), @"at launch, with the new model not in, the old one is kept");
+ CHECK(SGSingModelCurrentState() == SGSingModelReady && [SGSingModelURL().path isEqualToString:old.path] && SGSingModelUpdateAvailable(),
+ @"it is the model, Ready, and the download reads as its update (%@)", SGSingModelURL().lastPathComponent);
+ CHECK(SGSingModelProgress() == 0 && SGSingModelPausedBytes() == 0, @"the update is not downloaded");
+ printf("the model on the iPhone reads %s, its update %s\n", SGSingModelInUseSizeText().UTF8String, SGSingModelSizeText().UTF8String);
+
+ SGSingDownloadModelOverCellular();
+ __block BOOL readyThroughout = YES, sawUpdating = NO;
+ runUntil(^BOOL {
+ readyThroughout &= SGSingModelCurrentState() == SGSingModelReady;
+ if (SGSingModelUpdateDownloading()) {
+ sawUpdating = YES;
+ readyThroughout &= [SGSingModelURL().path isEqualToString:old.path];
+ }
+ return !SGSingModelUpdateDownloading() && sawUpdating;
+ }, 1800);
+ CHECK(sawUpdating && readyThroughout, @"while the update downloads the old model stays the model, Ready");
+ CHECK(!exists(old), @"the update in, the old model is deleted at once (%@)", SGSingModelError() ?: @"no error");
+ CHECK(SGSingModelCurrentState() == SGSingModelReady && [SGSingModelURL().lastPathComponent isEqualToString:@"separator-ane.mlmodelc"] && !SGSingModelUpdateAvailable(),
+ @"and the model is separator-ane.mlmodelc, no update left (%@)", SGSingModelURL().lastPathComponent);
+ NSArray *left = [NSFileManager.defaultManager contentsOfDirectoryAtPath:sing.path error:nil];
+ CHECK([left isEqualToArray:@[@"separator-ane.mlmodelc"]], @"nothing else in Sing/ (%@)", [left componentsJoinedByString:@", "]);
+
+ standIn(old, oldFiles);
+ SGSingRemoveOldModel();
+ CHECK(!exists(old) && exists([sing URLByAppendingPathComponent:@"separator-ane.mlmodelc"]), @"a launch that finds both deletes the old one");
+ printf("%s\n", failures ? "FAILED" : "all passed");
+ return failures ? 1 : 0;
+}
+
int main(int argc, char **argv) {
@autoreleasepool {
setvbuf(stdout, NULL, _IOLBF, 0);
@@ -21,32 +103,52 @@ int main(int argc, char **argv) {
NSURL *sing = [[[NSFileManager.defaultManager URLsForDirectory:NSApplicationSupportDirectory inDomains:NSUserDomainMask].firstObject
URLByAppendingPathComponent:@"Vitrine"] URLByAppendingPathComponent:@"Sing"];
printf("the model goes to %s\n", sing.path.UTF8String);
- CHECK(SGSingModelCurrentState() == SGSingModelMissing, @"no model to start with");
+ if (argc > 1 && !strcmp(argv[1], "dev")) return checkDev(sing);
- // The small files are 0.14% of the bytes, so the progress is nearly the weights' own.
- SGSingDownloadModel();
+ if (argc > 1 && !strcmp(argv[1], "update")) return checkUpdate(sing);
+
+ // A paused download of the old model, never finished.
+ standIn([sing URLByAppendingPathComponent:@"download"], @{@"model.mil": @669061});
+ standIn(sing, @{@"download-4.resume": @1000});
+ [NSUserDefaults.standardUserDefaults setInteger:97000000 forKey:@"spotifyglass.sing.downloadPaused"];
+ CHECK(SGSingModelCurrentState() == SGSingModelMissing && SGSingModelPausedBytes() == 97000000,
+ @"no model, and the old model's paused download reads %lld bytes", SGSingModelPausedBytes());
+ SGSingRemoveOldModel();
+ CHECK(!exists([sing URLByAppendingPathComponent:@"download"]) && !exists([sing URLByAppendingPathComponent:@"download-4.resume"]),
+ @"at launch the old download's staging folder and resume data are deleted");
+ CHECK(SGSingModelPausedBytes() == 0, @"and the paused bytes, which were the old model's, are forgotten");
+ printf("the model's size reads %s\n", SGSingModelSizeText().UTF8String);
+
+ // The small files are 0.65% of the bytes, so the progress is nearly the weights' own. This Mac's network may count
+ // as expensive, which the download otherwise waits out.
+ SGSingDownloadModelOverCellular();
runUntil(^BOOL { return SGSingModelProgress() >= stopAt || SGSingModelCurrentState() != SGSingModelDownloading; }, 900);
double stopped = SGSingModelProgress();
CHECK(SGSingModelCurrentState() == SGSingModelDownloading, @"the download runs into the weights (%.0f%%)", stopped * 100);
SGSingCancelModelDownload();
- NSURL *resume = [sing URLByAppendingPathComponent:@"download-4.resume"];
- runUntil(^BOOL { return [NSFileManager.defaultManager fileExistsAtPath:resume.path]; }, 10);
- CHECK([NSFileManager.defaultManager fileExistsAtPath:resume.path], @"the stop keeps the weights' resume data");
+ NSURL *resume = [sing URLByAppendingPathComponent:@"download-ane-4.resume"];
+ runUntil(^BOOL { return exists(resume); }, 10);
+ CHECK(exists(resume), @"the stop keeps the weights' resume data");
CHECK(SGSingModelCurrentState() == SGSingModelMissing, @"stopped, the model is still missing");
+ SGSingRemoveOldModel();
+ CHECK(exists(resume) && exists([sing URLByAppendingPathComponent:@"download-ane"]), @"the launch's cleanup leaves this model's stopped download alone");
// Started over, the first bytes read well under 1%; carried on, the stop's share.
SGSingDownloadModel();
__block double first = -1;
runUntil(^BOOL {
- if (first < 0 && SGSingModelProgress() > 0.005) first = SGSingModelProgress();
+ if (first < 0 && SGSingModelProgress() > 0.01) first = SGSingModelProgress();
return SGSingModelCurrentState() != SGSingModelDownloading;
}, 1800);
CHECK(first >= stopped - 0.01, @"the next download carries on from %.0f%%, not from the start", first * 100);
- CHECK(![NSFileManager.defaultManager fileExistsAtPath:resume.path], @"the resume data is used once");
+ CHECK(!exists(resume), @"the resume data is used once");
CHECK(SGSingModelCurrentState() == SGSingModelReady, @"the model is in place and checked (%@)", SGSingModelError() ?: @"no error");
+ CHECK([SGSingModelURL().lastPathComponent isEqualToString:@"separator-ane.mlmodelc"], @"it is separator-ane.mlmodelc (%@)", SGSingModelURL().path);
NSArray *inside = [NSFileManager.defaultManager contentsOfDirectoryAtPath:SGSingModelURL().path error:nil];
CHECK(inside && ![[inside componentsJoinedByString:@" "] containsString:@"resume"], @"nothing but the model in its folder (%@)",
[inside componentsJoinedByString:@", "]);
+ NSArray *left = [NSFileManager.defaultManager contentsOfDirectoryAtPath:sing.path error:nil];
+ CHECK([left isEqualToArray:@[@"separator-ane.mlmodelc"]], @"and nothing else in Sing/ (%@)", [left componentsJoinedByString:@", "]);
printf("%s\n", failures ? "FAILED" : "all passed");
return failures ? 1 : 0;
}
diff --git a/harness/sing/main.m b/harness/sing/main.m
index ce3cb57c..952ef283 100644
--- a/harness/sing/main.m
+++ b/harness/sing/main.m
@@ -3,11 +3,12 @@
//
// - the STFT: a sine's peak bin at the size torch.stft gives it (amplitude times the window's sum over two), and
// noise through the STFT and back unchanged;
-// - the loader (SGSingLoader.m): the CPU copy loaded and warmed first, then the faster copy on the compute units
-// asked for; a second want joining the load in flight; a load past its deadline abandoned, Sing Failed, and a
-// fresh load working while the abandoned one still runs, which is let go when it comes back; a purge during a
-// load dropping its result; the copies kept over a quick off and on, and dropped after the time kept; each
-// window on the faster copy only in the foreground; a window the faster copy fails done again on the CPU's;
+// - the loader (SGSingLoader.m): the CPU copy loaded and warmed first, then the Neural Engine copy of the same model
+// beside it (`cpu` for none); a second want joining the load in flight; a load past its deadline abandoned, Sing
+// Failed, and a fresh load working while the abandoned one still runs, which is let go when it comes back; a purge
+// during a load dropping its result; the copies kept over a quick off and on, and dropped after the time kept; every
+// window on the Neural Engine copy once it is in; a window it fails done again on the CPU's, and it not used again;
+// a Neural Engine copy past its deadline leaving Sing Ready on the CPU's, and dropped, as Sing.x drops it;
// - the model: its shapes, the time a window takes, and a voice mixed over chords taken apart: the vocals it finds
// against the voice, beside the mix's own score;
// - the engine: Spotify's mixer stood in for by the mix, pulled through SGSingEngineRender in IO buffers on a
@@ -24,8 +25,11 @@
// degrees left the same mirrored; back ahead it is exact again; and no turn clicks. Then the front the voice
// is held off: a turned head has it off to the side, the front catching up over 20 s, across +-180 degrees
// without a jump, and ahead again after a gap in the motion.
+// - the Neural Engine copy loading, without the model: copies that take a set time a window; the CPU's falling behind
+// while the budget is held spends none of it, and the budget starts full once the copy is in or has failed. A Neural
+// Engine copy that only falls behind reads unfailed; one that fails a window reads failed after CPU windows too.
//
-// ./build.sh && build/sing <separator.mlmodelc> <voice> <out dir> [all|cpu|gpu|ane] (the faster copy's units)
+// ./build.sh && build/sing <separator-ane.mlmodelc> <voice> <out dir> [ane|cpu] (with or without the Neural Engine copy)
// ./build.sh && build/sing spatial (the checks without the model: the lead, spatial voice, falling behind)
#import <Foundation/Foundation.h>
#import <Accelerate/Accelerate.h>
@@ -161,13 +165,6 @@ static void checkSTFT(void) {
#pragma mark - the model
-static MLComputeUnits unitsNamed(NSString *name) {
- if ([name isEqualToString:@"cpu"]) return MLComputeUnitsCPUOnly;
- if ([name isEqualToString:@"gpu"]) return MLComputeUnitsCPUAndGPU;
- if ([name isEqualToString:@"ane"]) return MLComputeUnitsCPUAndNeuralEngine;
- return MLComputeUnitsAll;
-}
-
static double now(void) {
return CFAbsoluteTimeGetCurrent();
}
@@ -210,37 +207,37 @@ static bool waitFor(double seconds, bool (^done)(void)) {
return done();
}
-// An MLModel whose every prediction fails, standing in for a GPU copy that iOS stops.
+// An MLModel whose every prediction fails, standing in for a Neural Engine copy whose window Core ML fails.
@interface SGFailingModel : MLModel
@end
@implementation SGFailingModel
- (id<MLFeatureProvider>)predictionFromFeatures:(id<MLFeatureProvider>)input error:(NSError **)error {
- if (error) *error = [NSError errorWithDomain:@"harness" code:1 userInfo:@{NSLocalizedDescriptionKey: @"refused, as iOS refuses the GPU in the background"}];
+ if (error) *error = [NSError errorWithDomain:@"harness" code:1 userInfo:@{NSLocalizedDescriptionKey: @"refused, as Core ML can refuse a window"}];
return nil;
}
@end
-// The loader through its cases; hands back the separator it leaves Ready, both copies in it.
-static SGSingSeparator *checkLoader(NSString *path, NSString *units) {
+// The loader through its cases; hands back the separator it leaves Ready, both copies in it. With `neural`, the
+// model's Neural Engine copy loads beside its CPU copy, as Sing.x has it on Automatic.
+static SGSingSeparator *checkLoader(NSString *path, bool neural) {
NSURL *url = [NSURL fileURLWithPath:path];
- MLComputeUnits fast = unitsNamed(units);
SGSingLoaderSetForeground(YES);
// A load past its deadline is abandoned and Sing is Failed; the mic off and on starts a fresh one at once,
// which works while the first is still out, and the first is let go when it comes back.
SGSingLoaderCPUDeadline = 0.2;
- SGSingLoaderWant(url, MLComputeUnitsCPUOnly);
+ SGSingLoaderWant(url, NO);
waitFor(5, ^bool { return SGSingLoaderCurrentState() != SGSingLoaderLoading; });
CHECK(SGSingLoaderCurrentState() == SGSingLoaderFailed && [SGSingLoaderError() containsString:@"did not load in"],
"a load past its deadline is abandoned, and Sing is Failed: %s", SGSingLoaderError().UTF8String ?: "(no reason)");
SGSingLoaderCPUDeadline = 120;
- SGSingLoaderWant(url, MLComputeUnitsCPUOnly);
+ SGSingLoaderWant(url, NO);
CHECK(SGSingLoaderCurrentState() == SGSingLoaderFailed && SGSingLoaderAttempts() == 1, "Failed stays Failed until the mic is switched off and on");
SGSingLoaderRelease();
double began = now();
- SGSingLoaderWant(url, fast);
- SGSingLoaderWant(url, fast);
+ SGSingLoaderWant(url, neural);
+ SGSingLoaderWant(url, neural);
CHECK(SGSingLoaderAttempts() == 2 && SGSingLoaderOutstanding() == 2, "off and on starts a fresh load beside the abandoned one, and a second want joins it (%u loads, %u out)",
SGSingLoaderAttempts(), SGSingLoaderOutstanding());
waitFor(300, ^bool { return SGSingLoaderCurrentState() != SGSingLoaderLoading; });
@@ -249,42 +246,56 @@ static SGSingSeparator *checkLoader(NSString *path, NSString *units) {
CHECK(SGSingLoaderCurrentState() == SGSingLoaderReady && separator, "the CPU copy loads and warms in %.1f s, and Sing is Ready", ready);
waitFor(300, ^bool { return SGSingLoaderOutstanding() == 0 && SGSingLoaderFastState() != SGSingFastLoading; });
CHECK(SGSingLoaderSeparator() == separator, "the abandoned load came back and was let go: the separator is still the fresh load's");
- if (fast != MLComputeUnitsCPUOnly) {
- CHECK(SGSingLoaderFastState() == SGSingFastReady, "then the %s copy loads and warms beside it, %.1f s after the want (state %ld)",
- SGSingUnitsName(fast).UTF8String, now() - began, (long)SGSingLoaderFastState());
+ if (neural) {
+ CHECK(SGSingLoaderFastState() == SGSingFastReady, "then the Neural Engine copy loads and warms beside it, %.1f s after the want (state %ld)",
+ now() - began, (long)SGSingLoaderFastState());
+ } else {
+ CHECK(SGSingLoaderFastState() == SGSingFastNone, "and no Neural Engine copy loads (state %ld)", (long)SGSingLoaderFastState());
}
// Kept over a quick off and on, dropped once the time kept has passed.
SGSingLoaderKeepSeconds = 1.5;
SGSingLoaderRelease();
waitFor(0.5, ^bool { return false; });
- SGSingLoaderWant(url, fast);
+ SGSingLoaderWant(url, neural);
waitFor(2, ^bool { return false; });
- CHECK(SGSingLoaderSeparator() == separator && SGSingLoaderAttempts() == 2 + (fast != MLComputeUnitsCPUOnly),
+ CHECK(SGSingLoaderSeparator() == separator && SGSingLoaderAttempts() == 2 + neural,
"switched off and on within the time kept, the copies are still there and nothing loads (%u loads)", SGSingLoaderAttempts());
SGSingLoaderRelease();
waitFor(3, ^bool { return SGSingLoaderCurrentState() == SGSingLoaderIdle; });
CHECK(SGSingLoaderCurrentState() == SGSingLoaderIdle && !SGSingLoaderSeparator(), "left off past the time kept, the copies are dropped");
// A purge during a load: what the load brings back is let go.
- SGSingLoaderWant(url, MLComputeUnitsCPUOnly);
+ SGSingLoaderWant(url, NO);
SGSingLoaderPurge(@"the harness purges during the load");
waitFor(300, ^bool { return SGSingLoaderOutstanding() == 0; });
waitFor(0.5, ^bool { return false; });
CHECK(SGSingLoaderCurrentState() == SGSingLoaderIdle && !SGSingLoaderSeparator(), "a load purged on its way is let go when it comes back");
- // The separator for the rest of the run, both copies in it, as Sing.x has it.
+ // Another model wanted (the update in place of the old one), here the same files through a link: the first model's
+ // copies go at once and the other loads.
SGSingLoaderKeepSeconds = 60;
- SGSingLoaderWant(url, fast);
- waitFor(300, ^bool { return SGSingLoaderCurrentState() != SGSingLoaderLoading
- && (fast == MLComputeUnitsCPUOnly || SGSingLoaderFastState() != SGSingFastLoading); });
+ NSString *link = [NSTemporaryDirectory() stringByAppendingPathComponent:[NSString stringWithFormat:@"sing-other-%d.mlmodelc", getpid()]];
+ [NSFileManager.defaultManager createSymbolicLinkAtPath:link withDestinationPath:path error:nil];
+ SGSingLoaderWant(url, NO);
+ waitFor(300, ^bool { return SGSingLoaderCurrentState() != SGSingLoaderLoading; });
+ SGSingSeparator *first = SGSingLoaderSeparator();
+ SGSingLoaderWant([NSURL fileURLWithPath:link], NO);
+ CHECK(first && SGSingLoaderCurrentState() == SGSingLoaderLoading && !SGSingLoaderSeparator(), "another model wanted: the first's copies go at once and it loads");
+ waitFor(300, ^bool { return SGSingLoaderCurrentState() != SGSingLoaderLoading; });
+ CHECK(SGSingLoaderSeparator() && SGSingLoaderSeparator() != first && [SGSingLoaderURL().path isEqualToString:link], "and Sing is Ready on it");
+ [NSFileManager.defaultManager removeItemAtPath:link error:nil];
+
+ // The separator for the rest of the run, both copies in it, as Sing.x has it (the first model again).
+ SGSingLoaderWant(url, neural);
+ waitFor(300, ^bool { return SGSingLoaderCurrentState() != SGSingLoaderLoading && SGSingLoaderFastState() != SGSingFastLoading; });
separator = SGSingLoaderSeparator();
- CHECK(separator != nil, "loaded again for the rest of the run");
+ CHECK(separator != nil && [SGSingLoaderURL() isEqual:url], "loaded again for the rest of the run");
return separator;
}
-// Each window on the copy that fits: the faster only in the foreground, a failed one done again on the CPU's, which
-// keeps it until the app has been in the background.
+// Each window on the faster copy while it is in; a failed one done again on the CPU's, which keeps the rest until
+// another faster copy is handed over.
static void checkCopies(SGSingSeparator *separator, Audio mix, bool hasFast) {
float *vl = calloc(kSGSingWindowFrames, sizeof(float)), *vr = calloc(kSGSingWindowFrames, sizeof(float));
NSError *error;
@@ -292,37 +303,74 @@ static void checkCopies(SGSingSeparator *separator, Audio mix, bool hasFast) {
if (hasFast) {
[separator separateLeft:mix.left right:mix.right vocalsLeft:vl vocalsRight:vr error:&error];
SGSingSeparatorStats stats = [separator stats];
- CHECK(stats.fast && stats.windows[1] == before.windows[1] + 1, "in the foreground a window goes to the faster copy");
- SGSingSeparatorSetForeground(false);
- [separator separateLeft:mix.left right:mix.right vocalsLeft:vl vocalsRight:vr error:&error];
- stats = [separator stats];
- CHECK(!stats.fast && stats.windows[0] == before.windows[0] + 1, "in the background it goes to the CPU's");
- SGSingSeparatorSetForeground(true);
+ CHECK(stats.fast && stats.windows[1] == before.windows[1] + 1, "a window goes to the Neural Engine copy");
before = stats;
}
- // The fast copy swapped for one that fails, then put back.
+ // The fast copy swapped for one that fails, then for a working one.
SGFailingModel *failing = [SGFailingModel alloc];
[separator setFastModel:failing named:@"failing"];
BOOL ok = [separator separateLeft:mix.left right:mix.right vocalsLeft:vl vocalsRight:vr error:&error];
SGSingSeparatorStats stats = [separator stats];
- CHECK(ok && stats.fallbacks == before.fallbacks + 1 && stats.windows[0] == before.windows[0] + 1 && !stats.fast,
+ CHECK(ok && stats.fallbacks == before.fallbacks + 1 && stats.windows[0] == before.windows[0] + 1 && !stats.fast && stats.fastFailed,
"a window the faster copy fails is done again on the CPU's (%s)", ok ? "separated" : error.localizedDescription.UTF8String);
[separator separateLeft:mix.left right:mix.right vocalsLeft:vl vocalsRight:vr error:&error];
stats = [separator stats];
- CHECK(stats.fallbacks == before.fallbacks + 1 && stats.windows[0] == before.windows[0] + 2, "and the faster copy rests for the next window");
- SGSingSeparatorSetForeground(false);
- SGSingSeparatorSetForeground(true);
- [separator separateLeft:mix.left right:mix.right vocalsLeft:vl vocalsRight:vr error:&error];
- CHECK([separator stats].fallbacks == before.fallbacks + 2, "until the app has been in the background, when it is tried again");
+ CHECK(stats.fallbacks == before.fallbacks + 1 && stats.windows[0] == before.windows[0] + 2, "and the failed copy is not tried again");
+ [separator setFastModel:nil named:nil];
free(vl);
free(vr);
}
+// The model's Neural Engine copy against its CPU copy on the same window: the vocals they find alike. The CPU's window
+// is had with the Neural Engine copy dropped, which is then loaded again (from Core ML's cache).
+static void checkNeuralMatches(SGSingSeparator *separator, Audio mix, NSString *path) {
+ size_t at = (size_t)kSGSingRate * 5;
+ Audio cpu = makeAudio(kSGSingWindowFrames), neural = makeAudio(kSGSingWindowFrames);
+ NSError *error;
+ SGSingLoaderDropFast(nil);
+ BOOL cpuOK = [separator separateLeft:mix.left + at right:mix.right + at vocalsLeft:cpu.left vocalsRight:cpu.right error:&error];
+ bool onCPU = ![separator stats].fast;
+ SGSingLoaderWant([NSURL fileURLWithPath:path], YES);
+ waitFor(300, ^bool { return SGSingLoaderFastState() != SGSingFastLoading; });
+ double began = now();
+ BOOL neuralOK = [separator separateLeft:mix.left + at right:mix.right + at vocalsLeft:neural.left vocalsRight:neural.right error:&error];
+ double took = now() - began;
+ bool onNeural = [separator stats].fast;
+ double alike = snr(cpu, neural, kSGSingWindowFrames / 8, kSGSingWindowFrames * 7 / 8);
+ CHECK(cpuOK && neuralOK && onCPU && onNeural && alike > 25,
+ "on the same window the Neural Engine copy (%.0f ms) finds the vocals the CPU copy does, %.1f dB apart", took * 1000, alike);
+}
+
+// The Neural Engine copy past its deadline: it times out and the CPU's carries on; asked for no Neural Engine copy then,
+// as Sing.x does once it has failed, the windows stay on the CPU's.
+static void checkNeuralFallback(NSString *path) {
+ NSURL *url = [NSURL fileURLWithPath:path];
+ SGSingSeparator *separator = SGSingLoaderSeparator();
+ SGSingLoaderWant(url, NO);
+ CHECK(SGSingLoaderFastState() == SGSingFastNone && ![separator stats].fast, "asked for none, the Neural Engine copy is dropped");
+ SGSingLoaderNeuralDeadline = 0.001;
+ SGSingLoaderWant(url, YES);
+ waitFor(60, ^bool { return SGSingLoaderFastState() != SGSingFastLoading; });
+ CHECK(SGSingLoaderFastState() == SGSingFastTimedOut && SGSingLoaderSeparator() == separator && SGSingLoaderCurrentState() == SGSingLoaderReady,
+ "the Neural Engine copy past its deadline times out, and Sing stays Ready on the CPU's");
+ SGSingLoaderNeuralDeadline = 600;
+ SGSingLoaderWant(url, NO);
+ float *vl = calloc(kSGSingWindowFrames, sizeof(float)), *vr = calloc(kSGSingWindowFrames, sizeof(float));
+ float *silence = calloc(kSGSingWindowFrames, sizeof(float));
+ NSError *error;
+ BOOL ok = [separator separateLeft:silence right:silence vocalsLeft:vl vocalsRight:vr error:&error];
+ CHECK(ok && ![separator stats].fast && SGSingLoaderFastState() == SGSingFastNone, "then asked for none, nothing loads and the windows go to the CPU's");
+ waitFor(300, ^bool { return SGSingLoaderOutstanding() == 0; });
+ free(vl);
+ free(vr);
+ free(silence);
+}
+
// Which device Core ML means to run each operation on, counted, where the OS can tell.
-static void describePlan(NSString *path, NSString *units) {
+static void describePlan(NSString *path, bool neural) {
if (@available(macOS 14.4, *)) {
MLModelConfiguration *configuration = [MLModelConfiguration new];
- configuration.computeUnits = unitsNamed(units);
+ configuration.computeUnits = neural ? MLComputeUnitsCPUAndNeuralEngine : MLComputeUnitsCPUOnly;
dispatch_semaphore_t done = dispatch_semaphore_create(0);
[MLComputePlan loadContentsOfURL:[NSURL fileURLWithPath:path] configuration:configuration completionHandler:^(MLComputePlan *plan, NSError *error) {
NSMutableDictionary<NSString *, NSNumber *> *counts = [NSMutableDictionary dictionary];
@@ -335,7 +383,7 @@ static void describePlan(NSString *path, NSString *units) {
: [device isKindOfClass:MLGPUComputeDevice.class] ? @"GPU" : @"CPU";
counts[name] = @(counts[name].intValue + 1);
}
- printf(" info the plan at %s: %s\n", units.UTF8String, counts.description.UTF8String);
+ printf(" info the plan on %s: %s\n", neural ? "the CPU and Neural Engine" : "the CPU", counts.description.UTF8String);
dispatch_semaphore_signal(done);
}];
dispatch_semaphore_wait(done, DISPATCH_TIME_FOREVER);
@@ -667,6 +715,132 @@ static void checkFallingBehind(void) {
SGSingEngineDestroy(poisoned);
}
+// A copy of the model without the model, on the compute units it is given: each prediction takes `seconds` and hands
+// back silence as the vocals, or fails.
+@interface SGMockModel : MLModel
+@property double seconds; // atomic: changed while the worker runs a window
+@property (nonatomic) bool fails;
+@property (nonatomic) MLComputeUnits units;
+@end
+
+@implementation SGMockModel
+- (MLModelConfiguration *)configuration {
+ MLModelConfiguration *configuration = [MLModelConfiguration new];
+ configuration.computeUnits = _units;
+ return configuration;
+}
+- (id<MLFeatureProvider>)predictionFromFeatures:(id<MLFeatureProvider>)input error:(NSError **)error {
+ usleep((useconds_t)(self.seconds * 1e6));
+ if (_fails) {
+ if (error) *error = [NSError errorWithDomain:@"harness" code:2 userInfo:@{NSLocalizedDescriptionKey: @"the mock copy fails every window"}];
+ return nil;
+ }
+ MLMultiArray *vocals = [[MLMultiArray alloc] initWithShape:@[@1, @(2 * kSGSingBins), @(kSGSingSTFTFrames), @2] dataType:MLMultiArrayDataTypeFloat32 error:error];
+ [vocals getMutableBytesWithHandler:^(void *bytes, NSInteger size, NSArray<NSNumber *> *strides) { memset(bytes, 0, (size_t)size); }];
+ return vocals ? [[MLDictionaryFeatureProvider alloc] initWithDictionary:@{@"vocals_spectrum": vocals} error:error] : nil;
+}
+@end
+
+static SGMockModel *mockCopy(MLComputeUnits units, double seconds) {
+ SGMockModel *model = [SGMockModel alloc];
+ model.units = units;
+ model.seconds = seconds;
+ return model;
+}
+
+// The Neural Engine copy loading while the CPU's falls behind (Sing.x holds the budget while the loader reads
+// Loading): a Neural Engine copy keeps up, is dropped at 6 s and another loads until 24 s (a first compile), the CPU's
+// taking 2.2 s a window meanwhile. Held, the engine never gives up; let go with the copy in, the vocals come back; let
+// go with the load failed, the budget starts then, full, and the engine gives up 8 s on, as before.
+static void checkFasterCopyLoading(void) {
+ size_t frames = 40 * kSGSingRate;
+ Audio tone = makeAudio(frames);
+ for (size_t i = 0; i < frames; i++) tone.left[i] = tone.right[i] = (float)(0.5 * sin(2 * M_PI * 440 * i / kSGSingRate));
+ for (int loads = 1; loads >= 0; loads--) {
+ SGSingSeparator *separator = [[SGSingSeparator alloc] initWithModel:mockCopy(MLComputeUnitsCPUOnly, 2.2)];
+ [separator setFastModel:mockCopy(MLComputeUnitsCPUAndNeuralEngine, 0.25) named:@"Neural Engine (mock)"];
+ SGSingEngine *engine = SGSingEngineCreate();
+ SGSingEngineSetSeparator(engine, separator);
+ SGSingEngineSetLevel(engine, 0);
+ SGSingEngineSetOn(engine, true);
+ Source source = {tone, 0};
+ __block bool inBefore = false, outWhileHeld = false, backAfter = false;
+ __block double gaveUpAt = 0, spentWhileHeld = 0;
+ play(engine, tone, &source, 1024, loads ? 32 : 36, ^(double played) {
+ SGSingEngineStats stats = SGSingEngineReadStats(engine);
+ if (played < 6) inBefore |= stats.mixing;
+ if (played >= 6 && !stats.budgetHeld && played < 24) {
+ [separator setFastModel:nil named:nil];
+ SGSingEngineHoldBudget(engine, true);
+ }
+ if (played >= 10 && played < 24) {
+ outWhileHeld |= !stats.mixing;
+ spentWhileHeld = fmax(spentWhileHeld, stats.budgetSpent);
+ }
+ if (played >= 24 && stats.budgetHeld) {
+ if (loads) [separator setFastModel:mockCopy(MLComputeUnitsCPUAndNeuralEngine, 0.3) named:@"Neural Engine (mock, compiled)"];
+ SGSingEngineHoldBudget(engine, false);
+ }
+ if (played >= 24) backAfter |= stats.mixing;
+ if (!gaveUpAt && SGSingEngineGaveUp(engine)) gaveUpAt = played;
+ });
+ if (loads) {
+ CHECK(inBefore && outWhileHeld && spentWhileHeld == 0 && (!gaveUpAt || gaveUpAt >= 24),
+ "a Neural Engine copy loading from 6 s to 24 s while the CPU's falls behind: the vocals went out, none of the 8 s was spent (%.1f s) and the engine did not give up",
+ spentWhileHeld);
+ CHECK(backAfter && !gaveUpAt && [separator stats].fast, "with the Neural Engine copy in at 24 s the vocals come back and the windows go to it");
+ } else {
+ CHECK(gaveUpAt >= 31.9 && gaveUpAt < 33,
+ "with that load failed at 24 s, the budget starts then, full: the engine gives up at %.1f s (8 s on), as a copy that cannot keep up does", gaveUpAt);
+ }
+ SGSingEngineDestroy(engine);
+ }
+}
+
+// What Sing.x drops a Neural Engine copy on (`fastFailed`): a failed window, not a slow one. A copy that falls behind
+// until the engine gives up still reads unfailed, the last window on it; one that fails a window reads failed after the
+// CPU's has run the windows since; a fresh copy reads unfailed again.
+static void checkNeuralFaults(void) {
+ size_t frames = 30 * kSGSingRate;
+ Audio tone = makeAudio(frames);
+ for (size_t i = 0; i < frames; i++) tone.left[i] = tone.right[i] = (float)(0.5 * sin(2 * M_PI * 440 * i / kSGSingRate));
+ SGSingSeparator *separator = [[SGSingSeparator alloc] initWithModel:mockCopy(MLComputeUnitsCPUOnly, 2.2)];
+ SGMockModel *slow = mockCopy(MLComputeUnitsCPUAndNeuralEngine, 0.25);
+ [separator setFastModel:slow named:@"Neural Engine (mock)"];
+ SGSingEngine *engine = SGSingEngineCreate();
+ SGSingEngineSetSeparator(engine, separator);
+ SGSingEngineSetOn(engine, true);
+ Source source = {tone, 0};
+ __block double gaveUpAt = 0;
+ play(engine, tone, &source, 1024, 24, ^(double played) {
+ if (played >= 6) slow.seconds = 2.2;
+ if (!gaveUpAt && SGSingEngineGaveUp(engine)) gaveUpAt = played;
+ });
+ SGSingEngineDestroy(engine);
+ SGSingSeparatorStats stats = [separator stats];
+ CHECK(gaveUpAt > 0 && stats.fast && !stats.fastFailed && stats.fallbacks == 0,
+ "a Neural Engine copy slowed to 2.2 s a window: the engine gives up (%.1f s), and the copy reads unfailed, so Sing.x keeps it", gaveUpAt);
+
+ float *vl = calloc(kSGSingWindowFrames, sizeof(float)), *vr = calloc(kSGSingWindowFrames, sizeof(float));
+ NSError *error;
+ SGSingSeparator *quick = [[SGSingSeparator alloc] initWithModel:mockCopy(MLComputeUnitsCPUOnly, 0)];
+ SGMockModel *failing = mockCopy(MLComputeUnitsCPUAndNeuralEngine, 0);
+ failing.fails = true;
+ [quick setFastModel:failing named:@"Neural Engine (mock, failing)"];
+ BOOL first = [quick separateLeft:tone.left right:tone.right vocalsLeft:vl vocalsRight:vr error:&error];
+ BOOL second = [quick separateLeft:tone.left right:tone.right vocalsLeft:vl vocalsRight:vr error:&error];
+ stats = [quick stats];
+ CHECK(first && second && stats.fallbacks == 1 && stats.windows[0] == 2 && !stats.fast && stats.fastFailed,
+ "a Neural Engine copy that fails a window: the window done on the CPU's, and after a second window on the CPU's "
+ "it still reads failed, so Sing.x drops it");
+ [quick setFastModel:mockCopy(MLComputeUnitsCPUAndNeuralEngine, 0) named:@"Neural Engine (mock, fresh)"];
+ [quick separateLeft:tone.left right:tone.right vocalsLeft:vl vocalsRight:vr error:&error];
+ stats = [quick stats];
+ CHECK(stats.fast && !stats.fastFailed, "a fresh Neural Engine copy put in its place reads unfailed and takes the window");
+ free(vl);
+ free(vr);
+}
+
static double energy(const float *samples, size_t from, size_t to) {
double sum = 0;
for (size_t i = from; i < to; i++) sum += (double)samples[i] * samples[i];
@@ -781,19 +955,23 @@ int main(int argc, char **argv) {
checkSpatial();
checkFront();
checkFallingBehind();
+ checkFasterCopyLoading();
+ checkNeuralFaults();
printf("%s\n", sg_failures ? "FAILED" : "all passed");
return sg_failures ? 1 : 0;
}
- if (argc < 4) {
- printf("usage: sing <separator.mlmodelc> <voice> <out dir> [all|cpu|gpu|ane] (the faster copy's units; cpu for none)\n sing spatial\n");
+ if (argc < 4 || (argc > 4 && strcmp(argv[4], "ane") && strcmp(argv[4], "cpu"))) {
+ printf("usage: sing <separator-ane.mlmodelc> <voice> <out dir> [ane|cpu] (with the Neural Engine copy beside the CPU's, the default, or the CPU's alone)\n"
+ " sing spatial\n");
return 2;
}
- NSString *modelPath = @(argv[1]), *units = argc > 4 ? @(argv[4]) : @"all";
+ NSString *modelPath = @(argv[1]);
+ bool neural = argc < 5 || !strcmp(argv[4], "ane");
sg_outDir = @(argv[3]);
[NSFileManager.defaultManager createDirectoryAtPath:sg_outDir withIntermediateDirectories:YES attributes:nil error:nil];
checkSTFT();
- SGSingSeparator *separator = checkLoader(modelPath, units);
+ SGSingSeparator *separator = checkLoader(modelPath, neural);
if (!separator) return 1;
MLModel *shapes = [MLModel modelWithContentsOfURL:[NSURL fileURLWithPath:modelPath] configuration:[MLModelConfiguration new] error:nil];
MLFeatureDescription *input = shapes.modelDescription.inputDescriptionsByName[@"spectrum"];
@@ -801,7 +979,7 @@ int main(int argc, char **argv) {
CHECK([input.multiArrayConstraint.shape isEqualToArray:(@[@1, @2050, @201, @2])] && [output.multiArrayConstraint.shape isEqualToArray:(@[@1, @2050, @201, @2])],
"its spectrum and vocals_spectrum are [1, 2050, 201, 2]");
shapes = nil;
- describePlan(modelPath, units);
+ describePlan(modelPath, neural);
Audio voice = readAudio(@(argv[2]));
size_t frames = (size_t)kSGSingRate * 30;
@@ -827,17 +1005,21 @@ int main(int argc, char **argv) {
}
printf(" info the mix less them scores %.1f dB against the chords, the mix itself %.1f dB\n", snr(backing, accompaniment, kSGSingRate, to),
snr(backing, mix, kSGSingRate, to));
+ if (neural) checkNeuralMatches(separator, mix, modelPath);
writeWAV(@"mix.wav", mix);
writeWAV(@"vocals.wav", vocals);
writeWAV(@"accompaniment.wav", accompaniment);
checkEngine(separator, mix, padded, vocals);
// Last, as it swaps the faster copy out.
- checkCopies(separator, mix, unitsNamed(units) != MLComputeUnitsCPUOnly);
+ checkCopies(separator, mix, neural);
+ if (neural) checkNeuralFallback(modelPath);
checkHold();
checkSpatial();
checkFront();
checkFallingBehind();
+ checkFasterCopyLoading();
+ checkNeuralFaults();
printf("%s\n", sg_failures ? "FAILED" : "all passed");
return sg_failures ? 1 : 0;
}
diff --git a/harness/spatial-page/stubs.m b/harness/spatial-page/stubs.m
index 29bb92fe..db057013 100644
--- a/harness/spatial-page/stubs.m
+++ b/harness/spatial-page/stubs.m
@@ -23,7 +23,7 @@ NSString *SGSingStatusText(void) {
NSString *SGSingStatusDetail(void) {
SGSingState state = SGSingCurrentState();
return state == SGSingStateFailed ? @"The voice model did not load in 120 s, so that load was given up. Switch Sing off and on again to try again."
- : state == SGSingStateSinging ? @"Karaoke runs on the GPU while Spotify is open, and on the CPU in the background." : nil;
+ : state == SGSingStateSinging ? @"Karaoke runs on the Neural Engine." : nil;
}
NSString *SGSingMissing(void) { return nil; }
BOOL SGSingOn(void) { return SGHidden(SGKeySing); }
@@ -72,11 +72,15 @@ BOOL SGSingModelOverCellular(void) { return NO; }
void SGSingDownloadModelOverCellular(void) {}
long long SGSingModelPausedBytes(void) { return SGSingModelCurrentState() == SGSingModelMissing ? sg_pausedBytes : 0; }
NSString *SGSingModelError(void) { return nil; }
-NSString *SGSingModelSizeText(void) { return @"489 MB"; }
+NSString *SGSingModelSizeText(void) { return @"210.4 MB"; }
+// `-update 1` on the launch line: the old model in place, its update not downloaded.
+BOOL SGSingModelUpdateAvailable(void) { return SGSingModelCurrentState() == SGSingModelReady && [NSUserDefaults.standardUserDefaults boolForKey:@"update"]; }
+BOOL SGSingModelUpdateDownloading(void) { return NO; }
+NSString *SGSingModelInUseSizeText(void) { return SGSingModelUpdateAvailable() ? @"489.7 MB" : @"210.4 MB"; }
void SGSingDownloadModel(void) {}
void SGSingCancelModelDownload(void) {}
void SGSingDeleteModel(void) {}
-NSArray<NSString *> *SGSingComputeUnitNames(void) { return @[@"Automatic", @"CPU only", @"GPU", @"Neural Engine", @"GPU and Neural Engine"]; }
+NSArray<NSString *> *SGSingComputeUnitNames(void) { return @[@"Automatic", @"CPU only"]; }
BOOL SGSingOSSupported(void) { return YES; }
BOOL SGSingDeviceSupported(void) { return YES; }
diff --git a/tweak/Sources/App/About/Licenses.m b/tweak/Sources/App/About/Licenses.m
index 83b5a259..d6602340 100644
--- a/tweak/Sources/App/About/Licenses.m
+++ b/tweak/Sources/App/About/Licenses.m
@@ -54,6 +54,7 @@ UIViewController *SGLicensesPage(void) {
@"Checkpoint: KimberleyJensen, https://huggingface.co/KimberleyJSN/melbandroformer (MIT).\n"
@"Core ML conversion: john-rocky / mlboydaisuke, https://huggingface.co/mlboydaisuke/MelBandRoformer-Vocal-CoreAI (MIT).\n"
@"Core ML export: Darkkos, https://huggingface.co/Darkkos/spoti-sing (MIT), mirrored unchanged.\n"
+ @"Neural Engine export, the model Karaoke downloads: Vitrine, from the same checkpoint, palettized to 6 bits.\n"
@"Reference architecture: lucidrains/BS-RoFormer. Training code: ZFTurbo.\n\n"
@"Copyright (c) 2023 Phil Wang\n\n" stringByAppendingString:kMIT]),
] footer:nil];
diff --git a/tweak/Sources/Shared/Sing/SGSingCard.m b/tweak/Sources/Shared/Sing/SGSingCard.m
index c108c02a..a144b709 100644
--- a/tweak/Sources/Shared/Sing/SGSingCard.m
+++ b/tweak/Sources/Shared/Sing/SGSingCard.m
@@ -10,6 +10,13 @@
// Animation alone, with no display link; while none come (a pause), they rest. With Sing off, loading or held they
// lie flat. Under Reduce Motion they do not scroll: they are redrawn in place every two seconds.
//
+// Reading the lines is how Sing knows the page shows ("the lines" in its log are these two, not the lyrics' lines):
+// at As sung with Spatial voice off, where Sing otherwise rests, it separates the playing song while they are read,
+// so the lines move and the slider and the three stops are heard at once, and rests again a second after the last
+// read. The timer stops with the page off screen (popped, or covered by another page) and reads nothing while
+// Spotify is in the background. At any other level Sing separates whether the page shows or not. Every other gate
+// (the switch, the model, the heat, memory, Runs on, the model kept a minute after resting) stays Sing.x's.
+//
// Threading: main thread.
#import "Core/SGCore.h"
#import "Core/SGGlass.h"
@@ -323,6 +330,8 @@ static CGFloat height(float rms) {
}
- (void)tick {
+ // Spotify in the background plays on and so runs this timer: no reads then, so Sing rests as with the page gone.
+ if (UIApplication.sharedApplication.applicationState == UIApplicationStateBackground) return;
// The words ten times a second, the lines at every read.
if (_ticks++ % 3 == 0) [self refresh];
[self drawLines:YES];
diff --git a/tweak/Sources/Shared/Sing/SGSingEngine.h b/tweak/Sources/Shared/Sing/SGSingEngine.h
index e7f60b0f..e0e563cd 100644
--- a/tweak/Sources/Shared/Sing/SGSingEngine.h
+++ b/tweak/Sources/Shared/Sing/SGSingEngine.h
@@ -80,6 +80,9 @@ static inline double SGSpatialVoiceAngle(SGSpatialFront *f, double yaw, double t
// The vocals fell short for longer than the engine's budget: it plays the song as it is, the lead kept as it is,
// until switched off and on again.
bool SGSingEngineGaveUp(SGSingEngine *engine);
+// A faster copy of the model is loading: the slower one falling behind meanwhile is not yet a reason to give up, so no
+// budget is spent, and it starts full once the hold is let go (the faster copy in, or failed).
+void SGSingEngineHoldBudget(SGSingEngine *engine, bool held);
// Holds the worker (a hot phone, or Sing resting at As sung): what plays is dry, the lead kept as it is.
void SGSingEngineSetPaused(SGSingEngine *engine, bool paused);
// Drops the sound held ahead, at the next render; the lead reads as dropped from now.
@@ -102,6 +105,7 @@ typedef struct {
unsigned long long written, played; // frames pulled from Spotify's mixer and played, since the engine was made
bool mixing; // the vocals are turned down now (the hysteresis in SGSingEngine.m)
double budgetSpent; // seconds short of vocals since they were last in, of the 8 s it gives up after
+ bool budgetHeld; // none spent while a faster copy loads (SGSingEngineHoldBudget)
} SGSingEngineStats;
SGSingEngineStats SGSingEngineReadStats(SGSingEngine *engine);
// The loudness (RMS) of the separated vocals and of the rest of the song, per tenth of a second, for the `count`
diff --git a/tweak/Sources/Shared/Sing/SGSingEngine.m b/tweak/Sources/Shared/Sing/SGSingEngine.m
index b91069ac..6b5761f4 100644
--- a/tweak/Sources/Shared/Sing/SGSingEngine.m
+++ b/tweak/Sources/Shared/Sing/SGSingEngine.m
@@ -23,6 +23,7 @@ enum { kHistory = 1024, kAsked = 8 };
// the fade), and comes back to them only with a full hop separated ahead, which the lead (a window, 1.5 times the
// model's time and the spare) reaches just as each window lands. Time spent so after the vocals were first in comes
// out of a budget, renewed by that long with the vocals in throughout; spent, the engine gives up (SGSingEngineGaveUp).
+// While a faster copy is on its way (SGSingEngineHoldBudget) none is spent, and the budget starts full once it is in.
static const uint64_t kReserve = kSGSingRate * 12 / 100;
static const uint64_t kReturn = kSGSingEngineHop;
static const uint64_t kBudget = kSGSingRate * 8;
@@ -49,6 +50,7 @@ struct SGSingEngine {
atomic_bool flushAsked, on, paused, hasSeparator;
atomic_bool mixing, gaveUp; // the render's, for the stats and Sing.x
atomic_bool freshBudget; // switched on again: the render starts the budget over
+ atomic_bool budgetHeld; // a faster copy is on its way: falling behind spends no budget
_Atomic uint64_t spent; // the render's `recovering`, for the stats
_Atomic uint64_t dropped; // frames of held sound never played: flushes
// What Spotify's clock had counted past what plays, at each render that changed it (written less played,
@@ -218,6 +220,8 @@ OSStatus SGSingEngineRender(SGSingEngine *engine, UInt32 frames, float *left, fl
if (mixing) {
engine->steady += frames;
if (engine->steady >= kBudget) engine->recovering = 0;
+ } else if (atomic_load_explicit(&engine->budgetHeld, memory_order_relaxed)) {
+ engine->recovering = 0;
} else if (working && engine->established) {
engine->recovering += frames;
if (engine->recovering >= kBudget) atomic_store_explicit(&engine->gaveUp, true, memory_order_relaxed);
@@ -466,6 +470,10 @@ void SGSingEngineSetOn(SGSingEngine *engine, bool on) {
atomic_store(&engine->on, on);
}
+void SGSingEngineHoldBudget(SGSingEngine *engine, bool held) {
+ atomic_store(&engine->budgetHeld, held);
+}
+
bool SGSingEngineGaveUp(SGSingEngine *engine) {
return atomic_load(&engine->gaveUp);
}
@@ -554,6 +562,7 @@ SGSingEngineStats SGSingEngineReadStats(SGSingEngine *engine) {
.played = played,
.mixing = atomic_load(&engine->mixing),
.budgetSpent = atomic_load(&engine->spent) / (double)kSGSingRate,
+ .budgetHeld = atomic_load(&engine->budgetHeld),
};
}
diff --git a/tweak/Sources/Shared/Sing/SGSingLoader.h b/tweak/Sources/Shared/Sing/SGSingLoader.h
index ee08167d..6ecc0b4a 100644
--- a/tweak/Sources/Shared/Sing/SGSingLoader.h
+++ b/tweak/Sources/Shared/Sing/SGSingLoader.h
@@ -1,15 +1,16 @@
// Sing's voice model in memory (Sing.h): loaded, warmed, kept a minute after the mic goes off, and dropped.
//
-// A copy on the CPU alone comes first, since that is the one Core ML has always loaded: once one prediction of
-// silence has warmed it, the separator is Ready and Sing works. Then, while Spotify is in the foreground, a faster
-// copy on the units asked for (the GPU unless the Sing page's Runs on says otherwise) loads beside it, warms the
-// same way and is handed to the separator, which uses it for the windows while Spotify is in the foreground. A
-// second copy loads only when the process has comfortably more memory left than a copy takes.
+// A copy on the CPU alone comes first, as it loads in seconds: once one prediction of silence has warmed it, the
+// separator is Ready and Sing works. Then, while Spotify is in the foreground and the Neural Engine is asked for, a
+// copy of the same model on the CPU and Neural Engine loads beside it, warms the same way and is handed to the
+// separator, which runs the windows on it from then on, in the background too. Its first load compiles the model for
+// the Neural Engine (32-46 s on an iPhone 15 Pro, after every install, as an install moves the app's container; then
+// 0.4 s), so it has a deadline of its own; it loads only when the process has the memory left for it.
//
// Core ML cannot cancel a load, and on an iPhone 15 Pro one has never come back. So each load runs on a queue of its
// own, has a deadline, and is abandoned when the deadline passes or the model is dropped: abandoned, it is left to
// finish or hang where it is, and whatever it brings back later is let go. A CPU copy past its deadline makes Sing
-// Failed until the mic is switched off and on; a faster copy past its deadline leaves Sing on the CPU's.
+// Failed until the mic is switched off and on; a Neural Engine copy past its deadline leaves Sing on the CPU's.
//
// Threading: main thread, but for SGSingLoaderSeparator's separator, which the engine's worker runs.
#import <CoreML/CoreML.h>
@@ -23,35 +24,37 @@ typedef NS_ENUM(NSInteger, SGSingLoaderState) {
SGSingLoaderFailed, // the CPU copy did not load (SGSingLoaderError says why); stays so until a retry
};
-// What became of the faster copy.
+// What became of the Neural Engine copy (the "fast" one).
typedef NS_ENUM(NSInteger, SGSingFastState) {
SGSingFastNone, // none asked for, or not yet started
SGSingFastLoading,
SGSingFastReady,
- SGSingFastSkipped, // too little memory for a second copy
+ SGSingFastSkipped, // too little memory for it
SGSingFastFailed, // Core ML refused it, or its warm-up failed
- SGSingFastTimedOut, // past its deadline; SGSingLoaderFastTimedOutActive says whether Spotify stayed active through it
+ SGSingFastTimedOut, // past its deadline
};
-// Run on the main thread whenever the state, the separator or the faster copy changes.
+// Run on the main thread whenever the state, the separator or the Neural Engine copy changes.
void SGSingLoaderSetChanged(void (^changed)(void));
-// Loads the model at `url` or joins the load already running, with `fast` the units of the faster copy
-// (MLComputeUnitsCPUOnly for none); a different `fast` than before drops the faster copy for the new one. Does
-// nothing while Failed. Call it only while Spotify is in the foreground.
-void SGSingLoaderWant(NSURL *url, MLComputeUnits fast);
+// Loads the model at `url` or joins the load already running, with a Neural Engine copy beside the CPU's when `neural`;
+// without it, any Neural Engine copy is dropped. Another `url` than the last drops the copies of that one (and a
+// Failed) first. Does nothing while Failed. Call it only while Spotify is in the foreground.
+void SGSingLoaderWant(NSURL *url, BOOL neural);
+// The model last wanted, nil before the first want.
+NSURL *SGSingLoaderURL(void);
// The mic is off: the copies are kept for SGSingLoaderKeepSeconds, then dropped unless wanted again by then.
// Failed is cleared, so the next want is a fresh load whatever an abandoned one is still doing.
void SGSingLoaderRelease(void);
// Both copies dropped at once and any load abandoned, `why` logged when there was something to drop.
void SGSingLoaderPurge(NSString *why);
-// The faster copy alone dropped (a memory warning), the CPU's kept.
+// The Neural Engine copy alone dropped, the CPU's kept.
void SGSingLoaderDropFast(NSString *why);
-// Spotify became active or stopped being active: the faster copy is used only while it is, and starts loading
-// when it becomes so.
+// Spotify became active or stopped being active: a load starts only while it is, the Neural Engine copy's when it
+// becomes so.
void SGSingLoaderSetForeground(BOOL foreground);
SGSingLoaderState SGSingLoaderCurrentState(void);
-// Whether the process has the memory left for a copy (always, where the OS does not say).
+// Whether the process has the memory left for a CPU copy (always, where the OS does not say).
BOOL SGSingLoaderHasRoom(void);
// Nil until Ready.
SGSingSeparator *SGSingLoaderSeparator(void);
@@ -59,13 +62,12 @@ NSString *SGSingLoaderError(void);
// Seconds since the CPU copy began loading, while Loading.
NSTimeInterval SGSingLoaderSeconds(void);
SGSingFastState SGSingLoaderFastState(void);
-BOOL SGSingLoaderFastTimedOutActive(void);
-MLComputeUnits SGSingLoaderFastUnits(void);
-// "GPU", "Neural Engine", "GPU and Neural Engine", "CPU".
-NSString *SGSingUnitsName(MLComputeUnits units);
+// Seconds since the Neural Engine copy began loading, while it loads.
+NSTimeInterval SGSingLoaderFastSeconds(void);
// Loads started since launch, and loads not come back yet (abandoned ones with them), for the harness.
unsigned SGSingLoaderAttempts(void);
unsigned SGSingLoaderOutstanding(void);
-// The deadlines and the minute kept, in seconds; changed only by the harness.
-extern double SGSingLoaderCPUDeadline, SGSingLoaderFastDeadline, SGSingLoaderKeepSeconds;
+// The deadlines (the Neural Engine copy's covers its first compile) and the minute kept, in seconds; changed only by the
+// harness.
+extern double SGSingLoaderCPUDeadline, SGSingLoaderNeuralDeadline, SGSingLoaderKeepSeconds;
diff --git a/tweak/Sources/Shared/Sing/SGSingLoader.m b/tweak/Sources/Shared/Sing/SGSingLoader.m
index 5889a462..525b66be 100644
--- a/tweak/Sources/Shared/Sing/SGSingLoader.m
+++ b/tweak/Sources/Shared/Sing/SGSingLoader.m
@@ -6,20 +6,24 @@
#import "SGSingLoader.h"
#import "SGSingSeparator.h"
-// The CPU copy loaded in 0.9-5 s on an M4 and the Neural Engine's in 44.6 s on an iPhone 15 Pro; a GPU load there
-// never came back in 14 min. The CPU copy is what Sing waits on, the faster one runs beside a working Sing.
-double SGSingLoaderCPUDeadline = 120, SGSingLoaderFastDeadline = 180, SGSingLoaderKeepSeconds = 60;
-// A second copy took 1.07-1.15 GB more of the process's footprint on an M4, so it loads only with this much left.
-static const unsigned long long kFastRoom = 1600ull * 1000 * 1000;
+// The CPU copy loaded in 0.9-5 s on an M4. The Neural Engine's first load compiles the model, which Core ML then
+// keeps: 24-63 s on an M4 and 32-46 s on an iPhone 15 Pro, then 0.4-0.7 s; it has ten minutes.
+double SGSingLoaderCPUDeadline = 120, SGSingLoaderNeuralDeadline = 600, SGSingLoaderKeepSeconds = 60;
+// A CPU copy loads beside an abandoned load still out (or again after a memory warning) only with this much left: a
+// second copy took 1.07-1.15 GB more of the process's footprint on an M4.
+static const unsigned long long kCopyRoom = 1600ull * 1000 * 1000;
+// The Neural Engine copy loads beside the CPU's only with this much left. On an iPhone 15 Pro it added 0.06-0.1 GB of
+// footprint (resident 1.06 to 1.13 GB); 0.5 GB is that five times over, for its first compile, which was not measured
+// apart, and for Spotify's own memory to grow without iOS closing it for a copy Sing can do without.
+static const unsigned long long kNeuralRoom = 500ull * 1000 * 1000;
static const int kStillEvery = 30;
// One load of one copy.
@interface SGSingLoad : NSObject
@property (nonatomic) unsigned number;
-@property (nonatomic) MLComputeUnits units;
+@property (nonatomic, copy) NSString *name; // the copy's, for the log
@property (nonatomic) CFAbsoluteTime began;
@property (nonatomic) BOOL finished, abandoned;
-@property (nonatomic) BOOL stayedActive; // Spotify was active from its start to its end or deadline
@end
@implementation SGSingLoad
@@ -33,12 +37,11 @@ static SGSingLoad *sg_cpuLoad, *sg_fastLoad; // the loads that count; abandone
static unsigned sg_outstanding; // loads not come back yet, abandoned ones with them
static unsigned sg_attempts;
static NSURL *sg_url;
-static MLComputeUnits sg_fastUnits = MLComputeUnitsCPUOnly;
+static BOOL sg_neural; // a Neural Engine copy is asked for
static SGSingFastState sg_fastState;
-static BOOL sg_fastTimedOutActive;
static BOOL sg_foreground;
static NSUInteger sg_keepToken; // counts the mic's releases and wants, so a stale minute drops nothing
-static BOOL sg_wanted; // the mic is on: a faster copy is worth loading
+static BOOL sg_wanted; // the mic is on: the Neural Engine copy is worth loading
// Run after the caller returns, so a listener that wants or purges again does not run inside it.
static void changed(void) {
@@ -47,15 +50,6 @@ static void changed(void) {
});
}
-NSString *SGSingUnitsName(MLComputeUnits units) {
- switch (units) {
- case MLComputeUnitsCPUAndGPU: return @"GPU";
- case MLComputeUnitsCPUAndNeuralEngine: return @"Neural Engine";
- case MLComputeUnitsAll: return @"GPU and Neural Engine";
- default: return @"CPU";
- }
-}
-
// The memory the process has left before iOS closes it; 0 where the OS does not say (the Mac).
static unsigned long long memoryLeft(void) {
#if TARGET_OS_IPHONE
@@ -70,48 +64,48 @@ static NSString *memoryText(void) {
return left ? [NSString stringWithFormat:@"%.2f GB left to the process", left / 1e9] : @"memory left unknown";
}
-// Whether a copy can load beside what is already in memory or still loading.
-static BOOL roomForAnother(void) {
+// Whether `room` is left for a copy beside what is already in memory or still loading.
+static BOOL roomFor(unsigned long long room) {
unsigned long long left = memoryLeft();
- return !left || left >= kFastRoom;
+ return !left || left >= room;
}
static void still(SGSingLoad *load, int seconds) {
dispatch_after(dispatch_time(DISPATCH_TIME_NOW, kStillEvery * NSEC_PER_SEC), dispatch_get_main_queue(), ^{
if (load.finished || load.abandoned) return;
- SGLog(@"sing: still loading the %@ copy (load %u), %d s", SGSingUnitsName(load.units), load.number, seconds + kStillEvery);
+ SGLog(@"sing: still loading the %@ copy (load %u), %d s", load.name, load.number, seconds + kStillEvery);
still(load, seconds + kStillEvery);
});
}
// Loads and warms a copy on a queue of its own; `done` runs on the main thread unless the load was abandoned
// first, and `late` when its deadline passes before it comes back.
-static SGSingLoad *startLoad(NSURL *url, MLComputeUnits units, double deadline, void (^done)(MLModel *model, NSError *error),
- void (^late)(SGSingLoad *load)) {
+static SGSingLoad *startLoad(NSURL *url, BOOL neural, double deadline, void (^done)(MLModel *model, NSError *error), void (^late)(void)) {
SGSingLoad *load = [SGSingLoad new];
load.number = ++sg_attempts;
- load.units = units;
+ load.name = neural ? @"Neural Engine" : @"CPU";
load.began = CFAbsoluteTimeGetCurrent();
- load.stayedActive = sg_foreground;
sg_outstanding++;
- SGLog(@"sing: load %u, the %@ copy, starts (Spotify %@, thermal state %s, %@, %u loads out)", load.number, SGSingUnitsName(units),
- sg_foreground ? @"active" : @"not active", SGSingThermalName(), memoryText(), sg_outstanding);
+ SGLog(@"sing: load %u, the %@ copy of %@, starts (Spotify %@, thermal state %s, %@, %@, %u loads out)", load.number, load.name,
+ url.lastPathComponent, sg_foreground ? @"active" : @"not active", SGSingThermalName(), memoryText(), SGSingMemoryText(), sg_outstanding);
NSString *name = [NSString stringWithFormat:@"spotifyglass.sing.load.%u", load.number];
dispatch_queue_t queue = dispatch_queue_create(name.UTF8String, dispatch_queue_attr_make_with_qos_class(DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INITIATED, 0));
dispatch_async(queue, ^{
MLModelConfiguration *configuration = [MLModelConfiguration new];
- configuration.computeUnits = units;
+ configuration.computeUnits = neural ? MLComputeUnitsCPUAndNeuralEngine : MLComputeUnitsCPUOnly;
NSError *error;
MLModel *model = [MLModel modelWithContentsOfURL:url configuration:configuration error:&error];
double loaded = CFAbsoluteTimeGetCurrent() - load.began;
+ NSString *loadedMemory = SGSingMemoryText();
double warm = model ? [SGSingSeparator warmUp:model error:&error] : -1;
+ NSString *warmMemory = SGSingMemoryText();
dispatch_async(dispatch_get_main_queue(), ^{
sg_outstanding--;
load.finished = YES;
NSString *outcome = !model ? [NSString stringWithFormat:@"did not load after %.1f s: %@", loaded, error.localizedDescription]
: warm < 0 ? [NSString stringWithFormat:@"loaded in %.1f s, and its warm-up failed: %@", loaded, error.localizedDescription]
: [NSString stringWithFormat:@"loaded in %.1f s, and its warm-up window of silence took %.1f s", loaded, warm];
- SGLog(@"sing: load %u, the %@ copy, %@%@", load.number, SGSingUnitsName(units), outcome,
+ SGLog(@"sing: load %u, the %@ copy, %@ (memory loaded: %@; warm: %@)%@", load.number, load.name, outcome, loadedMemory, warmMemory,
load.abandoned ? @"; it was abandoned before, so it is let go" : @"");
if (load.abandoned) return;
done(warm >= 0 ? model : nil, error);
@@ -121,39 +115,36 @@ static SGSingLoad *startLoad(NSURL *url, MLComputeUnits units, double deadline,
dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(deadline * NSEC_PER_SEC)), dispatch_get_main_queue(), ^{
if (load.finished || load.abandoned) return;
load.abandoned = YES;
- SGLog(@"sing: load %u, the %@ copy, has not come back in %.0f s (Spotify %@ throughout): abandoned", load.number, SGSingUnitsName(units),
- deadline, load.stayedActive ? @"active" : @"not always active");
- late(load);
+ SGLog(@"sing: load %u, the %@ copy, has not come back in %.0f s: abandoned", load.number, load.name, deadline);
+ late();
});
return load;
}
static void startFast(void) {
- if (!sg_wanted || sg_state != SGSingLoaderReady || !sg_foreground || sg_fastUnits == MLComputeUnitsCPUOnly || sg_fastState != SGSingFastNone) return;
- if (!roomForAnother()) {
+ if (!sg_wanted || sg_state != SGSingLoaderReady || !sg_foreground || !sg_neural || sg_fastState != SGSingFastNone) return;
+ if (!roomFor(kNeuralRoom)) {
sg_fastState = SGSingFastSkipped;
- SGLog(@"sing: no %@ copy: %@, a copy wants %.1f GB, so Sing stays on the CPU", SGSingUnitsName(sg_fastUnits), memoryText(), kFastRoom / 1e9);
+ SGLog(@"sing: no Neural Engine copy: %@, it wants %.1f GB, so Sing stays on the CPU", memoryText(), kNeuralRoom / 1e9);
changed();
return;
}
sg_fastState = SGSingFastLoading;
- sg_fastTimedOutActive = NO;
- sg_fastLoad = startLoad(sg_url, sg_fastUnits, SGSingLoaderFastDeadline, ^(MLModel *model, NSError *error) {
+ sg_fastLoad = startLoad(sg_url, YES, SGSingLoaderNeuralDeadline, ^(MLModel *model, NSError *error) {
sg_fastLoad = nil;
sg_fastState = model ? SGSingFastReady : SGSingFastFailed;
- if (model) [sg_separator setFastModel:model named:SGSingUnitsName(sg_fastUnits)];
+ if (model) [sg_separator setFastModel:model named:@"Neural Engine"];
changed();
- }, ^(SGSingLoad *load) {
+ }, ^{
sg_fastLoad = nil;
sg_fastState = SGSingFastTimedOut;
- sg_fastTimedOutActive = load.stayedActive;
changed();
});
changed();
}
static void startCPU(void) {
- if (sg_outstanding && !roomForAnother()) {
+ if (sg_outstanding && !roomFor(kCopyRoom)) {
sg_state = SGSingLoaderFailed;
sg_error = [NSString stringWithFormat:@"A load abandoned earlier still holds memory, and %@. Close Spotify and open it again to load the voice model.", memoryText()];
SGLog(@"sing: no new load: %@", sg_error);
@@ -162,7 +153,7 @@ static void startCPU(void) {
}
sg_state = SGSingLoaderLoading;
sg_error = nil;
- sg_cpuLoad = startLoad(sg_url, MLComputeUnitsCPUOnly, SGSingLoaderCPUDeadline, ^(MLModel *model, NSError *error) {
+ sg_cpuLoad = startLoad(sg_url, NO, SGSingLoaderCPUDeadline, ^(MLModel *model, NSError *error) {
sg_cpuLoad = nil;
sg_separator = model ? [[SGSingSeparator alloc] initWithModel:model] : nil;
if (sg_separator) {
@@ -174,7 +165,7 @@ static void startCPU(void) {
}
changed();
startFast();
- }, ^(SGSingLoad *load) {
+ }, ^{
sg_cpuLoad = nil;
sg_state = SGSingLoaderFailed;
sg_error = [NSString stringWithFormat:@"The voice model did not load in %.0f s, so that load was given up.", SGSingLoaderCPUDeadline];
@@ -197,20 +188,23 @@ void SGSingLoaderDropFast(NSString *why) {
sg_fastLoad = nil;
[sg_separator setFastModel:nil named:nil];
sg_fastState = SGSingFastNone;
- sg_fastTimedOutActive = NO;
if (!had) return;
- if (why) SGLog(@"sing: the %@ copy is dropped: %@", SGSingUnitsName(sg_fastUnits), why);
+ if (why) SGLog(@"sing: the Neural Engine copy is dropped: %@", why);
changed();
}
-void SGSingLoaderWant(NSURL *url, MLComputeUnits fast) {
+void SGSingLoaderWant(NSURL *url, BOOL neural) {
+ // Another model (its update in): the copies of the one before go, a failure with them, and this one loads.
+ if (sg_url && url && ![url isEqual:sg_url]) {
+ SGSingLoaderPurge([NSString stringWithFormat:@"the model is now %@", url.lastPathComponent]);
+ sg_state = SGSingLoaderIdle;
+ sg_error = nil;
+ }
sg_keepToken++;
sg_wanted = YES;
sg_url = url;
- if (fast != sg_fastUnits) {
- SGSingLoaderDropFast(@"Runs on changed");
- sg_fastUnits = fast;
- }
+ sg_neural = neural;
+ if (!neural) SGSingLoaderDropFast(@"Karaoke runs on the CPU alone now");
if (sg_state == SGSingLoaderIdle) startCPU();
else startFast();
}
@@ -243,13 +237,15 @@ void SGSingLoaderRelease(void) {
void SGSingLoaderSetForeground(BOOL foreground) {
sg_foreground = foreground;
- SGSingSeparatorSetForeground(foreground);
- if (!foreground) sg_fastLoad.stayedActive = NO;
- else startFast();
+ if (foreground) startFast();
}
BOOL SGSingLoaderHasRoom(void) {
- return roomForAnother();
+ return roomFor(kCopyRoom);
+}
+
+NSURL *SGSingLoaderURL(void) {
+ return sg_url;
}
SGSingLoaderState SGSingLoaderCurrentState(void) {
@@ -272,12 +268,8 @@ SGSingFastState SGSingLoaderFastState(void) {
return sg_fastState;
}
-BOOL SGSingLoaderFastTimedOutActive(void) {
- return sg_fastTimedOutActive;
-}
-
-MLComputeUnits SGSingLoaderFastUnits(void) {
- return sg_fastUnits;
+NSTimeInterval SGSingLoaderFastSeconds(void) {
+ return sg_fastLoad ? CFAbsoluteTimeGetCurrent() - sg_fastLoad.began : 0;
}
unsigned SGSingLoaderAttempts(void) {
diff --git a/tweak/Sources/Shared/Sing/SGSingModel.m b/tweak/Sources/Shared/Sing/SGSingModel.m
index f208cf2d..f8e628c8 100644
--- a/tweak/Sources/Shared/Sing/SGSingModel.m
+++ b/tweak/Sources/Shared/Sing/SGSingModel.m
@@ -1,16 +1,24 @@
-// The voice model on the phone (Sing.h): the five files of its compiled separator.mlmodelc, downloaded one by one
-// from the Hugging Face repo, each kept only once its size and SHA-256 are the ones pinned here, then moved
-// together into Application Support/Vitrine/Sing/separator.mlmodelc, which backups leave out (489 MB that can
-// be downloaded again). A dropped connection carries on from where it stopped, three times per file, and a stopped
-// download keeps the file it was in the middle of (NSURLSession's resume data, kept beside the staging folder)
-// for the next one to carry on from. A resumed request is answered 206, with the whole file handed over all the same.
+// The voice model on the phone (Sing.h): the five files of its compiled separator-ane.mlmodelc, downloaded one by one
+// from the Hugging Face repo's separator-ane.mlmodelc folder, each kept only once its size and SHA-256 are the ones
+// pinned here, then moved together into Application Support/Vitrine/Sing/separator-ane.mlmodelc, which backups leave
+// out (210 MB that can be downloaded again). A dropped connection carries on from where it stopped, three times per
+// file, and a stopped download keeps the file it was in the middle of (NSURLSession's resume data, kept beside the
+// staging folder) for the next one to carry on from. A resumed request is answered 206, with the whole file handed
+// over all the same. On a FLEX build a copy put in Sing/dev/ by hand comes first.
//
-// Threading: main thread; the session's delegate runs on a queue of its own and hands back to the main thread.
+// The model Karaoke downloaded before (separator.mlmodelc, 489 MB, the same checkpoint with operations in float32 that
+// the Neural Engine does not run) is kept and loaded until this one is in, so Karaoke works through the update: it is
+// then the model, on the CPU alone, and this one's download an update (SGSingModelUpdateAvailable). Once this one is in,
+// at launch or as its download ends, the old one is deleted (SGSingRemoveOldModel); what its own download left goes at
+// launch either way.
+//
+// Threading: main thread, but for SGSingRemoveOldModel; the session's delegate runs on a queue of its own and hands
+// back to the main thread.
#import <CommonCrypto/CommonDigest.h>
#import "Core/SGCore.h"
#import "Sing.h"
-static NSString *const kRepo = @"https://huggingface.co/My-Name-Is-Jeff/vitrine-sing/resolve/main/";
+static NSString *const kRepo = @"https://huggingface.co/My-Name-Is-Jeff/vitrine-sing/resolve/main/separator-ane.mlmodelc/";
static const int kRetries = 3;
// Bytes a stopped download has kept, for the row that offers to carry on with it or remove it.
static NSString *const kPausedKey = @"spotifyglass.sing.downloadPaused";
@@ -23,15 +31,24 @@ typedef struct {
const char *sha256;
} SGSingFile;
-// The model card's table (huggingface.co/My-Name-Is-Jeff/vitrine-sing, a mirror of Darkkos/spoti-sing), the weights last so the small files fail first.
+// The repo's separator-ane.mlmodelc (manifest.json beside the compiled folder; upload those exact files, as a
+// recompile changes coremldata.bin), the weights last so the small files fail first.
static const SGSingFile kFiles[] = {
- {@"metadata.json", 2431, "52a8d5e3f09e33236d495dbed5bbce1c75bac6f2a6b6097637cf214f37c1de53"},
- {@"coremldata.bin", 507, "2090acaf7a6df72ec83857cb88d101654023a6baad222a25d0173827d3347e28"},
- {@"analytics/coremldata.bin", 243, "f7ee4ec9b5cc1c97171bd5aad93af61e183aa0db1451ef3b775bd4e77f0b7cfd"},
- {@"model.mil", 669061, "966560ed5125174a98f19b94f5de04450a7112ade0e731f2236c202c0280a623"},
- {@"weights/weight.bin", 488986336, "970a99fb4b15724bf76d2918ceb177df592c69265d3e2fabaab6e5ba72738e62"},
+ {@"metadata.json", 2333, "101714d1a37ad29c70bf1705ec25ec99584c3cbce27f2245858ace06dacbfb44"},
+ {@"coremldata.bin", 388, "bcfd38a8b121681dfb9fdfe6f9d88ffb29068e0dd6d24d4012f4c3a7bdcbc81b"},
+ {@"analytics/coremldata.bin", 243, "55f78aab64f5b1250ce4d6016dff23aee43fb97feb77bbceab4e36b3d9ccffb3"},
+ {@"model.mil", 1346642, "957024c1075fe331f7ca3410b8bf5e2f83a58c61763caf154729829ce0a92d08"},
+ {@"weights/weight.bin", 209077208, "bb4a0effafb5121b9aaa5ea96371fa14d0d8dc30a5ae44a93e0c15a2f4077635"},
};
enum { kFileCount = sizeof kFiles / sizeof kFiles[0] };
+// The old model's, checked by size only: it was checked as it came in.
+static const SGSingFile kOldFiles[] = {
+ {@"metadata.json", 2431, NULL},
+ {@"coremldata.bin", 507, NULL},
+ {@"analytics/coremldata.bin", 243, NULL},
+ {@"model.mil", 669061, NULL},
+ {@"weights/weight.bin", 488986336, NULL},
+};
NSString *const SGSingChangedNotification = @"SGSingChangedNotification";
@@ -50,31 +67,43 @@ static NSURL *singFolder(void) {
}
static NSURL *modelFolder(void) {
+ return [singFolder() URLByAppendingPathComponent:@"separator-ane.mlmodelc" isDirectory:YES];
+}
+
+static NSURL *oldFolder(void) {
return [singFolder() URLByAppendingPathComponent:@"separator.mlmodelc" isDirectory:YES];
}
static NSURL *stagingFolder(void) {
- return [singFolder() URLByAppendingPathComponent:@"download" isDirectory:YES];
+ return [singFolder() URLByAppendingPathComponent:@"download-ane" isDirectory:YES];
}
// A stopped download's resume data for file `index`, beside the staging folder: everything in that folder is moved
// into the model.
static NSURL *resumeFile(int index) {
- return [singFolder() URLByAppendingPathComponent:[NSString stringWithFormat:@"download-%d.resume", index]];
+ return [singFolder() URLByAppendingPathComponent:[NSString stringWithFormat:@"download-ane-%d.resume", index]];
}
static long long sizeOf(NSURL *url) {
return [[NSFileManager.defaultManager attributesOfItemAtPath:url.path error:nil][NSFileSize] longLongValue];
}
-// Whether every file is in `folder` at its size; the hashes were checked as each came in.
-static BOOL complete(NSURL *folder, int upTo) {
- for (int i = 0; i < upTo; i++) {
- if (sizeOf([folder URLByAppendingPathComponent:kFiles[i].path]) != kFiles[i].size) return NO;
+// Whether every file of `files` is in `folder` at its size; the hashes were checked as each came in.
+static BOOL completeWith(NSURL *folder, const SGSingFile *files, int count) {
+ for (int i = 0; i < count; i++) {
+ if (sizeOf([folder URLByAppendingPathComponent:files[i].path]) != files[i].size) return NO;
}
return YES;
}
+static BOOL complete(NSURL *folder) {
+ return completeWith(folder, kFiles, kFileCount);
+}
+
+static BOOL oldComplete(void) {
+ return completeWith(oldFolder(), kOldFiles, sizeof kOldFiles / sizeof kOldFiles[0]);
+}
+
// The file's SHA-256, read a megabyte at a time: the weights are too big to read whole.
static NSString *sha256Of(NSURL *url) {
NSFileHandle *handle = [NSFileHandle fileHandleForReadingFromURL:url error:nil];
@@ -128,7 +157,7 @@ static BOOL sg_checking; // the weights' checksum is being read
NSURLSessionConfiguration *configuration = NSURLSessionConfiguration.defaultSessionConfiguration;
configuration.timeoutIntervalForRequest = 60;
// Offline, a request (a retry too) waits for the network rather than failing at once; and so it does on cellular
- // or a Low Data Mode network, 489 MB being a lot of a plan, until the user says it may use them.
+ // or a Low Data Mode network, 210 MB being a lot of a plan, until the user says it may use them.
configuration.waitsForConnectivity = YES;
configuration.allowsExpensiveNetworkAccess = sg_cellular;
configuration.allowsConstrainedNetworkAccess = sg_cellular;
@@ -208,6 +237,8 @@ static BOOL sg_checking; // the weights' checksum is being read
sg_waitingForNetwork = NO;
sg_lastError = error;
if (!error) [NSUserDefaults.standardUserDefaults removeObjectForKey:kPausedKey];
+ // The update in: the old model goes now, and Sing.x loads this one the next time it wants the model.
+ if (!error) SGSingRemoveOldModel();
sg_modelGeneration++;
announce();
});
@@ -308,23 +339,43 @@ static BOOL sg_checking; // the weights' checksum is being read
#pragma mark - what Sing asks
SGSingModelState SGSingModelCurrentState(void) {
- if (sg_downloader) return SGSingModelDownloading;
+ // The old model works on while its update downloads.
+ if (sg_downloader && !SGSingModelUpdateAvailable()) return SGSingModelDownloading;
return SGSingModelURL() ? SGSingModelReady : SGSingModelMissing;
}
NSURL *SGSingModelURL(void) {
// Checked by size once a launch, and again after a download or a delete changes it.
- static int known = -1;
+ static int known = -1, oldKnown = -1;
+ static NSURL *dev;
static NSUInteger knownGeneration = NSUIntegerMax;
if (knownGeneration != sg_modelGeneration) {
knownGeneration = sg_modelGeneration;
- known = complete(modelFolder(), kFileCount);
+ known = complete(modelFolder());
+ oldKnown = oldComplete();
+ // FLEX builds only (Diagnostics' SGIsDebugBuild, which the harnesses do not link): a copy put in Sing/dev/ by
+ // hand, for a model not on Hugging Face yet, neither downloaded nor checked. Core ML crashes the process on a
+ // compiled model with a file missing (model.mil, weights/weight.bin; a short file fails cleanly), so a copy
+ // cut off halfway is not loaded.
+ NSURL *folder = [[singFolder() URLByAppendingPathComponent:@"dev" isDirectory:YES] URLByAppendingPathComponent:@"separator-ane.mlmodelc" isDirectory:YES];
+ int present = 0;
+ for (int i = 0; i < kFileCount; i++) present += sizeOf([folder URLByAppendingPathComponent:kFiles[i].path]) > 0;
+ BOOL there = present == kFileCount, flex = NSClassFromString(@"FLEXManager") != nil;
+ if (there && flex && !dev) SGLog(@"sing: DEV MODEL: the voice model is the unchecked copy in %@, not a download (a FLEX build)", folder.path);
+ if (there && !flex) SGLog(@"sing: a dev model is in %@, and only a FLEX build loads it", folder.path);
+ if (present && !there) SGLog(@"sing: the dev model in %@ has %d of its %d files, so it is not loaded", folder.path, present, kFileCount);
+ dev = there && flex ? folder : nil;
}
- return known ? modelFolder() : nil;
+ return dev ?: known ? modelFolder() : oldKnown ? oldFolder() : nil;
+}
+
+// This model in place, the dev copy or the download: nothing to download.
+static BOOL hasModel(void) {
+ return SGSingModelURL() && !SGSingModelUpdateAvailable();
}
double SGSingModelProgress(void) {
- return sg_downloader ? sg_progress : SGSingModelURL() ? 1 : 0;
+ return sg_downloader ? sg_progress : hasModel() ? 1 : 0;
}
BOOL SGSingModelWaitingForNetwork(void) {
@@ -349,7 +400,7 @@ static long long bytesToCome(void) {
}
void SGSingDownloadModel(void) {
- if (sg_downloader || SGSingModelURL()) return;
+ if (sg_downloader || hasModel()) return;
// The volume Application Support is on, which the staging folder's parent may not exist on yet.
NSURL *support = singFolder().URLByDeletingLastPathComponent.URLByDeletingLastPathComponent;
NSNumber *free = [support resourceValuesForKeys:@[NSURLVolumeAvailableCapacityForImportantUsageKey] error:nil][NSURLVolumeAvailableCapacityForImportantUsageKey];
@@ -393,7 +444,7 @@ BOOL SGSingModelChecking(void) {
}
long long SGSingModelPausedBytes(void) {
- return sg_downloader || SGSingModelURL() ? 0 : [NSUserDefaults.standardUserDefaults integerForKey:kPausedKey];
+ return sg_downloader || hasModel() ? 0 : [NSUserDefaults.standardUserDefaults integerForKey:kPausedKey];
}
void SGSingCancelModelDownload(void) {
@@ -412,8 +463,55 @@ void SGSingDeleteModel(void) {
announce();
}
+// Deletes what of `urls` is there, adding the bytes freed to `freed`; whether there was any.
+static BOOL removeAll(NSArray<NSURL *> *urls, long long *freed) {
+ NSFileManager *files = NSFileManager.defaultManager;
+ BOOL found = NO;
+ for (NSURL *url in urls) {
+ if (![files fileExistsAtPath:url.path]) continue;
+ found = YES;
+ *freed += sizeOf(url);
+ for (NSURL *file in [files enumeratorAtURL:url includingPropertiesForKeys:@[NSURLFileSizeKey] options:0 errorHandler:nil]) *freed += sizeOf(file);
+ [files removeItemAtURL:url error:nil];
+ }
+ return found;
+}
+
+void SGSingRemoveOldModel(void) {
+ // What the old model's download left: its staging folder and resume data, one per file (named one by one:
+ // download-ane-N.resume is this model's own). The bytes a stopped download kept were the old model's.
+ NSMutableArray<NSURL *> *leftovers = [NSMutableArray arrayWithObject:[singFolder() URLByAppendingPathComponent:@"download"]];
+ for (int i = 0; i < 5; i++) [leftovers addObject:[singFolder() URLByAppendingPathComponent:[NSString stringWithFormat:@"download-%d.resume", i]]];
+ long long freed = 0;
+ BOOL left = removeAll(leftovers, &freed);
+ if (left) [NSUserDefaults.standardUserDefaults removeObjectForKey:kPausedKey];
+ // The old model itself only once this one is in: until then Karaoke runs on it.
+ BOOL there = [NSFileManager.defaultManager fileExistsAtPath:oldFolder().path], replaced = complete(modelFolder());
+ BOOL old = there && replaced && removeAll(@[oldFolder()], &freed);
+ NSString *size = [NSByteCountFormatter stringFromByteCount:freed countStyle:NSByteCountFormatterCountStyleFile];
+ if (old) SGLog(@"sing: the old voice model (separator.mlmodelc) is deleted, %@ freed; the model is separator-ane.mlmodelc", size);
+ else if (left) SGLog(@"sing: what the old voice model's download left is deleted, %@ freed", size);
+ if (there && !replaced) {
+ SGLog(@"sing: the old voice model (separator.mlmodelc) is kept until separator-ane.mlmodelc is downloaded, and Karaoke runs on it, on the CPU alone");
+ }
+}
+
+BOOL SGSingModelUpdateAvailable(void) {
+ return [SGSingModelURL() isEqual:oldFolder()];
+}
+
+BOOL SGSingModelUpdateDownloading(void) {
+ return sg_downloader && SGSingModelUpdateAvailable();
+}
+
+NSString *SGSingModelInUseSizeText(void) {
+ long long total = 0;
+ for (size_t i = 0; i < sizeof kOldFiles / sizeof kOldFiles[0]; i++) total += kOldFiles[i].size;
+ return [NSByteCountFormatter stringFromByteCount:SGSingModelUpdateAvailable() ? total : totalBytes() countStyle:NSByteCountFormatterCountStyleFile];
+}
+
NSArray<NSString *> *SGSingComputeUnitNames(void) {
- return @[@"Automatic", @"CPU only", @"GPU", @"Neural Engine", @"GPU and Neural Engine"];
+ return @[@"Automatic", @"CPU only"];
}
BOOL SGSingOSSupported(void) {
@@ -421,8 +519,9 @@ BOOL SGSingOSSupported(void) {
return NO;
}
-// The model is 489 MB of weights, held while Sing is on beside Spotify's own memory: an iPhone with less than
-// 6 GB (which report a little under 6) would have Spotify closed under it.
+// The model is 209 MB of weights, held while Sing is on beside Spotify's own memory (about 1.07 GB resident with its
+// CPU copy warm on an iPhone 15 Pro): an iPhone with less than 6 GB (which report a little under 6) would have Spotify
+// closed under it.
// ponytail: a memory floor stands in for a list of phones; measure on a 6 GB phone and a 4 GB one to place it.
BOOL SGSingDeviceSupported(void) {
return NSProcessInfo.processInfo.physicalMemory >= 5ull * 1000 * 1000 * 1000;
diff --git a/tweak/Sources/Shared/Sing/SGSingSeparator.h b/tweak/Sources/Shared/Sing/SGSingSeparator.h
index 9a953cf4..8df6ec1d 100644
--- a/tweak/Sources/Shared/Sing/SGSingSeparator.h
+++ b/tweak/Sources/Shared/Sing/SGSingSeparator.h
@@ -12,14 +12,13 @@
// The vocals come back from the model's complex mask times the spectrum, through the inverse STFT
// (torch.istft: overlap-add of the windowed frames divided by the window's squared sum).
//
-// Two copies of the model can stand behind one separator: the CPU's, which it is made with, and a faster one
-// (the GPU's or the Neural Engine's) handed over once it is loaded and warm. A window goes to the faster copy only
-// while Spotify is in the foreground (SGSingSeparatorSetForeground), since iOS refuses GPU work from the
-// background; a window the faster copy fails is done again on the CPU's, and the faster copy rests until the app
-// has left the foreground. The first windows on each copy are logged with their time.
+// Two copies of the model can stand behind one separator: the CPU's, which it is made with, and a faster one (the
+// Neural Engine's) handed over once it is loaded and warm, which then takes every window, in the background too. A
+// window the faster copy fails is done again on the CPU's, and the faster copy is not used again until another is
+// handed over (Sing.x drops it for the launch). The first windows on each copy are logged with their time.
//
-// Threading: one separator is used by one thread at a time (its buffers are its own); the faster copy and the
-// foreground can be set from any thread.
+// Threading: one separator is used by one thread at a time (its buffers are its own); the faster copy can be set from
+// any thread.
#import <Foundation/Foundation.h>
@class MLModel;
@@ -36,13 +35,15 @@ enum {
// The thermal state in a word, for the log.
const char *SGSingThermalName(void);
-// Whether Spotify is in the foreground, process-wide; the faster copy is used only then.
-void SGSingSeparatorSetForeground(bool foreground);
+// The process's memory for the log (TASK_VM_INFO): its footprint (phys_footprint, what iOS holds against its limit) and
+// what is resident, the model's weights mapped from their file with it.
+NSString *SGSingMemoryText(void);
typedef struct {
unsigned long long windows[2], fallbacks; // [0] the CPU's copy, [1] the faster one
double averageMS[2];
bool fast; // the last window went to the faster copy
+ bool fastFailed; // the faster copy now in failed a window, and is not used again
} SGSingSeparatorStats;
@interface SGSingSeparator : NSObject
diff --git a/tweak/Sources/Shared/Sing/SGSingSeparator.m b/tweak/Sources/Shared/Sing/SGSingSeparator.m
index e18d94eb..aaeffff7 100644
--- a/tweak/Sources/Shared/Sing/SGSingSeparator.m
+++ b/tweak/Sources/Shared/Sing/SGSingSeparator.m
@@ -1,7 +1,7 @@
#import <Accelerate/Accelerate.h>
#import <CoreML/CoreML.h>
+#import <mach/mach.h>
#import <os/lock.h>
-#import <stdatomic.h>
#import "Core/SGLog.h"
#import "SGSingSeparator.h"
@@ -15,14 +15,6 @@ static NSString *const kInput = @"spectrum", *const kOutput = @"vocals_spectrum"
// This many windows on each copy are logged with their time.
static const unsigned long long kLoggedWindows = 5;
-static atomic_bool sg_foreground;
-static atomic_uint sg_backgrounds; // counts the times the app left the foreground
-
-void SGSingSeparatorSetForeground(bool foreground) {
- if (!foreground && atomic_exchange(&sg_foreground, false)) atomic_fetch_add(&sg_backgrounds, 1);
- atomic_store(&sg_foreground, foreground);
-}
-
const char *SGSingThermalName(void) {
switch (NSProcessInfo.processInfo.thermalState) {
case NSProcessInfoThermalStateNominal: return "nominal";
@@ -33,6 +25,13 @@ const char *SGSingThermalName(void) {
return "?";
}
+NSString *SGSingMemoryText(void) {
+ task_vm_info_data_t info;
+ mach_msg_type_number_t count = TASK_VM_INFO_COUNT;
+ if (task_info(mach_task_self(), TASK_VM_INFO, (task_info_t)&info, &count) != KERN_SUCCESS) return @"footprint unknown";
+ return [NSString stringWithFormat:@"footprint %.2f GB, resident %.2f GB", info.phys_footprint / 1e9, info.resident_size / 1e9];
+}
+
// Where bin `bin` of channel `channel` at STFT frame `frame` starts in the model's layout (its real part;
// the imaginary part follows it).
static inline size_t at(int bin, int channel, int frame) {
@@ -44,8 +43,6 @@ static inline size_t at(int bin, int channel, int frame) {
os_unfair_lock _lock; // the faster copy and its name
MLModel *_fast;
NSString *_fastName;
- bool _fastResting; // failed a window: rests until the app has left the foreground
- unsigned _restingFrom; // sg_backgrounds when it failed
SGSingSeparatorStats _stats; // the worker's; read whole under the lock
MLMultiArray *_input;
vDSP_DFT_Setup _forward, _inverse;
@@ -191,7 +188,7 @@ static BOOL readOutput(MLMultiArray *array, float *into) {
os_unfair_lock_lock(&_lock);
_fast = model;
_fastName = name;
- _fastResting = false;
+ _stats.fastFailed = false;
_stats.windows[1] = 0;
_stats.averageMS[1] = 0;
os_unfair_lock_unlock(&_lock);
@@ -249,6 +246,7 @@ static BOOL predict(MLModel *model, MLDictionaryFeatureProvider *features, float
_stats.fast = copy == 1;
os_unfair_lock_unlock(&_lock);
if (n <= kLoggedWindows) SGLog(@"sing: window %llu on the %@ copy took %.0f ms (1500 ms keeps up), thermal state %s", n, name, ms, SGSingThermalName());
+ if (n == kLoggedWindows) SGLog(@"sing: memory after %llu windows on the %@ copy: %@", n, name, SGSingMemoryText());
}
- (BOOL)separateLeft:(const float *)left right:(const float *)right vocalsLeft:(float *)vocalsLeft
@@ -257,11 +255,8 @@ static BOOL predict(MLModel *model, MLDictionaryFeatureProvider *features, float
[self analyzeLeft:left right:right into:_spectrum];
MLDictionaryFeatureProvider *features = [[MLDictionaryFeatureProvider alloc] initWithDictionary:@{kInput: _input} error:error];
if (!features) return NO;
- bool foreground = atomic_load(&sg_foreground);
- unsigned backgrounds = atomic_load(&sg_backgrounds);
os_unfair_lock_lock(&_lock);
- if (_fastResting && backgrounds != _restingFrom) _fastResting = false;
- MLModel *fast = foreground && !_fastResting ? _fast : nil;
+ MLModel *fast = _stats.fastFailed ? nil : _fast;
NSString *fastName = _fastName;
os_unfair_lock_unlock(&_lock);
if (fast) {
@@ -273,13 +268,10 @@ static BOOL predict(MLModel *model, MLDictionaryFeatureProvider *features, float
return YES;
}
os_unfair_lock_lock(&_lock);
- if (_fast == fast) {
- _fastResting = true;
- _restingFrom = backgrounds;
- }
+ if (_fast == fast) _stats.fastFailed = true;
_stats.fallbacks++;
os_unfair_lock_unlock(&_lock);
- SGLog(@"sing: the %@ copy failed a window (%@), so it is done on the CPU's, and the %@ copy rests until Spotify has been in the background",
+ SGLog(@"sing: the %@ copy failed a window (%@), so it is done on the CPU's, and the %@ copy is not used again",
fastName, fastError.localizedDescription, fastName);
}
CFAbsoluteTime began = CFAbsoluteTimeGetCurrent();
diff --git a/tweak/Sources/Shared/Sing/Sing.h b/tweak/Sources/Shared/Sing/Sing.h
index 48ef26f0..c506c3ff 100644
--- a/tweak/Sources/Shared/Sing/Sing.h
+++ b/tweak/Sources/Shared/Sing/Sing.h
@@ -3,8 +3,8 @@
// under either look: the work is on Spotify's sound.
//
// SGSingModel.m the voice model: downloaded, each file checked against its size and SHA-256, loaded
-// SGSingLoader.m the model in memory: a CPU copy, then a faster one beside it, each load with a deadline
-// SGSingSeparator.m the STFT around the model: two seconds in, their vocals out, on the copy that fits
+// SGSingLoader.m the model in memory: a CPU copy, then a Neural Engine one beside it, each load with a deadline
+// SGSingSeparator.m the STFT around the model: two seconds in, their vocals out, on the faster copy in
// SGSingEngine.m Spotify's mixer pulled ahead of what plays, the vocals separated in the lead, mixed down
// and put where spatial voice holds them
// Sing.x the engine put between Spotify's mixer and its output (through Shared/Player/SpeedPitch.x's
@@ -17,8 +17,11 @@
//
// The voice model is public and MIT licensed: Mel-Band RoFormer (Ju-Chiang Wang, Wei-Tsung Lu, Minz Won) with
// KimberleyJensen's vocal checkpoint, its spectral core exported for Core ML with two-second windows, from
-// https://huggingface.co/My-Name-Is-Jeff/vitrine-sing, a mirror of Darkkos/spoti-sing (App/About/Licenses.m carries
-// its notice). Its floor is iOS 18, the Core ML it was built with. No audio leaves the phone.
+// https://huggingface.co/My-Name-Is-Jeff/vitrine-sing's separator-ane.mlmodelc (App/About/Licenses.m carries its
+// notice): Vitrine's export of that checkpoint for the Neural Engine (its layout (B, C, 1, S), 1x1 convolutions,
+// attention per head, an RMSNorm that divides by the row's largest value first so half precision does not overflow),
+// palettized to 6 bits. Every operation runs on the Neural Engine, and the CPU runs it too, more slowly. Its floor is
+// iOS 18, the Core ML it was built with. No audio leaves the phone.
//
// Threading: main thread, but for what Sing.x's render side and the engine say.
#import <UIKit/UIKit.h>
@@ -31,8 +34,9 @@
#define SGKeySingLevel @"spotifyglass.sing.level"
// Keeps the model running on a hot iPhone, which otherwise lets it go from the thermal state Serious up.
#define SGKeySingIgnoreHeat @"spotifyglass.sing.ignoreHeat"
-// Where the model runs beside the CPU: an index into SGSingComputeUnitNames(), Automatic (0) unless chosen. The
-// earlier key held the one place the model ran (GPU, GPU and Neural Engine, Neural Engine) and is carried over.
+// Where the model runs: an index into SGSingComputeUnitNames(), Automatic (0: a Neural Engine copy beside the CPU's,
+// where the iPhone has a Neural Engine) unless CPU only (1) is chosen. The GPU, Neural Engine and GPU and Neural Engine
+// stored before (2-4), and the earlier key's every value, are Automatic now, carried over at launch.
#define SGKeySingComputeUnits @"spotifyglass.sing.runsOn"
#define SGKeySingComputeUnitsBefore @"spotifyglass.sing.computeUnits"
// Spatial voice: through headphones that track the head, the vocals stay in front as it turns; off until
@@ -97,7 +101,7 @@ void SGSetSingSpatial(BOOL on);
@end
// Spatial voice's page, its preview and its switch: under Sing's page, and on Mod Settings' main page under Sing.
UIViewController *SGSpatialVoiceSettingsPage(void);
-// Loads the faster copy again on the compute units now stored, the CPU's kept.
+// Runs on changed: the Neural Engine copy loaded or dropped as it now says, the CPU's kept.
void SGSingComputeUnitsChanged(void);
// What the main page's Sing row reads out: On, Off, or how far the model has come.
NSString *SGSingSummary(void);
@@ -107,8 +111,8 @@ UIViewController *SGSingSettingsPage(void);
typedef NS_ENUM(NSInteger, SGSingModelState) {
SGSingModelMissing,
- SGSingModelDownloading,
- SGSingModelReady, // on the phone, every file checked
+ SGSingModelDownloading, // with no model to run meanwhile (the old one runs on through its update's download)
+ SGSingModelReady, // on the phone, every file checked: this model, or the old one until its update is in
};
SGSingModelState SGSingModelCurrentState(void);
// 0 to 1 while downloading.
@@ -130,8 +134,19 @@ NSString *SGSingModelSizeText(void);
void SGSingDownloadModel(void);
void SGSingCancelModelDownload(void);
void SGSingDeleteModel(void);
-// The compiled model's folder, nil until it is ready.
+// The compiled model's folder: separator-ane.mlmodelc once it is downloaded and checked, else the old model
+// (separator.mlmodelc) where it is still on the iPhone, else nil. On a FLEX build a copy put in Application
+// Support/Vitrine/Sing/dev/ by hand comes first, unchecked.
NSURL *SGSingModelURL(void);
+// The model is the old one: it runs on the CPU alone, and the download is its update.
+BOOL SGSingModelUpdateAvailable(void);
+// That update downloading (SGSingModelProgress and the rest say how far).
+BOOL SGSingModelUpdateDownloading(void);
+// The size of the model on the iPhone now, for the row that removes it.
+NSString *SGSingModelInUseSizeText(void);
+// Deletes what the old model's download left, and the old model (separator.mlmodelc) itself once separator-ane.mlmodelc
+// is in, logging the space freed; from any thread, at launch and as the download ends.
+void SGSingRemoveOldModel(void);
NSArray<NSString *> *SGSingComputeUnitNames(void);
// Whether this OS can load the model (iOS 18) and this iPhone was built to run it.
BOOL SGSingOSSupported(void);
diff --git a/tweak/Sources/Shared/Sing/Sing.x b/tweak/Sources/Shared/Sing/Sing.x
index cfe37415..6fb46beb 100644
--- a/tweak/Sources/Shared/Sing/Sing.x
+++ b/tweak/Sources/Shared/Sing/Sing.x
@@ -24,11 +24,12 @@
// before the last one ended: something new was played.
//
// The model is loaded by SGSingLoader.m, only while Spotify is active and the mic is on, and kept a minute after the
-// mic goes off. From the thermal state Serious up the engine is held, plays the song as it is and lets the model go,
+// mic goes off: a CPU copy, then on Automatic a Neural Engine copy beside it, which runs the windows once it is in. From the thermal state Serious up the engine is held, plays the song as it is and lets the model go,
// unless Ignore heat warnings is on; it loads again once the iPhone is back at Fair. The model is what heats it, so
// Sing does not keep a hot iPhone hot. A model that falls 8 s behind stops Sing for the rest of the song, which then
// plays as it is; Sing tries again with the next track, when the iPhone cools or when Runs on changes, and stays
-// stopped after three in a row with no song kept up.
+// stopped after three in a row with no song kept up. While the Neural Engine copy loads (its first compile among
+// them), the CPU's falling behind counts for none of that: the song plays with what vocals are in.
// With the vocals as sung, Spatial voice off and the Sing page's lines not on screen, what Sing plays is the song
// itself, so it rests: the engine is held as for the heat and the model kept a minute as for a mic switched off.
// Whenever Sing does not separate (stopped, held, resting, standing aside), the engine plays the sound it holds ahead
@@ -169,12 +170,16 @@ static void announce(void) {
[NSNotificationCenter.defaultCenter postNotificationName:SGSingChangedNotification object:nil];
}
-// Runs on: Automatic, CPU only, GPU, Neural Engine, GPU and Neural Engine. Automatic is the GPU beside the CPU: on an
-// M4 the GPU's copy runs a window in 405-525 ms against the CPU's 1.2 s, while the Neural Engine's copy, which
-// leaves the model's float32 operations to the CPU, took 6.25 s. On an iPhone 15 Pro the GPU's load never came
-// back from a background launch; it now loads only in the foreground, and a GPU load that runs past its deadline
-// with Spotify active throughout keeps Automatic on the CPU alone for this iOS, until Runs on is changed.
-static NSString *const kGPUStuckOn = @"spotifyglass.sing.gpuStuckOn";
+// Runs on: Automatic or CPU only. Automatic is a Neural Engine copy beside the CPU's, on an iPhone with a Neural Engine
+// (A12 on): on an iPhone 15 Pro it takes 231-325 ms a window there, against 632-723 ms on the CPU (which keeps up,
+// warmer). Once that copy has failed to load, warm up or run a window in this launch, Automatic is the CPU's alone
+// until Spotify is opened again or Runs on changes (falling behind is not a failure: that is the budget's, as on any
+// copy). A copy past its deadline is not remembered as failed: Core ML carries on compiling a load given up on and
+// keeps the result, so the next launch may load it at once.
+// The iOS and the folder the Neural Engine copy last loaded on and from: Core ML has kept its compiled form, so it
+// loads in seconds.
+static NSString *const kNeuralReadyOn = @"spotifyglass.sing.neuralReadyOn";
+static NSString *sg_neuralFailed; // why the Neural Engine copy is not used for the rest of this launch
static NSString *osBuild(void) {
return NSProcessInfo.processInfo.operatingSystemVersionString;
@@ -184,30 +189,76 @@ static NSInteger runsOn(void) {
return SGInt(SGKeySingComputeUnits, 0);
}
-static BOOL gpuStuck(void) {
- return [[NSUserDefaults.standardUserDefaults stringForKey:kGPUStuckOn] isEqualToString:osBuild()];
+static NSString *runsOnName(void) {
+ return SGSingComputeUnitNames()[runsOn() == 1];
}
-static MLComputeUnits fastUnits(void) {
- switch (runsOn()) {
- case 1: return MLComputeUnitsCPUOnly;
- case 2: return MLComputeUnitsCPUAndGPU;
- case 3: return MLComputeUnitsCPUAndNeuralEngine;
- case 4: return MLComputeUnitsAll;
- default: return gpuStuck() ? MLComputeUnitsCPUOnly : MLComputeUnitsCPUAndGPU;
+static BOOL hasNeuralEngine(void) {
+ static int has = -1;
+ if (has < 0) {
+ has = 0;
+ if (@available(iOS 17.0, *)) {
+ for (id<MLComputeDeviceProtocol> device in MLAllComputeDevices()) {
+ if ([device isKindOfClass:MLNeuralEngineComputeDevice.class]) has = 1;
+ }
+ }
}
+ return has;
+}
+
+// Runs on is Automatic, the iPhone has a Neural Engine, its copy has not failed in this launch, and the model is not the
+// old one, which the Neural Engine does not run.
+static BOOL neuralWanted(void) {
+ return runsOn() != 1 && hasNeuralEngine() && !sg_neuralFailed && !SGSingModelUpdateAvailable();
+}
+
+// What Core ML keeps the compiled model for: this iOS and the model's folder.
+static NSString *neuralCompiled(void) {
+ return [NSString stringWithFormat:@"%@ %@", osBuild(), SGSingModelURL().path];
+}
+
+static void wantModel(void) {
+ SGSingLoaderWant(SGSingModelURL(), neuralWanted());
}
-// The loader's news: the separator into the engine, a GPU load stuck in the foreground remembered.
+// The Neural Engine copy given up on for the rest of the launch: the CPU's carries on alone.
+static void apply(void);
+static void neuralFailed(NSString *why) {
+ if (sg_neuralFailed) return;
+ sg_neuralFailed = why;
+ SGLog(@"sing: the Neural Engine copy %@, so Karaoke runs on the CPU copy for the rest of this launch", why);
+ if (SGSingOn()) apply();
+}
+
+// The first load on this iOS, from this folder, compiles the model for the Neural Engine.
+static BOOL preparingNeural(void) {
+ return SGSingLoaderFastState() == SGSingFastLoading
+ && ![[NSUserDefaults.standardUserDefaults stringForKey:kNeuralReadyOn] isEqualToString:neuralCompiled()];
+}
+
+// While the Neural Engine copy loads or warms up, the CPU's falling behind is "not yet", not "cannot": the engine's
+// budget is held, so no song is given up and no strike counted, until the copy is in or has failed.
+static void holdBudget(SGSingEngine *engine) {
+ static BOOL held;
+ BOOL hold = SGSingLoaderFastState() == SGSingFastLoading;
+ if (engine) SGSingEngineHoldBudget(engine, hold);
+ if (hold == held) return;
+ held = hold;
+ SGLog(@"sing: %@", hold ? @"falling behind spends none of the 8 s while the Neural Engine copy loads" : @"falling behind counts again");
+}
+
+// The loader's news: the separator into the engine, the Neural Engine copy loaded (remembered) or failed (the CPU's
+// alone from then on).
static void loaderChanged(void) {
SGSingEngine *engine = atomic_load(&sg_engine);
if (engine) SGSingEngineSetSeparator(engine, SGSingLoaderSeparator());
- if (SGSingLoaderFastState() == SGSingFastTimedOut && SGSingLoaderFastTimedOutActive() && runsOn() == 0
- && SGSingLoaderFastUnits() == MLComputeUnitsCPUAndGPU && !gpuStuck()
- && NSProcessInfo.processInfo.thermalState < NSProcessInfoThermalStateSerious) {
- [NSUserDefaults.standardUserDefaults setObject:osBuild() forKey:kGPUStuckOn];
- SGLog(@"sing: the GPU did not load on %@ with Spotify active, so Automatic stays on the CPU on this iOS", osBuild());
+ switch (SGSingLoaderFastState()) {
+ case SGSingFastReady: [NSUserDefaults.standardUserDefaults setObject:neuralCompiled() forKey:kNeuralReadyOn]; break;
+ case SGSingFastFailed: neuralFailed(@"did not load"); break;
+ case SGSingFastTimedOut: neuralFailed([NSString stringWithFormat:@"did not load in %.0f minutes", SGSingLoaderNeuralDeadline / 60]); break;
+ default: break;
}
+ holdBudget(engine);
announce();
}
@@ -272,7 +323,8 @@ static void watch(void);
// Resting: nothing needs the model, so the engine is held as for the heat and the song plays straight. The vocals are
// as sung (which plays exactly the song), Spatial voice is off, and no page traces the lines: the Sing page's card
-// reads them ten times a second while it shows (SGSingReadLevels), so a second without a read is the page gone.
+// reads them thirty times a second while it shows with Spotify in front (SGSingReadLevels), so a second without a
+// read is the page gone (popped, covered or in the background). The lyrics page reads none: it is not a viewer.
static const CFTimeInterval kLinesGoneAfter = 1;
static CFAbsoluteTime sg_linesReadAt;
static BOOL sg_resting;
@@ -313,9 +365,10 @@ static void apply(void) {
else if (!on) SGSingLoaderRelease();
else if (held) SGSingLoaderPurge(@"the iPhone is too hot (thermal state serious or above)");
else if (rest) { if (toRest) SGSingLoaderRelease(); }
- else if (sg_active) SGSingLoaderWant(SGSingModelURL(), fastUnits());
+ else if (sg_active) wantModel();
if (!engine) return;
SGSingEngineSetSeparator(engine, SGSingLoaderSeparator());
+ holdBudget(engine);
SGSingEngineSetPaused(engine, held || rest);
SGSingEngineSetOn(engine, on);
double lead = SGSingEngineLead(engine);
@@ -384,9 +437,9 @@ void SGSetSingLevel(float level) {
}
void SGSingComputeUnitsChanged(void) {
- // A choice made is a new try, the GPU's included.
- [NSUserDefaults.standardUserDefaults removeObjectForKey:kGPUStuckOn];
- SGLog(@"sing: Runs on is now %@", SGSingComputeUnitNames()[(NSUInteger)MIN(MAX(runsOn(), 0), 4)]);
+ // A choice made is a new try, the Neural Engine's included.
+ sg_neuralFailed = nil;
+ SGLog(@"sing: Runs on is now %@", runsOnName());
// Somewhere else to run is a fresh try, however often it fell behind before.
sg_giveUps = 0;
if (sg_stopKind == SGSingStopForGood) sg_stopKind = SGSingStopBehind;
@@ -448,7 +501,9 @@ NSString *SGSingStatusText(void) {
case SGSingStateWaiting: return sg_resting ? @"As sung" : @"Ready";
case SGSingStateBuffering: return @"Listening ahead";
case SGSingStateSinging: return @"On";
- case SGSingStateBehind: return @"Too slow";
+ // The Neural Engine copy on its way: the budget is held (holdBudget), so this is not yet Too slow.
+ case SGSingStateBehind:
+ return SGSingLoaderFastState() == SGSingFastLoading ? @"Preparing for the Neural Engine" : @"Too slow";
case SGSingStateHot: return @"Held, too hot";
case SGSingStateFailed:
if (!sg_stopped && sg_refused) return [sg_refused containsString:@"AirPlay"] ? @"Off over AirPlay" : @"Songs only";
@@ -459,19 +514,27 @@ NSString *SGSingStatusText(void) {
// Where the model runs now, for the details of a working Sing.
static NSString *runsOnNote(void) {
- MLComputeUnits units = SGSingLoaderFastUnits();
- NSString *name = SGSingUnitsName(units);
- if (units == MLComputeUnitsCPUOnly) {
- return runsOn() == 0 ? @"The GPU did not load on this iOS before, so Karaoke runs on the CPU alone. Choosing GPU under Runs on tries it again."
- : @"Karaoke runs on the CPU alone, as Runs on says.";
+ if (SGSingModelUpdateAvailable()) {
+ return [NSString stringWithFormat:@"Karaoke runs on the CPU with the voice model it had before. Its update (%@, on the Voice model row) runs "
+ @"on the Neural Engine, faster and cooler.", SGSingModelSizeText()];
+ }
+ if (runsOn() == 1) return @"Karaoke runs on the CPU alone, as Runs on says.";
+ if (!hasNeuralEngine()) return @"This iPhone has no Neural Engine Karaoke can use, so it runs on the CPU.";
+ if (sg_neuralFailed) {
+ return [NSString stringWithFormat:@"The Neural Engine copy of the voice model %@, so Karaoke runs on the CPU until Spotify is opened again.", sg_neuralFailed];
+ }
+ if (preparingNeural()) {
+ return [NSString stringWithFormat:@"Preparing for the Neural Engine, first time only (%.0f s so far). Karaoke runs on the CPU until then.",
+ SGSingLoaderFastSeconds()];
}
switch (SGSingLoaderFastState()) {
- case SGSingFastNone: return [NSString stringWithFormat:@"Karaoke runs on the CPU; a copy for the %@ loads once Spotify is open.", name];
- case SGSingFastLoading: return [NSString stringWithFormat:@"Karaoke runs on the CPU while a copy for the %@ loads.", name];
- case SGSingFastReady: return [NSString stringWithFormat:@"Karaoke runs on the %@ while Spotify is open, and on the CPU in the background.", name];
- case SGSingFastSkipped: return [NSString stringWithFormat:@"Too little memory is left for a copy on the %@, so Karaoke runs on the CPU alone.", name];
- case SGSingFastFailed: return [NSString stringWithFormat:@"The %@ could not load the voice model, so Karaoke runs on the CPU alone.", name];
- case SGSingFastTimedOut: return [NSString stringWithFormat:@"The %@ did not load the voice model in %.0f minutes, so Karaoke runs on the CPU alone.", name, SGSingLoaderFastDeadline / 60];
+ case SGSingFastNone: return @"Karaoke runs on the CPU; the Neural Engine copy loads once Spotify is open.";
+ case SGSingFastLoading: return @"Karaoke runs on the CPU while the Neural Engine copy loads.";
+ case SGSingFastReady: return @"Karaoke runs on the Neural Engine.";
+ case SGSingFastSkipped: return @"Too little memory is left for the Neural Engine copy, so Karaoke runs on the CPU alone.";
+ case SGSingFastFailed: return @"The Neural Engine could not load the voice model, so Karaoke runs on the CPU alone.";
+ case SGSingFastTimedOut: return [NSString stringWithFormat:@"The Neural Engine did not load the voice model in %.0f minutes, so Karaoke runs on the CPU alone.",
+ SGSingLoaderNeuralDeadline / 60];
}
return nil;
}
@@ -706,6 +769,10 @@ static void tick(void) {
SGSingEngine *engine = atomic_load(&sg_engine);
logClock(engine);
if (state == SGSingStateSinging) sg_sangThisTrack = YES;
+ // A Neural Engine copy that failed a window (Core ML's error) is given up on for the launch. One that is only slow
+ // stays: falling behind is the budget's, as on any copy.
+ SGSingSeparator *current = SGSingLoaderSeparator();
+ if (current && [current stats].fastFailed) neuralFailed(@"failed a window");
if (engine && SGSingOn() && !sg_stopped && SGSingEngineGaveUp(engine)) {
BOOL forGood = ++sg_giveUps >= kGiveUpsKept;
sg_stopKind = forGood ? SGSingStopForGood : SGSingStopBehind;
@@ -723,17 +790,16 @@ static void tick(void) {
SGSingEngineStats stats = SGSingEngineReadStats(engine);
SGSingSeparator *separator = SGSingLoaderSeparator();
SGSingSeparatorStats copies = separator ? [separator stats] : (SGSingSeparatorStats){0};
- MLComputeUnits fast = SGSingLoaderFastUnits();
- NSString *fastPart = fast == MLComputeUnitsCPUOnly ? @"no faster copy"
- : [NSString stringWithFormat:@"%@ copy %@, %llu at %.0f ms", SGSingUnitsName(fast),
+ NSString *fastPart = !neuralWanted() && SGSingLoaderFastState() == SGSingFastNone ? @"no Neural Engine copy"
+ : [NSString stringWithFormat:@"Neural Engine copy %@, %llu at %.0f ms",
@[@"not started", @"loading", @"ready", @"skipped", @"failed", @"timed out"][(NSUInteger)SGSingLoaderFastState()], copies.windows[1], copies.averageMS[1]];
SGLog(@"sing: %@, %llu windows at %.0f ms each (1500 ms keeps up; CPU copy %llu at %.0f ms, %@, %llu redone on the CPU, the last on the %@), "
- @"lead %.2f s of %.2f s (the clock's %.2f s), %.2f s separated ahead, %llu frames dry, %.1f s dropped, %.1f s of 8 s short, %llu failures, Spotify %@, "
- @"thermal state %s, %@%@",
+ @"lead %.2f s of %.2f s (the clock's %.2f s), %.2f s separated ahead, %llu frames dry, %.1f s dropped, %.1f s of 8 s short%@, %llu failures, Spotify %@, "
+ @"thermal state %s, %@, Runs on %@%@",
SGSingStatusText(), stats.windows, stats.averageMS, copies.windows[0], copies.averageMS[0], fastPart,
- copies.fallbacks, copies.fast ? SGSingUnitsName(fast) : @"CPU", stats.lead, stats.targetLead, player ? SGSingLeadOf(player) : 0, stats.ready,
- stats.dryFrames, stats.dropped, stats.budgetSpent, stats.failures, sg_active ? @"active" : @"not active", SGSingThermalName(),
- [NSString stringWithFormat:@"Runs on %@", SGSingComputeUnitNames()[(NSUInteger)MIN(MAX(runsOn(), 0), 4)]],
+ copies.fallbacks, copies.fast ? @"Neural Engine" : @"CPU", stats.lead, stats.targetLead, player ? SGSingLeadOf(player) : 0, stats.ready,
+ stats.dryFrames, stats.dropped, stats.budgetSpent, stats.budgetHeld ? @" (held while the Neural Engine copy loads)" : @"", stats.failures, sg_active ? @"active" : @"not active", SGSingThermalName(),
+ SGSingMemoryText(), runsOnName(),
sg_spatialListening ? [NSString stringWithFormat:@", the voice %.0f degrees right", stats.voiceAngle * 180 / M_PI] : @"");
}
// Off, with the lead let go: nothing left to watch.
@@ -885,15 +951,15 @@ static void readHeat(void) {
SGPlayerSetStage(stage);
%init;
SGRequireClasses(@[@"SPTPlayerState", @"SPTEsperantoPlayer"]);
- // Runs on was GPU, GPU and Neural Engine, Neural Engine (unset: Neural Engine); a choice made then is the
- // faster copy's now, under its new place in the list, and unset is Automatic.
+ // Runs on is Automatic or CPU only: GPU, Neural Engine and GPU and Neural Engine (2-4) are Automatic now, as is every
+ // value of the earlier key (GPU, GPU and Neural Engine, Neural Engine). The GPU's memory of a stuck load and the
+ // one-model test's switch are gone with them.
NSUserDefaults *defaults = NSUserDefaults.standardUserDefaults;
- NSNumber *before = [defaults objectForKey:SGKeySingComputeUnitsBefore];
- if (before && ![defaults objectForKey:SGKeySingComputeUnits]) {
- NSInteger was = before.integerValue;
- [defaults setInteger:was == 0 ? 2 : was == 1 ? 4 : 3 forKey:SGKeySingComputeUnits];
- [defaults removeObjectForKey:SGKeySingComputeUnitsBefore];
- }
+ if (runsOn() > 1) SGLog(@"sing: Runs on was %ld, which is gone, so it is Automatic", (long)runsOn());
+ if (runsOn() != 1) [defaults removeObjectForKey:SGKeySingComputeUnits];
+ for (NSString *key in @[SGKeySingComputeUnitsBefore, @"spotifyglass.sing.gpuStuckOn", @"spotifyglass.sing.oneModel"]) [defaults removeObjectForKey:key];
+ // The 489 MB model this one replaces, deleted off the main thread once this one is in (kept and run until then).
+ dispatch_async(dispatch_get_global_queue(QOS_CLASS_UTILITY, 0), ^{ SGSingRemoveOldModel(); });
dispatch_async(dispatch_get_main_queue(), ^{
static SGSingTrackWatcher *watcher;
watcher = [SGSingTrackWatcher new];
@@ -917,8 +983,7 @@ static void readHeat(void) {
});
}];
}
- // A load starts only while Spotify is active, so never in a launch into the background, and the faster copy
- // runs the windows only then.
+ // A load starts only while Spotify is active, so never in a launch into the background.
[center addObserverForName:UIApplicationDidBecomeActiveNotification object:nil queue:NSOperationQueue.mainQueue usingBlock:^(NSNotification *note) {
sg_active = YES;
SGSingLoaderSetForeground(YES);
@@ -942,8 +1007,10 @@ static void readHeat(void) {
[center addObserverForName:SGSingChangedNotification object:nil queue:nil usingBlock:^(NSNotification *note) {
if (SGSingModelCurrentState() != SGSingModelReady) {
SGSingLoaderPurge(@"the voice model is gone from the iPhone");
- } else if (SGSingOn() && sg_active && !sg_resting && SGSingLoaderCurrentState() == SGSingLoaderIdle && !(sg_hot && !SGHidden(SGKeySingIgnoreHeat))) {
- // A download finishing with the mic already on loads the model.
+ } else if (SGSingOn() && sg_active && !sg_resting && !(sg_hot && !SGHidden(SGKeySingIgnoreHeat))
+ && (SGSingLoaderCurrentState() == SGSingLoaderIdle || ![SGSingLoaderURL() isEqual:SGSingModelURL()])) {
+ // A download finishing with the mic already on loads the model, or the update in place of the old one
+ // (whose files are gone; its copies in memory run until the new ones are in).
apply();
}
}];
diff --git a/tweak/Sources/Shared/Sing/SingSettings.m b/tweak/Sources/Shared/Sing/SingSettings.m
index 7b062fc0..78ec1c64 100644
--- a/tweak/Sources/Shared/Sing/SingSettings.m
+++ b/tweak/Sources/Shared/Sing/SingSettings.m
@@ -2,7 +2,9 @@
// (SGSingCard.m: the song, what Sing is doing, the vocals and the rest traced live, play and pause, the vocals'
// level and its three stops), Sing's switch, which turns the mic on and off at once, Ignore heat warnings, the voice
// model and its removal, spatial voice's page (its preview, SGSpatialPreview.m, and its switch), and Runs on under
-// Advanced.
+// Advanced. While the page is on screen with Spotify in front, the card's reads of its lines keep Sing separating
+// the playing song even at As sung, where it otherwise rests (SGSingReadLevels), so what the card shows and what
+// its slider sets are heard at once; popped, covered or in the background, the page no longer counts.
#import "Core/SGCore.h"
#import "Settings/SGModPage.h"
#import "Settings/SGPageStyle.h"
@@ -36,18 +38,28 @@ static NSString *bytes(long long count) {
return [NSByteCountFormatter stringFromByteCount:count countStyle:NSByteCountFormatterCountStyleFile];
}
+// The row's state: with the old model, its update reads as the model's download, worded as an update.
+static SGSingModelState rowState(void) {
+ if (SGSingModelUpdateDownloading()) return SGSingModelDownloading;
+ if (SGSingModelUpdateAvailable()) return SGSingModelMissing;
+ return SGSingModelCurrentState();
+}
+
static NSString *modelValue(void) {
if (!SGSingOSSupported() || !SGSingDeviceSupported()) return @"Unavailable";
- switch (SGSingModelCurrentState()) {
+ BOOL update = SGSingModelUpdateAvailable();
+ switch (rowState()) {
case SGSingModelDownloading:
if (SGSingModelChecking()) return @"Checking…";
if (SGSingModelWaitingForNetwork()) return SGSingModelOverCellular() ? @"Waiting for the network" : @"Waiting for Wi-Fi";
return [NSString stringWithFormat:@"%.0f%%", SGSingModelProgress() * 100];
case SGSingModelReady: return [NSString stringWithFormat:@"Downloaded · %@", SGSingModelSizeText()];
case SGSingModelMissing:
- if (SGSingModelError()) return @"Failed, try again";
- if (SGSingModelPausedBytes()) return [NSString stringWithFormat:@"Paused · %@ of %@", bytes(SGSingModelPausedBytes()), SGSingModelSizeText()];
- return [NSString stringWithFormat:@"Download, %@", SGSingModelSizeText()];
+ if (SGSingModelError()) return update ? @"Update failed, try again" : @"Failed, try again";
+ if (SGSingModelPausedBytes()) {
+ return [NSString stringWithFormat:@"%@ · %@ of %@", update ? @"Update paused" : @"Paused", bytes(SGSingModelPausedBytes()), SGSingModelSizeText()];
+ }
+ return [NSString stringWithFormat:update ? @"Update available · %@" : @"Download, %@", SGSingModelSizeText()];
}
return @"";
}
@@ -57,11 +69,16 @@ static void modelTapped(UITableViewController *page) {
tell(@"Voice model", SGSingMissing());
return;
}
- switch (SGSingModelCurrentState()) {
+ switch (rowState()) {
case SGSingModelMissing: {
- NSString *message = [NSString stringWithFormat:@"%@%@ from Hugging Face. It waits for Wi-Fi unless you let it use cellular. It stays on this iPhone, and no audio leaves it.",
- SGSingModelError() ? [NSString stringWithFormat:@"The last try failed: %@\n\n", SGSingModelError()] : @"", SGSingModelSizeText()];
- ask(SGSingModelPausedBytes() ? @"Carry on with the download?" : @"Download the voice model?", message, @"Download", NO, ^{ SGSingDownloadModel(); });
+ BOOL update = SGSingModelUpdateAvailable();
+ NSString *message = [NSString stringWithFormat:@"%@%@%@ from Hugging Face. It waits for Wi-Fi unless you let it use cellular. It stays on this iPhone, and no audio leaves it.",
+ SGSingModelError() ? [NSString stringWithFormat:@"The last try failed: %@\n\n", SGSingModelError()] : @"",
+ update ? @"The new voice model runs on the Neural Engine, faster and cooler, and Karaoke keeps the one it has until it is in. " : @"",
+ SGSingModelSizeText()];
+ NSString *title = SGSingModelPausedBytes() ? (update ? @"Carry on with the update?" : @"Carry on with the download?")
+ : (update ? @"Download the update?" : @"Download the voice model?");
+ ask(title, message, @"Download", NO, ^{ SGSingDownloadModel(); });
break;
}
case SGSingModelDownloading:
@@ -73,7 +90,7 @@ static void modelTapped(UITableViewController *page) {
ask(@"Stop the download?", @"What has come in is kept, so the next download carries on from it.", @"Stop", YES, ^{ SGSingCancelModelDownload(); });
break;
case SGSingModelReady:
- ask(@"Delete the voice model?", [NSString stringWithFormat:@"It frees %@. Karaoke needs it again to turn the vocals down.", SGSingModelSizeText()],
+ ask(@"Delete the voice model?", [NSString stringWithFormat:@"It frees %@. Karaoke needs it again to turn the vocals down.", SGSingModelInUseSizeText()],
@"Delete", YES, ^{
SGSetSingOn(NO);
SGSingDeleteModel();
@@ -165,7 +182,7 @@ UIViewController *SGSingSettingsPage(void) {
BOOL partial = SGSingModelCurrentState() != SGSingModelReady;
ask(partial ? @"Remove the paused download?" : @"Remove the voice model?",
partial ? [NSString stringWithFormat:@"It frees %@. A new download starts from the beginning.", bytes(SGSingModelPausedBytes())]
- : [NSString stringWithFormat:@"It frees %@. Karaoke needs it again to turn the vocals down.", SGSingModelSizeText()],
+ : [NSString stringWithFormat:@"It frees %@. Karaoke needs it again to turn the vocals down.", SGSingModelInUseSizeText()],
@"Remove", YES, ^{
SGSetSingOn(NO);
SGSingDeleteModel();
@@ -180,10 +197,11 @@ UIViewController *SGSingSettingsPage(void) {
// Facts about the iPhone, which do not change while the page shows.
spatial.visible = ^BOOL { return SGSingSpatialAvailable(); };
SGModRow *units = SGChoiceRow(@"Runs on", nil, SGKeySingComputeUnits, SGSingComputeUnitNames(), 0);
- units.choiceNotes = @[@"The GPU, unless it did not load on this iOS before", @"Slower, and the least memory", @"Tried every time, even after it did not load",
- @"Experimental: slower than the CPU on a Mac", @"Experimental"];
- units.choiceFooter = @"Karaoke loads the voice model on the CPU first and starts with it. A second copy then loads for the windows "
- @"played while Spotify is open; in the background they run on the CPU.";
+ units.choiceNotes = @[@"The Neural Engine, the fastest, with the CPU while it loads", @"Slower and warmer, and a little less memory"];
+ units.choiceFooter = @"Karaoke loads the voice model on the CPU first and starts with it. On Automatic a second copy then loads on "
+ @"the Neural Engine and takes over, in the background too. The iPhone prepares it the first time after Vitrine "
+ @"is installed or updated, for a minute or so, and Karaoke runs on the CPU until then. If it does not load, "
+ @"Karaoke stays on the CPU until Spotify is opened again.";
units.chosen = ^(NSInteger index) { SGSingComputeUnitsChanged(); };
SGSingPage *made = [[SGSingPage alloc] initWithTitle:@"Karaoke" intro:nil sections:@[
SGSection(nil, @[sing, heat, model, remove]),