Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.
C
The data volume saves itself, and ramfs stops calling a want of memory a want of room
vcvkka26b48fe9ad2
diff --git a/README.md b/README.mdindex d15b679..0f2b718 100644--- a/README.md+++ b/README.md@@ -1064,12 +1064,36 @@ inode встретился впервые. Не сохраняются сокелегли в то, что было выравнивающим зазором. Телефон один на несколько рабочих копий, и ядро, собранное дозаписи, читает через загрузчик, собранный после неё, и наоборот.+**Сохраняется само, просить не надо.** Задание `com.redwood.varstore` (`var_store watch`) остаётся на весь+прогон и объявляется ядру (`kern.redwood.var_save`). Когда прогон кончается по-хорошему — вышло время+`loader_return_after` или хозяин сказал `./xl stop`, — ядро в том самом потоке, который сейчас вернёт управление+загрузчику, просит том и ждёт ответа (`bsd/dev/redwood_ufs.c`, `redwood_var_save_on_return`), и только потом+отдаёт GPU, процессоры звука и Wi-Fi и уходит. Чего при этом не должно случиться и как это не случается:++- **Ждать некого.** Если никто не объявился, ядро не просит и не ждёт ни миллисекунды: прогон без хранителя+ возвращается точно как раньше. Паника этим путём не идёт вовсе — спрашивать уже некого, — и хранилище остаётся+ с тем, что в нём было.+- **Ждать слишком долго.** Не дольше 20 с (`var_save_ms=N`), и в журнале сказано, сколько ждали. Загрузчик кормит+ сторожевой таймер ещё минуту после срока прогона, а хозяин ждёт ответа сто секунд после него, так что эти+ секунды с запасом внутри обоих. Упаковка и запись 120–160 МБ занимают 6–7 с.+- **Два сохранения разом.** Запись в хранилище отказывается (`EPERM`), если пишущий не держит заявку+ (`kern.redwood.var_busy`), а заявку держит один процесс; заявку, оставленную умершим процессом, следующий+ забирает себе. Да и столкнувшись, двое испортили бы только тот слот, в который пишут, — а это никогда не тот, в+ котором лежит последний хороший архив.+- **Сохранение, прерванное на середине.** Заголовок слота сначала обнуляется, архив пишется, заголовок —+ последним; поэтому недописанный слот просто не проходит проверку, и `load` берёт предыдущий.+- **Прогон, который после сохранения продолжается.** Устройство открывается на время записи и сразу закрывается:+ список запросов этого ядра возвращается в контроллер, и iOS снова читает флеш через него, а не через загрузчик.++Сохранение «каждые N секунд» тоже есть (`var_store watch N`), но по умолчанию его нет: каждое стоит 6–7 с прогона+и сотню с лишним мегабайт записи во флеш, а пока оно идёт, накопитель читается через загрузчик.+**Как запускать.**```shCRYPTEXES=flash FLASHBOOT=1 GL=1 REST=1 RC_JOBS=rc.jobs.persist userland/build.sh./xl ramdisk userland/flashboot.apfs-XL_CMDLINE="root_apfs=ufs_darwin ufs=2 ufs_native=1 ufs_write=1 var_store=1 bootcpu=7 cpus=8 cpufreq=1 touchscreen=1 gpu=1 adsp=1" XL_SECS=220 ./xl run+XL_CMDLINE="root_apfs=ufs_darwin ufs=2 ufs_native=1 ufs_write=1 var_store=1 bootcpu=7 cpus=8 cpufreq=1 touchscreen=1 gpu=1 adsp=1" XL_SECS=200 ./xl run```**Что показал телефон.** Три прогона, между каждыми двумя — настоящий холодный сброс: `./xl reboot` в Android,@@ -1092,20 +1116,24 @@ XL_CMDLINE="root_apfs=ufs_darwin ufs=2 ufs_native=1 ufs_write=1 var_store=1 bootфайл-метка, который копит по строке за прогон, и то, что писала сама iOS — раскладка домашнего экрана(`IconState.plist`), `TodayViewArchive.plist`, `SBLastRestoreIdentifier`, контейнеры приложений.-Место в самом томе — не предел: `vol_ls fs /private/var` показывает 68 608 блоков по 16 КБ, то есть 1,07 ГБ-(четверть памяти телефона), а том при этом занимает 180 МБ. В прогоне 2 восемь файлов из 2401 всё же не-поместились (`No space left on device`, около 2,6 МБ в сумме): весь том идёт в память одним потоком, быстрее, чем-`ramfs` успевает находить страницы под массив страниц файла (`ramfs_grow_slots`, `Z_NOWAIT`), и она отвечает-отказом не по месту, а по странице. Теперь короткая запись выжидается и повторяется — в прогоне 3 не потерялось-ничего. Ссылки в прогоне 2 не восстанавливались вовсе: имя уже было создано шаблоном, и `symlink` отказывал, —-теперь имя шаблона уступает хранилищу.--`var_store=1` — `init` распаковывает хранилище при загрузке; `ufs_write=1` — ядру и загрузчику разрешено писать,-без него сохранить не получится, а распаковать — получится. Сам `var_store`: `save` — упаковать `/private/var`,-`load` — распаковать, `show` — что лежит в обоих слотах, `erase` — забыть оба слота, `stamp ПУТЬ` — дописать в-файл строку о том, когда (метка прогона). Сценарий `userland/rc.jobs.persist` делает всё это по кругу: смотрит-хранилище, печатает строки прошлых прогонов и содержимое `/var/mobile/Library/SpringBoard` с временами, ставит-свою метку, две минуты не мешает iOS и сохраняет том.+**Отказ не по месту, а по странице.** В прогоне 2 восемь файлов из 2401 не поместились (`No space left on+device`, около 2,6 МБ), хотя место было: `vol_ls fs /private/var` показывает 68 608 блоков по 16 КБ — 1,07 ГБ,+четверть памяти телефона, — а том занимал 180 МБ. Весь том идёт в память одним потоком, быстрее, чем `ramfs`+успевает находить память под массив страниц файла, а под замком тома она ждать не может: ожидание может кончиться+в демоне подкачки, который придёт сюда же за этим замком, — поэтому массив брался с `Z_NOWAIT`, и неудача+становилась `ENOSPC`. Теперь массив — забота того, кто пишет, и находится он без замка, как и запасная страница+рядом (`ramfs_grow_slots` отвечает `EAGAIN`: «принеси такой массив и приходи снова», `ramfs_write` его приносит).+Путь подкачки ждать по-прежнему не может и по-прежнему берёт с `Z_NOWAIT`, а его `EAGAIN` остаётся прежним+`ENOSPC` — страницы остаются грязными, и их предложат ещё раз. Ссылки в прогоне 2 не восстанавливались вовсе: имя+уже было создано шаблоном, и `symlink` отказывал, — теперь имя шаблона уступает хранилищу.++`var_store=1` — `init` распаковывает хранилище при загрузке, а задание `com.redwood.varstore` сохраняет его в+конце прогона; `ufs_write=1` — ядру и загрузчику разрешено писать, без него сохранить не получится, а+распаковать — получится. Сам `var_store`: `save` — упаковать `/private/var` сейчас, `watch [N]` — остаться и+сохранять само, `load` — распаковать, `show` — что лежит в обоих слотах, `erase` — забыть оба слота,+`stamp ПУТЬ` — дописать в файл строку о том, когда (метка прогона). Сценарий `userland/rc.jobs.persist` сам не+сохраняет ничего: он печатает хранилище, место в томе, строки прошлых прогонов и содержимое+`/var/mobile/Library/SpringBoard` с временами, ставит свою метку и две с половиной минуты не мешает iOS.## Снимки экранаdiff --git a/userland/build.sh b/userland/build.shindex bf18382..7ce02ff 100755--- a/userland/build.sh+++ b/userland/build.sh@@ -226,7 +226,7 @@ if [ -d $IOS/rootfs/usr/lib ]; thencp ${RC:-rc.launchd} root/etc/rccp ${RC_JOBS:-rc.jobs} root/etc/rc.jobs# the jobs the iOS launchd will see (/System/Library/LaunchDaemons is a symlink to this directory)- cp com.redwood.demo.plist com.redwood.wifi.plist root/LaunchDaemons/+ cp com.redwood.demo.plist com.redwood.wifi.plist com.redwood.varstore.plist root/LaunchDaemons/[ -n "$IOS_JOBS" ] && OSLOG=${OSLOG-com.apple.backboardd} ./mkjobs.py ${REST:+--rest} $IOS/rootfs root/LaunchDaemons $IOS_JOBSfi# firmware of the phone's own hardware, where somebody has pulled it from the phone's Android into ../fw-gpudiff --git a/userland/com.redwood.varstore.plist b/userland/com.redwood.varstore.plistnew file mode 100644index 0000000..42f6c3f--- /dev/null+++ b/userland/com.redwood.varstore.plist@@ -0,0 +1,23 @@+<?xml version="1.0" encoding="UTF-8"?>+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">+<plist version="1.0">+<dict>+ <key>Label</key>+ <string>com.redwood.varstore</string>+ <!-- with var_store=1 among the kernel's arguments: stays for the run and writes /private/var into+ the flash when the kernel says the run is ending (src/var_store.c, watch); without, it ends at+ once. An argument after "watch" is seconds between saves of its own accord as well; there is+ none by default, because each one costs seconds of the run and a hundred-odd megabytes. -->+ <key>ProgramArguments</key>+ <array>+ <string>/redwood/bin/var_store</string>+ <string>watch</string>+ </array>+ <key>RunAtLoad</key>+ <true/>+ <key>StandardOutPath</key>+ <string>/dev/console</string>+ <key>StandardErrorPath</key>+ <string>/dev/console</string>+</dict>+</plist>diff --git a/userland/rc.jobs.persist b/userland/rc.jobs.persistindex 53710e7..8d82b55 100644--- a/userland/rc.jobs.persist+++ b/userland/rc.jobs.persist@@ -1,21 +1,23 @@# Commands run by the launchd job com.redwood.demo (see com.redwood.demo.plist).# The data volume of iOS kept between boots (var_store=1 ufs_write=1; src/var_store.c): what the-# store held when this boot began, the line every run before this one left in /private/var and the-# names SpringBoard had made by then with the seconds they were written, then this run's own line;-# and after two minutes of iOS left to itself -- long enough for the home screen, the snapshots and-# whatever was touched -- the same names again and the whole of /private/var into the store.-# Run it twice over, with ./xl reboot in between: the second run's output holds the first run's-# line and the first run's seconds, which is the whole point.+# store held when this boot began, how much room the volume has, the line every run before this one+# left in /private/var and the names SpringBoard had made by then with the seconds they were+# written; then this run's own line, and after two and a half minutes of iOS left to itself the+# same names again.+# Nothing here saves. The job com.redwood.varstore stays for the run and writes the volume into the+# flash when the kernel says the run is ending -- which is the point: a run that was never told to+# save keeps what it wrote all the same. Run it twice over, with ./xl reboot in between; the second+# run's output holds the first run's line and the first run's seconds./redwood/bin/var_store show/redwood/bin/vol_ls fs /private/var/redwood/bin/vol_ls cat /private/var/redwood-stamp 4096/redwood/bin/vol_ls ls /private/var/mobile/Library/SpringBoard/redwood/bin/var_store stamp /private/var/redwood-stamp-sleep 120+sleep 150/redwood/bin/vol_ls ls /private/var/mobile/Library/SpringBoard/redwood/bin/vol_ls ls /private/var/mobile/Library/redwood/bin/vol_ls cat /private/var/redwood-stamp 4096-/redwood/bin/var_store save/redwood/bin/var_store show-# (how many writes went into darwin, and that each was read back, is in the loader's own log at the-# end of the run: "ufs: N writes into darwin, M bytes, read back the same; K refused")+# (what the saver then did, and how many writes went into darwin, is at the end of the run: the+# kernel's "var_store: the run is ending" and the loader's "ufs: N writes into darwin ... read+# back the same; K refused")diff --git a/userland/src/var_store.c b/userland/src/var_store.cindex f41b202..02b6fd7 100644--- a/userland/src/var_store.c+++ b/userland/src/var_store.c@@ -13,10 +13,24 @@** var_store save pack /private/var into the store* var_store load unpack the newest good store into /private/var (init, at boot)+ * var_store watch [SECS] stay and save by itself: when the kernel says the run is ending, and+ * every SECS seconds besides (a launchd job of ours does this)* var_store show what the store holds, both slots* var_store erase forget both slots (the next boot starts empty again)* var_store stamp PATH write a line into PATH saying when: a marker for a run*+ * Nobody has to remember to save. `watch` registers with the kernel (kern.redwood.var_save), and+ * on the orderly way back to the loader -- the run's time up, or "xl stop" -- the kernel asks it+ * for the volume and waits, bounded, before it goes (bsd/dev/redwood_ufs.c,+ * redwood_var_save_on_return). A save every SECS seconds is there as well but is not the default:+ * one costs seconds of the run and a hundred-odd megabytes of flash, and while it runs the+ * storage is read through the loader rather than through this kernel's own driver.+ *+ * One saver at a time: a write to the store is refused unless the writer holds the claim+ * (kern.redwood.var_busy), so two of them cannot interleave into one slot. Even if they did, the+ * slot being written is never the one holding the newest good archive, and a header is written+ * last and over a zeroed one -- so the worst an interrupted or doubled save can do is lose itself.+ ** Two slots, written in turn: the one not holding the newest good archive is the one written, its* header first zeroed, the archive next, the header last. So a save that does not finish -- the* phone reset in the middle of it -- loses nothing: the other slot still holds the archive from@@ -92,6 +106,81 @@ struct entry {uint64_t size; /* a file's contents, a link's target, a hard link's first name */};+/* ---- the kernel's side of it: the boot-arg, the handshake, the claim ---- */++#define VAR_SAVE_NONE 0+#define VAR_SAVE_IDLE 1+#define VAR_SAVE_WANTED 2+#define VAR_SAVE_DONE 3++/* Is NAME=VALUE among the kernel's arguments? (as init.c reads them) */+static int+boot_arg_is(const char *name, const char *value)+{+ static char args[1024];+ size_t len = sizeof(args) - 1, n = strlen(name), v = strlen(value);++ if (sysctlbyname("kern.bootargs", args, &len, NULL, 0) != 0) {+ return 0;+ }+ args[len < sizeof(args) ? len : sizeof(args) - 1] = 0;+ for (char *p = args; *p;) {+ while (*p == ' ') {+ p++;+ }+ if (strncmp(p, name, n) == 0 && p[n] == '=' && strncmp(p + n + 1, value, v) == 0 &&+ (p[n + 1 + v] == 0 || p[n + 1 + v] == ' ')) {+ return 1;+ }+ while (*p && *p != ' ') {+ p++;+ }+ }+ return 0;+}++static long+parse_number(const char *text)+{+ long value = 0;++ for (; *text >= '0' && *text <= '9'; text++) {+ value = value * 10 + (*text - '0');+ }+ return value;+}++static int+sysctl_int(const char *name, int *value)+{+ size_t len = sizeof(*value);++ return sysctlbyname(name, value, &len, NULL, 0);+}++static int+sysctl_set_int(const char *name, int value)+{+ return sysctlbyname(name, NULL, NULL, &value, sizeof(value));+}++/* The claim on the store: nothing may be written without it, and only one program may hold it. */+static int+claim(void)+{+ if (sysctl_set_int("kern.redwood.var_busy", 1) != 0) {+ printf("var_store: the store is claimed by another saver (%s)\n", strerror(errno));+ return 0;+ }+ return 1;+}++static void+release(void)+{+ (void)sysctl_set_int("kern.redwood.var_busy", 0);+}+/* ---- checksum (the same one the loader reads a write back with) ---- */static uint32_t crc_table[256];@@ -658,6 +747,9 @@ save(void)}stream_start(slot, 0);entries = 0;+ /* a watcher saves more than once: nothing of the save before is carried into this one */+ memset(links, 0, LINKS * sizeof(*links));+ skipped_kinds = skipped_big = 0;strcpy(path, ROOT);if (!pack_tree("") || !out_flush(1)) {return 0;@@ -697,6 +789,16 @@ save(void)/* ---- unpacking ---- */static uint64_t made_dirs, made_files, made_links, made_bytes, short_files, lost_bytes;+/*+ * A belt, after the braces: the memory filesystem no longer answers ENOSPC for want of a page+ * (bsd/miscfs/ramfs, ramfs_grow_slots now takes an array found without the mount lock), so a short+ * write here means the volume really is full. Waiting a little and trying again still earns its+ * place -- the pageout path cannot wait for an array either, and it may be what answered -- but+ * the whole restore spends at most RETRY_MS on it, so a volume that is truly full does not turn+ * the boot into a crawl.+ */+#define RETRY_MS 2000+static uint64_t retry_ms_left = RETRY_MS;static intwhole_path(const char *relative, uint32_t len)@@ -824,14 +926,12 @@ unpack_one(int make)*/while (out_fd >= 0 && done < want) {ssize_t wrote = write(out_fd, piece + done, (size_t)(want - done));- static int tries;if (wrote > 0) {done += (uint64_t)wrote;- tries = 0;continue;}- if (++tries > 200) {+ if (retry_ms_left < 5) {printf("var_store: %s: %llu of %llu bytes, no more would go in: %s\n",path, (unsigned long long)(in_file + done),(unsigned long long)e.size, strerror(errno));@@ -839,10 +939,10 @@ unpack_one(int make)lost_bytes += e.size - in_file - done;close(out_fd);out_fd = -1;- tries = 0;break;}usleep(5000);+ retry_ms_left -= 5;}in_file += done;left -= want;@@ -1032,6 +1132,84 @@ erase(void)return 1;}+/*+ * Stay and save by itself. The kernel is told there is a saver, and then this waits: for the+ * kernel to ask (the run is ending, and it waits for the answer before it goes), and, if seconds+ * is given, for that long to pass between saves of its own accord. A save that fails is said so+ * and the waiting goes on -- the one at the end of the run still gets its chance.+ */+static int+save_once(void)+{+ int ok;++ /*+ * The device is opened for the save and closed again, not held: while a write is in flight+ * this kernel's own transfer list is out of the controller, and it goes back in when the+ * device is closed (bsd/dev/redwood_ufs.c, ufs_cclose). A watcher that kept the device open+ * would leave iOS reading the flash through the loader for the rest of the run.+ */+ fd = open(DEVICE, O_RDWR, 0);+ if (fd < 0) {+ printf("var_store: %s: %s\n", DEVICE, strerror(errno));+ return 0;+ }+ ok = claim() && save();+ release();+ close(fd);+ fd = -1;+ return ok;+}++static int+watch(int seconds)+{+ uint64_t last = 0;+ int asked = 0;++ fd = open(DEVICE, O_RDWR, 0);+ if (fd < 0) {+ printf("var_store: %s cannot be written (%s): nothing would be saved\n", DEVICE,+ strerror(errno));+ return 0;+ }+ close(fd);+ fd = -1;+ if (sysctl_set_int("kern.redwood.var_save", VAR_SAVE_IDLE) != 0) {+ printf("var_store: this kernel does not keep the data volume (%s)\n", strerror(errno));+ return 0;+ }+ printf("var_store: watching: the volume is saved when the run ends%s\n",+ seconds > 0 ? ", and in between" : "");+ {+ struct timeval tv = { 0, 0 };++ gettimeofday(&tv, NULL);+ last = (uint64_t)tv.tv_sec;+ }+ for (;;) {+ struct timeval tv = { 0, 0 };+ int state = VAR_SAVE_NONE;++ if (sysctl_int("kern.redwood.var_save", &state) == 0 && state == VAR_SAVE_WANTED) {+ asked = 1;+ }+ gettimeofday(&tv, NULL);+ if (asked || (seconds > 0 && (uint64_t)tv.tv_sec - last >= (uint64_t)seconds)) {+ if (!save_once()) {+ printf("var_store: this save came to nothing; the store keeps what it had\n");+ }+ last = (uint64_t)tv.tv_sec;+ if (asked) {+ /* the kernel is waiting on this to go back to the loader */+ (void)sysctl_set_int("kern.redwood.var_save", VAR_SAVE_DONE);+ return 1;+ }+ }+ usleep(100000);+ }+}+/* A line saying when, for a run to leave behind and the next one to find. */static intstamp(const char *where)@@ -1091,6 +1269,13 @@ main(int argc, char **argv, char **envp)printf("var_store: no memory for the buffers\n");return 1;}+ if (strcmp(what, "watch") == 0) {+ if (!boot_arg_is("var_store", "1")) {+ printf("var_store: not asked to keep the data volume (var_store=1): nothing to watch\n");+ return 0;+ }+ return watch(argc > 2 ? (int)parse_number(argv[2]) : 0) ? 0 : 1;+ }fd = open(DEVICE, strcmp(what, "load") == 0 || strcmp(what, "show") == 0 ? O_RDONLY : O_RDWR, 0);if (fd < 0) {printf("var_store: %s: %s (the kernel takes writes with ufs_write=1)\n", DEVICE,@@ -1098,7 +1283,8 @@ main(int argc, char **argv, char **envp)return 1;}if (strcmp(what, "save") == 0) {- ok = save();+ ok = claim() && save();+ release();} else if (strcmp(what, "load") == 0) {ok = load();} else if (strcmp(what, "erase") == 0) {diff --git a/xnu/files/bsd/dev/redwood_ufs.c b/xnu/files/bsd/dev/redwood_ufs.cindex 7d9d2eb..62e06ee 100644--- a/xnu/files/bsd/dev/redwood_ufs.c+++ b/xnu/files/bsd/dev/redwood_ufs.c@@ -172,6 +172,36 @@ static LCK_MTX_DECLARE(ufs_lock, &ufs_lck_grp);/* One writer at a time; never held while the mailbox is, nor while user memory is touched. */static LCK_MTX_DECLARE(ufs_write_mtx, &ufs_lck_grp);+/*+ * Saving the data volume without being asked (userland/src/var_store.c).+ *+ * The volume is packed and written by a program, not by this kernel, so the kernel's part is to+ * say when and to let only one program at a time do it:+ *+ * kern.redwood.var_save the handshake. The saver writes VAR_SAVE_IDLE when it starts and is+ * then the one that will be asked; on the orderly way back to the loader+ * (loader_return_after expired, or "xl stop") the kernel writes+ * VAR_SAVE_WANTED and waits, and the saver writes VAR_SAVE_DONE when the+ * volume is in the flash. Nobody registered: nothing is asked and+ * nothing waited for, so a run without a saver returns as it always did.+ * A panic does not come this way at all -- there is no program left to+ * ask -- and then the store simply keeps what it had.+ * kern.redwood.var_busy the claim. A write to /dev/rufs_darwin is refused unless the writer+ * holds it, so two savers cannot interleave into one slot. It is held by+ * process number, and a claimant may take it from a process that is gone+ * (a saver killed in the middle leaves no lock behind for the rest of the+ * run). What the one that was killed half-wrote is not lost either: the+ * slot it was writing is not the one holding the newest good archive.+ */+#define VAR_SAVE_NONE 0+#define VAR_SAVE_IDLE 1+#define VAR_SAVE_WANTED 2+#define VAR_SAVE_DONE 3+#define VAR_SAVE_MS 20000 /* at most this long on the way out (var_save_ms=N) */++static volatile int ufs_var_save = VAR_SAVE_NONE;+static int ufs_var_busy_pid;+/* ufs_write=1 and the one partition it is for: everything else answers EROFS as it always did */static int ufs_write_part = -1;static volatile bool ufs_write_busy; /* a write of ours has this kernel's list out: not a failure */@@ -198,6 +228,71 @@ SYSCTL_QUAD(_kern_redwood, OID_AUTO, ufs_write_errors, CTLFLAG_RD | CTLFLAG_LOCKSYSCTL_QUAD(_kern_redwood, OID_AUTO, ufs_write_refused, CTLFLAG_RD | CTLFLAG_LOCKED, &ufs_write_refused,"writes refused here: not darwin, or not past what the build writes");+/* The handshake: 1 to register as the saver, 3 to say the volume is in the flash. 2 is the kernel's. */+static int+ufs_sysctl_var_save(__unused struct sysctl_oid *oidp, __unused void *arg1, __unused int arg2, struct sysctl_req *req)+{+ int value = ufs_var_save, error;++ error = sysctl_io_number(req, value, sizeof(value), &value, NULL);+ if (error != 0 || req->newptr == 0) {+ return error;+ }+ if (value == VAR_SAVE_IDLE && ufs_var_save != VAR_SAVE_WANTED) {+ ufs_var_save = VAR_SAVE_IDLE; /* a saver is there; it is the one that will be asked */+ } else if (value == VAR_SAVE_DONE && ufs_var_save == VAR_SAVE_WANTED) {+ ufs_var_save = VAR_SAVE_DONE; /* and it has finished: the way out may go on */+ } else if (value == VAR_SAVE_NONE) {+ ufs_var_save = VAR_SAVE_NONE;+ } else {+ return EINVAL;+ }+ return 0;+}++SYSCTL_PROC(_kern_redwood, OID_AUTO, var_save, CTLTYPE_INT | CTLFLAG_RW | CTLFLAG_LOCKED, NULL, 0,+ ufs_sysctl_var_save, "I", "the data volume's saver: 1 registers, 2 is the kernel asking, 3 is done");++/* The claim on the store: 1 takes it, 0 gives it back; reading says which process holds it. */+static int+ufs_sysctl_var_busy(__unused struct sysctl_oid *oidp, __unused void *arg1, __unused int arg2, struct sysctl_req *req)+{+ int value = ufs_var_busy_pid, error, me = proc_selfpid();++ error = sysctl_io_number(req, value, sizeof(value), &value, NULL);+ if (error != 0 || req->newptr == 0) {+ return error;+ }+ lck_mtx_lock(&ufs_write_mtx);+ if (value == 1) {+ if (ufs_var_busy_pid != 0 && ufs_var_busy_pid != me) {+ proc_t held = proc_find(ufs_var_busy_pid);++ if (held != NULL) {+ proc_rele(held);+ lck_mtx_unlock(&ufs_write_mtx);+ return EBUSY; /* another saver has it and is alive: one at a time */+ }+ printf("var_store: the claim on the store was left by process %d, which is gone\n",+ ufs_var_busy_pid);+ }+ ufs_var_busy_pid = me;+ } else if (value == 0) {+ if (ufs_var_busy_pid == me || ufs_var_busy_pid == 0) {+ ufs_var_busy_pid = 0;+ } else {+ error = EPERM;+ }+ } else {+ error = EINVAL;+ }+ lck_mtx_unlock(&ufs_write_mtx);+ return error;+}++SYSCTL_PROC(_kern_redwood, OID_AUTO, var_busy, CTLTYPE_INT | CTLFLAG_RW | CTLFLAG_LOCKED, NULL, 0,+ ufs_sysctl_var_busy, "I", "the claim a writer of /dev/rufs_darwin must hold: 1 takes it, 0 gives it back");+/** Native blocks [lba, lba + count) of partition part, one request of at most the buffer's size:* read into dst, or, with src, written from there. Called with ufs_lock held.@@ -735,6 +830,11 @@ ufs_cwrite(dev_t dev, struct uio *uio, __unused int ioflag)if (!ufs_ready || part >= ufs_parts || ufs_write_part < 0 || part != (uint32_t)ufs_write_part) {return EROFS;}+ if (ufs_var_busy_pid != proc_selfpid()) {+ /* one writer of the store at a time, and it says so first (kern.redwood.var_busy) */+ ufs_write_refused++;+ return EPERM;+ }nbs = ufs_page->part[part].block_size;if (uio_offset(uio) < 0 || (uint64_t)uio_offset(uio) % nbs != 0 || (uint64_t)uio_resid(uio) % nbs != 0) {return EINVAL;@@ -791,6 +891,37 @@ static const struct cdevsw ufs_cdevsw = {.d_type = D_DISK,};+/*+ * The orderly end of a run, from osfmk/arm64/loader_return.c, in the thread that is about to hand+ * control back: ask the saver for the data volume and wait for it, so that a run need not have+ * been told to save. Bounded, and only when a saver registered -- the loader goes on feeding the+ * watchdog for a minute past the run's own budget, and the host waits a hundred seconds past it,+ * so twenty seconds here is well inside both. Nothing else of the way out has happened yet: the+ * GPU, the DSPs and this kernel's transfer list are still as the run had them.+ */+void+redwood_var_save_on_return(void)+{+ uint32_t ms = VAR_SAVE_MS;+ uint64_t waited = 0;++ if (!ufs_ready || ufs_write_part < 0 || ufs_var_save != VAR_SAVE_IDLE) {+ return;+ }+ PE_parse_boot_argn("var_save_ms", &ms, sizeof(ms));+ printf("var_store: the run is ending: the data volume is asked for\n");+ ufs_var_save = VAR_SAVE_WANTED;+ while (ufs_var_save != VAR_SAVE_DONE && waited < ms) {+ IOSleep(20);+ waited += 20;+ }+ if (ufs_var_save == VAR_SAVE_DONE) {+ printf("var_store: the data volume is in the flash (%llu ms)\n", waited);+ } else {+ printf("var_store: the saver did not answer in %u ms: the store keeps what it had\n", ms);+ }+}+/* ---- the check: the partitions there are images of, hashed as the loader hashes them ---- */static const struct {diff --git a/xnu/files/bsd/miscfs/ramfs/ramfs.c b/xnu/files/bsd/miscfs/ramfs/ramfs.cindex 6510077..8776736 100644--- a/xnu/files/bsd/miscfs/ramfs/ramfs.c+++ b/xnu/files/bsd/miscfs/ramfs/ramfs.c@@ -358,43 +358,70 @@ ramfs_vget_locked(struct ramfs_mount *rm, struct ramfs_node *node, vnode_t dvp,/* ------------------------------------------------------------------------ *//* File contents */-/* Make sure the page array has a slot for page `index`. The mount lock is held. */+/* How many slots an array needs to hold page `index` of a file that has `have` of them. */+static uint64_t+ramfs_slots_wanted(uint64_t have, uint64_t index)+{+ uint64_t want = have ? have : 4;++ while (want <= index) {+ want *= 2;+ }+ return want;+}++/*+ * Make sure the page array has a slot for page `index`. The mount lock is held, and nothing waits+ * for memory under it -- a wait could end in the pageout daemon, which comes back here for the+ * lock. So the array is the caller's to find room for, without the lock: *slots of *nslots+ * entries, used and consumed when it is big enough. EAGAIN means only that: bring one of+ * ramfs_slots_wanted() entries and come again; it is not ENOSPC, and a caller that turns it into+ * one says a volume with room to spare is full. A caller that cannot wait either -- the pageout+ * path -- brings none, and then this tries without waiting, as it always did.+ */static int-ramfs_grow_slots(struct ramfs_node *node, uint64_t index)+ramfs_grow_slots(struct ramfs_node *node, uint64_t index, void ***slots, uint64_t *nslots){- uint64_t want = node->rn_nslots ? node->rn_nslots : 4;- void **slots;+ void **grown;+ uint64_t grown_n;if (index < node->rn_nslots) {return 0;}- while (want <= index) {- want *= 2;- }- slots = kalloc_type(void *, want, Z_NOWAIT | Z_ZERO);- if (slots == NULL) {- return ENOSPC;+ if (*slots != NULL && *nslots > index && *nslots >= node->rn_nslots) {+ grown = *slots;+ grown_n = *nslots;+ *slots = NULL;+ } else {+ grown_n = ramfs_slots_wanted(node->rn_nslots, index);+ grown = kalloc_type(void *, grown_n, Z_NOWAIT | Z_ZERO);+ if (grown == NULL) {+ return EAGAIN;+ }}if (node->rn_pages != NULL) {- memcpy(slots, node->rn_pages, node->rn_nslots * sizeof(void *));+ memcpy(grown, node->rn_pages, node->rn_nslots * sizeof(void *));kfree_type(void *, node->rn_nslots, node->rn_pages);}- node->rn_pages = slots;- node->rn_nslots = want;+ node->rn_pages = grown;+ node->rn_nslots = grown_n;return 0;}/** Copy len bytes (within one page) into the file at offset off, extending* the file if needed. *spare is a zeroed page the caller allocated without- * the lock; it is consumed if the slot was empty. The mount lock is held.+ * the lock; it is consumed if the slot was empty. *slots is the page array+ * the caller allocated the same way (see ramfs_grow_slots). The mount lock+ * is held.*/static int-ramfs_store(struct ramfs_node *node, uint64_t off, const void *src, size_t len, void **spare)+ramfs_store(struct ramfs_node *node, uint64_t off, const void *src, size_t len, void **spare,+ void ***slots, uint64_t *nslots){struct ramfs_mount *rm = node->rn_mount;uint64_t index = off / PAGE_SIZE;- int error = ramfs_grow_slots(node, index);+ int error = ramfs_grow_slots(node, index, slots, nslots);if (error) {return error;@@ -990,7 +1017,8 @@ ramfs_write(struct vnop_write_args *ap)struct uio *uio = ap->a_uio;struct ramfs_node *node = RAMFS_NODE(vp);struct ramfs_mount *rm = node->rn_mount;- void *spare = NULL;+ void *spare = NULL, **slots = NULL;+ uint64_t nslots = 0;char *bounce;uint64_t start, old_size;int error = 0;@@ -1026,9 +1054,32 @@ ramfs_write(struct vnop_write_args *ap)if (spare == NULL) {spare = kalloc_data(PAGE_SIZE, Z_WAITOK | Z_ZERO);}- lck_mtx_lock(&rm->rm_lock);- error = ramfs_store(node, off, bounce, len, &spare);- lck_mtx_unlock(&rm->rm_lock);+ /*+ * EAGAIN says the page array has to grow and that room for it is this side's to find,+ * which is done here, where waiting is allowed. rn_nslots is read without the lock as a+ * hint only; another writer growing the same file at the same time makes the hint too+ * small, and then this asks again with a fresh one.+ */+ for (int again = 0; again < 4; again++) {+ lck_mtx_lock(&rm->rm_lock);+ error = ramfs_store(node, off, bounce, len, &spare, &slots, &nslots);+ lck_mtx_unlock(&rm->rm_lock);+ if (error != EAGAIN) {+ break;+ }+ if (slots != NULL) {+ kfree_type(void *, nslots, slots);+ }+ nslots = ramfs_slots_wanted(node->rn_nslots, off / PAGE_SIZE);+ slots = kalloc_type(void *, nslots, Z_WAITOK | Z_ZERO);+ if (slots == NULL) {+ nslots = 0;+ break;+ }+ }+ if (error == EAGAIN) {+ error = ENOSPC; /* it never got the array: out of memory, not out of volume */+ }if (error) {/* nothing of this piece was stored: hand it back to the caller's count */uio_setoffset(uio, (off_t)off);@@ -1040,6 +1091,9 @@ ramfs_write(struct vnop_write_args *ap)if (spare != NULL) {kfree_data(spare, PAGE_SIZE);}+ if (slots != NULL) {+ kfree_type(void *, nslots, slots);+ }if ((uint64_t)uio_offset(uio) > start) {uint64_t new_size;@@ -1318,9 +1372,13 @@ ramfs_pageout(struct vnop_pageout_args *ap)for (upl_size_t done = 0; done < size && off + done < node->rn_size && !error; done += PAGE_SIZE) {size_t len = (size_t)MIN((uint64_t)PAGE_SIZE, node->rn_size - (off + done));/* this may be the pageout daemon looking for memory: never wait for more of it here */- void *spare = kalloc_data(PAGE_SIZE, Z_NOWAIT | Z_ZERO);+ void *spare = kalloc_data(PAGE_SIZE, Z_NOWAIT | Z_ZERO), **slots = NULL;+ uint64_t nslots = 0;- error = ramfs_store(node, off + done, (const void *)(addr + done), len, &spare);+ error = ramfs_store(node, off + done, (const void *)(addr + done), len, &spare, &slots, &nslots);+ if (error == EAGAIN) {+ error = ENOSPC; /* nothing here may wait for the array either: the pages stay dirty */+ }if (spare != NULL) {kfree_data(spare, PAGE_SIZE);}diff --git a/xnu/files/osfmk/arm64/loader_return.c b/xnu/files/osfmk/arm64/loader_return.cindex e7e2dbb..a058bb5 100644--- a/xnu/files/osfmk/arm64/loader_return.c+++ b/xnu/files/osfmk/arm64/loader_return.c@@ -57,6 +57,7 @@ extern void redwood_torch_quiesce(void);extern void redwood_camera_quiesce(void);extern void redwood_mic_quiesce(void);extern void redwood_ufs_quiesce(void);+extern void redwood_var_save_on_return(void); /* bsd/dev/redwood_ufs.c: the data volume into the flash */void loader_checkpoint(uint64_t n);/* Also referenced from the early exception vector in start.s. */@@ -435,6 +436,7 @@ loader_return_timer(__unused thread_call_param_t p0, __unused thread_call_param_}PE_parse_boot_argn("return_panic", &as_panic, sizeof(as_panic));if (!as_panic) {+ redwood_var_save_on_return(); /* the data volume kept, while the system is still whole */redwood_adreno_stop(); /* a GPU kept for the system is given back while there is a thread to wait in */redwood_wifi_stop(); /* and the Wi-Fi DSP, if it runs, shut down */redwood_adsp_stop(); /* and the audio DSP */@@ -480,6 +482,7 @@ loader_stop_poll(__unused thread_call_param_t p0, __unused thread_call_param_t pfor (const char *c = banner; *c; c++) {loader_log_putc(*c);}+ redwood_var_save_on_return();redwood_adreno_stop();redwood_wifi_stop();redwood_adsp_stop();