Skip to content

Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.

C

The data volume saves itself, and ramfs stops calling a want of memory a want of room

vcvkka26b48fe9ad2
diff --git a/README.md b/README.md
index d15b679..0f2b718 100644
--- a/README.md
+++ b/README.md
@@ -1064,12 +1064,36 @@ inode встретился впервые. Не сохраняются соке
легли в то, что было выравнивающим зазором. Телефон один на несколько рабочих копий, и ядро, собранное до
записи, читает через загрузчик, собранный после неё, и наоборот.
+**Сохраняется само, просить не надо.** Задание `com.redwood.varstore` (`var_store watch`) остаётся на весь
+прогон и объявляется ядру (`kern.redwood.var_save`). Когда прогон кончается по-хорошему — вышло время
+`loader_return_after` или хозяин сказал `./xl stop`, — ядро в том самом потоке, который сейчас вернёт управление
+загрузчику, просит том и ждёт ответа (`bsd/dev/redwood_ufs.c`, `redwood_var_save_on_return`), и только потом
+отдаёт GPU, процессоры звука и Wi-Fi и уходит. Чего при этом не должно случиться и как это не случается:
+
+- **Ждать некого.** Если никто не объявился, ядро не просит и не ждёт ни миллисекунды: прогон без хранителя
+ возвращается точно как раньше. Паника этим путём не идёт вовсе — спрашивать уже некого, — и хранилище остаётся
+ с тем, что в нём было.
+- **Ждать слишком долго.** Не дольше 20 с (`var_save_ms=N`), и в журнале сказано, сколько ждали. Загрузчик кормит
+ сторожевой таймер ещё минуту после срока прогона, а хозяин ждёт ответа сто секунд после него, так что эти
+ секунды с запасом внутри обоих. Упаковка и запись 120–160 МБ занимают 6–7 с.
+- **Два сохранения разом.** Запись в хранилище отказывается (`EPERM`), если пишущий не держит заявку
+ (`kern.redwood.var_busy`), а заявку держит один процесс; заявку, оставленную умершим процессом, следующий
+ забирает себе. Да и столкнувшись, двое испортили бы только тот слот, в который пишут, — а это никогда не тот, в
+ котором лежит последний хороший архив.
+- **Сохранение, прерванное на середине.** Заголовок слота сначала обнуляется, архив пишется, заголовок —
+ последним; поэтому недописанный слот просто не проходит проверку, и `load` берёт предыдущий.
+- **Прогон, который после сохранения продолжается.** Устройство открывается на время записи и сразу закрывается:
+ список запросов этого ядра возвращается в контроллер, и iOS снова читает флеш через него, а не через загрузчик.
+
+Сохранение «каждые N секунд» тоже есть (`var_store watch N`), но по умолчанию его нет: каждое стоит 6–7 с прогона
+и сотню с лишним мегабайт записи во флеш, а пока оно идёт, накопитель читается через загрузчик.
+
**Как запускать.**
```sh
CRYPTEXES=flash FLASHBOOT=1 GL=1 REST=1 RC_JOBS=rc.jobs.persist userland/build.sh
./xl ramdisk userland/flashboot.apfs
-XL_CMDLINE="root_apfs=ufs_darwin ufs=2 ufs_native=1 ufs_write=1 var_store=1 bootcpu=7 cpus=8 cpufreq=1 touchscreen=1 gpu=1 adsp=1" XL_SECS=220 ./xl run
+XL_CMDLINE="root_apfs=ufs_darwin ufs=2 ufs_native=1 ufs_write=1 var_store=1 bootcpu=7 cpus=8 cpufreq=1 touchscreen=1 gpu=1 adsp=1" XL_SECS=200 ./xl run
```
**Что показал телефон.** Три прогона, между каждыми двумя — настоящий холодный сброс: `./xl reboot` в Android,
@@ -1092,20 +1116,24 @@ XL_CMDLINE="root_apfs=ufs_darwin ufs=2 ufs_native=1 ufs_write=1 var_store=1 boot
файл-метка, который копит по строке за прогон, и то, что писала сама iOS — раскладка домашнего экрана
(`IconState.plist`), `TodayViewArchive.plist`, `SBLastRestoreIdentifier`, контейнеры приложений.
-Место в самом томе — не предел: `vol_ls fs /private/var` показывает 68 608 блоков по 16 КБ, то есть 1,07 ГБ
-(четверть памяти телефона), а том при этом занимает 180 МБ. В прогоне 2 восемь файлов из 2401 всё же не
-поместились (`No space left on device`, около 2,6 МБ в сумме): весь том идёт в память одним потоком, быстрее, чем
-`ramfs` успевает находить страницы под массив страниц файла (`ramfs_grow_slots`, `Z_NOWAIT`), и она отвечает
-отказом не по месту, а по странице. Теперь короткая запись выжидается и повторяется — в прогоне 3 не потерялось
-ничего. Ссылки в прогоне 2 не восстанавливались вовсе: имя уже было создано шаблоном, и `symlink` отказывал, —
-теперь имя шаблона уступает хранилищу.
-
-`var_store=1` — `init` распаковывает хранилище при загрузке; `ufs_write=1` — ядру и загрузчику разрешено писать,
-без него сохранить не получится, а распаковать — получится. Сам `var_store`: `save` — упаковать `/private/var`,
-`load` — распаковать, `show` — что лежит в обоих слотах, `erase` — забыть оба слота, `stamp ПУТЬ` — дописать в
-файл строку о том, когда (метка прогона). Сценарий `userland/rc.jobs.persist` делает всё это по кругу: смотрит
-хранилище, печатает строки прошлых прогонов и содержимое `/var/mobile/Library/SpringBoard` с временами, ставит
-свою метку, две минуты не мешает iOS и сохраняет том.
+**Отказ не по месту, а по странице.** В прогоне 2 восемь файлов из 2401 не поместились (`No space left on
+device`, около 2,6 МБ), хотя место было: `vol_ls fs /private/var` показывает 68 608 блоков по 16 КБ — 1,07 ГБ,
+четверть памяти телефона, — а том занимал 180 МБ. Весь том идёт в память одним потоком, быстрее, чем `ramfs`
+успевает находить память под массив страниц файла, а под замком тома она ждать не может: ожидание может кончиться
+в демоне подкачки, который придёт сюда же за этим замком, — поэтому массив брался с `Z_NOWAIT`, и неудача
+становилась `ENOSPC`. Теперь массив — забота того, кто пишет, и находится он без замка, как и запасная страница
+рядом (`ramfs_grow_slots` отвечает `EAGAIN`: «принеси такой массив и приходи снова», `ramfs_write` его приносит).
+Путь подкачки ждать по-прежнему не может и по-прежнему берёт с `Z_NOWAIT`, а его `EAGAIN` остаётся прежним
+`ENOSPC` — страницы остаются грязными, и их предложат ещё раз. Ссылки в прогоне 2 не восстанавливались вовсе: имя
+уже было создано шаблоном, и `symlink` отказывал, — теперь имя шаблона уступает хранилищу.
+
+`var_store=1` — `init` распаковывает хранилище при загрузке, а задание `com.redwood.varstore` сохраняет его в
+конце прогона; `ufs_write=1` — ядру и загрузчику разрешено писать, без него сохранить не получится, а
+распаковать — получится. Сам `var_store`: `save` — упаковать `/private/var` сейчас, `watch [N]` — остаться и
+сохранять само, `load` — распаковать, `show` — что лежит в обоих слотах, `erase` — забыть оба слота,
+`stamp ПУТЬ` — дописать в файл строку о том, когда (метка прогона). Сценарий `userland/rc.jobs.persist` сам не
+сохраняет ничего: он печатает хранилище, место в томе, строки прошлых прогонов и содержимое
+`/var/mobile/Library/SpringBoard` с временами, ставит свою метку и две с половиной минуты не мешает iOS.
## Снимки экрана
diff --git a/userland/build.sh b/userland/build.sh
index bf18382..7ce02ff 100755
--- a/userland/build.sh
+++ b/userland/build.sh
@@ -226,7 +226,7 @@ if [ -d $IOS/rootfs/usr/lib ]; then
cp ${RC:-rc.launchd} root/etc/rc
cp ${RC_JOBS:-rc.jobs} root/etc/rc.jobs
# the jobs the iOS launchd will see (/System/Library/LaunchDaemons is a symlink to this directory)
- cp com.redwood.demo.plist com.redwood.wifi.plist root/LaunchDaemons/
+ cp com.redwood.demo.plist com.redwood.wifi.plist com.redwood.varstore.plist root/LaunchDaemons/
[ -n "$IOS_JOBS" ] && OSLOG=${OSLOG-com.apple.backboardd} ./mkjobs.py ${REST:+--rest} $IOS/rootfs root/LaunchDaemons $IOS_JOBS
fi
# firmware of the phone's own hardware, where somebody has pulled it from the phone's Android into ../fw-gpu
diff --git a/userland/com.redwood.varstore.plist b/userland/com.redwood.varstore.plist
new file mode 100644
index 0000000..42f6c3f
--- /dev/null
+++ b/userland/com.redwood.varstore.plist
@@ -0,0 +1,23 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+ <key>Label</key>
+ <string>com.redwood.varstore</string>
+ <!-- with var_store=1 among the kernel's arguments: stays for the run and writes /private/var into
+ the flash when the kernel says the run is ending (src/var_store.c, watch); without, it ends at
+ once. An argument after "watch" is seconds between saves of its own accord as well; there is
+ none by default, because each one costs seconds of the run and a hundred-odd megabytes. -->
+ <key>ProgramArguments</key>
+ <array>
+ <string>/redwood/bin/var_store</string>
+ <string>watch</string>
+ </array>
+ <key>RunAtLoad</key>
+ <true/>
+ <key>StandardOutPath</key>
+ <string>/dev/console</string>
+ <key>StandardErrorPath</key>
+ <string>/dev/console</string>
+</dict>
+</plist>
diff --git a/userland/rc.jobs.persist b/userland/rc.jobs.persist
index 53710e7..8d82b55 100644
--- a/userland/rc.jobs.persist
+++ b/userland/rc.jobs.persist
@@ -1,21 +1,23 @@
# Commands run by the launchd job com.redwood.demo (see com.redwood.demo.plist).
# The data volume of iOS kept between boots (var_store=1 ufs_write=1; src/var_store.c): what the
-# store held when this boot began, the line every run before this one left in /private/var and the
-# names SpringBoard had made by then with the seconds they were written, then this run's own line;
-# and after two minutes of iOS left to itself -- long enough for the home screen, the snapshots and
-# whatever was touched -- the same names again and the whole of /private/var into the store.
-# Run it twice over, with ./xl reboot in between: the second run's output holds the first run's
-# line and the first run's seconds, which is the whole point.
+# store held when this boot began, how much room the volume has, the line every run before this one
+# left in /private/var and the names SpringBoard had made by then with the seconds they were
+# written; then this run's own line, and after two and a half minutes of iOS left to itself the
+# same names again.
+# Nothing here saves. The job com.redwood.varstore stays for the run and writes the volume into the
+# flash when the kernel says the run is ending -- which is the point: a run that was never told to
+# save keeps what it wrote all the same. Run it twice over, with ./xl reboot in between; the second
+# run's output holds the first run's line and the first run's seconds.
/redwood/bin/var_store show
/redwood/bin/vol_ls fs /private/var
/redwood/bin/vol_ls cat /private/var/redwood-stamp 4096
/redwood/bin/vol_ls ls /private/var/mobile/Library/SpringBoard
/redwood/bin/var_store stamp /private/var/redwood-stamp
-sleep 120
+sleep 150
/redwood/bin/vol_ls ls /private/var/mobile/Library/SpringBoard
/redwood/bin/vol_ls ls /private/var/mobile/Library
/redwood/bin/vol_ls cat /private/var/redwood-stamp 4096
-/redwood/bin/var_store save
/redwood/bin/var_store show
-# (how many writes went into darwin, and that each was read back, is in the loader's own log at the
-# end of the run: "ufs: N writes into darwin, M bytes, read back the same; K refused")
+# (what the saver then did, and how many writes went into darwin, is at the end of the run: the
+# kernel's "var_store: the run is ending" and the loader's "ufs: N writes into darwin ... read
+# back the same; K refused")
diff --git a/userland/src/var_store.c b/userland/src/var_store.c
index f41b202..02b6fd7 100644
--- a/userland/src/var_store.c
+++ b/userland/src/var_store.c
@@ -13,10 +13,24 @@
*
* var_store save pack /private/var into the store
* var_store load unpack the newest good store into /private/var (init, at boot)
+ * var_store watch [SECS] stay and save by itself: when the kernel says the run is ending, and
+ * every SECS seconds besides (a launchd job of ours does this)
* var_store show what the store holds, both slots
* var_store erase forget both slots (the next boot starts empty again)
* var_store stamp PATH write a line into PATH saying when: a marker for a run
*
+ * Nobody has to remember to save. `watch` registers with the kernel (kern.redwood.var_save), and
+ * on the orderly way back to the loader -- the run's time up, or "xl stop" -- the kernel asks it
+ * for the volume and waits, bounded, before it goes (bsd/dev/redwood_ufs.c,
+ * redwood_var_save_on_return). A save every SECS seconds is there as well but is not the default:
+ * one costs seconds of the run and a hundred-odd megabytes of flash, and while it runs the
+ * storage is read through the loader rather than through this kernel's own driver.
+ *
+ * One saver at a time: a write to the store is refused unless the writer holds the claim
+ * (kern.redwood.var_busy), so two of them cannot interleave into one slot. Even if they did, the
+ * slot being written is never the one holding the newest good archive, and a header is written
+ * last and over a zeroed one -- so the worst an interrupted or doubled save can do is lose itself.
+ *
* Two slots, written in turn: the one not holding the newest good archive is the one written, its
* header first zeroed, the archive next, the header last. So a save that does not finish -- the
* phone reset in the middle of it -- loses nothing: the other slot still holds the archive from
@@ -92,6 +106,81 @@ struct entry {
uint64_t size; /* a file's contents, a link's target, a hard link's first name */
};
+/* ---- the kernel's side of it: the boot-arg, the handshake, the claim ---- */
+
+#define VAR_SAVE_NONE 0
+#define VAR_SAVE_IDLE 1
+#define VAR_SAVE_WANTED 2
+#define VAR_SAVE_DONE 3
+
+/* Is NAME=VALUE among the kernel's arguments? (as init.c reads them) */
+static int
+boot_arg_is(const char *name, const char *value)
+{
+ static char args[1024];
+ size_t len = sizeof(args) - 1, n = strlen(name), v = strlen(value);
+
+ if (sysctlbyname("kern.bootargs", args, &len, NULL, 0) != 0) {
+ return 0;
+ }
+ args[len < sizeof(args) ? len : sizeof(args) - 1] = 0;
+ for (char *p = args; *p;) {
+ while (*p == ' ') {
+ p++;
+ }
+ if (strncmp(p, name, n) == 0 && p[n] == '=' && strncmp(p + n + 1, value, v) == 0 &&
+ (p[n + 1 + v] == 0 || p[n + 1 + v] == ' ')) {
+ return 1;
+ }
+ while (*p && *p != ' ') {
+ p++;
+ }
+ }
+ return 0;
+}
+
+static long
+parse_number(const char *text)
+{
+ long value = 0;
+
+ for (; *text >= '0' && *text <= '9'; text++) {
+ value = value * 10 + (*text - '0');
+ }
+ return value;
+}
+
+static int
+sysctl_int(const char *name, int *value)
+{
+ size_t len = sizeof(*value);
+
+ return sysctlbyname(name, value, &len, NULL, 0);
+}
+
+static int
+sysctl_set_int(const char *name, int value)
+{
+ return sysctlbyname(name, NULL, NULL, &value, sizeof(value));
+}
+
+/* The claim on the store: nothing may be written without it, and only one program may hold it. */
+static int
+claim(void)
+{
+ if (sysctl_set_int("kern.redwood.var_busy", 1) != 0) {
+ printf("var_store: the store is claimed by another saver (%s)\n", strerror(errno));
+ return 0;
+ }
+ return 1;
+}
+
+static void
+release(void)
+{
+ (void)sysctl_set_int("kern.redwood.var_busy", 0);
+}
+
/* ---- checksum (the same one the loader reads a write back with) ---- */
static uint32_t crc_table[256];
@@ -658,6 +747,9 @@ save(void)
}
stream_start(slot, 0);
entries = 0;
+ /* a watcher saves more than once: nothing of the save before is carried into this one */
+ memset(links, 0, LINKS * sizeof(*links));
+ skipped_kinds = skipped_big = 0;
strcpy(path, ROOT);
if (!pack_tree("") || !out_flush(1)) {
return 0;
@@ -697,6 +789,16 @@ save(void)
/* ---- unpacking ---- */
static uint64_t made_dirs, made_files, made_links, made_bytes, short_files, lost_bytes;
+/*
+ * A belt, after the braces: the memory filesystem no longer answers ENOSPC for want of a page
+ * (bsd/miscfs/ramfs, ramfs_grow_slots now takes an array found without the mount lock), so a short
+ * write here means the volume really is full. Waiting a little and trying again still earns its
+ * place -- the pageout path cannot wait for an array either, and it may be what answered -- but
+ * the whole restore spends at most RETRY_MS on it, so a volume that is truly full does not turn
+ * the boot into a crawl.
+ */
+#define RETRY_MS 2000
+static uint64_t retry_ms_left = RETRY_MS;
static int
whole_path(const char *relative, uint32_t len)
@@ -824,14 +926,12 @@ unpack_one(int make)
*/
while (out_fd >= 0 && done < want) {
ssize_t wrote = write(out_fd, piece + done, (size_t)(want - done));
- static int tries;
if (wrote > 0) {
done += (uint64_t)wrote;
- tries = 0;
continue;
}
- if (++tries > 200) {
+ if (retry_ms_left < 5) {
printf("var_store: %s: %llu of %llu bytes, no more would go in: %s\n",
path, (unsigned long long)(in_file + done),
(unsigned long long)e.size, strerror(errno));
@@ -839,10 +939,10 @@ unpack_one(int make)
lost_bytes += e.size - in_file - done;
close(out_fd);
out_fd = -1;
- tries = 0;
break;
}
usleep(5000);
+ retry_ms_left -= 5;
}
in_file += done;
left -= want;
@@ -1032,6 +1132,84 @@ erase(void)
return 1;
}
+/*
+ * Stay and save by itself. The kernel is told there is a saver, and then this waits: for the
+ * kernel to ask (the run is ending, and it waits for the answer before it goes), and, if seconds
+ * is given, for that long to pass between saves of its own accord. A save that fails is said so
+ * and the waiting goes on -- the one at the end of the run still gets its chance.
+ */
+static int
+save_once(void)
+{
+ int ok;
+
+ /*
+ * The device is opened for the save and closed again, not held: while a write is in flight
+ * this kernel's own transfer list is out of the controller, and it goes back in when the
+ * device is closed (bsd/dev/redwood_ufs.c, ufs_cclose). A watcher that kept the device open
+ * would leave iOS reading the flash through the loader for the rest of the run.
+ */
+ fd = open(DEVICE, O_RDWR, 0);
+ if (fd < 0) {
+ printf("var_store: %s: %s\n", DEVICE, strerror(errno));
+ return 0;
+ }
+ ok = claim() && save();
+ release();
+ close(fd);
+ fd = -1;
+ return ok;
+}
+
+static int
+watch(int seconds)
+{
+ uint64_t last = 0;
+ int asked = 0;
+
+ fd = open(DEVICE, O_RDWR, 0);
+ if (fd < 0) {
+ printf("var_store: %s cannot be written (%s): nothing would be saved\n", DEVICE,
+ strerror(errno));
+ return 0;
+ }
+ close(fd);
+ fd = -1;
+ if (sysctl_set_int("kern.redwood.var_save", VAR_SAVE_IDLE) != 0) {
+ printf("var_store: this kernel does not keep the data volume (%s)\n", strerror(errno));
+ return 0;
+ }
+ printf("var_store: watching: the volume is saved when the run ends%s\n",
+ seconds > 0 ? ", and in between" : "");
+ {
+ struct timeval tv = { 0, 0 };
+
+ gettimeofday(&tv, NULL);
+ last = (uint64_t)tv.tv_sec;
+ }
+ for (;;) {
+ struct timeval tv = { 0, 0 };
+ int state = VAR_SAVE_NONE;
+
+ if (sysctl_int("kern.redwood.var_save", &state) == 0 && state == VAR_SAVE_WANTED) {
+ asked = 1;
+ }
+ gettimeofday(&tv, NULL);
+ if (asked || (seconds > 0 && (uint64_t)tv.tv_sec - last >= (uint64_t)seconds)) {
+ if (!save_once()) {
+ printf("var_store: this save came to nothing; the store keeps what it had\n");
+ }
+ last = (uint64_t)tv.tv_sec;
+ if (asked) {
+ /* the kernel is waiting on this to go back to the loader */
+ (void)sysctl_set_int("kern.redwood.var_save", VAR_SAVE_DONE);
+ return 1;
+ }
+ }
+ usleep(100000);
+ }
+}
+
/* A line saying when, for a run to leave behind and the next one to find. */
static int
stamp(const char *where)
@@ -1091,6 +1269,13 @@ main(int argc, char **argv, char **envp)
printf("var_store: no memory for the buffers\n");
return 1;
}
+ if (strcmp(what, "watch") == 0) {
+ if (!boot_arg_is("var_store", "1")) {
+ printf("var_store: not asked to keep the data volume (var_store=1): nothing to watch\n");
+ return 0;
+ }
+ return watch(argc > 2 ? (int)parse_number(argv[2]) : 0) ? 0 : 1;
+ }
fd = open(DEVICE, strcmp(what, "load") == 0 || strcmp(what, "show") == 0 ? O_RDONLY : O_RDWR, 0);
if (fd < 0) {
printf("var_store: %s: %s (the kernel takes writes with ufs_write=1)\n", DEVICE,
@@ -1098,7 +1283,8 @@ main(int argc, char **argv, char **envp)
return 1;
}
if (strcmp(what, "save") == 0) {
- ok = save();
+ ok = claim() && save();
+ release();
} else if (strcmp(what, "load") == 0) {
ok = load();
} else if (strcmp(what, "erase") == 0) {
diff --git a/xnu/files/bsd/dev/redwood_ufs.c b/xnu/files/bsd/dev/redwood_ufs.c
index 7d9d2eb..62e06ee 100644
--- a/xnu/files/bsd/dev/redwood_ufs.c
+++ b/xnu/files/bsd/dev/redwood_ufs.c
@@ -172,6 +172,36 @@ static LCK_MTX_DECLARE(ufs_lock, &ufs_lck_grp);
/* One writer at a time; never held while the mailbox is, nor while user memory is touched. */
static LCK_MTX_DECLARE(ufs_write_mtx, &ufs_lck_grp);
+/*
+ * Saving the data volume without being asked (userland/src/var_store.c).
+ *
+ * The volume is packed and written by a program, not by this kernel, so the kernel's part is to
+ * say when and to let only one program at a time do it:
+ *
+ * kern.redwood.var_save the handshake. The saver writes VAR_SAVE_IDLE when it starts and is
+ * then the one that will be asked; on the orderly way back to the loader
+ * (loader_return_after expired, or "xl stop") the kernel writes
+ * VAR_SAVE_WANTED and waits, and the saver writes VAR_SAVE_DONE when the
+ * volume is in the flash. Nobody registered: nothing is asked and
+ * nothing waited for, so a run without a saver returns as it always did.
+ * A panic does not come this way at all -- there is no program left to
+ * ask -- and then the store simply keeps what it had.
+ * kern.redwood.var_busy the claim. A write to /dev/rufs_darwin is refused unless the writer
+ * holds it, so two savers cannot interleave into one slot. It is held by
+ * process number, and a claimant may take it from a process that is gone
+ * (a saver killed in the middle leaves no lock behind for the rest of the
+ * run). What the one that was killed half-wrote is not lost either: the
+ * slot it was writing is not the one holding the newest good archive.
+ */
+#define VAR_SAVE_NONE 0
+#define VAR_SAVE_IDLE 1
+#define VAR_SAVE_WANTED 2
+#define VAR_SAVE_DONE 3
+#define VAR_SAVE_MS 20000 /* at most this long on the way out (var_save_ms=N) */
+
+static volatile int ufs_var_save = VAR_SAVE_NONE;
+static int ufs_var_busy_pid;
+
/* ufs_write=1 and the one partition it is for: everything else answers EROFS as it always did */
static int ufs_write_part = -1;
static volatile bool ufs_write_busy; /* a write of ours has this kernel's list out: not a failure */
@@ -198,6 +228,71 @@ SYSCTL_QUAD(_kern_redwood, OID_AUTO, ufs_write_errors, CTLFLAG_RD | CTLFLAG_LOCK
SYSCTL_QUAD(_kern_redwood, OID_AUTO, ufs_write_refused, CTLFLAG_RD | CTLFLAG_LOCKED, &ufs_write_refused,
"writes refused here: not darwin, or not past what the build writes");
+/* The handshake: 1 to register as the saver, 3 to say the volume is in the flash. 2 is the kernel's. */
+static int
+ufs_sysctl_var_save(__unused struct sysctl_oid *oidp, __unused void *arg1, __unused int arg2, struct sysctl_req *req)
+{
+ int value = ufs_var_save, error;
+
+ error = sysctl_io_number(req, value, sizeof(value), &value, NULL);
+ if (error != 0 || req->newptr == 0) {
+ return error;
+ }
+ if (value == VAR_SAVE_IDLE && ufs_var_save != VAR_SAVE_WANTED) {
+ ufs_var_save = VAR_SAVE_IDLE; /* a saver is there; it is the one that will be asked */
+ } else if (value == VAR_SAVE_DONE && ufs_var_save == VAR_SAVE_WANTED) {
+ ufs_var_save = VAR_SAVE_DONE; /* and it has finished: the way out may go on */
+ } else if (value == VAR_SAVE_NONE) {
+ ufs_var_save = VAR_SAVE_NONE;
+ } else {
+ return EINVAL;
+ }
+ return 0;
+}
+
+SYSCTL_PROC(_kern_redwood, OID_AUTO, var_save, CTLTYPE_INT | CTLFLAG_RW | CTLFLAG_LOCKED, NULL, 0,
+ ufs_sysctl_var_save, "I", "the data volume's saver: 1 registers, 2 is the kernel asking, 3 is done");
+
+/* The claim on the store: 1 takes it, 0 gives it back; reading says which process holds it. */
+static int
+ufs_sysctl_var_busy(__unused struct sysctl_oid *oidp, __unused void *arg1, __unused int arg2, struct sysctl_req *req)
+{
+ int value = ufs_var_busy_pid, error, me = proc_selfpid();
+
+ error = sysctl_io_number(req, value, sizeof(value), &value, NULL);
+ if (error != 0 || req->newptr == 0) {
+ return error;
+ }
+ lck_mtx_lock(&ufs_write_mtx);
+ if (value == 1) {
+ if (ufs_var_busy_pid != 0 && ufs_var_busy_pid != me) {
+ proc_t held = proc_find(ufs_var_busy_pid);
+
+ if (held != NULL) {
+ proc_rele(held);
+ lck_mtx_unlock(&ufs_write_mtx);
+ return EBUSY; /* another saver has it and is alive: one at a time */
+ }
+ printf("var_store: the claim on the store was left by process %d, which is gone\n",
+ ufs_var_busy_pid);
+ }
+ ufs_var_busy_pid = me;
+ } else if (value == 0) {
+ if (ufs_var_busy_pid == me || ufs_var_busy_pid == 0) {
+ ufs_var_busy_pid = 0;
+ } else {
+ error = EPERM;
+ }
+ } else {
+ error = EINVAL;
+ }
+ lck_mtx_unlock(&ufs_write_mtx);
+ return error;
+}
+
+SYSCTL_PROC(_kern_redwood, OID_AUTO, var_busy, CTLTYPE_INT | CTLFLAG_RW | CTLFLAG_LOCKED, NULL, 0,
+ ufs_sysctl_var_busy, "I", "the claim a writer of /dev/rufs_darwin must hold: 1 takes it, 0 gives it back");
+
/*
* Native blocks [lba, lba + count) of partition part, one request of at most the buffer's size:
* read into dst, or, with src, written from there. Called with ufs_lock held.
@@ -735,6 +830,11 @@ ufs_cwrite(dev_t dev, struct uio *uio, __unused int ioflag)
if (!ufs_ready || part >= ufs_parts || ufs_write_part < 0 || part != (uint32_t)ufs_write_part) {
return EROFS;
}
+ if (ufs_var_busy_pid != proc_selfpid()) {
+ /* one writer of the store at a time, and it says so first (kern.redwood.var_busy) */
+ ufs_write_refused++;
+ return EPERM;
+ }
nbs = ufs_page->part[part].block_size;
if (uio_offset(uio) < 0 || (uint64_t)uio_offset(uio) % nbs != 0 || (uint64_t)uio_resid(uio) % nbs != 0) {
return EINVAL;
@@ -791,6 +891,37 @@ static const struct cdevsw ufs_cdevsw = {
.d_type = D_DISK,
};
+/*
+ * The orderly end of a run, from osfmk/arm64/loader_return.c, in the thread that is about to hand
+ * control back: ask the saver for the data volume and wait for it, so that a run need not have
+ * been told to save. Bounded, and only when a saver registered -- the loader goes on feeding the
+ * watchdog for a minute past the run's own budget, and the host waits a hundred seconds past it,
+ * so twenty seconds here is well inside both. Nothing else of the way out has happened yet: the
+ * GPU, the DSPs and this kernel's transfer list are still as the run had them.
+ */
+void
+redwood_var_save_on_return(void)
+{
+ uint32_t ms = VAR_SAVE_MS;
+ uint64_t waited = 0;
+
+ if (!ufs_ready || ufs_write_part < 0 || ufs_var_save != VAR_SAVE_IDLE) {
+ return;
+ }
+ PE_parse_boot_argn("var_save_ms", &ms, sizeof(ms));
+ printf("var_store: the run is ending: the data volume is asked for\n");
+ ufs_var_save = VAR_SAVE_WANTED;
+ while (ufs_var_save != VAR_SAVE_DONE && waited < ms) {
+ IOSleep(20);
+ waited += 20;
+ }
+ if (ufs_var_save == VAR_SAVE_DONE) {
+ printf("var_store: the data volume is in the flash (%llu ms)\n", waited);
+ } else {
+ printf("var_store: the saver did not answer in %u ms: the store keeps what it had\n", ms);
+ }
+}
+
/* ---- the check: the partitions there are images of, hashed as the loader hashes them ---- */
static const struct {
diff --git a/xnu/files/bsd/miscfs/ramfs/ramfs.c b/xnu/files/bsd/miscfs/ramfs/ramfs.c
index 6510077..8776736 100644
--- a/xnu/files/bsd/miscfs/ramfs/ramfs.c
+++ b/xnu/files/bsd/miscfs/ramfs/ramfs.c
@@ -358,43 +358,70 @@ ramfs_vget_locked(struct ramfs_mount *rm, struct ramfs_node *node, vnode_t dvp,
/* ------------------------------------------------------------------------ */
/* File contents */
-/* Make sure the page array has a slot for page `index`. The mount lock is held. */
+/* How many slots an array needs to hold page `index` of a file that has `have` of them. */
+static uint64_t
+ramfs_slots_wanted(uint64_t have, uint64_t index)
+{
+ uint64_t want = have ? have : 4;
+
+ while (want <= index) {
+ want *= 2;
+ }
+ return want;
+}
+
+/*
+ * Make sure the page array has a slot for page `index`. The mount lock is held, and nothing waits
+ * for memory under it -- a wait could end in the pageout daemon, which comes back here for the
+ * lock. So the array is the caller's to find room for, without the lock: *slots of *nslots
+ * entries, used and consumed when it is big enough. EAGAIN means only that: bring one of
+ * ramfs_slots_wanted() entries and come again; it is not ENOSPC, and a caller that turns it into
+ * one says a volume with room to spare is full. A caller that cannot wait either -- the pageout
+ * path -- brings none, and then this tries without waiting, as it always did.
+ */
static int
-ramfs_grow_slots(struct ramfs_node *node, uint64_t index)
+ramfs_grow_slots(struct ramfs_node *node, uint64_t index, void ***slots, uint64_t *nslots)
{
- uint64_t want = node->rn_nslots ? node->rn_nslots : 4;
- void **slots;
+ void **grown;
+ uint64_t grown_n;
if (index < node->rn_nslots) {
return 0;
}
- while (want <= index) {
- want *= 2;
- }
- slots = kalloc_type(void *, want, Z_NOWAIT | Z_ZERO);
- if (slots == NULL) {
- return ENOSPC;
+ if (*slots != NULL && *nslots > index && *nslots >= node->rn_nslots) {
+ grown = *slots;
+ grown_n = *nslots;
+ *slots = NULL;
+ } else {
+ grown_n = ramfs_slots_wanted(node->rn_nslots, index);
+ grown = kalloc_type(void *, grown_n, Z_NOWAIT | Z_ZERO);
+ if (grown == NULL) {
+ return EAGAIN;
+ }
}
if (node->rn_pages != NULL) {
- memcpy(slots, node->rn_pages, node->rn_nslots * sizeof(void *));
+ memcpy(grown, node->rn_pages, node->rn_nslots * sizeof(void *));
kfree_type(void *, node->rn_nslots, node->rn_pages);
}
- node->rn_pages = slots;
- node->rn_nslots = want;
+ node->rn_pages = grown;
+ node->rn_nslots = grown_n;
return 0;
}
/*
* Copy len bytes (within one page) into the file at offset off, extending
* the file if needed. *spare is a zeroed page the caller allocated without
- * the lock; it is consumed if the slot was empty. The mount lock is held.
+ * the lock; it is consumed if the slot was empty. *slots is the page array
+ * the caller allocated the same way (see ramfs_grow_slots). The mount lock
+ * is held.
*/
static int
-ramfs_store(struct ramfs_node *node, uint64_t off, const void *src, size_t len, void **spare)
+ramfs_store(struct ramfs_node *node, uint64_t off, const void *src, size_t len, void **spare,
+ void ***slots, uint64_t *nslots)
{
struct ramfs_mount *rm = node->rn_mount;
uint64_t index = off / PAGE_SIZE;
- int error = ramfs_grow_slots(node, index);
+ int error = ramfs_grow_slots(node, index, slots, nslots);
if (error) {
return error;
@@ -990,7 +1017,8 @@ ramfs_write(struct vnop_write_args *ap)
struct uio *uio = ap->a_uio;
struct ramfs_node *node = RAMFS_NODE(vp);
struct ramfs_mount *rm = node->rn_mount;
- void *spare = NULL;
+ void *spare = NULL, **slots = NULL;
+ uint64_t nslots = 0;
char *bounce;
uint64_t start, old_size;
int error = 0;
@@ -1026,9 +1054,32 @@ ramfs_write(struct vnop_write_args *ap)
if (spare == NULL) {
spare = kalloc_data(PAGE_SIZE, Z_WAITOK | Z_ZERO);
}
- lck_mtx_lock(&rm->rm_lock);
- error = ramfs_store(node, off, bounce, len, &spare);
- lck_mtx_unlock(&rm->rm_lock);
+ /*
+ * EAGAIN says the page array has to grow and that room for it is this side's to find,
+ * which is done here, where waiting is allowed. rn_nslots is read without the lock as a
+ * hint only; another writer growing the same file at the same time makes the hint too
+ * small, and then this asks again with a fresh one.
+ */
+ for (int again = 0; again < 4; again++) {
+ lck_mtx_lock(&rm->rm_lock);
+ error = ramfs_store(node, off, bounce, len, &spare, &slots, &nslots);
+ lck_mtx_unlock(&rm->rm_lock);
+ if (error != EAGAIN) {
+ break;
+ }
+ if (slots != NULL) {
+ kfree_type(void *, nslots, slots);
+ }
+ nslots = ramfs_slots_wanted(node->rn_nslots, off / PAGE_SIZE);
+ slots = kalloc_type(void *, nslots, Z_WAITOK | Z_ZERO);
+ if (slots == NULL) {
+ nslots = 0;
+ break;
+ }
+ }
+ if (error == EAGAIN) {
+ error = ENOSPC; /* it never got the array: out of memory, not out of volume */
+ }
if (error) {
/* nothing of this piece was stored: hand it back to the caller's count */
uio_setoffset(uio, (off_t)off);
@@ -1040,6 +1091,9 @@ ramfs_write(struct vnop_write_args *ap)
if (spare != NULL) {
kfree_data(spare, PAGE_SIZE);
}
+ if (slots != NULL) {
+ kfree_type(void *, nslots, slots);
+ }
if ((uint64_t)uio_offset(uio) > start) {
uint64_t new_size;
@@ -1318,9 +1372,13 @@ ramfs_pageout(struct vnop_pageout_args *ap)
for (upl_size_t done = 0; done < size && off + done < node->rn_size && !error; done += PAGE_SIZE) {
size_t len = (size_t)MIN((uint64_t)PAGE_SIZE, node->rn_size - (off + done));
/* this may be the pageout daemon looking for memory: never wait for more of it here */
- void *spare = kalloc_data(PAGE_SIZE, Z_NOWAIT | Z_ZERO);
+ void *spare = kalloc_data(PAGE_SIZE, Z_NOWAIT | Z_ZERO), **slots = NULL;
+ uint64_t nslots = 0;
- error = ramfs_store(node, off + done, (const void *)(addr + done), len, &spare);
+ error = ramfs_store(node, off + done, (const void *)(addr + done), len, &spare, &slots, &nslots);
+ if (error == EAGAIN) {
+ error = ENOSPC; /* nothing here may wait for the array either: the pages stay dirty */
+ }
if (spare != NULL) {
kfree_data(spare, PAGE_SIZE);
}
diff --git a/xnu/files/osfmk/arm64/loader_return.c b/xnu/files/osfmk/arm64/loader_return.c
index e7e2dbb..a058bb5 100644
--- a/xnu/files/osfmk/arm64/loader_return.c
+++ b/xnu/files/osfmk/arm64/loader_return.c
@@ -57,6 +57,7 @@ extern void redwood_torch_quiesce(void);
extern void redwood_camera_quiesce(void);
extern void redwood_mic_quiesce(void);
extern void redwood_ufs_quiesce(void);
+extern void redwood_var_save_on_return(void); /* bsd/dev/redwood_ufs.c: the data volume into the flash */
void loader_checkpoint(uint64_t n);
/* Also referenced from the early exception vector in start.s. */
@@ -435,6 +436,7 @@ loader_return_timer(__unused thread_call_param_t p0, __unused thread_call_param_
}
PE_parse_boot_argn("return_panic", &as_panic, sizeof(as_panic));
if (!as_panic) {
+ redwood_var_save_on_return(); /* the data volume kept, while the system is still whole */
redwood_adreno_stop(); /* a GPU kept for the system is given back while there is a thread to wait in */
redwood_wifi_stop(); /* and the Wi-Fi DSP, if it runs, shut down */
redwood_adsp_stop(); /* and the audio DSP */
@@ -480,6 +482,7 @@ loader_stop_poll(__unused thread_call_param_t p0, __unused thread_call_param_t p
for (const char *c = banner; *c; c++) {
loader_log_putc(*c);
}
+ redwood_var_save_on_return();
redwood_adreno_stop();
redwood_wifi_stop();
redwood_adsp_stop();