Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.
C
Sound, step 1: the audio DSP started through TrustZone, and linked (GLINK, QRTR)
vcvkk25c19a686dec
diff --git a/README.md b/README.mdindex 255462f..718d13c 100644--- a/README.md+++ b/README.md@@ -231,6 +231,7 @@ XL_SECS=900 XL_CMDLINE="touchscreen=1 cpufreq=1 bootcpu=7 cpus=4" ./xl run #| `wifi_file=0` | с `wifi=1`: сеть из `fw-wifi/network` при старте не подключается, радио ждёт, пока сеть выберут в Настройках (образ с `WIFID=1`) || `wifi_irq=0`, `wifi_ba=0`, `wifi_wide=0` | с `wifi=1`: без прерываний радио (его кольца опрашиваются раз в миллисекунду, как раньше), без блочных подтверждений (точка доступа шлёт кадры по одному — скорость впятеро ниже), без широкого канала (только 20 МГц) || `wifi_step=N` | то же по шагам, как это выяснялось: первые N шагов, и ядро ждёт, пока они пройдут. 1–5 только читают (общая память процессоров, почтовый ящик процессора питания, RPMh, шины питания, выводы), 6–11 кладут прошивку процессора Wi-Fi в его память, заводят в общей памяти нашу половину его сигнальных битов (SMP2P) и запускают его через TrustZone, 12–14 поднимают связь с ним (GLINK, канал IPCRTR) и QRTR поверх неё: процессор Wi-Fi объявляет свои службы, среди них службу прошивки Wi-Fi; 15 — разговор с этой службой, какой драйвер ведёт до того, как тронет радио (нужны описания сообщений из ath11k: `tools/mk-wlfw.sh`, затем пересобрать ядро): прошивка принимает сообщения ath11k, файл платы и базу регионов телефона, с `wifi_cal=1` калибруется и через 7 секунд говорит «готова»; с `wifi_mission=1` прошивка получает настройку и переходит в рабочий режим; 16 — чтение нескольких регистров радио (`wifi_radio_read=1`): в рабочем режиме они отвечают, вне его чтение кончается ошибкой шины, и шаг его не делает (`adb pull` файлов `wpss.mdt`, `wpss.b02`…`wpss.b07` из `/vendor/firmware_mnt/image` в каталог `fw-wifi/`; в git они не попадают, `userland/build.sh` кладёт их в малый образ). Каждый шаг пишет в журнал, что собирается тронуть и что прочёл. `wifi_keep=1` оставляет процессор Wi-Fi и связь с ним работать, пока идёт сеанс. С `wifi_lx=1` шаг 15 ведёт сам драйвер ath11k из Linux (см. «Драйвер Wi-Fi» ниже; ядро собирается после `tools/mk-ath11k.sh`): его `qmi.c` проходит весь разговор с прошивкой до рабочего режима, со своей настройкой и списком теневых регистров, а регистры радио не трогает. С `wifi_lx=2 wifi_smmu=2` потоку радио в SMMU даётся контекст с нашими таблицами страниц (`wifi_smmu=2` в одиночку — только это, с возвратом как было), `ce.c` и `hal.c` драйвера заводят в регистрах радио кольца его «копировальных машин», `htc.c` подключает по ним службу WMI, и прошивка по WMI рассказывает о радио (одно, две антенные цепи, 2,4 и 5 ГГц) и через четверть секунды после запуска шины сообщает, что готова к командам («unified ready»); собственного адреса у прошивки нет (нули). Прерываний при этом нет: кольца опрашиваются раз в миллисекунду. С `wifi_scan=1` драйвер после этого регистрирует радио, заводит интерфейс в режиме станции и просит прошивку просканировать эфир — только слушая, в эфир ничего не уходит: 26 каналов 2,4 и 5 ГГц за четыре секунды, и в журнале появляются точки доступа вокруг с каналами и уровнем сигнала (имена и адреса чужих сетей — только в журнале запуска на вашей машине). Адрес радио у прошивки свой не записан: он лежит одной строкой в `fw-wifi/mac` (в git не попадает), и `xl run` передаёт его ядру параметром `wifi_mac`. С `wifi_join=1` драйвер подключается к сети из `fw-wifi/network` (WPA2 с паролем): находит её точку доступа, проходит аутентификацию, ассоциацию и четырёхстороннее рукопожатие, отдаёт ключи прошивке и становится тем, что стоит за `en0`; с `wifi_keep=1` он остаётся работать, пока идёт сеанс, без него — сворачивается, когда шаги кончились (`wifi_hold=S` — на S секунд позже). Остановленный без спроса (ядро, которое до этого не дошло) второй раз без перезагрузки телефона не запускается — такой запуск сбрасывал телефон, и драйвер от него сам отказывается; шаги и остальные параметры описаны в начале `bsd/dev/redwood_wifi.c` |+| `adsp=1` | аудиопроцессор телефона (ADSP: звук, а у Xiaomi ещё и зарядка), шаг 1 звука: прошивка `adsp.mdt` и `adsp.b02`…`adsp.b36` с разделов телефона (`/vendor/firmware_mnt/image`, в каталог `fw-audio/`, в git не попадает; в малый образ её кладёт `ADSP=1 userland/build.sh`, и только без `GL=1` — вместе они не помещаются в 64 МБ, что берёт загрузчик) кладётся в его память, RPMh просят о его шинах питания LCX и LMX, пока он не попросит о них сам, TrustZone его запускает, через 41 мс он говорит «готов», поднимается GLINK (канал IPCRTR) и QRTR поверх него: процессор — узел 5 — объявляет 12 служб и открывает свои каналы, среди них `apr_audio_svc`, через который потом пойдёт звук. Рядом с загрузкой системы и рядом с `wifi=1`; в конце сеанса его просят остановиться и выключают. `adsp_step=N` — то же по шагам (первые N из 10, ядро ждёт), подробности в начале `bsd/dev/redwood_adsp.c` |Несколько ядер на телефоне проверены (2, 4 и 7 ядер запускаются и чисто возвращаются в загрузчик). Лучшийвариант — **`cpus=4`**: четыре больших ядра (7, 6, 5, 4). На одном и том же сценарии: 929 кадров вместо 671,diff --git a/userland/build.sh b/userland/build.shindex 0a7c267..0e74514 100755--- a/userland/build.sh+++ b/userland/build.sh@@ -30,6 +30,9 @@# and nothing changes (src/inject.c, opengl_es()).# GL=test: only the library and bin/gl_test (a triangle, src/gl_test.c;# RC_JOBS=rc.jobs.gltest runs it); the window server stays as it is.+# ADSP=1 with the audio DSP's firmware from ../fw-audio in fw/ (20 MB; for the+# kernel's adsp=1 / adsp_step=N, bsd/dev/redwood_adsp.c). Not beside GL=1:+# the two do not fit the 64 MB the loader takes with the kernel.# system.packfs the system image, the root filesystem (second RAM disk, /dev/md1): the part of# iOS 16.5 chosen by ../ios165/select-system.py at its real paths, the dyld# shared cache, and src/stage0.c as /sbin/launchd (the real one is@@ -207,6 +210,13 @@ if [ -f ../fw-wifi/wpss.mdt ]; thenrm -f root/fw/network[ -f ../fw-wifi/network ] && install -m 0600 ../fw-wifi/network root/fw/networkfi+# ... and with ADSP=1 the image of the audio DSP, from the same place into ../fw-audio (bsd/dev/redwood_adsp.c):+# adsp.mdt and adsp.b02 to adsp.b36, 20 MB. Only when asked for: beside GL=1 the payload would be more than the 64 MB+# the loader takes.+if [ -n "$ADSP" ] && [ -f ../fw-audio/adsp.mdt ]; then+ mkdir -p root/fw+ install -m 0644 ../fw-audio/adsp.mdt ../fw-audio/adsp.b0[2-9] ../fw-audio/adsp.b[1-9][0-9] root/fw/+fi../mkpackfs.py -o root.packfs root# --- the system image -----------------------------------------------------------------------------diff --git a/xnu/files/bsd/dev/redwood_adsp.c b/xnu/files/bsd/dev/redwood_adsp.cnew file mode 100644index 0000000..b4e278a--- /dev/null+++ b/xnu/files/bsd/dev/redwood_adsp.c@@ -0,0 +1,2607 @@+/*+ * The audio DSP of the POCO X5 Pro (redwood), started, and spoken to: step 1 of sound. Every+ * address below belongs to this phone, and nothing here is touched unless a boot-arg asks for it.+ *+ * The SoC's audio (LPASS: its clocks, its I2S and SoundWire, the codec's macros) does not belong+ * to the application processor. It is the audio DSP's -- a Hexagon of its own inside LPASS,+ * Android's "adsp" -- which stays in reset until somebody loads it, exactly as the Wi-Fi's does+ * (redwood_wifi.c): the phone's own signed image, adsp.mdt and adsp.b02 to adsp.b36 from+ * /vendor/firmware_mnt/image of its Android, goes into the 64 MB reserved for it, TrustZone is+ * asked to authenticate and start it, and from there everything is conversation through the+ * shared memory: SMP2P for "up", "given up", "stopped"; GLINK for channels; QRTR on the channel+ * "IPCRTR" for the services the DSP offers. (Reading an LPASS clock register from here ends the+ * kernel with an external abort: the DSP owns that block, and talks about it only this way.)+ * What audio needs of it later -- the audio services, the codec, the speaker's path -- is for the+ * steps after this one; this file only brings the DSP up and the link with it.+ *+ * The way is the Wi-Fi DSP's, taken over from redwood_wifi.c with the audio DSP's numbers: its+ * comments say why each thing is done so. What is different is said here:+ * - processor 2 (not 13) in the shared memory; its SMP2P items 443 (its) and 429 (ours), its+ * last words in item 423; GLINK's three items in our partition with it+ * - its doorbells are IPCC client 3 (LPASS): signal 0 GLINK, signal 2 SMP2P+ * - TrustZone knows it as image 1; its memory is 0x86700000, 64 MB (dt: qcom,lpass@3700000,+ * reserved-memory adsp@86700000), every byte of which the image takes+ * - the always-on processor is told of it as "adsp"+ * - the Wi-Fi DSP may be running at the same time (wifi=1): the mailbox to the always-on+ * processor and the lock for allocating in the shared memory are the two things both use,+ * each behind a lock both take (redwood_wifi.c); the doorbells are only counted here, and+ * only the audio DSP's are cleared from here+ * The code is a copy, not a shared library, for now: step 1 is not to change what the Wi-Fi does.+ *+ * Boot-arg adsp_step=N performs steps 1 to N and stops; as with wifi_step, a step says what it is+ * about to touch before it touches it, and a step that does not find what it needs ends the+ * sequence. adsp=1 is all of them, on a thread beside the system's start.+ *+ * 1 reads the shared memory: our partition with the DSP, its items, whether it has run+ * since the phone was started (its SMP2P item is there if so), its last words+ * 2 reads the firmware in the small image: is it whole, does it fit the reserved memory+ * 3 writes the always-on processor's mailbox open, unless it is (the Wi-Fi may have opened it)+ * 4 asks TrustZone to take the image's header (image 1) and where the image will be; a+ * DSP that a kernel before this one left running is asked to stop and shut down first+ * 5 writes the image into the DSP's memory, and reads it back+ * 6 writes our SMP2P item (429) into the shared memory, and our two marks of GLINK to zero+ * 7 asks RPMh for the DSP's rails, writes "adsp: on" to the always-on processor, asks+ * TrustZone to authenticate and start the DSP, and watches for its first words+ * (answering in our item what SMP2P has us answer)+ * 8 reads GLINK's queues+ * 9 writes our side of GLINK: the versions, the channel "IPCRTR" opened both ways, our intents;+ * from here a thread of ours listens to the DSP+ * 10 talks QRTR on that channel: hello both ways, and the services the DSP tells of, written+ * down for adsp_listen seconds (default 3)+ *+ * What is started is stopped: when the steps are over (unless adsp_keep=1, and then when the run+ * ends), and on a panic's way back to the loader. The DSP is asked first (bit 0 of our entry+ * "master-kernel"), then TrustZone shuts it down, then the always-on processor hears "adsp: off".+ * A DSP that a kernel before this one left running is found by its item and treated the same way+ * before this one is started; one that was shut down without having said it stopped is not+ * started again unless adsp_again=1 (the Wi-Fi DSP reset the phone, twice, when that was tried).+ *+ * More boot-args: adsp_new=K (steps K on have not been seen to work, and wait adsp_pause=MS after+ * each announcement, default 2200: what the loader needs to note the end of the log), adsp_wait=S+ * (how long step 7 watches for "up", default 10), adsp_keep=1, adsp_again=1, adsp_return=1 (back+ * to the loader when the steps are over).+ *+ * Nothing here writes a clock or changes a pin. One request goes to RPMh, as Linux makes it: the+ * DSP's two rails, LCX and LMX, at level 384 while it boots, given back when it says it has asked+ * for its own (see rails()).+ *+ * Addresses are from the phone's own device tree (qcom,lpass@3700000, qcom,smp2p-adsp,+ * qcom,glink/adsp, qcom,ipcc@408000, power-controller@c300000, reserved-memory), layouts from+ * Linux (see redwood_wifi.c for the files).+ */+#include <kern/clock.h>+#include <kern/kalloc.h>+#include <kern/locks.h>+#include <kern/thread.h>+#include <libkern/libkern.h>+#include <mach/mach_types.h>+#include <mach/vm_param.h>+#include <vm/memory_types.h>+#include <miscfs/packfs/packfs.h>+#include <pexpert/pexpert.h>+#include <sys/errno.h>+#include <sys/stat.h>+#include <sys/systm.h>+#include <vm/vm_kern.h>++extern vm_offset_t ml_io_map(vm_offset_t phys_addr, vm_size_t size);+extern vm_offset_t ml_io_map_unmappable(vm_offset_t phys_addr, vm_size_t size, unsigned int flags);+extern void ml_io_unmap(vm_offset_t addr, vm_size_t sz);+extern uint64_t kvtophys(vm_offset_t va);+extern void loader_return(void);+extern kern_return_t thread_terminate(thread_t thread);+extern vm_offset_t pe_arm_get_soc_base_phys(void);+extern dev_t mdevlookup(int devid);+extern int mdevgetkernelrange(dev_t dev, vm_offset_t *base, uint64_t *size);++/* What the Wi-Fi's DSP and this one both use (redwood_wifi.c). */+extern lck_mtx_t redwood_aop_mailbox_lock;+extern lck_mtx_t redwood_smem_alloc_lock;++static LCK_GRP_DECLARE(adsp_lock_group, "redwood-adsp");++#define STEPS 10+#define REDWOOD_SOC_BASE 0x00400000ul /* where the loader says the phone's peripherals begin */++static uint32_t last_step, pause_ms = 2200, listen_seconds = 3;+static uint32_t step; /* the one being performed */+static uint32_t new_from; /* the first of the steps that have not been seen to work: they wait */+static char last_write[96] = "nothing written";+static bool leaving; /* on the way back to the loader after a panic: no lock is taken, nothing sleeps */++/*+ * Every line goes to the log the loader reads from another core as well (as in redwood_wifi.c);+ * with adsp=1 only the few that say how it went (tell()), the rest to the kernel's own log.+ */+extern void loader_log_putc(char c);++static LCK_MTX_DECLARE(say_lock, &adsp_lock_group);+static bool quiet;++static void+say_line(const char *line, bool aloud)+{+ if (leaving) {+ for (const char *c = line; *c != 0; c++) {+ loader_log_putc(*c);+ }+ return;+ }+ lck_mtx_lock(&say_lock);+ if (!quiet || aloud) {+ for (const char *c = line; *c != 0; c++) {+ loader_log_putc(*c);+ }+ }+ printf("%s", line);+ lck_mtx_unlock(&say_lock);+}++static int say(const char *format, ...) __printflike(1, 2);+static int tell(const char *format, ...) __printflike(1, 2);++static int+say(const char *format, ...)+{+ char line[400];+ va_list list;+ int length;++ va_start(list, format);+ length = vsnprintf(line, sizeof(line), format, list);+ va_end(list);+ say_line(line, false);+ return length;+}++static int+tell(const char *format, ...)+{+ char line[400];+ va_list list;+ int length;++ va_start(list, format);+ length = vsnprintf(line, sizeof(line), format, list);+ va_end(list);+ say_line(line, true);+ return length;+}+#define printf say++static void+sleep_us(uint32_t us)+{+ delay_for_interval(us, NSEC_PER_USEC);+}++static void+sleep_ms(uint32_t ms)+{+ delay_for_interval(ms, NSEC_PER_MSEC);+}++static uint64_t+now_us(void)+{+ uint64_t nanoseconds;++ absolutetime_to_nanoseconds(mach_absolute_time(), &nanoseconds);+ return nanoseconds / 1000;+}++/* Until every write to memory so far has been carried out. */+static void+settle(void)+{+ __asm__ volatile ("dsb sy");+}++static bool+option(const char *name)+{+ uint32_t value = 0;++ return PE_parse_boot_argn(name, &value, sizeof(value)) && value != 0;+}++/* ---- register access (as in redwood_wifi.c) ------------------------------------------------------- */++static uint32_t __attribute__((noinline))+rd(vm_offset_t address)+{+ uint32_t value;++ __asm__ volatile ("ldr %w0, [%1]" : "=r" (value) : "r" (address) : "memory");+ return value;+}++static void __attribute__((noinline))+wr(vm_offset_t address, uint32_t value)+{+ __asm__ volatile ("str %w0, [%1]" : : "r" (value), "r" (address) : "memory");+}++struct block {+ const char *name;+ uint32_t phys, size;+ vm_offset_t base;+};++static void+need(struct block *b)+{+ if (b->base == 0) {+ b->base = ml_io_map(b->phys, b->size);+ }+}++/* Time for the loader to note what has just been said, where it is not yet known to be harmless. */+static void+hold(void)+{+ if (step >= new_from && pause_ms && !leaving) {+ sleep_ms(pause_ms);+ }+}++static void+announce(const char *what)+{+ printf("adsp: step %u: %s\n", step, what);+ hold();+}++static uint32_t+get(struct block *b, const char *name, uint32_t offset)+{+ uint32_t value;++ need(b);+ value = rd(b->base + offset);+ printf("adsp: %u: %s %s (%#x) = %#x\n", step, b->name, name, b->phys + offset, value);+ return value;+}++static void+put(struct block *b, const char *name, uint32_t offset, uint32_t value)+{+ need(b);+ printf("adsp: %u: about to write %s %s at %#x: %#x\n", step, b->name, name, b->phys + offset, value);+ snprintf(last_write, sizeof(last_write), "step %u wrote %#x to %s %s at %#x", step, value, b->name, name, b->phys + offset);+ wr(b->base + offset, value);+}++/* ---- the shared memory (redwood_wifi.c, step 1) --------------------------------------------------- */++#define SMEM_PHYS 0x80900000UL+#define SMEM_SIZE 0x00200000u+#define SMEM_US 0+#define SMEM_ADSP 2 /* dt: qcom,remote-pid of qcom,smp2p-adsp and qcom,glink/adsp */+#define SMEM_GLOBAL 0xfffe+#define SMEM_CANARY 0xa5a5++struct smem_table_entry {+ uint32_t offset, size, flags;+ uint16_t host0, host1;+ uint32_t cacheline, reserved[7];+};+struct smem_table {+ uint8_t magic[4];+ uint32_t version, count, reserved[5];+};+struct smem_partition {+ uint8_t magic[4];+ uint16_t host0, host1;+ uint32_t size, free_uncached, free_cached, reserved[3];+};+struct smem_entry {+ uint16_t canary, item;+ uint32_t size;+ uint16_t padding_data, padding_header;+ uint32_t reserved;+};+struct smem_item {+ uint32_t item, offset, size;+};++/*+ * The DSP's items (dt: qcom,smem of qcom,smp2p-adsp, <443 429>: the item the DSP writes first, then+ * ours -- the Wi-Fi's <617 616> are in the same order; taken the other way round, the DSP made+ * our item its own and never said "up" --, and qcom,smem-id of qcom,lpass@3700000).+ */+#define ITEM_SMP2P_OURS 429+#define ITEM_SMP2P_THEIRS 443+#define ITEM_CRASH_REASON 423++static const volatile uint8_t *smem;++static bool+smem_peek(uint64_t offset, void *to, uint32_t size)+{+ if (smem == NULL) {+ smem = (const volatile uint8_t *)ml_io_map_unmappable(SMEM_PHYS, SMEM_SIZE, VM_WIMG_WCOMB);+ }+ if (offset + size > SMEM_SIZE) {+ return false;+ }+ for (uint32_t i = 0; i < size; i++) {+ ((uint8_t *)to)[i] = smem[offset + i];+ }+ return true;+}++static void+smem_poke(uint32_t offset, uint32_t value)+{+ wr((vm_offset_t)smem + offset, value);+}++static uint32_t+smem_word(uint32_t offset)+{+ return rd((vm_offset_t)smem + offset);+}++static uint32_t+smem_partition(uint16_t one, uint16_t other, uint32_t *cacheline, uint32_t *size)+{+ struct smem_table table;+ struct smem_table_entry entry;+ struct smem_partition header;++ if (!smem_peek(SMEM_SIZE - 0x1000, &table, sizeof(table)) || memcmp(table.magic, "$TOC", 4) != 0 || table.version != 1 ||+ table.count > 64) {+ return 0;+ }+ for (uint32_t i = 0; i < table.count; i++) {+ if (!smem_peek(SMEM_SIZE - 0x1000 + sizeof(table) + i * sizeof(entry), &entry, sizeof(entry)) || entry.offset == 0 ||+ entry.size == 0) {+ continue;+ }+ if (!((entry.host0 == one && entry.host1 == other) || (entry.host0 == other && entry.host1 == one))) {+ continue;+ }+ if ((uint64_t)entry.offset + entry.size > SMEM_SIZE - 0x1000 || !smem_peek(entry.offset, &header, sizeof(header)) ||+ memcmp(header.magic, "$PRT", 4) != 0 || header.size != entry.size) {+ return 0;+ }+ *cacheline = entry.cacheline;+ *size = entry.size;+ return entry.offset;+ }+ return 0;+}++static unsigned+smem_items(uint32_t partition, uint32_t cacheline, struct smem_item *found, unsigned max)+{+ struct smem_partition header;+ struct smem_entry entry;+ uint32_t at, step_back;+ unsigned count = 0;++ if (partition == 0 || !smem_peek(partition, &header, sizeof(header)) || header.free_uncached > header.size ||+ header.free_cached > header.size) {+ return 0;+ }+ for (at = sizeof(header); at + sizeof(entry) <= header.free_uncached;) {+ if (!smem_peek(partition + at, &entry, sizeof(entry)) || entry.canary != SMEM_CANARY || entry.size > header.size) {+ break;+ }+ if (count < max) {+ found[count].item = entry.item;+ found[count].offset = partition + at + sizeof(entry) + entry.padding_header;+ found[count].size = entry.size - entry.padding_data;+ }+ count++;+ at += sizeof(entry) + entry.padding_header + entry.size;+ }+ if (cacheline == 0 || (cacheline & (cacheline - 1))) {+ return count;+ }+ step_back = ((uint32_t)sizeof(entry) + cacheline - 1) & ~(cacheline - 1);+ for (at = header.size - step_back; at > header.free_cached && at < header.size;) {+ if (!smem_peek(partition + at, &entry, sizeof(entry)) || entry.canary != SMEM_CANARY || entry.size > at) {+ break;+ }+ if (count < max) {+ found[count].item = entry.item;+ found[count].offset = partition + at - entry.size;+ found[count].size = entry.size - entry.padding_data;+ }+ count++;+ if (entry.size + step_back > at) {+ break;+ }+ at -= entry.size + step_back;+ }+ return count;+}++/* An item the DSP and we share: in the partition of the two, or else in the global one. Where it is; 0: nowhere. */+static uint32_t+smem_item(uint32_t item, uint32_t *size)+{+ static struct smem_item found[256];+ static const uint16_t hosts[2][2] = { { SMEM_US, SMEM_ADSP }, { SMEM_GLOBAL, SMEM_GLOBAL } };++ for (unsigned h = 0; h < 2; h++) {+ uint32_t cacheline = 0, partition_size = 0;+ uint32_t partition = smem_partition(hosts[h][0], hosts[h][1], &cacheline, &partition_size);+ unsigned count = smem_items(partition, cacheline, found, 256);++ for (unsigned i = 0; i < count && i < 256; i++) {+ if (found[i].item == item) {+ *size = found[i].size;+ return found[i].offset;+ }+ }+ }+ return 0;+}++static void+show_partition(void)+{+ static struct smem_item found[256];+ struct smem_partition header;+ uint32_t cacheline = 0, size = 0, partition = smem_partition(SMEM_US, SMEM_ADSP, &cacheline, &size);+ unsigned count;+ char line[320];++ if (partition == 0 || !smem_peek(partition, &header, sizeof(header))) {+ printf("adsp: %u: there is no partition for us and the audio DSP (0 and 2)\n", step);+ return;+ }+ count = smem_items(partition, cacheline, found, 256);+ printf("adsp: %u: the partition for us and the audio DSP (0 and 2): at %#lx, %#x bytes, of which %#x free; %u items\n", step,+ SMEM_PHYS + partition, size, header.free_cached - header.free_uncached, count);+ for (unsigned i = 0; i < count && i < 256;) {+ int at = 0;++ for (unsigned n = 0; n < 12 && i < count && i < 256; n++, i++) {+ at += snprintf(line + at, sizeof(line) - (size_t)at, " %u (%u)", found[i].item, found[i].size);+ }+ printf("adsp: %u: items (bytes):%s\n", step, line);+ }+}++/* ---- SMP2P ---------------------------------------------------------------------------------------- */++struct smp2p {+ uint32_t magic;+ uint32_t version;+ uint16_t local, remote, total, valid;+ uint32_t flags;+ struct {+ char name[16];+ uint32_t value;+ } entry[16];+};+#define SMP2P_MAGIC 0x504d5324+#define SMP2P_VERSION 1+#define SMP2P_RESTART_ACKS 1u+#define SMP2P_RESTART_DONE (1u << 0)+#define SMP2P_RESTART_ACK (1u << 1)+#define SMP2P_AT_VERSION 4+#define SMP2P_AT_PROCESSORS 8+#define SMP2P_AT_COUNTS 12+#define SMP2P_AT_FLAGS 16+#define SMP2P_AT_NAME(i) (20 + 20 * (i))+#define SMP2P_AT_VALUE(i) (36 + 20 * (i))++/* The bits of the DSP's entry "slave-kernel" (dt: the interrupts of qcom,lpass@3700000). */+#define DSP_FATAL (1u << 0)+#define DSP_READY (1u << 1)+#define DSP_HANDOVER (1u << 2)+#define DSP_STOP_ACK (1u << 3)+#define DSP_SHUTDOWN_ACK (1u << 7)++/* Our entry (dt: the child "master-kernel" of qcom,smp2p-adsp; "qcom,force-stop" of qcom,lpass@3700000). */+#define OUR_ENTRY_STOP "master-kernel"+#define DSP_STOP (1u << 0)++static bool+dsp_words(struct smp2p *words)+{+ static uint32_t at;+ uint32_t size = 0;++ if (at == 0) {+ at = smem_item(ITEM_SMP2P_THEIRS, &size);+ if (at == 0 || size < sizeof(*words)) {+ at = 0;+ return false;+ }+ }+ return smem_peek(at, words, sizeof(*words)) && words->magic == SMP2P_MAGIC;+}++static int+smp2p_entry(const struct smp2p *words, const char *name)+{+ for (unsigned i = 0; i < words->valid && i < 16; i++) {+ if (strncmp(words->entry[i].name, name, sizeof(words->entry[i].name)) == 0) {+ return (int)i;+ }+ }+ return -1;+}++static uint32_t+dsp_state(const struct smp2p *words)+{+ int entry = smp2p_entry(words, "slave-kernel");++ return entry < 0 ? 0 : words->entry[entry].value;+}++static void+show_words(const char *prefix, const struct smp2p *words)+{+ char line[320];+ int at = 0;++ line[0] = 0;+ for (unsigned i = 0; i < words->valid && i < 16 && at < (int)sizeof(line) - 40; i++) {+ at += snprintf(line + at, sizeof(line) - (size_t)at, " %.16s = %#x;", words->entry[i].name, words->entry[i].value);+ }+ printf("adsp: %u: %s: SMP2P version %u, features %#x, from processor %u to %u, flags %#x, %u of %u entries:%s\n", step, prefix,+ words->version & 0xff, words->version >> 8, words->local, words->remote, words->flags, words->valid, words->total, line);+}++static void+show_crash_reason(void)+{+ char text[81];+ uint32_t size = 0, at = smem_item(ITEM_CRASH_REASON, &size);++ if (at == 0 || size == 0) {+ printf("adsp: %u: item %u, the DSP's last words: not there\n", step, ITEM_CRASH_REASON);+ return;+ }+ bzero(text, sizeof(text));+ smem_peek(at, text, size < sizeof(text) - 1 ? size : sizeof(text) - 1);+ if (text[0] == 0) {+ printf("adsp: %u: item %u, the DSP's last words (%u bytes): none\n", step, ITEM_CRASH_REASON, size);+ } else {+ printf("adsp: %u: THE AUDIO DSP'S LAST WORDS (item %u, %u bytes): \"%s\"\n", step, ITEM_CRASH_REASON, size, text);+ }+}++/* ---- the doorbells, and the mailbox to the always-on processor (redwood_wifi.c, steps 2 and 7) ------ */++#define QMP_MAGIC 0x00+#define QMP_MAGIC_VALUE 0x4d41494c+#define QMP_VERSION 0x04+#define QMP_THEIR_LINK 0x0c+#define QMP_THEIR_LINK_ACK 0x10+#define QMP_THEIR_CHANNEL 0x14+#define QMP_THEIR_CHANNEL_ACK 0x18+#define QMP_OUR_LINK 0x24+#define QMP_OUR_LINK_ACK 0x28+#define QMP_OUR_CHANNEL 0x2c+#define QMP_OUR_CHANNEL_ACK 0x30+#define QMP_OUR_BOX_SIZE 0x34+#define QMP_OUR_BOX 0x38+#define QMP_UP 0x0000ffffu+#define QMP_DOWN 0xffff0000u+#define QMP_MESSAGE 64++static struct block aoss = { "mailbox", 0x0c300000, 0x400 };++#define IPCC_SEND 0x0c+#define IPCC_RECEIVED 0x10+#define IPCC_ENABLE 0x14+#define IPCC_DISABLE 0x18+#define IPCC_CLEAR 0x1c+#define IPCC_NOBODY 0xffffffffu+#define IPCC_AOP 0u+#define IPCC_LPASS 3u /* dt: mboxes of qcom,smp2p-adsp and qcom,glink/adsp */+#define IPCC_GLINK 0u+#define IPCC_SMP2P 2u++static struct block ipcc = { "ipcc", 0x00408000, 0x1000 };++static struct {+ uint32_t glink, smp2p, said;+ uint64_t since;+} doorbell;++/*+ * The audio DSP's rings, counted and cleared. Another's ring (the Wi-Fi DSP's, the always-on+ * processor's) is left for whoever listens for it: IPCC names the same one until it is cleared,+ * so the look ends there.+ */+static void+doorbells(void)+{+ for (unsigned i = 0; i < 8; i++) {+ uint32_t who = rd(ipcc.base + IPCC_RECEIVED);++ if (who == IPCC_NOBODY || who >> 16 != IPCC_LPASS) {+ return;+ }+ wr(ipcc.base + IPCC_CLEAR, who);+ doorbell.glink += (who & 0xffff) == IPCC_GLINK;+ doorbell.smp2p += (who & 0xffff) == IPCC_SMP2P;+ if (doorbell.said < 24) {+ uint64_t now = now_us() - doorbell.since;++ doorbell.said++;+ printf("adsp: doorbell: +%llu.%03llu ms: signal %u of the audio DSP; heard and cleared\n", now / 1000, now % 1000,+ who & 0xffff);+ }+ }+}++static uint32_t our_box; /* where in the message memory our mailbox is; 0: not open */++static bool+mailbox_wait(uint32_t offset, uint32_t value, uint32_t ms)+{+ for (uint32_t i = 0; i < ms * 10; i++) {+ if (rd(aoss.base + offset) == value) {+ return true;+ }+ sleep_us(100);+ }+ return rd(aoss.base + offset) == value;+}++static void+ring_aop(void)+{+ wr(ipcc.base + IPCC_SEND, IPCC_AOP << 16 | 0);+}++/* Linux's qmp_open(), as redwood_wifi.c does it; what is done already is found done. Under the mailbox's lock. */+static bool+mailbox_open(void)+{+ uint32_t box, size;++ need(&aoss);+ need(&ipcc);+ if (get(&aoss, "magic", QMP_MAGIC) != QMP_MAGIC_VALUE || get(&aoss, "version", QMP_VERSION) != 1) {+ printf("adsp: %u: not a mailbox this knows\n", step);+ return false;+ }+ box = get(&aoss, "our mailbox", QMP_OUR_BOX);+ size = get(&aoss, "our mailbox's size", QMP_OUR_BOX_SIZE);+ if (size < QMP_MESSAGE + 4 || box < 0x3c || box + size > aoss.size || (box & 3)) {+ printf("adsp: %u: our mailbox is not where a message fits\n", step);+ return false;+ }+ if (rd(aoss.base + QMP_OUR_LINK) == QMP_UP && rd(aoss.base + QMP_OUR_LINK_ACK) == QMP_UP &&+ rd(aoss.base + QMP_OUR_CHANNEL) == QMP_UP && rd(aoss.base + QMP_THEIR_CHANNEL) == QMP_UP &&+ rd(aoss.base + QMP_OUR_CHANNEL_ACK) == QMP_UP) {+ printf("adsp: %u: the mailbox is open already (the Wi-Fi's start, or a kernel before this one, opened it)\n", step);+ our_box = box;+ return true;+ }+ put(&aoss, "their link, seen by us", QMP_THEIR_LINK_ACK, rd(aoss.base + QMP_THEIR_LINK));+ put(&aoss, "our link", QMP_OUR_LINK, QMP_UP);+ ring_aop();+ if (!mailbox_wait(QMP_OUR_LINK_ACK, QMP_UP, 1000)) {+ printf("adsp: %u: THE ALWAYS-ON PROCESSOR DOES NOT ACKNOWLEDGE OUR LINK within a second; taken down again\n", step);+ put(&aoss, "our link", QMP_OUR_LINK, QMP_DOWN);+ ring_aop();+ return false;+ }+ put(&aoss, "our channel", QMP_OUR_CHANNEL, QMP_UP);+ ring_aop();+ if (!mailbox_wait(QMP_THEIR_CHANNEL, QMP_UP, 1000)) {+ printf("adsp: %u: THE ALWAYS-ON PROCESSOR DOES NOT OPEN ITS CHANNEL within a second; ours is taken down again\n", step);+ put(&aoss, "our channel", QMP_OUR_CHANNEL, QMP_DOWN);+ put(&aoss, "our link", QMP_OUR_LINK, QMP_DOWN);+ ring_aop();+ return false;+ }+ put(&aoss, "their channel, seen by us", QMP_THEIR_CHANNEL_ACK, QMP_UP);+ ring_aop();+ if (!mailbox_wait(QMP_OUR_CHANNEL_ACK, QMP_UP, 1000)) {+ printf("adsp: %u: THE ALWAYS-ON PROCESSOR DOES NOT ACKNOWLEDGE OUR CHANNEL within a second; taken down again\n", step);+ put(&aoss, "our channel", QMP_OUR_CHANNEL, QMP_DOWN);+ put(&aoss, "our link", QMP_OUR_LINK, QMP_DOWN);+ ring_aop();+ return false;+ }+ printf("adsp: %u: THE MAILBOX TO THE ALWAYS-ON PROCESSOR IS OPEN\n", step);+ our_box = box;+ return true;+}++/* A message to the always-on processor (Linux's qmp_send()), under the lock the Wi-Fi's start takes too. */+static bool+mailbox_send(const char *text)+{+ uint32_t words[QMP_MESSAGE / 4] = { 0 };+ bool sent = false;++ if (!leaving) {+ lck_mtx_lock(&redwood_aop_mailbox_lock);+ }+ if (our_box == 0 && (leaving || !mailbox_open())) {+ goto out;+ }+ strlcpy((char *)words, text, sizeof(words));+ printf("adsp: %u: about to tell the always-on processor \"%s\"\n", step, text);+ snprintf(last_write, sizeof(last_write), "step %u sent the always-on processor a message", step);+ if (rd(aoss.base + our_box) != 0) {+ printf("adsp: %u: our mailbox still holds a message of %u bytes: not sent\n", step, rd(aoss.base + our_box));+ goto out;+ }+ for (unsigned i = 0; i < QMP_MESSAGE / 4; i++) {+ wr(aoss.base + our_box + 4 + 4 * i, words[i]);+ }+ wr(aoss.base + our_box, QMP_MESSAGE);+ (void)rd(aoss.base + our_box);+ ring_aop();+ if (!mailbox_wait(our_box, 0, 1000)) {+ printf("adsp: %u: THE ALWAYS-ON PROCESSOR DOES NOT TAKE THE MESSAGE within a second; it is taken back\n", step);+ wr(aoss.base + our_box, 0);+ goto out;+ }+ printf("adsp: %u: the always-on processor has taken it\n", step);+ sent = true;+out:+ if (!leaving) {+ lck_mtx_unlock(&redwood_aop_mailbox_lock);+ }+ return sent;+}++/* ---- RPMh: the DSP's two rails, asked for while it boots ------------------------------------------------ */++/*+ * The audio DSP's logic and memories run from LPASS's own rails, LCX and LMX (dt: vdd_cx and+ * vdd_mx of qcom,lpass@3700000 are PM7325 L5 and L4 "level" regulators, RPMh resources lcx.lvl and+ * lmx.lvl), which the boot loader leaves at retention (16). Linux and Android ask for level 384+ * while the DSP boots, until it says it has asked for its own (bit 2, "proxy-unvote"); without it+ * the DSP rang once and left "SFR Init: wdog or kernel error suspected." in its last words.+ *+ * A request goes to RPMh through the application processor's sequencer (dt: rsc@18200000, DRV 2,+ * qcom,tcs-offset 0xd00, qcom,tcs-config: TCSs 0 and 1 are for requests made at once), as Linux's+ * rpmh-rsc.c writes one: the command's message id, address and value into a TCS that is idle, the+ * command enabled and marked to be waited for, the TCS put in "active" mode and triggered, and its+ * bit in the status of completions looked for. A rail asked for by level is asked by position in+ * the list of its levels that the boot loader's command database keeps (cmd-db.c), the first+ * position at or above the level wanted.+ */+#define CMD_DB_PHYS 0x80860000UL+#define CMD_DB_SIZE 0x20000UL++struct cmd_db_slave {+ uint16_t id, header_offset, data_offset, count, version, reserved[3];+};+struct cmd_db_header {+ uint32_t version;+ uint8_t magic[4];+ struct cmd_db_slave slave[8];+ uint32_t checksum, reserved;+ uint8_t data[];+};+struct cmd_db_entry {+ char id[8];+ uint32_t priority[2];+ uint32_t address;+ uint16_t length, offset;+};++static const struct cmd_db_header *cmd_db;++static struct block rsc = { "rsc", 0x18220000, 0x10000 };+#define RSC_TCS_BASE 0xd00+#define RSC_IRQ_STATUS (RSC_TCS_BASE + 0x04) /* a bit per TCS: done */+#define RSC_IRQ_CLEAR (RSC_TCS_BASE + 0x08)+#define RSC_TCS(n) (RSC_TCS_BASE + 672 * (n))+#define TCS_WAIT_FOR_DONE 0x10+#define TCS_CONTROL 0x14+#define TCS_STATUS 0x18 /* not zero: idle */+#define TCS_ENABLED 0x1c+#define TCS_COMMAND(c) (0x30 + 20 * (c)) /* message id, address, value, status */+#define TCS_ACTIVE_MODE (1u << 16)+#define TCS_TRIGGER (1u << 24)+#define MSGID_WRITE_WAITED (8u | 1u << 8 | 1u << 16) /* four bytes, an answer wanted, a write */+#define RPMH_TCS 0 /* the first of the two for requests made at once */++/* The level position RPMh takes for that rail and level; -1: not found. */+static int+rail_position(const char *name, uint32_t level, uint32_t *address)+{+ char id[8] = { 0 };++ if (cmd_db == NULL) {+ cmd_db = (const struct cmd_db_header *)ml_io_map_unmappable(CMD_DB_PHYS, CMD_DB_SIZE, VM_WIMG_WCOMB);+ }+ if (!(cmd_db->magic[0] == 0xdb && cmd_db->magic[1] == 0x30 && cmd_db->magic[2] == 0x03 && cmd_db->magic[3] == 0x0c)) {+ printf("adsp: %u: the boot loader's command database is not there\n", step);+ return -1;+ }+ strlcpy(id, name, sizeof(id));+ for (unsigned s = 0; s < 8; s++) {+ const struct cmd_db_slave *slave = &cmd_db->slave[s];+ const struct cmd_db_entry *entry = (const struct cmd_db_entry *)(cmd_db->data + slave->header_offset);++ if (slave->id == 0 || slave->count > 512) {+ continue;+ }+ for (unsigned i = 0; i < slave->count; i++) {+ const uint8_t *data;+ char levels[160];+ int at;++ if (memcmp(entry[i].id, id, sizeof(id)) != 0) {+ continue;+ }+ if ((unsigned)slave->data_offset + entry[i].offset + entry[i].length > CMD_DB_SIZE - 0x1000) {+ return -1;+ }+ data = cmd_db->data + slave->data_offset + entry[i].offset;+ at = snprintf(levels, sizeof(levels), "%s at %#x, levels:", name, entry[i].address);+ for (unsigned l = 0; l < entry[i].length / 2u && l < 16; l++) {+ unsigned value = data[2 * l] | data[2 * l + 1] << 8;++ if (l > 0 && value == 0) {+ break;+ }+ at += snprintf(levels + at, sizeof(levels) - (size_t)at, " %u", value);+ }+ printf("adsp: %u: RPMh has %s\n", step, levels);+ for (unsigned l = 0; l < entry[i].length / 2u && l < 16; l++) {+ if ((uint32_t)(data[2 * l] | data[2 * l + 1] << 8) >= level) {+ *address = entry[i].address;+ return (int)l;+ }+ }+ return -1;+ }+ }+ printf("adsp: %u: RPMh has no \"%s\"\n", step, name);+ return -1;+}++/* The two rails at that level (Linux's proxy request), or given back (level 0). True: RPMh has said done. */+static bool+rails(uint32_t level)+{+ static const char *const names[2] = { "lcx.lvl", "lmx.lvl" };+ uint32_t addresses[2] = { 0, 0 }, values[2] = { 0, 0 }, done = 0;+ uint64_t began;++ for (unsigned r = 0; r < 2; r++) {+ int position = level == 0 ? 0 : rail_position(names[r], level, &addresses[r]);++ if (level == 0) {+ rail_position(names[r], 0, &addresses[r]);+ }+ if (position < 0 || addresses[r] == 0) {+ printf("adsp: %u: no position for %s at level %u: the rails are not asked\n", step, names[r], level);+ return false;+ }+ values[r] = (uint32_t)position;+ }+ need(&rsc);+ /* (the boot loader leaves the trigger set after its last request, done: idle is what the status says) */+ if (rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_STATUS) == 0) {+ printf("adsp: %u: TCS %u of the sequencer is busy (status %#x, control %#x): the rails are not asked\n", step, RPMH_TCS,+ rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_STATUS), rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL));+ return false;+ }+ printf("adsp: %u: about to ask RPMh, through TCS %u, for lcx.lvl (%#x) = %u and lmx.lvl (%#x) = %u (level %u%s)\n", step,+ RPMH_TCS, addresses[0], values[0], addresses[1], values[1], level, level == 0 ? ": given back" : "");+ snprintf(last_write, sizeof(last_write), "step %u asked RPMh for the audio DSP's rails at level %u", step, level);+ hold();+ wr(rsc.base + RSC_IRQ_CLEAR, 1u << RPMH_TCS);+ for (unsigned r = 0; r < 2; r++) {+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(r), MSGID_WRITE_WAITED);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(r) + 4, addresses[r]);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(r) + 8, values[r]);+ }+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_WAIT_FOR_DONE, 3);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_ENABLED, 3);+ settle();+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL) & ~TCS_TRIGGER);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL) | TCS_ACTIVE_MODE);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL) | TCS_TRIGGER);+ began = now_us();+ while (now_us() - began < 100000) {+ done = rd(rsc.base + RSC_IRQ_STATUS) & (1u << RPMH_TCS);+ if (done) {+ break;+ }+ sleep_us(50);+ }+ printf("adsp: %u: RPMh %s after %llu us; the commands' status %#x %#x, the TCS's %#x\n", step, done ? "HAS DONE IT" :+ "HAS NOT SAID DONE", now_us() - began, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(0) + 12),+ rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(1) + 12), rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_STATUS));+ wr(rsc.base + RSC_IRQ_CLEAR, 1u << RPMH_TCS);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL) & ~(TCS_TRIGGER | TCS_ACTIVE_MODE));+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_ENABLED, 0);+ return done != 0;+}++/* ---- step 1: the shared memory --------------------------------------------------------------------- */++static bool+step_survey(void)+{+ struct smp2p words;+ uint32_t size = 0;++ announce("read our partition with the audio DSP in the shared memory (0x80900000), and its items");+ show_partition();+ if (dsp_words(&words)) {+ show_words("the DSP's item (443) is there: it has run since the phone was started", &words);+ } else {+ printf("adsp: %u: the DSP's SMP2P item (443) is not there: it has not run since the phone was started\n", step);+ }+ printf("adsp: %u: our SMP2P item (429): %s; GLINK's descriptor (478): %s, our queue (479): %s, the DSP's (480): %s\n", step,+ smem_item(ITEM_SMP2P_OURS, &size) ? "there" : "not there", smem_item(478, &size) ? "there" : "NOT THERE",+ smem_item(479, &size) ? "there" : "NOT THERE", smem_item(480, &size) ? "there" : "not there (the DSP makes it)");+ show_crash_reason();+ need(&ipcc);+ printf("adsp: %u: the doorbells say %#x\n", step, rd(ipcc.base + IPCC_RECEIVED));+ return smem_partition(SMEM_US, SMEM_ADSP, &size, &size) != 0;+}++/* ---- step 2: the firmware -------------------------------------------------------------------------- */++#define ADSP_MEMORY 0x86700000u /* dt: reserved-memory adsp@86700000, memory-region of qcom,lpass@3700000 */+#define ADSP_MEMORY_SIZE 0x04000000u+#define PAS_ADSP 1 /* dt: qcom,pas-id */++/* An entry of a directory of a packfs image in memory, by name (as in redwood_wifi.c). */+static const struct packfs_dnode *+image_entry(const uint8_t *image, const struct packfs_super *super, const struct packfs_dnode *directory, const char *name)+{+ const struct packfs_dnode *nodes = (const struct packfs_dnode *)(image + super->ps_nodes_off);+ size_t length = strlen(name);++ if (directory == NULL || !S_ISDIR(directory->pd_mode) || directory->pd_data + directory->pd_size > super->ps_node_count) {+ return NULL;+ }+ for (uint64_t i = 0; i < directory->pd_size; i++) {+ const struct packfs_dnode *node = &nodes[directory->pd_data + i];++ if (node->pd_name_len == length && (uint64_t)node->pd_name + length <= super->ps_strings_size &&+ memcmp(image + super->ps_strings_off + node->pd_name, name, length) == 0) {+ return node;+ }+ }+ return NULL;+}++/* A file of fw/ in the small image (userland/build.sh puts fw-audio/ there), or in the system image. */+static const uint8_t *+image_file(const char *name, uint64_t *size)+{+ for (int disk = 1; disk >= 0; disk--) {+ const struct packfs_super *super;+ const struct packfs_dnode *file;+ vm_offset_t base = 0;+ uint64_t length = 0;+ dev_t dev = mdevlookup(disk);++ if (dev == (dev_t)-1 || mdevgetkernelrange(dev, &base, &length) != 0 || length < PACKFS_SUPER_SIZE) {+ continue;+ }+ super = (const struct packfs_super *)base;+ if (memcmp(super->ps_magic, PACKFS_MAGIC, sizeof(super->ps_magic)) != 0 || super->ps_image_size > length ||+ super->ps_node_count == 0 ||+ super->ps_nodes_off + (uint64_t)super->ps_node_count * sizeof(struct packfs_dnode) > super->ps_image_size ||+ super->ps_strings_off + super->ps_strings_size > super->ps_image_size) {+ continue;+ }+ file = (const struct packfs_dnode *)(base + super->ps_nodes_off);+ file = image_entry((const uint8_t *)base, super, file, "fw");+ file = image_entry((const uint8_t *)base, super, file, name);+ if (file != NULL && S_ISREG(file->pd_mode) && !(file->pd_flags & PACKFS_F_CHUNKED) &&+ file->pd_data + file->pd_size <= super->ps_image_size) {+ *size = file->pd_size;+ return (const uint8_t *)base + file->pd_data;+ }+ }+ printf("adsp: %u: no fw/%s in the small image: put the phone's /vendor/firmware_mnt/image/%s into fw-audio/ and run "+ "userland/build.sh\n", step, name, name);+ return NULL;+}++#define MDT_HASH(flags) (((flags) >> 24 & 7) == 2)+#define MAX_SEGMENTS 48++struct elf_segment {+ uint32_t type, offset, vaddr, paddr, file_size, memory_size, flags, align;+};++static struct {+ const uint8_t *header;+ uint64_t header_size;+ struct {+ uint32_t at;+ uint32_t file_size, memory_size;+ const uint8_t *file;+ } segment[MAX_SEGMENTS];+ unsigned segments;+ uint32_t size;+} image;++static bool+step_firmware(void)+{+ struct elf_segment segment;+ const uint8_t *header;+ uint64_t size = 0, files = 0;+ uint32_t count, hashes = 0, lowest = ~0u, highest = 0;++ announce("find the audio DSP's image in the small image, and check it against its own header");+ image.segments = 0;+ header = image_file("adsp.mdt", &size);+ if (header == NULL) {+ return false;+ }+ count = size >= 52 + 2 * 32 ? header[44] | header[45] << 8 : 0;+ if (memcmp(header, "\177ELF\1", 5) != 0 || count < 3 || count > MAX_SEGMENTS || 52 + 32 * count > size || size > PAGE_SIZE) {+ printf("adsp: %u: fw/adsp.mdt is not such a header (%02x %02x %02x %02x, %u segments, %llu bytes)\n", step, header[0],+ header[1], header[2], header[3], count, size);+ return false;+ }+ for (uint32_t i = 0; i < count; i++) {+ char name[16];++ memcpy(&segment, header + 52 + 32 * i, sizeof(segment));+ if (MDT_HASH(segment.flags)) {+ hashes = segment.file_size;+ }+ if (segment.type != 1 || MDT_HASH(segment.flags) || segment.memory_size == 0) {+ continue;+ }+ if (segment.paddr < ADSP_MEMORY || segment.file_size > segment.memory_size ||+ (uint64_t)segment.paddr + segment.memory_size > (uint64_t)ADSP_MEMORY + ADSP_MEMORY_SIZE) {+ printf("adsp: %u: segment %u (%#x, %#x bytes) is not within the DSP's memory\n", step, i, segment.paddr,+ segment.memory_size);+ return false;+ }+ image.segment[image.segments].at = segment.paddr - ADSP_MEMORY;+ image.segment[image.segments].file_size = segment.file_size;+ image.segment[image.segments].memory_size = segment.memory_size;+ image.segment[image.segments].file = NULL;+ if (segment.file_size != 0) {+ uint64_t file_size = 0;++ snprintf(name, sizeof(name), "adsp.b%02u", i);+ image.segment[image.segments].file = image_file(name, &file_size);+ if (image.segment[image.segments].file == NULL) {+ return false;+ }+ if (file_size != segment.file_size) {+ printf("adsp: %u: fw/%s has %llu bytes, the header says %u\n", step, name, file_size, segment.file_size);+ return false;+ }+ files += file_size;+ }+ lowest = segment.paddr < lowest ? segment.paddr : lowest;+ highest = segment.paddr + segment.memory_size > highest ? segment.paddr + segment.memory_size : highest;+ image.segments++;+ }+ if (image.segments == 0 || lowest != ADSP_MEMORY || 52 + 32 * count + (uint64_t)hashes != size) {+ printf("adsp: %u: not what TrustZone takes: %u segments to load, the lowest at %#x; the header file has %llu bytes, "+ "the header %u and the hashes %u\n", step, image.segments, lowest, size, 52 + 32 * count, hashes);+ image.segments = 0;+ return false;+ }+ image.header = header;+ image.header_size = size;+ image.size = ((highest + 0xfff) & ~0xfffu) - ADSP_MEMORY;+ printf("adsp: %u: the image is whole: a header of %llu bytes (%u of hashes), %u segments, %llu bytes of files, %#x bytes "+ "of the DSP's %#x from %#x\n", step, size, hashes, image.segments, files, image.size, ADSP_MEMORY_SIZE, ADSP_MEMORY);+ return true;+}++/* ---- step 3: the mailbox --------------------------------------------------------------------------- */++static bool+step_mailbox(void)+{+ bool open;++ announce("open the mailbox to the always-on processor, unless it is open: words of its message memory, and its doorbell");+ lck_mtx_lock(&redwood_aop_mailbox_lock);+ open = mailbox_open();+ lck_mtx_unlock(&redwood_aop_mailbox_lock);+ return open;+}++/* ---- TrustZone, and stopping the DSP --------------------------------------------------------------- */++#define SCM_64 (1ull << 30)+#define SCM_SIP (2ull << 24)+#define SCM_CALL(service, command) ((uint64_t)(service) << 8 | (command))+#define SCM_PAS_INIT_IMAGE SCM_CALL(2, 1)+#define SCM_PAS_MEM_SETUP SCM_CALL(2, 2)+#define SCM_PAS_AUTH_AND_RESET SCM_CALL(2, 5)+#define SCM_PAS_SHUTDOWN SCM_CALL(2, 6)+#define SCM_ONE_VALUE 1+#define SCM_THREE_VALUES 3+#define SCM_VALUE_AND_BUFFER (2 | 2 << 6)+#define SCM_INTERRUPTED 1++static LCK_MTX_DECLARE(dsp_lock, &adsp_lock_group); /* the DSP being started or stopped */+static bool rails_asked; /* our request for its rails stands (until it asks for its own, or is shut down) */+static bool dsp_started;+static bool stopped_by_its_word;+static uint32_t our_item;+static bool smp2p_agreed, smp2p_restart_acks, smp2p_restart_acked;++static int32_t+scm(const char *what, uint64_t function, uint64_t arguments, uint64_t first, uint64_t second, uint64_t third, uint64_t *answer)+{+ uint64_t session = 0;++ function |= SCM_64 | SCM_SIP;+ printf("adsp: %u: about to ask TrustZone %s: smc %#llx, arguments %#llx, %#llx, %#llx\n", step, what, function, first,+ second, third);+ snprintf(last_write, sizeof(last_write), "step %u asked TrustZone %#llx(%#llx, %#llx)", step, function, first, second);+ for (int i = 0; i < 100; i++) {+ register uint64_t x0 __asm__("x0") = i == 0 ? function : SCM_INTERRUPTED;+ register uint64_t x1 __asm__("x1") = arguments;+ register uint64_t x2 __asm__("x2") = first;+ register uint64_t x3 __asm__("x3") = second;+ register uint64_t x4 __asm__("x4") = third;+ register uint64_t x5 __asm__("x5") = 0;+ register uint64_t x6 __asm__("x6") = session;+ register uint64_t x7 __asm__("x7") = 0;++ __asm__ volatile ("smc #0" : "+r" (x0), "+r" (x1), "+r" (x2), "+r" (x3), "+r" (x4), "+r" (x5), "+r" (x6),+ "+r" (x7) : : "x8", "x9", "x10", "x11", "x12", "x13", "x14", "x15", "x16", "x17", "memory");+ session = x6;+ if (x0 != SCM_INTERRUPTED) {+ printf("adsp: %u: TrustZone returns %d, answer %#llx\n", step, (int32_t)x0, x1);+ *answer = x1;+ return (int32_t)x0;+ }+ }+ printf("adsp: %u: TrustZone keeps saying the call was interrupted\n", step);+ return -1;+}++static uint32_t+our_item_find(void)+{+ uint32_t size = 0, at = smem_item(ITEM_SMP2P_OURS, &size);++ return at != 0 && size >= sizeof(struct smp2p) ? at : 0;+}++/* What Linux does when the DSP rings for SMP2P: agree on the features once, and answer a restart. */+static void+dsp_listen(const struct smp2p *theirs)+{+ struct smp2p ours;++ if (our_item == 0 || !smem_peek(our_item, &ours, sizeof(ours)) || ours.magic != SMP2P_MAGIC) {+ return;+ }+ if (!smp2p_agreed) {+ uint32_t features = (ours.version >> 8) & (theirs->version >> 8);++ if ((theirs->version & 0xff) != (ours.version & 0xff)) {+ return;+ }+ if (features != ours.version >> 8) {+ smem_poke(our_item + SMP2P_AT_VERSION, features << 8 | (ours.version & 0xff));+ }+ smp2p_restart_acks = features & SMP2P_RESTART_ACKS;+ smp2p_agreed = true;+ printf("adsp: %u: the DSP and we agree on SMP2P: version %u, features %#x\n", step, ours.version & 0xff, features);+ }+ if (smp2p_restart_acks && ((theirs->flags & SMP2P_RESTART_DONE) != 0) != smp2p_restart_acked) {+ smp2p_restart_acked = !smp2p_restart_acked;+ printf("adsp: %u: the DSP's flags say it has restarted (%#x): answered in our flags (bit 1 = %u), and its SMP2P doorbell\n",+ step, theirs->flags, smp2p_restart_acked);+ smem_poke(our_item + SMP2P_AT_FLAGS, (ours.flags & ~SMP2P_RESTART_ACK) | (smp2p_restart_acked ? SMP2P_RESTART_ACK : 0));+ settle();+ wr(ipcc.base + IPCC_SEND, IPCC_LPASS << 16 | IPCC_SMP2P);+ }+}++/* Linux's qcom_q6v5_request_stop(): our bit "stop", the DSP's "stop ack" waited for, our bit taken back. */+static bool+dsp_ask_to_stop(uint32_t bound_ms)+{+ struct smp2p ours, theirs;+ uint32_t at = our_item_find(), state, value_at;+ uint64_t asked, now;+ int entry = -1;++ if (at == 0 || !smem_peek(at, &ours, sizeof(ours)) || ours.magic != SMP2P_MAGIC || (ours.version & 0xff) != SMP2P_VERSION ||+ (entry = smp2p_entry(&ours, OUR_ENTRY_STOP)) < 0) {+ printf("adsp: %u: the DSP is not asked to stop first: there is no SMP2P item of ours to ask it with\n", step);+ return false;+ }+ if (!dsp_words(&theirs) || smp2p_entry(&theirs, "slave-kernel") < 0) {+ printf("adsp: %u: the DSP is not asked to stop first: its item has no entry to answer in\n", step);+ return false;+ }+ state = dsp_state(&theirs);+ if (state & DSP_STOP_ACK) {+ printf("adsp: %u: the DSP is not asked to stop: its entry says %#x, that it has stopped when asked before\n", step, state);+ return true;+ }+ if (!(state & DSP_READY) || (state & DSP_FATAL)) {+ printf("adsp: %u: the DSP is not asked to stop first: its entry says %#x, not \"up\"\n", step, state);+ return false;+ }+ need(&ipcc);+ value_at = at + SMP2P_AT_VALUE((uint32_t)entry);+ printf("adsp: %u: about to ask the DSP to stop: bit 0 of our entry \"%s\" at %#lx, and its SMP2P doorbell\n", step,+ OUR_ENTRY_STOP, SMEM_PHYS + value_at);+ snprintf(last_write, sizeof(last_write), "step %u asked the audio DSP to stop", step);+ smem_poke(value_at, ours.entry[entry].value | DSP_STOP);+ settle();+ wr(ipcc.base + IPCC_SEND, IPCC_LPASS << 16 | IPCC_SMP2P);+ asked = now = now_us();+ while (now - asked < bound_ms * 1000ull) {+ if (dsp_words(&theirs) && (dsp_state(&theirs) & (DSP_STOP_ACK | DSP_FATAL))) {+ break;+ }+ sleep_us(200);+ now = now_us();+ }+ state = dsp_words(&theirs) ? dsp_state(&theirs) : 0;+ printf("adsp: %u: %s after %llu.%03llu ms; its entry says %#x\n", step, state & DSP_STOP_ACK ?+ "THE AUDIO DSP ACKNOWLEDGES THE STOP" : "THE AUDIO DSP DOES NOT ACKNOWLEDGE THE STOP", (now - asked) / 1000,+ (now - asked) % 1000, state);+ smem_poke(value_at, ours.entry[entry].value & ~DSP_STOP);+ settle();+ wr(ipcc.base + IPCC_SEND, IPCC_LPASS << 16 | IPCC_SMP2P);+ return (state & DSP_STOP_ACK) != 0;+}++static void link_stop(void);++/* Asked to stop, then shut down by TrustZone, then the always-on processor told that the image is gone. */+static bool+dsp_shutdown(const char *why)+{+ uint64_t answer = ~0ull;+ int32_t result;++ printf("adsp: %u: %s: the audio DSP is asked to stop, then TrustZone to shut it down\n", step, why);+ if (!leaving) {+ link_stop();+ }+ stopped_by_its_word = dsp_ask_to_stop(leaving ? 2000 : 5000);+ result = scm("to shut the audio DSP down", SCM_PAS_SHUTDOWN, SCM_ONE_VALUE, PAS_ADSP, 0, 0, &answer);+ if (result != 0 || answer != 0) {+ printf("adsp: %u: TRUSTZONE DOES NOT SHUT THE AUDIO DSP DOWN: the call returns %d, its answer is %#llx\n", step, result,+ answer);+ return false;+ }+ printf("adsp: %u: THE AUDIO DSP IS SHUT DOWN\n", step);+ need(&ipcc);+ wr(ipcc.base + IPCC_DISABLE, IPCC_LPASS << 16 | IPCC_SMP2P);+ doorbells();+ dsp_started = false;+ mailbox_send("{class: image, res: load_state, name: adsp, val: off}");+ if (rails_asked) {+ rails(0);+ rails_asked = false;+ }+ return true;+}++/* ---- step 4: TrustZone takes the header ------------------------------------------------------------ */++static bool+step_header(void)+{+ struct smp2p words;+ vm_offset_t page = 0;+ uint64_t answer = ~0ull;+ int32_t result;++ if (image.segments == 0) {+ return false;+ }+ if (dsp_words(&words)) {+ show_words("the DSP has left words in the shared memory", &words);+ announce("ask TrustZone to shut down an audio DSP that a kernel before this one may have left running");+ if (!dsp_shutdown("it may be running")) {+ return false;+ }+ if (!stopped_by_its_word && !option("adsp_again")) {+ printf("adsp: %u: THE AUDIO DSP IS NOT STARTED AGAIN: it was shut down without having acknowledged a stop, and "+ "with the Wi-Fi's DSP a start after that reset the phone (adsp_again=1 to try all the same)\n", step);+ return false;+ }+ } else {+ printf("adsp: %u: the DSP has left nothing in the shared memory: it has not run since the phone was started\n", step);+ }++ announce("hand the image's header to TrustZone to check its signature (image 1), in a page of ours");+ if (kmem_alloc(kernel_map, &page, PAGE_SIZE, KMA_KOBJECT | KMA_ZERO | KMA_DATA, VM_KERN_MEMORY_IOKIT) != KERN_SUCCESS) {+ printf("adsp: %u: no memory\n", step);+ return false;+ }+ memcpy((void *)page, image.header, image.header_size);+ for (vm_offset_t line = page; line < page + PAGE_SIZE; line += 64) {+ __asm__ volatile ("dc civac, %0" : : "r" (line));+ }+ settle();+ result = scm("to take the audio DSP's header", SCM_PAS_INIT_IMAGE, SCM_VALUE_AND_BUFFER, PAS_ADSP, kvtophys(page), 0, &answer);+ if (result != 0 || answer != 0) {+ printf("adsp: %u: TRUSTZONE DOES NOT TAKE THE HEADER: the call returns %d, its answer is %#llx\n", step, result, answer);+ return false;+ }+ printf("adsp: %u: TRUSTZONE ACCEPTS THE AUDIO DSP'S SIGNATURE FROM THIS KERNEL\n", step);++ announce("tell TrustZone where the image will be");+ result = scm("where the image will be", SCM_PAS_MEM_SETUP, SCM_THREE_VALUES, PAS_ADSP, ADSP_MEMORY, image.size, &answer);+ if (result != 0 || answer != 0) {+ printf("adsp: %u: TRUSTZONE DOES NOT TAKE THE PLACE: the call returns %d, its answer is %#llx\n", step, result, answer);+ return false;+ }+ printf("adsp: %u: TrustZone takes the place: %#x bytes at %#x\n", step, image.size, ADSP_MEMORY);+ return true;+}++/* ---- step 5: the image, in the DSP's memory --------------------------------------------------------- */++static bool image_written;++static uint64_t+segment_word(unsigned s, uint32_t at)+{+ uint64_t word = 0;++ for (unsigned b = 0; b < 8 && at + b < image.segment[s].file_size; b++) {+ word |= (uint64_t)image.segment[s].file[at + b] << (8 * b);+ }+ return word;+}++static bool+step_image(void)+{+ volatile uint64_t *dsp_memory;+ uint64_t written = 0, cleared = 0, wrong = 0;++ if (image.segments == 0) {+ return false;+ }+ announce("put the image's segments into the audio DSP's memory (0x86700000, 64 MB: memory of nobody's until TrustZone "+ "starts the DSP), with zeroes where a segment is longer than its file, and read them back");+ dsp_memory = (volatile uint64_t *)ml_io_map_unmappable(ADSP_MEMORY, ADSP_MEMORY_SIZE, VM_WIMG_WCOMB);+ snprintf(last_write, sizeof(last_write), "step %u wrote the image at %#x", step, ADSP_MEMORY);+ for (unsigned s = 0; s < image.segments; s++) {+ volatile uint64_t *to = dsp_memory + image.segment[s].at / 8;++ if ((image.segment[s].at | image.segment[s].memory_size) & 7) {+ printf("adsp: %u: a segment that does not begin and end at a multiple of eight bytes: not written\n", step);+ ml_io_unmap((vm_offset_t)dsp_memory, ADSP_MEMORY_SIZE);+ return false;+ }+ for (uint32_t i = 0; i < image.segment[s].memory_size; i += 8) {+ to[i / 8] = segment_word(s, i);+ }+ written += image.segment[s].file_size;+ cleared += image.segment[s].memory_size - image.segment[s].file_size;+ }+ settle();+ for (unsigned s = 0; s < image.segments; s++) {+ volatile uint64_t *from = dsp_memory + image.segment[s].at / 8;++ for (uint32_t i = 0; i < image.segment[s].memory_size; i += 8) {+ wrong += from[i / 8] != segment_word(s, i);+ }+ }+ printf("adsp: %u: %u segments, %llu bytes written and %llu cleared; read back, %llu words of eight bytes differ\n", step,+ image.segments, written, cleared, wrong);+ settle();+ ml_io_unmap((vm_offset_t)dsp_memory, ADSP_MEMORY_SIZE);+ image_written = wrong == 0;+ return image_written;+}++/* ---- step 6: our SMP2P item ------------------------------------------------------------------------ */++static struct block smem_lock = { "shared memory's lock", 0x01f43000, 0x1000 };+#define LOCK_OWNER 0x000+#define LOCK_US 1u++/* Our item made in the partition, as Linux's qcom_smem_alloc_private() makes one (redwood_wifi.c: our_item_make()). */+static bool+our_item_make(uint32_t partition)+{+ struct smem_partition header;+ struct smem_entry entry;+ const uint32_t item_size = (sizeof(struct smp2p) + 7) & ~7u, need_bytes = sizeof(entry) + item_size;+ uint32_t owner = ~0u, at;+ bool made = false;++ lck_mtx_lock(&redwood_smem_alloc_lock); /* (the Wi-Fi's start allocates under the same hardware lock) */+ need(&smem_lock);+ printf("adsp: %u: about to take the lock (%#x), write an entry for item %u with %u bytes behind it at the partition's mark, "+ "move the mark by %u, and release the lock\n", step, smem_lock.phys + LOCK_OWNER, ITEM_SMP2P_OURS, item_size, need_bytes);+ snprintf(last_write, sizeof(last_write), "step %u took the shared memory's lock", step);+ for (uint32_t ms = 0; ms <= 1000; ms++) {+ wr(smem_lock.base + LOCK_OWNER, LOCK_US);+ owner = rd(smem_lock.base + LOCK_OWNER);+ if (owner == LOCK_US) {+ break;+ }+ sleep_ms(1);+ }+ if (owner != LOCK_US) {+ printf("adsp: %u: THE LOCK IS HELD BY PROCESSOR %u AND STAYS SO FOR A SECOND: nothing is written\n", step, owner);+ lck_mtx_unlock(&redwood_smem_alloc_lock);+ return false;+ }+ if (!smem_peek(partition, &header, sizeof(header)) || header.free_uncached > header.free_cached ||+ header.free_cached > header.size) {+ printf("adsp: %u: the partition's marks make no sense: nothing is written\n", step);+ goto out;+ }+ for (at = sizeof(header); at < header.free_uncached;) {+ if (!smem_peek(partition + at, &entry, sizeof(entry)) || entry.canary != SMEM_CANARY || entry.item == ITEM_SMP2P_OURS ||+ entry.size > header.size) {+ break;+ }+ at += sizeof(entry) + entry.padding_header + entry.size;+ }+ if (at != header.free_uncached) {+ printf("adsp: %u: the partition's entries do not end at its mark (+%#x, the mark is +%#x), or the item has appeared: "+ "nothing is written\n", step, at, header.free_uncached);+ goto out;+ }+ if (at + need_bytes > header.free_cached) {+ printf("adsp: %u: no room: %u bytes wanted, %u free: nothing is written\n", step, need_bytes, header.free_cached - at);+ goto out;+ }+ smem_poke(partition + at, SMEM_CANARY | ITEM_SMP2P_OURS << 16);+ smem_poke(partition + at + 4, item_size);+ smem_poke(partition + at + 8, item_size - (uint32_t)sizeof(struct smp2p));+ smem_poke(partition + at + 12, 0);+ settle();+ smem_poke(partition + __offsetof(struct smem_partition, free_uncached), at + need_bytes);+ settle();+ made = true;+out:+ wr(smem_lock.base + LOCK_OWNER, 0);+ printf("adsp: %u: the lock is released (it reads %u)\n", step, rd(smem_lock.base + LOCK_OWNER));+ lck_mtx_unlock(&redwood_smem_alloc_lock);+ return made;+}++static void glink_reset(void);++static bool+step_our_item(void)+{+ struct smem_partition header;+ struct smp2p ours;+ uint32_t cacheline = 0, size = 0, partition, name[4] = { 0 };++ announce("look for an SMP2P item of ours (429) in our partition with the audio DSP, make it if there is none, and fill it in");+ partition = smem_partition(SMEM_US, SMEM_ADSP, &cacheline, &size);+ if (partition == 0 || !smem_peek(partition, &header, sizeof(header))) {+ printf("adsp: %u: there is no such partition\n", step);+ return false;+ }+ our_item = our_item_find();+ if (our_item != 0) {+ printf("adsp: %u: item %u is there, at %#lx: a kernel before this one made it, and it is used again\n", step,+ ITEM_SMP2P_OURS, SMEM_PHYS + our_item);+ } else {+ if (!our_item_make(partition)) {+ return false;+ }+ our_item = our_item_find();+ if (our_item == 0) {+ printf("adsp: %u: THE ITEM IS NOT FOUND where it was just made\n", step);+ return false;+ }+ printf("adsp: %u: item %u made, at %#lx\n", step, ITEM_SMP2P_OURS, SMEM_PHYS + our_item);+ }+ /* zeroes; from processor 0 to 2, room for 16 entries, "master-kernel" at zero, feature bit 0; last the version */+ snprintf(last_write, sizeof(last_write), "step %u filled in our SMP2P item", step);+ for (uint32_t at = 0; at < sizeof(ours); at += 4) {+ smem_poke(our_item + at, 0);+ }+ settle();+ smem_poke(our_item, SMP2P_MAGIC);+ smem_poke(our_item + SMP2P_AT_PROCESSORS, SMEM_US | SMEM_ADSP << 16);+ strlcpy((char *)name, OUR_ENTRY_STOP, sizeof(name));+ for (uint32_t word = 0; word < 4; word++) {+ smem_poke(our_item + SMP2P_AT_NAME(0) + 4 * word, name[word]);+ }+ smem_poke(our_item + SMP2P_AT_COUNTS, 16 | 1u << 16);+ smem_poke(our_item + SMP2P_AT_VERSION, SMP2P_RESTART_ACKS << 8);+ settle();+ smem_poke(our_item + SMP2P_AT_VERSION, SMP2P_RESTART_ACKS << 8 | SMP2P_VERSION);+ settle();+ smp2p_agreed = smp2p_restart_acks = smp2p_restart_acked = false;+ need(&ipcc);+ wr(ipcc.base + IPCC_SEND, IPCC_LPASS << 16 | IPCC_SMP2P);+ if (!smem_peek(our_item, &ours, sizeof(ours)) || ours.magic != SMP2P_MAGIC || smp2p_entry(&ours, OUR_ENTRY_STOP) < 0) {+ printf("adsp: %u: OUR ITEM DOES NOT READ BACK AS WRITTEN\n", step);+ return false;+ }+ show_words("our item reads back", &ours);+ glink_reset();+ return true;+}++/* ---- step 7: the DSP, started ---------------------------------------------------------------------- */++/*+ * Android's kernel (this phone, dmesg): the image loaded at 8.92 s, "brought out of reset" at+ * 9.10, "error ready" 40 ms later; the audio services' protection domain up at 9.39.+ * Here (runs 085341 and later): the DSP's SMP2P item appears 17 ms after TrustZone returns, with+ * a ring of its SMP2P and GLINK doorbells; at 41 ms "slave-kernel" says 6, up and asking for its+ * own power, and our request for the rails is given back; at 2.2 s it adds "sleepstate_see".+ * Asked to stop, it acknowledges within 25 ms (0x10e), and it starts again in the same session.+ */+static bool+step_start(void)+{+ struct smp2p words, last;+ uint64_t answer = ~0ull, started, now, ready_at = 0, limit;+ uint32_t state = 0, rang, changes = 0, wait_seconds = 10;+ int32_t result;+ bool have, spoke = false, handed_over = false;++ if (!image_written) {+ printf("adsp: %u: the image is not in place\n", step);+ return false;+ }+ PE_parse_boot_argn("adsp_wait", &wait_seconds, sizeof(wait_seconds));+ bzero(&last, sizeof(last));+ have = dsp_words(&last);+ announce("listen for the audio DSP's doorbells, ask RPMh for its rails, tell the always-on processor that its image is "+ "loaded, and ask TrustZone to authenticate the image and let the DSP out of reset");+ need(&ipcc);+ for (unsigned i = 0; i < 8 && (rang = rd(ipcc.base + IPCC_RECEIVED)) != IPCC_NOBODY && rang >> 16 == IPCC_LPASS; i++) {+ put(&ipcc, "heard", IPCC_CLEAR, rang);+ }+ put(&ipcc, "listened for", IPCC_ENABLE, IPCC_LPASS << 16 | IPCC_SMP2P);+ put(&ipcc, "listened for", IPCC_ENABLE, IPCC_LPASS << 16 | IPCC_GLINK);+ bzero(&doorbell, sizeof(doorbell));+ lck_mtx_lock(&dsp_lock);+ if (!rails(384)) {+ lck_mtx_unlock(&dsp_lock);+ return false;+ }+ rails_asked = true;+ if (!mailbox_send("{class: image, res: load_state, name: adsp, val: on}")) {+ rails(0);+ rails_asked = false;+ lck_mtx_unlock(&dsp_lock);+ return false;+ }+ result = scm("to authenticate the audio DSP's image and start it", SCM_PAS_AUTH_AND_RESET, SCM_ONE_VALUE, PAS_ADSP, 0, 0,+ &answer);+ doorbell.since = started = now_us();+ image_written = false;+ if (result != 0 || answer != 0) {+ printf("adsp: %u: TRUSTZONE DOES NOT START THE AUDIO DSP: the call returns %d, its answer is %#llx\n", step, result,+ answer);+ mailbox_send("{class: image, res: load_state, name: adsp, val: off}");+ rails(0);+ rails_asked = false;+ lck_mtx_unlock(&dsp_lock);+ return false;+ }+ dsp_started = true;+ lck_mtx_unlock(&dsp_lock);+ printf("adsp: %u: TRUSTZONE HAS AUTHENTICATED THE IMAGE AND LET THE AUDIO DSP OUT OF RESET; watching for %u s\n", step,+ wait_seconds);+ limit = started + wait_seconds * 1000000ull;+ for (;;) {+ now = now_us();+ if (now >= limit || !dsp_started) {+ break;+ }+ doorbells();+ spoke = spoke || doorbell.glink + doorbell.smp2p != 0;+ if (dsp_words(&words) && (!have || memcmp(&words, &last, sizeof(words)) != 0)) {+ char when[64];++ snprintf(when, sizeof(when), "+%llu.%03llu ms: the DSP's item %s", (now - started) / 1000, (now - started) % 1000,+ have ? "changes" : "APPEARS");+ if (changes++ < 40) {+ show_words(when, &words);+ }+ last = words;+ have = spoke = true;+ dsp_listen(&words);+ state = dsp_state(&words);+ if ((state & DSP_READY) && ready_at == 0) {+ ready_at = now;+ limit = now + 3000000ull;+ printf("adsp: %u: THE AUDIO DSP SAYS IT IS UP: \"slave-kernel\" says %#x, %llu.%03llu ms after TrustZone started "+ "it\n", step, state, (now - started) / 1000, (now - started) % 1000);+ }+ if ((state & DSP_HANDOVER) && !handed_over) {+ handed_over = true;+ printf("adsp: %u: the DSP says it has asked for its own power (bit 2), %llu.%03llu ms after the start: ours is "+ "given back\n", step, (now - started) / 1000, (now - started) % 1000);+ lck_mtx_lock(&dsp_lock);+ if (rails_asked) {+ rails(0);+ rails_asked = false;+ }+ lck_mtx_unlock(&dsp_lock);+ }+ if (state & DSP_FATAL) {+ printf("adsp: %u: THE AUDIO DSP SAYS IT HAS GIVEN UP\n", step);+ show_crash_reason();+ break;+ }+ }+ sleep_us(200);+ }+ printf("adsp: %u: the watch is over after %llu ms; its doorbells heard: GLINK %u, SMP2P %u\n", step,+ (now_us() - started) / 1000, doorbell.glink, doorbell.smp2p);+ show_partition();+ if (our_item != 0 && smem_peek(our_item, &words, sizeof(words))) {+ show_words("our item (429) now", &words);+ }+ {+ /* (what is at the bottom of our partition, entry by entry, as it is now) */+ uint32_t cacheline = 0, size = 0, partition = smem_partition(SMEM_US, SMEM_ADSP, &cacheline, &size);+ struct smem_partition header;+ struct smem_entry entry;++ if (partition != 0 && smem_peek(partition, &header, sizeof(header))) {+ printf("adsp: %u: the partition's header: hosts %u and %u, size %#x, free from the bottom at +%#x, from the top at "+ "+%#x; the table's line size %u\n", step, header.host0, header.host1, header.size, header.free_uncached,+ header.free_cached, cacheline);+ for (uint32_t at = sizeof(header), n = 0; at + sizeof(entry) <= header.size && n < 8; n++) {+ smem_peek(partition + at, &entry, sizeof(entry));+ printf("adsp: %u: +%#x: canary %#x item %u size %u padding %u/%u\n", step, at, entry.canary, entry.item,+ entry.size, entry.padding_data, entry.padding_header);+ if (entry.canary != SMEM_CANARY) {+ break;+ }+ at += sizeof(entry) + entry.padding_header + entry.size;+ }+ }+ }+ if (state & DSP_FATAL) {+ return false;+ }+ if (ready_at == 0) {+ printf("adsp: %u: %s\n", step, spoke ? "THE AUDIO DSP RUNS BUT HAS NOT SAID \"UP\"" : "NOT A WORD FROM THE AUDIO DSP");+ show_crash_reason();+ }+ return ready_at != 0;+}++/* ---- GLINK (redwood_wifi.c; items 478 to 480 of our partition with the audio DSP) ------------------- */++#define ITEM_GLINK_DESCRIPTOR 478+#define ITEM_GLINK_OUR_QUEUE 479+#define ITEM_GLINK_THEIR_QUEUE 480+#define GLINK_OUR_TAIL 0+#define GLINK_OUR_HEAD 4+#define GLINK_THEIR_TAIL 8+#define GLINK_THEIR_HEAD 12+#define GLINK_RESERVE 16++enum {+ GLINK_VERSION = 0, GLINK_VERSION_ACK = 1, GLINK_OPEN = 2, GLINK_CLOSE = 3, GLINK_OPEN_ACK = 4, GLINK_INTENT = 5,+ GLINK_DONE = 6, GLINK_INTENT_REQUEST = 7, GLINK_INTENT_REQUEST_ACK = 8, GLINK_DATA = 9, GLINK_CLOSE_ACK = 11,+ GLINK_DATA_MORE = 12, GLINK_READ_NOTIFY = 13, GLINK_DONE_REUSE = 14, GLINK_SIGNALS = 15,+};+#define GLINK_VERSION_1 1+#define GLINK_INTENT_REUSE 1u+#define GLINK_CHANNEL "IPCRTR"+#define GLINK_US 1++struct glink_message {+ uint16_t command, param1;+ uint32_t param2;+};+struct glink_packet {+ struct glink_packet *next;+ uint32_t length;+ uint8_t data[];+};++static LCK_MTX_DECLARE(glink_lock, &adsp_lock_group);+static struct {+ uint32_t descriptor, ours, ours_size, theirs, theirs_size;+ bool up, broken;+ uint32_t features;+ bool agreed;+ uint32_t their_channel;+ bool asked, acknowledged, open;+ bool request_answered, request_granted;+ struct {+ uint32_t id, size;+ bool there, in_use;+ } their_intents[64];+ struct {+ uint32_t size, fill;+ uint8_t *data;+ } our_intents[24];+ uint32_t our_intent_count;+ struct glink_packet *first, *last;+ uint32_t said, heard, trace;+ char others[4][33]; /* the channels the DSP opens that nobody here speaks (for the log) */+} glink;++static bool+glink_find(void)+{+ uint32_t size = 0;++ glink.descriptor = smem_item(ITEM_GLINK_DESCRIPTOR, &size);+ if (glink.descriptor == 0 || size != 32) {+ return false;+ }+ glink.ours = smem_item(ITEM_GLINK_OUR_QUEUE, &glink.ours_size);+ glink.theirs = smem_item(ITEM_GLINK_THEIR_QUEUE, &glink.theirs_size);+ return glink.ours != 0 && glink.theirs != 0 && glink.ours_size >= 1024 && glink.theirs_size >= 1024 &&+ !(glink.ours_size & 7) && !(glink.theirs_size & 7);+}++static void+glink_reset(void)+{+ uint32_t size = 0, descriptor = smem_item(ITEM_GLINK_DESCRIPTOR, &size);++ if (descriptor == 0 || size != 32) {+ printf("adsp: %u: GLINK's descriptor (item %u) is not there: nothing to reset\n", step, ITEM_GLINK_DESCRIPTOR);+ return;+ }+ printf("adsp: %u: GLINK's descriptor at %#lx: our queue read to %#x and written to %#x, the DSP's read to %#x and written "+ "to %#x\n", step, SMEM_PHYS + descriptor, smem_word(descriptor + GLINK_OUR_TAIL), smem_word(descriptor + GLINK_OUR_HEAD),+ smem_word(descriptor + GLINK_THEIR_TAIL), smem_word(descriptor + GLINK_THEIR_HEAD));+ if (smem_word(descriptor + GLINK_OUR_HEAD) != 0 || smem_word(descriptor + GLINK_THEIR_TAIL) != 0) {+ printf("adsp: %u: about to write our two marks there (+4, +8): 0\n", step);+ smem_poke(descriptor + GLINK_OUR_HEAD, 0);+ smem_poke(descriptor + GLINK_THEIR_TAIL, 0);+ settle();+ }+}++static uint32_t+glink_waiting(void)+{+ uint32_t head = smem_word(glink.descriptor + GLINK_THEIR_HEAD), tail = smem_word(glink.descriptor + GLINK_THEIR_TAIL);++ if (head >= glink.theirs_size || tail >= glink.theirs_size) {+ return 0;+ }+ return head >= tail ? head - tail : glink.theirs_size - tail + head;+}++static void+glink_peek(void *to, uint32_t offset, uint32_t count)+{+ uint32_t at = (smem_word(glink.descriptor + GLINK_THEIR_TAIL) + offset) % glink.theirs_size;++ for (uint32_t i = 0; i < count; i++) {+ ((uint8_t *)to)[i] = smem[glink.theirs + (at + i) % glink.theirs_size];+ }+}++static void+glink_advance(uint32_t count)+{+ smem_poke(glink.descriptor + GLINK_THEIR_TAIL, (smem_word(glink.descriptor + GLINK_THEIR_TAIL) + count) % glink.theirs_size);+}++static uint32_t+glink_room(void)+{+ uint32_t head = smem_word(glink.descriptor + GLINK_OUR_HEAD), tail = smem_word(glink.descriptor + GLINK_OUR_TAIL), room;++ if (head >= glink.ours_size || tail >= glink.ours_size) {+ return 0;+ }+ room = tail <= head ? glink.ours_size - head + tail : tail - head;+ return room < GLINK_RESERVE ? 0 : room - GLINK_RESERVE;+}++static bool+glink_put(const void *first, uint32_t first_length, const void *second, uint32_t second_length)+{+ volatile uint8_t *queue = (volatile uint8_t *)(uintptr_t)smem + glink.ours;+ uint32_t head = smem_word(glink.descriptor + GLINK_OUR_HEAD), length = first_length + second_length;+ uint32_t padded = (length + 7) & ~7u;++ if (!glink.up || glink_room() < padded) {+ return false;+ }+ for (uint32_t i = 0; i < padded; i++) {+ queue[(head + i) % glink.ours_size] = i < first_length ? ((const uint8_t *)first)[i] :+ i < length ? ((const uint8_t *)second)[i - first_length] : 0;+ }+ settle();+ smem_poke(glink.descriptor + GLINK_OUR_HEAD, (head + padded) % glink.ours_size);+ settle();+ wr(ipcc.base + IPCC_SEND, IPCC_LPASS << 16 | IPCC_GLINK);+ glink.said++;+ return true;+}++static const char *+glink_name(unsigned command)+{+ static const char *const names[] = {+ "version", "version-ack", "open", "close", "open-ack", "intent", "done", "intent-request", "intent-request-ack",+ "data", "10", "close-ack", "data-more", "read-notify", "done-reuse", "signals"+ };++ return command < sizeof(names) / sizeof(names[0]) ? names[command] : "?";+}++static void+glink_say(unsigned command, uint32_t param1, uint32_t param2, const void *more, uint32_t more_length)+{+ struct glink_message message = { .command = (uint16_t)command, .param1 = (uint16_t)param1, .param2 = param2 };+ bool sent = glink_put(&message, sizeof(message), more, more_length);++ if (glink.trace > 0 || !sent) {+ glink.trace -= glink.trace > 0;+ printf("adsp: glink: > %s %u %#x%s\n", glink_name(command), param1, param2, sent ? "" : ": NO ROOM IN OUR QUEUE, not sent");+ }+}++static bool+glink_offer(uint32_t size)+{+ uint32_t words[2] = { size, glink.our_intent_count + 1 }, i = glink.our_intent_count;++ if (i >= sizeof(glink.our_intents) / sizeof(glink.our_intents[0]) || size == 0 || size > 0x10000) {+ return false;+ }+ if (glink.our_intents[i].data == NULL || glink.our_intents[i].size != size) {+ glink.our_intents[i].data = kalloc_data(size, Z_WAITOK | Z_ZERO);+ }+ if (glink.our_intents[i].data == NULL) {+ return false;+ }+ glink.our_intents[i].size = size;+ glink.our_intents[i].fill = 0;+ glink.our_intent_count++;+ glink_say(GLINK_INTENT, GLINK_US, 1, words, sizeof(words));+ return true;+}++/* Everything the DSP has put into its queue, taken out and acted on (Linux's qcom_glink_native_rx()). With glink_lock held. */+static void+glink_pump(void)+{+ static const struct {+ uint32_t size, count;+ } intents[] = { { 0x800, 5 }, { 0x2000, 3 }, { 0x4400, 2 } }; /* dt: qcom,intents of qcom,adsp_qrtr */+ struct glink_message message;+ uint32_t words[4];+ char name[33];++ while (glink.up && !glink.broken) {+ uint32_t waiting = glink_waiting(), length = sizeof(message);++ if (waiting < sizeof(message)) {+ return;+ }+ glink_peek(&message, 0, sizeof(message));+ if (message.command == GLINK_OPEN) {+ length += message.param2;+ } else if (message.command == GLINK_INTENT) {+ length += 8 * message.param2;+ } else if (message.command == GLINK_DATA || message.command == GLINK_DATA_MORE) {+ if (waiting < 16) {+ return;+ }+ glink_peek(words, 8, 8);+ length = 16 + words[0];+ }+ length = (length + 7) & ~7u;+ if (length > glink.theirs_size - 8) {+ printf("adsp: glink: A MESSAGE OF %u BYTES (%s %u %#x): not believed, and nothing more is read\n", length,+ glink_name(message.command), message.param1, message.param2);+ glink.broken = true;+ return;+ }+ if (waiting < length) {+ return;+ }+ glink.heard++;+ if (glink.trace > 0 && message.command != GLINK_DATA && message.command != GLINK_DATA_MORE) {+ glink.trace--;+ printf("adsp: glink: < %s %u %#x\n", glink_name(message.command), message.param1, message.param2);+ }+ switch (message.command) {+ case GLINK_VERSION:+ if (message.param1 == GLINK_VERSION_1) {+ glink.features &= message.param2;+ }+ glink_advance(length);+ glink_say(GLINK_VERSION_ACK, GLINK_VERSION_1, glink.features, NULL, 0);+ break;+ case GLINK_VERSION_ACK:+ glink_advance(length);+ if (message.param1 == GLINK_VERSION_1 && message.param2 == glink.features) {+ glink.agreed = true;+ } else if (message.param1 != 0) {+ glink.features &= message.param2;+ glink_say(GLINK_VERSION, GLINK_VERSION_1, glink.features, NULL, 0);+ }+ break;+ case GLINK_OPEN:+ bzero(name, sizeof(name));+ glink_peek(name, sizeof(message), message.param2 < sizeof(name) - 1 ? message.param2 : sizeof(name) - 1);+ glink_advance(length);+ if (strcmp(name, GLINK_CHANNEL) != 0) {+ printf("adsp: glink: the DSP opens \"%s\" as its channel %u: left unanswered, nobody here speaks it\n", name,+ message.param1);+ for (unsigned i = 0; i < 4; i++) {+ if (glink.others[i][0] == 0) {+ strlcpy(glink.others[i], name, sizeof(glink.others[i]));+ break;+ }+ }+ break;+ }+ printf("adsp: glink: the DSP opens \"%s\" as its channel %u: acknowledged, and opened from our side as channel %u\n",+ name, message.param1, GLINK_US);+ glink.their_channel = message.param1;+ glink_say(GLINK_OPEN_ACK, glink.their_channel, 0, NULL, 0);+ if (!glink.asked) {+ bzero(name, sizeof(name));+ strlcpy(name, GLINK_CHANNEL, sizeof(name));+ glink_say(GLINK_OPEN, GLINK_US, sizeof(GLINK_CHANNEL), name, (sizeof(GLINK_CHANNEL) + 7) & ~7u);+ glink.asked = true;+ }+ break;+ case GLINK_OPEN_ACK:+ glink_advance(length);+ if (message.param1 == GLINK_US) {+ glink.acknowledged = true;+ }+ break;+ case GLINK_INTENT:+ for (uint32_t i = 0; i < message.param2 && message.param1 == glink.their_channel; i++) {+ glink_peek(words, 8 + 8 * i, 8);+ for (unsigned n = 0; n < sizeof(glink.their_intents) / sizeof(glink.their_intents[0]); n++) {+ if (!glink.their_intents[n].there || glink.their_intents[n].id == words[1]) {+ glink.their_intents[n].there = true;+ glink.their_intents[n].in_use = false;+ glink.their_intents[n].size = words[0];+ glink.their_intents[n].id = words[1];+ break;+ }+ }+ }+ glink_advance(length);+ break;+ case GLINK_DONE:+ case GLINK_DONE_REUSE:+ for (unsigned n = 0; n < sizeof(glink.their_intents) / sizeof(glink.their_intents[0]); n++) {+ if (glink.their_intents[n].there && glink.their_intents[n].id == message.param2) {+ glink.their_intents[n].in_use = false;+ glink.their_intents[n].there = message.command == GLINK_DONE_REUSE;+ }+ }+ glink_advance(length);+ break;+ case GLINK_INTENT_REQUEST:+ glink_advance(length);+ if (message.param1 == glink.their_channel) {+ bool granted = glink_offer(message.param2);++ glink_say(GLINK_INTENT_REQUEST_ACK, GLINK_US, granted, NULL, 0);+ }+ break;+ case GLINK_INTENT_REQUEST_ACK:+ glink_advance(length);+ glink.request_answered = true;+ glink.request_granted = message.param2 != 0;+ break;+ case GLINK_DATA:+ case GLINK_DATA_MORE: {+ uint32_t intent = message.param2 - 1;++ if (message.param1 != glink.their_channel || intent >= glink.our_intent_count ||+ words[0] > glink.our_intents[intent].size - glink.our_intents[intent].fill) {+ printf("adsp: glink: < data of %u bytes on channel %u for intent %u: NOT OURS OR TOO MUCH, dropped\n", words[0],+ message.param1, message.param2);+ glink_advance(length);+ break;+ }+ glink_peek(glink.our_intents[intent].data + glink.our_intents[intent].fill, 16, words[0]);+ glink.our_intents[intent].fill += words[0];+ glink_advance(length);+ if (words[1] == 0) {+ struct glink_packet *packet = kalloc_data(sizeof(*packet) + glink.our_intents[intent].fill, Z_WAITOK);++ if (packet != NULL) {+ packet->next = NULL;+ packet->length = glink.our_intents[intent].fill;+ memcpy(packet->data, glink.our_intents[intent].data, packet->length);+ if (glink.last != NULL) {+ glink.last->next = packet;+ } else {+ glink.first = packet;+ }+ glink.last = packet;+ }+ glink.our_intents[intent].fill = 0;+ glink_say(GLINK_DONE_REUSE, GLINK_US, message.param2, NULL, 0);+ }+ break;+ }+ case GLINK_READ_NOTIFY:+ glink_advance(length);+ wr(ipcc.base + IPCC_SEND, IPCC_LPASS << 16 | IPCC_GLINK);+ break;+ case GLINK_CLOSE:+ glink_advance(length);+ glink_say(GLINK_CLOSE_ACK, message.param1, 0, NULL, 0);+ if (message.param1 == glink.their_channel) {+ printf("adsp: glink: THE DSP CLOSES \"%s\"\n", GLINK_CHANNEL);+ glink.open = false;+ glink.their_channel = 0;+ }+ break;+ case GLINK_CLOSE_ACK:+ case GLINK_SIGNALS:+ glink_advance(length);+ break;+ default:+ printf("adsp: glink: A MESSAGE THIS DOES NOT KNOW (%u, %u, %#x): nothing more is read\n", message.command,+ message.param1, message.param2);+ glink.broken = true;+ return;+ }+ if (glink.their_channel != 0 && glink.acknowledged && !glink.open) {+ glink.open = true;+ printf("adsp: glink: THE CHANNEL \"%s\" IS OPEN both ways (ours %u, the DSP's %u); telling the DSP our intents\n",+ GLINK_CHANNEL, GLINK_US, glink.their_channel);+ for (unsigned i = 0; i < sizeof(intents) / sizeof(intents[0]); i++) {+ for (unsigned n = 0; n < intents[i].count; n++) {+ glink_offer(intents[i].size);+ }+ }+ }+ }+}++/* One whole message to the DSP on the channel; waits up to five seconds for an intent of the DSP's that it fits. */+static int+glink_send(const void *data, uint32_t length)+{+ struct {+ struct glink_message message;+ uint32_t size, left;+ } header = { { GLINK_DATA, GLINK_US, 0 }, length, 0 };+ bool asked = false, fits;+ int found = -1;++ lck_mtx_lock(&glink_lock);+ if (glink.up && length + sizeof(header) + 8 > glink.ours_size - GLINK_RESERVE) {+ lck_mtx_unlock(&glink_lock);+ return EMSGSIZE;+ }+ for (unsigned waited = 0;; waited++) {+ glink_pump();+ if (!glink.up || !glink.open || glink.broken) {+ lck_mtx_unlock(&glink_lock);+ return ENOTCONN;+ }+ found = -1;+ fits = false;+ for (unsigned n = 0; n < sizeof(glink.their_intents) / sizeof(glink.their_intents[0]); n++) {+ if (!glink.their_intents[n].there || glink.their_intents[n].size < length) {+ continue;+ }+ fits = true;+ if (!glink.their_intents[n].in_use && (found < 0 || glink.their_intents[n].size < glink.their_intents[found].size)) {+ found = (int)n;+ }+ }+ if (found >= 0 && glink_room() >= ((length + sizeof(header) + 7) & ~7u)) {+ break;+ }+ if (!fits && !asked) {+ glink.request_answered = false;+ glink_say(GLINK_INTENT_REQUEST, GLINK_US, length, NULL, 0);+ asked = true;+ }+ if (waited >= 5000 || (!fits && asked && glink.request_answered && !glink.request_granted)) {+ lck_mtx_unlock(&glink_lock);+ return found < 0 ? ENOBUFS : EAGAIN;+ }+ lck_mtx_unlock(&glink_lock);+ sleep_ms(1);+ lck_mtx_lock(&glink_lock);+ }+ glink.their_intents[found].in_use = true;+ header.message.param2 = glink.their_intents[found].id;+ glink_put(&header, sizeof(header), data, length);+ lck_mtx_unlock(&glink_lock);+ return 0;+}++/* ---- QRTR: hello, and the services the DSP tells of (redwood_wifi.c, without ports of our own yet) ---- */++#define QRTR_US 1u+#define QRTR_CONTROL 0xfffffffeu+enum { QRTR_DATA = 1, QRTR_HELLO = 2, QRTR_BYE = 3, QRTR_NEW_SERVER = 4, QRTR_DEL_SERVER = 5, QRTR_DEL_CLIENT = 6, QRTR_RESUME_TX = 7,+ QRTR_NEW_LOOKUP = 10, QRTR_DEL_LOOKUP = 11 };++struct qrtr_header {+ uint32_t version, type, from_node, from_port, confirm, size, to_node, to_port;+};+struct qrtr_control {+ uint32_t command;+ uint32_t words[4];+};++static struct {+ uint32_t service, instance, node, port;+ bool there;+} qrtr_servers[64];+static uint32_t qrtr_peer, qrtr_in, qrtr_out, qrtr_hellos, qrtr_lookups;++static int+qrtr_out_packet(uint32_t type, uint32_t from_port, uint32_t to_node, uint32_t to_port, const void *data, uint32_t length)+{+ uint32_t padded = (length + 3) & ~3u;+ struct qrtr_header *packet;+ int error;++ packet = kalloc_data(sizeof(*packet) + padded, Z_WAITOK | Z_ZERO);+ if (packet == NULL) {+ return ENOMEM;+ }+ packet->version = 1;+ packet->type = type;+ packet->from_node = QRTR_US;+ packet->from_port = from_port;+ packet->size = length;+ packet->to_node = to_node;+ packet->to_port = to_port;+ memcpy(packet + 1, data, length);+ error = glink_send(packet, (uint32_t)sizeof(*packet) + padded);+ kfree_data(packet, sizeof(*packet) + padded);+ qrtr_out += error == 0;+ return error;+}++static void+qrtr_control(const struct qrtr_header *header, const struct qrtr_control *control)+{+ struct qrtr_control hello = { .command = QRTR_HELLO };+ int found = -1, error;++ switch (control->command) {+ case QRTR_HELLO:+ if (qrtr_hellos++ != 0) {+ break;+ }+ error = qrtr_out_packet(QRTR_HELLO, QRTR_CONTROL, header->from_node, QRTR_CONTROL, &hello, sizeof(hello));+ printf("adsp: qrtr: HELLO FROM NODE %u; ours to it: %s\n", header->from_node, error == 0 ? "SENT" : "NOT SENT");+ break;+ case QRTR_NEW_SERVER:+ for (unsigned i = 0; i < sizeof(qrtr_servers) / sizeof(qrtr_servers[0]); i++) {+ if (qrtr_servers[i].there && qrtr_servers[i].node == control->words[2] && qrtr_servers[i].port == control->words[3]) {+ found = (int)i;+ break;+ }+ if (!qrtr_servers[i].there && found < 0) {+ found = (int)i;+ }+ }+ if (found >= 0) {+ qrtr_servers[found].there = true;+ qrtr_servers[found].service = control->words[0];+ qrtr_servers[found].instance = control->words[1];+ qrtr_servers[found].node = control->words[2];+ qrtr_servers[found].port = control->words[3];+ }+ printf("adsp: qrtr: a service: %u (%#x), version %u, instance %u, at node %u port %u\n", control->words[0],+ control->words[0], control->words[1] & 0xff, control->words[1] >> 8, control->words[2], control->words[3]);+ break;+ case QRTR_DEL_SERVER:+ printf("adsp: qrtr: service %u at node %u port %u has gone\n", control->words[0], control->words[2], control->words[3]);+ for (unsigned i = 0; i < sizeof(qrtr_servers) / sizeof(qrtr_servers[0]); i++) {+ if (qrtr_servers[i].there && qrtr_servers[i].node == control->words[2] && qrtr_servers[i].port == control->words[3]) {+ qrtr_servers[i].there = false;+ }+ }+ break;+ case QRTR_NEW_LOOKUP:+ /* the DSP looks for a service of ours (a protection domain's locator, say): there are none yet */+ if (qrtr_lookups++ < 16) {+ printf("adsp: qrtr: the DSP looks for service %u (%#x), instance %u: nobody here offers it\n", control->words[0],+ control->words[0], control->words[1]);+ }+ break;+ case QRTR_BYE:+ printf("adsp: qrtr: BYE from node %u\n", header->from_node);+ bzero(qrtr_servers, sizeof(qrtr_servers));+ break;+ default:+ break;+ }+}++static void+qrtr_input(const uint8_t *data, uint32_t length)+{+ struct qrtr_header header;+ uint32_t at;++ if (length >= sizeof(header) && data[0] == 1) {+ memcpy(&header, data, sizeof(header));+ at = sizeof(header);+ } else if (length >= 16 && data[0] == 3) {+ bzero(&header, sizeof(header));+ header.type = data[1];+ header.confirm = data[2] & 1;+ memcpy(&header.size, data + 4, 4);+ header.from_node = (uint32_t)(data[8] | data[9] << 8);+ header.from_port = (uint32_t)(data[10] | data[11] << 8);+ header.to_node = (uint32_t)(data[12] | data[13] << 8);+ header.to_port = (uint32_t)(data[14] | data[15] << 8);+ header.from_port = header.from_port == 0xfffe ? QRTR_CONTROL : header.from_port;+ header.to_port = header.to_port == 0xfffe ? QRTR_CONTROL : header.to_port;+ at = 16 + data[3];+ } else {+ printf("adsp: qrtr: %u bytes that are no datagram (they begin %#x)\n", length, length ? data[0] : 0);+ return;+ }+ if (header.size == 0 || at + header.size > length) {+ printf("adsp: qrtr: a datagram of %u bytes in a message of %u: dropped\n", header.size, length);+ return;+ }+ qrtr_in++;+ if (qrtr_peer == 0) {+ qrtr_peer = header.from_node;+ }+ if (header.type == QRTR_RESUME_TX) {+ return;+ }+ if (header.confirm) {+ struct qrtr_control resume = { .command = QRTR_RESUME_TX, .words = { header.to_node, header.to_port } };++ qrtr_out_packet(QRTR_RESUME_TX, header.to_port, header.from_node, header.from_port, &resume, sizeof(resume));+ }+ if (header.to_port == QRTR_CONTROL || header.type == QRTR_HELLO || header.type == QRTR_BYE ||+ header.type == QRTR_NEW_SERVER || header.type == QRTR_DEL_SERVER || header.type == QRTR_NEW_LOOKUP) {+ struct qrtr_control control = { 0 };++ memcpy(&control, data + at, header.size < sizeof(control) ? header.size : sizeof(control));+ qrtr_control(&header, &control);+ return;+ }+ printf("adsp: qrtr: a datagram of %u bytes from node %u port %u to our port %u: nobody here listens there\n", header.size,+ header.from_node, header.from_port, header.to_port);+}++/* ---- the link's thread ----------------------------------------------------------------------------- */++static volatile bool link_wanted, link_runs;++static void+link_thread(__unused void *parameter, __unused wait_result_t result)+{+ struct smp2p words, last;+ struct glink_packet *packet;+ bool gave_up = false;++ bzero(&last, sizeof(last));+ dsp_words(&last);+ while (link_wanted) {+ sleep_us(1000);+ doorbells();+ if (dsp_words(&words) && memcmp(&words, &last, sizeof(words)) != 0) {+ last = words;+ dsp_listen(&words);+ if ((dsp_state(&words) & DSP_FATAL) && !gave_up) {+ gave_up = true;+ printf("adsp: link: THE AUDIO DSP SAYS IT HAS GIVEN UP (\"slave-kernel\" = %#x)\n", dsp_state(&words));+ show_crash_reason();+ }+ }+ for (;;) {+ lck_mtx_lock(&glink_lock);+ glink_pump();+ packet = glink.first;+ if (packet != NULL) {+ glink.first = packet->next;+ glink.last = glink.first == NULL ? NULL : glink.last;+ }+ lck_mtx_unlock(&glink_lock);+ if (packet == NULL) {+ break;+ }+ qrtr_input(packet->data, packet->length);+ kfree_data(packet, sizeof(*packet) + packet->length);+ }+ }+ link_runs = false;+ thread_terminate(current_thread());+}++static void+link_stop(void)+{+ struct glink_packet *packet;++ if (link_runs) {+ link_wanted = false;+ for (unsigned ms = 0; ms < 2000 && link_runs; ms++) {+ sleep_ms(1);+ }+ printf("adsp: link: %s; GLINK said %u and heard %u messages, QRTR sent %u and received %u datagrams; its doorbells heard: "+ "GLINK %u, SMP2P %u\n", link_runs ? "THE THREAD DOES NOT END" : "down", glink.said, glink.heard, qrtr_out, qrtr_in,+ doorbell.glink, doorbell.smp2p);+ }+ lck_mtx_lock(&glink_lock);+ glink.up = glink.open = false;+ while ((packet = glink.first) != NULL) {+ glink.first = packet->next;+ kfree_data(packet, sizeof(*packet) + packet->length);+ }+ glink.last = NULL;+ lck_mtx_unlock(&glink_lock);+ bzero(qrtr_servers, sizeof(qrtr_servers));+ qrtr_peer = 0;+}++/* ---- step 8: GLINK's queues, looked at -------------------------------------------------------------- */++static bool+step_queues(void)+{+ struct glink_message message;+ uint32_t head, tail;++ announce("find GLINK's descriptor and the two queues, and read what the audio DSP has put into its own");+ if (!glink_find()) {+ printf("adsp: %u: GLINK's items are not all there, or not of a shape this knows (descriptor %#x, ours %#x of %u bytes, "+ "the DSP's %#x of %u)\n", step, glink.descriptor, glink.ours, glink.ours_size, glink.theirs, glink.theirs_size);+ return false;+ }+ head = smem_word(glink.descriptor + GLINK_THEIR_HEAD);+ tail = smem_word(glink.descriptor + GLINK_THEIR_TAIL);+ printf("adsp: %u: our queue: %u bytes at %#lx, written to %#x, read by the DSP to %#x; the DSP's: %u bytes at %#lx, written "+ "to %#x, read by us to %#x\n", step, glink.ours_size, SMEM_PHYS + glink.ours, smem_word(glink.descriptor + GLINK_OUR_HEAD),+ smem_word(glink.descriptor + GLINK_OUR_TAIL), glink.theirs_size, SMEM_PHYS + glink.theirs, head, tail);+ for (uint32_t at = 0, shown = 0; at + sizeof(message) <= head && head < glink.theirs_size && tail == 0 && shown < 8; shown++) {+ char name[33] = "";+ uint32_t length = sizeof(message);++ glink_peek(&message, at, sizeof(message));+ if (message.command == GLINK_OPEN) {+ glink_peek(name, at + 8, message.param2 < 32 ? message.param2 : 32);+ length += message.param2;+ }+ printf("adsp: %u: in the DSP's queue at %#x: %s %u %#x %s\n", step, at, glink_name(message.command), message.param1,+ message.param2, name);+ if (message.command != GLINK_VERSION && message.command != GLINK_OPEN) {+ break;+ }+ at += (length + 7) & ~7u;+ }+ return true;+}++/* ---- step 9: GLINK, spoken -------------------------------------------------------------------------- */++static bool+step_link(void)+{+ thread_t thread;+ uint64_t began;++ if (!dsp_started || glink.descriptor == 0) {+ return false;+ }+ announce("take up GLINK with the audio DSP: our version, then whatever its messages call for; a thread of ours listens "+ "from here on, every millisecond");+ snprintf(last_write, sizeof(last_write), "step %u took up GLINK", step);+ lck_mtx_lock(&glink_lock);+ glink.features = GLINK_INTENT_REUSE;+ glink.agreed = glink.asked = glink.acknowledged = glink.open = glink.broken = false;+ glink.their_channel = glink.our_intent_count = glink.said = glink.heard = 0;+ bzero(glink.their_intents, sizeof(glink.their_intents));+ bzero(glink.others, sizeof(glink.others));+ glink.trace = 60;+ glink.up = true;+ qrtr_hellos = qrtr_in = qrtr_out = qrtr_lookups = 0;+ doorbell.since = began = now_us();+ glink_say(GLINK_VERSION, GLINK_VERSION_1, glink.features, NULL, 0);+ lck_mtx_unlock(&glink_lock);+ link_wanted = link_runs = true;+ if (kernel_thread_start(link_thread, NULL, &thread) != KERN_SUCCESS) {+ link_wanted = link_runs = false;+ printf("adsp: %u: no thread\n", step);+ return false;+ }+ thread_deallocate(thread);+ for (unsigned ms = 0; ms < 5000 && !glink.open && !glink.broken; ms++) {+ sleep_ms(1);+ }+ sleep_ms(50);+ printf("adsp: %u: %llu ms after our version: versions %s (features %#x); the channel \"%s\" is %s; the DSP also opened "+ "\"%s\" \"%s\" \"%s\" \"%s\"\n", step, (now_us() - began) / 1000, glink.agreed ? "agreed" : "NOT AGREED", glink.features,+ GLINK_CHANNEL, glink.open ? "OPEN" : "NOT OPEN", glink.others[0], glink.others[1], glink.others[2], glink.others[3]);+ return glink.open;+}++/* ---- step 10: QRTR ---------------------------------------------------------------------------------- */++static bool+step_qrtr(void)+{+ unsigned services = 0;++ if (!glink.open) {+ return false;+ }+ PE_parse_boot_argn("adsp_listen", &listen_seconds, sizeof(listen_seconds));+ announce("listen to QRTR on the channel: the DSP's hello (answered from the link's thread), and the services it tells of");+ for (unsigned ms = 0; ms < listen_seconds * 1000; ms++) {+ sleep_ms(1);+ }+ for (unsigned i = 0; i < sizeof(qrtr_servers) / sizeof(qrtr_servers[0]); i++) {+ services += qrtr_servers[i].there;+ }+ printf("adsp: %u: after %u s: %s; the DSP is node %u and has told of %u services; it looked for %u of ours; QRTR received %u "+ "and sent %u datagrams\n", step, listen_seconds, qrtr_hellos ? "HELLO BOTH WAYS" : "NO HELLO FROM THE DSP", qrtr_peer,+ services, qrtr_lookups, qrtr_in, qrtr_out);+ return qrtr_hellos != 0;+}++/* ---- the steps, and their end ----------------------------------------------------------------------- */++static bool (*const steps[STEPS])(void) = {+ step_survey, step_firmware, step_mailbox, step_header, step_image, step_our_item, step_start, step_queues, step_link,+ step_qrtr,+};++static bool keep; /* the DSP and the link stay up while the system runs (adsp_keep, adsp=1) */++static void+steps_over(uint32_t done)+{+ lck_mtx_lock(&dsp_lock);+ if (dsp_started && done == last_step && keep) {+ printf("adsp: THE AUDIO DSP STAYS ON while the system runs%s\n", link_runs ? ", and the link with it" : "");+ } else if (dsp_started) {+ dsp_shutdown("the steps are over");+ }+ lck_mtx_unlock(&dsp_lock);+}++/* When the run ends in order: a DSP that runs is shut down, for the next kernel to start again. */+void+redwood_adsp_stop(void)+{+ lck_mtx_lock(&dsp_lock);+ if (dsp_started) {+ unsigned services = 0;+ uint32_t node = qrtr_peer;+ bool down;++ for (unsigned i = 0; i < sizeof(qrtr_servers) / sizeof(qrtr_servers[0]); i++) {+ services += qrtr_servers[i].there;+ }+ down = dsp_shutdown("the run is about to end");+ /* (said again at the end: a long run's log keeps its beginning and its end, and the middle may be gone) */+ tell("adsp: the audio DSP is off: it had been up%s (node %u, %u services), it %s, and TrustZone %s\n", node ?+ " and linked" : ", NOT LINKED", node, services, stopped_by_its_word ? "stopped when asked" :+ "DID NOT SAY IT HAD STOPPED", down ? "has shut it down" : "DID NOT SHUT IT DOWN");+ }+ lck_mtx_unlock(&dsp_lock);+}++/* The same on a panic's way back to the loader: one processor, interrupts off, no lock, no sleep. */+void+redwood_adsp_quiesce(void)+{+ if (!dsp_started) {+ return;+ }+ leaving = true;+ new_from = ~0u;+ dsp_shutdown("the kernel is on its way back to the loader");+}++static void+run_steps(void)+{+ uint32_t done = 0;++ for (step = 1; step <= last_step; step++) {+ if (!steps[step - 1]()) {+ printf("adsp: step %u: FAILED; the steps after it are not attempted\n", step);+ break;+ }+ printf("adsp: step %u: done\n", step);+ done = step;+ }+ step = done;+ new_from = STEPS + 1;+ steps_over(done);+ printf("adsp: over after step %u; %s\n", done, last_write);+}++static void+whole_thread(__unused void *parameter, __unused wait_result_t result)+{+ uint64_t began = now_us();+ unsigned services = 0;++ run_steps();+ for (unsigned i = 0; i < sizeof(qrtr_servers) / sizeof(qrtr_servers[0]); i++) {+ services += qrtr_servers[i].there;+ }+ if (step == STEPS && dsp_started) {+ tell("adsp: THE AUDIO DSP IS UP AND LINKED, %llu ms after the kernel began on it: node %u, %u services; it stays on\n",+ (now_us() - began) / 1000, qrtr_peer, services);+ } else {+ tell("adsp: THE AUDIO DSP DID NOT COME UP: step %u failed (%s); see the kernel's own log for the steps\n", step + 1,+ last_write);+ }+ thread_terminate(current_thread());+}++/* Called while the system starts (bsd/dev/redwood_input.c), after the Wi-Fi's start has been set going. */+void+redwood_adsp_init(void)+{+ uint32_t on = 0;+ thread_t thread;++ PE_parse_boot_argn("adsp_step", &last_step, sizeof(last_step));+ if (last_step == 0 && (!PE_parse_boot_argn("adsp", &on, sizeof(on)) || on != 1)) {+ return;+ }+ if (pe_arm_get_soc_base_phys() != REDWOOD_SOC_BASE) {+ printf("adsp: adsp=1 and adsp_step=N are for the phone's audio DSP, and this is not the phone\n");+ return;+ }+ if (last_step == 0) {+ /* all of it, on a thread of its own, and left up */+ last_step = STEPS;+ keep = true;+ quiet = true;+ pause_ms = 0;+ listen_seconds = 1;+ new_from = STEPS + 1;+ if (kernel_thread_start(whole_thread, NULL, &thread) != KERN_SUCCESS) {+ printf("adsp: no thread: the audio DSP is not brought up\n");+ return;+ }+ thread_deallocate(thread);+ tell("adsp: the audio DSP is being brought up beside the system's start (adsp=1)\n");+ return;+ }+ if (last_step > STEPS) {+ last_step = STEPS;+ }+ keep = option("adsp_keep");+ new_from = last_step;+ PE_parse_boot_argn("adsp_new", &new_from, sizeof(new_from));+ PE_parse_boot_argn("adsp_pause", &pause_ms, sizeof(pause_ms));+ printf("adsp: steps up to %u; those from %u on are new and wait %u ms after each announcement\n", last_step, new_from,+ pause_ms);+ run_steps();+ if (option("adsp_return")) {+ printf("adsp: back to the loader (adsp_return)\n");+ loader_return();+ }+ printf("adsp: the system goes on starting\n");+}diff --git a/xnu/files/bsd/dev/redwood_input.c b/xnu/files/bsd/dev/redwood_input.cindex 8ac7a2a..e2155aa 100644--- a/xnu/files/bsd/dev/redwood_input.c+++ b/xnu/files/bsd/dev/redwood_input.c@@ -175,4 +175,5 @@ redwood_input_init(void)redwood_net_init();redwood_adreno_init();redwood_wifi_init();+ redwood_adsp_init();}diff --git a/xnu/files/bsd/dev/redwood_wifi.c b/xnu/files/bsd/dev/redwood_wifi.cindex ffb13ed..4709e7c 100644--- a/xnu/files/bsd/dev/redwood_wifi.c+++ b/xnu/files/bsd/dev/redwood_wifi.c@@ -182,6 +182,15 @@ extern kern_return_t thread_terminate(thread_t thread);static LCK_GRP_DECLARE(wifi_lock_group, "redwood-wifi");+/*+ * What the audio DSP's start (redwood_adsp.c) uses as well, while this may be at work beside it:+ * the mailbox to the always-on processor, one message at a time, and the allocation of items in+ * the shared memory (whose hardware lock both take under the same number, so it does not keep+ * the two apart).+ */+LCK_MTX_DECLARE(redwood_aop_mailbox_lock, &wifi_lock_group);+LCK_MTX_DECLARE(redwood_smem_alloc_lock, &wifi_lock_group);+static uint32_t last_step, pause_ms = 2200, fine, watch_seconds = 3;static uint32_t step; /* the one being performed */static uint32_t new_from; /* the first of the steps that have not been seen to work: they wait */@@ -1481,8 +1490,21 @@ doorbells(void)* acknowledgement of ours. Whatever of it has been done already -- by a kernel before this one:* the always-on processor is restarted only with the phone -- is found done and left.*/+static bool step_mailbox_open_locked(void);+static boolstep_mailbox_open(void)+{+ bool open;++ lck_mtx_lock(&redwood_aop_mailbox_lock);+ open = step_mailbox_open_locked();+ lck_mtx_unlock(&redwood_aop_mailbox_lock);+ return open;+}++static bool+step_mailbox_open_locked(void){uint32_t box, size;@@ -1541,8 +1563,24 @@ step_mailbox_open(void)}/* A message to the always-on processor (Linux's qmp_send()). */+static bool mailbox_send_locked(const char *text);+static boolmailbox_send(const char *text)+{+ bool sent;++ if (leaving) {+ return mailbox_send_locked(text); /* (one processor left, and no lock taken: see redwood_wifi_quiesce()) */+ }+ lck_mtx_lock(&redwood_aop_mailbox_lock);+ sent = mailbox_send_locked(text);+ lck_mtx_unlock(&redwood_aop_mailbox_lock);+ return sent;+}++static bool+mailbox_send_locked(const char *text){uint32_t words[QMP_MESSAGE / 4] = { 0 };@@ -1937,8 +1975,21 @@ static struct block smem_lock = { "shared memory's lock", 0x01f43000, 0x1000 };#define LOCK_US 1u/* Our item made in the partition, as Linux's qcom_smem_alloc_private() makes one. False, and why said: not made. */+static bool our_item_make_locked(uint32_t partition);+static boolour_item_make(uint32_t partition)+{+ bool made;++ lck_mtx_lock(&redwood_smem_alloc_lock);+ made = our_item_make_locked(partition);+ lck_mtx_unlock(&redwood_smem_alloc_lock);+ return made;+}++static bool+our_item_make_locked(uint32_t partition){struct smem_partition header;struct smem_entry entry;diff --git a/xnu/files/bsd/sys/redwood_input.h b/xnu/files/bsd/sys/redwood_input.hindex ec5a96f..e6fb113 100644--- a/xnu/files/bsd/sys/redwood_input.h+++ b/xnu/files/bsd/sys/redwood_input.h@@ -50,6 +50,7 @@ void redwood_adreno_quiesce(void); /* on the way back to the loader: the GPvoid redwood_adreno_stop(void); /* before that, where the ending is no panic's: the GPU given back in order */void redwood_adreno_minute(void); /* once a minute: a line about how warm the GPU is, if it is in use */void redwood_wifi_init(void); /* boot-arg wifi=1: the phone's Wi-Fi; wifi_step=N: the first N steps of it */+void redwood_adsp_init(void); /* boot-arg adsp=1: the phone's audio DSP started and linked; adsp_step=N: the first N steps */void redwood_wifi_stop(void); /* where a run ends in order: the Wi-Fi DSP, if it was started, shut down */int redwood_wifi_say(const char *format, ...) __printflike(1, 2); /* a line of the Wi-Fi steps' log */int redwood_wifi_tell(const char *format, ...) __printflike(1, 2); /* ... one that is said even with wifi=1 */diff --git a/xnu/files/osfmk/arm64/loader_return.c b/xnu/files/osfmk/arm64/loader_return.cindex e5081aa..ac2f743 100644--- a/xnu/files/osfmk/arm64/loader_return.c+++ b/xnu/files/osfmk/arm64/loader_return.c@@ -49,6 +49,8 @@ extern void redwood_adreno_stop(void);extern void redwood_adreno_minute(void);extern void redwood_wifi_stop(void);extern void redwood_wifi_quiesce(void);+extern void redwood_adsp_stop(void);+extern void redwood_adsp_quiesce(void);void loader_checkpoint(uint64_t n);/* Also referenced from the early exception vector in start.s. */@@ -328,6 +330,7 @@ loader_return(void)#endifredwood_adreno_quiesce(); /* the GPU block, if it was started, has tables in memory that is about to be reused */redwood_wifi_quiesce(); /* the Wi-Fi DSP, if it still runs (a panic), is not left to itself */+ redwood_adsp_quiesce(); /* nor the audio DSP *//** A panic's text is first collected in the panic buffer and only later* flushed to the console; if the console itself is what failed, that@@ -388,6 +391,7 @@ loader_return_timer(__unused thread_call_param_t p0, __unused thread_call_param_if (!as_panic) {redwood_adreno_stop(); /* a GPU kept for the system is given back while there is a thread to wait in */redwood_wifi_stop(); /* and the Wi-Fi DSP, if it runs, shut down */+ redwood_adsp_stop(); /* and the audio DSP */}if (PE_parse_boot_argn("return_cpu", &cpu, sizeof(cpu)) && cpu <= (uint32_t)ml_get_max_cpu_number() &&cpu_xcall((int)cpu, loader_return_test, (void *)(uintptr_t)(as_panic ? 2 : 1)) == KERN_SUCCESS) {@@ -432,6 +436,7 @@ loader_stop_poll(__unused thread_call_param_t p0, __unused thread_call_param_t p}redwood_adreno_stop();redwood_wifi_stop();+ redwood_adsp_stop();loader_return();}clock_interval_to_deadline(1, NSEC_PER_SEC, &deadline);diff --git a/xnu/redwood.patch b/xnu/redwood.patchindex 3d5edd6..dfbb965 100644--- a/xnu/redwood.patch+++ b/xnu/redwood.patch@@ -46,10 +46,10 @@ index 222158a93918..68bb0f3e56bf 100755####################################################################### HISTORYdiff --git a/bsd/conf/files b/bsd/conf/files-index 44274016a4cf..1750e10d3697 100644+index 44274016a4cf..f061b6eb9c2c 100644--- a/bsd/conf/files+++ b/bsd/conf/files-@@ -109,6 +109,24 @@ bsd/dev/munge.c optional config_requires_u32_munging+@@ -109,6 +109,25 @@ bsd/dev/munge.c optional config_requires_u32_mungingbsd/dev/unix_startup.c standardbsd/dev/monotonic.c optional monotonic@@ -67,6 +67,7 @@ index 44274016a4cf..1750e10d3697 100644+bsd/dev/redwood_virtio_sound.c standard+bsd/dev/redwood_speaker.c standard+bsd/dev/redwood_wifi.c standard++bsd/dev/redwood_adsp.c standard+bsd/dev/redwood_qmi.c standard+bsd/dev/redwood_wlfw.c standard+bsd/dev/redwood_lx.c standard@@ -74,7 +75,7 @@ index 44274016a4cf..1750e10d3697 100644bsd/dev/dev_perfmon.c optional config_perfmonbsd/libkern/crc32.c standard-@@ -155,6 +173,10 @@ bsd/miscfs/devfs/devfs_vfsops.c optional devfs+@@ -155,6 +174,10 @@ bsd/miscfs/devfs/devfs_vfsops.c optional devfsbsd/miscfs/mockfs/mockfs_fsnode.c optional mockfsbsd/miscfs/mockfs/mockfs_vfsops.c optional mockfsbsd/miscfs/mockfs/mockfs_vnops.c optional mockfs@@ -85,7 +86,7 @@ index 44274016a4cf..1750e10d3697 100644bsd/miscfs/routefs/routefs_ops.c optional routefs-@@ -361,6 +383,7 @@ bsd/security/audit/audit_syscalls.c standard+@@ -361,6 +384,7 @@ bsd/security/audit/audit_syscalls.c standardbsd/security/audit/audit_worker.c optional config_auditbsd/pthread/pthread_shims.c standard@@ -93,7 +94,7 @@ index 44274016a4cf..1750e10d3697 100644bsd/pthread/pthread_priority.c standardbsd/pthread/pthread_workqueue.c standard-@@ -383,6 +406,7 @@ bsd/kern/kern_debug.c standard+@@ -383,6 +407,7 @@ bsd/kern/kern_debug.c standardbsd/kern/kern_symfile.c standardbsd/kern/kern_descrip.c standardbsd/kern/kern_guarded.c standard