Skip to content

Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.

C

Sound, step 1: the audio DSP started through TrustZone, and linked (GLINK, QRTR)

vcvkk25c19a686dec
diff --git a/README.md b/README.md
index 255462f..718d13c 100644
--- a/README.md
+++ b/README.md
@@ -231,6 +231,7 @@ XL_SECS=900 XL_CMDLINE="touchscreen=1 cpufreq=1 bootcpu=7 cpus=4" ./xl run #
| `wifi_file=0` | с `wifi=1`: сеть из `fw-wifi/network` при старте не подключается, радио ждёт, пока сеть выберут в Настройках (образ с `WIFID=1`) |
| `wifi_irq=0`, `wifi_ba=0`, `wifi_wide=0` | с `wifi=1`: без прерываний радио (его кольца опрашиваются раз в миллисекунду, как раньше), без блочных подтверждений (точка доступа шлёт кадры по одному — скорость впятеро ниже), без широкого канала (только 20 МГц) |
| `wifi_step=N` | то же по шагам, как это выяснялось: первые N шагов, и ядро ждёт, пока они пройдут. 1–5 только читают (общая память процессоров, почтовый ящик процессора питания, RPMh, шины питания, выводы), 6–11 кладут прошивку процессора Wi-Fi в его память, заводят в общей памяти нашу половину его сигнальных битов (SMP2P) и запускают его через TrustZone, 12–14 поднимают связь с ним (GLINK, канал IPCRTR) и QRTR поверх неё: процессор Wi-Fi объявляет свои службы, среди них службу прошивки Wi-Fi; 15 — разговор с этой службой, какой драйвер ведёт до того, как тронет радио (нужны описания сообщений из ath11k: `tools/mk-wlfw.sh`, затем пересобрать ядро): прошивка принимает сообщения ath11k, файл платы и базу регионов телефона, с `wifi_cal=1` калибруется и через 7 секунд говорит «готова»; с `wifi_mission=1` прошивка получает настройку и переходит в рабочий режим; 16 — чтение нескольких регистров радио (`wifi_radio_read=1`): в рабочем режиме они отвечают, вне его чтение кончается ошибкой шины, и шаг его не делает (`adb pull` файлов `wpss.mdt`, `wpss.b02`…`wpss.b07` из `/vendor/firmware_mnt/image` в каталог `fw-wifi/`; в git они не попадают, `userland/build.sh` кладёт их в малый образ). Каждый шаг пишет в журнал, что собирается тронуть и что прочёл. `wifi_keep=1` оставляет процессор Wi-Fi и связь с ним работать, пока идёт сеанс. С `wifi_lx=1` шаг 15 ведёт сам драйвер ath11k из Linux (см. «Драйвер Wi-Fi» ниже; ядро собирается после `tools/mk-ath11k.sh`): его `qmi.c` проходит весь разговор с прошивкой до рабочего режима, со своей настройкой и списком теневых регистров, а регистры радио не трогает. С `wifi_lx=2 wifi_smmu=2` потоку радио в SMMU даётся контекст с нашими таблицами страниц (`wifi_smmu=2` в одиночку — только это, с возвратом как было), `ce.c` и `hal.c` драйвера заводят в регистрах радио кольца его «копировальных машин», `htc.c` подключает по ним службу WMI, и прошивка по WMI рассказывает о радио (одно, две антенные цепи, 2,4 и 5 ГГц) и через четверть секунды после запуска шины сообщает, что готова к командам («unified ready»); собственного адреса у прошивки нет (нули). Прерываний при этом нет: кольца опрашиваются раз в миллисекунду. С `wifi_scan=1` драйвер после этого регистрирует радио, заводит интерфейс в режиме станции и просит прошивку просканировать эфир — только слушая, в эфир ничего не уходит: 26 каналов 2,4 и 5 ГГц за четыре секунды, и в журнале появляются точки доступа вокруг с каналами и уровнем сигнала (имена и адреса чужих сетей — только в журнале запуска на вашей машине). Адрес радио у прошивки свой не записан: он лежит одной строкой в `fw-wifi/mac` (в git не попадает), и `xl run` передаёт его ядру параметром `wifi_mac`. С `wifi_join=1` драйвер подключается к сети из `fw-wifi/network` (WPA2 с паролем): находит её точку доступа, проходит аутентификацию, ассоциацию и четырёхстороннее рукопожатие, отдаёт ключи прошивке и становится тем, что стоит за `en0`; с `wifi_keep=1` он остаётся работать, пока идёт сеанс, без него — сворачивается, когда шаги кончились (`wifi_hold=S` — на S секунд позже). Остановленный без спроса (ядро, которое до этого не дошло) второй раз без перезагрузки телефона не запускается — такой запуск сбрасывал телефон, и драйвер от него сам отказывается; шаги и остальные параметры описаны в начале `bsd/dev/redwood_wifi.c` |
+| `adsp=1` | аудиопроцессор телефона (ADSP: звук, а у Xiaomi ещё и зарядка), шаг 1 звука: прошивка `adsp.mdt` и `adsp.b02`…`adsp.b36` с разделов телефона (`/vendor/firmware_mnt/image`, в каталог `fw-audio/`, в git не попадает; в малый образ её кладёт `ADSP=1 userland/build.sh`, и только без `GL=1` — вместе они не помещаются в 64 МБ, что берёт загрузчик) кладётся в его память, RPMh просят о его шинах питания LCX и LMX, пока он не попросит о них сам, TrustZone его запускает, через 41 мс он говорит «готов», поднимается GLINK (канал IPCRTR) и QRTR поверх него: процессор — узел 5 — объявляет 12 служб и открывает свои каналы, среди них `apr_audio_svc`, через который потом пойдёт звук. Рядом с загрузкой системы и рядом с `wifi=1`; в конце сеанса его просят остановиться и выключают. `adsp_step=N` — то же по шагам (первые N из 10, ядро ждёт), подробности в начале `bsd/dev/redwood_adsp.c` |
Несколько ядер на телефоне проверены (2, 4 и 7 ядер запускаются и чисто возвращаются в загрузчик). Лучший
вариант — **`cpus=4`**: четыре больших ядра (7, 6, 5, 4). На одном и том же сценарии: 929 кадров вместо 671,
diff --git a/userland/build.sh b/userland/build.sh
index 0a7c267..0e74514 100755
--- a/userland/build.sh
+++ b/userland/build.sh
@@ -30,6 +30,9 @@
# and nothing changes (src/inject.c, opengl_es()).
# GL=test: only the library and bin/gl_test (a triangle, src/gl_test.c;
# RC_JOBS=rc.jobs.gltest runs it); the window server stays as it is.
+# ADSP=1 with the audio DSP's firmware from ../fw-audio in fw/ (20 MB; for the
+# kernel's adsp=1 / adsp_step=N, bsd/dev/redwood_adsp.c). Not beside GL=1:
+# the two do not fit the 64 MB the loader takes with the kernel.
# system.packfs the system image, the root filesystem (second RAM disk, /dev/md1): the part of
# iOS 16.5 chosen by ../ios165/select-system.py at its real paths, the dyld
# shared cache, and src/stage0.c as /sbin/launchd (the real one is
@@ -207,6 +210,13 @@ if [ -f ../fw-wifi/wpss.mdt ]; then
rm -f root/fw/network
[ -f ../fw-wifi/network ] && install -m 0600 ../fw-wifi/network root/fw/network
fi
+# ... and with ADSP=1 the image of the audio DSP, from the same place into ../fw-audio (bsd/dev/redwood_adsp.c):
+# adsp.mdt and adsp.b02 to adsp.b36, 20 MB. Only when asked for: beside GL=1 the payload would be more than the 64 MB
+# the loader takes.
+if [ -n "$ADSP" ] && [ -f ../fw-audio/adsp.mdt ]; then
+ mkdir -p root/fw
+ install -m 0644 ../fw-audio/adsp.mdt ../fw-audio/adsp.b0[2-9] ../fw-audio/adsp.b[1-9][0-9] root/fw/
+fi
../mkpackfs.py -o root.packfs root
# --- the system image -----------------------------------------------------------------------------
diff --git a/xnu/files/bsd/dev/redwood_adsp.c b/xnu/files/bsd/dev/redwood_adsp.c
new file mode 100644
index 0000000..b4e278a
--- /dev/null
+++ b/xnu/files/bsd/dev/redwood_adsp.c
@@ -0,0 +1,2607 @@
+/*
+ * The audio DSP of the POCO X5 Pro (redwood), started, and spoken to: step 1 of sound. Every
+ * address below belongs to this phone, and nothing here is touched unless a boot-arg asks for it.
+ *
+ * The SoC's audio (LPASS: its clocks, its I2S and SoundWire, the codec's macros) does not belong
+ * to the application processor. It is the audio DSP's -- a Hexagon of its own inside LPASS,
+ * Android's "adsp" -- which stays in reset until somebody loads it, exactly as the Wi-Fi's does
+ * (redwood_wifi.c): the phone's own signed image, adsp.mdt and adsp.b02 to adsp.b36 from
+ * /vendor/firmware_mnt/image of its Android, goes into the 64 MB reserved for it, TrustZone is
+ * asked to authenticate and start it, and from there everything is conversation through the
+ * shared memory: SMP2P for "up", "given up", "stopped"; GLINK for channels; QRTR on the channel
+ * "IPCRTR" for the services the DSP offers. (Reading an LPASS clock register from here ends the
+ * kernel with an external abort: the DSP owns that block, and talks about it only this way.)
+ * What audio needs of it later -- the audio services, the codec, the speaker's path -- is for the
+ * steps after this one; this file only brings the DSP up and the link with it.
+ *
+ * The way is the Wi-Fi DSP's, taken over from redwood_wifi.c with the audio DSP's numbers: its
+ * comments say why each thing is done so. What is different is said here:
+ * - processor 2 (not 13) in the shared memory; its SMP2P items 443 (its) and 429 (ours), its
+ * last words in item 423; GLINK's three items in our partition with it
+ * - its doorbells are IPCC client 3 (LPASS): signal 0 GLINK, signal 2 SMP2P
+ * - TrustZone knows it as image 1; its memory is 0x86700000, 64 MB (dt: qcom,lpass@3700000,
+ * reserved-memory adsp@86700000), every byte of which the image takes
+ * - the always-on processor is told of it as "adsp"
+ * - the Wi-Fi DSP may be running at the same time (wifi=1): the mailbox to the always-on
+ * processor and the lock for allocating in the shared memory are the two things both use,
+ * each behind a lock both take (redwood_wifi.c); the doorbells are only counted here, and
+ * only the audio DSP's are cleared from here
+ * The code is a copy, not a shared library, for now: step 1 is not to change what the Wi-Fi does.
+ *
+ * Boot-arg adsp_step=N performs steps 1 to N and stops; as with wifi_step, a step says what it is
+ * about to touch before it touches it, and a step that does not find what it needs ends the
+ * sequence. adsp=1 is all of them, on a thread beside the system's start.
+ *
+ * 1 reads the shared memory: our partition with the DSP, its items, whether it has run
+ * since the phone was started (its SMP2P item is there if so), its last words
+ * 2 reads the firmware in the small image: is it whole, does it fit the reserved memory
+ * 3 writes the always-on processor's mailbox open, unless it is (the Wi-Fi may have opened it)
+ * 4 asks TrustZone to take the image's header (image 1) and where the image will be; a
+ * DSP that a kernel before this one left running is asked to stop and shut down first
+ * 5 writes the image into the DSP's memory, and reads it back
+ * 6 writes our SMP2P item (429) into the shared memory, and our two marks of GLINK to zero
+ * 7 asks RPMh for the DSP's rails, writes "adsp: on" to the always-on processor, asks
+ * TrustZone to authenticate and start the DSP, and watches for its first words
+ * (answering in our item what SMP2P has us answer)
+ * 8 reads GLINK's queues
+ * 9 writes our side of GLINK: the versions, the channel "IPCRTR" opened both ways, our intents;
+ * from here a thread of ours listens to the DSP
+ * 10 talks QRTR on that channel: hello both ways, and the services the DSP tells of, written
+ * down for adsp_listen seconds (default 3)
+ *
+ * What is started is stopped: when the steps are over (unless adsp_keep=1, and then when the run
+ * ends), and on a panic's way back to the loader. The DSP is asked first (bit 0 of our entry
+ * "master-kernel"), then TrustZone shuts it down, then the always-on processor hears "adsp: off".
+ * A DSP that a kernel before this one left running is found by its item and treated the same way
+ * before this one is started; one that was shut down without having said it stopped is not
+ * started again unless adsp_again=1 (the Wi-Fi DSP reset the phone, twice, when that was tried).
+ *
+ * More boot-args: adsp_new=K (steps K on have not been seen to work, and wait adsp_pause=MS after
+ * each announcement, default 2200: what the loader needs to note the end of the log), adsp_wait=S
+ * (how long step 7 watches for "up", default 10), adsp_keep=1, adsp_again=1, adsp_return=1 (back
+ * to the loader when the steps are over).
+ *
+ * Nothing here writes a clock or changes a pin. One request goes to RPMh, as Linux makes it: the
+ * DSP's two rails, LCX and LMX, at level 384 while it boots, given back when it says it has asked
+ * for its own (see rails()).
+ *
+ * Addresses are from the phone's own device tree (qcom,lpass@3700000, qcom,smp2p-adsp,
+ * qcom,glink/adsp, qcom,ipcc@408000, power-controller@c300000, reserved-memory), layouts from
+ * Linux (see redwood_wifi.c for the files).
+ */
+#include <kern/clock.h>
+#include <kern/kalloc.h>
+#include <kern/locks.h>
+#include <kern/thread.h>
+#include <libkern/libkern.h>
+#include <mach/mach_types.h>
+#include <mach/vm_param.h>
+#include <vm/memory_types.h>
+#include <miscfs/packfs/packfs.h>
+#include <pexpert/pexpert.h>
+#include <sys/errno.h>
+#include <sys/stat.h>
+#include <sys/systm.h>
+#include <vm/vm_kern.h>
+
+extern vm_offset_t ml_io_map(vm_offset_t phys_addr, vm_size_t size);
+extern vm_offset_t ml_io_map_unmappable(vm_offset_t phys_addr, vm_size_t size, unsigned int flags);
+extern void ml_io_unmap(vm_offset_t addr, vm_size_t sz);
+extern uint64_t kvtophys(vm_offset_t va);
+extern void loader_return(void);
+extern kern_return_t thread_terminate(thread_t thread);
+extern vm_offset_t pe_arm_get_soc_base_phys(void);
+extern dev_t mdevlookup(int devid);
+extern int mdevgetkernelrange(dev_t dev, vm_offset_t *base, uint64_t *size);
+
+/* What the Wi-Fi's DSP and this one both use (redwood_wifi.c). */
+extern lck_mtx_t redwood_aop_mailbox_lock;
+extern lck_mtx_t redwood_smem_alloc_lock;
+
+static LCK_GRP_DECLARE(adsp_lock_group, "redwood-adsp");
+
+#define STEPS 10
+#define REDWOOD_SOC_BASE 0x00400000ul /* where the loader says the phone's peripherals begin */
+
+static uint32_t last_step, pause_ms = 2200, listen_seconds = 3;
+static uint32_t step; /* the one being performed */
+static uint32_t new_from; /* the first of the steps that have not been seen to work: they wait */
+static char last_write[96] = "nothing written";
+static bool leaving; /* on the way back to the loader after a panic: no lock is taken, nothing sleeps */
+
+/*
+ * Every line goes to the log the loader reads from another core as well (as in redwood_wifi.c);
+ * with adsp=1 only the few that say how it went (tell()), the rest to the kernel's own log.
+ */
+extern void loader_log_putc(char c);
+
+static LCK_MTX_DECLARE(say_lock, &adsp_lock_group);
+static bool quiet;
+
+static void
+say_line(const char *line, bool aloud)
+{
+ if (leaving) {
+ for (const char *c = line; *c != 0; c++) {
+ loader_log_putc(*c);
+ }
+ return;
+ }
+ lck_mtx_lock(&say_lock);
+ if (!quiet || aloud) {
+ for (const char *c = line; *c != 0; c++) {
+ loader_log_putc(*c);
+ }
+ }
+ printf("%s", line);
+ lck_mtx_unlock(&say_lock);
+}
+
+static int say(const char *format, ...) __printflike(1, 2);
+static int tell(const char *format, ...) __printflike(1, 2);
+
+static int
+say(const char *format, ...)
+{
+ char line[400];
+ va_list list;
+ int length;
+
+ va_start(list, format);
+ length = vsnprintf(line, sizeof(line), format, list);
+ va_end(list);
+ say_line(line, false);
+ return length;
+}
+
+static int
+tell(const char *format, ...)
+{
+ char line[400];
+ va_list list;
+ int length;
+
+ va_start(list, format);
+ length = vsnprintf(line, sizeof(line), format, list);
+ va_end(list);
+ say_line(line, true);
+ return length;
+}
+#define printf say
+
+static void
+sleep_us(uint32_t us)
+{
+ delay_for_interval(us, NSEC_PER_USEC);
+}
+
+static void
+sleep_ms(uint32_t ms)
+{
+ delay_for_interval(ms, NSEC_PER_MSEC);
+}
+
+static uint64_t
+now_us(void)
+{
+ uint64_t nanoseconds;
+
+ absolutetime_to_nanoseconds(mach_absolute_time(), &nanoseconds);
+ return nanoseconds / 1000;
+}
+
+/* Until every write to memory so far has been carried out. */
+static void
+settle(void)
+{
+ __asm__ volatile ("dsb sy");
+}
+
+static bool
+option(const char *name)
+{
+ uint32_t value = 0;
+
+ return PE_parse_boot_argn(name, &value, sizeof(value)) && value != 0;
+}
+
+/* ---- register access (as in redwood_wifi.c) ------------------------------------------------------- */
+
+static uint32_t __attribute__((noinline))
+rd(vm_offset_t address)
+{
+ uint32_t value;
+
+ __asm__ volatile ("ldr %w0, [%1]" : "=r" (value) : "r" (address) : "memory");
+ return value;
+}
+
+static void __attribute__((noinline))
+wr(vm_offset_t address, uint32_t value)
+{
+ __asm__ volatile ("str %w0, [%1]" : : "r" (value), "r" (address) : "memory");
+}
+
+struct block {
+ const char *name;
+ uint32_t phys, size;
+ vm_offset_t base;
+};
+
+static void
+need(struct block *b)
+{
+ if (b->base == 0) {
+ b->base = ml_io_map(b->phys, b->size);
+ }
+}
+
+/* Time for the loader to note what has just been said, where it is not yet known to be harmless. */
+static void
+hold(void)
+{
+ if (step >= new_from && pause_ms && !leaving) {
+ sleep_ms(pause_ms);
+ }
+}
+
+static void
+announce(const char *what)
+{
+ printf("adsp: step %u: %s\n", step, what);
+ hold();
+}
+
+static uint32_t
+get(struct block *b, const char *name, uint32_t offset)
+{
+ uint32_t value;
+
+ need(b);
+ value = rd(b->base + offset);
+ printf("adsp: %u: %s %s (%#x) = %#x\n", step, b->name, name, b->phys + offset, value);
+ return value;
+}
+
+static void
+put(struct block *b, const char *name, uint32_t offset, uint32_t value)
+{
+ need(b);
+ printf("adsp: %u: about to write %s %s at %#x: %#x\n", step, b->name, name, b->phys + offset, value);
+ snprintf(last_write, sizeof(last_write), "step %u wrote %#x to %s %s at %#x", step, value, b->name, name, b->phys + offset);
+ wr(b->base + offset, value);
+}
+
+/* ---- the shared memory (redwood_wifi.c, step 1) --------------------------------------------------- */
+
+#define SMEM_PHYS 0x80900000UL
+#define SMEM_SIZE 0x00200000u
+#define SMEM_US 0
+#define SMEM_ADSP 2 /* dt: qcom,remote-pid of qcom,smp2p-adsp and qcom,glink/adsp */
+#define SMEM_GLOBAL 0xfffe
+#define SMEM_CANARY 0xa5a5
+
+struct smem_table_entry {
+ uint32_t offset, size, flags;
+ uint16_t host0, host1;
+ uint32_t cacheline, reserved[7];
+};
+struct smem_table {
+ uint8_t magic[4];
+ uint32_t version, count, reserved[5];
+};
+struct smem_partition {
+ uint8_t magic[4];
+ uint16_t host0, host1;
+ uint32_t size, free_uncached, free_cached, reserved[3];
+};
+struct smem_entry {
+ uint16_t canary, item;
+ uint32_t size;
+ uint16_t padding_data, padding_header;
+ uint32_t reserved;
+};
+struct smem_item {
+ uint32_t item, offset, size;
+};
+
+/*
+ * The DSP's items (dt: qcom,smem of qcom,smp2p-adsp, <443 429>: the item the DSP writes first, then
+ * ours -- the Wi-Fi's <617 616> are in the same order; taken the other way round, the DSP made
+ * our item its own and never said "up" --, and qcom,smem-id of qcom,lpass@3700000).
+ */
+#define ITEM_SMP2P_OURS 429
+#define ITEM_SMP2P_THEIRS 443
+#define ITEM_CRASH_REASON 423
+
+static const volatile uint8_t *smem;
+
+static bool
+smem_peek(uint64_t offset, void *to, uint32_t size)
+{
+ if (smem == NULL) {
+ smem = (const volatile uint8_t *)ml_io_map_unmappable(SMEM_PHYS, SMEM_SIZE, VM_WIMG_WCOMB);
+ }
+ if (offset + size > SMEM_SIZE) {
+ return false;
+ }
+ for (uint32_t i = 0; i < size; i++) {
+ ((uint8_t *)to)[i] = smem[offset + i];
+ }
+ return true;
+}
+
+static void
+smem_poke(uint32_t offset, uint32_t value)
+{
+ wr((vm_offset_t)smem + offset, value);
+}
+
+static uint32_t
+smem_word(uint32_t offset)
+{
+ return rd((vm_offset_t)smem + offset);
+}
+
+static uint32_t
+smem_partition(uint16_t one, uint16_t other, uint32_t *cacheline, uint32_t *size)
+{
+ struct smem_table table;
+ struct smem_table_entry entry;
+ struct smem_partition header;
+
+ if (!smem_peek(SMEM_SIZE - 0x1000, &table, sizeof(table)) || memcmp(table.magic, "$TOC", 4) != 0 || table.version != 1 ||
+ table.count > 64) {
+ return 0;
+ }
+ for (uint32_t i = 0; i < table.count; i++) {
+ if (!smem_peek(SMEM_SIZE - 0x1000 + sizeof(table) + i * sizeof(entry), &entry, sizeof(entry)) || entry.offset == 0 ||
+ entry.size == 0) {
+ continue;
+ }
+ if (!((entry.host0 == one && entry.host1 == other) || (entry.host0 == other && entry.host1 == one))) {
+ continue;
+ }
+ if ((uint64_t)entry.offset + entry.size > SMEM_SIZE - 0x1000 || !smem_peek(entry.offset, &header, sizeof(header)) ||
+ memcmp(header.magic, "$PRT", 4) != 0 || header.size != entry.size) {
+ return 0;
+ }
+ *cacheline = entry.cacheline;
+ *size = entry.size;
+ return entry.offset;
+ }
+ return 0;
+}
+
+static unsigned
+smem_items(uint32_t partition, uint32_t cacheline, struct smem_item *found, unsigned max)
+{
+ struct smem_partition header;
+ struct smem_entry entry;
+ uint32_t at, step_back;
+ unsigned count = 0;
+
+ if (partition == 0 || !smem_peek(partition, &header, sizeof(header)) || header.free_uncached > header.size ||
+ header.free_cached > header.size) {
+ return 0;
+ }
+ for (at = sizeof(header); at + sizeof(entry) <= header.free_uncached;) {
+ if (!smem_peek(partition + at, &entry, sizeof(entry)) || entry.canary != SMEM_CANARY || entry.size > header.size) {
+ break;
+ }
+ if (count < max) {
+ found[count].item = entry.item;
+ found[count].offset = partition + at + sizeof(entry) + entry.padding_header;
+ found[count].size = entry.size - entry.padding_data;
+ }
+ count++;
+ at += sizeof(entry) + entry.padding_header + entry.size;
+ }
+ if (cacheline == 0 || (cacheline & (cacheline - 1))) {
+ return count;
+ }
+ step_back = ((uint32_t)sizeof(entry) + cacheline - 1) & ~(cacheline - 1);
+ for (at = header.size - step_back; at > header.free_cached && at < header.size;) {
+ if (!smem_peek(partition + at, &entry, sizeof(entry)) || entry.canary != SMEM_CANARY || entry.size > at) {
+ break;
+ }
+ if (count < max) {
+ found[count].item = entry.item;
+ found[count].offset = partition + at - entry.size;
+ found[count].size = entry.size - entry.padding_data;
+ }
+ count++;
+ if (entry.size + step_back > at) {
+ break;
+ }
+ at -= entry.size + step_back;
+ }
+ return count;
+}
+
+/* An item the DSP and we share: in the partition of the two, or else in the global one. Where it is; 0: nowhere. */
+static uint32_t
+smem_item(uint32_t item, uint32_t *size)
+{
+ static struct smem_item found[256];
+ static const uint16_t hosts[2][2] = { { SMEM_US, SMEM_ADSP }, { SMEM_GLOBAL, SMEM_GLOBAL } };
+
+ for (unsigned h = 0; h < 2; h++) {
+ uint32_t cacheline = 0, partition_size = 0;
+ uint32_t partition = smem_partition(hosts[h][0], hosts[h][1], &cacheline, &partition_size);
+ unsigned count = smem_items(partition, cacheline, found, 256);
+
+ for (unsigned i = 0; i < count && i < 256; i++) {
+ if (found[i].item == item) {
+ *size = found[i].size;
+ return found[i].offset;
+ }
+ }
+ }
+ return 0;
+}
+
+static void
+show_partition(void)
+{
+ static struct smem_item found[256];
+ struct smem_partition header;
+ uint32_t cacheline = 0, size = 0, partition = smem_partition(SMEM_US, SMEM_ADSP, &cacheline, &size);
+ unsigned count;
+ char line[320];
+
+ if (partition == 0 || !smem_peek(partition, &header, sizeof(header))) {
+ printf("adsp: %u: there is no partition for us and the audio DSP (0 and 2)\n", step);
+ return;
+ }
+ count = smem_items(partition, cacheline, found, 256);
+ printf("adsp: %u: the partition for us and the audio DSP (0 and 2): at %#lx, %#x bytes, of which %#x free; %u items\n", step,
+ SMEM_PHYS + partition, size, header.free_cached - header.free_uncached, count);
+ for (unsigned i = 0; i < count && i < 256;) {
+ int at = 0;
+
+ for (unsigned n = 0; n < 12 && i < count && i < 256; n++, i++) {
+ at += snprintf(line + at, sizeof(line) - (size_t)at, " %u (%u)", found[i].item, found[i].size);
+ }
+ printf("adsp: %u: items (bytes):%s\n", step, line);
+ }
+}
+
+/* ---- SMP2P ---------------------------------------------------------------------------------------- */
+
+struct smp2p {
+ uint32_t magic;
+ uint32_t version;
+ uint16_t local, remote, total, valid;
+ uint32_t flags;
+ struct {
+ char name[16];
+ uint32_t value;
+ } entry[16];
+};
+#define SMP2P_MAGIC 0x504d5324
+#define SMP2P_VERSION 1
+#define SMP2P_RESTART_ACKS 1u
+#define SMP2P_RESTART_DONE (1u << 0)
+#define SMP2P_RESTART_ACK (1u << 1)
+#define SMP2P_AT_VERSION 4
+#define SMP2P_AT_PROCESSORS 8
+#define SMP2P_AT_COUNTS 12
+#define SMP2P_AT_FLAGS 16
+#define SMP2P_AT_NAME(i) (20 + 20 * (i))
+#define SMP2P_AT_VALUE(i) (36 + 20 * (i))
+
+/* The bits of the DSP's entry "slave-kernel" (dt: the interrupts of qcom,lpass@3700000). */
+#define DSP_FATAL (1u << 0)
+#define DSP_READY (1u << 1)
+#define DSP_HANDOVER (1u << 2)
+#define DSP_STOP_ACK (1u << 3)
+#define DSP_SHUTDOWN_ACK (1u << 7)
+
+/* Our entry (dt: the child "master-kernel" of qcom,smp2p-adsp; "qcom,force-stop" of qcom,lpass@3700000). */
+#define OUR_ENTRY_STOP "master-kernel"
+#define DSP_STOP (1u << 0)
+
+static bool
+dsp_words(struct smp2p *words)
+{
+ static uint32_t at;
+ uint32_t size = 0;
+
+ if (at == 0) {
+ at = smem_item(ITEM_SMP2P_THEIRS, &size);
+ if (at == 0 || size < sizeof(*words)) {
+ at = 0;
+ return false;
+ }
+ }
+ return smem_peek(at, words, sizeof(*words)) && words->magic == SMP2P_MAGIC;
+}
+
+static int
+smp2p_entry(const struct smp2p *words, const char *name)
+{
+ for (unsigned i = 0; i < words->valid && i < 16; i++) {
+ if (strncmp(words->entry[i].name, name, sizeof(words->entry[i].name)) == 0) {
+ return (int)i;
+ }
+ }
+ return -1;
+}
+
+static uint32_t
+dsp_state(const struct smp2p *words)
+{
+ int entry = smp2p_entry(words, "slave-kernel");
+
+ return entry < 0 ? 0 : words->entry[entry].value;
+}
+
+static void
+show_words(const char *prefix, const struct smp2p *words)
+{
+ char line[320];
+ int at = 0;
+
+ line[0] = 0;
+ for (unsigned i = 0; i < words->valid && i < 16 && at < (int)sizeof(line) - 40; i++) {
+ at += snprintf(line + at, sizeof(line) - (size_t)at, " %.16s = %#x;", words->entry[i].name, words->entry[i].value);
+ }
+ printf("adsp: %u: %s: SMP2P version %u, features %#x, from processor %u to %u, flags %#x, %u of %u entries:%s\n", step, prefix,
+ words->version & 0xff, words->version >> 8, words->local, words->remote, words->flags, words->valid, words->total, line);
+}
+
+static void
+show_crash_reason(void)
+{
+ char text[81];
+ uint32_t size = 0, at = smem_item(ITEM_CRASH_REASON, &size);
+
+ if (at == 0 || size == 0) {
+ printf("adsp: %u: item %u, the DSP's last words: not there\n", step, ITEM_CRASH_REASON);
+ return;
+ }
+ bzero(text, sizeof(text));
+ smem_peek(at, text, size < sizeof(text) - 1 ? size : sizeof(text) - 1);
+ if (text[0] == 0) {
+ printf("adsp: %u: item %u, the DSP's last words (%u bytes): none\n", step, ITEM_CRASH_REASON, size);
+ } else {
+ printf("adsp: %u: THE AUDIO DSP'S LAST WORDS (item %u, %u bytes): \"%s\"\n", step, ITEM_CRASH_REASON, size, text);
+ }
+}
+
+/* ---- the doorbells, and the mailbox to the always-on processor (redwood_wifi.c, steps 2 and 7) ------ */
+
+#define QMP_MAGIC 0x00
+#define QMP_MAGIC_VALUE 0x4d41494c
+#define QMP_VERSION 0x04
+#define QMP_THEIR_LINK 0x0c
+#define QMP_THEIR_LINK_ACK 0x10
+#define QMP_THEIR_CHANNEL 0x14
+#define QMP_THEIR_CHANNEL_ACK 0x18
+#define QMP_OUR_LINK 0x24
+#define QMP_OUR_LINK_ACK 0x28
+#define QMP_OUR_CHANNEL 0x2c
+#define QMP_OUR_CHANNEL_ACK 0x30
+#define QMP_OUR_BOX_SIZE 0x34
+#define QMP_OUR_BOX 0x38
+#define QMP_UP 0x0000ffffu
+#define QMP_DOWN 0xffff0000u
+#define QMP_MESSAGE 64
+
+static struct block aoss = { "mailbox", 0x0c300000, 0x400 };
+
+#define IPCC_SEND 0x0c
+#define IPCC_RECEIVED 0x10
+#define IPCC_ENABLE 0x14
+#define IPCC_DISABLE 0x18
+#define IPCC_CLEAR 0x1c
+#define IPCC_NOBODY 0xffffffffu
+#define IPCC_AOP 0u
+#define IPCC_LPASS 3u /* dt: mboxes of qcom,smp2p-adsp and qcom,glink/adsp */
+#define IPCC_GLINK 0u
+#define IPCC_SMP2P 2u
+
+static struct block ipcc = { "ipcc", 0x00408000, 0x1000 };
+
+static struct {
+ uint32_t glink, smp2p, said;
+ uint64_t since;
+} doorbell;
+
+/*
+ * The audio DSP's rings, counted and cleared. Another's ring (the Wi-Fi DSP's, the always-on
+ * processor's) is left for whoever listens for it: IPCC names the same one until it is cleared,
+ * so the look ends there.
+ */
+static void
+doorbells(void)
+{
+ for (unsigned i = 0; i < 8; i++) {
+ uint32_t who = rd(ipcc.base + IPCC_RECEIVED);
+
+ if (who == IPCC_NOBODY || who >> 16 != IPCC_LPASS) {
+ return;
+ }
+ wr(ipcc.base + IPCC_CLEAR, who);
+ doorbell.glink += (who & 0xffff) == IPCC_GLINK;
+ doorbell.smp2p += (who & 0xffff) == IPCC_SMP2P;
+ if (doorbell.said < 24) {
+ uint64_t now = now_us() - doorbell.since;
+
+ doorbell.said++;
+ printf("adsp: doorbell: +%llu.%03llu ms: signal %u of the audio DSP; heard and cleared\n", now / 1000, now % 1000,
+ who & 0xffff);
+ }
+ }
+}
+
+static uint32_t our_box; /* where in the message memory our mailbox is; 0: not open */
+
+static bool
+mailbox_wait(uint32_t offset, uint32_t value, uint32_t ms)
+{
+ for (uint32_t i = 0; i < ms * 10; i++) {
+ if (rd(aoss.base + offset) == value) {
+ return true;
+ }
+ sleep_us(100);
+ }
+ return rd(aoss.base + offset) == value;
+}
+
+static void
+ring_aop(void)
+{
+ wr(ipcc.base + IPCC_SEND, IPCC_AOP << 16 | 0);
+}
+
+/* Linux's qmp_open(), as redwood_wifi.c does it; what is done already is found done. Under the mailbox's lock. */
+static bool
+mailbox_open(void)
+{
+ uint32_t box, size;
+
+ need(&aoss);
+ need(&ipcc);
+ if (get(&aoss, "magic", QMP_MAGIC) != QMP_MAGIC_VALUE || get(&aoss, "version", QMP_VERSION) != 1) {
+ printf("adsp: %u: not a mailbox this knows\n", step);
+ return false;
+ }
+ box = get(&aoss, "our mailbox", QMP_OUR_BOX);
+ size = get(&aoss, "our mailbox's size", QMP_OUR_BOX_SIZE);
+ if (size < QMP_MESSAGE + 4 || box < 0x3c || box + size > aoss.size || (box & 3)) {
+ printf("adsp: %u: our mailbox is not where a message fits\n", step);
+ return false;
+ }
+ if (rd(aoss.base + QMP_OUR_LINK) == QMP_UP && rd(aoss.base + QMP_OUR_LINK_ACK) == QMP_UP &&
+ rd(aoss.base + QMP_OUR_CHANNEL) == QMP_UP && rd(aoss.base + QMP_THEIR_CHANNEL) == QMP_UP &&
+ rd(aoss.base + QMP_OUR_CHANNEL_ACK) == QMP_UP) {
+ printf("adsp: %u: the mailbox is open already (the Wi-Fi's start, or a kernel before this one, opened it)\n", step);
+ our_box = box;
+ return true;
+ }
+ put(&aoss, "their link, seen by us", QMP_THEIR_LINK_ACK, rd(aoss.base + QMP_THEIR_LINK));
+ put(&aoss, "our link", QMP_OUR_LINK, QMP_UP);
+ ring_aop();
+ if (!mailbox_wait(QMP_OUR_LINK_ACK, QMP_UP, 1000)) {
+ printf("adsp: %u: THE ALWAYS-ON PROCESSOR DOES NOT ACKNOWLEDGE OUR LINK within a second; taken down again\n", step);
+ put(&aoss, "our link", QMP_OUR_LINK, QMP_DOWN);
+ ring_aop();
+ return false;
+ }
+ put(&aoss, "our channel", QMP_OUR_CHANNEL, QMP_UP);
+ ring_aop();
+ if (!mailbox_wait(QMP_THEIR_CHANNEL, QMP_UP, 1000)) {
+ printf("adsp: %u: THE ALWAYS-ON PROCESSOR DOES NOT OPEN ITS CHANNEL within a second; ours is taken down again\n", step);
+ put(&aoss, "our channel", QMP_OUR_CHANNEL, QMP_DOWN);
+ put(&aoss, "our link", QMP_OUR_LINK, QMP_DOWN);
+ ring_aop();
+ return false;
+ }
+ put(&aoss, "their channel, seen by us", QMP_THEIR_CHANNEL_ACK, QMP_UP);
+ ring_aop();
+ if (!mailbox_wait(QMP_OUR_CHANNEL_ACK, QMP_UP, 1000)) {
+ printf("adsp: %u: THE ALWAYS-ON PROCESSOR DOES NOT ACKNOWLEDGE OUR CHANNEL within a second; taken down again\n", step);
+ put(&aoss, "our channel", QMP_OUR_CHANNEL, QMP_DOWN);
+ put(&aoss, "our link", QMP_OUR_LINK, QMP_DOWN);
+ ring_aop();
+ return false;
+ }
+ printf("adsp: %u: THE MAILBOX TO THE ALWAYS-ON PROCESSOR IS OPEN\n", step);
+ our_box = box;
+ return true;
+}
+
+/* A message to the always-on processor (Linux's qmp_send()), under the lock the Wi-Fi's start takes too. */
+static bool
+mailbox_send(const char *text)
+{
+ uint32_t words[QMP_MESSAGE / 4] = { 0 };
+ bool sent = false;
+
+ if (!leaving) {
+ lck_mtx_lock(&redwood_aop_mailbox_lock);
+ }
+ if (our_box == 0 && (leaving || !mailbox_open())) {
+ goto out;
+ }
+ strlcpy((char *)words, text, sizeof(words));
+ printf("adsp: %u: about to tell the always-on processor \"%s\"\n", step, text);
+ snprintf(last_write, sizeof(last_write), "step %u sent the always-on processor a message", step);
+ if (rd(aoss.base + our_box) != 0) {
+ printf("adsp: %u: our mailbox still holds a message of %u bytes: not sent\n", step, rd(aoss.base + our_box));
+ goto out;
+ }
+ for (unsigned i = 0; i < QMP_MESSAGE / 4; i++) {
+ wr(aoss.base + our_box + 4 + 4 * i, words[i]);
+ }
+ wr(aoss.base + our_box, QMP_MESSAGE);
+ (void)rd(aoss.base + our_box);
+ ring_aop();
+ if (!mailbox_wait(our_box, 0, 1000)) {
+ printf("adsp: %u: THE ALWAYS-ON PROCESSOR DOES NOT TAKE THE MESSAGE within a second; it is taken back\n", step);
+ wr(aoss.base + our_box, 0);
+ goto out;
+ }
+ printf("adsp: %u: the always-on processor has taken it\n", step);
+ sent = true;
+out:
+ if (!leaving) {
+ lck_mtx_unlock(&redwood_aop_mailbox_lock);
+ }
+ return sent;
+}
+
+/* ---- RPMh: the DSP's two rails, asked for while it boots ------------------------------------------------ */
+
+/*
+ * The audio DSP's logic and memories run from LPASS's own rails, LCX and LMX (dt: vdd_cx and
+ * vdd_mx of qcom,lpass@3700000 are PM7325 L5 and L4 "level" regulators, RPMh resources lcx.lvl and
+ * lmx.lvl), which the boot loader leaves at retention (16). Linux and Android ask for level 384
+ * while the DSP boots, until it says it has asked for its own (bit 2, "proxy-unvote"); without it
+ * the DSP rang once and left "SFR Init: wdog or kernel error suspected." in its last words.
+ *
+ * A request goes to RPMh through the application processor's sequencer (dt: rsc@18200000, DRV 2,
+ * qcom,tcs-offset 0xd00, qcom,tcs-config: TCSs 0 and 1 are for requests made at once), as Linux's
+ * rpmh-rsc.c writes one: the command's message id, address and value into a TCS that is idle, the
+ * command enabled and marked to be waited for, the TCS put in "active" mode and triggered, and its
+ * bit in the status of completions looked for. A rail asked for by level is asked by position in
+ * the list of its levels that the boot loader's command database keeps (cmd-db.c), the first
+ * position at or above the level wanted.
+ */
+#define CMD_DB_PHYS 0x80860000UL
+#define CMD_DB_SIZE 0x20000UL
+
+struct cmd_db_slave {
+ uint16_t id, header_offset, data_offset, count, version, reserved[3];
+};
+struct cmd_db_header {
+ uint32_t version;
+ uint8_t magic[4];
+ struct cmd_db_slave slave[8];
+ uint32_t checksum, reserved;
+ uint8_t data[];
+};
+struct cmd_db_entry {
+ char id[8];
+ uint32_t priority[2];
+ uint32_t address;
+ uint16_t length, offset;
+};
+
+static const struct cmd_db_header *cmd_db;
+
+static struct block rsc = { "rsc", 0x18220000, 0x10000 };
+#define RSC_TCS_BASE 0xd00
+#define RSC_IRQ_STATUS (RSC_TCS_BASE + 0x04) /* a bit per TCS: done */
+#define RSC_IRQ_CLEAR (RSC_TCS_BASE + 0x08)
+#define RSC_TCS(n) (RSC_TCS_BASE + 672 * (n))
+#define TCS_WAIT_FOR_DONE 0x10
+#define TCS_CONTROL 0x14
+#define TCS_STATUS 0x18 /* not zero: idle */
+#define TCS_ENABLED 0x1c
+#define TCS_COMMAND(c) (0x30 + 20 * (c)) /* message id, address, value, status */
+#define TCS_ACTIVE_MODE (1u << 16)
+#define TCS_TRIGGER (1u << 24)
+#define MSGID_WRITE_WAITED (8u | 1u << 8 | 1u << 16) /* four bytes, an answer wanted, a write */
+#define RPMH_TCS 0 /* the first of the two for requests made at once */
+
+/* The level position RPMh takes for that rail and level; -1: not found. */
+static int
+rail_position(const char *name, uint32_t level, uint32_t *address)
+{
+ char id[8] = { 0 };
+
+ if (cmd_db == NULL) {
+ cmd_db = (const struct cmd_db_header *)ml_io_map_unmappable(CMD_DB_PHYS, CMD_DB_SIZE, VM_WIMG_WCOMB);
+ }
+ if (!(cmd_db->magic[0] == 0xdb && cmd_db->magic[1] == 0x30 && cmd_db->magic[2] == 0x03 && cmd_db->magic[3] == 0x0c)) {
+ printf("adsp: %u: the boot loader's command database is not there\n", step);
+ return -1;
+ }
+ strlcpy(id, name, sizeof(id));
+ for (unsigned s = 0; s < 8; s++) {
+ const struct cmd_db_slave *slave = &cmd_db->slave[s];
+ const struct cmd_db_entry *entry = (const struct cmd_db_entry *)(cmd_db->data + slave->header_offset);
+
+ if (slave->id == 0 || slave->count > 512) {
+ continue;
+ }
+ for (unsigned i = 0; i < slave->count; i++) {
+ const uint8_t *data;
+ char levels[160];
+ int at;
+
+ if (memcmp(entry[i].id, id, sizeof(id)) != 0) {
+ continue;
+ }
+ if ((unsigned)slave->data_offset + entry[i].offset + entry[i].length > CMD_DB_SIZE - 0x1000) {
+ return -1;
+ }
+ data = cmd_db->data + slave->data_offset + entry[i].offset;
+ at = snprintf(levels, sizeof(levels), "%s at %#x, levels:", name, entry[i].address);
+ for (unsigned l = 0; l < entry[i].length / 2u && l < 16; l++) {
+ unsigned value = data[2 * l] | data[2 * l + 1] << 8;
+
+ if (l > 0 && value == 0) {
+ break;
+ }
+ at += snprintf(levels + at, sizeof(levels) - (size_t)at, " %u", value);
+ }
+ printf("adsp: %u: RPMh has %s\n", step, levels);
+ for (unsigned l = 0; l < entry[i].length / 2u && l < 16; l++) {
+ if ((uint32_t)(data[2 * l] | data[2 * l + 1] << 8) >= level) {
+ *address = entry[i].address;
+ return (int)l;
+ }
+ }
+ return -1;
+ }
+ }
+ printf("adsp: %u: RPMh has no \"%s\"\n", step, name);
+ return -1;
+}
+
+/* The two rails at that level (Linux's proxy request), or given back (level 0). True: RPMh has said done. */
+static bool
+rails(uint32_t level)
+{
+ static const char *const names[2] = { "lcx.lvl", "lmx.lvl" };
+ uint32_t addresses[2] = { 0, 0 }, values[2] = { 0, 0 }, done = 0;
+ uint64_t began;
+
+ for (unsigned r = 0; r < 2; r++) {
+ int position = level == 0 ? 0 : rail_position(names[r], level, &addresses[r]);
+
+ if (level == 0) {
+ rail_position(names[r], 0, &addresses[r]);
+ }
+ if (position < 0 || addresses[r] == 0) {
+ printf("adsp: %u: no position for %s at level %u: the rails are not asked\n", step, names[r], level);
+ return false;
+ }
+ values[r] = (uint32_t)position;
+ }
+ need(&rsc);
+ /* (the boot loader leaves the trigger set after its last request, done: idle is what the status says) */
+ if (rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_STATUS) == 0) {
+ printf("adsp: %u: TCS %u of the sequencer is busy (status %#x, control %#x): the rails are not asked\n", step, RPMH_TCS,
+ rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_STATUS), rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL));
+ return false;
+ }
+ printf("adsp: %u: about to ask RPMh, through TCS %u, for lcx.lvl (%#x) = %u and lmx.lvl (%#x) = %u (level %u%s)\n", step,
+ RPMH_TCS, addresses[0], values[0], addresses[1], values[1], level, level == 0 ? ": given back" : "");
+ snprintf(last_write, sizeof(last_write), "step %u asked RPMh for the audio DSP's rails at level %u", step, level);
+ hold();
+ wr(rsc.base + RSC_IRQ_CLEAR, 1u << RPMH_TCS);
+ for (unsigned r = 0; r < 2; r++) {
+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(r), MSGID_WRITE_WAITED);
+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(r) + 4, addresses[r]);
+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(r) + 8, values[r]);
+ }
+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_WAIT_FOR_DONE, 3);
+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_ENABLED, 3);
+ settle();
+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL) & ~TCS_TRIGGER);
+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL) | TCS_ACTIVE_MODE);
+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL) | TCS_TRIGGER);
+ began = now_us();
+ while (now_us() - began < 100000) {
+ done = rd(rsc.base + RSC_IRQ_STATUS) & (1u << RPMH_TCS);
+ if (done) {
+ break;
+ }
+ sleep_us(50);
+ }
+ printf("adsp: %u: RPMh %s after %llu us; the commands' status %#x %#x, the TCS's %#x\n", step, done ? "HAS DONE IT" :
+ "HAS NOT SAID DONE", now_us() - began, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(0) + 12),
+ rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(1) + 12), rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_STATUS));
+ wr(rsc.base + RSC_IRQ_CLEAR, 1u << RPMH_TCS);
+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL) & ~(TCS_TRIGGER | TCS_ACTIVE_MODE));
+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_ENABLED, 0);
+ return done != 0;
+}
+
+/* ---- step 1: the shared memory --------------------------------------------------------------------- */
+
+static bool
+step_survey(void)
+{
+ struct smp2p words;
+ uint32_t size = 0;
+
+ announce("read our partition with the audio DSP in the shared memory (0x80900000), and its items");
+ show_partition();
+ if (dsp_words(&words)) {
+ show_words("the DSP's item (443) is there: it has run since the phone was started", &words);
+ } else {
+ printf("adsp: %u: the DSP's SMP2P item (443) is not there: it has not run since the phone was started\n", step);
+ }
+ printf("adsp: %u: our SMP2P item (429): %s; GLINK's descriptor (478): %s, our queue (479): %s, the DSP's (480): %s\n", step,
+ smem_item(ITEM_SMP2P_OURS, &size) ? "there" : "not there", smem_item(478, &size) ? "there" : "NOT THERE",
+ smem_item(479, &size) ? "there" : "NOT THERE", smem_item(480, &size) ? "there" : "not there (the DSP makes it)");
+ show_crash_reason();
+ need(&ipcc);
+ printf("adsp: %u: the doorbells say %#x\n", step, rd(ipcc.base + IPCC_RECEIVED));
+ return smem_partition(SMEM_US, SMEM_ADSP, &size, &size) != 0;
+}
+
+/* ---- step 2: the firmware -------------------------------------------------------------------------- */
+
+#define ADSP_MEMORY 0x86700000u /* dt: reserved-memory adsp@86700000, memory-region of qcom,lpass@3700000 */
+#define ADSP_MEMORY_SIZE 0x04000000u
+#define PAS_ADSP 1 /* dt: qcom,pas-id */
+
+/* An entry of a directory of a packfs image in memory, by name (as in redwood_wifi.c). */
+static const struct packfs_dnode *
+image_entry(const uint8_t *image, const struct packfs_super *super, const struct packfs_dnode *directory, const char *name)
+{
+ const struct packfs_dnode *nodes = (const struct packfs_dnode *)(image + super->ps_nodes_off);
+ size_t length = strlen(name);
+
+ if (directory == NULL || !S_ISDIR(directory->pd_mode) || directory->pd_data + directory->pd_size > super->ps_node_count) {
+ return NULL;
+ }
+ for (uint64_t i = 0; i < directory->pd_size; i++) {
+ const struct packfs_dnode *node = &nodes[directory->pd_data + i];
+
+ if (node->pd_name_len == length && (uint64_t)node->pd_name + length <= super->ps_strings_size &&
+ memcmp(image + super->ps_strings_off + node->pd_name, name, length) == 0) {
+ return node;
+ }
+ }
+ return NULL;
+}
+
+/* A file of fw/ in the small image (userland/build.sh puts fw-audio/ there), or in the system image. */
+static const uint8_t *
+image_file(const char *name, uint64_t *size)
+{
+ for (int disk = 1; disk >= 0; disk--) {
+ const struct packfs_super *super;
+ const struct packfs_dnode *file;
+ vm_offset_t base = 0;
+ uint64_t length = 0;
+ dev_t dev = mdevlookup(disk);
+
+ if (dev == (dev_t)-1 || mdevgetkernelrange(dev, &base, &length) != 0 || length < PACKFS_SUPER_SIZE) {
+ continue;
+ }
+ super = (const struct packfs_super *)base;
+ if (memcmp(super->ps_magic, PACKFS_MAGIC, sizeof(super->ps_magic)) != 0 || super->ps_image_size > length ||
+ super->ps_node_count == 0 ||
+ super->ps_nodes_off + (uint64_t)super->ps_node_count * sizeof(struct packfs_dnode) > super->ps_image_size ||
+ super->ps_strings_off + super->ps_strings_size > super->ps_image_size) {
+ continue;
+ }
+ file = (const struct packfs_dnode *)(base + super->ps_nodes_off);
+ file = image_entry((const uint8_t *)base, super, file, "fw");
+ file = image_entry((const uint8_t *)base, super, file, name);
+ if (file != NULL && S_ISREG(file->pd_mode) && !(file->pd_flags & PACKFS_F_CHUNKED) &&
+ file->pd_data + file->pd_size <= super->ps_image_size) {
+ *size = file->pd_size;
+ return (const uint8_t *)base + file->pd_data;
+ }
+ }
+ printf("adsp: %u: no fw/%s in the small image: put the phone's /vendor/firmware_mnt/image/%s into fw-audio/ and run "
+ "userland/build.sh\n", step, name, name);
+ return NULL;
+}
+
+#define MDT_HASH(flags) (((flags) >> 24 & 7) == 2)
+#define MAX_SEGMENTS 48
+
+struct elf_segment {
+ uint32_t type, offset, vaddr, paddr, file_size, memory_size, flags, align;
+};
+
+static struct {
+ const uint8_t *header;
+ uint64_t header_size;
+ struct {
+ uint32_t at;
+ uint32_t file_size, memory_size;
+ const uint8_t *file;
+ } segment[MAX_SEGMENTS];
+ unsigned segments;
+ uint32_t size;
+} image;
+
+static bool
+step_firmware(void)
+{
+ struct elf_segment segment;
+ const uint8_t *header;
+ uint64_t size = 0, files = 0;
+ uint32_t count, hashes = 0, lowest = ~0u, highest = 0;
+
+ announce("find the audio DSP's image in the small image, and check it against its own header");
+ image.segments = 0;
+ header = image_file("adsp.mdt", &size);
+ if (header == NULL) {
+ return false;
+ }
+ count = size >= 52 + 2 * 32 ? header[44] | header[45] << 8 : 0;
+ if (memcmp(header, "\177ELF\1", 5) != 0 || count < 3 || count > MAX_SEGMENTS || 52 + 32 * count > size || size > PAGE_SIZE) {
+ printf("adsp: %u: fw/adsp.mdt is not such a header (%02x %02x %02x %02x, %u segments, %llu bytes)\n", step, header[0],
+ header[1], header[2], header[3], count, size);
+ return false;
+ }
+ for (uint32_t i = 0; i < count; i++) {
+ char name[16];
+
+ memcpy(&segment, header + 52 + 32 * i, sizeof(segment));
+ if (MDT_HASH(segment.flags)) {
+ hashes = segment.file_size;
+ }
+ if (segment.type != 1 || MDT_HASH(segment.flags) || segment.memory_size == 0) {
+ continue;
+ }
+ if (segment.paddr < ADSP_MEMORY || segment.file_size > segment.memory_size ||
+ (uint64_t)segment.paddr + segment.memory_size > (uint64_t)ADSP_MEMORY + ADSP_MEMORY_SIZE) {
+ printf("adsp: %u: segment %u (%#x, %#x bytes) is not within the DSP's memory\n", step, i, segment.paddr,
+ segment.memory_size);
+ return false;
+ }
+ image.segment[image.segments].at = segment.paddr - ADSP_MEMORY;
+ image.segment[image.segments].file_size = segment.file_size;
+ image.segment[image.segments].memory_size = segment.memory_size;
+ image.segment[image.segments].file = NULL;
+ if (segment.file_size != 0) {
+ uint64_t file_size = 0;
+
+ snprintf(name, sizeof(name), "adsp.b%02u", i);
+ image.segment[image.segments].file = image_file(name, &file_size);
+ if (image.segment[image.segments].file == NULL) {
+ return false;
+ }
+ if (file_size != segment.file_size) {
+ printf("adsp: %u: fw/%s has %llu bytes, the header says %u\n", step, name, file_size, segment.file_size);
+ return false;
+ }
+ files += file_size;
+ }
+ lowest = segment.paddr < lowest ? segment.paddr : lowest;
+ highest = segment.paddr + segment.memory_size > highest ? segment.paddr + segment.memory_size : highest;
+ image.segments++;
+ }
+ if (image.segments == 0 || lowest != ADSP_MEMORY || 52 + 32 * count + (uint64_t)hashes != size) {
+ printf("adsp: %u: not what TrustZone takes: %u segments to load, the lowest at %#x; the header file has %llu bytes, "
+ "the header %u and the hashes %u\n", step, image.segments, lowest, size, 52 + 32 * count, hashes);
+ image.segments = 0;
+ return false;
+ }
+ image.header = header;
+ image.header_size = size;
+ image.size = ((highest + 0xfff) & ~0xfffu) - ADSP_MEMORY;
+ printf("adsp: %u: the image is whole: a header of %llu bytes (%u of hashes), %u segments, %llu bytes of files, %#x bytes "
+ "of the DSP's %#x from %#x\n", step, size, hashes, image.segments, files, image.size, ADSP_MEMORY_SIZE, ADSP_MEMORY);
+ return true;
+}
+
+/* ---- step 3: the mailbox --------------------------------------------------------------------------- */
+
+static bool
+step_mailbox(void)
+{
+ bool open;
+
+ announce("open the mailbox to the always-on processor, unless it is open: words of its message memory, and its doorbell");
+ lck_mtx_lock(&redwood_aop_mailbox_lock);
+ open = mailbox_open();
+ lck_mtx_unlock(&redwood_aop_mailbox_lock);
+ return open;
+}
+
+/* ---- TrustZone, and stopping the DSP --------------------------------------------------------------- */
+
+#define SCM_64 (1ull << 30)
+#define SCM_SIP (2ull << 24)
+#define SCM_CALL(service, command) ((uint64_t)(service) << 8 | (command))
+#define SCM_PAS_INIT_IMAGE SCM_CALL(2, 1)
+#define SCM_PAS_MEM_SETUP SCM_CALL(2, 2)
+#define SCM_PAS_AUTH_AND_RESET SCM_CALL(2, 5)
+#define SCM_PAS_SHUTDOWN SCM_CALL(2, 6)
+#define SCM_ONE_VALUE 1
+#define SCM_THREE_VALUES 3
+#define SCM_VALUE_AND_BUFFER (2 | 2 << 6)
+#define SCM_INTERRUPTED 1
+
+static LCK_MTX_DECLARE(dsp_lock, &adsp_lock_group); /* the DSP being started or stopped */
+static bool rails_asked; /* our request for its rails stands (until it asks for its own, or is shut down) */
+static bool dsp_started;
+static bool stopped_by_its_word;
+static uint32_t our_item;
+static bool smp2p_agreed, smp2p_restart_acks, smp2p_restart_acked;
+
+static int32_t
+scm(const char *what, uint64_t function, uint64_t arguments, uint64_t first, uint64_t second, uint64_t third, uint64_t *answer)
+{
+ uint64_t session = 0;
+
+ function |= SCM_64 | SCM_SIP;
+ printf("adsp: %u: about to ask TrustZone %s: smc %#llx, arguments %#llx, %#llx, %#llx\n", step, what, function, first,
+ second, third);
+ snprintf(last_write, sizeof(last_write), "step %u asked TrustZone %#llx(%#llx, %#llx)", step, function, first, second);
+ for (int i = 0; i < 100; i++) {
+ register uint64_t x0 __asm__("x0") = i == 0 ? function : SCM_INTERRUPTED;
+ register uint64_t x1 __asm__("x1") = arguments;
+ register uint64_t x2 __asm__("x2") = first;
+ register uint64_t x3 __asm__("x3") = second;
+ register uint64_t x4 __asm__("x4") = third;
+ register uint64_t x5 __asm__("x5") = 0;
+ register uint64_t x6 __asm__("x6") = session;
+ register uint64_t x7 __asm__("x7") = 0;
+
+ __asm__ volatile ("smc #0" : "+r" (x0), "+r" (x1), "+r" (x2), "+r" (x3), "+r" (x4), "+r" (x5), "+r" (x6),
+ "+r" (x7) : : "x8", "x9", "x10", "x11", "x12", "x13", "x14", "x15", "x16", "x17", "memory");
+ session = x6;
+ if (x0 != SCM_INTERRUPTED) {
+ printf("adsp: %u: TrustZone returns %d, answer %#llx\n", step, (int32_t)x0, x1);
+ *answer = x1;
+ return (int32_t)x0;
+ }
+ }
+ printf("adsp: %u: TrustZone keeps saying the call was interrupted\n", step);
+ return -1;
+}
+
+static uint32_t
+our_item_find(void)
+{
+ uint32_t size = 0, at = smem_item(ITEM_SMP2P_OURS, &size);
+
+ return at != 0 && size >= sizeof(struct smp2p) ? at : 0;
+}
+
+/* What Linux does when the DSP rings for SMP2P: agree on the features once, and answer a restart. */
+static void
+dsp_listen(const struct smp2p *theirs)
+{
+ struct smp2p ours;
+
+ if (our_item == 0 || !smem_peek(our_item, &ours, sizeof(ours)) || ours.magic != SMP2P_MAGIC) {
+ return;
+ }
+ if (!smp2p_agreed) {
+ uint32_t features = (ours.version >> 8) & (theirs->version >> 8);
+
+ if ((theirs->version & 0xff) != (ours.version & 0xff)) {
+ return;
+ }
+ if (features != ours.version >> 8) {
+ smem_poke(our_item + SMP2P_AT_VERSION, features << 8 | (ours.version & 0xff));
+ }
+ smp2p_restart_acks = features & SMP2P_RESTART_ACKS;
+ smp2p_agreed = true;
+ printf("adsp: %u: the DSP and we agree on SMP2P: version %u, features %#x\n", step, ours.version & 0xff, features);
+ }
+ if (smp2p_restart_acks && ((theirs->flags & SMP2P_RESTART_DONE) != 0) != smp2p_restart_acked) {
+ smp2p_restart_acked = !smp2p_restart_acked;
+ printf("adsp: %u: the DSP's flags say it has restarted (%#x): answered in our flags (bit 1 = %u), and its SMP2P doorbell\n",
+ step, theirs->flags, smp2p_restart_acked);
+ smem_poke(our_item + SMP2P_AT_FLAGS, (ours.flags & ~SMP2P_RESTART_ACK) | (smp2p_restart_acked ? SMP2P_RESTART_ACK : 0));
+ settle();
+ wr(ipcc.base + IPCC_SEND, IPCC_LPASS << 16 | IPCC_SMP2P);
+ }
+}
+
+/* Linux's qcom_q6v5_request_stop(): our bit "stop", the DSP's "stop ack" waited for, our bit taken back. */
+static bool
+dsp_ask_to_stop(uint32_t bound_ms)
+{
+ struct smp2p ours, theirs;
+ uint32_t at = our_item_find(), state, value_at;
+ uint64_t asked, now;
+ int entry = -1;
+
+ if (at == 0 || !smem_peek(at, &ours, sizeof(ours)) || ours.magic != SMP2P_MAGIC || (ours.version & 0xff) != SMP2P_VERSION ||
+ (entry = smp2p_entry(&ours, OUR_ENTRY_STOP)) < 0) {
+ printf("adsp: %u: the DSP is not asked to stop first: there is no SMP2P item of ours to ask it with\n", step);
+ return false;
+ }
+ if (!dsp_words(&theirs) || smp2p_entry(&theirs, "slave-kernel") < 0) {
+ printf("adsp: %u: the DSP is not asked to stop first: its item has no entry to answer in\n", step);
+ return false;
+ }
+ state = dsp_state(&theirs);
+ if (state & DSP_STOP_ACK) {
+ printf("adsp: %u: the DSP is not asked to stop: its entry says %#x, that it has stopped when asked before\n", step, state);
+ return true;
+ }
+ if (!(state & DSP_READY) || (state & DSP_FATAL)) {
+ printf("adsp: %u: the DSP is not asked to stop first: its entry says %#x, not \"up\"\n", step, state);
+ return false;
+ }
+ need(&ipcc);
+ value_at = at + SMP2P_AT_VALUE((uint32_t)entry);
+ printf("adsp: %u: about to ask the DSP to stop: bit 0 of our entry \"%s\" at %#lx, and its SMP2P doorbell\n", step,
+ OUR_ENTRY_STOP, SMEM_PHYS + value_at);
+ snprintf(last_write, sizeof(last_write), "step %u asked the audio DSP to stop", step);
+ smem_poke(value_at, ours.entry[entry].value | DSP_STOP);
+ settle();
+ wr(ipcc.base + IPCC_SEND, IPCC_LPASS << 16 | IPCC_SMP2P);
+ asked = now = now_us();
+ while (now - asked < bound_ms * 1000ull) {
+ if (dsp_words(&theirs) && (dsp_state(&theirs) & (DSP_STOP_ACK | DSP_FATAL))) {
+ break;
+ }
+ sleep_us(200);
+ now = now_us();
+ }
+ state = dsp_words(&theirs) ? dsp_state(&theirs) : 0;
+ printf("adsp: %u: %s after %llu.%03llu ms; its entry says %#x\n", step, state & DSP_STOP_ACK ?
+ "THE AUDIO DSP ACKNOWLEDGES THE STOP" : "THE AUDIO DSP DOES NOT ACKNOWLEDGE THE STOP", (now - asked) / 1000,
+ (now - asked) % 1000, state);
+ smem_poke(value_at, ours.entry[entry].value & ~DSP_STOP);
+ settle();
+ wr(ipcc.base + IPCC_SEND, IPCC_LPASS << 16 | IPCC_SMP2P);
+ return (state & DSP_STOP_ACK) != 0;
+}
+
+static void link_stop(void);
+
+/* Asked to stop, then shut down by TrustZone, then the always-on processor told that the image is gone. */
+static bool
+dsp_shutdown(const char *why)
+{
+ uint64_t answer = ~0ull;
+ int32_t result;
+
+ printf("adsp: %u: %s: the audio DSP is asked to stop, then TrustZone to shut it down\n", step, why);
+ if (!leaving) {
+ link_stop();
+ }
+ stopped_by_its_word = dsp_ask_to_stop(leaving ? 2000 : 5000);
+ result = scm("to shut the audio DSP down", SCM_PAS_SHUTDOWN, SCM_ONE_VALUE, PAS_ADSP, 0, 0, &answer);
+ if (result != 0 || answer != 0) {
+ printf("adsp: %u: TRUSTZONE DOES NOT SHUT THE AUDIO DSP DOWN: the call returns %d, its answer is %#llx\n", step, result,
+ answer);
+ return false;
+ }
+ printf("adsp: %u: THE AUDIO DSP IS SHUT DOWN\n", step);
+ need(&ipcc);
+ wr(ipcc.base + IPCC_DISABLE, IPCC_LPASS << 16 | IPCC_SMP2P);
+ doorbells();
+ dsp_started = false;
+ mailbox_send("{class: image, res: load_state, name: adsp, val: off}");
+ if (rails_asked) {
+ rails(0);
+ rails_asked = false;
+ }
+ return true;
+}
+
+/* ---- step 4: TrustZone takes the header ------------------------------------------------------------ */
+
+static bool
+step_header(void)
+{
+ struct smp2p words;
+ vm_offset_t page = 0;
+ uint64_t answer = ~0ull;
+ int32_t result;
+
+ if (image.segments == 0) {
+ return false;
+ }
+ if (dsp_words(&words)) {
+ show_words("the DSP has left words in the shared memory", &words);
+ announce("ask TrustZone to shut down an audio DSP that a kernel before this one may have left running");
+ if (!dsp_shutdown("it may be running")) {
+ return false;
+ }
+ if (!stopped_by_its_word && !option("adsp_again")) {
+ printf("adsp: %u: THE AUDIO DSP IS NOT STARTED AGAIN: it was shut down without having acknowledged a stop, and "
+ "with the Wi-Fi's DSP a start after that reset the phone (adsp_again=1 to try all the same)\n", step);
+ return false;
+ }
+ } else {
+ printf("adsp: %u: the DSP has left nothing in the shared memory: it has not run since the phone was started\n", step);
+ }
+
+ announce("hand the image's header to TrustZone to check its signature (image 1), in a page of ours");
+ if (kmem_alloc(kernel_map, &page, PAGE_SIZE, KMA_KOBJECT | KMA_ZERO | KMA_DATA, VM_KERN_MEMORY_IOKIT) != KERN_SUCCESS) {
+ printf("adsp: %u: no memory\n", step);
+ return false;
+ }
+ memcpy((void *)page, image.header, image.header_size);
+ for (vm_offset_t line = page; line < page + PAGE_SIZE; line += 64) {
+ __asm__ volatile ("dc civac, %0" : : "r" (line));
+ }
+ settle();
+ result = scm("to take the audio DSP's header", SCM_PAS_INIT_IMAGE, SCM_VALUE_AND_BUFFER, PAS_ADSP, kvtophys(page), 0, &answer);
+ if (result != 0 || answer != 0) {
+ printf("adsp: %u: TRUSTZONE DOES NOT TAKE THE HEADER: the call returns %d, its answer is %#llx\n", step, result, answer);
+ return false;
+ }
+ printf("adsp: %u: TRUSTZONE ACCEPTS THE AUDIO DSP'S SIGNATURE FROM THIS KERNEL\n", step);
+
+ announce("tell TrustZone where the image will be");
+ result = scm("where the image will be", SCM_PAS_MEM_SETUP, SCM_THREE_VALUES, PAS_ADSP, ADSP_MEMORY, image.size, &answer);
+ if (result != 0 || answer != 0) {
+ printf("adsp: %u: TRUSTZONE DOES NOT TAKE THE PLACE: the call returns %d, its answer is %#llx\n", step, result, answer);
+ return false;
+ }
+ printf("adsp: %u: TrustZone takes the place: %#x bytes at %#x\n", step, image.size, ADSP_MEMORY);
+ return true;
+}
+
+/* ---- step 5: the image, in the DSP's memory --------------------------------------------------------- */
+
+static bool image_written;
+
+static uint64_t
+segment_word(unsigned s, uint32_t at)
+{
+ uint64_t word = 0;
+
+ for (unsigned b = 0; b < 8 && at + b < image.segment[s].file_size; b++) {
+ word |= (uint64_t)image.segment[s].file[at + b] << (8 * b);
+ }
+ return word;
+}
+
+static bool
+step_image(void)
+{
+ volatile uint64_t *dsp_memory;
+ uint64_t written = 0, cleared = 0, wrong = 0;
+
+ if (image.segments == 0) {
+ return false;
+ }
+ announce("put the image's segments into the audio DSP's memory (0x86700000, 64 MB: memory of nobody's until TrustZone "
+ "starts the DSP), with zeroes where a segment is longer than its file, and read them back");
+ dsp_memory = (volatile uint64_t *)ml_io_map_unmappable(ADSP_MEMORY, ADSP_MEMORY_SIZE, VM_WIMG_WCOMB);
+ snprintf(last_write, sizeof(last_write), "step %u wrote the image at %#x", step, ADSP_MEMORY);
+ for (unsigned s = 0; s < image.segments; s++) {
+ volatile uint64_t *to = dsp_memory + image.segment[s].at / 8;
+
+ if ((image.segment[s].at | image.segment[s].memory_size) & 7) {
+ printf("adsp: %u: a segment that does not begin and end at a multiple of eight bytes: not written\n", step);
+ ml_io_unmap((vm_offset_t)dsp_memory, ADSP_MEMORY_SIZE);
+ return false;
+ }
+ for (uint32_t i = 0; i < image.segment[s].memory_size; i += 8) {
+ to[i / 8] = segment_word(s, i);
+ }
+ written += image.segment[s].file_size;
+ cleared += image.segment[s].memory_size - image.segment[s].file_size;
+ }
+ settle();
+ for (unsigned s = 0; s < image.segments; s++) {
+ volatile uint64_t *from = dsp_memory + image.segment[s].at / 8;
+
+ for (uint32_t i = 0; i < image.segment[s].memory_size; i += 8) {
+ wrong += from[i / 8] != segment_word(s, i);
+ }
+ }
+ printf("adsp: %u: %u segments, %llu bytes written and %llu cleared; read back, %llu words of eight bytes differ\n", step,
+ image.segments, written, cleared, wrong);
+ settle();
+ ml_io_unmap((vm_offset_t)dsp_memory, ADSP_MEMORY_SIZE);
+ image_written = wrong == 0;
+ return image_written;
+}
+
+/* ---- step 6: our SMP2P item ------------------------------------------------------------------------ */
+
+static struct block smem_lock = { "shared memory's lock", 0x01f43000, 0x1000 };
+#define LOCK_OWNER 0x000
+#define LOCK_US 1u
+
+/* Our item made in the partition, as Linux's qcom_smem_alloc_private() makes one (redwood_wifi.c: our_item_make()). */
+static bool
+our_item_make(uint32_t partition)
+{
+ struct smem_partition header;
+ struct smem_entry entry;
+ const uint32_t item_size = (sizeof(struct smp2p) + 7) & ~7u, need_bytes = sizeof(entry) + item_size;
+ uint32_t owner = ~0u, at;
+ bool made = false;
+
+ lck_mtx_lock(&redwood_smem_alloc_lock); /* (the Wi-Fi's start allocates under the same hardware lock) */
+ need(&smem_lock);
+ printf("adsp: %u: about to take the lock (%#x), write an entry for item %u with %u bytes behind it at the partition's mark, "
+ "move the mark by %u, and release the lock\n", step, smem_lock.phys + LOCK_OWNER, ITEM_SMP2P_OURS, item_size, need_bytes);
+ snprintf(last_write, sizeof(last_write), "step %u took the shared memory's lock", step);
+ for (uint32_t ms = 0; ms <= 1000; ms++) {
+ wr(smem_lock.base + LOCK_OWNER, LOCK_US);
+ owner = rd(smem_lock.base + LOCK_OWNER);
+ if (owner == LOCK_US) {
+ break;
+ }
+ sleep_ms(1);
+ }
+ if (owner != LOCK_US) {
+ printf("adsp: %u: THE LOCK IS HELD BY PROCESSOR %u AND STAYS SO FOR A SECOND: nothing is written\n", step, owner);
+ lck_mtx_unlock(&redwood_smem_alloc_lock);
+ return false;
+ }
+ if (!smem_peek(partition, &header, sizeof(header)) || header.free_uncached > header.free_cached ||
+ header.free_cached > header.size) {
+ printf("adsp: %u: the partition's marks make no sense: nothing is written\n", step);
+ goto out;
+ }
+ for (at = sizeof(header); at < header.free_uncached;) {
+ if (!smem_peek(partition + at, &entry, sizeof(entry)) || entry.canary != SMEM_CANARY || entry.item == ITEM_SMP2P_OURS ||
+ entry.size > header.size) {
+ break;
+ }
+ at += sizeof(entry) + entry.padding_header + entry.size;
+ }
+ if (at != header.free_uncached) {
+ printf("adsp: %u: the partition's entries do not end at its mark (+%#x, the mark is +%#x), or the item has appeared: "
+ "nothing is written\n", step, at, header.free_uncached);
+ goto out;
+ }
+ if (at + need_bytes > header.free_cached) {
+ printf("adsp: %u: no room: %u bytes wanted, %u free: nothing is written\n", step, need_bytes, header.free_cached - at);
+ goto out;
+ }
+ smem_poke(partition + at, SMEM_CANARY | ITEM_SMP2P_OURS << 16);
+ smem_poke(partition + at + 4, item_size);
+ smem_poke(partition + at + 8, item_size - (uint32_t)sizeof(struct smp2p));
+ smem_poke(partition + at + 12, 0);
+ settle();
+ smem_poke(partition + __offsetof(struct smem_partition, free_uncached), at + need_bytes);
+ settle();
+ made = true;
+out:
+ wr(smem_lock.base + LOCK_OWNER, 0);
+ printf("adsp: %u: the lock is released (it reads %u)\n", step, rd(smem_lock.base + LOCK_OWNER));
+ lck_mtx_unlock(&redwood_smem_alloc_lock);
+ return made;
+}
+
+static void glink_reset(void);
+
+static bool
+step_our_item(void)
+{
+ struct smem_partition header;
+ struct smp2p ours;
+ uint32_t cacheline = 0, size = 0, partition, name[4] = { 0 };
+
+ announce("look for an SMP2P item of ours (429) in our partition with the audio DSP, make it if there is none, and fill it in");
+ partition = smem_partition(SMEM_US, SMEM_ADSP, &cacheline, &size);
+ if (partition == 0 || !smem_peek(partition, &header, sizeof(header))) {
+ printf("adsp: %u: there is no such partition\n", step);
+ return false;
+ }
+ our_item = our_item_find();
+ if (our_item != 0) {
+ printf("adsp: %u: item %u is there, at %#lx: a kernel before this one made it, and it is used again\n", step,
+ ITEM_SMP2P_OURS, SMEM_PHYS + our_item);
+ } else {
+ if (!our_item_make(partition)) {
+ return false;
+ }
+ our_item = our_item_find();
+ if (our_item == 0) {
+ printf("adsp: %u: THE ITEM IS NOT FOUND where it was just made\n", step);
+ return false;
+ }
+ printf("adsp: %u: item %u made, at %#lx\n", step, ITEM_SMP2P_OURS, SMEM_PHYS + our_item);
+ }
+ /* zeroes; from processor 0 to 2, room for 16 entries, "master-kernel" at zero, feature bit 0; last the version */
+ snprintf(last_write, sizeof(last_write), "step %u filled in our SMP2P item", step);
+ for (uint32_t at = 0; at < sizeof(ours); at += 4) {
+ smem_poke(our_item + at, 0);
+ }
+ settle();
+ smem_poke(our_item, SMP2P_MAGIC);
+ smem_poke(our_item + SMP2P_AT_PROCESSORS, SMEM_US | SMEM_ADSP << 16);
+ strlcpy((char *)name, OUR_ENTRY_STOP, sizeof(name));
+ for (uint32_t word = 0; word < 4; word++) {
+ smem_poke(our_item + SMP2P_AT_NAME(0) + 4 * word, name[word]);
+ }
+ smem_poke(our_item + SMP2P_AT_COUNTS, 16 | 1u << 16);
+ smem_poke(our_item + SMP2P_AT_VERSION, SMP2P_RESTART_ACKS << 8);
+ settle();
+ smem_poke(our_item + SMP2P_AT_VERSION, SMP2P_RESTART_ACKS << 8 | SMP2P_VERSION);
+ settle();
+ smp2p_agreed = smp2p_restart_acks = smp2p_restart_acked = false;
+ need(&ipcc);
+ wr(ipcc.base + IPCC_SEND, IPCC_LPASS << 16 | IPCC_SMP2P);
+ if (!smem_peek(our_item, &ours, sizeof(ours)) || ours.magic != SMP2P_MAGIC || smp2p_entry(&ours, OUR_ENTRY_STOP) < 0) {
+ printf("adsp: %u: OUR ITEM DOES NOT READ BACK AS WRITTEN\n", step);
+ return false;
+ }
+ show_words("our item reads back", &ours);
+ glink_reset();
+ return true;
+}
+
+/* ---- step 7: the DSP, started ---------------------------------------------------------------------- */
+
+/*
+ * Android's kernel (this phone, dmesg): the image loaded at 8.92 s, "brought out of reset" at
+ * 9.10, "error ready" 40 ms later; the audio services' protection domain up at 9.39.
+ * Here (runs 085341 and later): the DSP's SMP2P item appears 17 ms after TrustZone returns, with
+ * a ring of its SMP2P and GLINK doorbells; at 41 ms "slave-kernel" says 6, up and asking for its
+ * own power, and our request for the rails is given back; at 2.2 s it adds "sleepstate_see".
+ * Asked to stop, it acknowledges within 25 ms (0x10e), and it starts again in the same session.
+ */
+static bool
+step_start(void)
+{
+ struct smp2p words, last;
+ uint64_t answer = ~0ull, started, now, ready_at = 0, limit;
+ uint32_t state = 0, rang, changes = 0, wait_seconds = 10;
+ int32_t result;
+ bool have, spoke = false, handed_over = false;
+
+ if (!image_written) {
+ printf("adsp: %u: the image is not in place\n", step);
+ return false;
+ }
+ PE_parse_boot_argn("adsp_wait", &wait_seconds, sizeof(wait_seconds));
+ bzero(&last, sizeof(last));
+ have = dsp_words(&last);
+ announce("listen for the audio DSP's doorbells, ask RPMh for its rails, tell the always-on processor that its image is "
+ "loaded, and ask TrustZone to authenticate the image and let the DSP out of reset");
+ need(&ipcc);
+ for (unsigned i = 0; i < 8 && (rang = rd(ipcc.base + IPCC_RECEIVED)) != IPCC_NOBODY && rang >> 16 == IPCC_LPASS; i++) {
+ put(&ipcc, "heard", IPCC_CLEAR, rang);
+ }
+ put(&ipcc, "listened for", IPCC_ENABLE, IPCC_LPASS << 16 | IPCC_SMP2P);
+ put(&ipcc, "listened for", IPCC_ENABLE, IPCC_LPASS << 16 | IPCC_GLINK);
+ bzero(&doorbell, sizeof(doorbell));
+ lck_mtx_lock(&dsp_lock);
+ if (!rails(384)) {
+ lck_mtx_unlock(&dsp_lock);
+ return false;
+ }
+ rails_asked = true;
+ if (!mailbox_send("{class: image, res: load_state, name: adsp, val: on}")) {
+ rails(0);
+ rails_asked = false;
+ lck_mtx_unlock(&dsp_lock);
+ return false;
+ }
+ result = scm("to authenticate the audio DSP's image and start it", SCM_PAS_AUTH_AND_RESET, SCM_ONE_VALUE, PAS_ADSP, 0, 0,
+ &answer);
+ doorbell.since = started = now_us();
+ image_written = false;
+ if (result != 0 || answer != 0) {
+ printf("adsp: %u: TRUSTZONE DOES NOT START THE AUDIO DSP: the call returns %d, its answer is %#llx\n", step, result,
+ answer);
+ mailbox_send("{class: image, res: load_state, name: adsp, val: off}");
+ rails(0);
+ rails_asked = false;
+ lck_mtx_unlock(&dsp_lock);
+ return false;
+ }
+ dsp_started = true;
+ lck_mtx_unlock(&dsp_lock);
+ printf("adsp: %u: TRUSTZONE HAS AUTHENTICATED THE IMAGE AND LET THE AUDIO DSP OUT OF RESET; watching for %u s\n", step,
+ wait_seconds);
+ limit = started + wait_seconds * 1000000ull;
+ for (;;) {
+ now = now_us();
+ if (now >= limit || !dsp_started) {
+ break;
+ }
+ doorbells();
+ spoke = spoke || doorbell.glink + doorbell.smp2p != 0;
+ if (dsp_words(&words) && (!have || memcmp(&words, &last, sizeof(words)) != 0)) {
+ char when[64];
+
+ snprintf(when, sizeof(when), "+%llu.%03llu ms: the DSP's item %s", (now - started) / 1000, (now - started) % 1000,
+ have ? "changes" : "APPEARS");
+ if (changes++ < 40) {
+ show_words(when, &words);
+ }
+ last = words;
+ have = spoke = true;
+ dsp_listen(&words);
+ state = dsp_state(&words);
+ if ((state & DSP_READY) && ready_at == 0) {
+ ready_at = now;
+ limit = now + 3000000ull;
+ printf("adsp: %u: THE AUDIO DSP SAYS IT IS UP: \"slave-kernel\" says %#x, %llu.%03llu ms after TrustZone started "
+ "it\n", step, state, (now - started) / 1000, (now - started) % 1000);
+ }
+ if ((state & DSP_HANDOVER) && !handed_over) {
+ handed_over = true;
+ printf("adsp: %u: the DSP says it has asked for its own power (bit 2), %llu.%03llu ms after the start: ours is "
+ "given back\n", step, (now - started) / 1000, (now - started) % 1000);
+ lck_mtx_lock(&dsp_lock);
+ if (rails_asked) {
+ rails(0);
+ rails_asked = false;
+ }
+ lck_mtx_unlock(&dsp_lock);
+ }
+ if (state & DSP_FATAL) {
+ printf("adsp: %u: THE AUDIO DSP SAYS IT HAS GIVEN UP\n", step);
+ show_crash_reason();
+ break;
+ }
+ }
+ sleep_us(200);
+ }
+ printf("adsp: %u: the watch is over after %llu ms; its doorbells heard: GLINK %u, SMP2P %u\n", step,
+ (now_us() - started) / 1000, doorbell.glink, doorbell.smp2p);
+ show_partition();
+ if (our_item != 0 && smem_peek(our_item, &words, sizeof(words))) {
+ show_words("our item (429) now", &words);
+ }
+ {
+ /* (what is at the bottom of our partition, entry by entry, as it is now) */
+ uint32_t cacheline = 0, size = 0, partition = smem_partition(SMEM_US, SMEM_ADSP, &cacheline, &size);
+ struct smem_partition header;
+ struct smem_entry entry;
+
+ if (partition != 0 && smem_peek(partition, &header, sizeof(header))) {
+ printf("adsp: %u: the partition's header: hosts %u and %u, size %#x, free from the bottom at +%#x, from the top at "
+ "+%#x; the table's line size %u\n", step, header.host0, header.host1, header.size, header.free_uncached,
+ header.free_cached, cacheline);
+ for (uint32_t at = sizeof(header), n = 0; at + sizeof(entry) <= header.size && n < 8; n++) {
+ smem_peek(partition + at, &entry, sizeof(entry));
+ printf("adsp: %u: +%#x: canary %#x item %u size %u padding %u/%u\n", step, at, entry.canary, entry.item,
+ entry.size, entry.padding_data, entry.padding_header);
+ if (entry.canary != SMEM_CANARY) {
+ break;
+ }
+ at += sizeof(entry) + entry.padding_header + entry.size;
+ }
+ }
+ }
+ if (state & DSP_FATAL) {
+ return false;
+ }
+ if (ready_at == 0) {
+ printf("adsp: %u: %s\n", step, spoke ? "THE AUDIO DSP RUNS BUT HAS NOT SAID \"UP\"" : "NOT A WORD FROM THE AUDIO DSP");
+ show_crash_reason();
+ }
+ return ready_at != 0;
+}
+
+/* ---- GLINK (redwood_wifi.c; items 478 to 480 of our partition with the audio DSP) ------------------- */
+
+#define ITEM_GLINK_DESCRIPTOR 478
+#define ITEM_GLINK_OUR_QUEUE 479
+#define ITEM_GLINK_THEIR_QUEUE 480
+#define GLINK_OUR_TAIL 0
+#define GLINK_OUR_HEAD 4
+#define GLINK_THEIR_TAIL 8
+#define GLINK_THEIR_HEAD 12
+#define GLINK_RESERVE 16
+
+enum {
+ GLINK_VERSION = 0, GLINK_VERSION_ACK = 1, GLINK_OPEN = 2, GLINK_CLOSE = 3, GLINK_OPEN_ACK = 4, GLINK_INTENT = 5,
+ GLINK_DONE = 6, GLINK_INTENT_REQUEST = 7, GLINK_INTENT_REQUEST_ACK = 8, GLINK_DATA = 9, GLINK_CLOSE_ACK = 11,
+ GLINK_DATA_MORE = 12, GLINK_READ_NOTIFY = 13, GLINK_DONE_REUSE = 14, GLINK_SIGNALS = 15,
+};
+#define GLINK_VERSION_1 1
+#define GLINK_INTENT_REUSE 1u
+#define GLINK_CHANNEL "IPCRTR"
+#define GLINK_US 1
+
+struct glink_message {
+ uint16_t command, param1;
+ uint32_t param2;
+};
+struct glink_packet {
+ struct glink_packet *next;
+ uint32_t length;
+ uint8_t data[];
+};
+
+static LCK_MTX_DECLARE(glink_lock, &adsp_lock_group);
+static struct {
+ uint32_t descriptor, ours, ours_size, theirs, theirs_size;
+ bool up, broken;
+ uint32_t features;
+ bool agreed;
+ uint32_t their_channel;
+ bool asked, acknowledged, open;
+ bool request_answered, request_granted;
+ struct {
+ uint32_t id, size;
+ bool there, in_use;
+ } their_intents[64];
+ struct {
+ uint32_t size, fill;
+ uint8_t *data;
+ } our_intents[24];
+ uint32_t our_intent_count;
+ struct glink_packet *first, *last;
+ uint32_t said, heard, trace;
+ char others[4][33]; /* the channels the DSP opens that nobody here speaks (for the log) */
+} glink;
+
+static bool
+glink_find(void)
+{
+ uint32_t size = 0;
+
+ glink.descriptor = smem_item(ITEM_GLINK_DESCRIPTOR, &size);
+ if (glink.descriptor == 0 || size != 32) {
+ return false;
+ }
+ glink.ours = smem_item(ITEM_GLINK_OUR_QUEUE, &glink.ours_size);
+ glink.theirs = smem_item(ITEM_GLINK_THEIR_QUEUE, &glink.theirs_size);
+ return glink.ours != 0 && glink.theirs != 0 && glink.ours_size >= 1024 && glink.theirs_size >= 1024 &&
+ !(glink.ours_size & 7) && !(glink.theirs_size & 7);
+}
+
+static void
+glink_reset(void)
+{
+ uint32_t size = 0, descriptor = smem_item(ITEM_GLINK_DESCRIPTOR, &size);
+
+ if (descriptor == 0 || size != 32) {
+ printf("adsp: %u: GLINK's descriptor (item %u) is not there: nothing to reset\n", step, ITEM_GLINK_DESCRIPTOR);
+ return;
+ }
+ printf("adsp: %u: GLINK's descriptor at %#lx: our queue read to %#x and written to %#x, the DSP's read to %#x and written "
+ "to %#x\n", step, SMEM_PHYS + descriptor, smem_word(descriptor + GLINK_OUR_TAIL), smem_word(descriptor + GLINK_OUR_HEAD),
+ smem_word(descriptor + GLINK_THEIR_TAIL), smem_word(descriptor + GLINK_THEIR_HEAD));
+ if (smem_word(descriptor + GLINK_OUR_HEAD) != 0 || smem_word(descriptor + GLINK_THEIR_TAIL) != 0) {
+ printf("adsp: %u: about to write our two marks there (+4, +8): 0\n", step);
+ smem_poke(descriptor + GLINK_OUR_HEAD, 0);
+ smem_poke(descriptor + GLINK_THEIR_TAIL, 0);
+ settle();
+ }
+}
+
+static uint32_t
+glink_waiting(void)
+{
+ uint32_t head = smem_word(glink.descriptor + GLINK_THEIR_HEAD), tail = smem_word(glink.descriptor + GLINK_THEIR_TAIL);
+
+ if (head >= glink.theirs_size || tail >= glink.theirs_size) {
+ return 0;
+ }
+ return head >= tail ? head - tail : glink.theirs_size - tail + head;
+}
+
+static void
+glink_peek(void *to, uint32_t offset, uint32_t count)
+{
+ uint32_t at = (smem_word(glink.descriptor + GLINK_THEIR_TAIL) + offset) % glink.theirs_size;
+
+ for (uint32_t i = 0; i < count; i++) {
+ ((uint8_t *)to)[i] = smem[glink.theirs + (at + i) % glink.theirs_size];
+ }
+}
+
+static void
+glink_advance(uint32_t count)
+{
+ smem_poke(glink.descriptor + GLINK_THEIR_TAIL, (smem_word(glink.descriptor + GLINK_THEIR_TAIL) + count) % glink.theirs_size);
+}
+
+static uint32_t
+glink_room(void)
+{
+ uint32_t head = smem_word(glink.descriptor + GLINK_OUR_HEAD), tail = smem_word(glink.descriptor + GLINK_OUR_TAIL), room;
+
+ if (head >= glink.ours_size || tail >= glink.ours_size) {
+ return 0;
+ }
+ room = tail <= head ? glink.ours_size - head + tail : tail - head;
+ return room < GLINK_RESERVE ? 0 : room - GLINK_RESERVE;
+}
+
+static bool
+glink_put(const void *first, uint32_t first_length, const void *second, uint32_t second_length)
+{
+ volatile uint8_t *queue = (volatile uint8_t *)(uintptr_t)smem + glink.ours;
+ uint32_t head = smem_word(glink.descriptor + GLINK_OUR_HEAD), length = first_length + second_length;
+ uint32_t padded = (length + 7) & ~7u;
+
+ if (!glink.up || glink_room() < padded) {
+ return false;
+ }
+ for (uint32_t i = 0; i < padded; i++) {
+ queue[(head + i) % glink.ours_size] = i < first_length ? ((const uint8_t *)first)[i] :
+ i < length ? ((const uint8_t *)second)[i - first_length] : 0;
+ }
+ settle();
+ smem_poke(glink.descriptor + GLINK_OUR_HEAD, (head + padded) % glink.ours_size);
+ settle();
+ wr(ipcc.base + IPCC_SEND, IPCC_LPASS << 16 | IPCC_GLINK);
+ glink.said++;
+ return true;
+}
+
+static const char *
+glink_name(unsigned command)
+{
+ static const char *const names[] = {
+ "version", "version-ack", "open", "close", "open-ack", "intent", "done", "intent-request", "intent-request-ack",
+ "data", "10", "close-ack", "data-more", "read-notify", "done-reuse", "signals"
+ };
+
+ return command < sizeof(names) / sizeof(names[0]) ? names[command] : "?";
+}
+
+static void
+glink_say(unsigned command, uint32_t param1, uint32_t param2, const void *more, uint32_t more_length)
+{
+ struct glink_message message = { .command = (uint16_t)command, .param1 = (uint16_t)param1, .param2 = param2 };
+ bool sent = glink_put(&message, sizeof(message), more, more_length);
+
+ if (glink.trace > 0 || !sent) {
+ glink.trace -= glink.trace > 0;
+ printf("adsp: glink: > %s %u %#x%s\n", glink_name(command), param1, param2, sent ? "" : ": NO ROOM IN OUR QUEUE, not sent");
+ }
+}
+
+static bool
+glink_offer(uint32_t size)
+{
+ uint32_t words[2] = { size, glink.our_intent_count + 1 }, i = glink.our_intent_count;
+
+ if (i >= sizeof(glink.our_intents) / sizeof(glink.our_intents[0]) || size == 0 || size > 0x10000) {
+ return false;
+ }
+ if (glink.our_intents[i].data == NULL || glink.our_intents[i].size != size) {
+ glink.our_intents[i].data = kalloc_data(size, Z_WAITOK | Z_ZERO);
+ }
+ if (glink.our_intents[i].data == NULL) {
+ return false;
+ }
+ glink.our_intents[i].size = size;
+ glink.our_intents[i].fill = 0;
+ glink.our_intent_count++;
+ glink_say(GLINK_INTENT, GLINK_US, 1, words, sizeof(words));
+ return true;
+}
+
+/* Everything the DSP has put into its queue, taken out and acted on (Linux's qcom_glink_native_rx()). With glink_lock held. */
+static void
+glink_pump(void)
+{
+ static const struct {
+ uint32_t size, count;
+ } intents[] = { { 0x800, 5 }, { 0x2000, 3 }, { 0x4400, 2 } }; /* dt: qcom,intents of qcom,adsp_qrtr */
+ struct glink_message message;
+ uint32_t words[4];
+ char name[33];
+
+ while (glink.up && !glink.broken) {
+ uint32_t waiting = glink_waiting(), length = sizeof(message);
+
+ if (waiting < sizeof(message)) {
+ return;
+ }
+ glink_peek(&message, 0, sizeof(message));
+ if (message.command == GLINK_OPEN) {
+ length += message.param2;
+ } else if (message.command == GLINK_INTENT) {
+ length += 8 * message.param2;
+ } else if (message.command == GLINK_DATA || message.command == GLINK_DATA_MORE) {
+ if (waiting < 16) {
+ return;
+ }
+ glink_peek(words, 8, 8);
+ length = 16 + words[0];
+ }
+ length = (length + 7) & ~7u;
+ if (length > glink.theirs_size - 8) {
+ printf("adsp: glink: A MESSAGE OF %u BYTES (%s %u %#x): not believed, and nothing more is read\n", length,
+ glink_name(message.command), message.param1, message.param2);
+ glink.broken = true;
+ return;
+ }
+ if (waiting < length) {
+ return;
+ }
+ glink.heard++;
+ if (glink.trace > 0 && message.command != GLINK_DATA && message.command != GLINK_DATA_MORE) {
+ glink.trace--;
+ printf("adsp: glink: < %s %u %#x\n", glink_name(message.command), message.param1, message.param2);
+ }
+ switch (message.command) {
+ case GLINK_VERSION:
+ if (message.param1 == GLINK_VERSION_1) {
+ glink.features &= message.param2;
+ }
+ glink_advance(length);
+ glink_say(GLINK_VERSION_ACK, GLINK_VERSION_1, glink.features, NULL, 0);
+ break;
+ case GLINK_VERSION_ACK:
+ glink_advance(length);
+ if (message.param1 == GLINK_VERSION_1 && message.param2 == glink.features) {
+ glink.agreed = true;
+ } else if (message.param1 != 0) {
+ glink.features &= message.param2;
+ glink_say(GLINK_VERSION, GLINK_VERSION_1, glink.features, NULL, 0);
+ }
+ break;
+ case GLINK_OPEN:
+ bzero(name, sizeof(name));
+ glink_peek(name, sizeof(message), message.param2 < sizeof(name) - 1 ? message.param2 : sizeof(name) - 1);
+ glink_advance(length);
+ if (strcmp(name, GLINK_CHANNEL) != 0) {
+ printf("adsp: glink: the DSP opens \"%s\" as its channel %u: left unanswered, nobody here speaks it\n", name,
+ message.param1);
+ for (unsigned i = 0; i < 4; i++) {
+ if (glink.others[i][0] == 0) {
+ strlcpy(glink.others[i], name, sizeof(glink.others[i]));
+ break;
+ }
+ }
+ break;
+ }
+ printf("adsp: glink: the DSP opens \"%s\" as its channel %u: acknowledged, and opened from our side as channel %u\n",
+ name, message.param1, GLINK_US);
+ glink.their_channel = message.param1;
+ glink_say(GLINK_OPEN_ACK, glink.their_channel, 0, NULL, 0);
+ if (!glink.asked) {
+ bzero(name, sizeof(name));
+ strlcpy(name, GLINK_CHANNEL, sizeof(name));
+ glink_say(GLINK_OPEN, GLINK_US, sizeof(GLINK_CHANNEL), name, (sizeof(GLINK_CHANNEL) + 7) & ~7u);
+ glink.asked = true;
+ }
+ break;
+ case GLINK_OPEN_ACK:
+ glink_advance(length);
+ if (message.param1 == GLINK_US) {
+ glink.acknowledged = true;
+ }
+ break;
+ case GLINK_INTENT:
+ for (uint32_t i = 0; i < message.param2 && message.param1 == glink.their_channel; i++) {
+ glink_peek(words, 8 + 8 * i, 8);
+ for (unsigned n = 0; n < sizeof(glink.their_intents) / sizeof(glink.their_intents[0]); n++) {
+ if (!glink.their_intents[n].there || glink.their_intents[n].id == words[1]) {
+ glink.their_intents[n].there = true;
+ glink.their_intents[n].in_use = false;
+ glink.their_intents[n].size = words[0];
+ glink.their_intents[n].id = words[1];
+ break;
+ }
+ }
+ }
+ glink_advance(length);
+ break;
+ case GLINK_DONE:
+ case GLINK_DONE_REUSE:
+ for (unsigned n = 0; n < sizeof(glink.their_intents) / sizeof(glink.their_intents[0]); n++) {
+ if (glink.their_intents[n].there && glink.their_intents[n].id == message.param2) {
+ glink.their_intents[n].in_use = false;
+ glink.their_intents[n].there = message.command == GLINK_DONE_REUSE;
+ }
+ }
+ glink_advance(length);
+ break;
+ case GLINK_INTENT_REQUEST:
+ glink_advance(length);
+ if (message.param1 == glink.their_channel) {
+ bool granted = glink_offer(message.param2);
+
+ glink_say(GLINK_INTENT_REQUEST_ACK, GLINK_US, granted, NULL, 0);
+ }
+ break;
+ case GLINK_INTENT_REQUEST_ACK:
+ glink_advance(length);
+ glink.request_answered = true;
+ glink.request_granted = message.param2 != 0;
+ break;
+ case GLINK_DATA:
+ case GLINK_DATA_MORE: {
+ uint32_t intent = message.param2 - 1;
+
+ if (message.param1 != glink.their_channel || intent >= glink.our_intent_count ||
+ words[0] > glink.our_intents[intent].size - glink.our_intents[intent].fill) {
+ printf("adsp: glink: < data of %u bytes on channel %u for intent %u: NOT OURS OR TOO MUCH, dropped\n", words[0],
+ message.param1, message.param2);
+ glink_advance(length);
+ break;
+ }
+ glink_peek(glink.our_intents[intent].data + glink.our_intents[intent].fill, 16, words[0]);
+ glink.our_intents[intent].fill += words[0];
+ glink_advance(length);
+ if (words[1] == 0) {
+ struct glink_packet *packet = kalloc_data(sizeof(*packet) + glink.our_intents[intent].fill, Z_WAITOK);
+
+ if (packet != NULL) {
+ packet->next = NULL;
+ packet->length = glink.our_intents[intent].fill;
+ memcpy(packet->data, glink.our_intents[intent].data, packet->length);
+ if (glink.last != NULL) {
+ glink.last->next = packet;
+ } else {
+ glink.first = packet;
+ }
+ glink.last = packet;
+ }
+ glink.our_intents[intent].fill = 0;
+ glink_say(GLINK_DONE_REUSE, GLINK_US, message.param2, NULL, 0);
+ }
+ break;
+ }
+ case GLINK_READ_NOTIFY:
+ glink_advance(length);
+ wr(ipcc.base + IPCC_SEND, IPCC_LPASS << 16 | IPCC_GLINK);
+ break;
+ case GLINK_CLOSE:
+ glink_advance(length);
+ glink_say(GLINK_CLOSE_ACK, message.param1, 0, NULL, 0);
+ if (message.param1 == glink.their_channel) {
+ printf("adsp: glink: THE DSP CLOSES \"%s\"\n", GLINK_CHANNEL);
+ glink.open = false;
+ glink.their_channel = 0;
+ }
+ break;
+ case GLINK_CLOSE_ACK:
+ case GLINK_SIGNALS:
+ glink_advance(length);
+ break;
+ default:
+ printf("adsp: glink: A MESSAGE THIS DOES NOT KNOW (%u, %u, %#x): nothing more is read\n", message.command,
+ message.param1, message.param2);
+ glink.broken = true;
+ return;
+ }
+ if (glink.their_channel != 0 && glink.acknowledged && !glink.open) {
+ glink.open = true;
+ printf("adsp: glink: THE CHANNEL \"%s\" IS OPEN both ways (ours %u, the DSP's %u); telling the DSP our intents\n",
+ GLINK_CHANNEL, GLINK_US, glink.their_channel);
+ for (unsigned i = 0; i < sizeof(intents) / sizeof(intents[0]); i++) {
+ for (unsigned n = 0; n < intents[i].count; n++) {
+ glink_offer(intents[i].size);
+ }
+ }
+ }
+ }
+}
+
+/* One whole message to the DSP on the channel; waits up to five seconds for an intent of the DSP's that it fits. */
+static int
+glink_send(const void *data, uint32_t length)
+{
+ struct {
+ struct glink_message message;
+ uint32_t size, left;
+ } header = { { GLINK_DATA, GLINK_US, 0 }, length, 0 };
+ bool asked = false, fits;
+ int found = -1;
+
+ lck_mtx_lock(&glink_lock);
+ if (glink.up && length + sizeof(header) + 8 > glink.ours_size - GLINK_RESERVE) {
+ lck_mtx_unlock(&glink_lock);
+ return EMSGSIZE;
+ }
+ for (unsigned waited = 0;; waited++) {
+ glink_pump();
+ if (!glink.up || !glink.open || glink.broken) {
+ lck_mtx_unlock(&glink_lock);
+ return ENOTCONN;
+ }
+ found = -1;
+ fits = false;
+ for (unsigned n = 0; n < sizeof(glink.their_intents) / sizeof(glink.their_intents[0]); n++) {
+ if (!glink.their_intents[n].there || glink.their_intents[n].size < length) {
+ continue;
+ }
+ fits = true;
+ if (!glink.their_intents[n].in_use && (found < 0 || glink.their_intents[n].size < glink.their_intents[found].size)) {
+ found = (int)n;
+ }
+ }
+ if (found >= 0 && glink_room() >= ((length + sizeof(header) + 7) & ~7u)) {
+ break;
+ }
+ if (!fits && !asked) {
+ glink.request_answered = false;
+ glink_say(GLINK_INTENT_REQUEST, GLINK_US, length, NULL, 0);
+ asked = true;
+ }
+ if (waited >= 5000 || (!fits && asked && glink.request_answered && !glink.request_granted)) {
+ lck_mtx_unlock(&glink_lock);
+ return found < 0 ? ENOBUFS : EAGAIN;
+ }
+ lck_mtx_unlock(&glink_lock);
+ sleep_ms(1);
+ lck_mtx_lock(&glink_lock);
+ }
+ glink.their_intents[found].in_use = true;
+ header.message.param2 = glink.their_intents[found].id;
+ glink_put(&header, sizeof(header), data, length);
+ lck_mtx_unlock(&glink_lock);
+ return 0;
+}
+
+/* ---- QRTR: hello, and the services the DSP tells of (redwood_wifi.c, without ports of our own yet) ---- */
+
+#define QRTR_US 1u
+#define QRTR_CONTROL 0xfffffffeu
+enum { QRTR_DATA = 1, QRTR_HELLO = 2, QRTR_BYE = 3, QRTR_NEW_SERVER = 4, QRTR_DEL_SERVER = 5, QRTR_DEL_CLIENT = 6, QRTR_RESUME_TX = 7,
+ QRTR_NEW_LOOKUP = 10, QRTR_DEL_LOOKUP = 11 };
+
+struct qrtr_header {
+ uint32_t version, type, from_node, from_port, confirm, size, to_node, to_port;
+};
+struct qrtr_control {
+ uint32_t command;
+ uint32_t words[4];
+};
+
+static struct {
+ uint32_t service, instance, node, port;
+ bool there;
+} qrtr_servers[64];
+static uint32_t qrtr_peer, qrtr_in, qrtr_out, qrtr_hellos, qrtr_lookups;
+
+static int
+qrtr_out_packet(uint32_t type, uint32_t from_port, uint32_t to_node, uint32_t to_port, const void *data, uint32_t length)
+{
+ uint32_t padded = (length + 3) & ~3u;
+ struct qrtr_header *packet;
+ int error;
+
+ packet = kalloc_data(sizeof(*packet) + padded, Z_WAITOK | Z_ZERO);
+ if (packet == NULL) {
+ return ENOMEM;
+ }
+ packet->version = 1;
+ packet->type = type;
+ packet->from_node = QRTR_US;
+ packet->from_port = from_port;
+ packet->size = length;
+ packet->to_node = to_node;
+ packet->to_port = to_port;
+ memcpy(packet + 1, data, length);
+ error = glink_send(packet, (uint32_t)sizeof(*packet) + padded);
+ kfree_data(packet, sizeof(*packet) + padded);
+ qrtr_out += error == 0;
+ return error;
+}
+
+static void
+qrtr_control(const struct qrtr_header *header, const struct qrtr_control *control)
+{
+ struct qrtr_control hello = { .command = QRTR_HELLO };
+ int found = -1, error;
+
+ switch (control->command) {
+ case QRTR_HELLO:
+ if (qrtr_hellos++ != 0) {
+ break;
+ }
+ error = qrtr_out_packet(QRTR_HELLO, QRTR_CONTROL, header->from_node, QRTR_CONTROL, &hello, sizeof(hello));
+ printf("adsp: qrtr: HELLO FROM NODE %u; ours to it: %s\n", header->from_node, error == 0 ? "SENT" : "NOT SENT");
+ break;
+ case QRTR_NEW_SERVER:
+ for (unsigned i = 0; i < sizeof(qrtr_servers) / sizeof(qrtr_servers[0]); i++) {
+ if (qrtr_servers[i].there && qrtr_servers[i].node == control->words[2] && qrtr_servers[i].port == control->words[3]) {
+ found = (int)i;
+ break;
+ }
+ if (!qrtr_servers[i].there && found < 0) {
+ found = (int)i;
+ }
+ }
+ if (found >= 0) {
+ qrtr_servers[found].there = true;
+ qrtr_servers[found].service = control->words[0];
+ qrtr_servers[found].instance = control->words[1];
+ qrtr_servers[found].node = control->words[2];
+ qrtr_servers[found].port = control->words[3];
+ }
+ printf("adsp: qrtr: a service: %u (%#x), version %u, instance %u, at node %u port %u\n", control->words[0],
+ control->words[0], control->words[1] & 0xff, control->words[1] >> 8, control->words[2], control->words[3]);
+ break;
+ case QRTR_DEL_SERVER:
+ printf("adsp: qrtr: service %u at node %u port %u has gone\n", control->words[0], control->words[2], control->words[3]);
+ for (unsigned i = 0; i < sizeof(qrtr_servers) / sizeof(qrtr_servers[0]); i++) {
+ if (qrtr_servers[i].there && qrtr_servers[i].node == control->words[2] && qrtr_servers[i].port == control->words[3]) {
+ qrtr_servers[i].there = false;
+ }
+ }
+ break;
+ case QRTR_NEW_LOOKUP:
+ /* the DSP looks for a service of ours (a protection domain's locator, say): there are none yet */
+ if (qrtr_lookups++ < 16) {
+ printf("adsp: qrtr: the DSP looks for service %u (%#x), instance %u: nobody here offers it\n", control->words[0],
+ control->words[0], control->words[1]);
+ }
+ break;
+ case QRTR_BYE:
+ printf("adsp: qrtr: BYE from node %u\n", header->from_node);
+ bzero(qrtr_servers, sizeof(qrtr_servers));
+ break;
+ default:
+ break;
+ }
+}
+
+static void
+qrtr_input(const uint8_t *data, uint32_t length)
+{
+ struct qrtr_header header;
+ uint32_t at;
+
+ if (length >= sizeof(header) && data[0] == 1) {
+ memcpy(&header, data, sizeof(header));
+ at = sizeof(header);
+ } else if (length >= 16 && data[0] == 3) {
+ bzero(&header, sizeof(header));
+ header.type = data[1];
+ header.confirm = data[2] & 1;
+ memcpy(&header.size, data + 4, 4);
+ header.from_node = (uint32_t)(data[8] | data[9] << 8);
+ header.from_port = (uint32_t)(data[10] | data[11] << 8);
+ header.to_node = (uint32_t)(data[12] | data[13] << 8);
+ header.to_port = (uint32_t)(data[14] | data[15] << 8);
+ header.from_port = header.from_port == 0xfffe ? QRTR_CONTROL : header.from_port;
+ header.to_port = header.to_port == 0xfffe ? QRTR_CONTROL : header.to_port;
+ at = 16 + data[3];
+ } else {
+ printf("adsp: qrtr: %u bytes that are no datagram (they begin %#x)\n", length, length ? data[0] : 0);
+ return;
+ }
+ if (header.size == 0 || at + header.size > length) {
+ printf("adsp: qrtr: a datagram of %u bytes in a message of %u: dropped\n", header.size, length);
+ return;
+ }
+ qrtr_in++;
+ if (qrtr_peer == 0) {
+ qrtr_peer = header.from_node;
+ }
+ if (header.type == QRTR_RESUME_TX) {
+ return;
+ }
+ if (header.confirm) {
+ struct qrtr_control resume = { .command = QRTR_RESUME_TX, .words = { header.to_node, header.to_port } };
+
+ qrtr_out_packet(QRTR_RESUME_TX, header.to_port, header.from_node, header.from_port, &resume, sizeof(resume));
+ }
+ if (header.to_port == QRTR_CONTROL || header.type == QRTR_HELLO || header.type == QRTR_BYE ||
+ header.type == QRTR_NEW_SERVER || header.type == QRTR_DEL_SERVER || header.type == QRTR_NEW_LOOKUP) {
+ struct qrtr_control control = { 0 };
+
+ memcpy(&control, data + at, header.size < sizeof(control) ? header.size : sizeof(control));
+ qrtr_control(&header, &control);
+ return;
+ }
+ printf("adsp: qrtr: a datagram of %u bytes from node %u port %u to our port %u: nobody here listens there\n", header.size,
+ header.from_node, header.from_port, header.to_port);
+}
+
+/* ---- the link's thread ----------------------------------------------------------------------------- */
+
+static volatile bool link_wanted, link_runs;
+
+static void
+link_thread(__unused void *parameter, __unused wait_result_t result)
+{
+ struct smp2p words, last;
+ struct glink_packet *packet;
+ bool gave_up = false;
+
+ bzero(&last, sizeof(last));
+ dsp_words(&last);
+ while (link_wanted) {
+ sleep_us(1000);
+ doorbells();
+ if (dsp_words(&words) && memcmp(&words, &last, sizeof(words)) != 0) {
+ last = words;
+ dsp_listen(&words);
+ if ((dsp_state(&words) & DSP_FATAL) && !gave_up) {
+ gave_up = true;
+ printf("adsp: link: THE AUDIO DSP SAYS IT HAS GIVEN UP (\"slave-kernel\" = %#x)\n", dsp_state(&words));
+ show_crash_reason();
+ }
+ }
+ for (;;) {
+ lck_mtx_lock(&glink_lock);
+ glink_pump();
+ packet = glink.first;
+ if (packet != NULL) {
+ glink.first = packet->next;
+ glink.last = glink.first == NULL ? NULL : glink.last;
+ }
+ lck_mtx_unlock(&glink_lock);
+ if (packet == NULL) {
+ break;
+ }
+ qrtr_input(packet->data, packet->length);
+ kfree_data(packet, sizeof(*packet) + packet->length);
+ }
+ }
+ link_runs = false;
+ thread_terminate(current_thread());
+}
+
+static void
+link_stop(void)
+{
+ struct glink_packet *packet;
+
+ if (link_runs) {
+ link_wanted = false;
+ for (unsigned ms = 0; ms < 2000 && link_runs; ms++) {
+ sleep_ms(1);
+ }
+ printf("adsp: link: %s; GLINK said %u and heard %u messages, QRTR sent %u and received %u datagrams; its doorbells heard: "
+ "GLINK %u, SMP2P %u\n", link_runs ? "THE THREAD DOES NOT END" : "down", glink.said, glink.heard, qrtr_out, qrtr_in,
+ doorbell.glink, doorbell.smp2p);
+ }
+ lck_mtx_lock(&glink_lock);
+ glink.up = glink.open = false;
+ while ((packet = glink.first) != NULL) {
+ glink.first = packet->next;
+ kfree_data(packet, sizeof(*packet) + packet->length);
+ }
+ glink.last = NULL;
+ lck_mtx_unlock(&glink_lock);
+ bzero(qrtr_servers, sizeof(qrtr_servers));
+ qrtr_peer = 0;
+}
+
+/* ---- step 8: GLINK's queues, looked at -------------------------------------------------------------- */
+
+static bool
+step_queues(void)
+{
+ struct glink_message message;
+ uint32_t head, tail;
+
+ announce("find GLINK's descriptor and the two queues, and read what the audio DSP has put into its own");
+ if (!glink_find()) {
+ printf("adsp: %u: GLINK's items are not all there, or not of a shape this knows (descriptor %#x, ours %#x of %u bytes, "
+ "the DSP's %#x of %u)\n", step, glink.descriptor, glink.ours, glink.ours_size, glink.theirs, glink.theirs_size);
+ return false;
+ }
+ head = smem_word(glink.descriptor + GLINK_THEIR_HEAD);
+ tail = smem_word(glink.descriptor + GLINK_THEIR_TAIL);
+ printf("adsp: %u: our queue: %u bytes at %#lx, written to %#x, read by the DSP to %#x; the DSP's: %u bytes at %#lx, written "
+ "to %#x, read by us to %#x\n", step, glink.ours_size, SMEM_PHYS + glink.ours, smem_word(glink.descriptor + GLINK_OUR_HEAD),
+ smem_word(glink.descriptor + GLINK_OUR_TAIL), glink.theirs_size, SMEM_PHYS + glink.theirs, head, tail);
+ for (uint32_t at = 0, shown = 0; at + sizeof(message) <= head && head < glink.theirs_size && tail == 0 && shown < 8; shown++) {
+ char name[33] = "";
+ uint32_t length = sizeof(message);
+
+ glink_peek(&message, at, sizeof(message));
+ if (message.command == GLINK_OPEN) {
+ glink_peek(name, at + 8, message.param2 < 32 ? message.param2 : 32);
+ length += message.param2;
+ }
+ printf("adsp: %u: in the DSP's queue at %#x: %s %u %#x %s\n", step, at, glink_name(message.command), message.param1,
+ message.param2, name);
+ if (message.command != GLINK_VERSION && message.command != GLINK_OPEN) {
+ break;
+ }
+ at += (length + 7) & ~7u;
+ }
+ return true;
+}
+
+/* ---- step 9: GLINK, spoken -------------------------------------------------------------------------- */
+
+static bool
+step_link(void)
+{
+ thread_t thread;
+ uint64_t began;
+
+ if (!dsp_started || glink.descriptor == 0) {
+ return false;
+ }
+ announce("take up GLINK with the audio DSP: our version, then whatever its messages call for; a thread of ours listens "
+ "from here on, every millisecond");
+ snprintf(last_write, sizeof(last_write), "step %u took up GLINK", step);
+ lck_mtx_lock(&glink_lock);
+ glink.features = GLINK_INTENT_REUSE;
+ glink.agreed = glink.asked = glink.acknowledged = glink.open = glink.broken = false;
+ glink.their_channel = glink.our_intent_count = glink.said = glink.heard = 0;
+ bzero(glink.their_intents, sizeof(glink.their_intents));
+ bzero(glink.others, sizeof(glink.others));
+ glink.trace = 60;
+ glink.up = true;
+ qrtr_hellos = qrtr_in = qrtr_out = qrtr_lookups = 0;
+ doorbell.since = began = now_us();
+ glink_say(GLINK_VERSION, GLINK_VERSION_1, glink.features, NULL, 0);
+ lck_mtx_unlock(&glink_lock);
+ link_wanted = link_runs = true;
+ if (kernel_thread_start(link_thread, NULL, &thread) != KERN_SUCCESS) {
+ link_wanted = link_runs = false;
+ printf("adsp: %u: no thread\n", step);
+ return false;
+ }
+ thread_deallocate(thread);
+ for (unsigned ms = 0; ms < 5000 && !glink.open && !glink.broken; ms++) {
+ sleep_ms(1);
+ }
+ sleep_ms(50);
+ printf("adsp: %u: %llu ms after our version: versions %s (features %#x); the channel \"%s\" is %s; the DSP also opened "
+ "\"%s\" \"%s\" \"%s\" \"%s\"\n", step, (now_us() - began) / 1000, glink.agreed ? "agreed" : "NOT AGREED", glink.features,
+ GLINK_CHANNEL, glink.open ? "OPEN" : "NOT OPEN", glink.others[0], glink.others[1], glink.others[2], glink.others[3]);
+ return glink.open;
+}
+
+/* ---- step 10: QRTR ---------------------------------------------------------------------------------- */
+
+static bool
+step_qrtr(void)
+{
+ unsigned services = 0;
+
+ if (!glink.open) {
+ return false;
+ }
+ PE_parse_boot_argn("adsp_listen", &listen_seconds, sizeof(listen_seconds));
+ announce("listen to QRTR on the channel: the DSP's hello (answered from the link's thread), and the services it tells of");
+ for (unsigned ms = 0; ms < listen_seconds * 1000; ms++) {
+ sleep_ms(1);
+ }
+ for (unsigned i = 0; i < sizeof(qrtr_servers) / sizeof(qrtr_servers[0]); i++) {
+ services += qrtr_servers[i].there;
+ }
+ printf("adsp: %u: after %u s: %s; the DSP is node %u and has told of %u services; it looked for %u of ours; QRTR received %u "
+ "and sent %u datagrams\n", step, listen_seconds, qrtr_hellos ? "HELLO BOTH WAYS" : "NO HELLO FROM THE DSP", qrtr_peer,
+ services, qrtr_lookups, qrtr_in, qrtr_out);
+ return qrtr_hellos != 0;
+}
+
+/* ---- the steps, and their end ----------------------------------------------------------------------- */
+
+static bool (*const steps[STEPS])(void) = {
+ step_survey, step_firmware, step_mailbox, step_header, step_image, step_our_item, step_start, step_queues, step_link,
+ step_qrtr,
+};
+
+static bool keep; /* the DSP and the link stay up while the system runs (adsp_keep, adsp=1) */
+
+static void
+steps_over(uint32_t done)
+{
+ lck_mtx_lock(&dsp_lock);
+ if (dsp_started && done == last_step && keep) {
+ printf("adsp: THE AUDIO DSP STAYS ON while the system runs%s\n", link_runs ? ", and the link with it" : "");
+ } else if (dsp_started) {
+ dsp_shutdown("the steps are over");
+ }
+ lck_mtx_unlock(&dsp_lock);
+}
+
+/* When the run ends in order: a DSP that runs is shut down, for the next kernel to start again. */
+void
+redwood_adsp_stop(void)
+{
+ lck_mtx_lock(&dsp_lock);
+ if (dsp_started) {
+ unsigned services = 0;
+ uint32_t node = qrtr_peer;
+ bool down;
+
+ for (unsigned i = 0; i < sizeof(qrtr_servers) / sizeof(qrtr_servers[0]); i++) {
+ services += qrtr_servers[i].there;
+ }
+ down = dsp_shutdown("the run is about to end");
+ /* (said again at the end: a long run's log keeps its beginning and its end, and the middle may be gone) */
+ tell("adsp: the audio DSP is off: it had been up%s (node %u, %u services), it %s, and TrustZone %s\n", node ?
+ " and linked" : ", NOT LINKED", node, services, stopped_by_its_word ? "stopped when asked" :
+ "DID NOT SAY IT HAD STOPPED", down ? "has shut it down" : "DID NOT SHUT IT DOWN");
+ }
+ lck_mtx_unlock(&dsp_lock);
+}
+
+/* The same on a panic's way back to the loader: one processor, interrupts off, no lock, no sleep. */
+void
+redwood_adsp_quiesce(void)
+{
+ if (!dsp_started) {
+ return;
+ }
+ leaving = true;
+ new_from = ~0u;
+ dsp_shutdown("the kernel is on its way back to the loader");
+}
+
+static void
+run_steps(void)
+{
+ uint32_t done = 0;
+
+ for (step = 1; step <= last_step; step++) {
+ if (!steps[step - 1]()) {
+ printf("adsp: step %u: FAILED; the steps after it are not attempted\n", step);
+ break;
+ }
+ printf("adsp: step %u: done\n", step);
+ done = step;
+ }
+ step = done;
+ new_from = STEPS + 1;
+ steps_over(done);
+ printf("adsp: over after step %u; %s\n", done, last_write);
+}
+
+static void
+whole_thread(__unused void *parameter, __unused wait_result_t result)
+{
+ uint64_t began = now_us();
+ unsigned services = 0;
+
+ run_steps();
+ for (unsigned i = 0; i < sizeof(qrtr_servers) / sizeof(qrtr_servers[0]); i++) {
+ services += qrtr_servers[i].there;
+ }
+ if (step == STEPS && dsp_started) {
+ tell("adsp: THE AUDIO DSP IS UP AND LINKED, %llu ms after the kernel began on it: node %u, %u services; it stays on\n",
+ (now_us() - began) / 1000, qrtr_peer, services);
+ } else {
+ tell("adsp: THE AUDIO DSP DID NOT COME UP: step %u failed (%s); see the kernel's own log for the steps\n", step + 1,
+ last_write);
+ }
+ thread_terminate(current_thread());
+}
+
+/* Called while the system starts (bsd/dev/redwood_input.c), after the Wi-Fi's start has been set going. */
+void
+redwood_adsp_init(void)
+{
+ uint32_t on = 0;
+ thread_t thread;
+
+ PE_parse_boot_argn("adsp_step", &last_step, sizeof(last_step));
+ if (last_step == 0 && (!PE_parse_boot_argn("adsp", &on, sizeof(on)) || on != 1)) {
+ return;
+ }
+ if (pe_arm_get_soc_base_phys() != REDWOOD_SOC_BASE) {
+ printf("adsp: adsp=1 and adsp_step=N are for the phone's audio DSP, and this is not the phone\n");
+ return;
+ }
+ if (last_step == 0) {
+ /* all of it, on a thread of its own, and left up */
+ last_step = STEPS;
+ keep = true;
+ quiet = true;
+ pause_ms = 0;
+ listen_seconds = 1;
+ new_from = STEPS + 1;
+ if (kernel_thread_start(whole_thread, NULL, &thread) != KERN_SUCCESS) {
+ printf("adsp: no thread: the audio DSP is not brought up\n");
+ return;
+ }
+ thread_deallocate(thread);
+ tell("adsp: the audio DSP is being brought up beside the system's start (adsp=1)\n");
+ return;
+ }
+ if (last_step > STEPS) {
+ last_step = STEPS;
+ }
+ keep = option("adsp_keep");
+ new_from = last_step;
+ PE_parse_boot_argn("adsp_new", &new_from, sizeof(new_from));
+ PE_parse_boot_argn("adsp_pause", &pause_ms, sizeof(pause_ms));
+ printf("adsp: steps up to %u; those from %u on are new and wait %u ms after each announcement\n", last_step, new_from,
+ pause_ms);
+ run_steps();
+ if (option("adsp_return")) {
+ printf("adsp: back to the loader (adsp_return)\n");
+ loader_return();
+ }
+ printf("adsp: the system goes on starting\n");
+}
diff --git a/xnu/files/bsd/dev/redwood_input.c b/xnu/files/bsd/dev/redwood_input.c
index 8ac7a2a..e2155aa 100644
--- a/xnu/files/bsd/dev/redwood_input.c
+++ b/xnu/files/bsd/dev/redwood_input.c
@@ -175,4 +175,5 @@ redwood_input_init(void)
redwood_net_init();
redwood_adreno_init();
redwood_wifi_init();
+ redwood_adsp_init();
}
diff --git a/xnu/files/bsd/dev/redwood_wifi.c b/xnu/files/bsd/dev/redwood_wifi.c
index ffb13ed..4709e7c 100644
--- a/xnu/files/bsd/dev/redwood_wifi.c
+++ b/xnu/files/bsd/dev/redwood_wifi.c
@@ -182,6 +182,15 @@ extern kern_return_t thread_terminate(thread_t thread);
static LCK_GRP_DECLARE(wifi_lock_group, "redwood-wifi");
+/*
+ * What the audio DSP's start (redwood_adsp.c) uses as well, while this may be at work beside it:
+ * the mailbox to the always-on processor, one message at a time, and the allocation of items in
+ * the shared memory (whose hardware lock both take under the same number, so it does not keep
+ * the two apart).
+ */
+LCK_MTX_DECLARE(redwood_aop_mailbox_lock, &wifi_lock_group);
+LCK_MTX_DECLARE(redwood_smem_alloc_lock, &wifi_lock_group);
+
static uint32_t last_step, pause_ms = 2200, fine, watch_seconds = 3;
static uint32_t step; /* the one being performed */
static uint32_t new_from; /* the first of the steps that have not been seen to work: they wait */
@@ -1481,8 +1490,21 @@ doorbells(void)
* acknowledgement of ours. Whatever of it has been done already -- by a kernel before this one:
* the always-on processor is restarted only with the phone -- is found done and left.
*/
+static bool step_mailbox_open_locked(void);
+
static bool
step_mailbox_open(void)
+{
+ bool open;
+
+ lck_mtx_lock(&redwood_aop_mailbox_lock);
+ open = step_mailbox_open_locked();
+ lck_mtx_unlock(&redwood_aop_mailbox_lock);
+ return open;
+}
+
+static bool
+step_mailbox_open_locked(void)
{
uint32_t box, size;
@@ -1541,8 +1563,24 @@ step_mailbox_open(void)
}
/* A message to the always-on processor (Linux's qmp_send()). */
+static bool mailbox_send_locked(const char *text);
+
static bool
mailbox_send(const char *text)
+{
+ bool sent;
+
+ if (leaving) {
+ return mailbox_send_locked(text); /* (one processor left, and no lock taken: see redwood_wifi_quiesce()) */
+ }
+ lck_mtx_lock(&redwood_aop_mailbox_lock);
+ sent = mailbox_send_locked(text);
+ lck_mtx_unlock(&redwood_aop_mailbox_lock);
+ return sent;
+}
+
+static bool
+mailbox_send_locked(const char *text)
{
uint32_t words[QMP_MESSAGE / 4] = { 0 };
@@ -1937,8 +1975,21 @@ static struct block smem_lock = { "shared memory's lock", 0x01f43000, 0x1000 };
#define LOCK_US 1u
/* Our item made in the partition, as Linux's qcom_smem_alloc_private() makes one. False, and why said: not made. */
+static bool our_item_make_locked(uint32_t partition);
+
static bool
our_item_make(uint32_t partition)
+{
+ bool made;
+
+ lck_mtx_lock(&redwood_smem_alloc_lock);
+ made = our_item_make_locked(partition);
+ lck_mtx_unlock(&redwood_smem_alloc_lock);
+ return made;
+}
+
+static bool
+our_item_make_locked(uint32_t partition)
{
struct smem_partition header;
struct smem_entry entry;
diff --git a/xnu/files/bsd/sys/redwood_input.h b/xnu/files/bsd/sys/redwood_input.h
index ec5a96f..e6fb113 100644
--- a/xnu/files/bsd/sys/redwood_input.h
+++ b/xnu/files/bsd/sys/redwood_input.h
@@ -50,6 +50,7 @@ void redwood_adreno_quiesce(void); /* on the way back to the loader: the GP
void redwood_adreno_stop(void); /* before that, where the ending is no panic's: the GPU given back in order */
void redwood_adreno_minute(void); /* once a minute: a line about how warm the GPU is, if it is in use */
void redwood_wifi_init(void); /* boot-arg wifi=1: the phone's Wi-Fi; wifi_step=N: the first N steps of it */
+void redwood_adsp_init(void); /* boot-arg adsp=1: the phone's audio DSP started and linked; adsp_step=N: the first N steps */
void redwood_wifi_stop(void); /* where a run ends in order: the Wi-Fi DSP, if it was started, shut down */
int redwood_wifi_say(const char *format, ...) __printflike(1, 2); /* a line of the Wi-Fi steps' log */
int redwood_wifi_tell(const char *format, ...) __printflike(1, 2); /* ... one that is said even with wifi=1 */
diff --git a/xnu/files/osfmk/arm64/loader_return.c b/xnu/files/osfmk/arm64/loader_return.c
index e5081aa..ac2f743 100644
--- a/xnu/files/osfmk/arm64/loader_return.c
+++ b/xnu/files/osfmk/arm64/loader_return.c
@@ -49,6 +49,8 @@ extern void redwood_adreno_stop(void);
extern void redwood_adreno_minute(void);
extern void redwood_wifi_stop(void);
extern void redwood_wifi_quiesce(void);
+extern void redwood_adsp_stop(void);
+extern void redwood_adsp_quiesce(void);
void loader_checkpoint(uint64_t n);
/* Also referenced from the early exception vector in start.s. */
@@ -328,6 +330,7 @@ loader_return(void)
#endif
redwood_adreno_quiesce(); /* the GPU block, if it was started, has tables in memory that is about to be reused */
redwood_wifi_quiesce(); /* the Wi-Fi DSP, if it still runs (a panic), is not left to itself */
+ redwood_adsp_quiesce(); /* nor the audio DSP */
/*
* A panic's text is first collected in the panic buffer and only later
* flushed to the console; if the console itself is what failed, that
@@ -388,6 +391,7 @@ loader_return_timer(__unused thread_call_param_t p0, __unused thread_call_param_
if (!as_panic) {
redwood_adreno_stop(); /* a GPU kept for the system is given back while there is a thread to wait in */
redwood_wifi_stop(); /* and the Wi-Fi DSP, if it runs, shut down */
+ redwood_adsp_stop(); /* and the audio DSP */
}
if (PE_parse_boot_argn("return_cpu", &cpu, sizeof(cpu)) && cpu <= (uint32_t)ml_get_max_cpu_number() &&
cpu_xcall((int)cpu, loader_return_test, (void *)(uintptr_t)(as_panic ? 2 : 1)) == KERN_SUCCESS) {
@@ -432,6 +436,7 @@ loader_stop_poll(__unused thread_call_param_t p0, __unused thread_call_param_t p
}
redwood_adreno_stop();
redwood_wifi_stop();
+ redwood_adsp_stop();
loader_return();
}
clock_interval_to_deadline(1, NSEC_PER_SEC, &deadline);
diff --git a/xnu/redwood.patch b/xnu/redwood.patch
index 3d5edd6..dfbb965 100644
--- a/xnu/redwood.patch
+++ b/xnu/redwood.patch
@@ -46,10 +46,10 @@ index 222158a93918..68bb0f3e56bf 100755
######################################################################
# HISTORY
diff --git a/bsd/conf/files b/bsd/conf/files
-index 44274016a4cf..1750e10d3697 100644
+index 44274016a4cf..f061b6eb9c2c 100644
--- a/bsd/conf/files
+++ b/bsd/conf/files
-@@ -109,6 +109,24 @@ bsd/dev/munge.c optional config_requires_u32_munging
+@@ -109,6 +109,25 @@ bsd/dev/munge.c optional config_requires_u32_munging
bsd/dev/unix_startup.c standard
bsd/dev/monotonic.c optional monotonic
@@ -67,6 +67,7 @@ index 44274016a4cf..1750e10d3697 100644
+bsd/dev/redwood_virtio_sound.c standard
+bsd/dev/redwood_speaker.c standard
+bsd/dev/redwood_wifi.c standard
++bsd/dev/redwood_adsp.c standard
+bsd/dev/redwood_qmi.c standard
+bsd/dev/redwood_wlfw.c standard
+bsd/dev/redwood_lx.c standard
@@ -74,7 +75,7 @@ index 44274016a4cf..1750e10d3697 100644
bsd/dev/dev_perfmon.c optional config_perfmon
bsd/libkern/crc32.c standard
-@@ -155,6 +173,10 @@ bsd/miscfs/devfs/devfs_vfsops.c optional devfs
+@@ -155,6 +174,10 @@ bsd/miscfs/devfs/devfs_vfsops.c optional devfs
bsd/miscfs/mockfs/mockfs_fsnode.c optional mockfs
bsd/miscfs/mockfs/mockfs_vfsops.c optional mockfs
bsd/miscfs/mockfs/mockfs_vnops.c optional mockfs
@@ -85,7 +86,7 @@ index 44274016a4cf..1750e10d3697 100644
bsd/miscfs/routefs/routefs_ops.c optional routefs
-@@ -361,6 +383,7 @@ bsd/security/audit/audit_syscalls.c standard
+@@ -361,6 +384,7 @@ bsd/security/audit/audit_syscalls.c standard
bsd/security/audit/audit_worker.c optional config_audit
bsd/pthread/pthread_shims.c standard
@@ -93,7 +94,7 @@ index 44274016a4cf..1750e10d3697 100644
bsd/pthread/pthread_priority.c standard
bsd/pthread/pthread_workqueue.c standard
-@@ -383,6 +406,7 @@ bsd/kern/kern_debug.c standard
+@@ -383,6 +407,7 @@ bsd/kern/kern_debug.c standard
bsd/kern/kern_symfile.c standard
bsd/kern/kern_descrip.c standard
bsd/kern/kern_guarded.c standard