Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.
C
The modem's IPA: the hypervisor takes one SMMU match register per stream, and the handshake is Android's, not Linux's
shimoku74a19cf5ab2027
diff --git a/.gitignore b/.gitignoreindex 92cca1f..8fe3fbb 100644--- a/.gitignore+++ b/.gitignore@@ -79,3 +79,4 @@ __pycache__/!/xnu-deps/build/lzbitmap/libzbitmap.c!/xnu-deps/build/lzbitmap/libzbitmap.h/tools/android/f2fs-resize+/tools/macos-bin/platform-tools/diff --git a/README.md b/README.mdindex 91e5047..7466371 100644--- a/README.md+++ b/README.md@@ -2300,6 +2300,7 @@ SpringBoard спрашивает десять тысяч раз за все секаждые 15 секунд, того не стоит. Заглушить первый драйвер — работа того же рода, что `RedwoodBattery`; второй —это модем Apple на PCIe, а модем этого телефона сидит на самом кристалле и говорит по QMI с подсистемойQualcomm, так что за драйвером нет и того, к чему он ведёт.+Текущий план подключения MPSS и проверка аппаратных ресурсов: [docs/cellular.md](docs/cellular.md).**Остальные из того же списка «никогда не пробовали».** `cloudd` и `mobileassetd` не падают, и 42 ненайденныхобращения за сеанс с ними исчезают (`com.apple.cloudd` — 26, `com.apple.mobileassetd.v2` — 16). И всё-такиdiff --git a/docs/cellular.md b/docs/cellular.mdnew file mode 100644index 0000000..dee9610--- /dev/null+++ b/docs/cellular.md@@ -0,0 +1,185 @@+# Сотовая связь: состояние и план порта++Сейчас сотовая связь в HackiOS **не работает**, но модем уже запускается из XNU (см. «21 октября» ниже). Wi-Fi использует WPSS и свой GLINK/QRTR; модем MPSS — отдельный процессор с другим каналом. `redwood_qmi.c` кодирует сообщения QMI и теперь позволяет привязать клиент к заданному транспорту QRTR, но канал к MPSS ещё не реализован. У iOS также нет драйверов AppleBaseband и AppleIPAppender для этого аппаратного пути. Запуск `CommCenter` без них уже проверялся: он завершается аварийно (см. README, раздел «Телефония»).++Проверка `python3 tools/modem_firmware.py --dtb prebuilt/uefi/redwood.dtb --firmware-dir fw-modem --node /soc/qcom,mss@4080000` сопоставила 28 сегментов штатной прошивки с областью `mpss@8b800000` (PAS ID 4). `MODEM=1 SYSTEM=1 ... userland/build.sh` теперь может положить эти локальные файлы в системный APFS после проверки. Это только подготовка образа: наличие прошивки в `/fw` не запускает MPSS. Исходник `redwood_nas.c` содержит клиент QMI NAS для заданного транспорта QRTR: он ищет службу, запрашивает регистрацию и силу сигнала, разбирает ответы и сбрасывает состояние при исчезновении службы. Уровень `bars` остаётся нулевым без подтверждённой регистрации. MPSS-транспорта в XNU пока нет, поэтому этот клиент ещё не получает сообщения с телефона и не подключён к статус-бару iOS. Форматы NAS сверены с [описанием libqmi](https://github.com/linux-mobile-broadband/libqmi/blob/main/data/qmi-service-nas.json).++## Проверка на телефоне, 20 октября 2026++На redwood через `fastboot boot` временно запущены xnuloader и образ iOS 16.5. Системный APFS загружен в ОЗУ; разделы телефона не прошивались. `launchd` запустил `SpringBoard`, и на дисплее открылся интерфейс «Настройки». Это подтверждает загрузку iOS и работу экрана, но не работу модема. В журнале `mg_test` показывает `telephony = true` и `cellular-data = false`; запросы `com.apple.commcenter.xpc` и `com.apple.basebandd.xpc` остаются без службы. Значение `telephony = true` — ответ MobileGestalt о возможностях устройства, а не регистрация в сети. Тест выполнен с готовым ядром из `prebuilt`, поэтому изменения QMI в исходниках XNU в этой загрузке **не проверялись**. Дальнейший аппаратный тест требует сборки ядра с MPSS/QRTR и интеграции с iOS.++## Что установлено по redwood++`prebuilt/uefi/redwood.dtb` содержит зарезервированную память `mpss@8b800000`, `qcom,smp2p-modem`, `qcom,glink/modem/qcom,modem_qrtr`, `qcom,ipa@1e00000` и `qcom,rmnet-ipa`. Это разные части тракта: запуск прошивки модема, сигнализация, управляющий QMI-канал и передача IP-пакетов. Дерево Android [redwood](https://github.com/Redwood-AOSP/android_device_xiaomi_redwood) включает `full_base_telephony.mk`; его `init.qcom.rc` запускает `vendor.qcrild` и `vendor.qcrild2`. [Ветка araxys с ядром/DTB](https://github.com/Redwood-AOSP/android_device_xiaomi_redwood-kernel/tree/araxys) пригодна как аппаратный справочник. Бинарники Android RIL нельзя просто запустить в iOS: им нужны Android binder, HAL, службы и ABI.++Проверка по USB 20 октября 2026 года: fastboot определил продукт `redwood` и разблокированный загрузчик; Android сообщил `ro.boot.hardware.sku=redwood`, `ro.soc.model=SM7325`. Под Android подсистема `modem` и прошивка `yupik_ipa_fws` были `ONLINE`, `vendor.qcrild`, `vendor.qcrild2`, `vendor.qrtr-ns` — `running`; присутствовали `rmnet_ipa0` и `rmnet_data0`–`rmnet_data5`. Состояние SIM: `ABSENT,LOADED` для двух слотов. Временная загрузка `xnuloader` через `fastboot boot` и его `selftest` прошли (`selftest-ok`); это проверка загрузчика, а не сотовой связи в XNU.++Android `dmesg` уточняет последовательность: сначала поднимается `yupik_ipa_fws`, затем MPSS загружает `modem.mdt` и сегменты в `0x8b800000..0x9ae00000`, получает `Power/Clock ready`, после чего QRTR фиксирует готовность QMI. Штатные `modem.mdt`/`modem.b*` (около 85 МиБ) сохранены локально в игнорируемом Git каталоге `fw-modem/` и не входят в исходники. Их ELF-заголовок содержит 30 программных заголовков и 28 загружаемых сегментов; все файлы совпали по размеру, а сегменты лежат в зарезервированном окне. Загрузчик WPSS ограничен 16 сегментами; путь ADSP рассчитан на 48, но использует собственные ресурсы памяти, SMP2P и GLINK и не может запускать MPSS простым изменением имени прошивки.++## Границы общей реализации++Общими для устройств должны быть контракт транспорта QRTR, разбор QMI, состояние SIM/сети и интерфейс передачи IP-пакетов. Запуск прошивки MPSS, ресурсы памяти, прерывания, SMMU, IPA и прошивки остаются в профиле и драйвере конкретного SoC/устройства. По наличию `qcom,glink` нельзя объявлять поддержку всех Qualcomm или других телефонов.++Порядок доведения до работающей связи:++1. На подключённом Android собрать отчёт `python3 tools/cellular_probe.py --adb` и сверить DTB, версию прошивки и активные службы. Скрипт читает только открытые свойства и не собирает IMEI, IMSI и номера SIM.+2. Поднять MPSS в XNU с собственным GLINK/QRTR-каналом, отдельным от WPSS; проверить обнаружение QMI-служб, не отправляя команд регистрации или передачи данных.+3. Реализовать QMI DMS/UIM/NAS/WDS и состояние SIM/сети; отдельно включить путь IPA/rmnet для мобильных данных.+4. Подключить эти возможности к CoreTelephony/CommCenter через совместимый сервис и аппаратный драйвер, затем проверить SIM, регистрацию, входящие/исходящие вызовы, SMS и мобильные данные на телефоне. VoLTE/IMS — отдельная работа.++Пока пункт 2 не подтверждён на устройстве, не включать узел `baseband` и не переключать систему на cellular `CommCenter`: это создаёт аварийные циклы без радио.++## Команды до и после подключения++```sh+python3 tools/cellular_probe.py > modem-dtb.json+# После подключения телефона с разрешённой USB-отладкой:+python3 tools/cellular_probe.py --adb > modem-android.json+```++Для другого устройства передать его DTB через `--dtb PATH`; прошивку на телефон инструмент не записывает.++## 21 октября 2026: модем запущен из XNU++Ядро теперь собирается на этом Mac (см. `xnu/README.md`, «Сборка на macOS»), и драйвер+`bsd/dev/redwood_mpss.c` — копия пути аудио‑DSP (`redwood_adsp.c`) с числами модема — проверен+на телефоне через `xl run` с `modem_step=11 modem_return=1`:++- шаги 1–10 проходят: прошивка `modem.*` из `/fw` системного образа (85 МБ, 28 сегментов)+ записана в `0x8b800000` и прочитана обратно без расхождений; TrustZone принял подпись+ (образ 4) и область `0xf600000` байт; шины CX и MSS запрошены у RPMh на уровне 384; AOP+ получил `modem: on`; через 63 мс после старта модем сказал `slave-kernel = 0x6` (поднялся и+ взял питание сам); GLINK согласован, канал `IPCRTR` открыт в обе стороны; QRTR: hello в обе+ стороны, модем — узел 0;+- модем объявил только две службы: 43 (SSCTL, экземпляр 18) и 66 (реестр служб). DMS и NAS+ не появились за 40 с, поэтому шаг 11 (DMS «online», опрос NAS) не выполнен;+- по просьбе модем остановился за 43 мс (`0x10e`), TrustZone выключил его, AOP получил+ `modem: off`; телефон вернулся в загрузчик штатно.++Причина отсутствия DMS/NAS — у модема нет доступа к своим файловым системам: на Android их+отдаёт демон `rmtfs` (служба QMI 14 поверх QRTR, [linux-msm/rmtfs](https://github.com/linux-msm/rmtfs)).+В драйвер добавлена такая служба: четыре раздела (`modemst1`, `modemst2`, `fsc`, `fsg`)+читаются при старте шагов целиком в память ядра через `redwood_ufs_read_partition()`+(нужен `ufs=2` в аргументах ядра — блочная служба загрузчика), модем открывает их по путям+`/boot/modem_fs1` и т. д., получает буфер 2,5 МБ (`qcom,rmtfs_sharedmem@0`), в который+TrustZone пускает модем вызовом `SCM MP ASSIGN` (HLOS + MSS_MSA, чтение и запись), и читает+или пишет секторы через него. Записи остаются в памяти ядра: **разделы телефона не+записываются**. Служба объявляется модему в ответ на hello и на lookup (наша сторона+сервиса имён QRTR). Это ещё **не проверено на телефоне**: после предыдущего прогона телефон+исчез с USB (сторожевой таймер загрузчика), нужен ручной перезапуск в fastboot.++Запуск теста:++```sh+./xl enter && ./xl ramdisk userland/system.apfs # образ с MODEM=1 SYSTEM=1 (один раз после входа в загрузчик)+XL_CMDLINE="debug=0x8 rd=md0 hv_wdt_pet=1 bootcpu=7 cpus=8 cpufreq=1 sb_delay=600 ufs=2 modem_step=11 modem_return=1 modem_new=12 modem_nas_wait=60" XL_SECS=300 XL_FOLLOW=3 ./xl run+```++`modem=1` в обычной загрузке делает всё то же на своём потоке и оставляет модем включённым,+опрашивая NAS каждые две секунды; результат — в `sysctl kern.redwood.cellular_up`,+`cellular_registered`, `cellular_bars`, `cellular_dbm`, `cellular_radio`. Палочки в+статус‑баре iOS из этого ещё не получаются: нужна либо часть CommCenter, либо подмена+элемента статус‑бара в `libredwood-sb.dylib` по этим sysctl (следующий шаг).++## 23 октября 2026: буфер RMTFS и перезапуски телефона++Ночные прогоны (`tools/macos-bin`: adb из platform-tools r34 — сервер adb 37.0.1 на этом Mac+убивается через две секунды после запуска; `xl` возвращает телефон в Android после каждого+прогона) показали:++- буфер RMTFS должен лежать **ниже 4 ГБ**: адреса в запросах модема 32‑битные (с буфером+ `0x197508000` модем пишет «xpu lock failed» по усечённому адресу и сдаётся);+- память CDSP (`0x9c700000`) не годится: после её передачи TrustZone отказывается запускать+ модем (`-3145799`); память видеокодека (`0x8ae00000`) и `ramoops` (`0xb0000000`) —+ модем стартует, но телефон сбрасывается при первой нашей записи сектора; свободная память+ UEFI (`0xa8000000`, `modem_buffer=`) — то же, пока буфер отдан только HLOS и MSS;+- Android отдаёт буфер трём владельцам: HLOS, MSS_MSA (15) и NAV (0x2b) — в DTB телефона у+ `qcom,rmtfs_sharedmem@0` стоит `qcom,vm-nav-path`. С тремя владельцами запись в буфер после+ «замка» модема проходит без сброса (цикл 2: сектор записан, модему отказано — ядро вернулось+ штатно). Обратная передача (`{HLOS, MSS, NAV} → {HLOS}`) при остановке отказывает+ (`-2`, ответ `0x56`): видимо, пока держится замок модема; на перезагрузку это не влияет;+- опыты включаются ключами `modem_share_test=1` (пробная запись после передачи),+ `modem_rmtfs_rw_off=1` (отказ в чтении), `modem_rmtfs_rw_fake=1` (запись есть, ответ «нет»),+ `modem_rmtfs_delay=S` (пауза перед первым чтением), `modem_share_off=1`, `modem_no_nav=1`.++Паника `copyio.c:89 Assertion failed: pan != 0` в пользовательских процессах iOS встречалась в+двух прогонах с модемом; её причина не найдена.++## 23 октября 2026, вечер: модем читает файлы, выходит в ONLINE++С буфером в `0xa8000000`, отданным HLOS, MSS и NAV, модем читает свои файловые системы и за+~30 с объявляет все службы (88 объявлений: DMS 2, NAS 3, UIM 11, WDS 1, ...). Дальше он стоит в+режиме **OFFLINE** и на `DMS Set Operating Mode = online` отвечает «устройство не готово» (52),+на `low power` — «недопустимый переход» (60): он ждёт двух служб процессора приложений, которые+на Android дают демоны `pd-mapper` и `tqftpserv`:++- **реестр служб** (QMI 64, версия 257): модем спрашивает, где `tms/servreg` и другие; ответ —+ домены (`msm/modem/root_pd`, экземпляр 180) из `*.jsn` раздела прошивки;+- **TFTP поверх QRTR** (QMI 4096): модем пишет `/readwrite/server_check.txt`, читает+ `/readonly/firmware/image/modem_pr/so/665_0_0.mbn` кусками (опции `rsize`, `seek`, `wsize`,+ `blksize`), просит `/readwrite/ota_firewall/ruleset` (нет — не страшно).++Обе службы теперь в `redwood_mpss.c` (разделы «TFTP over QRTR» и «the service registry's+locator»): файлы берутся из `/fw/image` системного образа (`fw-modem/image` — копия+`/vendor/firmware_mnt/image` телефона, снятая `adb pull`; `MODEM=1 SYSTEM=1 userland/build.sh`+кладёт её в образ), записи модема хранятся в памяти ядра на время прогона. Сервер имён QRTR+объявляет наши службы модему при hello и отвечает на его lookup.++Результат (прогон 23:00): после обслуживания файлов модем **согласился перейти в ONLINE**, NAS+ответил «не зарегистрирован, CS/PS отключены», и через несколько секунд модем упал в задаче+`enl2_dl` (LTE L2, downlink) с нулевым адресом. Предположительно при выходе в сеть он обращается+к IPA, которого с нашей стороны нет; проверяется ответ «IPA недоступен» через SMP2P+(`modem_ipa=1`), дальше — либо запуск прошивки IPA и его тактов, либо иной обход.++## 24 октября 2026: IPA — прошивка, такт, SMMU++Падение `enl2_dl` повторяется в каждом прогоне через несколько секунд после ONLINE. Гипотеза —+модему нужен работающий IPA (пакетный ускоритель, через него идут данные LTE), и в+`redwood_mpss.c` появился его минимальный драйвер («the IPA» в файле), по образцу+`drivers/net/ipa` Linux (версия IPA 4.11, sc7280):++- **такт**: без голоса RPMh за BCM `IP0` (`ipa_clock()`) TrustZone отвечает на+ `AUTH_AND_RESET` образа 15 «Unhandled implementation specific error»; с голосом прошивка+ `yupik_ipa_fws` (перемещаемый ELF, кладётся от `0x8b700000`) стартует, микроконтроллер+ отвечает «booted» (`ipa_status()`);+- **память IPA**: `ipa_memory_init()` обнуляет локальную память (0x3000), пишет канарейки и базу+ контекстов обработки; раскладка — `ipa_data-v4.11.c`;+- **QMI 0x31**: наш хост (экземпляр 1) посылает модемному драйверу IPA `INIT_DRIVER` с раскладкой+ памяти (таблиц маршрутов 15, иначе модем падает на `num_indices >= rtng_tbl_max`), модем+ подписывается на индикацию; ответ `INIT_COMPLETE` ещё не дождались;+- **SMMU**: IPA ходит в память через SMMU процессора приложений потоками `0x480` (AP) и `0x482`+ (микроконтроллер). Поток без совпадения даёт отказ, обход (bypass) гипервизор не даёт.+ Контекст с нашими таблицами (тождественное отображение гигабайта DDR от `0x80000000` и imem+ `0x146a8000`) гипервизор принял (`sctlr` стал `0x127`, `s2cr` — наш), но **регистр+ совпадения с потоком `0x480` и маской `0x2` он сбросил в 0**. У Wi-Fi (`0x1c00`, маска 1,+ как в дереве устройств) прошло. В дереве устройств у потоков IPA маска 0 — гипервизор, видимо,+ сверяет пару (поток, маска) со списком хозяев. Прогон 21: по регистру совпадения на каждый+ поток, маска 0.++Телефон после каждого прогона возвращается в Android (`cycle.sh`); разделы не пишутся.++### 24 октября, ночь: что говорит сам Android++Android телефона рутован (KernelSU), и его `dmesg`, debugfs (`/d/ipa`, `/d/gsi`) и IPC-логи+(`/d/ipc_logging/ipa/log`, `gsi/log`) читаются по adb. По ним:++- SMMU: прогон 21 подтвердил догадку — **гипервизор принял по регистру совпадения на поток**+ (`0x480` и `0x482`, маска 0, оба в контекст 7), отказов IPA не было;+- модем **не отвечал на INIT_DRIVER** вовсе (13 с до падения `enl2_dl`). У Android ответ+ приходит через 11 мс, и запрос другой: `platform_type 3` (мы слали 1 — «data card»),+ `num_indices 7` (мы 14), область модема `0x27d8` размером `0x800` (мы `0x1f18/0x100c`,+ раскладка upstream v4.11 не совпадает с раскладкой `ipa_4_11_mem_part` из techpack/dataipa),+ контексты обработки `0xad0` размером `0xac0`, статистика квот `0x1d00/0x30`, дропов+ `0x27b0/0x20`; TLV 0x18 (`is_ssr_bootup`) при первой загрузке не посылается;+- после ответа модем шлёт `INIT_MODEM_DRIVER_CMPLT` (0x35), `CONFIG` (0x27),+ `INSTALL_FILTER_RULE` (0x23), `FILTER_INSTALLED_NOTIF` (0x24) — Android отвечает успехом;+ индикацию `INIT_COMPLETE` (0x22) шлёт сразу после ответа на INIT_DRIVER;+- микроконтроллер IPA загружает **модем** (его события «booted» приходят после ответа), нам его+ область (0x0..0x288) трогать не надо;+- прошивка IPA у Android кладётся с `0x8b710000` (reserved `ipa_fw@8b700000` 0x10000,+ `ipa_gsi@8b710000` 0xa000); окно SRAM — `ipa-base + 0x40000 + 0x10000 = 0x1e50000`, 0x3000+ байт (блок в драйвере был отображён на 4 КБ — исправлено на 0x4000).++Источник раскладки и обработчиков: `techpack/dataipa/drivers/platform/msm/ipa/ipa_v3`+(`ipa_utils.c: ipa_4_11_mem_part`, `ipa_qmi_service.c`), ядро+Atom-X-Devs/android_kernel_xiaomi_sm7325 (у MiCode ветка `redwood-s-oss`).diff --git a/ios165/fetch.sh b/ios165/fetch.shindex ef6f43d..ee15b25 100755--- a/ios165/fetch.sh+++ b/ios165/fetch.sh@@ -39,7 +39,7 @@ unzbm_all() { # unzbm_all DIR: expand every LZBITMAP file below DIRfind "$1" -type f -name '*:com.apple.decmpfs' | sed 's/:com\.apple\.decmpfs$//' | tr '\n' '\0' | xargs -0 -r -n 200 $UNZBMlocal leftleft=$(find "$1" -name '*:com.apple.*' | wc -l)- [ "$left" = 0 ] || { echo "fetch.sh: $left files in $1 are still compressed"; exit 1; }+ [ "$left" -eq 0 ] || { echo "fetch.sh: $left files in $1 are still compressed"; exit 1; }}# --- the root filesystem --------------------------------------------------------------------------@@ -82,7 +82,7 @@ if [ ! -f .done-cryptex ]; thenunzbm_all cryptex.cachemv cryptex.cache/System/Library/Caches/com.apple.dyld $CACHErm -rf cryptex.cache- [ "$(ls $CACHE | wc -l)" = 43 ] || { echo "fetch.sh: the dyld shared cache is incomplete"; exit 1; }+ [ "$(ls $CACHE | wc -l)" -eq 43 ] || { echo "fetch.sh: the dyld shared cache is incomplete"; exit 1; }touch .done-cryptex[ -n "$KEEP" ] || rm -f cryptex-os.dmgfidiff --git a/tools/cellular_probe.py b/tools/cellular_probe.pynew file mode 100755index 0000000..c6ccc76--- /dev/null+++ b/tools/cellular_probe.py@@ -0,0 +1,133 @@+#!/usr/bin/env python3+"""Read-only modem inventory from an FDT and, optionally, Android over ADB.++This is a discovery tool. It does not start MPSS or implement a cellular driver.+The output intentionally omits IMEI, IMSI, phone numbers and SIM identifiers.+"""++import argparse+import json+import pathlib+import struct+import subprocess++FDT_MAGIC = 0xD00DFEED+BEGIN_NODE, END_NODE, PROP, NOP, END = 1, 2, 3, 4, 9+KEYWORDS = ("modem", "mpss", "mss", "ipa", "qrtr", "glink", "smem", "smp2p", "remoteproc")+PROPERTY_NAMES = {"compatible", "status", "memory-region", "qcom,pas-id", "qcom,smem", "qcom,smem-id", "reg", "interrupts", "phandle", "linux,phandle"}+ANDROID_PROPERTIES = (+ "ro.product.device", "ro.boot.hardware.sku", "ro.board.platform", "ro.boot.hardware", "ro.boot.baseband",+ "gsm.version.baseband", "init.svc.vendor.qcrild", "init.svc.vendor.qcrild2",+ "init.svc.vendor.qrtr-ns", "init.svc.vendor.ssgqmigd", "gsm.sim.state",+)+++def fdt_nodes(data):+ if len(data) < 40:+ raise ValueError("DTB header is too short")+ magic, total, off_struct, off_strings, _, version, _, _, size_strings, size_struct = struct.unpack_from(">10I", data)+ if magic != FDT_MAGIC or version < 16 or total > len(data):+ raise ValueError("invalid or unsupported DTB")+ if off_struct + size_struct > total or off_strings + size_strings > total:+ raise ValueError("DTB sections exceed file size")+ strings = data[off_strings:off_strings + size_strings]+ pos, limit, stack, indexes, nodes = off_struct, off_struct + size_struct, [], [], []+ while pos + 4 <= limit:+ token = struct.unpack_from(">I", data, pos)[0]+ pos += 4+ if token == BEGIN_NODE:+ stop = data.find(b"\0", pos, limit)+ if stop < 0:+ raise ValueError("unterminated DTB node")+ name = data[pos:stop].decode("ascii", "replace")+ stack.append(name)+ nodes.append({"path": "/" + "/".join(x for x in stack if x), "properties": {}})+ indexes.append(len(nodes) - 1)+ pos = (stop + 4) & ~3+ elif token == END_NODE:+ if not stack:+ raise ValueError("unbalanced DTB nodes")+ stack.pop()+ indexes.pop()+ elif token == PROP:+ if pos + 8 > limit or not nodes or not stack:+ raise ValueError("invalid DTB property")+ length, nameoff = struct.unpack_from(">II", data, pos)+ pos += 8+ if pos + length > limit or nameoff >= len(strings):+ raise ValueError("DTB property exceeds section")+ name_end = strings.find(b"\0", nameoff)+ if name_end < 0:+ raise ValueError("unterminated DTB property name")+ name = strings[nameoff:name_end].decode("ascii", "replace")+ if name in PROPERTY_NAMES:+ raw = data[pos:pos + length]+ if name in ("compatible", "status"):+ value = [x.decode("ascii", "replace") for x in raw.rstrip(b"\0").split(b"\0")]+ else:+ value = raw.hex()+ nodes[indexes[-1]]["properties"][name] = value+ pos = (pos + length + 3) & ~3+ elif token == NOP:+ continue+ elif token == END:+ if stack:+ raise ValueError("unclosed DTB nodes")+ return nodes+ else:+ raise ValueError(f"invalid DTB token {token}")+ raise ValueError("DTB has no end token")+++def modem_nodes(data):+ nodes = fdt_nodes(data)+ return [node for node in nodes if any(+ word in (node["path"].rsplit("/", 1)[-1] + " " + " ".join(node["properties"].get("compatible", []))).lower()+ for word in KEYWORDS+ )]+++def adb_call(serial, *args):+ command = ["adb"] + (["-s", serial] if serial else []) + list(args)+ result = subprocess.run(command, capture_output=True, text=True, timeout=15, check=False)+ if result.returncode:+ raise RuntimeError(result.stderr.strip() or f"adb exited {result.returncode}")+ return result.stdout.strip()+++def android_inventory(serial):+ devices = [line.split()[0] for line in adb_call(None, "devices").splitlines()[1:]+ if len(line.split()) >= 2 and line.split()[1] == "device"]+ if serial is None:+ if len(devices) != 1:+ raise RuntimeError(f"expected one authorized Android device, found {len(devices)}; use --serial")+ serial = devices[0]+ elif serial not in devices:+ raise RuntimeError(f"Android device {serial!r} is not authorized or connected")+ properties = {key: adb_call(serial, "shell", "getprop", key) for key in ANDROID_PROPERTIES}+ subsystem_lines = adb_call(serial, "shell", "for d in /sys/bus/msm_subsys/devices/subsys*; do "+ "[ -r \"$d/name\" ] && [ -r \"$d/state\" ] && "+ "printf '%s:%s\\n' \"$(cat \"$d/name\")\" \"$(cat \"$d/state\")\"; done")+ subsystems = dict(line.split(":", 1) for line in subsystem_lines.splitlines() if ":" in line)+ interfaces = adb_call(serial, "shell", "ls", "/sys/class/net").split()+ return {"properties": properties, "subsystems": subsystems,+ "cellular_interfaces": [name for name in interfaces if name.startswith(("rmnet", "wwan", "ipa"))]}+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--dtb", type=pathlib.Path, default=pathlib.Path(__file__).resolve().parents[1] / "prebuilt/uefi/redwood.dtb")+ parser.add_argument("--adb", action="store_true", help="also query a connected Android phone without changing it")+ parser.add_argument("--serial", help="ADB serial when more than one phone is connected")+ args = parser.parse_args()+ try:+ report = {"dtb": str(args.dtb), "modem_related_nodes": modem_nodes(args.dtb.read_bytes())}+ if args.adb:+ report["android"] = android_inventory(args.serial)+ except (OSError, ValueError, RuntimeError, subprocess.TimeoutExpired) as error:+ parser.exit(1, f"cellular_probe: {error}\n")+ print(json.dumps(report, ensure_ascii=False, indent=2))+++if __name__ == "__main__":+ main()diff --git a/tools/macos-bin/README.md b/tools/macos-bin/README.mdnew file mode 100755index 0000000..78656e4--- /dev/null+++ b/tools/macos-bin/README.md@@ -0,0 +1,4 @@+What `xl` needs of Linux on a Mac: `flock`, GNU `timeout` and `stat` (MacPorts' gtimeout and gstat),+and Android's `fastboot` and `adb` (platform-tools, which `xl` finds here or in $PATH). Put the+platform-tools directory next to this file as `platform-tools/` (not in the repository).+r34 platform-tools (adb 34.0.5): the newer 37.0.1 adb server is killed within seconds on this Macdiff --git a/tools/macos-bin/flock b/tools/macos-bin/flocknew file mode 100755index 0000000..8ce9fff--- /dev/null+++ b/tools/macos-bin/flock@@ -0,0 +1,29 @@+#!/usr/bin/env python3+# flock(1) for macOS, as much of util-linux's as xl uses: flock [-n] [-w SECS] FILE COMMAND...,+# flock [-n] FD (the lock is on the open file description, which the shell keeps open), flock [-n] FILE -c CMD+import fcntl, os, subprocess, sys, time+args = sys.argv[1:]+nonblock = False; wait = None+while args and args[0].startswith('-'):+ a = args.pop(0)+ if a in ('-n', '--nonblock', '--nb'): nonblock = True+ elif a in ('-w', '--wait', '--timeout'): wait = float(args.pop(0))+ elif a in ('-x', '--exclusive', '-e'): pass+ elif a in ('-s', '--shared'): pass+ else: sys.exit('flock: unknown option ' + a)+if not args: sys.exit('flock: usage: flock [-n] [-w secs] FILE|FD [command...]')+target = args.pop(0)+if target.isdigit() and not args:+ fd = int(target)+else:+ fd = os.open(target, os.O_RDONLY | os.O_CREAT, 0o644)+deadline = None if wait is None else time.time() + wait+while True:+ try:+ fcntl.flock(fd, fcntl.LOCK_EX | (fcntl.LOCK_NB if (nonblock or deadline is not None) else 0))+ break+ except OSError:+ if nonblock or (deadline is not None and time.time() >= deadline): sys.exit(1)+ time.sleep(0.1)+if not args: sys.exit(0)+sys.exit(subprocess.call(args))diff --git a/tools/macos-bin/stat b/tools/macos-bin/statnew file mode 100755index 0000000..0424613--- /dev/null+++ b/tools/macos-bin/stat@@ -0,0 +1,2 @@+#!/bin/sh+exec /opt/local/bin/gstat "$@"diff --git a/tools/macos-bin/timeout b/tools/macos-bin/timeoutnew file mode 100755index 0000000..30d68d4--- /dev/null+++ b/tools/macos-bin/timeout@@ -0,0 +1,2 @@+#!/bin/sh+exec /opt/local/bin/gtimeout "$@"diff --git a/tools/modem_firmware.py b/tools/modem_firmware.pynew file mode 100755index 0000000..e93825a--- /dev/null+++ b/tools/modem_firmware.py@@ -0,0 +1,88 @@+#!/usr/bin/env python3+"""Validate a Qualcomm MDT firmware set against a device tree modem region.++This is an offline check. It does not authenticate, start, or contact the modem.+"""++import argparse+import json+import pathlib+import struct++from cellular_probe import fdt_nodes+++def cell(prop):+ raw = bytes.fromhex(prop)+ if len(raw) != 4:+ raise ValueError("expected one 32-bit device tree cell")+ return struct.unpack(">I", raw)[0]+++def inspect(dtb_path, firmware_dir, node_path):+ nodes = fdt_nodes(dtb_path.read_bytes())+ by_path = {node["path"]: node for node in nodes}+ if node_path not in by_path:+ raise ValueError(f"modem node {node_path} is absent")+ props = by_path[node_path]["properties"]+ if "memory-region" not in props or "qcom,pas-id" not in props:+ raise ValueError("modem node lacks memory-region or qcom,pas-id")+ memory_phandle = cell(props["memory-region"])+ matches = [node for node in nodes if any(+ cell(node["properties"][name]) == memory_phandle+ for name in ("phandle", "linux,phandle") if name in node["properties"])]+ if len(matches) != 1:+ raise ValueError(f"memory-region phandle {memory_phandle:#x} is not unique")+ region = matches[0]+ raw_reg = bytes.fromhex(region["properties"].get("reg", ""))+ if len(raw_reg) != 16:+ raise ValueError("expected a two-cell base and size for modem reserved memory")+ base, size = struct.unpack(">QQ", raw_reg)+ if size == 0:+ raise ValueError("empty modem reserved memory")++ header = (firmware_dir / "modem.mdt").read_bytes()+ if len(header) < 52 or header[:5] != b"\x7fELF\x01" or header[5] != 1:+ raise ValueError("modem.mdt is not little-endian ELF32")+ program_offset = struct.unpack_from("<I", header, 28)[0]+ entry_size, count = struct.unpack_from("<HH", header, 42)+ if entry_size != 32 or not 1 <= count <= 128 or program_offset + count * entry_size > len(header):+ raise ValueError("invalid program header table")+ segments = []+ total = 0+ for index in range(count):+ kind, _, _, address, file_size, memory_size, flags, _ = struct.unpack_from(+ "<8I", header, program_offset + index * entry_size)+ if kind != 1 or (flags >> 24) & 7 == 2 or memory_size == 0:+ continue+ if file_size > memory_size or address < base or address + memory_size > base + size:+ raise ValueError(f"segment {index} exceeds the modem reserved memory")+ name = f"modem.b{index:02d}"+ if file_size:+ actual = (firmware_dir / name).stat().st_size+ if actual != file_size:+ raise ValueError(f"{name}: {actual} bytes, expected {file_size}")+ total += actual+ segments.append({"index": index, "address": address, "file_bytes": file_size,+ "memory_bytes": memory_size})+ if not segments or min(item["address"] for item in segments) != base:+ raise ValueError("firmware has no segments starting at the modem memory base")+ return {"modem_node": node_path, "memory_node": region["path"], "memory_base": base,+ "memory_size": size, "pas_id": cell(props["qcom,pas-id"]), "program_headers": count,+ "loadable_segments": len(segments), "file_bytes": total, "segments": segments}+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--dtb", type=pathlib.Path, required=True)+ parser.add_argument("--firmware-dir", type=pathlib.Path, required=True)+ parser.add_argument("--node", required=True, help="modem remoteproc node in this device's DTB")+ args = parser.parse_args()+ try:+ print(json.dumps(inspect(args.dtb, args.firmware_dir, args.node), indent=2))+ except (OSError, ValueError) as error:+ parser.exit(1, f"modem_firmware: {error}\n")+++if __name__ == "__main__":+ main()diff --git a/tools/qmi-test/run-nas.sh b/tools/qmi-test/run-nas.shnew file mode 100755index 0000000..482b750--- /dev/null+++ b/tools/qmi-test/run-nas.sh@@ -0,0 +1,9 @@+#!/bin/sh+set -eu+root=$(CDPATH= cd -- "$(dirname -- "$0")/../.." && pwd)+out="$root/xnu-deps/build/qmi-test/nas-test"+mkdir -p "$(dirname "$out")"+${CC:-cc} -std=c11 -O1 -g -Wall -Wextra -Werror -fsanitize=address,undefined \+ -I"$root/xnu/files/bsd" -o "$out" \+ "$root/tools/qmi-test/test_nas.c" "$root/xnu/files/bsd/dev/redwood_nas.c"+"$out"diff --git a/tools/qmi-test/test_nas.c b/tools/qmi-test/test_nas.cnew file mode 100644index 0000000..7c3f24a--- /dev/null+++ b/tools/qmi-test/test_nas.c@@ -0,0 +1,92 @@+#include <assert.h>+#include <stdio.h>+#include <sys/redwood_nas.h>+#include <sys/redwood_qrtr.h>++/* Literal NAS frames using the public QMI TLV layout. */+static const unsigned char registered[] = {+ 2, 1, 0, 0x24, 0, 15, 0,+ 2, 4, 0, 0, 0, 0, 0,+ 1, 5, 0, 1, 2, 2, 2, 0+};+static const unsigned char disconnected[] = {+ 4, 1, 0, 0x24, 0, 8, 0,+ 1, 5, 0, 0, 3, 3, 0, 0+};+static const unsigned char lte_signal[] = {+ 2, 2, 0, 0x4f, 0, 16, 0,+ 2, 4, 0, 0, 0, 0, 0,+ 0x14, 6, 0, 0xba, 0xf6, 0xa1, 0xff, 0x64, 0+};+static const unsigned char failed_signal[] = {+ 2, 2, 0, 0x4f, 0, 16, 0,+ 2, 4, 0, 1, 0, 1, 0,+ 0x14, 6, 0, 0xba, 0xf6, 0xa1, 0xff, 0x64, 0+};++static redwood_qrtr_receive_t received;+static redwood_qrtr_server_t server;+static void *callback_context;+static int requests, updates;++static struct redwood_qrtr_port *open_port(redwood_qrtr_receive_t receive, redwood_qrtr_server_t on_server, void *context)+{+ received = receive; server = on_server; callback_context = context;+ return (struct redwood_qrtr_port *)(uintptr_t)1;+}+static int send_packet(struct redwood_qrtr_port *port, uint32_t node, uint32_t peer,+ const void *data, size_t length)+{+ const unsigned char *bytes = data;+ assert(port && node == 7 && peer == 9 && length == 7 && bytes[0] == 0);+ assert((requests == 0 && bytes[3] == 0x24) || (requests == 1 && bytes[3] == 0x4f));+ requests++;+ return 0;+}+static int lookup_port(struct redwood_qrtr_port *port, uint32_t service, uint32_t version, uint32_t instance)+{+ assert(port && service == 3 && version == 0 && instance == 0);+ server(callback_context, true, 3, 1, 0, 7, 9);+ return 0;+}+static void close_port(struct redwood_qrtr_port *port) { assert(port); }+static void changed(void *context, const struct redwood_nas_state *state)+{+ assert(context && state);+ updates++;+}+static const struct redwood_qrtr_ops transport = {+ .open = open_port, .send = send_packet, .lookup = lookup_port, .close = close_port+};++int main(void)+{+ struct redwood_nas_state state = {0};+ assert(redwood_nas_apply(&state, lte_signal, sizeof(lte_signal)) == 1);+ assert(state.has_signal && !state.registered && state.bars == 0);+ assert(redwood_nas_apply(&state, registered, sizeof(registered)) == 1);+ assert(state.registered && state.bars == 0);+ assert(redwood_nas_apply(&state, lte_signal, sizeof(lte_signal)) == 1);+ assert(state.radio_interface == 8 && state.dbm == -95 && state.bars == 4);+ assert(redwood_nas_apply(&state, failed_signal, sizeof(failed_signal)) == 0);+ assert(state.bars == 4);+ for (size_t n = 0; n < sizeof(lte_signal); n++) {+ assert(redwood_nas_apply(&state, lte_signal, n) == -1);+ assert(state.bars == 4);+ }+ assert(redwood_nas_apply(&state, disconnected, sizeof(disconnected)) == 1);+ assert(!state.registered && !state.has_signal && state.bars == 0);+ struct redwood_nas_client client;+ assert(redwood_nas_client_open(&client, &transport, changed, &client) == 0);+ assert(requests == 2);+ received(callback_context, 7, 9, registered, sizeof(registered));+ received(callback_context, 7, 9, lte_signal, sizeof(lte_signal));+ assert(client.state.registered && client.state.bars == 4 && updates == 2);+ received(callback_context, 8, 9, disconnected, sizeof(disconnected));+ assert(client.state.registered && updates == 2);+ server(callback_context, false, 3, 1, 0, 7, 9);+ assert(!client.state.registered && client.state.bars == 0 && updates == 3);+ redwood_nas_client_close(&client);+ puts("nas-test: registration gates bars; QRTR lookup, polling and disconnect verified");+ return 0;+}diff --git a/tools/qmi-test/test_qmi.c b/tools/qmi-test/test_qmi.cindex 1e432ff..9021950 100644--- a/tools/qmi-test/test_qmi.c+++ b/tools/qmi-test/test_qmi.c@@ -944,6 +944,9 @@ test_helpers(void)/* no router: no handle */redwood_qmi_set_transport(NULL);check(qmi_handle_init(&qmi, 8192, &handle_ops, handle_handlers) == -ENXIO && qmi.sock == NULL, "a handle is made with no router there");+ check(redwood_qmi_handle_init_transport(&qmi, 8192, &handle_ops, handle_handlers, &router_ops) == 0 &&+ qmi.sock != NULL, "an explicit transport cannot be used without the default router");+ qmi_handle_release(&qmi);redwood_qmi_set_transport(&router_ops);memset(&seen, 0, sizeof(seen));diff --git a/userland/build.sh b/userland/build.shindex 41570fe..6388c9d 100755--- a/userland/build.sh+++ b/userland/build.sh@@ -53,6 +53,8 @@# kernel's adsp_step=N, bsd/dev/redwood_adsp.c, which runs before the root is# mounted). Not beside GL=1: the two do not fit the 64 MB the loader takes with# the kernel. adsp=1 needs none of it: the system image has the firmware in /fw.+# MODEM=1 stage checked local MPSS firmware from ../fw-modem in system.apfs /fw.+# This does not start the modem; the XNU MPSS driver is still required.# system.apfs the system image, the root filesystem (the resident RAM disk, /dev/md0), an# APFS container: the part of iOS 16.5 chosen by ../ios165/select-system.py at# its real paths, the dyld shared cache, and src/stage0.c as /sbin/launchd (the@@ -336,9 +338,10 @@ if [ -d $IOS/rootfs/usr/lib ]; thencp $IOS/var.skel root/etc/var.skel # the directories init creates in /private/var# time zones: the template's links go in a circle until the time zone daemon has run (/usr/share/zoneinfo# points back into /var); point them at the data that ships with the system, and use this machine's zone- zone=$(timedatectl show -p Timezone --value 2>/dev/null); zone=${zone:-UTC}- sed -i -e 's|^l\t/usr/share/zoneinfo\t/private/var/db/timezone/zoneinfo$|l\t/usr/share/zoneinfo.default\t/private/var/db/timezone/zoneinfo|' \+ zone=$(timedatectl show -p Timezone --value 2>/dev/null || true); zone=${zone:-UTC}+ sed -i.bak -e 's|^l\t/usr/share/zoneinfo\t/private/var/db/timezone/zoneinfo$|l\t/usr/share/zoneinfo.default\t/private/var/db/timezone/zoneinfo|' \-e "s|^l\t/var/db/timezone/zoneinfo/[^\t]*\t/private/var/db/timezone/localtime\$|l\t/var/db/timezone/zoneinfo/$zone\t/private/var/db/timezone/localtime|" root/etc/var.skel+ rm -f root/etc/var.skel.bak# ... and the files of ours it puts there (var/): those under mobile/ belong to the user mobilecp -r var root/var(cd var && find . -type f | sed 's|^\./||' | sort) | while read -r f; do@@ -586,6 +589,21 @@ if [ -f $IOS/system.manifest ] && [ -f $CACHE/dyld_shared_cache_arm64 ] &&mkdir -p sysroot/fwcp ../fw-audio/adsp.mdt ../fw-audio/adsp.b0[2-9] ../fw-audio/adsp.b[1-9][0-9] sysroot/fw/fi+ # MPSS firmware is much too large for the small boot image. Keep it in the system APFS,+ # only when explicitly requested, after checking every loadable segment against this DTB.+ # This stages the files; a separate XNU MPSS driver is still required to start the modem.+ if [ -n "$MODEM" ]; then+ [ -f ../fw-modem/modem.mdt ] || { echo "build.sh: MODEM=1 needs ../fw-modem/modem.mdt" >&2; exit 1; }+ ../tools/modem_firmware.py --dtb ../prebuilt/uefi/redwood.dtb --firmware-dir ../fw-modem \+ --node /soc/qcom,mss@4080000 >/dev/null+ mkdir -p sysroot/fw+ cp ../fw-modem/modem.mdt ../fw-modem/modem.b[0-9][0-9] sysroot/fw/+ # ... and the firmware partition's image/ (its mcfg configurations and the registry's *.jsn, pulled from+ # the phone's /vendor/firmware_mnt/image into ../fw-modem/image), which the modem reads over TFTP (redwood_mpss.c)+ if [ -d ../fw-modem/image ]; then+ rm -rf sysroot/fw/image && cp -R ../fw-modem/image sysroot/fw/image+ fi+ fi# ... and the speaker amplifiers' configuration, the phone's /vendor/etc/aw882xx_acf.bin (bsd/dev/redwood_speaker.c,# for sound=3): Android's own register values, which are not in the repository eitherif [ -f ../fw-audio/aw882xx_acf.bin ]; thendiff --git a/userland/src/inject_sb.c b/userland/src/inject_sb.cindex c739bf6..6b31ec4 100644--- a/userland/src/inject_sb.c+++ b/userland/src/inject_sb.c@@ -1066,6 +1066,38 @@ redwood_springboard_init(void)snapshot_format ? "as JPEG" : "NOT changed", original_lock_long_press ? "checked" : "NOT checked",battery_lvl ? "from QG" : "NOT replaced", battery_st ? "from QG" : "NOT replaced",poster_switcher ? "not offered" : "NOT withheld");+ /*+ * Cellular, a survey first (the modem's bars come from the kernel: sysctl kern.redwood.cellular_*,+ * bsd/dev/redwood_mpss.c): where UIKit makes the status bar's cellular entry from the raw data,+ * and with which signature -- said in the log with the unslid address, for the hook to come.+ */+ {+ static const char *const classes[] = { "_UIStatusBarDataCellularEntry", "_UIStatusBarData", "_UIStatusBarDataEntry" };+ const char *selector = "_cellularEntryWithData:signalStrengthEnabled:serviceEnabled:dataNetworkEnabled:serviceString:"+ "serviceCrossfadeString:serviceBadgeString:serviceContentType:dataNetworkType:gsmSignalStrengthRaw:"+ "gsmSignalStrengthBars:callForwardingType:lowDataModeActive:";++ for (unsigned c = 0; c < sizeof(classes) / sizeof(classes[0]); c++) {+ Class cls = objc_getClass(classes[c]);+ Method plus = cls ? class_getClassMethod(cls, sel_registerName(selector)) : NULL;+ Method minus = cls ? class_getInstanceMethod(cls, sel_registerName(selector)) : NULL;++ if (plus) {+ say("redwood: cellular: +[%s %s] at %#lx, types \"%s\"\n", classes[c], selector,+ (uintptr_t)method_getImplementation(plus) - slide, method_getTypeEncoding(plus));+ }+ if (minus) {+ say("redwood: cellular: -[%s %s] at %#lx, types \"%s\"\n", classes[c], selector,+ (uintptr_t)method_getImplementation(minus) - slide, method_getTypeEncoding(minus));+ }+ if (cls && !plus && !minus) {+ say("redwood: cellular: %s has no such method\n", classes[c]);+ }+ }+ say("redwood: cellular: the kernel says up=%d registered=%d bars=%d dbm=%d radio=%d\n", sysctl_int("kern.redwood.cellular_up"),+ sysctl_int("kern.redwood.cellular_registered"), sysctl_int("kern.redwood.cellular_bars"),+ sysctl_int("kern.redwood.cellular_dbm"), sysctl_int("kern.redwood.cellular_radio"));+ }say("redwood: battery UI: shows%% %s, set-shows%% %s, bolt %s, chargePercent %s, setChargePercent %s, lockscreen ring %s\n",original_battery_view_shows_percentage ? "ok" : "NO",original_battery_view_set_shows_percentage ? "ok" : "NO",diff --git a/xl b/xlindex 11a79d5..1a3849e 100755--- a/xl+++ b/xl@@ -44,6 +44,8 @@# when the last upload was not from this copy. The record starts empty -- an upload by a copy without# this change leaves none -- and `run` says so rather than assuming.cd "$(dirname "$(readlink -f "$0")")" || exit 1+# On a Mac: flock, GNU timeout and stat, and Android's platform-tools, from tools/macos-bin (see its README)+[ "$(uname)" = Darwin ] && PATH=$PWD/tools/macos-bin:$PWD/tools/macos-bin/platform-tools:$PATH# the kernel and the loader built here, or else the ones that come with the repositoryK=xnu-src/xnu-8796.121.2/BUILD/obj/DEVELOPMENT_ARM64_BCM2837/mach.development.bcm2837[ -f $K ] || K=prebuilt/mach.development.bcm2837diff --git a/xnu-build.sh b/xnu-build.shindex 8f851bb..dc3c7c4 100755--- a/xnu-build.sh+++ b/xnu-build.sh@@ -6,11 +6,15 @@ cd "$XNU_SRC" || exit 1# does not see it change, so the linked kernel is thrown away when it has.LX=$XNUDEPS/build/ath11k/out/ath11k.oSTAMP=BUILD/obj/redwood-objects-if [ -f "$LX" ]; then now="$LX $(stat -c %Y "$LX")"; else LX= now=; fi+mtime() { if [ "$XNU_HOST_OS" = Darwin ]; then stat -f %m "$1"; else stat -c %Y "$1"; fi; }+if [ -f "$LX" ]; then now="$LX $(mtime "$LX")"; else LX= now=; fi+# the build tools of SETUP/ are Apple's: on macOS they build as they are, on Linux with a compatibility header+if [ "$XNU_HOST_OS" = Darwin ]; then HOSTCC=clang; else+HOSTCC="clang -include $XNUDEPS/hostinc/hostcompat.h -I$XNUDEPS/hostinc -Wno-unused-command-line-argument -Wno-error=implicit-function-declaration"; fiif [ -d BUILD/obj ] && [ "$(cat $STAMP 2>/dev/null)" != "$now" ]; thenrm -f BUILD/obj/*/mach.*.unstrippedecho "$now" > $STAMPfiexec make REDWOOD_OBJECTS="$LX" SDKROOT=iphoneos ARCH_CONFIGS=ARM64 MACHINE_CONFIGS=${MACHINE:-BCM2837} KERNEL_CONFIGS=DEVELOPMENT \ARCH_STRING_FOR_CURRENT_MACHINE_CONFIG=arm64 BUILD_WERROR=0 BUILD_LTO=0 DO_CTFMERGE=0 \- HOST_CC="clang -include $XNUDEPS/hostinc/hostcompat.h -I$XNUDEPS/hostinc -Wno-unused-command-line-argument -Wno-error=implicit-function-declaration" HOST_FLEX=flex HOST_BISON=bison HOST_GM4=m4 CC=xnu-clang CXX=xnu-clang++ LOGCOLORS=n "$@"+ HOST_CC="$HOSTCC" HOST_FLEX=flex HOST_BISON=bison HOST_GM4=m4 CC=xnu-clang CXX=xnu-clang++ LOGCOLORS=n "$@"diff --git a/xnu-deps/host/bin/xcrun b/xnu-deps/host/bin/xcrunindex c74c918..e29b378 100755--- a/xnu-deps/host/bin/xcrun+++ b/xnu-deps/host/bin/xcrun@@ -1,6 +1,9 @@#!/bin/bash# Minimal xcrun replacement for building XNU on LinuxHB="$(cd "$(dirname "$0")" && pwd)"+# On macOS the host side (the SDK and tools the build tools of SETUP/ are made with) is the real+# xcrun's business; only the kernel's own SDK is ours.+case " $* " in *" -sdk macosx"*) [ "$(uname)" = Darwin ] && exec /usr/bin/xcrun "$@" ;; esacsdk=""; find=""; show=""while [ $# -gt 0 ]; do case "$1" in-sdk) sdk="$2"; shift 2;; -verbose) shift;;@@ -18,6 +21,7 @@ case "$show" inesacif [ -n "$find" ]; thenfor d in "$HB" /usr/bin; do [ -x "$d/$find" ] && { echo "$d/$find"; exit 0; }; done+ [ "$(uname)" = Darwin ] && exec /usr/bin/xcrun -find "$find"exit 1fi[ $# -gt 0 ] && exec "$@"diff --git a/xnu-deps/hostinc/xnu_clang_compat.h b/xnu-deps/hostinc/xnu_clang_compat.hindex dd781d3..c95cd53 100644--- a/xnu-deps/hostinc/xnu_clang_compat.h+++ b/xnu-deps/hostinc/xnu_clang_compat.h@@ -16,6 +16,8 @@#else#define __builtin_xnu_types_compatible(a, b) __builtin_types_compatible_p(a, b)#endif+#else+#define XNU_NO_TYPE_BUILTINS 0 /* Apple's clang: the real ones */#endif#endif#endifdiff --git a/xnu-deps/setup.sh b/xnu-deps/setup.shindex 005dd06..ebf44f7 100755--- a/xnu-deps/setup.sh+++ b/xnu-deps/setup.sh@@ -26,7 +26,7 @@ set -e -o pipefailHERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)source "$HERE/../xnu-env.sh"cd "$XNUDEPS"-JOBS=${JOBS:-$(nproc)}+JOBS=${JOBS:-$(nproc 2>/dev/null || sysctl -n hw.ncpu)}CCTOOLS=$THEOS/toolchain/linux/iphone/binIOS_SDK=$THEOS/sdks/iPhoneOS16.5.sdkMACOS_SDK_URL=https://github.com/joseluisq/macosx-sdks/releases/download/13.3/MacOSX13.3.sdk.tar.xz@@ -58,9 +58,46 @@ update() {if cmp -s "$1.new" "$1"; then rm "$1.new"; else mv "$1.new" "$1"; fi}+# macOS: no theos, but Xcode has the same cctools and the SDK can be had from github.com/theos/sdks.+# theos-mac/ is laid out like theos so that the rest of this script does not care.+theos_mac() {+ local tool+ mkdir -p "$THEOS/toolchain/linux/iphone/bin" "$THEOS/sdks"+ for tool in ld nm otool strip lipo libtool nmedit ctf_insert codesign_allocate install_name_tool; do+ link "$(/usr/bin/xcrun -f $tool)" "$THEOS/toolchain/linux/iphone/bin/$tool"+ done+ if [ ! -d "$THEOS/sdks-git/iPhoneOS16.5.sdk/usr/include" ]; then+ log "fetching iPhoneOS16.5.sdk from github.com/theos/sdks (about 220 MB)"+ rm -rf "$THEOS/sdks-git"+ git clone -q --depth 1 --filter=blob:none --sparse https://github.com/theos/sdks.git "$THEOS/sdks-git" &&+ git -C "$THEOS/sdks-git" sparse-checkout set iPhoneOS16.5.sdk || die "cannot fetch the SDK"+ fi+ link ../sdks-git/iPhoneOS16.5.sdk "$THEOS/sdks/iPhoneOS16.5.sdk"+}++# linktree FROM TO: TO becomes a tree of links to the files of FROM (cp -rs, which BSD cp lacks)+linktree() {+ if [ "$XNU_HOST_OS" = Darwin ]; then+ python3 - "$1" "$2" <<-'PY'+ import os, sys+ src, dst = sys.argv[1], sys.argv[2]+ for root, dirs, files in os.walk(src):+ rel = os.path.relpath(root, src)+ out = dst if rel == '.' else os.path.join(dst, rel)+ os.makedirs(out, exist_ok=True)+ for name in files:+ os.symlink(os.path.join(root, name), os.path.join(out, name))+ PY+ else+ cp -rs "$1" "$2"+ fi+}+stage1() {local missing= tool- for tool in git curl tar xz make cc clang clang++ flex bison m4 perl python3 autoreconf automake; do+ local tools="git curl tar xz make cc clang clang++ flex bison m4 perl python3 autoreconf automake"+ [ "$XNU_HOST_OS" = Darwin ] && { theos_mac; tools="git curl tar make cc clang clang++ flex bison m4 perl python3"; }+ for tool in $tools; docommand -v $tool >/dev/null || missing="$missing $tool"done[ -z "$missing" ] || die "not installed:$missing"@@ -84,13 +121,21 @@ stage1() {if [ ! -d MacOSX13.3.sdk/System/Library/Frameworks/Kernel.framework/Headers/DriverKit ]; thenlog "downloading the macOS 13.3 SDK for its DriverKit headers"curl -L --fail --retry 3 -s -o macos-sdk.tar.xz "$MACOS_SDK_URL" || die "download failed: $MACOS_SDK_URL"- tar -xJf macos-sdk.tar.xz --wildcards '*MacOSX13.3.sdk/System/Library/Frameworks/Kernel.framework*' ||+ local wild=--wildcards; [ "$XNU_HOST_OS" = Darwin ] && wild= # bsdtar: patterns as they are+ tar -xJf macos-sdk.tar.xz $wild '*MacOSX13.3.sdk/System/Library/Frameworks/Kernel.framework*' ||die "cannot unpack the macOS SDK"rm -f macos-sdk.tar.xzfi# --- host tools ----------------------------------------------------------------------------mkdir -p host/bin+ if [ "$XNU_HOST_OS" = Darwin ]; then+ # macOS has it all: mig (Xcode's; it takes $MIGCC and $MIGCOM from the kernel build), unifdef, csh+ link "$(/usr/bin/xcrun -f mig)" host/bin/mig+ link "$(/usr/bin/xcrun -f migcom)" host/bin/migcom+ link /usr/bin/unifdef host/bin/unifdef+ link /bin/tcsh host/bin/tcsh+ fi# mig: Apple's own does not build on glibc; this is a port that doesif [ ! -x host/bin/migcom ] || [ ! -x host/bin/mig ]; thenlog "building mig"@@ -120,7 +165,12 @@ stage1() {for tool in nm otool strip lipo libtool nmedit ctf_insert codesign_allocate install_name_tool; dolink "$CCTOOLS/$tool" host/bin/$tooldone- link "$CCTOOLS/ld" host/bin/ld64+ if [ "$XNU_HOST_OS" = Darwin ]; then+ # (not a link: Xcode's ld looks itself up by the name it was called by, and knows no "ld64")+ printf '#!/bin/sh\nexec "%s" "$@"\n' "$(/usr/bin/xcrun -f ld)" > host/bin/ld64.new && chmod +x host/bin/ld64.new && mv host/bin/ld64.new host/bin/ld64+ else+ link "$CCTOOLS/ld" host/bin/ld64+ filink xnu-clang host/bin/xnu-clang++# --- the SDK -------------------------------------------------------------------------------@@ -129,7 +179,7 @@ stage1() {if [ ! -d "$XNU_SDK_PATH/usr/include" ]; thenlog "making the SDK: links to $IOS_SDK"rm -rf "$XNU_SDK_PATH"- cp -rs "$(cd "$IOS_SDK" && pwd)" "$XNU_SDK_PATH"+ linktree "$(cd "$IOS_SDK" && pwd -P)" "$XNU_SDK_PATH"filocal sdk_local=$XNU_SDK_PATH/usr/localmkdir -p $sdk_local/include/kernel/os $sdk_local/include/os $sdk_local/lib/kernel $sdk_local/libexecdiff --git a/xnu-env.sh b/xnu-env.shindex f2c6b83..4a2179d 100644--- a/xnu-env.sh+++ b/xnu-env.sh@@ -3,7 +3,14 @@# iOS SDK; XNU_SRC is the kernel tree (xnu/apply.sh creates it).DARWIN_REDWOOD=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)export DARWIN_REDWOOD-export THEOS=${THEOS:-$HOME/theos}+export XNU_HOST_OS=$(uname)+# On macOS there is no theos: xnu-deps/setup.sh lays out xnu-deps/theos-mac like one (links to the+# cctools of Xcode and to the iPhoneOS 16.5 SDK of github.com/theos/sdks).+if [ "$XNU_HOST_OS" = Darwin ]; then+ export THEOS=${THEOS:-$DARWIN_REDWOOD/xnu-deps/theos-mac}+else+ export THEOS=${THEOS:-$HOME/theos}+fiexport XNU_SRC=${XNU_SRC:-$DARWIN_REDWOOD/xnu-src/xnu-8796.121.2}export XNUDEPS=$DARWIN_REDWOOD/xnu-depsexport PATH=$XNUDEPS/host/bin:$PATHdiff --git a/xnu/README.md b/xnu/README.mdindex e3caa7b..da37ad0 100644--- a/xnu/README.md+++ b/xnu/README.md@@ -6,7 +6,7 @@- `files/` — новые файлы по их путям в дереве XNU (37 файлов);- `apply.sh` — берёт исходники у Apple и накладывает всё это; `export.sh` — обратное действие.-Собирается на Linux стоковым clang, без macOS и Xcode.+Собирается на Linux стоковым clang, без macOS и Xcode, — и на macOS с Xcode (ниже, «Сборка на macOS»).## Сборка@@ -41,6 +41,16 @@ xnu-deps/setup.sh # ~30 с: исходники зависимостей,`xnu-deps/gen-iig-stubs.py xnu-deps/iig-impl/undefined-all.txt xnu-deps/iig-impl qemu/kernel.syms`, затем `rm -rf BUILD`.- Для `qemu/sym.py` после пересборки: `nm -n …/mach.development.bcm2837.unstripped > qemu/kernel.syms`.+## Сборка на macOS++Те же три команды. theos не нужен: `xnu-deps/setup.sh` на Darwin сам складывает `xnu-deps/theos-mac`+(ссылки на cctools из Xcode, iPhoneOS16.5.sdk из github.com/theos/sdks частичным клоном, ~220 МБ)+и берёт mig, unifdef и csh системные. Нужны Xcode (проверено с 26.5, Apple clang 21) и MacPorts с+coreutils (`gstat`, `gtimeout` — для `xl`). `host/bin/ld64` там — обёртка, а не ссылка: ld из Xcode+ищет себя по имени, под которым вызван. Ядро, собранное так, загружается на телефоне и возвращается в+загрузчик как и собранное на Linux. `xl` на Mac берёт `flock`, `timeout`, `stat` и platform-tools из+`tools/macos-bin` (см. README там).+## Работа с деревомПравки делаются прямо в `xnu-src/xnu-8796.121.2` (обычное git-дерево Apple на ветке `redwood`). Перед коммитом@@ -79,6 +89,8 @@ xnu-deps/setup.sh # ~30 с: исходники зависимостей,| `bsd/dev/redwood_adreno.c`, `iokit/Kernel/RedwoodGPU.cpp` | GPU телефона (Adreno 642L): пошаговое включение (`adreno=N`: питание, GMU, zap-шейдер, микрокод), буферы и очередь команд; клиент IOKit с подмножеством msm DRM для Mesa || `bsd/dev/redwood_wifi.c`, `bsd/sys/redwood_qrtr.h` | Wi-Fi телефона (WCN6750), начало пути (`wifi=N`): общая память процессоров Qualcomm (SMEM: чтение и выделение под её замком), сигнальные биты между нами и процессором Wi-Fi (SMP2P, обе стороны), почтовый ящик процессора питания, загрузка прошивки процессора Wi-Fi, его запуск через TrustZone и остановка по просьбе, GLINK (очереди в общей памяти, канал IPCRTR) и QRTR поверх него; `redwood_qrtr.h` — порт датаграмм для остального ядра (открыть, послать, найти службу, закрыть) || `bsd/dev/redwood_qmi.c`, `bsd/sys/redwood_qmi.h` | QMI — протокол сообщений служб на процессорах Qualcomm: кодирование по таблицам, запросы с ответами, оповещения; интерфейс как у Linux (`qmi_*`), чтобы `qmi.c` из ath11k собирался без правок, реализация своя; проверяется на компьютере (`tools/qmi-test/run.sh`) |+| `bsd/dev/redwood_nas.c`, `bsd/sys/redwood_nas.h` | клиент QMI NAS поверх заданного QRTR: регистрация в сети и уровень сигнала; не показывает сигнал без регистрации. Проверка на компьютере: `tools/qmi-test/run-nas.sh`. Для работы на телефоне требуются отдельный транспорт MPSS и подключение к CoreTelephony |+| `bsd/dev/redwood_mpss.c` | модем (MPSS): путь аудио-DSP с числами модема (`modem=N`, `modem_step=N`): прошивка из `/fw`, TrustZone (образ 4), шины CX/MSS, SMP2P, GLINK, QRTR с портами для QMI; наша служба RMTFS (QMI 14: файловые системы модема из разделов UFS, в памяти, без записи на флеш) и сервис имён QRTR; DMS «online» и клиент NAS → `sysctl kern.redwood.cellular_*`. Проверено: модем стартует и отвечает; RMTFS ещё не проверен (docs/cellular.md) || `bsd/dev/redwood_wlfw.c` | временная замена `qmi.c` из ath11k: разговор со службой прошивки Wi-Fi вручную, в его порядке и его сообщениями (шаг 15 `wifi=N`); описания сообщений берутся из исходников ath11k при сборке (`tools/mk-wlfw.sh`) и в git не попадают || `bsd/dev/redwood_audio.c`, `redwood_virtio_sound.c`, `redwood_speaker.c` | звук: `/dev/redwood-audio` (куда наше устройство вывода в mediaserverd пишет то, что играет iOS), запись его в память, virtio-sound для QEMU; первый взгляд на усилители динамиков телефона || `bsd/pthread/pthread_kext_static.c` | pthread kext, слинкованный в ядро |diff --git a/xnu/export.sh b/xnu/export.shindex 0940781..8d3868c 100755--- a/xnu/export.sh+++ b/xnu/export.sh@@ -26,7 +26,7 @@ mkdir "$HERE/files.new"while IFS= read -r f; do[ -f "$f" ] || [ -L "$f" ] || continue # added in a commit, deleted sincemkdir -p "$HERE/files.new/$(dirname "$f")"- cp -P --preserve=mode "$f" "$HERE/files.new/$f"+ if [ "$(uname)" = Darwin ]; then cp -pP "$f" "$HERE/files.new/$f"; else cp -P --preserve=mode "$f" "$HERE/files.new/$f"; fidone < "$HERE/files.list"rm -rf "$HERE/files" "$HERE/files.list"mv "$HERE/files.new" "$HERE/files"diff --git a/xnu/files/bsd/dev/redwood_input.c b/xnu/files/bsd/dev/redwood_input.cindex 8076cf0..e5ec5cb 100644--- a/xnu/files/bsd/dev/redwood_input.c+++ b/xnu/files/bsd/dev/redwood_input.c@@ -224,4 +224,5 @@ redwood_input_init(void)redwood_adreno_init();redwood_wifi_init();redwood_adsp_init();+ redwood_mpss_init();}diff --git a/xnu/files/bsd/dev/redwood_mpss.c b/xnu/files/bsd/dev/redwood_mpss.cnew file mode 100644index 0000000..086900f--- /dev/null+++ b/xnu/files/bsd/dev/redwood_mpss.c@@ -0,0 +1,5438 @@+/*+ * The modem of the POCO X5 Pro (redwood), started, and asked about the network: step 1 of+ * cellular. Every address below belongs to this phone, and nothing here is touched unless a+ * boot-arg asks for it.+ *+ * The modem ("MPSS", Android's "modem") is a Hexagon of its own inside the SoC, exactly as the+ * Wi-Fi's and the audio's are (redwood_wifi.c, redwood_adsp.c): it stays in reset until somebody+ * loads its signed image -- modem.mdt and modem.b00 to modem.b29 from /vendor/firmware_mnt/image+ * of its Android, 85 MB, in the system image under /fw (MODEM=1 SYSTEM=1 userland/build.sh, from+ * fw-modem/) -- into the 246 MB reserved for it and asks TrustZone to authenticate and start it.+ * From there everything is conversation through the shared memory: SMP2P for "up", "given up",+ * "stopped"; GLINK for channels; QRTR on the channel "IPCRTR" for the modem's services, which+ * speak QMI (bsd/dev/redwood_qmi.c knows the protocol; here only two of its messages are made by+ * hand): the device management service is asked to put the modem online, and the network access+ * service what network it is on and how strong the signal is (bsd/dev/redwood_nas.c). What this+ * learns goes to sysctl kern.redwood.cellular_* for the rest of the system; calls, SMS and data+ * are for the steps after this one.+ *+ * The way is the modem's, which is the Wi-Fi DSP's: redwood_adsp.c's comments say why each+ * thing is done so; this file is that one with the modem's numbers (dt: qcom,mss@4080000,+ * qcom,smp2p-modem, qcom,glink/modem, reserved-memory mpss@8b800000):+ * - processor 1 in the shared memory; its SMP2P items 435 (its) and 428 (ours); its last words+ * in item 421; GLINK's three items in our partition with it+ * - its doorbells are IPCC client 2 (MPSS): signal 0 GLINK, signal 2 SMP2P+ * - TrustZone knows it as image 4; its memory is 0x8b800000, 246 MB, of which the image takes+ * the first 85 (TrustZone is told of all of it: the modem uses the rest)+ * - its rails are CX and MSS (cx.lvl and mss.lvl), asked for at level 384 while it boots+ * - the always-on processor is told of it as "modem"+ * - the channel is IPCRTR only; the modem's others (DS, DATA*, glink_ssr) are left unanswered+ * - the ports of QRTR are the Wi-Fi driver's (sys/redwood_qrtr.h): what the NAS client needs+ * The modem and the Wi-Fi DSP may run at the same time: the mailbox to the always-on+ * processor and the lock for allocating in the shared memory are shared, each behind a lock.+ *+ * Boot-arg modem_step=N performs steps 1 to N and stops; modem=1 is all of them, on a thread+ * beside the system's start, and the modem stays on, its network state asked every two seconds.+ *+ * 1 reads the shared memory: our partition with the modem, its items, whether it has run+ * 2 reads the firmware on the root (or in the small image): is it whole, does it fit+ * 3 writes the always-on processor's mailbox open, unless it is+ * 4 asks TrustZone to take the image's header (image 4) and where the image will be; a+ * modem that a kernel before this one left running is asked to stop and shut down first+ * 5 writes the image into the modem's memory, and reads it back+ * 6 writes our SMP2P item (428) into the shared memory, and our two marks of GLINK to zero+ * 7 asks RPMh for the modem's rails, writes "modem: on" to the always-on processor, asks+ * TrustZone to authenticate and start the modem, and watches for its first words+ * 8 reads GLINK's queues+ * 9 writes our side of GLINK: the versions, the channel "IPCRTR" opened both ways, our intents;+ * from here a thread of ours listens to the modem+ * 10 talks QRTR on that channel: hello both ways, and the services the modem tells of+ * 11 talks QMI: DMS "operating mode online" (modem_online=0: not), then NAS "serving system"+ * and "signal info" every second for modem_nas_wait seconds (default 30), or until+ * the modem says it is registered and how strong the signal is+ *+ * What is started is stopped: when the steps are over (unless modem_keep=1, and then when the run+ * ends), and on a panic's way back to the loader. More boot-args: modem_new=K, modem_pause=MS,+ * modem_wait=S (step 7's watch, default 10), modem_listen=S (step 10, default 3), modem_again=1,+ * modem_return=1, as the modem's.+ *+ * Nothing here writes a clock or changes a pin. One request goes to RPMh (the two rails), as+ * Linux's remoteproc makes it; the SIM is not touched, no call is made, no data session opened.+ */+#include <kern/clock.h>+#include <kern/kalloc.h>+#include <kern/locks.h>+#include <kern/thread.h>+#include <libkern/libkern.h>+#include <os/atomic_private.h>+#include <mach/mach_types.h>+#include <mach/vm_param.h>+#include <vm/memory_types.h>+#include <pexpert/pexpert.h>+#include <sys/errno.h>+#include <sys/kauth.h>+#include <sys/stat.h>+#include <sys/systm.h>+#include <sys/vnode.h>+#include <sys/vnode_internal.h>+#include <vm/vm_kern.h>+#include <sys/redwood_input.h>+#include <sys/redwood_nas.h>+#include <sys/redwood_qrtr.h>+#include <sys/sysctl.h>++extern vm_offset_t ml_io_map(vm_offset_t phys_addr, vm_size_t size);+extern vm_offset_t ml_io_map_unmappable(vm_offset_t phys_addr, vm_size_t size, unsigned int flags);+extern void ml_io_unmap(vm_offset_t addr, vm_size_t sz);+extern uint64_t kvtophys(vm_offset_t va);+extern void loader_return(void);+extern kern_return_t thread_terminate(thread_t thread);+extern vm_offset_t pe_arm_get_soc_base_phys(void);+extern dev_t mdevlookup(int devid);+extern int mdevgetkernelrange(dev_t dev, vm_offset_t *base, uint64_t *size);+extern void FlushPoC_DcacheRegion(vm_offset_t va, size_t length); /* osfmk/arm64/caches_asm.s */++/* What the Wi-Fi's DSP and this one both use (redwood_wifi.c). */+extern lck_mtx_t redwood_aop_mailbox_lock;+extern lck_mtx_t redwood_smem_alloc_lock;++static LCK_GRP_DECLARE(mpss_lock_group, "redwood-mpss");++/* the modem's file systems, served from here (rmtfs, below) */+static bool rmtfs_buffer_make(void);+static bool ipa_firmware_load(void);+static void ipa_firmware_stop(void);+static bool ipa_loaded; /* the IPA's firmware runs (ipa_firmware_load()) */+static uint32_t ipa_mem_offset; /* where the IPA's local memory begins, as its register says */+static bool rmtfs_share(bool in);+static bool rmtfs_start(void);+static void rmtfs_stop(void);++#define STEPS 11+#define REDWOOD_SOC_BASE 0x00400000ul /* where the loader says the phone's peripherals begin */++static uint32_t last_step, pause_ms = 2200, listen_seconds = 3;+static uint32_t step; /* the one being performed */+static uint32_t new_from; /* the first of the steps that have not been seen to work: they wait */+static char last_write[96] = "nothing written";+static bool leaving; /* on the way back to the loader after a panic: no lock is taken, nothing sleeps */++/*+ * Every line goes to the log the loader reads from another core as well (as in redwood_wifi.c);+ * with modem=1 only the few that say how it went (tell()), the rest to the kernel's own log.+ */+extern void loader_log_putc(char c);++static LCK_MTX_DECLARE(say_lock, &mpss_lock_group);+static bool quiet;++static void+say_line(const char *line, bool aloud)+{+ if (leaving) {+ for (const char *c = line; *c != 0; c++) {+ loader_log_putc(*c);+ }+ return;+ }+ lck_mtx_lock(&say_lock);+ if (!quiet || aloud) {+ for (const char *c = line; *c != 0; c++) {+ loader_log_putc(*c);+ }+ }+ printf("%s", line);+ lck_mtx_unlock(&say_lock);+}++static int say(const char *format, ...) __printflike(1, 2);+static int tell(const char *format, ...) __printflike(1, 2);++static int+say(const char *format, ...)+{+ char line[400];+ va_list list;+ int length;++ va_start(list, format);+ length = vsnprintf(line, sizeof(line), format, list);+ va_end(list);+ say_line(line, false);+ return length;+}++static int+tell(const char *format, ...)+{+ char line[400];+ va_list list;+ int length;++ va_start(list, format);+ length = vsnprintf(line, sizeof(line), format, list);+ va_end(list);+ say_line(line, true);+ return length;+}+#define printf say++static void+sleep_us(uint32_t us)+{+ delay_for_interval(us, NSEC_PER_USEC);+}++static void+sleep_ms(uint32_t ms)+{+ delay_for_interval(ms, NSEC_PER_MSEC);+}++static uint64_t+now_us(void)+{+ uint64_t nanoseconds;++ absolutetime_to_nanoseconds(mach_absolute_time(), &nanoseconds);+ return nanoseconds / 1000;+}++/* Until every write to memory so far has been carried out. */+static void+settle(void)+{+ __asm__ volatile ("dsb sy");+}++static bool+option(const char *name)+{+ uint32_t value = 0;++ return PE_parse_boot_argn(name, &value, sizeof(value)) && value != 0;+}++/* ---- register access (as in redwood_wifi.c) ------------------------------------------------------- */++static uint32_t __attribute__((noinline))+rd(vm_offset_t address)+{+ uint32_t value;++ __asm__ volatile ("ldr %w0, [%1]" : "=r" (value) : "r" (address) : "memory");+ return value;+}++static void __attribute__((noinline))+wr(vm_offset_t address, uint32_t value)+{+ __asm__ volatile ("str %w0, [%1]" : : "r" (value), "r" (address) : "memory");+}++struct block {+ const char *name;+ uint32_t phys, size;+ vm_offset_t base;+};++static void+need(struct block *b)+{+ if (b->base == 0) {+ b->base = ml_io_map(b->phys, b->size);+ }+}++/* Time for the loader to note what has just been said, where it is not yet known to be harmless. */+static void+hold(void)+{+ if (step >= new_from && pause_ms && !leaving) {+ sleep_ms(pause_ms);+ }+}++static void+announce(const char *what)+{+ printf("mpss: step %u: %s\n", step, what);+ hold();+}++static uint32_t+get(struct block *b, const char *name, uint32_t offset)+{+ uint32_t value;++ need(b);+ value = rd(b->base + offset);+ printf("mpss: %u: %s %s (%#x) = %#x\n", step, b->name, name, b->phys + offset, value);+ return value;+}++static void+put(struct block *b, const char *name, uint32_t offset, uint32_t value)+{+ need(b);+ printf("mpss: %u: about to write %s %s at %#x: %#x\n", step, b->name, name, b->phys + offset, value);+ snprintf(last_write, sizeof(last_write), "step %u wrote %#x to %s %s at %#x", step, value, b->name, name, b->phys + offset);+ wr(b->base + offset, value);+}++/* ---- the shared memory (redwood_wifi.c, step 1) --------------------------------------------------- */++#define SMEM_PHYS 0x80900000UL+#define SMEM_SIZE 0x00200000u+#define SMEM_US 0+#define SMEM_MPSS 1 /* dt: qcom,remote-pid of qcom,smp2p-modem and qcom,glink/modem */+#define SMEM_GLOBAL 0xfffe+#define SMEM_CANARY 0xa5a5++struct smem_table_entry {+ uint32_t offset, size, flags;+ uint16_t host0, host1;+ uint32_t cacheline, reserved[7];+};+struct smem_table {+ uint8_t magic[4];+ uint32_t version, count, reserved[5];+};+struct smem_partition {+ uint8_t magic[4];+ uint16_t host0, host1;+ uint32_t size, free_uncached, free_cached, reserved[3];+};+struct smem_entry {+ uint16_t canary, item;+ uint32_t size;+ uint16_t padding_data, padding_header;+ uint32_t reserved;+};+struct smem_item {+ uint32_t item, offset, size;+};++/*+ * The DSP's items (dt: qcom,smem of qcom,smp2p-modem, <435 428>: the item the DSP writes first, then+ * ours -- the Wi-Fi's <617 616> are in the same order; taken the other way round, the DSP made+ * our item its own and never said "up" --, and qcom,smem-id of qcom,mss@4080000).+ */+#define ITEM_SMP2P_OURS 428+#define ITEM_SMP2P_THEIRS 435+#define ITEM_CRASH_REASON 421++static const volatile uint8_t *smem;++static bool+smem_peek(uint64_t offset, void *to, uint32_t size)+{+ if (smem == NULL) {+ smem = (const volatile uint8_t *)ml_io_map_unmappable(SMEM_PHYS, SMEM_SIZE, VM_WIMG_WCOMB);+ }+ if (offset + size > SMEM_SIZE) {+ return false;+ }+ for (uint32_t i = 0; i < size; i++) {+ ((uint8_t *)to)[i] = smem[offset + i];+ }+ return true;+}++static void+smem_poke(uint32_t offset, uint32_t value)+{+ wr((vm_offset_t)smem + offset, value);+}++static uint32_t+smem_word(uint32_t offset)+{+ return rd((vm_offset_t)smem + offset);+}++static uint32_t+smem_partition(uint16_t one, uint16_t other, uint32_t *cacheline, uint32_t *size)+{+ struct smem_table table;+ struct smem_table_entry entry;+ struct smem_partition header;++ if (!smem_peek(SMEM_SIZE - 0x1000, &table, sizeof(table)) || memcmp(table.magic, "$TOC", 4) != 0 || table.version != 1 ||+ table.count > 64) {+ return 0;+ }+ for (uint32_t i = 0; i < table.count; i++) {+ if (!smem_peek(SMEM_SIZE - 0x1000 + sizeof(table) + i * sizeof(entry), &entry, sizeof(entry)) || entry.offset == 0 ||+ entry.size == 0) {+ continue;+ }+ if (!((entry.host0 == one && entry.host1 == other) || (entry.host0 == other && entry.host1 == one))) {+ continue;+ }+ if ((uint64_t)entry.offset + entry.size > SMEM_SIZE - 0x1000 || !smem_peek(entry.offset, &header, sizeof(header)) ||+ memcmp(header.magic, "$PRT", 4) != 0 || header.size != entry.size) {+ return 0;+ }+ *cacheline = entry.cacheline;+ *size = entry.size;+ return entry.offset;+ }+ return 0;+}++static unsigned+smem_items(uint32_t partition, uint32_t cacheline, struct smem_item *found, unsigned max)+{+ struct smem_partition header;+ struct smem_entry entry;+ uint32_t at, step_back;+ unsigned count = 0;++ if (partition == 0 || !smem_peek(partition, &header, sizeof(header)) || header.free_uncached > header.size ||+ header.free_cached > header.size) {+ return 0;+ }+ for (at = sizeof(header); at + sizeof(entry) <= header.free_uncached;) {+ if (!smem_peek(partition + at, &entry, sizeof(entry)) || entry.canary != SMEM_CANARY || entry.size > header.size) {+ break;+ }+ if (count < max) {+ found[count].item = entry.item;+ found[count].offset = partition + at + sizeof(entry) + entry.padding_header;+ found[count].size = entry.size - entry.padding_data;+ }+ count++;+ at += sizeof(entry) + entry.padding_header + entry.size;+ }+ if (cacheline == 0 || (cacheline & (cacheline - 1))) {+ return count;+ }+ step_back = ((uint32_t)sizeof(entry) + cacheline - 1) & ~(cacheline - 1);+ for (at = header.size - step_back; at > header.free_cached && at < header.size;) {+ if (!smem_peek(partition + at, &entry, sizeof(entry)) || entry.canary != SMEM_CANARY || entry.size > at) {+ break;+ }+ if (count < max) {+ found[count].item = entry.item;+ found[count].offset = partition + at - entry.size;+ found[count].size = entry.size - entry.padding_data;+ }+ count++;+ if (entry.size + step_back > at) {+ break;+ }+ at -= entry.size + step_back;+ }+ return count;+}++/* An item the DSP and we share: in the partition of the two, or else in the global one. Where it is; 0: nowhere. */+static uint32_t+smem_item(uint32_t item, uint32_t *size)+{+ static struct smem_item found[256];+ static const uint16_t hosts[2][2] = { { SMEM_US, SMEM_MPSS }, { SMEM_GLOBAL, SMEM_GLOBAL } };++ for (unsigned h = 0; h < 2; h++) {+ uint32_t cacheline = 0, partition_size = 0;+ uint32_t partition = smem_partition(hosts[h][0], hosts[h][1], &cacheline, &partition_size);+ unsigned count = smem_items(partition, cacheline, found, 256);++ for (unsigned i = 0; i < count && i < 256; i++) {+ if (found[i].item == item) {+ *size = found[i].size;+ return found[i].offset;+ }+ }+ }+ return 0;+}++static void+show_partition(void)+{+ static struct smem_item found[256];+ struct smem_partition header;+ uint32_t cacheline = 0, size = 0, partition = smem_partition(SMEM_US, SMEM_MPSS, &cacheline, &size);+ unsigned count;+ char line[320];++ if (partition == 0 || !smem_peek(partition, &header, sizeof(header))) {+ printf("mpss: %u: there is no partition for us and the modem (0 and 1)\n", step);+ return;+ }+ count = smem_items(partition, cacheline, found, 256);+ printf("mpss: %u: the partition for us and the modem (0 and 1): at %#lx, %#x bytes, of which %#x free; %u items\n", step,+ SMEM_PHYS + partition, size, header.free_cached - header.free_uncached, count);+ for (unsigned i = 0; i < count && i < 256;) {+ int at = 0;++ for (unsigned n = 0; n < 12 && i < count && i < 256; n++, i++) {+ at += snprintf(line + at, sizeof(line) - (size_t)at, " %u (%u)", found[i].item, found[i].size);+ }+ printf("mpss: %u: items (bytes):%s\n", step, line);+ }+}++/* ---- SMP2P ---------------------------------------------------------------------------------------- */++struct smp2p {+ uint32_t magic;+ uint32_t version;+ uint16_t local, remote, total, valid;+ uint32_t flags;+ struct {+ char name[16];+ uint32_t value;+ } entry[16];+};+#define SMP2P_MAGIC 0x504d5324+#define SMP2P_VERSION 1+#define SMP2P_RESTART_ACKS 1u+#define SMP2P_RESTART_DONE (1u << 0)+#define SMP2P_RESTART_ACK (1u << 1)+#define SMP2P_AT_VERSION 4+#define SMP2P_AT_PROCESSORS 8+#define SMP2P_AT_COUNTS 12+#define SMP2P_AT_FLAGS 16+#define SMP2P_AT_NAME(i) (20 + 20 * (i))+#define SMP2P_AT_VALUE(i) (36 + 20 * (i))++/* The bits of the DSP's entry "slave-kernel" (dt: the interrupts of qcom,lpass@3700000). */+#define DSP_FATAL (1u << 0)+#define DSP_READY (1u << 1)+#define DSP_HANDOVER (1u << 2)+#define DSP_STOP_ACK (1u << 3)+#define DSP_SHUTDOWN_ACK (1u << 7)++/* Our entry (dt: the child "master-kernel" of qcom,smp2p-adsp; "qcom,force-stop" of qcom,lpass@3700000). */+#define OUR_ENTRY_STOP "master-kernel"+#define DSP_STOP (1u << 0)++static bool+dsp_words(struct smp2p *words)+{+ static uint32_t at;+ uint32_t size = 0;++ if (at == 0) {+ at = smem_item(ITEM_SMP2P_THEIRS, &size);+ if (at == 0 || size < sizeof(*words)) {+ at = 0;+ return false;+ }+ }+ return smem_peek(at, words, sizeof(*words)) && words->magic == SMP2P_MAGIC;+}++static int+smp2p_entry(const struct smp2p *words, const char *name)+{+ for (unsigned i = 0; i < words->valid && i < 16; i++) {+ if (strncmp(words->entry[i].name, name, sizeof(words->entry[i].name)) == 0) {+ return (int)i;+ }+ }+ return -1;+}++static uint32_t+dsp_state(const struct smp2p *words)+{+ int entry = smp2p_entry(words, "slave-kernel");++ return entry < 0 ? 0 : words->entry[entry].value;+}++static void+show_words(const char *prefix, const struct smp2p *words)+{+ char line[320];+ int at = 0;++ line[0] = 0;+ for (unsigned i = 0; i < words->valid && i < 16 && at < (int)sizeof(line) - 40; i++) {+ at += snprintf(line + at, sizeof(line) - (size_t)at, " %.16s = %#x;", words->entry[i].name, words->entry[i].value);+ }+ printf("mpss: %u: %s: SMP2P version %u, features %#x, from processor %u to %u, flags %#x, %u of %u entries:%s\n", step, prefix,+ words->version & 0xff, words->version >> 8, words->local, words->remote, words->flags, words->valid, words->total, line);+}++static void+show_crash_reason(void)+{+ char text[257];+ uint32_t size = 0, at = smem_item(ITEM_CRASH_REASON, &size);++ if (at == 0 || size == 0) {+ printf("mpss: %u: item %u, the DSP's last words: not there\n", step, ITEM_CRASH_REASON);+ return;+ }+ bzero(text, sizeof(text));+ smem_peek(at, text, size < sizeof(text) - 1 ? size : sizeof(text) - 1);+ if (text[0] == 0) {+ printf("mpss: %u: item %u, the DSP's last words (%u bytes): none\n", step, ITEM_CRASH_REASON, size);+ } else {+ printf("mpss: %u: THE MODEM'S LAST WORDS (item %u, %u bytes): \"%s\"\n", step, ITEM_CRASH_REASON, size, text);+ }+}++/* ---- the doorbells, and the mailbox to the always-on processor (redwood_wifi.c, steps 2 and 7) ------ */++#define QMP_MAGIC 0x00+#define QMP_MAGIC_VALUE 0x4d41494c+#define QMP_VERSION 0x04+#define QMP_THEIR_LINK 0x0c+#define QMP_THEIR_LINK_ACK 0x10+#define QMP_THEIR_CHANNEL 0x14+#define QMP_THEIR_CHANNEL_ACK 0x18+#define QMP_OUR_LINK 0x24+#define QMP_OUR_LINK_ACK 0x28+#define QMP_OUR_CHANNEL 0x2c+#define QMP_OUR_CHANNEL_ACK 0x30+#define QMP_OUR_BOX_SIZE 0x34+#define QMP_OUR_BOX 0x38+#define QMP_UP 0x0000ffffu+#define QMP_DOWN 0xffff0000u+#define QMP_MESSAGE 64++static struct block aoss = { "mailbox", 0x0c300000, 0x400 };++#define IPCC_SEND 0x0c+#define IPCC_RECEIVED 0x10+#define IPCC_ENABLE 0x14+#define IPCC_DISABLE 0x18+#define IPCC_CLEAR 0x1c+#define IPCC_NOBODY 0xffffffffu+#define IPCC_AOP 0u+#define IPCC_MPSS 2u /* dt: mboxes of qcom,smp2p-modem and qcom,glink/modem */+#define IPCC_GLINK 0u+#define IPCC_SMP2P 2u++static struct block ipcc = { "ipcc", 0x00408000, 0x1000 };++static struct {+ uint32_t glink, smp2p, said;+ uint64_t since;+} doorbell;++/*+ * The modem's rings, counted and cleared. Another's ring (the Wi-Fi DSP's, the always-on+ * processor's) is left for whoever listens for it: IPCC names the same one until it is cleared,+ * so the look ends there.+ */+static void+doorbells(void)+{+ for (unsigned i = 0; i < 8; i++) {+ uint32_t who = rd(ipcc.base + IPCC_RECEIVED);++ if (who == IPCC_NOBODY || who >> 16 != IPCC_MPSS) {+ return;+ }+ wr(ipcc.base + IPCC_CLEAR, who);+ doorbell.glink += (who & 0xffff) == IPCC_GLINK;+ doorbell.smp2p += (who & 0xffff) == IPCC_SMP2P;+ if (doorbell.said < 24) {+ uint64_t now = now_us() - doorbell.since;++ doorbell.said++;+ printf("mpss: doorbell: +%llu.%03llu ms: signal %u of the modem; heard and cleared\n", now / 1000, now % 1000,+ who & 0xffff);+ }+ }+}++static uint32_t our_box; /* where in the message memory our mailbox is; 0: not open */++static bool+mailbox_wait(uint32_t offset, uint32_t value, uint32_t ms)+{+ for (uint32_t i = 0; i < ms * 10; i++) {+ if (rd(aoss.base + offset) == value) {+ return true;+ }+ sleep_us(100);+ }+ return rd(aoss.base + offset) == value;+}++static void+ring_aop(void)+{+ wr(ipcc.base + IPCC_SEND, IPCC_AOP << 16 | 0);+}++/* Linux's qmp_open(), as redwood_wifi.c does it; what is done already is found done. Under the mailbox's lock. */+static bool+mailbox_open(void)+{+ uint32_t box, size;++ need(&aoss);+ need(&ipcc);+ if (get(&aoss, "magic", QMP_MAGIC) != QMP_MAGIC_VALUE || get(&aoss, "version", QMP_VERSION) != 1) {+ printf("mpss: %u: not a mailbox this knows\n", step);+ return false;+ }+ box = get(&aoss, "our mailbox", QMP_OUR_BOX);+ size = get(&aoss, "our mailbox's size", QMP_OUR_BOX_SIZE);+ if (size < QMP_MESSAGE + 4 || box < 0x3c || box + size > aoss.size || (box & 3)) {+ printf("mpss: %u: our mailbox is not where a message fits\n", step);+ return false;+ }+ if (rd(aoss.base + QMP_OUR_LINK) == QMP_UP && rd(aoss.base + QMP_OUR_LINK_ACK) == QMP_UP &&+ rd(aoss.base + QMP_OUR_CHANNEL) == QMP_UP && rd(aoss.base + QMP_THEIR_CHANNEL) == QMP_UP &&+ rd(aoss.base + QMP_OUR_CHANNEL_ACK) == QMP_UP) {+ printf("mpss: %u: the mailbox is open already (the Wi-Fi's start, or a kernel before this one, opened it)\n", step);+ our_box = box;+ return true;+ }+ put(&aoss, "their link, seen by us", QMP_THEIR_LINK_ACK, rd(aoss.base + QMP_THEIR_LINK));+ put(&aoss, "our link", QMP_OUR_LINK, QMP_UP);+ ring_aop();+ if (!mailbox_wait(QMP_OUR_LINK_ACK, QMP_UP, 1000)) {+ printf("mpss: %u: THE ALWAYS-ON PROCESSOR DOES NOT ACKNOWLEDGE OUR LINK within a second; taken down again\n", step);+ put(&aoss, "our link", QMP_OUR_LINK, QMP_DOWN);+ ring_aop();+ return false;+ }+ put(&aoss, "our channel", QMP_OUR_CHANNEL, QMP_UP);+ ring_aop();+ if (!mailbox_wait(QMP_THEIR_CHANNEL, QMP_UP, 1000)) {+ printf("mpss: %u: THE ALWAYS-ON PROCESSOR DOES NOT OPEN ITS CHANNEL within a second; ours is taken down again\n", step);+ put(&aoss, "our channel", QMP_OUR_CHANNEL, QMP_DOWN);+ put(&aoss, "our link", QMP_OUR_LINK, QMP_DOWN);+ ring_aop();+ return false;+ }+ put(&aoss, "their channel, seen by us", QMP_THEIR_CHANNEL_ACK, QMP_UP);+ ring_aop();+ if (!mailbox_wait(QMP_OUR_CHANNEL_ACK, QMP_UP, 1000)) {+ printf("mpss: %u: THE ALWAYS-ON PROCESSOR DOES NOT ACKNOWLEDGE OUR CHANNEL within a second; taken down again\n", step);+ put(&aoss, "our channel", QMP_OUR_CHANNEL, QMP_DOWN);+ put(&aoss, "our link", QMP_OUR_LINK, QMP_DOWN);+ ring_aop();+ return false;+ }+ printf("mpss: %u: THE MAILBOX TO THE ALWAYS-ON PROCESSOR IS OPEN\n", step);+ our_box = box;+ return true;+}++/* A message to the always-on processor (Linux's qmp_send()), under the lock the Wi-Fi's start takes too. */+static bool+mailbox_send(const char *text)+{+ uint32_t words[QMP_MESSAGE / 4] = { 0 };+ bool sent = false;++ if (!leaving) {+ lck_mtx_lock(&redwood_aop_mailbox_lock);+ }+ if (our_box == 0 && (leaving || !mailbox_open())) {+ goto out;+ }+ strlcpy((char *)words, text, sizeof(words));+ printf("mpss: %u: about to tell the always-on processor \"%s\"\n", step, text);+ snprintf(last_write, sizeof(last_write), "step %u sent the always-on processor a message", step);+ if (rd(aoss.base + our_box) != 0) {+ printf("mpss: %u: our mailbox still holds a message of %u bytes: not sent\n", step, rd(aoss.base + our_box));+ goto out;+ }+ for (unsigned i = 0; i < QMP_MESSAGE / 4; i++) {+ wr(aoss.base + our_box + 4 + 4 * i, words[i]);+ }+ wr(aoss.base + our_box, QMP_MESSAGE);+ (void)rd(aoss.base + our_box);+ ring_aop();+ if (!mailbox_wait(our_box, 0, 1000)) {+ printf("mpss: %u: THE ALWAYS-ON PROCESSOR DOES NOT TAKE THE MESSAGE within a second; it is taken back\n", step);+ wr(aoss.base + our_box, 0);+ goto out;+ }+ printf("mpss: %u: the always-on processor has taken it\n", step);+ sent = true;+out:+ if (!leaving) {+ lck_mtx_unlock(&redwood_aop_mailbox_lock);+ }+ return sent;+}++/* ---- RPMh: the DSP's two rails, asked for while it boots ------------------------------------------------ */++/*+ * The modem's logic and memories run from LPASS's own rails, LCX and LMX (dt: vdd_cx and+ * vdd_mx of qcom,lpass@3700000 are PM7325 L5 and L4 "level" regulators, RPMh resources cx.lvl and+ * mss.lvl), which the boot loader leaves at retention (16). Linux and Android ask for level 384+ * while the DSP boots, until it says it has asked for its own (bit 2, "proxy-unvote"); without it+ * the DSP rang once and left "SFR Init: wdog or kernel error suspected." in its last words.+ *+ * A request goes to RPMh through the application processor's sequencer (dt: rsc@18200000, DRV 2,+ * qcom,tcs-offset 0xd00, qcom,tcs-config: TCSs 0 and 1 are for requests made at once), as Linux's+ * rpmh-rsc.c writes one: the command's message id, address and value into a TCS that is idle, the+ * command enabled and marked to be waited for, the TCS put in "active" mode and triggered, and its+ * bit in the status of completions looked for. A rail asked for by level is asked by position in+ * the list of its levels that the boot loader's command database keeps (cmd-db.c), the first+ * position at or above the level wanted.+ */+#define CMD_DB_PHYS 0x80860000UL+#define CMD_DB_SIZE 0x20000UL++struct cmd_db_slave {+ uint16_t id, header_offset, data_offset, count, version, reserved[3];+};+struct cmd_db_header {+ uint32_t version;+ uint8_t magic[4];+ struct cmd_db_slave slave[8];+ uint32_t checksum, reserved;+ uint8_t data[];+};+struct cmd_db_entry {+ char id[8];+ uint32_t priority[2];+ uint32_t address;+ uint16_t length, offset;+};++static const struct cmd_db_header *cmd_db;++static struct block rsc = { "rsc", 0x18220000, 0x10000 };+#define RSC_TCS_BASE 0xd00+#define RSC_IRQ_STATUS (RSC_TCS_BASE + 0x04) /* a bit per TCS: done */+#define RSC_IRQ_CLEAR (RSC_TCS_BASE + 0x08)+#define RSC_TCS(n) (RSC_TCS_BASE + 672 * (n))+#define TCS_WAIT_FOR_DONE 0x10+#define TCS_CONTROL 0x14+#define TCS_STATUS 0x18 /* not zero: idle */+#define TCS_ENABLED 0x1c+#define TCS_COMMAND(c) (0x30 + 20 * (c)) /* message id, address, value, status */+#define TCS_ACTIVE_MODE (1u << 16)+#define TCS_TRIGGER (1u << 24)+#define MSGID_WRITE_WAITED (8u | 1u << 8 | 1u << 16) /* four bytes, an answer wanted, a write */+#define RPMH_TCS 0 /* the first of the two for requests made at once */++/* The level position RPMh takes for that rail and level; -1: not found. */+static int+rail_position(const char *name, uint32_t level, uint32_t *address)+{+ char id[8] = { 0 };++ if (cmd_db == NULL) {+ cmd_db = (const struct cmd_db_header *)ml_io_map_unmappable(CMD_DB_PHYS, CMD_DB_SIZE, VM_WIMG_WCOMB);+ }+ if (!(cmd_db->magic[0] == 0xdb && cmd_db->magic[1] == 0x30 && cmd_db->magic[2] == 0x03 && cmd_db->magic[3] == 0x0c)) {+ printf("mpss: %u: the boot loader's command database is not there\n", step);+ return -1;+ }+ strlcpy(id, name, sizeof(id));+ for (unsigned s = 0; s < 8; s++) {+ const struct cmd_db_slave *slave = &cmd_db->slave[s];+ const struct cmd_db_entry *entry = (const struct cmd_db_entry *)(cmd_db->data + slave->header_offset);++ if (slave->id == 0 || slave->count > 512) {+ continue;+ }+ for (unsigned i = 0; i < slave->count; i++) {+ const uint8_t *data;+ char levels[160];+ int at;++ if (memcmp(entry[i].id, id, sizeof(id)) != 0) {+ continue;+ }+ if ((unsigned)slave->data_offset + entry[i].offset + entry[i].length > CMD_DB_SIZE - 0x1000) {+ return -1;+ }+ data = cmd_db->data + slave->data_offset + entry[i].offset;+ at = snprintf(levels, sizeof(levels), "%s at %#x, levels:", name, entry[i].address);+ for (unsigned l = 0; l < entry[i].length / 2u && l < 16; l++) {+ unsigned value = data[2 * l] | data[2 * l + 1] << 8;++ if (l > 0 && value == 0) {+ break;+ }+ at += snprintf(levels + at, sizeof(levels) - (size_t)at, " %u", value);+ }+ printf("mpss: %u: RPMh has %s\n", step, levels);+ for (unsigned l = 0; l < entry[i].length / 2u && l < 16; l++) {+ if ((uint32_t)(data[2 * l] | data[2 * l + 1] << 8) >= level) {+ *address = entry[i].address;+ return (int)l;+ }+ }+ return -1;+ }+ }+ printf("mpss: %u: RPMh has no \"%s\"\n", step, name);+ return -1;+}++/* The two rails at that level (Linux's proxy request), or given back (level 0). True: RPMh has said done. */+static bool+rails(uint32_t level)+{+ static const char *const names[2] = { "cx.lvl", "mss.lvl" };+ uint32_t addresses[2] = { 0, 0 }, values[2] = { 0, 0 }, done = 0;+ uint64_t began;++ for (unsigned r = 0; r < 2; r++) {+ int position = level == 0 ? 0 : rail_position(names[r], level, &addresses[r]);++ if (level == 0) {+ rail_position(names[r], 0, &addresses[r]);+ }+ if (position < 0 || addresses[r] == 0) {+ printf("mpss: %u: no position for %s at level %u: the rails are not asked\n", step, names[r], level);+ return false;+ }+ values[r] = (uint32_t)position;+ }+ need(&rsc);+ /* (the boot loader leaves the trigger set after its last request, done: idle is what the status says) */+ if (rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_STATUS) == 0) {+ printf("mpss: %u: TCS %u of the sequencer is busy (status %#x, control %#x): the rails are not asked\n", step, RPMH_TCS,+ rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_STATUS), rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL));+ return false;+ }+ printf("mpss: %u: about to ask RPMh, through TCS %u, for cx.lvl (%#x) = %u and mss.lvl (%#x) = %u (level %u%s)\n", step,+ RPMH_TCS, addresses[0], values[0], addresses[1], values[1], level, level == 0 ? ": given back" : "");+ snprintf(last_write, sizeof(last_write), "step %u asked RPMh for the modem's rails at level %u", step, level);+ hold();+ wr(rsc.base + RSC_IRQ_CLEAR, 1u << RPMH_TCS);+ for (unsigned r = 0; r < 2; r++) {+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(r), MSGID_WRITE_WAITED);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(r) + 4, addresses[r]);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(r) + 8, values[r]);+ }+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_WAIT_FOR_DONE, 3);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_ENABLED, 3);+ settle();+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL) & ~TCS_TRIGGER);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL) | TCS_ACTIVE_MODE);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL) | TCS_TRIGGER);+ began = now_us();+ while (now_us() - began < 100000) {+ done = rd(rsc.base + RSC_IRQ_STATUS) & (1u << RPMH_TCS);+ if (done) {+ break;+ }+ sleep_us(50);+ }+ printf("mpss: %u: RPMh %s after %llu us; the commands' status %#x %#x, the TCS's %#x\n", step, done ? "HAS DONE IT" :+ "HAS NOT SAID DONE", now_us() - began, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(0) + 12),+ rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(1) + 12), rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_STATUS));+ wr(rsc.base + RSC_IRQ_CLEAR, 1u << RPMH_TCS);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL) & ~(TCS_TRIGGER | TCS_ACTIVE_MODE));+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_ENABLED, 0);+ return done != 0;+}+++/*+ * One command to RPMh through the sequencer, as rails() sends its two: a resource's address from+ * the command database and the word for it. True: RPMh has said done. The IPA's clock is such a+ * resource, a "bus clock manager" named IP0 (Linux's clk-rpmh.c: BCM votes are a word of commit+ * (bit 30), valid (bit 29) and a vote of 14 bits).+ */+static bool+rpmh_command(const char *name, uint32_t value)+{+ uint32_t address = 0, done = 0;+ uint64_t began;++ if (rail_position(name, 0, &address) < 0 || address == 0) {+ printf("mpss: %u: RPMh has no \"%s\": nothing is asked\n", step, name);+ return false;+ }+ need(&rsc);+ if (rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_STATUS) == 0) {+ printf("mpss: %u: TCS %u of the sequencer is busy: \"%s\" is not asked\n", step, RPMH_TCS, name);+ return false;+ }+ printf("mpss: %u: about to ask RPMh, through TCS %u, for %s (%#x) = %#x\n", step, RPMH_TCS, name, address, value);+ snprintf(last_write, sizeof(last_write), "step %u asked RPMh for %s = %#x", step, name, value);+ hold();+ wr(rsc.base + RSC_IRQ_CLEAR, 1u << RPMH_TCS);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(0), MSGID_WRITE_WAITED);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(0) + 4, address);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(0) + 8, value);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_WAIT_FOR_DONE, 1);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_ENABLED, 1);+ settle();+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL) & ~TCS_TRIGGER);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL) | TCS_ACTIVE_MODE);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL) | TCS_TRIGGER);+ began = now_us();+ while (now_us() - began < 100000) {+ done = rd(rsc.base + RSC_IRQ_STATUS) & (1u << RPMH_TCS);+ if (done) {+ break;+ }+ sleep_us(50);+ }+ printf("mpss: %u: RPMh %s after %llu us for %s; the command's status %#x\n", step, done ? "HAS DONE IT" : "HAS NOT SAID DONE",+ now_us() - began, name, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_COMMAND(0) + 12));+ wr(rsc.base + RSC_IRQ_CLEAR, 1u << RPMH_TCS);+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL, rd(rsc.base + RSC_TCS(RPMH_TCS) + TCS_CONTROL) & ~(TCS_TRIGGER | TCS_ACTIVE_MODE));+ wr(rsc.base + RSC_TCS(RPMH_TCS) + TCS_ENABLED, 0);+ return done != 0;+}++#define BCM_VOTE(on, vote) (1u << 30 | (uint32_t)(on) << 29 | ((vote) & 0x3fffu))++/* The IPA's clock, on or off: the BCM "IP0" (dt of Linux: clocks = <&rpmhcc RPMH_IPA_CLK>). */+static bool+ipa_clock(bool on)+{+ return rpmh_command("IP0", BCM_VOTE(on, on ? 0x3fffu : 0));+}++/* ---- step 1: the shared memory --------------------------------------------------------------------- */++static bool+step_survey(void)+{+ struct smp2p words;+ uint32_t size = 0;++ announce("read our partition with the modem in the shared memory (0x80900000), and its items");+ show_partition();+ if (dsp_words(&words)) {+ show_words("the DSP's item (435) is there: it has run since the phone was started", &words);+ } else {+ printf("mpss: %u: the DSP's SMP2P item (435) is not there: it has not run since the phone was started\n", step);+ }+ printf("mpss: %u: our SMP2P item (428): %s; GLINK's descriptor (478): %s, our queue (479): %s, the DSP's (480): %s\n", step,+ smem_item(ITEM_SMP2P_OURS, &size) ? "there" : "not there", smem_item(478, &size) ? "there" : "NOT THERE",+ smem_item(479, &size) ? "there" : "NOT THERE", smem_item(480, &size) ? "there" : "not there (the DSP makes it)");+ show_crash_reason();+ need(&ipcc);+ printf("mpss: %u: the doorbells say %#x\n", step, rd(ipcc.base + IPCC_RECEIVED));+ return smem_partition(SMEM_US, SMEM_MPSS, &size, &size) != 0;+}++/* ---- step 2: the firmware -------------------------------------------------------------------------- */++#define MPSS_MEMORY 0x8b800000u /* dt: reserved-memory mpss@8b800000, memory-region of qcom,mss@4080000 */+#define MPSS_MEMORY_SIZE 0x0f600000u+#define PAS_MPSS 4 /* dt: qcom,pas-id */+++/*+ * A file of /fw on the root file system, read whole into memory of ours: the system image, where+ * userland/build.sh puts fw-modem/ when it makes that image. It is compressed there, which the file+ * system undoes and the reading of the small image's memory below does not; and it is there only+ * once the root is mounted, so only modem=1, which waits for that on its own thread, finds it so.+ */+static const uint8_t *+root_file(const char *name, uint64_t *size)+{+ char path[256];+ vfs_context_t context = vfs_context_kernel();+ vnode_t vnode = NULLVP;+ struct vnode_attr attributes;+ uint8_t *data;+ int left = 0, error;++ if (rootvnode == NULLVP) {+ return NULL;+ }+ snprintf(path, sizeof(path), "/fw/%s", name);+ if (vnode_lookup(path, 0, &vnode, context) != 0) {+ return NULL;+ }+ VATTR_INIT(&attributes);+ VATTR_WANTED(&attributes, va_data_size);+ if (vnode_getattr(vnode, &attributes, context) != 0 || attributes.va_data_size == 0 ||+ attributes.va_data_size > MPSS_MEMORY_SIZE) {+ vnode_put(vnode);+ return NULL;+ }+ data = kalloc_data(attributes.va_data_size, Z_WAITOK);+ if (data == NULL) {+ vnode_put(vnode);+ return NULL;+ }+ error = vn_rdwr(UIO_READ, vnode, (caddr_t)data, (int)attributes.va_data_size, 0, UIO_SYSSPACE, 0,+ vfs_context_ucred(context), &left, vfs_context_proc(context));+ vnode_put(vnode);+ if (error != 0 || left != 0) {+ printf("mpss: %u: %s: read error %d, %d bytes left\n", step, path, error, left);+ kfree_data(data, attributes.va_data_size);+ return NULL;+ }+ *size = attributes.va_data_size;+ return data;+}++static bool from_root; /* the image is in memory of ours (root_file()), to be given back once written */++/* A file of fw/ in the small image (MODEM=1 userland/build.sh puts fw-modem/ there), else of /fw on the root (above). */+extern const uint8_t *redwood_image_file(const char *dir, const char *name, uint64_t *size, int *disk_out);+static const uint8_t *+image_file(const char *name, uint64_t *size)+{+ {+ const uint8_t *data = redwood_image_file("fw", name, size, NULL); /* (redwood_image.c) */++ if (data != NULL) {+ return data;+ }+ }+ {+ const uint8_t *data = root_file(name, size);++ if (data != NULL) {+ from_root = true;+ return data;+ }+ }+ printf("mpss: %u: no fw/%s in the small image nor on the root: put the phone's /vendor/firmware_mnt/image/%s into "+ "fw-modem/ and run SYSTEM=1 userland/build.sh (or MODEM=1, for the small image)\n", step, name, name);+ return NULL;+}+++#define MDT_HASH(flags) (((flags) >> 24 & 7) == 2)+#define MAX_SEGMENTS 48++struct elf_segment {+ uint32_t type, offset, vaddr, paddr, file_size, memory_size, flags, align;+};++static struct {+ const uint8_t *header;+ uint64_t header_size;+ struct {+ uint32_t at;+ uint32_t file_size, memory_size;+ const uint8_t *file;+ } segment[MAX_SEGMENTS];+ unsigned segments;+ uint32_t size;+} image;++/* The image's files given back, if they were read into memory of ours. */+static void+image_forget(void)+{+ if (!from_root) {+ return;+ }+ if (image.header != NULL) {+ void *data = (void *)(uintptr_t)image.header;++ kfree_data(data, image.header_size);+ }+ for (unsigned s = 0; s < image.segments; s++) {+ if (image.segment[s].file != NULL) {+ void *data = (void *)(uintptr_t)image.segment[s].file;++ kfree_data(data, image.segment[s].file_size);+ }+ }+ bzero(&image, sizeof(image));+ from_root = false;+}++static unsigned rmtfs_load(void);+static void servreg_load(void);++static bool+step_firmware(void)+{+ struct elf_segment segment;+ const uint8_t *header;+ uint64_t size = 0, files = 0;+ uint32_t count, hashes = 0, lowest = ~0u, highest = 0;++ announce("find the modem's image in the small image or on the root, and check it against its own header");+ image_forget();+ image.segments = 0;+ header = image_file("modem.mdt", &size);+ if (header == NULL) {+ return false;+ }+ image.header = header;+ image.header_size = size;+ count = size >= 52 + 2 * 32 ? header[44] | header[45] << 8 : 0;+ if (memcmp(header, "\177ELF\1", 5) != 0 || count < 3 || count > MAX_SEGMENTS || 52 + 32 * count > size || size > PAGE_SIZE) {+ printf("mpss: %u: fw/modem.mdt is not such a header (%02x %02x %02x %02x, %u segments, %llu bytes)\n", step, header[0],+ header[1], header[2], header[3], count, size);+ goto fail;+ }+ for (uint32_t i = 0; i < count; i++) {+ char name[16];++ memcpy(&segment, header + 52 + 32 * i, sizeof(segment));+ if (MDT_HASH(segment.flags)) {+ hashes = segment.file_size;+ }+ if (segment.type != 1 || MDT_HASH(segment.flags) || segment.memory_size == 0) {+ continue;+ }+ if (segment.paddr < MPSS_MEMORY || segment.file_size > segment.memory_size ||+ (uint64_t)segment.paddr + segment.memory_size > (uint64_t)MPSS_MEMORY + MPSS_MEMORY_SIZE) {+ printf("mpss: %u: segment %u (%#x, %#x bytes) is not within the DSP's memory\n", step, i, segment.paddr,+ segment.memory_size);+ goto fail;+ }+ image.segment[image.segments].at = segment.paddr - MPSS_MEMORY;+ image.segment[image.segments].file_size = segment.file_size;+ image.segment[image.segments].memory_size = segment.memory_size;+ image.segment[image.segments].file = NULL;+ if (segment.file_size != 0) {+ uint64_t file_size = 0;++ snprintf(name, sizeof(name), "modem.b%02u", i);+ image.segment[image.segments].file = image_file(name, &file_size);+ if (image.segment[image.segments].file == NULL) {+ goto fail;+ }+ if (file_size != segment.file_size) {+ printf("mpss: %u: fw/%s has %llu bytes, the header says %u\n", step, name, file_size, segment.file_size);+ image.segments++; /* (for its file to be given back) */+ goto fail;+ }+ files += file_size;+ }+ lowest = segment.paddr < lowest ? segment.paddr : lowest;+ highest = segment.paddr + segment.memory_size > highest ? segment.paddr + segment.memory_size : highest;+ image.segments++;+ }+ if (image.segments == 0 || lowest != MPSS_MEMORY || 52 + 32 * count + (uint64_t)hashes != size) {+ printf("mpss: %u: not what TrustZone takes: %u segments to load, the lowest at %#x; the header file has %llu bytes, "+ "the header %u and the hashes %u\n", step, image.segments, lowest, size, 52 + 32 * count, hashes);+ goto fail;+ }+ image.header = header;+ image.header_size = size;+ image.size = ((highest + 0xfff) & ~0xfffu) - MPSS_MEMORY;+ printf("mpss: %u: the image is whole (%s): a header of %llu bytes (%u of hashes), %u segments, %llu bytes of files, %#x "+ "bytes of the DSP's %#x from %#x\n", step, from_root ? "from /fw on the root" : "from the small image", size, hashes,+ image.segments, files, image.size, MPSS_MEMORY_SIZE, MPSS_MEMORY);+ announce("read the modem's four file systems from the phone's storage into memory of ours (rmtfs below)");+ {+ unsigned loaded = rmtfs_load();++ printf("mpss: %u: %u of the modem's 4 file systems are here%s\n", step, loaded, loaded == 4 ? "" :+ ": WITHOUT THEM ALL THE MODEM OFFERS NO SERVICES BUT ITS OWN (DMS, NAS... stay away)");+ servreg_load();+ }+ return true;+fail:+ image_forget();+ image.segments = 0;+ return false;+}++/* ---- step 3: the mailbox --------------------------------------------------------------------------- */++static bool+step_mailbox(void)+{+ bool open;++ announce("open the mailbox to the always-on processor, unless it is open: words of its message memory, and its doorbell");+ lck_mtx_lock(&redwood_aop_mailbox_lock);+ open = mailbox_open();+ lck_mtx_unlock(&redwood_aop_mailbox_lock);+ return open;+}++/* ---- TrustZone, and stopping the DSP --------------------------------------------------------------- */++#define SCM_64 (1ull << 30)+#define SCM_SIP (2ull << 24)+#define SCM_CALL(service, command) ((uint64_t)(service) << 8 | (command))+#define SCM_PAS_INIT_IMAGE SCM_CALL(2, 1)+#define SCM_PAS_MEM_SETUP SCM_CALL(2, 2)+#define SCM_PAS_AUTH_AND_RESET SCM_CALL(2, 5)+#define SCM_PAS_SHUTDOWN SCM_CALL(2, 6)+#define SCM_ONE_VALUE 1+#define SCM_THREE_VALUES 3+#define SCM_VALUE_AND_BUFFER (2 | 2 << 6)+#define SCM_INTERRUPTED 1++static LCK_MTX_DECLARE(dsp_lock, &mpss_lock_group); /* the DSP being started or stopped */+static bool rails_asked; /* our request for its rails stands (until it asks for its own, or is shut down) */+static bool dsp_started;+static bool stopped_by_its_word;+static uint32_t our_item;+static bool smp2p_agreed, smp2p_restart_acks, smp2p_restart_acked;++static int32_t+scm(const char *what, uint64_t function, uint64_t arguments, uint64_t first, uint64_t second, uint64_t third, uint64_t fourth,+ uint64_t *answer)+{+ uint64_t session = 0;++ function |= SCM_64 | SCM_SIP;+ printf("mpss: %u: about to ask TrustZone %s: smc %#llx, arguments %#llx, %#llx, %#llx\n", step, what, function, first,+ second, third);+ snprintf(last_write, sizeof(last_write), "step %u asked TrustZone %#llx(%#llx, %#llx)", step, function, first, second);+ for (int i = 0; i < 100; i++) {+ register uint64_t x0 __asm__("x0") = i == 0 ? function : SCM_INTERRUPTED;+ register uint64_t x1 __asm__("x1") = arguments;+ register uint64_t x2 __asm__("x2") = first;+ register uint64_t x3 __asm__("x3") = second;+ register uint64_t x4 __asm__("x4") = third;+ register uint64_t x5 __asm__("x5") = fourth;+ register uint64_t x6 __asm__("x6") = session;+ register uint64_t x7 __asm__("x7") = 0;++ __asm__ volatile ("smc #0" : "+r" (x0), "+r" (x1), "+r" (x2), "+r" (x3), "+r" (x4), "+r" (x5), "+r" (x6),+ "+r" (x7) : : "x8", "x9", "x10", "x11", "x12", "x13", "x14", "x15", "x16", "x17", "memory");+ session = x6;+ if (x0 != SCM_INTERRUPTED) {+ printf("mpss: %u: TrustZone returns %d, answer %#llx\n", step, (int32_t)x0, x1);+ *answer = x1;+ return (int32_t)x0;+ }+ }+ printf("mpss: %u: TrustZone keeps saying the call was interrupted\n", step);+ return -1;+}++static uint32_t+our_item_find(void)+{+ uint32_t size = 0, at = smem_item(ITEM_SMP2P_OURS, &size);++ return at != 0 && size >= sizeof(struct smp2p) ? at : 0;+}++/* What Linux does when the DSP rings for SMP2P: agree on the features once, and answer a restart. */+static void+dsp_listen(const struct smp2p *theirs)+{+ struct smp2p ours;++ if (our_item == 0 || !smem_peek(our_item, &ours, sizeof(ours)) || ours.magic != SMP2P_MAGIC) {+ return;+ }+ if (!smp2p_agreed) {+ uint32_t features = (ours.version >> 8) & (theirs->version >> 8);++ if ((theirs->version & 0xff) != (ours.version & 0xff)) {+ return;+ }+ if (features != ours.version >> 8) {+ smem_poke(our_item + SMP2P_AT_VERSION, features << 8 | (ours.version & 0xff));+ }+ smp2p_restart_acks = features & SMP2P_RESTART_ACKS;+ smp2p_agreed = true;+ printf("mpss: %u: the DSP and we agree on SMP2P: version %u, features %#x\n", step, ours.version & 0xff, features);+ }+ if (smp2p_restart_acks && ((theirs->flags & SMP2P_RESTART_DONE) != 0) != smp2p_restart_acked) {+ smp2p_restart_acked = !smp2p_restart_acked;+ printf("mpss: %u: the DSP's flags say it has restarted (%#x): answered in our flags (bit 1 = %u), and its SMP2P doorbell\n",+ step, theirs->flags, smp2p_restart_acked);+ smem_poke(our_item + SMP2P_AT_FLAGS, (ours.flags & ~SMP2P_RESTART_ACK) | (smp2p_restart_acked ? SMP2P_RESTART_ACK : 0));+ settle();+ wr(ipcc.base + IPCC_SEND, IPCC_MPSS << 16 | IPCC_SMP2P);+ }+}++/* Linux's qcom_q6v5_request_stop(): our bit "stop", the DSP's "stop ack" waited for, our bit taken back. */+static bool+dsp_ask_to_stop(uint32_t bound_ms)+{+ struct smp2p ours, theirs;+ uint32_t at = our_item_find(), state, value_at;+ uint64_t asked, now;+ int entry = -1;++ if (at == 0 || !smem_peek(at, &ours, sizeof(ours)) || ours.magic != SMP2P_MAGIC || (ours.version & 0xff) != SMP2P_VERSION ||+ (entry = smp2p_entry(&ours, OUR_ENTRY_STOP)) < 0) {+ printf("mpss: %u: the DSP is not asked to stop first: there is no SMP2P item of ours to ask it with\n", step);+ return false;+ }+ if (!dsp_words(&theirs) || smp2p_entry(&theirs, "slave-kernel") < 0) {+ printf("mpss: %u: the DSP is not asked to stop first: its item has no entry to answer in\n", step);+ return false;+ }+ state = dsp_state(&theirs);+ if (state & DSP_STOP_ACK) {+ printf("mpss: %u: the DSP is not asked to stop: its entry says %#x, that it has stopped when asked before\n", step, state);+ return true;+ }+ if (!(state & DSP_READY) || (state & DSP_FATAL)) {+ printf("mpss: %u: the DSP is not asked to stop first: its entry says %#x, not \"up\"\n", step, state);+ return false;+ }+ need(&ipcc);+ value_at = at + SMP2P_AT_VALUE((uint32_t)entry);+ printf("mpss: %u: about to ask the DSP to stop: bit 0 of our entry \"%s\" at %#lx, and its SMP2P doorbell\n", step,+ OUR_ENTRY_STOP, SMEM_PHYS + value_at);+ snprintf(last_write, sizeof(last_write), "step %u asked the modem to stop", step);+ smem_poke(value_at, ours.entry[entry].value | DSP_STOP);+ settle();+ wr(ipcc.base + IPCC_SEND, IPCC_MPSS << 16 | IPCC_SMP2P);+ asked = now = now_us();+ while (now - asked < bound_ms * 1000ull) {+ if (dsp_words(&theirs) && (dsp_state(&theirs) & (DSP_STOP_ACK | DSP_FATAL))) {+ break;+ }+ sleep_us(200);+ now = now_us();+ }+ state = dsp_words(&theirs) ? dsp_state(&theirs) : 0;+ printf("mpss: %u: %s after %llu.%03llu ms; its entry says %#x\n", step, state & DSP_STOP_ACK ?+ "THE MODEM ACKNOWLEDGES THE STOP" : "THE MODEM DOES NOT ACKNOWLEDGE THE STOP", (now - asked) / 1000,+ (now - asked) % 1000, state);+ smem_poke(value_at, ours.entry[entry].value & ~DSP_STOP);+ settle();+ wr(ipcc.base + IPCC_SEND, IPCC_MPSS << 16 | IPCC_SMP2P);+ return (state & DSP_STOP_ACK) != 0;+}++static void link_stop(void);++/* Asked to stop, then shut down by TrustZone, then the always-on processor told that the image is gone. */+static bool+dsp_shutdown(const char *why)+{+ uint64_t answer = ~0ull;+ int32_t result;++ printf("mpss: %u: %s: the modem is asked to stop, then TrustZone to shut it down\n", step, why);+ if (!leaving) {+ link_stop();+ }+ stopped_by_its_word = dsp_ask_to_stop(leaving ? 2000 : 5000);+ result = scm("to shut the modem down", SCM_PAS_SHUTDOWN, SCM_ONE_VALUE, PAS_MPSS, 0, 0, 0, &answer);+ if (result != 0 || answer != 0) {+ printf("mpss: %u: TRUSTZONE DOES NOT SHUT THE MODEM DOWN: the call returns %d, its answer is %#llx\n", step, result,+ answer);+ return false;+ }+ printf("mpss: %u: THE MODEM IS SHUT DOWN\n", step);+ if (ipa_loaded) {+ ipa_firmware_stop();+ ipa_loaded = false;+ }+ if (!leaving) {+ rmtfs_share(false);+ }+ need(&ipcc);+ wr(ipcc.base + IPCC_DISABLE, IPCC_MPSS << 16 | IPCC_SMP2P);+ doorbells();+ dsp_started = false;+ mailbox_send("{class: image, res: load_state, name: modem, val: off}");+ if (rails_asked) {+ rails(0);+ rails_asked = false;+ }+ return true;+}++/* ---- step 4: TrustZone takes the header ------------------------------------------------------------ */++static bool+step_header(void)+{+ struct smp2p words;+ vm_offset_t page = 0;+ uint64_t answer = ~0ull;+ int32_t result;++ if (image.segments == 0) {+ return false;+ }+ if (dsp_words(&words)) {+ show_words("the DSP has left words in the shared memory", &words);+ announce("ask TrustZone to shut down an modem that a kernel before this one may have left running");+ if (!dsp_shutdown("it may be running")) {+ return false;+ }+ if (!stopped_by_its_word && !option("modem_again")) {+ printf("mpss: %u: THE MODEM IS NOT STARTED AGAIN: it was shut down without having acknowledged a stop, and "+ "with the Wi-Fi's DSP a start after that reset the phone (modem_again=1 to try all the same)\n", step);+ return false;+ }+ } else {+ printf("mpss: %u: the DSP has left nothing in the shared memory: it has not run since the phone was started\n", step);+ }++ announce("hand the image's header to TrustZone to check its signature (image 4), in a page of ours");+ if (kmem_alloc(kernel_map, &page, PAGE_SIZE, KMA_KOBJECT | KMA_ZERO | KMA_DATA, VM_KERN_MEMORY_IOKIT) != KERN_SUCCESS) {+ printf("mpss: %u: no memory\n", step);+ return false;+ }+ memcpy((void *)page, image.header, image.header_size);+ for (vm_offset_t line = page; line < page + PAGE_SIZE; line += 64) {+ __asm__ volatile ("dc civac, %0" : : "r" (line));+ }+ settle();+ result = scm("to take the modem's header", SCM_PAS_INIT_IMAGE, SCM_VALUE_AND_BUFFER, PAS_MPSS, kvtophys(page), 0, 0, &answer);+ if (result != 0 || answer != 0) {+ printf("mpss: %u: TRUSTZONE DOES NOT TAKE THE HEADER: the call returns %d, its answer is %#llx\n", step, result, answer);+ return false;+ }+ printf("mpss: %u: TRUSTZONE ACCEPTS THE MODEM'S SIGNATURE FROM THIS KERNEL\n", step);++ announce("tell TrustZone where the image will be: the whole of the modem's memory, as Linux does");+ result = scm("where the image will be", SCM_PAS_MEM_SETUP, SCM_THREE_VALUES, PAS_MPSS, MPSS_MEMORY, MPSS_MEMORY_SIZE, 0, &answer);+ if (result != 0 || answer != 0) {+ printf("mpss: %u: TRUSTZONE DOES NOT TAKE THE PLACE: the call returns %d, its answer is %#llx\n", step, result, answer);+ return false;+ }+ printf("mpss: %u: TrustZone takes the place: %#x bytes at %#x (the image takes %#x of them)\n", step, MPSS_MEMORY_SIZE, MPSS_MEMORY,+ image.size);+ return true;+}++/* ---- step 5: the image, in the DSP's memory --------------------------------------------------------- */++static bool image_written;++static uint64_t+segment_word(unsigned s, uint32_t at)+{+ uint64_t word = 0;++ for (unsigned b = 0; b < 8 && at + b < image.segment[s].file_size; b++) {+ word |= (uint64_t)image.segment[s].file[at + b] << (8 * b);+ }+ return word;+}++/* A segment written at its place: words of eight where it begins on one, bytes where it does not or at its end. */+static void+segment_put(volatile uint8_t *dsp_memory, unsigned s)+{+ uint32_t at = image.segment[s].at, size = image.segment[s].memory_size, i = 0;++ if ((at & 7) == 0) {+ volatile uint64_t *to = (volatile uint64_t *)(dsp_memory + at);++ for (; i + 8 <= size; i += 8) {+ to[i / 8] = segment_word(s, i);+ }+ }+ for (; i < size; i++) {+ dsp_memory[at + i] = i < image.segment[s].file_size ? image.segment[s].file[i] : 0;+ }+}++/* ... and read back: how many words of eight (or bytes, where it is not written by words) differ. */+static uint64_t+segment_check(volatile uint8_t *dsp_memory, unsigned s)+{+ uint32_t at = image.segment[s].at, size = image.segment[s].memory_size, i = 0;+ uint64_t wrong = 0;++ if ((at & 7) == 0) {+ volatile uint64_t *from = (volatile uint64_t *)(dsp_memory + at);++ for (; i + 8 <= size; i += 8) {+ wrong += from[i / 8] != segment_word(s, i);+ }+ }+ for (; i < size; i++) {+ wrong += dsp_memory[at + i] != (i < image.segment[s].file_size ? image.segment[s].file[i] : 0);+ }+ return wrong;+}++static bool+step_image(void)+{+ volatile uint8_t *dsp_memory;+ uint64_t written = 0, cleared = 0, wrong = 0, began;++ if (image.segments == 0) {+ return false;+ }+ announce("put the image's segments into the modem's memory (0x8b800000, 246 MB: memory of nobody's until TrustZone starts "+ "the modem), with zeroes where a segment is longer than its file, and read them back");+ dsp_memory = (volatile uint8_t *)ml_io_map_unmappable(MPSS_MEMORY, image.size, VM_WIMG_WCOMB);+ if (dsp_memory == NULL) {+ printf("mpss: %u: the modem's memory cannot be mapped (%#x bytes)\n", step, image.size);+ return false;+ }+ snprintf(last_write, sizeof(last_write), "step %u wrote the image at %#x", step, MPSS_MEMORY);+ began = now_us();+ for (unsigned s = 0; s < image.segments; s++) {+ segment_put(dsp_memory, s);+ written += image.segment[s].file_size;+ cleared += image.segment[s].memory_size - image.segment[s].file_size;+ }+ settle();+ for (unsigned s = 0; s < image.segments; s++) {+ wrong += segment_check(dsp_memory, s);+ }+ printf("mpss: %u: %u segments, %llu bytes written and %llu cleared in %llu ms; read back, %llu words or bytes differ\n", step,+ image.segments, written, cleared, (now_us() - began) / 1000, wrong);+ settle();+ ml_io_unmap((vm_offset_t)dsp_memory, image.size);+ image_written = wrong == 0;+ image_forget(); /* (it is in the modem's memory now: what was read of the root, given back) */+ return image_written;+}++/* ---- step 6: our SMP2P item ------------------------------------------------------------------------ */++static struct block smem_lock = { "shared memory's lock", 0x01f43000, 0x1000 };+#define LOCK_OWNER 0x000+#define LOCK_US 1u++/* Our item made in the partition, as Linux's qcom_smem_alloc_private() makes one (redwood_wifi.c: our_item_make()). */+static bool+our_item_make(uint32_t partition, uint32_t item, uint32_t bytes)+{+ struct smem_partition header;+ struct smem_entry entry;+ const uint32_t item_size = (bytes + 7) & ~7u, need_bytes = sizeof(entry) + item_size;+ uint32_t owner = ~0u, at;+ bool made = false;++ lck_mtx_lock(&redwood_smem_alloc_lock); /* (the Wi-Fi's start allocates under the same hardware lock) */+ need(&smem_lock);+ printf("mpss: %u: about to take the lock (%#x), write an entry for item %u with %u bytes behind it at the partition's mark, "+ "move the mark by %u, and release the lock\n", step, smem_lock.phys + LOCK_OWNER, item, item_size, need_bytes);+ snprintf(last_write, sizeof(last_write), "step %u took the shared memory's lock", step);+ for (uint32_t ms = 0; ms <= 1000; ms++) {+ wr(smem_lock.base + LOCK_OWNER, LOCK_US);+ owner = rd(smem_lock.base + LOCK_OWNER);+ if (owner == LOCK_US) {+ break;+ }+ sleep_ms(1);+ }+ if (owner != LOCK_US) {+ printf("mpss: %u: THE LOCK IS HELD BY PROCESSOR %u AND STAYS SO FOR A SECOND: nothing is written\n", step, owner);+ lck_mtx_unlock(&redwood_smem_alloc_lock);+ return false;+ }+ if (!smem_peek(partition, &header, sizeof(header)) || header.free_uncached > header.free_cached ||+ header.free_cached > header.size) {+ printf("mpss: %u: the partition's marks make no sense: nothing is written\n", step);+ goto out;+ }+ for (at = sizeof(header); at < header.free_uncached;) {+ if (!smem_peek(partition + at, &entry, sizeof(entry)) || entry.canary != SMEM_CANARY || entry.item == item ||+ entry.size > header.size) {+ break;+ }+ at += sizeof(entry) + entry.padding_header + entry.size;+ }+ if (at != header.free_uncached) {+ printf("mpss: %u: the partition's entries do not end at its mark (+%#x, the mark is +%#x), or the item has appeared: "+ "nothing is written\n", step, at, header.free_uncached);+ goto out;+ }+ if (at + need_bytes > header.free_cached) {+ printf("mpss: %u: no room: %u bytes wanted, %u free: nothing is written\n", step, need_bytes, header.free_cached - at);+ goto out;+ }+ smem_poke(partition + at, SMEM_CANARY | item << 16);+ smem_poke(partition + at + 4, item_size);+ smem_poke(partition + at + 8, item_size - bytes);+ for (uint32_t z = 0; z < item_size; z += 4) {+ smem_poke(partition + at + (uint32_t)sizeof(entry) + z, 0);+ }+ smem_poke(partition + at + 12, 0);+ settle();+ smem_poke(partition + __offsetof(struct smem_partition, free_uncached), at + need_bytes);+ settle();+ made = true;+out:+ wr(smem_lock.base + LOCK_OWNER, 0);+ printf("mpss: %u: the lock is released (it reads %u)\n", step, rd(smem_lock.base + LOCK_OWNER));+ lck_mtx_unlock(&redwood_smem_alloc_lock);+ return made;+}++static void glink_reset(void);++static bool+step_our_item(void)+{+ struct smem_partition header;+ struct smp2p ours;+ uint32_t cacheline = 0, size = 0, partition, name[4] = { 0 };++ announce("look for an SMP2P item of ours (428) in our partition with the modem, make it if there is none, and fill it in");+ partition = smem_partition(SMEM_US, SMEM_MPSS, &cacheline, &size);+ if (partition == 0 || !smem_peek(partition, &header, sizeof(header))) {+ printf("mpss: %u: there is no such partition\n", step);+ return false;+ }+ our_item = our_item_find();+ if (our_item != 0) {+ printf("mpss: %u: item %u is there, at %#lx: a kernel before this one made it, and it is used again\n", step,+ ITEM_SMP2P_OURS, SMEM_PHYS + our_item);+ } else {+ if (!our_item_make(partition, ITEM_SMP2P_OURS, sizeof(struct smp2p))) {+ return false;+ }+ our_item = our_item_find();+ if (our_item == 0) {+ printf("mpss: %u: THE ITEM IS NOT FOUND where it was just made\n", step);+ return false;+ }+ printf("mpss: %u: item %u made, at %#lx\n", step, ITEM_SMP2P_OURS, SMEM_PHYS + our_item);+ }+ /* zeroes; from processor 0 to 1, room for 16 entries, "master-kernel" at zero, feature bit 0; last the version */+ snprintf(last_write, sizeof(last_write), "step %u filled in our SMP2P item", step);+ for (uint32_t at = 0; at < sizeof(ours); at += 4) {+ smem_poke(our_item + at, 0);+ }+ settle();+ smem_poke(our_item, SMP2P_MAGIC);+ smem_poke(our_item + SMP2P_AT_PROCESSORS, SMEM_US | SMEM_MPSS << 16);+ strlcpy((char *)name, OUR_ENTRY_STOP, sizeof(name));+ for (uint32_t word = 0; word < 4; word++) {+ smem_poke(our_item + SMP2P_AT_NAME(0) + 4 * word, name[word]);+ }+ /*+ * A second entry, "ipa" (dt: qcom,smp2p-ipa-1-out of qcom,smp2p-modem): the IPA driver's answer to the+ * modem's question whether the IPA's clock is on, as Linux's ipa_smp2p.c gives it: bit 0 "the answer+ * is valid", bit 1 "the clock is on". The modem, told so, keeps away from the IPA (there is no IPA+ * driver here). Off unless modem_ipa=1 (valid, off) or 3 (on): with the entry the run of 23 October hung.+ */+ {+ uint32_t ipa = 0;++ PE_parse_boot_argn("modem_ipa", &ipa, sizeof(ipa));+ if (ipa != 0) {+ bzero(name, sizeof(name));+ strlcpy((char *)name, "ipa", sizeof(name));+ for (uint32_t word = 0; word < 4; word++) {+ smem_poke(our_item + SMP2P_AT_NAME(1) + 4 * word, name[word]);+ }+ smem_poke(our_item + SMP2P_AT_VALUE(1), ipa);+ printf("mpss: %u: our entry \"ipa\" = %#x\n", step, ipa);+ }+ smem_poke(our_item + SMP2P_AT_COUNTS, 16 | (ipa != 0 ? 2u : 1u) << 16);+ }+ smem_poke(our_item + SMP2P_AT_VERSION, SMP2P_RESTART_ACKS << 8);+ settle();+ smem_poke(our_item + SMP2P_AT_VERSION, SMP2P_RESTART_ACKS << 8 | SMP2P_VERSION);+ settle();+ smp2p_agreed = smp2p_restart_acks = smp2p_restart_acked = false;+ need(&ipcc);+ wr(ipcc.base + IPCC_SEND, IPCC_MPSS << 16 | IPCC_SMP2P);+ if (!smem_peek(our_item, &ours, sizeof(ours)) || ours.magic != SMP2P_MAGIC || smp2p_entry(&ours, OUR_ENTRY_STOP) < 0) {+ printf("mpss: %u: OUR ITEM DOES NOT READ BACK AS WRITTEN\n", step);+ return false;+ }+ show_words("our item reads back", &ours);+ /*+ * The IPA's item (dt of Linux: qcom,smem-id 497, qcom,smem-size 0x2000 of the IPA): memory the+ * application processor's IPA driver allocates in this partition for the modem's filter tables.+ * There is no IPA driver here, but the modem looks for the item when it goes online; made, zeroed,+ * so that it finds it (modem_ipa_item=0: not).+ */+ {+ uint32_t ipa_item = 0, ipa_size = 0; /* (497 made the modem give up before any service: off; modem_ipa_item=497 to try) */++ PE_parse_boot_argn("modem_ipa_item", &ipa_item, sizeof(ipa_item));+ if (ipa_item != 0) {+ if (smem_item(ipa_item, &ipa_size) != 0) {+ printf("mpss: %u: the IPA's item %u is there already (%u bytes)\n", step, ipa_item, ipa_size);+ } else if (our_item_make(partition, ipa_item, 0x2000)) {+ printf("mpss: %u: the IPA's item %u made, 0x2000 bytes, zeroed\n", step, ipa_item);+ } else {+ printf("mpss: %u: the IPA's item %u could not be made\n", step, ipa_item);+ }+ }+ }+ glink_reset();+ return true;+}+++/* ---- the IPA's firmware, loaded before the modem (Android's pil "ipa_fws") -------------------------- */++/*+ * The IPA (the SoC's packet engine, the modem's path for data) has a small firmware of its own,+ * which Android's IPA driver has TrustZone load before the modem starts (dt: qcom,ipa_fws, image+ * 15, yupik_ipa_fws.mdt + .b*, into reserved-memory ipa_fw@8b700000 and ipa_gsi@8b710000). A modem+ * that goes online without it dies in its LTE downlink task (23 October: "EX:kernel:0x0:enl2_dl",+ * a null pointer, twice of two). Nothing else of the IPA is done here yet: no clock, no tables.+ * modem_ipa_fw=0 leaves it out.+ */+#define PAS_IPA 15+#define IPA_FW_MEMORY 0x8b700000u+#define IPA_FW_MEMORY_SIZE 0x1a000u /* ipa_fw (0x10000) and ipa_gsi (0xa000), one after the other */+++/*+ * What the IPA says of itself, read with its clock on: its shared memory register (where the local+ * memory is), the microcontroller's area at the start of that memory (Linux's ipa_uc.c: command,+ * response -- 1 once it has booted --, event, error, hardware state, interface version) and GSI's+ * status (bit 0: the GSI firmware is enabled) and error log, for our execution environment.+ */+static struct block ipa_regs = { "ipa", 0x01e40000, 0x1000 };+static struct block ipa_sram = { "ipa-shared", 0x01e50000, 0x4000 }; /* the local memory: 0x3000 bytes (SHARED_MEM_SIZE) */+static struct block gsi_ee0 = { "gsi-ee0", 0x01e16000, 0x1000 };++static void+ipa_status(const char *when)+{+ uint32_t shared, uc0, uc3, uc5, uc7, uc8, gsi, error;++ need(&ipa_regs);+ need(&ipa_sram);+ need(&gsi_ee0);+ shared = rd(ipa_regs.base + 0x54);+ uc0 = rd(ipa_sram.base + 0x00); /* command (byte 0) */+ uc3 = rd(ipa_sram.base + 0x0c); /* response (byte 0) */+ uc5 = rd(ipa_sram.base + 0x14); /* event (byte 0) */+ uc7 = rd(ipa_sram.base + 0x1c); /* first error address */+ uc8 = rd(ipa_sram.base + 0x20); /* hw_state (byte 0), warning counter (byte 1) */+ gsi = rd(gsi_ee0.base + 0x000); /* GSI_STATUS */+ error = rd(gsi_ee0.base + 0x200); /* ERROR_LOG */+ tell("mpss: ipa: %s: shared memory %#x (offset %u, %u bytes); uC command %#x response %#x (1: booted) event %#x error address %#x "+ "hw_state %u warnings %u, interface %#x; GSI status %#x (%s), error log %#x\n", when, shared, 8 * (shared >> 16),+ 8 * (shared & 0xffff), uc0 & 0xff, uc3 & 0xff, uc5 & 0xff, uc7, uc8 & 0xff, (uc8 >> 8) & 0xff, rd(ipa_sram.base + 0x24) & 0xffff,+ gsi, gsi & 1 ? "ENABLED" : "NOT ENABLED", error);+ ipa_mem_offset = 8 * (shared >> 16);+}+++/*+ * The IPA's local memory as Linux's ipa_mem_config() and ipa_mem_setup() leave it before the modem+ * is told of it: zeroed, with "canaries" (0xdeadbeef) in the words before the regions that have+ * them (IPA v4.11: ipa_data-v4.11.c), and the register that says where the modem's processing+ * contexts begin. The modem reads this memory for the microcontroller's state and its own tables;+ * what it finds there after power-on is whatever was left.+ */+static void+ipa_memory_init(void)+{+ static const struct {+ uint16_t offset, canaries;+ } regions[] = {+ /* Android's _ipa_init_sram_v3() for IPA 4.11 (its ipa_4_11_mem_part): two before each table and the modem's+ * headers and processing contexts, one 16 bytes before the NAT table (0x17a0) and the modem's quota statistics+ * (0x1d00), two before the modem's region (0x27d8), one at the end */+ { 0x0288, 2 }, { 0x0308, 2 }, { 0x0388, 2 }, { 0x0408, 2 }, { 0x0488, 2 }, { 0x0508, 2 }, { 0x0588, 2 }, { 0x0608, 2 },+ { 0x0688, 2 }, { 0x0ad0, 2 }, { 0x1794, 1 }, { 0x1cf4, 1 }, { 0x27d8, 2 }, { 0x3000, 1 },+ };+ const uint32_t size = 0x3000;++ need(&ipa_sram);+ need(&ipa_regs);+ /* from 0x288: the microcontroller's own area (its shared words at 0, its info at 0x80) is left alone, as Linux leaves it */+ for (uint32_t at = 0x288; at < size; at += 4) {+ wr(ipa_sram.base + ipa_mem_offset + at, 0);+ }+ for (unsigned r = 0; r < sizeof(regions) / sizeof(regions[0]); r++) {+ for (unsigned c = 1; c <= regions[r].canaries; c++) {+ wr(ipa_sram.base + ipa_mem_offset + regions[r].offset - 4 * c, 0xdeadbeef);+ }+ }+ wr(ipa_regs.base + 0x1e8, ipa_mem_offset + 0x0ad0); /* LOCAL_PKT_PROC_CNTXT: the modem's processing contexts */+ settle();+ printf("mpss: ipa: its local memory zeroed (0x288..%#x at offset %#x), the canaries written, the processing contexts' base set\n",+ size, ipa_mem_offset);+}+++/*+ * What Android's IPA driver programs into the IPA after its firmware starts and before the modem+ * boots (techpack/dataipa, ipa3_post_init(): ipa3_init_hw(), ipa3_set_resorce_groups_min_max_limits(),+ * ipa3_cfg_route()), as read from its source for IPA 4.11 and from the live registers of the phone's+ * Android (debugfs gen_reg): the QSB masters' outstanding reads and writes, the Qtime timers, the+ * GSI's in-order bits of COMP_CFG, the idle indication for clock division, the resource groups'+ * limits (without them nothing can be processed: the modem's own commands to the IPA would never+ * complete, and its answer to INIT_DRIVER never come), and the default route. Nothing of the+ * application processor's own pipes: there is no data path on this side yet.+ */+#define IPA_COMP_CFG 0x3c+#define IPA_ROUTE 0x48+#define IPA_QSB_MAX_WRITES 0x74+#define IPA_QSB_MAX_READS 0x78+#define IPA_IDLE_INDICATION_CFG 0x240+#define IPA_QTIME_TIMESTAMP_CFG 0x24c+#define IPA_TIMERS_XO_CLK_DIV_CFG 0x250+#define IPA_TIMERS_PULSE_GRAN_CFG 0x254+#define IPA_SRC_RSRC_GRP_01(n) (0x400 + 0x20 * (n)) /* groups 0 and 1: x_min [5:0] x_max [13:8] y_min [21:16] y_max [29:24] */+#define IPA_SRC_RSRC_GRP_23(n) (0x404 + 0x20 * (n))+#define IPA_DST_RSRC_GRP_01(n) (0x500 + 0x20 * (n))+#define IPA_DST_RSRC_GRP_23(n) (0x504 + 0x20 * (n))+#define RSRC(xmin, xmax, ymin, ymax) ((uint32_t)(xmin) | (uint32_t)(xmax) << 8 | (uint32_t)(ymin) << 16 | (uint32_t)(ymax) << 24)++static void+ipa_hardware_init(void)+{+ static const uint8_t source[5] = { 6, 8, 18, 2, 15 }; /* packet contexts, descriptor lists, descriptor buffers, HPS DMARs, ACK entries: group 0 (UL/DL) */+ uint32_t value;++ need(&ipa_regs);+ /* QSB: 12 outstanding writes, 13 reads of 120 beats on the DDR master (ipa3_qmb_outstanding[IPA_4_11]) */+ value = rd(ipa_regs.base + IPA_QSB_MAX_WRITES);+ wr(ipa_regs.base + IPA_QSB_MAX_WRITES, (value & ~0xfu) | 12);+ value = rd(ipa_regs.base + IPA_QSB_MAX_READS);+ wr(ipa_regs.base + IPA_QSB_MAX_READS, (value & ~0x00ff000fu) | 13 | 120u << 16);+ /* Qtime: the divider off, the timestamps and pulse generators as Android leaves them, the divider 192 on */+ wr(ipa_regs.base + IPA_TIMERS_XO_CLK_DIV_CFG, 0);+ wr(ipa_regs.base + IPA_QTIME_TIMESTAMP_CFG, 0x180e8e);+ wr(ipa_regs.base + IPA_TIMERS_PULSE_GRAN_CFG, 0x123);+ wr(ipa_regs.base + IPA_TIMERS_XO_CLK_DIV_CFG, 0xbf);+ wr(ipa_regs.base + IPA_TIMERS_XO_CLK_DIV_CFG, 0x800000bf);+ /* COMP_CFG: GSI_MULTI_INORDER_RD_DIS and _WR_DIS */+ value = rd(ipa_regs.base + IPA_COMP_CFG);+ wr(ipa_regs.base + IPA_COMP_CFG, value | 1u << 7 | 1u << 8);+ /* dynamic clock division: enter idle after 256, no constant non-idle */+ wr(ipa_regs.base + IPA_IDLE_INDICATION_CFG, 256);+ /* the resource groups: the sources' five types for group 0 alone; the destinations' data sectors 3 for group 0+ * and 25 for group 2 (DRB_IP), DPS DMARs 2 for group 0 */+ for (unsigned n = 0; n < 5; n++) {+ wr(ipa_regs.base + IPA_SRC_RSRC_GRP_01(n), RSRC(source[n], source[n], 0, 0));+ }+ wr(ipa_regs.base + IPA_DST_RSRC_GRP_01(0), RSRC(3, 3, 0, 0));+ wr(ipa_regs.base + IPA_DST_RSRC_GRP_23(0), RSRC(25, 25, 0, 0));+ wr(ipa_regs.base + IPA_DST_RSRC_GRP_01(1), RSRC(2, 2, 0, 0));+ wr(ipa_regs.base + IPA_DST_RSRC_GRP_23(1), 0);+ /* the default route as Android's: not disabled, default pipe 9, header table 1, fragments to pipe 9, header retained */+ wr(ipa_regs.base + IPA_ROUTE, 0x1120052);+ settle();+ printf("mpss: ipa: the hardware set up as Android's driver does before the modem: QSB %#x %#x, COMP_CFG %#x, idle %#x, route %#x, "+ "source group 0 %u/%u/%u/%u/%u, destination sectors %#x %#x\n", rd(ipa_regs.base + IPA_QSB_MAX_WRITES),+ rd(ipa_regs.base + IPA_QSB_MAX_READS), rd(ipa_regs.base + IPA_COMP_CFG), rd(ipa_regs.base + IPA_IDLE_INDICATION_CFG),+ rd(ipa_regs.base + IPA_ROUTE), rd(ipa_regs.base + IPA_SRC_RSRC_GRP_01(0)) & 0x3f, rd(ipa_regs.base + IPA_SRC_RSRC_GRP_01(1)) & 0x3f,+ rd(ipa_regs.base + IPA_SRC_RSRC_GRP_01(2)) & 0x3f, rd(ipa_regs.base + IPA_SRC_RSRC_GRP_01(3)) & 0x3f,+ rd(ipa_regs.base + IPA_SRC_RSRC_GRP_01(4)) & 0x3f, rd(ipa_regs.base + IPA_DST_RSRC_GRP_01(0)), rd(ipa_regs.base + IPA_DST_RSRC_GRP_23(0)));+}+++/*+ * The IPA reaches memory through the application processors' SMMU (dt of Linux: iommus 0x480 and+ * 0x482 of apps_smmu), where a stream nobody matches faults, and the hypervisor grants no bypass+ * (tried on 24 October: the S2CR stayed "fault"). So, as the Wi-Fi's radio has (redwood_wifi.c,+ * radio_space_on(), where the formats and the order of programming are explained): a translation+ * context of our own, with the first free match register and the lowest free context, and page+ * tables of ours. The IPA, though, is shown memory as it is: one table of 2 MB blocks maps the+ * gigabyte of DDR from 0x80000000 (the shared memory, the modem's memory, our buffers) onto+ * itself, uncached, and one block the SoC's internal memory at 0x146a8000 (Linux's ipa_mem_data:+ * imem), as a device. Put back when the IPA is stopped (modem_ipa_smmu_off=1: not done at all).+ *+ * The hypervisor checks what is written to a match register against the masters it knows (the+ * device tree's "iommus" entries, stream and mask): on 24 October one register with stream 0x480+ * and the mask 0x2 (both streams at once) was refused -- the S2CR took, the SMR read back 0 --+ * while the Wi-Fi's register, whose mask 1 is the device tree's, was granted. The device tree+ * gives the IPA's streams the mask 0, so each takes a match register of its own, with the mask 0,+ * both sending to the one context.+ */+#define IPA_STREAM 0x480u+#define IPA_STREAM_UC 0x482u /* the IPA's microcontroller: its own match register */+#define IPA_SPACE_BASE 0x80000000ull /* a gigabyte from here, as it is */+#define IPA_IMEM 0x14600000ull /* the 2 MB block that holds 0x146a8000 */+static struct block apps_smmu = { "apps smmu", 0x15000000, 0x2000 };+static struct block apps_smmu_second = { "apps smmu", 0, 0x1000 };+static struct block apps_smmu_context = { "apps smmu, the IPA's context", 0, 0x1000 };+#define SMMU_SCR0 0x000+#define SMMU_IDR0 0x020+#define SMMU_IDR1 0x024+#define SMMU_SMR(n) (0x800 + 4 * (n))+#define SMMU_S2CR(n) (0xc00 + 4 * (n))+#define SMMU_SGFSR 0x048+#define SMMU_SGFSYNR0 0x050+#define SMMU_SGFSYNR1 0x054+#define SMMU_CBAR(n) (4 * (n))+#define SMMU_CBAR_STAGE_1 (1u << 16 | 3u << 8 | 0xfu << 12)+#define SMMU_CBA2R(n) (0x800 + 4 * (n))+#define SMMU_CBA2R_64_BIT 1+#define SMMU_CB_SCTLR 0x00+#define SMMU_CB_TCR2 0x10+#define SMMU_CB_TTBR0 0x20+#define SMMU_CB_TTBR1 0x28+#define SMMU_CB_TCR 0x30+#define SMMU_CB_MAIR0 0x38+#define SMMU_CB_MAIR1 0x3c+#define SMMU_CB_FSR 0x58+#define SMMU_CB_FAR 0x60+#define SMMU_CB_FSYNR0 0x68+#define SMMU_CB_TLBIASID 0x610+#define SMMU_CB_TLBSYNC 0x7f0+#define SMMU_CB_TLBSTATUS 0x7f4+#define SMMU_SCTLR_ON (1u << 0 | 1u << 1 | 1u << 2 | 1u << 5 | 1u << 8 | 1u << 12)+#define SMMU_TCR_39_BITS (25u | 1u << 8 | 1u << 10 | 3u << 12 | 1u << 23)+#define SMMU_TCR2 (1u | 1u << 4 | 7u << 15)+#define SMMU_MAIR0 0xf404ff44 /* 0 uncached, 1 cached, 2 device */+#define SMMU_FSR_ALL 0xc00001fe+#define IPA_BLOCK (1ull | 0ull << 2 | 1ull << 6 | 2ull << 8 | 1ull << 10 | 1ull << 11) /* a 2 MB block, uncached, RW, shared */+#define IPA_BLOCK_DEVICE (1ull | 2ull << 2 | 1ull << 6 | 2ull << 8 | 1ull << 10 | 1ull << 11)+#define IPA_TABLE 3ull /* a table entry of the root */+extern lck_mtx_t redwood_smmu_lock;+static struct {+ bool on;+ uint32_t page, pages, match, match_uc, context, old_smr, old_s2cr, old_smr_uc, old_s2cr_uc, old_sctlr;+ vm_offset_t tables; /* three pages: the root, the DDR gigabyte's blocks, the SoC's */+} ipa_space;++static void+ipa_space_faults(const char *when)+{+ uint32_t status, global;++ if (!ipa_space.on) {+ return;+ }+ status = rd(apps_smmu_context.base + SMMU_CB_FSR);+ global = rd(apps_smmu.base + SMMU_SGFSR);+ if ((status & SMMU_FSR_ALL) == 0 && global == 0) {+ printf("mpss: ipa: %s: the SMMU has refused the IPA nothing (context %u)\n", when, ipa_space.context);+ return;+ }+ tell("mpss: ipa: %s: THE SMMU HAS REFUSED A REQUEST OF THE IPA'S: context %u: status %#x, address %#llx, syndrome %#x; the SMMU "+ "as a whole: status %#x, syndromes %#x %#x\n", when, ipa_space.context, status,+ (uint64_t)rd(apps_smmu_context.base + SMMU_CB_FAR + 4) << 32 | rd(apps_smmu_context.base + SMMU_CB_FAR),+ rd(apps_smmu_context.base + SMMU_CB_FSYNR0), global, rd(apps_smmu.base + SMMU_SGFSYNR0), rd(apps_smmu.base + SMMU_SGFSYNR1));+ wr(apps_smmu_context.base + SMMU_CB_FSR, SMMU_FSR_ALL);+}++static bool+ipa_space_on(void)+{+ bool used[256] = { false };+ uint32_t scr0, idr0, idr1, matches, contexts, sees = 1u << 31 | IPA_STREAM, sees_uc = 1u << 31 | IPA_STREAM_UC;+ uint64_t *root, *ddr, *soc, ttbr;++ if (ipa_space.on) {+ return true;+ }+ lck_mtx_lock(&redwood_smmu_lock);+ need(&apps_smmu);+ scr0 = rd(apps_smmu.base + SMMU_SCR0);+ idr0 = rd(apps_smmu.base + SMMU_IDR0);+ idr1 = rd(apps_smmu.base + SMMU_IDR1);+ matches = idr0 & 0xff;+ contexts = idr1 & 0xff;+ ipa_space.page = idr1 >> 31 ? 0x10000 : 0x1000;+ ipa_space.pages = 2u << (idr1 >> 28 & 7);+ if ((scr0 & 1u << 3) || 2 * ipa_space.pages * ipa_space.page != 0x100000) {+ lck_mtx_unlock(&redwood_smmu_lock);+ printf("mpss: ipa: the SMMU is not of the shape this knows (scr0 %#x, idr1 %#x)\n", scr0, idr1);+ return false;+ }+ ipa_space.match = ipa_space.match_uc = ~0u;+ for (uint32_t n = 0; n < matches && SMMU_SMR(n) < 0xc00; n++) {+ uint32_t smr = rd(apps_smmu.base + SMMU_SMR(n)), s2cr = rd(apps_smmu.base + SMMU_S2CR(n));++ if (smr >> 31) {+ used[s2cr & 0xff] = true;+ } else if (ipa_space.match == ~0u) {+ ipa_space.match = n;+ ipa_space.old_smr = smr;+ ipa_space.old_s2cr = s2cr;+ } else if (ipa_space.match_uc == ~0u) {+ ipa_space.match_uc = n;+ ipa_space.old_smr_uc = smr;+ ipa_space.old_s2cr_uc = s2cr;+ }+ }+ for (ipa_space.context = 0; ipa_space.context < contexts && used[ipa_space.context]; ipa_space.context++) {+ }+ if (ipa_space.match == ~0u || ipa_space.match_uc == ~0u || ipa_space.context >= contexts) {+ lck_mtx_unlock(&redwood_smmu_lock);+ printf("mpss: ipa: the SMMU has not two match registers or no context left\n");+ return false;+ }+ apps_smmu_second.phys = apps_smmu.phys + ipa_space.page;+ apps_smmu_context.phys = apps_smmu.phys + (ipa_space.pages + ipa_space.context) * ipa_space.page;+ need(&apps_smmu_second);+ need(&apps_smmu_context);+ ipa_space.old_sctlr = rd(apps_smmu_context.base + SMMU_CB_SCTLR);+ if (ipa_space.old_sctlr & 1) {+ lck_mtx_unlock(&redwood_smmu_lock);+ printf("mpss: ipa: context %u of the SMMU is switched on although no stream goes through it: left alone\n", ipa_space.context);+ return false;+ }+ if (ipa_space.tables == 0 &&+ kmem_alloc(kernel_map, &ipa_space.tables, 3 * 4096, KMA_KOBJECT | KMA_ZERO | KMA_DATA, VM_KERN_MEMORY_IOKIT) != KERN_SUCCESS) {+ lck_mtx_unlock(&redwood_smmu_lock);+ return false;+ }+ root = (uint64_t *)ipa_space.tables;+ ddr = (uint64_t *)(ipa_space.tables + 4096);+ soc = (uint64_t *)(ipa_space.tables + 8192);+ bzero(root, 3 * 4096);+ for (unsigned i = 0; i < 512; i++) {+ ddr[i] = (IPA_SPACE_BASE + (uint64_t)i * 0x200000) | IPA_BLOCK;+ }+ soc[(IPA_IMEM >> 21) & 0x1ff] = IPA_IMEM | IPA_BLOCK_DEVICE;+ root[IPA_SPACE_BASE >> 30] = kvtophys((vm_offset_t)ddr) | IPA_TABLE;+ root[IPA_IMEM >> 30] = kvtophys((vm_offset_t)soc) | IPA_TABLE;+ FlushPoC_DcacheRegion(ipa_space.tables, 3 * 4096);+ settle();+ ttbr = kvtophys(ipa_space.tables) | (uint64_t)ipa_space.context << 48;+ printf("mpss: ipa: THE SMMU IS ABOUT TO BE WRITTEN: context %u gets our tables (root %#llx: DDR from %#llx as it is, 1 GB, and "+ "the SoC's memory at %#llx) and match registers %u and %u send streams %#x and %#x through it (before: smr %#x s2cr %#x, "+ "smr %#x s2cr %#x, sctlr %#x)\n", ipa_space.context, ttbr & 0x0000fffffffff000ull, IPA_SPACE_BASE, IPA_IMEM, ipa_space.match,+ ipa_space.match_uc, IPA_STREAM, IPA_STREAM_UC, ipa_space.old_smr, ipa_space.old_s2cr, ipa_space.old_smr_uc, ipa_space.old_s2cr_uc,+ ipa_space.old_sctlr);+ wr(apps_smmu_context.base + SMMU_CB_SCTLR, 0);+ wr(apps_smmu_second.base + SMMU_CBA2R(ipa_space.context), SMMU_CBA2R_64_BIT);+ wr(apps_smmu_second.base + SMMU_CBAR(ipa_space.context), SMMU_CBAR_STAGE_1);+ wr(apps_smmu_context.base + SMMU_CB_TCR2, SMMU_TCR2);+ wr(apps_smmu_context.base + SMMU_CB_TCR, SMMU_TCR_39_BITS);+ wr(apps_smmu_context.base + SMMU_CB_TTBR0, (uint32_t)ttbr);+ wr(apps_smmu_context.base + SMMU_CB_TTBR0 + 4, (uint32_t)(ttbr >> 32));+ wr(apps_smmu_context.base + SMMU_CB_TTBR1, 0);+ wr(apps_smmu_context.base + SMMU_CB_TTBR1 + 4, 0);+ wr(apps_smmu_context.base + SMMU_CB_MAIR0, SMMU_MAIR0);+ wr(apps_smmu_context.base + SMMU_CB_MAIR1, 0);+ wr(apps_smmu_context.base + SMMU_CB_FSR, SMMU_FSR_ALL);+ wr(apps_smmu_context.base + SMMU_CB_TLBIASID, ipa_space.context);+ wr(apps_smmu_context.base + SMMU_CB_TLBSYNC, 1);+ for (unsigned ms = 0; rd(apps_smmu_context.base + SMMU_CB_TLBSTATUS) & 1; ms++) {+ if (ms == 100) {+ lck_mtx_unlock(&redwood_smmu_lock);+ printf("mpss: ipa: the SMMU does not finish forgetting old translations\n");+ return false;+ }+ sleep_ms(1);+ }+ wr(apps_smmu_context.base + SMMU_CB_SCTLR, SMMU_SCTLR_ON);+ ipa_space.on = true;+ wr(apps_smmu.base + SMMU_S2CR(ipa_space.match), ipa_space.context);+ wr(apps_smmu.base + SMMU_SMR(ipa_space.match), sees);+ wr(apps_smmu.base + SMMU_S2CR(ipa_space.match_uc), ipa_space.context);+ wr(apps_smmu.base + SMMU_SMR(ipa_space.match_uc), sees_uc);+ if (rd(apps_smmu.base + SMMU_SMR(ipa_space.match)) != sees ||+ (rd(apps_smmu.base + SMMU_S2CR(ipa_space.match)) & 0x300ff) != ipa_space.context ||+ rd(apps_smmu.base + SMMU_SMR(ipa_space.match_uc)) != sees_uc ||+ (rd(apps_smmu.base + SMMU_S2CR(ipa_space.match_uc)) & 0x300ff) != ipa_space.context ||+ !(rd(apps_smmu_context.base + SMMU_CB_SCTLR) & 1) || rd(apps_smmu_context.base + SMMU_CB_TTBR0) != (uint32_t)ttbr) {+ lck_mtx_unlock(&redwood_smmu_lock);+ printf("mpss: ipa: THE SMMU DID NOT TAKE WHAT WAS WRITTEN: streams %#x and %#x are not both going through context %u (smr %#x "+ "s2cr %#x, smr %#x s2cr %#x, sctlr %#x)\n", IPA_STREAM, IPA_STREAM_UC, ipa_space.context,+ rd(apps_smmu.base + SMMU_SMR(ipa_space.match)), rd(apps_smmu.base + SMMU_S2CR(ipa_space.match)),+ rd(apps_smmu.base + SMMU_SMR(ipa_space.match_uc)), rd(apps_smmu.base + SMMU_S2CR(ipa_space.match_uc)),+ rd(apps_smmu_context.base + SMMU_CB_SCTLR));+ return false;+ }+ lck_mtx_unlock(&redwood_smmu_lock);+ tell("mpss: ipa: THE IPA'S STREAMS GO THROUGH CONTEXT %u OF THE SMMU, with our tables (match registers %u and %u)\n", ipa_space.context,+ ipa_space.match, ipa_space.match_uc);+ return true;+}++static void+ipa_space_off(void)+{+ if (!ipa_space.on) {+ return;+ }+ ipa_space_faults("at the end");+ lck_mtx_lock(&redwood_smmu_lock);+ wr(apps_smmu.base + SMMU_SMR(ipa_space.match), ipa_space.old_smr);+ wr(apps_smmu.base + SMMU_S2CR(ipa_space.match), ipa_space.old_s2cr);+ wr(apps_smmu.base + SMMU_SMR(ipa_space.match_uc), ipa_space.old_smr_uc);+ wr(apps_smmu.base + SMMU_S2CR(ipa_space.match_uc), ipa_space.old_s2cr_uc);+ wr(apps_smmu_context.base + SMMU_CB_SCTLR, ipa_space.old_sctlr);+ lck_mtx_unlock(&redwood_smmu_lock);+ ipa_space.on = false;+ printf("mpss: ipa: the IPA's streams and context %u of the SMMU are put back as they were found\n", ipa_space.context);+}++static bool+ipa_firmware_load(void)+{+ const uint8_t *header, *segment_file[8] = { NULL };+ uint64_t header_size = 0, segment_size[8] = { 0 }, answer = ~0ull;+ struct elf_segment segments[8];+ volatile uint8_t *memory;+ vm_offset_t page = 0;+ uint32_t count, loaded = 0, highest = 0, size;+ int32_t result;+ bool ok = false;++ announce("turn the IPA's clock on (RPMh), load the IPA's firmware (image 15) into its memory at 0x8b700000 and have TrustZone "+ "start it, as Android does before the modem");+ if (!ipa_clock(true)) {+ printf("mpss: %u: the IPA's clock is not on: its firmware is not loaded (a start without the clock ends the kernel)\n", step);+ return false;+ }+ if (!option("modem_ipa_smmu_off")) {+ ipa_space_on();+ }+ header = root_file("image/yupik_ipa_fws.mdt", &header_size);+ if (header == NULL) {+ printf("mpss: %u: no fw/image/yupik_ipa_fws.mdt: the IPA's firmware is not loaded\n", step);+ return false;+ }+ count = header_size >= 52 ? header[44] | header[45] << 8 : 0;+ if (memcmp(header, "\177ELF\1", 5) != 0 || count == 0 || count > 8 || 52 + 32 * count > header_size) {+ printf("mpss: %u: yupik_ipa_fws.mdt is not such a header (%u segments)\n", step, count);+ goto out;+ }+ /* (the image is relocatable, bit 27 of the flags: its addresses count from the lowest, which goes at the region's start) */+ for (uint32_t i = 0; i < count; i++) {+ memcpy(&segments[i], header + 52 + 32 * i, sizeof(segments[i]));+ }+ {+ uint32_t lowest = ~0u;+ bool relocatable = false;++ for (uint32_t i = 0; i < count; i++) {+ if (segments[i].type == 1 && !MDT_HASH(segments[i].flags) && segments[i].memory_size != 0) {+ relocatable |= (segments[i].flags & (1u << 27)) != 0;+ lowest = segments[i].paddr < lowest ? segments[i].paddr : lowest;+ }+ }+ if (relocatable && lowest != ~0u) {+ for (uint32_t i = 0; i < count; i++) {+ segments[i].paddr = segments[i].paddr - lowest + IPA_FW_MEMORY;+ }+ }+ }+ for (uint32_t i = 0; i < count; i++) {+ char name[48];++ if (segments[i].type != 1 || MDT_HASH(segments[i].flags) || segments[i].memory_size == 0) {+ continue;+ }+ if (segments[i].paddr < IPA_FW_MEMORY || (uint64_t)segments[i].paddr + segments[i].memory_size > IPA_FW_MEMORY + IPA_FW_MEMORY_SIZE) {+ printf("mpss: %u: the IPA's segment %u (%#x, %#x bytes) is not within its memory\n", step, i, segments[i].paddr,+ segments[i].memory_size);+ goto out;+ }+ if (segments[i].file_size != 0) {+ snprintf(name, sizeof(name), "image/yupik_ipa_fws.b%02u", i);+ segment_file[i] = root_file(name, &segment_size[i]);+ if (segment_file[i] == NULL || segment_size[i] != segments[i].file_size) {+ printf("mpss: %u: fw/%s: %s\n", step, name, segment_file[i] == NULL ? "not there" : "not the size the header says");+ goto out;+ }+ }+ if (segments[i].paddr + segments[i].memory_size > highest) {+ highest = segments[i].paddr + segments[i].memory_size;+ }+ loaded++;+ }+ size = ((highest + 0xfff) & ~0xfffu) - IPA_FW_MEMORY;+ printf("mpss: %u: the IPA's image: %u segments to load, %#x bytes from %#x\n", step, loaded, size, IPA_FW_MEMORY);+ if (kmem_alloc(kernel_map, &page, PAGE_SIZE, KMA_KOBJECT | KMA_ZERO | KMA_DATA, VM_KERN_MEMORY_IOKIT) != KERN_SUCCESS) {+ goto out;+ }+ memcpy((void *)page, header, header_size < PAGE_SIZE ? header_size : PAGE_SIZE);+ for (vm_offset_t line = page; line < page + PAGE_SIZE; line += 64) {+ __asm__ volatile ("dc civac, %0" : : "r" (line));+ }+ settle();+ result = scm("to take the IPA's header", SCM_PAS_INIT_IMAGE, SCM_VALUE_AND_BUFFER, PAS_IPA, kvtophys(page), 0, 0, &answer);+ if (result != 0 || answer != 0) {+ printf("mpss: %u: TRUSTZONE DOES NOT TAKE THE IPA'S HEADER (%d, %#llx)\n", step, result, answer);+ goto out;+ }+ result = scm("where the IPA's image will be", SCM_PAS_MEM_SETUP, SCM_THREE_VALUES, PAS_IPA, IPA_FW_MEMORY, size, 0, &answer);+ if (result != 0 || answer != 0) {+ printf("mpss: %u: TRUSTZONE DOES NOT TAKE THE IPA'S PLACE (%d, %#llx)\n", step, result, answer);+ goto out;+ }+ memory = (volatile uint8_t *)ml_io_map_unmappable(IPA_FW_MEMORY, size, VM_WIMG_WCOMB);+ if (memory == NULL) {+ goto out;+ }+ for (uint32_t i = 0; i < count; i++) {+ uint32_t at = segments[i].paddr - IPA_FW_MEMORY;++ if (segments[i].type != 1 || MDT_HASH(segments[i].flags) || segments[i].memory_size == 0) {+ continue;+ }+ for (uint32_t b = 0; b < segments[i].memory_size; b++) {+ memory[at + b] = b < segments[i].file_size ? segment_file[i][b] : 0;+ }+ }+ settle();+ ml_io_unmap((vm_offset_t)memory, size);+ result = scm("to authenticate the IPA's image and start it", SCM_PAS_AUTH_AND_RESET, SCM_ONE_VALUE, PAS_IPA, 0, 0, 0, &answer);+ if (result != 0 || answer != 0) {+ printf("mpss: %u: TRUSTZONE DOES NOT START THE IPA'S FIRMWARE (%d, %#llx)\n", step, result, answer);+ goto out;+ }+ tell("mpss: %u: THE IPA'S FIRMWARE IS LOADED AND STARTED (image 15, %u segments)\n", step, loaded);+ sleep_ms(50);+ if (!option("modem_ipa_quiet")) {+ ipa_status("after its firmware's start");+ }+ if (!option("modem_ipa_raw")) {+ ipa_memory_init();+ ipa_hardware_init();+ }+ ok = true;+out:+ if (page != 0) {+ kmem_free(kernel_map, page, PAGE_SIZE);+ }+ {+ uint8_t *gone = __DECONST(uint8_t *, header);++ kfree_data(gone, header_size);+ }+ for (uint32_t i = 0; i < 8; i++) {+ if (segment_file[i] != NULL) {+ uint8_t *gone = __DECONST(uint8_t *, segment_file[i]);++ kfree_data(gone, segment_size[i]);+ }+ }+ return ok;+}++static void+ipa_firmware_stop(void)+{+ uint64_t answer = ~0ull;++ (void)scm("to shut the IPA's firmware down", SCM_PAS_SHUTDOWN, SCM_ONE_VALUE, PAS_IPA, 0, 0, 0, &answer);+ if (!leaving) {+ ipa_space_off();+ ipa_clock(false);+ }+}++/* ---- step 7: the DSP, started ---------------------------------------------------------------------- */++/*+ * Android's kernel (this phone, dmesg): the IPA's firmware first, then modem.mdt and its segments+ * into 0x8b800000..0x9ae00000, "Power/Clock ready", and QRTR's "ready" after that. Here the modem+ * is watched for as the audio DSP is (redwood_adsp.c, step 7): its SMP2P item, its doorbells, its+ * "slave-kernel" word, and its last words in item 421 if it gives up.+ */+static bool+step_start(void)+{+ struct smp2p words, last;+ uint64_t answer = ~0ull, started, now, ready_at = 0, limit;+ uint32_t state = 0, rang, changes = 0, wait_seconds = 10;+ int32_t result;+ bool have, spoke = false, handed_over = false;++ if (!image_written) {+ printf("mpss: %u: the image is not in place\n", step);+ return false;+ }+ PE_parse_boot_argn("modem_wait", &wait_seconds, sizeof(wait_seconds));+ bzero(&last, sizeof(last));+ have = dsp_words(&last);+ announce("listen for the modem's doorbells, ask RPMh for its rails, tell the always-on processor that its image is "+ "loaded, and ask TrustZone to authenticate the image and let the DSP out of reset");+ need(&ipcc);+ for (unsigned i = 0; i < 8 && (rang = rd(ipcc.base + IPCC_RECEIVED)) != IPCC_NOBODY && rang >> 16 == IPCC_MPSS; i++) {+ put(&ipcc, "heard", IPCC_CLEAR, rang);+ }+ put(&ipcc, "listened for", IPCC_ENABLE, IPCC_MPSS << 16 | IPCC_SMP2P);+ put(&ipcc, "listened for", IPCC_ENABLE, IPCC_MPSS << 16 | IPCC_GLINK);+ bzero(&doorbell, sizeof(doorbell));+ lck_mtx_lock(&dsp_lock);+ {+ uint32_t ipa_fw = 1;++ PE_parse_boot_argn("modem_ipa_fw", &ipa_fw, sizeof(ipa_fw));+ ipa_loaded = ipa_fw != 0 && ipa_firmware_load();+ }+ if (option("modem_share_off")) {+ printf("mpss: %u: the buffer is not shared with the modem (modem_share_off)\n", step);+ } else if (!rmtfs_buffer_make() || !rmtfs_share(true)) {+ printf("mpss: %u: THE MODEM GETS NO BUFFER FOR ITS FILES: it is started all the same, and will offer little\n", step);+ }+ if (!rails(384)) {+ lck_mtx_unlock(&dsp_lock);+ return false;+ }+ rails_asked = true;+ if (!mailbox_send("{class: image, res: load_state, name: modem, val: on}")) {+ rails(0);+ rails_asked = false;+ lck_mtx_unlock(&dsp_lock);+ return false;+ }+ result = scm("to authenticate the modem's image and start it", SCM_PAS_AUTH_AND_RESET, SCM_ONE_VALUE, PAS_MPSS, 0, 0, 0,+ &answer);+ doorbell.since = started = now_us();+ image_written = false;+ if (result != 0 || answer != 0) {+ printf("mpss: %u: TRUSTZONE DOES NOT START THE MODEM: the call returns %d, its answer is %#llx\n", step, result,+ answer);+ mailbox_send("{class: image, res: load_state, name: modem, val: off}");+ rails(0);+ rails_asked = false;+ lck_mtx_unlock(&dsp_lock);+ return false;+ }+ dsp_started = true;+ lck_mtx_unlock(&dsp_lock);+ printf("mpss: %u: TRUSTZONE HAS AUTHENTICATED THE IMAGE AND LET THE MODEM OUT OF RESET; watching for %u s\n", step,+ wait_seconds);+ limit = started + wait_seconds * 1000000ull;+ for (;;) {+ now = now_us();+ if (now >= limit || !dsp_started) {+ break;+ }+ doorbells();+ spoke = spoke || doorbell.glink + doorbell.smp2p != 0;+ if (dsp_words(&words) && (!have || memcmp(&words, &last, sizeof(words)) != 0)) {+ char when[64];++ snprintf(when, sizeof(when), "+%llu.%03llu ms: the DSP's item %s", (now - started) / 1000, (now - started) % 1000,+ have ? "changes" : "APPEARS");+ if (changes++ < 40) {+ show_words(when, &words);+ }+ last = words;+ have = spoke = true;+ dsp_listen(&words);+ state = dsp_state(&words);+ if ((state & DSP_READY) && ready_at == 0) {+ ready_at = now;+ limit = now + 3000000ull;+ printf("mpss: %u: THE MODEM SAYS IT IS UP: \"slave-kernel\" says %#x, %llu.%03llu ms after TrustZone started "+ "it\n", step, state, (now - started) / 1000, (now - started) % 1000);+ }+ if ((state & DSP_HANDOVER) && !handed_over) {+ handed_over = true;+ printf("mpss: %u: the DSP says it has asked for its own power (bit 2), %llu.%03llu ms after the start: ours is "+ "given back\n", step, (now - started) / 1000, (now - started) % 1000);+ lck_mtx_lock(&dsp_lock);+ if (rails_asked) {+ rails(0);+ rails_asked = false;+ }+ lck_mtx_unlock(&dsp_lock);+ }+ if (state & DSP_FATAL) {+ printf("mpss: %u: THE MODEM SAYS IT HAS GIVEN UP\n", step);+ show_crash_reason();+ break;+ }+ }+ sleep_us(200);+ }+ printf("mpss: %u: the watch is over after %llu ms; its doorbells heard: GLINK %u, SMP2P %u\n", step,+ (now_us() - started) / 1000, doorbell.glink, doorbell.smp2p);+ show_partition();+ if (our_item != 0 && smem_peek(our_item, &words, sizeof(words))) {+ show_words("our item (428) now", &words);+ }+ {+ /* (what is at the bottom of our partition, entry by entry, as it is now) */+ uint32_t cacheline = 0, size = 0, partition = smem_partition(SMEM_US, SMEM_MPSS, &cacheline, &size);+ struct smem_partition header;+ struct smem_entry entry;++ if (partition != 0 && smem_peek(partition, &header, sizeof(header))) {+ printf("mpss: %u: the partition's header: hosts %u and %u, size %#x, free from the bottom at +%#x, from the top at "+ "+%#x; the table's line size %u\n", step, header.host0, header.host1, header.size, header.free_uncached,+ header.free_cached, cacheline);+ for (uint32_t at = sizeof(header), n = 0; at + sizeof(entry) <= header.size && n < 8; n++) {+ smem_peek(partition + at, &entry, sizeof(entry));+ printf("mpss: %u: +%#x: canary %#x item %u size %u padding %u/%u\n", step, at, entry.canary, entry.item,+ entry.size, entry.padding_data, entry.padding_header);+ if (entry.canary != SMEM_CANARY) {+ break;+ }+ at += sizeof(entry) + entry.padding_header + entry.size;+ }+ }+ }+ if (state & DSP_FATAL) {+ return false;+ }+ if (ready_at == 0) {+ printf("mpss: %u: %s\n", step, spoke ? "THE MODEM RUNS BUT HAS NOT SAID \"UP\"" : "NOT A WORD FROM THE MODEM");+ show_crash_reason();+ }+ return ready_at != 0;+}++/* ---- GLINK (redwood_wifi.c; items 478 to 480 of our partition with the modem) ------------------- */++#define ITEM_GLINK_DESCRIPTOR 478+#define ITEM_GLINK_OUR_QUEUE 479+#define ITEM_GLINK_THEIR_QUEUE 480+#define GLINK_OUR_TAIL 0+#define GLINK_OUR_HEAD 4+#define GLINK_THEIR_TAIL 8+#define GLINK_THEIR_HEAD 12+#define GLINK_RESERVE 16++enum {+ GLINK_VERSION = 0, GLINK_VERSION_ACK = 1, GLINK_OPEN = 2, GLINK_CLOSE = 3, GLINK_OPEN_ACK = 4, GLINK_INTENT = 5,+ GLINK_DONE = 6, GLINK_INTENT_REQUEST = 7, GLINK_INTENT_REQUEST_ACK = 8, GLINK_DATA = 9, GLINK_CLOSE_ACK = 11,+ GLINK_DATA_MORE = 12, GLINK_READ_NOTIFY = 13, GLINK_DONE_REUSE = 14, GLINK_SIGNALS = 15,+};+#define GLINK_VERSION_1 1+#define GLINK_INTENT_REUSE 1u++struct glink_message {+ uint16_t command, param1;+ uint32_t param2;+};+struct glink_packet {+ struct glink_packet *next;+ uint32_t length;+ uint8_t data[];+};++/* The one channel we answer when the modem opens it: IPCRTR, which carries QRTR (dt: qcom,intents of qcom,modem_qrtr). */+enum { CHANNEL_QRTR, CHANNELS };++struct glink_channel {+ const char *name;+ uint32_t ours; /* our number for it */+ struct {+ uint32_t size, count;+ } intents[3];+ uint32_t theirs; /* the DSP's number for it; 0: it has not opened it */+ bool asked, acknowledged, open; /* our open sent; acknowledged; both ways done and our intents told */+ bool request_answered, request_granted; /* our last request for an intent */+ struct {+ uint32_t id, size;+ bool there, in_use;+ } their_intents[64]; /* what we may send into */+ struct {+ uint32_t size, fill;+ uint8_t *data;+ } our_intents[24]; /* what the DSP may send into; an intent's number is its place here + 1 */+ uint32_t our_intent_count;+ struct glink_packet *first, *last; /* received, in order */+};++static LCK_MTX_DECLARE(glink_lock, &mpss_lock_group);+static struct {+ uint32_t descriptor, ours, ours_size, theirs, theirs_size;+ bool up, broken;+ uint32_t features;+ bool agreed;+ struct glink_channel channel[CHANNELS];+ uint32_t said, heard, trace;+ char others[4][33]; /* the channels the DSP opens that nobody here speaks (for the log) */+} glink = {+ .channel = {+ [CHANNEL_QRTR] = { .name = "IPCRTR", .ours = 1, .intents = { { 0x800, 5 }, { 0x2000, 3 }, { 0x4400, 2 } } },+ },+};++static struct glink_channel *+glink_theirs(uint32_t theirs)+{+ for (unsigned c = 0; c < CHANNELS; c++) {+ if (glink.channel[c].theirs != 0 && glink.channel[c].theirs == theirs) {+ return &glink.channel[c];+ }+ }+ return NULL;+}++static bool+glink_find(void)+{+ uint32_t size = 0;++ glink.descriptor = smem_item(ITEM_GLINK_DESCRIPTOR, &size);+ if (glink.descriptor == 0 || size != 32) {+ return false;+ }+ glink.ours = smem_item(ITEM_GLINK_OUR_QUEUE, &glink.ours_size);+ glink.theirs = smem_item(ITEM_GLINK_THEIR_QUEUE, &glink.theirs_size);+ return glink.ours != 0 && glink.theirs != 0 && glink.ours_size >= 1024 && glink.theirs_size >= 1024 &&+ !(glink.ours_size & 7) && !(glink.theirs_size & 7);+}++static void+glink_reset(void)+{+ uint32_t size = 0, descriptor = smem_item(ITEM_GLINK_DESCRIPTOR, &size);++ if (descriptor == 0 || size != 32) {+ printf("mpss: %u: GLINK's descriptor (item %u) is not there: nothing to reset\n", step, ITEM_GLINK_DESCRIPTOR);+ return;+ }+ printf("mpss: %u: GLINK's descriptor at %#lx: our queue read to %#x and written to %#x, the DSP's read to %#x and written "+ "to %#x\n", step, SMEM_PHYS + descriptor, smem_word(descriptor + GLINK_OUR_TAIL), smem_word(descriptor + GLINK_OUR_HEAD),+ smem_word(descriptor + GLINK_THEIR_TAIL), smem_word(descriptor + GLINK_THEIR_HEAD));+ if (smem_word(descriptor + GLINK_OUR_HEAD) != 0 || smem_word(descriptor + GLINK_THEIR_TAIL) != 0) {+ printf("mpss: %u: about to write our two marks there (+4, +8): 0\n", step);+ smem_poke(descriptor + GLINK_OUR_HEAD, 0);+ smem_poke(descriptor + GLINK_THEIR_TAIL, 0);+ settle();+ }+}++static uint32_t+glink_waiting(void)+{+ uint32_t head = smem_word(glink.descriptor + GLINK_THEIR_HEAD), tail = smem_word(glink.descriptor + GLINK_THEIR_TAIL);++ if (head >= glink.theirs_size || tail >= glink.theirs_size) {+ return 0;+ }+ return head >= tail ? head - tail : glink.theirs_size - tail + head;+}++static void+glink_peek(void *to, uint32_t offset, uint32_t count)+{+ uint32_t at = (smem_word(glink.descriptor + GLINK_THEIR_TAIL) + offset) % glink.theirs_size;++ for (uint32_t i = 0; i < count; i++) {+ ((uint8_t *)to)[i] = smem[glink.theirs + (at + i) % glink.theirs_size];+ }+}++static void+glink_advance(uint32_t count)+{+ smem_poke(glink.descriptor + GLINK_THEIR_TAIL, (smem_word(glink.descriptor + GLINK_THEIR_TAIL) + count) % glink.theirs_size);+}++static uint32_t+glink_room(void)+{+ uint32_t head = smem_word(glink.descriptor + GLINK_OUR_HEAD), tail = smem_word(glink.descriptor + GLINK_OUR_TAIL), room;++ if (head >= glink.ours_size || tail >= glink.ours_size) {+ return 0;+ }+ room = tail <= head ? glink.ours_size - head + tail : tail - head;+ return room < GLINK_RESERVE ? 0 : room - GLINK_RESERVE;+}++static bool+glink_put(const void *first, uint32_t first_length, const void *second, uint32_t second_length)+{+ volatile uint8_t *queue = (volatile uint8_t *)(uintptr_t)smem + glink.ours;+ uint32_t head = smem_word(glink.descriptor + GLINK_OUR_HEAD), length = first_length + second_length;+ uint32_t padded = (length + 7) & ~7u;++ if (!glink.up || glink_room() < padded) {+ return false;+ }+ for (uint32_t i = 0; i < padded; i++) {+ queue[(head + i) % glink.ours_size] = i < first_length ? ((const uint8_t *)first)[i] :+ i < length ? ((const uint8_t *)second)[i - first_length] : 0;+ }+ settle();+ smem_poke(glink.descriptor + GLINK_OUR_HEAD, (head + padded) % glink.ours_size);+ settle();+ wr(ipcc.base + IPCC_SEND, IPCC_MPSS << 16 | IPCC_GLINK);+ glink.said++;+ return true;+}++static const char *+glink_name(unsigned command)+{+ static const char *const names[] = {+ "version", "version-ack", "open", "close", "open-ack", "intent", "done", "intent-request", "intent-request-ack",+ "data", "10", "close-ack", "data-more", "read-notify", "done-reuse", "signals"+ };++ return command < sizeof(names) / sizeof(names[0]) ? names[command] : "?";+}++static void+glink_say(unsigned command, uint32_t param1, uint32_t param2, const void *more, uint32_t more_length)+{+ struct glink_message message = { .command = (uint16_t)command, .param1 = (uint16_t)param1, .param2 = param2 };+ bool sent = glink_put(&message, sizeof(message), more, more_length);++ if (glink.trace > 0 || !sent) {+ glink.trace -= glink.trace > 0;+ printf("mpss: glink: > %s %u %#x%s\n", glink_name(command), param1, param2, sent ? "" : ": NO ROOM IN OUR QUEUE, not sent");+ }+}++static bool+glink_offer(struct glink_channel *channel, uint32_t size)+{+ uint32_t words[2] = { size, channel->our_intent_count + 1 }, i = channel->our_intent_count;++ if (i >= sizeof(channel->our_intents) / sizeof(channel->our_intents[0]) || size == 0 || size > 0x10000) {+ return false;+ }+ if (channel->our_intents[i].data == NULL || channel->our_intents[i].size != size) {+ channel->our_intents[i].data = kalloc_data(size, Z_WAITOK | Z_ZERO);+ }+ if (channel->our_intents[i].data == NULL) {+ return false;+ }+ channel->our_intents[i].size = size;+ channel->our_intents[i].fill = 0;+ channel->our_intent_count++;+ glink_say(GLINK_INTENT, channel->ours, 1, words, sizeof(words));+ return true;+}++/* Everything the DSP has put into its queue, taken out and acted on (Linux's qcom_glink_native_rx()). With glink_lock held. */+static void+glink_pump(void)+{+ struct glink_message message;+ struct glink_channel *channel;+ uint32_t words[4];+ char name[33];++ while (glink.up && !glink.broken) {+ uint32_t waiting = glink_waiting(), length = sizeof(message);++ if (waiting < sizeof(message)) {+ return;+ }+ glink_peek(&message, 0, sizeof(message));+ if (message.command == GLINK_OPEN) {+ length += message.param2;+ } else if (message.command == GLINK_INTENT) {+ length += 8 * message.param2;+ } else if (message.command == GLINK_DATA || message.command == GLINK_DATA_MORE) {+ if (waiting < 16) {+ return;+ }+ glink_peek(words, 8, 8);+ length = 16 + words[0];+ }+ length = (length + 7) & ~7u;+ if (length > glink.theirs_size - 8) {+ printf("mpss: glink: A MESSAGE OF %u BYTES (%s %u %#x): not believed, and nothing more is read\n", length,+ glink_name(message.command), message.param1, message.param2);+ glink.broken = true;+ return;+ }+ if (waiting < length) {+ return;+ }+ glink.heard++;+ if (glink.trace > 0 && message.command != GLINK_DATA && message.command != GLINK_DATA_MORE &&+ message.command != GLINK_INTENT && message.command != GLINK_DONE_REUSE) {+ glink.trace--;+ printf("mpss: glink: < %s %u %#x\n", glink_name(message.command), message.param1, message.param2);+ }+ switch (message.command) {+ case GLINK_VERSION:+ if (message.param1 == GLINK_VERSION_1) {+ glink.features &= message.param2;+ }+ glink_advance(length);+ glink_say(GLINK_VERSION_ACK, GLINK_VERSION_1, glink.features, NULL, 0);+ break;+ case GLINK_VERSION_ACK:+ glink_advance(length);+ if (message.param1 == GLINK_VERSION_1 && message.param2 == glink.features) {+ glink.agreed = true;+ } else if (message.param1 != 0) {+ glink.features &= message.param2;+ glink_say(GLINK_VERSION, GLINK_VERSION_1, glink.features, NULL, 0);+ }+ break;+ case GLINK_OPEN:+ bzero(name, sizeof(name));+ glink_peek(name, sizeof(message), message.param2 < sizeof(name) - 1 ? message.param2 : sizeof(name) - 1);+ glink_advance(length);+ channel = NULL;+ for (unsigned c = 0; c < CHANNELS; c++) {+ if (strcmp(name, glink.channel[c].name) == 0) {+ channel = &glink.channel[c];+ }+ }+ if (channel == NULL) {+ printf("mpss: glink: the DSP opens \"%s\" as its channel %u: left unanswered, nobody here speaks it\n", name,+ message.param1);+ for (unsigned i = 0; i < 4; i++) {+ if (glink.others[i][0] == 0) {+ strlcpy(glink.others[i], name, sizeof(glink.others[i]));+ break;+ }+ }+ break;+ }+ printf("mpss: glink: the DSP opens \"%s\" as its channel %u: acknowledged, and opened from our side as channel %u\n",+ name, message.param1, channel->ours);+ channel->theirs = message.param1;+ glink_say(GLINK_OPEN_ACK, channel->theirs, 0, NULL, 0);+ if (!channel->asked) {+ uint32_t size = (uint32_t)strlen(channel->name) + 1;++ bzero(name, sizeof(name));+ strlcpy(name, channel->name, sizeof(name));+ glink_say(GLINK_OPEN, channel->ours, size, name, (size + 7) & ~7u);+ channel->asked = true;+ }+ break;+ case GLINK_OPEN_ACK:+ glink_advance(length);+ for (unsigned c = 0; c < CHANNELS; c++) {+ if (message.param1 == glink.channel[c].ours) {+ glink.channel[c].acknowledged = true;+ }+ }+ break;+ case GLINK_INTENT:+ channel = glink_theirs(message.param1);+ for (uint32_t i = 0; channel != NULL && i < message.param2; i++) {+ glink_peek(words, 8 + 8 * i, 8);+ for (unsigned n = 0; n < sizeof(channel->their_intents) / sizeof(channel->their_intents[0]); n++) {+ if (!channel->their_intents[n].there || channel->their_intents[n].id == words[1]) {+ channel->their_intents[n].there = true;+ channel->their_intents[n].in_use = false;+ channel->their_intents[n].size = words[0];+ channel->their_intents[n].id = words[1];+ break;+ }+ }+ }+ glink_advance(length);+ break;+ case GLINK_DONE:+ case GLINK_DONE_REUSE:+ channel = glink_theirs(message.param1);+ for (unsigned n = 0; channel != NULL && n < sizeof(channel->their_intents) / sizeof(channel->their_intents[0]); n++) {+ if (channel->their_intents[n].there && channel->their_intents[n].id == message.param2) {+ channel->their_intents[n].in_use = false;+ channel->their_intents[n].there = message.command == GLINK_DONE_REUSE;+ }+ }+ glink_advance(length);+ break;+ case GLINK_INTENT_REQUEST:+ glink_advance(length);+ if ((channel = glink_theirs(message.param1)) != NULL) {+ bool granted = glink_offer(channel, message.param2);++ glink_say(GLINK_INTENT_REQUEST_ACK, channel->ours, granted, NULL, 0);+ }+ break;+ case GLINK_INTENT_REQUEST_ACK:+ glink_advance(length);+ for (unsigned c = 0; c < CHANNELS; c++) {+ if (message.param1 == glink.channel[c].theirs || message.param1 == glink.channel[c].ours) {+ glink.channel[c].request_answered = true;+ glink.channel[c].request_granted = message.param2 != 0;+ }+ }+ break;+ case GLINK_DATA:+ case GLINK_DATA_MORE: {+ uint32_t intent = message.param2 - 1;++ channel = glink_theirs(message.param1);+ if (channel == NULL || intent >= channel->our_intent_count ||+ words[0] > channel->our_intents[intent].size - channel->our_intents[intent].fill) {+ printf("mpss: glink: < data of %u bytes on channel %u for intent %u: NOT OURS OR TOO MUCH, dropped\n", words[0],+ message.param1, message.param2);+ glink_advance(length);+ break;+ }+ glink_peek(channel->our_intents[intent].data + channel->our_intents[intent].fill, 16, words[0]);+ channel->our_intents[intent].fill += words[0];+ glink_advance(length);+ if (words[1] == 0) {+ struct glink_packet *packet = kalloc_data(sizeof(*packet) + channel->our_intents[intent].fill, Z_WAITOK);++ if (packet != NULL) {+ packet->next = NULL;+ packet->length = channel->our_intents[intent].fill;+ memcpy(packet->data, channel->our_intents[intent].data, packet->length);+ if (channel->last != NULL) {+ channel->last->next = packet;+ } else {+ channel->first = packet;+ }+ channel->last = packet;+ }+ channel->our_intents[intent].fill = 0;+ glink_say(GLINK_DONE_REUSE, channel->ours, message.param2, NULL, 0);+ }+ break;+ }+ case GLINK_READ_NOTIFY:+ glink_advance(length);+ wr(ipcc.base + IPCC_SEND, IPCC_MPSS << 16 | IPCC_GLINK);+ break;+ case GLINK_CLOSE:+ glink_advance(length);+ glink_say(GLINK_CLOSE_ACK, message.param1, 0, NULL, 0);+ if ((channel = glink_theirs(message.param1)) != NULL) {+ printf("mpss: glink: THE DSP CLOSES \"%s\"\n", channel->name);+ channel->open = false;+ channel->theirs = 0;+ }+ break;+ case GLINK_CLOSE_ACK:+ case GLINK_SIGNALS:+ glink_advance(length);+ break;+ default:+ printf("mpss: glink: A MESSAGE THIS DOES NOT KNOW (%u, %u, %#x): nothing more is read\n", message.command,+ message.param1, message.param2);+ glink.broken = true;+ return;+ }+ for (unsigned c = 0; c < CHANNELS; c++) {+ channel = &glink.channel[c];+ if (channel->theirs != 0 && channel->acknowledged && !channel->open) {+ channel->open = true;+ printf("mpss: glink: THE CHANNEL \"%s\" IS OPEN both ways (ours %u, the DSP's %u); telling the DSP our intents\n",+ channel->name, channel->ours, channel->theirs);+ for (unsigned i = 0; i < 3; i++) {+ for (unsigned n = 0; n < channel->intents[i].count; n++) {+ glink_offer(channel, channel->intents[i].size);+ }+ }+ }+ }+ }+}++/* One whole message to the DSP on a channel; waits up to five seconds for an intent of the DSP's that it fits. */+static int+glink_send(struct glink_channel *channel, const void *data, uint32_t length)+{+ struct {+ struct glink_message message;+ uint32_t size, left;+ } header = { { GLINK_DATA, (uint16_t)channel->ours, 0 }, length, 0 };+ bool asked = false, fits;+ int found = -1;++ lck_mtx_lock(&glink_lock);+ if (glink.up && length + sizeof(header) + 8 > glink.ours_size - GLINK_RESERVE) {+ lck_mtx_unlock(&glink_lock);+ return EMSGSIZE;+ }+ for (unsigned waited = 0;; waited++) {+ glink_pump();+ if (!glink.up || !channel->open || glink.broken) {+ lck_mtx_unlock(&glink_lock);+ return ENOTCONN;+ }+ found = -1;+ fits = false;+ for (unsigned n = 0; n < sizeof(channel->their_intents) / sizeof(channel->their_intents[0]); n++) {+ if (!channel->their_intents[n].there || channel->their_intents[n].size < length) {+ continue;+ }+ fits = true;+ if (!channel->their_intents[n].in_use &&+ (found < 0 || channel->their_intents[n].size < channel->their_intents[found].size)) {+ found = (int)n;+ }+ }+ if (found >= 0 && glink_room() >= ((length + sizeof(header) + 7) & ~7u)) {+ break;+ }+ if (!fits && !asked) {+ channel->request_answered = false;+ glink_say(GLINK_INTENT_REQUEST, channel->ours, length, NULL, 0);+ asked = true;+ }+ if (waited >= 5000 || (!fits && asked && channel->request_answered && !channel->request_granted)) {+ lck_mtx_unlock(&glink_lock);+ return found < 0 ? ENOBUFS : EAGAIN;+ }+ lck_mtx_unlock(&glink_lock);+ sleep_ms(1);+ lck_mtx_lock(&glink_lock);+ }+ channel->their_intents[found].in_use = true;+ header.message.param2 = channel->their_intents[found].id;+ glink_put(&header, sizeof(header), data, length);+ lck_mtx_unlock(&glink_lock);+ return 0;+}++/* ---- QRTR: the router's datagrams, and ports of ours (redwood_wifi.c) ------------------------------ */++/*+ * What goes over the channel: datagrams of Qualcomm's IPC router, each with a header that says+ * who sends it (a node, a port) and to whom, and of which kind. The control port hears of the+ * services the modem offers ("new server": a service, an instance, a node, a port), and later+ * their going. We are node 1 (Linux's default, and what this phone's Android is); the modem says+ * who it is in what it sends. The interface for the rest of the kernel is sys/redwood_qrtr.h:+ * the four calls at the end of this section, which the NAS client (redwood_nas.c) and the QMI+ * helpers (redwood_qmi.c) take.+ */+#define REDWOOD_QRTR_MAX 8192 /* the most a datagram holds, either way */+#define QRTR_US 1u+#define QRTR_CONTROL 0xfffffffeu /* the control port */+#define QRTR_FIRST_PORT 0x4000u /* where Linux begins numbering ports that ask for no number */+enum { QRTR_DATA = 1, QRTR_HELLO = 2, QRTR_BYE = 3, QRTR_NEW_SERVER = 4, QRTR_DEL_SERVER = 5, QRTR_DEL_CLIENT = 6, QRTR_RESUME_TX = 7,+ QRTR_NEW_LOOKUP = 10, QRTR_DEL_LOOKUP = 11 };++struct qrtr_header { /* version 1, which Linux sends; version 2 is read as well */+ uint32_t version, type, from_node, from_port, confirm, size, to_node, to_port;+};+struct qrtr_control {+ uint32_t command;+ uint32_t words[4]; /* a server: service, instance, node, port; a client: node, port */+};++struct redwood_qrtr_port {+ uint32_t number; /* 0: free */+ redwood_qrtr_receive_t receive;+ redwood_qrtr_server_t server;+ void *context;+ struct {+ uint32_t service, instance; /* the instance as on the wire: version | instance << 8; 0: any */+ bool asked;+ bool fresh; /* not yet told of the services that were there before it */+ } lookups[4];+};++static LCK_MTX_DECLARE(qrtr_lock, &mpss_lock_group); /* the tables below */+static LCK_MTX_DECLARE(qrtr_calling, &mpss_lock_group); /* held by the link's thread around a callback */+static struct redwood_qrtr_port qrtr_ports[16];+static struct {+ uint32_t service, instance, node, port;+ bool there;+} qrtr_servers[64];+static uint32_t qrtr_peer; /* the modem's node; 0: it has not said */+static uint32_t qrtr_next_port = QRTR_FIRST_PORT, qrtr_in, qrtr_out, qrtr_hellos, qrtr_lookups;+static thread_t link_thread_is;++/* The name service's side: our servers, told of on hello and on lookup (the one there is: rmtfs, below). */+static struct {+ uint32_t service, instance, port;+} our_servers[4];+static uint32_t rmtfs_requests, rmtfs_reads, rmtfs_writes, rmtfs_sectors_read, rmtfs_sectors_written, rmtfs_refused;+static uint32_t tftp_requests, tftp_served, tftp_refused, servreg_requests;++/* One datagram out, of any type. */+static int+qrtr_out_packet(uint32_t type, uint32_t from_port, uint32_t to_node, uint32_t to_port, const void *data, uint32_t length)+{+ uint32_t padded = (length + 3) & ~3u;+ struct qrtr_header *packet;+ int error;++ if (length == 0 || length > REDWOOD_QRTR_MAX) {+ return EMSGSIZE;+ }+ packet = kalloc_data(sizeof(*packet) + padded, Z_WAITOK | Z_ZERO);+ if (packet == NULL) {+ return ENOMEM;+ }+ packet->version = 1;+ packet->type = type;+ packet->from_node = QRTR_US;+ packet->from_port = from_port;+ packet->size = length;+ packet->to_node = to_node;+ packet->to_port = to_port;+ memcpy(packet + 1, data, length);+ error = glink_send(&glink.channel[CHANNEL_QRTR], packet, (uint32_t)sizeof(*packet) + padded);+ kfree_data(packet, sizeof(*packet) + padded);+ qrtr_out += error == 0;+ return error;+}++static bool+qrtr_matches(uint32_t service, uint32_t instance, unsigned server)+{+ return qrtr_servers[server].there && qrtr_servers[server].service == service &&+ (instance == 0 || qrtr_servers[server].instance == instance);+}++/* Everybody who looked a service up is told that it has come, or gone. From the link's thread. */+static void+qrtr_tell(uint32_t service, uint32_t instance, uint32_t node, uint32_t port, bool arrived)+{+ for (unsigned p = 0; p < sizeof(qrtr_ports) / sizeof(qrtr_ports[0]); p++) {+ for (unsigned l = 0; l < 4; l++) {+ redwood_qrtr_server_t server = NULL;+ void *context;++ lck_mtx_lock(&qrtr_lock);+ if (qrtr_ports[p].number != 0 && qrtr_ports[p].lookups[l].asked && !qrtr_ports[p].lookups[l].fresh &&+ qrtr_ports[p].lookups[l].service == service &&+ (qrtr_ports[p].lookups[l].instance == 0 || qrtr_ports[p].lookups[l].instance == instance)) {+ server = qrtr_ports[p].server;+ }+ context = qrtr_ports[p].context;+ lck_mtx_unlock(&qrtr_lock);+ if (server != NULL) {+ server(context, arrived, service, instance & 0xff, instance >> 8, node, port);+ }+ }+ }+}++static void qrtr_announce(uint32_t node, unsigned server, bool end);+static void qrtr_announce_all(uint32_t node);++/* The control port's part: Linux's name service, with our one service to tell of. */+static void+qrtr_control(const struct qrtr_header *header, const struct qrtr_control *control)+{+ struct qrtr_control hello = { .command = QRTR_HELLO };+ int found = -1, error;++ switch (control->command) {+ case QRTR_HELLO:+ if (qrtr_hellos++ != 0) {+ break; /* (answered once: two that answer every hello never end) */+ }+ error = qrtr_out_packet(QRTR_HELLO, QRTR_CONTROL, header->from_node, QRTR_CONTROL, &hello, sizeof(hello));+ printf("mpss: qrtr: HELLO FROM NODE %u; ours to it: %s\n", header->from_node, error == 0 ? "SENT" : "NOT SENT");+ qrtr_announce_all(header->from_node); /* (the name service's part: what we offer) */+ break;+ case QRTR_NEW_SERVER:+ lck_mtx_lock(&qrtr_lock);+ for (unsigned i = 0; i < sizeof(qrtr_servers) / sizeof(qrtr_servers[0]); i++) {+ if (qrtr_servers[i].there && qrtr_servers[i].node == control->words[2] && qrtr_servers[i].port == control->words[3]) {+ found = (int)i;+ break;+ }+ if (!qrtr_servers[i].there && found < 0) {+ found = (int)i;+ }+ }+ if (found >= 0) {+ qrtr_servers[found].there = true;+ qrtr_servers[found].service = control->words[0];+ qrtr_servers[found].instance = control->words[1];+ qrtr_servers[found].node = control->words[2];+ qrtr_servers[found].port = control->words[3];+ }+ lck_mtx_unlock(&qrtr_lock);+ printf("mpss: qrtr: a service: %u (%#x), version %u, instance %u, at node %u port %u\n", control->words[0],+ control->words[0], control->words[1] & 0xff, control->words[1] >> 8, control->words[2], control->words[3]);+ qrtr_tell(control->words[0], control->words[1], control->words[2], control->words[3], true);+ break;+ case QRTR_DEL_SERVER:+ case QRTR_BYE:+ for (unsigned i = 0; i < sizeof(qrtr_servers) / sizeof(qrtr_servers[0]); i++) {+ bool gone;++ lck_mtx_lock(&qrtr_lock);+ gone = qrtr_servers[i].there && (control->command == QRTR_BYE ? qrtr_servers[i].node == header->from_node :+ qrtr_servers[i].node == control->words[2] && qrtr_servers[i].port == control->words[3]);+ qrtr_servers[i].there = qrtr_servers[i].there && !gone;+ lck_mtx_unlock(&qrtr_lock);+ if (gone) {+ printf("mpss: qrtr: service %u at node %u port %u has gone\n", qrtr_servers[i].service, qrtr_servers[i].node,+ qrtr_servers[i].port);+ qrtr_tell(qrtr_servers[i].service, qrtr_servers[i].instance, qrtr_servers[i].node, qrtr_servers[i].port, false);+ }+ }+ break;+ case QRTR_NEW_LOOKUP: {+ /* the modem looks for a service: ours that match, then the empty "new server" that ends the answer */+ unsigned matches = 0;++ for (unsigned i = 0; i < sizeof(our_servers) / sizeof(our_servers[0]); i++) {+ if (our_servers[i].port != 0 && (control->words[0] == 0 || our_servers[i].service == control->words[0]) &&+ (control->words[1] == 0 || our_servers[i].instance == control->words[1])) {+ qrtr_announce(header->from_node, i, false);+ matches++;+ }+ }+ qrtr_announce(header->from_node, 0, true);+ if (qrtr_lookups++ < 16) {+ printf("mpss: qrtr: the modem looks for service %u (%#x), instance %u: %u of ours\n", control->words[0],+ control->words[0], control->words[1], matches);+ }+ break;+ }+ default:+ break; /* (a client's going, and what else there is: nothing here wants it) */+ }+}++/* A message from the channel: one datagram. From the link's thread. */+static void+qrtr_input(const uint8_t *data, uint32_t length)+{+ struct qrtr_header header;+ redwood_qrtr_receive_t receive = NULL;+ void *context = NULL;+ uint32_t at;++ if (length >= sizeof(header) && data[0] == 1) {+ memcpy(&header, data, sizeof(header));+ at = sizeof(header);+ } else if (length >= 16 && data[0] == 3) {+ /* version 2: version, type, flags, a length of options; the size; then four numbers of 16 bits */+ bzero(&header, sizeof(header));+ header.type = data[1];+ header.confirm = data[2] & 1;+ memcpy(&header.size, data + 4, 4);+ header.from_node = (uint32_t)(data[8] | data[9] << 8);+ header.from_port = (uint32_t)(data[10] | data[11] << 8);+ header.to_node = (uint32_t)(data[12] | data[13] << 8);+ header.to_port = (uint32_t)(data[14] | data[15] << 8);+ header.from_port = header.from_port == 0xfffe ? QRTR_CONTROL : header.from_port;+ header.to_port = header.to_port == 0xfffe ? QRTR_CONTROL : header.to_port;+ at = 16 + data[3];+ } else {+ printf("mpss: qrtr: %u bytes that are no datagram (they begin %#x)\n", length, length ? data[0] : 0);+ return;+ }+ if (header.size == 0 || at + header.size > length) {+ printf("mpss: qrtr: a datagram of %u bytes in a message of %u: dropped\n", header.size, length);+ return;+ }+ qrtr_in++;+ if (qrtr_peer == 0) {+ qrtr_peer = header.from_node;+ }+ if (header.type == QRTR_RESUME_TX) {+ return; /* (we never ask for one: GLINK's intents hold us back already) */+ }+ if (header.confirm) {+ /* the sender counts what it has sent to this port, and waits for this before it sends many more */+ struct qrtr_control resume = { .command = QRTR_RESUME_TX, .words = { header.to_node, header.to_port } };++ qrtr_out_packet(QRTR_RESUME_TX, header.to_port, header.from_node, header.from_port, &resume, sizeof(resume));+ }+ if (header.to_port == QRTR_CONTROL || header.type == QRTR_HELLO || header.type == QRTR_BYE ||+ header.type == QRTR_NEW_SERVER || header.type == QRTR_DEL_SERVER || header.type == QRTR_NEW_LOOKUP) {+ struct qrtr_control control = { 0 };++ memcpy(&control, data + at, header.size < sizeof(control) ? header.size : sizeof(control));+ qrtr_control(&header, &control);+ return;+ }+ lck_mtx_lock(&qrtr_lock);+ for (unsigned p = 0; p < sizeof(qrtr_ports) / sizeof(qrtr_ports[0]); p++) {+ if (qrtr_ports[p].number == header.to_port && header.type == QRTR_DATA) {+ receive = qrtr_ports[p].receive;+ context = qrtr_ports[p].context;+ }+ }+ lck_mtx_unlock(&qrtr_lock);+ if (receive != NULL) {+ receive(context, header.from_node, header.from_port, data + at, header.size);+ } else if (qrtr_in <= 64) {+ printf("mpss: qrtr: a datagram of %u bytes from node %u port %u to our port %u: nobody here listens there\n", header.size,+ header.from_node, header.from_port, header.to_port);+ }+}++/* Lookups made since the last look are told of the services that were there before them. From the link's thread. */+static void+qrtr_fresh(void)+{+ for (unsigned p = 0; p < sizeof(qrtr_ports) / sizeof(qrtr_ports[0]); p++) {+ for (unsigned l = 0; l < 4; l++) {+ redwood_qrtr_server_t server;+ uint32_t service, instance;+ void *context;++ lck_mtx_lock(&qrtr_lock);+ if (qrtr_ports[p].number == 0 || !qrtr_ports[p].lookups[l].fresh) {+ lck_mtx_unlock(&qrtr_lock);+ continue;+ }+ qrtr_ports[p].lookups[l].fresh = false;+ server = qrtr_ports[p].server;+ service = qrtr_ports[p].lookups[l].service;+ instance = qrtr_ports[p].lookups[l].instance;+ context = qrtr_ports[p].context;+ lck_mtx_unlock(&qrtr_lock);+ for (unsigned i = 0; i < sizeof(qrtr_servers) / sizeof(qrtr_servers[0]) && server != NULL; i++) {+ if (qrtr_matches(service, instance, i)) {+ server(context, true, qrtr_servers[i].service, qrtr_servers[i].instance & 0xff,+ qrtr_servers[i].instance >> 8, qrtr_servers[i].node, qrtr_servers[i].port);+ }+ }+ }+ }+}++static struct redwood_qrtr_port *+qrtr_open(redwood_qrtr_receive_t receive, redwood_qrtr_server_t server, void *context)+{+ struct redwood_qrtr_port *port = NULL;++ lck_mtx_lock(&qrtr_lock);+ for (unsigned p = 0; p < sizeof(qrtr_ports) / sizeof(qrtr_ports[0]) && port == NULL; p++) {+ if (qrtr_ports[p].number == 0) {+ port = &qrtr_ports[p];+ bzero(port, sizeof(*port));+ port->number = qrtr_next_port++;+ port->receive = receive;+ port->server = server;+ port->context = context;+ }+ }+ lck_mtx_unlock(&qrtr_lock);+ return port;+}++static int+qrtr_send(struct redwood_qrtr_port *port, uint32_t node, uint32_t to, const void *data, size_t length)+{+ if (port == NULL || port->number == 0 || length > REDWOOD_QRTR_MAX) {+ return EINVAL;+ }+ return qrtr_out_packet(QRTR_DATA, port->number, node, to, data, (uint32_t)length);+}++static int+qrtr_lookup(struct redwood_qrtr_port *port, uint32_t service, uint32_t version, uint32_t instance)+{+ int error = ENOSPC;++ lck_mtx_lock(&qrtr_lock);+ for (unsigned l = 0; l < 4 && error != 0 && port != NULL && port->number != 0; l++) {+ if (!port->lookups[l].asked) {+ port->lookups[l].service = service;+ port->lookups[l].instance = instance != 0 ? (version & 0xff) | instance << 8 : 0;+ port->lookups[l].asked = true;+ port->lookups[l].fresh = true;+ error = 0;+ }+ }+ lck_mtx_unlock(&qrtr_lock);+ return error;+}++static void+qrtr_close(struct redwood_qrtr_port *port)+{+ struct qrtr_control gone = { .command = QRTR_DEL_CLIENT, .words = { QRTR_US, port ? port->number : 0 } };+ bool outside = current_thread() != link_thread_is;++ if (port == NULL || port->number == 0) {+ return;+ }+ if (outside) {+ lck_mtx_lock(&qrtr_calling); /* (a callback of the port's that is running ends first) */+ }+ lck_mtx_lock(&qrtr_lock);+ port->number = 0;+ lck_mtx_unlock(&qrtr_lock);+ if (outside) {+ lck_mtx_unlock(&qrtr_calling);+ }+ if (qrtr_peer != 0 && glink.channel[CHANNEL_QRTR].open) {+ qrtr_out_packet(QRTR_DEL_CLIENT, gone.words[1], qrtr_peer, QRTR_CONTROL, &gone, sizeof(gone));+ }+}++static const struct redwood_qrtr_ops qrtr_ops = { .open = qrtr_open, .send = qrtr_send, .lookup = qrtr_lookup, .close = qrtr_close };++/* The modem's QRTR for the rest of the kernel: NULL until the link is up (sys/redwood_qrtr.h). */+const struct redwood_qrtr_ops *+redwood_mpss_qrtr(void)+{+ return glink.channel[CHANNEL_QRTR].open ? &qrtr_ops : NULL;+}++/* ---- the link's thread ----------------------------------------------------------------------------- */++static volatile bool link_wanted, link_runs;++/* Everything the modem says once it runs: its SMP2P item, its queue, and from that the datagrams, handed on from here. */+static void+link_thread(__unused void *parameter, __unused wait_result_t result)+{+ struct smp2p words, last;+ struct glink_packet *packet;+ bool gave_up = false;++ bzero(&last, sizeof(last));+ dsp_words(&last);+ while (link_wanted) {+ sleep_us(1000);+ doorbells();+ if (dsp_words(&words) && memcmp(&words, &last, sizeof(words)) != 0) {+ last = words;+ dsp_listen(&words);+ if ((dsp_state(&words) & DSP_FATAL) && !gave_up) {+ gave_up = true;+ printf("mpss: link: THE MODEM SAYS IT HAS GIVEN UP (\"slave-kernel\" = %#x)\n", dsp_state(&words));+ show_crash_reason();+ }+ }+ for (;;) {+ struct glink_channel *channel = &glink.channel[CHANNEL_QRTR];++ lck_mtx_lock(&glink_lock);+ glink_pump();+ packet = channel->first;+ if (packet != NULL) {+ channel->first = packet->next;+ channel->last = channel->first == NULL ? NULL : channel->last;+ }+ lck_mtx_unlock(&glink_lock);+ if (packet == NULL) {+ break;+ }+ lck_mtx_lock(&qrtr_calling);+ qrtr_input(packet->data, packet->length);+ lck_mtx_unlock(&qrtr_calling);+ kfree_data(packet, sizeof(*packet) + packet->length);+ }+ lck_mtx_lock(&qrtr_calling);+ qrtr_fresh();+ lck_mtx_unlock(&qrtr_calling);+ }+ link_runs = false;+ thread_terminate(current_thread());+}++/* The link given up, before the modem is stopped: Linux too takes GLINK down first and tells the modem nothing of it. */+static void+link_stop(void)+{+ struct glink_packet *packet;++ if (link_runs) {+ link_wanted = false;+ for (unsigned ms = 0; ms < 2000 && link_runs; ms++) {+ sleep_ms(1);+ }+ printf("mpss: link: %s; GLINK said %u and heard %u messages, QRTR sent %u and received %u datagrams; its doorbells heard: "+ "GLINK %u, SMP2P %u; rmtfs: %u requests, %u reads (%u sectors), %u writes (%u sectors), %u refused\n", link_runs ?+ "THE THREAD DOES NOT END" : "down", glink.said, glink.heard, qrtr_out, qrtr_in, doorbell.glink, doorbell.smp2p,+ rmtfs_requests, rmtfs_reads, rmtfs_sectors_read, rmtfs_writes, rmtfs_sectors_written, rmtfs_refused);+ printf("mpss: link: tftp: %u requests, %u served whole, %u refused; the registry: %u questions\n", tftp_requests, tftp_served,+ tftp_refused, servreg_requests);+ }+ lck_mtx_lock(&glink_lock);+ glink.up = false;+ for (unsigned c = 0; c < CHANNELS; c++) {+ struct glink_channel *channel = &glink.channel[c];++ channel->open = false;+ while ((packet = channel->first) != NULL) {+ channel->first = packet->next;+ kfree_data(packet, sizeof(*packet) + packet->length);+ }+ channel->last = NULL;+ }+ lck_mtx_unlock(&glink_lock);+ lck_mtx_lock(&qrtr_lock);+ bzero(qrtr_servers, sizeof(qrtr_servers));+ qrtr_peer = 0;+ lck_mtx_unlock(&qrtr_lock);+}++/* ---- the modem's file systems: RMTFS, a service of ours on QRTR ----------------------------------- */++/*+ * The modem keeps its own files -- its settings, its calibration, what it has learnt of the+ * networks -- on four partitions of the phone's storage, which it cannot reach itself: the+ * application processor reads and writes them for it, through QMI service 14, "RMTFS" (Linux:+ * github.com/linux-msm/rmtfs, which this follows). Until that service is there the modem offers+ * nothing beyond SSCTL and its service registry (service 43 and 66: what the first run saw), and+ * waits. The modem opens a file by path (/boot/modem_fs1 and so on), asks for a buffer (2.5 MB+ * of memory both can reach: dt qcom,rmtfs_sharedmem@0; TrustZone is asked to let the modem into+ * it, as Linux's rmtfs_mem and Android's sharedmem do, and out again when it is shut down), and+ * then reads and writes sectors of 512 bytes through that buffer.+ *+ * Here the four partitions are read whole into memory of ours when the steps begin (the phone's+ * storage through the loader, redwood_ufs.c: ufs=2, which the ordinary run has), the modem is+ * served out of that, and what it writes goes into that memory and no further: the storage is+ * not written. The modem is content with that for a run; what it would have saved is lost when+ * the run ends, as the next run reads the partitions afresh.+ *+ * QRTR's name service is the application processor's job as well (Linux: net/qrtr/ns.c): the+ * modem is told of our service when it says hello and when it looks for one, with an empty+ * "new server" to end an answer to a lookup, as Linux ends it.+ */+#define RMTFS_SERVICE 14+#define RMTFS_VERSION 1+#define RMTFS_INSTANCE 0+#define RMTFS_SECTOR 512+#define RMTFS_BUFFER_SIZE 0x280000u /* dt: reg of qcom,rmtfs_sharedmem@0 */+#define RMTFS_OPEN 1+#define RMTFS_CLOSE 2+#define RMTFS_RW_IOVEC 3+#define RMTFS_ALLOC_BUFF 4+#define RMTFS_GET_DEV_ERROR 5+#define QMI_REQUEST 0+#define QMI_RESPONSE 2+#define QMI_RESULT_FAILURE 1+#define QMI_ERR_INTERNAL 1+#define QMI_ERR_MALFORMED 2+#define SCM_MP_ASSIGN SCM_CALL(0xc, 0x16)+#define SCM_ASSIGN_ARGUMENTS 0x1117 /* seven: a buffer, a value, a buffer, a value, a buffer, two values */+#define VMID_HLOS 3+#define VMID_MSS_MSA 15+#define VMID_NAV 0x2b /* the modem's navigation side; dt: qcom,vm-nav-path of qcom,rmtfs_sharedmem@0 */+#define VM_PERM_RW 6++static struct {+ const char *path, *partition;+ uint8_t *data;+ uint64_t size;+ bool open;+} rmtfs_files[4] = {+ { "/boot/modem_fs1", "modemst1" }, { "/boot/modem_fs2", "modemst2" }, { "/boot/modem_fsc", "fsc" },+ { "/boot/modem_fsg", "fsg" },+};+static struct redwood_qrtr_port *rmtfs_port;+static vm_offset_t rmtfs_buffer; /* the buffer, as this kernel sees it; 0: none yet */+static volatile uint8_t *rmtfs_window; /* ... and the uncached one it is used through */+static uint64_t rmtfs_buffer_pa;+static bool rmtfs_shared; /* the modem has been let into it */++static unsigned+rmtfs_load(void)+{+ unsigned loaded = 0;++ for (unsigned f = 0; f < 4; f++) {+ uint64_t size = 0;+ int error = redwood_ufs_read_partition(rmtfs_files[f].partition, 0, NULL, 0, &size);++ if (error != 0 || size == 0 || size > (64u << 20)) {+ printf("mpss: %u: the partition %s: %s (%d)\n", step, rmtfs_files[f].partition, error == ENXIO ?+ "no storage to read it from (the loader was not asked for ufs=2)" : "not read", error);+ continue;+ }+ if (rmtfs_files[f].data == NULL || rmtfs_files[f].size != size) {+ if (rmtfs_files[f].data != NULL) {+ kfree_data(rmtfs_files[f].data, rmtfs_files[f].size);+ }+ rmtfs_files[f].data = kalloc_data(size, Z_WAITOK | Z_ZERO);+ rmtfs_files[f].size = rmtfs_files[f].data == NULL ? 0 : size;+ }+ if (rmtfs_files[f].data == NULL) {+ printf("mpss: %u: no memory for %s (%llu bytes)\n", step, rmtfs_files[f].partition, size);+ continue;+ }+ error = redwood_ufs_read_partition(rmtfs_files[f].partition, 0, rmtfs_files[f].data, size, NULL);+ if (error != 0) {+ printf("mpss: %u: the partition %s: read error %d\n", step, rmtfs_files[f].partition, error);+ continue;+ }+ printf("mpss: %u: the partition %s (%s): %llu bytes read into memory of ours\n", step, rmtfs_files[f].partition,+ rmtfs_files[f].path, size);+ loaded++;+ }+ return loaded;+}++/*+ * The buffer, once. The modem's requests carry addresses of 32 bits, so it has to be below 4 GB,+ * and the kernel's free memory there is not to be had in one piece of 2.5 MB; so it is the first+ * 2.5 MB of the memory reserved for the compute DSP (dt: reserved-memory cdsp@9c700000, 30 MB,+ * no-map: nothing of ours uses it, and the CDSP is never started here), through an uncached+ * window, as the modem reads it.+ */+#define RMTFS_BUFFER_PA 0x9c700000ull++static bool+rmtfs_buffer_make(void)+{+ if (rmtfs_window != NULL) {+ return true;+ }+ rmtfs_buffer_pa = RMTFS_BUFFER_PA;+ PE_parse_boot_argn("modem_buffer", &rmtfs_buffer_pa, sizeof(rmtfs_buffer_pa)); /* (another region, to try) */+ rmtfs_window = (volatile uint8_t *)ml_io_map_unmappable((vm_offset_t)rmtfs_buffer_pa, RMTFS_BUFFER_SIZE, VM_WIMG_WCOMB);+ if (rmtfs_window == NULL) {+ printf("mpss: %u: the modem's buffer at %#llx cannot be mapped\n", step, rmtfs_buffer_pa);+ return false;+ }+ rmtfs_buffer = (vm_offset_t)rmtfs_window;+ for (uint32_t i = 0; i < RMTFS_BUFFER_SIZE; i += 8) {+ *(volatile uint64_t *)(rmtfs_window + i) = 0;+ }+ settle();+ printf("mpss: %u: the modem's buffer: %u bytes at %#llx (the compute DSP's memory, unused here)\n", step, RMTFS_BUFFER_SIZE,+ rmtfs_buffer_pa);+ return true;+}++/*+ * TrustZone asked to let the modem into the buffer (Linux's qcom_scm_assign_mem(): the memory, who+ * has it now, who is to have it and how), or, with `in` false, out of it again.+ */+static bool+rmtfs_share(bool in)+{+ struct {+ uint64_t address, size; /* +0: the memory */+ uint64_t pad0[6];+ uint32_t sources[3]; /* +64: who has it */+ uint32_t pad1[13];+ struct { /* +128: who is to have it */+ uint32_t vmid, permission;+ uint64_t context;+ uint32_t context_size, unused;+ } destinations[3];+ uint8_t pad2[40];+ uint64_t more[7]; /* +240: the arguments beyond the registers */+ } *page;+ vm_offset_t memory = 0;+ uint64_t answer = ~0ull, pa;+ unsigned sources, destinations;+ int32_t result;++ if (rmtfs_buffer == 0 || rmtfs_shared == in) {+ return rmtfs_buffer != 0;+ }+ if (kmem_alloc(kernel_map, &memory, PAGE_SIZE, KMA_KOBJECT | KMA_ZERO | KMA_DATA, VM_KERN_MEMORY_IOKIT) != KERN_SUCCESS) {+ return false;+ }+ page = (void *)memory;+ page->address = rmtfs_buffer_pa;+ page->size = RMTFS_BUFFER_SIZE;+ if (in) {+ page->sources[0] = VMID_HLOS;+ sources = 1;+ page->destinations[0].vmid = VMID_HLOS;+ page->destinations[0].permission = VM_PERM_RW;+ page->destinations[1].vmid = VMID_MSS_MSA;+ page->destinations[1].permission = VM_PERM_RW;+ page->destinations[2].vmid = VMID_NAV;+ page->destinations[2].permission = VM_PERM_RW;+ destinations = option("modem_no_nav") ? 2 : 3;+ } else {+ page->sources[0] = VMID_HLOS;+ page->sources[1] = VMID_MSS_MSA;+ page->sources[2] = VMID_NAV;+ sources = option("modem_no_nav") ? 2 : 3;+ page->destinations[0].vmid = VMID_HLOS;+ page->destinations[0].permission = VM_PERM_RW;+ destinations = 1;+ }+ pa = kvtophys(memory);+ page->more[0] = sources * 4; /* argument 3: the size of the sources */+ page->more[1] = pa + 128; /* 4: the destinations */+ page->more[2] = destinations * sizeof(page->destinations[0]); /* 5: their size */+ page->more[3] = 0; /* 6: spare */+ for (vm_offset_t line = memory; line < memory + PAGE_SIZE; line += 64) {+ __asm__ volatile ("dc civac, %0" : : "r" (line));+ }+ settle();+ announce(in ? "ask TrustZone to let the modem into our buffer for its files (the memory assigned to HLOS and MSS, both "+ "read and write)" : "ask TrustZone to take the buffer back from the modem (assigned to HLOS alone)");+ result = scm(in ? "to share the buffer with the modem" : "to take the buffer back", SCM_MP_ASSIGN, SCM_ASSIGN_ARGUMENTS, pa, 16,+ pa + 64, pa + 240, &answer);+ kmem_free(kernel_map, memory, PAGE_SIZE);+ if (result != 0 || answer != 0) {+ printf("mpss: %u: TRUSTZONE DOES NOT %s: the call returns %d, its answer is %#llx\n", step, in ? "SHARE THE BUFFER" :+ "TAKE THE BUFFER BACK", result, answer);+ return false;+ }+ rmtfs_shared = in;+ printf("mpss: %u: %s\n", step, in ? "THE MODEM MAY USE THE BUFFER" : "the buffer is ours alone again");+ if (in && option("modem_share_test")) {+ /* (an experiment: is the buffer still ours to write, now that it is shared? said aloud before and after) */+ tell("mpss: %u: about to write a word into the shared buffer at %#llx, and read it back\n", step, rmtfs_buffer_pa + 0x200);+ *(volatile uint32_t *)(rmtfs_window + 0x200) = 0x52545346;+ settle();+ tell("mpss: %u: the shared buffer takes our write: it reads %#x\n", step, *(volatile uint32_t *)(rmtfs_window + 0x200));+ }+ return true;+}++/* A TLV of a QMI message: its value and length, or NULL. */+static const uint8_t *+qmi_tlv(const uint8_t *m, size_t length, uint8_t type, uint16_t *size)+{+ for (size_t at = 7; at + 3 <= length;) {+ uint16_t n = (uint16_t)(m[at + 1] | m[at + 2] << 8);++ if (at + 3 + n > length) {+ return NULL;+ }+ if (m[at] == type) {+ *size = n;+ return m + at + 3;+ }+ at += 3 + n;+ }+ return NULL;+}++static uint32_t+le32(const uint8_t *p)+{+ return (uint32_t)(p[0] | p[1] << 8 | p[2] << 16 | p[3] << 24);+}++/* The modem's request for its files: answered, the one TLV the answer carries besides the result made here. From the link's thread. */+static void+rmtfs_receive(__unused void *context, uint32_t node, uint32_t port, const void *data, size_t length)+{+ const uint8_t *m = data, *value;+ uint8_t answer[64], *tlv;+ uint16_t message, size = 0, result = 0, error = 0, extra = 0;+ unsigned file = 4;++ if (length < 7 || m[0] != QMI_REQUEST) {+ return;+ }+ message = (uint16_t)(m[3] | m[4] << 8);+ rmtfs_requests++;+ bzero(answer, sizeof(answer));+ answer[0] = QMI_RESPONSE;+ answer[1] = m[1];+ answer[2] = m[2];+ answer[3] = m[3];+ answer[4] = m[4];+ tlv = answer + 7 + 7; /* (the result TLV, 7 bytes, comes first) */+ switch (message) {+ case RMTFS_OPEN:+ /* (the path: the TLV's value is the string itself, with no length in front -- the modem sends it so) */+ value = qmi_tlv(m, length, 1, &size);+ for (unsigned f = 0; value != NULL && f < 4; f++) {+ if (size == strlen(rmtfs_files[f].path) && memcmp(value, rmtfs_files[f].path, size) == 0) {+ file = f;+ }+ }+ if (file == 4 || rmtfs_files[file].data == NULL) {+ printf("mpss: rmtfs: the modem opens \"%.*s\": NOT HERE\n", value != NULL ? (int)(size < 64 ? size : 64) : 0,+ value != NULL ? (const char *)value : "");+ result = QMI_RESULT_FAILURE;+ error = QMI_ERR_INTERNAL;+ rmtfs_refused++;+ break;+ }+ rmtfs_files[file].open = true;+ printf("mpss: rmtfs: the modem opens %s (%s, %llu bytes): caller %u\n", rmtfs_files[file].path, rmtfs_files[file].partition,+ rmtfs_files[file].size, file + 1);+ tlv[0] = 0x10; /* the caller's id: the file's number plus one */+ tlv[1] = 4;+ tlv[2] = 0;+ tlv[3] = (uint8_t)(file + 1);+ extra = 7;+ break;+ case RMTFS_CLOSE:+ value = qmi_tlv(m, length, 1, &size);+ file = value != NULL && size == 4 && le32(value) >= 1 && le32(value) <= 4 ? le32(value) - 1 : 4;+ if (file < 4) {+ rmtfs_files[file].open = false;+ printf("mpss: rmtfs: the modem closes %s\n", rmtfs_files[file].path);+ }+ break;+ case RMTFS_ALLOC_BUFF:+ value = qmi_tlv(m, length, 2, &size);+ if (value == NULL || size != 4 || le32(value) > RMTFS_BUFFER_SIZE || rmtfs_buffer == 0 || !rmtfs_shared) {+ printf("mpss: rmtfs: the modem asks for a buffer of %u bytes: NOT GIVEN (ours is %u, %s)\n", value != NULL && size == 4 ?+ le32(value) : 0, RMTFS_BUFFER_SIZE, rmtfs_shared ? "shared" : "NOT SHARED");+ result = QMI_RESULT_FAILURE;+ error = QMI_ERR_INTERNAL;+ rmtfs_refused++;+ break;+ }+ tell("mpss: rmtfs: the modem asks for a buffer of %u bytes: given ours at %#llx\n", le32(value), rmtfs_buffer_pa);+ tlv[0] = 0x10;+ tlv[1] = 8;+ tlv[2] = 0;+ for (unsigned b = 0; b < 8; b++) {+ tlv[3 + b] = (uint8_t)(rmtfs_buffer_pa >> (8 * b));+ }+ extra = 11;+ break;+ case RMTFS_RW_IOVEC: {+ const uint8_t *direction = qmi_tlv(m, length, 2, &size), *entries;+ uint16_t direction_size = size, entries_size = 0;+ bool write;+ unsigned count;++ value = qmi_tlv(m, length, 1, &size);+ entries = qmi_tlv(m, length, 3, &entries_size);+ file = value != NULL && size == 4 && le32(value) >= 1 && le32(value) <= 4 ? le32(value) - 1 : 4;+ if (file == 4 || !rmtfs_files[file].open || direction == NULL || direction_size != 1 || entries == NULL || entries_size < 1 ||+ entries_size != 1 + 12 * entries[0] || rmtfs_window == NULL) {+ result = QMI_RESULT_FAILURE;+ error = QMI_ERR_MALFORMED;+ rmtfs_refused++;+ break;+ }+ write = direction[0] != 0;+ count = entries[0];+ {+ /* (an experiment: the first request waited on, said aloud before and after, for the log to catch what follows) */+ static bool first = true;+ uint32_t delay = 0;++ if (first && PE_parse_boot_argn("modem_rmtfs_delay", &delay, sizeof(delay)) && delay != 0) {+ tell("mpss: rmtfs: the modem's first request (%s %u pieces of %s, from sector %u, %u sectors, to %#x): waiting %u s "+ "before it is served\n", write ? "write" : "read", count, rmtfs_files[file].partition, le32(entries + 1),+ le32(entries + 9), le32(entries + 5), delay);+ for (uint32_t ms = 0; ms < delay * 1000; ms++) {+ sleep_ms(1);+ }+ tell("mpss: rmtfs: serving it now\n");+ }+ first = false;+ }+ if (option("modem_rmtfs_rw_off")) {+ /* (an experiment: the modem is told no, and the buffer is not touched from here) */+ printf("mpss: rmtfs: the modem %s %u pieces of %s: REFUSED (modem_rmtfs_rw_off)\n", write ? "writes" : "reads", count,+ rmtfs_files[file].partition);+ result = QMI_RESULT_FAILURE;+ error = QMI_ERR_INTERNAL;+ rmtfs_refused++;+ break;+ }+ for (unsigned e = 0; e < count && result == 0; e++) {+ uint64_t sector = le32(entries + 1 + 12 * e), physical = le32(entries + 5 + 12 * e), sectors = le32(entries + 9 + 12 * e);+ uint64_t bytes = sectors * RMTFS_SECTOR;++ if (physical < rmtfs_buffer_pa || physical - rmtfs_buffer_pa + bytes > RMTFS_BUFFER_SIZE ||+ sector * RMTFS_SECTOR + bytes > rmtfs_files[file].size) {+ printf("mpss: rmtfs: %s %llu sectors from %llu of %s at %#llx: OUTSIDE THE FILE OR THE BUFFER\n", write ? "write" :+ "read", sectors, sector, rmtfs_files[file].partition, physical);+ result = QMI_RESULT_FAILURE;+ error = QMI_ERR_MALFORMED;+ rmtfs_refused++;+ break;+ }+ if (write) {+ for (uint64_t i = 0; i < bytes; i++) {+ rmtfs_files[file].data[sector * RMTFS_SECTOR + i] = rmtfs_window[physical - rmtfs_buffer_pa + i];+ }+ rmtfs_sectors_written += sectors;+ } else {+ for (uint64_t i = 0; i < bytes; i++) {+ rmtfs_window[physical - rmtfs_buffer_pa + i] = rmtfs_files[file].data[sector * RMTFS_SECTOR + i];+ }+ rmtfs_sectors_read += sectors;+ }+ }+ settle();+ if (option("modem_rmtfs_rw_fake")) {+ /* (an experiment: the buffer is filled as asked, but the modem is told no: does the filling alone end the run?) */+ tell("mpss: rmtfs: the buffer is filled for the modem's %s of %s, and it is told NO (modem_rmtfs_rw_fake)\n",+ write ? "write" : "read", rmtfs_files[file].partition);+ result = QMI_RESULT_FAILURE;+ error = QMI_ERR_INTERNAL;+ break;+ }+ rmtfs_reads += !write;+ rmtfs_writes += write;+ if (rmtfs_reads + rmtfs_writes <= 12) {+ tell("mpss: rmtfs: the modem %s %u pieces of %s%s: served\n", write ? "writes" : "reads", count, rmtfs_files[file].partition,+ write ? " (kept in memory, not on the storage)" : "");+ }+ break;+ }+ case RMTFS_GET_DEV_ERROR:+ tlv[0] = 0x10; /* the device's error status: none */+ tlv[1] = 1;+ tlv[2] = 0;+ tlv[3] = 0;+ extra = 4;+ break;+ default:+ printf("mpss: rmtfs: a request this does not know (%#x, %zu bytes)\n", message, length);+ result = QMI_RESULT_FAILURE;+ error = QMI_ERR_MALFORMED;+ break;+ }+ answer[5] = (uint8_t)(7 + extra);+ answer[6] = (uint8_t)((7 + extra) >> 8);+ answer[7] = 2; /* the result TLV */+ answer[8] = 4;+ answer[9] = 0;+ answer[10] = (uint8_t)result;+ answer[11] = (uint8_t)(result >> 8);+ answer[12] = (uint8_t)error;+ answer[13] = (uint8_t)(error >> 8);+ if (rmtfs_port != NULL) {+ int sent = qrtr_send(rmtfs_port, node, port, answer, 7 + 7 + extra);++ if (sent != 0) {+ printf("mpss: rmtfs: the answer to %#x is NOT SENT (%d)\n", message, sent);+ }+ }+}++/* Our service announced to a node (its control port): a "new server", or the empty one that ends an answer to a lookup. */+static void+qrtr_announce(uint32_t node, unsigned server, bool end)+{+ struct qrtr_control packet = { .command = QRTR_NEW_SERVER };++ if (!end) {+ packet.words[0] = our_servers[server].service;+ packet.words[1] = our_servers[server].instance;+ packet.words[2] = QRTR_US;+ packet.words[3] = our_servers[server].port;+ }+ qrtr_out_packet(QRTR_NEW_SERVER, QRTR_CONTROL, node, QRTR_CONTROL, &packet, sizeof(packet));+}++static void+qrtr_announce_all(uint32_t node)+{+ for (unsigned i = 0; i < sizeof(our_servers) / sizeof(our_servers[0]); i++) {+ if (our_servers[i].port != 0) {+ qrtr_announce(node, i, false);+ }+ }+}++static void tftp_receive(void *context, uint32_t node, uint32_t port, const void *data, size_t length);+static void servreg_receive(void *context, uint32_t node, uint32_t port, const void *data, size_t length);+static struct redwood_qrtr_port *tftp_port, *servreg_port;+static bool ipa_qmi_start(void);+static void ipa_qmi_stop(void);+static struct redwood_qrtr_port *ipa_host_port;++/* The port, and the service on it; the partitions are read in step 2, the buffer shared in step 7. */+static bool+rmtfs_start(void)+{+ if (rmtfs_port == NULL) {+ rmtfs_port = qrtr_open(rmtfs_receive, NULL, NULL);+ }+ if (rmtfs_port == NULL) {+ return false;+ }+ our_servers[0].service = RMTFS_SERVICE;+ our_servers[0].instance = RMTFS_VERSION | RMTFS_INSTANCE << 8;+ our_servers[0].port = rmtfs_port->number;+ if (!option("modem_no_tftp")) {+ if (tftp_port == NULL) {+ tftp_port = qrtr_open(tftp_receive, NULL, NULL);+ }+ if (tftp_port != NULL) {+ our_servers[1].service = 4096; /* TQFTP (below) */+ our_servers[1].instance = 1;+ our_servers[1].port = tftp_port->number;+ }+ if (servreg_port == NULL) {+ servreg_port = qrtr_open(servreg_receive, NULL, NULL);+ }+ if (servreg_port != NULL) {+ our_servers[2].service = 64; /* the service registry's locator (below) */+ our_servers[2].instance = 257;+ our_servers[2].port = servreg_port->number;+ }+ if (ipa_qmi_start()) {+ our_servers[3].service = 0x31; /* the IPA's host service (below) */+ our_servers[3].instance = 0x101;+ our_servers[3].port = ipa_host_port->number;+ }+ }+ return true;+}++static void+rmtfs_stop(void)+{+ for (unsigned i = 0; i < sizeof(our_servers) / sizeof(our_servers[0]); i++) {+ our_servers[i].port = 0;+ }+ if (rmtfs_port != NULL) {+ qrtr_close(rmtfs_port);+ rmtfs_port = NULL;+ }+ if (tftp_port != NULL) {+ qrtr_close(tftp_port);+ tftp_port = NULL;+ }+ if (servreg_port != NULL) {+ qrtr_close(servreg_port);+ servreg_port = NULL;+ }+ ipa_qmi_stop();+ for (unsigned f = 0; f < 4; f++) {+ rmtfs_files[f].open = false;+ }+}++/* ---- TFTP over QRTR: the modem's files of configuration (tqftpserv) --------------------------------- */++/*+ * Besides its file systems, the modem reads files of its firmware partition -- the carrier+ * configurations (mcfg), shared objects -- through TFTP carried in QRTR datagrams, from a+ * service numbered 4096 that Linux's tqftpserv and Android's provide; until it has them it+ * stays offline ("device not ready" to every request to go online). Here the partition's+ * directory image/ is in the system image under /fw/image (userland/build.sh: fw-modem/image,+ * pulled from the phone's /vendor/firmware_mnt/image), and served read-only: a read request+ * names a file as /readonly/firmware/image/..., its options (RFC 2347: blksize, tsize, timeoutms;+ * Qualcomm's wsize, a window of blocks sent without waiting, rsize, how much of the file, and+ * seek, from where) are answered in an OACK, and the blocks go from a port of ours opened for the+ * transfer, as TFTP wants. Writes (/readwrite/...) are refused: nothing of the modem's is kept+ * here between runs.+ */+#define TQFTP_SERVICE 4096+#define TQFTP_INSTANCE 1 /* version 1, instance 0 */+#define TFTP_RRQ 1+#define TFTP_WRQ 2+#define TFTP_DATA 3+#define TFTP_ACK 4+#define TFTP_ERROR 5+#define TFTP_OACK 6+#define TFTP_ENOENT 1+#define TFTP_EACCESS 2+#define TFTP_EBADOP 4+#define TFTP_END_OF_TRANSFER 9+#define TFTP_MAX_BLOCK 8000 /* a datagram of ours holds 8192; the modem's intents 0x4400 */+#define TFTP_MAX_WINDOW 4++struct tftp_transfer {+ struct redwood_qrtr_port *port;+ uint32_t node, peer;+ const uint8_t *data; /* a read: the file, whole, in memory of ours (root_file(), or tftp_files) */+ uint64_t size;+ uint32_t blksize, wsize, rsize, seek;+ uint32_t sent; /* blocks sent so far */+ bool done, writing, borrowed; /* borrowed: data is a tftp_files entry's, not to be freed */+ uint8_t *written; /* a write: what has come, up to TFTP_WRITE_MAX */+ uint32_t written_size, expected;+ char name[96];+};+static struct tftp_transfer tftp_transfers[4];++/* What the modem writes, kept for the run (it reads some of it back); not on the storage. */+#define TFTP_WRITE_MAX (1u << 20)+static struct {+ char name[96];+ uint8_t *data;+ uint32_t size;+} tftp_files[8];++static void+tftp_error(struct redwood_qrtr_port *from, uint32_t node, uint32_t port, uint16_t code, const char *text)+{+ uint8_t packet[64];+ size_t n = strlen(text) + 1;++ if (n > sizeof(packet) - 4) {+ n = sizeof(packet) - 4;+ }+ packet[0] = 0;+ packet[1] = TFTP_ERROR;+ packet[2] = (uint8_t)(code >> 8);+ packet[3] = (uint8_t)code;+ memcpy(packet + 4, text, n);+ packet[4 + n - 1] = 0;+ qrtr_send(from, node, port, packet, 4 + n);+}++static void+tftp_transfer_end(struct tftp_transfer *t)+{+ if (t->data != NULL && !t->borrowed) {+ uint8_t *data = __DECONST(uint8_t *, t->data);++ kfree_data(data, t->size);+ }+ t->data = NULL;+ if (t->written != NULL) {+ kfree_data(t->written, TFTP_WRITE_MAX);+ t->written = NULL;+ }+ if (t->port != NULL) {+ qrtr_close(t->port);+ t->port = NULL;+ }+ t->done = true;+}++/* Blocks `from` + 1 onward, a window of them: TFTP's blocks are numbered from 1. */+static void+tftp_send_window(struct tftp_transfer *t, uint32_t from)+{+ uint8_t *packet = kalloc_data(t->blksize + 4, Z_WAITOK);++ if (packet == NULL) {+ return;+ }+ for (uint32_t w = 0; w < t->wsize; w++) {+ uint32_t block = from + 1 + w;+ uint64_t offset = (uint64_t)t->seek + (uint64_t)(block - 1) * t->blksize;+ uint32_t n = offset >= t->size ? 0 : (uint32_t)(t->size - offset < t->blksize ? t->size - offset : t->blksize);++ if (t->rsize != 0 && (uint64_t)(block - 1) * t->blksize >= t->rsize) {+ break; /* (as much as was asked for has gone) */+ }+ if (t->rsize != 0 && (uint64_t)block * t->blksize > t->rsize) {+ uint32_t left = (uint32_t)(t->rsize - (uint64_t)(block - 1) * t->blksize);++ n = left < n ? left : n; /* (as much as was asked for, and no more than the file has) */+ }+ packet[0] = 0;+ packet[1] = TFTP_DATA;+ packet[2] = (uint8_t)(block >> 8);+ packet[3] = (uint8_t)block;+ if (n != 0) {+ memcpy(packet + 4, t->data + offset, n);+ }+ if (qrtr_send(t->port, t->node, t->peer, packet, 4 + n) != 0) {+ break;+ }+ t->sent = block;+ if (n < t->blksize) {+ break; /* the last block */+ }+ }+ kfree_data(packet, t->blksize + 4);+}++/* What the modem says to a transfer's port: acknowledgements, or an error that ends it. From the link's thread. */+static void+tftp_transfer_receive(void *context, uint32_t node, uint32_t port, const void *data, size_t length)+{+ struct tftp_transfer *t = context;+ const uint8_t *m = data;+ uint16_t opcode, block;++ if (t->done || node != t->node || port != t->peer || length < 4) {+ return;+ }+ opcode = (uint16_t)(m[0] << 8 | m[1]);+ block = (uint16_t)(m[2] << 8 | m[3]);+ if (t->writing && opcode == TFTP_DATA) {+ uint32_t n = (uint32_t)(length - 4);+ uint8_t ack[4] = { 0, TFTP_ACK, (uint8_t)(block >> 8), (uint8_t)block };++ if (block != t->expected) {+ tftp_error(t->port, node, port, TFTP_EBADOP, "block out of sequence");+ tftp_transfer_end(t);+ return;+ }+ if (t->written_size + n > TFTP_WRITE_MAX) {+ tftp_error(t->port, node, port, 3, "too much");+ tftp_transfer_end(t);+ return;+ }+ memcpy(t->written + t->written_size, m + 4, n);+ t->written_size += n;+ t->expected++;+ if (block % t->wsize == 0 || n < t->blksize) {+ qrtr_send(t->port, node, port, ack, sizeof(ack));+ }+ if (n < t->blksize) {+ /* the last block: the file is kept, under its name, for the run */+ unsigned f, free_slot = 8;++ for (f = 0; f < 8; f++) {+ if (tftp_files[f].data != NULL && strcmp(tftp_files[f].name, t->name) == 0) {+ break;+ }+ if (tftp_files[f].data == NULL && free_slot == 8) {+ free_slot = f;+ }+ }+ if (f == 8) {+ f = free_slot;+ }+ if (f < 8) {+ if (tftp_files[f].data != NULL) {+ kfree_data(tftp_files[f].data, tftp_files[f].size);+ }+ tftp_files[f].data = kalloc_data(t->written_size ? t->written_size : 1, Z_WAITOK);+ if (tftp_files[f].data != NULL) {+ memcpy(tftp_files[f].data, t->written, t->written_size);+ tftp_files[f].size = t->written_size;+ strlcpy(tftp_files[f].name, t->name, sizeof(tftp_files[f].name));+ }+ }+ printf("mpss: tftp: the modem has written \"%s\": %u bytes, kept%s\n", t->name, t->written_size, f < 8 ? "" :+ " NOWHERE (no room)");+ tftp_served++;+ tftp_transfer_end(t);+ }+ return;+ }+ if (opcode == TFTP_ERROR) {+ if (block != TFTP_END_OF_TRANSFER) {+ printf("mpss: tftp: %s: the modem ends the transfer with error %u \"%.*s\"\n", t->name, block, (int)(length - 4), m + 4);+ }+ tftp_transfer_end(t);+ return;+ }+ if (opcode != TFTP_ACK) {+ tftp_error(t->port, node, port, TFTP_EBADOP, "expected ACK");+ tftp_transfer_end(t);+ return;+ }+ if ((uint64_t)t->seek + (uint64_t)block * t->blksize >= t->size || (t->rsize != 0 && (uint64_t)block * t->blksize >= t->rsize)) {+ /* the acknowledged block was the last: over */+ tftp_served++;+ tftp_transfer_end(t);+ return;+ }+ tftp_send_window(t, block);+}++static const char *+tftp_file_of(const char *path)+{+ static const char *const prefixes[] = { "/readonly/firmware/image/", "/readonly/vendor/firmware_mnt/image/",+ "/readonly/vendor/firmware/", "/readonly/firmware/" };++ for (unsigned i = 0; i < sizeof(prefixes) / sizeof(prefixes[0]); i++) {+ size_t n = strlen(prefixes[i]);++ if (strncmp(path, prefixes[i], n) == 0 && strnstr(path + n, "..", strlen(path + n)) == NULL) {+ return path + n;+ }+ }+ return NULL;+}++/* A request to our TFTP port: a read starts a transfer on a port of its own; anything else is refused. From the link's thread. */+static void+tftp_receive(__unused void *context, uint32_t node, uint32_t port, const void *data, size_t length)+{+ const uint8_t *m = data, *end = m + length;+ const char *name, *mode, *p;+ struct tftp_transfer *t = NULL;+ uint32_t blksize = 512, wsize = 1, rsize = 0, seek = 0;+ bool tsize = false, options = false;+ char fwname[160];+ uint64_t size = 0;+ const uint8_t *file;+ uint8_t oack[160];+ unsigned at;++ bool writing;++ if (length < 4) {+ return;+ }+ tftp_requests++;+ writing = (m[0] << 8 | m[1]) == TFTP_WRQ;+ if ((m[0] << 8 | m[1]) != TFTP_RRQ && !writing) {+ printf("mpss: tftp: the modem's request %u: not a read nor a write, refused\n", m[0] << 8 | m[1]);+ tftp_error(tftp_port, node, port, TFTP_EBADOP, "not a read nor a write");+ tftp_refused++;+ return;+ }+ name = (const char *)m + 2;+ p = name;+ while (p < (const char *)end && *p != 0) {+ p++;+ }+ if (p >= (const char *)end) {+ return;+ }+ mode = p + 1;+ p = mode;+ while (p < (const char *)end && *p != 0) {+ p++;+ }+ if (p >= (const char *)end) {+ return;+ }+ p++;+ /* the options: name, value, name, value..., each ended by a zero */+ while (p < (const char *)end) {+ const char *option = p, *value;++ while (p < (const char *)end && *p != 0) {+ p++;+ }+ if (p >= (const char *)end) {+ break;+ }+ value = ++p;+ while (p < (const char *)end && *p != 0) {+ p++;+ }+ if (p >= (const char *)end) {+ break;+ }+ p++;+ options = true;+ if (strcmp(option, "blksize") == 0) {+ blksize = (uint32_t)strtoul(value, NULL, 10);+ } else if (strcmp(option, "wsize") == 0) {+ wsize = (uint32_t)strtoul(value, NULL, 10);+ } else if (strcmp(option, "rsize") == 0) {+ rsize = (uint32_t)strtoul(value, NULL, 10);+ } else if (strcmp(option, "seek") == 0) {+ seek = (uint32_t)strtoul(value, NULL, 10);+ } else if (strcmp(option, "tsize") == 0) {+ tsize = true;+ }+ }+ if (blksize < 8) {+ blksize = 8;+ }+ if (blksize > TFTP_MAX_BLOCK) {+ blksize = TFTP_MAX_BLOCK;+ }+ if (wsize < 1) {+ wsize = 1;+ }+ if (wsize > TFTP_MAX_WINDOW) {+ wsize = TFTP_MAX_WINDOW;+ }+ for (unsigned i = 0; i < sizeof(tftp_transfers) / sizeof(tftp_transfers[0]) && t == NULL; i++) {+ if (tftp_transfers[i].port == NULL) {+ t = &tftp_transfers[i];+ }+ }+ if (t == NULL) {+ printf("mpss: tftp: the modem %s \"%s\": NO FREE TRANSFER\n", writing ? "writes" : "reads", name);+ tftp_error(tftp_port, node, port, TFTP_ENOENT, "busy");+ tftp_refused++;+ return;+ }+ if (writing) {+ bzero(t, sizeof(*t));+ t->written = kalloc_data(TFTP_WRITE_MAX, Z_WAITOK);+ t->port = t->written == NULL ? NULL : qrtr_open(tftp_transfer_receive, NULL, t);+ if (t->port == NULL) {+ tftp_transfer_end(t);+ tftp_error(tftp_port, node, port, TFTP_ENOENT, "no port");+ return;+ }+ t->node = node;+ t->peer = port;+ t->writing = true;+ t->expected = 1;+ t->blksize = blksize;+ t->wsize = wsize;+ strlcpy(t->name, name, sizeof(t->name));+ printf("mpss: tftp: the modem writes \"%s\": blocks of %u, window %u; taken into memory\n", name, blksize, wsize);+ if (!options) {+ uint8_t ack[4] = { 0, TFTP_ACK, 0, 0 };++ qrtr_send(t->port, node, port, ack, sizeof(ack));+ return;+ }+ size = 0;+ goto oack;+ }+ /* a read: what the modem wrote earlier under that name, else the firmware partition's file */+ file = NULL;+ for (unsigned f = 0; f < 8 && file == NULL; f++) {+ if (tftp_files[f].data != NULL && strcmp(tftp_files[f].name, name) == 0) {+ file = tftp_files[f].data;+ size = tftp_files[f].size;+ t->borrowed = true;+ }+ }+ if (file == NULL && tftp_file_of(name) == NULL) {+ printf("mpss: tftp: the modem reads \"%s\": NOT A PATH SERVED HERE\n", name);+ tftp_error(tftp_port, node, port, TFTP_ENOENT, "file not found");+ tftp_refused++;+ return;+ }+ if (file == NULL) {+ snprintf(fwname, sizeof(fwname), "image/%s", tftp_file_of(name));+ file = root_file(fwname, &size);+ } else {+ strlcpy(fwname, "(written by the modem)", sizeof(fwname));+ }+ if (file == NULL) {+ printf("mpss: tftp: the modem reads \"%s\" (fw/%s): NOT HERE\n", name, fwname);+ tftp_error(tftp_port, node, port, TFTP_ENOENT, "file not found");+ tftp_refused++;+ return;+ }+ {+ bool borrowed = t->borrowed;++ bzero(t, sizeof(*t));+ t->borrowed = borrowed;+ }+ t->port = qrtr_open(tftp_transfer_receive, NULL, t);+ if (t->port == NULL) {+ if (!t->borrowed) {+ uint8_t *gone = __DECONST(uint8_t *, file);++ kfree_data(gone, size);+ }+ tftp_error(tftp_port, node, port, TFTP_ENOENT, "no port");+ return;+ }+ t->node = node;+ t->peer = port;+ t->data = file;+ t->size = size;+ t->blksize = blksize;+ t->wsize = wsize;+ t->rsize = rsize;+ t->seek = seek;+ strlcpy(t->name, name, sizeof(t->name));+ if (tftp_requests <= 40) {+ printf("mpss: tftp: the modem reads \"%s\": %llu bytes; blocks of %u, window %u, %u asked for, from %u%s\n", name, size, blksize,+ wsize, rsize, seek, options ? "" : " (no options)");+ }+ if (!options) {+ tftp_send_window(t, 0);+ return;+ }+oack:+ /* OACK: the options as taken, as tqftpserv answers them */+ oack[0] = 0;+ oack[1] = TFTP_OACK;+ at = 2;+ at += (unsigned)snprintf((char *)oack + at, sizeof(oack) - at, "blksize") + 1;+ at += (unsigned)snprintf((char *)oack + at, sizeof(oack) - at, "%u", blksize) + 1;+ at += (unsigned)snprintf((char *)oack + at, sizeof(oack) - at, "timeoutms") + 1;+ at += (unsigned)snprintf((char *)oack + at, sizeof(oack) - at, "1000") + 1;+ if (tsize && !t->writing) {+ at += (unsigned)snprintf((char *)oack + at, sizeof(oack) - at, "tsize") + 1;+ at += (unsigned)snprintf((char *)oack + at, sizeof(oack) - at, "%llu", size) + 1;+ }+ at += (unsigned)snprintf((char *)oack + at, sizeof(oack) - at, "wsize") + 1;+ at += (unsigned)snprintf((char *)oack + at, sizeof(oack) - at, "%u", wsize) + 1;+ if (rsize != 0) {+ at += (unsigned)snprintf((char *)oack + at, sizeof(oack) - at, "rsize") + 1;+ at += (unsigned)snprintf((char *)oack + at, sizeof(oack) - at, "%u", rsize) + 1;+ }+ if (seek != 0) {+ at += (unsigned)snprintf((char *)oack + at, sizeof(oack) - at, "seek") + 1;+ at += (unsigned)snprintf((char *)oack + at, sizeof(oack) - at, "%u", seek) + 1;+ }+ qrtr_send(t->port, node, port, oack, at);+}++/* ---- the service registry's locator (pd-mapper) ---------------------------------------------------- */++/*+ * The modem's protection domains register their services with a registry whose locator (QMI+ * service 64) is the application processor's: asked for a service by name ("tms/servreg",+ * "gps/gps_service"), it answers with the domains that provide it ("msm/modem/root_pd") and their+ * QMI instance. Linux's pd-mapper reads the table from the *.jsn files beside the firmware; here+ * they are read from /fw/image (the phone's own), each one domain with its services.+ */+#define SERVREG_SERVICE 64+#define SERVREG_INSTANCE 257 /* what pd-mapper publishes: version 257 */+#define SERVREG_GET_DOMAIN_LIST 0x21+#define SERVREG_PFR 0x24++static struct {+ char service[48]; /* provider/service */+ char domain[48]; /* soc/domain/subdomain */+ uint32_t instance;+} servreg_map[32];+static unsigned servreg_maps;++/* The text between the quotes after "key": in a JSON file of the registry's simple shape; NULL if there is none. */+static const char *+jsn_string(const char *from, const char *end, const char *key, char *out, size_t out_size)+{+ char quoted[48];+ const char *at;++ snprintf(quoted, sizeof(quoted), "\"%s\"", key);+ at = strnstr(from, quoted, (size_t)(end - from));+ if (at == NULL) {+ return NULL;+ }+ at += strlen(quoted);+ while (at < end && (*at == ':' || *at == ' ' || *at == '\t' || *at == '\n' || *at == '\r')) {+ at++;+ }+ if (at >= end) {+ return NULL;+ }+ if (*at == '"') {+ const char *close = at + 1;++ while (close < end && *close != '"') {+ close++;+ }+ if (close >= end) {+ return NULL;+ }+ snprintf(out, out_size, "%.*s", (int)(close - at - 1), at + 1);+ return close + 1;+ }+ /* a number */+ {+ const char *close = at;++ while (close < end && ((*close >= '0' && *close <= '9') || *close == '-')) {+ close++;+ }+ snprintf(out, out_size, "%.*s", (int)(close - at), at);+ return close;+ }+}++static void+servreg_load(void)+{+ static const char *const files[] = { "modemr.jsn", "adspr.jsn", "adsps.jsn", "adspua.jsn", "battmgr.jsn", "cdspr.jsn", "wpssr.jsn" };++ servreg_maps = 0;+ for (unsigned f = 0; f < sizeof(files) / sizeof(files[0]); f++) {+ char name[64], soc[24], domain[24], subdomain[24], instance[16], provider[24], service[24], full[48];+ uint64_t size = 0;+ const uint8_t *data;+ const char *text, *end, *at;++ snprintf(name, sizeof(name), "image/%s", files[f]);+ data = root_file(name, &size);+ if (data == NULL) {+ continue;+ }+ text = (const char *)data;+ end = text + size;+ if (jsn_string(text, end, "soc", soc, sizeof(soc)) == NULL || jsn_string(text, end, "domain", domain, sizeof(domain)) == NULL ||+ jsn_string(text, end, "subdomain", subdomain, sizeof(subdomain)) == NULL ||+ jsn_string(text, end, "qmi_instance_id", instance, sizeof(instance)) == NULL) {+ uint8_t *gone = __DECONST(uint8_t *, data);++ printf("mpss: servreg: fw/%s: not of the shape this reads\n", name);+ kfree_data(gone, size);+ continue;+ }+ snprintf(full, sizeof(full), "%s/%s/%s", soc, domain, subdomain);+ at = text;+ while (servreg_maps < sizeof(servreg_map) / sizeof(servreg_map[0]) &&+ (at = jsn_string(at, end, "provider", provider, sizeof(provider))) != NULL &&+ jsn_string(at, end, "service", service, sizeof(service)) != NULL) {+ snprintf(servreg_map[servreg_maps].service, sizeof(servreg_map[0].service), "%s/%s", provider, service);+ strlcpy(servreg_map[servreg_maps].domain, full, sizeof(servreg_map[0].domain));+ servreg_map[servreg_maps].instance = (uint32_t)strtoul(instance, NULL, 10);+ servreg_maps++;+ }+ {+ uint8_t *gone = __DECONST(uint8_t *, data);++ kfree_data(gone, size);+ }+ }+ printf("mpss: %u: the service registry: %u services of %u files\n", step, servreg_maps, (unsigned)(sizeof(files) / sizeof(files[0])));+ for (unsigned i = 0; i < servreg_maps; i++) {+ printf("mpss: %u: %s at %s, instance %u\n", step, servreg_map[i].service, servreg_map[i].domain, servreg_map[i].instance);+ }+}++/* A question to the locator, answered. From the link's thread. */+static void+servreg_receive(__unused void *context, uint32_t node, uint32_t port, const void *data, size_t length)+{+ const uint8_t *m = data, *value;+ uint8_t answer[512];+ uint16_t message, size = 0;+ unsigned at = 7 + 7, found = 0;+ char wanted[64] = "";++ if (length < 7 || m[0] != QMI_REQUEST) {+ return;+ }+ message = (uint16_t)(m[3] | m[4] << 8);+ servreg_requests++;+ bzero(answer, sizeof(answer));+ answer[0] = QMI_RESPONSE;+ answer[1] = m[1];+ answer[2] = m[2];+ answer[3] = m[3];+ answer[4] = m[4];+ answer[7] = 2; /* result: success */+ answer[8] = 4;+ if (message == SERVREG_GET_DOMAIN_LIST) {+ value = qmi_tlv(m, length, 1, &size);+ if (value != NULL) {+ snprintf(wanted, sizeof(wanted), "%.*s", (int)(size < 63 ? size : 63), (const char *)value);+ }+ for (unsigned i = 0; i < servreg_maps; i++) {+ found += strcmp(servreg_map[i].service, wanted) == 0;+ }+ /* TLV 0x10: how many domains in all; 0x11: the table's revision; 0x12: the list, each a string with a+ * byte of length, the instance (4 bytes), and the service data (a byte of validity, 4 bytes) */+ answer[at++] = 0x10;+ answer[at++] = 2;+ answer[at++] = 0;+ answer[at++] = (uint8_t)found;+ answer[at++] = (uint8_t)(found >> 8);+ answer[at++] = 0x11;+ answer[at++] = 2;+ answer[at++] = 0;+ answer[at++] = 1;+ answer[at++] = 0;+ if (found != 0) {+ unsigned list = at;++ answer[at++] = 0x12;+ at += 2; /* the length, filled in below */+ answer[at++] = (uint8_t)found;+ for (unsigned i = 0; i < servreg_maps && at + 64 < sizeof(answer); i++) {+ if (strcmp(servreg_map[i].service, wanted) != 0) {+ continue;+ }+ answer[at++] = (uint8_t)strlen(servreg_map[i].domain);+ memcpy(answer + at, servreg_map[i].domain, strlen(servreg_map[i].domain));+ at += (unsigned)strlen(servreg_map[i].domain);+ for (unsigned b = 0; b < 4; b++) {+ answer[at++] = (uint8_t)(servreg_map[i].instance >> (8 * b));+ }+ answer[at++] = 0; /* no service data */+ at += 4;+ }+ answer[list + 1] = (uint8_t)(at - list - 3);+ answer[list + 2] = (uint8_t)((at - list - 3) >> 8);+ }+ printf("mpss: servreg: the modem asks where \"%s\" is: %u domain%s\n", wanted, found, found == 1 ? "" : "s");+ } else if (message == SERVREG_PFR) {+ value = qmi_tlv(m, length, 1, &size);+ printf("mpss: servreg: the modem reports a failure of \"%.*s\"\n", value ? (int)(size < 63 ? size : 63) : 0,+ value ? (const char *)value : "");+ } else {+ printf("mpss: servreg: a request this does not know (%#x)\n", message);+ answer[10] = QMI_RESULT_FAILURE;+ answer[12] = QMI_ERR_MALFORMED;+ }+ answer[5] = (uint8_t)(at - 7);+ answer[6] = (uint8_t)((at - 7) >> 8);+ qrtr_send(servreg_port, node, port, answer, at);+}++/* ---- the IPA's QMI handshake with the modem (Linux's ipa_qmi.c) ------------------------------------- */++/*+ * The modem's IPA driver and the application processor's agree on the IPA's shared memory over+ * QMI before the modem uses the IPA: the application processor hosts service 0x31 instance 1 (the+ * modem registers for an indication there, 0x20, and says when its driver is up, 0x35), and asks+ * the modem's service 0x31 instance 2 to initialize its driver (0x21) with where the tables are in+ * the IPA's local memory -- the layout of IPA v4.11 (Linux: ipa_data-v4.11.c) --, then tells it+ * that all is complete (indication 0x22). Without this the modem's LTE downlink task dies on a+ * null pointer the moment the modem goes online (23 October: enl2_dl, three of three).+ */+#define IPA_QMI_SERVICE 0x31+#define IPA_HOST_INSTANCE 0x101 /* version 1, instance 1: ours */+#define IPA_MODEM_INSTANCE 0x201 /* version 1, instance 2: the modem's */+#define IPA_QMI_INDICATION_REGISTER 0x20+#define IPA_QMI_INIT_DRIVER 0x21+#define IPA_QMI_INIT_COMPLETE 0x22+#define IPA_QMI_INSTALL_FILTER_RULE 0x23+#define IPA_QMI_FILTER_INSTALLED_NOTIF 0x24+#define IPA_QMI_CONFIG 0x27+#define IPA_QMI_DRIVER_INIT_COMPLETE 0x35+#define QMI_INDICATION 4++static struct redwood_qrtr_port *ipa_client_port;+static uint32_t ipa_modem_node, ipa_modem_port; /* the modem's IPA service; 0: not seen */+static uint32_t ipa_register_node, ipa_register_port; /* who asked for the indication */+static bool ipa_indication_wanted, ipa_indication_sent, ipa_driver_ready, ipa_uc_ready;+static uint16_t ipa_transaction;++static unsigned+qmi_put_u32(uint8_t *at, uint8_t type, uint32_t value)+{+ at[0] = type;+ at[1] = 4;+ at[2] = 0;+ for (unsigned b = 0; b < 4; b++) {+ at[3 + b] = (uint8_t)(value >> (8 * b));+ }+ return 7;+}++static unsigned+qmi_put_pair(uint8_t *at, uint8_t type, uint32_t first, uint32_t second)+{+ at[0] = type;+ at[1] = 8;+ at[2] = 0;+ for (unsigned b = 0; b < 4; b++) {+ at[3 + b] = (uint8_t)(first >> (8 * b));+ at[7 + b] = (uint8_t)(second >> (8 * b));+ }+ return 11;+}++/* A QMI message with only the result TLV: an answer (flags 2, the transaction echoed) or an indication (flags 4). */+static void+ipa_qmi_result(struct redwood_qrtr_port *from, uint32_t node, uint32_t port, uint8_t flags, uint16_t transaction, uint16_t message)+{+ uint8_t packet[14] = { flags, (uint8_t)transaction, (uint8_t)(transaction >> 8), (uint8_t)message, (uint8_t)(message >> 8), 7, 0,+ 2, 4, 0, 0, 0, 0, 0 };++ qrtr_send(from, node, port, packet, sizeof(packet));+}++static void+ipa_init_complete(void)+{+ if (ipa_indication_wanted && !ipa_indication_sent && ipa_driver_ready) {+ ipa_qmi_result(ipa_host_port, ipa_register_node, ipa_register_port, QMI_INDICATION, 0, IPA_QMI_INIT_COMPLETE);+ ipa_indication_sent = true;+ tell("mpss: ipa: the modem is told that the IPA is all set up (indication %#x)\n", IPA_QMI_INIT_COMPLETE);+ }+}++/* The modem's requests to our IPA service. From the link's thread. */+static void+ipa_host_receive(__unused void *context, uint32_t node, uint32_t port, const void *data, size_t length)+{+ const uint8_t *m = data;+ uint16_t message, transaction;++ if (length < 7 || m[0] != QMI_REQUEST) {+ return;+ }+ transaction = (uint16_t)(m[1] | m[2] << 8);+ message = (uint16_t)(m[3] | m[4] << 8);+ switch (message) {+ case IPA_QMI_INDICATION_REGISTER:+ ipa_qmi_result(ipa_host_port, node, port, QMI_RESPONSE, transaction, message);+ ipa_register_node = node;+ ipa_register_port = port;+ ipa_indication_wanted = true;+ printf("mpss: ipa: the modem registers for the IPA's indication (node %u port %u)\n", node, port);+ ipa_init_complete();+ break;+ case IPA_QMI_DRIVER_INIT_COMPLETE:+ ipa_qmi_result(ipa_host_port, node, port, QMI_RESPONSE, transaction, message);+ ipa_uc_ready = true;+ printf("mpss: ipa: the modem says its IPA driver is up\n");+ ipa_init_complete();+ break;+ case IPA_QMI_CONFIG:+ case IPA_QMI_INSTALL_FILTER_RULE:+ case IPA_QMI_FILTER_INSTALLED_NOTIF:+ default:+ /*+ * After the handshake the modem asks for the IPA's configuration (0x27), has its uplink filter rules+ * installed (0x23) and tells that they are (0x24) -- Android's driver answers each with success and+ * little else. Everything it asks is answered so here (the filter rules are not programmed: there is+ * no data path yet).+ */+ ipa_qmi_result(ipa_host_port, node, port, QMI_RESPONSE, transaction, message);+ printf("mpss: ipa: the modem's request %#x (%s) answered with success\n", message,+ message == IPA_QMI_CONFIG ? "configuration" : message == IPA_QMI_INSTALL_FILTER_RULE ? "install filter rules" :+ message == IPA_QMI_FILTER_INSTALLED_NOTIF ? "filters installed" : "not known here");+ break;+ }+}++/* The modem's answer to our INIT_DRIVER. From the link's thread. */+static void+ipa_client_receive(__unused void *context, uint32_t node, uint32_t port, const void *data, size_t length)+{+ const uint8_t *m = data;+ uint16_t size = 0;+ const uint8_t *value;++ /* every datagram on this port is shown: on 24 October the modem's answer was waited for in vain, if it came at all */+ printf("mpss: ipa: from node %u port %u, %u bytes:", node, port, (unsigned)length);+ for (size_t i = 0; i < length && i < 24; i++) {+ printf(" %02x", m[i]);+ }+ printf("%s\n", length > 24 ? " ..." : "");+ if (length < 7 || m[0] != QMI_RESPONSE || node != ipa_modem_node || port != ipa_modem_port) {+ return;+ }+ value = qmi_tlv(m, length, 2, &size);+ if ((m[3] | m[4] << 8) == IPA_QMI_INIT_DRIVER && value != NULL && size == 4) {+ uint16_t result = (uint16_t)(value[0] | value[1] << 8), error = (uint16_t)(value[2] | value[3] << 8);++ tell("mpss: ipa: the modem %s our INIT_DRIVER (result %u, error %u)\n", result == 0 ? "ACCEPTS" : "REFUSES", result, error);+ ipa_driver_ready = result == 0;+ ipa_init_complete();+ }+}++/* The IPA's local memory as IPA v4.11 lays it out (Linux: ipa_data-v4.11.c), told to the modem. */+static void+ipa_init_driver(void)+{+ uint8_t packet[200];+ unsigned at = 7;+ uint32_t routes = 8; /* the route tables' entries the modem may use: indices 0..7 of 15 (Android's driver:+ * v4_modem_rt_index_hi 7; its log "num_indices 7"); modem_ipa_routes=N */+ const uint32_t o = ipa_mem_offset;++ PE_parse_boot_argn("modem_ipa_routes", &routes, sizeof(routes));++ ipa_transaction = (uint16_t)(ipa_transaction + 1 == 0 ? 1 : ipa_transaction + 1);+ packet[0] = QMI_REQUEST;+ packet[1] = (uint8_t)ipa_transaction;+ packet[2] = (uint8_t)(ipa_transaction >> 8);+ packet[3] = IPA_QMI_INIT_DRIVER;+ packet[4] = 0;+ at += qmi_put_u32(packet + at, 0x10, 3); /* platform: 3 = MSM Android (Android's log: "platform_type 3"; 1 is a data card) */+ at += qmi_put_pair(packet + at, 0x11, o + 0x0688, o + 0x0688 + 0x0240 - 1); /* the modem's headers */+ at += qmi_put_pair(packet + at, 0x12, o + 0x0508, routes - 1); /* IPv4 routes */+ at += qmi_put_pair(packet + at, 0x13, o + 0x0608, routes - 1); /* IPv6 routes */+ at += qmi_put_u32(packet + at, 0x14, o + 0x0308); /* IPv4 filters */+ at += qmi_put_u32(packet + at, 0x15, o + 0x0408); /* IPv6 filters */+ at += qmi_put_pair(packet + at, 0x16, o + 0x27d8, 0x800); /* the modem's own region (Android: modem_ofst 0x27d8, 0x800) */+ at += qmi_put_u32(packet + at, 0x17, 16); /* our endpoint for its control messages (AP_MODEM_RX) */+ {+ /*+ * TLV 0x18 is "is_ssr_bootup" to Android's driver (skip_uc_load to Linux's): whether the modem is+ * being restarted with the IPA's microcontroller already loaded. Android leaves it out on the first+ * boot (its log: "is_ssr_bootup 0", not sent), and the modem then loads the microcontroller itself+ * (its "booted" events follow the answer). Sent as 1 with modem_ipa_skip_uc=1 only.+ */+ uint32_t skip = 0;++ PE_parse_boot_argn("modem_ipa_skip_uc", &skip, sizeof(skip));+ if (skip != 0) {+ packet[at++] = 0x18;+ packet[at++] = 1;+ packet[at++] = 0;+ packet[at++] = 1;+ }+ }+ at += qmi_put_pair(packet + at, 0x19, o + 0x0ad0, o + 0x0ad0 + 0x0ac0 - 1); /* the modem's processing contexts (Android: 0xad0, 0xac0) */+ at += qmi_put_pair(packet + at, 0x1b, o + 0x0488, routes - 1); /* hashed IPv4 routes */+ at += qmi_put_pair(packet + at, 0x1c, o + 0x0588, routes - 1); /* hashed IPv6 routes */+ at += qmi_put_u32(packet + at, 0x1d, o + 0x0288); /* hashed IPv4 filters */+ at += qmi_put_u32(packet + at, 0x1e, o + 0x0388); /* hashed IPv6 filters */+ at += qmi_put_u32(packet + at, 0x1f, o + 0x1d00); /* the modem's quota statistics (Android: stats_quota_q6 0x1d00, 0x30) */+ at += qmi_put_u32(packet + at, 0x20, 0x30);+ at += qmi_put_u32(packet + at, 0x21, o + 0x27b0); /* drop statistics (Android: stats_drop 0x27b0, 0x20) */+ at += qmi_put_u32(packet + at, 0x22, 0x20);+ packet[5] = (uint8_t)(at - 7);+ packet[6] = (uint8_t)((at - 7) >> 8);+ printf("mpss: ipa: about to ask the modem's IPA driver (node %u port %u) to initialize with the IPA's memory layout (%u bytes, "+ "offset %#x)\n", ipa_modem_node, ipa_modem_port, at, o);+ if (qrtr_send(ipa_client_port, ipa_modem_node, ipa_modem_port, packet, at) != 0) {+ printf("mpss: ipa: INIT_DRIVER not sent\n");+ }+ if (!option("modem_ipa_no_probe")) {+ /*+ * A probe: a request of a message the service has no handler for. A QMI service answers it with an+ * error (invalid message id) at once -- or, if the modem's IPA driver is stuck before its service's+ * loop, not at all. Whether anything comes back tells which (24 October: INIT_DRIVER got no answer).+ */+ uint8_t probe[7] = { QMI_REQUEST, 0x7e, 0x00, 0xff, 0x7f, 0, 0 };++ if (qrtr_send(ipa_client_port, ipa_modem_node, ipa_modem_port, probe, sizeof(probe)) == 0) {+ printf("mpss: ipa: a probe sent to the modem's IPA service too (message 0x7fff, transaction 0x7e): an error back means it listens\n");+ }+ }+}++static void+ipa_client_server(__unused void *context, bool arrived, uint32_t service, __unused uint32_t version, uint32_t instance, uint32_t node,+ uint32_t port)+{+ if (service != IPA_QMI_SERVICE || instance != 2) {+ return;+ }+ if (arrived) {+ ipa_modem_node = node;+ ipa_modem_port = port;+ ipa_init_driver();+ } else {+ ipa_modem_node = ipa_modem_port = 0;+ ipa_driver_ready = ipa_indication_sent = ipa_indication_wanted = false;+ }+}++static bool+ipa_qmi_start(void)+{+ if (option("modem_no_ipa_qmi")) {+ return false;+ }+ if (ipa_host_port == NULL) {+ ipa_host_port = qrtr_open(ipa_host_receive, NULL, NULL);+ }+ if (ipa_client_port == NULL) {+ ipa_client_port = qrtr_open(ipa_client_receive, ipa_client_server, NULL);+ if (ipa_client_port != NULL) {+ qrtr_lookup(ipa_client_port, IPA_QMI_SERVICE, 1, 2);+ }+ }+ ipa_driver_ready = ipa_indication_sent = ipa_indication_wanted = ipa_uc_ready = false;+ return ipa_host_port != NULL && ipa_client_port != NULL;+}++static void+ipa_qmi_stop(void)+{+ if (ipa_host_port != NULL) {+ qrtr_close(ipa_host_port);+ ipa_host_port = NULL;+ }+ if (ipa_client_port != NULL) {+ qrtr_close(ipa_client_port);+ ipa_client_port = NULL;+ }+ ipa_modem_node = ipa_modem_port = 0;+}++/* ---- step 8: GLINK's queues, looked at -------------------------------------------------------------- */++static bool+step_queues(void)+{+ struct glink_message message;+ uint32_t head, tail;++ announce("find GLINK's descriptor and the two queues, and read what the modem has put into its own");+ if (!glink_find()) {+ printf("mpss: %u: GLINK's items are not all there, or not of a shape this knows (descriptor %#x, ours %#x of %u bytes, "+ "the DSP's %#x of %u)\n", step, glink.descriptor, glink.ours, glink.ours_size, glink.theirs, glink.theirs_size);+ return false;+ }+ head = smem_word(glink.descriptor + GLINK_THEIR_HEAD);+ tail = smem_word(glink.descriptor + GLINK_THEIR_TAIL);+ printf("mpss: %u: our queue: %u bytes at %#lx, written to %#x, read by the DSP to %#x; the DSP's: %u bytes at %#lx, written "+ "to %#x, read by us to %#x\n", step, glink.ours_size, SMEM_PHYS + glink.ours, smem_word(glink.descriptor + GLINK_OUR_HEAD),+ smem_word(glink.descriptor + GLINK_OUR_TAIL), glink.theirs_size, SMEM_PHYS + glink.theirs, head, tail);+ for (uint32_t at = 0, shown = 0; at + sizeof(message) <= head && head < glink.theirs_size && tail == 0 && shown < 8; shown++) {+ char name[33] = "";+ uint32_t length = sizeof(message);++ glink_peek(&message, at, sizeof(message));+ if (message.command == GLINK_OPEN) {+ glink_peek(name, at + 8, message.param2 < 32 ? message.param2 : 32);+ length += message.param2;+ }+ printf("mpss: %u: in the DSP's queue at %#x: %s %u %#x %s\n", step, at, glink_name(message.command), message.param1,+ message.param2, name);+ if (message.command != GLINK_VERSION && message.command != GLINK_OPEN) {+ break;+ }+ at += (length + 7) & ~7u;+ }+ return true;+}++/* ---- step 9: GLINK, spoken -------------------------------------------------------------------------- */++static bool+step_link(void)+{+ thread_t thread;+ uint64_t began;++ if (!dsp_started || glink.descriptor == 0) {+ return false;+ }+ announce("take up GLINK with the modem: our version, then whatever its messages call for; a thread of ours listens "+ "from here on, every millisecond");+ snprintf(last_write, sizeof(last_write), "step %u took up GLINK", step);+ if (!rmtfs_start()) {+ printf("mpss: %u: no port for the modem's file systems\n", step);+ }+ lck_mtx_lock(&glink_lock);+ glink.features = GLINK_INTENT_REUSE;+ glink.agreed = glink.broken = false;+ glink.said = glink.heard = 0;+ for (unsigned c = 0; c < CHANNELS; c++) {+ struct glink_channel *channel = &glink.channel[c];++ channel->asked = channel->acknowledged = channel->open = false;+ channel->theirs = channel->our_intent_count = 0;+ bzero(channel->their_intents, sizeof(channel->their_intents));+ }+ bzero(glink.others, sizeof(glink.others));+ glink.trace = 60;+ glink.up = true;+ qrtr_hellos = qrtr_in = qrtr_out = qrtr_lookups = 0;+ doorbell.since = began = now_us();+ glink_say(GLINK_VERSION, GLINK_VERSION_1, glink.features, NULL, 0);+ lck_mtx_unlock(&glink_lock);+ link_wanted = link_runs = true;+ if (kernel_thread_start(link_thread, NULL, &thread) != KERN_SUCCESS) {+ link_wanted = link_runs = false;+ printf("mpss: %u: no thread\n", step);+ return false;+ }+ link_thread_is = thread;+ thread_deallocate(thread);+ for (unsigned ms = 0; ms < 5000 && !glink.channel[CHANNEL_QRTR].open && !glink.broken; ms++) {+ sleep_ms(1);+ }+ sleep_ms(50);+ printf("mpss: %u: %llu ms after our version: versions %s (features %#x); the channel \"IPCRTR\" is %s; the modem also "+ "opened \"%s\" \"%s\" \"%s\" \"%s\"\n", step, (now_us() - began) / 1000, glink.agreed ? "agreed" : "NOT AGREED", glink.features,+ glink.channel[CHANNEL_QRTR].open ? "OPEN" : "NOT OPEN", glink.others[0], glink.others[1], glink.others[2], glink.others[3]);+ return glink.channel[CHANNEL_QRTR].open;+}++/* ---- step 10: QRTR ---------------------------------------------------------------------------------- */++static bool+step_qrtr(void)+{+ unsigned services = 0;++ if (!glink.channel[CHANNEL_QRTR].open) {+ return false;+ }+ PE_parse_boot_argn("modem_listen", &listen_seconds, sizeof(listen_seconds));+ announce("listen to QRTR on the channel: the DSP's hello (answered from the link's thread), and the services it tells of");+ for (unsigned ms = 0; ms < listen_seconds * 1000; ms++) {+ sleep_ms(1);+ }+ for (unsigned i = 0; i < sizeof(qrtr_servers) / sizeof(qrtr_servers[0]); i++) {+ services += qrtr_servers[i].there;+ }+ printf("mpss: %u: after %u s: %s; the DSP is node %u and has told of %u services; it looked for %u of ours; QRTR received %u "+ "and sent %u datagrams\n", step, listen_seconds, qrtr_hellos ? "HELLO BOTH WAYS" : "NO HELLO FROM THE DSP", qrtr_peer,+ services, qrtr_lookups, qrtr_in, qrtr_out);+ return qrtr_hellos != 0;+}++/* ---- step 11: the modem's services: online, and what the network says ------------------------------ */++/*+ * QMI, by hand: a message is a byte of flags (0 a request, 2 an answer, 4 a notice), a transaction+ * of two bytes, the message's number and the length of what follows, then TLVs (a type, a length+ * of two bytes, the value). The device management service (DMS, 2) is asked to put the modem+ * online -- message 0x2e, TLV 1 = 0 --: it starts in low power and stays there until somebody+ * does, which on Android is the RIL's first act. The network access service (NAS, 3) is then+ * asked, by redwood_nas.c, for the serving system (0x24: registered or not) and the signal (0x4f:+ * dBm, per radio); both are answers to questions, asked every second here and every two seconds+ * afterwards (modem=1), since no indication is asked for. What comes back is the kernel's to show:+ * sysctl kern.redwood.cellular_up, _registered, _bars, _dbm, _radio.+ */+#define QMI_DMS 2+#define QMI_NAS 3+#define DMS_SET_OPERATING_MODE 0x002e+#define DMS_GET_OPERATING_MODE 0x002d+#define DMS_ONLINE 0++static struct redwood_nas_client nas;+static bool nas_opened;+static struct redwood_qrtr_port *dms_port;+static uint32_t dms_node, dms_port_number; /* where DMS is; 0: not seen */+static volatile uint32_t dms_answers;+static volatile uint32_t dms_result, dms_error;+static uint16_t dms_transaction;+static int cellular_up, cellular_registered, cellular_bars, cellular_dbm, cellular_radio;+static uint32_t nas_polls, nas_changes;++SYSCTL_DECL(_kern_redwood);+SYSCTL_INT(_kern_redwood, OID_AUTO, cellular_up, CTLFLAG_RD | CTLFLAG_LOCKED, &cellular_up, 0,+ "the modem runs and its network access service answers");+SYSCTL_INT(_kern_redwood, OID_AUTO, cellular_registered, CTLFLAG_RD | CTLFLAG_LOCKED, &cellular_registered, 0,+ "registered with a network");+SYSCTL_INT(_kern_redwood, OID_AUTO, cellular_bars, CTLFLAG_RD | CTLFLAG_LOCKED, &cellular_bars, 0, "signal, 0 to 4 bars");+SYSCTL_INT(_kern_redwood, OID_AUTO, cellular_dbm, CTLFLAG_RD | CTLFLAG_LOCKED, &cellular_dbm, 0, "signal, dBm; 0: none read");+SYSCTL_INT(_kern_redwood, OID_AUTO, cellular_radio, CTLFLAG_RD | CTLFLAG_LOCKED, &cellular_radio, 0,+ "the radio (QMI NAS): 4 GSM, 5 UMTS, 8 LTE, 12 NR");++static const char *+radio_name(uint8_t radio)+{+ return radio == 4 ? "GSM" : radio == 5 ? "UMTS" : radio == 8 ? "LTE" : radio == 12 ? "NR" : "?";+}++/* From the link's thread, when what NAS says has changed. */+static void+nas_changed(__unused void *context, const struct redwood_nas_state *state)+{+ nas_changes++;+ cellular_registered = state->registered;+ cellular_bars = state->bars;+ cellular_dbm = state->has_signal ? state->dbm : 0;+ cellular_radio = state->has_signal ? state->radio_interface : 0;+ tell("mpss: nas: %s; signal %s%d dBm (%s): %u bars\n", state->registered ? "REGISTERED WITH A NETWORK" : "not registered",+ state->has_signal ? "" : "none read, ", state->has_signal ? state->dbm : 0, radio_name(state->radio_interface), state->bars);+}++/* A datagram for our DMS port: the answer to our one question. From the link's thread. */+static void+dms_receive(__unused void *context, __unused uint32_t node, __unused uint32_t port, const void *data, size_t length)+{+ const uint8_t *m = data;+ uint32_t result = ~0u, error = ~0u;++ if (length < 7 || m[0] != 2) {+ return;+ }+ for (size_t at = 7; at + 3 <= length;) {+ uint16_t size = (uint16_t)(m[at + 1] | m[at + 2] << 8);++ if (m[at] == 2 && size == 4 && at + 7 <= length) {+ result = (uint32_t)(m[at + 3] | m[at + 4] << 8);+ error = (uint32_t)(m[at + 5] | m[at + 6] << 8);+ }+ at += 3 + size;+ }+ printf("mpss: dms: < message %#x, transaction %u: result %u, error %u\n", m[3] | m[4] << 8, m[1] | m[2] << 8, result, error);+ if ((m[3] | m[4] << 8) == DMS_GET_OPERATING_MODE) {+ /* the mode (TLV 1: 0 online, 1 low power, 3 offline, 6 persistent low power, 7 mode-only low power), why offline (TLV 0x10,+ * bits: host image, PRI image, PRI version, memory full), and whether the hardware restricts it (TLV 0x11) */+ static const char *const modes[] = { "ONLINE", "LOW POWER", "factory test", "OFFLINE", "resetting", "shutting down",+ "PERSISTENT LOW POWER", "mode-only low power", "network test" };+ uint32_t mode = ~0u, reason = ~0u, restricted = ~0u;++ for (size_t at = 7; at + 3 <= length;) {+ uint16_t size = (uint16_t)(m[at + 1] | m[at + 2] << 8);++ if (at + 3 + size > length) {+ break;+ }+ if (m[at] == 1 && size >= 1) {+ mode = m[at + 3];+ } else if (m[at] == 0x10 && size >= 2) {+ reason = (uint32_t)(m[at + 3] | m[at + 4] << 8);+ } else if (m[at] == 0x11 && size >= 1) {+ restricted = m[at + 3];+ }+ at += 3 + size;+ }+ tell("mpss: dms: THE MODEM'S OPERATING MODE IS %u (%s); offline reason %#x%s; hardware restricted %u\n", mode,+ mode < sizeof(modes) / sizeof(modes[0]) ? modes[mode] : "?", reason, reason == ~0u ? " (not told)" : "", restricted);+ }+ dms_result = result;+ dms_error = error;+ dms_answers++;+}++static void+dms_server(__unused void *context, bool arrived, uint32_t service, __unused uint32_t version, __unused uint32_t instance,+ uint32_t node, uint32_t port)+{+ if (service != QMI_DMS) {+ return;+ }+ if (arrived) {+ dms_node = node;+ dms_port_number = port;+ } else if (node == dms_node && port == dms_port_number) {+ dms_node = dms_port_number = 0;+ }+}++/* DMS, once it is there, asked to put the modem online, and its answer waited for. */+static bool+dms_set_mode(uint32_t mode)+{+ uint8_t request[11] = { 0, 0, 0, DMS_SET_OPERATING_MODE & 0xff, DMS_SET_OPERATING_MODE >> 8, 4, 0, 1, 1, 0, (uint8_t)mode };+ uint32_t before;+ int error;++ if (dms_port == NULL || dms_port_number == 0) {+ return false;+ }+ {+ /* first, what mode the modem is in and why (the answer is said by dms_receive()) */+ uint8_t question[7] = { 0, 0, 0, DMS_GET_OPERATING_MODE & 0xff, DMS_GET_OPERATING_MODE >> 8, 0, 0 };++ dms_transaction = (uint16_t)(dms_transaction + 1 == 0 ? 1 : dms_transaction + 1);+ question[1] = (uint8_t)dms_transaction;+ question[2] = (uint8_t)(dms_transaction >> 8);+ before = dms_answers;+ if (qrtr_send(dms_port, dms_node, dms_port_number, question, sizeof(question)) == 0) {+ for (unsigned ms = 0; ms < 3000 && dms_answers == before; ms++) {+ sleep_ms(1);+ }+ }+ }+ dms_transaction = (uint16_t)(dms_transaction + 1 == 0 ? 1 : dms_transaction + 1);+ request[1] = (uint8_t)dms_transaction;+ request[2] = (uint8_t)(dms_transaction >> 8);+ before = dms_answers;+ printf("mpss: %u: about to ask DMS (node %u port %u) for operating mode %u (0 online, 1 low power): message %#x\n", step, dms_node,+ dms_port_number, mode, DMS_SET_OPERATING_MODE);+ snprintf(last_write, sizeof(last_write), "step %u asked the modem to go online", step);+ hold();+ error = qrtr_send(dms_port, dms_node, dms_port_number, request, sizeof(request));+ if (error != 0) {+ printf("mpss: %u: NOT SENT (%d)\n", step, error);+ return false;+ }+ for (unsigned ms = 0; ms < 5000 && dms_answers == before; ms++) {+ sleep_ms(1);+ }+ if (dms_answers == before) {+ printf("mpss: %u: DMS DOES NOT ANSWER within 5 s\n", step);+ return false;+ }+ tell("mpss: %u: mode %u: %s (result %u, error %u)\n", step, mode, dms_result == 0 ? "THE MODEM AGREES" : "THE MODEM REFUSES",+ dms_result, dms_error);+ return dms_result == 0;+}++/*+ * Online. A modem that boots offline (as this one does) does not go straight online: Qualcomm's+ * allow offline -> low power -> online (ModemManager and qmicli users meet the same refusal, error+ * 52, and go the long way), so that is tried when the short way is refused.+ */+static bool+dms_online(void)+{+ if (dms_set_mode(DMS_ONLINE)) {+ return true;+ }+ if (dms_error == 52 && dms_set_mode(1)) {+ sleep_ms(2000);+ return dms_set_mode(DMS_ONLINE);+ }+ return false;+}++static bool+step_nas(void)+{+ uint32_t wait_seconds = 90, online = 1;+ uint64_t began, asked_online = 0;+ bool registered = false, is_online = false;++ if (!glink.channel[CHANNEL_QRTR].open) {+ return false;+ }+ PE_parse_boot_argn("modem_nas_wait", &wait_seconds, sizeof(wait_seconds));+ PE_parse_boot_argn("modem_online", &online, sizeof(online));+ announce("look the modem's services up as they come (they take some 30 s): DMS asked to put the modem online as soon as it is "+ "there (modem_online=0: not), NAS asked for the serving system and the signal every second, until the modem is registered "+ "with a signal read or modem_nas_wait seconds (default 90) are over");+ dms_port = qrtr_open(dms_receive, dms_server, NULL);+ if (dms_port == NULL || qrtr_lookup(dms_port, QMI_DMS, 0, 0) != 0) {+ printf("mpss: %u: no port for DMS\n", step);+ return false;+ }+ if (redwood_nas_client_open(&nas, &qrtr_ops, nas_changed, NULL) != 0) {+ printf("mpss: %u: no port for NAS\n", step);+ return false;+ }+ nas_opened = true;+ began = now_us();+ while (now_us() - began < wait_seconds * 1000000ull) {+ if (online && !is_online && dms_port_number != 0 && now_us() - asked_online > 5000000ull) {+ asked_online = now_us();+ is_online = dms_online(); /* (asked again every five seconds until the modem agrees) */+ }+ if (nas.connected) {+ nas_polls += redwood_nas_client_poll(&nas) == 0;+ }+ sleep_ms(1000);+ if (nas.state.registered && nas.state.has_signal) {+ registered = true;+ break;+ }+ }+ cellular_up = nas.connected;+ if (ipa_loaded && !option("modem_ipa_quiet")) {+ ipa_status("at the end of step 11");+ ipa_space_faults("at the end of step 11");+ }+ {+ struct smp2p words;++ if (dsp_words(&words)) {+ show_words("the modem's item now", &words);+ }+ }+ tell("mpss: %u: after %llu ms: DMS %s, the modem %s; NAS %s (node %u port %u), asked %u times, answered %u, %u changes; %s; "+ "signal %s%d dBm (%s), %u bars\n", step, (now_us() - began) / 1000, dms_port_number ? "there" : "NOT THERE", is_online ?+ "online" : "NOT PUT ONLINE", nas.connected ? "there" : "NOT THERE", nas.peer_node, nas.peer_port, nas_polls, nas.answers,+ nas_changes, registered ? "REGISTERED WITH A NETWORK" : nas.state.registered ? "registered" : "not registered",+ nas.state.has_signal ? "" : "none read, ", nas.state.has_signal ? nas.state.dbm : 0, radio_name(nas.state.radio_interface),+ nas.state.bars);+ return nas.connected;+}++/* With modem=1: NAS asked every two seconds while the modem runs, on a thread of its own. */+static volatile bool nas_wanted, nas_runs;++static void+nas_thread(__unused void *parameter, __unused wait_result_t result)+{+ while (nas_wanted) {+ if (nas_opened && nas.connected) {+ nas_polls += redwood_nas_client_poll(&nas) == 0;+ }+ for (unsigned ms = 0; ms < 2000 && nas_wanted; ms += 100) {+ sleep_ms(100);+ }+ }+ nas_runs = false;+ thread_terminate(current_thread());+}++static void+nas_start(void)+{+ thread_t thread;++ nas_wanted = nas_runs = true;+ if (kernel_thread_start(nas_thread, NULL, &thread) != KERN_SUCCESS) {+ nas_wanted = nas_runs = false;+ return;+ }+ thread_deallocate(thread);+}++/* The questions end before the link does. */+static void+nas_stop(void)+{+ if (nas_runs) {+ nas_wanted = false;+ for (unsigned ms = 0; ms < 3000 && nas_runs; ms++) {+ sleep_ms(1);+ }+ }+ if (nas_opened) {+ redwood_nas_client_close(&nas);+ nas_opened = false;+ }+ if (dms_port != NULL) {+ qrtr_close(dms_port);+ dms_port = NULL;+ }+ rmtfs_stop();+ cellular_up = cellular_registered = cellular_bars = cellular_dbm = cellular_radio = 0;+}++/* ---- the steps, and their end ----------------------------------------------------------------------- */++static bool (*const steps[STEPS])(void) = {+ step_survey, step_firmware, step_mailbox, step_header, step_image, step_our_item, step_start, step_queues, step_link,+ step_qrtr, step_nas,+};++static bool keep; /* the DSP and the link stay up while the system runs (modem_keep, modem=1) */++static void+steps_over(uint32_t done)+{+ lck_mtx_lock(&dsp_lock);+ if (dsp_started && done == last_step && keep) {+ printf("mpss: THE MODEM STAYS ON while the system runs%s\n", link_runs ? ", and the link with it" : "");+ } else if (dsp_started) {+ dsp_shutdown("the steps are over");+ }+ lck_mtx_unlock(&dsp_lock);+}++/* When the run ends in order: a DSP that runs is shut down, for the next kernel to start again. */+void+redwood_mpss_stop(void)+{+ nas_stop(); /* the questions first, while the modem still listens */+ lck_mtx_lock(&dsp_lock);+ if (dsp_started) {+ unsigned services = 0;+ uint32_t node = qrtr_peer;+ bool down;++ for (unsigned i = 0; i < sizeof(qrtr_servers) / sizeof(qrtr_servers[0]); i++) {+ services += qrtr_servers[i].there;+ }+ down = dsp_shutdown("the run is about to end");+ /* (said again at the end: a long run's log keeps its beginning and its end, and the middle may be gone) */+ tell("mpss: the modem is off: it had been up%s (node %u, %u services; NAS %s, %s, %u bars), it %s, and TrustZone %s\n",+ node ? " and linked" : ", NOT LINKED", node, services, nas.connected ? "there" : "NOT THERE", nas.state.registered ?+ "registered" : "not registered", nas.state.bars, stopped_by_its_word ? "stopped when asked" :+ "DID NOT SAY IT HAD STOPPED", down ? "has shut it down" : "DID NOT SHUT IT DOWN");+ }+ lck_mtx_unlock(&dsp_lock);+}++/* The same on a panic's way back to the loader: one processor, interrupts off, no lock, no sleep. */+void+redwood_mpss_quiesce(void)+{+ if (!dsp_started) {+ return;+ }+ leaving = true;+ new_from = ~0u;+ dsp_shutdown("the kernel is on its way back to the loader");+}++static void+run_steps(void)+{+ uint32_t done = 0;++ for (step = 1; step <= last_step; step++) {+ if (!steps[step - 1]()) {+ printf("mpss: step %u: FAILED; the steps after it are not attempted\n", step);+ break;+ }+ printf("mpss: step %u: done\n", step);+ done = step;+ }+ step = done;+ new_from = STEPS + 1;+ steps_over(done);+ printf("mpss: over after step %u; %s\n", done, last_write);+}++/* modem_step=N: the steps once the root is there, then back to the loader if modem_return=1 says so. */+static void+steps_thread(__unused void *parameter, __unused wait_result_t result)+{+ for (unsigned waited = 0; waited < 300 && rootvnode == NULLVP; waited++) {+ sleep_ms(200);+ }+ if (rootvnode == NULLVP) {+ printf("mpss: no root file system within a minute: the steps are not performed\n");+ } else {+ run_steps();+ }+ if (option("modem_return")) {+ printf("mpss: back to the loader (modem_return)\n");+ loader_return();+ }+ thread_terminate(current_thread());+}++static void+whole_thread(__unused void *parameter, __unused wait_result_t result)+{+ uint64_t began = now_us();+ unsigned services = 0;++ /* (the firmware may be on the root file system, which is not mounted yet when this begins: a minute at most) */+ for (unsigned waited = 0; waited < 300 && rootvnode == NULLVP; waited++) {+ sleep_ms(200);+ }+ run_steps();+ for (unsigned i = 0; i < sizeof(qrtr_servers) / sizeof(qrtr_servers[0]); i++) {+ services += qrtr_servers[i].there;+ }+ if (step == STEPS && dsp_started) {+ tell("mpss: THE MODEM IS UP AND LINKED, %llu ms after the kernel began on it (the root's waited for included): node %u, %u "+ "services; NAS %s, %s, %u bars (%d dBm, %s); it stays on, asked every two seconds\n", (now_us() - began) / 1000, qrtr_peer,+ services, nas.connected ? "there" : "NOT THERE", nas.state.registered ? "REGISTERED" : "not registered", nas.state.bars,+ nas.state.has_signal ? nas.state.dbm : 0, radio_name(nas.state.radio_interface));+ nas_start();+ } else {+ tell("mpss: THE MODEM DID NOT COME UP: step %u failed (%s); see the kernel's own log for the steps\n", step + 1,+ last_write);+ }+ thread_terminate(current_thread());+}++/* Called while the system starts (bsd/dev/redwood_input.c), after the Wi-Fi's start has been set going. */+void+redwood_mpss_init(void)+{+ uint32_t on = 0;+ thread_t thread;++ PE_parse_boot_argn("modem_step", &last_step, sizeof(last_step));+ if (last_step == 0 && (!PE_parse_boot_argn("modem", &on, sizeof(on)) || on != 1)) {+ return;+ }+ if (pe_arm_get_soc_base_phys() != REDWOOD_SOC_BASE) {+ printf("mpss: modem=1 and modem_step=N are for the phone's modem, and this is not the phone\n");+ return;+ }+ if (last_step == 0) {+ /* all of it, on a thread of its own, and left up */+ last_step = STEPS;+ keep = true;+ quiet = true;+ pause_ms = 0;+ listen_seconds = 1;+ new_from = STEPS + 1;+ if (kernel_thread_start(whole_thread, NULL, &thread) != KERN_SUCCESS) {+ printf("mpss: no thread: the modem is not brought up\n");+ return;+ }+ thread_deallocate(thread);+ tell("mpss: the modem is being brought up beside the system's start (modem=1)\n");+ return;+ }+ if (last_step > STEPS) {+ last_step = STEPS;+ }+ keep = option("modem_keep");+ new_from = last_step;+ PE_parse_boot_argn("modem_new", &new_from, sizeof(new_from));+ PE_parse_boot_argn("modem_pause", &pause_ms, sizeof(pause_ms));+ printf("mpss: steps up to %u; those from %u on are new and wait %u ms after each announcement\n", last_step, new_from,+ pause_ms);+ /*+ * The image is on the root (85 MB: too much for the small image), which is not mounted yet: the+ * steps wait for it on a thread of their own, while the system goes on starting (sb_delay=N+ * keeps SpringBoard out of the way meanwhile).+ */+ if (kernel_thread_start(steps_thread, NULL, &thread) != KERN_SUCCESS) {+ printf("mpss: no thread: the steps are not performed\n");+ return;+ }+ thread_deallocate(thread);+ printf("mpss: the steps wait for the root file system, and the system goes on starting\n");+}diff --git a/xnu/files/bsd/dev/redwood_nas.c b/xnu/files/bsd/dev/redwood_nas.cnew file mode 100644index 0000000..4ea9105--- /dev/null+++ b/xnu/files/bsd/dev/redwood_nas.c@@ -0,0 +1,178 @@+/*+ * Small QMI NAS parser. Wire IDs and TLVs follow libqmi's qmi-service-nas.json:+ * Serving System (0x24), Get Signal Info (0x4f), Signal Info indication (0x51).+ * It only consumes reports. It never changes modem registration or radio power.+ */+#include <sys/redwood_nas.h>+#include <sys/systm.h>+#include <sys/redwood_qrtr.h>++static uint16_t+le16(const uint8_t *p)+{+ return (uint16_t)(p[0] | p[1] << 8);+}++static uint8_t+bars_for(int16_t dbm)+{+ if (dbm < -120) return 1;+ if (dbm < -110) return 2;+ if (dbm < -100) return 3;+ return 4;+}++int+redwood_nas_apply(struct redwood_nas_state *state, const void *frame, size_t length)+{+ const uint8_t *data = frame;+ struct redwood_nas_state next;+ uint16_t message, payload_size;+ size_t at;+ bool serving = false, signal = false, success = false, result_seen = false;+ int16_t level = 0;+ uint8_t radio = 0;++ if (state == NULL || data == NULL || length < 7) return -1;+ if (data[0] != 2 && data[0] != 4) return 0;+ message = le16(data + 3);+ if (message != 0x24 && message != 0x4f && message != 0x51) return 0;+ if ((message == 0x4f && data[0] != 2) || (message == 0x51 && data[0] != 4)) return 0;+ payload_size = le16(data + 5);+ if ((size_t)payload_size != length - 7) return -1;+ next = *state;+ for (at = 7; at < length;) {+ uint8_t type;+ uint16_t size;+ const uint8_t *value;+ if (length - at < 3) return -1;+ type = data[at];+ size = le16(data + at + 1);+ at += 3;+ if (size > length - at) return -1;+ value = data + at;+ at += size;+ if (type == 2 && data[0] == 2) {+ if (size != 4) return -1;+ result_seen = true;+ success = le16(value) == 0;+ } else if (message == 0x24 && type == 1) {+ static unsigned shown;++ if (size < 5 || size < (size_t)5 + value[4]) return -1;+ if (shown++ < 4) {+ printf("nas: serving system: registration %u (0 none, 1 registered, 2 searching, 3 denied, 4 unknown), "+ "CS %u, PS %u, network %u, %u radios%s%u\n", value[0], value[1], value[2], value[3], value[4],+ value[4] ? ", the first " : "", value[4] ? value[5] : 0);+ }+ next.registered = value[0] == 1;+ if (!next.registered) {+ next.has_signal = false;+ next.bars = 0;+ }+ serving = true;+ } else if ((message == 0x4f || message == 0x51) && type == 0x17 && size >= 4) {+ level = (int16_t)le16(value); radio = 12; signal = true;+ } else if ((message == 0x4f || message == 0x51) && type == 0x14 && size >= 6 && radio != 12) {+ level = (int16_t)le16(value + 2); radio = 8; signal = true;+ } else if ((message == 0x4f || message == 0x51) && type == 0x13 && size >= 3 && !signal) {+ level = (int8_t)value[0]; radio = 5; signal = true;+ } else if ((message == 0x4f || message == 0x51) && type == 0x12 && size >= 1 && !signal) {+ level = (int8_t)value[0]; radio = 4; signal = true;+ }+ }+ if (data[0] == 2 && (!result_seen || !success)) return 0;+ if (signal && level >= -150 && level <= -30) {+ next.has_signal = true;+ next.dbm = level;+ next.radio_interface = radio;+ next.bars = next.registered ? bars_for(level) : 0;+ }+ if (!serving && !signal) return 0;+ if (next.registered == state->registered && next.has_signal == state->has_signal &&+ next.radio_interface == state->radio_interface && next.dbm == state->dbm && next.bars == state->bars) return 0;+ *state = next;+ return 1;+}++static void+nas_receive(void *context, uint32_t node, uint32_t port, const void *data, size_t length)+{+ struct redwood_nas_client *client = context;++ if (!client->connected || node != client->peer_node || port != client->peer_port) return;+ client->answers++;+ if (redwood_nas_apply(&client->state, data, length) == 1 && client->changed != NULL)+ client->changed(client->context, &client->state);+}++static void+nas_server(void *context, bool arrived, uint32_t service, uint32_t version, uint32_t instance,+ uint32_t node, uint32_t port)+{+ struct redwood_nas_client *client = context;+ (void)version;+ (void)instance;+ if (service != 3) return; /* QMI NAS */+ if (arrived && !client->connected) {+ client->connected = true;+ client->peer_node = node;+ client->peer_port = port;+ if (client->port != NULL) (void)redwood_nas_client_poll(client);+ } else if (!arrived && client->connected && node == client->peer_node && port == client->peer_port) {+ client->connected = false;+ client->state = (struct redwood_nas_state){0};+ if (client->changed != NULL) client->changed(client->context, &client->state);+ }+}++int+redwood_nas_client_open(struct redwood_nas_client *client, const struct redwood_qrtr_ops *transport,+ void (*changed)(void *, const struct redwood_nas_state *), void *context)+{+ int error;+ if (client == NULL || transport == NULL || transport->open == NULL || transport->lookup == NULL ||+ transport->send == NULL || transport->close == NULL) return -1;+ *client = (struct redwood_nas_client){0};+ client->transport = transport;+ client->changed = changed;+ client->context = context;+ client->port = transport->open(nas_receive, nas_server, client);+ if (client->port == NULL) return -1;+ error = transport->lookup(client->port, 3, 0, 0);+ if (error != 0) {+ redwood_nas_client_close(client);+ return error;+ }+ return 0;+}++int+redwood_nas_client_poll(struct redwood_nas_client *client)+{+ uint8_t request[7] = {0};+ const uint16_t messages[2] = {0x24, 0x4f};+ int error;+ if (client == NULL || client->port == NULL || !client->connected) return -1;+ for (unsigned i = 0; i < 2; i++) {+ client->transaction++;+ if (client->transaction == 0) client->transaction++;+ request[1] = (uint8_t)client->transaction;+ request[2] = (uint8_t)(client->transaction >> 8);+ request[3] = (uint8_t)messages[i];+ request[4] = (uint8_t)(messages[i] >> 8);+ error = client->transport->send(client->port, client->peer_node, client->peer_port, request, sizeof(request));+ if (error != 0) return error;+ }+ return 0;+}++void+redwood_nas_client_close(struct redwood_nas_client *client)+{+ if (client != NULL && client->port != NULL) {+ client->transport->close(client->port);+ client->port = NULL;+ client->connected = false;+ }+}diff --git a/xnu/files/bsd/dev/redwood_qmi.c b/xnu/files/bsd/dev/redwood_qmi.cindex e88ec7b..130154d 100644--- a/xnu/files/bsd/dev/redwood_qmi.c+++ b/xnu/files/bsd/dev/redwood_qmi.c@@ -759,9 +759,9 @@ qmi_server(void *context, bool arrived, uint32_t service, uint32_t version, uint}int-qmi_handle_init(struct qmi_handle *qmi, size_t largest, const struct qmi_ops *ops, const struct qmi_msg_handler *handlers)+redwood_qmi_handle_init_transport(struct qmi_handle *qmi, size_t largest, const struct qmi_ops *ops,+ const struct qmi_msg_handler *handlers, const struct redwood_qrtr_ops *transport){- const struct redwood_qrtr_ops *transport = qmi_transport;struct qmi_state *s;qmi->sock = NULL;@@ -791,6 +791,12 @@ qmi_handle_init(struct qmi_handle *qmi, size_t largest, const struct qmi_ops *opreturn 0;}+int+qmi_handle_init(struct qmi_handle *qmi, size_t largest, const struct qmi_ops *ops, const struct qmi_msg_handler *handlers)+{+ return redwood_qmi_handle_init_transport(qmi, largest, ops, handlers, qmi_transport);+}+/* Nobody may be waiting in qmi_txn_wait() on this handle any more. */voidqmi_handle_release(struct qmi_handle *qmi)diff --git a/xnu/files/bsd/dev/redwood_ufs.c b/xnu/files/bsd/dev/redwood_ufs.cindex efa2fda..6d2d6da 100644--- a/xnu/files/bsd/dev/redwood_ufs.c+++ b/xnu/files/bsd/dev/redwood_ufs.c@@ -1062,6 +1062,41 @@ ufs_entry(const char *name)return BLOCK_MAX_PARTS;}+/*+ * For the kernel's own drivers (the modem's file systems, bsd/dev/redwood_mpss.c): bytes+ * [offset, offset + len) of the partition of that name (or of its _a or _b slot), and how big it+ * is. ENXIO without the storage, ENOENT without the partition; nothing is written.+ */+int+redwood_ufs_read_partition(const char *name, uint64_t offset, uint8_t *dst, uint64_t len, uint64_t *bytes)+{+ uint32_t part;++ if (ufs_page == NULL) {+ return ENXIO;+ }+ part = ufs_entry(name);+ for (const char *slot = "_a"; part == BLOCK_MAX_PARTS && slot[0] != 0; slot = slot[1] == 'a' ? "_b" : "") {+ char slotted[48];++ snprintf(slotted, sizeof(slotted), "%s%s", name, slot);+ part = ufs_entry(slotted);+ }+ if (part == BLOCK_MAX_PARTS) {+ return ENOENT;+ }+ if (bytes != NULL) {+ *bytes = ufs_part_bytes(part);+ }+ if (len == 0) {+ return 0;+ }+ if (offset + len > ufs_part_bytes(part)) {+ return EINVAL;+ }+ return ufs_read(part, offset, dst, len);+}+/** Which logical unit each unit the loader lists is (their device paths do not say): unit k is* taken to be LUN k (ordinary logical units only: never a well-known one), and it is so only ifdiff --git a/xnu/files/bsd/sys/redwood_input.h b/xnu/files/bsd/sys/redwood_input.hindex ff84b74..57c5122 100644--- a/xnu/files/bsd/sys/redwood_input.h+++ b/xnu/files/bsd/sys/redwood_input.h@@ -45,6 +45,7 @@ void redwood_buttons_init(void); /* boot-arg buttons=1: the phone's powervoid redwood_ufs_init(void); /* ufs=2 to xnuloader: the phone's storage, read only, through the loader (ufs_shared_pa=) */void redwood_ufs_report(const char *when); /* where reading the storage has spent its time so far */bool redwood_ufs_dev(dev_t dev); /* one of the storage's devices */+int redwood_ufs_read_partition(const char *name, uint64_t offset, uint8_t *dst, uint64_t len, uint64_t *bytes); /* for drivers */void redwood_battery_init(void); /* boot-arg battery=1: Qualcomm UEFI charger state shared by xnuloader */int redwood_battery_temperature(void); /* the battery's temperature in C, -273 while it is not known *//* one transfer at a time through the PMIC arbiter for this system (bsd/dev/redwood_buttons.c) */@@ -78,6 +79,7 @@ void redwood_adreno_stop(void); /* before that, where the ending is no pvoid redwood_adreno_minute(void); /* once a minute: a line about how warm the GPU is, if it is in use */void redwood_wifi_init(void); /* boot-arg wifi=1: the phone's Wi-Fi; wifi_step=N: the first N steps of it */void redwood_adsp_init(void); /* boot-arg adsp=1: the phone's audio DSP started and linked; adsp_step=N: the first N steps */+void redwood_mpss_init(void); /* boot-arg modem=1: the phone's modem started and linked; modem_step=N: the first N steps */void redwood_wifi_stop(void); /* where a run ends in order: the Wi-Fi DSP, if it was started, shut down */int redwood_wifi_say(const char *format, ...) __printflike(1, 2); /* a line of the Wi-Fi steps' log */int redwood_wifi_tell(const char *format, ...) __printflike(1, 2); /* ... one that is said even with wifi=1 */diff --git a/xnu/files/bsd/sys/redwood_nas.h b/xnu/files/bsd/sys/redwood_nas.hnew file mode 100644index 0000000..9fe7677--- /dev/null+++ b/xnu/files/bsd/sys/redwood_nas.h@@ -0,0 +1,43 @@+/* Minimal, transport-independent QMI NAS network state for cellular UI clients. */+#ifndef _SYS_REDWOOD_NAS_H_+#define _SYS_REDWOOD_NAS_H_++#include <stdbool.h>+#include <stddef.h>+#include <stdint.h>++struct redwood_nas_state {+ bool registered;+ bool has_signal;+ uint8_t radio_interface; /* QMI NAS: 4 GSM, 5 UMTS, 8 LTE, 12 NR */+ int16_t dbm; /* LTE/NR RSRP; GSM/UMTS RSSI */+ uint8_t bars; /* 0 until registration and a valid signal reading */+};++/* Apply one complete QMI NAS frame. 1: state changed; 0: ignored; -1: malformed. */+int redwood_nas_apply(struct redwood_nas_state *state, const void *frame, size_t length);++struct redwood_qrtr_ops;+struct redwood_qrtr_port;++/* One NAS service client on a specified processor's QRTR transport. Callbacks are serialized+ * by that transport; the caller must synchronize a state read on any other thread. */+struct redwood_nas_client {+ const struct redwood_qrtr_ops *transport;+ struct redwood_qrtr_port *port;+ struct redwood_nas_state state;+ void (*changed)(void *context, const struct redwood_nas_state *state);+ void *context;+ uint32_t peer_node, peer_port;+ uint16_t transaction;+ uint32_t answers; /* NAS frames received, changed or not */+ bool connected;+};++int redwood_nas_client_open(struct redwood_nas_client *client, const struct redwood_qrtr_ops *transport,+ void (*changed)(void *, const struct redwood_nas_state *), void *context);+/* Get Serving System and Get Signal Info. Poll regularly until indications are configured. */+int redwood_nas_client_poll(struct redwood_nas_client *client);+void redwood_nas_client_close(struct redwood_nas_client *client);++#endifdiff --git a/xnu/files/bsd/sys/redwood_qmi.h b/xnu/files/bsd/sys/redwood_qmi.hindex 2be5586..62c3b81 100644--- a/xnu/files/bsd/sys/redwood_qmi.h+++ b/xnu/files/bsd/sys/redwood_qmi.h@@ -151,6 +151,8 @@ struct qmi_handle {struct socket *sock; /* what kernel_connect() is given; not NULL while the handle is in use */};+struct redwood_qrtr_ops;+struct qmi_txn {struct qmi_handle *qmi;uint16_t id;@@ -166,6 +168,9 @@ struct qmi_txn {* errno: -ENXIO where there is no router (redwood_qmi_set_transport() has not been called).*/int qmi_handle_init(struct qmi_handle *qmi, size_t largest, const struct qmi_ops *ops, const struct qmi_msg_handler *handlers);+/* Bind a handle to a specific processor's QRTR transport (e.g. WPSS or MPSS). */+int redwood_qmi_handle_init_transport(struct qmi_handle *qmi, size_t largest, const struct qmi_ops *ops,+ const struct qmi_msg_handler *handlers, const struct redwood_qrtr_ops *transport);void qmi_handle_release(struct qmi_handle *qmi);/* Ask for a service: ops->new_server is called for every server of it there is and will be. */int qmi_add_lookup(struct qmi_handle *qmi, unsigned int service, unsigned int version, unsigned int instance);diff --git a/xnu/files/osfmk/arm64/loader_return.c b/xnu/files/osfmk/arm64/loader_return.cindex a058bb5..bde30ee 100644--- a/xnu/files/osfmk/arm64/loader_return.c+++ b/xnu/files/osfmk/arm64/loader_return.c@@ -51,6 +51,8 @@ extern void redwood_wifi_stop(void);extern void redwood_wifi_quiesce(void);extern void redwood_adsp_stop(void);extern void redwood_adsp_quiesce(void);+extern void redwood_mpss_stop(void);+extern void redwood_mpss_quiesce(void);extern void redwood_ddr_quiesce(void);extern void redwood_haptic_quiesce(void);extern void redwood_torch_quiesce(void);@@ -373,6 +375,7 @@ loader_return(void)redwood_adreno_quiesce(); /* the GPU block, if it was started, has tables in memory that is about to be reused */redwood_wifi_quiesce(); /* the Wi-Fi DSP, if it still runs (a panic), is not left to itself */redwood_adsp_quiesce(); /* nor the audio DSP */+ redwood_mpss_quiesce(); /* nor the modem */redwood_ddr_quiesce(); /* and the vote for the memory's bandwidth taken away */redwood_haptic_quiesce(); /* the vibration motor's chip back into reset, if it was let out */redwood_torch_quiesce(); /* the flashlight off, if it was lit */@@ -440,6 +443,7 @@ loader_return_timer(__unused thread_call_param_t p0, __unused thread_call_param_redwood_adreno_stop(); /* a GPU kept for the system is given back while there is a thread to wait in */redwood_wifi_stop(); /* and the Wi-Fi DSP, if it runs, shut down */redwood_adsp_stop(); /* and the audio DSP */+ redwood_mpss_stop(); /* and the modem */}if (PE_parse_boot_argn("return_cpu", &cpu, sizeof(cpu)) && cpu <= (uint32_t)ml_get_max_cpu_number() &&cpu_xcall((int)cpu, loader_return_test, (void *)(uintptr_t)(as_panic ? 2 : 1)) == KERN_SUCCESS) {diff --git a/xnu/redwood.patch b/xnu/redwood.patchindex 8f7dd87..2f6b425 100644--- a/xnu/redwood.patch+++ b/xnu/redwood.patch@@ -62,10 +62,10 @@ index a4d25040f38f..aad72993d139 100644kern_debug.o_INCFLAGS += -I$(SRCROOT)/osfmkdiff --git a/bsd/conf/files b/bsd/conf/files-index 44274016a4cf..9cd24b42caca 100644+index 44274016a4cf..08878fedb36e 100644--- a/bsd/conf/files+++ b/bsd/conf/files-@@ -109,6 +109,37 @@ bsd/dev/munge.c optional config_requires_u32_munging+@@ -109,6 +109,39 @@ bsd/dev/munge.c optional config_requires_u32_mungingbsd/dev/unix_startup.c standardbsd/dev/monotonic.c optional monotonic@@ -91,19 +91,21 @@ index 44274016a4cf..9cd24b42caca 100644+bsd/dev/redwood_mic.c standard+bsd/dev/redwood_wifi.c standard+bsd/dev/redwood_adsp.c standard++bsd/dev/redwood_mpss.c standard+bsd/dev/redwood_image.c standard+bsd/dev/redwood_virtio_blk.c standard+bsd/dev/redwood_sparse.c standard+bsd/dev/redwood_apfs_check.c optional apfs+bsd/dev/redwood_sound.c standard+bsd/dev/redwood_qmi.c standard++bsd/dev/redwood_nas.c standard+bsd/dev/redwood_wlfw.c standard+bsd/dev/redwood_lx.c standard+bsd/dev/redwood_sta.c standardbsd/dev/dev_perfmon.c optional config_perfmonbsd/libkern/crc32.c standard-@@ -155,6 +186,14 @@ bsd/miscfs/devfs/devfs_vfsops.c optional devfs+@@ -155,6 +188,14 @@ bsd/miscfs/devfs/devfs_vfsops.c optional devfsbsd/miscfs/mockfs/mockfs_fsnode.c optional mockfsbsd/miscfs/mockfs/mockfs_vfsops.c optional mockfsbsd/miscfs/mockfs/mockfs_vnops.c optional mockfs@@ -118,7 +120,7 @@ index 44274016a4cf..9cd24b42caca 100644bsd/miscfs/routefs/routefs_ops.c optional routefs-@@ -361,6 +400,7 @@ bsd/security/audit/audit_syscalls.c standard+@@ -361,6 +402,7 @@ bsd/security/audit/audit_syscalls.c standardbsd/security/audit/audit_worker.c optional config_auditbsd/pthread/pthread_shims.c standard@@ -126,7 +128,7 @@ index 44274016a4cf..9cd24b42caca 100644bsd/pthread/pthread_priority.c standardbsd/pthread/pthread_workqueue.c standard-@@ -383,6 +423,7 @@ bsd/kern/kern_debug.c standard+@@ -383,6 +425,7 @@ bsd/kern/kern_debug.c standardbsd/kern/kern_symfile.c standardbsd/kern/kern_descrip.c standardbsd/kern/kern_guarded.c standard@@ -1865,7 +1867,7 @@ index 5bd6f23dd10c..14e0f17409f5 100644ifneq ($(filter T6000 T8101 VMAPPLE T6000 T8101 VMAPPLE,$(CURRENT_MACHINE_CONFIG)),)LDFLAGS_KERNEL_ONLY += -rdynamic -Wl,-force_load,$(KDKROOT)/System/Library/KernelSupport/lib$(CURRENT_MACHINE_CONFIG).os.$(CURRENT_KERNEL_CONFIG).adiff --git a/makedefs/MakeInc.top b/makedefs/MakeInc.top-index 0316792a3f42..bbef0a4b26d4 100644+index 0316792a3f42..b9c195b09cd1 100644--- a/makedefs/MakeInc.top+++ b/makedefs/MakeInc.top@@ -285,7 +285,7 @@ endif@@ -1873,7 +1875,7 @@ index 0316792a3f42..bbef0a4b26d4 100644# $(warning VARIANT_CONFIGS is $(VARIANT_CONFIGS))-MEMORY_SIZE := $(shell /usr/sbin/sysctl -n hw.memsize)-+MEMORY_SIZE := $(shell awk '/MemTotal/{printf "%d", $$2*1024}' /proc/meminfo)++MEMORY_SIZE := $(shell if [ -r /proc/meminfo ]; then awk '/MemTotal/{printf "%d", $$2*1024}' /proc/meminfo; else sysctl -n hw.memsize; fi)# Assume LTO scaling by default, unless it is being explicitly passed on the command-lineLARGE_BUILD_FOOTPRINT := $(if $(BUILD_LTO),$(BUILD_LTO),1)