Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.
C
GPU: SpringBoard drawn by the Adreno through Mesa's freedreno
vcvkk825806aff752
diff --git a/README.md b/README.mdindex 94981ca..687d02c 100644--- a/README.md+++ b/README.md@@ -22,7 +22,7 @@|---|---|| Экран блокировки, разблокировка свайпом, домашний экран с полным набором значков (две страницы) | Звук, сеть (Wi-Fi/сотовая), Bluetooth, камера || Системные жесты: свайп снизу — «домой» и переключатель приложений, свайп от левого края — «назад», Пункт управления | Обои (чёрный фон) |-| Запуск приложений: Настройки (вглубь), Часы, Календарь, Фото, Заметки и др. | GPU для iOS: пока только программный рендер (сам Adreno уже заведён и заливает кадр, см. ниже) |+| Запуск приложений: Настройки (вглубь), Часы, Календарь, Фото, Заметки и др. | Metal (приложения, рисующие через него сами); отрисовка самим Adreno есть, но пока опытная — см. «GPU» || Сенсорный экран телефона (Goodix GT9916), мультитач, пробуждение касанием | Без `cpus=N` работает одно ядро процессора || Несколько ядер процессора (`cpus=4`: четыре больших ядра телефона) | || Кнопки «Домой»/«Блокировка»/громкость (в QEMU — клавишами) | Зарядка под нашим ядром (телефон только разряжается) |@@ -152,22 +152,37 @@ XL_SECS=900 XL_CMDLINE="touchscreen=1 cpufreq=1 bootcpu=7 cpus=4" ./xl run #что ядра 1–3 малые, и сажает на них интерфейс. Паника «Ticket spinlock … timeout» значила бы, что ядрам тесново времени (в QEMU загрузчик сам увеличивает эти сроки в 20 раз): помогает `ml-timeout-global-scale=4`.-## GPU (в работе)--Adreno 642L телефона заведён под нашим ядром: загружается прошивка управляющего процессора (GMU), включается-3D-ядро, TrustZone принимает zap-шейдер, ядро GPU исполняет команды и заливает кадр, который драйвер экрана выводит-на панель (`docs/gpu-first-frame.png`). Это опыт за параметром `adreno=N` (шаги описаны в начале-`bsd/dev/redwood_adreno.c`), iOS пока по-прежнему рисует процессором. Прошивки GPU берутся с самого телефона-(`adb pull /vendor/firmware/a660_{gmu.bin,sqe.fw,zap.mdt,zap.b00,zap.b01,zap.b02}` в каталог `fw-gpu/`, в git они-не попадают) и едут в малом образе.--Сторона iOS тоже проверена: в QuartzCore 16.5 есть рабочий рендер через OpenGL ES, на который сервер окон-переходит сам, если нет Metal. `GL=1 REST=1 userland/build.sh` (перед этим `tools/mk-mesa.sh` — Mesa 25.0.7 под-userland iOS, минута сборки) кладёт в малый образ Mesa и прослойку `userland/src/eagl.c`, которая подменяет шесть-методов `EAGLContext` и таблицу функций OpenGL ES: SpringBoard рисуется через этот путь и в QEMU, и на телефоне-(`docs/gles-on-phone.png`). Пока за ним стоит программный драйвер Mesa (softpipe), то есть это медленнее обычного-режима и нужно только как проверка; на экране блокировки при этом не рисуются часы и обои. Дальше — интерфейс-ядра под драйвер Adreno из Mesa (freedreno) и замена программного драйвера на него.+## GPU++**iOS на телефоне может рисовать графический процессор** — Adreno 642L, через открытый драйвер Mesa freedreno+и наш драйвер в ядре (`docs/adreno-springboard.png`: треугольник, экран блокировки, домашний экран).++Как это устроено:++- ядро (`bsd/dev/redwood_adreno.c`, `iokit/Kernel/RedwoodGPU.cpp`) поднимает Adreno — прошивка управляющего+ процессора (GMU), питание 3D-ядра, zap-шейдер через TrustZone, микрокод — и отдаёт его userland'у через+ клиент IOKit с подмножеством интерфейса msm DRM (буферы, отправка команд, ожидание);+- `tools/mk-mesa.sh` собирает Mesa 25.0.7 под userland iOS (freedreno и программный softpipe, минута сборки);+- `userland/src/eagl.c` встаёт за системный `EAGLContext`: в QuartzCore 16.5 есть рендер через OpenGL ES, на+ который сервер окон переходит сам, если нет Metal.++Запуск (прошивки GPU один раз снять со своего телефона:+`adb pull /vendor/firmware/a660_{gmu.bin,sqe.fw,zap.mdt,zap.b00,zap.b01,zap.b02}` в каталог `fw-gpu/`; в git они+не попадают):++```sh+tools/mk-mesa.sh+GL=1 REST=1 userland/build.sh+XL_CMDLINE="touchscreen=1 cpufreq=1 bootcpu=7 cpus=4 adreno=21 adreno_skip=0x1c00 adreno_pause=0 adreno_keep=1" ./xl run+```++При втором и следующих запусках за одну загрузку телефона добавлять `adreno_zap=0`. Без параметров `adreno…`+тот же образ рисует программным драйвером Mesa (в QEMU — всегда так).++Состояние — первый рабочий вариант, не режим по умолчанию: проверены экран блокировки и домашний экран; кадр+рисуется за 16–17 мс, но 14 из них — копирование готовой картинки (память кадра пока не общая с GPU); прерываний+нет (ожидание опросом); сеанс, оборванный паникой или `./xl stop`, оставляет GPU включённым — после него+`./xl reboot`. Приложения, использующие Metal сами, от этого ничего не получают.## Сборка из исходниковdiff --git a/prebuilt/SHA256SUMS b/prebuilt/SHA256SUMSindex 382bac3..38e69ae 100644--- a/prebuilt/SHA256SUMS+++ b/prebuilt/SHA256SUMS@@ -1,6 +1,6 @@-Built 2026-10-12 23:02 from this tree.+Built 2026-10-12 23:49 from this tree.-30e96b37407464e40391ef99a0b933eafebb5105129ba1cf4aa254636eca4772 prebuilt/mach.development.bcm2837+b79c92cb557d7b995cee74fb3aa6a895d07b191c3c69d8e40ea19ae148e36a84 prebuilt/mach.development.bcm28371332a598d2dec00a1d8e930044e4f00a12ad44c6e5e50eecf4b7fe7fae803bf2 prebuilt/loader.efid2a35a4a1d6658ffb3696914f727661b52c9333de8a3af8a31642b8023ce03ac prebuilt/userland/child1a413befd511384052e1962c1394adce20f718e1f2657ce63ab09828ebc4bd1b prebuilt/userland/cpu_testdiff --git a/prebuilt/mach.development.bcm2837 b/prebuilt/mach.development.bcm2837index b5dda07..e268a2c 100755Binary files a/prebuilt/mach.development.bcm2837 and b/prebuilt/mach.development.bcm2837 differdiff --git a/xnu/files/bsd/dev/redwood_adreno.c b/xnu/files/bsd/dev/redwood_adreno.cindex 7bd0fe7..0593e8d 100644--- a/xnu/files/bsd/dev/redwood_adreno.c+++ b/xnu/files/bsd/dev/redwood_adreno.c@@ -61,6 +61,9 @@* the end of our log to storage every two seconds, and after a silent reset* that note is all there is.* adreno_fine=1 the newest step waits like that before every single access+ * adreno_keep=1 with adreno=21: the GPU is not stopped again but kept for the system to draw+ * with, through the service RedwoodGPU (see "the GPU kept on" below); it is+ * given back five seconds before a run's time is up* adreno_zap=0 step 19 has been done since the phone started: not again* adreno_blind=1 step 20 starts the command processor with its microcode nowhere, and* adreno_stream1=N step 18 leaves the core's second stream out (0) or gives it an empty@@ -81,6 +84,7 @@* vendor kernel agrees on every offset used here, and is the only source for the frequency counter.*/#include <kern/clock.h>+#include <kern/thread_call.h>#include <libkern/libkern.h>#include <mach/mach_types.h>#include <mach/vm_param.h>@@ -97,6 +101,7 @@ extern uint64_t kvtophys(vm_offset_t va); /* pmap_paddr_t */static uint32_t last_step, skip_steps, pause_ms = 2200, fine, watch_seconds = 20;static uint32_t step; /* the one being performed */+static uint32_t speed_bin; /* which of the GPUs of this kind the fuses say this is (step 2) */static uint32_t newest_step; /* the highest one asked for: the one that has not been seen to work */static char last_write[96] = "nothing written";static bool quiet; /* repeating accesses that have been announced and seen to work: say nothing */@@ -428,6 +433,7 @@ step_fuses(void)bin = get8(&qfprom, "speed bin, low", QFPROM_CORRECTED + QFPROM_GPU_SPEED_BIN);bin |= get8(&qfprom, "speed bin, high", QFPROM_CORRECTED + QFPROM_GPU_SPEED_BIN + 1) << 8;bin = bin >> 5 & 0xff;+ speed_bin = bin;gaming = get8(&qfprom, "gaming bin", QFPROM_CORRECTED + QFPROM_GPU_GAMING_BIN) >> 5 & 1;/* dt: qcom,gpu-models: 0xac and 0xbe are sold as Adreno 643, anything else is the 642L */printf("adreno: %u: speed bin %u: an Adreno %s; gaming bin %u\n", step, bin, bin == 0xac || bin == 0xbe ? "643" : "642L",@@ -530,6 +536,8 @@ cx_is_on(void)return (rd(gpucc.base + GPUCC_CX_GDSCR) | rd(gpucc.base + GPUCC_CX_HW_CTRL)) & GDSC_POWER_ON;}+static void stale_stop(void);+/** The switch hangs on the rail "cx.lvl", which is up for as long as the SoC runs at all; the* vendor's driver (gdsc-regulator.c) clears the collapse bit and waits for bit 31 of the register@@ -547,8 +555,19 @@ step_cx_power(void)gdscr = get(&gpucc, "cx_gdscr", GPUCC_CX_GDSCR);get(&gpucc, "cx_hw_ctrl", GPUCC_CX_HW_CTRL);if (cx_is_on()) {- printf("adreno: %u: CX is on already\n", step);- return true;+ /*+ * A kernel that kept the GPU for the system to use (adreno_keep) leaves it running when+ * its run ends. What it left -- the GMU's processor, the 3D core's power, the GMU's+ * votes in RPMh, the SMMU's contexts pointing into memory that is somebody else's now --+ * is taken down as gpu_block_stop() would have, and the start is from nothing.+ */+ printf("adreno: %u: CX is on already: an earlier kernel left the GPU block running; stopping it first\n", step);+ stale_stop();+ if (cx_is_on()) {+ printf("adreno: %u: CX stays on: going on with it as it is\n", step);+ return true;+ }+ gdscr = rd(gpucc.base + GPUCC_CX_GDSCR);}if (gdscr & GDSC_HW_CONTROL) {printf("adreno: %u: the switch is under hardware control: not ours to operate\n", step);@@ -1185,7 +1204,7 @@ to_memory(vm_offset_t start, vm_size_t size)settle();}-#define SPACE_TABLES 48+#define SPACE_TABLES 1024 /* 2 GB of addresses, were they all in use */struct gpu_space {const char *name;@@ -1196,7 +1215,9 @@ struct gpu_space {struct {uint64_t phys;uint64_t *table;+ bool changed; /* since it was last written out of this processor's cache (space_commit()) */} tables[SPACE_TABLES];+ uint64_t *last_entry, *last_table; /* the level 2 entry looked up last, and its table */};static uint64_t *@@ -1223,6 +1244,30 @@ space_new_table(struct gpu_space *space)}/* The table an entry of a higher table leads to; made if the entry is empty. */+/* An entry of one of the tables has been written: that table is to go out to memory. */+static void+space_changed(struct gpu_space *space, const uint64_t *entry)+{+ for (uint32_t i = space->table_count; i-- > 0;) {+ if ((uintptr_t)entry - (uintptr_t)space->tables[i].table < 4096) {+ space->tables[i].changed = true;+ return;+ }+ }+}++/* The SMMU reads the tables from memory, not from this processor's cache. */+static void+space_commit(struct gpu_space *space)+{+ for (uint32_t i = 0; i < space->table_count; i++) {+ if (space->tables[i].changed) {+ to_memory((vm_offset_t)space->tables[i].table, 4096);+ space->tables[i].changed = false;+ }+ }+}+static uint64_t *space_next(struct gpu_space *space, uint64_t *entry){@@ -1239,6 +1284,7 @@ space_next(struct gpu_space *space, uint64_t *entry)table = space_new_table(space);if (table != NULL) {*entry = kvtophys((vm_offset_t)table) | PTE_TABLE;+ space_changed(space, entry);}return table;}@@ -1253,11 +1299,22 @@ space_map(struct gpu_space *space, uint64_t address, uint64_t phys, uint64_t attreturn false;}level2 = space_next(space, &space->root[address >> 30 & 0x1ff]);- level3 = level2 != NULL ? space_next(space, &level2[address >> 21 & 0x1ff]) : NULL;+ if (level2 == NULL) {+ return false;+ }+ if (space->last_entry == &level2[address >> 21 & 0x1ff]) {+ level3 = space->last_table; /* page after page of one buffer: the same table */+ } else {+ level3 = space_next(space, &level2[address >> 21 & 0x1ff]);+ space->last_entry = &level2[address >> 21 & 0x1ff];+ space->last_table = level3;+ }if (level3 == NULL) {+ space->last_entry = NULL;return false;}- level3[address >> 12 & 0x1ff] = phys | attributes | PTE_PAGE;+ level3[address >> 12 & 0x1ff] = phys == 0 && attributes == 0 ? 0 : phys | attributes | PTE_PAGE;+ space_changed(space, level3);return true;}@@ -2416,7 +2473,9 @@ static struct gpu_space second_space = {static bool second_space_is_on, second_stream_is_on;static struct gpu_block core_ring, core_sqe, core_scratch, core_target;static bool core_space_is_on, core_runs, zap_is_loaded;-static uint32_t ring_words; /* written to the ring so far; it is never filled here */+#define RING_WORDS (CORE_RING_SIZE / 4)+static uint32_t ring_at; /* where the next word goes: the ring goes round */+static uint32_t ring_sent; /* up to where the core has been told to read */static uint32_todd_parity(uint32_t v)@@ -2427,7 +2486,8 @@ odd_parity(uint32_t v)static voidring(uint32_t word){- ((uint32_t *)core_ring.memory)[ring_words++] = word;+ ((uint32_t *)core_ring.memory)[ring_at] = word;+ ring_at = (ring_at + 1) % RING_WORDS;}/* The header of a command with so many words after it. */@@ -2445,28 +2505,41 @@ ring_register(uint32_t number, uint32_t value)ring(value);}+/* What has been written to the ring since the last time: out to memory, and the core told how far to read. */+static void+ring_send(void)+{+ if (ring_at >= ring_sent) {+ to_memory(core_ring.memory + 4 * ring_sent, 4 * (ring_at - ring_sent));+ } else {+ to_memory(core_ring.memory + 4 * ring_sent, 4 * (RING_WORDS - ring_sent));+ to_memory(core_ring.memory, 4 * ring_at);+ }+ ring_sent = ring_at;+ wr(gpu.base + CP_RB_WPTR, ring_at);+}+/* Let the 3D core have what is in the ring, and wait until it has taken all of it and rests. */static boolring_run(const char *what){uint32_t taken = ~0u, status = ~0u;- to_memory(core_ring.memory, ring_words * 4);- printf("adreno: %u: about to have the 3D core %s: the ring up to word %u\n", step, what, ring_words);+ printf("adreno: %u: about to have the 3D core %s: the ring up to word %u\n", step, what, ring_at);snprintf(last_write, sizeof(last_write), "step %u had the 3D core %s", step, what);hold(true);- wr(gpu.base + CP_RB_WPTR, ring_words);- for (int i = 0; i < 20000 && (taken != ring_words || (status & ~1u)); i++) {+ ring_send();+ for (int i = 0; i < 20000 && (taken != ring_at || (status & ~1u)); i++) {sleep_us(100);taken = rd(gpu.base + CP_RB_RPTR);status = rd(gpu.base + RBBM_STATUS);}- if (taken == ring_words && !(status & ~1u) && rd(gpu.base + CP_HW_FAULT) == 0 && !space_faulted(&core_space)) {+ if (taken == ring_at && !(status & ~1u) && rd(gpu.base + CP_HW_FAULT) == 0 && !space_faulted(&core_space)) {printf("adreno: %u: the 3D core has done it (ring read up to %u, status %#x)\n", step, taken, status);return true;}printf("adreno: %u: the 3D core does not get through: ring read up to %u of %u, status %#x, interrupts %#x, "- "command processor: fault %#x, interrupts %#x, protection %#x\n", step, taken, ring_words, status,+ "command processor: fault %#x, interrupts %#x, protection %#x\n", step, taken, ring_at, status,rd(gpu.base + RBBM_INT_0_STATUS), rd(gpu.base + CP_HW_FAULT), rd(gpu.base + CP_INTERRUPT_STATUS),rd(gpu.base + CP_PROTECT_STATUS));space_faulted(&core_space);@@ -2641,6 +2714,16 @@ step_core_start(void)put(&gpu, "cp_mem_pool_size", CP_MEM_POOL_SIZE, 128);put(&gpu, "pc_dbg_eco_cntl", PC_DBG_ECO_CNTL, 0x00200200);put(&gpu, "cp_ahb_cntl", CP_AHB_CNTL, 1);+ /*+ * How textures in the GPU's own tiled and compressed layouts are laid out in memory (Linux's+ * a6xx_calc_ubwc_config, a6xx_set_ubwc_config for this GPU: highest bank bit 14, and the+ * rest as below); Mesa is told the same (redwood_gpu_parameter()).+ */+ put(&gpu, "rb_nc_mode_cntl", GPU_WORD(0x8e08), 1u << 11 | 1u << 4 | 1u << 1);+ put(&gpu, "tpl1_nc_mode_cntl", GPU_WORD(0xb604), 1u << 1);+ put(&gpu, "sp_nc_mode_cntl", GPU_WORD(0xae02), 2u << 4 | 1u << 1);+ put(&gpu, "uche_mode_cntl", GPU_WORD(0x0e01), 1u << 21);+ put(&gpu, "rbbm_nc_mode_cntl", GPU_WORD(0x0534), 1);put(&gpu, "rbbm_perfctr_cntl", RBBM_PERFCTR_CNTL, 1);put(&gpu, "rbbm_interface_hang_int_cntl", RBBM_INTERFACE_HANG_INT_CNTL, 1u << 30 | 0x1fffff);put(&gpu, "uche_client_pf", UCHE_CLIENT_PF, 1u << 7 | 1);@@ -2854,6 +2937,43 @@ step_frame(void)* core given back, the GMU told to drop its votes ("slumber") and stopped, its RPMh channel put* to sleep; then the processor into reset and its stream sent nowhere.*/+/* What step 4 does when it finds the GPU block running: see there. Nothing here has been set up by this kernel. */+static void+stale_stop(void)+{+ uint32_t gx;++ need(&gmu);+ need(&rscc);+ put(&gmu, "cm3_sysreset", GMU_CM3_SYSRESET, 1);+ gx = get(&gpucc, "gx_gdscr", GPUCC_GX_GDSCR);+ if (gx & GDSC_POWER_ON) {+ /* the 3D core's switch is the GMU's to operate, and the GMU is gone: vendor's a6xx_gmu_suspend */+ put(&gpucc, "gx_gdscr", GPUCC_GX_GDSCR, gx | GDSC_SW_COLLAPSE);+ wait_for(&gpucc, GPUCC_GX_GDSCR, GDSC_POWER_ON, 0, 100);+ get(&gpucc, "gx_gdscr", GPUCC_GX_GDSCR);+ }+ if (rd(rscc.base + RSCC_SEQ_MEM_0_DRV0) == 0xeaaae5a0 && !(rd(rscc.base + RSCC_RSC_STATUS0_DRV0) & RSCC_ASLEEP)) {+ put(&gmu, "rscc_control_req", GMU_RSCC_CONTROL_REQ, 1);+ wait_for(&rscc, RSCC_RSC_STATUS0_DRV0, RSCC_ASLEEP, RSCC_ASLEEP, 100);+ get(&rscc, "rsc_status0_drv0", RSCC_RSC_STATUS0_DRV0);+ put(&gmu, "rscc_control_req", GMU_RSCC_CONTROL_REQ, 0);+ }+ if (smmu_layout()) {+ for (uint32_t match = 0; match < 3; match++) {+ put(&smmu, "smr", SMMU_SMR(match), 0);+ put(&smmu, "s2cr", SMMU_S2CR(match), SMMU_S2CR_FAULT);+ put(&smmu, "cb sctlr", (smmu_pages + match) * smmu_page + SMMU_CB_SCTLR, 0);+ }+ }+ put(&gpucc, "cx_gdscr", GPUCC_CX_GDSCR, rd(gpucc.base + GPUCC_CX_GDSCR) | GDSC_SW_COLLAPSE);+ for (int i = 0; i < 1000 && cx_is_on(); i++) {+ sleep_us(100);+ }+ get(&gpucc, "cx_gdscr", GPUCC_CX_GDSCR);+ get(&gpucc, "cx_hw_ctrl", GPUCC_CX_HW_CTRL);+}+static voidgpu_block_stop(void){@@ -2955,6 +3075,272 @@ redwood_adreno_quiesce(void)}}+/* ---- the GPU kept on, for the system to draw with --------------------------------------------------- */++/*+ * With boot-arg adreno_keep=1 the steps are not undone: the GPU block stays as step 21 leaves it,+ * the 3D core powered and taking commands, and the service RedwoodGPU (iokit/Kernel/RedwoodGPU.cpp)+ * hands it to programs through the interface of Linux's msm driver, which is what Mesa's+ * freedreno speaks. That file keeps the buffers and the clients; what is the GPU's is here:+ * addresses in the 3D core's SMMU context, the ring, and the word by which the core says how+ * far it has got (a "fence": a submission's number, written by the core when it is through+ * with it). The caller holds one lock around all of these.+ */+#define GPU_CHIP_ID 0x06030500ull /* dt: qcom,chipid */+#define GPU_ADDRESSES_START 0x20000000ull /* above everything the steps use; below 4 GB, as Linux keeps them */+#define GPU_ADDRESSES_SIZE 0xe0000000ull+#define GPU_FENCE_AT 0x40 /* in the scratch page */+#define CP_SCRATCH_REG2 0x0885+#define CP_IB1_BASE GPU_WORD(0x0928)+#define CP_IB1_REM_SIZE GPU_WORD(0x092a)+#define CP_IB2_BASE GPU_WORD(0x092b)+#define CP_IB2_REM_SIZE GPU_WORD(0x092d)+#define CP_INDIRECT_BUFFER 0x3f+#define EVENT_INVALIDATE_DEPTH 0x18+#define EVENT_INVALIDATE_COLOR 0x19++static bool gpu_kept;+static uint32_t gpu_fence; /* the number of the last submission */+static struct {+ uint64_t start, size;+} gpu_free[1024]; /* the addresses not in use, in order */+static unsigned gpu_free_count;++extern void RedwoodGPUStart(void);+extern void RedwoodGPUStop(void); /* takes the clients' lock, then calls redwood_gpu_stop() */+static void gpu_block_stop(void);++/*+ * A run that the loader ends after so many seconds (boot-arg loader_return_after) gives the GPU+ * back a few seconds before that, in order and with time to do it: the next kernel then finds+ * the GPU block as the phone's own start leaves it. A run that ends otherwise (a panic, xl stop)+ * leaves it running, and the next kernel's step 4 has to deal with that.+ */+#define GPU_STOP_AHEAD 5++void+redwood_gpu_stop(void)+{+ if (gpu_kept) {+ gpu_kept = false;+ quiet = false;+ printf("adreno: the run is about to end: the GPU is given back\n");+ gpu_block_stop();+ }+}++static void+gpu_stop_timer(__unused thread_call_param_t p0, __unused thread_call_param_t p1)+{+ RedwoodGPUStop();+}++bool+redwood_gpu_ready(void)+{+ return gpu_kept;+}++/* The answers to msm's GET_PARAM (Linux's adreno_get_param for this GPU; the numbers are msm_drm.h's). */+bool+redwood_gpu_parameter(uint32_t which, uint64_t *value)+{+ switch (which) {+ case 0x01: /* GPU_ID: none, the chip ID says it */+ case 0x06: /* GMEM_BASE */+ case 0x08: /* PP_PGTABLE */+ case 0x09: /* FAULTS */+ case 0x0a: /* SUSPENDS */+ case 0x11: /* RAYTRACING */+ *value = 0;+ return true;+ case 0x02: /* GMEM_SIZE */+ *value = 512 * 1024;+ return true;+ case 0x03: /* CHIP_ID, with the speed bin above it */+ *value = GPU_CHIP_ID | (uint64_t)speed_bin << 32;+ return true;+ case 0x04: /* MAX_FREQ */+ *value = 315000000;+ return true;+ case 0x05: /* TIMESTAMP */+ *value = mach_absolute_time();+ return true;+ case 0x07: /* PRIORITIES */+ *value = 1;+ return true;+ case 0x10: /* HIGHEST_BANK_BIT */+ *value = 14;+ return true;+ case 0x12: /* UBWC_SWIZZLE */+ *value = 6;+ return true;+ case 0x13: /* MACROTILE_MODE */+ *value = 1;+ return true;+ default: /* among them VA_START, VA_SIZE: addresses are ours to give */+ return false;+ }+}++/* Addresses for a buffer of so many bytes; 0: none left. */+uint64_t+redwood_gpu_address(uint64_t size)+{+ size = (size + 0xffff) & ~0xffffull;+ for (unsigned i = 0; i < gpu_free_count; i++) {+ if (gpu_free[i].size >= size) {+ uint64_t address = gpu_free[i].start;++ gpu_free[i].start += size;+ gpu_free[i].size -= size;+ if (gpu_free[i].size == 0) {+ memmove(&gpu_free[i], &gpu_free[i + 1], (--gpu_free_count - i) * sizeof(gpu_free[0]));+ }+ return address;+ }+ }+ return 0;+}++void+redwood_gpu_address_free(uint64_t address, uint64_t size)+{+ unsigned at;++ size = (size + 0xffff) & ~0xffffull;+ for (at = 0; at < gpu_free_count && gpu_free[at].start < address; at++) {+ }+ if (at > 0 && gpu_free[at - 1].start + gpu_free[at - 1].size == address) {+ gpu_free[at - 1].size += size;+ if (at < gpu_free_count && address + size == gpu_free[at].start) {+ gpu_free[at - 1].size += gpu_free[at].size;+ memmove(&gpu_free[at], &gpu_free[at + 1], (--gpu_free_count - at) * sizeof(gpu_free[0]));+ }+ } else if (at < gpu_free_count && address + size == gpu_free[at].start) {+ gpu_free[at].start = address;+ gpu_free[at].size += size;+ } else if (gpu_free_count < sizeof(gpu_free) / sizeof(gpu_free[0])) {+ memmove(&gpu_free[at + 1], &gpu_free[at], (gpu_free_count++ - at) * sizeof(gpu_free[0]));+ gpu_free[at].start = address;+ gpu_free[at].size = size;+ } /* (or else these addresses are lost) */+}++/* One 4K page of ours at an address of the 3D core's; after the last of a buffer's, redwood_gpu_mapped(). */+bool+redwood_gpu_map(uint64_t address, uint64_t phys, bool read_only)+{+ return space_map(&core_space, address, phys, PTE_CORE | (read_only ? PTE_READ_ONLY : 0));+}++void+redwood_gpu_mapped(void)+{+ space_commit(&core_space);+}++void+redwood_gpu_unmap(uint64_t address, uint64_t size)+{+ uint32_t context = smmu_context(&core_space);++ for (uint64_t offset = 0; offset < size; offset += 4096) {+ space_map(&core_space, address + offset, 0, 0);+ }+ space_commit(&core_space);+ wr(smmu.base + context + SMMU_CB_TLBIASID, core_space.context);+ wr(smmu.base + context + SMMU_CB_TLBSYNC, 1);+ for (int i = 0; i < 1000 && (rd(smmu.base + context + SMMU_CB_TLBSTATUS) & 1); i++) {+ }+}++/* Memory that the 3D core and a program see without this processor's cache in between: out of that cache. */+void+redwood_gpu_to_memory(void *start, uint64_t size)+{+ to_memory((vm_offset_t)start, size);+}++/* How far the 3D core has got. */+bool+redwood_gpu_done(uint32_t fence)+{+ volatile uint32_t *word = (volatile uint32_t *)(core_scratch.memory + GPU_FENCE_AT);++ to_memory((vm_offset_t)word, 4);+ return (int32_t)(*word - fence) >= 0;+}++uint32_t+redwood_gpu_last(void)+{+ return gpu_fence;+}++/*+ * Command buffers for the 3D core to run, in order: each an address and a count of words. As+ * Linux's a6xx_submit puts them in the ring. Returns the submission's number; 0: the ring is+ * full and the core is not reading it.+ */+uint32_t+redwood_gpu_submit(const uint64_t *addresses, const uint32_t *words, uint32_t count)+{+ uint32_t wanted = 16 + 4 * count, room = 0;++ for (int i = 0; i < 20000; i++) {+ room = (rd(gpu.base + CP_RB_RPTR) + RING_WORDS - ring_at - 1) % RING_WORDS;+ if (room >= wanted) {+ break;+ }+ sleep_us(100);+ }+ if (room < wanted) {+ return 0;+ }+ ring_command(CP_EVENT_WRITE, 1);+ ring(EVENT_INVALIDATE_DEPTH);+ ring_command(CP_EVENT_WRITE, 1);+ ring(EVENT_INVALIDATE_COLOR);+ for (uint32_t i = 0; i < count; i++) {+ ring_command(CP_INDIRECT_BUFFER, 3);+ ring((uint32_t)addresses[i]);+ ring((uint32_t)(addresses[i] >> 32));+ ring(words[i]);+ }+ if (++gpu_fence == 0) {+ gpu_fence = 1;+ }+ ring_register(CP_SCRATCH_REG2, gpu_fence);+ ring_command(CP_EVENT_WRITE, 4);+ ring(EVENT_CACHE_FLUSH);+ ring(CORE_SCRATCH_ADDRESS + GPU_FENCE_AT);+ ring(0);+ ring(gpu_fence);+ ring_send();+ return gpu_fence;+}++/* A submission does not come back: what the 3D core and its SMMU context say. */+void+redwood_gpu_report(const char *why)+{+ uint32_t context = smmu_context(&core_space);++ printf("adreno: gpu: %s: submission %u is the last, the core has done %u; ring: read up to %u, written up to %u; status "+ "%#x, interrupts %#x; command processor: fault %#x, interrupts %#x, protection %#x\n", why, gpu_fence,+ *(volatile uint32_t *)(core_scratch.memory + GPU_FENCE_AT), rd(gpu.base + CP_RB_RPTR), ring_at,+ rd(gpu.base + RBBM_STATUS), rd(gpu.base + RBBM_INT_0_STATUS), rd(gpu.base + CP_HW_FAULT),+ rd(gpu.base + CP_INTERRUPT_STATUS), rd(gpu.base + CP_PROTECT_STATUS));+ printf("adreno: gpu: in the first command buffer at %#llx, %u words to go; in the second at %#llx, %u to go\n",+ (uint64_t)rd(gpu.base + CP_IB1_BASE + 4) << 32 | rd(gpu.base + CP_IB1_BASE), rd(gpu.base + CP_IB1_REM_SIZE),+ (uint64_t)rd(gpu.base + CP_IB2_BASE + 4) << 32 | rd(gpu.base + CP_IB2_BASE), rd(gpu.base + CP_IB2_REM_SIZE));+ step = 0;+ if (space_faulted(&core_space)) {+ wr(smmu.base + context + SMMU_CB_FSR, SMMU_FSR_ALL); /* so that the next one shows */+ }+}+/* ---- all of it together ---------------------------------------------------------------------------- */static bool(*const steps[])(void) = {@@ -3006,6 +3392,27 @@ redwood_adreno_init(void)printf("adreno: step %u: done\n", step);done = step;}+ if (PE_parse_boot_argn("adreno_keep", &only, sizeof(only)) && only) {+ if (done >= 21 && core_runs) {+ gpu_free[0].start = GPU_ADDRESSES_START;+ gpu_free[0].size = GPU_ADDRESSES_SIZE;+ gpu_free_count = 1;+ *(volatile uint32_t *)(core_scratch.memory + GPU_FENCE_AT) = 0;+ to_memory(core_scratch.memory, 4096);+ gpu_kept = true;+ quiet = true;+ printf("adreno: THE GPU STAYS ON for the system to draw with (adreno_keep): the service RedwoodGPU\n");+ RedwoodGPUStart();+ if (PE_parse_boot_argn("loader_return_after", &only, sizeof(only)) && only > GPU_STOP_AHEAD + 5) {+ uint64_t deadline;++ clock_interval_to_deadline(only - GPU_STOP_AHEAD, NSEC_PER_SEC, &deadline);+ thread_call_enter_delayed(thread_call_allocate(gpu_stop_timer, NULL), deadline);+ }+ return;+ }+ printf("adreno: not kept: the steps did not get as far as a 3D core that takes commands\n");+ }/** Punishment for a write may come late (a reset some 17 seconds after it has been seen):* keep the end of the log saying where things stand.diff --git a/xnu/files/bsd/sys/redwood_input.h b/xnu/files/bsd/sys/redwood_input.hindex 5855949..a12e979 100644--- a/xnu/files/bsd/sys/redwood_input.h+++ b/xnu/files/bsd/sys/redwood_input.h@@ -44,6 +44,24 @@ void redwood_goodix_init(void); /* boot-arg touchscreen=1: the phone's tvoid redwood_cpufreq_init(void); /* boot-arg cpufreq=1: the phone's processor at full speed */void redwood_adreno_init(void); /* boot-arg adreno=N: the first N steps of waking the phone's GPU */void redwood_adreno_quiesce(void); /* on the way back to the loader: the GPU block stopped, out of our memory */++/*+ * The GPU kept on (boot-arg adreno_keep=1), for iokit/Kernel/RedwoodGPU.cpp: see "the GPU kept on"+ * in bsd/dev/redwood_adreno.c. One caller at a time.+ */+bool redwood_gpu_ready(void);+bool redwood_gpu_parameter(uint32_t which, uint64_t *value);+uint64_t redwood_gpu_address(uint64_t size);+void redwood_gpu_address_free(uint64_t address, uint64_t size);+bool redwood_gpu_map(uint64_t address, uint64_t phys, bool read_only);+void redwood_gpu_mapped(void);+void redwood_gpu_unmap(uint64_t address, uint64_t size);+void redwood_gpu_to_memory(void *start, uint64_t size);+uint32_t redwood_gpu_submit(const uint64_t *addresses, const uint32_t *words, uint32_t count);+bool redwood_gpu_done(uint32_t fence);+uint32_t redwood_gpu_last(void);+void redwood_gpu_report(const char *why);+void redwood_gpu_stop(void);#endif#endif /* _SYS_REDWOOD_INPUT_H_ */diff --git a/xnu/files/iokit/Kernel/RedwoodGPU.cpp b/xnu/files/iokit/Kernel/RedwoodGPU.cppnew file mode 100644index 0000000..99b6b4b--- /dev/null+++ b/xnu/files/iokit/Kernel/RedwoodGPU.cpp@@ -0,0 +1,658 @@+/*+ * RedwoodGPU: the phone's Adreno for programs to draw with.+ *+ * bsd/dev/redwood_adreno.c wakes the GPU and, with boot-arg adreno_keep=1, keeps it on and+ * starts this service. A program that opens it speaks the interface of Linux's msm DRM driver,+ * the one Mesa's freedreno knows: every request of that interface is a call of the user client+ * with the request's number as the selector and the request's structure passed in and back+ * (userland: tools/mesa/redwood-drm, which stands in for libdrm inside Mesa). What Linux does+ * with mmap() on the device is the client memory of the type that is the buffer's handle.+ *+ * What there is of the interface is what freedreno uses to draw: parameters, buffers (made,+ * told their address in the GPU's address space, mapped, given back), submission of command+ * buffers, and waiting for one to be done. A buffer is wired kernel memory entered into the 3D+ * core's SMMU context; it is zero when made. The program has this processor's cache between+ * itself and the memory, the GPU has not: see submit().+ *+ * Not here yet: interrupts (waiting is asking again every millisecond), more than one program's+ * worth of care (every client's buffers are in the one address space; a client that goes takes+ * its own away), recovery of a GPU that has stopped answering (it is reported, see+ * redwood_gpu_report(), and stays stopped), fences and buffers as file descriptors.+ */++#include <IOKit/IOLib.h>+#include <IOKit/IOLocks.h>+#include <IOKit/IOKitKeys.h>+#include <IOKit/IOPlatformExpert.h>+#include <IOKit/IOService.h>+#include <IOKit/IOUserClient.h>+#include <IOKit/IOBufferMemoryDescriptor.h>+#include <kern/clock.h>+#include <mach/mach_time.h>+#include <mach/vm_param.h>+#include <sys/systm.h>++extern "C" {+/* bsd/dev/redwood_adreno.c ("the GPU kept on"; declared in bsd/sys/redwood_input.h, which is not seen from here) */+bool redwood_gpu_ready(void);+bool redwood_gpu_parameter(uint32_t which, uint64_t * value);+uint64_t redwood_gpu_address(uint64_t size);+void redwood_gpu_address_free(uint64_t address, uint64_t size);+bool redwood_gpu_map(uint64_t address, uint64_t phys, bool read_only);+void redwood_gpu_mapped(void);+void redwood_gpu_unmap(uint64_t address, uint64_t size);+void redwood_gpu_to_memory(void * start, uint64_t size);+uint32_t redwood_gpu_submit(const uint64_t * addresses, const uint32_t * words, uint32_t count);+bool redwood_gpu_done(uint32_t fence);+uint32_t redwood_gpu_last(void);+void redwood_gpu_report(const char * why);+void redwood_gpu_stop(void);++void RedwoodGPUStart(void);+void RedwoodGPUStop(void);+void MiniIOMobileFramebufferShow(const uint8_t * pixels, uint32_t width, uint32_t height, uint32_t rowBytes);+}++/* Linux's include/uapi/drm/msm_drm.h and drm.h: the requests' numbers and structures. */+enum {+ kDrmGemClose = 0x09,+ kMsmGetParam = 0x40,+ kMsmSetParam = 0x41,+ kMsmGemNew = 0x42,+ kMsmGemInfo = 0x43,+ kMsmGemCpuPrep = 0x44,+ kMsmGemCpuFini = 0x45,+ kMsmGemSubmit = 0x46,+ kMsmWaitFence = 0x47,+ kMsmGemMadvise = 0x48,+ kMsmSubmitQueueNew = 0x4a,+ kMsmSubmitQueueClose = 0x4b,+ kMsmSubmitQueueQuery = 0x4c,+ kRedwoodShow = 0x7f, /* ours: pixels of the caller's onto the panel, to see a test's result */+};++struct msm_timespec {+ int64_t seconds, nanoseconds; /* of the clock that goes on while the machine sleeps */+};+struct msm_param {+ uint32_t pipe, param;+ uint64_t value;+ uint32_t len, pad;+};+struct msm_gem_new {+ uint64_t size;+ uint32_t flags, handle;+};+struct msm_gem_info {+ uint32_t handle, info;+ uint64_t value;+ uint32_t len, pad;+};+struct msm_gem_cpu_prep {+ uint32_t handle, op;+ struct msm_timespec timeout;+};+struct msm_gem_submit_cmd {+ uint32_t type, submit_idx, submit_offset, size, pad, nr_relocs;+ uint64_t relocs;+};+struct msm_gem_submit_bo {+ uint32_t flags, handle;+ uint64_t presumed;+};+struct msm_gem_submit {+ uint32_t flags, fence, nr_bos, nr_cmds;+ uint64_t bos, cmds;+ int32_t fence_fd;+ uint32_t queueid;+ uint64_t in_syncobjs, out_syncobjs;+ uint32_t nr_in_syncobjs, nr_out_syncobjs, syncobj_stride, pad;+};+struct msm_wait_fence {+ uint32_t fence, flags;+ struct msm_timespec timeout;+ uint32_t queueid;+};+struct msm_gem_madvise {+ uint32_t handle, madv, retained;+};+struct msm_submitqueue {+ uint32_t flags, prio, id;+};+struct msm_submitqueue_query {+ uint64_t data;+ uint32_t id, param, len, pad;+};+struct drm_gem_close {+ uint32_t handle, pad;+};+struct redwood_show {+ uint64_t pixels;+ uint32_t width, height, rowBytes, pad;+};++#define MSM_INFO_GET_OFFSET 0+#define MSM_INFO_GET_IOVA 1+#define MSM_INFO_SET_NAME 2+#define MSM_INFO_GET_FLAGS 5+#define MSM_BO_GPU_READONLY 0x00000002+#define MSM_PREP_NOSYNC 0x04+#define MSM_SUBMIT_CMD_BUF 1+#define MSM_SUBMIT_FENCE_FD_OUT 0x20000000++#define GPU_BUFFERS 8192 /* a client's, at most */+#define GPU_BUFFER_MAX (256u << 20)+#define GPU_SUBMIT_CMDS 128+#define GPU_SUBMIT_BOS 8192+#define GPU_PATIENCE_MS 5000 /* for a buffer or a submission that somebody waits for without a limit */++static IOLock * gGPULock; /* around everything that is the GPU's (redwood_gpu_...) */+static bool gGPUReported;++/* Wait for the 3D core to be done with submission `fence`; with the lock held, which is let go of while waiting. */+static bool+gpuWait(uint32_t fence, uint64_t milliseconds)+{+ for (uint64_t waited = 0;; waited++) {+ if (redwood_gpu_done(fence)) {+ return true;+ }+ if (waited >= milliseconds) {+ break;+ }+ IOLockUnlock(gGPULock);+ IOSleep(1);+ IOLockLock(gGPULock);+ }+ if (milliseconds >= 1000 && !gGPUReported) {+ gGPUReported = true;+ redwood_gpu_report("no answer");+ }+ return false;+}++/* How long from now a time of msm's is, in milliseconds; 0: now or past. */+static uint64_t+millisecondsUntil(const struct msm_timespec * when)+{+ uint64_t now;+ int64_t left;++ absolutetime_to_nanoseconds(mach_continuous_time(), &now);+ left = when->seconds * (int64_t) NSEC_PER_SEC + when->nanoseconds - (int64_t) now;+ if (left <= 0) {+ return 0;+ }+ return (uint64_t) left / NSEC_PER_MSEC > GPU_PATIENCE_MS ? GPU_PATIENCE_MS : (uint64_t) left / NSEC_PER_MSEC;+}++/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */++class RedwoodGPU : public IOService+{+ OSDeclareDefaultStructors(RedwoodGPU);++public:+ virtual bool start(IOService * provider) APPLE_KEXT_OVERRIDE;+};++OSDefineMetaClassAndStructors(RedwoodGPU, IOService);++bool+RedwoodGPU::start(IOService * provider)+{+ if (!IOService::start(provider)) {+ return false;+ }+ setProperty(kIOUserClientClassKey, "RedwoodGPUUserClient");+ setName("RedwoodGPU");+ registerService();+ return true;+}++void+RedwoodGPUStart(void)+{+ IOService * provider = IOService::getPlatform();+ RedwoodGPU * gpu;++ gGPULock = IOLockAlloc();+ gpu = OSTypeAlloc(RedwoodGPU);+ if (gpu == NULL || provider == NULL || !gpu->init() || !gpu->attach(provider)) {+ OSSafeReleaseNULL(gpu);+ IOLog("RedwoodGPU: cannot create the service\n");+ return;+ }+ if (!gpu->start(provider)) {+ gpu->detach(provider);+ IOLog("RedwoodGPU: the service did not start\n");+ }+ gpu->release();+}++/* The run is ending (redwood_adreno.c): nobody is in the middle of anything when the GPU goes. */+void+RedwoodGPUStop(void)+{+ IOLockLock(gGPULock);+ redwood_gpu_stop();+ IOLockUnlock(gGPULock);+}++/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */++struct gpu_buffer {+ IOBufferMemoryDescriptor * memory; /* NULL: this handle is free */+ uint64_t address; /* the GPU's for it */+ uint64_t size;+ uint32_t flags;+};++class RedwoodGPUUserClient : public IOUserClient+{+ OSDeclareDefaultStructors(RedwoodGPUUserClient);++ task_t fTask;+ struct gpu_buffer * fBuffers; /* handle n is fBuffers[n - 1] */+ uint32_t fNextQueue;+ uint32_t fBufferCount;+ uint64_t fBufferBytes;+ uint32_t fSubmits;++ struct gpu_buffer * buffer(uint32_t handle);+ void freeBuffer(struct gpu_buffer * b);+ IOReturn gemNew(struct msm_gem_new * request);+ IOReturn gemInfo(struct msm_gem_info * request);+ IOReturn submit(struct msm_gem_submit * request);+ IOReturn show(const struct redwood_show * request);++public:+ virtual bool initWithTask(task_t owningTask, void * securityID, UInt32 type, OSDictionary * properties) APPLE_KEXT_OVERRIDE;+ virtual bool start(IOService * provider) APPLE_KEXT_OVERRIDE;+ virtual IOReturn clientClose(void) APPLE_KEXT_OVERRIDE;+ virtual void free(void) APPLE_KEXT_OVERRIDE;+ virtual IOReturn clientMemoryForType(UInt32 type, IOOptionBits * options, IOMemoryDescriptor ** memory) APPLE_KEXT_OVERRIDE;+ virtual IOReturn externalMethod(uint32_t selector, IOExternalMethodArguments * args,+ IOExternalMethodDispatch * dispatch, OSObject * target, void * reference) APPLE_KEXT_OVERRIDE;+};++OSDefineMetaClassAndStructors(RedwoodGPUUserClient, IOUserClient);++bool+RedwoodGPUUserClient::initWithTask(task_t owningTask, void * securityID, UInt32 type, OSDictionary * properties)+{+ if (!IOUserClient::initWithTask(owningTask, securityID, type, properties)) {+ return false;+ }+ fTask = owningTask;+ return true;+}++bool+RedwoodGPUUserClient::start(IOService * provider)+{+ if (!redwood_gpu_ready() || !IOUserClient::start(provider)) {+ return false;+ }+ fBuffers = IONewZero(struct gpu_buffer, GPU_BUFFERS);+ return fBuffers != NULL;+}++/* With the lock held. */+void+RedwoodGPUUserClient::freeBuffer(struct gpu_buffer * b)+{+ if (redwood_gpu_ready()) {+ redwood_gpu_unmap(b->address, b->size);+ redwood_gpu_address_free(b->address, b->size);+ }+ b->memory->release();+ b->memory = NULL;+ fBufferCount--;+ fBufferBytes -= b->size;+}++IOReturn+RedwoodGPUUserClient::clientClose(void)+{+ if (fBuffers != NULL) {+ IOLockLock(gGPULock);+ /* what the client had the GPU do last may still be going on, in these buffers */+ if (redwood_gpu_ready()) {+ gpuWait(redwood_gpu_last(), 1000);+ }+ for (uint32_t i = 0; i < GPU_BUFFERS; i++) {+ if (fBuffers[i].memory != NULL) {+ freeBuffer(&fBuffers[i]);+ }+ }+ IOLockUnlock(gGPULock);+ }+ if (!isInactive()) {+ terminate();+ }+ return kIOReturnSuccess;+}++void+RedwoodGPUUserClient::free(void)+{+ if (fBuffers != NULL) {+ IODelete(fBuffers, struct gpu_buffer, GPU_BUFFERS);+ fBuffers = NULL;+ }+ IOUserClient::free();+}++struct gpu_buffer *+RedwoodGPUUserClient::buffer(uint32_t handle)+{+ if (handle == 0 || handle > GPU_BUFFERS || fBuffers[handle - 1].memory == NULL) {+ return NULL;+ }+ return &fBuffers[handle - 1];+}++IOReturn+RedwoodGPUUserClient::gemNew(struct msm_gem_new * request)+{+ uint64_t size = round_page_64(request->size);+ struct gpu_buffer * b = NULL;+ IOBufferMemoryDescriptor * memory;+ bool mapped = true;++ if (size == 0 || size > GPU_BUFFER_MAX) {+ return kIOReturnBadArgument;+ }+ memory = IOBufferMemoryDescriptor::inTaskWithOptions(kernel_task, kIODirectionInOut | kIOMemoryKernelUserShared,+ (vm_size_t) size, PAGE_SIZE);+ if (memory == NULL) {+ IOLog("RedwoodGPU: no memory for a buffer of %llu bytes (%u buffers, %llu MB with this client)\n", size, fBufferCount,+ fBufferBytes >> 20);+ return kIOReturnNoMemory;+ }+ bzero(memory->getBytesNoCopy(), size);+ redwood_gpu_to_memory(memory->getBytesNoCopy(), size);++ IOLockLock(gGPULock);+ for (uint32_t i = 0; i < GPU_BUFFERS; i++) {+ if (fBuffers[i].memory == NULL) {+ b = &fBuffers[i];+ request->handle = i + 1;+ break;+ }+ }+ if (b == NULL || (b->address = redwood_gpu_address(size)) == 0) {+ IOLockUnlock(gGPULock);+ memory->release();+ IOLog("RedwoodGPU: no %s left for a buffer of %llu bytes\n", b ? "addresses" : "handles", size);+ return kIOReturnNoMemory;+ }+ for (uint64_t offset = 0; offset < size && mapped; offset += 4096) {+ IOByteCount length = 0;+ addr64_t phys = memory->getPhysicalSegment(offset, &length, kIOMemoryMapperNone);++ mapped = phys != 0 && redwood_gpu_map(b->address + offset, phys, request->flags & MSM_BO_GPU_READONLY);+ }+ redwood_gpu_mapped();+ if (!mapped) {+ redwood_gpu_unmap(b->address, size);+ redwood_gpu_address_free(b->address, size);+ IOLockUnlock(gGPULock);+ memory->release();+ IOLog("RedwoodGPU: a buffer of %llu bytes cannot be entered into the GPU's address space\n", size);+ return kIOReturnNoMemory;+ }+ b->memory = memory;+ b->size = size;+ b->flags = request->flags;+ fBufferCount++;+ fBufferBytes += size;+ IOLockUnlock(gGPULock);+ return kIOReturnSuccess;+}++IOReturn+RedwoodGPUUserClient::gemInfo(struct msm_gem_info * request)+{+ struct gpu_buffer * b;+ IOReturn ret = kIOReturnSuccess;++ IOLockLock(gGPULock);+ b = buffer(request->handle);+ if (b == NULL) {+ ret = kIOReturnBadArgument;+ } else if (request->info == MSM_INFO_GET_OFFSET) {+ request->value = (uint64_t) request->handle << 32; /* nobody maps by it here */+ } else if (request->info == MSM_INFO_GET_IOVA) {+ request->value = b->address;+ } else if (request->info == MSM_INFO_GET_FLAGS) {+ request->value = b->flags;+ } else if (request->info != MSM_INFO_SET_NAME) { /* a name for somebody's debugger: not kept */+ ret = kIOReturnUnsupported;+ }+ IOLockUnlock(gGPULock);+ return ret;+}++IOReturn+RedwoodGPUUserClient::submit(struct msm_gem_submit * request)+{+ struct msm_gem_submit_bo * bos = NULL;+ struct msm_gem_submit_cmd * cmds = NULL;+ uint64_t addresses[GPU_SUBMIT_CMDS];+ uint32_t words[GPU_SUBMIT_CMDS], count = 0;+ IOReturn ret = kIOReturnBadArgument;++ if (request->nr_cmds == 0 || request->nr_cmds > GPU_SUBMIT_CMDS || request->nr_bos == 0 || request->nr_bos > GPU_SUBMIT_BOS) {+ IOLog("RedwoodGPU: a submission of %u command buffers in %u buffers: not taken\n", request->nr_cmds, request->nr_bos);+ return kIOReturnBadArgument;+ }+ bos = IONew(struct msm_gem_submit_bo, request->nr_bos);+ cmds = IONew(struct msm_gem_submit_cmd, request->nr_cmds);+ if (bos == NULL || cmds == NULL) {+ ret = kIOReturnNoMemory;+ goto out;+ }+ if (copyin((user_addr_t) request->bos, bos, request->nr_bos * sizeof(*bos)) != 0 ||+ copyin((user_addr_t) request->cmds, cmds, request->nr_cmds * sizeof(*cmds)) != 0) {+ ret = kIOReturnVMError;+ goto out;+ }+ IOLockLock(gGPULock);+ /*+ * The program writes its buffers through this processor's cache and the GPU reads memory,+ * and the other way round for what the GPU draws: everything the submission names goes out+ * of the cache now. (Mapped write-combined, as on Linux, none of this would be needed: see+ * redwood_drm_map() in tools/mesa/redwood-drm.)+ */+ for (uint32_t i = 0; i < request->nr_bos; i++) {+ struct gpu_buffer * b = buffer(bos[i].handle);++ if (b != NULL) {+ redwood_gpu_to_memory(b->memory->getBytesNoCopy(), b->size);+ }+ }+ for (uint32_t i = 0; i < request->nr_cmds; i++) {+ struct gpu_buffer * b = cmds[i].submit_idx < request->nr_bos ? buffer(bos[cmds[i].submit_idx].handle) : NULL;++ if (cmds[i].type != MSM_SUBMIT_CMD_BUF) {+ continue; /* the other kinds are for a GPU shared between address spaces */+ }+ if (b == NULL || cmds[i].nr_relocs != 0 || (cmds[i].size & 3) || (cmds[i].submit_offset & 3) ||+ (uint64_t) cmds[i].submit_offset + cmds[i].size > b->size) {+ IOLog("RedwoodGPU: command buffer %u of a submission: buffer %u, offset %u, %u bytes, %u relocations: not taken\n",+ i, cmds[i].submit_idx, cmds[i].submit_offset, cmds[i].size, cmds[i].nr_relocs);+ IOLockUnlock(gGPULock);+ goto out;+ }+ addresses[count] = b->address + cmds[i].submit_offset;+ words[count++] = cmds[i].size / 4;+ }+ request->fence = redwood_gpu_submit(addresses, words, count);+ if (request->flags & MSM_SUBMIT_FENCE_FD_OUT) {+ request->fence_fd = -1;+ }+ if (request->fence == 0 && !gGPUReported) {+ gGPUReported = true;+ redwood_gpu_report("the ring is full");+ }+ if (fSubmits++ < 4) {+ IOLog("RedwoodGPU: submission %u: %u command buffers, the first at %#llx, %u words\n", request->fence, count,+ addresses[0], words[0]);+ }+ IOLockUnlock(gGPULock);+ ret = request->fence != 0 ? kIOReturnSuccess : kIOReturnTimeout;+out:+ if (bos != NULL) {+ IODelete(bos, struct msm_gem_submit_bo, request->nr_bos);+ }+ if (cmds != NULL) {+ IODelete(cmds, struct msm_gem_submit_cmd, request->nr_cmds);+ }+ return ret;+}++/* Pixels of the caller's (32 bits each, blue first) copied to the panel: for a test to show what it has drawn. */+IOReturn+RedwoodGPUUserClient::show(const struct redwood_show * request)+{+ uint64_t size = (uint64_t) request->rowBytes * request->height;+ uint8_t * pixels;+ IOReturn ret = kIOReturnSuccess;++ if (request->width == 0 || request->width > 4096 || request->height == 0 || request->height > 4096 ||+ request->rowBytes < 4 * request->width) {+ return kIOReturnBadArgument;+ }+ pixels = (uint8_t *) IOMallocData(size);+ if (pixels == NULL) {+ return kIOReturnNoMemory;+ }+ if (copyin((user_addr_t) request->pixels, pixels, size) != 0) {+ ret = kIOReturnVMError;+ } else {+ MiniIOMobileFramebufferShow(pixels, request->width, request->height, request->rowBytes);+ }+ IOFreeData(pixels, size);+ return ret;+}++IOReturn+RedwoodGPUUserClient::clientMemoryForType(UInt32 type, IOOptionBits * options, IOMemoryDescriptor ** memory)+{+ struct gpu_buffer * b;+ IOReturn ret = kIOReturnBadArgument;++ IOLockLock(gGPULock);+ b = buffer(type);+ if (b != NULL) {+ b->memory->retain();+ *memory = b->memory;+ *options = 0;+ ret = kIOReturnSuccess;+ }+ IOLockUnlock(gGPULock);+ return ret;+}++IOReturn+RedwoodGPUUserClient::externalMethod(uint32_t selector, IOExternalMethodArguments * args,+ IOExternalMethodDispatch * dispatch, OSObject * target, void * reference)+{+ union {+ struct msm_param param;+ struct msm_gem_new gemNew;+ struct msm_gem_info gemInfo;+ struct msm_gem_cpu_prep cpuPrep;+ struct msm_gem_submit submit;+ struct msm_wait_fence waitFence;+ struct msm_gem_madvise madvise;+ struct msm_submitqueue queue;+ struct msm_submitqueue_query query;+ struct drm_gem_close close;+ struct redwood_show show;+ } r;+ size_t size = args->structureInputSize;+ struct gpu_buffer * b;+ IOReturn ret = kIOReturnSuccess;+ uint64_t zero = 0;++ if (args->structureInput == NULL || size > sizeof(r) || fBuffers == NULL) {+ return kIOReturnBadArgument;+ }+ if (!redwood_gpu_ready() && selector != kRedwoodShow) {+ return kIOReturnNotReady; /* given back: the run is ending */+ }+ bzero(&r, sizeof(r));+ memcpy(&r, args->structureInput, size);++ switch (selector) {+ case kMsmGetParam:+ ret = redwood_gpu_parameter(r.param.param, &r.param.value) ? kIOReturnSuccess : kIOReturnBadArgument;+ break;+ case kMsmSetParam: /* the program's name, for Linux's lists of who uses the GPU */+ case kMsmGemCpuFini:+ case kMsmSubmitQueueClose:+ break;+ case kMsmGemNew:+ ret = gemNew(&r.gemNew);+ break;+ case kMsmGemInfo:+ ret = gemInfo(&r.gemInfo);+ break;+ case kMsmGemCpuPrep: /* is the GPU done with a buffer: with all it has been given, here */+ IOLockLock(gGPULock);+ if (!gpuWait(redwood_gpu_last(), (r.cpuPrep.op & MSM_PREP_NOSYNC) ? 0 : millisecondsUntil(&r.cpuPrep.timeout))) {+ ret = kIOReturnBusy;+ }+ IOLockUnlock(gGPULock);+ break;+ case kMsmGemSubmit:+ ret = submit(&r.submit);+ break;+ case kMsmWaitFence:+ IOLockLock(gGPULock);+ if (!gpuWait(r.waitFence.fence, millisecondsUntil(&r.waitFence.timeout))) {+ ret = kIOReturnTimeout;+ }+ IOLockUnlock(gGPULock);+ break;+ case kMsmGemMadvise:+ r.madvise.retained = 1;+ break;+ case kMsmSubmitQueueNew:+ r.queue.id = ++fNextQueue;+ break;+ case kMsmSubmitQueueQuery: /* how often the GPU has faulted on this client's commands */+ if (r.query.data != 0 && r.query.len >= sizeof(zero)) {+ copyout(&zero, (user_addr_t) r.query.data, sizeof(zero));+ }+ break;+ case kDrmGemClose:+ IOLockLock(gGPULock);+ b = buffer(r.close.handle);+ if (b == NULL) {+ ret = kIOReturnBadArgument;+ } else {+ gpuWait(redwood_gpu_last(), 1000);+ freeBuffer(b);+ }+ IOLockUnlock(gGPULock);+ break;+ case kRedwoodShow:+ ret = show(&r.show);+ break;+ default:+ IOLog("RedwoodGPU: request %#x (%zu bytes) is not known here\n", selector, size);+ ret = kIOReturnUnsupported;+ break;+ }+ if (ret == kIOReturnSuccess && args->structureOutput != NULL) {+ if (args->structureOutputSize > size) {+ args->structureOutputSize = (uint32_t) size;+ }+ memcpy(args->structureOutput, &r, args->structureOutputSize);+ }+ return ret;+}diff --git a/xnu/redwood.patch b/xnu/redwood.patchindex b01d62e..a197a43 100644--- a/xnu/redwood.patch+++ b/xnu/redwood.patch@@ -1319,16 +1319,17 @@ index 5219fb0d9ed6..29978e9cff43 100644}}diff --git a/iokit/conf/files b/iokit/conf/files-index 5fb0a52062e0..8dab3e0a364d 100644+index 5fb0a52062e0..5edb1f225d18 100644--- a/iokit/conf/files+++ b/iokit/conf/files-@@ -79,6 +79,10 @@ iokit/Kernel/IOSubMemoryDescriptor.cpp optional iokitcpp+@@ -79,6 +79,11 @@ iokit/Kernel/IOSubMemoryDescriptor.cpp optional iokitcppiokit/Kernel/IOGuardPageMemoryDescriptor.cpp optional iokitcppiokit/Kernel/IOPlatformExpert.cpp optional iokitcpp+iokit/Kernel/GenericARMPE.cpp optional iokitcpp+iokit/Kernel/MiniIOSurface.cpp optional iokitcpp+iokit/Kernel/MiniIOMobileFramebuffer.cpp optional iokitcpp++iokit/Kernel/RedwoodGPU.cpp optional iokitcpp+iokit/Kernel/MiniAppleKeyStore.cpp optional iokitcppiokit/Kernel/IOPlatformIO.cpp optional iokitcpp