Skip to content

Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.

C

GPU: SpringBoard drawn by the Adreno through Mesa's freedreno

vcvkk825806aff752
diff --git a/README.md b/README.md
index 94981ca..687d02c 100644
--- a/README.md
+++ b/README.md
@@ -22,7 +22,7 @@
|---|---|
| Экран блокировки, разблокировка свайпом, домашний экран с полным набором значков (две страницы) | Звук, сеть (Wi-Fi/сотовая), Bluetooth, камера |
| Системные жесты: свайп снизу — «домой» и переключатель приложений, свайп от левого края — «назад», Пункт управления | Обои (чёрный фон) |
-| Запуск приложений: Настройки (вглубь), Часы, Календарь, Фото, Заметки и др. | GPU для iOS: пока только программный рендер (сам Adreno уже заведён и заливает кадр, см. ниже) |
+| Запуск приложений: Настройки (вглубь), Часы, Календарь, Фото, Заметки и др. | Metal (приложения, рисующие через него сами); отрисовка самим Adreno есть, но пока опытная — см. «GPU» |
| Сенсорный экран телефона (Goodix GT9916), мультитач, пробуждение касанием | Без `cpus=N` работает одно ядро процессора |
| Несколько ядер процессора (`cpus=4`: четыре больших ядра телефона) | |
| Кнопки «Домой»/«Блокировка»/громкость (в QEMU — клавишами) | Зарядка под нашим ядром (телефон только разряжается) |
@@ -152,22 +152,37 @@ XL_SECS=900 XL_CMDLINE="touchscreen=1 cpufreq=1 bootcpu=7 cpus=4" ./xl run #
что ядра 1–3 малые, и сажает на них интерфейс. Паника «Ticket spinlock … timeout» значила бы, что ядрам тесно
во времени (в QEMU загрузчик сам увеличивает эти сроки в 20 раз): помогает `ml-timeout-global-scale=4`.
-## GPU (в работе)
-
-Adreno 642L телефона заведён под нашим ядром: загружается прошивка управляющего процессора (GMU), включается
-3D-ядро, TrustZone принимает zap-шейдер, ядро GPU исполняет команды и заливает кадр, который драйвер экрана выводит
-на панель (`docs/gpu-first-frame.png`). Это опыт за параметром `adreno=N` (шаги описаны в начале
-`bsd/dev/redwood_adreno.c`), iOS пока по-прежнему рисует процессором. Прошивки GPU берутся с самого телефона
-(`adb pull /vendor/firmware/a660_{gmu.bin,sqe.fw,zap.mdt,zap.b00,zap.b01,zap.b02}` в каталог `fw-gpu/`, в git они
-не попадают) и едут в малом образе.
-
-Сторона iOS тоже проверена: в QuartzCore 16.5 есть рабочий рендер через OpenGL ES, на который сервер окон
-переходит сам, если нет Metal. `GL=1 REST=1 userland/build.sh` (перед этим `tools/mk-mesa.sh` — Mesa 25.0.7 под
-userland iOS, минута сборки) кладёт в малый образ Mesa и прослойку `userland/src/eagl.c`, которая подменяет шесть
-методов `EAGLContext` и таблицу функций OpenGL ES: SpringBoard рисуется через этот путь и в QEMU, и на телефоне
-(`docs/gles-on-phone.png`). Пока за ним стоит программный драйвер Mesa (softpipe), то есть это медленнее обычного
-режима и нужно только как проверка; на экране блокировки при этом не рисуются часы и обои. Дальше — интерфейс
-ядра под драйвер Adreno из Mesa (freedreno) и замена программного драйвера на него.
+## GPU
+
+**iOS на телефоне может рисовать графический процессор** — Adreno 642L, через открытый драйвер Mesa freedreno
+и наш драйвер в ядре (`docs/adreno-springboard.png`: треугольник, экран блокировки, домашний экран).
+
+Как это устроено:
+
+- ядро (`bsd/dev/redwood_adreno.c`, `iokit/Kernel/RedwoodGPU.cpp`) поднимает Adreno — прошивка управляющего
+ процессора (GMU), питание 3D-ядра, zap-шейдер через TrustZone, микрокод — и отдаёт его userland'у через
+ клиент IOKit с подмножеством интерфейса msm DRM (буферы, отправка команд, ожидание);
+- `tools/mk-mesa.sh` собирает Mesa 25.0.7 под userland iOS (freedreno и программный softpipe, минута сборки);
+- `userland/src/eagl.c` встаёт за системный `EAGLContext`: в QuartzCore 16.5 есть рендер через OpenGL ES, на
+ который сервер окон переходит сам, если нет Metal.
+
+Запуск (прошивки GPU один раз снять со своего телефона:
+`adb pull /vendor/firmware/a660_{gmu.bin,sqe.fw,zap.mdt,zap.b00,zap.b01,zap.b02}` в каталог `fw-gpu/`; в git они
+не попадают):
+
+```sh
+tools/mk-mesa.sh
+GL=1 REST=1 userland/build.sh
+XL_CMDLINE="touchscreen=1 cpufreq=1 bootcpu=7 cpus=4 adreno=21 adreno_skip=0x1c00 adreno_pause=0 adreno_keep=1" ./xl run
+```
+
+При втором и следующих запусках за одну загрузку телефона добавлять `adreno_zap=0`. Без параметров `adreno…`
+тот же образ рисует программным драйвером Mesa (в QEMU — всегда так).
+
+Состояние — первый рабочий вариант, не режим по умолчанию: проверены экран блокировки и домашний экран; кадр
+рисуется за 16–17 мс, но 14 из них — копирование готовой картинки (память кадра пока не общая с GPU); прерываний
+нет (ожидание опросом); сеанс, оборванный паникой или `./xl stop`, оставляет GPU включённым — после него
+`./xl reboot`. Приложения, использующие Metal сами, от этого ничего не получают.
## Сборка из исходников
diff --git a/prebuilt/SHA256SUMS b/prebuilt/SHA256SUMS
index 382bac3..38e69ae 100644
--- a/prebuilt/SHA256SUMS
+++ b/prebuilt/SHA256SUMS
@@ -1,6 +1,6 @@
-Built 2026-10-12 23:02 from this tree.
+Built 2026-10-12 23:49 from this tree.
-30e96b37407464e40391ef99a0b933eafebb5105129ba1cf4aa254636eca4772 prebuilt/mach.development.bcm2837
+b79c92cb557d7b995cee74fb3aa6a895d07b191c3c69d8e40ea19ae148e36a84 prebuilt/mach.development.bcm2837
1332a598d2dec00a1d8e930044e4f00a12ad44c6e5e50eecf4b7fe7fae803bf2 prebuilt/loader.efi
d2a35a4a1d6658ffb3696914f727661b52c9333de8a3af8a31642b8023ce03ac prebuilt/userland/child
1a413befd511384052e1962c1394adce20f718e1f2657ce63ab09828ebc4bd1b prebuilt/userland/cpu_test
diff --git a/prebuilt/mach.development.bcm2837 b/prebuilt/mach.development.bcm2837
index b5dda07..e268a2c 100755
Binary files a/prebuilt/mach.development.bcm2837 and b/prebuilt/mach.development.bcm2837 differ
diff --git a/xnu/files/bsd/dev/redwood_adreno.c b/xnu/files/bsd/dev/redwood_adreno.c
index 7bd0fe7..0593e8d 100644
--- a/xnu/files/bsd/dev/redwood_adreno.c
+++ b/xnu/files/bsd/dev/redwood_adreno.c
@@ -61,6 +61,9 @@
* the end of our log to storage every two seconds, and after a silent reset
* that note is all there is.
* adreno_fine=1 the newest step waits like that before every single access
+ * adreno_keep=1 with adreno=21: the GPU is not stopped again but kept for the system to draw
+ * with, through the service RedwoodGPU (see "the GPU kept on" below); it is
+ * given back five seconds before a run's time is up
* adreno_zap=0 step 19 has been done since the phone started: not again
* adreno_blind=1 step 20 starts the command processor with its microcode nowhere, and
* adreno_stream1=N step 18 leaves the core's second stream out (0) or gives it an empty
@@ -81,6 +84,7 @@
* vendor kernel agrees on every offset used here, and is the only source for the frequency counter.
*/
#include <kern/clock.h>
+#include <kern/thread_call.h>
#include <libkern/libkern.h>
#include <mach/mach_types.h>
#include <mach/vm_param.h>
@@ -97,6 +101,7 @@ extern uint64_t kvtophys(vm_offset_t va); /* pmap_paddr_t */
static uint32_t last_step, skip_steps, pause_ms = 2200, fine, watch_seconds = 20;
static uint32_t step; /* the one being performed */
+static uint32_t speed_bin; /* which of the GPUs of this kind the fuses say this is (step 2) */
static uint32_t newest_step; /* the highest one asked for: the one that has not been seen to work */
static char last_write[96] = "nothing written";
static bool quiet; /* repeating accesses that have been announced and seen to work: say nothing */
@@ -428,6 +433,7 @@ step_fuses(void)
bin = get8(&qfprom, "speed bin, low", QFPROM_CORRECTED + QFPROM_GPU_SPEED_BIN);
bin |= get8(&qfprom, "speed bin, high", QFPROM_CORRECTED + QFPROM_GPU_SPEED_BIN + 1) << 8;
bin = bin >> 5 & 0xff;
+ speed_bin = bin;
gaming = get8(&qfprom, "gaming bin", QFPROM_CORRECTED + QFPROM_GPU_GAMING_BIN) >> 5 & 1;
/* dt: qcom,gpu-models: 0xac and 0xbe are sold as Adreno 643, anything else is the 642L */
printf("adreno: %u: speed bin %u: an Adreno %s; gaming bin %u\n", step, bin, bin == 0xac || bin == 0xbe ? "643" : "642L",
@@ -530,6 +536,8 @@ cx_is_on(void)
return (rd(gpucc.base + GPUCC_CX_GDSCR) | rd(gpucc.base + GPUCC_CX_HW_CTRL)) & GDSC_POWER_ON;
}
+static void stale_stop(void);
+
/*
* The switch hangs on the rail "cx.lvl", which is up for as long as the SoC runs at all; the
* vendor's driver (gdsc-regulator.c) clears the collapse bit and waits for bit 31 of the register
@@ -547,8 +555,19 @@ step_cx_power(void)
gdscr = get(&gpucc, "cx_gdscr", GPUCC_CX_GDSCR);
get(&gpucc, "cx_hw_ctrl", GPUCC_CX_HW_CTRL);
if (cx_is_on()) {
- printf("adreno: %u: CX is on already\n", step);
- return true;
+ /*
+ * A kernel that kept the GPU for the system to use (adreno_keep) leaves it running when
+ * its run ends. What it left -- the GMU's processor, the 3D core's power, the GMU's
+ * votes in RPMh, the SMMU's contexts pointing into memory that is somebody else's now --
+ * is taken down as gpu_block_stop() would have, and the start is from nothing.
+ */
+ printf("adreno: %u: CX is on already: an earlier kernel left the GPU block running; stopping it first\n", step);
+ stale_stop();
+ if (cx_is_on()) {
+ printf("adreno: %u: CX stays on: going on with it as it is\n", step);
+ return true;
+ }
+ gdscr = rd(gpucc.base + GPUCC_CX_GDSCR);
}
if (gdscr & GDSC_HW_CONTROL) {
printf("adreno: %u: the switch is under hardware control: not ours to operate\n", step);
@@ -1185,7 +1204,7 @@ to_memory(vm_offset_t start, vm_size_t size)
settle();
}
-#define SPACE_TABLES 48
+#define SPACE_TABLES 1024 /* 2 GB of addresses, were they all in use */
struct gpu_space {
const char *name;
@@ -1196,7 +1215,9 @@ struct gpu_space {
struct {
uint64_t phys;
uint64_t *table;
+ bool changed; /* since it was last written out of this processor's cache (space_commit()) */
} tables[SPACE_TABLES];
+ uint64_t *last_entry, *last_table; /* the level 2 entry looked up last, and its table */
};
static uint64_t *
@@ -1223,6 +1244,30 @@ space_new_table(struct gpu_space *space)
}
/* The table an entry of a higher table leads to; made if the entry is empty. */
+/* An entry of one of the tables has been written: that table is to go out to memory. */
+static void
+space_changed(struct gpu_space *space, const uint64_t *entry)
+{
+ for (uint32_t i = space->table_count; i-- > 0;) {
+ if ((uintptr_t)entry - (uintptr_t)space->tables[i].table < 4096) {
+ space->tables[i].changed = true;
+ return;
+ }
+ }
+}
+
+/* The SMMU reads the tables from memory, not from this processor's cache. */
+static void
+space_commit(struct gpu_space *space)
+{
+ for (uint32_t i = 0; i < space->table_count; i++) {
+ if (space->tables[i].changed) {
+ to_memory((vm_offset_t)space->tables[i].table, 4096);
+ space->tables[i].changed = false;
+ }
+ }
+}
+
static uint64_t *
space_next(struct gpu_space *space, uint64_t *entry)
{
@@ -1239,6 +1284,7 @@ space_next(struct gpu_space *space, uint64_t *entry)
table = space_new_table(space);
if (table != NULL) {
*entry = kvtophys((vm_offset_t)table) | PTE_TABLE;
+ space_changed(space, entry);
}
return table;
}
@@ -1253,11 +1299,22 @@ space_map(struct gpu_space *space, uint64_t address, uint64_t phys, uint64_t att
return false;
}
level2 = space_next(space, &space->root[address >> 30 & 0x1ff]);
- level3 = level2 != NULL ? space_next(space, &level2[address >> 21 & 0x1ff]) : NULL;
+ if (level2 == NULL) {
+ return false;
+ }
+ if (space->last_entry == &level2[address >> 21 & 0x1ff]) {
+ level3 = space->last_table; /* page after page of one buffer: the same table */
+ } else {
+ level3 = space_next(space, &level2[address >> 21 & 0x1ff]);
+ space->last_entry = &level2[address >> 21 & 0x1ff];
+ space->last_table = level3;
+ }
if (level3 == NULL) {
+ space->last_entry = NULL;
return false;
}
- level3[address >> 12 & 0x1ff] = phys | attributes | PTE_PAGE;
+ level3[address >> 12 & 0x1ff] = phys == 0 && attributes == 0 ? 0 : phys | attributes | PTE_PAGE;
+ space_changed(space, level3);
return true;
}
@@ -2416,7 +2473,9 @@ static struct gpu_space second_space = {
static bool second_space_is_on, second_stream_is_on;
static struct gpu_block core_ring, core_sqe, core_scratch, core_target;
static bool core_space_is_on, core_runs, zap_is_loaded;
-static uint32_t ring_words; /* written to the ring so far; it is never filled here */
+#define RING_WORDS (CORE_RING_SIZE / 4)
+static uint32_t ring_at; /* where the next word goes: the ring goes round */
+static uint32_t ring_sent; /* up to where the core has been told to read */
static uint32_t
odd_parity(uint32_t v)
@@ -2427,7 +2486,8 @@ odd_parity(uint32_t v)
static void
ring(uint32_t word)
{
- ((uint32_t *)core_ring.memory)[ring_words++] = word;
+ ((uint32_t *)core_ring.memory)[ring_at] = word;
+ ring_at = (ring_at + 1) % RING_WORDS;
}
/* The header of a command with so many words after it. */
@@ -2445,28 +2505,41 @@ ring_register(uint32_t number, uint32_t value)
ring(value);
}
+/* What has been written to the ring since the last time: out to memory, and the core told how far to read. */
+static void
+ring_send(void)
+{
+ if (ring_at >= ring_sent) {
+ to_memory(core_ring.memory + 4 * ring_sent, 4 * (ring_at - ring_sent));
+ } else {
+ to_memory(core_ring.memory + 4 * ring_sent, 4 * (RING_WORDS - ring_sent));
+ to_memory(core_ring.memory, 4 * ring_at);
+ }
+ ring_sent = ring_at;
+ wr(gpu.base + CP_RB_WPTR, ring_at);
+}
+
/* Let the 3D core have what is in the ring, and wait until it has taken all of it and rests. */
static bool
ring_run(const char *what)
{
uint32_t taken = ~0u, status = ~0u;
- to_memory(core_ring.memory, ring_words * 4);
- printf("adreno: %u: about to have the 3D core %s: the ring up to word %u\n", step, what, ring_words);
+ printf("adreno: %u: about to have the 3D core %s: the ring up to word %u\n", step, what, ring_at);
snprintf(last_write, sizeof(last_write), "step %u had the 3D core %s", step, what);
hold(true);
- wr(gpu.base + CP_RB_WPTR, ring_words);
- for (int i = 0; i < 20000 && (taken != ring_words || (status & ~1u)); i++) {
+ ring_send();
+ for (int i = 0; i < 20000 && (taken != ring_at || (status & ~1u)); i++) {
sleep_us(100);
taken = rd(gpu.base + CP_RB_RPTR);
status = rd(gpu.base + RBBM_STATUS);
}
- if (taken == ring_words && !(status & ~1u) && rd(gpu.base + CP_HW_FAULT) == 0 && !space_faulted(&core_space)) {
+ if (taken == ring_at && !(status & ~1u) && rd(gpu.base + CP_HW_FAULT) == 0 && !space_faulted(&core_space)) {
printf("adreno: %u: the 3D core has done it (ring read up to %u, status %#x)\n", step, taken, status);
return true;
}
printf("adreno: %u: the 3D core does not get through: ring read up to %u of %u, status %#x, interrupts %#x, "
- "command processor: fault %#x, interrupts %#x, protection %#x\n", step, taken, ring_words, status,
+ "command processor: fault %#x, interrupts %#x, protection %#x\n", step, taken, ring_at, status,
rd(gpu.base + RBBM_INT_0_STATUS), rd(gpu.base + CP_HW_FAULT), rd(gpu.base + CP_INTERRUPT_STATUS),
rd(gpu.base + CP_PROTECT_STATUS));
space_faulted(&core_space);
@@ -2641,6 +2714,16 @@ step_core_start(void)
put(&gpu, "cp_mem_pool_size", CP_MEM_POOL_SIZE, 128);
put(&gpu, "pc_dbg_eco_cntl", PC_DBG_ECO_CNTL, 0x00200200);
put(&gpu, "cp_ahb_cntl", CP_AHB_CNTL, 1);
+ /*
+ * How textures in the GPU's own tiled and compressed layouts are laid out in memory (Linux's
+ * a6xx_calc_ubwc_config, a6xx_set_ubwc_config for this GPU: highest bank bit 14, and the
+ * rest as below); Mesa is told the same (redwood_gpu_parameter()).
+ */
+ put(&gpu, "rb_nc_mode_cntl", GPU_WORD(0x8e08), 1u << 11 | 1u << 4 | 1u << 1);
+ put(&gpu, "tpl1_nc_mode_cntl", GPU_WORD(0xb604), 1u << 1);
+ put(&gpu, "sp_nc_mode_cntl", GPU_WORD(0xae02), 2u << 4 | 1u << 1);
+ put(&gpu, "uche_mode_cntl", GPU_WORD(0x0e01), 1u << 21);
+ put(&gpu, "rbbm_nc_mode_cntl", GPU_WORD(0x0534), 1);
put(&gpu, "rbbm_perfctr_cntl", RBBM_PERFCTR_CNTL, 1);
put(&gpu, "rbbm_interface_hang_int_cntl", RBBM_INTERFACE_HANG_INT_CNTL, 1u << 30 | 0x1fffff);
put(&gpu, "uche_client_pf", UCHE_CLIENT_PF, 1u << 7 | 1);
@@ -2854,6 +2937,43 @@ step_frame(void)
* core given back, the GMU told to drop its votes ("slumber") and stopped, its RPMh channel put
* to sleep; then the processor into reset and its stream sent nowhere.
*/
+/* What step 4 does when it finds the GPU block running: see there. Nothing here has been set up by this kernel. */
+static void
+stale_stop(void)
+{
+ uint32_t gx;
+
+ need(&gmu);
+ need(&rscc);
+ put(&gmu, "cm3_sysreset", GMU_CM3_SYSRESET, 1);
+ gx = get(&gpucc, "gx_gdscr", GPUCC_GX_GDSCR);
+ if (gx & GDSC_POWER_ON) {
+ /* the 3D core's switch is the GMU's to operate, and the GMU is gone: vendor's a6xx_gmu_suspend */
+ put(&gpucc, "gx_gdscr", GPUCC_GX_GDSCR, gx | GDSC_SW_COLLAPSE);
+ wait_for(&gpucc, GPUCC_GX_GDSCR, GDSC_POWER_ON, 0, 100);
+ get(&gpucc, "gx_gdscr", GPUCC_GX_GDSCR);
+ }
+ if (rd(rscc.base + RSCC_SEQ_MEM_0_DRV0) == 0xeaaae5a0 && !(rd(rscc.base + RSCC_RSC_STATUS0_DRV0) & RSCC_ASLEEP)) {
+ put(&gmu, "rscc_control_req", GMU_RSCC_CONTROL_REQ, 1);
+ wait_for(&rscc, RSCC_RSC_STATUS0_DRV0, RSCC_ASLEEP, RSCC_ASLEEP, 100);
+ get(&rscc, "rsc_status0_drv0", RSCC_RSC_STATUS0_DRV0);
+ put(&gmu, "rscc_control_req", GMU_RSCC_CONTROL_REQ, 0);
+ }
+ if (smmu_layout()) {
+ for (uint32_t match = 0; match < 3; match++) {
+ put(&smmu, "smr", SMMU_SMR(match), 0);
+ put(&smmu, "s2cr", SMMU_S2CR(match), SMMU_S2CR_FAULT);
+ put(&smmu, "cb sctlr", (smmu_pages + match) * smmu_page + SMMU_CB_SCTLR, 0);
+ }
+ }
+ put(&gpucc, "cx_gdscr", GPUCC_CX_GDSCR, rd(gpucc.base + GPUCC_CX_GDSCR) | GDSC_SW_COLLAPSE);
+ for (int i = 0; i < 1000 && cx_is_on(); i++) {
+ sleep_us(100);
+ }
+ get(&gpucc, "cx_gdscr", GPUCC_CX_GDSCR);
+ get(&gpucc, "cx_hw_ctrl", GPUCC_CX_HW_CTRL);
+}
+
static void
gpu_block_stop(void)
{
@@ -2955,6 +3075,272 @@ redwood_adreno_quiesce(void)
}
}
+/* ---- the GPU kept on, for the system to draw with --------------------------------------------------- */
+
+/*
+ * With boot-arg adreno_keep=1 the steps are not undone: the GPU block stays as step 21 leaves it,
+ * the 3D core powered and taking commands, and the service RedwoodGPU (iokit/Kernel/RedwoodGPU.cpp)
+ * hands it to programs through the interface of Linux's msm driver, which is what Mesa's
+ * freedreno speaks. That file keeps the buffers and the clients; what is the GPU's is here:
+ * addresses in the 3D core's SMMU context, the ring, and the word by which the core says how
+ * far it has got (a "fence": a submission's number, written by the core when it is through
+ * with it). The caller holds one lock around all of these.
+ */
+#define GPU_CHIP_ID 0x06030500ull /* dt: qcom,chipid */
+#define GPU_ADDRESSES_START 0x20000000ull /* above everything the steps use; below 4 GB, as Linux keeps them */
+#define GPU_ADDRESSES_SIZE 0xe0000000ull
+#define GPU_FENCE_AT 0x40 /* in the scratch page */
+#define CP_SCRATCH_REG2 0x0885
+#define CP_IB1_BASE GPU_WORD(0x0928)
+#define CP_IB1_REM_SIZE GPU_WORD(0x092a)
+#define CP_IB2_BASE GPU_WORD(0x092b)
+#define CP_IB2_REM_SIZE GPU_WORD(0x092d)
+#define CP_INDIRECT_BUFFER 0x3f
+#define EVENT_INVALIDATE_DEPTH 0x18
+#define EVENT_INVALIDATE_COLOR 0x19
+
+static bool gpu_kept;
+static uint32_t gpu_fence; /* the number of the last submission */
+static struct {
+ uint64_t start, size;
+} gpu_free[1024]; /* the addresses not in use, in order */
+static unsigned gpu_free_count;
+
+extern void RedwoodGPUStart(void);
+extern void RedwoodGPUStop(void); /* takes the clients' lock, then calls redwood_gpu_stop() */
+static void gpu_block_stop(void);
+
+/*
+ * A run that the loader ends after so many seconds (boot-arg loader_return_after) gives the GPU
+ * back a few seconds before that, in order and with time to do it: the next kernel then finds
+ * the GPU block as the phone's own start leaves it. A run that ends otherwise (a panic, xl stop)
+ * leaves it running, and the next kernel's step 4 has to deal with that.
+ */
+#define GPU_STOP_AHEAD 5
+
+void
+redwood_gpu_stop(void)
+{
+ if (gpu_kept) {
+ gpu_kept = false;
+ quiet = false;
+ printf("adreno: the run is about to end: the GPU is given back\n");
+ gpu_block_stop();
+ }
+}
+
+static void
+gpu_stop_timer(__unused thread_call_param_t p0, __unused thread_call_param_t p1)
+{
+ RedwoodGPUStop();
+}
+
+bool
+redwood_gpu_ready(void)
+{
+ return gpu_kept;
+}
+
+/* The answers to msm's GET_PARAM (Linux's adreno_get_param for this GPU; the numbers are msm_drm.h's). */
+bool
+redwood_gpu_parameter(uint32_t which, uint64_t *value)
+{
+ switch (which) {
+ case 0x01: /* GPU_ID: none, the chip ID says it */
+ case 0x06: /* GMEM_BASE */
+ case 0x08: /* PP_PGTABLE */
+ case 0x09: /* FAULTS */
+ case 0x0a: /* SUSPENDS */
+ case 0x11: /* RAYTRACING */
+ *value = 0;
+ return true;
+ case 0x02: /* GMEM_SIZE */
+ *value = 512 * 1024;
+ return true;
+ case 0x03: /* CHIP_ID, with the speed bin above it */
+ *value = GPU_CHIP_ID | (uint64_t)speed_bin << 32;
+ return true;
+ case 0x04: /* MAX_FREQ */
+ *value = 315000000;
+ return true;
+ case 0x05: /* TIMESTAMP */
+ *value = mach_absolute_time();
+ return true;
+ case 0x07: /* PRIORITIES */
+ *value = 1;
+ return true;
+ case 0x10: /* HIGHEST_BANK_BIT */
+ *value = 14;
+ return true;
+ case 0x12: /* UBWC_SWIZZLE */
+ *value = 6;
+ return true;
+ case 0x13: /* MACROTILE_MODE */
+ *value = 1;
+ return true;
+ default: /* among them VA_START, VA_SIZE: addresses are ours to give */
+ return false;
+ }
+}
+
+/* Addresses for a buffer of so many bytes; 0: none left. */
+uint64_t
+redwood_gpu_address(uint64_t size)
+{
+ size = (size + 0xffff) & ~0xffffull;
+ for (unsigned i = 0; i < gpu_free_count; i++) {
+ if (gpu_free[i].size >= size) {
+ uint64_t address = gpu_free[i].start;
+
+ gpu_free[i].start += size;
+ gpu_free[i].size -= size;
+ if (gpu_free[i].size == 0) {
+ memmove(&gpu_free[i], &gpu_free[i + 1], (--gpu_free_count - i) * sizeof(gpu_free[0]));
+ }
+ return address;
+ }
+ }
+ return 0;
+}
+
+void
+redwood_gpu_address_free(uint64_t address, uint64_t size)
+{
+ unsigned at;
+
+ size = (size + 0xffff) & ~0xffffull;
+ for (at = 0; at < gpu_free_count && gpu_free[at].start < address; at++) {
+ }
+ if (at > 0 && gpu_free[at - 1].start + gpu_free[at - 1].size == address) {
+ gpu_free[at - 1].size += size;
+ if (at < gpu_free_count && address + size == gpu_free[at].start) {
+ gpu_free[at - 1].size += gpu_free[at].size;
+ memmove(&gpu_free[at], &gpu_free[at + 1], (--gpu_free_count - at) * sizeof(gpu_free[0]));
+ }
+ } else if (at < gpu_free_count && address + size == gpu_free[at].start) {
+ gpu_free[at].start = address;
+ gpu_free[at].size += size;
+ } else if (gpu_free_count < sizeof(gpu_free) / sizeof(gpu_free[0])) {
+ memmove(&gpu_free[at + 1], &gpu_free[at], (gpu_free_count++ - at) * sizeof(gpu_free[0]));
+ gpu_free[at].start = address;
+ gpu_free[at].size = size;
+ } /* (or else these addresses are lost) */
+}
+
+/* One 4K page of ours at an address of the 3D core's; after the last of a buffer's, redwood_gpu_mapped(). */
+bool
+redwood_gpu_map(uint64_t address, uint64_t phys, bool read_only)
+{
+ return space_map(&core_space, address, phys, PTE_CORE | (read_only ? PTE_READ_ONLY : 0));
+}
+
+void
+redwood_gpu_mapped(void)
+{
+ space_commit(&core_space);
+}
+
+void
+redwood_gpu_unmap(uint64_t address, uint64_t size)
+{
+ uint32_t context = smmu_context(&core_space);
+
+ for (uint64_t offset = 0; offset < size; offset += 4096) {
+ space_map(&core_space, address + offset, 0, 0);
+ }
+ space_commit(&core_space);
+ wr(smmu.base + context + SMMU_CB_TLBIASID, core_space.context);
+ wr(smmu.base + context + SMMU_CB_TLBSYNC, 1);
+ for (int i = 0; i < 1000 && (rd(smmu.base + context + SMMU_CB_TLBSTATUS) & 1); i++) {
+ }
+}
+
+/* Memory that the 3D core and a program see without this processor's cache in between: out of that cache. */
+void
+redwood_gpu_to_memory(void *start, uint64_t size)
+{
+ to_memory((vm_offset_t)start, size);
+}
+
+/* How far the 3D core has got. */
+bool
+redwood_gpu_done(uint32_t fence)
+{
+ volatile uint32_t *word = (volatile uint32_t *)(core_scratch.memory + GPU_FENCE_AT);
+
+ to_memory((vm_offset_t)word, 4);
+ return (int32_t)(*word - fence) >= 0;
+}
+
+uint32_t
+redwood_gpu_last(void)
+{
+ return gpu_fence;
+}
+
+/*
+ * Command buffers for the 3D core to run, in order: each an address and a count of words. As
+ * Linux's a6xx_submit puts them in the ring. Returns the submission's number; 0: the ring is
+ * full and the core is not reading it.
+ */
+uint32_t
+redwood_gpu_submit(const uint64_t *addresses, const uint32_t *words, uint32_t count)
+{
+ uint32_t wanted = 16 + 4 * count, room = 0;
+
+ for (int i = 0; i < 20000; i++) {
+ room = (rd(gpu.base + CP_RB_RPTR) + RING_WORDS - ring_at - 1) % RING_WORDS;
+ if (room >= wanted) {
+ break;
+ }
+ sleep_us(100);
+ }
+ if (room < wanted) {
+ return 0;
+ }
+ ring_command(CP_EVENT_WRITE, 1);
+ ring(EVENT_INVALIDATE_DEPTH);
+ ring_command(CP_EVENT_WRITE, 1);
+ ring(EVENT_INVALIDATE_COLOR);
+ for (uint32_t i = 0; i < count; i++) {
+ ring_command(CP_INDIRECT_BUFFER, 3);
+ ring((uint32_t)addresses[i]);
+ ring((uint32_t)(addresses[i] >> 32));
+ ring(words[i]);
+ }
+ if (++gpu_fence == 0) {
+ gpu_fence = 1;
+ }
+ ring_register(CP_SCRATCH_REG2, gpu_fence);
+ ring_command(CP_EVENT_WRITE, 4);
+ ring(EVENT_CACHE_FLUSH);
+ ring(CORE_SCRATCH_ADDRESS + GPU_FENCE_AT);
+ ring(0);
+ ring(gpu_fence);
+ ring_send();
+ return gpu_fence;
+}
+
+/* A submission does not come back: what the 3D core and its SMMU context say. */
+void
+redwood_gpu_report(const char *why)
+{
+ uint32_t context = smmu_context(&core_space);
+
+ printf("adreno: gpu: %s: submission %u is the last, the core has done %u; ring: read up to %u, written up to %u; status "
+ "%#x, interrupts %#x; command processor: fault %#x, interrupts %#x, protection %#x\n", why, gpu_fence,
+ *(volatile uint32_t *)(core_scratch.memory + GPU_FENCE_AT), rd(gpu.base + CP_RB_RPTR), ring_at,
+ rd(gpu.base + RBBM_STATUS), rd(gpu.base + RBBM_INT_0_STATUS), rd(gpu.base + CP_HW_FAULT),
+ rd(gpu.base + CP_INTERRUPT_STATUS), rd(gpu.base + CP_PROTECT_STATUS));
+ printf("adreno: gpu: in the first command buffer at %#llx, %u words to go; in the second at %#llx, %u to go\n",
+ (uint64_t)rd(gpu.base + CP_IB1_BASE + 4) << 32 | rd(gpu.base + CP_IB1_BASE), rd(gpu.base + CP_IB1_REM_SIZE),
+ (uint64_t)rd(gpu.base + CP_IB2_BASE + 4) << 32 | rd(gpu.base + CP_IB2_BASE), rd(gpu.base + CP_IB2_REM_SIZE));
+ step = 0;
+ if (space_faulted(&core_space)) {
+ wr(smmu.base + context + SMMU_CB_FSR, SMMU_FSR_ALL); /* so that the next one shows */
+ }
+}
+
/* ---- all of it together ---------------------------------------------------------------------------- */
static bool(*const steps[])(void) = {
@@ -3006,6 +3392,27 @@ redwood_adreno_init(void)
printf("adreno: step %u: done\n", step);
done = step;
}
+ if (PE_parse_boot_argn("adreno_keep", &only, sizeof(only)) && only) {
+ if (done >= 21 && core_runs) {
+ gpu_free[0].start = GPU_ADDRESSES_START;
+ gpu_free[0].size = GPU_ADDRESSES_SIZE;
+ gpu_free_count = 1;
+ *(volatile uint32_t *)(core_scratch.memory + GPU_FENCE_AT) = 0;
+ to_memory(core_scratch.memory, 4096);
+ gpu_kept = true;
+ quiet = true;
+ printf("adreno: THE GPU STAYS ON for the system to draw with (adreno_keep): the service RedwoodGPU\n");
+ RedwoodGPUStart();
+ if (PE_parse_boot_argn("loader_return_after", &only, sizeof(only)) && only > GPU_STOP_AHEAD + 5) {
+ uint64_t deadline;
+
+ clock_interval_to_deadline(only - GPU_STOP_AHEAD, NSEC_PER_SEC, &deadline);
+ thread_call_enter_delayed(thread_call_allocate(gpu_stop_timer, NULL), deadline);
+ }
+ return;
+ }
+ printf("adreno: not kept: the steps did not get as far as a 3D core that takes commands\n");
+ }
/*
* Punishment for a write may come late (a reset some 17 seconds after it has been seen):
* keep the end of the log saying where things stand.
diff --git a/xnu/files/bsd/sys/redwood_input.h b/xnu/files/bsd/sys/redwood_input.h
index 5855949..a12e979 100644
--- a/xnu/files/bsd/sys/redwood_input.h
+++ b/xnu/files/bsd/sys/redwood_input.h
@@ -44,6 +44,24 @@ void redwood_goodix_init(void); /* boot-arg touchscreen=1: the phone's t
void redwood_cpufreq_init(void); /* boot-arg cpufreq=1: the phone's processor at full speed */
void redwood_adreno_init(void); /* boot-arg adreno=N: the first N steps of waking the phone's GPU */
void redwood_adreno_quiesce(void); /* on the way back to the loader: the GPU block stopped, out of our memory */
+
+/*
+ * The GPU kept on (boot-arg adreno_keep=1), for iokit/Kernel/RedwoodGPU.cpp: see "the GPU kept on"
+ * in bsd/dev/redwood_adreno.c. One caller at a time.
+ */
+bool redwood_gpu_ready(void);
+bool redwood_gpu_parameter(uint32_t which, uint64_t *value);
+uint64_t redwood_gpu_address(uint64_t size);
+void redwood_gpu_address_free(uint64_t address, uint64_t size);
+bool redwood_gpu_map(uint64_t address, uint64_t phys, bool read_only);
+void redwood_gpu_mapped(void);
+void redwood_gpu_unmap(uint64_t address, uint64_t size);
+void redwood_gpu_to_memory(void *start, uint64_t size);
+uint32_t redwood_gpu_submit(const uint64_t *addresses, const uint32_t *words, uint32_t count);
+bool redwood_gpu_done(uint32_t fence);
+uint32_t redwood_gpu_last(void);
+void redwood_gpu_report(const char *why);
+void redwood_gpu_stop(void);
#endif
#endif /* _SYS_REDWOOD_INPUT_H_ */
diff --git a/xnu/files/iokit/Kernel/RedwoodGPU.cpp b/xnu/files/iokit/Kernel/RedwoodGPU.cpp
new file mode 100644
index 0000000..99b6b4b
--- /dev/null
+++ b/xnu/files/iokit/Kernel/RedwoodGPU.cpp
@@ -0,0 +1,658 @@
+/*
+ * RedwoodGPU: the phone's Adreno for programs to draw with.
+ *
+ * bsd/dev/redwood_adreno.c wakes the GPU and, with boot-arg adreno_keep=1, keeps it on and
+ * starts this service. A program that opens it speaks the interface of Linux's msm DRM driver,
+ * the one Mesa's freedreno knows: every request of that interface is a call of the user client
+ * with the request's number as the selector and the request's structure passed in and back
+ * (userland: tools/mesa/redwood-drm, which stands in for libdrm inside Mesa). What Linux does
+ * with mmap() on the device is the client memory of the type that is the buffer's handle.
+ *
+ * What there is of the interface is what freedreno uses to draw: parameters, buffers (made,
+ * told their address in the GPU's address space, mapped, given back), submission of command
+ * buffers, and waiting for one to be done. A buffer is wired kernel memory entered into the 3D
+ * core's SMMU context; it is zero when made. The program has this processor's cache between
+ * itself and the memory, the GPU has not: see submit().
+ *
+ * Not here yet: interrupts (waiting is asking again every millisecond), more than one program's
+ * worth of care (every client's buffers are in the one address space; a client that goes takes
+ * its own away), recovery of a GPU that has stopped answering (it is reported, see
+ * redwood_gpu_report(), and stays stopped), fences and buffers as file descriptors.
+ */
+
+#include <IOKit/IOLib.h>
+#include <IOKit/IOLocks.h>
+#include <IOKit/IOKitKeys.h>
+#include <IOKit/IOPlatformExpert.h>
+#include <IOKit/IOService.h>
+#include <IOKit/IOUserClient.h>
+#include <IOKit/IOBufferMemoryDescriptor.h>
+#include <kern/clock.h>
+#include <mach/mach_time.h>
+#include <mach/vm_param.h>
+#include <sys/systm.h>
+
+extern "C" {
+/* bsd/dev/redwood_adreno.c ("the GPU kept on"; declared in bsd/sys/redwood_input.h, which is not seen from here) */
+bool redwood_gpu_ready(void);
+bool redwood_gpu_parameter(uint32_t which, uint64_t * value);
+uint64_t redwood_gpu_address(uint64_t size);
+void redwood_gpu_address_free(uint64_t address, uint64_t size);
+bool redwood_gpu_map(uint64_t address, uint64_t phys, bool read_only);
+void redwood_gpu_mapped(void);
+void redwood_gpu_unmap(uint64_t address, uint64_t size);
+void redwood_gpu_to_memory(void * start, uint64_t size);
+uint32_t redwood_gpu_submit(const uint64_t * addresses, const uint32_t * words, uint32_t count);
+bool redwood_gpu_done(uint32_t fence);
+uint32_t redwood_gpu_last(void);
+void redwood_gpu_report(const char * why);
+void redwood_gpu_stop(void);
+
+void RedwoodGPUStart(void);
+void RedwoodGPUStop(void);
+void MiniIOMobileFramebufferShow(const uint8_t * pixels, uint32_t width, uint32_t height, uint32_t rowBytes);
+}
+
+/* Linux's include/uapi/drm/msm_drm.h and drm.h: the requests' numbers and structures. */
+enum {
+ kDrmGemClose = 0x09,
+ kMsmGetParam = 0x40,
+ kMsmSetParam = 0x41,
+ kMsmGemNew = 0x42,
+ kMsmGemInfo = 0x43,
+ kMsmGemCpuPrep = 0x44,
+ kMsmGemCpuFini = 0x45,
+ kMsmGemSubmit = 0x46,
+ kMsmWaitFence = 0x47,
+ kMsmGemMadvise = 0x48,
+ kMsmSubmitQueueNew = 0x4a,
+ kMsmSubmitQueueClose = 0x4b,
+ kMsmSubmitQueueQuery = 0x4c,
+ kRedwoodShow = 0x7f, /* ours: pixels of the caller's onto the panel, to see a test's result */
+};
+
+struct msm_timespec {
+ int64_t seconds, nanoseconds; /* of the clock that goes on while the machine sleeps */
+};
+struct msm_param {
+ uint32_t pipe, param;
+ uint64_t value;
+ uint32_t len, pad;
+};
+struct msm_gem_new {
+ uint64_t size;
+ uint32_t flags, handle;
+};
+struct msm_gem_info {
+ uint32_t handle, info;
+ uint64_t value;
+ uint32_t len, pad;
+};
+struct msm_gem_cpu_prep {
+ uint32_t handle, op;
+ struct msm_timespec timeout;
+};
+struct msm_gem_submit_cmd {
+ uint32_t type, submit_idx, submit_offset, size, pad, nr_relocs;
+ uint64_t relocs;
+};
+struct msm_gem_submit_bo {
+ uint32_t flags, handle;
+ uint64_t presumed;
+};
+struct msm_gem_submit {
+ uint32_t flags, fence, nr_bos, nr_cmds;
+ uint64_t bos, cmds;
+ int32_t fence_fd;
+ uint32_t queueid;
+ uint64_t in_syncobjs, out_syncobjs;
+ uint32_t nr_in_syncobjs, nr_out_syncobjs, syncobj_stride, pad;
+};
+struct msm_wait_fence {
+ uint32_t fence, flags;
+ struct msm_timespec timeout;
+ uint32_t queueid;
+};
+struct msm_gem_madvise {
+ uint32_t handle, madv, retained;
+};
+struct msm_submitqueue {
+ uint32_t flags, prio, id;
+};
+struct msm_submitqueue_query {
+ uint64_t data;
+ uint32_t id, param, len, pad;
+};
+struct drm_gem_close {
+ uint32_t handle, pad;
+};
+struct redwood_show {
+ uint64_t pixels;
+ uint32_t width, height, rowBytes, pad;
+};
+
+#define MSM_INFO_GET_OFFSET 0
+#define MSM_INFO_GET_IOVA 1
+#define MSM_INFO_SET_NAME 2
+#define MSM_INFO_GET_FLAGS 5
+#define MSM_BO_GPU_READONLY 0x00000002
+#define MSM_PREP_NOSYNC 0x04
+#define MSM_SUBMIT_CMD_BUF 1
+#define MSM_SUBMIT_FENCE_FD_OUT 0x20000000
+
+#define GPU_BUFFERS 8192 /* a client's, at most */
+#define GPU_BUFFER_MAX (256u << 20)
+#define GPU_SUBMIT_CMDS 128
+#define GPU_SUBMIT_BOS 8192
+#define GPU_PATIENCE_MS 5000 /* for a buffer or a submission that somebody waits for without a limit */
+
+static IOLock * gGPULock; /* around everything that is the GPU's (redwood_gpu_...) */
+static bool gGPUReported;
+
+/* Wait for the 3D core to be done with submission `fence`; with the lock held, which is let go of while waiting. */
+static bool
+gpuWait(uint32_t fence, uint64_t milliseconds)
+{
+ for (uint64_t waited = 0;; waited++) {
+ if (redwood_gpu_done(fence)) {
+ return true;
+ }
+ if (waited >= milliseconds) {
+ break;
+ }
+ IOLockUnlock(gGPULock);
+ IOSleep(1);
+ IOLockLock(gGPULock);
+ }
+ if (milliseconds >= 1000 && !gGPUReported) {
+ gGPUReported = true;
+ redwood_gpu_report("no answer");
+ }
+ return false;
+}
+
+/* How long from now a time of msm's is, in milliseconds; 0: now or past. */
+static uint64_t
+millisecondsUntil(const struct msm_timespec * when)
+{
+ uint64_t now;
+ int64_t left;
+
+ absolutetime_to_nanoseconds(mach_continuous_time(), &now);
+ left = when->seconds * (int64_t) NSEC_PER_SEC + when->nanoseconds - (int64_t) now;
+ if (left <= 0) {
+ return 0;
+ }
+ return (uint64_t) left / NSEC_PER_MSEC > GPU_PATIENCE_MS ? GPU_PATIENCE_MS : (uint64_t) left / NSEC_PER_MSEC;
+}
+
+/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
+
+class RedwoodGPU : public IOService
+{
+ OSDeclareDefaultStructors(RedwoodGPU);
+
+public:
+ virtual bool start(IOService * provider) APPLE_KEXT_OVERRIDE;
+};
+
+OSDefineMetaClassAndStructors(RedwoodGPU, IOService);
+
+bool
+RedwoodGPU::start(IOService * provider)
+{
+ if (!IOService::start(provider)) {
+ return false;
+ }
+ setProperty(kIOUserClientClassKey, "RedwoodGPUUserClient");
+ setName("RedwoodGPU");
+ registerService();
+ return true;
+}
+
+void
+RedwoodGPUStart(void)
+{
+ IOService * provider = IOService::getPlatform();
+ RedwoodGPU * gpu;
+
+ gGPULock = IOLockAlloc();
+ gpu = OSTypeAlloc(RedwoodGPU);
+ if (gpu == NULL || provider == NULL || !gpu->init() || !gpu->attach(provider)) {
+ OSSafeReleaseNULL(gpu);
+ IOLog("RedwoodGPU: cannot create the service\n");
+ return;
+ }
+ if (!gpu->start(provider)) {
+ gpu->detach(provider);
+ IOLog("RedwoodGPU: the service did not start\n");
+ }
+ gpu->release();
+}
+
+/* The run is ending (redwood_adreno.c): nobody is in the middle of anything when the GPU goes. */
+void
+RedwoodGPUStop(void)
+{
+ IOLockLock(gGPULock);
+ redwood_gpu_stop();
+ IOLockUnlock(gGPULock);
+}
+
+/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
+
+struct gpu_buffer {
+ IOBufferMemoryDescriptor * memory; /* NULL: this handle is free */
+ uint64_t address; /* the GPU's for it */
+ uint64_t size;
+ uint32_t flags;
+};
+
+class RedwoodGPUUserClient : public IOUserClient
+{
+ OSDeclareDefaultStructors(RedwoodGPUUserClient);
+
+ task_t fTask;
+ struct gpu_buffer * fBuffers; /* handle n is fBuffers[n - 1] */
+ uint32_t fNextQueue;
+ uint32_t fBufferCount;
+ uint64_t fBufferBytes;
+ uint32_t fSubmits;
+
+ struct gpu_buffer * buffer(uint32_t handle);
+ void freeBuffer(struct gpu_buffer * b);
+ IOReturn gemNew(struct msm_gem_new * request);
+ IOReturn gemInfo(struct msm_gem_info * request);
+ IOReturn submit(struct msm_gem_submit * request);
+ IOReturn show(const struct redwood_show * request);
+
+public:
+ virtual bool initWithTask(task_t owningTask, void * securityID, UInt32 type, OSDictionary * properties) APPLE_KEXT_OVERRIDE;
+ virtual bool start(IOService * provider) APPLE_KEXT_OVERRIDE;
+ virtual IOReturn clientClose(void) APPLE_KEXT_OVERRIDE;
+ virtual void free(void) APPLE_KEXT_OVERRIDE;
+ virtual IOReturn clientMemoryForType(UInt32 type, IOOptionBits * options, IOMemoryDescriptor ** memory) APPLE_KEXT_OVERRIDE;
+ virtual IOReturn externalMethod(uint32_t selector, IOExternalMethodArguments * args,
+ IOExternalMethodDispatch * dispatch, OSObject * target, void * reference) APPLE_KEXT_OVERRIDE;
+};
+
+OSDefineMetaClassAndStructors(RedwoodGPUUserClient, IOUserClient);
+
+bool
+RedwoodGPUUserClient::initWithTask(task_t owningTask, void * securityID, UInt32 type, OSDictionary * properties)
+{
+ if (!IOUserClient::initWithTask(owningTask, securityID, type, properties)) {
+ return false;
+ }
+ fTask = owningTask;
+ return true;
+}
+
+bool
+RedwoodGPUUserClient::start(IOService * provider)
+{
+ if (!redwood_gpu_ready() || !IOUserClient::start(provider)) {
+ return false;
+ }
+ fBuffers = IONewZero(struct gpu_buffer, GPU_BUFFERS);
+ return fBuffers != NULL;
+}
+
+/* With the lock held. */
+void
+RedwoodGPUUserClient::freeBuffer(struct gpu_buffer * b)
+{
+ if (redwood_gpu_ready()) {
+ redwood_gpu_unmap(b->address, b->size);
+ redwood_gpu_address_free(b->address, b->size);
+ }
+ b->memory->release();
+ b->memory = NULL;
+ fBufferCount--;
+ fBufferBytes -= b->size;
+}
+
+IOReturn
+RedwoodGPUUserClient::clientClose(void)
+{
+ if (fBuffers != NULL) {
+ IOLockLock(gGPULock);
+ /* what the client had the GPU do last may still be going on, in these buffers */
+ if (redwood_gpu_ready()) {
+ gpuWait(redwood_gpu_last(), 1000);
+ }
+ for (uint32_t i = 0; i < GPU_BUFFERS; i++) {
+ if (fBuffers[i].memory != NULL) {
+ freeBuffer(&fBuffers[i]);
+ }
+ }
+ IOLockUnlock(gGPULock);
+ }
+ if (!isInactive()) {
+ terminate();
+ }
+ return kIOReturnSuccess;
+}
+
+void
+RedwoodGPUUserClient::free(void)
+{
+ if (fBuffers != NULL) {
+ IODelete(fBuffers, struct gpu_buffer, GPU_BUFFERS);
+ fBuffers = NULL;
+ }
+ IOUserClient::free();
+}
+
+struct gpu_buffer *
+RedwoodGPUUserClient::buffer(uint32_t handle)
+{
+ if (handle == 0 || handle > GPU_BUFFERS || fBuffers[handle - 1].memory == NULL) {
+ return NULL;
+ }
+ return &fBuffers[handle - 1];
+}
+
+IOReturn
+RedwoodGPUUserClient::gemNew(struct msm_gem_new * request)
+{
+ uint64_t size = round_page_64(request->size);
+ struct gpu_buffer * b = NULL;
+ IOBufferMemoryDescriptor * memory;
+ bool mapped = true;
+
+ if (size == 0 || size > GPU_BUFFER_MAX) {
+ return kIOReturnBadArgument;
+ }
+ memory = IOBufferMemoryDescriptor::inTaskWithOptions(kernel_task, kIODirectionInOut | kIOMemoryKernelUserShared,
+ (vm_size_t) size, PAGE_SIZE);
+ if (memory == NULL) {
+ IOLog("RedwoodGPU: no memory for a buffer of %llu bytes (%u buffers, %llu MB with this client)\n", size, fBufferCount,
+ fBufferBytes >> 20);
+ return kIOReturnNoMemory;
+ }
+ bzero(memory->getBytesNoCopy(), size);
+ redwood_gpu_to_memory(memory->getBytesNoCopy(), size);
+
+ IOLockLock(gGPULock);
+ for (uint32_t i = 0; i < GPU_BUFFERS; i++) {
+ if (fBuffers[i].memory == NULL) {
+ b = &fBuffers[i];
+ request->handle = i + 1;
+ break;
+ }
+ }
+ if (b == NULL || (b->address = redwood_gpu_address(size)) == 0) {
+ IOLockUnlock(gGPULock);
+ memory->release();
+ IOLog("RedwoodGPU: no %s left for a buffer of %llu bytes\n", b ? "addresses" : "handles", size);
+ return kIOReturnNoMemory;
+ }
+ for (uint64_t offset = 0; offset < size && mapped; offset += 4096) {
+ IOByteCount length = 0;
+ addr64_t phys = memory->getPhysicalSegment(offset, &length, kIOMemoryMapperNone);
+
+ mapped = phys != 0 && redwood_gpu_map(b->address + offset, phys, request->flags & MSM_BO_GPU_READONLY);
+ }
+ redwood_gpu_mapped();
+ if (!mapped) {
+ redwood_gpu_unmap(b->address, size);
+ redwood_gpu_address_free(b->address, size);
+ IOLockUnlock(gGPULock);
+ memory->release();
+ IOLog("RedwoodGPU: a buffer of %llu bytes cannot be entered into the GPU's address space\n", size);
+ return kIOReturnNoMemory;
+ }
+ b->memory = memory;
+ b->size = size;
+ b->flags = request->flags;
+ fBufferCount++;
+ fBufferBytes += size;
+ IOLockUnlock(gGPULock);
+ return kIOReturnSuccess;
+}
+
+IOReturn
+RedwoodGPUUserClient::gemInfo(struct msm_gem_info * request)
+{
+ struct gpu_buffer * b;
+ IOReturn ret = kIOReturnSuccess;
+
+ IOLockLock(gGPULock);
+ b = buffer(request->handle);
+ if (b == NULL) {
+ ret = kIOReturnBadArgument;
+ } else if (request->info == MSM_INFO_GET_OFFSET) {
+ request->value = (uint64_t) request->handle << 32; /* nobody maps by it here */
+ } else if (request->info == MSM_INFO_GET_IOVA) {
+ request->value = b->address;
+ } else if (request->info == MSM_INFO_GET_FLAGS) {
+ request->value = b->flags;
+ } else if (request->info != MSM_INFO_SET_NAME) { /* a name for somebody's debugger: not kept */
+ ret = kIOReturnUnsupported;
+ }
+ IOLockUnlock(gGPULock);
+ return ret;
+}
+
+IOReturn
+RedwoodGPUUserClient::submit(struct msm_gem_submit * request)
+{
+ struct msm_gem_submit_bo * bos = NULL;
+ struct msm_gem_submit_cmd * cmds = NULL;
+ uint64_t addresses[GPU_SUBMIT_CMDS];
+ uint32_t words[GPU_SUBMIT_CMDS], count = 0;
+ IOReturn ret = kIOReturnBadArgument;
+
+ if (request->nr_cmds == 0 || request->nr_cmds > GPU_SUBMIT_CMDS || request->nr_bos == 0 || request->nr_bos > GPU_SUBMIT_BOS) {
+ IOLog("RedwoodGPU: a submission of %u command buffers in %u buffers: not taken\n", request->nr_cmds, request->nr_bos);
+ return kIOReturnBadArgument;
+ }
+ bos = IONew(struct msm_gem_submit_bo, request->nr_bos);
+ cmds = IONew(struct msm_gem_submit_cmd, request->nr_cmds);
+ if (bos == NULL || cmds == NULL) {
+ ret = kIOReturnNoMemory;
+ goto out;
+ }
+ if (copyin((user_addr_t) request->bos, bos, request->nr_bos * sizeof(*bos)) != 0 ||
+ copyin((user_addr_t) request->cmds, cmds, request->nr_cmds * sizeof(*cmds)) != 0) {
+ ret = kIOReturnVMError;
+ goto out;
+ }
+ IOLockLock(gGPULock);
+ /*
+ * The program writes its buffers through this processor's cache and the GPU reads memory,
+ * and the other way round for what the GPU draws: everything the submission names goes out
+ * of the cache now. (Mapped write-combined, as on Linux, none of this would be needed: see
+ * redwood_drm_map() in tools/mesa/redwood-drm.)
+ */
+ for (uint32_t i = 0; i < request->nr_bos; i++) {
+ struct gpu_buffer * b = buffer(bos[i].handle);
+
+ if (b != NULL) {
+ redwood_gpu_to_memory(b->memory->getBytesNoCopy(), b->size);
+ }
+ }
+ for (uint32_t i = 0; i < request->nr_cmds; i++) {
+ struct gpu_buffer * b = cmds[i].submit_idx < request->nr_bos ? buffer(bos[cmds[i].submit_idx].handle) : NULL;
+
+ if (cmds[i].type != MSM_SUBMIT_CMD_BUF) {
+ continue; /* the other kinds are for a GPU shared between address spaces */
+ }
+ if (b == NULL || cmds[i].nr_relocs != 0 || (cmds[i].size & 3) || (cmds[i].submit_offset & 3) ||
+ (uint64_t) cmds[i].submit_offset + cmds[i].size > b->size) {
+ IOLog("RedwoodGPU: command buffer %u of a submission: buffer %u, offset %u, %u bytes, %u relocations: not taken\n",
+ i, cmds[i].submit_idx, cmds[i].submit_offset, cmds[i].size, cmds[i].nr_relocs);
+ IOLockUnlock(gGPULock);
+ goto out;
+ }
+ addresses[count] = b->address + cmds[i].submit_offset;
+ words[count++] = cmds[i].size / 4;
+ }
+ request->fence = redwood_gpu_submit(addresses, words, count);
+ if (request->flags & MSM_SUBMIT_FENCE_FD_OUT) {
+ request->fence_fd = -1;
+ }
+ if (request->fence == 0 && !gGPUReported) {
+ gGPUReported = true;
+ redwood_gpu_report("the ring is full");
+ }
+ if (fSubmits++ < 4) {
+ IOLog("RedwoodGPU: submission %u: %u command buffers, the first at %#llx, %u words\n", request->fence, count,
+ addresses[0], words[0]);
+ }
+ IOLockUnlock(gGPULock);
+ ret = request->fence != 0 ? kIOReturnSuccess : kIOReturnTimeout;
+out:
+ if (bos != NULL) {
+ IODelete(bos, struct msm_gem_submit_bo, request->nr_bos);
+ }
+ if (cmds != NULL) {
+ IODelete(cmds, struct msm_gem_submit_cmd, request->nr_cmds);
+ }
+ return ret;
+}
+
+/* Pixels of the caller's (32 bits each, blue first) copied to the panel: for a test to show what it has drawn. */
+IOReturn
+RedwoodGPUUserClient::show(const struct redwood_show * request)
+{
+ uint64_t size = (uint64_t) request->rowBytes * request->height;
+ uint8_t * pixels;
+ IOReturn ret = kIOReturnSuccess;
+
+ if (request->width == 0 || request->width > 4096 || request->height == 0 || request->height > 4096 ||
+ request->rowBytes < 4 * request->width) {
+ return kIOReturnBadArgument;
+ }
+ pixels = (uint8_t *) IOMallocData(size);
+ if (pixels == NULL) {
+ return kIOReturnNoMemory;
+ }
+ if (copyin((user_addr_t) request->pixels, pixels, size) != 0) {
+ ret = kIOReturnVMError;
+ } else {
+ MiniIOMobileFramebufferShow(pixels, request->width, request->height, request->rowBytes);
+ }
+ IOFreeData(pixels, size);
+ return ret;
+}
+
+IOReturn
+RedwoodGPUUserClient::clientMemoryForType(UInt32 type, IOOptionBits * options, IOMemoryDescriptor ** memory)
+{
+ struct gpu_buffer * b;
+ IOReturn ret = kIOReturnBadArgument;
+
+ IOLockLock(gGPULock);
+ b = buffer(type);
+ if (b != NULL) {
+ b->memory->retain();
+ *memory = b->memory;
+ *options = 0;
+ ret = kIOReturnSuccess;
+ }
+ IOLockUnlock(gGPULock);
+ return ret;
+}
+
+IOReturn
+RedwoodGPUUserClient::externalMethod(uint32_t selector, IOExternalMethodArguments * args,
+ IOExternalMethodDispatch * dispatch, OSObject * target, void * reference)
+{
+ union {
+ struct msm_param param;
+ struct msm_gem_new gemNew;
+ struct msm_gem_info gemInfo;
+ struct msm_gem_cpu_prep cpuPrep;
+ struct msm_gem_submit submit;
+ struct msm_wait_fence waitFence;
+ struct msm_gem_madvise madvise;
+ struct msm_submitqueue queue;
+ struct msm_submitqueue_query query;
+ struct drm_gem_close close;
+ struct redwood_show show;
+ } r;
+ size_t size = args->structureInputSize;
+ struct gpu_buffer * b;
+ IOReturn ret = kIOReturnSuccess;
+ uint64_t zero = 0;
+
+ if (args->structureInput == NULL || size > sizeof(r) || fBuffers == NULL) {
+ return kIOReturnBadArgument;
+ }
+ if (!redwood_gpu_ready() && selector != kRedwoodShow) {
+ return kIOReturnNotReady; /* given back: the run is ending */
+ }
+ bzero(&r, sizeof(r));
+ memcpy(&r, args->structureInput, size);
+
+ switch (selector) {
+ case kMsmGetParam:
+ ret = redwood_gpu_parameter(r.param.param, &r.param.value) ? kIOReturnSuccess : kIOReturnBadArgument;
+ break;
+ case kMsmSetParam: /* the program's name, for Linux's lists of who uses the GPU */
+ case kMsmGemCpuFini:
+ case kMsmSubmitQueueClose:
+ break;
+ case kMsmGemNew:
+ ret = gemNew(&r.gemNew);
+ break;
+ case kMsmGemInfo:
+ ret = gemInfo(&r.gemInfo);
+ break;
+ case kMsmGemCpuPrep: /* is the GPU done with a buffer: with all it has been given, here */
+ IOLockLock(gGPULock);
+ if (!gpuWait(redwood_gpu_last(), (r.cpuPrep.op & MSM_PREP_NOSYNC) ? 0 : millisecondsUntil(&r.cpuPrep.timeout))) {
+ ret = kIOReturnBusy;
+ }
+ IOLockUnlock(gGPULock);
+ break;
+ case kMsmGemSubmit:
+ ret = submit(&r.submit);
+ break;
+ case kMsmWaitFence:
+ IOLockLock(gGPULock);
+ if (!gpuWait(r.waitFence.fence, millisecondsUntil(&r.waitFence.timeout))) {
+ ret = kIOReturnTimeout;
+ }
+ IOLockUnlock(gGPULock);
+ break;
+ case kMsmGemMadvise:
+ r.madvise.retained = 1;
+ break;
+ case kMsmSubmitQueueNew:
+ r.queue.id = ++fNextQueue;
+ break;
+ case kMsmSubmitQueueQuery: /* how often the GPU has faulted on this client's commands */
+ if (r.query.data != 0 && r.query.len >= sizeof(zero)) {
+ copyout(&zero, (user_addr_t) r.query.data, sizeof(zero));
+ }
+ break;
+ case kDrmGemClose:
+ IOLockLock(gGPULock);
+ b = buffer(r.close.handle);
+ if (b == NULL) {
+ ret = kIOReturnBadArgument;
+ } else {
+ gpuWait(redwood_gpu_last(), 1000);
+ freeBuffer(b);
+ }
+ IOLockUnlock(gGPULock);
+ break;
+ case kRedwoodShow:
+ ret = show(&r.show);
+ break;
+ default:
+ IOLog("RedwoodGPU: request %#x (%zu bytes) is not known here\n", selector, size);
+ ret = kIOReturnUnsupported;
+ break;
+ }
+ if (ret == kIOReturnSuccess && args->structureOutput != NULL) {
+ if (args->structureOutputSize > size) {
+ args->structureOutputSize = (uint32_t) size;
+ }
+ memcpy(args->structureOutput, &r, args->structureOutputSize);
+ }
+ return ret;
+}
diff --git a/xnu/redwood.patch b/xnu/redwood.patch
index b01d62e..a197a43 100644
--- a/xnu/redwood.patch
+++ b/xnu/redwood.patch
@@ -1319,16 +1319,17 @@ index 5219fb0d9ed6..29978e9cff43 100644
}
}
diff --git a/iokit/conf/files b/iokit/conf/files
-index 5fb0a52062e0..8dab3e0a364d 100644
+index 5fb0a52062e0..5edb1f225d18 100644
--- a/iokit/conf/files
+++ b/iokit/conf/files
-@@ -79,6 +79,10 @@ iokit/Kernel/IOSubMemoryDescriptor.cpp optional iokitcpp
+@@ -79,6 +79,11 @@ iokit/Kernel/IOSubMemoryDescriptor.cpp optional iokitcpp
iokit/Kernel/IOGuardPageMemoryDescriptor.cpp optional iokitcpp
iokit/Kernel/IOPlatformExpert.cpp optional iokitcpp
+iokit/Kernel/GenericARMPE.cpp optional iokitcpp
+iokit/Kernel/MiniIOSurface.cpp optional iokitcpp
+iokit/Kernel/MiniIOMobileFramebuffer.cpp optional iokitcpp
++iokit/Kernel/RedwoodGPU.cpp optional iokitcpp
+iokit/Kernel/MiniAppleKeyStore.cpp optional iokitcpp
iokit/Kernel/IOPlatformIO.cpp optional iokitcpp