Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.
C
GPU: the Adreno comes up and fills a frame that reaches the panel
vcvkkf0282871daa2
diff --git a/README.md b/README.mdindex 445ec6e..d3afede 100644--- a/README.md+++ b/README.md@@ -22,7 +22,7 @@|---|---|| Экран блокировки, разблокировка свайпом, домашний экран с полным набором значков (две страницы) | Звук, сеть (Wi-Fi/сотовая), Bluetooth, камера || Системные жесты: свайп снизу — «домой» и переключатель приложений, свайп от левого края — «назад», Пункт управления | Обои (чёрный фон) |-| Запуск приложений: Настройки (вглубь), Часы, Календарь, Фото, Заметки и др. | GPU/Metal — только программный рендер |+| Запуск приложений: Настройки (вглубь), Часы, Календарь, Фото, Заметки и др. | GPU для iOS: пока только программный рендер (сам Adreno уже заведён и заливает кадр, см. ниже) || Сенсорный экран телефона (Goodix GT9916), мультитач, пробуждение касанием | Без `cpus=N` работает одно ядро процессора || Несколько ядер процессора (`cpus=4`: четыре больших ядра телефона) | || Кнопки «Домой»/«Блокировка»/громкость (в QEMU — клавишами) | Зарядка под нашим ядром (телефон только разряжается) |@@ -152,6 +152,16 @@ XL_SECS=900 XL_CMDLINE="touchscreen=1 cpufreq=1 bootcpu=7 cpus=4" ./xl run #что ядра 1–3 малые, и сажает на них интерфейс. Паника «Ticket spinlock … timeout» значила бы, что ядрам тесново времени (в QEMU загрузчик сам увеличивает эти сроки в 20 раз): помогает `ml-timeout-global-scale=4`.+## GPU (в работе)++Adreno 642L телефона заведён под нашим ядром: загружается прошивка управляющего процессора (GMU), включается+3D-ядро, TrustZone принимает zap-шейдер, ядро GPU исполняет команды и заливает кадр, который драйвер экрана выводит+на панель (`docs/gpu-first-frame.png`). Это опыт за параметром `adreno=N` (шаги описаны в начале+`bsd/dev/redwood_adreno.c`), iOS пока по-прежнему рисует процессором. Прошивки GPU берутся с самого телефона+(`adb pull /vendor/firmware/a660_{gmu.bin,sqe.fw,zap.mdt,zap.b00,zap.b01,zap.b02}` в каталог `fw-gpu/`, в git они+не попадают) и едут в малом образе. Дальше — настоящая 3D-отрисовка и OpenGL ES для CoreAnimation: в QuartzCore+16.5 есть рабочий рендер через OpenGL ES, на который сервер окон переходит сам, если нет Metal.+## Сборка из исходниковГотовые бинарники в `prebuilt/` собраны из этого репозитория; пересобрать всё можно самому.diff --git a/docs/gpu-first-frame.png b/docs/gpu-first-frame.pngnew file mode 100644index 0000000..adee2ebBinary files /dev/null and b/docs/gpu-first-frame.png differdiff --git a/prebuilt/SHA256SUMS b/prebuilt/SHA256SUMSindex 00a79d9..3971cd3 100644--- a/prebuilt/SHA256SUMS+++ b/prebuilt/SHA256SUMS@@ -1,6 +1,6 @@-Built 2026-10-12 18:12 from this tree.+Built 2026-10-12 22:05 from this tree.-0439a8605c0830be4b6c13bb59205b44e18bc3ea77af53bdb19434d8e2cbb375 prebuilt/mach.development.bcm2837+6a90f6523071ced387d80c6a71918a86f68055ffe7bb72e49e7754a0835344b4 prebuilt/mach.development.bcm28371332a598d2dec00a1d8e930044e4f00a12ad44c6e5e50eecf4b7fe7fae803bf2 prebuilt/loader.efid2a35a4a1d6658ffb3696914f727661b52c9333de8a3af8a31642b8023ce03ac prebuilt/userland/child1a413befd511384052e1962c1394adce20f718e1f2657ce63ab09828ebc4bd1b prebuilt/userland/cpu_testdiff --git a/prebuilt/mach.development.bcm2837 b/prebuilt/mach.development.bcm2837index 7fa6a34..82714d2 100755Binary files a/prebuilt/mach.development.bcm2837 and b/prebuilt/mach.development.bcm2837 differdiff --git a/xnu/files/bsd/dev/redwood_adreno.c b/xnu/files/bsd/dev/redwood_adreno.cindex c831ca1..7bd0fe7 100644--- a/xnu/files/bsd/dev/redwood_adreno.c+++ b/xnu/files/bsd/dev/redwood_adreno.c@@ -1,6 +1,7 @@/*- * The GPU of the POCO X5 Pro (redwood), an Adreno 642L, as far as it can be woken without its- * firmware. Only with boot-arg adreno=N: every address below belongs to this phone.+ * The GPU of the POCO X5 Pro (redwood), an Adreno 642L, woken step by step up to a picture that+ * it fills and the panel shows. Only with boot-arg adreno=N: every address below belongs to this+ * phone.** Nothing in this system uses the GPU yet, and how much of it a kernel under Qualcomm's* hypervisor may reach is not known: an access the hypervisor does not like ends the run, at@@ -28,13 +29,28 @@* 12 asks TrustZone to check the signature of the GPU's "zap shader", the signed image* without which the 3D core stays locked in secure mode: its header goes from the* small image (fw/a660_zap.mdt) into memory of ours that TrustZone is pointed at+ * 13 writes page tables for the memory the GMU's firmware asks for, and the SMMU: a+ * translation context with those tables, and the GMU's requests sent through it+ * 14 writes the GMU's firmware (fw/a660_gmu.bin) into its own memory and into that memory+ * 15 writes the GMU's configuration, and lets its processor out of reset: the firmware runs+ * 16 writes the GMU's connection to RPMh; starts its message queues ("HFI")+ * 17 talks to the GMU: the tables it needs, and the request to power the 3D core, whose+ * registers are then read+ * 18 writes the 3D core's translation context, with its ring and its microcode+ * (fw/a660_sqe.fw) in it+ * 19 asks TrustZone to load the zap shader for good (once in the life of a started phone:+ * adreno_zap=0 in the runs after)+ * 20 writes the 3D core's registers; starts its command processor and gives it its first+ * command+ * 21 commands the 3D core out of secure mode, and to write a word into a page of ours+ * 22 commands a picture filled with three colours, and shows it on the panel+ * When the steps are over, what they started is stopped again, in reverse, and the CX half+ * switched off: nothing of the GPU block outlives the run (its SMMU contexts point into this+ * kernel's memory), and the next one starts from what the phone's own start leaves.*- * What is left after that: a translation context for the GMU, its firmware (a660_gmu.bin) loaded- * and started, a conversation with it (HFI) that makes it power the 3D core; then the core's- * microcode (a660_sqe.fw), and the rest of the zap shader's loading. The firmware is the phone's- * own, from /vendor/firmware of its Android. There is no register that holds the chip ID- * (0x06030500, "qcom,chipid" in the device tree): the fuses of step 2 are all the hardware says- * about itself, and the GMU's version is a property of its firmware.+ * The firmware is the phone's own, from /vendor/firmware of its Android. There is no register that holds the chip ID (0x06030500,+ * "qcom,chipid" in the device tree): the fuses of step 2 are all the hardware says about+ * itself, and the GMU's version is a property of its firmware.** More boot-args:* adreno_only=K step K alone (what steps 6 to 10 need, they check for themselves: within@@ -45,8 +61,14 @@* the end of our log to storage every two seconds, and after a silent reset* that note is all there is.* adreno_fine=1 the newest step waits like that before every single access+ * adreno_zap=0 step 19 has been done since the phone started: not again+ * adreno_blind=1 step 20 starts the command processor with its microcode nowhere, and+ * adreno_stream1=N step 18 leaves the core's second stream out (0) or gives it an empty+ * context of its own (2): what the SMMU then records shows what the core+ * asks for. With 0 and the microcode in place the phone resets.* adreno_watch=S after the last step, say once a second for S seconds (default 20) that* the machine is still alive, and what was written last+ * adreno_return=1 when all is over, go back to the loader instead of starting the system* The system does not go on starting until all of this is over, so that the end of the log is* ours; the touch screen driver's start would talk across it (leave touchscreen=1 out).*@@ -71,6 +93,7 @@extern vm_offset_t ml_io_map(vm_offset_t phys_addr, vm_size_t size);extern vm_offset_t ml_io_map_wcomb(vm_offset_t phys_addr, vm_size_t size);+extern uint64_t kvtophys(vm_offset_t va); /* pmap_paddr_t */static uint32_t last_step, skip_steps, pause_ms = 2200, fine, watch_seconds = 20;static uint32_t step; /* the one being performed */@@ -78,6 +101,32 @@ static uint32_t newest_step; /* the highest one asked for: the one thstatic char last_write[96] = "nothing written";static bool quiet; /* repeating accesses that have been announced and seen to work: say nothing */+/*+ * The kernel's log reaches the loader when a run is over, and a run that ends in a reset+ * without a word leaves none. The loader also reads a log of the kernel's from another core+ * while it runs (loader_return.c; xl keeps it with XL_FOLLOW): every line said here goes there+ * as well, before the next register is touched.+ */+extern void loader_log_putc(char c);++static int say(const char *format, ...) __printflike(1, 2);++static int+say(const char *format, ...)+{+ char line[400];+ va_list list;++ va_start(list, format);+ vsnprintf(line, sizeof(line), format, list);+ va_end(list);+ for (const char *c = line; *c != 0; c++) {+ loader_log_putc(*c);+ }+ return printf("%s", line);+}+#define printf say+static voidsleep_us(uint32_t us){@@ -253,15 +302,27 @@ struct cmd_db_entry {uint16_t length, offset;};+static const struct cmd_db_header *cmd_db;+/*- * Say where RPMh has a resource and, for a supply rail, which levels it knows: the GMU's firmware- * is told the levels by their position in this list.+ * Where RPMh has a resource (0: nowhere) and, for a supply rail, the levels it knows: the GMU's+ * firmware is told the levels by their position in this list.*/-static void-show_rpmh_resource(const struct cmd_db_header *db, const char *name)+static uint32_t+rpmh_resource(const char *name, uint16_t *levels, unsigned *level_count){- char id[8] = { 0 }, levels[160];+ const struct cmd_db_header *db = cmd_db;+ char id[8] = { 0 };+ if (level_count != NULL) {+ *level_count = 0;+ }+ if (db == NULL) {+ db = cmd_db = (const struct cmd_db_header *)ml_io_map_wcomb(CMD_DB_PHYS, CMD_DB_SIZE); /* memory, not registers */+ }+ if (db->magic[0] != 0xdb || db->magic[1] != 0x30 || db->magic[2] != 0x03 || db->magic[3] != 0x0c) {+ return 0;+ }strlcpy(id, name, sizeof(id));for (unsigned s = 0; s < 8; s++) {const struct cmd_db_slave *slave = &db->slave[s];@@ -273,7 +334,6 @@ show_rpmh_resource(const struct cmd_db_header *db, const char *name)for (unsigned i = 0; i < slave->count; i++) {const uint8_t *data = db->data + slave->data_offset + entry[i].offset;unsigned count = entry[i].length / 2u;- int at = 0;if (memcmp(entry[i].id, id, sizeof(id)) != 0) {continue;@@ -281,27 +341,42 @@ show_rpmh_resource(const struct cmd_db_header *db, const char *name)if ((unsigned)slave->data_offset + entry[i].offset + entry[i].length > CMD_DB_SIZE - 0x1000) {count = 0;}- levels[0] = 0;- for (unsigned l = 0; l < count && l < 16 && at < (int)sizeof(levels) - 8; l++) {- unsigned level = data[2 * l] | data[2 * l + 1] << 8;-- if (l > 0 && level == 0) {+ for (unsigned l = 0; levels != NULL && l < count && l < 16; l++) {+ levels[l] = (uint16_t)(data[2 * l] | data[2 * l + 1] << 8);+ if (l > 0 && levels[l] == 0) {break;}- at += snprintf(levels + at, sizeof(levels) - (size_t)at, " %u", level);+ *level_count = l + 1;}- printf("adreno: %u: RPMh has \"%s\" at %#x; levels:%s\n", step, name, entry[i].address, levels);- return;+ return entry[i].address;}}- printf("adreno: %u: RPMh has no \"%s\"\n", step, name);+ return 0;+}++static void+show_rpmh_resource(const char *name)+{+ uint16_t level[16];+ unsigned count = 0;+ uint32_t address = rpmh_resource(name, level, &count);+ char levels[160];+ int at = 0;++ if (address == 0) {+ printf("adreno: %u: RPMh has no \"%s\"\n", step, name);+ return;+ }+ levels[0] = 0;+ for (unsigned l = 0; l < count && at < (int)sizeof(levels) - 8; l++) {+ at += snprintf(levels + at, sizeof(levels) - (size_t)at, " %u", level[l]);+ }+ printf("adreno: %u: RPMh has \"%s\" at %#x; levels:%s\n", step, name, address, levels);}static boolstep_outside(void){- const struct cmd_db_header *db;-announce("read the GPU's clocks in the global clock controller (registers the touch screen driver uses too)");get(&gcc, "pll votes", GCC_PLL_VOTE);get(&gcc, "gpu_cfg_ahb", GCC_GPU_CFG_AHB_CBCR);@@ -310,15 +385,9 @@ step_outside(void)get(&gcc, "ddrss_gpu_axi", GCC_DDRSS_GPU_AXI_CBCR);announce("look up the GPU's supply rails in the boot loader's list of RPMh resources (memory at 0x80860000)");- db = (const struct cmd_db_header *)ml_io_map_wcomb(CMD_DB_PHYS, CMD_DB_SIZE); /* memory, not registers */- if (db->magic[0] != 0xdb || db->magic[1] != 0x30 || db->magic[2] != 0x03 || db->magic[3] != 0x0c) {- printf("adreno: %u: no list there (magic %02x %02x %02x %02x)\n", step, db->magic[0], db->magic[1],- db->magic[2], db->magic[3]);- return true; /* nothing before step 10 depends on it */- }- show_rpmh_resource(db, "gfx.lvl"); /* the 3D core's own rail (dt: VDD_GFX_LEVEL, PM8350C S7) */- show_rpmh_resource(db, "cx.lvl"); /* the rail of the CX half, shared with most of the SoC */- show_rpmh_resource(db, "mx.lvl"); /* memories */+ show_rpmh_resource("gfx.lvl"); /* the 3D core's own rail (dt: VDD_GFX_LEVEL, PM8350C S7) */+ show_rpmh_resource("cx.lvl"); /* the rail of the CX half, shared with most of the SoC */+ show_rpmh_resource("mx.lvl"); /* memories */return true;}@@ -606,6 +675,7 @@ step_clocks(void)#define GPUCC_DEBUG_CBCR 0x1100#define GPUCC_DEBUG_MUX 0x1568 /* [7:0] which of the GPU's clocks */#define GPUCC_DEBUG_CXO_AON 0x0b+#define GPUCC_DEBUG_GX_GFX3D 0x0d /* the 3D core's own clock */#define GPUCC_DEBUG_GX_GMU 0x11#define GPUCC_DEBUG_AHB 0x12#define GPUCC_DEBUG_CX_GMU 0x1a@@ -866,12 +936,41 @@ step_first_words(void)* kernel asks RPMh for the touch screen's supply, but the GMU's own; through them its firmware* raises the 3D core's rail. Offsets in words: a6xx_gmu.xml, domain A6XX_RSCC.*/-static struct block rscc = { "rscc", 0x03de0000, 0x1000 };+static struct block rscc = { "rscc", 0x03de0000, 0x2000 };#define RSCC_RSC_STATUS0_DRV0 (4 * 0x004)#define RSCC_PDC_SLAVE_ID_DRV0 (4 * 0x00b)#define RSCC_SEQ_BUSY_DRV0 (4 * 0x101)#define RSCC_SEQ_MEM_0_DRV0 (4 * 0x180)#define RSCC_TCS0_DRV0_STATUS (4 * 0x346)+#define RSCC_ASLEEP (1u << 16) /* of rsc_status0: handed over to the always-on sequencer */+#define RSCC_DRV_CONFIG 0x00c+#define RSCC_SLOTS 0xd00 /* +0 interrupts enabled, +4 pending */+#define RSCC_SLOT_SIZE 0x2a0++/*+ * The channel's state and its four command slots (Linux's rpmh-rsc.c: slot n at 0xd00 + 0x2a0 n:+ * +0x10 commands whose completion is waited for, +0x14 control, +0x18 idle, +0x1c commands+ * enabled, then 20 bytes a command from +0x30: number, address, data, status).+ */+static void+rscc_show(void)+{+ get(&rscc, "drv config", RSCC_DRV_CONFIG);+ get(&rscc, "rsc_status0_drv0", RSCC_RSC_STATUS0_DRV0);+ get(&rscc, "seq_busy_drv0", RSCC_SEQ_BUSY_DRV0);+ get(&rscc, "slots' interrupts enabled", RSCC_SLOTS);+ get(&rscc, "slots' interrupts pending", RSCC_SLOTS + 4);+ for (uint32_t slot = 0; slot < 4; slot++) {+ vm_offset_t at = rscc.base + RSCC_SLOTS + RSCC_SLOT_SIZE * slot;++ printf("adreno: %u: rscc slot %u: wait %#x, control %#x, idle %#x, enabled %#x; commands (number, address, data, "+ "status):\n", step, slot, rd(at + 0x10), rd(at + 0x14), rd(at + 0x18), rd(at + 0x1c));+ printf("adreno: %u: %x %x %x %x | %x %x %x %x | %x %x %x %x | %x %x %x %x\n", step,+ rd(at + 0x30), rd(at + 0x34), rd(at + 0x38), rd(at + 0x3c), rd(at + 0x44), rd(at + 0x48), rd(at + 0x4c),+ rd(at + 0x50), rd(at + 0x58), rd(at + 0x5c), rd(at + 0x60), rd(at + 0x64), rd(at + 0x6c), rd(at + 0x70),+ rd(at + 0x74), rd(at + 0x78));+ }+}static boolstep_rscc(void)@@ -1025,6 +1124,287 @@ step_gmu_memory(void)return back[0] == pattern[0] && back[1] == pattern[1];}+/* ---- memory the GPU block reaches ------------------------------------------------------------------- */++/*+ * Whatever in the GPU block reads or writes memory -- the GMU's processor, later the 3D core --+ * does it through the SMMU, which makes of the address it uses ("the GPU's address" below) a+ * physical one by walking page tables of ours, as a processor's MMU does: here always three+ * levels of 4K pages, 39 bits of address (dt: qcom,use-3-lvl-tables). The SMMU has seven such+ * translations ("contexts") and a table of which stream of requests -- 0 is the 3D core, 5 the+ * GMU (dt: iommus) -- goes through which. Formats and the order of programming: Linux+ * drivers/iommu/io-pgtable-arm.c and arm-smmu.c.+ */+#define PTE_TABLE 3ull+#define PTE_PAGE 3ull+#define PTE_CACHED (1ull << 2) /* memory attribute 1 of SMMU_MAIR0; without it 0, uncached */+#define PTE_UNPRIVILEGED (1ull << 6) /* the 3D core's accesses are; the GMU's are not */+#define PTE_SHARED (2ull << 8)+#define PTE_ACCESSED (1ull << 10)+#define PTE_NOT_GLOBAL (1ull << 11)+#define PTE_ADDRESS 0x0000fffffffff000ull+#define PTE_GMU (PTE_SHARED | PTE_ACCESSED | PTE_NOT_GLOBAL)++#define SMMU_S2CR_FAULT (2u << 16)+#define SMMU_SMR_VALID (1u << 31)+#define SMMU_STREAM_MASK 0x400 /* dt: the second cell of every "iommus" of this SMMU */+#define SMMU_CBAR_STAGE_1 (1u << 16 | 3u << 8 | 0xfu << 12) /* our tables alone decide; weakest defaults */+#define SMMU_CBA2R(n) (0x800 + 4 * (n)) /* in the second page, like CBAR */+#define SMMU_CBA2R_64_BIT 1+#define SMMU_CB_TCR2 0x10+#define SMMU_CB_TTBR1 0x28+#define SMMU_CB_MAIR0 0x38+#define SMMU_CB_MAIR1 0x3c+#define SMMU_CB_FAR 0x60+#define SMMU_CB_FSYNR0 0x68+#define SMMU_CB_TLBIASID 0x610+#define SMMU_CB_TLBSYNC 0x7f0+#define SMMU_CB_TLBSTATUS 0x7f4+#define SMMU_SCTLR_ON (1u << 0 | 1u << 1 | 1u << 2 | 1u << 5 | 1u << 12) /* on; faults recorded, no interrupt */+#define SMMU_TCR_39_BITS (25u | 1u << 8 | 1u << 10 | 3u << 12 | 1u << 23) /* one table of 4K pages, cached walks */+#define SMMU_TCR2 (1u | 1u << 4 | 7u << 15) /* 36 bits of physical address, 16 of context tag */+#define SMMU_MAIR0 0xf404ff44 /* Linux's: 0 uncached, 1 cached, 2 device */+#define SMMU_FSR_ALL 0xc00001fe /* every kind of fault, and "several" */++extern void loader_return(void);++/* Until every write to memory so far has been carried out. */+static void+settle(void)+{+ __asm__ volatile ("dsb sy");+}++/* Push what this processor has written to a stretch of memory out of its cache, and forget it: the GPU block reads memory. */+static void+to_memory(vm_offset_t start, vm_size_t size)+{+ for (vm_offset_t line = start & ~(vm_offset_t)63; line < start + size; line += 64) {+ __asm__ volatile ("dc civac, %0" : : "r" (line));+ }+ settle();+}++#define SPACE_TABLES 48++struct gpu_space {+ const char *name;+ uint32_t context; /* which of the SMMU's */+ uint32_t control; /* bits of its control register besides the usual */+ uint32_t table_count;+ uint64_t *root;+ struct {+ uint64_t phys;+ uint64_t *table;+ } tables[SPACE_TABLES];+};++static uint64_t *+space_new_table(struct gpu_space *space)+{+ static vm_offset_t spare;+ static unsigned spare_left;+ uint64_t *table;++ if (space->table_count == SPACE_TABLES) {+ return NULL;+ }+ if (spare_left == 0) {+ if (kmem_alloc(kernel_map, &spare, PAGE_SIZE, KMA_KOBJECT | KMA_ZERO | KMA_DATA, VM_KERN_MEMORY_IOKIT) !=+ KERN_SUCCESS) {+ return NULL;+ }+ spare_left = PAGE_SIZE / 4096;+ }+ table = (uint64_t *)(spare + PAGE_SIZE - 4096 * spare_left--);+ space->tables[space->table_count].phys = kvtophys((vm_offset_t)table);+ space->tables[space->table_count++].table = table;+ return table;+}++/* The table an entry of a higher table leads to; made if the entry is empty. */+static uint64_t *+space_next(struct gpu_space *space, uint64_t *entry)+{+ uint64_t *table;++ if (*entry & 1) {+ for (uint32_t i = 0; i < space->table_count; i++) {+ if (space->tables[i].phys == (*entry & PTE_ADDRESS)) {+ return space->tables[i].table;+ }+ }+ return NULL;+ }+ table = space_new_table(space);+ if (table != NULL) {+ *entry = kvtophys((vm_offset_t)table) | PTE_TABLE;+ }+ return table;+}++/* One 4K page. */+static bool+space_map(struct gpu_space *space, uint64_t address, uint64_t phys, uint64_t attributes)+{+ uint64_t *level2, *level3;++ if (address >> 39 || (space->root == NULL && (space->root = space_new_table(space)) == NULL)) {+ return false;+ }+ level2 = space_next(space, &space->root[address >> 30 & 0x1ff]);+ level3 = level2 != NULL ? space_next(space, &level2[address >> 21 & 0x1ff]) : NULL;+ if (level3 == NULL) {+ return false;+ }+ level3[address >> 12 & 0x1ff] = phys | attributes | PTE_PAGE;+ return true;+}++/* Memory of ours that the GPU block sees at an address of its own. */+struct gpu_block {+ uint64_t address; /* the GPU's */+ uint32_t size;+ vm_offset_t memory; /* ours */+};++static bool+block_new(struct gpu_space *space, struct gpu_block *block, uint64_t address, uint32_t size, uint64_t attributes)+{+ block->address = address;+ block->size = (size + 4095) & ~4095u;+ if (kmem_alloc(kernel_map, &block->memory, round_page(block->size), KMA_KOBJECT | KMA_ZERO | KMA_DATA,+ VM_KERN_MEMORY_IOKIT) != KERN_SUCCESS) {+ return false;+ }+ for (uint32_t offset = 0; offset < block->size; offset += 4096) {+ if (!space_map(space, address + offset, kvtophys(block->memory + offset), attributes)) {+ return false;+ }+ }+ to_memory(block->memory, block->size);+ printf("adreno: %u: %s sees %#llx..%#llx in memory of ours (%#llx)\n", step, space->name, address,+ address + block->size - 1, kvtophys(block->memory));+ return true;+}++/*+ * A second view of a block without this processor's cache in between, for memory that both+ * sides write a word at a time. Only blocks of one page of ours: those are in one piece.+ */+static vm_offset_t+block_uncached(const struct gpu_block *block)+{+ uint64_t phys = kvtophys(block->memory);++ for (uint32_t offset = 0; offset < block->size; offset += 4096) {+ if (kvtophys(block->memory + offset) != phys + offset) {+ return 0;+ }+ }+ return ml_io_map_wcomb(phys, block->size);+}++static uint32_t+smmu_context(const struct gpu_space *space)+{+ return (smmu_pages + space->context) * smmu_page;+}++/* What a context has recorded of a request it could not serve. */+static bool+space_faulted(const struct gpu_space *space)+{+ uint32_t context = smmu_context(space), status = rd(smmu.base + context + SMMU_CB_FSR);++ if ((status & SMMU_FSR_ALL) == 0) { /* bits 10:9 only tell the format of the syndrome */+ return false;+ }+ printf("adreno: %u: THE SMMU REFUSED %s: status %#x, address %#llx, syndrome %#x\n", step, space->name, status,+ (uint64_t)rd(smmu.base + context + SMMU_CB_FAR + 4) << 32 | rd(smmu.base + context + SMMU_CB_FAR),+ rd(smmu.base + context + SMMU_CB_FSYNR0));+ return true;+}++/*+ * One more stream through a context that is on. A match is taken only as the device tree has+ * it, a stream and the mask 0x400: one entry for streams 0 and 1 (mask 0x401) reads back 0.+ */+static bool+space_stream(struct gpu_space *space, uint32_t match, uint32_t stream)+{+ put(&smmu, "s2cr", SMMU_S2CR(match), space->context);+ put(&smmu, "smr", SMMU_SMR(match), SMMU_SMR_VALID | SMMU_STREAM_MASK << 16 | stream);+ if (get(&smmu, "smr", SMMU_SMR(match)) != (SMMU_SMR_VALID | SMMU_STREAM_MASK << 16 | stream) ||+ (get(&smmu, "s2cr", SMMU_S2CR(match)) & 0x300ff) != space->context) {+ printf("adreno: %u: the SMMU did not take what was written: stream %u is not going through context %u\n", step,+ stream, space->context);+ return false;+ }+ return true;+}++/* Give a context our tables and send a stream through it; `match` is the entry of the stream table to use. */+static bool+space_on(struct gpu_space *space, uint32_t match, uint32_t stream)+{+ uint32_t context = smmu_context(space);+ uint64_t root = space->tables[0].phys | (uint64_t)space->context << 48; /* tagged with the context's number */++ for (uint32_t i = 0; i < space->table_count; i++) {+ to_memory((vm_offset_t)space->tables[i].table, 4096);+ }+ put(&smmu, "cb sctlr", context + SMMU_CB_SCTLR, 0);+ put(&smmu, "cba2r", smmu_page + SMMU_CBA2R(space->context), SMMU_CBA2R_64_BIT);+ put(&smmu, "cbar", smmu_page + SMMU_CBAR(space->context), SMMU_CBAR_STAGE_1);+ put(&smmu, "cb tcr2", context + SMMU_CB_TCR2, SMMU_TCR2);+ put(&smmu, "cb tcr", context + SMMU_CB_TCR, SMMU_TCR_39_BITS);+ put(&smmu, "cb ttbr0, low", context + SMMU_CB_TTBR0, (uint32_t)root);+ put(&smmu, "cb ttbr0, high", context + SMMU_CB_TTBR0 + 4, (uint32_t)(root >> 32));+ put(&smmu, "cb ttbr1, low", context + SMMU_CB_TTBR1, 0);+ put(&smmu, "cb ttbr1, high", context + SMMU_CB_TTBR1 + 4, 0);+ put(&smmu, "cb mair0", context + SMMU_CB_MAIR0, SMMU_MAIR0);+ put(&smmu, "cb mair1", context + SMMU_CB_MAIR1, 0);+ put(&smmu, "cb fsr", context + SMMU_CB_FSR, SMMU_FSR_ALL);+ put(&smmu, "cb tlbiasid", context + SMMU_CB_TLBIASID, space->context);+ put(&smmu, "cb tlbsync", context + SMMU_CB_TLBSYNC, 1);+ if (!wait_for(&smmu, context + SMMU_CB_TLBSTATUS, 1, 0, 100)) {+ printf("adreno: %u: the SMMU does not finish forgetting old translations (%#x)\n", step,+ rd(smmu.base + context + SMMU_CB_TLBSTATUS));+ return false;+ }+ put(&smmu, "cb sctlr", context + SMMU_CB_SCTLR, SMMU_SCTLR_ON | space->control);+ put(&smmu, "s2cr", SMMU_S2CR(match), space->context);+ put(&smmu, "smr", SMMU_SMR(match), SMMU_SMR_VALID | SMMU_STREAM_MASK << 16 | stream);++ /* the global ones are the hypervisor's to grant: see what it made of them */+ if (get(&smmu, "smr", SMMU_SMR(match)) != (SMMU_SMR_VALID | SMMU_STREAM_MASK << 16 | stream) ||+ (get(&smmu, "s2cr", SMMU_S2CR(match)) & 0x300ff) != space->context ||+ (get(&smmu, "cbar", smmu_page + SMMU_CBAR(space->context)) & 0x30000) != (SMMU_CBAR_STAGE_1 & 0x30000) ||+ !(get(&smmu, "cb sctlr", context + SMMU_CB_SCTLR) & 1) ||+ get(&smmu, "cb ttbr0, low", context + SMMU_CB_TTBR0) != (uint32_t)root ||+ get(&smmu, "cb ttbr0, high", context + SMMU_CB_TTBR0 + 4) != (uint32_t)(root >> 32)) {+ printf("adreno: %u: the SMMU did not take what was written: stream %u is not going through context %u\n", step,+ stream, space->context);+ return false;+ }+ get(&smmu, "cb tcr", context + SMMU_CB_TCR);+ get(&smmu, "cba2r", smmu_page + SMMU_CBA2R(space->context));+ get(&smmu, "cb tcr2", context + SMMU_CB_TCR2);+ get(&smmu, "cb fsr", context + SMMU_CB_FSR);+ get(&smmu, "sgfsr", SMMU_SGFSR);+ return true;+}++/* A stream goes nowhere again, and its context translates nothing. */+static void+space_off(struct gpu_space *space, uint32_t match)+{+ put(&smmu, "smr", SMMU_SMR(match), 0);+ put(&smmu, "s2cr", SMMU_S2CR(match), SMMU_S2CR_FAULT);+ put(&smmu, "cb sctlr", smmu_context(space) + SMMU_CB_SCTLR, 0);+}+/* ---- step 11: TrustZone -------------------------------------------------------------------------------- *//*@@ -1054,11 +1434,13 @@ step_gmu_memory(void)/* One call. What TrustZone returns: 0 or an error of its own, and a word of answer. */static int32_t-scm(const char *what, uint64_t function, uint64_t arguments, uint64_t first, uint64_t second, uint64_t *answer)+scm(const char *what, uint64_t function, uint64_t arguments, uint64_t first, uint64_t second, uint64_t third,+ uint64_t *answer){uint64_t session = 0;- printf("adreno: %u: about to ask TrustZone %s: smc %#llx, arguments %#llx, %#llx\n", step, what, function, first, second);+ printf("adreno: %u: about to ask TrustZone %s: smc %#llx, arguments %#llx, %#llx, %#llx\n", step, what, function, first,+ second, third);snprintf(last_write, sizeof(last_write), "step %u asked TrustZone %#llx(%#llx, %#llx)", step, function, first, second);hold(true);for (int i = 0; i < 100; i++) {@@ -1066,7 +1448,7 @@ scm(const char *what, uint64_t function, uint64_t arguments, uint64_t first, uinregister uint64_t x1 __asm__("x1") = arguments;register uint64_t x2 __asm__("x2") = first;register uint64_t x3 __asm__("x3") = second;- register uint64_t x4 __asm__("x4") = 0;+ register uint64_t x4 __asm__("x4") = third;register uint64_t x5 __asm__("x5") = 0;register uint64_t x6 __asm__("x6") = session;register uint64_t x7 __asm__("x7") = 0;@@ -1096,10 +1478,10 @@ step_trustzone(void)announce("ask TrustZone what Linux asks it first: is the call \"is this call available\" available");if (scm("whether it can be asked", how | SCM_IS_CALL_AVAILABLE, SCM_ONE_VALUE, SCM_SIP | SCM_IS_CALL_AVAILABLE, 0,- &answer) != 0 || answer != 1) {+ 0, &answer) != 0 || answer != 1) {how = SCM_FAST | SCM_SIP; /* the same with 32-bit arguments, Linux's second try */if (scm("the same, the 32-bit way", how | SCM_IS_CALL_AVAILABLE, SCM_ONE_VALUE, SCM_SIP | SCM_IS_CALL_AVAILABLE,- 0, &answer) != 0 || answer != 1) {+ 0, 0, &answer) != 0 || answer != 1) {printf("adreno: %u: TrustZone does not answer the way it answers Linux\n", step);return false;}@@ -1107,18 +1489,18 @@ step_trustzone(void)announce("ask TrustZone whether it has the calls that authenticate a firmware image");scm("for the call that takes an image's signature", how | SCM_IS_CALL_AVAILABLE, SCM_ONE_VALUE,- SCM_SIP | SCM_PAS_INIT_IMAGE, 0, &answer);+ SCM_SIP | SCM_PAS_INIT_IMAGE, 0, 0, &answer);scm("for the call that authenticates and starts an image", how | SCM_IS_CALL_AVAILABLE, SCM_ONE_VALUE,- SCM_SIP | SCM_PAS_AUTH_AND_RESET, 0, &answer);+ SCM_SIP | SCM_PAS_AUTH_AND_RESET, 0, 0, &answer);announce("ask TrustZone which images it \"supports\": three this phone starts through it, and the GPU's; each asked ""both ways, at once and (as Linux does) as a call that may be preempted");for (unsigned i = 0; i < 4; i++) {answer = 0;- fast[i] = scm(images[i].name, how | SCM_PAS_IS_SUPPORTED, SCM_ONE_VALUE, images[i].id, 0, &answer) == 0 && answer;+ fast[i] = scm(images[i].name, how | SCM_PAS_IS_SUPPORTED, SCM_ONE_VALUE, images[i].id, 0, 0, &answer) == 0 && answer;answer = 0;slow[i] = scm(images[i].name, (how & ~SCM_FAST) | SCM_PAS_IS_SUPPORTED, SCM_ONE_VALUE, images[i].id, 0,- &answer) == 0 && answer;+ 0, &answer) == 0 && answer;}printf("adreno: %u: \"supported\", asked at once / as Linux asks: audio %u/%u, modem %u/%u, video %u/%u, GPU %u/%u\n",step, fast[0], slow[0], fast[1], slow[1], fast[2], slow[2], fast[3], slow[3]);@@ -1133,7 +1515,6 @@ step_trustzone(void)extern dev_t mdevlookup(int devid);extern int mdevgetkernelrange(dev_t dev, vm_offset_t *base, uint64_t *size);-extern uint64_t kvtophys(vm_offset_t va); /* pmap_paddr_t *//* An entry of a directory of a packfs image in memory, by name. */static const struct packfs_dnode *@@ -1210,16 +1591,15 @@ image_file(const char *directory, const char *name, uint64_t *size)* ("creates SHM bridges on behalf of Linux"), and that TrustZone protects the memory while it* works: so it is not touched again here, and never freed.*/+/* The zap shader's header in a page of ours, as TrustZone wants it; where that page is in physical memory. */static bool-step_zap_signature(void)+zap_header(uint64_t *phys){const uint8_t *header;vm_offset_t page = 0;- uint64_t size = 0, answer = ~0ull;+ uint64_t size = 0;uint32_t segments, hashes;- int32_t result;- announce("find the zap shader's header, fw/a660_zap.mdt, in the small image that came with the kernel");header = image_file("fw", "a660_zap.mdt", &size);if (header == NULL) {return false;@@ -1243,15 +1623,24 @@ step_zap_signature(void)return false;}memcpy((void *)page, header, size);- /* TrustZone reads memory, not this processor's cache */- for (vm_offset_t line = page; line < page + PAGE_SIZE; line += 64) {- __asm__ volatile ("dc cvac, %0" : : "r" (line));- }- __asm__ volatile ("dsb sy");+ to_memory(page, PAGE_SIZE); /* TrustZone reads memory, not this processor's cache */+ *phys = kvtophys(page);+ return true;+}++static bool+step_zap_signature(void)+{+ uint64_t header = 0, answer = ~0ull;+ int32_t result;+ announce("find the zap shader's header, fw/a660_zap.mdt, in the small image that came with the kernel");+ if (!zap_header(&header)) {+ return false;+ }announce("hand the header to TrustZone to check its signature (image 13); a call that may be preempted, as Linux's is");result = scm("to take the zap shader's signature", SCM_64 | SCM_SIP | SCM_PAS_INIT_IMAGE, SCM_VALUE_AND_BUFFER, PAS_GPU,- kvtophys(page), &answer);+ header, 0, &answer);if (result == 0 && answer == 0) {printf("adreno: %u: TRUSTZONE ACCEPTS THE ZAP SHADER'S SIGNATURE FROM THIS KERNEL\n", step);return true;@@ -1260,62 +1649,1375 @@ step_zap_signature(void)return false;}-/* ---- all of it together ---------------------------------------------------------------------------- */--static bool(*const steps[])(void) = {- step_outside, step_fuses, step_gpucc, step_cx_power, step_clocks, step_first_words, step_rscc, step_smmu,- step_smmu_context, step_gmu_memory, step_trustzone, step_zap_signature,-};-#define STEPS ((uint32_t)(sizeof(steps) / sizeof(steps[0])))+/* ---- step 13: memory for the GMU ------------------------------------------------------------------------ *//*- * Called while the system starts, before there is a root file system, and the start waits: for- * as long as the steps and the watch after them last, nothing else writes to the log, so that- * its end -- all that the loader's notes keep -- is about what is being done here.+ * The GMU is a Cortex-M3 with 16K of code memory and 16K of data memory of its own (written+ * through registers, step 10); the rest of its firmware and everything it shares with us it+ * reaches through the SMMU, at addresses the firmware file names: a file of blocks, each "this+ * goes at address A" or, with no data, "I need N bytes at address A" (vendor's+ * adreno_a6xx_gmu.c: a6xx_gmu_parse_fw, a6xx_gmu_load_fw). Besides those, where the vendor's+ * driver puts them: 16K for the firmware's log and a page for the table and queues of the+ * message protocol ("HFI").*/-void-redwood_adreno_init(void)+#define GMU_ITCM_ADDRESS 0x00000000u+#define GMU_DTCM_ADDRESS 0x10004000u /* a6xx_gmu_fw_load: the 650 family's */+#define GMU_TCM_SIZE 0x4000u+#define GMU_LOG_ADDRESS 0x60002000u /* after the two pages the firmware asks for at 0x60000000 */+#define GMU_LOG_SIZE 0x4000u+#define GMU_HFI_ADDRESS 0x60006000u+#define GMU_STREAM 5+#define GMU_MATCH 1 /* entry of the stream table; 0 is kept for the 3D core */++struct gmu_firmware_block {+ uint32_t address, size, type, value;+};+#define GMU_FIRMWARE_NEEDS 1 /* types of blocks without data: memory wanted, value = bytes */+#define GMU_FIRMWARE_NEEDS_KEPT 7++static struct gpu_space gmu_space = { .name = "the GMU", .context = 1 };+static struct gpu_block gmu_blocks[12], gmu_log, gmu_hfi;+static unsigned gmu_block_count;+static const uint8_t *gmu_firmware;+static uint64_t gmu_firmware_size;+static bool gmu_space_is_on, gmu_runs, gmu_talks, core_is_on, rsc_awake, cx_off;+static void gmu_post_mortem(void);++static bool+in_tcm(uint32_t address){- uint32_t only = 0, done = 0;+ return address - GMU_ITCM_ADDRESS < GMU_TCM_SIZE || address - GMU_DTCM_ADDRESS < GMU_TCM_SIZE;+}- PE_parse_boot_argn("adreno", &last_step, sizeof(last_step));- PE_parse_boot_argn("adreno_skip", &skip_steps, sizeof(skip_steps));- if (PE_parse_boot_argn("adreno_only", &only, sizeof(only)) && only > 0 && only <= STEPS) {- last_step = only;- skip_steps = ~(1u << only);- }- if (last_step == 0) {- return;- }- if (last_step > STEPS) {- last_step = STEPS;+static bool+step_gmu_space(void)+{+ if (!cx_ready()) {+ return false;}- PE_parse_boot_argn("adreno_pause", &pause_ms, sizeof(pause_ms));- PE_parse_boot_argn("adreno_fine", &fine, sizeof(fine));- PE_parse_boot_argn("adreno_watch", &watch_seconds, sizeof(watch_seconds));- for (newest_step = last_step; newest_step > 0 && (skip_steps & 1u << newest_step); newest_step--) {+ announce("find the GMU's firmware, fw/a660_gmu.bin, in the small image, and make the memory it asks for");+ gmu_firmware = image_file("fw", "a660_gmu.bin", &gmu_firmware_size);+ if (gmu_firmware == NULL) {+ return false;}+ for (uint64_t at = 0; at + sizeof(struct gmu_firmware_block) <= gmu_firmware_size;) {+ struct gmu_firmware_block block;- printf("adreno: steps up to %u, without those in mask %#x; step %u is the new one and waits %u ms %s\n", last_step,- skip_steps, newest_step, pause_ms, fine ? "before every access" : "after each announcement");- for (step = 1; step <= last_step; step++) {- if (skip_steps & 1u << step) {+ memcpy(&block, gmu_firmware + at, sizeof(block));+ at += sizeof(block) + block.size;+ if (at > gmu_firmware_size) {+ printf("adreno: %u: the file ends inside a block\n", step);+ return false;+ }+ if (block.size != 0 || (block.type != GMU_FIRMWARE_NEEDS && block.type != GMU_FIRMWARE_NEEDS_KEPT) ||+ in_tcm(block.address)) {continue;}- if (!steps[step - 1]()) {- printf("adreno: step %u: FAILED; the steps after it are not attempted\n", step);- break;+ if (gmu_block_count == sizeof(gmu_blocks) / sizeof(gmu_blocks[0]) || (block.address & 4095) ||+ block.value > 0x100000 || !block_new(&gmu_space, &gmu_blocks[gmu_block_count++], block.address, block.value,+ PTE_GMU)) {+ printf("adreno: %u: cannot make %#x bytes at %#x for the firmware\n", step, block.value, block.address);+ return false;}- printf("adreno: step %u: done\n", step);- done = step;+ }+ if (!block_new(&gmu_space, &gmu_log, GMU_LOG_ADDRESS, GMU_LOG_SIZE, PTE_GMU) ||+ !block_new(&gmu_space, &gmu_hfi, GMU_HFI_ADDRESS, PAGE_SIZE, PTE_GMU)) {+ return false;+ }++ announce("give the SMMU's context 1 these tables and send the GMU's requests (stream 5) through it: the first "+ "writes to the SMMU, the global ones (cbar, cba2r, s2cr, smr) through the hypervisor");+ if (!smmu_layout()) {+ return false;+ }+ gmu_space_is_on = true; /* whatever of it has been written by the time something fails is taken back */+ return space_on(&gmu_space, GMU_MATCH, GMU_STREAM);+}++/* ---- step 14: the GMU's firmware ----------------------------------------------------------------------- */++#define GMU_ITCM GMU_WORD(0x0c00)++static bool+step_gmu_firmware(void)+{+ uint32_t wrong = 0;++ if (!cx_ready() || gmu_firmware == NULL) {+ return false;+ }+ announce("load the GMU's firmware: its first 16K of code and of data through the GMU's registers (0x3d6d000, "+ "0x3d71000), the rest into the memory just made");+ put(&gmu, "cm3_sysreset", GMU_CM3_SYSRESET, 1);+ for (int pass = 0; pass < 2; pass++) {+ for (uint64_t at = 0; at + sizeof(struct gmu_firmware_block) <= gmu_firmware_size;) {+ struct gmu_firmware_block block;+ const uint8_t *data = gmu_firmware + at + sizeof(block);++ memcpy(&block, gmu_firmware + at, sizeof(block));+ at += sizeof(block) + block.size;+ if (block.size == 0) {+ continue;+ }+ if (in_tcm(block.address) && in_tcm(block.address + block.size - 1)) {+ uint32_t registers = block.address < GMU_DTCM_ADDRESS ? GMU_ITCM + block.address - GMU_ITCM_ADDRESS :+ GMU_DTCM + block.address - GMU_DTCM_ADDRESS;++ for (uint32_t offset = 0; offset + 4 <= block.size; offset += 4) {+ uint32_t word;++ memcpy(&word, data + offset, sizeof(word));+ if (pass == 0) {+ wr(gmu.base + registers + offset, word);+ } else if (rd(gmu.base + registers + offset) != word) {+ wrong++;+ }+ }+ if (pass == 1) {+ printf("adreno: %u: %#x bytes at the GMU's %#x, through its registers\n", step, block.size,+ block.address);+ }+ continue;+ }+ for (unsigned i = 0; pass == 0 && i <= gmu_block_count; i++) {+ if (i == gmu_block_count) {+ printf("adreno: %u: the firmware has %#x bytes for %#x, where it asked for no memory\n", step,+ block.size, block.address);+ return false;+ }+ if (block.address >= gmu_blocks[i].address &&+ block.address + block.size <= gmu_blocks[i].address + gmu_blocks[i].size) {+ memcpy((void *)(gmu_blocks[i].memory + (block.address - gmu_blocks[i].address)), data, block.size);+ to_memory(gmu_blocks[i].memory, gmu_blocks[i].size);+ printf("adreno: %u: %#x bytes at the GMU's %#x, in memory\n", step, block.size, block.address);+ break;+ }+ }+ }+ }+ snprintf(last_write, sizeof(last_write), "step %u wrote the GMU's firmware into its code and data memory", step);+ if (wrong) {+ printf("adreno: %u: %u words of the GMU's own memory do not read back as written\n", step, wrong);+ return false;+ }+ printf("adreno: %u: THE GMU'S FIRMWARE IS LOADED; its versions, read back from the GMU: core %#x, power %#x, HFI %#x\n",+ step, rd(gmu.base + GMU_DTCM_VERSIONS), rd(gmu.base + GMU_DTCM_VERSIONS + 8), rd(gmu.base + GMU_DTCM_VERSIONS + 16));+ return true;+}++/* ---- step 15: start the GMU ---------------------------------------------------------------------------- */++#define GMU_ICACHE_CONFIG GMU_WORD(0x4c00)+#define GMU_DCACHE_CONFIG GMU_WORD(0x4c01)+#define GMU_SYS_BUS_CONFIG GMU_WORD(0x4c0f)+#define GMU_PWR_COL_INTER_FRAME_CTRL GMU_WORD(0x50c0)+#define GMU_RPMH_CTRL GMU_WORD(0x50e8)+#define GMU_RPMH_CTRL_ALL 0xf11 /* the RPMh interface, and every vote the firmware may cast */+#define GMU_CX_FAL_INTF GMU_WORD(0x50f0)+#define GMU_CX_FALNEXT_INTF GMU_WORD(0x50f1)+#define GMU_PWR_COL_CP_MSG GMU_WORD(0x5100) /* at start: where the log is, and its pages - 1 */+#define GMU_PWR_COL_CP_RESP GMU_WORD(0x5101) /* ... and how far it was written before */+#define GMU_HFI_SFR_ADDR GMU_WORD(0x5182) /* at start: which GPU this is */+#define GMU_HFI_QTBL_INFO GMU_WORD(0x5184)+#define GMU_HFI_QTBL_ADDR GMU_WORD(0x5185)+#define GMU_HFI_CTRL_INIT GMU_WORD(0x5186)+#define GMU_GMU2HOST_INTR_CLR GMU_WORD(0x5191)+#define GMU_GMU2HOST_INTR_INFO GMU_WORD(0x5192) /* bit 0: a message for us; bit 31: the 3D core is ours */+#define GMU_HOST2GMU_INTR_SET GMU_WORD(0x5194) /* bit 0: a message for the GMU; bit 30, 31: see step 17 */+#define GMU_RSCC_CONTROL_REQ GMU_WORD(0x9307)+#define GMU_AO_AHB_FENCE_CTRL GMU_WORD(0x9310)+#define GMU_AHB_FENCE_RANGE_0 GMU_WORD(0x9311)+#define GMU_CHIP 0x06035000 /* dt qcom,chipid 0x06030500 the way the firmware wants it */+#define GMU_STARTED_MASK 0x1ff /* of fw_init_result, for firmware newer than core 0x20010004 */+#define GMU_STARTED 0x100+#define GMU_FAILED 0xe00++/* The table at the head of the HFI page and the three queues after it, as the vendor's driver lays them out. */+struct hfi_queue {+ uint32_t status, start, type, words, message_words, unused[5], read, write;+};+struct hfi_table {+ uint32_t version, words, first_queue, queue_words, queues, active_queues;+ struct hfi_queue queue[17];+};+#define HFI_QUEUES 3 /* 0: ours to the GMU; 1: its answers; 2: its debugging messages */+#define HFI_QUEUE_WORDS 1024+#define HFI_QUEUE_AT(n) (((uint32_t)sizeof(struct hfi_table) + 15) / 16 * 16 + 4096 * (n))++static volatile struct hfi_table *hfi; /* the uncached view */++/*+ * Everything the vendor's driver writes before it lets the processor go (a6xx_gmu_register_config,+ * a6xx_gmu_power_config), in its order; then out of reset (a6xx_gmu_device_start).+ */+static bool+step_gmu_start(void)+{+ static const uint8_t priority[HFI_QUEUES] = { 10, 10, 40 };+ uint32_t result = 0;++ if (!cx_ready() || !gmu_space_is_on) {+ return false;+ }+ hfi = (volatile struct hfi_table *)block_uncached(&gmu_hfi);+ if (hfi == NULL) {+ printf("adreno: %u: the page for the messages is not in one piece\n", step);+ return false;+ }+ hfi->words = sizeof(struct hfi_table) / 4;+ hfi->first_queue = 6;+ hfi->queue_words = sizeof(struct hfi_queue) / 4;+ hfi->queues = hfi->active_queues = HFI_QUEUES;+ for (unsigned i = 0; i < HFI_QUEUES; i++) {+ hfi->queue[i].start = GMU_HFI_ADDRESS + HFI_QUEUE_AT(i);+ hfi->queue[i].type = i | (uint32_t)priority[i] << 8;+ hfi->queue[i].words = HFI_QUEUE_WORDS;+ hfi->queue[i].status = 1;+ }++ announce("tell the GMU where things are, by its registers: the message table, the log, which GPU this is; its "+ "caches and its use of RPMh; then let its processor out of reset");+ if (get(&gmu, "cm3_sysreset", GMU_CM3_SYSRESET) != 1) {+ printf("adreno: %u: the GMU's processor is not held in reset: not configuring it while it runs\n", step);+ return false;}/*- * Punishment for a write may come late (a reset some 17 seconds after it has been seen):- * keep the end of the log saying where things stand.+ * Our own leaving (gpu_block_stop) hands the GMU's RPMh channel to the always-on sequencer, and+ * until it is taken back nothing the GMU asks of RPMh gets there: the vendor's+ * a6xx_rscc_wakeup_sequence, before the firmware starts.*/- for (uint32_t second = 1; second <= watch_seconds; second++) {- sleep_ms(1000);- printf("adreno: alive %u s after step %u; %s\n", second, done, last_write);+ rscc_show();+ if (rd(rscc.base + RSCC_RSC_STATUS0_DRV0) & RSCC_ASLEEP) {+ put(&gmu, "rscc_control_req", GMU_RSCC_CONTROL_REQ, 2);+ if (!wait_for(&gmu, GMU_RSCC_CONTROL_ACK, 2, 2, 100) || !wait_for(&rscc, RSCC_SEQ_BUSY_DRV0, ~0u, 0, 100)) {+ printf("adreno: %u: the GMU's RPMh channel does not wake\n", step);+ }+ get(&gmu, "rscc_control_ack", GMU_RSCC_CONTROL_ACK);+ put(&gmu, "rscc_control_req", GMU_RSCC_CONTROL_REQ, 0);+ get(&rscc, "rsc_status0_drv0", RSCC_RSC_STATUS0_DRV0);+ get(&rscc, "seq_busy_drv0", RSCC_SEQ_BUSY_DRV0);+ }+ rsc_awake = true;+ put(&gmu, "cx_falnext_intf", GMU_CX_FALNEXT_INTF, 1);+ put(&gmu, "cx_fal_intf", GMU_CX_FAL_INTF, 1);+ put(&gmu, "general_7", GMU_GENERAL_7, 1);+ put(&gmu, "cm3_fw_init_result", GMU_CM3_FW_INIT_RESULT, 0);+ put(&gmu, "cm3_boot_config", GMU_CM3_BOOT_CONFIG, 2);+ put(&gmu, "hfi_qtbl_addr", GMU_HFI_QTBL_ADDR, GMU_HFI_ADDRESS);+ put(&gmu, "hfi_qtbl_info", GMU_HFI_QTBL_INFO, 1);+ put(&gmu, "ahb_fence_range_0", GMU_AHB_FENCE_RANGE_0, 1u << 31 | (0xa << 2) << 18 | 0x8a0);+ put(&gmu, "cm3_cfg", GMU_CM3_CFG, 0x4052);+ put(&gbif, "halt", GBIF_HALT, 0);+ put(&gmu, "pwr_col_cp_resp", GMU_PWR_COL_CP_RESP, 0);+ put(&gmu, "hfi_sfr_addr", GMU_HFI_SFR_ADDR, GMU_CHIP);+ put(&gmu, "pwr_col_cp_msg", GMU_PWR_COL_CP_MSG, GMU_LOG_ADDRESS | (GMU_LOG_SIZE / 4096 - 1));+ put(&gmu, "sys_bus_config", GMU_SYS_BUS_CONFIG, 1);+ put(&gmu, "icache_config", GMU_ICACHE_CONFIG, 1);+ put(&gmu, "dcache_config", GMU_DCACHE_CONFIG, 1);+ put(&gmu, "pwr_col_inter_frame_ctrl", GMU_PWR_COL_INTER_FRAME_CTRL, 0x9c40400);+ set(&gmu, "rpmh_ctrl", GMU_RPMH_CTRL, GMU_RPMH_CTRL_ALL);+ put(&gmu, "gmu2host_intr_clr", GMU_GMU2HOST_INTR_CLR, 0xffffffff);++ gmu_runs = true;+ put(&gmu, "cm3_sysreset", GMU_CM3_SYSRESET, 0);+ for (int i = 0; i < 5000 && (result & GMU_STARTED_MASK) != GMU_STARTED && !(result & GMU_FAILED); i++) {+ sleep_us(100);+ result = rd(gmu.base + GMU_CM3_FW_INIT_RESULT);+ }+ get(&gmu, "cm3_fw_init_result", GMU_CM3_FW_INIT_RESULT);+ get(&gmu, "gmu2host_intr_info", GMU_GMU2HOST_INTR_INFO);+ get(&gmu, "ao_host_interrupt_status", GMU_AO_HOST_INTERRUPT_STATUS);+ get(&gmu, "hfi_ctrl_status", GMU_HFI_CTRL_STATUS);+ if (space_faulted(&gmu_space) || (result & GMU_STARTED_MASK) != GMU_STARTED) {+ printf("adreno: %u: the GMU's firmware does not come up\n", step);+ gmu_post_mortem();+ return false;+ }+ printf("adreno: %u: THE GMU'S FIRMWARE RUNS\n", step);+ return true;+}++/* ---- step 16: the GMU's way to RPMh, and its message queues ------------------------------------------------ */++/*+ * The GMU raises and drops the 3D core's supply by itself, through command slots of its own+ * (step 7) and a sequencer in the always-on part of the SoC, "PDC". On this SoC the always-on+ * processor programs the PDC; left to us are its two enabling registers and the GMU side's+ * little program (vendor's a6xx_load_pdc_ucode with pdc_in_aop, a6xx_load_rsc_ucode: after the+ * firmware has started and before the message queues are).+ */+static struct block pdc = { "pdc", 0x0b290000, 0x10000 }; /* dt: qcom,gmu "kgsl_gmu_pdc_cfg" */+#define PDC_GPU_ENABLE (4 * 0x1140)+#define PDC_GPU_SEQ_START_ADDR (4 * 0x1148)+#define RSCC_PDC_SEQ_START_ADDR (4 * 0x008)+#define RSCC_PDC_MATCH_VALUE_LO (4 * 0x009)+#define RSCC_PDC_MATCH_VALUE_HI (4 * 0x00a)+#define RSCC_HIDDEN_TCS_CMD0_ADDR (4 * 0x00d)+#define RSCC_HIDDEN_TCS_CMD0_DATA (4 * 0x00e)+#define RSCC_OVERRIDE_START_ADDR (4 * 0x100)++static bool+step_gmu_queues(void)+{+ static const uint32_t program[5] = { 0xeaaae5a0, 0xe1a1ebab, 0xa2e0a581, 0xecac82e2, 0x0020edad };++ if (!cx_ready() || !gmu_runs) {+ return false;+ }+ announce("enable the GPU's part of the always-on sequencer (pdc, 0xb294500 and 0xb294520: a part of the SoC not "+ "touched before)");+ get(&pdc, "gpu_enable", PDC_GPU_ENABLE);+ get(&pdc, "gpu_seq_start_addr", PDC_GPU_SEQ_START_ADDR);+ put(&pdc, "gpu_seq_start_addr", PDC_GPU_SEQ_START_ADDR, 0);+ put(&pdc, "gpu_enable", PDC_GPU_ENABLE, 0x80000001);+ get(&pdc, "gpu_enable", PDC_GPU_ENABLE);++ announce("give the GMU's RPMh channel its sleep and wake program (rscc, 0x3de0000), unless it has it");+ if (get(&rscc, "seq_mem_0_drv0", RSCC_SEQ_MEM_0_DRV0) == program[0] &&+ get(&rscc, "hidden_tcs_cmd2_data", RSCC_HIDDEN_TCS_CMD0_DATA + 4 * 4) == 0x80000021) {+ goto programmed;+ }+ put(&rscc, "rsc_status0_drv0", RSCC_RSC_STATUS0_DRV0, 1u << 24);+ put(&rscc, "pdc_slave_id_drv0", RSCC_PDC_SLAVE_ID_DRV0, 1);+ put(&rscc, "hidden_tcs_cmd0_data", RSCC_HIDDEN_TCS_CMD0_DATA, 0);+ put(&rscc, "hidden_tcs_cmd0_addr", RSCC_HIDDEN_TCS_CMD0_ADDR, 0);+ put(&rscc, "hidden_tcs_cmd1_data", RSCC_HIDDEN_TCS_CMD0_DATA + 4 * 2, 0);+ put(&rscc, "hidden_tcs_cmd1_addr", RSCC_HIDDEN_TCS_CMD0_ADDR + 4 * 2, 0);+ put(&rscc, "hidden_tcs_cmd2_data", RSCC_HIDDEN_TCS_CMD0_DATA + 4 * 4, 0x80000021); /* a650 and later */+ put(&rscc, "hidden_tcs_cmd2_addr", RSCC_HIDDEN_TCS_CMD0_ADDR + 4 * 4, 0);+ put(&rscc, "override_start_addr", RSCC_OVERRIDE_START_ADDR, 0);+ put(&rscc, "pdc_seq_start_addr", RSCC_PDC_SEQ_START_ADDR, 0x4520);+ put(&rscc, "pdc_match_value_lo", RSCC_PDC_MATCH_VALUE_LO, 0x4510);+ put(&rscc, "pdc_match_value_hi", RSCC_PDC_MATCH_VALUE_HI, 0x4514);+ for (unsigned i = 0; i < 5; i++) {+ put(&rscc, "seq_mem_drv0", RSCC_SEQ_MEM_0_DRV0 + 4 * i, program[i]);+ }+ get(&rscc, "seq_mem_0_drv0", RSCC_SEQ_MEM_0_DRV0);++programmed:+ announce("ask the GMU to start its message queues");+ put(&gmu, "hfi_ctrl_init", GMU_HFI_CTRL_INIT, 1);+ if (!wait_for(&gmu, GMU_HFI_CTRL_STATUS, 1, 1, 500)) {+ get(&gmu, "hfi_ctrl_status", GMU_HFI_CTRL_STATUS);+ get(&gmu, "cm3_fw_init_result", GMU_CM3_FW_INIT_RESULT);+ space_faulted(&gmu_space);+ printf("adreno: %u: the GMU does not start its message queues\n", step);+ return false;+ }+ get(&gmu, "hfi_ctrl_status", GMU_HFI_CTRL_STATUS);+ get(&gmu, "hfi_version_info", GMU_HFI_VERSION_INFO);+ gmu_talks = true;+ printf("adreno: %u: THE GMU'S MESSAGE QUEUES ARE UP\n", step);+ return !space_faulted(&gmu_space);+}++/* ---- step 17: talk to the GMU; the 3D core powered ---------------------------------------------------------- */++/*+ * A message is words: the first says which (bits 7:0), how many words (15:8), a request or an+ * answer (19:16), and a number (31:20) that the answer repeats in its second word; the third word+ * of an answer is the GMU's verdict, 0 for done. Vendor's adreno_a6xx_hfi.c, Linux a6xx_hfi.c.+ */+#define HFI_BW_TABLE 3+#define HFI_PERF_TABLE 4+#define HFI_START 10+#define HFI_CORE_FW_START 14+#define HFI_GX_BW_PERF_VOTE 30+#define HFI_PREPARE_SLUMBER 33+#define HFI_ERROR 100+#define HFI_ANSWER 1 /* in bits 19:16 */+#define HFI_NO_INDEX 0xff++struct hfi_perf_table {+ uint32_t header, gpu_levels, gmu_levels;+ struct {+ uint32_t vote, acd, khz;+ } gpu[16];+ struct {+ uint32_t vote, khz;+ } gmu[4];+};+struct hfi_bw_table {+ uint32_t header, levels, cnoc_commands, ddr_commands, cnoc_wait, ddr_wait;+ uint32_t cnoc_address[6], cnoc_data[2][6], ddr_address[8], ddr_data[16][8];+};++static uint32_t hfi_number;+static bool post_mortem_done;++/* Everything waiting in a queue of the GMU's; true if the answer to the message numbered `number` was among it. */+static bool+hfi_receive(unsigned which, uint32_t number, uint32_t *verdict)+{+ volatile struct hfi_queue *queue = &hfi->queue[which];+ volatile uint32_t *data = (volatile uint32_t *)((vm_offset_t)hfi + HFI_QUEUE_AT(which));+ bool answered = false;++ while (queue->read != queue->write) {+ uint32_t message[8] = { 0 }, at = queue->read % HFI_QUEUE_WORDS, words = data[at] >> 8 & 0xff;++ for (uint32_t i = 0; i < words; i++) {+ if (i < 8) {+ message[i] = data[at];+ }+ at = (at + 1) % HFI_QUEUE_WORDS;+ }+ queue->read = (at + 3) / 4 * 4 % HFI_QUEUE_WORDS;+ if ((message[0] >> 16 & 0xf) == HFI_ANSWER && message[1] >> 20 == number) {+ *verdict = message[2];+ answered = true;+ } else {+ message[7] = 0;+ if ((message[0] & 0xff) == HFI_ERROR) { /* vendor's adreno_a6xx_receive_err_req */+ printf("adreno: %u: THE GMU REPORTS AN ERROR: %u, %u, \"%s\"\n", step, message[1] >> 16,+ message[1] & 0xffff, (const char *)&message[2]);+ } else {+ printf("adreno: %u: the GMU says (queue %u, %u words): %#x %#x %#x %#x %#x %#x\n", step, which, words,+ message[0], message[1], message[2], message[3], message[4], message[5]);+ }+ }+ if (words == 0) {+ break;+ }+ }+ return answered;+}++/* A word of the GMU's own data, by the GMU's address for it; only what lies in memory of ours. */+static uint32_t+gmu_data(uint32_t address)+{+ for (unsigned i = 0; i < gmu_block_count; i++) {+ if (address - (uint32_t)gmu_blocks[i].address < gmu_blocks[i].size - 3) {+ vm_offset_t at = gmu_blocks[i].memory + (address - (uint32_t)gmu_blocks[i].address);++ to_memory(at, 4); /* what the GMU wrote, not what our cache remembers */+ return *(volatile uint32_t *)at;+ }+ }+ return 0xdeadbeef;+}++/*+ * What can be seen of a GMU that has stopped answering: the tail of its log (16 bytes an event:+ * which, the crystal's counter, two values; the firmware's function at 0x1a98; where it writes+ * next is in pwr_col_cp_resp), its record of what it was asked for and what it has (firmware+ * 1.8's data at 0x108454: +0x28 is what is in flight, +0x40.. what is in force), its RPMh+ * channel, and the 3D core's switch and PLLs, which only the GMU's firmware writes.+ */+#define GMU_STATE_ADDRESS 0x108454++static void+gmu_post_mortem(void)+{+ uint32_t next = rd(gmu.base + GMU_PWR_COL_CP_RESP) & 0xffff, entries = GMU_LOG_SIZE / 16;+ const volatile uint32_t *log = (const volatile uint32_t *)gmu_log.memory;++ to_memory(gmu_log.memory, GMU_LOG_SIZE);+ printf("adreno: %u: the GMU's log (event, time, two values), %u events in, the last 40:\n", step, next);+ for (uint32_t i = next > 40 ? next - 40 : 0; i < next && i + 4 <= entries; i += 4) {+ printf("adreno: %u: %x %x %x %x | %x %x %x %x | %x %x %x %x | %x %x %x %x\n", step,+ log[4 * i], log[4 * i + 1], log[4 * i + 2], log[4 * i + 3],+ log[4 * i + 4], log[4 * i + 5], log[4 * i + 6], log[4 * i + 7],+ log[4 * i + 8], log[4 * i + 9], log[4 * i + 10], log[4 * i + 11],+ log[4 * i + 12], log[4 * i + 13], log[4 * i + 14], log[4 * i + 15]);+ }+ for (uint32_t i = 0; i < 24; i += 8) {+ printf("adreno: %u: the GMU's state +%#x: %x %x %x %x %x %x %x %x\n", step, 4 * i,+ gmu_data(GMU_STATE_ADDRESS + 4 * i), gmu_data(GMU_STATE_ADDRESS + 4 * i + 4),+ gmu_data(GMU_STATE_ADDRESS + 4 * i + 8), gmu_data(GMU_STATE_ADDRESS + 4 * i + 12),+ gmu_data(GMU_STATE_ADDRESS + 4 * i + 16), gmu_data(GMU_STATE_ADDRESS + 4 * i + 20),+ gmu_data(GMU_STATE_ADDRESS + 4 * i + 24), gmu_data(GMU_STATE_ADDRESS + 4 * i + 28));+ }+ rscc_show();+ get(&gpucc, "gx_gdscr", GPUCC_GX_GDSCR);+ get(&gpucc, "gx_sw_reset", GPUCC_GX_SW_RESET);+ get(&gpucc, "gx_domain_misc", GPUCC_GX_DOMAIN_MISC);+ get(&gpucc, "pll0 mode", GPUCC_PLL0_MODE);+ get(&gpucc, "pll0 multiplier", GPUCC_PLL0_MODE + 4);+ get(&gpucc, "pll1 mode", GPUCC_PLL1_MODE);+ get(&gpucc, "pll1 multiplier", GPUCC_PLL1_MODE + 4);+ get(&gmu, "rpmh_power_state", GMU_RPMH_POWER_STATE);+ get(&gmu, "sptprac_pwr_clk_status", GMU_SPTPRAC_PWR_CLK_STATUS);+ get(&gmu, "ao_host_interrupt_status", GMU_AO_HOST_INTERRUPT_STATUS);+ get(&gmu, "ahb_fence_status", GMU_AHB_FENCE_STATUS);+}++/* Send a message whose first word is yet to be made and wait for the GMU's verdict on it. */+static bool+hfi_send(const char *what, uint32_t id, uint32_t *message, uint32_t words)+{+ volatile struct hfi_queue *queue = &hfi->queue[0];+ volatile uint32_t *data = (volatile uint32_t *)((vm_offset_t)hfi + HFI_QUEUE_AT(0));+ uint32_t at = queue->write % HFI_QUEUE_WORDS, number = ++hfi_number & 0xfff, verdict = 0, waiting = 0;+ bool answered = false;++ message[0] = id | words << 8 | number << 20;+ printf("adreno: %u: about to send the GMU %s: message %u, %u words, number %u\n", step, what, id, words, number);+ snprintf(last_write, sizeof(last_write), "step %u sent the GMU message %u (%s)", step, id, what);+ hold(true);+ for (uint32_t i = 0; i < (words + 3) / 4 * 4; i++) {+ data[at] = i < words ? message[i] : 0xfafafafa;+ at = (at + 1) % HFI_QUEUE_WORDS;+ }+ queue->write = at;+ settle();+ wr(gmu.base + GMU_HOST2GMU_INTR_SET, 1);+ for (int i = 0; i < 10000 && !answered; i++) {+ sleep_us(100);+ waiting = rd(gmu.base + GMU_GMU2HOST_INTR_INFO);+ if (waiting & 1) {+ wr(gmu.base + GMU_GMU2HOST_INTR_CLR, 1);+ answered = hfi_receive(1, number, &verdict);+ }+ if (rd(gmu.base + GMU_CM3_FW_INIT_RESULT) & GMU_FAILED) {+ break;+ }+ }+ hfi_receive(2, ~0u, &waiting);+ if (!answered || verdict != 0) {+ printf("adreno: %u: the GMU %s (%#x) to %s; fw_init_result %#x, gmu2host_intr_info %#x\n", step,+ answered ? "says no" : "does not answer", verdict, what, rd(gmu.base + GMU_CM3_FW_INIT_RESULT),+ rd(gmu.base + GMU_GMU2HOST_INTR_INFO));+ space_faulted(&gmu_space);+ if (!post_mortem_done) {+ post_mortem_done = true;+ gmu_post_mortem();+ }+ return false;+ }+ printf("adreno: %u: the GMU accepts %s\n", step, what);+ return true;+}++/*+ * A vote for a level of a supply: the level, its place in RPMh's list for the rail, and the place+ * of the level the memories' rail must have with it (vendor's setup_volt_dependency_tbl).+ */+static bool+rail_vote(uint32_t level, const uint16_t *rail, unsigned rail_levels, const uint16_t *mx, unsigned mx_levels, uint32_t *vote)+{+ unsigned at, with = 0;++ for (at = 0; at < rail_levels && rail[at] < level; at++) {+ }+ if (at == rail_levels || mx_levels == 0) {+ return false;+ }+ while (with + 1 < mx_levels && mx[with] < rail[at]) {+ with++;+ }+ *vote = (uint32_t)rail[at] << 16 | with << 8 | at;+ return true;+}++/* dt: kgsl_3d0_reg_memory: the 3D core's registers; a6xx.xml counts in words */+static struct block gpu = { "gpu", 0x03d00000, 0x40000 };+#define GPU_WORD(n) (4u * (n))+#define RBBM_INT_0_STATUS GPU_WORD(0x0201)+#define RBBM_STATUS GPU_WORD(0x0210)+#define RBBM_CLOCK_CNTL GPU_WORD(0x00ae)+#define CP_RB_BASE GPU_WORD(0x0800)+#define GMU_CORE_REQUEST (1u << 30) /* of host2gmu_intr_set: keep the 3D core up for us; bit 31: no more */+#define GMU_CORE_GRANTED (1u << 31) /* of gmu2host_intr_info */++/*+ * What the vendor's driver sends at this point (a6xx_hfi_start; Linux the same with fewer+ * levels): the table of speeds of the 3D core with the supply level each needs -- the phone's+ * three lowest (dt: qcom,gpu-pwrlevels: 315, 450, 550 MHz at "levels" 64, 128, 192) -- and of the+ * GMU itself; the table of bus bandwidth votes, of which Linux gets by with the one for "off"+ * (a6xx_hfi.c: adreno_7c3_build_bw_table); "start"; and the speed wanted now. Then the request+ * to keep the 3D core powered while we use it (a6xx_gmu_oob_set), and its registers can be read.+ */+static bool+step_core_power(void)+{+ static const struct {+ uint32_t khz, level;+ } speeds[] = { { 0, 0 }, { 315000, 64 }, { 450000, 128 }, { 550000, 192 } };+ static struct hfi_perf_table perf;+ static struct hfi_bw_table bw;+ uint32_t core_start[2] = { 0, 0 }, start[1] = { 0 }, vote[4] = { 0, 1, 1, HFI_NO_INDEX }, granted = 0;+ uint16_t gfx[16], cx[16], mx[16];+ unsigned gfx_levels, cx_levels, mx_levels;++ if (!cx_ready() || !gmu_talks) {+ return false;+ }+ if (!rpmh_resource("gfx.lvl", gfx, &gfx_levels) || !rpmh_resource("cx.lvl", cx, &cx_levels) ||+ !rpmh_resource("mx.lvl", mx, &mx_levels)) {+ printf("adreno: %u: RPMh's list lacks one of the rails\n", step);+ return false;+ }+ perf.gpu_levels = sizeof(speeds) / sizeof(speeds[0]);+ for (unsigned i = 0; i < perf.gpu_levels; i++) {+ perf.gpu[i].khz = speeds[i].khz;+ perf.gpu[i].acd = 0xffffffff; /* no adaptive clocking */+ if (!rail_vote(speeds[i].level, gfx, gfx_levels, mx, mx_levels, &perf.gpu[i].vote)) {+ printf("adreno: %u: RPMh has no level %u for the 3D core's rail\n", step, speeds[i].level);+ return false;+ }+ }+ perf.gmu_levels = 2;+ perf.gmu[1].khz = 200000;+ if (!rail_vote(0, cx, cx_levels, mx, mx_levels, &perf.gmu[0].vote) ||+ !rail_vote(48, cx, cx_levels, mx, mx_levels, &perf.gmu[1].vote)) { /* "minimum SVS", the vendor's choice */+ return false;+ }+ printf("adreno: %u: votes: 3D core %#x %#x %#x %#x, GMU %#x %#x\n", step, perf.gpu[0].vote, perf.gpu[1].vote,+ perf.gpu[2].vote, perf.gpu[3].vote, perf.gmu[0].vote, perf.gmu[1].vote);++ bw.levels = 1;+ bw.ddr_commands = 3;+ bw.ddr_wait = 7;+ bw.ddr_address[0] = rpmh_resource("SH0", NULL, NULL);+ bw.ddr_address[1] = rpmh_resource("MC0", NULL, NULL);+ bw.ddr_address[2] = rpmh_resource("ACV", NULL, NULL);+ bw.ddr_data[0][0] = bw.ddr_data[0][1] = bw.ddr_data[0][2] = 0x40000000;+ bw.cnoc_commands = 1;+ bw.cnoc_wait = 1;+ bw.cnoc_address[0] = rpmh_resource("CN0", NULL, NULL);+ bw.cnoc_data[0][0] = 0x40000000;+ bw.cnoc_data[1][0] = 0x60000001;+ printf("adreno: %u: RPMh's bus resources: SH0 %#x, MC0 %#x, ACV %#x, CN0 %#x (Linux has 0x50004, 0x50000, 0x50088, "+ "0x5006c)\n", step, bw.ddr_address[0], bw.ddr_address[1], bw.ddr_address[2], bw.cnoc_address[0]);+ if (bw.ddr_address[0] == 0 || bw.ddr_address[1] == 0 || bw.ddr_address[2] == 0 || bw.cnoc_address[0] == 0) {+ return false;+ }++ announce("send the GMU its tables (speeds and supply levels, bus votes), \"start\", and the speed wanted: 315 MHz; "+ "from here on it casts votes in RPMh by itself");+ if (!hfi_send("the table of speeds", HFI_PERF_TABLE, &perf.header, sizeof(perf) / 4) ||+ !hfi_send("the table of bus votes", HFI_BW_TABLE, &bw.header, sizeof(bw) / 4) ||+ !hfi_send("\"start the core firmware\"", HFI_CORE_FW_START, core_start, 2) ||+ !hfi_send("the speed wanted", HFI_GX_BW_PERF_VOTE, vote, 4)) {+ return false;+ }++ announce("ask the GMU to power the 3D core and keep it powered");+ get(&gmu, "sptprac_pwr_clk_status", GMU_SPTPRAC_PWR_CLK_STATUS);+ get(&gpucc, "gx_gdscr", GPUCC_GX_GDSCR);+ put(&gmu, "host2gmu_intr_set", GMU_HOST2GMU_INTR_SET, GMU_CORE_REQUEST);+ for (int i = 0; i < 5000 && !(granted & GMU_CORE_GRANTED); i++) {+ sleep_us(100);+ granted = rd(gmu.base + GMU_GMU2HOST_INTR_INFO);+ }+ core_is_on = true; /* asked for, at least: to be given back */+ put(&gmu, "gmu2host_intr_clr", GMU_GMU2HOST_INTR_CLR, GMU_CORE_GRANTED);+ get(&gmu, "sptprac_pwr_clk_status", GMU_SPTPRAC_PWR_CLK_STATUS);+ get(&gpucc, "gx_gdscr", GPUCC_GX_GDSCR);+ get(&gmu, "rpmh_power_state", GMU_RPMH_POWER_STATE);+ if (!(granted & GMU_CORE_GRANTED) || (rd(gmu.base + GMU_SPTPRAC_PWR_CLK_STATUS) & 0xc0) != 0) {+ printf("adreno: %u: the GMU does not power the 3D core (gmu2host_intr_info %#x, fw_init_result %#x)\n", step,+ granted, rd(gmu.base + GMU_CM3_FW_INIT_RESULT));+ space_faulted(&gmu_space);+ return false;+ }+ if (!hfi_send("\"start\"", HFI_START, start, 1)) {+ return false;+ }+ printf("adreno: %u: measured: the 3D core's clock runs at %u Hz (315 MHz was asked for), the GMU's in it at %u Hz\n", step,+ measure_quietly(GPUCC_DEBUG_GX_GFX3D), measure_quietly(GPUCC_DEBUG_GX_GMU));++ announce("read registers of the 3D core itself (0x3d00000): the first time it has power");+ get(&gpu, "rbbm_status", RBBM_STATUS);+ get(&gpu, "rbbm_int_0_status", RBBM_INT_0_STATUS);+ get(&gpu, "rbbm_clock_cntl", RBBM_CLOCK_CNTL);+ get(&gpu, "cp_rb_base", CP_RB_BASE);+ printf("adreno: %u: THE 3D CORE HAS POWER AND ANSWERS\n", step);+ return !space_faulted(&gmu_space);+}++/* ---- step 18: the 3D core's memory and its microcode -------------------------------------------------------- */++/*+ * The 3D core takes its orders as words in a ring of memory, which a small processor inside it+ * ("SQE", with microcode of its own: a660_sqe.fw) reads up to where a register says we have+ * written. Like the GMU it reaches memory through the SMMU: stream 0, to which this gives+ * context 0, with tables of ours that hold the ring, the microcode and later what is drawn into.+ * What is written to the core's registers before its processor is started, and in what order,+ * is Linux's a6xx_gpu.c hw_init for this GPU, less what only matters once it draws (clock gating,+ * the protection of registers against user programs' commands, counters).+ */+#define CORE_STREAM 0+#define CORE_MATCH 0+#define PTE_CORE (PTE_GMU | PTE_UNPRIVILEGED)+#define PTE_READ_ONLY (1ull << 7) /* Linux gives the core its microcode so (MSM_BO_GPU_READONLY) */+#define CORE_RING_ADDRESS 0x01000000u+#define CORE_RING_SIZE 0x8000u /* Linux's; 8192 words */+#define CORE_SQE_ADDRESS 0x01100000u+#define CORE_SCRATCH_ADDRESS 0x01200000u+#define CORE_TARGET_ADDRESS 0x10000000u+#define CORE_NOWHERE 0x7f000000u /* never given anything */++#define GBIF_QSB_SIDE0 (4 * 0x03) /* of the bus interface (step 6) */+#define RBBM_GBIF_CLIENT_QOS_CNTL GPU_WORD(0x0011)+#define RBBM_GBIF_HALT GPU_WORD(0x0016)+#define RBBM_INTERFACE_HANG_INT_CNTL GPU_WORD(0x001f)+#define RBBM_PERFCTR_CNTL GPU_WORD(0x0500)+#define RBBM_PERFCTR_GPU_BUSY_MASKED GPU_WORD(0x050b)+#define CP_RB_CNTL GPU_WORD(0x0802)+#define CP_RB_RPTR GPU_WORD(0x0806)+#define CP_RB_WPTR GPU_WORD(0x0807)+#define CP_SQE_CNTL GPU_WORD(0x0808)+#define CP_HW_FAULT GPU_WORD(0x0821)+#define CP_INTERRUPT_STATUS GPU_WORD(0x0823)+#define CP_PROTECT_STATUS GPU_WORD(0x0824)+#define CP_SQE_INSTR_BASE GPU_WORD(0x0830)+#define CP_CHICKEN_DBG GPU_WORD(0x0841)+#define CP_ADDR_MODE_CNTL GPU_WORD(0x0842)+#define CP_APRIV_CNTL GPU_WORD(0x0844)+#define CP_ROQ_THRESHOLDS_1 GPU_WORD(0x08c1)+#define CP_ROQ_THRESHOLDS_2 GPU_WORD(0x08c2)+#define CP_MEM_POOL_SIZE GPU_WORD(0x08c3)+#define CP_AHB_CNTL GPU_WORD(0x098d)+#define CP_LPAC_PROG_FIFO_SIZE GPU_WORD(0x0b34)+#define UCHE_WRITE_RANGE_MAX GPU_WORD(0x0e05)+#define UCHE_WRITE_THRU_BASE GPU_WORD(0x0e07)+#define UCHE_TRAP_BASE GPU_WORD(0x0e09)+#define UCHE_CACHE_WAYS GPU_WORD(0x0e17)+#define UCHE_FILTER_CNTL GPU_WORD(0x0e18)+#define UCHE_CLIENT_PF GPU_WORD(0x0e19)+#define RB_DBG_ECO_CNTL GPU_WORD(0x8e04)+#define PC_DBG_ECO_CNTL GPU_WORD(0x9e00)+#define RBBM_SECVID_TSB_TRUSTED_BASE GPU_WORD(0xf800)+#define RBBM_SECVID_TSB_TRUSTED_SIZE GPU_WORD(0xf802)+#define RBBM_SECVID_TSB_CNTL GPU_WORD(0xf803)+#define RBBM_SECVID_TSB_ADDR_MODE_CNTL GPU_WORD(0xf810)+#define CP_RB_CNTL_VALUE (12u | 2u << 8 | 1u << 27) /* 2^12 pairs of words; do not copy the read position to memory */++/* commands (Mesa's adreno_pm4.xml); a packet is a header word and so many more */+#define CP_NOP 0x10+#define CP_WAIT_FOR_IDLE 0x26+#define CP_BLIT 0x2c+#define CP_MEM_WRITE 0x3d+#define CP_EVENT_WRITE 0x46+#define CP_ME_INIT 0x48+#define CP_SET_MARKER 0x65+#define CP_SET_SECURE_MODE 0x66++/*+ * Linux (qcom_adreno_smmu_write_sctlr): after a fault the core's other requests are served, or it+ * hangs. And the core has a second stream, 1 (dt: gfx3d_lpac; Linux's sc7280.dtsi gives the GPU+ * both): once started, the microcode starts a second command processor ("LPAC"), which fetches+ * its own part of the microcode as that stream at once -- seen on the phone as a request for+ * 0x7e60 into the microcode, refused by an empty context given to stream 1 (adreno_stream1=2).+ * With nothing at all set up for stream 1 the phone resets without a word the moment the+ * command processor is started (adreno_stream1=0); a start with the microcode nowhere+ * (adreno_blind=1) does not get that far.+ */+#define SMMU_SCTLR_SERVE_AFTER_FAULT (1u << 8)+static struct gpu_space core_space = { .name = "the 3D core", .context = 0, .control = SMMU_SCTLR_SERVE_AFTER_FAULT };+/* adreno_stream1=2: the second stream gets a context of its own with nothing in it, to see what it asks for */+static struct gpu_space second_space = {+ .name = "the 3D core's second stream", .context = 2, .control = SMMU_SCTLR_SERVE_AFTER_FAULT,+};+#define SECOND_MATCH 2+static bool second_space_is_on, second_stream_is_on;+static struct gpu_block core_ring, core_sqe, core_scratch, core_target;+static bool core_space_is_on, core_runs, zap_is_loaded;+static uint32_t ring_words; /* written to the ring so far; it is never filled here */++static uint32_t+odd_parity(uint32_t v)+{+ return 0x9669 >> (0xf & (v ^ v >> 4 ^ v >> 8 ^ v >> 12 ^ v >> 16 ^ v >> 20 ^ v >> 24 ^ v >> 28)) & 1;+}++static void+ring(uint32_t word)+{+ ((uint32_t *)core_ring.memory)[ring_words++] = word;+}++/* The header of a command with so many words after it. */+static void+ring_command(uint32_t command, uint32_t words)+{+ ring(7u << 28 | words | odd_parity(words) << 15 | command << 16 | odd_parity(command) << 23);+}++/* A register of the 3D core (by its word number) written by command, in the order of the other commands. */+static void+ring_register(uint32_t number, uint32_t value)+{+ ring(4u << 28 | 1 | odd_parity(1) << 7 | number << 8 | odd_parity(number) << 27);+ ring(value);+}++/* Let the 3D core have what is in the ring, and wait until it has taken all of it and rests. */+static bool+ring_run(const char *what)+{+ uint32_t taken = ~0u, status = ~0u;++ to_memory(core_ring.memory, ring_words * 4);+ printf("adreno: %u: about to have the 3D core %s: the ring up to word %u\n", step, what, ring_words);+ snprintf(last_write, sizeof(last_write), "step %u had the 3D core %s", step, what);+ hold(true);+ wr(gpu.base + CP_RB_WPTR, ring_words);+ for (int i = 0; i < 20000 && (taken != ring_words || (status & ~1u)); i++) {+ sleep_us(100);+ taken = rd(gpu.base + CP_RB_RPTR);+ status = rd(gpu.base + RBBM_STATUS);+ }+ if (taken == ring_words && !(status & ~1u) && rd(gpu.base + CP_HW_FAULT) == 0 && !space_faulted(&core_space)) {+ printf("adreno: %u: the 3D core has done it (ring read up to %u, status %#x)\n", step, taken, status);+ return true;+ }+ printf("adreno: %u: the 3D core does not get through: ring read up to %u of %u, status %#x, interrupts %#x, "+ "command processor: fault %#x, interrupts %#x, protection %#x\n", step, taken, ring_words, status,+ rd(gpu.base + RBBM_INT_0_STATUS), rd(gpu.base + CP_HW_FAULT), rd(gpu.base + CP_INTERRUPT_STATUS),+ rd(gpu.base + CP_PROTECT_STATUS));+ space_faulted(&core_space);+ return false;+}++static void+put64(struct block *b, const char *name, uint32_t offset, uint64_t value)+{+ put(b, name, offset, (uint32_t)value);+ put(b, name, offset + 4, (uint32_t)(value >> 32));+}++static bool+step_core_space(void)+{+ const uint8_t *microcode;+ uint64_t size = 0;+ uint32_t stream1 = 1;++ if (!cx_ready() || !core_is_on) {+ return false;+ }+ PE_parse_boot_argn("adreno_stream1", &stream1, sizeof(stream1));+ announce("make the 3D core's memory: its ring, its microcode (fw/a660_sqe.fw), a page to try writing to");+ microcode = image_file("fw", "a660_sqe.fw", &size);+ if (microcode == NULL || size < 8 || size > 0x40000 ||+ !block_new(&core_space, &core_ring, CORE_RING_ADDRESS, CORE_RING_SIZE, PTE_CORE) ||+ !block_new(&core_space, &core_sqe, CORE_SQE_ADDRESS, (uint32_t)size - 4, PTE_CORE | PTE_READ_ONLY) ||+ !block_new(&core_space, &core_scratch, CORE_SCRATCH_ADDRESS, 4096, PTE_CORE)) {+ return false;+ }+ memcpy((void *)core_sqe.memory, microcode + 4, size - 4); /* the file's first word is not microcode */+ to_memory(core_sqe.memory, core_sqe.size);++ announce("give the SMMU's context 0 these tables and send the 3D core's requests (stream 0) through it");+ core_space_is_on = true;+ if (!space_on(&core_space, CORE_MATCH, CORE_STREAM)) {+ return false;+ }+ if (stream1 == 2) {+ if ((second_space.root = space_new_table(&second_space)) == NULL) {+ return false;+ }+ second_space_is_on = true;+ if (!space_on(&second_space, SECOND_MATCH, 1)) {+ return false;+ }+ } else if (stream1 == 1) {+ second_stream_is_on = true;+ if (!space_stream(&core_space, SECOND_MATCH, 1)) {+ return false;+ }+ }+ get(&gpu, "rbbm_status", RBBM_STATUS);+ return !space_faulted(&core_space);+}++/* ---- step 19: the zap shader, loaded -------------------------------------------------------------------- */++/*+ * The rest of what Linux does with the zap shader (mdt_loader.c: qcom_mdt_pas_init,+ * qcom_mdt_load; adreno_gpu.c: zap_shader_load_mdt): after the header (step 12), TrustZone is+ * told where the image will be -- the two pages the device tree reserves for it+ * (reserved-memory gpu_micro_code@8b71a000, the zap-shader node's memory-region) --, the image's+ * one segment (the file a660_zap.b02, relocatable: at the start of that memory) is put there,+ * and TrustZone is asked to authenticate it and run it. The vendor's driver does this before it+ * starts the 3D core's command processor (a6xx_rb_start: a6xx_microcode_load), Linux after;+ * this phone's own order is kept. It is done once in the life of a started phone: afterwards+ * the memory is TrustZone's, and nothing here can see whether that is so. Our kernels come and+ * go without the phone restarting: boot-arg adreno_zap=0 says an earlier run has done it.+ */+#define ZAP_MEMORY 0x8b71a000ull+#define ZAP_MEMORY_SIZE 0x2000u+#define SCM_PAS_MEM_SETUP SCM_CALL(2, 2)+#define SCM_THREE_VALUES 3++static bool+step_zap_load(void)+{+ const uint8_t *segment;+ volatile uint8_t *memory;+ uint64_t size = 0, header = 0, answer = ~0ull;+ uint32_t load = 1;++ if (!cx_ready() || !core_is_on) {+ return false;+ }+ PE_parse_boot_argn("adreno_zap", &load, sizeof(load));+ if (!load) {+ printf("adreno: %u: told that the zap shader is loaded already (adreno_zap=0)\n", step);+ zap_is_loaded = true;+ return true;+ }+ announce("find the zap shader's code, fw/a660_zap.b02, and look at the memory reserved for it (0x8b71a000: memory "+ "outside what this kernel was given; read only)");+ segment = image_file("fw", "a660_zap.b02", &size);+ if (segment == NULL || size == 0 || size > 0x1000) {+ return false;+ }+ memory = (volatile uint8_t *)ml_io_map_wcomb(ZAP_MEMORY, ZAP_MEMORY_SIZE);+ printf("adreno: %u: the reserved memory begins %02x %02x %02x %02x, the zap shader %02x %02x %02x %02x\n", step,+ memory[0], memory[1], memory[2], memory[3], segment[0], segment[1], segment[2], segment[3]);++ announce("hand the zap shader's header to TrustZone, tell it where the image goes, put the image there (the first "+ "write to that memory), and ask TrustZone to authenticate and run it");+ if (!zap_header(&header)) {+ return false;+ }+ if (scm("to take the zap shader's signature", SCM_64 | SCM_SIP | SCM_PAS_INIT_IMAGE, SCM_VALUE_AND_BUFFER, PAS_GPU,+ header, 0, &answer) != 0 || answer != 0) {+ printf("adreno: %u: TrustZone does not take the signature (%#llx)\n", step, answer);+ return false;+ }+ if (scm("where the image will be", SCM_64 | SCM_SIP | SCM_PAS_MEM_SETUP, SCM_THREE_VALUES, PAS_GPU, ZAP_MEMORY, 0x1000,+ &answer) != 0 || answer != 0) {+ printf("adreno: %u: TrustZone does not take the place (%#llx)\n", step, answer);+ return false;+ }+ printf("adreno: %u: about to write %llu bytes at %#llx\n", step, size, ZAP_MEMORY);+ snprintf(last_write, sizeof(last_write), "step %u wrote the zap shader into memory at %#llx", step, ZAP_MEMORY);+ hold(true);+ for (uint64_t i = 0; i < 0x1000; i++) {+ memory[i] = i < size ? segment[i] : 0;+ }+ settle();+ if (scm("to authenticate the zap shader and run it", SCM_64 | SCM_SIP | SCM_PAS_AUTH_AND_RESET, SCM_ONE_VALUE, PAS_GPU, 0,+ 0, &answer) != 0 || answer != 0) {+ printf("adreno: %u: TrustZone does not authenticate the zap shader (%#llx)\n", step, answer);+ return false;+ }+ printf("adreno: %u: TRUSTZONE HAS AUTHENTICATED AND STARTED THE ZAP SHADER\n", step);+ zap_is_loaded = true;+ return true;+}++/* ---- step 20: the 3D core's registers, its command processor started, its first command ------------------- */++static bool+step_core_start(void)+{+ static const uint32_t address_modes[] = { 0x0842, 0x0c01, 0x8601, 0x8e05, 0x9e01, 0xbe05, 0xa601, 0x9601, 0x0e00, 0xae01,+ 0xb601, 0xf810 };+ uint32_t blind = 0;++ if (!cx_ready() || !core_is_on || !core_space_is_on) {+ return false;+ }+ announce("write the 3D core's registers as Linux does before it starts the command processor: the first writes "+ "to the core itself");+ put(&gbif, "halt", GBIF_HALT, 0);+ put(&gpu, "rbbm_gbif_halt", RBBM_GBIF_HALT, 0);+ put(&gpu, "rbbm_secvid_tsb_cntl", RBBM_SECVID_TSB_CNTL, 0);+ put64(&gpu, "rbbm_secvid_tsb_trusted_base", RBBM_SECVID_TSB_TRUSTED_BASE, 0);+ put(&gpu, "rbbm_secvid_tsb_trusted_size", RBBM_SECVID_TSB_TRUSTED_SIZE, 0);+ for (unsigned i = 0; i < sizeof(address_modes) / sizeof(address_modes[0]); i++) {+ put(&gpu, "address mode: 64 bits", GPU_WORD(address_modes[i]), 1);+ }+ for (unsigned i = 0; i < 4; i++) {+ put(&gbif, "qsb_side", GBIF_QSB_SIDE0 + 4 * i, 0x00071620);+ }+ put(&gpu, "rbbm_gbif_client_qos_cntl", RBBM_GBIF_CLIENT_QOS_CNTL, 3);+ put(&gpu, "rbbm_perfctr_gpu_busy_masked", RBBM_PERFCTR_GPU_BUSY_MASKED, 0xffffffff);+ put64(&gpu, "uche_write_range_max", UCHE_WRITE_RANGE_MAX, 0x0001ffffffffffc0ull);+ put64(&gpu, "uche_trap_base", UCHE_TRAP_BASE, 0x0001fffffffff000ull);+ put64(&gpu, "uche_write_thru_base", UCHE_WRITE_THRU_BASE, 0x0001fffffffff000ull);+ put(&gpu, "uche_filter_cntl", UCHE_FILTER_CNTL, 0x804);+ put(&gpu, "uche_cache_ways", UCHE_CACHE_WAYS, 0x4);+ put(&gpu, "cp_roq_thresholds_2", CP_ROQ_THRESHOLDS_2, 0x02000140);+ put(&gpu, "cp_roq_thresholds_1", CP_ROQ_THRESHOLDS_1, 0x8040362c);+ put(&gpu, "cp_lpac_prog_fifo_size", CP_LPAC_PROG_FIFO_SIZE, 0x20);+ put(&gpu, "cp_mem_pool_size", CP_MEM_POOL_SIZE, 128);+ put(&gpu, "pc_dbg_eco_cntl", PC_DBG_ECO_CNTL, 0x00200200);+ put(&gpu, "cp_ahb_cntl", CP_AHB_CNTL, 1);+ put(&gpu, "rbbm_perfctr_cntl", RBBM_PERFCTR_CNTL, 1);+ put(&gpu, "rbbm_interface_hang_int_cntl", RBBM_INTERFACE_HANG_INT_CNTL, 1u << 30 | 0x1fffff);+ put(&gpu, "uche_client_pf", UCHE_CLIENT_PF, 1u << 7 | 1);+ put(&gpu, "cp_chicken_dbg", CP_CHICKEN_DBG, 1);+ put(&gpu, "rbbm_gbif_client_qos_cntl", RBBM_GBIF_CLIENT_QOS_CNTL, 0);+ put(&gpu, "cp_apriv_cntl", CP_APRIV_CNTL, 1u << 6 | 1u << 5 | 1u << 3 | 1u << 2 | 1u << 1);+ put(&gpu, "rb_dbg_eco_cntl", RB_DBG_ECO_CNTL, 0x04100000); /* Mesa's value for this GPU */+ /*+ * adreno_blind=1: the command processor is started with its microcode at an address where+ * nothing is. All it can do then is ask the SMMU for it, and the SMMU's record of the refusal+ * shows that its requests come through context 0 as meant, and what they look like.+ */+ PE_parse_boot_argn("adreno_blind", &blind, sizeof(blind));+ put64(&gpu, "cp_sqe_instr_base", CP_SQE_INSTR_BASE, blind ? CORE_NOWHERE : CORE_SQE_ADDRESS);+ put64(&gpu, "cp_rb_base", CP_RB_BASE, CORE_RING_ADDRESS);+ put(&gpu, "cp_rb_cntl", CP_RB_CNTL, CP_RB_CNTL_VALUE);+ get(&gpu, "cp_sqe_instr_base", CP_SQE_INSTR_BASE);+ get(&gpu, "cp_rb_base", CP_RB_BASE);+ get(&gpu, "cp_rb_cntl", CP_RB_CNTL);+ get(&gpu, "cp_apriv_cntl", CP_APRIV_CNTL);+ get(&gpu, "rbbm_status", RBBM_STATUS);++ announce(blind ? "start the 3D core's command processor with its microcode nowhere" :+ "start the 3D core's command processor");+ core_runs = true;+ put(&gpu, "cp_sqe_cntl", CP_SQE_CNTL, 1);+ sleep_ms(20);+ printf("adreno: %u: alive after it; status %#x, interrupts %#x, command processor: fault %#x, interrupts %#x, "+ "protection %#x, ring read up to %u\n", step, rd(gpu.base + RBBM_STATUS), rd(gpu.base + RBBM_INT_0_STATUS),+ rd(gpu.base + CP_HW_FAULT), rd(gpu.base + CP_INTERRUPT_STATUS), rd(gpu.base + CP_PROTECT_STATUS),+ rd(gpu.base + CP_RB_RPTR));+ get(&smmu, "sgfsr", SMMU_SGFSR);+ get(&smmu, "sgfsynr0", SMMU_SGFSR + 8);+ get(&smmu, "sgfsynr1", SMMU_SGFSR + 12);+ if (blind) {+ return space_faulted(&core_space);+ }+ if (second_space_is_on) {+ sleep_ms(200);+ return space_faulted(&second_space) | !space_faulted(&core_space);+ }+ if (space_faulted(&core_space)) {+ return false;+ }++ announce("give the 3D core its first command, \"initialize\"");+ ring_command(CP_ME_INIT, 8); /* a6xx_cp_init */+ ring(0x0000002f);+ ring(0x00000003); /* several hardware contexts */+ ring(0x20000000); /* error detection */+ ring(0);+ ring(0);+ ring(0);+ ring(0);+ ring(0);+ if (!ring_run("initialize its command processor")) {+ return false;+ }+ printf("adreno: %u: THE 3D CORE TAKES COMMANDS\n", step);+ return true;+}++/* ---- step 21: out of secure mode; a word written on command ------------------------------------------------ */++static bool+step_core_commands(void)+{+ volatile uint32_t *scratch = (volatile uint32_t *)core_scratch.memory;++ if (!cx_ready() || !core_runs) {+ return false;+ }+ announce("command the 3D core out of secure mode (it asks the zap shader), then to write a word into a page of ours");+ ring_command(CP_SET_SECURE_MODE, 1);+ ring(0);+ if (!ring_run("leave secure mode")) {+ return false;+ }+ ring_command(CP_MEM_WRITE, 3);+ ring(CORE_SCRATCH_ADDRESS);+ ring(0);+ ring(0x600dc0de);+ if (!ring_run("write a word to memory")) {+ return false;+ }+ to_memory(core_scratch.memory, 64);+ printf("adreno: %u: the page reads %#x: %s\n", step, scratch[0], scratch[0] == 0x600dc0de ?+ "THE 3D CORE WRITES OUR MEMORY ON COMMAND" : "not what it was told to write");+ return scratch[0] == 0x600dc0de;+}++/* ---- step 22: a frame filled by the GPU, on the panel ------------------------------------------------------ */++/*+ * The simplest thing the 3D core draws: its 2D engine filling a rectangle with one colour (Mesa's+ * fd6_blitter.cc: fd6_clear_surface; the registers under their older names, as in Linux's+ * a6xx.xml), three times for three bands. The picture is 32-bit pixels in rows of a multiple of+ * 64 bytes, in memory of ours; the framebuffer driver copies it to the panel.+ */+#define RB_CCU_CNTL 0x8e07 /* register numbers: these are written by command */+#define GRAS_2D_BLIT_CNTL 0x8400+#define GRAS_2D_DST_TL 0x8405+#define GRAS_2D_DST_BR 0x8406+#define RB_2D_BLIT_CNTL 0x8c00+#define RB_2D_PIXEL_CNTL 0x8c01+#define RB_2D_DST_INFO 0x8c17+#define RB_2D_DST 0x8c18+#define RB_2D_DST_PITCH 0x8c1a+#define RB_2D_SRC_SOLID_C0 0x8c2c+#define SP_2D_DST_FORMAT 0xacc0+#define FORMAT_8888 0x30+#define BLIT_SOLID_8888 (1u << 7 | FORMAT_8888 << 8 | 0xfu << 20 | 0x10u << 24) /* one colour; all channels; 8-bit */+#define DST_BGRA_LINEAR (FORMAT_8888 | 1u << 10) /* rows of pixels, blue first in memory */+#define EVENT_FLUSH_COLOR 0x1d+#define EVENT_CACHE_FLUSH 0x04+#define TARGET_WIDTH 1080+#define TARGET_HEIGHT 2400+#define TARGET_ROW 4352 /* 1088 pixels: the next multiple of 64 bytes */++extern void MiniIOMobileFramebufferShow(const uint8_t *pixels, uint32_t width, uint32_t height, uint32_t row_bytes);++static void+ring_fill(uint32_t top, uint32_t bottom, uint32_t red, uint32_t green, uint32_t blue)+{+ ring_register(GRAS_2D_DST_TL, top << 16);+ ring_register(GRAS_2D_DST_BR, (TARGET_WIDTH - 1) | (bottom - 1) << 16);+ ring_register(RB_2D_SRC_SOLID_C0, red); /* a register for each part of the colour, 0 to 255 */+ ring_register(RB_2D_SRC_SOLID_C0 + 1, green);+ ring_register(RB_2D_SRC_SOLID_C0 + 2, blue);+ ring_register(RB_2D_SRC_SOLID_C0 + 3, 255);+ ring_register(RB_2D_BLIT_CNTL, BLIT_SOLID_8888);+ ring_register(GRAS_2D_BLIT_CNTL, BLIT_SOLID_8888);+ ring_register(SP_2D_DST_FORMAT, 1 | FORMAT_8888 << 3 | 0xfu << 12);+ ring_register(RB_2D_PIXEL_CNTL, 0);+ ring_register(RB_2D_DST_INFO, DST_BGRA_LINEAR);+ ring_register(RB_2D_DST, CORE_TARGET_ADDRESS);+ ring_register(RB_2D_DST + 1, 0);+ ring_register(RB_2D_DST_PITCH, TARGET_ROW >> 6);+ ring_command(CP_WAIT_FOR_IDLE, 0);+ ring_command(CP_BLIT, 1);+ ring(3); /* "scale": the 2D engine's one operation */+}++static bool+step_frame(void)+{+ const volatile uint8_t *pixel;++ if (!cx_ready() || !core_runs) {+ return false;+ }+ announce("make a picture's worth of memory for the 3D core (10 MB) and have it fill three bands of colour");+ if (!block_new(&core_space, &core_target, CORE_TARGET_ADDRESS, TARGET_ROW * TARGET_HEIGHT, PTE_CORE)) {+ return false;+ }+ /* tables that the SMMU has in use have grown: it has not seen these addresses, so nothing stale to forget */+ for (uint32_t i = 0; i < core_space.table_count; i++) {+ to_memory((vm_offset_t)core_space.tables[i].table, 4096);+ }+ ring_command(CP_SET_MARKER, 1);+ ring(12); /* "2D blits follow" */+ ring_register(RB_CCU_CNTL, 0x10000004); /* Mesa's for this GPU when it draws straight into memory */+ ring_fill(0, 800, 255, 128, 0); /* orange */+ ring_fill(800, 1600, 255, 255, 255);+ ring_fill(1600, TARGET_HEIGHT, 0, 128, 255);+ /*+ * What it has drawn out of its caches into memory. Each of the two events is followed by an+ * address and a word that the core writes there when it is done (Linux's a6xx_submit; Mesa's+ * tu6_emit_event_write: the events named ..._TS): the core takes those three words whether+ * they are there or not.+ */+ ring_command(CP_EVENT_WRITE, 4);+ ring(EVENT_FLUSH_COLOR);+ ring(CORE_SCRATCH_ADDRESS + 32);+ ring(0);+ ring(0x0c0c0c0c);+ ring_command(CP_EVENT_WRITE, 4);+ ring(EVENT_CACHE_FLUSH);+ ring(CORE_SCRATCH_ADDRESS + 16);+ ring(0);+ ring(0x0d0e0d0e);+ ring_command(CP_WAIT_FOR_IDLE, 0);+ if (!ring_run("fill a picture")) {+ return false;+ }+ to_memory(core_scratch.memory, 64);+ printf("adreno: %u: the words it writes when its caches are in memory: %#x, %#x (told: 0xc0c0c0c, 0xd0e0d0e)\n", step,+ ((volatile uint32_t *)core_scratch.memory)[8], ((volatile uint32_t *)core_scratch.memory)[4]);+ to_memory(core_target.memory, core_target.size);+ pixel = (const volatile uint8_t *)core_target.memory;+ printf("adreno: %u: the picture's first pixel is %02x %02x %02x %02x (blue green red alpha; told: 00 80 ff ff), one in "+ "the middle %02x %02x %02x %02x, one in the last row %02x %02x %02x %02x\n", step, pixel[0], pixel[1], pixel[2],+ pixel[3], pixel[1200 * TARGET_ROW + 2160], pixel[1200 * TARGET_ROW + 2161], pixel[1200 * TARGET_ROW + 2162],+ pixel[1200 * TARGET_ROW + 2163], pixel[2399 * TARGET_ROW + 4316], pixel[2399 * TARGET_ROW + 4317],+ pixel[2399 * TARGET_ROW + 4318], pixel[2399 * TARGET_ROW + 4319]);+ if ((pixel[0] | pixel[1] | pixel[2]) == 0 || pixel[1200 * TARGET_ROW + 2160] == 0 || (pixel[2399 * TARGET_ROW + 4316] |+ pixel[2399 * TARGET_ROW + 4317] | pixel[2399 * TARGET_ROW + 4318]) == 0) {+ printf("adreno: %u: the picture is not what the 3D core was told to draw\n", step);+ return false;+ }+ MiniIOMobileFramebufferShow((const uint8_t *)core_target.memory, TARGET_WIDTH, TARGET_HEIGHT, TARGET_ROW);+ printf("adreno: %u: A PICTURE DRAWN BY THE GPU IS ON THE PANEL: orange, white, blue from the top\n", step);+ return true;+}++/* ---- the way back ------------------------------------------------------------------------------------- */++/*+ * Nothing of the GPU block may outlive the kernel that started it: the memory its SMMU context+ * points into is the next kernel's to use. In the order of Linux's a6xx_gmu_shutdown: the 3D+ * core given back, the GMU told to drop its votes ("slumber") and stopped, its RPMh channel put+ * to sleep; then the processor into reset and its stream sent nowhere.+ */+static void+gpu_block_stop(void)+{+ uint32_t slumber[3] = { 0, 0, 0 };+ bool power_off = false;++ step = 0;+ newest_step = ~0u;+ if (core_is_on) {+ put(&gmu, "host2gmu_intr_set", GMU_HOST2GMU_INTR_SET, 1u << 31);+ core_is_on = false;+ }+ if (gmu_talks) {+ hfi_send("\"slumber\"", HFI_PREPARE_SLUMBER, slumber, 3);+ if (!wait_for(&gmu, GMU_AO_GPU_CX_BUSY_STATUS, 1u << 23, 0, 100)) {+ get(&gmu, "ao_gpu_cx_busy_status", GMU_AO_GPU_CX_BUSY_STATUS);+ }+ get(&gmu, "sptprac_pwr_clk_status", GMU_SPTPRAC_PWR_CLK_STATUS);+ get(&gpucc, "gx_gdscr", GPUCC_GX_GDSCR);+ gmu_talks = false;+ }+ if (gmu_runs) {+ put(&gmu, "cm3_sysreset", GMU_CM3_SYSRESET, 1);+ gmu_runs = false;+ }+ /* vendor's a6xx_rscc_sleep_sequence: the GMU's votes in RPMh are dropped, by the always-on sequencer */+ if (rsc_awake && rd(rscc.base + RSCC_SEQ_MEM_0_DRV0) == 0xeaaae5a0) {+ put(&gmu, "rscc_control_req", GMU_RSCC_CONTROL_REQ, 1);+ if (!wait_for(&rscc, RSCC_RSC_STATUS0_DRV0, RSCC_ASLEEP, RSCC_ASLEEP, 100)) {+ printf("adreno: %u: the GMU's RPMh channel does not go to sleep\n", step);+ }+ get(&rscc, "rsc_status0_drv0", RSCC_RSC_STATUS0_DRV0);+ put(&gmu, "rscc_control_req", GMU_RSCC_CONTROL_REQ, 0);+ }+ rsc_awake = false;+ if (gmu_space_is_on) {+ space_faulted(&gmu_space);+ space_off(&gmu_space, GMU_MATCH);+ gmu_space_is_on = false;+ power_off = true;+ }+ if (second_space_is_on) {+ space_faulted(&second_space);+ space_off(&second_space, SECOND_MATCH);+ second_space_is_on = false;+ }+ if (second_stream_is_on) {+ put(&smmu, "smr", SMMU_SMR(SECOND_MATCH), 0);+ put(&smmu, "s2cr", SMMU_S2CR(SECOND_MATCH), SMMU_S2CR_FAULT);+ second_stream_is_on = false;+ }+ if (core_space_is_on) {+ space_faulted(&core_space);+ space_off(&core_space, CORE_MATCH);+ core_space_is_on = core_runs = false;+ }+ /*+ * And the CX half off again, as it was when the phone started: the GMU with all it remembers+ * and the SMMU lose their power, and the next time starts from nothing (vendor's+ * a6xx_gmu_suspend). The switch takes votes; ours is all this withdraws.+ */+ if (power_off && gpucc_answers()) {+ put(&gpucc, "cx_gdscr", GPUCC_CX_GDSCR, rd(gpucc.base + GPUCC_CX_GDSCR) | GDSC_SW_COLLAPSE);+ for (int i = 0; i < 1000 && cx_is_on(); i++) {+ sleep_us(100);+ }+ get(&gpucc, "cx_gdscr", GPUCC_CX_GDSCR);+ get(&gpucc, "cx_hw_ctrl", GPUCC_CX_HW_CTRL);+ cx_off = !cx_is_on();+ }+}++/*+ * The same when there is no time for asking: the kernel is on its way back to the loader (the+ * end of a run, a panic). Only the two writes that keep the GMU out of memory that is no longer+ * its to use.+ */+void+redwood_adreno_quiesce(void)+{+ if (cx_off) {+ return;+ }+ if (gmu_runs) {+ wr(gmu.base + GMU_CM3_SYSRESET, 1);+ gmu_runs = false;+ }+ if (gmu_space_is_on) {+ wr(smmu.base + smmu_context(&gmu_space) + SMMU_CB_SCTLR, 0);+ gmu_space_is_on = false;+ }+ if (core_space_is_on) {+ wr(smmu.base + smmu_context(&core_space) + SMMU_CB_SCTLR, 0);+ core_space_is_on = false;+ }+ if (second_space_is_on) {+ wr(smmu.base + smmu_context(&second_space) + SMMU_CB_SCTLR, 0);+ second_space_is_on = false;+ }+}++/* ---- all of it together ---------------------------------------------------------------------------- */++static bool(*const steps[])(void) = {+ step_outside, step_fuses, step_gpucc, step_cx_power, step_clocks, step_first_words, step_rscc, step_smmu,+ step_smmu_context, step_gmu_memory, step_trustzone, step_zap_signature, step_gmu_space, step_gmu_firmware,+ step_gmu_start, step_gmu_queues, step_core_power, step_core_space, step_zap_load, step_core_start, step_core_commands,+ step_frame,+};+#define STEPS ((uint32_t)(sizeof(steps) / sizeof(steps[0])))++/*+ * Called while the system starts, before there is a root file system, and the start waits: for+ * as long as the steps and the watch after them last, nothing else writes to the log, so that+ * its end -- all that the loader's notes keep -- is about what is being done here.+ */+void+redwood_adreno_init(void)+{+ uint32_t only = 0, done = 0;++ PE_parse_boot_argn("adreno", &last_step, sizeof(last_step));+ PE_parse_boot_argn("adreno_skip", &skip_steps, sizeof(skip_steps));+ if (PE_parse_boot_argn("adreno_only", &only, sizeof(only)) && only > 0 && only <= STEPS) {+ last_step = only;+ skip_steps = ~(1u << only);+ }+ if (last_step == 0) {+ return;+ }+ if (last_step > STEPS) {+ last_step = STEPS;+ }+ PE_parse_boot_argn("adreno_pause", &pause_ms, sizeof(pause_ms));+ PE_parse_boot_argn("adreno_fine", &fine, sizeof(fine));+ PE_parse_boot_argn("adreno_watch", &watch_seconds, sizeof(watch_seconds));+ for (newest_step = last_step; newest_step > 0 && (skip_steps & 1u << newest_step); newest_step--) {+ }++ printf("adreno: steps up to %u, without those in mask %#x; step %u is the new one and waits %u ms %s\n", last_step,+ skip_steps, newest_step, pause_ms, fine ? "before every access" : "after each announcement");+ for (step = 1; step <= last_step; step++) {+ if (skip_steps & 1u << step) {+ continue;+ }+ if (!steps[step - 1]()) {+ printf("adreno: step %u: FAILED; the steps after it are not attempted\n", step);+ break;+ }+ printf("adreno: step %u: done\n", step);+ done = step;+ }+ /*+ * Punishment for a write may come late (a reset some 17 seconds after it has been seen):+ * keep the end of the log saying where things stand.+ */+ for (uint32_t second = 1; second <= watch_seconds; second++) {+ sleep_ms(1000);+ printf("adreno: alive %u s after step %u; %s\n", second, done, last_write);+ }+ gpu_block_stop();+ if (PE_parse_boot_argn("adreno_return", &only, sizeof(only)) && only) {+ printf("adreno: over; back to the loader (adreno_return)\n");+ loader_return();}printf("adreno: over; the system goes on starting\n");}diff --git a/xnu/files/bsd/sys/redwood_input.h b/xnu/files/bsd/sys/redwood_input.hindex b865560..5855949 100644--- a/xnu/files/bsd/sys/redwood_input.h+++ b/xnu/files/bsd/sys/redwood_input.h@@ -43,6 +43,7 @@ void redwood_virtio_input_init(void); /* boot-arg virtio_input=1: QEMU's virtivoid redwood_goodix_init(void); /* boot-arg touchscreen=1: the phone's touch screen */void redwood_cpufreq_init(void); /* boot-arg cpufreq=1: the phone's processor at full speed */void redwood_adreno_init(void); /* boot-arg adreno=N: the first N steps of waking the phone's GPU */+void redwood_adreno_quiesce(void); /* on the way back to the loader: the GPU block stopped, out of our memory */#endif#endif /* _SYS_REDWOOD_INPUT_H_ */diff --git a/xnu/files/iokit/Kernel/MiniIOMobileFramebuffer.cpp b/xnu/files/iokit/Kernel/MiniIOMobileFramebuffer.cppindex 2876605..e829074 100644--- a/xnu/files/iokit/Kernel/MiniIOMobileFramebuffer.cpp+++ b/xnu/files/iokit/Kernel/MiniIOMobileFramebuffer.cpp@@ -889,6 +889,25 @@ publishAccelerator(IOService * provider)OSSafeReleaseNULL(accelerator);}+/*+ * A picture from inside the kernel (the GPU driver's first ones): rows of 32-bit pixels, put at+ * the top left corner of the panel as they are and cut to its size.+ */+extern "C" void+MiniIOMobileFramebufferShow(const uint8_t * pixels, uint32_t width, uint32_t height, uint32_t rowBytes)+{+ uint32_t columns = width < vinfo.v_width ? width : (uint32_t) vinfo.v_width;+ uint32_t rows = height < vinfo.v_height ? height : (uint32_t) vinfo.v_height;++ if (vinfo.v_baseaddr == 0 || vinfo.v_depth != 32) {+ return;+ }+ for (uint32_t y = 0; y < rows; y++) {+ memcpy((void *) (vinfo.v_baseaddr + (size_t) y * vinfo.v_rowbytes), pixels + (size_t) y * rowBytes,+ (size_t) columns * 4);+ }+}+voidMiniIOMobileFramebufferStart(IOService * provider){diff --git a/xnu/files/osfmk/arm64/loader_return.c b/xnu/files/osfmk/arm64/loader_return.cindex ed871e7..a2810f2 100644--- a/xnu/files/osfmk/arm64/loader_return.c+++ b/xnu/files/osfmk/arm64/loader_return.c@@ -43,6 +43,7 @@ extern void gic_v3_irq_quiesce(void);void loader_log_putc(char c);void loader_return(void);+extern void redwood_adreno_quiesce(void);void loader_checkpoint(uint64_t n);/* Also referenced from the early exception vector in start.s. */@@ -299,6 +300,7 @@ loader_return(void)*/psci_smp_stop_others();#endif+ redwood_adreno_quiesce(); /* the GPU block, if it was started, has tables in memory that is about to be reused *//** A panic's text is first collected in the panic buffer and only later* flushed to the console; if the console itself is what failed, that