Skip to content

Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.

C

GPU: the Adreno comes up and fills a frame that reaches the panel

vcvkkf0282871daa2
diff --git a/README.md b/README.md
index 445ec6e..d3afede 100644
--- a/README.md
+++ b/README.md
@@ -22,7 +22,7 @@
|---|---|
| Экран блокировки, разблокировка свайпом, домашний экран с полным набором значков (две страницы) | Звук, сеть (Wi-Fi/сотовая), Bluetooth, камера |
| Системные жесты: свайп снизу — «домой» и переключатель приложений, свайп от левого края — «назад», Пункт управления | Обои (чёрный фон) |
-| Запуск приложений: Настройки (вглубь), Часы, Календарь, Фото, Заметки и др. | GPU/Metal — только программный рендер |
+| Запуск приложений: Настройки (вглубь), Часы, Календарь, Фото, Заметки и др. | GPU для iOS: пока только программный рендер (сам Adreno уже заведён и заливает кадр, см. ниже) |
| Сенсорный экран телефона (Goodix GT9916), мультитач, пробуждение касанием | Без `cpus=N` работает одно ядро процессора |
| Несколько ядер процессора (`cpus=4`: четыре больших ядра телефона) | |
| Кнопки «Домой»/«Блокировка»/громкость (в QEMU — клавишами) | Зарядка под нашим ядром (телефон только разряжается) |
@@ -152,6 +152,16 @@ XL_SECS=900 XL_CMDLINE="touchscreen=1 cpufreq=1 bootcpu=7 cpus=4" ./xl run #
что ядра 1–3 малые, и сажает на них интерфейс. Паника «Ticket spinlock … timeout» значила бы, что ядрам тесно
во времени (в QEMU загрузчик сам увеличивает эти сроки в 20 раз): помогает `ml-timeout-global-scale=4`.
+## GPU (в работе)
+
+Adreno 642L телефона заведён под нашим ядром: загружается прошивка управляющего процессора (GMU), включается
+3D-ядро, TrustZone принимает zap-шейдер, ядро GPU исполняет команды и заливает кадр, который драйвер экрана выводит
+на панель (`docs/gpu-first-frame.png`). Это опыт за параметром `adreno=N` (шаги описаны в начале
+`bsd/dev/redwood_adreno.c`), iOS пока по-прежнему рисует процессором. Прошивки GPU берутся с самого телефона
+(`adb pull /vendor/firmware/a660_{gmu.bin,sqe.fw,zap.mdt,zap.b00,zap.b01,zap.b02}` в каталог `fw-gpu/`, в git они
+не попадают) и едут в малом образе. Дальше — настоящая 3D-отрисовка и OpenGL ES для CoreAnimation: в QuartzCore
+16.5 есть рабочий рендер через OpenGL ES, на который сервер окон переходит сам, если нет Metal.
+
## Сборка из исходников
Готовые бинарники в `prebuilt/` собраны из этого репозитория; пересобрать всё можно самому.
diff --git a/docs/gpu-first-frame.png b/docs/gpu-first-frame.png
new file mode 100644
index 0000000..adee2eb
Binary files /dev/null and b/docs/gpu-first-frame.png differ
diff --git a/prebuilt/SHA256SUMS b/prebuilt/SHA256SUMS
index 00a79d9..3971cd3 100644
--- a/prebuilt/SHA256SUMS
+++ b/prebuilt/SHA256SUMS
@@ -1,6 +1,6 @@
-Built 2026-10-12 18:12 from this tree.
+Built 2026-10-12 22:05 from this tree.
-0439a8605c0830be4b6c13bb59205b44e18bc3ea77af53bdb19434d8e2cbb375 prebuilt/mach.development.bcm2837
+6a90f6523071ced387d80c6a71918a86f68055ffe7bb72e49e7754a0835344b4 prebuilt/mach.development.bcm2837
1332a598d2dec00a1d8e930044e4f00a12ad44c6e5e50eecf4b7fe7fae803bf2 prebuilt/loader.efi
d2a35a4a1d6658ffb3696914f727661b52c9333de8a3af8a31642b8023ce03ac prebuilt/userland/child
1a413befd511384052e1962c1394adce20f718e1f2657ce63ab09828ebc4bd1b prebuilt/userland/cpu_test
diff --git a/prebuilt/mach.development.bcm2837 b/prebuilt/mach.development.bcm2837
index 7fa6a34..82714d2 100755
Binary files a/prebuilt/mach.development.bcm2837 and b/prebuilt/mach.development.bcm2837 differ
diff --git a/xnu/files/bsd/dev/redwood_adreno.c b/xnu/files/bsd/dev/redwood_adreno.c
index c831ca1..7bd0fe7 100644
--- a/xnu/files/bsd/dev/redwood_adreno.c
+++ b/xnu/files/bsd/dev/redwood_adreno.c
@@ -1,6 +1,7 @@
/*
- * The GPU of the POCO X5 Pro (redwood), an Adreno 642L, as far as it can be woken without its
- * firmware. Only with boot-arg adreno=N: every address below belongs to this phone.
+ * The GPU of the POCO X5 Pro (redwood), an Adreno 642L, woken step by step up to a picture that
+ * it fills and the panel shows. Only with boot-arg adreno=N: every address below belongs to this
+ * phone.
*
* Nothing in this system uses the GPU yet, and how much of it a kernel under Qualcomm's
* hypervisor may reach is not known: an access the hypervisor does not like ends the run, at
@@ -28,13 +29,28 @@
* 12 asks TrustZone to check the signature of the GPU's "zap shader", the signed image
* without which the 3D core stays locked in secure mode: its header goes from the
* small image (fw/a660_zap.mdt) into memory of ours that TrustZone is pointed at
+ * 13 writes page tables for the memory the GMU's firmware asks for, and the SMMU: a
+ * translation context with those tables, and the GMU's requests sent through it
+ * 14 writes the GMU's firmware (fw/a660_gmu.bin) into its own memory and into that memory
+ * 15 writes the GMU's configuration, and lets its processor out of reset: the firmware runs
+ * 16 writes the GMU's connection to RPMh; starts its message queues ("HFI")
+ * 17 talks to the GMU: the tables it needs, and the request to power the 3D core, whose
+ * registers are then read
+ * 18 writes the 3D core's translation context, with its ring and its microcode
+ * (fw/a660_sqe.fw) in it
+ * 19 asks TrustZone to load the zap shader for good (once in the life of a started phone:
+ * adreno_zap=0 in the runs after)
+ * 20 writes the 3D core's registers; starts its command processor and gives it its first
+ * command
+ * 21 commands the 3D core out of secure mode, and to write a word into a page of ours
+ * 22 commands a picture filled with three colours, and shows it on the panel
+ * When the steps are over, what they started is stopped again, in reverse, and the CX half
+ * switched off: nothing of the GPU block outlives the run (its SMMU contexts point into this
+ * kernel's memory), and the next one starts from what the phone's own start leaves.
*
- * What is left after that: a translation context for the GMU, its firmware (a660_gmu.bin) loaded
- * and started, a conversation with it (HFI) that makes it power the 3D core; then the core's
- * microcode (a660_sqe.fw), and the rest of the zap shader's loading. The firmware is the phone's
- * own, from /vendor/firmware of its Android. There is no register that holds the chip ID
- * (0x06030500, "qcom,chipid" in the device tree): the fuses of step 2 are all the hardware says
- * about itself, and the GMU's version is a property of its firmware.
+ * The firmware is the phone's own, from /vendor/firmware of its Android. There is no register that holds the chip ID (0x06030500,
+ * "qcom,chipid" in the device tree): the fuses of step 2 are all the hardware says about
+ * itself, and the GMU's version is a property of its firmware.
*
* More boot-args:
* adreno_only=K step K alone (what steps 6 to 10 need, they check for themselves: within
@@ -45,8 +61,14 @@
* the end of our log to storage every two seconds, and after a silent reset
* that note is all there is.
* adreno_fine=1 the newest step waits like that before every single access
+ * adreno_zap=0 step 19 has been done since the phone started: not again
+ * adreno_blind=1 step 20 starts the command processor with its microcode nowhere, and
+ * adreno_stream1=N step 18 leaves the core's second stream out (0) or gives it an empty
+ * context of its own (2): what the SMMU then records shows what the core
+ * asks for. With 0 and the microcode in place the phone resets.
* adreno_watch=S after the last step, say once a second for S seconds (default 20) that
* the machine is still alive, and what was written last
+ * adreno_return=1 when all is over, go back to the loader instead of starting the system
* The system does not go on starting until all of this is over, so that the end of the log is
* ours; the touch screen driver's start would talk across it (leave touchscreen=1 out).
*
@@ -71,6 +93,7 @@
extern vm_offset_t ml_io_map(vm_offset_t phys_addr, vm_size_t size);
extern vm_offset_t ml_io_map_wcomb(vm_offset_t phys_addr, vm_size_t size);
+extern uint64_t kvtophys(vm_offset_t va); /* pmap_paddr_t */
static uint32_t last_step, skip_steps, pause_ms = 2200, fine, watch_seconds = 20;
static uint32_t step; /* the one being performed */
@@ -78,6 +101,32 @@ static uint32_t newest_step; /* the highest one asked for: the one th
static char last_write[96] = "nothing written";
static bool quiet; /* repeating accesses that have been announced and seen to work: say nothing */
+/*
+ * The kernel's log reaches the loader when a run is over, and a run that ends in a reset
+ * without a word leaves none. The loader also reads a log of the kernel's from another core
+ * while it runs (loader_return.c; xl keeps it with XL_FOLLOW): every line said here goes there
+ * as well, before the next register is touched.
+ */
+extern void loader_log_putc(char c);
+
+static int say(const char *format, ...) __printflike(1, 2);
+
+static int
+say(const char *format, ...)
+{
+ char line[400];
+ va_list list;
+
+ va_start(list, format);
+ vsnprintf(line, sizeof(line), format, list);
+ va_end(list);
+ for (const char *c = line; *c != 0; c++) {
+ loader_log_putc(*c);
+ }
+ return printf("%s", line);
+}
+#define printf say
+
static void
sleep_us(uint32_t us)
{
@@ -253,15 +302,27 @@ struct cmd_db_entry {
uint16_t length, offset;
};
+static const struct cmd_db_header *cmd_db;
+
/*
- * Say where RPMh has a resource and, for a supply rail, which levels it knows: the GMU's firmware
- * is told the levels by their position in this list.
+ * Where RPMh has a resource (0: nowhere) and, for a supply rail, the levels it knows: the GMU's
+ * firmware is told the levels by their position in this list.
*/
-static void
-show_rpmh_resource(const struct cmd_db_header *db, const char *name)
+static uint32_t
+rpmh_resource(const char *name, uint16_t *levels, unsigned *level_count)
{
- char id[8] = { 0 }, levels[160];
+ const struct cmd_db_header *db = cmd_db;
+ char id[8] = { 0 };
+ if (level_count != NULL) {
+ *level_count = 0;
+ }
+ if (db == NULL) {
+ db = cmd_db = (const struct cmd_db_header *)ml_io_map_wcomb(CMD_DB_PHYS, CMD_DB_SIZE); /* memory, not registers */
+ }
+ if (db->magic[0] != 0xdb || db->magic[1] != 0x30 || db->magic[2] != 0x03 || db->magic[3] != 0x0c) {
+ return 0;
+ }
strlcpy(id, name, sizeof(id));
for (unsigned s = 0; s < 8; s++) {
const struct cmd_db_slave *slave = &db->slave[s];
@@ -273,7 +334,6 @@ show_rpmh_resource(const struct cmd_db_header *db, const char *name)
for (unsigned i = 0; i < slave->count; i++) {
const uint8_t *data = db->data + slave->data_offset + entry[i].offset;
unsigned count = entry[i].length / 2u;
- int at = 0;
if (memcmp(entry[i].id, id, sizeof(id)) != 0) {
continue;
@@ -281,27 +341,42 @@ show_rpmh_resource(const struct cmd_db_header *db, const char *name)
if ((unsigned)slave->data_offset + entry[i].offset + entry[i].length > CMD_DB_SIZE - 0x1000) {
count = 0;
}
- levels[0] = 0;
- for (unsigned l = 0; l < count && l < 16 && at < (int)sizeof(levels) - 8; l++) {
- unsigned level = data[2 * l] | data[2 * l + 1] << 8;
-
- if (l > 0 && level == 0) {
+ for (unsigned l = 0; levels != NULL && l < count && l < 16; l++) {
+ levels[l] = (uint16_t)(data[2 * l] | data[2 * l + 1] << 8);
+ if (l > 0 && levels[l] == 0) {
break;
}
- at += snprintf(levels + at, sizeof(levels) - (size_t)at, " %u", level);
+ *level_count = l + 1;
}
- printf("adreno: %u: RPMh has \"%s\" at %#x; levels:%s\n", step, name, entry[i].address, levels);
- return;
+ return entry[i].address;
}
}
- printf("adreno: %u: RPMh has no \"%s\"\n", step, name);
+ return 0;
+}
+
+static void
+show_rpmh_resource(const char *name)
+{
+ uint16_t level[16];
+ unsigned count = 0;
+ uint32_t address = rpmh_resource(name, level, &count);
+ char levels[160];
+ int at = 0;
+
+ if (address == 0) {
+ printf("adreno: %u: RPMh has no \"%s\"\n", step, name);
+ return;
+ }
+ levels[0] = 0;
+ for (unsigned l = 0; l < count && at < (int)sizeof(levels) - 8; l++) {
+ at += snprintf(levels + at, sizeof(levels) - (size_t)at, " %u", level[l]);
+ }
+ printf("adreno: %u: RPMh has \"%s\" at %#x; levels:%s\n", step, name, address, levels);
}
static bool
step_outside(void)
{
- const struct cmd_db_header *db;
-
announce("read the GPU's clocks in the global clock controller (registers the touch screen driver uses too)");
get(&gcc, "pll votes", GCC_PLL_VOTE);
get(&gcc, "gpu_cfg_ahb", GCC_GPU_CFG_AHB_CBCR);
@@ -310,15 +385,9 @@ step_outside(void)
get(&gcc, "ddrss_gpu_axi", GCC_DDRSS_GPU_AXI_CBCR);
announce("look up the GPU's supply rails in the boot loader's list of RPMh resources (memory at 0x80860000)");
- db = (const struct cmd_db_header *)ml_io_map_wcomb(CMD_DB_PHYS, CMD_DB_SIZE); /* memory, not registers */
- if (db->magic[0] != 0xdb || db->magic[1] != 0x30 || db->magic[2] != 0x03 || db->magic[3] != 0x0c) {
- printf("adreno: %u: no list there (magic %02x %02x %02x %02x)\n", step, db->magic[0], db->magic[1],
- db->magic[2], db->magic[3]);
- return true; /* nothing before step 10 depends on it */
- }
- show_rpmh_resource(db, "gfx.lvl"); /* the 3D core's own rail (dt: VDD_GFX_LEVEL, PM8350C S7) */
- show_rpmh_resource(db, "cx.lvl"); /* the rail of the CX half, shared with most of the SoC */
- show_rpmh_resource(db, "mx.lvl"); /* memories */
+ show_rpmh_resource("gfx.lvl"); /* the 3D core's own rail (dt: VDD_GFX_LEVEL, PM8350C S7) */
+ show_rpmh_resource("cx.lvl"); /* the rail of the CX half, shared with most of the SoC */
+ show_rpmh_resource("mx.lvl"); /* memories */
return true;
}
@@ -606,6 +675,7 @@ step_clocks(void)
#define GPUCC_DEBUG_CBCR 0x1100
#define GPUCC_DEBUG_MUX 0x1568 /* [7:0] which of the GPU's clocks */
#define GPUCC_DEBUG_CXO_AON 0x0b
+#define GPUCC_DEBUG_GX_GFX3D 0x0d /* the 3D core's own clock */
#define GPUCC_DEBUG_GX_GMU 0x11
#define GPUCC_DEBUG_AHB 0x12
#define GPUCC_DEBUG_CX_GMU 0x1a
@@ -866,12 +936,41 @@ step_first_words(void)
* kernel asks RPMh for the touch screen's supply, but the GMU's own; through them its firmware
* raises the 3D core's rail. Offsets in words: a6xx_gmu.xml, domain A6XX_RSCC.
*/
-static struct block rscc = { "rscc", 0x03de0000, 0x1000 };
+static struct block rscc = { "rscc", 0x03de0000, 0x2000 };
#define RSCC_RSC_STATUS0_DRV0 (4 * 0x004)
#define RSCC_PDC_SLAVE_ID_DRV0 (4 * 0x00b)
#define RSCC_SEQ_BUSY_DRV0 (4 * 0x101)
#define RSCC_SEQ_MEM_0_DRV0 (4 * 0x180)
#define RSCC_TCS0_DRV0_STATUS (4 * 0x346)
+#define RSCC_ASLEEP (1u << 16) /* of rsc_status0: handed over to the always-on sequencer */
+#define RSCC_DRV_CONFIG 0x00c
+#define RSCC_SLOTS 0xd00 /* +0 interrupts enabled, +4 pending */
+#define RSCC_SLOT_SIZE 0x2a0
+
+/*
+ * The channel's state and its four command slots (Linux's rpmh-rsc.c: slot n at 0xd00 + 0x2a0 n:
+ * +0x10 commands whose completion is waited for, +0x14 control, +0x18 idle, +0x1c commands
+ * enabled, then 20 bytes a command from +0x30: number, address, data, status).
+ */
+static void
+rscc_show(void)
+{
+ get(&rscc, "drv config", RSCC_DRV_CONFIG);
+ get(&rscc, "rsc_status0_drv0", RSCC_RSC_STATUS0_DRV0);
+ get(&rscc, "seq_busy_drv0", RSCC_SEQ_BUSY_DRV0);
+ get(&rscc, "slots' interrupts enabled", RSCC_SLOTS);
+ get(&rscc, "slots' interrupts pending", RSCC_SLOTS + 4);
+ for (uint32_t slot = 0; slot < 4; slot++) {
+ vm_offset_t at = rscc.base + RSCC_SLOTS + RSCC_SLOT_SIZE * slot;
+
+ printf("adreno: %u: rscc slot %u: wait %#x, control %#x, idle %#x, enabled %#x; commands (number, address, data, "
+ "status):\n", step, slot, rd(at + 0x10), rd(at + 0x14), rd(at + 0x18), rd(at + 0x1c));
+ printf("adreno: %u: %x %x %x %x | %x %x %x %x | %x %x %x %x | %x %x %x %x\n", step,
+ rd(at + 0x30), rd(at + 0x34), rd(at + 0x38), rd(at + 0x3c), rd(at + 0x44), rd(at + 0x48), rd(at + 0x4c),
+ rd(at + 0x50), rd(at + 0x58), rd(at + 0x5c), rd(at + 0x60), rd(at + 0x64), rd(at + 0x6c), rd(at + 0x70),
+ rd(at + 0x74), rd(at + 0x78));
+ }
+}
static bool
step_rscc(void)
@@ -1025,6 +1124,287 @@ step_gmu_memory(void)
return back[0] == pattern[0] && back[1] == pattern[1];
}
+/* ---- memory the GPU block reaches ------------------------------------------------------------------- */
+
+/*
+ * Whatever in the GPU block reads or writes memory -- the GMU's processor, later the 3D core --
+ * does it through the SMMU, which makes of the address it uses ("the GPU's address" below) a
+ * physical one by walking page tables of ours, as a processor's MMU does: here always three
+ * levels of 4K pages, 39 bits of address (dt: qcom,use-3-lvl-tables). The SMMU has seven such
+ * translations ("contexts") and a table of which stream of requests -- 0 is the 3D core, 5 the
+ * GMU (dt: iommus) -- goes through which. Formats and the order of programming: Linux
+ * drivers/iommu/io-pgtable-arm.c and arm-smmu.c.
+ */
+#define PTE_TABLE 3ull
+#define PTE_PAGE 3ull
+#define PTE_CACHED (1ull << 2) /* memory attribute 1 of SMMU_MAIR0; without it 0, uncached */
+#define PTE_UNPRIVILEGED (1ull << 6) /* the 3D core's accesses are; the GMU's are not */
+#define PTE_SHARED (2ull << 8)
+#define PTE_ACCESSED (1ull << 10)
+#define PTE_NOT_GLOBAL (1ull << 11)
+#define PTE_ADDRESS 0x0000fffffffff000ull
+#define PTE_GMU (PTE_SHARED | PTE_ACCESSED | PTE_NOT_GLOBAL)
+
+#define SMMU_S2CR_FAULT (2u << 16)
+#define SMMU_SMR_VALID (1u << 31)
+#define SMMU_STREAM_MASK 0x400 /* dt: the second cell of every "iommus" of this SMMU */
+#define SMMU_CBAR_STAGE_1 (1u << 16 | 3u << 8 | 0xfu << 12) /* our tables alone decide; weakest defaults */
+#define SMMU_CBA2R(n) (0x800 + 4 * (n)) /* in the second page, like CBAR */
+#define SMMU_CBA2R_64_BIT 1
+#define SMMU_CB_TCR2 0x10
+#define SMMU_CB_TTBR1 0x28
+#define SMMU_CB_MAIR0 0x38
+#define SMMU_CB_MAIR1 0x3c
+#define SMMU_CB_FAR 0x60
+#define SMMU_CB_FSYNR0 0x68
+#define SMMU_CB_TLBIASID 0x610
+#define SMMU_CB_TLBSYNC 0x7f0
+#define SMMU_CB_TLBSTATUS 0x7f4
+#define SMMU_SCTLR_ON (1u << 0 | 1u << 1 | 1u << 2 | 1u << 5 | 1u << 12) /* on; faults recorded, no interrupt */
+#define SMMU_TCR_39_BITS (25u | 1u << 8 | 1u << 10 | 3u << 12 | 1u << 23) /* one table of 4K pages, cached walks */
+#define SMMU_TCR2 (1u | 1u << 4 | 7u << 15) /* 36 bits of physical address, 16 of context tag */
+#define SMMU_MAIR0 0xf404ff44 /* Linux's: 0 uncached, 1 cached, 2 device */
+#define SMMU_FSR_ALL 0xc00001fe /* every kind of fault, and "several" */
+
+extern void loader_return(void);
+
+/* Until every write to memory so far has been carried out. */
+static void
+settle(void)
+{
+ __asm__ volatile ("dsb sy");
+}
+
+/* Push what this processor has written to a stretch of memory out of its cache, and forget it: the GPU block reads memory. */
+static void
+to_memory(vm_offset_t start, vm_size_t size)
+{
+ for (vm_offset_t line = start & ~(vm_offset_t)63; line < start + size; line += 64) {
+ __asm__ volatile ("dc civac, %0" : : "r" (line));
+ }
+ settle();
+}
+
+#define SPACE_TABLES 48
+
+struct gpu_space {
+ const char *name;
+ uint32_t context; /* which of the SMMU's */
+ uint32_t control; /* bits of its control register besides the usual */
+ uint32_t table_count;
+ uint64_t *root;
+ struct {
+ uint64_t phys;
+ uint64_t *table;
+ } tables[SPACE_TABLES];
+};
+
+static uint64_t *
+space_new_table(struct gpu_space *space)
+{
+ static vm_offset_t spare;
+ static unsigned spare_left;
+ uint64_t *table;
+
+ if (space->table_count == SPACE_TABLES) {
+ return NULL;
+ }
+ if (spare_left == 0) {
+ if (kmem_alloc(kernel_map, &spare, PAGE_SIZE, KMA_KOBJECT | KMA_ZERO | KMA_DATA, VM_KERN_MEMORY_IOKIT) !=
+ KERN_SUCCESS) {
+ return NULL;
+ }
+ spare_left = PAGE_SIZE / 4096;
+ }
+ table = (uint64_t *)(spare + PAGE_SIZE - 4096 * spare_left--);
+ space->tables[space->table_count].phys = kvtophys((vm_offset_t)table);
+ space->tables[space->table_count++].table = table;
+ return table;
+}
+
+/* The table an entry of a higher table leads to; made if the entry is empty. */
+static uint64_t *
+space_next(struct gpu_space *space, uint64_t *entry)
+{
+ uint64_t *table;
+
+ if (*entry & 1) {
+ for (uint32_t i = 0; i < space->table_count; i++) {
+ if (space->tables[i].phys == (*entry & PTE_ADDRESS)) {
+ return space->tables[i].table;
+ }
+ }
+ return NULL;
+ }
+ table = space_new_table(space);
+ if (table != NULL) {
+ *entry = kvtophys((vm_offset_t)table) | PTE_TABLE;
+ }
+ return table;
+}
+
+/* One 4K page. */
+static bool
+space_map(struct gpu_space *space, uint64_t address, uint64_t phys, uint64_t attributes)
+{
+ uint64_t *level2, *level3;
+
+ if (address >> 39 || (space->root == NULL && (space->root = space_new_table(space)) == NULL)) {
+ return false;
+ }
+ level2 = space_next(space, &space->root[address >> 30 & 0x1ff]);
+ level3 = level2 != NULL ? space_next(space, &level2[address >> 21 & 0x1ff]) : NULL;
+ if (level3 == NULL) {
+ return false;
+ }
+ level3[address >> 12 & 0x1ff] = phys | attributes | PTE_PAGE;
+ return true;
+}
+
+/* Memory of ours that the GPU block sees at an address of its own. */
+struct gpu_block {
+ uint64_t address; /* the GPU's */
+ uint32_t size;
+ vm_offset_t memory; /* ours */
+};
+
+static bool
+block_new(struct gpu_space *space, struct gpu_block *block, uint64_t address, uint32_t size, uint64_t attributes)
+{
+ block->address = address;
+ block->size = (size + 4095) & ~4095u;
+ if (kmem_alloc(kernel_map, &block->memory, round_page(block->size), KMA_KOBJECT | KMA_ZERO | KMA_DATA,
+ VM_KERN_MEMORY_IOKIT) != KERN_SUCCESS) {
+ return false;
+ }
+ for (uint32_t offset = 0; offset < block->size; offset += 4096) {
+ if (!space_map(space, address + offset, kvtophys(block->memory + offset), attributes)) {
+ return false;
+ }
+ }
+ to_memory(block->memory, block->size);
+ printf("adreno: %u: %s sees %#llx..%#llx in memory of ours (%#llx)\n", step, space->name, address,
+ address + block->size - 1, kvtophys(block->memory));
+ return true;
+}
+
+/*
+ * A second view of a block without this processor's cache in between, for memory that both
+ * sides write a word at a time. Only blocks of one page of ours: those are in one piece.
+ */
+static vm_offset_t
+block_uncached(const struct gpu_block *block)
+{
+ uint64_t phys = kvtophys(block->memory);
+
+ for (uint32_t offset = 0; offset < block->size; offset += 4096) {
+ if (kvtophys(block->memory + offset) != phys + offset) {
+ return 0;
+ }
+ }
+ return ml_io_map_wcomb(phys, block->size);
+}
+
+static uint32_t
+smmu_context(const struct gpu_space *space)
+{
+ return (smmu_pages + space->context) * smmu_page;
+}
+
+/* What a context has recorded of a request it could not serve. */
+static bool
+space_faulted(const struct gpu_space *space)
+{
+ uint32_t context = smmu_context(space), status = rd(smmu.base + context + SMMU_CB_FSR);
+
+ if ((status & SMMU_FSR_ALL) == 0) { /* bits 10:9 only tell the format of the syndrome */
+ return false;
+ }
+ printf("adreno: %u: THE SMMU REFUSED %s: status %#x, address %#llx, syndrome %#x\n", step, space->name, status,
+ (uint64_t)rd(smmu.base + context + SMMU_CB_FAR + 4) << 32 | rd(smmu.base + context + SMMU_CB_FAR),
+ rd(smmu.base + context + SMMU_CB_FSYNR0));
+ return true;
+}
+
+/*
+ * One more stream through a context that is on. A match is taken only as the device tree has
+ * it, a stream and the mask 0x400: one entry for streams 0 and 1 (mask 0x401) reads back 0.
+ */
+static bool
+space_stream(struct gpu_space *space, uint32_t match, uint32_t stream)
+{
+ put(&smmu, "s2cr", SMMU_S2CR(match), space->context);
+ put(&smmu, "smr", SMMU_SMR(match), SMMU_SMR_VALID | SMMU_STREAM_MASK << 16 | stream);
+ if (get(&smmu, "smr", SMMU_SMR(match)) != (SMMU_SMR_VALID | SMMU_STREAM_MASK << 16 | stream) ||
+ (get(&smmu, "s2cr", SMMU_S2CR(match)) & 0x300ff) != space->context) {
+ printf("adreno: %u: the SMMU did not take what was written: stream %u is not going through context %u\n", step,
+ stream, space->context);
+ return false;
+ }
+ return true;
+}
+
+/* Give a context our tables and send a stream through it; `match` is the entry of the stream table to use. */
+static bool
+space_on(struct gpu_space *space, uint32_t match, uint32_t stream)
+{
+ uint32_t context = smmu_context(space);
+ uint64_t root = space->tables[0].phys | (uint64_t)space->context << 48; /* tagged with the context's number */
+
+ for (uint32_t i = 0; i < space->table_count; i++) {
+ to_memory((vm_offset_t)space->tables[i].table, 4096);
+ }
+ put(&smmu, "cb sctlr", context + SMMU_CB_SCTLR, 0);
+ put(&smmu, "cba2r", smmu_page + SMMU_CBA2R(space->context), SMMU_CBA2R_64_BIT);
+ put(&smmu, "cbar", smmu_page + SMMU_CBAR(space->context), SMMU_CBAR_STAGE_1);
+ put(&smmu, "cb tcr2", context + SMMU_CB_TCR2, SMMU_TCR2);
+ put(&smmu, "cb tcr", context + SMMU_CB_TCR, SMMU_TCR_39_BITS);
+ put(&smmu, "cb ttbr0, low", context + SMMU_CB_TTBR0, (uint32_t)root);
+ put(&smmu, "cb ttbr0, high", context + SMMU_CB_TTBR0 + 4, (uint32_t)(root >> 32));
+ put(&smmu, "cb ttbr1, low", context + SMMU_CB_TTBR1, 0);
+ put(&smmu, "cb ttbr1, high", context + SMMU_CB_TTBR1 + 4, 0);
+ put(&smmu, "cb mair0", context + SMMU_CB_MAIR0, SMMU_MAIR0);
+ put(&smmu, "cb mair1", context + SMMU_CB_MAIR1, 0);
+ put(&smmu, "cb fsr", context + SMMU_CB_FSR, SMMU_FSR_ALL);
+ put(&smmu, "cb tlbiasid", context + SMMU_CB_TLBIASID, space->context);
+ put(&smmu, "cb tlbsync", context + SMMU_CB_TLBSYNC, 1);
+ if (!wait_for(&smmu, context + SMMU_CB_TLBSTATUS, 1, 0, 100)) {
+ printf("adreno: %u: the SMMU does not finish forgetting old translations (%#x)\n", step,
+ rd(smmu.base + context + SMMU_CB_TLBSTATUS));
+ return false;
+ }
+ put(&smmu, "cb sctlr", context + SMMU_CB_SCTLR, SMMU_SCTLR_ON | space->control);
+ put(&smmu, "s2cr", SMMU_S2CR(match), space->context);
+ put(&smmu, "smr", SMMU_SMR(match), SMMU_SMR_VALID | SMMU_STREAM_MASK << 16 | stream);
+
+ /* the global ones are the hypervisor's to grant: see what it made of them */
+ if (get(&smmu, "smr", SMMU_SMR(match)) != (SMMU_SMR_VALID | SMMU_STREAM_MASK << 16 | stream) ||
+ (get(&smmu, "s2cr", SMMU_S2CR(match)) & 0x300ff) != space->context ||
+ (get(&smmu, "cbar", smmu_page + SMMU_CBAR(space->context)) & 0x30000) != (SMMU_CBAR_STAGE_1 & 0x30000) ||
+ !(get(&smmu, "cb sctlr", context + SMMU_CB_SCTLR) & 1) ||
+ get(&smmu, "cb ttbr0, low", context + SMMU_CB_TTBR0) != (uint32_t)root ||
+ get(&smmu, "cb ttbr0, high", context + SMMU_CB_TTBR0 + 4) != (uint32_t)(root >> 32)) {
+ printf("adreno: %u: the SMMU did not take what was written: stream %u is not going through context %u\n", step,
+ stream, space->context);
+ return false;
+ }
+ get(&smmu, "cb tcr", context + SMMU_CB_TCR);
+ get(&smmu, "cba2r", smmu_page + SMMU_CBA2R(space->context));
+ get(&smmu, "cb tcr2", context + SMMU_CB_TCR2);
+ get(&smmu, "cb fsr", context + SMMU_CB_FSR);
+ get(&smmu, "sgfsr", SMMU_SGFSR);
+ return true;
+}
+
+/* A stream goes nowhere again, and its context translates nothing. */
+static void
+space_off(struct gpu_space *space, uint32_t match)
+{
+ put(&smmu, "smr", SMMU_SMR(match), 0);
+ put(&smmu, "s2cr", SMMU_S2CR(match), SMMU_S2CR_FAULT);
+ put(&smmu, "cb sctlr", smmu_context(space) + SMMU_CB_SCTLR, 0);
+}
+
/* ---- step 11: TrustZone -------------------------------------------------------------------------------- */
/*
@@ -1054,11 +1434,13 @@ step_gmu_memory(void)
/* One call. What TrustZone returns: 0 or an error of its own, and a word of answer. */
static int32_t
-scm(const char *what, uint64_t function, uint64_t arguments, uint64_t first, uint64_t second, uint64_t *answer)
+scm(const char *what, uint64_t function, uint64_t arguments, uint64_t first, uint64_t second, uint64_t third,
+ uint64_t *answer)
{
uint64_t session = 0;
- printf("adreno: %u: about to ask TrustZone %s: smc %#llx, arguments %#llx, %#llx\n", step, what, function, first, second);
+ printf("adreno: %u: about to ask TrustZone %s: smc %#llx, arguments %#llx, %#llx, %#llx\n", step, what, function, first,
+ second, third);
snprintf(last_write, sizeof(last_write), "step %u asked TrustZone %#llx(%#llx, %#llx)", step, function, first, second);
hold(true);
for (int i = 0; i < 100; i++) {
@@ -1066,7 +1448,7 @@ scm(const char *what, uint64_t function, uint64_t arguments, uint64_t first, uin
register uint64_t x1 __asm__("x1") = arguments;
register uint64_t x2 __asm__("x2") = first;
register uint64_t x3 __asm__("x3") = second;
- register uint64_t x4 __asm__("x4") = 0;
+ register uint64_t x4 __asm__("x4") = third;
register uint64_t x5 __asm__("x5") = 0;
register uint64_t x6 __asm__("x6") = session;
register uint64_t x7 __asm__("x7") = 0;
@@ -1096,10 +1478,10 @@ step_trustzone(void)
announce("ask TrustZone what Linux asks it first: is the call \"is this call available\" available");
if (scm("whether it can be asked", how | SCM_IS_CALL_AVAILABLE, SCM_ONE_VALUE, SCM_SIP | SCM_IS_CALL_AVAILABLE, 0,
- &answer) != 0 || answer != 1) {
+ 0, &answer) != 0 || answer != 1) {
how = SCM_FAST | SCM_SIP; /* the same with 32-bit arguments, Linux's second try */
if (scm("the same, the 32-bit way", how | SCM_IS_CALL_AVAILABLE, SCM_ONE_VALUE, SCM_SIP | SCM_IS_CALL_AVAILABLE,
- 0, &answer) != 0 || answer != 1) {
+ 0, 0, &answer) != 0 || answer != 1) {
printf("adreno: %u: TrustZone does not answer the way it answers Linux\n", step);
return false;
}
@@ -1107,18 +1489,18 @@ step_trustzone(void)
announce("ask TrustZone whether it has the calls that authenticate a firmware image");
scm("for the call that takes an image's signature", how | SCM_IS_CALL_AVAILABLE, SCM_ONE_VALUE,
- SCM_SIP | SCM_PAS_INIT_IMAGE, 0, &answer);
+ SCM_SIP | SCM_PAS_INIT_IMAGE, 0, 0, &answer);
scm("for the call that authenticates and starts an image", how | SCM_IS_CALL_AVAILABLE, SCM_ONE_VALUE,
- SCM_SIP | SCM_PAS_AUTH_AND_RESET, 0, &answer);
+ SCM_SIP | SCM_PAS_AUTH_AND_RESET, 0, 0, &answer);
announce("ask TrustZone which images it \"supports\": three this phone starts through it, and the GPU's; each asked "
"both ways, at once and (as Linux does) as a call that may be preempted");
for (unsigned i = 0; i < 4; i++) {
answer = 0;
- fast[i] = scm(images[i].name, how | SCM_PAS_IS_SUPPORTED, SCM_ONE_VALUE, images[i].id, 0, &answer) == 0 && answer;
+ fast[i] = scm(images[i].name, how | SCM_PAS_IS_SUPPORTED, SCM_ONE_VALUE, images[i].id, 0, 0, &answer) == 0 && answer;
answer = 0;
slow[i] = scm(images[i].name, (how & ~SCM_FAST) | SCM_PAS_IS_SUPPORTED, SCM_ONE_VALUE, images[i].id, 0,
- &answer) == 0 && answer;
+ 0, &answer) == 0 && answer;
}
printf("adreno: %u: \"supported\", asked at once / as Linux asks: audio %u/%u, modem %u/%u, video %u/%u, GPU %u/%u\n",
step, fast[0], slow[0], fast[1], slow[1], fast[2], slow[2], fast[3], slow[3]);
@@ -1133,7 +1515,6 @@ step_trustzone(void)
extern dev_t mdevlookup(int devid);
extern int mdevgetkernelrange(dev_t dev, vm_offset_t *base, uint64_t *size);
-extern uint64_t kvtophys(vm_offset_t va); /* pmap_paddr_t */
/* An entry of a directory of a packfs image in memory, by name. */
static const struct packfs_dnode *
@@ -1210,16 +1591,15 @@ image_file(const char *directory, const char *name, uint64_t *size)
* ("creates SHM bridges on behalf of Linux"), and that TrustZone protects the memory while it
* works: so it is not touched again here, and never freed.
*/
+/* The zap shader's header in a page of ours, as TrustZone wants it; where that page is in physical memory. */
static bool
-step_zap_signature(void)
+zap_header(uint64_t *phys)
{
const uint8_t *header;
vm_offset_t page = 0;
- uint64_t size = 0, answer = ~0ull;
+ uint64_t size = 0;
uint32_t segments, hashes;
- int32_t result;
- announce("find the zap shader's header, fw/a660_zap.mdt, in the small image that came with the kernel");
header = image_file("fw", "a660_zap.mdt", &size);
if (header == NULL) {
return false;
@@ -1243,15 +1623,24 @@ step_zap_signature(void)
return false;
}
memcpy((void *)page, header, size);
- /* TrustZone reads memory, not this processor's cache */
- for (vm_offset_t line = page; line < page + PAGE_SIZE; line += 64) {
- __asm__ volatile ("dc cvac, %0" : : "r" (line));
- }
- __asm__ volatile ("dsb sy");
+ to_memory(page, PAGE_SIZE); /* TrustZone reads memory, not this processor's cache */
+ *phys = kvtophys(page);
+ return true;
+}
+
+static bool
+step_zap_signature(void)
+{
+ uint64_t header = 0, answer = ~0ull;
+ int32_t result;
+ announce("find the zap shader's header, fw/a660_zap.mdt, in the small image that came with the kernel");
+ if (!zap_header(&header)) {
+ return false;
+ }
announce("hand the header to TrustZone to check its signature (image 13); a call that may be preempted, as Linux's is");
result = scm("to take the zap shader's signature", SCM_64 | SCM_SIP | SCM_PAS_INIT_IMAGE, SCM_VALUE_AND_BUFFER, PAS_GPU,
- kvtophys(page), &answer);
+ header, 0, &answer);
if (result == 0 && answer == 0) {
printf("adreno: %u: TRUSTZONE ACCEPTS THE ZAP SHADER'S SIGNATURE FROM THIS KERNEL\n", step);
return true;
@@ -1260,62 +1649,1375 @@ step_zap_signature(void)
return false;
}
-/* ---- all of it together ---------------------------------------------------------------------------- */
-
-static bool(*const steps[])(void) = {
- step_outside, step_fuses, step_gpucc, step_cx_power, step_clocks, step_first_words, step_rscc, step_smmu,
- step_smmu_context, step_gmu_memory, step_trustzone, step_zap_signature,
-};
-#define STEPS ((uint32_t)(sizeof(steps) / sizeof(steps[0])))
+/* ---- step 13: memory for the GMU ------------------------------------------------------------------------ */
/*
- * Called while the system starts, before there is a root file system, and the start waits: for
- * as long as the steps and the watch after them last, nothing else writes to the log, so that
- * its end -- all that the loader's notes keep -- is about what is being done here.
+ * The GMU is a Cortex-M3 with 16K of code memory and 16K of data memory of its own (written
+ * through registers, step 10); the rest of its firmware and everything it shares with us it
+ * reaches through the SMMU, at addresses the firmware file names: a file of blocks, each "this
+ * goes at address A" or, with no data, "I need N bytes at address A" (vendor's
+ * adreno_a6xx_gmu.c: a6xx_gmu_parse_fw, a6xx_gmu_load_fw). Besides those, where the vendor's
+ * driver puts them: 16K for the firmware's log and a page for the table and queues of the
+ * message protocol ("HFI").
*/
-void
-redwood_adreno_init(void)
+#define GMU_ITCM_ADDRESS 0x00000000u
+#define GMU_DTCM_ADDRESS 0x10004000u /* a6xx_gmu_fw_load: the 650 family's */
+#define GMU_TCM_SIZE 0x4000u
+#define GMU_LOG_ADDRESS 0x60002000u /* after the two pages the firmware asks for at 0x60000000 */
+#define GMU_LOG_SIZE 0x4000u
+#define GMU_HFI_ADDRESS 0x60006000u
+#define GMU_STREAM 5
+#define GMU_MATCH 1 /* entry of the stream table; 0 is kept for the 3D core */
+
+struct gmu_firmware_block {
+ uint32_t address, size, type, value;
+};
+#define GMU_FIRMWARE_NEEDS 1 /* types of blocks without data: memory wanted, value = bytes */
+#define GMU_FIRMWARE_NEEDS_KEPT 7
+
+static struct gpu_space gmu_space = { .name = "the GMU", .context = 1 };
+static struct gpu_block gmu_blocks[12], gmu_log, gmu_hfi;
+static unsigned gmu_block_count;
+static const uint8_t *gmu_firmware;
+static uint64_t gmu_firmware_size;
+static bool gmu_space_is_on, gmu_runs, gmu_talks, core_is_on, rsc_awake, cx_off;
+static void gmu_post_mortem(void);
+
+static bool
+in_tcm(uint32_t address)
{
- uint32_t only = 0, done = 0;
+ return address - GMU_ITCM_ADDRESS < GMU_TCM_SIZE || address - GMU_DTCM_ADDRESS < GMU_TCM_SIZE;
+}
- PE_parse_boot_argn("adreno", &last_step, sizeof(last_step));
- PE_parse_boot_argn("adreno_skip", &skip_steps, sizeof(skip_steps));
- if (PE_parse_boot_argn("adreno_only", &only, sizeof(only)) && only > 0 && only <= STEPS) {
- last_step = only;
- skip_steps = ~(1u << only);
- }
- if (last_step == 0) {
- return;
- }
- if (last_step > STEPS) {
- last_step = STEPS;
+static bool
+step_gmu_space(void)
+{
+ if (!cx_ready()) {
+ return false;
}
- PE_parse_boot_argn("adreno_pause", &pause_ms, sizeof(pause_ms));
- PE_parse_boot_argn("adreno_fine", &fine, sizeof(fine));
- PE_parse_boot_argn("adreno_watch", &watch_seconds, sizeof(watch_seconds));
- for (newest_step = last_step; newest_step > 0 && (skip_steps & 1u << newest_step); newest_step--) {
+ announce("find the GMU's firmware, fw/a660_gmu.bin, in the small image, and make the memory it asks for");
+ gmu_firmware = image_file("fw", "a660_gmu.bin", &gmu_firmware_size);
+ if (gmu_firmware == NULL) {
+ return false;
}
+ for (uint64_t at = 0; at + sizeof(struct gmu_firmware_block) <= gmu_firmware_size;) {
+ struct gmu_firmware_block block;
- printf("adreno: steps up to %u, without those in mask %#x; step %u is the new one and waits %u ms %s\n", last_step,
- skip_steps, newest_step, pause_ms, fine ? "before every access" : "after each announcement");
- for (step = 1; step <= last_step; step++) {
- if (skip_steps & 1u << step) {
+ memcpy(&block, gmu_firmware + at, sizeof(block));
+ at += sizeof(block) + block.size;
+ if (at > gmu_firmware_size) {
+ printf("adreno: %u: the file ends inside a block\n", step);
+ return false;
+ }
+ if (block.size != 0 || (block.type != GMU_FIRMWARE_NEEDS && block.type != GMU_FIRMWARE_NEEDS_KEPT) ||
+ in_tcm(block.address)) {
continue;
}
- if (!steps[step - 1]()) {
- printf("adreno: step %u: FAILED; the steps after it are not attempted\n", step);
- break;
+ if (gmu_block_count == sizeof(gmu_blocks) / sizeof(gmu_blocks[0]) || (block.address & 4095) ||
+ block.value > 0x100000 || !block_new(&gmu_space, &gmu_blocks[gmu_block_count++], block.address, block.value,
+ PTE_GMU)) {
+ printf("adreno: %u: cannot make %#x bytes at %#x for the firmware\n", step, block.value, block.address);
+ return false;
}
- printf("adreno: step %u: done\n", step);
- done = step;
+ }
+ if (!block_new(&gmu_space, &gmu_log, GMU_LOG_ADDRESS, GMU_LOG_SIZE, PTE_GMU) ||
+ !block_new(&gmu_space, &gmu_hfi, GMU_HFI_ADDRESS, PAGE_SIZE, PTE_GMU)) {
+ return false;
+ }
+
+ announce("give the SMMU's context 1 these tables and send the GMU's requests (stream 5) through it: the first "
+ "writes to the SMMU, the global ones (cbar, cba2r, s2cr, smr) through the hypervisor");
+ if (!smmu_layout()) {
+ return false;
+ }
+ gmu_space_is_on = true; /* whatever of it has been written by the time something fails is taken back */
+ return space_on(&gmu_space, GMU_MATCH, GMU_STREAM);
+}
+
+/* ---- step 14: the GMU's firmware ----------------------------------------------------------------------- */
+
+#define GMU_ITCM GMU_WORD(0x0c00)
+
+static bool
+step_gmu_firmware(void)
+{
+ uint32_t wrong = 0;
+
+ if (!cx_ready() || gmu_firmware == NULL) {
+ return false;
+ }
+ announce("load the GMU's firmware: its first 16K of code and of data through the GMU's registers (0x3d6d000, "
+ "0x3d71000), the rest into the memory just made");
+ put(&gmu, "cm3_sysreset", GMU_CM3_SYSRESET, 1);
+ for (int pass = 0; pass < 2; pass++) {
+ for (uint64_t at = 0; at + sizeof(struct gmu_firmware_block) <= gmu_firmware_size;) {
+ struct gmu_firmware_block block;
+ const uint8_t *data = gmu_firmware + at + sizeof(block);
+
+ memcpy(&block, gmu_firmware + at, sizeof(block));
+ at += sizeof(block) + block.size;
+ if (block.size == 0) {
+ continue;
+ }
+ if (in_tcm(block.address) && in_tcm(block.address + block.size - 1)) {
+ uint32_t registers = block.address < GMU_DTCM_ADDRESS ? GMU_ITCM + block.address - GMU_ITCM_ADDRESS :
+ GMU_DTCM + block.address - GMU_DTCM_ADDRESS;
+
+ for (uint32_t offset = 0; offset + 4 <= block.size; offset += 4) {
+ uint32_t word;
+
+ memcpy(&word, data + offset, sizeof(word));
+ if (pass == 0) {
+ wr(gmu.base + registers + offset, word);
+ } else if (rd(gmu.base + registers + offset) != word) {
+ wrong++;
+ }
+ }
+ if (pass == 1) {
+ printf("adreno: %u: %#x bytes at the GMU's %#x, through its registers\n", step, block.size,
+ block.address);
+ }
+ continue;
+ }
+ for (unsigned i = 0; pass == 0 && i <= gmu_block_count; i++) {
+ if (i == gmu_block_count) {
+ printf("adreno: %u: the firmware has %#x bytes for %#x, where it asked for no memory\n", step,
+ block.size, block.address);
+ return false;
+ }
+ if (block.address >= gmu_blocks[i].address &&
+ block.address + block.size <= gmu_blocks[i].address + gmu_blocks[i].size) {
+ memcpy((void *)(gmu_blocks[i].memory + (block.address - gmu_blocks[i].address)), data, block.size);
+ to_memory(gmu_blocks[i].memory, gmu_blocks[i].size);
+ printf("adreno: %u: %#x bytes at the GMU's %#x, in memory\n", step, block.size, block.address);
+ break;
+ }
+ }
+ }
+ }
+ snprintf(last_write, sizeof(last_write), "step %u wrote the GMU's firmware into its code and data memory", step);
+ if (wrong) {
+ printf("adreno: %u: %u words of the GMU's own memory do not read back as written\n", step, wrong);
+ return false;
+ }
+ printf("adreno: %u: THE GMU'S FIRMWARE IS LOADED; its versions, read back from the GMU: core %#x, power %#x, HFI %#x\n",
+ step, rd(gmu.base + GMU_DTCM_VERSIONS), rd(gmu.base + GMU_DTCM_VERSIONS + 8), rd(gmu.base + GMU_DTCM_VERSIONS + 16));
+ return true;
+}
+
+/* ---- step 15: start the GMU ---------------------------------------------------------------------------- */
+
+#define GMU_ICACHE_CONFIG GMU_WORD(0x4c00)
+#define GMU_DCACHE_CONFIG GMU_WORD(0x4c01)
+#define GMU_SYS_BUS_CONFIG GMU_WORD(0x4c0f)
+#define GMU_PWR_COL_INTER_FRAME_CTRL GMU_WORD(0x50c0)
+#define GMU_RPMH_CTRL GMU_WORD(0x50e8)
+#define GMU_RPMH_CTRL_ALL 0xf11 /* the RPMh interface, and every vote the firmware may cast */
+#define GMU_CX_FAL_INTF GMU_WORD(0x50f0)
+#define GMU_CX_FALNEXT_INTF GMU_WORD(0x50f1)
+#define GMU_PWR_COL_CP_MSG GMU_WORD(0x5100) /* at start: where the log is, and its pages - 1 */
+#define GMU_PWR_COL_CP_RESP GMU_WORD(0x5101) /* ... and how far it was written before */
+#define GMU_HFI_SFR_ADDR GMU_WORD(0x5182) /* at start: which GPU this is */
+#define GMU_HFI_QTBL_INFO GMU_WORD(0x5184)
+#define GMU_HFI_QTBL_ADDR GMU_WORD(0x5185)
+#define GMU_HFI_CTRL_INIT GMU_WORD(0x5186)
+#define GMU_GMU2HOST_INTR_CLR GMU_WORD(0x5191)
+#define GMU_GMU2HOST_INTR_INFO GMU_WORD(0x5192) /* bit 0: a message for us; bit 31: the 3D core is ours */
+#define GMU_HOST2GMU_INTR_SET GMU_WORD(0x5194) /* bit 0: a message for the GMU; bit 30, 31: see step 17 */
+#define GMU_RSCC_CONTROL_REQ GMU_WORD(0x9307)
+#define GMU_AO_AHB_FENCE_CTRL GMU_WORD(0x9310)
+#define GMU_AHB_FENCE_RANGE_0 GMU_WORD(0x9311)
+#define GMU_CHIP 0x06035000 /* dt qcom,chipid 0x06030500 the way the firmware wants it */
+#define GMU_STARTED_MASK 0x1ff /* of fw_init_result, for firmware newer than core 0x20010004 */
+#define GMU_STARTED 0x100
+#define GMU_FAILED 0xe00
+
+/* The table at the head of the HFI page and the three queues after it, as the vendor's driver lays them out. */
+struct hfi_queue {
+ uint32_t status, start, type, words, message_words, unused[5], read, write;
+};
+struct hfi_table {
+ uint32_t version, words, first_queue, queue_words, queues, active_queues;
+ struct hfi_queue queue[17];
+};
+#define HFI_QUEUES 3 /* 0: ours to the GMU; 1: its answers; 2: its debugging messages */
+#define HFI_QUEUE_WORDS 1024
+#define HFI_QUEUE_AT(n) (((uint32_t)sizeof(struct hfi_table) + 15) / 16 * 16 + 4096 * (n))
+
+static volatile struct hfi_table *hfi; /* the uncached view */
+
+/*
+ * Everything the vendor's driver writes before it lets the processor go (a6xx_gmu_register_config,
+ * a6xx_gmu_power_config), in its order; then out of reset (a6xx_gmu_device_start).
+ */
+static bool
+step_gmu_start(void)
+{
+ static const uint8_t priority[HFI_QUEUES] = { 10, 10, 40 };
+ uint32_t result = 0;
+
+ if (!cx_ready() || !gmu_space_is_on) {
+ return false;
+ }
+ hfi = (volatile struct hfi_table *)block_uncached(&gmu_hfi);
+ if (hfi == NULL) {
+ printf("adreno: %u: the page for the messages is not in one piece\n", step);
+ return false;
+ }
+ hfi->words = sizeof(struct hfi_table) / 4;
+ hfi->first_queue = 6;
+ hfi->queue_words = sizeof(struct hfi_queue) / 4;
+ hfi->queues = hfi->active_queues = HFI_QUEUES;
+ for (unsigned i = 0; i < HFI_QUEUES; i++) {
+ hfi->queue[i].start = GMU_HFI_ADDRESS + HFI_QUEUE_AT(i);
+ hfi->queue[i].type = i | (uint32_t)priority[i] << 8;
+ hfi->queue[i].words = HFI_QUEUE_WORDS;
+ hfi->queue[i].status = 1;
+ }
+
+ announce("tell the GMU where things are, by its registers: the message table, the log, which GPU this is; its "
+ "caches and its use of RPMh; then let its processor out of reset");
+ if (get(&gmu, "cm3_sysreset", GMU_CM3_SYSRESET) != 1) {
+ printf("adreno: %u: the GMU's processor is not held in reset: not configuring it while it runs\n", step);
+ return false;
}
/*
- * Punishment for a write may come late (a reset some 17 seconds after it has been seen):
- * keep the end of the log saying where things stand.
+ * Our own leaving (gpu_block_stop) hands the GMU's RPMh channel to the always-on sequencer, and
+ * until it is taken back nothing the GMU asks of RPMh gets there: the vendor's
+ * a6xx_rscc_wakeup_sequence, before the firmware starts.
*/
- for (uint32_t second = 1; second <= watch_seconds; second++) {
- sleep_ms(1000);
- printf("adreno: alive %u s after step %u; %s\n", second, done, last_write);
+ rscc_show();
+ if (rd(rscc.base + RSCC_RSC_STATUS0_DRV0) & RSCC_ASLEEP) {
+ put(&gmu, "rscc_control_req", GMU_RSCC_CONTROL_REQ, 2);
+ if (!wait_for(&gmu, GMU_RSCC_CONTROL_ACK, 2, 2, 100) || !wait_for(&rscc, RSCC_SEQ_BUSY_DRV0, ~0u, 0, 100)) {
+ printf("adreno: %u: the GMU's RPMh channel does not wake\n", step);
+ }
+ get(&gmu, "rscc_control_ack", GMU_RSCC_CONTROL_ACK);
+ put(&gmu, "rscc_control_req", GMU_RSCC_CONTROL_REQ, 0);
+ get(&rscc, "rsc_status0_drv0", RSCC_RSC_STATUS0_DRV0);
+ get(&rscc, "seq_busy_drv0", RSCC_SEQ_BUSY_DRV0);
+ }
+ rsc_awake = true;
+ put(&gmu, "cx_falnext_intf", GMU_CX_FALNEXT_INTF, 1);
+ put(&gmu, "cx_fal_intf", GMU_CX_FAL_INTF, 1);
+ put(&gmu, "general_7", GMU_GENERAL_7, 1);
+ put(&gmu, "cm3_fw_init_result", GMU_CM3_FW_INIT_RESULT, 0);
+ put(&gmu, "cm3_boot_config", GMU_CM3_BOOT_CONFIG, 2);
+ put(&gmu, "hfi_qtbl_addr", GMU_HFI_QTBL_ADDR, GMU_HFI_ADDRESS);
+ put(&gmu, "hfi_qtbl_info", GMU_HFI_QTBL_INFO, 1);
+ put(&gmu, "ahb_fence_range_0", GMU_AHB_FENCE_RANGE_0, 1u << 31 | (0xa << 2) << 18 | 0x8a0);
+ put(&gmu, "cm3_cfg", GMU_CM3_CFG, 0x4052);
+ put(&gbif, "halt", GBIF_HALT, 0);
+ put(&gmu, "pwr_col_cp_resp", GMU_PWR_COL_CP_RESP, 0);
+ put(&gmu, "hfi_sfr_addr", GMU_HFI_SFR_ADDR, GMU_CHIP);
+ put(&gmu, "pwr_col_cp_msg", GMU_PWR_COL_CP_MSG, GMU_LOG_ADDRESS | (GMU_LOG_SIZE / 4096 - 1));
+ put(&gmu, "sys_bus_config", GMU_SYS_BUS_CONFIG, 1);
+ put(&gmu, "icache_config", GMU_ICACHE_CONFIG, 1);
+ put(&gmu, "dcache_config", GMU_DCACHE_CONFIG, 1);
+ put(&gmu, "pwr_col_inter_frame_ctrl", GMU_PWR_COL_INTER_FRAME_CTRL, 0x9c40400);
+ set(&gmu, "rpmh_ctrl", GMU_RPMH_CTRL, GMU_RPMH_CTRL_ALL);
+ put(&gmu, "gmu2host_intr_clr", GMU_GMU2HOST_INTR_CLR, 0xffffffff);
+
+ gmu_runs = true;
+ put(&gmu, "cm3_sysreset", GMU_CM3_SYSRESET, 0);
+ for (int i = 0; i < 5000 && (result & GMU_STARTED_MASK) != GMU_STARTED && !(result & GMU_FAILED); i++) {
+ sleep_us(100);
+ result = rd(gmu.base + GMU_CM3_FW_INIT_RESULT);
+ }
+ get(&gmu, "cm3_fw_init_result", GMU_CM3_FW_INIT_RESULT);
+ get(&gmu, "gmu2host_intr_info", GMU_GMU2HOST_INTR_INFO);
+ get(&gmu, "ao_host_interrupt_status", GMU_AO_HOST_INTERRUPT_STATUS);
+ get(&gmu, "hfi_ctrl_status", GMU_HFI_CTRL_STATUS);
+ if (space_faulted(&gmu_space) || (result & GMU_STARTED_MASK) != GMU_STARTED) {
+ printf("adreno: %u: the GMU's firmware does not come up\n", step);
+ gmu_post_mortem();
+ return false;
+ }
+ printf("adreno: %u: THE GMU'S FIRMWARE RUNS\n", step);
+ return true;
+}
+
+/* ---- step 16: the GMU's way to RPMh, and its message queues ------------------------------------------------ */
+
+/*
+ * The GMU raises and drops the 3D core's supply by itself, through command slots of its own
+ * (step 7) and a sequencer in the always-on part of the SoC, "PDC". On this SoC the always-on
+ * processor programs the PDC; left to us are its two enabling registers and the GMU side's
+ * little program (vendor's a6xx_load_pdc_ucode with pdc_in_aop, a6xx_load_rsc_ucode: after the
+ * firmware has started and before the message queues are).
+ */
+static struct block pdc = { "pdc", 0x0b290000, 0x10000 }; /* dt: qcom,gmu "kgsl_gmu_pdc_cfg" */
+#define PDC_GPU_ENABLE (4 * 0x1140)
+#define PDC_GPU_SEQ_START_ADDR (4 * 0x1148)
+#define RSCC_PDC_SEQ_START_ADDR (4 * 0x008)
+#define RSCC_PDC_MATCH_VALUE_LO (4 * 0x009)
+#define RSCC_PDC_MATCH_VALUE_HI (4 * 0x00a)
+#define RSCC_HIDDEN_TCS_CMD0_ADDR (4 * 0x00d)
+#define RSCC_HIDDEN_TCS_CMD0_DATA (4 * 0x00e)
+#define RSCC_OVERRIDE_START_ADDR (4 * 0x100)
+
+static bool
+step_gmu_queues(void)
+{
+ static const uint32_t program[5] = { 0xeaaae5a0, 0xe1a1ebab, 0xa2e0a581, 0xecac82e2, 0x0020edad };
+
+ if (!cx_ready() || !gmu_runs) {
+ return false;
+ }
+ announce("enable the GPU's part of the always-on sequencer (pdc, 0xb294500 and 0xb294520: a part of the SoC not "
+ "touched before)");
+ get(&pdc, "gpu_enable", PDC_GPU_ENABLE);
+ get(&pdc, "gpu_seq_start_addr", PDC_GPU_SEQ_START_ADDR);
+ put(&pdc, "gpu_seq_start_addr", PDC_GPU_SEQ_START_ADDR, 0);
+ put(&pdc, "gpu_enable", PDC_GPU_ENABLE, 0x80000001);
+ get(&pdc, "gpu_enable", PDC_GPU_ENABLE);
+
+ announce("give the GMU's RPMh channel its sleep and wake program (rscc, 0x3de0000), unless it has it");
+ if (get(&rscc, "seq_mem_0_drv0", RSCC_SEQ_MEM_0_DRV0) == program[0] &&
+ get(&rscc, "hidden_tcs_cmd2_data", RSCC_HIDDEN_TCS_CMD0_DATA + 4 * 4) == 0x80000021) {
+ goto programmed;
+ }
+ put(&rscc, "rsc_status0_drv0", RSCC_RSC_STATUS0_DRV0, 1u << 24);
+ put(&rscc, "pdc_slave_id_drv0", RSCC_PDC_SLAVE_ID_DRV0, 1);
+ put(&rscc, "hidden_tcs_cmd0_data", RSCC_HIDDEN_TCS_CMD0_DATA, 0);
+ put(&rscc, "hidden_tcs_cmd0_addr", RSCC_HIDDEN_TCS_CMD0_ADDR, 0);
+ put(&rscc, "hidden_tcs_cmd1_data", RSCC_HIDDEN_TCS_CMD0_DATA + 4 * 2, 0);
+ put(&rscc, "hidden_tcs_cmd1_addr", RSCC_HIDDEN_TCS_CMD0_ADDR + 4 * 2, 0);
+ put(&rscc, "hidden_tcs_cmd2_data", RSCC_HIDDEN_TCS_CMD0_DATA + 4 * 4, 0x80000021); /* a650 and later */
+ put(&rscc, "hidden_tcs_cmd2_addr", RSCC_HIDDEN_TCS_CMD0_ADDR + 4 * 4, 0);
+ put(&rscc, "override_start_addr", RSCC_OVERRIDE_START_ADDR, 0);
+ put(&rscc, "pdc_seq_start_addr", RSCC_PDC_SEQ_START_ADDR, 0x4520);
+ put(&rscc, "pdc_match_value_lo", RSCC_PDC_MATCH_VALUE_LO, 0x4510);
+ put(&rscc, "pdc_match_value_hi", RSCC_PDC_MATCH_VALUE_HI, 0x4514);
+ for (unsigned i = 0; i < 5; i++) {
+ put(&rscc, "seq_mem_drv0", RSCC_SEQ_MEM_0_DRV0 + 4 * i, program[i]);
+ }
+ get(&rscc, "seq_mem_0_drv0", RSCC_SEQ_MEM_0_DRV0);
+
+programmed:
+ announce("ask the GMU to start its message queues");
+ put(&gmu, "hfi_ctrl_init", GMU_HFI_CTRL_INIT, 1);
+ if (!wait_for(&gmu, GMU_HFI_CTRL_STATUS, 1, 1, 500)) {
+ get(&gmu, "hfi_ctrl_status", GMU_HFI_CTRL_STATUS);
+ get(&gmu, "cm3_fw_init_result", GMU_CM3_FW_INIT_RESULT);
+ space_faulted(&gmu_space);
+ printf("adreno: %u: the GMU does not start its message queues\n", step);
+ return false;
+ }
+ get(&gmu, "hfi_ctrl_status", GMU_HFI_CTRL_STATUS);
+ get(&gmu, "hfi_version_info", GMU_HFI_VERSION_INFO);
+ gmu_talks = true;
+ printf("adreno: %u: THE GMU'S MESSAGE QUEUES ARE UP\n", step);
+ return !space_faulted(&gmu_space);
+}
+
+/* ---- step 17: talk to the GMU; the 3D core powered ---------------------------------------------------------- */
+
+/*
+ * A message is words: the first says which (bits 7:0), how many words (15:8), a request or an
+ * answer (19:16), and a number (31:20) that the answer repeats in its second word; the third word
+ * of an answer is the GMU's verdict, 0 for done. Vendor's adreno_a6xx_hfi.c, Linux a6xx_hfi.c.
+ */
+#define HFI_BW_TABLE 3
+#define HFI_PERF_TABLE 4
+#define HFI_START 10
+#define HFI_CORE_FW_START 14
+#define HFI_GX_BW_PERF_VOTE 30
+#define HFI_PREPARE_SLUMBER 33
+#define HFI_ERROR 100
+#define HFI_ANSWER 1 /* in bits 19:16 */
+#define HFI_NO_INDEX 0xff
+
+struct hfi_perf_table {
+ uint32_t header, gpu_levels, gmu_levels;
+ struct {
+ uint32_t vote, acd, khz;
+ } gpu[16];
+ struct {
+ uint32_t vote, khz;
+ } gmu[4];
+};
+struct hfi_bw_table {
+ uint32_t header, levels, cnoc_commands, ddr_commands, cnoc_wait, ddr_wait;
+ uint32_t cnoc_address[6], cnoc_data[2][6], ddr_address[8], ddr_data[16][8];
+};
+
+static uint32_t hfi_number;
+static bool post_mortem_done;
+
+/* Everything waiting in a queue of the GMU's; true if the answer to the message numbered `number` was among it. */
+static bool
+hfi_receive(unsigned which, uint32_t number, uint32_t *verdict)
+{
+ volatile struct hfi_queue *queue = &hfi->queue[which];
+ volatile uint32_t *data = (volatile uint32_t *)((vm_offset_t)hfi + HFI_QUEUE_AT(which));
+ bool answered = false;
+
+ while (queue->read != queue->write) {
+ uint32_t message[8] = { 0 }, at = queue->read % HFI_QUEUE_WORDS, words = data[at] >> 8 & 0xff;
+
+ for (uint32_t i = 0; i < words; i++) {
+ if (i < 8) {
+ message[i] = data[at];
+ }
+ at = (at + 1) % HFI_QUEUE_WORDS;
+ }
+ queue->read = (at + 3) / 4 * 4 % HFI_QUEUE_WORDS;
+ if ((message[0] >> 16 & 0xf) == HFI_ANSWER && message[1] >> 20 == number) {
+ *verdict = message[2];
+ answered = true;
+ } else {
+ message[7] = 0;
+ if ((message[0] & 0xff) == HFI_ERROR) { /* vendor's adreno_a6xx_receive_err_req */
+ printf("adreno: %u: THE GMU REPORTS AN ERROR: %u, %u, \"%s\"\n", step, message[1] >> 16,
+ message[1] & 0xffff, (const char *)&message[2]);
+ } else {
+ printf("adreno: %u: the GMU says (queue %u, %u words): %#x %#x %#x %#x %#x %#x\n", step, which, words,
+ message[0], message[1], message[2], message[3], message[4], message[5]);
+ }
+ }
+ if (words == 0) {
+ break;
+ }
+ }
+ return answered;
+}
+
+/* A word of the GMU's own data, by the GMU's address for it; only what lies in memory of ours. */
+static uint32_t
+gmu_data(uint32_t address)
+{
+ for (unsigned i = 0; i < gmu_block_count; i++) {
+ if (address - (uint32_t)gmu_blocks[i].address < gmu_blocks[i].size - 3) {
+ vm_offset_t at = gmu_blocks[i].memory + (address - (uint32_t)gmu_blocks[i].address);
+
+ to_memory(at, 4); /* what the GMU wrote, not what our cache remembers */
+ return *(volatile uint32_t *)at;
+ }
+ }
+ return 0xdeadbeef;
+}
+
+/*
+ * What can be seen of a GMU that has stopped answering: the tail of its log (16 bytes an event:
+ * which, the crystal's counter, two values; the firmware's function at 0x1a98; where it writes
+ * next is in pwr_col_cp_resp), its record of what it was asked for and what it has (firmware
+ * 1.8's data at 0x108454: +0x28 is what is in flight, +0x40.. what is in force), its RPMh
+ * channel, and the 3D core's switch and PLLs, which only the GMU's firmware writes.
+ */
+#define GMU_STATE_ADDRESS 0x108454
+
+static void
+gmu_post_mortem(void)
+{
+ uint32_t next = rd(gmu.base + GMU_PWR_COL_CP_RESP) & 0xffff, entries = GMU_LOG_SIZE / 16;
+ const volatile uint32_t *log = (const volatile uint32_t *)gmu_log.memory;
+
+ to_memory(gmu_log.memory, GMU_LOG_SIZE);
+ printf("adreno: %u: the GMU's log (event, time, two values), %u events in, the last 40:\n", step, next);
+ for (uint32_t i = next > 40 ? next - 40 : 0; i < next && i + 4 <= entries; i += 4) {
+ printf("adreno: %u: %x %x %x %x | %x %x %x %x | %x %x %x %x | %x %x %x %x\n", step,
+ log[4 * i], log[4 * i + 1], log[4 * i + 2], log[4 * i + 3],
+ log[4 * i + 4], log[4 * i + 5], log[4 * i + 6], log[4 * i + 7],
+ log[4 * i + 8], log[4 * i + 9], log[4 * i + 10], log[4 * i + 11],
+ log[4 * i + 12], log[4 * i + 13], log[4 * i + 14], log[4 * i + 15]);
+ }
+ for (uint32_t i = 0; i < 24; i += 8) {
+ printf("adreno: %u: the GMU's state +%#x: %x %x %x %x %x %x %x %x\n", step, 4 * i,
+ gmu_data(GMU_STATE_ADDRESS + 4 * i), gmu_data(GMU_STATE_ADDRESS + 4 * i + 4),
+ gmu_data(GMU_STATE_ADDRESS + 4 * i + 8), gmu_data(GMU_STATE_ADDRESS + 4 * i + 12),
+ gmu_data(GMU_STATE_ADDRESS + 4 * i + 16), gmu_data(GMU_STATE_ADDRESS + 4 * i + 20),
+ gmu_data(GMU_STATE_ADDRESS + 4 * i + 24), gmu_data(GMU_STATE_ADDRESS + 4 * i + 28));
+ }
+ rscc_show();
+ get(&gpucc, "gx_gdscr", GPUCC_GX_GDSCR);
+ get(&gpucc, "gx_sw_reset", GPUCC_GX_SW_RESET);
+ get(&gpucc, "gx_domain_misc", GPUCC_GX_DOMAIN_MISC);
+ get(&gpucc, "pll0 mode", GPUCC_PLL0_MODE);
+ get(&gpucc, "pll0 multiplier", GPUCC_PLL0_MODE + 4);
+ get(&gpucc, "pll1 mode", GPUCC_PLL1_MODE);
+ get(&gpucc, "pll1 multiplier", GPUCC_PLL1_MODE + 4);
+ get(&gmu, "rpmh_power_state", GMU_RPMH_POWER_STATE);
+ get(&gmu, "sptprac_pwr_clk_status", GMU_SPTPRAC_PWR_CLK_STATUS);
+ get(&gmu, "ao_host_interrupt_status", GMU_AO_HOST_INTERRUPT_STATUS);
+ get(&gmu, "ahb_fence_status", GMU_AHB_FENCE_STATUS);
+}
+
+/* Send a message whose first word is yet to be made and wait for the GMU's verdict on it. */
+static bool
+hfi_send(const char *what, uint32_t id, uint32_t *message, uint32_t words)
+{
+ volatile struct hfi_queue *queue = &hfi->queue[0];
+ volatile uint32_t *data = (volatile uint32_t *)((vm_offset_t)hfi + HFI_QUEUE_AT(0));
+ uint32_t at = queue->write % HFI_QUEUE_WORDS, number = ++hfi_number & 0xfff, verdict = 0, waiting = 0;
+ bool answered = false;
+
+ message[0] = id | words << 8 | number << 20;
+ printf("adreno: %u: about to send the GMU %s: message %u, %u words, number %u\n", step, what, id, words, number);
+ snprintf(last_write, sizeof(last_write), "step %u sent the GMU message %u (%s)", step, id, what);
+ hold(true);
+ for (uint32_t i = 0; i < (words + 3) / 4 * 4; i++) {
+ data[at] = i < words ? message[i] : 0xfafafafa;
+ at = (at + 1) % HFI_QUEUE_WORDS;
+ }
+ queue->write = at;
+ settle();
+ wr(gmu.base + GMU_HOST2GMU_INTR_SET, 1);
+ for (int i = 0; i < 10000 && !answered; i++) {
+ sleep_us(100);
+ waiting = rd(gmu.base + GMU_GMU2HOST_INTR_INFO);
+ if (waiting & 1) {
+ wr(gmu.base + GMU_GMU2HOST_INTR_CLR, 1);
+ answered = hfi_receive(1, number, &verdict);
+ }
+ if (rd(gmu.base + GMU_CM3_FW_INIT_RESULT) & GMU_FAILED) {
+ break;
+ }
+ }
+ hfi_receive(2, ~0u, &waiting);
+ if (!answered || verdict != 0) {
+ printf("adreno: %u: the GMU %s (%#x) to %s; fw_init_result %#x, gmu2host_intr_info %#x\n", step,
+ answered ? "says no" : "does not answer", verdict, what, rd(gmu.base + GMU_CM3_FW_INIT_RESULT),
+ rd(gmu.base + GMU_GMU2HOST_INTR_INFO));
+ space_faulted(&gmu_space);
+ if (!post_mortem_done) {
+ post_mortem_done = true;
+ gmu_post_mortem();
+ }
+ return false;
+ }
+ printf("adreno: %u: the GMU accepts %s\n", step, what);
+ return true;
+}
+
+/*
+ * A vote for a level of a supply: the level, its place in RPMh's list for the rail, and the place
+ * of the level the memories' rail must have with it (vendor's setup_volt_dependency_tbl).
+ */
+static bool
+rail_vote(uint32_t level, const uint16_t *rail, unsigned rail_levels, const uint16_t *mx, unsigned mx_levels, uint32_t *vote)
+{
+ unsigned at, with = 0;
+
+ for (at = 0; at < rail_levels && rail[at] < level; at++) {
+ }
+ if (at == rail_levels || mx_levels == 0) {
+ return false;
+ }
+ while (with + 1 < mx_levels && mx[with] < rail[at]) {
+ with++;
+ }
+ *vote = (uint32_t)rail[at] << 16 | with << 8 | at;
+ return true;
+}
+
+/* dt: kgsl_3d0_reg_memory: the 3D core's registers; a6xx.xml counts in words */
+static struct block gpu = { "gpu", 0x03d00000, 0x40000 };
+#define GPU_WORD(n) (4u * (n))
+#define RBBM_INT_0_STATUS GPU_WORD(0x0201)
+#define RBBM_STATUS GPU_WORD(0x0210)
+#define RBBM_CLOCK_CNTL GPU_WORD(0x00ae)
+#define CP_RB_BASE GPU_WORD(0x0800)
+#define GMU_CORE_REQUEST (1u << 30) /* of host2gmu_intr_set: keep the 3D core up for us; bit 31: no more */
+#define GMU_CORE_GRANTED (1u << 31) /* of gmu2host_intr_info */
+
+/*
+ * What the vendor's driver sends at this point (a6xx_hfi_start; Linux the same with fewer
+ * levels): the table of speeds of the 3D core with the supply level each needs -- the phone's
+ * three lowest (dt: qcom,gpu-pwrlevels: 315, 450, 550 MHz at "levels" 64, 128, 192) -- and of the
+ * GMU itself; the table of bus bandwidth votes, of which Linux gets by with the one for "off"
+ * (a6xx_hfi.c: adreno_7c3_build_bw_table); "start"; and the speed wanted now. Then the request
+ * to keep the 3D core powered while we use it (a6xx_gmu_oob_set), and its registers can be read.
+ */
+static bool
+step_core_power(void)
+{
+ static const struct {
+ uint32_t khz, level;
+ } speeds[] = { { 0, 0 }, { 315000, 64 }, { 450000, 128 }, { 550000, 192 } };
+ static struct hfi_perf_table perf;
+ static struct hfi_bw_table bw;
+ uint32_t core_start[2] = { 0, 0 }, start[1] = { 0 }, vote[4] = { 0, 1, 1, HFI_NO_INDEX }, granted = 0;
+ uint16_t gfx[16], cx[16], mx[16];
+ unsigned gfx_levels, cx_levels, mx_levels;
+
+ if (!cx_ready() || !gmu_talks) {
+ return false;
+ }
+ if (!rpmh_resource("gfx.lvl", gfx, &gfx_levels) || !rpmh_resource("cx.lvl", cx, &cx_levels) ||
+ !rpmh_resource("mx.lvl", mx, &mx_levels)) {
+ printf("adreno: %u: RPMh's list lacks one of the rails\n", step);
+ return false;
+ }
+ perf.gpu_levels = sizeof(speeds) / sizeof(speeds[0]);
+ for (unsigned i = 0; i < perf.gpu_levels; i++) {
+ perf.gpu[i].khz = speeds[i].khz;
+ perf.gpu[i].acd = 0xffffffff; /* no adaptive clocking */
+ if (!rail_vote(speeds[i].level, gfx, gfx_levels, mx, mx_levels, &perf.gpu[i].vote)) {
+ printf("adreno: %u: RPMh has no level %u for the 3D core's rail\n", step, speeds[i].level);
+ return false;
+ }
+ }
+ perf.gmu_levels = 2;
+ perf.gmu[1].khz = 200000;
+ if (!rail_vote(0, cx, cx_levels, mx, mx_levels, &perf.gmu[0].vote) ||
+ !rail_vote(48, cx, cx_levels, mx, mx_levels, &perf.gmu[1].vote)) { /* "minimum SVS", the vendor's choice */
+ return false;
+ }
+ printf("adreno: %u: votes: 3D core %#x %#x %#x %#x, GMU %#x %#x\n", step, perf.gpu[0].vote, perf.gpu[1].vote,
+ perf.gpu[2].vote, perf.gpu[3].vote, perf.gmu[0].vote, perf.gmu[1].vote);
+
+ bw.levels = 1;
+ bw.ddr_commands = 3;
+ bw.ddr_wait = 7;
+ bw.ddr_address[0] = rpmh_resource("SH0", NULL, NULL);
+ bw.ddr_address[1] = rpmh_resource("MC0", NULL, NULL);
+ bw.ddr_address[2] = rpmh_resource("ACV", NULL, NULL);
+ bw.ddr_data[0][0] = bw.ddr_data[0][1] = bw.ddr_data[0][2] = 0x40000000;
+ bw.cnoc_commands = 1;
+ bw.cnoc_wait = 1;
+ bw.cnoc_address[0] = rpmh_resource("CN0", NULL, NULL);
+ bw.cnoc_data[0][0] = 0x40000000;
+ bw.cnoc_data[1][0] = 0x60000001;
+ printf("adreno: %u: RPMh's bus resources: SH0 %#x, MC0 %#x, ACV %#x, CN0 %#x (Linux has 0x50004, 0x50000, 0x50088, "
+ "0x5006c)\n", step, bw.ddr_address[0], bw.ddr_address[1], bw.ddr_address[2], bw.cnoc_address[0]);
+ if (bw.ddr_address[0] == 0 || bw.ddr_address[1] == 0 || bw.ddr_address[2] == 0 || bw.cnoc_address[0] == 0) {
+ return false;
+ }
+
+ announce("send the GMU its tables (speeds and supply levels, bus votes), \"start\", and the speed wanted: 315 MHz; "
+ "from here on it casts votes in RPMh by itself");
+ if (!hfi_send("the table of speeds", HFI_PERF_TABLE, &perf.header, sizeof(perf) / 4) ||
+ !hfi_send("the table of bus votes", HFI_BW_TABLE, &bw.header, sizeof(bw) / 4) ||
+ !hfi_send("\"start the core firmware\"", HFI_CORE_FW_START, core_start, 2) ||
+ !hfi_send("the speed wanted", HFI_GX_BW_PERF_VOTE, vote, 4)) {
+ return false;
+ }
+
+ announce("ask the GMU to power the 3D core and keep it powered");
+ get(&gmu, "sptprac_pwr_clk_status", GMU_SPTPRAC_PWR_CLK_STATUS);
+ get(&gpucc, "gx_gdscr", GPUCC_GX_GDSCR);
+ put(&gmu, "host2gmu_intr_set", GMU_HOST2GMU_INTR_SET, GMU_CORE_REQUEST);
+ for (int i = 0; i < 5000 && !(granted & GMU_CORE_GRANTED); i++) {
+ sleep_us(100);
+ granted = rd(gmu.base + GMU_GMU2HOST_INTR_INFO);
+ }
+ core_is_on = true; /* asked for, at least: to be given back */
+ put(&gmu, "gmu2host_intr_clr", GMU_GMU2HOST_INTR_CLR, GMU_CORE_GRANTED);
+ get(&gmu, "sptprac_pwr_clk_status", GMU_SPTPRAC_PWR_CLK_STATUS);
+ get(&gpucc, "gx_gdscr", GPUCC_GX_GDSCR);
+ get(&gmu, "rpmh_power_state", GMU_RPMH_POWER_STATE);
+ if (!(granted & GMU_CORE_GRANTED) || (rd(gmu.base + GMU_SPTPRAC_PWR_CLK_STATUS) & 0xc0) != 0) {
+ printf("adreno: %u: the GMU does not power the 3D core (gmu2host_intr_info %#x, fw_init_result %#x)\n", step,
+ granted, rd(gmu.base + GMU_CM3_FW_INIT_RESULT));
+ space_faulted(&gmu_space);
+ return false;
+ }
+ if (!hfi_send("\"start\"", HFI_START, start, 1)) {
+ return false;
+ }
+ printf("adreno: %u: measured: the 3D core's clock runs at %u Hz (315 MHz was asked for), the GMU's in it at %u Hz\n", step,
+ measure_quietly(GPUCC_DEBUG_GX_GFX3D), measure_quietly(GPUCC_DEBUG_GX_GMU));
+
+ announce("read registers of the 3D core itself (0x3d00000): the first time it has power");
+ get(&gpu, "rbbm_status", RBBM_STATUS);
+ get(&gpu, "rbbm_int_0_status", RBBM_INT_0_STATUS);
+ get(&gpu, "rbbm_clock_cntl", RBBM_CLOCK_CNTL);
+ get(&gpu, "cp_rb_base", CP_RB_BASE);
+ printf("adreno: %u: THE 3D CORE HAS POWER AND ANSWERS\n", step);
+ return !space_faulted(&gmu_space);
+}
+
+/* ---- step 18: the 3D core's memory and its microcode -------------------------------------------------------- */
+
+/*
+ * The 3D core takes its orders as words in a ring of memory, which a small processor inside it
+ * ("SQE", with microcode of its own: a660_sqe.fw) reads up to where a register says we have
+ * written. Like the GMU it reaches memory through the SMMU: stream 0, to which this gives
+ * context 0, with tables of ours that hold the ring, the microcode and later what is drawn into.
+ * What is written to the core's registers before its processor is started, and in what order,
+ * is Linux's a6xx_gpu.c hw_init for this GPU, less what only matters once it draws (clock gating,
+ * the protection of registers against user programs' commands, counters).
+ */
+#define CORE_STREAM 0
+#define CORE_MATCH 0
+#define PTE_CORE (PTE_GMU | PTE_UNPRIVILEGED)
+#define PTE_READ_ONLY (1ull << 7) /* Linux gives the core its microcode so (MSM_BO_GPU_READONLY) */
+#define CORE_RING_ADDRESS 0x01000000u
+#define CORE_RING_SIZE 0x8000u /* Linux's; 8192 words */
+#define CORE_SQE_ADDRESS 0x01100000u
+#define CORE_SCRATCH_ADDRESS 0x01200000u
+#define CORE_TARGET_ADDRESS 0x10000000u
+#define CORE_NOWHERE 0x7f000000u /* never given anything */
+
+#define GBIF_QSB_SIDE0 (4 * 0x03) /* of the bus interface (step 6) */
+#define RBBM_GBIF_CLIENT_QOS_CNTL GPU_WORD(0x0011)
+#define RBBM_GBIF_HALT GPU_WORD(0x0016)
+#define RBBM_INTERFACE_HANG_INT_CNTL GPU_WORD(0x001f)
+#define RBBM_PERFCTR_CNTL GPU_WORD(0x0500)
+#define RBBM_PERFCTR_GPU_BUSY_MASKED GPU_WORD(0x050b)
+#define CP_RB_CNTL GPU_WORD(0x0802)
+#define CP_RB_RPTR GPU_WORD(0x0806)
+#define CP_RB_WPTR GPU_WORD(0x0807)
+#define CP_SQE_CNTL GPU_WORD(0x0808)
+#define CP_HW_FAULT GPU_WORD(0x0821)
+#define CP_INTERRUPT_STATUS GPU_WORD(0x0823)
+#define CP_PROTECT_STATUS GPU_WORD(0x0824)
+#define CP_SQE_INSTR_BASE GPU_WORD(0x0830)
+#define CP_CHICKEN_DBG GPU_WORD(0x0841)
+#define CP_ADDR_MODE_CNTL GPU_WORD(0x0842)
+#define CP_APRIV_CNTL GPU_WORD(0x0844)
+#define CP_ROQ_THRESHOLDS_1 GPU_WORD(0x08c1)
+#define CP_ROQ_THRESHOLDS_2 GPU_WORD(0x08c2)
+#define CP_MEM_POOL_SIZE GPU_WORD(0x08c3)
+#define CP_AHB_CNTL GPU_WORD(0x098d)
+#define CP_LPAC_PROG_FIFO_SIZE GPU_WORD(0x0b34)
+#define UCHE_WRITE_RANGE_MAX GPU_WORD(0x0e05)
+#define UCHE_WRITE_THRU_BASE GPU_WORD(0x0e07)
+#define UCHE_TRAP_BASE GPU_WORD(0x0e09)
+#define UCHE_CACHE_WAYS GPU_WORD(0x0e17)
+#define UCHE_FILTER_CNTL GPU_WORD(0x0e18)
+#define UCHE_CLIENT_PF GPU_WORD(0x0e19)
+#define RB_DBG_ECO_CNTL GPU_WORD(0x8e04)
+#define PC_DBG_ECO_CNTL GPU_WORD(0x9e00)
+#define RBBM_SECVID_TSB_TRUSTED_BASE GPU_WORD(0xf800)
+#define RBBM_SECVID_TSB_TRUSTED_SIZE GPU_WORD(0xf802)
+#define RBBM_SECVID_TSB_CNTL GPU_WORD(0xf803)
+#define RBBM_SECVID_TSB_ADDR_MODE_CNTL GPU_WORD(0xf810)
+#define CP_RB_CNTL_VALUE (12u | 2u << 8 | 1u << 27) /* 2^12 pairs of words; do not copy the read position to memory */
+
+/* commands (Mesa's adreno_pm4.xml); a packet is a header word and so many more */
+#define CP_NOP 0x10
+#define CP_WAIT_FOR_IDLE 0x26
+#define CP_BLIT 0x2c
+#define CP_MEM_WRITE 0x3d
+#define CP_EVENT_WRITE 0x46
+#define CP_ME_INIT 0x48
+#define CP_SET_MARKER 0x65
+#define CP_SET_SECURE_MODE 0x66
+
+/*
+ * Linux (qcom_adreno_smmu_write_sctlr): after a fault the core's other requests are served, or it
+ * hangs. And the core has a second stream, 1 (dt: gfx3d_lpac; Linux's sc7280.dtsi gives the GPU
+ * both): once started, the microcode starts a second command processor ("LPAC"), which fetches
+ * its own part of the microcode as that stream at once -- seen on the phone as a request for
+ * 0x7e60 into the microcode, refused by an empty context given to stream 1 (adreno_stream1=2).
+ * With nothing at all set up for stream 1 the phone resets without a word the moment the
+ * command processor is started (adreno_stream1=0); a start with the microcode nowhere
+ * (adreno_blind=1) does not get that far.
+ */
+#define SMMU_SCTLR_SERVE_AFTER_FAULT (1u << 8)
+static struct gpu_space core_space = { .name = "the 3D core", .context = 0, .control = SMMU_SCTLR_SERVE_AFTER_FAULT };
+/* adreno_stream1=2: the second stream gets a context of its own with nothing in it, to see what it asks for */
+static struct gpu_space second_space = {
+ .name = "the 3D core's second stream", .context = 2, .control = SMMU_SCTLR_SERVE_AFTER_FAULT,
+};
+#define SECOND_MATCH 2
+static bool second_space_is_on, second_stream_is_on;
+static struct gpu_block core_ring, core_sqe, core_scratch, core_target;
+static bool core_space_is_on, core_runs, zap_is_loaded;
+static uint32_t ring_words; /* written to the ring so far; it is never filled here */
+
+static uint32_t
+odd_parity(uint32_t v)
+{
+ return 0x9669 >> (0xf & (v ^ v >> 4 ^ v >> 8 ^ v >> 12 ^ v >> 16 ^ v >> 20 ^ v >> 24 ^ v >> 28)) & 1;
+}
+
+static void
+ring(uint32_t word)
+{
+ ((uint32_t *)core_ring.memory)[ring_words++] = word;
+}
+
+/* The header of a command with so many words after it. */
+static void
+ring_command(uint32_t command, uint32_t words)
+{
+ ring(7u << 28 | words | odd_parity(words) << 15 | command << 16 | odd_parity(command) << 23);
+}
+
+/* A register of the 3D core (by its word number) written by command, in the order of the other commands. */
+static void
+ring_register(uint32_t number, uint32_t value)
+{
+ ring(4u << 28 | 1 | odd_parity(1) << 7 | number << 8 | odd_parity(number) << 27);
+ ring(value);
+}
+
+/* Let the 3D core have what is in the ring, and wait until it has taken all of it and rests. */
+static bool
+ring_run(const char *what)
+{
+ uint32_t taken = ~0u, status = ~0u;
+
+ to_memory(core_ring.memory, ring_words * 4);
+ printf("adreno: %u: about to have the 3D core %s: the ring up to word %u\n", step, what, ring_words);
+ snprintf(last_write, sizeof(last_write), "step %u had the 3D core %s", step, what);
+ hold(true);
+ wr(gpu.base + CP_RB_WPTR, ring_words);
+ for (int i = 0; i < 20000 && (taken != ring_words || (status & ~1u)); i++) {
+ sleep_us(100);
+ taken = rd(gpu.base + CP_RB_RPTR);
+ status = rd(gpu.base + RBBM_STATUS);
+ }
+ if (taken == ring_words && !(status & ~1u) && rd(gpu.base + CP_HW_FAULT) == 0 && !space_faulted(&core_space)) {
+ printf("adreno: %u: the 3D core has done it (ring read up to %u, status %#x)\n", step, taken, status);
+ return true;
+ }
+ printf("adreno: %u: the 3D core does not get through: ring read up to %u of %u, status %#x, interrupts %#x, "
+ "command processor: fault %#x, interrupts %#x, protection %#x\n", step, taken, ring_words, status,
+ rd(gpu.base + RBBM_INT_0_STATUS), rd(gpu.base + CP_HW_FAULT), rd(gpu.base + CP_INTERRUPT_STATUS),
+ rd(gpu.base + CP_PROTECT_STATUS));
+ space_faulted(&core_space);
+ return false;
+}
+
+static void
+put64(struct block *b, const char *name, uint32_t offset, uint64_t value)
+{
+ put(b, name, offset, (uint32_t)value);
+ put(b, name, offset + 4, (uint32_t)(value >> 32));
+}
+
+static bool
+step_core_space(void)
+{
+ const uint8_t *microcode;
+ uint64_t size = 0;
+ uint32_t stream1 = 1;
+
+ if (!cx_ready() || !core_is_on) {
+ return false;
+ }
+ PE_parse_boot_argn("adreno_stream1", &stream1, sizeof(stream1));
+ announce("make the 3D core's memory: its ring, its microcode (fw/a660_sqe.fw), a page to try writing to");
+ microcode = image_file("fw", "a660_sqe.fw", &size);
+ if (microcode == NULL || size < 8 || size > 0x40000 ||
+ !block_new(&core_space, &core_ring, CORE_RING_ADDRESS, CORE_RING_SIZE, PTE_CORE) ||
+ !block_new(&core_space, &core_sqe, CORE_SQE_ADDRESS, (uint32_t)size - 4, PTE_CORE | PTE_READ_ONLY) ||
+ !block_new(&core_space, &core_scratch, CORE_SCRATCH_ADDRESS, 4096, PTE_CORE)) {
+ return false;
+ }
+ memcpy((void *)core_sqe.memory, microcode + 4, size - 4); /* the file's first word is not microcode */
+ to_memory(core_sqe.memory, core_sqe.size);
+
+ announce("give the SMMU's context 0 these tables and send the 3D core's requests (stream 0) through it");
+ core_space_is_on = true;
+ if (!space_on(&core_space, CORE_MATCH, CORE_STREAM)) {
+ return false;
+ }
+ if (stream1 == 2) {
+ if ((second_space.root = space_new_table(&second_space)) == NULL) {
+ return false;
+ }
+ second_space_is_on = true;
+ if (!space_on(&second_space, SECOND_MATCH, 1)) {
+ return false;
+ }
+ } else if (stream1 == 1) {
+ second_stream_is_on = true;
+ if (!space_stream(&core_space, SECOND_MATCH, 1)) {
+ return false;
+ }
+ }
+ get(&gpu, "rbbm_status", RBBM_STATUS);
+ return !space_faulted(&core_space);
+}
+
+/* ---- step 19: the zap shader, loaded -------------------------------------------------------------------- */
+
+/*
+ * The rest of what Linux does with the zap shader (mdt_loader.c: qcom_mdt_pas_init,
+ * qcom_mdt_load; adreno_gpu.c: zap_shader_load_mdt): after the header (step 12), TrustZone is
+ * told where the image will be -- the two pages the device tree reserves for it
+ * (reserved-memory gpu_micro_code@8b71a000, the zap-shader node's memory-region) --, the image's
+ * one segment (the file a660_zap.b02, relocatable: at the start of that memory) is put there,
+ * and TrustZone is asked to authenticate it and run it. The vendor's driver does this before it
+ * starts the 3D core's command processor (a6xx_rb_start: a6xx_microcode_load), Linux after;
+ * this phone's own order is kept. It is done once in the life of a started phone: afterwards
+ * the memory is TrustZone's, and nothing here can see whether that is so. Our kernels come and
+ * go without the phone restarting: boot-arg adreno_zap=0 says an earlier run has done it.
+ */
+#define ZAP_MEMORY 0x8b71a000ull
+#define ZAP_MEMORY_SIZE 0x2000u
+#define SCM_PAS_MEM_SETUP SCM_CALL(2, 2)
+#define SCM_THREE_VALUES 3
+
+static bool
+step_zap_load(void)
+{
+ const uint8_t *segment;
+ volatile uint8_t *memory;
+ uint64_t size = 0, header = 0, answer = ~0ull;
+ uint32_t load = 1;
+
+ if (!cx_ready() || !core_is_on) {
+ return false;
+ }
+ PE_parse_boot_argn("adreno_zap", &load, sizeof(load));
+ if (!load) {
+ printf("adreno: %u: told that the zap shader is loaded already (adreno_zap=0)\n", step);
+ zap_is_loaded = true;
+ return true;
+ }
+ announce("find the zap shader's code, fw/a660_zap.b02, and look at the memory reserved for it (0x8b71a000: memory "
+ "outside what this kernel was given; read only)");
+ segment = image_file("fw", "a660_zap.b02", &size);
+ if (segment == NULL || size == 0 || size > 0x1000) {
+ return false;
+ }
+ memory = (volatile uint8_t *)ml_io_map_wcomb(ZAP_MEMORY, ZAP_MEMORY_SIZE);
+ printf("adreno: %u: the reserved memory begins %02x %02x %02x %02x, the zap shader %02x %02x %02x %02x\n", step,
+ memory[0], memory[1], memory[2], memory[3], segment[0], segment[1], segment[2], segment[3]);
+
+ announce("hand the zap shader's header to TrustZone, tell it where the image goes, put the image there (the first "
+ "write to that memory), and ask TrustZone to authenticate and run it");
+ if (!zap_header(&header)) {
+ return false;
+ }
+ if (scm("to take the zap shader's signature", SCM_64 | SCM_SIP | SCM_PAS_INIT_IMAGE, SCM_VALUE_AND_BUFFER, PAS_GPU,
+ header, 0, &answer) != 0 || answer != 0) {
+ printf("adreno: %u: TrustZone does not take the signature (%#llx)\n", step, answer);
+ return false;
+ }
+ if (scm("where the image will be", SCM_64 | SCM_SIP | SCM_PAS_MEM_SETUP, SCM_THREE_VALUES, PAS_GPU, ZAP_MEMORY, 0x1000,
+ &answer) != 0 || answer != 0) {
+ printf("adreno: %u: TrustZone does not take the place (%#llx)\n", step, answer);
+ return false;
+ }
+ printf("adreno: %u: about to write %llu bytes at %#llx\n", step, size, ZAP_MEMORY);
+ snprintf(last_write, sizeof(last_write), "step %u wrote the zap shader into memory at %#llx", step, ZAP_MEMORY);
+ hold(true);
+ for (uint64_t i = 0; i < 0x1000; i++) {
+ memory[i] = i < size ? segment[i] : 0;
+ }
+ settle();
+ if (scm("to authenticate the zap shader and run it", SCM_64 | SCM_SIP | SCM_PAS_AUTH_AND_RESET, SCM_ONE_VALUE, PAS_GPU, 0,
+ 0, &answer) != 0 || answer != 0) {
+ printf("adreno: %u: TrustZone does not authenticate the zap shader (%#llx)\n", step, answer);
+ return false;
+ }
+ printf("adreno: %u: TRUSTZONE HAS AUTHENTICATED AND STARTED THE ZAP SHADER\n", step);
+ zap_is_loaded = true;
+ return true;
+}
+
+/* ---- step 20: the 3D core's registers, its command processor started, its first command ------------------- */
+
+static bool
+step_core_start(void)
+{
+ static const uint32_t address_modes[] = { 0x0842, 0x0c01, 0x8601, 0x8e05, 0x9e01, 0xbe05, 0xa601, 0x9601, 0x0e00, 0xae01,
+ 0xb601, 0xf810 };
+ uint32_t blind = 0;
+
+ if (!cx_ready() || !core_is_on || !core_space_is_on) {
+ return false;
+ }
+ announce("write the 3D core's registers as Linux does before it starts the command processor: the first writes "
+ "to the core itself");
+ put(&gbif, "halt", GBIF_HALT, 0);
+ put(&gpu, "rbbm_gbif_halt", RBBM_GBIF_HALT, 0);
+ put(&gpu, "rbbm_secvid_tsb_cntl", RBBM_SECVID_TSB_CNTL, 0);
+ put64(&gpu, "rbbm_secvid_tsb_trusted_base", RBBM_SECVID_TSB_TRUSTED_BASE, 0);
+ put(&gpu, "rbbm_secvid_tsb_trusted_size", RBBM_SECVID_TSB_TRUSTED_SIZE, 0);
+ for (unsigned i = 0; i < sizeof(address_modes) / sizeof(address_modes[0]); i++) {
+ put(&gpu, "address mode: 64 bits", GPU_WORD(address_modes[i]), 1);
+ }
+ for (unsigned i = 0; i < 4; i++) {
+ put(&gbif, "qsb_side", GBIF_QSB_SIDE0 + 4 * i, 0x00071620);
+ }
+ put(&gpu, "rbbm_gbif_client_qos_cntl", RBBM_GBIF_CLIENT_QOS_CNTL, 3);
+ put(&gpu, "rbbm_perfctr_gpu_busy_masked", RBBM_PERFCTR_GPU_BUSY_MASKED, 0xffffffff);
+ put64(&gpu, "uche_write_range_max", UCHE_WRITE_RANGE_MAX, 0x0001ffffffffffc0ull);
+ put64(&gpu, "uche_trap_base", UCHE_TRAP_BASE, 0x0001fffffffff000ull);
+ put64(&gpu, "uche_write_thru_base", UCHE_WRITE_THRU_BASE, 0x0001fffffffff000ull);
+ put(&gpu, "uche_filter_cntl", UCHE_FILTER_CNTL, 0x804);
+ put(&gpu, "uche_cache_ways", UCHE_CACHE_WAYS, 0x4);
+ put(&gpu, "cp_roq_thresholds_2", CP_ROQ_THRESHOLDS_2, 0x02000140);
+ put(&gpu, "cp_roq_thresholds_1", CP_ROQ_THRESHOLDS_1, 0x8040362c);
+ put(&gpu, "cp_lpac_prog_fifo_size", CP_LPAC_PROG_FIFO_SIZE, 0x20);
+ put(&gpu, "cp_mem_pool_size", CP_MEM_POOL_SIZE, 128);
+ put(&gpu, "pc_dbg_eco_cntl", PC_DBG_ECO_CNTL, 0x00200200);
+ put(&gpu, "cp_ahb_cntl", CP_AHB_CNTL, 1);
+ put(&gpu, "rbbm_perfctr_cntl", RBBM_PERFCTR_CNTL, 1);
+ put(&gpu, "rbbm_interface_hang_int_cntl", RBBM_INTERFACE_HANG_INT_CNTL, 1u << 30 | 0x1fffff);
+ put(&gpu, "uche_client_pf", UCHE_CLIENT_PF, 1u << 7 | 1);
+ put(&gpu, "cp_chicken_dbg", CP_CHICKEN_DBG, 1);
+ put(&gpu, "rbbm_gbif_client_qos_cntl", RBBM_GBIF_CLIENT_QOS_CNTL, 0);
+ put(&gpu, "cp_apriv_cntl", CP_APRIV_CNTL, 1u << 6 | 1u << 5 | 1u << 3 | 1u << 2 | 1u << 1);
+ put(&gpu, "rb_dbg_eco_cntl", RB_DBG_ECO_CNTL, 0x04100000); /* Mesa's value for this GPU */
+ /*
+ * adreno_blind=1: the command processor is started with its microcode at an address where
+ * nothing is. All it can do then is ask the SMMU for it, and the SMMU's record of the refusal
+ * shows that its requests come through context 0 as meant, and what they look like.
+ */
+ PE_parse_boot_argn("adreno_blind", &blind, sizeof(blind));
+ put64(&gpu, "cp_sqe_instr_base", CP_SQE_INSTR_BASE, blind ? CORE_NOWHERE : CORE_SQE_ADDRESS);
+ put64(&gpu, "cp_rb_base", CP_RB_BASE, CORE_RING_ADDRESS);
+ put(&gpu, "cp_rb_cntl", CP_RB_CNTL, CP_RB_CNTL_VALUE);
+ get(&gpu, "cp_sqe_instr_base", CP_SQE_INSTR_BASE);
+ get(&gpu, "cp_rb_base", CP_RB_BASE);
+ get(&gpu, "cp_rb_cntl", CP_RB_CNTL);
+ get(&gpu, "cp_apriv_cntl", CP_APRIV_CNTL);
+ get(&gpu, "rbbm_status", RBBM_STATUS);
+
+ announce(blind ? "start the 3D core's command processor with its microcode nowhere" :
+ "start the 3D core's command processor");
+ core_runs = true;
+ put(&gpu, "cp_sqe_cntl", CP_SQE_CNTL, 1);
+ sleep_ms(20);
+ printf("adreno: %u: alive after it; status %#x, interrupts %#x, command processor: fault %#x, interrupts %#x, "
+ "protection %#x, ring read up to %u\n", step, rd(gpu.base + RBBM_STATUS), rd(gpu.base + RBBM_INT_0_STATUS),
+ rd(gpu.base + CP_HW_FAULT), rd(gpu.base + CP_INTERRUPT_STATUS), rd(gpu.base + CP_PROTECT_STATUS),
+ rd(gpu.base + CP_RB_RPTR));
+ get(&smmu, "sgfsr", SMMU_SGFSR);
+ get(&smmu, "sgfsynr0", SMMU_SGFSR + 8);
+ get(&smmu, "sgfsynr1", SMMU_SGFSR + 12);
+ if (blind) {
+ return space_faulted(&core_space);
+ }
+ if (second_space_is_on) {
+ sleep_ms(200);
+ return space_faulted(&second_space) | !space_faulted(&core_space);
+ }
+ if (space_faulted(&core_space)) {
+ return false;
+ }
+
+ announce("give the 3D core its first command, \"initialize\"");
+ ring_command(CP_ME_INIT, 8); /* a6xx_cp_init */
+ ring(0x0000002f);
+ ring(0x00000003); /* several hardware contexts */
+ ring(0x20000000); /* error detection */
+ ring(0);
+ ring(0);
+ ring(0);
+ ring(0);
+ ring(0);
+ if (!ring_run("initialize its command processor")) {
+ return false;
+ }
+ printf("adreno: %u: THE 3D CORE TAKES COMMANDS\n", step);
+ return true;
+}
+
+/* ---- step 21: out of secure mode; a word written on command ------------------------------------------------ */
+
+static bool
+step_core_commands(void)
+{
+ volatile uint32_t *scratch = (volatile uint32_t *)core_scratch.memory;
+
+ if (!cx_ready() || !core_runs) {
+ return false;
+ }
+ announce("command the 3D core out of secure mode (it asks the zap shader), then to write a word into a page of ours");
+ ring_command(CP_SET_SECURE_MODE, 1);
+ ring(0);
+ if (!ring_run("leave secure mode")) {
+ return false;
+ }
+ ring_command(CP_MEM_WRITE, 3);
+ ring(CORE_SCRATCH_ADDRESS);
+ ring(0);
+ ring(0x600dc0de);
+ if (!ring_run("write a word to memory")) {
+ return false;
+ }
+ to_memory(core_scratch.memory, 64);
+ printf("adreno: %u: the page reads %#x: %s\n", step, scratch[0], scratch[0] == 0x600dc0de ?
+ "THE 3D CORE WRITES OUR MEMORY ON COMMAND" : "not what it was told to write");
+ return scratch[0] == 0x600dc0de;
+}
+
+/* ---- step 22: a frame filled by the GPU, on the panel ------------------------------------------------------ */
+
+/*
+ * The simplest thing the 3D core draws: its 2D engine filling a rectangle with one colour (Mesa's
+ * fd6_blitter.cc: fd6_clear_surface; the registers under their older names, as in Linux's
+ * a6xx.xml), three times for three bands. The picture is 32-bit pixels in rows of a multiple of
+ * 64 bytes, in memory of ours; the framebuffer driver copies it to the panel.
+ */
+#define RB_CCU_CNTL 0x8e07 /* register numbers: these are written by command */
+#define GRAS_2D_BLIT_CNTL 0x8400
+#define GRAS_2D_DST_TL 0x8405
+#define GRAS_2D_DST_BR 0x8406
+#define RB_2D_BLIT_CNTL 0x8c00
+#define RB_2D_PIXEL_CNTL 0x8c01
+#define RB_2D_DST_INFO 0x8c17
+#define RB_2D_DST 0x8c18
+#define RB_2D_DST_PITCH 0x8c1a
+#define RB_2D_SRC_SOLID_C0 0x8c2c
+#define SP_2D_DST_FORMAT 0xacc0
+#define FORMAT_8888 0x30
+#define BLIT_SOLID_8888 (1u << 7 | FORMAT_8888 << 8 | 0xfu << 20 | 0x10u << 24) /* one colour; all channels; 8-bit */
+#define DST_BGRA_LINEAR (FORMAT_8888 | 1u << 10) /* rows of pixels, blue first in memory */
+#define EVENT_FLUSH_COLOR 0x1d
+#define EVENT_CACHE_FLUSH 0x04
+#define TARGET_WIDTH 1080
+#define TARGET_HEIGHT 2400
+#define TARGET_ROW 4352 /* 1088 pixels: the next multiple of 64 bytes */
+
+extern void MiniIOMobileFramebufferShow(const uint8_t *pixels, uint32_t width, uint32_t height, uint32_t row_bytes);
+
+static void
+ring_fill(uint32_t top, uint32_t bottom, uint32_t red, uint32_t green, uint32_t blue)
+{
+ ring_register(GRAS_2D_DST_TL, top << 16);
+ ring_register(GRAS_2D_DST_BR, (TARGET_WIDTH - 1) | (bottom - 1) << 16);
+ ring_register(RB_2D_SRC_SOLID_C0, red); /* a register for each part of the colour, 0 to 255 */
+ ring_register(RB_2D_SRC_SOLID_C0 + 1, green);
+ ring_register(RB_2D_SRC_SOLID_C0 + 2, blue);
+ ring_register(RB_2D_SRC_SOLID_C0 + 3, 255);
+ ring_register(RB_2D_BLIT_CNTL, BLIT_SOLID_8888);
+ ring_register(GRAS_2D_BLIT_CNTL, BLIT_SOLID_8888);
+ ring_register(SP_2D_DST_FORMAT, 1 | FORMAT_8888 << 3 | 0xfu << 12);
+ ring_register(RB_2D_PIXEL_CNTL, 0);
+ ring_register(RB_2D_DST_INFO, DST_BGRA_LINEAR);
+ ring_register(RB_2D_DST, CORE_TARGET_ADDRESS);
+ ring_register(RB_2D_DST + 1, 0);
+ ring_register(RB_2D_DST_PITCH, TARGET_ROW >> 6);
+ ring_command(CP_WAIT_FOR_IDLE, 0);
+ ring_command(CP_BLIT, 1);
+ ring(3); /* "scale": the 2D engine's one operation */
+}
+
+static bool
+step_frame(void)
+{
+ const volatile uint8_t *pixel;
+
+ if (!cx_ready() || !core_runs) {
+ return false;
+ }
+ announce("make a picture's worth of memory for the 3D core (10 MB) and have it fill three bands of colour");
+ if (!block_new(&core_space, &core_target, CORE_TARGET_ADDRESS, TARGET_ROW * TARGET_HEIGHT, PTE_CORE)) {
+ return false;
+ }
+ /* tables that the SMMU has in use have grown: it has not seen these addresses, so nothing stale to forget */
+ for (uint32_t i = 0; i < core_space.table_count; i++) {
+ to_memory((vm_offset_t)core_space.tables[i].table, 4096);
+ }
+ ring_command(CP_SET_MARKER, 1);
+ ring(12); /* "2D blits follow" */
+ ring_register(RB_CCU_CNTL, 0x10000004); /* Mesa's for this GPU when it draws straight into memory */
+ ring_fill(0, 800, 255, 128, 0); /* orange */
+ ring_fill(800, 1600, 255, 255, 255);
+ ring_fill(1600, TARGET_HEIGHT, 0, 128, 255);
+ /*
+ * What it has drawn out of its caches into memory. Each of the two events is followed by an
+ * address and a word that the core writes there when it is done (Linux's a6xx_submit; Mesa's
+ * tu6_emit_event_write: the events named ..._TS): the core takes those three words whether
+ * they are there or not.
+ */
+ ring_command(CP_EVENT_WRITE, 4);
+ ring(EVENT_FLUSH_COLOR);
+ ring(CORE_SCRATCH_ADDRESS + 32);
+ ring(0);
+ ring(0x0c0c0c0c);
+ ring_command(CP_EVENT_WRITE, 4);
+ ring(EVENT_CACHE_FLUSH);
+ ring(CORE_SCRATCH_ADDRESS + 16);
+ ring(0);
+ ring(0x0d0e0d0e);
+ ring_command(CP_WAIT_FOR_IDLE, 0);
+ if (!ring_run("fill a picture")) {
+ return false;
+ }
+ to_memory(core_scratch.memory, 64);
+ printf("adreno: %u: the words it writes when its caches are in memory: %#x, %#x (told: 0xc0c0c0c, 0xd0e0d0e)\n", step,
+ ((volatile uint32_t *)core_scratch.memory)[8], ((volatile uint32_t *)core_scratch.memory)[4]);
+ to_memory(core_target.memory, core_target.size);
+ pixel = (const volatile uint8_t *)core_target.memory;
+ printf("adreno: %u: the picture's first pixel is %02x %02x %02x %02x (blue green red alpha; told: 00 80 ff ff), one in "
+ "the middle %02x %02x %02x %02x, one in the last row %02x %02x %02x %02x\n", step, pixel[0], pixel[1], pixel[2],
+ pixel[3], pixel[1200 * TARGET_ROW + 2160], pixel[1200 * TARGET_ROW + 2161], pixel[1200 * TARGET_ROW + 2162],
+ pixel[1200 * TARGET_ROW + 2163], pixel[2399 * TARGET_ROW + 4316], pixel[2399 * TARGET_ROW + 4317],
+ pixel[2399 * TARGET_ROW + 4318], pixel[2399 * TARGET_ROW + 4319]);
+ if ((pixel[0] | pixel[1] | pixel[2]) == 0 || pixel[1200 * TARGET_ROW + 2160] == 0 || (pixel[2399 * TARGET_ROW + 4316] |
+ pixel[2399 * TARGET_ROW + 4317] | pixel[2399 * TARGET_ROW + 4318]) == 0) {
+ printf("adreno: %u: the picture is not what the 3D core was told to draw\n", step);
+ return false;
+ }
+ MiniIOMobileFramebufferShow((const uint8_t *)core_target.memory, TARGET_WIDTH, TARGET_HEIGHT, TARGET_ROW);
+ printf("adreno: %u: A PICTURE DRAWN BY THE GPU IS ON THE PANEL: orange, white, blue from the top\n", step);
+ return true;
+}
+
+/* ---- the way back ------------------------------------------------------------------------------------- */
+
+/*
+ * Nothing of the GPU block may outlive the kernel that started it: the memory its SMMU context
+ * points into is the next kernel's to use. In the order of Linux's a6xx_gmu_shutdown: the 3D
+ * core given back, the GMU told to drop its votes ("slumber") and stopped, its RPMh channel put
+ * to sleep; then the processor into reset and its stream sent nowhere.
+ */
+static void
+gpu_block_stop(void)
+{
+ uint32_t slumber[3] = { 0, 0, 0 };
+ bool power_off = false;
+
+ step = 0;
+ newest_step = ~0u;
+ if (core_is_on) {
+ put(&gmu, "host2gmu_intr_set", GMU_HOST2GMU_INTR_SET, 1u << 31);
+ core_is_on = false;
+ }
+ if (gmu_talks) {
+ hfi_send("\"slumber\"", HFI_PREPARE_SLUMBER, slumber, 3);
+ if (!wait_for(&gmu, GMU_AO_GPU_CX_BUSY_STATUS, 1u << 23, 0, 100)) {
+ get(&gmu, "ao_gpu_cx_busy_status", GMU_AO_GPU_CX_BUSY_STATUS);
+ }
+ get(&gmu, "sptprac_pwr_clk_status", GMU_SPTPRAC_PWR_CLK_STATUS);
+ get(&gpucc, "gx_gdscr", GPUCC_GX_GDSCR);
+ gmu_talks = false;
+ }
+ if (gmu_runs) {
+ put(&gmu, "cm3_sysreset", GMU_CM3_SYSRESET, 1);
+ gmu_runs = false;
+ }
+ /* vendor's a6xx_rscc_sleep_sequence: the GMU's votes in RPMh are dropped, by the always-on sequencer */
+ if (rsc_awake && rd(rscc.base + RSCC_SEQ_MEM_0_DRV0) == 0xeaaae5a0) {
+ put(&gmu, "rscc_control_req", GMU_RSCC_CONTROL_REQ, 1);
+ if (!wait_for(&rscc, RSCC_RSC_STATUS0_DRV0, RSCC_ASLEEP, RSCC_ASLEEP, 100)) {
+ printf("adreno: %u: the GMU's RPMh channel does not go to sleep\n", step);
+ }
+ get(&rscc, "rsc_status0_drv0", RSCC_RSC_STATUS0_DRV0);
+ put(&gmu, "rscc_control_req", GMU_RSCC_CONTROL_REQ, 0);
+ }
+ rsc_awake = false;
+ if (gmu_space_is_on) {
+ space_faulted(&gmu_space);
+ space_off(&gmu_space, GMU_MATCH);
+ gmu_space_is_on = false;
+ power_off = true;
+ }
+ if (second_space_is_on) {
+ space_faulted(&second_space);
+ space_off(&second_space, SECOND_MATCH);
+ second_space_is_on = false;
+ }
+ if (second_stream_is_on) {
+ put(&smmu, "smr", SMMU_SMR(SECOND_MATCH), 0);
+ put(&smmu, "s2cr", SMMU_S2CR(SECOND_MATCH), SMMU_S2CR_FAULT);
+ second_stream_is_on = false;
+ }
+ if (core_space_is_on) {
+ space_faulted(&core_space);
+ space_off(&core_space, CORE_MATCH);
+ core_space_is_on = core_runs = false;
+ }
+ /*
+ * And the CX half off again, as it was when the phone started: the GMU with all it remembers
+ * and the SMMU lose their power, and the next time starts from nothing (vendor's
+ * a6xx_gmu_suspend). The switch takes votes; ours is all this withdraws.
+ */
+ if (power_off && gpucc_answers()) {
+ put(&gpucc, "cx_gdscr", GPUCC_CX_GDSCR, rd(gpucc.base + GPUCC_CX_GDSCR) | GDSC_SW_COLLAPSE);
+ for (int i = 0; i < 1000 && cx_is_on(); i++) {
+ sleep_us(100);
+ }
+ get(&gpucc, "cx_gdscr", GPUCC_CX_GDSCR);
+ get(&gpucc, "cx_hw_ctrl", GPUCC_CX_HW_CTRL);
+ cx_off = !cx_is_on();
+ }
+}
+
+/*
+ * The same when there is no time for asking: the kernel is on its way back to the loader (the
+ * end of a run, a panic). Only the two writes that keep the GMU out of memory that is no longer
+ * its to use.
+ */
+void
+redwood_adreno_quiesce(void)
+{
+ if (cx_off) {
+ return;
+ }
+ if (gmu_runs) {
+ wr(gmu.base + GMU_CM3_SYSRESET, 1);
+ gmu_runs = false;
+ }
+ if (gmu_space_is_on) {
+ wr(smmu.base + smmu_context(&gmu_space) + SMMU_CB_SCTLR, 0);
+ gmu_space_is_on = false;
+ }
+ if (core_space_is_on) {
+ wr(smmu.base + smmu_context(&core_space) + SMMU_CB_SCTLR, 0);
+ core_space_is_on = false;
+ }
+ if (second_space_is_on) {
+ wr(smmu.base + smmu_context(&second_space) + SMMU_CB_SCTLR, 0);
+ second_space_is_on = false;
+ }
+}
+
+/* ---- all of it together ---------------------------------------------------------------------------- */
+
+static bool(*const steps[])(void) = {
+ step_outside, step_fuses, step_gpucc, step_cx_power, step_clocks, step_first_words, step_rscc, step_smmu,
+ step_smmu_context, step_gmu_memory, step_trustzone, step_zap_signature, step_gmu_space, step_gmu_firmware,
+ step_gmu_start, step_gmu_queues, step_core_power, step_core_space, step_zap_load, step_core_start, step_core_commands,
+ step_frame,
+};
+#define STEPS ((uint32_t)(sizeof(steps) / sizeof(steps[0])))
+
+/*
+ * Called while the system starts, before there is a root file system, and the start waits: for
+ * as long as the steps and the watch after them last, nothing else writes to the log, so that
+ * its end -- all that the loader's notes keep -- is about what is being done here.
+ */
+void
+redwood_adreno_init(void)
+{
+ uint32_t only = 0, done = 0;
+
+ PE_parse_boot_argn("adreno", &last_step, sizeof(last_step));
+ PE_parse_boot_argn("adreno_skip", &skip_steps, sizeof(skip_steps));
+ if (PE_parse_boot_argn("adreno_only", &only, sizeof(only)) && only > 0 && only <= STEPS) {
+ last_step = only;
+ skip_steps = ~(1u << only);
+ }
+ if (last_step == 0) {
+ return;
+ }
+ if (last_step > STEPS) {
+ last_step = STEPS;
+ }
+ PE_parse_boot_argn("adreno_pause", &pause_ms, sizeof(pause_ms));
+ PE_parse_boot_argn("adreno_fine", &fine, sizeof(fine));
+ PE_parse_boot_argn("adreno_watch", &watch_seconds, sizeof(watch_seconds));
+ for (newest_step = last_step; newest_step > 0 && (skip_steps & 1u << newest_step); newest_step--) {
+ }
+
+ printf("adreno: steps up to %u, without those in mask %#x; step %u is the new one and waits %u ms %s\n", last_step,
+ skip_steps, newest_step, pause_ms, fine ? "before every access" : "after each announcement");
+ for (step = 1; step <= last_step; step++) {
+ if (skip_steps & 1u << step) {
+ continue;
+ }
+ if (!steps[step - 1]()) {
+ printf("adreno: step %u: FAILED; the steps after it are not attempted\n", step);
+ break;
+ }
+ printf("adreno: step %u: done\n", step);
+ done = step;
+ }
+ /*
+ * Punishment for a write may come late (a reset some 17 seconds after it has been seen):
+ * keep the end of the log saying where things stand.
+ */
+ for (uint32_t second = 1; second <= watch_seconds; second++) {
+ sleep_ms(1000);
+ printf("adreno: alive %u s after step %u; %s\n", second, done, last_write);
+ }
+ gpu_block_stop();
+ if (PE_parse_boot_argn("adreno_return", &only, sizeof(only)) && only) {
+ printf("adreno: over; back to the loader (adreno_return)\n");
+ loader_return();
}
printf("adreno: over; the system goes on starting\n");
}
diff --git a/xnu/files/bsd/sys/redwood_input.h b/xnu/files/bsd/sys/redwood_input.h
index b865560..5855949 100644
--- a/xnu/files/bsd/sys/redwood_input.h
+++ b/xnu/files/bsd/sys/redwood_input.h
@@ -43,6 +43,7 @@ void redwood_virtio_input_init(void); /* boot-arg virtio_input=1: QEMU's virti
void redwood_goodix_init(void); /* boot-arg touchscreen=1: the phone's touch screen */
void redwood_cpufreq_init(void); /* boot-arg cpufreq=1: the phone's processor at full speed */
void redwood_adreno_init(void); /* boot-arg adreno=N: the first N steps of waking the phone's GPU */
+void redwood_adreno_quiesce(void); /* on the way back to the loader: the GPU block stopped, out of our memory */
#endif
#endif /* _SYS_REDWOOD_INPUT_H_ */
diff --git a/xnu/files/iokit/Kernel/MiniIOMobileFramebuffer.cpp b/xnu/files/iokit/Kernel/MiniIOMobileFramebuffer.cpp
index 2876605..e829074 100644
--- a/xnu/files/iokit/Kernel/MiniIOMobileFramebuffer.cpp
+++ b/xnu/files/iokit/Kernel/MiniIOMobileFramebuffer.cpp
@@ -889,6 +889,25 @@ publishAccelerator(IOService * provider)
OSSafeReleaseNULL(accelerator);
}
+/*
+ * A picture from inside the kernel (the GPU driver's first ones): rows of 32-bit pixels, put at
+ * the top left corner of the panel as they are and cut to its size.
+ */
+extern "C" void
+MiniIOMobileFramebufferShow(const uint8_t * pixels, uint32_t width, uint32_t height, uint32_t rowBytes)
+{
+ uint32_t columns = width < vinfo.v_width ? width : (uint32_t) vinfo.v_width;
+ uint32_t rows = height < vinfo.v_height ? height : (uint32_t) vinfo.v_height;
+
+ if (vinfo.v_baseaddr == 0 || vinfo.v_depth != 32) {
+ return;
+ }
+ for (uint32_t y = 0; y < rows; y++) {
+ memcpy((void *) (vinfo.v_baseaddr + (size_t) y * vinfo.v_rowbytes), pixels + (size_t) y * rowBytes,
+ (size_t) columns * 4);
+ }
+}
+
void
MiniIOMobileFramebufferStart(IOService * provider)
{
diff --git a/xnu/files/osfmk/arm64/loader_return.c b/xnu/files/osfmk/arm64/loader_return.c
index ed871e7..a2810f2 100644
--- a/xnu/files/osfmk/arm64/loader_return.c
+++ b/xnu/files/osfmk/arm64/loader_return.c
@@ -43,6 +43,7 @@ extern void gic_v3_irq_quiesce(void);
void loader_log_putc(char c);
void loader_return(void);
+extern void redwood_adreno_quiesce(void);
void loader_checkpoint(uint64_t n);
/* Also referenced from the early exception vector in start.s. */
@@ -299,6 +300,7 @@ loader_return(void)
*/
psci_smp_stop_others();
#endif
+ redwood_adreno_quiesce(); /* the GPU block, if it was started, has tables in memory that is about to be reused */
/*
* A panic's text is first collected in the panic buffer and only later
* flushed to the console; if the console itself is what failed, that