Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.
C
air: a module function that takes a texture, via splitting translate_globals
vcvkkc2fc71ad190b
diff --git a/docs/air.md b/docs/air.mdindex eb63994..545495b 100644--- a/docs/air.md+++ b/docs/air.md@@ -20,8 +20,8 @@ Of the 456 modules of the device's system `.metallib` files (`tools/air/check.sh| | of 456 || --- | --- || read by our own bitcode reader, byte for byte as `llvm-dis` reads them | 456 |-| translated to NIR, passing `nir_validate` | 315 |-| compiled by ir3 into Adreno machine code | 315 |+| translated to NIR, passing `nir_validate` | 318 |+| compiled by ir3 into Adreno machine code | 318 || numerically identical to LLVM's own execution, and drawn | 255 || **run on the phone's GPU** | **7** |@@ -41,7 +41,7 @@ And of the 119 distinct `air.*` operations those modules call (`tools/air/intrin| texture operations with a case softpipe cannot run: a 16-bit result | 7 || texture operations with no case: multisample, depth, write | 11 |-315 of 456 is 315 of 321 entry points — 98.1% — because 135 modules are `air.visible` library+318 of 456 is 318 of 321 entry points — 99.1% — because 135 modules are `air.visible` libraryfunctions and not entry points at all.**What those numbers are not**, because they will be quoted without this paragraph otherwise:@@ -309,52 +309,35 @@ functions and not entry points at all.they are evidence about is binding, descriptors and ir3's texture instructions — exactly whatthey were built to be — and silence about everything Metal's sampler state decides. Two modulesof the corpus declare a `constexpr sampler`; everywhere else the API supplies it.-- **Four modules call a function that takes a texture, and that is attempted and not done.**- `air.convert`-style arguments pass through a `nir_call` happily; a texture does not, because- this frontend resolves a texture at translation time into a binding and `tex_handle` wants an- `AIR_VAL_PTR`, not an SSA def. The attempt, kept here so it is not repeated from the start:-- - binding the pointer-class arguments from the caller's own resolution and passing only the- rest as NIR parameters **works** — "argument 0 is not a texture" goes away, and one of the- four, `FilmGrainChromaKernel_Uint`, gets past the call to a different limit (a shuffle mask- that is not constant);- - the module's own values have to be carried into the callee's fresh value table rather than- re-registered, because `translate_globals` is not idempotent: it appends to- `info->resources` and adds to `shared_size`, so a second run invents an inline sampler and- doubles the threadgroup memory. Copying the `module_values` set is the right carry;- - and then three of the four **crash**, in `nir_lower_goto_ifs` → `nir_repair_ssa` →- `nir_block_is_unreachable`, on an assertion that dominance metadata is valid. Marking the- hand-built callee impl's metadata invalid (`nir_metadata_preserve(impl, nir_metadata_none)`)- does not fix it, and the obvious hypothesis — that the callee has unreachable blocks, which- is what that predicate is about — is **wrong**: these callees have no `unreachable`- terminator at all.-- **The diagnosis is now complete, and it is the second bullet above that causes the third.** A- peer session read `nir_dominance.c` and pointed out what the assertion actually says:- `nir_block_is_unreachable` asks *the block's own* `nir_function_impl` for its dominance- metadata, and `nir_repair_ssa` requires that metadata only on the impl it was called for. So a- use sitting in another function's block makes it ask the wrong function — nothing to do with a- block being unreachable, which is why that hypothesis failed.-- Their proposed test, `nir_validate_shader` before `nir_lower_goto_ifs`, did not settle it and- taught something anyway: it fails identically on `fbs_slice_trilinear`, which translates- perfectly. Its complaints are about block indices in an unstructured impl, which nothing indexes- before `nir_lower_goto_ifs` — pre-existing and benign, and they masked the real thing.-- The carrier is the `module_values` copy. A probe printing any copied value that holds an- `nir_def` found five in `FilmGrainLumaKernel_Uint` alone, of both kinds: a pointer with a byte- offset and a plain SSA value. Those defs were created in the *entry point's* impl by- `translate_globals`, so copying the values into a callee leaves the callee using the caller's- values — exactly the cross-function def/use the peer described, arriving through the globals- rather than through an argument. An argument cannot carry one: a guard refusing a pointer- argument with an offset never fired, so every pointer argument here has none.-- **So the fix is known and is not a third guess**: `translate_globals` must be split, the part- that assigns bindings and appends to `info->resources` running once, and the part that- materialises `nir_def`s running per impl, so each callee builds its own. That is a change to- shared code that handles six kinds of global, and it is not started. The state above is the- committed one; each of the four modules has exactly one call site, so specialising per site- costs nothing in duplication once this is done.+- **A function of the module that takes a texture: done, and the diagnosis was the work.** Three+ of the four such entry points now translate; the fourth stops on the non-constant shuffle mask+ it would have stopped on anyway. The fix is a split of `translate_globals`, and arriving at it+ took two reverts and three failed guesses, which is the part worth keeping.++ A texture is not a value in this frontend: it is resolved at translation time into a binding,+ and `tex_handle` wants an `AIR_VAL_PTR` rather than an SSA def. So a pointer-class argument is+ bound from the caller's own resolution and takes no NIR parameter, which specialises the callee+ to its call site — free here, since each of the four has exactly one. That much worked at once.++ What did not was everything after. The callee needs the module's globals, and copying the entry+ point's values into it crashed three of the four in `nir_lower_goto_ifs` → `nir_repair_ssa` →+ `nir_block_is_unreachable`. **Two hypotheses died with evidence before the right one arrived**:+ the callee has no unreachable block at all, and `nir_validate_shader` — the test a peer session+ proposed for it — fails identically on `fbs_slice_trilinear`, which translates perfectly, so+ its complaint about block indices in an unstructured impl was true and about something else.+ The peer's reading of the assertion was right and their route was not: it asks *the block's own*+ impl for dominance metadata, so a use in another function's block asks the wrong function — but+ the cross-function value arrived through the globals, not through an argument. A guard refusing+ a pointer argument with a byte offset was evaluated 26 times over the corpus and never fired, so+ arguments are not the carrier; the count is the evidence, because a silent guard and an+ unreachable guard look the same.++ So `translate_globals` is now two. `plan_globals` decides what each global is and runs **once**,+ because it appends inline samplers to `info->resources` and lays out shared and scratch memory.+ `materialise_globals` builds the values and runs **per function**, because a byte offset is an+ `nir_def` and a def belongs to the impl that made it. Module-level constants needed neither:+ `air_get` already materialises one in whichever builder asks, which was checked rather than+ assumed.- **Of Apple's own 47 kernels, 6 are dispatched, all on software.** `tools/air/kernelcheck.py`prints where each one stops, as a table whose rows are checked against the total before it is@@ -364,9 +347,14 @@ functions and not entry points at all.| --- | --- || dispatched and read back | 6 || refused by this harness | 0 |- | softpipe's `nir_to_tgsi`: scratch and intrinsics it lacks (ir3 has these) | 34 |- | calls an `air.visible` library function in the module | 5 |- | a shuffle mask that is not constant | 2 |+ | softpipe's `nir_to_tgsi`: scratch and intrinsics it lacks (ir3 has these) | 38 |+ | calls a function the module defines | 0 |+ | a shuffle mask that is not constant | 3 |++ The row for a call of a module-local function was 5 until those calls were translated; it is now+ 0, and its five went to the two rows below it — four to softpipe and one to the shuffle mask. The+ dispatched count did not move, which is the same shape as before: the wall is no longer ours and+ the number behind it is unchanged.The 6 is the honest number and not the interesting one. Until 20 October, 24 of these wererefused by `air_run` itself — it bound textures and images for a draw and not for a dispatch —diff --git a/tools/mesa/air/air_private.h b/tools/mesa/air/air_private.hindex 61ad621..e8760dd 100644--- a/tools/mesa/air/air_private.h+++ b/tools/mesa/air/air_private.h@@ -85,6 +85,24 @@ struct air_ctx {struct hash_table *fns;nir_deref_instr *ret_deref;+ /* What each module global resolves to, decided once (plan_globals) and+ * turned into a value in whichever function needs it (materialise_globals).+ * The two are separate because the deciding half has side effects that must+ * happen once -- it appends inline samplers to info->resources and lays out+ * shared and scratch memory -- while the making half emits an nir_def for+ * the byte offset, and a def belongs to the impl that created it. Copying+ * the entry point's values into a callee instead left the callee using the+ * caller's defs, which is a cross-function def/use: nir_repair_ssa then+ * asks the wrong function for its dominance metadata and asserts.+ */+ struct air_global_plan {+ enum air_ptr_class class;+ unsigned binding;+ unsigned offset; /* bytes, when has_offset */+ bool has_offset;+ bool skip; /* llvm.* lists, and anything that failed */+ } *global_plan;+/* what the entry point's parameters are */unsigned num_params;struct air_val **params;diff --git a/tools/mesa/air/air_to_nir.c b/tools/mesa/air/air_to_nir.cindex eeb53a9..9011bcf 100644--- a/tools/mesa/air/air_to_nir.c+++ b/tools/mesa/air/air_to_nir.c@@ -1325,6 +1325,7 @@ reverse_post_order(struct air_ctx *c, const struct air_function *f, unsigned *or}static bool translate_body(struct air_ctx *c);+static void materialise_globals(struct air_ctx *c);/* A function of the module, translated as a nir_function so that* nir_inline_functions can fold it into its callers afterwards.@@ -1343,9 +1344,33 @@ static bool translate_body(struct air_ctx *c);* entries in place would let a callee resolve a value it cannot see.*/static nir_function *-translate_callee(struct air_ctx *c, const struct air_function *f)+translate_callee(struct air_ctx *c, const struct air_function *f,+ struct air_val **argv, bool *is_param){- struct hash_entry *e = _mesa_hash_table_search(c->fns, f);+ /* A texture, sampler or buffer argument is not a value in this frontend: it+ * is resolved at translation time into a binding, and tex_handle wants an+ * AIR_VAL_PTR rather than an SSA def. Such an argument is bound from the+ * caller's own resolution and takes no NIR parameter, which specialises the+ * callee to its call site -- so it is not cached. Each of the four modules+ * that needs this has exactly one call site, counted beforehand.+ *+ * A pointer carrying a byte offset is refused instead: that offset is an+ * nir_def of the CALLER's impl, and using it here would be the same+ * cross-function def/use that the globals used to cause.+ */+ bool specialised = false;+ for (unsigned i = 0; i < f->num_args; i++) {+ bool ptr = argv && argv[i] && argv[i]->kind == AIR_VAL_PTR;+ if (ptr && argv[i]->ptr.offset) {+ air_error(c, "intrinsic: a call passing a pointer at a computed offset as "+ "argument %u, which would use the caller's own value here", i);+ return NULL;+ }+ is_param[i] = !ptr;+ specialised |= ptr;+ }++ struct hash_entry *e = specialised ? NULL : _mesa_hash_table_search(c->fns, f);if (e)return e->data;@@ -1378,16 +1403,22 @@ translate_callee(struct air_ctx *c, const struct air_function *f)};}for (unsigned i = 0; i < f->num_args; i++) {+ if (!is_param[i])+ continue;const struct air_type *at = f->args[i].type;fn->params[idx++] = (nir_parameter){.num_components = (uint8_t)air_nir_components(at),.bit_size = (uint8_t)air_nir_bit_size(c, at),};}+ fn->num_params = idx; /* the pointers took no parameter *//* cached before the body is translated, so that a callee which calls itself* meets its own declaration rather than being translated for ever */- _mesa_hash_table_insert(c->fns, f, fn);+ if (!specialised)+ _mesa_hash_table_insert(c->fns, f, fn);+ else+ _mesa_hash_table_insert(c->fns, fn, fn);nir_builder saved_b = c->b;const struct air_function *saved_fn = c->fn;@@ -1408,6 +1439,9 @@ translate_callee(struct air_ctx *c, const struct air_function *f)c->b.impl->structured = false;c->ret_deref = NULL;+ /* this function's own globals, built in its own builder */+ materialise_globals(c);+bool ok = true;if (has_ret) {/* The caller passes the deref's own def and the callee casts it back,@@ -1418,9 +1452,13 @@ translate_callee(struct air_ctx *c, const struct air_function *f)c->ret_deref = nir_build_deref_cast(&c->b, nir_load_param(&c->b, 0),nir_var_function_temp, ret_glsl, 0);}+ unsigned pidx = has_ret ? 1 : 0;for (unsigned i = 0; i < f->num_args && ok; i++) {- nir_def *p = nir_load_param(&c->b, i + (has_ret ? 1 : 0));- air_set(c, &f->args[i], air_val_ssa(c, p));+ if (!is_param[i]) {+ air_set(c, &f->args[i], argv[i]); /* the caller's own resolution */+ continue;+ }+ air_set(c, &f->args[i], air_val_ssa(c, nir_load_param(&c->b, pidx++)));}if (ok)ok = translate_body(c);@@ -1460,15 +1498,26 @@ air_call_in_module(struct air_ctx *c, const struct air_inst *in, const char *namreturn air_error(c, "intrinsic: a call of %s, which is not a function of this ""module", name);- nir_function *fn = translate_callee(c, f);+ if (in->num_operands - 1 != f->num_args)+ return air_error(c, "intrinsic: a call of %s with %u arguments, wanting %u", name,+ in->num_operands - 1, f->num_args);++ /* resolved in the CALLER's value map, which is the live one here */+ struct air_val **argv = rzalloc_array(c->mem_ctx, struct air_val *,+ MAX2(f->num_args, 1));+ bool *is_param = rzalloc_array(c->mem_ctx, bool, MAX2(f->num_args, 1));+ for (unsigned i = 0; i < f->num_args; i++) {+ argv[i] = air_get(c, in->operands[i + 1]);+ if (!argv[i])+ return false;+ }++ nir_function *fn = translate_callee(c, f, argv, is_param);if (!fn)return false; /* translate_callee has said why */const struct air_type *rt = f->fn_type->elem; /* FUNCTION: the return type */bool has_ret = rt && rt->kind != AIR_TYPE_VOID;- if (in->num_operands - 1 != f->num_args)- return air_error(c, "intrinsic: a call of %s with %u arguments, wanting %u", name,- in->num_operands - 1, f->num_args);nir_variable *tmp = NULL;nir_call_instr *call = nir_call_instr_create(c->nir, fn);@@ -1484,6 +1533,8 @@ air_call_in_module(struct air_ctx *c, const struct air_inst *in, const char *nam}/* operand 0 is the callee; the arguments follow it */for (unsigned i = 0; i < f->num_args; i++) {+ if (!is_param[i])+ continue;nir_def *a = air_get_ssa(c, in->operands[i + 1]);if (!a)return air_error(c, "intrinsic: argument %u of %s is not a value this can pass",@@ -1650,20 +1701,28 @@ function_constant_index(struct air_ctx *c, const struct air_global *g)return -1;}+/* Decide what every global is: once per shader, because this appends inline+ * samplers to info->resources and lays out the shared and scratch memory. It+ * records rather than builds -- making the value needs a builder, and which+ * function's builder matters (see materialise_globals).+ */static void-translate_globals(struct air_ctx *c)+plan_globals(struct air_ctx *c){+ c->global_plan = rzalloc_array(c->mem_ctx, struct air_global_plan,+ MAX2(c->m->num_globals, 1));for (unsigned i = 0; i < c->m->num_globals; i++) {+ struct air_global_plan *pl = &c->global_plan[i];const struct air_global *g = c->m->globals[i];const char *name = g->base.name ? g->base.name : "";/* llvm.compiler.used and llvm.global_ctors are lists for the linker,* not memory the shader reads*/- if (!strncmp(name, "llvm.", 5))+ if (!strncmp(name, "llvm.", 5)) {+ pl->skip = true;continue;-- struct air_ptr p = { .pointee = g->value_type };+ }if (g->externally_initialized) {int idx = function_constant_index(c, g);@@ -1672,8 +1731,8 @@ translate_globals(struct air_ctx *c)"function constant");return;}- p.class = AIR_PTR_FUNCTION_CONSTANT;- p.binding = idx;+ pl->class = AIR_PTR_FUNCTION_CONSTANT;+ pl->binding = idx;} else if (!strncmp(name, "__air_sampler_state", 19)) {/* A sampler the shader declares itself. It needs a real index, not a* sentinel: the Adreno's texture instruction holds the sampler in a@@ -1685,8 +1744,8 @@ translate_globals(struct air_ctx *c)for (unsigned k = 0; k < c->info->num_resources; k++)if (c->info->resources[k].kind == AIR_RESOURCE_SAMPLER)next = MAX2(next, c->info->resources[k].index + 1);- p.class = AIR_PTR_SAMPLER;- p.binding = next;+ pl->class = AIR_PTR_SAMPLER;+ pl->binding = next;uint64_t word = 0;if (g->init && !air_const_uint(g->init, &word)) {@@ -1705,8 +1764,9 @@ translate_globals(struct air_ctx *c)/* a static threadgroup array */unsigned align = MAX2(g->align, air_type_abi_align(c->m, g->value_type));c->shared_size = ALIGN_POT(c->shared_size, MAX2(align, 1));- p.class = AIR_PTR_SHARED;- p.offset = nir_imm_int(&c->b, c->shared_size);+ pl->class = AIR_PTR_SHARED;+ pl->offset = c->shared_size;+ pl->has_offset = true;c->shared_size += air_type_alloc_size(c->m, g->value_type);} else {/* a program-scope variable: private to the invocation, written by@@ -1714,10 +1774,34 @@ translate_globals(struct air_ctx *c)*/unsigned align = MAX2(g->align, air_type_abi_align(c->m, g->value_type));c->scratch_size = ALIGN_POT(c->scratch_size, MAX2(align, 1));- p.class = AIR_PTR_SCRATCH;- p.offset = nir_imm_int(&c->b, c->scratch_size);+ pl->class = AIR_PTR_SCRATCH;+ pl->offset = c->scratch_size;+ pl->has_offset = true;c->scratch_size += air_type_alloc_size(c->m, g->value_type);}+ }+}++/* Build the globals as values of the function being translated now. Called for+ * the entry point and again for every callee, because the byte offset is an+ * nir_def and a def belongs to one impl: handing a callee the entry point's+ * defs is what crashed nir_repair_ssa. Module-level constants need no such+ * care -- air_get materialises one on demand in whatever builder asks.+ */+static void+materialise_globals(struct air_ctx *c)+{+ for (unsigned i = 0; i < c->m->num_globals; i++) {+ const struct air_global_plan *pl = &c->global_plan[i];+ if (pl->skip)+ continue;+ const struct air_global *g = c->m->globals[i];+ struct air_ptr p = {+ .pointee = g->value_type,+ .class = pl->class,+ .binding = pl->binding,+ .offset = pl->has_offset ? nir_imm_int(&c->b, pl->offset) : NULL,+ };air_set(c, &g->base, air_val_ptr(c, p));}}@@ -1873,7 +1957,9 @@ air_to_nir(void *mem_ctx, const void *bitcode, size_t size, const char *entry_po}if (!c.error[0])- translate_globals(&c);+ plan_globals(&c);+ if (!c.error[0])+ materialise_globals(&c);if (!c.error[0])run_static_initializers(&c);