Skip to content

Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.

C

air: a module function that takes a texture, via splitting translate_globals

vcvkkc2fc71ad190b
diff --git a/docs/air.md b/docs/air.md
index eb63994..545495b 100644
--- a/docs/air.md
+++ b/docs/air.md
@@ -20,8 +20,8 @@ Of the 456 modules of the device's system `.metallib` files (`tools/air/check.sh
| | of 456 |
| --- | --- |
| read by our own bitcode reader, byte for byte as `llvm-dis` reads them | 456 |
-| translated to NIR, passing `nir_validate` | 315 |
-| compiled by ir3 into Adreno machine code | 315 |
+| translated to NIR, passing `nir_validate` | 318 |
+| compiled by ir3 into Adreno machine code | 318 |
| numerically identical to LLVM's own execution, and drawn | 255 |
| **run on the phone's GPU** | **7** |
@@ -41,7 +41,7 @@ And of the 119 distinct `air.*` operations those modules call (`tools/air/intrin
| texture operations with a case softpipe cannot run: a 16-bit result | 7 |
| texture operations with no case: multisample, depth, write | 11 |
-315 of 456 is 315 of 321 entry points — 98.1% — because 135 modules are `air.visible` library
+318 of 456 is 318 of 321 entry points — 99.1% — because 135 modules are `air.visible` library
functions and not entry points at all.
**What those numbers are not**, because they will be quoted without this paragraph otherwise:
@@ -309,52 +309,35 @@ functions and not entry points at all.
they are evidence about is binding, descriptors and ir3's texture instructions — exactly what
they were built to be — and silence about everything Metal's sampler state decides. Two modules
of the corpus declare a `constexpr sampler`; everywhere else the API supplies it.
-- **Four modules call a function that takes a texture, and that is attempted and not done.**
- `air.convert`-style arguments pass through a `nir_call` happily; a texture does not, because
- this frontend resolves a texture at translation time into a binding and `tex_handle` wants an
- `AIR_VAL_PTR`, not an SSA def. The attempt, kept here so it is not repeated from the start:
-
- - binding the pointer-class arguments from the caller's own resolution and passing only the
- rest as NIR parameters **works** — "argument 0 is not a texture" goes away, and one of the
- four, `FilmGrainChromaKernel_Uint`, gets past the call to a different limit (a shuffle mask
- that is not constant);
- - the module's own values have to be carried into the callee's fresh value table rather than
- re-registered, because `translate_globals` is not idempotent: it appends to
- `info->resources` and adds to `shared_size`, so a second run invents an inline sampler and
- doubles the threadgroup memory. Copying the `module_values` set is the right carry;
- - and then three of the four **crash**, in `nir_lower_goto_ifs` → `nir_repair_ssa` →
- `nir_block_is_unreachable`, on an assertion that dominance metadata is valid. Marking the
- hand-built callee impl's metadata invalid (`nir_metadata_preserve(impl, nir_metadata_none)`)
- does not fix it, and the obvious hypothesis — that the callee has unreachable blocks, which
- is what that predicate is about — is **wrong**: these callees have no `unreachable`
- terminator at all.
-
- **The diagnosis is now complete, and it is the second bullet above that causes the third.** A
- peer session read `nir_dominance.c` and pointed out what the assertion actually says:
- `nir_block_is_unreachable` asks *the block's own* `nir_function_impl` for its dominance
- metadata, and `nir_repair_ssa` requires that metadata only on the impl it was called for. So a
- use sitting in another function's block makes it ask the wrong function — nothing to do with a
- block being unreachable, which is why that hypothesis failed.
-
- Their proposed test, `nir_validate_shader` before `nir_lower_goto_ifs`, did not settle it and
- taught something anyway: it fails identically on `fbs_slice_trilinear`, which translates
- perfectly. Its complaints are about block indices in an unstructured impl, which nothing indexes
- before `nir_lower_goto_ifs` — pre-existing and benign, and they masked the real thing.
-
- The carrier is the `module_values` copy. A probe printing any copied value that holds an
- `nir_def` found five in `FilmGrainLumaKernel_Uint` alone, of both kinds: a pointer with a byte
- offset and a plain SSA value. Those defs were created in the *entry point's* impl by
- `translate_globals`, so copying the values into a callee leaves the callee using the caller's
- values — exactly the cross-function def/use the peer described, arriving through the globals
- rather than through an argument. An argument cannot carry one: a guard refusing a pointer
- argument with an offset never fired, so every pointer argument here has none.
-
- **So the fix is known and is not a third guess**: `translate_globals` must be split, the part
- that assigns bindings and appends to `info->resources` running once, and the part that
- materialises `nir_def`s running per impl, so each callee builds its own. That is a change to
- shared code that handles six kinds of global, and it is not started. The state above is the
- committed one; each of the four modules has exactly one call site, so specialising per site
- costs nothing in duplication once this is done.
+- **A function of the module that takes a texture: done, and the diagnosis was the work.** Three
+ of the four such entry points now translate; the fourth stops on the non-constant shuffle mask
+ it would have stopped on anyway. The fix is a split of `translate_globals`, and arriving at it
+ took two reverts and three failed guesses, which is the part worth keeping.
+
+ A texture is not a value in this frontend: it is resolved at translation time into a binding,
+ and `tex_handle` wants an `AIR_VAL_PTR` rather than an SSA def. So a pointer-class argument is
+ bound from the caller's own resolution and takes no NIR parameter, which specialises the callee
+ to its call site — free here, since each of the four has exactly one. That much worked at once.
+
+ What did not was everything after. The callee needs the module's globals, and copying the entry
+ point's values into it crashed three of the four in `nir_lower_goto_ifs` → `nir_repair_ssa` →
+ `nir_block_is_unreachable`. **Two hypotheses died with evidence before the right one arrived**:
+ the callee has no unreachable block at all, and `nir_validate_shader` — the test a peer session
+ proposed for it — fails identically on `fbs_slice_trilinear`, which translates perfectly, so
+ its complaint about block indices in an unstructured impl was true and about something else.
+ The peer's reading of the assertion was right and their route was not: it asks *the block's own*
+ impl for dominance metadata, so a use in another function's block asks the wrong function — but
+ the cross-function value arrived through the globals, not through an argument. A guard refusing
+ a pointer argument with a byte offset was evaluated 26 times over the corpus and never fired, so
+ arguments are not the carrier; the count is the evidence, because a silent guard and an
+ unreachable guard look the same.
+
+ So `translate_globals` is now two. `plan_globals` decides what each global is and runs **once**,
+ because it appends inline samplers to `info->resources` and lays out shared and scratch memory.
+ `materialise_globals` builds the values and runs **per function**, because a byte offset is an
+ `nir_def` and a def belongs to the impl that made it. Module-level constants needed neither:
+ `air_get` already materialises one in whichever builder asks, which was checked rather than
+ assumed.
- **Of Apple's own 47 kernels, 6 are dispatched, all on software.** `tools/air/kernelcheck.py`
prints where each one stops, as a table whose rows are checked against the total before it is
@@ -364,9 +347,14 @@ functions and not entry points at all.
| --- | --- |
| dispatched and read back | 6 |
| refused by this harness | 0 |
- | softpipe's `nir_to_tgsi`: scratch and intrinsics it lacks (ir3 has these) | 34 |
- | calls an `air.visible` library function in the module | 5 |
- | a shuffle mask that is not constant | 2 |
+ | softpipe's `nir_to_tgsi`: scratch and intrinsics it lacks (ir3 has these) | 38 |
+ | calls a function the module defines | 0 |
+ | a shuffle mask that is not constant | 3 |
+
+ The row for a call of a module-local function was 5 until those calls were translated; it is now
+ 0, and its five went to the two rows below it — four to softpipe and one to the shuffle mask. The
+ dispatched count did not move, which is the same shape as before: the wall is no longer ours and
+ the number behind it is unchanged.
The 6 is the honest number and not the interesting one. Until 20 October, 24 of these were
refused by `air_run` itself — it bound textures and images for a draw and not for a dispatch —
diff --git a/tools/mesa/air/air_private.h b/tools/mesa/air/air_private.h
index 61ad621..e8760dd 100644
--- a/tools/mesa/air/air_private.h
+++ b/tools/mesa/air/air_private.h
@@ -85,6 +85,24 @@ struct air_ctx {
struct hash_table *fns;
nir_deref_instr *ret_deref;
+ /* What each module global resolves to, decided once (plan_globals) and
+ * turned into a value in whichever function needs it (materialise_globals).
+ * The two are separate because the deciding half has side effects that must
+ * happen once -- it appends inline samplers to info->resources and lays out
+ * shared and scratch memory -- while the making half emits an nir_def for
+ * the byte offset, and a def belongs to the impl that created it. Copying
+ * the entry point's values into a callee instead left the callee using the
+ * caller's defs, which is a cross-function def/use: nir_repair_ssa then
+ * asks the wrong function for its dominance metadata and asserts.
+ */
+ struct air_global_plan {
+ enum air_ptr_class class;
+ unsigned binding;
+ unsigned offset; /* bytes, when has_offset */
+ bool has_offset;
+ bool skip; /* llvm.* lists, and anything that failed */
+ } *global_plan;
+
/* what the entry point's parameters are */
unsigned num_params;
struct air_val **params;
diff --git a/tools/mesa/air/air_to_nir.c b/tools/mesa/air/air_to_nir.c
index eeb53a9..9011bcf 100644
--- a/tools/mesa/air/air_to_nir.c
+++ b/tools/mesa/air/air_to_nir.c
@@ -1325,6 +1325,7 @@ reverse_post_order(struct air_ctx *c, const struct air_function *f, unsigned *or
}
static bool translate_body(struct air_ctx *c);
+static void materialise_globals(struct air_ctx *c);
/* A function of the module, translated as a nir_function so that
* nir_inline_functions can fold it into its callers afterwards.
@@ -1343,9 +1344,33 @@ static bool translate_body(struct air_ctx *c);
* entries in place would let a callee resolve a value it cannot see.
*/
static nir_function *
-translate_callee(struct air_ctx *c, const struct air_function *f)
+translate_callee(struct air_ctx *c, const struct air_function *f,
+ struct air_val **argv, bool *is_param)
{
- struct hash_entry *e = _mesa_hash_table_search(c->fns, f);
+ /* A texture, sampler or buffer argument is not a value in this frontend: it
+ * is resolved at translation time into a binding, and tex_handle wants an
+ * AIR_VAL_PTR rather than an SSA def. Such an argument is bound from the
+ * caller's own resolution and takes no NIR parameter, which specialises the
+ * callee to its call site -- so it is not cached. Each of the four modules
+ * that needs this has exactly one call site, counted beforehand.
+ *
+ * A pointer carrying a byte offset is refused instead: that offset is an
+ * nir_def of the CALLER's impl, and using it here would be the same
+ * cross-function def/use that the globals used to cause.
+ */
+ bool specialised = false;
+ for (unsigned i = 0; i < f->num_args; i++) {
+ bool ptr = argv && argv[i] && argv[i]->kind == AIR_VAL_PTR;
+ if (ptr && argv[i]->ptr.offset) {
+ air_error(c, "intrinsic: a call passing a pointer at a computed offset as "
+ "argument %u, which would use the caller's own value here", i);
+ return NULL;
+ }
+ is_param[i] = !ptr;
+ specialised |= ptr;
+ }
+
+ struct hash_entry *e = specialised ? NULL : _mesa_hash_table_search(c->fns, f);
if (e)
return e->data;
@@ -1378,16 +1403,22 @@ translate_callee(struct air_ctx *c, const struct air_function *f)
};
}
for (unsigned i = 0; i < f->num_args; i++) {
+ if (!is_param[i])
+ continue;
const struct air_type *at = f->args[i].type;
fn->params[idx++] = (nir_parameter){
.num_components = (uint8_t)air_nir_components(at),
.bit_size = (uint8_t)air_nir_bit_size(c, at),
};
}
+ fn->num_params = idx; /* the pointers took no parameter */
/* cached before the body is translated, so that a callee which calls itself
* meets its own declaration rather than being translated for ever */
- _mesa_hash_table_insert(c->fns, f, fn);
+ if (!specialised)
+ _mesa_hash_table_insert(c->fns, f, fn);
+ else
+ _mesa_hash_table_insert(c->fns, fn, fn);
nir_builder saved_b = c->b;
const struct air_function *saved_fn = c->fn;
@@ -1408,6 +1439,9 @@ translate_callee(struct air_ctx *c, const struct air_function *f)
c->b.impl->structured = false;
c->ret_deref = NULL;
+ /* this function's own globals, built in its own builder */
+ materialise_globals(c);
+
bool ok = true;
if (has_ret) {
/* The caller passes the deref's own def and the callee casts it back,
@@ -1418,9 +1452,13 @@ translate_callee(struct air_ctx *c, const struct air_function *f)
c->ret_deref = nir_build_deref_cast(&c->b, nir_load_param(&c->b, 0),
nir_var_function_temp, ret_glsl, 0);
}
+ unsigned pidx = has_ret ? 1 : 0;
for (unsigned i = 0; i < f->num_args && ok; i++) {
- nir_def *p = nir_load_param(&c->b, i + (has_ret ? 1 : 0));
- air_set(c, &f->args[i], air_val_ssa(c, p));
+ if (!is_param[i]) {
+ air_set(c, &f->args[i], argv[i]); /* the caller's own resolution */
+ continue;
+ }
+ air_set(c, &f->args[i], air_val_ssa(c, nir_load_param(&c->b, pidx++)));
}
if (ok)
ok = translate_body(c);
@@ -1460,15 +1498,26 @@ air_call_in_module(struct air_ctx *c, const struct air_inst *in, const char *nam
return air_error(c, "intrinsic: a call of %s, which is not a function of this "
"module", name);
- nir_function *fn = translate_callee(c, f);
+ if (in->num_operands - 1 != f->num_args)
+ return air_error(c, "intrinsic: a call of %s with %u arguments, wanting %u", name,
+ in->num_operands - 1, f->num_args);
+
+ /* resolved in the CALLER's value map, which is the live one here */
+ struct air_val **argv = rzalloc_array(c->mem_ctx, struct air_val *,
+ MAX2(f->num_args, 1));
+ bool *is_param = rzalloc_array(c->mem_ctx, bool, MAX2(f->num_args, 1));
+ for (unsigned i = 0; i < f->num_args; i++) {
+ argv[i] = air_get(c, in->operands[i + 1]);
+ if (!argv[i])
+ return false;
+ }
+
+ nir_function *fn = translate_callee(c, f, argv, is_param);
if (!fn)
return false; /* translate_callee has said why */
const struct air_type *rt = f->fn_type->elem; /* FUNCTION: the return type */
bool has_ret = rt && rt->kind != AIR_TYPE_VOID;
- if (in->num_operands - 1 != f->num_args)
- return air_error(c, "intrinsic: a call of %s with %u arguments, wanting %u", name,
- in->num_operands - 1, f->num_args);
nir_variable *tmp = NULL;
nir_call_instr *call = nir_call_instr_create(c->nir, fn);
@@ -1484,6 +1533,8 @@ air_call_in_module(struct air_ctx *c, const struct air_inst *in, const char *nam
}
/* operand 0 is the callee; the arguments follow it */
for (unsigned i = 0; i < f->num_args; i++) {
+ if (!is_param[i])
+ continue;
nir_def *a = air_get_ssa(c, in->operands[i + 1]);
if (!a)
return air_error(c, "intrinsic: argument %u of %s is not a value this can pass",
@@ -1650,20 +1701,28 @@ function_constant_index(struct air_ctx *c, const struct air_global *g)
return -1;
}
+/* Decide what every global is: once per shader, because this appends inline
+ * samplers to info->resources and lays out the shared and scratch memory. It
+ * records rather than builds -- making the value needs a builder, and which
+ * function's builder matters (see materialise_globals).
+ */
static void
-translate_globals(struct air_ctx *c)
+plan_globals(struct air_ctx *c)
{
+ c->global_plan = rzalloc_array(c->mem_ctx, struct air_global_plan,
+ MAX2(c->m->num_globals, 1));
for (unsigned i = 0; i < c->m->num_globals; i++) {
+ struct air_global_plan *pl = &c->global_plan[i];
const struct air_global *g = c->m->globals[i];
const char *name = g->base.name ? g->base.name : "";
/* llvm.compiler.used and llvm.global_ctors are lists for the linker,
* not memory the shader reads
*/
- if (!strncmp(name, "llvm.", 5))
+ if (!strncmp(name, "llvm.", 5)) {
+ pl->skip = true;
continue;
-
- struct air_ptr p = { .pointee = g->value_type };
+ }
if (g->externally_initialized) {
int idx = function_constant_index(c, g);
@@ -1672,8 +1731,8 @@ translate_globals(struct air_ctx *c)
"function constant");
return;
}
- p.class = AIR_PTR_FUNCTION_CONSTANT;
- p.binding = idx;
+ pl->class = AIR_PTR_FUNCTION_CONSTANT;
+ pl->binding = idx;
} else if (!strncmp(name, "__air_sampler_state", 19)) {
/* A sampler the shader declares itself. It needs a real index, not a
* sentinel: the Adreno's texture instruction holds the sampler in a
@@ -1685,8 +1744,8 @@ translate_globals(struct air_ctx *c)
for (unsigned k = 0; k < c->info->num_resources; k++)
if (c->info->resources[k].kind == AIR_RESOURCE_SAMPLER)
next = MAX2(next, c->info->resources[k].index + 1);
- p.class = AIR_PTR_SAMPLER;
- p.binding = next;
+ pl->class = AIR_PTR_SAMPLER;
+ pl->binding = next;
uint64_t word = 0;
if (g->init && !air_const_uint(g->init, &word)) {
@@ -1705,8 +1764,9 @@ translate_globals(struct air_ctx *c)
/* a static threadgroup array */
unsigned align = MAX2(g->align, air_type_abi_align(c->m, g->value_type));
c->shared_size = ALIGN_POT(c->shared_size, MAX2(align, 1));
- p.class = AIR_PTR_SHARED;
- p.offset = nir_imm_int(&c->b, c->shared_size);
+ pl->class = AIR_PTR_SHARED;
+ pl->offset = c->shared_size;
+ pl->has_offset = true;
c->shared_size += air_type_alloc_size(c->m, g->value_type);
} else {
/* a program-scope variable: private to the invocation, written by
@@ -1714,10 +1774,34 @@ translate_globals(struct air_ctx *c)
*/
unsigned align = MAX2(g->align, air_type_abi_align(c->m, g->value_type));
c->scratch_size = ALIGN_POT(c->scratch_size, MAX2(align, 1));
- p.class = AIR_PTR_SCRATCH;
- p.offset = nir_imm_int(&c->b, c->scratch_size);
+ pl->class = AIR_PTR_SCRATCH;
+ pl->offset = c->scratch_size;
+ pl->has_offset = true;
c->scratch_size += air_type_alloc_size(c->m, g->value_type);
}
+ }
+}
+
+/* Build the globals as values of the function being translated now. Called for
+ * the entry point and again for every callee, because the byte offset is an
+ * nir_def and a def belongs to one impl: handing a callee the entry point's
+ * defs is what crashed nir_repair_ssa. Module-level constants need no such
+ * care -- air_get materialises one on demand in whatever builder asks.
+ */
+static void
+materialise_globals(struct air_ctx *c)
+{
+ for (unsigned i = 0; i < c->m->num_globals; i++) {
+ const struct air_global_plan *pl = &c->global_plan[i];
+ if (pl->skip)
+ continue;
+ const struct air_global *g = c->m->globals[i];
+ struct air_ptr p = {
+ .pointee = g->value_type,
+ .class = pl->class,
+ .binding = pl->binding,
+ .offset = pl->has_offset ? nir_imm_int(&c->b, pl->offset) : NULL,
+ };
air_set(c, &g->base, air_val_ptr(c, p));
}
}
@@ -1873,7 +1957,9 @@ air_to_nir(void *mem_ctx, const void *bitcode, size_t size, const char *entry_po
}
if (!c.error[0])
- translate_globals(&c);
+ plan_globals(&c);
+ if (!c.error[0])
+ materialise_globals(&c);
if (!c.error[0])
run_static_initializers(&c);