Skip to content

Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.

C

Apple's own compiler settles the two guesses, and says one of them was wrong

vcvkk8b6e9d67712d
diff --git a/docs/air.md b/docs/air.md
index 53a238a..38fde65 100644
--- a/docs/air.md
+++ b/docs/air.md
@@ -209,6 +209,58 @@ LLVM parameter type, and the NIR value is cut or converted to it.
| `fast` flags on an instruction | the NIR ALU is not `exact`; without them it is |
| `air.<op>.<type suffix>` | one handler per op in `air_intrinsics.c`; the suffix is checked against the LLVM types, not trusted |
+### 3.2 The vocabulary, read off Apple's own compiler
+
+What each `air.*` name means was a guess until this was read, and one of the guesses was wrong.
+`AGXCompilerCore` — the back end of Apple's own shader compiler, in **this device's** shared
+cache (`ios165/20F66__iPhone10,3_6`, never another device's) — carries its whole operation
+vocabulary as a run of C strings at file offsets `0x16cf9f`–`0x16d400`, which is virtual
+`0x1bbf43f9f`–`0x1bbf44400` (`__cstring` begins at file 1 482 768 for `0x1bbf41010`). 137 names,
+and they are suffixes: the full function is `air.` + name + `.` + a type suffix, which is why
+grepping the library for `air.fast_exp` finds nothing.
+
+```
+.add .sub .or .xor .and .min .max global local .s .u .max.u.i64 .min.u.i64 .xchg.i32
+.cmpxchg.weak.i32 .store.i32 .load.i32 global.{add,sub,xchg,cmpxchg.weak,store,load}.f32
+acos asin atan atan2 acosh asinh atanh pow powr cosh sinh tanh exp10 log10 cos sin tan sincos
+rint round fmod ceil exp exp2 fabs floor fma fmax fmin fract ldexp log log2 mad rsqrt sqrt trunc
+fast_{acos asin atan atan2 exp10 log10 acosh asinh atanh pow powr cosh sinh tanh cos sin tan
+ sincos round fmod mod exp exp2 ldexp log log2 rsqrt sqrt ceil fabs floor fract rint trunc
+ fmax fmin clamp saturate}
+gamma_expand gamma_compress gamma_expand_xr gamma_compress_xr convert pack unpack
+abs abs_diff add_sat sub_sat hadd rhadd clamp clz ctz mad_hi mad_sat mul_hi rotate popcount
+extract_bits insert_bits reverse_bits any all any_msb all_msb saturate mix sign dot
+dfdx dfdy fwidth atomic assert get_simd_width get_thread_execution_width
+```
+
+**This settles the two operations whose meaning was a guess, and corrects one of them.** The list
+distinguishes exactly the functions the Metal Shading Language Specification defines, one name per
+function:
+
+| the question | what the list says | what we had |
+| --- | --- | --- |
+| is `air.fast_exp` the natural `exp` or `exp2`? | `exp` and `exp2` are separate names, and so are `fast_exp` and `fast_exp2` | right: natural `exp` |
+| does `air.fast_round` take a half away from zero (`round`) or to even (`rint`)? | `round` and `rint` are separate, **and so are `fast_round` and `fast_rint`** | **wrong**: it was `nir_fround_even`, which is `rint` |
+
+`air.fast_round` is now `trunc(x)`, stepped one away from zero when `|x - trunc(x)| >= 0.5`.
+Not `trunc(x + copysign(0.5, x))`, which is the obvious formula and is wrong: for an `x` just
+below a half that addition itself rounds up and the answer comes out 1 where it should be 0.
+The subtraction is exact for every float, so the version used needs no magnitude guard and keeps
+the sign of a negative zero; it was checked bit-exact against C's `roundf` over the halves, the
+sub-halves, both zeros and 2^24.
+
+Reading the list also closed gaps that had not been noticed: `exp10`, `log10`, `powr`, `fmod`,
+`mod`, `mad`, `abs`, `abs_diff`, `add_sat`, `sub_sat`, `mul_hi`, `popcount`, `clz`, `ctz`,
+`rotate`, `reverse_bits`, `extract_bits`, `insert_bits`, `dfdx`, `dfdy`, `fwidth`,
+`get_simd_width` and `rint` itself are all implemented now. None of them occurs in the 456-module
+corpus, so none is tested by anything here; they are in because the list says they exist and
+guessing at them later would be the same mistake again.
+
+Still unimplemented, and now known by name rather than by surprise: the inverse and hyperbolic
+trigonometry (`acos`, `asin`, `atan`, `acosh`, `asinh`, `atanh`, `cosh`, `sinh`, `tanh`),
+`sincos` (two results from one call), `hadd`, `rhadd`, `mad_hi`, `mad_sat`, `any_msb`/`all_msb`,
+the `gamma_*` conversions, `pack`/`unpack`, and `assert`.
+
## 4. How it is proven right
A shader is reported as *working* only when its output was compared, never because it compiled.
@@ -244,6 +296,13 @@ Three references, each independent of the translator:
implementations would encode the same beliefs as the translator, and a reference that agrees
with you by construction is not a reference.
+ There is one oracle for the intrinsics that needs no hardware and is not circular, and
+ section 3.2 is it: **Apple's own compiler, which knows what each name means because it
+ compiles it.** Its vocabulary settled the two guesses and found one of them wrong. What that
+ oracle gives is each operation's *identity*, not its numerical result, so it decides
+ `round` against `rint` and `exp` against `exp2` and cannot decide how many ULP Apple's
+ `fast_sin` is allowed to be out by. For that, M4's buffer readback is still the test.
+
What it cannot decide, and says so rather than passing: a shader with a non-buffer argument —
a varying, `air.position`, a thread id — because what is in it is the GPU's business (19 in
AppleMetalGLRenderer, all of them the texture ones), and a compute kernel, which returns
diff --git a/tools/mesa/air/air_intrinsics.c b/tools/mesa/air/air_intrinsics.c
index 10bdbda..b6d680c 100644
--- a/tools/mesa/air/air_intrinsics.c
+++ b/tools/mesa/air/air_intrinsics.c
@@ -10,10 +10,13 @@
* for anything: the types come from the call's own LLVM types, and the suffix
* is only what picks the handler.
*
- * Where Metal's definition of an operation is not certain from its name --
- * air.fast_exp could be exp or exp2 -- the choice is marked UNVERIFIED and
- * has to be settled against the CPU reference (docs/air.md section 4) before
- * any shader using it is called correct.
+ * What each name means is not guessed: it is read off Apple's own compiler.
+ * AGXCompilerCore carries the whole vocabulary as a table of names at
+ * 0x16d082..0x16d400 of the library in this device's shared cache (the list is
+ * in docs/air.md section 3.2), and that table distinguishes exactly the
+ * functions the Metal Shading Language Specification defines -- round from
+ * rint, exp from exp2, fmod from mod, pow from powr. Two of this file's
+ * handlers were guesses before that table was read; one of them was wrong.
*/
#include "air_private.h"
@@ -100,8 +103,24 @@ math_op(struct air_ctx *c, const struct air_inst *in, const struct air_name *n,
} else if (!strcmp(op, "fast_ceil") || !strcmp(op, "ceil")) {
r = nir_fceil(&c->b, a[0]);
} else if (!strcmp(op, "fast_round") || !strcmp(op, "round")) {
- /* Metal's round() goes away from zero at .5; rint() is the even one */
- r = nir_fround_even(&c->b, a[0]); /* UNVERIFIED: round vs rint */
+ /* round() takes a half away from zero; rint() takes it to even, and
+ * Apple's compiler has both air.fast_round and air.fast_rint, so this
+ * one is not the even one. (It was nir_fround_even here until the
+ * vocabulary table said otherwise.)
+ *
+ * Truncate, then step one away from zero if the remainder reaches a
+ * half. Not trunc(x + copysign(0.5, x)): that addition rounds for an x
+ * just under a half and answers 1 where the answer is 0. x - trunc(x)
+ * is exact for every float, so this needs no magnitude guard, and it
+ * keeps the sign of a negative zero. Checked bit-exact against C's
+ * roundf over the halves, the sub-halves, both zeros and 2^24.
+ */
+ nir_def *t = nir_ftrunc(&c->b, a[0]);
+ nir_def *d = nir_fabs(&c->b, nir_fsub(&c->b, a[0], t));
+ r = nir_bcsel(&c->b, nir_fge_imm(&c->b, d, 0.5),
+ nir_fadd(&c->b, t, nir_fsign(&c->b, a[0])), t);
+ } else if (!strcmp(op, "fast_rint") || !strcmp(op, "rint")) {
+ r = nir_fround_even(&c->b, a[0]);
} else if (!strcmp(op, "fast_trunc") || !strcmp(op, "trunc")) {
r = nir_ftrunc(&c->b, a[0]);
} else if (!strcmp(op, "fast_fract") || !strcmp(op, "fract")) {
@@ -115,18 +134,25 @@ math_op(struct air_ctx *c, const struct air_inst *in, const struct air_name *n,
} else if (!strcmp(op, "fast_atan2") || !strcmp(op, "atan2")) {
r = nir_atan2(&c->b, a[0], a[1]);
} else if (!strcmp(op, "fast_exp") || !strcmp(op, "exp")) {
- /* UNVERIFIED: exp or exp2. Taken as the natural exp, which is what
- * Metal's exp() is; exp2 has its own name in the language.
- */
+ /* the natural one: Apple's compiler names exp and exp2 separately */
r = nir_fexp2(&c->b, nir_fmul_imm(&c->b, a[0], M_LOG2E));
} else if (!strcmp(op, "fast_exp2") || !strcmp(op, "exp2")) {
r = nir_fexp2(&c->b, a[0]);
+ } else if (!strcmp(op, "fast_exp10") || !strcmp(op, "exp10")) {
+ r = nir_fexp2(&c->b, nir_fmul_imm(&c->b, a[0], 3.321928094887362)); /* log2(10) */
} else if (!strcmp(op, "fast_log") || !strcmp(op, "log")) {
r = nir_fmul_imm(&c->b, nir_flog2(&c->b, a[0]), 1.0 / M_LOG2E);
} else if (!strcmp(op, "fast_log2") || !strcmp(op, "log2")) {
r = nir_flog2(&c->b, a[0]);
- } else if (!strcmp(op, "fast_pow") || !strcmp(op, "pow")) {
+ } else if (!strcmp(op, "fast_log10") || !strcmp(op, "log10")) {
+ r = nir_fmul_imm(&c->b, nir_flog2(&c->b, a[0]), 0.30102999566398120); /* 1/log2(10) */
+ } else if (!strcmp(op, "fast_pow") || !strcmp(op, "pow") ||
+ !strcmp(op, "fast_powr") || !strcmp(op, "powr")) {
+ /* powr is pow with x >= 0 promised, which is what nir_fpow computes */
r = nir_fpow(&c->b, a[0], a[1]);
+ } else if (!strcmp(op, "fast_fmod") || !strcmp(op, "fmod") ||
+ !strcmp(op, "fast_mod") || !strcmp(op, "mod")) {
+ r = nir_frem(&c->b, a[0], a[1]);
} else if (!strcmp(op, "fast_divide") || !strcmp(op, "divide")) {
r = nir_fdiv(&c->b, a[0], a[1]);
} else if (!strcmp(op, "saturate") || !strcmp(op, "fast_saturate")) {
@@ -154,8 +180,50 @@ math_op(struct air_ctx *c, const struct air_inst *in, const struct air_name *n,
r = nir_umin(&c->b, nir_umax(&c->b, a[0], a[1]), a[2]);
} else if (!strcmp(op, "fast_ldexp") || !strcmp(op, "ldexp")) {
r = nir_ldexp(&c->b, a[0], nir_i2i32(&c->b, a[1]));
- } else if (!strcmp(op, "fast_fma") || !strcmp(op, "fma")) {
+ } else if (!strcmp(op, "fast_fma") || !strcmp(op, "fma") || !strcmp(op, "mad")) {
r = nir_ffma(&c->b, a[0], a[1], a[2]);
+ } else if (!strcmp(op, "abs")) {
+ r = is_float ? nir_fabs(&c->b, a[0]) : nir_iabs(&c->b, a[0]);
+ } else if (!strcmp(op, "abs_diff")) {
+ r = is_signed ? nir_iabs(&c->b, nir_isub(&c->b, a[0], a[1]))
+ : nir_isub(&c->b, nir_umax(&c->b, a[0], a[1]),
+ nir_umin(&c->b, a[0], a[1]));
+ } else if (!strcmp(op, "add_sat")) {
+ r = is_signed ? nir_iadd_sat(&c->b, a[0], a[1]) : nir_uadd_sat(&c->b, a[0], a[1]);
+ } else if (!strcmp(op, "sub_sat")) {
+ r = is_signed ? nir_isub_sat(&c->b, a[0], a[1]) : nir_usub_sat(&c->b, a[0], a[1]);
+ } else if (!strcmp(op, "mul_hi")) {
+ r = is_signed ? nir_imul_high(&c->b, a[0], a[1]) : nir_umul_high(&c->b, a[0], a[1]);
+ } else if (!strcmp(op, "popcount")) {
+ r = nir_bit_count(&c->b, a[0]);
+ } else if (!strcmp(op, "clz")) {
+ r = nir_uclz(&c->b, a[0]);
+ } else if (!strcmp(op, "ctz")) {
+ /* Metal's ctz returns the type's width when the value is zero */
+ nir_def *lsb = nir_find_lsb(&c->b, a[0]);
+ r = nir_bcsel(&c->b, nir_ieq_imm(&c->b, a[0], 0),
+ nir_imm_intN_t(&c->b, a[0]->bit_size, a[0]->bit_size), lsb);
+ } else if (!strcmp(op, "reverse_bits")) {
+ r = nir_bitfield_reverse(&c->b, a[0]);
+ } else if (!strcmp(op, "rotate")) {
+ r = nir_urol(&c->b, a[0], nir_u2u32(&c->b, a[1]));
+ } else if (!strcmp(op, "extract_bits")) {
+ r = is_signed ? nir_ibitfield_extract(&c->b, a[0], nir_u2u32(&c->b, a[1]),
+ nir_u2u32(&c->b, a[2]))
+ : nir_ubitfield_extract(&c->b, a[0], nir_u2u32(&c->b, a[1]),
+ nir_u2u32(&c->b, a[2]));
+ } else if (!strcmp(op, "insert_bits")) {
+ r = nir_bitfield_insert(&c->b, a[0], a[1], nir_u2u32(&c->b, a[2]),
+ nir_u2u32(&c->b, a[3]));
+ } else if (!strcmp(op, "dfdx")) {
+ r = nir_ddx(&c->b, a[0]);
+ } else if (!strcmp(op, "dfdy")) {
+ r = nir_ddy(&c->b, a[0]);
+ } else if (!strcmp(op, "fwidth")) {
+ r = nir_fadd(&c->b, nir_fabs(&c->b, nir_ddx(&c->b, a[0])),
+ nir_fabs(&c->b, nir_ddy(&c->b, a[0])));
+ } else if (!strcmp(op, "get_simd_width") || !strcmp(op, "get_thread_execution_width")) {
+ r = nir_load_subgroup_size(&c->b);
} else if (!strcmp(op, "all")) {
r = nir_ball(&c->b, a[0]);
} else if (!strcmp(op, "any")) {