Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.
C
Apple's own compiler settles the two guesses, and says one of them was wrong
vcvkk8b6e9d67712d
diff --git a/docs/air.md b/docs/air.mdindex 53a238a..38fde65 100644--- a/docs/air.md+++ b/docs/air.md@@ -209,6 +209,58 @@ LLVM parameter type, and the NIR value is cut or converted to it.| `fast` flags on an instruction | the NIR ALU is not `exact`; without them it is || `air.<op>.<type suffix>` | one handler per op in `air_intrinsics.c`; the suffix is checked against the LLVM types, not trusted |+### 3.2 The vocabulary, read off Apple's own compiler++What each `air.*` name means was a guess until this was read, and one of the guesses was wrong.+`AGXCompilerCore` — the back end of Apple's own shader compiler, in **this device's** shared+cache (`ios165/20F66__iPhone10,3_6`, never another device's) — carries its whole operation+vocabulary as a run of C strings at file offsets `0x16cf9f`–`0x16d400`, which is virtual+`0x1bbf43f9f`–`0x1bbf44400` (`__cstring` begins at file 1 482 768 for `0x1bbf41010`). 137 names,+and they are suffixes: the full function is `air.` + name + `.` + a type suffix, which is why+grepping the library for `air.fast_exp` finds nothing.++```+.add .sub .or .xor .and .min .max global local .s .u .max.u.i64 .min.u.i64 .xchg.i32+.cmpxchg.weak.i32 .store.i32 .load.i32 global.{add,sub,xchg,cmpxchg.weak,store,load}.f32+acos asin atan atan2 acosh asinh atanh pow powr cosh sinh tanh exp10 log10 cos sin tan sincos+rint round fmod ceil exp exp2 fabs floor fma fmax fmin fract ldexp log log2 mad rsqrt sqrt trunc+fast_{acos asin atan atan2 exp10 log10 acosh asinh atanh pow powr cosh sinh tanh cos sin tan+ sincos round fmod mod exp exp2 ldexp log log2 rsqrt sqrt ceil fabs floor fract rint trunc+ fmax fmin clamp saturate}+gamma_expand gamma_compress gamma_expand_xr gamma_compress_xr convert pack unpack+abs abs_diff add_sat sub_sat hadd rhadd clamp clz ctz mad_hi mad_sat mul_hi rotate popcount+extract_bits insert_bits reverse_bits any all any_msb all_msb saturate mix sign dot+dfdx dfdy fwidth atomic assert get_simd_width get_thread_execution_width+```++**This settles the two operations whose meaning was a guess, and corrects one of them.** The list+distinguishes exactly the functions the Metal Shading Language Specification defines, one name per+function:++| the question | what the list says | what we had |+| --- | --- | --- |+| is `air.fast_exp` the natural `exp` or `exp2`? | `exp` and `exp2` are separate names, and so are `fast_exp` and `fast_exp2` | right: natural `exp` |+| does `air.fast_round` take a half away from zero (`round`) or to even (`rint`)? | `round` and `rint` are separate, **and so are `fast_round` and `fast_rint`** | **wrong**: it was `nir_fround_even`, which is `rint` |++`air.fast_round` is now `trunc(x)`, stepped one away from zero when `|x - trunc(x)| >= 0.5`.+Not `trunc(x + copysign(0.5, x))`, which is the obvious formula and is wrong: for an `x` just+below a half that addition itself rounds up and the answer comes out 1 where it should be 0.+The subtraction is exact for every float, so the version used needs no magnitude guard and keeps+the sign of a negative zero; it was checked bit-exact against C's `roundf` over the halves, the+sub-halves, both zeros and 2^24.++Reading the list also closed gaps that had not been noticed: `exp10`, `log10`, `powr`, `fmod`,+`mod`, `mad`, `abs`, `abs_diff`, `add_sat`, `sub_sat`, `mul_hi`, `popcount`, `clz`, `ctz`,+`rotate`, `reverse_bits`, `extract_bits`, `insert_bits`, `dfdx`, `dfdy`, `fwidth`,+`get_simd_width` and `rint` itself are all implemented now. None of them occurs in the 456-module+corpus, so none is tested by anything here; they are in because the list says they exist and+guessing at them later would be the same mistake again.++Still unimplemented, and now known by name rather than by surprise: the inverse and hyperbolic+trigonometry (`acos`, `asin`, `atan`, `acosh`, `asinh`, `atanh`, `cosh`, `sinh`, `tanh`),+`sincos` (two results from one call), `hadd`, `rhadd`, `mad_hi`, `mad_sat`, `any_msb`/`all_msb`,+the `gamma_*` conversions, `pack`/`unpack`, and `assert`.+## 4. How it is proven rightA shader is reported as *working* only when its output was compared, never because it compiled.@@ -244,6 +296,13 @@ Three references, each independent of the translator:implementations would encode the same beliefs as the translator, and a reference that agreeswith you by construction is not a reference.+ There is one oracle for the intrinsics that needs no hardware and is not circular, and+ section 3.2 is it: **Apple's own compiler, which knows what each name means because it+ compiles it.** Its vocabulary settled the two guesses and found one of them wrong. What that+ oracle gives is each operation's *identity*, not its numerical result, so it decides+ `round` against `rint` and `exp` against `exp2` and cannot decide how many ULP Apple's+ `fast_sin` is allowed to be out by. For that, M4's buffer readback is still the test.+What it cannot decide, and says so rather than passing: a shader with a non-buffer argument —a varying, `air.position`, a thread id — because what is in it is the GPU's business (19 inAppleMetalGLRenderer, all of them the texture ones), and a compute kernel, which returnsdiff --git a/tools/mesa/air/air_intrinsics.c b/tools/mesa/air/air_intrinsics.cindex 10bdbda..b6d680c 100644--- a/tools/mesa/air/air_intrinsics.c+++ b/tools/mesa/air/air_intrinsics.c@@ -10,10 +10,13 @@* for anything: the types come from the call's own LLVM types, and the suffix* is only what picks the handler.*- * Where Metal's definition of an operation is not certain from its name --- * air.fast_exp could be exp or exp2 -- the choice is marked UNVERIFIED and- * has to be settled against the CPU reference (docs/air.md section 4) before- * any shader using it is called correct.+ * What each name means is not guessed: it is read off Apple's own compiler.+ * AGXCompilerCore carries the whole vocabulary as a table of names at+ * 0x16d082..0x16d400 of the library in this device's shared cache (the list is+ * in docs/air.md section 3.2), and that table distinguishes exactly the+ * functions the Metal Shading Language Specification defines -- round from+ * rint, exp from exp2, fmod from mod, pow from powr. Two of this file's+ * handlers were guesses before that table was read; one of them was wrong.*/#include "air_private.h"@@ -100,8 +103,24 @@ math_op(struct air_ctx *c, const struct air_inst *in, const struct air_name *n,} else if (!strcmp(op, "fast_ceil") || !strcmp(op, "ceil")) {r = nir_fceil(&c->b, a[0]);} else if (!strcmp(op, "fast_round") || !strcmp(op, "round")) {- /* Metal's round() goes away from zero at .5; rint() is the even one */- r = nir_fround_even(&c->b, a[0]); /* UNVERIFIED: round vs rint */+ /* round() takes a half away from zero; rint() takes it to even, and+ * Apple's compiler has both air.fast_round and air.fast_rint, so this+ * one is not the even one. (It was nir_fround_even here until the+ * vocabulary table said otherwise.)+ *+ * Truncate, then step one away from zero if the remainder reaches a+ * half. Not trunc(x + copysign(0.5, x)): that addition rounds for an x+ * just under a half and answers 1 where the answer is 0. x - trunc(x)+ * is exact for every float, so this needs no magnitude guard, and it+ * keeps the sign of a negative zero. Checked bit-exact against C's+ * roundf over the halves, the sub-halves, both zeros and 2^24.+ */+ nir_def *t = nir_ftrunc(&c->b, a[0]);+ nir_def *d = nir_fabs(&c->b, nir_fsub(&c->b, a[0], t));+ r = nir_bcsel(&c->b, nir_fge_imm(&c->b, d, 0.5),+ nir_fadd(&c->b, t, nir_fsign(&c->b, a[0])), t);+ } else if (!strcmp(op, "fast_rint") || !strcmp(op, "rint")) {+ r = nir_fround_even(&c->b, a[0]);} else if (!strcmp(op, "fast_trunc") || !strcmp(op, "trunc")) {r = nir_ftrunc(&c->b, a[0]);} else if (!strcmp(op, "fast_fract") || !strcmp(op, "fract")) {@@ -115,18 +134,25 @@ math_op(struct air_ctx *c, const struct air_inst *in, const struct air_name *n,} else if (!strcmp(op, "fast_atan2") || !strcmp(op, "atan2")) {r = nir_atan2(&c->b, a[0], a[1]);} else if (!strcmp(op, "fast_exp") || !strcmp(op, "exp")) {- /* UNVERIFIED: exp or exp2. Taken as the natural exp, which is what- * Metal's exp() is; exp2 has its own name in the language.- */+ /* the natural one: Apple's compiler names exp and exp2 separately */r = nir_fexp2(&c->b, nir_fmul_imm(&c->b, a[0], M_LOG2E));} else if (!strcmp(op, "fast_exp2") || !strcmp(op, "exp2")) {r = nir_fexp2(&c->b, a[0]);+ } else if (!strcmp(op, "fast_exp10") || !strcmp(op, "exp10")) {+ r = nir_fexp2(&c->b, nir_fmul_imm(&c->b, a[0], 3.321928094887362)); /* log2(10) */} else if (!strcmp(op, "fast_log") || !strcmp(op, "log")) {r = nir_fmul_imm(&c->b, nir_flog2(&c->b, a[0]), 1.0 / M_LOG2E);} else if (!strcmp(op, "fast_log2") || !strcmp(op, "log2")) {r = nir_flog2(&c->b, a[0]);- } else if (!strcmp(op, "fast_pow") || !strcmp(op, "pow")) {+ } else if (!strcmp(op, "fast_log10") || !strcmp(op, "log10")) {+ r = nir_fmul_imm(&c->b, nir_flog2(&c->b, a[0]), 0.30102999566398120); /* 1/log2(10) */+ } else if (!strcmp(op, "fast_pow") || !strcmp(op, "pow") ||+ !strcmp(op, "fast_powr") || !strcmp(op, "powr")) {+ /* powr is pow with x >= 0 promised, which is what nir_fpow computes */r = nir_fpow(&c->b, a[0], a[1]);+ } else if (!strcmp(op, "fast_fmod") || !strcmp(op, "fmod") ||+ !strcmp(op, "fast_mod") || !strcmp(op, "mod")) {+ r = nir_frem(&c->b, a[0], a[1]);} else if (!strcmp(op, "fast_divide") || !strcmp(op, "divide")) {r = nir_fdiv(&c->b, a[0], a[1]);} else if (!strcmp(op, "saturate") || !strcmp(op, "fast_saturate")) {@@ -154,8 +180,50 @@ math_op(struct air_ctx *c, const struct air_inst *in, const struct air_name *n,r = nir_umin(&c->b, nir_umax(&c->b, a[0], a[1]), a[2]);} else if (!strcmp(op, "fast_ldexp") || !strcmp(op, "ldexp")) {r = nir_ldexp(&c->b, a[0], nir_i2i32(&c->b, a[1]));- } else if (!strcmp(op, "fast_fma") || !strcmp(op, "fma")) {+ } else if (!strcmp(op, "fast_fma") || !strcmp(op, "fma") || !strcmp(op, "mad")) {r = nir_ffma(&c->b, a[0], a[1], a[2]);+ } else if (!strcmp(op, "abs")) {+ r = is_float ? nir_fabs(&c->b, a[0]) : nir_iabs(&c->b, a[0]);+ } else if (!strcmp(op, "abs_diff")) {+ r = is_signed ? nir_iabs(&c->b, nir_isub(&c->b, a[0], a[1]))+ : nir_isub(&c->b, nir_umax(&c->b, a[0], a[1]),+ nir_umin(&c->b, a[0], a[1]));+ } else if (!strcmp(op, "add_sat")) {+ r = is_signed ? nir_iadd_sat(&c->b, a[0], a[1]) : nir_uadd_sat(&c->b, a[0], a[1]);+ } else if (!strcmp(op, "sub_sat")) {+ r = is_signed ? nir_isub_sat(&c->b, a[0], a[1]) : nir_usub_sat(&c->b, a[0], a[1]);+ } else if (!strcmp(op, "mul_hi")) {+ r = is_signed ? nir_imul_high(&c->b, a[0], a[1]) : nir_umul_high(&c->b, a[0], a[1]);+ } else if (!strcmp(op, "popcount")) {+ r = nir_bit_count(&c->b, a[0]);+ } else if (!strcmp(op, "clz")) {+ r = nir_uclz(&c->b, a[0]);+ } else if (!strcmp(op, "ctz")) {+ /* Metal's ctz returns the type's width when the value is zero */+ nir_def *lsb = nir_find_lsb(&c->b, a[0]);+ r = nir_bcsel(&c->b, nir_ieq_imm(&c->b, a[0], 0),+ nir_imm_intN_t(&c->b, a[0]->bit_size, a[0]->bit_size), lsb);+ } else if (!strcmp(op, "reverse_bits")) {+ r = nir_bitfield_reverse(&c->b, a[0]);+ } else if (!strcmp(op, "rotate")) {+ r = nir_urol(&c->b, a[0], nir_u2u32(&c->b, a[1]));+ } else if (!strcmp(op, "extract_bits")) {+ r = is_signed ? nir_ibitfield_extract(&c->b, a[0], nir_u2u32(&c->b, a[1]),+ nir_u2u32(&c->b, a[2]))+ : nir_ubitfield_extract(&c->b, a[0], nir_u2u32(&c->b, a[1]),+ nir_u2u32(&c->b, a[2]));+ } else if (!strcmp(op, "insert_bits")) {+ r = nir_bitfield_insert(&c->b, a[0], a[1], nir_u2u32(&c->b, a[2]),+ nir_u2u32(&c->b, a[3]));+ } else if (!strcmp(op, "dfdx")) {+ r = nir_ddx(&c->b, a[0]);+ } else if (!strcmp(op, "dfdy")) {+ r = nir_ddy(&c->b, a[0]);+ } else if (!strcmp(op, "fwidth")) {+ r = nir_fadd(&c->b, nir_fabs(&c->b, nir_ddx(&c->b, a[0])),+ nir_fabs(&c->b, nir_ddy(&c->b, a[0])));+ } else if (!strcmp(op, "get_simd_width") || !strcmp(op, "get_thread_execution_width")) {+ r = nir_load_subgroup_size(&c->b);} else if (!strcmp(op, "all")) {r = nir_ball(&c->b, a[0]);} else if (!strcmp(op, "any")) {