Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.
C
air: the threadgroup atomics and the barrier, and softpipe runs four at a time
vcvkkfb6d3806c4a5
diff --git a/docs/air.md b/docs/air.mdindex f68b84e..2eadbe3 100644--- a/docs/air.md+++ b/docs/air.md@@ -49,21 +49,22 @@ functions and not entry points at all.24 narrow-type declarations softpipe cannot run at all are exactly the ones where ir3 differsmost from it. None of the seven shaders that have run on the phone calls an `air.*` function:**no operation has been run on the GPU.**-- **The atomics compute the right thing; whether they are atomic is still open.**- `tools/air/atomiccheck.py` dispatches a kernel written for the purpose (section 4, item 5) and- the five operations the corpus uses — global `add.s`, `load`, `max.s`, `min.s`, `store` — agree,- including a signed minimum taken across negative values, which is where an unsigned one would- pass unnoticed. Sixteen threads each incremented a counter once and each was handed a different- predecessor count, so no update was lost. But it ran in software, which need not have run two- threads at once, so that is evidence about the arithmetic and none about atomicity. Two other- things are settled without running anything: every atomic in the corpus asks for- `memory_order_relaxed`, and a stronger order is now refused by name instead of being ignored,- which is what the lowering used to do.-- **Still unverified by anything: the barrier, the derivatives, the simd reductions and the- threadgroup atomics** — what they mean depends on more than one thread or on neighbouring- fragments, and neither a folded constant nor a software dispatch can say it. The texture- operations are covered differently and partly, by the six shaders that returned the exact texel- on the Adreno.+- **The atomics compute the right thing and the barrier waits; whether the atomics are *atomic*+ is still open.** `tools/air/atomiccheck.py` dispatches two kernels written for the purpose+ (section 4, item 5) and all eight operations the corpus uses agree — five on device memory+ (`add.s`, `load`, `max.s`, `min.s`, `store`), three on threadgroup memory (`add.u`, `load`,+ `store`) across two `air.wg.barrier`s. Among them a signed minimum taken across negative+ values, which is where an unsigned one passes unnoticed. The barrier is genuinely under test:+ softpipe advances its invocations in groups of four, so making the barrier a no-op leaves every+ thread seeing 4 of 16 increments, and the check says exactly that. **Atomicity is not under+ test**, because softpipe applies a group's atomic lane by lane and so cannot lose an update+ however the lowering is written. That needs the phone. Two things are settled without running+ anything: every atomic in the corpus asks for `memory_order_relaxed`, and a stronger order is+ now refused by name instead of ignored, which is what the lowering used to do.+- **Still unverified by anything: the derivatives and the simd reductions** — what they mean+ depends on neighbouring fragments or on the width of a simdgroup, and a folded constant cannot+ say it. The texture operations are covered differently and partly, by the six shaders that+ returned the exact texel on the Adreno.- **Nothing here tests Metal's own filtering or addressing.** The six texture shaders that drew onthe Adreno took their sampler from the API, which this harness chose: nearest, clamped, sampledat a texel centre. That is evidence about binding, descriptors and ir3's texture instructions —@@ -571,16 +572,29 @@ Three references, each independent of the translator:written as `old * 16 + index + 1`, so a slot no thread wrote cannot look like a result and athread writing another thread's slot does not read as correct either.- **The five operations agree, and this is still not evidence of atomicity**, because it ran on- softpipe, which need not have run two threads at once — serial execution produces a correct- permutation too. The check passes there by being right rather than by being unopposed. What- would make it evidence is the same kernel on the Adreno, where the threads do run at once, and- that has not happened. It is also only the five operations the corpus proves exist, read off- its own declarations; `air_intrinsics.c` also handles `and`, `or`, `xor` and `xchg`, and those- are untested because the names Apple would emit for them are a guess, and a test of a guessed- name is a test of the guess. The threadgroup atomics (`air.atomic.local.*`, three declarations)- are untested too: they go through `nir_shared_atomic` and a different path for the memory, and- that path has not been exercised at all.+ A second kernel does the threadgroup atomics and the barrier, which the first cannot reach —+ they go through `nir_shared_atomic` and a different path for the memory. Threadgroup memory is+ not zeroed at dispatch, and neither Metal nor Mesa promises it is, so the kernel initialises+ it: every thread storing 0 to the same two words does that without needing a branch. Then each+ thread adds 1 to one word and `1 << its own index` to the other, and after the second barrier+ reads both back into its own slot of two device buffers. Every slot must hold 16 and `0xffff`,+ and the two ways it can fail are told apart by *which* slot is wrong: a lost add clears a bit+ in every slot and the bit names the thread whose increment went missing, while a barrier that+ did not wait leaves a smaller number in the slot of the thread that ran ahead.++ **All eight operations agree.** The barrier is genuinely under test, which was a surprise worth+ recording: softpipe does not advance its invocations one at a time but in groups of four, so+ with `barrier_op` made a no-op every thread sees 4 of 16 increments and the check says so,+ naming the twelve threads whose bits are missing.++ **Atomicity is still not under test.** Softpipe applies a group's atomic lane by lane, so it+ cannot lose an update however the lowering is written, and the permutation comes out correct by+ construction. The check passes there by being right rather than by being unopposed. What would+ make it evidence is the same two kernels on the Adreno, where sixteen threads really do race,+ and that has not happened. It is also only the eight operations the corpus proves exist, read+ off its own declarations; `air_intrinsics.c` also handles `and`, `or`, `xor` and `xchg`, and+ those stay untested because the names Apple would emit for them are a guess, and a test of a+ guessed name is a test of the guess.Where a shader has a known GLSL equivalent (AppleMetalGLRenderer's shaders *are* the GLpipeline's fixed functions), rendering both through the port's GLES and comparing images is adiff --git a/tools/air/atomiccheck.py b/tools/air/atomiccheck.pyindex 8433b34..d5539b3 100755--- a/tools/air/atomiccheck.py+++ b/tools/air/atomiccheck.py@@ -114,6 +114,74 @@ define void @atomics(ptr addrspace(1) %0, ptr addrspace(1) %1, i32 %2) local_unn'''+# The threadgroup atomics and the barrier, which the kernel above cannot reach:+# they go through nir_shared_atomic and a different path for the memory.+#+# Threadgroup memory is not zeroed at dispatch -- neither Metal nor Mesa+# promises that -- so it has to be initialised, and every thread storing 0 to+# the same two words does it without needing a branch.+#+# Then every thread adds 1 to one word and 1 << its own index to the other, and+# after the second barrier reads both back and writes them to its own slot of+# two device buffers. So each of the sixteen slots must hold 16 and 0xffff:+#+# a lost add leaves a bit clear in every slot, and the bit names the thread+# whose increment went missing;+# a barrier that did not wait leaves a *smaller* number in the slot of the+# thread that ran ahead, and names that one instead.+LOCAL_KERNEL = r"""source_filename = "local_atomics"+target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v24:32:32-v32:32:32-v48:64:64-v64:64:64-v96:128:128-v128:128:128-v192:256:256-v256:256:256-v512:512:512-v1024:1024:1024-n8:16:32"+target triple = "air64-apple-ios16.5.0"++@tg = internal addrspace(3) global [2 x i32] undef++declare i32 @air.atomic.local.add.u.i32(ptr addrspace(3), i32, i32, i32, i1) local_unnamed_addr+declare i32 @air.atomic.local.load.i32(ptr addrspace(3), i32, i32, i1) local_unnamed_addr+declare void @air.atomic.local.store.i32(ptr addrspace(3), i32, i32, i32, i1) local_unnamed_addr+declare void @air.wg.barrier(i32, i32) local_unnamed_addr++define void @local_atomics(ptr addrspace(1) %0, ptr addrspace(1) %1, i32 %2) local_unnamed_addr {+ %t0 = getelementptr inbounds [2 x i32], ptr addrspace(3) @tg, i64 0, i64 0+ %t1 = getelementptr inbounds [2 x i32], ptr addrspace(3) @tg, i64 0, i64 1++ ; every thread stores 0 to both: idempotent, so no branch is needed to make+ ; one thread the initialiser+ call void @air.atomic.local.store.i32(ptr addrspace(3) %t0, i32 0, i32 0, i32 1, i1 true)+ call void @air.atomic.local.store.i32(ptr addrspace(3) %t1, i32 0, i32 0, i32 1, i1 true)+ call void @air.wg.barrier(i32 2, i32 1)++ %bit = shl i32 1, %2+ %o0 = call i32 @air.atomic.local.add.u.i32(ptr addrspace(3) %t0, i32 1, i32 0, i32 1, i1 true)+ %o1 = call i32 @air.atomic.local.add.u.i32(ptr addrspace(3) %t1, i32 %bit, i32 0, i32 1, i1 true)+ call void @air.wg.barrier(i32 2, i32 1)++ %v0 = call i32 @air.atomic.local.load.i32(ptr addrspace(3) %t0, i32 0, i32 1, i1 true)+ %v1 = call i32 @air.atomic.local.load.i32(ptr addrspace(3) %t1, i32 0, i32 1, i1 true)+ %w = zext i32 %2 to i64+ %p0 = getelementptr inbounds i32, ptr addrspace(1) %0, i64 %w+ %p1 = getelementptr inbounds i32, ptr addrspace(1) %1, i64 %w+ store i32 %v0, ptr addrspace(1) %p0, align 4+ store i32 %v1, ptr addrspace(1) %p1, align 4+ ret void+}++!llvm.module.flags = !{!0}+!air.version = !{!1}+!air.language_version = !{!2}+!air.kernel = !{!3}++!0 = !{i32 1, !"wchar_size", i32 4}+!1 = !{i32 2, i32 5, i32 0}+!2 = !{!"Metal", i32 3, i32 0, i32 0}+!3 = !{ptr @local_atomics, !4, !5}+!4 = !{}+!5 = !{!6, !7, !8}+!6 = !{i32 0, !"air.buffer", !"air.buffer_size", i32 64, !"air.location_index", i32 0, i32 1, !"air.read_write", !"air.address_space", i32 1, !"air.arg_type_size", i32 4, !"air.arg_type_align_size", i32 4, !"air.arg_type_name", !"int", !"air.arg_name", !"counts"}+!7 = !{i32 1, !"air.buffer", !"air.buffer_size", i32 64, !"air.location_index", i32 1, i32 1, !"air.read_write", !"air.address_space", i32 1, !"air.arg_type_size", i32 4, !"air.arg_type_align_size", i32 4, !"air.arg_type_name", !"int", !"air.arg_name", !"masks"}+!8 = !{i32 2, !"air.thread_index_in_threadgroup", !"air.arg_type_name", !"uint", !"air.arg_name", !"tid"}+"""++def initial_bytes():"""What both buffers start as. Every word is chosen, none is left to theharness's pattern, so a slot nothing wrote reads as something invalid@@ -150,27 +218,16 @@ def signed(u):return u - (1 << 32) if u >> 31 else u-def main(argv):- keep = None- if len(argv) >= 2 and argv[0] == "--keep":- keep = argv[1]- if not os.path.exists(AIR_RUN):- print("atomiccheck: no air_run at %s (run tools/air/mk-host.sh)" % AIR_RUN,- file=sys.stderr)- return 1-- d = keep or tempfile.mkdtemp(prefix="atomiccheck.")- os.makedirs(d, exist_ok=True)- ll, bc, bin_ = (os.path.join(d, n) for n in ("atomics.ll", "atomics.bc", "in.bin"))+def run_kernel(d, stem, source, data, env_extra=None):+ """Assemble one kernel, dispatch it, and return (buffers, screen, output)."""+ ll, bc, bin_ = (os.path.join(d, stem + e) for e in (".ll", ".bc", ".bin"))with open(ll, "w") as f:- f.write(KERNEL)+ f.write(source)with open(bin_, "wb") as f:- f.write(initial_bytes())+ f.write(data)q = subprocess.run(["llvm-as", "-o", bc, ll], capture_output=True)if q.returncode:- print("atomiccheck: llvm-as: " + q.stderr.decode().strip(), file=sys.stderr)- return 1-+ return None, None, "llvm-as: " + q.stderr.decode().strip()q = subprocess.run([AIR_RUN, bc], capture_output=True,env=dict(os.environ, AIR_RUN_BUFFER=bin_))out = q.stdout.decode() + q.stderr.decode()@@ -178,11 +235,48 @@ def main(argv):m = re.search(r"air_run: \S+ on (.+?), as a compute kernel", out)if m:screen = m.group(1)+ return parse(out), screen, out+++def check_local(d):+ """The threadgroup atomics and the barrier. Returns a list of complaints."""+ # the initial bytes do not matter here: every slot is written by its own+ # thread and the threadgroup words are initialised by the kernel+ bufs, screen, out = run_kernel(d, "local", LOCAL_KERNEL,+ b"\0" * 64)+ if bufs is None or 0 not in bufs or 1 not in bufs:+ tail = "\n ".join((out or "").strip().splitlines()[-5:])+ return ["the threadgroup kernel did not read both buffers back:\n " + tail]+ bad = []+ for i in range(THREADS):+ if bufs[0][i] != THREADS:+ bad.append("thread %-2d saw %d of %d increments: the barrier did not wait"+ % (i, bufs[0][i], THREADS))+ want = (1 << THREADS) - 1+ if bufs[1][i] != want:+ missing = [b for b in range(THREADS) if not bufs[1][i] & (1 << b)]+ bad.append("thread %-2d saw mask 0x%08x, want 0x%08x: nothing from thread%s %s"+ % (i, bufs[1][i], want, "" if len(missing) == 1 else "s",+ ", ".join(str(b) for b in missing)))+ return bad- bufs = parse(out)- if 0 not in bufs or 1 not in bufs:++def main(argv):+ keep = None+ if len(argv) >= 2 and argv[0] == "--keep":+ keep = argv[1]+ if not os.path.exists(AIR_RUN):+ print("atomiccheck: no air_run at %s (run tools/air/mk-host.sh)" % AIR_RUN,+ file=sys.stderr)+ return 1++ d = keep or tempfile.mkdtemp(prefix="atomiccheck.")+ os.makedirs(d, exist_ok=True)++ bufs, screen, out = run_kernel(d, "atomics", KERNEL, initial_bytes())+ if bufs is None or 0 not in bufs or 1 not in bufs:print("atomiccheck: the dispatch did not read both buffers back:")- for line in out.strip().splitlines()[-6:]:+ for line in (out or "").strip().splitlines()[-6:]:print(" " + line)return 1@@ -230,20 +324,28 @@ def main(argv):if o not in seen:bad.append("no thread was given %d: that increment went missing" % o)+ local_bad = check_local(d)+ bad += local_bad+for line in bad:print("WRONG " + line)print()if bad:print("atomiccheck: %d of the checks failed on %s" % (len(bad), screen))return 1- print("atomiccheck: the five atomic operations the corpus uses agree, on %s" % screen)+ print("atomiccheck: the eight atomic operations the corpus uses agree, on %s" % screen)+ print("atomiccheck: five on device memory, three on threadgroup memory across two")+ print("atomiccheck: barriers, every thread seeing all sixteen contributions")print("atomiccheck: %d threads each incremented once and were given a distinct"% THREADS)print("atomiccheck: predecessor count, 0 to %d, so no update was lost" % (THREADS - 1))if "softpipe" in screen or "software" in screen:- print("atomiccheck: but this ran in software, which may not have run two threads")- print("atomiccheck: at once -- so it is evidence the operations compute the right")- print("atomiccheck: thing and NO evidence that they are atomic. That needs the GPU.")+ print("atomiccheck: the two barriers are really under test here: softpipe advances its")+ print("atomiccheck: invocations in groups of four, so with the barrier made a no-op")+ print("atomiccheck: every thread sees 4 of 16 increments and this says so.")+ print("atomiccheck: ATOMICITY still is not tested. Softpipe applies a group's atomic")+ print("atomiccheck: lane by lane, so it cannot lose an update however the lowering is")+ print("atomiccheck: written. That needs the GPU, where sixteen threads really do race.")if keep:print("atomiccheck: kept in %s" % d)return 0