Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.
C
air: a kernel of our own, so a lost atomic update would show
vcvkk97eadb0310b2
diff --git a/docs/air.md b/docs/air.mdindex 5c4293e..f68b84e 100644--- a/docs/air.md+++ b/docs/air.md@@ -49,13 +49,21 @@ functions and not entry points at all.24 narrow-type declarations softpipe cannot run at all are exactly the ones where ir3 differsmost from it. None of the seven shaders that have run on the phone calls an `air.*` function:**no operation has been run on the GPU.**-- **Still unverified by anything: the atomics, the barrier, the derivatives and the simd- reductions** — 41 declarations with no reference in `intrincheck.py`, because what they mean- depends on more than one thread or on neighbouring fragments and a folded constant cannot say- it. The texture operations among them are covered differently and partly, by the six shaders- that returned the exact texel on the Adreno. One thing *is* now settled about the atomics- without running any: every one in the corpus asks for `memory_order_relaxed`, and the lowering- refuses any other order by name instead of ignoring the argument, which it used to do.+- **The atomics compute the right thing; whether they are atomic is still open.**+ `tools/air/atomiccheck.py` dispatches a kernel written for the purpose (section 4, item 5) and+ the five operations the corpus uses — global `add.s`, `load`, `max.s`, `min.s`, `store` — agree,+ including a signed minimum taken across negative values, which is where an unsigned one would+ pass unnoticed. Sixteen threads each incremented a counter once and each was handed a different+ predecessor count, so no update was lost. But it ran in software, which need not have run two+ threads at once, so that is evidence about the arithmetic and none about atomicity. Two other+ things are settled without running anything: every atomic in the corpus asks for+ `memory_order_relaxed`, and a stronger order is now refused by name instead of being ignored,+ which is what the lowering used to do.+- **Still unverified by anything: the barrier, the derivatives, the simd reductions and the+ threadgroup atomics** — what they mean depends on more than one thread or on neighbouring+ fragments, and neither a folded constant nor a software dispatch can say it. The texture+ operations are covered differently and partly, by the six shaders that returned the exact texel+ on the Adreno.- **Nothing here tests Metal's own filtering or addressing.** The six texture shaders that drew onthe Adreno took their sampler from the API, which this harness chose: nearest, clamped, sampledat a texel centre. That is evidence about binding, descriptors and ir3's texture instructions —@@ -541,6 +549,39 @@ Three references, each independent of the translator:session all night. **So every number above is from an x86 host, and the bottom row of thetable at the top of this document still reads 7.**+5. **The atomics, with a kernel written to make a lost update show.** This document used to say+ that none of Apple's kernels could test an atomic and why: "a sum written by threads that never+ raced looks exactly like a sum written by threads that did. Detecting a lost update needs the+ expected total to depend on every thread having contributed — which needs a kernel whose+ arithmetic is known, i.e. one written for the purpose rather than one of Apple's."+ `tools/air/atomiccheck.py` is that kernel, generated as AIR, assembled with `llvm-as` and+ dispatched by `air_run` over its 4×4 threads, with both device buffers' initial bytes chosen+ rather than left to the harness's pattern.++ Four of its five checks are ordinary: the counter reaches the thread count, the atomic load+ returns what was put there, the signed maximum is the largest thread index, and the signed+ minimum of `index - 100` is `-100` — that last one is where an unsigned minimum would answer+ `0xffffff9c` as a huge number, and where it did when the lowering was broken on purpose to see+ whether this check works.++ The fifth is the one that can see a race, and it needs no knowledge of the order the threads+ ran in. Each thread writes the value its own `fetch_add` returned, so if the increment is+ atomic those sixteen returned values are the numbers 0 to 15 in *some* order. A lost update+ leaves a duplicate and a gap, and the gap names the increment that went missing. Each is+ written as `old * 16 + index + 1`, so a slot no thread wrote cannot look like a result and a+ thread writing another thread's slot does not read as correct either.++ **The five operations agree, and this is still not evidence of atomicity**, because it ran on+ softpipe, which need not have run two threads at once — serial execution produces a correct+ permutation too. The check passes there by being right rather than by being unopposed. What+ would make it evidence is the same kernel on the Adreno, where the threads do run at once, and+ that has not happened. It is also only the five operations the corpus proves exist, read off+ its own declarations; `air_intrinsics.c` also handles `and`, `or`, `xor` and `xchg`, and those+ are untested because the names Apple would emit for them are a guess, and a test of a guessed+ name is a test of the guess. The threadgroup atomics (`air.atomic.local.*`, three declarations)+ are untested too: they go through `nir_shared_atomic` and a different path for the memory, and+ that path has not been exercised at all.+Where a shader has a known GLSL equivalent (AppleMetalGLRenderer's shaders *are* the GLpipeline's fixed functions), rendering both through the port's GLES and comparing images is afourth check; it is not built yet.@@ -868,6 +909,7 @@ tools/air/intrincheck.py [-v] [OP ...] # one air.* operationtools/air/intrincheck.py --run --few [OP ...] # the same, drawn by a real drivertools/air/intrincheck.py --emit DIR --few # modules + inputs + run.sh, to carrytools/air/intrincheck.py --check DIR RUN.LOG # judge what the runner printed+tools/air/atomiccheck.py [--keep DIR] # the atomics, with its own kernelbuild/mesa-host/build/src/gallium/drivers/freedreno/air_run \FILE[.metallib] [FUNCTION] # draw with it, print the pixeltools/air/texcheck.sh [FILE.metallib ...] # every texture shader, the texel it returnsdiff --git a/tools/air/atomiccheck.py b/tools/air/atomiccheck.pynew file mode 100755index 0000000..8433b34--- /dev/null+++ b/tools/air/atomiccheck.py@@ -0,0 +1,253 @@+#!/usr/bin/env python3+"""Check the atomics with a kernel written for the purpose.++docs/air.md said what was missing and why none of Apple's kernels could supply+it: "a sum written by threads that never raced looks exactly like a sum written+by threads that did. Detecting a lost update needs the expected total to depend+on every thread having contributed -- which needs a kernel whose arithmetic is+known, i.e. one written for the purpose rather than one of Apple's." This is+that kernel.++It is generated as AIR, assembled with llvm-as and dispatched by air_run, which+runs 4x4 threads in one workgroup. Every thread does the same five things to two+device buffers whose initial bytes this script chooses (AIR_RUN_BUFFER).++What each part can show:++ [0] += 1 by every thread the total is the thread count+ [1] += the value loaded from [4] atomic load returns what is there+ [2] = max(thread index) signed max, and the right operand+ [3] = min(thread index - 100) signed min across negatives, which+ is where an unsigned min would+ answer 0xffffff9c-as-huge instead+ buf1[i] = old*16 + i + 1 see below++The last one is the only part that can detect a lost update, and it does so+without depending on the order the threads ran in. If fetch_add is atomic, the+sixteen values it returned are the numbers 0..15 in some order -- a permutation+-- because each thread saw a different predecessor count. If two threads read+the same old value, the permutation has a duplicate and a gap, and the gap says+which increment was lost. Serial execution also produces a permutation, so this+passes on software by being correct rather than by being unopposed; what it+cannot do is *prove* atomicity on a machine that never ran the threads at once.+The +1 is so that a slot no thread wrote cannot look like a result, and the+*16 + i is so that a thread writing another thread's slot does not read as+correct either.++The operations tested are only the ones the corpus proves exist, read off its+own declarations: global add.s, load, max.s, min.s and store. air_intrinsics.c+also handles and, or, xor and xchg, and those are not tested here, because the+names Apple would emit for them are a guess and a test of a guessed name is a+test of the guess.++usage: atomiccheck.py [--keep DIR]+"""++import os+import re+import struct+import subprocess+import sys+import tempfile++ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))+AIR_RUN = os.environ.get(+ "AIR_RUN",+ os.path.join(ROOT, "build/mesa-host/build/src/gallium/drivers/freedreno/air_run"),+)++THREADS = 16 # air_run dispatches GRID x GRID = 4x4 in one workgroup+LOADED = 7 # what sits in [4] for every thread to read++KERNEL = r'''source_filename = "atomics"+target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v24:32:32-v32:32:32-v48:64:64-v64:64:64-v96:128:128-v128:128:128-v192:256:256-v256:256:256-v512:512:512-v1024:1024:1024-n8:16:32"+target triple = "air64-apple-ios16.5.0"++declare i32 @air.atomic.global.add.s.i32(ptr addrspace(1), i32, i32, i32, i1) local_unnamed_addr+declare i32 @air.atomic.global.load.i32(ptr addrspace(1), i32, i32, i1) local_unnamed_addr+declare i32 @air.atomic.global.max.s.i32(ptr addrspace(1), i32, i32, i32, i1) local_unnamed_addr+declare i32 @air.atomic.global.min.s.i32(ptr addrspace(1), i32, i32, i32, i1) local_unnamed_addr+declare void @air.atomic.global.store.i32(ptr addrspace(1), i32, i32, i32, i1) local_unnamed_addr++define void @atomics(ptr addrspace(1) %0, ptr addrspace(1) %1, i32 %2) local_unnamed_addr {+ ; [0] += 1: the returned old value is this thread's place in the queue+ %c0 = getelementptr inbounds i32, ptr addrspace(1) %0, i64 0+ %old = call i32 @air.atomic.global.add.s.i32(ptr addrspace(1) %c0, i32 1, i32 0, i32 2, i1 true)++ ; [1] += whatever [4] holds, loaded atomically+ %c4 = getelementptr inbounds i32, ptr addrspace(1) %0, i64 4+ %got = call i32 @air.atomic.global.load.i32(ptr addrspace(1) %c4, i32 0, i32 2, i1 true)+ %c1 = getelementptr inbounds i32, ptr addrspace(1) %0, i64 1+ %sum = call i32 @air.atomic.global.add.s.i32(ptr addrspace(1) %c1, i32 %got, i32 0, i32 2, i1 true)++ ; [2] = max(thread index), [3] = min(thread index - 100)+ %c2 = getelementptr inbounds i32, ptr addrspace(1) %0, i64 2+ %mx = call i32 @air.atomic.global.max.s.i32(ptr addrspace(1) %c2, i32 %2, i32 0, i32 2, i1 true)+ %neg = sub i32 %2, 100+ %c3 = getelementptr inbounds i32, ptr addrspace(1) %0, i64 3+ %mn = call i32 @air.atomic.global.min.s.i32(ptr addrspace(1) %c3, i32 %neg, i32 0, i32 2, i1 true)++ ; buf1[index] = old * 16 + index + 1, stored atomically+ %t0 = mul i32 %old, 16+ %t1 = add i32 %t0, %2+ %t2 = add i32 %t1, 1+ %w = zext i32 %2 to i64+ %p1 = getelementptr inbounds i32, ptr addrspace(1) %1, i64 %w+ call void @air.atomic.global.store.i32(ptr addrspace(1) %p1, i32 %t2, i32 0, i32 2, i1 true)+ ret void+}++!llvm.module.flags = !{!0}+!air.version = !{!1}+!air.language_version = !{!2}+!air.kernel = !{!3}++!0 = !{i32 1, !"wchar_size", i32 4}+!1 = !{i32 2, i32 5, i32 0}+!2 = !{!"Metal", i32 3, i32 0, i32 0}+!3 = !{ptr @atomics, !4, !5}+!4 = !{}+!5 = !{!6, !7, !8}+!6 = !{i32 0, !"air.buffer", !"air.buffer_size", i32 64, !"air.location_index", i32 0, i32 1, !"air.read_write", !"air.address_space", i32 1, !"air.arg_type_size", i32 4, !"air.arg_type_align_size", i32 4, !"air.arg_type_name", !"int", !"air.arg_name", !"counters"}+!7 = !{i32 1, !"air.buffer", !"air.buffer_size", i32 64, !"air.location_index", i32 1, i32 1, !"air.read_write", !"air.address_space", i32 1, !"air.arg_type_size", i32 4, !"air.arg_type_align_size", i32 4, !"air.arg_type_name", !"int", !"air.arg_name", !"queue"}+!8 = !{i32 2, !"air.thread_index_in_threadgroup", !"air.arg_type_name", !"uint", !"air.arg_name", !"tid"}+'''+++def initial_bytes():+ """What both buffers start as. Every word is chosen, none is left to the+ harness's pattern, so a slot nothing wrote reads as something invalid+ rather than as something plausible."""+ words = [0] * 16+ words[0] = 0 # the counter+ words[1] = 0 # the sum of the loaded values+ words[2] = 0 # max starts at 0; every thread index is >= 0+ words[3] = 2147483647 # min starts at INT_MAX+ words[4] = LOADED # what every thread loads+ return b"".join(struct.pack("<i", w) for w in words)+++def parse(out):+ """{buffer index: [words]} from air_run's readback."""+ bufs = {}+ cur = None+ for line in out.splitlines():+ m = re.match(r"air_run: device buffer (\d+) after the dispatch", line)+ if m:+ cur = int(m.group(1))+ bufs[cur] = []+ continue+ if cur is not None:+ m = re.match(r"\s*\[\s*(\d+)\]\s+(\d+)\s+0x([0-9a-f]+)", line)+ if m:+ bufs[cur].append(int(m.group(3), 16))+ elif line.startswith("air_run:") and "device buffer" not in line:+ cur = None+ return bufs+++def signed(u):+ return u - (1 << 32) if u >> 31 else u+++def main(argv):+ keep = None+ if len(argv) >= 2 and argv[0] == "--keep":+ keep = argv[1]+ if not os.path.exists(AIR_RUN):+ print("atomiccheck: no air_run at %s (run tools/air/mk-host.sh)" % AIR_RUN,+ file=sys.stderr)+ return 1++ d = keep or tempfile.mkdtemp(prefix="atomiccheck.")+ os.makedirs(d, exist_ok=True)+ ll, bc, bin_ = (os.path.join(d, n) for n in ("atomics.ll", "atomics.bc", "in.bin"))+ with open(ll, "w") as f:+ f.write(KERNEL)+ with open(bin_, "wb") as f:+ f.write(initial_bytes())+ q = subprocess.run(["llvm-as", "-o", bc, ll], capture_output=True)+ if q.returncode:+ print("atomiccheck: llvm-as: " + q.stderr.decode().strip(), file=sys.stderr)+ return 1++ q = subprocess.run([AIR_RUN, bc], capture_output=True,+ env=dict(os.environ, AIR_RUN_BUFFER=bin_))+ out = q.stdout.decode() + q.stderr.decode()+ screen = "an unnamed screen"+ m = re.search(r"air_run: \S+ on (.+?), as a compute kernel", out)+ if m:+ screen = m.group(1)++ bufs = parse(out)+ if 0 not in bufs or 1 not in bufs:+ print("atomiccheck: the dispatch did not read both buffers back:")+ for line in out.strip().splitlines()[-6:]:+ print(" " + line)+ return 1++ c, queue = bufs[0], bufs[1]+ bad = []++ def want(label, got, exp):+ if got != exp:+ bad.append("%-34s got %-12s want %s" % (label, got, exp))++ want("[0] threads that incremented", c[0], THREADS)+ want("[1] sum of the loaded value", c[1], THREADS * LOADED)+ want("[2] max of the thread indices", c[2], THREADS - 1)+ want("[3] min of index - 100", signed(c[3]), -100)+ want("[4] the loaded value, untouched", c[4], LOADED)++ # the permutation: every slot written by its own thread, and the old values+ # the sixteen fetch_adds returned being 0..15 with nothing repeated+ olds = []+ for i in range(THREADS):+ v = queue[i]+ if v == 0:+ bad.append("queue[%-2d] no thread wrote it" % i)+ continue+ if (v - 1) % 16 != i:+ bad.append("queue[%-2d] was written by thread %d" % (i, (v - 1) % 16))+ continue+ o = (v - 1) // 16+ if not 0 <= o < THREADS:+ # the whole set being shifted is what returning the new value+ # instead of the old looks like, so say that rather than leave it+ # to be read off a missing element+ bad.append("queue[%-2d] was given %d, which is not a predecessor count "+ "of %d threads" % (i, o, THREADS))+ olds.append(o)+ if len(olds) == THREADS and sorted(olds) != list(range(THREADS)):+ seen = {}+ for i, o in enumerate(olds):+ seen.setdefault(o, []).append(i)+ for o in sorted(seen):+ if len(seen[o]) > 1:+ bad.append("fetch_add returned %d to threads %s: a lost update"+ % (o, ", ".join(str(t) for t in seen[o])))+ for o in range(THREADS):+ if o not in seen:+ bad.append("no thread was given %d: that increment went missing" % o)++ for line in bad:+ print("WRONG " + line)+ print()+ if bad:+ print("atomiccheck: %d of the checks failed on %s" % (len(bad), screen))+ return 1+ print("atomiccheck: the five atomic operations the corpus uses agree, on %s" % screen)+ print("atomiccheck: %d threads each incremented once and were given a distinct"+ % THREADS)+ print("atomiccheck: predecessor count, 0 to %d, so no update was lost" % (THREADS - 1))+ if "softpipe" in screen or "software" in screen:+ print("atomiccheck: but this ran in software, which may not have run two threads")+ print("atomiccheck: at once -- so it is evidence the operations compute the right")+ print("atomiccheck: thing and NO evidence that they are atomic. That needs the GPU.")+ if keep:+ print("atomiccheck: kept in %s" % d)+ return 0+++if __name__ == "__main__":+ sys.exit(main(sys.argv[1:]))diff --git a/tools/air/check.sh b/tools/air/check.shindex 53621bf..d3a91f4 100755--- a/tools/air/check.sh+++ b/tools/air/check.sh@@ -92,4 +92,13 @@ echo "== the same operations run, not folded"python3 "$ROOT/tools/air/intrincheck.py" --run --cases 4 2>&1 | tail -6run_rc=${PIPESTATUS[0]}-exit $((reader_rc | cpu_rc | intrin_rc | run_rc))+echo+echo "== the atomics, with a kernel written to make a lost update show"+# None of Apple's kernels can test an atomic: a sum written by threads that+# never raced reads the same as one written by threads that did. This one+# dispatches sixteen threads whose fetch_add results must be the numbers 0..15+# with nothing repeated, so a lost update leaves a duplicate and a gap.+python3 "$ROOT/tools/air/atomiccheck.py" 2>&1 | tail -8+atomic_rc=${PIPESTATUS[0]}++exit $((reader_rc | cpu_rc | intrin_rc | run_rc | atomic_rc))diff --git a/tools/mesa/air/air_run.c b/tools/mesa/air/air_run.cindex be885c0..c257d76 100644--- a/tools/mesa/air/air_run.c+++ b/tools/mesa/air/air_run.c@@ -269,14 +269,15 @@ finalize_for_gallium(nir_shader *nir)* pattern), so a thread reading the wrong offset produces a wrong value* and not a plausible one.*- * What this does NOT yet do is give atomics a test that can fail. Seventeen of- * the forty kernels that translate use a global atomic, and a sum written by+ * A kernel of Apple's that uses atomics and produces sensible numbers here is+ * evidence that it ran and no evidence about atomicity: a sum written by* threads that never raced looks exactly like a sum written by threads that- * did. Detecting a lost update needs the expected total to depend on every- * thread having contributed -- which needs a kernel whose arithmetic is known,- * i.e. one written for the purpose rather than one of Apple's. Until that- * exists, a kernel that uses atomics and produces sensible numbers here is- * evidence that it ran, and no evidence at all about atomicity.+ * did. The kernel that can tell them apart is written for the purpose rather+ * than Apple's -- tools/air/atomiccheck.py, which makes each thread report the+ * value its own fetch_add returned, so the sixteen of them must be 0..15 with+ * nothing repeated whatever order the threads ran in. Even that is only+ * evidence of atomicity where the threads really do run at once, which here+ * they need not.*/#define GRID 4