Skip to content

Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.

C

air: a kernel of our own, so a lost atomic update would show

vcvkk97eadb0310b2
diff --git a/docs/air.md b/docs/air.md
index 5c4293e..f68b84e 100644
--- a/docs/air.md
+++ b/docs/air.md
@@ -49,13 +49,21 @@ functions and not entry points at all.
24 narrow-type declarations softpipe cannot run at all are exactly the ones where ir3 differs
most from it. None of the seven shaders that have run on the phone calls an `air.*` function:
**no operation has been run on the GPU.**
-- **Still unverified by anything: the atomics, the barrier, the derivatives and the simd
- reductions** — 41 declarations with no reference in `intrincheck.py`, because what they mean
- depends on more than one thread or on neighbouring fragments and a folded constant cannot say
- it. The texture operations among them are covered differently and partly, by the six shaders
- that returned the exact texel on the Adreno. One thing *is* now settled about the atomics
- without running any: every one in the corpus asks for `memory_order_relaxed`, and the lowering
- refuses any other order by name instead of ignoring the argument, which it used to do.
+- **The atomics compute the right thing; whether they are atomic is still open.**
+ `tools/air/atomiccheck.py` dispatches a kernel written for the purpose (section 4, item 5) and
+ the five operations the corpus uses — global `add.s`, `load`, `max.s`, `min.s`, `store` — agree,
+ including a signed minimum taken across negative values, which is where an unsigned one would
+ pass unnoticed. Sixteen threads each incremented a counter once and each was handed a different
+ predecessor count, so no update was lost. But it ran in software, which need not have run two
+ threads at once, so that is evidence about the arithmetic and none about atomicity. Two other
+ things are settled without running anything: every atomic in the corpus asks for
+ `memory_order_relaxed`, and a stronger order is now refused by name instead of being ignored,
+ which is what the lowering used to do.
+- **Still unverified by anything: the barrier, the derivatives, the simd reductions and the
+ threadgroup atomics** — what they mean depends on more than one thread or on neighbouring
+ fragments, and neither a folded constant nor a software dispatch can say it. The texture
+ operations are covered differently and partly, by the six shaders that returned the exact texel
+ on the Adreno.
- **Nothing here tests Metal's own filtering or addressing.** The six texture shaders that drew on
the Adreno took their sampler from the API, which this harness chose: nearest, clamped, sampled
at a texel centre. That is evidence about binding, descriptors and ir3's texture instructions —
@@ -541,6 +549,39 @@ Three references, each independent of the translator:
session all night. **So every number above is from an x86 host, and the bottom row of the
table at the top of this document still reads 7.**
+5. **The atomics, with a kernel written to make a lost update show.** This document used to say
+ that none of Apple's kernels could test an atomic and why: "a sum written by threads that never
+ raced looks exactly like a sum written by threads that did. Detecting a lost update needs the
+ expected total to depend on every thread having contributed — which needs a kernel whose
+ arithmetic is known, i.e. one written for the purpose rather than one of Apple's."
+ `tools/air/atomiccheck.py` is that kernel, generated as AIR, assembled with `llvm-as` and
+ dispatched by `air_run` over its 4×4 threads, with both device buffers' initial bytes chosen
+ rather than left to the harness's pattern.
+
+ Four of its five checks are ordinary: the counter reaches the thread count, the atomic load
+ returns what was put there, the signed maximum is the largest thread index, and the signed
+ minimum of `index - 100` is `-100` — that last one is where an unsigned minimum would answer
+ `0xffffff9c` as a huge number, and where it did when the lowering was broken on purpose to see
+ whether this check works.
+
+ The fifth is the one that can see a race, and it needs no knowledge of the order the threads
+ ran in. Each thread writes the value its own `fetch_add` returned, so if the increment is
+ atomic those sixteen returned values are the numbers 0 to 15 in *some* order. A lost update
+ leaves a duplicate and a gap, and the gap names the increment that went missing. Each is
+ written as `old * 16 + index + 1`, so a slot no thread wrote cannot look like a result and a
+ thread writing another thread's slot does not read as correct either.
+
+ **The five operations agree, and this is still not evidence of atomicity**, because it ran on
+ softpipe, which need not have run two threads at once — serial execution produces a correct
+ permutation too. The check passes there by being right rather than by being unopposed. What
+ would make it evidence is the same kernel on the Adreno, where the threads do run at once, and
+ that has not happened. It is also only the five operations the corpus proves exist, read off
+ its own declarations; `air_intrinsics.c` also handles `and`, `or`, `xor` and `xchg`, and those
+ are untested because the names Apple would emit for them are a guess, and a test of a guessed
+ name is a test of the guess. The threadgroup atomics (`air.atomic.local.*`, three declarations)
+ are untested too: they go through `nir_shared_atomic` and a different path for the memory, and
+ that path has not been exercised at all.
+
Where a shader has a known GLSL equivalent (AppleMetalGLRenderer's shaders *are* the GL
pipeline's fixed functions), rendering both through the port's GLES and comparing images is a
fourth check; it is not built yet.
@@ -868,6 +909,7 @@ tools/air/intrincheck.py [-v] [OP ...] # one air.* operation
tools/air/intrincheck.py --run --few [OP ...] # the same, drawn by a real driver
tools/air/intrincheck.py --emit DIR --few # modules + inputs + run.sh, to carry
tools/air/intrincheck.py --check DIR RUN.LOG # judge what the runner printed
+tools/air/atomiccheck.py [--keep DIR] # the atomics, with its own kernel
build/mesa-host/build/src/gallium/drivers/freedreno/air_run \
FILE[.metallib] [FUNCTION] # draw with it, print the pixel
tools/air/texcheck.sh [FILE.metallib ...] # every texture shader, the texel it returns
diff --git a/tools/air/atomiccheck.py b/tools/air/atomiccheck.py
new file mode 100755
index 0000000..8433b34
--- /dev/null
+++ b/tools/air/atomiccheck.py
@@ -0,0 +1,253 @@
+#!/usr/bin/env python3
+"""Check the atomics with a kernel written for the purpose.
+
+docs/air.md said what was missing and why none of Apple's kernels could supply
+it: "a sum written by threads that never raced looks exactly like a sum written
+by threads that did. Detecting a lost update needs the expected total to depend
+on every thread having contributed -- which needs a kernel whose arithmetic is
+known, i.e. one written for the purpose rather than one of Apple's." This is
+that kernel.
+
+It is generated as AIR, assembled with llvm-as and dispatched by air_run, which
+runs 4x4 threads in one workgroup. Every thread does the same five things to two
+device buffers whose initial bytes this script chooses (AIR_RUN_BUFFER).
+
+What each part can show:
+
+ [0] += 1 by every thread the total is the thread count
+ [1] += the value loaded from [4] atomic load returns what is there
+ [2] = max(thread index) signed max, and the right operand
+ [3] = min(thread index - 100) signed min across negatives, which
+ is where an unsigned min would
+ answer 0xffffff9c-as-huge instead
+ buf1[i] = old*16 + i + 1 see below
+
+The last one is the only part that can detect a lost update, and it does so
+without depending on the order the threads ran in. If fetch_add is atomic, the
+sixteen values it returned are the numbers 0..15 in some order -- a permutation
+-- because each thread saw a different predecessor count. If two threads read
+the same old value, the permutation has a duplicate and a gap, and the gap says
+which increment was lost. Serial execution also produces a permutation, so this
+passes on software by being correct rather than by being unopposed; what it
+cannot do is *prove* atomicity on a machine that never ran the threads at once.
+The +1 is so that a slot no thread wrote cannot look like a result, and the
+*16 + i is so that a thread writing another thread's slot does not read as
+correct either.
+
+The operations tested are only the ones the corpus proves exist, read off its
+own declarations: global add.s, load, max.s, min.s and store. air_intrinsics.c
+also handles and, or, xor and xchg, and those are not tested here, because the
+names Apple would emit for them are a guess and a test of a guessed name is a
+test of the guess.
+
+usage: atomiccheck.py [--keep DIR]
+"""
+
+import os
+import re
+import struct
+import subprocess
+import sys
+import tempfile
+
+ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+AIR_RUN = os.environ.get(
+ "AIR_RUN",
+ os.path.join(ROOT, "build/mesa-host/build/src/gallium/drivers/freedreno/air_run"),
+)
+
+THREADS = 16 # air_run dispatches GRID x GRID = 4x4 in one workgroup
+LOADED = 7 # what sits in [4] for every thread to read
+
+KERNEL = r'''source_filename = "atomics"
+target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v24:32:32-v32:32:32-v48:64:64-v64:64:64-v96:128:128-v128:128:128-v192:256:256-v256:256:256-v512:512:512-v1024:1024:1024-n8:16:32"
+target triple = "air64-apple-ios16.5.0"
+
+declare i32 @air.atomic.global.add.s.i32(ptr addrspace(1), i32, i32, i32, i1) local_unnamed_addr
+declare i32 @air.atomic.global.load.i32(ptr addrspace(1), i32, i32, i1) local_unnamed_addr
+declare i32 @air.atomic.global.max.s.i32(ptr addrspace(1), i32, i32, i32, i1) local_unnamed_addr
+declare i32 @air.atomic.global.min.s.i32(ptr addrspace(1), i32, i32, i32, i1) local_unnamed_addr
+declare void @air.atomic.global.store.i32(ptr addrspace(1), i32, i32, i32, i1) local_unnamed_addr
+
+define void @atomics(ptr addrspace(1) %0, ptr addrspace(1) %1, i32 %2) local_unnamed_addr {
+ ; [0] += 1: the returned old value is this thread's place in the queue
+ %c0 = getelementptr inbounds i32, ptr addrspace(1) %0, i64 0
+ %old = call i32 @air.atomic.global.add.s.i32(ptr addrspace(1) %c0, i32 1, i32 0, i32 2, i1 true)
+
+ ; [1] += whatever [4] holds, loaded atomically
+ %c4 = getelementptr inbounds i32, ptr addrspace(1) %0, i64 4
+ %got = call i32 @air.atomic.global.load.i32(ptr addrspace(1) %c4, i32 0, i32 2, i1 true)
+ %c1 = getelementptr inbounds i32, ptr addrspace(1) %0, i64 1
+ %sum = call i32 @air.atomic.global.add.s.i32(ptr addrspace(1) %c1, i32 %got, i32 0, i32 2, i1 true)
+
+ ; [2] = max(thread index), [3] = min(thread index - 100)
+ %c2 = getelementptr inbounds i32, ptr addrspace(1) %0, i64 2
+ %mx = call i32 @air.atomic.global.max.s.i32(ptr addrspace(1) %c2, i32 %2, i32 0, i32 2, i1 true)
+ %neg = sub i32 %2, 100
+ %c3 = getelementptr inbounds i32, ptr addrspace(1) %0, i64 3
+ %mn = call i32 @air.atomic.global.min.s.i32(ptr addrspace(1) %c3, i32 %neg, i32 0, i32 2, i1 true)
+
+ ; buf1[index] = old * 16 + index + 1, stored atomically
+ %t0 = mul i32 %old, 16
+ %t1 = add i32 %t0, %2
+ %t2 = add i32 %t1, 1
+ %w = zext i32 %2 to i64
+ %p1 = getelementptr inbounds i32, ptr addrspace(1) %1, i64 %w
+ call void @air.atomic.global.store.i32(ptr addrspace(1) %p1, i32 %t2, i32 0, i32 2, i1 true)
+ ret void
+}
+
+!llvm.module.flags = !{!0}
+!air.version = !{!1}
+!air.language_version = !{!2}
+!air.kernel = !{!3}
+
+!0 = !{i32 1, !"wchar_size", i32 4}
+!1 = !{i32 2, i32 5, i32 0}
+!2 = !{!"Metal", i32 3, i32 0, i32 0}
+!3 = !{ptr @atomics, !4, !5}
+!4 = !{}
+!5 = !{!6, !7, !8}
+!6 = !{i32 0, !"air.buffer", !"air.buffer_size", i32 64, !"air.location_index", i32 0, i32 1, !"air.read_write", !"air.address_space", i32 1, !"air.arg_type_size", i32 4, !"air.arg_type_align_size", i32 4, !"air.arg_type_name", !"int", !"air.arg_name", !"counters"}
+!7 = !{i32 1, !"air.buffer", !"air.buffer_size", i32 64, !"air.location_index", i32 1, i32 1, !"air.read_write", !"air.address_space", i32 1, !"air.arg_type_size", i32 4, !"air.arg_type_align_size", i32 4, !"air.arg_type_name", !"int", !"air.arg_name", !"queue"}
+!8 = !{i32 2, !"air.thread_index_in_threadgroup", !"air.arg_type_name", !"uint", !"air.arg_name", !"tid"}
+'''
+
+
+def initial_bytes():
+ """What both buffers start as. Every word is chosen, none is left to the
+ harness's pattern, so a slot nothing wrote reads as something invalid
+ rather than as something plausible."""
+ words = [0] * 16
+ words[0] = 0 # the counter
+ words[1] = 0 # the sum of the loaded values
+ words[2] = 0 # max starts at 0; every thread index is >= 0
+ words[3] = 2147483647 # min starts at INT_MAX
+ words[4] = LOADED # what every thread loads
+ return b"".join(struct.pack("<i", w) for w in words)
+
+
+def parse(out):
+ """{buffer index: [words]} from air_run's readback."""
+ bufs = {}
+ cur = None
+ for line in out.splitlines():
+ m = re.match(r"air_run: device buffer (\d+) after the dispatch", line)
+ if m:
+ cur = int(m.group(1))
+ bufs[cur] = []
+ continue
+ if cur is not None:
+ m = re.match(r"\s*\[\s*(\d+)\]\s+(\d+)\s+0x([0-9a-f]+)", line)
+ if m:
+ bufs[cur].append(int(m.group(3), 16))
+ elif line.startswith("air_run:") and "device buffer" not in line:
+ cur = None
+ return bufs
+
+
+def signed(u):
+ return u - (1 << 32) if u >> 31 else u
+
+
+def main(argv):
+ keep = None
+ if len(argv) >= 2 and argv[0] == "--keep":
+ keep = argv[1]
+ if not os.path.exists(AIR_RUN):
+ print("atomiccheck: no air_run at %s (run tools/air/mk-host.sh)" % AIR_RUN,
+ file=sys.stderr)
+ return 1
+
+ d = keep or tempfile.mkdtemp(prefix="atomiccheck.")
+ os.makedirs(d, exist_ok=True)
+ ll, bc, bin_ = (os.path.join(d, n) for n in ("atomics.ll", "atomics.bc", "in.bin"))
+ with open(ll, "w") as f:
+ f.write(KERNEL)
+ with open(bin_, "wb") as f:
+ f.write(initial_bytes())
+ q = subprocess.run(["llvm-as", "-o", bc, ll], capture_output=True)
+ if q.returncode:
+ print("atomiccheck: llvm-as: " + q.stderr.decode().strip(), file=sys.stderr)
+ return 1
+
+ q = subprocess.run([AIR_RUN, bc], capture_output=True,
+ env=dict(os.environ, AIR_RUN_BUFFER=bin_))
+ out = q.stdout.decode() + q.stderr.decode()
+ screen = "an unnamed screen"
+ m = re.search(r"air_run: \S+ on (.+?), as a compute kernel", out)
+ if m:
+ screen = m.group(1)
+
+ bufs = parse(out)
+ if 0 not in bufs or 1 not in bufs:
+ print("atomiccheck: the dispatch did not read both buffers back:")
+ for line in out.strip().splitlines()[-6:]:
+ print(" " + line)
+ return 1
+
+ c, queue = bufs[0], bufs[1]
+ bad = []
+
+ def want(label, got, exp):
+ if got != exp:
+ bad.append("%-34s got %-12s want %s" % (label, got, exp))
+
+ want("[0] threads that incremented", c[0], THREADS)
+ want("[1] sum of the loaded value", c[1], THREADS * LOADED)
+ want("[2] max of the thread indices", c[2], THREADS - 1)
+ want("[3] min of index - 100", signed(c[3]), -100)
+ want("[4] the loaded value, untouched", c[4], LOADED)
+
+ # the permutation: every slot written by its own thread, and the old values
+ # the sixteen fetch_adds returned being 0..15 with nothing repeated
+ olds = []
+ for i in range(THREADS):
+ v = queue[i]
+ if v == 0:
+ bad.append("queue[%-2d] no thread wrote it" % i)
+ continue
+ if (v - 1) % 16 != i:
+ bad.append("queue[%-2d] was written by thread %d" % (i, (v - 1) % 16))
+ continue
+ o = (v - 1) // 16
+ if not 0 <= o < THREADS:
+ # the whole set being shifted is what returning the new value
+ # instead of the old looks like, so say that rather than leave it
+ # to be read off a missing element
+ bad.append("queue[%-2d] was given %d, which is not a predecessor count "
+ "of %d threads" % (i, o, THREADS))
+ olds.append(o)
+ if len(olds) == THREADS and sorted(olds) != list(range(THREADS)):
+ seen = {}
+ for i, o in enumerate(olds):
+ seen.setdefault(o, []).append(i)
+ for o in sorted(seen):
+ if len(seen[o]) > 1:
+ bad.append("fetch_add returned %d to threads %s: a lost update"
+ % (o, ", ".join(str(t) for t in seen[o])))
+ for o in range(THREADS):
+ if o not in seen:
+ bad.append("no thread was given %d: that increment went missing" % o)
+
+ for line in bad:
+ print("WRONG " + line)
+ print()
+ if bad:
+ print("atomiccheck: %d of the checks failed on %s" % (len(bad), screen))
+ return 1
+ print("atomiccheck: the five atomic operations the corpus uses agree, on %s" % screen)
+ print("atomiccheck: %d threads each incremented once and were given a distinct"
+ % THREADS)
+ print("atomiccheck: predecessor count, 0 to %d, so no update was lost" % (THREADS - 1))
+ if "softpipe" in screen or "software" in screen:
+ print("atomiccheck: but this ran in software, which may not have run two threads")
+ print("atomiccheck: at once -- so it is evidence the operations compute the right")
+ print("atomiccheck: thing and NO evidence that they are atomic. That needs the GPU.")
+ if keep:
+ print("atomiccheck: kept in %s" % d)
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main(sys.argv[1:]))
diff --git a/tools/air/check.sh b/tools/air/check.sh
index 53621bf..d3a91f4 100755
--- a/tools/air/check.sh
+++ b/tools/air/check.sh
@@ -92,4 +92,13 @@ echo "== the same operations run, not folded"
python3 "$ROOT/tools/air/intrincheck.py" --run --cases 4 2>&1 | tail -6
run_rc=${PIPESTATUS[0]}
-exit $((reader_rc | cpu_rc | intrin_rc | run_rc))
+echo
+echo "== the atomics, with a kernel written to make a lost update show"
+# None of Apple's kernels can test an atomic: a sum written by threads that
+# never raced reads the same as one written by threads that did. This one
+# dispatches sixteen threads whose fetch_add results must be the numbers 0..15
+# with nothing repeated, so a lost update leaves a duplicate and a gap.
+python3 "$ROOT/tools/air/atomiccheck.py" 2>&1 | tail -8
+atomic_rc=${PIPESTATUS[0]}
+
+exit $((reader_rc | cpu_rc | intrin_rc | run_rc | atomic_rc))
diff --git a/tools/mesa/air/air_run.c b/tools/mesa/air/air_run.c
index be885c0..c257d76 100644
--- a/tools/mesa/air/air_run.c
+++ b/tools/mesa/air/air_run.c
@@ -269,14 +269,15 @@ finalize_for_gallium(nir_shader *nir)
* pattern), so a thread reading the wrong offset produces a wrong value
* and not a plausible one.
*
- * What this does NOT yet do is give atomics a test that can fail. Seventeen of
- * the forty kernels that translate use a global atomic, and a sum written by
+ * A kernel of Apple's that uses atomics and produces sensible numbers here is
+ * evidence that it ran and no evidence about atomicity: a sum written by
* threads that never raced looks exactly like a sum written by threads that
- * did. Detecting a lost update needs the expected total to depend on every
- * thread having contributed -- which needs a kernel whose arithmetic is known,
- * i.e. one written for the purpose rather than one of Apple's. Until that
- * exists, a kernel that uses atomics and produces sensible numbers here is
- * evidence that it ran, and no evidence at all about atomicity.
+ * did. The kernel that can tell them apart is written for the purpose rather
+ * than Apple's -- tools/air/atomiccheck.py, which makes each thread report the
+ * value its own fetch_add returned, so the sixteen of them must be 0..15 with
+ * nothing repeated whatever order the threads ran in. Even that is only
+ * evidence of atomicity where the threads really do run at once, which here
+ * they need not.
*/
#define GRID 4