Skip to content

Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.

C

air: the threadgroup atomics and the barrier, and softpipe runs four at a time

vcvkkfb6d3806c4a5
diff --git a/docs/air.md b/docs/air.md
index f68b84e..2eadbe3 100644
--- a/docs/air.md
+++ b/docs/air.md
@@ -49,21 +49,22 @@ functions and not entry points at all.
24 narrow-type declarations softpipe cannot run at all are exactly the ones where ir3 differs
most from it. None of the seven shaders that have run on the phone calls an `air.*` function:
**no operation has been run on the GPU.**
-- **The atomics compute the right thing; whether they are atomic is still open.**
- `tools/air/atomiccheck.py` dispatches a kernel written for the purpose (section 4, item 5) and
- the five operations the corpus uses — global `add.s`, `load`, `max.s`, `min.s`, `store` — agree,
- including a signed minimum taken across negative values, which is where an unsigned one would
- pass unnoticed. Sixteen threads each incremented a counter once and each was handed a different
- predecessor count, so no update was lost. But it ran in software, which need not have run two
- threads at once, so that is evidence about the arithmetic and none about atomicity. Two other
- things are settled without running anything: every atomic in the corpus asks for
- `memory_order_relaxed`, and a stronger order is now refused by name instead of being ignored,
- which is what the lowering used to do.
-- **Still unverified by anything: the barrier, the derivatives, the simd reductions and the
- threadgroup atomics** — what they mean depends on more than one thread or on neighbouring
- fragments, and neither a folded constant nor a software dispatch can say it. The texture
- operations are covered differently and partly, by the six shaders that returned the exact texel
- on the Adreno.
+- **The atomics compute the right thing and the barrier waits; whether the atomics are *atomic*
+ is still open.** `tools/air/atomiccheck.py` dispatches two kernels written for the purpose
+ (section 4, item 5) and all eight operations the corpus uses agree — five on device memory
+ (`add.s`, `load`, `max.s`, `min.s`, `store`), three on threadgroup memory (`add.u`, `load`,
+ `store`) across two `air.wg.barrier`s. Among them a signed minimum taken across negative
+ values, which is where an unsigned one passes unnoticed. The barrier is genuinely under test:
+ softpipe advances its invocations in groups of four, so making the barrier a no-op leaves every
+ thread seeing 4 of 16 increments, and the check says exactly that. **Atomicity is not under
+ test**, because softpipe applies a group's atomic lane by lane and so cannot lose an update
+ however the lowering is written. That needs the phone. Two things are settled without running
+ anything: every atomic in the corpus asks for `memory_order_relaxed`, and a stronger order is
+ now refused by name instead of ignored, which is what the lowering used to do.
+- **Still unverified by anything: the derivatives and the simd reductions** — what they mean
+ depends on neighbouring fragments or on the width of a simdgroup, and a folded constant cannot
+ say it. The texture operations are covered differently and partly, by the six shaders that
+ returned the exact texel on the Adreno.
- **Nothing here tests Metal's own filtering or addressing.** The six texture shaders that drew on
the Adreno took their sampler from the API, which this harness chose: nearest, clamped, sampled
at a texel centre. That is evidence about binding, descriptors and ir3's texture instructions —
@@ -571,16 +572,29 @@ Three references, each independent of the translator:
written as `old * 16 + index + 1`, so a slot no thread wrote cannot look like a result and a
thread writing another thread's slot does not read as correct either.
- **The five operations agree, and this is still not evidence of atomicity**, because it ran on
- softpipe, which need not have run two threads at once — serial execution produces a correct
- permutation too. The check passes there by being right rather than by being unopposed. What
- would make it evidence is the same kernel on the Adreno, where the threads do run at once, and
- that has not happened. It is also only the five operations the corpus proves exist, read off
- its own declarations; `air_intrinsics.c` also handles `and`, `or`, `xor` and `xchg`, and those
- are untested because the names Apple would emit for them are a guess, and a test of a guessed
- name is a test of the guess. The threadgroup atomics (`air.atomic.local.*`, three declarations)
- are untested too: they go through `nir_shared_atomic` and a different path for the memory, and
- that path has not been exercised at all.
+ A second kernel does the threadgroup atomics and the barrier, which the first cannot reach —
+ they go through `nir_shared_atomic` and a different path for the memory. Threadgroup memory is
+ not zeroed at dispatch, and neither Metal nor Mesa promises it is, so the kernel initialises
+ it: every thread storing 0 to the same two words does that without needing a branch. Then each
+ thread adds 1 to one word and `1 << its own index` to the other, and after the second barrier
+ reads both back into its own slot of two device buffers. Every slot must hold 16 and `0xffff`,
+ and the two ways it can fail are told apart by *which* slot is wrong: a lost add clears a bit
+ in every slot and the bit names the thread whose increment went missing, while a barrier that
+ did not wait leaves a smaller number in the slot of the thread that ran ahead.
+
+ **All eight operations agree.** The barrier is genuinely under test, which was a surprise worth
+ recording: softpipe does not advance its invocations one at a time but in groups of four, so
+ with `barrier_op` made a no-op every thread sees 4 of 16 increments and the check says so,
+ naming the twelve threads whose bits are missing.
+
+ **Atomicity is still not under test.** Softpipe applies a group's atomic lane by lane, so it
+ cannot lose an update however the lowering is written, and the permutation comes out correct by
+ construction. The check passes there by being right rather than by being unopposed. What would
+ make it evidence is the same two kernels on the Adreno, where sixteen threads really do race,
+ and that has not happened. It is also only the eight operations the corpus proves exist, read
+ off its own declarations; `air_intrinsics.c` also handles `and`, `or`, `xor` and `xchg`, and
+ those stay untested because the names Apple would emit for them are a guess, and a test of a
+ guessed name is a test of the guess.
Where a shader has a known GLSL equivalent (AppleMetalGLRenderer's shaders *are* the GL
pipeline's fixed functions), rendering both through the port's GLES and comparing images is a
diff --git a/tools/air/atomiccheck.py b/tools/air/atomiccheck.py
index 8433b34..d5539b3 100755
--- a/tools/air/atomiccheck.py
+++ b/tools/air/atomiccheck.py
@@ -114,6 +114,74 @@ define void @atomics(ptr addrspace(1) %0, ptr addrspace(1) %1, i32 %2) local_unn
'''
+# The threadgroup atomics and the barrier, which the kernel above cannot reach:
+# they go through nir_shared_atomic and a different path for the memory.
+#
+# Threadgroup memory is not zeroed at dispatch -- neither Metal nor Mesa
+# promises that -- so it has to be initialised, and every thread storing 0 to
+# the same two words does it without needing a branch.
+#
+# Then every thread adds 1 to one word and 1 << its own index to the other, and
+# after the second barrier reads both back and writes them to its own slot of
+# two device buffers. So each of the sixteen slots must hold 16 and 0xffff:
+#
+# a lost add leaves a bit clear in every slot, and the bit names the thread
+# whose increment went missing;
+# a barrier that did not wait leaves a *smaller* number in the slot of the
+# thread that ran ahead, and names that one instead.
+LOCAL_KERNEL = r"""source_filename = "local_atomics"
+target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v24:32:32-v32:32:32-v48:64:64-v64:64:64-v96:128:128-v128:128:128-v192:256:256-v256:256:256-v512:512:512-v1024:1024:1024-n8:16:32"
+target triple = "air64-apple-ios16.5.0"
+
+@tg = internal addrspace(3) global [2 x i32] undef
+
+declare i32 @air.atomic.local.add.u.i32(ptr addrspace(3), i32, i32, i32, i1) local_unnamed_addr
+declare i32 @air.atomic.local.load.i32(ptr addrspace(3), i32, i32, i1) local_unnamed_addr
+declare void @air.atomic.local.store.i32(ptr addrspace(3), i32, i32, i32, i1) local_unnamed_addr
+declare void @air.wg.barrier(i32, i32) local_unnamed_addr
+
+define void @local_atomics(ptr addrspace(1) %0, ptr addrspace(1) %1, i32 %2) local_unnamed_addr {
+ %t0 = getelementptr inbounds [2 x i32], ptr addrspace(3) @tg, i64 0, i64 0
+ %t1 = getelementptr inbounds [2 x i32], ptr addrspace(3) @tg, i64 0, i64 1
+
+ ; every thread stores 0 to both: idempotent, so no branch is needed to make
+ ; one thread the initialiser
+ call void @air.atomic.local.store.i32(ptr addrspace(3) %t0, i32 0, i32 0, i32 1, i1 true)
+ call void @air.atomic.local.store.i32(ptr addrspace(3) %t1, i32 0, i32 0, i32 1, i1 true)
+ call void @air.wg.barrier(i32 2, i32 1)
+
+ %bit = shl i32 1, %2
+ %o0 = call i32 @air.atomic.local.add.u.i32(ptr addrspace(3) %t0, i32 1, i32 0, i32 1, i1 true)
+ %o1 = call i32 @air.atomic.local.add.u.i32(ptr addrspace(3) %t1, i32 %bit, i32 0, i32 1, i1 true)
+ call void @air.wg.barrier(i32 2, i32 1)
+
+ %v0 = call i32 @air.atomic.local.load.i32(ptr addrspace(3) %t0, i32 0, i32 1, i1 true)
+ %v1 = call i32 @air.atomic.local.load.i32(ptr addrspace(3) %t1, i32 0, i32 1, i1 true)
+ %w = zext i32 %2 to i64
+ %p0 = getelementptr inbounds i32, ptr addrspace(1) %0, i64 %w
+ %p1 = getelementptr inbounds i32, ptr addrspace(1) %1, i64 %w
+ store i32 %v0, ptr addrspace(1) %p0, align 4
+ store i32 %v1, ptr addrspace(1) %p1, align 4
+ ret void
+}
+
+!llvm.module.flags = !{!0}
+!air.version = !{!1}
+!air.language_version = !{!2}
+!air.kernel = !{!3}
+
+!0 = !{i32 1, !"wchar_size", i32 4}
+!1 = !{i32 2, i32 5, i32 0}
+!2 = !{!"Metal", i32 3, i32 0, i32 0}
+!3 = !{ptr @local_atomics, !4, !5}
+!4 = !{}
+!5 = !{!6, !7, !8}
+!6 = !{i32 0, !"air.buffer", !"air.buffer_size", i32 64, !"air.location_index", i32 0, i32 1, !"air.read_write", !"air.address_space", i32 1, !"air.arg_type_size", i32 4, !"air.arg_type_align_size", i32 4, !"air.arg_type_name", !"int", !"air.arg_name", !"counts"}
+!7 = !{i32 1, !"air.buffer", !"air.buffer_size", i32 64, !"air.location_index", i32 1, i32 1, !"air.read_write", !"air.address_space", i32 1, !"air.arg_type_size", i32 4, !"air.arg_type_align_size", i32 4, !"air.arg_type_name", !"int", !"air.arg_name", !"masks"}
+!8 = !{i32 2, !"air.thread_index_in_threadgroup", !"air.arg_type_name", !"uint", !"air.arg_name", !"tid"}
+"""
+
+
def initial_bytes():
"""What both buffers start as. Every word is chosen, none is left to the
harness's pattern, so a slot nothing wrote reads as something invalid
@@ -150,27 +218,16 @@ def signed(u):
return u - (1 << 32) if u >> 31 else u
-def main(argv):
- keep = None
- if len(argv) >= 2 and argv[0] == "--keep":
- keep = argv[1]
- if not os.path.exists(AIR_RUN):
- print("atomiccheck: no air_run at %s (run tools/air/mk-host.sh)" % AIR_RUN,
- file=sys.stderr)
- return 1
-
- d = keep or tempfile.mkdtemp(prefix="atomiccheck.")
- os.makedirs(d, exist_ok=True)
- ll, bc, bin_ = (os.path.join(d, n) for n in ("atomics.ll", "atomics.bc", "in.bin"))
+def run_kernel(d, stem, source, data, env_extra=None):
+ """Assemble one kernel, dispatch it, and return (buffers, screen, output)."""
+ ll, bc, bin_ = (os.path.join(d, stem + e) for e in (".ll", ".bc", ".bin"))
with open(ll, "w") as f:
- f.write(KERNEL)
+ f.write(source)
with open(bin_, "wb") as f:
- f.write(initial_bytes())
+ f.write(data)
q = subprocess.run(["llvm-as", "-o", bc, ll], capture_output=True)
if q.returncode:
- print("atomiccheck: llvm-as: " + q.stderr.decode().strip(), file=sys.stderr)
- return 1
-
+ return None, None, "llvm-as: " + q.stderr.decode().strip()
q = subprocess.run([AIR_RUN, bc], capture_output=True,
env=dict(os.environ, AIR_RUN_BUFFER=bin_))
out = q.stdout.decode() + q.stderr.decode()
@@ -178,11 +235,48 @@ def main(argv):
m = re.search(r"air_run: \S+ on (.+?), as a compute kernel", out)
if m:
screen = m.group(1)
+ return parse(out), screen, out
+
+
+def check_local(d):
+ """The threadgroup atomics and the barrier. Returns a list of complaints."""
+ # the initial bytes do not matter here: every slot is written by its own
+ # thread and the threadgroup words are initialised by the kernel
+ bufs, screen, out = run_kernel(d, "local", LOCAL_KERNEL,
+ b"\0" * 64)
+ if bufs is None or 0 not in bufs or 1 not in bufs:
+ tail = "\n ".join((out or "").strip().splitlines()[-5:])
+ return ["the threadgroup kernel did not read both buffers back:\n " + tail]
+ bad = []
+ for i in range(THREADS):
+ if bufs[0][i] != THREADS:
+ bad.append("thread %-2d saw %d of %d increments: the barrier did not wait"
+ % (i, bufs[0][i], THREADS))
+ want = (1 << THREADS) - 1
+ if bufs[1][i] != want:
+ missing = [b for b in range(THREADS) if not bufs[1][i] & (1 << b)]
+ bad.append("thread %-2d saw mask 0x%08x, want 0x%08x: nothing from thread%s %s"
+ % (i, bufs[1][i], want, "" if len(missing) == 1 else "s",
+ ", ".join(str(b) for b in missing)))
+ return bad
- bufs = parse(out)
- if 0 not in bufs or 1 not in bufs:
+
+def main(argv):
+ keep = None
+ if len(argv) >= 2 and argv[0] == "--keep":
+ keep = argv[1]
+ if not os.path.exists(AIR_RUN):
+ print("atomiccheck: no air_run at %s (run tools/air/mk-host.sh)" % AIR_RUN,
+ file=sys.stderr)
+ return 1
+
+ d = keep or tempfile.mkdtemp(prefix="atomiccheck.")
+ os.makedirs(d, exist_ok=True)
+
+ bufs, screen, out = run_kernel(d, "atomics", KERNEL, initial_bytes())
+ if bufs is None or 0 not in bufs or 1 not in bufs:
print("atomiccheck: the dispatch did not read both buffers back:")
- for line in out.strip().splitlines()[-6:]:
+ for line in (out or "").strip().splitlines()[-6:]:
print(" " + line)
return 1
@@ -230,20 +324,28 @@ def main(argv):
if o not in seen:
bad.append("no thread was given %d: that increment went missing" % o)
+ local_bad = check_local(d)
+ bad += local_bad
+
for line in bad:
print("WRONG " + line)
print()
if bad:
print("atomiccheck: %d of the checks failed on %s" % (len(bad), screen))
return 1
- print("atomiccheck: the five atomic operations the corpus uses agree, on %s" % screen)
+ print("atomiccheck: the eight atomic operations the corpus uses agree, on %s" % screen)
+ print("atomiccheck: five on device memory, three on threadgroup memory across two")
+ print("atomiccheck: barriers, every thread seeing all sixteen contributions")
print("atomiccheck: %d threads each incremented once and were given a distinct"
% THREADS)
print("atomiccheck: predecessor count, 0 to %d, so no update was lost" % (THREADS - 1))
if "softpipe" in screen or "software" in screen:
- print("atomiccheck: but this ran in software, which may not have run two threads")
- print("atomiccheck: at once -- so it is evidence the operations compute the right")
- print("atomiccheck: thing and NO evidence that they are atomic. That needs the GPU.")
+ print("atomiccheck: the two barriers are really under test here: softpipe advances its")
+ print("atomiccheck: invocations in groups of four, so with the barrier made a no-op")
+ print("atomiccheck: every thread sees 4 of 16 increments and this says so.")
+ print("atomiccheck: ATOMICITY still is not tested. Softpipe applies a group's atomic")
+ print("atomiccheck: lane by lane, so it cannot lose an update however the lowering is")
+ print("atomiccheck: written. That needs the GPU, where sixteen threads really do race.")
if keep:
print("atomiccheck: kept in %s" % d)
return 0