Apple's open-source XNU kernel on Android phones: its own loader and drivers, Metal on the Adreno. Runs from RAM; a reboot gives the phone back to Android.
C
air: run the 135 library functions by wrapping them, and use the decoder we had
vcvkke9ea9f0185fb
diff --git a/docs/air.md b/docs/air.mdindex faa4b1d..d0e4f88 100644--- a/docs/air.md+++ b/docs/air.md@@ -41,8 +41,42 @@ And of the 119 distinct `air.*` operations those modules call (`tools/air/intrin| texture operations with a case softpipe cannot run: a 16-bit result | 7 || texture operations with no case: multisample, depth, write | 11 |-318 of 456 is 318 of 321 entry points — 99.1% — because 135 modules are `air.visible` library-functions and not entry points at all.+321 of 456 is **every entry point** of the corpus: 271 fragment shaders, 47 compute kernels and 3+vertex shaders. The other 135 modules are `air.visible` library functions, not entry points at+all, and they have a measurement of their own:++| the 135 library functions | |+| --- | --- |+| wrapped in a generated entry point and translated (`--library`) | 124 |+| refused: a texture, sampler or matrix argument the wrapper cannot pass | 11 |+| dispatched by a Gallium driver and read back | 124 |+| of those, agreeing with our own constant folding of the same shader | 104 |+| of those, run but disagreeing | 0 |+| ran, but the folder did not reduce the result to a constant | 20 |+| **agreeing with LLVM on this machine, bit for bit** | **21 of 21** |++`tools/air/libcheck.py` prints that table and refuses to print it when the rows do not add up.+The last row is the one that counts, and it is small for a stated reason: LLVM can be the+reference only for a function that calls no `air.*` at all, because an `air.*` call is an+undefined symbol on the host and defining one would put our own reading of that operation on+both sides of the comparison. 57 more call only exactly defined `air.*` operations and could be+compared that way if those definitions were written; 46 call an approximate one+(`air.fast_pow` and the like), which nothing compares bit-exactly anywhere.++**What a library function's measurement is not: evidence about Metal's stitching.** Metal links+these into a caller at pipeline time through the ABI their own metadata describes+(`air.stitching_info`, with the four functions that do it and each argument's size and+alignment). That ABI is the runtime's, not the compiler's, and it is not implemented. The wrapper+is this project's own calling convention, invented to make the bodies runnable.++**And what the 456 itself is not: the number of shaders on the device.** It is the number of+modules in the `.metallib` *files* under `ios165/rootfs`. A library can also be a blob inside a+dylib — `GPUCompiler`'s own `GNR.videoprocessor` carries one in `__DATA`, found while looking for+something else — so every statement here of the form "the corpus contains no X" is bounded by how+the corpus was collected, not by what the system contains. Where that distinction matters, this+document says so; extending the corpus to embedded libraries would move this denominator and+every percentage computed from it, so it is a change to make in one commit and recount, not a+figure to adjust.**What those numbers are not**, because they will be quoted without this paragraph otherwise:@@ -311,8 +345,16 @@ functions and not entry points at all.corner, where nearest filtering has no weights to round and clamping has no arithmetic.The six texture shaders that drew on the Adreno took their sampler from the API too, so whatthey are evidence about is binding, descriptors and ir3's texture instructions — exactly what- they were built to be — and silence about everything Metal's sampler state decides. Two modules- of the corpus declare a `constexpr sampler`; everywhere else the API supplies it.+ they were built to be — and silence about everything Metal's sampler state decides. **Ten**+ modules of the corpus declare a `constexpr sampler`, carrying three distinct state words, all+ ten in `PortraitFilters.cifilter/default.metallib`; everywhere else the API supplies it. This+ paragraph said "two" for a while, while the mapping table in section 7 said ten: two numbers for+ one fact in one file, and the disagreement was enough to settle it without any new evidence.+ Where the "two" came from is not established. What is worth recording is how the ten were+ counted, because the obvious way does not work: a plain `grep` for `__air_sampler_state` over+ the 456 modules finds **none**, since bitcode is bit-packed and a name matches only when it+ happens to land on byte boundaries. The count above is from `llvm-dis` output, module by+ module.- **A function of the module that takes a texture: done, and the diagnosis was the work.** Allfour such entry points translate — three with this fix, the fourth once the shuffle mask it alsostopped on was decoded. The fix is a split of `translate_globals`, and arriving at it took two@@ -379,12 +421,17 @@ functions and not entry points at all.encodings meet only in this case, and our reader accepted only the second. We refused on a zerowe could read perfectly well, and recorded it as a limit of the shader.- The fix is not three cases but one: `air_const_as_uint` (`air_module.h`) reads either spelling,- and the three other places that took an integer constant off a record — `llvm.memcpy` and- `llvm.memset` lengths, `air.wg.barrier` flags — now go through it. None of those three was- reachable in this corpus, so this is not a correction of any number; it closes a class. Two- spellings of zero is a property of the format, and a format detail belongs in one decoder- rather than in every reader of it.+ **And the decoder for it already existed.** `air_const_uint` (`air_module.h`) has read both+ spellings since long before this, with a comment beside one of its callers saying that a zero+ arrives as `CST_CODE_NULL` about as often as not. Four places tested `ckind == AIR_CONST_INT`+ by hand instead of calling it — the shuffle mask, the `llvm.memcpy` and `llvm.memset` lengths,+ and the `air.wg.barrier` flags — and the shuffle mask is the one where it cost something. All+ four go through the helper now, which removed more code than it added.++ The first write-up of this fix claimed it "closes a class" by adding that decoder. It did not:+ the class was already closed and one reader had opted out of it, which is a different and less+ flattering story. Worth keeping for the shape — a hand-rolled test beside an existing helper+ that does the same job, in a file big enough that neither author sees the other.The four that used to crash now dispatch, and the crash was this harness's:diff --git a/tools/air/cpuref.py b/tools/air/cpuref.pyindex 541b137..ceaf1fa 100755--- a/tools/air/cpuref.py+++ b/tools/air/cpuref.py@@ -78,6 +78,20 @@ def parse_type(t):return (base, 1) if base else (None, 0)+def param_type(t):+ """The type out of a define's parameter: "<4 x float> %0" -> "<4 x float>".++ Taken as the leading type rather than by stripping a list of attribute+ words: LLVM has dozens of parameter attributes and they sit between the+ type and the name, so a blacklist misses one -- "returned" was the one that+ made a library function unreadable here, and the next version of LLVM would+ have found another.+ """+ m = re.match(r"\s*(<\s*\d+\s+x\s+[A-Za-z_0-9]+\s*>|\[[^\]]*\]|[A-Za-z_0-9]+)",+ t)+ return re.sub(r"\s+", " ", m.group(1)).strip() if m else t.strip()++def split_fields(s):"""The top-level comma-separated members of a struct body."""out, depth, cur = [], 0, ""@@ -157,7 +171,61 @@ def buffer_args(ll, entry):return out-def build_and_run(ll, entry, bufs, workdir):+# Apple's name for each type, from the air.visible_input/output metadata beside+# a library function. It is a second statement of what the arguments are: the+# LLVM signature says <4 x float>, the metadata says float4, and a harness that+# agrees with only one of them is guessing. parse_type reads the LLVM side; this+# reads Apple's, and the two are checked against each other before anything runs.+APPLE_TYPES = {+ "float": ("float", 1), "float2": ("float", 2),+ "float3": ("float", 3), "float4": ("float", 4),+ "half": ("half", 1), "half2": ("half", 2),+ "half3": ("half", 3), "half4": ("half", 4),+}+++def library_types(ll):+ """(inputs, output) as Apple names them, in parameter order."""+ ins = {}+ for body in re.findall(r"^!\d+ = !\{(.*)\}$", ll, re.M):+ if '"air.visible_input"' in body:+ pm = re.match(r"\s*i32 (\d+)", body)+ tn = re.search(r'"air\.arg_type_name", !"([^"]*)"', body)+ if pm and tn:+ ins[int(pm.group(1))] = tn.group(1)+ out = None+ for body in re.findall(r"^!\d+ = !\{(.*)\}$", ll, re.M):+ if '"air.visible_output"' in body:+ tn = re.search(r'"air\.arg_type_name", !"([^"]*)"', body)+ if tn:+ out = tn.group(1)+ return [ins.get(i) for i in range(len(ins))], out+++def check_against_metadata(entry, ll):+ """Refuse when the LLVM types and Apple's names do not say the same thing."""+ names, out = library_types(ll)+ if len(names) != len(entry["params"]):+ return ("the metadata names %d arguments and the signature has %d"+ % (len(names), len(entry["params"])))+ for i, (t, name) in enumerate(zip(entry["params"], names)):+ base, n = parse_type(param_type(t))+ if base is None:+ return "argument %d is %s, which this harness cannot make" % (i, param_type(t))+ if name is None:+ return "argument %d has no air.arg_type_name" % i+ want = APPLE_TYPES.get(name)+ if want is None:+ return "argument %d is called %s, a type this harness does not make" % (i, name)+ if (want[0] == "float") != (base.fmt == "f") or want[1] != n:+ return ("argument %d is %s in the signature and %s in the metadata"+ % (i, param_type(t), name))+ if out is not None and out not in APPLE_TYPES:+ return "the result is called %s, which this harness cannot read" % out+ return None+++def build_and_run(ll, entry, bufs, workdir, library=False):"""Compile the module for this machine and call the entry point.The call is made from a wrapper written in LLVM IR, not from C. A shader@@ -179,6 +247,23 @@ def build_and_run(ll, entry, bufs, workdir):if not fields:return None, "the entry point returns nothing to compare"+ # A library function takes values, not buffers, so the wrapper reads each+ # argument out of one buffer at byte 16*i -- the same slot the frontend's+ # generated entry point reads it from (wrap_library_function in+ # air_to_nir.c). Both sides read the same offsets, so this comparison+ # cannot catch a slot layout that is wrong in the same way twice; what it+ # does catch is the two sides disagreeing about which slot is which+ # argument, because the pattern makes every slot's bytes different.+ if library:+ wrap_params, call_args, pre = ["ptr %in"], [], []+ for i, t in enumerate(entry["params"]):+ t = param_type(t)+ pre.append(" %%g%d = getelementptr i8, ptr %%in, i64 %d" % (i, 16 * i))+ pre.append(" %%a%d = load %s, ptr %%g%d, align 16" % (i, t, i))+ call_args.append("%s %%a%d" % (t, i))+ return _finish(ll, entry, {0: 16 * max(len(entry["params"]), 1)}, workdir,+ fields, ret, wrap_params, call_args, pre, True)+# the wrapper's own arguments: one pointer per buffer, and the outputwrap_params, call_args, pre = [], [], []for i, t in enumerate(entry["params"]):@@ -202,10 +287,23 @@ def build_and_run(ll, entry, bufs, workdir):# really is only the GPU knows, so such a shader is not comparedreturn None, "parameter %d is %s, not a buffer" % (i, t)- # store each field at its own 16-byte slot of the output buffer+ return _finish(ll, entry, bufs, workdir, fields, ret, wrap_params, call_args, pre)+++def _finish(ll, entry, bufs, workdir, fields, ret, wrap_params, call_args, pre,+ library=False):+ """Compile the wrapper and the C harness around it, and run them."""+ # store each field at its own 16-byte slot of the output buffer. A shader+ # returns a struct of fields and each one is extracted; a library function+ # returns one value and there is nothing to extract -- extractvalue on a+ # vector is not valid IR, and llc says so with a caret and no sentence.+ is_struct = bool(re.match(r"<?\{", ret.strip()))stores, slots = [], []for i, (base, n, t) in enumerate(fields):- stores.append(" %%f%d = extractvalue %s %%r, %d" % (i, ret, i))+ if is_struct:+ stores.append(" %%f%d = extractvalue %s %%r, %d" % (i, ret, i))+ else:+ stores.append(" %%f%d = bitcast %s %%r to %s" % (i, t, t))stores.append(" %%o%d = getelementptr i8, ptr %%out, i64 %d" % (i, i * 16))stores.append(" store %s %%f%d, ptr %%o%d, align 1" % (t, i, i))slots.append((base, n))@@ -235,8 +333,18 @@ define void @__cpuref_call(%s, ptr %%out) {setup, args = [], []for i in sorted(bufs):setup.append(" static unsigned char buf%d[%d];" % (i, bufs[i]))- setup.append(" for (int i = 0; i < %d; i++) buf%d[i] = (i * 7 + 13) & 0xff;"- % (bufs[i], i))+ if library:+ # floats, not bytes: these modules are compiled with no-nans and+ # no-infs, so arbitrary bytes put the call outside the contract it+ # was compiled under. The same words air2nir and air_run use+ # (library_byte in both).+ setup.append(" for (int w = 0; w < %d; w++) {"+ " float f = (float)((w * 13) %% 29 - 14) / 4.0f;"+ " __builtin_memcpy(buf%d + 4 * w, &f, 4); }"+ % (bufs[i] // 4, i))+ else:+ setup.append(" for (int i = 0; i < %d; i++) buf%d[i] = (i * 7 + 13) & 0xff;"+ % (bufs[i], i))args.append("buf%d" % i)decl = ", ".join(["const void *"] * len(args) + ["void *"]) if args else "void *"@@ -325,14 +433,25 @@ def main():entry = entry_of(ll)if not entry:sys.exit("cpuref.py: no entry point found")+ library = entry["stage"] == "visible"bufs = buffer_args(ll, entry)- print("cpuref: %s (%s), %d parameters, %d of them buffers" %- (entry["name"], entry["stage"], len(entry["params"]), len(bufs)))+ if library:+ why = check_against_metadata(entry, ll)+ if why:+ print("cpuref: cannot run this one: %s" % why)+ return 1+ bufs = {0: 16 * max(len(entry["params"]), 1)}+ print("cpuref: %s (a library function), %d arguments, each read from "+ "byte 16*i of one buffer" % (entry["name"], len(entry["params"])))+ else:+ print("cpuref: %s (%s), %d parameters, %d of them buffers" %+ (entry["name"], entry["stage"], len(entry["params"]), len(bufs)))for i, n in sorted(bufs.items()):- print(" parameter %d: a buffer of %d bytes, filled with (i * 7 + 13) & 0xff"- % (i, n))+ print(" parameter %d: a buffer of %d bytes, filled with %s"+ % (i, n, "the floats ((w * 13) % 29 - 14) / 4" if library+ else "(i * 7 + 13) & 0xff"))- values, why = build_and_run(ll, entry, bufs, work)+ values, why = build_and_run(ll, entry, bufs, work, library)if why:print("cpuref: cannot run this one: %s" % why)return 1@@ -341,7 +460,7 @@ def main():print(" %s" % v)if args.compare:- ours, why = evaluate_ours(args.file, entry["name"])+ ours, why = evaluate_ours(args.file, entry["name"], library)if why:print("cpuref: ours could not be evaluated: %s" % why)return 1@@ -372,10 +491,10 @@ def draw_ours(path, function):return (values, None) if values else (None, "air_run printed no pixel")-def evaluate_ours(path, function):+def evaluate_ours(path, function, library=False):"""air2nir --eval: the same shader folded over the same buffer bytes."""- out = subprocess.run([AIR2NIR, "--eval", path, function],- capture_output=True, text=True)+ cmd = [AIR2NIR] + (["--library"] if library else []) + ["--eval", path, function]+ out = subprocess.run(cmd, capture_output=True, text=True)if out.returncode:return None, (out.stderr or out.stdout).strip().splitlines()[-1][:160]values = []diff --git a/tools/air/libcheck.py b/tools/air/libcheck.pynew file mode 100644index 0000000..0f822f0--- /dev/null+++ b/tools/air/libcheck.py@@ -0,0 +1,264 @@+#!/usr/bin/env python3+"""The 135 library functions: how far each one gets, and against what.++An air.visible function is not a shader. Metal links one into a caller at+pipeline time through the ABI its own metadata describes (air.stitching_info),+which this project does not implement -- so a third of the corpus had nothing+that could be dispatched and was counted only as "not an entry point".++air_to_nir's --library wraps one in a generated entry point instead: argument i+read from device buffer 0 at byte 16*i, the return value written to buffer 1+(wrap_library_function in air_to_nir.c). That makes a library function runnable+and comparable, and this script reports what each comparison is worth, because+they are not worth the same thing:++ LLVM the function compiled for this machine by llc and called with+ the same bytes. Independent of everything of ours, and the+ strongest evidence here. Available only for a function that+ calls no air.* at all: an air.* call is an undefined symbol+ on the host, and defining one would put our own reading of+ that operation on both sides of the comparison.+ folded vs run our NIR constant-folded on this machine against our NIR run+ by a Gallium driver. Not independent of the frontend -- a+ mistranslation is on both sides -- but it does catch the+ backend and the hardware disagreeing with the folder, which+ is how the ir3 u32-to-f32 rounding bug was found.++Both sides read argument i from byte 16*i, so neither comparison can catch a+slot layout that is wrong in the same way twice. What they do catch is the two+sides disagreeing about which slot is which argument, because the fill pattern+makes every slot's bytes different. Apple's own air.arg_type_name for each+argument is checked against the LLVM signature before anything runs+(cpuref.py, check_against_metadata), which is the one part of the layout that+has a second source.++usage: libcheck.py [--list] [FILE.metallib...]+ (no files: portrait_filters.metallib, the corpus's library)+"""++import os+import re+import struct+import subprocess+import sys++ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))+AIR2NIR = os.environ.get(+ "AIR2NIR", os.path.join(ROOT, "build/mesa-host/build/src/compiler/air/air2nir"))+AIR_RUN = os.environ.get(+ "AIR_RUN",+ os.path.join(ROOT, "build/mesa-host/build/src/gallium/drivers/freedreno/air_run"))+CPUREF = os.path.join(ROOT, "tools/air/cpuref.py")++LIBS = ["ios165/rootfs/System/Library/CoreImage/PortraitFilters.cifilter/"+ "portrait_filters.metallib"]++# The air.* operations whose result is exactly defined, by the family name in+# the declaration. The split matters: a function calling an approximate one+# cannot be compared bit-exactly against anything, here or anywhere, and saying+# so is not the same as saying it is wrong (docs/air.md section 4).+APPROX = {"fast_pow", "fast_powr", "fast_exp", "fast_exp2", "fast_exp10", "fast_log",+ "fast_log2", "fast_log10", "fast_sin", "fast_cos", "fast_tan", "fast_atan2",+ "fast_rsqrt", "fast_divide", "fast_asin", "fast_acos", "fast_atan",+ "fast_sinh", "fast_cosh", "fast_tanh"}+++def report(lib):+ """air2nir --library --report: one row per module of the library."""+ out = subprocess.run([AIR2NIR, "--library", "--report", lib],+ capture_output=True, text=True)+ rows = []+ for line in out.stdout.splitlines():+ if not line.startswith("module="):+ continue+ r = dict(kv.split("=", 1) for kv in line.split("\t") if "=" in kv)+ r["calls"] = [c for c in r.get("calls", "").split(",") if c]+ rows.append(r)+ return rows+++def families(row):+ return {c.split(".")[1] for c in row["calls"] if c.startswith("air.")}+++def folded(lib, name):+ """What our frontend folds the wrapper to, as 32-bit words."""+ out = subprocess.run([AIR2NIR, "--library", "--eval", lib, name],+ capture_output=True, text=True)+ if out.returncode:+ return None, "air2nir --eval failed"+ for line in out.stdout.splitlines():+ if not line.startswith(" buffer"):+ continue+ if "did not fold" in line:+ return None, "the store did not fold to a constant"+ return [bits_of(v) for v in line.split()[1:]], None+ return None, "no buffer store in the folded shader"+++def bits_of(text):+ """A float as printed by --eval, back to its 32 bits. NaN keeps no payload."""+ f = float(text)+ if f != f:+ return "nan"+ return struct.unpack("<I", struct.pack("<f", f))[0]+++def ran(lib, name):+ """What a Gallium driver writes into buffer 1, as 32-bit words."""+ out = subprocess.run([AIR_RUN, "--library", lib, name],+ capture_output=True, text=True)+ if out.returncode:+ last = (out.stderr or out.stdout).strip().splitlines()+ return None, (last[-1][:120] if last else "air_run failed")+ words, taking = [], False+ for line in out.stdout.splitlines():+ if line.startswith("air_run: device buffer 1 after"):+ taking = True+ continue+ if taking:+ m = re.match(r"\s*\[\s*\d+\]\s+\S+\s+0x([0-9a-f]+)", line)+ if not m:+ break+ words.append(int(m.group(1), 16))+ return (words, None) if words else (None, "air_run printed no buffer")+++def same(a, b):+ """Word for word, with any NaN equal to any NaN and nothing else loose."""+ if len(a) != len(b):+ return False+ for x, y in zip(a, b):+ if x == "nan" or y == "nan":+ # a NaN's payload is not specified by Metal and not preserved by+ # every lowering, so two NaNs agree and a NaN beside a number does+ # not. Counting them apart is the caller's job.+ if not (is_nan(x) and is_nan(y)):+ return False+ elif x != y:+ return False+ return True+++def is_nan(w):+ return w == "nan" or (isinstance(w, int) and (w & 0x7f800000) == 0x7f800000+ and (w & 0x7fffff))+++def llvm_agrees(lib, name):+ """cpuref.py --compare: LLVM on this machine against our folded NIR."""+ out = subprocess.run([sys.executable, CPUREF, "--compare", lib, name],+ capture_output=True, text=True)+ text = out.stdout + out.stderr+ if "they agree" in text:+ return True, None+ if "they DIFFER" in text:+ return False, text.strip().splitlines()[-1][:120]+ return None, text.strip().splitlines()[-1][:120] if text.strip() else "no output"+++def main(argv):+ want_list = "--list" in argv+ libs = [a for a in argv if not a.startswith("--")] or LIBS+ for prog in (AIR2NIR, AIR_RUN):+ if not os.path.exists(prog):+ print("libcheck: run tools/air/mk-host.sh first (%s)" % prog, file=sys.stderr)+ return 1++ rows = []+ for lib in libs:+ path = lib if os.path.isabs(lib) else os.path.join(ROOT, lib)+ if not os.path.exists(path):+ print("libcheck: missing %s" % lib, file=sys.stderr)+ return 1+ for r in report(path):+ r["path"] = path+ rows.append(r)++ wrapped = [r for r in rows if r.get("translate") == "ok"]+ refused = [r for r in rows if r.get("translate") != "ok"]++ plain, exact, approx = [], [], []+ for r in wrapped:+ fam = families(r)+ (plain if not fam else approx if fam & APPROX else exact).append(r)++ # what each one is worth, measured rather than assumed+ llvm_ok, llvm_bad, llvm_other = [], [], []+ for r in plain:+ ok, why = llvm_agrees(r["path"], r["module"])+ (llvm_ok if ok else llvm_bad if ok is False else llvm_other).append((r, why))++ run_ok, run_bad, no_fold, no_run, nan_rows = [], [], [], [], []+ for r in wrapped:+ theirs, why2 = ran(r["path"], r["module"])+ if theirs is None:+ no_run.append((r, why2))+ continue+ ours, why1 = folded(r["path"], r["module"])+ if ours is None:+ # it ran: what is missing is the host side of the comparison, not+ # the dispatch. Folding stops at a loop or at control flow the+ # folder cannot resolve, which says nothing about the shader.+ no_fold.append((r, why1))+ continue+ if any(is_nan(w) for w in ours):+ nan_rows.append(r)+ (run_ok if same(ours, theirs) else run_bad).append(+ (r, "folded %s, ran %s" % (ours, theirs)))++ print("libcheck: %d library functions in %d librar%s"+ % (len(rows), len(libs), "y" if len(libs) == 1 else "ies"))+ print(" %3d wrapped in a generated entry point and translated" % len(wrapped))+ print(" %3d refused: an argument the wrapper cannot pass" % len(refused))+ print()+ print(" of the %d wrapped, what a comparison can be worth:" % len(wrapped))+ print(" %3d call no air.* at all, so LLVM on this machine is a full reference"+ % len(plain))+ print(" %3d call only exactly defined air.*: no host definition for those yet"+ % len(exact))+ print(" %3d call an approximate air.*, which nothing compares bit-exactly"+ % len(approx))+ print()+ print(" %3d of the %d agree with LLVM bit for bit" % (len(llvm_ok), len(plain)))+ print(" %3d DIFFER from LLVM" % len(llvm_bad))+ print(" %3d could not be compared with LLVM" % len(llvm_other))+ print(" %3d of the %d dispatched on softpipe and read their result back"+ % (len(run_ok) + len(run_bad) + len(no_fold), len(wrapped)))+ print(" %3d of those match our own folding of the same shader" % len(run_ok))+ print(" %3d run but DIFFER from our own folding" % len(run_bad))+ print(" %3d ran, but our folding did not reduce to a constant, so there was "+ "nothing to compare" % len(no_fold))+ print(" %3d could not be dispatched at all" % len(no_run))+ if nan_rows:+ print(" (%d of them return a NaN, where only the quiet bit is compared)"+ % len(nan_rows))++ if want_list:+ for r in refused:+ print(" refused %-44s %s" % (r["module"], r.get("error", "")[:80]))+ for r, why in llvm_bad + run_bad + llvm_other + no_run + no_fold:+ print(" %-44s %s" % (r["module"], why))++ # the rows have to account for every module before the numbers are read+ if len(wrapped) + len(refused) != len(rows):+ print("libcheck: the rows add to %d and there are %d functions"+ % (len(wrapped) + len(refused), len(rows)), file=sys.stderr)+ return 1+ if len(run_ok) + len(run_bad) + len(no_fold) + len(no_run) != len(wrapped):+ print("libcheck: the run rows add to %d and %d were wrapped"+ % (len(run_ok) + len(run_bad) + len(no_fold) + len(no_run), len(wrapped)),+ file=sys.stderr)+ return 1+ if len(plain) + len(exact) + len(approx) != len(wrapped):+ print("libcheck: the reference rows add to %d and %d were wrapped"+ % (len(plain) + len(exact) + len(approx), len(wrapped)), file=sys.stderr)+ return 1+ print("libcheck: every function is in exactly one row of each group. "+ "Wrapping is not stitching: Metal's own ABI for these functions is not "+ "implemented, and nothing here is evidence about it.")+ return 1 if (llvm_bad or run_bad) else 0+++if __name__ == "__main__":+ sys.exit(main(sys.argv[1:]))diff --git a/tools/mesa/air/air2nir.c b/tools/mesa/air/air2nir.cindex 4676172..c4f89ea 100644--- a/tools/mesa/air/air2nir.c+++ b/tools/mesa/air/air2nir.c@@ -34,6 +34,12 @@#include "air_module.h"#include "util/ralloc.h"+/* --library: translate an air.visible library function inside a generated+ * entry point (nir_air.h). Off by default, because a wrapped function is not+ * an entry point of the corpus and must not be counted as one.+ */+static bool library_wrapper = false;+static const nir_shader_compiler_options default_options = {.lower_fdiv = false,};@@ -116,11 +122,36 @@ stage_name(int stage)* strongest one, and it needs no GPU.*/+/* The argument buffer of a wrapped library function holds floats, not the byte+ * pattern.+ *+ * Every one of these functions is compiled with no-nans-fp-math and+ * no-infs-fp-math: the module promises its inputs are finite. Arbitrary bytes+ * are mostly not -- they are infinities, NaNs and denormals -- so the inputs+ * would land outside the contract the shader was compiled under, where+ * 0 * inf may legally be either NaN or inf and a disagreement means nothing.+ * One case did disagree exactly there (folded NaN, softpipe +inf) and said+ * nothing about either side.+ *+ * Word w is ((w * 13) % 29 - 14) / 4: finite, exactly representable in+ * binary32, and different for 29 words running -- so a harness that mixed up+ * two slots, or two lanes of one slot, would get a different answer. cpuref.py+ * fills the same words.+ */+static uint8_t+library_byte(unsigned i)+{+ float f = (float)((int)((i / 4 * 13) % 29) - 14) / 4.0f;+ uint32_t u;+ memcpy(&u, &f, sizeof(u));+ return (u >> (8 * (i % 4))) & 0xff;+}+/* the same pattern cpuref.py fills its buffers with */static uint8_tpattern_byte(unsigned i){- return (i * 7 + 13) & 0xff;+ return library_wrapper ? library_byte(i) : (i * 7 + 13) & 0xff;}struct eval_state {@@ -206,6 +237,62 @@ print_output(nir_variable *var, nir_def *value)printf("\n");}+/* A wrapped library function returns its value by storing it to a buffer, not+ * to a shader output, so --eval has to read it from there. The type name comes+ * from Apple's own air.visible_output metadata, carried on the resource: how+ * to print the bytes is not something this can infer from a store.+ */+static unsigned+print_buffer_stores(nir_shader *nir, const struct air_shader_info *info)+{+ unsigned printed = 0;+ nir_foreach_function_impl(impl, nir) {+ nir_foreach_block(block, impl) {+ nir_foreach_instr(instr, block) {+ if (instr->type != nir_instr_type_intrinsic)+ continue;+ nir_intrinsic_instr *intr = nir_instr_as_intrinsic(instr);+ if (intr->intrinsic != nir_intrinsic_store_ssbo)+ continue;+ if (!nir_src_is_const(intr->src[1]))+ continue;+ unsigned index = nir_src_as_uint(intr->src[1]);+ const char *type_name = NULL;+ for (unsigned i = 0; i < info->num_resources; i++) {+ const struct air_resource *r = &info->resources[i];+ if (r->kind == AIR_RESOURCE_DEVICE_BUFFER && r->index == index && r->write)+ type_name = r->type_name;+ }+ nir_def *value = intr->src[0].ssa;+ printf(" buffer%u%*s", index, 18, "");+ if (!nir_src_is_const(nir_src_for_ssa(value))) {+ printf("did not fold to a constant\n");+ printed++;+ continue;+ }+ bool is_float = !type_name || strstr(type_name, "float") ||+ strstr(type_name, "half");+ for (unsigned c = 0; c < value->num_components; c++) {+ uint64_t bits = nir_src_comp_as_uint(nir_src_for_ssa(value), c);+ if (is_float && value->bit_size == 32) {+ float f;+ uint32_t u = bits;+ memcpy(&f, &u, 4);+ printf("%s%.9g", c ? " " : "", (double)f);+ } else if (is_float && value->bit_size == 16) {+ printf("%s0x%04" PRIx64, c ? " " : "", bits);+ } else {+ printf("%s%" PRIu64, c ? " " : "", bits);+ }+ }+ printf("\n");+ printed++;+ }+ }+ }+ return printed;+}+static inteval_module(const struct module_ref *mod){@@ -213,7 +300,8 @@ eval_module(const struct module_ref *mod)const char *err = NULL;struct air_constant_value *fc;unsigned num_fc = air_module_function_constants(ctx, mod->bitcode, mod->size, &fc);- struct air_to_nir_options options = { .constants = fc, .num_constants = num_fc };+ struct air_to_nir_options options = { .constants = fc, .num_constants = num_fc,+ .library_wrapper = library_wrapper };struct air_shader_info info;nir_shader *nir = air_to_nir(ctx, mod->bitcode, mod->size, NULL, &options,&default_options, &info, &err);@@ -251,6 +339,7 @@ eval_module(const struct module_ref *mod)printf("eval: the outputs this shader stores:\n");unsigned printed = 0;+ printed += print_buffer_stores(nir, &info);nir_foreach_function_impl(impl, nir) {nir_foreach_block(block, impl) {nir_foreach_instr(instr, block) {@@ -311,7 +400,8 @@ report_one(const struct module_ref *mod)*/struct air_constant_value *fc;unsigned num_fc = air_module_function_constants(ctx, mod->bitcode, mod->size, &fc);- struct air_to_nir_options options = { .constants = fc, .num_constants = num_fc };+ struct air_to_nir_options options = { .constants = fc, .num_constants = num_fc,+ .library_wrapper = library_wrapper };if (num_fc)printf("\tfc=%u", num_fc);@@ -343,6 +433,8 @@ main(int argc, char **argv){enum { TRANSLATE, LL, LIST, REPORT, EVAL } mode = TRANSLATE;int i = 1;+ while (i < argc && !strcmp(argv[i], "--library"))+ library_wrapper = true, i++;if (i < argc && !strcmp(argv[i], "--ll"))mode = LL, i++;else if (i < argc && !strcmp(argv[i], "--list"))@@ -353,7 +445,8 @@ main(int argc, char **argv)mode = EVAL, i++;if (i >= argc) {fprintf(stderr,- "usage: air2nir [--ll | --list | --report | --eval] FILE [FUNCTION]\n");+ "usage: air2nir [--library] [--ll | --list | --report | --eval] "+ "FILE [FUNCTION]\n");return 2;}@@ -416,7 +509,8 @@ main(int argc, char **argv)}struct air_constant_value *fc;unsigned num_fc = air_module_function_constants(ctx, mods[k].bitcode, mods[k].size, &fc);- struct air_to_nir_options options = { .constants = fc, .num_constants = num_fc };+ struct air_to_nir_options options = { .constants = fc, .num_constants = num_fc,+ .library_wrapper = library_wrapper };struct air_shader_info info;nir_shader *nir = air_to_nir(ctx, mods[k].bitcode, mods[k].size, NULL, &options,&default_options, &info, &err);diff --git a/tools/mesa/air/air_intrinsics.c b/tools/mesa/air/air_intrinsics.cindex 107acac..b9b5e31 100644--- a/tools/mesa/air/air_intrinsics.c+++ b/tools/mesa/air/air_intrinsics.c@@ -410,9 +410,8 @@ barrier_op(struct air_ctx *c, const struct air_inst *in)* ever passes 2 with scope 1 (the threadgroup).*/uint64_t flags = 2;- const struct air_constant *k =- in->num_operands > 1 ? air_value_as_constant(in->operands[1]) : NULL;- air_const_as_uint(k, &flags); /* leaves flags alone when it is not readable */+ if (in->num_operands > 1)+ air_const_uint(in->operands[1], &flags); /* left alone when not readable */nir_variable_mode modes = 0;if (flags & 1)@@ -441,13 +440,12 @@ llvm_op(struct air_ctx *c, const struct air_inst *in, const char *name)/* a copy between two of our pointers, of a constant length */struct air_val *dst = air_get(c, in->operands[1]);struct air_val *src = air_get(c, in->operands[2]);- const struct air_constant *len = air_value_as_constant(in->operands[3]);if (!dst || !src)return false;if (dst->kind != AIR_VAL_PTR || src->kind != AIR_VAL_PTR)return air_error(c, "intrinsic: memcpy that is not between pointers");uint64_t bytes;- if (!air_const_as_uint(len, &bytes))+ if (!air_const_uint(in->operands[3], &bytes))return air_error(c, "intrinsic: memcpy of a run-time length");/* in 4-byte pieces, which is what every memcpy in the corpus is a* multiple of (they copy float vectors out of constant buffers)@@ -467,13 +465,12 @@ llvm_op(struct air_ctx *c, const struct air_inst *in, const char *name)if (!strncmp(name, "llvm.memset.", 12)) {struct air_val *dst = air_get(c, in->operands[1]);nir_def *byte = air_get_ssa(c, in->operands[2]);- const struct air_constant *len = air_value_as_constant(in->operands[3]);if (!dst || !byte)return false;if (dst->kind != AIR_VAL_PTR)return air_error(c, "intrinsic: memset of something that is not a pointer");uint64_t bytes;- if (!air_const_as_uint(len, &bytes) || bytes % 4)+ if (!air_const_uint(in->operands[3], &bytes) || bytes % 4)return air_error(c, "intrinsic: memset of a length that is not a multiple of 4");if (!nir_src_is_const(nir_src_for_ssa(byte)))return air_error(c, "intrinsic: memset of a run-time byte");diff --git a/tools/mesa/air/air_module.h b/tools/mesa/air/air_module.hindex e791e6f..7ea7c95 100644--- a/tools/mesa/air/air_module.h+++ b/tools/mesa/air/air_module.h@@ -316,32 +316,6 @@ air_value_as_constant(const struct air_value *v)return v && v->kind == AIR_VALUE_CONSTANT ? (const struct air_constant *)v : NULL;}-/* An integer constant's value, whichever way the writer spelled it.- *- * LLVM has two encodings for a zero of a scalar type: CST_CODE_INTEGER with a- * value of 0, and CST_CODE_NULL. Which one a module uses is not free -- a zero- * that appears inside an aggregate constant is always NULL -- so anything that- * reads an integer constant has to accept both. Taking only INTEGER is why- * three of Apple's kernels stopped on a shuffle mask whose first lane was 0,- * and the same trap sits under every other reader of a constant integer.- */-static inline bool-air_const_as_uint(const struct air_constant *c, uint64_t *out)-{- if (!c)- return false;- if (c->ckind == AIR_CONST_INT) {- *out = c->u;- return true;- }- if (c->ckind == AIR_CONST_NULL && c->base.type &&- c->base.type->kind == AIR_TYPE_INT) {- *out = 0;- return true;- }- return false;-}-static inline const struct air_inst *air_value_as_inst(const struct air_value *v){diff --git a/tools/mesa/air/air_run.c b/tools/mesa/air/air_run.cindex cc02281..2697b14 100644--- a/tools/mesa/air/air_run.c+++ b/tools/mesa/air/air_run.c@@ -153,13 +153,41 @@ watch_for_faults(void)static void watch_for_faults(void) {}#endif+/* set by --library: the fill below is then floats, for the reason given there */+static bool library_fill = false;++/* The argument buffer of a wrapped library function holds floats, not the byte+ * pattern.+ *+ * Every one of these functions is compiled with no-nans-fp-math and+ * no-infs-fp-math: the module promises its inputs are finite. Arbitrary bytes+ * are mostly not -- they are infinities, NaNs and denormals -- so the inputs+ * would land outside the contract the shader was compiled under, where+ * 0 * inf may legally be either NaN or inf and a disagreement means nothing.+ * One case did disagree exactly there (folded NaN, softpipe +inf) and said+ * nothing about either side.+ *+ * Word w is ((w * 13) % 29 - 14) / 4: finite, exactly representable in+ * binary32, and different for 29 words running -- so a harness that mixed up+ * two slots, or two lanes of one slot, would get a different answer. cpuref.py+ * fills the same words.+ */+static uint8_t+library_byte(unsigned i)+{+ float f = (float)((int)((i / 4 * 13) % 29) - 14) / 4.0f;+ uint32_t u;+ memcpy(&u, &f, sizeof(u));+ return (u >> (8 * (i % 4))) & 0xff;+}+/* the pattern cpuref.py and air2nir --eval fill their buffers with, so that* all three agree on what the shader was given*/static uint8_tpattern_byte(unsigned i){- return (i * 7 + 13) & 0xff;+ return library_fill ? library_byte(i) : (i * 7 + 13) & 0xff;}/* Fill a buffer the shader is about to read.@@ -728,6 +756,11 @@ main(int argc, char **argv)* creation, so it has to be set before the screen is made.*/bool want_disasm = false;+ /* --library runs an air.visible library function through the generated+ * wrapper (nir_air.h): there is nothing to dispatch otherwise, so these 135+ * modules were never run at all.+ */+ bool want_library = false;/* --buffer FILE is AIR_RUN_BUFFER as an argument, and it exists because the* device has no shell to set one with. iOS 16.5's own root carries no sh at* all -- /bin holds df and ps -- and the job list that runs this is read by@@ -742,6 +775,10 @@ main(int argc, char **argv)setenv("IR3_SHADER_DEBUG", "disasm", 1);argv++;argc--;+ } else if (!strcmp(argv[1], "--library")) {+ want_library = library_fill = true;+ argv++;+ argc--;} else if (!strcmp(argv[1], "--buffer") && argc > 2) {setenv("AIR_RUN_BUFFER", argv[2], 1);argv += 2;@@ -751,7 +788,7 @@ main(int argc, char **argv)}}if (argc < 2) {- fprintf(stderr, "usage: air_run [--disasm] [--buffer FILE] "+ fprintf(stderr, "usage: air_run [--disasm] [--library] [--buffer FILE] ""FILE[.metallib] [FUNCTION]\n");return 2;}@@ -861,7 +898,8 @@ main(int argc, char **argv)/* the shader */struct air_constant_value *fc;unsigned num_fc = air_module_function_constants(ctx, bitcode, bitcode_size, &fc);- struct air_to_nir_options air_options = { .constants = fc, .num_constants = num_fc };+ struct air_to_nir_options air_options = { .constants = fc, .num_constants = num_fc,+ .library_wrapper = want_library };struct air_shader_info info;const char *err;nir_shader *fs = air_to_nir(ctx, bitcode, bitcode_size, NULL, &air_options, options,diff --git a/tools/mesa/air/air_to_nir.c b/tools/mesa/air/air_to_nir.cindex 61cbc0e..26478c5 100644--- a/tools/mesa/air/air_to_nir.c+++ b/tools/mesa/air/air_to_nir.c@@ -1147,10 +1147,12 @@ translate_instruction(struct air_ctx *c, const struct air_inst *in)/* The mask is an aggregate rather than plain data whenever a lane* is poison, so <i32 0, i32 poison, ...> -- what a Metal* half2-to-half4 widening compiles to -- mixes LLVM's two- * spellings of a constant integer. air_const_as_uint reads both.+ * spellings of a constant integer. air_const_uint reads both; this+ * code used to test ckind == AIR_CONST_INT by hand instead, which+ * is the whole bug.*/uint64_t lane = 0;- bool have = e && air_const_as_uint(e, &lane);+ bool have = air_const_uint(mk->elems[i], &lane);if (!e || (!have && e->ckind != AIR_CONST_UNDEF &&e->ckind != AIR_CONST_POISON))return air_error(c, "instruction: element %u of a shuffle mask is %s",@@ -1593,6 +1595,108 @@ air_call_in_module(struct air_ctx *c, const struct air_inst *in, const char *namreturn true;}+/* An air.visible library function, inside an entry point we generate for it.+ *+ * A library function is not a shader. Metal links one into a caller at+ * pipeline time through the ABI its own metadata describes -- air.stitching_info+ * names the four functions that do it (load_argument, copy_from_buffer,+ * store_return_value, destroy) and gives each argument's size and alignment --+ * and none of that is implemented here: it belongs to the runtime, not to the+ * compiler. So 135 modules of the corpus, a third of it, had nothing that+ * could be dispatched and were counted only as refused.+ *+ * This wraps one instead: argument i is read from device buffer 0 at byte+ * 16*i, the function is called, and the return value is written to device+ * buffer 1 at offset 0, by a single thread. The 16-byte slot is Apple's own+ * size and alignment for a float4 (the air.vector_type node beside each+ * stitching argument says 16 and 16), and a fixed slot makes the layout+ * derivable from the signature alone -- which is what lets a harness on this+ * machine fill the same bytes and compare the result.+ *+ * What that measures is the body: its arithmetic, its control flow, its calls.+ * What it does not touch is the stitching ABI, so a wrapped function is+ * reported in a row of its own and never counted among the corpus's entry+ * points.+ */+static bool+wrap_library_function(struct air_ctx *c, const struct air_md *out_md)+{+ const struct air_function *f = c->fn;++ /* The result buffer carries the function's own air.arg_type_name ("float4"+ * and the like, from air.visible_output): a harness that fills the argument+ * buffer has to know how to read the bytes that come back, and Apple's name+ * for the type is a better source for that than our reading of the LLVM+ * type -- the two can then be checked against each other.+ */+ add_resource(c, AIR_RESOURCE_DEVICE_BUFFER, 0, 16 * MAX2(f->num_args, 1),+ true, false, ~0u, NULL);+ add_resource(c, AIR_RESOURCE_DEVICE_BUFFER, 1, 16, false, true, ~0u, out_md);++ struct air_val **argv = rzalloc_array(c->mem_ctx, struct air_val *,+ MAX2(f->num_args, 1));+ bool *is_param = rzalloc_array(c->mem_ctx, bool, MAX2(f->num_args, 1));++ for (unsigned i = 0; i < f->num_args; i++) {+ const struct air_type *t = f->args[i].type;+ const struct air_type *e = t->kind == AIR_TYPE_VECTOR ? t->elem : t;+ bool numeric = e && (e->kind == AIR_TYPE_FLOAT || e->kind == AIR_TYPE_HALF ||+ e->kind == AIR_TYPE_INT);+ if (!numeric || (t->kind != AIR_TYPE_VECTOR && t->kind != AIR_TYPE_FLOAT &&+ t->kind != AIR_TYPE_HALF && t->kind != AIR_TYPE_INT))+ return air_error(c, "library: argument %u is neither a scalar nor a vector, "+ "and this wrapper passes each argument in a 16-byte slot "+ "of one buffer", i);+ if (air_nir_components(t) * air_nir_bit_size(c, t) > 16 * 8)+ return air_error(c, "library: argument %u does not fit the wrapper's 16-byte "+ "slot", i);++ struct air_ptr p = { .class = AIR_PTR_SSBO, .binding = 0, .pointee = t };+ nir_def *v = air_load_ptr(c, air_ptr_add(c, p, nir_imm_int(&c->b, 16 * i)), t, 16);+ if (!v)+ return false;+ argv[i] = air_val_ssa(c, v);+ is_param[i] = true;+ }++ const struct air_type *rt = f->fn_type->elem;+ const struct air_type *re = rt && rt->kind == AIR_TYPE_VECTOR ? rt->elem : rt;+ if (!rt || rt->kind == AIR_TYPE_VOID)+ return air_error(c, "library: the function returns nothing, so a wrapper would "+ "have nothing to write");+ if (!re || (re->kind != AIR_TYPE_FLOAT && re->kind != AIR_TYPE_HALF &&+ re->kind != AIR_TYPE_INT))+ return air_error(c, "library: the return value is neither a scalar nor a vector");++ nir_function *fn = translate_callee(c, f, argv, is_param);+ if (!fn)+ return false; /* translate_callee has said why */++ unsigned bits = air_nir_bit_size(c, rt);+ unsigned comps = air_nir_components(rt);+ const struct glsl_type *gt =+ glsl_vector_type(bits == 16 ? GLSL_TYPE_FLOAT16 :+ bits == 64 ? GLSL_TYPE_UINT64 : GLSL_TYPE_UINT, comps);+ nir_variable *tmp = nir_local_variable_create(c->b.impl, gt, "libret");++ nir_call_instr *call = nir_call_instr_create(c->nir, fn);+ unsigned idx = 0;+ call->params[idx++] = nir_src_for_ssa(&nir_build_deref_var(&c->b, tmp)->def);+ for (unsigned i = 0; i < f->num_args; i++)+ call->params[idx++] = nir_src_for_ssa(argv[i]->def);+ nir_builder_instr_insert(&c->b, &call->instr);++ struct air_ptr out = { .class = AIR_PTR_SSBO, .binding = 1, .pointee = rt };+ if (!air_store_ptr(c, out, nir_load_var(&c->b, tmp), 16))+ return false;++ /* the generated block ends the way a translated one does: this impl is+ * unstructured until nir_lower_goto_ifs runs over it+ */+ nir_goto(&c->b, c->b.impl->end_block);+ return true;+}+static booltranslate_body(struct air_ctx *c){@@ -1912,14 +2016,22 @@ air_to_nir(void *mem_ctx, const void *bitcode, size_t size, const char *entry_poralloc_free(ctx);return NULL;}+ /* air.visible: a library function, linked into its caller at pipeline time+ * (docs/air.md section 7.5). Not an entry point, so not a shader -- unless+ * the caller asks for the generated wrapper, which makes one compute shader+ * whose only job is to call it (wrap_library_function).+ */+ bool wrap = false;if (stage == MESA_SHADER_NONE) {- /* air.visible: a library function, linked into its caller at pipeline- * time (docs/air.md section 7.5). Not an entry point, so not a shader.- */- if (error)- *error = ralloc_strdup(mem_ctx, "stage: air.visible, a library function (not yet)");- ralloc_free(ctx);- return NULL;+ if (!options || !options->library_wrapper) {+ if (error)+ *error = ralloc_strdup(mem_ctx,+ "stage: air.visible, a library function (not yet)");+ ralloc_free(ctx);+ return NULL;+ }+ stage = MESA_SHADER_COMPUTE;+ wrap = true;}if (!entry->num_ops || !entry->ops[0] || entry->ops[0]->kind != AIR_MD_VALUE ||@@ -1965,10 +2077,15 @@ air_to_nir(void *mem_ctx, const void *bitcode, size_t size, const char *entry_poc.info->stage = stage;c.info->entry_point = fn->base.name;- /* the outputs: fields of the returned struct, in order */- const struct air_md *out_node = entry->num_ops > 1 ? entry->ops[1] : NULL;+ /* the outputs: fields of the returned struct, in order. A wrapped library+ * function has none of this: its result is a value the wrapper stores to a+ * buffer, not a shader output, and its own metadata uses air.visible_input+ * and air.visible_output rather than the keys translate_input reads.+ */+ const struct air_md *out_node = (!wrap && entry->num_ops > 1) ? entry->ops[1] : NULL;const struct air_type *rt = fn->fn_type->elem;- unsigned num_fields = rt->kind == AIR_TYPE_STRUCT ? rt->num_members :+ unsigned num_fields = wrap ? 0 :+ rt->kind == AIR_TYPE_STRUCT ? rt->num_members :rt->kind == AIR_TYPE_VOID ? 0 : 1;c.num_outputs = num_fields;c.outputs = rzalloc_array(ctx, __typeof__(*c.outputs), MAX2(num_fields, 1));@@ -1989,12 +2106,12 @@ air_to_nir(void *mem_ctx, const void *bitcode, size_t size, const char *entry_po}/* the inputs */- const struct air_md *in_node = entry->num_ops > 2 ? entry->ops[2] : NULL;+ const struct air_md *in_node = (!wrap && entry->num_ops > 2) ? entry->ops[2] : NULL;for (unsigned i = 0; i < (in_node ? in_node->num_ops : 0) && !c.error[0]; i++)translate_input(&c, in_node->ops[i], &next_in_varying);/* the entry point's parameters, as the body will read them */- for (unsigned i = 0; i < c.num_params && !c.error[0]; i++) {+ for (unsigned i = 0; i < (wrap ? 0 : c.num_params) && !c.error[0]; i++) {if (c.params[i])air_set(&c, &fn->args[i], c.params[i]);else@@ -2008,8 +2125,17 @@ air_to_nir(void *mem_ctx, const void *bitcode, size_t size, const char *entry_poif (!c.error[0])run_static_initializers(&c);- if (!c.error[0])- translate_body(&c);+ if (!c.error[0]) {+ if (wrap) {+ /* air.visible's second operand is the result node, whose one entry+ * carries air.visible_output and the type's name+ */+ const struct air_md *rn = entry->num_ops > 1 ? entry->ops[1] : NULL;+ wrap_library_function(&c, rn && rn->num_ops ? rn->ops[0] : NULL);+ }+ else+ translate_body(&c);+ }if (c.error[0]) {if (error)diff --git a/tools/mesa/air/nir_air.h b/tools/mesa/air/nir_air.hindex 2a69ab5..5e24918 100644--- a/tools/mesa/air/nir_air.h+++ b/tools/mesa/air/nir_air.h@@ -101,6 +101,17 @@ struct air_constant_value {struct air_to_nir_options {const struct air_constant_value *constants;unsigned num_constants;++ /* Translate an air.visible library function by generating an entry point+ * that calls it: argument i read from device buffer 0 at byte 16*i, the+ * return value written to device buffer 1 at offset 0, one thread. Metal's+ * own way of using these functions is to stitch them into a caller at+ * pipeline time (air.stitching_info in their metadata), which this frontend+ * does not implement; the wrapper exists so that a library function's body+ * can be run and compared at all. A shader produced this way is not an+ * entry point of the corpus and is counted separately.+ */+ bool library_wrapper;};/* Translates one AIR module (one function of a .metallib). entry_point may