Edit photos by describing the change, on your own GPU. A free, open-source alternative to Nano Banana and ChatGPT image editing: photos stay on your computer, no daily limit. One installer for Windows, Mac and Linux sets up the best open model (Qwen-Image 2.1). MCP server for AI agents.
TypeScript ai-photo-editorcomfyui-alternativegenerative-fillimage-editingimage-generationinpaintinglinuxlocal-aimacosmcpmcp-servernano-banana-alternativeofflinephoto-editorprivacyqwen-imageqwen-image-2-1stable-diffusion-cppvulkanwindows
Mac engine: the target's attention on the M5's neural accelerators (#50)
Cesare Davide Pacef4233ef8793b
diff --git a/engine-mac/README.md b/engine-mac/README.mdindex b34b64f..85d1ff4 100644--- a/engine-mac/README.md+++ b/engine-mac/README.md@@ -63,7 +63,7 @@ the kernels were found.| `EngineServer.swift`, `HTTPServer.swift` | `serve`: sd-server's `/sdcpp/v1` API (capabilities, img_gen, jobs, cancel, text_gen) and its log lines || `QwenTextGeneration.swift`, `Kernels/decode.metal` | Prompt+: the text encoder's LLM writes, K/V cache, products on the Q4_K / Q6_K blocks, sd.cpp's sampler || `Chip.swift` | this Mac's chip, GPU family, cores and memory; each unit's estimated speed against the M2 Pro's |-| `NeuralAccelerators.swift`, `Kernels/metal4/na_matmul.metal` | the GPU's rows of a product on Metal 4 tensor operations (the M5's neural accelerators) |+| `NeuralAccelerators.swift`, `Kernels/metal4/na_matmul.metal`, `Kernels/metal4/na_attention.metal` | the GPU's rows of a product and the target's attention on Metal 4 tensor operations (the M5's neural accelerators) || `Bench.swift` | `bench`: the report testers paste in GitHub |Tests check the CPU and GPU unpacking against gguf-py's values (`tools/make_quant_fixtures.py` writes the@@ -432,6 +432,16 @@ correctness test that runs on any Mac (`PerChipTests`). Only the M2 Pro has meas**On the M5 Pro they are the neural accelerators**, and MPS reaches them too: 31.4 against 32.2 TFLOP/s at1536 × 4096 × 4096, the same result to the bit, against at most ~10 for its 20 GPU cores without them; MLX 0.32reaches 28–31 on the DiT's shapes. So `KOKON_NA=0` does not turn them off; it was ahead because of the tile below.+- **The target's attention on the neural accelerators** (`na_flash_attention`, `KOKON_NA_ATTENTION=0` off, `=1` on+ any Metal 4 GPU). With the products on the accelerators, the attention was half of a step on the M5 Pro+ (`KOKON_PROFILE` at 1024²: products 1.92 s of an evaluation, attention 1.91 s), on the shader cores' 8×8 SIMD-group+ matrices at ~4.6 TFLOP/s. The new kernel does Q·Kᵀ and P·V with MPP's `matmul2d` (each SIMD group 16 query rows,+ tiles of 64 keys, the softmax on MPP's row reductions), on the same packed fp16 K and V: `probe-attention` 60.0 →+ 28.1 ms, head 0 6.3 · 10⁻⁴ from exact for both. Chosen by capability (Apple10 GPU and Metal 4), so the M5, M5 Pro+ and M5 Max; M1–M4 keep the engine's kernel. M5 Pro 20-core, 24 GB, macOS 27.2, 3 alternated rounds, median:+ `time-dit` third evaluation 3.64 → 2.99 s; test set 1024² 33.3 → 28.9 s, 1344×768 33.7 → 27.7 s, edit 36.2 →+ 31.3 s, faster in every round; bench velocity error 4.4 · 10⁻⁴ (3.0 · 10⁻⁴ with it off in the same run, 4.1 ·+ 10⁻⁴ the run before). 2 SIMD groups of 32 rows, 8 of 8 rows or tiles of 32 keys were 1.7–3× slower.- **2 SIMD groups per `na_matmul` tile on the M5 Pro** (`KOKON_NA_SG2=0` off, `=1` on elsewhere). With 4, the MLP'sout (2048 × 12288 × 4096) took 9.6 ms against MPS's 6.7 and 4096 × 4096 × 4096 6.0 against 4.4 (`check-na`, bestof 5); with 2, 7.0 and 4.4, the same result to the bit. A sweep of tiles 32–128 and 1–8 SIMD groups found nodiff --git a/engine-mac/Sources/KokonEngine/AttentionProbe.swift b/engine-mac/Sources/KokonEngine/AttentionProbe.swiftindex 25b5470..74aef11 100644--- a/engine-mac/Sources/KokonEngine/AttentionProbe.swift+++ b/engine-mac/Sources/KokonEngine/AttentionProbe.swift@@ -99,6 +99,23 @@ extension Probe {print(String(format: "GPU %@%@, 32 heads: %.1f ms, %.2f ms/head, %.2f TFLOP/s; head 0 rel %.1e", kernel, note,gpu * 1e3, gpu * 1e3 / Double(heads), flops / gpu / 1e12, gpuError))}+ // The same packed attention on Metal 4 tensor operations (the M5's neural accelerators), where it compiles.+ if let na = NeuralAttention.make(device: context.device) {+ let gpu = try best(3) {+ let commands = context.queue.makeCommandBuffer()!+ try pack(commands)+ try na.encode(q: (qb, offset * stride * 4), kp: (kp, 0), vp: (vp, 0), out: (ob, offset * stride * 4),+ queries: queries, keys: keys, paddedKeys: paddedKeys, heads: heads, stride: stride,+ scale: scale * Float(M_LOG2E), commands: commands)+ return try MetalContext.finish(commands, "na attention")+ }+ let flops: Double = 4 * Double(queries) * Double(keys) * Double(d) * Double(heads)+ print(String(format: "GPU na_flash_attention (with packing)%@, 32 heads: %.1f ms, %.2f ms/head, %.2f TFLOP/s; head 0 rel %.1e",+ NeuralAttention.selected(device: context.device) ? " (selected)" : "", gpu * 1e3, gpu * 1e3 / Double(heads),+ flops / gpu / 1e12, error { i, c in f(ob)[(offset + i) * stride + c] }))+ } else {+ print("GPU na_flash_attention unavailable: \(NeuralAttention.unavailable ?? "?")")+ }if ProcessInfo.processInfo.environment["KOKON_PROBE_GPU_ONLY"] == "1" { return }// The Neural Engine.diff --git a/engine-mac/Sources/KokonEngine/Bench.swift b/engine-mac/Sources/KokonEngine/Bench.swiftindex d4fd365..cffab33 100644--- a/engine-mac/Sources/KokonEngine/Bench.swift+++ b/engine-mac/Sources/KokonEngine/Bench.swift@@ -31,6 +31,9 @@ public enum Bench {paths.append(("those products on 2 SIMD groups per tile (M5 Pro)", "KOKON_NA_SG2", "0"))}}+ if NeuralAttention.selected(device: device), NeuralAttention.make(device: device) != nil {+ paths.append(("the target's attention on Metal 4 tensor operations (the M5's neural accelerators)", "KOKON_NA_ATTENTION", "0"))+ }if Coexecutor.flushReleaseSelected(context) {paths.append(("each product released to the AMX and the Neural Engine at once (this Mac's signals arrive late)", "KOKON_FLUSH_READY", "0"))}diff --git a/engine-mac/Sources/KokonEngine/Kernels/metal4/na_attention.metal b/engine-mac/Sources/KokonEngine/Kernels/metal4/na_attention.metalnew file mode 100644index 0000000..ed9e3b2--- /dev/null+++ b/engine-mac/Sources/KokonEngine/Kernels/metal4/na_attention.metal@@ -0,0 +1,124 @@+#include <metal_stdlib>+#include <metal_tensor>+#include <MetalPerformancePrimitives/MetalPerformancePrimitives.h>+using namespace metal;+using namespace mpp::tensor_ops;++// The target's attention with its two products, Q·Kᵀ and P·V, on Metal 4 tensor operations (MetalPerformancePrimitives'+// matmul2d): on the M5 they run on the GPU's neural accelerators, where `flash_attention_packed`'s 8×8 SIMD-group+// matrices run on the shader cores. Compiled apart from the other kernels (Metal 4, `NeuralAccelerators.swift`).+//+// The same interface as `flash_attention_packed` (`attention.metal`), not causal: q fp32 rows of `stride` floats (head+// h at columns [128 h, 128 h + 128)), K and V packed by `attention_pack_kv` (fp16, `[head][key][128]`, keys padded with+// zeros to `paddedKeys`, a multiple of 64), out fp32 rows like q. The same arithmetic too: q, K, V and the+// probabilities in fp16, scores, softmax and O accumulated in fp32, an online softmax over tiles of 64 keys.+//+// Each SIMD group owns 16 query rows (MPP's row reductions work within one SIMD group), 4 per threadgroup: 64 rows,+// as `flash_attention_packed` (rows past `queries` are computed, never stored). Its Q rows (fp16) and probabilities sit+// in threadgroup memory as tensor operands; the scores and O stay in cooperative tensors, in registers; each row's+// running max, sum and rescale in threadgroup memory, read back by the elements of that row.+//+// M5 Pro 20-core, macOS 27.2, 1024² (4 096 queries × 4 128 keys, 32 heads): 27.4 ms against 59.7 for+// `flash_attention_packed` with its packing, 4.8e-4 from an fp64 attention of the same fp16 K and V. 2 SIMD groups of+// 32 rows 48.0 ms, 8 of 8 81.6, tiles of 32 keys 48.8 (median of 7).++struct FlashParams {+ uint queries;+ uint keys;+ uint causal;+ uint stride;+ float scale; // 1/sqrt(head_dim) * log2(e)+ uint headBase;+};++constant constexpr int ND = 128;++template <int SG, int MQ, int NK>+kernel void na_flash_attention(device const float *q [[buffer(0)]], device half *kp [[buffer(1)]], device half *vp [[buffer(2)]],+ device float *out [[buffer(3)]], constant FlashParams &p [[buffer(4)]],+ constant uint &paddedKeys [[buffer(5)]],+ uint2 tg [[threadgroup_position_in_grid]], uint sg [[simdgroup_index_in_threadgroup]],+ uint lane [[thread_index_in_simdgroup]]) {+ threadgroup half qsAll[SG * MQ * ND];+ threadgroup half psAll[SG * MQ * NK];+ threadgroup float statsAll[SG * 4 * MQ]; // per row: max, sum, alpha, block value+ threadgroup half *qs = qsAll + sg * MQ * ND;+ threadgroup half *ps = psAll + sg * MQ * NK;+ threadgroup float *rowMax = statsAll + sg * 4 * MQ, *rowSum = rowMax + MQ, *rowAlpha = rowSum + MQ, *block = rowAlpha + MQ;++ const uint head = p.headBase + tg.y, q0 = (tg.x * SG + sg) * MQ;+ for (uint i = lane; i < MQ * ND; i += 32) {+ qs[i] = half(q[(ulong)(q0 + i / ND) * p.stride + head * ND + i % ND]);+ }+ if (lane < MQ) { rowMax[lane] = -INFINITY; rowSum[lane] = 0.0f; }+ simdgroup_barrier(mem_flags::mem_threadgroup);++ auto tQ = tensor<threadgroup half, extents<int32_t, ND, MQ>, tensor_inline>(qs, extents<int32_t, ND, MQ>());+ auto tP = tensor<threadgroup half, extents<int32_t, NK, MQ>, tensor_inline>(ps, extents<int32_t, NK, MQ>());+ device half *kh = kp + (ulong)head * paddedKeys * ND;+ device half *vh = vp + (ulong)head * paddedKeys * ND;+ auto tK = tensor<device half, dextents<int32_t, 2>, tensor_inline>(kh, dextents<int32_t, 2>(ND, paddedKeys));+ auto tV = tensor<device half, dextents<int32_t, 2>, tensor_inline>(vh, dextents<int32_t, 2>(ND, paddedKeys));++ constexpr auto qk = matmul2d_descriptor(MQ, NK, ND, false, true, false, matmul2d_descriptor::mode::multiply);+ matmul2d<qk, execution_simdgroup> opS;+ constexpr auto pv = matmul2d_descriptor(MQ, ND, NK, false, false, false, matmul2d_descriptor::mode::multiply_accumulate);+ matmul2d<pv, execution_simdgroup> opO;+ using KT = remove_addrspace_t<decltype(tK.slice(0, 0))>;+ using VT = remove_addrspace_t<decltype(tV.slice(0, 0))>;++ auto S = opS.template get_destination_cooperative_tensor<decltype(tQ), KT, float>();+ auto rows = opS.template get_row_reduction_destination_cooperative_tensor<decltype(tQ), KT, float>();+ auto O = opO.template get_destination_cooperative_tensor<decltype(tP), VT, float>();+ for (uint16_t i = 0; i < O.get_capacity(); ++i) if (O.is_valid_element(i)) O[i] = 0.0f;++ for (uint j0 = 0; j0 < p.keys; j0 += NK) {+ auto mK = tK.slice(0, int(j0));+ opS.run(tQ, mK, S);+ for (uint16_t i = 0; i < S.get_capacity(); ++i) {+ if (!S.is_valid_element(i)) continue;+ S[i] = j0 + uint(S.get_multidimensional_index(i)[0]) < p.keys ? S[i] * p.scale : -INFINITY;+ }+ reduce_rows(S, rows, reduction_operation::max, reduction_operation_identity<float>::max_identity);+ for (uint16_t i = 0; i < rows.get_capacity(); ++i) {+ if (rows.is_valid_element(i)) block[rows.get_multidimensional_index(i)[0]] = rows[i];+ }+ simdgroup_barrier(mem_flags::mem_threadgroup);+ if (lane < MQ) {+ float mOld = rowMax[lane], mNew = max(mOld, block[lane]);+ rowAlpha[lane] = (mNew == -INFINITY) ? 1.0f : exp2(mOld - mNew);+ rowMax[lane] = mNew;+ }+ simdgroup_barrier(mem_flags::mem_threadgroup);+ for (uint16_t i = 0; i < S.get_capacity(); ++i) {+ if (!S.is_valid_element(i)) continue;+ auto at = S.get_multidimensional_index(i); // (key, query) within the tile+ float m = rowMax[at[1]];+ float e = (m == -INFINITY) ? 0.0f : exp2(S[i] - m);+ S[i] = e;+ ps[at[1] * NK + at[0]] = half(e);+ }+ reduce_rows(S, rows, reduction_operation::sum, reduction_operation_identity<float>::sum_identity);+ for (uint16_t i = 0; i < rows.get_capacity(); ++i) {+ if (rows.is_valid_element(i)) block[rows.get_multidimensional_index(i)[0]] = rows[i];+ }+ simdgroup_barrier(mem_flags::mem_threadgroup);+ if (lane < MQ) rowSum[lane] = rowSum[lane] * rowAlpha[lane] + block[lane];+ for (uint16_t i = 0; i < O.get_capacity(); ++i) {+ if (O.is_valid_element(i)) O[i] *= rowAlpha[O.get_multidimensional_index(i)[1]];+ }+ auto mV = tV.slice(0, int(j0));+ opO.run(tP, mV, O);+ simdgroup_barrier(mem_flags::mem_threadgroup);+ }++ for (uint16_t i = 0; i < O.get_capacity(); ++i) {+ if (!O.is_valid_element(i)) continue;+ auto at = O.get_multidimensional_index(i); // (d, query)+ uint qi = q0 + uint(at[1]);+ if (qi < p.queries) out[(ulong)qi * p.stride + head * ND + uint(at[0])] = O[i] / rowSum[at[1]];+ }+}++template [[host_name("na_flash_attention")]] kernel void na_flash_attention<4, 16, 64>(device const float *, device half *,+ device half *, device float *, constant FlashParams &, constant uint &, uint2, uint, uint);diff --git a/engine-mac/Sources/KokonEngine/NeuralAccelerators.swift b/engine-mac/Sources/KokonEngine/NeuralAccelerators.swiftindex 3f5ebfc..b8fcd84 100644--- a/engine-mac/Sources/KokonEngine/NeuralAccelerators.swift+++ b/engine-mac/Sources/KokonEngine/NeuralAccelerators.swift@@ -44,12 +44,7 @@ public final class NeuralAccelerators: @unchecked Sendable {return nil}do {- guard let url = EngineBundle.kernels?- .appendingPathComponent("metal4/na_matmul.metal") else { throw MetalContext.Failure.noKernelSource }- let options = MTLCompileOptions()- options.languageVersion = .version4_0- options.mathMode = .safe- let library = try device.makeLibrary(source: try String(contentsOf: url, encoding: .utf8), options: options)+ let library = try metal4Library(device: device, file: "na_matmul.metal")let groups = simdgroups ?? NeuralAccelerators.simdgroups(chip: Chip.current.name,setting: ProcessInfo.processInfo.environment["KOKON_NA_SG2"])let name = groups == 2 ? "na_matmul_sg2" : "na_matmul"@@ -61,6 +56,16 @@ public final class NeuralAccelerators: @unchecked Sendable {}}+ /// Compiles one of `Kernels/metal4/` (Metal 4, `mathMode = .safe` as the other kernels).+ @available(macOS 26, *)+ static func metal4Library(device: MTLDevice, file: String) throws -> MTLLibrary {+ guard let url = EngineBundle.kernels?.appendingPathComponent("metal4/\(file)") else { throw MetalContext.Failure.noKernelSource }+ let options = MTLCompileOptions()+ options.languageVersion = .version4_0+ options.mathMode = .safe+ return try device.makeLibrary(source: try String(contentsOf: url, encoding: .utf8), options: options)+ }+init(pipeline: MTLComputePipelineState, simdgroups: Int) {self.pipeline = pipelineself.simdgroups = simdgroups@@ -96,3 +101,82 @@ public final class NeuralAccelerators: @unchecked Sendable {e.endEncoding()}}++/// **The target's attention on the neural accelerators** (`Kernels/metal4/na_attention.metal`): Q·Kᵀ and P·V on Metal+/// 4's tensor operations, where `flash_attention_packed` uses 8×8 SIMD-group matrices on the shader cores. With the+/// products on the accelerators, the attention became half of a step on the M5 Pro (`KOKON_PROFILE`, 1024²: products+/// 1.92 s of an evaluation, the attention 1.91 s). The kernel alone at 1024²: 27.4 ms against 59.7 (20-core, macOS+/// 27.2). Same inputs as `flash_attention_packed` (K and V packed by `attention_pack_kv`), not causal: the text+/// segments' causal attentions stay on the engine's kernels.+///+/// Chosen by capability, not by name: GPUs of the Apple10 family (the M5's, with neural accelerators) with Metal 4.+/// Older Metal 4 GPUs (M1–M4) would run the same source on their shader cores, where it is not measured: they keep+/// `flash_attention_packed`. `KOKON_NA_ATTENTION=0` off, `=1` on any Metal 4 GPU (its results are checked there too).+public final class NeuralAttention: @unchecked Sendable {+ let pipeline: MTLComputePipelineState+ nonisolated(unsafe) public private(set) static var unavailable: String?+ /// Query rows per threadgroup: 4 SIMD groups of 16 (the kernel's), as `flash_attention_packed`'s 64.+ static let rowsPerGroup = 64, threads = 128++ /// Mirrors `FlashParams` in `na_attention.metal` (and `attention.metal`).+ struct Params { var queries, keys, causal, stride: UInt32; var scale: Float; var headBase: UInt32 }++ static func selected(gpuFamily: Int, metal4: Bool, setting: String?) -> Bool {+ switch setting {+ case "0": return false+ case "1": return metal4+ default: return metal4 && gpuFamily >= 10+ }+ }++ public static func selected(device: MTLDevice) -> Bool {+ var metal4 = false+ if #available(macOS 26, *) { metal4 = device.supportsFamily(.metal4) }+ return selected(gpuFamily: device.supportsFamily(.apple10) ? 10 : 0, metal4: metal4,+ setting: ProcessInfo.processInfo.environment["KOKON_NA_ATTENTION"])+ }++ /// Compiles the kernel; nil without Metal 4 or when the compiler refuses the source (the engine's kernels then).+ public static func make(device: MTLDevice) -> NeuralAttention? {+ guard #available(macOS 26, *), device.supportsFamily(.metal4) else {+ unavailable = "no Metal 4 on this GPU or macOS"+ return nil+ }+ do {+ let library = try NeuralAccelerators.metal4Library(device: device, file: "na_attention.metal")+ guard let function = library.makeFunction(name: "na_flash_attention") else {+ throw MetalContext.Failure.kernel("na_flash_attention not found")+ }+ return NeuralAttention(pipeline: try device.makeComputePipelineState(function: function))+ } catch {+ unavailable = "the kernel did not compile: \(error.localizedDescription)"+ return nil+ }+ }++ init(pipeline: MTLComputePipelineState) { self.pipeline = pipeline }++ /// Encodes heads `[0, heads)` of `queries` query rows against `keys` keys; q and out rows of `stride` floats, K and+ /// V packed with `paddedKeys` keys per head. `scale`: 1/√d · log2(e).+ func encode(q: (MTLBuffer, Int), kp: (MTLBuffer, Int), vp: (MTLBuffer, Int), out: (MTLBuffer, Int),+ queries: Int, keys: Int, paddedKeys: Int, heads: Int, stride: Int, scale: Float,+ commands: MTLCommandBuffer) throws {+ guard paddedKeys % 64 == 0, paddedKeys >= keys, let e = commands.makeComputeCommandEncoder() else {+ throw MetalContext.Failure.kernel("na_flash_attention \(queries)×\(keys)")+ }+ e.label = "na_flash_attention"+ e.setComputePipelineState(pipeline)+ e.setBuffer(q.0, offset: q.1, index: 0)+ e.setBuffer(kp.0, offset: kp.1, index: 1)+ e.setBuffer(vp.0, offset: vp.1, index: 2)+ e.setBuffer(out.0, offset: out.1, index: 3)+ var p = Params(queries: UInt32(queries), keys: UInt32(keys), causal: 0, stride: UInt32(stride), scale: scale, headBase: 0)+ e.setBytes(&p, length: MemoryLayout<Params>.stride, index: 4)+ var padded = UInt32(paddedKeys)+ e.setBytes(&padded, length: 4, index: 5)+ e.dispatchThreadgroups(MTLSize(width: (queries + NeuralAttention.rowsPerGroup - 1) / NeuralAttention.rowsPerGroup,+ height: heads, depth: 1),+ threadsPerThreadgroup: MTLSize(width: NeuralAttention.threads, height: 1, depth: 1))+ e.endEncoding()+ }+}diff --git a/engine-mac/Sources/KokonEngine/QwenDiT.swift b/engine-mac/Sources/KokonEngine/QwenDiT.swiftindex b5bc194..e494917 100644--- a/engine-mac/Sources/KokonEngine/QwenDiT.swift+++ b/engine-mac/Sources/KokonEngine/QwenDiT.swift@@ -930,6 +930,9 @@ public final class QwenDiT {/// M4 Max 4.57 → 3.88, M5 Pro 5.79 → 5.34 (on the M2 Pro 142 ms against 91 for the attention alone). Off by default/// on every chip; `KOKON_ATTENTION_DIRECT=1` to try it.lazy var packedKernel = QwenDiT.directAttention(device: context.device) ? "flash_attention_direct" : "flash_attention_packed"+ /// The packed attention on the neural accelerators where the GPU has them (`NeuralAttention`), else nil.+ lazy var neuralAttention: NeuralAttention? = !QwenDiT.directAttention(device: context.device)+ && NeuralAttention.selected(device: context.device) ? NeuralAttention.make(device: context.device) : nilstatic func directAttention(device: MTLDevice) -> Bool {ProcessInfo.processInfo.environment["KOKON_ATTENTION_DIRECT"] == "1"}@@ -982,14 +985,24 @@ public final class QwenDiT {threadsPerThreadgroup: MTLSize(width: 32, height: 8, depth: 1))}}+ let scale = 1 / Float(QwenDiT.headDim).squareRoot() * Float(M_LOG2E)+ if packed, let neuralAttention {+ castInput = nil+ try gpu("na_flash_attention", commands) {+ try neuralAttention.encode(q: (arena.buffer, q.offset + queries * row), kp: (arena.buffer, kPacked.offset),+ vp: (arena.buffer, vPacked.offset), out: (arena.buffer, attended.offset + queries * row),+ queries: count, keys: keys, paddedKeys: paddedKeys, heads: heads,+ stride: QwenDiT.dim, scale: scale, commands: $0)+ }+ return+ }try compute(packed ? packedKernel : flashKernel, commands: commands) { e ine.setBuffer(arena.buffer, offset: q.offset + queries * row, index: 0)e.setBuffer(arena.buffer, offset: packed ? kPacked.offset : k.offset, index: 1)e.setBuffer(arena.buffer, offset: packed ? vPacked.offset : v.offset, index: 2)e.setBuffer(arena.buffer, offset: attended.offset + queries * row, index: 3)var params = FlashParams(queries: UInt32(count), keys: UInt32(keys), causal: causal ? 1 : 0,- stride: UInt32(QwenDiT.dim), scale: 1 / Float(QwenDiT.headDim).squareRoot() * Float(M_LOG2E),- headBase: 0)+ stride: UInt32(QwenDiT.dim), scale: scale, headBase: 0)e.setBytes(¶ms, length: MemoryLayout<FlashParams>.stride, index: 4)var padded = UInt32(paddedKeys)if packed { e.setBytes(&padded, length: 4, index: 5) }diff --git a/engine-mac/Tests/KokonEngineTests/AttentionTests.swift b/engine-mac/Tests/KokonEngineTests/AttentionTests.swiftindex f86708e..72931e6 100644--- a/engine-mac/Tests/KokonEngineTests/AttentionTests.swift+++ b/engine-mac/Tests/KokonEngineTests/AttentionTests.swift@@ -9,9 +9,10 @@ final class AttentionTests: XCTestCase {struct PackParams { var keys, paddedKeys, stride, heads: UInt32 }- /// `kernel` on K and V packed by `attention_pack_kv` when `packed`.+ /// `kernel` on K and V packed by `attention_pack_kv` when `packed`; `neural`: the packed attention on Metal 4 tensor+ /// operations instead (`NeuralAttention`).func run(queries: Int, keys: Int, causal: Bool, heads: Int = 2, magnitude: Float = 4,- kernel: String = "flash_attention", packed: Bool = false) throws -> (worst: Double, untouched: Bool) {+ kernel: String = "flash_attention", packed: Bool = false, neural: NeuralAttention? = nil) throws -> (worst: Double, untouched: Bool) {let context = try MetalContext()let stride = heads * 128, padding = 64let rows = max(queries, keys) + padding@@ -42,6 +43,10 @@ final class AttentionTests: XCTestCase {e.dispatchThreads(MTLSize(width: 32, height: paddedKeys, depth: heads), threadsPerThreadgroup: MTLSize(width: 32, height: 8, depth: 1))e.endEncoding()}+ if let neural {+ try neural.encode(q: (qb, 0), kp: (kp, 0), vp: (vp, 0), out: (ob, 0), queries: queries, keys: keys, paddedKeys: paddedKeys,+ heads: heads, stride: stride, scale: 1 / Float(128).squareRoot() * Float(M_LOG2E), commands: commands)+ } else {let e = commands.makeComputeCommandEncoder()!e.setComputePipelineState(try context.pipeline(kernel))for (i, b) in [qb, packed ? kp : kb, packed ? vp : vb, ob].enumerated() { e.setBuffer(b, offset: 0, index: i) }@@ -53,6 +58,7 @@ final class AttentionTests: XCTestCase {e.dispatchThreadgroups(MTLSize(width: (queries + 63) / 64, height: heads, depth: 1),threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))e.endEncoding()+ }try MetalContext.finish(commands, "flash")let o = ob.contents().assumingMemoryBound(to: Float.self)var worst = 0.0@@ -116,4 +122,21 @@ final class AttentionTests: XCTestCase {}}}++ /// The packed attention on Metal 4 tensor operations (`NeuralAttention`, the M5's neural accelerators), forced on+ /// any Metal 4 GPU: within fp16's rounding of exact as `flash_attention_packed`, and no further from exact than it+ /// (the same fp16 operands), on sizes that are not multiples of its tiles and on a peaked softmax.+ func testNeuralAttention() throws {+ let device = try XCTUnwrap(MTLCreateSystemDefaultDevice())+ guard let neural = NeuralAttention.make(device: device) else {+ throw XCTSkip("Metal 4 tensor operations unavailable: \(NeuralAttention.unavailable ?? "?")")+ }+ for (q, k, magnitude): (Int, Int, Float) in [(32, 32, 4), (40, 70, 4), (100, 130, 4), (7, 300, 4), (64, 96, 16), (128, 200, 4)] {+ let r = try run(queries: q, keys: k, causal: false, magnitude: magnitude, packed: true, neural: neural)+ let reference = try run(queries: q, keys: k, causal: false, magnitude: magnitude, kernel: "flash_attention_packed", packed: true)+ if magnitude == 4 { XCTAssertLessThan(r.worst, 5e-3, "\(q)×\(k)") } // as testPacked; the peaked case by the line below+ XCTAssertLessThan(r.worst, 2 * reference.worst + 1e-4, "\(q)×\(k): \(r.worst) against the packed kernel's \(reference.worst)")+ XCTAssertTrue(r.untouched, "\(q)×\(k) wrote past its queries")+ }+ }}diff --git a/engine-mac/Tests/KokonEngineTests/PerChipTests.swift b/engine-mac/Tests/KokonEngineTests/PerChipTests.swiftindex da460de..83e5e1f 100644--- a/engine-mac/Tests/KokonEngineTests/PerChipTests.swift+++ b/engine-mac/Tests/KokonEngineTests/PerChipTests.swift@@ -62,6 +62,18 @@ final class PerChipTests: XCTestCase {}}+ /// Which GPUs take the attention on the neural accelerators: Apple10 with Metal 4 (the M5's), unless the setting+ /// says otherwise; never without Metal 4.+ func testNeuralAttentionSelection() {+ XCTAssertTrue(NeuralAttention.selected(gpuFamily: 10, metal4: true, setting: nil))+ XCTAssertFalse(NeuralAttention.selected(gpuFamily: 10, metal4: true, setting: "0"))+ for family in [7, 8, 9] { // M1–M4+ XCTAssertFalse(NeuralAttention.selected(gpuFamily: family, metal4: true, setting: nil), "Apple\(family)")+ XCTAssertTrue(NeuralAttention.selected(gpuFamily: family, metal4: true, setting: "1"), "Apple\(family)")+ }+ XCTAssertFalse(NeuralAttention.selected(gpuFamily: 10, metal4: false, setting: "1"))+ }+func testChipNames() {XCTAssertEqual(Chip.generation(of: "Apple M2 Pro"), 2)XCTAssertEqual(Chip.generation(of: "Apple M4 Max"), 4)diff --git a/engine-mac/chips.md b/engine-mac/chips.mdindex 3848274..fe74bd6 100644--- a/engine-mac/chips.md+++ b/engine-mac/chips.md@@ -8,7 +8,7 @@ Before working on a chip, look here and in [docs/mac-engine.md](docs/mac-engine.| Chip | Mac (model, cooling) | Memory | macOS | Paths on by default | 1024² text to image | Velocity error (default) | Source ||---|---|---|---|---|---|---|---|| M2 Pro, 19-core GPU | MacBook Pro (Mac14,10), fans | 16 GB | 27 | Metal 4 products (7.57 → 7.48 s per evaluation); direct attention off (142 ms vs 91) | ~64–67 s; 1 MP edit ~106 s, peak 5 GB | ~6–7 · 10⁻⁴ | [README.md](README.md), [docs/mac-engine.md](docs/mac-engine.md) chapters 26, 28, 30 |-| M5 Pro, 20-core GPU | MacBook Pro 16" (Mac17,8), fans | 24 GB | 27.2 | Metal 4 products on 2 SIMD groups per tile; each product released at once (signals arrive late: still on macOS 27.2) | 34.4 s (test set, median of 3, macOS 27.2; 37.2 s with 4 SIMD groups); edit 39.0 s, peak 5.2–5.3 GB | 4.1 · 10⁻⁴ (bench, vs the engine's fp32 path) | [README.md](README.md) "Per-chip paths", [docs/mac-engine.md](docs/mac-engine.md) chapters 13, 26 |+| M5 Pro, 20-core GPU | MacBook Pro 16" (Mac17,8), fans | 24 GB | 27.2 | Metal 4 products on 2 SIMD groups per tile; the target's attention on Metal 4 tensor operations; each product released at once (signals arrive late: still on macOS 27.2) | 28.9 s (test set, median of 3, macOS 27.2; 33.3 s with the engine's attention); edit 31.3 s, peak 5.3 GB | 4.4 · 10⁻⁴ (bench, vs the engine's fp32 path) | [README.md](README.md) "Per-chip paths", [docs/mac-engine.md](docs/mac-engine.md) chapters 13, 26 |## Per-chip notes@@ -31,6 +31,8 @@ Before working on a chip, look here and in [docs/mac-engine.md](docs/mac-engine.28–31). `na_matmul` was behind on long K and many rows with 4 SIMD groups per tile; with 2 (`KOKON_NA_SG2`) a stepat 1024² 4.21 → 3.86 s, level with MPS (3.89), test set 1024² 37.2 → 34.4 s (3 alternated rounds, macOS 27.2).- On macOS 27.2 the late signal is still measured here (`KOKON_FLUSH_READY` selected), unlike the other Macs on 27.+- The attention on the neural accelerators (`na_flash_attention`): 60.0 → 28.1 ms at 1024², a step 3.64 → 2.99 s; the+ balancer then gives the CPU 3 of 32 heads again.- The balancer keeps about 2 600 of 4 096 rows on the GPU, 1 280 on the Neural Engine and 192–256 on the AMX, witheither tile: its remembered shares do not tell the GPU paths apart, so whether a faster GPU should take more rowsis not measured yet.diff --git a/engine-mac/docs/mac-engine.md b/engine-mac/docs/mac-engine.mdindex 9673388..59ad84d 100644--- a/engine-mac/docs/mac-engine.md+++ b/engine-mac/docs/mac-engine.md@@ -1275,6 +1275,7 @@ any Mac.**| Path | Chosen | M2 Pro result | Switch ||---|---|---|---|| **GPU rows on Metal 4 tensor operations** (`na_matmul`, MetalPerformancePrimitives' `matmul2d`): on the M5 the GPU cores' matrix units, before it the shader cores | wherever Metal 4 runs | 6.3 vs MPS 5.6 TFLOP/s, bit for bit the same; an evaluation 7.57 → 7.48 s | `KOKON_NA=0` |+| **The target's attention on Metal 4 tensor operations** (`na_flash_attention`: Q·Kᵀ and P·V with MPP's `matmul2d`, the softmax on its row reductions; same packed K and V and the same fp16 arithmetic as `flash_attention_packed`) | GPUs of the Apple10 family with Metal 4 (the M5's neural accelerators), by capability | not selected (M5 Pro: the attention at 1024² 60.0 → 28.1 ms, the same error; a step 3.64 → 2.99 s, 1024² image 33.3 → 28.9 s) | `KOKON_NA_ATTENTION=0/1` || **2 SIMD groups per `na_matmul` tile** (4 elsewhere): with 4 the MLP's out (K 12 288) and 4 096-row products ran 20–26 % below MPS on the M5 Pro's neural accelerators; with 2 every DiT shape is at their ~32 TFLOP/s fp16 ceiling (MPS's and MLX's too) | M5 Pro only | not selected (M5 Pro: a step at 1024² 4.21 → 3.86 s, 1024² image 37.2 → 34.4 s, the same result to the bit) | `KOKON_NA_SG2=0/1` || **Attention reading K/V from memory** (`flash_attention_direct`) | off on every chip (since 0.1.1) | slower on every Mac measured (M2 Pro 142 ms vs 91; M3 Max, M4 Pro, M4 Max, M5 Pro: 8–18 % per step) | `KOKON_ATTENTION_DIRECT=1` || **int8 products on the Neural Engine** (libnnc's M3/M4 path) | opt-in on every chip | a tile 10.1 → 7.0 ms, but 15–50× the error | `KOKON_ANE_INT8=1` |@@ -1454,6 +1455,7 @@ All are environment variables read once at start. Defaults are what users run.| `KOKON_ANE_ATTENTION` | off | `1`: Neural Engine heads too || `KOKON_ANE_INT8` | off | `1`: int8 weights and rows on the Neural Engine || `KOKON_NA` | on with Metal 4 | `0`: MPS for the GPU's rows; `1`: force Metal 4 tensor ops |+| `KOKON_NA_ATTENTION` | on Apple10 GPUs with Metal 4 | `0`: the engine's packed attention; `1`: force it on any Metal 4 GPU || `KOKON_NA_SG2` | on the M5 Pro | `0`/`1`: 2 SIMD groups per `na_matmul` tile instead of 4 || `KOKON_WEIGHTS` | fp16 | `fp32`: unpack weights to fp32 (checks) || `KOKON_ACTIVATIONS` | fp16 | `fp32`: fp32 inputs to the GPU's and AMX's products (checks) |@@ -1490,7 +1492,7 @@ All are environment variables read once at start. Defaults are what users run.| `QwenTextGeneration.swift`, `Kernels/decode.metal` | [23](#23-prompt-writing-text) || `Noise.swift` | [24](#24-seeds-and-noise) || `EngineServer.swift`, `HTTPServer.swift` | [25](#25-the-server-and-the-app) |-| `NeuralAccelerators.swift`, `Kernels/metal4/na_matmul.metal` | [26](#26-per-chip-paths) |+| `NeuralAccelerators.swift`, `Kernels/metal4/na_matmul.metal`, `Kernels/metal4/na_attention.metal` | [26](#26-per-chip-paths) || `Probe.swift`, `AttentionProbe.swift`, `Bench.swift` | [27](#27-measuring-a-mac-bench-and-the-tools) || `Checks.swift`, `TextChecks.swift`, `tools/reference_*.py`, `Tests/` | [28](#28-how-we-know-it-is-right) || `Sources/kokon-engine-mac/main.swift` | every command in one place |