Skip to content

armgit / kokon

Public Imported from GitHub
0

Edit photos by describing the change, on your own GPU. A free, open-source alternative to Nano Banana and ChatGPT image editing: photos stay on your computer, no daily limit. One installer for Windows, Mac and Linux sets up the best open model (Qwen-Image 2.1). MCP server for AI agents.

Mac engine: the target's attention on the M5's neural accelerators (#50)

Cesare Davide Pacef4233ef8793b
diff --git a/engine-mac/README.md b/engine-mac/README.md
index b34b64f..85d1ff4 100644
--- a/engine-mac/README.md
+++ b/engine-mac/README.md
@@ -63,7 +63,7 @@ the kernels were found.
| `EngineServer.swift`, `HTTPServer.swift` | `serve`: sd-server's `/sdcpp/v1` API (capabilities, img_gen, jobs, cancel, text_gen) and its log lines |
| `QwenTextGeneration.swift`, `Kernels/decode.metal` | Prompt+: the text encoder's LLM writes, K/V cache, products on the Q4_K / Q6_K blocks, sd.cpp's sampler |
| `Chip.swift` | this Mac's chip, GPU family, cores and memory; each unit's estimated speed against the M2 Pro's |
-| `NeuralAccelerators.swift`, `Kernels/metal4/na_matmul.metal` | the GPU's rows of a product on Metal 4 tensor operations (the M5's neural accelerators) |
+| `NeuralAccelerators.swift`, `Kernels/metal4/na_matmul.metal`, `Kernels/metal4/na_attention.metal` | the GPU's rows of a product and the target's attention on Metal 4 tensor operations (the M5's neural accelerators) |
| `Bench.swift` | `bench`: the report testers paste in GitHub |
Tests check the CPU and GPU unpacking against gguf-py's values (`tools/make_quant_fixtures.py` writes the
@@ -432,6 +432,16 @@ correctness test that runs on any Mac (`PerChipTests`). Only the M2 Pro has meas
**On the M5 Pro they are the neural accelerators**, and MPS reaches them too: 31.4 against 32.2 TFLOP/s at
1536 × 4096 × 4096, the same result to the bit, against at most ~10 for its 20 GPU cores without them; MLX 0.32
reaches 28–31 on the DiT's shapes. So `KOKON_NA=0` does not turn them off; it was ahead because of the tile below.
+- **The target's attention on the neural accelerators** (`na_flash_attention`, `KOKON_NA_ATTENTION=0` off, `=1` on
+ any Metal 4 GPU). With the products on the accelerators, the attention was half of a step on the M5 Pro
+ (`KOKON_PROFILE` at 1024²: products 1.92 s of an evaluation, attention 1.91 s), on the shader cores' 8×8 SIMD-group
+ matrices at ~4.6 TFLOP/s. The new kernel does Q·Kᵀ and P·V with MPP's `matmul2d` (each SIMD group 16 query rows,
+ tiles of 64 keys, the softmax on MPP's row reductions), on the same packed fp16 K and V: `probe-attention` 60.0 →
+ 28.1 ms, head 0 6.3 · 10⁻⁴ from exact for both. Chosen by capability (Apple10 GPU and Metal 4), so the M5, M5 Pro
+ and M5 Max; M1–M4 keep the engine's kernel. M5 Pro 20-core, 24 GB, macOS 27.2, 3 alternated rounds, median:
+ `time-dit` third evaluation 3.64 → 2.99 s; test set 1024² 33.3 → 28.9 s, 1344×768 33.7 → 27.7 s, edit 36.2 →
+ 31.3 s, faster in every round; bench velocity error 4.4 · 10⁻⁴ (3.0 · 10⁻⁴ with it off in the same run, 4.1 ·
+ 10⁻⁴ the run before). 2 SIMD groups of 32 rows, 8 of 8 rows or tiles of 32 keys were 1.7–3× slower.
- **2 SIMD groups per `na_matmul` tile on the M5 Pro** (`KOKON_NA_SG2=0` off, `=1` on elsewhere). With 4, the MLP's
out (2048 × 12288 × 4096) took 9.6 ms against MPS's 6.7 and 4096 × 4096 × 4096 6.0 against 4.4 (`check-na`, best
of 5); with 2, 7.0 and 4.4, the same result to the bit. A sweep of tiles 32–128 and 1–8 SIMD groups found no
diff --git a/engine-mac/Sources/KokonEngine/AttentionProbe.swift b/engine-mac/Sources/KokonEngine/AttentionProbe.swift
index 25b5470..74aef11 100644
--- a/engine-mac/Sources/KokonEngine/AttentionProbe.swift
+++ b/engine-mac/Sources/KokonEngine/AttentionProbe.swift
@@ -99,6 +99,23 @@ extension Probe {
print(String(format: "GPU %@%@, 32 heads: %.1f ms, %.2f ms/head, %.2f TFLOP/s; head 0 rel %.1e", kernel, note,
gpu * 1e3, gpu * 1e3 / Double(heads), flops / gpu / 1e12, gpuError))
}
+ // The same packed attention on Metal 4 tensor operations (the M5's neural accelerators), where it compiles.
+ if let na = NeuralAttention.make(device: context.device) {
+ let gpu = try best(3) {
+ let commands = context.queue.makeCommandBuffer()!
+ try pack(commands)
+ try na.encode(q: (qb, offset * stride * 4), kp: (kp, 0), vp: (vp, 0), out: (ob, offset * stride * 4),
+ queries: queries, keys: keys, paddedKeys: paddedKeys, heads: heads, stride: stride,
+ scale: scale * Float(M_LOG2E), commands: commands)
+ return try MetalContext.finish(commands, "na attention")
+ }
+ let flops: Double = 4 * Double(queries) * Double(keys) * Double(d) * Double(heads)
+ print(String(format: "GPU na_flash_attention (with packing)%@, 32 heads: %.1f ms, %.2f ms/head, %.2f TFLOP/s; head 0 rel %.1e",
+ NeuralAttention.selected(device: context.device) ? " (selected)" : "", gpu * 1e3, gpu * 1e3 / Double(heads),
+ flops / gpu / 1e12, error { i, c in f(ob)[(offset + i) * stride + c] }))
+ } else {
+ print("GPU na_flash_attention unavailable: \(NeuralAttention.unavailable ?? "?")")
+ }
if ProcessInfo.processInfo.environment["KOKON_PROBE_GPU_ONLY"] == "1" { return }
// The Neural Engine.
diff --git a/engine-mac/Sources/KokonEngine/Bench.swift b/engine-mac/Sources/KokonEngine/Bench.swift
index d4fd365..cffab33 100644
--- a/engine-mac/Sources/KokonEngine/Bench.swift
+++ b/engine-mac/Sources/KokonEngine/Bench.swift
@@ -31,6 +31,9 @@ public enum Bench {
paths.append(("those products on 2 SIMD groups per tile (M5 Pro)", "KOKON_NA_SG2", "0"))
}
}
+ if NeuralAttention.selected(device: device), NeuralAttention.make(device: device) != nil {
+ paths.append(("the target's attention on Metal 4 tensor operations (the M5's neural accelerators)", "KOKON_NA_ATTENTION", "0"))
+ }
if Coexecutor.flushReleaseSelected(context) {
paths.append(("each product released to the AMX and the Neural Engine at once (this Mac's signals arrive late)", "KOKON_FLUSH_READY", "0"))
}
diff --git a/engine-mac/Sources/KokonEngine/Kernels/metal4/na_attention.metal b/engine-mac/Sources/KokonEngine/Kernels/metal4/na_attention.metal
new file mode 100644
index 0000000..ed9e3b2
--- /dev/null
+++ b/engine-mac/Sources/KokonEngine/Kernels/metal4/na_attention.metal
@@ -0,0 +1,124 @@
+#include <metal_stdlib>
+#include <metal_tensor>
+#include <MetalPerformancePrimitives/MetalPerformancePrimitives.h>
+using namespace metal;
+using namespace mpp::tensor_ops;
+
+// The target's attention with its two products, Q·Kᵀ and P·V, on Metal 4 tensor operations (MetalPerformancePrimitives'
+// matmul2d): on the M5 they run on the GPU's neural accelerators, where `flash_attention_packed`'s 8×8 SIMD-group
+// matrices run on the shader cores. Compiled apart from the other kernels (Metal 4, `NeuralAccelerators.swift`).
+//
+// The same interface as `flash_attention_packed` (`attention.metal`), not causal: q fp32 rows of `stride` floats (head
+// h at columns [128 h, 128 h + 128)), K and V packed by `attention_pack_kv` (fp16, `[head][key][128]`, keys padded with
+// zeros to `paddedKeys`, a multiple of 64), out fp32 rows like q. The same arithmetic too: q, K, V and the
+// probabilities in fp16, scores, softmax and O accumulated in fp32, an online softmax over tiles of 64 keys.
+//
+// Each SIMD group owns 16 query rows (MPP's row reductions work within one SIMD group), 4 per threadgroup: 64 rows,
+// as `flash_attention_packed` (rows past `queries` are computed, never stored). Its Q rows (fp16) and probabilities sit
+// in threadgroup memory as tensor operands; the scores and O stay in cooperative tensors, in registers; each row's
+// running max, sum and rescale in threadgroup memory, read back by the elements of that row.
+//
+// M5 Pro 20-core, macOS 27.2, 1024² (4 096 queries × 4 128 keys, 32 heads): 27.4 ms against 59.7 for
+// `flash_attention_packed` with its packing, 4.8e-4 from an fp64 attention of the same fp16 K and V. 2 SIMD groups of
+// 32 rows 48.0 ms, 8 of 8 81.6, tiles of 32 keys 48.8 (median of 7).
+
+struct FlashParams {
+ uint queries;
+ uint keys;
+ uint causal;
+ uint stride;
+ float scale; // 1/sqrt(head_dim) * log2(e)
+ uint headBase;
+};
+
+constant constexpr int ND = 128;
+
+template <int SG, int MQ, int NK>
+kernel void na_flash_attention(device const float *q [[buffer(0)]], device half *kp [[buffer(1)]], device half *vp [[buffer(2)]],
+ device float *out [[buffer(3)]], constant FlashParams &p [[buffer(4)]],
+ constant uint &paddedKeys [[buffer(5)]],
+ uint2 tg [[threadgroup_position_in_grid]], uint sg [[simdgroup_index_in_threadgroup]],
+ uint lane [[thread_index_in_simdgroup]]) {
+ threadgroup half qsAll[SG * MQ * ND];
+ threadgroup half psAll[SG * MQ * NK];
+ threadgroup float statsAll[SG * 4 * MQ]; // per row: max, sum, alpha, block value
+ threadgroup half *qs = qsAll + sg * MQ * ND;
+ threadgroup half *ps = psAll + sg * MQ * NK;
+ threadgroup float *rowMax = statsAll + sg * 4 * MQ, *rowSum = rowMax + MQ, *rowAlpha = rowSum + MQ, *block = rowAlpha + MQ;
+
+ const uint head = p.headBase + tg.y, q0 = (tg.x * SG + sg) * MQ;
+ for (uint i = lane; i < MQ * ND; i += 32) {
+ qs[i] = half(q[(ulong)(q0 + i / ND) * p.stride + head * ND + i % ND]);
+ }
+ if (lane < MQ) { rowMax[lane] = -INFINITY; rowSum[lane] = 0.0f; }
+ simdgroup_barrier(mem_flags::mem_threadgroup);
+
+ auto tQ = tensor<threadgroup half, extents<int32_t, ND, MQ>, tensor_inline>(qs, extents<int32_t, ND, MQ>());
+ auto tP = tensor<threadgroup half, extents<int32_t, NK, MQ>, tensor_inline>(ps, extents<int32_t, NK, MQ>());
+ device half *kh = kp + (ulong)head * paddedKeys * ND;
+ device half *vh = vp + (ulong)head * paddedKeys * ND;
+ auto tK = tensor<device half, dextents<int32_t, 2>, tensor_inline>(kh, dextents<int32_t, 2>(ND, paddedKeys));
+ auto tV = tensor<device half, dextents<int32_t, 2>, tensor_inline>(vh, dextents<int32_t, 2>(ND, paddedKeys));
+
+ constexpr auto qk = matmul2d_descriptor(MQ, NK, ND, false, true, false, matmul2d_descriptor::mode::multiply);
+ matmul2d<qk, execution_simdgroup> opS;
+ constexpr auto pv = matmul2d_descriptor(MQ, ND, NK, false, false, false, matmul2d_descriptor::mode::multiply_accumulate);
+ matmul2d<pv, execution_simdgroup> opO;
+ using KT = remove_addrspace_t<decltype(tK.slice(0, 0))>;
+ using VT = remove_addrspace_t<decltype(tV.slice(0, 0))>;
+
+ auto S = opS.template get_destination_cooperative_tensor<decltype(tQ), KT, float>();
+ auto rows = opS.template get_row_reduction_destination_cooperative_tensor<decltype(tQ), KT, float>();
+ auto O = opO.template get_destination_cooperative_tensor<decltype(tP), VT, float>();
+ for (uint16_t i = 0; i < O.get_capacity(); ++i) if (O.is_valid_element(i)) O[i] = 0.0f;
+
+ for (uint j0 = 0; j0 < p.keys; j0 += NK) {
+ auto mK = tK.slice(0, int(j0));
+ opS.run(tQ, mK, S);
+ for (uint16_t i = 0; i < S.get_capacity(); ++i) {
+ if (!S.is_valid_element(i)) continue;
+ S[i] = j0 + uint(S.get_multidimensional_index(i)[0]) < p.keys ? S[i] * p.scale : -INFINITY;
+ }
+ reduce_rows(S, rows, reduction_operation::max, reduction_operation_identity<float>::max_identity);
+ for (uint16_t i = 0; i < rows.get_capacity(); ++i) {
+ if (rows.is_valid_element(i)) block[rows.get_multidimensional_index(i)[0]] = rows[i];
+ }
+ simdgroup_barrier(mem_flags::mem_threadgroup);
+ if (lane < MQ) {
+ float mOld = rowMax[lane], mNew = max(mOld, block[lane]);
+ rowAlpha[lane] = (mNew == -INFINITY) ? 1.0f : exp2(mOld - mNew);
+ rowMax[lane] = mNew;
+ }
+ simdgroup_barrier(mem_flags::mem_threadgroup);
+ for (uint16_t i = 0; i < S.get_capacity(); ++i) {
+ if (!S.is_valid_element(i)) continue;
+ auto at = S.get_multidimensional_index(i); // (key, query) within the tile
+ float m = rowMax[at[1]];
+ float e = (m == -INFINITY) ? 0.0f : exp2(S[i] - m);
+ S[i] = e;
+ ps[at[1] * NK + at[0]] = half(e);
+ }
+ reduce_rows(S, rows, reduction_operation::sum, reduction_operation_identity<float>::sum_identity);
+ for (uint16_t i = 0; i < rows.get_capacity(); ++i) {
+ if (rows.is_valid_element(i)) block[rows.get_multidimensional_index(i)[0]] = rows[i];
+ }
+ simdgroup_barrier(mem_flags::mem_threadgroup);
+ if (lane < MQ) rowSum[lane] = rowSum[lane] * rowAlpha[lane] + block[lane];
+ for (uint16_t i = 0; i < O.get_capacity(); ++i) {
+ if (O.is_valid_element(i)) O[i] *= rowAlpha[O.get_multidimensional_index(i)[1]];
+ }
+ auto mV = tV.slice(0, int(j0));
+ opO.run(tP, mV, O);
+ simdgroup_barrier(mem_flags::mem_threadgroup);
+ }
+
+ for (uint16_t i = 0; i < O.get_capacity(); ++i) {
+ if (!O.is_valid_element(i)) continue;
+ auto at = O.get_multidimensional_index(i); // (d, query)
+ uint qi = q0 + uint(at[1]);
+ if (qi < p.queries) out[(ulong)qi * p.stride + head * ND + uint(at[0])] = O[i] / rowSum[at[1]];
+ }
+}
+
+template [[host_name("na_flash_attention")]] kernel void na_flash_attention<4, 16, 64>(device const float *, device half *,
+ device half *, device float *, constant FlashParams &, constant uint &, uint2, uint, uint);
diff --git a/engine-mac/Sources/KokonEngine/NeuralAccelerators.swift b/engine-mac/Sources/KokonEngine/NeuralAccelerators.swift
index 3f5ebfc..b8fcd84 100644
--- a/engine-mac/Sources/KokonEngine/NeuralAccelerators.swift
+++ b/engine-mac/Sources/KokonEngine/NeuralAccelerators.swift
@@ -44,12 +44,7 @@ public final class NeuralAccelerators: @unchecked Sendable {
return nil
}
do {
- guard let url = EngineBundle.kernels?
- .appendingPathComponent("metal4/na_matmul.metal") else { throw MetalContext.Failure.noKernelSource }
- let options = MTLCompileOptions()
- options.languageVersion = .version4_0
- options.mathMode = .safe
- let library = try device.makeLibrary(source: try String(contentsOf: url, encoding: .utf8), options: options)
+ let library = try metal4Library(device: device, file: "na_matmul.metal")
let groups = simdgroups ?? NeuralAccelerators.simdgroups(chip: Chip.current.name,
setting: ProcessInfo.processInfo.environment["KOKON_NA_SG2"])
let name = groups == 2 ? "na_matmul_sg2" : "na_matmul"
@@ -61,6 +56,16 @@ public final class NeuralAccelerators: @unchecked Sendable {
}
}
+ /// Compiles one of `Kernels/metal4/` (Metal 4, `mathMode = .safe` as the other kernels).
+ @available(macOS 26, *)
+ static func metal4Library(device: MTLDevice, file: String) throws -> MTLLibrary {
+ guard let url = EngineBundle.kernels?.appendingPathComponent("metal4/\(file)") else { throw MetalContext.Failure.noKernelSource }
+ let options = MTLCompileOptions()
+ options.languageVersion = .version4_0
+ options.mathMode = .safe
+ return try device.makeLibrary(source: try String(contentsOf: url, encoding: .utf8), options: options)
+ }
+
init(pipeline: MTLComputePipelineState, simdgroups: Int) {
self.pipeline = pipeline
self.simdgroups = simdgroups
@@ -96,3 +101,82 @@ public final class NeuralAccelerators: @unchecked Sendable {
e.endEncoding()
}
}
+
+/// **The target's attention on the neural accelerators** (`Kernels/metal4/na_attention.metal`): Q·Kᵀ and P·V on Metal
+/// 4's tensor operations, where `flash_attention_packed` uses 8×8 SIMD-group matrices on the shader cores. With the
+/// products on the accelerators, the attention became half of a step on the M5 Pro (`KOKON_PROFILE`, 1024²: products
+/// 1.92 s of an evaluation, the attention 1.91 s). The kernel alone at 1024²: 27.4 ms against 59.7 (20-core, macOS
+/// 27.2). Same inputs as `flash_attention_packed` (K and V packed by `attention_pack_kv`), not causal: the text
+/// segments' causal attentions stay on the engine's kernels.
+///
+/// Chosen by capability, not by name: GPUs of the Apple10 family (the M5's, with neural accelerators) with Metal 4.
+/// Older Metal 4 GPUs (M1–M4) would run the same source on their shader cores, where it is not measured: they keep
+/// `flash_attention_packed`. `KOKON_NA_ATTENTION=0` off, `=1` on any Metal 4 GPU (its results are checked there too).
+public final class NeuralAttention: @unchecked Sendable {
+ let pipeline: MTLComputePipelineState
+ nonisolated(unsafe) public private(set) static var unavailable: String?
+ /// Query rows per threadgroup: 4 SIMD groups of 16 (the kernel's), as `flash_attention_packed`'s 64.
+ static let rowsPerGroup = 64, threads = 128
+
+ /// Mirrors `FlashParams` in `na_attention.metal` (and `attention.metal`).
+ struct Params { var queries, keys, causal, stride: UInt32; var scale: Float; var headBase: UInt32 }
+
+ static func selected(gpuFamily: Int, metal4: Bool, setting: String?) -> Bool {
+ switch setting {
+ case "0": return false
+ case "1": return metal4
+ default: return metal4 && gpuFamily >= 10
+ }
+ }
+
+ public static func selected(device: MTLDevice) -> Bool {
+ var metal4 = false
+ if #available(macOS 26, *) { metal4 = device.supportsFamily(.metal4) }
+ return selected(gpuFamily: device.supportsFamily(.apple10) ? 10 : 0, metal4: metal4,
+ setting: ProcessInfo.processInfo.environment["KOKON_NA_ATTENTION"])
+ }
+
+ /// Compiles the kernel; nil without Metal 4 or when the compiler refuses the source (the engine's kernels then).
+ public static func make(device: MTLDevice) -> NeuralAttention? {
+ guard #available(macOS 26, *), device.supportsFamily(.metal4) else {
+ unavailable = "no Metal 4 on this GPU or macOS"
+ return nil
+ }
+ do {
+ let library = try NeuralAccelerators.metal4Library(device: device, file: "na_attention.metal")
+ guard let function = library.makeFunction(name: "na_flash_attention") else {
+ throw MetalContext.Failure.kernel("na_flash_attention not found")
+ }
+ return NeuralAttention(pipeline: try device.makeComputePipelineState(function: function))
+ } catch {
+ unavailable = "the kernel did not compile: \(error.localizedDescription)"
+ return nil
+ }
+ }
+
+ init(pipeline: MTLComputePipelineState) { self.pipeline = pipeline }
+
+ /// Encodes heads `[0, heads)` of `queries` query rows against `keys` keys; q and out rows of `stride` floats, K and
+ /// V packed with `paddedKeys` keys per head. `scale`: 1/√d · log2(e).
+ func encode(q: (MTLBuffer, Int), kp: (MTLBuffer, Int), vp: (MTLBuffer, Int), out: (MTLBuffer, Int),
+ queries: Int, keys: Int, paddedKeys: Int, heads: Int, stride: Int, scale: Float,
+ commands: MTLCommandBuffer) throws {
+ guard paddedKeys % 64 == 0, paddedKeys >= keys, let e = commands.makeComputeCommandEncoder() else {
+ throw MetalContext.Failure.kernel("na_flash_attention \(queries)×\(keys)")
+ }
+ e.label = "na_flash_attention"
+ e.setComputePipelineState(pipeline)
+ e.setBuffer(q.0, offset: q.1, index: 0)
+ e.setBuffer(kp.0, offset: kp.1, index: 1)
+ e.setBuffer(vp.0, offset: vp.1, index: 2)
+ e.setBuffer(out.0, offset: out.1, index: 3)
+ var p = Params(queries: UInt32(queries), keys: UInt32(keys), causal: 0, stride: UInt32(stride), scale: scale, headBase: 0)
+ e.setBytes(&p, length: MemoryLayout<Params>.stride, index: 4)
+ var padded = UInt32(paddedKeys)
+ e.setBytes(&padded, length: 4, index: 5)
+ e.dispatchThreadgroups(MTLSize(width: (queries + NeuralAttention.rowsPerGroup - 1) / NeuralAttention.rowsPerGroup,
+ height: heads, depth: 1),
+ threadsPerThreadgroup: MTLSize(width: NeuralAttention.threads, height: 1, depth: 1))
+ e.endEncoding()
+ }
+}
diff --git a/engine-mac/Sources/KokonEngine/QwenDiT.swift b/engine-mac/Sources/KokonEngine/QwenDiT.swift
index b5bc194..e494917 100644
--- a/engine-mac/Sources/KokonEngine/QwenDiT.swift
+++ b/engine-mac/Sources/KokonEngine/QwenDiT.swift
@@ -930,6 +930,9 @@ public final class QwenDiT {
/// M4 Max 4.57 → 3.88, M5 Pro 5.79 → 5.34 (on the M2 Pro 142 ms against 91 for the attention alone). Off by default
/// on every chip; `KOKON_ATTENTION_DIRECT=1` to try it.
lazy var packedKernel = QwenDiT.directAttention(device: context.device) ? "flash_attention_direct" : "flash_attention_packed"
+ /// The packed attention on the neural accelerators where the GPU has them (`NeuralAttention`), else nil.
+ lazy var neuralAttention: NeuralAttention? = !QwenDiT.directAttention(device: context.device)
+ && NeuralAttention.selected(device: context.device) ? NeuralAttention.make(device: context.device) : nil
static func directAttention(device: MTLDevice) -> Bool {
ProcessInfo.processInfo.environment["KOKON_ATTENTION_DIRECT"] == "1"
}
@@ -982,14 +985,24 @@ public final class QwenDiT {
threadsPerThreadgroup: MTLSize(width: 32, height: 8, depth: 1))
}
}
+ let scale = 1 / Float(QwenDiT.headDim).squareRoot() * Float(M_LOG2E)
+ if packed, let neuralAttention {
+ castInput = nil
+ try gpu("na_flash_attention", commands) {
+ try neuralAttention.encode(q: (arena.buffer, q.offset + queries * row), kp: (arena.buffer, kPacked.offset),
+ vp: (arena.buffer, vPacked.offset), out: (arena.buffer, attended.offset + queries * row),
+ queries: count, keys: keys, paddedKeys: paddedKeys, heads: heads,
+ stride: QwenDiT.dim, scale: scale, commands: $0)
+ }
+ return
+ }
try compute(packed ? packedKernel : flashKernel, commands: commands) { e in
e.setBuffer(arena.buffer, offset: q.offset + queries * row, index: 0)
e.setBuffer(arena.buffer, offset: packed ? kPacked.offset : k.offset, index: 1)
e.setBuffer(arena.buffer, offset: packed ? vPacked.offset : v.offset, index: 2)
e.setBuffer(arena.buffer, offset: attended.offset + queries * row, index: 3)
var params = FlashParams(queries: UInt32(count), keys: UInt32(keys), causal: causal ? 1 : 0,
- stride: UInt32(QwenDiT.dim), scale: 1 / Float(QwenDiT.headDim).squareRoot() * Float(M_LOG2E),
- headBase: 0)
+ stride: UInt32(QwenDiT.dim), scale: scale, headBase: 0)
e.setBytes(&params, length: MemoryLayout<FlashParams>.stride, index: 4)
var padded = UInt32(paddedKeys)
if packed { e.setBytes(&padded, length: 4, index: 5) }
diff --git a/engine-mac/Tests/KokonEngineTests/AttentionTests.swift b/engine-mac/Tests/KokonEngineTests/AttentionTests.swift
index f86708e..72931e6 100644
--- a/engine-mac/Tests/KokonEngineTests/AttentionTests.swift
+++ b/engine-mac/Tests/KokonEngineTests/AttentionTests.swift
@@ -9,9 +9,10 @@ final class AttentionTests: XCTestCase {
struct PackParams { var keys, paddedKeys, stride, heads: UInt32 }
- /// `kernel` on K and V packed by `attention_pack_kv` when `packed`.
+ /// `kernel` on K and V packed by `attention_pack_kv` when `packed`; `neural`: the packed attention on Metal 4 tensor
+ /// operations instead (`NeuralAttention`).
func run(queries: Int, keys: Int, causal: Bool, heads: Int = 2, magnitude: Float = 4,
- kernel: String = "flash_attention", packed: Bool = false) throws -> (worst: Double, untouched: Bool) {
+ kernel: String = "flash_attention", packed: Bool = false, neural: NeuralAttention? = nil) throws -> (worst: Double, untouched: Bool) {
let context = try MetalContext()
let stride = heads * 128, padding = 64
let rows = max(queries, keys) + padding
@@ -42,6 +43,10 @@ final class AttentionTests: XCTestCase {
e.dispatchThreads(MTLSize(width: 32, height: paddedKeys, depth: heads), threadsPerThreadgroup: MTLSize(width: 32, height: 8, depth: 1))
e.endEncoding()
}
+ if let neural {
+ try neural.encode(q: (qb, 0), kp: (kp, 0), vp: (vp, 0), out: (ob, 0), queries: queries, keys: keys, paddedKeys: paddedKeys,
+ heads: heads, stride: stride, scale: 1 / Float(128).squareRoot() * Float(M_LOG2E), commands: commands)
+ } else {
let e = commands.makeComputeCommandEncoder()!
e.setComputePipelineState(try context.pipeline(kernel))
for (i, b) in [qb, packed ? kp : kb, packed ? vp : vb, ob].enumerated() { e.setBuffer(b, offset: 0, index: i) }
@@ -53,6 +58,7 @@ final class AttentionTests: XCTestCase {
e.dispatchThreadgroups(MTLSize(width: (queries + 63) / 64, height: heads, depth: 1),
threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
e.endEncoding()
+ }
try MetalContext.finish(commands, "flash")
let o = ob.contents().assumingMemoryBound(to: Float.self)
var worst = 0.0
@@ -116,4 +122,21 @@ final class AttentionTests: XCTestCase {
}
}
}
+
+ /// The packed attention on Metal 4 tensor operations (`NeuralAttention`, the M5's neural accelerators), forced on
+ /// any Metal 4 GPU: within fp16's rounding of exact as `flash_attention_packed`, and no further from exact than it
+ /// (the same fp16 operands), on sizes that are not multiples of its tiles and on a peaked softmax.
+ func testNeuralAttention() throws {
+ let device = try XCTUnwrap(MTLCreateSystemDefaultDevice())
+ guard let neural = NeuralAttention.make(device: device) else {
+ throw XCTSkip("Metal 4 tensor operations unavailable: \(NeuralAttention.unavailable ?? "?")")
+ }
+ for (q, k, magnitude): (Int, Int, Float) in [(32, 32, 4), (40, 70, 4), (100, 130, 4), (7, 300, 4), (64, 96, 16), (128, 200, 4)] {
+ let r = try run(queries: q, keys: k, causal: false, magnitude: magnitude, packed: true, neural: neural)
+ let reference = try run(queries: q, keys: k, causal: false, magnitude: magnitude, kernel: "flash_attention_packed", packed: true)
+ if magnitude == 4 { XCTAssertLessThan(r.worst, 5e-3, "\(q)×\(k)") } // as testPacked; the peaked case by the line below
+ XCTAssertLessThan(r.worst, 2 * reference.worst + 1e-4, "\(q)×\(k): \(r.worst) against the packed kernel's \(reference.worst)")
+ XCTAssertTrue(r.untouched, "\(q)×\(k) wrote past its queries")
+ }
+ }
}
diff --git a/engine-mac/Tests/KokonEngineTests/PerChipTests.swift b/engine-mac/Tests/KokonEngineTests/PerChipTests.swift
index da460de..83e5e1f 100644
--- a/engine-mac/Tests/KokonEngineTests/PerChipTests.swift
+++ b/engine-mac/Tests/KokonEngineTests/PerChipTests.swift
@@ -62,6 +62,18 @@ final class PerChipTests: XCTestCase {
}
}
+ /// Which GPUs take the attention on the neural accelerators: Apple10 with Metal 4 (the M5's), unless the setting
+ /// says otherwise; never without Metal 4.
+ func testNeuralAttentionSelection() {
+ XCTAssertTrue(NeuralAttention.selected(gpuFamily: 10, metal4: true, setting: nil))
+ XCTAssertFalse(NeuralAttention.selected(gpuFamily: 10, metal4: true, setting: "0"))
+ for family in [7, 8, 9] { // M1–M4
+ XCTAssertFalse(NeuralAttention.selected(gpuFamily: family, metal4: true, setting: nil), "Apple\(family)")
+ XCTAssertTrue(NeuralAttention.selected(gpuFamily: family, metal4: true, setting: "1"), "Apple\(family)")
+ }
+ XCTAssertFalse(NeuralAttention.selected(gpuFamily: 10, metal4: false, setting: "1"))
+ }
+
func testChipNames() {
XCTAssertEqual(Chip.generation(of: "Apple M2 Pro"), 2)
XCTAssertEqual(Chip.generation(of: "Apple M4 Max"), 4)
diff --git a/engine-mac/chips.md b/engine-mac/chips.md
index 3848274..fe74bd6 100644
--- a/engine-mac/chips.md
+++ b/engine-mac/chips.md
@@ -8,7 +8,7 @@ Before working on a chip, look here and in [docs/mac-engine.md](docs/mac-engine.
| Chip | Mac (model, cooling) | Memory | macOS | Paths on by default | 1024² text to image | Velocity error (default) | Source |
|---|---|---|---|---|---|---|---|
| M2 Pro, 19-core GPU | MacBook Pro (Mac14,10), fans | 16 GB | 27 | Metal 4 products (7.57 → 7.48 s per evaluation); direct attention off (142 ms vs 91) | ~64–67 s; 1 MP edit ~106 s, peak 5 GB | ~6–7 · 10⁻⁴ | [README.md](README.md), [docs/mac-engine.md](docs/mac-engine.md) chapters 26, 28, 30 |
-| M5 Pro, 20-core GPU | MacBook Pro 16" (Mac17,8), fans | 24 GB | 27.2 | Metal 4 products on 2 SIMD groups per tile; each product released at once (signals arrive late: still on macOS 27.2) | 34.4 s (test set, median of 3, macOS 27.2; 37.2 s with 4 SIMD groups); edit 39.0 s, peak 5.2–5.3 GB | 4.1 · 10⁻⁴ (bench, vs the engine's fp32 path) | [README.md](README.md) "Per-chip paths", [docs/mac-engine.md](docs/mac-engine.md) chapters 13, 26 |
+| M5 Pro, 20-core GPU | MacBook Pro 16" (Mac17,8), fans | 24 GB | 27.2 | Metal 4 products on 2 SIMD groups per tile; the target's attention on Metal 4 tensor operations; each product released at once (signals arrive late: still on macOS 27.2) | 28.9 s (test set, median of 3, macOS 27.2; 33.3 s with the engine's attention); edit 31.3 s, peak 5.3 GB | 4.4 · 10⁻⁴ (bench, vs the engine's fp32 path) | [README.md](README.md) "Per-chip paths", [docs/mac-engine.md](docs/mac-engine.md) chapters 13, 26 |
## Per-chip notes
@@ -31,6 +31,8 @@ Before working on a chip, look here and in [docs/mac-engine.md](docs/mac-engine.
28–31). `na_matmul` was behind on long K and many rows with 4 SIMD groups per tile; with 2 (`KOKON_NA_SG2`) a step
at 1024² 4.21 → 3.86 s, level with MPS (3.89), test set 1024² 37.2 → 34.4 s (3 alternated rounds, macOS 27.2).
- On macOS 27.2 the late signal is still measured here (`KOKON_FLUSH_READY` selected), unlike the other Macs on 27.
+- The attention on the neural accelerators (`na_flash_attention`): 60.0 → 28.1 ms at 1024², a step 3.64 → 2.99 s; the
+ balancer then gives the CPU 3 of 32 heads again.
- The balancer keeps about 2 600 of 4 096 rows on the GPU, 1 280 on the Neural Engine and 192–256 on the AMX, with
either tile: its remembered shares do not tell the GPU paths apart, so whether a faster GPU should take more rows
is not measured yet.
diff --git a/engine-mac/docs/mac-engine.md b/engine-mac/docs/mac-engine.md
index 9673388..59ad84d 100644
--- a/engine-mac/docs/mac-engine.md
+++ b/engine-mac/docs/mac-engine.md
@@ -1275,6 +1275,7 @@ any Mac.**
| Path | Chosen | M2 Pro result | Switch |
|---|---|---|---|
| **GPU rows on Metal 4 tensor operations** (`na_matmul`, MetalPerformancePrimitives' `matmul2d`): on the M5 the GPU cores' matrix units, before it the shader cores | wherever Metal 4 runs | 6.3 vs MPS 5.6 TFLOP/s, bit for bit the same; an evaluation 7.57 → 7.48 s | `KOKON_NA=0` |
+| **The target's attention on Metal 4 tensor operations** (`na_flash_attention`: Q·Kᵀ and P·V with MPP's `matmul2d`, the softmax on its row reductions; same packed K and V and the same fp16 arithmetic as `flash_attention_packed`) | GPUs of the Apple10 family with Metal 4 (the M5's neural accelerators), by capability | not selected (M5 Pro: the attention at 1024² 60.0 → 28.1 ms, the same error; a step 3.64 → 2.99 s, 1024² image 33.3 → 28.9 s) | `KOKON_NA_ATTENTION=0/1` |
| **2 SIMD groups per `na_matmul` tile** (4 elsewhere): with 4 the MLP's out (K 12 288) and 4 096-row products ran 20–26 % below MPS on the M5 Pro's neural accelerators; with 2 every DiT shape is at their ~32 TFLOP/s fp16 ceiling (MPS's and MLX's too) | M5 Pro only | not selected (M5 Pro: a step at 1024² 4.21 → 3.86 s, 1024² image 37.2 → 34.4 s, the same result to the bit) | `KOKON_NA_SG2=0/1` |
| **Attention reading K/V from memory** (`flash_attention_direct`) | off on every chip (since 0.1.1) | slower on every Mac measured (M2 Pro 142 ms vs 91; M3 Max, M4 Pro, M4 Max, M5 Pro: 8–18 % per step) | `KOKON_ATTENTION_DIRECT=1` |
| **int8 products on the Neural Engine** (libnnc's M3/M4 path) | opt-in on every chip | a tile 10.1 → 7.0 ms, but 15–50× the error | `KOKON_ANE_INT8=1` |
@@ -1454,6 +1455,7 @@ All are environment variables read once at start. Defaults are what users run.
| `KOKON_ANE_ATTENTION` | off | `1`: Neural Engine heads too |
| `KOKON_ANE_INT8` | off | `1`: int8 weights and rows on the Neural Engine |
| `KOKON_NA` | on with Metal 4 | `0`: MPS for the GPU's rows; `1`: force Metal 4 tensor ops |
+| `KOKON_NA_ATTENTION` | on Apple10 GPUs with Metal 4 | `0`: the engine's packed attention; `1`: force it on any Metal 4 GPU |
| `KOKON_NA_SG2` | on the M5 Pro | `0`/`1`: 2 SIMD groups per `na_matmul` tile instead of 4 |
| `KOKON_WEIGHTS` | fp16 | `fp32`: unpack weights to fp32 (checks) |
| `KOKON_ACTIVATIONS` | fp16 | `fp32`: fp32 inputs to the GPU's and AMX's products (checks) |
@@ -1490,7 +1492,7 @@ All are environment variables read once at start. Defaults are what users run.
| `QwenTextGeneration.swift`, `Kernels/decode.metal` | [23](#23-prompt-writing-text) |
| `Noise.swift` | [24](#24-seeds-and-noise) |
| `EngineServer.swift`, `HTTPServer.swift` | [25](#25-the-server-and-the-app) |
-| `NeuralAccelerators.swift`, `Kernels/metal4/na_matmul.metal` | [26](#26-per-chip-paths) |
+| `NeuralAccelerators.swift`, `Kernels/metal4/na_matmul.metal`, `Kernels/metal4/na_attention.metal` | [26](#26-per-chip-paths) |
| `Probe.swift`, `AttentionProbe.swift`, `Bench.swift` | [27](#27-measuring-a-mac-bench-and-the-tools) |
| `Checks.swift`, `TextChecks.swift`, `tools/reference_*.py`, `Tests/` | [28](#28-how-we-know-it-is-right) |
| `Sources/kokon-engine-mac/main.swift` | every command in one place |