Edit photos by describing the change, on your own GPU. A free, open-source alternative to Nano Banana and ChatGPT image editing: photos stay on your computer, no daily limit. One installer for Windows, Mac and Linux sets up the best open model (Qwen-Image 2.1). MCP server for AI agents.
TypeScript ai-photo-editorcomfyui-alternativegenerative-fillimage-editingimage-generationinpaintinglinuxlocal-aimacosmcpmcp-servernano-banana-alternativeofflinephoto-editorprivacyqwen-imageqwen-image-2-1stable-diffusion-cppvulkanwindows
Mac engine: attention cut by heads, balancing loop, faster flash kernel
Gabriele Lozupone237da48c7693
diff --git a/engine-mac/README.md b/engine-mac/README.mdindex b496ca6..74433ee 100644--- a/engine-mac/README.md+++ b/engine-mac/README.md@@ -10,6 +10,7 @@ cd engine-macswift build -c releaseswift test # no model needed.build/release/kokon-engine-mac probe # this Mac's matrix throughput, ~1 min, no model+.build/release/kokon-engine-mac probe-attention # the target's attention at 1024² on the GPU, Neural Engine, CPU.build/release/kokon-engine-mac inspect <file.gguf>.build/release/kokon-engine-mac check-dequant <file.gguf> [tensor...].build/release/kokon-engine-mac generate <encoder.gguf> <dit.gguf> <vae.safetensors> "a red fox in the snow" \@@ -27,8 +28,11 @@ swift test # no model needed| `Probe.swift` | GPU (MPS), AMX (Accelerate) and Neural Engine (Core ML) on Qwen's product shapes, alone and together || `QwenSequence.swift` | the joint text/image sequence, block-causal segments and RoPE positions || `QwenDiT.swift`, `Kernels/dit.metal`, `Attention.swift`, `Arena.swift` | the DiT: modulation on the CPU, everything per token on the GPU |-| `Coexecution.swift`, `Kernels/ane.metal` | the AMX and the Neural Engine beside the GPU on the DiT's products |+| `Coexecution.swift`, `Kernels/ane.metal` | the AMX and the Neural Engine beside the GPU on the DiT's products; the timeline of each split operation |+| `Balancer.swift` | the balancing loop: each evaluation's timeline sets the next one's rows and heads per unit || `Kernels/attention.metal` | fused block-causal attention |+| `AttentionSplit.swift`, `CPUAttention.swift` | the target's attention cut by heads: GPU, CPU (AMX + cores), optionally the Neural Engine |+| `AttentionProbe.swift` | `probe-attention`: the target's attention on each unit alone || `QwenVAE.swift`, `Safetensors.swift` | the VAE decoder as one MPSGraph, from Kokon's VAE file (Wan 2.2 names) || `QwenTokenizer.swift` | Qwen's byte-level BPE tokenizer, read from the encoder GGUF's metadata || `QwenTextEncoder.swift`, `Kernels/text.metal` | the text encoder: Qwen3-VL-8B's text decoder on the GPU, diffusers' prompt template |@@ -76,10 +80,21 @@ processor) and saves its noise; `check-generate` starts the engine from that noi```bash~/kokon-bench/venv-oracle/bin/python -I tools/reference_generate.py <dit.gguf> <vae.safetensors> ~/kokon-bench/ref/text/fox <repo dir> ~/kokon-bench/ref/generate-256 # ~5 min-.build/release/kokon-engine-mac check-generate <encoder.gguf> <dit.gguf> <vae.safetensors> ~/kokon-bench/ref/generate-256+.build/release/kokon-engine-mac check-generate <encoder.gguf> <dit.gguf> <vae.safetensors> ~/kokon-bench/ref/generate-256 [~/kokon-bench/ref/text/fox]+.build/release/kokon-engine-mac check-steps <dit.gguf> ~/kokon-bench/ref/generate-256 ~/kokon-bench/ref/text/fox```-256², 2026-10-25: each step's latent within 6.7e-6 relative, the image within 1.0e-5, 97 of 262 144 8-bit values+`check-generate` with the text reference starts from its conditioning instead of the engine's encoder;+`check-steps` runs each step alone on the reference's own latent, so one evaluation's error is not mixed with what+the trajectory makes of the steps before.++Unsloth's DiT + the texture-fix VAE, 256², fp64 reference (`--dtype float64`), 2026-10-25: each step alone within+1.4–2.9e-6 of fp64 except step 4 (σ 0.93) at 6.6e-5, an ill-conditioned evaluation (torch's fp32: 2.7e-5 there).+Along the trajectory the final latent ends within 5.8e-5 (torch fp32 1.2e-5), the image within 7.6e-5, 782 of+262 144 8-bit values off by 1; with the engine's own text encoder (5e-6 off) the same order. These shift by a few+times with any change of rounding order (the old flash kernel: 1.9e-5), as torch's fp32 shifts from fp64.++Old uniform Q4_K DiT + original VAE, 256², fp32 reference: each step's latent within 6.7e-6 relative, the image within 1.0e-5, 97 of 262 144 8-bit valuesdiffer by 1. **First 1024² image (M2 Pro, `generate`): 100.7 s in all**, text encoder 1.7 s, 8 steps of 11.4–12.9 s,VAE 5.4 s with its graph build; peak memory 4.1 GB (Kokon's sd.cpp: ~4.5 min).@@ -120,12 +135,37 @@ Engine 2048 at 1024²): **11.3 s per evaluation at 1024²**, 8.8 s at 896². Theoperands straight into its IOSurfaces and unpacks its output, so the CPU only runs the AMX. Accuracy at 896²against diffusers fp32: the velocity within 3.6e-4 relative (GPU alone 4.4e-6); running the model in bf16, asmost tools do, deviates by roughly 1e-2.-**Attention** is a fused (flash) kernel of our own (`Kernels/attention.metal`, fp32, online softmax, block-causal-mask): 4.5 s per 1024² evaluation against MPSGraph's 4.1 s, but without MPSGraph's score matrix (2.2 GB at 1 MP,-~34 GB at 4 MP). `KOKON_ATTENTION=graph` keeps MPSGraph for comparison. The other units are idle during it: next.--**VAE decoder** (`check-vae`, against `tools/reference_vae.py`, diffusers fp32): within 1.1e-5; 1024² in 3.1 s-(plus ~6 s to build its graph the first time).+**Attention** is a fused (flash) kernel of our own (`Kernels/attention.metal`, fp32, online softmax in registers,+block-causal mask, 64 keys a tile): 120 ms for the target's 32 heads at 1024² alone (2.3 TFLOP/s; the first version+145 ms, MPSGraph ~128 ms) and without MPSGraph's score matrix (2.2 GB at 1 MP, ~34 GB at 4 MP).+`KOKON_ATTENTION=graph` keeps MPSGraph for comparison.++**The target's attention is cut by heads** (`AttentionSplit.swift`) as the products are by rows: GPU heads in the+flash kernel, CPU heads in `CPUAttention` (AMX for the two products, vvexpf on 8 threads for the softmax, fp32,+written straight into the GPU's output). `probe-attention`, ms per head alone: GPU 3.8, CPU 6.6, Neural Engine 5.1+in programs of 4 heads × 1 024 queries (head 0 within 2.2e-3 of exact: fp16). Together at 1024², per layer: GPU+alone 145 ms; GPU 19 / CPU 13 91 ms at no cost in accuracy; with Neural Engine heads ~1.5 % more, because the GPU's+heads slow from 4.5 to 6.3–7.0 ms beside it (memory and cache), for 2.5× the velocity error (9.3e-4 against 3.6e-4+at 896²): Neural Engine heads only with `KOKON_ANE_ATTENTION=1`.++**The balancing loop** (`Balancer.swift`): each evaluation's timeline (when the GPU released each split operation,+when each unit finished its share) gives each unit's rate per kind of operation beside the others, and the next+evaluation's shares follow: the AMX's rows per product shape (steps of 64, remembering measured counts: its rate+is not linear in rows), the Neural Engine's rows (steps of 256, a program set per count), the heads. On the M2 Pro at+1024² it moves from 1856 / 256 / 2048 rows and 19 / 13 heads to about 1728 / 320–448 / 2048 and 21 / 11 in two+evaluations; every product's units then finish within ~0.3 ms of each other (before: the GPU 3–15 ms after the+AMX). `KOKON_BALANCE=0` keeps the starting shares; `KOKON_HEADS=g,a,c` fixes the heads.++**One evaluation at 1024², all three units, unsloth's DiT: 8.8 s** (start of this pass 11.6 s): products ~5.2 s+(≈11 TFLOP/s across the units, the probe's ceiling), attention 32 × 78 ms, the GPU alone between them ~1 s. The+Neural Engine's per-column weight scales are kept per weight (computed once), and the modulation of all of the+pipeline's timesteps is computed once (it was 0.2 s of CPU per evaluation).++**VAE decoder** (`check-vae`, against `tools/reference_vae.py`, diffusers fp32): within 1.1e-5; 1024² in 2.75 s. Its+graph builds and compiles in 0.3 s, but its first run at a size costs ~3.7 s more, GPU time (~1 s of CPU): it cannot+hide under the DiT, so the pipeline keeps its VAE between images and a long-lived engine pays it once per size.+fp16 convolutions (norms and attention kept fp32) were wrong (2.7e-1, an overflow) and 5.7× slower (MPSGraph tries to+move them to the Neural Engine): the VAE stays fp32.Credits: the Neural Engine program technique and the GPU + AMX + Neural Engine split come from[Siliconed](https://github.com/gwenn-ha-dev/Siliconed) (Apache-2.0), written again here.diff --git a/engine-mac/Sources/KokonEngine/AttentionProbe.swift b/engine-mac/Sources/KokonEngine/AttentionProbe.swiftnew file mode 100644index 0000000..2a58623--- /dev/null+++ b/engine-mac/Sources/KokonEngine/AttentionProbe.swift@@ -0,0 +1,116 @@+import Accelerate+import Foundation+import Metal++/// **Where the target's attention can run** at 1024² (4 096 queries on 4 128 keys, 32 heads of 128): the GPU's flash+/// kernel, the Neural Engine (one Core ML program per shape) and the CPU (`CPUAttention`), each timed alone, ms per+/// head, and the Neural Engine and the CPU compared with an exact attention of head 0. q and k uniform in ±3 so the+/// scores spread over ±12 and the softmax is peaked as a model's is.+extension Probe {+ public static func attention() throws {+ let context = try MetalContext()+ let (heads, queries, keys, d) = (32, 4096, 4128, 128)+ let stride = heads * d+ let scale = 1 / Float(d).squareRoot()+ var seed: UInt64 = 7+ func fill(_ p: UnsafeMutablePointer<Float>, _ count: Int, _ range: Float) {+ for i in 0..<count {+ seed = seed &* 6364136223846793005 &+ 1442695040888963407+ p[i] = (Float(seed >> 40) / Float(1 << 24) * 2 - 1) * range+ }+ }+ // q holds the keys' rows too (the target's queries are the last 4 096 rows); 64 rows of slack after each.+ let rows = keys + 64+ let qb = try context.scratch(rows * stride * 4, label: "q"), kb = try context.scratch(rows * stride * 4, label: "k")+ let vb = try context.scratch(rows * stride * 4, label: "v"), ob = try context.scratch(rows * stride * 4, label: "out")+ func f(_ b: MTLBuffer) -> UnsafeMutablePointer<Float> { b.contents().assumingMemoryBound(to: Float.self) }+ fill(f(qb), keys * stride, 3); fill(f(kb), keys * stride, 3); fill(f(vb), keys * stride, 1)+ let offset = keys - queries++ // Exact head 0 (fp64 sums) for the accuracy columns.+ let checkRows = 64+ var exact = [Double](repeating: 0, count: checkRows * d)+ for i in 0..<checkRows {+ let qi = f(qb) + (offset + i) * stride+ var s = [Double](repeating: 0, count: keys), peak = -Double.infinity+ for j in 0..<keys {+ var dot = 0.0+ for c in 0..<d { dot += Double(qi[c]) * Double(f(kb)[j * stride + c]) }+ s[j] = dot * Double(scale); peak = max(peak, s[j])+ }+ var sum = 0.0+ for j in 0..<keys { s[j] = exp(s[j] - peak); sum += s[j] }+ for j in 0..<keys { for c in 0..<d { exact[i * d + c] += s[j] * Double(f(vb)[j * stride + c]) / sum } }+ }+ func error(_ got: (Int, Int) -> Float) -> Double {+ var diff = 0.0, norm = 0.0+ for i in 0..<checkRows { for c in 0..<d { let e = Double(got(i, c)) - exact[i * d + c]; diff += e * e; norm += exact[i * d + c] * exact[i * d + c] } }+ return (diff / norm).squareRoot()+ }+ print("target attention at 1024²: \(queries) queries × \(keys) keys, \(heads) heads of \(d); ms per head, best of 3")++ // The GPU's flash kernel, all heads.+ struct FlashParams { var queries, keys, causal, stride: UInt32; var scale: Float; var headBase: UInt32 }+ let gpu = try best(3) {+ let commands = context.queue.makeCommandBuffer()!+ let e = commands.makeComputeCommandEncoder()!+ e.setComputePipelineState(try context.pipeline("flash_attention"))+ e.setBuffer(qb, offset: offset * stride * 4, index: 0); e.setBuffer(kb, offset: 0, index: 1)+ e.setBuffer(vb, offset: 0, index: 2); e.setBuffer(ob, offset: offset * stride * 4, index: 3)+ var p = FlashParams(queries: UInt32(queries), keys: UInt32(keys), causal: 0, stride: UInt32(stride),+ scale: scale * Float(M_LOG2E), headBase: 0)+ e.setBytes(&p, length: MemoryLayout<FlashParams>.stride, index: 4)+ e.dispatchThreadgroups(MTLSize(width: (queries + 63) / 64, height: heads, depth: 1),+ threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))+ e.endEncoding()+ return try MetalContext.finish(commands, "flash")+ }+ let gpuError = error { i, c in f(ob)[(offset + i) * stride + c] }+ print(String(format: "GPU flash kernel, 32 heads: %.1f ms, %.2f ms/head, %.2f TFLOP/s; head 0 rel %.1e",+ gpu * 1e3, gpu * 1e3 / Double(heads), 4 * Double(queries) * Double(keys) * Double(d) * Double(heads) / gpu / 1e12, gpuError))+ if ProcessInfo.processInfo.environment["KOKON_PROBE_GPU_ONLY"] == "1" { return }++ // The Neural Engine.+ for (h, l) in [(1, 4096), (4, 4096), (8, 4096), (1, 1024), (4, 1024), (8, 1024), (16, 512)] {+ let name = "Neural Engine \(h) heads × \(l) queries"+ do {+ let a = try NeuralEngineAttention(heads: h, queries: l, keys: keys)+ func write(_ surface: CVPixelBuffer, _ source: UnsafeMutablePointer<Float>, _ first: Int, _ count: Int, _ factor: Float) {+ CVPixelBufferLockBaseAddress(surface, [])+ let dst = CVPixelBufferGetBaseAddress(surface)!.assumingMemoryBound(to: Float16.self)+ for hh in 0..<h { for r in 0..<count { for c in 0..<d {+ dst[(hh * count + r) * d + c] = Float16(source[(first + r) * stride + hh * d + c] * factor)+ } } }+ CVPixelBufferUnlockBaseAddress(surface, [])+ }+ write(a.q, f(qb), offset, l, scale); write(a.k, f(kb), 0, keys, 1); write(a.v, f(vb), 0, keys, 1)+ try a.run()+ let t = try best(3) { let s = now(); try a.run(); return now() - s }+ CVPixelBufferLockBaseAddress(a.y, .readOnly)+ let y = CVPixelBufferGetBaseAddress(a.y)!.assumingMemoryBound(to: Float16.self)+ let e = error { i, c in Float(y[i * d + c]) }+ CVPixelBufferUnlockBaseAddress(a.y, .readOnly)+ let perHead = t * 1e3 / Double(h) * Double(queries) / Double(l)+ print(String(format: "%-48@ %7.1f ms a run, %.2f ms/head (all queries); head 0 rel %.1e", name as NSString, t * 1e3, perHead, e))+ } catch {+ print("\(name): \(error)")+ }+ }++ // The CPU.+ for (threads, single) in [(8, true), (12, true), (8, false), (4, true)] {+ BLASSetThreading(single ? BLAS_THREADING_SINGLE_THREADED : BLAS_THREADING_MULTI_THREADED)+ let h = 4+ let t = best(3) {+ let s = now()+ CPUAttention.attend(q: f(qb) + offset * stride, k: f(kb), v: f(vb), out: f(ob) + offset * stride, stride: stride,+ queries: queries, keys: keys, heads: 0..<h, scale: scale, threads: threads)+ return now() - s+ }+ let e = error { i, c in f(ob)[(offset + i) * stride + c] }+ print(String(format: "CPU %2d threads, BLAS %@: %.1f ms for %d heads, %.2f ms/head; head 0 rel %.1e",+ threads, single ? "single-threaded" : "multi-threaded ", t * 1e3, h, t * 1e3 / Double(h), e))+ }+ BLASSetThreading(BLAS_THREADING_MULTI_THREADED)+ }+}diff --git a/engine-mac/Sources/KokonEngine/AttentionSplit.swift b/engine-mac/Sources/KokonEngine/AttentionSplit.swiftnew file mode 100644index 0000000..d38b85f--- /dev/null+++ b/engine-mac/Sources/KokonEngine/AttentionSplit.swift@@ -0,0 +1,163 @@+import Foundation+import Metal++/// **The target's attention cut by heads** between the GPU, the CPU and (optionally) the Neural Engine, beside one+/// another as the products are cut by rows (`Coexecution.swift`); the units would otherwise idle through ~40 % of a step.+///+/// Measured at 1024² (M2 Pro, `time-dit`, 2026-10-25), ms per layer and velocity against diffusers fp32 at 896²:+/// GPU alone 145 ms (3.7e-4); GPU 19 / CPU 13 heads 91 ms (3.6e-4: the CPU's heads are fp32); GPU 14 / Neural Engine 8 /+/// CPU 10 88 ms (9.3e-4); GPU 16 / Neural Engine 16 112 ms (1.6e-3). Beside the Neural Engine the GPU's heads slow+/// from 4.5 to 6.3–7.0 ms each (they share the memory and its cache), so its heads buy ~1.5 % for 2.5× the error:+/// off unless `KOKON_ANE_ATTENTION=1`.+///+/// heads [0, g) GPU, `flash_attention` with its grid on those heads+/// heads [g, g + a) Neural Engine, fp16, in programs of 4 heads × 1 024 queries (`NeuralEngineAttention`):+/// the GPU packs q (scaled), k, v head-first into their surfaces and unpacks y+/// heads [g + a, 32) CPU, `CPUAttention` on the AMX worker's thread, straight into `attended`+///+/// Each head is computed by one unit, so the cut reorders no sum. Only the target's queries are cut: a text segment+/// is 32 rows, microseconds on the GPU.+struct HeadSplit: CustomStringConvertible {+ var gpu: Int, ane: Int, cpu: Int+ var description: String { "GPU \(gpu) / ANE \(ane) / CPU \(cpu) heads" }++ /// From `time-dit` and `probe-attention` on the M2 Pro (ms per head at 1024² beside the others: GPU 4.5, CPU 7.0;+ /// the Neural Engine 6.0 and the GPU 6.3 when it runs), the Neural Engine in groups of 4. The balancing loop moves it.+ static func `default`(ane: Bool, cpu: Bool) -> HeadSplit {+ switch (ane, cpu) {+ case (true, true): return HeadSplit(gpu: 14, ane: 8, cpu: 10)+ case (true, false): return HeadSplit(gpu: 16, ane: 16, cpu: 0)+ case (false, true): return HeadSplit(gpu: 19, ane: 0, cpu: 13)+ case (false, false): return HeadSplit(gpu: QwenDiT.heads, ane: 0, cpu: 0)+ }+ }++ /// `KOKON_HEADS=gpu,ane,cpu` (the Neural Engine's a multiple of 4, the three summing to 32).+ static func fromEnvironment() -> HeadSplit? {+ guard let text = ProcessInfo.processInfo.environment["KOKON_HEADS"] else { return nil }+ let n = text.split(separator: ",").compactMap { Int($0) }+ guard n.count == 3, n.allSatisfy({ $0 >= 0 }), n.reduce(0, +) == QwenDiT.heads, n[1] % Coexecutor.aneHeadGroup == 0 else {+ FileHandle.standardError.write(Data("KOKON_HEADS=\(text) ignored: gpu,ane,cpu summing to 32, ane a multiple of 4\n".utf8))+ return nil+ }+ return HeadSplit(gpu: n[0], ane: n[1], cpu: n[2])+ }+}++/// One Neural Engine attention program and GPU views of its surfaces.+struct AttentionSlot {+ let program: NeuralEngineAttention+ let q: MTLBuffer, k: MTLBuffer, v: MTLBuffer, y: MTLBuffer+}++extension Coexecutor {+ static let aneHeadGroup = 4, aneQueryChunk = 1024+ /// The performance cores: `probe-attention` on the M2 Pro, 4 heads: 8 threads 27 ms, 12 the same, 4 threads 35.+ static let cpuThreads: Int = {+ var n: Int32 = 0, size = MemoryLayout<Int32>.size+ return sysctlbyname("hw.perflevel0.physicalcpu", &n, &size, nil, 0) == 0 && n > 0 ? Int(n) : 4+ }()++ /// Whether the target's attention takes the cut (below 1 024 queries the rendezvous costs more than it brings).+ func attentionApplies(queries: Int) -> Bool { queries >= 1024 && heads.gpu < QwenDiT.heads }++ /// The Neural Engine's programs for `queries` target rows on `keys` keys: per group of 4 heads, one program per+ /// chunk of 1 024 queries (the last one shorter, a multiple of 64), sharing the group's k and v surfaces.+ /// Groups are added or dropped to follow `heads.ane` (a group's programs do not depend on which heads it takes).+ func prepareAttention(queries: Int, keys: Int) throws {+ if attentionShape != (queries, keys) { attentionGroups = [] }+ attentionShape = (queries, keys)+ let wanted = heads.ane / Coexecutor.aneHeadGroup+ if attentionGroups.count > wanted { attentionGroups.removeLast(attentionGroups.count - wanted) }+ let device = context.device+ while attentionGroups.count < wanted {+ var slots: [AttentionSlot] = []+ for start in stride(from: 0, to: queries, by: Coexecutor.aneQueryChunk) {+ let rows = (min(Coexecutor.aneQueryChunk, queries - start) + 63) / 64 * 64+ let p = try NeuralEngineAttention(heads: Coexecutor.aneHeadGroup, queries: rows, keys: keys, sharing: slots.first?.program)+ let first = slots.first+ slots.append(AttentionSlot(program: p, q: try p.gpuBuffer(p.q, device: device),+ k: try first?.k ?? p.gpuBuffer(p.k, device: device),+ v: try first?.v ?? p.gpuBuffer(p.v, device: device),+ y: try p.gpuBuffer(p.y, device: device)))+ try p.run() // compile and warm+ }+ attentionGroups.append(slots)+ }+ }++ struct HeadsShape { var stride, column0, rows, valid, row0, heads: UInt32; var factor: Float }+ typealias Operand = (buffer: MTLBuffer, offset: Int, pointer: UnsafeMutablePointer<Float>)++ /// Encodes the cut attention of `queries` target rows (`q`, `out` at the first of them) on `keys` keys (`k`, `v`+ /// at row 0), rows of `stride` floats. `gpuHeads` encodes the GPU's heads `[0, g)`.+ func encodeAttention(q: Operand, k: Operand, v: Operand, out: Operand, queries: Int, keys: Int, stride: Int,+ commands: MTLCommandBuffer, gpuHeads: (Int) throws -> Void) throws {+ try prepareAttention(queries: queries, keys: keys)+ let split = heads, d = NeuralEngineAttention.headDim, group = Coexecutor.aneHeadGroup+ let scale = 1 / Float(d).squareRoot()+ counter += 1+ let p = counter+ open(p, kind: "attention", shares: (split.gpu, split.cpu, split.ane))+ func move(_ kernel: String, _ src: (MTLBuffer, Int), _ dst: (MTLBuffer, Int), _ shape: HeadsShape) throws {+ var s = shape+ let e = commands.makeComputeCommandEncoder()!+ e.label = kernel+ e.setComputePipelineState(try context.pipeline(kernel))+ e.setBuffer(src.0, offset: src.1, index: 0); e.setBuffer(dst.0, offset: dst.1, index: 1)+ e.setBytes(&s, length: MemoryLayout<HeadsShape>.stride, index: 2)+ e.dispatchThreads(MTLSize(width: d, height: Int(kernel == "ane_heads_pack" ? s.rows : s.valid), depth: Int(s.heads)),+ threadsPerThreadgroup: MTLSize(width: 128, height: 2, depth: 1))+ e.endEncoding()+ }+ func shape(_ g: Int, rows: Int, valid: Int, row0: Int, factor: Float = 1) -> HeadsShape {+ HeadsShape(stride: UInt32(stride), column0: UInt32((split.gpu + g * group) * d), rows: UInt32(rows),+ valid: UInt32(valid), row0: UInt32(row0), heads: UInt32(group), factor: factor)+ }+ for (g, slots) in attentionGroups.enumerated() {+ try move("ane_heads_pack", (k.buffer, k.offset), (slots[0].k, 0), shape(g, rows: keys, valid: keys, row0: 0))+ try move("ane_heads_pack", (v.buffer, v.offset), (slots[0].v, 0), shape(g, rows: keys, valid: keys, row0: 0))+ for (c, slot) in slots.enumerated() {+ let row0 = c * Coexecutor.aneQueryChunk+ try move("ane_heads_pack", (q.buffer, q.offset), (slot.q, 0),+ shape(g, rows: slot.program.queries, valid: min(slot.program.queries, queries - row0), row0: row0, factor: scale))+ }+ }+ if split.ane > 0 || split.cpu > 0 { commands.encodeSignalEvent(ready, value: p) }+ if split.cpu > 0 {+ nonisolated(unsafe) let (pq, pk, pv, po) = (q.pointer, k.pointer, v.pointer, out.pointer)+ let range = (split.gpu + split.ane)..<QwenDiT.heads+ amxWorker.submit { [self] in+ defer { amxDone.signaledValue = p }+ guard wait(ready, p, "CPU attention") else { return }+ let t = DispatchTime.now().uptimeNanoseconds+ CPUAttention.attend(q: pq, k: pk, v: pv, out: po, stride: stride, queries: queries, keys: keys,+ heads: range, scale: scale, threads: Coexecutor.cpuThreads)+ finished(p, \.amx, since: t)+ }+ }+ if split.ane > 0 {+ let programs = attentionGroups.flatMap { $0.map(\.program) }+ aneWorker.submit { [self] in+ defer { aneDone.signaledValue = p }+ guard wait(ready, p, "Neural Engine attention") else { return }+ let t = DispatchTime.now().uptimeNanoseconds+ do { for program in programs { try program.run() } } catch { fail("Neural Engine attention: \(error)") }+ finished(p, \.ane, since: t)+ }+ }+ try gpuHeads(split.gpu)+ gpuFinished(p, commands)+ if split.cpu > 0 { commands.encodeWaitForEvent(amxDone, value: p) }+ if split.ane > 0 {+ commands.encodeWaitForEvent(aneDone, value: p)+ for (g, slots) in attentionGroups.enumerated() {+ for (c, slot) in slots.enumerated() {+ let row0 = c * Coexecutor.aneQueryChunk+ try move("ane_heads_unpack", (slot.y, 0), (out.buffer, out.offset),+ shape(g, rows: slot.program.queries, valid: min(slot.program.queries, queries - row0), row0: row0))+ }+ }+ }+ }+}diff --git a/engine-mac/Sources/KokonEngine/Balancer.swift b/engine-mac/Sources/KokonEngine/Balancer.swiftnew file mode 100644index 0000000..362be3f--- /dev/null+++ b/engine-mac/Sources/KokonEngine/Balancer.swift@@ -0,0 +1,124 @@+import Foundation++/// **The balancing loop.** After each evaluation the timeline gives each unit's rate on each kind of operation,+/// measured beside the others (the AMX runs at about half its solo rate there), and the next evaluation's shares are+/// set so the units finish together:+///+/// - products: the Neural Engine's rows are one count for every product (its programs are compiled per count, in+/// steps of 256), chosen to minimize the predicted time of all products; the AMX's rows per kind of product (steps+/// of 64) split the rest with the GPU in proportion to their rates;+/// - the target's attention: every (Neural Engine in groups of 4, CPU, GPU) cut of the 32 heads, the one whose+/// slowest unit is fastest.+///+/// A new cut is taken only when it is predicted 1.5 % faster, so the shares settle instead of flickering. Rates are+/// smoothed over evaluations and start from the M2 Pro's, so a unit without a share keeps an estimate. Moving a+/// share moves which rows go through the Neural Engine's fp16: images stay within its accuracy, not bit for bit.+struct Balancer {+ /// Rows (heads, for attention) per millisecond.+ struct Rates { var gpu: Double, amx: Double, ane: Double }+ var rates: [String: Rates]+ /// The AMX's measured rate at each row count tried, per kind: it is not linear in rows (MLP out on the M2 Pro:+ /// 320 rows 31 ms, 384 rows 43 ms), so a measured count beats the extrapolated rate.+ var amxAt: [String: [Int: Double]] = [:]+ let aneAttention: Bool+ static let smoothing = 0.5, margin = 0.985++ /// Seeds from `time-dit` at 1024² (1856 / 256 / 2048 rows; attention 19 GPU / 13 CPU heads) and `probe-attention`+ /// (the Neural Engine's heads), M2 Pro, 2026-10-25.+ init(aneAttention: Bool) {+ self.aneAttention = aneAttention+ rates = ["4096×4096": Rates(gpu: 149, amx: 28, ane: 172), "4096×12288": Rates(gpu: 48, amx: 9.8, ane: 58),+ "12288×4096": Rates(gpu: 43.5, amx: 8.8, ane: 57.5), "attention": Rates(gpu: 0.224, amx: 0.142, ane: 0.195)]+ }++ /// Folds an evaluation's timeline into the rates: per kind, each unit's rows over its seconds.+ mutating func observe(_ entries: [Timeline.Entry]) {+ var sums: [String: (rows: Rates, ms: Rates)] = [:]+ for e in entries {+ var s = sums[e.kind] ?? (Rates(gpu: 0, amx: 0, ane: 0), Rates(gpu: 0, amx: 0, ane: 0))+ s.rows.gpu += Double(e.shares.gpu); s.ms.gpu += (e.gpu - e.released) * 1e3+ s.rows.amx += Double(e.shares.amx); s.ms.amx += e.amx.seconds * 1e3+ s.rows.ane += Double(e.shares.ane); s.ms.ane += e.ane.seconds * 1e3+ sums[e.kind] = s+ }+ for (kind, s) in sums {+ var r = rates[kind] ?? Rates(gpu: s.rows.gpu / max(s.ms.gpu, 1e-3), amx: 0, ane: 0)+ func fold(_ old: Double, _ rows: Double, _ ms: Double) -> Double {+ rows > 0 && ms > 0 ? Balancer.smoothing * old + (1 - Balancer.smoothing) * rows / ms : old+ }+ r.gpu = fold(r.gpu, s.rows.gpu, s.ms.gpu)+ r.amx = fold(r.amx, s.rows.amx, s.ms.amx)+ r.ane = fold(r.ane, s.rows.ane, s.ms.ane)+ rates[kind] = r+ if kind != "attention", s.rows.amx > 0, s.ms.amx > 0, let first = entries.first(where: { $0.kind == kind }) {+ amxAt[kind, default: [:]][first.shares.amx] = s.rows.amx / s.ms.amx+ }+ }+ }++ /// Predicted milliseconds of `counts` products of each kind with `ane` rows of `rows` on the Neural Engine and the+ /// rest cut ideally between the GPU and (when `amx`) the AMX.+ func productCost(rows: Int, ane: Int, amx: Bool, counts: [String: Int]) -> Double {+ counts.reduce(0) { sum, kc in+ guard let r = rates[kc.key] else { return sum }+ let rest = Double(rows - ane) / (r.gpu + (amx ? r.amx : 0))+ return sum + Double(kc.value) * max(ane > 0 ? Double(ane) / r.ane : 0, rest)+ }+ }++ /// Predicted milliseconds of one cut attention.+ func attentionCost(_ h: HeadSplit) -> Double {+ guard let r = rates["attention"] else { return .infinity }+ return max(Double(h.gpu) / r.gpu, h.ane > 0 ? Double(h.ane) / r.ane : 0, h.cpu > 0 ? Double(h.cpu) / r.amx : 0)+ }+}++extension Coexecutor {+ /// Reads the last evaluation's timeline and sets the next one's shares. Between evaluations only.+ func rebalance() throws {+ lastTimeline = timeline.drain()+ balancer.observe(lastTimeline)++ var counts: [String: Int] = [:]+ for e in lastTimeline where e.kind != "attention" { counts[e.kind, default: 0] += 1 }+ if !counts.isEmpty {+ if aneEnabled {+ func cost(_ e: Int) -> Double { balancer.productCost(rows: rows, ane: e, amx: amxEnabled, counts: counts) }+ let best = stride(from: 1024, through: rows - 512, by: 256).min { cost($0) < cost($1) } ?? aneRows+ if cost(best) < cost(aneRows) * Balancer.margin { try setAneRows(best) }+ }+ for kind in counts.keys {+ guard amxEnabled, let r = balancer.rates[kind] else { continue }+ // Of the two multiples of 64 around the ideal cut, the one whose slower unit is faster.+ let rest = rows - aneRows+ let ideal = Double(rest) * r.amx / (r.gpu + r.amx)+ let low = Int(ideal / 64) * 64+ func time(_ a: Int) -> Double {+ max(a > 0 ? Double(a) / (balancer.amxAt[kind]?[a] ?? r.amx) : 0, Double(rest - a) / r.gpu)+ }+ amxRows[kind] = [low - 64, low, low + 64].filter { $0 >= 0 }.min { time($0) < time($1) }!+ }+ }++ if lastTimeline.contains(where: { $0.kind == "attention" }), HeadSplit.fromEnvironment() == nil {+ let maxAne = aneEnabled && balancer.aneAttention ? QwenDiT.heads : 0+ var best = heads+ for ane in stride(from: 0, through: maxAne, by: Coexecutor.aneHeadGroup) {+ for cpu in 0...(amxEnabled ? QwenDiT.heads - 1 - ane : 0) where ane + cpu < QwenDiT.heads {+ let h = HeadSplit(gpu: QwenDiT.heads - ane - cpu, ane: ane, cpu: cpu)+ if balancer.attentionCost(h) < balancer.attentionCost(best) { best = h }+ }+ }+ if balancer.attentionCost(best) < balancer.attentionCost(heads) * Balancer.margin {+ heads = best+ try prepareAttention(queries: attentionShape.queries, keys: attentionShape.keys)+ }+ }+ }++ /// The shares the next evaluation takes.+ var shares: String {+ let amx = amxRows.sorted { $0.key < $1.key }.map { "\($0.key) \($0.value)" }.joined(separator: ", ")+ return "products ANE \(aneRows) rows, AMX \(amx.isEmpty ? "default" : amx); attention \(heads)"+ }+}diff --git a/engine-mac/Sources/KokonEngine/CPUAttention.swift b/engine-mac/Sources/KokonEngine/CPUAttention.swiftnew file mode 100644index 0000000..5bbc8cd--- /dev/null+++ b/engine-mac/Sources/KokonEngine/CPUAttention.swift@@ -0,0 +1,73 @@+import Accelerate+import Foundation++/// **Attention of some heads on the CPU** (the AMX for the two products, the cores for the softmax), fp32, on the+/// layout the products write: rows of `stride` floats, head h at columns `[128 h, 128 h + 128)`. No mask: the+/// target's queries see every key.+///+/// Work items are (head, block of `blockRows` queries), cut into one contiguous run per thread, so the bits do not+/// depend on which thread computed a block. Per item:+///+/// S = scale · Q_b · K_hᵀ cblas_sgemm [blockRows, keys]+/// P = exp(S − max S), l = Σ P vDSP, vvexpf per row+/// O_b = (P · V_h) / l cblas_sgemm, then the rows scaled into `out`+///+/// The operands are packed first (Q_b, and K_h, V_h once per head of a run): read in place a head's rows are 16 KB+/// apart, a power of two, and land in the same cache sets (Siliconed measured a head 1.8× slower so).+enum CPUAttention {+ static let blockRows = 128, headDim = 128++ /// Heads `heads` of `queries` query rows against `keys` key rows; `q` points at the first query row.+ static func attend(q: UnsafePointer<Float>, k: UnsafePointer<Float>, v: UnsafePointer<Float>,+ out: UnsafeMutablePointer<Float>, stride: Int, queries: Int, keys: Int, heads: Range<Int>,+ scale: Float, threads: Int) {+ let blocks = (queries + blockRows - 1) / blockRows+ let items = heads.count * blocks+ guard items > 0 else { return }+ let threads = max(1, min(threads, items))+ nonisolated(unsafe) let (q, k, v, out) = (q, k, v, out)+ DispatchQueue.concurrentPerform(iterations: threads) { t in+ let first = items * t / threads, end = items * (t + 1) / threads+ let d = headDim+ let kh = UnsafeMutablePointer<Float>.allocate(capacity: keys * d)+ let vh = UnsafeMutablePointer<Float>.allocate(capacity: keys * d)+ let qb = UnsafeMutablePointer<Float>.allocate(capacity: blockRows * d)+ let s = UnsafeMutablePointer<Float>.allocate(capacity: blockRows * keys)+ let ob = UnsafeMutablePointer<Float>.allocate(capacity: blockRows * d)+ defer { kh.deallocate(); vh.deallocate(); qb.deallocate(); s.deallocate(); ob.deallocate() }+ var packed = -1+ for item in first..<end {+ let head = heads.lowerBound + item / blocks, block = item % blocks+ let column = head * d+ if head != packed {+ for r in 0..<keys {+ (kh + r * d).update(from: k + r * stride + column, count: d)+ (vh + r * d).update(from: v + r * stride + column, count: d)+ }+ packed = head+ }+ let row0 = block * blockRows, rows = min(blockRows, queries - row0)+ for r in 0..<rows { (qb + r * d).update(from: q + (row0 + r) * stride + column, count: d) }+ cblas_sgemm(CblasRowMajor, CblasNoTrans, CblasTrans, Int32(rows), Int32(keys), Int32(d), scale,+ qb, Int32(d), kh, Int32(d), 0, s, Int32(keys))+ var sums = [Float](repeating: 0, count: rows)+ var n = Int32(keys)+ for r in 0..<rows {+ let row = s + r * keys+ var peak: Float = 0+ vDSP_maxv(row, 1, &peak, vDSP_Length(keys))+ var shift = -peak+ vDSP_vsadd(row, 1, &shift, row, 1, vDSP_Length(keys))+ vvexpf(row, row, &n)+ vDSP_sve(row, 1, &sums[r], vDSP_Length(keys))+ }+ cblas_sgemm(CblasRowMajor, CblasNoTrans, CblasNoTrans, Int32(rows), Int32(d), Int32(keys), 1,+ s, Int32(keys), vh, Int32(d), 0, ob, Int32(d))+ for r in 0..<rows {+ var inverse = 1 / sums[r]+ vDSP_vsmul(ob + r * d, 1, &inverse, out + (row0 + r) * stride + column, 1, vDSP_Length(d))+ }+ }+ }+ }+}diff --git a/engine-mac/Sources/KokonEngine/Checks.swift b/engine-mac/Sources/KokonEngine/Checks.swiftindex 2677bbd..7154aae 100644--- a/engine-mac/Sources/KokonEngine/Checks.swift+++ b/engine-mac/Sources/KokonEngine/Checks.swift@@ -118,15 +118,17 @@ extension Checks {}let encoder = random(text * QwenDiT.dim, 8), latent = random(sequence.target * QwenDiT.channels, 2)print("sequence: \(sequence.count) rows (\(sequence.prefix) text, \(sequence.target) target)"- + (dit.coexecutor.map { ", products split \($0.split(m: QwenDiT.padded(sequence.count)))" } ?? ", GPU only"))+ + (dit.coexecutor.map { ", products split \($0.split(m: QwenDiT.padded(sequence.count), k: 4096, n: 4096)), attention \($0.heads)" } ?? ", GPU only"))for run in 0..<runs {let started = DispatchTime.now().uptimeNanosecondslet out = try dit.forward(encoder: encoder, target: latent, sigma: 0.5)let seconds = Double(DispatchTime.now().uptimeNanoseconds - started) * 1e-9print(String(format: "evaluation %d: %.2f s%@", run + 1, seconds, out.allSatisfy(\.isFinite) ? "" : " NON-FINITE OUTPUT"))if let c = dit.coexecutor {- print(String(format: " AMX busy %.2f s, Neural Engine busy %.2f s%@", c.amxSeconds, c.aneSeconds,+ print(String(format: " AMX/CPU busy %.2f s, Neural Engine busy %.2f s%@", c.amxSeconds, c.aneSeconds,c.failures.isEmpty ? "" : "; FAILURES: \(c.failures.prefix(3))"))+ print(Timeline.summary(dit.balancing ? c.lastTimeline : c.timeline.drain()))+ if dit.balancing { print(" next: \(c.shares)") }c.resetTimings()}}@@ -196,12 +198,22 @@ extension Checks {print(String(format: "VAE %d² → %d²: rel %.2e, max |Δ| %.3g%@ (%.2f s with the graph build)", grid, grid * 16,r.relative, r.maxAbs, r.finite ? "" : " NON-FINITE", first))if let t = timeGrid {+ started = DispatchTime.now().uptimeNanoseconds+ try vae.prepare(height: t, width: t)+ print(String(format: "build and compile %d²: %.2f s", t * 16, Double(DispatchTime.now().uptimeNanoseconds - started) * 1e-9))let random = (0..<(64 * t * t)).map { Float(($0 * 7919) % 1000) / 500 - 1 }+ func cpu() -> (user: Double, system: Double) {+ var u = rusage()+ getrusage(RUSAGE_SELF, &u)+ return (Double(u.ru_utime.tv_sec) + Double(u.ru_utime.tv_usec) * 1e-6, Double(u.ru_stime.tv_sec) + Double(u.ru_stime.tv_usec) * 1e-6)+ }for run in 0..<2 {started = DispatchTime.now().uptimeNanoseconds+ let before = cpu()_ = try vae.decode(random, height: t, width: t)- print(String(format: "decode %d² (run %d): %.2f s", t * 16, run + 1,- Double(DispatchTime.now().uptimeNanoseconds - started) * 1e-9))+ let after = cpu()+ print(String(format: "decode %d² (run %d): %.2f s (CPU user %.2f s, system %.2f s)", t * 16, run + 1,+ Double(DispatchTime.now().uptimeNanoseconds - started) * 1e-9, after.user - before.user, after.system - before.system))}}}diff --git a/engine-mac/Sources/KokonEngine/Coexecution.swift b/engine-mac/Sources/KokonEngine/Coexecution.swiftindex 270faa6..fa3610d 100644--- a/engine-mac/Sources/KokonEngine/Coexecution.swift+++ b/engine-mac/Sources/KokonEngine/Coexecution.swift@@ -22,40 +22,81 @@ final class Coexecutor {static let tile = 4096let context: MetalContext- let aneRows: Int- private let slots: [(product: NeuralEngineProduct, x: MTLBuffer, w: MTLBuffer, y: MTLBuffer)]- private let ready: MTLSharedEvent, amxDone: MTLSharedEvent, aneDone: MTLSharedEvent- private var counter: UInt64 = 0- private let amxWorker = Worker(name: "kokon.amx"), aneWorker = Worker(name: "kokon.ane")- /// λ (12 288), lifted norms (12 288), L, σ (aneRows): the Neural Engine's scales.+ /// The rows of every split product (the DiT's padded sequence).+ let rows: Int+ /// The Neural Engine's rows, the same for every product (its programs are compiled per row count), and the+ /// AMX's per kind of product (`"k×n"`): set by the balancing loop (`Balancer.swift`).+ private(set) var aneRows: Int+ var amxRows: [String: Int] = [:]+ typealias ProductSlot = (product: NeuralEngineProduct, x: MTLBuffer, w: MTLBuffer, y: MTLBuffer)+ private var slots: [ProductSlot]+ /// Programs of other row counts, kept while the balancing loop may come back to them.+ private var spareSlots: [Int: [ProductSlot]] = [:]+ var balancer: Balancer+ /// The timeline of the last evaluation, once the balancing loop has read it.+ var lastTimeline: [Timeline.Entry] = []+ let ready: MTLSharedEvent, amxDone: MTLSharedEvent, aneDone: MTLSharedEvent+ /// Signaled after the GPU's own share of each operation, for the timeline only.+ let gpuDone: MTLSharedEvent+ let listener = MTLSharedEventListener(dispatchQueue: DispatchQueue(label: "kokon.timeline"))+ let timeline = Timeline()+ var counter: UInt64 = 0+ let amxWorker = Worker(name: "kokon.amx"), aneWorker = Worker(name: "kokon.ane")+ /// σ of the Neural Engine's rows (`rows` at most).private let scales: MTLBuffer- /// The share of rows each unit takes; `KOKON_SPLIT=gpu,amx,ane` (fractions) overrides.- var amxShare: Double, aneEnabled: Bool, amxEnabled: Bool+ /// Per weight, its columns' λ, their lifted norms and L (`[λ (n) | norms (n) | L]`): they depend on the weight+ /// alone, so they are computed the first time it is used and kept (~11 MB for the DiT). Two full reads of every+ /// weight per evaluation, by only n threads, were most of the GPU's time alone between products.+ private var columnScales: [String: MTLBuffer] = [:]+ let aneEnabled: Bool, amxEnabled: Bool/// Seconds each worker spent computing (not waiting), and failures (a failure leaves rows unwritten).- private(set) var amxSeconds = 0.0, aneSeconds = 0.0- private(set) var failures: [String] = []- private let lock = NSLock()+ var amxSeconds = 0.0, aneSeconds = 0.0+ var failures: [String] = []+ let lock = NSLock()+ /// The target's attention cut by heads (`AttentionSplit.swift`), and the Neural Engine's programs for it.+ var heads: HeadSplit+ var attentionGroups: [[AttentionSlot]] = []+ var attentionShape = (queries: 0, keys: 0)- /// Defaults measured on the M2 Pro 16 GB, 1024² + 32 text rows (`time-dit`, 2026-10-25): AMX 512 rows 12.8 s per- /// evaluation (the AMX the slowest, ~35 % under its solo rate beside the others), 256 rows 11.3 s, none 11.8 s;- /// GPU alone 16.2 s. A balancing loop per shape replaces the fixed share later.- init(context: MetalContext, aneRows: Int = 2048, amxShare: Double = 0.0625, ane: Bool = true, amx: Bool = true) throws {+ /// Starting shares measured on the M2 Pro 16 GB, 1024² + 32 text rows (`time-dit`, 2026-10-25): 2 048 rows on the+ /// Neural Engine and 1/16 on the AMX (AMX 512 rows 12.8 s per evaluation, 256 rows 11.3 s, none 11.8 s; GPU alone+ /// 16.2 s). The balancing loop moves them from the first evaluation on.+ init(context: MetalContext, rows: Int, ane: Bool = true, amx: Bool = true, aneAttention: Bool = false) throws {self.context = context- self.aneRows = aneRows- self.amxShare = amxShare+ self.rows = rowsself.aneEnabled = aneself.amxEnabled = amx+ aneRows = ane ? min(2048, max(0, (rows - 512) / 256 * 256)) : 0+ heads = HeadSplit.fromEnvironment() ?? HeadSplit.default(ane: ane && aneAttention, cpu: amx)+ balancer = Balancer(aneAttention: ane && aneAttention)let device = context.device- guard let r = device.makeSharedEvent(), let a = device.makeSharedEvent(), let e = device.makeSharedEvent() else {+ guard let r = device.makeSharedEvent(), let a = device.makeSharedEvent(), let e = device.makeSharedEvent(),+ let g = device.makeSharedEvent() else {throw MetalContext.Failure.buffer("shared events")}- (ready, amxDone, aneDone) = (r, a, e)- scales = try context.scratch((2 * 12288 + 16 + aneRows) * 4, label: "ane scales")- slots = ane ? try (0..<3).map { _ in- let p = try NeuralEngineProduct(rows: aneRows, k: Coexecutor.tile, n: Coexecutor.tile)- try p.run() // compile and warm+ (ready, amxDone, aneDone, gpuDone) = (r, a, e, g)+ scales = try context.scratch(rows * 4, label: "ane row scales")+ slots = []+ slots = try productSlots(aneRows)+ }++ /// Three programs of `rows` rows (the tiles of a product, at most 3), compiled and warmed.+ private func productSlots(_ rows: Int) throws -> [ProductSlot] {+ guard rows > 0 else { return [] }+ let device = context.device+ return try (0..<3).map { _ in+ let p = try NeuralEngineProduct(rows: rows, k: Coexecutor.tile, n: Coexecutor.tile)+ try p.run()return (p, try p.gpuBuffer(p.x, device: device), try p.gpuBuffer(p.w, device: device), try p.gpuBuffer(p.y, device: device))- } : []+ }+ }++ /// Moves the Neural Engine to `rows` rows per product; between evaluations only (the GPU must be idle).+ func setAneRows(_ rows: Int) throws {+ guard rows != aneRows else { return }+ let next = try spareSlots.removeValue(forKey: rows) ?? productSlots(rows)+ if aneRows > 0 { spareSlots = [aneRows: slots] } // the last one only: 192 MB of surfaces at 2 048 rows+ (slots, aneRows) = (next, rows)}/// Whether a product takes the split: the Neural Engine's tiles need K and N in multiples of 4 096 with at@@ -64,24 +105,26 @@ final class Coexecutor {m >= 1024 && k % Coexecutor.tile == 0 && n % Coexecutor.tile == 0 && (k / Coexecutor.tile) * (n / Coexecutor.tile) <= 3}- func split(m: Int) -> Split {- let ane = aneEnabled && m >= aneRows + 512 ? aneRows : 0- let amx = amxEnabled ? Int((Double(m) * amxShare / 256).rounded()) * 256 : 0+ func split(m: Int, k: Int, n: Int) -> Split {+ let ane = aneEnabled && m == rows ? aneRows : 0+ let amx = amxEnabled ? amxRows["\(k)×\(n)"] ?? Int((Double(m) / 16 / 64).rounded()) * 64 : 0return Split(gpu: m - ane - amx, amx: amx, ane: ane)}struct Shape { var k, n, k0, kPart, n0, nPart, row0, rows, kParts: UInt32 }- /// Encodes the split product. `gpuRows` encodes the GPU's own rows `[0, g)` into the command buffer.- func encode(a: (buffer: MTLBuffer, offset: Int, pointer: UnsafeMutablePointer<Float>),+ /// Encodes the split product of the weight named `weight`. `gpuRows` encodes the GPU's own rows `[0, g)` into the+ /// command buffer.+ func encode(weight: String, a: (buffer: MTLBuffer, offset: Int, pointer: UnsafeMutablePointer<Float>),w: (buffer: MTLBuffer, offset: Int, pointer: UnsafeMutablePointer<Float>),c: (buffer: MTLBuffer, offset: Int, pointer: UnsafeMutablePointer<Float>),m: Int, k: Int, n: Int, commands: MTLCommandBuffer, gpuRows: (Int) throws -> Void) throws {- let s = split(m: m)+ let s = split(m: m, k: k, n: n)counter += 1let p = counter+ open(p, kind: "\(k)×\(n)", shares: (s.gpu, s.amx, s.ane))let (kParts, nParts) = (k / Coexecutor.tile, n / Coexecutor.tile)- let lambda = 0, norm = 12288 * 4, state = 2 * 12288 * 4, sigma = 2 * 12288 * 4 + 64+ let (lambda, norm, state, sigma) = (0, n * 4, 2 * n * 4, 0)func shape(kp: Int = 0, np: Int = 0) -> Shape {Shape(k: UInt32(k), n: UInt32(n), k0: UInt32(kp * Coexecutor.tile), kPart: UInt32(Coexecutor.tile),n0: UInt32(np * Coexecutor.tile), nPart: UInt32(Coexecutor.tile), row0: UInt32(s.gpu + s.amx),@@ -94,21 +137,26 @@ final class Coexecutor {try body(e)e.endEncoding()}+ // (Read only when the Neural Engine takes rows.)+ let columns = s.ane > 0 ? try columnScales[weight] ?? context.scratch((2 * n + 16) * 4, label: "λ \(weight)") : scalesif s.ane > 0 {var sh = shape()- try kernel("ane_columns") { e in- e.setBuffer(w.buffer, offset: w.offset, index: 0)- e.setBuffer(scales, offset: lambda, index: 1); e.setBuffer(scales, offset: norm, index: 2)- e.setBytes(&sh, length: MemoryLayout<Shape>.stride, index: 3)- e.dispatchThreads(MTLSize(width: n, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 64, height: 1, depth: 1))- }- try kernel("ane_norm") { e in- e.setBuffer(scales, offset: norm, index: 0); e.setBuffer(scales, offset: state, index: 1)- e.setBytes(&sh, length: MemoryLayout<Shape>.stride, index: 2)- e.dispatchThreadgroups(MTLSize(width: 1, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 1024, height: 1, depth: 1))+ if columnScales[weight] == nil {+ columnScales[weight] = columns+ try kernel("ane_columns") { e in+ e.setBuffer(w.buffer, offset: w.offset, index: 0)+ e.setBuffer(columns, offset: lambda, index: 1); e.setBuffer(columns, offset: norm, index: 2)+ e.setBytes(&sh, length: MemoryLayout<Shape>.stride, index: 3)+ e.dispatchThreads(MTLSize(width: n, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 64, height: 1, depth: 1))+ }+ try kernel("ane_norm") { e in+ e.setBuffer(columns, offset: norm, index: 0); e.setBuffer(columns, offset: state, index: 1)+ e.setBytes(&sh, length: MemoryLayout<Shape>.stride, index: 2)+ e.dispatchThreadgroups(MTLSize(width: 1, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 1024, height: 1, depth: 1))+ }}try kernel("ane_row_scales") { e in- e.setBuffer(a.buffer, offset: a.offset, index: 0); e.setBuffer(scales, offset: state, index: 1)+ e.setBuffer(a.buffer, offset: a.offset, index: 0); e.setBuffer(columns, offset: state, index: 1)e.setBuffer(scales, offset: sigma, index: 2)e.setBytes(&sh, length: MemoryLayout<Shape>.stride, index: 3)e.dispatchThreadgroups(MTLSize(width: s.ane, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))@@ -118,7 +166,7 @@ final class Coexecutor {let slot = slots[kp * nParts + np]var t = shape(kp: kp, np: np)try kernel("ane_pack_weight") { e in- e.setBuffer(w.buffer, offset: w.offset, index: 0); e.setBuffer(scales, offset: lambda, index: 1)+ e.setBuffer(w.buffer, offset: w.offset, index: 0); e.setBuffer(columns, offset: lambda, index: 1)e.setBuffer(slot.w, offset: 0, index: 2)e.setBytes(&t, length: MemoryLayout<Shape>.stride, index: 3)e.dispatchThreads(MTLSize(width: Coexecutor.tile, height: Coexecutor.tile, depth: 1),@@ -144,7 +192,7 @@ final class Coexecutor {let t = DispatchTime.now().uptimeNanosecondscblas_sgemm(CblasRowMajor, CblasNoTrans, CblasNoTrans, Int32(rows), Int32(n), Int32(k), 1,pa, Int32(k), pw, Int32(n), 0, pc, Int32(n))- lock.withLock { amxSeconds += Double(DispatchTime.now().uptimeNanoseconds - t) * 1e-9 }+ finished(p, \.amx, since: t)}}if s.ane > 0 {@@ -154,10 +202,11 @@ final class Coexecutor {guard wait(ready, p, "ANE") else { return }let t = DispatchTime.now().uptimeNanosecondsdo { for tile in tiles { try tile.run() } } catch { fail("ANE: \(error)") }- lock.withLock { aneSeconds += Double(DispatchTime.now().uptimeNanoseconds - t) * 1e-9 }+ finished(p, \.ane, since: t)}}try gpuRows(s.gpu)+ gpuFinished(p, commands)if s.amx > 0 { commands.encodeWaitForEvent(amxDone, value: p) }if s.ane > 0 {commands.encodeWaitForEvent(aneDone, value: p)@@ -166,7 +215,7 @@ final class Coexecutor {let ys = (0..<3).map { kp in slots[min(kp, kParts - 1) * nParts + np].y }try kernel("ane_unpack") { e infor (i, y) in ys.enumerated() { e.setBuffer(y, offset: 0, index: i) }- e.setBuffer(scales, offset: lambda, index: 3); e.setBuffer(scales, offset: sigma, index: 4)+ e.setBuffer(columns, offset: lambda, index: 3); e.setBuffer(scales, offset: sigma, index: 4)e.setBuffer(c.buffer, offset: c.offset, index: 5)e.setBytes(&t, length: MemoryLayout<Shape>.stride, index: 6)e.dispatchThreads(MTLSize(width: Coexecutor.tile, height: s.ane, depth: 1),@@ -176,17 +225,96 @@ final class Coexecutor {}}- private func fail(_ message: String) { lock.withLock { failures.append(message) } }+ func fail(_ message: String) { lock.withLock { failures.append(message) } }/// Waits for the GPU to reach `value`; false (and a failure) after a minute, so a broken command buffer never/// hangs a worker for good.- private func wait(_ event: MTLSharedEvent, _ value: UInt64, _ who: String) -> Bool {+ func wait(_ event: MTLSharedEvent, _ value: UInt64, _ who: String) -> Bool {if event.wait(untilSignaledValue: value, timeoutMS: 60_000) { return true }fail("\(who): the GPU never reached product \(value)")return false}func resetTimings() { lock.withLock { amxSeconds = 0; aneSeconds = 0 } }++ // MARK: - the timeline++ /// Starts operation `p`'s entry and notes when the GPU releases it (`ready` reaches `p`).+ func open(_ p: UInt64, kind: String, shares: (gpu: Int, amx: Int, ane: Int)) {+ timeline.open(p, kind: kind, shares: shares)+ ready.notify(listener, atValue: p) { [timeline] _, _ in timeline.mark(p, \.released, Timeline.now()) }+ }++ /// Encodes the GPU's end-of-share signal for operation `p` and notes when it fires.+ func gpuFinished(_ p: UInt64, _ commands: MTLCommandBuffer) {+ commands.encodeSignalEvent(gpuDone, value: p)+ gpuDone.notify(listener, atValue: p) { [timeline] _, _ in timeline.mark(p, \.gpu, Timeline.now()) }+ }++ /// A worker's share of `p` done: its busy seconds, and its start and end in the timeline.+ func finished(_ p: UInt64, _ unit: WritableKeyPath<Timeline.Entry, Timeline.Span>, since start: UInt64) {+ let end = DispatchTime.now().uptimeNanoseconds+ let seconds = Double(end - start) * 1e-9+ lock.withLock { if unit == \Timeline.Entry.amx { amxSeconds += seconds } else { aneSeconds += seconds } }+ timeline.mark(p, unit, Timeline.Span(start: Double(start) * 1e-9, end: Double(end) * 1e-9))+ }+}++/// **When each split operation ran on each unit**, from the GPU's release (`ready`) to each unit's end: what the+/// balancing loop reads. Times are seconds of the uptime clock; the GPU's are noted by an event listener (tens of+/// microseconds late, against operations of milliseconds).+final class Timeline: @unchecked Sendable {+ struct Span { var start = 0.0, end = 0.0; var seconds: Double { end - start } }+ struct Entry {+ var kind: String+ var shares: (gpu: Int, amx: Int, ane: Int)+ var released = 0.0, gpu = 0.0+ var amx = Span(), ane = Span()+ /// Seconds from the release to the end of each unit's share (0 for a unit without one).+ var walls: (gpu: Double, amx: Double, ane: Double) {+ (gpu - released, shares.amx > 0 ? amx.end - released : 0, shares.ane > 0 ? ane.end - released : 0)+ }+ }+ private var entries: [UInt64: Entry] = [:]+ private let lock = NSLock()++ static func now() -> Double { Double(DispatchTime.now().uptimeNanoseconds) * 1e-9 }++ func open(_ p: UInt64, kind: String, shares: (gpu: Int, amx: Int, ane: Int)) {+ lock.withLock { entries[p] = Entry(kind: kind, shares: shares) }+ }++ func mark<T>(_ p: UInt64, _ field: WritableKeyPath<Entry, T>, _ value: T) {+ lock.withLock { entries[p]?[keyPath: field] = value }+ }++ /// The complete entries, in order, and forgets them all.+ func drain() -> [Entry] {+ lock.withLock {+ defer { entries = [:] }+ return entries.sorted { $0.key < $1.key }.map(\.value).filter { $0.released > 0 && $0.gpu > 0 }+ }+ }++ /// Per kind: operations, mean milliseconds of each unit's share from its own start (GPU: from the release),+ /// and the mean wall from release to the last unit's end.+ static func summary(_ entries: [Entry]) -> String {+ var kinds: [String] = []+ var groups: [String: [Entry]] = [:]+ for e in entries {+ if groups[e.kind] == nil { kinds.append(e.kind) }+ groups[e.kind, default: []].append(e)+ }+ return kinds.map { kind in+ let g = groups[kind]!, n = Double(g.count), s = g[0].shares+ func mean(_ f: (Entry) -> Double) -> Double { g.map(f).reduce(0, +) / n * 1e3 }+ return String(format: " %-12@ ×%3d shares %d/%d/%d GPU %6.2f ms AMX/CPU %6.2f ms (starts +%.2f) ANE %6.2f ms (starts +%.2f) wall %6.2f ms",+ kind as NSString, g.count, s.gpu, s.amx, s.ane, mean { $0.gpu - $0.released },+ mean { $0.amx.seconds }, mean { $0.shares.amx > 0 ? $0.amx.start - $0.released : 0 },+ mean { $0.ane.seconds }, mean { $0.shares.ane > 0 ? $0.ane.start - $0.released : 0 },+ mean { max($0.walls.gpu, $0.walls.amx, $0.walls.ane) })+ }.joined(separator: "\n")+ }}/// A thread running submitted jobs in order.diff --git a/engine-mac/Sources/KokonEngine/Kernels/ane.metal b/engine-mac/Sources/KokonEngine/Kernels/ane.metalindex 3328586..0e7e6a5 100644--- a/engine-mac/Sources/KokonEngine/Kernels/ane.metal+++ b/engine-mac/Sources/KokonEngine/Kernels/ane.metal@@ -111,3 +111,30 @@ kernel void ane_unpack(device const half *y0 [[buffer(0)]], device const half *yuint j = s.n0 + jj;c[(ulong)(s.row0 + i) * s.n + j] = (sum / lambda[j]) / sigma[i];}++// Attention heads to and from the Neural Engine's surfaces [heads, rows, 128] fp16 (heads first), from the rows of+// `stride` floats the products write (head h at columns column0 + 128 h). Surface rows past `valid` are zeros.+struct HeadsShape {+ uint stride;+ uint column0;+ uint rows; // rows per head in the surface+ uint valid; // of which exist in the fp32 tensor, from row0+ uint row0;+ uint heads;+ float factor; // q: 1/sqrt(128)+};++kernel void ane_heads_pack(device const float *src [[buffer(0)]], device half *dst [[buffer(1)]],+ constant HeadsShape &s [[buffer(2)]], uint3 g [[thread_position_in_grid]]) {+ uint c = g.x, r = g.y, h = g.z;+ if (c >= 128 || r >= s.rows || h >= s.heads) return;+ float x = r < s.valid ? src[(ulong)(s.row0 + r) * s.stride + s.column0 + h * 128 + c] * s.factor : 0.0f;+ dst[((ulong)h * s.rows + r) * 128 + c] = half(x);+}++kernel void ane_heads_unpack(device const half *src [[buffer(0)]], device float *dst [[buffer(1)]],+ constant HeadsShape &s [[buffer(2)]], uint3 g [[thread_position_in_grid]]) {+ uint c = g.x, r = g.y, h = g.z;+ if (c >= 128 || r >= s.valid || h >= s.heads) return;+ dst[(ulong)(s.row0 + r) * s.stride + s.column0 + h * 128 + c] = float(src[((ulong)h * s.rows + r) * 128 + c]);+}diff --git a/engine-mac/Sources/KokonEngine/Kernels/attention.metal b/engine-mac/Sources/KokonEngine/Kernels/attention.metalindex 1b7e784..263faf7 100644--- a/engine-mac/Sources/KokonEngine/Kernels/attention.metal+++ b/engine-mac/Sources/KokonEngine/Kernels/attention.metal@@ -5,10 +5,20 @@ using namespace metal;// Fused scaled dot-product attention, fp32, head_dim 128, on the layout the products write: q, k, v and out are// rows of `stride` floats (all heads side by side), head h at columns [128 h, 128 h + 128).//-// A threadgroup takes 64 query rows of one head (8 SIMD groups of 8 rows) and walks the keys 32 at a time with an+// A threadgroup takes 64 query rows of one head (8 SIMD groups of 8 rows) and walks the keys 64 at a time with an// online softmax (running max and sum per row), so the [queries, keys] score matrix never exists in memory. Each-// key tile, then value tile, is loaded once into threadgroup memory by all 256 threads (coalesced rows) and shared-// by the 8 SIMD groups; products on 8×8 SIMD-group matrices.+// key tile, then its value tile, is loaded once into threadgroup memory by all 256 threads (coalesced rows) and+// shared by the 8 SIMD groups; products on 8×8 SIMD-group matrices.+//+// The softmax stays in registers: each lane holds two elements of every 8×8 matrix, row (lane / 16) * 4 ++// (lane / 2) % 4, columns ((lane / 8) % 2) * 4 + (lane % 2) * 2 and the next (`thread_elements`, the layout of+// Apple's GPUs); the four lanes of a row differ in lane bits 0 and 3, so a row's max and sum take two shuffles, and+// O is rescaled in place.+//+// Measured at 1024² (4 096 queries × 4 128 keys, 32 heads, M2 Pro, `probe-attention`, 2026-10-25): 120 ms; the+// first version (32 keys a tile, scores through threadgroup memory, O rescaled by a diagonal product) 145 ms; with+// separate 32-key K and V tiles 135; 16-key tiles or 4 SIMD groups 1.5-2.6× slower, 16 SIMD groups 178, K and V read+// straight from memory by each SIMD group 520.//// causal: the queries are the last `queries` of the keys; query i sees keys j <= keys - queries + i (a text// segment of Qwen-Image 2.1's block-causal mask). Rows past `queries` are computed but never stored and keys past@@ -17,7 +27,7 @@ using namespace metal;// Barriers are threadgroup barriers: with simdgroup barriers alone the lanes' writes to threadgroup memory were// sometimes not seen by the SIMD-group loads that followed (1 run in 4 wrong, M2 Pro, macOS 27, AttentionTests).-constant uint BQ = 64, BK = 32, D = 128, GROUPS = 8;+constant uint BQ = 64, BK = 64, D = 128;struct FlashParams {uint queries;@@ -28,6 +38,11 @@ struct FlashParams {uint headBase;};+// A lane's own two elements of an 8×8 matrix (`thread_elements` is typed as the whole matrix).+static inline thread float2 &elements(thread simdgroup_float8x8 &m) {+ return *(thread float2 *)&m.thread_elements();+}+kernel void flash_attention(device const float *q [[buffer(0)]], device const float *k [[buffer(1)]],device const float *v [[buffer(2)]], device float *out [[buffer(3)]],constant FlashParams &p [[buffer(4)]],@@ -35,11 +50,7 @@ kernel void flash_attention(device const float *q [[buffer(0)]], device const fluint tid [[thread_index_in_threadgroup]],uint sg [[simdgroup_index_in_threadgroup]],uint lane [[thread_index_in_simdgroup]]) {- threadgroup float tile[BK * D]; // 16 KB: the key tile, then the value tile- threadgroup float stageAll[GROUPS][8 * BK]; // 8 KB: each SIMD group's scores / probabilities- threadgroup float diagAll[GROUPS][64]; // 2 KB: each SIMD group's per-row scale, as a diagonal- threadgroup float *stage = stageAll[sg];- threadgroup float *diag = diagAll[sg];+ threadgroup float tile[BK * D]; // 32 KB: the key tile, then the value tileconst uint head = p.headBase + tg.y;const uint q0 = tg.x * BQ + sg * 8;@@ -52,11 +63,9 @@ kernel void flash_attention(device const float *q [[buffer(0)]], device const flsimdgroup_load(Q[d], q + (ulong)q0 * stride + head * D + 8 * d, stride);O[d] = make_filled_simdgroup_matrix<float, 8, 8>(0.0f);}- diag[lane * 2] = 0.0f;- diag[lane * 2 + 1] = 0.0f;- // Each lane owns 8 scores of one row: row lane / 4, columns (lane % 4) * 8 ... + 8.- const uint row = lane / 4, cbase = (lane % 4) * 8;+ const uint row = (lane / 16) * 4 + (lane / 2) % 4;+ const uint col = ((lane / 8) % 2) * 4 + (lane % 2) * 2;const uint qi = q0 + row;const uint offset = p.keys - p.queries;const int limit = p.causal ? int(offset + qi) : int(p.keys) - 1;@@ -66,94 +75,76 @@ kernel void flash_attention(device const float *q [[buffer(0)]], device const flif (p.causal) kEnd = min(p.keys, offset + min(tg.x * BQ + BQ - 1, p.queries - 1) + 1);for (uint j0 = 0; j0 < kEnd; j0 += BK) {- // Key tile: 32 rows × 128, 256 threads × 4 float4.+ // Key tile: 64 rows × 128, 256 threads × 8 float4.for (uint i = tid; i < BK * D / 4; i += 256) {uint r = i / (D / 4), c = i % (D / 4);((threadgroup float4 *)tile)[i] = *((device const float4 *)(kh + (ulong)(j0 + r) * stride) + c);}threadgroup_barrier(mem_flags::mem_threadgroup);- simdgroup_float8x8 S[4];- for (uint c = 0; c < 4; c++) S[c] = make_filled_simdgroup_matrix<float, 8, 8>(0.0f);+ simdgroup_float8x8 S[8];+ for (uint c = 0; c < 8; c++) S[c] = make_filled_simdgroup_matrix<float, 8, 8>(0.0f);for (uint d = 0; d < 16; d++) {- for (uint c = 0; c < 4; c++) {+ for (uint c = 0; c < 8; c++) {simdgroup_float8x8 Kt;simdgroup_load(Kt, tile + 8 * c * D + 8 * d, D, ulong2(0, 0), true);simdgroup_multiply_accumulate(S[c], Q[d], Kt, S[c]);}}- for (uint c = 0; c < 4; c++) simdgroup_store(S[c], stage + 8 * c, BK);- threadgroup_barrier(mem_flags::mem_threadgroup);- // Value tile into the same memory (everyone is done with the keys), softmax meanwhile.- for (uint i = tid; i < BK * D / 4; i += 256) {- uint r = i / (D / 4), c = i % (D / 4);- ((threadgroup float4 *)tile)[i] = *((device const float4 *)(vh + (ulong)(j0 + r) * stride) + c);- }- float s[8];+ // Online softmax on the lane's two scores of each of the 8 matrices.+ float2 s[8];float blockMax = -INFINITY;- for (uint t = 0; t < 8; t++) {- int j = int(j0 + cbase + t);- float x = stage[row * BK + cbase + t] * p.scale;- if (j > limit) x = -INFINITY;- s[t] = x;- blockMax = max(blockMax, x);+ for (uint c = 0; c < 8; c++) {+ s[c] = elements(S[c]) * p.scale;+ int j = int(j0 + 8 * c + col);+ if (j > limit) s[c].x = -INFINITY;+ if (j + 1 > limit) s[c].y = -INFINITY;+ blockMax = max(blockMax, max(s[c].x, s[c].y));}blockMax = max(blockMax, simd_shuffle_xor(blockMax, 1));- blockMax = max(blockMax, simd_shuffle_xor(blockMax, 2));+ blockMax = max(blockMax, simd_shuffle_xor(blockMax, 8));float mNew = max(m, blockMax);float alpha = (mNew == -INFINITY) ? 1.0f : exp2(m - mNew);float sum = 0.0f;- for (uint t = 0; t < 8; t++) {- float e = (mNew == -INFINITY) ? 0.0f : exp2(s[t] - mNew);- stage[row * BK + cbase + t] = e;- sum += e;+ for (uint c = 0; c < 8; c++) {+ float2 e = (mNew == -INFINITY) ? float2(0.0f) : exp2(s[c] - mNew);+ elements(S[c]) = e;+ sum += e.x + e.y;}sum += simd_shuffle_xor(sum, 1);- sum += simd_shuffle_xor(sum, 2);+ sum += simd_shuffle_xor(sum, 8);l = l * alpha + sum;m = mNew;- if (lane % 4 == 0) diag[row * 8 + row] = alpha;- const bool rescale = !simd_all(alpha == 1.0f);- threadgroup_barrier(mem_flags::mem_threadgroup);+ if (!simd_all(alpha == 1.0f)) {+ for (uint d = 0; d < 16; d++) elements(O[d]) *= alpha;+ }- simdgroup_float8x8 P[4];- for (uint c = 0; c < 4; c++) simdgroup_load(P[c], stage + 8 * c, BK);- if (rescale) {- simdgroup_float8x8 Dg;- simdgroup_load(Dg, diag, 8);- for (uint d = 0; d < 16; d++) {- simdgroup_float8x8 scaled;- simdgroup_multiply(scaled, Dg, O[d]);- O[d] = scaled;- }+ // Value tile into the same memory, once every SIMD group is done with the keys.+ threadgroup_barrier(mem_flags::mem_threadgroup);+ for (uint i = tid; i < BK * D / 4; i += 256) {+ uint r = i / (D / 4), c = i % (D / 4);+ ((threadgroup float4 *)tile)[i] = *((device const float4 *)(vh + (ulong)(j0 + r) * stride) + c);}+ threadgroup_barrier(mem_flags::mem_threadgroup);for (uint d = 0; d < 16; d++) {- for (uint c = 0; c < 4; c++) {+ for (uint c = 0; c < 8; c++) {simdgroup_float8x8 V;simdgroup_load(V, tile + 8 * c * D + 8 * d, D);- simdgroup_multiply_accumulate(O[d], P[c], V, O[d]);+ simdgroup_multiply_accumulate(O[d], S[c], V, O[d]);}}threadgroup_barrier(mem_flags::mem_threadgroup);}- // O / l per row, then the valid rows to memory.- if (lane % 4 == 0) diag[row * 8 + row] = 1.0f / l;- threadgroup_barrier(mem_flags::mem_threadgroup);- simdgroup_float8x8 Dg;- simdgroup_load(Dg, diag, 8);- for (uint part = 0; part < 4; part++) {- for (uint c = 0; c < 4; c++) {- simdgroup_float8x8 r;- simdgroup_multiply(r, Dg, O[part * 4 + c]);- simdgroup_store(r, stage + 8 * c, BK);- }- threadgroup_barrier(mem_flags::mem_threadgroup);- if (qi < p.queries) {- device float *o = out + (ulong)qi * stride + head * D + part * 32 + cbase;- for (uint t = 0; t < 8; t++) o[t] = stage[row * BK + cbase + t];+ // O / l, the valid rows to memory.+ if (qi < p.queries) {+ const float inv = 1.0f / l;+ device float *o = out + (ulong)qi * stride + head * D + col;+ for (uint d = 0; d < 16; d++) {+ float2 e = elements(O[d]) * inv;+ o[8 * d] = e.x;+ o[8 * d + 1] = e.y;}- threadgroup_barrier(mem_flags::mem_threadgroup);}}diff --git a/engine-mac/Sources/KokonEngine/NeuralEngine.swift b/engine-mac/Sources/KokonEngine/NeuralEngine.swiftindex ae20099..7402970 100644--- a/engine-mac/Sources/KokonEngine/NeuralEngine.swift+++ b/engine-mac/Sources/KokonEngine/NeuralEngine.swift@@ -39,21 +39,9 @@ public final class NeuralEngineProduct: @unchecked Sendable {public init(rows: Int, k: Int, n: Int) throws {self.rows = rows; self.k = k; self.n = n- directory = FileManager.default.temporaryDirectory- .appendingPathComponent("kokon-ane-\(ProcessInfo.processInfo.processIdentifier)-\(rows)x\(k)x\(n)-\(UUID().uuidString)")- .appendingPathExtension("mlmodelc")- try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)- try NeuralEngineProduct.program(rows: rows, k: k, n: n)- .write(to: directory.appendingPathComponent("model.mil"), atomically: true, encoding: .utf8)- try NeuralEngineProduct.description(rows: rows, k: k, n: n)- .write(to: directory.appendingPathComponent("coremldata.bin"))- let configuration = MLModelConfiguration()- configuration.computeUnits = .cpuAndNeuralEngine- do {- model = try MLModel(contentsOf: directory, configuration: configuration)- } catch {- throw Failure.program("\(rows)×\(k)×\(n): \(error.localizedDescription)")- }+ (model, directory) = try NeuralEngineProduct.compile(+ "\(rows)x\(k)x\(n)", program: NeuralEngineProduct.program(rows: rows, k: k, n: n),+ inputs: [("x", [rows, k]), ("w", [k, n])], outputs: [("y", [rows, n])])x = try NeuralEngineProduct.surface(rows: rows, columns: k)w = try NeuralEngineProduct.surface(rows: k, columns: n)y = try NeuralEngineProduct.surface(rows: rows, columns: n)@@ -114,9 +102,29 @@ public final class NeuralEngineProduct: @unchecked Sendable {"""}+ /// Writes `program` and its description as a compiled model in a temporary directory and loads it for the+ /// CPU and the Neural Engine. `tag` names the directory.+ static func compile(_ tag: String, program: String, inputs: [(String, [Int])],+ outputs: [(String, [Int])]) throws -> (MLModel, URL) {+ let directory = FileManager.default.temporaryDirectory+ .appendingPathComponent("kokon-ane-\(ProcessInfo.processInfo.processIdentifier)-\(tag)-\(UUID().uuidString)")+ .appendingPathExtension("mlmodelc")+ try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)+ try program.write(to: directory.appendingPathComponent("model.mil"), atomically: true, encoding: .utf8)+ try description(inputs: inputs, outputs: outputs).write(to: directory.appendingPathComponent("coremldata.bin"))+ let configuration = MLModelConfiguration()+ configuration.computeUnits = .cpuAndNeuralEngine+ do {+ return (try MLModel(contentsOf: directory, configuration: configuration), directory)+ } catch {+ try? FileManager.default.removeItem(at: directory)+ throw Failure.program("\(tag): \(error.localizedDescription)")+ }+ }+/// `coremldata.bin`: a fixed header with the description's length (little-endian u64 at 0x4b), the- /// model description as protobuf (inputs `x`, `w`, output `y`, fp16 multi-arrays), a fixed trailer.- static func description(rows: Int, k: Int, n: Int) -> Data {+ /// model description as protobuf (named fp16 multi-arrays in and out), a fixed trailer.+ static func description(inputs: [(String, [Int])], outputs: [(String, [Int])]) -> Data {func varint(_ value: Int) -> [UInt8] {var v = UInt64(value), out: [UInt8] = []repeat {@@ -133,7 +141,7 @@ public final class NeuralEngineProduct: @unchecked Sendable {let array = field(0x2a, shape + [0x10] + varint(65_552))return field(tag, field(0x0a, Array(name.utf8)) + field(0x1a, array))}- let payload = feature(0x0a, "x", [rows, k]) + feature(0x0a, "w", [k, n]) + feature(0x52, "y", [rows, n])+ let payload: [UInt8] = inputs.flatMap { feature(0x0a, $0.0, $0.1) } + outputs.flatMap { feature(0x52, $0.0, $0.1) }+ [0xa2, 0x06, 0x00]var header = [UInt8](repeating: 0, count: 0x53)header[0] = 0xf6; header[1] = 0x01; header[4] = 0x09; header[0x1c] = 0x07@@ -150,6 +158,10 @@ extension NeuralEngineProduct {/// A GPU buffer over a surface's memory, no copy: the GPU writes the Neural Engine's operands in place./// The surface stays locked for the product's life (unified memory: the lock only pins the address).public func gpuBuffer(_ surface: CVPixelBuffer, device: MTLDevice) throws -> MTLBuffer {+ try NeuralEngineProduct.gpuBuffer(surface, device: device)+ }++ static func gpuBuffer(_ surface: CVPixelBuffer, device: MTLDevice) throws -> MTLBuffer {CVPixelBufferLockBaseAddress(surface, [])guard let io = CVPixelBufferGetIOSurface(surface)?.takeUnretainedValue(),let base = CVPixelBufferGetBaseAddress(surface) else { throw Failure.surface("no IOSurface") }@@ -163,3 +175,72 @@ extension NeuralEngineProduct {return buffer}}++/// **Attention on the Neural Engine**: `y[h] = softmax(q[h] · k[h]ᵀ) · v[h]` for `heads` heads of 128, fp16, `q`+/// already scaled by 1/√128. One program per shape, operands in surfaces `[heads · rows, 128]` (heads first),+/// bound once, as for the product.+public final class NeuralEngineAttention: @unchecked Sendable {+ public let heads: Int, queries: Int, keys: Int+ public static let headDim = 128+ private let model: MLModel+ private let features: MLFeatureProvider+ private let options: MLPredictionOptions+ public let q: CVPixelBuffer, k: CVPixelBuffer, v: CVPixelBuffer, y: CVPixelBuffer+ private let directory: URL++ /// Measured on the M2 Pro (`probe-attention`, 4 096 queries × 4 128 keys, 2026-10-25): 4 heads × 1 024 queries a+ /// run 5.3 ms per head of all queries, 8 × 1 024 5.5, 1 × 4 096 6.5, 8 × 4 096 10.5 (the score matrix grows past+ /// what the Neural Engine keeps close); MIL's fused `scaled_dot_product_attention` 33 ms per head. Head 0 within+ /// 2.2e-3 of an exact attention (fp16 scores and probabilities).+ ///+ /// `k`, `v`: surfaces of another program of the same heads and keys to share (the query chunks of one head group).+ public init(heads: Int, queries: Int, keys: Int, sharing: NeuralEngineAttention? = nil) throws {+ self.heads = heads; self.queries = queries; self.keys = keys+ let d = NeuralEngineAttention.headDim+ (model, directory) = try NeuralEngineProduct.compile(+ "attention-\(heads)x\(queries)x\(keys)", program: NeuralEngineAttention.program(heads: heads, queries: queries, keys: keys),+ inputs: [("q", [1, heads, queries, d]), ("k", [1, heads, keys, d]), ("v", [1, heads, keys, d])],+ outputs: [("y", [1, heads, queries, d])])+ q = try NeuralEngineProduct.surface(rows: heads * queries, columns: d)+ k = try sharing?.k ?? NeuralEngineProduct.surface(rows: heads * keys, columns: d)+ v = try sharing?.v ?? NeuralEngineProduct.surface(rows: heads * keys, columns: d)+ y = try NeuralEngineProduct.surface(rows: heads * queries, columns: d)+ func shape(_ dims: Int...) -> [NSNumber] { dims.map { NSNumber(value: $0) } }+ options = MLPredictionOptions()+ options.outputBackings = ["y": MLMultiArray(pixelBuffer: y, shape: shape(1, heads, queries, d))]+ features = try MLDictionaryFeatureProvider(dictionary: [+ "q": MLFeatureValue(multiArray: MLMultiArray(pixelBuffer: q, shape: shape(1, heads, queries, d))),+ "k": MLFeatureValue(multiArray: MLMultiArray(pixelBuffer: k, shape: shape(1, heads, keys, d))),+ "v": MLFeatureValue(multiArray: MLMultiArray(pixelBuffer: v, shape: shape(1, heads, keys, d))),+ ])+ }++ deinit { try? FileManager.default.removeItem(at: directory) }++ public func run() throws {+ _ = try model.prediction(from: features, options: options)+ }++ /// A GPU buffer over one of its surfaces (`NeuralEngineProduct.gpuBuffer`).+ public func gpuBuffer(_ surface: CVPixelBuffer, device: MTLDevice) throws -> MTLBuffer {+ try NeuralEngineProduct.gpuBuffer(surface, device: device)+ }++ static func program(heads h: Int, queries l: Int, keys s: Int) -> String {+ let d = NeuralEngineAttention.headDim+ return """+ program(1.3)+ [buildInfo = dict<string, string>({{"coremlc-component-MIL", "3600.16.1"}, {"coremlc-version", "3600.25.2"}})]+ {+ func main<ios18>(tensor<fp16, [1, \(h), \(l), \(d)]> q, tensor<fp16, [1, \(h), \(s), \(d)]> k, tensor<fp16, [1, \(h), \(s), \(d)]> v) {+ bool f = const()[name = string("f"), val = bool(false)];+ bool t = const()[name = string("t"), val = bool(true)];+ tensor<fp16, [1, \(h), \(l), \(s)]> scores = matmul(transpose_x = f, transpose_y = t, x = q, y = k)[name = string("scores")];+ int32 axis = const()[name = string("axis"), val = int32(-1)];+ tensor<fp16, [1, \(h), \(l), \(s)]> p = softmax(axis = axis, x = scores)[name = string("p")];+ tensor<fp16, [1, \(h), \(l), \(d)]> y = matmul(transpose_x = f, transpose_y = f, x = p, y = v)[name = string("y")];+ } -> (y);+ }+ """+ }+}diff --git a/engine-mac/Sources/KokonEngine/QwenDiT.swift b/engine-mac/Sources/KokonEngine/QwenDiT.swiftindex 82e309c..d50039c 100644--- a/engine-mac/Sources/KokonEngine/QwenDiT.swift+++ b/engine-mac/Sources/KokonEngine/QwenDiT.swift@@ -44,8 +44,9 @@ public final class QwenDiT {let rope, modulation, qkNorms, textNorm, textIn, textOut, latent, output: Arena.Sliceprivate var nextReserve = 0private var products: [String: MPSMatrixMultiplication] = [:]- /// The AMX and the Neural Engine beside the GPU on the large products (`Coexecution.swift`); nil: GPU alone.- /// `KOKON_COEXEC=0` turns it off, `KOKON_ANE=0` / `KOKON_AMX=0` one unit, `KOKON_AMX_SHARE` the AMX's rows.+ /// The AMX and the Neural Engine beside the GPU on the large products and the target's attention+ /// (`Coexecution.swift`, `AttentionSplit.swift`); nil: GPU alone. `KOKON_COEXEC=0` turns it off, `KOKON_ANE=0` /+ /// `KOKON_AMX=0` one unit, `KOKON_ANE_ATTENTION=1` heads on the Neural Engine too, `KOKON_BALANCE=0` the balancing loop.let coexecutor: Coexecutor?private var attentions: [String: GraphAttention] = [:]/// Seconds per phase of the last evaluation.@@ -74,12 +75,15 @@ public final class QwenDiT {let environment = ProcessInfo.processInfo.environmentif environment["KOKON_COEXEC"] != "0", sequence.count >= 1024 {- coexecutor = try Coexecutor(context: context,- amxShare: Double(environment["KOKON_AMX_SHARE"] ?? "") ?? 0.0625,- ane: environment["KOKON_ANE"] != "0", amx: environment["KOKON_AMX"] != "0")+ coexecutor = try Coexecutor(context: context, rows: QwenDiT.padded(sequence.count),+ ane: environment["KOKON_ANE"] != "0", amx: environment["KOKON_AMX"] != "0",+ aneAttention: environment["KOKON_ANE_ATTENTION"] == "1")} else {coexecutor = nil}+ if let coexecutor, coexecutor.attentionApplies(queries: sequence.target) {+ try coexecutor.prepareAttention(queries: sequence.target, keys: sequence.count)+ }let table = sequence.ropeTable()arena.pointer(rope).update(from: table, count: table.count)// Small weights unpacked once on the CPU: the QK norms of every layer, txt_in's norm as (w + 1).@@ -111,26 +115,38 @@ public final class QwenDiT {static func silu(_ x: [Float]) -> [Float] { x.map { $0 / (1 + expf(-$0)) } }- /// `y = W x` for a Linear weight `[out, in]` of the file, unpacked on the CPU.- func linearCPU(_ name: String, _ x: [Float]) throws -> [Float] {+ /// `y = W x` for each `x`, W a Linear weight `[out, in]` of the file unpacked once on the CPU.+ func linearCPU(_ name: String, _ xs: [[Float]]) throws -> [[Float]] {let t = try file.tensor(QwenDiT.prefix + name)let w = try Dequant.array(file, t.name)- var y = [Float](repeating: 0, count: t.rows)- cblas_sgemv(CblasRowMajor, CblasNoTrans, Int32(t.rows), Int32(t.columns), 1, w, Int32(t.columns), x, 1, 0, &y, 1)- return y+ return xs.map { x in+ var y = [Float](repeating: 0, count: t.rows)+ cblas_sgemv(CblasRowMajor, CblasNoTrans, Int32(t.rows), Int32(t.columns), 1, w, Int32(t.columns), x, 1, 0, &y, 1)+ return y+ }+ }++ /// The stored modulation rows of each timestep seen (`modulationRows`): unpacking the four weights takes ~0.1 s+ /// (`modulation.1` is 268 MB in fp32), so the pipeline prepares all its timesteps at once.+ private var modulationCache: [Float: [Float]] = [:]++ /// Computes and keeps the modulation rows of `timesteps` not yet seen.+ public func prepareModulation(_ timesteps: [Float]) throws {+ let missing = Array(Set(timesteps.filter { modulationCache[$0] == nil }))+ guard !missing.isEmpty else { return }+ for (t, row) in zip(missing, try modulationRows(timesteps: missing).stored) { modulationCache[t] = row }}/// The temb, block modulation and final scale of each timestep, as stored for the kernels:/// `[1 + s₁, tanh g₁, 1 + s₂, tanh g₂, 1 + s_final]`, `dim` floats each.public func modulationRows(timesteps: [Float]) throws -> (temb: [[Float]], raw: [[Float]], stored: [[Float]]) {let d = QwenDiT.dim- var tembs: [[Float]] = [], raws: [[Float]] = [], stored: [[Float]] = []- for t in timesteps {- let h = QwenDiT.silu(try linearCPU("time_text_embed.timestep_embedder.linear_1.weight", QwenDiT.sinusoid(t)))- let temb = try linearCPU("time_text_embed.timestep_embedder.linear_2.weight", h)- let active = QwenDiT.silu(temb)- let m = try linearCPU("modulation.1.weight", active)- let f = try linearCPU("norm_out.linear.weight", active)+ var raws: [[Float]] = [], stored: [[Float]] = []+ let h = try linearCPU("time_text_embed.timestep_embedder.linear_1.weight", timesteps.map(QwenDiT.sinusoid)).map(QwenDiT.silu)+ let tembs = try linearCPU("time_text_embed.timestep_embedder.linear_2.weight", h)+ let active = tembs.map(QwenDiT.silu)+ let ms = try linearCPU("modulation.1.weight", active), fs = try linearCPU("norm_out.linear.weight", active)+ for (m, f) in zip(ms, fs) {var row = [Float](repeating: 0, count: 5 * d)for j in 0..<d {row[j] = 1 + m[j]@@ -139,7 +155,7 @@ public final class QwenDiT {row[3 * d + j] = tanhf(m[3 * d + j])row[4 * d + j] = 1 + f[j]}- tembs.append(temb); raws.append(m); stored.append(row)+ raws.append(m); stored.append(row)}return (tembs, raws, stored)}@@ -156,8 +172,8 @@ public final class QwenDiT {let d = QwenDiT.dim, n = sequence.count, p = sequence.prefix, s = sequence.targetlet started = DispatchTime.now().uptimeNanosecondstimings = [:]- let rows = try modulationRows(timesteps: [0, sigma]).stored- arena.pointer(modulation).update(from: rows[0] + rows[1], count: 10 * d)+ try prepareModulation([0, sigma])+ arena.pointer(modulation).update(from: modulationCache[0]! + modulationCache[sigma]!, count: 10 * d)timings["modulation"] = elapsed(since: started)// Inputs: the text rows gathered in encoder order, the target tokens.@@ -209,6 +225,7 @@ public final class QwenDiT {if let probe { try finish(commands); probe("norm_out", arena.pointer(normed), n * d); commands = try begin() }try linear("proj_out.weight", a: normed, aRow: p, c: output, m: s, commands: commands)try finish(commands)+ if let coexecutor, !profiling, balancing { try coexecutor.rebalance() }let out = Array(UnsafeBufferPointer(start: arena.pointer(output), count: s * QwenDiT.channels))probe?("model_out", arena.pointer(output), s * QwenDiT.channels)timings["total"] = elapsed(since: started)@@ -255,6 +272,9 @@ public final class QwenDiT {// MARK: - encoders+ /// The balancing loop after each evaluation (`Balancer.swift`); `KOKON_BALANCE=0` keeps the starting shares.+ public var balancing = ProcessInfo.processInfo.environment["KOKON_BALANCE"] != "0"+/// When set, every operation runs in a command buffer of its own and its GPU time is added to `timings`/// under its kind (slower overall: the GPU idles between operations).public var profiling = false@@ -333,7 +353,7 @@ public final class QwenDiT {func operand(_ slice: Arena.Slice, _ row: Int, _ width: Int) -> (MTLBuffer, Int, UnsafeMutablePointer<Float>) {(arena.buffer, slice.offset + row * width * 4, arena.pointer(slice, floatOffset: row * width))}- try coexecutor.encode(a: operand(a, aRow, kk), w: operand(reserve, 0, nn), c: operand(c, cRow, nn),+ try coexecutor.encode(weight: name, a: operand(a, aRow, kk), w: operand(reserve, 0, nn), c: operand(c, cRow, nn),m: m, k: kk, n: nn, commands: commands, gpuRows: gpuRows)} else {try gpuRows(m)@@ -419,24 +439,24 @@ public final class QwenDiT {let graphAttention = ProcessInfo.processInfo.environment["KOKON_ATTENTION"] == "graph"struct FlashParams { var queries, keys, causal, stride: UInt32; var scale: Float; var headBase: UInt32 }- /// Queries `[queries, queries + count)` against keys `[0, keys)`, into the same rows of `attended`.+ /// Queries `[queries, queries + count)` against keys `[0, keys)`, into the same rows of `attended`. The target's+ /// queries are cut by heads with the Neural Engine and the CPU when the co-executor runs (`AttentionSplit.swift`).func attend(queries: Int, count: Int, keys: Int, causal: Bool, commands: MTLCommandBuffer) throws {let row = QwenDiT.dim * 4- if !graphAttention {- try compute("flash_attention", commands: commands) { e in- e.setBuffer(arena.buffer, offset: q.offset + queries * row, index: 0)- e.setBuffer(arena.buffer, offset: k.offset, index: 1)- e.setBuffer(arena.buffer, offset: v.offset, index: 2)- e.setBuffer(arena.buffer, offset: attended.offset + queries * row, index: 3)- var params = FlashParams(queries: UInt32(count), keys: UInt32(keys), causal: causal ? 1 : 0,- stride: UInt32(QwenDiT.dim), scale: 1 / Float(QwenDiT.headDim).squareRoot() * Float(M_LOG2E),- headBase: 0)- e.setBytes(¶ms, length: MemoryLayout<FlashParams>.stride, index: 4)- e.dispatchThreadgroups(MTLSize(width: (count + 63) / 64, height: QwenDiT.heads, depth: 1),- threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))+ if let coexecutor, !profiling, !graphAttention, !causal, coexecutor.attentionApplies(queries: count) {+ func operand(_ slice: Arena.Slice, _ first: Int) -> Coexecutor.Operand {+ (arena.buffer, slice.offset + first * row, arena.pointer(slice, floatOffset: first * QwenDiT.dim))+ }+ try coexecutor.encodeAttention(q: operand(q, queries), k: operand(k, 0), v: operand(v, 0), out: operand(attended, queries),+ queries: count, keys: keys, stride: QwenDiT.dim, commands: commands) { gpuHeads in+ try flash(queries: queries, count: count, keys: keys, causal: false, heads: gpuHeads, commands: commands)}return}+ if !graphAttention {+ try flash(queries: queries, count: count, keys: keys, causal: causal, heads: QwenDiT.heads, commands: commands)+ return+ }let key = "\(count)×\(keys)\(causal ? "c" : "")"let attention: GraphAttentionif let a = attentions[key] { attention = a } else {@@ -450,4 +470,22 @@ public final class QwenDiT {out: (b, attended.offset + queries * row))}}++ /// `flash_attention` on heads `[0, heads)`.+ func flash(queries: Int, count: Int, keys: Int, causal: Bool, heads: Int, commands: MTLCommandBuffer) throws {+ guard heads > 0 else { return }+ let row = QwenDiT.dim * 4+ try compute("flash_attention", commands: commands) { e in+ e.setBuffer(arena.buffer, offset: q.offset + queries * row, index: 0)+ e.setBuffer(arena.buffer, offset: k.offset, index: 1)+ e.setBuffer(arena.buffer, offset: v.offset, index: 2)+ e.setBuffer(arena.buffer, offset: attended.offset + queries * row, index: 3)+ var params = FlashParams(queries: UInt32(count), keys: UInt32(keys), causal: causal ? 1 : 0,+ stride: UInt32(QwenDiT.dim), scale: 1 / Float(QwenDiT.headDim).squareRoot() * Float(M_LOG2E),+ headBase: 0)+ e.setBytes(¶ms, length: MemoryLayout<FlashParams>.stride, index: 4)+ e.dispatchThreadgroups(MTLSize(width: (count + 63) / 64, height: heads, depth: 1),+ threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))+ }+ }}diff --git a/engine-mac/Sources/KokonEngine/QwenPipeline.swift b/engine-mac/Sources/KokonEngine/QwenPipeline.swiftindex 3b7ba7c..d7db681 100644--- a/engine-mac/Sources/KokonEngine/QwenPipeline.swift+++ b/engine-mac/Sources/KokonEngine/QwenPipeline.swift@@ -21,6 +21,9 @@ public final class QwenPipeline {public var log: (String) -> Void = { print($0) }/// Seconds per phase of the last run: encode, steps (each), dit load, vae, total.public private(set) var timings: [String: Double] = [:]+ /// Kept between images: its first decode of a size costs ~3.7 s more than the next ones at 1024² (M2 Pro), GPU+ /// time (~1 s of CPU), so it cannot hide under the DiT; a long-lived engine pays it once per size.+ private var vae: QwenVAE?public init(encoder: String, dit: String, vae: String) throws {encoderPath = encoder@@ -35,10 +38,11 @@ public final class QwenPipeline {}/// Text to image at `width` × `height` (multiples of 32). `noise`: a latent `[64, h, w]` to start from instead of- /// the seed's; `step`: called after each step with its index and the packed latent `[h·w, 64]`.+ /// the seed's; `conditioning`: the encoder's output `[T, 4096]` to use instead of encoding `prompt` (the checks);+ /// `step`: called after each step with its index and the packed latent `[h·w, 64]`./// Returns the decoded image, RGBA `[4, height, width]` in [−1, 1].public func textToImage(prompt: String, width: Int, height: Int, seed: UInt64, noise: [Float]? = nil,- sigmas: [Float] = QwenPipeline.turboSigmas,+ conditioning given: [Float]? = nil, sigmas: [Float] = QwenPipeline.turboSigmas,step: ((Int, [Float]) -> Void)? = nil) throws -> [Float] {guard width % 32 == 0, height % 32 == 0, width >= 64, height >= 64 else {throw QwenSequence.Failure.layout("\(width)×\(height): each side must be a multiple of 32, at least 64")@@ -52,6 +56,7 @@ public final class QwenPipeline {// 1. The prompt. The encoder and its 5 GB mapping are gone when this scope ends.var t = DispatchTime.now().uptimeNanosecondslet (conditioning, rows): ([Float], Int) = try {+ if let given { return (given, given.count / QwenDiT.dim) }let file = try GGUFFile(path: encoderPath)let encoder = try QwenTextEncoder(file: file, context: context)defer { encoder.release() }@@ -74,6 +79,7 @@ public final class QwenPipeline {let file = try GGUFFile(path: ditPath)defer { context.release(file) }let dit = try QwenDiT(file: file, context: context, sequence: sequence)+ try dit.prepareModulation([0] + sigmas.dropLast())timings["dit load"] = since(t)for i in 0..<(sigmas.count - 1) {let stepStart = DispatchTime.now().uptimeNanoseconds@@ -88,8 +94,8 @@ public final class QwenPipeline {// 4. The image.t = DispatchTime.now().uptimeNanoseconds- let vae = try QwenVAE(path: vaePath, context: context)- let image = try vae.decode(QwenPipeline.unpack(x, height: h, width: w), height: h, width: w)+ if vae == nil { vae = try QwenVAE(path: vaePath, context: context) }+ let image = try vae!.decode(QwenPipeline.unpack(x, height: h, width: w), height: h, width: w)timings["vae"] = since(t)timings["total"] = since(started)log(String(format: "VAE: %.2f s; total %.1f s", timings["vae"]!, timings["total"]!))diff --git a/engine-mac/Sources/KokonEngine/QwenVAE.swift b/engine-mac/Sources/KokonEngine/QwenVAE.swiftindex 22e97fd..b242a57 100644--- a/engine-mac/Sources/KokonEngine/QwenVAE.swift+++ b/engine-mac/Sources/KokonEngine/QwenVAE.swift@@ -35,7 +35,12 @@ public final class QwenVAE {let context: MetalContextlet file: SafetensorsFile- private var graphs: [String: (graph: MPSGraph, input: MPSGraphTensor, output: MPSGraphTensor)] = [:]+ /// One compiled graph per latent size. Building and compiling the 1024² one takes 0.3 s; its first run then costs+ /// ~3.7 s more than the next ones (M2 Pro), GPU time with ~1 s of CPU, so the pipeline keeps its VAE between+ /// images rather than building it beside the DiT. fp16 convolutions (norms and attention in fp32) were wrong+ /// (2.7e-1: an overflow) and 5.7× slower, MPSGraph trying to move them to the Neural Engine: the VAE stays fp32.+ private var executables: [String: MPSGraphExecutable] = [:]+ private let lock = NSLock()public init(path: String, context: MetalContext) throws {file = try SafetensorsFile(path: path)@@ -48,20 +53,33 @@ public final class QwenVAE {/// Decodes a normalized latent `[64, h, w]` (the DiT's packed tokens transposed) into RGBA `[4, 16h, 16w]` in/// [−1, 1].public func decode(_ latent: [Float], height h: Int, width w: Int) throws -> [Float] {- let key = "\(h)×\(w)"- let built = try graphs[key] ?? build(height: h, width: w)- graphs[key] = built+ let executable = try prepare(height: h, width: w)let input = MPSGraphTensorData(device: MPSGraphDevice(mtlDevice: context.device),data: latent.withUnsafeBytes { Data($0) },shape: [1, 64, NSNumber(value: h), NSNumber(value: w)], dataType: .float32)- let result = built.graph.run(with: context.queue, feeds: [built.input: input], targetTensors: [built.output],- targetOperations: nil)- guard let data = result[built.output] else { throw MetalContext.Failure.kernel("VAE decode") }+ guard let data = executable.run(with: context.queue, inputs: [input], results: nil, executionDescriptor: nil).first else {+ throw MetalContext.Failure.kernel("VAE decode")+ }var out = [Float](repeating: 0, count: 4 * 16 * h * 16 * w)out.withUnsafeMutableBytes { data.mpsndarray().readBytes($0.baseAddress!, strideBytes: nil) }return out}+ /// Builds and compiles the decoder for an `h` × `w` latent, once.+ @discardableResult+ public func prepare(height h: Int, width w: Int) throws -> MPSGraphExecutable {+ try lock.withLock {+ let key = "\(h)×\(w)"+ if let e = executables[key] { return e }+ let built = try build(height: h, width: w)+ let shape = MPSGraphShapedType(shape: [1, 64, NSNumber(value: h), NSNumber(value: w)], dataType: .float32)+ let e = built.graph.compile(with: MPSGraphDevice(mtlDevice: context.device), feeds: [built.input: shape],+ targetTensors: [built.output], targetOperations: nil, compilationDescriptor: nil)+ executables[key] = e+ return e+ }+ }+// MARK: - the graphprivate func build(height h: Int, width w: Int) throws -> (graph: MPSGraph, input: MPSGraphTensor, output: MPSGraphTensor) {diff --git a/engine-mac/Sources/KokonEngine/TextChecks.swift b/engine-mac/Sources/KokonEngine/TextChecks.swiftindex 6bb16f1..d1a6a4e 100644--- a/engine-mac/Sources/KokonEngine/TextChecks.swift+++ b/engine-mac/Sources/KokonEngine/TextChecks.swift@@ -60,7 +60,9 @@ extension Checks {extension Checks {/// Text to image against `tools/reference_generate.py`, from its noise: each step's latent, the decoded image,/// the 8-bit pixels.- public static func generate(encoder: String, dit: String, vae: String, reference directory: String) throws {+ /// `text`: a `reference_text_encoder.py` directory whose conditioning replaces the engine's encoder, to measure+ /// the DiT and the sampler alone.+ public static func generate(encoder: String, dit: String, vae: String, reference directory: String, text: String? = nil) throws {let meta = try JSONSerialization.jsonObject(with: Data(contentsOf: URL(fileURLWithPath: directory).appendingPathComponent("meta.json"))) as! [String: Any]let prompt = meta["prompt"] as! String, grid = meta["grid"] as! [Int]@@ -71,7 +73,8 @@ extension Checks {print(String(format: " %-12@ rel %.2e max |Δ| %.3g%@", name as NSString, r.relative, r.maxAbs, r.finite ? "" : " NON-FINITE"))}let image = try pipeline.textToImage(prompt: prompt, width: w * 16, height: h * 16, seed: 0,- noise: try reference(directory, "noise")) { i, latent in+ noise: try reference(directory, "noise"),+ conditioning: try text.map { try reference($0, "conditioning") }) { i, latent inlatent.withUnsafeBufferPointer { report("latent\(i)", $0.baseAddress!, try! reference(directory, "latent\(i)")) }}image.withUnsafeBufferPointer { report("image", $0.baseAddress!, try! reference(directory, "image")) }@@ -93,6 +96,27 @@ extension Checks {}extension Checks {+ /// Each step of a `reference_generate.py` run on its own: the DiT on the reference's latent before the step (its+ /// noise for the first), with the text reference's conditioning, against the reference's velocity. A trajectory+ /// amplifies rounding from step to step; this measures one evaluation's error on real inputs.+ public static func steps(dit path: String, reference directory: String, text: String) throws {+ let meta = try JSONSerialization.jsonObject(with: Data(contentsOf: URL(fileURLWithPath: directory)+ .appendingPathComponent("meta.json"))) as! [String: Any]+ let grid = meta["grid"] as! [Int], sigmas = (meta["sigmas"] as! [Double]).map(Float.init)+ let conditioning = try reference(text, "conditioning")+ let file = try GGUFFile(path: path)+ let sequence = try QwenSequence.textToImage(text: conditioning.count / QwenDiT.dim,+ target: .init(height: grid[0], width: grid[1]))+ let dit = try QwenDiT(file: file, context: try MetalContext(), sequence: sequence)+ for i in 0..<(sigmas.count - 1) {+ let input = i == 0 ? QwenPipeline.pack(try reference(directory, "noise"), height: grid[0], width: grid[1])+ : try reference(directory, "latent\(i - 1)")+ let velocity = try dit.forward(encoder: conditioning, target: input, sigma: sigmas[i])+ let r = velocity.withUnsafeBufferPointer { compare($0.baseAddress!, try! reference(directory, "velocity\(i)")) }+ print(String(format: " step %d (σ %.4f): velocity rel %.2e max |Δ| %.3g", i + 1, sigmas[i], r.relative, r.maxAbs))+ }+ }+/// The seed's noise against sd.cpp's own generator (`tools/philox_reference.cpp`): every value, bit for bit.public static func noise(dump: String) throws {let data = try Data(contentsOf: URL(fileURLWithPath: dump))diff --git a/engine-mac/Sources/kokon-engine-mac/main.swift b/engine-mac/Sources/kokon-engine-mac/main.swiftindex 0436ea3..6a26109 100644--- a/engine-mac/Sources/kokon-engine-mac/main.swift+++ b/engine-mac/Sources/kokon-engine-mac/main.swift@@ -11,12 +11,16 @@ kokon-engine-mac: Kokon's Apple Silicon engine (in development)check-vae <vae.safetensors> <reference dir> [grid] the VAE decoder against tools/reference_vae.pycheck-tokenizer <encoder.gguf> <fixture.json> the tokenizer against tools/reference_tokenizer.pycheck-text <encoder.gguf> <reference dir> the text encoder against tools/reference_text_encoder.py- check-generate <encoder.gguf> <dit.gguf> <vae.safetensors> <reference dir>+ check-generate <encoder.gguf> <dit.gguf> <vae.safetensors> <reference dir> [text reference dir]text to image against tools/reference_generate.py, from its noise+ (and from the text reference's conditioning when given)+ check-steps <dit.gguf> <generate reference dir> <text reference dir>+ each step alone against tools/reference_generate.py (its latents in)check-noise <philox.bin> the seed's noise against sd.cpp's generator (tools/philox_reference.cpp)generate <encoder.gguf> <dit.gguf> <vae.safetensors> "<prompt>" [--size WxH] [--seed N] [--out file.png]text to image, Turbo's 8 steps (default 1024x1024, seed 42, out.png)probe this Mac's matrix throughput: GPU, AMX, Neural Engine, together+ probe-attention the target's attention at 1024² on the GPU, the Neural Engine, the CPU"""func fail(_ message: String) -> Never {@@ -66,8 +70,9 @@ do {guard arguments.count == 3 else { fail(usage) }try Checks.textEncoder(path: arguments[1], reference: arguments[2])case "check-generate":- guard arguments.count == 5 else { fail(usage) }- try Checks.generate(encoder: arguments[1], dit: arguments[2], vae: arguments[3], reference: arguments[4])+ guard arguments.count == 5 || arguments.count == 6 else { fail(usage) }+ try Checks.generate(encoder: arguments[1], dit: arguments[2], vae: arguments[3], reference: arguments[4],+ text: arguments.count == 6 ? arguments[5] : nil)case "generate":guard arguments.count >= 5 else { fail(usage) }var (width, height, seed, out) = (1024, 1024, UInt64(42), "out.png")@@ -91,11 +96,16 @@ do {try QwenPipeline.writePNG(QwenPipeline.pixels(image, width: size.width, height: size.height),width: size.width, height: size.height, to: out)print("wrote \(out) (\(size.width)×\(size.height), seed \(seed))")+ case "check-steps":+ guard arguments.count == 4 else { fail(usage) }+ try Checks.steps(dit: arguments[1], reference: arguments[2], text: arguments[3])case "check-noise":guard arguments.count == 2 else { fail(usage) }try Checks.noise(dump: arguments[1])case "probe":try Probe.run()+ case "probe-attention":+ try Probe.attention()default:fail(usage)}