Skip to content

armgit / kokon

Public Imported from GitHub
0

Edit photos by describing the change, on your own GPU. A free, open-source alternative to Nano Banana and ChatGPT image editing: photos stay on your computer, no daily limit. One installer for Windows, Mac and Linux sets up the best open model (Qwen-Image 2.1). MCP server for AI agents.

Mac engine: attention cut by heads, balancing loop, faster flash kernel

Gabriele Lozupone237da48c7693
diff --git a/engine-mac/README.md b/engine-mac/README.md
index b496ca6..74433ee 100644
--- a/engine-mac/README.md
+++ b/engine-mac/README.md
@@ -10,6 +10,7 @@ cd engine-mac
swift build -c release
swift test # no model needed
.build/release/kokon-engine-mac probe # this Mac's matrix throughput, ~1 min, no model
+.build/release/kokon-engine-mac probe-attention # the target's attention at 1024² on the GPU, Neural Engine, CPU
.build/release/kokon-engine-mac inspect <file.gguf>
.build/release/kokon-engine-mac check-dequant <file.gguf> [tensor...]
.build/release/kokon-engine-mac generate <encoder.gguf> <dit.gguf> <vae.safetensors> "a red fox in the snow" \
@@ -27,8 +28,11 @@ swift test # no model needed
| `Probe.swift` | GPU (MPS), AMX (Accelerate) and Neural Engine (Core ML) on Qwen's product shapes, alone and together |
| `QwenSequence.swift` | the joint text/image sequence, block-causal segments and RoPE positions |
| `QwenDiT.swift`, `Kernels/dit.metal`, `Attention.swift`, `Arena.swift` | the DiT: modulation on the CPU, everything per token on the GPU |
-| `Coexecution.swift`, `Kernels/ane.metal` | the AMX and the Neural Engine beside the GPU on the DiT's products |
+| `Coexecution.swift`, `Kernels/ane.metal` | the AMX and the Neural Engine beside the GPU on the DiT's products; the timeline of each split operation |
+| `Balancer.swift` | the balancing loop: each evaluation's timeline sets the next one's rows and heads per unit |
| `Kernels/attention.metal` | fused block-causal attention |
+| `AttentionSplit.swift`, `CPUAttention.swift` | the target's attention cut by heads: GPU, CPU (AMX + cores), optionally the Neural Engine |
+| `AttentionProbe.swift` | `probe-attention`: the target's attention on each unit alone |
| `QwenVAE.swift`, `Safetensors.swift` | the VAE decoder as one MPSGraph, from Kokon's VAE file (Wan 2.2 names) |
| `QwenTokenizer.swift` | Qwen's byte-level BPE tokenizer, read from the encoder GGUF's metadata |
| `QwenTextEncoder.swift`, `Kernels/text.metal` | the text encoder: Qwen3-VL-8B's text decoder on the GPU, diffusers' prompt template |
@@ -76,10 +80,21 @@ processor) and saves its noise; `check-generate` starts the engine from that noi
```bash
~/kokon-bench/venv-oracle/bin/python -I tools/reference_generate.py <dit.gguf> <vae.safetensors> ~/kokon-bench/ref/text/fox <repo dir> ~/kokon-bench/ref/generate-256 # ~5 min
-.build/release/kokon-engine-mac check-generate <encoder.gguf> <dit.gguf> <vae.safetensors> ~/kokon-bench/ref/generate-256
+.build/release/kokon-engine-mac check-generate <encoder.gguf> <dit.gguf> <vae.safetensors> ~/kokon-bench/ref/generate-256 [~/kokon-bench/ref/text/fox]
+.build/release/kokon-engine-mac check-steps <dit.gguf> ~/kokon-bench/ref/generate-256 ~/kokon-bench/ref/text/fox
```
-256², 2026-10-25: each step's latent within 6.7e-6 relative, the image within 1.0e-5, 97 of 262 144 8-bit values
+`check-generate` with the text reference starts from its conditioning instead of the engine's encoder;
+`check-steps` runs each step alone on the reference's own latent, so one evaluation's error is not mixed with what
+the trajectory makes of the steps before.
+
+Unsloth's DiT + the texture-fix VAE, 256², fp64 reference (`--dtype float64`), 2026-10-25: each step alone within
+1.4–2.9e-6 of fp64 except step 4 (σ 0.93) at 6.6e-5, an ill-conditioned evaluation (torch's fp32: 2.7e-5 there).
+Along the trajectory the final latent ends within 5.8e-5 (torch fp32 1.2e-5), the image within 7.6e-5, 782 of
+262 144 8-bit values off by 1; with the engine's own text encoder (5e-6 off) the same order. These shift by a few
+times with any change of rounding order (the old flash kernel: 1.9e-5), as torch's fp32 shifts from fp64.
+
+Old uniform Q4_K DiT + original VAE, 256², fp32 reference: each step's latent within 6.7e-6 relative, the image within 1.0e-5, 97 of 262 144 8-bit values
differ by 1. **First 1024² image (M2 Pro, `generate`): 100.7 s in all**, text encoder 1.7 s, 8 steps of 11.4–12.9 s,
VAE 5.4 s with its graph build; peak memory 4.1 GB (Kokon's sd.cpp: ~4.5 min).
@@ -120,12 +135,37 @@ Engine 2048 at 1024²): **11.3 s per evaluation at 1024²**, 8.8 s at 896². The
operands straight into its IOSurfaces and unpacks its output, so the CPU only runs the AMX. Accuracy at 896²
against diffusers fp32: the velocity within 3.6e-4 relative (GPU alone 4.4e-6); running the model in bf16, as
most tools do, deviates by roughly 1e-2.
-**Attention** is a fused (flash) kernel of our own (`Kernels/attention.metal`, fp32, online softmax, block-causal
-mask): 4.5 s per 1024² evaluation against MPSGraph's 4.1 s, but without MPSGraph's score matrix (2.2 GB at 1 MP,
-~34 GB at 4 MP). `KOKON_ATTENTION=graph` keeps MPSGraph for comparison. The other units are idle during it: next.
-
-**VAE decoder** (`check-vae`, against `tools/reference_vae.py`, diffusers fp32): within 1.1e-5; 1024² in 3.1 s
-(plus ~6 s to build its graph the first time).
+**Attention** is a fused (flash) kernel of our own (`Kernels/attention.metal`, fp32, online softmax in registers,
+block-causal mask, 64 keys a tile): 120 ms for the target's 32 heads at 1024² alone (2.3 TFLOP/s; the first version
+145 ms, MPSGraph ~128 ms) and without MPSGraph's score matrix (2.2 GB at 1 MP, ~34 GB at 4 MP).
+`KOKON_ATTENTION=graph` keeps MPSGraph for comparison.
+
+**The target's attention is cut by heads** (`AttentionSplit.swift`) as the products are by rows: GPU heads in the
+flash kernel, CPU heads in `CPUAttention` (AMX for the two products, vvexpf on 8 threads for the softmax, fp32,
+written straight into the GPU's output). `probe-attention`, ms per head alone: GPU 3.8, CPU 6.6, Neural Engine 5.1
+in programs of 4 heads × 1 024 queries (head 0 within 2.2e-3 of exact: fp16). Together at 1024², per layer: GPU
+alone 145 ms; GPU 19 / CPU 13 91 ms at no cost in accuracy; with Neural Engine heads ~1.5 % more, because the GPU's
+heads slow from 4.5 to 6.3–7.0 ms beside it (memory and cache), for 2.5× the velocity error (9.3e-4 against 3.6e-4
+at 896²): Neural Engine heads only with `KOKON_ANE_ATTENTION=1`.
+
+**The balancing loop** (`Balancer.swift`): each evaluation's timeline (when the GPU released each split operation,
+when each unit finished its share) gives each unit's rate per kind of operation beside the others, and the next
+evaluation's shares follow: the AMX's rows per product shape (steps of 64, remembering measured counts: its rate
+is not linear in rows), the Neural Engine's rows (steps of 256, a program set per count), the heads. On the M2 Pro at
+1024² it moves from 1856 / 256 / 2048 rows and 19 / 13 heads to about 1728 / 320–448 / 2048 and 21 / 11 in two
+evaluations; every product's units then finish within ~0.3 ms of each other (before: the GPU 3–15 ms after the
+AMX). `KOKON_BALANCE=0` keeps the starting shares; `KOKON_HEADS=g,a,c` fixes the heads.
+
+**One evaluation at 1024², all three units, unsloth's DiT: 8.8 s** (start of this pass 11.6 s): products ~5.2 s
+(≈11 TFLOP/s across the units, the probe's ceiling), attention 32 × 78 ms, the GPU alone between them ~1 s. The
+Neural Engine's per-column weight scales are kept per weight (computed once), and the modulation of all of the
+pipeline's timesteps is computed once (it was 0.2 s of CPU per evaluation).
+
+**VAE decoder** (`check-vae`, against `tools/reference_vae.py`, diffusers fp32): within 1.1e-5; 1024² in 2.75 s. Its
+graph builds and compiles in 0.3 s, but its first run at a size costs ~3.7 s more, GPU time (~1 s of CPU): it cannot
+hide under the DiT, so the pipeline keeps its VAE between images and a long-lived engine pays it once per size.
+fp16 convolutions (norms and attention kept fp32) were wrong (2.7e-1, an overflow) and 5.7× slower (MPSGraph tries to
+move them to the Neural Engine): the VAE stays fp32.
Credits: the Neural Engine program technique and the GPU + AMX + Neural Engine split come from
[Siliconed](https://github.com/gwenn-ha-dev/Siliconed) (Apache-2.0), written again here.
diff --git a/engine-mac/Sources/KokonEngine/AttentionProbe.swift b/engine-mac/Sources/KokonEngine/AttentionProbe.swift
new file mode 100644
index 0000000..2a58623
--- /dev/null
+++ b/engine-mac/Sources/KokonEngine/AttentionProbe.swift
@@ -0,0 +1,116 @@
+import Accelerate
+import Foundation
+import Metal
+
+/// **Where the target's attention can run** at 1024² (4 096 queries on 4 128 keys, 32 heads of 128): the GPU's flash
+/// kernel, the Neural Engine (one Core ML program per shape) and the CPU (`CPUAttention`), each timed alone, ms per
+/// head, and the Neural Engine and the CPU compared with an exact attention of head 0. q and k uniform in ±3 so the
+/// scores spread over ±12 and the softmax is peaked as a model's is.
+extension Probe {
+ public static func attention() throws {
+ let context = try MetalContext()
+ let (heads, queries, keys, d) = (32, 4096, 4128, 128)
+ let stride = heads * d
+ let scale = 1 / Float(d).squareRoot()
+ var seed: UInt64 = 7
+ func fill(_ p: UnsafeMutablePointer<Float>, _ count: Int, _ range: Float) {
+ for i in 0..<count {
+ seed = seed &* 6364136223846793005 &+ 1442695040888963407
+ p[i] = (Float(seed >> 40) / Float(1 << 24) * 2 - 1) * range
+ }
+ }
+ // q holds the keys' rows too (the target's queries are the last 4 096 rows); 64 rows of slack after each.
+ let rows = keys + 64
+ let qb = try context.scratch(rows * stride * 4, label: "q"), kb = try context.scratch(rows * stride * 4, label: "k")
+ let vb = try context.scratch(rows * stride * 4, label: "v"), ob = try context.scratch(rows * stride * 4, label: "out")
+ func f(_ b: MTLBuffer) -> UnsafeMutablePointer<Float> { b.contents().assumingMemoryBound(to: Float.self) }
+ fill(f(qb), keys * stride, 3); fill(f(kb), keys * stride, 3); fill(f(vb), keys * stride, 1)
+ let offset = keys - queries
+
+ // Exact head 0 (fp64 sums) for the accuracy columns.
+ let checkRows = 64
+ var exact = [Double](repeating: 0, count: checkRows * d)
+ for i in 0..<checkRows {
+ let qi = f(qb) + (offset + i) * stride
+ var s = [Double](repeating: 0, count: keys), peak = -Double.infinity
+ for j in 0..<keys {
+ var dot = 0.0
+ for c in 0..<d { dot += Double(qi[c]) * Double(f(kb)[j * stride + c]) }
+ s[j] = dot * Double(scale); peak = max(peak, s[j])
+ }
+ var sum = 0.0
+ for j in 0..<keys { s[j] = exp(s[j] - peak); sum += s[j] }
+ for j in 0..<keys { for c in 0..<d { exact[i * d + c] += s[j] * Double(f(vb)[j * stride + c]) / sum } }
+ }
+ func error(_ got: (Int, Int) -> Float) -> Double {
+ var diff = 0.0, norm = 0.0
+ for i in 0..<checkRows { for c in 0..<d { let e = Double(got(i, c)) - exact[i * d + c]; diff += e * e; norm += exact[i * d + c] * exact[i * d + c] } }
+ return (diff / norm).squareRoot()
+ }
+ print("target attention at 1024²: \(queries) queries × \(keys) keys, \(heads) heads of \(d); ms per head, best of 3")
+
+ // The GPU's flash kernel, all heads.
+ struct FlashParams { var queries, keys, causal, stride: UInt32; var scale: Float; var headBase: UInt32 }
+ let gpu = try best(3) {
+ let commands = context.queue.makeCommandBuffer()!
+ let e = commands.makeComputeCommandEncoder()!
+ e.setComputePipelineState(try context.pipeline("flash_attention"))
+ e.setBuffer(qb, offset: offset * stride * 4, index: 0); e.setBuffer(kb, offset: 0, index: 1)
+ e.setBuffer(vb, offset: 0, index: 2); e.setBuffer(ob, offset: offset * stride * 4, index: 3)
+ var p = FlashParams(queries: UInt32(queries), keys: UInt32(keys), causal: 0, stride: UInt32(stride),
+ scale: scale * Float(M_LOG2E), headBase: 0)
+ e.setBytes(&p, length: MemoryLayout<FlashParams>.stride, index: 4)
+ e.dispatchThreadgroups(MTLSize(width: (queries + 63) / 64, height: heads, depth: 1),
+ threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
+ e.endEncoding()
+ return try MetalContext.finish(commands, "flash")
+ }
+ let gpuError = error { i, c in f(ob)[(offset + i) * stride + c] }
+ print(String(format: "GPU flash kernel, 32 heads: %.1f ms, %.2f ms/head, %.2f TFLOP/s; head 0 rel %.1e",
+ gpu * 1e3, gpu * 1e3 / Double(heads), 4 * Double(queries) * Double(keys) * Double(d) * Double(heads) / gpu / 1e12, gpuError))
+ if ProcessInfo.processInfo.environment["KOKON_PROBE_GPU_ONLY"] == "1" { return }
+
+ // The Neural Engine.
+ for (h, l) in [(1, 4096), (4, 4096), (8, 4096), (1, 1024), (4, 1024), (8, 1024), (16, 512)] {
+ let name = "Neural Engine \(h) heads × \(l) queries"
+ do {
+ let a = try NeuralEngineAttention(heads: h, queries: l, keys: keys)
+ func write(_ surface: CVPixelBuffer, _ source: UnsafeMutablePointer<Float>, _ first: Int, _ count: Int, _ factor: Float) {
+ CVPixelBufferLockBaseAddress(surface, [])
+ let dst = CVPixelBufferGetBaseAddress(surface)!.assumingMemoryBound(to: Float16.self)
+ for hh in 0..<h { for r in 0..<count { for c in 0..<d {
+ dst[(hh * count + r) * d + c] = Float16(source[(first + r) * stride + hh * d + c] * factor)
+ } } }
+ CVPixelBufferUnlockBaseAddress(surface, [])
+ }
+ write(a.q, f(qb), offset, l, scale); write(a.k, f(kb), 0, keys, 1); write(a.v, f(vb), 0, keys, 1)
+ try a.run()
+ let t = try best(3) { let s = now(); try a.run(); return now() - s }
+ CVPixelBufferLockBaseAddress(a.y, .readOnly)
+ let y = CVPixelBufferGetBaseAddress(a.y)!.assumingMemoryBound(to: Float16.self)
+ let e = error { i, c in Float(y[i * d + c]) }
+ CVPixelBufferUnlockBaseAddress(a.y, .readOnly)
+ let perHead = t * 1e3 / Double(h) * Double(queries) / Double(l)
+ print(String(format: "%-48@ %7.1f ms a run, %.2f ms/head (all queries); head 0 rel %.1e", name as NSString, t * 1e3, perHead, e))
+ } catch {
+ print("\(name): \(error)")
+ }
+ }
+
+ // The CPU.
+ for (threads, single) in [(8, true), (12, true), (8, false), (4, true)] {
+ BLASSetThreading(single ? BLAS_THREADING_SINGLE_THREADED : BLAS_THREADING_MULTI_THREADED)
+ let h = 4
+ let t = best(3) {
+ let s = now()
+ CPUAttention.attend(q: f(qb) + offset * stride, k: f(kb), v: f(vb), out: f(ob) + offset * stride, stride: stride,
+ queries: queries, keys: keys, heads: 0..<h, scale: scale, threads: threads)
+ return now() - s
+ }
+ let e = error { i, c in f(ob)[(offset + i) * stride + c] }
+ print(String(format: "CPU %2d threads, BLAS %@: %.1f ms for %d heads, %.2f ms/head; head 0 rel %.1e",
+ threads, single ? "single-threaded" : "multi-threaded ", t * 1e3, h, t * 1e3 / Double(h), e))
+ }
+ BLASSetThreading(BLAS_THREADING_MULTI_THREADED)
+ }
+}
diff --git a/engine-mac/Sources/KokonEngine/AttentionSplit.swift b/engine-mac/Sources/KokonEngine/AttentionSplit.swift
new file mode 100644
index 0000000..d38b85f
--- /dev/null
+++ b/engine-mac/Sources/KokonEngine/AttentionSplit.swift
@@ -0,0 +1,163 @@
+import Foundation
+import Metal
+
+/// **The target's attention cut by heads** between the GPU, the CPU and (optionally) the Neural Engine, beside one
+/// another as the products are cut by rows (`Coexecution.swift`); the units would otherwise idle through ~40 % of a step.
+///
+/// Measured at 1024² (M2 Pro, `time-dit`, 2026-10-25), ms per layer and velocity against diffusers fp32 at 896²:
+/// GPU alone 145 ms (3.7e-4); GPU 19 / CPU 13 heads 91 ms (3.6e-4: the CPU's heads are fp32); GPU 14 / Neural Engine 8 /
+/// CPU 10 88 ms (9.3e-4); GPU 16 / Neural Engine 16 112 ms (1.6e-3). Beside the Neural Engine the GPU's heads slow
+/// from 4.5 to 6.3–7.0 ms each (they share the memory and its cache), so its heads buy ~1.5 % for 2.5× the error:
+/// off unless `KOKON_ANE_ATTENTION=1`.
+///
+/// heads [0, g) GPU, `flash_attention` with its grid on those heads
+/// heads [g, g + a) Neural Engine, fp16, in programs of 4 heads × 1 024 queries (`NeuralEngineAttention`):
+/// the GPU packs q (scaled), k, v head-first into their surfaces and unpacks y
+/// heads [g + a, 32) CPU, `CPUAttention` on the AMX worker's thread, straight into `attended`
+///
+/// Each head is computed by one unit, so the cut reorders no sum. Only the target's queries are cut: a text segment
+/// is 32 rows, microseconds on the GPU.
+struct HeadSplit: CustomStringConvertible {
+ var gpu: Int, ane: Int, cpu: Int
+ var description: String { "GPU \(gpu) / ANE \(ane) / CPU \(cpu) heads" }
+
+ /// From `time-dit` and `probe-attention` on the M2 Pro (ms per head at 1024² beside the others: GPU 4.5, CPU 7.0;
+ /// the Neural Engine 6.0 and the GPU 6.3 when it runs), the Neural Engine in groups of 4. The balancing loop moves it.
+ static func `default`(ane: Bool, cpu: Bool) -> HeadSplit {
+ switch (ane, cpu) {
+ case (true, true): return HeadSplit(gpu: 14, ane: 8, cpu: 10)
+ case (true, false): return HeadSplit(gpu: 16, ane: 16, cpu: 0)
+ case (false, true): return HeadSplit(gpu: 19, ane: 0, cpu: 13)
+ case (false, false): return HeadSplit(gpu: QwenDiT.heads, ane: 0, cpu: 0)
+ }
+ }
+
+ /// `KOKON_HEADS=gpu,ane,cpu` (the Neural Engine's a multiple of 4, the three summing to 32).
+ static func fromEnvironment() -> HeadSplit? {
+ guard let text = ProcessInfo.processInfo.environment["KOKON_HEADS"] else { return nil }
+ let n = text.split(separator: ",").compactMap { Int($0) }
+ guard n.count == 3, n.allSatisfy({ $0 >= 0 }), n.reduce(0, +) == QwenDiT.heads, n[1] % Coexecutor.aneHeadGroup == 0 else {
+ FileHandle.standardError.write(Data("KOKON_HEADS=\(text) ignored: gpu,ane,cpu summing to 32, ane a multiple of 4\n".utf8))
+ return nil
+ }
+ return HeadSplit(gpu: n[0], ane: n[1], cpu: n[2])
+ }
+}
+
+/// One Neural Engine attention program and GPU views of its surfaces.
+struct AttentionSlot {
+ let program: NeuralEngineAttention
+ let q: MTLBuffer, k: MTLBuffer, v: MTLBuffer, y: MTLBuffer
+}
+
+extension Coexecutor {
+ static let aneHeadGroup = 4, aneQueryChunk = 1024
+ /// The performance cores: `probe-attention` on the M2 Pro, 4 heads: 8 threads 27 ms, 12 the same, 4 threads 35.
+ static let cpuThreads: Int = {
+ var n: Int32 = 0, size = MemoryLayout<Int32>.size
+ return sysctlbyname("hw.perflevel0.physicalcpu", &n, &size, nil, 0) == 0 && n > 0 ? Int(n) : 4
+ }()
+
+ /// Whether the target's attention takes the cut (below 1 024 queries the rendezvous costs more than it brings).
+ func attentionApplies(queries: Int) -> Bool { queries >= 1024 && heads.gpu < QwenDiT.heads }
+
+ /// The Neural Engine's programs for `queries` target rows on `keys` keys: per group of 4 heads, one program per
+ /// chunk of 1 024 queries (the last one shorter, a multiple of 64), sharing the group's k and v surfaces.
+ /// Groups are added or dropped to follow `heads.ane` (a group's programs do not depend on which heads it takes).
+ func prepareAttention(queries: Int, keys: Int) throws {
+ if attentionShape != (queries, keys) { attentionGroups = [] }
+ attentionShape = (queries, keys)
+ let wanted = heads.ane / Coexecutor.aneHeadGroup
+ if attentionGroups.count > wanted { attentionGroups.removeLast(attentionGroups.count - wanted) }
+ let device = context.device
+ while attentionGroups.count < wanted {
+ var slots: [AttentionSlot] = []
+ for start in stride(from: 0, to: queries, by: Coexecutor.aneQueryChunk) {
+ let rows = (min(Coexecutor.aneQueryChunk, queries - start) + 63) / 64 * 64
+ let p = try NeuralEngineAttention(heads: Coexecutor.aneHeadGroup, queries: rows, keys: keys, sharing: slots.first?.program)
+ let first = slots.first
+ slots.append(AttentionSlot(program: p, q: try p.gpuBuffer(p.q, device: device),
+ k: try first?.k ?? p.gpuBuffer(p.k, device: device),
+ v: try first?.v ?? p.gpuBuffer(p.v, device: device),
+ y: try p.gpuBuffer(p.y, device: device)))
+ try p.run() // compile and warm
+ }
+ attentionGroups.append(slots)
+ }
+ }
+
+ struct HeadsShape { var stride, column0, rows, valid, row0, heads: UInt32; var factor: Float }
+ typealias Operand = (buffer: MTLBuffer, offset: Int, pointer: UnsafeMutablePointer<Float>)
+
+ /// Encodes the cut attention of `queries` target rows (`q`, `out` at the first of them) on `keys` keys (`k`, `v`
+ /// at row 0), rows of `stride` floats. `gpuHeads` encodes the GPU's heads `[0, g)`.
+ func encodeAttention(q: Operand, k: Operand, v: Operand, out: Operand, queries: Int, keys: Int, stride: Int,
+ commands: MTLCommandBuffer, gpuHeads: (Int) throws -> Void) throws {
+ try prepareAttention(queries: queries, keys: keys)
+ let split = heads, d = NeuralEngineAttention.headDim, group = Coexecutor.aneHeadGroup
+ let scale = 1 / Float(d).squareRoot()
+ counter += 1
+ let p = counter
+ open(p, kind: "attention", shares: (split.gpu, split.cpu, split.ane))
+ func move(_ kernel: String, _ src: (MTLBuffer, Int), _ dst: (MTLBuffer, Int), _ shape: HeadsShape) throws {
+ var s = shape
+ let e = commands.makeComputeCommandEncoder()!
+ e.label = kernel
+ e.setComputePipelineState(try context.pipeline(kernel))
+ e.setBuffer(src.0, offset: src.1, index: 0); e.setBuffer(dst.0, offset: dst.1, index: 1)
+ e.setBytes(&s, length: MemoryLayout<HeadsShape>.stride, index: 2)
+ e.dispatchThreads(MTLSize(width: d, height: Int(kernel == "ane_heads_pack" ? s.rows : s.valid), depth: Int(s.heads)),
+ threadsPerThreadgroup: MTLSize(width: 128, height: 2, depth: 1))
+ e.endEncoding()
+ }
+ func shape(_ g: Int, rows: Int, valid: Int, row0: Int, factor: Float = 1) -> HeadsShape {
+ HeadsShape(stride: UInt32(stride), column0: UInt32((split.gpu + g * group) * d), rows: UInt32(rows),
+ valid: UInt32(valid), row0: UInt32(row0), heads: UInt32(group), factor: factor)
+ }
+ for (g, slots) in attentionGroups.enumerated() {
+ try move("ane_heads_pack", (k.buffer, k.offset), (slots[0].k, 0), shape(g, rows: keys, valid: keys, row0: 0))
+ try move("ane_heads_pack", (v.buffer, v.offset), (slots[0].v, 0), shape(g, rows: keys, valid: keys, row0: 0))
+ for (c, slot) in slots.enumerated() {
+ let row0 = c * Coexecutor.aneQueryChunk
+ try move("ane_heads_pack", (q.buffer, q.offset), (slot.q, 0),
+ shape(g, rows: slot.program.queries, valid: min(slot.program.queries, queries - row0), row0: row0, factor: scale))
+ }
+ }
+ if split.ane > 0 || split.cpu > 0 { commands.encodeSignalEvent(ready, value: p) }
+ if split.cpu > 0 {
+ nonisolated(unsafe) let (pq, pk, pv, po) = (q.pointer, k.pointer, v.pointer, out.pointer)
+ let range = (split.gpu + split.ane)..<QwenDiT.heads
+ amxWorker.submit { [self] in
+ defer { amxDone.signaledValue = p }
+ guard wait(ready, p, "CPU attention") else { return }
+ let t = DispatchTime.now().uptimeNanoseconds
+ CPUAttention.attend(q: pq, k: pk, v: pv, out: po, stride: stride, queries: queries, keys: keys,
+ heads: range, scale: scale, threads: Coexecutor.cpuThreads)
+ finished(p, \.amx, since: t)
+ }
+ }
+ if split.ane > 0 {
+ let programs = attentionGroups.flatMap { $0.map(\.program) }
+ aneWorker.submit { [self] in
+ defer { aneDone.signaledValue = p }
+ guard wait(ready, p, "Neural Engine attention") else { return }
+ let t = DispatchTime.now().uptimeNanoseconds
+ do { for program in programs { try program.run() } } catch { fail("Neural Engine attention: \(error)") }
+ finished(p, \.ane, since: t)
+ }
+ }
+ try gpuHeads(split.gpu)
+ gpuFinished(p, commands)
+ if split.cpu > 0 { commands.encodeWaitForEvent(amxDone, value: p) }
+ if split.ane > 0 {
+ commands.encodeWaitForEvent(aneDone, value: p)
+ for (g, slots) in attentionGroups.enumerated() {
+ for (c, slot) in slots.enumerated() {
+ let row0 = c * Coexecutor.aneQueryChunk
+ try move("ane_heads_unpack", (slot.y, 0), (out.buffer, out.offset),
+ shape(g, rows: slot.program.queries, valid: min(slot.program.queries, queries - row0), row0: row0))
+ }
+ }
+ }
+ }
+}
diff --git a/engine-mac/Sources/KokonEngine/Balancer.swift b/engine-mac/Sources/KokonEngine/Balancer.swift
new file mode 100644
index 0000000..362be3f
--- /dev/null
+++ b/engine-mac/Sources/KokonEngine/Balancer.swift
@@ -0,0 +1,124 @@
+import Foundation
+
+/// **The balancing loop.** After each evaluation the timeline gives each unit's rate on each kind of operation,
+/// measured beside the others (the AMX runs at about half its solo rate there), and the next evaluation's shares are
+/// set so the units finish together:
+///
+/// - products: the Neural Engine's rows are one count for every product (its programs are compiled per count, in
+/// steps of 256), chosen to minimize the predicted time of all products; the AMX's rows per kind of product (steps
+/// of 64) split the rest with the GPU in proportion to their rates;
+/// - the target's attention: every (Neural Engine in groups of 4, CPU, GPU) cut of the 32 heads, the one whose
+/// slowest unit is fastest.
+///
+/// A new cut is taken only when it is predicted 1.5 % faster, so the shares settle instead of flickering. Rates are
+/// smoothed over evaluations and start from the M2 Pro's, so a unit without a share keeps an estimate. Moving a
+/// share moves which rows go through the Neural Engine's fp16: images stay within its accuracy, not bit for bit.
+struct Balancer {
+ /// Rows (heads, for attention) per millisecond.
+ struct Rates { var gpu: Double, amx: Double, ane: Double }
+ var rates: [String: Rates]
+ /// The AMX's measured rate at each row count tried, per kind: it is not linear in rows (MLP out on the M2 Pro:
+ /// 320 rows 31 ms, 384 rows 43 ms), so a measured count beats the extrapolated rate.
+ var amxAt: [String: [Int: Double]] = [:]
+ let aneAttention: Bool
+ static let smoothing = 0.5, margin = 0.985
+
+ /// Seeds from `time-dit` at 1024² (1856 / 256 / 2048 rows; attention 19 GPU / 13 CPU heads) and `probe-attention`
+ /// (the Neural Engine's heads), M2 Pro, 2026-10-25.
+ init(aneAttention: Bool) {
+ self.aneAttention = aneAttention
+ rates = ["4096×4096": Rates(gpu: 149, amx: 28, ane: 172), "4096×12288": Rates(gpu: 48, amx: 9.8, ane: 58),
+ "12288×4096": Rates(gpu: 43.5, amx: 8.8, ane: 57.5), "attention": Rates(gpu: 0.224, amx: 0.142, ane: 0.195)]
+ }
+
+ /// Folds an evaluation's timeline into the rates: per kind, each unit's rows over its seconds.
+ mutating func observe(_ entries: [Timeline.Entry]) {
+ var sums: [String: (rows: Rates, ms: Rates)] = [:]
+ for e in entries {
+ var s = sums[e.kind] ?? (Rates(gpu: 0, amx: 0, ane: 0), Rates(gpu: 0, amx: 0, ane: 0))
+ s.rows.gpu += Double(e.shares.gpu); s.ms.gpu += (e.gpu - e.released) * 1e3
+ s.rows.amx += Double(e.shares.amx); s.ms.amx += e.amx.seconds * 1e3
+ s.rows.ane += Double(e.shares.ane); s.ms.ane += e.ane.seconds * 1e3
+ sums[e.kind] = s
+ }
+ for (kind, s) in sums {
+ var r = rates[kind] ?? Rates(gpu: s.rows.gpu / max(s.ms.gpu, 1e-3), amx: 0, ane: 0)
+ func fold(_ old: Double, _ rows: Double, _ ms: Double) -> Double {
+ rows > 0 && ms > 0 ? Balancer.smoothing * old + (1 - Balancer.smoothing) * rows / ms : old
+ }
+ r.gpu = fold(r.gpu, s.rows.gpu, s.ms.gpu)
+ r.amx = fold(r.amx, s.rows.amx, s.ms.amx)
+ r.ane = fold(r.ane, s.rows.ane, s.ms.ane)
+ rates[kind] = r
+ if kind != "attention", s.rows.amx > 0, s.ms.amx > 0, let first = entries.first(where: { $0.kind == kind }) {
+ amxAt[kind, default: [:]][first.shares.amx] = s.rows.amx / s.ms.amx
+ }
+ }
+ }
+
+ /// Predicted milliseconds of `counts` products of each kind with `ane` rows of `rows` on the Neural Engine and the
+ /// rest cut ideally between the GPU and (when `amx`) the AMX.
+ func productCost(rows: Int, ane: Int, amx: Bool, counts: [String: Int]) -> Double {
+ counts.reduce(0) { sum, kc in
+ guard let r = rates[kc.key] else { return sum }
+ let rest = Double(rows - ane) / (r.gpu + (amx ? r.amx : 0))
+ return sum + Double(kc.value) * max(ane > 0 ? Double(ane) / r.ane : 0, rest)
+ }
+ }
+
+ /// Predicted milliseconds of one cut attention.
+ func attentionCost(_ h: HeadSplit) -> Double {
+ guard let r = rates["attention"] else { return .infinity }
+ return max(Double(h.gpu) / r.gpu, h.ane > 0 ? Double(h.ane) / r.ane : 0, h.cpu > 0 ? Double(h.cpu) / r.amx : 0)
+ }
+}
+
+extension Coexecutor {
+ /// Reads the last evaluation's timeline and sets the next one's shares. Between evaluations only.
+ func rebalance() throws {
+ lastTimeline = timeline.drain()
+ balancer.observe(lastTimeline)
+
+ var counts: [String: Int] = [:]
+ for e in lastTimeline where e.kind != "attention" { counts[e.kind, default: 0] += 1 }
+ if !counts.isEmpty {
+ if aneEnabled {
+ func cost(_ e: Int) -> Double { balancer.productCost(rows: rows, ane: e, amx: amxEnabled, counts: counts) }
+ let best = stride(from: 1024, through: rows - 512, by: 256).min { cost($0) < cost($1) } ?? aneRows
+ if cost(best) < cost(aneRows) * Balancer.margin { try setAneRows(best) }
+ }
+ for kind in counts.keys {
+ guard amxEnabled, let r = balancer.rates[kind] else { continue }
+ // Of the two multiples of 64 around the ideal cut, the one whose slower unit is faster.
+ let rest = rows - aneRows
+ let ideal = Double(rest) * r.amx / (r.gpu + r.amx)
+ let low = Int(ideal / 64) * 64
+ func time(_ a: Int) -> Double {
+ max(a > 0 ? Double(a) / (balancer.amxAt[kind]?[a] ?? r.amx) : 0, Double(rest - a) / r.gpu)
+ }
+ amxRows[kind] = [low - 64, low, low + 64].filter { $0 >= 0 }.min { time($0) < time($1) }!
+ }
+ }
+
+ if lastTimeline.contains(where: { $0.kind == "attention" }), HeadSplit.fromEnvironment() == nil {
+ let maxAne = aneEnabled && balancer.aneAttention ? QwenDiT.heads : 0
+ var best = heads
+ for ane in stride(from: 0, through: maxAne, by: Coexecutor.aneHeadGroup) {
+ for cpu in 0...(amxEnabled ? QwenDiT.heads - 1 - ane : 0) where ane + cpu < QwenDiT.heads {
+ let h = HeadSplit(gpu: QwenDiT.heads - ane - cpu, ane: ane, cpu: cpu)
+ if balancer.attentionCost(h) < balancer.attentionCost(best) { best = h }
+ }
+ }
+ if balancer.attentionCost(best) < balancer.attentionCost(heads) * Balancer.margin {
+ heads = best
+ try prepareAttention(queries: attentionShape.queries, keys: attentionShape.keys)
+ }
+ }
+ }
+
+ /// The shares the next evaluation takes.
+ var shares: String {
+ let amx = amxRows.sorted { $0.key < $1.key }.map { "\($0.key) \($0.value)" }.joined(separator: ", ")
+ return "products ANE \(aneRows) rows, AMX \(amx.isEmpty ? "default" : amx); attention \(heads)"
+ }
+}
diff --git a/engine-mac/Sources/KokonEngine/CPUAttention.swift b/engine-mac/Sources/KokonEngine/CPUAttention.swift
new file mode 100644
index 0000000..5bbc8cd
--- /dev/null
+++ b/engine-mac/Sources/KokonEngine/CPUAttention.swift
@@ -0,0 +1,73 @@
+import Accelerate
+import Foundation
+
+/// **Attention of some heads on the CPU** (the AMX for the two products, the cores for the softmax), fp32, on the
+/// layout the products write: rows of `stride` floats, head h at columns `[128 h, 128 h + 128)`. No mask: the
+/// target's queries see every key.
+///
+/// Work items are (head, block of `blockRows` queries), cut into one contiguous run per thread, so the bits do not
+/// depend on which thread computed a block. Per item:
+///
+/// S = scale · Q_b · K_hᵀ cblas_sgemm [blockRows, keys]
+/// P = exp(S − max S), l = Σ P vDSP, vvexpf per row
+/// O_b = (P · V_h) / l cblas_sgemm, then the rows scaled into `out`
+///
+/// The operands are packed first (Q_b, and K_h, V_h once per head of a run): read in place a head's rows are 16 KB
+/// apart, a power of two, and land in the same cache sets (Siliconed measured a head 1.8× slower so).
+enum CPUAttention {
+ static let blockRows = 128, headDim = 128
+
+ /// Heads `heads` of `queries` query rows against `keys` key rows; `q` points at the first query row.
+ static func attend(q: UnsafePointer<Float>, k: UnsafePointer<Float>, v: UnsafePointer<Float>,
+ out: UnsafeMutablePointer<Float>, stride: Int, queries: Int, keys: Int, heads: Range<Int>,
+ scale: Float, threads: Int) {
+ let blocks = (queries + blockRows - 1) / blockRows
+ let items = heads.count * blocks
+ guard items > 0 else { return }
+ let threads = max(1, min(threads, items))
+ nonisolated(unsafe) let (q, k, v, out) = (q, k, v, out)
+ DispatchQueue.concurrentPerform(iterations: threads) { t in
+ let first = items * t / threads, end = items * (t + 1) / threads
+ let d = headDim
+ let kh = UnsafeMutablePointer<Float>.allocate(capacity: keys * d)
+ let vh = UnsafeMutablePointer<Float>.allocate(capacity: keys * d)
+ let qb = UnsafeMutablePointer<Float>.allocate(capacity: blockRows * d)
+ let s = UnsafeMutablePointer<Float>.allocate(capacity: blockRows * keys)
+ let ob = UnsafeMutablePointer<Float>.allocate(capacity: blockRows * d)
+ defer { kh.deallocate(); vh.deallocate(); qb.deallocate(); s.deallocate(); ob.deallocate() }
+ var packed = -1
+ for item in first..<end {
+ let head = heads.lowerBound + item / blocks, block = item % blocks
+ let column = head * d
+ if head != packed {
+ for r in 0..<keys {
+ (kh + r * d).update(from: k + r * stride + column, count: d)
+ (vh + r * d).update(from: v + r * stride + column, count: d)
+ }
+ packed = head
+ }
+ let row0 = block * blockRows, rows = min(blockRows, queries - row0)
+ for r in 0..<rows { (qb + r * d).update(from: q + (row0 + r) * stride + column, count: d) }
+ cblas_sgemm(CblasRowMajor, CblasNoTrans, CblasTrans, Int32(rows), Int32(keys), Int32(d), scale,
+ qb, Int32(d), kh, Int32(d), 0, s, Int32(keys))
+ var sums = [Float](repeating: 0, count: rows)
+ var n = Int32(keys)
+ for r in 0..<rows {
+ let row = s + r * keys
+ var peak: Float = 0
+ vDSP_maxv(row, 1, &peak, vDSP_Length(keys))
+ var shift = -peak
+ vDSP_vsadd(row, 1, &shift, row, 1, vDSP_Length(keys))
+ vvexpf(row, row, &n)
+ vDSP_sve(row, 1, &sums[r], vDSP_Length(keys))
+ }
+ cblas_sgemm(CblasRowMajor, CblasNoTrans, CblasNoTrans, Int32(rows), Int32(d), Int32(keys), 1,
+ s, Int32(keys), vh, Int32(d), 0, ob, Int32(d))
+ for r in 0..<rows {
+ var inverse = 1 / sums[r]
+ vDSP_vsmul(ob + r * d, 1, &inverse, out + (row0 + r) * stride + column, 1, vDSP_Length(d))
+ }
+ }
+ }
+ }
+}
diff --git a/engine-mac/Sources/KokonEngine/Checks.swift b/engine-mac/Sources/KokonEngine/Checks.swift
index 2677bbd..7154aae 100644
--- a/engine-mac/Sources/KokonEngine/Checks.swift
+++ b/engine-mac/Sources/KokonEngine/Checks.swift
@@ -118,15 +118,17 @@ extension Checks {
}
let encoder = random(text * QwenDiT.dim, 8), latent = random(sequence.target * QwenDiT.channels, 2)
print("sequence: \(sequence.count) rows (\(sequence.prefix) text, \(sequence.target) target)"
- + (dit.coexecutor.map { ", products split \($0.split(m: QwenDiT.padded(sequence.count)))" } ?? ", GPU only"))
+ + (dit.coexecutor.map { ", products split \($0.split(m: QwenDiT.padded(sequence.count), k: 4096, n: 4096)), attention \($0.heads)" } ?? ", GPU only"))
for run in 0..<runs {
let started = DispatchTime.now().uptimeNanoseconds
let out = try dit.forward(encoder: encoder, target: latent, sigma: 0.5)
let seconds = Double(DispatchTime.now().uptimeNanoseconds - started) * 1e-9
print(String(format: "evaluation %d: %.2f s%@", run + 1, seconds, out.allSatisfy(\.isFinite) ? "" : " NON-FINITE OUTPUT"))
if let c = dit.coexecutor {
- print(String(format: " AMX busy %.2f s, Neural Engine busy %.2f s%@", c.amxSeconds, c.aneSeconds,
+ print(String(format: " AMX/CPU busy %.2f s, Neural Engine busy %.2f s%@", c.amxSeconds, c.aneSeconds,
c.failures.isEmpty ? "" : "; FAILURES: \(c.failures.prefix(3))"))
+ print(Timeline.summary(dit.balancing ? c.lastTimeline : c.timeline.drain()))
+ if dit.balancing { print(" next: \(c.shares)") }
c.resetTimings()
}
}
@@ -196,12 +198,22 @@ extension Checks {
print(String(format: "VAE %d² → %d²: rel %.2e, max |Δ| %.3g%@ (%.2f s with the graph build)", grid, grid * 16,
r.relative, r.maxAbs, r.finite ? "" : " NON-FINITE", first))
if let t = timeGrid {
+ started = DispatchTime.now().uptimeNanoseconds
+ try vae.prepare(height: t, width: t)
+ print(String(format: "build and compile %d²: %.2f s", t * 16, Double(DispatchTime.now().uptimeNanoseconds - started) * 1e-9))
let random = (0..<(64 * t * t)).map { Float(($0 * 7919) % 1000) / 500 - 1 }
+ func cpu() -> (user: Double, system: Double) {
+ var u = rusage()
+ getrusage(RUSAGE_SELF, &u)
+ return (Double(u.ru_utime.tv_sec) + Double(u.ru_utime.tv_usec) * 1e-6, Double(u.ru_stime.tv_sec) + Double(u.ru_stime.tv_usec) * 1e-6)
+ }
for run in 0..<2 {
started = DispatchTime.now().uptimeNanoseconds
+ let before = cpu()
_ = try vae.decode(random, height: t, width: t)
- print(String(format: "decode %d² (run %d): %.2f s", t * 16, run + 1,
- Double(DispatchTime.now().uptimeNanoseconds - started) * 1e-9))
+ let after = cpu()
+ print(String(format: "decode %d² (run %d): %.2f s (CPU user %.2f s, system %.2f s)", t * 16, run + 1,
+ Double(DispatchTime.now().uptimeNanoseconds - started) * 1e-9, after.user - before.user, after.system - before.system))
}
}
}
diff --git a/engine-mac/Sources/KokonEngine/Coexecution.swift b/engine-mac/Sources/KokonEngine/Coexecution.swift
index 270faa6..fa3610d 100644
--- a/engine-mac/Sources/KokonEngine/Coexecution.swift
+++ b/engine-mac/Sources/KokonEngine/Coexecution.swift
@@ -22,40 +22,81 @@ final class Coexecutor {
static let tile = 4096
let context: MetalContext
- let aneRows: Int
- private let slots: [(product: NeuralEngineProduct, x: MTLBuffer, w: MTLBuffer, y: MTLBuffer)]
- private let ready: MTLSharedEvent, amxDone: MTLSharedEvent, aneDone: MTLSharedEvent
- private var counter: UInt64 = 0
- private let amxWorker = Worker(name: "kokon.amx"), aneWorker = Worker(name: "kokon.ane")
- /// λ (12 288), lifted norms (12 288), L, σ (aneRows): the Neural Engine's scales.
+ /// The rows of every split product (the DiT's padded sequence).
+ let rows: Int
+ /// The Neural Engine's rows, the same for every product (its programs are compiled per row count), and the
+ /// AMX's per kind of product (`"k×n"`): set by the balancing loop (`Balancer.swift`).
+ private(set) var aneRows: Int
+ var amxRows: [String: Int] = [:]
+ typealias ProductSlot = (product: NeuralEngineProduct, x: MTLBuffer, w: MTLBuffer, y: MTLBuffer)
+ private var slots: [ProductSlot]
+ /// Programs of other row counts, kept while the balancing loop may come back to them.
+ private var spareSlots: [Int: [ProductSlot]] = [:]
+ var balancer: Balancer
+ /// The timeline of the last evaluation, once the balancing loop has read it.
+ var lastTimeline: [Timeline.Entry] = []
+ let ready: MTLSharedEvent, amxDone: MTLSharedEvent, aneDone: MTLSharedEvent
+ /// Signaled after the GPU's own share of each operation, for the timeline only.
+ let gpuDone: MTLSharedEvent
+ let listener = MTLSharedEventListener(dispatchQueue: DispatchQueue(label: "kokon.timeline"))
+ let timeline = Timeline()
+ var counter: UInt64 = 0
+ let amxWorker = Worker(name: "kokon.amx"), aneWorker = Worker(name: "kokon.ane")
+ /// σ of the Neural Engine's rows (`rows` at most).
private let scales: MTLBuffer
- /// The share of rows each unit takes; `KOKON_SPLIT=gpu,amx,ane` (fractions) overrides.
- var amxShare: Double, aneEnabled: Bool, amxEnabled: Bool
+ /// Per weight, its columns' λ, their lifted norms and L (`[λ (n) | norms (n) | L]`): they depend on the weight
+ /// alone, so they are computed the first time it is used and kept (~11 MB for the DiT). Two full reads of every
+ /// weight per evaluation, by only n threads, were most of the GPU's time alone between products.
+ private var columnScales: [String: MTLBuffer] = [:]
+ let aneEnabled: Bool, amxEnabled: Bool
/// Seconds each worker spent computing (not waiting), and failures (a failure leaves rows unwritten).
- private(set) var amxSeconds = 0.0, aneSeconds = 0.0
- private(set) var failures: [String] = []
- private let lock = NSLock()
+ var amxSeconds = 0.0, aneSeconds = 0.0
+ var failures: [String] = []
+ let lock = NSLock()
+ /// The target's attention cut by heads (`AttentionSplit.swift`), and the Neural Engine's programs for it.
+ var heads: HeadSplit
+ var attentionGroups: [[AttentionSlot]] = []
+ var attentionShape = (queries: 0, keys: 0)
- /// Defaults measured on the M2 Pro 16 GB, 1024² + 32 text rows (`time-dit`, 2026-10-25): AMX 512 rows 12.8 s per
- /// evaluation (the AMX the slowest, ~35 % under its solo rate beside the others), 256 rows 11.3 s, none 11.8 s;
- /// GPU alone 16.2 s. A balancing loop per shape replaces the fixed share later.
- init(context: MetalContext, aneRows: Int = 2048, amxShare: Double = 0.0625, ane: Bool = true, amx: Bool = true) throws {
+ /// Starting shares measured on the M2 Pro 16 GB, 1024² + 32 text rows (`time-dit`, 2026-10-25): 2 048 rows on the
+ /// Neural Engine and 1/16 on the AMX (AMX 512 rows 12.8 s per evaluation, 256 rows 11.3 s, none 11.8 s; GPU alone
+ /// 16.2 s). The balancing loop moves them from the first evaluation on.
+ init(context: MetalContext, rows: Int, ane: Bool = true, amx: Bool = true, aneAttention: Bool = false) throws {
self.context = context
- self.aneRows = aneRows
- self.amxShare = amxShare
+ self.rows = rows
self.aneEnabled = ane
self.amxEnabled = amx
+ aneRows = ane ? min(2048, max(0, (rows - 512) / 256 * 256)) : 0
+ heads = HeadSplit.fromEnvironment() ?? HeadSplit.default(ane: ane && aneAttention, cpu: amx)
+ balancer = Balancer(aneAttention: ane && aneAttention)
let device = context.device
- guard let r = device.makeSharedEvent(), let a = device.makeSharedEvent(), let e = device.makeSharedEvent() else {
+ guard let r = device.makeSharedEvent(), let a = device.makeSharedEvent(), let e = device.makeSharedEvent(),
+ let g = device.makeSharedEvent() else {
throw MetalContext.Failure.buffer("shared events")
}
- (ready, amxDone, aneDone) = (r, a, e)
- scales = try context.scratch((2 * 12288 + 16 + aneRows) * 4, label: "ane scales")
- slots = ane ? try (0..<3).map { _ in
- let p = try NeuralEngineProduct(rows: aneRows, k: Coexecutor.tile, n: Coexecutor.tile)
- try p.run() // compile and warm
+ (ready, amxDone, aneDone, gpuDone) = (r, a, e, g)
+ scales = try context.scratch(rows * 4, label: "ane row scales")
+ slots = []
+ slots = try productSlots(aneRows)
+ }
+
+ /// Three programs of `rows` rows (the tiles of a product, at most 3), compiled and warmed.
+ private func productSlots(_ rows: Int) throws -> [ProductSlot] {
+ guard rows > 0 else { return [] }
+ let device = context.device
+ return try (0..<3).map { _ in
+ let p = try NeuralEngineProduct(rows: rows, k: Coexecutor.tile, n: Coexecutor.tile)
+ try p.run()
return (p, try p.gpuBuffer(p.x, device: device), try p.gpuBuffer(p.w, device: device), try p.gpuBuffer(p.y, device: device))
- } : []
+ }
+ }
+
+ /// Moves the Neural Engine to `rows` rows per product; between evaluations only (the GPU must be idle).
+ func setAneRows(_ rows: Int) throws {
+ guard rows != aneRows else { return }
+ let next = try spareSlots.removeValue(forKey: rows) ?? productSlots(rows)
+ if aneRows > 0 { spareSlots = [aneRows: slots] } // the last one only: 192 MB of surfaces at 2 048 rows
+ (slots, aneRows) = (next, rows)
}
/// Whether a product takes the split: the Neural Engine's tiles need K and N in multiples of 4 096 with at
@@ -64,24 +105,26 @@ final class Coexecutor {
m >= 1024 && k % Coexecutor.tile == 0 && n % Coexecutor.tile == 0 && (k / Coexecutor.tile) * (n / Coexecutor.tile) <= 3
}
- func split(m: Int) -> Split {
- let ane = aneEnabled && m >= aneRows + 512 ? aneRows : 0
- let amx = amxEnabled ? Int((Double(m) * amxShare / 256).rounded()) * 256 : 0
+ func split(m: Int, k: Int, n: Int) -> Split {
+ let ane = aneEnabled && m == rows ? aneRows : 0
+ let amx = amxEnabled ? amxRows["\(k)×\(n)"] ?? Int((Double(m) / 16 / 64).rounded()) * 64 : 0
return Split(gpu: m - ane - amx, amx: amx, ane: ane)
}
struct Shape { var k, n, k0, kPart, n0, nPart, row0, rows, kParts: UInt32 }
- /// Encodes the split product. `gpuRows` encodes the GPU's own rows `[0, g)` into the command buffer.
- func encode(a: (buffer: MTLBuffer, offset: Int, pointer: UnsafeMutablePointer<Float>),
+ /// Encodes the split product of the weight named `weight`. `gpuRows` encodes the GPU's own rows `[0, g)` into the
+ /// command buffer.
+ func encode(weight: String, a: (buffer: MTLBuffer, offset: Int, pointer: UnsafeMutablePointer<Float>),
w: (buffer: MTLBuffer, offset: Int, pointer: UnsafeMutablePointer<Float>),
c: (buffer: MTLBuffer, offset: Int, pointer: UnsafeMutablePointer<Float>),
m: Int, k: Int, n: Int, commands: MTLCommandBuffer, gpuRows: (Int) throws -> Void) throws {
- let s = split(m: m)
+ let s = split(m: m, k: k, n: n)
counter += 1
let p = counter
+ open(p, kind: "\(k)×\(n)", shares: (s.gpu, s.amx, s.ane))
let (kParts, nParts) = (k / Coexecutor.tile, n / Coexecutor.tile)
- let lambda = 0, norm = 12288 * 4, state = 2 * 12288 * 4, sigma = 2 * 12288 * 4 + 64
+ let (lambda, norm, state, sigma) = (0, n * 4, 2 * n * 4, 0)
func shape(kp: Int = 0, np: Int = 0) -> Shape {
Shape(k: UInt32(k), n: UInt32(n), k0: UInt32(kp * Coexecutor.tile), kPart: UInt32(Coexecutor.tile),
n0: UInt32(np * Coexecutor.tile), nPart: UInt32(Coexecutor.tile), row0: UInt32(s.gpu + s.amx),
@@ -94,21 +137,26 @@ final class Coexecutor {
try body(e)
e.endEncoding()
}
+ // (Read only when the Neural Engine takes rows.)
+ let columns = s.ane > 0 ? try columnScales[weight] ?? context.scratch((2 * n + 16) * 4, label: "λ \(weight)") : scales
if s.ane > 0 {
var sh = shape()
- try kernel("ane_columns") { e in
- e.setBuffer(w.buffer, offset: w.offset, index: 0)
- e.setBuffer(scales, offset: lambda, index: 1); e.setBuffer(scales, offset: norm, index: 2)
- e.setBytes(&sh, length: MemoryLayout<Shape>.stride, index: 3)
- e.dispatchThreads(MTLSize(width: n, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 64, height: 1, depth: 1))
- }
- try kernel("ane_norm") { e in
- e.setBuffer(scales, offset: norm, index: 0); e.setBuffer(scales, offset: state, index: 1)
- e.setBytes(&sh, length: MemoryLayout<Shape>.stride, index: 2)
- e.dispatchThreadgroups(MTLSize(width: 1, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 1024, height: 1, depth: 1))
+ if columnScales[weight] == nil {
+ columnScales[weight] = columns
+ try kernel("ane_columns") { e in
+ e.setBuffer(w.buffer, offset: w.offset, index: 0)
+ e.setBuffer(columns, offset: lambda, index: 1); e.setBuffer(columns, offset: norm, index: 2)
+ e.setBytes(&sh, length: MemoryLayout<Shape>.stride, index: 3)
+ e.dispatchThreads(MTLSize(width: n, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 64, height: 1, depth: 1))
+ }
+ try kernel("ane_norm") { e in
+ e.setBuffer(columns, offset: norm, index: 0); e.setBuffer(columns, offset: state, index: 1)
+ e.setBytes(&sh, length: MemoryLayout<Shape>.stride, index: 2)
+ e.dispatchThreadgroups(MTLSize(width: 1, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 1024, height: 1, depth: 1))
+ }
}
try kernel("ane_row_scales") { e in
- e.setBuffer(a.buffer, offset: a.offset, index: 0); e.setBuffer(scales, offset: state, index: 1)
+ e.setBuffer(a.buffer, offset: a.offset, index: 0); e.setBuffer(columns, offset: state, index: 1)
e.setBuffer(scales, offset: sigma, index: 2)
e.setBytes(&sh, length: MemoryLayout<Shape>.stride, index: 3)
e.dispatchThreadgroups(MTLSize(width: s.ane, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
@@ -118,7 +166,7 @@ final class Coexecutor {
let slot = slots[kp * nParts + np]
var t = shape(kp: kp, np: np)
try kernel("ane_pack_weight") { e in
- e.setBuffer(w.buffer, offset: w.offset, index: 0); e.setBuffer(scales, offset: lambda, index: 1)
+ e.setBuffer(w.buffer, offset: w.offset, index: 0); e.setBuffer(columns, offset: lambda, index: 1)
e.setBuffer(slot.w, offset: 0, index: 2)
e.setBytes(&t, length: MemoryLayout<Shape>.stride, index: 3)
e.dispatchThreads(MTLSize(width: Coexecutor.tile, height: Coexecutor.tile, depth: 1),
@@ -144,7 +192,7 @@ final class Coexecutor {
let t = DispatchTime.now().uptimeNanoseconds
cblas_sgemm(CblasRowMajor, CblasNoTrans, CblasNoTrans, Int32(rows), Int32(n), Int32(k), 1,
pa, Int32(k), pw, Int32(n), 0, pc, Int32(n))
- lock.withLock { amxSeconds += Double(DispatchTime.now().uptimeNanoseconds - t) * 1e-9 }
+ finished(p, \.amx, since: t)
}
}
if s.ane > 0 {
@@ -154,10 +202,11 @@ final class Coexecutor {
guard wait(ready, p, "ANE") else { return }
let t = DispatchTime.now().uptimeNanoseconds
do { for tile in tiles { try tile.run() } } catch { fail("ANE: \(error)") }
- lock.withLock { aneSeconds += Double(DispatchTime.now().uptimeNanoseconds - t) * 1e-9 }
+ finished(p, \.ane, since: t)
}
}
try gpuRows(s.gpu)
+ gpuFinished(p, commands)
if s.amx > 0 { commands.encodeWaitForEvent(amxDone, value: p) }
if s.ane > 0 {
commands.encodeWaitForEvent(aneDone, value: p)
@@ -166,7 +215,7 @@ final class Coexecutor {
let ys = (0..<3).map { kp in slots[min(kp, kParts - 1) * nParts + np].y }
try kernel("ane_unpack") { e in
for (i, y) in ys.enumerated() { e.setBuffer(y, offset: 0, index: i) }
- e.setBuffer(scales, offset: lambda, index: 3); e.setBuffer(scales, offset: sigma, index: 4)
+ e.setBuffer(columns, offset: lambda, index: 3); e.setBuffer(scales, offset: sigma, index: 4)
e.setBuffer(c.buffer, offset: c.offset, index: 5)
e.setBytes(&t, length: MemoryLayout<Shape>.stride, index: 6)
e.dispatchThreads(MTLSize(width: Coexecutor.tile, height: s.ane, depth: 1),
@@ -176,17 +225,96 @@ final class Coexecutor {
}
}
- private func fail(_ message: String) { lock.withLock { failures.append(message) } }
+ func fail(_ message: String) { lock.withLock { failures.append(message) } }
/// Waits for the GPU to reach `value`; false (and a failure) after a minute, so a broken command buffer never
/// hangs a worker for good.
- private func wait(_ event: MTLSharedEvent, _ value: UInt64, _ who: String) -> Bool {
+ func wait(_ event: MTLSharedEvent, _ value: UInt64, _ who: String) -> Bool {
if event.wait(untilSignaledValue: value, timeoutMS: 60_000) { return true }
fail("\(who): the GPU never reached product \(value)")
return false
}
func resetTimings() { lock.withLock { amxSeconds = 0; aneSeconds = 0 } }
+
+ // MARK: - the timeline
+
+ /// Starts operation `p`'s entry and notes when the GPU releases it (`ready` reaches `p`).
+ func open(_ p: UInt64, kind: String, shares: (gpu: Int, amx: Int, ane: Int)) {
+ timeline.open(p, kind: kind, shares: shares)
+ ready.notify(listener, atValue: p) { [timeline] _, _ in timeline.mark(p, \.released, Timeline.now()) }
+ }
+
+ /// Encodes the GPU's end-of-share signal for operation `p` and notes when it fires.
+ func gpuFinished(_ p: UInt64, _ commands: MTLCommandBuffer) {
+ commands.encodeSignalEvent(gpuDone, value: p)
+ gpuDone.notify(listener, atValue: p) { [timeline] _, _ in timeline.mark(p, \.gpu, Timeline.now()) }
+ }
+
+ /// A worker's share of `p` done: its busy seconds, and its start and end in the timeline.
+ func finished(_ p: UInt64, _ unit: WritableKeyPath<Timeline.Entry, Timeline.Span>, since start: UInt64) {
+ let end = DispatchTime.now().uptimeNanoseconds
+ let seconds = Double(end - start) * 1e-9
+ lock.withLock { if unit == \Timeline.Entry.amx { amxSeconds += seconds } else { aneSeconds += seconds } }
+ timeline.mark(p, unit, Timeline.Span(start: Double(start) * 1e-9, end: Double(end) * 1e-9))
+ }
+}
+
+/// **When each split operation ran on each unit**, from the GPU's release (`ready`) to each unit's end: what the
+/// balancing loop reads. Times are seconds of the uptime clock; the GPU's are noted by an event listener (tens of
+/// microseconds late, against operations of milliseconds).
+final class Timeline: @unchecked Sendable {
+ struct Span { var start = 0.0, end = 0.0; var seconds: Double { end - start } }
+ struct Entry {
+ var kind: String
+ var shares: (gpu: Int, amx: Int, ane: Int)
+ var released = 0.0, gpu = 0.0
+ var amx = Span(), ane = Span()
+ /// Seconds from the release to the end of each unit's share (0 for a unit without one).
+ var walls: (gpu: Double, amx: Double, ane: Double) {
+ (gpu - released, shares.amx > 0 ? amx.end - released : 0, shares.ane > 0 ? ane.end - released : 0)
+ }
+ }
+ private var entries: [UInt64: Entry] = [:]
+ private let lock = NSLock()
+
+ static func now() -> Double { Double(DispatchTime.now().uptimeNanoseconds) * 1e-9 }
+
+ func open(_ p: UInt64, kind: String, shares: (gpu: Int, amx: Int, ane: Int)) {
+ lock.withLock { entries[p] = Entry(kind: kind, shares: shares) }
+ }
+
+ func mark<T>(_ p: UInt64, _ field: WritableKeyPath<Entry, T>, _ value: T) {
+ lock.withLock { entries[p]?[keyPath: field] = value }
+ }
+
+ /// The complete entries, in order, and forgets them all.
+ func drain() -> [Entry] {
+ lock.withLock {
+ defer { entries = [:] }
+ return entries.sorted { $0.key < $1.key }.map(\.value).filter { $0.released > 0 && $0.gpu > 0 }
+ }
+ }
+
+ /// Per kind: operations, mean milliseconds of each unit's share from its own start (GPU: from the release),
+ /// and the mean wall from release to the last unit's end.
+ static func summary(_ entries: [Entry]) -> String {
+ var kinds: [String] = []
+ var groups: [String: [Entry]] = [:]
+ for e in entries {
+ if groups[e.kind] == nil { kinds.append(e.kind) }
+ groups[e.kind, default: []].append(e)
+ }
+ return kinds.map { kind in
+ let g = groups[kind]!, n = Double(g.count), s = g[0].shares
+ func mean(_ f: (Entry) -> Double) -> Double { g.map(f).reduce(0, +) / n * 1e3 }
+ return String(format: " %-12@ ×%3d shares %d/%d/%d GPU %6.2f ms AMX/CPU %6.2f ms (starts +%.2f) ANE %6.2f ms (starts +%.2f) wall %6.2f ms",
+ kind as NSString, g.count, s.gpu, s.amx, s.ane, mean { $0.gpu - $0.released },
+ mean { $0.amx.seconds }, mean { $0.shares.amx > 0 ? $0.amx.start - $0.released : 0 },
+ mean { $0.ane.seconds }, mean { $0.shares.ane > 0 ? $0.ane.start - $0.released : 0 },
+ mean { max($0.walls.gpu, $0.walls.amx, $0.walls.ane) })
+ }.joined(separator: "\n")
+ }
}
/// A thread running submitted jobs in order.
diff --git a/engine-mac/Sources/KokonEngine/Kernels/ane.metal b/engine-mac/Sources/KokonEngine/Kernels/ane.metal
index 3328586..0e7e6a5 100644
--- a/engine-mac/Sources/KokonEngine/Kernels/ane.metal
+++ b/engine-mac/Sources/KokonEngine/Kernels/ane.metal
@@ -111,3 +111,30 @@ kernel void ane_unpack(device const half *y0 [[buffer(0)]], device const half *y
uint j = s.n0 + jj;
c[(ulong)(s.row0 + i) * s.n + j] = (sum / lambda[j]) / sigma[i];
}
+
+// Attention heads to and from the Neural Engine's surfaces [heads, rows, 128] fp16 (heads first), from the rows of
+// `stride` floats the products write (head h at columns column0 + 128 h). Surface rows past `valid` are zeros.
+struct HeadsShape {
+ uint stride;
+ uint column0;
+ uint rows; // rows per head in the surface
+ uint valid; // of which exist in the fp32 tensor, from row0
+ uint row0;
+ uint heads;
+ float factor; // q: 1/sqrt(128)
+};
+
+kernel void ane_heads_pack(device const float *src [[buffer(0)]], device half *dst [[buffer(1)]],
+ constant HeadsShape &s [[buffer(2)]], uint3 g [[thread_position_in_grid]]) {
+ uint c = g.x, r = g.y, h = g.z;
+ if (c >= 128 || r >= s.rows || h >= s.heads) return;
+ float x = r < s.valid ? src[(ulong)(s.row0 + r) * s.stride + s.column0 + h * 128 + c] * s.factor : 0.0f;
+ dst[((ulong)h * s.rows + r) * 128 + c] = half(x);
+}
+
+kernel void ane_heads_unpack(device const half *src [[buffer(0)]], device float *dst [[buffer(1)]],
+ constant HeadsShape &s [[buffer(2)]], uint3 g [[thread_position_in_grid]]) {
+ uint c = g.x, r = g.y, h = g.z;
+ if (c >= 128 || r >= s.valid || h >= s.heads) return;
+ dst[(ulong)(s.row0 + r) * s.stride + s.column0 + h * 128 + c] = float(src[((ulong)h * s.rows + r) * 128 + c]);
+}
diff --git a/engine-mac/Sources/KokonEngine/Kernels/attention.metal b/engine-mac/Sources/KokonEngine/Kernels/attention.metal
index 1b7e784..263faf7 100644
--- a/engine-mac/Sources/KokonEngine/Kernels/attention.metal
+++ b/engine-mac/Sources/KokonEngine/Kernels/attention.metal
@@ -5,10 +5,20 @@ using namespace metal;
// Fused scaled dot-product attention, fp32, head_dim 128, on the layout the products write: q, k, v and out are
// rows of `stride` floats (all heads side by side), head h at columns [128 h, 128 h + 128).
//
-// A threadgroup takes 64 query rows of one head (8 SIMD groups of 8 rows) and walks the keys 32 at a time with an
+// A threadgroup takes 64 query rows of one head (8 SIMD groups of 8 rows) and walks the keys 64 at a time with an
// online softmax (running max and sum per row), so the [queries, keys] score matrix never exists in memory. Each
-// key tile, then value tile, is loaded once into threadgroup memory by all 256 threads (coalesced rows) and shared
-// by the 8 SIMD groups; products on 8×8 SIMD-group matrices.
+// key tile, then its value tile, is loaded once into threadgroup memory by all 256 threads (coalesced rows) and
+// shared by the 8 SIMD groups; products on 8×8 SIMD-group matrices.
+//
+// The softmax stays in registers: each lane holds two elements of every 8×8 matrix, row (lane / 16) * 4 +
+// (lane / 2) % 4, columns ((lane / 8) % 2) * 4 + (lane % 2) * 2 and the next (`thread_elements`, the layout of
+// Apple's GPUs); the four lanes of a row differ in lane bits 0 and 3, so a row's max and sum take two shuffles, and
+// O is rescaled in place.
+//
+// Measured at 1024² (4 096 queries × 4 128 keys, 32 heads, M2 Pro, `probe-attention`, 2026-10-25): 120 ms; the
+// first version (32 keys a tile, scores through threadgroup memory, O rescaled by a diagonal product) 145 ms; with
+// separate 32-key K and V tiles 135; 16-key tiles or 4 SIMD groups 1.5-2.6× slower, 16 SIMD groups 178, K and V read
+// straight from memory by each SIMD group 520.
//
// causal: the queries are the last `queries` of the keys; query i sees keys j <= keys - queries + i (a text
// segment of Qwen-Image 2.1's block-causal mask). Rows past `queries` are computed but never stored and keys past
@@ -17,7 +27,7 @@ using namespace metal;
// Barriers are threadgroup barriers: with simdgroup barriers alone the lanes' writes to threadgroup memory were
// sometimes not seen by the SIMD-group loads that followed (1 run in 4 wrong, M2 Pro, macOS 27, AttentionTests).
-constant uint BQ = 64, BK = 32, D = 128, GROUPS = 8;
+constant uint BQ = 64, BK = 64, D = 128;
struct FlashParams {
uint queries;
@@ -28,6 +38,11 @@ struct FlashParams {
uint headBase;
};
+// A lane's own two elements of an 8×8 matrix (`thread_elements` is typed as the whole matrix).
+static inline thread float2 &elements(thread simdgroup_float8x8 &m) {
+ return *(thread float2 *)&m.thread_elements();
+}
+
kernel void flash_attention(device const float *q [[buffer(0)]], device const float *k [[buffer(1)]],
device const float *v [[buffer(2)]], device float *out [[buffer(3)]],
constant FlashParams &p [[buffer(4)]],
@@ -35,11 +50,7 @@ kernel void flash_attention(device const float *q [[buffer(0)]], device const fl
uint tid [[thread_index_in_threadgroup]],
uint sg [[simdgroup_index_in_threadgroup]],
uint lane [[thread_index_in_simdgroup]]) {
- threadgroup float tile[BK * D]; // 16 KB: the key tile, then the value tile
- threadgroup float stageAll[GROUPS][8 * BK]; // 8 KB: each SIMD group's scores / probabilities
- threadgroup float diagAll[GROUPS][64]; // 2 KB: each SIMD group's per-row scale, as a diagonal
- threadgroup float *stage = stageAll[sg];
- threadgroup float *diag = diagAll[sg];
+ threadgroup float tile[BK * D]; // 32 KB: the key tile, then the value tile
const uint head = p.headBase + tg.y;
const uint q0 = tg.x * BQ + sg * 8;
@@ -52,11 +63,9 @@ kernel void flash_attention(device const float *q [[buffer(0)]], device const fl
simdgroup_load(Q[d], q + (ulong)q0 * stride + head * D + 8 * d, stride);
O[d] = make_filled_simdgroup_matrix<float, 8, 8>(0.0f);
}
- diag[lane * 2] = 0.0f;
- diag[lane * 2 + 1] = 0.0f;
- // Each lane owns 8 scores of one row: row lane / 4, columns (lane % 4) * 8 ... + 8.
- const uint row = lane / 4, cbase = (lane % 4) * 8;
+ const uint row = (lane / 16) * 4 + (lane / 2) % 4;
+ const uint col = ((lane / 8) % 2) * 4 + (lane % 2) * 2;
const uint qi = q0 + row;
const uint offset = p.keys - p.queries;
const int limit = p.causal ? int(offset + qi) : int(p.keys) - 1;
@@ -66,94 +75,76 @@ kernel void flash_attention(device const float *q [[buffer(0)]], device const fl
if (p.causal) kEnd = min(p.keys, offset + min(tg.x * BQ + BQ - 1, p.queries - 1) + 1);
for (uint j0 = 0; j0 < kEnd; j0 += BK) {
- // Key tile: 32 rows × 128, 256 threads × 4 float4.
+ // Key tile: 64 rows × 128, 256 threads × 8 float4.
for (uint i = tid; i < BK * D / 4; i += 256) {
uint r = i / (D / 4), c = i % (D / 4);
((threadgroup float4 *)tile)[i] = *((device const float4 *)(kh + (ulong)(j0 + r) * stride) + c);
}
threadgroup_barrier(mem_flags::mem_threadgroup);
- simdgroup_float8x8 S[4];
- for (uint c = 0; c < 4; c++) S[c] = make_filled_simdgroup_matrix<float, 8, 8>(0.0f);
+ simdgroup_float8x8 S[8];
+ for (uint c = 0; c < 8; c++) S[c] = make_filled_simdgroup_matrix<float, 8, 8>(0.0f);
for (uint d = 0; d < 16; d++) {
- for (uint c = 0; c < 4; c++) {
+ for (uint c = 0; c < 8; c++) {
simdgroup_float8x8 Kt;
simdgroup_load(Kt, tile + 8 * c * D + 8 * d, D, ulong2(0, 0), true);
simdgroup_multiply_accumulate(S[c], Q[d], Kt, S[c]);
}
}
- for (uint c = 0; c < 4; c++) simdgroup_store(S[c], stage + 8 * c, BK);
- threadgroup_barrier(mem_flags::mem_threadgroup);
- // Value tile into the same memory (everyone is done with the keys), softmax meanwhile.
- for (uint i = tid; i < BK * D / 4; i += 256) {
- uint r = i / (D / 4), c = i % (D / 4);
- ((threadgroup float4 *)tile)[i] = *((device const float4 *)(vh + (ulong)(j0 + r) * stride) + c);
- }
- float s[8];
+ // Online softmax on the lane's two scores of each of the 8 matrices.
+ float2 s[8];
float blockMax = -INFINITY;
- for (uint t = 0; t < 8; t++) {
- int j = int(j0 + cbase + t);
- float x = stage[row * BK + cbase + t] * p.scale;
- if (j > limit) x = -INFINITY;
- s[t] = x;
- blockMax = max(blockMax, x);
+ for (uint c = 0; c < 8; c++) {
+ s[c] = elements(S[c]) * p.scale;
+ int j = int(j0 + 8 * c + col);
+ if (j > limit) s[c].x = -INFINITY;
+ if (j + 1 > limit) s[c].y = -INFINITY;
+ blockMax = max(blockMax, max(s[c].x, s[c].y));
}
blockMax = max(blockMax, simd_shuffle_xor(blockMax, 1));
- blockMax = max(blockMax, simd_shuffle_xor(blockMax, 2));
+ blockMax = max(blockMax, simd_shuffle_xor(blockMax, 8));
float mNew = max(m, blockMax);
float alpha = (mNew == -INFINITY) ? 1.0f : exp2(m - mNew);
float sum = 0.0f;
- for (uint t = 0; t < 8; t++) {
- float e = (mNew == -INFINITY) ? 0.0f : exp2(s[t] - mNew);
- stage[row * BK + cbase + t] = e;
- sum += e;
+ for (uint c = 0; c < 8; c++) {
+ float2 e = (mNew == -INFINITY) ? float2(0.0f) : exp2(s[c] - mNew);
+ elements(S[c]) = e;
+ sum += e.x + e.y;
}
sum += simd_shuffle_xor(sum, 1);
- sum += simd_shuffle_xor(sum, 2);
+ sum += simd_shuffle_xor(sum, 8);
l = l * alpha + sum;
m = mNew;
- if (lane % 4 == 0) diag[row * 8 + row] = alpha;
- const bool rescale = !simd_all(alpha == 1.0f);
- threadgroup_barrier(mem_flags::mem_threadgroup);
+ if (!simd_all(alpha == 1.0f)) {
+ for (uint d = 0; d < 16; d++) elements(O[d]) *= alpha;
+ }
- simdgroup_float8x8 P[4];
- for (uint c = 0; c < 4; c++) simdgroup_load(P[c], stage + 8 * c, BK);
- if (rescale) {
- simdgroup_float8x8 Dg;
- simdgroup_load(Dg, diag, 8);
- for (uint d = 0; d < 16; d++) {
- simdgroup_float8x8 scaled;
- simdgroup_multiply(scaled, Dg, O[d]);
- O[d] = scaled;
- }
+ // Value tile into the same memory, once every SIMD group is done with the keys.
+ threadgroup_barrier(mem_flags::mem_threadgroup);
+ for (uint i = tid; i < BK * D / 4; i += 256) {
+ uint r = i / (D / 4), c = i % (D / 4);
+ ((threadgroup float4 *)tile)[i] = *((device const float4 *)(vh + (ulong)(j0 + r) * stride) + c);
}
+ threadgroup_barrier(mem_flags::mem_threadgroup);
for (uint d = 0; d < 16; d++) {
- for (uint c = 0; c < 4; c++) {
+ for (uint c = 0; c < 8; c++) {
simdgroup_float8x8 V;
simdgroup_load(V, tile + 8 * c * D + 8 * d, D);
- simdgroup_multiply_accumulate(O[d], P[c], V, O[d]);
+ simdgroup_multiply_accumulate(O[d], S[c], V, O[d]);
}
}
threadgroup_barrier(mem_flags::mem_threadgroup);
}
- // O / l per row, then the valid rows to memory.
- if (lane % 4 == 0) diag[row * 8 + row] = 1.0f / l;
- threadgroup_barrier(mem_flags::mem_threadgroup);
- simdgroup_float8x8 Dg;
- simdgroup_load(Dg, diag, 8);
- for (uint part = 0; part < 4; part++) {
- for (uint c = 0; c < 4; c++) {
- simdgroup_float8x8 r;
- simdgroup_multiply(r, Dg, O[part * 4 + c]);
- simdgroup_store(r, stage + 8 * c, BK);
- }
- threadgroup_barrier(mem_flags::mem_threadgroup);
- if (qi < p.queries) {
- device float *o = out + (ulong)qi * stride + head * D + part * 32 + cbase;
- for (uint t = 0; t < 8; t++) o[t] = stage[row * BK + cbase + t];
+ // O / l, the valid rows to memory.
+ if (qi < p.queries) {
+ const float inv = 1.0f / l;
+ device float *o = out + (ulong)qi * stride + head * D + col;
+ for (uint d = 0; d < 16; d++) {
+ float2 e = elements(O[d]) * inv;
+ o[8 * d] = e.x;
+ o[8 * d + 1] = e.y;
}
- threadgroup_barrier(mem_flags::mem_threadgroup);
}
}
diff --git a/engine-mac/Sources/KokonEngine/NeuralEngine.swift b/engine-mac/Sources/KokonEngine/NeuralEngine.swift
index ae20099..7402970 100644
--- a/engine-mac/Sources/KokonEngine/NeuralEngine.swift
+++ b/engine-mac/Sources/KokonEngine/NeuralEngine.swift
@@ -39,21 +39,9 @@ public final class NeuralEngineProduct: @unchecked Sendable {
public init(rows: Int, k: Int, n: Int) throws {
self.rows = rows; self.k = k; self.n = n
- directory = FileManager.default.temporaryDirectory
- .appendingPathComponent("kokon-ane-\(ProcessInfo.processInfo.processIdentifier)-\(rows)x\(k)x\(n)-\(UUID().uuidString)")
- .appendingPathExtension("mlmodelc")
- try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
- try NeuralEngineProduct.program(rows: rows, k: k, n: n)
- .write(to: directory.appendingPathComponent("model.mil"), atomically: true, encoding: .utf8)
- try NeuralEngineProduct.description(rows: rows, k: k, n: n)
- .write(to: directory.appendingPathComponent("coremldata.bin"))
- let configuration = MLModelConfiguration()
- configuration.computeUnits = .cpuAndNeuralEngine
- do {
- model = try MLModel(contentsOf: directory, configuration: configuration)
- } catch {
- throw Failure.program("\(rows)×\(k)×\(n): \(error.localizedDescription)")
- }
+ (model, directory) = try NeuralEngineProduct.compile(
+ "\(rows)x\(k)x\(n)", program: NeuralEngineProduct.program(rows: rows, k: k, n: n),
+ inputs: [("x", [rows, k]), ("w", [k, n])], outputs: [("y", [rows, n])])
x = try NeuralEngineProduct.surface(rows: rows, columns: k)
w = try NeuralEngineProduct.surface(rows: k, columns: n)
y = try NeuralEngineProduct.surface(rows: rows, columns: n)
@@ -114,9 +102,29 @@ public final class NeuralEngineProduct: @unchecked Sendable {
"""
}
+ /// Writes `program` and its description as a compiled model in a temporary directory and loads it for the
+ /// CPU and the Neural Engine. `tag` names the directory.
+ static func compile(_ tag: String, program: String, inputs: [(String, [Int])],
+ outputs: [(String, [Int])]) throws -> (MLModel, URL) {
+ let directory = FileManager.default.temporaryDirectory
+ .appendingPathComponent("kokon-ane-\(ProcessInfo.processInfo.processIdentifier)-\(tag)-\(UUID().uuidString)")
+ .appendingPathExtension("mlmodelc")
+ try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
+ try program.write(to: directory.appendingPathComponent("model.mil"), atomically: true, encoding: .utf8)
+ try description(inputs: inputs, outputs: outputs).write(to: directory.appendingPathComponent("coremldata.bin"))
+ let configuration = MLModelConfiguration()
+ configuration.computeUnits = .cpuAndNeuralEngine
+ do {
+ return (try MLModel(contentsOf: directory, configuration: configuration), directory)
+ } catch {
+ try? FileManager.default.removeItem(at: directory)
+ throw Failure.program("\(tag): \(error.localizedDescription)")
+ }
+ }
+
/// `coremldata.bin`: a fixed header with the description's length (little-endian u64 at 0x4b), the
- /// model description as protobuf (inputs `x`, `w`, output `y`, fp16 multi-arrays), a fixed trailer.
- static func description(rows: Int, k: Int, n: Int) -> Data {
+ /// model description as protobuf (named fp16 multi-arrays in and out), a fixed trailer.
+ static func description(inputs: [(String, [Int])], outputs: [(String, [Int])]) -> Data {
func varint(_ value: Int) -> [UInt8] {
var v = UInt64(value), out: [UInt8] = []
repeat {
@@ -133,7 +141,7 @@ public final class NeuralEngineProduct: @unchecked Sendable {
let array = field(0x2a, shape + [0x10] + varint(65_552))
return field(tag, field(0x0a, Array(name.utf8)) + field(0x1a, array))
}
- let payload = feature(0x0a, "x", [rows, k]) + feature(0x0a, "w", [k, n]) + feature(0x52, "y", [rows, n])
+ let payload: [UInt8] = inputs.flatMap { feature(0x0a, $0.0, $0.1) } + outputs.flatMap { feature(0x52, $0.0, $0.1) }
+ [0xa2, 0x06, 0x00]
var header = [UInt8](repeating: 0, count: 0x53)
header[0] = 0xf6; header[1] = 0x01; header[4] = 0x09; header[0x1c] = 0x07
@@ -150,6 +158,10 @@ extension NeuralEngineProduct {
/// A GPU buffer over a surface's memory, no copy: the GPU writes the Neural Engine's operands in place.
/// The surface stays locked for the product's life (unified memory: the lock only pins the address).
public func gpuBuffer(_ surface: CVPixelBuffer, device: MTLDevice) throws -> MTLBuffer {
+ try NeuralEngineProduct.gpuBuffer(surface, device: device)
+ }
+
+ static func gpuBuffer(_ surface: CVPixelBuffer, device: MTLDevice) throws -> MTLBuffer {
CVPixelBufferLockBaseAddress(surface, [])
guard let io = CVPixelBufferGetIOSurface(surface)?.takeUnretainedValue(),
let base = CVPixelBufferGetBaseAddress(surface) else { throw Failure.surface("no IOSurface") }
@@ -163,3 +175,72 @@ extension NeuralEngineProduct {
return buffer
}
}
+
+/// **Attention on the Neural Engine**: `y[h] = softmax(q[h] · k[h]ᵀ) · v[h]` for `heads` heads of 128, fp16, `q`
+/// already scaled by 1/√128. One program per shape, operands in surfaces `[heads · rows, 128]` (heads first),
+/// bound once, as for the product.
+public final class NeuralEngineAttention: @unchecked Sendable {
+ public let heads: Int, queries: Int, keys: Int
+ public static let headDim = 128
+ private let model: MLModel
+ private let features: MLFeatureProvider
+ private let options: MLPredictionOptions
+ public let q: CVPixelBuffer, k: CVPixelBuffer, v: CVPixelBuffer, y: CVPixelBuffer
+ private let directory: URL
+
+ /// Measured on the M2 Pro (`probe-attention`, 4 096 queries × 4 128 keys, 2026-10-25): 4 heads × 1 024 queries a
+ /// run 5.3 ms per head of all queries, 8 × 1 024 5.5, 1 × 4 096 6.5, 8 × 4 096 10.5 (the score matrix grows past
+ /// what the Neural Engine keeps close); MIL's fused `scaled_dot_product_attention` 33 ms per head. Head 0 within
+ /// 2.2e-3 of an exact attention (fp16 scores and probabilities).
+ ///
+ /// `k`, `v`: surfaces of another program of the same heads and keys to share (the query chunks of one head group).
+ public init(heads: Int, queries: Int, keys: Int, sharing: NeuralEngineAttention? = nil) throws {
+ self.heads = heads; self.queries = queries; self.keys = keys
+ let d = NeuralEngineAttention.headDim
+ (model, directory) = try NeuralEngineProduct.compile(
+ "attention-\(heads)x\(queries)x\(keys)", program: NeuralEngineAttention.program(heads: heads, queries: queries, keys: keys),
+ inputs: [("q", [1, heads, queries, d]), ("k", [1, heads, keys, d]), ("v", [1, heads, keys, d])],
+ outputs: [("y", [1, heads, queries, d])])
+ q = try NeuralEngineProduct.surface(rows: heads * queries, columns: d)
+ k = try sharing?.k ?? NeuralEngineProduct.surface(rows: heads * keys, columns: d)
+ v = try sharing?.v ?? NeuralEngineProduct.surface(rows: heads * keys, columns: d)
+ y = try NeuralEngineProduct.surface(rows: heads * queries, columns: d)
+ func shape(_ dims: Int...) -> [NSNumber] { dims.map { NSNumber(value: $0) } }
+ options = MLPredictionOptions()
+ options.outputBackings = ["y": MLMultiArray(pixelBuffer: y, shape: shape(1, heads, queries, d))]
+ features = try MLDictionaryFeatureProvider(dictionary: [
+ "q": MLFeatureValue(multiArray: MLMultiArray(pixelBuffer: q, shape: shape(1, heads, queries, d))),
+ "k": MLFeatureValue(multiArray: MLMultiArray(pixelBuffer: k, shape: shape(1, heads, keys, d))),
+ "v": MLFeatureValue(multiArray: MLMultiArray(pixelBuffer: v, shape: shape(1, heads, keys, d))),
+ ])
+ }
+
+ deinit { try? FileManager.default.removeItem(at: directory) }
+
+ public func run() throws {
+ _ = try model.prediction(from: features, options: options)
+ }
+
+ /// A GPU buffer over one of its surfaces (`NeuralEngineProduct.gpuBuffer`).
+ public func gpuBuffer(_ surface: CVPixelBuffer, device: MTLDevice) throws -> MTLBuffer {
+ try NeuralEngineProduct.gpuBuffer(surface, device: device)
+ }
+
+ static func program(heads h: Int, queries l: Int, keys s: Int) -> String {
+ let d = NeuralEngineAttention.headDim
+ return """
+ program(1.3)
+ [buildInfo = dict<string, string>({{"coremlc-component-MIL", "3600.16.1"}, {"coremlc-version", "3600.25.2"}})]
+ {
+ func main<ios18>(tensor<fp16, [1, \(h), \(l), \(d)]> q, tensor<fp16, [1, \(h), \(s), \(d)]> k, tensor<fp16, [1, \(h), \(s), \(d)]> v) {
+ bool f = const()[name = string("f"), val = bool(false)];
+ bool t = const()[name = string("t"), val = bool(true)];
+ tensor<fp16, [1, \(h), \(l), \(s)]> scores = matmul(transpose_x = f, transpose_y = t, x = q, y = k)[name = string("scores")];
+ int32 axis = const()[name = string("axis"), val = int32(-1)];
+ tensor<fp16, [1, \(h), \(l), \(s)]> p = softmax(axis = axis, x = scores)[name = string("p")];
+ tensor<fp16, [1, \(h), \(l), \(d)]> y = matmul(transpose_x = f, transpose_y = f, x = p, y = v)[name = string("y")];
+ } -> (y);
+ }
+ """
+ }
+}
diff --git a/engine-mac/Sources/KokonEngine/QwenDiT.swift b/engine-mac/Sources/KokonEngine/QwenDiT.swift
index 82e309c..d50039c 100644
--- a/engine-mac/Sources/KokonEngine/QwenDiT.swift
+++ b/engine-mac/Sources/KokonEngine/QwenDiT.swift
@@ -44,8 +44,9 @@ public final class QwenDiT {
let rope, modulation, qkNorms, textNorm, textIn, textOut, latent, output: Arena.Slice
private var nextReserve = 0
private var products: [String: MPSMatrixMultiplication] = [:]
- /// The AMX and the Neural Engine beside the GPU on the large products (`Coexecution.swift`); nil: GPU alone.
- /// `KOKON_COEXEC=0` turns it off, `KOKON_ANE=0` / `KOKON_AMX=0` one unit, `KOKON_AMX_SHARE` the AMX's rows.
+ /// The AMX and the Neural Engine beside the GPU on the large products and the target's attention
+ /// (`Coexecution.swift`, `AttentionSplit.swift`); nil: GPU alone. `KOKON_COEXEC=0` turns it off, `KOKON_ANE=0` /
+ /// `KOKON_AMX=0` one unit, `KOKON_ANE_ATTENTION=1` heads on the Neural Engine too, `KOKON_BALANCE=0` the balancing loop.
let coexecutor: Coexecutor?
private var attentions: [String: GraphAttention] = [:]
/// Seconds per phase of the last evaluation.
@@ -74,12 +75,15 @@ public final class QwenDiT {
let environment = ProcessInfo.processInfo.environment
if environment["KOKON_COEXEC"] != "0", sequence.count >= 1024 {
- coexecutor = try Coexecutor(context: context,
- amxShare: Double(environment["KOKON_AMX_SHARE"] ?? "") ?? 0.0625,
- ane: environment["KOKON_ANE"] != "0", amx: environment["KOKON_AMX"] != "0")
+ coexecutor = try Coexecutor(context: context, rows: QwenDiT.padded(sequence.count),
+ ane: environment["KOKON_ANE"] != "0", amx: environment["KOKON_AMX"] != "0",
+ aneAttention: environment["KOKON_ANE_ATTENTION"] == "1")
} else {
coexecutor = nil
}
+ if let coexecutor, coexecutor.attentionApplies(queries: sequence.target) {
+ try coexecutor.prepareAttention(queries: sequence.target, keys: sequence.count)
+ }
let table = sequence.ropeTable()
arena.pointer(rope).update(from: table, count: table.count)
// Small weights unpacked once on the CPU: the QK norms of every layer, txt_in's norm as (w + 1).
@@ -111,26 +115,38 @@ public final class QwenDiT {
static func silu(_ x: [Float]) -> [Float] { x.map { $0 / (1 + expf(-$0)) } }
- /// `y = W x` for a Linear weight `[out, in]` of the file, unpacked on the CPU.
- func linearCPU(_ name: String, _ x: [Float]) throws -> [Float] {
+ /// `y = W x` for each `x`, W a Linear weight `[out, in]` of the file unpacked once on the CPU.
+ func linearCPU(_ name: String, _ xs: [[Float]]) throws -> [[Float]] {
let t = try file.tensor(QwenDiT.prefix + name)
let w = try Dequant.array(file, t.name)
- var y = [Float](repeating: 0, count: t.rows)
- cblas_sgemv(CblasRowMajor, CblasNoTrans, Int32(t.rows), Int32(t.columns), 1, w, Int32(t.columns), x, 1, 0, &y, 1)
- return y
+ return xs.map { x in
+ var y = [Float](repeating: 0, count: t.rows)
+ cblas_sgemv(CblasRowMajor, CblasNoTrans, Int32(t.rows), Int32(t.columns), 1, w, Int32(t.columns), x, 1, 0, &y, 1)
+ return y
+ }
+ }
+
+ /// The stored modulation rows of each timestep seen (`modulationRows`): unpacking the four weights takes ~0.1 s
+ /// (`modulation.1` is 268 MB in fp32), so the pipeline prepares all its timesteps at once.
+ private var modulationCache: [Float: [Float]] = [:]
+
+ /// Computes and keeps the modulation rows of `timesteps` not yet seen.
+ public func prepareModulation(_ timesteps: [Float]) throws {
+ let missing = Array(Set(timesteps.filter { modulationCache[$0] == nil }))
+ guard !missing.isEmpty else { return }
+ for (t, row) in zip(missing, try modulationRows(timesteps: missing).stored) { modulationCache[t] = row }
}
/// The temb, block modulation and final scale of each timestep, as stored for the kernels:
/// `[1 + s₁, tanh g₁, 1 + s₂, tanh g₂, 1 + s_final]`, `dim` floats each.
public func modulationRows(timesteps: [Float]) throws -> (temb: [[Float]], raw: [[Float]], stored: [[Float]]) {
let d = QwenDiT.dim
- var tembs: [[Float]] = [], raws: [[Float]] = [], stored: [[Float]] = []
- for t in timesteps {
- let h = QwenDiT.silu(try linearCPU("time_text_embed.timestep_embedder.linear_1.weight", QwenDiT.sinusoid(t)))
- let temb = try linearCPU("time_text_embed.timestep_embedder.linear_2.weight", h)
- let active = QwenDiT.silu(temb)
- let m = try linearCPU("modulation.1.weight", active)
- let f = try linearCPU("norm_out.linear.weight", active)
+ var raws: [[Float]] = [], stored: [[Float]] = []
+ let h = try linearCPU("time_text_embed.timestep_embedder.linear_1.weight", timesteps.map(QwenDiT.sinusoid)).map(QwenDiT.silu)
+ let tembs = try linearCPU("time_text_embed.timestep_embedder.linear_2.weight", h)
+ let active = tembs.map(QwenDiT.silu)
+ let ms = try linearCPU("modulation.1.weight", active), fs = try linearCPU("norm_out.linear.weight", active)
+ for (m, f) in zip(ms, fs) {
var row = [Float](repeating: 0, count: 5 * d)
for j in 0..<d {
row[j] = 1 + m[j]
@@ -139,7 +155,7 @@ public final class QwenDiT {
row[3 * d + j] = tanhf(m[3 * d + j])
row[4 * d + j] = 1 + f[j]
}
- tembs.append(temb); raws.append(m); stored.append(row)
+ raws.append(m); stored.append(row)
}
return (tembs, raws, stored)
}
@@ -156,8 +172,8 @@ public final class QwenDiT {
let d = QwenDiT.dim, n = sequence.count, p = sequence.prefix, s = sequence.target
let started = DispatchTime.now().uptimeNanoseconds
timings = [:]
- let rows = try modulationRows(timesteps: [0, sigma]).stored
- arena.pointer(modulation).update(from: rows[0] + rows[1], count: 10 * d)
+ try prepareModulation([0, sigma])
+ arena.pointer(modulation).update(from: modulationCache[0]! + modulationCache[sigma]!, count: 10 * d)
timings["modulation"] = elapsed(since: started)
// Inputs: the text rows gathered in encoder order, the target tokens.
@@ -209,6 +225,7 @@ public final class QwenDiT {
if let probe { try finish(commands); probe("norm_out", arena.pointer(normed), n * d); commands = try begin() }
try linear("proj_out.weight", a: normed, aRow: p, c: output, m: s, commands: commands)
try finish(commands)
+ if let coexecutor, !profiling, balancing { try coexecutor.rebalance() }
let out = Array(UnsafeBufferPointer(start: arena.pointer(output), count: s * QwenDiT.channels))
probe?("model_out", arena.pointer(output), s * QwenDiT.channels)
timings["total"] = elapsed(since: started)
@@ -255,6 +272,9 @@ public final class QwenDiT {
// MARK: - encoders
+ /// The balancing loop after each evaluation (`Balancer.swift`); `KOKON_BALANCE=0` keeps the starting shares.
+ public var balancing = ProcessInfo.processInfo.environment["KOKON_BALANCE"] != "0"
+
/// When set, every operation runs in a command buffer of its own and its GPU time is added to `timings`
/// under its kind (slower overall: the GPU idles between operations).
public var profiling = false
@@ -333,7 +353,7 @@ public final class QwenDiT {
func operand(_ slice: Arena.Slice, _ row: Int, _ width: Int) -> (MTLBuffer, Int, UnsafeMutablePointer<Float>) {
(arena.buffer, slice.offset + row * width * 4, arena.pointer(slice, floatOffset: row * width))
}
- try coexecutor.encode(a: operand(a, aRow, kk), w: operand(reserve, 0, nn), c: operand(c, cRow, nn),
+ try coexecutor.encode(weight: name, a: operand(a, aRow, kk), w: operand(reserve, 0, nn), c: operand(c, cRow, nn),
m: m, k: kk, n: nn, commands: commands, gpuRows: gpuRows)
} else {
try gpuRows(m)
@@ -419,24 +439,24 @@ public final class QwenDiT {
let graphAttention = ProcessInfo.processInfo.environment["KOKON_ATTENTION"] == "graph"
struct FlashParams { var queries, keys, causal, stride: UInt32; var scale: Float; var headBase: UInt32 }
- /// Queries `[queries, queries + count)` against keys `[0, keys)`, into the same rows of `attended`.
+ /// Queries `[queries, queries + count)` against keys `[0, keys)`, into the same rows of `attended`. The target's
+ /// queries are cut by heads with the Neural Engine and the CPU when the co-executor runs (`AttentionSplit.swift`).
func attend(queries: Int, count: Int, keys: Int, causal: Bool, commands: MTLCommandBuffer) throws {
let row = QwenDiT.dim * 4
- if !graphAttention {
- try compute("flash_attention", commands: commands) { e in
- e.setBuffer(arena.buffer, offset: q.offset + queries * row, index: 0)
- e.setBuffer(arena.buffer, offset: k.offset, index: 1)
- e.setBuffer(arena.buffer, offset: v.offset, index: 2)
- e.setBuffer(arena.buffer, offset: attended.offset + queries * row, index: 3)
- var params = FlashParams(queries: UInt32(count), keys: UInt32(keys), causal: causal ? 1 : 0,
- stride: UInt32(QwenDiT.dim), scale: 1 / Float(QwenDiT.headDim).squareRoot() * Float(M_LOG2E),
- headBase: 0)
- e.setBytes(&params, length: MemoryLayout<FlashParams>.stride, index: 4)
- e.dispatchThreadgroups(MTLSize(width: (count + 63) / 64, height: QwenDiT.heads, depth: 1),
- threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
+ if let coexecutor, !profiling, !graphAttention, !causal, coexecutor.attentionApplies(queries: count) {
+ func operand(_ slice: Arena.Slice, _ first: Int) -> Coexecutor.Operand {
+ (arena.buffer, slice.offset + first * row, arena.pointer(slice, floatOffset: first * QwenDiT.dim))
+ }
+ try coexecutor.encodeAttention(q: operand(q, queries), k: operand(k, 0), v: operand(v, 0), out: operand(attended, queries),
+ queries: count, keys: keys, stride: QwenDiT.dim, commands: commands) { gpuHeads in
+ try flash(queries: queries, count: count, keys: keys, causal: false, heads: gpuHeads, commands: commands)
}
return
}
+ if !graphAttention {
+ try flash(queries: queries, count: count, keys: keys, causal: causal, heads: QwenDiT.heads, commands: commands)
+ return
+ }
let key = "\(count)×\(keys)\(causal ? "c" : "")"
let attention: GraphAttention
if let a = attentions[key] { attention = a } else {
@@ -450,4 +470,22 @@ public final class QwenDiT {
out: (b, attended.offset + queries * row))
}
}
+
+ /// `flash_attention` on heads `[0, heads)`.
+ func flash(queries: Int, count: Int, keys: Int, causal: Bool, heads: Int, commands: MTLCommandBuffer) throws {
+ guard heads > 0 else { return }
+ let row = QwenDiT.dim * 4
+ try compute("flash_attention", commands: commands) { e in
+ e.setBuffer(arena.buffer, offset: q.offset + queries * row, index: 0)
+ e.setBuffer(arena.buffer, offset: k.offset, index: 1)
+ e.setBuffer(arena.buffer, offset: v.offset, index: 2)
+ e.setBuffer(arena.buffer, offset: attended.offset + queries * row, index: 3)
+ var params = FlashParams(queries: UInt32(count), keys: UInt32(keys), causal: causal ? 1 : 0,
+ stride: UInt32(QwenDiT.dim), scale: 1 / Float(QwenDiT.headDim).squareRoot() * Float(M_LOG2E),
+ headBase: 0)
+ e.setBytes(&params, length: MemoryLayout<FlashParams>.stride, index: 4)
+ e.dispatchThreadgroups(MTLSize(width: (count + 63) / 64, height: heads, depth: 1),
+ threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
+ }
+ }
}
diff --git a/engine-mac/Sources/KokonEngine/QwenPipeline.swift b/engine-mac/Sources/KokonEngine/QwenPipeline.swift
index 3b7ba7c..d7db681 100644
--- a/engine-mac/Sources/KokonEngine/QwenPipeline.swift
+++ b/engine-mac/Sources/KokonEngine/QwenPipeline.swift
@@ -21,6 +21,9 @@ public final class QwenPipeline {
public var log: (String) -> Void = { print($0) }
/// Seconds per phase of the last run: encode, steps (each), dit load, vae, total.
public private(set) var timings: [String: Double] = [:]
+ /// Kept between images: its first decode of a size costs ~3.7 s more than the next ones at 1024² (M2 Pro), GPU
+ /// time (~1 s of CPU), so it cannot hide under the DiT; a long-lived engine pays it once per size.
+ private var vae: QwenVAE?
public init(encoder: String, dit: String, vae: String) throws {
encoderPath = encoder
@@ -35,10 +38,11 @@ public final class QwenPipeline {
}
/// Text to image at `width` × `height` (multiples of 32). `noise`: a latent `[64, h, w]` to start from instead of
- /// the seed's; `step`: called after each step with its index and the packed latent `[h·w, 64]`.
+ /// the seed's; `conditioning`: the encoder's output `[T, 4096]` to use instead of encoding `prompt` (the checks);
+ /// `step`: called after each step with its index and the packed latent `[h·w, 64]`.
/// Returns the decoded image, RGBA `[4, height, width]` in [−1, 1].
public func textToImage(prompt: String, width: Int, height: Int, seed: UInt64, noise: [Float]? = nil,
- sigmas: [Float] = QwenPipeline.turboSigmas,
+ conditioning given: [Float]? = nil, sigmas: [Float] = QwenPipeline.turboSigmas,
step: ((Int, [Float]) -> Void)? = nil) throws -> [Float] {
guard width % 32 == 0, height % 32 == 0, width >= 64, height >= 64 else {
throw QwenSequence.Failure.layout("\(width)×\(height): each side must be a multiple of 32, at least 64")
@@ -52,6 +56,7 @@ public final class QwenPipeline {
// 1. The prompt. The encoder and its 5 GB mapping are gone when this scope ends.
var t = DispatchTime.now().uptimeNanoseconds
let (conditioning, rows): ([Float], Int) = try {
+ if let given { return (given, given.count / QwenDiT.dim) }
let file = try GGUFFile(path: encoderPath)
let encoder = try QwenTextEncoder(file: file, context: context)
defer { encoder.release() }
@@ -74,6 +79,7 @@ public final class QwenPipeline {
let file = try GGUFFile(path: ditPath)
defer { context.release(file) }
let dit = try QwenDiT(file: file, context: context, sequence: sequence)
+ try dit.prepareModulation([0] + sigmas.dropLast())
timings["dit load"] = since(t)
for i in 0..<(sigmas.count - 1) {
let stepStart = DispatchTime.now().uptimeNanoseconds
@@ -88,8 +94,8 @@ public final class QwenPipeline {
// 4. The image.
t = DispatchTime.now().uptimeNanoseconds
- let vae = try QwenVAE(path: vaePath, context: context)
- let image = try vae.decode(QwenPipeline.unpack(x, height: h, width: w), height: h, width: w)
+ if vae == nil { vae = try QwenVAE(path: vaePath, context: context) }
+ let image = try vae!.decode(QwenPipeline.unpack(x, height: h, width: w), height: h, width: w)
timings["vae"] = since(t)
timings["total"] = since(started)
log(String(format: "VAE: %.2f s; total %.1f s", timings["vae"]!, timings["total"]!))
diff --git a/engine-mac/Sources/KokonEngine/QwenVAE.swift b/engine-mac/Sources/KokonEngine/QwenVAE.swift
index 22e97fd..b242a57 100644
--- a/engine-mac/Sources/KokonEngine/QwenVAE.swift
+++ b/engine-mac/Sources/KokonEngine/QwenVAE.swift
@@ -35,7 +35,12 @@ public final class QwenVAE {
let context: MetalContext
let file: SafetensorsFile
- private var graphs: [String: (graph: MPSGraph, input: MPSGraphTensor, output: MPSGraphTensor)] = [:]
+ /// One compiled graph per latent size. Building and compiling the 1024² one takes 0.3 s; its first run then costs
+ /// ~3.7 s more than the next ones (M2 Pro), GPU time with ~1 s of CPU, so the pipeline keeps its VAE between
+ /// images rather than building it beside the DiT. fp16 convolutions (norms and attention in fp32) were wrong
+ /// (2.7e-1: an overflow) and 5.7× slower, MPSGraph trying to move them to the Neural Engine: the VAE stays fp32.
+ private var executables: [String: MPSGraphExecutable] = [:]
+ private let lock = NSLock()
public init(path: String, context: MetalContext) throws {
file = try SafetensorsFile(path: path)
@@ -48,20 +53,33 @@ public final class QwenVAE {
/// Decodes a normalized latent `[64, h, w]` (the DiT's packed tokens transposed) into RGBA `[4, 16h, 16w]` in
/// [−1, 1].
public func decode(_ latent: [Float], height h: Int, width w: Int) throws -> [Float] {
- let key = "\(h)×\(w)"
- let built = try graphs[key] ?? build(height: h, width: w)
- graphs[key] = built
+ let executable = try prepare(height: h, width: w)
let input = MPSGraphTensorData(device: MPSGraphDevice(mtlDevice: context.device),
data: latent.withUnsafeBytes { Data($0) },
shape: [1, 64, NSNumber(value: h), NSNumber(value: w)], dataType: .float32)
- let result = built.graph.run(with: context.queue, feeds: [built.input: input], targetTensors: [built.output],
- targetOperations: nil)
- guard let data = result[built.output] else { throw MetalContext.Failure.kernel("VAE decode") }
+ guard let data = executable.run(with: context.queue, inputs: [input], results: nil, executionDescriptor: nil).first else {
+ throw MetalContext.Failure.kernel("VAE decode")
+ }
var out = [Float](repeating: 0, count: 4 * 16 * h * 16 * w)
out.withUnsafeMutableBytes { data.mpsndarray().readBytes($0.baseAddress!, strideBytes: nil) }
return out
}
+ /// Builds and compiles the decoder for an `h` × `w` latent, once.
+ @discardableResult
+ public func prepare(height h: Int, width w: Int) throws -> MPSGraphExecutable {
+ try lock.withLock {
+ let key = "\(h)×\(w)"
+ if let e = executables[key] { return e }
+ let built = try build(height: h, width: w)
+ let shape = MPSGraphShapedType(shape: [1, 64, NSNumber(value: h), NSNumber(value: w)], dataType: .float32)
+ let e = built.graph.compile(with: MPSGraphDevice(mtlDevice: context.device), feeds: [built.input: shape],
+ targetTensors: [built.output], targetOperations: nil, compilationDescriptor: nil)
+ executables[key] = e
+ return e
+ }
+ }
+
// MARK: - the graph
private func build(height h: Int, width w: Int) throws -> (graph: MPSGraph, input: MPSGraphTensor, output: MPSGraphTensor) {
diff --git a/engine-mac/Sources/KokonEngine/TextChecks.swift b/engine-mac/Sources/KokonEngine/TextChecks.swift
index 6bb16f1..d1a6a4e 100644
--- a/engine-mac/Sources/KokonEngine/TextChecks.swift
+++ b/engine-mac/Sources/KokonEngine/TextChecks.swift
@@ -60,7 +60,9 @@ extension Checks {
extension Checks {
/// Text to image against `tools/reference_generate.py`, from its noise: each step's latent, the decoded image,
/// the 8-bit pixels.
- public static func generate(encoder: String, dit: String, vae: String, reference directory: String) throws {
+ /// `text`: a `reference_text_encoder.py` directory whose conditioning replaces the engine's encoder, to measure
+ /// the DiT and the sampler alone.
+ public static func generate(encoder: String, dit: String, vae: String, reference directory: String, text: String? = nil) throws {
let meta = try JSONSerialization.jsonObject(with: Data(contentsOf: URL(fileURLWithPath: directory)
.appendingPathComponent("meta.json"))) as! [String: Any]
let prompt = meta["prompt"] as! String, grid = meta["grid"] as! [Int]
@@ -71,7 +73,8 @@ extension Checks {
print(String(format: " %-12@ rel %.2e max |Δ| %.3g%@", name as NSString, r.relative, r.maxAbs, r.finite ? "" : " NON-FINITE"))
}
let image = try pipeline.textToImage(prompt: prompt, width: w * 16, height: h * 16, seed: 0,
- noise: try reference(directory, "noise")) { i, latent in
+ noise: try reference(directory, "noise"),
+ conditioning: try text.map { try reference($0, "conditioning") }) { i, latent in
latent.withUnsafeBufferPointer { report("latent\(i)", $0.baseAddress!, try! reference(directory, "latent\(i)")) }
}
image.withUnsafeBufferPointer { report("image", $0.baseAddress!, try! reference(directory, "image")) }
@@ -93,6 +96,27 @@ extension Checks {
}
extension Checks {
+ /// Each step of a `reference_generate.py` run on its own: the DiT on the reference's latent before the step (its
+ /// noise for the first), with the text reference's conditioning, against the reference's velocity. A trajectory
+ /// amplifies rounding from step to step; this measures one evaluation's error on real inputs.
+ public static func steps(dit path: String, reference directory: String, text: String) throws {
+ let meta = try JSONSerialization.jsonObject(with: Data(contentsOf: URL(fileURLWithPath: directory)
+ .appendingPathComponent("meta.json"))) as! [String: Any]
+ let grid = meta["grid"] as! [Int], sigmas = (meta["sigmas"] as! [Double]).map(Float.init)
+ let conditioning = try reference(text, "conditioning")
+ let file = try GGUFFile(path: path)
+ let sequence = try QwenSequence.textToImage(text: conditioning.count / QwenDiT.dim,
+ target: .init(height: grid[0], width: grid[1]))
+ let dit = try QwenDiT(file: file, context: try MetalContext(), sequence: sequence)
+ for i in 0..<(sigmas.count - 1) {
+ let input = i == 0 ? QwenPipeline.pack(try reference(directory, "noise"), height: grid[0], width: grid[1])
+ : try reference(directory, "latent\(i - 1)")
+ let velocity = try dit.forward(encoder: conditioning, target: input, sigma: sigmas[i])
+ let r = velocity.withUnsafeBufferPointer { compare($0.baseAddress!, try! reference(directory, "velocity\(i)")) }
+ print(String(format: " step %d (σ %.4f): velocity rel %.2e max |Δ| %.3g", i + 1, sigmas[i], r.relative, r.maxAbs))
+ }
+ }
+
/// The seed's noise against sd.cpp's own generator (`tools/philox_reference.cpp`): every value, bit for bit.
public static func noise(dump: String) throws {
let data = try Data(contentsOf: URL(fileURLWithPath: dump))
diff --git a/engine-mac/Sources/kokon-engine-mac/main.swift b/engine-mac/Sources/kokon-engine-mac/main.swift
index 0436ea3..6a26109 100644
--- a/engine-mac/Sources/kokon-engine-mac/main.swift
+++ b/engine-mac/Sources/kokon-engine-mac/main.swift
@@ -11,12 +11,16 @@ kokon-engine-mac: Kokon's Apple Silicon engine (in development)
check-vae <vae.safetensors> <reference dir> [grid] the VAE decoder against tools/reference_vae.py
check-tokenizer <encoder.gguf> <fixture.json> the tokenizer against tools/reference_tokenizer.py
check-text <encoder.gguf> <reference dir> the text encoder against tools/reference_text_encoder.py
- check-generate <encoder.gguf> <dit.gguf> <vae.safetensors> <reference dir>
+ check-generate <encoder.gguf> <dit.gguf> <vae.safetensors> <reference dir> [text reference dir]
text to image against tools/reference_generate.py, from its noise
+ (and from the text reference's conditioning when given)
+ check-steps <dit.gguf> <generate reference dir> <text reference dir>
+ each step alone against tools/reference_generate.py (its latents in)
check-noise <philox.bin> the seed's noise against sd.cpp's generator (tools/philox_reference.cpp)
generate <encoder.gguf> <dit.gguf> <vae.safetensors> "<prompt>" [--size WxH] [--seed N] [--out file.png]
text to image, Turbo's 8 steps (default 1024x1024, seed 42, out.png)
probe this Mac's matrix throughput: GPU, AMX, Neural Engine, together
+ probe-attention the target's attention at 1024² on the GPU, the Neural Engine, the CPU
"""
func fail(_ message: String) -> Never {
@@ -66,8 +70,9 @@ do {
guard arguments.count == 3 else { fail(usage) }
try Checks.textEncoder(path: arguments[1], reference: arguments[2])
case "check-generate":
- guard arguments.count == 5 else { fail(usage) }
- try Checks.generate(encoder: arguments[1], dit: arguments[2], vae: arguments[3], reference: arguments[4])
+ guard arguments.count == 5 || arguments.count == 6 else { fail(usage) }
+ try Checks.generate(encoder: arguments[1], dit: arguments[2], vae: arguments[3], reference: arguments[4],
+ text: arguments.count == 6 ? arguments[5] : nil)
case "generate":
guard arguments.count >= 5 else { fail(usage) }
var (width, height, seed, out) = (1024, 1024, UInt64(42), "out.png")
@@ -91,11 +96,16 @@ do {
try QwenPipeline.writePNG(QwenPipeline.pixels(image, width: size.width, height: size.height),
width: size.width, height: size.height, to: out)
print("wrote \(out) (\(size.width)×\(size.height), seed \(seed))")
+ case "check-steps":
+ guard arguments.count == 4 else { fail(usage) }
+ try Checks.steps(dit: arguments[1], reference: arguments[2], text: arguments[3])
case "check-noise":
guard arguments.count == 2 else { fail(usage) }
try Checks.noise(dump: arguments[1])
case "probe":
try Probe.run()
+ case "probe-attention":
+ try Probe.attention()
default:
fail(usage)
}