Edit photos by describing the change, on your own GPU. A free, open-source alternative to Nano Banana and ChatGPT image editing: photos stay on your computer, no daily limit. One installer for Windows, Mac and Linux sets up the best open model (Qwen-Image 2.1). MCP server for AI agents.
TypeScript ai-photo-editorcomfyui-alternativegenerative-fillimage-editingimage-generationinpaintinglinuxlocal-aimacosmcpmcp-servernano-banana-alternativeofflinephoto-editorprivacyqwen-imageqwen-image-2-1stable-diffusion-cppvulkanwindows
Mac engine 0.1.1: release flush chosen by measuring, direct attention off, lower worker priority
Gabriele Lozuponed1955ea06188
diff --git a/crates/qls-core/src/engine.rs b/crates/qls-core/src/engine.rsindex f5b82a2..4424d80 100644--- a/crates/qls-core/src/engine.rs+++ b/crates/qls-core/src/engine.rs@@ -545,7 +545,7 @@ impl Engine {self.emit_server();let report = mb::report_path(&self.data_dir);let args = mb::bench_args(cfg.profile(), &report, quick);- let envs = vec![("KOKON_ENGINE_VERSION".to_string(), format!("0.1 (experimental), Kokon {APP_VERSION}"))];+ let envs = vec![("KOKON_ENGINE_VERSION".to_string(), format!("0.1.1 (experimental), Kokon {APP_VERSION}"))];let command = mb::BenchCommand { exe: cfg.server_exe(), args, envs, report, quick };let (e1, e2) = (self.clone(), self.clone());let r = self.mac_bench.start(diff --git a/engine-mac/README.md b/engine-mac/README.mdindex ec170b7..ff1f724 100644--- a/engine-mac/README.md+++ b/engine-mac/README.md@@ -229,8 +229,9 @@ packed once per layer to fp16, heads first (`attention_pack_kv`, 1.1 ms), and thM1/M2 shape (4 SIMD groups × 8 rows, 128-key tiles, the head in chunks of 32) on the same packed operands: 105 ms;8 × 8 rows with 128-key tiles in chunks of 32: 94; 64-key tiles in chunks of 64: 104; so the kernel keeps 8 × 8rows, 64-key tiles, K and V loaded together. MFA's shapes rest on its async copies, which run-time compiled Metal-cannot use. On Apple9+ GPUs (M3 and later) the engine picks `flash_attention_direct`, each SIMD group reading the-packed K and V straight from memory as MFA's tables prefer there: 142 ms on the M2 Pro, not yet measured on an M3+.+cannot use. `flash_attention_direct` (`KOKON_ATTENTION_DIRECT=1`), each SIMD group reading the packed K and V+straight from memory as MFA's tables prefer on Apple9+ GPUs, is off on every chip since engine 0.1.1: 142 ms on the+M2 Pro, and slower on every M3, M4 and M5 measured (a step at 1024² in the bench reports of [Discussion #43](https://github.com/GabrieleLozupone/kokon/discussions/43): M3 Max 5.45 → 4.83 s without it, M4 Pro 7.27 → 5.96, M4 Max 4.57 → 3.88, M5 Pro 5.79 → 5.34).**Prefix K/V cache** (`KOKON_PREFIX_CACHE=0` turns it off): the prefix (text; later the reference images) sees onlyits own keys and the modulation of t = 0, so its K and V are saved per layer during the first evaluation of a@@ -433,13 +434,20 @@ correctness test that runs on any Mac (`PerChipTests`). Only the M2 Pro has measin fp16, 8.3 ms with int8 weights, 7.0 ms in int8 (`check-ane8`); per evaluation 7.42 → 7.20 / 7.00 s; but thevelocity at 896² goes from 7.6e-4 to 1.1e-2 / 4.1e-2 (one int8 scale per 4 096 weights where Q4_K has one per 32;activation rows with outliers). **Finer scales do not fix it** (2026-10-25, below): int8 stays opt-in.-- **Attention reading K and V straight from memory** (`flash_attention_direct`) on Apple9+ GPUs (M3 and later), as- MFA's tables prefer there (`KOKON_ATTENTION_DIRECT=0` off); 142 ms on the M2 Pro against 91, unmeasured on an M3+.-- **Each product released at once** on the M5 Pro (`KOKON_FLUSH_READY=0` off): a wait on `ready` right after its- signal. On the M5 Pro (macOS 26.6) a signal inside a command buffer reaches the CPU only at the next wait the+- **int8 weights alone** (`KOKON_ANE_INT8=w8`) are broken in a whole generation: on every Mac in the bench reports and+ on the M2 Pro (2026-10-28) the image comes out black (non-finite) at 512² to 1024², with either DiT file, while one tile+ (`check-ane8`, 5.8e-3), `check-dit` at 896² (1.1e-2) and `time-dit` stay finite; turning off the prefix cache, the+ fused kernels, shared rows, the AMX, fp16 activations or Metal 4 does not fix it. Cause not found; out of the bench.+- **Each product released at once** where signals arrive late (`KOKON_FLUSH_READY=0` off, `=1` on): a wait on+ `ready` right after its signal, chosen by a ~50 ms measurement at the engine's start (`Coexecutor.signalArrivesLate`),+ every start, so a macOS update is seen. On the M5 Pro (macOS 26.6) a signal inside a command buffer reaches the CPU only at the next wait thecommand buffer encodes, which was after the GPU's own rows: the AMX and the Neural Engine ran after the GPU instead ofbeside it, and the balancer, reading ~1 ms for the GPU's rows, gave them almost nothing. M5 Pro 20-core, 24 GB,- `time-dit` at 1024², 3 alternated rounds (third evaluation): 5.50–5.89 → 4.44–4.46 s.+ `time-dit` at 1024², 3 alternated rounds (third evaluation): 5.50–5.89 → 4.44–4.46 s. An M2 Max on macOS 26.2 shows+ the same starved shares; the Macs on macOS 27 do not, and there the wait costs ~2.5 % (M2 Pro, macOS 27.2: 6.83 →+ 7.00 s), so it is not on by default.+- **Attention reading K and V straight from memory** (`flash_attention_direct`): off on every chip since engine 0.1.1+ (`KOKON_ATTENTION_DIRECT=1` to try it); a step at 1024² in the bench reports of [Discussion #43](https://github.com/GabrieleLozupone/kokon/discussions/43): M3 Max 5.45 → 4.83 s without it, M4 Pro 7.27 → 5.96, M4 Max 4.57 → 3.88, M5 Pro 5.79 → 5.34.- **Starting shares per chip**: the balancer's M2 Pro rates scaled by the chip's estimated units (GPU cores × clock,the Neural Engine's published TOPS, performance cores; `Chip.unitScale`), so the first image starts near balanced;the loop measures the real rates from the first evaluation on.diff --git a/engine-mac/Sources/KokonEngine/Bench.swift b/engine-mac/Sources/KokonEngine/Bench.swiftindex f46a4cd..59dedb0 100644--- a/engine-mac/Sources/KokonEngine/Bench.swift+++ b/engine-mac/Sources/KokonEngine/Bench.swift@@ -19,8 +19,8 @@ public enum Bench {struct Run { let name: String; let environment: [String: String] }/// The experimental paths this Mac selects, as (what, the setting that turns it off).- static func selectedPaths(device: MTLDevice) -> [(String, String, String)] {- let chip = Chip.current+ static func selectedPaths(context: MetalContext) -> [(String, String, String)] {+ let (chip, device) = (Chip.current, context.device)var paths: [(String, String, String)] = []if Coexecutor.int8Selected(chip), #available(macOS 26, *) {paths.append(("int8 products on the Neural Engine (M3/M4 path)", "KOKON_ANE_INT8", "0"))@@ -28,11 +28,11 @@ public enum Bench {if NeuralAccelerators.selected(device: device), NeuralAccelerators.make(device: device) != nil {paths.append(("GPU products on Metal 4 tensor operations (the M5's neural accelerators; shader cores before)", "KOKON_NA", "0"))}- if Coexecutor.flushReleaseSelected(chipName: chip.name) {- paths.append(("each product released to the AMX and the Neural Engine at once (M5 Pro path)", "KOKON_FLUSH_READY", "0"))+ if Coexecutor.flushReleaseSelected(context) {+ paths.append(("each product released to the AMX and the Neural Engine at once (this Mac's signals arrive late)", "KOKON_FLUSH_READY", "0"))}if QwenDiT.directAttention(device: device) {- paths.append(("attention reading K/V straight from memory (Apple9+ path)", "KOKON_ATTENTION_DIRECT", "0"))+ paths.append(("attention reading K/V straight from memory", "KOKON_ATTENTION_DIRECT", "0"))}return paths}@@ -49,7 +49,7 @@ public enum Bench {public static func run(encoder: String, dit: String, vae: String, quick: Bool, out: String?) throws {let context = try MetalContext()let chip = Chip.current- let paths = selectedPaths(device: context.device)+ let paths = selectedPaths(context: context)print("kokon-engine-mac bench: \(chip)")print("selected: " + (paths.isEmpty ? "the M2 path (nothing chip-specific)" : paths.map(\.0).joined(separator: "; ")))let scratch = FileManager.default.temporaryDirectory.appendingPathComponent("kokon-bench-\(ProcessInfo.processInfo.processIdentifier)")@@ -58,9 +58,10 @@ public enum Bench {var runs = [Run(name: "default (paths above)", environment: [:])]for path in paths { runs.append(Run(name: "\(path.1)=\(path.2)", environment: [path.1: path.2])) }- // The opt-in int8 Neural Engine path where it may pay (libnnc's M3/M4 path; less accurate everywhere).+ // The opt-in int8 Neural Engine path where it may pay (libnnc's M3/M4 path; less accurate everywhere). Not `w8`:+ // its images come out black in a whole generation on every Mac (README, "int8 weights alone").if (chip.generation ?? 5) >= 3, #available(macOS 26, *) {- for mode in ["w8", "1"] { runs.append(Run(name: "KOKON_ANE_INT8=\(mode)", environment: ["KOKON_ANE_INT8": mode])) }+ runs.append(Run(name: "KOKON_ANE_INT8=1", environment: ["KOKON_ANE_INT8": "1"]))}// `PROGRESS k/N <what>` before each child, for Kokon's Benchmark button (stdout is a pipe there).@@ -257,5 +258,5 @@ public enum Bench {/// The engine's version in reports (the bench script adds the commit).public enum EngineVersion {- public static let string = ProcessInfo.processInfo.environment["KOKON_ENGINE_VERSION"] ?? "0.1 (experimental)"+ public static let string = ProcessInfo.processInfo.environment["KOKON_ENGINE_VERSION"] ?? "0.1.1 (experimental)"}diff --git a/engine-mac/Sources/KokonEngine/Coexecution.swift b/engine-mac/Sources/KokonEngine/Coexecution.swiftindex 5977c9b..245c155 100644--- a/engine-mac/Sources/KokonEngine/Coexecution.swift+++ b/engine-mac/Sources/KokonEngine/Coexecution.swift@@ -1,6 +1,7 @@import Accelerateimport Foundationimport Metal+import QuartzCore/// **One product on the GPU, the AMX and the Neural Engine at the same time**, cut by rows:///@@ -87,16 +88,68 @@ final class Coexecutor {/// (a 3 × 20 ms test: the CPU saw the three signals at 60 / 60 / 60 ms, with a wait after each at 20 / 40 / 60)./// Here the next wait is after the GPU's own rows, so the AMX and the Neural Engine started only when the GPU was/// done, one after the other instead of together, and the timeline read the GPU's rows as ~1 ms. The wait is on a- /// value already reached: the GPU does not stop. Measured on the M5 Pro only; `KOKON_FLUSH_READY=0` / `=1`.+ /// value already reached, but it is not free where signals arrive on time: on the M2 Pro (macOS 27.2) an+ /// evaluation 6.83 → 7.00 s. So it is chosen by measuring (`signalArrivesLate`), not by chip: the M5 Pro on+ /// macOS 26.6 and an M2 Max on 26.2 show the late signal, the Macs on macOS 27 did not. `KOKON_FLUSH_READY=0` / `=1`.let flushRelease: Bool- static func flushReleaseSelected(chipName: String) -> Bool {+ static func flushReleaseSelected(_ context: MetalContext) -> Bool {switch ProcessInfo.processInfo.environment["KOKON_FLUSH_READY"] {case "0": return falsecase "1": return true- default: return chipName == "Apple M5 Pro"+ default: return signalArrivesLate(context)}}+ nonisolated(unsafe) private static var lateSignal: Bool?+ private static let lateSignalLock = NSLock()+ /// **Whether this Mac delivers a signal inside a command buffer late** (at the command buffer's next wait or its+ /// end): ~20 ms of GPU work, a signal, ~20 ms more; the CPU sees it about halfway when signals arrive on time, near+ /// the end when they are late. Measured once per process (~50 ms, at the first co-executor), so a macOS update is+ /// seen at the next start. A failed measurement counts as on time (the path before the flush existed).+ static func signalArrivesLate(_ context: MetalContext) -> Bool {+ lateSignalLock.withLock {+ if let late = lateSignal { return late }+ let late = (try? measureLateSignal(context)) ?? false+ lateSignal = late+ return late+ }+ }++ private static func measureLateSignal(_ context: MetalContext) throws -> Bool {+ let pipeline = try context.pipeline("spin")+ let out = try context.scratch(4 << 20, label: "spin")+ func spin(_ commands: MTLCommandBuffer, _ n: UInt32) {+ var n = n+ let e = commands.makeComputeCommandEncoder()!+ e.setComputePipelineState(pipeline); e.setBuffer(out, offset: 0, index: 0); e.setBytes(&n, length: 4, index: 1)+ e.dispatchThreads(MTLSize(width: 1 << 20, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))+ e.endEncoding()+ }+ // Sized to ~20 ms on this GPU.+ var n: UInt32 = 64+ for _ in 0..<3 {+ guard let c = context.queue.makeCommandBuffer() else { return false }+ spin(c, n)+ let seconds = try MetalContext.finish(c, "spin")+ n = UInt32(min(Double(1 << 24), Double(n) * 0.02 / max(seconds, 1e-5)))+ }+ guard let event = context.device.makeSharedEvent(), let commands = context.queue.makeCommandBuffer() else { return false }+ spin(commands, n)+ release(event, 1, flush: false, commands)+ spin(commands, n)+ let seen = DispatchSemaphore(value: 0)+ nonisolated(unsafe) var at = 0.0+ Thread {+ _ = event.wait(untilSignaledValue: 1, timeoutMS: 10_000)+ at = CACurrentMediaTime()+ seen.signal()+ }.start()+ try MetalContext.finish(commands, "late signal probe")+ guard seen.wait(timeout: .now() + 10) == .success else { return false }+ let (start, end) = (commands.gpuStartTime, commands.gpuEndTime)+ return at > end - 0.25 * (end - start)+ }+/// Releases operation `p` to the AMX and the Neural Engine.func release(_ p: UInt64, _ commands: MTLCommandBuffer) {Coexecutor.release(ready, p, flush: flushRelease, commands)@@ -136,7 +189,7 @@ final class Coexecutor {self.amxEnabled = amxaneInt8 = ane && Coexecutor.int8Selected(Chip.current)aneHalfRows = !Coexecutor.int8Rows- flushRelease = Coexecutor.flushReleaseSelected(chipName: Chip.current.name)+ flushRelease = Coexecutor.flushReleaseSelected(context)// The first evaluation's shares from the cost model on this chip's estimated rates (`Chip.unitScale`; on the// M2 Pro its measured ones). Remembered shares replace them (`recall`).balancer = Balancer(aneAttention: ane && aneAttention, chip: Chip.current)@@ -558,7 +611,10 @@ final class Timeline: @unchecked Sendable {}}-/// A thread running submitted jobs in order.+/// A thread running submitted jobs in order, at `.userInitiated` like the engine's own work thread (`EngineServer`):+/// under `.userInteractive`, the level of the apps' interface threads, the AMX's and the CPU's products (BNNS and+/// `concurrentPerform` inherit it) competed with every window on the Mac as equals, and the interface stuttered while+/// an image was made. The priority only matters when the CPU is contended; alone, the engine runs as fast.final class Worker: @unchecked Sendable {private var jobs: [() -> Void] = []private let lock = NSLock(), available = DispatchSemaphore(value: 0)@@ -572,7 +628,7 @@ final class Worker: @unchecked Sendable {}}thread.name = name- thread.qualityOfService = .userInteractive+ thread.qualityOfService = .userInitiatedthread.start()}diff --git a/engine-mac/Sources/KokonEngine/Kernels/ane.metal b/engine-mac/Sources/KokonEngine/Kernels/ane.metalindex 5c1e008..8535c80 100644--- a/engine-mac/Sources/KokonEngine/Kernels/ane.metal+++ b/engine-mac/Sources/KokonEngine/Kernels/ane.metal@@ -11,6 +11,14 @@ using namespace metal;// The product runs in tiles of at most 4 096 × 4 096 (one Core ML program each, the Neural Engine's fast range);// tiles along K are summed here in fp32.+// A few milliseconds of arithmetic per thread and nothing else: the GPU work around the probe's signal+// (`Coexecutor.signalArrivesLate`).+kernel void spin(device float *o [[buffer(0)]], constant uint &n [[buffer(1)]], uint i [[thread_position_in_grid]]) {+ float x = float(i);+ for (uint k = 0; k < n; k++) x = fma(x, 1.0000001f, 0.5f);+ o[i] = x;+}+// fp32 -> fp16 (check-ane).kernel void pack_half(device const float *src [[buffer(0)]], device half *dst [[buffer(1)]],constant uint &count [[buffer(2)]], uint i [[thread_position_in_grid]]) {diff --git a/engine-mac/Sources/KokonEngine/QwenDiT.swift b/engine-mac/Sources/KokonEngine/QwenDiT.swiftindex cf51680..b5bc194 100644--- a/engine-mac/Sources/KokonEngine/QwenDiT.swift+++ b/engine-mac/Sources/KokonEngine/QwenDiT.swift@@ -924,16 +924,14 @@ public final class QwenDiT {struct PackParams { var keys, paddedKeys, stride, heads: UInt32 }/// The target's attention (not causal) on K and V packed first to fp16, heads first: 1024², 32 heads, M2 Pro/// (`probe-attention`, 2026-10-25): 91.0 ms with the packing against 95.2 for `flash_attention_half`, the same- /// accuracy. Apple9+ GPUs (M3 and later) read the packed K and V straight from memory (`flash_attention_direct`),- /// as MFA's and libnnc's tables do there; on the M2 Pro that variant takes 142 ms. Not yet measured on an M3+.+ /// accuracy. `flash_attention_direct` reads the packed K and V straight from memory, as MFA's and libnnc's tables+ /// do on Apple9+ GPUs; it was the default there until the bench reports of Discussion #43 (2026-10-27) found it+ /// slower on every one: a step at 1024², direct against packed, M3 Max 5.45 → 4.83 s, M4 Pro 7.27 → 5.96,+ /// M4 Max 4.57 → 3.88, M5 Pro 5.79 → 5.34 (on the M2 Pro 142 ms against 91 for the attention alone). Off by default+ /// on every chip; `KOKON_ATTENTION_DIRECT=1` to try it.lazy var packedKernel = QwenDiT.directAttention(device: context.device) ? "flash_attention_direct" : "flash_attention_packed"- /// `flash_attention_direct` on Apple9+ GPUs; `KOKON_ATTENTION_DIRECT=0` / `=1` to choose.static func directAttention(device: MTLDevice) -> Bool {- switch ProcessInfo.processInfo.environment["KOKON_ATTENTION_DIRECT"] {- case "0": return false- case "1": return true- default: return device.supportsFamily(.apple9)- }+ ProcessInfo.processInfo.environment["KOKON_ATTENTION_DIRECT"] == "1"}/// Queries `[queries, queries + count)` against keys `[0, keys)`, into the same rows of `attended`. The target'sdiff --git a/engine-mac/Tests/KokonEngineTests/PerChipTests.swift b/engine-mac/Tests/KokonEngineTests/PerChipTests.swiftindex d43b2f9..5aac247 100644--- a/engine-mac/Tests/KokonEngineTests/PerChipTests.swift+++ b/engine-mac/Tests/KokonEngineTests/PerChipTests.swift@@ -107,13 +107,18 @@ extension PerChipTests {}extension PerChipTests {- /// The release made visible at once (`KOKON_FLUSH_READY`): chosen on the M5 Pro only, where it was measured.+ /// The release made visible at once (`KOKON_FLUSH_READY`): chosen by measuring this Mac, once per process.func testFlushReleaseSelection() throws {guard ProcessInfo.processInfo.environment["KOKON_FLUSH_READY"] == nil else { throw XCTSkip("KOKON_FLUSH_READY is set") }- XCTAssertTrue(Coexecutor.flushReleaseSelected(chipName: "Apple M5 Pro"))- for name in ["Apple M1", "Apple M2 Pro", "Apple M3 Max", "Apple M4 Pro", "Apple M5", "Apple M5 Max", "Apple M6 Pro", ""] {- XCTAssertFalse(Coexecutor.flushReleaseSelected(chipName: name), name)- }+ let device = try XCTUnwrap(MTLCreateSystemDefaultDevice())+ if device.name.contains("Paravirtual") { throw XCTSkip("a virtual GPU's timing says nothing") }+ let context = try MetalContext()+ let start = CACurrentMediaTime()+ let late = Coexecutor.signalArrivesLate(context)+ let seconds = CACurrentMediaTime() - start+ print("\(Chip.current): signals arrive \(late ? "late (flush on)" : "on time (flush off)"), measured in \(Int(seconds * 1000)) ms")+ XCTAssertEqual(Coexecutor.flushReleaseSelected(context), late)+ XCTAssertLessThan(seconds, 0.5)}/// With the flush, the CPU sees the release while the GPU's work after it is still running, not at the commanddiff --git a/engine-mac/chips.md b/engine-mac/chips.mdindex abbee9a..8088ab8 100644--- a/engine-mac/chips.md+++ b/engine-mac/chips.md@@ -8,7 +8,7 @@ Before working on a chip, look here and in [docs/mac-engine.md](docs/mac-engine.| Chip | Mac (model, cooling) | Memory | macOS | Paths on by default | 1024² text to image | Velocity error (default) | Source ||---|---|---|---|---|---|---|---|| M2 Pro, 19-core GPU | MacBook Pro (Mac14,10), fans | 16 GB | 27 | Metal 4 products (7.57 → 7.48 s per evaluation); direct attention off (142 ms vs 91) | ~64–67 s; 1 MP edit ~106 s, peak 5 GB | ~6–7 · 10⁻⁴ | [README.md](README.md), [docs/mac-engine.md](docs/mac-engine.md) chapters 26, 28, 30 |-| M5 Pro, 20-core GPU | MacBook Pro 16" (Mac17,8), fans | 24 GB | 26.6.2 | Metal 4 products; direct attention; each product released at once (`KOKON_FLUSH_READY`) | ~40 s (test set, median of 3; master 49.0 s); edit ~43 s, peak 4.3–4.6 GB | 3.1 · 10⁻⁴ (bench, vs the engine's fp32 path) | [README.md](README.md) "Per-chip paths", [docs/mac-engine.md](docs/mac-engine.md) chapters 13, 26 |+| M5 Pro, 20-core GPU | MacBook Pro 16" (Mac17,8), fans | 24 GB | 26.6.2 | Metal 4 products; each product released at once (signals arrive late: macOS 26) | ~40 s (test set, median of 3; master 49.0 s); edit ~43 s, peak 4.3–4.6 GB | 3.1 · 10⁻⁴ (bench, vs the engine's fp32 path) | [README.md](README.md) "Per-chip paths", [docs/mac-engine.md](docs/mac-engine.md) chapters 13, 26 |## Per-chip notes@@ -20,9 +20,10 @@ Before working on a chip, look here and in [docs/mac-engine.md](docs/mac-engine.### M5 Pro, 20-core GPU, 24 GB-- A signal inside a command buffer reaches the CPU only at the command buffer's next wait (chapter 13): before+- A signal inside a command buffer reaches the CPU only at the command buffer's next wait (chapter 13; macOS 26, not+ the chip: an M2 Max on 26.2 too, the Macs on 27 not; measured at every start since 0.1.1): before`KOKON_FLUSH_READY` the AMX and the Neural Engine ran after the GPU's rows, and the balancer gave them almost nothing.With it, test set (3 alternated rounds, median): 1024² 49.0 → 39.6 s, 1344×768 48.3 → 39.9 s, edit 51.6 → 43.0 s.- With the release fixed, `bench` still shows `KOKON_NA=0` (3.92 s per step) and `KOKON_ATTENTION_DIRECT=0` (3.86 s)- ahead of the default (4.08 s): one run each, not yet confirmed with alternated rounds.-- `KOKON_ANE_INT8=w8` gives a NaN velocity and a wrong image (192 levels mean) on this Mac; `=1` works (1.7 · 10⁻²).+ ahead of the default (4.08 s): one run each. Direct attention is off on every chip since 0.1.1.+- `KOKON_ANE_INT8=w8` gives a NaN velocity and a black image here, as on every Mac (README, "int8 weights alone"); `=1` works (1.7 · 10⁻²).diff --git a/engine-mac/docs/mac-engine.md b/engine-mac/docs/mac-engine.mdindex f09c046..060fcf6 100644--- a/engine-mac/docs/mac-engine.md+++ b/engine-mac/docs/mac-engine.md@@ -653,10 +653,13 @@ read it. That is why there are two reserves and two sets of Neural Engine prograthe next wait the command buffer encodes, or at its end: GPU work of 3 × 20 ms with a signal after each, the CPU sawthem at 60 / 60 / 60 ms, and with a wait after each signal at 20 / 40 / 60. In the sequence above the first wait comesafter the GPU's rows, so there the AMX and the Neural Engine started only when the GPU was done, and the timeline read-the GPU's rows as ~1 ms (the balancer then gave the AMX nothing and the CPU no heads). On that chip `release` encodes a+the GPU's rows as ~1 ms (the balancer then gave the AMX nothing and the CPU no heads). An M2 Max on macOS 26.2 showed+the same; the Macs on macOS 27 did not. So the engine measures it at every start (`Coexecutor.signalArrivesLate`,+~50 ms: ~20 ms of GPU work, a signal, ~20 ms more, and when the CPU sees it), and where it is late `release` encodes await on `ready` right after its signal, on a value already reached ([chapter 26](#26-per-chip-paths),-`KOKON_FLUSH_READY`). `PerChipTests` checks on every Mac that the flushed release reaches the CPU before the command-buffer's end.+`KOKON_FLUSH_READY`). Where signals arrive on time the wait is not free (M2 Pro, macOS 27.2: an evaluation 6.83 →+7.00 s), so it is not used there. `PerChipTests` checks on every Mac that the flushed release reaches the CPU before+the command buffer's end.## 14. The Neural Engine in detail@@ -839,9 +842,10 @@ fp32: 94 ms against 120 for the fp32 kernel. The velocity stays within 6 · 10keys padded to 64, in 1.1 ms. Each threadgroup then reads half the bytes, contiguous and unconverted:`flash_attention_packed`, 91 ms against 95.-**On M3 and later** (Apple9+ GPUs) the engine picks `flash_attention_direct`: each SIMD group reads the packed K and V-straight from memory, with no threadgroup tiles and no barriers, as Metal FlashAttention's tables prefer on those GPUs.-On the M2 Pro it takes 142 ms, so it is chosen only from Apple9 on. Not yet measured on an M3 or later.+**`flash_attention_direct`** (`KOKON_ATTENTION_DIRECT=1`): each SIMD group reads the packed K and V straight from+memory, with no threadgroup tiles and no barriers, as Metal FlashAttention's tables prefer on Apple9+ GPUs. It was the+default from Apple9 on until engine 0.1.1; the bench reports showed it slower on every M3, M4 and M5 measured+(a step at 1024² in the bench reports of [Discussion #43](https://github.com/GabrieleLozupone/kokon/discussions/43): M3 Max 5.45 → 4.83 s without it, M4 Pro 7.27 → 5.96, M4 Max 4.57 → 3.88, M5 Pro 5.79 → 5.34), and on the M2 Pro it takes 142 ms against 91. Off on every chip.A note in the kernel's comments that cost a day: with SIMD-group barriers only, the lanes' writes to threadgroup memorywere sometimes not seen by the loads that followed (1 run in 4 wrong). The kernel uses threadgroup barriers.@@ -1271,9 +1275,9 @@ any Mac.**| Path | Chosen | M2 Pro result | Switch ||---|---|---|---|| **GPU rows on Metal 4 tensor operations** (`na_matmul`, MetalPerformancePrimitives' `matmul2d`): on the M5 the GPU cores' matrix units, before it the shader cores | wherever Metal 4 runs | 6.3 vs MPS 5.6 TFLOP/s, bit for bit the same; an evaluation 7.57 → 7.48 s | `KOKON_NA=0` |-| **Attention reading K/V from memory** (`flash_attention_direct`) | Apple9+ GPUs (M3 and later) | 142 ms vs 91 (so not on M2) | `KOKON_ATTENTION_DIRECT=0/1` |+| **Attention reading K/V from memory** (`flash_attention_direct`) | off on every chip (since 0.1.1) | slower on every Mac measured (M2 Pro 142 ms vs 91; M3 Max, M4 Pro, M4 Max, M5 Pro: 8–18 % per step) | `KOKON_ATTENTION_DIRECT=1` || **int8 products on the Neural Engine** (libnnc's M3/M4 path) | opt-in on every chip | a tile 10.1 → 7.0 ms, but 15–50× the error | `KOKON_ANE_INT8=1` or `w8` |-| **Each product released at once** (`Coexecutor.release`: a wait on `ready` right after its signal, so the CPU sees it before the GPU's own rows) | the M5 Pro only (`Chip.name`) | not selected (M5 Pro: an evaluation 5.50–5.89 → 4.44–4.46 s) | `KOKON_FLUSH_READY=0/1` |+| **Each product released at once** (`Coexecutor.release`: a wait on `ready` right after its signal, so the CPU sees it before the GPU's own rows) | where signals arrive late, measured at every start (`signalArrivesLate`: M5 Pro on macOS 26.6, M2 Max on 26.2) | M5 Pro: an evaluation 5.50–5.89 → 4.44–4.46 s; where on time it costs ~2.5 % (M2 Pro, macOS 27.2) | `KOKON_FLUSH_READY=0/1` || **Starting shares** scaled by the chip's estimated units | always | — | `KOKON_BALANCE=0` keeps the defaults |Only the M2 Pro has measured these. `bench` exists to measure them on the Macs we do not have.@@ -1452,8 +1456,8 @@ All are environment variables read once at start. Defaults are what users run.| `KOKON_WEIGHTS` | fp16 | `fp32`: unpack weights to fp32 (checks) || `KOKON_ACTIVATIONS` | fp16 | `fp32`: fp32 inputs to the GPU's and AMX's products (checks) || `KOKON_ATTENTION` | fp16 packed kernel | `fp32`: fp32 kernel; `graph`: MPSGraph; `unpacked`: no K/V packing |-| `KOKON_ATTENTION_DIRECT` | on from Apple9 | `0`/`1`: choose the direct-load attention |-| `KOKON_FLUSH_READY` | on the M5 Pro | `0`/`1`: a wait right after each release, so the AMX and the Neural Engine see it at once |+| `KOKON_ATTENTION_DIRECT` | off | `1`: the direct-load attention |+| `KOKON_FLUSH_READY` | where signals arrive late (measured at start) | `0`/`1`: a wait right after each release, so the AMX and the Neural Engine see it at once || `KOKON_FUSED_CAST`, `KOKON_FUSED_PACK`, `KOKON_FUSED_RESIDUAL`, `KOKON_SHARED_ROWS` | on | `0`: the separate passes, for comparison || `KOKON_PREFIX_CACHE` | fp16 or q8 | `0`: off; `fp32`, `fp16`, `q8`: choose the format || `KOKON_PREFIX_CACHE_MB` | 1536 below 32 GB, memory/8 above | the cache's limit |diff --git a/engine-mac/tools/bench.sh b/engine-mac/tools/bench.shindex a59377e..3a5a21b 100755--- a/engine-mac/tools/bench.sh+++ b/engine-mac/tools/bench.sh@@ -11,4 +11,4 @@ here="$(cd "$(dirname "$0")/.." && pwd)"(cd "$here" && swift build -c release 2>&1 | tail -1)commit="$(git -C "$here" rev-parse --short HEAD 2>/dev/null || echo unknown)"out="$PWD/kokon-bench-report.md"-KOKON_ENGINE_VERSION="0.1 (experimental), commit $commit" "$here/.build/release/kokon-engine-mac" bench --out "$out" "$@"+KOKON_ENGINE_VERSION="0.1.1 (experimental), commit $commit" "$here/.build/release/kokon-engine-mac" bench --out "$out" "$@"