Edit photos by describing the change, on your own GPU. A free, open-source alternative to Nano Banana and ChatGPT image editing: photos stay on your computer, no daily limit. One installer for Windows, Mac and Linux sets up the best open model (Qwen-Image 2.1). MCP server for AI agents.
TypeScript ai-photo-editorcomfyui-alternativegenerative-fillimage-editingimage-generationinpaintinglinuxlocal-aimacosmcpmcp-servernano-banana-alternativeofflinephoto-editorprivacyqwen-imageqwen-image-2-1stable-diffusion-cppvulkanwindows
Mac engine: row kernels pack the Neural Engine's rows, residual fused with the LN after it
Gabriele Lozupone840fa90baa33
diff --git a/engine-mac/README.md b/engine-mac/README.mdindex 4df9feb..f0d8a2a 100644--- a/engine-mac/README.md+++ b/engine-mac/README.md@@ -246,6 +246,18 @@ worse). Between operations 0.60 → 0.53 s, but per evaluation, 3 alternated rouinstead of 66). With the three units at their shared limit, the GPU-alone time moved into the products; what paysis fewer bytes moved, as with the fp16 cast above.+**The row kernels pack the Neural Engine's rows, and the residual goes with the LN after it** (2026-10-26;+`KOKON_FUSED_PACK=0`, `KOKON_FUSED_RESIDUAL=0` for the comparison). From the second evaluation (every weight's L known):+the LNs and the SwiGLU compute the Neural Engine's rows' σ themselves and write them scaled and packed into its input+surfaces (`layernorm_modulate_ane`, `swiglu_ane`), where they wrote fp32 rows that `ane_row_scales` and `ane_pack_rows`+read back; gate's and proj's Neural Engine rows stay in their output surfaces (each program set has its own now) and the+SwiGLU reads them there instead of two fp32 unpackings; the residual after to_out and the one after the MLP out (carried+into the next layer's LN1) run in the LN's pass (`residual_layernorm`, `residual_layernorm_ane`), x read once and the+Neural Engine's rows of `o` taken from the product's surfaces. The velocity is the same to the bit (1024², fixed shares,+three evaluations, `KOKON_DUMP=<file> time-dit` with each switch; `check-generate` unchanged to every digit). Between operations an evaluation: 0.60–0.66 → 0.41–0.45+s (proj → MLP out 5.4 → 3.3 ms, gate → proj 1.3–2.6 → 0.07, to_out → gate 3.3 → 1.7–2.0, MLP out → q 3.4 → 2.5). Per+evaluation at 1024², 5 alternated rounds of 4 (M2 Pro, other apps open, ~4.5 GB of page-ins a run): median 7.71 → 7.34 s.+**The Neural Engine's rows packed once per input** (`KOKON_SHARED_ROWS=0` for the comparison): q, k and v read thesame rows, and so do gate and proj, but each product scaled and packed them again with its own σ. From the secondevaluation on (when every weight's L is known), an input's rows get one σ from the largest L of its readers and arediff --git a/engine-mac/Sources/KokonEngine/Checks.swift b/engine-mac/Sources/KokonEngine/Checks.swiftindex 58c8e0b..74c885a 100644--- a/engine-mac/Sources/KokonEngine/Checks.swift+++ b/engine-mac/Sources/KokonEngine/Checks.swift@@ -128,6 +128,9 @@ extension Checks {let started = DispatchTime.now().uptimeNanosecondslet out = try dit.forward(encoder: encoder, target: latent, sigma: 0.5)let seconds = Double(DispatchTime.now().uptimeNanoseconds - started) * 1e-9+ if let dump = ProcessInfo.processInfo.environment["KOKON_DUMP"] { // each evaluation's velocity, to compare builds+ try out.withUnsafeBytes { try Data($0).write(to: URL(fileURLWithPath: "\(dump).\(run + 1)")) }+ }print(String(format: "evaluation %d: %.2f s%@%@", run + 1, seconds, dit.timings["prefix"].map { String(format: " (prefix pass %.2f s)", $0) } ?? "",out.allSatisfy(\.isFinite) ? "" : " NON-FINITE OUTPUT"))if let c = dit.coexecutor {diff --git a/engine-mac/Sources/KokonEngine/Coexecution.swift b/engine-mac/Sources/KokonEngine/Coexecution.swiftindex ccb55da..308c36a 100644--- a/engine-mac/Sources/KokonEngine/Coexecution.swift+++ b/engine-mac/Sources/KokonEngine/Coexecution.swift@@ -30,8 +30,9 @@ final class Coexecutor {var amxRows: [String: Int] = [:]typealias ProductSlot = (product: NeuralEngineProduct, x: MTLBuffer, w: MTLBuffer, y: MTLBuffer)/// Two sets of the three tile programs, alternating product by product: the next product's weight is packed into- /// one set while the Neural Engine reads the other (`prepareWeight`). The sets share their row surfaces.- private var slots: [[ProductSlot]]+ /// one set while the Neural Engine reads the other (`prepareWeight`). The sets share their input surfaces; each has+ /// its own outputs, so gate's stay beside proj's for the SwiGLU (`keepOutput`).+ private(set) var slots: [[ProductSlot]]/// Programs of other row counts, kept while the balancing loop may come back to them.private var spareSlots: [Int: [[ProductSlot]]] = [:]/// The set of the last product encoded; the weight packed ahead and its set.@@ -148,9 +149,9 @@ final class Coexecutor {}let second: [ProductSlot] = try first.map { f inlet p = try NeuralEngineProduct(rows: rows, k: Coexecutor.tile, n: Coexecutor.tile, int8: aneInt8, halfActivations: aneHalfRows,- sharing: f.product)+ sharing: f.product, sharingOutput: false)try p.run()- return (p, f.x, try p.gpuBuffer(p.w, device: device), f.y)+ return (p, f.x, try p.gpuBuffer(p.w, device: device), try p.gpuBuffer(p.y, device: device))}return [first, second]}@@ -190,13 +191,46 @@ final class Coexecutor {return (aHalf, m - split(m: m, k: k, n: n).ane)}+ /// **The row kernel before a split product packs its Neural Engine rows** (`layernorm_modulate_ane`, `swiglu_ane`):+ /// σ and the scaled fp16 rows straight into the input surfaces, instead of fp32 rows that `ane_row_scales` and+ /// `ane_pack_rows` read back. Once every reader's L is known (from the first evaluation on), fp16, `sharedRows` for+ /// several readers. `KOKON_FUSED_PACK=0`: the separate passes, for comparison.+ struct RowPack {+ let readers: [String], surfaces: [MTLBuffer], bounds: [MTLBuffer], boundOffset: Int, sigma: MTLBuffer+ /// The first Neural Engine row (from the product's first row), their count, K split into the tiles (MLP out).+ let first: Int, rows: Int, kSplit: Bool+ }+ let fusedPack = ProcessInfo.processInfo.environment["KOKON_FUSED_PACK"] != "0"+ /// The input (A's offset) a single-reader product finds packed by its row kernel.+ var prepacked: Int?++ func rowPack(m: Int, k: Int, n: Int, readers: [String]) -> RowPack? {+ guard fusedPack, aneEnabled, !aneInt8, halfActivations, !readers.isEmpty, readers.count == 1 || sharedRows,+ applies(m: m, k: k, n: n), m <= maxRows, m >= rows, k == Coexecutor.tile || n == Coexecutor.tile else { return nil }+ let s = split(m: m, k: k, n: n)+ let bounds = readers.compactMap { columnScales[$0] }+ guard s.ane > 0, bounds.count == readers.count else { return nil }+ let tiles = k / Coexecutor.tile * (n / Coexecutor.tile)+ return RowPack(readers: readers, surfaces: slots[0].prefix(tiles).map(\.x), bounds: bounds, boundOffset: 2 * n * 4, sigma: scales,+ first: m - s.ane, rows: s.ane, kSplit: k > Coexecutor.tile)+ }++ /// The set of the last product encoded, whose output surfaces hold its Neural Engine rows (`keepOutput`).+ var lastSet: Int { currentSet }+ func columns(_ weight: String) -> MTLBuffer? { columnScales[weight] }++ /// Notes that the rows at `offset` were packed by their row kernel for `readers`.+ func packedByRowKernel(offset: Int, readers: [String]) {+ if readers.count > 1 { packedRows = (offset, readers) } else { prepacked = offset }+ }+/// Encodes the split product of the weight named `weight`. `gpuRows` encodes the GPU's own rows `[0, g)` into the/// command buffer.func encode(weight: String, a: (buffer: MTLBuffer, offset: Int, pointer: UnsafeMutablePointer<Float>),w: (buffer: MTLBuffer, offset: Int, pointer: UnsafeMutablePointer<Float>),c: (buffer: MTLBuffer, offset: Int, pointer: UnsafeMutablePointer<Float>),m: Int, k: Int, n: Int, commands: MTLCommandBuffer, readers: [String] = [],- castReady: Bool = false, ahead: ((MTLCommandBuffer) throws -> Void)? = nil,+ castReady: Bool = false, keepOutput: Bool = false, ahead: ((MTLCommandBuffer) throws -> Void)? = nil,gpuRows: (Int, MTLBuffer?) throws -> Void) throws {let s = split(m: m, k: k, n: n)counter += 1@@ -232,6 +266,8 @@ final class Coexecutor {let shared = sharedRows && readers.count > 1 && readers.count <= 3 && readers.contains(weight)&& (aneInt8 || readers.allSatisfy { columnScales[$0] != nil })let packed = shared && packedRows.map { $0.offset == a.offset && $0.readers == readers } ?? false+ || prepacked == a.offset+ prepacked = nilif s.ane > 0, !packed {var sh = shape()let bounds = shared && !aneInt8 ? readers.map { columnScales[$0]! } : [columns] // L: fp16 only@@ -306,6 +342,8 @@ final class Coexecutor {if s.amx > 0 { commands.encodeWaitForEvent(amxDone, value: p) }if s.ane > 0 {commands.encodeWaitForEvent(aneDone, value: p)+ // Left in its set's output surfaces for the kernel that reads them (gate and proj for the SwiGLU).+ if keepOutput { return }for np in 0..<nParts {var t = shape(np: np)let ys = (0..<3).map { kp in slots[min(kp, kParts - 1) * nParts + np].y }diff --git a/engine-mac/Sources/KokonEngine/Kernels/dit.metal b/engine-mac/Sources/KokonEngine/Kernels/dit.metalindex 66b361f..00eae1c 100644--- a/engine-mac/Sources/KokonEngine/Kernels/dit.metal+++ b/engine-mac/Sources/KokonEngine/Kernels/dit.metal@@ -185,3 +185,223 @@ kernel void widen_q8(device float4 *dst [[buffer(0)]], device const char4 *codesconstant uint &lanes [[buffer(4)]], uint i [[thread_position_in_grid]]) {if (i / 32 < units) dst[i] = float4(codes[i]) * scales[i / lanes];}++// The Neural Engine's rows of a row kernel's output, written straight into the next split product's input surfaces+// (`Coexecution.swift`, `RowPack`): each row's σ from its largest |value| and the readers' bound L, as `ane_row_scales`+// computes it, then half(v · σ) as `ane_pack_rows` writes it, into `surfaces` tiles of 4 096 columns (the same row+// into each of an N-split product's tiles; a K-split product's parts in turn). Rows from `real` on are the product's+// padding: zeros (σ 1), as packing never-written rows gave.+struct AnePack {+ uint row0; // the first Neural Engine row, from the kernel's first row+ uint real; // rows of the kernel that are real, from its first row+ uint surfaces; // tiles written: 1 to 3+ uint kSplit; // 1: column j goes to tile j / 4096 (K parts); 0: every tile gets the whole row (N parts)+};++static inline float row_max(float v, threadgroup float *shared, uint tid) {+ v = simd_max(v);+ if (tid % 32 == 0) shared[tid / 32] = v;+ threadgroup_barrier(mem_flags::mem_threadgroup);+ float m = 0;+ for (uint i = 0; i < ROW_THREADS / 32; i++) m = max(m, shared[i]);+ threadgroup_barrier(mem_flags::mem_threadgroup);+ return m;+}++static inline float ane_sigma(float peak, device const float *l0, device const float *l1, device const float *l2) {+ float bound = peak * max(l0[0], max(l1[0], l2[0]));+ return (bound > 0 && isfinite(bound)) ? exp2(floor(log2(32768.0f / bound))) : 1.0f;+}++// layernorm_modulate on the Neural Engine's rows (4 096 columns), packed. One threadgroup per row.+kernel void layernorm_modulate_ane(device const float *x [[buffer(0)]], device const float *prefixMod [[buffer(1)]],+ device const float *targetMod [[buffer(2)]], constant NormParams &p [[buffer(3)]],+ constant AnePack &a [[buffer(4)]], device float *sigma [[buffer(5)]],+ device const float *l0 [[buffer(6)]], device const float *l1 [[buffer(7)]],+ device const float *l2 [[buffer(8)]], device half *s0 [[buffer(9)]],+ device half *s1 [[buffer(10)]], device half *s2 [[buffer(11)]],+ uint i [[threadgroup_position_in_grid]], uint tid [[thread_index_in_threadgroup]]) {+ threadgroup float shared[ROW_THREADS / 32];+ uint row = a.row0 + i;+ device half *out[3] = {s0 + (ulong)i * 4096, s1 + (ulong)i * 4096, s2 + (ulong)i * 4096};+ if (row >= a.real) {+ for (uint t = 0; t < a.surfaces; t++) for (uint j = tid; j < 4096; j += ROW_THREADS) out[t][j] = 0;+ if (tid == 0) sigma[i] = 1;+ return;+ }+ device const float *r = x + (ulong)row * p.columns;+ float s = 0;+ for (uint j = tid; j < p.columns; j += ROW_THREADS) s += r[j];+ float mean = row_sum(s, shared, tid) / float(p.columns);+ float q = 0;+ for (uint j = tid; j < p.columns; j += ROW_THREADS) { float c = r[j] - mean; q += c * c; }+ float inv = rsqrt(row_sum(q, shared, tid) / float(p.columns) + p.eps);+ device const float *m = row < p.targetStart ? prefixMod : targetMod;+ float v[16], peak = 0;+ for (uint c = 0; c < 16; c++) {+ uint j = tid + c * ROW_THREADS;+ v[c] = ((r[j] - mean) * inv) * m[j];+ peak = max(peak, fabs(v[c]));+ }+ float sg = ane_sigma(row_max(peak, shared, tid), l0, l1, l2);+ if (tid == 0) sigma[i] = sg;+ for (uint c = 0; c < 16; c++) {+ half h = half(v[c] * sg);+ for (uint t = 0; t < a.surfaces; t++) out[t][tid + c * ROW_THREADS] = h;+ }+}++// swiglu on the Neural Engine's rows (12 288 columns: the MLP out's K parts), packed. One threadgroup per row.+// Padding rows pack the gate's value, which is what `swiglu` leaves in them.+kernel void swiglu_ane(device const float *gate [[buffer(0)]], device const float *proj [[buffer(1)]],+ constant AnePack &a [[buffer(2)]], device float *sigma [[buffer(3)]],+ device const float *l0 [[buffer(4)]], device const float *l1 [[buffer(5)]],+ device const float *l2 [[buffer(6)]], device half *s0 [[buffer(7)]],+ device half *s1 [[buffer(8)]], device half *s2 [[buffer(9)]],+ uint i [[threadgroup_position_in_grid]], uint tid [[thread_index_in_threadgroup]]) {+ threadgroup float shared[ROW_THREADS / 32];+ uint row = a.row0 + i;+ device const float *g = gate + (ulong)row * 12288, *pr = proj + (ulong)row * 12288;+ float v[48], peak = 0;+ for (uint c = 0; c < 48; c++) {+ uint j = tid + c * ROW_THREADS;+ float x = g[j];+ v[c] = row < a.real ? (x / (1.0f + exp(-x))) * pr[j] : x;+ peak = max(peak, fabs(v[c]));+ }+ float sg = ane_sigma(row_max(peak, shared, tid), l0, l1, l2);+ if (tid == 0) sigma[i] = sg;+ device half *out[3] = {s0 + (ulong)i * 4096, s1 + (ulong)i * 4096, s2 + (ulong)i * 4096};+ for (uint c = 0; c < 48; c++) {+ uint j = tid + c * ROW_THREADS;+ out[j / 4096][j % 4096] = half(v[c] * sg);+ }+}++// swiglu_ane reading gate's and proj's Neural Engine outputs where the products left them (`keepOutput`): each+// value unscaled as `ane_unpack` does ((y / λ) / σ, σ the rows' shared one, then replaced by the MLP out's), the gate+// tiles `g0..g2` and proj tiles `p0..p2` of 4 096 columns each. Value for value the unpacks, `swiglu`, then the packing.+kernel void swiglu_ane_tiles(device const half *g0 [[buffer(0)]], device const half *g1 [[buffer(1)]],+ device const half *g2 [[buffer(2)]], device const half *p0 [[buffer(3)]],+ device const half *p1 [[buffer(4)]], device const half *p2 [[buffer(5)]],+ device const float *lambdaG [[buffer(6)]], device const float *lambdaP [[buffer(7)]],+ constant AnePack &a [[buffer(8)]], device float *sigma [[buffer(9)]],+ device const float *l0 [[buffer(10)]], device const float *l1 [[buffer(11)]],+ device const float *l2 [[buffer(12)]], device half *s0 [[buffer(13)]],+ device half *s1 [[buffer(14)]], device half *s2 [[buffer(15)]],+ uint i [[threadgroup_position_in_grid]], uint tid [[thread_index_in_threadgroup]]) {+ threadgroup float shared[ROW_THREADS / 32];+ uint row = a.row0 + i;+ float sgIn = sigma[i]; // read before the barriers in row_max; thread 0 rewrites it after them+ device const half *gs[3] = {g0, g1, g2}, *ps[3] = {p0, p1, p2};+ float v[48], peak = 0;+ for (uint c = 0; c < 48; c++) {+ uint j = tid + c * ROW_THREADS, t = j / 4096;+ ulong at = (ulong)i * 4096 + j % 4096;+ float x = (float(gs[t][at]) / lambdaG[j]) / sgIn;+ v[c] = row < a.real ? (x / (1.0f + exp(-x))) * ((float(ps[t][at]) / lambdaP[j]) / sgIn) : x;+ peak = max(peak, fabs(v[c]));+ }+ float sg = ane_sigma(row_max(peak, shared, tid), l0, l1, l2);+ if (tid == 0) sigma[i] = sg;+ device half *out[3] = {s0 + (ulong)i * 4096, s1 + (ulong)i * 4096, s2 + (ulong)i * 4096};+ for (uint c = 0; c < 48; c++) {+ uint j = tid + c * ROW_THREADS;+ out[j / 4096][j % 4096] = half(v[c] * sg);+ }+}++// gated_residual then layernorm_modulate in one pass over each row: x += tanh(gate) · o (written back), then the+// norm of the new x, as the two kernels compute them. Rows from `real` on are padding: no residual, zeros in the fp16+// copy. One threadgroup per row, 4 096 columns.+kernel void residual_layernorm(device float *x [[buffer(0)]], device float *out [[buffer(1)]],+ device const float *prefixMod [[buffer(2)]], device const float *targetMod [[buffer(3)]],+ constant NormParams &p [[buffer(4)]], device half *outHalf [[buffer(5)]],+ device const float *o [[buffer(6)]], device const float *prefixGate [[buffer(7)]],+ device const float *targetGate [[buffer(8)]],+ uint row [[threadgroup_position_in_grid]], uint tid [[thread_index_in_threadgroup]]) {+ threadgroup float shared[ROW_THREADS / 32];+ device half *h = outHalf + (ulong)row * p.columns;+ if (row >= p.half_out.rows) {+ for (uint j = tid; j < p.columns; j += ROW_THREADS) h[j] = 0;+ return;+ }+ device float *r = x + (ulong)row * p.columns;+ device const float *orow = o + (ulong)row * p.columns;+ device const float *g = row < p.targetStart ? prefixGate : targetGate;+ float v[16], s = 0;+ for (uint c = 0; c < 16; c++) {+ uint j = tid + c * ROW_THREADS;+ v[c] = r[j] + g[j] * orow[j];+ r[j] = v[c];+ s += v[c];+ }+ float mean = row_sum(s, shared, tid) / float(p.columns);+ float q = 0;+ for (uint c = 0; c < 16; c++) { float d = v[c] - mean; q += d * d; }+ float inv = rsqrt(row_sum(q, shared, tid) / float(p.columns) + p.eps);+ device const float *m = row < p.targetStart ? prefixMod : targetMod;+ device float *w = out + (ulong)row * p.columns;+ bool toHalf = row < p.half_out.halfRows, toFull = row >= p.half_out.fullFrom;+ for (uint c = 0; c < 16; c++) {+ uint j = tid + c * ROW_THREADS;+ float y = ((v[c] - mean) * inv) * m[j];+ if (toFull) w[j] = y;+ if (toHalf) h[j] = half(y);+ }+}++// The same on the Neural Engine's rows, o read from the product's output tiles (`y0..y2`: its K parts, summed in+// fp32 and unscaled as `ane_unpack` does, σ the product's, then replaced by the next one's), the norm packed as in+// layernorm_modulate_ane.+kernel void residual_layernorm_ane(device float *x [[buffer(0)]], device const float *prefixMod [[buffer(1)]],+ device const float *targetMod [[buffer(2)]], constant NormParams &p [[buffer(3)]],+ constant AnePack &a [[buffer(4)]], device float *sigma [[buffer(5)]],+ device const float *l0 [[buffer(6)]], device const float *l1 [[buffer(7)]],+ device const float *l2 [[buffer(8)]], device half *s0 [[buffer(9)]],+ device half *s1 [[buffer(10)]], device half *s2 [[buffer(11)]],+ device const half *y0 [[buffer(12)]], device const half *y1 [[buffer(13)]],+ device const half *y2 [[buffer(14)]], device const float *lambda [[buffer(15)]],+ constant uint &kParts [[buffer(16)]], device const float *prefixGate [[buffer(17)]],+ device const float *targetGate [[buffer(18)]],+ uint i [[threadgroup_position_in_grid]], uint tid [[thread_index_in_threadgroup]]) {+ threadgroup float shared[ROW_THREADS / 32];+ uint row = a.row0 + i;+ device half *out[3] = {s0 + (ulong)i * 4096, s1 + (ulong)i * 4096, s2 + (ulong)i * 4096};+ float sgIn = sigma[i]; // read before the barriers; thread 0 rewrites it after them+ if (row >= a.real) {+ for (uint t = 0; t < a.surfaces; t++) for (uint j = tid; j < 4096; j += ROW_THREADS) out[t][j] = 0;+ threadgroup_barrier(mem_flags::mem_threadgroup);+ if (tid == 0) sigma[i] = 1;+ return;+ }+ device float *r = x + (ulong)row * p.columns;+ device const float *g = row < p.targetStart ? prefixGate : targetGate;+ float v[16], s = 0;+ for (uint c = 0; c < 16; c++) {+ uint j = tid + c * ROW_THREADS;+ ulong at = (ulong)i * 4096 + j;+ float sum = float(y0[at]);+ if (kParts > 1) sum += float(y1[at]);+ if (kParts > 2) sum += float(y2[at]);+ v[c] = r[j] + g[j] * ((sum / lambda[j]) / sgIn);+ r[j] = v[c];+ s += v[c];+ }+ float mean = row_sum(s, shared, tid) / float(p.columns);+ float q = 0;+ for (uint c = 0; c < 16; c++) { float d = v[c] - mean; q += d * d; }+ float inv = rsqrt(row_sum(q, shared, tid) / float(p.columns) + p.eps);+ device const float *m = row < p.targetStart ? prefixMod : targetMod;+ float peak = 0;+ for (uint c = 0; c < 16; c++) {+ v[c] = ((v[c] - mean) * inv) * m[tid + c * ROW_THREADS];+ peak = max(peak, fabs(v[c]));+ }+ float sg = ane_sigma(row_max(peak, shared, tid), l0, l1, l2);+ if (tid == 0) sigma[i] = sg;+ for (uint c = 0; c < 16; c++) {+ half hv = half(v[c] * sg);+ for (uint t = 0; t < a.surfaces; t++) out[t][tid + c * ROW_THREADS] = hv;+ }+}diff --git a/engine-mac/Sources/KokonEngine/NeuralEngine.swift b/engine-mac/Sources/KokonEngine/NeuralEngine.swiftindex cda9410..57a2832 100644--- a/engine-mac/Sources/KokonEngine/NeuralEngine.swift+++ b/engine-mac/Sources/KokonEngine/NeuralEngine.swift@@ -39,8 +39,9 @@ public final class NeuralEngineProduct: @unchecked Sendable {}}- /// `sharing`: a program of the same shape whose `x` and `y` surfaces this one uses too (its own weight only).- public init(rows: Int, k: Int, n: Int, int8: Bool = false, halfActivations: Bool = false, sharing: NeuralEngineProduct? = nil) throws {+ /// `sharing`: a program of the same shape whose `x` surface (and `y`, with `sharingOutput`) this one uses too.+ public init(rows: Int, k: Int, n: Int, int8: Bool = false, halfActivations: Bool = false, sharing: NeuralEngineProduct? = nil,+ sharingOutput: Bool = true) throws {self.rows = rows; self.k = k; self.n = n; self.int8 = int8; self.halfActivations = int8 && halfActivationslet int8x = int8 && !halfActivations(model, directory) = try NeuralEngineProduct.compile(@@ -49,7 +50,7 @@ public final class NeuralEngineProduct: @unchecked Sendable {inputs: [("x", [rows, k]), ("w", [k, n])], outputs: [("y", [rows, n])], int8Inputs: int8 ? (int8x ? ["x", "w"] : ["w"]) : [])x = try sharing?.x ?? NeuralEngineProduct.surface(rows: rows, columns: k, int8: int8x)w = try NeuralEngineProduct.surface(rows: k, columns: n, int8: int8)- y = try sharing?.y ?? NeuralEngineProduct.surface(rows: rows, columns: n)+ y = try (sharingOutput ? sharing?.y : nil) ?? NeuralEngineProduct.surface(rows: rows, columns: n)func shape(_ dims: Int...) -> [NSNumber] { dims.map { NSNumber(value: $0) } }options = MLPredictionOptions()options.outputBackings = ["y": MLMultiArray(pixelBuffer: y, shape: shape(rows, n))]diff --git a/engine-mac/Sources/KokonEngine/QwenDiT.swift b/engine-mac/Sources/KokonEngine/QwenDiT.swiftindex 887c9c7..cf51680 100644--- a/engine-mac/Sources/KokonEngine/QwenDiT.swift+++ b/engine-mac/Sources/KokonEngine/QwenDiT.swift@@ -329,6 +329,7 @@ public final class QwenDiT {let started = DispatchTime.now().uptimeNanosecondstimings = [:]castInput = nil+ deferredResidual = nil// The cached path when there is a cache and no probe (the probes compare every row of every stage).// A new conditioning: the whole sequence once, the prefix's K and V saved on the way. A pass of the prefix// alone took 1.86 s at 1024² (products of 64 rows run far below the GPU's rate); riding along the first@@ -489,9 +490,16 @@ public final class QwenDiT {let (first, count) = (rows.lowerBound, rows.count)let block = "transformer_blocks.\(layer)."let band = 5 * d // the target's modulation row follows the prefix's- try layerNorm(x, into: normed, rows: rows, prefixMod: 0, targetMod: band,- half: halfOutput(count: count, k: d, n: d), commands: commands)let qkv = ["to_q", "to_k", "to_v"].map { block + "attn.\($0).weight" }+ let qkvPack = rowPack(count: count, k: d, n: d, readers: qkv)+ if let deferred = deferredResidual { // the layer before's MLP residual, with this LN1+ deferredResidual = nil+ try residualNorm(gate: 3 * d, rows: rows, prefixMod: 0, targetMod: band, half: halfOutput(count: count, k: d, n: d),+ pack: qkvPack, tiles: deferred.tiles, commands: commands)+ } else {+ try layerNorm(x, into: normed, rows: rows, prefixMod: 0, targetMod: band,+ half: halfOutput(count: count, k: d, n: d), pack: qkvPack, commands: commands)+ }try linear(qkv[0], a: normed, aRow: first, c: q, m: count, cRow: first, next: qkv[1], readers: qkv, commands: commands)try linear(qkv[1], a: normed, aRow: first, c: k, m: count, cRow: first, next: qkv[2], readers: qkv, commands: commands)try mark("norm1", normed)@@ -508,22 +516,50 @@ public final class QwenDiT {if mode == .saving { try copyPrefix(layer: layer, save: true, commands: commands) }if mode == .target { try copyPrefix(layer: layer, save: false, commands: commands) }try attend(queries: p, count: n - p, keys: n, causal: false, commands: commands)- try linear(block + "attn.to_out.0.weight", a: attended, aRow: first, c: o, m: count, cRow: first,- next: block + "img_mlp.gate_layer.weight", commands: commands)- try mark("attn_out", o)- try residual(gate: d, rows: rows, commands: commands)- try mark("res1", x)- try layerNorm(x, into: normed, rows: rows, prefixMod: 2 * d, targetMod: band + 2 * d,- half: halfOutput(count: count, k: d, n: QwenDiT.hidden), commands: commands)- try mark("norm2", normed)let gateProj = ["gate_layer", "proj"].map { block + "img_mlp.\($0).weight" }- try linear(gateProj[0], a: normed, aRow: first, c: h1, m: count, cRow: first, next: gateProj[1], readers: gateProj, commands: commands)- try linear(gateProj[1], a: normed, aRow: first, c: h2, m: count, cRow: first, next: block + "img_mlp.out.weight", readers: gateProj, commands: commands)- try swiglu(first: first, count: count, half: halfOutput(count: count, k: QwenDiT.hidden, n: d), commands: commands)- try linear(block + "img_mlp.out.weight", a: h1, aRow: first, c: o, m: count, cRow: first,- next: layer + 1 < QwenDiT.layers ? "transformer_blocks.\(layer + 1).attn.to_q.weight" : nil, commands: commands)+ let toOut = block + "attn.to_out.0.weight"+ let gpPack = rowPack(count: count, k: d, n: QwenDiT.hidden, readers: gateProj)+ // With LN2's rows packed, to_out's Neural Engine rows stay in its output surfaces for the fused residual.+ let fuse = fusedResidual && stage == nil+ try linear(toOut, a: attended, aRow: first, c: o, m: count, cRow: first, next: gateProj[0],+ keepOutput: fuse && gpPack != nil, commands: commands)+ if fuse {+ try residualNorm(gate: d, rows: rows, prefixMod: 2 * d, targetMod: band + 2 * d,+ half: halfOutput(count: count, k: d, n: QwenDiT.hidden), pack: gpPack,+ tiles: gpPack != nil ? (coexecutor!.lastSet, toOut, 1) : nil, commands: commands)+ } else {+ try mark("attn_out", o)+ try residual(gate: d, rows: rows, commands: commands)+ try mark("res1", x)+ try layerNorm(x, into: normed, rows: rows, prefixMod: 2 * d, targetMod: band + 2 * d,+ half: halfOutput(count: count, k: d, n: QwenDiT.hidden), pack: gpPack, commands: commands)+ }+ try mark("norm2", normed)+ // With the MLP out's rows packed by the SwiGLU, gate's and proj's Neural Engine rows stay in their output+ // surfaces and the SwiGLU reads them there (no fp32 unpacking of 2 × 88 MB at 1024²).+ let mlpOut = block + "img_mlp.out.weight"+ let mlpPack = rowPack(count: count, k: QwenDiT.hidden, n: d, readers: [mlpOut])+ let keep = mlpPack != nil+ try linear(gateProj[0], a: normed, aRow: first, c: h1, m: count, cRow: first, next: gateProj[1], readers: gateProj,+ keepOutput: keep, commands: commands)+ let gateSet = coexecutor?.lastSet ?? 0+ try linear(gateProj[1], a: normed, aRow: first, c: h2, m: count, cRow: first, next: mlpOut, readers: gateProj,+ keepOutput: keep, commands: commands)+ try swiglu(first: first, count: count, half: halfOutput(count: count, k: QwenDiT.hidden, n: d), pack: mlpPack,+ tiles: keep ? (gateSet, coexecutor?.lastSet ?? 0, gateProj) : nil, commands: commands)+ // The MLP's residual goes with the next layer's LN1 (`deferredResidual`), the Neural Engine's rows of the MLP+ // out read there from its output surfaces when that LN1 packs them.+ let nextQKV = ["to_q", "to_k", "to_v"].map { "transformer_blocks.\(layer + 1).attn.\($0).weight" }+ let carry = fuse && layer + 1 < QwenDiT.layers+ let keepMLP = carry && rowPack(count: count, k: d, n: d, readers: nextQKV) != nil+ try linear(mlpOut, a: h1, aRow: first, c: o, m: count, cRow: first,+ next: layer + 1 < QwenDiT.layers ? nextQKV[0] : nil, keepOutput: keepMLP, commands: commands)try mark("mlp", o)- try residual(gate: 3 * d, rows: rows, commands: commands)+ if carry {+ deferredResidual = Deferred(tiles: keepMLP ? (coexecutor!.lastSet, mlpOut, 3) : nil)+ } else {+ try residual(gate: 3 * d, rows: rows, commands: commands)+ }}// MARK: - encoders@@ -582,7 +618,7 @@ public final class QwenDiT {/// GPU's ~1.8 ms per 4096 × 4096 weight otherwise ran alone between products)./// `readers`: every weight that reads this same A (q, k, v; gate, proj), so the Neural Engine's rows are packed once.func linear(_ name: String, a: Arena.Slice, aRow: Int = 0, c: Arena.Slice, m: Int, cRow: Int = 0,- next: String? = nil, readers: [String] = [], commands: MTLCommandBuffer) throws {+ next: String? = nil, readers: [String] = [], keepOutput: Bool = false, commands: MTLCommandBuffer) throws {let tensor = try weight(name)let (kk, nn) = (tensor.columns, tensor.rows)let m = QwenDiT.padded(m)@@ -644,7 +680,8 @@ public final class QwenDiT {}let ready = castInput.map { $0 == (a.offset, aRow, kk) } ?? falsetry coexecutor.encode(weight: name, a: operand(a, aRow, kk), w: operand(reserve, 0, nn), c: operand(c, cRow, nn),- m: m, k: kk, n: nn, commands: commands, readers: readers, castReady: ready, ahead: ahead, gpuRows: gpuRows)+ m: m, k: kk, n: nn, commands: commands, readers: readers, castReady: ready, keepOutput: keepOutput,+ ahead: ahead, gpuRows: gpuRows)castInput = c.offset == a.offset ? nil : (a.offset, aRow, kk)} else {try gpuRows(m)@@ -681,6 +718,79 @@ public final class QwenDiT {return (h.buffer, HalfOut(rows: UInt32(count), halfRows: UInt32(h.rows), fullFrom: probing ? 0 : UInt32(h.rows)))}+ /// Where a row kernel of `count` real rows packs the Neural Engine's rows of the split product that follows+ /// (`Coexecutor.RowPack`); nil: they are written in fp32 and packed by the product.+ func rowPack(count: Int, k: Int, n: Int, readers: [String]) -> Coexecutor.RowPack? {+ guard fusedCast, !profiling, !probing, let coexecutor else { return nil }+ return coexecutor.rowPack(m: QwenDiT.padded(count), k: k, n: n, readers: readers)+ }++ struct AnePack { var row0, real, surfaces, kSplit: UInt32 }++ /// Binds a row pack's σ, bounds and surfaces at `index` on.+ func bind(_ pack: Coexecutor.RowPack, _ e: MTLComputeCommandEncoder, real: Int, from index: Int) {+ var a = AnePack(row0: UInt32(pack.first), real: UInt32(real), surfaces: UInt32(pack.surfaces.count), kSplit: pack.kSplit ? 1 : 0)+ e.setBytes(&a, length: MemoryLayout<AnePack>.stride, index: index)+ e.setBuffer(pack.sigma, offset: 0, index: index + 1)+ for i in 0..<3 { e.setBuffer(pack.bounds[min(i, pack.bounds.count - 1)], offset: pack.boundOffset, index: index + 2 + i) }+ for i in 0..<3 { e.setBuffer(pack.surfaces[min(i, pack.surfaces.count - 1)], offset: 0, index: index + 5 + i) }+ }++ /// **The residual and the LN after it in one pass** (`residual_layernorm`, `residual_layernorm_ane`): x is read+ /// once instead of twice, and the Neural Engine's rows of `o` come from the product's output surfaces (`tiles`: set,+ /// weight, K parts) instead of an fp32 unpacking. After to_out (with LN2) and after the MLP out (with the next+ /// layer's LN1). The same values to the bit. `KOKON_FUSED_RESIDUAL=0`: the separate kernels.+ let fusedResidual = ProcessInfo.processInfo.environment["KOKON_FUSED_RESIDUAL"] != "0"+ struct Deferred { let tiles: (set: Int, weight: String, kParts: Int)? }+ /// The MLP residual of the layer before, encoded with this layer's LN1.+ var deferredResidual: Deferred?++ /// `x += tanh(gate) · o` then `LN(x) · mod` on `rows` (see `fusedResidual`).+ func residualNorm(gate: Int, rows: Range<Int>, prefixMod: Int, targetMod: Int, half: (buffer: MTLBuffer, out: HalfOut)?,+ pack: Coexecutor.RowPack?, tiles: (set: Int, weight: String, kParts: Int)?, commands: MTLCommandBuffer) throws {+ let d = QwenDiT.dim+ // Packing the Neural Engine's rows needs them from the product's surfaces; otherwise the residual kernel takes+ // every row (the Neural Engine's unpacked in fp32 by the product).+ guard (pack == nil) == (tiles == nil) else {+ try residual(gate: gate, rows: rows, commands: commands)+ return try layerNorm(x, into: normed, rows: rows, prefixMod: prefixMod, targetMod: targetMod, half: half, pack: pack, commands: commands)+ }+ let at = rows.lowerBound * d * 4+ let h = half?.out ?? QwenDiT.fullOnly(rows.count)+ var params = NormParams(columns: UInt32(d), targetStart: UInt32(max(0, sequence.prefix - rows.lowerBound)), eps: QwenDiT.eps, half: h)+ try compute("residual_layernorm", commands: commands) { e in+ e.setBuffer(arena.buffer, offset: x.offset + at, index: 0)+ e.setBuffer(arena.buffer, offset: normed.offset + at, index: 1)+ e.setBuffer(arena.buffer, offset: modulation.offset + prefixMod * 4, index: 2)+ e.setBuffer(arena.buffer, offset: modulation.offset + targetMod * 4, index: 3)+ e.setBytes(¶ms, length: MemoryLayout<NormParams>.stride, index: 4)+ e.setBuffer(half?.buffer ?? arena.buffer, offset: half == nil ? normed.offset + at : 0, index: 5)+ e.setBuffer(arena.buffer, offset: o.offset + at, index: 6)+ e.setBuffer(arena.buffer, offset: modulation.offset + gate * 4, index: 7)+ e.setBuffer(arena.buffer, offset: modulation.offset + (5 * d + gate) * 4, index: 8)+ e.dispatchThreadgroups(MTLSize(width: pack.map { $0.first } ?? max(rows.count, Int(h.halfRows)), height: 1, depth: 1),+ threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))+ }+ if let pack, let tiles, let c = coexecutor {+ try compute("residual_layernorm_ane", commands: commands) { e in+ e.setBuffer(arena.buffer, offset: x.offset + at, index: 0)+ e.setBuffer(arena.buffer, offset: modulation.offset + prefixMod * 4, index: 1)+ e.setBuffer(arena.buffer, offset: modulation.offset + targetMod * 4, index: 2)+ e.setBytes(¶ms, length: MemoryLayout<NormParams>.stride, index: 3)+ bind(pack, e, real: rows.count, from: 4)+ for t in 0..<3 { e.setBuffer(c.slots[tiles.set][min(t, tiles.kParts - 1)].y, offset: 0, index: 12 + t) }+ e.setBuffer(c.columns(tiles.weight), offset: 0, index: 15)+ var parts = UInt32(tiles.kParts)+ e.setBytes(&parts, length: 4, index: 16)+ e.setBuffer(arena.buffer, offset: modulation.offset + gate * 4, index: 17)+ e.setBuffer(arena.buffer, offset: modulation.offset + (5 * d + gate) * 4, index: 18)+ e.dispatchThreadgroups(MTLSize(width: pack.rows, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))+ }+ c.packedByRowKernel(offset: normed.offset + at, readers: pack.readers)+ }+ if half != nil { castInput = (normed.offset, rows.lowerBound, d) }+ }+/// No fp16 copy: every row in fp32.static func fullOnly(_ rows: Int) -> HalfOut { HalfOut(rows: UInt32(rows), halfRows: 0, fullFrom: 0) }struct HeadParams { var heads: UInt32; var rows: UInt32; var eps: Float }@@ -689,7 +799,7 @@ public final class QwenDiT {/// `out = LN(x) · mod[band]`, `prefixMod` / `targetMod` floats into the modulation slice; `half`: the next/// product's fp16 input too (`halfOutput`).func layerNorm(_ input: Arena.Slice, into out: Arena.Slice, rows: Range<Int>, prefixMod: Int, targetMod: Int,- half: (buffer: MTLBuffer, out: HalfOut)? = nil, commands: MTLCommandBuffer) throws {+ half: (buffer: MTLBuffer, out: HalfOut)? = nil, pack: Coexecutor.RowPack? = nil, commands: MTLCommandBuffer) throws {let at = rows.lowerBound * QwenDiT.dim * 4let h = half?.out ?? QwenDiT.fullOnly(rows.count)try compute("layernorm_modulate", commands: commands) { e in@@ -701,14 +811,29 @@ public final class QwenDiT {eps: QwenDiT.eps, half: h)e.setBytes(¶ms, length: MemoryLayout<NormParams>.stride, index: 4)e.setBuffer(half?.buffer ?? arena.buffer, offset: half == nil ? out.offset + at : 0, index: 5)- e.dispatchThreadgroups(MTLSize(width: max(rows.count, Int(h.halfRows)), height: 1, depth: 1),+ e.dispatchThreadgroups(MTLSize(width: pack.map { $0.first } ?? max(rows.count, Int(h.halfRows)), height: 1, depth: 1),threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))}+ if let pack {+ try compute("layernorm_modulate_ane", commands: commands) { e in+ e.setBuffer(arena.buffer, offset: input.offset + at, index: 0)+ e.setBuffer(arena.buffer, offset: modulation.offset + prefixMod * 4, index: 1)+ e.setBuffer(arena.buffer, offset: modulation.offset + targetMod * 4, index: 2)+ var params = NormParams(columns: UInt32(QwenDiT.dim), targetStart: UInt32(max(0, sequence.prefix - rows.lowerBound)),+ eps: QwenDiT.eps, half: h)+ e.setBytes(¶ms, length: MemoryLayout<NormParams>.stride, index: 3)+ bind(pack, e, real: rows.count, from: 4)+ e.dispatchThreadgroups(MTLSize(width: pack.rows, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))+ }+ coexecutor?.packedByRowKernel(offset: out.offset + at, readers: pack.readers)+ }if half != nil { castInput = (out.offset, rows.lowerBound, QwenDiT.dim) }}/// `h1 = SiLU(h1) ⊙ h2` on rows `[first, first + count)`; `half`: the MLP out's fp16 input too.- func swiglu(first: Int, count: Int, half: (buffer: MTLBuffer, out: HalfOut)?, commands: MTLCommandBuffer) throws {+ /// `tiles`: gate's and proj's Neural Engine rows read from their output surfaces (sets, weights) instead of h1, h2.+ func swiglu(first: Int, count: Int, half: (buffer: MTLBuffer, out: HalfOut)?, pack: Coexecutor.RowPack? = nil,+ tiles: (gate: Int, proj: Int, weights: [String])? = nil, commands: MTLCommandBuffer) throws {let w = QwenDiT.hiddenlet h = half?.out ?? QwenDiT.fullOnly(count)try compute("swiglu", commands: commands) { e in@@ -717,9 +842,28 @@ public final class QwenDiT {var params = SwigluParams(columns: UInt32(w), half: h)e.setBytes(¶ms, length: MemoryLayout<SwigluParams>.stride, index: 2)e.setBuffer(half?.buffer ?? arena.buffer, offset: half == nil ? h1.offset : 0, index: 3)- e.dispatchThreads(MTLSize(width: max(count, Int(h.halfRows)) * w, height: 1, depth: 1),+ e.dispatchThreads(MTLSize(width: (pack.map { $0.first } ?? max(count, Int(h.halfRows))) * w, height: 1, depth: 1),threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))}+ if let pack, let tiles, let c = coexecutor {+ try compute("swiglu_ane_tiles", commands: commands) { e in+ for t in 0..<3 { e.setBuffer(c.slots[tiles.gate][t].y, offset: 0, index: t) }+ for t in 0..<3 { e.setBuffer(c.slots[tiles.proj][t].y, offset: 0, index: 3 + t) }+ e.setBuffer(c.columns(tiles.weights[0]), offset: 0, index: 6)+ e.setBuffer(c.columns(tiles.weights[1]), offset: 0, index: 7)+ bind(pack, e, real: count, from: 8)+ e.dispatchThreadgroups(MTLSize(width: pack.rows, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))+ }+ c.packedByRowKernel(offset: h1.offset + first * w * 4, readers: pack.readers)+ } else if let pack {+ try compute("swiglu_ane", commands: commands) { e in+ e.setBuffer(arena.buffer, offset: h1.offset + first * w * 4, index: 0)+ e.setBuffer(arena.buffer, offset: h2.offset + first * w * 4, index: 1)+ bind(pack, e, real: count, from: 2)+ e.dispatchThreadgroups(MTLSize(width: pack.rows, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))+ }+ coexecutor?.packedByRowKernel(offset: h1.offset + first * w * 4, readers: pack.readers)+ }if half != nil { castInput = (h1.offset, first, w) }}