Skip to content

armgit / kokon

Public Imported from GitHub
0

Edit photos by describing the change, on your own GPU. A free, open-source alternative to Nano Banana and ChatGPT image editing: photos stay on your computer, no daily limit. One installer for Windows, Mac and Linux sets up the best open model (Qwen-Image 2.1). MCP server for AI agents.

Mac engine: row kernels pack the Neural Engine's rows, residual fused with the LN after it

Gabriele Lozupone840fa90baa33
diff --git a/engine-mac/README.md b/engine-mac/README.md
index 4df9feb..f0d8a2a 100644
--- a/engine-mac/README.md
+++ b/engine-mac/README.md
@@ -246,6 +246,18 @@ worse). Between operations 0.60 → 0.53 s, but per evaluation, 3 alternated rou
instead of 66). With the three units at their shared limit, the GPU-alone time moved into the products; what pays
is fewer bytes moved, as with the fp16 cast above.
+**The row kernels pack the Neural Engine's rows, and the residual goes with the LN after it** (2026-10-26;
+`KOKON_FUSED_PACK=0`, `KOKON_FUSED_RESIDUAL=0` for the comparison). From the second evaluation (every weight's L known):
+the LNs and the SwiGLU compute the Neural Engine's rows' σ themselves and write them scaled and packed into its input
+surfaces (`layernorm_modulate_ane`, `swiglu_ane`), where they wrote fp32 rows that `ane_row_scales` and `ane_pack_rows`
+read back; gate's and proj's Neural Engine rows stay in their output surfaces (each program set has its own now) and the
+SwiGLU reads them there instead of two fp32 unpackings; the residual after to_out and the one after the MLP out (carried
+into the next layer's LN1) run in the LN's pass (`residual_layernorm`, `residual_layernorm_ane`), x read once and the
+Neural Engine's rows of `o` taken from the product's surfaces. The velocity is the same to the bit (1024², fixed shares,
+three evaluations, `KOKON_DUMP=<file> time-dit` with each switch; `check-generate` unchanged to every digit). Between operations an evaluation: 0.60–0.66 → 0.41–0.45
+s (proj → MLP out 5.4 → 3.3 ms, gate → proj 1.3–2.6 → 0.07, to_out → gate 3.3 → 1.7–2.0, MLP out → q 3.4 → 2.5). Per
+evaluation at 1024², 5 alternated rounds of 4 (M2 Pro, other apps open, ~4.5 GB of page-ins a run): median 7.71 → 7.34 s.
+
**The Neural Engine's rows packed once per input** (`KOKON_SHARED_ROWS=0` for the comparison): q, k and v read the
same rows, and so do gate and proj, but each product scaled and packed them again with its own σ. From the second
evaluation on (when every weight's L is known), an input's rows get one σ from the largest L of its readers and are
diff --git a/engine-mac/Sources/KokonEngine/Checks.swift b/engine-mac/Sources/KokonEngine/Checks.swift
index 58c8e0b..74c885a 100644
--- a/engine-mac/Sources/KokonEngine/Checks.swift
+++ b/engine-mac/Sources/KokonEngine/Checks.swift
@@ -128,6 +128,9 @@ extension Checks {
let started = DispatchTime.now().uptimeNanoseconds
let out = try dit.forward(encoder: encoder, target: latent, sigma: 0.5)
let seconds = Double(DispatchTime.now().uptimeNanoseconds - started) * 1e-9
+ if let dump = ProcessInfo.processInfo.environment["KOKON_DUMP"] { // each evaluation's velocity, to compare builds
+ try out.withUnsafeBytes { try Data($0).write(to: URL(fileURLWithPath: "\(dump).\(run + 1)")) }
+ }
print(String(format: "evaluation %d: %.2f s%@%@", run + 1, seconds, dit.timings["prefix"].map { String(format: " (prefix pass %.2f s)", $0) } ?? "",
out.allSatisfy(\.isFinite) ? "" : " NON-FINITE OUTPUT"))
if let c = dit.coexecutor {
diff --git a/engine-mac/Sources/KokonEngine/Coexecution.swift b/engine-mac/Sources/KokonEngine/Coexecution.swift
index ccb55da..308c36a 100644
--- a/engine-mac/Sources/KokonEngine/Coexecution.swift
+++ b/engine-mac/Sources/KokonEngine/Coexecution.swift
@@ -30,8 +30,9 @@ final class Coexecutor {
var amxRows: [String: Int] = [:]
typealias ProductSlot = (product: NeuralEngineProduct, x: MTLBuffer, w: MTLBuffer, y: MTLBuffer)
/// Two sets of the three tile programs, alternating product by product: the next product's weight is packed into
- /// one set while the Neural Engine reads the other (`prepareWeight`). The sets share their row surfaces.
- private var slots: [[ProductSlot]]
+ /// one set while the Neural Engine reads the other (`prepareWeight`). The sets share their input surfaces; each has
+ /// its own outputs, so gate's stay beside proj's for the SwiGLU (`keepOutput`).
+ private(set) var slots: [[ProductSlot]]
/// Programs of other row counts, kept while the balancing loop may come back to them.
private var spareSlots: [Int: [[ProductSlot]]] = [:]
/// The set of the last product encoded; the weight packed ahead and its set.
@@ -148,9 +149,9 @@ final class Coexecutor {
}
let second: [ProductSlot] = try first.map { f in
let p = try NeuralEngineProduct(rows: rows, k: Coexecutor.tile, n: Coexecutor.tile, int8: aneInt8, halfActivations: aneHalfRows,
- sharing: f.product)
+ sharing: f.product, sharingOutput: false)
try p.run()
- return (p, f.x, try p.gpuBuffer(p.w, device: device), f.y)
+ return (p, f.x, try p.gpuBuffer(p.w, device: device), try p.gpuBuffer(p.y, device: device))
}
return [first, second]
}
@@ -190,13 +191,46 @@ final class Coexecutor {
return (aHalf, m - split(m: m, k: k, n: n).ane)
}
+ /// **The row kernel before a split product packs its Neural Engine rows** (`layernorm_modulate_ane`, `swiglu_ane`):
+ /// σ and the scaled fp16 rows straight into the input surfaces, instead of fp32 rows that `ane_row_scales` and
+ /// `ane_pack_rows` read back. Once every reader's L is known (from the first evaluation on), fp16, `sharedRows` for
+ /// several readers. `KOKON_FUSED_PACK=0`: the separate passes, for comparison.
+ struct RowPack {
+ let readers: [String], surfaces: [MTLBuffer], bounds: [MTLBuffer], boundOffset: Int, sigma: MTLBuffer
+ /// The first Neural Engine row (from the product's first row), their count, K split into the tiles (MLP out).
+ let first: Int, rows: Int, kSplit: Bool
+ }
+ let fusedPack = ProcessInfo.processInfo.environment["KOKON_FUSED_PACK"] != "0"
+ /// The input (A's offset) a single-reader product finds packed by its row kernel.
+ var prepacked: Int?
+
+ func rowPack(m: Int, k: Int, n: Int, readers: [String]) -> RowPack? {
+ guard fusedPack, aneEnabled, !aneInt8, halfActivations, !readers.isEmpty, readers.count == 1 || sharedRows,
+ applies(m: m, k: k, n: n), m <= maxRows, m >= rows, k == Coexecutor.tile || n == Coexecutor.tile else { return nil }
+ let s = split(m: m, k: k, n: n)
+ let bounds = readers.compactMap { columnScales[$0] }
+ guard s.ane > 0, bounds.count == readers.count else { return nil }
+ let tiles = k / Coexecutor.tile * (n / Coexecutor.tile)
+ return RowPack(readers: readers, surfaces: slots[0].prefix(tiles).map(\.x), bounds: bounds, boundOffset: 2 * n * 4, sigma: scales,
+ first: m - s.ane, rows: s.ane, kSplit: k > Coexecutor.tile)
+ }
+
+ /// The set of the last product encoded, whose output surfaces hold its Neural Engine rows (`keepOutput`).
+ var lastSet: Int { currentSet }
+ func columns(_ weight: String) -> MTLBuffer? { columnScales[weight] }
+
+ /// Notes that the rows at `offset` were packed by their row kernel for `readers`.
+ func packedByRowKernel(offset: Int, readers: [String]) {
+ if readers.count > 1 { packedRows = (offset, readers) } else { prepacked = offset }
+ }
+
/// Encodes the split product of the weight named `weight`. `gpuRows` encodes the GPU's own rows `[0, g)` into the
/// command buffer.
func encode(weight: String, a: (buffer: MTLBuffer, offset: Int, pointer: UnsafeMutablePointer<Float>),
w: (buffer: MTLBuffer, offset: Int, pointer: UnsafeMutablePointer<Float>),
c: (buffer: MTLBuffer, offset: Int, pointer: UnsafeMutablePointer<Float>),
m: Int, k: Int, n: Int, commands: MTLCommandBuffer, readers: [String] = [],
- castReady: Bool = false, ahead: ((MTLCommandBuffer) throws -> Void)? = nil,
+ castReady: Bool = false, keepOutput: Bool = false, ahead: ((MTLCommandBuffer) throws -> Void)? = nil,
gpuRows: (Int, MTLBuffer?) throws -> Void) throws {
let s = split(m: m, k: k, n: n)
counter += 1
@@ -232,6 +266,8 @@ final class Coexecutor {
let shared = sharedRows && readers.count > 1 && readers.count <= 3 && readers.contains(weight)
&& (aneInt8 || readers.allSatisfy { columnScales[$0] != nil })
let packed = shared && packedRows.map { $0.offset == a.offset && $0.readers == readers } ?? false
+ || prepacked == a.offset
+ prepacked = nil
if s.ane > 0, !packed {
var sh = shape()
let bounds = shared && !aneInt8 ? readers.map { columnScales[$0]! } : [columns] // L: fp16 only
@@ -306,6 +342,8 @@ final class Coexecutor {
if s.amx > 0 { commands.encodeWaitForEvent(amxDone, value: p) }
if s.ane > 0 {
commands.encodeWaitForEvent(aneDone, value: p)
+ // Left in its set's output surfaces for the kernel that reads them (gate and proj for the SwiGLU).
+ if keepOutput { return }
for np in 0..<nParts {
var t = shape(np: np)
let ys = (0..<3).map { kp in slots[min(kp, kParts - 1) * nParts + np].y }
diff --git a/engine-mac/Sources/KokonEngine/Kernels/dit.metal b/engine-mac/Sources/KokonEngine/Kernels/dit.metal
index 66b361f..00eae1c 100644
--- a/engine-mac/Sources/KokonEngine/Kernels/dit.metal
+++ b/engine-mac/Sources/KokonEngine/Kernels/dit.metal
@@ -185,3 +185,223 @@ kernel void widen_q8(device float4 *dst [[buffer(0)]], device const char4 *codes
constant uint &lanes [[buffer(4)]], uint i [[thread_position_in_grid]]) {
if (i / 32 < units) dst[i] = float4(codes[i]) * scales[i / lanes];
}
+
+// The Neural Engine's rows of a row kernel's output, written straight into the next split product's input surfaces
+// (`Coexecution.swift`, `RowPack`): each row's σ from its largest |value| and the readers' bound L, as `ane_row_scales`
+// computes it, then half(v · σ) as `ane_pack_rows` writes it, into `surfaces` tiles of 4 096 columns (the same row
+// into each of an N-split product's tiles; a K-split product's parts in turn). Rows from `real` on are the product's
+// padding: zeros (σ 1), as packing never-written rows gave.
+struct AnePack {
+ uint row0; // the first Neural Engine row, from the kernel's first row
+ uint real; // rows of the kernel that are real, from its first row
+ uint surfaces; // tiles written: 1 to 3
+ uint kSplit; // 1: column j goes to tile j / 4096 (K parts); 0: every tile gets the whole row (N parts)
+};
+
+static inline float row_max(float v, threadgroup float *shared, uint tid) {
+ v = simd_max(v);
+ if (tid % 32 == 0) shared[tid / 32] = v;
+ threadgroup_barrier(mem_flags::mem_threadgroup);
+ float m = 0;
+ for (uint i = 0; i < ROW_THREADS / 32; i++) m = max(m, shared[i]);
+ threadgroup_barrier(mem_flags::mem_threadgroup);
+ return m;
+}
+
+static inline float ane_sigma(float peak, device const float *l0, device const float *l1, device const float *l2) {
+ float bound = peak * max(l0[0], max(l1[0], l2[0]));
+ return (bound > 0 && isfinite(bound)) ? exp2(floor(log2(32768.0f / bound))) : 1.0f;
+}
+
+// layernorm_modulate on the Neural Engine's rows (4 096 columns), packed. One threadgroup per row.
+kernel void layernorm_modulate_ane(device const float *x [[buffer(0)]], device const float *prefixMod [[buffer(1)]],
+ device const float *targetMod [[buffer(2)]], constant NormParams &p [[buffer(3)]],
+ constant AnePack &a [[buffer(4)]], device float *sigma [[buffer(5)]],
+ device const float *l0 [[buffer(6)]], device const float *l1 [[buffer(7)]],
+ device const float *l2 [[buffer(8)]], device half *s0 [[buffer(9)]],
+ device half *s1 [[buffer(10)]], device half *s2 [[buffer(11)]],
+ uint i [[threadgroup_position_in_grid]], uint tid [[thread_index_in_threadgroup]]) {
+ threadgroup float shared[ROW_THREADS / 32];
+ uint row = a.row0 + i;
+ device half *out[3] = {s0 + (ulong)i * 4096, s1 + (ulong)i * 4096, s2 + (ulong)i * 4096};
+ if (row >= a.real) {
+ for (uint t = 0; t < a.surfaces; t++) for (uint j = tid; j < 4096; j += ROW_THREADS) out[t][j] = 0;
+ if (tid == 0) sigma[i] = 1;
+ return;
+ }
+ device const float *r = x + (ulong)row * p.columns;
+ float s = 0;
+ for (uint j = tid; j < p.columns; j += ROW_THREADS) s += r[j];
+ float mean = row_sum(s, shared, tid) / float(p.columns);
+ float q = 0;
+ for (uint j = tid; j < p.columns; j += ROW_THREADS) { float c = r[j] - mean; q += c * c; }
+ float inv = rsqrt(row_sum(q, shared, tid) / float(p.columns) + p.eps);
+ device const float *m = row < p.targetStart ? prefixMod : targetMod;
+ float v[16], peak = 0;
+ for (uint c = 0; c < 16; c++) {
+ uint j = tid + c * ROW_THREADS;
+ v[c] = ((r[j] - mean) * inv) * m[j];
+ peak = max(peak, fabs(v[c]));
+ }
+ float sg = ane_sigma(row_max(peak, shared, tid), l0, l1, l2);
+ if (tid == 0) sigma[i] = sg;
+ for (uint c = 0; c < 16; c++) {
+ half h = half(v[c] * sg);
+ for (uint t = 0; t < a.surfaces; t++) out[t][tid + c * ROW_THREADS] = h;
+ }
+}
+
+// swiglu on the Neural Engine's rows (12 288 columns: the MLP out's K parts), packed. One threadgroup per row.
+// Padding rows pack the gate's value, which is what `swiglu` leaves in them.
+kernel void swiglu_ane(device const float *gate [[buffer(0)]], device const float *proj [[buffer(1)]],
+ constant AnePack &a [[buffer(2)]], device float *sigma [[buffer(3)]],
+ device const float *l0 [[buffer(4)]], device const float *l1 [[buffer(5)]],
+ device const float *l2 [[buffer(6)]], device half *s0 [[buffer(7)]],
+ device half *s1 [[buffer(8)]], device half *s2 [[buffer(9)]],
+ uint i [[threadgroup_position_in_grid]], uint tid [[thread_index_in_threadgroup]]) {
+ threadgroup float shared[ROW_THREADS / 32];
+ uint row = a.row0 + i;
+ device const float *g = gate + (ulong)row * 12288, *pr = proj + (ulong)row * 12288;
+ float v[48], peak = 0;
+ for (uint c = 0; c < 48; c++) {
+ uint j = tid + c * ROW_THREADS;
+ float x = g[j];
+ v[c] = row < a.real ? (x / (1.0f + exp(-x))) * pr[j] : x;
+ peak = max(peak, fabs(v[c]));
+ }
+ float sg = ane_sigma(row_max(peak, shared, tid), l0, l1, l2);
+ if (tid == 0) sigma[i] = sg;
+ device half *out[3] = {s0 + (ulong)i * 4096, s1 + (ulong)i * 4096, s2 + (ulong)i * 4096};
+ for (uint c = 0; c < 48; c++) {
+ uint j = tid + c * ROW_THREADS;
+ out[j / 4096][j % 4096] = half(v[c] * sg);
+ }
+}
+
+// swiglu_ane reading gate's and proj's Neural Engine outputs where the products left them (`keepOutput`): each
+// value unscaled as `ane_unpack` does ((y / λ) / σ, σ the rows' shared one, then replaced by the MLP out's), the gate
+// tiles `g0..g2` and proj tiles `p0..p2` of 4 096 columns each. Value for value the unpacks, `swiglu`, then the packing.
+kernel void swiglu_ane_tiles(device const half *g0 [[buffer(0)]], device const half *g1 [[buffer(1)]],
+ device const half *g2 [[buffer(2)]], device const half *p0 [[buffer(3)]],
+ device const half *p1 [[buffer(4)]], device const half *p2 [[buffer(5)]],
+ device const float *lambdaG [[buffer(6)]], device const float *lambdaP [[buffer(7)]],
+ constant AnePack &a [[buffer(8)]], device float *sigma [[buffer(9)]],
+ device const float *l0 [[buffer(10)]], device const float *l1 [[buffer(11)]],
+ device const float *l2 [[buffer(12)]], device half *s0 [[buffer(13)]],
+ device half *s1 [[buffer(14)]], device half *s2 [[buffer(15)]],
+ uint i [[threadgroup_position_in_grid]], uint tid [[thread_index_in_threadgroup]]) {
+ threadgroup float shared[ROW_THREADS / 32];
+ uint row = a.row0 + i;
+ float sgIn = sigma[i]; // read before the barriers in row_max; thread 0 rewrites it after them
+ device const half *gs[3] = {g0, g1, g2}, *ps[3] = {p0, p1, p2};
+ float v[48], peak = 0;
+ for (uint c = 0; c < 48; c++) {
+ uint j = tid + c * ROW_THREADS, t = j / 4096;
+ ulong at = (ulong)i * 4096 + j % 4096;
+ float x = (float(gs[t][at]) / lambdaG[j]) / sgIn;
+ v[c] = row < a.real ? (x / (1.0f + exp(-x))) * ((float(ps[t][at]) / lambdaP[j]) / sgIn) : x;
+ peak = max(peak, fabs(v[c]));
+ }
+ float sg = ane_sigma(row_max(peak, shared, tid), l0, l1, l2);
+ if (tid == 0) sigma[i] = sg;
+ device half *out[3] = {s0 + (ulong)i * 4096, s1 + (ulong)i * 4096, s2 + (ulong)i * 4096};
+ for (uint c = 0; c < 48; c++) {
+ uint j = tid + c * ROW_THREADS;
+ out[j / 4096][j % 4096] = half(v[c] * sg);
+ }
+}
+
+// gated_residual then layernorm_modulate in one pass over each row: x += tanh(gate) · o (written back), then the
+// norm of the new x, as the two kernels compute them. Rows from `real` on are padding: no residual, zeros in the fp16
+// copy. One threadgroup per row, 4 096 columns.
+kernel void residual_layernorm(device float *x [[buffer(0)]], device float *out [[buffer(1)]],
+ device const float *prefixMod [[buffer(2)]], device const float *targetMod [[buffer(3)]],
+ constant NormParams &p [[buffer(4)]], device half *outHalf [[buffer(5)]],
+ device const float *o [[buffer(6)]], device const float *prefixGate [[buffer(7)]],
+ device const float *targetGate [[buffer(8)]],
+ uint row [[threadgroup_position_in_grid]], uint tid [[thread_index_in_threadgroup]]) {
+ threadgroup float shared[ROW_THREADS / 32];
+ device half *h = outHalf + (ulong)row * p.columns;
+ if (row >= p.half_out.rows) {
+ for (uint j = tid; j < p.columns; j += ROW_THREADS) h[j] = 0;
+ return;
+ }
+ device float *r = x + (ulong)row * p.columns;
+ device const float *orow = o + (ulong)row * p.columns;
+ device const float *g = row < p.targetStart ? prefixGate : targetGate;
+ float v[16], s = 0;
+ for (uint c = 0; c < 16; c++) {
+ uint j = tid + c * ROW_THREADS;
+ v[c] = r[j] + g[j] * orow[j];
+ r[j] = v[c];
+ s += v[c];
+ }
+ float mean = row_sum(s, shared, tid) / float(p.columns);
+ float q = 0;
+ for (uint c = 0; c < 16; c++) { float d = v[c] - mean; q += d * d; }
+ float inv = rsqrt(row_sum(q, shared, tid) / float(p.columns) + p.eps);
+ device const float *m = row < p.targetStart ? prefixMod : targetMod;
+ device float *w = out + (ulong)row * p.columns;
+ bool toHalf = row < p.half_out.halfRows, toFull = row >= p.half_out.fullFrom;
+ for (uint c = 0; c < 16; c++) {
+ uint j = tid + c * ROW_THREADS;
+ float y = ((v[c] - mean) * inv) * m[j];
+ if (toFull) w[j] = y;
+ if (toHalf) h[j] = half(y);
+ }
+}
+
+// The same on the Neural Engine's rows, o read from the product's output tiles (`y0..y2`: its K parts, summed in
+// fp32 and unscaled as `ane_unpack` does, σ the product's, then replaced by the next one's), the norm packed as in
+// layernorm_modulate_ane.
+kernel void residual_layernorm_ane(device float *x [[buffer(0)]], device const float *prefixMod [[buffer(1)]],
+ device const float *targetMod [[buffer(2)]], constant NormParams &p [[buffer(3)]],
+ constant AnePack &a [[buffer(4)]], device float *sigma [[buffer(5)]],
+ device const float *l0 [[buffer(6)]], device const float *l1 [[buffer(7)]],
+ device const float *l2 [[buffer(8)]], device half *s0 [[buffer(9)]],
+ device half *s1 [[buffer(10)]], device half *s2 [[buffer(11)]],
+ device const half *y0 [[buffer(12)]], device const half *y1 [[buffer(13)]],
+ device const half *y2 [[buffer(14)]], device const float *lambda [[buffer(15)]],
+ constant uint &kParts [[buffer(16)]], device const float *prefixGate [[buffer(17)]],
+ device const float *targetGate [[buffer(18)]],
+ uint i [[threadgroup_position_in_grid]], uint tid [[thread_index_in_threadgroup]]) {
+ threadgroup float shared[ROW_THREADS / 32];
+ uint row = a.row0 + i;
+ device half *out[3] = {s0 + (ulong)i * 4096, s1 + (ulong)i * 4096, s2 + (ulong)i * 4096};
+ float sgIn = sigma[i]; // read before the barriers; thread 0 rewrites it after them
+ if (row >= a.real) {
+ for (uint t = 0; t < a.surfaces; t++) for (uint j = tid; j < 4096; j += ROW_THREADS) out[t][j] = 0;
+ threadgroup_barrier(mem_flags::mem_threadgroup);
+ if (tid == 0) sigma[i] = 1;
+ return;
+ }
+ device float *r = x + (ulong)row * p.columns;
+ device const float *g = row < p.targetStart ? prefixGate : targetGate;
+ float v[16], s = 0;
+ for (uint c = 0; c < 16; c++) {
+ uint j = tid + c * ROW_THREADS;
+ ulong at = (ulong)i * 4096 + j;
+ float sum = float(y0[at]);
+ if (kParts > 1) sum += float(y1[at]);
+ if (kParts > 2) sum += float(y2[at]);
+ v[c] = r[j] + g[j] * ((sum / lambda[j]) / sgIn);
+ r[j] = v[c];
+ s += v[c];
+ }
+ float mean = row_sum(s, shared, tid) / float(p.columns);
+ float q = 0;
+ for (uint c = 0; c < 16; c++) { float d = v[c] - mean; q += d * d; }
+ float inv = rsqrt(row_sum(q, shared, tid) / float(p.columns) + p.eps);
+ device const float *m = row < p.targetStart ? prefixMod : targetMod;
+ float peak = 0;
+ for (uint c = 0; c < 16; c++) {
+ v[c] = ((v[c] - mean) * inv) * m[tid + c * ROW_THREADS];
+ peak = max(peak, fabs(v[c]));
+ }
+ float sg = ane_sigma(row_max(peak, shared, tid), l0, l1, l2);
+ if (tid == 0) sigma[i] = sg;
+ for (uint c = 0; c < 16; c++) {
+ half hv = half(v[c] * sg);
+ for (uint t = 0; t < a.surfaces; t++) out[t][tid + c * ROW_THREADS] = hv;
+ }
+}
diff --git a/engine-mac/Sources/KokonEngine/NeuralEngine.swift b/engine-mac/Sources/KokonEngine/NeuralEngine.swift
index cda9410..57a2832 100644
--- a/engine-mac/Sources/KokonEngine/NeuralEngine.swift
+++ b/engine-mac/Sources/KokonEngine/NeuralEngine.swift
@@ -39,8 +39,9 @@ public final class NeuralEngineProduct: @unchecked Sendable {
}
}
- /// `sharing`: a program of the same shape whose `x` and `y` surfaces this one uses too (its own weight only).
- public init(rows: Int, k: Int, n: Int, int8: Bool = false, halfActivations: Bool = false, sharing: NeuralEngineProduct? = nil) throws {
+ /// `sharing`: a program of the same shape whose `x` surface (and `y`, with `sharingOutput`) this one uses too.
+ public init(rows: Int, k: Int, n: Int, int8: Bool = false, halfActivations: Bool = false, sharing: NeuralEngineProduct? = nil,
+ sharingOutput: Bool = true) throws {
self.rows = rows; self.k = k; self.n = n; self.int8 = int8; self.halfActivations = int8 && halfActivations
let int8x = int8 && !halfActivations
(model, directory) = try NeuralEngineProduct.compile(
@@ -49,7 +50,7 @@ public final class NeuralEngineProduct: @unchecked Sendable {
inputs: [("x", [rows, k]), ("w", [k, n])], outputs: [("y", [rows, n])], int8Inputs: int8 ? (int8x ? ["x", "w"] : ["w"]) : [])
x = try sharing?.x ?? NeuralEngineProduct.surface(rows: rows, columns: k, int8: int8x)
w = try NeuralEngineProduct.surface(rows: k, columns: n, int8: int8)
- y = try sharing?.y ?? NeuralEngineProduct.surface(rows: rows, columns: n)
+ y = try (sharingOutput ? sharing?.y : nil) ?? NeuralEngineProduct.surface(rows: rows, columns: n)
func shape(_ dims: Int...) -> [NSNumber] { dims.map { NSNumber(value: $0) } }
options = MLPredictionOptions()
options.outputBackings = ["y": MLMultiArray(pixelBuffer: y, shape: shape(rows, n))]
diff --git a/engine-mac/Sources/KokonEngine/QwenDiT.swift b/engine-mac/Sources/KokonEngine/QwenDiT.swift
index 887c9c7..cf51680 100644
--- a/engine-mac/Sources/KokonEngine/QwenDiT.swift
+++ b/engine-mac/Sources/KokonEngine/QwenDiT.swift
@@ -329,6 +329,7 @@ public final class QwenDiT {
let started = DispatchTime.now().uptimeNanoseconds
timings = [:]
castInput = nil
+ deferredResidual = nil
// The cached path when there is a cache and no probe (the probes compare every row of every stage).
// A new conditioning: the whole sequence once, the prefix's K and V saved on the way. A pass of the prefix
// alone took 1.86 s at 1024² (products of 64 rows run far below the GPU's rate); riding along the first
@@ -489,9 +490,16 @@ public final class QwenDiT {
let (first, count) = (rows.lowerBound, rows.count)
let block = "transformer_blocks.\(layer)."
let band = 5 * d // the target's modulation row follows the prefix's
- try layerNorm(x, into: normed, rows: rows, prefixMod: 0, targetMod: band,
- half: halfOutput(count: count, k: d, n: d), commands: commands)
let qkv = ["to_q", "to_k", "to_v"].map { block + "attn.\($0).weight" }
+ let qkvPack = rowPack(count: count, k: d, n: d, readers: qkv)
+ if let deferred = deferredResidual { // the layer before's MLP residual, with this LN1
+ deferredResidual = nil
+ try residualNorm(gate: 3 * d, rows: rows, prefixMod: 0, targetMod: band, half: halfOutput(count: count, k: d, n: d),
+ pack: qkvPack, tiles: deferred.tiles, commands: commands)
+ } else {
+ try layerNorm(x, into: normed, rows: rows, prefixMod: 0, targetMod: band,
+ half: halfOutput(count: count, k: d, n: d), pack: qkvPack, commands: commands)
+ }
try linear(qkv[0], a: normed, aRow: first, c: q, m: count, cRow: first, next: qkv[1], readers: qkv, commands: commands)
try linear(qkv[1], a: normed, aRow: first, c: k, m: count, cRow: first, next: qkv[2], readers: qkv, commands: commands)
try mark("norm1", normed)
@@ -508,22 +516,50 @@ public final class QwenDiT {
if mode == .saving { try copyPrefix(layer: layer, save: true, commands: commands) }
if mode == .target { try copyPrefix(layer: layer, save: false, commands: commands) }
try attend(queries: p, count: n - p, keys: n, causal: false, commands: commands)
- try linear(block + "attn.to_out.0.weight", a: attended, aRow: first, c: o, m: count, cRow: first,
- next: block + "img_mlp.gate_layer.weight", commands: commands)
- try mark("attn_out", o)
- try residual(gate: d, rows: rows, commands: commands)
- try mark("res1", x)
- try layerNorm(x, into: normed, rows: rows, prefixMod: 2 * d, targetMod: band + 2 * d,
- half: halfOutput(count: count, k: d, n: QwenDiT.hidden), commands: commands)
- try mark("norm2", normed)
let gateProj = ["gate_layer", "proj"].map { block + "img_mlp.\($0).weight" }
- try linear(gateProj[0], a: normed, aRow: first, c: h1, m: count, cRow: first, next: gateProj[1], readers: gateProj, commands: commands)
- try linear(gateProj[1], a: normed, aRow: first, c: h2, m: count, cRow: first, next: block + "img_mlp.out.weight", readers: gateProj, commands: commands)
- try swiglu(first: first, count: count, half: halfOutput(count: count, k: QwenDiT.hidden, n: d), commands: commands)
- try linear(block + "img_mlp.out.weight", a: h1, aRow: first, c: o, m: count, cRow: first,
- next: layer + 1 < QwenDiT.layers ? "transformer_blocks.\(layer + 1).attn.to_q.weight" : nil, commands: commands)
+ let toOut = block + "attn.to_out.0.weight"
+ let gpPack = rowPack(count: count, k: d, n: QwenDiT.hidden, readers: gateProj)
+ // With LN2's rows packed, to_out's Neural Engine rows stay in its output surfaces for the fused residual.
+ let fuse = fusedResidual && stage == nil
+ try linear(toOut, a: attended, aRow: first, c: o, m: count, cRow: first, next: gateProj[0],
+ keepOutput: fuse && gpPack != nil, commands: commands)
+ if fuse {
+ try residualNorm(gate: d, rows: rows, prefixMod: 2 * d, targetMod: band + 2 * d,
+ half: halfOutput(count: count, k: d, n: QwenDiT.hidden), pack: gpPack,
+ tiles: gpPack != nil ? (coexecutor!.lastSet, toOut, 1) : nil, commands: commands)
+ } else {
+ try mark("attn_out", o)
+ try residual(gate: d, rows: rows, commands: commands)
+ try mark("res1", x)
+ try layerNorm(x, into: normed, rows: rows, prefixMod: 2 * d, targetMod: band + 2 * d,
+ half: halfOutput(count: count, k: d, n: QwenDiT.hidden), pack: gpPack, commands: commands)
+ }
+ try mark("norm2", normed)
+ // With the MLP out's rows packed by the SwiGLU, gate's and proj's Neural Engine rows stay in their output
+ // surfaces and the SwiGLU reads them there (no fp32 unpacking of 2 × 88 MB at 1024²).
+ let mlpOut = block + "img_mlp.out.weight"
+ let mlpPack = rowPack(count: count, k: QwenDiT.hidden, n: d, readers: [mlpOut])
+ let keep = mlpPack != nil
+ try linear(gateProj[0], a: normed, aRow: first, c: h1, m: count, cRow: first, next: gateProj[1], readers: gateProj,
+ keepOutput: keep, commands: commands)
+ let gateSet = coexecutor?.lastSet ?? 0
+ try linear(gateProj[1], a: normed, aRow: first, c: h2, m: count, cRow: first, next: mlpOut, readers: gateProj,
+ keepOutput: keep, commands: commands)
+ try swiglu(first: first, count: count, half: halfOutput(count: count, k: QwenDiT.hidden, n: d), pack: mlpPack,
+ tiles: keep ? (gateSet, coexecutor?.lastSet ?? 0, gateProj) : nil, commands: commands)
+ // The MLP's residual goes with the next layer's LN1 (`deferredResidual`), the Neural Engine's rows of the MLP
+ // out read there from its output surfaces when that LN1 packs them.
+ let nextQKV = ["to_q", "to_k", "to_v"].map { "transformer_blocks.\(layer + 1).attn.\($0).weight" }
+ let carry = fuse && layer + 1 < QwenDiT.layers
+ let keepMLP = carry && rowPack(count: count, k: d, n: d, readers: nextQKV) != nil
+ try linear(mlpOut, a: h1, aRow: first, c: o, m: count, cRow: first,
+ next: layer + 1 < QwenDiT.layers ? nextQKV[0] : nil, keepOutput: keepMLP, commands: commands)
try mark("mlp", o)
- try residual(gate: 3 * d, rows: rows, commands: commands)
+ if carry {
+ deferredResidual = Deferred(tiles: keepMLP ? (coexecutor!.lastSet, mlpOut, 3) : nil)
+ } else {
+ try residual(gate: 3 * d, rows: rows, commands: commands)
+ }
}
// MARK: - encoders
@@ -582,7 +618,7 @@ public final class QwenDiT {
/// GPU's ~1.8 ms per 4096 × 4096 weight otherwise ran alone between products).
/// `readers`: every weight that reads this same A (q, k, v; gate, proj), so the Neural Engine's rows are packed once.
func linear(_ name: String, a: Arena.Slice, aRow: Int = 0, c: Arena.Slice, m: Int, cRow: Int = 0,
- next: String? = nil, readers: [String] = [], commands: MTLCommandBuffer) throws {
+ next: String? = nil, readers: [String] = [], keepOutput: Bool = false, commands: MTLCommandBuffer) throws {
let tensor = try weight(name)
let (kk, nn) = (tensor.columns, tensor.rows)
let m = QwenDiT.padded(m)
@@ -644,7 +680,8 @@ public final class QwenDiT {
}
let ready = castInput.map { $0 == (a.offset, aRow, kk) } ?? false
try coexecutor.encode(weight: name, a: operand(a, aRow, kk), w: operand(reserve, 0, nn), c: operand(c, cRow, nn),
- m: m, k: kk, n: nn, commands: commands, readers: readers, castReady: ready, ahead: ahead, gpuRows: gpuRows)
+ m: m, k: kk, n: nn, commands: commands, readers: readers, castReady: ready, keepOutput: keepOutput,
+ ahead: ahead, gpuRows: gpuRows)
castInput = c.offset == a.offset ? nil : (a.offset, aRow, kk)
} else {
try gpuRows(m)
@@ -681,6 +718,79 @@ public final class QwenDiT {
return (h.buffer, HalfOut(rows: UInt32(count), halfRows: UInt32(h.rows), fullFrom: probing ? 0 : UInt32(h.rows)))
}
+ /// Where a row kernel of `count` real rows packs the Neural Engine's rows of the split product that follows
+ /// (`Coexecutor.RowPack`); nil: they are written in fp32 and packed by the product.
+ func rowPack(count: Int, k: Int, n: Int, readers: [String]) -> Coexecutor.RowPack? {
+ guard fusedCast, !profiling, !probing, let coexecutor else { return nil }
+ return coexecutor.rowPack(m: QwenDiT.padded(count), k: k, n: n, readers: readers)
+ }
+
+ struct AnePack { var row0, real, surfaces, kSplit: UInt32 }
+
+ /// Binds a row pack's σ, bounds and surfaces at `index` on.
+ func bind(_ pack: Coexecutor.RowPack, _ e: MTLComputeCommandEncoder, real: Int, from index: Int) {
+ var a = AnePack(row0: UInt32(pack.first), real: UInt32(real), surfaces: UInt32(pack.surfaces.count), kSplit: pack.kSplit ? 1 : 0)
+ e.setBytes(&a, length: MemoryLayout<AnePack>.stride, index: index)
+ e.setBuffer(pack.sigma, offset: 0, index: index + 1)
+ for i in 0..<3 { e.setBuffer(pack.bounds[min(i, pack.bounds.count - 1)], offset: pack.boundOffset, index: index + 2 + i) }
+ for i in 0..<3 { e.setBuffer(pack.surfaces[min(i, pack.surfaces.count - 1)], offset: 0, index: index + 5 + i) }
+ }
+
+ /// **The residual and the LN after it in one pass** (`residual_layernorm`, `residual_layernorm_ane`): x is read
+ /// once instead of twice, and the Neural Engine's rows of `o` come from the product's output surfaces (`tiles`: set,
+ /// weight, K parts) instead of an fp32 unpacking. After to_out (with LN2) and after the MLP out (with the next
+ /// layer's LN1). The same values to the bit. `KOKON_FUSED_RESIDUAL=0`: the separate kernels.
+ let fusedResidual = ProcessInfo.processInfo.environment["KOKON_FUSED_RESIDUAL"] != "0"
+ struct Deferred { let tiles: (set: Int, weight: String, kParts: Int)? }
+ /// The MLP residual of the layer before, encoded with this layer's LN1.
+ var deferredResidual: Deferred?
+
+ /// `x += tanh(gate) · o` then `LN(x) · mod` on `rows` (see `fusedResidual`).
+ func residualNorm(gate: Int, rows: Range<Int>, prefixMod: Int, targetMod: Int, half: (buffer: MTLBuffer, out: HalfOut)?,
+ pack: Coexecutor.RowPack?, tiles: (set: Int, weight: String, kParts: Int)?, commands: MTLCommandBuffer) throws {
+ let d = QwenDiT.dim
+ // Packing the Neural Engine's rows needs them from the product's surfaces; otherwise the residual kernel takes
+ // every row (the Neural Engine's unpacked in fp32 by the product).
+ guard (pack == nil) == (tiles == nil) else {
+ try residual(gate: gate, rows: rows, commands: commands)
+ return try layerNorm(x, into: normed, rows: rows, prefixMod: prefixMod, targetMod: targetMod, half: half, pack: pack, commands: commands)
+ }
+ let at = rows.lowerBound * d * 4
+ let h = half?.out ?? QwenDiT.fullOnly(rows.count)
+ var params = NormParams(columns: UInt32(d), targetStart: UInt32(max(0, sequence.prefix - rows.lowerBound)), eps: QwenDiT.eps, half: h)
+ try compute("residual_layernorm", commands: commands) { e in
+ e.setBuffer(arena.buffer, offset: x.offset + at, index: 0)
+ e.setBuffer(arena.buffer, offset: normed.offset + at, index: 1)
+ e.setBuffer(arena.buffer, offset: modulation.offset + prefixMod * 4, index: 2)
+ e.setBuffer(arena.buffer, offset: modulation.offset + targetMod * 4, index: 3)
+ e.setBytes(&params, length: MemoryLayout<NormParams>.stride, index: 4)
+ e.setBuffer(half?.buffer ?? arena.buffer, offset: half == nil ? normed.offset + at : 0, index: 5)
+ e.setBuffer(arena.buffer, offset: o.offset + at, index: 6)
+ e.setBuffer(arena.buffer, offset: modulation.offset + gate * 4, index: 7)
+ e.setBuffer(arena.buffer, offset: modulation.offset + (5 * d + gate) * 4, index: 8)
+ e.dispatchThreadgroups(MTLSize(width: pack.map { $0.first } ?? max(rows.count, Int(h.halfRows)), height: 1, depth: 1),
+ threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
+ }
+ if let pack, let tiles, let c = coexecutor {
+ try compute("residual_layernorm_ane", commands: commands) { e in
+ e.setBuffer(arena.buffer, offset: x.offset + at, index: 0)
+ e.setBuffer(arena.buffer, offset: modulation.offset + prefixMod * 4, index: 1)
+ e.setBuffer(arena.buffer, offset: modulation.offset + targetMod * 4, index: 2)
+ e.setBytes(&params, length: MemoryLayout<NormParams>.stride, index: 3)
+ bind(pack, e, real: rows.count, from: 4)
+ for t in 0..<3 { e.setBuffer(c.slots[tiles.set][min(t, tiles.kParts - 1)].y, offset: 0, index: 12 + t) }
+ e.setBuffer(c.columns(tiles.weight), offset: 0, index: 15)
+ var parts = UInt32(tiles.kParts)
+ e.setBytes(&parts, length: 4, index: 16)
+ e.setBuffer(arena.buffer, offset: modulation.offset + gate * 4, index: 17)
+ e.setBuffer(arena.buffer, offset: modulation.offset + (5 * d + gate) * 4, index: 18)
+ e.dispatchThreadgroups(MTLSize(width: pack.rows, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
+ }
+ c.packedByRowKernel(offset: normed.offset + at, readers: pack.readers)
+ }
+ if half != nil { castInput = (normed.offset, rows.lowerBound, d) }
+ }
+
/// No fp16 copy: every row in fp32.
static func fullOnly(_ rows: Int) -> HalfOut { HalfOut(rows: UInt32(rows), halfRows: 0, fullFrom: 0) }
struct HeadParams { var heads: UInt32; var rows: UInt32; var eps: Float }
@@ -689,7 +799,7 @@ public final class QwenDiT {
/// `out = LN(x) · mod[band]`, `prefixMod` / `targetMod` floats into the modulation slice; `half`: the next
/// product's fp16 input too (`halfOutput`).
func layerNorm(_ input: Arena.Slice, into out: Arena.Slice, rows: Range<Int>, prefixMod: Int, targetMod: Int,
- half: (buffer: MTLBuffer, out: HalfOut)? = nil, commands: MTLCommandBuffer) throws {
+ half: (buffer: MTLBuffer, out: HalfOut)? = nil, pack: Coexecutor.RowPack? = nil, commands: MTLCommandBuffer) throws {
let at = rows.lowerBound * QwenDiT.dim * 4
let h = half?.out ?? QwenDiT.fullOnly(rows.count)
try compute("layernorm_modulate", commands: commands) { e in
@@ -701,14 +811,29 @@ public final class QwenDiT {
eps: QwenDiT.eps, half: h)
e.setBytes(&params, length: MemoryLayout<NormParams>.stride, index: 4)
e.setBuffer(half?.buffer ?? arena.buffer, offset: half == nil ? out.offset + at : 0, index: 5)
- e.dispatchThreadgroups(MTLSize(width: max(rows.count, Int(h.halfRows)), height: 1, depth: 1),
+ e.dispatchThreadgroups(MTLSize(width: pack.map { $0.first } ?? max(rows.count, Int(h.halfRows)), height: 1, depth: 1),
threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
}
+ if let pack {
+ try compute("layernorm_modulate_ane", commands: commands) { e in
+ e.setBuffer(arena.buffer, offset: input.offset + at, index: 0)
+ e.setBuffer(arena.buffer, offset: modulation.offset + prefixMod * 4, index: 1)
+ e.setBuffer(arena.buffer, offset: modulation.offset + targetMod * 4, index: 2)
+ var params = NormParams(columns: UInt32(QwenDiT.dim), targetStart: UInt32(max(0, sequence.prefix - rows.lowerBound)),
+ eps: QwenDiT.eps, half: h)
+ e.setBytes(&params, length: MemoryLayout<NormParams>.stride, index: 3)
+ bind(pack, e, real: rows.count, from: 4)
+ e.dispatchThreadgroups(MTLSize(width: pack.rows, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
+ }
+ coexecutor?.packedByRowKernel(offset: out.offset + at, readers: pack.readers)
+ }
if half != nil { castInput = (out.offset, rows.lowerBound, QwenDiT.dim) }
}
/// `h1 = SiLU(h1) ⊙ h2` on rows `[first, first + count)`; `half`: the MLP out's fp16 input too.
- func swiglu(first: Int, count: Int, half: (buffer: MTLBuffer, out: HalfOut)?, commands: MTLCommandBuffer) throws {
+ /// `tiles`: gate's and proj's Neural Engine rows read from their output surfaces (sets, weights) instead of h1, h2.
+ func swiglu(first: Int, count: Int, half: (buffer: MTLBuffer, out: HalfOut)?, pack: Coexecutor.RowPack? = nil,
+ tiles: (gate: Int, proj: Int, weights: [String])? = nil, commands: MTLCommandBuffer) throws {
let w = QwenDiT.hidden
let h = half?.out ?? QwenDiT.fullOnly(count)
try compute("swiglu", commands: commands) { e in
@@ -717,9 +842,28 @@ public final class QwenDiT {
var params = SwigluParams(columns: UInt32(w), half: h)
e.setBytes(&params, length: MemoryLayout<SwigluParams>.stride, index: 2)
e.setBuffer(half?.buffer ?? arena.buffer, offset: half == nil ? h1.offset : 0, index: 3)
- e.dispatchThreads(MTLSize(width: max(count, Int(h.halfRows)) * w, height: 1, depth: 1),
+ e.dispatchThreads(MTLSize(width: (pack.map { $0.first } ?? max(count, Int(h.halfRows))) * w, height: 1, depth: 1),
threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
}
+ if let pack, let tiles, let c = coexecutor {
+ try compute("swiglu_ane_tiles", commands: commands) { e in
+ for t in 0..<3 { e.setBuffer(c.slots[tiles.gate][t].y, offset: 0, index: t) }
+ for t in 0..<3 { e.setBuffer(c.slots[tiles.proj][t].y, offset: 0, index: 3 + t) }
+ e.setBuffer(c.columns(tiles.weights[0]), offset: 0, index: 6)
+ e.setBuffer(c.columns(tiles.weights[1]), offset: 0, index: 7)
+ bind(pack, e, real: count, from: 8)
+ e.dispatchThreadgroups(MTLSize(width: pack.rows, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
+ }
+ c.packedByRowKernel(offset: h1.offset + first * w * 4, readers: pack.readers)
+ } else if let pack {
+ try compute("swiglu_ane", commands: commands) { e in
+ e.setBuffer(arena.buffer, offset: h1.offset + first * w * 4, index: 0)
+ e.setBuffer(arena.buffer, offset: h2.offset + first * w * 4, index: 1)
+ bind(pack, e, real: count, from: 2)
+ e.dispatchThreadgroups(MTLSize(width: pack.rows, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
+ }
+ coexecutor?.packedByRowKernel(offset: h1.offset + first * w * 4, readers: pack.readers)
+ }
if half != nil { castInput = (h1.offset, first, w) }
}