Skip to content

armgit / kokon

Public Imported from GitHub
0

Edit photos by describing the change, on your own GPU. A free, open-source alternative to Nano Banana and ChatGPT image editing: photos stay on your computer, no daily limit. One installer for Windows, Mac and Linux sets up the best open model (Qwen-Image 2.1). MCP server for AI agents.

Mac engine: int8 Neural Engine with finer scales measured and dropped

Gabriele Lozuponefb9cc5f7700d
diff --git a/engine-mac/README.md b/engine-mac/README.md
index 685e7dc..7c5d2d9 100644
--- a/engine-mac/README.md
+++ b/engine-mac/README.md
@@ -271,7 +271,7 @@ correctness test that runs on any Mac (`PerChipTests`). Only the M2 Pro has meas
rows). Opt-in on every chip, as their error does not depend on the chip. M2 Pro: one 1792 × 4096 × 4096 tile 10.1 ms
in fp16, 8.3 ms with int8 weights, 7.0 ms in int8 (`check-ane8`); per evaluation 7.42 → 7.20 / 7.00 s; but the
velocity at 896² goes from 7.6e-4 to 1.1e-2 / 4.1e-2 (one int8 scale per 4 096 weights where Q4_K has one per 32;
- activation rows with outliers). Finer weight scales are the way to try next.
+ activation rows with outliers). **Finer scales do not fix it** (2026-10-25, below): int8 stays opt-in.
- **Attention reading K and V straight from memory** (`flash_attention_direct`) on Apple9+ GPUs (M3 and later), as
MFA's tables prefer there (`KOKON_ATTENTION_DIRECT=0` off); 142 ms on the M2 Pro against 91, unmeasured on an M3+.
- **Starting shares per chip**: the balancer's M2 Pro rates scaled by the chip's estimated units (GPU cores × clock,
@@ -297,6 +297,24 @@ chip. Close other apps and keep the Mac plugged in: the units share one power an
## Tried and not kept, the VAE, credits
+**int8 on the Neural Engine with finer scales: measured, not kept** (M2 Pro, 2026-10-25). First the accuracy, with
+int8 rounding simulated on the fp16 path (each column's weights, or each row's values, rounded to 127 levels of its
+own piece of K, then packed as fp16), velocity at 896² against diffusers fp32 (`dit-t2i-896`; fp16 path 7.6e-4):
+
+| int8 grid | weights | rows |
+|---|---|---|
+| one scale per column / row (today's int8 path) | 1.1e-2 (the measured int8 program's) | 3.0e-2 |
+| pieces of 512 / 256 / 128 | 1.3e-2 / 6.4e-3 / 7.7e-3 | – / 1.4e-2 / – |
+| pieces of 64 / 32 | – / 4.8e-3 | 8.1e-3 / – |
+| weights and rows, pieces of 256 | 1.7e-2 | |
+
+So even one scale per 32 weights stays 6× fp16's error: Q4_K's 4-bit grid rounded again to 127 levels. Only Q4_K's own
+codes with its per-32 scale and offset would be exact, and the scales of a weight given at run time must be program
+inputs; timed on one 1792 × 4096 × 4096 tile, the Neural Engine then runs slower than in fp16: fp16 8.2 ms, int8 with
+one constant scale 4.8, int8 with scales per 256 as an input 10.0, per 32 10.6, per 32 with offsets 11.2 (K in two
+pieces of 2 048: 10.9 / 11.7). The int8 path stays opt-in (`KOKON_ANE_INT8`); activation smoothing was not tried,
+since the weights alone already miss the target.
+
MFA's GEMM (libnnc, 48 × 48 blocks, 2 × 2 SIMD groups) was tried against MPS: its async copies are an AIR intrinsic
that macOS 27's run-time compiler refuses (libnnc ships prebuilt metallibs for it), and without them it ran at
3.6–4.0 TFLOP/s against MPS's 5.1–5.6 at 1472 × 4096 × 4096, so MPS stays.
diff --git a/engine-mac/Sources/KokonEngine/Coexecution.swift b/engine-mac/Sources/KokonEngine/Coexecution.swift
index 1614c68..ccb55da 100644
--- a/engine-mac/Sources/KokonEngine/Coexecution.swift
+++ b/engine-mac/Sources/KokonEngine/Coexecution.swift
@@ -68,6 +68,8 @@ final class Coexecutor {
/// chip: the error does not depend on the chip, and on the M2 Pro (2026-10-25) one 1792 × 4096 × 4096 tile ran
/// 7.0 ms in int8, 8.3 ms with int8 weights, 10.1 ms in fp16 (`check-ane8`), but the velocity at 896² went from
/// 7.6e-4 to 4.1e-2 (int8) and 1.1e-2 (int8 weights; one scale per 4 096 values where Q4_K has one per 32).
+ /// Finer scales do not close the gap (README, "Tried and not kept": 4.8e-3 with one per 32, and scales given as
+ /// program inputs run slower than fp16).
/// `bench` measures both on M3 and later. Falls back to fp16 when its programs cannot be built (macOS 26+).
private(set) var aneInt8: Bool
/// With `aneInt8`, the rows stay fp16 (int8 weights only, W8A16: `KOKON_ANE_INT8=w8`).