Edit photos by describing the change, on your own GPU. A free, open-source alternative to Nano Banana and ChatGPT image editing: photos stay on your computer, no daily limit. One installer for Windows, Mac and Linux sets up the best open model (Qwen-Image 2.1). MCP server for AI agents.
TypeScript ai-photo-editorcomfyui-alternativegenerative-fillimage-editingimage-generationinpaintinglinuxlocal-aimacosmcpmcp-servernano-banana-alternativeofflinephoto-editorprivacyqwen-imageqwen-image-2-1stable-diffusion-cppvulkanwindows
Mac engine: Prompt+ (text_gen) on the engine
Gabriele Lozupone2e0e235cdcf3
diff --git a/docs/STATUS.md b/docs/STATUS.mdindex de83013..0fbea5a 100644--- a/docs/STATUS.md+++ b/docs/STATUS.md@@ -24,9 +24,10 @@ Windows, macOS (Apple Silicon, 15+) and Linux (`.deb`, `.rpm`). Installed copies- **LoRA is beta** and tested on Windows only; reports from macOS and Linux are welcome([#33](https://github.com/GabrieleLozupone/kokon/issues/33) has a product-shot test).- **Macs are slow** compared with NVIDIA cards (about 4 minutes for 1 MP on an M2 Pro 16 GB). Kokon's own Mac- engine is in development on the `macos-optimization` branch (`engine-mac/`, not released): text to image only,- about 64–67 s for 1 MP on the same Mac through the app's API (peak memory 5 GB), tested from the app, as an experimental switch in Settings → Performance- (off by default). Edits, LoRAs, Prompt+ and the upscaler still need the standard engine.+ engine is in development on the `macos-optimization` branch (`engine-mac/`, not released): text to image and+ Prompt+ (a prompt written in about 7–9 s), about 64–67 s for 1 MP on the same Mac through the app's API (peak memory 5 GB),+ tested from the app, as an experimental switch in Settings → Performance (off by default). Edits, LoRAs and the+ upscaler still need the standard engine.- The Windows installer is not code-signed: SmartScreen or Smart App Control may block it (README FAQ: Unblock).- HEIC photos can't be opened on Linux yet.diff --git a/engine-mac/README.md b/engine-mac/README.mdindex 7c5d2d9..8cd2365 100644--- a/engine-mac/README.md+++ b/engine-mac/README.md@@ -16,7 +16,7 @@ swift test # no model needed.build/release/kokon-engine-mac generate <encoder.gguf> <dit.gguf> <vae.safetensors> "a red fox in the snow" \[--size 1024x1024] [--seed 42] [--out out.png].build/release/kokon-engine-mac serve --diffusion-model <dit.gguf> --vae <vae.safetensors> --llm <encoder.gguf> \- [--listen-port 1234] # sd-server's /sdcpp/v1 API, text to image+ [--listen-port 1234] # sd-server's /sdcpp/v1 API, text to image and Prompt+.build/release/kokon-engine-mac bench # a few minutes: speed and accuracy of this Mac's paths, a report.build/release/kokon-engine-mac check-na # the Metal 4 matrix product against MPS.build/release/kokon-engine-mac check-ane8 # the int8 Neural Engine programs against the fp16 one@@ -47,7 +47,8 @@ Windows and Linux.| `QwenTokenizer.swift` | Qwen's byte-level BPE tokenizer, read from the encoder GGUF's metadata || `QwenTextEncoder.swift`, `Kernels/text.metal` | the text encoder: Qwen3-VL-8B's text decoder on the GPU, diffusers' prompt template || `QwenPipeline.swift` | text to image: noise, Turbo's 8 flow-Euler steps (`planner::TURBO_SIGMAS`), VAE, PNG; models kept loaded for the server |-| `EngineServer.swift`, `HTTPServer.swift` | `serve`: sd-server's `/sdcpp/v1` API (capabilities, img_gen, jobs, cancel) and its log lines |+| `EngineServer.swift`, `HTTPServer.swift` | `serve`: sd-server's `/sdcpp/v1` API (capabilities, img_gen, jobs, cancel, text_gen) and its log lines |+| `QwenTextGeneration.swift`, `Kernels/decode.metal` | Prompt+: the text encoder's LLM writes, K/V cache, products on the Q4_K / Q6_K blocks, sd.cpp's sampler || `Chip.swift` | this Mac's chip, GPU family, cores and memory; each unit's estimated speed against the M2 Pro's || `NeuralAccelerators.swift`, `Kernels/metal4/na_matmul.metal` | the GPU's rows of a product on Metal 4 tensor operations (the M5's neural accelerators) || `Bench.swift` | `bench`: the report testers paste in GitHub |@@ -248,7 +249,8 @@ Engine's programs, are not pipelines an archive holds. Only an M2 Pro has been m`kokon-engine-mac serve` answers sd-server's `/sdcpp/v1` API (capabilities, img_gen, jobs/{id}, cancel), soqls-core, the app, remote access and MCP use it as they use sd-server; it prints the log lines qls-core's`ProgressParser` reads. Text to image only for now: reference images, masks, LoRAs and second passes are refused-with 400 (qls-core says so before submitting), upscaling falls back to Lanczos, Prompt+ is off. A running image+with 400 (qls-core says so before submitting), upscaling falls back to Lanczos. Prompt+ works on text-to-image prompts+(`text_gen`, below); with images (edits, Compose) it needs the vision tower. A running imagecancels within a DiT layer (sd-server answers 409 and qls-core restarts it). A new prompt length rebuilds the DiT'sworking memory in ~0.5 s.@@ -258,6 +260,39 @@ operations and tiles (below): **first image 67.0 s, the next ones 63.9–64.5 s,peak 9.6 GB). Keeping the compiled VAE between images now costs only 1.7 GB, but it did not pay on 16 GB: its decode3.2 s instead of 4.0, the images 68.4–70.3 s (with the DiT kept too 70.8 / 68.3 s; single runs in a row, not alternated).+**Prompt+ (`POST /sdcpp/v1/text_gen`)**: the text encoder's LLM writes the prompt, as Kokon's sd.cpp does, on the same+Qwen3-VL Q4_K_M file (`QwenTextGeneration.swift`, `Kernels/decode.metal`): sd.cpp's chat template and stop tokens, its+sampler step for step (presence penalty, top-k, temperature, top-p, one draw from mt19937_64: a seed writes the same+text where no two logits nearly tie), its stream of whole UTF-8 characters, its Server-Sent Events, and a dropped+connection stops it. The prompt runs through the encoder's forward, which keeps each layer's K and V in an fp16 cache+(144 KB a token); then each token reads every weight once, so the decode's matrix-vector products read the Q4_K and+Q6_K blocks where they lie in the mapped file. Text and image generation take turns on the GPU.++Against transformers (`check-textgen` + `tools/reference_text_generation.py`: the engine writes greedily, transformers'+fp32 decoder layers then run once on the prompt and those tokens; 68-token prompt, 48 tokens, M2 Pro, 2026-10-25): the+prompt's ids identical, the first token's logits within 3.2e-6, the next ones within 4.6e-4 (the fp16 cache: with an fp32+cache every step within 3.2e-6), the greedy choice the same at all 48 (one near-tie at 0.04 among them).++```bash+.build/release/kokon-engine-mac check-textgen <encoder.gguf> ~/kokon-bench/ref/textgen-cat # writes engine.json+~/kokon-bench/venv-oracle/bin/python -I tools/reference_text_generation.py <encoder.gguf> <processor dir> <text_encoder dir> ~/kokon-bench/ref/textgen-cat+.build/release/kokon-engine-mac check-textgen <encoder.gguf> ~/kokon-bench/ref/textgen-cat # compares+.build/release/kokon-engine-mac time-matvec <encoder.gguf> # the decode's products alone, GB/s+```++Speed (M2 Pro 16 GB, 2026-10-25): **a token in 32 ms (31 tokens/s; ~4.9 GB read a token, ~155 GB/s)**; Kokon's+built-in system prompt and a short request, through the server: 481 prompt tokens, 142 written, 6.6–7.8 s (prefill+1.8–2.4 s, 26.8–29.5 tokens/s as the cache grows; 8.2 s while the sampler looked the presence penalty up in a set for+each of the 151 936 tokens, a dense array now as sd.cpp's). The products' kernels (`time-matvec`, GB/s on the file's weights): one lane per 64+values, one row per SIMD group: Q4_K 25–49, Q6_K 39–53, 10.5 tokens/s; 8 lanes a block with wide loads, 2 rows: Q6_K+115–182; Q4_K 4 rows: 51–68; its scales and mins unpacked from one 16-byte load without a branch (they were half its+time): 112–120. One compute encoder per token instead of one per kernel (~500): no change (32.5–34.9 ms against+31.8–32.5), not kept. Of a 12-token prefill's 2.2–2.5 s, ~2 s is opening the encoder (0.46 s with it warm): when the+encoder is not kept between images (below 32 GB), the one Prompt+ opened is handed to the next image's encode. Through+the API at 1024², alternated (`tools/api_check.py --enhance`): the image's text encoder 0.88 s after Prompt+ against+2.1–2.65 s for a plain image, peak footprint 5.1 GB either way; Prompt+ adds 8.7–9.0 s to an image (75.4–77.2 s against+65.6–71.3).+## Per-chip pathsSelected at run time from the GPU family and the chip (`Chip.swift`), each with the M2 path as its fallback and adiff --git a/engine-mac/Sources/KokonEngine/EngineServer.swift b/engine-mac/Sources/KokonEngine/EngineServer.swiftindex 1b4c0b9..7394acf 100644--- a/engine-mac/Sources/KokonEngine/EngineServer.swift+++ b/engine-mac/Sources/KokonEngine/EngineServer.swift@@ -7,10 +7,11 @@ import Foundation/// POST /sdcpp/v1/img_gen 202 {id, kind, status, created, poll_url}: the job is queued/// GET /sdcpp/v1/jobs/{id} {status: queued|generating|completed|failed|cancelled, result, error}/// POST /sdcpp/v1/jobs/{id}/cancel 200 {status: cancelled}: a running image stops within a DiT layer+/// POST /sdcpp/v1/text_gen Prompt+: the text encoder's LLM writes, streamed as Server-Sent Events////// Text to image only for now: requests with reference images, a mask, an init image, LoRAs or a second pass are-/// refused with 400 and say so. Jobs run one at a time; the text encoder, the DiT and the compiled VAE stay loaded-/// between them (`QwenPipeline.keepModels`).+/// refused with 400 and say so, and so is text generation with images. Jobs and text generation run one at a time;+/// what stays loaded between them depends on the Mac (`QwenPipeline.keepModels`).////// Progress goes to stdout as sd-server's log lines, the ones qls-core's `ProgressParser` reads (process.rs):/// `generate_image WxH`, `sampling using`, `generating image: i/n - seed s`, step bars `| i/n - x.xxs/it`,@@ -95,6 +96,8 @@ public final class EngineServer: @unchecked Sendable {private var running: Job?private let lock = NSLock()private let wake = DispatchSemaphore(value: 0)+ /// Held by a running image or text generation: one GPU user at a time.+ private let gpu = NSLock()/// Lines to stdout (qls-core reads them from the log file it redirects stdout to).public var log: @Sendable (String) -> Void = { line inFileHandle.standardOutput.write(Data((line + "\n").utf8))@@ -125,7 +128,8 @@ public final class EngineServer: @unchecked Sendable {case ("GET", let p) where p.count == 2 && p[0] == "jobs":return lock.withLock { jobs[p[1]].map { .json(200, describe($0)) } ?? .error(404, "unknown job \(p[1])") }case ("POST", let p) where p.count == 3 && p[0] == "jobs" && p[2] == "cancel": return cancel(p[1])- case (_, ["upscale"]), (_, ["text_gen"]):+ case ("POST", ["text_gen"]): return textGen(request.body)+ case (_, ["upscale"]):return .error(404, "Kokon's Mac engine has no \(parts[2]) endpoint yet")default: return .error(404, "not found: \(request.method) \(request.path)")}@@ -143,12 +147,13 @@ public final class EngineServer: @unchecked Sendable {"schedulers": [String](),"output_formats": ["png"],"limits": ["min_width": 64, "max_width": 4096, "min_height": 64, "max_height": 4096,- "max_batch_count": EngineServer.maxBatch, "max_queue_size": EngineServer.maxQueue],+ "max_batch_count": EngineServer.maxBatch, "max_queue_size": EngineServer.maxQueue,+ "max_text_tokens": EngineServer.maxTextTokens, "max_text_images": 0],"features_by_mode": ["img_gen": feature],"defaults_by_mode": ["img_gen": ["sample_steps": QwenPipeline.turboSigmas.count - 1, "txt_cfg": 1]],"upscalers": [Any](),"upscale": false,- "text_generation": false,+ "text_generation": true,]}@@ -193,6 +198,97 @@ public final class EngineServer: @unchecked Sendable {return out}+ // MARK: - text generation (Prompt+)++ /// The fields of `text_gen` (sd-server's names and checks, `examples/server/routes_sdcpp.cpp`).+ struct TextRequest: Sendable {+ let system, prompt: String+ let maxTokens: Int+ let sampling: TextSampler.Parameters+ let seed: Int64+ let stream: Bool++ static func parse(_ body: Data) -> (TextRequest?, String?) {+ guard let json = (try? JSONSerialization.jsonObject(with: body)) as? [String: Any] else { return (nil, "body must be an object") }+ guard let prompt = json["prompt"] as? String, !prompt.isEmpty else { return (nil, "prompt is required") }+ guard let system = json["system_prompt"] as? String else {+ return (nil, "system_prompt is required (an empty string means no system prompt)")+ }+ let maxTokens = json["max_tokens"] as? Int ?? 512+ guard (1...EngineServer.maxTextTokens).contains(maxTokens) else { return (nil, "max_tokens must be between 1 and \(EngineServer.maxTextTokens)") }+ if let images = json["images"] as? [Any], !images.isEmpty {+ return (nil, "Kokon's Mac engine cannot look at images yet (no vision tower): Prompt+ works on text-to-image prompts only")+ }+ var sampling = TextSampler.Parameters(temperature: 0, topP: 1, topK: 0, presencePenalty: 0)+ var seed: Int64 = -1+ if let s = json["sampling"] {+ guard let s = s as? [String: Any] else { return (nil, "sampling must be an object") }+ func number(_ key: String) -> Double? { (s[key] as? NSNumber)?.doubleValue }+ sampling.temperature = Float(number("temperature") ?? 0)+ sampling.topP = Float(number("top_p") ?? 1)+ sampling.topK = Int(number("top_k") ?? 0)+ sampling.presencePenalty = Float(number("presence_penalty") ?? 0)+ seed = (s["seed"] as? NSNumber)?.int64Value ?? -1+ guard sampling.temperature.isFinite, sampling.temperature >= 0, sampling.topP.isFinite, sampling.topP > 0, sampling.topP <= 1,+ sampling.topK >= 0, sampling.presencePenalty.isFinite else {+ return (nil, "sampling needs temperature >= 0, 0 < top_p <= 1, top_k >= 0 and a finite presence_penalty")+ }+ }+ return (TextRequest(system: system, prompt: prompt, maxTokens: maxTokens, sampling: sampling, seed: seed,+ stream: json["stream"] as? Bool ?? true), nil)+ }+ }++ static let maxTextTokens = 4096++ /// Synchronous like sd-server's: it waits for a running image (one GPU user at a time) and holds the models while it+ /// writes. Streamed (the default) as Server-Sent Events: `delta` {text} per piece, then `done` {text, finish_reason,+ /// seed, usage, timings} or `error` {error}; closing the connection stops it.+ func textGen(_ body: Data) -> HTTPServer.Response {+ let (parsed, reason) = TextRequest.parse(body)+ guard let request = parsed else { return .error(400, reason ?? "bad request") }+ guard request.stream else {+ let outcome = Result { try gpu.withLock { try generateText(request) { _ in true } } }+ switch outcome {+ case .success(let result): return .json(200, EngineServer.done(result))+ case .failure(let error):+ log("[ERROR ] engine-mac - text generation failed: \(error)")+ return .error(500, "text generation failed")+ }+ }+ return .streaming(contentType: "text/event-stream") { [self] write in+ // Before waiting for a running image, so the client gets the headers at once.+ guard write(Data(": waiting for the model\n\n".utf8)) else { return }+ func event(_ name: String, _ object: Any) -> Bool {+ let data = (try? JSONSerialization.data(withJSONObject: object)) ?? Data("{}".utf8)+ return write(Data("event: \(name)\ndata: ".utf8) + data + Data("\n\n".utf8))+ }+ let outcome = Result { try gpu.withLock { try generateText(request) { piece in event("delta", ["text": piece]) } } }+ switch outcome {+ case .success(let result): _ = event("done", EngineServer.done(result))+ case .failure(let error):+ log("[ERROR ] engine-mac - text generation failed: \(error)")+ _ = event("error", ["error": "text generation failed"])+ }+ }+ }++ private func generateText(_ r: TextRequest, onPiece: (String) -> Bool) throws -> QwenTextGenerator.Result {+ let result = try pipeline.generateText(system: r.system, prompt: r.prompt, maxTokens: r.maxTokens, sampling: r.sampling,+ seed: r.seed, onPiece: onPiece)+ log(String(format: "[INFO ] engine-mac - text generation: %d prompt tokens, %d generated in %.2fs (prefill %.2fs, %.1f tok/s), seed %lld",+ result.promptTokens, result.generatedTokens, result.prepare + result.prefill + result.decode, result.prefill,+ result.decode > 0 ? Double(result.tokens.count - 1) / result.decode : 0, result.seed))+ return result+ }++ /// sd-server's `done` object.+ static func done(_ r: QwenTextGenerator.Result) -> [String: Any] {+ ["text": r.text, "finish_reason": r.finish.rawValue, "seed": r.seed,+ "usage": ["prompt_tokens": r.promptTokens, "completion_tokens": r.generatedTokens],+ "timings": ["prepare_s": r.prepare, "image_s": 0, "prefill_s": r.prefill, "decode_s": r.decode]]+ }+// MARK: - the jobsprivate func work() {@@ -206,7 +302,7 @@ public final class EngineServer: @unchecked Sendable {running = jobreturn job}) else { continue }- let outcome: Result<[String], Error> = autoreleasepool { Result { try generate(job) } }+ let outcome: Result<[String], Error> = autoreleasepool { Result { try gpu.withLock { try generate(job) } } }lock.withLock {running = niljob.completed = Int(Date().timeIntervalSince1970)diff --git a/engine-mac/Sources/KokonEngine/HTTPServer.swift b/engine-mac/Sources/KokonEngine/HTTPServer.swiftindex 2e5fbcd..af6c8c3 100644--- a/engine-mac/Sources/KokonEngine/HTTPServer.swift+++ b/engine-mac/Sources/KokonEngine/HTTPServer.swift@@ -1,8 +1,9 @@import Foundation/// **A minimal HTTP/1.1 server on a loopback port**, enough for qls-core's client (reqwest): one request per-/// connection (`Connection: close`), a `Content-Length` body, JSON in and out. POSIX sockets, a thread per-/// connection: the engine serves one local client, and the slow part of every request is the GPU, not the socket.+/// connection (`Connection: close`), a `Content-Length` body, JSON in and out, or a body written as it is made+/// (`Response.stream`, chunked: `text_gen`'s Server-Sent Events). POSIX sockets, a thread per connection: the engine+/// serves one local client, and the slow part of every request is the GPU, not the socket.public final class HTTPServer: @unchecked Sendable {public struct Request: Sendable {public let method: String@@ -14,6 +15,9 @@ public final class HTTPServer: @unchecked Sendable {public var status: Intpublic var body: Datapublic var contentType = "application/json"+ /// A body written piece by piece (chunked transfer): `stream` gets a writer, which returns false once the client+ /// has gone (the connection closed).+ public var stream: (@Sendable (_ write: (Data) -> Bool) -> Void)?public init(status: Int, body: Data, contentType: String = "application/json") {self.status = status@@ -21,6 +25,12 @@ public final class HTTPServer: @unchecked Sendable {self.contentType = contentType}+ public static func streaming(contentType: String, _ stream: @escaping @Sendable (_ write: (Data) -> Bool) -> Void) -> Response {+ var r = Response(status: 200, body: Data(), contentType: contentType)+ r.stream = stream+ return r+ }+public static func json(_ status: Int, _ object: Any) -> Response {let data = (try? JSONSerialization.data(withJSONObject: object, options: [.sortedKeys])) ?? Data("{}".utf8)return Response(status: status, body: data)@@ -136,15 +146,35 @@ public final class HTTPServer: @unchecked Sendable {static func write(_ fd: Int32, _ response: Response) {let reason = [200: "OK", 202: "Accepted", 400: "Bad Request", 404: "Not Found", 409: "Conflict",429: "Too Many Requests", 500: "Internal Server Error"][response.status] ?? "Status"+ if let stream = response.stream {+ guard send(fd, Data("HTTP/1.1 \(response.status) \(reason)\r\nContent-Type: \(response.contentType)\r\nCache-Control: no-cache\r\nTransfer-Encoding: chunked\r\nConnection: close\r\n\r\n".utf8)) else { return }+ var open = true+ stream { piece in+ guard open, !piece.isEmpty else { return open }+ var chunk = Data((String(piece.count, radix: 16) + "\r\n").utf8)+ chunk.append(piece)+ chunk.append(contentsOf: "\r\n".utf8)+ open = send(fd, chunk)+ return open+ }+ if open { _ = send(fd, Data("0\r\n\r\n".utf8)) }+ return+ }var out = Data("HTTP/1.1 \(response.status) \(reason)\r\nContent-Type: \(response.contentType)\r\nContent-Length: \(response.body.count)\r\nConnection: close\r\n\r\n".utf8)out.append(response.body)- out.withUnsafeBytes { raw in+ _ = send(fd, out)+ }++ /// All of `data`; false when the connection is gone.+ static func send(_ fd: Int32, _ data: Data) -> Bool {+ data.withUnsafeBytes { raw invar sent = 0while sent < raw.count {- let n = send(fd, raw.baseAddress! + sent, raw.count - sent, 0)- if n <= 0 { return }+ let n = Darwin.send(fd, raw.baseAddress! + sent, raw.count - sent, 0)+ if n <= 0 { return false }sent += n}+ return true}}}diff --git a/engine-mac/Sources/KokonEngine/Kernels/decode.metal b/engine-mac/Sources/KokonEngine/Kernels/decode.metalnew file mode 100644index 0000000..34a277c--- /dev/null+++ b/engine-mac/Sources/KokonEngine/Kernels/decode.metal@@ -0,0 +1,166 @@+#include <metal_stdlib>+using namespace metal;++// Text generation (Prompt+, QwenTextGeneration.swift): one token at a time, so every product is a matrix-vector+// product that reads the whole weight once. They read the GGUF's Q4_K and Q6_K blocks where they lie (the mapped+// file), with ggml's formulas (dequant.metal), and add in fp32: the decode is bound by those bytes.++struct MatvecParams {+ uint rows; // the weight's ggml rows (output features)+ uint blocksPerRow; // 256-value blocks along the input+ uint rowBytes;+};++// y[r] = W[r, :] · x. Two rows per SIMD group, 4 SIMD groups a threadgroup. 8 lanes share a block, each taking 32 of+// its values (Q4_K: one 16-byte load of nibbles, low and high halves from two sub-blocks; Q6_K: 8 bytes of each of+// its three byte runs, 2-byte aligned as its 210-byte blocks are), so the 32 lanes read 4 neighbouring blocks of a row+// at once; the activations a lane needs are loaded once for both rows. M2 Pro, 2026-10-25 (`time-matvec`): one lane+// per 64 values and one row per SIMD group read 25–53 GB/s.+#define MATVEC_ROWS 2+#define MATVEC_Q4_ROWS 4++static inline float4 nibbles(uint v) { return float4(as_type<uchar4>(v & 0x0F0F0F0Fu)); }++kernel void matvec_q4_K(device const uchar *w [[buffer(0)]], device const float *x [[buffer(1)]],+ device float *y [[buffer(2)]], constant MatvecParams &p [[buffer(3)]],+ uint group [[threadgroup_position_in_grid]], uint lane [[thread_index_in_simdgroup]],+ uint simd [[simdgroup_index_in_threadgroup]], uint groups [[simdgroups_per_threadgroup]]) {+ uint row0 = (group * groups + simd) * MATVEC_Q4_ROWS;+ if (row0 >= p.rows) return;+ uint pair = (lane % 8) / 2, part = lane % 2; // sub-blocks 2·pair (low nibbles) and 2·pair + 1 (high)+ float sums[MATVEC_Q4_ROWS] = {0};+ for (uint b = lane / 8; b < p.blocksPerRow; b += 4) {+ device const float4 *xl = (device const float4 *)(x + b * 256 + 64 * pair + 16 * part);+ float4 a0 = xl[0], a1 = xl[1], a2 = xl[2], a3 = xl[3], c0 = xl[8], c1 = xl[9], c2 = xl[10], c3 = xl[11];+ float sa = dot(a0 + a1 + a2 + a3, float4(1)), sc = dot(c0 + c1 + c2 + c3, float4(1));+ for (uint r = 0; r < MATVEC_Q4_ROWS; r++) {+ if (row0 + r >= p.rows) break;+ device const uchar *block = w + (ulong)(row0 + r) * p.rowBytes + b * 144;+ // d, dmin and the 12 bytes of 6-bit scales and mins in one load; get_scale_min_k4 for all 8 sub-blocks at+ // once (bytes 0–3 of each word: sub-blocks 0–3, then 4–7), of which this lane takes 2·pair and 2·pair + 1.+ uint4 head = *((device const uint4 *)block);+ half2 dd = as_type<half2>(head.x);+ uint scales = pair < 2 ? head.y & 0x3F3F3F3Fu : (head.w & 0x0F0F0F0Fu) | (((head.y >> 6) & 0x03030303u) << 4);+ uint mins = pair < 2 ? head.z & 0x3F3F3F3Fu : ((head.w >> 4) & 0x0F0F0F0Fu) | (((head.z >> 6) & 0x03030303u) << 4);+ uint shift = 16 * (pair % 2);+ float d = float(dd.x), dmin = float(dd.y);+ float sc1 = float((scales >> shift) & 0xFF), sc2 = float((scales >> (shift + 8)) & 0xFF);+ float m1 = float((mins >> shift) & 0xFF), m2 = float((mins >> (shift + 8)) & 0xFF);+ uint4 q = *((device const uint4 *)(block + 16 + 32 * pair + 16 * part));+ float lo = dot(nibbles(q.x), a0) + dot(nibbles(q.y), a1) + dot(nibbles(q.z), a2) + dot(nibbles(q.w), a3);+ float hi = dot(nibbles(q.x >> 4), c0) + dot(nibbles(q.y >> 4), c1) + dot(nibbles(q.z >> 4), c2) + dot(nibbles(q.w >> 4), c3);+ sums[r] += d * (sc1 * lo + sc2 * hi) - dmin * (m1 * sa + m2 * sc);+ }+ }+ for (uint r = 0; r < MATVEC_Q4_ROWS; r++) {+ float total = simd_sum(sums[r]);+ if (lane == 0 && row0 + r < p.rows) y[row0 + r] = total;+ }+}++// 8 bytes from a 2-byte aligned address, as two words.+static inline uint2 load8(device const uchar *p) {+ device const ushort *h = (device const ushort *)p;+ return uint2(uint(h[0]) | (uint(h[1]) << 16), uint(h[2]) | (uint(h[3]) << 16));+}++static inline float4 q6(uint low, uint high) { return float4(as_type<uchar4>(low | (high << 4))) - 32.0f; }++kernel void matvec_q6_K(device const uchar *w [[buffer(0)]], device const float *x [[buffer(1)]],+ device float *y [[buffer(2)]], constant MatvecParams &p [[buffer(3)]],+ uint group [[threadgroup_position_in_grid]], uint lane [[thread_index_in_simdgroup]],+ uint simd [[simdgroup_index_in_threadgroup]], uint groups [[simdgroups_per_threadgroup]]) {+ uint row0 = (group * groups + simd) * MATVEC_ROWS;+ if (row0 >= p.rows) return;+ // Half h of the block (values 128h …), l from l0 to l0 + 7 with l0 = 16g + 8·part: scales sc[g], sc[g + 2], …+ uint j = lane % 8, h = j / 4, g = (j / 2) % 2, l0 = 16 * g + 8 * (j % 2);+ float sums[MATVEC_ROWS] = {0};+ for (uint b = lane / 8; b < p.blocksPerRow; b += 4) {+ device const float4 *xl = (device const float4 *)(x + b * 256 + 128 * h + l0);+ float4 x1a = xl[0], x1b = xl[1], x2a = xl[8], x2b = xl[9], x3a = xl[16], x3b = xl[17], x4a = xl[24], x4b = xl[25];+ for (uint r = 0; r < MATVEC_ROWS; r++) {+ if (row0 + r >= p.rows) break;+ device const uchar *block = w + (ulong)(row0 + r) * p.rowBytes + b * 210;+ float d = float(*((device const half *)(block + 208)));+ device const char *sc = (device const char *)(block + 192 + 8 * h + g);+ uint2 la = load8(block + 64 * h + l0), lb = load8(block + 64 * h + 32 + l0), qh = load8(block + 128 + 32 * h + l0);+ const uint m4 = 0x0F0F0F0Fu, m2 = 0x03030303u;+ float s1 = dot(q6(la.x & m4, qh.x & m2), x1a) + dot(q6(la.y & m4, qh.y & m2), x1b);+ float s2 = dot(q6(lb.x & m4, (qh.x >> 2) & m2), x2a) + dot(q6(lb.y & m4, (qh.y >> 2) & m2), x2b);+ float s3 = dot(q6((la.x >> 4) & m4, (qh.x >> 4) & m2), x3a) + dot(q6((la.y >> 4) & m4, (qh.y >> 4) & m2), x3b);+ float s4 = dot(q6((lb.x >> 4) & m4, (qh.x >> 6) & m2), x4a) + dot(q6((lb.y >> 4) & m4, (qh.y >> 6) & m2), x4b);+ sums[r] += d * (float(sc[0]) * s1 + float(sc[2]) * s2 + float(sc[4]) * s3 + float(sc[6]) * s4);+ }+ }+ for (uint r = 0; r < MATVEC_ROWS; r++) {+ float total = simd_sum(sums[r]);+ if (lane == 0 && row0 + r < p.rows) y[row0 + r] = total;+ }+}++struct CacheParams {+ uint rows; // rows of k and v [rows, kvHeads · 128]+ uint kvHeads;+ uint capacity; // positions per head in the cache+ uint position; // of the first row+};++// K (after its norm and RoPE) and V of `rows` positions into one layer's cache, fp16 [kvHeads, capacity, 128] each.+kernel void kv_store(device const float *k [[buffer(0)]], device const float *v [[buffer(1)]],+ device half *kc [[buffer(2)]], device half *vc [[buffer(3)]], constant CacheParams &p [[buffer(4)]],+ uint2 g [[thread_position_in_grid]]) {+ uint c = g.x, row = g.y;+ if (c >= p.kvHeads * 128 || row >= p.rows) return;+ uint head = c / 128;+ ulong at = ((ulong)head * p.capacity + p.position + row) * 128 + c % 128;+ kc[at] = half(k[(ulong)row * p.kvHeads * 128 + c]);+ vc[at] = half(v[(ulong)row * p.kvHeads * 128 + c]);+}++struct DecodeAttentionParams {+ uint keys; // positions 0 … keys − 1 in the cache+ uint capacity;+ uint groups; // query heads per key/value head+ float scale; // 1 / √128+};++// One query per head against the cached keys (causal: all of them), as Qwen3's grouped attention: query head h reads+// key/value head h / groups. A threadgroup per query head, 8 SIMD groups each taking every 8th key with its own online+// softmax, each lane 4 of the 128 dimensions; the 8 partial results are merged at the end.+kernel void decode_attention(device const float *q [[buffer(0)]], device const half *kc [[buffer(1)]],+ device const half *vc [[buffer(2)]], device float *out [[buffer(3)]],+ constant DecodeAttentionParams &p [[buffer(4)]],+ uint head [[threadgroup_position_in_grid]], uint lane [[thread_index_in_simdgroup]],+ uint simd [[simdgroup_index_in_threadgroup]]) {+ threadgroup float partM[8], partL[8];+ threadgroup float4 partAcc[8][32];+ float4 qv = ((device const float4 *)(q + head * 128))[lane] * p.scale;+ ulong base = (ulong)(head / p.groups) * p.capacity * 128;+ float m = -INFINITY, l = 0;+ float4 acc = 0;+ for (uint j = simd; j < p.keys; j += 8) {+ float4 kv = float4(((device const half4 *)(kc + base + (ulong)j * 128))[lane]);+ float s = simd_sum(dot(qv, kv));+ float mNew = max(m, s);+ float correction = exp(m - mNew), weight = exp(s - mNew);+ l = l * correction + weight;+ acc = acc * correction + weight * float4(((device const half4 *)(vc + base + (ulong)j * 128))[lane]);+ m = mNew;+ }+ if (lane == 0) { partM[simd] = m; partL[simd] = l; }+ partAcc[simd][lane] = acc;+ threadgroup_barrier(mem_flags::mem_threadgroup);+ if (simd == 0) {+ float mAll = -INFINITY;+ for (uint i = 0; i < 8; i++) mAll = max(mAll, partM[i]);+ float lAll = 0;+ float4 total = 0;+ for (uint i = 0; i < 8; i++) {+ if (partL[i] == 0) continue; // a SIMD group without keys+ float f = exp(partM[i] - mAll);+ lAll += partL[i] * f;+ total += partAcc[i][lane] * f;+ }+ ((device float4 *)(out + head * 128))[lane] = total / lAll;+ }+}diff --git a/engine-mac/Sources/KokonEngine/QwenPipeline.swift b/engine-mac/Sources/KokonEngine/QwenPipeline.swiftindex 0ae5063..642cbc8 100644--- a/engine-mac/Sources/KokonEngine/QwenPipeline.swift+++ b/engine-mac/Sources/KokonEngine/QwenPipeline.swift@@ -51,6 +51,11 @@ public final class QwenPipeline {}}private var encoderFile: GGUFFile?, encoder: QwenTextEncoder?+ /// **Prompt+ hands its encoder to the next image**: Prompt+ writes the prompt the next image encodes, on the same file,+ /// so when the encoder is not kept (`keepEncoder`), the one text generation opened stays open until that image's+ /// encode, which then drops it. Its pages are the file's, which macOS can reclaim. M2 Pro 16 GB, 2026-10-25: opening+ /// the encoder and its first pass cost ~2 s of a 12-token prefill of 2.2–2.5 s (0.46 s once warm).+ private var handedOver: QwenTextEncoder?private var ditFile: GGUFFile?, dit: QwenDiT?/// Checked before each step and after each DiT layer: true stops the image (`QwenDiT.Cancelled`).public var cancelled: () -> Bool = { false }@@ -100,8 +105,8 @@ public final class QwenPipeline {defer { encoder!.release() }return try encoder!.conditioning(prompt: prompt)}- let file = try GGUFFile(path: encoderPath)- let encoder = try QwenTextEncoder(file: file, context: context)+ let encoder = try handedOver ?? QwenTextEncoder(file: try GGUFFile(path: encoderPath), context: context)+ handedOver = nildefer { encoder.release() }return try encoder.conditioning(prompt: prompt)}()@@ -164,6 +169,30 @@ public final class QwenPipeline {return image}+ /// **Prompt+**: the text encoder's LLM writes the assistant's reply to `prompt` (`QwenTextGenerator`), on the same+ /// file and with the same keeping rules as the image's encoder.+ public func generateText(system: String, prompt: String, maxTokens: Int, sampling: TextSampler.Parameters, seed: Int64,+ onPiece: (String) -> Bool) throws -> QwenTextGenerator.Result {+ let started = DispatchTime.now().uptimeNanoseconds+ let encoder: QwenTextEncoder+ if keepModels && keepEncoder {+ if self.encoder == nil {+ let file = try GGUFFile(path: encoderPath)+ (encoderFile, self.encoder) = (file, try QwenTextEncoder(file: file, context: context))+ }+ encoder = self.encoder!+ } else {+ encoder = try handedOver ?? QwenTextEncoder(file: try GGUFFile(path: encoderPath), context: context)+ handedOver = encoder // for the image that follows+ }+ defer { encoder.release() }+ var result = try QwenTextGenerator(encoder: encoder).generate(system: system, prompt: prompt, maxTokens: maxTokens,+ sampling: sampling, seed: seed, onPiece: onPiece)+ // With opening the file and reading its tokenizer.+ result.prepare = Double(DispatchTime.now().uptimeNanoseconds - started) * 1e-9 - result.prefill - result.decode+ return result+ }+// MARK: - latents/// `[64, h, w]` → tokens `[h·w, 64]` (`_pack_latents`: a spatial flatten, then channels last).diff --git a/engine-mac/Sources/KokonEngine/QwenTextEncoder.swift b/engine-mac/Sources/KokonEngine/QwenTextEncoder.swiftindex c31b62d..fae6d4a 100644--- a/engine-mac/Sources/KokonEngine/QwenTextEncoder.swift+++ b/engine-mac/Sources/KokonEngine/QwenTextEncoder.swift@@ -75,8 +75,9 @@ public final class QwenTextEncoder {}private var nextReserve = 0- /// The last decoder layer's output for every token, `[ids.count, 4096]`.- public func forward(ids: [Int], probe: Probe? = nil) throws -> [Float] {+ /// The last decoder layer's output for every token, `[ids.count, 4096]`. `cache`: each layer's K (after its norm+ /// and RoPE) and V are kept there too, for text generation (`QwenTextGenerator`).+ public func forward(ids: [Int], probe: Probe? = nil, cache: KVCache? = nil) throws -> [Float] {let d = QwenTextEncoder.dim, t = ids.count, kvWidth = QwenTextEncoder.kvHeads * QwenTextEncoder.headDimlet started = DispatchTime.now().uptimeNanosecondstimings = [:]@@ -120,7 +121,7 @@ public final class QwenTextEncoder {var commands = try begin()for layer in 0..<QwenTextEncoder.layers {- try encodeLayer(layer, rows: t, b, commands: commands)+ try encodeLayer(layer, rows: t, b, commands: commands, cache: cache)if let probe, [0, 1, 17, QwenTextEncoder.layers - 1].contains(layer) {try MetalContext.finish(commands, "text encoder")probe("layer\(layer)_out", arena.pointer(b.x), t * d)@@ -149,7 +150,7 @@ public final class QwenTextEncoder {return out}- func encodeLayer(_ layer: Int, rows t: Int, _ b: Buffers, commands: MTLCommandBuffer) throws {+ func encodeLayer(_ layer: Int, rows t: Int, _ b: Buffers, commands: MTLCommandBuffer, cache: KVCache? = nil) throws {let d = QwenTextEncoder.dim, block = "blk.\(layer)."let norms = layer * 4 * dtry rmsNorm(b, b.x, into: b.normed, rows: t, weight: norms, commands: commands)@@ -159,6 +160,8 @@ public final class QwenTextEncoder {try headNormRope(b, b.q, heads: QwenTextEncoder.heads, weight: norms + 2 * d, rows: t, commands: commands)try headNormRope(b, b.k, heads: QwenTextEncoder.kvHeads, weight: norms + 2 * d + QwenTextEncoder.headDim,rows: t, commands: commands)+ try cache?.store(layer: layer, k: (b.arena.buffer, b.k.offset), v: (b.arena.buffer, b.v.offset), rows: t, position: 0,+ context: context, commands: commands)try repeatKV(b, b.k, into: b.kx, rows: t, commands: commands)try repeatKV(b, b.v, into: b.vx, rows: t, commands: commands)try attend(b, rows: t, commands: commands)diff --git a/engine-mac/Sources/KokonEngine/QwenTextGeneration.swift b/engine-mac/Sources/KokonEngine/QwenTextGeneration.swiftnew file mode 100644index 0000000..d5326c7--- /dev/null+++ b/engine-mac/Sources/KokonEngine/QwenTextGeneration.swift@@ -0,0 +1,489 @@+import Foundation+import Metal++/// **Text generation with the text encoder's LLM** (Prompt+): Qwen3-VL-8B's text decoder writes the prompt that the+/// image model gets, as Kokon's sd.cpp does (`src/pipeline/text.cpp`, `POST /sdcpp/v1/text_gen`), on the same file:+///+/// chat template ─ tokenizer ─ prefill: QwenTextEncoder.forward on every token, each layer's K and V into the cache+/// then per token: embedding row ─ 36 × { RMS ─ q, k, v ─ norms, RoPE at its position ─ into the cache ─ attention on+/// the cache ─ o ─ RMS ─ gate, up ─ SwiGLU ─ down } ─ final RMSNorm ─ LM head ─ sampler ─ next token+///+/// The decode reads every weight once per token, so its products read the Q4_K / Q6_K blocks where they lie in the+/// mapped file (`Kernels/decode.metal`), with no unpacking. Text only: images (edit Prompt+, Compose labels) need the+/// vision tower.+public final class QwenTextGenerator {+ /// sd.cpp's limit on prompt + generated tokens (`TEXT_GEN_MAX_CONTEXT`).+ public static let maxContext = 16384+ public static let vocabulary = 151_936++ public enum Finish: String { case stop, length, cancelled }+ public struct Result {+ public var text = ""+ public var finish = Finish.stop+ public var seed: Int64 = 0+ public var promptTokens = 0, generatedTokens = 0+ /// Seconds: tokenizing and buffers, prefill (to the first token), decode (first token to last).+ public var prepare = 0.0, prefill = 0.0, decode = 0.0+ /// Every token sampled, the stop token included.+ public var tokens: [Int] = []+ }++ let encoder: QwenTextEncoder+ let context: MetalContext+ var file: GGUFFile { encoder.file }++ public init(encoder: QwenTextEncoder) {+ self.encoder = encoder+ context = encoder.context+ }++ /// sd.cpp's chat template for text generation without images: the system turn when `system` is not empty.+ public static func template(system: String, prompt: String) -> String {+ (system.isEmpty ? "" : "<|im_start|>system\n\(system)<|im_end|>\n") + "<|im_start|>user\n\(prompt)<|im_end|>\n<|im_start|>assistant\n"+ }++ /// Writes the assistant's reply to `prompt`. `onPiece` gets the text as it is written (whole UTF-8 characters);+ /// returning false stops it (`cancelled`). `logits`: each step's logits, for the checks.+ public func generate(system: String, prompt: String, maxTokens: Int, sampling: TextSampler.Parameters, seed: Int64,+ onPiece: (String) -> Bool = { _ in true },+ logits: ((Int, UnsafePointer<Float>) -> Void)? = nil) throws -> Result {+ var result = Result()+ let started = DispatchTime.now().uptimeNanoseconds+ func since(_ t: UInt64) -> Double { Double(DispatchTime.now().uptimeNanoseconds - t) * 1e-9 }+ let tokenizer = encoder.tokenizer+ guard let imEnd = tokenizer.id(of: "<|im_end|>"), let endOfText = tokenizer.id(of: "<|endoftext|>") else {+ throw GGUFFile.Failure.unsupported("the tokenizer has no <|im_end|> or <|endoftext|>")+ }+ let stops: Set<Int> = [imEnd, endOfText]+ let ids = tokenizer.encode(QwenTextGenerator.template(system: system, prompt: prompt))+ let capacity = ids.count + maxTokens+ guard capacity <= QwenTextGenerator.maxContext else {+ throw GGUFFile.Failure.unsupported("prompt (\(ids.count) tokens) + max_tokens (\(maxTokens)) exceeds the \(QwenTextGenerator.maxContext)-token limit")+ }+ result.promptTokens = ids.count+ result.seed = seed >= 0 ? seed : Int64.random(in: 0...0x7fff_ffff)+ var sampler = TextSampler(sampling, seed: UInt64(bitPattern: result.seed))+ let cache = try KVCache(context: context, layers: QwenTextEncoder.layers, kvHeads: QwenTextEncoder.kvHeads, capacity: capacity)+ let step = try Decoder(generator: self, cache: cache)+ result.prepare = since(started)++ // Prefill: the prompt's K and V into the cache, the last row's hidden state to the head.+ var t = DispatchTime.now().uptimeNanoseconds+ let hidden = try encoder.forward(ids: ids, cache: cache)+ var next = try step.head(hidden: Array(hidden[((ids.count - 1) * QwenTextEncoder.dim)...]))+ var stream = TokenStreamDecoder(tokenizer: tokenizer)+ while true {+ logits?(result.tokens.count, next)+ let token = sampler.sample(next, count: QwenTextGenerator.vocabulary)+ result.tokens.append(token)+ if result.tokens.count == 1 { result.prefill = since(t); t = DispatchTime.now().uptimeNanoseconds }+ if stops.contains(token) { result.finish = .stop; break }+ let piece = stream.push(token)+ if !piece.isEmpty {+ result.text += piece+ if !onPiece(piece) { result.finish = .cancelled; break }+ }+ if result.tokens.count == maxTokens { result.finish = .length; break }+ next = try step.token(token, position: ids.count + result.tokens.count - 1)+ }+ if result.finish != .cancelled {+ let rest = stream.flush()+ if !rest.isEmpty {+ result.text += rest+ if !onPiece(rest) { result.finish = .cancelled }+ }+ }+ result.decode = since(t)+ result.generatedTokens = result.tokens.count - (result.finish == .stop ? 1 : 0)+ context.release(file)+ return result+ }++ /// One token through the decoder layers and the head, on the GPU; the buffers live as long as one reply.+ final class Decoder {+ let generator: QwenTextGenerator+ let cache: KVCache+ let context: MetalContext+ let file: GGUFFile+ /// [x | normed | q | k | v | attended | o | gate | up] fp32, the layers' norm weights, the RoPE table, the logits.+ let state: MTLBuffer, norms: MTLBuffer, rope: MTLBuffer, logits: MTLBuffer+ let offsets: [String: Int]+ let embedding: GGUFTensor+ /// The GPU's seconds for the last token.+ var lastGPU = 0.0++ init(generator: QwenTextGenerator, cache: KVCache) throws {+ self.generator = generator+ self.cache = cache+ context = generator.context+ file = generator.file+ let d = QwenTextEncoder.dim, kv = QwenTextEncoder.kvHeads * QwenTextEncoder.headDim, h = QwenTextEncoder.hidden+ var offsets: [String: Int] = [:], at = 0+ for (name, floats) in [("x", d), ("normed", d), ("q", d), ("k", kv), ("v", kv), ("attended", d), ("o", d),+ ("gate", h), ("up", h)] {+ offsets[name] = at+ at += floats * 4+ }+ self.offsets = offsets+ state = try context.scratch(at, label: "decode state")+ // Per layer: input norm, post-attention norm, q norm (128), k norm (128); then the final norm.+ norms = try context.scratch((QwenTextEncoder.layers * (2 * d + 256) + d) * 4, label: "decode norms")+ let n = norms.contents().assumingMemoryBound(to: Float.self)+ for layer in 0..<QwenTextEncoder.layers {+ for (i, (name, count)) in [("attn_norm", d), ("ffn_norm", d), ("attn_q_norm", 128), ("attn_k_norm", 128)].enumerated() {+ let w = try Dequant.array(file, "blk.\(layer).\(name).weight")+ (n + layer * (2 * d + 256) + [0, d, 2 * d, 2 * d + 128][i]).update(from: w, count: count)+ }+ }+ (n + QwenTextEncoder.layers * (2 * d + 256)).update(from: try Dequant.array(file, "output_norm.weight"), count: d)+ let table = QwenTextEncoder.ropeTable(rows: cache.capacity)+ rope = try context.scratch(table.count * 4, label: "decode rope")+ rope.contents().copyMemory(from: table, byteCount: table.count * 4)+ logits = try context.scratch(QwenTextGenerator.vocabulary * 4, label: "logits")+ embedding = try file.tensor("token_embd.weight")+ }++ func pointer(_ name: String) -> UnsafeMutablePointer<Float> {+ (state.contents() + offsets[name]!).assumingMemoryBound(to: Float.self)+ }++ /// The final norm and the LM head on `hidden` (one row): the logits.+ func head(hidden: [Float]) throws -> UnsafePointer<Float> {+ pointer("x").update(from: hidden, count: QwenTextEncoder.dim)+ let commands = try generator.encoder.begin()+ try encodeHead(commands)+ try MetalContext.finish(commands, "text generation head")+ return UnsafePointer(logits.contents().assumingMemoryBound(to: Float.self))+ }++ /// `token` at `position`: its K and V join the cache; the logits of the next one.+ func token(_ token: Int, position: Int) throws -> UnsafePointer<Float> {+ let d = QwenTextEncoder.dim+ guard let type = embedding.type, token >= 0, token < embedding.rows else { throw GGUFFile.Failure.unsupported("token id \(token)") }+ Dequant.row(file.data(embedding) + token * type.bytes(count: d), type: type, count: d, into: pointer("x"))+ let commands = try generator.encoder.begin()+ for layer in 0..<QwenTextEncoder.layers { try encodeLayer(layer, position: position, commands) }+ try encodeHead(commands)+ lastGPU = try MetalContext.finish(commands, "text generation step")+ return UnsafePointer(logits.contents().assumingMemoryBound(to: Float.self))+ }++ func encodeLayer(_ layer: Int, position: Int, _ commands: MTLCommandBuffer) throws {+ let d = QwenTextEncoder.dim, block = "blk.\(layer).", norm = layer * (2 * d + 256) * 4+ try rmsNorm("x", into: "normed", weight: norm, commands)+ try matvec(block + "attn_q.weight", "normed", "q", commands)+ try matvec(block + "attn_k.weight", "normed", "k", commands)+ try matvec(block + "attn_v.weight", "normed", "v", commands)+ for (name, heads, weight) in [("q", QwenTextEncoder.heads, norm + 2 * d * 4), ("k", QwenTextEncoder.kvHeads, norm + (2 * d + 128) * 4)] {+ try kernel("head_rmsnorm_rope_neox", commands) { e in+ e.setBuffer(state, offset: offsets[name]!, index: 0)+ e.setBuffer(norms, offset: weight, index: 1)+ e.setBuffer(rope, offset: position * 128 * 4, index: 2)+ var params = QwenDiT.HeadParams(heads: UInt32(heads), rows: 1, eps: QwenTextEncoder.eps)+ e.setBytes(¶ms, length: MemoryLayout<QwenDiT.HeadParams>.stride, index: 3)+ e.dispatchThreadgroups(MTLSize(width: (heads + 7) / 8, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))+ }+ }+ try cache.store(layer: layer, k: (state, offsets["k"]!), v: (state, offsets["v"]!), rows: 1, position: position,+ context: context, commands: commands)+ try kernel("decode_attention", commands) { e in+ e.setBuffer(state, offset: offsets["q"]!, index: 0)+ e.setBuffer(cache.buffer, offset: cache.offset(layer: layer, value: false), index: 1)+ e.setBuffer(cache.buffer, offset: cache.offset(layer: layer, value: true), index: 2)+ e.setBuffer(state, offset: offsets["attended"]!, index: 3)+ var params = DecodeAttentionParams(keys: UInt32(position + 1), capacity: UInt32(cache.capacity),+ groups: UInt32(QwenTextEncoder.heads / QwenTextEncoder.kvHeads),+ scale: 1 / Float(QwenTextEncoder.headDim).squareRoot())+ e.setBytes(¶ms, length: MemoryLayout<DecodeAttentionParams>.stride, index: 4)+ e.dispatchThreadgroups(MTLSize(width: QwenTextEncoder.heads, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))+ }+ try matvec(block + "attn_output.weight", "attended", "o", commands)+ try add("o", count: d, commands)+ try rmsNorm("x", into: "normed", weight: norm + d * 4, commands)+ try matvec(block + "ffn_gate.weight", "normed", "gate", commands)+ try matvec(block + "ffn_up.weight", "normed", "up", commands)+ try kernel("swiglu", commands) { e in+ e.setBuffer(state, offset: offsets["gate"]!, index: 0)+ e.setBuffer(state, offset: offsets["up"]!, index: 1)+ var params = QwenDiT.SwigluParams(columns: UInt32(QwenTextEncoder.hidden), half: QwenDiT.fullOnly(1))+ e.setBytes(¶ms, length: MemoryLayout<QwenDiT.SwigluParams>.stride, index: 2)+ e.setBuffer(state, offset: offsets["gate"]!, index: 3)+ e.dispatchThreads(MTLSize(width: QwenTextEncoder.hidden, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))+ }+ try matvec(block + "ffn_down.weight", "gate", "o", commands)+ try add("o", count: d, commands)+ }++ func encodeHead(_ commands: MTLCommandBuffer) throws {+ try rmsNorm("x", into: "normed", weight: QwenTextEncoder.layers * (2 * QwenTextEncoder.dim + 256) * 4, commands)+ let tensor = try file.tensor("output.weight")+ try matvec(tensor, x: (state, offsets["normed"]!), y: (logits, 0), commands)+ }++ func kernel(_ name: String, _ commands: MTLCommandBuffer, _ body: (MTLComputeCommandEncoder) throws -> Void) throws {+ try generator.encoder.compute(name, commands: commands, body)+ }++ func rmsNorm(_ input: String, into out: String, weight: Int, _ commands: MTLCommandBuffer) throws {+ try kernel("rmsnorm_rows", commands) { e in+ e.setBuffer(state, offset: offsets[input]!, index: 0)+ e.setBuffer(state, offset: offsets[out]!, index: 1)+ e.setBuffer(norms, offset: weight, index: 2)+ var params = QwenDiT.RowParams(columns: UInt32(QwenTextEncoder.dim), targetStart: 0, eps: QwenTextEncoder.eps)+ e.setBytes(¶ms, length: MemoryLayout<QwenDiT.RowParams>.stride, index: 3)+ e.dispatchThreadgroups(MTLSize(width: 1, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))+ }+ }++ func add(_ y: String, count: Int, _ commands: MTLCommandBuffer) throws {+ try kernel("add_inplace", commands) { e in+ e.setBuffer(state, offset: offsets["x"]!, index: 0)+ e.setBuffer(state, offset: offsets[y]!, index: 1)+ var c = UInt32(count)+ e.setBytes(&c, length: 4, index: 2)+ e.dispatchThreads(MTLSize(width: count, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))+ }+ }++ func matvec(_ name: String, _ x: String, _ y: String, _ commands: MTLCommandBuffer) throws {+ try matvec(try file.tensor(name), x: (state, offsets[x]!), y: (state, offsets[y]!), commands)+ }++ /// `y = W · x` on the weight's blocks in the mapped file.+ func matvec(_ tensor: GGUFTensor, x: (MTLBuffer, Int), y: (MTLBuffer, Int), _ commands: MTLCommandBuffer) throws {+ guard let type = tensor.type, type == .q4_K || type == .q6_K else {+ throw GGUFFile.Failure.unsupported("\(tensor.name): \(tensor.type?.name ?? "?"), the decode reads Q4_K and Q6_K")+ }+ let (source, offset) = try context.buffer(for: tensor, in: file)+ try kernel(type == .q4_K ? "matvec_q4_K" : "matvec_q6_K", commands) { e in+ e.setBuffer(source, offset: offset, index: 0)+ e.setBuffer(x.0, offset: x.1, index: 1)+ e.setBuffer(y.0, offset: y.1, index: 2)+ var params = MatvecParams(rows: UInt32(tensor.rows), blocksPerRow: UInt32(tensor.columns / 256),+ rowBytes: UInt32(type.bytes(count: tensor.columns)))+ e.setBytes(¶ms, length: MemoryLayout<MatvecParams>.stride, index: 3)+ let rows = QwenTextGenerator.matvecRows(type)+ e.dispatchThreadgroups(MTLSize(width: (tensor.rows + rows - 1) / rows, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 128, height: 1, depth: 1))+ }+ }+ }++ struct MatvecParams { var rows, blocksPerRow, rowBytes: UInt32 }+ /// Rows per threadgroup of `matvec_*` (4 SIMD groups of MATVEC_Q4_ROWS or MATVEC_ROWS).+ static func matvecRows(_ type: QuantType) -> Int { type == .q4_K ? 16 : 8 }+ struct DecodeAttentionParams { var keys, capacity, groups: UInt32; var scale: Float }+}++/// **The K/V cache** of text generation: per layer, K and V of every position so far, fp16 `[kvHeads, capacity, 128]`+/// each (144 KB a position for Qwen3-VL-8B, as sd.cpp's).+public final class KVCache {+ public let layers: Int, kvHeads: Int, capacity: Int+ let buffer: MTLBuffer++ init(context: MetalContext, layers: Int, kvHeads: Int, capacity: Int) throws {+ (self.layers, self.kvHeads, self.capacity) = (layers, kvHeads, capacity)+ buffer = try context.scratch(layers * 2 * kvHeads * capacity * 128 * 2, label: "K/V cache")+ }++ func offset(layer: Int, value: Bool) -> Int { ((layer * 2 + (value ? 1 : 0)) * kvHeads * capacity * 128) * 2 }++ struct Params { var rows, kvHeads, capacity, position: UInt32 }++ /// Encodes K and V of `rows` positions from `position` (rows `[rows, kvHeads · 128]` fp32) into layer `layer`.+ func store(layer: Int, k: (MTLBuffer, Int), v: (MTLBuffer, Int), rows: Int, position: Int, context: MetalContext,+ commands: MTLCommandBuffer) throws {+ guard position + rows <= capacity else { throw MetalContext.Failure.buffer("K/V cache: position \(position + rows) of \(capacity)") }+ let e = commands.makeComputeCommandEncoder()!+ e.label = "kv_store"+ e.setComputePipelineState(try context.pipeline("kv_store"))+ e.setBuffer(k.0, offset: k.1, index: 0)+ e.setBuffer(v.0, offset: v.1, index: 1)+ e.setBuffer(buffer, offset: offset(layer: layer, value: false), index: 2)+ e.setBuffer(buffer, offset: offset(layer: layer, value: true), index: 3)+ var params = Params(rows: UInt32(rows), kvHeads: UInt32(kvHeads), capacity: UInt32(capacity), position: UInt32(position))+ e.setBytes(¶ms, length: MemoryLayout<Params>.stride, index: 4)+ e.dispatchThreads(MTLSize(width: kvHeads * 128, height: rows, depth: 1), threadsPerThreadgroup: MTLSize(width: 128, height: 1, depth: 1))+ e.endEncoding()+ }+}++/// **The next-token sampler**, sd.cpp's (`src/runtime/text_sampler.cpp`) step for step so a seed writes the same text:+/// presence penalty (generated tokens only), then top-k, temperature, top-p, one draw from mt19937_64; temperature 0 is+/// greedy. Ties keep the lower token id first.+public struct TextSampler {+ public struct Parameters: Sendable {+ public var temperature: Float, topP: Float, topK: Int, presencePenalty: Float+ public init(temperature: Float, topP: Float, topK: Int, presencePenalty: Float) {+ (self.temperature, self.topP, self.topK, self.presencePenalty) = (temperature, topP, topK, presencePenalty)+ }+ public static let greedy = Parameters(temperature: 0, topP: 1, topK: 0, presencePenalty: 0)+ }++ let parameters: Parameters+ var random: MersenneTwister64+ /// Tokens generated so far, by id (dense, as sd.cpp's: the penalty is looked up for every token of the vocabulary).+ var seen: [Bool] = []++ public init(_ parameters: Parameters, seed: UInt64) {+ self.parameters = parameters+ random = MersenneTwister64(seed: seed)+ }++ var greedy: Bool { parameters.temperature <= 0 }++ /// A double in [0, 1) from 53 random bits.+ mutating func uniform() -> Double { Double(random.next() >> 11) * (1.0 / 9_007_199_254_740_992.0) }++ func penalized(_ logits: UnsafePointer<Float>, _ token: Int) -> Float {+ parameters.presencePenalty != 0 && token < seen.count && seen[token] ? logits[token] - parameters.presencePenalty : logits[token]+ }++ /// Probabilities after the penalty, top-k, temperature and top-p, by decreasing probability.+ func candidates(_ logits: UnsafePointer<Float>, count: Int) -> [(token: Int, p: Double)] {+ func before(_ a: (Int, Float), _ b: (Int, Float)) -> Bool { a.1 > b.1 || (a.1 == b.1 && a.0 < b.0) }+ var kept: [(Int, Float)]+ if parameters.topK > 0 && parameters.topK < count {+ // The k best in one pass, kept sorted (k is small, the vocabulary ~150k).+ let k = parameters.topK+ kept = []+ kept.reserveCapacity(k + 1)+ for i in 0..<count {+ let c = (i, penalized(logits, i))+ if kept.count == k, !before(c, kept[k - 1]) { continue }+ var at = kept.count+ while at > 0 && before(c, kept[at - 1]) { at -= 1 }+ kept.insert(c, at: at)+ if kept.count > k { kept.removeLast() }+ }+ } else {+ kept = (0..<count).map { ($0, penalized(logits, $0)) }+ kept.sort(by: before)+ }+ let inverse = parameters.temperature > 0 ? 1 / Double(parameters.temperature) : 1+ let top = Double(kept[0].1)+ var probs = kept.map { ($0.0, exp((Double($0.1) - top) * inverse)) }+ let sum = probs.reduce(0) { $0 + $1.1 }+ probs = probs.map { ($0.0, $0.1 / sum) }+ if parameters.topP < 1 {+ var cumulative = 0.0, keep = probs.count+ for (i, p) in probs.enumerated() {+ cumulative += p.1+ if cumulative >= Double(parameters.topP) { keep = i + 1; break }+ }+ probs.removeLast(probs.count - keep)+ let total = probs.reduce(0) { $0 + $1.1 }+ probs = probs.map { ($0.0, $0.1 / total) }+ }+ return probs.map { (token: $0.0, p: $0.1) }+ }++ /// Picks the next token and remembers it for the presence penalty.+ public mutating func sample(_ logits: UnsafePointer<Float>, count: Int) -> Int {+ var token = 0+ if greedy {+ var best = penalized(logits, 0)+ for i in 1..<count {+ let l = penalized(logits, i)+ if l > best { best = l; token = i }+ }+ } else {+ let probs = candidates(logits, count: count)+ let u = uniform()+ var cumulative = 0.0+ token = probs.last!.token+ for p in probs {+ cumulative += p.p+ if u < cumulative { token = p.token; break }+ }+ }+ if token >= seen.count { seen += [Bool](repeating: false, count: token + 1 - seen.count) }+ seen[token] = true+ return token+ }+}++/// `std::mt19937_64`, whose output the C++ standard fixes: the same seed draws the same numbers as sd.cpp.+public struct MersenneTwister64 {+ private var state = [UInt64](repeating: 0, count: 312)+ private var index = 312++ public init(seed: UInt64) {+ state[0] = seed+ for i in 1..<312 { state[i] = 6_364_136_223_846_793_005 &* (state[i - 1] ^ (state[i - 1] >> 62)) &+ UInt64(i) }+ }++ public mutating func next() -> UInt64 {+ if index >= 312 {+ for i in 0..<312 {+ let y = (state[i] & 0xFFFF_FFFF_8000_0000) | (state[(i + 1) % 312] & 0x7FFF_FFFF)+ var v = state[(i + 156) % 312] ^ (y >> 1)+ if y & 1 != 0 { v ^= 0xB502_6F5A_A966_19E9 }+ state[i] = v+ }+ index = 0+ }+ var x = state[index]+ index += 1+ x ^= (x >> 29) & 0x5555_5555_5555_5555+ x ^= (x << 17) & 0x71D6_7FFF_EDA6_0000+ x ^= (x << 37) & 0xFFF7_EEE0_0000_0000+ x ^= x >> 43+ return x+ }+}++/// Tokens to text as they come, sd.cpp's `TokenStreamDecoder`: special tokens skipped, bytes held back until they+/// complete a UTF-8 character, invalid bytes as U+FFFD.+public struct TokenStreamDecoder {+ let tokenizer: QwenTokenizer+ var pending: [UInt8] = []++ public init(tokenizer: QwenTokenizer) { self.tokenizer = tokenizer }++ public mutating func push(_ token: Int) -> String {+ guard let bytes = tokenizer.bytes(of: token) else { return "" } // an added (special) token+ pending += bytes+ return take(end: false)+ }++ public mutating func flush() -> String { take(end: true) }++ mutating func take(end: Bool) -> String {+ var out: [UInt8] = []+ var at = 0+ while at < pending.count {+ let length = TokenStreamDecoder.sequenceLength(pending, at)+ if length > 0 {+ out += pending[at..<(at + length)]+ at += length+ } else if length == 0 && !end {+ break+ } else {+ out += [0xEF, 0xBF, 0xBD]+ at = length == 0 ? pending.count : at - length+ }+ }+ pending.removeFirst(at)+ return String(decoding: out, as: UTF8.self)+ }++ /// The length of the valid UTF-8 sequence at `at`; 0 if the bytes end inside a valid prefix; −n if its first n bytes+ /// are an invalid prefix (one U+FFFD, as Python does). sd.cpp's `utf8_sequence_length`.+ static func sequenceLength(_ s: [UInt8], _ at: Int) -> Int {+ let lead = s[at]+ if lead < 0x80 { return 1 }+ var (length, lo, hi): (Int, UInt8, UInt8) = (0, 0x80, 0xBF)+ switch lead {+ case 0xC2...0xDF: length = 2+ case 0xE0...0xEF: (length, lo, hi) = (3, lead == 0xE0 ? 0xA0 : 0x80, lead == 0xED ? 0x9F : 0xBF)+ case 0xF0...0xF4: (length, lo, hi) = (4, lead == 0xF0 ? 0x90 : 0x80, lead == 0xF4 ? 0x8F : 0xBF)+ default: return -1+ }+ for i in 1..<length {+ if at + i >= s.count { return 0 }+ if s[at + i] < lo || s[at + i] > hi { return -i }+ (lo, hi) = (0x80, 0xBF)+ }+ return length+ }+}diff --git a/engine-mac/Sources/KokonEngine/QwenTokenizer.swift b/engine-mac/Sources/KokonEngine/QwenTokenizer.swiftindex 85dfe93..6a84c32 100644--- a/engine-mac/Sources/KokonEngine/QwenTokenizer.swift+++ b/engine-mac/Sources/KokonEngine/QwenTokenizer.swift@@ -121,15 +121,30 @@ public final class QwenTokenizer: @unchecked Sendable {return result}+ static let byteValues: [Character: UInt8] = {+ var reverse: [Character: UInt8] = [:]+ for (b, c) in byteCharacters.enumerated() { reverse[c] = UInt8(b) }+ return reverse+ }()+ private lazy var addedIDs = Set(added.map(\.id))+/// Token ids back to text (byte-level characters to bytes); for messages and tests.public func decode(_ ids: [Int]) -> String {- var reverse: [Character: UInt8] = [:]- for (b, c) in QwenTokenizer.byteCharacters.enumerated() { reverse[c] = UInt8(b) }var bytes: [UInt8] = []for id in ids where id >= 0 && id < tokens.count {- if added.contains(where: { $0.id == id }) { bytes += Array(tokens[id].utf8); continue }- for c in tokens[id] { if let b = reverse[c] { bytes.append(b) } }+ bytes += self.bytes(of: id) ?? Array(tokens[id].utf8)}return String(decoding: bytes, as: UTF8.self)}++ /// A token's bytes, nil for an added token (`<|im_end|>`, `<think>`, …: the ones text generation leaves out, as+ /// sd.cpp's Qwen2 special tokens).+ public func bytes(of id: Int) -> [UInt8]? {+ guard id >= 0, id < tokens.count else { return [] }+ lock.lock()+ let isAdded = addedIDs.contains(id)+ lock.unlock()+ if isAdded { return nil }+ return tokens[id].compactMap { QwenTokenizer.byteValues[$0] }+ }}diff --git a/engine-mac/Sources/KokonEngine/TextChecks.swift b/engine-mac/Sources/KokonEngine/TextChecks.swiftindex d1a6a4e..95c7ce3 100644--- a/engine-mac/Sources/KokonEngine/TextChecks.swift+++ b/engine-mac/Sources/KokonEngine/TextChecks.swift@@ -134,3 +134,93 @@ extension Checks {if differ > 0 || offset != want.count { throw GGUFFile.Failure.unsupported("noise differs from sd.cpp's") }}}++extension Checks {+ /// Text generation (Prompt+) against `tools/reference_text_generation.py`: the engine writes greedily (its tokens to+ /// `<dir>/engine.json`, its logits beside); once the reference has run on those tokens, each step's logits are+ /// compared with transformers' (relative error, the greedy choice, the margin between the two best) and the+ /// engine's speed is reported. `system` / `prompt` from `<dir>/engine.json` when it exists.+ public static func textGeneration(path: String, directory: String, tokens: Int = 48) throws {+ let dir = URL(fileURLWithPath: directory)+ try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true)+ let previous = (try? JSONSerialization.jsonObject(with: Data(contentsOf: dir.appendingPathComponent("engine.json")))) as? [String: Any]+ let system = previous?["system"] as? String+ ?? "You write prompts for an image generation model. Expand the user's request into one detailed English paragraph that describes the finished image: subject, setting, composition, lighting and style. Output only the prompt."+ let prompt = previous?["prompt"] as? String ?? "un gatto rosso che dorme su un davanzale, sera"+ let file = try GGUFFile(path: path)+ let context = try MetalContext()+ let encoder = try QwenTextEncoder(file: file, context: context)+ let generator = QwenTextGenerator(encoder: encoder)+ let vocabulary = QwenTextGenerator.vocabulary+ var logits = [Float](repeating: 0, count: tokens * vocabulary)+ let result = try generator.generate(system: system, prompt: prompt, maxTokens: tokens, sampling: .greedy, seed: 0, logits: { step, l in+ logits.withUnsafeMutableBufferPointer { ($0.baseAddress! + step * vocabulary).update(from: l, count: vocabulary) }+ })+ let steps = result.tokens.count+ let promptIDs = encoder.tokenizer.encode(QwenTextGenerator.template(system: system, prompt: prompt))+ print(String(format: "%d prompt tokens, %d generated (%@): prefill %.2f s, decode %.2f s, %.1f tokens/s",+ result.promptTokens, result.generatedTokens, result.finish.rawValue, result.prefill, result.decode,+ Double(steps - 1) / max(result.decode, 1e-9)))+ print("text: \(result.text)")+ if let previous, let generated = previous["generated"] as? [Int], generated != result.tokens {+ print("NOTE: not the tokens of the engine's previous run (\(generated.count) then, \(steps) now): the reference must be run again")+ }+ let meta: [String: Any] = ["system": system, "prompt": prompt, "prompt_ids": promptIDs, "generated": result.tokens, "text": result.text]+ try JSONSerialization.data(withJSONObject: meta, options: [.prettyPrinted, .sortedKeys]).write(to: dir.appendingPathComponent("engine.json"))+ try logits.prefix(steps * vocabulary).withUnsafeBufferPointer { Data(buffer: $0) }.write(to: dir.appendingPathComponent("engine_logits.f32"))++ guard let want = try? reference(directory, "reference_logits"), want.count == steps * vocabulary,+ (previous?["generated"] as? [Int]) == result.tokens else {+ print("no reference for these tokens yet: run tools/reference_text_generation.py on \(directory), then this again")+ return+ }+ var worst = 0.0, agree = 0, ties = 0+ for s in 0..<steps {+ let r = logits.withUnsafeBufferPointer { compare($0.baseAddress! + s * vocabulary, Array(want[(s * vocabulary)..<((s + 1) * vocabulary)])) }+ worst = max(worst, r.relative)+ let row = want[(s * vocabulary)..<((s + 1) * vocabulary)]+ let sorted = row.enumerated().sorted { $0.element > $1.element }+ let (best, second) = (sorted[0], sorted[1])+ if best.offset == result.tokens[s] { agree += 1 } else if best.element - second.element < 0.05 { ties += 1 }+ if s < 4 || s == steps - 1 || best.offset != result.tokens[s] {+ print(String(format: " step %2d rel %.2e max |Δ| %.3g engine %6d reference %6d (margin %.3f)", s, r.relative, r.maxAbs,+ result.tokens[s], best.offset, best.element - second.element))+ }+ }+ print(String(format: "logits within %.2e relative at every step; the greedy choice agrees at %d of %d (%d near-ties)", worst, agree, steps, ties))+ }+}++extension Checks {+ /// The decode's matrix-vector products on the encoder's real weights, alone: milliseconds and GB/s per shape (the+ /// decode reads every weight once a token, so these rates bound its speed), then one whole token's GPU time.+ public static func timeMatvec(path: String) throws {+ let file = try GGUFFile(path: path)+ let context = try MetalContext()+ let encoder = try QwenTextEncoder(file: file, context: context)+ let generator = QwenTextGenerator(encoder: encoder)+ let cache = try KVCache(context: context, layers: QwenTextEncoder.layers, kvHeads: QwenTextEncoder.kvHeads, capacity: 1024)+ let decoder = try QwenTextGenerator.Decoder(generator: generator, cache: cache)+ let x = try context.scratch(QwenTextEncoder.hidden * 4, label: "x"), y = try context.scratch(QwenTextGenerator.vocabulary * 4, label: "y")+ for i in 0..<QwenTextEncoder.hidden { x.contents().assumingMemoryBound(to: Float.self)[i] = Float(i % 7) * 0.1 - 0.3 }+ for name in ["blk.0.attn_q.weight", "blk.0.attn_k.weight", "blk.0.attn_v.weight", "blk.0.ffn_gate.weight", "blk.0.ffn_down.weight", "output.weight"] {+ let tensor = try file.tensor(name)+ let runs = 20+ // Warm (the file's pages), then timed: one command buffer of `runs` products.+ for timed in [false, true] {+ let commands = try encoder.begin()+ for _ in 0..<(timed ? runs : 2) { try decoder.matvec(tensor, x: (x, 0), y: (y, 0), commands) }+ let seconds = try MetalContext.finish(commands, name) / Double(timed ? runs : 2)+ if timed {+ print(String(format: " %-24@ %@ %6d × %5d %.3f ms %.0f GB/s", name as NSString, (tensor.type?.name ?? "?") as NSString,+ tensor.rows, tensor.columns, seconds * 1e3, Double(tensor.bytes) / seconds / 1e9))+ }+ }+ }+ for run in 0..<5 {+ let t = DispatchTime.now().uptimeNanoseconds+ _ = try decoder.token(1000 + run, position: run)+ print(String(format: "token %d: %.1f ms (GPU %.1f ms)", run, Double(DispatchTime.now().uptimeNanoseconds - t) * 1e-6, decoder.lastGPU * 1e3))+ }+ }+}diff --git a/engine-mac/Sources/kokon-engine-mac/main.swift b/engine-mac/Sources/kokon-engine-mac/main.swiftindex 1a6a7f6..0bfed05 100644--- a/engine-mac/Sources/kokon-engine-mac/main.swift+++ b/engine-mac/Sources/kokon-engine-mac/main.swift@@ -15,6 +15,8 @@ kokon-engine-mac: Kokon's Apple Silicon engine (in development)check-generate <encoder.gguf> <dit.gguf> <vae.safetensors> <reference dir> [text reference dir]text to image against tools/reference_generate.py, from its noise(and from the text reference's conditioning when given)+ check-textgen <encoder.gguf> <dir> [tokens] text generation (Prompt+), greedy, against tools/reference_text_generation.py+ time-matvec <encoder.gguf> the decode's products on the encoder's weights: ms and GB/scheck-steps <dit.gguf> <generate reference dir> <text reference dir>each step alone against tools/reference_generate.py (its latents in)check-noise <philox.bin> the seed's noise against sd.cpp's generator (tools/philox_reference.cpp)@@ -66,6 +68,12 @@ do {let numbers = arguments.dropFirst(2).compactMap { Int($0) }try Checks.timeDiT(path: arguments[1], grid: numbers.first ?? 64, text: numbers.dropFirst().first ?? 32,runs: numbers.dropFirst(2).first ?? 3)+ case "check-textgen":+ guard arguments.count >= 3 else { fail(usage) }+ try Checks.textGeneration(path: arguments[1], directory: arguments[2], tokens: arguments.count > 3 ? Int(arguments[3]) ?? 48 : 48)+ case "time-matvec":+ guard arguments.count == 2 else { fail(usage) }+ try Checks.timeMatvec(path: arguments[1])case "check-ane":try Checks.aneSurfaces()case "check-vae":diff --git a/engine-mac/Tests/KokonEngineTests/QwenTokenizerTests.swift b/engine-mac/Tests/KokonEngineTests/QwenTokenizerTests.swiftindex 6317fb5..83cfce6 100644--- a/engine-mac/Tests/KokonEngineTests/QwenTokenizerTests.swift+++ b/engine-mac/Tests/KokonEngineTests/QwenTokenizerTests.swift@@ -68,4 +68,20 @@ final class QwenTokenizerTests: XCTestCase {XCTAssertEqual(tokenizer.encode("e\u{301}"), [byte(0xC3), byte(0xA9)])XCTAssertEqual(tokenizer.decode(tokenizer.encode("the cat é")), "the cat é")}++ /// Text generation's stream: whole UTF-8 characters only, added tokens left out, invalid bytes as U+FFFD (sd.cpp's+ /// TokenStreamDecoder).+ func testStreamDecoderHoldsPartialCharacters() {+ var stream = TokenStreamDecoder(tokenizer: tokenizer)+ XCTAssertEqual(stream.push(id("the")), "the")+ XCTAssertEqual(stream.push(byte(0xC3)), "") // the first byte of é+ XCTAssertEqual(stream.push(id("<|im_start|>")), "") // an added token: nothing+ XCTAssertEqual(stream.push(byte(0xA9)), "é")+ XCTAssertEqual(stream.push(byte(0xFF)), "\u{FFFD}") // never valid+ XCTAssertEqual(stream.push(byte(0xE2)), "")+ XCTAssertEqual(stream.push(byte(UInt8(ascii: "a"))), "\u{FFFD}a") // a broken sequence, then the next character+ XCTAssertEqual(stream.push(byte(0xF0)), "")+ XCTAssertEqual(stream.flush(), "\u{FFFD}") // unfinished at the end+ XCTAssertEqual(stream.flush(), "")+ }}diff --git a/engine-mac/Tests/KokonEngineTests/TextGenerationTests.swift b/engine-mac/Tests/KokonEngineTests/TextGenerationTests.swiftnew file mode 100644index 0000000..aeedbe6--- /dev/null+++ b/engine-mac/Tests/KokonEngineTests/TextGenerationTests.swift@@ -0,0 +1,102 @@+import XCTest+@testable import KokonEngine++/// Text generation's sampler, which must draw what sd.cpp's draws for a seed (`src/runtime/text_sampler.cpp`).+final class TextGenerationTests: XCTestCase {+ /// The C++ standard fixes mt19937_64's output: the 10 000th number from the default seed 5489.+ func testMersenneTwisterIsTheStandards() {+ var random = MersenneTwister64(seed: 5489)+ XCTAssertEqual(random.next(), 14_514_284_786_278_117_030)+ for _ in 2..<10_000 { _ = random.next() }+ XCTAssertEqual(random.next(), 9_981_545_732_273_789_042)+ }++ func testGreedyTakesTheLowestIDOnTiesAndPenalizesSeenTokens() {+ var sampler = TextSampler(TextSampler.Parameters(temperature: 0, topP: 1, topK: 0, presencePenalty: 1.5), seed: 1)+ let logits: [Float] = [1, 3, 3, 2]+ XCTAssertEqual(logits.withUnsafeBufferPointer { sampler.sample($0.baseAddress!, count: 4) }, 1)+ // 1 was generated: 3 − 1.5 < 3, so 2 wins.+ XCTAssertEqual(logits.withUnsafeBufferPointer { sampler.sample($0.baseAddress!, count: 4) }, 2)+ // Both penalized to 1.5: 3 (logit 2) wins.+ XCTAssertEqual(logits.withUnsafeBufferPointer { sampler.sample($0.baseAddress!, count: 4) }, 3)+ }++ func testCandidatesTopKTemperatureTopP() {+ let sampler = TextSampler(TextSampler.Parameters(temperature: 0.5, topP: 0.8, topK: 3, presencePenalty: 0), seed: 1)+ let logits: [Float] = [0, 2, 1, 2, -1]+ let c = logits.withUnsafeBufferPointer { sampler.candidates($0.baseAddress!, count: 5) }+ // Top 3 by logit (ties by id): 1, 3, 2; at T 0.5 their weights are 1, 1, e^-2; 1 + 1 reach 0.8 of the sum.+ XCTAssertEqual(c.map(\.token), [1, 3])+ XCTAssertEqual(c[0].p, 0.5, accuracy: 1e-12)+ let all = TextSampler(TextSampler.Parameters(temperature: 1, topP: 1, topK: 0, presencePenalty: 0), seed: 1)+ let p = logits.withUnsafeBufferPointer { all.candidates($0.baseAddress!, count: 5) }+ XCTAssertEqual(p.map(\.token), [1, 3, 2, 0, 4])+ XCTAssertEqual(p.reduce(0) { $0 + $1.p }, 1, accuracy: 1e-12)+ XCTAssertEqual(p[2].p / p[0].p, exp(-1), accuracy: 1e-12)+ }++ /// A draw is the first candidate whose cumulative probability passes a uniform double of 53 bits from the seed.+ func testDrawFollowsTheSeed() {+ let parameters = TextSampler.Parameters(temperature: 1, topP: 1, topK: 0, presencePenalty: 0)+ let logits: [Float] = [0, 0] // 0.5 each: token 0 when u < 0.5+ for seed: UInt64 in [1, 2, 3, 42] {+ var sampler = TextSampler(parameters, seed: seed)+ var random = MersenneTwister64(seed: seed)+ let u = Double(random.next() >> 11) / 9_007_199_254_740_992.0+ XCTAssertEqual(logits.withUnsafeBufferPointer { sampler.sample($0.baseAddress!, count: 2) }, u < 0.5 ? 0 : 1)+ }+ }++ func testTemplateIsSdcpps() {+ XCTAssertEqual(QwenTextGenerator.template(system: "S", prompt: "P"),+ "<|im_start|>system\nS<|im_end|>\n<|im_start|>user\nP<|im_end|>\n<|im_start|>assistant\n")+ XCTAssertEqual(QwenTextGenerator.template(system: "", prompt: "P"), "<|im_start|>user\nP<|im_end|>\n<|im_start|>assistant\n")+ }+}++/// The decode's matrix-vector products on Q4_K and Q6_K blocks against the CPU's unpacking (`Dequant.row`) and a dot+/// product in double: random blocks with sane scales, 3 blocks a row (ragged across the SIMD group's lanes).+final class MatvecKernelTests: XCTestCase {+ func testMatvecMatchesTheUnpackedRows() throws {+ let context = try MetalContext()+ var s: UInt64 = 7+ func next() -> UInt64 { s = s &* 6364136223846793005 &+ 1442695040888963407; return s >> 33 }+ for (type, kernel) in [(QuantType.q4_K, "matvec_q4_K"), (.q6_K, "matvec_q6_K")] {+ let rows = 37, columns = 3 * 256, rowBytes = type.bytes(count: columns)+ var bytes = (0..<(rows * rowBytes)).map { _ in UInt8(truncatingIfNeeded: next()) }+ // The fp16 scales of every block: small positive values (random bytes may be NaN or huge).+ for r in 0..<rows {+ for b in 0..<3 {+ let block = r * rowBytes + b * type.blockBytes+ let scaleOffsets = type == .q4_K ? [0, 2] : [208]+ for o in scaleOffsets {+ let v = Float16(Float(next() % 1000 + 1) * 1e-5)+ withUnsafeBytes(of: v.bitPattern.littleEndian) { bytes[block + o] = $0[0]; bytes[block + o + 1] = $0[1] }+ }+ }+ }+ let x = (0..<columns).map { _ in Float(Int(next() % 2001) - 1000) / 500 }+ let w = try context.scratch(bytes.count, label: "w"), xb = try context.scratch(columns * 4, label: "x"), y = try context.scratch(rows * 4, label: "y")+ w.contents().copyMemory(from: bytes, byteCount: bytes.count)+ xb.contents().copyMemory(from: x, byteCount: columns * 4)+ let commands = context.queue.makeCommandBuffer()!+ let e = commands.makeComputeCommandEncoder()!+ e.setComputePipelineState(try context.pipeline(kernel))+ e.setBuffer(w, offset: 0, index: 0); e.setBuffer(xb, offset: 0, index: 1); e.setBuffer(y, offset: 0, index: 2)+ var params = QwenTextGenerator.MatvecParams(rows: UInt32(rows), blocksPerRow: 3, rowBytes: UInt32(rowBytes))+ e.setBytes(¶ms, length: MemoryLayout<QwenTextGenerator.MatvecParams>.stride, index: 3)+ let perGroup = QwenTextGenerator.matvecRows(type)+ e.dispatchThreadgroups(MTLSize(width: (rows + perGroup - 1) / perGroup, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 128, height: 1, depth: 1))+ e.endEncoding()+ try MetalContext.finish(commands, kernel)+ let got = y.contents().assumingMemoryBound(to: Float.self)+ var unpacked = [Float](repeating: 0, count: columns)+ for r in 0..<rows {+ bytes.withUnsafeBytes { Dequant.row($0.baseAddress! + r * rowBytes, type: type, count: columns, into: &unpacked) }+ let want = zip(unpacked, x).reduce(0.0) { $0 + Double($1.0) * Double($1.1) }+ let scale = zip(unpacked, x).reduce(0.0) { $0 + abs(Double($1.0) * Double($1.1)) }+ XCTAssertEqual(Double(got[r]), want, accuracy: 1e-5 * scale + 1e-9, "\(kernel) row \(r)")+ }+ }+ }+}diff --git a/engine-mac/tools/api_check.py b/engine-mac/tools/api_check.pyindex 3f09f77..8c1d2aa 100644--- a/engine-mac/tools/api_check.py+++ b/engine-mac/tools/api_check.py@@ -1,10 +1,12 @@#!/usr/bin/env python3"""Text to image through a running `kokon-engine-mac serve` (or sd-server) as qls-core sends it: capabilities,-img_gen, polling jobs/{id}, optionally a cancel. Prints the time of each image and writes the PNGs.+img_gen, polling jobs/{id}, optionally a cancel; with `--enhance`, Prompt+ first (text_gen with Kokon's built-in+system prompt, read from crates/qls-core/src/enhancer.rs, and its sampling). Prints the time of each image and writes+the PNGs.- python3 -I tools/api_check.py --port 1234 --images 2 --out /tmp/api [--size 1024x1024] [--cancel-after 10]+ python3 -I tools/api_check.py --port 1234 --images 2 --out /tmp/api [--size 1024x1024] [--cancel-after 10] [--enhance]"""-import argparse, base64, json, os, time, urllib.request, urllib.error+import argparse, base64, json, os, re, time, urllib.request, urllib.errorTURBO_SIGMAS = [1.0, 0.978453, 0.95418, 0.926626, 0.89508, 0.845148, 0.704534, 0.414568, 0.0]@@ -29,17 +31,30 @@ def main():p.add_argument("--seed", type=int, default=42)p.add_argument("--out", default=".")p.add_argument("--cancel-after", type=float, default=0, help="cancel the last image after this many seconds")+ p.add_argument("--enhance", action="store_true", help="Prompt+ writes each image's prompt first")a = p.parse_args()+ enhancer = os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../crates/qls-core/src/enhancer.rs")+ system = re.search(r'pub const T2I_SYSTEM_PROMPT: &str = r#"(.*?)"#;', open(enhancer).read(), re.S).group(1) if a.enhance else ""width, height = (int(v) for v in a.size.split("x"))status, caps = call(a.port, "GET", "/sdcpp/v1/capabilities", timeout=10)print(f"capabilities {status}: engine {caps.get('engine', 'sd-server')}, model {caps.get('model', {}).get('name')}")os.makedirs(a.out, exist_ok=True)for i in range(a.images):- body = {"prompt": a.prompt, "negative_prompt": "", "width": width, "height": height, "seed": a.seed + i,+ started = time.time()+ prompt = a.prompt+ if a.enhance:+ status, done = call(a.port, "POST", "/sdcpp/v1/text_gen", {+ "prompt": a.prompt, "system_prompt": system, "max_tokens": 1024, "stream": False,+ "sampling": {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "presence_penalty": 1.5, "seed": a.seed + i}}, timeout=300)+ if status != 200:+ print(f"text_gen {status}: {done}")+ return 1+ prompt = done["text"]+ print(f"Prompt+ {time.time() - started:.1f} s, {done['usage']['completion_tokens']} tokens, {done['timings']}: {prompt[:80]}…")+ body = {"prompt": prompt, "negative_prompt": "", "width": width, "height": height, "seed": a.seed + i,"batch_count": 1, "sample_params": {"sample_method": "euler", "sample_steps": 8,"custom_sigmas": TURBO_SIGMAS, "guidance": {"txt_cfg": 1.0}},"output_format": "png", "embed_image_metadata": False}- started = time.time()status, accepted = call(a.port, "POST", "/sdcpp/v1/img_gen", body)if status != 202:print(f"img_gen {status}: {accepted}")diff --git a/engine-mac/tools/reference_text_generation.py b/engine-mac/tools/reference_text_generation.pynew file mode 100644index 0000000..20f567d--- /dev/null+++ b/engine-mac/tools/reference_text_generation.py@@ -0,0 +1,67 @@+"""The text generation reference for the Mac engine (Prompt+): transformers' own Qwen3-VL text decoder layers in fp32+on the CPU, on the GGUF's weights (as `reference_text_encoder.py`), run once on the prompt followed by the tokens the+engine generated (teacher forcing), with the final RMSNorm and the language model head on every generated position.++ kokon-engine-mac check-textgen <encoder.gguf> <dir> # 1. the engine, greedy: <dir>/engine.json + its logits+ python -I engine-mac/tools/reference_text_generation.py <encoder.gguf> <processor dir> <text_encoder dir> <dir>+ kokon-engine-mac check-textgen <encoder.gguf> <dir> # 2. the same tokens again, compared with the reference++The prompt is tokenized by the official processor as sd.cpp's template writes it (system turn, user turn, assistant),+and must give the engine's ids. Writes `<dir>/reference_logits.f32` ([generated, vocabulary] fp32: the logits that+predict each generated token) and `<dir>/reference.json` (its own greedy choice at each position).+"""+import argparse+import json+import os+import sys++import torch+from transformers import AutoConfig, AutoProcessor+from transformers.models.qwen3_vl.modeling_qwen3_vl import Qwen3VLTextRMSNorm++sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))+from reference_text_encoder import Weights, run # noqa: E402+++def main():+ parser = argparse.ArgumentParser()+ parser.add_argument("gguf")+ parser.add_argument("processor")+ parser.add_argument("text_encoder", help="text_encoder/ of the official repository (config.json)")+ parser.add_argument("dir", help="where check-textgen wrote engine.json")+ args = parser.parse_args()+ torch.set_grad_enabled(False)+ with open(os.path.join(args.dir, "engine.json")) as f:+ engine = json.load(f)+ config = AutoConfig.from_pretrained(args.text_encoder).text_config+ config._attn_implementation = "eager"+ processor = AutoProcessor.from_pretrained(args.processor)++ # sd.cpp's template (src/pipeline/text.cpp), text only.+ system, prompt = engine["system"], engine["prompt"]+ text = (f"<|im_start|>system\n{system}<|im_end|>\n" if system else "") + f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"+ ids = processor(text=[text], return_tensors="pt").input_ids[0].tolist()+ if ids != engine["prompt_ids"]:+ sys.exit(f"prompt ids differ from the engine's: {len(ids)} against {len(engine['prompt_ids'])}")+ generated = engine["generated"]+ sequence = ids + generated[:-1] # each position predicts the next token++ weights = Weights(args.gguf)+ hidden = run(weights, config, sequence)[0]+ norm = Qwen3VLTextRMSNorm(config.hidden_size, eps=config.rms_norm_eps)+ norm.weight.copy_(weights.get("output_norm.weight"))+ head = weights.get("output.weight")+ logits = head @ norm(hidden[len(ids) - 1:]).T # [vocabulary, generated]+ logits = logits.T.contiguous()+ logits.numpy().astype("float32").tofile(os.path.join(args.dir, "reference_logits.f32"))+ greedy = logits.argmax(dim=1).tolist()+ with open(os.path.join(args.dir, "reference.json"), "w") as f:+ json.dump({"greedy": greedy, "positions": len(generated), "transformers": __import__("transformers").__version__,+ "torch": torch.__version__}, f, indent=1)+ same = sum(int(a == b) for a, b in zip(greedy, generated))+ print(f"{len(ids)} prompt tokens, {len(generated)} generated; the reference's greedy choice agrees at {same}")+ print("reference text:", processor.tokenizer.decode(greedy))+++if __name__ == "__main__":+ main()