Skip to content

armgit / kokon

Public Imported from GitHub
0

Edit photos by describing the change, on your own GPU. A free, open-source alternative to Nano Banana and ChatGPT image editing: photos stay on your computer, no daily limit. One installer for Windows, Mac and Linux sets up the best open model (Qwen-Image 2.1). MCP server for AI agents.

Mac engine: Prompt+ (text_gen) on the engine

Gabriele Lozupone2e0e235cdcf3
diff --git a/docs/STATUS.md b/docs/STATUS.md
index de83013..0fbea5a 100644
--- a/docs/STATUS.md
+++ b/docs/STATUS.md
@@ -24,9 +24,10 @@ Windows, macOS (Apple Silicon, 15+) and Linux (`.deb`, `.rpm`). Installed copies
- **LoRA is beta** and tested on Windows only; reports from macOS and Linux are welcome
([#33](https://github.com/GabrieleLozupone/kokon/issues/33) has a product-shot test).
- **Macs are slow** compared with NVIDIA cards (about 4 minutes for 1 MP on an M2 Pro 16 GB). Kokon's own Mac
- engine is in development on the `macos-optimization` branch (`engine-mac/`, not released): text to image only,
- about 64–67 s for 1 MP on the same Mac through the app's API (peak memory 5 GB), tested from the app, as an experimental switch in Settings → Performance
- (off by default). Edits, LoRAs, Prompt+ and the upscaler still need the standard engine.
+ engine is in development on the `macos-optimization` branch (`engine-mac/`, not released): text to image and
+ Prompt+ (a prompt written in about 7–9 s), about 64–67 s for 1 MP on the same Mac through the app's API (peak memory 5 GB),
+ tested from the app, as an experimental switch in Settings → Performance (off by default). Edits, LoRAs and the
+ upscaler still need the standard engine.
- The Windows installer is not code-signed: SmartScreen or Smart App Control may block it (README FAQ: Unblock).
- HEIC photos can't be opened on Linux yet.
diff --git a/engine-mac/README.md b/engine-mac/README.md
index 7c5d2d9..8cd2365 100644
--- a/engine-mac/README.md
+++ b/engine-mac/README.md
@@ -16,7 +16,7 @@ swift test # no model needed
.build/release/kokon-engine-mac generate <encoder.gguf> <dit.gguf> <vae.safetensors> "a red fox in the snow" \
[--size 1024x1024] [--seed 42] [--out out.png]
.build/release/kokon-engine-mac serve --diffusion-model <dit.gguf> --vae <vae.safetensors> --llm <encoder.gguf> \
- [--listen-port 1234] # sd-server's /sdcpp/v1 API, text to image
+ [--listen-port 1234] # sd-server's /sdcpp/v1 API, text to image and Prompt+
.build/release/kokon-engine-mac bench # a few minutes: speed and accuracy of this Mac's paths, a report
.build/release/kokon-engine-mac check-na # the Metal 4 matrix product against MPS
.build/release/kokon-engine-mac check-ane8 # the int8 Neural Engine programs against the fp16 one
@@ -47,7 +47,8 @@ Windows and Linux.
| `QwenTokenizer.swift` | Qwen's byte-level BPE tokenizer, read from the encoder GGUF's metadata |
| `QwenTextEncoder.swift`, `Kernels/text.metal` | the text encoder: Qwen3-VL-8B's text decoder on the GPU, diffusers' prompt template |
| `QwenPipeline.swift` | text to image: noise, Turbo's 8 flow-Euler steps (`planner::TURBO_SIGMAS`), VAE, PNG; models kept loaded for the server |
-| `EngineServer.swift`, `HTTPServer.swift` | `serve`: sd-server's `/sdcpp/v1` API (capabilities, img_gen, jobs, cancel) and its log lines |
+| `EngineServer.swift`, `HTTPServer.swift` | `serve`: sd-server's `/sdcpp/v1` API (capabilities, img_gen, jobs, cancel, text_gen) and its log lines |
+| `QwenTextGeneration.swift`, `Kernels/decode.metal` | Prompt+: the text encoder's LLM writes, K/V cache, products on the Q4_K / Q6_K blocks, sd.cpp's sampler |
| `Chip.swift` | this Mac's chip, GPU family, cores and memory; each unit's estimated speed against the M2 Pro's |
| `NeuralAccelerators.swift`, `Kernels/metal4/na_matmul.metal` | the GPU's rows of a product on Metal 4 tensor operations (the M5's neural accelerators) |
| `Bench.swift` | `bench`: the report testers paste in GitHub |
@@ -248,7 +249,8 @@ Engine's programs, are not pipelines an archive holds. Only an M2 Pro has been m
`kokon-engine-mac serve` answers sd-server's `/sdcpp/v1` API (capabilities, img_gen, jobs/{id}, cancel), so
qls-core, the app, remote access and MCP use it as they use sd-server; it prints the log lines qls-core's
`ProgressParser` reads. Text to image only for now: reference images, masks, LoRAs and second passes are refused
-with 400 (qls-core says so before submitting), upscaling falls back to Lanczos, Prompt+ is off. A running image
+with 400 (qls-core says so before submitting), upscaling falls back to Lanczos. Prompt+ works on text-to-image prompts
+(`text_gen`, below); with images (edits, Compose) it needs the vision tower. A running image
cancels within a DiT layer (sd-server answers 409 and qls-core restarts it). A new prompt length rebuilds the DiT's
working memory in ~0.5 s.
@@ -258,6 +260,39 @@ operations and tiles (below): **first image 67.0 s, the next ones 63.9–64.5 s,
peak 9.6 GB). Keeping the compiled VAE between images now costs only 1.7 GB, but it did not pay on 16 GB: its decode
3.2 s instead of 4.0, the images 68.4–70.3 s (with the DiT kept too 70.8 / 68.3 s; single runs in a row, not alternated).
+**Prompt+ (`POST /sdcpp/v1/text_gen`)**: the text encoder's LLM writes the prompt, as Kokon's sd.cpp does, on the same
+Qwen3-VL Q4_K_M file (`QwenTextGeneration.swift`, `Kernels/decode.metal`): sd.cpp's chat template and stop tokens, its
+sampler step for step (presence penalty, top-k, temperature, top-p, one draw from mt19937_64: a seed writes the same
+text where no two logits nearly tie), its stream of whole UTF-8 characters, its Server-Sent Events, and a dropped
+connection stops it. The prompt runs through the encoder's forward, which keeps each layer's K and V in an fp16 cache
+(144 KB a token); then each token reads every weight once, so the decode's matrix-vector products read the Q4_K and
+Q6_K blocks where they lie in the mapped file. Text and image generation take turns on the GPU.
+
+Against transformers (`check-textgen` + `tools/reference_text_generation.py`: the engine writes greedily, transformers'
+fp32 decoder layers then run once on the prompt and those tokens; 68-token prompt, 48 tokens, M2 Pro, 2026-10-25): the
+prompt's ids identical, the first token's logits within 3.2e-6, the next ones within 4.6e-4 (the fp16 cache: with an fp32
+cache every step within 3.2e-6), the greedy choice the same at all 48 (one near-tie at 0.04 among them).
+
+```bash
+.build/release/kokon-engine-mac check-textgen <encoder.gguf> ~/kokon-bench/ref/textgen-cat # writes engine.json
+~/kokon-bench/venv-oracle/bin/python -I tools/reference_text_generation.py <encoder.gguf> <processor dir> <text_encoder dir> ~/kokon-bench/ref/textgen-cat
+.build/release/kokon-engine-mac check-textgen <encoder.gguf> ~/kokon-bench/ref/textgen-cat # compares
+.build/release/kokon-engine-mac time-matvec <encoder.gguf> # the decode's products alone, GB/s
+```
+
+Speed (M2 Pro 16 GB, 2026-10-25): **a token in 32 ms (31 tokens/s; ~4.9 GB read a token, ~155 GB/s)**; Kokon's
+built-in system prompt and a short request, through the server: 481 prompt tokens, 142 written, 6.6–7.8 s (prefill
+1.8–2.4 s, 26.8–29.5 tokens/s as the cache grows; 8.2 s while the sampler looked the presence penalty up in a set for
+each of the 151 936 tokens, a dense array now as sd.cpp's). The products' kernels (`time-matvec`, GB/s on the file's weights): one lane per 64
+values, one row per SIMD group: Q4_K 25–49, Q6_K 39–53, 10.5 tokens/s; 8 lanes a block with wide loads, 2 rows: Q6_K
+115–182; Q4_K 4 rows: 51–68; its scales and mins unpacked from one 16-byte load without a branch (they were half its
+time): 112–120. One compute encoder per token instead of one per kernel (~500): no change (32.5–34.9 ms against
+31.8–32.5), not kept. Of a 12-token prefill's 2.2–2.5 s, ~2 s is opening the encoder (0.46 s with it warm): when the
+encoder is not kept between images (below 32 GB), the one Prompt+ opened is handed to the next image's encode. Through
+the API at 1024², alternated (`tools/api_check.py --enhance`): the image's text encoder 0.88 s after Prompt+ against
+2.1–2.65 s for a plain image, peak footprint 5.1 GB either way; Prompt+ adds 8.7–9.0 s to an image (75.4–77.2 s against
+65.6–71.3).
+
## Per-chip paths
Selected at run time from the GPU family and the chip (`Chip.swift`), each with the M2 path as its fallback and a
diff --git a/engine-mac/Sources/KokonEngine/EngineServer.swift b/engine-mac/Sources/KokonEngine/EngineServer.swift
index 1b4c0b9..7394acf 100644
--- a/engine-mac/Sources/KokonEngine/EngineServer.swift
+++ b/engine-mac/Sources/KokonEngine/EngineServer.swift
@@ -7,10 +7,11 @@ import Foundation
/// POST /sdcpp/v1/img_gen 202 {id, kind, status, created, poll_url}: the job is queued
/// GET /sdcpp/v1/jobs/{id} {status: queued|generating|completed|failed|cancelled, result, error}
/// POST /sdcpp/v1/jobs/{id}/cancel 200 {status: cancelled}: a running image stops within a DiT layer
+/// POST /sdcpp/v1/text_gen Prompt+: the text encoder's LLM writes, streamed as Server-Sent Events
///
/// Text to image only for now: requests with reference images, a mask, an init image, LoRAs or a second pass are
-/// refused with 400 and say so. Jobs run one at a time; the text encoder, the DiT and the compiled VAE stay loaded
-/// between them (`QwenPipeline.keepModels`).
+/// refused with 400 and say so, and so is text generation with images. Jobs and text generation run one at a time;
+/// what stays loaded between them depends on the Mac (`QwenPipeline.keepModels`).
///
/// Progress goes to stdout as sd-server's log lines, the ones qls-core's `ProgressParser` reads (process.rs):
/// `generate_image WxH`, `sampling using`, `generating image: i/n - seed s`, step bars `| i/n - x.xxs/it`,
@@ -95,6 +96,8 @@ public final class EngineServer: @unchecked Sendable {
private var running: Job?
private let lock = NSLock()
private let wake = DispatchSemaphore(value: 0)
+ /// Held by a running image or text generation: one GPU user at a time.
+ private let gpu = NSLock()
/// Lines to stdout (qls-core reads them from the log file it redirects stdout to).
public var log: @Sendable (String) -> Void = { line in
FileHandle.standardOutput.write(Data((line + "\n").utf8))
@@ -125,7 +128,8 @@ public final class EngineServer: @unchecked Sendable {
case ("GET", let p) where p.count == 2 && p[0] == "jobs":
return lock.withLock { jobs[p[1]].map { .json(200, describe($0)) } ?? .error(404, "unknown job \(p[1])") }
case ("POST", let p) where p.count == 3 && p[0] == "jobs" && p[2] == "cancel": return cancel(p[1])
- case (_, ["upscale"]), (_, ["text_gen"]):
+ case ("POST", ["text_gen"]): return textGen(request.body)
+ case (_, ["upscale"]):
return .error(404, "Kokon's Mac engine has no \(parts[2]) endpoint yet")
default: return .error(404, "not found: \(request.method) \(request.path)")
}
@@ -143,12 +147,13 @@ public final class EngineServer: @unchecked Sendable {
"schedulers": [String](),
"output_formats": ["png"],
"limits": ["min_width": 64, "max_width": 4096, "min_height": 64, "max_height": 4096,
- "max_batch_count": EngineServer.maxBatch, "max_queue_size": EngineServer.maxQueue],
+ "max_batch_count": EngineServer.maxBatch, "max_queue_size": EngineServer.maxQueue,
+ "max_text_tokens": EngineServer.maxTextTokens, "max_text_images": 0],
"features_by_mode": ["img_gen": feature],
"defaults_by_mode": ["img_gen": ["sample_steps": QwenPipeline.turboSigmas.count - 1, "txt_cfg": 1]],
"upscalers": [Any](),
"upscale": false,
- "text_generation": false,
+ "text_generation": true,
]
}
@@ -193,6 +198,97 @@ public final class EngineServer: @unchecked Sendable {
return out
}
+ // MARK: - text generation (Prompt+)
+
+ /// The fields of `text_gen` (sd-server's names and checks, `examples/server/routes_sdcpp.cpp`).
+ struct TextRequest: Sendable {
+ let system, prompt: String
+ let maxTokens: Int
+ let sampling: TextSampler.Parameters
+ let seed: Int64
+ let stream: Bool
+
+ static func parse(_ body: Data) -> (TextRequest?, String?) {
+ guard let json = (try? JSONSerialization.jsonObject(with: body)) as? [String: Any] else { return (nil, "body must be an object") }
+ guard let prompt = json["prompt"] as? String, !prompt.isEmpty else { return (nil, "prompt is required") }
+ guard let system = json["system_prompt"] as? String else {
+ return (nil, "system_prompt is required (an empty string means no system prompt)")
+ }
+ let maxTokens = json["max_tokens"] as? Int ?? 512
+ guard (1...EngineServer.maxTextTokens).contains(maxTokens) else { return (nil, "max_tokens must be between 1 and \(EngineServer.maxTextTokens)") }
+ if let images = json["images"] as? [Any], !images.isEmpty {
+ return (nil, "Kokon's Mac engine cannot look at images yet (no vision tower): Prompt+ works on text-to-image prompts only")
+ }
+ var sampling = TextSampler.Parameters(temperature: 0, topP: 1, topK: 0, presencePenalty: 0)
+ var seed: Int64 = -1
+ if let s = json["sampling"] {
+ guard let s = s as? [String: Any] else { return (nil, "sampling must be an object") }
+ func number(_ key: String) -> Double? { (s[key] as? NSNumber)?.doubleValue }
+ sampling.temperature = Float(number("temperature") ?? 0)
+ sampling.topP = Float(number("top_p") ?? 1)
+ sampling.topK = Int(number("top_k") ?? 0)
+ sampling.presencePenalty = Float(number("presence_penalty") ?? 0)
+ seed = (s["seed"] as? NSNumber)?.int64Value ?? -1
+ guard sampling.temperature.isFinite, sampling.temperature >= 0, sampling.topP.isFinite, sampling.topP > 0, sampling.topP <= 1,
+ sampling.topK >= 0, sampling.presencePenalty.isFinite else {
+ return (nil, "sampling needs temperature >= 0, 0 < top_p <= 1, top_k >= 0 and a finite presence_penalty")
+ }
+ }
+ return (TextRequest(system: system, prompt: prompt, maxTokens: maxTokens, sampling: sampling, seed: seed,
+ stream: json["stream"] as? Bool ?? true), nil)
+ }
+ }
+
+ static let maxTextTokens = 4096
+
+ /// Synchronous like sd-server's: it waits for a running image (one GPU user at a time) and holds the models while it
+ /// writes. Streamed (the default) as Server-Sent Events: `delta` {text} per piece, then `done` {text, finish_reason,
+ /// seed, usage, timings} or `error` {error}; closing the connection stops it.
+ func textGen(_ body: Data) -> HTTPServer.Response {
+ let (parsed, reason) = TextRequest.parse(body)
+ guard let request = parsed else { return .error(400, reason ?? "bad request") }
+ guard request.stream else {
+ let outcome = Result { try gpu.withLock { try generateText(request) { _ in true } } }
+ switch outcome {
+ case .success(let result): return .json(200, EngineServer.done(result))
+ case .failure(let error):
+ log("[ERROR ] engine-mac - text generation failed: \(error)")
+ return .error(500, "text generation failed")
+ }
+ }
+ return .streaming(contentType: "text/event-stream") { [self] write in
+ // Before waiting for a running image, so the client gets the headers at once.
+ guard write(Data(": waiting for the model\n\n".utf8)) else { return }
+ func event(_ name: String, _ object: Any) -> Bool {
+ let data = (try? JSONSerialization.data(withJSONObject: object)) ?? Data("{}".utf8)
+ return write(Data("event: \(name)\ndata: ".utf8) + data + Data("\n\n".utf8))
+ }
+ let outcome = Result { try gpu.withLock { try generateText(request) { piece in event("delta", ["text": piece]) } } }
+ switch outcome {
+ case .success(let result): _ = event("done", EngineServer.done(result))
+ case .failure(let error):
+ log("[ERROR ] engine-mac - text generation failed: \(error)")
+ _ = event("error", ["error": "text generation failed"])
+ }
+ }
+ }
+
+ private func generateText(_ r: TextRequest, onPiece: (String) -> Bool) throws -> QwenTextGenerator.Result {
+ let result = try pipeline.generateText(system: r.system, prompt: r.prompt, maxTokens: r.maxTokens, sampling: r.sampling,
+ seed: r.seed, onPiece: onPiece)
+ log(String(format: "[INFO ] engine-mac - text generation: %d prompt tokens, %d generated in %.2fs (prefill %.2fs, %.1f tok/s), seed %lld",
+ result.promptTokens, result.generatedTokens, result.prepare + result.prefill + result.decode, result.prefill,
+ result.decode > 0 ? Double(result.tokens.count - 1) / result.decode : 0, result.seed))
+ return result
+ }
+
+ /// sd-server's `done` object.
+ static func done(_ r: QwenTextGenerator.Result) -> [String: Any] {
+ ["text": r.text, "finish_reason": r.finish.rawValue, "seed": r.seed,
+ "usage": ["prompt_tokens": r.promptTokens, "completion_tokens": r.generatedTokens],
+ "timings": ["prepare_s": r.prepare, "image_s": 0, "prefill_s": r.prefill, "decode_s": r.decode]]
+ }
+
// MARK: - the jobs
private func work() {
@@ -206,7 +302,7 @@ public final class EngineServer: @unchecked Sendable {
running = job
return job
}) else { continue }
- let outcome: Result<[String], Error> = autoreleasepool { Result { try generate(job) } }
+ let outcome: Result<[String], Error> = autoreleasepool { Result { try gpu.withLock { try generate(job) } } }
lock.withLock {
running = nil
job.completed = Int(Date().timeIntervalSince1970)
diff --git a/engine-mac/Sources/KokonEngine/HTTPServer.swift b/engine-mac/Sources/KokonEngine/HTTPServer.swift
index 2e5fbcd..af6c8c3 100644
--- a/engine-mac/Sources/KokonEngine/HTTPServer.swift
+++ b/engine-mac/Sources/KokonEngine/HTTPServer.swift
@@ -1,8 +1,9 @@
import Foundation
/// **A minimal HTTP/1.1 server on a loopback port**, enough for qls-core's client (reqwest): one request per
-/// connection (`Connection: close`), a `Content-Length` body, JSON in and out. POSIX sockets, a thread per
-/// connection: the engine serves one local client, and the slow part of every request is the GPU, not the socket.
+/// connection (`Connection: close`), a `Content-Length` body, JSON in and out, or a body written as it is made
+/// (`Response.stream`, chunked: `text_gen`'s Server-Sent Events). POSIX sockets, a thread per connection: the engine
+/// serves one local client, and the slow part of every request is the GPU, not the socket.
public final class HTTPServer: @unchecked Sendable {
public struct Request: Sendable {
public let method: String
@@ -14,6 +15,9 @@ public final class HTTPServer: @unchecked Sendable {
public var status: Int
public var body: Data
public var contentType = "application/json"
+ /// A body written piece by piece (chunked transfer): `stream` gets a writer, which returns false once the client
+ /// has gone (the connection closed).
+ public var stream: (@Sendable (_ write: (Data) -> Bool) -> Void)?
public init(status: Int, body: Data, contentType: String = "application/json") {
self.status = status
@@ -21,6 +25,12 @@ public final class HTTPServer: @unchecked Sendable {
self.contentType = contentType
}
+ public static func streaming(contentType: String, _ stream: @escaping @Sendable (_ write: (Data) -> Bool) -> Void) -> Response {
+ var r = Response(status: 200, body: Data(), contentType: contentType)
+ r.stream = stream
+ return r
+ }
+
public static func json(_ status: Int, _ object: Any) -> Response {
let data = (try? JSONSerialization.data(withJSONObject: object, options: [.sortedKeys])) ?? Data("{}".utf8)
return Response(status: status, body: data)
@@ -136,15 +146,35 @@ public final class HTTPServer: @unchecked Sendable {
static func write(_ fd: Int32, _ response: Response) {
let reason = [200: "OK", 202: "Accepted", 400: "Bad Request", 404: "Not Found", 409: "Conflict",
429: "Too Many Requests", 500: "Internal Server Error"][response.status] ?? "Status"
+ if let stream = response.stream {
+ guard send(fd, Data("HTTP/1.1 \(response.status) \(reason)\r\nContent-Type: \(response.contentType)\r\nCache-Control: no-cache\r\nTransfer-Encoding: chunked\r\nConnection: close\r\n\r\n".utf8)) else { return }
+ var open = true
+ stream { piece in
+ guard open, !piece.isEmpty else { return open }
+ var chunk = Data((String(piece.count, radix: 16) + "\r\n").utf8)
+ chunk.append(piece)
+ chunk.append(contentsOf: "\r\n".utf8)
+ open = send(fd, chunk)
+ return open
+ }
+ if open { _ = send(fd, Data("0\r\n\r\n".utf8)) }
+ return
+ }
var out = Data("HTTP/1.1 \(response.status) \(reason)\r\nContent-Type: \(response.contentType)\r\nContent-Length: \(response.body.count)\r\nConnection: close\r\n\r\n".utf8)
out.append(response.body)
- out.withUnsafeBytes { raw in
+ _ = send(fd, out)
+ }
+
+ /// All of `data`; false when the connection is gone.
+ static func send(_ fd: Int32, _ data: Data) -> Bool {
+ data.withUnsafeBytes { raw in
var sent = 0
while sent < raw.count {
- let n = send(fd, raw.baseAddress! + sent, raw.count - sent, 0)
- if n <= 0 { return }
+ let n = Darwin.send(fd, raw.baseAddress! + sent, raw.count - sent, 0)
+ if n <= 0 { return false }
sent += n
}
+ return true
}
}
}
diff --git a/engine-mac/Sources/KokonEngine/Kernels/decode.metal b/engine-mac/Sources/KokonEngine/Kernels/decode.metal
new file mode 100644
index 0000000..34a277c
--- /dev/null
+++ b/engine-mac/Sources/KokonEngine/Kernels/decode.metal
@@ -0,0 +1,166 @@
+#include <metal_stdlib>
+using namespace metal;
+
+// Text generation (Prompt+, QwenTextGeneration.swift): one token at a time, so every product is a matrix-vector
+// product that reads the whole weight once. They read the GGUF's Q4_K and Q6_K blocks where they lie (the mapped
+// file), with ggml's formulas (dequant.metal), and add in fp32: the decode is bound by those bytes.
+
+struct MatvecParams {
+ uint rows; // the weight's ggml rows (output features)
+ uint blocksPerRow; // 256-value blocks along the input
+ uint rowBytes;
+};
+
+// y[r] = W[r, :] · x. Two rows per SIMD group, 4 SIMD groups a threadgroup. 8 lanes share a block, each taking 32 of
+// its values (Q4_K: one 16-byte load of nibbles, low and high halves from two sub-blocks; Q6_K: 8 bytes of each of
+// its three byte runs, 2-byte aligned as its 210-byte blocks are), so the 32 lanes read 4 neighbouring blocks of a row
+// at once; the activations a lane needs are loaded once for both rows. M2 Pro, 2026-10-25 (`time-matvec`): one lane
+// per 64 values and one row per SIMD group read 25–53 GB/s.
+#define MATVEC_ROWS 2
+#define MATVEC_Q4_ROWS 4
+
+static inline float4 nibbles(uint v) { return float4(as_type<uchar4>(v & 0x0F0F0F0Fu)); }
+
+kernel void matvec_q4_K(device const uchar *w [[buffer(0)]], device const float *x [[buffer(1)]],
+ device float *y [[buffer(2)]], constant MatvecParams &p [[buffer(3)]],
+ uint group [[threadgroup_position_in_grid]], uint lane [[thread_index_in_simdgroup]],
+ uint simd [[simdgroup_index_in_threadgroup]], uint groups [[simdgroups_per_threadgroup]]) {
+ uint row0 = (group * groups + simd) * MATVEC_Q4_ROWS;
+ if (row0 >= p.rows) return;
+ uint pair = (lane % 8) / 2, part = lane % 2; // sub-blocks 2·pair (low nibbles) and 2·pair + 1 (high)
+ float sums[MATVEC_Q4_ROWS] = {0};
+ for (uint b = lane / 8; b < p.blocksPerRow; b += 4) {
+ device const float4 *xl = (device const float4 *)(x + b * 256 + 64 * pair + 16 * part);
+ float4 a0 = xl[0], a1 = xl[1], a2 = xl[2], a3 = xl[3], c0 = xl[8], c1 = xl[9], c2 = xl[10], c3 = xl[11];
+ float sa = dot(a0 + a1 + a2 + a3, float4(1)), sc = dot(c0 + c1 + c2 + c3, float4(1));
+ for (uint r = 0; r < MATVEC_Q4_ROWS; r++) {
+ if (row0 + r >= p.rows) break;
+ device const uchar *block = w + (ulong)(row0 + r) * p.rowBytes + b * 144;
+ // d, dmin and the 12 bytes of 6-bit scales and mins in one load; get_scale_min_k4 for all 8 sub-blocks at
+ // once (bytes 0–3 of each word: sub-blocks 0–3, then 4–7), of which this lane takes 2·pair and 2·pair + 1.
+ uint4 head = *((device const uint4 *)block);
+ half2 dd = as_type<half2>(head.x);
+ uint scales = pair < 2 ? head.y & 0x3F3F3F3Fu : (head.w & 0x0F0F0F0Fu) | (((head.y >> 6) & 0x03030303u) << 4);
+ uint mins = pair < 2 ? head.z & 0x3F3F3F3Fu : ((head.w >> 4) & 0x0F0F0F0Fu) | (((head.z >> 6) & 0x03030303u) << 4);
+ uint shift = 16 * (pair % 2);
+ float d = float(dd.x), dmin = float(dd.y);
+ float sc1 = float((scales >> shift) & 0xFF), sc2 = float((scales >> (shift + 8)) & 0xFF);
+ float m1 = float((mins >> shift) & 0xFF), m2 = float((mins >> (shift + 8)) & 0xFF);
+ uint4 q = *((device const uint4 *)(block + 16 + 32 * pair + 16 * part));
+ float lo = dot(nibbles(q.x), a0) + dot(nibbles(q.y), a1) + dot(nibbles(q.z), a2) + dot(nibbles(q.w), a3);
+ float hi = dot(nibbles(q.x >> 4), c0) + dot(nibbles(q.y >> 4), c1) + dot(nibbles(q.z >> 4), c2) + dot(nibbles(q.w >> 4), c3);
+ sums[r] += d * (sc1 * lo + sc2 * hi) - dmin * (m1 * sa + m2 * sc);
+ }
+ }
+ for (uint r = 0; r < MATVEC_Q4_ROWS; r++) {
+ float total = simd_sum(sums[r]);
+ if (lane == 0 && row0 + r < p.rows) y[row0 + r] = total;
+ }
+}
+
+// 8 bytes from a 2-byte aligned address, as two words.
+static inline uint2 load8(device const uchar *p) {
+ device const ushort *h = (device const ushort *)p;
+ return uint2(uint(h[0]) | (uint(h[1]) << 16), uint(h[2]) | (uint(h[3]) << 16));
+}
+
+static inline float4 q6(uint low, uint high) { return float4(as_type<uchar4>(low | (high << 4))) - 32.0f; }
+
+kernel void matvec_q6_K(device const uchar *w [[buffer(0)]], device const float *x [[buffer(1)]],
+ device float *y [[buffer(2)]], constant MatvecParams &p [[buffer(3)]],
+ uint group [[threadgroup_position_in_grid]], uint lane [[thread_index_in_simdgroup]],
+ uint simd [[simdgroup_index_in_threadgroup]], uint groups [[simdgroups_per_threadgroup]]) {
+ uint row0 = (group * groups + simd) * MATVEC_ROWS;
+ if (row0 >= p.rows) return;
+ // Half h of the block (values 128h …), l from l0 to l0 + 7 with l0 = 16g + 8·part: scales sc[g], sc[g + 2], …
+ uint j = lane % 8, h = j / 4, g = (j / 2) % 2, l0 = 16 * g + 8 * (j % 2);
+ float sums[MATVEC_ROWS] = {0};
+ for (uint b = lane / 8; b < p.blocksPerRow; b += 4) {
+ device const float4 *xl = (device const float4 *)(x + b * 256 + 128 * h + l0);
+ float4 x1a = xl[0], x1b = xl[1], x2a = xl[8], x2b = xl[9], x3a = xl[16], x3b = xl[17], x4a = xl[24], x4b = xl[25];
+ for (uint r = 0; r < MATVEC_ROWS; r++) {
+ if (row0 + r >= p.rows) break;
+ device const uchar *block = w + (ulong)(row0 + r) * p.rowBytes + b * 210;
+ float d = float(*((device const half *)(block + 208)));
+ device const char *sc = (device const char *)(block + 192 + 8 * h + g);
+ uint2 la = load8(block + 64 * h + l0), lb = load8(block + 64 * h + 32 + l0), qh = load8(block + 128 + 32 * h + l0);
+ const uint m4 = 0x0F0F0F0Fu, m2 = 0x03030303u;
+ float s1 = dot(q6(la.x & m4, qh.x & m2), x1a) + dot(q6(la.y & m4, qh.y & m2), x1b);
+ float s2 = dot(q6(lb.x & m4, (qh.x >> 2) & m2), x2a) + dot(q6(lb.y & m4, (qh.y >> 2) & m2), x2b);
+ float s3 = dot(q6((la.x >> 4) & m4, (qh.x >> 4) & m2), x3a) + dot(q6((la.y >> 4) & m4, (qh.y >> 4) & m2), x3b);
+ float s4 = dot(q6((lb.x >> 4) & m4, (qh.x >> 6) & m2), x4a) + dot(q6((lb.y >> 4) & m4, (qh.y >> 6) & m2), x4b);
+ sums[r] += d * (float(sc[0]) * s1 + float(sc[2]) * s2 + float(sc[4]) * s3 + float(sc[6]) * s4);
+ }
+ }
+ for (uint r = 0; r < MATVEC_ROWS; r++) {
+ float total = simd_sum(sums[r]);
+ if (lane == 0 && row0 + r < p.rows) y[row0 + r] = total;
+ }
+}
+
+struct CacheParams {
+ uint rows; // rows of k and v [rows, kvHeads · 128]
+ uint kvHeads;
+ uint capacity; // positions per head in the cache
+ uint position; // of the first row
+};
+
+// K (after its norm and RoPE) and V of `rows` positions into one layer's cache, fp16 [kvHeads, capacity, 128] each.
+kernel void kv_store(device const float *k [[buffer(0)]], device const float *v [[buffer(1)]],
+ device half *kc [[buffer(2)]], device half *vc [[buffer(3)]], constant CacheParams &p [[buffer(4)]],
+ uint2 g [[thread_position_in_grid]]) {
+ uint c = g.x, row = g.y;
+ if (c >= p.kvHeads * 128 || row >= p.rows) return;
+ uint head = c / 128;
+ ulong at = ((ulong)head * p.capacity + p.position + row) * 128 + c % 128;
+ kc[at] = half(k[(ulong)row * p.kvHeads * 128 + c]);
+ vc[at] = half(v[(ulong)row * p.kvHeads * 128 + c]);
+}
+
+struct DecodeAttentionParams {
+ uint keys; // positions 0 … keys − 1 in the cache
+ uint capacity;
+ uint groups; // query heads per key/value head
+ float scale; // 1 / √128
+};
+
+// One query per head against the cached keys (causal: all of them), as Qwen3's grouped attention: query head h reads
+// key/value head h / groups. A threadgroup per query head, 8 SIMD groups each taking every 8th key with its own online
+// softmax, each lane 4 of the 128 dimensions; the 8 partial results are merged at the end.
+kernel void decode_attention(device const float *q [[buffer(0)]], device const half *kc [[buffer(1)]],
+ device const half *vc [[buffer(2)]], device float *out [[buffer(3)]],
+ constant DecodeAttentionParams &p [[buffer(4)]],
+ uint head [[threadgroup_position_in_grid]], uint lane [[thread_index_in_simdgroup]],
+ uint simd [[simdgroup_index_in_threadgroup]]) {
+ threadgroup float partM[8], partL[8];
+ threadgroup float4 partAcc[8][32];
+ float4 qv = ((device const float4 *)(q + head * 128))[lane] * p.scale;
+ ulong base = (ulong)(head / p.groups) * p.capacity * 128;
+ float m = -INFINITY, l = 0;
+ float4 acc = 0;
+ for (uint j = simd; j < p.keys; j += 8) {
+ float4 kv = float4(((device const half4 *)(kc + base + (ulong)j * 128))[lane]);
+ float s = simd_sum(dot(qv, kv));
+ float mNew = max(m, s);
+ float correction = exp(m - mNew), weight = exp(s - mNew);
+ l = l * correction + weight;
+ acc = acc * correction + weight * float4(((device const half4 *)(vc + base + (ulong)j * 128))[lane]);
+ m = mNew;
+ }
+ if (lane == 0) { partM[simd] = m; partL[simd] = l; }
+ partAcc[simd][lane] = acc;
+ threadgroup_barrier(mem_flags::mem_threadgroup);
+ if (simd == 0) {
+ float mAll = -INFINITY;
+ for (uint i = 0; i < 8; i++) mAll = max(mAll, partM[i]);
+ float lAll = 0;
+ float4 total = 0;
+ for (uint i = 0; i < 8; i++) {
+ if (partL[i] == 0) continue; // a SIMD group without keys
+ float f = exp(partM[i] - mAll);
+ lAll += partL[i] * f;
+ total += partAcc[i][lane] * f;
+ }
+ ((device float4 *)(out + head * 128))[lane] = total / lAll;
+ }
+}
diff --git a/engine-mac/Sources/KokonEngine/QwenPipeline.swift b/engine-mac/Sources/KokonEngine/QwenPipeline.swift
index 0ae5063..642cbc8 100644
--- a/engine-mac/Sources/KokonEngine/QwenPipeline.swift
+++ b/engine-mac/Sources/KokonEngine/QwenPipeline.swift
@@ -51,6 +51,11 @@ public final class QwenPipeline {
}
}
private var encoderFile: GGUFFile?, encoder: QwenTextEncoder?
+ /// **Prompt+ hands its encoder to the next image**: Prompt+ writes the prompt the next image encodes, on the same file,
+ /// so when the encoder is not kept (`keepEncoder`), the one text generation opened stays open until that image's
+ /// encode, which then drops it. Its pages are the file's, which macOS can reclaim. M2 Pro 16 GB, 2026-10-25: opening
+ /// the encoder and its first pass cost ~2 s of a 12-token prefill of 2.2–2.5 s (0.46 s once warm).
+ private var handedOver: QwenTextEncoder?
private var ditFile: GGUFFile?, dit: QwenDiT?
/// Checked before each step and after each DiT layer: true stops the image (`QwenDiT.Cancelled`).
public var cancelled: () -> Bool = { false }
@@ -100,8 +105,8 @@ public final class QwenPipeline {
defer { encoder!.release() }
return try encoder!.conditioning(prompt: prompt)
}
- let file = try GGUFFile(path: encoderPath)
- let encoder = try QwenTextEncoder(file: file, context: context)
+ let encoder = try handedOver ?? QwenTextEncoder(file: try GGUFFile(path: encoderPath), context: context)
+ handedOver = nil
defer { encoder.release() }
return try encoder.conditioning(prompt: prompt)
}()
@@ -164,6 +169,30 @@ public final class QwenPipeline {
return image
}
+ /// **Prompt+**: the text encoder's LLM writes the assistant's reply to `prompt` (`QwenTextGenerator`), on the same
+ /// file and with the same keeping rules as the image's encoder.
+ public func generateText(system: String, prompt: String, maxTokens: Int, sampling: TextSampler.Parameters, seed: Int64,
+ onPiece: (String) -> Bool) throws -> QwenTextGenerator.Result {
+ let started = DispatchTime.now().uptimeNanoseconds
+ let encoder: QwenTextEncoder
+ if keepModels && keepEncoder {
+ if self.encoder == nil {
+ let file = try GGUFFile(path: encoderPath)
+ (encoderFile, self.encoder) = (file, try QwenTextEncoder(file: file, context: context))
+ }
+ encoder = self.encoder!
+ } else {
+ encoder = try handedOver ?? QwenTextEncoder(file: try GGUFFile(path: encoderPath), context: context)
+ handedOver = encoder // for the image that follows
+ }
+ defer { encoder.release() }
+ var result = try QwenTextGenerator(encoder: encoder).generate(system: system, prompt: prompt, maxTokens: maxTokens,
+ sampling: sampling, seed: seed, onPiece: onPiece)
+ // With opening the file and reading its tokenizer.
+ result.prepare = Double(DispatchTime.now().uptimeNanoseconds - started) * 1e-9 - result.prefill - result.decode
+ return result
+ }
+
// MARK: - latents
/// `[64, h, w]` → tokens `[h·w, 64]` (`_pack_latents`: a spatial flatten, then channels last).
diff --git a/engine-mac/Sources/KokonEngine/QwenTextEncoder.swift b/engine-mac/Sources/KokonEngine/QwenTextEncoder.swift
index c31b62d..fae6d4a 100644
--- a/engine-mac/Sources/KokonEngine/QwenTextEncoder.swift
+++ b/engine-mac/Sources/KokonEngine/QwenTextEncoder.swift
@@ -75,8 +75,9 @@ public final class QwenTextEncoder {
}
private var nextReserve = 0
- /// The last decoder layer's output for every token, `[ids.count, 4096]`.
- public func forward(ids: [Int], probe: Probe? = nil) throws -> [Float] {
+ /// The last decoder layer's output for every token, `[ids.count, 4096]`. `cache`: each layer's K (after its norm
+ /// and RoPE) and V are kept there too, for text generation (`QwenTextGenerator`).
+ public func forward(ids: [Int], probe: Probe? = nil, cache: KVCache? = nil) throws -> [Float] {
let d = QwenTextEncoder.dim, t = ids.count, kvWidth = QwenTextEncoder.kvHeads * QwenTextEncoder.headDim
let started = DispatchTime.now().uptimeNanoseconds
timings = [:]
@@ -120,7 +121,7 @@ public final class QwenTextEncoder {
var commands = try begin()
for layer in 0..<QwenTextEncoder.layers {
- try encodeLayer(layer, rows: t, b, commands: commands)
+ try encodeLayer(layer, rows: t, b, commands: commands, cache: cache)
if let probe, [0, 1, 17, QwenTextEncoder.layers - 1].contains(layer) {
try MetalContext.finish(commands, "text encoder")
probe("layer\(layer)_out", arena.pointer(b.x), t * d)
@@ -149,7 +150,7 @@ public final class QwenTextEncoder {
return out
}
- func encodeLayer(_ layer: Int, rows t: Int, _ b: Buffers, commands: MTLCommandBuffer) throws {
+ func encodeLayer(_ layer: Int, rows t: Int, _ b: Buffers, commands: MTLCommandBuffer, cache: KVCache? = nil) throws {
let d = QwenTextEncoder.dim, block = "blk.\(layer)."
let norms = layer * 4 * d
try rmsNorm(b, b.x, into: b.normed, rows: t, weight: norms, commands: commands)
@@ -159,6 +160,8 @@ public final class QwenTextEncoder {
try headNormRope(b, b.q, heads: QwenTextEncoder.heads, weight: norms + 2 * d, rows: t, commands: commands)
try headNormRope(b, b.k, heads: QwenTextEncoder.kvHeads, weight: norms + 2 * d + QwenTextEncoder.headDim,
rows: t, commands: commands)
+ try cache?.store(layer: layer, k: (b.arena.buffer, b.k.offset), v: (b.arena.buffer, b.v.offset), rows: t, position: 0,
+ context: context, commands: commands)
try repeatKV(b, b.k, into: b.kx, rows: t, commands: commands)
try repeatKV(b, b.v, into: b.vx, rows: t, commands: commands)
try attend(b, rows: t, commands: commands)
diff --git a/engine-mac/Sources/KokonEngine/QwenTextGeneration.swift b/engine-mac/Sources/KokonEngine/QwenTextGeneration.swift
new file mode 100644
index 0000000..d5326c7
--- /dev/null
+++ b/engine-mac/Sources/KokonEngine/QwenTextGeneration.swift
@@ -0,0 +1,489 @@
+import Foundation
+import Metal
+
+/// **Text generation with the text encoder's LLM** (Prompt+): Qwen3-VL-8B's text decoder writes the prompt that the
+/// image model gets, as Kokon's sd.cpp does (`src/pipeline/text.cpp`, `POST /sdcpp/v1/text_gen`), on the same file:
+///
+/// chat template ─ tokenizer ─ prefill: QwenTextEncoder.forward on every token, each layer's K and V into the cache
+/// then per token: embedding row ─ 36 × { RMS ─ q, k, v ─ norms, RoPE at its position ─ into the cache ─ attention on
+/// the cache ─ o ─ RMS ─ gate, up ─ SwiGLU ─ down } ─ final RMSNorm ─ LM head ─ sampler ─ next token
+///
+/// The decode reads every weight once per token, so its products read the Q4_K / Q6_K blocks where they lie in the
+/// mapped file (`Kernels/decode.metal`), with no unpacking. Text only: images (edit Prompt+, Compose labels) need the
+/// vision tower.
+public final class QwenTextGenerator {
+ /// sd.cpp's limit on prompt + generated tokens (`TEXT_GEN_MAX_CONTEXT`).
+ public static let maxContext = 16384
+ public static let vocabulary = 151_936
+
+ public enum Finish: String { case stop, length, cancelled }
+ public struct Result {
+ public var text = ""
+ public var finish = Finish.stop
+ public var seed: Int64 = 0
+ public var promptTokens = 0, generatedTokens = 0
+ /// Seconds: tokenizing and buffers, prefill (to the first token), decode (first token to last).
+ public var prepare = 0.0, prefill = 0.0, decode = 0.0
+ /// Every token sampled, the stop token included.
+ public var tokens: [Int] = []
+ }
+
+ let encoder: QwenTextEncoder
+ let context: MetalContext
+ var file: GGUFFile { encoder.file }
+
+ public init(encoder: QwenTextEncoder) {
+ self.encoder = encoder
+ context = encoder.context
+ }
+
+ /// sd.cpp's chat template for text generation without images: the system turn when `system` is not empty.
+ public static func template(system: String, prompt: String) -> String {
+ (system.isEmpty ? "" : "<|im_start|>system\n\(system)<|im_end|>\n") + "<|im_start|>user\n\(prompt)<|im_end|>\n<|im_start|>assistant\n"
+ }
+
+ /// Writes the assistant's reply to `prompt`. `onPiece` gets the text as it is written (whole UTF-8 characters);
+ /// returning false stops it (`cancelled`). `logits`: each step's logits, for the checks.
+ public func generate(system: String, prompt: String, maxTokens: Int, sampling: TextSampler.Parameters, seed: Int64,
+ onPiece: (String) -> Bool = { _ in true },
+ logits: ((Int, UnsafePointer<Float>) -> Void)? = nil) throws -> Result {
+ var result = Result()
+ let started = DispatchTime.now().uptimeNanoseconds
+ func since(_ t: UInt64) -> Double { Double(DispatchTime.now().uptimeNanoseconds - t) * 1e-9 }
+ let tokenizer = encoder.tokenizer
+ guard let imEnd = tokenizer.id(of: "<|im_end|>"), let endOfText = tokenizer.id(of: "<|endoftext|>") else {
+ throw GGUFFile.Failure.unsupported("the tokenizer has no <|im_end|> or <|endoftext|>")
+ }
+ let stops: Set<Int> = [imEnd, endOfText]
+ let ids = tokenizer.encode(QwenTextGenerator.template(system: system, prompt: prompt))
+ let capacity = ids.count + maxTokens
+ guard capacity <= QwenTextGenerator.maxContext else {
+ throw GGUFFile.Failure.unsupported("prompt (\(ids.count) tokens) + max_tokens (\(maxTokens)) exceeds the \(QwenTextGenerator.maxContext)-token limit")
+ }
+ result.promptTokens = ids.count
+ result.seed = seed >= 0 ? seed : Int64.random(in: 0...0x7fff_ffff)
+ var sampler = TextSampler(sampling, seed: UInt64(bitPattern: result.seed))
+ let cache = try KVCache(context: context, layers: QwenTextEncoder.layers, kvHeads: QwenTextEncoder.kvHeads, capacity: capacity)
+ let step = try Decoder(generator: self, cache: cache)
+ result.prepare = since(started)
+
+ // Prefill: the prompt's K and V into the cache, the last row's hidden state to the head.
+ var t = DispatchTime.now().uptimeNanoseconds
+ let hidden = try encoder.forward(ids: ids, cache: cache)
+ var next = try step.head(hidden: Array(hidden[((ids.count - 1) * QwenTextEncoder.dim)...]))
+ var stream = TokenStreamDecoder(tokenizer: tokenizer)
+ while true {
+ logits?(result.tokens.count, next)
+ let token = sampler.sample(next, count: QwenTextGenerator.vocabulary)
+ result.tokens.append(token)
+ if result.tokens.count == 1 { result.prefill = since(t); t = DispatchTime.now().uptimeNanoseconds }
+ if stops.contains(token) { result.finish = .stop; break }
+ let piece = stream.push(token)
+ if !piece.isEmpty {
+ result.text += piece
+ if !onPiece(piece) { result.finish = .cancelled; break }
+ }
+ if result.tokens.count == maxTokens { result.finish = .length; break }
+ next = try step.token(token, position: ids.count + result.tokens.count - 1)
+ }
+ if result.finish != .cancelled {
+ let rest = stream.flush()
+ if !rest.isEmpty {
+ result.text += rest
+ if !onPiece(rest) { result.finish = .cancelled }
+ }
+ }
+ result.decode = since(t)
+ result.generatedTokens = result.tokens.count - (result.finish == .stop ? 1 : 0)
+ context.release(file)
+ return result
+ }
+
+ /// One token through the decoder layers and the head, on the GPU; the buffers live as long as one reply.
+ final class Decoder {
+ let generator: QwenTextGenerator
+ let cache: KVCache
+ let context: MetalContext
+ let file: GGUFFile
+ /// [x | normed | q | k | v | attended | o | gate | up] fp32, the layers' norm weights, the RoPE table, the logits.
+ let state: MTLBuffer, norms: MTLBuffer, rope: MTLBuffer, logits: MTLBuffer
+ let offsets: [String: Int]
+ let embedding: GGUFTensor
+ /// The GPU's seconds for the last token.
+ var lastGPU = 0.0
+
+ init(generator: QwenTextGenerator, cache: KVCache) throws {
+ self.generator = generator
+ self.cache = cache
+ context = generator.context
+ file = generator.file
+ let d = QwenTextEncoder.dim, kv = QwenTextEncoder.kvHeads * QwenTextEncoder.headDim, h = QwenTextEncoder.hidden
+ var offsets: [String: Int] = [:], at = 0
+ for (name, floats) in [("x", d), ("normed", d), ("q", d), ("k", kv), ("v", kv), ("attended", d), ("o", d),
+ ("gate", h), ("up", h)] {
+ offsets[name] = at
+ at += floats * 4
+ }
+ self.offsets = offsets
+ state = try context.scratch(at, label: "decode state")
+ // Per layer: input norm, post-attention norm, q norm (128), k norm (128); then the final norm.
+ norms = try context.scratch((QwenTextEncoder.layers * (2 * d + 256) + d) * 4, label: "decode norms")
+ let n = norms.contents().assumingMemoryBound(to: Float.self)
+ for layer in 0..<QwenTextEncoder.layers {
+ for (i, (name, count)) in [("attn_norm", d), ("ffn_norm", d), ("attn_q_norm", 128), ("attn_k_norm", 128)].enumerated() {
+ let w = try Dequant.array(file, "blk.\(layer).\(name).weight")
+ (n + layer * (2 * d + 256) + [0, d, 2 * d, 2 * d + 128][i]).update(from: w, count: count)
+ }
+ }
+ (n + QwenTextEncoder.layers * (2 * d + 256)).update(from: try Dequant.array(file, "output_norm.weight"), count: d)
+ let table = QwenTextEncoder.ropeTable(rows: cache.capacity)
+ rope = try context.scratch(table.count * 4, label: "decode rope")
+ rope.contents().copyMemory(from: table, byteCount: table.count * 4)
+ logits = try context.scratch(QwenTextGenerator.vocabulary * 4, label: "logits")
+ embedding = try file.tensor("token_embd.weight")
+ }
+
+ func pointer(_ name: String) -> UnsafeMutablePointer<Float> {
+ (state.contents() + offsets[name]!).assumingMemoryBound(to: Float.self)
+ }
+
+ /// The final norm and the LM head on `hidden` (one row): the logits.
+ func head(hidden: [Float]) throws -> UnsafePointer<Float> {
+ pointer("x").update(from: hidden, count: QwenTextEncoder.dim)
+ let commands = try generator.encoder.begin()
+ try encodeHead(commands)
+ try MetalContext.finish(commands, "text generation head")
+ return UnsafePointer(logits.contents().assumingMemoryBound(to: Float.self))
+ }
+
+ /// `token` at `position`: its K and V join the cache; the logits of the next one.
+ func token(_ token: Int, position: Int) throws -> UnsafePointer<Float> {
+ let d = QwenTextEncoder.dim
+ guard let type = embedding.type, token >= 0, token < embedding.rows else { throw GGUFFile.Failure.unsupported("token id \(token)") }
+ Dequant.row(file.data(embedding) + token * type.bytes(count: d), type: type, count: d, into: pointer("x"))
+ let commands = try generator.encoder.begin()
+ for layer in 0..<QwenTextEncoder.layers { try encodeLayer(layer, position: position, commands) }
+ try encodeHead(commands)
+ lastGPU = try MetalContext.finish(commands, "text generation step")
+ return UnsafePointer(logits.contents().assumingMemoryBound(to: Float.self))
+ }
+
+ func encodeLayer(_ layer: Int, position: Int, _ commands: MTLCommandBuffer) throws {
+ let d = QwenTextEncoder.dim, block = "blk.\(layer).", norm = layer * (2 * d + 256) * 4
+ try rmsNorm("x", into: "normed", weight: norm, commands)
+ try matvec(block + "attn_q.weight", "normed", "q", commands)
+ try matvec(block + "attn_k.weight", "normed", "k", commands)
+ try matvec(block + "attn_v.weight", "normed", "v", commands)
+ for (name, heads, weight) in [("q", QwenTextEncoder.heads, norm + 2 * d * 4), ("k", QwenTextEncoder.kvHeads, norm + (2 * d + 128) * 4)] {
+ try kernel("head_rmsnorm_rope_neox", commands) { e in
+ e.setBuffer(state, offset: offsets[name]!, index: 0)
+ e.setBuffer(norms, offset: weight, index: 1)
+ e.setBuffer(rope, offset: position * 128 * 4, index: 2)
+ var params = QwenDiT.HeadParams(heads: UInt32(heads), rows: 1, eps: QwenTextEncoder.eps)
+ e.setBytes(&params, length: MemoryLayout<QwenDiT.HeadParams>.stride, index: 3)
+ e.dispatchThreadgroups(MTLSize(width: (heads + 7) / 8, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
+ }
+ }
+ try cache.store(layer: layer, k: (state, offsets["k"]!), v: (state, offsets["v"]!), rows: 1, position: position,
+ context: context, commands: commands)
+ try kernel("decode_attention", commands) { e in
+ e.setBuffer(state, offset: offsets["q"]!, index: 0)
+ e.setBuffer(cache.buffer, offset: cache.offset(layer: layer, value: false), index: 1)
+ e.setBuffer(cache.buffer, offset: cache.offset(layer: layer, value: true), index: 2)
+ e.setBuffer(state, offset: offsets["attended"]!, index: 3)
+ var params = DecodeAttentionParams(keys: UInt32(position + 1), capacity: UInt32(cache.capacity),
+ groups: UInt32(QwenTextEncoder.heads / QwenTextEncoder.kvHeads),
+ scale: 1 / Float(QwenTextEncoder.headDim).squareRoot())
+ e.setBytes(&params, length: MemoryLayout<DecodeAttentionParams>.stride, index: 4)
+ e.dispatchThreadgroups(MTLSize(width: QwenTextEncoder.heads, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
+ }
+ try matvec(block + "attn_output.weight", "attended", "o", commands)
+ try add("o", count: d, commands)
+ try rmsNorm("x", into: "normed", weight: norm + d * 4, commands)
+ try matvec(block + "ffn_gate.weight", "normed", "gate", commands)
+ try matvec(block + "ffn_up.weight", "normed", "up", commands)
+ try kernel("swiglu", commands) { e in
+ e.setBuffer(state, offset: offsets["gate"]!, index: 0)
+ e.setBuffer(state, offset: offsets["up"]!, index: 1)
+ var params = QwenDiT.SwigluParams(columns: UInt32(QwenTextEncoder.hidden), half: QwenDiT.fullOnly(1))
+ e.setBytes(&params, length: MemoryLayout<QwenDiT.SwigluParams>.stride, index: 2)
+ e.setBuffer(state, offset: offsets["gate"]!, index: 3)
+ e.dispatchThreads(MTLSize(width: QwenTextEncoder.hidden, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
+ }
+ try matvec(block + "ffn_down.weight", "gate", "o", commands)
+ try add("o", count: d, commands)
+ }
+
+ func encodeHead(_ commands: MTLCommandBuffer) throws {
+ try rmsNorm("x", into: "normed", weight: QwenTextEncoder.layers * (2 * QwenTextEncoder.dim + 256) * 4, commands)
+ let tensor = try file.tensor("output.weight")
+ try matvec(tensor, x: (state, offsets["normed"]!), y: (logits, 0), commands)
+ }
+
+ func kernel(_ name: String, _ commands: MTLCommandBuffer, _ body: (MTLComputeCommandEncoder) throws -> Void) throws {
+ try generator.encoder.compute(name, commands: commands, body)
+ }
+
+ func rmsNorm(_ input: String, into out: String, weight: Int, _ commands: MTLCommandBuffer) throws {
+ try kernel("rmsnorm_rows", commands) { e in
+ e.setBuffer(state, offset: offsets[input]!, index: 0)
+ e.setBuffer(state, offset: offsets[out]!, index: 1)
+ e.setBuffer(norms, offset: weight, index: 2)
+ var params = QwenDiT.RowParams(columns: UInt32(QwenTextEncoder.dim), targetStart: 0, eps: QwenTextEncoder.eps)
+ e.setBytes(&params, length: MemoryLayout<QwenDiT.RowParams>.stride, index: 3)
+ e.dispatchThreadgroups(MTLSize(width: 1, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
+ }
+ }
+
+ func add(_ y: String, count: Int, _ commands: MTLCommandBuffer) throws {
+ try kernel("add_inplace", commands) { e in
+ e.setBuffer(state, offset: offsets["x"]!, index: 0)
+ e.setBuffer(state, offset: offsets[y]!, index: 1)
+ var c = UInt32(count)
+ e.setBytes(&c, length: 4, index: 2)
+ e.dispatchThreads(MTLSize(width: count, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 256, height: 1, depth: 1))
+ }
+ }
+
+ func matvec(_ name: String, _ x: String, _ y: String, _ commands: MTLCommandBuffer) throws {
+ try matvec(try file.tensor(name), x: (state, offsets[x]!), y: (state, offsets[y]!), commands)
+ }
+
+ /// `y = W · x` on the weight's blocks in the mapped file.
+ func matvec(_ tensor: GGUFTensor, x: (MTLBuffer, Int), y: (MTLBuffer, Int), _ commands: MTLCommandBuffer) throws {
+ guard let type = tensor.type, type == .q4_K || type == .q6_K else {
+ throw GGUFFile.Failure.unsupported("\(tensor.name): \(tensor.type?.name ?? "?"), the decode reads Q4_K and Q6_K")
+ }
+ let (source, offset) = try context.buffer(for: tensor, in: file)
+ try kernel(type == .q4_K ? "matvec_q4_K" : "matvec_q6_K", commands) { e in
+ e.setBuffer(source, offset: offset, index: 0)
+ e.setBuffer(x.0, offset: x.1, index: 1)
+ e.setBuffer(y.0, offset: y.1, index: 2)
+ var params = MatvecParams(rows: UInt32(tensor.rows), blocksPerRow: UInt32(tensor.columns / 256),
+ rowBytes: UInt32(type.bytes(count: tensor.columns)))
+ e.setBytes(&params, length: MemoryLayout<MatvecParams>.stride, index: 3)
+ let rows = QwenTextGenerator.matvecRows(type)
+ e.dispatchThreadgroups(MTLSize(width: (tensor.rows + rows - 1) / rows, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 128, height: 1, depth: 1))
+ }
+ }
+ }
+
+ struct MatvecParams { var rows, blocksPerRow, rowBytes: UInt32 }
+ /// Rows per threadgroup of `matvec_*` (4 SIMD groups of MATVEC_Q4_ROWS or MATVEC_ROWS).
+ static func matvecRows(_ type: QuantType) -> Int { type == .q4_K ? 16 : 8 }
+ struct DecodeAttentionParams { var keys, capacity, groups: UInt32; var scale: Float }
+}
+
+/// **The K/V cache** of text generation: per layer, K and V of every position so far, fp16 `[kvHeads, capacity, 128]`
+/// each (144 KB a position for Qwen3-VL-8B, as sd.cpp's).
+public final class KVCache {
+ public let layers: Int, kvHeads: Int, capacity: Int
+ let buffer: MTLBuffer
+
+ init(context: MetalContext, layers: Int, kvHeads: Int, capacity: Int) throws {
+ (self.layers, self.kvHeads, self.capacity) = (layers, kvHeads, capacity)
+ buffer = try context.scratch(layers * 2 * kvHeads * capacity * 128 * 2, label: "K/V cache")
+ }
+
+ func offset(layer: Int, value: Bool) -> Int { ((layer * 2 + (value ? 1 : 0)) * kvHeads * capacity * 128) * 2 }
+
+ struct Params { var rows, kvHeads, capacity, position: UInt32 }
+
+ /// Encodes K and V of `rows` positions from `position` (rows `[rows, kvHeads · 128]` fp32) into layer `layer`.
+ func store(layer: Int, k: (MTLBuffer, Int), v: (MTLBuffer, Int), rows: Int, position: Int, context: MetalContext,
+ commands: MTLCommandBuffer) throws {
+ guard position + rows <= capacity else { throw MetalContext.Failure.buffer("K/V cache: position \(position + rows) of \(capacity)") }
+ let e = commands.makeComputeCommandEncoder()!
+ e.label = "kv_store"
+ e.setComputePipelineState(try context.pipeline("kv_store"))
+ e.setBuffer(k.0, offset: k.1, index: 0)
+ e.setBuffer(v.0, offset: v.1, index: 1)
+ e.setBuffer(buffer, offset: offset(layer: layer, value: false), index: 2)
+ e.setBuffer(buffer, offset: offset(layer: layer, value: true), index: 3)
+ var params = Params(rows: UInt32(rows), kvHeads: UInt32(kvHeads), capacity: UInt32(capacity), position: UInt32(position))
+ e.setBytes(&params, length: MemoryLayout<Params>.stride, index: 4)
+ e.dispatchThreads(MTLSize(width: kvHeads * 128, height: rows, depth: 1), threadsPerThreadgroup: MTLSize(width: 128, height: 1, depth: 1))
+ e.endEncoding()
+ }
+}
+
+/// **The next-token sampler**, sd.cpp's (`src/runtime/text_sampler.cpp`) step for step so a seed writes the same text:
+/// presence penalty (generated tokens only), then top-k, temperature, top-p, one draw from mt19937_64; temperature 0 is
+/// greedy. Ties keep the lower token id first.
+public struct TextSampler {
+ public struct Parameters: Sendable {
+ public var temperature: Float, topP: Float, topK: Int, presencePenalty: Float
+ public init(temperature: Float, topP: Float, topK: Int, presencePenalty: Float) {
+ (self.temperature, self.topP, self.topK, self.presencePenalty) = (temperature, topP, topK, presencePenalty)
+ }
+ public static let greedy = Parameters(temperature: 0, topP: 1, topK: 0, presencePenalty: 0)
+ }
+
+ let parameters: Parameters
+ var random: MersenneTwister64
+ /// Tokens generated so far, by id (dense, as sd.cpp's: the penalty is looked up for every token of the vocabulary).
+ var seen: [Bool] = []
+
+ public init(_ parameters: Parameters, seed: UInt64) {
+ self.parameters = parameters
+ random = MersenneTwister64(seed: seed)
+ }
+
+ var greedy: Bool { parameters.temperature <= 0 }
+
+ /// A double in [0, 1) from 53 random bits.
+ mutating func uniform() -> Double { Double(random.next() >> 11) * (1.0 / 9_007_199_254_740_992.0) }
+
+ func penalized(_ logits: UnsafePointer<Float>, _ token: Int) -> Float {
+ parameters.presencePenalty != 0 && token < seen.count && seen[token] ? logits[token] - parameters.presencePenalty : logits[token]
+ }
+
+ /// Probabilities after the penalty, top-k, temperature and top-p, by decreasing probability.
+ func candidates(_ logits: UnsafePointer<Float>, count: Int) -> [(token: Int, p: Double)] {
+ func before(_ a: (Int, Float), _ b: (Int, Float)) -> Bool { a.1 > b.1 || (a.1 == b.1 && a.0 < b.0) }
+ var kept: [(Int, Float)]
+ if parameters.topK > 0 && parameters.topK < count {
+ // The k best in one pass, kept sorted (k is small, the vocabulary ~150k).
+ let k = parameters.topK
+ kept = []
+ kept.reserveCapacity(k + 1)
+ for i in 0..<count {
+ let c = (i, penalized(logits, i))
+ if kept.count == k, !before(c, kept[k - 1]) { continue }
+ var at = kept.count
+ while at > 0 && before(c, kept[at - 1]) { at -= 1 }
+ kept.insert(c, at: at)
+ if kept.count > k { kept.removeLast() }
+ }
+ } else {
+ kept = (0..<count).map { ($0, penalized(logits, $0)) }
+ kept.sort(by: before)
+ }
+ let inverse = parameters.temperature > 0 ? 1 / Double(parameters.temperature) : 1
+ let top = Double(kept[0].1)
+ var probs = kept.map { ($0.0, exp((Double($0.1) - top) * inverse)) }
+ let sum = probs.reduce(0) { $0 + $1.1 }
+ probs = probs.map { ($0.0, $0.1 / sum) }
+ if parameters.topP < 1 {
+ var cumulative = 0.0, keep = probs.count
+ for (i, p) in probs.enumerated() {
+ cumulative += p.1
+ if cumulative >= Double(parameters.topP) { keep = i + 1; break }
+ }
+ probs.removeLast(probs.count - keep)
+ let total = probs.reduce(0) { $0 + $1.1 }
+ probs = probs.map { ($0.0, $0.1 / total) }
+ }
+ return probs.map { (token: $0.0, p: $0.1) }
+ }
+
+ /// Picks the next token and remembers it for the presence penalty.
+ public mutating func sample(_ logits: UnsafePointer<Float>, count: Int) -> Int {
+ var token = 0
+ if greedy {
+ var best = penalized(logits, 0)
+ for i in 1..<count {
+ let l = penalized(logits, i)
+ if l > best { best = l; token = i }
+ }
+ } else {
+ let probs = candidates(logits, count: count)
+ let u = uniform()
+ var cumulative = 0.0
+ token = probs.last!.token
+ for p in probs {
+ cumulative += p.p
+ if u < cumulative { token = p.token; break }
+ }
+ }
+ if token >= seen.count { seen += [Bool](repeating: false, count: token + 1 - seen.count) }
+ seen[token] = true
+ return token
+ }
+}
+
+/// `std::mt19937_64`, whose output the C++ standard fixes: the same seed draws the same numbers as sd.cpp.
+public struct MersenneTwister64 {
+ private var state = [UInt64](repeating: 0, count: 312)
+ private var index = 312
+
+ public init(seed: UInt64) {
+ state[0] = seed
+ for i in 1..<312 { state[i] = 6_364_136_223_846_793_005 &* (state[i - 1] ^ (state[i - 1] >> 62)) &+ UInt64(i) }
+ }
+
+ public mutating func next() -> UInt64 {
+ if index >= 312 {
+ for i in 0..<312 {
+ let y = (state[i] & 0xFFFF_FFFF_8000_0000) | (state[(i + 1) % 312] & 0x7FFF_FFFF)
+ var v = state[(i + 156) % 312] ^ (y >> 1)
+ if y & 1 != 0 { v ^= 0xB502_6F5A_A966_19E9 }
+ state[i] = v
+ }
+ index = 0
+ }
+ var x = state[index]
+ index += 1
+ x ^= (x >> 29) & 0x5555_5555_5555_5555
+ x ^= (x << 17) & 0x71D6_7FFF_EDA6_0000
+ x ^= (x << 37) & 0xFFF7_EEE0_0000_0000
+ x ^= x >> 43
+ return x
+ }
+}
+
+/// Tokens to text as they come, sd.cpp's `TokenStreamDecoder`: special tokens skipped, bytes held back until they
+/// complete a UTF-8 character, invalid bytes as U+FFFD.
+public struct TokenStreamDecoder {
+ let tokenizer: QwenTokenizer
+ var pending: [UInt8] = []
+
+ public init(tokenizer: QwenTokenizer) { self.tokenizer = tokenizer }
+
+ public mutating func push(_ token: Int) -> String {
+ guard let bytes = tokenizer.bytes(of: token) else { return "" } // an added (special) token
+ pending += bytes
+ return take(end: false)
+ }
+
+ public mutating func flush() -> String { take(end: true) }
+
+ mutating func take(end: Bool) -> String {
+ var out: [UInt8] = []
+ var at = 0
+ while at < pending.count {
+ let length = TokenStreamDecoder.sequenceLength(pending, at)
+ if length > 0 {
+ out += pending[at..<(at + length)]
+ at += length
+ } else if length == 0 && !end {
+ break
+ } else {
+ out += [0xEF, 0xBF, 0xBD]
+ at = length == 0 ? pending.count : at - length
+ }
+ }
+ pending.removeFirst(at)
+ return String(decoding: out, as: UTF8.self)
+ }
+
+ /// The length of the valid UTF-8 sequence at `at`; 0 if the bytes end inside a valid prefix; −n if its first n bytes
+ /// are an invalid prefix (one U+FFFD, as Python does). sd.cpp's `utf8_sequence_length`.
+ static func sequenceLength(_ s: [UInt8], _ at: Int) -> Int {
+ let lead = s[at]
+ if lead < 0x80 { return 1 }
+ var (length, lo, hi): (Int, UInt8, UInt8) = (0, 0x80, 0xBF)
+ switch lead {
+ case 0xC2...0xDF: length = 2
+ case 0xE0...0xEF: (length, lo, hi) = (3, lead == 0xE0 ? 0xA0 : 0x80, lead == 0xED ? 0x9F : 0xBF)
+ case 0xF0...0xF4: (length, lo, hi) = (4, lead == 0xF0 ? 0x90 : 0x80, lead == 0xF4 ? 0x8F : 0xBF)
+ default: return -1
+ }
+ for i in 1..<length {
+ if at + i >= s.count { return 0 }
+ if s[at + i] < lo || s[at + i] > hi { return -i }
+ (lo, hi) = (0x80, 0xBF)
+ }
+ return length
+ }
+}
diff --git a/engine-mac/Sources/KokonEngine/QwenTokenizer.swift b/engine-mac/Sources/KokonEngine/QwenTokenizer.swift
index 85dfe93..6a84c32 100644
--- a/engine-mac/Sources/KokonEngine/QwenTokenizer.swift
+++ b/engine-mac/Sources/KokonEngine/QwenTokenizer.swift
@@ -121,15 +121,30 @@ public final class QwenTokenizer: @unchecked Sendable {
return result
}
+ static let byteValues: [Character: UInt8] = {
+ var reverse: [Character: UInt8] = [:]
+ for (b, c) in byteCharacters.enumerated() { reverse[c] = UInt8(b) }
+ return reverse
+ }()
+ private lazy var addedIDs = Set(added.map(\.id))
+
/// Token ids back to text (byte-level characters to bytes); for messages and tests.
public func decode(_ ids: [Int]) -> String {
- var reverse: [Character: UInt8] = [:]
- for (b, c) in QwenTokenizer.byteCharacters.enumerated() { reverse[c] = UInt8(b) }
var bytes: [UInt8] = []
for id in ids where id >= 0 && id < tokens.count {
- if added.contains(where: { $0.id == id }) { bytes += Array(tokens[id].utf8); continue }
- for c in tokens[id] { if let b = reverse[c] { bytes.append(b) } }
+ bytes += self.bytes(of: id) ?? Array(tokens[id].utf8)
}
return String(decoding: bytes, as: UTF8.self)
}
+
+ /// A token's bytes, nil for an added token (`<|im_end|>`, `<think>`, …: the ones text generation leaves out, as
+ /// sd.cpp's Qwen2 special tokens).
+ public func bytes(of id: Int) -> [UInt8]? {
+ guard id >= 0, id < tokens.count else { return [] }
+ lock.lock()
+ let isAdded = addedIDs.contains(id)
+ lock.unlock()
+ if isAdded { return nil }
+ return tokens[id].compactMap { QwenTokenizer.byteValues[$0] }
+ }
}
diff --git a/engine-mac/Sources/KokonEngine/TextChecks.swift b/engine-mac/Sources/KokonEngine/TextChecks.swift
index d1a6a4e..95c7ce3 100644
--- a/engine-mac/Sources/KokonEngine/TextChecks.swift
+++ b/engine-mac/Sources/KokonEngine/TextChecks.swift
@@ -134,3 +134,93 @@ extension Checks {
if differ > 0 || offset != want.count { throw GGUFFile.Failure.unsupported("noise differs from sd.cpp's") }
}
}
+
+extension Checks {
+ /// Text generation (Prompt+) against `tools/reference_text_generation.py`: the engine writes greedily (its tokens to
+ /// `<dir>/engine.json`, its logits beside); once the reference has run on those tokens, each step's logits are
+ /// compared with transformers' (relative error, the greedy choice, the margin between the two best) and the
+ /// engine's speed is reported. `system` / `prompt` from `<dir>/engine.json` when it exists.
+ public static func textGeneration(path: String, directory: String, tokens: Int = 48) throws {
+ let dir = URL(fileURLWithPath: directory)
+ try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true)
+ let previous = (try? JSONSerialization.jsonObject(with: Data(contentsOf: dir.appendingPathComponent("engine.json")))) as? [String: Any]
+ let system = previous?["system"] as? String
+ ?? "You write prompts for an image generation model. Expand the user's request into one detailed English paragraph that describes the finished image: subject, setting, composition, lighting and style. Output only the prompt."
+ let prompt = previous?["prompt"] as? String ?? "un gatto rosso che dorme su un davanzale, sera"
+ let file = try GGUFFile(path: path)
+ let context = try MetalContext()
+ let encoder = try QwenTextEncoder(file: file, context: context)
+ let generator = QwenTextGenerator(encoder: encoder)
+ let vocabulary = QwenTextGenerator.vocabulary
+ var logits = [Float](repeating: 0, count: tokens * vocabulary)
+ let result = try generator.generate(system: system, prompt: prompt, maxTokens: tokens, sampling: .greedy, seed: 0, logits: { step, l in
+ logits.withUnsafeMutableBufferPointer { ($0.baseAddress! + step * vocabulary).update(from: l, count: vocabulary) }
+ })
+ let steps = result.tokens.count
+ let promptIDs = encoder.tokenizer.encode(QwenTextGenerator.template(system: system, prompt: prompt))
+ print(String(format: "%d prompt tokens, %d generated (%@): prefill %.2f s, decode %.2f s, %.1f tokens/s",
+ result.promptTokens, result.generatedTokens, result.finish.rawValue, result.prefill, result.decode,
+ Double(steps - 1) / max(result.decode, 1e-9)))
+ print("text: \(result.text)")
+ if let previous, let generated = previous["generated"] as? [Int], generated != result.tokens {
+ print("NOTE: not the tokens of the engine's previous run (\(generated.count) then, \(steps) now): the reference must be run again")
+ }
+ let meta: [String: Any] = ["system": system, "prompt": prompt, "prompt_ids": promptIDs, "generated": result.tokens, "text": result.text]
+ try JSONSerialization.data(withJSONObject: meta, options: [.prettyPrinted, .sortedKeys]).write(to: dir.appendingPathComponent("engine.json"))
+ try logits.prefix(steps * vocabulary).withUnsafeBufferPointer { Data(buffer: $0) }.write(to: dir.appendingPathComponent("engine_logits.f32"))
+
+ guard let want = try? reference(directory, "reference_logits"), want.count == steps * vocabulary,
+ (previous?["generated"] as? [Int]) == result.tokens else {
+ print("no reference for these tokens yet: run tools/reference_text_generation.py on \(directory), then this again")
+ return
+ }
+ var worst = 0.0, agree = 0, ties = 0
+ for s in 0..<steps {
+ let r = logits.withUnsafeBufferPointer { compare($0.baseAddress! + s * vocabulary, Array(want[(s * vocabulary)..<((s + 1) * vocabulary)])) }
+ worst = max(worst, r.relative)
+ let row = want[(s * vocabulary)..<((s + 1) * vocabulary)]
+ let sorted = row.enumerated().sorted { $0.element > $1.element }
+ let (best, second) = (sorted[0], sorted[1])
+ if best.offset == result.tokens[s] { agree += 1 } else if best.element - second.element < 0.05 { ties += 1 }
+ if s < 4 || s == steps - 1 || best.offset != result.tokens[s] {
+ print(String(format: " step %2d rel %.2e max |Δ| %.3g engine %6d reference %6d (margin %.3f)", s, r.relative, r.maxAbs,
+ result.tokens[s], best.offset, best.element - second.element))
+ }
+ }
+ print(String(format: "logits within %.2e relative at every step; the greedy choice agrees at %d of %d (%d near-ties)", worst, agree, steps, ties))
+ }
+}
+
+extension Checks {
+ /// The decode's matrix-vector products on the encoder's real weights, alone: milliseconds and GB/s per shape (the
+ /// decode reads every weight once a token, so these rates bound its speed), then one whole token's GPU time.
+ public static func timeMatvec(path: String) throws {
+ let file = try GGUFFile(path: path)
+ let context = try MetalContext()
+ let encoder = try QwenTextEncoder(file: file, context: context)
+ let generator = QwenTextGenerator(encoder: encoder)
+ let cache = try KVCache(context: context, layers: QwenTextEncoder.layers, kvHeads: QwenTextEncoder.kvHeads, capacity: 1024)
+ let decoder = try QwenTextGenerator.Decoder(generator: generator, cache: cache)
+ let x = try context.scratch(QwenTextEncoder.hidden * 4, label: "x"), y = try context.scratch(QwenTextGenerator.vocabulary * 4, label: "y")
+ for i in 0..<QwenTextEncoder.hidden { x.contents().assumingMemoryBound(to: Float.self)[i] = Float(i % 7) * 0.1 - 0.3 }
+ for name in ["blk.0.attn_q.weight", "blk.0.attn_k.weight", "blk.0.attn_v.weight", "blk.0.ffn_gate.weight", "blk.0.ffn_down.weight", "output.weight"] {
+ let tensor = try file.tensor(name)
+ let runs = 20
+ // Warm (the file's pages), then timed: one command buffer of `runs` products.
+ for timed in [false, true] {
+ let commands = try encoder.begin()
+ for _ in 0..<(timed ? runs : 2) { try decoder.matvec(tensor, x: (x, 0), y: (y, 0), commands) }
+ let seconds = try MetalContext.finish(commands, name) / Double(timed ? runs : 2)
+ if timed {
+ print(String(format: " %-24@ %@ %6d × %5d %.3f ms %.0f GB/s", name as NSString, (tensor.type?.name ?? "?") as NSString,
+ tensor.rows, tensor.columns, seconds * 1e3, Double(tensor.bytes) / seconds / 1e9))
+ }
+ }
+ }
+ for run in 0..<5 {
+ let t = DispatchTime.now().uptimeNanoseconds
+ _ = try decoder.token(1000 + run, position: run)
+ print(String(format: "token %d: %.1f ms (GPU %.1f ms)", run, Double(DispatchTime.now().uptimeNanoseconds - t) * 1e-6, decoder.lastGPU * 1e3))
+ }
+ }
+}
diff --git a/engine-mac/Sources/kokon-engine-mac/main.swift b/engine-mac/Sources/kokon-engine-mac/main.swift
index 1a6a7f6..0bfed05 100644
--- a/engine-mac/Sources/kokon-engine-mac/main.swift
+++ b/engine-mac/Sources/kokon-engine-mac/main.swift
@@ -15,6 +15,8 @@ kokon-engine-mac: Kokon's Apple Silicon engine (in development)
check-generate <encoder.gguf> <dit.gguf> <vae.safetensors> <reference dir> [text reference dir]
text to image against tools/reference_generate.py, from its noise
(and from the text reference's conditioning when given)
+ check-textgen <encoder.gguf> <dir> [tokens] text generation (Prompt+), greedy, against tools/reference_text_generation.py
+ time-matvec <encoder.gguf> the decode's products on the encoder's weights: ms and GB/s
check-steps <dit.gguf> <generate reference dir> <text reference dir>
each step alone against tools/reference_generate.py (its latents in)
check-noise <philox.bin> the seed's noise against sd.cpp's generator (tools/philox_reference.cpp)
@@ -66,6 +68,12 @@ do {
let numbers = arguments.dropFirst(2).compactMap { Int($0) }
try Checks.timeDiT(path: arguments[1], grid: numbers.first ?? 64, text: numbers.dropFirst().first ?? 32,
runs: numbers.dropFirst(2).first ?? 3)
+ case "check-textgen":
+ guard arguments.count >= 3 else { fail(usage) }
+ try Checks.textGeneration(path: arguments[1], directory: arguments[2], tokens: arguments.count > 3 ? Int(arguments[3]) ?? 48 : 48)
+ case "time-matvec":
+ guard arguments.count == 2 else { fail(usage) }
+ try Checks.timeMatvec(path: arguments[1])
case "check-ane":
try Checks.aneSurfaces()
case "check-vae":
diff --git a/engine-mac/Tests/KokonEngineTests/QwenTokenizerTests.swift b/engine-mac/Tests/KokonEngineTests/QwenTokenizerTests.swift
index 6317fb5..83cfce6 100644
--- a/engine-mac/Tests/KokonEngineTests/QwenTokenizerTests.swift
+++ b/engine-mac/Tests/KokonEngineTests/QwenTokenizerTests.swift
@@ -68,4 +68,20 @@ final class QwenTokenizerTests: XCTestCase {
XCTAssertEqual(tokenizer.encode("e\u{301}"), [byte(0xC3), byte(0xA9)])
XCTAssertEqual(tokenizer.decode(tokenizer.encode("the cat é")), "the cat é")
}
+
+ /// Text generation's stream: whole UTF-8 characters only, added tokens left out, invalid bytes as U+FFFD (sd.cpp's
+ /// TokenStreamDecoder).
+ func testStreamDecoderHoldsPartialCharacters() {
+ var stream = TokenStreamDecoder(tokenizer: tokenizer)
+ XCTAssertEqual(stream.push(id("the")), "the")
+ XCTAssertEqual(stream.push(byte(0xC3)), "") // the first byte of é
+ XCTAssertEqual(stream.push(id("<|im_start|>")), "") // an added token: nothing
+ XCTAssertEqual(stream.push(byte(0xA9)), "é")
+ XCTAssertEqual(stream.push(byte(0xFF)), "\u{FFFD}") // never valid
+ XCTAssertEqual(stream.push(byte(0xE2)), "")
+ XCTAssertEqual(stream.push(byte(UInt8(ascii: "a"))), "\u{FFFD}a") // a broken sequence, then the next character
+ XCTAssertEqual(stream.push(byte(0xF0)), "")
+ XCTAssertEqual(stream.flush(), "\u{FFFD}") // unfinished at the end
+ XCTAssertEqual(stream.flush(), "")
+ }
}
diff --git a/engine-mac/Tests/KokonEngineTests/TextGenerationTests.swift b/engine-mac/Tests/KokonEngineTests/TextGenerationTests.swift
new file mode 100644
index 0000000..aeedbe6
--- /dev/null
+++ b/engine-mac/Tests/KokonEngineTests/TextGenerationTests.swift
@@ -0,0 +1,102 @@
+import XCTest
+@testable import KokonEngine
+
+/// Text generation's sampler, which must draw what sd.cpp's draws for a seed (`src/runtime/text_sampler.cpp`).
+final class TextGenerationTests: XCTestCase {
+ /// The C++ standard fixes mt19937_64's output: the 10 000th number from the default seed 5489.
+ func testMersenneTwisterIsTheStandards() {
+ var random = MersenneTwister64(seed: 5489)
+ XCTAssertEqual(random.next(), 14_514_284_786_278_117_030)
+ for _ in 2..<10_000 { _ = random.next() }
+ XCTAssertEqual(random.next(), 9_981_545_732_273_789_042)
+ }
+
+ func testGreedyTakesTheLowestIDOnTiesAndPenalizesSeenTokens() {
+ var sampler = TextSampler(TextSampler.Parameters(temperature: 0, topP: 1, topK: 0, presencePenalty: 1.5), seed: 1)
+ let logits: [Float] = [1, 3, 3, 2]
+ XCTAssertEqual(logits.withUnsafeBufferPointer { sampler.sample($0.baseAddress!, count: 4) }, 1)
+ // 1 was generated: 3 − 1.5 < 3, so 2 wins.
+ XCTAssertEqual(logits.withUnsafeBufferPointer { sampler.sample($0.baseAddress!, count: 4) }, 2)
+ // Both penalized to 1.5: 3 (logit 2) wins.
+ XCTAssertEqual(logits.withUnsafeBufferPointer { sampler.sample($0.baseAddress!, count: 4) }, 3)
+ }
+
+ func testCandidatesTopKTemperatureTopP() {
+ let sampler = TextSampler(TextSampler.Parameters(temperature: 0.5, topP: 0.8, topK: 3, presencePenalty: 0), seed: 1)
+ let logits: [Float] = [0, 2, 1, 2, -1]
+ let c = logits.withUnsafeBufferPointer { sampler.candidates($0.baseAddress!, count: 5) }
+ // Top 3 by logit (ties by id): 1, 3, 2; at T 0.5 their weights are 1, 1, e^-2; 1 + 1 reach 0.8 of the sum.
+ XCTAssertEqual(c.map(\.token), [1, 3])
+ XCTAssertEqual(c[0].p, 0.5, accuracy: 1e-12)
+ let all = TextSampler(TextSampler.Parameters(temperature: 1, topP: 1, topK: 0, presencePenalty: 0), seed: 1)
+ let p = logits.withUnsafeBufferPointer { all.candidates($0.baseAddress!, count: 5) }
+ XCTAssertEqual(p.map(\.token), [1, 3, 2, 0, 4])
+ XCTAssertEqual(p.reduce(0) { $0 + $1.p }, 1, accuracy: 1e-12)
+ XCTAssertEqual(p[2].p / p[0].p, exp(-1), accuracy: 1e-12)
+ }
+
+ /// A draw is the first candidate whose cumulative probability passes a uniform double of 53 bits from the seed.
+ func testDrawFollowsTheSeed() {
+ let parameters = TextSampler.Parameters(temperature: 1, topP: 1, topK: 0, presencePenalty: 0)
+ let logits: [Float] = [0, 0] // 0.5 each: token 0 when u < 0.5
+ for seed: UInt64 in [1, 2, 3, 42] {
+ var sampler = TextSampler(parameters, seed: seed)
+ var random = MersenneTwister64(seed: seed)
+ let u = Double(random.next() >> 11) / 9_007_199_254_740_992.0
+ XCTAssertEqual(logits.withUnsafeBufferPointer { sampler.sample($0.baseAddress!, count: 2) }, u < 0.5 ? 0 : 1)
+ }
+ }
+
+ func testTemplateIsSdcpps() {
+ XCTAssertEqual(QwenTextGenerator.template(system: "S", prompt: "P"),
+ "<|im_start|>system\nS<|im_end|>\n<|im_start|>user\nP<|im_end|>\n<|im_start|>assistant\n")
+ XCTAssertEqual(QwenTextGenerator.template(system: "", prompt: "P"), "<|im_start|>user\nP<|im_end|>\n<|im_start|>assistant\n")
+ }
+}
+
+/// The decode's matrix-vector products on Q4_K and Q6_K blocks against the CPU's unpacking (`Dequant.row`) and a dot
+/// product in double: random blocks with sane scales, 3 blocks a row (ragged across the SIMD group's lanes).
+final class MatvecKernelTests: XCTestCase {
+ func testMatvecMatchesTheUnpackedRows() throws {
+ let context = try MetalContext()
+ var s: UInt64 = 7
+ func next() -> UInt64 { s = s &* 6364136223846793005 &+ 1442695040888963407; return s >> 33 }
+ for (type, kernel) in [(QuantType.q4_K, "matvec_q4_K"), (.q6_K, "matvec_q6_K")] {
+ let rows = 37, columns = 3 * 256, rowBytes = type.bytes(count: columns)
+ var bytes = (0..<(rows * rowBytes)).map { _ in UInt8(truncatingIfNeeded: next()) }
+ // The fp16 scales of every block: small positive values (random bytes may be NaN or huge).
+ for r in 0..<rows {
+ for b in 0..<3 {
+ let block = r * rowBytes + b * type.blockBytes
+ let scaleOffsets = type == .q4_K ? [0, 2] : [208]
+ for o in scaleOffsets {
+ let v = Float16(Float(next() % 1000 + 1) * 1e-5)
+ withUnsafeBytes(of: v.bitPattern.littleEndian) { bytes[block + o] = $0[0]; bytes[block + o + 1] = $0[1] }
+ }
+ }
+ }
+ let x = (0..<columns).map { _ in Float(Int(next() % 2001) - 1000) / 500 }
+ let w = try context.scratch(bytes.count, label: "w"), xb = try context.scratch(columns * 4, label: "x"), y = try context.scratch(rows * 4, label: "y")
+ w.contents().copyMemory(from: bytes, byteCount: bytes.count)
+ xb.contents().copyMemory(from: x, byteCount: columns * 4)
+ let commands = context.queue.makeCommandBuffer()!
+ let e = commands.makeComputeCommandEncoder()!
+ e.setComputePipelineState(try context.pipeline(kernel))
+ e.setBuffer(w, offset: 0, index: 0); e.setBuffer(xb, offset: 0, index: 1); e.setBuffer(y, offset: 0, index: 2)
+ var params = QwenTextGenerator.MatvecParams(rows: UInt32(rows), blocksPerRow: 3, rowBytes: UInt32(rowBytes))
+ e.setBytes(&params, length: MemoryLayout<QwenTextGenerator.MatvecParams>.stride, index: 3)
+ let perGroup = QwenTextGenerator.matvecRows(type)
+ e.dispatchThreadgroups(MTLSize(width: (rows + perGroup - 1) / perGroup, height: 1, depth: 1), threadsPerThreadgroup: MTLSize(width: 128, height: 1, depth: 1))
+ e.endEncoding()
+ try MetalContext.finish(commands, kernel)
+ let got = y.contents().assumingMemoryBound(to: Float.self)
+ var unpacked = [Float](repeating: 0, count: columns)
+ for r in 0..<rows {
+ bytes.withUnsafeBytes { Dequant.row($0.baseAddress! + r * rowBytes, type: type, count: columns, into: &unpacked) }
+ let want = zip(unpacked, x).reduce(0.0) { $0 + Double($1.0) * Double($1.1) }
+ let scale = zip(unpacked, x).reduce(0.0) { $0 + abs(Double($1.0) * Double($1.1)) }
+ XCTAssertEqual(Double(got[r]), want, accuracy: 1e-5 * scale + 1e-9, "\(kernel) row \(r)")
+ }
+ }
+ }
+}
diff --git a/engine-mac/tools/api_check.py b/engine-mac/tools/api_check.py
index 3f09f77..8c1d2aa 100644
--- a/engine-mac/tools/api_check.py
+++ b/engine-mac/tools/api_check.py
@@ -1,10 +1,12 @@
#!/usr/bin/env python3
"""Text to image through a running `kokon-engine-mac serve` (or sd-server) as qls-core sends it: capabilities,
-img_gen, polling jobs/{id}, optionally a cancel. Prints the time of each image and writes the PNGs.
+img_gen, polling jobs/{id}, optionally a cancel; with `--enhance`, Prompt+ first (text_gen with Kokon's built-in
+system prompt, read from crates/qls-core/src/enhancer.rs, and its sampling). Prints the time of each image and writes
+the PNGs.
- python3 -I tools/api_check.py --port 1234 --images 2 --out /tmp/api [--size 1024x1024] [--cancel-after 10]
+ python3 -I tools/api_check.py --port 1234 --images 2 --out /tmp/api [--size 1024x1024] [--cancel-after 10] [--enhance]
"""
-import argparse, base64, json, os, time, urllib.request, urllib.error
+import argparse, base64, json, os, re, time, urllib.request, urllib.error
TURBO_SIGMAS = [1.0, 0.978453, 0.95418, 0.926626, 0.89508, 0.845148, 0.704534, 0.414568, 0.0]
@@ -29,17 +31,30 @@ def main():
p.add_argument("--seed", type=int, default=42)
p.add_argument("--out", default=".")
p.add_argument("--cancel-after", type=float, default=0, help="cancel the last image after this many seconds")
+ p.add_argument("--enhance", action="store_true", help="Prompt+ writes each image's prompt first")
a = p.parse_args()
+ enhancer = os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../crates/qls-core/src/enhancer.rs")
+ system = re.search(r'pub const T2I_SYSTEM_PROMPT: &str = r#"(.*?)"#;', open(enhancer).read(), re.S).group(1) if a.enhance else ""
width, height = (int(v) for v in a.size.split("x"))
status, caps = call(a.port, "GET", "/sdcpp/v1/capabilities", timeout=10)
print(f"capabilities {status}: engine {caps.get('engine', 'sd-server')}, model {caps.get('model', {}).get('name')}")
os.makedirs(a.out, exist_ok=True)
for i in range(a.images):
- body = {"prompt": a.prompt, "negative_prompt": "", "width": width, "height": height, "seed": a.seed + i,
+ started = time.time()
+ prompt = a.prompt
+ if a.enhance:
+ status, done = call(a.port, "POST", "/sdcpp/v1/text_gen", {
+ "prompt": a.prompt, "system_prompt": system, "max_tokens": 1024, "stream": False,
+ "sampling": {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "presence_penalty": 1.5, "seed": a.seed + i}}, timeout=300)
+ if status != 200:
+ print(f"text_gen {status}: {done}")
+ return 1
+ prompt = done["text"]
+ print(f"Prompt+ {time.time() - started:.1f} s, {done['usage']['completion_tokens']} tokens, {done['timings']}: {prompt[:80]}…")
+ body = {"prompt": prompt, "negative_prompt": "", "width": width, "height": height, "seed": a.seed + i,
"batch_count": 1, "sample_params": {"sample_method": "euler", "sample_steps": 8,
"custom_sigmas": TURBO_SIGMAS, "guidance": {"txt_cfg": 1.0}},
"output_format": "png", "embed_image_metadata": False}
- started = time.time()
status, accepted = call(a.port, "POST", "/sdcpp/v1/img_gen", body)
if status != 202:
print(f"img_gen {status}: {accepted}")
diff --git a/engine-mac/tools/reference_text_generation.py b/engine-mac/tools/reference_text_generation.py
new file mode 100644
index 0000000..20f567d
--- /dev/null
+++ b/engine-mac/tools/reference_text_generation.py
@@ -0,0 +1,67 @@
+"""The text generation reference for the Mac engine (Prompt+): transformers' own Qwen3-VL text decoder layers in fp32
+on the CPU, on the GGUF's weights (as `reference_text_encoder.py`), run once on the prompt followed by the tokens the
+engine generated (teacher forcing), with the final RMSNorm and the language model head on every generated position.
+
+ kokon-engine-mac check-textgen <encoder.gguf> <dir> # 1. the engine, greedy: <dir>/engine.json + its logits
+ python -I engine-mac/tools/reference_text_generation.py <encoder.gguf> <processor dir> <text_encoder dir> <dir>
+ kokon-engine-mac check-textgen <encoder.gguf> <dir> # 2. the same tokens again, compared with the reference
+
+The prompt is tokenized by the official processor as sd.cpp's template writes it (system turn, user turn, assistant),
+and must give the engine's ids. Writes `<dir>/reference_logits.f32` ([generated, vocabulary] fp32: the logits that
+predict each generated token) and `<dir>/reference.json` (its own greedy choice at each position).
+"""
+import argparse
+import json
+import os
+import sys
+
+import torch
+from transformers import AutoConfig, AutoProcessor
+from transformers.models.qwen3_vl.modeling_qwen3_vl import Qwen3VLTextRMSNorm
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+from reference_text_encoder import Weights, run # noqa: E402
+
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("gguf")
+ parser.add_argument("processor")
+ parser.add_argument("text_encoder", help="text_encoder/ of the official repository (config.json)")
+ parser.add_argument("dir", help="where check-textgen wrote engine.json")
+ args = parser.parse_args()
+ torch.set_grad_enabled(False)
+ with open(os.path.join(args.dir, "engine.json")) as f:
+ engine = json.load(f)
+ config = AutoConfig.from_pretrained(args.text_encoder).text_config
+ config._attn_implementation = "eager"
+ processor = AutoProcessor.from_pretrained(args.processor)
+
+ # sd.cpp's template (src/pipeline/text.cpp), text only.
+ system, prompt = engine["system"], engine["prompt"]
+ text = (f"<|im_start|>system\n{system}<|im_end|>\n" if system else "") + f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"
+ ids = processor(text=[text], return_tensors="pt").input_ids[0].tolist()
+ if ids != engine["prompt_ids"]:
+ sys.exit(f"prompt ids differ from the engine's: {len(ids)} against {len(engine['prompt_ids'])}")
+ generated = engine["generated"]
+ sequence = ids + generated[:-1] # each position predicts the next token
+
+ weights = Weights(args.gguf)
+ hidden = run(weights, config, sequence)[0]
+ norm = Qwen3VLTextRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+ norm.weight.copy_(weights.get("output_norm.weight"))
+ head = weights.get("output.weight")
+ logits = head @ norm(hidden[len(ids) - 1:]).T # [vocabulary, generated]
+ logits = logits.T.contiguous()
+ logits.numpy().astype("float32").tofile(os.path.join(args.dir, "reference_logits.f32"))
+ greedy = logits.argmax(dim=1).tolist()
+ with open(os.path.join(args.dir, "reference.json"), "w") as f:
+ json.dump({"greedy": greedy, "positions": len(generated), "transformers": __import__("transformers").__version__,
+ "torch": torch.__version__}, f, indent=1)
+ same = sum(int(a == b) for a, b in zip(greedy, generated))
+ print(f"{len(ids)} prompt tokens, {len(generated)} generated; the reference's greedy choice agrees at {same}")
+ print("reference text:", processor.tokenizer.decode(greedy))
+
+
+if __name__ == "__main__":
+ main()