Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

dsv41 cudalab: gfx12 WMMA port recipe of the trellis GEMV: host emulation of the data flow (matches the reference), HIP kernel source, layout probe

Niko1221d251b2c0db6d
diff --git a/tools/dsv41_lab/hip/README.md b/tools/dsv41_lab/hip/README.md
new file mode 100644
index 000000000..e998b1ae2
--- /dev/null
+++ b/tools/dsv41_lab/hip/README.md
@@ -0,0 +1,38 @@
+# HIP port of the trellis GEMV (gfx1201, R9700) - recipe
+
+State: `wmma_emu.hpp` (host, lane by lane) reproduces the double reference to 3e-8 for K4..K7, widths 1/3/5, table codebook and the
+computed code `3single` (`wmma_emu_test`, no GPU needed). `tqk_gemv_wmma.hip.cpp` is its line-by-line HIP form. It was **not
+compiled or run** anywhere. Do the steps in this order; each one is a hard stop if it fails.
+
+1. **Layout probe** (30 s). `hipcc --offload-arch=gfx1201 -O2 hip/wmma_probe.hip.cpp -o wmma_probe && HIP_VISIBLE_DEVICES=<n> ./wmma_probe`.
+ It must print `layout as assumed`. The assumption (wave32, `__builtin_amdgcn_wmma_f32_16x16x16_f16_w32_gfx12`, 8 halves per lane for A and B,
+ 8 floats for D, no replication between the two lane halves):
+ `A[m][k]` in lane `l`, element `j`: `m = l % 16, k = 8 * (l / 16) + j`; `B[k][n]`: `k = 8 * (l / 16) + j, n = l % 16`; `D[m][n]`: `m = 8 * (l / 16) + i, n = l % 16`.
+ If it does not match, change `wmma_emu.hpp` (the specification) and `wmma_emu_test` until it passes with the layout the probe found, then
+ mirror the change in the kernel. (gfx11 replicated A/B between the halves and striped D by `2 i + l / 16`; gfx12 does not, as far as the
+ LLVM documentation says, but nobody here has measured it.)
+2. **Emulator test**: `g++ -O2 -std=c++17 -I third_party/ggml tools/dsv41_lab/wmma_emu_test.cpp tools/trellis/tqenc.cpp -o wmma_emu_test` (and with `TQE_CODE=3single`).
+3. **Kernel**: the data flow, all of it in `wmma_emu.hpp::gemv_tqw`:
+ * *Why a new layout.* mma.m16n8k16 gives a lane 2 rows of A and 4 floats of D; WMMA gives a lane 1 row of A (8 consecutive k) and 8 rows of D.
+ So lane `l = r + 16 kg` owns row `r` of the tile and **16 consecutive steps** (`u = 0..15`, steps `t = 16 kg + u`) of every 128-weight block:
+ stream bits `[(16 - 16 kg) K, (32 - 16 kg) K)`. The two blocks of a pair give 2 chunks of `16 K` bits = `K` words per lane
+ (`words[((tile * NP + P) * K + j) * 32 + lane]`, coalesced). Scales `sc[((tile*NP+P)*16 + row)*2 + bk]`: a lane reads rows `8 kg..8 kg+7`.
+ The bytes are the same as the file layout; the permutation is done at load (a bit shuffle per lane, `repack_tqw`).
+ * *k order.* WMMA k-slot `(kg, j)` of mma `m` = weight `4 (16 kg + 2 m) + j` of the block (steps `u = 2m, 2m+1` of the lane). A and B must use the same
+ map; B lane `(n, kg)` loads `x[n][blk*128 + 4(16 kg + 2m) + j]`, `j = 0..7` (2 float4), split `x = hi + lo` in fp16 (two WMMAs, error 1e-7).
+ `prep_b_tqw` does this once per call into 32 contiguous bytes per (block, m, kg, column).
+ * *States.* local step `u` is at chunk-relative bit `(15 - u) K`. Window 0 (u 0..7) = chunk bits `[8K, 16K)` + the low `15 - K` bits of the **partner lane's
+ chunk** (`lane ^ 16`, same row and block; circular wrap: for kg = 0 the partner is kg = 1) - one `__shfl_xor(x, 16)`; window 1 (u 8..15) = chunk bits `[0, 7K + 15)`.
+ `state(u) = (window >> offset) & 0x7fff`. Both windows fit 64 bits for K <= 7.
+ * *A fragment.* computed code `3single`: `w0 = ((s*89226354 + 64248484) & 0x8FFF8FFF) ^ 0x3B603B60`, `w1 = ((s*0x9E3779B1 + 0x7F4A7C15) & ...) ^ ...`;
+ the four half values of a step are `lo(w0), hi(w0), lo(w1), hi(w1)` = 2 words of the 8-half A vector. Table codebook: `x = s*0x9e3779b1`,
+ words `lut[x >> 21]`, `lut[(x >> 10) & 2047]` from an 8 KB LDS table of half2 (the table is exact in fp16: 0 inexact entries).
+ * *Accumulate.* D (8 floats) after the 8 (16 with lo) WMMAs of a block times the row scale (`fp16 d`, 8 rows per lane) is added to the running `tot[8]`; at the end lane
+ `(n = l % 16 < NB, kg)` stores rows `8 kg .. 8 kg + 7` of column `n`.
+ * Wave32 (the gfx12 default); 128-thread blocks (4 waves), `T = 2` tiles per wave first; VGPR budget: B fragments of one block half (8 m x 8 regs = 64), records `T K`,
+ windows, A (4), D (8) - fits well under 256.
+4. **Compare with the emulator** on the same random streams; the CUDA harness (`tqk_lab.cu`) has the benchmark shape (12 gate/up + 6 down matrices) to copy.
+ Speed targets on gfx1201: K5/3single at 640 GB/s is a 3.7 Tw/s roof; WMMA hi+lo costs 2 x 8 WMMAs per 2048 weights per wave, the ALU side (2 IMAD + 2 LOP3 a step) is
+ light, so expect the memory system to be the limit - the opposite of the A4000, where the tensor pipe is at 75 percent.
+5. Not decided here: whether the R9700 build should keep the table codebook (K6/K7 are byte-compatible with agentionai and must) or the computed code (K4/K5 new types) -
+ see notes-dsv41-cudalab.md.
diff --git a/tools/dsv41_lab/hip/tqk_gemv_wmma.hip.cpp b/tools/dsv41_lab/hip/tqk_gemv_wmma.hip.cpp
new file mode 100644
index 000000000..e9227e112
--- /dev/null
+++ b/tools/dsv41_lab/hip/tqk_gemv_wmma.hip.cpp
@@ -0,0 +1,117 @@
+// tools/dsv41_lab/hip/tqk_gemv_wmma.hip.cpp - gfx12 (RDNA4, R9700 = gfx1201) port of gemv_v3 / gemv_v3c. NOT COMPILED OR RUN on any
+// machine yet: it is the line-by-line HIP form of tools/dsv41_lab/wmma_emu.hpp, whose host emulation matches the double reference
+// (wmma_emu_test). Port steps: (1) run wmma_probe.hip.cpp; (2) hipcc this file with the test main you need; (3) compare against
+// the emulator (same layout TQW, same inputs) lane by lane if anything differs.
+// Differences to the CUDA path, all forced by the instruction: mma.m16n8k16 (A 16x16 as 4 regs, B 16x8 as 2 regs, D 4 floats) becomes
+// v_wmma_f32_16x16x16_f16 (A 8 halves, B 8 halves, D 8 floats, 16 columns of which NB <= 5 are used); a lane holds ONE row of A
+// and 8 rows of D (not two rows of A / D); the lane owns 16 consecutive steps of a block (not 8); hence layout TQW, not TQL.
+#include <hip/hip_runtime.h>
+#include <cstdint>
+
+typedef _Float16 half8_t __attribute__((ext_vector_type(8)));
+typedef float float8_t __attribute__((ext_vector_type(8)));
+
+namespace tqw_hip {
+
+constexpr uint32_t kMask = 0x7fffu;
+
+struct Job {
+ const uint32_t* words; // TQW words [tile][P][K][32 lanes]
+ const uint16_t* scales; // [tile][P][16 rows][2 blocks] fp16
+ const half8_t* xb; // prepared B fragments (prep_b_tqw)
+ float* y; // [NB][ldy]
+ int rows, in, ldy;
+};
+
+// 3single computed code (computed_codes.hpp): a step's four weights are two LCG words, read as two half2 each. For the table
+// codebook replace it by two lookups into an 8 KB LDS table of half2 (lut[x >> 21], lut[(x >> 10) & 2047], x = s * 0x9e3779b1).
+__device__ __forceinline__ void step_words(uint32_t s, uint32_t& w0, uint32_t& w1) {
+ w0 = ((s * 89226354u + 64248484u) & 0x8FFF8FFFu) ^ 0x3B603B60u;
+ w1 = ((s * 0x9E3779B1u + 0x7F4A7C15u) & 0x8FFF8FFFu) ^ 0x3B603B60u;
+}
+
+__device__ __forceinline__ uint32_t funnel_r(uint32_t lo, uint32_t hi, int sh) { return (uint32_t) ((((uint64_t) hi << 32) | lo) >> sh); }
+
+// activations -> B fragments. out[(((blk * 8 + m) * 2 + kg) * nb + n) * 2 + {0 = hi, 1 = lo}] = 8 halves of x[n][blk * 128 + 4 * (16 kg + 2 m) + j]
+__global__ void prep_b_tqw(const float* x, int ldx, int nb, int in, half8_t* out) {
+ const int nblk = in / 128;
+ const size_t gid = (size_t) blockIdx.x * blockDim.x + threadIdx.x, n_all = (size_t) nblk * 8 * 2 * nb;
+ if (gid >= n_all) return;
+ const int n = (int) (gid % nb), kg = (int) ((gid / nb) % 2), m = (int) ((gid / (2 * nb)) % 8), blk = (int) (gid / (16 * nb));
+ const float* p = x + (size_t) n * ldx + blk * 128 + 4 * (16 * kg + 2 * m);
+ half8_t hi, lo;
+ for (int j = 0; j < 8; ++j) { hi[j] = (_Float16) p[j]; lo[j] = (_Float16) (p[j] - (float) hi[j]); }
+ out[gid * 2] = hi;
+ out[gid * 2 + 1] = lo;
+}
+
+// one wave = T tiles of 16 rows. K = bits per step (4..7), NB = columns (<= 16), T tiles per wave, EXACT: hi + lo activations.
+template <int K, int NB, int T, int EXACT>
+__global__ void __launch_bounds__(128) gemv(const Job* jobs) {
+ const Job job = jobs[blockIdx.y];
+ const int wave = threadIdx.x >> 5, lane = threadIdx.x & 31, r = lane & 15, kg = lane >> 4;
+ const int tile0 = (blockIdx.x * (blockDim.x >> 5) + wave) * T;
+ if (tile0 * 16 >= job.rows) return;
+ const int nblk = job.in / 128, NP = nblk / 2;
+ float tot[T][8];
+ for (int t = 0; t < T; ++t)
+ for (int i = 0; i < 8; ++i) tot[t][i] = 0.f;
+ for (int P = 0; P < NP; ++P) {
+ uint32_t rec[T][K];
+ for (int t = 0; t < T; ++t)
+ for (int j = 0; j < K; ++j) rec[t][j] = __builtin_nontemporal_load(job.words + (((size_t) (tile0 + t) * NP + P) * K + j) * 32 + lane);
+ for (int bk = 0; bk < 2; ++bk) {
+ half8_t bh[8], bl[8];
+ if (r < NB) {
+ for (int m = 0; m < 8; ++m) {
+ const half8_t* p = job.xb + (((((size_t) (2 * P + bk) * 8 + m) * 2 + kg) * NB + r) * 2);
+ bh[m] = p[0];
+ if (EXACT) bl[m] = p[1];
+ }
+ } else {
+ for (int m = 0; m < 8; ++m) { bh[m] = half8_t{0, 0, 0, 0, 0, 0, 0, 0}; bl[m] = bh[m]; }
+ }
+ for (int t = 0; t < T; ++t) {
+ // chunk bk of the record: bits [16 K bk, 16 K bk + 16 K); window 0 = steps u 0..7 (+ 15-K bits of the partner chunk),
+ // window 1 = steps u 8..15
+ const int b0 = 16 * K * bk;
+ auto bits = [&](int pos, int n) -> uint64_t { // n <= 56 bits at record bit pos, from the register words
+ const int a = pos >> 5, sh = pos & 31;
+ uint32_t w0 = rec[t][a], w1 = a + 1 < K ? rec[t][a + 1 < K ? a + 1 : 0] : 0u, w2 = a + 2 < K ? rec[t][a + 2 < K ? a + 2 : 0] : 0u;
+ const uint64_t v = ((uint64_t) funnel_r(w1, w2, sh) << 32) | funnel_r(w0, w1, sh);
+ return v & ((1ull << n) - 1ull);
+ };
+ // (the compiler folds the constant positions; a, sh are compile-time after unrolling bk and the loops below)
+ const uint64_t lo8 = bits(b0, 8 * K), hi8 = bits(b0 + 8 * K, 8 * K);
+ const uint32_t mine = (uint32_t) (lo8 & 0xffffffffu); // the bits the PARTNER needs: this lane's chunk bits [0, 15 - K)
+ const uint32_t pc = __shfl_xor((int) mine, 16);
+ const uint64_t win0 = hi8 | ((uint64_t) (pc & ((1u << (15 - K)) - 1u)) << (8 * K));
+ const uint64_t win1 = lo8 | ((hi8 & ((1ull << (15 - K)) - 1ull)) << (8 * K));
+ float8_t acc = {0, 0, 0, 0, 0, 0, 0, 0};
+ for (int m = 0; m < 8; ++m) {
+ half8_t a;
+ for (int s2 = 0; s2 < 2; ++s2) {
+ const int u = 2 * m + s2;
+ const uint32_t st = (uint32_t) (u < 8 ? win0 >> ((7 - u) * K) : win1 >> ((15 - u) * K)) & kMask;
+ uint32_t w0, w1;
+ step_words(st, w0, w1);
+ a[4 * s2 + 0] = __builtin_bit_cast(_Float16, (uint16_t) (w0 & 0xffffu));
+ a[4 * s2 + 1] = __builtin_bit_cast(_Float16, (uint16_t) (w0 >> 16));
+ a[4 * s2 + 2] = __builtin_bit_cast(_Float16, (uint16_t) (w1 & 0xffffu));
+ a[4 * s2 + 3] = __builtin_bit_cast(_Float16, (uint16_t) (w1 >> 16));
+ }
+ acc = __builtin_amdgcn_wmma_f32_16x16x16_f16_w32_gfx12(a, bh[m], acc);
+ if (EXACT) acc = __builtin_amdgcn_wmma_f32_16x16x16_f16_w32_gfx12(a, bl[m], acc);
+ }
+ // scales of rows 8 kg .. 8 kg + 7 of this block: 8 consecutive (row, bk) pairs of 2 -> stride 2
+ const uint16_t* sp = job.scales + (((size_t) (tile0 + t) * NP + P) * 16 + 8 * kg) * 2 + bk;
+ for (int i = 0; i < 8; ++i) tot[t][i] += (float) __builtin_bit_cast(_Float16, sp[2 * i]) * acc[i];
+ }
+ }
+ }
+ for (int t = 0; t < T; ++t)
+ if (r < NB)
+ for (int i = 0; i < 8; ++i) job.y[(size_t) r * job.ldy + (tile0 + t) * 16 + 8 * kg + i] = tot[t][i];
+}
+
+} // namespace tqw_hip
diff --git a/tools/dsv41_lab/hip/wmma_probe.hip.cpp b/tools/dsv41_lab/hip/wmma_probe.hip.cpp
new file mode 100644
index 000000000..7b3ccb66d
--- /dev/null
+++ b/tools/dsv41_lab/hip/wmma_probe.hip.cpp
@@ -0,0 +1,40 @@
+// tools/dsv41_lab/hip/wmma_probe.hip.cpp - run this FIRST on the gfx1201 box (one launch, no inputs): it checks the lane layout of
+// __builtin_amdgcn_wmma_f32_16x16x16_f16_w32_gfx12 that wmma_emu.hpp and tqk_gemv_wmma.hip.cpp assume:
+// A[m][k]: lane l holds a[j] = A[l % 16][8 * (l / 16) + j]; B[k][n]: lane l holds b[j] = B[8 * (l / 16) + j][l % 16]
+// D[m][n]: lane l holds c[i] = D[8 * (l / 16) + i][l % 16]
+// Probe 1: A[m][k] = m * 16 + k (exact in fp16), B = identity => D = A, so lane l's c[i] must be (8 (l/16) + i) * 16 + (l % 16).
+// Probe 2: A = identity, B[k][n] = k * 16 + n => D = B, same expectation.
+// A mismatch in probe 1 only means the A (or D) layout differs; in probe 2 only B (or D): the printout says which.
+// hipcc --offload-arch=gfx1201 -O2 wmma_probe.hip.cpp -o wmma_probe && HIP_VISIBLE_DEVICES=<card> ./wmma_probe
+#include <hip/hip_runtime.h>
+#include <cstdio>
+
+typedef _Float16 half8_t __attribute__((ext_vector_type(8)));
+typedef float float8_t __attribute__((ext_vector_type(8)));
+
+__global__ void probe(float* out, int which) {
+ const int l = threadIdx.x; // wave32
+ half8_t a, b;
+ for (int j = 0; j < 8; ++j) {
+ const int m = l % 16, k = 8 * (l / 16) + j; // the layout under test, for A
+ const int k2 = 8 * (l / 16) + j, n = l % 16; // for B
+ a[j] = (_Float16) (which == 0 ? (float) (m * 16 + k) : (m == k ? 1.f : 0.f));
+ b[j] = (_Float16) (which == 0 ? (k2 == n ? 1.f : 0.f) : (float) (k2 * 16 + n));
+ }
+ float8_t c = {0, 0, 0, 0, 0, 0, 0, 0};
+ c = __builtin_amdgcn_wmma_f32_16x16x16_f16_w32_gfx12(a, b, c);
+ for (int i = 0; i < 8; ++i) out[(which * 32 + l) * 8 + i] = c[i];
+}
+
+int main() {
+ float* d; hipMalloc(&d, 2 * 32 * 8 * sizeof(float));
+ probe<<<1, 32>>>(d, 0); probe<<<1, 32>>>(d, 1);
+ float h[2 * 32 * 8]; hipMemcpy(h, d, sizeof h, hipMemcpyDeviceToHost);
+ int bad[2] = {0, 0};
+ for (int w = 0; w < 2; ++w)
+ for (int l = 0; l < 32; ++l)
+ for (int i = 0; i < 8; ++i) bad[w] += h[(w * 32 + l) * 8 + i] != (float) ((8 * (l / 16) + i) * 16 + (l % 16));
+ std::printf("probe 1 (A layout, D layout): %d of 256 values wrong\nprobe 2 (B layout, D layout): %d of 256 wrong\n%s\n", bad[0], bad[1],
+ (bad[0] || bad[1]) ? "LAYOUT DIFFERS: fix wmma_emu.hpp first (the emulator is the specification), then the kernel" : "layout as assumed");
+ return bad[0] || bad[1];
+}
diff --git a/tools/dsv41_lab/wmma_emu.hpp b/tools/dsv41_lab/wmma_emu.hpp
new file mode 100644
index 000000000..968e4dd43
--- /dev/null
+++ b/tools/dsv41_lab/wmma_emu.hpp
@@ -0,0 +1,168 @@
+// tools/dsv41_lab/wmma_emu.hpp - host emulation of the gfx12 (RDNA4, R9700 = gfx1201) port of gemv_v3, written lane by lane exactly
+// as the HIP kernel in hip/tqk_gemv_wmma.hip.cpp does it, so the data flow (layout "TQW", step -> k-slot map, window bits, shuffles, B
+// fragments, D scaling) is checked on any machine against the double reference BEFORE it runs on the card. The only assumption
+// that needs the card is the lane layout of v_wmma_f32_16x16x16_f16 (hip/wmma_probe.hip.cpp checks it in one launch).
+//
+// Assumed layout, wave32, __builtin_amdgcn_wmma_f32_16x16x16_f16_w32_gfx12(v8h a, v8h b, v8f c):
+// A[m][k]: lane l holds a[j] = A[m = l % 16][k = 8 * (l / 16) + j], j = 0..7 (no replication between the lane halves)
+// B[k][n]: lane l holds b[j] = B[k = 8 * (l / 16) + j][n = l % 16]
+// D[m][n]: lane l holds c[i] = D[m = 8 * (l / 16) + i][n = l % 16], i = 0..7
+#pragma once
+#include "computed_codes.hpp"
+
+#include <cstdint>
+#include <cstring>
+#include <vector>
+
+namespace tqw {
+
+struct Half8 { uint16_t h[8]; };
+struct Float8 { float f[8]; };
+
+// D = A * B + C on 32 lanes
+inline void wmma_f32_16x16x16_f16(const Half8 a[32], const Half8 b[32], Float8 c[32]) {
+ float A[16][16], B[16][16], D[16][16];
+ for (int l = 0; l < 32; ++l)
+ for (int j = 0; j < 8; ++j) {
+ A[l % 16][8 * (l / 16) + j] = tqcode::h2f(a[l].h[j]);
+ B[8 * (l / 16) + j][l % 16] = tqcode::h2f(b[l].h[j]);
+ }
+ for (int l = 0; l < 32; ++l)
+ for (int i = 0; i < 8; ++i) D[8 * (l / 16) + i][l % 16] = c[l].f[i];
+ for (int m = 0; m < 16; ++m)
+ for (int n = 0; n < 16; ++n) {
+ double s = D[m][n];
+ for (int k = 0; k < 16; ++k) s += (double) A[m][k] * B[k][n];
+ D[m][n] = (float) s;
+ }
+ for (int l = 0; l < 32; ++l)
+ for (int i = 0; i < 8; ++i) c[l].f[i] = D[8 * (l / 16) + i][l % 16];
+}
+
+// ---- layout TQW (gfx12): a tile = 16 rows, a pair P = blocks 2P, 2P+1. Lane l = r + 16 kg of the wave owns row r and, of every
+// block, the 16 steps u = 0..15 of the half kg (steps t = 16 kg + u): stream bits [(16 - 16 kg) K, (32 - 16 kg) K) of the block.
+// Its two chunks (bk = 0, 1) are 32 K bits = K words: words[((tile * NP + P) * K + j) * 32 + lane].
+// Scales: sc[((tile * NP + P) * 16 + r) * 2 + bk] (u16 fp16 d of row r, block 2P + bk): a lane reads the 8 rows 8 kg .. 8 kg + 7 = 32 bytes.
+inline void repack_tqw(int K, const uint8_t* w, int rows, int nblk, std::vector<uint32_t>& words, std::vector<uint16_t>& sc) {
+ const int bsz = 2 + 4 * K, NP = nblk / 2, NT = rows / 16;
+ words.assign((size_t) NT * NP * K * 32, 0u);
+ sc.assign((size_t) NT * NP * 32, 0);
+ for (int t = 0; t < NT; ++t)
+ for (int P = 0; P < NP; ++P) {
+ const size_t base = (size_t) t * NP + P;
+ for (int r = 0; r < 16; ++r)
+ for (int bk = 0; bk < 2; ++bk) {
+ const uint8_t* blk = w + ((size_t) (t * 16 + r) * nblk + 2 * P + bk) * bsz;
+ std::memcpy(&sc[(base * 16 + r) * 2 + bk], blk, 2);
+ for (int kg = 0; kg < 2; ++kg) {
+ const int lane = r + 16 * kg;
+ for (int b = 0; b < 16 * K; ++b) { // chunk bit b -> record bit 16 K bk + b
+ const int src = (16 - 16 * kg) * K + b;
+ const uint32_t bit = (blk[2 + (src >> 3)] >> (src & 7)) & 1u;
+ const int dst = 16 * K * bk + b;
+ words[(base * K + (dst >> 5)) * 32 + lane] |= bit << (dst & 31);
+ }
+ }
+ }
+ }
+}
+
+// B fragments (prep_b of the CUDA path, gfx12 form): for column n < nb, block blk, mma m, k-group kg: 8 halves hi and 8 halves lo of
+// x[n][blk * 128 + 4 * (16 kg + 2 m) + j], j = 0..7 (hi = fp16(x), lo = fp16(x - hi))
+inline uint16_t f2h_rn(float f) { // fp16 round to nearest even (host)
+ uint32_t u; std::memcpy(&u, &f, 4);
+ const uint32_t s = (u >> 16) & 0x8000u; int e = (int) ((u >> 23) & 0xff) - 127 + 15; uint32_t m = u & 0x7fffffu;
+ if (((u >> 23) & 0xff) == 0xff) return (uint16_t) (s | 0x7c00u | (m ? 0x200u : 0));
+ if (e >= 31) return (uint16_t) (s | 0x7c00u);
+ if (e <= 0) {
+ if (e < -10) return (uint16_t) s;
+ m |= 0x800000u; const int sh = 14 - e; uint32_t hm = m >> sh; const uint32_t rem = m & ((1u << sh) - 1), half = 1u << (sh - 1);
+ if (rem > half || (rem == half && (hm & 1))) ++hm;
+ return (uint16_t) (s | hm);
+ }
+ uint32_t hm = m >> 13; const uint32_t rem = m & 0x1fffu;
+ uint32_t v = ((uint32_t) e << 10) | hm;
+ if (rem > 0x1000u || (rem == 0x1000u && (v & 1))) ++v;
+ return (uint16_t) (s | v);
+}
+
+// the A fragment of a lane for one trellis step: (word 0, word 1) -> 4 fp16 (weights 4t .. 4t+3). computed code "3single":
+// the two LCG words as they are. table: the two half2 entries of the hash.
+using StepFn = void (*)(uint32_t state, uint32_t out[2]);
+
+// The kernel, wave by wave. x: nb columns of F32 activations [nb][in]. Returns y[nb][rows].
+inline void gemv_tqw(int K, int nb, int rows, int in, const std::vector<uint32_t>& words, const std::vector<uint16_t>& sc,
+ const std::vector<float>& x, StepFn step, std::vector<float>& y, bool exact = true) {
+ const int nblk = in / 128, NP = nblk / 2, NT = rows / 16;
+ y.assign((size_t) nb * rows, 0.f);
+ for (int t = 0; t < NT; ++t) {
+ float tot[32][8] = {};
+ for (int P = 0; P < NP; ++P) {
+ const size_t base = (size_t) t * NP + P;
+ // per lane: the record, the two chunk windows (8 steps each) with the partner bits
+ uint64_t win[32][2][2]; // [lane][bk][half of the chunk: 0 = steps u 0..7, 1 = u 8..15]
+ for (int lane = 0; lane < 32; ++lane) {
+ uint32_t r[8];
+ for (int j = 0; j < K; ++j) r[j] = words[(base * K + j) * 32 + lane];
+ for (int bk = 0; bk < 2; ++bk) {
+ // chunk = record bits [16 K bk, 16 K (bk + 1)), up to 80 bits: read as two 64-bit pieces
+ auto bits = [&](int pos, int n) { // n <= 64 bits at record bit pos
+ uint64_t v = 0;
+ for (int i = 0; i < n; ++i) { const int p = pos + i; v |= (uint64_t) ((r[p >> 5] >> (p & 31)) & 1u) << i; }
+ return v;
+ };
+ const uint64_t lo8 = bits(16 * K * bk, 8 * K); // chunk bits [0, 8K) -> steps u 8..15
+ const uint64_t hi8 = bits(16 * K * bk + 8 * K, 8 * K); // chunk bits [8K, 16K) -> steps u 0..7
+ win[lane][bk][1] = lo8 | (hi8 & ((1ull << (15 - K)) - 1ull)) << (8 * K); // u 8..15 end at bit 7K + 15 <= 16K - 1
+ win[lane][bk][0] = hi8; // partner bits added below
+ }
+ }
+ // partner bits: the low 15 - K bits of the chunk of lane ^ 16 (same bk)
+ for (int lane = 0; lane < 32; ++lane)
+ for (int bk = 0; bk < 2; ++bk) {
+ const uint64_t pc = win[lane ^ 16][bk][1] & ((1ull << (8 * K)) - 1ull); // partner chunk bits [0, 8K) (its steps u 8..15)
+ win[lane][bk][0] |= (pc & ((1ull << (15 - K)) - 1ull)) << (8 * K);
+ }
+ for (int bk = 0; bk < 2; ++bk) {
+ Float8 acc[32] = {};
+ for (int m = 0; m < 8; ++m) {
+ Half8 a[32], bh[32], bl[32];
+ for (int lane = 0; lane < 32; ++lane) {
+ const int kg = lane / 16;
+ for (int s2 = 0; s2 < 2; ++s2) {
+ const int u = 2 * m + s2; // local step 0..15
+ // u 0..7 live in window 0 at offset (7 - u) K, u 8..15 in window 1 at offset (15 - u) K
+ const uint32_t st = (uint32_t) ((u < 8 ? win[lane][bk][0] >> ((7 - u) * K) : win[lane][bk][1] >> ((15 - u) * K))) & 0x7fffu;
+ uint32_t w2[2];
+ step(st, w2);
+ a[lane].h[4 * s2 + 0] = (uint16_t) (w2[0] & 0xffffu); a[lane].h[4 * s2 + 1] = (uint16_t) (w2[0] >> 16);
+ a[lane].h[4 * s2 + 2] = (uint16_t) (w2[1] & 0xffffu); a[lane].h[4 * s2 + 3] = (uint16_t) (w2[1] >> 16);
+ }
+ const int n = lane % 16;
+ for (int j = 0; j < 8; ++j) {
+ float xv = 0.f;
+ if (n < nb) xv = x[(size_t) n * in + (2 * P + bk) * 128 + 4 * (16 * kg + 2 * m) + j];
+ const uint16_t hi = f2h_rn(xv);
+ bh[lane].h[j] = hi;
+ bl[lane].h[j] = f2h_rn(xv - tqcode::h2f(hi));
+ }
+ }
+ wmma_f32_16x16x16_f16(a, bh, acc);
+ if (exact) wmma_f32_16x16x16_f16(a, bl, acc);
+ }
+ for (int lane = 0; lane < 32; ++lane)
+ for (int i = 0; i < 8; ++i) {
+ const int row = 8 * (lane / 16) + i;
+ tot[lane][i] += tqcode::h2f(sc[(base * 16 + row) * 2 + bk]) * acc[lane].f[i];
+ }
+ }
+ }
+ for (int lane = 0; lane < 32; ++lane) {
+ const int n = lane % 16;
+ if (n >= nb) continue;
+ for (int i = 0; i < 8; ++i) y[(size_t) n * rows + t * 16 + 8 * (lane / 16) + i] = tot[lane][i];
+ }
+ }
+}
+
+} // namespace tqw
diff --git a/tools/dsv41_lab/wmma_emu_test.cpp b/tools/dsv41_lab/wmma_emu_test.cpp
new file mode 100644
index 000000000..f0ad75edf
--- /dev/null
+++ b/tools/dsv41_lab/wmma_emu_test.cpp
@@ -0,0 +1,64 @@
+// tools/dsv41_lab/wmma_emu_test.cpp - the gfx12 port's data flow (wmma_emu.hpp) against the double reference on decoded weights.
+// ./wmma_emu_test table codebook (shipped K5/K6/K7)
+// TQE_CODE=3single ./wmma_emu_test computed code
+// Host only; no GPU needed.
+#include "../trellis/tqenc.hpp"
+#include "wmma_emu.hpp"
+
+#include <algorithm>
+#include <cmath>
+#include <cstdio>
+#include <cstdlib>
+#include <random>
+#include <vector>
+
+static void step_table(uint32_t s, uint32_t out[2]) {
+ static std::vector<uint32_t> lut;
+ if (lut.empty()) {
+ const float* pt = tqenc::point_table();
+ lut.resize(2048);
+ for (int i = 0; i < 2048; ++i) lut[i] = tqw::f2h_rn(pt[2 * i]) | ((uint32_t) tqw::f2h_rn(pt[2 * i + 1]) << 16);
+ }
+ const uint32_t x = (s & 0x7fffu) * 0x9e3779b1u;
+ out[0] = lut[x >> 21];
+ out[1] = lut[(x >> 10) & 2047u];
+}
+static void step_3single(uint32_t s, uint32_t out[2]) { out[0] = tqcode::word3(s & 0x7fffu, 0); out[1] = tqcode::word3(s & 0x7fffu, 1); }
+
+int main() {
+ const bool computed = tqcode::kind_of(std::getenv("TQE_CODE")) == tqcode::k3single;
+ if (std::getenv("TQE_CODE") && !computed) { std::printf("only TQE_CODE=3single is wired here\n"); return 2; }
+ std::mt19937_64 rng(7);
+ int fails = 0;
+ const int Ks[4] = {4, 5, 6, 7}, types[4] = {225, 224, 145, 146};
+ for (int ki = 0; ki < 4; ++ki) {
+ const int K = Ks[ki], type = types[ki], bsz = 2 + 4 * K;
+ const int rows = 48, in = 768, nblk = in / 128;
+ std::vector<uint8_t> w((size_t) rows * nblk * bsz);
+ for (auto& b : w) b = (uint8_t) rng();
+ for (int64_t b = 0; b < (int64_t) rows * nblk; ++b) { const uint16_t d = tqw::f2h_rn(0.02f + 0.001f * (float) (rng() % 400)); std::memcpy(&w[(size_t) b * bsz], &d, 2); }
+ std::vector<float> dec((size_t) rows * in);
+ tqenc::decode_blocks(type, w.data(), (int64_t) rows * nblk, dec.data());
+ std::vector<uint32_t> words; std::vector<uint16_t> sc;
+ tqw::repack_tqw(K, w.data(), rows, nblk, words, sc);
+ for (int nb : {1, 3, 5}) {
+ std::vector<float> x((size_t) nb * in);
+ std::uniform_real_distribution<float> u(-1.f, 1.f);
+ for (auto& v : x) v = u(rng);
+ std::vector<float> y;
+ tqw::gemv_tqw(K, nb, rows, in, words, sc, x, computed ? step_3single : step_table, y);
+ double worst = 0;
+ for (int c = 0; c < nb; ++c)
+ for (int r = 0; r < rows; ++r) {
+ double s = 0, sa = 0;
+ for (int i = 0; i < in; ++i) { const double t = (double) dec[(size_t) r * in + i] * x[(size_t) c * in + i]; s += t; sa += std::fabs(t); }
+ worst = std::max(worst, std::fabs(y[(size_t) c * rows + r] - s) / (sa + 1e-30));
+ }
+ const bool ok = worst < 3e-7;
+ std::printf("K=%d nb=%d (%s): emulated gfx12 flow vs double reference %.2e %s\n", K, nb, computed ? "3single" : "table", worst, ok ? "ok" : "FAIL");
+ fails += !ok;
+ }
+ }
+ std::printf("%s\n", fails ? "FAILED" : "ALL OK");
+ return fails;
+}