Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
dsv41 cudalab: gfx12 WMMA port recipe of the trellis GEMV: host emulation of the data flow (matches the reference), HIP kernel source, layout probe
Niko1221d251b2c0db6d
diff --git a/tools/dsv41_lab/hip/README.md b/tools/dsv41_lab/hip/README.mdnew file mode 100644index 000000000..e998b1ae2--- /dev/null+++ b/tools/dsv41_lab/hip/README.md@@ -0,0 +1,38 @@+# HIP port of the trellis GEMV (gfx1201, R9700) - recipe++State: `wmma_emu.hpp` (host, lane by lane) reproduces the double reference to 3e-8 for K4..K7, widths 1/3/5, table codebook and the+computed code `3single` (`wmma_emu_test`, no GPU needed). `tqk_gemv_wmma.hip.cpp` is its line-by-line HIP form. It was **not+compiled or run** anywhere. Do the steps in this order; each one is a hard stop if it fails.++1. **Layout probe** (30 s). `hipcc --offload-arch=gfx1201 -O2 hip/wmma_probe.hip.cpp -o wmma_probe && HIP_VISIBLE_DEVICES=<n> ./wmma_probe`.+ It must print `layout as assumed`. The assumption (wave32, `__builtin_amdgcn_wmma_f32_16x16x16_f16_w32_gfx12`, 8 halves per lane for A and B,+ 8 floats for D, no replication between the two lane halves):+ `A[m][k]` in lane `l`, element `j`: `m = l % 16, k = 8 * (l / 16) + j`; `B[k][n]`: `k = 8 * (l / 16) + j, n = l % 16`; `D[m][n]`: `m = 8 * (l / 16) + i, n = l % 16`.+ If it does not match, change `wmma_emu.hpp` (the specification) and `wmma_emu_test` until it passes with the layout the probe found, then+ mirror the change in the kernel. (gfx11 replicated A/B between the halves and striped D by `2 i + l / 16`; gfx12 does not, as far as the+ LLVM documentation says, but nobody here has measured it.)+2. **Emulator test**: `g++ -O2 -std=c++17 -I third_party/ggml tools/dsv41_lab/wmma_emu_test.cpp tools/trellis/tqenc.cpp -o wmma_emu_test` (and with `TQE_CODE=3single`).+3. **Kernel**: the data flow, all of it in `wmma_emu.hpp::gemv_tqw`:+ * *Why a new layout.* mma.m16n8k16 gives a lane 2 rows of A and 4 floats of D; WMMA gives a lane 1 row of A (8 consecutive k) and 8 rows of D.+ So lane `l = r + 16 kg` owns row `r` of the tile and **16 consecutive steps** (`u = 0..15`, steps `t = 16 kg + u`) of every 128-weight block:+ stream bits `[(16 - 16 kg) K, (32 - 16 kg) K)`. The two blocks of a pair give 2 chunks of `16 K` bits = `K` words per lane+ (`words[((tile * NP + P) * K + j) * 32 + lane]`, coalesced). Scales `sc[((tile*NP+P)*16 + row)*2 + bk]`: a lane reads rows `8 kg..8 kg+7`.+ The bytes are the same as the file layout; the permutation is done at load (a bit shuffle per lane, `repack_tqw`).+ * *k order.* WMMA k-slot `(kg, j)` of mma `m` = weight `4 (16 kg + 2 m) + j` of the block (steps `u = 2m, 2m+1` of the lane). A and B must use the same+ map; B lane `(n, kg)` loads `x[n][blk*128 + 4(16 kg + 2m) + j]`, `j = 0..7` (2 float4), split `x = hi + lo` in fp16 (two WMMAs, error 1e-7).+ `prep_b_tqw` does this once per call into 32 contiguous bytes per (block, m, kg, column).+ * *States.* local step `u` is at chunk-relative bit `(15 - u) K`. Window 0 (u 0..7) = chunk bits `[8K, 16K)` + the low `15 - K` bits of the **partner lane's+ chunk** (`lane ^ 16`, same row and block; circular wrap: for kg = 0 the partner is kg = 1) - one `__shfl_xor(x, 16)`; window 1 (u 8..15) = chunk bits `[0, 7K + 15)`.+ `state(u) = (window >> offset) & 0x7fff`. Both windows fit 64 bits for K <= 7.+ * *A fragment.* computed code `3single`: `w0 = ((s*89226354 + 64248484) & 0x8FFF8FFF) ^ 0x3B603B60`, `w1 = ((s*0x9E3779B1 + 0x7F4A7C15) & ...) ^ ...`;+ the four half values of a step are `lo(w0), hi(w0), lo(w1), hi(w1)` = 2 words of the 8-half A vector. Table codebook: `x = s*0x9e3779b1`,+ words `lut[x >> 21]`, `lut[(x >> 10) & 2047]` from an 8 KB LDS table of half2 (the table is exact in fp16: 0 inexact entries).+ * *Accumulate.* D (8 floats) after the 8 (16 with lo) WMMAs of a block times the row scale (`fp16 d`, 8 rows per lane) is added to the running `tot[8]`; at the end lane+ `(n = l % 16 < NB, kg)` stores rows `8 kg .. 8 kg + 7` of column `n`.+ * Wave32 (the gfx12 default); 128-thread blocks (4 waves), `T = 2` tiles per wave first; VGPR budget: B fragments of one block half (8 m x 8 regs = 64), records `T K`,+ windows, A (4), D (8) - fits well under 256.+4. **Compare with the emulator** on the same random streams; the CUDA harness (`tqk_lab.cu`) has the benchmark shape (12 gate/up + 6 down matrices) to copy.+ Speed targets on gfx1201: K5/3single at 640 GB/s is a 3.7 Tw/s roof; WMMA hi+lo costs 2 x 8 WMMAs per 2048 weights per wave, the ALU side (2 IMAD + 2 LOP3 a step) is+ light, so expect the memory system to be the limit - the opposite of the A4000, where the tensor pipe is at 75 percent.+5. Not decided here: whether the R9700 build should keep the table codebook (K6/K7 are byte-compatible with agentionai and must) or the computed code (K4/K5 new types) -+ see notes-dsv41-cudalab.md.diff --git a/tools/dsv41_lab/hip/tqk_gemv_wmma.hip.cpp b/tools/dsv41_lab/hip/tqk_gemv_wmma.hip.cppnew file mode 100644index 000000000..e9227e112--- /dev/null+++ b/tools/dsv41_lab/hip/tqk_gemv_wmma.hip.cpp@@ -0,0 +1,117 @@+// tools/dsv41_lab/hip/tqk_gemv_wmma.hip.cpp - gfx12 (RDNA4, R9700 = gfx1201) port of gemv_v3 / gemv_v3c. NOT COMPILED OR RUN on any+// machine yet: it is the line-by-line HIP form of tools/dsv41_lab/wmma_emu.hpp, whose host emulation matches the double reference+// (wmma_emu_test). Port steps: (1) run wmma_probe.hip.cpp; (2) hipcc this file with the test main you need; (3) compare against+// the emulator (same layout TQW, same inputs) lane by lane if anything differs.+// Differences to the CUDA path, all forced by the instruction: mma.m16n8k16 (A 16x16 as 4 regs, B 16x8 as 2 regs, D 4 floats) becomes+// v_wmma_f32_16x16x16_f16 (A 8 halves, B 8 halves, D 8 floats, 16 columns of which NB <= 5 are used); a lane holds ONE row of A+// and 8 rows of D (not two rows of A / D); the lane owns 16 consecutive steps of a block (not 8); hence layout TQW, not TQL.+#include <hip/hip_runtime.h>+#include <cstdint>++typedef _Float16 half8_t __attribute__((ext_vector_type(8)));+typedef float float8_t __attribute__((ext_vector_type(8)));++namespace tqw_hip {++constexpr uint32_t kMask = 0x7fffu;++struct Job {+ const uint32_t* words; // TQW words [tile][P][K][32 lanes]+ const uint16_t* scales; // [tile][P][16 rows][2 blocks] fp16+ const half8_t* xb; // prepared B fragments (prep_b_tqw)+ float* y; // [NB][ldy]+ int rows, in, ldy;+};++// 3single computed code (computed_codes.hpp): a step's four weights are two LCG words, read as two half2 each. For the table+// codebook replace it by two lookups into an 8 KB LDS table of half2 (lut[x >> 21], lut[(x >> 10) & 2047], x = s * 0x9e3779b1).+__device__ __forceinline__ void step_words(uint32_t s, uint32_t& w0, uint32_t& w1) {+ w0 = ((s * 89226354u + 64248484u) & 0x8FFF8FFFu) ^ 0x3B603B60u;+ w1 = ((s * 0x9E3779B1u + 0x7F4A7C15u) & 0x8FFF8FFFu) ^ 0x3B603B60u;+}++__device__ __forceinline__ uint32_t funnel_r(uint32_t lo, uint32_t hi, int sh) { return (uint32_t) ((((uint64_t) hi << 32) | lo) >> sh); }++// activations -> B fragments. out[(((blk * 8 + m) * 2 + kg) * nb + n) * 2 + {0 = hi, 1 = lo}] = 8 halves of x[n][blk * 128 + 4 * (16 kg + 2 m) + j]+__global__ void prep_b_tqw(const float* x, int ldx, int nb, int in, half8_t* out) {+ const int nblk = in / 128;+ const size_t gid = (size_t) blockIdx.x * blockDim.x + threadIdx.x, n_all = (size_t) nblk * 8 * 2 * nb;+ if (gid >= n_all) return;+ const int n = (int) (gid % nb), kg = (int) ((gid / nb) % 2), m = (int) ((gid / (2 * nb)) % 8), blk = (int) (gid / (16 * nb));+ const float* p = x + (size_t) n * ldx + blk * 128 + 4 * (16 * kg + 2 * m);+ half8_t hi, lo;+ for (int j = 0; j < 8; ++j) { hi[j] = (_Float16) p[j]; lo[j] = (_Float16) (p[j] - (float) hi[j]); }+ out[gid * 2] = hi;+ out[gid * 2 + 1] = lo;+}++// one wave = T tiles of 16 rows. K = bits per step (4..7), NB = columns (<= 16), T tiles per wave, EXACT: hi + lo activations.+template <int K, int NB, int T, int EXACT>+__global__ void __launch_bounds__(128) gemv(const Job* jobs) {+ const Job job = jobs[blockIdx.y];+ const int wave = threadIdx.x >> 5, lane = threadIdx.x & 31, r = lane & 15, kg = lane >> 4;+ const int tile0 = (blockIdx.x * (blockDim.x >> 5) + wave) * T;+ if (tile0 * 16 >= job.rows) return;+ const int nblk = job.in / 128, NP = nblk / 2;+ float tot[T][8];+ for (int t = 0; t < T; ++t)+ for (int i = 0; i < 8; ++i) tot[t][i] = 0.f;+ for (int P = 0; P < NP; ++P) {+ uint32_t rec[T][K];+ for (int t = 0; t < T; ++t)+ for (int j = 0; j < K; ++j) rec[t][j] = __builtin_nontemporal_load(job.words + (((size_t) (tile0 + t) * NP + P) * K + j) * 32 + lane);+ for (int bk = 0; bk < 2; ++bk) {+ half8_t bh[8], bl[8];+ if (r < NB) {+ for (int m = 0; m < 8; ++m) {+ const half8_t* p = job.xb + (((((size_t) (2 * P + bk) * 8 + m) * 2 + kg) * NB + r) * 2);+ bh[m] = p[0];+ if (EXACT) bl[m] = p[1];+ }+ } else {+ for (int m = 0; m < 8; ++m) { bh[m] = half8_t{0, 0, 0, 0, 0, 0, 0, 0}; bl[m] = bh[m]; }+ }+ for (int t = 0; t < T; ++t) {+ // chunk bk of the record: bits [16 K bk, 16 K bk + 16 K); window 0 = steps u 0..7 (+ 15-K bits of the partner chunk),+ // window 1 = steps u 8..15+ const int b0 = 16 * K * bk;+ auto bits = [&](int pos, int n) -> uint64_t { // n <= 56 bits at record bit pos, from the register words+ const int a = pos >> 5, sh = pos & 31;+ uint32_t w0 = rec[t][a], w1 = a + 1 < K ? rec[t][a + 1 < K ? a + 1 : 0] : 0u, w2 = a + 2 < K ? rec[t][a + 2 < K ? a + 2 : 0] : 0u;+ const uint64_t v = ((uint64_t) funnel_r(w1, w2, sh) << 32) | funnel_r(w0, w1, sh);+ return v & ((1ull << n) - 1ull);+ };+ // (the compiler folds the constant positions; a, sh are compile-time after unrolling bk and the loops below)+ const uint64_t lo8 = bits(b0, 8 * K), hi8 = bits(b0 + 8 * K, 8 * K);+ const uint32_t mine = (uint32_t) (lo8 & 0xffffffffu); // the bits the PARTNER needs: this lane's chunk bits [0, 15 - K)+ const uint32_t pc = __shfl_xor((int) mine, 16);+ const uint64_t win0 = hi8 | ((uint64_t) (pc & ((1u << (15 - K)) - 1u)) << (8 * K));+ const uint64_t win1 = lo8 | ((hi8 & ((1ull << (15 - K)) - 1ull)) << (8 * K));+ float8_t acc = {0, 0, 0, 0, 0, 0, 0, 0};+ for (int m = 0; m < 8; ++m) {+ half8_t a;+ for (int s2 = 0; s2 < 2; ++s2) {+ const int u = 2 * m + s2;+ const uint32_t st = (uint32_t) (u < 8 ? win0 >> ((7 - u) * K) : win1 >> ((15 - u) * K)) & kMask;+ uint32_t w0, w1;+ step_words(st, w0, w1);+ a[4 * s2 + 0] = __builtin_bit_cast(_Float16, (uint16_t) (w0 & 0xffffu));+ a[4 * s2 + 1] = __builtin_bit_cast(_Float16, (uint16_t) (w0 >> 16));+ a[4 * s2 + 2] = __builtin_bit_cast(_Float16, (uint16_t) (w1 & 0xffffu));+ a[4 * s2 + 3] = __builtin_bit_cast(_Float16, (uint16_t) (w1 >> 16));+ }+ acc = __builtin_amdgcn_wmma_f32_16x16x16_f16_w32_gfx12(a, bh[m], acc);+ if (EXACT) acc = __builtin_amdgcn_wmma_f32_16x16x16_f16_w32_gfx12(a, bl[m], acc);+ }+ // scales of rows 8 kg .. 8 kg + 7 of this block: 8 consecutive (row, bk) pairs of 2 -> stride 2+ const uint16_t* sp = job.scales + (((size_t) (tile0 + t) * NP + P) * 16 + 8 * kg) * 2 + bk;+ for (int i = 0; i < 8; ++i) tot[t][i] += (float) __builtin_bit_cast(_Float16, sp[2 * i]) * acc[i];+ }+ }+ }+ for (int t = 0; t < T; ++t)+ if (r < NB)+ for (int i = 0; i < 8; ++i) job.y[(size_t) r * job.ldy + (tile0 + t) * 16 + 8 * kg + i] = tot[t][i];+}++} // namespace tqw_hipdiff --git a/tools/dsv41_lab/hip/wmma_probe.hip.cpp b/tools/dsv41_lab/hip/wmma_probe.hip.cppnew file mode 100644index 000000000..7b3ccb66d--- /dev/null+++ b/tools/dsv41_lab/hip/wmma_probe.hip.cpp@@ -0,0 +1,40 @@+// tools/dsv41_lab/hip/wmma_probe.hip.cpp - run this FIRST on the gfx1201 box (one launch, no inputs): it checks the lane layout of+// __builtin_amdgcn_wmma_f32_16x16x16_f16_w32_gfx12 that wmma_emu.hpp and tqk_gemv_wmma.hip.cpp assume:+// A[m][k]: lane l holds a[j] = A[l % 16][8 * (l / 16) + j]; B[k][n]: lane l holds b[j] = B[8 * (l / 16) + j][l % 16]+// D[m][n]: lane l holds c[i] = D[8 * (l / 16) + i][l % 16]+// Probe 1: A[m][k] = m * 16 + k (exact in fp16), B = identity => D = A, so lane l's c[i] must be (8 (l/16) + i) * 16 + (l % 16).+// Probe 2: A = identity, B[k][n] = k * 16 + n => D = B, same expectation.+// A mismatch in probe 1 only means the A (or D) layout differs; in probe 2 only B (or D): the printout says which.+// hipcc --offload-arch=gfx1201 -O2 wmma_probe.hip.cpp -o wmma_probe && HIP_VISIBLE_DEVICES=<card> ./wmma_probe+#include <hip/hip_runtime.h>+#include <cstdio>++typedef _Float16 half8_t __attribute__((ext_vector_type(8)));+typedef float float8_t __attribute__((ext_vector_type(8)));++__global__ void probe(float* out, int which) {+ const int l = threadIdx.x; // wave32+ half8_t a, b;+ for (int j = 0; j < 8; ++j) {+ const int m = l % 16, k = 8 * (l / 16) + j; // the layout under test, for A+ const int k2 = 8 * (l / 16) + j, n = l % 16; // for B+ a[j] = (_Float16) (which == 0 ? (float) (m * 16 + k) : (m == k ? 1.f : 0.f));+ b[j] = (_Float16) (which == 0 ? (k2 == n ? 1.f : 0.f) : (float) (k2 * 16 + n));+ }+ float8_t c = {0, 0, 0, 0, 0, 0, 0, 0};+ c = __builtin_amdgcn_wmma_f32_16x16x16_f16_w32_gfx12(a, b, c);+ for (int i = 0; i < 8; ++i) out[(which * 32 + l) * 8 + i] = c[i];+}++int main() {+ float* d; hipMalloc(&d, 2 * 32 * 8 * sizeof(float));+ probe<<<1, 32>>>(d, 0); probe<<<1, 32>>>(d, 1);+ float h[2 * 32 * 8]; hipMemcpy(h, d, sizeof h, hipMemcpyDeviceToHost);+ int bad[2] = {0, 0};+ for (int w = 0; w < 2; ++w)+ for (int l = 0; l < 32; ++l)+ for (int i = 0; i < 8; ++i) bad[w] += h[(w * 32 + l) * 8 + i] != (float) ((8 * (l / 16) + i) * 16 + (l % 16));+ std::printf("probe 1 (A layout, D layout): %d of 256 values wrong\nprobe 2 (B layout, D layout): %d of 256 wrong\n%s\n", bad[0], bad[1],+ (bad[0] || bad[1]) ? "LAYOUT DIFFERS: fix wmma_emu.hpp first (the emulator is the specification), then the kernel" : "layout as assumed");+ return bad[0] || bad[1];+}diff --git a/tools/dsv41_lab/wmma_emu.hpp b/tools/dsv41_lab/wmma_emu.hppnew file mode 100644index 000000000..968e4dd43--- /dev/null+++ b/tools/dsv41_lab/wmma_emu.hpp@@ -0,0 +1,168 @@+// tools/dsv41_lab/wmma_emu.hpp - host emulation of the gfx12 (RDNA4, R9700 = gfx1201) port of gemv_v3, written lane by lane exactly+// as the HIP kernel in hip/tqk_gemv_wmma.hip.cpp does it, so the data flow (layout "TQW", step -> k-slot map, window bits, shuffles, B+// fragments, D scaling) is checked on any machine against the double reference BEFORE it runs on the card. The only assumption+// that needs the card is the lane layout of v_wmma_f32_16x16x16_f16 (hip/wmma_probe.hip.cpp checks it in one launch).+//+// Assumed layout, wave32, __builtin_amdgcn_wmma_f32_16x16x16_f16_w32_gfx12(v8h a, v8h b, v8f c):+// A[m][k]: lane l holds a[j] = A[m = l % 16][k = 8 * (l / 16) + j], j = 0..7 (no replication between the lane halves)+// B[k][n]: lane l holds b[j] = B[k = 8 * (l / 16) + j][n = l % 16]+// D[m][n]: lane l holds c[i] = D[m = 8 * (l / 16) + i][n = l % 16], i = 0..7+#pragma once+#include "computed_codes.hpp"++#include <cstdint>+#include <cstring>+#include <vector>++namespace tqw {++struct Half8 { uint16_t h[8]; };+struct Float8 { float f[8]; };++// D = A * B + C on 32 lanes+inline void wmma_f32_16x16x16_f16(const Half8 a[32], const Half8 b[32], Float8 c[32]) {+ float A[16][16], B[16][16], D[16][16];+ for (int l = 0; l < 32; ++l)+ for (int j = 0; j < 8; ++j) {+ A[l % 16][8 * (l / 16) + j] = tqcode::h2f(a[l].h[j]);+ B[8 * (l / 16) + j][l % 16] = tqcode::h2f(b[l].h[j]);+ }+ for (int l = 0; l < 32; ++l)+ for (int i = 0; i < 8; ++i) D[8 * (l / 16) + i][l % 16] = c[l].f[i];+ for (int m = 0; m < 16; ++m)+ for (int n = 0; n < 16; ++n) {+ double s = D[m][n];+ for (int k = 0; k < 16; ++k) s += (double) A[m][k] * B[k][n];+ D[m][n] = (float) s;+ }+ for (int l = 0; l < 32; ++l)+ for (int i = 0; i < 8; ++i) c[l].f[i] = D[8 * (l / 16) + i][l % 16];+}++// ---- layout TQW (gfx12): a tile = 16 rows, a pair P = blocks 2P, 2P+1. Lane l = r + 16 kg of the wave owns row r and, of every+// block, the 16 steps u = 0..15 of the half kg (steps t = 16 kg + u): stream bits [(16 - 16 kg) K, (32 - 16 kg) K) of the block.+// Its two chunks (bk = 0, 1) are 32 K bits = K words: words[((tile * NP + P) * K + j) * 32 + lane].+// Scales: sc[((tile * NP + P) * 16 + r) * 2 + bk] (u16 fp16 d of row r, block 2P + bk): a lane reads the 8 rows 8 kg .. 8 kg + 7 = 32 bytes.+inline void repack_tqw(int K, const uint8_t* w, int rows, int nblk, std::vector<uint32_t>& words, std::vector<uint16_t>& sc) {+ const int bsz = 2 + 4 * K, NP = nblk / 2, NT = rows / 16;+ words.assign((size_t) NT * NP * K * 32, 0u);+ sc.assign((size_t) NT * NP * 32, 0);+ for (int t = 0; t < NT; ++t)+ for (int P = 0; P < NP; ++P) {+ const size_t base = (size_t) t * NP + P;+ for (int r = 0; r < 16; ++r)+ for (int bk = 0; bk < 2; ++bk) {+ const uint8_t* blk = w + ((size_t) (t * 16 + r) * nblk + 2 * P + bk) * bsz;+ std::memcpy(&sc[(base * 16 + r) * 2 + bk], blk, 2);+ for (int kg = 0; kg < 2; ++kg) {+ const int lane = r + 16 * kg;+ for (int b = 0; b < 16 * K; ++b) { // chunk bit b -> record bit 16 K bk + b+ const int src = (16 - 16 * kg) * K + b;+ const uint32_t bit = (blk[2 + (src >> 3)] >> (src & 7)) & 1u;+ const int dst = 16 * K * bk + b;+ words[(base * K + (dst >> 5)) * 32 + lane] |= bit << (dst & 31);+ }+ }+ }+ }+}++// B fragments (prep_b of the CUDA path, gfx12 form): for column n < nb, block blk, mma m, k-group kg: 8 halves hi and 8 halves lo of+// x[n][blk * 128 + 4 * (16 kg + 2 m) + j], j = 0..7 (hi = fp16(x), lo = fp16(x - hi))+inline uint16_t f2h_rn(float f) { // fp16 round to nearest even (host)+ uint32_t u; std::memcpy(&u, &f, 4);+ const uint32_t s = (u >> 16) & 0x8000u; int e = (int) ((u >> 23) & 0xff) - 127 + 15; uint32_t m = u & 0x7fffffu;+ if (((u >> 23) & 0xff) == 0xff) return (uint16_t) (s | 0x7c00u | (m ? 0x200u : 0));+ if (e >= 31) return (uint16_t) (s | 0x7c00u);+ if (e <= 0) {+ if (e < -10) return (uint16_t) s;+ m |= 0x800000u; const int sh = 14 - e; uint32_t hm = m >> sh; const uint32_t rem = m & ((1u << sh) - 1), half = 1u << (sh - 1);+ if (rem > half || (rem == half && (hm & 1))) ++hm;+ return (uint16_t) (s | hm);+ }+ uint32_t hm = m >> 13; const uint32_t rem = m & 0x1fffu;+ uint32_t v = ((uint32_t) e << 10) | hm;+ if (rem > 0x1000u || (rem == 0x1000u && (v & 1))) ++v;+ return (uint16_t) (s | v);+}++// the A fragment of a lane for one trellis step: (word 0, word 1) -> 4 fp16 (weights 4t .. 4t+3). computed code "3single":+// the two LCG words as they are. table: the two half2 entries of the hash.+using StepFn = void (*)(uint32_t state, uint32_t out[2]);++// The kernel, wave by wave. x: nb columns of F32 activations [nb][in]. Returns y[nb][rows].+inline void gemv_tqw(int K, int nb, int rows, int in, const std::vector<uint32_t>& words, const std::vector<uint16_t>& sc,+ const std::vector<float>& x, StepFn step, std::vector<float>& y, bool exact = true) {+ const int nblk = in / 128, NP = nblk / 2, NT = rows / 16;+ y.assign((size_t) nb * rows, 0.f);+ for (int t = 0; t < NT; ++t) {+ float tot[32][8] = {};+ for (int P = 0; P < NP; ++P) {+ const size_t base = (size_t) t * NP + P;+ // per lane: the record, the two chunk windows (8 steps each) with the partner bits+ uint64_t win[32][2][2]; // [lane][bk][half of the chunk: 0 = steps u 0..7, 1 = u 8..15]+ for (int lane = 0; lane < 32; ++lane) {+ uint32_t r[8];+ for (int j = 0; j < K; ++j) r[j] = words[(base * K + j) * 32 + lane];+ for (int bk = 0; bk < 2; ++bk) {+ // chunk = record bits [16 K bk, 16 K (bk + 1)), up to 80 bits: read as two 64-bit pieces+ auto bits = [&](int pos, int n) { // n <= 64 bits at record bit pos+ uint64_t v = 0;+ for (int i = 0; i < n; ++i) { const int p = pos + i; v |= (uint64_t) ((r[p >> 5] >> (p & 31)) & 1u) << i; }+ return v;+ };+ const uint64_t lo8 = bits(16 * K * bk, 8 * K); // chunk bits [0, 8K) -> steps u 8..15+ const uint64_t hi8 = bits(16 * K * bk + 8 * K, 8 * K); // chunk bits [8K, 16K) -> steps u 0..7+ win[lane][bk][1] = lo8 | (hi8 & ((1ull << (15 - K)) - 1ull)) << (8 * K); // u 8..15 end at bit 7K + 15 <= 16K - 1+ win[lane][bk][0] = hi8; // partner bits added below+ }+ }+ // partner bits: the low 15 - K bits of the chunk of lane ^ 16 (same bk)+ for (int lane = 0; lane < 32; ++lane)+ for (int bk = 0; bk < 2; ++bk) {+ const uint64_t pc = win[lane ^ 16][bk][1] & ((1ull << (8 * K)) - 1ull); // partner chunk bits [0, 8K) (its steps u 8..15)+ win[lane][bk][0] |= (pc & ((1ull << (15 - K)) - 1ull)) << (8 * K);+ }+ for (int bk = 0; bk < 2; ++bk) {+ Float8 acc[32] = {};+ for (int m = 0; m < 8; ++m) {+ Half8 a[32], bh[32], bl[32];+ for (int lane = 0; lane < 32; ++lane) {+ const int kg = lane / 16;+ for (int s2 = 0; s2 < 2; ++s2) {+ const int u = 2 * m + s2; // local step 0..15+ // u 0..7 live in window 0 at offset (7 - u) K, u 8..15 in window 1 at offset (15 - u) K+ const uint32_t st = (uint32_t) ((u < 8 ? win[lane][bk][0] >> ((7 - u) * K) : win[lane][bk][1] >> ((15 - u) * K))) & 0x7fffu;+ uint32_t w2[2];+ step(st, w2);+ a[lane].h[4 * s2 + 0] = (uint16_t) (w2[0] & 0xffffu); a[lane].h[4 * s2 + 1] = (uint16_t) (w2[0] >> 16);+ a[lane].h[4 * s2 + 2] = (uint16_t) (w2[1] & 0xffffu); a[lane].h[4 * s2 + 3] = (uint16_t) (w2[1] >> 16);+ }+ const int n = lane % 16;+ for (int j = 0; j < 8; ++j) {+ float xv = 0.f;+ if (n < nb) xv = x[(size_t) n * in + (2 * P + bk) * 128 + 4 * (16 * kg + 2 * m) + j];+ const uint16_t hi = f2h_rn(xv);+ bh[lane].h[j] = hi;+ bl[lane].h[j] = f2h_rn(xv - tqcode::h2f(hi));+ }+ }+ wmma_f32_16x16x16_f16(a, bh, acc);+ if (exact) wmma_f32_16x16x16_f16(a, bl, acc);+ }+ for (int lane = 0; lane < 32; ++lane)+ for (int i = 0; i < 8; ++i) {+ const int row = 8 * (lane / 16) + i;+ tot[lane][i] += tqcode::h2f(sc[(base * 16 + row) * 2 + bk]) * acc[lane].f[i];+ }+ }+ }+ for (int lane = 0; lane < 32; ++lane) {+ const int n = lane % 16;+ if (n >= nb) continue;+ for (int i = 0; i < 8; ++i) y[(size_t) n * rows + t * 16 + 8 * (lane / 16) + i] = tot[lane][i];+ }+ }+}++} // namespace tqwdiff --git a/tools/dsv41_lab/wmma_emu_test.cpp b/tools/dsv41_lab/wmma_emu_test.cppnew file mode 100644index 000000000..f0ad75edf--- /dev/null+++ b/tools/dsv41_lab/wmma_emu_test.cpp@@ -0,0 +1,64 @@+// tools/dsv41_lab/wmma_emu_test.cpp - the gfx12 port's data flow (wmma_emu.hpp) against the double reference on decoded weights.+// ./wmma_emu_test table codebook (shipped K5/K6/K7)+// TQE_CODE=3single ./wmma_emu_test computed code+// Host only; no GPU needed.+#include "../trellis/tqenc.hpp"+#include "wmma_emu.hpp"++#include <algorithm>+#include <cmath>+#include <cstdio>+#include <cstdlib>+#include <random>+#include <vector>++static void step_table(uint32_t s, uint32_t out[2]) {+ static std::vector<uint32_t> lut;+ if (lut.empty()) {+ const float* pt = tqenc::point_table();+ lut.resize(2048);+ for (int i = 0; i < 2048; ++i) lut[i] = tqw::f2h_rn(pt[2 * i]) | ((uint32_t) tqw::f2h_rn(pt[2 * i + 1]) << 16);+ }+ const uint32_t x = (s & 0x7fffu) * 0x9e3779b1u;+ out[0] = lut[x >> 21];+ out[1] = lut[(x >> 10) & 2047u];+}+static void step_3single(uint32_t s, uint32_t out[2]) { out[0] = tqcode::word3(s & 0x7fffu, 0); out[1] = tqcode::word3(s & 0x7fffu, 1); }++int main() {+ const bool computed = tqcode::kind_of(std::getenv("TQE_CODE")) == tqcode::k3single;+ if (std::getenv("TQE_CODE") && !computed) { std::printf("only TQE_CODE=3single is wired here\n"); return 2; }+ std::mt19937_64 rng(7);+ int fails = 0;+ const int Ks[4] = {4, 5, 6, 7}, types[4] = {225, 224, 145, 146};+ for (int ki = 0; ki < 4; ++ki) {+ const int K = Ks[ki], type = types[ki], bsz = 2 + 4 * K;+ const int rows = 48, in = 768, nblk = in / 128;+ std::vector<uint8_t> w((size_t) rows * nblk * bsz);+ for (auto& b : w) b = (uint8_t) rng();+ for (int64_t b = 0; b < (int64_t) rows * nblk; ++b) { const uint16_t d = tqw::f2h_rn(0.02f + 0.001f * (float) (rng() % 400)); std::memcpy(&w[(size_t) b * bsz], &d, 2); }+ std::vector<float> dec((size_t) rows * in);+ tqenc::decode_blocks(type, w.data(), (int64_t) rows * nblk, dec.data());+ std::vector<uint32_t> words; std::vector<uint16_t> sc;+ tqw::repack_tqw(K, w.data(), rows, nblk, words, sc);+ for (int nb : {1, 3, 5}) {+ std::vector<float> x((size_t) nb * in);+ std::uniform_real_distribution<float> u(-1.f, 1.f);+ for (auto& v : x) v = u(rng);+ std::vector<float> y;+ tqw::gemv_tqw(K, nb, rows, in, words, sc, x, computed ? step_3single : step_table, y);+ double worst = 0;+ for (int c = 0; c < nb; ++c)+ for (int r = 0; r < rows; ++r) {+ double s = 0, sa = 0;+ for (int i = 0; i < in; ++i) { const double t = (double) dec[(size_t) r * in + i] * x[(size_t) c * in + i]; s += t; sa += std::fabs(t); }+ worst = std::max(worst, std::fabs(y[(size_t) c * rows + r] - s) / (sa + 1e-30));+ }+ const bool ok = worst < 3e-7;+ std::printf("K=%d nb=%d (%s): emulated gfx12 flow vs double reference %.2e %s\n", K, nb, computed ? "3single" : "table", worst, ok ? "ok" : "FAIL");+ fails += !ok;+ }+ }+ std::printf("%s\n", fails ? "FAILED" : "ALL OK");+ return fails;+}