Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
HIP port of the trellis GEMV (gfx1201, R9700) - recipe
State: wmma_emu.hpp (host, lane by lane) reproduces the double reference to 3e-8 for K4..K7, widths 1/3/5, table codebook and the
computed code 3single (wmma_emu_test, no GPU needed). tqk_gemv_wmma.hip.cpp is its line-by-line HIP form. It was not
compiled or run anywhere. Do the steps in this order; each one is a hard stop if it fails.
- Layout probe (30 s).
hipcc --offload-arch=gfx1201 -O2 hip/wmma_probe.hip.cpp -o wmma_probe && HIP_VISIBLE_DEVICES=<n> ./wmma_probe. It must printlayout as assumed. The assumption (wave32,__builtin_amdgcn_wmma_f32_16x16x16_f16_w32_gfx12, 8 halves per lane for A and B, 8 floats for D, no replication between the two lane halves):A[m][k]in lanel, elementj:m = l % 16, k = 8 * (l / 16) + j;B[k][n]:k = 8 * (l / 16) + j, n = l % 16;D[m][n]:m = 8 * (l / 16) + i, n = l % 16. If it does not match, changewmma_emu.hpp(the specification) andwmma_emu_testuntil it passes with the layout the probe found, then mirror the change in the kernel. (gfx11 replicated A/B between the halves and striped D by2 i + l / 16; gfx12 does not, as far as the LLVM documentation says, but nobody here has measured it.) - Emulator test:
g++ -O2 -std=c++17 -I third_party/ggml tools/dsv41_lab/wmma_emu_test.cpp tools/trellis/tqenc.cpp -o wmma_emu_test(and withTQE_CODE=3single). - Kernel: the data flow, all of it in
wmma_emu.hpp::gemv_tqw:- Why a new layout. mma.m16n8k16 gives a lane 2 rows of A and 4 floats of D; WMMA gives a lane 1 row of A (8 consecutive k) and 8 rows of D.
So lane
l = r + 16 kgowns rowrof the tile and 16 consecutive steps (u = 0..15, stepst = 16 kg + u) of every 128-weight block: stream bits[(16 - 16 kg) K, (32 - 16 kg) K). The two blocks of a pair give 2 chunks of16 Kbits =Kwords per lane (words[((tile * NP + P) * K + j) * 32 + lane], coalesced). Scalessc[((tile*NP+P)*16 + row)*2 + bk]: a lane reads rows8 kg..8 kg+7. The bytes are the same as the file layout; the permutation is done at load (a bit shuffle per lane,repack_tqw). - k order. WMMA k-slot
(kg, j)of mmam= weight4 (16 kg + 2 m) + jof the block (stepsu = 2m, 2m+1of the lane). A and B must use the same map; B lane(n, kg)loadsx[n][blk*128 + 4(16 kg + 2m) + j],j = 0..7(2 float4), splitx = hi + loin fp16 (two WMMAs, error 1e-7).prep_b_tqwdoes this once per call into 32 contiguous bytes per (block, m, kg, column). - States. local step
uis at chunk-relative bit(15 - u) K. Window 0 (u 0..7) = chunk bits[8K, 16K)+ the low15 - Kbits of the partner lane's chunk (lane ^ 16, same row and block; circular wrap: for kg = 0 the partner is kg = 1) - one__shfl_xor(x, 16); window 1 (u 8..15) = chunk bits[0, 7K + 15).state(u) = (window >> offset) & 0x7fff. Both windows fit 64 bits for K <= 7. - A fragment. computed code
3single:w0 = ((s*89226354 + 64248484) & 0x8FFF8FFF) ^ 0x3B603B60,w1 = ((s*0x9E3779B1 + 0x7F4A7C15) & ...) ^ ...; the four half values of a step arelo(w0), hi(w0), lo(w1), hi(w1)= 2 words of the 8-half A vector. Table codebook:x = s*0x9e3779b1, wordslut[x >> 21],lut[(x >> 10) & 2047]from an 8 KB LDS table of half2 (the table is exact in fp16: 0 inexact entries). - Accumulate. D (8 floats) after the 8 (16 with lo) WMMAs of a block times the row scale (
fp16 d, 8 rows per lane) is added to the runningtot[8]; at the end lane(n = l % 16 < NB, kg)stores rows8 kg .. 8 kg + 7of columnn. - Wave32 (the gfx12 default); 128-thread blocks (4 waves),
T = 2tiles per wave first; VGPR budget: B fragments of one block half (8 m x 8 regs = 64), recordsT K, windows, A (4), D (8) - fits well under 256.
- Why a new layout. mma.m16n8k16 gives a lane 2 rows of A and 4 floats of D; WMMA gives a lane 1 row of A (8 consecutive k) and 8 rows of D.
So lane
- Compare with the emulator on the same random streams; the CUDA harness (
tqk_lab.cu) has the benchmark shape (12 gate/up + 6 down matrices) to copy. Speed targets on gfx1201: K5/3single at 640 GB/s is a 3.7 Tw/s roof; WMMA hi+lo costs 2 x 8 WMMAs per 2048 weights per wave, the ALU side (2 IMAD + 2 LOP3 a step) is light, so expect the memory system to be the limit - the opposite of the A4000, where the tensor pipe is at 75 percent. - Not decided here: whether the R9700 build should keep the table codebook (K6/K7 are byte-compatible with agentionai and must) or the computed code (K4/K5 new types) - see notes-dsv41-cudalab.md.