Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
qsa_pattn: sm_6x prompt attention without partials, opt-in STRATA_PATTN_SM60=1 (#3943)
ruibeikaaa33899018638
diff --git a/CMakeLists.txt b/CMakeLists.txtindex ed296bab1..ff5494d28 100644--- a/CMakeLists.txt+++ b/CMakeLists.txt@@ -430,6 +430,7 @@ if(STRATA_ENABLE_CUDA OR STRATA_ENABLE_HIP)src/kernels/cuda/fused_gdn.cusrc/kernels/cuda/dequant_bf16.cusrc/kernels/cuda/qsa_select.cu+ src/kernels/cuda/qsa_pattn.cusrc/kernels/cuda/verify_kernels.cusrc/kernels/cuda/dflash2.cusrc/kernels/cuda/iq_kernels.cu@@ -459,6 +460,16 @@ if(STRATA_ENABLE_CUDA OR STRATA_ENABLE_HIP)target_include_directories(qsa_prompt_attn_parity PRIVATE ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES})target_link_libraries(qsa_prompt_attn_parity PRIVATE strata_kernels CUDA::cudart)endif()+ # PR #3943: STRATA_PATTN_SM60's score and value kernels against the decode kernels' output (the Daily's chunk kernel and+ # the merge), bitwise, on random int8 K/V (GPU, synthetic, no model). Only in the experimental sm_60 build; it skips+ # (77) on a build without the kernels.+ if(STRATA_BUILD_TESTS AND STRATA_ENABLE_CUDA AND STRATA_EXPERIMENTAL_SM60)+ add_executable(qsa_pattn_parity src/kernels/qsa_pattn_parity.cpp)+ target_include_directories(qsa_pattn_parity PRIVATE ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES})+ target_link_libraries(qsa_pattn_parity PRIVATE strata_kernels CUDA::cudart)+ add_test(NAME qsa_pattn_parity COMMAND qsa_pattn_parity)+ set_tests_properties(qsa_pattn_parity PROPERTIES SKIP_RETURN_CODE 77)+ endif()# PR #600: the Volta (sm_70) prompt attention's latency pipeline against the FP32 kernel it replaces. Only in the# experimental sm_60/sm_70 build, where prompt_attn_v70_kernel is compiled in.if(STRATA_BUILD_TESTS AND STRATA_ENABLE_CUDA AND STRATA_EXPERIMENTAL_SM60)diff --git a/docs/OLDER_GPUS.md b/docs/OLDER_GPUS.mdindex 1ddd99a64..93a0aa7c3 100644--- a/docs/OLDER_GPUS.md+++ b/docs/OLDER_GPUS.md@@ -129,7 +129,7 @@ in [DETAILS.md](DETAILS.md#rtx-30-series-sm_86-and-tesla-p100-sm_60-bitwise-deco## Pascal GP100: a Q8_0 decode kernel (opt-in)GP100 (P100, PH402) is the one Pascal chip without `__dp4a`. Its dense projections read their weights at 40-80 GB/s of about-730, decoding K- and i-quant blocks one int at a time. Three opt-in changes, all measured on a PH402 only:+730, decoding K- and i-quant blocks one int at a time. Four opt-in changes, all measured on a PH402 only:- **`STRATA_Q8_SM60=1`** packs every Q8_0 dense matrix and the head into the planes of `STRATA_Q8_PACKED` and runs them on`src/kernels/cuda/q8_sm60.cuh`: a warp per row, 16-byte weight loads, activations copied once per workgroup into shared memory,@@ -147,6 +147,14 @@ GP100 (P100, PH402) is the one Pascal chip without `__dp4a`. Its dense projectioon one PH402 die, Swift 1.5 IQ3_S, one-token window: IQ4_NL 72.6 us (mode 8, the cc 6.0 default) and 54.5 us (mode 0) become42.3 us; Q2_0 53.5 / 65.4 us become 38.6 us. It replaces the down step in either expert mode. Not measured end to end on asingle card here.+- **`STRATA_PATTN_SM60=1`** (prompt reading with int8 K/V, CUDA build with `STRATA_EXPERIMENTAL_SM60`, compute capability 6.x): below+ sm_70 the prompt reads each QSA layer's attention through the decode kernels, 32 queries a call, with 817 KB of partial results a+ query and a merge kernel. The switch runs a score kernel and a value kernel that merges as it folds+ (`src/kernels/cuda/qsa_pattn.cu`), in the same scratch and with the same arithmetic in the same order, so the output matches the+ decode kernels bit for bit. If a launch fails or the shape does not fit, the usual loop runs. The author measured a 1,536-query+ chunk's attention at 43.3 -> 30.8 ms per QSA layer at 26K context on one GP100 die, and 2-3% less time on 8K-150K token prompts+ on four dies; we have not measured it here, and nobody has measured compute capability 6.1. `qsa_pattn_parity` (built with the+ tests in this build) checks the bytes on random data.None of these changes is measured on compute capability 6.1, which has `__dp4a`. Two practical notes for GP100 cards: the defaultapplication clock is 759 MHz and a layer split leaves each die idle most of a window, so lock the clocksdiff --git a/include/strata/kernels/qsa_pattn.hpp b/include/strata/kernels/qsa_pattn.hppnew file mode 100644index 000000000..3fd8de0ba--- /dev/null+++ b/include/strata/kernels/qsa_pattn.hpp@@ -0,0 +1,46 @@+// include/strata/kernels/qsa_pattn.hpp - STRATA_PATTN_SM60: the prompt path's QSA attention on sm_6x, bit for bit+// what prefill.cpp gets today from qsa_decode_attn_batch (attn_chunk_kernel_pre75<1, 2> + attn_merge_kernel or+// attn_merge_v2_kernel, 32 queries a call) with int8 K/V.+//+// On sm_6x qsa_prompt_attn_batch returns false (no tensor cores) and the prompt reads every QSA layer's attention+// through the decode kernels: one CTA per (64-cell chunk, KV head, query), the partials of all 33 chunks written to+// scratch (817 KB a query) and merged by a second kernel. The kernels here keep every operation of that path in its+// order - each lane's 8-dimension products, warp_sum's tree over the 32 lanes, the chunk softmax (warp_max / __expf /+// warp_sum pairs), the ascending fmaf value fold, the merge's chunk order (fmaxf, __expf(m - M), fmaf(l, w, L),+// fmaf(acc, w, acc), acc / L) - and change only where operands live and which thread does what:+// - ps_score_x_kernel, one CTA per (chunk, KV head, query): q in registers (loaded once in a per-lane slot order),+// the chunk's K staged with 16-byte loads, two cells at a time through one reduce-scatter whose 24 sums are+// warp_sum's tree without reduce12's selects; writes the chunk's probabilities, rows, m and l to the scratch;+// - ps_value_d4_kernel, one CTA per (KV head, query): four dims x 12 heads a thread, V four bytes a load, the chunks+// in ascending order, each merged into the output as the merge kernel does (no partial accumulators, no merge+// kernel).+// The caller's attention scratch holds the blocks of ~100 queries; a prompt chunk runs in batches of that size.+//+// Off unless STRATA_PATTN_SM60=1, and compiled only in the CUDA build with STRATA_EXPERIMENTAL_SM60 (every other build+// links stubs that answer false, so the usual attention runs).+#pragma once++#include "strata/kernels/qsa.hpp"+#include "strata/kernels/qsa_decode_attn.hpp"++#include <cstdint>++namespace strata::kernels {++/// STRATA_PATTN_SM60: 0 off (default, and in a build without the sm_6x kernels); 1 on.+int pattn_mode();++/// qsa_decode_attn_batch for all `n_q` queries of a prompt chunk in one call (int8 K/V, sm_6x, experimental build).+/// `scratch` / `scratch_floats`: the caller's attention scratch (prefill.cpp: attn_batch x qsa_decode_attn_scratch_floats).+/// False: not applicable or a launch failed (nothing the caller can rely on was written; it runs the usual loop).+bool qsa_pattn_batch(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, int64_t cap,+ const QsaShapes& s, float* scratch, uint64_t scratch_floats, float* attn, int64_t n_q,+ void* stream);++/// The same without the compute-capability test, for src/kernels/qsa_pattn_parity.cpp (it runs the kernels on any card+/// to compare them with the decode kernels' output). False in a build without the sm_6x kernels.+bool qsa_pattn_batch_any_arch(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps,+ int64_t cap, const QsaShapes& s, float* scratch, uint64_t scratch_floats, float* attn,+ int64_t n_q, void* stream);++} // namespace strata::kernelsdiff --git a/src/kernels/cuda/qsa_pattn.cu b/src/kernels/cuda/qsa_pattn.cunew file mode 100644index 000000000..0b1174a79--- /dev/null+++ b/src/kernels/cuda/qsa_pattn.cu@@ -0,0 +1,458 @@+// src/kernels/cuda/qsa_pattn.cu - see include/strata/kernels/qsa_pattn.hpp.+//+// Every value below is computed by the operation the production path uses, on the same operands, in the same order:+// - score of (cell, head): each lane l's partial k8[0] * q[8l] + ... + k8[7] * q[8l + 7] with k8[e] = (float) code * sc,+// the 32 partials summed in warp_sum's pairs (reduce12's tree), times `scale`;+// - chunk softmax: m = warp_max over fmaxf(s[lane], s[lane + 32]); e = __expf(s - m) (0 for a masked or absent cell);+// l = warp_sum(e[lane] + e[lane + 32]);+// - chunk value: acc[h] = fmaf(e[c][h], v, acc[h]) over the chunk's cells in ascending order, masked cells skipped,+// v = (float) code * sc;+// - merge (attn_merge_kernel): M = fmaxf over the chunks' m; for each chunk in order with m != -FLT_MAX:+// w = __expf(m - M), L = fmaf(l, w, L), acc = fmaf(acc_c, w, acc); out = L > 0 ? acc / L : 0.+// qsa_pattn_parity compares the outputs with qsa_decode_attn_batch's byte for byte on synthetic selections.+// Only the default variant of the contributor's PR (#3943) is kept: ps_score_x_kernel + ps_value_d4_kernel.+#include "strata/kernels/qsa_pattn.hpp"+#include "strata/kernels/kv_q8.hpp"+#include "strata/core/geometry.hpp"++#include <cuda_runtime.h>++#include <cfloat>+#include <cstdio>+#include <cstdlib>+#include <cstring>++#if !defined(__HIPCC__) && defined(STRATA_EXPERIMENTAL_SM60)+#define STRATA_PATTN_BUILT 1 // the CUDA build with STRATA_EXPERIMENTAL_SM60 only (not HIP, not Metal)+#include <cuda_fp16.h>+#else+#define STRATA_PATTN_BUILT 0+#endif++namespace strata::kernels {++// STRATA_PATTN_SM60=1 turns it on (a tester switch, read once). In a build without the sm_6x kernels it says so and stays off.+int pattn_mode() {+ static const int m = [] {+ const char* v = std::getenv("STRATA_PATTN_SM60");+ const int x = v != nullptr && *v != '\0' ? std::atoi(v) : 0;+ if (x == 0) return 0;+#if STRATA_PATTN_BUILT+ std::fprintf(stderr, "strata pattn: STRATA_PATTN_SM60=%d: the prompt path's QSA attention on the sm_6x score / "+ "value kernels (bit for bit qsa_decode_attn_batch; int8 K/V)\n", x);+ return 1;+#else+ std::fprintf(stderr, "strata pattn: STRATA_PATTN_SM60 is set, but this build has no sm_6x kernels "+ "(CUDA with STRATA_EXPERIMENTAL_SM60 only); the usual attention runs\n");+ return 0;+#endif+ }();+ return m;+}++#if STRATA_PATTN_BUILT+namespace {++constexpr int HD = 256;+constexpr int G = 12;+static_assert(HD == strata::core::kCompiledGeometry.head_dim &&+ G == strata::core::kCompiledGeometry.n_head / strata::core::kCompiledGeometry.n_head_kv,+ "qsa_pattn is compiled for core/geometry.hpp");+constexpr int CHUNK = 64;+constexpr int THREADS = 256;+constexpr int WARPS = THREADS / 32;+constexpr int NSC = HD / KV_Q8_GROUP; // scales per row (4)++__device__ __forceinline__ float warp_sum_x(float v) {+ v += __shfl_xor_sync(0xffffffffu, v, 16); v += __shfl_xor_sync(0xffffffffu, v, 8);+ v += __shfl_xor_sync(0xffffffffu, v, 4); v += __shfl_xor_sync(0xffffffffu, v, 2);+ v += __shfl_xor_sync(0xffffffffu, v, 1);+ return v;+}+__device__ __forceinline__ float warp_max_x(float v) {+ v = fmaxf(v, __shfl_xor_sync(0xffffffffu, v, 16)); v = fmaxf(v, __shfl_xor_sync(0xffffffffu, v, 8));+ v = fmaxf(v, __shfl_xor_sync(0xffffffffu, v, 4)); v = fmaxf(v, __shfl_xor_sync(0xffffffffu, v, 2));+ v = fmaxf(v, __shfl_xor_sync(0xffffffffu, v, 1));+ return v;+}++// pool row of cell `c0 + t` of query selection `ids` (attn_chunk_kernel_pre75's srow), -1 masked / absent+__device__ __forceinline__ long long cell_row(const QsaAttnPools& p, const int32_t* __restrict__ ids, int c0, int t,+ int n_here, int n_kv_heads, int kvh, int page_size) {+ long long r = -1;+ if (t < n_here) {+ const int cell = __ldg(ids + c0 + t);+ const long long page = (long long) __ldg(p.page_table + cell / page_size);+ if (page >= 0) r = (page * n_kv_heads + kvh) * page_size + (cell % page_size);+ }+ return r;+}++// the chunk's K codes (16-byte loads) and scales (as float) into shared memory; masked / absent cells read row 0+__device__ __forceinline__ void stage_rows(const int8_t* __restrict__ codes, const uint16_t* __restrict__ scales,+ const long long* srow, int n_here, int t, int8_t (*sk)[HD],+ float (*sks)[NSC]) {+ uint4 kv[4];+#pragma unroll+ for (int j = 0; j < 4; ++j) {+ const int idx = t + THREADS * j, c = idx >> 4, part = idx & 15;+ const long long row = srow[c];+ const long long rr = (c < n_here && row >= 0) ? row : 0;+ kv[j] = __ldg(reinterpret_cast<const uint4*>(codes + rr * HD) + part);+ }+ float sc;+ {+ const int c = t >> 2, g = t & 3;+ const long long row = srow[c];+ const long long rr = (c < n_here && row >= 0) ? row : 0;+ sc = __half2float(__ushort_as_half(__ldg(scales + rr * NSC + g)));+ }+#pragma unroll+ for (int j = 0; j < 4; ++j) reinterpret_cast<uint4*>(&sk[0][0])[t + THREADS * j] = kv[j];+ sks[t >> 2][t & 3] = sc;+}++constexpr int PS_BLK = G * CHUNK + 2 * CHUNK + 2 * G; // floats per (query, KV head, chunk): e, rows (int64), m, l+constexpr int NCM = 33; // chunks the value kernel keeps (cap <= 2112)++// ---------------------------------------------------------------------------------------------------------------------+// ps_score_x_kernel: the score phase two cells at a time with a select-free reduce-scatter. reduce12's selects exist+// because the two lanes of an exchange send different slots; here every lane keeps the same slots and sends the same+// slots, and the lane-dependent part is moved into which query head each register slot holds (q is loaded once, so the+// permutation costs nothing per cell) and which of the two cells is "first":+// slot i (0..11) of lane L holds head P_L(i) = 6 ((i / 6) ^ b3) + ((i % 6 + 3 b2) % 6) (b_k: bit k of L)+// first = cell B if b4 else cell A, second = the other one; part1[i] / part2[i] = lane partials of (first / second,+// head P_L(i)) - the same expression as attn_chunk_kernel_pre75.+// level 16: r[i] = part1[i] + shfl(part2[i], 16) (partner: b4 flipped, so its second is my first, same P)+// level 8 : r6[i] = r[i] + shfl(r[i + 6], 8) (P_{L^8}(i) = P_L(i + 6))+// level 4 : r3[i] = r6[i] + shfl(r6[i + 3], 4) (P_{L^4}(i) = P_L(i + 3), i < 3)+// level 2 / 1: reduce24's (heads 6 b3 + 3 b2 + {0, 1, 2}; o0 head +2 b1, o1 head +1 where b1 = 0)+// At every level the value at lane L is its partial at L plus its partial at L ^ o, the same operands warp_sum adds,+// so every score is reduce12's bit for bit. A pair with an absent / masked cell runs the same code on row 0's staged+// codes and that cell's outputs are not stored (it gets -FLT_MAX). 25 shuffles, 25 adds and 4 selects per pair+// instead of 2 x (15 + 15 + ~30).+__device__ __forceinline__ int px_head(int i, int lane) {+ const int b3 = (lane >> 3) & 1, b2 = (lane >> 2) & 1;+ return 6 * ((i / 6) ^ b3) + ((i % 6 + 3 * b2) % 6);+}+__device__ __forceinline__ void load_q_perm(const float* __restrict__ qh, int lane, float4 (&qa)[G], float4 (&qb)[G]) {+#pragma unroll+ for (int i = 0; i < G; ++i) {+ const float4* p4 = reinterpret_cast<const float4*>(qh + (size_t) px_head(i, lane) * HD + lane * 8);+ qa[i] = __ldg(p4);+ qb[i] = __ldg(p4 + 1);+ }+}+__device__ __forceinline__ void dequant8(const int8_t* krow, float sc, int lane, float (&k8)[8]) {+ const uint2 raw = *reinterpret_cast<const uint2*>(krow + lane * 8);+ const int8_t* cb = reinterpret_cast<const int8_t*>(&raw);+#pragma unroll+ for (int e = 0; e < 8; ++e) k8[e] = (float) cb[e] * sc;+}+__device__ __forceinline__ float dot8(const float (&k8)[8], const float4& qa, const float4& qb) {+ return k8[0] * qa.x + k8[1] * qa.y + k8[2] * qa.z + k8[3] * qa.w + k8[4] * qb.x + k8[5] * qb.y + k8[6] * qb.z ++ k8[7] * qb.w;+}+template <int UNR = 1>+__device__ __forceinline__ void score_chunk_x(const float4 (&qa)[G], const float4 (&qb)[G], const int8_t (*sk)[HD],+ const float (*sks)[NSC], const long long* srow, float (*sp)[CHUNK],+ int n_here, float scale, int warp, int lane) {+ const bool h16 = (lane & 16) != 0, h2 = (lane & 2) != 0, even = (lane & 1) == 0;+ const int hb = 6 * ((lane >> 3) & 1) + 3 * ((lane >> 2) & 1);+ const int h0 = hb + (h2 ? 2 : 0), h1 = hb + 1;+#pragma unroll UNR+ for (int i = 0; i < CHUNK / WARPS; i += 2) {+ const int ca = warp + WARPS * i, cb = warp + WARPS * (i + 1);+ const bool va = ca < n_here && srow[ca] >= 0, vb = cb < n_here && srow[cb] >= 0;+ if (!va && !vb) {+ if (lane < G) { sp[lane][ca] = -FLT_MAX; sp[lane][cb] = -FLT_MAX; }+ continue;+ }+ const int c1 = h16 ? cb : ca, c2 = h16 ? ca : cb; // first / second (the staged rows of absent cells are row 0's)+ float r[G];+ {+ float k8[8];+ dequant8(sk[c2], sks[c2][lane >> 3], lane, k8);+ float p2[G];+#pragma unroll+ for (int s = 0; s < G; ++s) p2[s] = dot8(k8, qa[s], qb[s]);+ dequant8(sk[c1], sks[c1][lane >> 3], lane, k8);+#pragma unroll+ for (int s = 0; s < G; ++s) r[s] = dot8(k8, qa[s], qb[s]) + __shfl_xor_sync(0xffffffffu, p2[s], 16);+ }+ float r6[6];+#pragma unroll+ for (int s = 0; s < 6; ++s) r6[s] = r[s] + __shfl_xor_sync(0xffffffffu, r[s + 6], 8);+ float r3[3];+#pragma unroll+ for (int s = 0; s < 3; ++s) r3[s] = r6[s] + __shfl_xor_sync(0xffffffffu, r6[s + 3], 4);+ const float x0 = (h2 ? r3[2] : r3[0]) + __shfl_xor_sync(0xffffffffu, h2 ? r3[0] : r3[2], 2);+ const float x1 = r3[1] + __shfl_xor_sync(0xffffffffu, r3[1], 2);+ const float o0 = x0 + __shfl_xor_sync(0xffffffffu, x0, 1);+ const float o1 = x1 + __shfl_xor_sync(0xffffffffu, x1, 1);+ const bool vmine = h16 ? vb : va;+ if (even && vmine) {+ sp[h0][c1] = o0 * scale;+ if (!h2) sp[h1][c1] = o1 * scale;+ }+ if (lane < G) {+ if (!va) sp[lane][ca] = -FLT_MAX;+ if (!vb) sp[lane][cb] = -FLT_MAX;+ }+ }+}++// ps_score_kernel with score_chunk_x (q loaded in the permuted slot order)+template <int UNR>+__global__ void __launch_bounds__(THREADS, 2) ps_score_x_kernel(const float* __restrict__ q, QsaAttnPools p,+ const int32_t* __restrict__ ids,+ const int32_t* __restrict__ step, int n_kv_heads,+ int page_size, float scale, float* __restrict__ blocks,+ int n_chunks, int cap) {+ q += (size_t) blockIdx.z * (size_t) (n_kv_heads * G) * HD;+ ids += (size_t) blockIdx.z * (size_t) cap;+ step += (size_t) blockIdx.z * kStepCount;+ const int chunk = blockIdx.x, kvh = blockIdx.y;+ float* const blk = blocks + (((size_t) blockIdx.z * n_kv_heads + kvh) * n_chunks + chunk) * PS_BLK;+ __shared__ __align__(16) int8_t sk[CHUNK][HD];+ __shared__ __align__(16) float sks[CHUNK][NSC];+ __shared__ float sp[G][CHUNK];+ __shared__ long long srow[CHUNK];+ const int n_ids = __ldg(step + kStepWidth);+ const int t = threadIdx.x, lane = t & 31, warp = t >> 5;+ const int c0 = chunk * CHUNK;+ const int n_here = min(CHUNK, n_ids - c0);+ if (n_here <= 0) {+ if (t < G) { blk[G * CHUNK + 2 * CHUNK + t] = -FLT_MAX; blk[G * CHUNK + 2 * CHUNK + G + t] = 0.0f; }+ return;+ }+ if (t < CHUNK) {+ const long long r = cell_row(p, ids, c0, t, n_here, n_kv_heads, kvh, page_size);+ srow[t] = r;+ reinterpret_cast<long long*>(blk + G * CHUNK)[t] = r;+ }+ float4 qa[G], qb[G];+ load_q_perm(q + (size_t) (kvh * G) * HD, lane, qa, qb);+ __syncthreads();+ stage_rows(p.k_q, p.k_scale, srow, n_here, t, sk, sks);+ __syncthreads();+ score_chunk_x<UNR>(qa, qb, sk, sks, srow, sp, n_here, scale, warp, lane);+ __syncthreads();+ for (int h = warp; h < G; h += WARPS) {+ const float a = sp[h][lane], b = sp[h][lane + 32];+ const float m = warp_max_x(fmaxf(a, b));+ const float ea = (lane < n_here && srow[lane] >= 0) ? __expf(a - m) : 0.0f;+ const float eb = (lane + 32 < n_here && srow[lane + 32] >= 0) ? __expf(b - m) : 0.0f;+ blk[h * CHUNK + lane] = ea;+ blk[h * CHUNK + lane + 32] = eb;+ const float l = warp_sum_x(ea + eb);+ if (lane == 0) { blk[G * CHUNK + 2 * CHUNK + h] = m; blk[G * CHUNK + 2 * CHUNK + G + h] = l; }+ }+}++// ps_value_d4_kernel: the value kernel with four dims a thread (64 threads, one CTA per (KV head, query)): a cell's four+// codes are one 32-bit load (a warp reads 128 contiguous bytes), its 12 weights three broadcast float4 reads for 48+// fmaf. V loads run four cells ahead. Per (dim, head) the fold and the merge are ps_value_kernel's.+__global__ void __launch_bounds__(64) ps_value_d4_kernel(QsaAttnPools p, const int32_t* __restrict__ steps,+ const float* __restrict__ blocks, int n_kv_heads, int n_chunks,+ float* __restrict__ attn) {+ constexpr int NT = 64;+ const int kvh = blockIdx.x, z = blockIdx.y;+ const int t = threadIdx.x;+ const float* const blk0 = blocks + ((size_t) z * n_kv_heads + kvh) * n_chunks * PS_BLK;+ __shared__ float sm_m[NCM][G], sm_w[NCM][G], sm_l[NCM][G];+ __shared__ float sL[G];+ __shared__ __align__(16) float spt[CHUNK][G];+ __shared__ long long srow[CHUNK];+ __shared__ int s_masked;+ const int width = __ldg(steps + (size_t) z * kStepCount + kStepWidth);+ const int n_act = width > 0 ? min(n_chunks, (width - 1) / CHUNK + 1) : 0;+ for (int i = t; i < n_act * G; i += NT) {+ const int c = i / G, h = i % G;+ sm_m[c][h] = __ldg(blk0 + (size_t) c * PS_BLK + G * CHUNK + 2 * CHUNK + h);+ sm_l[c][h] = __ldg(blk0 + (size_t) c * PS_BLK + G * CHUNK + 2 * CHUNK + G + h);+ }+ __syncthreads();+ if (t < G) {+ float M = -FLT_MAX;+ for (int c = 0; c < n_act; ++c) M = fmaxf(M, sm_m[c][t]);+ float L = 0.0f;+ for (int c = 0; c < n_act; ++c) {+ const float m = sm_m[c][t];+ const float w = __expf(m - M);+ sm_w[c][t] = w;+ const float L_next = fmaf(sm_l[c][t], w, L);+ L = (m == -FLT_MAX) ? L : L_next;+ }+ sL[t] = L;+ }+ const int d0 = 4 * t, g = d0 / KV_Q8_GROUP;+ float acc[4][G];+#pragma unroll+ for (int h = 0; h < G; ++h)+#pragma unroll+ for (int d = 0; d < 4; ++d) acc[d][h] = 0.0f;+ for (int c = 0; c < n_act; ++c) {+ const float* blk = blk0 + (size_t) c * PS_BLK;+ const int n_here = min(CHUNK, width - c * CHUNK);+ __syncthreads();+ if (t == 0) s_masked = 0;+ __syncthreads();+ {+ const long long r = __ldg(reinterpret_cast<const long long*>(blk + G * CHUNK) + t);+ srow[t] = r;+ if (t < n_here && r < 0) s_masked = 1;+ }+#pragma unroll+ for (int j = 0; j < G * CHUNK / NT; ++j) {+ const int idx = t + NT * j, h = idx / CHUNK, cell = idx % CHUNK;+ spt[cell][h] = __ldg(blk + idx);+ }+ __syncthreads();+ float a[4][G];+#pragma unroll+ for (int h = 0; h < G; ++h)+#pragma unroll+ for (int d = 0; d < 4; ++d) a[d][h] = 0.0f;+ const auto fold4 = [&](int cl, uint32_t codes, float sc) {+ const float v0 = (float) (int8_t) (codes & 0xFF) * sc, v1 = (float) (int8_t) ((codes >> 8) & 0xFF) * sc;+ const float v2 = (float) (int8_t) ((codes >> 16) & 0xFF) * sc, v3 = (float) (int8_t) (codes >> 24) * sc;+ const float4* w4 = reinterpret_cast<const float4*>(spt[cl]);+ const float4 w0 = w4[0], w1 = w4[1], w2 = w4[2];+ const float ww[G] = {w0.x, w0.y, w0.z, w0.w, w1.x, w1.y, w1.z, w1.w, w2.x, w2.y, w2.z, w2.w};+#pragma unroll+ for (int h = 0; h < G; ++h) {+ a[0][h] = fmaf(ww[h], v0, a[0][h]);+ a[1][h] = fmaf(ww[h], v1, a[1][h]);+ a[2][h] = fmaf(ww[h], v2, a[2][h]);+ a[3][h] = fmaf(ww[h], v3, a[3][h]);+ }+ };+ const auto vld = [&](int cl, uint32_t& codes, float& sc) {+ const long long r = srow[cl];+ const long long rr = r >= 0 ? r : 0;+ codes = __ldg(reinterpret_cast<const uint32_t*>(p.v_q + rr * HD + d0));+ sc = __half2float(__ushort_as_half(__ldg(p.v_scale + rr * NSC + g)));+ };+ if (!s_masked) {+ int cl = 0;+ for (; cl + 4 <= n_here; cl += 4) {+ uint32_t k0, k1, k2, k3;+ float s0, s1, s2, s3;+ vld(cl, k0, s0); vld(cl + 1, k1, s1); vld(cl + 2, k2, s2); vld(cl + 3, k3, s3);+ fold4(cl, k0, s0);+ fold4(cl + 1, k1, s1);+ fold4(cl + 2, k2, s2);+ fold4(cl + 3, k3, s3);+ }+ for (; cl < n_here; ++cl) {+ uint32_t k0;+ float s0;+ vld(cl, k0, s0);+ fold4(cl, k0, s0);+ }+ } else {+ for (int cl = 0; cl < n_here; ++cl) {+ if (srow[cl] < 0) continue;+ uint32_t k0;+ float s0;+ vld(cl, k0, s0);+ fold4(cl, k0, s0);+ }+ }+#pragma unroll+ for (int h = 0; h < G; ++h) {+ const bool live = !(sm_m[c][h] == -FLT_MAX);+ const float w = sm_w[c][h];+#pragma unroll+ for (int d = 0; d < 4; ++d) {+ const float acc_next = fmaf(a[d][h], w, acc[d][h]);+ acc[d][h] = live ? acc_next : acc[d][h];+ }+ }+ }+ __syncthreads();+ float* out = attn + ((size_t) z * n_kv_heads + kvh) * G * HD;+#pragma unroll+ for (int h = 0; h < G; ++h) {+ const float L = sL[h];+ *reinterpret_cast<float4*>(out + (size_t) h * HD + d0) =+ make_float4(L > 0.0f ? acc[0][h] / L : 0.0f, L > 0.0f ? acc[1][h] / L : 0.0f,+ L > 0.0f ? acc[2][h] / L : 0.0f, L > 0.0f ? acc[3][h] / L : 0.0f);+ }+}++bool sm6x_device() {+ static int cc[64] = {};+ int dev = 0;+ if (cudaGetDevice(&dev) != cudaSuccess || dev < 0 || dev >= 64) { cudaGetLastError(); return false; }+ if (cc[dev] == 0) {+ int major = 0;+ if (cudaDeviceGetAttribute(&major, cudaDevAttrComputeCapabilityMajor, dev) != cudaSuccess) {+ cudaGetLastError();+ return false;+ }+ cc[dev] = major;+ }+ return cc[dev] == 6;+}++} // namespace++bool qsa_pattn_batch(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, int64_t cap,+ const QsaShapes& s, float* scratch, uint64_t scratch_floats, float* attn, int64_t n_q,+ void* stream) {+ return sm6x_device() && qsa_pattn_batch_any_arch(q, pools, ids, steps, cap, s, scratch, scratch_floats, attn, n_q, stream);+}++bool qsa_pattn_batch_any_arch(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps,+ int64_t cap, const QsaShapes& s, float* scratch, uint64_t scratch_floats, float* attn,+ int64_t n_q, void* stream) {+ if (n_q <= 0) return true;+ if (s.head_dim != HD || s.n_head != (int64_t) G * s.n_head_kv || cap <= 0 || !scratch || !ids || !steps ||+ !pools.page_table || pools.k_q == nullptr || pools.v_q == nullptr || pools.k_q4 != nullptr ||+ pools.v_q4 != nullptr || !pools.k_scale || !pools.v_scale)+ return false;+ const cudaStream_t st = (cudaStream_t) stream;+ const int n_chunks = (int) ((cap + CHUNK - 1) / CHUNK);+ const float scale = 1.0f / sqrtf((float) HD);+ if (n_chunks > NCM) return false; // a longer selection keeps the usual path+ // the scratch holds the blocks of one batch of queries (probabilities, rows, m and l of every chunk), evenly split+ const uint64_t per_q = (uint64_t) s.n_head_kv * (uint64_t) n_chunks * PS_BLK;+ const int64_t max_batch = (int64_t) (scratch_floats / per_q);+ if (max_batch < 1) return false;+ const int64_t n_batches = (n_q + max_batch - 1) / max_batch;+ const int64_t batch = (n_q + n_batches - 1) / n_batches;+ for (int64_t t0 = 0; t0 < n_q; t0 += batch) {+ const int64_t nb = n_q - t0 < batch ? n_q - t0 : batch;+ const dim3 g1((unsigned) n_chunks, (unsigned) s.n_head_kv, (unsigned) nb);+ const dim3 g2((unsigned) s.n_head_kv, (unsigned) nb);+ const float* qb = q + (size_t) t0 * s.n_head * HD;+ const int32_t* ib = ids + (size_t) t0 * cap;+ const int32_t* sb = steps + (size_t) t0 * kStepCount;+ ps_score_x_kernel<1><<<g1, THREADS, 0, st>>>(qb, pools, ib, sb, (int) s.n_head_kv, (int) s.page_size, scale,+ scratch, n_chunks, (int) cap);+ ps_value_d4_kernel<<<g2, 64, 0, st>>>(pools, sb, scratch, (int) s.n_head_kv, n_chunks,+ attn + (size_t) t0 * s.n_head * HD);+ }+ const cudaError_t e = cudaGetLastError();+ if (e != cudaSuccess) {+ // a launch that did not start: say so once, and let the caller run the usual attention over the whole chunk+ static bool warned = false;+ if (!warned) {+ warned = true;+ std::fprintf(stderr, "strata pattn: launch failed (%s); the usual attention runs instead\n", cudaGetErrorString(e));+ }+ return false;+ }+ return true;+}++#else // not the experimental sm_60 CUDA build+bool qsa_pattn_batch(const float*, const QsaAttnPools&, const int32_t*, const int32_t*, int64_t, const QsaShapes&,+ float*, uint64_t, float*, int64_t, void*) { return false; }+bool qsa_pattn_batch_any_arch(const float*, const QsaAttnPools&, const int32_t*, const int32_t*, int64_t, const QsaShapes&,+ float*, uint64_t, float*, int64_t, void*) { return false; }+#endif++} // namespace strata::kernelsdiff --git a/src/kernels/qsa_pattn_parity.cpp b/src/kernels/qsa_pattn_parity.cppnew file mode 100644index 000000000..ab67038d8--- /dev/null+++ b/src/kernels/qsa_pattn_parity.cpp@@ -0,0 +1,260 @@+// src/kernels/qsa_pattn_parity.cpp - STRATA_PATTN_SM60's score and value kernels (qsa_pattn.cu, PR #3943) against the+// decode kernels' output, bitwise (GPU, synthetic data, no model).+//+// The reference is what the prompt path reads on sm_6x today: the chunk pass of attn_chunk_kernel_pre75 (the Daily's+// chunk kernel, qsa_decode_attn_chunks_only with daily_chunk 1, which runs on any card) and attn_merge_kernel+// (qsa_decode_attn_merge_only, v2 0). The candidate is qsa_pattn_batch_any_arch (no compute-capability test), so the+// same bytes can be checked on an Ampere card too: the two paths do the same operations in the same order, and a+// pass there shows the indexing, masking and merge are right. The last word is a run on a Pascal card (P100): there+// the checked entry qsa_pattn_batch must take the shapes and the production qsa_decode_attn_batch must give the same+// bytes as the reference.+//+// Data: random int8 K/V codes, fp16 scales in a safe range, normal queries, a shuffled page table with ~3% unresident+// (-1) pages and one run of 64 unresident pages (so whole chunks are masked and some queries read nothing), selections+// shaped like the drafter's window, the indexer's (ascending with gaps) and unordered with repeats, caps 64 / 130 /+// 2051 / 2112 (1, 3 and 33 chunks), widths 0 .. cap, n_q up to 40 with a scratch that holds only 2 queries of the+// decode layout (so qsa_pattn_batch runs several batches). The pattn scratch and the output are filled with+// junk before each call, so a read of a block that was not written shows up as a difference.+//+// qsa_pattn_parity [--device N] exit 0: every case is bit-identical; 77: no GPU or a build without the kernels+#include "strata/kernels/qsa.hpp"+#include "strata/kernels/qsa_decode_attn.hpp"+#include "strata/kernels/qsa_pattn.hpp"++#include <cuda_runtime.h>++#include <algorithm>+#include <cmath>+#include <cstdint>+#include <cstdio>+#include <cstdlib>+#include <cstring>+#include <random>+#include <vector>++namespace k = strata::kernels;++namespace {+void ck(cudaError_t e, const char* what) {+ if (e != cudaSuccess) {+ std::fprintf(stderr, "qsa_pattn_parity: %s: %s\n", what, cudaGetErrorString(e));+ std::exit(2);+ }+}+template <typename T> T* dalloc(size_t n) {+ T* p = nullptr;+ ck(cudaMalloc((void**) &p, n * sizeof(T) + 256), "cudaMalloc");+ ck(cudaMemset(p, 0, n * sizeof(T) + 256), "cudaMemset");+ return p;+}+template <typename T> T* upload(const std::vector<T>& h) {+ T* d = dalloc<T>(h.size());+ ck(cudaMemcpy(d, h.data(), h.size() * sizeof(T), cudaMemcpyHostToDevice), "upload");+ return d;+}+// positive fp16 scale in [1/64, 1/32): codes up to 127 give |x| < 4+uint16_t f16_scale_q8(uint32_t r) { return (uint16_t) ((9u << 10) | (r & 0x3FFu)); }++constexpr int64_t kCells = 40960; // the cells this harness can address+constexpr int64_t kHole = 8192; // 64 unresident pages in a row: cells [kHole, kHole + 256)++void selection(int32_t* ids, int64_t cap, int width, int shape, std::mt19937& rng) {+ const int64_t w = std::max<int64_t>(0, std::min<int64_t>(width, cap));+ if (shape == 0) { // the drafter's window: consecutive cells, often across the unresident run+ int64_t start;+ if (rng() % 2 == 0) start = std::max<int64_t>(0, kHole - 64 * (int64_t) (rng() % 8));+ else start = (int64_t) (rng() % (uint32_t) (kCells - std::max<int64_t>(w, 1)));+ if (start + w > kCells) start = kCells - w;+ for (int64_t j = 0; j < cap; ++j) ids[j] = (int32_t) ((start + j) % kCells);+ } else if (shape == 1) { // the indexer's: ascending with gaps+ std::vector<int32_t> v((size_t) cap);+ for (auto& x : v) x = (int32_t) (rng() % (uint32_t) kCells);+ std::sort(v.begin(), v.begin() + w);+ std::copy(v.begin(), v.end(), ids);+ } else { // unordered, with repeats+ for (int64_t j = 0; j < cap; ++j) ids[j] = (int32_t) (rng() % (uint32_t) kCells);+ }+}++struct Harness {+ k::QsaShapes s = k::qsa_real_shapes();+ k::QsaAttnPools pools;+ int32_t* table = nullptr;+ float *q = nullptr, *scratch_ref = nullptr, *scratch_new = nullptr, *attn_ref = nullptr, *attn_new = nullptr;+ int32_t *ids = nullptr, *steps = nullptr;+ int64_t max_q = 0, max_cap = 0;+ std::vector<void*> to_free;++ template <typename T> T* keep(T* p) { to_free.push_back(p); return p; }++ void init(std::mt19937& rng, int64_t max_q_, int64_t max_cap_) {+ max_q = max_q_;+ max_cap = max_cap_;+ const int64_t PS = s.page_size, pages = kCells / PS, H = s.n_head_kv, D = s.head_dim;+ std::vector<int32_t> tab((size_t) pages); // logical page -> physical page: a permutation, with holes+ for (int64_t i = 0; i < pages; ++i) tab[(size_t) i] = (int32_t) i;+ std::shuffle(tab.begin(), tab.end(), rng);+ for (int64_t i = 0; i < pages; ++i)+ if (rng() % 32u == 0) tab[(size_t) i] = -1;+ for (int64_t i = kHole / PS; i < (kHole + 256) / PS; ++i) tab[(size_t) i] = -1;+ table = keep(upload(tab));+ const int64_t rows = pages * H * PS;+ pools.page_table = table;+ std::vector<int8_t> c((size_t) (rows * D));+ for (auto& x : c) x = (int8_t) ((int) (rng() % 255u) - 127);+ pools.k_q = keep(upload(c));+ for (auto& x : c) x = (int8_t) ((int) (rng() % 255u) - 127);+ pools.v_q = keep(upload(c));+ std::vector<uint16_t> sc((size_t) (rows * (D / 64)));+ for (auto& x : sc) x = f16_scale_q8(rng());+ pools.k_scale = keep(upload(sc));+ for (auto& x : sc) x = f16_scale_q8(rng());+ pools.v_scale = keep(upload(sc));+ q = keep(dalloc<float>((size_t) (max_q * s.n_head * D)));+ ids = keep(dalloc<int32_t>((size_t) (max_q * max_cap)));+ steps = keep(dalloc<int32_t>((size_t) (max_q * k::kStepCount)));+ const size_t sf = (size_t) (k::qsa_decode_attn_scratch_floats(max_cap, s) * (uint64_t) max_q);+ scratch_ref = keep(dalloc<float>(sf));+ scratch_new = keep(dalloc<float>(sf));+ attn_ref = keep(dalloc<float>((size_t) (max_q * s.n_head * D)));+ attn_new = keep(dalloc<float>((size_t) (max_q * s.n_head * D)));+ }+ ~Harness() {+ for (void* p : to_free) cudaFree(p);+ }++ // returns 0 identical, 1 differs, -1 the candidate declined (build without the kernels)+ int run(int64_t cap, int n_q, int scr_q, const std::vector<int32_t>& widths, std::mt19937& rng, bool on_pascal,+ long long& nonzero) {+ const int64_t D = s.head_dim, NH = s.n_head;+ std::normal_distribution<float> nd(0.0f, 1.0f);+ std::vector<float> hq((size_t) (n_q * NH * D));+ for (auto& x : hq) x = nd(rng);+ ck(cudaMemcpy(q, hq.data(), hq.size() * 4, cudaMemcpyHostToDevice), "q");+ std::vector<int32_t> hid((size_t) (n_q * cap));+ std::vector<int32_t> hst((size_t) (n_q * k::kStepCount), 0);+ for (int i = 0; i < n_q; ++i) {+ selection(hid.data() + (size_t) i * cap, cap, widths[(size_t) i], (int) (rng() % 3u), rng);+ hst[(size_t) i * k::kStepCount + k::kStepWidth] = widths[(size_t) i];+ }+ ck(cudaMemcpy(ids, hid.data(), hid.size() * 4, cudaMemcpyHostToDevice), "ids");+ ck(cudaMemcpy(steps, hst.data(), hst.size() * 4, cudaMemcpyHostToDevice), "steps");+ const size_t n = (size_t) (n_q * NH * D);+ const size_t sbytes = (size_t) k::qsa_decode_attn_scratch_floats(cap, s) * (size_t) n_q * 4;++ // the reference: the pre-sm_75 chunk kernel and attn_merge_kernel+ ck(cudaMemset(scratch_ref, 0xFF, sbytes), "scratch_ref fill");+ ck(cudaMemset(attn_ref, 0xFF, n * 4), "attn_ref fill");+ k::qsa_decode_attn_chunks_only(q, pools, ids, steps, cap, s, scratch_ref, n_q, 1, nullptr);+ k::qsa_decode_attn_merge_only(steps, cap, s, scratch_ref, attn_ref, n_q, 0, nullptr);+ ck(cudaDeviceSynchronize(), "reference");+ std::vector<float> a(n), b(n);+ ck(cudaMemcpy(a.data(), attn_ref, n * 4, cudaMemcpyDeviceToHost), "read reference");+ nonzero = 0;+ for (size_t i = 0; i < n; ++i) {+ if (!std::isfinite(a[i])) { std::printf("FAIL: the reference has a non-finite value\n"); return 1; }+ if (a[i] != 0.0f) ++nonzero;+ }++ // the candidate: the scratch holds scr_q queries of the decode layout (the caller's attn_batch), junk in it+ ck(cudaMemset(scratch_new, 0xFF, sbytes), "scratch_new fill");+ ck(cudaMemset(attn_new, 0x7F, n * 4), "attn_new fill");+ const uint64_t scratch_floats = (uint64_t) scr_q * k::qsa_decode_attn_scratch_floats(cap, s);+ const bool took = k::qsa_pattn_batch_any_arch(q, pools, ids, steps, cap, s, scratch_new, scratch_floats, attn_new,+ n_q, nullptr);+ if (!took) return -1;+ ck(cudaDeviceSynchronize(), "candidate");+ ck(cudaMemcpy(b.data(), attn_new, n * 4, cudaMemcpyDeviceToHost), "read candidate");+ bool same = std::memcmp(a.data(), b.data(), n * 4) == 0;+ if (!same) {+ size_t first = 0;+ while (first < n && std::memcmp(&a[first], &b[first], 4) == 0) ++first;+ std::printf("FAIL cap %lld n_q %d scratch %d queries, widths", (long long) cap, n_q, scr_q);+ for (int32_t w : widths) std::printf(" %d", w);+ std::printf(": first difference at query %lld head %lld dim %lld: %.9g vs %.9g\n",+ (long long) (first / (NH * D)), (long long) (first / D % NH), (long long) (first % D), a[first],+ b[first]);+ }+ if (on_pascal) { // the checked entry takes the shapes, and the production path is the reference+ ck(cudaMemset(scratch_new, 0xFF, sbytes), "scratch_new refill");+ ck(cudaMemset(attn_new, 0x7F, n * 4), "attn_new refill");+ if (!k::qsa_pattn_batch(q, pools, ids, steps, cap, s, scratch_new, scratch_floats, attn_new, n_q, nullptr)) {+ std::printf("FAIL: qsa_pattn_batch declined on a compute capability 6 card (cap %lld n_q %d)\n",+ (long long) cap, n_q);+ same = false;+ } else {+ ck(cudaDeviceSynchronize(), "checked entry");+ ck(cudaMemcpy(b.data(), attn_new, n * 4, cudaMemcpyDeviceToHost), "read checked entry");+ if (std::memcmp(a.data(), b.data(), n * 4) != 0) {+ std::printf("FAIL: qsa_pattn_batch differs from the reference (cap %lld n_q %d)\n", (long long) cap, n_q);+ same = false;+ }+ }+ ck(cudaMemset(scratch_new, 0xFF, sbytes), "scratch_new refill");+ k::qsa_decode_attn_batch(q, pools, ids, steps, cap, s, scratch_new, attn_new, n_q, nullptr);+ ck(cudaDeviceSynchronize(), "production");+ ck(cudaMemcpy(b.data(), attn_new, n * 4, cudaMemcpyDeviceToHost), "read production");+ if (std::memcmp(a.data(), b.data(), n * 4) != 0) {+ std::printf("FAIL: qsa_decode_attn_batch differs from the reference (cap %lld n_q %d)\n", (long long) cap,+ n_q);+ same = false;+ }+ }+ return same ? 0 : 1;+ }+};+} // namespace++int main(int argc, char** argv) {+ int device = 0;+ for (int i = 1; i < argc; ++i) {+ if (std::strcmp(argv[i], "--device") == 0 && i + 1 < argc) device = std::atoi(argv[++i]);+ else {+ std::fprintf(stderr, "usage: qsa_pattn_parity [--device N]\n");+ return 2;+ }+ }+ int devices = 0;+ if (cudaGetDeviceCount(&devices) != cudaSuccess || devices == 0) {+ std::puts("SKIP: no GPU");+ return 77;+ }+ ck(cudaSetDevice(device), "cudaSetDevice");+ cudaDeviceProp prop{};+ ck(cudaGetDeviceProperties(&prop, device), "cudaGetDeviceProperties");+ const bool on_pascal = prop.major == 6;+ std::printf("qsa_pattn_parity on device %d (%s, %d.%d)%s\n", device, prop.name, prop.major, prop.minor,+ on_pascal ? ": the checked entry and the production path are compared too" : "");++ std::mt19937 rng(20261026u);+ Harness hx;+ hx.init(rng, 40, 2112);++ const int32_t base_widths[] = {0, 1, 63, 64, 65, 1023, 1024, 1025, 2051, 2112};+ const int n_base = (int) (sizeof(base_widths) / sizeof(base_widths[0]));+ long long cases = 0, failed = 0, nonzero_total = 0;+ for (int64_t cap : {(int64_t) 64, (int64_t) 130, (int64_t) 2051, (int64_t) 2112}) {+ for (int n_q : {1, 2, 3, 7, 40}) {+ for (int scr_q : {n_q, 2}) {+ if (scr_q > n_q) continue;+ for (int wc = 0; wc < n_base + 3; ++wc) {+ std::vector<int32_t> widths;+ for (int i = 0; i < n_q; ++i)+ widths.push_back(std::min<int32_t>((int32_t) cap, wc < n_base ? base_widths[wc] : base_widths[rng() % (uint32_t) n_base]));+ long long nz = 0;+ const int r = hx.run(cap, n_q, scr_q, widths, rng, on_pascal, nz);+ if (r < 0) {+ std::puts("SKIP: this build has no sm_6x kernels (CUDA with STRATA_EXPERIMENTAL_SM60)");+ return 77;+ }+ nonzero_total += nz;+ ++cases;+ if (r != 0) ++failed;+ }+ }+ }+ }+ std::printf("qsa_pattn_parity: %lld cases, %lld differ, %lld nonzero output values in all -> %s\n", cases, failed,+ nonzero_total, failed == 0 && nonzero_total > 0 ? "PASS" : "FAIL");+ return failed == 0 && nonzero_total > 0 ? 0 : 1;+}diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cppindex 08b529e6f..a9d121f94 100644--- a/src/prefill/prefill.cpp+++ b/src/prefill/prefill.cpp@@ -38,6 +38,7 @@#include "strata/core/native_head.hpp"#include "strata/kernels/verify_kernels.hpp"#include "strata/kernels/qsa_decode_attn.hpp"+#include "strata/kernels/qsa_pattn.hpp"#include "strata/kernels/qsa_prompt_attn.hpp"#include "strata/kernels/qsa_select.hpp"#include "strata/prefill/gemm.hpp"@@ -3723,8 +3724,18 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri// decode kernel, 32 queries at a time (K8V4 runs the tensor kernel's mode 3: INT8 K,// V dequantized from its q4_0 blocks to fp16 at gather)static const bool old_attn = std::getenv("STRATA_PROMPT_ATTN_OLD") != nullptr;+ // STRATA_PATTN_SM60=1 (qsa_pattn.hpp, sm_6x, int8 K/V): the decode kernels' output bit for bit from a+ // score kernel and a value kernel that merges as it folds (no partials, no merge kernel), ~100+ // queries a batch in the same scratch; any refusal or launch failure runs the loop belowif (old_attn || !strata::kernels::qsa_prompt_attn_batch(m.q, pools, m.sel_ids, m.steps_dev, m.cap, s,m.attn, T, m.cs))+#if !defined(STRATA_GPU_METAL) // qsa_pattn.cu is in the CUDA / HIP kernel library only+ if (!(strata::kernels::pattn_mode() != 0 &&+ strata::kernels::qsa_pattn_batch(m.q, pools, m.sel_ids, m.steps_dev, m.cap, s, m.attn_scratch,+ (uint64_t) m.attn_batch *+ strata::kernels::qsa_decode_attn_scratch_floats(m.cap, s),+ m.attn, T, m.cs)))+#endiffor (int64_t t0 = 0; t0 < T; t0 += m.attn_batch) {const int64_t nb = std::min(m.attn_batch, T - t0);strata::kernels::qsa_decode_attn_batch(m.q + t0 * ZV, pools, m.sel_ids + t0 * m.cap,