Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

qsa_pattn: sm_6x prompt attention without partials, opt-in STRATA_PATTN_SM60=1 (#3943)

ruibeikaaa33899018638
diff --git a/CMakeLists.txt b/CMakeLists.txt
index ed296bab1..ff5494d28 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -430,6 +430,7 @@ if(STRATA_ENABLE_CUDA OR STRATA_ENABLE_HIP)
src/kernels/cuda/fused_gdn.cu
src/kernels/cuda/dequant_bf16.cu
src/kernels/cuda/qsa_select.cu
+ src/kernels/cuda/qsa_pattn.cu
src/kernels/cuda/verify_kernels.cu
src/kernels/cuda/dflash2.cu
src/kernels/cuda/iq_kernels.cu
@@ -459,6 +460,16 @@ if(STRATA_ENABLE_CUDA OR STRATA_ENABLE_HIP)
target_include_directories(qsa_prompt_attn_parity PRIVATE ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES})
target_link_libraries(qsa_prompt_attn_parity PRIVATE strata_kernels CUDA::cudart)
endif()
+ # PR #3943: STRATA_PATTN_SM60's score and value kernels against the decode kernels' output (the Daily's chunk kernel and
+ # the merge), bitwise, on random int8 K/V (GPU, synthetic, no model). Only in the experimental sm_60 build; it skips
+ # (77) on a build without the kernels.
+ if(STRATA_BUILD_TESTS AND STRATA_ENABLE_CUDA AND STRATA_EXPERIMENTAL_SM60)
+ add_executable(qsa_pattn_parity src/kernels/qsa_pattn_parity.cpp)
+ target_include_directories(qsa_pattn_parity PRIVATE ${CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES})
+ target_link_libraries(qsa_pattn_parity PRIVATE strata_kernels CUDA::cudart)
+ add_test(NAME qsa_pattn_parity COMMAND qsa_pattn_parity)
+ set_tests_properties(qsa_pattn_parity PROPERTIES SKIP_RETURN_CODE 77)
+ endif()
# PR #600: the Volta (sm_70) prompt attention's latency pipeline against the FP32 kernel it replaces. Only in the
# experimental sm_60/sm_70 build, where prompt_attn_v70_kernel is compiled in.
if(STRATA_BUILD_TESTS AND STRATA_ENABLE_CUDA AND STRATA_EXPERIMENTAL_SM60)
diff --git a/docs/OLDER_GPUS.md b/docs/OLDER_GPUS.md
index 1ddd99a64..93a0aa7c3 100644
--- a/docs/OLDER_GPUS.md
+++ b/docs/OLDER_GPUS.md
@@ -129,7 +129,7 @@ in [DETAILS.md](DETAILS.md#rtx-30-series-sm_86-and-tesla-p100-sm_60-bitwise-deco
## Pascal GP100: a Q8_0 decode kernel (opt-in)
GP100 (P100, PH402) is the one Pascal chip without `__dp4a`. Its dense projections read their weights at 40-80 GB/s of about
-730, decoding K- and i-quant blocks one int at a time. Three opt-in changes, all measured on a PH402 only:
+730, decoding K- and i-quant blocks one int at a time. Four opt-in changes, all measured on a PH402 only:
- **`STRATA_Q8_SM60=1`** packs every Q8_0 dense matrix and the head into the planes of `STRATA_Q8_PACKED` and runs them on
`src/kernels/cuda/q8_sm60.cuh`: a warp per row, 16-byte weight loads, activations copied once per workgroup into shared memory,
@@ -147,6 +147,14 @@ GP100 (P100, PH402) is the one Pascal chip without `__dp4a`. Its dense projectio
on one PH402 die, Swift 1.5 IQ3_S, one-token window: IQ4_NL 72.6 us (mode 8, the cc 6.0 default) and 54.5 us (mode 0) become
42.3 us; Q2_0 53.5 / 65.4 us become 38.6 us. It replaces the down step in either expert mode. Not measured end to end on a
single card here.
+- **`STRATA_PATTN_SM60=1`** (prompt reading with int8 K/V, CUDA build with `STRATA_EXPERIMENTAL_SM60`, compute capability 6.x): below
+ sm_70 the prompt reads each QSA layer's attention through the decode kernels, 32 queries a call, with 817 KB of partial results a
+ query and a merge kernel. The switch runs a score kernel and a value kernel that merges as it folds
+ (`src/kernels/cuda/qsa_pattn.cu`), in the same scratch and with the same arithmetic in the same order, so the output matches the
+ decode kernels bit for bit. If a launch fails or the shape does not fit, the usual loop runs. The author measured a 1,536-query
+ chunk's attention at 43.3 -> 30.8 ms per QSA layer at 26K context on one GP100 die, and 2-3% less time on 8K-150K token prompts
+ on four dies; we have not measured it here, and nobody has measured compute capability 6.1. `qsa_pattn_parity` (built with the
+ tests in this build) checks the bytes on random data.
None of these changes is measured on compute capability 6.1, which has `__dp4a`. Two practical notes for GP100 cards: the default
application clock is 759 MHz and a layer split leaves each die idle most of a window, so lock the clocks
diff --git a/include/strata/kernels/qsa_pattn.hpp b/include/strata/kernels/qsa_pattn.hpp
new file mode 100644
index 000000000..3fd8de0ba
--- /dev/null
+++ b/include/strata/kernels/qsa_pattn.hpp
@@ -0,0 +1,46 @@
+// include/strata/kernels/qsa_pattn.hpp - STRATA_PATTN_SM60: the prompt path's QSA attention on sm_6x, bit for bit
+// what prefill.cpp gets today from qsa_decode_attn_batch (attn_chunk_kernel_pre75<1, 2> + attn_merge_kernel or
+// attn_merge_v2_kernel, 32 queries a call) with int8 K/V.
+//
+// On sm_6x qsa_prompt_attn_batch returns false (no tensor cores) and the prompt reads every QSA layer's attention
+// through the decode kernels: one CTA per (64-cell chunk, KV head, query), the partials of all 33 chunks written to
+// scratch (817 KB a query) and merged by a second kernel. The kernels here keep every operation of that path in its
+// order - each lane's 8-dimension products, warp_sum's tree over the 32 lanes, the chunk softmax (warp_max / __expf /
+// warp_sum pairs), the ascending fmaf value fold, the merge's chunk order (fmaxf, __expf(m - M), fmaf(l, w, L),
+// fmaf(acc, w, acc), acc / L) - and change only where operands live and which thread does what:
+// - ps_score_x_kernel, one CTA per (chunk, KV head, query): q in registers (loaded once in a per-lane slot order),
+// the chunk's K staged with 16-byte loads, two cells at a time through one reduce-scatter whose 24 sums are
+// warp_sum's tree without reduce12's selects; writes the chunk's probabilities, rows, m and l to the scratch;
+// - ps_value_d4_kernel, one CTA per (KV head, query): four dims x 12 heads a thread, V four bytes a load, the chunks
+// in ascending order, each merged into the output as the merge kernel does (no partial accumulators, no merge
+// kernel).
+// The caller's attention scratch holds the blocks of ~100 queries; a prompt chunk runs in batches of that size.
+//
+// Off unless STRATA_PATTN_SM60=1, and compiled only in the CUDA build with STRATA_EXPERIMENTAL_SM60 (every other build
+// links stubs that answer false, so the usual attention runs).
+#pragma once
+
+#include "strata/kernels/qsa.hpp"
+#include "strata/kernels/qsa_decode_attn.hpp"
+
+#include <cstdint>
+
+namespace strata::kernels {
+
+/// STRATA_PATTN_SM60: 0 off (default, and in a build without the sm_6x kernels); 1 on.
+int pattn_mode();
+
+/// qsa_decode_attn_batch for all `n_q` queries of a prompt chunk in one call (int8 K/V, sm_6x, experimental build).
+/// `scratch` / `scratch_floats`: the caller's attention scratch (prefill.cpp: attn_batch x qsa_decode_attn_scratch_floats).
+/// False: not applicable or a launch failed (nothing the caller can rely on was written; it runs the usual loop).
+bool qsa_pattn_batch(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, int64_t cap,
+ const QsaShapes& s, float* scratch, uint64_t scratch_floats, float* attn, int64_t n_q,
+ void* stream);
+
+/// The same without the compute-capability test, for src/kernels/qsa_pattn_parity.cpp (it runs the kernels on any card
+/// to compare them with the decode kernels' output). False in a build without the sm_6x kernels.
+bool qsa_pattn_batch_any_arch(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps,
+ int64_t cap, const QsaShapes& s, float* scratch, uint64_t scratch_floats, float* attn,
+ int64_t n_q, void* stream);
+
+} // namespace strata::kernels
diff --git a/src/kernels/cuda/qsa_pattn.cu b/src/kernels/cuda/qsa_pattn.cu
new file mode 100644
index 000000000..0b1174a79
--- /dev/null
+++ b/src/kernels/cuda/qsa_pattn.cu
@@ -0,0 +1,458 @@
+// src/kernels/cuda/qsa_pattn.cu - see include/strata/kernels/qsa_pattn.hpp.
+//
+// Every value below is computed by the operation the production path uses, on the same operands, in the same order:
+// - score of (cell, head): each lane l's partial k8[0] * q[8l] + ... + k8[7] * q[8l + 7] with k8[e] = (float) code * sc,
+// the 32 partials summed in warp_sum's pairs (reduce12's tree), times `scale`;
+// - chunk softmax: m = warp_max over fmaxf(s[lane], s[lane + 32]); e = __expf(s - m) (0 for a masked or absent cell);
+// l = warp_sum(e[lane] + e[lane + 32]);
+// - chunk value: acc[h] = fmaf(e[c][h], v, acc[h]) over the chunk's cells in ascending order, masked cells skipped,
+// v = (float) code * sc;
+// - merge (attn_merge_kernel): M = fmaxf over the chunks' m; for each chunk in order with m != -FLT_MAX:
+// w = __expf(m - M), L = fmaf(l, w, L), acc = fmaf(acc_c, w, acc); out = L > 0 ? acc / L : 0.
+// qsa_pattn_parity compares the outputs with qsa_decode_attn_batch's byte for byte on synthetic selections.
+// Only the default variant of the contributor's PR (#3943) is kept: ps_score_x_kernel + ps_value_d4_kernel.
+#include "strata/kernels/qsa_pattn.hpp"
+#include "strata/kernels/kv_q8.hpp"
+#include "strata/core/geometry.hpp"
+
+#include <cuda_runtime.h>
+
+#include <cfloat>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+
+#if !defined(__HIPCC__) && defined(STRATA_EXPERIMENTAL_SM60)
+#define STRATA_PATTN_BUILT 1 // the CUDA build with STRATA_EXPERIMENTAL_SM60 only (not HIP, not Metal)
+#include <cuda_fp16.h>
+#else
+#define STRATA_PATTN_BUILT 0
+#endif
+
+namespace strata::kernels {
+
+// STRATA_PATTN_SM60=1 turns it on (a tester switch, read once). In a build without the sm_6x kernels it says so and stays off.
+int pattn_mode() {
+ static const int m = [] {
+ const char* v = std::getenv("STRATA_PATTN_SM60");
+ const int x = v != nullptr && *v != '\0' ? std::atoi(v) : 0;
+ if (x == 0) return 0;
+#if STRATA_PATTN_BUILT
+ std::fprintf(stderr, "strata pattn: STRATA_PATTN_SM60=%d: the prompt path's QSA attention on the sm_6x score / "
+ "value kernels (bit for bit qsa_decode_attn_batch; int8 K/V)\n", x);
+ return 1;
+#else
+ std::fprintf(stderr, "strata pattn: STRATA_PATTN_SM60 is set, but this build has no sm_6x kernels "
+ "(CUDA with STRATA_EXPERIMENTAL_SM60 only); the usual attention runs\n");
+ return 0;
+#endif
+ }();
+ return m;
+}
+
+#if STRATA_PATTN_BUILT
+namespace {
+
+constexpr int HD = 256;
+constexpr int G = 12;
+static_assert(HD == strata::core::kCompiledGeometry.head_dim &&
+ G == strata::core::kCompiledGeometry.n_head / strata::core::kCompiledGeometry.n_head_kv,
+ "qsa_pattn is compiled for core/geometry.hpp");
+constexpr int CHUNK = 64;
+constexpr int THREADS = 256;
+constexpr int WARPS = THREADS / 32;
+constexpr int NSC = HD / KV_Q8_GROUP; // scales per row (4)
+
+__device__ __forceinline__ float warp_sum_x(float v) {
+ v += __shfl_xor_sync(0xffffffffu, v, 16); v += __shfl_xor_sync(0xffffffffu, v, 8);
+ v += __shfl_xor_sync(0xffffffffu, v, 4); v += __shfl_xor_sync(0xffffffffu, v, 2);
+ v += __shfl_xor_sync(0xffffffffu, v, 1);
+ return v;
+}
+__device__ __forceinline__ float warp_max_x(float v) {
+ v = fmaxf(v, __shfl_xor_sync(0xffffffffu, v, 16)); v = fmaxf(v, __shfl_xor_sync(0xffffffffu, v, 8));
+ v = fmaxf(v, __shfl_xor_sync(0xffffffffu, v, 4)); v = fmaxf(v, __shfl_xor_sync(0xffffffffu, v, 2));
+ v = fmaxf(v, __shfl_xor_sync(0xffffffffu, v, 1));
+ return v;
+}
+
+// pool row of cell `c0 + t` of query selection `ids` (attn_chunk_kernel_pre75's srow), -1 masked / absent
+__device__ __forceinline__ long long cell_row(const QsaAttnPools& p, const int32_t* __restrict__ ids, int c0, int t,
+ int n_here, int n_kv_heads, int kvh, int page_size) {
+ long long r = -1;
+ if (t < n_here) {
+ const int cell = __ldg(ids + c0 + t);
+ const long long page = (long long) __ldg(p.page_table + cell / page_size);
+ if (page >= 0) r = (page * n_kv_heads + kvh) * page_size + (cell % page_size);
+ }
+ return r;
+}
+
+// the chunk's K codes (16-byte loads) and scales (as float) into shared memory; masked / absent cells read row 0
+__device__ __forceinline__ void stage_rows(const int8_t* __restrict__ codes, const uint16_t* __restrict__ scales,
+ const long long* srow, int n_here, int t, int8_t (*sk)[HD],
+ float (*sks)[NSC]) {
+ uint4 kv[4];
+#pragma unroll
+ for (int j = 0; j < 4; ++j) {
+ const int idx = t + THREADS * j, c = idx >> 4, part = idx & 15;
+ const long long row = srow[c];
+ const long long rr = (c < n_here && row >= 0) ? row : 0;
+ kv[j] = __ldg(reinterpret_cast<const uint4*>(codes + rr * HD) + part);
+ }
+ float sc;
+ {
+ const int c = t >> 2, g = t & 3;
+ const long long row = srow[c];
+ const long long rr = (c < n_here && row >= 0) ? row : 0;
+ sc = __half2float(__ushort_as_half(__ldg(scales + rr * NSC + g)));
+ }
+#pragma unroll
+ for (int j = 0; j < 4; ++j) reinterpret_cast<uint4*>(&sk[0][0])[t + THREADS * j] = kv[j];
+ sks[t >> 2][t & 3] = sc;
+}
+
+constexpr int PS_BLK = G * CHUNK + 2 * CHUNK + 2 * G; // floats per (query, KV head, chunk): e, rows (int64), m, l
+constexpr int NCM = 33; // chunks the value kernel keeps (cap <= 2112)
+
+// ---------------------------------------------------------------------------------------------------------------------
+// ps_score_x_kernel: the score phase two cells at a time with a select-free reduce-scatter. reduce12's selects exist
+// because the two lanes of an exchange send different slots; here every lane keeps the same slots and sends the same
+// slots, and the lane-dependent part is moved into which query head each register slot holds (q is loaded once, so the
+// permutation costs nothing per cell) and which of the two cells is "first":
+// slot i (0..11) of lane L holds head P_L(i) = 6 ((i / 6) ^ b3) + ((i % 6 + 3 b2) % 6) (b_k: bit k of L)
+// first = cell B if b4 else cell A, second = the other one; part1[i] / part2[i] = lane partials of (first / second,
+// head P_L(i)) - the same expression as attn_chunk_kernel_pre75.
+// level 16: r[i] = part1[i] + shfl(part2[i], 16) (partner: b4 flipped, so its second is my first, same P)
+// level 8 : r6[i] = r[i] + shfl(r[i + 6], 8) (P_{L^8}(i) = P_L(i + 6))
+// level 4 : r3[i] = r6[i] + shfl(r6[i + 3], 4) (P_{L^4}(i) = P_L(i + 3), i < 3)
+// level 2 / 1: reduce24's (heads 6 b3 + 3 b2 + {0, 1, 2}; o0 head +2 b1, o1 head +1 where b1 = 0)
+// At every level the value at lane L is its partial at L plus its partial at L ^ o, the same operands warp_sum adds,
+// so every score is reduce12's bit for bit. A pair with an absent / masked cell runs the same code on row 0's staged
+// codes and that cell's outputs are not stored (it gets -FLT_MAX). 25 shuffles, 25 adds and 4 selects per pair
+// instead of 2 x (15 + 15 + ~30).
+__device__ __forceinline__ int px_head(int i, int lane) {
+ const int b3 = (lane >> 3) & 1, b2 = (lane >> 2) & 1;
+ return 6 * ((i / 6) ^ b3) + ((i % 6 + 3 * b2) % 6);
+}
+__device__ __forceinline__ void load_q_perm(const float* __restrict__ qh, int lane, float4 (&qa)[G], float4 (&qb)[G]) {
+#pragma unroll
+ for (int i = 0; i < G; ++i) {
+ const float4* p4 = reinterpret_cast<const float4*>(qh + (size_t) px_head(i, lane) * HD + lane * 8);
+ qa[i] = __ldg(p4);
+ qb[i] = __ldg(p4 + 1);
+ }
+}
+__device__ __forceinline__ void dequant8(const int8_t* krow, float sc, int lane, float (&k8)[8]) {
+ const uint2 raw = *reinterpret_cast<const uint2*>(krow + lane * 8);
+ const int8_t* cb = reinterpret_cast<const int8_t*>(&raw);
+#pragma unroll
+ for (int e = 0; e < 8; ++e) k8[e] = (float) cb[e] * sc;
+}
+__device__ __forceinline__ float dot8(const float (&k8)[8], const float4& qa, const float4& qb) {
+ return k8[0] * qa.x + k8[1] * qa.y + k8[2] * qa.z + k8[3] * qa.w + k8[4] * qb.x + k8[5] * qb.y + k8[6] * qb.z +
+ k8[7] * qb.w;
+}
+template <int UNR = 1>
+__device__ __forceinline__ void score_chunk_x(const float4 (&qa)[G], const float4 (&qb)[G], const int8_t (*sk)[HD],
+ const float (*sks)[NSC], const long long* srow, float (*sp)[CHUNK],
+ int n_here, float scale, int warp, int lane) {
+ const bool h16 = (lane & 16) != 0, h2 = (lane & 2) != 0, even = (lane & 1) == 0;
+ const int hb = 6 * ((lane >> 3) & 1) + 3 * ((lane >> 2) & 1);
+ const int h0 = hb + (h2 ? 2 : 0), h1 = hb + 1;
+#pragma unroll UNR
+ for (int i = 0; i < CHUNK / WARPS; i += 2) {
+ const int ca = warp + WARPS * i, cb = warp + WARPS * (i + 1);
+ const bool va = ca < n_here && srow[ca] >= 0, vb = cb < n_here && srow[cb] >= 0;
+ if (!va && !vb) {
+ if (lane < G) { sp[lane][ca] = -FLT_MAX; sp[lane][cb] = -FLT_MAX; }
+ continue;
+ }
+ const int c1 = h16 ? cb : ca, c2 = h16 ? ca : cb; // first / second (the staged rows of absent cells are row 0's)
+ float r[G];
+ {
+ float k8[8];
+ dequant8(sk[c2], sks[c2][lane >> 3], lane, k8);
+ float p2[G];
+#pragma unroll
+ for (int s = 0; s < G; ++s) p2[s] = dot8(k8, qa[s], qb[s]);
+ dequant8(sk[c1], sks[c1][lane >> 3], lane, k8);
+#pragma unroll
+ for (int s = 0; s < G; ++s) r[s] = dot8(k8, qa[s], qb[s]) + __shfl_xor_sync(0xffffffffu, p2[s], 16);
+ }
+ float r6[6];
+#pragma unroll
+ for (int s = 0; s < 6; ++s) r6[s] = r[s] + __shfl_xor_sync(0xffffffffu, r[s + 6], 8);
+ float r3[3];
+#pragma unroll
+ for (int s = 0; s < 3; ++s) r3[s] = r6[s] + __shfl_xor_sync(0xffffffffu, r6[s + 3], 4);
+ const float x0 = (h2 ? r3[2] : r3[0]) + __shfl_xor_sync(0xffffffffu, h2 ? r3[0] : r3[2], 2);
+ const float x1 = r3[1] + __shfl_xor_sync(0xffffffffu, r3[1], 2);
+ const float o0 = x0 + __shfl_xor_sync(0xffffffffu, x0, 1);
+ const float o1 = x1 + __shfl_xor_sync(0xffffffffu, x1, 1);
+ const bool vmine = h16 ? vb : va;
+ if (even && vmine) {
+ sp[h0][c1] = o0 * scale;
+ if (!h2) sp[h1][c1] = o1 * scale;
+ }
+ if (lane < G) {
+ if (!va) sp[lane][ca] = -FLT_MAX;
+ if (!vb) sp[lane][cb] = -FLT_MAX;
+ }
+ }
+}
+
+// ps_score_kernel with score_chunk_x (q loaded in the permuted slot order)
+template <int UNR>
+__global__ void __launch_bounds__(THREADS, 2) ps_score_x_kernel(const float* __restrict__ q, QsaAttnPools p,
+ const int32_t* __restrict__ ids,
+ const int32_t* __restrict__ step, int n_kv_heads,
+ int page_size, float scale, float* __restrict__ blocks,
+ int n_chunks, int cap) {
+ q += (size_t) blockIdx.z * (size_t) (n_kv_heads * G) * HD;
+ ids += (size_t) blockIdx.z * (size_t) cap;
+ step += (size_t) blockIdx.z * kStepCount;
+ const int chunk = blockIdx.x, kvh = blockIdx.y;
+ float* const blk = blocks + (((size_t) blockIdx.z * n_kv_heads + kvh) * n_chunks + chunk) * PS_BLK;
+ __shared__ __align__(16) int8_t sk[CHUNK][HD];
+ __shared__ __align__(16) float sks[CHUNK][NSC];
+ __shared__ float sp[G][CHUNK];
+ __shared__ long long srow[CHUNK];
+ const int n_ids = __ldg(step + kStepWidth);
+ const int t = threadIdx.x, lane = t & 31, warp = t >> 5;
+ const int c0 = chunk * CHUNK;
+ const int n_here = min(CHUNK, n_ids - c0);
+ if (n_here <= 0) {
+ if (t < G) { blk[G * CHUNK + 2 * CHUNK + t] = -FLT_MAX; blk[G * CHUNK + 2 * CHUNK + G + t] = 0.0f; }
+ return;
+ }
+ if (t < CHUNK) {
+ const long long r = cell_row(p, ids, c0, t, n_here, n_kv_heads, kvh, page_size);
+ srow[t] = r;
+ reinterpret_cast<long long*>(blk + G * CHUNK)[t] = r;
+ }
+ float4 qa[G], qb[G];
+ load_q_perm(q + (size_t) (kvh * G) * HD, lane, qa, qb);
+ __syncthreads();
+ stage_rows(p.k_q, p.k_scale, srow, n_here, t, sk, sks);
+ __syncthreads();
+ score_chunk_x<UNR>(qa, qb, sk, sks, srow, sp, n_here, scale, warp, lane);
+ __syncthreads();
+ for (int h = warp; h < G; h += WARPS) {
+ const float a = sp[h][lane], b = sp[h][lane + 32];
+ const float m = warp_max_x(fmaxf(a, b));
+ const float ea = (lane < n_here && srow[lane] >= 0) ? __expf(a - m) : 0.0f;
+ const float eb = (lane + 32 < n_here && srow[lane + 32] >= 0) ? __expf(b - m) : 0.0f;
+ blk[h * CHUNK + lane] = ea;
+ blk[h * CHUNK + lane + 32] = eb;
+ const float l = warp_sum_x(ea + eb);
+ if (lane == 0) { blk[G * CHUNK + 2 * CHUNK + h] = m; blk[G * CHUNK + 2 * CHUNK + G + h] = l; }
+ }
+}
+
+// ps_value_d4_kernel: the value kernel with four dims a thread (64 threads, one CTA per (KV head, query)): a cell's four
+// codes are one 32-bit load (a warp reads 128 contiguous bytes), its 12 weights three broadcast float4 reads for 48
+// fmaf. V loads run four cells ahead. Per (dim, head) the fold and the merge are ps_value_kernel's.
+__global__ void __launch_bounds__(64) ps_value_d4_kernel(QsaAttnPools p, const int32_t* __restrict__ steps,
+ const float* __restrict__ blocks, int n_kv_heads, int n_chunks,
+ float* __restrict__ attn) {
+ constexpr int NT = 64;
+ const int kvh = blockIdx.x, z = blockIdx.y;
+ const int t = threadIdx.x;
+ const float* const blk0 = blocks + ((size_t) z * n_kv_heads + kvh) * n_chunks * PS_BLK;
+ __shared__ float sm_m[NCM][G], sm_w[NCM][G], sm_l[NCM][G];
+ __shared__ float sL[G];
+ __shared__ __align__(16) float spt[CHUNK][G];
+ __shared__ long long srow[CHUNK];
+ __shared__ int s_masked;
+ const int width = __ldg(steps + (size_t) z * kStepCount + kStepWidth);
+ const int n_act = width > 0 ? min(n_chunks, (width - 1) / CHUNK + 1) : 0;
+ for (int i = t; i < n_act * G; i += NT) {
+ const int c = i / G, h = i % G;
+ sm_m[c][h] = __ldg(blk0 + (size_t) c * PS_BLK + G * CHUNK + 2 * CHUNK + h);
+ sm_l[c][h] = __ldg(blk0 + (size_t) c * PS_BLK + G * CHUNK + 2 * CHUNK + G + h);
+ }
+ __syncthreads();
+ if (t < G) {
+ float M = -FLT_MAX;
+ for (int c = 0; c < n_act; ++c) M = fmaxf(M, sm_m[c][t]);
+ float L = 0.0f;
+ for (int c = 0; c < n_act; ++c) {
+ const float m = sm_m[c][t];
+ const float w = __expf(m - M);
+ sm_w[c][t] = w;
+ const float L_next = fmaf(sm_l[c][t], w, L);
+ L = (m == -FLT_MAX) ? L : L_next;
+ }
+ sL[t] = L;
+ }
+ const int d0 = 4 * t, g = d0 / KV_Q8_GROUP;
+ float acc[4][G];
+#pragma unroll
+ for (int h = 0; h < G; ++h)
+#pragma unroll
+ for (int d = 0; d < 4; ++d) acc[d][h] = 0.0f;
+ for (int c = 0; c < n_act; ++c) {
+ const float* blk = blk0 + (size_t) c * PS_BLK;
+ const int n_here = min(CHUNK, width - c * CHUNK);
+ __syncthreads();
+ if (t == 0) s_masked = 0;
+ __syncthreads();
+ {
+ const long long r = __ldg(reinterpret_cast<const long long*>(blk + G * CHUNK) + t);
+ srow[t] = r;
+ if (t < n_here && r < 0) s_masked = 1;
+ }
+#pragma unroll
+ for (int j = 0; j < G * CHUNK / NT; ++j) {
+ const int idx = t + NT * j, h = idx / CHUNK, cell = idx % CHUNK;
+ spt[cell][h] = __ldg(blk + idx);
+ }
+ __syncthreads();
+ float a[4][G];
+#pragma unroll
+ for (int h = 0; h < G; ++h)
+#pragma unroll
+ for (int d = 0; d < 4; ++d) a[d][h] = 0.0f;
+ const auto fold4 = [&](int cl, uint32_t codes, float sc) {
+ const float v0 = (float) (int8_t) (codes & 0xFF) * sc, v1 = (float) (int8_t) ((codes >> 8) & 0xFF) * sc;
+ const float v2 = (float) (int8_t) ((codes >> 16) & 0xFF) * sc, v3 = (float) (int8_t) (codes >> 24) * sc;
+ const float4* w4 = reinterpret_cast<const float4*>(spt[cl]);
+ const float4 w0 = w4[0], w1 = w4[1], w2 = w4[2];
+ const float ww[G] = {w0.x, w0.y, w0.z, w0.w, w1.x, w1.y, w1.z, w1.w, w2.x, w2.y, w2.z, w2.w};
+#pragma unroll
+ for (int h = 0; h < G; ++h) {
+ a[0][h] = fmaf(ww[h], v0, a[0][h]);
+ a[1][h] = fmaf(ww[h], v1, a[1][h]);
+ a[2][h] = fmaf(ww[h], v2, a[2][h]);
+ a[3][h] = fmaf(ww[h], v3, a[3][h]);
+ }
+ };
+ const auto vld = [&](int cl, uint32_t& codes, float& sc) {
+ const long long r = srow[cl];
+ const long long rr = r >= 0 ? r : 0;
+ codes = __ldg(reinterpret_cast<const uint32_t*>(p.v_q + rr * HD + d0));
+ sc = __half2float(__ushort_as_half(__ldg(p.v_scale + rr * NSC + g)));
+ };
+ if (!s_masked) {
+ int cl = 0;
+ for (; cl + 4 <= n_here; cl += 4) {
+ uint32_t k0, k1, k2, k3;
+ float s0, s1, s2, s3;
+ vld(cl, k0, s0); vld(cl + 1, k1, s1); vld(cl + 2, k2, s2); vld(cl + 3, k3, s3);
+ fold4(cl, k0, s0);
+ fold4(cl + 1, k1, s1);
+ fold4(cl + 2, k2, s2);
+ fold4(cl + 3, k3, s3);
+ }
+ for (; cl < n_here; ++cl) {
+ uint32_t k0;
+ float s0;
+ vld(cl, k0, s0);
+ fold4(cl, k0, s0);
+ }
+ } else {
+ for (int cl = 0; cl < n_here; ++cl) {
+ if (srow[cl] < 0) continue;
+ uint32_t k0;
+ float s0;
+ vld(cl, k0, s0);
+ fold4(cl, k0, s0);
+ }
+ }
+#pragma unroll
+ for (int h = 0; h < G; ++h) {
+ const bool live = !(sm_m[c][h] == -FLT_MAX);
+ const float w = sm_w[c][h];
+#pragma unroll
+ for (int d = 0; d < 4; ++d) {
+ const float acc_next = fmaf(a[d][h], w, acc[d][h]);
+ acc[d][h] = live ? acc_next : acc[d][h];
+ }
+ }
+ }
+ __syncthreads();
+ float* out = attn + ((size_t) z * n_kv_heads + kvh) * G * HD;
+#pragma unroll
+ for (int h = 0; h < G; ++h) {
+ const float L = sL[h];
+ *reinterpret_cast<float4*>(out + (size_t) h * HD + d0) =
+ make_float4(L > 0.0f ? acc[0][h] / L : 0.0f, L > 0.0f ? acc[1][h] / L : 0.0f,
+ L > 0.0f ? acc[2][h] / L : 0.0f, L > 0.0f ? acc[3][h] / L : 0.0f);
+ }
+}
+
+bool sm6x_device() {
+ static int cc[64] = {};
+ int dev = 0;
+ if (cudaGetDevice(&dev) != cudaSuccess || dev < 0 || dev >= 64) { cudaGetLastError(); return false; }
+ if (cc[dev] == 0) {
+ int major = 0;
+ if (cudaDeviceGetAttribute(&major, cudaDevAttrComputeCapabilityMajor, dev) != cudaSuccess) {
+ cudaGetLastError();
+ return false;
+ }
+ cc[dev] = major;
+ }
+ return cc[dev] == 6;
+}
+
+} // namespace
+
+bool qsa_pattn_batch(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps, int64_t cap,
+ const QsaShapes& s, float* scratch, uint64_t scratch_floats, float* attn, int64_t n_q,
+ void* stream) {
+ return sm6x_device() && qsa_pattn_batch_any_arch(q, pools, ids, steps, cap, s, scratch, scratch_floats, attn, n_q, stream);
+}
+
+bool qsa_pattn_batch_any_arch(const float* q, const QsaAttnPools& pools, const int32_t* ids, const int32_t* steps,
+ int64_t cap, const QsaShapes& s, float* scratch, uint64_t scratch_floats, float* attn,
+ int64_t n_q, void* stream) {
+ if (n_q <= 0) return true;
+ if (s.head_dim != HD || s.n_head != (int64_t) G * s.n_head_kv || cap <= 0 || !scratch || !ids || !steps ||
+ !pools.page_table || pools.k_q == nullptr || pools.v_q == nullptr || pools.k_q4 != nullptr ||
+ pools.v_q4 != nullptr || !pools.k_scale || !pools.v_scale)
+ return false;
+ const cudaStream_t st = (cudaStream_t) stream;
+ const int n_chunks = (int) ((cap + CHUNK - 1) / CHUNK);
+ const float scale = 1.0f / sqrtf((float) HD);
+ if (n_chunks > NCM) return false; // a longer selection keeps the usual path
+ // the scratch holds the blocks of one batch of queries (probabilities, rows, m and l of every chunk), evenly split
+ const uint64_t per_q = (uint64_t) s.n_head_kv * (uint64_t) n_chunks * PS_BLK;
+ const int64_t max_batch = (int64_t) (scratch_floats / per_q);
+ if (max_batch < 1) return false;
+ const int64_t n_batches = (n_q + max_batch - 1) / max_batch;
+ const int64_t batch = (n_q + n_batches - 1) / n_batches;
+ for (int64_t t0 = 0; t0 < n_q; t0 += batch) {
+ const int64_t nb = n_q - t0 < batch ? n_q - t0 : batch;
+ const dim3 g1((unsigned) n_chunks, (unsigned) s.n_head_kv, (unsigned) nb);
+ const dim3 g2((unsigned) s.n_head_kv, (unsigned) nb);
+ const float* qb = q + (size_t) t0 * s.n_head * HD;
+ const int32_t* ib = ids + (size_t) t0 * cap;
+ const int32_t* sb = steps + (size_t) t0 * kStepCount;
+ ps_score_x_kernel<1><<<g1, THREADS, 0, st>>>(qb, pools, ib, sb, (int) s.n_head_kv, (int) s.page_size, scale,
+ scratch, n_chunks, (int) cap);
+ ps_value_d4_kernel<<<g2, 64, 0, st>>>(pools, sb, scratch, (int) s.n_head_kv, n_chunks,
+ attn + (size_t) t0 * s.n_head * HD);
+ }
+ const cudaError_t e = cudaGetLastError();
+ if (e != cudaSuccess) {
+ // a launch that did not start: say so once, and let the caller run the usual attention over the whole chunk
+ static bool warned = false;
+ if (!warned) {
+ warned = true;
+ std::fprintf(stderr, "strata pattn: launch failed (%s); the usual attention runs instead\n", cudaGetErrorString(e));
+ }
+ return false;
+ }
+ return true;
+}
+
+#else // not the experimental sm_60 CUDA build
+bool qsa_pattn_batch(const float*, const QsaAttnPools&, const int32_t*, const int32_t*, int64_t, const QsaShapes&,
+ float*, uint64_t, float*, int64_t, void*) { return false; }
+bool qsa_pattn_batch_any_arch(const float*, const QsaAttnPools&, const int32_t*, const int32_t*, int64_t, const QsaShapes&,
+ float*, uint64_t, float*, int64_t, void*) { return false; }
+#endif
+
+} // namespace strata::kernels
diff --git a/src/kernels/qsa_pattn_parity.cpp b/src/kernels/qsa_pattn_parity.cpp
new file mode 100644
index 000000000..ab67038d8
--- /dev/null
+++ b/src/kernels/qsa_pattn_parity.cpp
@@ -0,0 +1,260 @@
+// src/kernels/qsa_pattn_parity.cpp - STRATA_PATTN_SM60's score and value kernels (qsa_pattn.cu, PR #3943) against the
+// decode kernels' output, bitwise (GPU, synthetic data, no model).
+//
+// The reference is what the prompt path reads on sm_6x today: the chunk pass of attn_chunk_kernel_pre75 (the Daily's
+// chunk kernel, qsa_decode_attn_chunks_only with daily_chunk 1, which runs on any card) and attn_merge_kernel
+// (qsa_decode_attn_merge_only, v2 0). The candidate is qsa_pattn_batch_any_arch (no compute-capability test), so the
+// same bytes can be checked on an Ampere card too: the two paths do the same operations in the same order, and a
+// pass there shows the indexing, masking and merge are right. The last word is a run on a Pascal card (P100): there
+// the checked entry qsa_pattn_batch must take the shapes and the production qsa_decode_attn_batch must give the same
+// bytes as the reference.
+//
+// Data: random int8 K/V codes, fp16 scales in a safe range, normal queries, a shuffled page table with ~3% unresident
+// (-1) pages and one run of 64 unresident pages (so whole chunks are masked and some queries read nothing), selections
+// shaped like the drafter's window, the indexer's (ascending with gaps) and unordered with repeats, caps 64 / 130 /
+// 2051 / 2112 (1, 3 and 33 chunks), widths 0 .. cap, n_q up to 40 with a scratch that holds only 2 queries of the
+// decode layout (so qsa_pattn_batch runs several batches). The pattn scratch and the output are filled with
+// junk before each call, so a read of a block that was not written shows up as a difference.
+//
+// qsa_pattn_parity [--device N] exit 0: every case is bit-identical; 77: no GPU or a build without the kernels
+#include "strata/kernels/qsa.hpp"
+#include "strata/kernels/qsa_decode_attn.hpp"
+#include "strata/kernels/qsa_pattn.hpp"
+
+#include <cuda_runtime.h>
+
+#include <algorithm>
+#include <cmath>
+#include <cstdint>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <random>
+#include <vector>
+
+namespace k = strata::kernels;
+
+namespace {
+void ck(cudaError_t e, const char* what) {
+ if (e != cudaSuccess) {
+ std::fprintf(stderr, "qsa_pattn_parity: %s: %s\n", what, cudaGetErrorString(e));
+ std::exit(2);
+ }
+}
+template <typename T> T* dalloc(size_t n) {
+ T* p = nullptr;
+ ck(cudaMalloc((void**) &p, n * sizeof(T) + 256), "cudaMalloc");
+ ck(cudaMemset(p, 0, n * sizeof(T) + 256), "cudaMemset");
+ return p;
+}
+template <typename T> T* upload(const std::vector<T>& h) {
+ T* d = dalloc<T>(h.size());
+ ck(cudaMemcpy(d, h.data(), h.size() * sizeof(T), cudaMemcpyHostToDevice), "upload");
+ return d;
+}
+// positive fp16 scale in [1/64, 1/32): codes up to 127 give |x| < 4
+uint16_t f16_scale_q8(uint32_t r) { return (uint16_t) ((9u << 10) | (r & 0x3FFu)); }
+
+constexpr int64_t kCells = 40960; // the cells this harness can address
+constexpr int64_t kHole = 8192; // 64 unresident pages in a row: cells [kHole, kHole + 256)
+
+void selection(int32_t* ids, int64_t cap, int width, int shape, std::mt19937& rng) {
+ const int64_t w = std::max<int64_t>(0, std::min<int64_t>(width, cap));
+ if (shape == 0) { // the drafter's window: consecutive cells, often across the unresident run
+ int64_t start;
+ if (rng() % 2 == 0) start = std::max<int64_t>(0, kHole - 64 * (int64_t) (rng() % 8));
+ else start = (int64_t) (rng() % (uint32_t) (kCells - std::max<int64_t>(w, 1)));
+ if (start + w > kCells) start = kCells - w;
+ for (int64_t j = 0; j < cap; ++j) ids[j] = (int32_t) ((start + j) % kCells);
+ } else if (shape == 1) { // the indexer's: ascending with gaps
+ std::vector<int32_t> v((size_t) cap);
+ for (auto& x : v) x = (int32_t) (rng() % (uint32_t) kCells);
+ std::sort(v.begin(), v.begin() + w);
+ std::copy(v.begin(), v.end(), ids);
+ } else { // unordered, with repeats
+ for (int64_t j = 0; j < cap; ++j) ids[j] = (int32_t) (rng() % (uint32_t) kCells);
+ }
+}
+
+struct Harness {
+ k::QsaShapes s = k::qsa_real_shapes();
+ k::QsaAttnPools pools;
+ int32_t* table = nullptr;
+ float *q = nullptr, *scratch_ref = nullptr, *scratch_new = nullptr, *attn_ref = nullptr, *attn_new = nullptr;
+ int32_t *ids = nullptr, *steps = nullptr;
+ int64_t max_q = 0, max_cap = 0;
+ std::vector<void*> to_free;
+
+ template <typename T> T* keep(T* p) { to_free.push_back(p); return p; }
+
+ void init(std::mt19937& rng, int64_t max_q_, int64_t max_cap_) {
+ max_q = max_q_;
+ max_cap = max_cap_;
+ const int64_t PS = s.page_size, pages = kCells / PS, H = s.n_head_kv, D = s.head_dim;
+ std::vector<int32_t> tab((size_t) pages); // logical page -> physical page: a permutation, with holes
+ for (int64_t i = 0; i < pages; ++i) tab[(size_t) i] = (int32_t) i;
+ std::shuffle(tab.begin(), tab.end(), rng);
+ for (int64_t i = 0; i < pages; ++i)
+ if (rng() % 32u == 0) tab[(size_t) i] = -1;
+ for (int64_t i = kHole / PS; i < (kHole + 256) / PS; ++i) tab[(size_t) i] = -1;
+ table = keep(upload(tab));
+ const int64_t rows = pages * H * PS;
+ pools.page_table = table;
+ std::vector<int8_t> c((size_t) (rows * D));
+ for (auto& x : c) x = (int8_t) ((int) (rng() % 255u) - 127);
+ pools.k_q = keep(upload(c));
+ for (auto& x : c) x = (int8_t) ((int) (rng() % 255u) - 127);
+ pools.v_q = keep(upload(c));
+ std::vector<uint16_t> sc((size_t) (rows * (D / 64)));
+ for (auto& x : sc) x = f16_scale_q8(rng());
+ pools.k_scale = keep(upload(sc));
+ for (auto& x : sc) x = f16_scale_q8(rng());
+ pools.v_scale = keep(upload(sc));
+ q = keep(dalloc<float>((size_t) (max_q * s.n_head * D)));
+ ids = keep(dalloc<int32_t>((size_t) (max_q * max_cap)));
+ steps = keep(dalloc<int32_t>((size_t) (max_q * k::kStepCount)));
+ const size_t sf = (size_t) (k::qsa_decode_attn_scratch_floats(max_cap, s) * (uint64_t) max_q);
+ scratch_ref = keep(dalloc<float>(sf));
+ scratch_new = keep(dalloc<float>(sf));
+ attn_ref = keep(dalloc<float>((size_t) (max_q * s.n_head * D)));
+ attn_new = keep(dalloc<float>((size_t) (max_q * s.n_head * D)));
+ }
+ ~Harness() {
+ for (void* p : to_free) cudaFree(p);
+ }
+
+ // returns 0 identical, 1 differs, -1 the candidate declined (build without the kernels)
+ int run(int64_t cap, int n_q, int scr_q, const std::vector<int32_t>& widths, std::mt19937& rng, bool on_pascal,
+ long long& nonzero) {
+ const int64_t D = s.head_dim, NH = s.n_head;
+ std::normal_distribution<float> nd(0.0f, 1.0f);
+ std::vector<float> hq((size_t) (n_q * NH * D));
+ for (auto& x : hq) x = nd(rng);
+ ck(cudaMemcpy(q, hq.data(), hq.size() * 4, cudaMemcpyHostToDevice), "q");
+ std::vector<int32_t> hid((size_t) (n_q * cap));
+ std::vector<int32_t> hst((size_t) (n_q * k::kStepCount), 0);
+ for (int i = 0; i < n_q; ++i) {
+ selection(hid.data() + (size_t) i * cap, cap, widths[(size_t) i], (int) (rng() % 3u), rng);
+ hst[(size_t) i * k::kStepCount + k::kStepWidth] = widths[(size_t) i];
+ }
+ ck(cudaMemcpy(ids, hid.data(), hid.size() * 4, cudaMemcpyHostToDevice), "ids");
+ ck(cudaMemcpy(steps, hst.data(), hst.size() * 4, cudaMemcpyHostToDevice), "steps");
+ const size_t n = (size_t) (n_q * NH * D);
+ const size_t sbytes = (size_t) k::qsa_decode_attn_scratch_floats(cap, s) * (size_t) n_q * 4;
+
+ // the reference: the pre-sm_75 chunk kernel and attn_merge_kernel
+ ck(cudaMemset(scratch_ref, 0xFF, sbytes), "scratch_ref fill");
+ ck(cudaMemset(attn_ref, 0xFF, n * 4), "attn_ref fill");
+ k::qsa_decode_attn_chunks_only(q, pools, ids, steps, cap, s, scratch_ref, n_q, 1, nullptr);
+ k::qsa_decode_attn_merge_only(steps, cap, s, scratch_ref, attn_ref, n_q, 0, nullptr);
+ ck(cudaDeviceSynchronize(), "reference");
+ std::vector<float> a(n), b(n);
+ ck(cudaMemcpy(a.data(), attn_ref, n * 4, cudaMemcpyDeviceToHost), "read reference");
+ nonzero = 0;
+ for (size_t i = 0; i < n; ++i) {
+ if (!std::isfinite(a[i])) { std::printf("FAIL: the reference has a non-finite value\n"); return 1; }
+ if (a[i] != 0.0f) ++nonzero;
+ }
+
+ // the candidate: the scratch holds scr_q queries of the decode layout (the caller's attn_batch), junk in it
+ ck(cudaMemset(scratch_new, 0xFF, sbytes), "scratch_new fill");
+ ck(cudaMemset(attn_new, 0x7F, n * 4), "attn_new fill");
+ const uint64_t scratch_floats = (uint64_t) scr_q * k::qsa_decode_attn_scratch_floats(cap, s);
+ const bool took = k::qsa_pattn_batch_any_arch(q, pools, ids, steps, cap, s, scratch_new, scratch_floats, attn_new,
+ n_q, nullptr);
+ if (!took) return -1;
+ ck(cudaDeviceSynchronize(), "candidate");
+ ck(cudaMemcpy(b.data(), attn_new, n * 4, cudaMemcpyDeviceToHost), "read candidate");
+ bool same = std::memcmp(a.data(), b.data(), n * 4) == 0;
+ if (!same) {
+ size_t first = 0;
+ while (first < n && std::memcmp(&a[first], &b[first], 4) == 0) ++first;
+ std::printf("FAIL cap %lld n_q %d scratch %d queries, widths", (long long) cap, n_q, scr_q);
+ for (int32_t w : widths) std::printf(" %d", w);
+ std::printf(": first difference at query %lld head %lld dim %lld: %.9g vs %.9g\n",
+ (long long) (first / (NH * D)), (long long) (first / D % NH), (long long) (first % D), a[first],
+ b[first]);
+ }
+ if (on_pascal) { // the checked entry takes the shapes, and the production path is the reference
+ ck(cudaMemset(scratch_new, 0xFF, sbytes), "scratch_new refill");
+ ck(cudaMemset(attn_new, 0x7F, n * 4), "attn_new refill");
+ if (!k::qsa_pattn_batch(q, pools, ids, steps, cap, s, scratch_new, scratch_floats, attn_new, n_q, nullptr)) {
+ std::printf("FAIL: qsa_pattn_batch declined on a compute capability 6 card (cap %lld n_q %d)\n",
+ (long long) cap, n_q);
+ same = false;
+ } else {
+ ck(cudaDeviceSynchronize(), "checked entry");
+ ck(cudaMemcpy(b.data(), attn_new, n * 4, cudaMemcpyDeviceToHost), "read checked entry");
+ if (std::memcmp(a.data(), b.data(), n * 4) != 0) {
+ std::printf("FAIL: qsa_pattn_batch differs from the reference (cap %lld n_q %d)\n", (long long) cap, n_q);
+ same = false;
+ }
+ }
+ ck(cudaMemset(scratch_new, 0xFF, sbytes), "scratch_new refill");
+ k::qsa_decode_attn_batch(q, pools, ids, steps, cap, s, scratch_new, attn_new, n_q, nullptr);
+ ck(cudaDeviceSynchronize(), "production");
+ ck(cudaMemcpy(b.data(), attn_new, n * 4, cudaMemcpyDeviceToHost), "read production");
+ if (std::memcmp(a.data(), b.data(), n * 4) != 0) {
+ std::printf("FAIL: qsa_decode_attn_batch differs from the reference (cap %lld n_q %d)\n", (long long) cap,
+ n_q);
+ same = false;
+ }
+ }
+ return same ? 0 : 1;
+ }
+};
+} // namespace
+
+int main(int argc, char** argv) {
+ int device = 0;
+ for (int i = 1; i < argc; ++i) {
+ if (std::strcmp(argv[i], "--device") == 0 && i + 1 < argc) device = std::atoi(argv[++i]);
+ else {
+ std::fprintf(stderr, "usage: qsa_pattn_parity [--device N]\n");
+ return 2;
+ }
+ }
+ int devices = 0;
+ if (cudaGetDeviceCount(&devices) != cudaSuccess || devices == 0) {
+ std::puts("SKIP: no GPU");
+ return 77;
+ }
+ ck(cudaSetDevice(device), "cudaSetDevice");
+ cudaDeviceProp prop{};
+ ck(cudaGetDeviceProperties(&prop, device), "cudaGetDeviceProperties");
+ const bool on_pascal = prop.major == 6;
+ std::printf("qsa_pattn_parity on device %d (%s, %d.%d)%s\n", device, prop.name, prop.major, prop.minor,
+ on_pascal ? ": the checked entry and the production path are compared too" : "");
+
+ std::mt19937 rng(20261026u);
+ Harness hx;
+ hx.init(rng, 40, 2112);
+
+ const int32_t base_widths[] = {0, 1, 63, 64, 65, 1023, 1024, 1025, 2051, 2112};
+ const int n_base = (int) (sizeof(base_widths) / sizeof(base_widths[0]));
+ long long cases = 0, failed = 0, nonzero_total = 0;
+ for (int64_t cap : {(int64_t) 64, (int64_t) 130, (int64_t) 2051, (int64_t) 2112}) {
+ for (int n_q : {1, 2, 3, 7, 40}) {
+ for (int scr_q : {n_q, 2}) {
+ if (scr_q > n_q) continue;
+ for (int wc = 0; wc < n_base + 3; ++wc) {
+ std::vector<int32_t> widths;
+ for (int i = 0; i < n_q; ++i)
+ widths.push_back(std::min<int32_t>((int32_t) cap, wc < n_base ? base_widths[wc] : base_widths[rng() % (uint32_t) n_base]));
+ long long nz = 0;
+ const int r = hx.run(cap, n_q, scr_q, widths, rng, on_pascal, nz);
+ if (r < 0) {
+ std::puts("SKIP: this build has no sm_6x kernels (CUDA with STRATA_EXPERIMENTAL_SM60)");
+ return 77;
+ }
+ nonzero_total += nz;
+ ++cases;
+ if (r != 0) ++failed;
+ }
+ }
+ }
+ }
+ std::printf("qsa_pattn_parity: %lld cases, %lld differ, %lld nonzero output values in all -> %s\n", cases, failed,
+ nonzero_total, failed == 0 && nonzero_total > 0 ? "PASS" : "FAIL");
+ return failed == 0 && nonzero_total > 0 ? 0 : 1;
+}
diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp
index 08b529e6f..a9d121f94 100644
--- a/src/prefill/prefill.cpp
+++ b/src/prefill/prefill.cpp
@@ -38,6 +38,7 @@
#include "strata/core/native_head.hpp"
#include "strata/kernels/verify_kernels.hpp"
#include "strata/kernels/qsa_decode_attn.hpp"
+#include "strata/kernels/qsa_pattn.hpp"
#include "strata/kernels/qsa_prompt_attn.hpp"
#include "strata/kernels/qsa_select.hpp"
#include "strata/prefill/gemm.hpp"
@@ -3723,8 +3724,18 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
// decode kernel, 32 queries at a time (K8V4 runs the tensor kernel's mode 3: INT8 K,
// V dequantized from its q4_0 blocks to fp16 at gather)
static const bool old_attn = std::getenv("STRATA_PROMPT_ATTN_OLD") != nullptr;
+ // STRATA_PATTN_SM60=1 (qsa_pattn.hpp, sm_6x, int8 K/V): the decode kernels' output bit for bit from a
+ // score kernel and a value kernel that merges as it folds (no partials, no merge kernel), ~100
+ // queries a batch in the same scratch; any refusal or launch failure runs the loop below
if (old_attn || !strata::kernels::qsa_prompt_attn_batch(m.q, pools, m.sel_ids, m.steps_dev, m.cap, s,
m.attn, T, m.cs))
+#if !defined(STRATA_GPU_METAL) // qsa_pattn.cu is in the CUDA / HIP kernel library only
+ if (!(strata::kernels::pattn_mode() != 0 &&
+ strata::kernels::qsa_pattn_batch(m.q, pools, m.sel_ids, m.steps_dev, m.cap, s, m.attn_scratch,
+ (uint64_t) m.attn_batch *
+ strata::kernels::qsa_decode_attn_scratch_floats(m.cap, s),
+ m.attn, T, m.cs)))
+#endif
for (int64_t t0 = 0; t0 < T; t0 += m.attn_batch) {
const int64_t nb = std::min(m.attn_batch, T - t0);
strata::kernels::qsa_decode_attn_batch(m.q + t0 * ZV, pools, m.sel_ids + t0 * m.cap,