Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
iq_kernels: GP100 4-bit gate/up expert slots, opt-in STRATA_EXPERT_NIBBLE=1 (#3455)
ruibeikaa9067cc64d317
diff --git a/CMakeLists.txt b/CMakeLists.txtindex efb231216..60a25467f 100644--- a/CMakeLists.txt+++ b/CMakeLists.txt@@ -2247,6 +2247,20 @@ if(STRATA_NATIVE_EXPERTS)add_executable(native_expert_bench src/kernels/native_expert_bench.cpp)target_include_directories(native_expert_bench PRIVATE ${_strata_gpu_include_directories})target_link_libraries(native_expert_bench PRIVATE strata_kernels strata_kernels_cpu strata_artifact ${_strata_gpu_runtime_target})+ if(STRATA_ENABLE_CUDA AND STRATA_EXPERIMENTAL_SM60)+ # STRATA_EXPERT_NIBBLE (include/strata/core/expert_nibble.hpp): the 4-bit gate/up expert slots against the+ # production kernels and the GGUF blobs, bit for bit, on a real model's layers (needs a GP100); --header: the+ # contract's host half, no GPU+ add_executable(iq_nibble_parity src/kernels/iq_nibble_parity.cpp)+ target_include_directories(iq_nibble_parity PRIVATE ${_strata_gpu_include_directories})+ target_link_libraries(iq_nibble_parity PRIVATE strata_kernels strata_kernels_cpu strata_artifact ggml-base ${_strata_gpu_runtime_target})+ if(STRATA_BUILD_TESTS)+ add_test(NAME iq_nibble_parity_header COMMAND iq_nibble_parity --header)+ # the synthetic experts of every gate/up x down format pair, no model (skips with 77 on a card that is not a GP100)+ add_test(NAME iq_nibble_parity_synthetic COMMAND iq_nibble_parity --synthetic-only)+ set_tests_properties(iq_nibble_parity_synthetic PROPERTIES SKIP_RETURN_CODE 77 TIMEOUT 900)+ endif()+ endif()# MODEL-UPDATE: the dense Q8_0 GEMVs at the engine's shapes against the memory bandwidth (a benchmark, no test)add_executable(q8_gemv_bench src/kernels/q8_gemv_bench.cpp)target_include_directories(q8_gemv_bench PRIVATE ${_strata_gpu_include_directories})diff --git a/docs/OLDER_GPUS.md b/docs/OLDER_GPUS.mdindex 58118fafc..88f9f9758 100644--- a/docs/OLDER_GPUS.md+++ b/docs/OLDER_GPUS.md@@ -182,6 +182,50 @@ decode; keep two engines on different logical CPUs.`tools/q8_sm60_check.cu` checks the kernel against a CPU reference on these shapes (1-8 columns).+### Pascal GP100: 4-bit gate/up expert slots (opt-in, `STRATA_EXPERT_NIBBLE=1`)++On a GP100 die the verify window's gate/up expert kernels wait on each block's serial chain (the weights, two codebook+lookups per four weights, the emulated dp4a, the reduction), not on memory bandwidth. IQ2_S, IQ3_XXS and IQ3_S have+only 3 or 8 magnitudes in their grids, so every weight is one of at most 16 signed values: `STRATA_EXPERT_NIBBLE=1`+keeps those layers' gate/up experts in VRAM as 16 bytes of 4-bit codes per 32 weights (one 16-byte load), with the+integer scale and the fp16 `d` of each block kept as they are, and decodes four weights in 1-4 integer instructions. The+down rows stay GGUF blocks. The host copy, the pack and its tools do not change: the conversion runs on the GPU while+the slots are filled. Only the CUDA build with `STRATA_EXPERIMENTAL_SM60` has the kernels.++- **The answers are meant to be bit for bit the default's.** The 4-bit kernels run the production integer chain and float tail in+ the production order (`native_gu_multi_kernel`'s sub-16 rows), and SwiGLU, the q8_1 step and the down projection are+ the layout's own (mode 8 on a GP100). At every start each item is decoded with the production loader, encoded,+ decoded from the written code and compared, and the GGUF blocks are rebuilt from every slot and compared with the+ blob byte for byte; any difference refuses the start and names the layer. The prompt path's MMQ gathers the GGUF+ blocks rebuilt from the slots (the copy it makes today, with the 4-bit codes read instead): the same bytes. The kernels that+ existed before this change have the same SASS in the sm_60 build with and without it (446 kernels in `iq_kernels.cu` compared).+ `iq_nibble_parity --synthetic-only` checks the codec, the rebuilt blocks and the expert step on random experts of every format+ pair with no model file; `iq_nibble_parity <model shard>` does the same on a model's layers, against the GGUF blobs in mode 8 and+ in the CUDA layout (`STRATA_SM60_TABLE=0`). Both need a GP100.+- **It needs every expert in VRAM** on the card that runs its layer: a profile that ranks every pair, room for the+ larger slots (per layer of 512 experts: +175 MiB for IQ3_S, +250 MiB for IQ3_XXS, +350 MiB for IQ2_S), and a prompt+ path with its own buffers, no loan of cache slots (`--no-prefill-borrow` on one GPU; a layer split's stages keep+ their own when their cards have room, #340). `STRATA_EXPERT_NIBBLE_SKIP=<layers>` keeps layers in their GGUF slots;+ `STRATA_EXPERT_NIBBLE_FMTS=22,18,21` picks the formats.+- **It does not start with what it cannot serve.** With the switch set, a build without `STRATA_EXPERIMENTAL_SM60`, a card other+ than compute capability 6.0, `--peer-device`, `--expert-cache-device1..3`, `--kv-grow`, `--vram-elastic`, no `--expert-profile`,+ `--expert-cache 0`, `STRATA_MMQ_DIRECT`, `STRATA_PREFILL_MMQ=0`, `STRATA_PF_FUSED=1`, `STRATA_FS_SLOTS`, `STRATA_EXPERT_V2=1` or+ `STRATA_EXP_MODE` other than 0 or 8 make the engine say so in a warning and start with the usual GGUF slots. What is only known+ later (a layer with an expert out of VRAM, a loan of cache slots, a prompt chunk other than the one the run without the switch+ reads) still refuses the start with the reason. The adaptive tier is off with the switch. `STRATA_EXPERT_NIBBLE_RECHECK=1` (debug)+ hashes every 4-bit slot again after each request.+- **Opt-in `STRATA_NIB_GU_PROTO=1`** (with `STRATA_EXPERT_NIBBLE=1`): the 4-bit gate/up runs `nibx_gu_kernel`+ (`nib_gu_proto.cuh`), the same row dot with the group's activations staged once per two row blocks by 16-byte+ loads. The author measured, on one GP100 die, a layer's gate/up at 54-57 us instead of 67-72 us at one token (x1.24-1.27) and+ x1.22-1.39 at two to four tokens.+- The author measured the switch on a PH402 SKU 200 (two boards, four GP100 dies of 48 SMs and 32 GB each, clocks locked at 715 /+ 1050 MHz), Windows 11, CUDA 12.9, engine 0.1.45.3, Swift 1.5 IQ3_S (abliterated) pack, layer split 12,24,36,+ `--trim-stage-weights`, MTP with `--spec-min-p 0.85`, `--pipeline-windows 2`, 1M context, every expert resident: 47 of 48 layers+ converted (+12.7 GiB of VRAM over the four dies, +3.1 s of start-up). Decode of eight prompts (8K, 26K and 88K-token contexts),+ token-weighted: 47.3 -> 51.9 tok/s (+9.7%; each prompt +9.5..+10.9%). Prompt reading 2.3..3.0% slower (the gather rebuilds the+ GGUF blocks). The replies of 15 requests, including a 260K-token read, matched the engine without the switch. We have not+ measured it on this code.+## Tesla P40 (sm_61): what was measured and changedThree things run differently on compute capability 6.1 by default and a fourth is opt-in. Each gives the same bits as what itdiff --git a/include/strata/core/expert_cache.hpp b/include/strata/core/expert_cache.hppindex 9685aa4d5..7de956b99 100644--- a/include/strata/core/expert_cache.hpp+++ b/include/strata/core/expert_cache.hpp@@ -217,6 +217,14 @@ public:/// Slots filled so far, for the startup report.int64_t fills() const { return fills_; }+ /// STRATA_EXPERT_NIBBLE=1 (strata/core/expert_nibble.hpp): called once the startup fill has written every slot+ /// (some of them as 4-bit codes, not GGUF blocks) and checked them. From then on no slot may be written through+ /// this class: the fill_slot calls stop the engine (they would write GGUF bytes, possibly into a 4-bit slot, and a+ /// wrong expert gives a plausible token, never an error), and `admit` claims nothing. Never called with the switch+ /// off: an unsealed cache is the one above.+ void seal() { sealed_ = true; }+ bool sealed() const { return sealed_; }+/// The elastic K/V (--kv-grow): the arena in a VMM range (vmm.hpp) instead of one cudaMalloc, so the K/V can take/// single chunks of it and give them back. Applies to the next `open`; ignored where VMM is not available.static void set_vmm(bool enabled);@@ -266,6 +274,7 @@ private:std::vector<int32_t> layer_next_; ///< [n_layers] -> that layer's next free slotstd::vector<uint64_t> off_; ///< plan v0.3 P6: slot offsets (slots + 1 entries) when sizedint64_t admitted_ = 0;+ bool sealed_ = false; ///< STRATA_EXPERT_NIBBLE: `seal` was called (no slot is written after it)};} // namespace strata::corediff --git a/include/strata/core/expert_nibble.hpp b/include/strata/core/expert_nibble.hppnew file mode 100644index 000000000..920019567--- /dev/null+++ b/include/strata/core/expert_nibble.hpp@@ -0,0 +1,457 @@+// include/strata/core/expert_nibble.hpp - STRATA_EXPERT_NIBBLE=1 (opt-in, default off): the gate/up experts of+// IQ2_S / IQ3_XXS / IQ3_S layers kept as 4-bit codes in their VRAM slots, for Pascal GP100 (sm_60).+//+// **WHY.** On a GP100 the decode window's gate/up expert kernels are not held by memory bandwidth: they use a small+// part of it and wait on each block's serial chain (the weights, the codebook lookups, the emulated dp4a, the+// reduction). Re-encoding each 32-weight item of these three formats as sixteen bytes of 4-bit codes (one aligned+// 16-byte load) and keeping the integer scale and the fp16 d as they are makes the slot 1.25-1.68x the GGUF bytes and+// the decode of four weights 1-4 integer instructions instead of two codebook lookups.+//+// **WHY IT IS LOSSLESS.** Their grids have only 3 (IQ2_S) or 8 (IQ3_XXS, IQ3_S) magnitudes, so each int8 weight the+// production `Split<TG>::load` decodes is one of at most 16 signed values. At every start the transcoder decodes every+// item of every converted slot with the production load, encodes it, decodes the written code again and compares; and+// the GGUF blocks are rebuilt from every slot and compared with the blob byte for byte (`check_gguf`). The decode+// kernels run the production integer chain and float tail in the production order: every gate / up value is the+// production kernel's, bit for bit. The prompt path's MMQ reads the GGUF blocks rebuilt from the slot (`gather_gguf`,+// in place of the copy it does today): the same bytes, so the same bits.+//+// **OFF IS TODAY.** Unset or 0: no layer is converted, `slot_bytes(l)` is `expert_layout().blob_bytes(l)`, nothing is+// transcoded, sealed or hashed, and none of the new branches is entered. Only a CUDA build with+// STRATA_EXPERIMENTAL_SM60 has the kernels (STRATA_EXPERT_NIBBLE_BUILT); every other build refuses the switch at start.+#pragma once++#include "strata/kernels/cpu/expert_layout.hpp"++#include <cstddef>+#include <cstdint>+#include <cstdio>+#include <cstdlib>+#include <cstring>+#include <string>+#include <vector>++#if defined(STRATA_EXPERIMENTAL_SM60) && !defined(__HIPCC__) && !defined(STRATA_USE_HIP) && !defined(STRATA_HIP_GFX906) && \+ !defined(STRATA_GPU_METAL)+#define STRATA_EXPERT_NIBBLE_BUILT 1+#else+#define STRATA_EXPERT_NIBBLE_BUILT 0+#endif++#if defined(__CUDACC__) || defined(__HIPCC__)+#define STRATA_NIB_HD __host__ __device__+#else+#define STRATA_NIB_HD+#endif++namespace strata::core::nibble {++// ------------------------------------------------------------------------------------------------ the slot layout+//+// A converted layer's expert slot, in order (the host arena and the pack keep the GGUF blocks):+//+// codes kRows x kItems x 16 B row r, item k at (r * kItems + k) * 16; rows 0..kNFf-1 gate, kNFf..kRows-1 up+// scales kRows x kItems x 1 B row r, item k at kScOff + r * kItems + k (the production scale step's input)+// d kRows x kSuper x 2 B row r, superblock s at kDOff + (r * kSuper + s) * 2: the block's fp16 d, bit for bit+// ------ kGuBytes = 1,766,400 B (6900 x 256: the down part starts 256-aligned)+// down the GGUF down rows, copied verbatim from the raw blob's [down_off, bytes)+// ------ slot_bytes = round_up(kGuBytes + down bytes, kSlotAlign)+//+// One item = 32 weights of a row, one 16-byte code (four 32-bit words, see `encode_item`). Only the expert shape+// n_embd 2560, n_ff 640 is converted: every other layer keeps its GGUF blob and its kernels.++inline constexpr int64_t kNEmbd = 2560; ///< the gate/up rows' width (the activation)+inline constexpr int64_t kNFf = 640; ///< gate rows = up rows = n_ff+inline constexpr int64_t kRows = 2 * kNFf; ///< 1280: gate, then up+inline constexpr int64_t kItems = kNEmbd / 32; ///< 80 items of 32 weights per row+inline constexpr int64_t kSuper = kNEmbd / 256; ///< 10 superblocks (one fp16 d each) per row+inline constexpr uint64_t kItemBytes = 16; ///< one item's code: four words, one 16-byte load++inline constexpr uint64_t kCodesOff = 0;+inline constexpr uint64_t kUpCodesOff = (uint64_t) kNFf * kItems * kItemBytes; ///< 819,200: row kNFf+inline constexpr uint64_t kScOff = (uint64_t) kRows * kItems * kItemBytes; ///< 1,638,400+inline constexpr uint64_t kDOff = kScOff + (uint64_t) kRows * kItems; ///< 1,740,800+inline constexpr uint64_t kGuBytes = kDOff + (uint64_t) kRows * kSuper * 2; ///< 1,766,400+inline constexpr uint64_t kDownOff = kGuBytes; ///< the down rows+inline constexpr uint64_t kSlotAlign = 256; ///< every slot's size++static_assert(kGuBytes == 1766400, "the slot layout above");+static_assert(kGuBytes % kSlotAlign == 0, "the down part starts 256-aligned");+static_assert(kScOff % 16 == 0 && kDOff % 16 == 0, "each region starts 16-aligned");++/// Byte offset of row `row` (0..kRows-1; up rows are kNFf + r) item `k`'s code inside the slot.+STRATA_NIB_HD constexpr uint64_t code_off(int64_t row, int64_t k) {+ return kCodesOff + ((uint64_t) row * kItems + (uint64_t) k) * kItemBytes;+}+/// Byte offset of row `row` item `k`'s scale byte.+STRATA_NIB_HD constexpr uint64_t scale_off(int64_t row, int64_t k) { return kScOff + (uint64_t) row * kItems + (uint64_t) k; }+/// Byte offset of row `row` superblock `s`'s fp16 d (s = k / 8 for item k).+STRATA_NIB_HD constexpr uint64_t d_off(int64_t row, int64_t s) { return kDOff + ((uint64_t) row * kSuper + (uint64_t) s) * 2; }+/// A converted slot's size for a raw blob whose down rows are `down_bytes` long.+STRATA_NIB_HD constexpr uint64_t slot_bytes_for(uint64_t down_bytes) {+ return (kGuBytes + down_bytes + kSlotAlign - 1) / kSlotAlign * kSlotAlign;+}++// ------------------------------------------------------------------------------------------------ the formats++inline constexpr int kIQ3_XXS = 18;+inline constexpr int kIQ3_S = 21;+inline constexpr int kIQ2_S = 22;++/// Whether `gu_type` is converted (the three formats whose weights take at most 16 values).+STRATA_NIB_HD constexpr bool fmt_built(int gu_type) { return gu_type == kIQ2_S || gu_type == kIQ3_XXS || gu_type == kIQ3_S; }+/// The GGUF gate/up row bytes of `gu_type` at kNEmbd (a converted layer's raw blob must agree; 0 = not a format here).+STRATA_NIB_HD constexpr uint64_t raw_gu_row(int gu_type) {+ return gu_type == kIQ2_S ? 820 : gu_type == kIQ3_XXS ? 980 : gu_type == kIQ3_S ? 1100 : 0;+}+inline const char* fmt_name(int gu_type) {+ switch (gu_type) {+ case kIQ3_XXS: return "IQ3_XXS";+ case kIQ3_S: return "IQ3_S";+ case kIQ2_S: return "IQ2_S";+ default: return "?";+ }+}++// ------------------------------------------------------------------------------------------------ the item code+//+// `g[32]` below is one item as the production `Split<TG>::load` leaves it: g[4 * j + b] is byte b of W.g[j]+// (j = 0..7, the dp4a order against the q8_1 block's qs words). Word m (0..3) of the 16-byte code holds W.g[2m]+// and W.g[2m + 1]:+//+// IQ2_S bits 4b..4b+3 = c(g[8m + b]), bits 16+4b.. = c(g[8m + 4 + b]), b = 0..3+// c = magnitude index of {8, 25, 43} | sign << 2: the PRMT selector into {8,25,43,0} {-8,-25,-43,0}+// IQ3_XXS same nibble positions; c = index of {4, 12, 20, 28, 36, 44, 52, 62} | sign << 3+// IQ3_S byte b = u(g[8m + b]) | u(g[8m + 4 + b]) << 4, u = (v + 15) / 2 (v odd, -15..15); v = 2u - 15+//+// The scale byte keeps the production integer scale step's input:+// IQ2_S the block's scales byte for the item: ls0 | ls1 << 4 (W.ls0, W.ls1)+// IQ3_XXS aux32 >> 28 (0..15, W.ls)+// IQ3_S 1 + 2 * nibble (1..31, W.ls)++/// Encodes one item; false when a weight is outside the format's code set (the conversion then refuses to start).+inline bool encode_item(int gu_type, const int8_t g[32], uint32_t q[4]) {+ bool ok = true;+ auto code = [&](int8_t v) -> uint32_t {+ const int a = v < 0 ? -(int) v : (int) v;+ if (gu_type == kIQ3_S) {+ if ((v & 1) == 0 || v < -15 || v > 15) { ok = false; return 0; }+ return (uint32_t) (((int) v + 15) / 2);+ }+ if (gu_type == kIQ3_XXS) {+ static constexpr int mag[8] = { 4, 12, 20, 28, 36, 44, 52, 62 };+ for (uint32_t i = 0; i < 8; ++i)+ if (a == mag[i]) return i | (v < 0 ? 8u : 0u);+ ok = false;+ return 0;+ }+ const uint32_t m = a == 8 ? 0u : a == 25 ? 1u : a == 43 ? 2u : 3u; // IQ2_S+ if (m == 3u) { ok = false; return 0; }+ return m | (v < 0 ? 4u : 0u);+ };+ for (int m = 0; m < 4; ++m) {+ uint32_t w = 0;+ for (int b = 0; b < 4; ++b) {+ const uint32_t c0 = code(g[8 * m + b]), c1 = code(g[8 * m + 4 + b]);+ if (gu_type == kIQ3_S) w |= (c0 | c1 << 4) << (8 * b);+ else w |= c0 << (4 * b) | c1 << (16 + 4 * b);+ }+ q[m] = w;+ }+ return ok;+}++/// Decodes one item's code back to the production W.g bytes (the host reference the parity program compares with;+/// the device decoders give the same bytes). Codes outside the set decode as the PRMT does: IQ2_S index 3 -> 0.+inline void decode_item(int gu_type, const uint32_t q[4], int8_t g[32]) {+ static constexpr int8_t iq2[8] = { 8, 25, 43, 0, -8, -25, -43, 0 };+ static constexpr int8_t iq3[8] = { 4, 12, 20, 28, 36, 44, 52, 62 };+ for (int m = 0; m < 4; ++m)+ for (int b = 0; b < 4; ++b) {+ uint32_t c0, c1;+ if (gu_type == kIQ3_S) {+ c0 = q[m] >> (8 * b) & 15u;+ c1 = q[m] >> (8 * b + 4) & 15u;+ } else {+ c0 = q[m] >> (4 * b) & 15u;+ c1 = q[m] >> (16 + 4 * b) & 15u;+ }+ auto val = [&](uint32_t c) -> int8_t {+ if (gu_type == kIQ3_S) return (int8_t) (2 * (int) c - 15);+ if (gu_type == kIQ3_XXS) return (int8_t) ((c & 8u) ? -iq3[c & 7u] : iq3[c & 7u]);+ return iq2[c & 7u];+ };+ g[8 * m + b] = val(c0);+ g[8 * m + 4 + b] = val(c1);+ }+}++// ------------------------------------------------------------------------------------------------ the slot hash+//+// STRATA_EXPERT_NIBBLE_RECHECK's fingerprint of a slot: over the slot's first `hashed` bytes (kGuBytes plus the down+// rows; the alignment tail is not written and not hashed) as little-endian 32-bit words w_i, the last one+// zero-extended, H = sum_i (uint64) (w_i ^ i) * kHashMul mod 2^64. A sum, so the device's order of adding does not+// change it; the index term makes a moved word count.++inline constexpr uint64_t kHashMul = 0x9E3779B97F4A7C15ull;++/// The host reference of the slot hash (the device hash gives the same value for the same bytes).+inline uint64_t hash_host(const void* p, uint64_t bytes) {+ const uint8_t* s = (const uint8_t*) p;+ uint64_t h = 0;+ const uint64_t n = (bytes + 3) / 4;+ for (uint64_t i = 0; i < n; ++i) {+ uint32_t w = 0;+ const uint64_t at = 4 * i, take = bytes - at < 4 ? bytes - at : 4;+ std::memcpy(&w, s + at, (size_t) take); // little-endian hosts only+ h += (uint64_t) (w ^ (uint32_t) i) * kHashMul;+ }+ return h;+}++// ------------------------------------------------------------------------------------------------ the switches++struct Config {+ bool on = false; ///< STRATA_EXPERT_NIBBLE=1+ bool recheck = false; ///< STRATA_EXPERT_NIBBLE_RECHECK=1 (debug: every slot's hash per request)+ std::vector<int> fmts{ kIQ2_S, kIQ3_XXS, kIQ3_S }; ///< STRATA_EXPERT_NIBBLE_FMTS (default 22,18,21)+ std::vector<int64_t> skip; ///< STRATA_EXPERT_NIBBLE_SKIP: layers kept in their GGUF blocks+ std::string error; ///< non-empty: a malformed value - the start is refused+};++namespace detail {+/// "a,b,c" -> integers; false on anything else (empty = an empty list).+inline bool parse_list(const char* s, std::vector<int64_t>& out) {+ out.clear();+ while (s != nullptr && *s != '\0') {+ char* end = nullptr;+ const long long v = std::strtoll(s, &end, 10);+ if (end == s) return false;+ out.push_back((int64_t) v);+ s = end;+ if (*s == ',') ++s;+ else if (*s != '\0') return false;+ }+ return true;+}+} // namespace detail++/// The switches, read once (generate.cpp reads them before anything is allocated).+inline const Config& config() {+ static const Config c = [] {+ Config r;+ const char* e = std::getenv("STRATA_EXPERT_NIBBLE");+ if (e != nullptr && e[0] != '\0' && std::strcmp(e, "0") != 0) {+ if (std::strcmp(e, "1") == 0) r.on = true;+ else r.error = std::string("STRATA_EXPERT_NIBBLE=") + e + ": expected 0 or 1";+ }+ const char* rc = std::getenv("STRATA_EXPERT_NIBBLE_RECHECK");+ r.recheck = rc != nullptr && std::strcmp(rc, "1") == 0;+ std::vector<int64_t> v;+ if (const char* f = std::getenv("STRATA_EXPERT_NIBBLE_FMTS"); f != nullptr && f[0] != '\0') {+ if (!detail::parse_list(f, v)) {+ if (r.error.empty()) r.error = std::string("STRATA_EXPERT_NIBBLE_FMTS=") + f + ": expected ggml types, e.g. 22,18,21";+ } else {+ r.fmts.clear();+ for (const int64_t t : v) {+ if (!fmt_built((int) t) && r.error.empty())+ r.error = "STRATA_EXPERT_NIBBLE_FMTS: type " + std::to_string(t) ++ " is not converted (22 IQ2_S, 18 IQ3_XXS, 21 IQ3_S)";+ r.fmts.push_back((int) t);+ }+ }+ }+ if (const char* k = std::getenv("STRATA_EXPERT_NIBBLE_SKIP"); k != nullptr && k[0] != '\0') {+ if (!detail::parse_list(k, r.skip) && r.error.empty())+ r.error = std::string("STRATA_EXPERT_NIBBLE_SKIP=") + k + ": expected layer numbers, e.g. 40,44";+ }+ return r;+ }();+ return c;+}++/// With the switch on, the first environment setting it refuses, as a message (empty = none): the paths that would+/// read a converted slot as GGUF blocks, write GGUF bytes into one, or run other kernels for the converted layers+/// than the ones the 4-bit kernels are bit for bit. The command-line half (--peer-device, remote expert caches,+/// --kv-grow, --vram-elastic), the cc 6.0 check, full residency and no loan are generate.cpp's checks; the prompt+/// path's own (MMQ on every converted layer) is Prefill::nibble_prompt_check.+inline std::string env_conflicts() {+ auto set = [](const char* n) -> const char* {+ const char* v = std::getenv(n);+ return v != nullptr && v[0] != '\0' ? v : nullptr;+ };+ const char* v = set("STRATA_MMQ_DIRECT");+ if (v != nullptr && std::atoi(v) != 0)+ return "STRATA_MMQ_DIRECT (the prompt MMQ would read a converted slot in place as GGUF blocks)";+ v = set("STRATA_PREFILL_MMQ");+ if (v != nullptr && std::atoi(v) == 0) return "STRATA_PREFILL_MMQ=0 (the FP16 prompt path reads GGUF blocks)";+ v = set("STRATA_PF_FUSED");+ if (v != nullptr && v[0] == '1') return "STRATA_PF_FUSED=1 (the fused prompt kernels read GGUF blocks)";+ v = set("STRATA_FS_SLOTS");+ if (v != nullptr && std::atoi(v) > 0) return "STRATA_FS_SLOTS (Foresight swaps GGUF blobs into slots)";+ v = set("STRATA_EXPERT_V2");+ if (v != nullptr && v[0] == '1') return "STRATA_EXPERT_V2=1 (the S26 kernels, not the CUDA layout, on IQ3_S)";+ v = set("STRATA_KV_GROW");+ if (v != nullptr && v[0] != '0') return "STRATA_KV_GROW (the elastic K/V moves slots at their GGUF size)";+ v = set("STRATA_EXP_MODE");+ if (v != nullptr && std::atoi(v) != 0 && std::atoi(v) != 8)+ return "STRATA_EXP_MODE other than 0 or 8 (another layout's gate/up kernels)";+ return {};+}++// ------------------------------------------------------------------------------------------------ the VRAM table++/// One layer's slots. `on` false: the GGUF blob, `slot_bytes == raw_bytes`, the production kernels.+struct LayerVram {+ bool on = false;+ int gu_type = -1, d_type = -1;+ uint64_t down_off = 0; ///< in the slot: kDownOff when on, else the raw blob's down_off+ uint64_t down_bytes = 0; ///< the down rows (raw_bytes - raw_down_off)+ uint64_t hashed = 0; ///< bytes the slot hash covers: kGuBytes + down_bytes when on, else 0+ uint64_t slot_bytes = 0; ///< what the expert cache allocates per slot (`open_sized`, every estimate)+ // the raw blob as the arena and the pack hold it (expert_layout().fmt[l]); the transcoder reads these+ uint64_t raw_bytes = 0, raw_gu_row = 0, raw_up_off = 0, raw_down_off = 0;+};++/// One layer's entry from its GGUF format and the switches (pure; `plan_layers` calls it per layer).+inline LayerVram plan_layer(int64_t l, const strata::kernels::cpu::NativeFmt& f, const Config& c) {+ LayerVram v;+ v.gu_type = f.gu_type;+ v.d_type = f.d_type;+ v.raw_bytes = f.bytes;+ v.raw_gu_row = f.gu_row;+ v.raw_up_off = f.up_off;+ v.raw_down_off = f.down_off;+ v.down_bytes = f.bytes - f.down_off;+ v.down_off = f.down_off;+ v.slot_bytes = f.bytes;+ bool listed = false, skipped = false;+ for (const int t : c.fmts) listed = listed || t == f.gu_type;+ for (const int64_t s : c.skip) skipped = skipped || s == l;+ // [gate | up | down] exactly, no NVFP4 tail, the down rows a whole number of 16-byte words (the gathers copy them so)+ const bool shape = f.n_embd == kNEmbd && f.n_ff == kNFf && f.gu_row == raw_gu_row(f.gu_type) &&+ f.up_off == (uint64_t) kNFf * f.gu_row && f.down_off == 2 * f.up_off && f.bytes > f.down_off &&+ f.tail_off == 0 && (f.bytes - f.down_off) % 16 == 0;+ if (c.on && c.error.empty() && listed && !skipped && fmt_built(f.gu_type) && shape) {+ v.on = true;+ v.down_off = kDownOff;+ v.hashed = kGuBytes + v.down_bytes;+ v.slot_bytes = slot_bytes_for(v.down_bytes);+ }+ return v;+}++/// Every layer's entry for this pack (all off for a Q2_0 pack or with the switch off). `err`: a SKIP layer out of+/// range, or the switch on with a Q2_0 pack.+inline std::vector<LayerVram> plan_layers(const strata::kernels::cpu::ExpertLayout& lay, std::string& err) {+ const Config& c = config();+ std::vector<LayerVram> out((size_t) (lay.n_layers > 0 ? lay.n_layers : 0));+ for (int64_t l = 0; l < (int64_t) out.size(); ++l) {+ if (lay.native && (size_t) l < lay.fmt.size()) out[(size_t) l] = plan_layer(l, lay.fmt[(size_t) l], c);+ else out[(size_t) l].slot_bytes = out[(size_t) l].raw_bytes = lay.blob_bytes(l);+ }+ if (c.on && !lay.native) err = "STRATA_EXPERT_NIBBLE needs a native pack (native_experts.txt)";+ for (const int64_t s : c.skip)+ if (c.on && (s < 0 || s >= lay.n_layers) && err.empty())+ err = "STRATA_EXPERT_NIBBLE_SKIP: layer " + std::to_string(s) + " is not in the model";+ return out;+}++namespace detail {+inline std::vector<LayerVram>& table() {+ static std::vector<LayerVram> t;+ return t;+}+} // namespace detail++/// generate.cpp only: the table, once, after `expert_layout_load` and BEFORE the first slot estimate (placement+/// search, can_start, the cache's auto size, every `open_sized`). Not thread-safe: set before any other thread.+inline void set_layers(std::vector<LayerVram> v) { detail::table() = std::move(v); }++/// Layer `l`'s entry (an off entry when the table is not set or `l` is outside it). Read-only after `set_layers`.+inline const LayerVram& layer(int64_t l) {+ static const LayerVram off;+ const auto& t = detail::table();+ return l >= 0 && (size_t) l < t.size() ? t[(size_t) l] : off;+}++/// Whether any layer is converted (the switch on and the table set).+inline bool any_on() {+ for (const LayerVram& v : detail::table())+ if (v.on) return true;+ return false;+}++/// Bytes of one of layer `l`'s VRAM slots: the converted size for a converted layer, else the layout's blob bytes+/// (exactly `expert_layout().blob_bytes(l)`: with the switch off every caller sizes what it sized before).+inline uint64_t slot_bytes(int64_t l) {+ const auto& t = detail::table();+ if (l >= 0 && (size_t) l < t.size()) return t[(size_t) l].slot_bytes;+ return strata::kernels::cpu::expert_layout().blob_bytes(l);+}++// ------------------------------------------------------------------------------------------------ fatal paths++/// A converted layer reached a path that would compute or copy it as GGUF blocks: say where and stop. Never a+/// silent wrong answer.+[[noreturn]] inline void fatal(int64_t l, const char* where, const char* why) {+ std::fprintf(stderr, "strata: STRATA_EXPERT_NIBBLE: layer %lld (%s, 4-bit slots) reached %s: %s - stopping\n",+ (long long) l, fmt_name(layer(l).gu_type), where != nullptr ? where : "?", why != nullptr ? why : "");+ std::fflush(stderr);+ std::abort();+}++/// Before each raw slot copy that bypasses the ExpertCache API (the adaptive tier's copies, the elastic K/V's slot+/// moves): fatal when `l` is a converted layer. The startup refusals keep these paths off; this is the backstop+/// (with `ExpertCache::seal()` for the API's own fills).+inline void guard_raw(int64_t l, const char* where) {+ if (layer(l).on) fatal(l, where, "a raw slot copy at the GGUF size");+}++// ------------------------------------------------------------------------------------------------ the decode kernels+//+// Two skeletons in the shape of native_gu_multi_kernel (16 lanes a row, a group's entries GRP_NC at a time), both bit+// for bit the production gate/up: S stages the entries' q8_1 blocks in shared memory and decodes each item where it+// is used (IQ3_S: v = 2u - 15 restored per byte, then the production apply); P loads the lane's codes, scales and d+// before staging the qs words as 16-byte rows (IQ3_S: sum = 2 dot(u, x) - 15 sum(x), the same int32). Per format the+// faster one on 4x GP100 at T = 1..6 (native_expert_bench --nibble): IQ2_S P, IQ3_XXS S, IQ3_S S.++inline constexpr int kSkelS = 0, kSkelP = 1;+STRATA_NIB_HD constexpr int gu_skeleton(int gu_type) { return gu_type == kIQ2_S ? kSkelP : kSkelS; }++// ------------------------------------------------------------------------------------------------ iq_kernels.cu+//+// Device addresses are `unsigned long long` (as native_expert_grouped's grp_ptr), streams `void*`; each returns the+// CUDA error code (0 = success) so the caller can name the layer. A build without STRATA_EXPERT_NIBBLE_BUILT+// returns cudaErrorInvalidValue (1) from each.++/// The reverse codebooks (grid magnitudes -> GGUF grid index) `check_gguf` and `gather_gguf` use, built on the CURRENT+/// device (once per device; call it at load, before any capture). Non-zero also when a grid entry is not unique.+int prepare(void* stream);++/// Converts `n` raw blobs of layer `L` (device `staging`, `L.raw_bytes` apart, as the host arena holds them) into+/// the converted slots at `slot_ptrs[0..n)` (a device array): codes, scales, d. The down rows are not touched (the+/// caller copies them to `slot + kDownOff`). Every item is decoded with the production `Split<TG>::load`, encoded,+/// decoded from the written code again and compared with W.g[0..7], the scale and d; each item that differs or holds+/// a weight outside the code set adds 1 to `*bad` (device). L.on must be true.+int transcode(const LayerVram& L, const void* staging, int64_t n, const unsigned long long* slot_ptrs,+ unsigned long long* bad, void* stream);++/// The GGUF gate/up blocks rebuilt from each of the `n` (<= 64) converted slots at `slot_ptrs` and its down rows,+/// compared with the raw blobs in `staging` (`L.raw_bytes` apart) byte for byte: each 32-bit word that differs adds 1+/// to `*bad`. `prepare` first.+int check_gguf(const LayerVram& L, const void* staging, int64_t n, const unsigned long long* slot_ptrs,+ unsigned long long* bad, void* stream);++/// The prompt path's gather of a converted layer's experts (prefill.cpp, MMQ): for q in [first, n) (n <= 16), the+/// GGUF [gate | up] blocks rebuilt from slot `slots[q]` (host array of device addresses) into gu_dst + q * gu_stride -+/// the bytes the raw blob holds, byte for byte - and its down rows copied to d_dst + q * d_stride; `zero_bytes` zeroed+/// after expert n - 1's gate/up and down (MMQ's tail). Every address, stride and size 16-byte aligned. `prepare`+/// on this device first.+int gather_gguf(const LayerVram& L, const uint8_t* const* slots, int first, int n, void* gu_dst, uint64_t gu_stride,+ void* d_dst, uint64_t d_stride, uint64_t zero_bytes, void* stream);++/// The slot hash (`hash_host`'s value) of the `L.hashed` bytes at each `slot_ptrs[i]`, into device `out[0..n)`.+int hash_slots(const LayerVram& L, const unsigned long long* slot_ptrs, int64_t n, unsigned long long* out,+ void* stream);++} // namespace strata::core::nibblediff --git a/include/strata/kernels/iq_kernels.hpp b/include/strata/kernels/iq_kernels.hppindex ce06a9fe7..b1f5ae5ac 100644--- a/include/strata/kernels/iq_kernels.hpp+++ b/include/strata/kernels/iq_kernels.hpp@@ -51,9 +51,16 @@ struct NativeExpertLayout {// APR (prism.hadamard): the down weights are Hadamard-folded, so native_expert_grouped rotates h before// quantizing it (hadamard_rows). had_block 0 = none; d_signs is a DEVICE vector of n_ff signs or nullptr.int had_block = 0;+ bool gu_nib = false; // STRATA_EXPERT_NIBBLE: gate/up as 4-bit codes (strata/core/expert_nibble.hpp's+ // slot); set only by native_expert_layout_nibconst float* d_signs = nullptr;};NativeExpertLayout native_expert_layout(int gu_type, int d_type, int64_t n_embd, int64_t n_ff);+/// STRATA_EXPERT_NIBBLE: a converted VRAM slot - gate/up as 4-bit codes, scales and d (expert_nibble.hpp's layout:+/// up_off = the up rows' codes, down_off = kGuBytes, bytes = the slot), the GGUF down rows after them. IQ2_S /+/// IQ3_XXS / IQ3_S at n_embd 2560, n_ff 640 only, and only in a build with the kernels (`native_expert_nib_supported`).+NativeExpertLayout native_expert_layout_nib(int gu_type, int d_type, int64_t n_embd, int64_t n_ff);+bool native_expert_nib_supported(int gu_type, int d_type, int64_t n_embd, int64_t n_ff) noexcept;/// Whether `native_expert_grouped` has kernels for this gate/up and down type pair at these dimensions, and the/// prompt path's dequantizer takes both (checked for every layer at startup, before anything is allocated).bool native_expert_supported(int gu_type, int d_type, int64_t n_embd, int64_t n_ff) noexcept;@@ -83,9 +90,15 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long/// per possible group, SwiGLU and the q8_1 quantization as two kernels over all cap_entries. Bitwise the same results/// (native_grouped_parity checks it); kept for A/B timing. Set before graph capture; captured graphs keep theirs.void native_grouped_set_v1(bool v1);+/// The bench / parity only: the 4-bit gate/up kernel of every format (-1 = expert_nibble.hpp's gu_skeleton, the+/// default; 0 = the S skeleton; 1 = the P skeleton). Bitwise the same results; set before graph capture.+void native_expert_nib_set_variant(int v);+/// gu-proto (nib_gu_proto.cuh): variant v's registers, local bytes and resident 256-thread blocks per SM for gu_type;+/// false when v is not a gu-proto id (>= 10) or the type has none.+bool native_expert_nib_variant_attr(int v, int gu_type, int* regs, int* local_bytes, int* blocks_per_sm);/// The bench only: the AMD kernel layout (STRATA_EXP_MODE values; -1 = the environment's) and the phase-/// (0 all, 1 gate/up + SwiGLU + quantize, 2 down).+/// (0 all, 1 gate/up + SwiGLU + quantize, 2 down, 3 gate/up only).void native_expert_set_mode(int mode, int phase);/// APR (prism.hadamard): y = H_block (signs * x) per block of `block` values of each of `rows` rows of `width`diff --git a/include/strata/prefill/prefill.hpp b/include/strata/prefill/prefill.hppindex 39ed85577..fa75c31d6 100644--- a/include/strata/prefill/prefill.hpp+++ b/include/strata/prefill/prefill.hpp@@ -169,6 +169,11 @@ public:/// build with the pointer-table MMQ (STRATA_MMQ_TAB).static size_t slot_guard_bytes();+ /// STRATA_EXPERT_NIBBLE=1: whether every converted layer's experts reach the prompt path's MMQ gather only (the+ /// GGUF blocks rebuilt from the 4-bit slots); false with the first layer and the reason in `why`. Pure: the+ /// layers' formats, the environment and the visible GPUs (generate.cpp asks once, before anything is allocated).+ static bool nibble_prompt_check(std::string& why);+/// Positions [pos0, pos0 + n) holding `tokens`; `ss.ple_prev` must be the two tokens before pos0 (oldest/// first, -1 for none) and is advanced to the last two of these.bool run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& err);diff --git a/src/core/expert_cache.cpp b/src/core/expert_cache.cppindex bd98ad711..72523b9cd 100644--- a/src/core/expert_cache.cpp+++ b/src/core/expert_cache.cpp@@ -24,6 +24,17 @@namespace strata::core {+namespace {+// STRATA_EXPERT_NIBBLE: a write into a sealed cache (see ExpertCache::seal). Says which call and stops: the slots hold+// 4-bit codes the GGUF bytes would overwrite, or a GGUF expert where a 4-bit one is expected - a wrong reply, not an error.+[[noreturn]] void sealed_write(const char* call, int32_t slot) {+ std::fprintf(stderr, "strata: STRATA_EXPERT_NIBBLE: ExpertCache::%s(slot %d) after the cache was sealed - a GGUF "+ "blob would be written into the 4-bit expert slots: stopping\n", call, (int) slot);+ std::fflush(stderr);+ std::abort();+}+} // namespace+// See the header. NVIDIA's DGX Spark Porting Guide (section 5.5, "Memory reporting on UMA systems") recommends the same:// not to rely on cudaMemGetInfo alone but to count the memory the OS can reclaim. Swap is not counted here (unlike// NVIDIA's reference snippet): an expert cache that pushes the system into swap would be far slower than a smaller one.@@ -621,6 +632,7 @@ void ExpertCache::close() {next_free_ = 0;fills_ = 0;admitted_ = 0;+ sealed_ = false;layer_next_.clear();}@@ -641,6 +653,7 @@ int32_t ExpertCache::slot_of(int64_t layer, int64_t expert) const {}int32_t ExpertCache::admit(int64_t layer, int64_t expert) {+ if (sealed_) return kNotResident; // STRATA_EXPERT_NIBBLE: nothing is admitted (and so filled) after the startup fillif (layer < 0 || layer >= n_layers_ || expert < 0 || expert >= n_expert_) return kNotResident;const size_t at = (size_t) (layer * n_expert_ + expert);if (residency_[at] != kNotResident) return residency_[at];@@ -673,6 +686,7 @@ const uint8_t* ExpertCache::device_slot(int32_t slot) const {}bool ExpertCache::fill_slot(int32_t slot, const uint8_t* host_blob, void* stream, std::string& err, int64_t bytes) {+ if (sealed_) sealed_write("fill_slot", slot);const size_t n = (size_t) (bytes > 0 && bytes <= blob_ ? bytes : blob_);uint8_t* dst = device_slot(slot);if (dst == nullptr) {@@ -695,6 +709,7 @@ bool ExpertCache::fill_slot(int32_t slot, const uint8_t* host_blob, void* stream}bool ExpertCache::fill_slot_blocking(int32_t slot, const uint8_t* host_blob, std::string& err, int64_t bytes) {+ if (sealed_) sealed_write("fill_slot_blocking", slot);const size_t n = (size_t) (bytes > 0 && bytes <= blob_ ? bytes : blob_);uint8_t* dst = device_slot(slot);if (dst == nullptr) {@@ -726,6 +741,7 @@ bool ExpertCache::fill_slot_blocking(int32_t slot, const uint8_t* host_blob, std}bool ExpertCache::fill_slot_queued(int32_t slot, const uint8_t* host_blob, std::string& err, int64_t bytes) {+ if (sealed_) sealed_write("fill_slot_queued", slot);const size_t n = (size_t) (bytes > 0 && bytes <= blob_ ? bytes : blob_);uint8_t* dst = device_slot(slot);if (dst == nullptr || host_blob == nullptr) {@@ -743,6 +759,7 @@ bool ExpertCache::fill_slot_queued(int32_t slot, const uint8_t* host_blob, std::}bool ExpertCache::fill_slot_queued_parts(int32_t slot, const BlobPart* parts, int n, std::string& err) {+ if (sealed_) sealed_write("fill_slot_queued_parts", slot);uint8_t* dst = device_slot(slot);if (dst == nullptr || parts == nullptr || n <= 0) {err = "ExpertCache::fill_slot_queued_parts: slot outside the arena or no pieces";diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cppindex b8ad8fb42..ec258c89a 100644--- a/src/core/expert_source.cpp+++ b/src/core/expert_source.cpp@@ -1,5 +1,6 @@// src/core/expert_source.cpp - the adapter. See the header for the three clauses of the contract.#include "strata/core/expert_source.hpp"+#include "strata/core/expert_nibble.hpp"#include "strata/core/foresight_swap.hpp"#include "strata/core/on_device.hpp"#include "strata/platform/aux_cpus.hpp"@@ -5221,6 +5222,14 @@ void expert_hit_run(void* user, void* stream, HitPhase phase, const int32_t* idsd.hit_pending = false;if (!d.hits_ready() || ids == nullptr || k <= 0) return;if ((int64_t) d.is_hit.size() < k) d.is_hit.resize((size_t) k);+ // STRATA_EXPERT_NIBBLE=1: this split computes a resident expert with the GGUF-block kernels and admits + fills+ // a missed one with its GGUF blob - neither may meet a layer whose slots hold 4-bit codes (a native pack's+ // experts run in the verify windows, so this is the backstop, never the path)+#if STRATA_EXPERT_NIBBLE_BUILT // compiled out of every other build: nothing is added to the hit split there+ if (strata::core::nibble::layer(d.layers).on)+ strata::core::nibble::fatal(d.layers, "the expert source's hit split (expert_hit_run)",+ "it computes resident experts as GGUF blocks and fills slots with GGUF blobs");+#endif// ================================ THE DECISION, ONCE, ON THIS LAYER'S IDS ================================//diff --git a/src/core/verify.cpp b/src/core/verify.cppindex 06306a239..727fea0e1 100644--- a/src/core/verify.cpp+++ b/src/core/verify.cpp@@ -4,6 +4,7 @@#include "strata/core/sm120_series.hpp"#include "strata/kernels/gfx_arch.hpp"#include "strata/core/verify.hpp"+#include "strata/core/expert_nibble.hpp"#include "strata/core/pin_budget.hpp"#if !defined(STRATA_USE_HIP) && !defined(STRATA_HIP_GFX906)#include "strata/core/host_graph_launch.hpp"@@ -2127,11 +2128,21 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {const auto& lay = strata::kernels::cpu::expert_layout();// plan v0.3 P6: the VRAM groups now; the PCIe groups once the copy engine has landed them in staging.// `gy`: the native launch's groups side by side (0: cap, one block row per possible group).- auto grouped = [&](const unsigned long long* gp, const int32_t* gs, const int32_t* gn, int64_t gy, float* dst_buf) {+ // `vram`: the groups point into this GPU's expert slots - a layer STRATA_EXPERT_NIBBLE converted holds its+ // gate/up there as 4-bit codes (expert_nibble.hpp); the PCIe share's blobs (host memory, staging) never do.+ auto grouped = [&](const unsigned long long* gp, const int32_t* gs, const int32_t* gn, int64_t gy, float* dst_buf,+ bool vram) {if (lay.native) {// the layer's GGUF formats (i-quant gate/up, Q2_0 / IQ4_NL down)const auto& f = lay.fmt[(size_t) l];+#if STRATA_EXPERT_NIBBLE_BUILT+ NativeExpertLayout L = vram && strata::core::nibble::layer(l).on+ ? native_expert_layout_nib(f.gu_type, f.d_type, f.n_embd, f.n_ff)+ : native_expert_layout(f.gu_type, f.d_type, f.n_embd, f.n_ff);+#else+ (void) vram;NativeExpertLayout L = native_expert_layout(f.gu_type, f.d_type, f.n_embd, f.n_ff);+#endifif (f.had_d) { L.had_block = f.had_block; L.d_signs = had_sh_; } // APR: h is rotated in therenative_expert_grouped(L, gp, gs, gn, p_dst, p_tok, cap, cap,nat_xq_ + (size_t) tb * (N / 32) * 36, hit_scratch_, dst_buf, cs, gy);@@ -2143,7 +2154,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {bool hits_in_combine = false;if (ar_on()) {stamp(l, 19, grp);- grouped(p_ptr, p_start, p_counts, 0, parts_out);+ grouped(p_ptr, p_start, p_counts, 0, parts_out, true);stamp(l, 20, grp);} else {if (device_plan_) { // E-6: skipped when the device planned this group (all its experts resident)@@ -2161,7 +2172,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {}}stamp(l, 19, grp);- grouped(p_ptr, p_start, p_counts, 0, hit_out);+ grouped(p_ptr, p_start, p_counts, 0, hit_out, true);stamp(l, 20, grp);if (device_plan_) wait_flag_ge_or(m_flagB_, ring, skip_ + grp, cs);else if (m_bal_ != nullptr) // set_pcie_balance: the PCIe part starts@@ -2176,7 +2187,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {stamp(l, 21, grp);// the PCIe share is pcie_frac of the misses: a few groups when the cache is cold, usually none (always none at// pcie_frac 0), so its launch is kPcieGroupRows block rows striding over the groups, not cap of them- grouped(p_ptr2, p_start2, p_counts + 2, kPcieGroupRows, hit_out);+ grouped(p_ptr2, p_start2, p_counts + 2, kPcieGroupRows, hit_out, false);stamp(l, 22, grp);if (device_plan_) { // no CPU share when the device planned the group: its rows are zeroswait_flag_ge_or(m_flag_, ring, skip_ + grp, cs);diff --git a/src/kernels/cuda/iq_kernels.cu b/src/kernels/cuda/iq_kernels.cuindex 9cb6894fb..50f0a839f 100644--- a/src/kernels/cuda/iq_kernels.cu+++ b/src/kernels/cuda/iq_kernels.cu@@ -5,6 +5,7 @@// MIT license, third_party/ggml/LICENSE). The block structs and codebook grids come from its ggml-common.h,// included unchanged.#include "strata/kernels/iq_kernels.hpp"+#include "strata/core/expert_nibble.hpp"#include "strata/core/sm75_series.hpp"#include "strata/core/sm120_series.hpp"#include "strata/kernels/gfx_arch.hpp"@@ -4476,6 +4477,704 @@ void launch_down_ws(const NativeExpertLayout& L, int64_t gy, int64_t cap_groups,#endif // STRATA_DOWN_WS_BUILT+// ---------------------------------------------------------------- STRATA_EXPERT_NIBBLE: gate/up on 4-bit slots+// (opt-in, default off; strata/core/expert_nibble.hpp has the slot layout and the item code). A converted layer's+// VRAM slot holds each 32-weight item of a gate/up row as 16 bytes of 4-bit codes (one 16-byte load), its integer+// scale byte and the superblock's fp16 d bit for bit; the kernels below decode an item into the production+// Split<TG>::W (or IQ3_S's offset form) with 1-4 instructions per four weights instead of the codebook lookups, then+// run the production integer chain and float tail in the production lane / item / addition order (row_dot_80_sub16:+// k = t, t+32, t+64 into s, t+16, t+48 into s16, __fadd_rn, xor 8, 4, 2, 1), so every gate / up value is bit for bit+// native_gu_multi_kernel<TG, *, true>'s. Two skeletons in the shape of native_gu_multi_kernel (grid (80, gy), 256+// threads, 16 lanes a row, a group's entries GRP_NC at a time, the group stride):+// S the entries' q8_1 blocks staged in shared memory, each item's code loaded and decoded where it is used, the+// production apply on it (IQ3_S: v = 2u - 15 restored per byte - no sum of x)+// P the lane's five items' codes, scales and d loaded before the staging (in flight during it), the entries' qs+// words staged as 16-byte rows and ds as a float, the production chain written out on the registers (IQ3_S:+// sumi = 2 dot(u, x) - 15 sum(x) with sum(x) from the staged words: the same int32)+// Then the load-time transcoder, the GGUF rebuild (the start-up check and the prompt path's gather) and the slot hash.+// Built only with STRATA_EXPERT_NIBBLE_BUILT (a CUDA build with STRATA_EXPERIMENTAL_SM60); elsewhere a gu_nib layout+// stops in native_expert_grouped, and the switch is refused at start.+#if STRATA_EXPERT_NIBBLE_BUILT+namespace nibk = strata::core::nibble;++__device__ __forceinline__ uint32_t nib_prmt(uint32_t a, uint32_t b, uint32_t sel) {+ uint32_t r;+ asm("prmt.b32 %0, %1, %2, %3;" : "=r"(r) : "r"(a), "r"(b), "r"(sel)); // the sign-replicate modes too+ return r;+}++// One item's code -> the production W (what Split<TG>::load gives for the same GGUF item, byte for byte).+template<int TG> struct NibLoad;+template<> struct NibLoad<22> { // IQ2_S: the nibble is the PRMT selector into {8,25,43,0} {-8,-25,-43,0}+ using W = Split<22>::W;+ __device__ static W load(uint4 q, uint32_t sc, __half d) {+ constexpr uint32_t T0 = 0x002B1908u, T1 = 0x00D5E7F8u;+ W r;+ const uint32_t qw[4] = { q.x, q.y, q.z, q.w };+#pragma unroll+ for (int m = 0; m < 4; ++m) {+ r.g[2 * m + 0] = (int) nib_prmt(T0, T1, qw[m]);+ r.g[2 * m + 1] = (int) nib_prmt(T0, T1, qw[m] >> 16);+ }+ r.ls0 = sc & 0x0F;+ r.ls1 = sc >> 4;+ r.dw = __half2float(d);+ return r;+ }+};+template<> struct NibLoad<18> { // IQ3_XXS: magnitude PRMT into {4..62}, sign-replicated byte masks, (g ^ m) + (m & 1s)+ using W = Split<18>::W;+ __device__ static W load(uint4 q, uint32_t sc, __half d) {+ constexpr uint32_t T0 = 0x1C140C04u, T1 = 0x3E342C24u; // {4,12,20,28} {36,44,52,62}+ W r;+ const uint32_t qw[4] = { q.x, q.y, q.z, q.w };+#pragma unroll+ for (int m = 0; m < 4; ++m) {+ const uint32_t w = qw[m], ms = w & 0x77777777u, a = w << 4;+ const uint32_t m0 = nib_prmt(a, w, 0xD9C8u), m1 = nib_prmt(a, w, 0xFBEAu);+ const uint32_t g0 = nib_prmt(T0, T1, ms), g1 = nib_prmt(T0, T1, ms >> 16);+ r.g[2 * m + 0] = (int) ((g0 ^ m0) + (m0 & 0x01010101u));+ r.g[2 * m + 1] = (int) ((g1 ^ m1) + (m1 & 0x01010101u));+ }+ r.ls = (int) sc;+ r.dw = __half2float(d);+ return r;+ }+};+template<> struct NibLoad<21> { // IQ3_S: v = 2u - 15 per byte, ((2u | 0x80) - 0x0F) ^ 0x80 (every byte >= 0x80: no borrow)+ using W = Split<21>::W;+ __device__ static W load(uint4 q, uint32_t sc, __half d) {+ W r;+ const uint32_t qw[4] = { q.x, q.y, q.z, q.w };+#pragma unroll+ for (int m = 0; m < 4; ++m) {+ const uint32_t lo2 = (qw[m] << 1) & 0x1E1E1E1Eu, hi2 = (qw[m] >> 3) & 0x1E1E1E1Eu; // 2u per byte+ r.g[2 * m + 0] = (int) (((lo2 | 0x80808080u) - 0x0F0F0F0Fu) ^ 0x80808080u);+ r.g[2 * m + 1] = (int) (((hi2 | 0x80808080u) - 0x0F0F0F0Fu) ^ 0x80808080u);+ }+ r.ls = (int) sc;+ r.dw = __half2float(d);+ return r;+ }+};++// S skeleton row dot over one converted row (codes, scales, d of row `row`), x the staged blocks+template<int TG, int NC>+__device__ __forceinline__ void nib_row_dot_s(const uint4* __restrict__ crow, const uint8_t* __restrict__ srow,+ const __half* __restrict__ drow, const block_q8_1* x, const int (&off)[NC],+ int t, float (&s)[NC]) {+ using Q = NibLoad<TG>;+ const int iqs = 2 * (t & 7), kb = t >> 3;+ const block_q8_1* xb[NC];+#pragma unroll+ for (int c = 0; c < NC; ++c) { s[c] = 0.0f; xb[c] = x + off[c] + kb * 8; }+#pragma unroll+ for (int i = 0; i < 3; ++i) {+ const typename Q::W w = Q::load(crow[t + 32 * i], srow[t + 32 * i], drow[kb + 4 * i]);+#pragma unroll+ for (int c = 0; c < NC; ++c) s[c] += Split<TG>::apply(w, xb[c] + 32 * i, iqs);+ }+ float s16[NC];+#pragma unroll+ for (int c = 0; c < NC; ++c) s16[c] = 0.0f;+#pragma unroll+ for (int i = 0; i < 2; ++i) {+ const typename Q::W w = Q::load(crow[t + 16 + 32 * i], srow[t + 16 + 32 * i], drow[kb + 2 + 4 * i]);+#pragma unroll+ for (int c = 0; c < NC; ++c) s16[c] += Split<TG>::apply(w, xb[c] + 16 + 32 * i, iqs);+ }+#pragma unroll+ for (int c = 0; c < NC; ++c) {+ s[c] = __fadd_rn(s[c], s16[c]);+#pragma unroll+ for (int o = 8; o > 0; o >>= 1) s[c] += __shfl_xor_sync(0xffffffffu, s[c], o);+ }+}++// One pass of the S skeleton: the n (1..GRP_NC) entries from e staged, then this thread's row against them.+// Block-uniform (e, n): every thread reaches both barriers.+template<int TG>+__device__ __forceinline__ void nib_pass_s(block_q8_1* s_x, const uint4* crow, const uint8_t* srow, const __half* drow,+ const int32_t* __restrict__ ent_tok, const block_q8_1* __restrict__ xq,+ int e, int n, int t, float* dst, int r) {+ int tok[GRP_NC];+#pragma unroll+ for (int c = 0; c < GRP_NC; ++c) tok[c] = ent_tok[e + min(c, n - 1)];+ __syncthreads();+ for (int i = threadIdx.x; i < n * 80 * 9; i += 256) { // 80 blocks x 36 B an entry, as 9 words a block+ const int c = i / 720, rem = i % 720;+ ((int*) s_x)[c * 720 + rem] = ((const int*) (xq + (size_t) tok[c] * 80))[rem];+ }+ __syncthreads();+ if (n == 1) {+ const int o[1] = { 0 }; float s[1];+ nib_row_dot_s<TG, 1>(crow, srow, drow, s_x, o, t, s);+ if (t == 0) dst[(size_t) e * nibk::kNFf + r] = s[0];+ } else if (n == 2) {+ const int o[2] = { 0, 80 }; float s[2];+ nib_row_dot_s<TG, 2>(crow, srow, drow, s_x, o, t, s);+ if (t < 2) dst[(size_t) (e + t) * nibk::kNFf + r] = s[t];+ } else if (n == 3) {+ const int o[3] = { 0, 80, 160 }; float s[3];+ nib_row_dot_s<TG, 3>(crow, srow, drow, s_x, o, t, s);+ if (t < 3) dst[(size_t) (e + t) * nibk::kNFf + r] = s[t];+ } else {+ const int o[4] = { 0, 80, 160, 240 }; float s[4];+ nib_row_dot_s<TG, 4>(crow, srow, drow, s_x, o, t, s);+ if (t < 4) dst[(size_t) (e + t) * nibk::kNFf + r] = s[t];+ }+}++// P skeleton: x as two LDS.128 per item and entry, ds as the float __low2float gives+struct NibXQ { int4 a, b; };++template<int TG>+__device__ __forceinline__ float nib_p_apply(const uint4& q, uint32_t sc, __half dh, const NibXQ& x, float dx, int xsum) {+ if constexpr (TG == 22) { // SplitLs2::apply on the registers+ const typename NibLoad<22>::W w = NibLoad<22>::load(q, sc, dh);+ int sumi0 = 0, sumi1 = 0;+ sumi0 = ggml_cuda_dp4a(w.g[0], x.a.x, sumi0); sumi0 = ggml_cuda_dp4a(w.g[1], x.a.y, sumi0);+ sumi0 = ggml_cuda_dp4a(w.g[2], x.a.z, sumi0); sumi0 = ggml_cuda_dp4a(w.g[3], x.a.w, sumi0);+ sumi1 = ggml_cuda_dp4a(w.g[4], x.b.x, sumi1); sumi1 = ggml_cuda_dp4a(w.g[5], x.b.y, sumi1);+ sumi1 = ggml_cuda_dp4a(w.g[6], x.b.z, sumi1); sumi1 = ggml_cuda_dp4a(w.g[7], x.b.w, sumi1);+ const int sumi = (sumi0 * w.ls0 + sumi1 * w.ls1 + (sumi0 + sumi1) / 2) / 4;+ const float d = w.dw * dx;+ return d * sumi;+ } else if constexpr (TG == 18) { // Split<18>::apply on the registers+ const typename NibLoad<18>::W w = NibLoad<18>::load(q, sc, dh);+ int sumi = 0;+ sumi = ggml_cuda_dp4a(w.g[0], x.a.x, sumi); sumi = ggml_cuda_dp4a(w.g[1], x.a.y, sumi);+ sumi = ggml_cuda_dp4a(w.g[2], x.a.z, sumi); sumi = ggml_cuda_dp4a(w.g[3], x.a.w, sumi);+ sumi = ggml_cuda_dp4a(w.g[4], x.b.x, sumi); sumi = ggml_cuda_dp4a(w.g[5], x.b.y, sumi);+ sumi = ggml_cuda_dp4a(w.g[6], x.b.z, sumi); sumi = ggml_cuda_dp4a(w.g[7], x.b.w, sumi);+ sumi = (w.ls * sumi + sumi / 2) / 2;+ const float d = w.dw * dx;+ return d * sumi;+ } else { // IQ3_S, offset form: dot(v, x) = 2 dot(u, x) - 15 sum(x), exact (|2 dot(u, x)| < 2^17); then Split<21>'s+ int u[8];+ const uint32_t qw[4] = { q.x, q.y, q.z, q.w };+#pragma unroll+ for (int m = 0; m < 4; ++m) {+ u[2 * m + 0] = (int) (qw[m] & 0x0F0F0F0Fu);+ u[2 * m + 1] = (int) ((qw[m] >> 4) & 0x0F0F0F0Fu);+ }+ int sumu = 0;+ sumu = ggml_cuda_dp4a(u[0], x.a.x, sumu); sumu = ggml_cuda_dp4a(u[1], x.a.y, sumu);+ sumu = ggml_cuda_dp4a(u[2], x.a.z, sumu); sumu = ggml_cuda_dp4a(u[3], x.a.w, sumu);+ sumu = ggml_cuda_dp4a(u[4], x.b.x, sumu); sumu = ggml_cuda_dp4a(u[5], x.b.y, sumu);+ sumu = ggml_cuda_dp4a(u[6], x.b.z, sumu); sumu = ggml_cuda_dp4a(u[7], x.b.w, sumu);+ int sumi = 2 * sumu - 15 * xsum;+ sumi *= (int) sc;+ const float d = __half2float(dh) * dx;+ return d * sumi;+ }+}++// the lane's items in the production order: k = t, t+32, t+64 (sum s), then t+16, t+48 (sum s16)+template<int TG, int NC>+__device__ __forceinline__ void nib_row_dot_p(const uint4 (&q)[5], const uint32_t (&sc)[5], const __half (&dh)[5],+ const NibXQ* __restrict__ sx, const float* __restrict__ sd,+ const int* __restrict__ sxs, int t, float (&s)[NC]) {+ constexpr int K[5] = { 0, 32, 64, 16, 48 };+#pragma unroll+ for (int c = 0; c < NC; ++c) s[c] = 0.0f;+#pragma unroll+ for (int i = 0; i < 3; ++i) {+ const int k = t + K[i];+#pragma unroll+ for (int c = 0; c < NC; ++c)+ s[c] += nib_p_apply<TG>(q[i], sc[i], dh[i], sx[c * 80 + k], sd[c * 80 + k], TG == 21 ? sxs[c * 80 + k] : 0);+ }+ float s16[NC];+#pragma unroll+ for (int c = 0; c < NC; ++c) s16[c] = 0.0f;+#pragma unroll+ for (int i = 3; i < 5; ++i) {+ const int k = t + K[i];+#pragma unroll+ for (int c = 0; c < NC; ++c)+ s16[c] += nib_p_apply<TG>(q[i], sc[i], dh[i], sx[c * 80 + k], sd[c * 80 + k], TG == 21 ? sxs[c * 80 + k] : 0);+ }+#pragma unroll+ for (int c = 0; c < NC; ++c) {+ s[c] = __fadd_rn(s[c], s16[c]);+#pragma unroll+ for (int o = 8; o > 0; o >>= 1) s[c] += __shfl_xor_sync(0xffffffffu, s[c], o);+ }+}++// The P skeleton's work on one group: this lane's five items into registers, then the passes. Block-uniform.+template<int TG>+__device__ __forceinline__ void nib_group_p(NibXQ* s_x, float* s_d, int* s_xs, const uint4* crow, const uint8_t* srow,+ const __half* drow, const int32_t* __restrict__ ent_tok,+ const block_q8_1* __restrict__ xq, int e0, int e1, int t, float* dst, int r) {+ const int kb = t >> 3;+ uint4 q[5]; uint32_t sc[5]; __half dh[5];+ q[0] = crow[t]; q[1] = crow[t + 32]; q[2] = crow[t + 64]; q[3] = crow[t + 16]; q[4] = crow[t + 48];+ sc[0] = srow[t]; sc[1] = srow[t + 32]; sc[2] = srow[t + 64]; sc[3] = srow[t + 16]; sc[4] = srow[t + 48];+ dh[0] = drow[kb]; dh[1] = drow[kb + 4]; dh[2] = drow[kb + 8]; dh[3] = drow[kb + 2]; dh[4] = drow[kb + 6];+ for (int e = e0; e < e1; e += GRP_NC) {+ const int n = min(GRP_NC, e1 - e);+ __syncthreads();+ for (int i = threadIdx.x; i < n * 80; i += 256) {+ const int c = i >= 240 ? 3 : i >= 160 ? 2 : i >= 80 ? 1 : 0, kk = i - 80 * c;+ const int tok = ent_tok[e + c];+ const int* src = (const int*) (xq + (size_t) tok * 80 + kk);+ NibXQ v;+ v.a = make_int4(src[1], src[2], src[3], src[4]);+ v.b = make_int4(src[5], src[6], src[7], src[8]);+ s_x[i] = v;+ s_d[i] = __low2float(xq[(size_t) tok * 80 + kk].ds);+ if constexpr (TG == 21) { // sum(x): the production dp4a against 0x01010101 over the 8 qs words+ int xs = 0;+ xs = ggml_cuda_dp4a(v.a.x, 0x01010101, xs); xs = ggml_cuda_dp4a(v.a.y, 0x01010101, xs);+ xs = ggml_cuda_dp4a(v.a.z, 0x01010101, xs); xs = ggml_cuda_dp4a(v.a.w, 0x01010101, xs);+ xs = ggml_cuda_dp4a(v.b.x, 0x01010101, xs); xs = ggml_cuda_dp4a(v.b.y, 0x01010101, xs);+ xs = ggml_cuda_dp4a(v.b.z, 0x01010101, xs); xs = ggml_cuda_dp4a(v.b.w, 0x01010101, xs);+ s_xs[i] = xs;+ }+ }+ __syncthreads();+ if (n == 1) {+ float s[1]; nib_row_dot_p<TG, 1>(q, sc, dh, s_x, s_d, s_xs, t, s);+ if (t == 0) dst[(size_t) e * nibk::kNFf + r] = s[0];+ } else if (n == 2) {+ float s[2]; nib_row_dot_p<TG, 2>(q, sc, dh, s_x, s_d, s_xs, t, s);+ if (t < 2) dst[(size_t) (e + t) * nibk::kNFf + r] = s[t];+ } else if (n == 3) {+ float s[3]; nib_row_dot_p<TG, 3>(q, sc, dh, s_x, s_d, s_xs, t, s);+ if (t < 3) dst[(size_t) (e + t) * nibk::kNFf + r] = s[t];+ } else {+ float s[4]; nib_row_dot_p<TG, 4>(q, sc, dh, s_x, s_d, s_xs, t, s);+ if (t < 4) dst[(size_t) (e + t) * nibk::kNFf + r] = s[t];+ }+ }+}++// SKEL kSkelS or kSkelP for every group. grid (80, gy): 2 x 640 rows = 80 x 16, so every thread has a row and+// reaches every barrier.+constexpr int kNibSWords = GRP_NC * 80 * (int) (sizeof(block_q8_1) / 4); // S: the blocks+constexpr int kNibXqWords = GRP_NC * 80 * (int) (sizeof(NibXQ) / 4); // P: the qs rows, then ds, then sum(x)+template<int TG, int SKEL>+inline constexpr int kNibSmemWords = SKEL == nibk::kSkelS ? kNibSWords : kNibXqWords + GRP_NC * 80 + (TG == 21 ? GRP_NC * 80 : 0);+template<int TG, int SKEL>+__global__ void __launch_bounds__(256) nib_gu_kernel(const unsigned long long* __restrict__ grp_ptr,+ const int32_t* __restrict__ grp_start,+ const int32_t* __restrict__ n_groups,+ const int32_t* __restrict__ ent_tok,+ const block_q8_1* __restrict__ xq, float* __restrict__ gate,+ float* __restrict__ up) {+ const int ng = *n_groups;+ if (blockIdx.y >= (unsigned) ng) return;+ STRATA_SHARED_ALIGN16_U32(s_buf, kNibSmemWords<TG, SKEL>);+ const int subwarp = threadIdx.x >> 4, t = threadIdx.x & 15;+ const int row = blockIdx.x * 16 + subwarp; // 0 .. 2 n_ff+ const bool is_up = row >= nibk::kNFf;+ const int r = is_up ? row - (int) nibk::kNFf : row;+ float* dst = is_up ? up : gate;+ for (int g = blockIdx.y; g < ng; g += gridDim.y) {+ const uint8_t* slot = (const uint8_t*) grp_ptr[g];+ const uint4* crow = (const uint4*) (slot + nibk::kCodesOff) + (size_t) row * nibk::kItems;+ const uint8_t* srow = slot + nibk::kScOff + (size_t) row * nibk::kItems;+ const __half* drow = (const __half*) (slot + nibk::kDOff) + (size_t) row * nibk::kSuper;+ const int e0 = grp_start[g], e1 = grp_start[g + 1];+ if constexpr (SKEL == nibk::kSkelP) {+ nib_group_p<TG>((NibXQ*) s_buf, (float*) (s_buf + kNibXqWords), (int*) (s_buf + kNibXqWords + GRP_NC * 80),+ crow, srow, drow, ent_tok, xq, e0, e1, t, dst, r);+ } else {+ for (int e = e0; e < e1; e += GRP_NC)+ nib_pass_s<TG>((block_q8_1*) s_buf, crow, srow, drow, ent_tok, xq, e, min(GRP_NC, e1 - e), t, dst, r);+ }+ }+}++#include "nib_gu_proto.cuh" // STRATA_NIB_GU_PROTO: the staged-once 4-bit gate/up kernel (id 31)++int g_nib_variant = -1; // native_expert_nib_set_variant (the bench / parity); -1 = expert_nibble.hpp's gu_skeleton++// STRATA_NIB_GU_PROTO: unset / 0 = off (the S / P skeleton per format); 1 (or 31) = gu-proto 31 (the activations staged+// once per two row blocks with 16-byte loads, the production row dot). Read once.+int nib_gu_proto_env() {+ static const int v = [] {+ const char* e = std::getenv("STRATA_NIB_GU_PROTO");+ int id = e != nullptr ? std::atoi(e) : 0;+ if (id == 1) id = 31;+ if (id != 31) id = 0;+ if (id != 0) std::fprintf(stderr, "native_expert_grouped: STRATA_NIB_GU_PROTO: 4-bit gate/up kernel gu-proto %d\n", id);+ return id;+ }();+ return v;+}++// The gate/up of a converted layer's VRAM groups: native_gu_multi_kernel's grid and outputs.+void launch_gu_nib(const NativeExpertLayout& L, dim3 grid, cudaStream_t s, const unsigned long long* grp_ptr,+ const int32_t* grp_start, const int32_t* n_groups, const int32_t* ent_tok, const block_q8_1* X,+ float* gate, float* up) {+ if (g_nib_variant >= 10) { // gu-proto (nib_gu_proto.cuh), forced by the bench / parity+ if (((uintptr_t) X & 15) != 0 ||+ !nibx_launch(L.gu_type, g_nib_variant, grid, s, grp_ptr, grp_start, n_groups, ent_tok, X, gate, up)) {+ std::fprintf(stderr, "native_expert_grouped: gu-proto variant %d for type %d is not built (or x is not 16-aligned)\n",+ g_nib_variant, L.gu_type);+ std::exit(1);+ }+ return;+ }+ // STRATA_NIB_GU_PROTO (opt-in, default off): the gu-proto kernel in place of the S / P skeleton - bitwise the+ // same gate / up (iq_nibble_parity (7), native_expert_bench --nibble --vs). The 16-byte staging needs x 16-aligned+ // (verify's nat_xq_ rows are: a 256-aligned buffer, 2,880 B a token); otherwise the production kernel runs.+ if (g_nib_variant < 0) {+ const int pv = nib_gu_proto_env();+ if (pv != 0 && ((uintptr_t) X & 15) == 0 &&+ nibx_launch(L.gu_type, pv, grid, s, grp_ptr, grp_start, n_groups, ent_tok, X, gate, up))+ return;+ }+ const int skel = g_nib_variant >= 0 ? (g_nib_variant == 1 ? nibk::kSkelP : nibk::kSkelS) : nibk::gu_skeleton(L.gu_type);+#define STRATA_NIB_GU(TG, S) \+ case TG * 2 + S: nib_gu_kernel<TG, S><<<grid, 256, 0, s>>>(grp_ptr, grp_start, n_groups, ent_tok, X, gate, up); break;+ switch (L.gu_type * 2 + skel) {+ STRATA_NIB_GU(22, 0) STRATA_NIB_GU(22, 1)+ STRATA_NIB_GU(18, 0) STRATA_NIB_GU(18, 1)+ STRATA_NIB_GU(21, 0) STRATA_NIB_GU(21, 1)+ default:+ std::fprintf(stderr, "native_expert_grouped: 4-bit gate/up type %d at %lld x %lld is not built\n", L.gu_type,+ (long long) L.n_embd, (long long) L.n_ff);+ std::exit(1);+ }+#undef STRATA_NIB_GU+}++// ---- the load-time transcoder (expert_nibble.hpp's transcode)+template<int TG>+__device__ __forceinline__ uint32_t nib_code(int v, bool& ok) {+ if constexpr (TG == 21) { // v odd in -15..15 -> u = (v + 15) / 2+ if ((v & 1) == 0 || v < -15 || v > 15) { ok = false; return 0; }+ return (uint32_t) ((v + 15) >> 1);+ } else if constexpr (TG == 18) { // index of {4, 12, ..., 52, 62} | sign << 3+ const int a = v < 0 ? -v : v;+ const int i = a == 62 ? 7 : ((a & 7) == 4 && a <= 52) ? (a - 4) >> 3 : -1;+ if (i < 0) { ok = false; return 0; }+ return (uint32_t) i | (v < 0 ? 8u : 0u);+ } else { // IQ2_S: index of {8, 25, 43} | sign << 2+ const int a = v < 0 ? -v : v;+ const int i = a == 8 ? 0 : a == 25 ? 1 : a == 43 ? 2 : -1;+ if (i < 0) { ok = false; return 0; }+ return (uint32_t) i | (v < 0 ? 4u : 0u);+ }+}++// The GGUF blocks (ggml-common.h): their size and the fp16 d at their start+template<int TG> struct NibBlk;+template<> struct NibBlk<22> { using B = block_iq2_s; };+template<> struct NibBlk<18> { using B = block_iq3_xxs; };+template<> struct NibBlk<21> { using B = block_iq3_s; };++// One thread per (expert, row, item): the production load, the code, scale and d written, read back from memory and+// decoded with the kernels' own decoders, compared with the load's W (and, for IQ3_S, the P skeleton's u against v);+// `bad` counts the items that differ or hold a weight outside the code set.+template<int TG>+__global__ void __launch_bounds__(256) nib_transcode_kernel(const uint8_t* __restrict__ src, uint64_t raw_bytes,+ uint64_t gu_row, uint64_t up_off,+ const unsigned long long* __restrict__ slots, long long n,+ unsigned long long* __restrict__ bad) {+ const long long i = (long long) blockIdx.x * blockDim.x + threadIdx.x;+ if (i >= n * nibk::kRows * nibk::kItems) return;+ const int k = (int) (i % nibk::kItems);+ const int row = (int) ((i / nibk::kItems) % nibk::kRows);+ const long long ex = i / (nibk::kItems * nibk::kRows);+ const bool is_up = row >= nibk::kNFf;+ const int r = is_up ? row - (int) nibk::kNFf : row;+ const uint8_t* wr = src + (size_t) ex * raw_bytes + (is_up ? up_off : 0) + (size_t) r * gu_row;+ const int kbx = k / Fmt<TG>::ipb, iqs = Fmt<TG>::step * (k % Fmt<TG>::ipb);+ using S = Split<TG>;+ const typename S::W w = S::template load<false>(wr, kbx, iqs, nullptr);+ bool ok = true;+ uint32_t words[4];+#pragma unroll+ for (int m = 0; m < 4; ++m) {+ const uint32_t g0 = (uint32_t) w.g[2 * m], g1 = (uint32_t) w.g[2 * m + 1];+ uint32_t word = 0;+#pragma unroll+ for (int b = 0; b < 4; ++b) {+ const uint32_t c0 = nib_code<TG>((int) (int8_t) (g0 >> (8 * b)), ok);+ const uint32_t c1 = nib_code<TG>((int) (int8_t) (g1 >> (8 * b)), ok);+ if constexpr (TG == 21) word |= (c0 | c1 << 4) << (8 * b);+ else word |= c0 << (4 * b) | c1 << (16 + 4 * b);+ }+ words[m] = word;+ }+ uint32_t sc;+ if constexpr (TG == 22) sc = (uint32_t) (w.ls0 | w.ls1 << 4);+ else sc = (uint32_t) w.ls;+ ok = ok && sc < 256u;+ uint8_t* slot = (uint8_t*) slots[ex];+ uint4* cp = (uint4*) (slot + nibk::code_off(row, k));+ uint8_t* sp = slot + nibk::scale_off(row, k);+ *cp = make_uint4(words[0], words[1], words[2], words[3]);+ *sp = (uint8_t) sc;+ const __half dsrc = ((const typename NibBlk<TG>::B*) wr + kbx)->d; // the block's d, bit for bit+ __half dread = dsrc;+ if (k % Fmt<TG>::ipb == 0) { // the superblock's first item writes its d+ __half* dp = (__half*) (slot + nibk::d_off(row, kbx));+ *dp = dsrc;+ dread = __ushort_as_half(*(volatile const unsigned short*) dp);+ ok = ok && __half_as_ushort(dread) == __half_as_ushort(dsrc);+ }+ // the code and the scale read back from memory and decoded as the kernels do+ const uint4 qb = __ldcv(cp);+ const uint32_t sb = *(volatile const uint8_t*) sp;+ const typename NibLoad<TG>::W v = NibLoad<TG>::load(qb, sb, dread);+#pragma unroll+ for (int j = 0; j < 8; ++j) ok = ok && v.g[j] == w.g[j];+ if constexpr (TG == 22) ok = ok && v.ls0 == w.ls0 && v.ls1 == w.ls1;+ else ok = ok && v.ls == w.ls;+ ok = ok && __float_as_uint(v.dw) == __float_as_uint(w.dw);+ if constexpr (TG == 21) { // the P skeleton's u: 2u - 15 per byte is v+ const uint32_t qw[4] = { qb.x, qb.y, qb.z, qb.w };+#pragma unroll+ for (int m = 0; m < 4; ++m) {+ const uint32_t u0 = qw[m] & 0x0F0F0F0Fu, u1 = (qw[m] >> 4) & 0x0F0F0F0Fu;+ ok = ok && __vsub4(__vadd4(u0, u0), 0x0F0F0F0Fu) == (uint32_t) w.g[2 * m] &&+ __vsub4(__vadd4(u1, u1), 0x0F0F0F0Fu) == (uint32_t) w.g[2 * m + 1];+ }+ }+ if (!ok) atomicAdd(bad, 1ull);+}++// ---- the GGUF rebuild (expert_nibble.hpp's check_gguf and gather_gguf)+// The reverse codebooks: a group's magnitude pattern -> its grid index (0xFFFFFFFF: not in the grid). 32-bit+// entries for atomicCAS, which also counts a pattern two grid entries share (none in ggml's grids; then refused).+constexpr int kNibRev22 = 6561, kNibRev18 = 4096, kNibRev21 = 4096; // 3^8 (IQ2_S: 8 weights), 8^4, 8^4 (4 weights)+__device__ uint32_t g_nib_rev22[kNibRev22];+__device__ uint32_t g_nib_rev18[kNibRev18];+__device__ uint32_t g_nib_rev21[kNibRev21];+__device__ unsigned long long g_nib_rev_bad;++__device__ __forceinline__ int nib_mag22(uint32_t b) { return b == 8 ? 0 : b == 25 ? 1 : b == 43 ? 2 : -1; }+__device__ __forceinline__ int nib_mag18(uint32_t b) { return b == 62 ? 7 : ((b & 7) == 4 && b <= 52) ? (int) (b - 4) >> 3 : -1; }+__device__ __forceinline__ int nib_mag21(uint32_t b) { return (b & 1) && b <= 15 ? (int) (b - 1) >> 1 : -1; }++__global__ void nib_rev_clear_kernel() {+ const int i = blockIdx.x * blockDim.x + threadIdx.x;+ if (i < kNibRev22) g_nib_rev22[i] = 0xFFFFFFFFu;+ if (i < kNibRev18) g_nib_rev18[i] = 0xFFFFFFFFu;+ if (i < kNibRev21) g_nib_rev21[i] = 0xFFFFFFFFu;+ if (i == 0) g_nib_rev_bad = 0;+}+__global__ void nib_rev_fill_kernel() {+ const int i = blockIdx.x * blockDim.x + threadIdx.x;+ uint32_t* tab = nullptr;+ int key = 0;+ bool ok = true;+ uint32_t idx = 0;+ if (i < 1024) { // iq2s_grid: 8 bytes of {8, 25, 43}+ const uint64_t g = iq2s_grid[i];+ for (int j = 7; j >= 0; --j) { const int m = nib_mag22((uint32_t) (g >> (8 * j)) & 0xFF); ok = ok && m >= 0; key = key * 3 + (m < 0 ? 0 : m); }+ tab = g_nib_rev22; idx = (uint32_t) i;+ } else if (i < 1024 + 256) { // iq3xxs_grid: 4 bytes of {4, ..., 62}+ const uint32_t g = iq3xxs_grid[i - 1024];+ for (int j = 3; j >= 0; --j) { const int m = nib_mag18((g >> (8 * j)) & 0xFF); ok = ok && m >= 0; key = key * 8 + (m < 0 ? 0 : m); }+ tab = g_nib_rev18; idx = (uint32_t) (i - 1024);+ } else if (i < 1024 + 256 + 512) { // iq3s_grid: 4 bytes of {1, 3, ..., 15}+ const uint32_t g = iq3s_grid[i - 1280];+ for (int j = 3; j >= 0; --j) { const int m = nib_mag21((g >> (8 * j)) & 0xFF); ok = ok && m >= 0; key = key * 8 + (m < 0 ? 0 : m); }+ tab = g_nib_rev21; idx = (uint32_t) (i - 1280);+ } else {+ return;+ }+ if (!ok || atomicCAS(tab + key, 0xFFFFFFFFu, idx) != 0xFFFFFFFFu) atomicAdd(&g_nib_rev_bad, 1ull);+}++// The bytes of GGUF block `s` of row `row`'s item `ib` (0..7) rebuilt from the slot into `blk` (shared memory, the+// block's first byte): the codes give each weight's magnitude index and sign, the reverse codebook the grid index,+// the scale byte and d are the block's own. Disjoint bytes per item (IQ3_S: the even item writes the shared scale+// byte); item 0 writes d.+template<int TG>+__device__ __forceinline__ void nib_gguf_item(uint8_t* blk, int ib, const uint8_t* __restrict__ slot, int row, int k) {+ const uint4 q4 = *(const uint4*) (slot + nibk::code_off(row, k));+ const uint32_t w[4] = { q4.x, q4.y, q4.z, q4.w };+ const uint32_t sc = slot[nibk::scale_off(row, k)];+ if (ib == 0) {+ const uint8_t* dp = slot + nibk::d_off(row, k / 8);+ blk[0] = dp[0];+ blk[1] = dp[1];+ }+ if constexpr (TG == 22) { // block_iq2_s: d, qs[32] grid low bits, qs[32] signs, qh[8], scales[8]+ uint32_t qh = 0;+#pragma unroll+ for (int l = 0; l < 4; ++l) {+ int key = 0;+ uint32_t sg = 0;+#pragma unroll+ for (int j = 7; j >= 0; --j) {+ const uint32_t c = j < 4 ? (w[l] >> (4 * j)) & 15u : (w[l] >> (16 + 4 * (j - 4))) & 15u;+ key = key * 3 + (int) (c & 3u);+ sg |= ((c >> 2) & 1u) << j;+ }+ const uint32_t idx = key < kNibRev22 ? g_nib_rev22[key] : 0xFFFFFFFFu;+ blk[2 + 4 * ib + l] = (uint8_t) idx;+ blk[2 + 32 + 4 * ib + l] = (uint8_t) sg;+ qh |= ((idx >> 8) & 3u) << (2 * l);+ }+ blk[66 + ib] = (uint8_t) qh;+ blk[74 + ib] = (uint8_t) sc;+ } else if constexpr (TG == 18) { // block_iq3_xxs: d, qs[64] grid indices, 8 x aux32 (4 x 7 sign bits | scale << 28)+ uint32_t aux = sc << 28;+#pragma unroll+ for (int l = 0; l < 4; ++l) {+ uint32_t sg = 0;+#pragma unroll+ for (int h = 0; h < 2; ++h) {+ int key = 0;+#pragma unroll+ for (int j = 3; j >= 0; --j) {+ const uint32_t c = (w[l] >> (16 * h + 4 * j)) & 15u;+ key = key * 8 + (int) (c & 7u);+ sg |= ((c >> 3) & 1u) << (4 * h + j);+ }+ blk[2 + 8 * ib + 2 * l + h] = (uint8_t) g_nib_rev18[key];+ }+ aux |= (sg & 0x7Fu) << (7 * l); // the 8th sign is the parity of the other 7 (ksigns_iq2xs)+ }+ blk[2 + 64 + 4 * ib + 0] = (uint8_t) aux;+ blk[2 + 64 + 4 * ib + 1] = (uint8_t) (aux >> 8);+ blk[2 + 64 + 4 * ib + 2] = (uint8_t) (aux >> 16);+ blk[2 + 64 + 4 * ib + 3] = (uint8_t) (aux >> 24);+ } else { // block_iq3_s: d, qs[64] grid low bits, qh[8], signs[32], scales[4] (two items' nibbles each)+ uint32_t qh = 0;+#pragma unroll+ for (int l = 0; l < 4; ++l) {+ uint32_t sg = 0;+#pragma unroll+ for (int h = 0; h < 2; ++h) {+ int key = 0;+#pragma unroll+ for (int j = 3; j >= 0; --j) {+ const uint32_t u = (w[l] >> (8 * j + 4 * h)) & 15u; // v = 2u - 15: negative for u <= 7+ key = key * 8 + (int) (u <= 7u ? 7u - u : u - 8u);+ sg |= (u <= 7u ? 1u : 0u) << (4 * h + j);+ }+ const uint32_t idx = g_nib_rev21[key];+ blk[2 + 8 * ib + 2 * l + h] = (uint8_t) idx;+ qh |= ((idx >> 8) & 1u) << (2 * l + h);+ }+ blk[74 + 4 * ib + l] = (uint8_t) sg;+ }+ blk[66 + ib] = (uint8_t) qh;+ if ((ib & 1) == 0) {+ const uint32_t sc1 = slot[nibk::scale_off(row, k + 1)];+ blk[106 + ib / 2] = (uint8_t) (((sc - 1u) >> 1) | (((sc1 - 1u) >> 1) << 4));+ }+ }+}++constexpr int kNibGgufRows = 8; // gate/up rows a block rebuilds (in shared memory, then one coalesced pass)+constexpr int kNibGgufThreads = 128;+constexpr int kNibGgufWords = 8; // uint4 per thread in the down / tail part+struct NibGgufArgs {+ const unsigned long long* dev; // the slots' device addresses (check), or nullptr: `p`+ const uint8_t* p[16]; // the slots (gather: an MMQ group, at most mmq::kGatherGroupMax)+ int first, n;+ uint8_t* gu; // gather: the rebuilt [gate | up]; check: the raw blobs' first byte+ uint64_t gu_stride;+ uint8_t* d; // gather: the down rows; check: the raw blobs' down rows+ uint64_t d_stride;+ uint64_t gu_row, down16, zero16; // GGUF row bytes; down rows and MMQ tail in uint4+ unsigned long long* bad; // check: 32-bit words / uint4 that differ+};+// blockIdx.y: expert first + y. blockIdx.x < kRows / kNibGgufRows: rows [8x, 8x + 8) rebuilt into shared memory,+// then written (gather) or compared (check) as 32-bit words; the rest: the down rows as uint4, and (gather, the last+// expert) the zeroed tails after gate/up and down.+template<int TG, bool CHECK>+__global__ void __launch_bounds__(kNibGgufThreads) nib_gguf_kernel(NibGgufArgs a) {+ constexpr int BB = (int) sizeof(typename NibBlk<TG>::B);+ __shared__ uint32_t s_rows[kNibGgufRows * 1100 / 4];+ const int q = a.first + (int) blockIdx.y;+ const uint8_t* slot = a.dev != nullptr ? (const uint8_t*) a.dev[q] : a.p[q];+ constexpr int kRowBlocks = (int) (nibk::kRows / kNibGgufRows);+ unsigned long long nbad = 0;+ if ((int) blockIdx.x < kRowBlocks) {+ const int r0 = (int) blockIdx.x * kNibGgufRows;+ uint8_t* sb = (uint8_t*) s_rows;+ for (int u = threadIdx.x; u < kNibGgufRows * (int) nibk::kItems; u += kNibGgufThreads) {+ const int rr = u / (int) nibk::kItems, k = u % (int) nibk::kItems;+ nib_gguf_item<TG>(sb + (size_t) rr * a.gu_row + (size_t) (k >> 3) * BB, k & 7, slot, r0 + rr, k);+ }+ __syncthreads();+ const int nw = (int) (kNibGgufRows * a.gu_row / 4);+ uint32_t* dst = (uint32_t*) (a.gu + (size_t) q * a.gu_stride + (size_t) r0 * a.gu_row);+ for (int i = threadIdx.x; i < nw; i += kNibGgufThreads) {+ if constexpr (CHECK) nbad += dst[i] != s_rows[i];+ else dst[i] = s_rows[i];+ }+ } else {+ const uint4* src = (const uint4*) (slot + nibk::kDownOff);+ uint4* dd = (uint4*) (a.d + (size_t) q * a.d_stride);+ const int64_t base = ((int64_t) blockIdx.x - kRowBlocks) * (kNibGgufThreads * kNibGgufWords);+#pragma unroll+ for (int w = 0; w < kNibGgufWords; ++w) {+ const int64_t i = base + (int64_t) w * kNibGgufThreads + threadIdx.x;+ if (i < (int64_t) a.down16) {+ if constexpr (CHECK) {+ const uint4 x = src[i], y = dd[i];+ nbad += x.x != y.x || x.y != y.y || x.z != y.z || x.w != y.w;+ } else {+ dd[i] = src[i];+ }+ } else if (!CHECK && q == a.n - 1 && i < (int64_t) (a.down16 + 2 * a.zero16)) {+ const int64_t z = i - (int64_t) a.down16;+ if (z < (int64_t) a.zero16) ((uint4*) (a.gu + (size_t) a.n * a.gu_stride))[z] = make_uint4(0, 0, 0, 0);+ else ((uint4*) (a.d + (size_t) a.n * a.d_stride))[z - (int64_t) a.zero16] = make_uint4(0, 0, 0, 0);+ }+ }+ }+ if constexpr (CHECK) {+ if (nbad) atomicAdd(a.bad, nbad);+ }+}++// sum_i (w_i ^ i) * kHashMul mod 2^64 over the first `bytes` bytes of each slot (little-endian words, the last one+// zero-extended): expert_nibble.hpp's hash_host. A sum, so the blocks' order of adding does not change it.+__global__ void __launch_bounds__(256) nib_hash_kernel(const unsigned long long* __restrict__ slots, uint64_t bytes,+ unsigned long long* __restrict__ out) {+ const uint8_t* p = (const uint8_t*) slots[blockIdx.y];+ const uint64_t n16 = bytes / 16, nw = (bytes + 3) / 4;+ uint64_t h = 0;+ for (uint64_t q = (uint64_t) blockIdx.x * blockDim.x + threadIdx.x; q < n16; q += (uint64_t) gridDim.x * blockDim.x) {+ const uint4 v = ((const uint4*) p)[q];+ const uint32_t i = (uint32_t) (4 * q);+ h += (uint64_t) (v.x ^ i) * nibk::kHashMul + (uint64_t) (v.y ^ (i + 1)) * nibk::kHashMul ++ (uint64_t) (v.z ^ (i + 2)) * nibk::kHashMul + (uint64_t) (v.w ^ (i + 3)) * nibk::kHashMul;+ }+ if (blockIdx.x == 0)+ for (uint64_t i = 4 * n16 + threadIdx.x; i < nw; i += blockDim.x) {+ uint32_t w = 0;+ for (uint64_t b = 4 * i; b < 4 * i + 4 && b < bytes; ++b) w |= (uint32_t) p[b] << (8 * (b - 4 * i));+ h += (uint64_t) (w ^ (uint32_t) i) * nibk::kHashMul;+ }+#pragma unroll+ for (int o = 16; o > 0; o >>= 1) h += __shfl_xor_sync(0xffffffffu, h, o);+ __shared__ uint64_t s_h[8];+ if ((threadIdx.x & 31) == 0) s_h[threadIdx.x >> 5] = h;+ __syncthreads();+ if (threadIdx.x == 0) {+ uint64_t t = 0;+ for (int w = 0; w < (int) (blockDim.x >> 5); ++w) t += s_h[w];+ atomicAdd(out + blockIdx.y, (unsigned long long) t);+ }+}+#endif // STRATA_EXPERT_NIBBLE_BUILT++// STRATA_EXPERT_NIBBLE: whether a layout's gate/up is a converted slot's 4-bit codes (constant false in a build+// without the kernels, so its dispatch below is today's code)+constexpr bool nib_layout(const NativeExpertLayout& L) {+#if STRATA_EXPERT_NIBBLE_BUILT+ return L.gu_nib;+#else+ static_cast<void>(L);+ return false;+#endif+}+} // namespacevoid iq_set_old_kernels(bool old) { g_old_kernels = old; }@@ -4660,6 +5359,50 @@ NativeExpertLayout native_expert_layout(int gu_type, int d_type, int64_t n_embd,return L;}+NativeExpertLayout native_expert_layout_nib(int gu_type, int d_type, int64_t n_embd, int64_t n_ff) {+ namespace nb = strata::core::nibble;+ NativeExpertLayout L = native_expert_layout(gu_type, d_type, n_embd, n_ff);+ L.gu_nib = true;+ L.gu_row = (size_t) (nb::kItems * nb::kItemBytes); // a row's codes (the scales and d follow all the codes)+ L.up_off = nb::kUpCodesOff;+ L.down_off = nb::kDownOff;+ L.bytes = nb::slot_bytes_for((uint64_t) n_embd * L.d_row);+ return L;+}++bool native_expert_nib_supported(int gu_type, int d_type, int64_t n_embd, int64_t n_ff) noexcept {+#if STRATA_EXPERT_NIBBLE_BUILT+ namespace nb = strata::core::nibble;+ return nb::fmt_built(gu_type) && n_embd == nb::kNEmbd && n_ff == nb::kNFf && d_type != 40 &&+ native_expert_supported(gu_type, d_type, n_embd, n_ff);+#else+ (void) gu_type; (void) d_type; (void) n_embd; (void) n_ff;+ return false;+#endif+}++void native_expert_nib_set_variant(int v) {+#if STRATA_EXPERT_NIBBLE_BUILT+ g_nib_variant = v;+#else+ (void) v;+#endif+}++bool native_expert_nib_variant_attr(int v, int gu_type, int* regs, int* local_bytes, int* blocks_per_sm) {+#if STRATA_EXPERT_NIBBLE_BUILT+ cudaFuncAttributes a{};+ if (!nibx_launch(gu_type, v, dim3(80, 1), nullptr, nullptr, nullptr, nullptr, nullptr, nullptr, nullptr, nullptr, &a)) return false;+ *regs = a.numRegs;+ *local_bytes = (int) a.localSizeBytes;+ *blocks_per_sm = g_nibx_occ;+ return true;+#else+ (void) v; (void) gu_type; (void) regs; (void) local_bytes; (void) blocks_per_sm;+ return false;+#endif+}+size_t native_expert_scratch_bytes(int64_t cap, int64_t n_ff) {const size_t f = (size_t) cap * (size_t) n_ff * sizeof(float);return 3 * ((f + 255) & ~(size_t) 255) + (((size_t) cap * (size_t) (n_ff / 32) * sizeof(block_q8_1) + 255) & ~(size_t) 255);@@ -5315,7 +6058,7 @@ int exp_mode() {#endif}#endif-int g_exp_phase = 0; // the bench: 0 all, 1 gate/up + swiglu + quantize only, 2 down only+int g_exp_phase = 0; // the bench: 0 all, 1 gate/up + swiglu + quantize only, 2 down only, 3 gate/up only#if STRATA_EXP_LAYOUTS && !defined(__HIPCC__)// The down projection's mode on Turing's default: mode 4 is slower than the CUDA layout once groups have 3+ entries// (12 x 3: 36.5 -> 40.8 us, 16 x 4: 56.9 -> 68.9) - windows of 4+ tokens (cap_groups > 30) keep mode 0 there.@@ -5389,8 +6132,16 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long longfloat* h = (float*) ((uint8_t*) scratch + 2 * fa);block_q8_1* hq = (block_q8_1*) ((uint8_t*) scratch + 3 * fa);const auto* X = (const block_q8_1*) x_q8_1;+#if !STRATA_EXPERT_NIBBLE_BUILT+ if (L.gu_nib) { // STRATA_EXPERT_NIBBLE's slots: the switch is refused at start in a build without the kernels+ std::fprintf(stderr, "native_expert_grouped: 4-bit gate/up slots are not built into this engine\n");+ std::exit(1);+ }+#endif+ // STRATA_EXPERT_NIBBLE (nib_layout): only the gate/up step below takes another kernel (launch_gu_nib, in place of+ // launch_gu); SwiGLU, the q8_1 quantization and the down rows (GGUF, at L.down_off) are the layout's ownstatic const bool v2 = [] { const char* v = std::getenv("STRATA_EXPERT_V2"); return v && v[0] == '1'; }();- if (v2 && L.gu_type == 21 && L.d_type == 20 && L.n_embd == 2560 && L.n_ff == 640) { // S26: see s26_gu_l_kernel+ if (v2 && !nib_layout(L) && L.gu_type == 21 && L.d_type == 20 && L.n_embd == 2560 && L.n_ff == 640) { // S26: see s26_gu_l_kernel// S26 STRATA_TSUM=1: the sums as one transposed butterfly per warp (s26_tsum.cuh; bitwise the same values)static const bool ts = [] { const char* v = std::getenv("STRATA_TSUM"); return v && v[0] == '1'; }();// (+ down rows' items spread evenly over the lanes, one load group: bitwise, harness 1.09-1.12x vs 1.03-1.06x;@@ -5460,7 +6211,7 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long#endif#endif#if STRATA_EXP_LAYOUTS- const bool fused_gu = em0 == 8 && ((kExpLds16 && L.gu_type == 16) || L.gu_type == 18 || L.gu_type == 21 || L.gu_type == 22 || L.gu_type == 23) &&+ const bool fused_gu = !nib_layout(L) && em0 == 8 && ((kExpLds16 && L.gu_type == 16) || L.gu_type == 18 || L.gu_type == 21 || L.gu_type == 22 || L.gu_type == 23) &&L.n_ff % 32 == 0;if (fused_gu && g_exp_phase != 2) {const dim3 gl((unsigned) (L.n_ff / 32), (unsigned) cap_groups);@@ -5482,7 +6233,7 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long#elseconst bool gs_layout = true;#endif- if (gs_layout && g_exp_phase == 0 && !v1 && gu_swiglu_on() && !(expdec_mode() && !g_old_kernels && expdec_gu_ok(L))) {+ if (gs_layout && !nib_layout(L) && g_exp_phase == 0 && !v1 && gu_swiglu_on() && !(expdec_mode() && !g_old_kernels && expdec_gu_ok(L))) {switch (L.gu_type) {#define STRATA_GS(T) case T: gs_done = launch_gu_swiglu<T>(L, gy, s, grp_ptr, grp_start, n_groups, ent_tok, X, hq); break;STRATA_GU_FMTS(STRATA_GS)@@ -5497,7 +6248,7 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long longif (g_exp_phase != 2 && !gs_done) {#endif#if STRATA_EXP_LAYOUTS- const bool lds_gu = (em0 == 5 || em0 == 6 || em0 == 7 || em0 == 8) && ((kExpLds16 && L.gu_type == 16) || L.gu_type == 18 || L.gu_type == 21 || L.gu_type == 22 || L.gu_type == 23);+ const bool lds_gu = !nib_layout(L) && (em0 == 5 || em0 == 6 || em0 == 7 || em0 == 8) && ((kExpLds16 && L.gu_type == 16) || L.gu_type == 18 || L.gu_type == 21 || L.gu_type == 22 || L.gu_type == 23);if (lds_gu) {const dim3 gl((unsigned) ((2 * L.n_ff + LDS_RB - 1) / LDS_RB), (unsigned) cap_groups);const size_t sh = (size_t) LDS_NT * (size_t) (L.n_embd / 32) * sizeof(block_q8_1);@@ -5518,13 +6269,19 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long// the AMD layouts cover the IQ packs' gate/up types; Unsloth's Q4_K / Q5_K take the CUDA layout belowconst bool amd_gu = L.gu_type == 16 || L.gu_type == 17 || L.gu_type == 18 || L.gu_type == 21 || L.gu_type == 22 ||L.gu_type == 23 || L.gu_type == 29 || L.gu_type == 42;- if ((em == 1 || em == 2 || em == 4) && amd_gu) {+ if ((em == 1 || em == 2 || em == 4) && amd_gu && !nib_layout(L)) {switch (L.gu_type) {STRATA_GU_AMD(16) STRATA_GU_AMD(17) STRATA_GU_AMD(18) STRATA_GU_AMD(21) STRATA_GU_AMD(22) STRATA_GU_AMD(23)STRATA_GU_AMD(29) STRATA_GU_AMD(42)}} else#undef STRATA_GU_AMD+#endif+#if STRATA_EXPERT_NIBBLE_BUILT+ if (nib_layout(L)) // STRATA_EXPERT_NIBBLE: a converted layer's VRAM slots, native_gu_multi_kernel's grid and bits+ launch_gu_nib(L, dim3((unsigned) (strata::core::nibble::kRows / 16), (unsigned) gy), s, grp_ptr, grp_start,+ n_groups, ent_tok, X, gate, up);+ else#endifif (!expdec_gu(L, gy, cap_groups, s, grp_ptr, grp_start, n_groups, ent_tok, X, gate, up)) // STRATA_EXPDEC=1switch (L.gu_type) {@@ -5537,6 +6294,7 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long}#endifcheck("native_expert_grouped/gu");+ if (g_exp_phase == 3) return; // the bench: gate/up onlyconst long long nh = (long long) cap_entries * L.n_ff;#if defined(STRATA_HIP_GFX906)// gfx906: the separate SwiGLU and q8_1 passes stay the default (the A/B baseline); the RDNA one-pass@@ -5611,6 +6369,164 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long} // namespace strata::kernels+// ---------------------------------------------------------------- STRATA_EXPERT_NIBBLE: expert_nibble.hpp's functions+namespace strata::core::nibble {++#if STRATA_EXPERT_NIBBLE_BUILT+namespace {+int g_rev_ready[64]; // per device: 0 not built, 1 built, 2 a grid entry not unique (refused)++bool rev_ready() {+ int dev = 0;+ return cudaGetDevice(&dev) == cudaSuccess && dev >= 0 && dev < 64 && g_rev_ready[dev] == 1;+}+} // namespace+#endif++int prepare(void* stream) {+#if STRATA_EXPERT_NIBBLE_BUILT+ int dev = 0;+ if (cudaGetDevice(&dev) != cudaSuccess || dev < 0 || dev >= 64) return (int) cudaErrorInvalidDevice;+ if (g_rev_ready[dev] == 1) return 0;+ if (g_rev_ready[dev] == 2) return (int) cudaErrorInvalidValue;+ cudaStream_t s = (cudaStream_t) stream;+ strata::kernels::nib_rev_clear_kernel<<<(strata::kernels::kNibRev22 + 255) / 256, 256, 0, s>>>();+ strata::kernels::nib_rev_fill_kernel<<<(1024 + 256 + 512 + 255) / 256, 256, 0, s>>>();+ cudaError_t e = cudaGetLastError();+ unsigned long long bad = 0;+ if (e == cudaSuccess) e = cudaStreamSynchronize(s);+ if (e == cudaSuccess) e = cudaMemcpyFromSymbol(&bad, strata::kernels::g_nib_rev_bad, sizeof bad);+ if (e != cudaSuccess) return (int) e;+ g_rev_ready[dev] = bad == 0 ? 1 : 2;+ return bad == 0 ? 0 : (int) cudaErrorInvalidValue;+#else+ (void) stream;+ return 1; // cudaErrorInvalidValue: not in this build+#endif+}++int transcode(const LayerVram& L, const void* staging, int64_t n, const unsigned long long* slot_ptrs,+ unsigned long long* bad, void* stream) {+#if STRATA_EXPERT_NIBBLE_BUILT+ if (n <= 0) return 0;+ if (!L.on || !fmt_built(L.gu_type) || L.raw_gu_row != raw_gu_row(L.gu_type) ||+ L.raw_up_off != (uint64_t) kNFf * L.raw_gu_row || L.raw_bytes < 2 * L.raw_up_off)+ return (int) cudaErrorInvalidValue;+ const long long nthr = (long long) n * kRows * kItems;+ const dim3 grid((unsigned) ((nthr + 255) / 256));+ cudaStream_t s = (cudaStream_t) stream;+ const uint8_t* src = (const uint8_t*) staging;+#define STRATA_NIB_TC(TG) \+ strata::kernels::nib_transcode_kernel<TG><<<grid, 256, 0, s>>>(src, L.raw_bytes, L.raw_gu_row, L.raw_up_off, slot_ptrs, n, bad)+ switch (L.gu_type) {+ case kIQ2_S: STRATA_NIB_TC(22); break;+ case kIQ3_XXS: STRATA_NIB_TC(18); break;+ case kIQ3_S: STRATA_NIB_TC(21); break;+ default: return (int) cudaErrorInvalidValue;+ }+#undef STRATA_NIB_TC+ return (int) cudaGetLastError();+#else+ (void) L; (void) staging; (void) n; (void) slot_ptrs; (void) bad; (void) stream;+ return 1; // cudaErrorInvalidValue: not in this build+#endif+}++#if STRATA_EXPERT_NIBBLE_BUILT+namespace {+int launch_gguf(const LayerVram& L, strata::kernels::NibGgufArgs& a, bool check, cudaStream_t s) {+ namespace K = strata::kernels;+ if (!L.on || !fmt_built(L.gu_type) || L.raw_gu_row != raw_gu_row(L.gu_type) || L.down_bytes % 16 != 0 ||+ L.raw_up_off != (uint64_t) kNFf * L.raw_gu_row || L.raw_down_off != 2 * L.raw_up_off)+ return (int) cudaErrorInvalidValue;+ if (!rev_ready()) return (int) cudaErrorNotReady; // prepare() was not run on this device+ a.gu_row = L.raw_gu_row;+ a.down16 = L.down_bytes / 16;+ const int64_t tail = (int64_t) (a.down16 + 2 * a.zero16);+ const int64_t per = (int64_t) K::kNibGgufThreads * K::kNibGgufWords;+ const dim3 grid((unsigned) (kRows / K::kNibGgufRows + (tail + per - 1) / per), (unsigned) (a.n - a.first));+#define STRATA_NIB_GG(TG) \+ (check ? K::nib_gguf_kernel<TG, true><<<grid, K::kNibGgufThreads, 0, s>>>(a) \+ : K::nib_gguf_kernel<TG, false><<<grid, K::kNibGgufThreads, 0, s>>>(a))+ switch (L.gu_type) {+ case kIQ2_S: STRATA_NIB_GG(22); break;+ case kIQ3_XXS: STRATA_NIB_GG(18); break;+ case kIQ3_S: STRATA_NIB_GG(21); break;+ default: return (int) cudaErrorInvalidValue;+ }+#undef STRATA_NIB_GG+ return (int) cudaGetLastError();+}+} // namespace+#endif++int check_gguf(const LayerVram& L, const void* staging, int64_t n, const unsigned long long* slot_ptrs,+ unsigned long long* bad, void* stream) {+#if STRATA_EXPERT_NIBBLE_BUILT+ if (n <= 0) return 0;+ if (n > 65535) return (int) cudaErrorInvalidValue;+ strata::kernels::NibGgufArgs a{};+ a.dev = slot_ptrs;+ a.first = 0;+ a.n = (int) n;+ a.gu = (uint8_t*) staging;+ a.gu_stride = L.raw_bytes;+ a.d = (uint8_t*) staging + L.raw_down_off;+ a.d_stride = L.raw_bytes;+ a.zero16 = 0;+ a.bad = bad;+ if (((uintptr_t) staging | L.raw_bytes) % 16 != 0) return (int) cudaErrorInvalidValue;+ return launch_gguf(L, a, true, (cudaStream_t) stream);+#else+ (void) L; (void) staging; (void) n; (void) slot_ptrs; (void) bad; (void) stream;+ return 1; // cudaErrorInvalidValue: not in this build+#endif+}++int gather_gguf(const LayerVram& L, const uint8_t* const* slots, int first, int n, void* gu_dst, uint64_t gu_stride,+ void* d_dst, uint64_t d_stride, uint64_t zero_bytes, void* stream) {+#if STRATA_EXPERT_NIBBLE_BUILT+ if (first < 0 || n <= first || n > 16) return (int) cudaErrorInvalidValue;+ uintptr_t al = (uintptr_t) gu_dst | (uintptr_t) d_dst | gu_stride | d_stride | zero_bytes;+ for (int q = first; q < n; ++q) al |= (uintptr_t) slots[q];+ if (al % 16 != 0 || gu_stride < (uint64_t) kRows * L.raw_gu_row || d_stride < L.down_bytes) return (int) cudaErrorInvalidValue;+ strata::kernels::NibGgufArgs a{};+ a.dev = nullptr;+ for (int q = first; q < n; ++q) a.p[q] = slots[q];+ a.first = first;+ a.n = n;+ a.gu = (uint8_t*) gu_dst;+ a.gu_stride = gu_stride;+ a.d = (uint8_t*) d_dst;+ a.d_stride = d_stride;+ a.zero16 = zero_bytes / 16;+ a.bad = nullptr;+ return launch_gguf(L, a, false, (cudaStream_t) stream);+#else+ (void) L; (void) slots; (void) first; (void) n; (void) gu_dst; (void) gu_stride; (void) d_dst; (void) d_stride;+ (void) zero_bytes; (void) stream;+ return 1; // cudaErrorInvalidValue: not in this build+#endif+}++int hash_slots(const LayerVram& L, const unsigned long long* slot_ptrs, int64_t n, unsigned long long* out, void* stream) {+#if STRATA_EXPERT_NIBBLE_BUILT+ if (n <= 0) return 0;+ if (L.hashed == 0) return (int) cudaErrorInvalidValue;+ cudaStream_t s = (cudaStream_t) stream;+ const cudaError_t e = cudaMemsetAsync(out, 0, (size_t) n * sizeof(unsigned long long), s);+ if (e != cudaSuccess) return (int) e;+ // ~2.7 MB a slot: 32 blocks of 256 threads each, ~21 16-byte loads a thread+ strata::kernels::nib_hash_kernel<<<dim3(32, (unsigned) n), 256, 0, s>>>(slot_ptrs, L.hashed, out);+ return (int) cudaGetLastError();+#else+ (void) L; (void) slot_ptrs; (void) n; (void) out; (void) stream;+ return 1; // cudaErrorInvalidValue: not in this build+#endif+}++} // namespace strata::core::nibble+// ---- Volta prompt experts (STRATA_PF_WMMA): a group's experts as one launch of FP16 tensor-core tiles, the weights// dequantized into shared memory by dq_dispatch (the FP16 path's own formulas), the activations FP16, FP32 sums.// Replaces MMQ's dp4a products on sm_70, and the FP16 path's dequantize-to-global + one cuBLAS call per expert.diff --git a/src/kernels/cuda/nib_gu_proto.cuh b/src/kernels/cuda/nib_gu_proto.cuhnew file mode 100644index 000000000..368eeb335--- /dev/null+++ b/src/kernels/cuda/nib_gu_proto.cuh@@ -0,0 +1,137 @@+// src/kernels/cuda/nib_gu_proto.cuh - STRATA_NIB_GU_PROTO: a staging variant of nib_gu_kernel (STRATA_EXPERT_NIBBLE's+// 4-bit gate/up), included by iq_kernels.cu inside its anonymous namespace after nib_gu_kernel.+//+// The kernel keeps the S skeleton's per-row arithmetic by calling nib_row_dot_s itself (16 lanes a row, lane t's items+// k = t, t+32, t+64 into s and t+16, t+48 into s16, NibLoad<TG>::load then Split<TG>::apply on the staged block_q8_1,+// __fadd_rn(s, s16), xor 8, 4, 2, 1). Only the staging and the block -> (group, rows) mapping change:+// - a block takes RB row blocks (16 rows each) of one group: grid (80 / RB, gy). The group's entries are staged+// once and the RB row blocks run one after another on them;+// - the staging reads the entries' q8_1 rows (2,880 B = 180 x 16 B each, 16-aligned when xq is) as 16-byte loads,+// every load issued before the first store, so only the stores sit between the two barriers. The S skeleton's+// word loop (nib_pass_s) has a serial load -> store chain there that stalls all eight warps of the block.+// The outputs are written without nib_pass_s's dynamically indexed s[t] (a select on t), the same values.+// Id 31 = RB 2, the one STRATA_NIB_GU_PROTO=1 selects: one GP100 die (PH402), T=1, gate/up alone, IQ2_S / IQ3_XXS /+// IQ3_S x1.25-1.27 against the S / P skeleton, every gate and up row bitwise equal (iq_nibble_parity (7),+// native_expert_bench --nibble --vs 31).+#pragma once++// The n (1..GRP_NC) entries from e into s_x (block_q8_1 rows of 80). Block-uniform (e, n): every thread reaches both+// barriers. 16-byte loads, all in flight before the first store.+__device__ __forceinline__ void nibx_stage(block_q8_1* s_x, const int32_t* __restrict__ ent_tok,+ const block_q8_1* __restrict__ xq, int e, int n) {+ constexpr int kV = (GRP_NC * 180 + 255) / 256; // 3+ const int nv = n * 180;+ uint4 v[kV];+#pragma unroll+ for (int j = 0; j < kV; ++j) {+ const int i = threadIdx.x + 256 * j;+ if (i < nv) {+ const int c = i >= 540 ? 3 : i >= 360 ? 2 : i >= 180 ? 1 : 0;+ v[j] = ((const uint4*) (xq + (size_t) ent_tok[e + c] * 80))[i - 180 * c];+ }+ }+ __syncthreads();+#pragma unroll+ for (int j = 0; j < kV; ++j) {+ const int i = threadIdx.x + 256 * j;+ if (i < nv) ((uint4*) s_x)[i] = v[j];+ }+ __syncthreads();+}++// nib_pass_s's compute on the staged entries (nib_row_dot_s, the production function), stored without the dynamically+// indexed s[t]+template<int TG>+__device__ __forceinline__ void nibx_run(const uint8_t* slot, int row, const block_q8_1* s_x, int e, int n, int t,+ float* dst, int r) {+ const uint4* crow = (const uint4*) (slot + nibk::kCodesOff) + (size_t) row * nibk::kItems;+ const uint8_t* srow = slot + nibk::kScOff + (size_t) row * nibk::kItems;+ const __half* drow = (const __half*) (slot + nibk::kDOff) + (size_t) row * nibk::kSuper;+ if (n == 1) {+ const int o[1] = { 0 }; float s[1];+ nib_row_dot_s<TG, 1>(crow, srow, drow, s_x, o, t, s);+ if (t == 0) dst[(size_t) e * nibk::kNFf + r] = s[0];+ } else if (n == 2) {+ const int o[2] = { 0, 80 }; float s[2];+ nib_row_dot_s<TG, 2>(crow, srow, drow, s_x, o, t, s);+ if (t < 2) dst[(size_t) (e + t) * nibk::kNFf + r] = t == 0 ? s[0] : s[1];+ } else if (n == 3) {+ const int o[3] = { 0, 80, 160 }; float s[3];+ nib_row_dot_s<TG, 3>(crow, srow, drow, s_x, o, t, s);+ if (t < 3) dst[(size_t) (e + t) * nibk::kNFf + r] = t == 0 ? s[0] : t == 1 ? s[1] : s[2];+ } else {+ const int o[4] = { 0, 80, 160, 240 }; float s[4];+ nib_row_dot_s<TG, 4>(crow, srow, drow, s_x, o, t, s);+ if (t < 4) dst[(size_t) (e + t) * nibk::kNFf + r] = t == 0 ? s[0] : t == 1 ? s[1] : t == 2 ? s[2] : s[3];+ }+}++template<int TG, int RB>+__global__ void __launch_bounds__(256) nibx_gu_kernel(const unsigned long long* __restrict__ grp_ptr,+ const int32_t* __restrict__ grp_start,+ const int32_t* __restrict__ n_groups,+ const int32_t* __restrict__ ent_tok,+ const block_q8_1* __restrict__ xq, float* __restrict__ gate,+ float* __restrict__ up) {+ static_assert(80 % RB == 0, "row blocks per block divide 80");+ const int ng = *n_groups;+ if (blockIdx.y >= (unsigned) ng) return;+ STRATA_SHARED_ALIGN16_U32(s_buf, kNibSWords);+ block_q8_1* s_x = (block_q8_1*) s_buf;+ const int subwarp = threadIdx.x >> 4, t = threadIdx.x & 15;+ const int rb0 = blockIdx.x * RB;+ for (int g = blockIdx.y; g < ng; g += gridDim.y) {+ const uint8_t* slot = (const uint8_t*) grp_ptr[g];+ const int e0 = grp_start[g], e1 = grp_start[g + 1];+ const int nt = e1 - e0;+ if (nt <= 0) continue; // an empty group: nothing to write (block-uniform, no barrier skipped unevenly)+ if (nt <= GRP_NC) {+ nibx_stage(s_x, ent_tok, xq, e0, nt);+#pragma unroll 1+ for (int j = 0; j < RB; ++j) {+ const int row = (rb0 + j) * 16 + subwarp;+ const bool is_up = row >= nibk::kNFf;+ nibx_run<TG>(slot, row, s_x, e0, nt, t, is_up ? up : gate, is_up ? row - (int) nibk::kNFf : row);+ }+ } else { // more entries than one pass holds: per row block, the passes as nib_pass_s runs them+#pragma unroll 1+ for (int j = 0; j < RB; ++j) {+ const int row = (rb0 + j) * 16 + subwarp;+ const bool is_up = row >= nibk::kNFf;+ for (int e = e0; e < e1; e += GRP_NC) {+ const int n = min(GRP_NC, e1 - e);+ nibx_stage(s_x, ent_tok, xq, e, n);+ nibx_run<TG>(slot, row, s_x, e, n, t, is_up ? up : gate, is_up ? row - (int) nibk::kNFf : row);+ }+ }+ }+ }+}++int g_nibx_occ = 0; // the last attribute query's resident blocks per SM++template<int TG, int RB>+void nibx_launch_one(dim3 grid, cudaStream_t s, const unsigned long long* grp_ptr, const int32_t* grp_start,+ const int32_t* n_groups, const int32_t* ent_tok, const block_q8_1* X, float* gate, float* up,+ cudaFuncAttributes* attr) {+ if (attr) {+ cudaFuncGetAttributes(attr, nibx_gu_kernel<TG, RB>);+ cudaOccupancyMaxActiveBlocksPerMultiprocessor(&g_nibx_occ, nibx_gu_kernel<TG, RB>, 256, 0);+ return;+ }+ nibx_gu_kernel<TG, RB><<<dim3(80 / RB, grid.y), 256, 0, s>>>(grp_ptr, grp_start, n_groups, ent_tok, X, gate, up);+}++// false: not the gu-proto id (31) or not a converted format. attr != nullptr: only the kernel's attributes, nothing+// launched.+bool nibx_launch(int gu_type, int id, dim3 grid, cudaStream_t s, const unsigned long long* grp_ptr,+ const int32_t* grp_start, const int32_t* n_groups, const int32_t* ent_tok, const block_q8_1* X,+ float* gate, float* up, cudaFuncAttributes* attr = nullptr) {+ if (id != 31) return false;+ switch (gu_type) {+ case 22: nibx_launch_one<22, 2>(grid, s, grp_ptr, grp_start, n_groups, ent_tok, X, gate, up, attr); return true;+ case 18: nibx_launch_one<18, 2>(grid, s, grp_ptr, grp_start, n_groups, ent_tok, X, gate, up, attr); return true;+ case 21: nibx_launch_one<21, 2>(grid, s, grp_ptr, grp_start, n_groups, ent_tok, X, gate, up, attr); return true;+ default: return false;+ }+}diff --git a/src/kernels/iq_nibble_parity.cpp b/src/kernels/iq_nibble_parity.cppnew file mode 100644index 000000000..40196bdd0--- /dev/null+++ b/src/kernels/iq_nibble_parity.cpp@@ -0,0 +1,954 @@+// src/kernels/iq_nibble_parity.cpp - STRATA_EXPERT_NIBBLE (strata/core/expert_nibble.hpp): the 4-bit gate/up expert+// slots against the production kernels and the GGUF blobs, bit for bit.+//+// build/iq_nibble_parity --header the host half of expert_nibble.hpp (no GPU)+// build/iq_nibble_parity <shard.gguf> [<shard.gguf> ...] [--layers-per-fmt 2] [--layers 1,17] [--ne 512]+// [--plans 8] [--synthetic 64] [--no-ggml] [--negative]+// build/iq_nibble_parity --synthetic-only [--synthetic 64] [--plans 8] [--no-ggml] no model, no shard+//+// Each model (any shard of it; its other shards are opened beside it) gives `--layers-per-fmt` layers of each+// converted format (IQ2_S, IQ3_XXS, IQ3_S; or `--layers`), `--ne` experts each, read from the GGUF tensors into the+// blob the pack holds (gate rows, up rows, down rows). Per layer:+// (1) nibble::transcode, 64 blobs a batch as the startup fill runs it: 0 items that do not round-trip;+// (2) every gate/up byte of a slot is written (8 slots transcoded again into another pre-fill byte);+// (3) nibble::hash_slots against nibble::hash_host on every slot read back (and a flipped byte must change it);+// (4) the host codec on every item: decode_item then encode_item gives the code back;+// (5) ggml's reference dequantizer (to_float) on every GGUF row against the weight the 4-bit slot gives through+// dequantize_row_iq2_s / _iq3_xxs / _iq3_s's own expression (code, scale byte, fp16 d): every weight, bit for bit;+// (6) nibble::check_gguf (the start-up check: the GGUF blocks rebuilt from every slot against the blob): 0 words+// differ; nibble::gather_gguf (the prompt path's MMQ gather) in groups of 16 and of 1..15 from a later first+// position: the rebuilt [gate | up] and the down rows byte for byte the blob's, the MMQ tail after the group zero;+// (7) native_expert_grouped on the converted slots (native_expert_layout_nib) against the same call on the GGUF+// blobs (native_expert_layout): hq and out byte for byte for every entry, and gate / up where the reference+// layout writes them - in the card's default layout (a GP100: mode 8, whose gate/up kernel writes hq) and in the+// CUDA layout (native_expert_set_mode(0)), each with native_grouped_set_v1 off and on (h too, in mode 0 + v1);+// top-10 routing T = 1..6 at 0% and 50% sharing (`--plans` each), one group of T entries (G = 1) per T, plans of+// groups with 0..11 entries (more than one GRP_NC pass, empty groups); grid_groups 0 (cap) and 4 (the PCIe+// call's stride); the default kernel per format and each skeleton forced (S, P); q8_1 activations random, all+// +127, all -127 and alternating +-127 (the sum of x at its extremes, +-4064).+// `--synthetic N`: per format, N experts whose gate/up blocks are random bytes (every grid index, sign byte and scale+// nibble the format has; the coverage is counted and must be complete) and whose fp16 d cycle through +-the smallest+// subnormal, +-65504, +-1, the smallest normal and two ordinary values; checks (1)-(7) on them.+// `--synthetic-only`: the synthetic experts alone, for every gate/up format x both down types (IQ4_NL, Q2_0), the down rows+// random bytes with ordinary fp16 scales: checks (1)-(7) with no model file (a GP100 is still needed).+// `--negative`: the control - one code byte per slot changed after the transcode; (6) and (7) must then report differences.+// Exit 0 when every check is equal; 1 otherwise; 2 on a usage or setup error. A build without STRATA_EXPERT_NIBBLE_BUILT+// (the CUDA build with STRATA_EXPERIMENTAL_SM60) has no 4-bit kernels: --header only.+#include "strata/artifact/gguf_reader.hpp"+#include "strata/core/expert_nibble.hpp"+#include "strata/kernels/cpu/native_expert.hpp"+#include "strata/kernels/iq_kernels.hpp"++#include "ggml.h"++#include <cuda_runtime.h>++#include <algorithm>+#include <atomic>+#include <cstdio>+#include <cstdlib>+#include <cstring>+#include <memory>+#include <random>+#include <set>+#include <string>+#include <thread>+#include <vector>++namespace {++namespace nib = strata::core::nibble;+namespace K = strata::kernels;+namespace cpu = strata::kernels::cpu;++int fails = 0;+void expect(bool ok, const char* what) {+ if (!ok) {+ std::printf("FAIL %s\n", what);+ ++fails;+ }+}++// Every value the format holds at every one of an item's 32 positions (the other positions cycle through the set),+// encoded, decoded and compared; and one value outside the set must be refused.+void round_trip(int fmt, const std::vector<int8_t>& set, int8_t outside) {+ int8_t g[32], back[32];+ uint32_t q[4];+ for (size_t shift = 0; shift < set.size(); ++shift) {+ for (int i = 0; i < 32; ++i) g[i] = set[(shift + (size_t) i) % set.size()];+ const bool ok = nib::encode_item(fmt, g, q);+ nib::decode_item(fmt, q, back);+ expect(ok && std::memcmp(g, back, 32) == 0, (std::string(nib::fmt_name(fmt)) + " round trip").c_str());+ }+ for (int i = 0; i < 32; ++i) g[i] = set[0];+ g[17] = outside;+ expect(!nib::encode_item(fmt, g, q), (std::string(nib::fmt_name(fmt)) + " refuses a value outside the set").c_str());+}++int header_check() {+ expect(nib::kGuBytes == 1766400 && nib::kScOff == 1638400 && nib::kDOff == 1740800, "layout sizes");+ expect(nib::code_off(nib::kNFf, 0) == nib::kUpCodesOff, "up rows' codes");+ expect(nib::slot_bytes_for(921600) % nib::kSlotAlign == 0, "slot alignment");+ round_trip(nib::kIQ2_S, { 8, 25, 43, -8, -25, -43 }, 9);+ round_trip(nib::kIQ3_XXS, { 4, 12, 20, 28, 36, 44, 52, 62, -4, -12, -20, -28, -36, -44, -52, -62 }, 5);+ std::vector<int8_t> s3;+ for (int v = -15; v <= 15; v += 2) s3.push_back((int8_t) v);+ round_trip(nib::kIQ3_S, s3, 2);+ // IQ2_S's nibble is the decode kernel's PRMT selector into {8,25,43,0} {-8,-25,-43,0}+ {+ int8_t g[32] = {}, back[32];+ uint32_t q[4];+ for (int i = 0; i < 32; ++i) g[i] = (int8_t) (i % 2 ? -25 : 43);+ nib::encode_item(nib::kIQ2_S, g, q);+ expect((q[0] & 0xFu) == 2u && (q[0] >> 4 & 0xFu) == 5u, "IQ2_S selector nibbles");+ nib::decode_item(nib::kIQ2_S, q, back);+ expect(std::memcmp(g, back, 32) == 0, "IQ2_S selector decode");+ }+ expect(nib::gu_skeleton(nib::kIQ2_S) == nib::kSkelP && nib::gu_skeleton(nib::kIQ3_XXS) == nib::kSkelS &&+ nib::gu_skeleton(nib::kIQ3_S) == nib::kSkelS,+ "kernel table");+ {+ const uint8_t b[6] = { 1, 2, 3, 4, 5, 6 };+ const uint64_t h = nib::hash_host(b, 6);+ const uint64_t want = (uint64_t) (0x04030201u ^ 0u) * nib::kHashMul + (uint64_t) (0x00000605u ^ 1u) * nib::kHashMul;+ expect(h == want, "hash_host");+ }+ std::printf("%s (%d failures)\n", fails == 0 ? "header OK" : "header FAILED", fails);+ return fails == 0 ? 0 : 1;+}++// ------------------------------------------------------------------------------------------------ the GPU parity++#define CK(x) \+ do { \+ cudaError_t e_ = (x); \+ if (e_ != cudaSuccess) { \+ std::fprintf(stderr, "%s:%d %s: %s\n", __FILE__, __LINE__, #x, cudaGetErrorString(e_)); \+ std::exit(2); \+ } \+ } while (0)++struct Q81 { // block_q8_1: fp16 d, fp16 s (= d * sum), 32 int8+ uint16_t d, s;+ int8_t qs[32];+};+static_assert(sizeof(Q81) == 36, "block_q8_1");++constexpr int64_t kEmbd = nib::kNEmbd, kFf = nib::kNFf;+constexpr int kTok = 12; // activation rows: T <= 6 for routing plans, 12 for the 0..11-entry plans++// One layer: `ne` GGUF blobs (gate rows | up rows | down rows, as the pack and the host arena hold them) on the host+// and the device, and their converted slots.+struct Layer {+ std::string tag;+ int layer = -1;+ cpu::NativeFmt f;+ nib::LayerVram V;+ K::NativeExpertLayout Lr, Ln;+ int ne = 0;+ std::vector<uint8_t> host; // ne x f.bytes+ uint8_t* raw = nullptr; // device copy of `host`+ uint8_t* slots = nullptr; // ne x V.slot_bytes (the last one only `hashed` long: nothing past it is readable)+};++bool layer_fmt(const strata::GgufModel& m, int l, int& gt, int& dt) {+ const auto* g = m.find("blk." + std::to_string(l) + ".ffn_gate_exps.weight");+ const auto* d = m.find("blk." + std::to_string(l) + ".ffn_down_exps.weight");+ if (g == nullptr || d == nullptr) return false;+ gt = (int) g->type;+ dt = (int) d->type;+ return true;+}++// The blobs of experts [0, ne) of layer l.+bool read_layer(const strata::GgufModel& m, int l, int ne, Layer& Y) {+ const char* roles[3] = { "gate", "up", "down" };+ const strata::TensorInfo* t[3] = {};+ size_t sh[3] = {};+ for (int r = 0; r < 3; ++r) t[r] = m.find("blk." + std::to_string(l) + ".ffn_" + roles[r] + "_exps.weight", &sh[r]);+ if (!t[0] || !t[1] || !t[2]) return false;+ std::string err;+ if (!cpu::native_fmt((int) t[0]->type, (int) t[2]->type, kEmbd, kFf, Y.f, err)) {+ std::printf("layer %d: %s\n", l, err.c_str());+ return false;+ }+ const size_t gu_role = Y.f.up_off, dn_role = Y.f.bytes - Y.f.down_off;+ Y.layer = l;+ Y.ne = ne;+ Y.host.resize((size_t) ne * Y.f.bytes);+ const uint8_t* G = m.shard(sh[0]).tensor_data(*t[0]);+ const uint8_t* U = m.shard(sh[1]).tensor_data(*t[1]);+ const uint8_t* D = m.shard(sh[2]).tensor_data(*t[2]);+ for (int e = 0; e < ne; ++e) {+ uint8_t* b = Y.host.data() + (size_t) e * Y.f.bytes;+ std::memcpy(b, G + (size_t) e * gu_role, gu_role);+ std::memcpy(b + Y.f.up_off, U + (size_t) e * gu_role, gu_role);+ std::memcpy(b + Y.f.down_off, D + (size_t) e * dn_role, dn_role);+ }+ return true;+}++// check (6): nibble::check_gguf over every slot (the start-up check), and nibble::gather_gguf (the prompt path's gather)+// in groups of 16 and from later first positions, against the blobs byte for byte (and the MMQ tail zeroed).+// `expect_diff`: the negative control (only the report says so: the counts must then be > 0).+long long gguf_parity(const Layer& Y, const unsigned long long* dp, bool expect_diff) {+ long long bad_total = 0;+ const int ne = Y.ne;+ unsigned long long* dbad = nullptr;+ CK(cudaMalloc(&dbad, 8));+ CK(cudaMemset(dbad, 0, 8));+ for (int e0 = 0; e0 < ne; e0 += 64) {+ const int rc = nib::check_gguf(Y.V, Y.raw + (size_t) e0 * Y.V.raw_bytes, std::min(64, ne - e0), dp + e0, dbad, nullptr);+ if (rc != 0) {+ std::printf("check_gguf: %s\n", cudaGetErrorString((cudaError_t) rc));+ std::exit(2);+ }+ }+ CK(cudaDeviceSynchronize());+ unsigned long long cbad = 0;+ CK(cudaMemcpy(&cbad, dbad, 8, cudaMemcpyDeviceToHost));+ CK(cudaFree(dbad));+ // the gathers: MMQ's group buffers (16 experts of [gate | up] and of down, each followed by the 4096-byte tail)+ const uint64_t gub = (uint64_t) nib::kRows * Y.f.gu_row, db = Y.V.down_bytes, zero = 4096;+ uint8_t *ggu = nullptr, *gd = nullptr;+ CK(cudaMalloc(&ggu, 16 * gub + zero));+ CK(cudaMalloc(&gd, 16 * db + zero));+ std::vector<uint8_t> hgu(16 * gub + zero), hd(16 * db + zero);+ std::vector<unsigned long long> hp(ne);+ CK(cudaMemcpy(hp.data(), dp, (size_t) ne * 8, cudaMemcpyDeviceToHost));+ long long gbad = 0, groups = 0;+ for (int e0 = 0; e0 < ne; e0 += 16) {+ const int first = (e0 / 16) % 4 == 3 ? 1 + (e0 / 64) % 15 : 0; // every 4th group from a later position+ const int n = std::min(16, ne - e0);+ if (first >= n) continue;+ const uint8_t* sl[16] = {};+ for (int q = first; q < n; ++q) sl[q] = (const uint8_t*) (uintptr_t) hp[(size_t) (e0 + q)];+ CK(cudaMemset(ggu, 0xA5, 16 * gub + zero));+ CK(cudaMemset(gd, 0xA5, 16 * db + zero));+ const int rc = nib::gather_gguf(Y.V, sl, first, n, ggu, gub, gd, db, zero, nullptr);+ if (rc != 0) {+ std::printf("gather_gguf: %s\n", cudaGetErrorString((cudaError_t) rc));+ std::exit(2);+ }+ CK(cudaDeviceSynchronize());+ CK(cudaMemcpy(hgu.data(), ggu, hgu.size(), cudaMemcpyDeviceToHost));+ CK(cudaMemcpy(hd.data(), gd, hd.size(), cudaMemcpyDeviceToHost));+ for (int q = first; q < n; ++q) {+ const uint8_t* raw = Y.host.data() + (size_t) (e0 + q) * Y.f.bytes;+ gbad += std::memcmp(hgu.data() + (size_t) q * gub, raw, gub) != 0;+ gbad += std::memcmp(hd.data() + (size_t) q * db, raw + Y.f.down_off, db) != 0;+ }+ for (uint64_t i = 0; i < zero; ++i) gbad += hgu[(size_t) n * gub + i] != 0 || hd[(size_t) n * db + i] != 0 ? 1 : 0;+ ++groups;+ }+ CK(cudaFree(ggu));+ CK(cudaFree(gd));+ std::printf(" check_gguf: %llu words differ; gather_gguf (%lld groups, the tail zeroed): %lld matrices or tail bytes differ%s\n",+ cbad, groups, gbad, expect_diff ? " (NEGATIVE CONTROL: both must be > 0)" : "");+ bad_total += (long long) cbad + gbad; // the negative control fails the run, as (7) does: it must+ return bad_total;+}++// Plans the layer, uploads the blobs, transcodes them into slots (checks 1-6).+long long convert(Layer& Y, bool ggml_check) {+ long long bad_total = 0;+ nib::Config c;+ c.on = true;+ Y.V = nib::plan_layer(Y.layer, Y.f, c);+ Y.Lr = K::native_expert_layout(Y.f.gu_type, Y.f.d_type, kEmbd, kFf);+ Y.Ln = K::native_expert_layout_nib(Y.f.gu_type, Y.f.d_type, kEmbd, kFf);+ if (!Y.V.on || Y.V.slot_bytes != Y.Ln.bytes || Y.V.down_off != Y.Ln.down_off || Y.Lr.bytes != Y.f.bytes) {+ std::printf("layer %d: plan_layer / native_expert_layout_nib disagree (on %d, slot %llu vs %zu)\n", Y.layer,+ (int) Y.V.on, (unsigned long long) Y.V.slot_bytes, Y.Ln.bytes);+ return 1;+ }+ const int ne = Y.ne;+ const size_t slots_bytes = (size_t) (ne - 1) * Y.V.slot_bytes + Y.V.hashed;+ CK(cudaMalloc(&Y.raw, Y.host.size()));+ CK(cudaMemcpy(Y.raw, Y.host.data(), Y.host.size(), cudaMemcpyHostToDevice));+ CK(cudaMalloc(&Y.slots, slots_bytes));+ CK(cudaMemset(Y.slots, 0xA5, slots_bytes));+ std::vector<unsigned long long> hp(ne);+ for (int e = 0; e < ne; ++e) hp[e] = (unsigned long long) (Y.slots + (size_t) e * Y.V.slot_bytes);+ unsigned long long *dp = nullptr, *dbad = nullptr, *dhash = nullptr;+ CK(cudaMalloc(&dp, (size_t) ne * 8));+ CK(cudaMalloc(&dbad, 8));+ CK(cudaMalloc(&dhash, (size_t) ne * 8));+ CK(cudaMemcpy(dp, hp.data(), (size_t) ne * 8, cudaMemcpyHostToDevice));+ CK(cudaMemset(dbad, 0, 8));+ // (1) the startup fill's batches: 64 blobs at a time, then the down rows behind the codes+ for (int e0 = 0; e0 < ne; e0 += 64) {+ const int n = std::min(64, ne - e0);+ const int rc = nib::transcode(Y.V, Y.raw + (size_t) e0 * Y.V.raw_bytes, n, dp + e0, dbad, nullptr);+ if (rc != 0) {+ std::printf("transcode: %s\n", cudaGetErrorString((cudaError_t) rc));+ std::exit(2);+ }+ }+ for (int e = 0; e < ne; ++e)+ CK(cudaMemcpyAsync(Y.slots + (size_t) e * Y.V.slot_bytes + nib::kDownOff,+ Y.raw + (size_t) e * Y.V.raw_bytes + Y.V.raw_down_off, Y.V.down_bytes, cudaMemcpyDeviceToDevice));+ CK(cudaDeviceSynchronize());+ unsigned long long bad = 0;+ CK(cudaMemcpy(&bad, dbad, 8, cudaMemcpyDeviceToHost));+ std::printf("%s layer %2d %-7s / down %-6s: %d experts, blob %zu B, slot %llu B (x%.3f); transcode: %llu of %lld items do not round-trip\n",+ Y.tag.c_str(), Y.layer, nib::fmt_name(Y.f.gu_type), ggml_type_name((ggml_type) Y.f.d_type), ne, Y.f.bytes,+ (unsigned long long) Y.V.slot_bytes, (double) Y.V.slot_bytes / (double) Y.f.bytes, bad,+ (long long) ne * nib::kRows * nib::kItems);+ bad_total += (long long) bad;+ bad_total += gguf_parity(Y, dp, false);+ // (2) every gate/up byte written: the first 8 blobs again into slots pre-filled with 0x5A+ {+ const int n8 = std::min(8, ne);+ uint8_t* alt = nullptr;+ unsigned long long* ap = nullptr;+ CK(cudaMalloc(&alt, (size_t) n8 * Y.V.slot_bytes));+ CK(cudaMemset(alt, 0x5A, (size_t) n8 * Y.V.slot_bytes));+ std::vector<unsigned long long> ah(n8);+ for (int e = 0; e < n8; ++e) ah[e] = (unsigned long long) (alt + (size_t) e * Y.V.slot_bytes);+ CK(cudaMalloc(&ap, (size_t) n8 * 8));+ CK(cudaMemcpy(ap, ah.data(), (size_t) n8 * 8, cudaMemcpyHostToDevice));+ CK(cudaMemset(dbad, 0, 8));+ if (nib::transcode(Y.V, Y.raw, n8, ap, dbad, nullptr) != 0) std::exit(2);+ CK(cudaDeviceSynchronize());+ std::vector<uint8_t> a(nib::kGuBytes), b(nib::kGuBytes);+ long long wbad = 0;+ for (int e = 0; e < n8; ++e) {+ CK(cudaMemcpy(a.data(), Y.slots + (size_t) e * Y.V.slot_bytes, nib::kGuBytes, cudaMemcpyDeviceToHost));+ CK(cudaMemcpy(b.data(), alt + (size_t) e * Y.V.slot_bytes, nib::kGuBytes, cudaMemcpyDeviceToHost));+ wbad += std::memcmp(a.data(), b.data(), nib::kGuBytes) != 0;+ }+ std::printf(" every gate/up byte written (0xA5 vs 0x5A pre-fill, %d slots): %lld differ\n", n8, wbad);+ bad_total += wbad;+ CK(cudaFree(alt));+ CK(cudaFree(ap));+ }+ // (3) the device hash of every slot against the host reference over the slot read back+ const int rc = nib::hash_slots(Y.V, dp, ne, dhash, nullptr);+ if (rc != 0) {+ std::printf("hash_slots: %s\n", cudaGetErrorString((cudaError_t) rc));+ std::exit(2);+ }+ std::vector<unsigned long long> hh(ne);+ CK(cudaMemcpy(hh.data(), dhash, (size_t) ne * 8, cudaMemcpyDeviceToHost));+ std::vector<uint8_t> sl(slots_bytes);+ CK(cudaMemcpy(sl.data(), Y.slots, slots_bytes, cudaMemcpyDeviceToHost));+ std::atomic<long long> hbad{ 0 }, cbad{ 0 }, dnbad{ 0 }, gbad{ 0 };+ std::atomic<long long> grows{ 0 };+ std::string first_ggml;+ std::atomic<int> first_set{ 0 };+ const ggml_type_traits* tr = ggml_get_type_traits((ggml_type) Y.f.gu_type);+ const unsigned nth = std::max(1u, std::min(16u, std::thread::hardware_concurrency()));+ std::vector<std::thread> th;+ for (unsigned ti = 0; ti < nth; ++ti)+ th.emplace_back([&, ti]() {+ std::vector<float> ref((size_t) kEmbd), got((size_t) kEmbd);+ for (int e = (int) ti; e < ne; e += (int) nth) {+ const uint8_t* s = sl.data() + (size_t) e * Y.V.slot_bytes;+ const uint8_t* raw = Y.host.data() + (size_t) e * Y.f.bytes;+ hbad += nib::hash_host(s, Y.V.hashed) != hh[e];+ if (e == 0) { // the hash sees every byte: one flipped must change it+ std::vector<uint8_t> t2(s, s + Y.V.hashed);+ t2[Y.V.hashed - 1] ^= 1;+ hbad += nib::hash_host(t2.data(), Y.V.hashed) == hh[e];+ }+ dnbad += std::memcmp(s + nib::kDownOff, raw + Y.V.raw_down_off, Y.V.down_bytes) != 0;+ for (int64_t r = 0; r < nib::kRows; ++r) {+ // (4) the host codec+ for (int64_t k = 0; k < nib::kItems; ++k) {+ uint32_t q[4], q2[4];+ int8_t g[32];+ std::memcpy(q, s + nib::code_off(r, k), 16);+ nib::decode_item(Y.f.gu_type, q, g);+ const bool ok = nib::encode_item(Y.f.gu_type, g, q2);+ cbad += !ok || std::memcmp(q, q2, 16) != 0;+ }+ if (!ggml_check) continue;+ // (5) ggml's dequantizer of the GGUF row against the slot's weights through its expression+ const uint8_t* row = raw + (r < kFf ? (size_t) r * Y.f.gu_row : Y.f.up_off + (size_t) (r - kFf) * Y.f.gu_row);+ tr->to_float(row, ref.data(), kEmbd);+ for (int64_t k = 0; k < nib::kItems; ++k) {+ uint32_t q[4];+ std::memcpy(q, s + nib::code_off(r, k), 16);+ int8_t gv[32];+ nib::decode_item(Y.f.gu_type, q, gv);+ const int sc = s[nib::scale_off(r, k)];+ ggml_fp16_t dh;+ std::memcpy(&dh, s + nib::d_off(r, k / 8), sizeof dh);+ const float d = ggml_fp16_to_fp32(dh);+ for (int j = 0; j < 32; ++j) {+ const float db = Y.f.gu_type == nib::kIQ2_S ? d * (0.5f + ((j < 16 ? sc : sc >> 4) & 0xf)) * 0.25f+ : Y.f.gu_type == nib::kIQ3_XXS ? d * (0.5f + sc) * 0.5f+ : d * sc;+ const int a = gv[j] < 0 ? -(int) gv[j] : (int) gv[j];+ float v = db * a;+ if (gv[j] < 0) v = -v;+ got[(size_t) (k * 32 + j)] = v;+ }+ }+ ++grows;+ if (std::memcmp(ref.data(), got.data(), ref.size() * sizeof(float)) != 0) {+ ++gbad;+ if (first_set.exchange(1) == 0) {+ size_t cc = 0;+ while (cc < ref.size() && std::memcmp(&ref[cc], &got[cc], 4) == 0) ++cc;+ char buf[200];+ std::snprintf(buf, sizeof buf, "expert %d row %lld column %zu: slot %.9g, GGUF %.9g", e,+ (long long) r, cc, (double) got[cc], (double) ref[cc]);+ first_ggml = buf;+ }+ }+ }+ }+ });+ for (auto& t : th) t.join();+ std::printf(" hash_slots vs hash_host: %lld of %d slots differ (+1 flipped-byte check); down rows: %lld of %d slots differ;"+ " host codec: %lld of %lld items differ\n",+ hbad.load(), ne, dnbad.load(), ne, cbad.load(), (long long) ne * nib::kRows * nib::kItems);+ if (ggml_check)+ std::printf(" ggml to_float of the GGUF rows vs the slot's weights: %lld of %lld rows differ%s%s\n", gbad.load(),+ grows.load(), first_ggml.empty() ? "" : "; first: ", first_ggml.c_str());+ bad_total += hbad + dnbad + cbad + gbad;+ CK(cudaFree(dp));+ CK(cudaFree(dbad));+ CK(cudaFree(dhash));+ return bad_total;+}++// ---- plans: synthetic top-10 routing and resident_plan_kernel's grouping (groups by first appearance)+struct Plan {+ int ng = 0, ne = 0, ntok = 0, maxn = 0;+ std::vector<int> gid, start, tok, dst;+};++void make_routing(std::mt19937& rng, int NE, int T, int share_pct, std::vector<int>& ids) {+ ids.clear();+ std::vector<int> prev, cur;+ for (int t = 0; t < T; ++t) {+ cur.clear();+ const bool sh = t > 0 && share_pct > 0;+ if (sh) {+ std::vector<int> p = prev;+ std::shuffle(p.begin(), p.end(), rng);+ cur.assign(p.begin(), p.begin() + 10 * share_pct / 100);+ }+ while ((int) cur.size() < 10) {+ const int e = (int) (rng() % (unsigned) NE);+ if (std::find(cur.begin(), cur.end(), e) != cur.end()) continue;+ if (sh && std::find(prev.begin(), prev.end(), e) != prev.end()) continue;+ cur.push_back(e);+ }+ std::shuffle(cur.begin(), cur.end(), rng);+ ids.insert(ids.end(), cur.begin(), cur.end());+ prev = cur;+ }+}++Plan build_plan(const std::vector<int>& ids) {+ const int n = (int) ids.size();+ Plan p;+ std::vector<int> first(n), cnt, grp_of(n, -1);+ for (int j = 0; j < n; ++j) {+ int f = j;+ for (int i = 0; i < j; ++i)+ if (ids[i] == ids[j]) { f = i; break; }+ first[j] = f;+ }+ for (int j = 0; j < n; ++j)+ if (first[j] == j) {+ grp_of[j] = p.ng++;+ p.gid.push_back(ids[j]);+ cnt.push_back(0);+ }+ for (int j = 0; j < n; ++j) cnt[grp_of[first[j]]]++;+ p.start.assign(p.ng + 1, 0);+ int acc = 0;+ for (int g = 0; g < p.ng; ++g) {+ p.start[g] = acc;+ acc += cnt[g];+ p.maxn = std::max(p.maxn, cnt[g]);+ }+ p.start[p.ng] = acc;+ p.tok.assign(n, 0);+ p.dst.assign(n, 0);+ std::vector<int> fill(p.ng, 0);+ for (int j = 0; j < n; ++j) {+ const int g = grp_of[first[j]];+ const int at = p.start[g] + fill[g]++;+ p.tok[at] = j / 10;+ p.dst[at] = j;+ }+ p.ne = n;+ p.ntok = n / 10;+ return p;+}++// one group of T entries (G = 1): an expert every token of the window routes to+Plan single_group(std::mt19937& rng, int NE, int T) {+ Plan p;+ p.ntok = T;+ p.ng = 1;+ p.gid.push_back((int) (rng() % (unsigned) NE));+ p.start = { 0, T };+ for (int t = 0; t < T; ++t) {+ p.tok.push_back(t);+ p.dst.push_back(T - 1 - t);+ }+ p.ne = T;+ p.maxn = T;+ return p;+}++// groups of 0..11 entries (more than one GRP_NC pass, empty groups), 12 tokens, distinct tokens within a group,+// destination rows in any order+Plan synthetic_plan(std::mt19937& rng, int NE) {+ Plan p;+ p.ntok = kTok;+ std::vector<int> ids(NE);+ for (int i = 0; i < NE; ++i) ids[i] = i;+ std::shuffle(ids.begin(), ids.end(), rng);+ const int sizes[] = { 3, 0, 11, 1, 5, 8, 0, 2, 4, 9, 6, 7, 10, 1 };+ int acc = 0;+ for (int g = 0; g < (int) (sizeof sizes / sizeof sizes[0]); ++g) {+ p.gid.push_back(ids[(size_t) g % ids.size()]);+ p.start.push_back(acc);+ std::vector<int> toks(kTok);+ for (int i = 0; i < kTok; ++i) toks[i] = i;+ std::shuffle(toks.begin(), toks.end(), rng);+ std::sort(toks.begin(), toks.begin() + sizes[g]);+ for (int i = 0; i < sizes[g]; ++i) {+ p.tok.push_back(toks[i]);+ p.dst.push_back(acc + i);+ }+ acc += sizes[g];+ p.maxn = std::max(p.maxn, sizes[g]);+ }+ p.start.push_back(acc);+ p.ng = (int) p.gid.size();+ p.ne = acc;+ std::vector<int> perm(acc);+ for (int i = 0; i < acc; ++i) perm[i] = i;+ std::shuffle(perm.begin(), perm.end(), rng);+ for (int i = 0; i < acc; ++i) p.dst[i] = perm[i];+ return p;+}++std::vector<Q81> make_x(std::mt19937& rng) {+ std::vector<Q81> x((size_t) kTok * (kEmbd / 32));+ for (size_t i = 0; i < x.size(); ++i) {+ Q81& b = x[i];+ const int tok = (int) (i / (kEmbd / 32));+ const float d = 0.002f + 0.03f * (float) (rng() % 1000) / 1000.0f;+ int sum = 0;+ for (int j = 0; j < 32; ++j) {+ int q = (int) (rng() % 255) - 127;+ if (tok == 1) q = 127; // the sum of x at its extremes (+-4064)+ if (tok == 2) q = -127;+ if (tok == 3 && (i % (kEmbd / 32)) < 8) q = (j & 1) ? 127 : -127;+ b.qs[j] = (int8_t) q;+ sum += q;+ }+ b.d = ggml_fp32_to_fp16(d);+ b.s = ggml_fp32_to_fp16(d * (float) sum);+ }+ return x;+}++struct Dev {+ unsigned long long* ptr = nullptr;+ int32_t *start = nullptr, *ng = nullptr, *tok = nullptr, *dst = nullptr;+};+Dev upload(const Plan& p, const Layer& Y, bool conv) {+ Dev d;+ std::vector<unsigned long long> hp(std::max(1, p.ng));+ for (int g = 0; g < p.ng; ++g)+ hp[g] = conv ? (unsigned long long) (Y.slots + (size_t) p.gid[g] * Y.V.slot_bytes)+ : (unsigned long long) (Y.raw + (size_t) p.gid[g] * Y.f.bytes);+ CK(cudaMalloc(&d.ptr, (size_t) std::max(1, p.ng) * 8));+ CK(cudaMalloc(&d.start, (size_t) (p.ng + 1) * 4));+ CK(cudaMalloc(&d.ng, 4));+ CK(cudaMalloc(&d.tok, (size_t) std::max(1, p.ne) * 4));+ CK(cudaMalloc(&d.dst, (size_t) std::max(1, p.ne) * 4));+ CK(cudaMemcpy(d.ptr, hp.data(), hp.size() * 8, cudaMemcpyHostToDevice));+ CK(cudaMemcpy(d.start, p.start.data(), (size_t) (p.ng + 1) * 4, cudaMemcpyHostToDevice));+ CK(cudaMemcpy(d.ng, &p.ng, 4, cudaMemcpyHostToDevice));+ if (p.ne) {+ CK(cudaMemcpy(d.tok, p.tok.data(), (size_t) p.ne * 4, cudaMemcpyHostToDevice));+ CK(cudaMemcpy(d.dst, p.dst.data(), (size_t) p.ne * 4, cudaMemcpyHostToDevice));+ }+ return d;+}+void release(Dev& d) {+ CK(cudaFree(d.ptr));+ CK(cudaFree(d.start));+ CK(cudaFree(d.ng));+ CK(cudaFree(d.tok));+ CK(cudaFree(d.dst));+}++struct Outs {+ std::vector<uint8_t> gate, up, h, hq, out;+};+// one native_expert_grouped call; scratch and out pre-filled with `fill` so an unwritten value shows+Outs run(const Layer& Y, const Plan& p, bool conv, const void* dx, void* scr, float* dout, int64_t gy, int mode,+ uint8_t fill) {+ // the window's capacity, as verify sizes it: T x 10 groups and entries (the blocks past the plan's groups idle)+ const int64_t cap_e = std::max<int64_t>((int64_t) p.ntok * 10, std::max(p.ne, p.ng)), cap_g = cap_e;+ Dev d = upload(p, Y, conv);+ CK(cudaMemset(scr, fill, K::native_expert_scratch_bytes(cap_e, kFf)));+ CK(cudaMemset(dout, fill, (size_t) cap_e * kEmbd * 4));+ K::native_expert_set_mode(mode, 0);+ K::native_expert_grouped(conv ? Y.Ln : Y.Lr, d.ptr, d.start, d.ng, d.dst, d.tok, cap_g, cap_e, dx, scr, dout, nullptr, gy);+ K::native_expert_set_mode(-1, 0);+ CK(cudaGetLastError());+ CK(cudaDeviceSynchronize());+ const size_t f = (size_t) cap_e * kFf * 4, fa = (f + 255) & ~(size_t) 255;+ Outs o;+ o.gate.resize((size_t) p.ne * kFf * 4);+ o.up.resize(o.gate.size());+ o.h.resize(o.gate.size());+ o.hq.resize((size_t) p.ne * (kFf / 32) * 36);+ o.out.resize((size_t) p.ne * kEmbd * 4);+ const uint8_t* s = (const uint8_t*) scr;+ if (p.ne) {+ CK(cudaMemcpy(o.gate.data(), s, o.gate.size(), cudaMemcpyDeviceToHost));+ CK(cudaMemcpy(o.up.data(), s + fa, o.up.size(), cudaMemcpyDeviceToHost));+ CK(cudaMemcpy(o.h.data(), s + 2 * fa, o.h.size(), cudaMemcpyDeviceToHost));+ CK(cudaMemcpy(o.hq.data(), s + 3 * fa, o.hq.size(), cudaMemcpyDeviceToHost));+ std::vector<uint8_t> all((size_t) cap_e * kEmbd * 4);+ CK(cudaMemcpy(all.data(), dout, all.size(), cudaMemcpyDeviceToHost));+ for (int e = 0; e < p.ne; ++e)+ std::memcpy(o.out.data() + (size_t) e * kEmbd * 4, all.data() + (size_t) p.dst[e] * kEmbd * 4, kEmbd * 4);+ }+ release(d);+ return o;+}++long long diff_rows(const std::vector<uint8_t>& a, const std::vector<uint8_t>& b, size_t row) {+ long long n = 0;+ for (size_t i = 0; i + row <= a.size(); i += row) n += std::memcmp(a.data() + i, b.data() + i, row) != 0;+ return n;+}++// check (7): every plan, both grid strides, every kernel choice, both layouts, v1 off / on, against the GGUF blobs+// through the production kernels+long long grouped_parity(const Layer& Y, std::mt19937& rng, int plans, const void* dx, void* scr, float* dout) {+ struct Case {+ Plan p;+ int T;+ };+ std::vector<Case> cases;+ for (int T = 1; T <= 6; ++T) {+ for (int share : { 0, 50 })+ for (int i = 0; i < plans; ++i) {+ std::vector<int> ids;+ make_routing(rng, Y.ne, T, share, ids);+ cases.push_back({ build_plan(ids), T });+ }+ cases.push_back({ single_group(rng, Y.ne, T), T });+ }+ for (int i = 0; i < 4; ++i) cases.push_back({ synthetic_plan(rng, Y.ne), kTok });+ int hist[13] = {};+ for (const Case& cs : cases)+ for (int g = 0; g < cs.p.ng; ++g) hist[std::min(12, cs.p.start[g + 1] - cs.p.start[g])]++;+ struct Var {+ int v, mode;+ bool v1;+ const char* name;+ };+ // mode -1: the card's default layout (a GP100: mode 8 - its reference gate/up kernel writes hq, not gate / up);+ // mode 0: the CUDA layout+ const Var vars[] = { { -1, -1, false, "default, card" }, { 0, -1, false, "S, card" }, { 1, -1, false, "P, card" },+ { -1, -1, true, "default, card, V1" }, { -1, 0, false, "default, mode 0" },+ { 0, 0, false, "S, mode 0" }, { 1, 0, false, "P, mode 0" }, { -1, 0, true, "default, mode 0, V1" },+ // STRATA_NIB_GU_PROTO (nib_gu_proto.cuh): the staged-once kernel, id 31+ { 31, -1, false, "gu-proto 31, card" }, { 31, 0, false, "gu-proto 31, mode 0" },+ { 31, 0, true, "gu-proto 31, mode 0, V1" } };+ constexpr int NV = (int) (sizeof vars / sizeof vars[0]);+ long long rows[NV] = {}, bad[NV][5] = {}, total = 0;+ for (const Case& cs : cases)+ for (int64_t gy : { (int64_t) 0, (int64_t) 4 }) {+ Outs ref[2][2]; // [mode 0][v1]+ bool have[2][2] = {};+ for (int vi = 0; vi < NV; ++vi) {+ const int m0 = vars[vi].mode == 0 ? 1 : 0, r1 = vars[vi].v1 ? 1 : 0;+ K::native_grouped_set_v1(vars[vi].v1);+ if (!have[m0][r1]) {+ K::native_expert_nib_set_variant(-1);+ ref[m0][r1] = run(Y, cs.p, false, dx, scr, dout, gy, vars[vi].mode, 0xFF);+ have[m0][r1] = true;+ }+ K::native_expert_nib_set_variant(vars[vi].v);+ const Outs o = run(Y, cs.p, true, dx, scr, dout, gy, vars[vi].mode, 0x00);+ rows[vi] += cs.p.ne;+ // gate / up: written by both arms in the CUDA layout, and by every arm of a striding call (gy 4: mode 0)+ const bool gu = vars[vi].mode == 0 || gy == 4;+ if (gu) {+ bad[vi][0] += diff_rows(ref[m0][r1].gate, o.gate, kFf * 4);+ bad[vi][1] += diff_rows(ref[m0][r1].up, o.up, kFf * 4);+ }+ if (vars[vi].v1 && gu) bad[vi][2] += diff_rows(ref[m0][r1].h, o.h, kFf * 4);+ bad[vi][3] += diff_rows(ref[m0][r1].hq, o.hq, (kFf / 32) * 36);+ bad[vi][4] += diff_rows(ref[m0][r1].out, o.out, kEmbd * 4);+ }+ }+ K::native_expert_nib_set_variant(-1);+ K::native_grouped_set_v1(false);+ std::printf(" plans: %zu (T 1..6 x sharing 0/50%% x %d, G=1 per T, 4 of 0..11 entries); groups by entries:", cases.size(),+ plans);+ for (int n = 0; n <= 12; ++n) std::printf(" %d:%d", n, hist[n]);+ std::printf("\n");+ for (int vi = 0; vi < NV; ++vi) {+ const long long b = bad[vi][0] + bad[vi][1] + bad[vi][2] + bad[vi][3] + bad[vi][4];+ std::printf(" native_expert_grouped %-20s: %6lld entries x 2 strides, rows differing: gate %lld, up %lld, h %s, hq %lld, out %lld\n",+ vars[vi].name, rows[vi], bad[vi][0], bad[vi][1],+ vars[vi].v1 ? std::to_string(bad[vi][2]).c_str() : "-", bad[vi][3], bad[vi][4]);+ total += b;+ }+ return total;+}++// --synthetic: random gate/up blocks, d cycling through the fp16 extremes; the coverage of each field is counted+long long synthesize(Layer& Y, std::mt19937& rng) {+ static const uint16_t kD[] = { 0x0001, 0x8001, 0x7BFF, 0xFBFF, 0x3C00, 0xBC00, 0x0400, 0x1A00, 0x2400 };+ const int gt = Y.f.gu_type;+ const size_t bb = gt == nib::kIQ2_S ? 82 : gt == nib::kIQ3_XXS ? 98 : 110; // block_iq2_s / iq3_xxs / iq3_s+ const size_t nblk = (size_t) nib::kRows * nib::kSuper;+ std::set<int> grid, sign, scale, dset;+ for (int e = 0; e < Y.ne; ++e) {+ uint8_t* b = Y.host.data() + (size_t) e * Y.f.bytes;+ for (size_t i = 0; i < 2 * Y.f.up_off; ++i) b[i] = (uint8_t) rng();+ for (size_t k = 0; k < nblk; ++k) {+ uint8_t* blk = b + k * bb;+ const uint16_t d = kD[((size_t) e * nblk + k) % (sizeof kD / sizeof kD[0])];+ std::memcpy(blk, &d, 2);+ dset.insert(d);+ const uint8_t* qs = blk + 2;+ for (int ib = 0; ib < 8; ++ib) {+ if (gt == nib::kIQ2_S) {+ const uint8_t* qh = blk + 66;+ const uint8_t* sc = blk + 74;+ for (int l = 0; l < 4; ++l) {+ grid.insert(qs[4 * ib + l] | ((qh[ib] << (8 - 2 * l)) & 0x300));+ sign.insert(qs[32 + 4 * ib + l]);+ }+ scale.insert(sc[ib] & 15);+ scale.insert(sc[ib] >> 4);+ } else if (gt == nib::kIQ3_XXS) {+ uint32_t aux;+ std::memcpy(&aux, blk + 66 + 4 * ib, 4);+ for (int l = 0; l < 8; ++l) grid.insert(qs[8 * ib + l]);+ for (int l = 0; l < 4; ++l) sign.insert((int) ((aux >> (7 * l)) & 127));+ scale.insert((int) (aux >> 28));+ } else {+ const uint8_t qh = blk[66 + ib];+ const uint8_t* sg = blk + 74;+ const uint8_t* sc = blk + 106;+ for (int l = 0; l < 4; ++l) {+ grid.insert(qs[8 * ib + 2 * l] | ((qh << (8 - 2 * l)) & 256));+ grid.insert(qs[8 * ib + 2 * l + 1] | ((qh << (7 - 2 * l)) & 256));+ sign.insert(sg[4 * ib + l]);+ }+ scale.insert((sc[ib / 2] >> (4 * (ib & 1))) & 15);+ }+ }+ }+ }+ const size_t want_grid = gt == nib::kIQ2_S ? 1024 : gt == nib::kIQ3_XXS ? 256 : 512;+ const size_t want_sign = gt == nib::kIQ3_XXS ? 128 : 256;+ const bool ok = grid.size() == want_grid && sign.size() == want_sign && scale.size() == 16 &&+ dset.size() == sizeof kD / sizeof kD[0];+ std::printf("synthetic %-7s: %d experts, coverage: grid indices %zu/%zu, sign %s %zu/%zu, scale nibbles %zu/16, d values %zu/%zu"+ " (+-min subnormal, +-65504, +-1, min normal, 2 ordinary) %s\n",+ nib::fmt_name(gt), Y.ne, grid.size(), want_grid, gt == nib::kIQ3_XXS ? "indices" : "bytes", sign.size(),+ want_sign, scale.size(), dset.size(), sizeof kD / sizeof kD[0], ok ? "complete" : "INCOMPLETE");+ return ok ? 0 : 1;+}++std::vector<int> split_ints(const std::string& s) {+ std::vector<int> v;+ size_t p = 0;+ while (p < s.size()) {+ v.push_back(std::atoi(s.c_str() + p));+ p = s.find(',', p);+ if (p == std::string::npos) break;+ ++p;+ }+ return v;+}++} // namespace++int main(int argc, char** argv) {+ setvbuf(stdout, nullptr, _IONBF, 0);+ if (argc == 2 && std::strcmp(argv[1], "--header") == 0) return header_check();+ std::vector<std::string> shards;+ int per_fmt = 2, NE = 512, plans = 8, syn = 64;+ bool ggml_check = true, negative = false, syn_only = false;+ std::vector<int> only;+ for (int i = 1; i < argc; ++i) {+ const std::string a = argv[i];+ auto next = [&]() -> const char* {+ if (i + 1 >= argc) {+ std::fprintf(stderr, "%s needs a value\n", a.c_str());+ std::exit(2);+ }+ return argv[++i];+ };+ if (a == "--layers-per-fmt") per_fmt = std::atoi(next());+ else if (a == "--layers") only = split_ints(next());+ else if (a == "--ne") NE = std::atoi(next());+ else if (a == "--plans") plans = std::atoi(next());+ else if (a == "--synthetic") syn = std::atoi(next());+ else if (a == "--no-ggml") ggml_check = false;+ else if (a == "--negative") negative = true;+ else if (a == "--synthetic-only") syn_only = true;+ else if (a.size() > 5 && a.substr(a.size() - 5) == ".gguf") shards.push_back(a);+ else {+ std::fprintf(stderr,+ "usage: iq_nibble_parity --header\n"+ " iq_nibble_parity <shard.gguf> [<shard.gguf> ...] [--layers-per-fmt 2] [--layers 1,17] [--ne 512]\n"+ " [--plans 8] [--synthetic 64] [--no-ggml] [--negative]\n"+ " iq_nibble_parity --synthetic-only [--synthetic 64] [--plans 8] [--no-ggml]\n");+ return 2;+ }+ }+ if ((shards.empty() && !syn_only) || NE < 16 || NE > 512) {+ std::fprintf(stderr, "iq_nibble_parity: give at least one model shard (.gguf); --ne 16..512\n");+ return 2;+ }+ CK(cudaSetDevice(0));+ cudaDeviceProp prop{};+ CK(cudaGetDeviceProperties(&prop, 0));+ char bus[64] = {};+ CK(cudaDeviceGetPCIBusId(bus, sizeof bus, 0));+ std::printf("device 0: %s, bus %s, cc %d.%d, %d SMs\n", prop.name, bus, prop.major, prop.minor, prop.multiProcessorCount);+ if (prop.major != 6 || prop.minor != 0) {+ std::fprintf(stderr, "iq_nibble_parity: the 4-bit kernels are the sm_60 build's; this GPU is cc %d.%d (skipped)\n",+ prop.major, prop.minor);+ return 77;+ }+ if (!STRATA_EXPERT_NIBBLE_BUILT || nib::prepare(nullptr) != 0) {+ std::fprintf(stderr, "iq_nibble_parity: no 4-bit kernels in this build, or the GGUF codebooks are not unique\n");+ return 2;+ }+ std::mt19937 rng(20261019);+ void* scr = nullptr;+ float* dout = nullptr;+ void* dx = nullptr;+ const int64_t capmax = (int64_t) kTok * 10;+ CK(cudaMalloc(&scr, K::native_expert_scratch_bytes(capmax, kFf)));+ CK(cudaMalloc((void**) &dout, (size_t) capmax * kEmbd * 4));+ const std::vector<Q81> hx = make_x(rng);+ CK(cudaMalloc(&dx, hx.size() * sizeof(Q81)));+ CK(cudaMemcpy(dx, hx.data(), hx.size() * sizeof(Q81), cudaMemcpyHostToDevice));++ if (syn_only) { // no model: every gate/up format with each down type, random experts+ long long total_s = 0;+ int done = 0;+ for (const int gt : {nib::kIQ2_S, nib::kIQ3_XXS, nib::kIQ3_S})+ for (const int dt : {20, 42}) { // the packs' down types: IQ4_NL, Q2_0+ Layer S;+ std::string err;+ if (!cpu::native_fmt(gt, dt, kEmbd, kFf, S.f, err)) {+ std::printf("%s / down type %d: %s\n", nib::fmt_name(gt), dt, err.c_str());+ ++total_s;+ continue;+ }+ S.tag = "syn";+ S.layer = 0;+ S.ne = std::max(16, std::min(syn, NE));+ S.host.resize((size_t) S.ne * S.f.bytes);+ for (auto& b : S.host) b = (uint8_t) rng();+ // block_iq4_nl and block_q2_0 are 18 bytes, the fp16 scale first: an ordinary scale in every block of the down rows+ for (int e = 0; e < S.ne; ++e)+ for (size_t o = S.f.down_off; o + 18 <= S.f.bytes; o += 18) {+ const uint16_t sc = (uint16_t) ((9u << 10) | (rng() & 0x3FFu));+ std::memcpy(S.host.data() + (size_t) e * S.f.bytes + o, &sc, 2);+ }+ long long sb = synthesize(S, rng);+ sb += convert(S, ggml_check);+ sb += grouped_parity(S, rng, plans, dx, scr, dout);+ total_s += sb;+ ++done;+ CK(cudaFree(S.raw));+ CK(cudaFree(S.slots));+ }+ std::printf("\n%s: %d synthetic format pairs, %lld differences\n", total_s == 0 && done == 6 ? "PASS" : "FAIL", done, total_s);+ return total_s == 0 && done == 6 ? 0 : 1;+ }+ long long total = 0;+ int layers_done = 0;+ std::vector<int> syn_done;+ for (size_t si = 0; si < shards.size(); ++si) {+ std::unique_ptr<strata::GgufModel> model;+ try {+ model = std::make_unique<strata::GgufModel>(strata::GgufModel::open(shards[si]));+ } catch (const std::exception& ex) {+ std::fprintf(stderr, "iq_nibble_parity: %s\n", ex.what());+ return 2;+ }+ const std::string tag(1, (char) ('A' + si));+ std::printf("\n#### model %s: %s (%zu shards)\n", tag.c_str(), shards[si].c_str(), model->size());+ std::vector<int> pick;+ int count[64] = {};+ for (int l = 0; l < 256; ++l) {+ int gt = -1, dt = -1;+ if (!layer_fmt(*model, l, gt, dt)) break;+ if (!nib::fmt_built(gt)) continue;+ const bool want = only.empty() ? count[gt]++ < per_fmt : std::find(only.begin(), only.end(), l) != only.end();+ if (want) pick.push_back(l);+ }+ for (int l : pick) {+ Layer Y;+ Y.tag = tag;+ if (!read_layer(*model, l, NE, Y)) {+ std::printf("layer %d: cannot read its experts\n", l);+ ++total;+ continue;+ }+ long long b = convert(Y, ggml_check);+ if (negative) { // the control: one code byte changed per slot (item 3 of row 5), which (6) must see+ for (int e = 0; e < Y.ne; ++e) {+ uint8_t* at = Y.slots + (size_t) e * Y.V.slot_bytes + nib::code_off(5, 3);+ uint8_t v = 0;+ CK(cudaMemcpy(&v, at, 1, cudaMemcpyDeviceToHost));+ v ^= 0x11;+ CK(cudaMemcpy(at, &v, 1, cudaMemcpyHostToDevice));+ }+ std::printf(" NEGATIVE CONTROL: one code byte changed per slot - the rows below must differ\n");+ std::vector<unsigned long long> hp(Y.ne);+ for (int e = 0; e < Y.ne; ++e) hp[e] = (unsigned long long) (Y.slots + (size_t) e * Y.V.slot_bytes);+ unsigned long long* dp = nullptr;+ CK(cudaMalloc(&dp, (size_t) Y.ne * 8));+ CK(cudaMemcpy(dp, hp.data(), (size_t) Y.ne * 8, cudaMemcpyHostToDevice));+ b += gguf_parity(Y, dp, true);+ CK(cudaFree(dp));+ }+ b += grouped_parity(Y, rng, plans, dx, scr, dout);+ total += b;+ ++layers_done;+ // the synthetic experts of this format, once: this layer's blobs with random gate/up blocks+ if (syn > 0 && std::find(syn_done.begin(), syn_done.end(), Y.f.gu_type) == syn_done.end() && !negative) {+ syn_done.push_back(Y.f.gu_type);+ Layer S;+ S.tag = "syn";+ S.layer = Y.layer;+ S.f = Y.f;+ S.ne = std::min(syn, Y.ne);+ S.host.assign(Y.host.begin(), Y.host.begin() + (ptrdiff_t) ((size_t) S.ne * Y.f.bytes));+ long long sb = synthesize(S, rng);+ sb += convert(S, ggml_check);+ sb += grouped_parity(S, rng, std::max(1, plans / 2), dx, scr, dout);+ total += sb;+ CK(cudaFree(S.raw));+ CK(cudaFree(S.slots));+ }+ CK(cudaFree(Y.raw));+ CK(cudaFree(Y.slots));+ }+ }+ std::printf("\n%s: %d layers (+ %zu synthetic formats), %lld differences\n", total == 0 ? "PASS" : "FAIL", layers_done,+ syn_done.size(), total);+ return total == 0 ? 0 : 1;+}diff --git a/src/kernels/native_expert_bench.cpp b/src/kernels/native_expert_bench.cppindex 3263ee7ae..e51bf7742 100644--- a/src/kernels/native_expert_bench.cpp+++ b/src/kernels/native_expert_bench.cpp@@ -2,11 +2,21 @@// another, bit for bit, and each one's time.//// build/native_expert_bench <shard1.gguf> <layer[,layer...]> <groups> <tokens per group> <ref mode> <mode> [iters]+// build/native_expert_bench --nibble <shard.gguf> <layer[,layer...]> [--t 1,2,3,4,5,6] [--share 0,50] [--plans 200]+// [--trials 7] [--mode -1|0] [--variant -1|0|1]//// Modes are STRATA_EXP_MODE values. A group is one expert (distinct blobs, ~2 MB each, so G >= 8 does not fit// in L2); its entries read distinct tokens of an 8-token window. The output must be bitwise equal to the// reference mode's: the verify window's text depends on it.+//+// --nibble (STRATA_EXPERT_NIBBLE, strata/core/expert_nibble.hpp): the layer's 512 experts as GGUF blobs (P, the+// production kernels) and as 4-bit slots made by the engine's transcoder (N: native_expert_layout_nib); the verify+// window's whole expert step (gate/up + SwiGLU/q8_1 + down) and its gate/up phase alone, per (T, sharing) over+// `--plans` top-10 routings (0% or 50% of a token's experts shared with the previous token's), every plan's out rows+// compared byte for byte first; interleaved trials, medians. `--mode`: both arms' layout (-1 the card's default, a+// GP100's mode 8; 0 the CUDA layout); `--variant`: the 4-bit gate/up kernel (-1 per format, 0 S, 1 P).#include "strata/artifact/gguf_reader.hpp"+#include "strata/core/expert_nibble.hpp"#include "strata/kernels/cpu/native_expert.hpp"#include "strata/kernels/iq_kernels.hpp"@@ -20,6 +30,8 @@#include <cstdio>#include <cstdlib>#include <cstring>+#include <map>+#include <memory>#include <random>#include <sstream>#include <string>@@ -28,8 +40,11 @@namespace cpu = strata::kernels::cpu;namespace K = strata::kernels;+int nibble_main(int argc, char** argv);+int main(int argc, char** argv) {setvbuf(stdout, nullptr, _IONBF, 0);+ if (argc >= 2 && std::strcmp(argv[1], "--nibble") == 0) return nibble_main(argc, argv);if (argc < 7) {std::fprintf(stderr, "usage: native_expert_bench <shard1.gguf> <layer[,layer]> <groups> <tokens> <ref mode> <mode> [iters]\n");return 2;@@ -175,3 +190,407 @@ int main(int argc, char** argv) {std::printf("native_expert_bench: %d differing\n", failures);return failures ? 1 : 0;}++// ------------------------------------------------------------------------------------------------ --nibble+#if STRATA_EXPERT_NIBBLE_BUILT+namespace {++namespace nib = strata::core::nibble;++#define NB_CK(x) \+ do { \+ cudaError_t e_ = (x); \+ if (e_ != cudaSuccess) { \+ std::fprintf(stderr, "%s:%d %s: %s\n", __FILE__, __LINE__, #x, cudaGetErrorString(e_)); \+ std::exit(2); \+ } \+ } while (0)++struct NbPlan {+ int ng = 0, ne = 0, t = 0;+ std::vector<int> gid, start, tok, dst;+};++// synthetic routing: t tokens of top-10, `share` % of each token's experts the previous token's; groups by first+// appearance (resident_plan_kernel's order), entries in token order+NbPlan nb_route(std::mt19937& rng, int t, int share) {+ std::vector<int> ids, prev, cur;+ for (int k = 0; k < t; ++k) {+ cur.clear();+ const bool sh = k > 0 && share > 0;+ if (sh) {+ std::vector<int> p = prev;+ std::shuffle(p.begin(), p.end(), rng);+ cur.assign(p.begin(), p.begin() + 10 * share / 100);+ }+ while ((int) cur.size() < 10) {+ const int e = (int) (rng() % 512u);+ if (std::find(cur.begin(), cur.end(), e) != cur.end()) continue;+ if (sh && std::find(prev.begin(), prev.end(), e) != prev.end()) continue;+ cur.push_back(e);+ }+ std::shuffle(cur.begin(), cur.end(), rng);+ ids.insert(ids.end(), cur.begin(), cur.end());+ prev = cur;+ }+ NbPlan p;+ p.t = t;+ std::map<int, std::vector<int>> by; // expert -> its entries (j = token * 10 + k)+ std::vector<int> first;+ for (int j = 0; j < (int) ids.size(); ++j) {+ if (by.find(ids[j]) == by.end()) first.push_back(ids[j]);+ by[ids[j]].push_back(j);+ }+ for (int e : first) {+ p.gid.push_back(e);+ p.start.push_back(p.ne);+ for (int j : by[e]) {+ p.tok.push_back(j / 10);+ p.dst.push_back(j);+ ++p.ne;+ }+ }+ p.start.push_back(p.ne);+ p.ng = (int) p.gid.size();+ return p;+}++struct NbDev {+ unsigned long long* ptr = nullptr;+ int32_t *start = nullptr, *ng = nullptr, *tok = nullptr, *dst = nullptr;+};+NbDev nb_upload(const NbPlan& p, const uint8_t* base, size_t stride) {+ NbDev d;+ std::vector<unsigned long long> hp(std::max(1, p.ng));+ for (int g = 0; g < p.ng; ++g) hp[g] = (unsigned long long) (base + (size_t) p.gid[g] * stride);+ NB_CK(cudaMalloc((void**) &d.ptr, hp.size() * 8));+ NB_CK(cudaMalloc((void**) &d.start, (size_t) (p.ng + 1) * 4));+ NB_CK(cudaMalloc((void**) &d.ng, 4));+ NB_CK(cudaMalloc((void**) &d.tok, (size_t) std::max(1, p.ne) * 4));+ NB_CK(cudaMalloc((void**) &d.dst, (size_t) std::max(1, p.ne) * 4));+ NB_CK(cudaMemcpy(d.ptr, hp.data(), hp.size() * 8, cudaMemcpyHostToDevice));+ NB_CK(cudaMemcpy(d.start, p.start.data(), (size_t) (p.ng + 1) * 4, cudaMemcpyHostToDevice));+ NB_CK(cudaMemcpy(d.ng, &p.ng, 4, cudaMemcpyHostToDevice));+ if (p.ne) {+ NB_CK(cudaMemcpy(d.tok, p.tok.data(), (size_t) p.ne * 4, cudaMemcpyHostToDevice));+ NB_CK(cudaMemcpy(d.dst, p.dst.data(), (size_t) p.ne * 4, cudaMemcpyHostToDevice));+ }+ return d;+}+void nb_free(NbDev& d) {+ cudaFree(d.ptr);+ cudaFree(d.start);+ cudaFree(d.ng);+ cudaFree(d.tok);+ cudaFree(d.dst);+}++std::vector<int> nb_ints(const std::string& s) {+ std::vector<int> v;+ std::stringstream ss(s);+ std::string t;+ while (std::getline(ss, t, ',')) v.push_back(std::atoi(t.c_str()));+ return v;+}++} // namespace++int nibble_main(int argc, char** argv) {+ if (argc < 4) {+ std::fprintf(stderr, "usage: native_expert_bench --nibble <shard.gguf> <layer[,layer]> [--t 1,2,3,4,5,6] [--share 0,50] "+ "[--plans 200] [--trials 7] [--mode -1|0] [--variant -1|0|1]\n");+ return 2;+ }+ const std::string shard = argv[2];+ const std::vector<int> layers = nb_ints(argv[3]);+ std::vector<int> Ts = { 1, 2, 3, 4, 5, 6 }, shares = { 0, 50 };+ int plans = 200, trials = 7, mode = -1, variant = -1;+ std::vector<int> vs; // gu-proto: the nibble gate/up kernels compared with the production one (-1), gate/up only+ for (int i = 4; i + 1 < argc; i += 2) {+ const std::string a = argv[i];+ if (a == "--t") Ts = nb_ints(argv[i + 1]);+ else if (a == "--share") shares = nb_ints(argv[i + 1]);+ else if (a == "--plans") plans = std::atoi(argv[i + 1]);+ else if (a == "--trials") trials = std::atoi(argv[i + 1]);+ else if (a == "--mode") mode = std::atoi(argv[i + 1]);+ else if (a == "--variant") variant = std::atoi(argv[i + 1]);+ else if (a == "--vs") vs = nb_ints(argv[i + 1]);+ else {+ std::fprintf(stderr, "unknown option %s\n", a.c_str());+ return 2;+ }+ }+ std::unique_ptr<strata::GgufModel> model;+ try {+ model = std::make_unique<strata::GgufModel>(strata::GgufModel::open(shard));+ } catch (const std::exception& ex) {+ std::fprintf(stderr, "native_expert_bench: %s\n", ex.what());+ return 2;+ }+ cudaDeviceProp prop{};+ NB_CK(cudaGetDeviceProperties(&prop, 0));+ char bus[64] = {};+ NB_CK(cudaDeviceGetPCIBusId(bus, sizeof bus, 0));+ std::printf("device 0: %s, bus %s, cc %d.%d; layout %s, 4-bit kernel %s\n", prop.name, bus, prop.major, prop.minor,+ mode < 0 ? "the card's default" : ("mode " + std::to_string(mode)).c_str(),+ variant < 0 ? "per format" : variant == 1 ? "P" : "S");+ K::native_expert_nib_set_variant(variant);+ const int64_t H = nib::kNEmbd, FF = nib::kNFf;+ constexpr int kTokMax = 8;+ int failures = 0;+ cudaStream_t s;+ NB_CK(cudaStreamCreate(&s));+ cudaEvent_t e0, e1;+ NB_CK(cudaEventCreate(&e0));+ NB_CK(cudaEventCreate(&e1));+ // the activations: kTokMax tokens of normal floats, quantized as the verify window quantizes them+ std::mt19937 rng(20261019);+ std::normal_distribution<float> nd(0.f, 1.f);+ std::vector<float> x((size_t) kTokMax * H);+ for (auto& v : x) v = nd(rng);+ void *dx = nullptr, *dxq = nullptr, *dscr = nullptr;+ float* dout = nullptr;+ const int64_t cap = (int64_t) kTokMax * 10;+ NB_CK(cudaMalloc(&dx, x.size() * 4));+ NB_CK(cudaMalloc(&dxq, (size_t) kTokMax * H / 32 * 36));+ NB_CK(cudaMalloc(&dscr, K::native_expert_scratch_bytes(cap, FF)));+ NB_CK(cudaMalloc((void**) &dout, (size_t) cap * H * 4));+ NB_CK(cudaMemcpy(dx, x.data(), x.size() * 4, cudaMemcpyHostToDevice));+ K::quantize_q8_1_rows((const float*) dx, kTokMax, H, dxq, s);+ for (int l : layers) {+ const char* roles[3] = { "gate", "up", "down" };+ const strata::TensorInfo* t[3] = {};+ size_t sh[3] = {};+ for (int r = 0; r < 3; ++r) t[r] = model->find("blk." + std::to_string(l) + ".ffn_" + roles[r] + "_exps.weight", &sh[r]);+ if (!t[0] || !t[1] || !t[2]) {+ std::printf("layer %d: no expert tensors\n", l);+ ++failures;+ continue;+ }+ cpu::NativeFmt f;+ std::string err;+ if (!cpu::native_fmt((int) t[0]->type, (int) t[2]->type, H, FF, f, err)) {+ std::printf("layer %d: %s\n", l, err.c_str());+ ++failures;+ continue;+ }+ nib::Config c;+ c.on = true;+ const nib::LayerVram V = nib::plan_layer(l, f, c);+ if (!V.on) {+ std::printf("layer %d: %s is not converted\n", l, ggml_type_name((ggml_type) f.gu_type));+ continue;+ }+ // the 512 blobs (P) and their 4-bit slots (N, the engine's transcoder)+ std::vector<uint8_t> host((size_t) 512 * f.bytes);+ const size_t dn_role = f.bytes - f.down_off;+ for (int e = 0; e < 512; ++e) {+ uint8_t* b = host.data() + (size_t) e * f.bytes;+ std::memcpy(b, model->shard(sh[0]).tensor_data(*t[0]) + (size_t) e * f.up_off, f.up_off);+ std::memcpy(b + f.up_off, model->shard(sh[1]).tensor_data(*t[1]) + (size_t) e * f.up_off, f.up_off);+ std::memcpy(b + f.down_off, model->shard(sh[2]).tensor_data(*t[2]) + (size_t) e * dn_role, dn_role);+ }+ uint8_t *raw = nullptr, *slots = nullptr;+ unsigned long long *dp = nullptr, *dbad = nullptr;+ NB_CK(cudaMalloc((void**) &raw, host.size()));+ NB_CK(cudaMalloc((void**) &slots, (size_t) 512 * V.slot_bytes));+ NB_CK(cudaMalloc((void**) &dp, 512 * 8));+ NB_CK(cudaMalloc((void**) &dbad, 8));+ NB_CK(cudaMemcpy(raw, host.data(), host.size(), cudaMemcpyHostToDevice));+ std::vector<unsigned long long> hp(512);+ for (int e = 0; e < 512; ++e) hp[e] = (unsigned long long) (slots + (size_t) e * V.slot_bytes);+ NB_CK(cudaMemcpy(dp, hp.data(), 512 * 8, cudaMemcpyHostToDevice));+ NB_CK(cudaMemset(dbad, 0, 8));+ for (int b0 = 0; b0 < 512; b0 += 64)+ if (nib::transcode(V, raw + (size_t) b0 * V.raw_bytes, 64, dp + b0, dbad, s) != 0) return 2;+ for (int e = 0; e < 512; ++e)+ NB_CK(cudaMemcpyAsync(slots + (size_t) e * V.slot_bytes + nib::kDownOff, raw + (size_t) e * V.raw_bytes + V.raw_down_off,+ V.down_bytes, cudaMemcpyDeviceToDevice, s));+ unsigned long long bad = 0;+ NB_CK(cudaMemcpyAsync(&bad, dbad, 8, cudaMemcpyDeviceToHost, s));+ NB_CK(cudaStreamSynchronize(s));+ if (bad) {+ std::printf("layer %d: %llu items do not round-trip\n", l, bad);+ ++failures;+ continue;+ }+ const auto Lr = K::native_expert_layout(f.gu_type, f.d_type, H, FF);+ const auto Ln = K::native_expert_layout_nib(f.gu_type, f.d_type, H, FF);+ std::printf("layer %2d %-7s/%-6s: 512 experts, blob %zu B, 4-bit slot %llu B\n", l, nib::fmt_name(f.gu_type),+ ggml_type_name((ggml_type) f.d_type), f.bytes, (unsigned long long) V.slot_bytes);+ // one set of plans: every plan's out rows compared first, then P and N timed interleaved, phase 0 (the whole+ // step) and 1 (gate/up + SwiGLU/q8_1), us per launch (median of the trials)+ auto bench = [&](const std::vector<NbPlan>& ps, int tt, double us[2][2]) -> long long {+ std::vector<NbDev> dr(ps.size()), dn(ps.size());+ for (size_t i = 0; i < ps.size(); ++i) {+ dr[i] = nb_upload(ps[i], raw, f.bytes);+ dn[i] = nb_upload(ps[i], slots, V.slot_bytes);+ }+ const int64_t cp = std::max<int64_t>((int64_t) tt * 10, 1);+ auto launch = [&](int arm, size_t i) {+ const NbDev& d = arm ? dn[i] : dr[i];+ K::native_expert_grouped(arm ? Ln : Lr, d.ptr, d.start, d.ng, d.dst, d.tok, std::max<int64_t>(cp, ps[i].ng),+ std::max<int64_t>(cp, ps[i].ne), dxq, dscr, dout, s, 0);+ };+ long long diff = 0;+ std::vector<uint8_t> a((size_t) cap * H * 4), b(a.size());+ K::native_expert_set_mode(mode, 0);+ for (size_t i = 0; i < ps.size(); ++i) {+ const size_t n = (size_t) ps[i].ne * H * 4;+ NB_CK(cudaMemsetAsync(dout, 0xFF, a.size(), s));+ launch(0, i);+ NB_CK(cudaMemcpyAsync(a.data(), dout, n, cudaMemcpyDeviceToHost, s));+ NB_CK(cudaMemsetAsync(dout, 0x00, a.size(), s));+ launch(1, i);+ NB_CK(cudaMemcpyAsync(b.data(), dout, n, cudaMemcpyDeviceToHost, s));+ NB_CK(cudaStreamSynchronize(s));+ diff += std::memcmp(a.data(), b.data(), n) != 0;+ }+ for (int ph = 0; ph < 2; ++ph) {+ K::native_expert_set_mode(mode, ph);+ std::vector<float> tm[2];+ for (int arm = 0; arm < 2; ++arm)+ for (size_t i = 0; i < std::min<size_t>(ps.size(), 20); ++i) launch(arm, i);+ for (int tr = 0; tr < trials; ++tr)+ for (int arm = 0; arm < 2; ++arm) {+ NB_CK(cudaEventRecord(e0, s));+ for (size_t i = 0; i < ps.size(); ++i) launch(arm, i);+ NB_CK(cudaEventRecord(e1, s));+ NB_CK(cudaEventSynchronize(e1));+ float ms = 0;+ NB_CK(cudaEventElapsedTime(&ms, e0, e1));+ tm[arm].push_back(1e3f * ms / (float) ps.size());+ }+ for (int arm = 0; arm < 2; ++arm) {+ std::sort(tm[arm].begin(), tm[arm].end());+ us[ph][arm] = tm[arm][tm[arm].size() / 2];+ }+ }+ K::native_expert_set_mode(mode, 0);+ for (size_t i = 0; i < ps.size(); ++i) {+ nb_free(dr[i]);+ nb_free(dn[i]);+ }+ return diff;+ };+ if (!vs.empty()) { // gu-proto: nibble production (-1) against each id in --vs, gate/up only (phase 3)+ std::vector<int> arms = { -1 };+ arms.insert(arms.end(), vs.begin(), vs.end());+ for (int v : vs) {+ int regs = 0, loc = 0, occ = 0;+ if (K::native_expert_nib_variant_attr(v, f.gu_type, ®s, &loc, &occ))+ std::printf(" variant %d: %d registers, %d B local, %d blocks/SM\n", v, regs, loc, occ);+ }+ const size_t fsz = (size_t) cap * FF * 4, fa = (fsz + 255) & ~(size_t) 255;+ for (int tt : Ts) {+ if (tt < 1 || tt > kTokMax) continue;+ for (int share : shares) {+ std::vector<NbPlan> ps;+ double G = 0;+ for (int i = 0; i < plans; ++i) {+ ps.push_back(nb_route(rng, tt, share));+ G += ps.back().ng;+ }+ std::vector<NbDev> dn(ps.size());+ for (size_t i = 0; i < ps.size(); ++i) dn[i] = nb_upload(ps[i], slots, V.slot_bytes);+ const int64_t cp = std::max<int64_t>((int64_t) tt * 10, 1);+ auto launch = [&](int v, size_t i) {+ K::native_expert_nib_set_variant(v);+ K::native_expert_grouped(Ln, dn[i].ptr, dn[i].start, dn[i].ng, dn[i].dst, dn[i].tok,+ std::max<int64_t>(cp, ps[i].ng), std::max<int64_t>(cp, ps[i].ne), dxq, dscr,+ dout, s, 0);+ };+ K::native_expert_set_mode(mode, 3);+ // every plan: each arm's gate and up rows (the plan's entries) byte for byte the production arm's+ std::vector<long long> drows(arms.size(), 0);+ std::vector<uint8_t> ref(2 * fsz), got(2 * fsz);+ for (size_t i = 0; i < ps.size(); ++i) {+ const size_t n = (size_t) ps[i].ne * FF * 4;+ // native_expert_grouped's up rows start after cap_entries gate rows (256-aligned)+ const size_t fi = (size_t) std::max<int64_t>(cp, ps[i].ne) * FF * 4, fai = (fi + 255) & ~(size_t) 255;+ for (size_t a = 0; a < arms.size(); ++a) {+ NB_CK(cudaMemsetAsync(dscr, a == 0 ? 0xFF : 0x00, 2 * fa, s));+ launch(arms[a], i);+ std::vector<uint8_t>& o = a == 0 ? ref : got;+ NB_CK(cudaMemcpyAsync(o.data(), dscr, n, cudaMemcpyDeviceToHost, s));+ NB_CK(cudaMemcpyAsync(o.data() + fsz, (const uint8_t*) dscr + fai, n, cudaMemcpyDeviceToHost, s));+ NB_CK(cudaStreamSynchronize(s));+ if (a == 0) continue;+ for (size_t r = 0; r < (size_t) ps[i].ne; ++r) {+ drows[a] += std::memcmp(ref.data() + r * FF * 4, got.data() + r * FF * 4, FF * 4) != 0;+ drows[a] += std::memcmp(ref.data() + fsz + r * FF * 4, got.data() + fsz + r * FF * 4, FF * 4) != 0;+ }+ }+ }+ long long rows = 0;+ for (const NbPlan& p : ps) rows += 2LL * p.ne;+ std::vector<std::vector<float>> tm(arms.size());+ for (size_t a = 0; a < arms.size(); ++a)+ for (size_t i = 0; i < std::min<size_t>(ps.size(), 20); ++i) launch(arms[a], i);+ for (int tr = 0; tr < trials; ++tr)+ for (size_t a = 0; a < arms.size(); ++a) {+ NB_CK(cudaEventRecord(e0, s));+ for (size_t i = 0; i < ps.size(); ++i) launch(arms[a], i);+ NB_CK(cudaEventRecord(e1, s));+ NB_CK(cudaEventSynchronize(e1));+ float ms = 0;+ NB_CK(cudaEventElapsedTime(&ms, e0, e1));+ tm[a].push_back(1e3f * ms / (float) ps.size());+ }+ K::native_expert_set_mode(mode, 0);+ K::native_expert_nib_set_variant(variant);+ std::printf(" T=%d share %2d%% (G %.2f, %lld gate/up rows a plan set), gate/up us per launch (median, min..max of %d):\n",+ tt, share, G / plans, rows, trials);+ double base = 0;+ for (size_t a = 0; a < arms.size(); ++a) {+ std::sort(tm[a].begin(), tm[a].end());+ const double med = tm[a][tm[a].size() / 2];+ if (a == 0) base = med;+ std::printf(" %4d: %8.2f (%7.2f..%7.2f) x%.3f %s\n", arms[a], med, tm[a].front(), tm[a].back(),+ base / med, a == 0 ? "reference" : drows[a] ? ("rows differ: " + std::to_string(drows[a])).c_str() : "bitwise equal");+ }+ for (size_t i = 0; i < ps.size(); ++i) nb_free(dn[i]);+ }+ }+ cudaFree(raw);+ cudaFree(slots);+ cudaFree(dp);+ cudaFree(dbad);+ continue;+ }+ for (int tt : Ts) {+ if (tt < 1 || tt > kTokMax) continue;+ for (int share : shares) {+ std::vector<NbPlan> ps;+ double G = 0;+ for (int i = 0; i < plans; ++i) {+ ps.push_back(nb_route(rng, tt, share));+ G += ps.back().ng;+ }+ double us[2][2];+ const long long diff = bench(ps, tt, us);+ std::printf(" T=%d share %2d%% (G %.2f): step P %7.2f N %7.2f us x%.3f | gate/up+swiglu P %7.2f N %7.2f us x%.3f | %s\n",+ tt, share, G / plans, us[0][0], us[0][1], us[0][0] / us[0][1], us[1][0], us[1][1],+ us[1][0] / us[1][1], diff ? "DIFFERS" : "bitwise equal");+ if (diff) {+ std::printf(" %lld of %d plans differ\n", diff, plans);+ ++failures;+ }+ }+ }+ cudaFree(raw);+ cudaFree(slots);+ cudaFree(dp);+ cudaFree(dbad);+ }+ K::native_expert_set_mode(-1, 0);+ K::native_expert_nib_set_variant(-1);+ std::printf("native_expert_bench --nibble: %d failing\n", failures);+ return failures ? 1 : 0;+}+#else+int nibble_main(int, char**) {+ std::fprintf(stderr, "native_expert_bench --nibble: no 4-bit kernels in this build (STRATA_EXPERIMENTAL_SM60, CUDA)\n");+ return 2;+}+#endifdiff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cppindex 8a2eb6a30..2e628967a 100644--- a/src/prefill/prefill.cpp+++ b/src/prefill/prefill.cpp@@ -47,6 +47,7 @@#include "strata/prefill/moe_fused_iq.hpp"#include "strata/prefill/moe_mmq.hpp"#include "strata/core/peer_experts.hpp"+#include "strata/core/expert_nibble.hpp"#include "strata/prefill/kernels.hpp"#include "strata/prefill/kva.hpp" // STRATA_KVA: the optional approximate-prefill plan (research, off by default)@@ -2467,6 +2468,31 @@ void Prefill::arm_cpu_share(bool applies, bool by_default) {bool Prefill::ring_bytes_enabled() { return ring_bytes_on(); }size_t Prefill::slot_guard_bytes() { return mmq::direct_slot_guard(); }+bool Prefill::nibble_prompt_check(std::string& why) {+ const auto& lay = strata::kernels::cpu::expert_layout();+ if (mmq::direct_mode() != 0) {+ why = "STRATA_MMQ_DIRECT (the prompt MMQ would read the 4-bit slots in place)";+ return false;+ }+ for (int64_t l = 0; lay.native && l < (int64_t) lay.fmt.size(); ++l) {+ if (!core::nibble::layer(l).on) continue;+ const auto& f = lay.fmt[(size_t) l];+ const char* w = !mmq_plan().any || (size_t) l >= mmq_plan().layer.size() || !mmq_plan().layer[(size_t) l]+ ? "no prompt MMQ for it (the FP16 path dequantizes GGUF blocks)"+ : mmq_plan().wmma_only[(size_t) l] ? "the WMMA tiles read GGUF blocks"+ : fused::native_supported(f.gu_type, f.d_type) ? "the fused prompt kernels read GGUF blocks"+ : mmq::matrix_bytes(f.gu_type, 1280, N) != (size_t) core::nibble::kRows * f.gu_row ||+ mmq::matrix_bytes(f.d_type, N, 640) != f.bytes - f.down_off+ ? "MMQ's matrices are not the blob's [gate | up] and down"+ : nullptr;+ if (w != nullptr) {+ why = "layer " + std::to_string(l) + ": " + w;+ return false;+ }+ }+ return true;+}+namespace {const core::WeightRef* need(const core::LayerView& v, const char* suffix, std::string& err) {@@ -3881,6 +3907,12 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::striconst bool had_gu = lay.native && lay.fmt[(size_t) l].had_gu;const bool had_d = lay.native && lay.fmt[(size_t) l].had_d;const bool use_mmq = mmq_plan().any && mmq_plan().layer[(size_t) l];+ // STRATA_EXPERT_NIBBLE=1: a layer whose gate/up experts are 4-bit codes in their slots is gathered+ // for MMQ as the GGUF blocks rebuilt from them (core::nibble::gather_gguf); every other expert path+ // reads GGUF blocks in the slot, so it stops there instead (never a wrong answer)+#if STRATA_EXPERT_NIBBLE_BUILT // every guard of this switch is compiled out of the other builds+ const core::nibble::LayerVram& nib_l = core::nibble::layer(l);+#endifconst int had_block = lay.native ? lay.fmt[(size_t) l].had_block : 0;const float* had_sx = had_gu ? m.had_sx[(size_t) l] : nullptr; // resolved in initconst float* had_sh = had_d ? m.had_sh[(size_t) l] : nullptr;@@ -3918,6 +3950,9 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stribool cpu_arm = true; // auto: this layer may share (false: the arm without)int cpu_set = -1; // auto: the wall events' set, -1 when not measuredbool cpu_cold = false; // auto: one-time costs in the window (no reading)+#if STRATA_EXPERT_NIBBLE_BUILT+ if (fused_l && nib_l.on) core::nibble::fatal(l, "the prompt path's fused experts", "they read GGUF blocks");+#endifif (fused_l) {if (static bool said = false; !said) {said = true;@@ -4823,6 +4858,13 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stricudaStreamWaitEvent(m.cs, m.copied[gg_slots[gg_nslots - 1]], 0);pt.mark(kPfDequant, cs);}+#if STRATA_EXPERT_NIBBLE_BUILT+ if (nib_l.on) { // the GGUF blocks rebuilt from the 4-bit slots: the bytes the copy gives+ if (const int rc = core::nibble::gather_gguf(nib_l, gg.blob, gg.first, gg.n, m.grp_gu, mmq_gub,+ m.grp_d, mmq_db, zero, m.cs); rc != 0)+ core::nibble::fatal(l, "the prompt path's group gather", cudaGetErrorString((cudaError_t) rc));+ } else+#endif// NVFP4: each expert's scales go next to its group slot in the same launchif (!mmq::gather_native_group(gg, f.up_off, mmq_gub / 2, f.down_off, mmq_db, m.grp_gu, mmq_gub,m.grp_d, mmq_db, m.cs, f.tail_off,@@ -4848,6 +4890,17 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri// a ring slot or the peer's), which STRATA_MMQ_DIRECT reads in placeauto compute = [&](size_t j, const uint8_t* blob_dev, int slot, int32_t cslot) -> bool {const int32_t e = order[j];+#if STRATA_EXPERT_NIBBLE_BUILT+ if (nib_l.on && (!use_mmq || direct || wmma_l || cslot < 0 || !lay.native ||+ mmq_gub != (size_t) core::nibble::kRows * nib_l.raw_gu_row ||+ mmq_db != nib_l.down_bytes))+ core::nibble::fatal(l, "the prompt path",+ !use_mmq ? "no MMQ for the layer (the FP16 path dequantizes GGUF blocks)"+ : direct ? "STRATA_MMQ_DIRECT reads the slots in place"+ : wmma_l ? "the WMMA tiles read GGUF blocks"+ : cslot < 0 ? "an expert not in this GPU's slots (streamed or the peer's)"+ : "the MMQ group buffers are not the blob's [gate | up] and down");+#endifpt.mark(kPfDequant, cs);if (use_mmq) {// gather the expert into its group slot (GGUF blocks, unchanged or converted)@@ -4894,6 +4947,16 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::strigg = mmq::GatherGroup{};} else if (lay.native) {const auto& f = lay.fmt[(size_t) l];+#if STRATA_EXPERT_NIBBLE_BUILT+ if (nib_l.on) { // as the group gather (flush), one expert at its group position q+ const uint8_t* one[mmq::kGatherGroupMax] = {};+ one[q] = blob_dev;+ if (const int rc = core::nibble::gather_gguf(nib_l, one, (int) q, (int) q + 1, m.grp_gu,+ mmq_gub, m.grp_d, mmq_db,+ last_of_group ? MMQ_TAIL : 0, m.cs); rc != 0)+ core::nibble::fatal(l, "the prompt path's gather", cudaGetErrorString((cudaError_t) rc));+ } else+#endif// NVFP4: this expert's scales go next to its group slot in the same launchmmq::gather_native(blob_dev, blob_dev + f.up_off, mmq_gub / 2, blob_dev + f.down_off,mmq_db, m.grp_gu + q * mmq_gub, m.grp_d + q * mmq_db, m.cs,diff --git a/src/program/generate.cpp b/src/program/generate.cppindex 75d404ac9..d0de8560c 100644--- a/src/program/generate.cpp+++ b/src/program/generate.cpp@@ -25,6 +25,7 @@#include "strata/core/device.hpp"#include "strata/core/remote_expert_opt.hpp"#include "strata/core/expert_cache.hpp"+#include "strata/core/expert_nibble.hpp"#include "strata/core/conversation_snapshot.hpp"#include "strata/core/conversation_file.hpp"#include "strata/core/session_save_reclaim.hpp"@@ -156,6 +157,7 @@ namespace strata::metal_engine { void set_model_shard(const char* shard1); } /#include <vector>namespace {+namespace nib = strata::core::nibble;// Windows' WDDM driver model: native Windows, or WSL2 (its GPU goes through /dev/dxg to the Windows driver). There,// pinning a large arena into two CUDA contexts leaves WDDM refusing every later allocation (the 5080 + 3090 rig);// a Linux driver has no such limit (#253: the 8 GiB cap cost a 4090 + 3060 split 3x of its prompt speed).@@ -388,6 +390,7 @@ bool resident_stage_swaps(strata::core::FileExpertSource& src, const std::vectorif (slot < 0) continue;const SwapHome home = locate(s.layer);const strata::core::OnDevice on(home.dev);+ nib::guard_raw(s.layer, "the adaptive tier's copy back (resident_stage_swaps)");if (const cudaError_t e = cudaMemcpyAsync(src.exchange_buffer(q, s.layer), home.cache->device_slot(slot),(size_t) strata::kernels::cpu::expert_layout().blob_bytes(s.layer), cudaMemcpyDeviceToHost,home.stream); e != cudaSuccess) {@@ -2248,6 +2251,297 @@ struct PcieAuto {}};+// ---- STRATA_EXPERT_NIBBLE=1 (strata/core/expert_nibble.hpp): the gate/up experts of IQ2_S / IQ3_XXS / IQ3_S layers as+// 4-bit codes in their VRAM slots. Everything below runs only with the switch on.++/// The largest slot an expert cache can hold for this pack: the GGUF `max_blob`, or a converted layer's 4-bit slot when+/// that is larger (the sizings that count slots of the largest blob would otherwise come out short of a full cache and+/// leave converted experts out of VRAM, which the switch refuses). `max_blob` exactly with the switch off.+uint64_t nibble_max_slot_bytes() {+ const auto& lay = strata::kernels::cpu::expert_layout();+ uint64_t m = lay.max_blob;+ if (nib::any_on())+ for (int64_t l = 0; l < lay.n_layers; ++l) m = std::max<uint64_t>(m, nib::slot_bytes(l));+ return m;+}++#if STRATA_EXPERT_NIBBLE_BUILT+/// One expert cache's converted slots. The startup fill hands a converted layer's pairs to `add` instead of+/// `fill_slot_blocking`; per layer, `kBatch` GGUF blobs at a time go H2D into one staging buffer (the bytes+/// fill_slot_blocking would copy, from the source's `blob_stable`, which stays valid) and, per batch, the+/// transcoder writes the codes, scales and d into the slots and round-trips every item, the GGUF down rows are copied+/// behind them, the GGUF blocks are rebuilt from every slot and compared with the blob (both count against the layer+/// in `d_bad`), and each slot's hash is recorded for STRATA_EXPERT_NIBBLE_RECHECK. One stream, so the staging+/// buffer's reuse is ordered without events. The device arrays live as long as the process (RECHECK reads them);+/// `finish` frees the staging buffer.+struct NibbleCache {+ static constexpr int64_t kBatch = 64;+ struct Pending { int32_t expert, slot; const uint8_t* blob; };+ struct Batch { int32_t layer; int64_t at, n; };+ strata::core::ExpertCache* cache = nullptr;+ int dev = 0; ///< the cache's device+ std::vector<std::vector<Pending>> pending; ///< per layer: queued, at most kBatch+ std::vector<Batch> batches; ///< what was transcoded and hashed, in that order+ std::vector<std::pair<int32_t, int32_t>> pairs; ///< (layer, expert) of each converted slot, in that order+ std::vector<int32_t> slots; ///< and its slot+ unsigned long long* h_ptrs = nullptr; ///< pinned: the slots' device addresses, in that order+ unsigned long long* d_ptrs = nullptr; ///< the same on the device (the transcoder's and hash's input)+ unsigned long long* d_ref = nullptr; ///< each slot's hash as loaded+ unsigned long long* d_now = nullptr; ///< RECHECK's+ unsigned long long* d_bad = nullptr; ///< per layer: items that did not round-trip, words that differ+ uint8_t* staging = nullptr;+ uint64_t raw_max = 0;+ cudaStream_t stream = nullptr;+ int64_t cap = 0, n = 0;+ double seconds = 0;++ bool active() const { return cache != nullptr; }++ static std::string cuda_fail(const char* what, cudaError_t e) { return std::string(what) + ": " + cudaGetErrorString(e); }++ /// For `c` on the current device, `want` converted pairs at most.+ bool begin(strata::core::ExpertCache& c, int64_t want, std::string& err) {+ if (cudaGetDevice(&dev) != cudaSuccess) dev = 0;+ cache = &c;+ cap = want > 0 ? want : 1;+ const auto& lay = strata::kernels::cpu::expert_layout();+ pending.assign((size_t) lay.n_layers, {});+ for (int64_t l = 0; l < lay.n_layers; ++l)+ if (nib::layer(l).on) raw_max = std::max<uint64_t>(raw_max, nib::layer(l).raw_bytes);+ cudaError_t e = cudaStreamCreateWithFlags(&stream, cudaStreamNonBlocking);+ if (e == cudaSuccess) e = cudaMallocHost((void**) &h_ptrs, (size_t) cap * 8);+ if (e == cudaSuccess) e = cudaMalloc((void**) &d_ptrs, (size_t) cap * 8);+ if (e == cudaSuccess) e = cudaMalloc((void**) &d_ref, (size_t) cap * 8);+ if (e == cudaSuccess) e = cudaMalloc((void**) &d_now, (size_t) cap * 8);+ if (e == cudaSuccess) e = cudaMalloc((void**) &d_bad, (size_t) lay.n_layers * 8);+ if (e == cudaSuccess) e = cudaMemsetAsync(d_bad, 0, (size_t) lay.n_layers * 8, stream);+ if (e == cudaSuccess) e = cudaMalloc((void**) &staging, (size_t) ((uint64_t) kBatch * raw_max));+ if (e != cudaSuccess) {+ err = cuda_fail("the 4-bit fill's buffers", e);+ return false;+ }+ if (const int rc = nib::prepare((void*) stream); rc != 0) {+ err = std::string("the 4-bit fill's GGUF codebooks: ") ++ (rc == (int) cudaErrorInvalidValue ? "a grid entry is not unique" : cudaGetErrorString((cudaError_t) rc));+ return false;+ }+ return true;+ }++ bool add(int32_t layer, int32_t expert, int32_t slot, const uint8_t* blob, std::string& err) {+ auto& q = pending[(size_t) layer];+ q.push_back({expert, slot, blob});+ return (int64_t) q.size() < kBatch || flush(layer, err);+ }++ bool flush(int32_t layer, std::string& err) {+ auto& q = pending[(size_t) layer];+ if (q.empty()) return true;+ const auto t0 = std::chrono::steady_clock::now();+ const nib::LayerVram& L = nib::layer(layer);+ const int64_t k = (int64_t) q.size();+ if (n + k > cap) {+ err = "the 4-bit fill was given more converted pairs than it counted";+ return false;+ }+ cudaError_t e = cudaSuccess;+ for (int64_t i = 0; i < k && e == cudaSuccess; ++i) {+ uint8_t* dst = cache->device_slot(q[(size_t) i].slot);+ if (dst == nullptr || cache->slot_room(q[(size_t) i].slot) < L.hashed) { // a slot sized for the GGUF blob+ err = "the 4-bit fill: slot " + std::to_string(q[(size_t) i].slot) ++ (dst == nullptr ? " is outside the cache" : " is smaller than a 4-bit slot");+ return false;+ }+ h_ptrs[n + i] = (unsigned long long) (uintptr_t) dst;+ e = cudaMemcpyAsync(staging + (uint64_t) i * L.raw_bytes, q[(size_t) i].blob, (size_t) L.raw_bytes,+ cudaMemcpyHostToDevice, stream);+ }+ if (e == cudaSuccess) e = cudaMemcpyAsync(d_ptrs + n, h_ptrs + n, (size_t) k * 8, cudaMemcpyHostToDevice, stream);+ if (e != cudaSuccess) {+ err = cuda_fail("the 4-bit fill's copies in", e);+ return false;+ }+ auto m2 = [&](const char* what, int rc) -> bool { // iq_kernels.cu's: a CUDA error code+ if (rc == 0) return true;+ err = std::string("the 4-bit ") + what + ": " + cudaGetErrorString((cudaError_t) rc);+ return false;+ };+ if (!m2("transcoder", nib::transcode(L, staging, k, d_ptrs + n, d_bad + layer, (void*) stream))) return false;+ for (int64_t i = 0; i < k && e == cudaSuccess; ++i)+ e = cudaMemcpyAsync((uint8_t*) (uintptr_t) h_ptrs[n + i] + nib::kDownOff,+ staging + (uint64_t) i * L.raw_bytes + L.raw_down_off, (size_t) L.down_bytes,+ cudaMemcpyDeviceToDevice, stream);+ if (e != cudaSuccess) {+ err = cuda_fail("the 4-bit fill's down rows", e);+ return false;+ }+ if (!m2("GGUF check", nib::check_gguf(L, staging, k, d_ptrs + n, d_bad + layer, (void*) stream))) return false;+ if (!m2("slot hash", nib::hash_slots(L, d_ptrs + n, k, d_ref + n, (void*) stream))) return false;+ // the next batch's copies in land in the same staging bytes: they wait for this batch's kernels (one stream)+ for (const Pending& p : q) {+ pairs.emplace_back(layer, p.expert);+ slots.push_back(p.slot);+ }+ batches.push_back({layer, n, k});+ n += k;+ q.clear();+ seconds += std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count();+ return true;+ }++ /// The rest of the queue, then waits: false (with `err` naming every layer) when an item did not round-trip or a+ /// rebuilt GGUF block differs from its blob.+ bool finish(std::string& err) {+ for (int32_t l = 0; l < (int32_t) pending.size(); ++l)+ if (!flush(l, err)) return false;+ const auto t0 = std::chrono::steady_clock::now();+ cudaError_t e = cudaStreamSynchronize(stream);+ std::vector<unsigned long long> bad(pending.size(), 0);+ if (e == cudaSuccess) e = cudaMemcpy(bad.data(), d_bad, bad.size() * 8, cudaMemcpyDeviceToHost);+ if (staging != nullptr) (void) cudaFree(staging);+ staging = nullptr;+ seconds += std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count();+ if (e != cudaSuccess) {+ err = cuda_fail("the 4-bit fill", e);+ return false;+ }+ for (size_t l = 0; l < bad.size(); ++l)+ if (bad[l] != 0) {+ char b[192];+ std::snprintf(b, sizeof b, "%slayer %zu (%s): %llu items or words do not round-trip", err.empty() ? "" : "; ",+ l, nib::fmt_name(nib::layer((int64_t) l).gu_type), bad[l]);+ err += b;+ }+ if (!err.empty()) {+ err += " - the 4-bit code does not hold these weights (STRATA_EXPERT_NIBBLE_SKIP=<layers> keeps a layer's "+ "GGUF blocks)";+ return false;+ }+ return true;+ }++ /// The end-to-end check of the fill as it is wired, on converted pair (layer, expert) with `raw` its GGUF blob: the+ /// slot read back, its hash against expert_nibble.hpp's host reference of what the device recorded, its down rows+ /// against the blob's. (Every item and every GGUF block of every slot was checked on the device by `finish`.)+ bool check(int32_t layer, int32_t expert, const uint8_t* raw, std::string& err) const {+ int64_t at = -1;+ for (size_t i = 0; i < pairs.size() && at < 0; ++i)+ if (pairs[i].first == layer && pairs[i].second == expert) at = (int64_t) i;+ if (at < 0 || raw == nullptr) {+ err = "the 4-bit check: no converted slot or no GGUF blob for layer " + std::to_string(layer) + " expert " ++ std::to_string(expert);+ return false;+ }+ const nib::LayerVram& L = nib::layer(layer);+ std::vector<uint8_t> got((size_t) L.hashed);+ unsigned long long h = 0;+ cudaError_t e = cudaMemcpy(got.data(), cache->device_slot(slots[(size_t) at]), got.size(), cudaMemcpyDeviceToHost);+ if (e == cudaSuccess) e = cudaMemcpy(&h, d_ref + at, 8, cudaMemcpyDeviceToHost);+ if (e != cudaSuccess) {+ err = cuda_fail("the 4-bit check's read back", e);+ return false;+ }+ const char* why = nib::hash_host(got.data(), L.hashed) != h ? "the device slot hash differs from the host reference"+ : std::memcmp(got.data() + nib::kDownOff, raw + L.raw_down_off, (size_t) L.down_bytes) != 0+ ? "the down rows differ from the GGUF blob's"+ : nullptr;+ if (why != nullptr) {+ err = "layer " + std::to_string(layer) + " expert " + std::to_string(expert) + " (" + nib::fmt_name(L.gu_type) ++ "): " + why;+ return false;+ }+ return true;+ }++ /// STRATA_EXPERT_NIBBLE_RECHECK: every converted slot's hash again, on the device, against the one recorded at load.+ /// False with the first slot that differs (or a CUDA error) in `err`; `bad_layer` its layer (-1: an error).+ bool recheck(std::string& err, int64_t& bad_layer) {+ bad_layer = -1;+ if (n == 0) return true;+ const strata::core::OnDevice on(dev);+ for (const Batch& b : batches) {+ const int rc = nib::hash_slots(nib::layer(b.layer), d_ptrs + b.at, b.n, d_now + b.at, (void*) stream);+ if (rc != 0) {+ err = cuda_fail("the slot hash", (cudaError_t) rc);+ return false;+ }+ }+ std::vector<unsigned long long> ref((size_t) n), now((size_t) n);+ cudaError_t e = cudaStreamSynchronize(stream);+ if (e == cudaSuccess) e = cudaMemcpy(ref.data(), d_ref, (size_t) n * 8, cudaMemcpyDeviceToHost);+ if (e == cudaSuccess) e = cudaMemcpy(now.data(), d_now, (size_t) n * 8, cudaMemcpyDeviceToHost);+ if (e != cudaSuccess) {+ err = cuda_fail("the slot hash read back", e);+ return false;+ }+ for (int64_t i = 0; i < n; ++i)+ if (ref[(size_t) i] != now[(size_t) i]) {+ char b[224];+ std::snprintf(b, sizeof b, "CUDA%d: layer %d expert %d (slot %d) hashes %016llx, %016llx at load", dev,+ (int) pairs[(size_t) i].first, (int) pairs[(size_t) i].second, (int) slots[(size_t) i],+ now[(size_t) i], ref[(size_t) i]);+ err = b;+ bad_layer = pairs[(size_t) i].first;+ return false;+ }+ return true;+ }+};+#else+// A build without the 4-bit kernels (STRATA_EXPERT_NIBBLE_BUILT 0: HIP, Metal, a CUDA build without+// STRATA_EXPERIMENTAL_SM60): nibble_refusal refuses the switch, so no cache is ever converted and none of this is entered.+struct NibbleCache {+ int64_t n = 0;+ double seconds = 0;+ std::vector<std::pair<int32_t, int32_t>> pairs;+ bool active() const { return false; }+ bool begin(strata::core::ExpertCache&, int64_t, std::string& err) { err = "the 4-bit slots are not in this build"; return false; }+ bool add(int32_t, int32_t, int32_t, const uint8_t*, std::string& err) { err = "the 4-bit slots are not in this build"; return false; }+ bool finish(std::string&) { return true; }+ bool check(int32_t, int32_t, const uint8_t*, std::string& err) const { err = "the 4-bit slots are not in this build"; return false; }+ bool recheck(std::string&, int64_t& bad_layer) { bad_layer = -1; return true; }+};+#endif++/// With STRATA_EXPERT_NIBBLE=1, why this start cannot have 4-bit slots (empty: nothing here stops it): what the build,+/// the GPUs, the environment and the options decide, before anything is allocated. The full residency, the loans and+/// the prompt chunk are checked where they are known.+std::string nibble_refusal(const Options& o) {+ const nib::Config& c = nib::config();+ if (!c.error.empty()) return c.error;+#if !STRATA_EXPERT_NIBBLE_BUILT || !defined(STRATA_NATIVE_EXPERTS)+ (void) o;+ return "the 4-bit slots and their kernels are in the CUDA build with STRATA_EXPERIMENTAL_SM60 and native experts only";+#else+ int count = 0;+ if (cudaGetDeviceCount(&count) != cudaSuccess || count <= 0) {+ (void) cudaGetLastError();+ return "no CUDA device";+ }+ for (int d = 0; d < count; ++d) {+ cudaDeviceProp p{};+ if (cudaGetDeviceProperties(&p, d) != cudaSuccess) {+ (void) cudaGetLastError();+ return "cannot read CUDA" + std::to_string(d) + "'s properties";+ }+ if (p.major != 6 || p.minor != 0)+ return "CUDA" + std::to_string(d) + " (" + p.name + ") is compute capability " + std::to_string(p.major) + "." ++ std::to_string(p.minor) + ": the 4-bit slots are for compute capability 6.0 (GP100) only";+ }+ if (std::string e = nib::env_conflicts(); !e.empty()) return e;+ if (o.peer_device >= 1) return "--peer-device (its cache holds GGUF blobs)";+ for (size_t r = 0; r < o.expert_cache_remote.size(); ++r)+ if (o.expert_cache_remote[r] > 0 || o.expert_cache_remote_auto[r])+ return "remote expert caches (--expert-cache-device1..3 hold GGUF blobs)";+ if (o.kv_grow) return "--kv-grow (the elastic K/V moves slots at their GGUF size)";+ if (o.vram_elastic) return "--vram-elastic (it refills slots with GGUF blobs)";+ if (o.expert_profile.empty())+ return "no --expert-profile (every expert must be VRAM-resident from the profile fill)";+ if (o.expert_cache == 0) return "--expert-cache 0 (no VRAM slots to convert)";+ return {};+#endif+}+} // namespace// #1425 #1549 #1629: --ple-io direct (the default) does unbuffered random reads of the n-gram table; on some drives@@ -3564,6 +3858,49 @@ int main(int argc, char** argv) {}std::fprintf(stderr, "strata generate: primary device from STRATA_PRIMARY_DEVICE: CUDA%d\n", primary_device);}+ // ---- STRATA_EXPERT_NIBBLE=1 keeps the gate/up experts of IQ2_S / IQ3_XXS / IQ3_S layers as 4-bit codes in their VRAM+ // slots (strata/core/expert_nibble.hpp). Unset or 0: nothing here runs, no table is set, and every slot size below is+ // the GGUF blob's, as before. On: if anything would read a converted slot as GGUF blocks or write GGUF bytes into+ // one, the switch is ignored for this start with a warning (before anything is allocated); else the per-layer table+ // is set before the first slot estimate and the adaptive tier is off (it swaps GGUF blobs in and out of slots).+ // The residency, the loans and the prompt chunk are checked where they are known (those refuse the start).+ const bool nib_asked = nib::config().on || nib::config().error.rfind("STRATA_EXPERT_NIBBLE=", 0) == 0;+ const int64_t nib_chunk_asked = o.prefill_chunk; // an explicit --prefill N must stay N (the chunking is in the bits)+ if (nib_asked) {+ std::string why = nibble_refusal(o);+ if (why.empty()) {+ nib::set_layers(nib::plan_layers(strata::kernels::cpu::expert_layout(), why));+ if (why.empty() && !nib::any_on())+ why = "no layer of this pack is IQ2_S / IQ3_XXS / IQ3_S at 2560 x 640 among STRATA_EXPERT_NIBBLE_FMTS "+ "and outside STRATA_EXPERT_NIBBLE_SKIP";+ if (why.empty()) (void) strata::prefill::Prefill::nibble_prompt_check(why);+ }+ if (!why.empty()) {+ // a warning, not a refusal: this start goes on as it would without the switch (the usual GGUF slots)+ std::fprintf(stderr, "strata generate: WARNING: STRATA_EXPERT_NIBBLE is off for this start: %s; the expert slots "+ "stay GGUF blocks\n", why.c_str());+ nib::set_layers({});+ } else {+ o.adapt_swaps = 0;+ o.peer_adapt_swaps = 0;+ const auto& lay = strata::kernels::cpu::expert_layout();+ int64_t n_on = 0, by_fmt[3] = {0, 0, 0};+ uint64_t more = 0;+ for (int64_t l = 0; l < lay.n_layers; ++l) {+ const nib::LayerVram& v = nib::layer(l);+ if (!v.on) continue;+ ++n_on;+ ++by_fmt[v.gu_type == nib::kIQ2_S ? 0 : v.gu_type == nib::kIQ3_XXS ? 1 : 2];+ more += (v.slot_bytes - (v.raw_bytes + 255) / 256 * 256) * (uint64_t) lay.n_expert;+ }+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: %lld of %lld layers keep their gate/up experts as 4-bit "+ "codes in VRAM (IQ2_S %lld, IQ3_XXS %lld, IQ3_S %lld), %.2f GiB more VRAM than their GGUF "+ "blobs for all %lld experts of each; the adaptive tier is off (--adapt-swaps 0)%s\n",+ (long long) n_on, (long long) lay.n_layers, (long long) by_fmt[0], (long long) by_fmt[1],+ (long long) by_fmt[2], (double) more / 1073741824.0, (long long) lay.n_expert,+ nib::config().recheck ? "; STRATA_EXPERT_NIBBLE_RECHECK: every slot's hash after each request" : "");+ }+ }// STRATA_EARLY_REMOTE_CONTEXTS=1: create EVERY secondary context here, like CUDA1's. Under WSL2 the driver's// pinned/mapped host budget (dxg gpadl, ~1 GiB) is spent by CUDA0's weights and MTP before the later loop runs,// and a new context then fails with cudaErrorMemoryAllocation (CUDA2: "cudaSetDevice(2) failed: out of memory").@@ -4291,7 +4628,7 @@ int main(int argc, char** argv) {const auto& lay = strata::kernels::cpu::expert_layout();int64_t pairs_bytes = 0;for (const auto& pr : prof)- pairs_bytes += native_pack ? strata::core::ExpertCache::slot_stride((int64_t) lay.blob_bytes(pr.first), slot_guard)+ pairs_bytes += native_pack ? strata::core::ExpertCache::slot_stride((int64_t) nib::slot_bytes(pr.first), slot_guard): (int64_t) lay.max_blob;size_t fb = 0, tb = 0;cudaMemGetInfo(&fb, &tb);@@ -4754,7 +5091,7 @@ int main(int argc, char** argv) {total_mass += (mass[r] = hi - lo);}auto cost = [&](int64_t l) -> int64_t {- return native_pack ? strata::core::ExpertCache::slot_stride((int64_t) lay.blob_bytes(l), slot_guard)+ return native_pack ? strata::core::ExpertCache::slot_stride((int64_t) nib::slot_bytes(l), slot_guard): (int64_t) lay.max_blob;};// the predicted window time (ms) of a placement, and the routed mass its caches hold@@ -4870,7 +5207,7 @@ int main(int argc, char** argv) {};struct PartStat { int64_t lb, le; int64_t used; int64_t cnt; int64_t cap; };auto can_start = [&](const std::vector<PartStat>& parts) -> bool {- const int64_t kMaxBlob = (int64_t) strata::kernels::cpu::expert_layout().max_blob;+ const int64_t kMaxBlob = (int64_t) nibble_max_slot_bytes(); // max_blob (or a larger 4-bit slot)auto slots_est_of = [&](int i, uint64_t cn) -> int64_t {if (parts[i].cnt <= 0) return -1;const uint64_t avg = (parts[i].used > 0)@@ -5966,7 +6303,7 @@ int main(int argc, char** argv) {// which only ever grants MORE slots)uint64_t cap_bytes;if (o.expert_cache > 0) {- cap_bytes = (uint64_t) o.expert_cache * pl.max_blob;+ cap_bytes = (uint64_t) o.expert_cache * nibble_max_slot_bytes();} else {const int64_t pf_mib = (o.prefill_chunk > 0 && !pf_borrow) ? 160 + (o.prefill_chunk * 680) / 1024 : 0;int64_t mtp_b = (!o.mtp.empty() && native_head.loaded())@@ -5986,7 +6323,7 @@ int main(int argc, char** argv) {uint64_t used = 0;skip = 0;for (const auto& pr : profile) {- const uint64_t b = pl.native ? (pl.blob_bytes(pr.first) + 255) / 256 * 256 : (uint64_t) pl.max_blob;+ const uint64_t b = pl.native ? (nib::slot_bytes(pr.first) + 255) / 256 * 256 : (uint64_t) pl.max_blob;if (used + b > cap_bytes) break;used += b;++skip;@@ -6222,7 +6559,7 @@ int main(int argc, char** argv) {if (use_df2 && native_head.loaded()) mtp_bind += (int64_t) mtp.bind_bytes(native_head.row_bytes(), n_vocab) + (48 << 20); // DFlash2's own draft head and logits, bound after this#endifconst int64_t reserve = (((int64_t) o.vram_reserve_mib + prefill_mib) << 20) + mtp_bind + pipe_first;- const int64_t blob = (int64_t) strata::kernels::cpu::expert_layout().max_blob;+ const int64_t blob = (int64_t) nibble_max_slot_bytes(); // max_blob (STRATA_EXPERT_NIBBLE: or a larger 4-bit slot)int64_t slots = ((int64_t) free_b - reserve) / blob;if (!profile.empty()) slots = std::min<int64_t>(slots, (int64_t) profile.size());#if defined(STRATA_GPU_METAL)@@ -6320,7 +6657,7 @@ int main(int argc, char** argv) {free_b = strata::core::device_free_bytes(); (void) total_b;const int64_t prefill_mib = owned_prefill_mib();const int64_t reserve = (((int64_t) o.vram_reserve_mib + prefill_mib) << 20) + pipe_first;- const int64_t fit = std::max<int64_t>(((int64_t) free_b - reserve) / (int64_t) strata::kernels::cpu::expert_layout().max_blob, 0);+ const int64_t fit = std::max<int64_t>(((int64_t) free_b - reserve) / (int64_t) nibble_max_slot_bytes(), 0);if (o.expert_cache > fit) {// a WARNING that names the knob: the user asked for this size, and gets fewer slotsstd::fprintf(stderr, "strata generate: WARNING: layer split: --expert-cache %d leaves no room for the "@@ -6342,19 +6679,19 @@ int main(int argc, char** argv) {const size_t free_b = strata::core::device_free_bytes(); // the same reading the shared-cache sizing usesconst auto& lay = strata::kernels::cpu::expert_layout();const int asked = o.expert_cache;- const uint64_t budget = (uint64_t) asked * lay.max_blob;+ const uint64_t budget = (uint64_t) asked * nibble_max_slot_bytes();size_t free_room = free_b > ((size_t) o.vram_reserve_mib << 20) ? free_b - ((size_t) o.vram_reserve_mib << 20) : 0;const uint64_t cap = std::min<uint64_t>(budget, (uint64_t) free_room);uint64_t sum = 0;for (int64_t l = 0; l < g.n_layers; ++l)- sum += (uint64_t) strata::core::ExpertCache::slot_stride((int64_t) lay.blob_bytes(l), slot_guard);+ sum += (uint64_t) strata::core::ExpertCache::slot_stride((int64_t) nib::slot_bytes(l), slot_guard);int64_t q = sum > 0 ? (int64_t) (cap / sum) : 0;if (q > g.n_expert) q = g.n_expert;if (q > 0) {per_layer_bytes = sum;sized_slots.reserve((size_t) q * (size_t) g.n_layers);for (int64_t l = 0; l < g.n_layers; ++l) {- const int64_t b = (int64_t) lay.blob_bytes(l);+ const int64_t b = (int64_t) nib::slot_bytes(l);for (int64_t s = 0; s < q; ++s) sized_slots.push_back(b);}o.expert_cache = (int) sized_slots.size();@@ -6370,16 +6707,16 @@ int main(int argc, char** argv) {// what the uniform sizing granted, in the 256-byte slots the loop below counts: the blob formats of 1280 / 2560// rows are whole multiples of 256 already; an NVFP4 blob's 16-byte scale tail is not, and the unrounded budget// held one slot fewer than asked (--expert-cache 1: none)- const uint64_t budget = (uint64_t) o.expert_cache * ((lay.max_blob + 255) / 256 * 256);+ const uint64_t budget = (uint64_t) o.expert_cache * ((nibble_max_slot_bytes() + 255) / 256 * 256);uint64_t used = 0;const size_t keep_free = ((size_t) o.vram_reserve_mib << 20) + (size_t) pipe_first;size_t free_room = free_b > keep_free ? free_b - keep_free : 0;const uint64_t cap = std::min<uint64_t>(budget, (uint64_t) free_room);for (const auto& pr : profile) {- const uint64_t b = (uint64_t) strata::core::ExpertCache::slot_stride((int64_t) lay.blob_bytes(pr.first), slot_guard);+ const uint64_t b = (uint64_t) strata::core::ExpertCache::slot_stride((int64_t) nib::slot_bytes(pr.first), slot_guard);if (used + b > cap) break;used += b;- sized_slots.push_back((int64_t) lay.blob_bytes(pr.first));+ sized_slots.push_back((int64_t) nib::slot_bytes(pr.first));}// #831: the sized-slots loop walks the profile's pairs, so the profile's length is the ceiling; an explicit// --expert-cache above it used to be cut with no word. A warning only (what is allocated does not change).@@ -6429,7 +6766,7 @@ int main(int argc, char** argv) {sized_slots.clear();if (q <= 0) { o.expert_cache = 0; return false; }for (int64_t l = 0; l < g.n_layers; ++l) {- const int64_t b = (int64_t) lay.blob_bytes(l);+ const int64_t b = (int64_t) nib::slot_bytes(l);for (int64_t s = 0; s < q; ++s) sized_slots.push_back(b);}o.expert_cache = (int) sized_slots.size();@@ -6554,6 +6891,10 @@ int main(int argc, char** argv) {// admission finds no room and every non-profiled expert stays a CPU miss. That is what makes the profile// the policy rather than a hint.int64_t prefilled = 0;+ // STRATA_EXPERT_NIBBLE: each cache's converted slots (RECHECK reads them per request)+ NibbleCache nib_x;+ std::vector<NibbleCache> nib_st;+ nib_st.reserve(stages.size());if (!profile.empty() && srcp != nullptr) {// #369 (dag08): per layer, a full layer skips only its own pairs - each layer takes its hottest experts until// its range is full (one full layer used to end the whole fill, leaving most layers empty)@@ -6574,6 +6915,14 @@ int main(int argc, char** argv) {}const auto fill_t0 = std::chrono::steady_clock::now();uint64_t fill_bytes = 0;+ if (nib::any_on()) { // STRATA_EXPERT_NIBBLE: the converted layers' pairs go through the 4-bit fill instead+ int64_t conv = 0;+ for (int64_t i = 0; i < want; ++i) conv += nib::layer(profile[(size_t) i].first).on ? 1 : 0;+ if (conv > 0 && !nib_x.begin(xcache, conv, err)) {+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: %s\n", err.c_str());+ return 1;+ }+ }for (int64_t i = 0; i < want; ++i) {if (!per_layer && srcp == &src && src.unbuffered() && i % 64 == 0) {if (ahead.valid()) ahead.get();@@ -6587,8 +6936,10 @@ int main(int argc, char** argv) {break;}const uint8_t* b = srcp->blob_stable(profile[(size_t) i].first, profile[(size_t) i].second);- if (b == nullptr || !xcache.fill_slot_blocking(slot, b, err,- (int64_t) strata::kernels::cpu::expert_layout().blob_bytes(profile[(size_t) i].first))) {+ const bool nib_pair = nib_x.active() && nib::layer(profile[(size_t) i].first).on; // STRATA_EXPERT_NIBBLE+ if (b == nullptr || !(nib_pair ? nib_x.add(profile[(size_t) i].first, profile[(size_t) i].second, slot, b, err)+ : xcache.fill_slot_blocking(slot, b, err,+ (int64_t) strata::kernels::cpu::expert_layout().blob_bytes(profile[(size_t) i].first)))) {std::fprintf(stderr, "strata generate: the profile fill failed at pair %lld: %s\n",(long long) i, err.c_str());return 1;@@ -6596,16 +6947,39 @@ int main(int argc, char** argv) {fill_bytes += strata::kernels::cpu::expert_layout().blob_bytes(profile[(size_t) i].first);++prefilled;}+ if (nib_x.active() && !nib_x.finish(err)) {+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: CUDA0: %s\n", err.c_str());+ return 1;+ }const double fill_s = std::chrono::duration<double>(std::chrono::steady_clock::now() - fill_t0).count();// **AND ONE SLOT IS READ BACK AND COMPARED.** A residency table that is right about indices and wrong// about bytes produces a plausible token, which is this project's most expensive failure mode; the// cache's own `verify_slot` is the check and it costs one 1.38 MB D2H at startup.- if (prefilled > 0 && !xcache.verify_slot(xcache.slot_of(profile[0].first, profile[0].second),+ // (STRATA_EXPERT_NIBBLE: a converted slot holds 4-bit codes, not the blob's bytes - the 4-bit check below is its check)+ if (prefilled > 0 && !nib::layer(profile[0].first).on &&+ !xcache.verify_slot(xcache.slot_of(profile[0].first, profile[0].second),srcp->blob_stable(profile[0].first, profile[0].second), err,(int64_t) strata::kernels::cpu::expert_layout().blob_bytes(profile[0].first))) {std::fprintf(stderr, "strata generate: %s\n", err.c_str());return 1;}+ if (nib_x.active() && nib_x.n > 0) {+ // STRATA_EXPERT_NIBBLE: the hottest converted pair read back (its hash, its down rows); every item and every+ // rebuilt GGUF block of every converted slot was already checked on the device+ size_t c = 0;+ while (c < (size_t) want && !nib::layer(profile[c].first).on) ++c;+ if (!nib_x.check(profile[c].first, profile[c].second, srcp->blob_stable(profile[c].first, profile[c].second),+ err)) {+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: CUDA0: %s\n", err.c_str());+ return 1;+ }+ std::set<int32_t> ls;+ for (const auto& pr : nib_x.pairs) ls.insert(pr.first);+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: CUDA0: %lld slots of %zu layers transcoded to 4-bit "+ "codes (%.2f s; every item round-tripped, every slot's GGUF blocks rebuilt and compared, "+ "slot hashes recorded); layer %d expert %d read back\n", (long long) nib_x.n, ls.size(),+ nib_x.seconds, (int) profile[c].first, (int) profile[c].second);+ }mem_mark("the profile fill");std::fprintf(stderr, "strata generate: pre-filled %lld of %lld slots from the profile in %.1f s (%.0f MB/s); ""slot 0 verified\n",@@ -6662,11 +7036,11 @@ int main(int argc, char** argv) {std::vector<int64_t> sized;int64_t used = 0;for (const auto& pr : st.profile) {- const int64_t b = native_pack ? strata::core::ExpertCache::slot_stride((int64_t) lay.blob_bytes(pr.first), slot_guard)+ const int64_t b = native_pack ? strata::core::ExpertCache::slot_stride((int64_t) nib::slot_bytes(pr.first), slot_guard): (int64_t) lay.max_blob;if (used + b > room) break;used += b;- sized.push_back((int64_t) lay.blob_bytes(pr.first));+ sized.push_back((int64_t) nib::slot_bytes(pr.first));}// #841: a stage's cache can fail to open with the VRAM free (the card's allocation is also charged to the// Windows commit, or the free figure ran high): it is tried again with 90% of the slots, up to 3 times, each@@ -6691,25 +7065,57 @@ int main(int argc, char** argv) {return 1;}int64_t filled = 0;+ NibbleCache& nib_c = nib_st.emplace_back(); // STRATA_EXPERT_NIBBLE: this stage's converted slots+ if (nib::any_on()) {+ int64_t conv = 0;+ for (const auto& pr : st.profile) conv += nib::layer(pr.first).on ? 1 : 0;+ if (conv > 0 && !nib_c.begin(st.cache, conv, err)) {+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: CUDA%d: %s\n", st.dev, err.c_str());+ return 1;+ }+ }for (const auto& pr : st.profile) {if (filled >= st.cache.slots()) break;const int32_t slot = st.cache.admit(pr.first, pr.second);if (slot == strata::core::kNotResident) break;const uint8_t* b = srcp->blob_stable(pr.first, pr.second);- if (b == nullptr || !st.cache.fill_slot_blocking(slot, b, err, (int64_t) lay.blob_bytes(pr.first))) {+ const bool nib_pair = nib_c.active() && nib::layer(pr.first).on; // STRATA_EXPERT_NIBBLE+ if (b == nullptr || !(nib_pair ? nib_c.add(pr.first, pr.second, slot, b, err)+ : st.cache.fill_slot_blocking(slot, b, err, (int64_t) lay.blob_bytes(pr.first)))) {std::fprintf(stderr, "strata generate: layer split, CUDA%d profile fill failed at pair %lld: %s\n",st.dev, (long long) filled, err.c_str());return 1;}++filled;}- if (filled == 0 || !st.cache.verify_slot(st.cache.slot_of(st.profile[0].first, st.profile[0].second),- srcp->blob_stable(st.profile[0].first, st.profile[0].second), err,- (int64_t) lay.blob_bytes(st.profile[0].first))) {+ if (nib_c.active() && !nib_c.finish(err)) {+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: CUDA%d: %s\n", st.dev, err.c_str());+ return 1;+ }+ // (STRATA_EXPERT_NIBBLE: a converted slot holds 4-bit codes, not the blob's bytes - the 4-bit check below is its check)+ if (filled == 0 || (!nib::layer(st.profile[0].first).on &&+ !st.cache.verify_slot(st.cache.slot_of(st.profile[0].first, st.profile[0].second),+ srcp->blob_stable(st.profile[0].first, st.profile[0].second), err,+ (int64_t) lay.blob_bytes(st.profile[0].first)))) {std::fprintf(stderr, "strata generate: layer split, CUDA%d expert cache: %s\n", st.dev,filled == 0 ? "nothing filled" : err.c_str());return 1;}+ if (nib_c.active() && nib_c.n > 0) { // STRATA_EXPERT_NIBBLE: as CUDA0's above+ size_t c = 0;+ while (c < st.profile.size() && !nib::layer(st.profile[c].first).on) ++c;+ const auto& pc = st.profile[c];+ if (!nib_c.check(pc.first, pc.second, srcp->blob_stable(pc.first, pc.second), err)) {+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: CUDA%d: %s\n", st.dev, err.c_str());+ return 1;+ }+ std::set<int32_t> ls;+ for (const auto& pr : nib_c.pairs) ls.insert(pr.first);+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: CUDA%d: %lld slots of %zu layers transcoded to 4-bit "+ "codes (%.2f s; every item round-tripped, every slot's GGUF blocks rebuilt and compared, "+ "slot hashes recorded); layer %d expert %d read back\n", st.dev, (long long) nib_c.n,+ ls.size(), nib_c.seconds, (int) pc.first, (int) pc.second);+ }std::fprintf(stderr, "strata generate: layer split: CUDA%d runs layers %lld-%lld, expert cache %lld slots ""(%.2f GiB), %lld of its %zu profiled pairs; slot 0 verified\n",st.dev, (long long) st.lb, (long long) (st.le - 1), (long long) st.cache.slots(), st.cache.gib(),@@ -6717,6 +7123,40 @@ int main(int argc, char** argv) {}if (multi_gpu)std::fprintf(stderr, "strata generate: layer split: CUDA0 runs layers 0-%lld\n", (long long) (split_at[0] - 1));+ if (nib::any_on()) {+ // STRATA_EXPERT_NIBBLE: every expert of every layer must be in VRAM, in the cache of the card that runs the layer.+ // A converted layer's miss would be computed from the GGUF blob by the CPU or over PCIe, and a non-converted+ // layer's residency would differ from the switch-off run's (its smaller slots leave room for more), so the+ // replies' bits would too. Then no slot is written again.+ for (int64_t l = 0; l < g.n_layers; ++l) {+ const strata::core::ExpertCache* owner = &xcache;+ int owner_dev = 0;+ if (multi_gpu && !split_at.empty() && l >= split_at[0]) {+ owner = nullptr;+ for (const auto& stp : stages)+ if (l >= stp->lb && l < stp->le) { owner = &stp->cache; owner_dev = stp->dev; }+ }+ int64_t missing = 0, first = -1;+ for (int64_t e = 0; e < g.n_expert; ++e)+ if (owner == nullptr || owner->slot_of(l, e) == strata::core::kNotResident) {+ if (first < 0) first = e;+ ++missing;+ }+ if (missing > 0) {+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE refuses the start: layer %lld has %lld of its %lld "+ "experts out of CUDA%d's VRAM (expert %lld the first) - every expert must be resident "+ "(a profile that ranks every pair and room on the card; STRATA_EXPERT_NIBBLE_SKIP="+ "<layers> keeps some layers' smaller GGUF slots)\n", (long long) l, (long long) missing,+ (long long) g.n_expert, owner_dev, (long long) first);+ return 1;+ }+ }+ xcache.seal();+ for (auto& stp : stages) stp->cache.seal();+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: all %lld experts of all %lld layers are VRAM-resident; "+ "the expert caches are sealed (nothing writes a slot from here on)\n",+ (long long) g.n_expert, (long long) g.n_layers);+ }// ---- LAYER 2: exact startability probe on the live free figures. Every session, cache and batch slot is// carved by now, so `cudaMemGetInfo` reads what serve's own_fits (5855-5869) will read. Run serve's own@@ -8043,6 +8483,7 @@ int main(int argc, char** argv) {if (!cold.empty() && cold.back().first < heat(oi)) {vi = cold.back().second;const int32_t vs = host_res[(size_t) vi];+ nib::guard_raw(layer, "the elastic K/V's slot move (kvg_ensure)");if (xcache.slot_offset(vs + 1) - xcache.slot_offset(vs) < lay.blob_bytes(layer) ||cudaMemcpy(xcache.device_slot(vs), xcache.device_slot((int32_t) kvg.lo),(size_t) lay.blob_bytes(layer), cudaMemcpyDeviceToDevice) != cudaSuccess)@@ -8145,6 +8586,7 @@ int main(int argc, char** argv) {if (host_res[i] >= 0 || lay.blob_bytes(l) > room) continue;const uint8_t* b = srcp->blob_stable(l, e);if (b == nullptr) continue;+ nib::guard_raw(l, "the elastic K/V's refill (kvg_trim)");cudaMemcpyAsync(xcache.device_slot((int32_t) s), b, (size_t) lay.blob_bytes(l), cudaMemcpyHostToDevice,nullptr);filled.emplace_back(i, (int32_t) s);@@ -8872,6 +9314,37 @@ int main(int argc, char** argv) {#endifbool any_loan = borrow != nullptr;for (size_t i = 1; i < pf_parts.size(); ++i) any_loan = any_loan || pf_parts[i].first >= 0;+ if (nib::any_on()) {+ // STRATA_EXPERT_NIBBLE: the prompt path keeps its own buffers (a loan is refilled with GGUF blobs) and reads in+ // the chunk it reads without the switch (the chunking is part of the replies' bits: auto's ceiling, or+ // --prefill N, and no host-aware plan of smaller chunks). Otherwise refused, never a quietly different reply.+ std::string why;+ const int64_t expect = o.prefill_auto ? auto_ceiling : nib_chunk_asked;+ if (any_loan) why = "the prompt path borrows expert-cache slots (a loan is refilled with GGUF blobs)";+ else if (o.prefill_chunk != expect)+ why = "the prompt chunk is " + std::to_string(o.prefill_chunk) + " tokens, not the " + std::to_string(expect) ++ " it is without the switch when nothing limits it (that would change the replies' bits; an explicit "+ "--prefill N that fits keeps N)";+ else if (pf_plan_chunk > 0)+ why = "the host-aware plan reads long prompts in chunks of " + std::to_string(pf_plan_chunk) + " tokens";+ std::string room;+ for (size_t i = 0; why.empty() && i <= stages.size(); ++i) {+ const int dev = i == 0 ? -1 : stages[i - 1]->dev;+ const strata::core::OnDevice on(dev);+ size_t fb = 0, tb = 0;+ if (cudaMemGetInfo(&fb, &tb) != cudaSuccess) { (void) cudaGetLastError(); continue; }+ char b[64];+ std::snprintf(b, sizeof b, "CUDA%d %.2f GiB", dev < 0 ? 0 : dev, (double) fb / 1073741824.0);+ room += (room.empty() ? "" : ", ") + std::string(b);+ }+ if (!why.empty()) {+ std::fprintf(stderr, "strata serve: STRATA_EXPERT_NIBBLE refuses the start: %s (STRATA_EXPERT_NIBBLE_SKIP=<layers> "+ "keeps some layers' GGUF blocks and their smaller slots)\n", why.c_str());+ return 1;+ }+ std::fprintf(stderr, "strata serve: STRATA_EXPERT_NIBBLE: prompt chunk %lld, no loan; free after the prompt "+ "buffers: %s\n", (long long) o.prefill_chunk, room.c_str());+ }if (any_loan) {if (borrow != nullptr)std::fprintf(stderr, "strata serve: the prompt path borrows %lld CUDA0 cache slots (%.2f GiB)\n",@@ -10544,6 +11017,7 @@ int main(int argc, char** argv) {const size_t in = (size_t) s.layer * g.n_expert + s.in, out = (size_t) s.layer * g.n_expert + s.out;const int32_t slot = fenced_slots.empty() ? host_res[out] : fenced_slots[si];const uint8_t* b = srcp->blob_stable(s.layer, s.in);+ nib::guard_raw(s.layer, "the adaptive tier's copy in");const int stn = multi_gpu ? stage_of(s.layer) : 0; // the swap stays in the layer's own cacheGpuStage* gs = stn > 0 ? stages[(size_t) stn - 1].get() : nullptr;const strata::core::OnDevice on(gs ? gs->dev : -1);@@ -10778,6 +11252,7 @@ int main(int argc, char** argv) {const strata::core::OnDevice on(h.dev);for (const ASwap& w : aswaps) {if (w.home != hi || !w.exchange) continue;+ nib::guard_raw(w.layer, "the async adaptive tier's copy back (a_back)");if (const cudaError_t e = cudaMemcpyAsync(src.exchange_buffer(w.j, w.layer), h.cache->device_slot(w.slot),(size_t) strata::kernels::cpu::expert_layout().blob_bytes(w.layer),cudaMemcpyDeviceToHost, h.stream); e != cudaSuccess) {@@ -10815,6 +11290,7 @@ int main(int argc, char** argv) {auto a_in_one = [&](const ASwap& w, const uint8_t* b) { // step 2: one copy in (H2D), on its home's streamAHome& h = ahomes[(size_t) w.home];const strata::core::OnDevice on(h.dev);+ nib::guard_raw(w.layer, "the async adaptive tier's copy in (a_in_one)");if (const cudaError_t e = cudaMemcpyAsync(h.cache->device_slot(w.slot), b,(size_t) strata::kernels::cpu::expert_layout().blob_bytes(w.layer),cudaMemcpyHostToDevice, h.stream); e != cudaSuccess) {@@ -15459,6 +15935,27 @@ int main(int argc, char** argv) {std::fprintf(stderr, "strata serve: cancelled request keeps its conversation: %lld tokens live\n",(long long) pp_reached);}+ if (nib::config().recheck && nib::any_on() && !cancelled) {+ // STRATA_EXPERT_NIBBLE_RECHECK=1 (debug): every converted slot's hash again on its own device, against the+ // hash recorded at load - a path that wrote a slot after the seal is found here+ std::string rerr;+ int64_t bad_layer = -1;+ const auto rt0 = Clock::now();+ bool ok = nib_x.recheck(rerr, bad_layer);+ int64_t checked = nib_x.n;+ for (size_t i = 0; ok && i < nib_st.size(); ++i) {+ ok = nib_st[i].recheck(rerr, bad_layer);+ checked += nib_st[i].n;+ }+ if (!ok) {+ std::fprintf(stderr, "strata serve: STRATA_EXPERT_NIBBLE_RECHECK: %s\n", rerr.c_str());+ if (bad_layer >= 0) nib::fatal(bad_layer, "STRATA_EXPERT_NIBBLE_RECHECK", "a slot changed after the load");+ std::fflush(stderr);+ std::abort();+ }+ std::fprintf(stderr, "strata serve: STRATA_EXPERT_NIBBLE_RECHECK: %lld 4-bit slots hash as at load (%.0f ms)\n",+ (long long) checked, std::chrono::duration<double, std::milli>(Clock::now() - rt0).count());+ }static const bool state_hash = std::getenv("STRATA_STATE_HASH") != nullptr;if (state_hash && !cancelled && o.prompt_cache > 0) { // every finished request, ckpt=0 too (parity gates)// DEBUG: a fingerprint of every part of the session over the positions it holds ([0, L)), and@@ -15967,6 +16464,11 @@ int main(int argc, char** argv) {} else if (chunk != o.prefill_chunk) {k = 0; // a fixed chunk that does not fit: its own buffers, as before}+ if (k > 0 && nib::any_on()) { // STRATA_EXPERT_NIBBLE: the lent slots would be refilled with GGUF blobs+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE refuses the start: the prompt path would borrow %lld "+ "cache slots (--no-prefill-borrow keeps its own buffers)\n", (long long) k);+ return 1;+ }const int64_t blob = (int64_t) strata::kernels::cpu::expert_layout().max_blob;#if defined(STRATA_GPU_METAL)k = 0; // Metal: no loan of expert-cache slots (the prompt buffers are the Metal window's own); the refill after it copied into given-back allocations@@ -16555,6 +17057,7 @@ int main(int argc, char** argv) {const size_t in = (size_t) s.layer * g.n_expert + s.in, out = (size_t) s.layer * g.n_expert + s.out;const int32_t slot = host_res[out];const uint8_t* b = srcp->blob_stable(s.layer, s.in);+ nib::guard_raw(s.layer, "the adaptive refill's copy in");const size_t blob_n = (size_t) strata::kernels::cpu::expert_layout().blob_bytes(s.layer);// asynchronous: the copies run while the MTP drafts; the next window waits for themif (slot < 0 || b == nullptr) {