Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

iq_kernels: GP100 4-bit gate/up expert slots, opt-in STRATA_EXPERT_NIBBLE=1 (#3455)

ruibeikaa9067cc64d317
diff --git a/CMakeLists.txt b/CMakeLists.txt
index efb231216..60a25467f 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -2247,6 +2247,20 @@ if(STRATA_NATIVE_EXPERTS)
add_executable(native_expert_bench src/kernels/native_expert_bench.cpp)
target_include_directories(native_expert_bench PRIVATE ${_strata_gpu_include_directories})
target_link_libraries(native_expert_bench PRIVATE strata_kernels strata_kernels_cpu strata_artifact ${_strata_gpu_runtime_target})
+ if(STRATA_ENABLE_CUDA AND STRATA_EXPERIMENTAL_SM60)
+ # STRATA_EXPERT_NIBBLE (include/strata/core/expert_nibble.hpp): the 4-bit gate/up expert slots against the
+ # production kernels and the GGUF blobs, bit for bit, on a real model's layers (needs a GP100); --header: the
+ # contract's host half, no GPU
+ add_executable(iq_nibble_parity src/kernels/iq_nibble_parity.cpp)
+ target_include_directories(iq_nibble_parity PRIVATE ${_strata_gpu_include_directories})
+ target_link_libraries(iq_nibble_parity PRIVATE strata_kernels strata_kernels_cpu strata_artifact ggml-base ${_strata_gpu_runtime_target})
+ if(STRATA_BUILD_TESTS)
+ add_test(NAME iq_nibble_parity_header COMMAND iq_nibble_parity --header)
+ # the synthetic experts of every gate/up x down format pair, no model (skips with 77 on a card that is not a GP100)
+ add_test(NAME iq_nibble_parity_synthetic COMMAND iq_nibble_parity --synthetic-only)
+ set_tests_properties(iq_nibble_parity_synthetic PROPERTIES SKIP_RETURN_CODE 77 TIMEOUT 900)
+ endif()
+ endif()
# MODEL-UPDATE: the dense Q8_0 GEMVs at the engine's shapes against the memory bandwidth (a benchmark, no test)
add_executable(q8_gemv_bench src/kernels/q8_gemv_bench.cpp)
target_include_directories(q8_gemv_bench PRIVATE ${_strata_gpu_include_directories})
diff --git a/docs/OLDER_GPUS.md b/docs/OLDER_GPUS.md
index 58118fafc..88f9f9758 100644
--- a/docs/OLDER_GPUS.md
+++ b/docs/OLDER_GPUS.md
@@ -182,6 +182,50 @@ decode; keep two engines on different logical CPUs.
`tools/q8_sm60_check.cu` checks the kernel against a CPU reference on these shapes (1-8 columns).
+### Pascal GP100: 4-bit gate/up expert slots (opt-in, `STRATA_EXPERT_NIBBLE=1`)
+
+On a GP100 die the verify window's gate/up expert kernels wait on each block's serial chain (the weights, two codebook
+lookups per four weights, the emulated dp4a, the reduction), not on memory bandwidth. IQ2_S, IQ3_XXS and IQ3_S have
+only 3 or 8 magnitudes in their grids, so every weight is one of at most 16 signed values: `STRATA_EXPERT_NIBBLE=1`
+keeps those layers' gate/up experts in VRAM as 16 bytes of 4-bit codes per 32 weights (one 16-byte load), with the
+integer scale and the fp16 `d` of each block kept as they are, and decodes four weights in 1-4 integer instructions. The
+down rows stay GGUF blocks. The host copy, the pack and its tools do not change: the conversion runs on the GPU while
+the slots are filled. Only the CUDA build with `STRATA_EXPERIMENTAL_SM60` has the kernels.
+
+- **The answers are meant to be bit for bit the default's.** The 4-bit kernels run the production integer chain and float tail in
+ the production order (`native_gu_multi_kernel`'s sub-16 rows), and SwiGLU, the q8_1 step and the down projection are
+ the layout's own (mode 8 on a GP100). At every start each item is decoded with the production loader, encoded,
+ decoded from the written code and compared, and the GGUF blocks are rebuilt from every slot and compared with the
+ blob byte for byte; any difference refuses the start and names the layer. The prompt path's MMQ gathers the GGUF
+ blocks rebuilt from the slots (the copy it makes today, with the 4-bit codes read instead): the same bytes. The kernels that
+ existed before this change have the same SASS in the sm_60 build with and without it (446 kernels in `iq_kernels.cu` compared).
+ `iq_nibble_parity --synthetic-only` checks the codec, the rebuilt blocks and the expert step on random experts of every format
+ pair with no model file; `iq_nibble_parity <model shard>` does the same on a model's layers, against the GGUF blobs in mode 8 and
+ in the CUDA layout (`STRATA_SM60_TABLE=0`). Both need a GP100.
+- **It needs every expert in VRAM** on the card that runs its layer: a profile that ranks every pair, room for the
+ larger slots (per layer of 512 experts: +175 MiB for IQ3_S, +250 MiB for IQ3_XXS, +350 MiB for IQ2_S), and a prompt
+ path with its own buffers, no loan of cache slots (`--no-prefill-borrow` on one GPU; a layer split's stages keep
+ their own when their cards have room, #340). `STRATA_EXPERT_NIBBLE_SKIP=<layers>` keeps layers in their GGUF slots;
+ `STRATA_EXPERT_NIBBLE_FMTS=22,18,21` picks the formats.
+- **It does not start with what it cannot serve.** With the switch set, a build without `STRATA_EXPERIMENTAL_SM60`, a card other
+ than compute capability 6.0, `--peer-device`, `--expert-cache-device1..3`, `--kv-grow`, `--vram-elastic`, no `--expert-profile`,
+ `--expert-cache 0`, `STRATA_MMQ_DIRECT`, `STRATA_PREFILL_MMQ=0`, `STRATA_PF_FUSED=1`, `STRATA_FS_SLOTS`, `STRATA_EXPERT_V2=1` or
+ `STRATA_EXP_MODE` other than 0 or 8 make the engine say so in a warning and start with the usual GGUF slots. What is only known
+ later (a layer with an expert out of VRAM, a loan of cache slots, a prompt chunk other than the one the run without the switch
+ reads) still refuses the start with the reason. The adaptive tier is off with the switch. `STRATA_EXPERT_NIBBLE_RECHECK=1` (debug)
+ hashes every 4-bit slot again after each request.
+- **Opt-in `STRATA_NIB_GU_PROTO=1`** (with `STRATA_EXPERT_NIBBLE=1`): the 4-bit gate/up runs `nibx_gu_kernel`
+ (`nib_gu_proto.cuh`), the same row dot with the group's activations staged once per two row blocks by 16-byte
+ loads. The author measured, on one GP100 die, a layer's gate/up at 54-57 us instead of 67-72 us at one token (x1.24-1.27) and
+ x1.22-1.39 at two to four tokens.
+- The author measured the switch on a PH402 SKU 200 (two boards, four GP100 dies of 48 SMs and 32 GB each, clocks locked at 715 /
+ 1050 MHz), Windows 11, CUDA 12.9, engine 0.1.45.3, Swift 1.5 IQ3_S (abliterated) pack, layer split 12,24,36,
+ `--trim-stage-weights`, MTP with `--spec-min-p 0.85`, `--pipeline-windows 2`, 1M context, every expert resident: 47 of 48 layers
+ converted (+12.7 GiB of VRAM over the four dies, +3.1 s of start-up). Decode of eight prompts (8K, 26K and 88K-token contexts),
+ token-weighted: 47.3 -> 51.9 tok/s (+9.7%; each prompt +9.5..+10.9%). Prompt reading 2.3..3.0% slower (the gather rebuilds the
+ GGUF blocks). The replies of 15 requests, including a 260K-token read, matched the engine without the switch. We have not
+ measured it on this code.
+
## Tesla P40 (sm_61): what was measured and changed
Three things run differently on compute capability 6.1 by default and a fourth is opt-in. Each gives the same bits as what it
diff --git a/include/strata/core/expert_cache.hpp b/include/strata/core/expert_cache.hpp
index 9685aa4d5..7de956b99 100644
--- a/include/strata/core/expert_cache.hpp
+++ b/include/strata/core/expert_cache.hpp
@@ -217,6 +217,14 @@ public:
/// Slots filled so far, for the startup report.
int64_t fills() const { return fills_; }
+ /// STRATA_EXPERT_NIBBLE=1 (strata/core/expert_nibble.hpp): called once the startup fill has written every slot
+ /// (some of them as 4-bit codes, not GGUF blocks) and checked them. From then on no slot may be written through
+ /// this class: the fill_slot calls stop the engine (they would write GGUF bytes, possibly into a 4-bit slot, and a
+ /// wrong expert gives a plausible token, never an error), and `admit` claims nothing. Never called with the switch
+ /// off: an unsealed cache is the one above.
+ void seal() { sealed_ = true; }
+ bool sealed() const { return sealed_; }
+
/// The elastic K/V (--kv-grow): the arena in a VMM range (vmm.hpp) instead of one cudaMalloc, so the K/V can take
/// single chunks of it and give them back. Applies to the next `open`; ignored where VMM is not available.
static void set_vmm(bool enabled);
@@ -266,6 +274,7 @@ private:
std::vector<int32_t> layer_next_; ///< [n_layers] -> that layer's next free slot
std::vector<uint64_t> off_; ///< plan v0.3 P6: slot offsets (slots + 1 entries) when sized
int64_t admitted_ = 0;
+ bool sealed_ = false; ///< STRATA_EXPERT_NIBBLE: `seal` was called (no slot is written after it)
};
} // namespace strata::core
diff --git a/include/strata/core/expert_nibble.hpp b/include/strata/core/expert_nibble.hpp
new file mode 100644
index 000000000..920019567
--- /dev/null
+++ b/include/strata/core/expert_nibble.hpp
@@ -0,0 +1,457 @@
+// include/strata/core/expert_nibble.hpp - STRATA_EXPERT_NIBBLE=1 (opt-in, default off): the gate/up experts of
+// IQ2_S / IQ3_XXS / IQ3_S layers kept as 4-bit codes in their VRAM slots, for Pascal GP100 (sm_60).
+//
+// **WHY.** On a GP100 the decode window's gate/up expert kernels are not held by memory bandwidth: they use a small
+// part of it and wait on each block's serial chain (the weights, the codebook lookups, the emulated dp4a, the
+// reduction). Re-encoding each 32-weight item of these three formats as sixteen bytes of 4-bit codes (one aligned
+// 16-byte load) and keeping the integer scale and the fp16 d as they are makes the slot 1.25-1.68x the GGUF bytes and
+// the decode of four weights 1-4 integer instructions instead of two codebook lookups.
+//
+// **WHY IT IS LOSSLESS.** Their grids have only 3 (IQ2_S) or 8 (IQ3_XXS, IQ3_S) magnitudes, so each int8 weight the
+// production `Split<TG>::load` decodes is one of at most 16 signed values. At every start the transcoder decodes every
+// item of every converted slot with the production load, encodes it, decodes the written code again and compares; and
+// the GGUF blocks are rebuilt from every slot and compared with the blob byte for byte (`check_gguf`). The decode
+// kernels run the production integer chain and float tail in the production order: every gate / up value is the
+// production kernel's, bit for bit. The prompt path's MMQ reads the GGUF blocks rebuilt from the slot (`gather_gguf`,
+// in place of the copy it does today): the same bytes, so the same bits.
+//
+// **OFF IS TODAY.** Unset or 0: no layer is converted, `slot_bytes(l)` is `expert_layout().blob_bytes(l)`, nothing is
+// transcoded, sealed or hashed, and none of the new branches is entered. Only a CUDA build with
+// STRATA_EXPERIMENTAL_SM60 has the kernels (STRATA_EXPERT_NIBBLE_BUILT); every other build refuses the switch at start.
+#pragma once
+
+#include "strata/kernels/cpu/expert_layout.hpp"
+
+#include <cstddef>
+#include <cstdint>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <string>
+#include <vector>
+
+#if defined(STRATA_EXPERIMENTAL_SM60) && !defined(__HIPCC__) && !defined(STRATA_USE_HIP) && !defined(STRATA_HIP_GFX906) && \
+ !defined(STRATA_GPU_METAL)
+#define STRATA_EXPERT_NIBBLE_BUILT 1
+#else
+#define STRATA_EXPERT_NIBBLE_BUILT 0
+#endif
+
+#if defined(__CUDACC__) || defined(__HIPCC__)
+#define STRATA_NIB_HD __host__ __device__
+#else
+#define STRATA_NIB_HD
+#endif
+
+namespace strata::core::nibble {
+
+// ------------------------------------------------------------------------------------------------ the slot layout
+//
+// A converted layer's expert slot, in order (the host arena and the pack keep the GGUF blocks):
+//
+// codes kRows x kItems x 16 B row r, item k at (r * kItems + k) * 16; rows 0..kNFf-1 gate, kNFf..kRows-1 up
+// scales kRows x kItems x 1 B row r, item k at kScOff + r * kItems + k (the production scale step's input)
+// d kRows x kSuper x 2 B row r, superblock s at kDOff + (r * kSuper + s) * 2: the block's fp16 d, bit for bit
+// ------ kGuBytes = 1,766,400 B (6900 x 256: the down part starts 256-aligned)
+// down the GGUF down rows, copied verbatim from the raw blob's [down_off, bytes)
+// ------ slot_bytes = round_up(kGuBytes + down bytes, kSlotAlign)
+//
+// One item = 32 weights of a row, one 16-byte code (four 32-bit words, see `encode_item`). Only the expert shape
+// n_embd 2560, n_ff 640 is converted: every other layer keeps its GGUF blob and its kernels.
+
+inline constexpr int64_t kNEmbd = 2560; ///< the gate/up rows' width (the activation)
+inline constexpr int64_t kNFf = 640; ///< gate rows = up rows = n_ff
+inline constexpr int64_t kRows = 2 * kNFf; ///< 1280: gate, then up
+inline constexpr int64_t kItems = kNEmbd / 32; ///< 80 items of 32 weights per row
+inline constexpr int64_t kSuper = kNEmbd / 256; ///< 10 superblocks (one fp16 d each) per row
+inline constexpr uint64_t kItemBytes = 16; ///< one item's code: four words, one 16-byte load
+
+inline constexpr uint64_t kCodesOff = 0;
+inline constexpr uint64_t kUpCodesOff = (uint64_t) kNFf * kItems * kItemBytes; ///< 819,200: row kNFf
+inline constexpr uint64_t kScOff = (uint64_t) kRows * kItems * kItemBytes; ///< 1,638,400
+inline constexpr uint64_t kDOff = kScOff + (uint64_t) kRows * kItems; ///< 1,740,800
+inline constexpr uint64_t kGuBytes = kDOff + (uint64_t) kRows * kSuper * 2; ///< 1,766,400
+inline constexpr uint64_t kDownOff = kGuBytes; ///< the down rows
+inline constexpr uint64_t kSlotAlign = 256; ///< every slot's size
+
+static_assert(kGuBytes == 1766400, "the slot layout above");
+static_assert(kGuBytes % kSlotAlign == 0, "the down part starts 256-aligned");
+static_assert(kScOff % 16 == 0 && kDOff % 16 == 0, "each region starts 16-aligned");
+
+/// Byte offset of row `row` (0..kRows-1; up rows are kNFf + r) item `k`'s code inside the slot.
+STRATA_NIB_HD constexpr uint64_t code_off(int64_t row, int64_t k) {
+ return kCodesOff + ((uint64_t) row * kItems + (uint64_t) k) * kItemBytes;
+}
+/// Byte offset of row `row` item `k`'s scale byte.
+STRATA_NIB_HD constexpr uint64_t scale_off(int64_t row, int64_t k) { return kScOff + (uint64_t) row * kItems + (uint64_t) k; }
+/// Byte offset of row `row` superblock `s`'s fp16 d (s = k / 8 for item k).
+STRATA_NIB_HD constexpr uint64_t d_off(int64_t row, int64_t s) { return kDOff + ((uint64_t) row * kSuper + (uint64_t) s) * 2; }
+/// A converted slot's size for a raw blob whose down rows are `down_bytes` long.
+STRATA_NIB_HD constexpr uint64_t slot_bytes_for(uint64_t down_bytes) {
+ return (kGuBytes + down_bytes + kSlotAlign - 1) / kSlotAlign * kSlotAlign;
+}
+
+// ------------------------------------------------------------------------------------------------ the formats
+
+inline constexpr int kIQ3_XXS = 18;
+inline constexpr int kIQ3_S = 21;
+inline constexpr int kIQ2_S = 22;
+
+/// Whether `gu_type` is converted (the three formats whose weights take at most 16 values).
+STRATA_NIB_HD constexpr bool fmt_built(int gu_type) { return gu_type == kIQ2_S || gu_type == kIQ3_XXS || gu_type == kIQ3_S; }
+/// The GGUF gate/up row bytes of `gu_type` at kNEmbd (a converted layer's raw blob must agree; 0 = not a format here).
+STRATA_NIB_HD constexpr uint64_t raw_gu_row(int gu_type) {
+ return gu_type == kIQ2_S ? 820 : gu_type == kIQ3_XXS ? 980 : gu_type == kIQ3_S ? 1100 : 0;
+}
+inline const char* fmt_name(int gu_type) {
+ switch (gu_type) {
+ case kIQ3_XXS: return "IQ3_XXS";
+ case kIQ3_S: return "IQ3_S";
+ case kIQ2_S: return "IQ2_S";
+ default: return "?";
+ }
+}
+
+// ------------------------------------------------------------------------------------------------ the item code
+//
+// `g[32]` below is one item as the production `Split<TG>::load` leaves it: g[4 * j + b] is byte b of W.g[j]
+// (j = 0..7, the dp4a order against the q8_1 block's qs words). Word m (0..3) of the 16-byte code holds W.g[2m]
+// and W.g[2m + 1]:
+//
+// IQ2_S bits 4b..4b+3 = c(g[8m + b]), bits 16+4b.. = c(g[8m + 4 + b]), b = 0..3
+// c = magnitude index of {8, 25, 43} | sign << 2: the PRMT selector into {8,25,43,0} {-8,-25,-43,0}
+// IQ3_XXS same nibble positions; c = index of {4, 12, 20, 28, 36, 44, 52, 62} | sign << 3
+// IQ3_S byte b = u(g[8m + b]) | u(g[8m + 4 + b]) << 4, u = (v + 15) / 2 (v odd, -15..15); v = 2u - 15
+//
+// The scale byte keeps the production integer scale step's input:
+// IQ2_S the block's scales byte for the item: ls0 | ls1 << 4 (W.ls0, W.ls1)
+// IQ3_XXS aux32 >> 28 (0..15, W.ls)
+// IQ3_S 1 + 2 * nibble (1..31, W.ls)
+
+/// Encodes one item; false when a weight is outside the format's code set (the conversion then refuses to start).
+inline bool encode_item(int gu_type, const int8_t g[32], uint32_t q[4]) {
+ bool ok = true;
+ auto code = [&](int8_t v) -> uint32_t {
+ const int a = v < 0 ? -(int) v : (int) v;
+ if (gu_type == kIQ3_S) {
+ if ((v & 1) == 0 || v < -15 || v > 15) { ok = false; return 0; }
+ return (uint32_t) (((int) v + 15) / 2);
+ }
+ if (gu_type == kIQ3_XXS) {
+ static constexpr int mag[8] = { 4, 12, 20, 28, 36, 44, 52, 62 };
+ for (uint32_t i = 0; i < 8; ++i)
+ if (a == mag[i]) return i | (v < 0 ? 8u : 0u);
+ ok = false;
+ return 0;
+ }
+ const uint32_t m = a == 8 ? 0u : a == 25 ? 1u : a == 43 ? 2u : 3u; // IQ2_S
+ if (m == 3u) { ok = false; return 0; }
+ return m | (v < 0 ? 4u : 0u);
+ };
+ for (int m = 0; m < 4; ++m) {
+ uint32_t w = 0;
+ for (int b = 0; b < 4; ++b) {
+ const uint32_t c0 = code(g[8 * m + b]), c1 = code(g[8 * m + 4 + b]);
+ if (gu_type == kIQ3_S) w |= (c0 | c1 << 4) << (8 * b);
+ else w |= c0 << (4 * b) | c1 << (16 + 4 * b);
+ }
+ q[m] = w;
+ }
+ return ok;
+}
+
+/// Decodes one item's code back to the production W.g bytes (the host reference the parity program compares with;
+/// the device decoders give the same bytes). Codes outside the set decode as the PRMT does: IQ2_S index 3 -> 0.
+inline void decode_item(int gu_type, const uint32_t q[4], int8_t g[32]) {
+ static constexpr int8_t iq2[8] = { 8, 25, 43, 0, -8, -25, -43, 0 };
+ static constexpr int8_t iq3[8] = { 4, 12, 20, 28, 36, 44, 52, 62 };
+ for (int m = 0; m < 4; ++m)
+ for (int b = 0; b < 4; ++b) {
+ uint32_t c0, c1;
+ if (gu_type == kIQ3_S) {
+ c0 = q[m] >> (8 * b) & 15u;
+ c1 = q[m] >> (8 * b + 4) & 15u;
+ } else {
+ c0 = q[m] >> (4 * b) & 15u;
+ c1 = q[m] >> (16 + 4 * b) & 15u;
+ }
+ auto val = [&](uint32_t c) -> int8_t {
+ if (gu_type == kIQ3_S) return (int8_t) (2 * (int) c - 15);
+ if (gu_type == kIQ3_XXS) return (int8_t) ((c & 8u) ? -iq3[c & 7u] : iq3[c & 7u]);
+ return iq2[c & 7u];
+ };
+ g[8 * m + b] = val(c0);
+ g[8 * m + 4 + b] = val(c1);
+ }
+}
+
+// ------------------------------------------------------------------------------------------------ the slot hash
+//
+// STRATA_EXPERT_NIBBLE_RECHECK's fingerprint of a slot: over the slot's first `hashed` bytes (kGuBytes plus the down
+// rows; the alignment tail is not written and not hashed) as little-endian 32-bit words w_i, the last one
+// zero-extended, H = sum_i (uint64) (w_i ^ i) * kHashMul mod 2^64. A sum, so the device's order of adding does not
+// change it; the index term makes a moved word count.
+
+inline constexpr uint64_t kHashMul = 0x9E3779B97F4A7C15ull;
+
+/// The host reference of the slot hash (the device hash gives the same value for the same bytes).
+inline uint64_t hash_host(const void* p, uint64_t bytes) {
+ const uint8_t* s = (const uint8_t*) p;
+ uint64_t h = 0;
+ const uint64_t n = (bytes + 3) / 4;
+ for (uint64_t i = 0; i < n; ++i) {
+ uint32_t w = 0;
+ const uint64_t at = 4 * i, take = bytes - at < 4 ? bytes - at : 4;
+ std::memcpy(&w, s + at, (size_t) take); // little-endian hosts only
+ h += (uint64_t) (w ^ (uint32_t) i) * kHashMul;
+ }
+ return h;
+}
+
+// ------------------------------------------------------------------------------------------------ the switches
+
+struct Config {
+ bool on = false; ///< STRATA_EXPERT_NIBBLE=1
+ bool recheck = false; ///< STRATA_EXPERT_NIBBLE_RECHECK=1 (debug: every slot's hash per request)
+ std::vector<int> fmts{ kIQ2_S, kIQ3_XXS, kIQ3_S }; ///< STRATA_EXPERT_NIBBLE_FMTS (default 22,18,21)
+ std::vector<int64_t> skip; ///< STRATA_EXPERT_NIBBLE_SKIP: layers kept in their GGUF blocks
+ std::string error; ///< non-empty: a malformed value - the start is refused
+};
+
+namespace detail {
+/// "a,b,c" -> integers; false on anything else (empty = an empty list).
+inline bool parse_list(const char* s, std::vector<int64_t>& out) {
+ out.clear();
+ while (s != nullptr && *s != '\0') {
+ char* end = nullptr;
+ const long long v = std::strtoll(s, &end, 10);
+ if (end == s) return false;
+ out.push_back((int64_t) v);
+ s = end;
+ if (*s == ',') ++s;
+ else if (*s != '\0') return false;
+ }
+ return true;
+}
+} // namespace detail
+
+/// The switches, read once (generate.cpp reads them before anything is allocated).
+inline const Config& config() {
+ static const Config c = [] {
+ Config r;
+ const char* e = std::getenv("STRATA_EXPERT_NIBBLE");
+ if (e != nullptr && e[0] != '\0' && std::strcmp(e, "0") != 0) {
+ if (std::strcmp(e, "1") == 0) r.on = true;
+ else r.error = std::string("STRATA_EXPERT_NIBBLE=") + e + ": expected 0 or 1";
+ }
+ const char* rc = std::getenv("STRATA_EXPERT_NIBBLE_RECHECK");
+ r.recheck = rc != nullptr && std::strcmp(rc, "1") == 0;
+ std::vector<int64_t> v;
+ if (const char* f = std::getenv("STRATA_EXPERT_NIBBLE_FMTS"); f != nullptr && f[0] != '\0') {
+ if (!detail::parse_list(f, v)) {
+ if (r.error.empty()) r.error = std::string("STRATA_EXPERT_NIBBLE_FMTS=") + f + ": expected ggml types, e.g. 22,18,21";
+ } else {
+ r.fmts.clear();
+ for (const int64_t t : v) {
+ if (!fmt_built((int) t) && r.error.empty())
+ r.error = "STRATA_EXPERT_NIBBLE_FMTS: type " + std::to_string(t) +
+ " is not converted (22 IQ2_S, 18 IQ3_XXS, 21 IQ3_S)";
+ r.fmts.push_back((int) t);
+ }
+ }
+ }
+ if (const char* k = std::getenv("STRATA_EXPERT_NIBBLE_SKIP"); k != nullptr && k[0] != '\0') {
+ if (!detail::parse_list(k, r.skip) && r.error.empty())
+ r.error = std::string("STRATA_EXPERT_NIBBLE_SKIP=") + k + ": expected layer numbers, e.g. 40,44";
+ }
+ return r;
+ }();
+ return c;
+}
+
+/// With the switch on, the first environment setting it refuses, as a message (empty = none): the paths that would
+/// read a converted slot as GGUF blocks, write GGUF bytes into one, or run other kernels for the converted layers
+/// than the ones the 4-bit kernels are bit for bit. The command-line half (--peer-device, remote expert caches,
+/// --kv-grow, --vram-elastic), the cc 6.0 check, full residency and no loan are generate.cpp's checks; the prompt
+/// path's own (MMQ on every converted layer) is Prefill::nibble_prompt_check.
+inline std::string env_conflicts() {
+ auto set = [](const char* n) -> const char* {
+ const char* v = std::getenv(n);
+ return v != nullptr && v[0] != '\0' ? v : nullptr;
+ };
+ const char* v = set("STRATA_MMQ_DIRECT");
+ if (v != nullptr && std::atoi(v) != 0)
+ return "STRATA_MMQ_DIRECT (the prompt MMQ would read a converted slot in place as GGUF blocks)";
+ v = set("STRATA_PREFILL_MMQ");
+ if (v != nullptr && std::atoi(v) == 0) return "STRATA_PREFILL_MMQ=0 (the FP16 prompt path reads GGUF blocks)";
+ v = set("STRATA_PF_FUSED");
+ if (v != nullptr && v[0] == '1') return "STRATA_PF_FUSED=1 (the fused prompt kernels read GGUF blocks)";
+ v = set("STRATA_FS_SLOTS");
+ if (v != nullptr && std::atoi(v) > 0) return "STRATA_FS_SLOTS (Foresight swaps GGUF blobs into slots)";
+ v = set("STRATA_EXPERT_V2");
+ if (v != nullptr && v[0] == '1') return "STRATA_EXPERT_V2=1 (the S26 kernels, not the CUDA layout, on IQ3_S)";
+ v = set("STRATA_KV_GROW");
+ if (v != nullptr && v[0] != '0') return "STRATA_KV_GROW (the elastic K/V moves slots at their GGUF size)";
+ v = set("STRATA_EXP_MODE");
+ if (v != nullptr && std::atoi(v) != 0 && std::atoi(v) != 8)
+ return "STRATA_EXP_MODE other than 0 or 8 (another layout's gate/up kernels)";
+ return {};
+}
+
+// ------------------------------------------------------------------------------------------------ the VRAM table
+
+/// One layer's slots. `on` false: the GGUF blob, `slot_bytes == raw_bytes`, the production kernels.
+struct LayerVram {
+ bool on = false;
+ int gu_type = -1, d_type = -1;
+ uint64_t down_off = 0; ///< in the slot: kDownOff when on, else the raw blob's down_off
+ uint64_t down_bytes = 0; ///< the down rows (raw_bytes - raw_down_off)
+ uint64_t hashed = 0; ///< bytes the slot hash covers: kGuBytes + down_bytes when on, else 0
+ uint64_t slot_bytes = 0; ///< what the expert cache allocates per slot (`open_sized`, every estimate)
+ // the raw blob as the arena and the pack hold it (expert_layout().fmt[l]); the transcoder reads these
+ uint64_t raw_bytes = 0, raw_gu_row = 0, raw_up_off = 0, raw_down_off = 0;
+};
+
+/// One layer's entry from its GGUF format and the switches (pure; `plan_layers` calls it per layer).
+inline LayerVram plan_layer(int64_t l, const strata::kernels::cpu::NativeFmt& f, const Config& c) {
+ LayerVram v;
+ v.gu_type = f.gu_type;
+ v.d_type = f.d_type;
+ v.raw_bytes = f.bytes;
+ v.raw_gu_row = f.gu_row;
+ v.raw_up_off = f.up_off;
+ v.raw_down_off = f.down_off;
+ v.down_bytes = f.bytes - f.down_off;
+ v.down_off = f.down_off;
+ v.slot_bytes = f.bytes;
+ bool listed = false, skipped = false;
+ for (const int t : c.fmts) listed = listed || t == f.gu_type;
+ for (const int64_t s : c.skip) skipped = skipped || s == l;
+ // [gate | up | down] exactly, no NVFP4 tail, the down rows a whole number of 16-byte words (the gathers copy them so)
+ const bool shape = f.n_embd == kNEmbd && f.n_ff == kNFf && f.gu_row == raw_gu_row(f.gu_type) &&
+ f.up_off == (uint64_t) kNFf * f.gu_row && f.down_off == 2 * f.up_off && f.bytes > f.down_off &&
+ f.tail_off == 0 && (f.bytes - f.down_off) % 16 == 0;
+ if (c.on && c.error.empty() && listed && !skipped && fmt_built(f.gu_type) && shape) {
+ v.on = true;
+ v.down_off = kDownOff;
+ v.hashed = kGuBytes + v.down_bytes;
+ v.slot_bytes = slot_bytes_for(v.down_bytes);
+ }
+ return v;
+}
+
+/// Every layer's entry for this pack (all off for a Q2_0 pack or with the switch off). `err`: a SKIP layer out of
+/// range, or the switch on with a Q2_0 pack.
+inline std::vector<LayerVram> plan_layers(const strata::kernels::cpu::ExpertLayout& lay, std::string& err) {
+ const Config& c = config();
+ std::vector<LayerVram> out((size_t) (lay.n_layers > 0 ? lay.n_layers : 0));
+ for (int64_t l = 0; l < (int64_t) out.size(); ++l) {
+ if (lay.native && (size_t) l < lay.fmt.size()) out[(size_t) l] = plan_layer(l, lay.fmt[(size_t) l], c);
+ else out[(size_t) l].slot_bytes = out[(size_t) l].raw_bytes = lay.blob_bytes(l);
+ }
+ if (c.on && !lay.native) err = "STRATA_EXPERT_NIBBLE needs a native pack (native_experts.txt)";
+ for (const int64_t s : c.skip)
+ if (c.on && (s < 0 || s >= lay.n_layers) && err.empty())
+ err = "STRATA_EXPERT_NIBBLE_SKIP: layer " + std::to_string(s) + " is not in the model";
+ return out;
+}
+
+namespace detail {
+inline std::vector<LayerVram>& table() {
+ static std::vector<LayerVram> t;
+ return t;
+}
+} // namespace detail
+
+/// generate.cpp only: the table, once, after `expert_layout_load` and BEFORE the first slot estimate (placement
+/// search, can_start, the cache's auto size, every `open_sized`). Not thread-safe: set before any other thread.
+inline void set_layers(std::vector<LayerVram> v) { detail::table() = std::move(v); }
+
+/// Layer `l`'s entry (an off entry when the table is not set or `l` is outside it). Read-only after `set_layers`.
+inline const LayerVram& layer(int64_t l) {
+ static const LayerVram off;
+ const auto& t = detail::table();
+ return l >= 0 && (size_t) l < t.size() ? t[(size_t) l] : off;
+}
+
+/// Whether any layer is converted (the switch on and the table set).
+inline bool any_on() {
+ for (const LayerVram& v : detail::table())
+ if (v.on) return true;
+ return false;
+}
+
+/// Bytes of one of layer `l`'s VRAM slots: the converted size for a converted layer, else the layout's blob bytes
+/// (exactly `expert_layout().blob_bytes(l)`: with the switch off every caller sizes what it sized before).
+inline uint64_t slot_bytes(int64_t l) {
+ const auto& t = detail::table();
+ if (l >= 0 && (size_t) l < t.size()) return t[(size_t) l].slot_bytes;
+ return strata::kernels::cpu::expert_layout().blob_bytes(l);
+}
+
+// ------------------------------------------------------------------------------------------------ fatal paths
+
+/// A converted layer reached a path that would compute or copy it as GGUF blocks: say where and stop. Never a
+/// silent wrong answer.
+[[noreturn]] inline void fatal(int64_t l, const char* where, const char* why) {
+ std::fprintf(stderr, "strata: STRATA_EXPERT_NIBBLE: layer %lld (%s, 4-bit slots) reached %s: %s - stopping\n",
+ (long long) l, fmt_name(layer(l).gu_type), where != nullptr ? where : "?", why != nullptr ? why : "");
+ std::fflush(stderr);
+ std::abort();
+}
+
+/// Before each raw slot copy that bypasses the ExpertCache API (the adaptive tier's copies, the elastic K/V's slot
+/// moves): fatal when `l` is a converted layer. The startup refusals keep these paths off; this is the backstop
+/// (with `ExpertCache::seal()` for the API's own fills).
+inline void guard_raw(int64_t l, const char* where) {
+ if (layer(l).on) fatal(l, where, "a raw slot copy at the GGUF size");
+}
+
+// ------------------------------------------------------------------------------------------------ the decode kernels
+//
+// Two skeletons in the shape of native_gu_multi_kernel (16 lanes a row, a group's entries GRP_NC at a time), both bit
+// for bit the production gate/up: S stages the entries' q8_1 blocks in shared memory and decodes each item where it
+// is used (IQ3_S: v = 2u - 15 restored per byte, then the production apply); P loads the lane's codes, scales and d
+// before staging the qs words as 16-byte rows (IQ3_S: sum = 2 dot(u, x) - 15 sum(x), the same int32). Per format the
+// faster one on 4x GP100 at T = 1..6 (native_expert_bench --nibble): IQ2_S P, IQ3_XXS S, IQ3_S S.
+
+inline constexpr int kSkelS = 0, kSkelP = 1;
+STRATA_NIB_HD constexpr int gu_skeleton(int gu_type) { return gu_type == kIQ2_S ? kSkelP : kSkelS; }
+
+// ------------------------------------------------------------------------------------------------ iq_kernels.cu
+//
+// Device addresses are `unsigned long long` (as native_expert_grouped's grp_ptr), streams `void*`; each returns the
+// CUDA error code (0 = success) so the caller can name the layer. A build without STRATA_EXPERT_NIBBLE_BUILT
+// returns cudaErrorInvalidValue (1) from each.
+
+/// The reverse codebooks (grid magnitudes -> GGUF grid index) `check_gguf` and `gather_gguf` use, built on the CURRENT
+/// device (once per device; call it at load, before any capture). Non-zero also when a grid entry is not unique.
+int prepare(void* stream);
+
+/// Converts `n` raw blobs of layer `L` (device `staging`, `L.raw_bytes` apart, as the host arena holds them) into
+/// the converted slots at `slot_ptrs[0..n)` (a device array): codes, scales, d. The down rows are not touched (the
+/// caller copies them to `slot + kDownOff`). Every item is decoded with the production `Split<TG>::load`, encoded,
+/// decoded from the written code again and compared with W.g[0..7], the scale and d; each item that differs or holds
+/// a weight outside the code set adds 1 to `*bad` (device). L.on must be true.
+int transcode(const LayerVram& L, const void* staging, int64_t n, const unsigned long long* slot_ptrs,
+ unsigned long long* bad, void* stream);
+
+/// The GGUF gate/up blocks rebuilt from each of the `n` (<= 64) converted slots at `slot_ptrs` and its down rows,
+/// compared with the raw blobs in `staging` (`L.raw_bytes` apart) byte for byte: each 32-bit word that differs adds 1
+/// to `*bad`. `prepare` first.
+int check_gguf(const LayerVram& L, const void* staging, int64_t n, const unsigned long long* slot_ptrs,
+ unsigned long long* bad, void* stream);
+
+/// The prompt path's gather of a converted layer's experts (prefill.cpp, MMQ): for q in [first, n) (n <= 16), the
+/// GGUF [gate | up] blocks rebuilt from slot `slots[q]` (host array of device addresses) into gu_dst + q * gu_stride -
+/// the bytes the raw blob holds, byte for byte - and its down rows copied to d_dst + q * d_stride; `zero_bytes` zeroed
+/// after expert n - 1's gate/up and down (MMQ's tail). Every address, stride and size 16-byte aligned. `prepare`
+/// on this device first.
+int gather_gguf(const LayerVram& L, const uint8_t* const* slots, int first, int n, void* gu_dst, uint64_t gu_stride,
+ void* d_dst, uint64_t d_stride, uint64_t zero_bytes, void* stream);
+
+/// The slot hash (`hash_host`'s value) of the `L.hashed` bytes at each `slot_ptrs[i]`, into device `out[0..n)`.
+int hash_slots(const LayerVram& L, const unsigned long long* slot_ptrs, int64_t n, unsigned long long* out,
+ void* stream);
+
+} // namespace strata::core::nibble
diff --git a/include/strata/kernels/iq_kernels.hpp b/include/strata/kernels/iq_kernels.hpp
index ce06a9fe7..b1f5ae5ac 100644
--- a/include/strata/kernels/iq_kernels.hpp
+++ b/include/strata/kernels/iq_kernels.hpp
@@ -51,9 +51,16 @@ struct NativeExpertLayout {
// APR (prism.hadamard): the down weights are Hadamard-folded, so native_expert_grouped rotates h before
// quantizing it (hadamard_rows). had_block 0 = none; d_signs is a DEVICE vector of n_ff signs or nullptr.
int had_block = 0;
+ bool gu_nib = false; // STRATA_EXPERT_NIBBLE: gate/up as 4-bit codes (strata/core/expert_nibble.hpp's
+ // slot); set only by native_expert_layout_nib
const float* d_signs = nullptr;
};
NativeExpertLayout native_expert_layout(int gu_type, int d_type, int64_t n_embd, int64_t n_ff);
+/// STRATA_EXPERT_NIBBLE: a converted VRAM slot - gate/up as 4-bit codes, scales and d (expert_nibble.hpp's layout:
+/// up_off = the up rows' codes, down_off = kGuBytes, bytes = the slot), the GGUF down rows after them. IQ2_S /
+/// IQ3_XXS / IQ3_S at n_embd 2560, n_ff 640 only, and only in a build with the kernels (`native_expert_nib_supported`).
+NativeExpertLayout native_expert_layout_nib(int gu_type, int d_type, int64_t n_embd, int64_t n_ff);
+bool native_expert_nib_supported(int gu_type, int d_type, int64_t n_embd, int64_t n_ff) noexcept;
/// Whether `native_expert_grouped` has kernels for this gate/up and down type pair at these dimensions, and the
/// prompt path's dequantizer takes both (checked for every layer at startup, before anything is allocated).
bool native_expert_supported(int gu_type, int d_type, int64_t n_embd, int64_t n_ff) noexcept;
@@ -83,9 +90,15 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long
/// per possible group, SwiGLU and the q8_1 quantization as two kernels over all cap_entries. Bitwise the same results
/// (native_grouped_parity checks it); kept for A/B timing. Set before graph capture; captured graphs keep theirs.
void native_grouped_set_v1(bool v1);
+/// The bench / parity only: the 4-bit gate/up kernel of every format (-1 = expert_nibble.hpp's gu_skeleton, the
+/// default; 0 = the S skeleton; 1 = the P skeleton). Bitwise the same results; set before graph capture.
+void native_expert_nib_set_variant(int v);
+/// gu-proto (nib_gu_proto.cuh): variant v's registers, local bytes and resident 256-thread blocks per SM for gu_type;
+/// false when v is not a gu-proto id (>= 10) or the type has none.
+bool native_expert_nib_variant_attr(int v, int gu_type, int* regs, int* local_bytes, int* blocks_per_sm);
/// The bench only: the AMD kernel layout (STRATA_EXP_MODE values; -1 = the environment's) and the phase
-/// (0 all, 1 gate/up + SwiGLU + quantize, 2 down).
+/// (0 all, 1 gate/up + SwiGLU + quantize, 2 down, 3 gate/up only).
void native_expert_set_mode(int mode, int phase);
/// APR (prism.hadamard): y = H_block (signs * x) per block of `block` values of each of `rows` rows of `width`
diff --git a/include/strata/prefill/prefill.hpp b/include/strata/prefill/prefill.hpp
index 39ed85577..fa75c31d6 100644
--- a/include/strata/prefill/prefill.hpp
+++ b/include/strata/prefill/prefill.hpp
@@ -169,6 +169,11 @@ public:
/// build with the pointer-table MMQ (STRATA_MMQ_TAB).
static size_t slot_guard_bytes();
+ /// STRATA_EXPERT_NIBBLE=1: whether every converted layer's experts reach the prompt path's MMQ gather only (the
+ /// GGUF blocks rebuilt from the 4-bit slots); false with the first layer and the reason in `why`. Pure: the
+ /// layers' formats, the environment and the visible GPUs (generate.cpp asks once, before anything is allocated).
+ static bool nibble_prompt_check(std::string& why);
+
/// Positions [pos0, pos0 + n) holding `tokens`; `ss.ple_prev` must be the two tokens before pos0 (oldest
/// first, -1 for none) and is advanced to the last two of these.
bool run(const int64_t* tokens, int64_t n, int64_t pos0, std::string& err);
diff --git a/src/core/expert_cache.cpp b/src/core/expert_cache.cpp
index bd98ad711..72523b9cd 100644
--- a/src/core/expert_cache.cpp
+++ b/src/core/expert_cache.cpp
@@ -24,6 +24,17 @@
namespace strata::core {
+namespace {
+// STRATA_EXPERT_NIBBLE: a write into a sealed cache (see ExpertCache::seal). Says which call and stops: the slots hold
+// 4-bit codes the GGUF bytes would overwrite, or a GGUF expert where a 4-bit one is expected - a wrong reply, not an error.
+[[noreturn]] void sealed_write(const char* call, int32_t slot) {
+ std::fprintf(stderr, "strata: STRATA_EXPERT_NIBBLE: ExpertCache::%s(slot %d) after the cache was sealed - a GGUF "
+ "blob would be written into the 4-bit expert slots: stopping\n", call, (int) slot);
+ std::fflush(stderr);
+ std::abort();
+}
+} // namespace
+
// See the header. NVIDIA's DGX Spark Porting Guide (section 5.5, "Memory reporting on UMA systems") recommends the same:
// not to rely on cudaMemGetInfo alone but to count the memory the OS can reclaim. Swap is not counted here (unlike
// NVIDIA's reference snippet): an expert cache that pushes the system into swap would be far slower than a smaller one.
@@ -621,6 +632,7 @@ void ExpertCache::close() {
next_free_ = 0;
fills_ = 0;
admitted_ = 0;
+ sealed_ = false;
layer_next_.clear();
}
@@ -641,6 +653,7 @@ int32_t ExpertCache::slot_of(int64_t layer, int64_t expert) const {
}
int32_t ExpertCache::admit(int64_t layer, int64_t expert) {
+ if (sealed_) return kNotResident; // STRATA_EXPERT_NIBBLE: nothing is admitted (and so filled) after the startup fill
if (layer < 0 || layer >= n_layers_ || expert < 0 || expert >= n_expert_) return kNotResident;
const size_t at = (size_t) (layer * n_expert_ + expert);
if (residency_[at] != kNotResident) return residency_[at];
@@ -673,6 +686,7 @@ const uint8_t* ExpertCache::device_slot(int32_t slot) const {
}
bool ExpertCache::fill_slot(int32_t slot, const uint8_t* host_blob, void* stream, std::string& err, int64_t bytes) {
+ if (sealed_) sealed_write("fill_slot", slot);
const size_t n = (size_t) (bytes > 0 && bytes <= blob_ ? bytes : blob_);
uint8_t* dst = device_slot(slot);
if (dst == nullptr) {
@@ -695,6 +709,7 @@ bool ExpertCache::fill_slot(int32_t slot, const uint8_t* host_blob, void* stream
}
bool ExpertCache::fill_slot_blocking(int32_t slot, const uint8_t* host_blob, std::string& err, int64_t bytes) {
+ if (sealed_) sealed_write("fill_slot_blocking", slot);
const size_t n = (size_t) (bytes > 0 && bytes <= blob_ ? bytes : blob_);
uint8_t* dst = device_slot(slot);
if (dst == nullptr) {
@@ -726,6 +741,7 @@ bool ExpertCache::fill_slot_blocking(int32_t slot, const uint8_t* host_blob, std
}
bool ExpertCache::fill_slot_queued(int32_t slot, const uint8_t* host_blob, std::string& err, int64_t bytes) {
+ if (sealed_) sealed_write("fill_slot_queued", slot);
const size_t n = (size_t) (bytes > 0 && bytes <= blob_ ? bytes : blob_);
uint8_t* dst = device_slot(slot);
if (dst == nullptr || host_blob == nullptr) {
@@ -743,6 +759,7 @@ bool ExpertCache::fill_slot_queued(int32_t slot, const uint8_t* host_blob, std::
}
bool ExpertCache::fill_slot_queued_parts(int32_t slot, const BlobPart* parts, int n, std::string& err) {
+ if (sealed_) sealed_write("fill_slot_queued_parts", slot);
uint8_t* dst = device_slot(slot);
if (dst == nullptr || parts == nullptr || n <= 0) {
err = "ExpertCache::fill_slot_queued_parts: slot outside the arena or no pieces";
diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp
index b8ad8fb42..ec258c89a 100644
--- a/src/core/expert_source.cpp
+++ b/src/core/expert_source.cpp
@@ -1,5 +1,6 @@
// src/core/expert_source.cpp - the adapter. See the header for the three clauses of the contract.
#include "strata/core/expert_source.hpp"
+#include "strata/core/expert_nibble.hpp"
#include "strata/core/foresight_swap.hpp"
#include "strata/core/on_device.hpp"
#include "strata/platform/aux_cpus.hpp"
@@ -5221,6 +5222,14 @@ void expert_hit_run(void* user, void* stream, HitPhase phase, const int32_t* ids
d.hit_pending = false;
if (!d.hits_ready() || ids == nullptr || k <= 0) return;
if ((int64_t) d.is_hit.size() < k) d.is_hit.resize((size_t) k);
+ // STRATA_EXPERT_NIBBLE=1: this split computes a resident expert with the GGUF-block kernels and admits + fills
+ // a missed one with its GGUF blob - neither may meet a layer whose slots hold 4-bit codes (a native pack's
+ // experts run in the verify windows, so this is the backstop, never the path)
+#if STRATA_EXPERT_NIBBLE_BUILT // compiled out of every other build: nothing is added to the hit split there
+ if (strata::core::nibble::layer(d.layers).on)
+ strata::core::nibble::fatal(d.layers, "the expert source's hit split (expert_hit_run)",
+ "it computes resident experts as GGUF blocks and fills slots with GGUF blobs");
+#endif
// ================================ THE DECISION, ONCE, ON THIS LAYER'S IDS ================================
//
diff --git a/src/core/verify.cpp b/src/core/verify.cpp
index 06306a239..727fea0e1 100644
--- a/src/core/verify.cpp
+++ b/src/core/verify.cpp
@@ -4,6 +4,7 @@
#include "strata/core/sm120_series.hpp"
#include "strata/kernels/gfx_arch.hpp"
#include "strata/core/verify.hpp"
+#include "strata/core/expert_nibble.hpp"
#include "strata/core/pin_budget.hpp"
#if !defined(STRATA_USE_HIP) && !defined(STRATA_HIP_GFX906)
#include "strata/core/host_graph_launch.hpp"
@@ -2127,11 +2128,21 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {
const auto& lay = strata::kernels::cpu::expert_layout();
// plan v0.3 P6: the VRAM groups now; the PCIe groups once the copy engine has landed them in staging.
// `gy`: the native launch's groups side by side (0: cap, one block row per possible group).
- auto grouped = [&](const unsigned long long* gp, const int32_t* gs, const int32_t* gn, int64_t gy, float* dst_buf) {
+ // `vram`: the groups point into this GPU's expert slots - a layer STRATA_EXPERT_NIBBLE converted holds its
+ // gate/up there as 4-bit codes (expert_nibble.hpp); the PCIe share's blobs (host memory, staging) never do.
+ auto grouped = [&](const unsigned long long* gp, const int32_t* gs, const int32_t* gn, int64_t gy, float* dst_buf,
+ bool vram) {
if (lay.native) {
// the layer's GGUF formats (i-quant gate/up, Q2_0 / IQ4_NL down)
const auto& f = lay.fmt[(size_t) l];
+#if STRATA_EXPERT_NIBBLE_BUILT
+ NativeExpertLayout L = vram && strata::core::nibble::layer(l).on
+ ? native_expert_layout_nib(f.gu_type, f.d_type, f.n_embd, f.n_ff)
+ : native_expert_layout(f.gu_type, f.d_type, f.n_embd, f.n_ff);
+#else
+ (void) vram;
NativeExpertLayout L = native_expert_layout(f.gu_type, f.d_type, f.n_embd, f.n_ff);
+#endif
if (f.had_d) { L.had_block = f.had_block; L.d_signs = had_sh_; } // APR: h is rotated in there
native_expert_grouped(L, gp, gs, gn, p_dst, p_tok, cap, cap,
nat_xq_ + (size_t) tb * (N / 32) * 36, hit_scratch_, dst_buf, cs, gy);
@@ -2143,7 +2154,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {
bool hits_in_combine = false;
if (ar_on()) {
stamp(l, 19, grp);
- grouped(p_ptr, p_start, p_counts, 0, parts_out);
+ grouped(p_ptr, p_start, p_counts, 0, parts_out, true);
stamp(l, 20, grp);
} else {
if (device_plan_) { // E-6: skipped when the device planned this group (all its experts resident)
@@ -2161,7 +2172,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {
}
}
stamp(l, 19, grp);
- grouped(p_ptr, p_start, p_counts, 0, hit_out);
+ grouped(p_ptr, p_start, p_counts, 0, hit_out, true);
stamp(l, 20, grp);
if (device_plan_) wait_flag_ge_or(m_flagB_, ring, skip_ + grp, cs);
else if (m_bal_ != nullptr) // set_pcie_balance: the PCIe part starts
@@ -2176,7 +2187,7 @@ bool Verifier::record_window(int T, cudaStream_t cs, std::string& err) {
stamp(l, 21, grp);
// the PCIe share is pcie_frac of the misses: a few groups when the cache is cold, usually none (always none at
// pcie_frac 0), so its launch is kPcieGroupRows block rows striding over the groups, not cap of them
- grouped(p_ptr2, p_start2, p_counts + 2, kPcieGroupRows, hit_out);
+ grouped(p_ptr2, p_start2, p_counts + 2, kPcieGroupRows, hit_out, false);
stamp(l, 22, grp);
if (device_plan_) { // no CPU share when the device planned the group: its rows are zeros
wait_flag_ge_or(m_flag_, ring, skip_ + grp, cs);
diff --git a/src/kernels/cuda/iq_kernels.cu b/src/kernels/cuda/iq_kernels.cu
index 9cb6894fb..50f0a839f 100644
--- a/src/kernels/cuda/iq_kernels.cu
+++ b/src/kernels/cuda/iq_kernels.cu
@@ -5,6 +5,7 @@
// MIT license, third_party/ggml/LICENSE). The block structs and codebook grids come from its ggml-common.h,
// included unchanged.
#include "strata/kernels/iq_kernels.hpp"
+#include "strata/core/expert_nibble.hpp"
#include "strata/core/sm75_series.hpp"
#include "strata/core/sm120_series.hpp"
#include "strata/kernels/gfx_arch.hpp"
@@ -4476,6 +4477,704 @@ void launch_down_ws(const NativeExpertLayout& L, int64_t gy, int64_t cap_groups,
#endif // STRATA_DOWN_WS_BUILT
+// ---------------------------------------------------------------- STRATA_EXPERT_NIBBLE: gate/up on 4-bit slots
+// (opt-in, default off; strata/core/expert_nibble.hpp has the slot layout and the item code). A converted layer's
+// VRAM slot holds each 32-weight item of a gate/up row as 16 bytes of 4-bit codes (one 16-byte load), its integer
+// scale byte and the superblock's fp16 d bit for bit; the kernels below decode an item into the production
+// Split<TG>::W (or IQ3_S's offset form) with 1-4 instructions per four weights instead of the codebook lookups, then
+// run the production integer chain and float tail in the production lane / item / addition order (row_dot_80_sub16:
+// k = t, t+32, t+64 into s, t+16, t+48 into s16, __fadd_rn, xor 8, 4, 2, 1), so every gate / up value is bit for bit
+// native_gu_multi_kernel<TG, *, true>'s. Two skeletons in the shape of native_gu_multi_kernel (grid (80, gy), 256
+// threads, 16 lanes a row, a group's entries GRP_NC at a time, the group stride):
+// S the entries' q8_1 blocks staged in shared memory, each item's code loaded and decoded where it is used, the
+// production apply on it (IQ3_S: v = 2u - 15 restored per byte - no sum of x)
+// P the lane's five items' codes, scales and d loaded before the staging (in flight during it), the entries' qs
+// words staged as 16-byte rows and ds as a float, the production chain written out on the registers (IQ3_S:
+// sumi = 2 dot(u, x) - 15 sum(x) with sum(x) from the staged words: the same int32)
+// Then the load-time transcoder, the GGUF rebuild (the start-up check and the prompt path's gather) and the slot hash.
+// Built only with STRATA_EXPERT_NIBBLE_BUILT (a CUDA build with STRATA_EXPERIMENTAL_SM60); elsewhere a gu_nib layout
+// stops in native_expert_grouped, and the switch is refused at start.
+#if STRATA_EXPERT_NIBBLE_BUILT
+namespace nibk = strata::core::nibble;
+
+__device__ __forceinline__ uint32_t nib_prmt(uint32_t a, uint32_t b, uint32_t sel) {
+ uint32_t r;
+ asm("prmt.b32 %0, %1, %2, %3;" : "=r"(r) : "r"(a), "r"(b), "r"(sel)); // the sign-replicate modes too
+ return r;
+}
+
+// One item's code -> the production W (what Split<TG>::load gives for the same GGUF item, byte for byte).
+template<int TG> struct NibLoad;
+template<> struct NibLoad<22> { // IQ2_S: the nibble is the PRMT selector into {8,25,43,0} {-8,-25,-43,0}
+ using W = Split<22>::W;
+ __device__ static W load(uint4 q, uint32_t sc, __half d) {
+ constexpr uint32_t T0 = 0x002B1908u, T1 = 0x00D5E7F8u;
+ W r;
+ const uint32_t qw[4] = { q.x, q.y, q.z, q.w };
+#pragma unroll
+ for (int m = 0; m < 4; ++m) {
+ r.g[2 * m + 0] = (int) nib_prmt(T0, T1, qw[m]);
+ r.g[2 * m + 1] = (int) nib_prmt(T0, T1, qw[m] >> 16);
+ }
+ r.ls0 = sc & 0x0F;
+ r.ls1 = sc >> 4;
+ r.dw = __half2float(d);
+ return r;
+ }
+};
+template<> struct NibLoad<18> { // IQ3_XXS: magnitude PRMT into {4..62}, sign-replicated byte masks, (g ^ m) + (m & 1s)
+ using W = Split<18>::W;
+ __device__ static W load(uint4 q, uint32_t sc, __half d) {
+ constexpr uint32_t T0 = 0x1C140C04u, T1 = 0x3E342C24u; // {4,12,20,28} {36,44,52,62}
+ W r;
+ const uint32_t qw[4] = { q.x, q.y, q.z, q.w };
+#pragma unroll
+ for (int m = 0; m < 4; ++m) {
+ const uint32_t w = qw[m], ms = w & 0x77777777u, a = w << 4;
+ const uint32_t m0 = nib_prmt(a, w, 0xD9C8u), m1 = nib_prmt(a, w, 0xFBEAu);
+ const uint32_t g0 = nib_prmt(T0, T1, ms), g1 = nib_prmt(T0, T1, ms >> 16);
+ r.g[2 * m + 0] = (int) ((g0 ^ m0) + (m0 & 0x01010101u));
+ r.g[2 * m + 1] = (int) ((g1 ^ m1) + (m1 & 0x01010101u));
+ }
+ r.ls = (int) sc;
+ r.dw = __half2float(d);
+ return r;
+ }
+};
+template<> struct NibLoad<21> { // IQ3_S: v = 2u - 15 per byte, ((2u | 0x80) - 0x0F) ^ 0x80 (every byte >= 0x80: no borrow)
+ using W = Split<21>::W;
+ __device__ static W load(uint4 q, uint32_t sc, __half d) {
+ W r;
+ const uint32_t qw[4] = { q.x, q.y, q.z, q.w };
+#pragma unroll
+ for (int m = 0; m < 4; ++m) {
+ const uint32_t lo2 = (qw[m] << 1) & 0x1E1E1E1Eu, hi2 = (qw[m] >> 3) & 0x1E1E1E1Eu; // 2u per byte
+ r.g[2 * m + 0] = (int) (((lo2 | 0x80808080u) - 0x0F0F0F0Fu) ^ 0x80808080u);
+ r.g[2 * m + 1] = (int) (((hi2 | 0x80808080u) - 0x0F0F0F0Fu) ^ 0x80808080u);
+ }
+ r.ls = (int) sc;
+ r.dw = __half2float(d);
+ return r;
+ }
+};
+
+// S skeleton row dot over one converted row (codes, scales, d of row `row`), x the staged blocks
+template<int TG, int NC>
+__device__ __forceinline__ void nib_row_dot_s(const uint4* __restrict__ crow, const uint8_t* __restrict__ srow,
+ const __half* __restrict__ drow, const block_q8_1* x, const int (&off)[NC],
+ int t, float (&s)[NC]) {
+ using Q = NibLoad<TG>;
+ const int iqs = 2 * (t & 7), kb = t >> 3;
+ const block_q8_1* xb[NC];
+#pragma unroll
+ for (int c = 0; c < NC; ++c) { s[c] = 0.0f; xb[c] = x + off[c] + kb * 8; }
+#pragma unroll
+ for (int i = 0; i < 3; ++i) {
+ const typename Q::W w = Q::load(crow[t + 32 * i], srow[t + 32 * i], drow[kb + 4 * i]);
+#pragma unroll
+ for (int c = 0; c < NC; ++c) s[c] += Split<TG>::apply(w, xb[c] + 32 * i, iqs);
+ }
+ float s16[NC];
+#pragma unroll
+ for (int c = 0; c < NC; ++c) s16[c] = 0.0f;
+#pragma unroll
+ for (int i = 0; i < 2; ++i) {
+ const typename Q::W w = Q::load(crow[t + 16 + 32 * i], srow[t + 16 + 32 * i], drow[kb + 2 + 4 * i]);
+#pragma unroll
+ for (int c = 0; c < NC; ++c) s16[c] += Split<TG>::apply(w, xb[c] + 16 + 32 * i, iqs);
+ }
+#pragma unroll
+ for (int c = 0; c < NC; ++c) {
+ s[c] = __fadd_rn(s[c], s16[c]);
+#pragma unroll
+ for (int o = 8; o > 0; o >>= 1) s[c] += __shfl_xor_sync(0xffffffffu, s[c], o);
+ }
+}
+
+// One pass of the S skeleton: the n (1..GRP_NC) entries from e staged, then this thread's row against them.
+// Block-uniform (e, n): every thread reaches both barriers.
+template<int TG>
+__device__ __forceinline__ void nib_pass_s(block_q8_1* s_x, const uint4* crow, const uint8_t* srow, const __half* drow,
+ const int32_t* __restrict__ ent_tok, const block_q8_1* __restrict__ xq,
+ int e, int n, int t, float* dst, int r) {
+ int tok[GRP_NC];
+#pragma unroll
+ for (int c = 0; c < GRP_NC; ++c) tok[c] = ent_tok[e + min(c, n - 1)];
+ __syncthreads();
+ for (int i = threadIdx.x; i < n * 80 * 9; i += 256) { // 80 blocks x 36 B an entry, as 9 words a block
+ const int c = i / 720, rem = i % 720;
+ ((int*) s_x)[c * 720 + rem] = ((const int*) (xq + (size_t) tok[c] * 80))[rem];
+ }
+ __syncthreads();
+ if (n == 1) {
+ const int o[1] = { 0 }; float s[1];
+ nib_row_dot_s<TG, 1>(crow, srow, drow, s_x, o, t, s);
+ if (t == 0) dst[(size_t) e * nibk::kNFf + r] = s[0];
+ } else if (n == 2) {
+ const int o[2] = { 0, 80 }; float s[2];
+ nib_row_dot_s<TG, 2>(crow, srow, drow, s_x, o, t, s);
+ if (t < 2) dst[(size_t) (e + t) * nibk::kNFf + r] = s[t];
+ } else if (n == 3) {
+ const int o[3] = { 0, 80, 160 }; float s[3];
+ nib_row_dot_s<TG, 3>(crow, srow, drow, s_x, o, t, s);
+ if (t < 3) dst[(size_t) (e + t) * nibk::kNFf + r] = s[t];
+ } else {
+ const int o[4] = { 0, 80, 160, 240 }; float s[4];
+ nib_row_dot_s<TG, 4>(crow, srow, drow, s_x, o, t, s);
+ if (t < 4) dst[(size_t) (e + t) * nibk::kNFf + r] = s[t];
+ }
+}
+
+// P skeleton: x as two LDS.128 per item and entry, ds as the float __low2float gives
+struct NibXQ { int4 a, b; };
+
+template<int TG>
+__device__ __forceinline__ float nib_p_apply(const uint4& q, uint32_t sc, __half dh, const NibXQ& x, float dx, int xsum) {
+ if constexpr (TG == 22) { // SplitLs2::apply on the registers
+ const typename NibLoad<22>::W w = NibLoad<22>::load(q, sc, dh);
+ int sumi0 = 0, sumi1 = 0;
+ sumi0 = ggml_cuda_dp4a(w.g[0], x.a.x, sumi0); sumi0 = ggml_cuda_dp4a(w.g[1], x.a.y, sumi0);
+ sumi0 = ggml_cuda_dp4a(w.g[2], x.a.z, sumi0); sumi0 = ggml_cuda_dp4a(w.g[3], x.a.w, sumi0);
+ sumi1 = ggml_cuda_dp4a(w.g[4], x.b.x, sumi1); sumi1 = ggml_cuda_dp4a(w.g[5], x.b.y, sumi1);
+ sumi1 = ggml_cuda_dp4a(w.g[6], x.b.z, sumi1); sumi1 = ggml_cuda_dp4a(w.g[7], x.b.w, sumi1);
+ const int sumi = (sumi0 * w.ls0 + sumi1 * w.ls1 + (sumi0 + sumi1) / 2) / 4;
+ const float d = w.dw * dx;
+ return d * sumi;
+ } else if constexpr (TG == 18) { // Split<18>::apply on the registers
+ const typename NibLoad<18>::W w = NibLoad<18>::load(q, sc, dh);
+ int sumi = 0;
+ sumi = ggml_cuda_dp4a(w.g[0], x.a.x, sumi); sumi = ggml_cuda_dp4a(w.g[1], x.a.y, sumi);
+ sumi = ggml_cuda_dp4a(w.g[2], x.a.z, sumi); sumi = ggml_cuda_dp4a(w.g[3], x.a.w, sumi);
+ sumi = ggml_cuda_dp4a(w.g[4], x.b.x, sumi); sumi = ggml_cuda_dp4a(w.g[5], x.b.y, sumi);
+ sumi = ggml_cuda_dp4a(w.g[6], x.b.z, sumi); sumi = ggml_cuda_dp4a(w.g[7], x.b.w, sumi);
+ sumi = (w.ls * sumi + sumi / 2) / 2;
+ const float d = w.dw * dx;
+ return d * sumi;
+ } else { // IQ3_S, offset form: dot(v, x) = 2 dot(u, x) - 15 sum(x), exact (|2 dot(u, x)| < 2^17); then Split<21>'s
+ int u[8];
+ const uint32_t qw[4] = { q.x, q.y, q.z, q.w };
+#pragma unroll
+ for (int m = 0; m < 4; ++m) {
+ u[2 * m + 0] = (int) (qw[m] & 0x0F0F0F0Fu);
+ u[2 * m + 1] = (int) ((qw[m] >> 4) & 0x0F0F0F0Fu);
+ }
+ int sumu = 0;
+ sumu = ggml_cuda_dp4a(u[0], x.a.x, sumu); sumu = ggml_cuda_dp4a(u[1], x.a.y, sumu);
+ sumu = ggml_cuda_dp4a(u[2], x.a.z, sumu); sumu = ggml_cuda_dp4a(u[3], x.a.w, sumu);
+ sumu = ggml_cuda_dp4a(u[4], x.b.x, sumu); sumu = ggml_cuda_dp4a(u[5], x.b.y, sumu);
+ sumu = ggml_cuda_dp4a(u[6], x.b.z, sumu); sumu = ggml_cuda_dp4a(u[7], x.b.w, sumu);
+ int sumi = 2 * sumu - 15 * xsum;
+ sumi *= (int) sc;
+ const float d = __half2float(dh) * dx;
+ return d * sumi;
+ }
+}
+
+// the lane's items in the production order: k = t, t+32, t+64 (sum s), then t+16, t+48 (sum s16)
+template<int TG, int NC>
+__device__ __forceinline__ void nib_row_dot_p(const uint4 (&q)[5], const uint32_t (&sc)[5], const __half (&dh)[5],
+ const NibXQ* __restrict__ sx, const float* __restrict__ sd,
+ const int* __restrict__ sxs, int t, float (&s)[NC]) {
+ constexpr int K[5] = { 0, 32, 64, 16, 48 };
+#pragma unroll
+ for (int c = 0; c < NC; ++c) s[c] = 0.0f;
+#pragma unroll
+ for (int i = 0; i < 3; ++i) {
+ const int k = t + K[i];
+#pragma unroll
+ for (int c = 0; c < NC; ++c)
+ s[c] += nib_p_apply<TG>(q[i], sc[i], dh[i], sx[c * 80 + k], sd[c * 80 + k], TG == 21 ? sxs[c * 80 + k] : 0);
+ }
+ float s16[NC];
+#pragma unroll
+ for (int c = 0; c < NC; ++c) s16[c] = 0.0f;
+#pragma unroll
+ for (int i = 3; i < 5; ++i) {
+ const int k = t + K[i];
+#pragma unroll
+ for (int c = 0; c < NC; ++c)
+ s16[c] += nib_p_apply<TG>(q[i], sc[i], dh[i], sx[c * 80 + k], sd[c * 80 + k], TG == 21 ? sxs[c * 80 + k] : 0);
+ }
+#pragma unroll
+ for (int c = 0; c < NC; ++c) {
+ s[c] = __fadd_rn(s[c], s16[c]);
+#pragma unroll
+ for (int o = 8; o > 0; o >>= 1) s[c] += __shfl_xor_sync(0xffffffffu, s[c], o);
+ }
+}
+
+// The P skeleton's work on one group: this lane's five items into registers, then the passes. Block-uniform.
+template<int TG>
+__device__ __forceinline__ void nib_group_p(NibXQ* s_x, float* s_d, int* s_xs, const uint4* crow, const uint8_t* srow,
+ const __half* drow, const int32_t* __restrict__ ent_tok,
+ const block_q8_1* __restrict__ xq, int e0, int e1, int t, float* dst, int r) {
+ const int kb = t >> 3;
+ uint4 q[5]; uint32_t sc[5]; __half dh[5];
+ q[0] = crow[t]; q[1] = crow[t + 32]; q[2] = crow[t + 64]; q[3] = crow[t + 16]; q[4] = crow[t + 48];
+ sc[0] = srow[t]; sc[1] = srow[t + 32]; sc[2] = srow[t + 64]; sc[3] = srow[t + 16]; sc[4] = srow[t + 48];
+ dh[0] = drow[kb]; dh[1] = drow[kb + 4]; dh[2] = drow[kb + 8]; dh[3] = drow[kb + 2]; dh[4] = drow[kb + 6];
+ for (int e = e0; e < e1; e += GRP_NC) {
+ const int n = min(GRP_NC, e1 - e);
+ __syncthreads();
+ for (int i = threadIdx.x; i < n * 80; i += 256) {
+ const int c = i >= 240 ? 3 : i >= 160 ? 2 : i >= 80 ? 1 : 0, kk = i - 80 * c;
+ const int tok = ent_tok[e + c];
+ const int* src = (const int*) (xq + (size_t) tok * 80 + kk);
+ NibXQ v;
+ v.a = make_int4(src[1], src[2], src[3], src[4]);
+ v.b = make_int4(src[5], src[6], src[7], src[8]);
+ s_x[i] = v;
+ s_d[i] = __low2float(xq[(size_t) tok * 80 + kk].ds);
+ if constexpr (TG == 21) { // sum(x): the production dp4a against 0x01010101 over the 8 qs words
+ int xs = 0;
+ xs = ggml_cuda_dp4a(v.a.x, 0x01010101, xs); xs = ggml_cuda_dp4a(v.a.y, 0x01010101, xs);
+ xs = ggml_cuda_dp4a(v.a.z, 0x01010101, xs); xs = ggml_cuda_dp4a(v.a.w, 0x01010101, xs);
+ xs = ggml_cuda_dp4a(v.b.x, 0x01010101, xs); xs = ggml_cuda_dp4a(v.b.y, 0x01010101, xs);
+ xs = ggml_cuda_dp4a(v.b.z, 0x01010101, xs); xs = ggml_cuda_dp4a(v.b.w, 0x01010101, xs);
+ s_xs[i] = xs;
+ }
+ }
+ __syncthreads();
+ if (n == 1) {
+ float s[1]; nib_row_dot_p<TG, 1>(q, sc, dh, s_x, s_d, s_xs, t, s);
+ if (t == 0) dst[(size_t) e * nibk::kNFf + r] = s[0];
+ } else if (n == 2) {
+ float s[2]; nib_row_dot_p<TG, 2>(q, sc, dh, s_x, s_d, s_xs, t, s);
+ if (t < 2) dst[(size_t) (e + t) * nibk::kNFf + r] = s[t];
+ } else if (n == 3) {
+ float s[3]; nib_row_dot_p<TG, 3>(q, sc, dh, s_x, s_d, s_xs, t, s);
+ if (t < 3) dst[(size_t) (e + t) * nibk::kNFf + r] = s[t];
+ } else {
+ float s[4]; nib_row_dot_p<TG, 4>(q, sc, dh, s_x, s_d, s_xs, t, s);
+ if (t < 4) dst[(size_t) (e + t) * nibk::kNFf + r] = s[t];
+ }
+ }
+}
+
+// SKEL kSkelS or kSkelP for every group. grid (80, gy): 2 x 640 rows = 80 x 16, so every thread has a row and
+// reaches every barrier.
+constexpr int kNibSWords = GRP_NC * 80 * (int) (sizeof(block_q8_1) / 4); // S: the blocks
+constexpr int kNibXqWords = GRP_NC * 80 * (int) (sizeof(NibXQ) / 4); // P: the qs rows, then ds, then sum(x)
+template<int TG, int SKEL>
+inline constexpr int kNibSmemWords = SKEL == nibk::kSkelS ? kNibSWords : kNibXqWords + GRP_NC * 80 + (TG == 21 ? GRP_NC * 80 : 0);
+template<int TG, int SKEL>
+__global__ void __launch_bounds__(256) nib_gu_kernel(const unsigned long long* __restrict__ grp_ptr,
+ const int32_t* __restrict__ grp_start,
+ const int32_t* __restrict__ n_groups,
+ const int32_t* __restrict__ ent_tok,
+ const block_q8_1* __restrict__ xq, float* __restrict__ gate,
+ float* __restrict__ up) {
+ const int ng = *n_groups;
+ if (blockIdx.y >= (unsigned) ng) return;
+ STRATA_SHARED_ALIGN16_U32(s_buf, kNibSmemWords<TG, SKEL>);
+ const int subwarp = threadIdx.x >> 4, t = threadIdx.x & 15;
+ const int row = blockIdx.x * 16 + subwarp; // 0 .. 2 n_ff
+ const bool is_up = row >= nibk::kNFf;
+ const int r = is_up ? row - (int) nibk::kNFf : row;
+ float* dst = is_up ? up : gate;
+ for (int g = blockIdx.y; g < ng; g += gridDim.y) {
+ const uint8_t* slot = (const uint8_t*) grp_ptr[g];
+ const uint4* crow = (const uint4*) (slot + nibk::kCodesOff) + (size_t) row * nibk::kItems;
+ const uint8_t* srow = slot + nibk::kScOff + (size_t) row * nibk::kItems;
+ const __half* drow = (const __half*) (slot + nibk::kDOff) + (size_t) row * nibk::kSuper;
+ const int e0 = grp_start[g], e1 = grp_start[g + 1];
+ if constexpr (SKEL == nibk::kSkelP) {
+ nib_group_p<TG>((NibXQ*) s_buf, (float*) (s_buf + kNibXqWords), (int*) (s_buf + kNibXqWords + GRP_NC * 80),
+ crow, srow, drow, ent_tok, xq, e0, e1, t, dst, r);
+ } else {
+ for (int e = e0; e < e1; e += GRP_NC)
+ nib_pass_s<TG>((block_q8_1*) s_buf, crow, srow, drow, ent_tok, xq, e, min(GRP_NC, e1 - e), t, dst, r);
+ }
+ }
+}
+
+#include "nib_gu_proto.cuh" // STRATA_NIB_GU_PROTO: the staged-once 4-bit gate/up kernel (id 31)
+
+int g_nib_variant = -1; // native_expert_nib_set_variant (the bench / parity); -1 = expert_nibble.hpp's gu_skeleton
+
+// STRATA_NIB_GU_PROTO: unset / 0 = off (the S / P skeleton per format); 1 (or 31) = gu-proto 31 (the activations staged
+// once per two row blocks with 16-byte loads, the production row dot). Read once.
+int nib_gu_proto_env() {
+ static const int v = [] {
+ const char* e = std::getenv("STRATA_NIB_GU_PROTO");
+ int id = e != nullptr ? std::atoi(e) : 0;
+ if (id == 1) id = 31;
+ if (id != 31) id = 0;
+ if (id != 0) std::fprintf(stderr, "native_expert_grouped: STRATA_NIB_GU_PROTO: 4-bit gate/up kernel gu-proto %d\n", id);
+ return id;
+ }();
+ return v;
+}
+
+// The gate/up of a converted layer's VRAM groups: native_gu_multi_kernel's grid and outputs.
+void launch_gu_nib(const NativeExpertLayout& L, dim3 grid, cudaStream_t s, const unsigned long long* grp_ptr,
+ const int32_t* grp_start, const int32_t* n_groups, const int32_t* ent_tok, const block_q8_1* X,
+ float* gate, float* up) {
+ if (g_nib_variant >= 10) { // gu-proto (nib_gu_proto.cuh), forced by the bench / parity
+ if (((uintptr_t) X & 15) != 0 ||
+ !nibx_launch(L.gu_type, g_nib_variant, grid, s, grp_ptr, grp_start, n_groups, ent_tok, X, gate, up)) {
+ std::fprintf(stderr, "native_expert_grouped: gu-proto variant %d for type %d is not built (or x is not 16-aligned)\n",
+ g_nib_variant, L.gu_type);
+ std::exit(1);
+ }
+ return;
+ }
+ // STRATA_NIB_GU_PROTO (opt-in, default off): the gu-proto kernel in place of the S / P skeleton - bitwise the
+ // same gate / up (iq_nibble_parity (7), native_expert_bench --nibble --vs). The 16-byte staging needs x 16-aligned
+ // (verify's nat_xq_ rows are: a 256-aligned buffer, 2,880 B a token); otherwise the production kernel runs.
+ if (g_nib_variant < 0) {
+ const int pv = nib_gu_proto_env();
+ if (pv != 0 && ((uintptr_t) X & 15) == 0 &&
+ nibx_launch(L.gu_type, pv, grid, s, grp_ptr, grp_start, n_groups, ent_tok, X, gate, up))
+ return;
+ }
+ const int skel = g_nib_variant >= 0 ? (g_nib_variant == 1 ? nibk::kSkelP : nibk::kSkelS) : nibk::gu_skeleton(L.gu_type);
+#define STRATA_NIB_GU(TG, S) \
+ case TG * 2 + S: nib_gu_kernel<TG, S><<<grid, 256, 0, s>>>(grp_ptr, grp_start, n_groups, ent_tok, X, gate, up); break;
+ switch (L.gu_type * 2 + skel) {
+ STRATA_NIB_GU(22, 0) STRATA_NIB_GU(22, 1)
+ STRATA_NIB_GU(18, 0) STRATA_NIB_GU(18, 1)
+ STRATA_NIB_GU(21, 0) STRATA_NIB_GU(21, 1)
+ default:
+ std::fprintf(stderr, "native_expert_grouped: 4-bit gate/up type %d at %lld x %lld is not built\n", L.gu_type,
+ (long long) L.n_embd, (long long) L.n_ff);
+ std::exit(1);
+ }
+#undef STRATA_NIB_GU
+}
+
+// ---- the load-time transcoder (expert_nibble.hpp's transcode)
+template<int TG>
+__device__ __forceinline__ uint32_t nib_code(int v, bool& ok) {
+ if constexpr (TG == 21) { // v odd in -15..15 -> u = (v + 15) / 2
+ if ((v & 1) == 0 || v < -15 || v > 15) { ok = false; return 0; }
+ return (uint32_t) ((v + 15) >> 1);
+ } else if constexpr (TG == 18) { // index of {4, 12, ..., 52, 62} | sign << 3
+ const int a = v < 0 ? -v : v;
+ const int i = a == 62 ? 7 : ((a & 7) == 4 && a <= 52) ? (a - 4) >> 3 : -1;
+ if (i < 0) { ok = false; return 0; }
+ return (uint32_t) i | (v < 0 ? 8u : 0u);
+ } else { // IQ2_S: index of {8, 25, 43} | sign << 2
+ const int a = v < 0 ? -v : v;
+ const int i = a == 8 ? 0 : a == 25 ? 1 : a == 43 ? 2 : -1;
+ if (i < 0) { ok = false; return 0; }
+ return (uint32_t) i | (v < 0 ? 4u : 0u);
+ }
+}
+
+// The GGUF blocks (ggml-common.h): their size and the fp16 d at their start
+template<int TG> struct NibBlk;
+template<> struct NibBlk<22> { using B = block_iq2_s; };
+template<> struct NibBlk<18> { using B = block_iq3_xxs; };
+template<> struct NibBlk<21> { using B = block_iq3_s; };
+
+// One thread per (expert, row, item): the production load, the code, scale and d written, read back from memory and
+// decoded with the kernels' own decoders, compared with the load's W (and, for IQ3_S, the P skeleton's u against v);
+// `bad` counts the items that differ or hold a weight outside the code set.
+template<int TG>
+__global__ void __launch_bounds__(256) nib_transcode_kernel(const uint8_t* __restrict__ src, uint64_t raw_bytes,
+ uint64_t gu_row, uint64_t up_off,
+ const unsigned long long* __restrict__ slots, long long n,
+ unsigned long long* __restrict__ bad) {
+ const long long i = (long long) blockIdx.x * blockDim.x + threadIdx.x;
+ if (i >= n * nibk::kRows * nibk::kItems) return;
+ const int k = (int) (i % nibk::kItems);
+ const int row = (int) ((i / nibk::kItems) % nibk::kRows);
+ const long long ex = i / (nibk::kItems * nibk::kRows);
+ const bool is_up = row >= nibk::kNFf;
+ const int r = is_up ? row - (int) nibk::kNFf : row;
+ const uint8_t* wr = src + (size_t) ex * raw_bytes + (is_up ? up_off : 0) + (size_t) r * gu_row;
+ const int kbx = k / Fmt<TG>::ipb, iqs = Fmt<TG>::step * (k % Fmt<TG>::ipb);
+ using S = Split<TG>;
+ const typename S::W w = S::template load<false>(wr, kbx, iqs, nullptr);
+ bool ok = true;
+ uint32_t words[4];
+#pragma unroll
+ for (int m = 0; m < 4; ++m) {
+ const uint32_t g0 = (uint32_t) w.g[2 * m], g1 = (uint32_t) w.g[2 * m + 1];
+ uint32_t word = 0;
+#pragma unroll
+ for (int b = 0; b < 4; ++b) {
+ const uint32_t c0 = nib_code<TG>((int) (int8_t) (g0 >> (8 * b)), ok);
+ const uint32_t c1 = nib_code<TG>((int) (int8_t) (g1 >> (8 * b)), ok);
+ if constexpr (TG == 21) word |= (c0 | c1 << 4) << (8 * b);
+ else word |= c0 << (4 * b) | c1 << (16 + 4 * b);
+ }
+ words[m] = word;
+ }
+ uint32_t sc;
+ if constexpr (TG == 22) sc = (uint32_t) (w.ls0 | w.ls1 << 4);
+ else sc = (uint32_t) w.ls;
+ ok = ok && sc < 256u;
+ uint8_t* slot = (uint8_t*) slots[ex];
+ uint4* cp = (uint4*) (slot + nibk::code_off(row, k));
+ uint8_t* sp = slot + nibk::scale_off(row, k);
+ *cp = make_uint4(words[0], words[1], words[2], words[3]);
+ *sp = (uint8_t) sc;
+ const __half dsrc = ((const typename NibBlk<TG>::B*) wr + kbx)->d; // the block's d, bit for bit
+ __half dread = dsrc;
+ if (k % Fmt<TG>::ipb == 0) { // the superblock's first item writes its d
+ __half* dp = (__half*) (slot + nibk::d_off(row, kbx));
+ *dp = dsrc;
+ dread = __ushort_as_half(*(volatile const unsigned short*) dp);
+ ok = ok && __half_as_ushort(dread) == __half_as_ushort(dsrc);
+ }
+ // the code and the scale read back from memory and decoded as the kernels do
+ const uint4 qb = __ldcv(cp);
+ const uint32_t sb = *(volatile const uint8_t*) sp;
+ const typename NibLoad<TG>::W v = NibLoad<TG>::load(qb, sb, dread);
+#pragma unroll
+ for (int j = 0; j < 8; ++j) ok = ok && v.g[j] == w.g[j];
+ if constexpr (TG == 22) ok = ok && v.ls0 == w.ls0 && v.ls1 == w.ls1;
+ else ok = ok && v.ls == w.ls;
+ ok = ok && __float_as_uint(v.dw) == __float_as_uint(w.dw);
+ if constexpr (TG == 21) { // the P skeleton's u: 2u - 15 per byte is v
+ const uint32_t qw[4] = { qb.x, qb.y, qb.z, qb.w };
+#pragma unroll
+ for (int m = 0; m < 4; ++m) {
+ const uint32_t u0 = qw[m] & 0x0F0F0F0Fu, u1 = (qw[m] >> 4) & 0x0F0F0F0Fu;
+ ok = ok && __vsub4(__vadd4(u0, u0), 0x0F0F0F0Fu) == (uint32_t) w.g[2 * m] &&
+ __vsub4(__vadd4(u1, u1), 0x0F0F0F0Fu) == (uint32_t) w.g[2 * m + 1];
+ }
+ }
+ if (!ok) atomicAdd(bad, 1ull);
+}
+
+// ---- the GGUF rebuild (expert_nibble.hpp's check_gguf and gather_gguf)
+// The reverse codebooks: a group's magnitude pattern -> its grid index (0xFFFFFFFF: not in the grid). 32-bit
+// entries for atomicCAS, which also counts a pattern two grid entries share (none in ggml's grids; then refused).
+constexpr int kNibRev22 = 6561, kNibRev18 = 4096, kNibRev21 = 4096; // 3^8 (IQ2_S: 8 weights), 8^4, 8^4 (4 weights)
+__device__ uint32_t g_nib_rev22[kNibRev22];
+__device__ uint32_t g_nib_rev18[kNibRev18];
+__device__ uint32_t g_nib_rev21[kNibRev21];
+__device__ unsigned long long g_nib_rev_bad;
+
+__device__ __forceinline__ int nib_mag22(uint32_t b) { return b == 8 ? 0 : b == 25 ? 1 : b == 43 ? 2 : -1; }
+__device__ __forceinline__ int nib_mag18(uint32_t b) { return b == 62 ? 7 : ((b & 7) == 4 && b <= 52) ? (int) (b - 4) >> 3 : -1; }
+__device__ __forceinline__ int nib_mag21(uint32_t b) { return (b & 1) && b <= 15 ? (int) (b - 1) >> 1 : -1; }
+
+__global__ void nib_rev_clear_kernel() {
+ const int i = blockIdx.x * blockDim.x + threadIdx.x;
+ if (i < kNibRev22) g_nib_rev22[i] = 0xFFFFFFFFu;
+ if (i < kNibRev18) g_nib_rev18[i] = 0xFFFFFFFFu;
+ if (i < kNibRev21) g_nib_rev21[i] = 0xFFFFFFFFu;
+ if (i == 0) g_nib_rev_bad = 0;
+}
+__global__ void nib_rev_fill_kernel() {
+ const int i = blockIdx.x * blockDim.x + threadIdx.x;
+ uint32_t* tab = nullptr;
+ int key = 0;
+ bool ok = true;
+ uint32_t idx = 0;
+ if (i < 1024) { // iq2s_grid: 8 bytes of {8, 25, 43}
+ const uint64_t g = iq2s_grid[i];
+ for (int j = 7; j >= 0; --j) { const int m = nib_mag22((uint32_t) (g >> (8 * j)) & 0xFF); ok = ok && m >= 0; key = key * 3 + (m < 0 ? 0 : m); }
+ tab = g_nib_rev22; idx = (uint32_t) i;
+ } else if (i < 1024 + 256) { // iq3xxs_grid: 4 bytes of {4, ..., 62}
+ const uint32_t g = iq3xxs_grid[i - 1024];
+ for (int j = 3; j >= 0; --j) { const int m = nib_mag18((g >> (8 * j)) & 0xFF); ok = ok && m >= 0; key = key * 8 + (m < 0 ? 0 : m); }
+ tab = g_nib_rev18; idx = (uint32_t) (i - 1024);
+ } else if (i < 1024 + 256 + 512) { // iq3s_grid: 4 bytes of {1, 3, ..., 15}
+ const uint32_t g = iq3s_grid[i - 1280];
+ for (int j = 3; j >= 0; --j) { const int m = nib_mag21((g >> (8 * j)) & 0xFF); ok = ok && m >= 0; key = key * 8 + (m < 0 ? 0 : m); }
+ tab = g_nib_rev21; idx = (uint32_t) (i - 1280);
+ } else {
+ return;
+ }
+ if (!ok || atomicCAS(tab + key, 0xFFFFFFFFu, idx) != 0xFFFFFFFFu) atomicAdd(&g_nib_rev_bad, 1ull);
+}
+
+// The bytes of GGUF block `s` of row `row`'s item `ib` (0..7) rebuilt from the slot into `blk` (shared memory, the
+// block's first byte): the codes give each weight's magnitude index and sign, the reverse codebook the grid index,
+// the scale byte and d are the block's own. Disjoint bytes per item (IQ3_S: the even item writes the shared scale
+// byte); item 0 writes d.
+template<int TG>
+__device__ __forceinline__ void nib_gguf_item(uint8_t* blk, int ib, const uint8_t* __restrict__ slot, int row, int k) {
+ const uint4 q4 = *(const uint4*) (slot + nibk::code_off(row, k));
+ const uint32_t w[4] = { q4.x, q4.y, q4.z, q4.w };
+ const uint32_t sc = slot[nibk::scale_off(row, k)];
+ if (ib == 0) {
+ const uint8_t* dp = slot + nibk::d_off(row, k / 8);
+ blk[0] = dp[0];
+ blk[1] = dp[1];
+ }
+ if constexpr (TG == 22) { // block_iq2_s: d, qs[32] grid low bits, qs[32] signs, qh[8], scales[8]
+ uint32_t qh = 0;
+#pragma unroll
+ for (int l = 0; l < 4; ++l) {
+ int key = 0;
+ uint32_t sg = 0;
+#pragma unroll
+ for (int j = 7; j >= 0; --j) {
+ const uint32_t c = j < 4 ? (w[l] >> (4 * j)) & 15u : (w[l] >> (16 + 4 * (j - 4))) & 15u;
+ key = key * 3 + (int) (c & 3u);
+ sg |= ((c >> 2) & 1u) << j;
+ }
+ const uint32_t idx = key < kNibRev22 ? g_nib_rev22[key] : 0xFFFFFFFFu;
+ blk[2 + 4 * ib + l] = (uint8_t) idx;
+ blk[2 + 32 + 4 * ib + l] = (uint8_t) sg;
+ qh |= ((idx >> 8) & 3u) << (2 * l);
+ }
+ blk[66 + ib] = (uint8_t) qh;
+ blk[74 + ib] = (uint8_t) sc;
+ } else if constexpr (TG == 18) { // block_iq3_xxs: d, qs[64] grid indices, 8 x aux32 (4 x 7 sign bits | scale << 28)
+ uint32_t aux = sc << 28;
+#pragma unroll
+ for (int l = 0; l < 4; ++l) {
+ uint32_t sg = 0;
+#pragma unroll
+ for (int h = 0; h < 2; ++h) {
+ int key = 0;
+#pragma unroll
+ for (int j = 3; j >= 0; --j) {
+ const uint32_t c = (w[l] >> (16 * h + 4 * j)) & 15u;
+ key = key * 8 + (int) (c & 7u);
+ sg |= ((c >> 3) & 1u) << (4 * h + j);
+ }
+ blk[2 + 8 * ib + 2 * l + h] = (uint8_t) g_nib_rev18[key];
+ }
+ aux |= (sg & 0x7Fu) << (7 * l); // the 8th sign is the parity of the other 7 (ksigns_iq2xs)
+ }
+ blk[2 + 64 + 4 * ib + 0] = (uint8_t) aux;
+ blk[2 + 64 + 4 * ib + 1] = (uint8_t) (aux >> 8);
+ blk[2 + 64 + 4 * ib + 2] = (uint8_t) (aux >> 16);
+ blk[2 + 64 + 4 * ib + 3] = (uint8_t) (aux >> 24);
+ } else { // block_iq3_s: d, qs[64] grid low bits, qh[8], signs[32], scales[4] (two items' nibbles each)
+ uint32_t qh = 0;
+#pragma unroll
+ for (int l = 0; l < 4; ++l) {
+ uint32_t sg = 0;
+#pragma unroll
+ for (int h = 0; h < 2; ++h) {
+ int key = 0;
+#pragma unroll
+ for (int j = 3; j >= 0; --j) {
+ const uint32_t u = (w[l] >> (8 * j + 4 * h)) & 15u; // v = 2u - 15: negative for u <= 7
+ key = key * 8 + (int) (u <= 7u ? 7u - u : u - 8u);
+ sg |= (u <= 7u ? 1u : 0u) << (4 * h + j);
+ }
+ const uint32_t idx = g_nib_rev21[key];
+ blk[2 + 8 * ib + 2 * l + h] = (uint8_t) idx;
+ qh |= ((idx >> 8) & 1u) << (2 * l + h);
+ }
+ blk[74 + 4 * ib + l] = (uint8_t) sg;
+ }
+ blk[66 + ib] = (uint8_t) qh;
+ if ((ib & 1) == 0) {
+ const uint32_t sc1 = slot[nibk::scale_off(row, k + 1)];
+ blk[106 + ib / 2] = (uint8_t) (((sc - 1u) >> 1) | (((sc1 - 1u) >> 1) << 4));
+ }
+ }
+}
+
+constexpr int kNibGgufRows = 8; // gate/up rows a block rebuilds (in shared memory, then one coalesced pass)
+constexpr int kNibGgufThreads = 128;
+constexpr int kNibGgufWords = 8; // uint4 per thread in the down / tail part
+struct NibGgufArgs {
+ const unsigned long long* dev; // the slots' device addresses (check), or nullptr: `p`
+ const uint8_t* p[16]; // the slots (gather: an MMQ group, at most mmq::kGatherGroupMax)
+ int first, n;
+ uint8_t* gu; // gather: the rebuilt [gate | up]; check: the raw blobs' first byte
+ uint64_t gu_stride;
+ uint8_t* d; // gather: the down rows; check: the raw blobs' down rows
+ uint64_t d_stride;
+ uint64_t gu_row, down16, zero16; // GGUF row bytes; down rows and MMQ tail in uint4
+ unsigned long long* bad; // check: 32-bit words / uint4 that differ
+};
+// blockIdx.y: expert first + y. blockIdx.x < kRows / kNibGgufRows: rows [8x, 8x + 8) rebuilt into shared memory,
+// then written (gather) or compared (check) as 32-bit words; the rest: the down rows as uint4, and (gather, the last
+// expert) the zeroed tails after gate/up and down.
+template<int TG, bool CHECK>
+__global__ void __launch_bounds__(kNibGgufThreads) nib_gguf_kernel(NibGgufArgs a) {
+ constexpr int BB = (int) sizeof(typename NibBlk<TG>::B);
+ __shared__ uint32_t s_rows[kNibGgufRows * 1100 / 4];
+ const int q = a.first + (int) blockIdx.y;
+ const uint8_t* slot = a.dev != nullptr ? (const uint8_t*) a.dev[q] : a.p[q];
+ constexpr int kRowBlocks = (int) (nibk::kRows / kNibGgufRows);
+ unsigned long long nbad = 0;
+ if ((int) blockIdx.x < kRowBlocks) {
+ const int r0 = (int) blockIdx.x * kNibGgufRows;
+ uint8_t* sb = (uint8_t*) s_rows;
+ for (int u = threadIdx.x; u < kNibGgufRows * (int) nibk::kItems; u += kNibGgufThreads) {
+ const int rr = u / (int) nibk::kItems, k = u % (int) nibk::kItems;
+ nib_gguf_item<TG>(sb + (size_t) rr * a.gu_row + (size_t) (k >> 3) * BB, k & 7, slot, r0 + rr, k);
+ }
+ __syncthreads();
+ const int nw = (int) (kNibGgufRows * a.gu_row / 4);
+ uint32_t* dst = (uint32_t*) (a.gu + (size_t) q * a.gu_stride + (size_t) r0 * a.gu_row);
+ for (int i = threadIdx.x; i < nw; i += kNibGgufThreads) {
+ if constexpr (CHECK) nbad += dst[i] != s_rows[i];
+ else dst[i] = s_rows[i];
+ }
+ } else {
+ const uint4* src = (const uint4*) (slot + nibk::kDownOff);
+ uint4* dd = (uint4*) (a.d + (size_t) q * a.d_stride);
+ const int64_t base = ((int64_t) blockIdx.x - kRowBlocks) * (kNibGgufThreads * kNibGgufWords);
+#pragma unroll
+ for (int w = 0; w < kNibGgufWords; ++w) {
+ const int64_t i = base + (int64_t) w * kNibGgufThreads + threadIdx.x;
+ if (i < (int64_t) a.down16) {
+ if constexpr (CHECK) {
+ const uint4 x = src[i], y = dd[i];
+ nbad += x.x != y.x || x.y != y.y || x.z != y.z || x.w != y.w;
+ } else {
+ dd[i] = src[i];
+ }
+ } else if (!CHECK && q == a.n - 1 && i < (int64_t) (a.down16 + 2 * a.zero16)) {
+ const int64_t z = i - (int64_t) a.down16;
+ if (z < (int64_t) a.zero16) ((uint4*) (a.gu + (size_t) a.n * a.gu_stride))[z] = make_uint4(0, 0, 0, 0);
+ else ((uint4*) (a.d + (size_t) a.n * a.d_stride))[z - (int64_t) a.zero16] = make_uint4(0, 0, 0, 0);
+ }
+ }
+ }
+ if constexpr (CHECK) {
+ if (nbad) atomicAdd(a.bad, nbad);
+ }
+}
+
+// sum_i (w_i ^ i) * kHashMul mod 2^64 over the first `bytes` bytes of each slot (little-endian words, the last one
+// zero-extended): expert_nibble.hpp's hash_host. A sum, so the blocks' order of adding does not change it.
+__global__ void __launch_bounds__(256) nib_hash_kernel(const unsigned long long* __restrict__ slots, uint64_t bytes,
+ unsigned long long* __restrict__ out) {
+ const uint8_t* p = (const uint8_t*) slots[blockIdx.y];
+ const uint64_t n16 = bytes / 16, nw = (bytes + 3) / 4;
+ uint64_t h = 0;
+ for (uint64_t q = (uint64_t) blockIdx.x * blockDim.x + threadIdx.x; q < n16; q += (uint64_t) gridDim.x * blockDim.x) {
+ const uint4 v = ((const uint4*) p)[q];
+ const uint32_t i = (uint32_t) (4 * q);
+ h += (uint64_t) (v.x ^ i) * nibk::kHashMul + (uint64_t) (v.y ^ (i + 1)) * nibk::kHashMul +
+ (uint64_t) (v.z ^ (i + 2)) * nibk::kHashMul + (uint64_t) (v.w ^ (i + 3)) * nibk::kHashMul;
+ }
+ if (blockIdx.x == 0)
+ for (uint64_t i = 4 * n16 + threadIdx.x; i < nw; i += blockDim.x) {
+ uint32_t w = 0;
+ for (uint64_t b = 4 * i; b < 4 * i + 4 && b < bytes; ++b) w |= (uint32_t) p[b] << (8 * (b - 4 * i));
+ h += (uint64_t) (w ^ (uint32_t) i) * nibk::kHashMul;
+ }
+#pragma unroll
+ for (int o = 16; o > 0; o >>= 1) h += __shfl_xor_sync(0xffffffffu, h, o);
+ __shared__ uint64_t s_h[8];
+ if ((threadIdx.x & 31) == 0) s_h[threadIdx.x >> 5] = h;
+ __syncthreads();
+ if (threadIdx.x == 0) {
+ uint64_t t = 0;
+ for (int w = 0; w < (int) (blockDim.x >> 5); ++w) t += s_h[w];
+ atomicAdd(out + blockIdx.y, (unsigned long long) t);
+ }
+}
+#endif // STRATA_EXPERT_NIBBLE_BUILT
+
+// STRATA_EXPERT_NIBBLE: whether a layout's gate/up is a converted slot's 4-bit codes (constant false in a build
+// without the kernels, so its dispatch below is today's code)
+constexpr bool nib_layout(const NativeExpertLayout& L) {
+#if STRATA_EXPERT_NIBBLE_BUILT
+ return L.gu_nib;
+#else
+ static_cast<void>(L);
+ return false;
+#endif
+}
+
} // namespace
void iq_set_old_kernels(bool old) { g_old_kernels = old; }
@@ -4660,6 +5359,50 @@ NativeExpertLayout native_expert_layout(int gu_type, int d_type, int64_t n_embd,
return L;
}
+NativeExpertLayout native_expert_layout_nib(int gu_type, int d_type, int64_t n_embd, int64_t n_ff) {
+ namespace nb = strata::core::nibble;
+ NativeExpertLayout L = native_expert_layout(gu_type, d_type, n_embd, n_ff);
+ L.gu_nib = true;
+ L.gu_row = (size_t) (nb::kItems * nb::kItemBytes); // a row's codes (the scales and d follow all the codes)
+ L.up_off = nb::kUpCodesOff;
+ L.down_off = nb::kDownOff;
+ L.bytes = nb::slot_bytes_for((uint64_t) n_embd * L.d_row);
+ return L;
+}
+
+bool native_expert_nib_supported(int gu_type, int d_type, int64_t n_embd, int64_t n_ff) noexcept {
+#if STRATA_EXPERT_NIBBLE_BUILT
+ namespace nb = strata::core::nibble;
+ return nb::fmt_built(gu_type) && n_embd == nb::kNEmbd && n_ff == nb::kNFf && d_type != 40 &&
+ native_expert_supported(gu_type, d_type, n_embd, n_ff);
+#else
+ (void) gu_type; (void) d_type; (void) n_embd; (void) n_ff;
+ return false;
+#endif
+}
+
+void native_expert_nib_set_variant(int v) {
+#if STRATA_EXPERT_NIBBLE_BUILT
+ g_nib_variant = v;
+#else
+ (void) v;
+#endif
+}
+
+bool native_expert_nib_variant_attr(int v, int gu_type, int* regs, int* local_bytes, int* blocks_per_sm) {
+#if STRATA_EXPERT_NIBBLE_BUILT
+ cudaFuncAttributes a{};
+ if (!nibx_launch(gu_type, v, dim3(80, 1), nullptr, nullptr, nullptr, nullptr, nullptr, nullptr, nullptr, nullptr, &a)) return false;
+ *regs = a.numRegs;
+ *local_bytes = (int) a.localSizeBytes;
+ *blocks_per_sm = g_nibx_occ;
+ return true;
+#else
+ (void) v; (void) gu_type; (void) regs; (void) local_bytes; (void) blocks_per_sm;
+ return false;
+#endif
+}
+
size_t native_expert_scratch_bytes(int64_t cap, int64_t n_ff) {
const size_t f = (size_t) cap * (size_t) n_ff * sizeof(float);
return 3 * ((f + 255) & ~(size_t) 255) + (((size_t) cap * (size_t) (n_ff / 32) * sizeof(block_q8_1) + 255) & ~(size_t) 255);
@@ -5315,7 +6058,7 @@ int exp_mode() {
#endif
}
#endif
-int g_exp_phase = 0; // the bench: 0 all, 1 gate/up + swiglu + quantize only, 2 down only
+int g_exp_phase = 0; // the bench: 0 all, 1 gate/up + swiglu + quantize only, 2 down only, 3 gate/up only
#if STRATA_EXP_LAYOUTS && !defined(__HIPCC__)
// The down projection's mode on Turing's default: mode 4 is slower than the CUDA layout once groups have 3+ entries
// (12 x 3: 36.5 -> 40.8 us, 16 x 4: 56.9 -> 68.9) - windows of 4+ tokens (cap_groups > 30) keep mode 0 there.
@@ -5389,8 +6132,16 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long
float* h = (float*) ((uint8_t*) scratch + 2 * fa);
block_q8_1* hq = (block_q8_1*) ((uint8_t*) scratch + 3 * fa);
const auto* X = (const block_q8_1*) x_q8_1;
+#if !STRATA_EXPERT_NIBBLE_BUILT
+ if (L.gu_nib) { // STRATA_EXPERT_NIBBLE's slots: the switch is refused at start in a build without the kernels
+ std::fprintf(stderr, "native_expert_grouped: 4-bit gate/up slots are not built into this engine\n");
+ std::exit(1);
+ }
+#endif
+ // STRATA_EXPERT_NIBBLE (nib_layout): only the gate/up step below takes another kernel (launch_gu_nib, in place of
+ // launch_gu); SwiGLU, the q8_1 quantization and the down rows (GGUF, at L.down_off) are the layout's own
static const bool v2 = [] { const char* v = std::getenv("STRATA_EXPERT_V2"); return v && v[0] == '1'; }();
- if (v2 && L.gu_type == 21 && L.d_type == 20 && L.n_embd == 2560 && L.n_ff == 640) { // S26: see s26_gu_l_kernel
+ if (v2 && !nib_layout(L) && L.gu_type == 21 && L.d_type == 20 && L.n_embd == 2560 && L.n_ff == 640) { // S26: see s26_gu_l_kernel
// S26 STRATA_TSUM=1: the sums as one transposed butterfly per warp (s26_tsum.cuh; bitwise the same values)
static const bool ts = [] { const char* v = std::getenv("STRATA_TSUM"); return v && v[0] == '1'; }();
// (+ down rows' items spread evenly over the lanes, one load group: bitwise, harness 1.09-1.12x vs 1.03-1.06x;
@@ -5460,7 +6211,7 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long
#endif
#endif
#if STRATA_EXP_LAYOUTS
- const bool fused_gu = em0 == 8 && ((kExpLds16 && L.gu_type == 16) || L.gu_type == 18 || L.gu_type == 21 || L.gu_type == 22 || L.gu_type == 23) &&
+ const bool fused_gu = !nib_layout(L) && em0 == 8 && ((kExpLds16 && L.gu_type == 16) || L.gu_type == 18 || L.gu_type == 21 || L.gu_type == 22 || L.gu_type == 23) &&
L.n_ff % 32 == 0;
if (fused_gu && g_exp_phase != 2) {
const dim3 gl((unsigned) (L.n_ff / 32), (unsigned) cap_groups);
@@ -5482,7 +6233,7 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long
#else
const bool gs_layout = true;
#endif
- if (gs_layout && g_exp_phase == 0 && !v1 && gu_swiglu_on() && !(expdec_mode() && !g_old_kernels && expdec_gu_ok(L))) {
+ if (gs_layout && !nib_layout(L) && g_exp_phase == 0 && !v1 && gu_swiglu_on() && !(expdec_mode() && !g_old_kernels && expdec_gu_ok(L))) {
switch (L.gu_type) {
#define STRATA_GS(T) case T: gs_done = launch_gu_swiglu<T>(L, gy, s, grp_ptr, grp_start, n_groups, ent_tok, X, hq); break;
STRATA_GU_FMTS(STRATA_GS)
@@ -5497,7 +6248,7 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long
if (g_exp_phase != 2 && !gs_done) {
#endif
#if STRATA_EXP_LAYOUTS
- const bool lds_gu = (em0 == 5 || em0 == 6 || em0 == 7 || em0 == 8) && ((kExpLds16 && L.gu_type == 16) || L.gu_type == 18 || L.gu_type == 21 || L.gu_type == 22 || L.gu_type == 23);
+ const bool lds_gu = !nib_layout(L) && (em0 == 5 || em0 == 6 || em0 == 7 || em0 == 8) && ((kExpLds16 && L.gu_type == 16) || L.gu_type == 18 || L.gu_type == 21 || L.gu_type == 22 || L.gu_type == 23);
if (lds_gu) {
const dim3 gl((unsigned) ((2 * L.n_ff + LDS_RB - 1) / LDS_RB), (unsigned) cap_groups);
const size_t sh = (size_t) LDS_NT * (size_t) (L.n_embd / 32) * sizeof(block_q8_1);
@@ -5518,13 +6269,19 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long
// the AMD layouts cover the IQ packs' gate/up types; Unsloth's Q4_K / Q5_K take the CUDA layout below
const bool amd_gu = L.gu_type == 16 || L.gu_type == 17 || L.gu_type == 18 || L.gu_type == 21 || L.gu_type == 22 ||
L.gu_type == 23 || L.gu_type == 29 || L.gu_type == 42;
- if ((em == 1 || em == 2 || em == 4) && amd_gu) {
+ if ((em == 1 || em == 2 || em == 4) && amd_gu && !nib_layout(L)) {
switch (L.gu_type) {
STRATA_GU_AMD(16) STRATA_GU_AMD(17) STRATA_GU_AMD(18) STRATA_GU_AMD(21) STRATA_GU_AMD(22) STRATA_GU_AMD(23)
STRATA_GU_AMD(29) STRATA_GU_AMD(42)
}
} else
#undef STRATA_GU_AMD
+#endif
+#if STRATA_EXPERT_NIBBLE_BUILT
+ if (nib_layout(L)) // STRATA_EXPERT_NIBBLE: a converted layer's VRAM slots, native_gu_multi_kernel's grid and bits
+ launch_gu_nib(L, dim3((unsigned) (strata::core::nibble::kRows / 16), (unsigned) gy), s, grp_ptr, grp_start,
+ n_groups, ent_tok, X, gate, up);
+ else
#endif
if (!expdec_gu(L, gy, cap_groups, s, grp_ptr, grp_start, n_groups, ent_tok, X, gate, up)) // STRATA_EXPDEC=1
switch (L.gu_type) {
@@ -5537,6 +6294,7 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long
}
#endif
check("native_expert_grouped/gu");
+ if (g_exp_phase == 3) return; // the bench: gate/up only
const long long nh = (long long) cap_entries * L.n_ff;
#if defined(STRATA_HIP_GFX906)
// gfx906: the separate SwiGLU and q8_1 passes stay the default (the A/B baseline); the RDNA one-pass
@@ -5611,6 +6369,164 @@ void native_expert_grouped(const NativeExpertLayout& L, const unsigned long long
} // namespace strata::kernels
+// ---------------------------------------------------------------- STRATA_EXPERT_NIBBLE: expert_nibble.hpp's functions
+namespace strata::core::nibble {
+
+#if STRATA_EXPERT_NIBBLE_BUILT
+namespace {
+int g_rev_ready[64]; // per device: 0 not built, 1 built, 2 a grid entry not unique (refused)
+
+bool rev_ready() {
+ int dev = 0;
+ return cudaGetDevice(&dev) == cudaSuccess && dev >= 0 && dev < 64 && g_rev_ready[dev] == 1;
+}
+} // namespace
+#endif
+
+int prepare(void* stream) {
+#if STRATA_EXPERT_NIBBLE_BUILT
+ int dev = 0;
+ if (cudaGetDevice(&dev) != cudaSuccess || dev < 0 || dev >= 64) return (int) cudaErrorInvalidDevice;
+ if (g_rev_ready[dev] == 1) return 0;
+ if (g_rev_ready[dev] == 2) return (int) cudaErrorInvalidValue;
+ cudaStream_t s = (cudaStream_t) stream;
+ strata::kernels::nib_rev_clear_kernel<<<(strata::kernels::kNibRev22 + 255) / 256, 256, 0, s>>>();
+ strata::kernels::nib_rev_fill_kernel<<<(1024 + 256 + 512 + 255) / 256, 256, 0, s>>>();
+ cudaError_t e = cudaGetLastError();
+ unsigned long long bad = 0;
+ if (e == cudaSuccess) e = cudaStreamSynchronize(s);
+ if (e == cudaSuccess) e = cudaMemcpyFromSymbol(&bad, strata::kernels::g_nib_rev_bad, sizeof bad);
+ if (e != cudaSuccess) return (int) e;
+ g_rev_ready[dev] = bad == 0 ? 1 : 2;
+ return bad == 0 ? 0 : (int) cudaErrorInvalidValue;
+#else
+ (void) stream;
+ return 1; // cudaErrorInvalidValue: not in this build
+#endif
+}
+
+int transcode(const LayerVram& L, const void* staging, int64_t n, const unsigned long long* slot_ptrs,
+ unsigned long long* bad, void* stream) {
+#if STRATA_EXPERT_NIBBLE_BUILT
+ if (n <= 0) return 0;
+ if (!L.on || !fmt_built(L.gu_type) || L.raw_gu_row != raw_gu_row(L.gu_type) ||
+ L.raw_up_off != (uint64_t) kNFf * L.raw_gu_row || L.raw_bytes < 2 * L.raw_up_off)
+ return (int) cudaErrorInvalidValue;
+ const long long nthr = (long long) n * kRows * kItems;
+ const dim3 grid((unsigned) ((nthr + 255) / 256));
+ cudaStream_t s = (cudaStream_t) stream;
+ const uint8_t* src = (const uint8_t*) staging;
+#define STRATA_NIB_TC(TG) \
+ strata::kernels::nib_transcode_kernel<TG><<<grid, 256, 0, s>>>(src, L.raw_bytes, L.raw_gu_row, L.raw_up_off, slot_ptrs, n, bad)
+ switch (L.gu_type) {
+ case kIQ2_S: STRATA_NIB_TC(22); break;
+ case kIQ3_XXS: STRATA_NIB_TC(18); break;
+ case kIQ3_S: STRATA_NIB_TC(21); break;
+ default: return (int) cudaErrorInvalidValue;
+ }
+#undef STRATA_NIB_TC
+ return (int) cudaGetLastError();
+#else
+ (void) L; (void) staging; (void) n; (void) slot_ptrs; (void) bad; (void) stream;
+ return 1; // cudaErrorInvalidValue: not in this build
+#endif
+}
+
+#if STRATA_EXPERT_NIBBLE_BUILT
+namespace {
+int launch_gguf(const LayerVram& L, strata::kernels::NibGgufArgs& a, bool check, cudaStream_t s) {
+ namespace K = strata::kernels;
+ if (!L.on || !fmt_built(L.gu_type) || L.raw_gu_row != raw_gu_row(L.gu_type) || L.down_bytes % 16 != 0 ||
+ L.raw_up_off != (uint64_t) kNFf * L.raw_gu_row || L.raw_down_off != 2 * L.raw_up_off)
+ return (int) cudaErrorInvalidValue;
+ if (!rev_ready()) return (int) cudaErrorNotReady; // prepare() was not run on this device
+ a.gu_row = L.raw_gu_row;
+ a.down16 = L.down_bytes / 16;
+ const int64_t tail = (int64_t) (a.down16 + 2 * a.zero16);
+ const int64_t per = (int64_t) K::kNibGgufThreads * K::kNibGgufWords;
+ const dim3 grid((unsigned) (kRows / K::kNibGgufRows + (tail + per - 1) / per), (unsigned) (a.n - a.first));
+#define STRATA_NIB_GG(TG) \
+ (check ? K::nib_gguf_kernel<TG, true><<<grid, K::kNibGgufThreads, 0, s>>>(a) \
+ : K::nib_gguf_kernel<TG, false><<<grid, K::kNibGgufThreads, 0, s>>>(a))
+ switch (L.gu_type) {
+ case kIQ2_S: STRATA_NIB_GG(22); break;
+ case kIQ3_XXS: STRATA_NIB_GG(18); break;
+ case kIQ3_S: STRATA_NIB_GG(21); break;
+ default: return (int) cudaErrorInvalidValue;
+ }
+#undef STRATA_NIB_GG
+ return (int) cudaGetLastError();
+}
+} // namespace
+#endif
+
+int check_gguf(const LayerVram& L, const void* staging, int64_t n, const unsigned long long* slot_ptrs,
+ unsigned long long* bad, void* stream) {
+#if STRATA_EXPERT_NIBBLE_BUILT
+ if (n <= 0) return 0;
+ if (n > 65535) return (int) cudaErrorInvalidValue;
+ strata::kernels::NibGgufArgs a{};
+ a.dev = slot_ptrs;
+ a.first = 0;
+ a.n = (int) n;
+ a.gu = (uint8_t*) staging;
+ a.gu_stride = L.raw_bytes;
+ a.d = (uint8_t*) staging + L.raw_down_off;
+ a.d_stride = L.raw_bytes;
+ a.zero16 = 0;
+ a.bad = bad;
+ if (((uintptr_t) staging | L.raw_bytes) % 16 != 0) return (int) cudaErrorInvalidValue;
+ return launch_gguf(L, a, true, (cudaStream_t) stream);
+#else
+ (void) L; (void) staging; (void) n; (void) slot_ptrs; (void) bad; (void) stream;
+ return 1; // cudaErrorInvalidValue: not in this build
+#endif
+}
+
+int gather_gguf(const LayerVram& L, const uint8_t* const* slots, int first, int n, void* gu_dst, uint64_t gu_stride,
+ void* d_dst, uint64_t d_stride, uint64_t zero_bytes, void* stream) {
+#if STRATA_EXPERT_NIBBLE_BUILT
+ if (first < 0 || n <= first || n > 16) return (int) cudaErrorInvalidValue;
+ uintptr_t al = (uintptr_t) gu_dst | (uintptr_t) d_dst | gu_stride | d_stride | zero_bytes;
+ for (int q = first; q < n; ++q) al |= (uintptr_t) slots[q];
+ if (al % 16 != 0 || gu_stride < (uint64_t) kRows * L.raw_gu_row || d_stride < L.down_bytes) return (int) cudaErrorInvalidValue;
+ strata::kernels::NibGgufArgs a{};
+ a.dev = nullptr;
+ for (int q = first; q < n; ++q) a.p[q] = slots[q];
+ a.first = first;
+ a.n = n;
+ a.gu = (uint8_t*) gu_dst;
+ a.gu_stride = gu_stride;
+ a.d = (uint8_t*) d_dst;
+ a.d_stride = d_stride;
+ a.zero16 = zero_bytes / 16;
+ a.bad = nullptr;
+ return launch_gguf(L, a, false, (cudaStream_t) stream);
+#else
+ (void) L; (void) slots; (void) first; (void) n; (void) gu_dst; (void) gu_stride; (void) d_dst; (void) d_stride;
+ (void) zero_bytes; (void) stream;
+ return 1; // cudaErrorInvalidValue: not in this build
+#endif
+}
+
+int hash_slots(const LayerVram& L, const unsigned long long* slot_ptrs, int64_t n, unsigned long long* out, void* stream) {
+#if STRATA_EXPERT_NIBBLE_BUILT
+ if (n <= 0) return 0;
+ if (L.hashed == 0) return (int) cudaErrorInvalidValue;
+ cudaStream_t s = (cudaStream_t) stream;
+ const cudaError_t e = cudaMemsetAsync(out, 0, (size_t) n * sizeof(unsigned long long), s);
+ if (e != cudaSuccess) return (int) e;
+ // ~2.7 MB a slot: 32 blocks of 256 threads each, ~21 16-byte loads a thread
+ strata::kernels::nib_hash_kernel<<<dim3(32, (unsigned) n), 256, 0, s>>>(slot_ptrs, L.hashed, out);
+ return (int) cudaGetLastError();
+#else
+ (void) L; (void) slot_ptrs; (void) n; (void) out; (void) stream;
+ return 1; // cudaErrorInvalidValue: not in this build
+#endif
+}
+
+} // namespace strata::core::nibble
+
// ---- Volta prompt experts (STRATA_PF_WMMA): a group's experts as one launch of FP16 tensor-core tiles, the weights
// dequantized into shared memory by dq_dispatch (the FP16 path's own formulas), the activations FP16, FP32 sums.
// Replaces MMQ's dp4a products on sm_70, and the FP16 path's dequantize-to-global + one cuBLAS call per expert.
diff --git a/src/kernels/cuda/nib_gu_proto.cuh b/src/kernels/cuda/nib_gu_proto.cuh
new file mode 100644
index 000000000..368eeb335
--- /dev/null
+++ b/src/kernels/cuda/nib_gu_proto.cuh
@@ -0,0 +1,137 @@
+// src/kernels/cuda/nib_gu_proto.cuh - STRATA_NIB_GU_PROTO: a staging variant of nib_gu_kernel (STRATA_EXPERT_NIBBLE's
+// 4-bit gate/up), included by iq_kernels.cu inside its anonymous namespace after nib_gu_kernel.
+//
+// The kernel keeps the S skeleton's per-row arithmetic by calling nib_row_dot_s itself (16 lanes a row, lane t's items
+// k = t, t+32, t+64 into s and t+16, t+48 into s16, NibLoad<TG>::load then Split<TG>::apply on the staged block_q8_1,
+// __fadd_rn(s, s16), xor 8, 4, 2, 1). Only the staging and the block -> (group, rows) mapping change:
+// - a block takes RB row blocks (16 rows each) of one group: grid (80 / RB, gy). The group's entries are staged
+// once and the RB row blocks run one after another on them;
+// - the staging reads the entries' q8_1 rows (2,880 B = 180 x 16 B each, 16-aligned when xq is) as 16-byte loads,
+// every load issued before the first store, so only the stores sit between the two barriers. The S skeleton's
+// word loop (nib_pass_s) has a serial load -> store chain there that stalls all eight warps of the block.
+// The outputs are written without nib_pass_s's dynamically indexed s[t] (a select on t), the same values.
+// Id 31 = RB 2, the one STRATA_NIB_GU_PROTO=1 selects: one GP100 die (PH402), T=1, gate/up alone, IQ2_S / IQ3_XXS /
+// IQ3_S x1.25-1.27 against the S / P skeleton, every gate and up row bitwise equal (iq_nibble_parity (7),
+// native_expert_bench --nibble --vs 31).
+#pragma once
+
+// The n (1..GRP_NC) entries from e into s_x (block_q8_1 rows of 80). Block-uniform (e, n): every thread reaches both
+// barriers. 16-byte loads, all in flight before the first store.
+__device__ __forceinline__ void nibx_stage(block_q8_1* s_x, const int32_t* __restrict__ ent_tok,
+ const block_q8_1* __restrict__ xq, int e, int n) {
+ constexpr int kV = (GRP_NC * 180 + 255) / 256; // 3
+ const int nv = n * 180;
+ uint4 v[kV];
+#pragma unroll
+ for (int j = 0; j < kV; ++j) {
+ const int i = threadIdx.x + 256 * j;
+ if (i < nv) {
+ const int c = i >= 540 ? 3 : i >= 360 ? 2 : i >= 180 ? 1 : 0;
+ v[j] = ((const uint4*) (xq + (size_t) ent_tok[e + c] * 80))[i - 180 * c];
+ }
+ }
+ __syncthreads();
+#pragma unroll
+ for (int j = 0; j < kV; ++j) {
+ const int i = threadIdx.x + 256 * j;
+ if (i < nv) ((uint4*) s_x)[i] = v[j];
+ }
+ __syncthreads();
+}
+
+// nib_pass_s's compute on the staged entries (nib_row_dot_s, the production function), stored without the dynamically
+// indexed s[t]
+template<int TG>
+__device__ __forceinline__ void nibx_run(const uint8_t* slot, int row, const block_q8_1* s_x, int e, int n, int t,
+ float* dst, int r) {
+ const uint4* crow = (const uint4*) (slot + nibk::kCodesOff) + (size_t) row * nibk::kItems;
+ const uint8_t* srow = slot + nibk::kScOff + (size_t) row * nibk::kItems;
+ const __half* drow = (const __half*) (slot + nibk::kDOff) + (size_t) row * nibk::kSuper;
+ if (n == 1) {
+ const int o[1] = { 0 }; float s[1];
+ nib_row_dot_s<TG, 1>(crow, srow, drow, s_x, o, t, s);
+ if (t == 0) dst[(size_t) e * nibk::kNFf + r] = s[0];
+ } else if (n == 2) {
+ const int o[2] = { 0, 80 }; float s[2];
+ nib_row_dot_s<TG, 2>(crow, srow, drow, s_x, o, t, s);
+ if (t < 2) dst[(size_t) (e + t) * nibk::kNFf + r] = t == 0 ? s[0] : s[1];
+ } else if (n == 3) {
+ const int o[3] = { 0, 80, 160 }; float s[3];
+ nib_row_dot_s<TG, 3>(crow, srow, drow, s_x, o, t, s);
+ if (t < 3) dst[(size_t) (e + t) * nibk::kNFf + r] = t == 0 ? s[0] : t == 1 ? s[1] : s[2];
+ } else {
+ const int o[4] = { 0, 80, 160, 240 }; float s[4];
+ nib_row_dot_s<TG, 4>(crow, srow, drow, s_x, o, t, s);
+ if (t < 4) dst[(size_t) (e + t) * nibk::kNFf + r] = t == 0 ? s[0] : t == 1 ? s[1] : t == 2 ? s[2] : s[3];
+ }
+}
+
+template<int TG, int RB>
+__global__ void __launch_bounds__(256) nibx_gu_kernel(const unsigned long long* __restrict__ grp_ptr,
+ const int32_t* __restrict__ grp_start,
+ const int32_t* __restrict__ n_groups,
+ const int32_t* __restrict__ ent_tok,
+ const block_q8_1* __restrict__ xq, float* __restrict__ gate,
+ float* __restrict__ up) {
+ static_assert(80 % RB == 0, "row blocks per block divide 80");
+ const int ng = *n_groups;
+ if (blockIdx.y >= (unsigned) ng) return;
+ STRATA_SHARED_ALIGN16_U32(s_buf, kNibSWords);
+ block_q8_1* s_x = (block_q8_1*) s_buf;
+ const int subwarp = threadIdx.x >> 4, t = threadIdx.x & 15;
+ const int rb0 = blockIdx.x * RB;
+ for (int g = blockIdx.y; g < ng; g += gridDim.y) {
+ const uint8_t* slot = (const uint8_t*) grp_ptr[g];
+ const int e0 = grp_start[g], e1 = grp_start[g + 1];
+ const int nt = e1 - e0;
+ if (nt <= 0) continue; // an empty group: nothing to write (block-uniform, no barrier skipped unevenly)
+ if (nt <= GRP_NC) {
+ nibx_stage(s_x, ent_tok, xq, e0, nt);
+#pragma unroll 1
+ for (int j = 0; j < RB; ++j) {
+ const int row = (rb0 + j) * 16 + subwarp;
+ const bool is_up = row >= nibk::kNFf;
+ nibx_run<TG>(slot, row, s_x, e0, nt, t, is_up ? up : gate, is_up ? row - (int) nibk::kNFf : row);
+ }
+ } else { // more entries than one pass holds: per row block, the passes as nib_pass_s runs them
+#pragma unroll 1
+ for (int j = 0; j < RB; ++j) {
+ const int row = (rb0 + j) * 16 + subwarp;
+ const bool is_up = row >= nibk::kNFf;
+ for (int e = e0; e < e1; e += GRP_NC) {
+ const int n = min(GRP_NC, e1 - e);
+ nibx_stage(s_x, ent_tok, xq, e, n);
+ nibx_run<TG>(slot, row, s_x, e, n, t, is_up ? up : gate, is_up ? row - (int) nibk::kNFf : row);
+ }
+ }
+ }
+ }
+}
+
+int g_nibx_occ = 0; // the last attribute query's resident blocks per SM
+
+template<int TG, int RB>
+void nibx_launch_one(dim3 grid, cudaStream_t s, const unsigned long long* grp_ptr, const int32_t* grp_start,
+ const int32_t* n_groups, const int32_t* ent_tok, const block_q8_1* X, float* gate, float* up,
+ cudaFuncAttributes* attr) {
+ if (attr) {
+ cudaFuncGetAttributes(attr, nibx_gu_kernel<TG, RB>);
+ cudaOccupancyMaxActiveBlocksPerMultiprocessor(&g_nibx_occ, nibx_gu_kernel<TG, RB>, 256, 0);
+ return;
+ }
+ nibx_gu_kernel<TG, RB><<<dim3(80 / RB, grid.y), 256, 0, s>>>(grp_ptr, grp_start, n_groups, ent_tok, X, gate, up);
+}
+
+// false: not the gu-proto id (31) or not a converted format. attr != nullptr: only the kernel's attributes, nothing
+// launched.
+bool nibx_launch(int gu_type, int id, dim3 grid, cudaStream_t s, const unsigned long long* grp_ptr,
+ const int32_t* grp_start, const int32_t* n_groups, const int32_t* ent_tok, const block_q8_1* X,
+ float* gate, float* up, cudaFuncAttributes* attr = nullptr) {
+ if (id != 31) return false;
+ switch (gu_type) {
+ case 22: nibx_launch_one<22, 2>(grid, s, grp_ptr, grp_start, n_groups, ent_tok, X, gate, up, attr); return true;
+ case 18: nibx_launch_one<18, 2>(grid, s, grp_ptr, grp_start, n_groups, ent_tok, X, gate, up, attr); return true;
+ case 21: nibx_launch_one<21, 2>(grid, s, grp_ptr, grp_start, n_groups, ent_tok, X, gate, up, attr); return true;
+ default: return false;
+ }
+}
diff --git a/src/kernels/iq_nibble_parity.cpp b/src/kernels/iq_nibble_parity.cpp
new file mode 100644
index 000000000..40196bdd0
--- /dev/null
+++ b/src/kernels/iq_nibble_parity.cpp
@@ -0,0 +1,954 @@
+// src/kernels/iq_nibble_parity.cpp - STRATA_EXPERT_NIBBLE (strata/core/expert_nibble.hpp): the 4-bit gate/up expert
+// slots against the production kernels and the GGUF blobs, bit for bit.
+//
+// build/iq_nibble_parity --header the host half of expert_nibble.hpp (no GPU)
+// build/iq_nibble_parity <shard.gguf> [<shard.gguf> ...] [--layers-per-fmt 2] [--layers 1,17] [--ne 512]
+// [--plans 8] [--synthetic 64] [--no-ggml] [--negative]
+// build/iq_nibble_parity --synthetic-only [--synthetic 64] [--plans 8] [--no-ggml] no model, no shard
+//
+// Each model (any shard of it; its other shards are opened beside it) gives `--layers-per-fmt` layers of each
+// converted format (IQ2_S, IQ3_XXS, IQ3_S; or `--layers`), `--ne` experts each, read from the GGUF tensors into the
+// blob the pack holds (gate rows, up rows, down rows). Per layer:
+// (1) nibble::transcode, 64 blobs a batch as the startup fill runs it: 0 items that do not round-trip;
+// (2) every gate/up byte of a slot is written (8 slots transcoded again into another pre-fill byte);
+// (3) nibble::hash_slots against nibble::hash_host on every slot read back (and a flipped byte must change it);
+// (4) the host codec on every item: decode_item then encode_item gives the code back;
+// (5) ggml's reference dequantizer (to_float) on every GGUF row against the weight the 4-bit slot gives through
+// dequantize_row_iq2_s / _iq3_xxs / _iq3_s's own expression (code, scale byte, fp16 d): every weight, bit for bit;
+// (6) nibble::check_gguf (the start-up check: the GGUF blocks rebuilt from every slot against the blob): 0 words
+// differ; nibble::gather_gguf (the prompt path's MMQ gather) in groups of 16 and of 1..15 from a later first
+// position: the rebuilt [gate | up] and the down rows byte for byte the blob's, the MMQ tail after the group zero;
+// (7) native_expert_grouped on the converted slots (native_expert_layout_nib) against the same call on the GGUF
+// blobs (native_expert_layout): hq and out byte for byte for every entry, and gate / up where the reference
+// layout writes them - in the card's default layout (a GP100: mode 8, whose gate/up kernel writes hq) and in the
+// CUDA layout (native_expert_set_mode(0)), each with native_grouped_set_v1 off and on (h too, in mode 0 + v1);
+// top-10 routing T = 1..6 at 0% and 50% sharing (`--plans` each), one group of T entries (G = 1) per T, plans of
+// groups with 0..11 entries (more than one GRP_NC pass, empty groups); grid_groups 0 (cap) and 4 (the PCIe
+// call's stride); the default kernel per format and each skeleton forced (S, P); q8_1 activations random, all
+// +127, all -127 and alternating +-127 (the sum of x at its extremes, +-4064).
+// `--synthetic N`: per format, N experts whose gate/up blocks are random bytes (every grid index, sign byte and scale
+// nibble the format has; the coverage is counted and must be complete) and whose fp16 d cycle through +-the smallest
+// subnormal, +-65504, +-1, the smallest normal and two ordinary values; checks (1)-(7) on them.
+// `--synthetic-only`: the synthetic experts alone, for every gate/up format x both down types (IQ4_NL, Q2_0), the down rows
+// random bytes with ordinary fp16 scales: checks (1)-(7) with no model file (a GP100 is still needed).
+// `--negative`: the control - one code byte per slot changed after the transcode; (6) and (7) must then report differences.
+// Exit 0 when every check is equal; 1 otherwise; 2 on a usage or setup error. A build without STRATA_EXPERT_NIBBLE_BUILT
+// (the CUDA build with STRATA_EXPERIMENTAL_SM60) has no 4-bit kernels: --header only.
+#include "strata/artifact/gguf_reader.hpp"
+#include "strata/core/expert_nibble.hpp"
+#include "strata/kernels/cpu/native_expert.hpp"
+#include "strata/kernels/iq_kernels.hpp"
+
+#include "ggml.h"
+
+#include <cuda_runtime.h>
+
+#include <algorithm>
+#include <atomic>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <memory>
+#include <random>
+#include <set>
+#include <string>
+#include <thread>
+#include <vector>
+
+namespace {
+
+namespace nib = strata::core::nibble;
+namespace K = strata::kernels;
+namespace cpu = strata::kernels::cpu;
+
+int fails = 0;
+void expect(bool ok, const char* what) {
+ if (!ok) {
+ std::printf("FAIL %s\n", what);
+ ++fails;
+ }
+}
+
+// Every value the format holds at every one of an item's 32 positions (the other positions cycle through the set),
+// encoded, decoded and compared; and one value outside the set must be refused.
+void round_trip(int fmt, const std::vector<int8_t>& set, int8_t outside) {
+ int8_t g[32], back[32];
+ uint32_t q[4];
+ for (size_t shift = 0; shift < set.size(); ++shift) {
+ for (int i = 0; i < 32; ++i) g[i] = set[(shift + (size_t) i) % set.size()];
+ const bool ok = nib::encode_item(fmt, g, q);
+ nib::decode_item(fmt, q, back);
+ expect(ok && std::memcmp(g, back, 32) == 0, (std::string(nib::fmt_name(fmt)) + " round trip").c_str());
+ }
+ for (int i = 0; i < 32; ++i) g[i] = set[0];
+ g[17] = outside;
+ expect(!nib::encode_item(fmt, g, q), (std::string(nib::fmt_name(fmt)) + " refuses a value outside the set").c_str());
+}
+
+int header_check() {
+ expect(nib::kGuBytes == 1766400 && nib::kScOff == 1638400 && nib::kDOff == 1740800, "layout sizes");
+ expect(nib::code_off(nib::kNFf, 0) == nib::kUpCodesOff, "up rows' codes");
+ expect(nib::slot_bytes_for(921600) % nib::kSlotAlign == 0, "slot alignment");
+ round_trip(nib::kIQ2_S, { 8, 25, 43, -8, -25, -43 }, 9);
+ round_trip(nib::kIQ3_XXS, { 4, 12, 20, 28, 36, 44, 52, 62, -4, -12, -20, -28, -36, -44, -52, -62 }, 5);
+ std::vector<int8_t> s3;
+ for (int v = -15; v <= 15; v += 2) s3.push_back((int8_t) v);
+ round_trip(nib::kIQ3_S, s3, 2);
+ // IQ2_S's nibble is the decode kernel's PRMT selector into {8,25,43,0} {-8,-25,-43,0}
+ {
+ int8_t g[32] = {}, back[32];
+ uint32_t q[4];
+ for (int i = 0; i < 32; ++i) g[i] = (int8_t) (i % 2 ? -25 : 43);
+ nib::encode_item(nib::kIQ2_S, g, q);
+ expect((q[0] & 0xFu) == 2u && (q[0] >> 4 & 0xFu) == 5u, "IQ2_S selector nibbles");
+ nib::decode_item(nib::kIQ2_S, q, back);
+ expect(std::memcmp(g, back, 32) == 0, "IQ2_S selector decode");
+ }
+ expect(nib::gu_skeleton(nib::kIQ2_S) == nib::kSkelP && nib::gu_skeleton(nib::kIQ3_XXS) == nib::kSkelS &&
+ nib::gu_skeleton(nib::kIQ3_S) == nib::kSkelS,
+ "kernel table");
+ {
+ const uint8_t b[6] = { 1, 2, 3, 4, 5, 6 };
+ const uint64_t h = nib::hash_host(b, 6);
+ const uint64_t want = (uint64_t) (0x04030201u ^ 0u) * nib::kHashMul + (uint64_t) (0x00000605u ^ 1u) * nib::kHashMul;
+ expect(h == want, "hash_host");
+ }
+ std::printf("%s (%d failures)\n", fails == 0 ? "header OK" : "header FAILED", fails);
+ return fails == 0 ? 0 : 1;
+}
+
+// ------------------------------------------------------------------------------------------------ the GPU parity
+
+#define CK(x) \
+ do { \
+ cudaError_t e_ = (x); \
+ if (e_ != cudaSuccess) { \
+ std::fprintf(stderr, "%s:%d %s: %s\n", __FILE__, __LINE__, #x, cudaGetErrorString(e_)); \
+ std::exit(2); \
+ } \
+ } while (0)
+
+struct Q81 { // block_q8_1: fp16 d, fp16 s (= d * sum), 32 int8
+ uint16_t d, s;
+ int8_t qs[32];
+};
+static_assert(sizeof(Q81) == 36, "block_q8_1");
+
+constexpr int64_t kEmbd = nib::kNEmbd, kFf = nib::kNFf;
+constexpr int kTok = 12; // activation rows: T <= 6 for routing plans, 12 for the 0..11-entry plans
+
+// One layer: `ne` GGUF blobs (gate rows | up rows | down rows, as the pack and the host arena hold them) on the host
+// and the device, and their converted slots.
+struct Layer {
+ std::string tag;
+ int layer = -1;
+ cpu::NativeFmt f;
+ nib::LayerVram V;
+ K::NativeExpertLayout Lr, Ln;
+ int ne = 0;
+ std::vector<uint8_t> host; // ne x f.bytes
+ uint8_t* raw = nullptr; // device copy of `host`
+ uint8_t* slots = nullptr; // ne x V.slot_bytes (the last one only `hashed` long: nothing past it is readable)
+};
+
+bool layer_fmt(const strata::GgufModel& m, int l, int& gt, int& dt) {
+ const auto* g = m.find("blk." + std::to_string(l) + ".ffn_gate_exps.weight");
+ const auto* d = m.find("blk." + std::to_string(l) + ".ffn_down_exps.weight");
+ if (g == nullptr || d == nullptr) return false;
+ gt = (int) g->type;
+ dt = (int) d->type;
+ return true;
+}
+
+// The blobs of experts [0, ne) of layer l.
+bool read_layer(const strata::GgufModel& m, int l, int ne, Layer& Y) {
+ const char* roles[3] = { "gate", "up", "down" };
+ const strata::TensorInfo* t[3] = {};
+ size_t sh[3] = {};
+ for (int r = 0; r < 3; ++r) t[r] = m.find("blk." + std::to_string(l) + ".ffn_" + roles[r] + "_exps.weight", &sh[r]);
+ if (!t[0] || !t[1] || !t[2]) return false;
+ std::string err;
+ if (!cpu::native_fmt((int) t[0]->type, (int) t[2]->type, kEmbd, kFf, Y.f, err)) {
+ std::printf("layer %d: %s\n", l, err.c_str());
+ return false;
+ }
+ const size_t gu_role = Y.f.up_off, dn_role = Y.f.bytes - Y.f.down_off;
+ Y.layer = l;
+ Y.ne = ne;
+ Y.host.resize((size_t) ne * Y.f.bytes);
+ const uint8_t* G = m.shard(sh[0]).tensor_data(*t[0]);
+ const uint8_t* U = m.shard(sh[1]).tensor_data(*t[1]);
+ const uint8_t* D = m.shard(sh[2]).tensor_data(*t[2]);
+ for (int e = 0; e < ne; ++e) {
+ uint8_t* b = Y.host.data() + (size_t) e * Y.f.bytes;
+ std::memcpy(b, G + (size_t) e * gu_role, gu_role);
+ std::memcpy(b + Y.f.up_off, U + (size_t) e * gu_role, gu_role);
+ std::memcpy(b + Y.f.down_off, D + (size_t) e * dn_role, dn_role);
+ }
+ return true;
+}
+
+// check (6): nibble::check_gguf over every slot (the start-up check), and nibble::gather_gguf (the prompt path's gather)
+// in groups of 16 and from later first positions, against the blobs byte for byte (and the MMQ tail zeroed).
+// `expect_diff`: the negative control (only the report says so: the counts must then be > 0).
+long long gguf_parity(const Layer& Y, const unsigned long long* dp, bool expect_diff) {
+ long long bad_total = 0;
+ const int ne = Y.ne;
+ unsigned long long* dbad = nullptr;
+ CK(cudaMalloc(&dbad, 8));
+ CK(cudaMemset(dbad, 0, 8));
+ for (int e0 = 0; e0 < ne; e0 += 64) {
+ const int rc = nib::check_gguf(Y.V, Y.raw + (size_t) e0 * Y.V.raw_bytes, std::min(64, ne - e0), dp + e0, dbad, nullptr);
+ if (rc != 0) {
+ std::printf("check_gguf: %s\n", cudaGetErrorString((cudaError_t) rc));
+ std::exit(2);
+ }
+ }
+ CK(cudaDeviceSynchronize());
+ unsigned long long cbad = 0;
+ CK(cudaMemcpy(&cbad, dbad, 8, cudaMemcpyDeviceToHost));
+ CK(cudaFree(dbad));
+ // the gathers: MMQ's group buffers (16 experts of [gate | up] and of down, each followed by the 4096-byte tail)
+ const uint64_t gub = (uint64_t) nib::kRows * Y.f.gu_row, db = Y.V.down_bytes, zero = 4096;
+ uint8_t *ggu = nullptr, *gd = nullptr;
+ CK(cudaMalloc(&ggu, 16 * gub + zero));
+ CK(cudaMalloc(&gd, 16 * db + zero));
+ std::vector<uint8_t> hgu(16 * gub + zero), hd(16 * db + zero);
+ std::vector<unsigned long long> hp(ne);
+ CK(cudaMemcpy(hp.data(), dp, (size_t) ne * 8, cudaMemcpyDeviceToHost));
+ long long gbad = 0, groups = 0;
+ for (int e0 = 0; e0 < ne; e0 += 16) {
+ const int first = (e0 / 16) % 4 == 3 ? 1 + (e0 / 64) % 15 : 0; // every 4th group from a later position
+ const int n = std::min(16, ne - e0);
+ if (first >= n) continue;
+ const uint8_t* sl[16] = {};
+ for (int q = first; q < n; ++q) sl[q] = (const uint8_t*) (uintptr_t) hp[(size_t) (e0 + q)];
+ CK(cudaMemset(ggu, 0xA5, 16 * gub + zero));
+ CK(cudaMemset(gd, 0xA5, 16 * db + zero));
+ const int rc = nib::gather_gguf(Y.V, sl, first, n, ggu, gub, gd, db, zero, nullptr);
+ if (rc != 0) {
+ std::printf("gather_gguf: %s\n", cudaGetErrorString((cudaError_t) rc));
+ std::exit(2);
+ }
+ CK(cudaDeviceSynchronize());
+ CK(cudaMemcpy(hgu.data(), ggu, hgu.size(), cudaMemcpyDeviceToHost));
+ CK(cudaMemcpy(hd.data(), gd, hd.size(), cudaMemcpyDeviceToHost));
+ for (int q = first; q < n; ++q) {
+ const uint8_t* raw = Y.host.data() + (size_t) (e0 + q) * Y.f.bytes;
+ gbad += std::memcmp(hgu.data() + (size_t) q * gub, raw, gub) != 0;
+ gbad += std::memcmp(hd.data() + (size_t) q * db, raw + Y.f.down_off, db) != 0;
+ }
+ for (uint64_t i = 0; i < zero; ++i) gbad += hgu[(size_t) n * gub + i] != 0 || hd[(size_t) n * db + i] != 0 ? 1 : 0;
+ ++groups;
+ }
+ CK(cudaFree(ggu));
+ CK(cudaFree(gd));
+ std::printf(" check_gguf: %llu words differ; gather_gguf (%lld groups, the tail zeroed): %lld matrices or tail bytes differ%s\n",
+ cbad, groups, gbad, expect_diff ? " (NEGATIVE CONTROL: both must be > 0)" : "");
+ bad_total += (long long) cbad + gbad; // the negative control fails the run, as (7) does: it must
+ return bad_total;
+}
+
+// Plans the layer, uploads the blobs, transcodes them into slots (checks 1-6).
+long long convert(Layer& Y, bool ggml_check) {
+ long long bad_total = 0;
+ nib::Config c;
+ c.on = true;
+ Y.V = nib::plan_layer(Y.layer, Y.f, c);
+ Y.Lr = K::native_expert_layout(Y.f.gu_type, Y.f.d_type, kEmbd, kFf);
+ Y.Ln = K::native_expert_layout_nib(Y.f.gu_type, Y.f.d_type, kEmbd, kFf);
+ if (!Y.V.on || Y.V.slot_bytes != Y.Ln.bytes || Y.V.down_off != Y.Ln.down_off || Y.Lr.bytes != Y.f.bytes) {
+ std::printf("layer %d: plan_layer / native_expert_layout_nib disagree (on %d, slot %llu vs %zu)\n", Y.layer,
+ (int) Y.V.on, (unsigned long long) Y.V.slot_bytes, Y.Ln.bytes);
+ return 1;
+ }
+ const int ne = Y.ne;
+ const size_t slots_bytes = (size_t) (ne - 1) * Y.V.slot_bytes + Y.V.hashed;
+ CK(cudaMalloc(&Y.raw, Y.host.size()));
+ CK(cudaMemcpy(Y.raw, Y.host.data(), Y.host.size(), cudaMemcpyHostToDevice));
+ CK(cudaMalloc(&Y.slots, slots_bytes));
+ CK(cudaMemset(Y.slots, 0xA5, slots_bytes));
+ std::vector<unsigned long long> hp(ne);
+ for (int e = 0; e < ne; ++e) hp[e] = (unsigned long long) (Y.slots + (size_t) e * Y.V.slot_bytes);
+ unsigned long long *dp = nullptr, *dbad = nullptr, *dhash = nullptr;
+ CK(cudaMalloc(&dp, (size_t) ne * 8));
+ CK(cudaMalloc(&dbad, 8));
+ CK(cudaMalloc(&dhash, (size_t) ne * 8));
+ CK(cudaMemcpy(dp, hp.data(), (size_t) ne * 8, cudaMemcpyHostToDevice));
+ CK(cudaMemset(dbad, 0, 8));
+ // (1) the startup fill's batches: 64 blobs at a time, then the down rows behind the codes
+ for (int e0 = 0; e0 < ne; e0 += 64) {
+ const int n = std::min(64, ne - e0);
+ const int rc = nib::transcode(Y.V, Y.raw + (size_t) e0 * Y.V.raw_bytes, n, dp + e0, dbad, nullptr);
+ if (rc != 0) {
+ std::printf("transcode: %s\n", cudaGetErrorString((cudaError_t) rc));
+ std::exit(2);
+ }
+ }
+ for (int e = 0; e < ne; ++e)
+ CK(cudaMemcpyAsync(Y.slots + (size_t) e * Y.V.slot_bytes + nib::kDownOff,
+ Y.raw + (size_t) e * Y.V.raw_bytes + Y.V.raw_down_off, Y.V.down_bytes, cudaMemcpyDeviceToDevice));
+ CK(cudaDeviceSynchronize());
+ unsigned long long bad = 0;
+ CK(cudaMemcpy(&bad, dbad, 8, cudaMemcpyDeviceToHost));
+ std::printf("%s layer %2d %-7s / down %-6s: %d experts, blob %zu B, slot %llu B (x%.3f); transcode: %llu of %lld items do not round-trip\n",
+ Y.tag.c_str(), Y.layer, nib::fmt_name(Y.f.gu_type), ggml_type_name((ggml_type) Y.f.d_type), ne, Y.f.bytes,
+ (unsigned long long) Y.V.slot_bytes, (double) Y.V.slot_bytes / (double) Y.f.bytes, bad,
+ (long long) ne * nib::kRows * nib::kItems);
+ bad_total += (long long) bad;
+ bad_total += gguf_parity(Y, dp, false);
+ // (2) every gate/up byte written: the first 8 blobs again into slots pre-filled with 0x5A
+ {
+ const int n8 = std::min(8, ne);
+ uint8_t* alt = nullptr;
+ unsigned long long* ap = nullptr;
+ CK(cudaMalloc(&alt, (size_t) n8 * Y.V.slot_bytes));
+ CK(cudaMemset(alt, 0x5A, (size_t) n8 * Y.V.slot_bytes));
+ std::vector<unsigned long long> ah(n8);
+ for (int e = 0; e < n8; ++e) ah[e] = (unsigned long long) (alt + (size_t) e * Y.V.slot_bytes);
+ CK(cudaMalloc(&ap, (size_t) n8 * 8));
+ CK(cudaMemcpy(ap, ah.data(), (size_t) n8 * 8, cudaMemcpyHostToDevice));
+ CK(cudaMemset(dbad, 0, 8));
+ if (nib::transcode(Y.V, Y.raw, n8, ap, dbad, nullptr) != 0) std::exit(2);
+ CK(cudaDeviceSynchronize());
+ std::vector<uint8_t> a(nib::kGuBytes), b(nib::kGuBytes);
+ long long wbad = 0;
+ for (int e = 0; e < n8; ++e) {
+ CK(cudaMemcpy(a.data(), Y.slots + (size_t) e * Y.V.slot_bytes, nib::kGuBytes, cudaMemcpyDeviceToHost));
+ CK(cudaMemcpy(b.data(), alt + (size_t) e * Y.V.slot_bytes, nib::kGuBytes, cudaMemcpyDeviceToHost));
+ wbad += std::memcmp(a.data(), b.data(), nib::kGuBytes) != 0;
+ }
+ std::printf(" every gate/up byte written (0xA5 vs 0x5A pre-fill, %d slots): %lld differ\n", n8, wbad);
+ bad_total += wbad;
+ CK(cudaFree(alt));
+ CK(cudaFree(ap));
+ }
+ // (3) the device hash of every slot against the host reference over the slot read back
+ const int rc = nib::hash_slots(Y.V, dp, ne, dhash, nullptr);
+ if (rc != 0) {
+ std::printf("hash_slots: %s\n", cudaGetErrorString((cudaError_t) rc));
+ std::exit(2);
+ }
+ std::vector<unsigned long long> hh(ne);
+ CK(cudaMemcpy(hh.data(), dhash, (size_t) ne * 8, cudaMemcpyDeviceToHost));
+ std::vector<uint8_t> sl(slots_bytes);
+ CK(cudaMemcpy(sl.data(), Y.slots, slots_bytes, cudaMemcpyDeviceToHost));
+ std::atomic<long long> hbad{ 0 }, cbad{ 0 }, dnbad{ 0 }, gbad{ 0 };
+ std::atomic<long long> grows{ 0 };
+ std::string first_ggml;
+ std::atomic<int> first_set{ 0 };
+ const ggml_type_traits* tr = ggml_get_type_traits((ggml_type) Y.f.gu_type);
+ const unsigned nth = std::max(1u, std::min(16u, std::thread::hardware_concurrency()));
+ std::vector<std::thread> th;
+ for (unsigned ti = 0; ti < nth; ++ti)
+ th.emplace_back([&, ti]() {
+ std::vector<float> ref((size_t) kEmbd), got((size_t) kEmbd);
+ for (int e = (int) ti; e < ne; e += (int) nth) {
+ const uint8_t* s = sl.data() + (size_t) e * Y.V.slot_bytes;
+ const uint8_t* raw = Y.host.data() + (size_t) e * Y.f.bytes;
+ hbad += nib::hash_host(s, Y.V.hashed) != hh[e];
+ if (e == 0) { // the hash sees every byte: one flipped must change it
+ std::vector<uint8_t> t2(s, s + Y.V.hashed);
+ t2[Y.V.hashed - 1] ^= 1;
+ hbad += nib::hash_host(t2.data(), Y.V.hashed) == hh[e];
+ }
+ dnbad += std::memcmp(s + nib::kDownOff, raw + Y.V.raw_down_off, Y.V.down_bytes) != 0;
+ for (int64_t r = 0; r < nib::kRows; ++r) {
+ // (4) the host codec
+ for (int64_t k = 0; k < nib::kItems; ++k) {
+ uint32_t q[4], q2[4];
+ int8_t g[32];
+ std::memcpy(q, s + nib::code_off(r, k), 16);
+ nib::decode_item(Y.f.gu_type, q, g);
+ const bool ok = nib::encode_item(Y.f.gu_type, g, q2);
+ cbad += !ok || std::memcmp(q, q2, 16) != 0;
+ }
+ if (!ggml_check) continue;
+ // (5) ggml's dequantizer of the GGUF row against the slot's weights through its expression
+ const uint8_t* row = raw + (r < kFf ? (size_t) r * Y.f.gu_row : Y.f.up_off + (size_t) (r - kFf) * Y.f.gu_row);
+ tr->to_float(row, ref.data(), kEmbd);
+ for (int64_t k = 0; k < nib::kItems; ++k) {
+ uint32_t q[4];
+ std::memcpy(q, s + nib::code_off(r, k), 16);
+ int8_t gv[32];
+ nib::decode_item(Y.f.gu_type, q, gv);
+ const int sc = s[nib::scale_off(r, k)];
+ ggml_fp16_t dh;
+ std::memcpy(&dh, s + nib::d_off(r, k / 8), sizeof dh);
+ const float d = ggml_fp16_to_fp32(dh);
+ for (int j = 0; j < 32; ++j) {
+ const float db = Y.f.gu_type == nib::kIQ2_S ? d * (0.5f + ((j < 16 ? sc : sc >> 4) & 0xf)) * 0.25f
+ : Y.f.gu_type == nib::kIQ3_XXS ? d * (0.5f + sc) * 0.5f
+ : d * sc;
+ const int a = gv[j] < 0 ? -(int) gv[j] : (int) gv[j];
+ float v = db * a;
+ if (gv[j] < 0) v = -v;
+ got[(size_t) (k * 32 + j)] = v;
+ }
+ }
+ ++grows;
+ if (std::memcmp(ref.data(), got.data(), ref.size() * sizeof(float)) != 0) {
+ ++gbad;
+ if (first_set.exchange(1) == 0) {
+ size_t cc = 0;
+ while (cc < ref.size() && std::memcmp(&ref[cc], &got[cc], 4) == 0) ++cc;
+ char buf[200];
+ std::snprintf(buf, sizeof buf, "expert %d row %lld column %zu: slot %.9g, GGUF %.9g", e,
+ (long long) r, cc, (double) got[cc], (double) ref[cc]);
+ first_ggml = buf;
+ }
+ }
+ }
+ }
+ });
+ for (auto& t : th) t.join();
+ std::printf(" hash_slots vs hash_host: %lld of %d slots differ (+1 flipped-byte check); down rows: %lld of %d slots differ;"
+ " host codec: %lld of %lld items differ\n",
+ hbad.load(), ne, dnbad.load(), ne, cbad.load(), (long long) ne * nib::kRows * nib::kItems);
+ if (ggml_check)
+ std::printf(" ggml to_float of the GGUF rows vs the slot's weights: %lld of %lld rows differ%s%s\n", gbad.load(),
+ grows.load(), first_ggml.empty() ? "" : "; first: ", first_ggml.c_str());
+ bad_total += hbad + dnbad + cbad + gbad;
+ CK(cudaFree(dp));
+ CK(cudaFree(dbad));
+ CK(cudaFree(dhash));
+ return bad_total;
+}
+
+// ---- plans: synthetic top-10 routing and resident_plan_kernel's grouping (groups by first appearance)
+struct Plan {
+ int ng = 0, ne = 0, ntok = 0, maxn = 0;
+ std::vector<int> gid, start, tok, dst;
+};
+
+void make_routing(std::mt19937& rng, int NE, int T, int share_pct, std::vector<int>& ids) {
+ ids.clear();
+ std::vector<int> prev, cur;
+ for (int t = 0; t < T; ++t) {
+ cur.clear();
+ const bool sh = t > 0 && share_pct > 0;
+ if (sh) {
+ std::vector<int> p = prev;
+ std::shuffle(p.begin(), p.end(), rng);
+ cur.assign(p.begin(), p.begin() + 10 * share_pct / 100);
+ }
+ while ((int) cur.size() < 10) {
+ const int e = (int) (rng() % (unsigned) NE);
+ if (std::find(cur.begin(), cur.end(), e) != cur.end()) continue;
+ if (sh && std::find(prev.begin(), prev.end(), e) != prev.end()) continue;
+ cur.push_back(e);
+ }
+ std::shuffle(cur.begin(), cur.end(), rng);
+ ids.insert(ids.end(), cur.begin(), cur.end());
+ prev = cur;
+ }
+}
+
+Plan build_plan(const std::vector<int>& ids) {
+ const int n = (int) ids.size();
+ Plan p;
+ std::vector<int> first(n), cnt, grp_of(n, -1);
+ for (int j = 0; j < n; ++j) {
+ int f = j;
+ for (int i = 0; i < j; ++i)
+ if (ids[i] == ids[j]) { f = i; break; }
+ first[j] = f;
+ }
+ for (int j = 0; j < n; ++j)
+ if (first[j] == j) {
+ grp_of[j] = p.ng++;
+ p.gid.push_back(ids[j]);
+ cnt.push_back(0);
+ }
+ for (int j = 0; j < n; ++j) cnt[grp_of[first[j]]]++;
+ p.start.assign(p.ng + 1, 0);
+ int acc = 0;
+ for (int g = 0; g < p.ng; ++g) {
+ p.start[g] = acc;
+ acc += cnt[g];
+ p.maxn = std::max(p.maxn, cnt[g]);
+ }
+ p.start[p.ng] = acc;
+ p.tok.assign(n, 0);
+ p.dst.assign(n, 0);
+ std::vector<int> fill(p.ng, 0);
+ for (int j = 0; j < n; ++j) {
+ const int g = grp_of[first[j]];
+ const int at = p.start[g] + fill[g]++;
+ p.tok[at] = j / 10;
+ p.dst[at] = j;
+ }
+ p.ne = n;
+ p.ntok = n / 10;
+ return p;
+}
+
+// one group of T entries (G = 1): an expert every token of the window routes to
+Plan single_group(std::mt19937& rng, int NE, int T) {
+ Plan p;
+ p.ntok = T;
+ p.ng = 1;
+ p.gid.push_back((int) (rng() % (unsigned) NE));
+ p.start = { 0, T };
+ for (int t = 0; t < T; ++t) {
+ p.tok.push_back(t);
+ p.dst.push_back(T - 1 - t);
+ }
+ p.ne = T;
+ p.maxn = T;
+ return p;
+}
+
+// groups of 0..11 entries (more than one GRP_NC pass, empty groups), 12 tokens, distinct tokens within a group,
+// destination rows in any order
+Plan synthetic_plan(std::mt19937& rng, int NE) {
+ Plan p;
+ p.ntok = kTok;
+ std::vector<int> ids(NE);
+ for (int i = 0; i < NE; ++i) ids[i] = i;
+ std::shuffle(ids.begin(), ids.end(), rng);
+ const int sizes[] = { 3, 0, 11, 1, 5, 8, 0, 2, 4, 9, 6, 7, 10, 1 };
+ int acc = 0;
+ for (int g = 0; g < (int) (sizeof sizes / sizeof sizes[0]); ++g) {
+ p.gid.push_back(ids[(size_t) g % ids.size()]);
+ p.start.push_back(acc);
+ std::vector<int> toks(kTok);
+ for (int i = 0; i < kTok; ++i) toks[i] = i;
+ std::shuffle(toks.begin(), toks.end(), rng);
+ std::sort(toks.begin(), toks.begin() + sizes[g]);
+ for (int i = 0; i < sizes[g]; ++i) {
+ p.tok.push_back(toks[i]);
+ p.dst.push_back(acc + i);
+ }
+ acc += sizes[g];
+ p.maxn = std::max(p.maxn, sizes[g]);
+ }
+ p.start.push_back(acc);
+ p.ng = (int) p.gid.size();
+ p.ne = acc;
+ std::vector<int> perm(acc);
+ for (int i = 0; i < acc; ++i) perm[i] = i;
+ std::shuffle(perm.begin(), perm.end(), rng);
+ for (int i = 0; i < acc; ++i) p.dst[i] = perm[i];
+ return p;
+}
+
+std::vector<Q81> make_x(std::mt19937& rng) {
+ std::vector<Q81> x((size_t) kTok * (kEmbd / 32));
+ for (size_t i = 0; i < x.size(); ++i) {
+ Q81& b = x[i];
+ const int tok = (int) (i / (kEmbd / 32));
+ const float d = 0.002f + 0.03f * (float) (rng() % 1000) / 1000.0f;
+ int sum = 0;
+ for (int j = 0; j < 32; ++j) {
+ int q = (int) (rng() % 255) - 127;
+ if (tok == 1) q = 127; // the sum of x at its extremes (+-4064)
+ if (tok == 2) q = -127;
+ if (tok == 3 && (i % (kEmbd / 32)) < 8) q = (j & 1) ? 127 : -127;
+ b.qs[j] = (int8_t) q;
+ sum += q;
+ }
+ b.d = ggml_fp32_to_fp16(d);
+ b.s = ggml_fp32_to_fp16(d * (float) sum);
+ }
+ return x;
+}
+
+struct Dev {
+ unsigned long long* ptr = nullptr;
+ int32_t *start = nullptr, *ng = nullptr, *tok = nullptr, *dst = nullptr;
+};
+Dev upload(const Plan& p, const Layer& Y, bool conv) {
+ Dev d;
+ std::vector<unsigned long long> hp(std::max(1, p.ng));
+ for (int g = 0; g < p.ng; ++g)
+ hp[g] = conv ? (unsigned long long) (Y.slots + (size_t) p.gid[g] * Y.V.slot_bytes)
+ : (unsigned long long) (Y.raw + (size_t) p.gid[g] * Y.f.bytes);
+ CK(cudaMalloc(&d.ptr, (size_t) std::max(1, p.ng) * 8));
+ CK(cudaMalloc(&d.start, (size_t) (p.ng + 1) * 4));
+ CK(cudaMalloc(&d.ng, 4));
+ CK(cudaMalloc(&d.tok, (size_t) std::max(1, p.ne) * 4));
+ CK(cudaMalloc(&d.dst, (size_t) std::max(1, p.ne) * 4));
+ CK(cudaMemcpy(d.ptr, hp.data(), hp.size() * 8, cudaMemcpyHostToDevice));
+ CK(cudaMemcpy(d.start, p.start.data(), (size_t) (p.ng + 1) * 4, cudaMemcpyHostToDevice));
+ CK(cudaMemcpy(d.ng, &p.ng, 4, cudaMemcpyHostToDevice));
+ if (p.ne) {
+ CK(cudaMemcpy(d.tok, p.tok.data(), (size_t) p.ne * 4, cudaMemcpyHostToDevice));
+ CK(cudaMemcpy(d.dst, p.dst.data(), (size_t) p.ne * 4, cudaMemcpyHostToDevice));
+ }
+ return d;
+}
+void release(Dev& d) {
+ CK(cudaFree(d.ptr));
+ CK(cudaFree(d.start));
+ CK(cudaFree(d.ng));
+ CK(cudaFree(d.tok));
+ CK(cudaFree(d.dst));
+}
+
+struct Outs {
+ std::vector<uint8_t> gate, up, h, hq, out;
+};
+// one native_expert_grouped call; scratch and out pre-filled with `fill` so an unwritten value shows
+Outs run(const Layer& Y, const Plan& p, bool conv, const void* dx, void* scr, float* dout, int64_t gy, int mode,
+ uint8_t fill) {
+ // the window's capacity, as verify sizes it: T x 10 groups and entries (the blocks past the plan's groups idle)
+ const int64_t cap_e = std::max<int64_t>((int64_t) p.ntok * 10, std::max(p.ne, p.ng)), cap_g = cap_e;
+ Dev d = upload(p, Y, conv);
+ CK(cudaMemset(scr, fill, K::native_expert_scratch_bytes(cap_e, kFf)));
+ CK(cudaMemset(dout, fill, (size_t) cap_e * kEmbd * 4));
+ K::native_expert_set_mode(mode, 0);
+ K::native_expert_grouped(conv ? Y.Ln : Y.Lr, d.ptr, d.start, d.ng, d.dst, d.tok, cap_g, cap_e, dx, scr, dout, nullptr, gy);
+ K::native_expert_set_mode(-1, 0);
+ CK(cudaGetLastError());
+ CK(cudaDeviceSynchronize());
+ const size_t f = (size_t) cap_e * kFf * 4, fa = (f + 255) & ~(size_t) 255;
+ Outs o;
+ o.gate.resize((size_t) p.ne * kFf * 4);
+ o.up.resize(o.gate.size());
+ o.h.resize(o.gate.size());
+ o.hq.resize((size_t) p.ne * (kFf / 32) * 36);
+ o.out.resize((size_t) p.ne * kEmbd * 4);
+ const uint8_t* s = (const uint8_t*) scr;
+ if (p.ne) {
+ CK(cudaMemcpy(o.gate.data(), s, o.gate.size(), cudaMemcpyDeviceToHost));
+ CK(cudaMemcpy(o.up.data(), s + fa, o.up.size(), cudaMemcpyDeviceToHost));
+ CK(cudaMemcpy(o.h.data(), s + 2 * fa, o.h.size(), cudaMemcpyDeviceToHost));
+ CK(cudaMemcpy(o.hq.data(), s + 3 * fa, o.hq.size(), cudaMemcpyDeviceToHost));
+ std::vector<uint8_t> all((size_t) cap_e * kEmbd * 4);
+ CK(cudaMemcpy(all.data(), dout, all.size(), cudaMemcpyDeviceToHost));
+ for (int e = 0; e < p.ne; ++e)
+ std::memcpy(o.out.data() + (size_t) e * kEmbd * 4, all.data() + (size_t) p.dst[e] * kEmbd * 4, kEmbd * 4);
+ }
+ release(d);
+ return o;
+}
+
+long long diff_rows(const std::vector<uint8_t>& a, const std::vector<uint8_t>& b, size_t row) {
+ long long n = 0;
+ for (size_t i = 0; i + row <= a.size(); i += row) n += std::memcmp(a.data() + i, b.data() + i, row) != 0;
+ return n;
+}
+
+// check (7): every plan, both grid strides, every kernel choice, both layouts, v1 off / on, against the GGUF blobs
+// through the production kernels
+long long grouped_parity(const Layer& Y, std::mt19937& rng, int plans, const void* dx, void* scr, float* dout) {
+ struct Case {
+ Plan p;
+ int T;
+ };
+ std::vector<Case> cases;
+ for (int T = 1; T <= 6; ++T) {
+ for (int share : { 0, 50 })
+ for (int i = 0; i < plans; ++i) {
+ std::vector<int> ids;
+ make_routing(rng, Y.ne, T, share, ids);
+ cases.push_back({ build_plan(ids), T });
+ }
+ cases.push_back({ single_group(rng, Y.ne, T), T });
+ }
+ for (int i = 0; i < 4; ++i) cases.push_back({ synthetic_plan(rng, Y.ne), kTok });
+ int hist[13] = {};
+ for (const Case& cs : cases)
+ for (int g = 0; g < cs.p.ng; ++g) hist[std::min(12, cs.p.start[g + 1] - cs.p.start[g])]++;
+ struct Var {
+ int v, mode;
+ bool v1;
+ const char* name;
+ };
+ // mode -1: the card's default layout (a GP100: mode 8 - its reference gate/up kernel writes hq, not gate / up);
+ // mode 0: the CUDA layout
+ const Var vars[] = { { -1, -1, false, "default, card" }, { 0, -1, false, "S, card" }, { 1, -1, false, "P, card" },
+ { -1, -1, true, "default, card, V1" }, { -1, 0, false, "default, mode 0" },
+ { 0, 0, false, "S, mode 0" }, { 1, 0, false, "P, mode 0" }, { -1, 0, true, "default, mode 0, V1" },
+ // STRATA_NIB_GU_PROTO (nib_gu_proto.cuh): the staged-once kernel, id 31
+ { 31, -1, false, "gu-proto 31, card" }, { 31, 0, false, "gu-proto 31, mode 0" },
+ { 31, 0, true, "gu-proto 31, mode 0, V1" } };
+ constexpr int NV = (int) (sizeof vars / sizeof vars[0]);
+ long long rows[NV] = {}, bad[NV][5] = {}, total = 0;
+ for (const Case& cs : cases)
+ for (int64_t gy : { (int64_t) 0, (int64_t) 4 }) {
+ Outs ref[2][2]; // [mode 0][v1]
+ bool have[2][2] = {};
+ for (int vi = 0; vi < NV; ++vi) {
+ const int m0 = vars[vi].mode == 0 ? 1 : 0, r1 = vars[vi].v1 ? 1 : 0;
+ K::native_grouped_set_v1(vars[vi].v1);
+ if (!have[m0][r1]) {
+ K::native_expert_nib_set_variant(-1);
+ ref[m0][r1] = run(Y, cs.p, false, dx, scr, dout, gy, vars[vi].mode, 0xFF);
+ have[m0][r1] = true;
+ }
+ K::native_expert_nib_set_variant(vars[vi].v);
+ const Outs o = run(Y, cs.p, true, dx, scr, dout, gy, vars[vi].mode, 0x00);
+ rows[vi] += cs.p.ne;
+ // gate / up: written by both arms in the CUDA layout, and by every arm of a striding call (gy 4: mode 0)
+ const bool gu = vars[vi].mode == 0 || gy == 4;
+ if (gu) {
+ bad[vi][0] += diff_rows(ref[m0][r1].gate, o.gate, kFf * 4);
+ bad[vi][1] += diff_rows(ref[m0][r1].up, o.up, kFf * 4);
+ }
+ if (vars[vi].v1 && gu) bad[vi][2] += diff_rows(ref[m0][r1].h, o.h, kFf * 4);
+ bad[vi][3] += diff_rows(ref[m0][r1].hq, o.hq, (kFf / 32) * 36);
+ bad[vi][4] += diff_rows(ref[m0][r1].out, o.out, kEmbd * 4);
+ }
+ }
+ K::native_expert_nib_set_variant(-1);
+ K::native_grouped_set_v1(false);
+ std::printf(" plans: %zu (T 1..6 x sharing 0/50%% x %d, G=1 per T, 4 of 0..11 entries); groups by entries:", cases.size(),
+ plans);
+ for (int n = 0; n <= 12; ++n) std::printf(" %d:%d", n, hist[n]);
+ std::printf("\n");
+ for (int vi = 0; vi < NV; ++vi) {
+ const long long b = bad[vi][0] + bad[vi][1] + bad[vi][2] + bad[vi][3] + bad[vi][4];
+ std::printf(" native_expert_grouped %-20s: %6lld entries x 2 strides, rows differing: gate %lld, up %lld, h %s, hq %lld, out %lld\n",
+ vars[vi].name, rows[vi], bad[vi][0], bad[vi][1],
+ vars[vi].v1 ? std::to_string(bad[vi][2]).c_str() : "-", bad[vi][3], bad[vi][4]);
+ total += b;
+ }
+ return total;
+}
+
+// --synthetic: random gate/up blocks, d cycling through the fp16 extremes; the coverage of each field is counted
+long long synthesize(Layer& Y, std::mt19937& rng) {
+ static const uint16_t kD[] = { 0x0001, 0x8001, 0x7BFF, 0xFBFF, 0x3C00, 0xBC00, 0x0400, 0x1A00, 0x2400 };
+ const int gt = Y.f.gu_type;
+ const size_t bb = gt == nib::kIQ2_S ? 82 : gt == nib::kIQ3_XXS ? 98 : 110; // block_iq2_s / iq3_xxs / iq3_s
+ const size_t nblk = (size_t) nib::kRows * nib::kSuper;
+ std::set<int> grid, sign, scale, dset;
+ for (int e = 0; e < Y.ne; ++e) {
+ uint8_t* b = Y.host.data() + (size_t) e * Y.f.bytes;
+ for (size_t i = 0; i < 2 * Y.f.up_off; ++i) b[i] = (uint8_t) rng();
+ for (size_t k = 0; k < nblk; ++k) {
+ uint8_t* blk = b + k * bb;
+ const uint16_t d = kD[((size_t) e * nblk + k) % (sizeof kD / sizeof kD[0])];
+ std::memcpy(blk, &d, 2);
+ dset.insert(d);
+ const uint8_t* qs = blk + 2;
+ for (int ib = 0; ib < 8; ++ib) {
+ if (gt == nib::kIQ2_S) {
+ const uint8_t* qh = blk + 66;
+ const uint8_t* sc = blk + 74;
+ for (int l = 0; l < 4; ++l) {
+ grid.insert(qs[4 * ib + l] | ((qh[ib] << (8 - 2 * l)) & 0x300));
+ sign.insert(qs[32 + 4 * ib + l]);
+ }
+ scale.insert(sc[ib] & 15);
+ scale.insert(sc[ib] >> 4);
+ } else if (gt == nib::kIQ3_XXS) {
+ uint32_t aux;
+ std::memcpy(&aux, blk + 66 + 4 * ib, 4);
+ for (int l = 0; l < 8; ++l) grid.insert(qs[8 * ib + l]);
+ for (int l = 0; l < 4; ++l) sign.insert((int) ((aux >> (7 * l)) & 127));
+ scale.insert((int) (aux >> 28));
+ } else {
+ const uint8_t qh = blk[66 + ib];
+ const uint8_t* sg = blk + 74;
+ const uint8_t* sc = blk + 106;
+ for (int l = 0; l < 4; ++l) {
+ grid.insert(qs[8 * ib + 2 * l] | ((qh << (8 - 2 * l)) & 256));
+ grid.insert(qs[8 * ib + 2 * l + 1] | ((qh << (7 - 2 * l)) & 256));
+ sign.insert(sg[4 * ib + l]);
+ }
+ scale.insert((sc[ib / 2] >> (4 * (ib & 1))) & 15);
+ }
+ }
+ }
+ }
+ const size_t want_grid = gt == nib::kIQ2_S ? 1024 : gt == nib::kIQ3_XXS ? 256 : 512;
+ const size_t want_sign = gt == nib::kIQ3_XXS ? 128 : 256;
+ const bool ok = grid.size() == want_grid && sign.size() == want_sign && scale.size() == 16 &&
+ dset.size() == sizeof kD / sizeof kD[0];
+ std::printf("synthetic %-7s: %d experts, coverage: grid indices %zu/%zu, sign %s %zu/%zu, scale nibbles %zu/16, d values %zu/%zu"
+ " (+-min subnormal, +-65504, +-1, min normal, 2 ordinary) %s\n",
+ nib::fmt_name(gt), Y.ne, grid.size(), want_grid, gt == nib::kIQ3_XXS ? "indices" : "bytes", sign.size(),
+ want_sign, scale.size(), dset.size(), sizeof kD / sizeof kD[0], ok ? "complete" : "INCOMPLETE");
+ return ok ? 0 : 1;
+}
+
+std::vector<int> split_ints(const std::string& s) {
+ std::vector<int> v;
+ size_t p = 0;
+ while (p < s.size()) {
+ v.push_back(std::atoi(s.c_str() + p));
+ p = s.find(',', p);
+ if (p == std::string::npos) break;
+ ++p;
+ }
+ return v;
+}
+
+} // namespace
+
+int main(int argc, char** argv) {
+ setvbuf(stdout, nullptr, _IONBF, 0);
+ if (argc == 2 && std::strcmp(argv[1], "--header") == 0) return header_check();
+ std::vector<std::string> shards;
+ int per_fmt = 2, NE = 512, plans = 8, syn = 64;
+ bool ggml_check = true, negative = false, syn_only = false;
+ std::vector<int> only;
+ for (int i = 1; i < argc; ++i) {
+ const std::string a = argv[i];
+ auto next = [&]() -> const char* {
+ if (i + 1 >= argc) {
+ std::fprintf(stderr, "%s needs a value\n", a.c_str());
+ std::exit(2);
+ }
+ return argv[++i];
+ };
+ if (a == "--layers-per-fmt") per_fmt = std::atoi(next());
+ else if (a == "--layers") only = split_ints(next());
+ else if (a == "--ne") NE = std::atoi(next());
+ else if (a == "--plans") plans = std::atoi(next());
+ else if (a == "--synthetic") syn = std::atoi(next());
+ else if (a == "--no-ggml") ggml_check = false;
+ else if (a == "--negative") negative = true;
+ else if (a == "--synthetic-only") syn_only = true;
+ else if (a.size() > 5 && a.substr(a.size() - 5) == ".gguf") shards.push_back(a);
+ else {
+ std::fprintf(stderr,
+ "usage: iq_nibble_parity --header\n"
+ " iq_nibble_parity <shard.gguf> [<shard.gguf> ...] [--layers-per-fmt 2] [--layers 1,17] [--ne 512]\n"
+ " [--plans 8] [--synthetic 64] [--no-ggml] [--negative]\n"
+ " iq_nibble_parity --synthetic-only [--synthetic 64] [--plans 8] [--no-ggml]\n");
+ return 2;
+ }
+ }
+ if ((shards.empty() && !syn_only) || NE < 16 || NE > 512) {
+ std::fprintf(stderr, "iq_nibble_parity: give at least one model shard (.gguf); --ne 16..512\n");
+ return 2;
+ }
+ CK(cudaSetDevice(0));
+ cudaDeviceProp prop{};
+ CK(cudaGetDeviceProperties(&prop, 0));
+ char bus[64] = {};
+ CK(cudaDeviceGetPCIBusId(bus, sizeof bus, 0));
+ std::printf("device 0: %s, bus %s, cc %d.%d, %d SMs\n", prop.name, bus, prop.major, prop.minor, prop.multiProcessorCount);
+ if (prop.major != 6 || prop.minor != 0) {
+ std::fprintf(stderr, "iq_nibble_parity: the 4-bit kernels are the sm_60 build's; this GPU is cc %d.%d (skipped)\n",
+ prop.major, prop.minor);
+ return 77;
+ }
+ if (!STRATA_EXPERT_NIBBLE_BUILT || nib::prepare(nullptr) != 0) {
+ std::fprintf(stderr, "iq_nibble_parity: no 4-bit kernels in this build, or the GGUF codebooks are not unique\n");
+ return 2;
+ }
+ std::mt19937 rng(20261019);
+ void* scr = nullptr;
+ float* dout = nullptr;
+ void* dx = nullptr;
+ const int64_t capmax = (int64_t) kTok * 10;
+ CK(cudaMalloc(&scr, K::native_expert_scratch_bytes(capmax, kFf)));
+ CK(cudaMalloc((void**) &dout, (size_t) capmax * kEmbd * 4));
+ const std::vector<Q81> hx = make_x(rng);
+ CK(cudaMalloc(&dx, hx.size() * sizeof(Q81)));
+ CK(cudaMemcpy(dx, hx.data(), hx.size() * sizeof(Q81), cudaMemcpyHostToDevice));
+
+ if (syn_only) { // no model: every gate/up format with each down type, random experts
+ long long total_s = 0;
+ int done = 0;
+ for (const int gt : {nib::kIQ2_S, nib::kIQ3_XXS, nib::kIQ3_S})
+ for (const int dt : {20, 42}) { // the packs' down types: IQ4_NL, Q2_0
+ Layer S;
+ std::string err;
+ if (!cpu::native_fmt(gt, dt, kEmbd, kFf, S.f, err)) {
+ std::printf("%s / down type %d: %s\n", nib::fmt_name(gt), dt, err.c_str());
+ ++total_s;
+ continue;
+ }
+ S.tag = "syn";
+ S.layer = 0;
+ S.ne = std::max(16, std::min(syn, NE));
+ S.host.resize((size_t) S.ne * S.f.bytes);
+ for (auto& b : S.host) b = (uint8_t) rng();
+ // block_iq4_nl and block_q2_0 are 18 bytes, the fp16 scale first: an ordinary scale in every block of the down rows
+ for (int e = 0; e < S.ne; ++e)
+ for (size_t o = S.f.down_off; o + 18 <= S.f.bytes; o += 18) {
+ const uint16_t sc = (uint16_t) ((9u << 10) | (rng() & 0x3FFu));
+ std::memcpy(S.host.data() + (size_t) e * S.f.bytes + o, &sc, 2);
+ }
+ long long sb = synthesize(S, rng);
+ sb += convert(S, ggml_check);
+ sb += grouped_parity(S, rng, plans, dx, scr, dout);
+ total_s += sb;
+ ++done;
+ CK(cudaFree(S.raw));
+ CK(cudaFree(S.slots));
+ }
+ std::printf("\n%s: %d synthetic format pairs, %lld differences\n", total_s == 0 && done == 6 ? "PASS" : "FAIL", done, total_s);
+ return total_s == 0 && done == 6 ? 0 : 1;
+ }
+ long long total = 0;
+ int layers_done = 0;
+ std::vector<int> syn_done;
+ for (size_t si = 0; si < shards.size(); ++si) {
+ std::unique_ptr<strata::GgufModel> model;
+ try {
+ model = std::make_unique<strata::GgufModel>(strata::GgufModel::open(shards[si]));
+ } catch (const std::exception& ex) {
+ std::fprintf(stderr, "iq_nibble_parity: %s\n", ex.what());
+ return 2;
+ }
+ const std::string tag(1, (char) ('A' + si));
+ std::printf("\n#### model %s: %s (%zu shards)\n", tag.c_str(), shards[si].c_str(), model->size());
+ std::vector<int> pick;
+ int count[64] = {};
+ for (int l = 0; l < 256; ++l) {
+ int gt = -1, dt = -1;
+ if (!layer_fmt(*model, l, gt, dt)) break;
+ if (!nib::fmt_built(gt)) continue;
+ const bool want = only.empty() ? count[gt]++ < per_fmt : std::find(only.begin(), only.end(), l) != only.end();
+ if (want) pick.push_back(l);
+ }
+ for (int l : pick) {
+ Layer Y;
+ Y.tag = tag;
+ if (!read_layer(*model, l, NE, Y)) {
+ std::printf("layer %d: cannot read its experts\n", l);
+ ++total;
+ continue;
+ }
+ long long b = convert(Y, ggml_check);
+ if (negative) { // the control: one code byte changed per slot (item 3 of row 5), which (6) must see
+ for (int e = 0; e < Y.ne; ++e) {
+ uint8_t* at = Y.slots + (size_t) e * Y.V.slot_bytes + nib::code_off(5, 3);
+ uint8_t v = 0;
+ CK(cudaMemcpy(&v, at, 1, cudaMemcpyDeviceToHost));
+ v ^= 0x11;
+ CK(cudaMemcpy(at, &v, 1, cudaMemcpyHostToDevice));
+ }
+ std::printf(" NEGATIVE CONTROL: one code byte changed per slot - the rows below must differ\n");
+ std::vector<unsigned long long> hp(Y.ne);
+ for (int e = 0; e < Y.ne; ++e) hp[e] = (unsigned long long) (Y.slots + (size_t) e * Y.V.slot_bytes);
+ unsigned long long* dp = nullptr;
+ CK(cudaMalloc(&dp, (size_t) Y.ne * 8));
+ CK(cudaMemcpy(dp, hp.data(), (size_t) Y.ne * 8, cudaMemcpyHostToDevice));
+ b += gguf_parity(Y, dp, true);
+ CK(cudaFree(dp));
+ }
+ b += grouped_parity(Y, rng, plans, dx, scr, dout);
+ total += b;
+ ++layers_done;
+ // the synthetic experts of this format, once: this layer's blobs with random gate/up blocks
+ if (syn > 0 && std::find(syn_done.begin(), syn_done.end(), Y.f.gu_type) == syn_done.end() && !negative) {
+ syn_done.push_back(Y.f.gu_type);
+ Layer S;
+ S.tag = "syn";
+ S.layer = Y.layer;
+ S.f = Y.f;
+ S.ne = std::min(syn, Y.ne);
+ S.host.assign(Y.host.begin(), Y.host.begin() + (ptrdiff_t) ((size_t) S.ne * Y.f.bytes));
+ long long sb = synthesize(S, rng);
+ sb += convert(S, ggml_check);
+ sb += grouped_parity(S, rng, std::max(1, plans / 2), dx, scr, dout);
+ total += sb;
+ CK(cudaFree(S.raw));
+ CK(cudaFree(S.slots));
+ }
+ CK(cudaFree(Y.raw));
+ CK(cudaFree(Y.slots));
+ }
+ }
+ std::printf("\n%s: %d layers (+ %zu synthetic formats), %lld differences\n", total == 0 ? "PASS" : "FAIL", layers_done,
+ syn_done.size(), total);
+ return total == 0 ? 0 : 1;
+}
diff --git a/src/kernels/native_expert_bench.cpp b/src/kernels/native_expert_bench.cpp
index 3263ee7ae..e51bf7742 100644
--- a/src/kernels/native_expert_bench.cpp
+++ b/src/kernels/native_expert_bench.cpp
@@ -2,11 +2,21 @@
// another, bit for bit, and each one's time.
//
// build/native_expert_bench <shard1.gguf> <layer[,layer...]> <groups> <tokens per group> <ref mode> <mode> [iters]
+// build/native_expert_bench --nibble <shard.gguf> <layer[,layer...]> [--t 1,2,3,4,5,6] [--share 0,50] [--plans 200]
+// [--trials 7] [--mode -1|0] [--variant -1|0|1]
//
// Modes are STRATA_EXP_MODE values. A group is one expert (distinct blobs, ~2 MB each, so G >= 8 does not fit
// in L2); its entries read distinct tokens of an 8-token window. The output must be bitwise equal to the
// reference mode's: the verify window's text depends on it.
+//
+// --nibble (STRATA_EXPERT_NIBBLE, strata/core/expert_nibble.hpp): the layer's 512 experts as GGUF blobs (P, the
+// production kernels) and as 4-bit slots made by the engine's transcoder (N: native_expert_layout_nib); the verify
+// window's whole expert step (gate/up + SwiGLU/q8_1 + down) and its gate/up phase alone, per (T, sharing) over
+// `--plans` top-10 routings (0% or 50% of a token's experts shared with the previous token's), every plan's out rows
+// compared byte for byte first; interleaved trials, medians. `--mode`: both arms' layout (-1 the card's default, a
+// GP100's mode 8; 0 the CUDA layout); `--variant`: the 4-bit gate/up kernel (-1 per format, 0 S, 1 P).
#include "strata/artifact/gguf_reader.hpp"
+#include "strata/core/expert_nibble.hpp"
#include "strata/kernels/cpu/native_expert.hpp"
#include "strata/kernels/iq_kernels.hpp"
@@ -20,6 +30,8 @@
#include <cstdio>
#include <cstdlib>
#include <cstring>
+#include <map>
+#include <memory>
#include <random>
#include <sstream>
#include <string>
@@ -28,8 +40,11 @@
namespace cpu = strata::kernels::cpu;
namespace K = strata::kernels;
+int nibble_main(int argc, char** argv);
+
int main(int argc, char** argv) {
setvbuf(stdout, nullptr, _IONBF, 0);
+ if (argc >= 2 && std::strcmp(argv[1], "--nibble") == 0) return nibble_main(argc, argv);
if (argc < 7) {
std::fprintf(stderr, "usage: native_expert_bench <shard1.gguf> <layer[,layer]> <groups> <tokens> <ref mode> <mode> [iters]\n");
return 2;
@@ -175,3 +190,407 @@ int main(int argc, char** argv) {
std::printf("native_expert_bench: %d differing\n", failures);
return failures ? 1 : 0;
}
+
+// ------------------------------------------------------------------------------------------------ --nibble
+#if STRATA_EXPERT_NIBBLE_BUILT
+namespace {
+
+namespace nib = strata::core::nibble;
+
+#define NB_CK(x) \
+ do { \
+ cudaError_t e_ = (x); \
+ if (e_ != cudaSuccess) { \
+ std::fprintf(stderr, "%s:%d %s: %s\n", __FILE__, __LINE__, #x, cudaGetErrorString(e_)); \
+ std::exit(2); \
+ } \
+ } while (0)
+
+struct NbPlan {
+ int ng = 0, ne = 0, t = 0;
+ std::vector<int> gid, start, tok, dst;
+};
+
+// synthetic routing: t tokens of top-10, `share` % of each token's experts the previous token's; groups by first
+// appearance (resident_plan_kernel's order), entries in token order
+NbPlan nb_route(std::mt19937& rng, int t, int share) {
+ std::vector<int> ids, prev, cur;
+ for (int k = 0; k < t; ++k) {
+ cur.clear();
+ const bool sh = k > 0 && share > 0;
+ if (sh) {
+ std::vector<int> p = prev;
+ std::shuffle(p.begin(), p.end(), rng);
+ cur.assign(p.begin(), p.begin() + 10 * share / 100);
+ }
+ while ((int) cur.size() < 10) {
+ const int e = (int) (rng() % 512u);
+ if (std::find(cur.begin(), cur.end(), e) != cur.end()) continue;
+ if (sh && std::find(prev.begin(), prev.end(), e) != prev.end()) continue;
+ cur.push_back(e);
+ }
+ std::shuffle(cur.begin(), cur.end(), rng);
+ ids.insert(ids.end(), cur.begin(), cur.end());
+ prev = cur;
+ }
+ NbPlan p;
+ p.t = t;
+ std::map<int, std::vector<int>> by; // expert -> its entries (j = token * 10 + k)
+ std::vector<int> first;
+ for (int j = 0; j < (int) ids.size(); ++j) {
+ if (by.find(ids[j]) == by.end()) first.push_back(ids[j]);
+ by[ids[j]].push_back(j);
+ }
+ for (int e : first) {
+ p.gid.push_back(e);
+ p.start.push_back(p.ne);
+ for (int j : by[e]) {
+ p.tok.push_back(j / 10);
+ p.dst.push_back(j);
+ ++p.ne;
+ }
+ }
+ p.start.push_back(p.ne);
+ p.ng = (int) p.gid.size();
+ return p;
+}
+
+struct NbDev {
+ unsigned long long* ptr = nullptr;
+ int32_t *start = nullptr, *ng = nullptr, *tok = nullptr, *dst = nullptr;
+};
+NbDev nb_upload(const NbPlan& p, const uint8_t* base, size_t stride) {
+ NbDev d;
+ std::vector<unsigned long long> hp(std::max(1, p.ng));
+ for (int g = 0; g < p.ng; ++g) hp[g] = (unsigned long long) (base + (size_t) p.gid[g] * stride);
+ NB_CK(cudaMalloc((void**) &d.ptr, hp.size() * 8));
+ NB_CK(cudaMalloc((void**) &d.start, (size_t) (p.ng + 1) * 4));
+ NB_CK(cudaMalloc((void**) &d.ng, 4));
+ NB_CK(cudaMalloc((void**) &d.tok, (size_t) std::max(1, p.ne) * 4));
+ NB_CK(cudaMalloc((void**) &d.dst, (size_t) std::max(1, p.ne) * 4));
+ NB_CK(cudaMemcpy(d.ptr, hp.data(), hp.size() * 8, cudaMemcpyHostToDevice));
+ NB_CK(cudaMemcpy(d.start, p.start.data(), (size_t) (p.ng + 1) * 4, cudaMemcpyHostToDevice));
+ NB_CK(cudaMemcpy(d.ng, &p.ng, 4, cudaMemcpyHostToDevice));
+ if (p.ne) {
+ NB_CK(cudaMemcpy(d.tok, p.tok.data(), (size_t) p.ne * 4, cudaMemcpyHostToDevice));
+ NB_CK(cudaMemcpy(d.dst, p.dst.data(), (size_t) p.ne * 4, cudaMemcpyHostToDevice));
+ }
+ return d;
+}
+void nb_free(NbDev& d) {
+ cudaFree(d.ptr);
+ cudaFree(d.start);
+ cudaFree(d.ng);
+ cudaFree(d.tok);
+ cudaFree(d.dst);
+}
+
+std::vector<int> nb_ints(const std::string& s) {
+ std::vector<int> v;
+ std::stringstream ss(s);
+ std::string t;
+ while (std::getline(ss, t, ',')) v.push_back(std::atoi(t.c_str()));
+ return v;
+}
+
+} // namespace
+
+int nibble_main(int argc, char** argv) {
+ if (argc < 4) {
+ std::fprintf(stderr, "usage: native_expert_bench --nibble <shard.gguf> <layer[,layer]> [--t 1,2,3,4,5,6] [--share 0,50] "
+ "[--plans 200] [--trials 7] [--mode -1|0] [--variant -1|0|1]\n");
+ return 2;
+ }
+ const std::string shard = argv[2];
+ const std::vector<int> layers = nb_ints(argv[3]);
+ std::vector<int> Ts = { 1, 2, 3, 4, 5, 6 }, shares = { 0, 50 };
+ int plans = 200, trials = 7, mode = -1, variant = -1;
+ std::vector<int> vs; // gu-proto: the nibble gate/up kernels compared with the production one (-1), gate/up only
+ for (int i = 4; i + 1 < argc; i += 2) {
+ const std::string a = argv[i];
+ if (a == "--t") Ts = nb_ints(argv[i + 1]);
+ else if (a == "--share") shares = nb_ints(argv[i + 1]);
+ else if (a == "--plans") plans = std::atoi(argv[i + 1]);
+ else if (a == "--trials") trials = std::atoi(argv[i + 1]);
+ else if (a == "--mode") mode = std::atoi(argv[i + 1]);
+ else if (a == "--variant") variant = std::atoi(argv[i + 1]);
+ else if (a == "--vs") vs = nb_ints(argv[i + 1]);
+ else {
+ std::fprintf(stderr, "unknown option %s\n", a.c_str());
+ return 2;
+ }
+ }
+ std::unique_ptr<strata::GgufModel> model;
+ try {
+ model = std::make_unique<strata::GgufModel>(strata::GgufModel::open(shard));
+ } catch (const std::exception& ex) {
+ std::fprintf(stderr, "native_expert_bench: %s\n", ex.what());
+ return 2;
+ }
+ cudaDeviceProp prop{};
+ NB_CK(cudaGetDeviceProperties(&prop, 0));
+ char bus[64] = {};
+ NB_CK(cudaDeviceGetPCIBusId(bus, sizeof bus, 0));
+ std::printf("device 0: %s, bus %s, cc %d.%d; layout %s, 4-bit kernel %s\n", prop.name, bus, prop.major, prop.minor,
+ mode < 0 ? "the card's default" : ("mode " + std::to_string(mode)).c_str(),
+ variant < 0 ? "per format" : variant == 1 ? "P" : "S");
+ K::native_expert_nib_set_variant(variant);
+ const int64_t H = nib::kNEmbd, FF = nib::kNFf;
+ constexpr int kTokMax = 8;
+ int failures = 0;
+ cudaStream_t s;
+ NB_CK(cudaStreamCreate(&s));
+ cudaEvent_t e0, e1;
+ NB_CK(cudaEventCreate(&e0));
+ NB_CK(cudaEventCreate(&e1));
+ // the activations: kTokMax tokens of normal floats, quantized as the verify window quantizes them
+ std::mt19937 rng(20261019);
+ std::normal_distribution<float> nd(0.f, 1.f);
+ std::vector<float> x((size_t) kTokMax * H);
+ for (auto& v : x) v = nd(rng);
+ void *dx = nullptr, *dxq = nullptr, *dscr = nullptr;
+ float* dout = nullptr;
+ const int64_t cap = (int64_t) kTokMax * 10;
+ NB_CK(cudaMalloc(&dx, x.size() * 4));
+ NB_CK(cudaMalloc(&dxq, (size_t) kTokMax * H / 32 * 36));
+ NB_CK(cudaMalloc(&dscr, K::native_expert_scratch_bytes(cap, FF)));
+ NB_CK(cudaMalloc((void**) &dout, (size_t) cap * H * 4));
+ NB_CK(cudaMemcpy(dx, x.data(), x.size() * 4, cudaMemcpyHostToDevice));
+ K::quantize_q8_1_rows((const float*) dx, kTokMax, H, dxq, s);
+ for (int l : layers) {
+ const char* roles[3] = { "gate", "up", "down" };
+ const strata::TensorInfo* t[3] = {};
+ size_t sh[3] = {};
+ for (int r = 0; r < 3; ++r) t[r] = model->find("blk." + std::to_string(l) + ".ffn_" + roles[r] + "_exps.weight", &sh[r]);
+ if (!t[0] || !t[1] || !t[2]) {
+ std::printf("layer %d: no expert tensors\n", l);
+ ++failures;
+ continue;
+ }
+ cpu::NativeFmt f;
+ std::string err;
+ if (!cpu::native_fmt((int) t[0]->type, (int) t[2]->type, H, FF, f, err)) {
+ std::printf("layer %d: %s\n", l, err.c_str());
+ ++failures;
+ continue;
+ }
+ nib::Config c;
+ c.on = true;
+ const nib::LayerVram V = nib::plan_layer(l, f, c);
+ if (!V.on) {
+ std::printf("layer %d: %s is not converted\n", l, ggml_type_name((ggml_type) f.gu_type));
+ continue;
+ }
+ // the 512 blobs (P) and their 4-bit slots (N, the engine's transcoder)
+ std::vector<uint8_t> host((size_t) 512 * f.bytes);
+ const size_t dn_role = f.bytes - f.down_off;
+ for (int e = 0; e < 512; ++e) {
+ uint8_t* b = host.data() + (size_t) e * f.bytes;
+ std::memcpy(b, model->shard(sh[0]).tensor_data(*t[0]) + (size_t) e * f.up_off, f.up_off);
+ std::memcpy(b + f.up_off, model->shard(sh[1]).tensor_data(*t[1]) + (size_t) e * f.up_off, f.up_off);
+ std::memcpy(b + f.down_off, model->shard(sh[2]).tensor_data(*t[2]) + (size_t) e * dn_role, dn_role);
+ }
+ uint8_t *raw = nullptr, *slots = nullptr;
+ unsigned long long *dp = nullptr, *dbad = nullptr;
+ NB_CK(cudaMalloc((void**) &raw, host.size()));
+ NB_CK(cudaMalloc((void**) &slots, (size_t) 512 * V.slot_bytes));
+ NB_CK(cudaMalloc((void**) &dp, 512 * 8));
+ NB_CK(cudaMalloc((void**) &dbad, 8));
+ NB_CK(cudaMemcpy(raw, host.data(), host.size(), cudaMemcpyHostToDevice));
+ std::vector<unsigned long long> hp(512);
+ for (int e = 0; e < 512; ++e) hp[e] = (unsigned long long) (slots + (size_t) e * V.slot_bytes);
+ NB_CK(cudaMemcpy(dp, hp.data(), 512 * 8, cudaMemcpyHostToDevice));
+ NB_CK(cudaMemset(dbad, 0, 8));
+ for (int b0 = 0; b0 < 512; b0 += 64)
+ if (nib::transcode(V, raw + (size_t) b0 * V.raw_bytes, 64, dp + b0, dbad, s) != 0) return 2;
+ for (int e = 0; e < 512; ++e)
+ NB_CK(cudaMemcpyAsync(slots + (size_t) e * V.slot_bytes + nib::kDownOff, raw + (size_t) e * V.raw_bytes + V.raw_down_off,
+ V.down_bytes, cudaMemcpyDeviceToDevice, s));
+ unsigned long long bad = 0;
+ NB_CK(cudaMemcpyAsync(&bad, dbad, 8, cudaMemcpyDeviceToHost, s));
+ NB_CK(cudaStreamSynchronize(s));
+ if (bad) {
+ std::printf("layer %d: %llu items do not round-trip\n", l, bad);
+ ++failures;
+ continue;
+ }
+ const auto Lr = K::native_expert_layout(f.gu_type, f.d_type, H, FF);
+ const auto Ln = K::native_expert_layout_nib(f.gu_type, f.d_type, H, FF);
+ std::printf("layer %2d %-7s/%-6s: 512 experts, blob %zu B, 4-bit slot %llu B\n", l, nib::fmt_name(f.gu_type),
+ ggml_type_name((ggml_type) f.d_type), f.bytes, (unsigned long long) V.slot_bytes);
+ // one set of plans: every plan's out rows compared first, then P and N timed interleaved, phase 0 (the whole
+ // step) and 1 (gate/up + SwiGLU/q8_1), us per launch (median of the trials)
+ auto bench = [&](const std::vector<NbPlan>& ps, int tt, double us[2][2]) -> long long {
+ std::vector<NbDev> dr(ps.size()), dn(ps.size());
+ for (size_t i = 0; i < ps.size(); ++i) {
+ dr[i] = nb_upload(ps[i], raw, f.bytes);
+ dn[i] = nb_upload(ps[i], slots, V.slot_bytes);
+ }
+ const int64_t cp = std::max<int64_t>((int64_t) tt * 10, 1);
+ auto launch = [&](int arm, size_t i) {
+ const NbDev& d = arm ? dn[i] : dr[i];
+ K::native_expert_grouped(arm ? Ln : Lr, d.ptr, d.start, d.ng, d.dst, d.tok, std::max<int64_t>(cp, ps[i].ng),
+ std::max<int64_t>(cp, ps[i].ne), dxq, dscr, dout, s, 0);
+ };
+ long long diff = 0;
+ std::vector<uint8_t> a((size_t) cap * H * 4), b(a.size());
+ K::native_expert_set_mode(mode, 0);
+ for (size_t i = 0; i < ps.size(); ++i) {
+ const size_t n = (size_t) ps[i].ne * H * 4;
+ NB_CK(cudaMemsetAsync(dout, 0xFF, a.size(), s));
+ launch(0, i);
+ NB_CK(cudaMemcpyAsync(a.data(), dout, n, cudaMemcpyDeviceToHost, s));
+ NB_CK(cudaMemsetAsync(dout, 0x00, a.size(), s));
+ launch(1, i);
+ NB_CK(cudaMemcpyAsync(b.data(), dout, n, cudaMemcpyDeviceToHost, s));
+ NB_CK(cudaStreamSynchronize(s));
+ diff += std::memcmp(a.data(), b.data(), n) != 0;
+ }
+ for (int ph = 0; ph < 2; ++ph) {
+ K::native_expert_set_mode(mode, ph);
+ std::vector<float> tm[2];
+ for (int arm = 0; arm < 2; ++arm)
+ for (size_t i = 0; i < std::min<size_t>(ps.size(), 20); ++i) launch(arm, i);
+ for (int tr = 0; tr < trials; ++tr)
+ for (int arm = 0; arm < 2; ++arm) {
+ NB_CK(cudaEventRecord(e0, s));
+ for (size_t i = 0; i < ps.size(); ++i) launch(arm, i);
+ NB_CK(cudaEventRecord(e1, s));
+ NB_CK(cudaEventSynchronize(e1));
+ float ms = 0;
+ NB_CK(cudaEventElapsedTime(&ms, e0, e1));
+ tm[arm].push_back(1e3f * ms / (float) ps.size());
+ }
+ for (int arm = 0; arm < 2; ++arm) {
+ std::sort(tm[arm].begin(), tm[arm].end());
+ us[ph][arm] = tm[arm][tm[arm].size() / 2];
+ }
+ }
+ K::native_expert_set_mode(mode, 0);
+ for (size_t i = 0; i < ps.size(); ++i) {
+ nb_free(dr[i]);
+ nb_free(dn[i]);
+ }
+ return diff;
+ };
+ if (!vs.empty()) { // gu-proto: nibble production (-1) against each id in --vs, gate/up only (phase 3)
+ std::vector<int> arms = { -1 };
+ arms.insert(arms.end(), vs.begin(), vs.end());
+ for (int v : vs) {
+ int regs = 0, loc = 0, occ = 0;
+ if (K::native_expert_nib_variant_attr(v, f.gu_type, &regs, &loc, &occ))
+ std::printf(" variant %d: %d registers, %d B local, %d blocks/SM\n", v, regs, loc, occ);
+ }
+ const size_t fsz = (size_t) cap * FF * 4, fa = (fsz + 255) & ~(size_t) 255;
+ for (int tt : Ts) {
+ if (tt < 1 || tt > kTokMax) continue;
+ for (int share : shares) {
+ std::vector<NbPlan> ps;
+ double G = 0;
+ for (int i = 0; i < plans; ++i) {
+ ps.push_back(nb_route(rng, tt, share));
+ G += ps.back().ng;
+ }
+ std::vector<NbDev> dn(ps.size());
+ for (size_t i = 0; i < ps.size(); ++i) dn[i] = nb_upload(ps[i], slots, V.slot_bytes);
+ const int64_t cp = std::max<int64_t>((int64_t) tt * 10, 1);
+ auto launch = [&](int v, size_t i) {
+ K::native_expert_nib_set_variant(v);
+ K::native_expert_grouped(Ln, dn[i].ptr, dn[i].start, dn[i].ng, dn[i].dst, dn[i].tok,
+ std::max<int64_t>(cp, ps[i].ng), std::max<int64_t>(cp, ps[i].ne), dxq, dscr,
+ dout, s, 0);
+ };
+ K::native_expert_set_mode(mode, 3);
+ // every plan: each arm's gate and up rows (the plan's entries) byte for byte the production arm's
+ std::vector<long long> drows(arms.size(), 0);
+ std::vector<uint8_t> ref(2 * fsz), got(2 * fsz);
+ for (size_t i = 0; i < ps.size(); ++i) {
+ const size_t n = (size_t) ps[i].ne * FF * 4;
+ // native_expert_grouped's up rows start after cap_entries gate rows (256-aligned)
+ const size_t fi = (size_t) std::max<int64_t>(cp, ps[i].ne) * FF * 4, fai = (fi + 255) & ~(size_t) 255;
+ for (size_t a = 0; a < arms.size(); ++a) {
+ NB_CK(cudaMemsetAsync(dscr, a == 0 ? 0xFF : 0x00, 2 * fa, s));
+ launch(arms[a], i);
+ std::vector<uint8_t>& o = a == 0 ? ref : got;
+ NB_CK(cudaMemcpyAsync(o.data(), dscr, n, cudaMemcpyDeviceToHost, s));
+ NB_CK(cudaMemcpyAsync(o.data() + fsz, (const uint8_t*) dscr + fai, n, cudaMemcpyDeviceToHost, s));
+ NB_CK(cudaStreamSynchronize(s));
+ if (a == 0) continue;
+ for (size_t r = 0; r < (size_t) ps[i].ne; ++r) {
+ drows[a] += std::memcmp(ref.data() + r * FF * 4, got.data() + r * FF * 4, FF * 4) != 0;
+ drows[a] += std::memcmp(ref.data() + fsz + r * FF * 4, got.data() + fsz + r * FF * 4, FF * 4) != 0;
+ }
+ }
+ }
+ long long rows = 0;
+ for (const NbPlan& p : ps) rows += 2LL * p.ne;
+ std::vector<std::vector<float>> tm(arms.size());
+ for (size_t a = 0; a < arms.size(); ++a)
+ for (size_t i = 0; i < std::min<size_t>(ps.size(), 20); ++i) launch(arms[a], i);
+ for (int tr = 0; tr < trials; ++tr)
+ for (size_t a = 0; a < arms.size(); ++a) {
+ NB_CK(cudaEventRecord(e0, s));
+ for (size_t i = 0; i < ps.size(); ++i) launch(arms[a], i);
+ NB_CK(cudaEventRecord(e1, s));
+ NB_CK(cudaEventSynchronize(e1));
+ float ms = 0;
+ NB_CK(cudaEventElapsedTime(&ms, e0, e1));
+ tm[a].push_back(1e3f * ms / (float) ps.size());
+ }
+ K::native_expert_set_mode(mode, 0);
+ K::native_expert_nib_set_variant(variant);
+ std::printf(" T=%d share %2d%% (G %.2f, %lld gate/up rows a plan set), gate/up us per launch (median, min..max of %d):\n",
+ tt, share, G / plans, rows, trials);
+ double base = 0;
+ for (size_t a = 0; a < arms.size(); ++a) {
+ std::sort(tm[a].begin(), tm[a].end());
+ const double med = tm[a][tm[a].size() / 2];
+ if (a == 0) base = med;
+ std::printf(" %4d: %8.2f (%7.2f..%7.2f) x%.3f %s\n", arms[a], med, tm[a].front(), tm[a].back(),
+ base / med, a == 0 ? "reference" : drows[a] ? ("rows differ: " + std::to_string(drows[a])).c_str() : "bitwise equal");
+ }
+ for (size_t i = 0; i < ps.size(); ++i) nb_free(dn[i]);
+ }
+ }
+ cudaFree(raw);
+ cudaFree(slots);
+ cudaFree(dp);
+ cudaFree(dbad);
+ continue;
+ }
+ for (int tt : Ts) {
+ if (tt < 1 || tt > kTokMax) continue;
+ for (int share : shares) {
+ std::vector<NbPlan> ps;
+ double G = 0;
+ for (int i = 0; i < plans; ++i) {
+ ps.push_back(nb_route(rng, tt, share));
+ G += ps.back().ng;
+ }
+ double us[2][2];
+ const long long diff = bench(ps, tt, us);
+ std::printf(" T=%d share %2d%% (G %.2f): step P %7.2f N %7.2f us x%.3f | gate/up+swiglu P %7.2f N %7.2f us x%.3f | %s\n",
+ tt, share, G / plans, us[0][0], us[0][1], us[0][0] / us[0][1], us[1][0], us[1][1],
+ us[1][0] / us[1][1], diff ? "DIFFERS" : "bitwise equal");
+ if (diff) {
+ std::printf(" %lld of %d plans differ\n", diff, plans);
+ ++failures;
+ }
+ }
+ }
+ cudaFree(raw);
+ cudaFree(slots);
+ cudaFree(dp);
+ cudaFree(dbad);
+ }
+ K::native_expert_set_mode(-1, 0);
+ K::native_expert_nib_set_variant(-1);
+ std::printf("native_expert_bench --nibble: %d failing\n", failures);
+ return failures ? 1 : 0;
+}
+#else
+int nibble_main(int, char**) {
+ std::fprintf(stderr, "native_expert_bench --nibble: no 4-bit kernels in this build (STRATA_EXPERIMENTAL_SM60, CUDA)\n");
+ return 2;
+}
+#endif
diff --git a/src/prefill/prefill.cpp b/src/prefill/prefill.cpp
index 8a2eb6a30..2e628967a 100644
--- a/src/prefill/prefill.cpp
+++ b/src/prefill/prefill.cpp
@@ -47,6 +47,7 @@
#include "strata/prefill/moe_fused_iq.hpp"
#include "strata/prefill/moe_mmq.hpp"
#include "strata/core/peer_experts.hpp"
+#include "strata/core/expert_nibble.hpp"
#include "strata/prefill/kernels.hpp"
#include "strata/prefill/kva.hpp" // STRATA_KVA: the optional approximate-prefill plan (research, off by default)
@@ -2467,6 +2468,31 @@ void Prefill::arm_cpu_share(bool applies, bool by_default) {
bool Prefill::ring_bytes_enabled() { return ring_bytes_on(); }
size_t Prefill::slot_guard_bytes() { return mmq::direct_slot_guard(); }
+bool Prefill::nibble_prompt_check(std::string& why) {
+ const auto& lay = strata::kernels::cpu::expert_layout();
+ if (mmq::direct_mode() != 0) {
+ why = "STRATA_MMQ_DIRECT (the prompt MMQ would read the 4-bit slots in place)";
+ return false;
+ }
+ for (int64_t l = 0; lay.native && l < (int64_t) lay.fmt.size(); ++l) {
+ if (!core::nibble::layer(l).on) continue;
+ const auto& f = lay.fmt[(size_t) l];
+ const char* w = !mmq_plan().any || (size_t) l >= mmq_plan().layer.size() || !mmq_plan().layer[(size_t) l]
+ ? "no prompt MMQ for it (the FP16 path dequantizes GGUF blocks)"
+ : mmq_plan().wmma_only[(size_t) l] ? "the WMMA tiles read GGUF blocks"
+ : fused::native_supported(f.gu_type, f.d_type) ? "the fused prompt kernels read GGUF blocks"
+ : mmq::matrix_bytes(f.gu_type, 1280, N) != (size_t) core::nibble::kRows * f.gu_row ||
+ mmq::matrix_bytes(f.d_type, N, 640) != f.bytes - f.down_off
+ ? "MMQ's matrices are not the blob's [gate | up] and down"
+ : nullptr;
+ if (w != nullptr) {
+ why = "layer " + std::to_string(l) + ": " + w;
+ return false;
+ }
+ }
+ return true;
+}
+
namespace {
const core::WeightRef* need(const core::LayerView& v, const char* suffix, std::string& err) {
@@ -3881,6 +3907,12 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
const bool had_gu = lay.native && lay.fmt[(size_t) l].had_gu;
const bool had_d = lay.native && lay.fmt[(size_t) l].had_d;
const bool use_mmq = mmq_plan().any && mmq_plan().layer[(size_t) l];
+ // STRATA_EXPERT_NIBBLE=1: a layer whose gate/up experts are 4-bit codes in their slots is gathered
+ // for MMQ as the GGUF blocks rebuilt from them (core::nibble::gather_gguf); every other expert path
+ // reads GGUF blocks in the slot, so it stops there instead (never a wrong answer)
+#if STRATA_EXPERT_NIBBLE_BUILT // every guard of this switch is compiled out of the other builds
+ const core::nibble::LayerVram& nib_l = core::nibble::layer(l);
+#endif
const int had_block = lay.native ? lay.fmt[(size_t) l].had_block : 0;
const float* had_sx = had_gu ? m.had_sx[(size_t) l] : nullptr; // resolved in init
const float* had_sh = had_d ? m.had_sh[(size_t) l] : nullptr;
@@ -3918,6 +3950,9 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
bool cpu_arm = true; // auto: this layer may share (false: the arm without)
int cpu_set = -1; // auto: the wall events' set, -1 when not measured
bool cpu_cold = false; // auto: one-time costs in the window (no reading)
+#if STRATA_EXPERT_NIBBLE_BUILT
+ if (fused_l && nib_l.on) core::nibble::fatal(l, "the prompt path's fused experts", "they read GGUF blocks");
+#endif
if (fused_l) {
if (static bool said = false; !said) {
said = true;
@@ -4823,6 +4858,13 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
cudaStreamWaitEvent(m.cs, m.copied[gg_slots[gg_nslots - 1]], 0);
pt.mark(kPfDequant, cs);
}
+#if STRATA_EXPERT_NIBBLE_BUILT
+ if (nib_l.on) { // the GGUF blocks rebuilt from the 4-bit slots: the bytes the copy gives
+ if (const int rc = core::nibble::gather_gguf(nib_l, gg.blob, gg.first, gg.n, m.grp_gu, mmq_gub,
+ m.grp_d, mmq_db, zero, m.cs); rc != 0)
+ core::nibble::fatal(l, "the prompt path's group gather", cudaGetErrorString((cudaError_t) rc));
+ } else
+#endif
// NVFP4: each expert's scales go next to its group slot in the same launch
if (!mmq::gather_native_group(gg, f.up_off, mmq_gub / 2, f.down_off, mmq_db, m.grp_gu, mmq_gub,
m.grp_d, mmq_db, m.cs, f.tail_off,
@@ -4848,6 +4890,17 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
// a ring slot or the peer's), which STRATA_MMQ_DIRECT reads in place
auto compute = [&](size_t j, const uint8_t* blob_dev, int slot, int32_t cslot) -> bool {
const int32_t e = order[j];
+#if STRATA_EXPERT_NIBBLE_BUILT
+ if (nib_l.on && (!use_mmq || direct || wmma_l || cslot < 0 || !lay.native ||
+ mmq_gub != (size_t) core::nibble::kRows * nib_l.raw_gu_row ||
+ mmq_db != nib_l.down_bytes))
+ core::nibble::fatal(l, "the prompt path",
+ !use_mmq ? "no MMQ for the layer (the FP16 path dequantizes GGUF blocks)"
+ : direct ? "STRATA_MMQ_DIRECT reads the slots in place"
+ : wmma_l ? "the WMMA tiles read GGUF blocks"
+ : cslot < 0 ? "an expert not in this GPU's slots (streamed or the peer's)"
+ : "the MMQ group buffers are not the blob's [gate | up] and down");
+#endif
pt.mark(kPfDequant, cs);
if (use_mmq) {
// gather the expert into its group slot (GGUF blocks, unchanged or converted)
@@ -4894,6 +4947,16 @@ bool Prefill::run_impl(const int64_t* tokens, int64_t n, int64_t pos0, std::stri
gg = mmq::GatherGroup{};
} else if (lay.native) {
const auto& f = lay.fmt[(size_t) l];
+#if STRATA_EXPERT_NIBBLE_BUILT
+ if (nib_l.on) { // as the group gather (flush), one expert at its group position q
+ const uint8_t* one[mmq::kGatherGroupMax] = {};
+ one[q] = blob_dev;
+ if (const int rc = core::nibble::gather_gguf(nib_l, one, (int) q, (int) q + 1, m.grp_gu,
+ mmq_gub, m.grp_d, mmq_db,
+ last_of_group ? MMQ_TAIL : 0, m.cs); rc != 0)
+ core::nibble::fatal(l, "the prompt path's gather", cudaGetErrorString((cudaError_t) rc));
+ } else
+#endif
// NVFP4: this expert's scales go next to its group slot in the same launch
mmq::gather_native(blob_dev, blob_dev + f.up_off, mmq_gub / 2, blob_dev + f.down_off,
mmq_db, m.grp_gu + q * mmq_gub, m.grp_d + q * mmq_db, m.cs,
diff --git a/src/program/generate.cpp b/src/program/generate.cpp
index 75d404ac9..d0de8560c 100644
--- a/src/program/generate.cpp
+++ b/src/program/generate.cpp
@@ -25,6 +25,7 @@
#include "strata/core/device.hpp"
#include "strata/core/remote_expert_opt.hpp"
#include "strata/core/expert_cache.hpp"
+#include "strata/core/expert_nibble.hpp"
#include "strata/core/conversation_snapshot.hpp"
#include "strata/core/conversation_file.hpp"
#include "strata/core/session_save_reclaim.hpp"
@@ -156,6 +157,7 @@ namespace strata::metal_engine { void set_model_shard(const char* shard1); } /
#include <vector>
namespace {
+namespace nib = strata::core::nibble;
// Windows' WDDM driver model: native Windows, or WSL2 (its GPU goes through /dev/dxg to the Windows driver). There,
// pinning a large arena into two CUDA contexts leaves WDDM refusing every later allocation (the 5080 + 3090 rig);
// a Linux driver has no such limit (#253: the 8 GiB cap cost a 4090 + 3060 split 3x of its prompt speed).
@@ -388,6 +390,7 @@ bool resident_stage_swaps(strata::core::FileExpertSource& src, const std::vector
if (slot < 0) continue;
const SwapHome home = locate(s.layer);
const strata::core::OnDevice on(home.dev);
+ nib::guard_raw(s.layer, "the adaptive tier's copy back (resident_stage_swaps)");
if (const cudaError_t e = cudaMemcpyAsync(src.exchange_buffer(q, s.layer), home.cache->device_slot(slot),
(size_t) strata::kernels::cpu::expert_layout().blob_bytes(s.layer), cudaMemcpyDeviceToHost,
home.stream); e != cudaSuccess) {
@@ -2248,6 +2251,297 @@ struct PcieAuto {
}
};
+// ---- STRATA_EXPERT_NIBBLE=1 (strata/core/expert_nibble.hpp): the gate/up experts of IQ2_S / IQ3_XXS / IQ3_S layers as
+// 4-bit codes in their VRAM slots. Everything below runs only with the switch on.
+
+/// The largest slot an expert cache can hold for this pack: the GGUF `max_blob`, or a converted layer's 4-bit slot when
+/// that is larger (the sizings that count slots of the largest blob would otherwise come out short of a full cache and
+/// leave converted experts out of VRAM, which the switch refuses). `max_blob` exactly with the switch off.
+uint64_t nibble_max_slot_bytes() {
+ const auto& lay = strata::kernels::cpu::expert_layout();
+ uint64_t m = lay.max_blob;
+ if (nib::any_on())
+ for (int64_t l = 0; l < lay.n_layers; ++l) m = std::max<uint64_t>(m, nib::slot_bytes(l));
+ return m;
+}
+
+#if STRATA_EXPERT_NIBBLE_BUILT
+/// One expert cache's converted slots. The startup fill hands a converted layer's pairs to `add` instead of
+/// `fill_slot_blocking`; per layer, `kBatch` GGUF blobs at a time go H2D into one staging buffer (the bytes
+/// fill_slot_blocking would copy, from the source's `blob_stable`, which stays valid) and, per batch, the
+/// transcoder writes the codes, scales and d into the slots and round-trips every item, the GGUF down rows are copied
+/// behind them, the GGUF blocks are rebuilt from every slot and compared with the blob (both count against the layer
+/// in `d_bad`), and each slot's hash is recorded for STRATA_EXPERT_NIBBLE_RECHECK. One stream, so the staging
+/// buffer's reuse is ordered without events. The device arrays live as long as the process (RECHECK reads them);
+/// `finish` frees the staging buffer.
+struct NibbleCache {
+ static constexpr int64_t kBatch = 64;
+ struct Pending { int32_t expert, slot; const uint8_t* blob; };
+ struct Batch { int32_t layer; int64_t at, n; };
+ strata::core::ExpertCache* cache = nullptr;
+ int dev = 0; ///< the cache's device
+ std::vector<std::vector<Pending>> pending; ///< per layer: queued, at most kBatch
+ std::vector<Batch> batches; ///< what was transcoded and hashed, in that order
+ std::vector<std::pair<int32_t, int32_t>> pairs; ///< (layer, expert) of each converted slot, in that order
+ std::vector<int32_t> slots; ///< and its slot
+ unsigned long long* h_ptrs = nullptr; ///< pinned: the slots' device addresses, in that order
+ unsigned long long* d_ptrs = nullptr; ///< the same on the device (the transcoder's and hash's input)
+ unsigned long long* d_ref = nullptr; ///< each slot's hash as loaded
+ unsigned long long* d_now = nullptr; ///< RECHECK's
+ unsigned long long* d_bad = nullptr; ///< per layer: items that did not round-trip, words that differ
+ uint8_t* staging = nullptr;
+ uint64_t raw_max = 0;
+ cudaStream_t stream = nullptr;
+ int64_t cap = 0, n = 0;
+ double seconds = 0;
+
+ bool active() const { return cache != nullptr; }
+
+ static std::string cuda_fail(const char* what, cudaError_t e) { return std::string(what) + ": " + cudaGetErrorString(e); }
+
+ /// For `c` on the current device, `want` converted pairs at most.
+ bool begin(strata::core::ExpertCache& c, int64_t want, std::string& err) {
+ if (cudaGetDevice(&dev) != cudaSuccess) dev = 0;
+ cache = &c;
+ cap = want > 0 ? want : 1;
+ const auto& lay = strata::kernels::cpu::expert_layout();
+ pending.assign((size_t) lay.n_layers, {});
+ for (int64_t l = 0; l < lay.n_layers; ++l)
+ if (nib::layer(l).on) raw_max = std::max<uint64_t>(raw_max, nib::layer(l).raw_bytes);
+ cudaError_t e = cudaStreamCreateWithFlags(&stream, cudaStreamNonBlocking);
+ if (e == cudaSuccess) e = cudaMallocHost((void**) &h_ptrs, (size_t) cap * 8);
+ if (e == cudaSuccess) e = cudaMalloc((void**) &d_ptrs, (size_t) cap * 8);
+ if (e == cudaSuccess) e = cudaMalloc((void**) &d_ref, (size_t) cap * 8);
+ if (e == cudaSuccess) e = cudaMalloc((void**) &d_now, (size_t) cap * 8);
+ if (e == cudaSuccess) e = cudaMalloc((void**) &d_bad, (size_t) lay.n_layers * 8);
+ if (e == cudaSuccess) e = cudaMemsetAsync(d_bad, 0, (size_t) lay.n_layers * 8, stream);
+ if (e == cudaSuccess) e = cudaMalloc((void**) &staging, (size_t) ((uint64_t) kBatch * raw_max));
+ if (e != cudaSuccess) {
+ err = cuda_fail("the 4-bit fill's buffers", e);
+ return false;
+ }
+ if (const int rc = nib::prepare((void*) stream); rc != 0) {
+ err = std::string("the 4-bit fill's GGUF codebooks: ") +
+ (rc == (int) cudaErrorInvalidValue ? "a grid entry is not unique" : cudaGetErrorString((cudaError_t) rc));
+ return false;
+ }
+ return true;
+ }
+
+ bool add(int32_t layer, int32_t expert, int32_t slot, const uint8_t* blob, std::string& err) {
+ auto& q = pending[(size_t) layer];
+ q.push_back({expert, slot, blob});
+ return (int64_t) q.size() < kBatch || flush(layer, err);
+ }
+
+ bool flush(int32_t layer, std::string& err) {
+ auto& q = pending[(size_t) layer];
+ if (q.empty()) return true;
+ const auto t0 = std::chrono::steady_clock::now();
+ const nib::LayerVram& L = nib::layer(layer);
+ const int64_t k = (int64_t) q.size();
+ if (n + k > cap) {
+ err = "the 4-bit fill was given more converted pairs than it counted";
+ return false;
+ }
+ cudaError_t e = cudaSuccess;
+ for (int64_t i = 0; i < k && e == cudaSuccess; ++i) {
+ uint8_t* dst = cache->device_slot(q[(size_t) i].slot);
+ if (dst == nullptr || cache->slot_room(q[(size_t) i].slot) < L.hashed) { // a slot sized for the GGUF blob
+ err = "the 4-bit fill: slot " + std::to_string(q[(size_t) i].slot) +
+ (dst == nullptr ? " is outside the cache" : " is smaller than a 4-bit slot");
+ return false;
+ }
+ h_ptrs[n + i] = (unsigned long long) (uintptr_t) dst;
+ e = cudaMemcpyAsync(staging + (uint64_t) i * L.raw_bytes, q[(size_t) i].blob, (size_t) L.raw_bytes,
+ cudaMemcpyHostToDevice, stream);
+ }
+ if (e == cudaSuccess) e = cudaMemcpyAsync(d_ptrs + n, h_ptrs + n, (size_t) k * 8, cudaMemcpyHostToDevice, stream);
+ if (e != cudaSuccess) {
+ err = cuda_fail("the 4-bit fill's copies in", e);
+ return false;
+ }
+ auto m2 = [&](const char* what, int rc) -> bool { // iq_kernels.cu's: a CUDA error code
+ if (rc == 0) return true;
+ err = std::string("the 4-bit ") + what + ": " + cudaGetErrorString((cudaError_t) rc);
+ return false;
+ };
+ if (!m2("transcoder", nib::transcode(L, staging, k, d_ptrs + n, d_bad + layer, (void*) stream))) return false;
+ for (int64_t i = 0; i < k && e == cudaSuccess; ++i)
+ e = cudaMemcpyAsync((uint8_t*) (uintptr_t) h_ptrs[n + i] + nib::kDownOff,
+ staging + (uint64_t) i * L.raw_bytes + L.raw_down_off, (size_t) L.down_bytes,
+ cudaMemcpyDeviceToDevice, stream);
+ if (e != cudaSuccess) {
+ err = cuda_fail("the 4-bit fill's down rows", e);
+ return false;
+ }
+ if (!m2("GGUF check", nib::check_gguf(L, staging, k, d_ptrs + n, d_bad + layer, (void*) stream))) return false;
+ if (!m2("slot hash", nib::hash_slots(L, d_ptrs + n, k, d_ref + n, (void*) stream))) return false;
+ // the next batch's copies in land in the same staging bytes: they wait for this batch's kernels (one stream)
+ for (const Pending& p : q) {
+ pairs.emplace_back(layer, p.expert);
+ slots.push_back(p.slot);
+ }
+ batches.push_back({layer, n, k});
+ n += k;
+ q.clear();
+ seconds += std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count();
+ return true;
+ }
+
+ /// The rest of the queue, then waits: false (with `err` naming every layer) when an item did not round-trip or a
+ /// rebuilt GGUF block differs from its blob.
+ bool finish(std::string& err) {
+ for (int32_t l = 0; l < (int32_t) pending.size(); ++l)
+ if (!flush(l, err)) return false;
+ const auto t0 = std::chrono::steady_clock::now();
+ cudaError_t e = cudaStreamSynchronize(stream);
+ std::vector<unsigned long long> bad(pending.size(), 0);
+ if (e == cudaSuccess) e = cudaMemcpy(bad.data(), d_bad, bad.size() * 8, cudaMemcpyDeviceToHost);
+ if (staging != nullptr) (void) cudaFree(staging);
+ staging = nullptr;
+ seconds += std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count();
+ if (e != cudaSuccess) {
+ err = cuda_fail("the 4-bit fill", e);
+ return false;
+ }
+ for (size_t l = 0; l < bad.size(); ++l)
+ if (bad[l] != 0) {
+ char b[192];
+ std::snprintf(b, sizeof b, "%slayer %zu (%s): %llu items or words do not round-trip", err.empty() ? "" : "; ",
+ l, nib::fmt_name(nib::layer((int64_t) l).gu_type), bad[l]);
+ err += b;
+ }
+ if (!err.empty()) {
+ err += " - the 4-bit code does not hold these weights (STRATA_EXPERT_NIBBLE_SKIP=<layers> keeps a layer's "
+ "GGUF blocks)";
+ return false;
+ }
+ return true;
+ }
+
+ /// The end-to-end check of the fill as it is wired, on converted pair (layer, expert) with `raw` its GGUF blob: the
+ /// slot read back, its hash against expert_nibble.hpp's host reference of what the device recorded, its down rows
+ /// against the blob's. (Every item and every GGUF block of every slot was checked on the device by `finish`.)
+ bool check(int32_t layer, int32_t expert, const uint8_t* raw, std::string& err) const {
+ int64_t at = -1;
+ for (size_t i = 0; i < pairs.size() && at < 0; ++i)
+ if (pairs[i].first == layer && pairs[i].second == expert) at = (int64_t) i;
+ if (at < 0 || raw == nullptr) {
+ err = "the 4-bit check: no converted slot or no GGUF blob for layer " + std::to_string(layer) + " expert " +
+ std::to_string(expert);
+ return false;
+ }
+ const nib::LayerVram& L = nib::layer(layer);
+ std::vector<uint8_t> got((size_t) L.hashed);
+ unsigned long long h = 0;
+ cudaError_t e = cudaMemcpy(got.data(), cache->device_slot(slots[(size_t) at]), got.size(), cudaMemcpyDeviceToHost);
+ if (e == cudaSuccess) e = cudaMemcpy(&h, d_ref + at, 8, cudaMemcpyDeviceToHost);
+ if (e != cudaSuccess) {
+ err = cuda_fail("the 4-bit check's read back", e);
+ return false;
+ }
+ const char* why = nib::hash_host(got.data(), L.hashed) != h ? "the device slot hash differs from the host reference"
+ : std::memcmp(got.data() + nib::kDownOff, raw + L.raw_down_off, (size_t) L.down_bytes) != 0
+ ? "the down rows differ from the GGUF blob's"
+ : nullptr;
+ if (why != nullptr) {
+ err = "layer " + std::to_string(layer) + " expert " + std::to_string(expert) + " (" + nib::fmt_name(L.gu_type) +
+ "): " + why;
+ return false;
+ }
+ return true;
+ }
+
+ /// STRATA_EXPERT_NIBBLE_RECHECK: every converted slot's hash again, on the device, against the one recorded at load.
+ /// False with the first slot that differs (or a CUDA error) in `err`; `bad_layer` its layer (-1: an error).
+ bool recheck(std::string& err, int64_t& bad_layer) {
+ bad_layer = -1;
+ if (n == 0) return true;
+ const strata::core::OnDevice on(dev);
+ for (const Batch& b : batches) {
+ const int rc = nib::hash_slots(nib::layer(b.layer), d_ptrs + b.at, b.n, d_now + b.at, (void*) stream);
+ if (rc != 0) {
+ err = cuda_fail("the slot hash", (cudaError_t) rc);
+ return false;
+ }
+ }
+ std::vector<unsigned long long> ref((size_t) n), now((size_t) n);
+ cudaError_t e = cudaStreamSynchronize(stream);
+ if (e == cudaSuccess) e = cudaMemcpy(ref.data(), d_ref, (size_t) n * 8, cudaMemcpyDeviceToHost);
+ if (e == cudaSuccess) e = cudaMemcpy(now.data(), d_now, (size_t) n * 8, cudaMemcpyDeviceToHost);
+ if (e != cudaSuccess) {
+ err = cuda_fail("the slot hash read back", e);
+ return false;
+ }
+ for (int64_t i = 0; i < n; ++i)
+ if (ref[(size_t) i] != now[(size_t) i]) {
+ char b[224];
+ std::snprintf(b, sizeof b, "CUDA%d: layer %d expert %d (slot %d) hashes %016llx, %016llx at load", dev,
+ (int) pairs[(size_t) i].first, (int) pairs[(size_t) i].second, (int) slots[(size_t) i],
+ now[(size_t) i], ref[(size_t) i]);
+ err = b;
+ bad_layer = pairs[(size_t) i].first;
+ return false;
+ }
+ return true;
+ }
+};
+#else
+// A build without the 4-bit kernels (STRATA_EXPERT_NIBBLE_BUILT 0: HIP, Metal, a CUDA build without
+// STRATA_EXPERIMENTAL_SM60): nibble_refusal refuses the switch, so no cache is ever converted and none of this is entered.
+struct NibbleCache {
+ int64_t n = 0;
+ double seconds = 0;
+ std::vector<std::pair<int32_t, int32_t>> pairs;
+ bool active() const { return false; }
+ bool begin(strata::core::ExpertCache&, int64_t, std::string& err) { err = "the 4-bit slots are not in this build"; return false; }
+ bool add(int32_t, int32_t, int32_t, const uint8_t*, std::string& err) { err = "the 4-bit slots are not in this build"; return false; }
+ bool finish(std::string&) { return true; }
+ bool check(int32_t, int32_t, const uint8_t*, std::string& err) const { err = "the 4-bit slots are not in this build"; return false; }
+ bool recheck(std::string&, int64_t& bad_layer) { bad_layer = -1; return true; }
+};
+#endif
+
+/// With STRATA_EXPERT_NIBBLE=1, why this start cannot have 4-bit slots (empty: nothing here stops it): what the build,
+/// the GPUs, the environment and the options decide, before anything is allocated. The full residency, the loans and
+/// the prompt chunk are checked where they are known.
+std::string nibble_refusal(const Options& o) {
+ const nib::Config& c = nib::config();
+ if (!c.error.empty()) return c.error;
+#if !STRATA_EXPERT_NIBBLE_BUILT || !defined(STRATA_NATIVE_EXPERTS)
+ (void) o;
+ return "the 4-bit slots and their kernels are in the CUDA build with STRATA_EXPERIMENTAL_SM60 and native experts only";
+#else
+ int count = 0;
+ if (cudaGetDeviceCount(&count) != cudaSuccess || count <= 0) {
+ (void) cudaGetLastError();
+ return "no CUDA device";
+ }
+ for (int d = 0; d < count; ++d) {
+ cudaDeviceProp p{};
+ if (cudaGetDeviceProperties(&p, d) != cudaSuccess) {
+ (void) cudaGetLastError();
+ return "cannot read CUDA" + std::to_string(d) + "'s properties";
+ }
+ if (p.major != 6 || p.minor != 0)
+ return "CUDA" + std::to_string(d) + " (" + p.name + ") is compute capability " + std::to_string(p.major) + "." +
+ std::to_string(p.minor) + ": the 4-bit slots are for compute capability 6.0 (GP100) only";
+ }
+ if (std::string e = nib::env_conflicts(); !e.empty()) return e;
+ if (o.peer_device >= 1) return "--peer-device (its cache holds GGUF blobs)";
+ for (size_t r = 0; r < o.expert_cache_remote.size(); ++r)
+ if (o.expert_cache_remote[r] > 0 || o.expert_cache_remote_auto[r])
+ return "remote expert caches (--expert-cache-device1..3 hold GGUF blobs)";
+ if (o.kv_grow) return "--kv-grow (the elastic K/V moves slots at their GGUF size)";
+ if (o.vram_elastic) return "--vram-elastic (it refills slots with GGUF blobs)";
+ if (o.expert_profile.empty())
+ return "no --expert-profile (every expert must be VRAM-resident from the profile fill)";
+ if (o.expert_cache == 0) return "--expert-cache 0 (no VRAM slots to convert)";
+ return {};
+#endif
+}
+
} // namespace
// #1425 #1549 #1629: --ple-io direct (the default) does unbuffered random reads of the n-gram table; on some drives
@@ -3564,6 +3858,49 @@ int main(int argc, char** argv) {
}
std::fprintf(stderr, "strata generate: primary device from STRATA_PRIMARY_DEVICE: CUDA%d\n", primary_device);
}
+ // ---- STRATA_EXPERT_NIBBLE=1 keeps the gate/up experts of IQ2_S / IQ3_XXS / IQ3_S layers as 4-bit codes in their VRAM
+ // slots (strata/core/expert_nibble.hpp). Unset or 0: nothing here runs, no table is set, and every slot size below is
+ // the GGUF blob's, as before. On: if anything would read a converted slot as GGUF blocks or write GGUF bytes into
+ // one, the switch is ignored for this start with a warning (before anything is allocated); else the per-layer table
+ // is set before the first slot estimate and the adaptive tier is off (it swaps GGUF blobs in and out of slots).
+ // The residency, the loans and the prompt chunk are checked where they are known (those refuse the start).
+ const bool nib_asked = nib::config().on || nib::config().error.rfind("STRATA_EXPERT_NIBBLE=", 0) == 0;
+ const int64_t nib_chunk_asked = o.prefill_chunk; // an explicit --prefill N must stay N (the chunking is in the bits)
+ if (nib_asked) {
+ std::string why = nibble_refusal(o);
+ if (why.empty()) {
+ nib::set_layers(nib::plan_layers(strata::kernels::cpu::expert_layout(), why));
+ if (why.empty() && !nib::any_on())
+ why = "no layer of this pack is IQ2_S / IQ3_XXS / IQ3_S at 2560 x 640 among STRATA_EXPERT_NIBBLE_FMTS "
+ "and outside STRATA_EXPERT_NIBBLE_SKIP";
+ if (why.empty()) (void) strata::prefill::Prefill::nibble_prompt_check(why);
+ }
+ if (!why.empty()) {
+ // a warning, not a refusal: this start goes on as it would without the switch (the usual GGUF slots)
+ std::fprintf(stderr, "strata generate: WARNING: STRATA_EXPERT_NIBBLE is off for this start: %s; the expert slots "
+ "stay GGUF blocks\n", why.c_str());
+ nib::set_layers({});
+ } else {
+ o.adapt_swaps = 0;
+ o.peer_adapt_swaps = 0;
+ const auto& lay = strata::kernels::cpu::expert_layout();
+ int64_t n_on = 0, by_fmt[3] = {0, 0, 0};
+ uint64_t more = 0;
+ for (int64_t l = 0; l < lay.n_layers; ++l) {
+ const nib::LayerVram& v = nib::layer(l);
+ if (!v.on) continue;
+ ++n_on;
+ ++by_fmt[v.gu_type == nib::kIQ2_S ? 0 : v.gu_type == nib::kIQ3_XXS ? 1 : 2];
+ more += (v.slot_bytes - (v.raw_bytes + 255) / 256 * 256) * (uint64_t) lay.n_expert;
+ }
+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: %lld of %lld layers keep their gate/up experts as 4-bit "
+ "codes in VRAM (IQ2_S %lld, IQ3_XXS %lld, IQ3_S %lld), %.2f GiB more VRAM than their GGUF "
+ "blobs for all %lld experts of each; the adaptive tier is off (--adapt-swaps 0)%s\n",
+ (long long) n_on, (long long) lay.n_layers, (long long) by_fmt[0], (long long) by_fmt[1],
+ (long long) by_fmt[2], (double) more / 1073741824.0, (long long) lay.n_expert,
+ nib::config().recheck ? "; STRATA_EXPERT_NIBBLE_RECHECK: every slot's hash after each request" : "");
+ }
+ }
// STRATA_EARLY_REMOTE_CONTEXTS=1: create EVERY secondary context here, like CUDA1's. Under WSL2 the driver's
// pinned/mapped host budget (dxg gpadl, ~1 GiB) is spent by CUDA0's weights and MTP before the later loop runs,
// and a new context then fails with cudaErrorMemoryAllocation (CUDA2: "cudaSetDevice(2) failed: out of memory").
@@ -4291,7 +4628,7 @@ int main(int argc, char** argv) {
const auto& lay = strata::kernels::cpu::expert_layout();
int64_t pairs_bytes = 0;
for (const auto& pr : prof)
- pairs_bytes += native_pack ? strata::core::ExpertCache::slot_stride((int64_t) lay.blob_bytes(pr.first), slot_guard)
+ pairs_bytes += native_pack ? strata::core::ExpertCache::slot_stride((int64_t) nib::slot_bytes(pr.first), slot_guard)
: (int64_t) lay.max_blob;
size_t fb = 0, tb = 0;
cudaMemGetInfo(&fb, &tb);
@@ -4754,7 +5091,7 @@ int main(int argc, char** argv) {
total_mass += (mass[r] = hi - lo);
}
auto cost = [&](int64_t l) -> int64_t {
- return native_pack ? strata::core::ExpertCache::slot_stride((int64_t) lay.blob_bytes(l), slot_guard)
+ return native_pack ? strata::core::ExpertCache::slot_stride((int64_t) nib::slot_bytes(l), slot_guard)
: (int64_t) lay.max_blob;
};
// the predicted window time (ms) of a placement, and the routed mass its caches hold
@@ -4870,7 +5207,7 @@ int main(int argc, char** argv) {
};
struct PartStat { int64_t lb, le; int64_t used; int64_t cnt; int64_t cap; };
auto can_start = [&](const std::vector<PartStat>& parts) -> bool {
- const int64_t kMaxBlob = (int64_t) strata::kernels::cpu::expert_layout().max_blob;
+ const int64_t kMaxBlob = (int64_t) nibble_max_slot_bytes(); // max_blob (or a larger 4-bit slot)
auto slots_est_of = [&](int i, uint64_t cn) -> int64_t {
if (parts[i].cnt <= 0) return -1;
const uint64_t avg = (parts[i].used > 0)
@@ -5966,7 +6303,7 @@ int main(int argc, char** argv) {
// which only ever grants MORE slots)
uint64_t cap_bytes;
if (o.expert_cache > 0) {
- cap_bytes = (uint64_t) o.expert_cache * pl.max_blob;
+ cap_bytes = (uint64_t) o.expert_cache * nibble_max_slot_bytes();
} else {
const int64_t pf_mib = (o.prefill_chunk > 0 && !pf_borrow) ? 160 + (o.prefill_chunk * 680) / 1024 : 0;
int64_t mtp_b = (!o.mtp.empty() && native_head.loaded())
@@ -5986,7 +6323,7 @@ int main(int argc, char** argv) {
uint64_t used = 0;
skip = 0;
for (const auto& pr : profile) {
- const uint64_t b = pl.native ? (pl.blob_bytes(pr.first) + 255) / 256 * 256 : (uint64_t) pl.max_blob;
+ const uint64_t b = pl.native ? (nib::slot_bytes(pr.first) + 255) / 256 * 256 : (uint64_t) pl.max_blob;
if (used + b > cap_bytes) break;
used += b;
++skip;
@@ -6222,7 +6559,7 @@ int main(int argc, char** argv) {
if (use_df2 && native_head.loaded()) mtp_bind += (int64_t) mtp.bind_bytes(native_head.row_bytes(), n_vocab) + (48 << 20); // DFlash2's own draft head and logits, bound after this
#endif
const int64_t reserve = (((int64_t) o.vram_reserve_mib + prefill_mib) << 20) + mtp_bind + pipe_first;
- const int64_t blob = (int64_t) strata::kernels::cpu::expert_layout().max_blob;
+ const int64_t blob = (int64_t) nibble_max_slot_bytes(); // max_blob (STRATA_EXPERT_NIBBLE: or a larger 4-bit slot)
int64_t slots = ((int64_t) free_b - reserve) / blob;
if (!profile.empty()) slots = std::min<int64_t>(slots, (int64_t) profile.size());
#if defined(STRATA_GPU_METAL)
@@ -6320,7 +6657,7 @@ int main(int argc, char** argv) {
free_b = strata::core::device_free_bytes(); (void) total_b;
const int64_t prefill_mib = owned_prefill_mib();
const int64_t reserve = (((int64_t) o.vram_reserve_mib + prefill_mib) << 20) + pipe_first;
- const int64_t fit = std::max<int64_t>(((int64_t) free_b - reserve) / (int64_t) strata::kernels::cpu::expert_layout().max_blob, 0);
+ const int64_t fit = std::max<int64_t>(((int64_t) free_b - reserve) / (int64_t) nibble_max_slot_bytes(), 0);
if (o.expert_cache > fit) {
// a WARNING that names the knob: the user asked for this size, and gets fewer slots
std::fprintf(stderr, "strata generate: WARNING: layer split: --expert-cache %d leaves no room for the "
@@ -6342,19 +6679,19 @@ int main(int argc, char** argv) {
const size_t free_b = strata::core::device_free_bytes(); // the same reading the shared-cache sizing uses
const auto& lay = strata::kernels::cpu::expert_layout();
const int asked = o.expert_cache;
- const uint64_t budget = (uint64_t) asked * lay.max_blob;
+ const uint64_t budget = (uint64_t) asked * nibble_max_slot_bytes();
size_t free_room = free_b > ((size_t) o.vram_reserve_mib << 20) ? free_b - ((size_t) o.vram_reserve_mib << 20) : 0;
const uint64_t cap = std::min<uint64_t>(budget, (uint64_t) free_room);
uint64_t sum = 0;
for (int64_t l = 0; l < g.n_layers; ++l)
- sum += (uint64_t) strata::core::ExpertCache::slot_stride((int64_t) lay.blob_bytes(l), slot_guard);
+ sum += (uint64_t) strata::core::ExpertCache::slot_stride((int64_t) nib::slot_bytes(l), slot_guard);
int64_t q = sum > 0 ? (int64_t) (cap / sum) : 0;
if (q > g.n_expert) q = g.n_expert;
if (q > 0) {
per_layer_bytes = sum;
sized_slots.reserve((size_t) q * (size_t) g.n_layers);
for (int64_t l = 0; l < g.n_layers; ++l) {
- const int64_t b = (int64_t) lay.blob_bytes(l);
+ const int64_t b = (int64_t) nib::slot_bytes(l);
for (int64_t s = 0; s < q; ++s) sized_slots.push_back(b);
}
o.expert_cache = (int) sized_slots.size();
@@ -6370,16 +6707,16 @@ int main(int argc, char** argv) {
// what the uniform sizing granted, in the 256-byte slots the loop below counts: the blob formats of 1280 / 2560
// rows are whole multiples of 256 already; an NVFP4 blob's 16-byte scale tail is not, and the unrounded budget
// held one slot fewer than asked (--expert-cache 1: none)
- const uint64_t budget = (uint64_t) o.expert_cache * ((lay.max_blob + 255) / 256 * 256);
+ const uint64_t budget = (uint64_t) o.expert_cache * ((nibble_max_slot_bytes() + 255) / 256 * 256);
uint64_t used = 0;
const size_t keep_free = ((size_t) o.vram_reserve_mib << 20) + (size_t) pipe_first;
size_t free_room = free_b > keep_free ? free_b - keep_free : 0;
const uint64_t cap = std::min<uint64_t>(budget, (uint64_t) free_room);
for (const auto& pr : profile) {
- const uint64_t b = (uint64_t) strata::core::ExpertCache::slot_stride((int64_t) lay.blob_bytes(pr.first), slot_guard);
+ const uint64_t b = (uint64_t) strata::core::ExpertCache::slot_stride((int64_t) nib::slot_bytes(pr.first), slot_guard);
if (used + b > cap) break;
used += b;
- sized_slots.push_back((int64_t) lay.blob_bytes(pr.first));
+ sized_slots.push_back((int64_t) nib::slot_bytes(pr.first));
}
// #831: the sized-slots loop walks the profile's pairs, so the profile's length is the ceiling; an explicit
// --expert-cache above it used to be cut with no word. A warning only (what is allocated does not change).
@@ -6429,7 +6766,7 @@ int main(int argc, char** argv) {
sized_slots.clear();
if (q <= 0) { o.expert_cache = 0; return false; }
for (int64_t l = 0; l < g.n_layers; ++l) {
- const int64_t b = (int64_t) lay.blob_bytes(l);
+ const int64_t b = (int64_t) nib::slot_bytes(l);
for (int64_t s = 0; s < q; ++s) sized_slots.push_back(b);
}
o.expert_cache = (int) sized_slots.size();
@@ -6554,6 +6891,10 @@ int main(int argc, char** argv) {
// admission finds no room and every non-profiled expert stays a CPU miss. That is what makes the profile
// the policy rather than a hint.
int64_t prefilled = 0;
+ // STRATA_EXPERT_NIBBLE: each cache's converted slots (RECHECK reads them per request)
+ NibbleCache nib_x;
+ std::vector<NibbleCache> nib_st;
+ nib_st.reserve(stages.size());
if (!profile.empty() && srcp != nullptr) {
// #369 (dag08): per layer, a full layer skips only its own pairs - each layer takes its hottest experts until
// its range is full (one full layer used to end the whole fill, leaving most layers empty)
@@ -6574,6 +6915,14 @@ int main(int argc, char** argv) {
}
const auto fill_t0 = std::chrono::steady_clock::now();
uint64_t fill_bytes = 0;
+ if (nib::any_on()) { // STRATA_EXPERT_NIBBLE: the converted layers' pairs go through the 4-bit fill instead
+ int64_t conv = 0;
+ for (int64_t i = 0; i < want; ++i) conv += nib::layer(profile[(size_t) i].first).on ? 1 : 0;
+ if (conv > 0 && !nib_x.begin(xcache, conv, err)) {
+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: %s\n", err.c_str());
+ return 1;
+ }
+ }
for (int64_t i = 0; i < want; ++i) {
if (!per_layer && srcp == &src && src.unbuffered() && i % 64 == 0) {
if (ahead.valid()) ahead.get();
@@ -6587,8 +6936,10 @@ int main(int argc, char** argv) {
break;
}
const uint8_t* b = srcp->blob_stable(profile[(size_t) i].first, profile[(size_t) i].second);
- if (b == nullptr || !xcache.fill_slot_blocking(slot, b, err,
- (int64_t) strata::kernels::cpu::expert_layout().blob_bytes(profile[(size_t) i].first))) {
+ const bool nib_pair = nib_x.active() && nib::layer(profile[(size_t) i].first).on; // STRATA_EXPERT_NIBBLE
+ if (b == nullptr || !(nib_pair ? nib_x.add(profile[(size_t) i].first, profile[(size_t) i].second, slot, b, err)
+ : xcache.fill_slot_blocking(slot, b, err,
+ (int64_t) strata::kernels::cpu::expert_layout().blob_bytes(profile[(size_t) i].first)))) {
std::fprintf(stderr, "strata generate: the profile fill failed at pair %lld: %s\n",
(long long) i, err.c_str());
return 1;
@@ -6596,16 +6947,39 @@ int main(int argc, char** argv) {
fill_bytes += strata::kernels::cpu::expert_layout().blob_bytes(profile[(size_t) i].first);
++prefilled;
}
+ if (nib_x.active() && !nib_x.finish(err)) {
+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: CUDA0: %s\n", err.c_str());
+ return 1;
+ }
const double fill_s = std::chrono::duration<double>(std::chrono::steady_clock::now() - fill_t0).count();
// **AND ONE SLOT IS READ BACK AND COMPARED.** A residency table that is right about indices and wrong
// about bytes produces a plausible token, which is this project's most expensive failure mode; the
// cache's own `verify_slot` is the check and it costs one 1.38 MB D2H at startup.
- if (prefilled > 0 && !xcache.verify_slot(xcache.slot_of(profile[0].first, profile[0].second),
+ // (STRATA_EXPERT_NIBBLE: a converted slot holds 4-bit codes, not the blob's bytes - the 4-bit check below is its check)
+ if (prefilled > 0 && !nib::layer(profile[0].first).on &&
+ !xcache.verify_slot(xcache.slot_of(profile[0].first, profile[0].second),
srcp->blob_stable(profile[0].first, profile[0].second), err,
(int64_t) strata::kernels::cpu::expert_layout().blob_bytes(profile[0].first))) {
std::fprintf(stderr, "strata generate: %s\n", err.c_str());
return 1;
}
+ if (nib_x.active() && nib_x.n > 0) {
+ // STRATA_EXPERT_NIBBLE: the hottest converted pair read back (its hash, its down rows); every item and every
+ // rebuilt GGUF block of every converted slot was already checked on the device
+ size_t c = 0;
+ while (c < (size_t) want && !nib::layer(profile[c].first).on) ++c;
+ if (!nib_x.check(profile[c].first, profile[c].second, srcp->blob_stable(profile[c].first, profile[c].second),
+ err)) {
+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: CUDA0: %s\n", err.c_str());
+ return 1;
+ }
+ std::set<int32_t> ls;
+ for (const auto& pr : nib_x.pairs) ls.insert(pr.first);
+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: CUDA0: %lld slots of %zu layers transcoded to 4-bit "
+ "codes (%.2f s; every item round-tripped, every slot's GGUF blocks rebuilt and compared, "
+ "slot hashes recorded); layer %d expert %d read back\n", (long long) nib_x.n, ls.size(),
+ nib_x.seconds, (int) profile[c].first, (int) profile[c].second);
+ }
mem_mark("the profile fill");
std::fprintf(stderr, "strata generate: pre-filled %lld of %lld slots from the profile in %.1f s (%.0f MB/s); "
"slot 0 verified\n",
@@ -6662,11 +7036,11 @@ int main(int argc, char** argv) {
std::vector<int64_t> sized;
int64_t used = 0;
for (const auto& pr : st.profile) {
- const int64_t b = native_pack ? strata::core::ExpertCache::slot_stride((int64_t) lay.blob_bytes(pr.first), slot_guard)
+ const int64_t b = native_pack ? strata::core::ExpertCache::slot_stride((int64_t) nib::slot_bytes(pr.first), slot_guard)
: (int64_t) lay.max_blob;
if (used + b > room) break;
used += b;
- sized.push_back((int64_t) lay.blob_bytes(pr.first));
+ sized.push_back((int64_t) nib::slot_bytes(pr.first));
}
// #841: a stage's cache can fail to open with the VRAM free (the card's allocation is also charged to the
// Windows commit, or the free figure ran high): it is tried again with 90% of the slots, up to 3 times, each
@@ -6691,25 +7065,57 @@ int main(int argc, char** argv) {
return 1;
}
int64_t filled = 0;
+ NibbleCache& nib_c = nib_st.emplace_back(); // STRATA_EXPERT_NIBBLE: this stage's converted slots
+ if (nib::any_on()) {
+ int64_t conv = 0;
+ for (const auto& pr : st.profile) conv += nib::layer(pr.first).on ? 1 : 0;
+ if (conv > 0 && !nib_c.begin(st.cache, conv, err)) {
+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: CUDA%d: %s\n", st.dev, err.c_str());
+ return 1;
+ }
+ }
for (const auto& pr : st.profile) {
if (filled >= st.cache.slots()) break;
const int32_t slot = st.cache.admit(pr.first, pr.second);
if (slot == strata::core::kNotResident) break;
const uint8_t* b = srcp->blob_stable(pr.first, pr.second);
- if (b == nullptr || !st.cache.fill_slot_blocking(slot, b, err, (int64_t) lay.blob_bytes(pr.first))) {
+ const bool nib_pair = nib_c.active() && nib::layer(pr.first).on; // STRATA_EXPERT_NIBBLE
+ if (b == nullptr || !(nib_pair ? nib_c.add(pr.first, pr.second, slot, b, err)
+ : st.cache.fill_slot_blocking(slot, b, err, (int64_t) lay.blob_bytes(pr.first)))) {
std::fprintf(stderr, "strata generate: layer split, CUDA%d profile fill failed at pair %lld: %s\n",
st.dev, (long long) filled, err.c_str());
return 1;
}
++filled;
}
- if (filled == 0 || !st.cache.verify_slot(st.cache.slot_of(st.profile[0].first, st.profile[0].second),
- srcp->blob_stable(st.profile[0].first, st.profile[0].second), err,
- (int64_t) lay.blob_bytes(st.profile[0].first))) {
+ if (nib_c.active() && !nib_c.finish(err)) {
+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: CUDA%d: %s\n", st.dev, err.c_str());
+ return 1;
+ }
+ // (STRATA_EXPERT_NIBBLE: a converted slot holds 4-bit codes, not the blob's bytes - the 4-bit check below is its check)
+ if (filled == 0 || (!nib::layer(st.profile[0].first).on &&
+ !st.cache.verify_slot(st.cache.slot_of(st.profile[0].first, st.profile[0].second),
+ srcp->blob_stable(st.profile[0].first, st.profile[0].second), err,
+ (int64_t) lay.blob_bytes(st.profile[0].first)))) {
std::fprintf(stderr, "strata generate: layer split, CUDA%d expert cache: %s\n", st.dev,
filled == 0 ? "nothing filled" : err.c_str());
return 1;
}
+ if (nib_c.active() && nib_c.n > 0) { // STRATA_EXPERT_NIBBLE: as CUDA0's above
+ size_t c = 0;
+ while (c < st.profile.size() && !nib::layer(st.profile[c].first).on) ++c;
+ const auto& pc = st.profile[c];
+ if (!nib_c.check(pc.first, pc.second, srcp->blob_stable(pc.first, pc.second), err)) {
+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: CUDA%d: %s\n", st.dev, err.c_str());
+ return 1;
+ }
+ std::set<int32_t> ls;
+ for (const auto& pr : nib_c.pairs) ls.insert(pr.first);
+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: CUDA%d: %lld slots of %zu layers transcoded to 4-bit "
+ "codes (%.2f s; every item round-tripped, every slot's GGUF blocks rebuilt and compared, "
+ "slot hashes recorded); layer %d expert %d read back\n", st.dev, (long long) nib_c.n,
+ ls.size(), nib_c.seconds, (int) pc.first, (int) pc.second);
+ }
std::fprintf(stderr, "strata generate: layer split: CUDA%d runs layers %lld-%lld, expert cache %lld slots "
"(%.2f GiB), %lld of its %zu profiled pairs; slot 0 verified\n",
st.dev, (long long) st.lb, (long long) (st.le - 1), (long long) st.cache.slots(), st.cache.gib(),
@@ -6717,6 +7123,40 @@ int main(int argc, char** argv) {
}
if (multi_gpu)
std::fprintf(stderr, "strata generate: layer split: CUDA0 runs layers 0-%lld\n", (long long) (split_at[0] - 1));
+ if (nib::any_on()) {
+ // STRATA_EXPERT_NIBBLE: every expert of every layer must be in VRAM, in the cache of the card that runs the layer.
+ // A converted layer's miss would be computed from the GGUF blob by the CPU or over PCIe, and a non-converted
+ // layer's residency would differ from the switch-off run's (its smaller slots leave room for more), so the
+ // replies' bits would too. Then no slot is written again.
+ for (int64_t l = 0; l < g.n_layers; ++l) {
+ const strata::core::ExpertCache* owner = &xcache;
+ int owner_dev = 0;
+ if (multi_gpu && !split_at.empty() && l >= split_at[0]) {
+ owner = nullptr;
+ for (const auto& stp : stages)
+ if (l >= stp->lb && l < stp->le) { owner = &stp->cache; owner_dev = stp->dev; }
+ }
+ int64_t missing = 0, first = -1;
+ for (int64_t e = 0; e < g.n_expert; ++e)
+ if (owner == nullptr || owner->slot_of(l, e) == strata::core::kNotResident) {
+ if (first < 0) first = e;
+ ++missing;
+ }
+ if (missing > 0) {
+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE refuses the start: layer %lld has %lld of its %lld "
+ "experts out of CUDA%d's VRAM (expert %lld the first) - every expert must be resident "
+ "(a profile that ranks every pair and room on the card; STRATA_EXPERT_NIBBLE_SKIP="
+ "<layers> keeps some layers' smaller GGUF slots)\n", (long long) l, (long long) missing,
+ (long long) g.n_expert, owner_dev, (long long) first);
+ return 1;
+ }
+ }
+ xcache.seal();
+ for (auto& stp : stages) stp->cache.seal();
+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE: all %lld experts of all %lld layers are VRAM-resident; "
+ "the expert caches are sealed (nothing writes a slot from here on)\n",
+ (long long) g.n_expert, (long long) g.n_layers);
+ }
// ---- LAYER 2: exact startability probe on the live free figures. Every session, cache and batch slot is
// carved by now, so `cudaMemGetInfo` reads what serve's own_fits (5855-5869) will read. Run serve's own
@@ -8043,6 +8483,7 @@ int main(int argc, char** argv) {
if (!cold.empty() && cold.back().first < heat(oi)) {
vi = cold.back().second;
const int32_t vs = host_res[(size_t) vi];
+ nib::guard_raw(layer, "the elastic K/V's slot move (kvg_ensure)");
if (xcache.slot_offset(vs + 1) - xcache.slot_offset(vs) < lay.blob_bytes(layer) ||
cudaMemcpy(xcache.device_slot(vs), xcache.device_slot((int32_t) kvg.lo),
(size_t) lay.blob_bytes(layer), cudaMemcpyDeviceToDevice) != cudaSuccess)
@@ -8145,6 +8586,7 @@ int main(int argc, char** argv) {
if (host_res[i] >= 0 || lay.blob_bytes(l) > room) continue;
const uint8_t* b = srcp->blob_stable(l, e);
if (b == nullptr) continue;
+ nib::guard_raw(l, "the elastic K/V's refill (kvg_trim)");
cudaMemcpyAsync(xcache.device_slot((int32_t) s), b, (size_t) lay.blob_bytes(l), cudaMemcpyHostToDevice,
nullptr);
filled.emplace_back(i, (int32_t) s);
@@ -8872,6 +9314,37 @@ int main(int argc, char** argv) {
#endif
bool any_loan = borrow != nullptr;
for (size_t i = 1; i < pf_parts.size(); ++i) any_loan = any_loan || pf_parts[i].first >= 0;
+ if (nib::any_on()) {
+ // STRATA_EXPERT_NIBBLE: the prompt path keeps its own buffers (a loan is refilled with GGUF blobs) and reads in
+ // the chunk it reads without the switch (the chunking is part of the replies' bits: auto's ceiling, or
+ // --prefill N, and no host-aware plan of smaller chunks). Otherwise refused, never a quietly different reply.
+ std::string why;
+ const int64_t expect = o.prefill_auto ? auto_ceiling : nib_chunk_asked;
+ if (any_loan) why = "the prompt path borrows expert-cache slots (a loan is refilled with GGUF blobs)";
+ else if (o.prefill_chunk != expect)
+ why = "the prompt chunk is " + std::to_string(o.prefill_chunk) + " tokens, not the " + std::to_string(expect) +
+ " it is without the switch when nothing limits it (that would change the replies' bits; an explicit "
+ "--prefill N that fits keeps N)";
+ else if (pf_plan_chunk > 0)
+ why = "the host-aware plan reads long prompts in chunks of " + std::to_string(pf_plan_chunk) + " tokens";
+ std::string room;
+ for (size_t i = 0; why.empty() && i <= stages.size(); ++i) {
+ const int dev = i == 0 ? -1 : stages[i - 1]->dev;
+ const strata::core::OnDevice on(dev);
+ size_t fb = 0, tb = 0;
+ if (cudaMemGetInfo(&fb, &tb) != cudaSuccess) { (void) cudaGetLastError(); continue; }
+ char b[64];
+ std::snprintf(b, sizeof b, "CUDA%d %.2f GiB", dev < 0 ? 0 : dev, (double) fb / 1073741824.0);
+ room += (room.empty() ? "" : ", ") + std::string(b);
+ }
+ if (!why.empty()) {
+ std::fprintf(stderr, "strata serve: STRATA_EXPERT_NIBBLE refuses the start: %s (STRATA_EXPERT_NIBBLE_SKIP=<layers> "
+ "keeps some layers' GGUF blocks and their smaller slots)\n", why.c_str());
+ return 1;
+ }
+ std::fprintf(stderr, "strata serve: STRATA_EXPERT_NIBBLE: prompt chunk %lld, no loan; free after the prompt "
+ "buffers: %s\n", (long long) o.prefill_chunk, room.c_str());
+ }
if (any_loan) {
if (borrow != nullptr)
std::fprintf(stderr, "strata serve: the prompt path borrows %lld CUDA0 cache slots (%.2f GiB)\n",
@@ -10544,6 +11017,7 @@ int main(int argc, char** argv) {
const size_t in = (size_t) s.layer * g.n_expert + s.in, out = (size_t) s.layer * g.n_expert + s.out;
const int32_t slot = fenced_slots.empty() ? host_res[out] : fenced_slots[si];
const uint8_t* b = srcp->blob_stable(s.layer, s.in);
+ nib::guard_raw(s.layer, "the adaptive tier's copy in");
const int stn = multi_gpu ? stage_of(s.layer) : 0; // the swap stays in the layer's own cache
GpuStage* gs = stn > 0 ? stages[(size_t) stn - 1].get() : nullptr;
const strata::core::OnDevice on(gs ? gs->dev : -1);
@@ -10778,6 +11252,7 @@ int main(int argc, char** argv) {
const strata::core::OnDevice on(h.dev);
for (const ASwap& w : aswaps) {
if (w.home != hi || !w.exchange) continue;
+ nib::guard_raw(w.layer, "the async adaptive tier's copy back (a_back)");
if (const cudaError_t e = cudaMemcpyAsync(src.exchange_buffer(w.j, w.layer), h.cache->device_slot(w.slot),
(size_t) strata::kernels::cpu::expert_layout().blob_bytes(w.layer),
cudaMemcpyDeviceToHost, h.stream); e != cudaSuccess) {
@@ -10815,6 +11290,7 @@ int main(int argc, char** argv) {
auto a_in_one = [&](const ASwap& w, const uint8_t* b) { // step 2: one copy in (H2D), on its home's stream
AHome& h = ahomes[(size_t) w.home];
const strata::core::OnDevice on(h.dev);
+ nib::guard_raw(w.layer, "the async adaptive tier's copy in (a_in_one)");
if (const cudaError_t e = cudaMemcpyAsync(h.cache->device_slot(w.slot), b,
(size_t) strata::kernels::cpu::expert_layout().blob_bytes(w.layer),
cudaMemcpyHostToDevice, h.stream); e != cudaSuccess) {
@@ -15459,6 +15935,27 @@ int main(int argc, char** argv) {
std::fprintf(stderr, "strata serve: cancelled request keeps its conversation: %lld tokens live\n",
(long long) pp_reached);
}
+ if (nib::config().recheck && nib::any_on() && !cancelled) {
+ // STRATA_EXPERT_NIBBLE_RECHECK=1 (debug): every converted slot's hash again on its own device, against the
+ // hash recorded at load - a path that wrote a slot after the seal is found here
+ std::string rerr;
+ int64_t bad_layer = -1;
+ const auto rt0 = Clock::now();
+ bool ok = nib_x.recheck(rerr, bad_layer);
+ int64_t checked = nib_x.n;
+ for (size_t i = 0; ok && i < nib_st.size(); ++i) {
+ ok = nib_st[i].recheck(rerr, bad_layer);
+ checked += nib_st[i].n;
+ }
+ if (!ok) {
+ std::fprintf(stderr, "strata serve: STRATA_EXPERT_NIBBLE_RECHECK: %s\n", rerr.c_str());
+ if (bad_layer >= 0) nib::fatal(bad_layer, "STRATA_EXPERT_NIBBLE_RECHECK", "a slot changed after the load");
+ std::fflush(stderr);
+ std::abort();
+ }
+ std::fprintf(stderr, "strata serve: STRATA_EXPERT_NIBBLE_RECHECK: %lld 4-bit slots hash as at load (%.0f ms)\n",
+ (long long) checked, std::chrono::duration<double, std::milli>(Clock::now() - rt0).count());
+ }
static const bool state_hash = std::getenv("STRATA_STATE_HASH") != nullptr;
if (state_hash && !cancelled && o.prompt_cache > 0) { // every finished request, ckpt=0 too (parity gates)
// DEBUG: a fingerprint of every part of the session over the positions it holds ([0, L)), and
@@ -15967,6 +16464,11 @@ int main(int argc, char** argv) {
} else if (chunk != o.prefill_chunk) {
k = 0; // a fixed chunk that does not fit: its own buffers, as before
}
+ if (k > 0 && nib::any_on()) { // STRATA_EXPERT_NIBBLE: the lent slots would be refilled with GGUF blobs
+ std::fprintf(stderr, "strata generate: STRATA_EXPERT_NIBBLE refuses the start: the prompt path would borrow %lld "
+ "cache slots (--no-prefill-borrow keeps its own buffers)\n", (long long) k);
+ return 1;
+ }
const int64_t blob = (int64_t) strata::kernels::cpu::expert_layout().max_blob;
#if defined(STRATA_GPU_METAL)
k = 0; // Metal: no loan of expert-cache slots (the prompt buffers are the Metal window's own); the refill after it copied into given-back allocations
@@ -16555,6 +17057,7 @@ int main(int argc, char** argv) {
const size_t in = (size_t) s.layer * g.n_expert + s.in, out = (size_t) s.layer * g.n_expert + s.out;
const int32_t slot = host_res[out];
const uint8_t* b = srcp->blob_stable(s.layer, s.in);
+ nib::guard_raw(s.layer, "the adaptive refill's copy in");
const size_t blob_n = (size_t) strata::kernels::cpu::expert_layout().blob_bytes(s.layer);
// asynchronous: the copies run while the MTP drafts; the next window waits for them
if (slot < 0 || b == nullptr) {