Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

dsv41 CPU lane: IQ1_M uses Strata's exact rows from one token (ggml's IQ1_M dot runs at 1.8 GB/s on AMD Zen 2: _pdep_u64 is microcoded there); portable ggml-cpu build has BMI2 off (IQ1_S 2.3 -> 37 GB/s, IQ1_M 1.8 -> 32 GB/s on an EPYC 7452)

Niko12216953d896d06e
diff --git a/CMakeLists.txt b/CMakeLists.txt
index acc764463..3efdcc464 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -1664,9 +1664,12 @@ if(STRATA_NATIVE_EXPERTS)
set(GGML_BUILD_EXAMPLES OFF CACHE BOOL "" FORCE)
if(STRATA_PORTABLE)
set(GGML_NATIVE OFF CACHE BOOL "" FORCE)
- foreach(_f GGML_AVX GGML_AVX2 GGML_FMA GGML_F16C GGML_BMI2)
+ foreach(_f GGML_AVX GGML_AVX2 GGML_FMA GGML_F16C)
set(${_f} ON CACHE BOOL "" FORCE)
endforeach()
+ # BMI2 off: ggml's IQ1_S / IQ1_M dots use _pdep_u64 with it, and AMD Zen 1/2 run PDEP in microcode (EPYC 7452, IQ1_M one-token
+ # dot 1.8 GB/s against 32 GB/s without it; IQ1_S 2.3 against 37). Intel loses nothing measurable: no other ggml-cpu quant uses it.
+ set(GGML_BMI2 OFF CACHE BOOL "" FORCE)
elseif(STRATA_ISA_FLOOR)
# the experimental older-CPU build (STRATA_ISA_FLOOR above): one ggml-cpu for the floor, ggml's "sandybridge"
# feature set (SSE4.2 + AVX) or SSE4.2 alone - not the build host's, which may be newer than the PC it runs on.
diff --git a/src/dsv41/ds4_gg_cpu.cpp b/src/dsv41/ds4_gg_cpu.cpp
index e303694b3..cede307a9 100644
--- a/src/dsv41/ds4_gg_cpu.cpp
+++ b/src/dsv41/ds4_gg_cpu.cpp
@@ -92,7 +92,10 @@ void gg_cpu_quant(int act, const float* x, int k, void* dst) {
// min_tokens(type) and these rows from there. DS4_GG_IQ256=0 never takes them, =1 always (the old opt-in).
static int g_iq256 = -1; // -1: from the environment and the CPU, 0 off, 1 always, 2 from min_tokens(type)
void gg_cpu_set_iq256(int mode) { g_iq256 = mode; }
-static int min_tokens(int type) { return type == 21 ? 1 : type == 29 ? 3 : 2; }
+// IQ1_M from one token too: ggml's own IQ1_M dot (and IQ1_S's) uses _pdep_u64 where the build has BMI2, which AMD Zen 1/2 run in
+// microcode - 1.8 GB/s against these rows' 33 GB/s on an EPYC 7452 (a 2.00 set decodes 3x slower on one card), 19-31 GB/s on Intel.
+// The rows are bit-identical, so taking them at every token count only changes the speed.
+static int min_tokens(int type) { return type == 21 ? 1 : type == 29 ? 1 : 2; }
static bool own_ok(int type, int impl) {
static std::once_flag once[2][32];
static bool ok[2][32];