Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
sycl: fused int8 DPAS expert products for the prompt path (ESIMD, Battlemage): IQ3_S gate/up + IQ4_NL down, int8 q8_1 activations, parity test, opt-in STRATA_PREFILL_MMQ=1; first correct version, not yet faster than dequantize + oneMKL
Niko12212856a8045eed
diff --git a/docs/INTEL_INT8_EXPERTS.md b/docs/INTEL_INT8_EXPERTS.mdnew file mode 100644index 000000000..37e2b67dd--- /dev/null+++ b/docs/INTEL_INT8_EXPERTS.md@@ -0,0 +1,57 @@+# Fused int8 DPAS expert products on Battlemage (branch f-151-b70)++Behaviour note, written before the kernel (2026-10-18), kept in step with `sycl/src/prefill/esimd_moe.dp.cpp`.++## What it replaces++The prompt path computes a routed expert as dequantize (gate/up and down to FP16, 9.8 MB written) then three oneMKL / SwiGLU+launches. On the Arc Pro B70 (IQ3_S, 4K prompt) that chain is ~38 us of GPU time per expert (dequant 25.6 us, GEMMs and+SwiGLU ~12 us) and 23% of the prompt. The fused path decodes the quantized weights in registers straight into the DPAS B+operand, so the FP16 copy never exists and the weights are read once, compressed (2.1 MB per expert instead of 9.8 MB written and+read back).++## Tiles and thresholds++* One hardware thread (ESIMD, sub-group 16, 256 GRF where the device builds it) owns 16 output rows of one expert, all of K, and+ `MT x 8` activation rows. `MT` = 2, 4, 8 or 12, chosen per launch from the mean number of 8-row tiles per expert in the group+ (<=2 -> 2, <=4 -> 4, <=8 -> 8, else 12; `STRATA_I8_MT` overrides). 12 keeps one decode for up to 96 rows.+* An expert with more rows than `8 MT` is split over row-block threads; each decodes the same weights again, so the redundancy is+ `ceil(rows / (8 MT))`.+* A 128-output work-group tile (the 16/32/64 rows x 128 outputs of the plan) is eight such threads side by side. There is no+ shared memory and no barrier, because the decode is per thread and the activations are read straight from L1/L2 as ready-made+ A tiles; this was chosen over SLM staging because the K step is one DPAS (32), the B fragment never leaves registers, and the+ tiled activations make each A operand one 256-byte block load.+* K step 32 (one s8 DPAS 8x16x32) with the 128-wide unrolling inside the superblock loop for IQ3_S (8 steps per 256).+* Group of up to 16 experts per launch (the same group the FP16 grouped XMX path holds in the prompt walk).++## Scales++* Activations: llama.cpp's q8_1 grid. Per row and 32 of K, `d = amax / 127`, `q = round(x / d)` half away from zero, stored as int8+ in DPAS A-operand order (`[8-row tile][K/32][8 rows x 32 bytes]`), with the 8 row scales of a tile contiguous.+* Weights: IQ3_S `d (1 + 2 s)` per 32 (exact in float), IQ4_NL `d` per 32; the int8 values are the codebook entries with their+ signs (IQ3_S: odd 1..15, IQ4_NL: -127..113), never rounded.+* Each 32-block's int32 dot product (<= 32 * 127 * 127 < 2^24: exact in int32 and exactly convertible to float) is converted and+ accumulated as `acc += float(dot) * (sw[n] * dx[m])` in float, K ascending. The only roundings are the float multiply/add and the+ activation quantization.+* After gate/up, `silu(gate) * up` (the FP16 path's formula, in float, not rounded to FP16) is quantized per 32 of the 640 FF+ values into the down input.++## Edge cases and fallback rules++* Row counts are padded to tiles of 8 with zero rows (stores are masked to the real rows). An expert with no rows is not in the+ group.+* IQ3_S rows are 110-byte blocks (2-byte aligned only): every field is read as dwords, two gathers and a funnel shift when it is+ not dword aligned, and the second gather is clamped to the matrix end (an unclamped read past the last block faulted the GPU once).+* Supported now: gate/up IQ3_S (ggml type 21) with K a multiple of 256, down IQ4_NL (20) with K a multiple of 32, output rows a+ multiple of 16. Any other type or shape makes `gate_up()` / `down()` return false and the caller keeps the FP16 path.+* `available()` is false without the ESIMD aspect and an s8 8x16x32 matrix combination (Alchemist has 8x8x32; non-Intel devices),+ `STRATA_I8_FORCE=0` turns it off.+* Selection (`prefill.cpp`): `STRATA_PREFILL_MMQ=1` (alias `STRATA_PF_I8=1`) runs the layers whose experts are the supported+ types through it, `=0` keeps the FP16 path, `=auto` uses the measured crossover (below). Default off until the owner decides after+ the quality check.++## Different bits++The activation quantization makes the result differ from the dequantize + oneMKL path by about 0.3-0.5% rms per product (the+q8_1 error), which is also what llama.cpp's MMQ path carries. `esimd_moe_parity` reports it against the FP16 path and against an+FP64 reference; KL and the task suites are in `phaseA-tests/notes-151-b70.md`.diff --git a/sycl/CMakeLists.txt b/sycl/CMakeLists.txtindex ba639519b..39be503d0 100644--- a/sycl/CMakeLists.txt+++ b/sycl/CMakeLists.txt@@ -245,7 +245,8 @@ target_link_libraries(strata_engine PUBLIC strata_core strata_kernels strata_kerstrata_resolve(_prefill src/prefill/gemm.cu src/prefill/kernels.cu src/prefill/prefill.cpp)add_library(strata_prefill STATIC ${_prefill}${CMAKE_CURRENT_SOURCE_DIR}/src/prefill/moe_fused_stub.cpp # upstream's fused int8 kernels: not built here- ${CMAKE_CURRENT_SOURCE_DIR}/src/prefill/xmx_moe.dp.cpp) # A770 port: grouped XMX expert GEMM+ ${CMAKE_CURRENT_SOURCE_DIR}/src/prefill/xmx_moe.dp.cpp # A770 port: grouped XMX expert GEMM+ ${CMAKE_CURRENT_SOURCE_DIR}/src/prefill/esimd_moe.dp.cpp) # 151-b70: fused int8 DPAS expert products (ESIMD, Battlemage)target_compile_options(strata_prefill PUBLIC -qmkl=sequential)target_link_options(strata_prefill PUBLIC -qmkl=sequential)target_link_libraries(strata_prefill PUBLIC strata_engine)@@ -387,6 +388,10 @@ if(STRATA_SYCL_PARITY)target_link_libraries(xmx_gemm_bench PRIVATE strata_kernels strata_prefill) # -qmkl for the oneMKL referenceadd_executable(xmx_group_parity ${CMAKE_CURRENT_SOURCE_DIR}/src/kernels/xmx_group_parity.cpp) # A770: fused grouped XMXtarget_link_libraries(xmx_group_parity PRIVATE strata_kernels strata_prefill)+ add_executable(esimd_moe_parity ${CMAKE_CURRENT_SOURCE_DIR}/src/kernels/esimd_moe_parity.cpp) # 151-b70: fused int8 expert products vs dequant+oneMKL and FP64+ target_link_libraries(esimd_moe_parity PRIVATE strata_kernels strata_prefill)+ add_executable(esimd_dpas_probe ${CMAKE_CURRENT_SOURCE_DIR}/src/kernels/esimd_dpas_probe.cpp) # ESIMD DPAS peak (registers only)+ add_executable(esimd_i8_bench ${CMAKE_CURRENT_SOURCE_DIR}/src/kernels/esimd_i8_bench.cpp) # ESIMD int8 DPAS GEMM on pre-decoded weights# one native expert three ways on real GGUF rows (ggml reference, CPU path, GPU native_expert_grouped):# native_expert_parity <shard1.gguf> [layer ...]if(STRATA_NATIVE_EXPERTS)diff --git a/sycl/include/strata/prefill/esimd_moe.hpp b/sycl/include/strata/prefill/esimd_moe.hppnew file mode 100644index 000000000..5243c0b19--- /dev/null+++ b/sycl/include/strata/prefill/esimd_moe.hpp@@ -0,0 +1,62 @@+// 151-b70: fused int8 DPAS expert products for the prompt path of the SYCL port (Battlemage, ESIMD).+//+// A group of experts' gate/up and down products straight from their quantized blobs: the weights are decoded in registers into+// the DPAS B operand (int8, one decode per 16 output rows and 32 of K), the activations are int8 with a float scale per 32 of K+// (llama.cpp's q8_1 grid, rounding half away from zero), each 32-block's int32 dot product is converted and scaled in float.+// The FP16 weights never exist in memory. The bits differ from the dequantize + oneMKL path (activations are quantized): the+// caller keeps it opt-in until a quality check (docs/INTEL.md, "int8 expert path").+//+// Row layout. A group's rows are the experts' rows back to back (cnt[0], cnt[1], ... rows, as the FP16 path has them). The+// int8 activations are stored per expert padded to a multiple of 8 rows, in DPAS A-operand order:+// Xq [padded rows / 8][K / 32][8 rows x 8 dwords] dx [padded rows / 8][K / 32][8 floats]+// so one 256-byte block load is an 8x32 A tile. Pad rows are zero.+#pragma once+#include <cstddef>+#include <cstdint>++namespace strata::prefill::i8 {++constexpr int kMaxGroup = 16;++/// Per group: the experts' row counts and where their (8-row padded) tiles start.+struct Layout {+ int n = 0;+ int cnt[kMaxGroup] = {};+ int row0[kMaxGroup + 1] = {}; // first row of expert i (unpadded), row0[n] = total rows+ int tile0[kMaxGroup + 1] = {}; // first 8-row tile of expert i in the padded layout, tile0[n] = total tiles+};+void make_layout(Layout& lo, const int32_t* cnt, int n);++/// True when this device can run the kernels (a Battlemage-class device: sub-group 16 and the s8 DPAS 8x16x32).+bool available(void* stream);++size_t xq_bytes(const Layout& lo, int K); // int8 activations, padded tiles+size_t dx_bytes(const Layout& lo, int K); // their scales (floats)++/// Scratch for one group: int8 activation tiles + scales for the gate/up input (K) and for the down input (n_ff). Grow-only,+/// owned by the library (device USM on the queue's device).+struct Workspace {+ int8_t* xq = nullptr;+ float* dx = nullptr;+ int8_t* hq = nullptr;+ float* dh = nullptr;+};+bool workspace(const Layout& lo, int K, int n_ff, Workspace& ws, void* stream);++/// FP16 rows X[row0[n]][K] -> int8 tiles + scales.+void quantize_rows(const uint16_t* X, const Layout& lo, int K, int8_t* Xq, float* dx, void* stream);++/// GU[row][v] (FP32, row stride 2 * n_ff), v = 2 r gate row r, 2 r + 1 up row r (the interleave swiglu_interleaved reads), for the+/// rows of the group. blob[i] holds expert i's gate rows at 0 and its up rows at up_off (ggml IQ3_S rows, K a multiple of 256).+/// Returns false when the type / shape is not handled (the caller keeps its other path).+bool gate_up(int gu_type, const uint8_t* const* blob, size_t up_off, int n_ff, const Layout& lo, int K, const int8_t* Xq,+ const float* dx, float* GU, void* stream);++/// Hq / dh: silu(gate) * up from GU (FP32, interleaved) quantized per 32 of n_ff into the same tile layout (K = n_ff).+void swiglu_quantize(const float* GU, const Layout& lo, int n_ff, int8_t* Hq, float* dh, void* stream);++/// Dm[row][n_embd] (FP32) = Hq . down^T, down rows (n_embd of them, K = n_ff) in ggml type d_type at blob[i] + down_off.+bool down(int d_type, const uint8_t* const* blob, size_t down_off, int n_embd, const Layout& lo, int n_ff, const int8_t* Hq,+ const float* dh, float* Dm, void* stream);++} // namespace strata::prefill::i8diff --git a/sycl/src/kernels/esimd_dpas_probe.cpp b/sycl/src/kernels/esimd_dpas_probe.cppnew file mode 100644index 000000000..4f5128809--- /dev/null+++ b/sycl/src/kernels/esimd_dpas_probe.cpp@@ -0,0 +1,68 @@+// esimd_dpas_probe (151-b70): the DPAS peak an ESIMD kernel reaches on one device, registers only.+// esimd_dpas_probe [reps=4096] [threads=8192]+// Modes: fp16 (8x16x16, fp32 acc), s8 (8x16x32, int32 acc). RC=8 rows per DPAS, MT row tiles x NT column tiles of+// accumulators per hardware thread; the A/B operands are loaded once (random bits), so the loop is pure DPAS.+// Build: icpx -fsycl -fsycl-targets=intel_gpu_bmg_g31 -O3 esimd_dpas_probe.cpp+#include <sycl/sycl.hpp>+#include <sycl/ext/intel/esimd.hpp>+#include <sycl/ext/intel/experimental/grf_size_properties.hpp>+#include <chrono>+#include <cstdio>+#include <cstdlib>+#include <vector>++using namespace sycl::ext::intel::esimd;+namespace xmx = sycl::ext::intel::esimd::xmx;++template <bool S8, int MT, int NT, int GRF>+double run(sycl::queue& q, int reps, int threads, uint32_t* src, float* out) {+ constexpr int KW = S8 ? 8 : 8; // dwords per row of A (K=32 s8 / K=16 fp16: 8 dwords)+ constexpr auto PA = S8 ? xmx::dpas_argument_type::s8 : xmx::dpas_argument_type::fp16;+ using CT = std::conditional_t<S8, int, float>;+ auto props = sycl::ext::oneapi::experimental::properties{sycl::ext::intel::experimental::grf_size<GRF>};+ auto t0 = std::chrono::steady_clock::now();+ q.submit([&](sycl::handler& h) {+ h.parallel_for(sycl::nd_range<1>(threads, 8), props, [=](sycl::nd_item<1> it) SYCL_ESIMD_KERNEL {+ const int tid = it.get_global_id(0);+ simd<int, 8 * KW> a[MT];+ simd<int, 16 * KW> b[NT];+ simd<CT, 128> acc[MT][NT];+ for (int i = 0; i < MT; ++i) a[i] = simd<int, 8 * KW>(0x3c003c00u + i * 0x01010101 + tid, 1);+ for (int j = 0; j < NT; ++j) b[j] = simd<int, 16 * KW>(0x3c003c00u + j * 0x00010001, 1);+ for (int i = 0; i < MT; ++i) for (int j = 0; j < NT; ++j) acc[i][j] = 0;+ for (int r = 0; r < reps; ++r) {+#pragma unroll+ for (int i = 0; i < MT; ++i)+#pragma unroll+ for (int j = 0; j < NT; ++j)+ acc[i][j] = xmx::dpas<8, 8, CT, CT, int, int, PA, PA>(acc[i][j], b[j], a[i]);+ }+ float s = 0;+ for (int i = 0; i < MT; ++i) for (int j = 0; j < NT; ++j) { simd<float, 128> f = acc[i][j]; s += reduce<float>(f, std::plus<>()); }+ out[tid] = s;+ });+ }).wait();+ return std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count();+}++int main(int argc, char** argv) {+ int reps = argc > 1 ? atoi(argv[1]) : 4096, threads = argc > 2 ? atoi(argv[2]) : 8192;+ sycl::queue q{sycl::gpu_selector_v, sycl::property::queue::in_order()};+ std::printf("device %s\n", q.get_device().get_info<sycl::info::device::name>().c_str());+ float* out = sycl::malloc_device<float>(threads, q);+ uint32_t* src = nullptr;+ auto rep = [&](const char* nm, double (*fn)(sycl::queue&, int, int, uint32_t*, float*), int mt, int nt, bool s8) {+ fn(q, 64, threads, src, out);+ double best = 1e30;+ for (int k = 0; k < 3; ++k) best = std::min(best, fn(q, reps, threads, src, out));+ const double ops = 2.0 * 8 * 16 * (s8 ? 32 : 16) * (double) mt * nt * reps * threads;+ std::printf("%-22s %.1f T%s/s (%.3f s)\n", nm, ops / best / 1e12, s8 ? "OPS" : "FLOPS", best);+ };+ rep("fp16 4x2 grf256", run<false, 4, 2, 256>, 4, 2, false);+ rep("fp16 4x2 grf128", run<false, 4, 2, 128>, 4, 2, false);+ rep("fp16 2x2 grf256", run<false, 2, 2, 256>, 2, 2, false);+ rep("s8 4x2 grf256", run<true, 4, 2, 256>, 4, 2, true);+ rep("s8 4x2 grf128", run<true, 4, 2, 128>, 4, 2, true);+ rep("s8 2x2 grf256", run<true, 2, 2, 256>, 2, 2, true);+ return 0;+}diff --git a/sycl/src/kernels/esimd_i8_bench.cpp b/sycl/src/kernels/esimd_i8_bench.cppnew file mode 100644index 000000000..c1444c784--- /dev/null+++ b/sycl/src/kernels/esimd_i8_bench.cpp@@ -0,0 +1,204 @@+// esimd_i8_bench (151-b70): an ESIMD int8 DPAS expert GEMM on pre-decoded weights, us per expert.+// esimd_i8_bench [M=80] [experts=64] [reps=10]+// Y[M][N] = sum_kb dx[m][kb] * sw[n][kb] * (Xq[m][kb*32..] . Wq[n][kb*32..]) (int8 x int8 -> int32 per 32 of K, then a float+// scale: the shape of a quantized-weight x q8_1 product). Operands are pre-tiled in DPAS register order:+// Xq [E][M/8][K/32][64 dwords] (8 rows x 32 bytes: an 8x32 A tile, row-major)+// dx [E][M/8][K/32][8 floats]+// Wq [E][N/16][K/32][8][16 dwords] (the B tile: dword r of column n at r*16+n)+// sw [E][N/16][K/32][16 floats]+// Build: icpx -fsycl -fsycl-targets=intel_gpu_bmg_g31 -O3 esimd_i8_bench.cpp+#include <sycl/sycl.hpp>+#include <sycl/ext/intel/esimd.hpp>+#include <sycl/ext/intel/experimental/grf_size_properties.hpp>+#include <chrono>+#include <cmath>+#include <cstdio>+#include <cstdlib>+#include <random>+#include <vector>+#include <utility>+#include <type_traits>++#ifndef EPIV+#define EPIV 1+#endif+using namespace sycl::ext::intel::esimd;+namespace xmx = sycl::ext::intel::esimd::xmx;++struct Args {+ const uint32_t* Xq; const float* dx; const uint32_t* Wq; const float* sw; float* Y;+ int M, N, K, MTILES, NTILES, alias;+};++template <int N, class F>+SYCL_ESIMD_FUNCTION inline void sfor(F&& f) {+ [&]<int... I>(std::integer_sequence<int, I...>) SYCL_ESIMD_FUNCTION {+ (f(std::integral_constant<int, I>{}), ...);+ }(std::make_integer_sequence<int, N>{});+}++template <int MT, int NT, int GRF, int EPI>+void run(sycl::queue& q, const Args a, int E) {+ const int nkb = a.K / 32;+ const int mtg = (a.MTILES + MT - 1) / MT, ntg = a.NTILES / NT;+ const size_t nthr = (size_t) E * mtg * ntg;+ auto props = sycl::ext::oneapi::experimental::properties{sycl::ext::intel::experimental::grf_size<GRF>};+ q.submit([&](sycl::handler& h) {+ h.parallel_for(sycl::range<1>(nthr), props, [=](sycl::id<1> id) SYCL_ESIMD_KERNEL {+ struct St {+ simd<uint32_t, 128 * NT> b; simd<float, 16 * NT> sw; simd<uint32_t, 64 * MT> a; simd<float, 8 * MT> dx;+ };+ const int t = (int) id[0];+ const int nt0 = (t % ntg) * NT, mt0 = ((t / ntg) % mtg) * MT, e = t / (ntg * mtg);+ simd<float, 128 * MT * NT> acc = 0.f;+ simd<int, 128 * MT * NT> acci = 0;+ const uint32_t* Xe = a.Xq + (size_t) e * a.MTILES * nkb * 64;+ const float* dxe = a.dx + (size_t) e * a.MTILES * nkb * 8;+ const uint32_t* We = a.Wq + (size_t) (a.alias ? 0 : e) * a.NTILES * nkb * 128;+ const float* swe = a.sw + (size_t) (a.alias ? 0 : e) * a.NTILES * nkb * 16;+ St sA, sB;+ auto ld = [&](int kb, St& s) SYCL_ESIMD_FUNCTION {+ sfor<NT>([&](auto jj) SYCL_ESIMD_FUNCTION {+ constexpr int j = decltype(jj)::value;+ s.b.template select<128, 1>(128 * j) = block_load<uint32_t, 128>(We + ((size_t) (nt0 + j) * nkb + kb) * 128);+ s.sw.template select<16, 1>(16 * j) = block_load<float, 16>(swe + ((size_t) (nt0 + j) * nkb + kb) * 16);+ });+ sfor<MT>([&](auto ii) SYCL_ESIMD_FUNCTION {+ constexpr int i = decltype(ii)::value;+ const int mi = (mt0 + i < a.MTILES) ? mt0 + i : a.MTILES - 1;+ s.a.template select<64, 1>(64 * i) = block_load<uint32_t, 64>(Xe + ((size_t) mi * nkb + kb) * 64);+ s.dx.template select<8, 1>(8 * i) = block_load<float, 8>(dxe + ((size_t) mi * nkb + kb) * 8);+ });+ };+ auto cmp = [&](int kb, St& s) SYCL_ESIMD_FUNCTION {+ sfor<MT>([&](auto ii) SYCL_ESIMD_FUNCTION {+ constexpr int i = decltype(ii)::value;+ sfor<NT>([&](auto jj) SYCL_ESIMD_FUNCTION {+ constexpr int j = decltype(jj)::value;+ constexpr int o = 128 * (i * NT + j);+ simd<int, 128> bv = s.b.template select<128, 1>(128 * j).template bit_cast_view<int>();+ simd<int, 64> av = s.a.template select<64, 1>(64 * i).template bit_cast_view<int>();+ if constexpr (EPI == 0) {+ simd<int, 128> c0 = acci.template select<128, 1>(o);+ acci.template select<128, 1>(o) = xmx::dpas<8, 8, int, int, int, int, xmx::dpas_argument_type::s8, xmx::dpas_argument_type::s8>(c0, bv, av);+ } else if constexpr (EPI == 1) {+ simd<int, 128> tmp = xmx::dpas<8, 8, int, int, int, int, xmx::dpas_argument_type::s8, xmx::dpas_argument_type::s8>(+ simd<int, 128>(0), bv, av);+ simd<float, 128> tf = tmp;+ simd<float, 16> sw = s.sw.template select<16, 1>(16 * j);+#pragma unroll+ for (int r = 0; r < 8; ++r) {+ const float dxr = s.dx[8 * i + r];+ acc.template select<16, 1>(o + 16 * r) += tf.template select<16, 1>(16 * r) * (sw * dxr);+ }+ } else {+ simd<int, 128> tmp = xmx::dpas<8, 8, int, int, int, int, xmx::dpas_argument_type::s8, xmx::dpas_argument_type::s8>(+ simd<int, 128>(0), bv, av);+ simd<float, 16> sw = s.sw.template select<16, 1>(16 * j);+ simd<int, 16> isw = convert<int>(sw * 50.f);+#pragma unroll+ for (int r = 0; r < 8; ++r)+ acci.template select<16, 1>(o + 16 * r) += tmp.template select<16, 1>(16 * r) * isw;+ if ((kb & 3) == 3) {+ simd<float, 128> tf = acci.template select<128, 1>(o);+ acci.template select<128, 1>(o) = 0;+#pragma unroll+ for (int r = 0; r < 8; ++r) {+ const float dxr = s.dx[8 * i + r];+ acc.template select<16, 1>(o + 16 * r) += tf.template select<16, 1>(16 * r) * (sw * dxr);+ }+ }+ }+ });+ });+ };+ ld(0, sA);+ for (int kb = 0; kb < nkb; kb += 2) {+ ld(kb + 1, sB);+ cmp(kb, sA);+ if (kb + 2 < nkb) ld(kb + 2, sA);+ cmp(kb + 1, sB);+ }+ if constexpr (EPI == 0) acc = acci;+ sfor<MT>([&](auto ii) SYCL_ESIMD_FUNCTION {+ constexpr int i = decltype(ii)::value;+ sfor<NT>([&](auto jj) SYCL_ESIMD_FUNCTION {+ constexpr int j = decltype(jj)::value;+#pragma unroll+ for (int r = 0; r < 8; ++r) {+ const int m = (mt0 + i) * 8 + r;+ if (m < a.M) block_store<float, 16>(a.Y + ((size_t) e * a.M + m) * a.N + (nt0 + j) * 16,+ acc.template select<16, 1>(128 * (i * NT + j) + 16 * r));+ }+ });+ });+ });+ });+}++int main(int argc, char** argv) {+ const int M = argc > 1 ? std::atoi(argv[1]) : 80;+ const int E = argc > 2 ? std::atoi(argv[2]) : 64;+ const int reps = argc > 3 ? std::atoi(argv[3]) : 10;+ sycl::queue q{sycl::gpu_selector_v, sycl::property::queue::in_order()};+ std::printf("device %s, M %d, %d experts\n", q.get_device().get_info<sycl::info::device::name>().c_str(), M, E);+ struct Shape { const char* nm; int N, K; } shapes[2] = {{"gate+up", 1280, 2560}, {"down", 2560, 640}};+ std::mt19937 rng(7);+ for (const Shape& sh : shapes) {+ const int N = sh.N, K = sh.K, nkb = K / 32, MTILES = (M + 7) / 8, NTILES = N / 16;+ const size_t nX = (size_t) E * MTILES * nkb * 64, nW = (size_t) E * NTILES * nkb * 128;+ std::vector<uint32_t> hX(nX), hW(nW);+ std::vector<float> hdx((size_t) E * MTILES * nkb * 8), hsw((size_t) E * NTILES * nkb * 16);+ for (auto& x : hX) x = rng(); for (auto& x : hW) x = rng() & 0x0f0f0f0fu; // small weights, |w| <= 15 like a grid+ for (auto& x : hdx) x = 0.01f * (1 + rng() % 8);+ for (auto& x : hsw) x = 0.02f * (1 + rng() % 8);+ uint32_t *dX = sycl::malloc_device<uint32_t>(nX, q), *dW = sycl::malloc_device<uint32_t>(nW, q);+ float *ddx = sycl::malloc_device<float>(hdx.size(), q), *dsw = sycl::malloc_device<float>(hsw.size(), q);+ float* dY = sycl::malloc_device<float>((size_t) E * M * N, q);+ q.memcpy(dX, hX.data(), nX * 4); q.memcpy(dW, hW.data(), nW * 4);+ q.memcpy(ddx, hdx.data(), hdx.size() * 4); q.memcpy(dsw, hsw.data(), hsw.size() * 4); q.wait();+ Args a{dX, ddx, dW, dsw, dY, M, N, K, MTILES, NTILES, argc > 4 ? std::atoi(argv[4]) : 0};+ auto time = [&](auto&& fn) {+ fn(); q.wait();+ double best = 1e30;+ for (int r = 0; r < reps; ++r) {+ const auto t0 = std::chrono::steady_clock::now();+ fn(); q.wait();+ best = std::min(best, std::chrono::duration<double, std::micro>(std::chrono::steady_clock::now() - t0).count());+ }+ return best / E;+ };+ const double ops = 2.0 * M * N * K;+ struct V { const char* nm; double t; } vs[6];+ vs[0] = {"2x2g256", time([&] { run<2, 2, 256, EPIV>(q, a, E); })};+ vs[1] = {"2x2g128", time([&] { run<2, 2, 128, EPIV>(q, a, E); })};+ vs[2] = {"1x2g128", time([&] { run<1, 2, 128, EPIV>(q, a, E); })};+ vs[3] = {"2x1g128", time([&] { run<2, 1, 128, EPIV>(q, a, E); })};+ vs[4] = {"1x1g128", time([&] { run<1, 1, 128, EPIV>(q, a, E); })};+ vs[5] = {"1x4g256", time([&] { run<1, 4, 256, EPIV>(q, a, E); })};+ // check expert 3 (last run's output: MT8 NT1) against a host reference+ std::vector<float> hY((size_t) M * N);+ q.memcpy(hY.data(), dY + (size_t) 3 * M * N, hY.size() * 4).wait();+ double maxerr = 0, maxref = 0;+ for (int m = 0; m < M; m += 5) for (int n = 0; n < N; n += 7) {+ double ref = 0;+ const int mt = m / 8, r = m % 8, nt = n / 16, c = n % 16;+ for (int kb = 0; kb < nkb; ++kb) {+ long long s = 0;+ for (int k = 0; k < 32; ++k) {+ // A tile: row r at dwords r*8 .. ; B tile: dword (k/4) of column c at (k/4)*16 + c+ const uint32_t aw = hX[(((size_t) 3 * MTILES + mt) * nkb + kb) * 64 + r * 8 + k / 4];+ const uint32_t bw = hW[(((size_t) 3 * NTILES + nt) * nkb + kb) * 128 + (k / 4) * 16 + c];+ s += (long long) (int8_t) (aw >> (8 * (k % 4))) * (int8_t) (bw >> (8 * (k % 4)));+ }+ ref += (double) s * hdx[(((size_t) 3 * MTILES + mt) * nkb + kb) * 8 + r] * hsw[(((size_t) 3 * NTILES + nt) * nkb + kb) * 16 + c];+ }+ maxref = std::max(maxref, std::fabs(ref)); maxerr = std::max(maxerr, std::fabs(ref - hY[(size_t) m * N + n]));+ }+ std::printf("%-8s N %d K %d:", sh.nm, N, K);+ for (auto& v : vs) std::printf(" %s %.1f us (%.0f TOPS)", v.nm, v.t, ops / v.t / 1e6);+ std::printf("\n max err %.3g of %.3g\n", maxerr, maxref);+ sycl::free(dX, q); sycl::free(dW, q); sycl::free(ddx, q); sycl::free(dsw, q); sycl::free(dY, q);+ }+ return 0;+}diff --git a/sycl/src/kernels/esimd_iq3s_decode_bench.cpp b/sycl/src/kernels/esimd_iq3s_decode_bench.cppnew file mode 100644index 000000000..09188bb9b--- /dev/null+++ b/sycl/src/kernels/esimd_iq3s_decode_bench.cpp@@ -0,0 +1,168 @@+// esimd_iq3s_decode_bench (151-b70): IQ3_S rows decoded to DPAS B fragments (int8) by an ESIMD kernel, lanes = 16 rows.+// esimd_iq3s_decode_bench [N=1280] [K=2560] [experts=16] [write=1]+// Output per expert: Wq [N/16][K/32][8][16] dwords (the B tile of a DPAS s8 8x16x32: dword r of row n at r*16+n), sw [N/16][K/32][16] floats+// (d * (1 + 2 s) per row and 32-block). Checked against a host decode with ggml's own grid. Reports us per expert and GB/s of+// compressed weights consumed. write=0 keeps the fragments in registers (an xor checksum) to separate decode from the plane write.+#include <sycl/sycl.hpp>+#include <sycl/ext/intel/esimd.hpp>+#include <sycl/ext/intel/experimental/grf_size_properties.hpp>+#include <sycl/ext/intel/experimental/esimd/memory.hpp>+#include <chrono>+#include <cmath>+#include <cstdio>+#include <cstdlib>+#include <cstring>+#include <random>+#include <vector>+#include <utility>+#include <type_traits>++#define GGML_COMMON_DECL_CPP+#define GGML_COMMON_IMPL_CPP+#include "ggml-common.h"++using namespace sycl::ext::intel::esimd;++constexpr int BLK = 110; // sizeof(block_iq3_s)++template <int N, class F>+SYCL_ESIMD_FUNCTION __attribute__((always_inline)) inline void sfor(F&& f) {+ [&]<int... I>(std::integer_sequence<int, I...>) SYCL_ESIMD_FUNCTION {+ (f(std::integral_constant<int, I>{}), ...);+ }(std::make_integer_sequence<int, N>{});+}++// unaligned 32-bit read at a 2-byte aligned byte offset (per lane) through two dword gathers+SYCL_ESIMD_FUNCTION inline simd<uint32_t, 16> ld32(const uint8_t* base, simd<uint32_t, 16> off) {+ simd<uint32_t, 16> a = off & ~3u;+ simd<uint32_t, 16> lo = gather<uint32_t, 16>(reinterpret_cast<const uint32_t*>(base), a);+ simd<uint32_t, 16> hi = gather<uint32_t, 16>(reinterpret_cast<const uint32_t*>(base), a + 4u);+ simd<uint32_t, 16> sh = (off & 2u) << 3; // 0 or 16+ return (lo >> sh) | ((hi << 16) << (16u - sh));+}+SYCL_ESIMD_FUNCTION inline simd<uint32_t, 16> ld16(const uint8_t* base, simd<uint32_t, 16> off) {+ simd<uint32_t, 16> a = off & ~3u;+ simd<uint32_t, 16> lo = gather<uint32_t, 16>(reinterpret_cast<const uint32_t*>(base), a);+ return (lo >> ((off & 2u) << 3)) & 0xffffu;+}++template <int WRITE>+void run(sycl::queue& q, const uint8_t* W, const uint32_t* grid, uint32_t* Wq, float* sw, uint32_t* sink, int N, int K, int E) {+ const int nsb = K / 256, ntiles = N / 16, rowb = nsb * BLK;+ const size_t nthr = (size_t) E * ntiles;+ auto props = sycl::ext::oneapi::experimental::properties{sycl::ext::intel::experimental::grf_size<128>};+ q.submit([&](sycl::handler& h) {+ h.parallel_for(sycl::range<1>(nthr), props, [=](sycl::id<1> id) SYCL_ESIMD_KERNEL {+ const int t = (int) id[0], nt = t % ntiles, e = t / ntiles;+ const uint8_t* We = W + (size_t) e * N * rowb;+ simd<uint32_t, 16> rowoff = (simd<uint32_t, 16>(0, 1) + (uint32_t) (nt * 16)) * (uint32_t) rowb;+ simd<uint32_t, 16> chk = 0;+ for (int sb = 0; sb < nsb; ++sb) {+ const uint32_t P = sb * BLK;+ simd<uint32_t, 16> d16 = ld16(We, rowoff + P);+ simd<uint16_t, 16> dlo = d16.template bit_cast_view<uint16_t>().template select<16, 2>(0);+ simd<sycl::half, 16> dh16 = dlo.template bit_cast_view<sycl::half>();+ simd<float, 16> d = dh16;+ simd<uint32_t, 16> sc4 = ld32(We, rowoff + (P + 106));+ simd<uint32_t, 16> qh4 = ld32(We, rowoff + (P + 66));+ simd<uint32_t, 16> qh8 = ld32(We, rowoff + (P + 70));+#pragma unroll+ for (int ib = 0; ib < 8; ++ib) {+ simd<uint32_t, 16> q0 = ld32(We, rowoff + (P + 2 + 8 * ib));+ simd<uint32_t, 16> q1 = ld32(We, rowoff + (P + 6 + 8 * ib));+ simd<uint32_t, 16> sg = ld32(We, rowoff + (P + 74 + 4 * ib));+ simd<uint32_t, 16> qhv = ((ib < 4 ? qh4 : qh8) >> (8 * (ib & 3))) & 0xffu;+ simd<uint32_t, 16> scn = ((sc4 >> (8 * (ib >> 1))) >> (4 * (ib & 1))) & 0xfu;+ simd<float, 16> swv = d * simd<float, 16>(simd<int, 16>(scn * 2u + 1u));+ simd<uint32_t, 128> frag;+#pragma unroll+ for (int r = 0; r < 8; ++r) {+ simd<uint32_t, 16> qsw = r < 4 ? q0 : q1;+ simd<uint32_t, 16> idx = ((qsw >> (8 * (r & 3))) & 0xffu) | (((qhv >> r) & 1u) << 8);+ simd<uint32_t, 16> g = gather<uint32_t, 16>(grid, idx << 2);+ simd<uint32_t, 16> nib = ((sg >> (8 * (r >> 1))) >> (4 * (r & 1))) & 0xfu;+ simd<uint32_t, 16> m = ((nib * 0x00204081u) & 0x01010101u);+ simd<uint32_t, 16> mask = m * 255u;+ frag.template select<16, 1>(16 * r) = (g ^ mask) + m;+ }+ if constexpr (WRITE) {+ const size_t fi = ((size_t) e * ntiles + nt) * (K / 32) + sb * 8 + ib;+ block_store<uint32_t, 128>(Wq + fi * 128, frag);+ block_store<float, 16>(sw + fi * 16, swv);+ } else {+ simd<uint32_t, 16> x = swv.template bit_cast_view<uint32_t>();+#pragma unroll+ for (int r = 0; r < 8; ++r) x ^= frag.template select<16, 1>(16 * r);+ chk ^= x;+ }+ }+ }+ if constexpr (!WRITE) block_store<uint32_t, 16>(sink + (size_t) t * 16, chk);+ });+ });+}++int main(int argc, char** argv) {+ const int N = argc > 1 ? std::atoi(argv[1]) : 1280, K = argc > 2 ? std::atoi(argv[2]) : 2560;+ const int E = argc > 3 ? std::atoi(argv[3]) : 16, WR = argc > 4 ? std::atoi(argv[4]) : 1;+ sycl::queue q{sycl::gpu_selector_v, sycl::property::queue::in_order()};+ std::printf("device %s, N %d K %d, %d experts, write %d\n", q.get_device().get_info<sycl::info::device::name>().c_str(), N, K, E, WR);+ const int nsb = K / 256, ntiles = N / 16, rowb = nsb * BLK;+ std::mt19937 rng(11);+ std::vector<uint8_t> hW((size_t) E * N * rowb);+ for (size_t i = 0; i < hW.size(); ++i) hW[i] = (uint8_t) rng();+ for (size_t b = 0; b < hW.size() / BLK; ++b) { // sane fp16 d+ const sycl::half dh = sycl::half(0.002f + 0.0001f * (rng() % 50));+ std::memcpy(&hW[b * BLK], &dh, 2);+ }+ uint8_t* dW = sycl::malloc_device<uint8_t>(hW.size() + 4096, q);+ uint32_t* dgrid = sycl::malloc_device<uint32_t>(512, q);+ uint32_t* dWq = sycl::malloc_device<uint32_t>((size_t) E * ntiles * (K / 32) * 128, q);+ float* dsw = sycl::malloc_device<float>((size_t) E * ntiles * (K / 32) * 16, q);+ uint32_t* dsink = sycl::malloc_device<uint32_t>((size_t) E * ntiles * 16, q);+ q.memcpy(dW, hW.data(), hW.size()); q.memcpy(dgrid, iq3s_grid, 512 * 4); q.wait();+ auto go = [&] { if (WR) run<1>(q, dW, dgrid, dWq, dsw, dsink, N, K, E); else run<0>(q, dW, dgrid, dWq, dsw, dsink, N, K, E); q.wait(); };+ go();+ double best = 1e30;+ for (int r = 0; r < 10; ++r) {+ const auto t0 = std::chrono::steady_clock::now();+ go();+ best = std::min(best, std::chrono::duration<double, std::micro>(std::chrono::steady_clock::now() - t0).count());+ }+ std::printf("decode: %.1f us per expert (%.0f GB/s of compressed weights)\n", best / E, (double) N * rowb / (best / E) / 1e3);+ if (WR) { // host check of expert 1+ const int e = E > 1 ? 1 : 0;+ std::vector<uint32_t> hQ((size_t) ntiles * (K / 32) * 128);+ std::vector<float> hS((size_t) ntiles * (K / 32) * 16);+ q.memcpy(hQ.data(), dWq + (size_t) e * hQ.size(), hQ.size() * 4).wait();+ q.memcpy(hS.data(), dsw + (size_t) e * hS.size(), hS.size() * 4).wait();+ long bad = 0, badS = 0;+ for (int n = 0; n < N; n += 3) {+ for (int sb = 0; sb < nsb; ++sb) {+ const uint8_t* blk = &hW[((size_t) e * N + n) * rowb + (size_t) sb * BLK];+ sycl::half dh; std::memcpy(&dh, blk, 2);+ const uint8_t *qs = blk + 2, *qh = blk + 66, *sgn = blk + 74, *scl = blk + 106;+ for (int ib = 0; ib < 8; ++ib) {+ const int sc = (scl[ib / 2] >> (4 * (ib % 2))) & 0xf;+ const float swr = (float) dh * (1 + 2 * sc);+ const size_t fi = ((size_t) (n / 16) * (K / 32) + sb * 8 + ib);+ if (std::fabs(hS[fi * 16 + n % 16] - swr) > 1e-6f * std::fabs(swr)) ++badS;+ for (int r = 0; r < 8; ++r) {+ const int idx = qs[8 * ib + r] | (((qh[ib] >> r) & 1) << 8);+ const uint8_t* g = (const uint8_t*) &iq3s_grid[idx];+ const uint8_t sbyte = sgn[4 * ib + r / 2];+ uint32_t want = 0;+ for (int j = 0; j < 4; ++j) {+ const int kk = 4 * (r & 1) + j;+ const int v = (sbyte >> kk & 1) ? -(int) g[j] : (int) g[j];+ want |= (uint32_t) (uint8_t) (int8_t) v << (8 * j);+ }+ if (hQ[fi * 128 + r * 16 + n % 16] != want) ++bad;+ }+ }+ }+ }+ std::printf("check expert %d: %ld bad dwords, %ld bad scales\n", e, bad, badS);+ }+ return 0;+}diff --git a/sycl/src/kernels/esimd_moe_parity.cpp b/sycl/src/kernels/esimd_moe_parity.cppnew file mode 100644index 000000000..269e67284--- /dev/null+++ b/sycl/src/kernels/esimd_moe_parity.cpp@@ -0,0 +1,263 @@+// esimd_moe_parity (151-b70): the fused int8 DPAS expert products (prefill/esimd_moe.dp.cpp) against+// (a) the engine's dequantize + oneMKL path (FP16 weights, FP16 activations), the path they replace,+// (b) an FP64 host reference from the blobs (IQ3_S gate/up, IQ4_NL down) with the unquantized activations,+// (c) an exact host reference on the SAME int8 activations the device produced (isolates the kernel arithmetic from the+// activation quantization: only the float accumulation order differs).+// and times both paths per expert. esimd_moe_parity [mean_rows=80] [experts=16] [reps=10] [seed=1]+// Exit status 1 when (c) is off by more than 1e-5 of the row's scale or (a)/(b) exceed the int8 activation error budget.+#include "strata/kernels/iq_kernels.hpp"+#include "strata/kernels/iq_dequant_expert.hpp"+#include "strata/prefill/esimd_moe.hpp"+#include "strata/prefill/kernels.hpp"+#include "strata/sycl_queue.hpp"+#include <sycl/sycl.hpp>+#include <dpct/dpct.hpp>+#include <dpct/blas_utils.hpp>+#include <algorithm>+#include <chrono>+#include <cmath>+#include <cstdio>+#include <cstdlib>+#include <cstring>+#include <random>+#include <vector>++#define GGML_TABLE_BEGIN(type, name, size) static const type name[size] = {+#define GGML_TABLE_END() };+#include "../prefill/iq_tables.inc"++namespace i8 = strata::prefill::i8;+static uint16_t f2h(float f) { return sycl::bit_cast<uint16_t>(sycl::half(f)); }+static float h2f(uint16_t h) { return (float) sycl::bit_cast<sycl::half>(h); }++// host decode of one IQ3_S row (K values) -> double+static void dec_iq3s(const uint8_t* row, int K, double* out) {+ for (int sb = 0; sb < K / 256; ++sb) {+ const uint8_t* blk = row + sb * 110;+ uint16_t dh; std::memcpy(&dh, blk, 2);+ const uint8_t *qs = blk + 2, *qh = blk + 66, *sgn = blk + 74, *scl = blk + 106;+ for (int ib = 0; ib < 8; ++ib) {+ const double d = (double) h2f(dh) * (1 + 2 * ((scl[ib / 2] >> (4 * (ib % 2))) & 0xf));+ for (int r = 0; r < 8; ++r) {+ const int idx = qs[8 * ib + r] | (((qh[ib] >> r) & 1) << 8);+ const uint8_t* g = (const uint8_t*) &iq3s_grid[idx];+ const uint8_t sbyte = sgn[4 * ib + r / 2];+ for (int j = 0; j < 4; ++j) {+ const int kk = 4 * (r & 1) + j;+ out[sb * 256 + ib * 32 + 4 * r + j] = d * ((sbyte >> kk & 1) ? -(int) g[j] : (int) g[j]);+ }+ }+ }+ }+}+static void dec_iq4nl(const uint8_t* row, int K, double* out) {+ for (int b = 0; b < K / 32; ++b) {+ const uint8_t* blk = row + b * 18;+ uint16_t dh; std::memcpy(&dh, blk, 2);+ for (int j = 0; j < 16; ++j) {+ out[b * 32 + j] = (double) h2f(dh) * kvalues_iq4nl[blk[2 + j] & 15];+ out[b * 32 + 16 + j] = (double) h2f(dh) * kvalues_iq4nl[blk[2 + j] >> 4];+ }+ }+}++int main(int argc, char** argv) {+ const int mean = argc > 1 ? std::atoi(argv[1]) : 80;+ const int n = argc > 2 ? std::atoi(argv[2]) : 16;+ const int reps = argc > 3 ? std::atoi(argv[3]) : 10;+ const int seed = argc > 4 ? std::atoi(argv[4]) : 1;+ sycl::queue* q = &dpct::get_in_order_queue();+ if (!i8::available(q)) { std::printf("esimd_moe_parity: SKIP (no sub-group 16 / s8 DPAS device)\n"); return 0; }+ dpct::blas::descriptor_ptr hd = new dpct::blas::descriptor();+ hd->set_queue(q);+ const int K = 2560, NFF = 640, NE = 2560;+ std::mt19937 rng(seed);+ std::vector<int32_t> cnt(n);+ std::gamma_distribution<double> gd(1.5, mean / 1.5);+ int tot = 0;+ for (auto& c : cnt) { c = std::max(1, (int) gd(rng)); tot += c; }+ if (n >= 3) { cnt[0] = 1; cnt[1] = 7; cnt[2] = 9; tot = 0; for (int c : cnt) tot += c; } // pad edge cases+ i8::Layout lo;+ i8::make_layout(lo, cnt.data(), n);++ const size_t gate_b = (size_t) NFF * (K / 256) * 110, down_b = (size_t) NE * (NFF / 32) * 18;+ const size_t blob_b = 2 * gate_b + down_b;+ std::vector<std::vector<uint8_t>> hb(n);+ std::vector<uint8_t*> db(n);+ for (int i = 0; i < n; ++i) {+ hb[i].resize(blob_b);+ for (auto& x : hb[i]) x = (uint8_t) rng();+ for (size_t o = 0; o < 2 * gate_b; o += 110) { const uint16_t d = f2h(0.0004f + 0.00002f * (rng() % 50)); std::memcpy(&hb[i][o], &d, 2); }+ for (size_t o = 2 * gate_b; o < blob_b; o += 18) { const uint16_t d = f2h(0.0004f + 0.00002f * (rng() % 50)); std::memcpy(&hb[i][o], &d, 2); }+ db[i] = sycl::malloc_device<uint8_t>(blob_b + 256, *q);+ q->memcpy(db[i], hb[i].data(), blob_b);+ }+ std::vector<uint16_t> hx((size_t) tot * K);+ std::normal_distribution<float> nd(0.f, 1.f);+ for (auto& x : hx) { float v = nd(rng) * 0.5f; if (rng() % 97 == 0) v *= 12.f; x = f2h(v); } // a few outliers+ uint16_t* dxh = sycl::malloc_device<uint16_t>(hx.size(), *q);+ q->memcpy(dxh, hx.data(), hx.size() * 2);+ int8_t* Xq = sycl::malloc_device<int8_t>(i8::xq_bytes(lo, K), *q);+ float* dX = sycl::malloc_device<float>(i8::dx_bytes(lo, K) / 4, *q);+ int8_t* Hq = sycl::malloc_device<int8_t>(i8::xq_bytes(lo, NFF), *q);+ float* dH = sycl::malloc_device<float>(i8::dx_bytes(lo, NFF) / 4, *q);+ float *GU = sycl::malloc_device<float>((size_t) tot * 2 * NFF, *q), *Dm = sycl::malloc_device<float>((size_t) tot * NE, *q);+ float *GUr = sycl::malloc_device<float>((size_t) tot * 2 * NFF, *q), *Dmr = sycl::malloc_device<float>((size_t) tot * NE, *q);+ uint16_t *Hh = sycl::malloc_device<uint16_t>((size_t) tot * NFF, *q);+ uint16_t* dwgu = sycl::malloc_device<uint16_t>((size_t) 2 * NFF * K, *q);+ uint16_t* dwd = sycl::malloc_device<uint16_t>((size_t) NE * NFF, *q);+ q->wait();+ std::vector<const uint8_t*> pb(n);+ if (std::getenv("I8_STAGE")) { for (int i = 0; i < n && i < 4; ++i) std::printf("blob %d at %p size %zu\n", i, (void*) db[i], blob_b); }+ for (int i = 0; i < n; ++i) pb[i] = db[i];++ auto fused = [&](bool all = true) {+ i8::quantize_rows(dxh, lo, K, Xq, dX, q);+ if (!i8::gate_up(21, pb.data(), 2 * 0 + gate_b, NFF, lo, K, Xq, dX, GU, q)) { std::printf("gate_up refused\n"); std::exit(2); }+ i8::swiglu_quantize(GU, lo, NFF, Hq, dH, q);+ if (!i8::down(20, pb.data(), 2 * gate_b, NE, lo, NFF, Hq, dH, Dm, q)) { std::printf("down refused\n"); std::exit(2); }+ (void) all;+ };+ auto ref = [&]() {+ int64_t o = 0;+ for (int i = 0; i < n; ++i) {+ strata::kernels::iq_dequant_expert_f16(21, 20, db[i], db[i] + gate_b, db[i] + 2 * gate_b, NFF, K, dwgu, dwd, q);+ const float a = 1.f, b = 0.f;+ dpct::blas::gemm(hd, oneapi::mkl::transpose::trans, oneapi::mkl::transpose::nontrans, 2 * NFF, cnt[i], K, &a, dwgu,+ dpct::library_data_t::real_half, K, dxh + o * K, dpct::library_data_t::real_half, K, &b,+ GUr + o * 2 * NFF, dpct::library_data_t::real_float, 2 * NFF, dpct::compute_type::f32);+ strata::prefill::swiglu_interleaved(GUr + o * 2 * NFF, Hh + o * NFF, cnt[i], q);+ dpct::blas::gemm(hd, oneapi::mkl::transpose::trans, oneapi::mkl::transpose::nontrans, NE, cnt[i], NFF, &a, dwd,+ dpct::library_data_t::real_half, NFF, Hh + o * NFF, dpct::library_data_t::real_half, NFF, &b,+ Dmr + o * NE, dpct::library_data_t::real_float, NE, dpct::compute_type::f32);+ o += cnt[i];+ }+ };+ auto tm = [&](auto f) {+ f(); q->wait();+ const auto t0 = std::chrono::steady_clock::now();+ while (std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count() < 0.5) { f(); q->wait(); } // GPU clocks up+ const auto a = std::chrono::steady_clock::now();+ for (int r = 0; r < reps; ++r) f();+ q->wait();+ return std::chrono::duration<double, std::micro>(std::chrono::steady_clock::now() - a).count() / reps;+ };++ if (const char* st = std::getenv("I8_STAGE")) { // bring-up: run the first N stages, synchronising after each+ const int ns = std::atoi(st);+ i8::quantize_rows(dxh, lo, K, Xq, dX, q); q->wait(); std::printf("stage 1 quantize ok%c", 10);+ if (ns >= 2) { i8::gate_up(21, pb.data(), gate_b, NFF, lo, K, Xq, dX, GU, q); q->wait(); std::printf("stage 2 gate_up ok%c", 10); }+ if (ns >= 3) { i8::swiglu_quantize(GU, lo, NFF, Hq, dH, q); q->wait(); std::printf("stage 3 swiglu ok%c", 10); }+ if (ns >= 4) { i8::down(20, pb.data(), 2 * gate_b, NE, lo, NFF, Hq, dH, Dm, q); q->wait(); std::printf("stage 4 down ok%c", 10); }+ if (const char* dd = std::getenv("STRATA_I8_DBG"); dd && (std::atoi(dd) & 128)) { // the down kernel stored rowoff + P per thread+ std::vector<uint32_t> o((size_t) 480 * 16);+ q->memcpy(o.data(), Dm, o.size() * 4).wait();+ uint32_t mx = 0, mn = ~0u;+ for (int th = 0; th < 480; ++th) for (int l = 0; l < 16; ++l) { mx = std::max(mx, o[th * 16 + l]); mn = std::min(mn, o[th * 16 + l]); }+ std::printf("rowoff+P: min %u max %u%c", mn, mx, 10); for (int th : {0, 1, 158, 159, 160, 161, 319, 320, 321, 479}) std::printf(" t%d lane0 %u row %u%c", th, o[th * 16], (o[th * 16] - 1408000) / 360, 10);+ }+ return 0;+ }+ fused(); q->wait();+ // ---- (c) exact reference on the device's int8 activations+ std::vector<int8_t> hXq(i8::xq_bytes(lo, K));+ std::vector<float> hdX(i8::dx_bytes(lo, K) / 4), hGU((size_t) tot * 2 * NFF);+ q->memcpy(hXq.data(), Xq, hXq.size());+ q->memcpy(hdX.data(), dX, hdX.size() * 4);+ q->memcpy(hGU.data(), GU, hGU.size() * 4).wait();+ const int nkb = K / 32;+ auto xq_at = [&](int tile, int r, int k) { return (int) hXq[(((size_t) tile * nkb + k / 32) * 8 + r) * 32 + (k % 32)]; };+ auto dx_at = [&](int tile, int r, int kb) { return (double) hdX[((size_t) tile * nkb + kb) * 8 + r]; };+ int fails = 0;+ double err_c = 0, mx_c = 0, err_b = 0, rms_b = 0, rms_bref = 0, mx_b = 0;+ std::vector<double> wrow(K);+ {+ int sampled = 0;+ for (int e = 0; e < n; ++e) {+ for (int rl = 0; rl < cnt[e]; rl += (cnt[e] > 12 ? 5 : 1)) {+ const int tile = lo.tile0[e] + rl / 8, r = rl % 8, row = lo.row0[e] + rl;+ for (int v = 0; v < 2 * NFF; v += 37) {+ dec_iq3s(&hb[e][(size_t) (v & 1 ? gate_b : 0) + (size_t) (v >> 1) * (K / 256) * 110], K, wrow.data());+ double refc = 0, refb = 0;+ for (int kb = 0; kb < nkb; ++kb) {+ double s = 0;+ for (int k = 0; k < 32; ++k) s += (double) xq_at(tile, r, kb * 32 + k) * wrow[kb * 32 + k];+ refc += s * dx_at(tile, r, kb);+ }+ for (int k = 0; k < K; ++k) refb += (double) h2f(hx[(size_t) row * K + k]) * wrow[k];+ const double got = hGU[(size_t) row * 2 * NFF + v];+ err_c = std::max(err_c, std::fabs(got - refc)); mx_c = std::max(mx_c, std::fabs(refc));+ err_b += (got - refb) * (got - refb); rms_bref += refb * refb; mx_b = std::max(mx_b, std::fabs(refb));+ ++sampled;+ }+ }+ }+ const bool ok = err_c <= 2e-5 * mx_c && std::sqrt(err_b / rms_bref) < 0.02;+ fails += !ok;+ std::printf("gate/up IQ3_S : exact-int8 reference max err %.2e of %.2e (%.1e) vs FP64 true: rms rel err %.2e (%d samples) %s\n",+ err_c, mx_c, err_c / mx_c, std::sqrt(err_b / rms_bref), sampled, ok ? "OK" : "FAIL");+ }+ { // down: needs the device's Hq / dH and the true H from the device GU (float math on the host)+ std::vector<int8_t> hHq(i8::xq_bytes(lo, NFF));+ std::vector<float> hdH(i8::dx_bytes(lo, NFF) / 4), hDm((size_t) tot * NE);+ q->memcpy(hHq.data(), Hq, hHq.size());+ q->memcpy(hdH.data(), dH, hdH.size() * 4);+ q->memcpy(hDm.data(), Dm, hDm.size() * 4).wait();+ const int nkh = NFF / 32;+ double ec = 0, mc = 0, eb = 0, rb = 0, hq_bad = 0;+ std::vector<double> drow(NFF);+ int sampled = 0;+ for (int e = 0; e < n; ++e)+ for (int rl = 0; rl < cnt[e]; rl += (cnt[e] > 12 ? 5 : 1)) {+ const int tile = lo.tile0[e] + rl / 8, r = rl % 8, row = lo.row0[e] + rl;+ // the true H (double) of this row, and a check of the device's swiglu quantization+ std::vector<double> H(NFF);+ for (int j = 0; j < NFF; ++j) {+ const double g = hGU[(size_t) row * 2 * NFF + 2 * j], u = hGU[(size_t) row * 2 * NFF + 2 * j + 1];+ H[j] = g / (1.0 + std::exp(-g)) * u;+ }+ for (int kb = 0; kb < nkh; ++kb) {+ double amax = 0; for (int k = 0; k < 32; ++k) amax = std::max(amax, std::fabs(H[kb * 32 + k]));+ const double dd = hdH[((size_t) tile * nkh + kb) * 8 + r];+ if (std::fabs(dd - amax / 127.0) > 1e-5 * amax + 1e-30) hq_bad += 1;+ }+ for (int v = 0; v < NE; v += 41) {+ dec_iq4nl(&hb[e][2 * gate_b + (size_t) v * nkh * 18], NFF, drow.data());+ double refc = 0, refb = 0;+ for (int kb = 0; kb < nkh; ++kb) {+ double s = 0;+ for (int k = 0; k < 32; ++k) s += (double) hHq[(((size_t) tile * nkh + kb) * 8 + r) * 32 + k] * drow[kb * 32 + k];+ refc += s * hdH[((size_t) tile * nkh + kb) * 8 + r];+ }+ for (int k = 0; k < NFF; ++k) refb += H[k] * drow[k];+ const double got = hDm[(size_t) row * NE + v];+ ec = std::max(ec, std::fabs(got - refc)); mc = std::max(mc, std::fabs(refc));+ eb += (got - refb) * (got - refb); rb += refb * refb;+ ++sampled;+ }+ }+ const bool ok = ec <= 2e-5 * mc && std::sqrt(eb / rb) < 0.03 && hq_bad == 0;+ fails += !ok;+ std::printf("down IQ4_NL : exact-int8 reference max err %.2e of %.2e (%.1e) vs FP64 true: rms rel err %.2e, swiglu scale mismatches %.0f (%d samples) %s\n",+ ec, mc, ec / mc, std::sqrt(eb / rb), hq_bad, sampled, ok ? "OK" : "FAIL");+ }+ // ---- (a) against the dequantize + oneMKL path, whole chain+ ref(); q->wait();+ {+ std::vector<float> a((size_t) tot * NE), b((size_t) tot * NE);+ q->memcpy(a.data(), Dmr, a.size() * 4);+ q->memcpy(b.data(), Dm, b.size() * 4).wait();+ double e2 = 0, r2 = 0;+ for (size_t i = 0; i < a.size(); ++i) { e2 += (double) (a[i] - b[i]) * (a[i] - b[i]); r2 += (double) a[i] * a[i]; }+ std::printf("chain vs dequant + oneMKL (FP16 path): rms rel diff %.2e (rows %d)\n", std::sqrt(e2 / r2), tot);+ }+ const double t_ref = tm(ref), t_f = tm([&] { fused(); });+ const double t_q = tm([&] { i8::quantize_rows(dxh, lo, K, Xq, dX, q); });+ const double t_gu = tm([&] { i8::gate_up(21, pb.data(), gate_b, NFF, lo, K, Xq, dX, GU, q); });+ const double t_sw = tm([&] { i8::swiglu_quantize(GU, lo, NFF, Hq, dH, q); });+ const double t_dn = tm([&] { i8::down(20, pb.data(), 2 * gate_b, NE, lo, NFF, Hq, dH, Dm, q); });+ std::printf("rows %d (%d experts, mean %.0f): dequant+oneMKL chain %.1f us/expert | fused int8 chain %.1f us/expert (%.2fx) [quantize %.1f, gate/up %.1f, swiglu+q %.1f, down %.1f]\n",+ tot, n, (double) tot / n, t_ref / n, t_f / n, t_ref / t_f, t_q / n, t_gu / n, t_sw / n, t_dn / n);+ std::printf("%s\n", fails ? "FAILED" : "all OK");+ return fails ? 1 : 0;+}diff --git a/sycl/src/prefill/esimd_moe.dp.cpp b/sycl/src/prefill/esimd_moe.dp.cppnew file mode 100644index 000000000..40794a832--- /dev/null+++ b/sycl/src/prefill/esimd_moe.dp.cpp@@ -0,0 +1,545 @@+// 151-b70: fused int8 DPAS expert products (ESIMD, Battlemage) - see include/strata/prefill/esimd_moe.hpp.+//+// Behaviour note (docs/INTEL.md, "int8 expert path"):+// * Tile. One hardware thread owns 16 output rows (a DPAS N tile; for gate/up the 16 virtual rows alternate gate/up) and up to+// MT x 8 activation rows (MT = 2, 4, 8 or 12, picked per group from the experts' mean row count). It walks K in steps of 32:+// decode the 16 weight rows' 32 values into the B operand (registers only), then for each of its row tiles one s8 DPAS 8x16x32+// into a zeroed int32 tile, converted to float and scaled by (weight scale of the row and 32-block) x (activation scale of the+// row and 32-block) into a float accumulator. A 128-wide work-group tile (the owner's 16/32/64 rows x 128 outputs) is eight of+// these threads side by side; there is no shared memory and no barrier.+// * Decode once per tile: the B operand is decoded once per thread and applied to all its MT row tiles. An expert with more rows+// than 8 x MT is split over row-block threads, each decoding again (the redundancy is ceil(rows / (8 MT)); MT 12 keeps it 1 up+// to 96 rows).+// * Scales. Activations: q8_1 as llama.cpp (d = amax / 127, q = round half away from zero), one float per row and 32 of K.+// Weights: IQ3_S d * (1 + 2 s) per 32 (exact in float), IQ4_NL d per 32; the int8 values are the codebook entries with their+// signs, never rounded. The int32 dot product of a 32-block is at most 32 * 127 * 15 (IQ3_S) so it is exact, and int32 -> float+// is exact below 2^24. The float accumulation order is fixed (K ascending).+// * Edge cases. Rows are padded to tiles of 8 with zeros (stores are masked to the real rows); a group of up to 16 experts per+// launch; K a multiple of 256 (IQ3_S) / 32 (IQ4_NL); output rows a multiple of 16. Blob reads never go past a matrix end+// (the last dword of a block is read as two dwords only when the block is not dword aligned).+// * Fallback. available() is false without sub-group 16 DPAS (a non-Battlemage device) and the caller keeps the dequantize ++// oneMKL path; gate_up()/down() return false for a type / shape they do not handle.+#include <sycl/sycl.hpp>+#include <sycl/ext/intel/esimd.hpp>+#include <sycl/ext/intel/experimental/grf_size_properties.hpp>+#include <sycl/ext/oneapi/matrix/matrix.hpp>+#include <algorithm>+#include <cstdio>+#include <cstdlib>+#include <cstring>+#include <mutex>+#include <type_traits>+#include <string>+#include <unordered_map>+#include "strata/sycl_queue.hpp"+#include "strata/prefill/esimd_moe.hpp"++namespace strata::prefill::i8 {+namespace {+#define GGML_TABLE_BEGIN(type, name, size) static const type name[size] = {+#define GGML_TABLE_END() };+#include "iq_tables.inc"+#undef GGML_TABLE_BEGIN+#undef GGML_TABLE_END++using namespace sycl::ext::intel::esimd;+namespace xmx = sycl::ext::intel::esimd::xmx;+using half = sycl::half;++constexpr int BLK_IQ3S = 110; // sizeof(block_iq3_s)+constexpr int BLK_IQ4NL = 18; // sizeof(block_iq4_nl)++struct Tables {+ uint32_t* grid3s = nullptr; // iq3s_grid[512]+ uint32_t* iq4 = nullptr; // [256]: byte b -> kvalues[b & 15] in byte 0, kvalues[b >> 4] in byte 1+};+Tables& tables(sycl::queue& q) {+ static std::mutex mu;+ static Tables t; // one device per process in the port+ std::lock_guard<std::mutex> lk(mu);+ if (!t.grid3s) {+ t.grid3s = sycl::malloc_device<uint32_t>(512, q);+ t.iq4 = sycl::malloc_device<uint32_t>(256, q);+ uint32_t h4[256];+ for (int b = 0; b < 256; ++b)+ h4[b] = (uint32_t) (uint8_t) kvalues_iq4nl[b & 15] | ((uint32_t) (uint8_t) kvalues_iq4nl[b >> 4] << 8);+ q.memcpy(t.grid3s, iq3s_grid, 512 * 4);+ q.memcpy(t.iq4, h4, 256 * 4);+ q.wait();+ }+ return t;+}++// unaligned 32-bit read at a 2-byte aligned byte offset (per lane) through two dword gathers+SYCL_ESIMD_FUNCTION inline simd<uint32_t, 16> ld32(const uint8_t* base, simd<uint32_t, 16> off) {+ simd<uint32_t, 16> a = off & ~3u;+ simd<uint32_t, 16> lo = gather<uint32_t, 16>(reinterpret_cast<const uint32_t*>(base), a);+ simd<uint32_t, 16> hi = gather<uint32_t, 16>(reinterpret_cast<const uint32_t*>(base), a + 4u);+ simd<uint32_t, 16> sh = (off & 2u) << 3; // 0 or 16+ return (lo >> sh) | ((hi << 16) << (16u - sh));+}+SYCL_ESIMD_FUNCTION inline simd<uint32_t, 16> ld16(const uint8_t* base, simd<uint32_t, 16> off) {+ simd<uint32_t, 16> a = off & ~3u;+ simd<uint32_t, 16> lo = gather<uint32_t, 16>(reinterpret_cast<const uint32_t*>(base), a);+ return (lo >> ((off & 2u) << 3)) & 0xffffu;+}+SYCL_ESIMD_FUNCTION inline simd<float, 16> half_of(simd<uint32_t, 16> h16) {+ simd<uint16_t, 16> lo = h16.template bit_cast_view<uint16_t>().template select<16, 2>(0);+ simd<half, 16> hh = lo.template bit_cast_view<half>();+ simd<float, 16> f = hh;+ return f;+}+SYCL_ESIMD_FUNCTION inline simd<int, 128> dpas_s8(simd<uint32_t, 128> b, simd<int, 64> a) {+ simd<int, 128> bi = b.template bit_cast_view<int>();+ return xmx::dpas<8, 8, int, int, int, int, xmx::dpas_argument_type::s8, xmx::dpas_argument_type::s8>(simd<int, 128>(0), bi, a);+}++struct GArgs {+ const uint8_t* blob[kMaxGroup];+ uint32_t sec_off; // gate/up: byte offset of the up rows; down: byte offset of the down rows+ int n;+ int row0[kMaxGroup + 1];+ int tile0[kMaxGroup + 1];+ int rb0[kMaxGroup + 1]; // first row block of expert i+ const int8_t* Xq;+ const float* dx;+ float* Y;+ const uint32_t* tab;+ int ntiles; // output tiles of 16+ int nkb; // K / 32+ int rowb; // bytes per weight row+ int ldy;+ int dbg; // bring-up switches (STRATA_I8_DBG): 1 skip the decode, 2 skip the activation loads, 4 skip the stores+};+++// The expert a work item belongs to, with constant-index reads of the kernel argument arrays (a dynamic index into the argument+// struct gave a garbage pointer in an ESIMD kernel on the B70: the first weight read faulted the GPU).+struct Pick { int e, row0, cnt, tile0, rb0; const uint8_t* blob; };+SYCL_ESIMD_FUNCTION inline Pick pick_expert(const GArgs& a, int rbg) {+ Pick p{0, a.row0[0], a.row0[1] - a.row0[0], a.tile0[0], a.rb0[0], a.blob[0]};+#pragma unroll+ for (int i = 1; i < kMaxGroup; ++i)+ if (i < a.n && rbg >= a.rb0[i]) p = Pick{i, a.row0[i], a.row0[i + 1] - a.row0[i], a.tile0[i], a.rb0[i], a.blob[i]};+ return p;+}++struct LaunchInfo { int nthreads; };+template <int MT>+void fill_args(GArgs& a, const Layout& lo, int ntiles_out, int& nthreads) {+ int rb = 0;+ for (int i = 0; i < lo.n; ++i) {+ a.row0[i] = lo.row0[i];+ a.tile0[i] = lo.tile0[i];+ a.rb0[i] = rb;+ const int ntl = (lo.cnt[i] + 7) / 8;+ rb += (ntl + MT - 1) / MT;+ }+ a.row0[lo.n] = lo.row0[lo.n];+ a.tile0[lo.n] = lo.tile0[lo.n];+ a.rb0[lo.n] = rb;+ a.n = lo.n;+ nthreads = rb * ntiles_out;+}++template <int MT> struct TagGuS3;+template <int MT> struct TagDnNL;++// ------------------------------------------------------------------------------------------------ gate/up, IQ3_S+template <int MT>+void launch_gu_iq3s(sycl::queue& q, const GArgs a, int nthreads, const uint32_t* grid) {+ auto props = sycl::ext::oneapi::experimental::properties{sycl::ext::intel::experimental::grf_size<256>};+ q.submit([&](sycl::handler& h) {+ h.parallel_for<TagGuS3<MT>>(sycl::range<2>((size_t) (nthreads / a.ntiles), (size_t) a.ntiles), props, [=](sycl::id<2> id) SYCL_ESIMD_KERNEL {+ // a 2-D grid: this device's ESIMD compiler gave `t % n` = t for a run-time n (integer / and % by a non-constant are not usable here)+ const int rbg = (int) id[0], nt = (int) id[1];+ const int t = rbg * a.ntiles + nt;+ const Pick pk = pick_expert(a, rbg);+ const int e = pk.e;+ const int cnt = pk.cnt;+ const int ntl = (cnt + 7) >> 3;+ const int tl0 = (rbg - pk.rb0) * MT;+ const int nvalid = (ntl - tl0) < MT ? (ntl - tl0) : MT;+ const int tbase = pk.tile0 + tl0;+ const uint8_t* We = pk.blob;+ const int nsb = a.nkb >> 3;+ simd<uint32_t, 16> lane(0, 1);+ simd<uint32_t, 16> rowoff = (lane & 1u) * a.sec_off + ((lane >> 1) + (uint32_t) (8 * nt)) * (uint32_t) a.rowb;+ simd<float, 128 * MT> acc = 0.f;+ const int* Xi = reinterpret_cast<const int*>(a.Xq);+ for (int sb = 0; sb < ((a.dbg & 8) ? 0 : nsb); ++sb) {+ const uint32_t P = (uint32_t) sb * BLK_IQ3S;+ simd<float, 16> d = half_of(ld16(We, rowoff + P));+ simd<uint32_t, 16> sc4 = ld32(We, rowoff + (P + 106));+ simd<uint32_t, 16> qh4 = ld32(We, rowoff + (P + 66));+ simd<uint32_t, 16> qh8 = ld32(We, rowoff + (P + 70));+#pragma unroll+ for (int ib = 0; ib < 8; ++ib) {+ simd<uint32_t, 128> frag;+ simd<float, 16> swv = d;+ if (a.dbg & 1) {+ frag = 0x01010101u;+ } else {+ simd<uint32_t, 16> q0 = ld32(We, rowoff + (P + 2 + 8 * ib));+ simd<uint32_t, 16> q1 = ld32(We, rowoff + (P + 6 + 8 * ib));+ simd<uint32_t, 16> sg = ld32(We, rowoff + (P + 74 + 4 * ib));+ simd<uint32_t, 16> qhv = ((ib < 4 ? qh4 : qh8) >> (8 * (ib & 3))) & 0xffu;+ simd<uint32_t, 16> scn = ((sc4 >> (8 * (ib >> 1))) >> (4 * (ib & 1))) & 0xfu;+ swv = d * simd<float, 16>(simd<int, 16>(scn * 2u + 1u));+#pragma unroll+ for (int r = 0; r < 8; ++r) {+ simd<uint32_t, 16> qsw = r < 4 ? q0 : q1;+ simd<uint32_t, 16> idx = ((qsw >> (8 * (r & 3))) & 0xffu) | (((qhv >> r) & 1u) << 8);+ simd<uint32_t, 16> g = gather<uint32_t, 16>(grid, idx << 2);+ simd<uint32_t, 16> nib = ((sg >> (8 * (r >> 1))) >> (4 * (r & 1))) & 0xfu;+ simd<uint32_t, 16> m = ((nib * 0x00204081u) & 0x01010101u);+ simd<uint32_t, 16> mask = m * 255u;+ frag.template select<16, 1>(16 * r) = (g ^ mask) + m;+ }+ }+ const int kb = sb * 8 + ib;+#pragma unroll+ for (int i = 0; i < MT; ++i) {+ if (i < nvalid) {+ simd<int, 64> av = 0x01010101;+ simd<float, 8> dxv = 1.f;+ if (!(a.dbg & 2)) {+ av = block_load<int, 64>(Xi + ((size_t) (tbase + i) * a.nkb + kb) * 64);+ dxv = block_load<float, 8>(a.dx + ((size_t) (tbase + i) * a.nkb + kb) * 8);+ }+ simd<float, 128> tf = dpas_s8(frag, av);+#pragma unroll+ for (int r = 0; r < 8; ++r) {+ const float dxr = dxv[r];+ acc.template select<16, 1>(128 * i + 16 * r) += tf.template select<16, 1>(16 * r) * (swv * dxr);+ }+ }+ }+ }+ }+#pragma unroll+ for (int i = 0; i < MT; ++i) {+ if (i < nvalid) {+#pragma unroll+ for (int r = 0; r < 8; ++r) {+ const int rl = (tl0 + i) * 8 + r;+ if (rl < cnt && !(a.dbg & 4))+ block_store<float, 16>(a.Y + ((size_t) (pk.row0 + rl)) * a.ldy + 16 * nt,+ acc.template select<16, 1>(128 * i + 16 * r));+ }+ }+ }+ });+ });+}++// ------------------------------------------------------------------------------------------------ down, IQ4_NL+template <int MT>+void launch_dn_iq4nl(sycl::queue& q, const GArgs a, int nthreads, const uint32_t* tab) {+ auto props = sycl::ext::oneapi::experimental::properties{sycl::ext::intel::experimental::grf_size<256>};+ q.submit([&](sycl::handler& h) {+ h.parallel_for<TagDnNL<MT>>(sycl::range<2>((size_t) (nthreads / a.ntiles), (size_t) a.ntiles), props, [=](sycl::id<2> id) SYCL_ESIMD_KERNEL {+ const int rbg = (int) id[0], nt = (int) id[1];+ const int t = rbg * a.ntiles + nt;+ const Pick pk = (a.dbg & 32) ? Pick{0, a.row0[0], a.row0[1] - a.row0[0], a.tile0[0], a.rb0[0], a.blob[0]} : pick_expert(a, rbg);+ const int e = pk.e;+ const int cnt = pk.cnt;+ const int ntl = (cnt + 7) >> 3;+ const int tl0 = (rbg - pk.rb0) * MT;+ const int nvalid = (ntl - tl0) < MT ? (ntl - tl0) : MT;+ const int tbase = pk.tile0 + tl0;+ const uint8_t* We = pk.blob;+ simd<uint32_t, 16> lane(0, 1);+ simd<uint32_t, 16> rowoff = (lane + (uint32_t) (16 * nt)) * (uint32_t) a.rowb + a.sec_off;+ simd<float, 128 * MT> acc = 0.f;+ const int* Xi = reinterpret_cast<const int*>(a.Xq);+ for (int kb = 0; kb < ((a.dbg & 8) ? 0 : a.nkb); ++kb) {+ const uint32_t P = (uint32_t) kb * BLK_IQ4NL;+ simd<float, 16> swv = 1.f;+ if (a.dbg & 128) { if (kb == 0) block_store<uint32_t, 16>(reinterpret_cast<uint32_t*>(a.Y) + (size_t) t * 16, rowoff + P); }+ else if (!(a.dbg & 16)) swv = half_of(ld16(We, rowoff + P));+ simd<uint32_t, 128> frag = 0x01010101u;+#pragma unroll+ for (int j = 0; j < ((a.dbg & 1) ? 0 : 4); ++j) {+ simd<uint32_t, 16> qd = ld32(We, rowoff + (P + 2 + 4 * j));+ // four byte-pair lookups (lo value in byte 0, hi value in byte 1), repacked into the two dwords+ simd<uint32_t, 16> t0 = gather<uint32_t, 16>(tab, (qd & 0xffu) << 2);+ simd<uint32_t, 16> t1 = gather<uint32_t, 16>(tab, ((qd >> 8) & 0xffu) << 2);+ simd<uint32_t, 16> t2 = gather<uint32_t, 16>(tab, ((qd >> 16) & 0xffu) << 2);+ simd<uint32_t, 16> t3 = gather<uint32_t, 16>(tab, (qd >> 24) << 2);+ frag.template select<16, 1>(16 * j) = (t0 & 0xffu) | ((t1 & 0xffu) << 8) | ((t2 & 0xffu) << 16) | (t3 << 24);+ frag.template select<16, 1>(16 * (j + 4)) = ((t0 >> 8) & 0xffu) | (t1 & 0xff00u) | ((t2 & 0xff00u) << 8) | ((t3 & 0xff00u) << 16);+ }+#pragma unroll+ for (int i = 0; i < MT; ++i) {+ if (i < nvalid) {+ simd<int, 64> av = 0x01010101;+ simd<float, 8> dxv = 1.f;+ if (!(a.dbg & 2)) {+ av = block_load<int, 64>(Xi + ((size_t) (tbase + i) * a.nkb + kb) * 64);+ dxv = block_load<float, 8>(a.dx + ((size_t) (tbase + i) * a.nkb + kb) * 8);+ }+ simd<float, 128> tf = dpas_s8(frag, av);+#pragma unroll+ for (int r = 0; r < 8; ++r) {+ const float dxr = dxv[r];+ acc.template select<16, 1>(128 * i + 16 * r) += tf.template select<16, 1>(16 * r) * (swv * dxr);+ }+ }+ }+ }+#pragma unroll+ for (int i = 0; i < MT; ++i) {+ if (i < nvalid) {+#pragma unroll+ for (int r = 0; r < 8; ++r) {+ const int rl = (tl0 + i) * 8 + r;+ if (rl < cnt && !(a.dbg & 4))+ block_store<float, 16>(a.Y + ((size_t) (pk.row0 + rl)) * a.ldy + 16 * nt,+ acc.template select<16, 1>(128 * i + 16 * r));+ }+ }+ }+ });+ });+}++int pick_mt(const Layout& lo) {+ int tiles = lo.tile0[lo.n];+ const double mean = lo.n ? (double) tiles / lo.n : 0;+ if (mean <= 2.0) return 2;+ if (mean <= 4.0) return 4;+ if (mean <= 8.0) return 8;+ return 12;+}+int dbg_flags() {+ static const int v = [] { const char* s = std::getenv("STRATA_I8_DBG"); return s ? std::atoi(s) : 0; }();+ return v;+}+int mt_override() {+ static const int v = [] { const char* s = std::getenv("STRATA_I8_MT"); return s ? std::atoi(s) : 0; }();+ return v;+}++template <class F>+void dispatch_mt(int mt, F&& f) {+ switch (mt) {+ case 2: f(std::integral_constant<int, 2>{}); break;+ case 4: f(std::integral_constant<int, 4>{}); break;+ case 8: f(std::integral_constant<int, 8>{}); break;+ default: f(std::integral_constant<int, 12>{}); break;+ }+}++// ------------------------------------------------------------------------------------------------ quantize / swiglu+struct QArgs {+ int n;+ int row0[kMaxGroup + 1];+ int tile0[kMaxGroup + 1];+ int nkb;+ int ld; // row length in elements of the source+};+class QuantTag;+class SwigluTag;++} // namespace++void make_layout(Layout& lo, const int32_t* cnt, int n) {+ lo.n = n;+ int r = 0, t = 0;+ for (int i = 0; i < n; ++i) {+ lo.cnt[i] = cnt[i];+ lo.row0[i] = r;+ lo.tile0[i] = t;+ r += cnt[i];+ t += (cnt[i] + 7) / 8;+ }+ lo.row0[n] = r;+ lo.tile0[n] = t;+}++bool available(void* stream) {+ sycl::queue& q = *strata::q_of(stream);+ static const bool ok = [&] {+ const char* v = std::getenv("STRATA_I8_FORCE");+ if (v && v[0] == '0') return false;+ sycl::device d = q.get_device();+ if (!d.has(sycl::aspect::ext_intel_esimd)) return false;+ namespace mx = sycl::ext::oneapi::experimental::matrix;+ bool s8 = false;+ for (const auto& c : d.get_info<sycl::ext::oneapi::experimental::info::device::matrix_combinations>())+ if (c.atype == mx::matrix_type::sint8 && c.btype == mx::matrix_type::sint8 && c.nsize == 16 && c.ksize == 32) s8 = true;+ return s8;+ }();+ return ok;+}++size_t xq_bytes(const Layout& lo, int K) { return (size_t) lo.tile0[lo.n] * 8 * K; }+size_t dx_bytes(const Layout& lo, int K) { return (size_t) lo.tile0[lo.n] * (K / 32) * 8 * sizeof(float); }++bool workspace(const Layout& lo, int K, int n_ff, Workspace& ws, void* stream) {+ sycl::queue& q = *strata::q_of(stream);+ static std::mutex mu;+ static Workspace cur;+ static size_t cap_x = 0, cap_h = 0;+ std::lock_guard<std::mutex> lk(mu);+ const size_t nx = xq_bytes(lo, K), nh = xq_bytes(lo, n_ff);+ if (nx > cap_x || nh > cap_h) {+ q.wait();+ if (cur.xq) { sycl::free(cur.xq, q); sycl::free(cur.dx, q); sycl::free(cur.hq, q); sycl::free(cur.dh, q); }+ cap_x = std::max(nx, cap_x + cap_x / 4);+ cap_h = std::max(nh, cap_h + cap_h / 4);+ cur.xq = sycl::malloc_device<int8_t>(cap_x, q);+ cur.dx = sycl::malloc_device<float>(cap_x / 32 / 8 * 8 + 64, q); // rows * K/32 floats = bytes / 32+ cur.hq = sycl::malloc_device<int8_t>(cap_h, q);+ cur.dh = sycl::malloc_device<float>(cap_h / 32 + 64, q);+ if (!cur.xq || !cur.dx || !cur.hq || !cur.dh) { cap_x = cap_h = 0; cur = Workspace{}; ws = Workspace{}; return false; }+ }+ ws = cur;+ return true;+}++void quantize_rows(const uint16_t* X, const Layout& lo, int K, int8_t* Xq, float* dx, void* stream) {+ if (lo.tile0[lo.n] == 0) return;+ sycl::queue& q = *strata::q_of(stream);+ QArgs a{};+ a.n = lo.n;+ for (int i = 0; i <= lo.n; ++i) { a.row0[i] = lo.row0[i]; a.tile0[i] = lo.tile0[i]; }+ a.nkb = K / 32;+ a.ld = K;+ const size_t nthreads = (size_t) lo.tile0[lo.n] * a.nkb;+ const half* Xh = reinterpret_cast<const half*>(X);+ uint32_t* Xd = reinterpret_cast<uint32_t*>(Xq);+ q.submit([&](sycl::handler& h) {+ h.parallel_for<QuantTag>(sycl::range<2>((size_t) lo.tile0[lo.n], (size_t) a.nkb), [=](sycl::id<2> id) SYCL_ESIMD_KERNEL {+ const int T = (int) id[0], kb = (int) id[1];+ int e = 0;+ while (e + 1 < a.n && T >= a.tile0[e + 1]) ++e;+ const int cnt = a.row0[e + 1] - a.row0[e];+ simd<float, 8> dxv = 0.f;+#pragma unroll+ for (int r = 0; r < 8; ++r) {+ const int rl = (T - a.tile0[e]) * 8 + r;+ simd<uint32_t, 8> qd = 0u;+ if (rl < cnt) {+ simd<half, 32> xh = block_load<half, 32>(Xh + ((size_t) (a.row0[e] + rl)) * a.ld + kb * 32);+ simd<float, 32> x = xh;+ float amax = hmax<float>(abs(x));+ const float dd = amax / 127.f;+ const float id127 = amax > 0.f ? 127.f / amax : 0.f;+ simd<float, 32> y = x * id127;+ simd<float, 32> adj = 0.5f;+ adj.merge(simd<float, 32>(-0.5f), y < 0.f);+ simd<int, 32> qi = convert<int>(y + adj);+ simd<int8_t, 32> q8 = convert<int8_t>(qi);+ qd = q8.template bit_cast_view<uint32_t>();+ dxv[r] = dd;+ }+ block_store<uint32_t, 8>(Xd + ((size_t) T * a.nkb + kb) * 64 + r * 8, qd);+ }+ block_store<float, 8>(dx + ((size_t) T * a.nkb + kb) * 8, dxv);+ });+ });+}++void swiglu_quantize(const float* GU, const Layout& lo, int n_ff, int8_t* Hq, float* dh, void* stream) {+ if (lo.tile0[lo.n] == 0) return;+ sycl::queue& q = *strata::q_of(stream);+ QArgs a{};+ a.n = lo.n;+ for (int i = 0; i <= lo.n; ++i) { a.row0[i] = lo.row0[i]; a.tile0[i] = lo.tile0[i]; }+ a.nkb = n_ff / 32;+ a.ld = 2 * n_ff;+ const size_t nthreads = (size_t) lo.tile0[lo.n] * a.nkb;+ uint32_t* Hd = reinterpret_cast<uint32_t*>(Hq);+ q.submit([&](sycl::handler& h) {+ h.parallel_for<SwigluTag>(sycl::range<2>((size_t) lo.tile0[lo.n], (size_t) a.nkb), [=](sycl::id<2> id) SYCL_ESIMD_KERNEL {+ const int T = (int) id[0], kb = (int) id[1];+ int e = 0;+ while (e + 1 < a.n && T >= a.tile0[e + 1]) ++e;+ const int cnt = a.row0[e + 1] - a.row0[e];+ simd<float, 8> dxv = 0.f;+#pragma unroll+ for (int r = 0; r < 8; ++r) {+ const int rl = (T - a.tile0[e]) * 8 + r;+ simd<uint32_t, 8> qd = 0u;+ if (rl < cnt) {+ simd<float, 64> v = block_load<float, 64>(GU + ((size_t) (a.row0[e] + rl)) * a.ld + kb * 64);+ simd<float, 32> g = v.template select<32, 2>(0);+ simd<float, 32> u = v.template select<32, 2>(1);+ simd<float, 32> x = g * u / (1.f + exp(-g));+ float amax = hmax<float>(abs(x));+ const float dd = amax / 127.f;+ const float id127 = amax > 0.f ? 127.f / amax : 0.f;+ simd<float, 32> y = x * id127;+ simd<float, 32> adj = 0.5f;+ adj.merge(simd<float, 32>(-0.5f), y < 0.f);+ simd<int, 32> qi = convert<int>(y + adj);+ simd<int8_t, 32> q8 = convert<int8_t>(qi);+ qd = q8.template bit_cast_view<uint32_t>();+ dxv[r] = dd;+ }+ block_store<uint32_t, 8>(Hd + ((size_t) T * a.nkb + kb) * 64 + r * 8, qd);+ }+ block_store<float, 8>(dh + ((size_t) T * a.nkb + kb) * 8, dxv);+ });+ });+}++bool gate_up(int gu_type, const uint8_t* const* blob, size_t up_off, int n_ff, const Layout& lo, int K, const int8_t* Xq,+ const float* dx, float* GU, void* stream) {+ if (lo.n == 0 || lo.tile0[lo.n] == 0) return true;+ if (gu_type != 21 || K % 256 != 0 || (2 * n_ff) % 16 != 0 || (up_off & 3) != 0) return false;+ sycl::queue& q = *strata::q_of(stream);+ Tables& tb = tables(q);+ GArgs a{};+ for (int i = 0; i < lo.n; ++i) a.blob[i] = blob[i];+ a.sec_off = (uint32_t) up_off;+ a.Xq = Xq; a.dx = dx; a.Y = GU;+ a.ntiles = 2 * n_ff / 16;+ a.nkb = K / 32;+ a.rowb = (K / 256) * BLK_IQ3S;+ a.ldy = 2 * n_ff;+ a.dbg = dbg_flags();+ const int mt = mt_override() ? mt_override() : pick_mt(lo);+ dispatch_mt(mt, [&](auto mtc) {+ constexpr int MT = decltype(mtc)::value;+ int nthreads = 0;+ fill_args<MT>(a, lo, a.ntiles, nthreads);+ launch_gu_iq3s<MT>(q, a, nthreads, tb.grid3s);+ });+ return true;+}++bool down(int d_type, const uint8_t* const* blob, size_t down_off, int n_embd, const Layout& lo, int n_ff, const int8_t* Hq,+ const float* dh, float* Dm, void* stream) {+ if (lo.n == 0 || lo.tile0[lo.n] == 0) return true;+ if (d_type != 20 || n_ff % 32 != 0 || n_embd % 16 != 0 || (down_off & 3) != 0) return false;+ sycl::queue& q = *strata::q_of(stream);+ Tables& tb = tables(q);+ GArgs a{};+ for (int i = 0; i < lo.n; ++i) a.blob[i] = blob[i];+ a.sec_off = (uint32_t) down_off;+ a.Xq = Hq; a.dx = dh; a.Y = Dm;+ a.ntiles = n_embd / 16;+ a.nkb = n_ff / 32;+ a.rowb = (n_ff / 32) * BLK_IQ4NL;+ a.ldy = n_embd;+ a.dbg = dbg_flags();+ const int mt = mt_override() ? mt_override() : pick_mt(lo);+ dispatch_mt(mt, [&](auto mtc) {+ constexpr int MT = decltype(mtc)::value;+ int nthreads = 0;+ fill_args<MT>(a, lo, a.ntiles, nthreads);+ if (a.dbg & 64) {+ std::fprintf(stderr, "i8 down: MT %d n %d threads %d ntiles %d nkb %d rowb %d ldy %d sec_off %u blob0 %p Xq %p dx %p Y %p tab %p\n",+ MT, a.n, nthreads, a.ntiles, a.nkb, a.rowb, a.ldy, a.sec_off, (const void*) a.blob[0], (const void*) a.Xq,+ (const void*) a.dx, (void*) a.Y, (const void*) tb.iq4);+ for (int i = 0; i <= a.n; ++i) std::fprintf(stderr, " e%d row0 %d tile0 %d rb0 %d blob %p\n", i, a.row0[i], a.tile0[i], a.rb0[i], i < a.n ? (const void*) a.blob[i] : nullptr);+ }+ launch_dn_iq4nl<MT>(q, a, nthreads, tb.iq4);+ });+ return true;+}++} // namespace strata::prefill::i8diff --git a/sycl/src/prefill/iq_tables.inc b/sycl/src/prefill/iq_tables.incnew file mode 100644index 000000000..0e9c2cd49--- /dev/null+++ b/sycl/src/prefill/iq_tables.inc@@ -0,0 +1,119 @@+// IQ lookup tables copied from llama.cpp ggml/src/ggml-common.h (MIT License, Copyright (c) 2023-2026 The ggml authors):+// kmask_iq2xs, ksigns_iq2xs, iq3xxs_grid, iq3s_grid, kvalues_iq4nl. Same values the dequantize kernels read.+// Included by esimd_moe.dp.cpp after defining GGML_TABLE_BEGIN / GGML_TABLE_END.+GGML_TABLE_BEGIN(uint8_t, kmask_iq2xs, 8)+ 1, 2, 4, 8, 16, 32, 64, 128+GGML_TABLE_END()+GGML_TABLE_BEGIN(uint8_t, ksigns_iq2xs, 128)+ 0, 129, 130, 3, 132, 5, 6, 135, 136, 9, 10, 139, 12, 141, 142, 15,+ 144, 17, 18, 147, 20, 149, 150, 23, 24, 153, 154, 27, 156, 29, 30, 159,+ 160, 33, 34, 163, 36, 165, 166, 39, 40, 169, 170, 43, 172, 45, 46, 175,+ 48, 177, 178, 51, 180, 53, 54, 183, 184, 57, 58, 187, 60, 189, 190, 63,+ 192, 65, 66, 195, 68, 197, 198, 71, 72, 201, 202, 75, 204, 77, 78, 207,+ 80, 209, 210, 83, 212, 85, 86, 215, 216, 89, 90, 219, 92, 221, 222, 95,+ 96, 225, 226, 99, 228, 101, 102, 231, 232, 105, 106, 235, 108, 237, 238, 111,+ 240, 113, 114, 243, 116, 245, 246, 119, 120, 249, 250, 123, 252, 125, 126, 255,+GGML_TABLE_END()+GGML_TABLE_BEGIN(uint32_t, iq3xxs_grid, 256)+ 0x04040404, 0x04040414, 0x04040424, 0x04040c0c, 0x04040c1c, 0x04040c3e, 0x04041404, 0x04041414,+ 0x04041c0c, 0x04042414, 0x04043e1c, 0x04043e2c, 0x040c040c, 0x040c041c, 0x040c0c04, 0x040c0c14,+ 0x040c140c, 0x040c142c, 0x040c1c04, 0x040c1c14, 0x040c240c, 0x040c2c24, 0x040c3e04, 0x04140404,+ 0x04140414, 0x04140424, 0x04140c0c, 0x04141404, 0x04141414, 0x04141c0c, 0x04141c1c, 0x04141c3e,+ 0x04142c0c, 0x04142c3e, 0x04143e2c, 0x041c040c, 0x041c043e, 0x041c0c04, 0x041c0c14, 0x041c142c,+ 0x041c3e04, 0x04240c1c, 0x04241c3e, 0x04242424, 0x04242c3e, 0x04243e1c, 0x04243e2c, 0x042c040c,+ 0x042c043e, 0x042c1c14, 0x042c2c14, 0x04341c2c, 0x04343424, 0x043e0c04, 0x043e0c24, 0x043e0c34,+ 0x043e241c, 0x043e340c, 0x0c04040c, 0x0c04041c, 0x0c040c04, 0x0c040c14, 0x0c04140c, 0x0c04141c,+ 0x0c041c04, 0x0c041c14, 0x0c041c24, 0x0c04243e, 0x0c042c04, 0x0c0c0404, 0x0c0c0414, 0x0c0c0c0c,+ 0x0c0c1404, 0x0c0c1414, 0x0c14040c, 0x0c14041c, 0x0c140c04, 0x0c140c14, 0x0c14140c, 0x0c141c04,+ 0x0c143e14, 0x0c1c0404, 0x0c1c0414, 0x0c1c1404, 0x0c1c1c0c, 0x0c1c2434, 0x0c1c3434, 0x0c24040c,+ 0x0c24042c, 0x0c242c04, 0x0c2c1404, 0x0c2c1424, 0x0c2c2434, 0x0c2c3e0c, 0x0c34042c, 0x0c3e1414,+ 0x0c3e2404, 0x14040404, 0x14040414, 0x14040c0c, 0x14040c1c, 0x14041404, 0x14041414, 0x14041434,+ 0x14041c0c, 0x14042414, 0x140c040c, 0x140c041c, 0x140c042c, 0x140c0c04, 0x140c0c14, 0x140c140c,+ 0x140c1c04, 0x140c341c, 0x140c343e, 0x140c3e04, 0x14140404, 0x14140414, 0x14140c0c, 0x14140c3e,+ 0x14141404, 0x14141414, 0x14141c3e, 0x14142404, 0x14142c2c, 0x141c040c, 0x141c0c04, 0x141c0c24,+ 0x141c3e04, 0x141c3e24, 0x14241c2c, 0x14242c1c, 0x142c041c, 0x142c143e, 0x142c240c, 0x142c3e24,+ 0x143e040c, 0x143e041c, 0x143e0c34, 0x143e242c, 0x1c04040c, 0x1c040c04, 0x1c040c14, 0x1c04140c,+ 0x1c04141c, 0x1c042c04, 0x1c04342c, 0x1c043e14, 0x1c0c0404, 0x1c0c0414, 0x1c0c1404, 0x1c0c1c0c,+ 0x1c0c2424, 0x1c0c2434, 0x1c14040c, 0x1c14041c, 0x1c140c04, 0x1c14142c, 0x1c142c14, 0x1c143e14,+ 0x1c1c0c0c, 0x1c1c1c1c, 0x1c241c04, 0x1c24243e, 0x1c243e14, 0x1c2c0404, 0x1c2c0434, 0x1c2c1414,+ 0x1c2c2c2c, 0x1c340c24, 0x1c341c34, 0x1c34341c, 0x1c3e1c1c, 0x1c3e3404, 0x24040424, 0x24040c3e,+ 0x24041c2c, 0x24041c3e, 0x24042c1c, 0x24042c3e, 0x240c3e24, 0x24141404, 0x24141c3e, 0x24142404,+ 0x24143404, 0x24143434, 0x241c043e, 0x241c242c, 0x24240424, 0x24242c0c, 0x24243424, 0x242c142c,+ 0x242c241c, 0x242c3e04, 0x243e042c, 0x243e0c04, 0x243e0c14, 0x243e1c04, 0x2c040c14, 0x2c04240c,+ 0x2c043e04, 0x2c0c0404, 0x2c0c0434, 0x2c0c1434, 0x2c0c2c2c, 0x2c140c24, 0x2c141c14, 0x2c143e14,+ 0x2c1c0414, 0x2c1c2c1c, 0x2c240c04, 0x2c24141c, 0x2c24143e, 0x2c243e14, 0x2c2c0414, 0x2c2c1c0c,+ 0x2c342c04, 0x2c3e1424, 0x2c3e2414, 0x34041424, 0x34042424, 0x34042434, 0x34043424, 0x340c140c,+ 0x340c340c, 0x34140c3e, 0x34143424, 0x341c1c04, 0x341c1c34, 0x34242424, 0x342c042c, 0x342c2c14,+ 0x34341c1c, 0x343e041c, 0x343e140c, 0x3e04041c, 0x3e04042c, 0x3e04043e, 0x3e040c04, 0x3e041c14,+ 0x3e042c14, 0x3e0c1434, 0x3e0c2404, 0x3e140c14, 0x3e14242c, 0x3e142c14, 0x3e1c0404, 0x3e1c0c2c,+ 0x3e1c1c1c, 0x3e1c3404, 0x3e24140c, 0x3e24240c, 0x3e2c0404, 0x3e2c0414, 0x3e2c1424, 0x3e341c04,+GGML_TABLE_END()+GGML_TABLE_BEGIN(uint32_t, iq3s_grid, 512)+ 0x01010101, 0x01010103, 0x01010105, 0x0101010b, 0x0101010f, 0x01010301, 0x01010303, 0x01010305,+ 0x01010309, 0x0101030d, 0x01010501, 0x01010503, 0x0101050b, 0x01010707, 0x01010901, 0x01010905,+ 0x0101090b, 0x0101090f, 0x01010b03, 0x01010b07, 0x01010d01, 0x01010d05, 0x01010f03, 0x01010f09,+ 0x01010f0f, 0x01030101, 0x01030103, 0x01030105, 0x01030109, 0x01030301, 0x01030303, 0x0103030b,+ 0x01030501, 0x01030507, 0x0103050f, 0x01030703, 0x0103070b, 0x01030909, 0x01030d03, 0x01030d0b,+ 0x01030f05, 0x01050101, 0x01050103, 0x0105010b, 0x0105010f, 0x01050301, 0x01050307, 0x0105030d,+ 0x01050503, 0x0105050b, 0x01050701, 0x01050709, 0x01050905, 0x0105090b, 0x0105090f, 0x01050b03,+ 0x01050b07, 0x01050f01, 0x01050f07, 0x01070107, 0x01070303, 0x0107030b, 0x01070501, 0x01070505,+ 0x01070703, 0x01070707, 0x0107070d, 0x01070909, 0x01070b01, 0x01070b05, 0x01070d0f, 0x01070f03,+ 0x01070f0b, 0x01090101, 0x01090307, 0x0109030f, 0x01090503, 0x01090509, 0x01090705, 0x01090901,+ 0x01090907, 0x01090b03, 0x01090f01, 0x010b0105, 0x010b0109, 0x010b0501, 0x010b0505, 0x010b050d,+ 0x010b0707, 0x010b0903, 0x010b090b, 0x010b090f, 0x010b0d0d, 0x010b0f07, 0x010d010d, 0x010d0303,+ 0x010d0307, 0x010d0703, 0x010d0b05, 0x010d0f03, 0x010f0101, 0x010f0105, 0x010f0109, 0x010f0501,+ 0x010f0505, 0x010f050d, 0x010f0707, 0x010f0b01, 0x010f0b09, 0x03010101, 0x03010103, 0x03010105,+ 0x03010109, 0x03010301, 0x03010303, 0x03010307, 0x0301030b, 0x0301030f, 0x03010501, 0x03010505,+ 0x03010703, 0x03010709, 0x0301070d, 0x03010b09, 0x03010b0d, 0x03010d03, 0x03010f05, 0x03030101,+ 0x03030103, 0x03030107, 0x0303010d, 0x03030301, 0x03030309, 0x03030503, 0x03030701, 0x03030707,+ 0x03030903, 0x03030b01, 0x03030b05, 0x03030f01, 0x03030f0d, 0x03050101, 0x03050305, 0x0305030b,+ 0x0305030f, 0x03050501, 0x03050509, 0x03050705, 0x03050901, 0x03050907, 0x03050b0b, 0x03050d01,+ 0x03050f05, 0x03070103, 0x03070109, 0x0307010f, 0x03070301, 0x03070307, 0x03070503, 0x0307050f,+ 0x03070701, 0x03070709, 0x03070903, 0x03070d05, 0x03070f01, 0x03090107, 0x0309010b, 0x03090305,+ 0x03090309, 0x03090703, 0x03090707, 0x03090905, 0x0309090d, 0x03090b01, 0x03090b09, 0x030b0103,+ 0x030b0301, 0x030b0307, 0x030b0503, 0x030b0701, 0x030b0705, 0x030b0b03, 0x030d0501, 0x030d0509,+ 0x030d050f, 0x030d0909, 0x030d090d, 0x030f0103, 0x030f0107, 0x030f0301, 0x030f0305, 0x030f0503,+ 0x030f070b, 0x030f0903, 0x030f0d05, 0x030f0f01, 0x05010101, 0x05010103, 0x05010107, 0x0501010b,+ 0x0501010f, 0x05010301, 0x05010305, 0x05010309, 0x0501030d, 0x05010503, 0x05010507, 0x0501050f,+ 0x05010701, 0x05010705, 0x05010903, 0x05010907, 0x0501090b, 0x05010b01, 0x05010b05, 0x05010d0f,+ 0x05010f01, 0x05010f07, 0x05010f0b, 0x05030101, 0x05030105, 0x05030301, 0x05030307, 0x0503030f,+ 0x05030505, 0x0503050b, 0x05030703, 0x05030709, 0x05030905, 0x05030b03, 0x05050103, 0x05050109,+ 0x0505010f, 0x05050503, 0x05050507, 0x05050701, 0x0505070f, 0x05050903, 0x05050b07, 0x05050b0f,+ 0x05050f03, 0x05050f09, 0x05070101, 0x05070105, 0x0507010b, 0x05070303, 0x05070505, 0x05070509,+ 0x05070703, 0x05070707, 0x05070905, 0x05070b01, 0x05070d0d, 0x05090103, 0x0509010f, 0x05090501,+ 0x05090507, 0x05090705, 0x0509070b, 0x05090903, 0x05090f05, 0x05090f0b, 0x050b0109, 0x050b0303,+ 0x050b0505, 0x050b070f, 0x050b0901, 0x050b0b07, 0x050b0f01, 0x050d0101, 0x050d0105, 0x050d010f,+ 0x050d0503, 0x050d0b0b, 0x050d0d03, 0x050f010b, 0x050f0303, 0x050f050d, 0x050f0701, 0x050f0907,+ 0x050f0b01, 0x07010105, 0x07010303, 0x07010307, 0x0701030b, 0x0701030f, 0x07010505, 0x07010703,+ 0x07010707, 0x0701070b, 0x07010905, 0x07010909, 0x0701090f, 0x07010b03, 0x07010d07, 0x07010f03,+ 0x07030103, 0x07030107, 0x0703010b, 0x07030309, 0x07030503, 0x07030507, 0x07030901, 0x07030d01,+ 0x07030f05, 0x07030f0d, 0x07050101, 0x07050305, 0x07050501, 0x07050705, 0x07050709, 0x07050b01,+ 0x07070103, 0x07070301, 0x07070309, 0x07070503, 0x07070507, 0x0707050f, 0x07070701, 0x07070903,+ 0x07070907, 0x0707090f, 0x07070b0b, 0x07070f07, 0x07090107, 0x07090303, 0x0709030d, 0x07090505,+ 0x07090703, 0x07090b05, 0x07090d01, 0x07090d09, 0x070b0103, 0x070b0301, 0x070b0305, 0x070b050b,+ 0x070b0705, 0x070b0909, 0x070b0b0d, 0x070b0f07, 0x070d030d, 0x070d0903, 0x070f0103, 0x070f0107,+ 0x070f0501, 0x070f0505, 0x070f070b, 0x09010101, 0x09010109, 0x09010305, 0x09010501, 0x09010509,+ 0x0901050f, 0x09010705, 0x09010903, 0x09010b01, 0x09010f01, 0x09030105, 0x0903010f, 0x09030303,+ 0x09030307, 0x09030505, 0x09030701, 0x0903070b, 0x09030907, 0x09030b03, 0x09030b0b, 0x09050103,+ 0x09050107, 0x09050301, 0x0905030b, 0x09050503, 0x09050707, 0x09050901, 0x09050b0f, 0x09050d05,+ 0x09050f01, 0x09070109, 0x09070303, 0x09070307, 0x09070501, 0x09070505, 0x09070703, 0x0907070b,+ 0x09090101, 0x09090105, 0x09090509, 0x0909070f, 0x09090901, 0x09090f03, 0x090b010b, 0x090b010f,+ 0x090b0503, 0x090b0d05, 0x090d0307, 0x090d0709, 0x090d0d01, 0x090f0301, 0x090f030b, 0x090f0701,+ 0x090f0907, 0x090f0b03, 0x0b010105, 0x0b010301, 0x0b010309, 0x0b010505, 0x0b010901, 0x0b010909,+ 0x0b01090f, 0x0b010b05, 0x0b010d0d, 0x0b010f09, 0x0b030103, 0x0b030107, 0x0b03010b, 0x0b030305,+ 0x0b030503, 0x0b030705, 0x0b030f05, 0x0b050101, 0x0b050303, 0x0b050507, 0x0b050701, 0x0b05070d,+ 0x0b050b07, 0x0b070105, 0x0b07010f, 0x0b070301, 0x0b07050f, 0x0b070909, 0x0b070b03, 0x0b070d0b,+ 0x0b070f07, 0x0b090103, 0x0b090109, 0x0b090501, 0x0b090705, 0x0b09090d, 0x0b0b0305, 0x0b0b050d,+ 0x0b0b0b03, 0x0b0b0b07, 0x0b0d0905, 0x0b0f0105, 0x0b0f0109, 0x0b0f0505, 0x0d010303, 0x0d010307,+ 0x0d01030b, 0x0d010703, 0x0d010707, 0x0d010d01, 0x0d030101, 0x0d030501, 0x0d03050f, 0x0d030d09,+ 0x0d050305, 0x0d050709, 0x0d050905, 0x0d050b0b, 0x0d050d05, 0x0d050f01, 0x0d070101, 0x0d070309,+ 0x0d070503, 0x0d070901, 0x0d09050b, 0x0d090907, 0x0d090d05, 0x0d0b0101, 0x0d0b0107, 0x0d0b0709,+ 0x0d0b0d01, 0x0d0d010b, 0x0d0d0901, 0x0d0f0303, 0x0d0f0307, 0x0f010101, 0x0f010109, 0x0f01010f,+ 0x0f010501, 0x0f010505, 0x0f01070d, 0x0f010901, 0x0f010b09, 0x0f010d05, 0x0f030105, 0x0f030303,+ 0x0f030509, 0x0f030907, 0x0f03090b, 0x0f050103, 0x0f050109, 0x0f050301, 0x0f05030d, 0x0f050503,+ 0x0f050701, 0x0f050b03, 0x0f070105, 0x0f070705, 0x0f07070b, 0x0f070b07, 0x0f090103, 0x0f09010b,+ 0x0f090307, 0x0f090501, 0x0f090b01, 0x0f0b0505, 0x0f0b0905, 0x0f0d0105, 0x0f0d0703, 0x0f0f0101,+GGML_TABLE_END()+GGML_TABLE_BEGIN(int8_t, kvalues_iq4nl, 16)+ -127, -104, -83, -65, -49, -35, -22, -10, 1, 13, 25, 38, 53, 69, 89, 113,+GGML_TABLE_END()diff --git a/sycl/src/prefill/prefill.cpp b/sycl/src/prefill/prefill.cppindex 9b7c125b5..236136fd8 100644--- a/sycl/src/prefill/prefill.cpp+++ b/sycl/src/prefill/prefill.cpp@@ -5,6 +5,7 @@#include "strata/sycl_queue.hpp"#include "strata/copy_queue.hpp"#include "strata/prefill/xmx_moe.hpp"+#include "strata/prefill/esimd_moe.hpp"#include "strata/prefill/prefill.hpp"#include "strata/core/gguf_expert_source.hpp"#include "../../../src/prefill/mmq_resident_sort.hpp"@@ -230,7 +231,20 @@ inline int ring_slots(size_t T) {}// A770 port: STRATA_PF_XMX=2 holds up to a group of staged experts until its launch, so routed-only staging needs// that many more slots for the lookahead to keep copying (the slots are only reused once the group's event is in)-inline int stage_slots() { return strata::prefill::xmx::mode() == 2 ? STAGE + strata::prefill::xmx::kMaxGroup : STAGE; }+// 151-b70: the fused int8 DPAS expert products (prefill/esimd_moe.dp.cpp, Battlemage). STRATA_PREFILL_MMQ=1 (or STRATA_PF_I8=1) turns+// them on for the layers whose experts are IQ3_S gate/up + IQ4_NL down, STRATA_PREFILL_MMQ=0 keeps the FP16 path; off by default+// until the owner has the quality check (the activations are quantized, so the bits differ from the FP16 path).+inline int i8_mode() {+ static const int v = [] {+ const char* e = std::getenv("STRATA_PREFILL_MMQ");+ if (e == nullptr || e[0] == 0) e = std::getenv("STRATA_PF_I8");+ if (e == nullptr || e[0] == 0) return 0;+ if (e[0] == 'a' || e[0] == 'A') return 2;+ return std::atoi(e) != 0 ? 1 : 0;+ }();+ return v;+}+inline int stage_slots() { return (strata::prefill::xmx::mode() == 2 || i8_mode() != 0) ? STAGE + strata::prefill::xmx::kMaxGroup : STAGE; }constexpr int DQ = 2; // dequantized-expert ring (FP16 gate/up + down)// A770 port: STRATA_PF_XMX=1 dequantizes a whole group of experts, then one grouped XMX launch per productinline int dq_slots() { return strata::prefill::xmx::mode() == 1 ? strata::prefill::xmx::kMaxGroup : DQ; }@@ -3945,7 +3959,14 @@ bool Prefill::run_impl(const int64_t *tokens, int64_t n, int64_t pos0,const int gt = lay.fmt[(size_t) l].gu_type, dt = lay.fmt[(size_t) l].d_type;return (gt == 18 || gt == 21 || gt == 22 || gt == 23) && (dt == 20 || dt == 42);};- const bool xmx2 = strata::prefill::xmx::mode() == 2 && !use_mmq && xmx2_fmt();+ const bool xmx2_only = strata::prefill::xmx::mode() == 2 && !use_mmq && xmx2_fmt();+ // 151-b70: the fused int8 DPAS products (IQ3_S gate/up, IQ4_NL down, the 640 x 2560 expert), the same+ // held group of staged experts as above+ const bool i8on = !xmx2_only && i8_mode() != 0 && !use_mmq && lay.native &&+ lay.fmt[(size_t) l].gu_type == 21 && lay.fmt[(size_t) l].d_type == 20 &&+ lay.fmt[(size_t) l].n_ff == 640 && lay.fmt[(size_t) l].n_embd == 2560 &&+ strata::prefill::i8::available(m.cs);+ const bool xmx2 = xmx2_only || i8on;if (strata::prefill::xmx::mode() == 2 && !xmx2) {static bool said = false;if (!said) {@@ -3965,13 +3986,46 @@ bool Prefill::run_impl(const int64_t *tokens, int64_t n, int64_t pos0,int64_t xg_r0 = 0;auto xflush = [&]() {if (xg_n == 0) return;+ const auto& f = lay.fmt[(size_t) l];+ int64_t nr = 0;+ for (int i = 0; i < xg_n; ++i) nr += xg_c[i];+ if (i8on) {+ if (static bool said = false; !said) {+ said = true;+ std::fprintf(stderr, "strata: prompt experts on the fused int8 DPAS products (STRATA_PREFILL_MMQ=1)\n");+ }+ namespace i8 = strata::prefill::i8;+ i8::Layout lo8;+ i8::make_layout(lo8, xg_c, xg_n);+ i8::Workspace ws8;+ const bool ws_ok = i8::workspace(lo8, (int) N, f.n_ff, ws8, m.cs);+ pt.mark(kPfGemmGU, cs);+ if (ws_ok &&+ (i8::quantize_rows(m.Xs + xg_r0 * N, lo8, (int) N, ws8.xq, ws8.dx, m.cs), true) &&+ i8::gate_up(f.gu_type, xg_g, f.up_off, f.n_ff, lo8, (int) N, ws8.xq, ws8.dx,+ m.GU + xg_r0 * 1280, m.cs)) {+ i8::swiglu_quantize(m.GU + xg_r0 * 1280, lo8, f.n_ff, ws8.hq, ws8.dh, m.cs);+ pt.mark(kPfGemmD, cs);+ i8::down(f.d_type, xg_g, f.down_off, (int) N, lo8, f.n_ff, ws8.hq, ws8.dh,+ m.Dm + xg_r0 * N, m.cs);+ } else {+ std::fprintf(stderr, "strata: the int8 expert products refused layer %lld (workspace %d)\n",+ (long long) l, (int) ws_ok);+ std::exit(1);+ }+ if (xg_ns > 0) {+ const int rel = xg_slots[xg_ns - 1];+ dpct::sync_barrier(m.used[rel], m.cs);+ for (int i = 0; i < xg_ns; ++i) m.used_of[xg_slots[i]] = rel;+ }+ xg_n = 0;+ xg_ns = 0;+ return;+ }if (static bool said = false; !said) {said = true;std::fprintf(stderr, "strata: prompt experts on the fused grouped XMX (STRATA_PF_XMX=2)\n");}- const auto& f = lay.fmt[(size_t) l];- int64_t nr = 0;- for (int i = 0; i < xg_n; ++i) nr += xg_c[i];// measured (xmx_group_parity, 16 experts): the fused launch wins below ~200 rows per// expert, dequantize + oneMKL above (gate/up 0.67-0.76x at 320); STRATA_PF_XMX_ROWS moves itstatic const int xg_max = [] {