LLM inference in C/C++
C++ ggml
Multi-column on Intel XMX matrix engines refactored (#29864)
budionyd81d0d8fb95e
diff --git a/docs/backend/SYCL.md b/docs/backend/SYCL.mdindex 81d293e4fe..249498450d 100644--- a/docs/backend/SYCL.md+++ b/docs/backend/SYCL.md@@ -819,6 +819,9 @@ User can use the device management in [docs/multi-gpu.md](https://github.com/ggm| GGML_SYCL_MKL_FA_DIAG | 0 (default) or 1 | Enable output fingerprinting for MKL flash attention. Dumps the first 64 float output values for the first 6 FA calls with n_kv ≥ 1024, labeled with kernel type (MKL/TILE/VEC) for cross-kernel comparison. || GGML_SYCL_ENABLE_FUSION | 0 or 1 (default) | Enable fused-kernel dispatch in graph compute. Unsupported types and layouts fall back to the standalone op kernels. See `ggml_sycl_can_fuse()`. || GGML_SYCL_ENABLE_ESIMD | 0 or 1 (default)| Enable ESIMD kernels when available. |+| GGML_SYCL_ENABLE_XMX_ESIMD | 0 or 1 (default) | Use the XMX (DPAS) multi-column mat-vec kernels for reordered Q2_K to Q6_K and Q8_0 weights on Xe2 GPUs. Speeds up speculative decoding and short prompt batches. |+| GGML_SYCL_XMX_MAX_COLS | non-negative integer, 80 (default) | Most activation columns (batch size) the `GGML_SYCL_ENABLE_XMX_ESIMD` kernels take; wider batches use the other mat-mul kernels. A larger value keeps XMX for bigger batches (as repeated launches), `0` never uses it. |+| GGML_SYCL_XMX_GLU_MAX_COLS | 0 to 16, 16 (default) | Most columns the fused gate and up XMX kernel takes; wider batches run gate and up as separate mat-vecs. 16 is the most the kernel is built for, larger values are clamped to 16 and `0` turns the fused kernel off. || GGML_SYCL_XMX_GATHER_TYPES | decimal bitmask, all bits set (default) | Weight formats that may use the XMX dequant-GEMM paths, which dequantize weights straight into the XMX tiles. This speeds up prompt processing of MoE models on GPUs with XMX units (Arc A- and B-series, Arc Pro, Data Center GPU Max), for example pp512 of Qwen3-30B-A3B UD-IQ3_XXS by about 50% on an Arc Pro B60. Bits:<br>* 1: IQ4_NL, 2: IQ3_S, 4: IQ4_XS, 8: IQ3_XXS, 16: IQ2_XXS, 32: IQ2_XS, 64: IQ2_S, 128: IQ1_S, 256: IQ1_M<br>* 512: Q8_0, 1024: Q4_K, 2048: Q5_K, 4096: Q6_K (MoE `MUL_MAT_ID` only)<br>Add values to combine them, for example `3` for IQ4_NL and IQ3_S; `0` disables the paths. A set bit does not force the path: batches of more than 64 tokens per expert or row lengths that are not a multiple of 256 (32 for IQ4_NL and Q8_0) use the library GEMM. || GGML_SYCL_XMX_GATHER_SHAPES | decimal bitmask, 255 (default) | XMX `joint_matrix` combinations the paths of `GGML_SYCL_XMX_GATHER_TYPES` may use; the operand type comes from `GGML_SYCL_DYNAMIC_PRECISION` and the best supported combination is picked automatically (logged as `fg_pick_combo`). Bits:<br>* Xe2, Xe3, Xe-HPC: 1: f16 8x16x16, 2: f16 16x16x16, 4: f16 32x64x16, 8: f16 32x64x32, 32: tf32 8x16x8, 64: bf16 8x16x16<br>* Xe-HPG (Arc A770, ARL-H): 16: f16 8x8x16, 128: bf16 8x8x16<br>Clear a bit to exclude a combination, or set a single bit to force one for testing. || GGML_SYCL_DYNAMIC_PRECISION | `F16` (default with `GGML_SYCL_F16=ON`), `BF16`, `TF32` or `F32` (default otherwise) | Operand type of the XMX dequant-GEMM paths (`GGML_SYCL_XMX_GATHER_TYPES`); accumulation is always f32. `F16` is the fastest, but activations above 65504 overflow. `BF16` keeps the f32 range at a 7-bit mantissa, `TF32` keeps the range and the f16 mantissa but is about 30% slower and needs Xe2, Xe3 or Xe-HPC, and `F32` turns the XMX paths off. Ops that request a higher src1 precision ([TAG_GGML_PREC]) get it regardless of this setting. |diff --git a/ggml/src/ggml-sycl/common.hpp b/ggml/src/ggml-sycl/common.hppindex 0bf6facec0..c6f285f963 100644--- a/ggml/src/ggml-sycl/common.hpp+++ b/ggml/src/ggml-sycl/common.hpp@@ -68,6 +68,9 @@ extern int g_ggml_sycl_enable_optimize;extern int g_ggml_sycl_enable_fusion;extern int g_ggml_sycl_enable_esimd;extern int g_ggml_sycl_mmvq_wide;+extern int g_ggml_sycl_enable_xmx_esimd;+extern int g_ggml_sycl_xmx_max_cols;+extern int g_ggml_sycl_xmx_glu_max_cols;extern int g_ggml_sycl_prioritize_dmmv;// Which quantized weight formats may take the XMX dequant-GEMM paths. A bitmask rather than one@@ -227,6 +230,41 @@ typedef sycl::float2 dfloat2;#define MMVQ_MAX_BATCH_SIZE 8+#if defined(__INTEL_LLVM_COMPILER) && __has_include(<sycl/ext/intel/esimd/xmx/dpas.hpp>)+# define GGML_SYCL_MMVQ_HAS_XMX+#endif // __INTEL_LLVM_COMPILER++// default of the most columns the XMX mul_mat_vec_q handles (GGML_SYCL_XMX_MAX_COLS), wider batches use other kernels+#define GGML_SYCL_XMX_MAX_COLS_DEFAULT 80++// most columns the fused gate and up XMX kernel is built for, GGML_SYCL_XMX_GLU_MAX_COLS can only lower this limit+#define GGML_SYCL_XMX_GLU_MAX_COLS 16++static inline bool ggml_sycl_xmx_supports_type(ggml_type type) {+ switch (type) {+ case GGML_TYPE_Q2_K:+ case GGML_TYPE_Q3_K:+ case GGML_TYPE_Q4_K:+ case GGML_TYPE_Q5_K:+ case GGML_TYPE_Q6_K:+ case GGML_TYPE_Q8_0:+ return true;+ default:+ return false;+ }+}++// the DPAS shapes and 2D block loads used by the XMX kernel are those of Xe2+static inline bool ggml_sycl_xmx_supports_arch(gpu_arch arch) {+ return arch == gpu_arch::intel_gpu_bmg_g21 || arch == gpu_arch::intel_gpu_bmg_g31 ||+ arch == gpu_arch::intel_gpu_lnl_m;+}++// fewest columns for which the XMX kernel is faster than the other kernels+static inline int ggml_sycl_xmx_min_cols(ggml_type type) {+ return type == GGML_TYPE_Q2_K || type == GGML_TYPE_Q3_K || type == GGML_TYPE_Q4_K ? 1 : 2;+}+static int g_all_sycl_device_count = -1;static bool g_ggml_backend_sycl_buffer_type_initialized = false;@@ -310,6 +348,10 @@ struct ggml_sycl_device_info {const ggml_sycl_device_info & ggml_sycl_info();+static inline bool ggml_sycl_xmx_enabled(int device) {+ return g_ggml_sycl_enable_xmx_esimd && ggml_sycl_xmx_supports_arch(ggml_sycl_info().devices[device].hw_info.arch);+}+static constexpr size_t SYCL_BUFFER_ALIGNMENT = 128;struct ggml_sycl_pool {diff --git a/ggml/src/ggml-sycl/esimd.hpp b/ggml/src/ggml-sycl/esimd.hppindex 0485ff0cee..85cc93e21e 100644--- a/ggml/src/ggml-sycl/esimd.hpp+++ b/ggml/src/ggml-sycl/esimd.hpp@@ -2,6 +2,8 @@#define GGML_SYCL_ESIMD_HPP#include <sycl/ext/intel/esimd.hpp>+#include <sycl/ext/intel/esimd/xmx/dpas.hpp>+#include <sycl/ext/intel/experimental/grf_size_properties.hpp>#include "common.hpp"@@ -584,6 +586,814 @@ template <> struct esimd_reorder_q_traits<GGML_TYPE_Q6_K> {}};+//+// Multi-column mul_mat_vec for reordered K-quant weights on the XMX engines (int8 DPAS).+// A thread owns 16 weight rows (B operand, one dword is 4 weights of a row), the q8_1 activation+// columns are the A operand. A super-block is 8 groups of 32, run as 2 parts of 4 groups. A quant+// type supplies its loads and unpack as a traits struct.+//+using sycl::ext::intel::esimd::simd;+constexpr int XMX_ROWS = 16; // weight rows per thread (DPAS N)+constexpr int XMX_CHUNK = 64; // widest single launch, more columns run as repeated launches+constexpr int XMX_FUSED_CHUNK = GGML_SYCL_XMX_GLU_MAX_COLS; // widest launch of the fused gate and up kernel++// threads that split K for one tile, sized so that the shared memory stays small+template <int NC> constexpr int xmx_n_split() {+ return NC <= 8 ? 32 : (NC == 16 ? 16 : 8);+}++inline bool xmx_supported(const void * vx, const int ncols) {+ // 2D block loads need a 64 byte aligned base+ return ncols % QK_K == 0 && (uintptr_t) vx % 64 == 0;+}++// transposed 2D block load of 16 rows, dword (kd, n) lands at [kd * 16 + n]+// 2D block loads need rows of at least 64 bytes, shorter rows (SR) are read row by row+template <int W, bool SR>+ESIMD_INLINE simd<uint32_t, W * XMX_ROWS> xmx_load_t(const void * base, unsigned pitch, unsigned h, int x, int y) {+ using namespace sycl::ext::intel::esimd;+ if constexpr (SR) {+ simd<uint32_t, W * XMX_ROWS> r;+ for (int n = 0; n < XMX_ROWS; ++n) {+ const unsigned row = y + n < (int) h ? y + n : h;+ const char * p = (const char *) base + (size_t) row * (pitch + 1);+ r.template select<W, XMX_ROWS>(n) = block_load<uint32_t, W>((const uint32_t *) p + x);+ }+ return r;+ } else {+ return load_2d<uint32_t, W, XMX_ROWS, 1, true, false>((const uint32_t *) base, pitch, h, pitch, x, y);+ }+}++// per byte (q - zero) as int8 for q in 0..127, no borrow between bytes+ESIMD_INLINE simd<uint32_t, 128> xmx_bytes_sub(simd<uint32_t, 128> x, uint32_t zero_x4) {+ return ((x | 0x80808080u) - zero_x4) ^ 0x80808080u;+}++// How the scales of a quant type enter the DPAS epilogue of a group of 32 (see xmx_group)+enum class xmx_scale_kind {+ split, // two scales per group, one per 16 wide half: the weights are masked to one half for each DPAS+ min, // one scale and one min per group: a second DPAS sums the activations for the min term+ min16, // a scale and a min per 16 wide half: the scales are folded into the DPAS operands+ plain, // one scale per group+};++// What every quant type states about itself. nc_wide is the tile width from which it takes the large register file.+template <bool SR, int NC_WIDE, xmx_scale_kind KIND> struct xmx_traits_base {+ static constexpr bool short_rows = SR;+ static constexpr int nc_wide = NC_WIDE;+ static constexpr bool has_min = KIND == xmx_scale_kind::min;+ static constexpr bool plain = KIND == xmx_scale_kind::plain;+ static constexpr bool min16 = KIND == xmx_scale_kind::min16;+};++// The part of a context every quant type shares: the last row (the clamp of 2D loads), this thread's first row and+// the number of super-blocks per row. A type adds its array pointers and 2D load pitches.+struct xmx_ctx_base {+ unsigned h;+ int row0;+ int bpr;+};++ESIMD_INLINE void xmx_ctx_init(xmx_ctx_base & c, int nrows, int bpr, int row0) {+ c.h = nrows - 1;+ c.row0 = row0;+ c.bpr = bpr;+}++// pitch of a 2D load over an array with `bytes` bytes per super-block+constexpr unsigned xmx_pitch(int bpr, int bytes) {+ return bpr * bytes - 1;+}++// per group row scales of the 16 rows, one vector per group+template <int N> struct xmx_row_scales {+ simd<float, 16> v[N];++ ESIMD_INLINE simd<float, 16> get(int i) { return v[i]; }+};++// the 12 packed scale bytes of a super-block (Q3_K, Q4_K, Q5_K) of the 16 rows as three dwords+ESIMD_INLINE void xmx_load_sc12(const uint8_t * sc,+ const simd<uint32_t, 16> & bi,+ simd<uint32_t, 16> & w0,+ simd<uint32_t, 16> & w1,+ simd<uint32_t, 16> & w2) {+ using namespace sycl::ext::intel::esimd;+ w0 = gather<uint32_t, 16>((const uint32_t *) sc, bi * 12);+ w1 = gather<uint32_t, 16>((const uint32_t *) sc, bi * 12 + 4);+ w2 = gather<uint32_t, 16>((const uint32_t *) sc, bi * 12 + 8);+}++// d and dmin (a half2 per super-block, Q2_K, Q4_K, Q5_K) of the 16 rows+ESIMD_INLINE void xmx_load_dm(const sycl::half * dm,+ const simd<uint32_t, 16> & bi,+ simd<float, 16> & d,+ simd<float, 16> & dmin) {+ using namespace sycl::ext::intel::esimd;+ d = convert<float>(gather<sycl::half, 16>(dm, bi * (2 * (uint32_t) sizeof(sycl::half))));+ dmin = convert<float>(+ gather<sycl::half, 16>(dm, bi * (2 * (uint32_t) sizeof(sycl::half)) + (uint32_t) sizeof(sycl::half)));+}++// Q6_K reorder layout: [ql: nb*128] [qh: nb*64] [scales int8: nb*16] [d: nb*2]+template <bool SR> struct xmx_traits_q6_k : xmx_traits_base<SR, 7, xmx_scale_kind::split> {+ struct ctx : xmx_ctx_base {+ const uint8_t * ql;+ const uint8_t * qh;+ const int8_t * scales;+ const sycl::half * dh;+ unsigned pl, ph, ps;+ };++ static ESIMD_INLINE ctx make(const void * vx, int nrows, int bpr, int row0) {+ const size_t nb = (size_t) nrows * bpr;+ ctx c;+ xmx_ctx_init(c, nrows, bpr, row0);+ c.ql = (const uint8_t *) vx;+ c.qh = c.ql + nb * (QK_K / 2);+ c.scales = (const int8_t *) (c.qh + nb * (QK_K / 4));+ c.dh = (const sycl::half *) (c.scales + nb * (QK_K / 16));+ c.pl = xmx_pitch(bpr, QK_K / 2);+ c.ph = xmx_pitch(bpr, QK_K / 4);+ c.ps = xmx_pitch(bpr, QK_K / 16);+ return c;+ }++ // row scales of the 16 scale groups of a super-block+ struct scales_t {+ simd<int8_t, 256> sc8;+ simd<float, 16> dv;++ ESIMD_INLINE simd<float, 16> get(int gr) {+ simd<int8_t, 16> s = sc8.template select<16, 4>(64 * (gr / 4) + (gr % 4));+ return sycl::ext::intel::esimd::convert<float>(s) * dv;+ }+ };++ // rows: clamped row index of each of the 16 lanes+ static ESIMD_INLINE scales_t load_scales(const ctx & c, int sb, const simd<uint32_t, 16> & rows) {+ using namespace sycl::ext::intel::esimd;+ scales_t s;+ s.dv = convert<float>(gather<sycl::half, 16>(c.dh, (rows * c.bpr + sb) * (uint32_t) sizeof(sycl::half)));+ simd<uint32_t, 64> raw = xmx_load_t<4, SR>(c.scales, c.ps, c.h, sb * 4, c.row0);+ s.sc8 = raw.template bit_cast_view<int8_t>();+ return s;+ }++ struct part_t {+ simd<uint32_t, 128> a, b, h;+ };++ static ESIMD_INLINE part_t load_part(const ctx & c, int sb, int part) {+ part_t p;+ p.a = xmx_load_t<8, SR>(c.ql, c.pl, c.h, sb * 32 + part * 16, c.row0);+ p.b = xmx_load_t<8, SR>(c.ql, c.pl, c.h, sb * 32 + part * 16 + 8, c.row0);+ p.h = xmx_load_t<8, SR>(c.qh, c.ph, c.h, sb * 16 + part * 8, c.row0);+ return p;+ }++ static ESIMD_INLINE simd<uint32_t, 128> unpack(const part_t & p, int g, int /*part*/) {+ simd<uint32_t, 128> qs = (g & 1) ? p.b : p.a;+ simd<uint32_t, 128> x;+ if (g == 0) {+ x = (qs & 0x0F0F0F0Fu) | ((p.h << 4) & 0x30303030u);+ } else if (g == 1) {+ x = (qs & 0x0F0F0F0Fu) | ((p.h << 2) & 0x30303030u);+ } else if (g == 2) {+ x = ((qs >> 4) & 0x0F0F0F0Fu) | (p.h & 0x30303030u);+ } else {+ x = ((qs >> 4) & 0x0F0F0F0Fu) | ((p.h >> 2) & 0x30303030u);+ }+ return xmx_bytes_sub(x, 0x20202020u);+ }+};++// Q3_K reorder layout: [qs: nb*64] [hmask: nb*32] [scales: nb*12] [d: nb*2]+// chunk s of 32: 2 bits from qs byte 32*(s/4)+l at shift 2*(s%4), high bit is hmask[l] bit s, value is q - 4+template <bool SR> struct xmx_traits_q3_k : xmx_traits_base<SR, 16, xmx_scale_kind::split> {+ struct ctx : xmx_ctx_base {+ const uint8_t * qs;+ const uint8_t * hm;+ const uint8_t * sc;+ const sycl::half * dh;+ unsigned pq, ph;+ };++ static ESIMD_INLINE ctx make(const void * vx, int nrows, int bpr, int row0) {+ const size_t nb = (size_t) nrows * bpr;+ ctx c;+ xmx_ctx_init(c, nrows, bpr, row0);+ c.qs = (const uint8_t *) vx;+ c.hm = c.qs + nb * (QK_K / 4);+ c.sc = c.hm + nb * (QK_K / 8);+ c.dh = (const sycl::half *) (c.sc + nb * 12);+ c.pq = xmx_pitch(bpr, QK_K / 4);+ c.ph = xmx_pitch(bpr, QK_K / 8);+ return c;+ }++ using scales_t = xmx_row_scales<16>;++ static ESIMD_INLINE scales_t load_scales(const ctx & c, int sb, const simd<uint32_t, 16> & rows) {+ using namespace sycl::ext::intel::esimd;+ scales_t s;+ simd<uint32_t, 16> bi = rows * c.bpr + sb;+ simd<uint32_t, 16> w0, w1, w2;+ xmx_load_sc12(c.sc, bi, w0, w1, w2);+ simd<float, 16> dv = convert<float>(gather<sycl::half, 16>(c.dh, bi * (uint32_t) sizeof(sycl::half)));+#pragma unroll+ for (int i = 0; i < 16; ++i) {+ const simd<uint32_t, 16> lo = (i % 8) < 4 ? w0 : w1;+ const int bl = i % 4;+ simd<uint32_t, 16> nib = (lo >> (8 * bl + (i < 8 ? 0 : 4))) & 0xFu;+ simd<uint32_t, 16> hi = ((w2 >> (8 * bl + 2 * (i / 4))) & 3u) << 4;+ simd<int, 16> code = nib | hi;+ s.v[i] = convert<float>(code - 32) * dv;+ }+ return s;+ }++ struct part_t {+ simd<uint32_t, 128> q, h;+ };++ static ESIMD_INLINE part_t load_part(const ctx & c, int sb, int part) {+ part_t p;+ p.q = xmx_load_t<8, SR>(c.qs, c.pq, c.h, sb * 16 + part * 8, c.row0);+ p.h = xmx_load_t<8, SR>(c.hm, c.ph, c.h, sb * 8, c.row0);+ return p;+ }++ static ESIMD_INLINE simd<uint32_t, 128> unpack(const part_t & p, int g, int part) {+ const int s = 4 * part + g;+ simd<uint32_t, 128> x = ((p.q >> (2 * g)) & 0x03030303u) | (((p.h >> s) & 0x01010101u) << 2);+ return xmx_bytes_sub(x, 0x04040404u);+ }+};++// Q4_K and Q5_K (QH) reorder layout: [qs: nb*128] ([qh: nb*32]) [scales: nb*12] [dm: nb*4]+// group j of 32: nibble of qs byte 32*(j/2)+l (low for even j), Q5_K adds 16 * bit j of qh[l]+// value is dall * sc_j * q - dmin * m_j, one scale and one min per group+template <bool QH, bool SR> struct xmx_traits_q45_k : xmx_traits_base<SR, QH ? 16 : 7, xmx_scale_kind::min> {+ struct ctx : xmx_ctx_base {+ const uint8_t * qs;+ const uint8_t * qh;+ const uint8_t * sc;+ const sycl::half * dm;+ unsigned pq, ph;+ };++ static ESIMD_INLINE ctx make(const void * vx, int nrows, int bpr, int row0) {+ const size_t nb = (size_t) nrows * bpr;+ ctx c;+ xmx_ctx_init(c, nrows, bpr, row0);+ c.qs = (const uint8_t *) vx;+ c.qh = c.qs + nb * (QK_K / 2);+ c.sc = QH ? c.qh + nb * (QK_K / 8) : c.qh;+ c.dm = (const sycl::half *) (c.sc + nb * 12);+ c.pq = xmx_pitch(bpr, QK_K / 2);+ c.ph = xmx_pitch(bpr, QK_K / 8);+ return c;+ }++ struct scales_t {+ xmx_row_scales<8> d; // dall * scale of each group+ xmx_row_scales<8> m; // dmin * min of each group+ };++ static ESIMD_INLINE scales_t load_scales(const ctx & c, int sb, const simd<uint32_t, 16> & rows) {+ using namespace sycl::ext::intel::esimd;+ scales_t s;+ simd<uint32_t, 16> bi = rows * c.bpr + sb;+ simd<uint32_t, 16> w0, w1, w2;+ xmx_load_sc12(c.sc, bi, w0, w1, w2);+ simd<float, 16> d, dmin;+ xmx_load_dm(c.dm, bi, d, dmin);+#pragma unroll+ for (int j = 0; j < 8; ++j) {+ const int jj = j % 4;+ simd<uint32_t, 16> sc, m;+ if (j < 4) {+ sc = (w0 >> (8 * jj)) & 63u;+ m = (w1 >> (8 * jj)) & 63u;+ } else {+ sc = ((w2 >> (8 * jj)) & 0xFu) | (((w0 >> (8 * jj + 6)) & 3u) << 4);+ m = ((w2 >> (8 * jj + 4)) & 0xFu) | (((w1 >> (8 * jj + 6)) & 3u) << 4);+ }+ s.d.v[j] = convert<float>(simd<int, 16>(sc)) * d;+ s.m.v[j] = convert<float>(simd<int, 16>(m)) * dmin;+ }+ return s;+ }++ struct part_t {+ simd<uint32_t, 128> a, b, h;+ };++ static ESIMD_INLINE part_t load_part(const ctx & c, int sb, int part) {+ part_t p;+ p.a = xmx_load_t<8, SR>(c.qs, c.pq, c.h, sb * 32 + part * 16, c.row0);+ p.b = xmx_load_t<8, SR>(c.qs, c.pq, c.h, sb * 32 + part * 16 + 8, c.row0);+ if constexpr (QH) {+ p.h = xmx_load_t<8, SR>(c.qh, c.ph, c.h, sb * 8, c.row0);+ }+ return p;+ }++ static ESIMD_INLINE simd<uint32_t, 128> unpack(const part_t & p, int g, int part) {+ simd<uint32_t, 128> qs = (g < 2) ? p.a : p.b;+ simd<uint32_t, 128> x = (qs >> (4 * (g & 1))) & 0x0F0F0F0Fu;+ if constexpr (QH) {+ x |= ((p.h >> (4 * part + g)) & 0x01010101u) << 4;+ }+ return x;+ }+};++// Q2_K reorder layout: [qs: nb*64] [scales: nb*16] [dm: nb*4]+// chunk s of 32: 2 bits from qs byte 32*(s/4)+l at shift 2*(s%4), each 16 wide half has a 4 bit scale and min+// value is d * (code & 15) * q - dmin * (code >> 4)+template <bool SR> struct xmx_traits_q2_k : xmx_traits_base<SR, 16, xmx_scale_kind::min16> {+ struct ctx : xmx_ctx_base {+ const uint8_t * qs;+ const uint8_t * sc;+ const sycl::half * dm;+ unsigned pq, ps;+ };++ static ESIMD_INLINE ctx make(const void * vx, int nrows, int bpr, int row0) {+ const size_t nb = (size_t) nrows * bpr;+ ctx c;+ xmx_ctx_init(c, nrows, bpr, row0);+ c.qs = (const uint8_t *) vx;+ c.sc = c.qs + nb * (QK_K / 4);+ c.dm = (const sycl::half *) (c.sc + nb * (QK_K / 16));+ c.pq = xmx_pitch(bpr, QK_K / 4);+ c.ps = xmx_pitch(bpr, QK_K / 16);+ return c;+ }++ struct scales_t {+ simd<uint8_t, 256> code; // 4 bit scale (low) and min (high) of each 16 wide half+ simd<float, 16> d, dmin;++ ESIMD_INLINE simd<uint8_t, 16> byte(int gr) { return code.template select<16, 4>(64 * (gr / 4) + (gr % 4)); }+ };++ static ESIMD_INLINE scales_t load_scales(const ctx & c, int sb, const simd<uint32_t, 16> & rows) {+ using namespace sycl::ext::intel::esimd;+ scales_t s;+ simd<uint32_t, 16> bi = rows * c.bpr + sb;+ simd<uint32_t, 64> raw = xmx_load_t<4, SR>(c.sc, c.ps, c.h, sb * 4, c.row0);+ s.code = raw.template bit_cast_view<uint8_t>();+ xmx_load_dm(c.dm, bi, s.d, s.dmin);+ return s;+ }++ struct part_t {+ simd<uint32_t, 128> q;+ };++ static ESIMD_INLINE part_t load_part(const ctx & c, int sb, int part) {+ part_t p;+ p.q = xmx_load_t<8, SR>(c.qs, c.pq, c.h, sb * 16 + part * 8, c.row0);+ return p;+ }++ static ESIMD_INLINE simd<uint32_t, 128> unpack(const part_t & p, int g, int /*part*/) {+ return (p.q >> (2 * g)) & 0x03030303u;+ }+};++// Q8_0 reorder layout: [qs: nrows*K bytes] [d: one half per 32 weights]+// the int8 weights feed DPAS as they are, a group is one block with one scale+template <bool SR> struct xmx_traits_q8_0 : xmx_traits_base<SR, 16, xmx_scale_kind::plain> {+ struct ctx : xmx_ctx_base {+ const uint8_t * qs;+ const sycl::half * dh;+ unsigned pq;+ };++ static ESIMD_INLINE ctx make(const void * vx, int nrows, int bpr, int row0) {+ ctx c;+ xmx_ctx_init(c, nrows, bpr, row0);+ c.qs = (const uint8_t *) vx;+ c.dh = (const sycl::half *) (c.qs + (size_t) nrows * bpr * QK_K);+ c.pq = xmx_pitch(bpr, QK_K);+ return c;+ }++ struct scales_t {+ xmx_row_scales<8> d; // scale of each of the 8 blocks of a super-block+ };++ static ESIMD_INLINE scales_t load_scales(const ctx & c, int sb, const simd<uint32_t, 16> & rows) {+ using namespace sycl::ext::intel::esimd;+ scales_t s;+ simd<uint32_t, 16> blk = (rows * c.bpr + sb) * (QK_K / QK8_0);+#pragma unroll+ for (int j = 0; j < 8; ++j) {+ s.d.v[j] = convert<float>(gather<sycl::half, 16>(c.dh, (blk + j) * (uint32_t) sizeof(sycl::half)));+ }+ return s;+ }++ struct part_t {+ simd<uint32_t, 128> q[4];+ };++ static ESIMD_INLINE part_t load_part(const ctx & c, int sb, int part) {+ part_t p;+#pragma unroll+ for (int g = 0; g < 4; ++g) {+ p.q[g] = xmx_load_t<8, SR>(c.qs, c.pq, c.h, sb * 64 + 8 * (4 * part + g), c.row0);+ }+ return p;+ }++ static ESIMD_INLINE simd<uint32_t, 128> unpack(const part_t & p, int g, int /*part*/) { return p.q[g]; }+};++template <bool SR> using xmx_traits_q4_k = xmx_traits_q45_k<false, SR>;+template <bool SR> using xmx_traits_q5_k = xmx_traits_q45_k<true, SR>;++// registers per thread, the large file halves the resident threads but avoids spills in wide tiles, the fused+// kernel keeps two sets of weights and scales and always takes it+template <typename T, int NC, bool FUSED> constexpr int xmx_grf() {+ return FUSED || NC >= T::nc_wide ? 256 : 128;+}++// One DPAS of a block of RC columns: the int32 products of the weights B and the activations A, as float and scaled+// per weight row. Every quant type sums one or two of these per group of 32.+template <int RC, int NT>+ESIMD_INLINE simd<float, RC * NT> xmx_term(const simd<int8_t, 512> & B,+ const simd<int8_t, RC * 32> & A,+ const simd<float, NT> & s) {+ using namespace sycl::ext::intel::esimd;+ namespace xmx = sycl::ext::intel::esimd::xmx;++ simd<int, RC * NT> C = xmx::dpas<8, RC, int>(B, A);+ return convert<float>(C) * s.template replicate<RC>();+}++// one group of 32 of one weight set, accumulated into Cf for all the columns of the tile+template <typename T, int NC, typename A2T, typename D8T>+ESIMD_INLINE void xmx_group(simd<float, NC * XMX_ROWS> & Cf,+ const typename T::part_t & pd,+ typename T::scales_t & sc,+ A2T & A2,+ D8T & d8f,+ simd<int8_t, 512> & Bl,+ simd<int8_t, 512> & Bh,+ const simd<int8_t, 512> & Ol,+ const int g,+ const int part) {+ using namespace sycl::ext::intel::esimd;++ constexpr int NT = XMX_ROWS;+ constexpr int RC = NC < 8 ? NC : 8;+ constexpr int NB = NC / RC;+ constexpr bool split = !T::has_min && !T::plain && !T::min16; // two scales per group of 32+ constexpr bool pair = (RC * 32) % 64 == 0; // a block of RC rows fills whole registers++ const int gi = 4 * part + g;++ simd<uint32_t, 128> x = T::unpack(pd, g, part);+ simd<int8_t, 512> B = x.template bit_cast_view<int8_t>();+ if constexpr (split) {+ Bl.template select<256, 1>(0) = B.template select<256, 1>(0);+ Bh.template select<256, 1>(256) = B.template select<256, 1>(256);+ }++ simd<float, NT> rd;+ simd<float, NT> rm;+ simd<float, NT> rs0;+ simd<float, NT> rs1;++ // Q2_K: scale and min of each 16 wide half are 4 bit codes, the weight is d * scl * q - dmin * mc. scl * q+ // is at most 45, so a dword multiply keeps the bytes apart and the scaled weights go to DPAS as int8. The+ // min term sums a * mc over K, the codes mc take the place of the weights in a second DPAS.+ simd<int8_t, 512> Bs;+ simd<int8_t, 512> Bm;+ if constexpr (T::min16) {+ const simd<uint8_t, NT> c0 = sc.byte(2 * gi);+ const simd<uint8_t, NT> c1 = sc.byte(2 * gi + 1);+ simd<uint32_t, NT> sl0 = convert<uint32_t>(simd<uint8_t, NT>(c0 & (uint8_t) 15));+ simd<uint32_t, NT> sl1 = convert<uint32_t>(simd<uint8_t, NT>(c1 & (uint8_t) 15));+ simd<uint32_t, NT> mc0 = convert<uint32_t>(simd<uint8_t, NT>(c0 >> (uint8_t) 4)) * 0x01010101u;+ simd<uint32_t, NT> mc1 = convert<uint32_t>(simd<uint8_t, NT>(c1 >> (uint8_t) 4)) * 0x01010101u;+ simd<uint32_t, 128> ws;+ simd<uint32_t, 128> wm;+ ws.template select<64, 1>(0) = x.template select<64, 1>(0) * sl0.template replicate<4>();+ ws.template select<64, 1>(64) = x.template select<64, 1>(64) * sl1.template replicate<4>();+ wm.template select<64, 1>(0) = mc0.template replicate<4>();+ wm.template select<64, 1>(64) = mc1.template replicate<4>();+ Bs = ws.template bit_cast_view<int8_t>();+ Bm = wm.template bit_cast_view<int8_t>();+ }+ if constexpr (T::plain) {+ rd = sc.d.get(gi);+ } else if constexpr (T::has_min) {+ rd = sc.d.get(gi);+ rm = sc.m.get(gi);+ } else if constexpr (T::min16) {+ // scales are applied through Bs and Bm+ } else {+ rs0 = sc.get(2 * gi);+ rs1 = sc.get(2 * gi + 1);+ }++#pragma unroll+ for (int b = 0; b < NB; ++b) {+ simd<int8_t, RC * 32> A =+ A2[b].template select<RC * 8, 1>(pair ? (g % 2) * RC * 8 : 0).template bit_cast_view<int8_t>();+ simd<float, RC * NT> d8r =+ d8f[b].template select<RC, RC == 1 ? 1 : 8>(gi).template replicate_vs_w_hs<RC, 1, NT, 0>(0);+ auto Cb = Cf.template select<RC * NT, 1>(b * RC * NT);++ simd<float, RC * NT> z;+ if constexpr (T::plain) {+ z = xmx_term<RC, NT>(B, A, rd);+ } else if constexpr (T::has_min) {+ // the second DPAS (all ones against the activations) is the exact integer sum of the activations+ z = xmx_term<RC, NT>(B, A, rd) - xmx_term<RC, NT>(Ol, A, rm);+ } else if constexpr (T::min16) {+ z = xmx_term<RC, NT>(Bs, A, sc.d) - xmx_term<RC, NT>(Bm, A, sc.dmin);+ } else {+ z = xmx_term<RC, NT>(Bl, A, rs0) + xmx_term<RC, NT>(Bh, A, rs1);+ }+ Cb += z * d8r;+ }+}++// activation of the GLU applied to the gate, uniform across the launch+template <int N> ESIMD_INLINE simd<float, N> xmx_glu_act(const simd<float, N> & gate, const int glu_op) {+ using namespace sycl::ext::intel::esimd;+ if (glu_op == GGML_GLU_OP_SWIGLU) {+ return gate / (1.0f + exp(-gate));+ }+ // GEGLU, the tanh is written with exp+ simd<float, N> a = 0.79788456f * gate * (1.0f + 0.044715f * gate * gate);+ simd<float, N> t = 1.0f - 2.0f / (exp(2.0f * a) + 1.0f);+ return 0.5f * gate * (1.0f + t);+}++// activations come in through 2D block loads, one for the (d, s) pairs of a super-block (s is unused) and one per two+// groups for the quants, the DPAS A operand needs no repacking. Groups with two scales (16 wide) mask the+// weights instead, zeros in B drop the other half of K.+// KS is the most threads that split K for one tile, the launch may use fewer.+// FUSED computes up * act(gate) from the weights vx (up) and vg (gate), reading the activations once.+template <typename T, int NC, int KS, bool FUSED>+ESIMD_INLINE void xmx_mul_mat(const void * vx,+ const void * vg,+ const char * vy,+ float * dst,+ const int ncols,+ const int nrows,+ const int ncv,+ const int stride_y_bytes,+ const int ldd,+ const int glu_op,+ const int tile,+ const int lid,+ const int ks) {+ using namespace sycl::ext::intel::esimd;++ constexpr int NT = XMX_ROWS;+ constexpr int RC = NC < 8 ? NC : 8;+ constexpr int NB = NC / RC;+ constexpr int N_ACC = FUSED ? 2 : 1;+ constexpr bool pair = (RC * 32) % 64 == 0; // a block of RC rows fills whole registers++ if constexpr (KS > 1) {+ slm_init<N_ACC * KS * NC * NT * sizeof(float)>();+ }++ const int bpr = ncols / QK_K;+ const int row0 = tile * NT;+ const auto c = T::make(vx, nrows, bpr, row0);+ const auto cg = T::make(FUSED ? vg : vx, nrows, bpr, row0);++ simd<uint32_t, NT> rows;+#pragma unroll+ for (int n = 0; n < NT; ++n) {+ rows[n] = row0 + n < nrows ? row0 + n : nrows - 1;+ }++ const uint32_t * yq = (const uint32_t *) vy;+ const uint32_t * yd = (const uint32_t *) (vy + ncols);+ const unsigned yqw = ncols - 1;+ const unsigned ydw = ncols / QK8_1 * sizeof(sycl::half2) - 1;+ const unsigned yh = ncv - 1;+ const unsigned yp = stride_y_bytes - 1;++ simd<float, NC * NT> Cf = 0.0f;+ simd<float, NC * NT> Cfg = 0.0f; // gate, only used when fused+ simd<int8_t, 512> Bl = int8_t(0);+ simd<int8_t, 512> Bh = int8_t(0);+ simd<int8_t, 512> Ol = int8_t(0);+ if constexpr (T::has_min) {+ Ol = int8_t(1);+ }++ for (int sb = lid; sb < bpr; sb += ks) {+ auto sc = T::load_scales(c, sb, rows);+ typename T::scales_t scg;+ if constexpr (FUSED) {+ scg = T::load_scales(cg, sb, rows);+ }++ // d of the 8 groups of the super-block, [column * 8 + group]+ simd<float, 8 * RC> d8f[NB];+#pragma unroll+ for (int b = 0; b < NB; ++b) {+ simd<uint32_t, 8 * RC> ds = load_2d<uint32_t, 8, RC, 1, false, false>(yd, ydw, yh, yp, sb * 8, b * RC);+ simd<sycl::half, 16 * RC> dh = ds.template bit_cast_view<sycl::half>();+ d8f[b] = convert<float>(simd<sycl::half, 8 * RC>(dh.template select<8 * RC, 2>(0)));+ }++#pragma unroll+ for (int part = 0; part < 2; ++part) {+ const auto pd = T::load_part(c, sb, part);+ typename T::part_t pdg;+ if constexpr (FUSED) {+ pdg = T::load_part(cg, sb, part);+ }++ simd<uint32_t, 16 * RC> A2[NB]; // quants of two groups++#pragma unroll+ for (int g = 0; g < 4; ++g) {+ const int gi = 4 * part + g;+ if constexpr (pair) {+ if (g % 2 == 0) {+#pragma unroll+ for (int b = 0; b < NB; ++b) {+ A2[b] =+ load_2d<uint32_t, 8, RC, 2, false, false>(yq, yqw, yh, yp, sb * 64 + gi * 8, b * RC);+ }+ }+ } else {+#pragma unroll+ for (int b = 0; b < NB; ++b) {+ A2[b].template select<RC * 8, 1>(0) =+ load_2d<uint32_t, 8, RC, 1, false, false>(yq, yqw, yh, yp, sb * 64 + gi * 8, b * RC);+ }+ }++ xmx_group<T, NC>(Cf, pd, sc, A2, d8f, Bl, Bh, Ol, g, part);+ if constexpr (FUSED) {+ xmx_group<T, NC>(Cfg, pdg, scg, A2, d8f, Bl, Bh, Ol, g, part);+ }+ }+ }+ }++ if constexpr (KS > 1) {+ slm_block_store<float, NC * NT>(lid * NC * NT * sizeof(float), Cf);+ if constexpr (FUSED) {+ slm_block_store<float, NC * NT>((KS + lid) * NC * NT * sizeof(float), Cfg);+ }+ barrier();+ if (lid != 0) {+ return;+ }+ Cf = 0.0f;+ Cfg = 0.0f;+#pragma unroll 4+ for (int j = 0; j < ks; ++j) {+ Cf += slm_block_load<float, NC * NT>(j * NC * NT * sizeof(float));+ if constexpr (FUSED) {+ Cfg += slm_block_load<float, NC * NT>((KS + j) * NC * NT * sizeof(float));+ }+ }+ }++ if constexpr (FUSED) {+ Cf *= xmx_glu_act<NC * NT>(Cfg, glu_op);+ }++ if (row0 + NT <= nrows && ldd % 4 == 0 && (uintptr_t) dst % 16 == 0) {+#pragma unroll+ for (int m = 0; m < NC; ++m) {+ if (NC <= 8 || m < ncv) {+ block_store<float, NT>(dst + (size_t) m * ldd + row0, Cf.template select<NT, 1>(m * NT));+ }+ }+ } else {+ // tail tile or unaligned output+ simd<uint32_t, NT> lane(0, 1);+ simd_mask<NT> in_rows = (lane + row0) < (uint32_t) nrows;+#pragma unroll+ for (int m = 0; m < NC; ++m) {+ if (NC <= 8 || m < ncv) {+ scatter<float, NT>(dst + (size_t) m * ldd + row0, lane * (uint32_t) sizeof(float),+ Cf.template select<NT, 1>(m * NT), in_rows);+ }+ }+ }+}++// KS_MAX bounds the threads that split K for one tile, the launch uses as few as keep every thread to the same+// number of super-blocks as the busiest one+template <typename T, int NC, int KS_MAX, bool FUSED>+static void xmx_mul_mat_launch_ks(const void * vx,+ const void * vg,+ const char * vy,+ float * dst,+ const int ncols,+ const int nrows,+ const int ncv,+ const int stride_y_bytes,+ const int ldd,+ const int glu_op,+ dpct::queue_ptr stream) {+ GGML_ASSERT(ncols % QK_K == 0);+ const size_t n_tiles = (nrows + XMX_ROWS - 1) / XMX_ROWS;+ const int bpr = ncols / QK_K;+ const int rounds = (bpr + KS_MAX - 1) / KS_MAX;+ const int ks = (bpr + rounds - 1) / rounds;+ stream->submit([&](sycl::handler & cgh) {+ cgh.parallel_for(sycl::nd_range<1>(sycl::range<1>(n_tiles * ks), sycl::range<1>(ks)),+ sycl::ext::oneapi::experimental::properties{+ sycl::ext::intel::experimental::grf_size<xmx_grf<T, NC, FUSED>()> },+ [=](sycl::nd_item<1> it) [[intel::sycl_explicit_simd]] {+ xmx_mul_mat<T, NC, KS_MAX, FUSED>(vx, vg, vy, dst, ncols, nrows, ncv, stride_y_bytes, ldd,+ glu_op, (int) it.get_group(0), (int) it.get_local_id(0),+ ks);+ });+ });+}++template <typename T, int NC, bool FUSED>+static void xmx_mul_mat_launch(const void * vx,+ const void * vg,+ const char * vy,+ float * dst,+ const int ncols,+ const int nrows,+ const int ncv,+ const int stride_y_bytes,+ const int ldd,+ const int glu_op,+ dpct::queue_ptr stream) {+ // the fused kernel keeps two sets of accumulators in shared memory, half the threads for the same size+ constexpr int ks_max = xmx_n_split<NC>() / (FUSED ? 2 : 1);+ xmx_mul_mat_launch_ks<T, NC, ks_max, FUSED>(vx, vg, vy, dst, ncols, nrows, ncv, stride_y_bytes, ldd, glu_op,+ stream);+}++// FUSED: vx and vg are the up and gate weights, dst gets up * act(gate) with glu_op a ggml_glu_op+template <typename T, bool FUSED = false>+static void xmx_mul_mat_ncols(const void * vx,+ const void * vg,+ const char * vy,+ float * dst,+ const int ncols,+ const int nrows,+ const int ncols_dst,+ const int stride_y_bytes,+ const int ldd,+ const int glu_op,+ dpct::queue_ptr stream) {+ constexpr int chunk_max = FUSED ? XMX_FUSED_CHUNK : XMX_CHUNK;+ for (int c0 = 0; c0 < ncols_dst; c0 += chunk_max) {+ const int n = std::min(chunk_max, ncols_dst - c0);+ const char * y = vy + (size_t) c0 * stride_y_bytes;+ float * d = dst + (size_t) c0 * ldd;+ const int nc = n > 32 ? 64 : (n > 16 ? 32 : (n > 8 ? 16 : n));+ switch (nc) {+#define XMX_CASE(N) \+ case N: \+ xmx_mul_mat_launch<T, N, FUSED>(vx, vg, y, d, ncols, nrows, n, stride_y_bytes, ldd, glu_op, stream); \+ break+ XMX_CASE(1);+ XMX_CASE(2);+ XMX_CASE(3);+ XMX_CASE(4);+ XMX_CASE(5);+ XMX_CASE(6);+ XMX_CASE(7);+ XMX_CASE(8);+#undef XMX_CASE+ // short row variants are for tests, not built for the wide tiles+#define XMX_CASE_WIDE(N) \+ case N: \+ if constexpr (!T::short_rows && (!FUSED || N <= XMX_FUSED_CHUNK)) { \+ xmx_mul_mat_launch<T, N, FUSED>(vx, vg, y, d, ncols, nrows, n, stride_y_bytes, ldd, glu_op, stream); \+ } \+ break+ XMX_CASE_WIDE(16);+ XMX_CASE_WIDE(32);+ XMX_CASE_WIDE(64);+#undef XMX_CASE_WIDE+ default:+ GGML_ABORT("unsupported ncols_dst=%d for XMX MMV", n);+ }+ }+}+} // namespace ggml_sycl_esimd#endif // GGML_SYCL_ESIMD_HPPdiff --git a/ggml/src/ggml-sycl/fusion.cpp b/ggml/src/ggml-sycl/fusion.cppindex cb4274e796..9b553829be 100644--- a/ggml/src/ggml-sycl/fusion.cpp+++ b/ggml/src/ggml-sycl/fusion.cpp@@ -37,6 +37,13 @@ static bool ggml_sycl_should_fuse_mul_mat_glu(const ggml_tensor * gate, const ggconst bool reorder_pair = wu->type == GGML_TYPE_Q4_K && wg->type == GGML_TYPE_Q4_K;const bool plain_pair = (wu->type == GGML_TYPE_Q5_K || wu->type == GGML_TYPE_IQ4_XS) &&(wg->type == GGML_TYPE_Q5_K || wg->type == GGML_TYPE_IQ4_XS);+#ifdef GGML_SYCL_MMVQ_HAS_XMX+ // same-type pairs of the types of the XMX kernels, which also take a few more columns+ // (the XMX kernels walk whole QK_K super-blocks, so they need rows of a multiple of QK_K)+ const bool xmx_pair = wu->type == wg->type && ggml_sycl_xmx_supports_type(wu->type) && wu->ne[0] % QK_K == 0;+#else+ const bool xmx_pair = false;+#endif // GGML_SYCL_MMVQ_HAS_XMX// the ESIMD kernel covers same-type q2_K..q6_K and q8_0, single column only (as unfused DMMV)#ifdef GGML_SYCL_DMMV_HAS_ESIMDconst bool esimd_pair = g_ggml_sycl_enable_esimd && wu->type == wg->type && act->ne[1] == 1 &&@@ -47,7 +54,7 @@ static bool ggml_sycl_should_fuse_mul_mat_glu(const ggml_tensor * gate, const gg#endif// q8_0 rows are walked in 32-wide blocks, so they only need to be a multiple of QK8_0const int64_t row_align = esimd_pair && wu->type == GGML_TYPE_Q8_0 ? QK8_0 : QK_K;- if ((!reorder_pair && !plain_pair && !esimd_pair) || wu->ne[0] % row_align != 0) {+ if ((!reorder_pair && !plain_pair && !xmx_pair && !esimd_pair) || wu->ne[0] % row_align != 0) {return false;}@@ -67,7 +74,8 @@ static bool ggml_sycl_should_fuse_mul_mat_glu(const ggml_tensor * gate, const ggreturn false;}// mat-vec only: one column per decoded token, up to the batch the reorder kernels cover- if (act->ne[1] > MMVQ_MAX_BATCH_SIZE) {+ const int max_cols = xmx_pair ? std::max(MMVQ_MAX_BATCH_SIZE, g_ggml_sycl_xmx_glu_max_cols) : MMVQ_MAX_BATCH_SIZE;+ if (act->ne[1] > max_cols) {return false;}diff --git a/ggml/src/ggml-sycl/ggml-sycl.cpp b/ggml/src/ggml-sycl/ggml-sycl.cppindex bfe53c5d7f..e07a18bdc2 100644--- a/ggml/src/ggml-sycl/ggml-sycl.cpp+++ b/ggml/src/ggml-sycl/ggml-sycl.cpp@@ -103,6 +103,9 @@ int g_ggml_sycl_enable_vmm = 1;int g_ggml_sycl_enable_fusion = 1;int g_ggml_sycl_enable_esimd = 1;int g_ggml_sycl_mmvq_wide = 1;+int g_ggml_sycl_enable_xmx_esimd = 1;+int g_ggml_sycl_xmx_max_cols = GGML_SYCL_XMX_MAX_COLS_DEFAULT;+int g_ggml_sycl_xmx_glu_max_cols = GGML_SYCL_XMX_GLU_MAX_COLS;int g_ggml_sycl_prioritize_dmmv = 0;int g_ggml_sycl_xmx_gather_types = GGML_SYCL_XMX_GATHER_TYPES_DEFAULT;int g_ggml_sycl_xmx_gather_shapes = GGML_SYCL_XMX_GATHER_SHAPES_DEFAULT;@@ -424,6 +427,10 @@ static void ggml_check_sycl() try {g_ggml_sycl_enable_fusion = ggml_sycl_get_env("GGML_SYCL_ENABLE_FUSION", 1);g_ggml_sycl_enable_esimd = ggml_sycl_get_env("GGML_SYCL_ENABLE_ESIMD", 1);g_ggml_sycl_mmvq_wide = ggml_sycl_get_env("GGML_SYCL_MMVQ_WIDE", 1);+ g_ggml_sycl_enable_xmx_esimd = ggml_sycl_get_env("GGML_SYCL_ENABLE_XMX_ESIMD", 1);+ g_ggml_sycl_xmx_max_cols = std::max(0, ggml_sycl_get_env("GGML_SYCL_XMX_MAX_COLS", GGML_SYCL_XMX_MAX_COLS_DEFAULT));+ g_ggml_sycl_xmx_glu_max_cols =+ std::clamp(ggml_sycl_get_env("GGML_SYCL_XMX_GLU_MAX_COLS", GGML_SYCL_XMX_GLU_MAX_COLS), 0, GGML_SYCL_XMX_GLU_MAX_COLS);g_ggml_sycl_prioritize_dmmv = ggml_sycl_get_env("GGML_SYCL_PRIORITIZE_DMMV", 0);g_ggml_sycl_xmx_gather_types = ggml_sycl_get_env("GGML_SYCL_XMX_GATHER_TYPES", GGML_SYCL_XMX_GATHER_TYPES_DEFAULT);g_ggml_sycl_xmx_gather_shapes = ggml_sycl_get_env("GGML_SYCL_XMX_GATHER_SHAPES", GGML_SYCL_XMX_GATHER_SHAPES_DEFAULT);@@ -561,7 +568,13 @@ static void ggml_check_sycl() try {#elseGGML_LOG_INFO(" GGML_SYCL_ENABLE_ESIMD: %d disabled by compile flag\n", g_ggml_sycl_enable_esimd);#endif+GGML_LOG_INFO(" GGML_SYCL_MMVQ_WIDE: %d\n", g_ggml_sycl_mmvq_wide);++ GGML_LOG_INFO(" GGML_SYCL_ENABLE_XMX_ESIMD: %d\n", g_ggml_sycl_enable_xmx_esimd);+ GGML_LOG_INFO(" GGML_SYCL_XMX_MAX_COLS: %d\n", g_ggml_sycl_xmx_max_cols);+ GGML_LOG_INFO(" GGML_SYCL_XMX_GLU_MAX_COLS: %d\n", g_ggml_sycl_xmx_glu_max_cols);+GGML_LOG_INFO(" GGML_SYCL_PRIORITIZE_DMMV: %d\n", g_ggml_sycl_prioritize_dmmv);g_ggml_sycl_use_async_mem_op_requested = ggml_sycl_get_env("GGML_SYCL_USE_ASYNC_MEM_OP", 1);@@ -5165,9 +5178,24 @@ static bool can_use_dequantize_mul_mat_vec(const ggml_tensor * src0, const ggml_src0->ne[0] % dmmv_x_required == 0 && src1->ne[1] == 1;}-static bool can_use_mul_mat_vec_q(const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) {+// reordered weights of the types in ggml_sycl_xmx_supports_type() take the XMX kernel for wider batches+static bool can_use_xmx_batch(int device, const ggml_tensor * src0, const ggml_tensor * src1) {+#ifdef GGML_SYCL_MMVQ_HAS_XMX+ const auto * extra = static_cast<const ggml_tensor_extra_gpu *>(src0->extra);+ return ggml_sycl_xmx_enabled(device) && ggml_sycl_xmx_supports_type(src0->type) && extra &&+ extra->optimized_feature.reorder && src1->ne[1] <= g_ggml_sycl_xmx_max_cols && src1->ne[2] == 1 &&+ src1->ne[3] == 1;+#else+ GGML_UNUSED(device);+ GGML_UNUSED(src0);+ GGML_UNUSED(src1);+ return false;+#endif // GGML_SYCL_MMVQ_HAS_XMX+}++static bool can_use_mul_mat_vec_q(int device, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) {return ggml_is_quantized(src0->type) && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32 &&- src1->ne[1] <= MMVQ_MAX_BATCH_SIZE;+ (src1->ne[1] <= MMVQ_MAX_BATCH_SIZE || can_use_xmx_batch(device, src0, src1));}static void ggml_sycl_mul_mat(ggml_backend_sycl_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) {@@ -5207,12 +5235,15 @@ static void ggml_sycl_mul_mat(ggml_backend_sycl_context & ctx, const ggml_tensor// check data types and tensor shapes for custom matrix multiplication kernels:bool use_dequantize_mul_mat_vec = can_use_dequantize_mul_mat_vec(src0, src1, dst);+ if (src1->ne[1] == 1 && ggml_sycl_xmx_min_cols(src0->type) == 1 && can_use_xmx_batch(ctx.device, src0, src1)) {+ use_dequantize_mul_mat_vec = false;+ }#ifdef GGML_SYCL_F16// dmmv may convert src1 to f16 in this build [TAG_GGML_PREC]use_dequantize_mul_mat_vec = use_dequantize_mul_mat_vec && ggml_sycl_src1_f16_ok(dst);#endif- bool use_mul_mat_vec_q = can_use_mul_mat_vec_q(src0, src1, dst);+ bool use_mul_mat_vec_q = can_use_mul_mat_vec_q(ctx.device, src0, src1, dst);bool use_mul_mat_q = ggml_sycl_supports_mmq(src0->type)&& src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32;@@ -5321,6 +5352,35 @@ static bool ggml_sycl_mul_mat_glu_mmvq_plain(ggml_backend_sycl_context & ctx, ggGGML_UNUSED(gate);}+// Gate and up weights of one type on the XMX engines: one launch reads the activations once and applies the GLU.+// Both weights must already be in the reorder layout. Returns false if the kernel declined.+static bool ggml_sycl_mul_mat_glu_xmx(ggml_backend_sycl_context & ctx,+ ggml_tensor * glu,+ ggml_tensor * up,+ const ggml_tensor * wu,+ const ggml_tensor * wg,+ const ggml_tensor * act) {+ const int64_t ne00 = wu->ne[0];+ const int64_t ne11 = act->ne[1];++ const queue_ptr stream = ctx.stream();+ const int src1_padded_cols = GGML_PAD((int) ne00, MATRIX_ROW_PADDING);+ const int stride_y_bytes = src1_padded_cols * (int) sizeof(block_q8_1) / QK8_1;++ // log the up mat-mul: glu's own srcs are the two intermediates the fusion never materialises+ scope_op_debug_print scope_dbg_print(__func__, up, /*num_src=*/2, " : fused with gate + GLU (XMX)");++ ggml_sycl_pool_alloc<char> src1_q8_alloc(ctx.pool(), (size_t) ne11 * stride_y_bytes);+ char * src1_ddq = src1_q8_alloc.get();++ quantize_row_q8_1_sycl<quantize_and_reorder_q8_1_soa>((const float *) act->data, src1_ddq, (int) ne00, (int) ne11,+ src1_padded_cols, stream);++ return ggml_sycl_mul_mat_vec_q_glu_xmx(ctx.device, wu->type, ggml_get_glu_op(glu), wu->data, wg->data, src1_ddq,+ (float *) glu->data, (int) ne00, (int) wu->ne[1], (int) ne11, stride_y_bytes,+ /*stride_col_dst=*/(int) glu->ne[0], stream);+}+#ifdef GGML_SYCL_DMMV_HAS_ESIMD// {mul_mat(gate), mul_mat(up), GLU} on the reorder layout via the ESIMD DMMV kernel, for// same-type K-quant or Q8_0 pairs and a single activation column. It reads the F32 activation@@ -5389,11 +5449,33 @@ static bool ggml_sycl_mul_mat_glu_fused(ggml_backend_sycl_context & ctx, ggml_cgreturn false;}+ // gate and up of one type the XMX kernel handles, it needs the reorder layout like the unfused mmvq path+ if (wg->type == wu->type && gate->src[1] == act && act->ne[1] >= ggml_sycl_xmx_min_cols(wu->type) &&+ act->ne[1] <= g_ggml_sycl_xmx_glu_max_cols) {+ opt_for_reorder(&ctx, wu, act, up, mul_mat_algo::MMVQ);+ opt_for_reorder(&ctx, wg, act, gate, mul_mat_algo::MMVQ);+ if (can_use_xmx_batch(ctx.device, wu, act) && can_use_xmx_batch(ctx.device, wg, act) &&+ ggml_sycl_mul_mat_glu_xmx(ctx, glu, up, wu, wg, act)) {+ return true;+ }+ }++ // the kernels below take up to MMVQ_MAX_BATCH_SIZE columns, a wider batch got here for the XMX kernel+ if (act->ne[1] > MMVQ_MAX_BATCH_SIZE) {+ return false;+ }+// quant pairs the reorder kernel does not serve (mixed gate/up types, q5_K off BMG) take the// standard-layout fused path instead; same-type q4_K / q5_K keep the reorder path belowconst bool reorder_pair = wg->type == wu->type &&(wu->type == GGML_TYPE_Q4_K || (wu->type == GGML_TYPE_Q5_K && ggml_sycl_q5_k_mmvq_reuse(ctx.device)));if (!reorder_pair) {+ const auto is_plain_type = [](ggml_type type) {+ return type == GGML_TYPE_Q5_K || type == GGML_TYPE_IQ4_XS;+ };+ if (!is_plain_type(wu->type) || !is_plain_type(wg->type)) {+ return false;+ }return ggml_sycl_mul_mat_glu_mmvq_plain(ctx, glu, gate, up, wu, wg, act);}diff --git a/ggml/src/ggml-sycl/mmvq.cpp b/ggml/src/ggml-sycl/mmvq.cppindex e9057777cb..ccb0363bd1 100644--- a/ggml/src/ggml-sycl/mmvq.cpp+++ b/ggml/src/ggml-sycl/mmvq.cpp@@ -6,6 +6,10 @@#include "quants.hpp"#include "vecdotq.hpp"+#ifdef GGML_SYCL_MMVQ_HAS_XMX+# include "esimd.hpp"+#endif // GGML_SYCL_MMVQ_HAS_XMX+// vec_dot_q_sycl_t adapters for the IQ vec_dots that take their codebook tables as extra// arguments: bind the constant tables here (as vec_dot_iq2_s_q8_1 / vec_dot_iq1_m_q8_1 already do// internally) so they can be used as template arguments of mul_mat_vec_q_moe.@@ -2557,6 +2561,158 @@ static void mul_mat_vec_iq4_xs_q8_1_sycl_switch_ncols(}}+#ifdef GGML_SYCL_MMVQ_HAS_XMX+// vgate is the gate weight of the fused gate and up kernel, nullptr for a plain mat-vec+template <template <bool> class Traits>+static void ggml_sycl_mul_mat_vec_q_xmx_run(bool short_rows,+ const void * vx,+ const void * vgate,+ const char * vy,+ float * dst,+ int ne00,+ int64_t nrows,+ int64_t ncols_dst,+ int stride_y_bytes,+ int ldd,+ int glu_op,+ dpct::queue_ptr stream) {+ namespace esimd = ggml_sycl_esimd;+ if (vgate) {+ if (short_rows) {+ esimd::xmx_mul_mat_ncols<Traits<true>, true>(vx, vgate, vy, dst, ne00, nrows, ncols_dst, stride_y_bytes,+ ldd, glu_op, stream);+ } else {+ esimd::xmx_mul_mat_ncols<Traits<false>, true>(vx, vgate, vy, dst, ne00, nrows, ncols_dst, stride_y_bytes,+ ldd, glu_op, stream);+ }+ } else if (short_rows) {+ esimd::xmx_mul_mat_ncols<Traits<true>>(vx, nullptr, vy, dst, ne00, nrows, ncols_dst, stride_y_bytes, ldd, 0,+ stream);+ } else {+ esimd::xmx_mul_mat_ncols<Traits<false>>(vx, nullptr, vy, dst, ne00, nrows, ncols_dst, stride_y_bytes, ldd, 0,+ stream);+ }+}++static bool ggml_sycl_mul_mat_vec_q_xmx_type(ggml_type type,+ bool short_rows,+ const void * vx,+ const void * vgate,+ const char * vy,+ float * dst,+ int ne00,+ int64_t nrows,+ int64_t ncols_dst,+ int stride_y_bytes,+ int ldd,+ int glu_op,+ dpct::queue_ptr stream) {+# define XMX_RUN(TRAITS) \+ ggml_sycl_mul_mat_vec_q_xmx_run<ggml_sycl_esimd::TRAITS>(short_rows, vx, vgate, vy, dst, ne00, nrows, \+ ncols_dst, stride_y_bytes, ldd, glu_op, stream); \+ return true+ switch (type) {+ case GGML_TYPE_Q2_K:+ XMX_RUN(xmx_traits_q2_k);+ case GGML_TYPE_Q3_K:+ XMX_RUN(xmx_traits_q3_k);+ case GGML_TYPE_Q4_K:+ XMX_RUN(xmx_traits_q4_k);+ case GGML_TYPE_Q5_K:+ XMX_RUN(xmx_traits_q5_k);+ case GGML_TYPE_Q6_K:+ XMX_RUN(xmx_traits_q6_k);+ case GGML_TYPE_Q8_0:+ XMX_RUN(xmx_traits_q8_0);+ default:+ return false;+ }+# undef XMX_RUN+}++// multi-column XMX path for reordered weights, returns false when it does not apply+static bool ggml_sycl_mul_mat_vec_q_xmx(int device,+ const ggml_tensor * src0,+ const ggml_tensor * dst,+ const char * src0_dd_i,+ const char * src1_ddq_i,+ float * dst_dd_i,+ int64_t row_diff,+ int64_t src1_ncols,+ int64_t src1_padded_col_size,+ dpct::queue_ptr stream) {+ const auto * extra = static_cast<const ggml_tensor_extra_gpu *>(src0->extra);+ if (!ggml_sycl_xmx_enabled(device) || !ggml_sycl_xmx_supports_type(src0->type) || !extra ||+ !extra->optimized_feature.reorder) {+ return false;+ }+ if (src1_ncols < ggml_sycl_xmx_min_cols(src0->type) || src1_ncols > g_ggml_sycl_xmx_max_cols) {+ return false;+ }++ const int ne00 = src0->ne[0];+ const int ldd = dst->ne[0];+ if (!ggml_sycl_esimd::xmx_supported(src0_dd_i, ne00)) {+ return false;+ }++ // rows under 64 bytes use the variant without 2D block loads, it is not built for the widest tile+ const bool short_rows = ne00 < 4 * QK_K;+ if (short_rows && src1_ncols > 8) {+ return false;+ }++ const int stride_y_bytes = src1_padded_col_size * sizeof(block_q8_1) / QK8_1;+ // the 2D block loads of the activations need a 64 byte aligned base and a pitch of 16 bytes+ if ((uintptr_t) src1_ddq_i % 64 != 0 || stride_y_bytes % 16 != 0) {+ return false;+ }+ return ggml_sycl_mul_mat_vec_q_xmx_type(src0->type, short_rows, src0_dd_i, nullptr, src1_ddq_i, dst_dd_i, ne00,+ row_diff, src1_ncols, stride_y_bytes, ldd, 0, stream);+}+#endif // GGML_SYCL_MMVQ_HAS_XMX++bool ggml_sycl_mul_mat_vec_q_glu_xmx(int device,+ enum ggml_type src0_type,+ enum ggml_glu_op glu_op,+ const void * vx,+ const void * vgate,+ const void * vy,+ float * dst,+ int ncols,+ int nrows,+ int ncols_dst,+ int stride_col_y_bytes,+ int stride_col_dst,+ dpct::queue_ptr stream) {+#ifdef GGML_SYCL_MMVQ_HAS_XMX+ if (!ggml_sycl_xmx_enabled(device) || !ggml_sycl_xmx_supports_type(src0_type)) {+ return false;+ }+ if (glu_op != GGML_GLU_OP_SWIGLU && glu_op != GGML_GLU_OP_GEGLU) {+ return false;+ }+ if (ncols_dst < ggml_sycl_xmx_min_cols(src0_type) || ncols_dst > g_ggml_sycl_xmx_glu_max_cols) {+ return false;+ }+ if (!ggml_sycl_esimd::xmx_supported(vx, ncols) || !ggml_sycl_esimd::xmx_supported(vgate, ncols) ||+ (uintptr_t) vy % 64 != 0 || stride_col_y_bytes % 16 != 0) {+ return false;+ }+ // rows under 64 bytes use the variant without 2D block loads, it is not built for the wide tiles+ const bool short_rows = ncols < 4 * QK_K;+ if (short_rows && ncols_dst > 8) {+ return false;+ }+ return ggml_sycl_mul_mat_vec_q_xmx_type(src0_type, short_rows, vx, vgate, (const char *) vy, dst, ncols, nrows,+ ncols_dst, stride_col_y_bytes, stride_col_dst, (int) glu_op, stream);+#else+ GGML_UNUSED_VARS(device, src0_type, glu_op, vx, vgate, vy, dst, ncols, nrows, ncols_dst, stride_col_y_bytes,+ stride_col_dst, stream);+ return false;+#endif // GGML_SYCL_MMVQ_HAS_XMX+}+void ggml_sycl_op_mul_mat_vec_q(ggml_backend_sycl_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1,ggml_tensor * dst, const char * src0_dd_i, const float * src1_ddf_i,const char * src1_ddq_i, float * dst_dd_i, const int64_t row_low,@@ -2568,6 +2724,13 @@ void ggml_sycl_op_mul_mat_vec_q(ggml_backend_sycl_context & ctx, const ggml_tensconst int64_t ne00 = src0->ne[0];const int64_t row_diff = row_high - row_low;+#ifdef GGML_SYCL_MMVQ_HAS_XMX+ if (ggml_sycl_mul_mat_vec_q_xmx(ctx.device, src0, dst, src0_dd_i, src1_ddq_i, dst_dd_i, row_diff, src1_ncols,+ src1_padded_col_size, stream)) {+ return;+ }+#endif // GGML_SYCL_MMVQ_HAS_XMX+int id;SYCL_CHECK(CHECK_TRY_ERROR(id = get_current_device_id()));const size_t q8_1_ts = sizeof(block_q8_1);diff --git a/ggml/src/ggml-sycl/mmvq.hpp b/ggml/src/ggml-sycl/mmvq.hppindex 3a0ef07d67..5649cbcd26 100644--- a/ggml/src/ggml-sycl/mmvq.hpp+++ b/ggml/src/ggml-sycl/mmvq.hpp@@ -80,6 +80,24 @@ bool ggml_sycl_mul_mat_vec_q_glu_reorder(dpct::queue_ptr stream);+// Fused dense-FFN GEMV on the XMX engines for reordered weights of one type, up * act(gate) for up to+// GGML_SYCL_XMX_GLU_MAX_COLS (at most) activation columns. Returns false if it does not apply, the caller falls back.+// vy: src1 quantized with quantize_and_reorder_q8_1_soa.+bool ggml_sycl_mul_mat_vec_q_glu_xmx(+ int device,+ enum ggml_type src0_type,+ enum ggml_glu_op glu_op,+ const void * vx, // up weights+ const void * vgate, // gate weights+ const void * vy,+ float * dst,+ int ncols, // K, shared by both weights+ int nrows, // output rows, i.e. weight ne[1]+ int ncols_dst, // activation columns+ int stride_col_y_bytes, // bytes between activation columns in vy+ int stride_col_dst, // floats between output columns in dst+ dpct::queue_ptr stream);+// Fused dense-FFN GEMV + GLU over the standard (non-reorder) layout; the gate and up// weights may carry different block types (q5_K / iq4_xs, mixed included).// vy: src1 quantized with plain quantize_q8_1 (padded rows). stride_col_y is indiff --git a/tests/test-backend-ops.cpp b/tests/test-backend-ops.cppindex 9f2d1f54db..f490d201c7 100644--- a/tests/test-backend-ops.cpp+++ b/tests/test-backend-ops.cpp@@ -12246,6 +12246,18 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {true, 16, 8, b, false, true, false));}+ // Gate and up on the SYCL XMX kernels: rows long enough for 2D block loads, a last tile of 8 rows, and+ // columns on both sides of the 8 and 16 column tiles+ for (ggml_type type : {GGML_TYPE_Q2_K, GGML_TYPE_Q3_K, GGML_TYPE_Q4_K, GGML_TYPE_Q5_K, GGML_TYPE_Q6_K,+ GGML_TYPE_Q8_0}) {+ for (ggml_glu_op glu_op : {GGML_GLU_OP_SWIGLU, GGML_GLU_OP_GEGLU}) {+ for (int64_t m_batch : {1, 2, 3, 5, 8, 9, 16, 17}) {+ test_cases.emplace_back(new test_mul_mat_vec_fusion(type, glu_op, m_batch, 1000, 1024,+ false, 16, 8, false, false, true, false, { 1, 1 }));+ }+ }+ }+for (int64_t k : {96, 896, 1024, 2880}) {for (ggml_glu_op glu_op : {GGML_GLU_OP_SWIGLU, GGML_GLU_OP_GEGLU}) {test_cases.emplace_back(new test_mul_mat_vec_fusion(GGML_TYPE_Q8_0, glu_op, 1, 32, k,