Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

Reduce SYCL-TLA Compiler Memory (#2193)

Luo Yuff2fd25c644d
diff --git a/auto_round_extension/ark/README.md b/auto_round_extension/ark/README.md
index 6e24fd355..47dd20b50 100644
--- a/auto_round_extension/ark/README.md
+++ b/auto_round_extension/ark/README.md
@@ -225,6 +225,28 @@ python setup.py bdist_wheel; pip install dist/*
Build with MoE / SageAttention support requires `ARK_SYCL_TLA=ON`.
+For oneAPI/SYCL-TLA builds, template compilation can require several gigabytes
+per compiler process. If the build host runs out of memory, limit concurrent
+XPU compilations with `ARK_SYCL_TLA_JOBS`:
+
+```bash
+ARK_SYCL_TLA_JOBS=1 pip install . --no-build-isolation
+```
+
+On a oneAPI-equipped build host, configure an XPU build with compile commands
+enabled and measure each translation unit before choosing a higher job count:
+
+```bash
+python tools/measure_sycl_tla_compile_memory.py \
+ --build-dir /path/to/ark/xbuild \
+ --output /tmp/ark-sycl-tla-rss.json
+```
+
+The measurement tool runs compiler commands serially and reports peak RSS,
+direct dispatch count, and reachable template declarations. The current
+environment must provide `icx` and a generated `compile_commands.json` for
+these measurements to represent the SYCL-TLA build.
+
---
## Tests
diff --git a/auto_round_extension/ark/README_CN.md b/auto_round_extension/ark/README_CN.md
new file mode 100644
index 000000000..8c631e56d
--- /dev/null
+++ b/auto_round_extension/ark/README_CN.md
@@ -0,0 +1,231 @@
+## 什么是 AutoRound Kernel(ARK)?
+
+AutoRound Kernel(ARK)是面向 Intel 平台的低比特加速库,为大语言模型推理提供三类优化算子。
+
+| 算子类别 | CPU | XPU(Battlemage) |
+|:--|:--:|:--:|
+| **仅权重量化线性层**(INT4/INT8/FP8/FP4) | ✅ | ✅ |
+| **MoE 分组 GEMM** | ❌ | ✅ |
+| **SageAttention**(SDPA / SAGE) | ❌ | ✅ |
+
+**已验证 CPU:** Intel Xeon Scalable(Sapphire Rapids / Emerald Rapids)、Intel Xeon 6(Sierra Forest / Granite Rapids)<br>
+**已验证 GPU:** Intel Arc B 系列 / Arc Pro B 系列(Battlemage)
+
+### 生态集成
+
+ARK 算子已集成到以下项目:
+
+| 项目 | 集成 | 说明 |
+|--|--|--|
+| [vllm](https://github.com/vllm-project/vllm) | [`inc_wna16_linear.py`](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/inc/schemes/inc_wna16_linear.py#L379-L470) | `INCXPUARKLinearMethod` 通过 `auto_round_kernel.qlinear.QuantLinear` 在 XPU 上提供仅权重量化线性层。 |
+| [vllm-omni](https://github.com/vllm-project/vllm-omni) | [`sage_attn.py`](https://github.com/vllm-project/vllm-omni/blob/f8340d078e4e9c3b793bd92d55d891b29703f0a8/vllm_omni/diffusion/attention/backends/sage_attn.py#L27) | `SageAttentionBackend` 通过 `ARK.sagev1` 在 XPU 上加速扩散模型注意力。 |
+| [Transformers](https://github.com/huggingface/transformers)(通过 [auto-round](https://github.com/intel/auto-round)) | [`backend.py`](https://github.com/intel/auto-round/blob/main/auto_round/inference/backend.py#L531) | AutoRound 量化的模型默认在 CPU/XPU 上使用 ARK,无需额外配置。 |
+
+---
+
+## 1. 线性层(仅权重量化 GEMM)
+
+面向大语言模型推理的低比特仅权重量化线性层,支持 CPU 和 XPU。
+
+### API
+
+| API | 说明 | 平台 |
+|--|--|--|
+| `QuantLinear` | 统一的 PyTorch 模块(GPTQ/AWQ/原始量化检查点) | CPU / XPU |
+| `QuantLinearGPTQ` | GPTQ 格式检查点加载器 | CPU / XPU |
+| `QuantLinearAWQ` | AWQ 格式检查点加载器 | CPU / XPU |
+| `QuantLinearFP8` | FP8 仅权重量化线性层 | CPU / XPU |
+| `woqgemm` | 低层仅权重量化 GEMM(打包格式) | CPU / XPU |
+| `woqgemm_s8` | 带 scale 的低层 INT8 权重 GEMM | CPU / XPU |
+| `_repack_quantized_weight` | 将原始 qweight/qzero/scale 重新打包为 ARK 格式 | CPU / XPU |
+| `_unpack_weight` | 将 ARK 格式权重解包回全精度 | CPU / XPU |
+
+### 关键特性
+
+> **W4A8 / W2A8 重缩放(QQQ 风格)**:XPU 上支持将低比特权重(INT2/INT4)重缩放为 INT8,并通过 INT8 GEMM 计算,避免 FP16 反量化。该功能通过环境变量 `ARK_AUTO_S8` 自动启用,详见 [xpu_wrapper.hpp](auto_round_kernel/wrapper/include/xpu_wrapper.hpp)。
+
+### 支持的数据类型
+
+#### CPU
+
+| 权重类型 | 计算类型 | Scale 类型 | 算法 |
+|--|:--:|:--:|:--:|
+| INT1–INT8 | INT8<sup>[1]</sup> / BF16 / FP32 | BF16 / FP32 | 对称 / 非对称 |
+| FP8(E4M3、E5M2) | BF16 / FP32 | FP32 / FP8(E8M0) | 不适用 |
+| FP4(E2M1) | BF16 / FP32 | BF16 / FP32 | 不适用 |
+
+#### XPU
+
+| 权重类型 | 计算类型 | Scale 类型 | 算法 |
+|--|:--:|:--:|:--:|
+| INT4、INT8 | INT8 / FP16 | FP16 | 对称 |
+| FP8(E4M3、E5M2) | FP16 | FP16 / FP8(E8M0) | 不适用 |
+
+<sup>[1]</sup> INT8 计算包含动态激活量化,结果会反量化为浮点类型。
+
+### 示例
+
+```python
+import auto_round_kernel as ark
+
+# 准备量化权重:qweight [K, N] int4/int2,scale [K/G, N] fp16/fp32,zp [K/G, N] int4/int2
+packw = ark.repack_quantized_weight(
+ qweight,
+ scale,
+ zp,
+ blocksize=128,
+ compute_type="fp16",
+ weight_type="int4",
+ scale_type="fp16",
+ asym=False,
+)
+
+# 运行仅权重量化 GEMM:activation [M, K] -> output [M, N]
+output = ark.woqgemm(
+ activation,
+ packw,
+ bias,
+ n,
+ k,
+ groupsize=128,
+ compute_type="fp16",
+ weight_type="int4",
+ scale_type="fp16",
+ asym=False,
+)
+
+# 解包回全精度以便验证
+_decompressed = ark.unpack_weight(
+ packw,
+ dtype=torch.float16,
+ n=n,
+ k=k,
+ groupsize=128,
+ compute_type="fp16",
+ weight_type="int4",
+ scale_type="fp16",
+ asym=False,
+)
+```
+
+详见 [test_weightonly.py](test/test_weightonly.py),其中包含 CPU 和 XPU 上的权重重新打包、验证和 `woqgemm` 运行示例。
+
+---
+
+## 2. MoE(混合专家分组 GEMM)
+
+面向 MoE 层的分组 GEMM,不同专家可以处理不同数量的 token。
+
+### API
+
+| 函数 | 说明 | 平台 | 激活类型 | 权重类型 |
+|--|--|--|:--:|:--:|
+| `ark.moe_gemm(...)` | 专家分组 GEMM | XPU | FP16 / BF16 | FP16 / BF16 |
+| `ark.moe_gemm(...)`(开发中) | INT4 权重分组 GEMM | XPU | FP16 / BF16 | INT4 🚧 |
+| `ark.moe_gemm(...)`(开发中) | INT2 权重分组 GEMM | XPU | FP16 / BF16 | INT2 🚧 |
+| `ark.moe_gemm(...)`(开发中) | INT8 权重分组 GEMM | XPU | FP16 / BF16 | INT8 🚧 |
+
+> 🚧 INT2 / INT4 / INT8 权重支持仍在开发中,详见 [#PR](https://github.com/intel/auto-round/pull)。
+
+### 细节
+
+| 参数 | 形状 | 类型 |
+|--|--|--|
+| activations | `[total_tokens, K]` | FP16 / BF16 |
+| weights | `[num_experts, K, N]`(行主序) | FP16 / BF16 |
+| num_tokens_per_expert | `[num_experts]` | INT32 |
+| scales(可选) | `[num_experts, N]` | FP16 / BF16 |
+| **output** | `[total_tokens, N]` | 与激活相同 |
+
+### 示例
+
+```python
+# FP16/BF16 MoE
+output = ark.moe_gemm(activations, weights, num_tokens_per_expert)
+
+# INT4 MoE(即将支持)
+# output = ark.moe_gemm(activations, q4_weights, num_tokens_per_expert, scales=scales)
+```
+
+构建要求:`ARK_SYCL_TLA=ON`。详见 [test_moe.py](test/test_moe.py)。
+
+---
+
+## 3. SageAttention(XPU SDPA 加速)
+
+ARK 在 XPU 上提供完整的缩放点积注意力内核族,从原生 FP16 SDPA 到 INT8 量化 SageAttention 变体均可使用。
+
+### API 概览
+
+| 函数 | 说明 | Q/K/V 输入 | PV 精度 | Head Dim |
+|--|--|--|:--:|:--:|
+| `ark.sdpa` | FP16/BF16 SDPA(flash attention) | FP16 / BF16 | FP16 | 64、96、128、192 |
+| `ark.sage` | 低层 INT8 SAGE(预量化 Q/K) | INT8(Q/K)、FP16(V) | FP16 | 64、128 |
+| `ark.sage_pvi8` | 低层 INT8 SAGE(预量化 Q/K/V) | INT8 | INT8 | 64、128 |
+| `ark.sagev1` | 高层 FP16 -> 内部 Q/K 量化 -> SAGE | FP16 / BF16 | FP16 | 64、128 |
+| `ark.sagev1_pvi8` | 高层 FP16 -> 内部 Q/K/V 量化 -> SAGE PV INT8 | FP16 / BF16 | INT8 | 64、128 |
+| `ark.sageattn` | SageAttention 兼容 API 的分发器 | FP16 / BF16 | FP16 / INT8 | 64、128 |
+| `ark.sage_dynquant` | 动态 INT8 分块 Q/K 量化 -> SAGE(可替换 SDPA) | FP16 / BF16 | FP16 | 64、128 |
+
+### 约束
+
+| 约束 | `sdpa` | `sagev1` / `sagev1_pvi8` / `sage_dynquant` |
+|--|:--:|:--:|
+| Q/K/V 类型 | FP16、BF16 | FP16、BF16 |
+| Head dim | 64、96、128、192 | 64、128 |
+| `dropout_p` | 必须为 0.0 | 必须为 0.0 |
+| 布尔 mask | 回退到 PyTorch | 回退到 PyTorch |
+| 加性 mask 形状 | `[B, 1, Sq, Skv]` FP32 | `[B, 1, Sq, Skv]` FP32 |
+| `quant_block_size` | 不适用 | 1(逐 token)或 >=32 |
+
+---
+
+## 安装
+
+### 通过 pip 安装
+
+```bash
+pip install auto-round-lib
+```
+
+### 从源码安装
+
+```bash
+pip install . --no-build-isolation
+# 或
+python setup.py bdist_wheel; pip install dist/*
+```
+
+启用 MoE / SageAttention 支持需要 `ARK_SYCL_TLA=ON`。
+
+对于 oneAPI/SYCL-TLA 构建,每个编译进程的模板编译可能需要数 GB 内存。如果构建主机内存不足,可以通过 `ARK_SYCL_TLA_JOBS` 限制并发的 XPU 编译数量:
+
+```bash
+ARK_SYCL_TLA_JOBS=1 pip install . --no-build-isolation
+```
+
+在配备 oneAPI 的构建主机上,建议启用编译命令并测量每个翻译单元,再选择更高的并发数:
+
+```bash
+python tools/measure_sycl_tla_compile_memory.py \
+ --build-dir /path/to/ark/xbuild \
+ --output /tmp/ark-sycl-tla-rss.json
+```
+
+该工具会串行运行编译器命令,并报告峰值 RSS、直接分发数量和可达模板声明数量。要使测量结果反映 SYCL-TLA 构建,当前环境必须提供 `icx` 以及生成的 `compile_commands.json`。
+
+---
+
+## 测试
+
+| 测试 | 说明 |
+|--|--|
+| [test_weightonly.py](test/test_weightonly.py) | WOQ GEMM 打包、解包和运行(CPU/XPU) |
+| [test_moe.py](test/test_moe.py) | MoE 分组 GEMM |
+| [test_flash_attn.py](test/test_flash_attn.py) | SDPA(flash attention)prefill |
+| [test_sdpa.py](test/test_sdpa.py) | SDPA 基准测试 |
+| [test_sdpa_parity.py](test/test_sdpa_parity.py) | SDPA 与 PyTorch 的一致性测试 |
+| [test_sage_dynquant.py](test/test_sage_dynquant.py) | SageAttention 动态 INT8 量化基准测试 |
+| [test_bench_bmg.py](test/test_bench_bmg.py) | BMG SDPA / SageAttention 基准测试 |
+| [test_matmul.py](test/test_matmul.py) | 低层 matmul |
+| [test_packq.py](test/test_packq.py) | 权重打包工具 |
diff --git a/auto_round_extension/ark/auto_round_kernel/CMakeLists.txt b/auto_round_extension/ark/auto_round_kernel/CMakeLists.txt
index 0d915d7f3..718ea46dd 100755
--- a/auto_round_extension/ark/auto_round_kernel/CMakeLists.txt
+++ b/auto_round_extension/ark/auto_round_kernel/CMakeLists.txt
@@ -101,6 +101,7 @@ if(NOT ARK_XPU)
endif()
set(SDPA_GENERATED_SRCS)
set(SDPA_KERNEL_DECLARATIONS)
+set(SYCL_TLA_GENERATED_SRCS)
include_directories(wrapper/include)
include_directories(${CMAKE_CURRENT_SOURCE_DIR})
@@ -161,9 +162,10 @@ if(ARK_XPU AND ARK_SYCL_TLA)
${CUTLASS_DIR}/examples/12_xe20_moe_gemm_cute_interface
)
include(${CMAKE_CURRENT_LIST_DIR}/sdpa_generation.cmake)
+ include(${CMAKE_CURRENT_LIST_DIR}/sycl_tla_generation.cmake)
endif()
-pybind11_add_module(${PY_NAME} ${SRCS} ${HEADERS} ${SDPA_GENERATED_SRCS})
+pybind11_add_module(${PY_NAME} ${SRCS} ${HEADERS} ${SDPA_GENERATED_SRCS} ${SYCL_TLA_GENERATED_SRCS})
target_compile_features(${PY_NAME} PRIVATE cxx_std_17)
target_compile_definitions(${PY_NAME} PRIVATE PY_NAME=${PY_NAME} ${ARK_TYPE}=1)
if(ARK_DNNL)
diff --git a/auto_round_extension/ark/auto_round_kernel/ark.cpp b/auto_round_extension/ark/auto_round_kernel/ark.cpp
index aa843020d..94510ce8d 100755
--- a/auto_round_extension/ark/auto_round_kernel/ark.cpp
+++ b/auto_round_extension/ark/auto_round_kernel/ark.cpp
@@ -28,13 +28,7 @@ typedef uintptr_t torch_ptr;
#include "xpu_wrapper.hpp"
#include "sycl_s8_wrapper.hpp"
#if ARK_SYCL_TLA
-// Only include declarations, implementations are in separate .cpp files
#include "sycl_tla_common.hpp"
-#include "sycl_tla_moe.hpp"
-#include "sycl_tla_moe_decode.hpp"
-#include "sycl_tla_moe_mixed.hpp"
-#include "sycl_tla_sdpa.hpp"
-#include "sycl_tla_dense_gemm.hpp"
#endif
#else
#include "ark/cpu/sdpa.h"
diff --git a/auto_round_extension/ark/auto_round_kernel/sdpa.cpp b/auto_round_extension/ark/auto_round_kernel/sdpa.cpp
index 2d9b5f424..6220d0cfa 100644
--- a/auto_round_extension/ark/auto_round_kernel/sdpa.cpp
+++ b/auto_round_extension/ark/auto_round_kernel/sdpa.cpp
@@ -114,31 +114,103 @@ KernelLauncher select_sage_prefill_launcher(BTLA_DTYPE dtype, int head_dim) {
}
-KernelLauncher select_prefill_launcher(BTLA_DTYPE dtype, int head_dim) {
+KernelLauncher select_prefill_launcher(BTLA_DTYPE dtype, int head_dim, bool is_causal, bool has_mask, bool is_varlen) {
switch (dtype) {
case BTLA_DTYPE::F16:
switch (head_dim) {
case 128:
- return detail::launch_prefill_kernel_f16_128;
+ if (is_causal) {
+ return is_varlen ? detail::launch_prefill_kernel_f16_128_causal_varlen
+ : detail::launch_prefill_kernel_f16_128_causal;
+ }
+ if (has_mask) {
+ return is_varlen ? detail::launch_prefill_kernel_f16_128_noncausal_masked_varlen
+ : detail::launch_prefill_kernel_f16_128_noncausal_masked;
+ }
+ return is_varlen ? detail::launch_prefill_kernel_f16_128_noncausal_unmasked_varlen
+ : detail::launch_prefill_kernel_f16_128_noncausal_unmasked;
case 64:
- return detail::launch_prefill_kernel_f16_64;
+ if (is_causal) {
+ return is_varlen ? detail::launch_prefill_kernel_f16_64_causal_varlen
+ : detail::launch_prefill_kernel_f16_64_causal;
+ }
+ if (has_mask) {
+ return is_varlen ? detail::launch_prefill_kernel_f16_64_noncausal_masked_varlen
+ : detail::launch_prefill_kernel_f16_64_noncausal_masked;
+ }
+ return is_varlen ? detail::launch_prefill_kernel_f16_64_noncausal_unmasked_varlen
+ : detail::launch_prefill_kernel_f16_64_noncausal_unmasked;
case 96:
- return detail::launch_prefill_kernel_f16_96;
+ if (is_causal) {
+ return is_varlen ? detail::launch_prefill_kernel_f16_96_causal_varlen
+ : detail::launch_prefill_kernel_f16_96_causal;
+ }
+ if (has_mask) {
+ return is_varlen ? detail::launch_prefill_kernel_f16_96_noncausal_masked_varlen
+ : detail::launch_prefill_kernel_f16_96_noncausal_masked;
+ }
+ return is_varlen ? detail::launch_prefill_kernel_f16_96_noncausal_unmasked_varlen
+ : detail::launch_prefill_kernel_f16_96_noncausal_unmasked;
case 192:
- return detail::launch_prefill_kernel_f16_192;
+ if (is_causal) {
+ return is_varlen ? detail::launch_prefill_kernel_f16_192_causal_varlen
+ : detail::launch_prefill_kernel_f16_192_causal;
+ }
+ if (has_mask) {
+ return is_varlen ? detail::launch_prefill_kernel_f16_192_noncausal_masked_varlen
+ : detail::launch_prefill_kernel_f16_192_noncausal_masked;
+ }
+ return is_varlen ? detail::launch_prefill_kernel_f16_192_noncausal_unmasked_varlen
+ : detail::launch_prefill_kernel_f16_192_noncausal_unmasked;
default:
return nullptr;
}
case BTLA_DTYPE::BF16:
switch (head_dim) {
case 64:
- return detail::launch_prefill_kernel_bf16_64;
+ if (is_causal) {
+ return is_varlen ? detail::launch_prefill_kernel_bf16_64_causal_varlen
+ : detail::launch_prefill_kernel_bf16_64_causal;
+ }
+ if (has_mask) {
+ return is_varlen ? detail::launch_prefill_kernel_bf16_64_noncausal_masked_varlen
+ : detail::launch_prefill_kernel_bf16_64_noncausal_masked;
+ }
+ return is_varlen ? detail::launch_prefill_kernel_bf16_64_noncausal_unmasked_varlen
+ : detail::launch_prefill_kernel_bf16_64_noncausal_unmasked;
case 96:
- return detail::launch_prefill_kernel_bf16_96;
+ if (is_causal) {
+ return is_varlen ? detail::launch_prefill_kernel_bf16_96_causal_varlen
+ : detail::launch_prefill_kernel_bf16_96_causal;
+ }
+ if (has_mask) {
+ return is_varlen ? detail::launch_prefill_kernel_bf16_96_noncausal_masked_varlen
+ : detail::launch_prefill_kernel_bf16_96_noncausal_masked;
+ }
+ return is_varlen ? detail::launch_prefill_kernel_bf16_96_noncausal_unmasked_varlen
+ : detail::launch_prefill_kernel_bf16_96_noncausal_unmasked;
case 128:
- return detail::launch_prefill_kernel_bf16_128;
+ if (is_causal) {
+ return is_varlen ? detail::launch_prefill_kernel_bf16_128_causal_varlen
+ : detail::launch_prefill_kernel_bf16_128_causal;
+ }
+ if (has_mask) {
+ return is_varlen ? detail::launch_prefill_kernel_bf16_128_noncausal_masked_varlen
+ : detail::launch_prefill_kernel_bf16_128_noncausal_masked;
+ }
+ return is_varlen ? detail::launch_prefill_kernel_bf16_128_noncausal_unmasked_varlen
+ : detail::launch_prefill_kernel_bf16_128_noncausal_unmasked;
case 192:
- return detail::launch_prefill_kernel_bf16_192;
+ if (is_causal) {
+ return is_varlen ? detail::launch_prefill_kernel_bf16_192_causal_varlen
+ : detail::launch_prefill_kernel_bf16_192_causal;
+ }
+ if (has_mask) {
+ return is_varlen ? detail::launch_prefill_kernel_bf16_192_noncausal_masked_varlen
+ : detail::launch_prefill_kernel_bf16_192_noncausal_masked;
+ }
+ return is_varlen ? detail::launch_prefill_kernel_bf16_192_noncausal_unmasked_varlen
+ : detail::launch_prefill_kernel_bf16_192_noncausal_unmasked;
default:
return nullptr;
}
@@ -147,31 +219,47 @@ KernelLauncher select_prefill_launcher(BTLA_DTYPE dtype, int head_dim) {
}
}
-KernelLauncher select_decode_launcher(BTLA_DTYPE dtype, int head_dim) {
+KernelLauncher select_decode_launcher(BTLA_DTYPE dtype, int head_dim, bool is_causal, bool has_mask) {
switch (dtype) {
case BTLA_DTYPE::F16:
switch (head_dim) {
case 128:
- return detail::launch_decode_kernel_f16_128;
+ if (is_causal) return detail::launch_decode_kernel_f16_128_causal;
+ if (has_mask) return detail::launch_decode_kernel_f16_128_noncausal_masked;
+ return detail::launch_decode_kernel_f16_128_noncausal_unmasked;
case 64:
- return detail::launch_decode_kernel_f16_64;
+ if (is_causal) return detail::launch_decode_kernel_f16_64_causal;
+ if (has_mask) return detail::launch_decode_kernel_f16_64_noncausal_masked;
+ return detail::launch_decode_kernel_f16_64_noncausal_unmasked;
case 96:
- return detail::launch_decode_kernel_f16_96;
+ if (is_causal) return detail::launch_decode_kernel_f16_96_causal;
+ if (has_mask) return detail::launch_decode_kernel_f16_96_noncausal_masked;
+ return detail::launch_decode_kernel_f16_96_noncausal_unmasked;
case 192:
- return detail::launch_decode_kernel_f16_192;
+ if (is_causal) return detail::launch_decode_kernel_f16_192_causal;
+ if (has_mask) return detail::launch_decode_kernel_f16_192_noncausal_masked;
+ return detail::launch_decode_kernel_f16_192_noncausal_unmasked;
default:
return nullptr;
}
case BTLA_DTYPE::BF16:
switch (head_dim) {
case 64:
- return detail::launch_decode_kernel_bf16_64;
+ if (is_causal) return detail::launch_decode_kernel_bf16_64_causal;
+ if (has_mask) return detail::launch_decode_kernel_bf16_64_noncausal_masked;
+ return detail::launch_decode_kernel_bf16_64_noncausal_unmasked;
case 96:
- return detail::launch_decode_kernel_bf16_96;
+ if (is_causal) return detail::launch_decode_kernel_bf16_96_causal;
+ if (has_mask) return detail::launch_decode_kernel_bf16_96_noncausal_masked;
+ return detail::launch_decode_kernel_bf16_96_noncausal_unmasked;
case 128:
- return detail::launch_decode_kernel_bf16_128;
+ if (is_causal) return detail::launch_decode_kernel_bf16_128_causal;
+ if (has_mask) return detail::launch_decode_kernel_bf16_128_noncausal_masked;
+ return detail::launch_decode_kernel_bf16_128_noncausal_unmasked;
case 192:
- return detail::launch_decode_kernel_bf16_192;
+ if (is_causal) return detail::launch_decode_kernel_bf16_192_causal;
+ if (has_mask) return detail::launch_decode_kernel_bf16_192_noncausal_masked;
+ return detail::launch_decode_kernel_bf16_192_noncausal_unmasked;
default:
return nullptr;
}
@@ -269,7 +357,7 @@ void flash_attn_prefill(sycl::queue* q, void* Q_ptr, void* K_ptr, void* V_ptr, v
options.lse = lse;
compat::set_default_queue(*q);
- KernelLauncher launcher = select_prefill_launcher(q_dtype, head_dim);
+ KernelLauncher launcher = select_prefill_launcher(q_dtype, head_dim, is_causal, mask != nullptr, false);
if (launcher == nullptr) {
throw std::runtime_error(
"Unsupported dtype or head dimension for flash_attn_prefill (only F16/BF16 and 64/96/128/192 are supported)");
@@ -293,7 +381,7 @@ void flash_attn_decode(sycl::queue* q, void* Q_ptr, void* K_ptr, void* V_ptr, vo
options.lse = lse;
compat::set_default_queue(*q);
- KernelLauncher launcher = select_decode_launcher(q_dtype, head_dim);
+ KernelLauncher launcher = select_decode_launcher(q_dtype, head_dim, is_causal, mask != nullptr);
if (launcher == nullptr) {
throw std::runtime_error(
"Unsupported dtype or head dimension for flash_attn_decode (only F16/BF16 and 64/96/128/192 are supported)");
@@ -520,7 +608,7 @@ void sdpa_varlen_impl(sycl::queue* q, void* Q_ptr, void* K_ptr, void* V_ptr, voi
options.cu_seqlens_kv_cache = zero_cu_buf;
options.use_tensor_strides = true;
- KernelLauncher launcher = select_prefill_launcher(q_dtype, head_dim);
+ KernelLauncher launcher = select_prefill_launcher(q_dtype, head_dim, is_causal, mask != nullptr, true);
if (launcher == nullptr) {
throw std::runtime_error(
"Unsupported dtype or head dimension for sdpa_varlen_impl (only F16/BF16 and 64/96/128/192 are supported)");
diff --git a/auto_round_extension/ark/auto_round_kernel/sdpa_generation.cmake b/auto_round_extension/ark/auto_round_kernel/sdpa_generation.cmake
index 051de0c68..aebbbf8f8 100644
--- a/auto_round_extension/ark/auto_round_kernel/sdpa_generation.cmake
+++ b/auto_round_extension/ark/auto_round_kernel/sdpa_generation.cmake
@@ -1,14 +1,19 @@
file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/generated/sdpa)
set(SDPA_DECLARATIONS_FILE ${CMAKE_CURRENT_BINARY_DIR}/generated/sdpa/sdpa_kernel_declarations.hpp)
-function(generate_sdpa_instantiation mode dtype_name dtype_type head_dim template_name function_name)
+function(generate_sdpa_instantiation mode dtype_name dtype_type head_dim variant_name causal_value has_mask_value
+ is_varlen_value)
+ set(function_name launch_${mode}_kernel_${dtype_name}_${head_dim}_${variant_name})
set(SDPA_FUNCTION_NAME ${function_name})
- set(SDPA_TEMPLATE_NAME ${template_name})
+ set(SDPA_TEMPLATE_NAME launch_${mode}_kernel_${head_dim})
+ set(SDPA_CAUSAL ${causal_value})
+ set(SDPA_HAS_MASK ${has_mask_value})
+ set(SDPA_IS_VARLEN ${is_varlen_value})
set(SDPA_ELEMENT_TYPE ${dtype_type})
string(APPEND SDPA_KERNEL_DECLARATIONS "int ${function_name}(Options const& options);\n")
set(SDPA_KERNEL_DECLARATIONS ${SDPA_KERNEL_DECLARATIONS} PARENT_SCOPE)
set(_generated_src
- ${CMAKE_CURRENT_BINARY_DIR}/generated/sdpa/${mode}_${dtype_name}_${head_dim}.cpp)
+ ${CMAKE_CURRENT_BINARY_DIR}/generated/sdpa/${mode}_${dtype_name}_${head_dim}_${variant_name}.cpp)
configure_file(${CMAKE_CURRENT_LIST_DIR}/sdpa_kernel_instantiation.cpp.in ${_generated_src} @ONLY)
list(APPEND SDPA_GENERATED_SRCS ${_generated_src})
set(SDPA_GENERATED_SRCS ${SDPA_GENERATED_SRCS} PARENT_SCOPE)
@@ -23,9 +28,44 @@ foreach(mode IN ITEMS prefill decode)
endif()
foreach(head_dim IN ITEMS 128 64 96 192)
- set(template_name launch_${mode}_kernel_${head_dim})
- set(function_name launch_${mode}_kernel_${dtype_name}_${head_dim})
- generate_sdpa_instantiation(${mode} ${dtype_name} "${dtype_type}" ${head_dim} ${template_name} ${function_name})
+ foreach(varlen_name IN ITEMS normal varlen)
+ if(varlen_name STREQUAL "varlen")
+ if(mode STREQUAL "decode")
+ continue()
+ endif()
+ set(is_varlen_value true)
+ set(varlen_suffix "_varlen")
+ else()
+ set(is_varlen_value false)
+ set(varlen_suffix "")
+ endif()
+
+ foreach(causal_name IN ITEMS causal noncausal)
+ if(causal_name STREQUAL "causal")
+ set(causal_value true)
+ set(mask_names unmasked)
+ else()
+ set(causal_value false)
+ set(mask_names masked unmasked)
+ endif()
+
+ foreach(mask_name IN LISTS mask_names)
+ if(mask_name STREQUAL "masked")
+ set(has_mask_value true)
+ else()
+ set(has_mask_value false)
+ endif()
+
+ set(variant_name ${causal_name})
+ if(causal_name STREQUAL "noncausal")
+ set(variant_name noncausal_${mask_name})
+ endif()
+ string(APPEND variant_name ${varlen_suffix})
+ generate_sdpa_instantiation(${mode} ${dtype_name} "${dtype_type}" ${head_dim} ${variant_name}
+ ${causal_value} ${has_mask_value} ${is_varlen_value})
+ endforeach()
+ endforeach()
+ endforeach()
endforeach()
endforeach()
endforeach()
diff --git a/auto_round_extension/ark/auto_round_kernel/sdpa_kernel_instantiation.cpp.in b/auto_round_extension/ark/auto_round_kernel/sdpa_kernel_instantiation.cpp.in
index b9f7f004e..7305a70f8 100644
--- a/auto_round_extension/ark/auto_round_kernel/sdpa_kernel_instantiation.cpp.in
+++ b/auto_round_extension/ark/auto_round_kernel/sdpa_kernel_instantiation.cpp.in
@@ -9,7 +9,8 @@ namespace ark {
namespace detail {
int @SDPA_FUNCTION_NAME@(Options const& options) {
- return @SDPA_TEMPLATE_NAME@<@SDPA_ELEMENT_TYPE@, @SDPA_ELEMENT_TYPE@, @SDPA_ELEMENT_TYPE@>(options);
+ return @SDPA_TEMPLATE_NAME@<@SDPA_CAUSAL@, @SDPA_HAS_MASK@, @SDPA_IS_VARLEN@, @SDPA_ELEMENT_TYPE@,
+ @SDPA_ELEMENT_TYPE@, @SDPA_ELEMENT_TYPE@>(options);
}
} // namespace detail
diff --git a/auto_round_extension/ark/auto_round_kernel/sycl_tla_gemm.cpp.in b/auto_round_extension/ark/auto_round_kernel/sycl_tla_gemm.cpp.in
new file mode 100644
index 000000000..60a99e768
--- /dev/null
+++ b/auto_round_extension/ark/auto_round_kernel/sycl_tla_gemm.cpp.in
@@ -0,0 +1,167 @@
+// Generated by CMake. Do not edit directly.
+
+#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)
+
+#if @DENSE_SOURCE_MODE@ > 0
+
+#if @DENSE_SOURCE_MODE@ == 1
+
+#include "sycl_tla_dense_gemm.hpp"
+#include "sycl_tla_dense_gemm_helpers.hpp"
+
+namespace ark {
+namespace dense_gemm_detail {
+
+void @DENSE_FUNCTION_NAME@(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c@DENSE_BIAS_PARAMETER@) {
+ run_dense_gemm_fixed_tile<@DENSE_HAS_BIAS@, @DENSE_TILE_M@, @DENSE_TILE_N@, @DENSE_LAYOUT@>(
+ q, m, n, k, static_cast<const @DENSE_ELEMENT_TYPE@*>(a), static_cast<const @DENSE_ELEMENT_TYPE@*>(b),
+ static_cast<@DENSE_ELEMENT_TYPE@*>(c), @DENSE_BIAS_ARGUMENT@);
+}
+
+} // namespace dense_gemm_detail
+} // namespace ark
+
+#elif @DENSE_SOURCE_MODE@ == 2
+
+#include "sycl_tla_common.hpp"
+
+namespace ark {
+
+void sycl_tla_dense_gemm(sycl::queue* q, int m, int n, int k, const void* a, BTLA_DTYPE at, const void* b,
+ BTLA_DTYPE bt, void* c, BTLA_DTYPE ct, const void* bias, bool BT) {
+ if (!q) {
+ throw std::invalid_argument("sycl_tla_dense_gemm: stream must be a valid SYCL queue");
+ }
+ if (!BT) {
+ throw std::invalid_argument("sycl_tla_dense_gemm: only the A @ B.T contract is supported");
+ }
+ if (!a || !b || !c) {
+ throw std::invalid_argument("sycl_tla_dense_gemm: input and output pointers must not be null");
+ }
+ if (m <= 0 || n <= 0 || k <= 0) return;
+ if (at != bt) {
+ throw std::invalid_argument("sycl_tla_dense_gemm: A and B must use the same dtype");
+ }
+ if (ct != at) {
+ throw std::invalid_argument("sycl_tla_dense_gemm: output dtype must match input dtype");
+ }
+
+ switch (at) {
+ case BTLA_DTYPE::F32:
+ sycl_tla_dense_gemm_f32(q, m, n, k, a, b, c, bias);
+ return;
+ case BTLA_DTYPE::F16:
+ sycl_tla_dense_gemm_f16(q, m, n, k, a, b, c, bias);
+ return;
+ case BTLA_DTYPE::BF16:
+ sycl_tla_dense_gemm_bf16(q, m, n, k, a, b, c, bias);
+ return;
+ default:
+ throw std::invalid_argument("sycl_tla_dense_gemm: only FP32, FP16 and BF16 are supported");
+ }
+}
+
+} // namespace ark
+
+#elif @DENSE_SOURCE_MODE@ == 3
+
+#include "sycl_tla_dense_gemm_helpers.hpp"
+
+namespace ark {
+
+void sycl_tla_dense_gemm_@DENSE_DISPATCH_DTYPE@(sycl::queue* q, int m, int n, int k, const void* a, const void* b,
+ void* c, const void* bias) {
+ if (bias != nullptr) {
+ if (m < 32) dense_gemm_detail::run_@DENSE_DISPATCH_DTYPE@_with_bias_small(q, m, n, k, a, b, c, bias);
+ else if (m < 128) dense_gemm_detail::run_@DENSE_DISPATCH_DTYPE@_with_bias_mid(q, m, n, k, a, b, c, bias);
+ else if (m <= 1024) dense_gemm_detail::run_@DENSE_DISPATCH_DTYPE@_with_bias_medium(q, m, n, k, a, b, c, bias);
+ else dense_gemm_detail::run_@DENSE_DISPATCH_DTYPE@_with_bias_large(q, m, n, k, a, b, c, bias);
+ } else {
+ if (m < 32) dense_gemm_detail::run_@DENSE_DISPATCH_DTYPE@_without_bias_small(q, m, n, k, a, b, c);
+ else if (m < 128) dense_gemm_detail::run_@DENSE_DISPATCH_DTYPE@_without_bias_mid(q, m, n, k, a, b, c);
+ else if (m <= 1024) dense_gemm_detail::run_@DENSE_DISPATCH_DTYPE@_without_bias_medium(q, m, n, k, a, b, c);
+ else dense_gemm_detail::run_@DENSE_DISPATCH_DTYPE@_without_bias_large(q, m, n, k, a, b, c);
+ }
+}
+
+} // namespace ark
+
+#endif
+
+#elif @S8_SOURCE_MODE@ > 0
+
+#if @S8_SOURCE_MODE@ == 1
+
+#include "sycl_tla_s8_gemm_fixed.hpp"
+
+@S8_ROUTE_MACRO@(@S8_FUNCTION_NAME@, @S8_ELEMENT_TYPE@, @S8_TILE_M@, @S8_TILE_N@, @S8_LAYOUT@)
+
+#elif @S8_SOURCE_MODE@ == 2
+
+#include "sycl_tla_common.hpp"
+
+namespace ark {
+
+void sycl_tla_igemm_s8s8_dequant(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,
+ BTLA_DTYPE ct, const void* scale_a, const void* scale_b, const void* bias,
+ int blocksize) {
+ if (!q) throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: queue must not be null");
+ if (!a || !b || !c || !scale_a || !scale_b) {
+ throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: input pointers must not be null");
+ }
+ if (m <= 0 || n <= 0 || k <= 0) return;
+ if (ct == BTLA_DTYPE::BF16 && !(blocksize == k || blocksize == -1)) {
+ throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: k-block path supports only F32/F16 output");
+ }
+
+ switch (ct) {
+ case BTLA_DTYPE::F32:
+ sycl_tla_igemm_s8s8_dequant_f32(q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize);
+ return;
+ case BTLA_DTYPE::F16:
+ sycl_tla_igemm_s8s8_dequant_f16(q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize);
+ return;
+ case BTLA_DTYPE::BF16:
+ sycl_tla_igemm_s8s8_dequant_bf16(q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize);
+ return;
+ default:
+ throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: unsupported output dtype");
+ }
+}
+
+} // namespace ark
+
+#elif @S8_SOURCE_MODE@ == 3
+
+#include "sycl_tla_s8_gemm_helpers.hpp"
+
+namespace ark {
+
+void sycl_tla_igemm_s8s8_dequant_@S8_DISPATCH_DTYPE@(sycl::queue* q, int m, int n, int k, const void* a,
+ const void* b, void* c, const void* scale_a, const void* scale_b,
+ const void* bias, int blocksize) {
+ if (blocksize == k || blocksize == -1) {
+ if (m < 16) sycl_tla_s8_detail::run_@S8_DISPATCH_DTYPE@_normal_small(q, m, n, k, a, b, c, scale_a, scale_b, bias);
+ else if (m < 128) sycl_tla_s8_detail::run_@S8_DISPATCH_DTYPE@_normal_mid(q, m, n, k, a, b, c, scale_a, scale_b, bias);
+ else if (m <= 1024) sycl_tla_s8_detail::run_@S8_DISPATCH_DTYPE@_normal_medium(q, m, n, k, a, b, c, scale_a, scale_b, bias);
+ else sycl_tla_s8_detail::run_@S8_DISPATCH_DTYPE@_normal_large(q, m, n, k, a, b, c, scale_a, scale_b, bias);
+ return;
+ }
+#if @S8_KBLOCK_SUPPORTED@
+ sycl_tla_s8_detail::validate_kblock_args(k, blocksize);
+ if (m < 16) sycl_tla_s8_detail::run_@S8_DISPATCH_DTYPE@_kblock_small(q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize);
+ else if (m < 128) sycl_tla_s8_detail::run_@S8_DISPATCH_DTYPE@_kblock_mid(q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize);
+ else if (m <= 1024) sycl_tla_s8_detail::run_@S8_DISPATCH_DTYPE@_kblock_medium(q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize);
+ else sycl_tla_s8_detail::run_@S8_DISPATCH_DTYPE@_kblock_large(q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize);
+#else
+ throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: k-block path supports only F32/F16 output");
+#endif
+}
+
+} // namespace ark
+
+#endif
+
+#endif
+
+#endif // ARK_XPU && ARK_SYCL_TLA
\ No newline at end of file
diff --git a/auto_round_extension/ark/auto_round_kernel/sycl_tla_generation.cmake b/auto_round_extension/ark/auto_round_kernel/sycl_tla_generation.cmake
new file mode 100644
index 000000000..811f781a0
--- /dev/null
+++ b/auto_round_extension/ark/auto_round_kernel/sycl_tla_generation.cmake
@@ -0,0 +1,401 @@
+set(SYCL_TLA_GENERATED_DIR ${CMAKE_CURRENT_BINARY_DIR}/generated/sycl_tla)
+set(SYCL_TLA_TEMPLATE_DIR ${CMAKE_CURRENT_LIST_DIR})
+file(MAKE_DIRECTORY ${SYCL_TLA_GENERATED_DIR})
+
+function(generate_sycl_tla_source template_name output_name)
+ set(_generated_src ${SYCL_TLA_GENERATED_DIR}/${output_name})
+ configure_file(${SYCL_TLA_TEMPLATE_DIR}/${template_name} ${_generated_src} @ONLY)
+ list(APPEND SYCL_TLA_GENERATED_SRCS ${_generated_src})
+ set(SYCL_TLA_GENERATED_SRCS ${SYCL_TLA_GENERATED_SRCS} PARENT_SCOPE)
+endfunction()
+
+set(DENSE_SOURCE_MODE 1)
+set(S8_SOURCE_MODE 0)
+set(_dense_dtypes f16 bf16 f32)
+foreach(_dense_dtype IN LISTS _dense_dtypes)
+ if(_dense_dtype STREQUAL "f16")
+ set(DENSE_ELEMENT_TYPE "cute::half_t")
+ elseif(_dense_dtype STREQUAL "bf16")
+ set(DENSE_ELEMENT_TYPE "cute::bfloat16_t")
+ else()
+ set(DENSE_ELEMENT_TYPE "float")
+ endif()
+
+ foreach(_dense_bias IN ITEMS bias no_bias)
+ if(_dense_bias STREQUAL "bias")
+ set(DENSE_HAS_BIAS true)
+ set(DENSE_FUNCTION_BIAS with_bias)
+ set(DENSE_BIAS_PARAMETER ", const void* bias")
+ set(DENSE_BIAS_ARGUMENT "static_cast<const ${DENSE_ELEMENT_TYPE}*>(bias)")
+ else()
+ set(DENSE_HAS_BIAS false)
+ set(DENSE_FUNCTION_BIAS without_bias)
+ set(DENSE_BIAS_PARAMETER "")
+ set(DENSE_BIAS_ARGUMENT "static_cast<const ${DENSE_ELEMENT_TYPE}*>(nullptr)")
+ endif()
+
+ foreach(_dense_tile IN ITEMS small mid medium large)
+ if(_dense_tile STREQUAL "small")
+ set(DENSE_TILE_M 8)
+ set(DENSE_TILE_N 128)
+ set(DENSE_LAYOUT DenseSmallTileSG)
+ set(_dense_output sycl_tla_dense_gemm_${_dense_dtype}_${_dense_bias}.cpp)
+ elseif(_dense_tile STREQUAL "mid")
+ set(DENSE_TILE_M 64)
+ set(DENSE_TILE_N 128)
+ set(DENSE_LAYOUT DenseSmallMidTileSG)
+ set(_dense_output sycl_tla_dense_gemm_${_dense_dtype}_${_dense_bias}_mid.cpp)
+ elseif(_dense_tile STREQUAL "medium")
+ set(DENSE_TILE_M 128)
+ set(DENSE_TILE_N 128)
+ set(DENSE_LAYOUT DenseMediumTileSG)
+ set(_dense_output sycl_tla_dense_gemm_${_dense_dtype}_${_dense_bias}_medium.cpp)
+ else()
+ set(DENSE_TILE_M 256)
+ set(DENSE_TILE_N 256)
+ set(DENSE_LAYOUT DenseLargeTileSG)
+ set(_dense_output sycl_tla_dense_gemm_${_dense_dtype}_${_dense_bias}_large.cpp)
+ endif()
+
+ set(DENSE_FUNCTION_NAME run_${_dense_dtype}_${DENSE_FUNCTION_BIAS}_${_dense_tile})
+ generate_sycl_tla_source(sycl_tla_gemm.cpp.in ${_dense_output})
+ endforeach()
+ endforeach()
+endforeach()
+
+set(DENSE_SOURCE_MODE 0)
+set(S8_SOURCE_MODE 1)
+set(S8_KBLOCK_SUPPORTED 0)
+set(_s8_dtypes f16 bf16 f32)
+foreach(_s8_dtype IN LISTS _s8_dtypes)
+ if(_s8_dtype STREQUAL "f16")
+ set(S8_ELEMENT_TYPE "cute::half_t")
+ elseif(_s8_dtype STREQUAL "bf16")
+ set(S8_ELEMENT_TYPE "cute::bfloat16_t")
+ else()
+ set(S8_ELEMENT_TYPE "float")
+ endif()
+
+ set(_s8_paths normal)
+ if(NOT _s8_dtype STREQUAL "bf16")
+ list(APPEND _s8_paths kblock)
+ endif()
+
+ foreach(_s8_path IN LISTS _s8_paths)
+ if(_s8_path STREQUAL "normal")
+ set(S8_ROUTE_MACRO ARK_DEFINE_S8_NORMAL_ROUTE)
+ else()
+ set(S8_ROUTE_MACRO ARK_DEFINE_S8_KBLOCK_ROUTE)
+ endif()
+
+ foreach(_s8_tile IN ITEMS small mid medium large)
+ if(_s8_tile STREQUAL "small")
+ set(S8_TILE_M 8)
+ set(S8_TILE_N 128)
+ elseif(_s8_tile STREQUAL "mid")
+ set(S8_TILE_M 64)
+ set(S8_TILE_N 128)
+ elseif(_s8_tile STREQUAL "medium")
+ set(S8_TILE_M 128)
+ set(S8_TILE_N 128)
+ elseif(_s8_path STREQUAL "normal")
+ set(S8_TILE_M 256)
+ set(S8_TILE_N 256)
+ else()
+ set(S8_TILE_M 256)
+ set(S8_TILE_N 128)
+ endif()
+
+ if(_s8_tile STREQUAL "small")
+ set(_s8_output_suffix small)
+ set(S8_LAYOUT SmallTileSG)
+ elseif(_s8_tile STREQUAL "mid")
+ set(_s8_output_suffix mid)
+ set(S8_LAYOUT SmallMidTileSG)
+ elseif(_s8_tile STREQUAL "medium")
+ set(_s8_output_suffix medium)
+ set(S8_LAYOUT MediumTileSG)
+ else()
+ set(_s8_output_suffix large)
+ set(S8_LAYOUT LargeTileSG)
+ endif()
+
+ set(S8_FUNCTION_NAME run_${_s8_dtype}_${_s8_path}_${_s8_tile})
+ set(_s8_output sycl_tla_s8_gemm_${_s8_dtype}_${_s8_path}_${_s8_output_suffix}.cpp)
+ generate_sycl_tla_source(sycl_tla_gemm.cpp.in ${_s8_output})
+ endforeach()
+ endforeach()
+endforeach()
+
+set(MOE_SOURCE_MODE 8)
+set(MOE_DEFINE_FP8_HELPERS 0)
+set(MOE_DEFINE_INT8_HELPERS 0)
+set(MOE_DEFINE_S4_HELPERS 0)
+set(_moe_tiles small mid large)
+foreach(_moe_tile IN LISTS _moe_tiles)
+ if(_moe_tile STREQUAL "small")
+ set(MOE_POLICY_SUFFIX small)
+ set(MOE_POLICY_NAME dpas_w8a16_policy_m_16)
+ elseif(_moe_tile STREQUAL "mid")
+ set(MOE_POLICY_SUFFIX mid)
+ set(MOE_POLICY_NAME dpas_w8a16_policy_m_32)
+ else()
+ set(MOE_POLICY_SUFFIX large)
+ set(MOE_POLICY_NAME dpas_w8a16_policy)
+ endif()
+
+ foreach(_moe_dtype IN ITEMS f16 bf16)
+ if(_moe_dtype STREQUAL "f16")
+ set(MOE_SCALAR_TYPE sycl::half)
+ else()
+ set(MOE_SCALAR_TYPE sycl::ext::oneapi::bfloat16)
+ endif()
+
+ foreach(_moe_format IN ITEMS e4m3 e5m2)
+ if(_moe_format STREQUAL "e4m3")
+ set(MOE_FORMAT_ARGUMENT true)
+ else()
+ set(MOE_FORMAT_ARGUMENT false)
+ endif()
+ set(MOE_POLICY_ARGS "${MOE_FORMAT_ARGUMENT}, moe_dpas_fp8::${MOE_POLICY_NAME}")
+ set(MOE_FUNCTION_NAME dispatch_${_moe_dtype}_${_moe_format}_${MOE_POLICY_SUFFIX})
+ if(_moe_tile STREQUAL "small")
+ set(_moe_output_suffix "")
+ else()
+ set(_moe_output_suffix _${MOE_POLICY_SUFFIX})
+ endif()
+ set(_moe_output sycl_tla_moe_prefill_fp8_${_moe_dtype}_${_moe_format}${_moe_output_suffix}.cpp)
+ set(MOE_HELPER_HEADER sycl_tla_moe_prefill_fp8_helpers.hpp)
+ set(MOE_DPAS_HEADER sycl_tla_moe_prefill_fp8_dpas.hpp)
+ set(MOE_NAMESPACE moe_prefill_fp8_detail)
+ set(MOE_DPAS_NAMESPACE moe_dpas_fp8)
+ set(MOE_DISPATCH_FUNCTION moe_prefill_fp8_dpas_per_group_dispatch_policy)
+ set(MOE_WEIGHT_TYPE uint8_t)
+ set(MOE_DEFINE_FP8_HELPERS 0)
+ set(MOE_DEFINE_INT8_HELPERS 0)
+ set(MOE_DEFINE_S4_HELPERS 0)
+ if(_moe_tile STREQUAL "small" AND _moe_dtype STREQUAL "f16" AND _moe_format STREQUAL "e4m3")
+ set(MOE_DEFINE_FP8_HELPERS 1)
+ endif()
+ generate_sycl_tla_source(sycl_tla_moe.cpp.in ${_moe_output})
+ endforeach()
+
+ set(MOE_POLICY_ARGS "moe_dpas_int::${MOE_POLICY_NAME}")
+ set(MOE_FUNCTION_NAME dispatch_${_moe_dtype}_${MOE_POLICY_SUFFIX})
+ if(_moe_tile STREQUAL "small")
+ set(_moe_output_suffix "")
+ else()
+ set(_moe_output_suffix _${MOE_POLICY_SUFFIX})
+ endif()
+ set(_moe_output sycl_tla_moe_prefill_int8_${_moe_dtype}${_moe_output_suffix}.cpp)
+ set(MOE_HELPER_HEADER sycl_tla_moe_prefill_int8_helpers.hpp)
+ set(MOE_DPAS_HEADER sycl_tla_moe_prefill_int_dpas.hpp)
+ set(MOE_NAMESPACE moe_prefill_int8_detail)
+ set(MOE_DPAS_NAMESPACE moe_dpas_int)
+ set(MOE_DISPATCH_FUNCTION moe_prefill_int_dpas_per_group_dispatch_policy)
+ set(MOE_WEIGHT_TYPE int8_t)
+ set(MOE_DEFINE_FP8_HELPERS 0)
+ set(MOE_DEFINE_INT8_HELPERS 0)
+ set(MOE_DEFINE_S4_HELPERS 0)
+ if(_moe_tile STREQUAL "small" AND _moe_dtype STREQUAL "f16")
+ set(MOE_DEFINE_INT8_HELPERS 1)
+ endif()
+ generate_sycl_tla_source(sycl_tla_moe.cpp.in ${_moe_output})
+
+ set(MOE_POLICY_ARGS "moe_dpas_s4::${MOE_POLICY_NAME}")
+ set(MOE_FUNCTION_NAME dispatch_${_moe_dtype}_${MOE_POLICY_SUFFIX})
+ set(_moe_output sycl_tla_moe_prefill_s4_${_moe_dtype}${_moe_output_suffix}.cpp)
+ set(MOE_HELPER_HEADER sycl_tla_moe_prefill_s4_helpers.hpp)
+ set(MOE_DPAS_HEADER sycl_tla_moe_prefill_s4_dpas.hpp)
+ set(MOE_NAMESPACE moe_prefill_s4_detail)
+ set(MOE_DPAS_NAMESPACE moe_dpas_s4)
+ set(MOE_DISPATCH_FUNCTION moe_prefill_s4_dpas_per_group_dispatch_policy)
+ set(MOE_WEIGHT_TYPE uint8_t)
+ set(MOE_DEFINE_FP8_HELPERS 0)
+ set(MOE_DEFINE_INT8_HELPERS 0)
+ set(MOE_DEFINE_S4_HELPERS 0)
+ if(_moe_tile STREQUAL "small" AND _moe_dtype STREQUAL "f16")
+ set(MOE_DEFINE_S4_HELPERS 1)
+ endif()
+ generate_sycl_tla_source(sycl_tla_moe.cpp.in ${_moe_output})
+ endforeach()
+endforeach()
+
+set(MOE_SOURCE_MODE 9)
+foreach(_moe_dtype IN ITEMS f16 bf16)
+ if(_moe_dtype STREQUAL "f16")
+ set(MOE_SCALAR_TYPE sycl::half)
+ else()
+ set(MOE_SCALAR_TYPE sycl::ext::oneapi::bfloat16)
+ endif()
+ foreach(_moe_tile IN LISTS _moe_tiles)
+ if(_moe_tile STREQUAL "small")
+ set(MOE_POLICY_SUFFIX small)
+ set(MOE_POLICY_NAME dpas_w8a16_policy_m_16)
+ elseif(_moe_tile STREQUAL "mid")
+ set(MOE_POLICY_SUFFIX mid)
+ set(MOE_POLICY_NAME dpas_w8a16_policy_m_32)
+ else()
+ set(MOE_POLICY_SUFFIX large)
+ set(MOE_POLICY_NAME dpas_w8a16_policy)
+ endif()
+ set(MOE_POLICY_ARGS "moe_dpas_int::${MOE_POLICY_NAME}")
+ set(MOE_FUNCTION_NAME dispatch_${_moe_dtype}_${MOE_POLICY_SUFFIX})
+ set(_moe_output sycl_tla_moe_prefill_int_tensor_${_moe_dtype}_${MOE_POLICY_SUFFIX}.cpp)
+ set(MOE_HELPER_HEADER sycl_tla_moe_prefill_int_tensor_helpers.hpp)
+ set(MOE_DPAS_HEADER sycl_tla_moe_prefill_int_dpas.hpp)
+ set(MOE_NAMESPACE moe_prefill_int_tensor_detail)
+ set(MOE_DPAS_NAMESPACE moe_dpas_int)
+ set(MOE_DISPATCH_FUNCTION moe_prefill_int_dpas_per_tensor_dispatch_policy)
+ set(MOE_WEIGHT_TYPE int8_t)
+ set(MOE_SCALE_TYPE float)
+ generate_sycl_tla_source(sycl_tla_moe.cpp.in ${_moe_output})
+ endforeach()
+endforeach()
+
+set(DENSE_SOURCE_MODE 2)
+set(S8_SOURCE_MODE 0)
+generate_sycl_tla_source(sycl_tla_gemm.cpp.in sycl_tla_dense_gemm.cpp)
+foreach(_dense_dispatch_dtype IN ITEMS f16 bf16 f32)
+ set(DENSE_SOURCE_MODE 3)
+ set(DENSE_DISPATCH_DTYPE ${_dense_dispatch_dtype})
+ generate_sycl_tla_source(sycl_tla_gemm.cpp.in sycl_tla_dense_gemm_${_dense_dispatch_dtype}.cpp)
+endforeach()
+
+set(DENSE_SOURCE_MODE 0)
+set(S8_SOURCE_MODE 2)
+generate_sycl_tla_source(sycl_tla_gemm.cpp.in sycl_tla_s8_gemm.cpp)
+foreach(_s8_dispatch_dtype IN ITEMS f16 bf16 f32)
+ set(S8_SOURCE_MODE 3)
+ set(S8_DISPATCH_DTYPE ${_s8_dispatch_dtype})
+ if(_s8_dispatch_dtype STREQUAL "bf16")
+ set(S8_KBLOCK_SUPPORTED 0)
+ else()
+ set(S8_KBLOCK_SUPPORTED 1)
+ endif()
+ generate_sycl_tla_source(sycl_tla_gemm.cpp.in sycl_tla_s8_gemm_${_s8_dispatch_dtype}.cpp)
+endforeach()
+
+set(MOE_SOURCE_MODE 1)
+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe.cpp)
+foreach(_moe_dispatch_dtype IN ITEMS f16 bf16)
+ set(MOE_DISPATCH_DTYPE ${_moe_dispatch_dtype})
+ if(_moe_dispatch_dtype STREQUAL "f16")
+ set(MOE_DISPATCH_ELEMENT cutlass::half_t)
+ else()
+ set(MOE_DISPATCH_ELEMENT cutlass::bfloat16_t)
+ endif()
+ set(MOE_SOURCE_MODE 2)
+ generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_${_moe_dispatch_dtype}.cpp)
+endforeach()
+
+set(MOE_SOURCE_MODE 3)
+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_decode.cpp)
+set(MOE_SOURCE_MODE 4)
+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_decode_fp.cpp)
+set(MOE_SOURCE_MODE 5)
+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_decode_fp8.cpp)
+foreach(_decode_int_kind IN ITEMS int2 int4 int8)
+ set(DECODE_INT_KIND ${_decode_int_kind})
+ set(MOE_SOURCE_MODE 6)
+ generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_decode_${_decode_int_kind}.cpp)
+endforeach()
+
+set(MOE_SOURCE_MODE 7)
+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_mixed.cpp)
+set(MOE_SOURCE_MODE 11)
+foreach(_moe_dequant_dtype IN ITEMS f16 bf16)
+ set(MOE_DEQUANT_DTYPE ${_moe_dequant_dtype})
+ if(_moe_dequant_dtype STREQUAL "f16")
+ set(MOE_DEQUANT_SCALAR sycl::half)
+ else()
+ set(MOE_DEQUANT_SCALAR sycl::ext::oneapi::bfloat16)
+ endif()
+ generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_prefill_dequant_${_moe_dequant_dtype}.cpp)
+endforeach()
+set(MOE_SOURCE_MODE 12)
+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_prefill_fp8.cpp)
+set(MOE_SOURCE_MODE 13)
+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_prefill_fp8_tensor.cpp)
+set(MOE_SOURCE_MODE 14)
+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_prefill_int8.cpp)
+set(MOE_SOURCE_MODE 15)
+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_prefill_int_tensor.cpp)
+set(MOE_SOURCE_MODE 16)
+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_prefill_s4.cpp)
+set(MOE_SOURCE_MODE 17)
+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_prefill_native_fp8.cpp)
+
+foreach(_moe_dtype IN ITEMS f16 bf16)
+ if(_moe_dtype STREQUAL "f16")
+ set(MOE_SCALAR_TYPE sycl::half)
+ else()
+ set(MOE_SCALAR_TYPE sycl::ext::oneapi::bfloat16)
+ endif()
+ foreach(_moe_format IN ITEMS e4m3 e5m2)
+ if(_moe_format STREQUAL "e4m3")
+ set(MOE_FORMAT_ARGUMENT true)
+ else()
+ set(MOE_FORMAT_ARGUMENT false)
+ endif()
+ foreach(_moe_tile IN LISTS _moe_tiles)
+ if(_moe_tile STREQUAL "small")
+ set(MOE_POLICY_SUFFIX small)
+ set(MOE_POLICY_NAME dpas_w8a16_policy_m_16)
+ elseif(_moe_tile STREQUAL "mid")
+ set(MOE_POLICY_SUFFIX mid)
+ set(MOE_POLICY_NAME dpas_w8a16_policy_m_32)
+ else()
+ set(MOE_POLICY_SUFFIX large)
+ set(MOE_POLICY_NAME dpas_w8a16_policy)
+ endif()
+ set(MOE_POLICY_ARGS "${MOE_FORMAT_ARGUMENT}, moe_dpas_fp8::${MOE_POLICY_NAME}")
+ set(MOE_FUNCTION_NAME dispatch_${_moe_dtype}_${_moe_format}_${MOE_POLICY_SUFFIX})
+ set(_moe_output sycl_tla_moe_prefill_fp8_tensor_${_moe_dtype}_${_moe_format}_${MOE_POLICY_SUFFIX}.cpp)
+ set(MOE_HELPER_HEADER sycl_tla_moe_prefill_fp8_tensor_helpers.hpp)
+ set(MOE_DPAS_HEADER sycl_tla_moe_prefill_fp8_dpas.hpp)
+ set(MOE_NAMESPACE moe_prefill_fp8_tensor_detail)
+ set(MOE_DPAS_NAMESPACE moe_dpas_fp8)
+ set(MOE_DISPATCH_FUNCTION moe_prefill_fp8_dpas_per_tensor_dispatch_policy)
+ set(MOE_WEIGHT_TYPE uint8_t)
+ set(MOE_SCALE_TYPE float)
+ set(MOE_SOURCE_MODE 9)
+ generate_sycl_tla_source(sycl_tla_moe.cpp.in ${_moe_output})
+ endforeach()
+ endforeach()
+endforeach()
+
+foreach(_moe_dtype IN ITEMS f16 bf16)
+ if(_moe_dtype STREQUAL "f16")
+ set(MOE_SCALAR_TYPE sycl::half)
+ else()
+ set(MOE_SCALAR_TYPE sycl::ext::oneapi::bfloat16)
+ endif()
+ foreach(_moe_tile IN LISTS _moe_tiles)
+ if(_moe_tile STREQUAL "small")
+ set(MOE_POLICY_SUFFIX small)
+ set(MOE_POLICY_NAME dpas_w8a16_policy_m_16)
+ elseif(_moe_tile STREQUAL "mid")
+ set(MOE_POLICY_SUFFIX mid)
+ set(MOE_POLICY_NAME dpas_w8a16_policy_m_32)
+ else()
+ set(MOE_POLICY_SUFFIX large)
+ set(MOE_POLICY_NAME dpas_w8a16_policy)
+ endif()
+ set(MOE_POLICY_ARGS "moe_dpas_int::${MOE_POLICY_NAME}")
+ set(MOE_FUNCTION_NAME dispatch_lowbit_${_moe_dtype}_${MOE_POLICY_SUFFIX})
+ if(_moe_tile STREQUAL "small")
+ set(_moe_output_suffix "")
+ else()
+ set(_moe_output_suffix _${MOE_POLICY_SUFFIX})
+ endif()
+ set(_moe_output sycl_tla_moe_prefill_int8_lowbit_${_moe_dtype}${_moe_output_suffix}.cpp)
+ set(MOE_HELPER_HEADER sycl_tla_moe_prefill_int8_helpers.hpp)
+ set(MOE_DPAS_HEADER sycl_tla_moe_prefill_int_dpas.hpp)
+ set(MOE_NAMESPACE moe_prefill_int8_detail)
+ set(MOE_DPAS_NAMESPACE moe_dpas_int)
+ set(MOE_DISPATCH_FUNCTION moe_prefill_int_dpas_per_group_dispatch_policy)
+ set(MOE_SOURCE_MODE 10)
+ generate_sycl_tla_source(sycl_tla_moe.cpp.in ${_moe_output})
+ endforeach()
+endforeach()
\ No newline at end of file
diff --git a/auto_round_extension/ark/auto_round_kernel/sycl_tla_moe.cpp.in b/auto_round_extension/ark/auto_round_kernel/sycl_tla_moe.cpp.in
new file mode 100644
index 000000000..bc81a9714
--- /dev/null
+++ b/auto_round_extension/ark/auto_round_kernel/sycl_tla_moe.cpp.in
@@ -0,0 +1,701 @@
+// Generated by CMake. Do not edit directly.
+
+#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)
+
+#if @MOE_SOURCE_MODE@ == 1
+
+#include "sycl_tla_common.hpp"
+
+namespace ark {
+
+void moe_gemm(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs, BTLA_DTYPE dtype,
+ int N, int K, int* num_tokens_per_expert, int num_experts) {
+ switch (dtype) {
+ case BTLA_DTYPE::F16:
+ sycl_tla_moe_gemm_f16(q, activations, weights, scales, outputs, N, K, num_tokens_per_expert, num_experts);
+ return;
+ case BTLA_DTYPE::BF16:
+ sycl_tla_moe_gemm_bf16(q, activations, weights, scales, outputs, N, K, num_tokens_per_expert, num_experts);
+ return;
+ default:
+ throw std::runtime_error("moe_gemm: unsupported dtype, only BF16/FP16 supported");
+ }
+}
+
+} // namespace ark
+
+#elif @MOE_SOURCE_MODE@ == 2
+
+#include "sycl_tla_common.hpp"
+#include "sycl_tla_moe.hpp"
+
+namespace ark {
+
+void sycl_tla_moe_gemm_@MOE_DISPATCH_DTYPE@(sycl::queue* q, void* activations, void* weights, void* scales,
+ void* outputs, int N, int K, int* num_tokens_per_expert,
+ int num_experts) {
+ using Element = @MOE_DISPATCH_ELEMENT@;
+ moe_detail::moe_gemm_launcher<'R', 'R', Element, Element, Element, Element>(
+ q, static_cast<const Element*>(activations), static_cast<const Element*>(weights),
+ static_cast<const Element*>(scales), static_cast<Element*>(outputs), N, K, num_tokens_per_expert, num_experts);
+}
+
+} // namespace ark
+
+#elif @MOE_SOURCE_MODE@ == 3
+
+#include "sycl_tla_common.hpp"
+
+namespace ark {
+
+void sycl_tla_moe_decode_fill_expert_id(sycl::queue* q, int* expert_id_per_token,
+ const int* num_tokens_per_expert, int num_experts, int total_tokens) {
+ if (total_tokens == 0) return;
+ q->parallel_for(sycl::range<1>(static_cast<size_t>(total_tokens)), [=](sycl::id<1> idx) {
+ const int token = static_cast<int>(idx[0]);
+ int offset = 0;
+ int expert = num_experts - 1;
+ for (int expert_id = 0; expert_id < num_experts; ++expert_id) {
+ const int token_count = num_tokens_per_expert[expert_id];
+ if (token < offset + token_count) {
+ expert = expert_id;
+ break;
+ }
+ offset += token_count;
+ }
+ expert_id_per_token[token] = expert;
+ });
+}
+
+void moe_gemm_decode(sycl::queue* q, void* activations, void* weights, void* scales, void* zeros, void* outputs,
+ int* expert_id_per_token_buf, BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,
+ int group_size, int* num_tokens_per_expert, int num_experts, int total_tokens, bool asym) {
+ sycl_tla_moe_decode_fill_expert_id(q, expert_id_per_token_buf, num_tokens_per_expert, num_experts, total_tokens);
+
+ MoeDecodeParams params{q, activations, weights, scales, zeros, outputs, expert_id_per_token_buf, act_dtype,
+ weight_dtype, N, K, group_size, total_tokens, asym};
+
+ if (weight_dtype == BTLA_DTYPE::F16 || weight_dtype == BTLA_DTYPE::BF16) {
+ if (weight_dtype != act_dtype) {
+ throw std::invalid_argument("moe_gemm_decode: unquantized weight_dtype must match act_dtype");
+ }
+ sycl_tla_moe_decode_fp(params);
+ return;
+ }
+ if (weight_dtype == BTLA_DTYPE::S4_CLIP) {
+ sycl_tla_moe_decode_int4(params);
+ return;
+ }
+ if (weight_dtype == BTLA_DTYPE::S8) {
+ sycl_tla_moe_decode_int8(params);
+ return;
+ }
+ if (weight_dtype == BTLA_DTYPE::S2_CLIP) {
+ sycl_tla_moe_decode_int2(params);
+ return;
+ }
+ if (weight_dtype == BTLA_DTYPE::F8_E4M3 || weight_dtype == BTLA_DTYPE::F8_E5M2) {
+ if (asym) {
+ throw std::invalid_argument("moe_gemm_decode(fp8): asym mode is not supported");
+ }
+ sycl_tla_moe_decode_fp8(params);
+ return;
+ }
+ throw std::invalid_argument(
+ "moe_gemm_decode: unsupported weight_dtype (supported: F16, BF16, S8, S4_CLIP, S2_CLIP, F8_E4M3, F8_E5M2)");
+}
+
+} // namespace ark
+
+#elif @MOE_SOURCE_MODE@ == 4
+
+#include "sycl_tla_common.hpp"
+#include "sycl_tla_moe_decode.hpp"
+
+namespace ark {
+
+void sycl_tla_moe_decode_fp(const MoeDecodeParams& params) {
+ if (params.act_dtype == BTLA_DTYPE::F16) {
+ moe_decode_detail::launch_fp<sycl::half>(
+ params.q, static_cast<const sycl::half*>(params.activations), static_cast<const sycl::half*>(params.weights),
+ static_cast<sycl::half*>(params.outputs), params.expert_id_per_token, params.total_tokens, params.N,
+ params.K);
+ return;
+ }
+ if (params.act_dtype == BTLA_DTYPE::BF16) {
+ using BF = sycl::ext::oneapi::bfloat16;
+ moe_decode_detail::launch_fp<BF>(
+ params.q, static_cast<const BF*>(params.activations), static_cast<const BF*>(params.weights),
+ static_cast<BF*>(params.outputs), params.expert_id_per_token, params.total_tokens, params.N, params.K);
+ return;
+ }
+ throw std::invalid_argument("moe_gemm_decode(fp): act_dtype must be FP16 or BF16");
+}
+
+} // namespace ark
+
+#elif @MOE_SOURCE_MODE@ == 5
+
+#include "sycl_tla_common.hpp"
+#include "sycl_tla_moe_decode.hpp"
+
+namespace ark {
+
+void sycl_tla_moe_decode_fp8(const MoeDecodeParams& params) {
+ const bool is_e4m3 = params.weight_dtype == BTLA_DTYPE::F8_E4M3;
+ const bool use_lut = moe_decode_detail::fp8_decode_use_lut();
+ if (params.act_dtype == BTLA_DTYPE::F16) {
+ if (is_e4m3) {
+ if (use_lut) {
+ moe_decode_detail::launch_fp8<sycl::half, true, true>(
+ params.q, static_cast<const sycl::half*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const sycl::half*>(params.scales), static_cast<sycl::half*>(params.outputs),
+ params.expert_id_per_token, params.total_tokens, params.N, params.K, params.group_size);
+ } else {
+ moe_decode_detail::launch_fp8<sycl::half, true, false>(
+ params.q, static_cast<const sycl::half*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const sycl::half*>(params.scales), static_cast<sycl::half*>(params.outputs),
+ params.expert_id_per_token, params.total_tokens, params.N, params.K, params.group_size);
+ }
+ } else if (use_lut) {
+ moe_decode_detail::launch_fp8<sycl::half, false, true>(
+ params.q, static_cast<const sycl::half*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const sycl::half*>(params.scales), static_cast<sycl::half*>(params.outputs),
+ params.expert_id_per_token, params.total_tokens, params.N, params.K, params.group_size);
+ } else {
+ moe_decode_detail::launch_fp8<sycl::half, false, false>(
+ params.q, static_cast<const sycl::half*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const sycl::half*>(params.scales), static_cast<sycl::half*>(params.outputs),
+ params.expert_id_per_token, params.total_tokens, params.N, params.K, params.group_size);
+ }
+ return;
+ }
+ if (params.act_dtype == BTLA_DTYPE::BF16) {
+ using BF = sycl::ext::oneapi::bfloat16;
+ if (is_e4m3) {
+ if (use_lut) {
+ moe_decode_detail::launch_fp8<BF, true, true>(
+ params.q, static_cast<const BF*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const BF*>(params.scales), static_cast<BF*>(params.outputs), params.expert_id_per_token,
+ params.total_tokens, params.N, params.K, params.group_size);
+ } else {
+ moe_decode_detail::launch_fp8<BF, true, false>(
+ params.q, static_cast<const BF*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const BF*>(params.scales), static_cast<BF*>(params.outputs), params.expert_id_per_token,
+ params.total_tokens, params.N, params.K, params.group_size);
+ }
+ } else if (use_lut) {
+ moe_decode_detail::launch_fp8<BF, false, true>(
+ params.q, static_cast<const BF*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const BF*>(params.scales), static_cast<BF*>(params.outputs), params.expert_id_per_token,
+ params.total_tokens, params.N, params.K, params.group_size);
+ } else {
+ moe_decode_detail::launch_fp8<BF, false, false>(
+ params.q, static_cast<const BF*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const BF*>(params.scales), static_cast<BF*>(params.outputs), params.expert_id_per_token,
+ params.total_tokens, params.N, params.K, params.group_size);
+ }
+ return;
+ }
+ throw std::invalid_argument("moe_gemm_decode(fp8): act_dtype must be FP16 or BF16");
+}
+
+} // namespace ark
+
+#elif @MOE_SOURCE_MODE@ == 6
+
+#include "sycl_tla_common.hpp"
+#include "sycl_tla_moe_decode.hpp"
+
+namespace ark {
+
+void sycl_tla_moe_decode_@DECODE_INT_KIND@(const MoeDecodeParams& params) {
+ if (params.act_dtype == BTLA_DTYPE::F16) {
+ if (params.asym) {
+ moe_decode_detail::launch_@DECODE_INT_KIND@<sycl::half, true>(
+ params.q, static_cast<const sycl::half*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const sycl::half*>(params.scales), static_cast<const sycl::half*>(params.zeros),
+ static_cast<sycl::half*>(params.outputs), params.expert_id_per_token, params.total_tokens, params.N,
+ params.K, params.group_size);
+ } else {
+ moe_decode_detail::launch_@DECODE_INT_KIND@<sycl::half, false>(
+ params.q, static_cast<const sycl::half*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const sycl::half*>(params.scales), static_cast<const sycl::half*>(params.zeros),
+ static_cast<sycl::half*>(params.outputs), params.expert_id_per_token, params.total_tokens, params.N,
+ params.K, params.group_size);
+ }
+ return;
+ }
+ if (params.act_dtype == BTLA_DTYPE::BF16) {
+ using BF = sycl::ext::oneapi::bfloat16;
+ if (params.asym) {
+ moe_decode_detail::launch_@DECODE_INT_KIND@<BF, true>(
+ params.q, static_cast<const BF*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const BF*>(params.scales), static_cast<const BF*>(params.zeros), static_cast<BF*>(params.outputs),
+ params.expert_id_per_token, params.total_tokens, params.N, params.K, params.group_size);
+ } else {
+ moe_decode_detail::launch_@DECODE_INT_KIND@<BF, false>(
+ params.q, static_cast<const BF*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const BF*>(params.scales), static_cast<const BF*>(params.zeros), static_cast<BF*>(params.outputs),
+ params.expert_id_per_token, params.total_tokens, params.N, params.K, params.group_size);
+ }
+ return;
+ }
+ throw std::invalid_argument("moe_gemm_decode(@DECODE_INT_KIND@): act_dtype must be FP16 or BF16");
+}
+
+} // namespace ark
+
+#elif @MOE_SOURCE_MODE@ == 7
+
+#include "sycl_tla_common.hpp"
+
+namespace ark {
+
+void moe_gemm_prefill(sycl::queue* q, void* activations, void* weights, void* scales, void* zeros, void* outputs,
+ void* dequant_workspace, BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,
+ int group_size, int* num_tokens_per_expert, int num_experts, int total_tokens, bool asym) {
+ if (total_tokens == 0) return;
+
+ if (weight_dtype == act_dtype && weight_dtype == BTLA_DTYPE::F16) {
+ sycl_tla_moe_gemm_f16(q, activations, weights, scales, outputs, N, K, num_tokens_per_expert, num_experts);
+ return;
+ }
+ if (weight_dtype == act_dtype && weight_dtype == BTLA_DTYPE::BF16) {
+ sycl_tla_moe_gemm_bf16(q, activations, weights, scales, outputs, N, K, num_tokens_per_expert, num_experts);
+ return;
+ }
+
+ MoePrefillParams params{q, activations, weights, scales, zeros, outputs, dequant_workspace, act_dtype, weight_dtype,
+ N, K, group_size, num_tokens_per_expert, num_experts, total_tokens, asym};
+
+ if (sycl_tla_moe_prefill_int8_dpas(params)) return;
+ if (sycl_tla_moe_prefill_s4_dpas(params)) return;
+ if (sycl_tla_moe_prefill_lowbit_int8_dpas(params)) return;
+ if (sycl_tla_moe_prefill_fp8_dpas(params)) return;
+ if (sycl_tla_moe_prefill_native_fp8(params)) return;
+
+ if (dequant_workspace == nullptr) {
+ throw std::invalid_argument("moe_gemm_prefill: dequant_workspace must be non-null for quantized paths");
+ }
+ if (act_dtype == BTLA_DTYPE::F16) {
+ sycl_tla_moe_prefill_dequant_f16(params);
+ return;
+ }
+ if (act_dtype == BTLA_DTYPE::BF16) {
+ sycl_tla_moe_prefill_dequant_bf16(params);
+ return;
+ }
+ throw std::invalid_argument("moe_gemm_prefill: act_dtype must be F16 or BF16");
+}
+
+} // namespace ark
+
+#elif @MOE_SOURCE_MODE@ == 8
+
+#include "@MOE_HELPER_HEADER@"
+#include "@MOE_DPAS_HEADER@"
+
+namespace ark {
+namespace @MOE_NAMESPACE@ {
+
+#if @MOE_DEFINE_FP8_HELPERS@
+bool enabled() { return moe_dpas_fp8::moe_prefill_dpas_fp8_enabled(); }
+bool shape_ok(int N, int K, int group_size) {
+ return moe_dpas_fp8::moe_prefill_dpas_fp8_pergroup_shape_ok(N, K, group_size);
+}
+#endif
+#if @MOE_DEFINE_INT8_HELPERS@
+bool enabled() { return moe_dpas_int::moe_prefill_dpas_int_enabled(); }
+bool shape_ok(int N, int K, int group_size) {
+ return moe_dpas_int::moe_prefill_dpas_int_pergroup_shape_ok(N, K, group_size);
+}
+#endif
+#if @MOE_DEFINE_S4_HELPERS@
+bool enabled() { return moe_dpas_s4::moe_prefill_dpas_s4_enabled(); }
+bool shape_ok(int N, int K, int group_size) {
+ return moe_dpas_s4::moe_prefill_dpas_s4_pergroup_shape_ok(N, K, group_size);
+}
+#endif
+
+void @MOE_FUNCTION_NAME@(const MoePrefillParams& params) {
+ using ScalarT = @MOE_SCALAR_TYPE@;
+ @MOE_DPAS_NAMESPACE@::@MOE_DISPATCH_FUNCTION@<ScalarT, @MOE_POLICY_ARGS@>(
+ params.q, static_cast<const ScalarT*>(params.activations),
+ static_cast<const @MOE_WEIGHT_TYPE@*>(params.weights), static_cast<const ScalarT*>(params.scales),
+ static_cast<ScalarT*>(params.outputs), params.num_tokens_per_expert, params.num_experts, params.N, params.K,
+ params.group_size, params.total_tokens);
+}
+
+} // namespace @MOE_NAMESPACE@
+} // namespace ark
+
+#elif @MOE_SOURCE_MODE@ == 9
+
+#include "@MOE_HELPER_HEADER@"
+#include "@MOE_DPAS_HEADER@"
+
+namespace ark {
+namespace @MOE_NAMESPACE@ {
+
+void @MOE_FUNCTION_NAME@(const MoePrefillParams& params) {
+ using ScalarT = @MOE_SCALAR_TYPE@;
+ @MOE_DPAS_NAMESPACE@::@MOE_DISPATCH_FUNCTION@<ScalarT, @MOE_POLICY_ARGS@>(
+ params.q, static_cast<const ScalarT*>(params.activations),
+ static_cast<const @MOE_WEIGHT_TYPE@*>(params.weights), static_cast<const @MOE_SCALE_TYPE@*>(params.scales),
+ static_cast<ScalarT*>(params.outputs), params.num_tokens_per_expert, params.num_experts, params.N, params.K,
+ params.total_tokens);
+}
+
+} // namespace @MOE_NAMESPACE@
+} // namespace ark
+
+#elif @MOE_SOURCE_MODE@ == 10
+
+#include "@MOE_HELPER_HEADER@"
+#include "@MOE_DPAS_HEADER@"
+#include "sycl_tla_moe_prefill_upcast.hpp"
+
+namespace ark {
+namespace @MOE_NAMESPACE@ {
+
+void @MOE_FUNCTION_NAME@(const MoePrefillParams& params) {
+ using ScalarT = @MOE_SCALAR_TYPE@;
+ auto* upcast_i8 = static_cast<int8_t*>(params.dequant_workspace);
+ if (params.weight_dtype == BTLA_DTYPE::S4_CLIP) {
+ moe_mixed_detail::launch_upcast_int4_sym_to_int8(
+ params.q, static_cast<const uint8_t*>(params.weights), upcast_i8, params.num_experts, params.N, params.K,
+ params.num_tokens_per_expert);
+ } else {
+ moe_mixed_detail::launch_upcast_int2_sym_to_int8(
+ params.q, static_cast<const uint8_t*>(params.weights), upcast_i8, params.num_experts, params.N, params.K,
+ params.num_tokens_per_expert);
+ }
+ @MOE_DPAS_NAMESPACE@::@MOE_DISPATCH_FUNCTION@<ScalarT, @MOE_POLICY_ARGS@>(
+ params.q, static_cast<const ScalarT*>(params.activations), upcast_i8, static_cast<const ScalarT*>(params.scales),
+ static_cast<ScalarT*>(params.outputs), params.num_tokens_per_expert, params.num_experts, params.N, params.K,
+ params.group_size, params.total_tokens);
+}
+
+} // namespace @MOE_NAMESPACE@
+} // namespace ark
+
+#elif @MOE_SOURCE_MODE@ == 11
+
+#include "sycl_tla_common.hpp"
+#include "sycl_tla_moe_mixed.hpp"
+
+namespace ark {
+
+void sycl_tla_moe_prefill_dequant_@MOE_DEQUANT_DTYPE@(const MoePrefillParams& params) {
+ using ScalarT = @MOE_DEQUANT_SCALAR@;
+ auto* weights_kn = static_cast<ScalarT*>(params.dequant_workspace);
+ moe_mixed_detail::dequant_to_KN<ScalarT>(
+ params.q, params.weights, params.scales, params.zeros, weights_kn, params.weight_dtype, params.num_experts,
+ params.N, params.K, params.group_size, params.asym, params.num_tokens_per_expert);
+ moe_gemm(params.q, params.activations, weights_kn, nullptr, params.outputs, params.act_dtype, params.N, params.K,
+ params.num_tokens_per_expert, params.num_experts);
+}
+
+} // namespace ark
+
+#elif @MOE_SOURCE_MODE@ == 12
+
+#include "sycl_tla_common.hpp"
+#include "sycl_tla_moe_prefill_fp8_helpers.hpp"
+
+namespace ark {
+
+bool sycl_tla_moe_prefill_fp8_dpas(const MoePrefillParams& params) {
+ if ((params.weight_dtype != BTLA_DTYPE::F8_E4M3 && params.weight_dtype != BTLA_DTYPE::F8_E5M2) || params.asym ||
+ !moe_prefill_fp8_detail::enabled() ||
+ !moe_prefill_fp8_detail::shape_ok(params.N, params.K, params.group_size)) {
+ return false;
+ }
+ const bool is_e4m3 = params.weight_dtype == BTLA_DTYPE::F8_E4M3;
+ const int average_tokens = params.num_experts > 0 ? params.total_tokens / params.num_experts : 0;
+ if (params.act_dtype == BTLA_DTYPE::F16) {
+ if (is_e4m3) {
+ if (average_tokens <= 8) moe_prefill_fp8_detail::dispatch_f16_e4m3_small(params);
+ else if (average_tokens <= 32) moe_prefill_fp8_detail::dispatch_f16_e4m3_mid(params);
+ else moe_prefill_fp8_detail::dispatch_f16_e4m3_large(params);
+ } else {
+ if (average_tokens <= 8) moe_prefill_fp8_detail::dispatch_f16_e5m2_small(params);
+ else if (average_tokens <= 32) moe_prefill_fp8_detail::dispatch_f16_e5m2_mid(params);
+ else moe_prefill_fp8_detail::dispatch_f16_e5m2_large(params);
+ }
+ return true;
+ }
+ if (params.act_dtype == BTLA_DTYPE::BF16) {
+ if (is_e4m3) {
+ if (average_tokens <= 8) moe_prefill_fp8_detail::dispatch_bf16_e4m3_small(params);
+ else if (average_tokens <= 32) moe_prefill_fp8_detail::dispatch_bf16_e4m3_mid(params);
+ else moe_prefill_fp8_detail::dispatch_bf16_e4m3_large(params);
+ } else {
+ if (average_tokens <= 8) moe_prefill_fp8_detail::dispatch_bf16_e5m2_small(params);
+ else if (average_tokens <= 32) moe_prefill_fp8_detail::dispatch_bf16_e5m2_mid(params);
+ else moe_prefill_fp8_detail::dispatch_bf16_e5m2_large(params);
+ }
+ return true;
+ }
+ return false;
+}
+
+} // namespace ark
+
+#elif @MOE_SOURCE_MODE@ == 13
+
+#include "sycl_tla_common.hpp"
+#include "sycl_tla_moe_prefill_fp8_tensor_helpers.hpp"
+
+namespace ark {
+
+void moe_gemm_prefill_fp8_dpas(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs,
+ BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,
+ int* num_tokens_per_expert, int num_experts, int total_tokens) {
+ if (total_tokens == 0) return;
+ if (weight_dtype != BTLA_DTYPE::F8_E4M3 && weight_dtype != BTLA_DTYPE::F8_E5M2) {
+ throw std::invalid_argument("moe_gemm_prefill_fp8_dpas: weight_dtype must be F8_E4M3 or F8_E5M2");
+ }
+ MoePrefillParams params{q, activations, weights, scales, nullptr, outputs, nullptr, act_dtype, weight_dtype, N, K,
+ 0, num_tokens_per_expert, num_experts, total_tokens, false};
+ const int average_tokens = num_experts > 0 ? total_tokens / num_experts : 0;
+ const bool is_e4m3 = weight_dtype == BTLA_DTYPE::F8_E4M3;
+ if (act_dtype == BTLA_DTYPE::F16) {
+ if (is_e4m3) {
+ if (average_tokens <= 8) moe_prefill_fp8_tensor_detail::dispatch_f16_e4m3_small(params);
+ else if (average_tokens <= 32) moe_prefill_fp8_tensor_detail::dispatch_f16_e4m3_mid(params);
+ else moe_prefill_fp8_tensor_detail::dispatch_f16_e4m3_large(params);
+ } else {
+ if (average_tokens <= 8) moe_prefill_fp8_tensor_detail::dispatch_f16_e5m2_small(params);
+ else if (average_tokens <= 32) moe_prefill_fp8_tensor_detail::dispatch_f16_e5m2_mid(params);
+ else moe_prefill_fp8_tensor_detail::dispatch_f16_e5m2_large(params);
+ }
+ return;
+ }
+ if (act_dtype == BTLA_DTYPE::BF16) {
+ if (is_e4m3) {
+ if (average_tokens <= 8) moe_prefill_fp8_tensor_detail::dispatch_bf16_e4m3_small(params);
+ else if (average_tokens <= 32) moe_prefill_fp8_tensor_detail::dispatch_bf16_e4m3_mid(params);
+ else moe_prefill_fp8_tensor_detail::dispatch_bf16_e4m3_large(params);
+ } else {
+ if (average_tokens <= 8) moe_prefill_fp8_tensor_detail::dispatch_bf16_e5m2_small(params);
+ else if (average_tokens <= 32) moe_prefill_fp8_tensor_detail::dispatch_bf16_e5m2_mid(params);
+ else moe_prefill_fp8_tensor_detail::dispatch_bf16_e5m2_large(params);
+ }
+ return;
+ }
+ throw std::invalid_argument("moe_gemm_prefill_fp8_dpas: act_dtype must be F16 or BF16");
+}
+
+} // namespace ark
+
+#elif @MOE_SOURCE_MODE@ == 14
+
+#include "sycl_tla_common.hpp"
+#include "sycl_tla_moe_prefill_int8_helpers.hpp"
+
+namespace ark {
+
+bool sycl_tla_moe_prefill_int8_dpas(const MoePrefillParams& params) {
+ if (params.weight_dtype == BTLA_DTYPE::S8 && !params.asym &&
+ moe_prefill_int8_detail::enabled() &&
+ moe_prefill_int8_detail::shape_ok(params.N, params.K, params.group_size)) {
+ const int average_tokens = params.num_experts > 0 ? params.total_tokens / params.num_experts : 0;
+ if (params.act_dtype == BTLA_DTYPE::F16) {
+ if (average_tokens <= 8) moe_prefill_int8_detail::dispatch_f16_small(params);
+ else if (average_tokens <= 32) moe_prefill_int8_detail::dispatch_f16_mid(params);
+ else moe_prefill_int8_detail::dispatch_f16_large(params);
+ return true;
+ }
+ if (params.act_dtype == BTLA_DTYPE::BF16) {
+ if (average_tokens <= 8) moe_prefill_int8_detail::dispatch_bf16_small(params);
+ else if (average_tokens <= 32) moe_prefill_int8_detail::dispatch_bf16_mid(params);
+ else moe_prefill_int8_detail::dispatch_bf16_large(params);
+ return true;
+ }
+ }
+ return false;
+}
+
+bool sycl_tla_moe_prefill_lowbit_int8_dpas(const MoePrefillParams& params) {
+ if ((params.weight_dtype != BTLA_DTYPE::S4_CLIP && params.weight_dtype != BTLA_DTYPE::S2_CLIP) || params.asym ||
+ !moe_prefill_int8_detail::enabled() ||
+ !moe_prefill_int8_detail::shape_ok(params.N, params.K, params.group_size) ||
+ params.dequant_workspace == nullptr ||
+ (params.act_dtype != BTLA_DTYPE::F16 && params.act_dtype != BTLA_DTYPE::BF16)) {
+ return false;
+ }
+ if (params.act_dtype == BTLA_DTYPE::F16) {
+ const int average_tokens = params.num_experts > 0 ? params.total_tokens / params.num_experts : 0;
+ if (average_tokens <= 8) moe_prefill_int8_detail::dispatch_lowbit_f16_small(params);
+ else if (average_tokens <= 32) moe_prefill_int8_detail::dispatch_lowbit_f16_mid(params);
+ else moe_prefill_int8_detail::dispatch_lowbit_f16_large(params);
+ } else {
+ const int average_tokens = params.num_experts > 0 ? params.total_tokens / params.num_experts : 0;
+ if (average_tokens <= 8) moe_prefill_int8_detail::dispatch_lowbit_bf16_small(params);
+ else if (average_tokens <= 32) moe_prefill_int8_detail::dispatch_lowbit_bf16_mid(params);
+ else moe_prefill_int8_detail::dispatch_lowbit_bf16_large(params);
+ }
+ return true;
+}
+
+} // namespace ark
+
+#elif @MOE_SOURCE_MODE@ == 15
+
+#include "sycl_tla_common.hpp"
+#include "sycl_tla_moe_prefill_int_tensor_helpers.hpp"
+
+namespace ark {
+
+void moe_gemm_prefill_int_dpas(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs,
+ BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,
+ int* num_tokens_per_expert, int num_experts, int total_tokens) {
+ if (total_tokens == 0) return;
+ if (weight_dtype != BTLA_DTYPE::S8) {
+ throw std::invalid_argument("moe_gemm_prefill_int_dpas: weight_dtype must be S8");
+ }
+ MoePrefillParams params{q, activations, weights, scales, nullptr, outputs, nullptr, act_dtype, weight_dtype, N, K,
+ 0, num_tokens_per_expert, num_experts, total_tokens, false};
+ const int average_tokens = num_experts > 0 ? total_tokens / num_experts : 0;
+ if (act_dtype == BTLA_DTYPE::F16) {
+ if (average_tokens <= 8) moe_prefill_int_tensor_detail::dispatch_f16_small(params);
+ else if (average_tokens <= 32) moe_prefill_int_tensor_detail::dispatch_f16_mid(params);
+ else moe_prefill_int_tensor_detail::dispatch_f16_large(params);
+ return;
+ }
+ if (act_dtype == BTLA_DTYPE::BF16) {
+ if (average_tokens <= 8) moe_prefill_int_tensor_detail::dispatch_bf16_small(params);
+ else if (average_tokens <= 32) moe_prefill_int_tensor_detail::dispatch_bf16_mid(params);
+ else moe_prefill_int_tensor_detail::dispatch_bf16_large(params);
+ return;
+ }
+ throw std::invalid_argument("moe_gemm_prefill_int_dpas: act_dtype must be F16 or BF16");
+}
+
+} // namespace ark
+
+#elif @MOE_SOURCE_MODE@ == 16
+
+#include "sycl_tla_common.hpp"
+#include "sycl_tla_moe_prefill_s4_helpers.hpp"
+
+namespace ark {
+
+bool sycl_tla_moe_prefill_s4_dpas(const MoePrefillParams& params) {
+ if (params.weight_dtype != BTLA_DTYPE::S4_CLIP || params.asym ||
+ !moe_prefill_s4_detail::enabled() ||
+ !moe_prefill_s4_detail::shape_ok(params.N, params.K, params.group_size) ||
+ (params.act_dtype != BTLA_DTYPE::F16 && params.act_dtype != BTLA_DTYPE::BF16)) {
+ return false;
+ }
+ if (params.act_dtype == BTLA_DTYPE::F16) {
+ const int average_tokens = params.num_experts > 0 ? params.total_tokens / params.num_experts : 0;
+ if (average_tokens <= 8) {
+ moe_prefill_s4_detail::dispatch_f16_small(params);
+ } else if (average_tokens <= 32) {
+ moe_prefill_s4_detail::dispatch_f16_mid(params);
+ } else {
+ moe_prefill_s4_detail::dispatch_f16_large(params);
+ }
+ } else {
+ const int average_tokens = params.num_experts > 0 ? params.total_tokens / params.num_experts : 0;
+ if (average_tokens <= 8) {
+ moe_prefill_s4_detail::dispatch_bf16_small(params);
+ } else if (average_tokens <= 32) {
+ moe_prefill_s4_detail::dispatch_bf16_mid(params);
+ } else {
+ moe_prefill_s4_detail::dispatch_bf16_large(params);
+ }
+ }
+ return true;
+}
+
+} // namespace ark
+
+#elif @MOE_SOURCE_MODE@ == 17
+
+#include "sycl_tla_common.hpp"
+#include "sycl_tla_moe_prefill_fp8_native.hpp"
+
+namespace ark {
+
+bool sycl_tla_moe_prefill_native_fp8(const MoePrefillParams& params) {
+ if ((params.weight_dtype != BTLA_DTYPE::F8_E4M3 && params.weight_dtype != BTLA_DTYPE::F8_E5M2) || params.asym ||
+ !moe_prefill_native_fp8_detail::moe_prefill_native_fp8_enabled() ||
+ !moe_prefill_native_fp8_detail::moe_prefill_native_fp8_shape_ok(params.N, params.K, params.group_size)) {
+ return false;
+ }
+ const bool is_e4m3 = params.weight_dtype == BTLA_DTYPE::F8_E4M3;
+ const bool use_lut = moe_dequant::fp8_decode_use_lut();
+ if (params.act_dtype == BTLA_DTYPE::F16) {
+ using ScalarT = sycl::half;
+ if (is_e4m3) {
+ if (use_lut) {
+ moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, true, true>(
+ params.q, static_cast<const ScalarT*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const ScalarT*>(params.scales), static_cast<ScalarT*>(params.outputs),
+ params.num_tokens_per_expert, params.num_experts, params.N, params.K, params.group_size,
+ params.total_tokens);
+ } else {
+ moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, true, false>(
+ params.q, static_cast<const ScalarT*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const ScalarT*>(params.scales), static_cast<ScalarT*>(params.outputs),
+ params.num_tokens_per_expert, params.num_experts, params.N, params.K, params.group_size,
+ params.total_tokens);
+ }
+ } else if (use_lut) {
+ moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, false, true>(
+ params.q, static_cast<const ScalarT*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const ScalarT*>(params.scales), static_cast<ScalarT*>(params.outputs),
+ params.num_tokens_per_expert, params.num_experts, params.N, params.K, params.group_size,
+ params.total_tokens);
+ } else {
+ moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, false, false>(
+ params.q, static_cast<const ScalarT*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const ScalarT*>(params.scales), static_cast<ScalarT*>(params.outputs),
+ params.num_tokens_per_expert, params.num_experts, params.N, params.K, params.group_size,
+ params.total_tokens);
+ }
+ return true;
+ }
+ if (params.act_dtype == BTLA_DTYPE::BF16) {
+ using ScalarT = sycl::ext::oneapi::bfloat16;
+ if (is_e4m3) {
+ if (use_lut) {
+ moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, true, true>(
+ params.q, static_cast<const ScalarT*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const ScalarT*>(params.scales), static_cast<ScalarT*>(params.outputs),
+ params.num_tokens_per_expert, params.num_experts, params.N, params.K, params.group_size,
+ params.total_tokens);
+ } else {
+ moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, true, false>(
+ params.q, static_cast<const ScalarT*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const ScalarT*>(params.scales), static_cast<ScalarT*>(params.outputs),
+ params.num_tokens_per_expert, params.num_experts, params.N, params.K, params.group_size,
+ params.total_tokens);
+ }
+ } else if (use_lut) {
+ moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, false, true>(
+ params.q, static_cast<const ScalarT*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const ScalarT*>(params.scales), static_cast<ScalarT*>(params.outputs),
+ params.num_tokens_per_expert, params.num_experts, params.N, params.K, params.group_size,
+ params.total_tokens);
+ } else {
+ moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, false, false>(
+ params.q, static_cast<const ScalarT*>(params.activations), static_cast<const uint8_t*>(params.weights),
+ static_cast<const ScalarT*>(params.scales), static_cast<ScalarT*>(params.outputs),
+ params.num_tokens_per_expert, params.num_experts, params.N, params.K, params.group_size,
+ params.total_tokens);
+ }
+ return true;
+ }
+ return false;
+}
+
+} // namespace ark
+
+#endif
+
+#endif // ARK_XPU && ARK_SYCL_TLA
\ No newline at end of file
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_s8_wrapper.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_s8_wrapper.hpp
index a77dcf2c7..31435a661 100644
--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_s8_wrapper.hpp
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_s8_wrapper.hpp
@@ -16,7 +16,7 @@
#if ARK_XPU
#if ARK_SYCL_TLA
-#include "sycl_tla_s8_gemm.hpp"
+#include "sycl_tla_common.hpp"
#endif
namespace ark {
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_common.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_common.hpp
index 824053d1b..390b5f890 100644
--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_common.hpp
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_common.hpp
@@ -34,13 +34,17 @@ namespace ark {
*/
void moe_gemm(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs, BTLA_DTYPE dtype, int N,
int K, int* num_tokens_per_expert, int num_experts);
+void sycl_tla_moe_gemm_f16(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs, int N, int K,
+ int* num_tokens_per_expert, int num_experts);
+void sycl_tla_moe_gemm_bf16(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs, int N, int K,
+ int* num_tokens_per_expert, int num_experts);
/**
* @brief MoE GEMV optimized for the decode phase (M per expert is typically
* 1-2 tokens). Supports unquantized FP16/BF16 weights and int4 (S4_CLIP)
* weights with group-wise scales and optional zero-points.
*
- * Implementation is header-only in `sycl_tla_moe_decode.hpp`.
+ * Heavy template implementations are split across the decode family source files.
*
* @param q SYCL queue
* @param activations [total_tokens, K] in `act_dtype`
@@ -69,6 +73,31 @@ void moe_gemm_decode(sycl::queue* q, void* activations, void* weights, void* sca
int* expert_id_per_token_buf, BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,
int group_size, int* num_tokens_per_expert, int num_experts, int total_tokens, bool asym);
+struct MoeDecodeParams {
+ sycl::queue* q;
+ void* activations;
+ void* weights;
+ void* scales;
+ void* zeros;
+ void* outputs;
+ int* expert_id_per_token;
+ BTLA_DTYPE act_dtype;
+ BTLA_DTYPE weight_dtype;
+ int N;
+ int K;
+ int group_size;
+ int total_tokens;
+ bool asym;
+};
+
+void sycl_tla_moe_decode_fill_expert_id(sycl::queue* q, int* expert_id_per_token,
+ const int* num_tokens_per_expert, int num_experts, int total_tokens);
+void sycl_tla_moe_decode_fp(const MoeDecodeParams& params);
+void sycl_tla_moe_decode_int4(const MoeDecodeParams& params);
+void sycl_tla_moe_decode_int8(const MoeDecodeParams& params);
+void sycl_tla_moe_decode_int2(const MoeDecodeParams& params);
+void sycl_tla_moe_decode_fp8(const MoeDecodeParams& params);
+
/**
* @brief MoE Grouped GEMM optimized for the prefill phase, supporting the
* same set of weight encodings as `moe_gemm_decode` (FP16/BF16, INT8 sym/asym,
@@ -80,7 +109,7 @@ void moe_gemm_decode(sycl::queue* q, void* activations, void* weights, void* sca
* existing `moe_gemm` baseline. This guarantees numerical parity with the
* decode path. Mainloop fusion is the follow-up perf-tuning step.
*
- * Implementation is header-only in `sycl_tla_moe_mixed.hpp`.
+ * Heavy template implementations are split across the prefill family source files.
*
* Layout convention (matches `moe_gemm_decode`):
* - activations: [total_tokens, K] in act_dtype
@@ -96,6 +125,33 @@ void moe_gemm_prefill(sycl::queue* q, void* activations, void* weights, void* sc
void* dequant_workspace, BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,
int group_size, int* num_tokens_per_expert, int num_experts, int total_tokens, bool asym);
+struct MoePrefillParams {
+ sycl::queue* q;
+ void* activations;
+ void* weights;
+ void* scales;
+ void* zeros;
+ void* outputs;
+ void* dequant_workspace;
+ BTLA_DTYPE act_dtype;
+ BTLA_DTYPE weight_dtype;
+ int N;
+ int K;
+ int group_size;
+ int* num_tokens_per_expert;
+ int num_experts;
+ int total_tokens;
+ bool asym;
+};
+
+bool sycl_tla_moe_prefill_int8_dpas(const MoePrefillParams& params);
+bool sycl_tla_moe_prefill_s4_dpas(const MoePrefillParams& params);
+bool sycl_tla_moe_prefill_lowbit_int8_dpas(const MoePrefillParams& params);
+bool sycl_tla_moe_prefill_fp8_dpas(const MoePrefillParams& params);
+bool sycl_tla_moe_prefill_native_fp8(const MoePrefillParams& params);
+void sycl_tla_moe_prefill_dequant_f16(const MoePrefillParams& params);
+void sycl_tla_moe_prefill_dequant_bf16(const MoePrefillParams& params);
+
/**
* @brief MoE prefill Grouped GEMM -- FP8 per-tensor mixed-input DPAS
* (Variant A of the vllm-xpu-kernels FP8 port).
@@ -109,7 +165,7 @@ void moe_gemm_prefill(sycl::queue* q, void* activations, void* weights, void* sc
* `sycl_tla_moe_prefill_fp8_dpas.hpp` for the port's provenance & the
* on-hardware TODOs.
*
- * Implementation is header-only in `sycl_tla_moe_prefill_fp8_dpas.hpp`.
+ * The public entry point is implemented in `sycl_tla_moe_prefill_fp8_tensor.cpp`.
*/
void moe_gemm_prefill_fp8_dpas(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs,
BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,
@@ -133,12 +189,31 @@ void moe_gemm_prefill_fp8_dpas(sycl::queue* q, void* activations, void* weights,
* `sycl_tla_moe_prefill_int_dpas.hpp` for the port's provenance & the
* on-hardware TODOs.
*
- * Implementation is header-only in `sycl_tla_moe_prefill_int_dpas.hpp`.
+ * The public entry point is implemented in `sycl_tla_moe_prefill_int_tensor.cpp`.
*/
void moe_gemm_prefill_int_dpas(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs,
BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,
int* num_tokens_per_expert, int num_experts, int total_tokens);
+void sycl_tla_dense_gemm(sycl::queue* q, int m, int n, int k, const void* a, BTLA_DTYPE at, const void* b,
+ BTLA_DTYPE bt, void* c, BTLA_DTYPE ct, const void* bias, bool BT);
+void sycl_tla_dense_gemm_f32(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,
+ const void* bias);
+void sycl_tla_dense_gemm_f16(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,
+ const void* bias);
+void sycl_tla_dense_gemm_bf16(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,
+ const void* bias);
+
+void sycl_tla_igemm_s8s8_dequant(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,
+ BTLA_DTYPE ct, const void* scale_a, const void* scale_b, const void* bias,
+ int blocksize);
+void sycl_tla_igemm_s8s8_dequant_f32(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,
+ const void* scale_a, const void* scale_b, const void* bias, int blocksize);
+void sycl_tla_igemm_s8s8_dequant_f16(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,
+ const void* scale_a, const void* scale_b, const void* bias, int blocksize);
+void sycl_tla_igemm_s8s8_dequant_bf16(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,
+ const void* scale_a, const void* scale_b, const void* bias, int blocksize);
+
// ========================================================================
// Public API
// ========================================================================
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm.hpp
index 8db797453..70450879d 100644
--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm.hpp
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm.hpp
@@ -30,6 +30,7 @@
#endif
#include "utils.hpp"
+#include "sycl_tla_common.hpp"
namespace ark {
@@ -210,6 +211,21 @@ void gemm_cute_store_tile(sycl::queue* q, ATensor const& A, BTensor const& B, CT
}
+using DenseSmallTileSG = Layout<Shape<_1, _4, _1>, Stride<_0, _1, _0>>;
+using DenseSmallMidTileSG = Layout<Shape<_2, _4, _1>, Stride<_4, _1, _0>>;
+using DenseMediumTileSG = Layout<Shape<_4, _4, _1>, Stride<_4, _1, _0>>;
+using DenseLargeTileSG = Layout<Shape<_8, _4, _1>, Stride<_4, _1, _0>>;
+
+template <bool HasBias, int TileM, int TileN, class SGLayout, typename Element>
+void run_dense_gemm_fixed_tile(sycl::queue* q, int m, int n, int k, const Element* a_ptr, const Element* b_ptr,
+ Element* c_ptr, const Element* bias_ptr) {
+ auto A = make_tensor(make_gmem_ptr(const_cast<Element*>(a_ptr)), make_shape(m, k), make_stride(k, _1{}));
+ auto B = make_tensor(make_gmem_ptr(const_cast<Element*>(b_ptr)), make_shape(n, k), make_stride(k, _1{}));
+ auto C = make_tensor(make_gmem_ptr(c_ptr), make_shape(m, n), make_stride(n, _1{}));
+ gemm_cute_store_tile<HasBias, TileM, TileN, SGLayout, decltype(A), decltype(B), decltype(C), Element, Element, Element,
+ 'R', 'R'>(q, A, B, C, bias_ptr);
+}
+
template <bool HasBias, typename Element>
void run_dense_gemm_tuned_tile(sycl::queue* q, int m, int n, int k, const Element* a_ptr, const Element* b_ptr,
@@ -250,52 +266,6 @@ void run_dense_gemm(sycl::queue* q, int m, int n, int k, const Element* a_ptr, c
} // namespace dense_gemm_detail
-inline void sycl_tla_dense_gemm(sycl::queue* q, int m, int n, int k, const void* a, BTLA_DTYPE at,
- const void* b, BTLA_DTYPE bt, void* c, BTLA_DTYPE ct, const void* bias,
- bool BT) {
- if (!q) {
- throw std::invalid_argument("sycl_tla_dense_gemm: stream must be a valid SYCL queue");
- }
- if (!BT) {
- throw std::invalid_argument("sycl_tla_dense_gemm: only the A @ B.T contract is supported");
- }
- if (!a || !b || !c) {
- throw std::invalid_argument("sycl_tla_dense_gemm: input and output pointers must not be null");
- }
- if (m <= 0 || n <= 0 || k <= 0) {
- return;
- }
- if (at != bt) {
- throw std::invalid_argument("sycl_tla_dense_gemm: A and B must use the same dtype");
- }
- if (ct != at) {
- throw std::invalid_argument("sycl_tla_dense_gemm: output dtype must match input dtype");
- }
-
- switch (at) {
- case BTLA_DTYPE::F32:
- dense_gemm_detail::run_dense_gemm(
- q, m, n, k, static_cast<const float*>(a), static_cast<const float*>(b),
- static_cast<float*>(c), static_cast<const float*>(bias));
- return;
-
- case BTLA_DTYPE::F16:
- dense_gemm_detail::run_dense_gemm(
- q, m, n, k, static_cast<const cute::half_t*>(a), static_cast<const cute::half_t*>(b),
- static_cast<cute::half_t*>(c), static_cast<const cute::half_t*>(bias));
- return;
-
- case BTLA_DTYPE::BF16:
- dense_gemm_detail::run_dense_gemm(
- q, m, n, k, static_cast<const cute::bfloat16_t*>(a), static_cast<const cute::bfloat16_t*>(b),
- static_cast<cute::bfloat16_t*>(c), static_cast<const cute::bfloat16_t*>(bias));
- return;
-
- default:
- throw std::invalid_argument("sycl_tla_dense_gemm: only FP32, FP16 and BF16 are supported");
- }
-}
-
#endif // ARK_XPU && ARK_SYCL_TLA
} // namespace ark
\ No newline at end of file
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm_helpers.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm_helpers.hpp
new file mode 100644
index 000000000..086a284d0
--- /dev/null
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm_helpers.hpp
@@ -0,0 +1,33 @@
+#pragma once
+
+#include "sycl_tla_common.hpp"
+
+#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)
+
+namespace ark {
+namespace dense_gemm_detail {
+
+#define ARK_DECLARE_DENSE_GEMM_ROUTES(prefix) \
+ void prefix##_with_bias_small(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \
+ const void* bias); \
+ void prefix##_with_bias_mid(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \
+ const void* bias); \
+ void prefix##_with_bias_medium(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \
+ const void* bias); \
+ void prefix##_with_bias_large(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \
+ const void* bias); \
+ void prefix##_without_bias_small(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c); \
+ void prefix##_without_bias_mid(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c); \
+ void prefix##_without_bias_medium(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c); \
+ void prefix##_without_bias_large(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c)
+
+ARK_DECLARE_DENSE_GEMM_ROUTES(run_f16);
+ARK_DECLARE_DENSE_GEMM_ROUTES(run_bf16);
+ARK_DECLARE_DENSE_GEMM_ROUTES(run_f32);
+
+#undef ARK_DECLARE_DENSE_GEMM_ROUTES
+
+} // namespace dense_gemm_detail
+} // namespace ark
+
+#endif
\ No newline at end of file
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe.hpp
index cff5626de..d2a0b0ae5 100644
--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe.hpp
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe.hpp
@@ -123,31 +123,6 @@ void moe_gemm_launcher(sycl::queue* q, const ElementA* activations, const Elemen
} // namespace moe_detail
-// Public MOE GEMM API
-inline void moe_gemm(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs, BTLA_DTYPE dtype,
- int N, int K, int* num_tokens_per_expert, int num_experts) {
- switch (dtype) {
- case BTLA_DTYPE::BF16: {
- using Element = cutlass::bfloat16_t;
- moe_detail::moe_gemm_launcher<'R', 'R', Element, Element, Element, Element>(
- q, static_cast<const Element*>(activations), static_cast<const Element*>(weights),
- static_cast<const Element*>(scales), static_cast<Element*>(outputs), N, K, num_tokens_per_expert,
- num_experts);
- break;
- }
- case BTLA_DTYPE::F16: {
- using Element = cutlass::half_t;
- moe_detail::moe_gemm_launcher<'R', 'R', Element, Element, Element, Element>(
- q, static_cast<const Element*>(activations), static_cast<const Element*>(weights),
- static_cast<const Element*>(scales), static_cast<Element*>(outputs), N, K, num_tokens_per_expert,
- num_experts);
- break;
- }
- default:
- throw std::runtime_error("moe_gemm: unsupported dtype, only BF16/FP16 supported");
- }
-}
-
#endif // ARK_SYCL_TLA
} // namespace ark
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_decode.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_decode.hpp
index 1b8a4feb0..4a79c5034 100644
--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_decode.hpp
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_decode.hpp
@@ -39,6 +39,7 @@
#include "bestla/bestla.h"
#include "sycl_tla_moe_dequant.hpp"
+#include "sycl_tla_common.hpp"
#ifdef ARK_XPU
#include <sycl/sycl.hpp>
@@ -637,196 +638,6 @@ void launch_fp8(sycl::queue* q, const ScalarT* activations, const uint8_t* weigh
// group-wise scales, no zero-points
// act_dtype: F16 or BF16 (must match scales/outputs dtype)
// ----------------------------------------------------------------------------
-inline void moe_gemm_decode(sycl::queue* q, void* activations, void* weights, void* scales, void* zeros,
- void* outputs, int* expert_id_per_token_buf, BTLA_DTYPE act_dtype,
- BTLA_DTYPE weight_dtype, int N, int K, int group_size, int* num_tokens_per_expert,
- int num_experts, int total_tokens, bool asym) {
- moe_decode_detail::fill_expert_id_per_token(q, expert_id_per_token_buf, num_tokens_per_expert, num_experts,
- total_tokens);
-
- if (weight_dtype == BTLA_DTYPE::F16 || weight_dtype == BTLA_DTYPE::BF16) {
- if (weight_dtype != act_dtype) {
- throw std::invalid_argument("moe_gemm_decode: unquantized weight_dtype must match act_dtype");
- }
- if (act_dtype == BTLA_DTYPE::F16) {
- moe_decode_detail::launch_fp<sycl::half>(q, static_cast<const sycl::half*>(activations),
- static_cast<const sycl::half*>(weights),
- static_cast<sycl::half*>(outputs), expert_id_per_token_buf,
- total_tokens, N, K);
- } else {
- moe_decode_detail::launch_fp<sycl::ext::oneapi::bfloat16>(
- q, static_cast<const sycl::ext::oneapi::bfloat16*>(activations),
- static_cast<const sycl::ext::oneapi::bfloat16*>(weights),
- static_cast<sycl::ext::oneapi::bfloat16*>(outputs), expert_id_per_token_buf, total_tokens, N, K);
- }
- return;
- }
-
- if (weight_dtype == BTLA_DTYPE::S4_CLIP) {
- if (act_dtype == BTLA_DTYPE::F16) {
- if (asym) {
- moe_decode_detail::launch_int4<sycl::half, true>(
- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const sycl::half*>(scales), static_cast<const sycl::half*>(zeros),
- static_cast<sycl::half*>(outputs), expert_id_per_token_buf, total_tokens, N, K, group_size);
- } else {
- moe_decode_detail::launch_int4<sycl::half, false>(
- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const sycl::half*>(scales), static_cast<const sycl::half*>(zeros),
- static_cast<sycl::half*>(outputs), expert_id_per_token_buf, total_tokens, N, K, group_size);
- }
- } else if (act_dtype == BTLA_DTYPE::BF16) {
- using BF = sycl::ext::oneapi::bfloat16;
- if (asym) {
- moe_decode_detail::launch_int4<BF, true>(
- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const BF*>(scales), static_cast<const BF*>(zeros), static_cast<BF*>(outputs),
- expert_id_per_token_buf, total_tokens, N, K, group_size);
- } else {
- moe_decode_detail::launch_int4<BF, false>(
- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const BF*>(scales), static_cast<const BF*>(zeros), static_cast<BF*>(outputs),
- expert_id_per_token_buf, total_tokens, N, K, group_size);
- }
- } else {
- throw std::invalid_argument("moe_gemm_decode(int4): act_dtype must be FP16 or BF16");
- }
- return;
- }
-
- if (weight_dtype == BTLA_DTYPE::S8) {
- if (act_dtype == BTLA_DTYPE::F16) {
- if (asym) {
- moe_decode_detail::launch_int8<sycl::half, true>(
- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const sycl::half*>(scales), static_cast<const sycl::half*>(zeros),
- static_cast<sycl::half*>(outputs), expert_id_per_token_buf, total_tokens, N, K, group_size);
- } else {
- moe_decode_detail::launch_int8<sycl::half, false>(
- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const sycl::half*>(scales), static_cast<const sycl::half*>(zeros),
- static_cast<sycl::half*>(outputs), expert_id_per_token_buf, total_tokens, N, K, group_size);
- }
- } else if (act_dtype == BTLA_DTYPE::BF16) {
- using BF = sycl::ext::oneapi::bfloat16;
- if (asym) {
- moe_decode_detail::launch_int8<BF, true>(
- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const BF*>(scales), static_cast<const BF*>(zeros), static_cast<BF*>(outputs),
- expert_id_per_token_buf, total_tokens, N, K, group_size);
- } else {
- moe_decode_detail::launch_int8<BF, false>(
- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const BF*>(scales), static_cast<const BF*>(zeros), static_cast<BF*>(outputs),
- expert_id_per_token_buf, total_tokens, N, K, group_size);
- }
- } else {
- throw std::invalid_argument("moe_gemm_decode(int8): act_dtype must be FP16 or BF16");
- }
- return;
- }
-
- if (weight_dtype == BTLA_DTYPE::S2_CLIP) {
- if (act_dtype == BTLA_DTYPE::F16) {
- if (asym) {
- moe_decode_detail::launch_int2<sycl::half, true>(
- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const sycl::half*>(scales), static_cast<const sycl::half*>(zeros),
- static_cast<sycl::half*>(outputs), expert_id_per_token_buf, total_tokens, N, K, group_size);
- } else {
- moe_decode_detail::launch_int2<sycl::half, false>(
- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const sycl::half*>(scales), static_cast<const sycl::half*>(zeros),
- static_cast<sycl::half*>(outputs), expert_id_per_token_buf, total_tokens, N, K, group_size);
- }
- } else if (act_dtype == BTLA_DTYPE::BF16) {
- using BF = sycl::ext::oneapi::bfloat16;
- if (asym) {
- moe_decode_detail::launch_int2<BF, true>(
- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const BF*>(scales), static_cast<const BF*>(zeros), static_cast<BF*>(outputs),
- expert_id_per_token_buf, total_tokens, N, K, group_size);
- } else {
- moe_decode_detail::launch_int2<BF, false>(
- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const BF*>(scales), static_cast<const BF*>(zeros), static_cast<BF*>(outputs),
- expert_id_per_token_buf, total_tokens, N, K, group_size);
- }
- } else {
- throw std::invalid_argument("moe_gemm_decode(int2): act_dtype must be FP16 or BF16");
- }
- return;
- }
-
- if (weight_dtype == BTLA_DTYPE::F8_E4M3 || weight_dtype == BTLA_DTYPE::F8_E5M2) {
- if (asym) {
- throw std::invalid_argument("moe_gemm_decode(fp8): asym mode is not supported");
- }
- const bool is_e4m3 = (weight_dtype == BTLA_DTYPE::F8_E4M3);
- const bool use_lut = moe_decode_detail::fp8_decode_use_lut();
- if (act_dtype == BTLA_DTYPE::F16) {
- if (is_e4m3) {
- if (use_lut) {
- moe_decode_detail::launch_fp8<sycl::half, true, true>(
- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const sycl::half*>(scales), static_cast<sycl::half*>(outputs), expert_id_per_token_buf,
- total_tokens, N, K, group_size);
- } else {
- moe_decode_detail::launch_fp8<sycl::half, true, false>(
- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const sycl::half*>(scales), static_cast<sycl::half*>(outputs), expert_id_per_token_buf,
- total_tokens, N, K, group_size);
- }
- } else {
- if (use_lut) {
- moe_decode_detail::launch_fp8<sycl::half, false, true>(
- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const sycl::half*>(scales), static_cast<sycl::half*>(outputs), expert_id_per_token_buf,
- total_tokens, N, K, group_size);
- } else {
- moe_decode_detail::launch_fp8<sycl::half, false, false>(
- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const sycl::half*>(scales), static_cast<sycl::half*>(outputs), expert_id_per_token_buf,
- total_tokens, N, K, group_size);
- }
- }
- } else if (act_dtype == BTLA_DTYPE::BF16) {
- using BF = sycl::ext::oneapi::bfloat16;
- if (is_e4m3) {
- if (use_lut) {
- moe_decode_detail::launch_fp8<BF, true, true>(
- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const BF*>(scales), static_cast<BF*>(outputs), expert_id_per_token_buf, total_tokens, N, K,
- group_size);
- } else {
- moe_decode_detail::launch_fp8<BF, true, false>(
- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const BF*>(scales), static_cast<BF*>(outputs), expert_id_per_token_buf, total_tokens, N, K,
- group_size);
- }
- } else {
- if (use_lut) {
- moe_decode_detail::launch_fp8<BF, false, true>(
- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const BF*>(scales), static_cast<BF*>(outputs), expert_id_per_token_buf, total_tokens, N, K,
- group_size);
- } else {
- moe_decode_detail::launch_fp8<BF, false, false>(
- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const BF*>(scales), static_cast<BF*>(outputs), expert_id_per_token_buf, total_tokens, N, K,
- group_size);
- }
- }
- } else {
- throw std::invalid_argument("moe_gemm_decode(fp8): act_dtype must be FP16 or BF16");
- }
- return;
- }
-
- throw std::invalid_argument(
- "moe_gemm_decode: unsupported weight_dtype (supported: F16, BF16, S8, S4_CLIP, S2_CLIP, F8_E4M3, F8_E5M2)");
-}
-
} // namespace ark
#endif // ARK_XPU && ARK_SYCL_TLA
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_mixed.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_mixed.hpp
index e6a3ececc..51d218a13 100644
--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_mixed.hpp
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_mixed.hpp
@@ -40,13 +40,10 @@
#include <stdexcept>
#include "bestla/bestla.h"
-#include "sycl_tla_moe.hpp"
#include "sycl_tla_moe_dequant.hpp"
-#include "sycl_tla_moe_prefill_fp8_dpas.hpp"
-#include "sycl_tla_moe_prefill_int_dpas.hpp"
-#include "sycl_tla_moe_prefill_s4_dpas.hpp"
-#include "sycl_tla_moe_prefill_fp8_native.hpp"
#include "sycl_tla_moe_prefill_fused.hpp"
+#include "sycl_tla_moe_prefill_upcast.hpp"
+#include "sycl_tla_common.hpp"
#ifdef ARK_XPU
#include <sycl/sycl.hpp>
@@ -88,21 +85,6 @@ class MoEDequantKernelInt2Fast;
template <typename ScalarT, bool IsE4M3, bool UseLut>
class MoEDequantKernelFP8;
-// Kernel name tags for the low-bit-width -> int8 sym upcast helpers used
-// by the INT DPAS grouped-GEMM path. Output layout is `[E, N, K]` int8
-// row-major (auto-round decode convention) so the caller can hand the
-// resulting buffer to `moe_prefill_int_dpas_per_group_dispatch` unchanged.
-class MoEUpcastInt4SymToInt8Kernel;
-class MoEUpcastInt2SymToInt8Kernel;
-
-// PR-1 fast-path counterparts of the sym->int8 upcast kernels: one
-// work-item processes 4 packed bytes (INT4) / 2 packed bytes (INT2) = 8
-// consecutive int8 outputs. Same bit-level decode as the scalar path so
-// the downstream `moe_prefill_int_dpas_per_group_dispatch` sees identical
-// bytes.
-class MoEUpcastInt4SymToInt8KernelFast;
-class MoEUpcastInt2SymToInt8KernelFast;
-
// Tile sizes for the dequant kernels.
//
// Each work-group covers a (PACK_K x WG_N) tile in (k, n) and writes PACK_K
@@ -131,8 +113,6 @@ class MoEUpcastInt2SymToInt8KernelFast;
constexpr int WG_N = 32;
constexpr int PACK_K_FP = 4;
constexpr int PACK_K_INT8 = 4;
-constexpr int PACK_K_INT4 = 2;
-constexpr int PACK_K_INT2 = 4;
constexpr int PACK_K_FP8 = 4;
// PR-1 fast-path PACK_K: one work-item handles 4 packed bytes for INT4
@@ -146,9 +126,6 @@ constexpr int PACK_K_FP8 = 4;
// suite (group_size is 32/64/128 and K is a multiple of group_size), but
// the kernels fall back to the byte-per-work-item path when either
// constraint is violated so short-K unit tests keep working.
-constexpr int PACK_K_INT4_FAST = 8;
-constexpr int PACK_K_INT2_FAST = 8;
-
// ----------------------------------------------------------------------------
// FP16 / BF16 weight reshape: in-place transpose [E, N, K] -> [E, K, N].
// Implemented as a generic dequant pass with identity scale; used so the
@@ -479,168 +456,6 @@ void launch_dequant_int2(sycl::queue* q, const uint8_t* weights_NKp, const Scala
});
}
-// ----------------------------------------------------------------------------
-// INT4 sym -> INT8 sym upcast: [E, N, K/2] uint8 packed -> [E, N, K] int8.
-//
-// Sign-extends each 4-bit signed field to a full `int8_t` (range [-8, 7])
-// preserving the auto-round `[E, N, K]` decode-style layout so the result
-// can be fed directly to `moe_prefill_int_dpas_per_group_dispatch`, which
-// treats its `int8_t` weight input as sym-packed values that the per-group
-// scale (already `[E, N, K/group_size]` in act dtype) rescales at each
-// group boundary. Algebraically:
-// dequant(w_int4)[e, n, k] = int4_signed(w_int4[e, n, k]) * scale[e, n, g]
-// = int8_signed(upcast[e, n, k]) * scale[e, n, g]
-// so the same `scale` tensor packs unmodified into the DPAS mainloop.
-// ----------------------------------------------------------------------------
-inline void launch_upcast_int4_sym_to_int8(sycl::queue* q, const uint8_t* weights_NKp, int8_t* weights_i8_NK,
- int E, int N, int K,
- const int* num_tokens_per_expert = nullptr) {
- if (E == 0 || N == 0 || K == 0) return;
- if ((K & 1) != 0) {
- throw std::invalid_argument("moe_gemm_prefill(int4->int8 upcast): K must be even");
- }
- const int k_packed = K / 2;
-
- // PR-1 fast path: one work-item processes a 4-byte packed word = 8 int8
- // outputs, matching the launch geometry of the int4 dequant fast path so
- // the two kernels share cache-hit patterns for the packed weight buffer.
- // Output writes here are contiguous along K (buffer layout `[E, N, K]`),
- // so 8 consecutive int8 stores form an aligned 8-byte burst.
- if ((K % PACK_K_INT4_FAST) == 0) {
- const int k_words = K / PACK_K_INT4_FAST;
- sycl::range<3> global_fast{static_cast<size_t>(E), static_cast<size_t>(k_words),
- static_cast<size_t>((N + WG_N - 1) / WG_N) * WG_N};
- sycl::range<3> local_fast{1, 1, static_cast<size_t>(WG_N)};
-
- q->parallel_for<MoEUpcastInt4SymToInt8KernelFast>(
- sycl::nd_range<3>(global_fast, local_fast), [=](sycl::nd_item<3> it) {
- const int e = static_cast<int>(it.get_global_id(0));
- if (num_tokens_per_expert != nullptr && num_tokens_per_expert[e] == 0) return;
- const int kw = static_cast<int>(it.get_global_id(1));
- const int n = static_cast<int>(it.get_global_id(2));
- if (n >= N) return;
- const int k_base = kw * PACK_K_INT4_FAST;
- const size_t row_kp_base = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * k_packed +
- static_cast<size_t>(kw) * 4;
- const uint32_t packed =
- static_cast<uint32_t>(weights_NKp[row_kp_base + 0]) |
- (static_cast<uint32_t>(weights_NKp[row_kp_base + 1]) << 8) |
- (static_cast<uint32_t>(weights_NKp[row_kp_base + 2]) << 16) |
- (static_cast<uint32_t>(weights_NKp[row_kp_base + 3]) << 24);
- int qv[8];
- moe_dequant::decode_int4_octet</*Asym=*/false>(packed, qv);
- const size_t out_row = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * K;
-#pragma unroll
- for (int j = 0; j < PACK_K_INT4_FAST; ++j) {
- weights_i8_NK[out_row + static_cast<size_t>(k_base + j)] = static_cast<int8_t>(qv[j]);
- }
- });
- return;
- }
-
- // Fallback byte-per-work-item path (short-K configurations).
- // One work-item per packed byte writes PACK_K_INT4 (=2) consecutive int8
- // outputs for a single (e, n). Matches the launch geometry of
- // `launch_dequant_int4` so the two paths share cache-hit patterns for
- // the packed-weight buffer.
- sycl::range<3> global{static_cast<size_t>(E), static_cast<size_t>(k_packed),
- static_cast<size_t>((N + WG_N - 1) / WG_N) * WG_N};
- sycl::range<3> local{1, 1, static_cast<size_t>(WG_N)};
-
- q->parallel_for<MoEUpcastInt4SymToInt8Kernel>(
- sycl::nd_range<3>(global, local), [=](sycl::nd_item<3> it) {
- const int e = static_cast<int>(it.get_global_id(0));
- if (num_tokens_per_expert != nullptr && num_tokens_per_expert[e] == 0) return;
- const int kp = static_cast<int>(it.get_global_id(1));
- const int n = static_cast<int>(it.get_global_id(2));
- if (n >= N) return;
- const int k_base = kp * PACK_K_INT4;
- const uint8_t packed =
- weights_NKp[(static_cast<size_t>(e) * N + static_cast<size_t>(n)) * k_packed +
- static_cast<size_t>(kp)];
- int q_lo, q_hi;
- moe_dequant::decode_int4_pair</*Asym=*/false>(packed, q_lo, q_hi);
- const size_t out_row = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * K;
- weights_i8_NK[out_row + static_cast<size_t>(k_base)] = static_cast<int8_t>(q_lo);
- weights_i8_NK[out_row + static_cast<size_t>(k_base + 1)] = static_cast<int8_t>(q_hi);
- });
-}
-
-// ----------------------------------------------------------------------------
-// INT2 sym -> INT8 sym upcast: [E, N, K/4] uint8 packed -> [E, N, K] int8.
-//
-// Sign-extends each 2-bit signed field to a full `int8_t` (range [-2, 1]).
-// Same rationale as `launch_upcast_int4_sym_to_int8` -- the DPAS per-group
-// mainloop consumes the resulting `[E, N, K]` int8 buffer with the packed
-// int2 scale tensor unmodified.
-// ----------------------------------------------------------------------------
-inline void launch_upcast_int2_sym_to_int8(sycl::queue* q, const uint8_t* weights_NKp, int8_t* weights_i8_NK,
- int E, int N, int K,
- const int* num_tokens_per_expert = nullptr) {
- if (E == 0 || N == 0 || K == 0) return;
- if ((K & 3) != 0) {
- throw std::invalid_argument("moe_gemm_prefill(int2->int8 upcast): K must be a multiple of 4");
- }
- const int k_packed = K / 4;
-
- // PR-1 fast path: one work-item processes a 2-byte packed word = 8 int8
- // outputs. Same rationale as the int4 upcast fast path above.
- if ((K % PACK_K_INT2_FAST) == 0) {
- const int k_words = K / PACK_K_INT2_FAST;
- sycl::range<3> global_fast{static_cast<size_t>(E), static_cast<size_t>(k_words),
- static_cast<size_t>((N + WG_N - 1) / WG_N) * WG_N};
- sycl::range<3> local_fast{1, 1, static_cast<size_t>(WG_N)};
-
- q->parallel_for<MoEUpcastInt2SymToInt8KernelFast>(
- sycl::nd_range<3>(global_fast, local_fast), [=](sycl::nd_item<3> it) {
- const int e = static_cast<int>(it.get_global_id(0));
- if (num_tokens_per_expert != nullptr && num_tokens_per_expert[e] == 0) return;
- const int kw = static_cast<int>(it.get_global_id(1));
- const int n = static_cast<int>(it.get_global_id(2));
- if (n >= N) return;
- const int k_base = kw * PACK_K_INT2_FAST;
- const size_t row_kp_base = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * k_packed +
- static_cast<size_t>(kw) * 2;
- const uint16_t packed =
- static_cast<uint16_t>(static_cast<uint32_t>(weights_NKp[row_kp_base + 0]) |
- (static_cast<uint32_t>(weights_NKp[row_kp_base + 1]) << 8));
- int qv[8];
- moe_dequant::decode_int2_octet</*Asym=*/false>(packed, qv);
- const size_t out_row = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * K;
-#pragma unroll
- for (int j = 0; j < PACK_K_INT2_FAST; ++j) {
- weights_i8_NK[out_row + static_cast<size_t>(k_base + j)] = static_cast<int8_t>(qv[j]);
- }
- });
- return;
- }
-
- // Fallback byte-per-work-item path (short-K configurations).
- sycl::range<3> global{static_cast<size_t>(E), static_cast<size_t>(k_packed),
- static_cast<size_t>((N + WG_N - 1) / WG_N) * WG_N};
- sycl::range<3> local{1, 1, static_cast<size_t>(WG_N)};
-
- q->parallel_for<MoEUpcastInt2SymToInt8Kernel>(
- sycl::nd_range<3>(global, local), [=](sycl::nd_item<3> it) {
- const int e = static_cast<int>(it.get_global_id(0));
- if (num_tokens_per_expert != nullptr && num_tokens_per_expert[e] == 0) return;
- const int kp = static_cast<int>(it.get_global_id(1));
- const int n = static_cast<int>(it.get_global_id(2));
- if (n >= N) return;
- const int k_base = kp * PACK_K_INT2;
- const uint8_t packed =
- weights_NKp[(static_cast<size_t>(e) * N + static_cast<size_t>(n)) * k_packed +
- static_cast<size_t>(kp)];
- int qv[4];
- moe_dequant::decode_int2_quad</*Asym=*/false>(packed, qv);
- const size_t out_row = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * K;
-#pragma unroll
- for (int j = 0; j < PACK_K_INT2; ++j) {
- weights_i8_NK[out_row + static_cast<size_t>(k_base + j)] = static_cast<int8_t>(qv[j]);
- }
- });
-}
-
// ----------------------------------------------------------------------------
// FP8 (E4M3 / E5M2) dequant: [E, N, K] uint8 -> [E, K, N] ScalarT.
// Per-group scale applied; no zero-points (caller must enforce sym).
@@ -811,290 +626,6 @@ void dequant_to_KN(sycl::queue* q, const void* weights, const void* scales, cons
// under the same shape contract as `moe_gemm` and the quantized paths
// under the same shape contract as `moe_gemm_decode`.
// ----------------------------------------------------------------------------
-inline void moe_gemm_prefill(sycl::queue* q, void* activations, void* weights, void* scales, void* zeros,
- void* outputs, void* dequant_workspace, BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype,
- int N, int K, int group_size, int* num_tokens_per_expert, int num_experts,
- int total_tokens, bool asym) {
- if (total_tokens == 0) return;
-
- // Unquantized fast path: forward directly to the existing prefill GEMM.
- // The caller is responsible for passing weights in `[E, K, N]` layout
- // (matching the existing `moe_gemm` contract).
- if (weight_dtype == act_dtype && (weight_dtype == BTLA_DTYPE::F16 || weight_dtype == BTLA_DTYPE::BF16)) {
- moe_gemm(q, activations, weights, scales, outputs, act_dtype, N, K, num_tokens_per_expert, num_experts);
- return;
- }
-
- // Native FP8 fused GEMM (Variant A: fp8 weight x bf16/fp16 activation,
- // in-register upcast, group-boundary scale fold, no `[E, K, N]` workspace).
- // Opt-in via `ARK_MOE_PREFILL_NATIVE_FP8=1`; silent fallback if the shape
- // doesn't satisfy the tile preconditions or if `asym=true` is requested.
- // Precedence (highest first): dpas -> native -> fused-dequant -> v1 dequant.
- // See `sycl_tla_moe_prefill_fp8_native.hpp` for the tile / SLM design.
-
- // INT8 mixed-input DPAS grouped GEMM (Variant B, sym only): per-K-group
- // scale, in-register int8->act upcast, XMX MMA. Opt-in default via
- // `ARK_MOE_PREFILL_DPAS_INT8` (default ON, matching the FP8 DPAS gate).
- // Shape gate mirrors the FP8 per-group predicate exactly. Only `asym=false`
- // is accelerated here -- the asym port used a per-M-row per-K-group
- // activation-sum precompute and regressed below the dequant fallback on
- // hardware, so asym S8 falls through to the dequant + `moe_gemm` path
- // below.
- //
- // Scales are already the activation dtype (`[E, N, K/group_size]` half /
- // bfloat16) in auto-round's INT8 per-group layout, so no conversion is
- // needed relative to the dequant fallback -- the kernel just consumes the
- // same scale tensor.
- //
- // STATUS: NEEDS-HARDWARE-VALIDATION. See
- // `sycl_tla_moe_prefill_int_dpas.hpp` for the port's provenance & the
- // on-hardware TODOs.
- if (weight_dtype == BTLA_DTYPE::S8 && !asym &&
- moe_dpas_int::moe_prefill_dpas_int_enabled() &&
- moe_dpas_int::moe_prefill_dpas_int_pergroup_shape_ok(N, K, group_size)) {
- const auto* weights_i8 = reinterpret_cast<const int8_t*>(weights);
- if (act_dtype == BTLA_DTYPE::F16) {
- using ScalarT = sycl::half;
- moe_dpas_int::moe_prefill_int_dpas_per_group_dispatch<ScalarT>(
- q, static_cast<const ScalarT*>(activations), weights_i8,
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs),
- num_tokens_per_expert, num_experts, N, K, group_size, total_tokens);
- return;
- } else if (act_dtype == BTLA_DTYPE::BF16) {
- using ScalarT = sycl::ext::oneapi::bfloat16;
- moe_dpas_int::moe_prefill_int_dpas_per_group_dispatch<ScalarT>(
- q, static_cast<const ScalarT*>(activations), weights_i8,
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs),
- num_tokens_per_expert, num_experts, N, K, group_size, total_tokens);
- return;
- }
- // Unsupported act_dtype falls through to the dequant branch below.
- }
-
- // S4-sym single-pass DPAS grouped GEMM (Variant B: per-K-group scale,
- // in-register nibble->act upcast, XMX MMA). This is the newer path that
- // reads packed `[E, N, K/2]` uint8_t nibbles directly and folds the
- // upcast into the DPAS mainloop via CuTe's `reorder(tBrB, tCrB)`, so
- // the B-side global traffic is halved vs. the S4->S8 upcast branch
- // below. Opt-in default via `ARK_MOE_PREFILL_DPAS_S4` (default ON);
- // silent fallback to the S4->S8 upcast branch (which is itself gated
- // by `ARK_MOE_PREFILL_DPAS_INT8`) or to the generic dequant path if
- // the shape gate rejects the tile geometry.
- //
- // STATUS: NEEDS-HARDWARE-VALIDATION. See
- // `sycl_tla_moe_prefill_s4_dpas.hpp` for the port's provenance & the
- // on-hardware TODOs (chief among them: `NumericArrayConverter
- // <ElementA, cutlass::int4b_t, N>` availability in the pinned
- // cutlass-sycl).
- if (weight_dtype == BTLA_DTYPE::S4_CLIP && !asym &&
- moe_dpas_s4::moe_prefill_dpas_s4_enabled() &&
- moe_dpas_s4::moe_prefill_dpas_s4_pergroup_shape_ok(N, K, group_size) &&
- (act_dtype == BTLA_DTYPE::F16 || act_dtype == BTLA_DTYPE::BF16)) {
- if (act_dtype == BTLA_DTYPE::F16) {
- using ScalarT = sycl::half;
- moe_dpas_s4::moe_prefill_s4_dpas_per_group_dispatch<ScalarT>(
- q, static_cast<const ScalarT*>(activations),
- static_cast<const uint8_t*>(weights),
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs),
- num_tokens_per_expert, num_experts, N, K, group_size, total_tokens);
- } else {
- using ScalarT = sycl::ext::oneapi::bfloat16;
- moe_dpas_s4::moe_prefill_s4_dpas_per_group_dispatch<ScalarT>(
- q, static_cast<const ScalarT*>(activations),
- static_cast<const uint8_t*>(weights),
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs),
- num_tokens_per_expert, num_experts, N, K, group_size, total_tokens);
- }
- return;
- }
-
- // INT4-sym / INT2-sym via INT8 DPAS. Rather than dequantise packed
- // nibbles/crumbs into a bf16/fp16 `[E, K, N]` workspace and then run a
- // bf16 x bf16 GEMM, we upcast the low-bit-width weights to `int8_t` in
- // a `[E, N, K]` layout and hand the resulting buffer to the same
- // per-group INT8 DPAS mainloop the S8 branch above uses. The upcast
- // writes 1 byte per element (vs. 2 bytes for the bf16/fp16 dequant),
- // and the DPAS mainloop then folds the per-K-group scale exactly the
- // same way as the S8-sym path -- reusing the packed scale tensor
- // unmodified. Silent fallback to the generic dequant path if the shape
- // predicate rejects the tile geometry, if `asym=true`, or if the caller
- // opted out via `ARK_MOE_PREFILL_DPAS_INT8=0`.
- //
- // For S4-sym specifically this branch is the *fallback* for the
- // single-pass S4 DPAS path above -- callers who disable
- // `ARK_MOE_PREFILL_DPAS_S4` land here instead of on the generic
- // dequant path, so the two-pass INT4->INT8 pipeline stays available
- // as a runtime kill-switch until the single-pass mainloop is
- // hardware-validated.
- //
- // The dequant workspace pointer we reinterpret as `int8_t*` is the same
- // caller-owned buffer used by the bf16/fp16 dequant fallback: since it
- // is sized to `E * K * N * sizeof(act_dtype)` (>= 2 bytes/element) and
- // the int8 view needs exactly `E * K * N` bytes, the reinterpretation is
- // safe and does not require a separate allocation.
- if ((weight_dtype == BTLA_DTYPE::S4_CLIP || weight_dtype == BTLA_DTYPE::S2_CLIP) && !asym &&
- moe_dpas_int::moe_prefill_dpas_int_enabled() &&
- moe_dpas_int::moe_prefill_dpas_int_pergroup_shape_ok(N, K, group_size) &&
- dequant_workspace != nullptr &&
- (act_dtype == BTLA_DTYPE::F16 || act_dtype == BTLA_DTYPE::BF16)) {
- auto* upcast_i8 = static_cast<int8_t*>(dequant_workspace);
- if (weight_dtype == BTLA_DTYPE::S4_CLIP) {
- moe_mixed_detail::launch_upcast_int4_sym_to_int8(
- q, static_cast<const uint8_t*>(weights), upcast_i8, num_experts, N, K,
- num_tokens_per_expert);
- } else {
- moe_mixed_detail::launch_upcast_int2_sym_to_int8(
- q, static_cast<const uint8_t*>(weights), upcast_i8, num_experts, N, K,
- num_tokens_per_expert);
- }
- if (act_dtype == BTLA_DTYPE::F16) {
- using ScalarT = sycl::half;
- moe_dpas_int::moe_prefill_int_dpas_per_group_dispatch<ScalarT>(
- q, static_cast<const ScalarT*>(activations), upcast_i8,
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs),
- num_tokens_per_expert, num_experts, N, K, group_size, total_tokens);
- } else {
- using ScalarT = sycl::ext::oneapi::bfloat16;
- moe_dpas_int::moe_prefill_int_dpas_per_group_dispatch<ScalarT>(
- q, static_cast<const ScalarT*>(activations), upcast_i8,
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs),
- num_tokens_per_expert, num_experts, N, K, group_size, total_tokens);
- }
- return;
- }
-
- // FP8 mixed-input DPAS grouped GEMM (Variant B: per-K-group scale,
- // in-register fp8->act upcast, XMX MMA). Opt-in default via
- // `ARK_MOE_PREFILL_DPAS_FP8` (default ON); shape gate is stricter than
- // the native path -- silent fallback to the native/dequant paths for any
- // shape that doesn't satisfy the tile preconditions. Drop-in for
- // existing auto-round FP8 checkpoints -- no re-calibration needed.
- //
- // STATUS: NEEDS-HARDWARE-VALIDATION. The port is untested; on-hardware
- // parity against `test_moe_prefill_accuracy.py::test_accuracy_fp8` must
- // be verified before this branch is trusted. See the header preamble
- // in `sycl_tla_moe_prefill_fp8_dpas.hpp` for the open questions.
- if ((weight_dtype == BTLA_DTYPE::F8_E4M3 || weight_dtype == BTLA_DTYPE::F8_E5M2) && !asym &&
- moe_dpas_fp8::moe_prefill_dpas_fp8_enabled() &&
- moe_dpas_fp8::moe_prefill_dpas_fp8_pergroup_shape_ok(N, K, group_size)) {
- const bool is_e4m3 = (weight_dtype == BTLA_DTYPE::F8_E4M3);
- if (act_dtype == BTLA_DTYPE::F16) {
- using ScalarT = sycl::half;
- if (is_e4m3) {
- moe_dpas_fp8::moe_prefill_fp8_dpas_per_group_dispatch<ScalarT, true>(
- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,
- N, K, group_size, total_tokens);
- } else {
- moe_dpas_fp8::moe_prefill_fp8_dpas_per_group_dispatch<ScalarT, false>(
- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,
- N, K, group_size, total_tokens);
- }
- return;
- } else if (act_dtype == BTLA_DTYPE::BF16) {
- using ScalarT = sycl::ext::oneapi::bfloat16;
- if (is_e4m3) {
- moe_dpas_fp8::moe_prefill_fp8_dpas_per_group_dispatch<ScalarT, true>(
- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,
- N, K, group_size, total_tokens);
- } else {
- moe_dpas_fp8::moe_prefill_fp8_dpas_per_group_dispatch<ScalarT, false>(
- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,
- N, K, group_size, total_tokens);
- }
- return;
- }
- // Unsupported act_dtype falls through to the native / dequant branches.
- }
-
- if ((weight_dtype == BTLA_DTYPE::F8_E4M3 || weight_dtype == BTLA_DTYPE::F8_E5M2) && !asym &&
- moe_prefill_native_fp8_detail::moe_prefill_native_fp8_enabled() &&
- moe_prefill_native_fp8_detail::moe_prefill_native_fp8_shape_ok(N, K, group_size)) {
- const bool is_e4m3 = (weight_dtype == BTLA_DTYPE::F8_E4M3);
- const bool use_lut = moe_dequant::fp8_decode_use_lut();
- if (act_dtype == BTLA_DTYPE::F16) {
- using ScalarT = sycl::half;
- if (is_e4m3) {
- if (use_lut) {
- moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, true, true>(
- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,
- N, K, group_size, total_tokens);
- } else {
- moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, true, false>(
- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,
- N, K, group_size, total_tokens);
- }
- } else {
- if (use_lut) {
- moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, false, true>(
- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,
- N, K, group_size, total_tokens);
- } else {
- moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, false, false>(
- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,
- N, K, group_size, total_tokens);
- }
- }
- return;
- } else if (act_dtype == BTLA_DTYPE::BF16) {
- using ScalarT = sycl::ext::oneapi::bfloat16;
- if (is_e4m3) {
- if (use_lut) {
- moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, true, true>(
- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,
- N, K, group_size, total_tokens);
- } else {
- moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, true, false>(
- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,
- N, K, group_size, total_tokens);
- }
- } else {
- if (use_lut) {
- moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, false, true>(
- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,
- N, K, group_size, total_tokens);
- } else {
- moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, false, false>(
- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,
- N, K, group_size, total_tokens);
- }
- }
- return;
- }
- // Unsupported act_dtype falls through to the generic dequant check below.
- }
-
- if (dequant_workspace == nullptr) {
- throw std::invalid_argument("moe_gemm_prefill: dequant_workspace must be non-null for quantized paths");
- }
-
- if (act_dtype == BTLA_DTYPE::F16) {
- auto* w_kn = static_cast<sycl::half*>(dequant_workspace);
- moe_mixed_detail::dequant_to_KN<sycl::half>(q, weights, scales, zeros, w_kn, weight_dtype, num_experts, N, K,
- group_size, asym, num_tokens_per_expert);
- moe_gemm(q, activations, w_kn, /*scales=*/nullptr, outputs, act_dtype, N, K, num_tokens_per_expert, num_experts);
- } else if (act_dtype == BTLA_DTYPE::BF16) {
- using BF = sycl::ext::oneapi::bfloat16;
- auto* w_kn = static_cast<BF*>(dequant_workspace);
- moe_mixed_detail::dequant_to_KN<BF>(q, weights, scales, zeros, w_kn, weight_dtype, num_experts, N, K, group_size,
- asym, num_tokens_per_expert);
- moe_gemm(q, activations, w_kn, /*scales=*/nullptr, outputs, act_dtype, N, K, num_tokens_per_expert, num_experts);
- } else {
- throw std::invalid_argument("moe_gemm_prefill: act_dtype must be F16 or BF16");
- }
-}
-
// ----------------------------------------------------------------------------
// MoE prefill Grouped GEMM -- FP8 per-tensor DPAS (Variant A).
//
@@ -1107,46 +638,6 @@ inline void moe_gemm_prefill(sycl::queue* q, void* activations, void* weights, v
// `sycl_tla_moe_prefill_fp8_dpas.hpp` for the port's provenance & the
// on-hardware TODOs.
// ----------------------------------------------------------------------------
-inline void moe_gemm_prefill_fp8_dpas(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs,
- BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,
- int* num_tokens_per_expert, int num_experts, int total_tokens) {
- if (total_tokens == 0) return;
- if (weight_dtype != BTLA_DTYPE::F8_E4M3 && weight_dtype != BTLA_DTYPE::F8_E5M2) {
- throw std::invalid_argument(
- "moe_gemm_prefill_fp8_dpas: weight_dtype must be F8_E4M3 or F8_E5M2");
- }
- const bool is_e4m3 = (weight_dtype == BTLA_DTYPE::F8_E4M3);
- if (act_dtype == BTLA_DTYPE::F16) {
- using ScalarT = sycl::half;
- if (is_e4m3) {
- moe_dpas_fp8::moe_prefill_fp8_dpas_per_tensor_dispatch<ScalarT, true>(
- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const float*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts, N,
- K, total_tokens);
- } else {
- moe_dpas_fp8::moe_prefill_fp8_dpas_per_tensor_dispatch<ScalarT, false>(
- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const float*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts, N,
- K, total_tokens);
- }
- } else if (act_dtype == BTLA_DTYPE::BF16) {
- using ScalarT = sycl::ext::oneapi::bfloat16;
- if (is_e4m3) {
- moe_dpas_fp8::moe_prefill_fp8_dpas_per_tensor_dispatch<ScalarT, true>(
- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const float*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts, N,
- K, total_tokens);
- } else {
- moe_dpas_fp8::moe_prefill_fp8_dpas_per_tensor_dispatch<ScalarT, false>(
- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),
- static_cast<const float*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts, N,
- K, total_tokens);
- }
- } else {
- throw std::invalid_argument("moe_gemm_prefill_fp8_dpas: act_dtype must be F16 or BF16");
- }
-}
-
// ----------------------------------------------------------------------------
// MoE prefill Grouped GEMM -- INT8 per-tensor DPAS (Variant A).
//
@@ -1160,31 +651,6 @@ inline void moe_gemm_prefill_fp8_dpas(sycl::queue* q, void* activations, void* w
// `sycl_tla_moe_prefill_int_dpas.hpp` for the port's provenance & the
// on-hardware TODOs.
// ----------------------------------------------------------------------------
-inline void moe_gemm_prefill_int_dpas(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs,
- BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,
- int* num_tokens_per_expert, int num_experts, int total_tokens) {
- if (total_tokens == 0) return;
- if (weight_dtype != BTLA_DTYPE::S8) {
- throw std::invalid_argument(
- "moe_gemm_prefill_int_dpas: weight_dtype must be S8");
- }
- if (act_dtype == BTLA_DTYPE::F16) {
- using ScalarT = sycl::half;
- moe_dpas_int::moe_prefill_int_dpas_per_tensor_dispatch<ScalarT>(
- q, static_cast<const ScalarT*>(activations), static_cast<const int8_t*>(weights),
- static_cast<const float*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts, N,
- K, total_tokens);
- } else if (act_dtype == BTLA_DTYPE::BF16) {
- using ScalarT = sycl::ext::oneapi::bfloat16;
- moe_dpas_int::moe_prefill_int_dpas_per_tensor_dispatch<ScalarT>(
- q, static_cast<const ScalarT*>(activations), static_cast<const int8_t*>(weights),
- static_cast<const float*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts, N,
- K, total_tokens);
- } else {
- throw std::invalid_argument("moe_gemm_prefill_int_dpas: act_dtype must be F16 or BF16");
- }
-}
-
} // namespace ark
#endif // ARK_XPU && ARK_SYCL_TLA
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_dpas.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_dpas.hpp
index 1638bf7af..5946c9f1c 100644
--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_dpas.hpp
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_dpas.hpp
@@ -679,11 +679,12 @@ CUTE_DEVICE void xe_gemm_fp8_pergroup(
template <typename T, char LayoutKind>
CUTE_DEVICE auto make_moe_tensor(T* ptr, int r, int c) {
auto shape = make_shape(r, c);
+ auto gmem_ptr = make_gmem_ptr(ptr);
if constexpr (LayoutKind == 'C')
- return make_tensor(make_gmem_ptr(ptr),
+ return make_tensor(gmem_ptr,
make_layout(shape, make_stride(_1{}, r)));
else
- return make_tensor(make_gmem_ptr(ptr),
+ return make_tensor(gmem_ptr,
make_layout(shape, make_stride(c, _1{})));
}
@@ -910,6 +911,36 @@ void MoEGEMMLauncher(sycl::queue& stream, const ElementA* activations,
// pointer `[E]` FP32.
// ---------------------------------------------------------------------------
+template <typename ScalarT, bool IsE4M3, class Policy>
+void moe_prefill_fp8_dpas_per_tensor_dispatch_policy(
+ sycl::queue* q, const ScalarT* activations, const uint8_t* weights_KN,
+ const float* scales_e, ScalarT* outputs, const int* num_tokens_per_expert,
+ int E, int N, int K, int total_tokens) {
+ if (E == 0 || N == 0 || K == 0 || total_tokens == 0) return;
+
+ compat::set_default_queue(*q);
+
+ using ElementB = std::conditional_t<IsE4M3, cutlass::float_e4m3_t,
+ cutlass::float_e5m2_t>;
+ using ElementA = cute_scalar_t<ScalarT>;
+ const auto* activations_ca =
+ reinterpret_cast<const ElementA*>(activations);
+ auto* outputs_ca = reinterpret_cast<ElementA*>(outputs);
+
+ int32_t* atomic_buffer = sycl::malloc_device<int32_t>(1, *q);
+ if (atomic_buffer == nullptr) {
+ throw std::runtime_error(
+ "moe_prefill_fp8_dpas(per-tensor): failed to allocate atomic buffer");
+ }
+
+ MoEGEMMLauncher<'R', 'R', Policy, ScaleMode::kPerTensor>(
+ *q, activations_ca, reinterpret_cast<const ElementB*>(weights_KN),
+ scales_e, static_cast<const ElementA*>(nullptr), outputs_ca, N, K,
+ num_tokens_per_expert, E, /*group_size=*/0, atomic_buffer);
+
+ sycl::free(atomic_buffer, *q);
+}
+
template <typename ScalarT, bool IsE4M3>
void moe_prefill_fp8_dpas_per_tensor_dispatch(
sycl::queue* q, const ScalarT* activations, const uint8_t* weights_KN,
@@ -963,6 +994,41 @@ void moe_prefill_fp8_dpas_per_tensor_dispatch(
// Scales `[E, N, K/group_size]` in act dtype.
// ---------------------------------------------------------------------------
+template <typename ScalarT, bool IsE4M3, class Policy>
+void moe_prefill_fp8_dpas_per_group_dispatch_policy(
+ sycl::queue* q, const ScalarT* activations, const uint8_t* weights_NK,
+ const ScalarT* scales, ScalarT* outputs, const int* num_tokens_per_expert,
+ int E, int N, int K, int group_size, int total_tokens) {
+ if (E == 0 || N == 0 || K == 0 || total_tokens == 0) return;
+ if (K % group_size != 0) {
+ throw std::invalid_argument(
+ "moe_prefill_fp8_dpas(per-group): K must be a multiple of group_size");
+ }
+
+ compat::set_default_queue(*q);
+
+ using ElementB = std::conditional_t<IsE4M3, cutlass::float_e4m3_t,
+ cutlass::float_e5m2_t>;
+ using ElementA = cute_scalar_t<ScalarT>;
+ const auto* activations_ca =
+ reinterpret_cast<const ElementA*>(activations);
+ const auto* scales_ca = reinterpret_cast<const ElementA*>(scales);
+ auto* outputs_ca = reinterpret_cast<ElementA*>(outputs);
+
+ int32_t* atomic_buffer = sycl::malloc_device<int32_t>(1, *q);
+ if (atomic_buffer == nullptr) {
+ throw std::runtime_error(
+ "moe_prefill_fp8_dpas(per-group): failed to allocate atomic buffer");
+ }
+
+ MoEGEMMLauncher<'R', 'C', Policy, ScaleMode::kPerGroup>(
+ *q, activations_ca, reinterpret_cast<const ElementB*>(weights_NK),
+ scales_ca, static_cast<const ElementA*>(nullptr), outputs_ca, N, K,
+ num_tokens_per_expert, E, group_size, atomic_buffer);
+
+ sycl::free(atomic_buffer, *q);
+}
+
template <typename ScalarT, bool IsE4M3>
void moe_prefill_fp8_dpas_per_group_dispatch(
sycl::queue* q, const ScalarT* activations, const uint8_t* weights_NK,
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_helpers.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_helpers.hpp
new file mode 100644
index 000000000..eda62655c
--- /dev/null
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_helpers.hpp
@@ -0,0 +1,28 @@
+#pragma once
+
+#include "sycl_tla_common.hpp"
+
+#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)
+
+namespace ark {
+namespace moe_prefill_fp8_detail {
+
+bool enabled();
+bool shape_ok(int N, int K, int group_size);
+void dispatch_f16_e4m3_small(const MoePrefillParams& params);
+void dispatch_f16_e4m3_mid(const MoePrefillParams& params);
+void dispatch_f16_e4m3_large(const MoePrefillParams& params);
+void dispatch_f16_e5m2_small(const MoePrefillParams& params);
+void dispatch_f16_e5m2_mid(const MoePrefillParams& params);
+void dispatch_f16_e5m2_large(const MoePrefillParams& params);
+void dispatch_bf16_e4m3_small(const MoePrefillParams& params);
+void dispatch_bf16_e4m3_mid(const MoePrefillParams& params);
+void dispatch_bf16_e4m3_large(const MoePrefillParams& params);
+void dispatch_bf16_e5m2_small(const MoePrefillParams& params);
+void dispatch_bf16_e5m2_mid(const MoePrefillParams& params);
+void dispatch_bf16_e5m2_large(const MoePrefillParams& params);
+
+} // namespace moe_prefill_fp8_detail
+} // namespace ark
+
+#endif
\ No newline at end of file
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_tensor_helpers.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_tensor_helpers.hpp
new file mode 100644
index 000000000..2dd584970
--- /dev/null
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_tensor_helpers.hpp
@@ -0,0 +1,26 @@
+#pragma once
+
+#include "sycl_tla_common.hpp"
+
+#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)
+
+namespace ark {
+namespace moe_prefill_fp8_tensor_detail {
+
+void dispatch_f16_e4m3_small(const MoePrefillParams& params);
+void dispatch_f16_e4m3_mid(const MoePrefillParams& params);
+void dispatch_f16_e4m3_large(const MoePrefillParams& params);
+void dispatch_f16_e5m2_small(const MoePrefillParams& params);
+void dispatch_f16_e5m2_mid(const MoePrefillParams& params);
+void dispatch_f16_e5m2_large(const MoePrefillParams& params);
+void dispatch_bf16_e4m3_small(const MoePrefillParams& params);
+void dispatch_bf16_e4m3_mid(const MoePrefillParams& params);
+void dispatch_bf16_e4m3_large(const MoePrefillParams& params);
+void dispatch_bf16_e5m2_small(const MoePrefillParams& params);
+void dispatch_bf16_e5m2_mid(const MoePrefillParams& params);
+void dispatch_bf16_e5m2_large(const MoePrefillParams& params);
+
+} // namespace moe_prefill_fp8_tensor_detail
+} // namespace ark
+
+#endif
\ No newline at end of file
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int8_helpers.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int8_helpers.hpp
new file mode 100644
index 000000000..67d75928d
--- /dev/null
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int8_helpers.hpp
@@ -0,0 +1,28 @@
+#pragma once
+
+#include "sycl_tla_common.hpp"
+
+#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)
+
+namespace ark {
+namespace moe_prefill_int8_detail {
+
+bool enabled();
+bool shape_ok(int N, int K, int group_size);
+void dispatch_f16_small(const MoePrefillParams& params);
+void dispatch_f16_mid(const MoePrefillParams& params);
+void dispatch_f16_large(const MoePrefillParams& params);
+void dispatch_bf16_small(const MoePrefillParams& params);
+void dispatch_bf16_mid(const MoePrefillParams& params);
+void dispatch_bf16_large(const MoePrefillParams& params);
+void dispatch_lowbit_f16_small(const MoePrefillParams& params);
+void dispatch_lowbit_f16_mid(const MoePrefillParams& params);
+void dispatch_lowbit_f16_large(const MoePrefillParams& params);
+void dispatch_lowbit_bf16_small(const MoePrefillParams& params);
+void dispatch_lowbit_bf16_mid(const MoePrefillParams& params);
+void dispatch_lowbit_bf16_large(const MoePrefillParams& params);
+
+} // namespace moe_prefill_int8_detail
+} // namespace ark
+
+#endif
\ No newline at end of file
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_dpas.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_dpas.hpp
index 2e05dfa44..d7f976011 100644
--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_dpas.hpp
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_dpas.hpp
@@ -761,6 +761,34 @@ void MoEGEMMLauncher_int(sycl::queue& stream, const ElementA* activations,
// Scale pointer `[E]` FP32 (one per-expert scalar).
// ---------------------------------------------------------------------------
+template <typename ScalarT, class Policy>
+void moe_prefill_int_dpas_per_tensor_dispatch_policy(
+ sycl::queue* q, const ScalarT* activations, const int8_t* weights_KN,
+ const float* scales_e, ScalarT* outputs, const int* num_tokens_per_expert,
+ int E, int N, int K, int total_tokens) {
+ if (E == 0 || N == 0 || K == 0 || total_tokens == 0) return;
+
+ compat::set_default_queue(*q);
+
+ using ElementA = cute_scalar_t<ScalarT>;
+ const auto* activations_ca =
+ reinterpret_cast<const ElementA*>(activations);
+ auto* outputs_ca = reinterpret_cast<ElementA*>(outputs);
+
+ int32_t* atomic_buffer = sycl::malloc_device<int32_t>(1, *q);
+ if (atomic_buffer == nullptr) {
+ throw std::runtime_error(
+ "moe_prefill_int_dpas(per-tensor): failed to allocate atomic buffer");
+ }
+
+ MoEGEMMLauncher_int<'R', 'R', Policy, ScaleMode::kPerTensor>(
+ *q, activations_ca, weights_KN, scales_e,
+ static_cast<const ElementA*>(nullptr), outputs_ca, N, K,
+ num_tokens_per_expert, E, /*group_size=*/0, atomic_buffer);
+
+ sycl::free(atomic_buffer, *q);
+}
+
template <typename ScalarT>
void moe_prefill_int_dpas_per_tensor_dispatch(
sycl::queue* q, const ScalarT* activations, const int8_t* weights_KN,
@@ -819,6 +847,40 @@ void moe_prefill_int_dpas_per_tensor_dispatch(
// as the FP8 per-group path.
// ---------------------------------------------------------------------------
+template <typename ScalarT, class Policy>
+void moe_prefill_int_dpas_per_group_dispatch_policy(
+ sycl::queue* q, const ScalarT* activations, const int8_t* weights_NK,
+ const ScalarT* scales, ScalarT* outputs,
+ const int* num_tokens_per_expert, int E, int N, int K, int group_size,
+ int total_tokens) {
+ if (E == 0 || N == 0 || K == 0 || total_tokens == 0) return;
+ if (K % group_size != 0) {
+ throw std::invalid_argument(
+ "moe_prefill_int_dpas(per-group): K must be a multiple of group_size");
+ }
+
+ compat::set_default_queue(*q);
+
+ using ElementA = cute_scalar_t<ScalarT>;
+ const auto* activations_ca =
+ reinterpret_cast<const ElementA*>(activations);
+ const auto* scales_ca = reinterpret_cast<const ElementA*>(scales);
+ auto* outputs_ca = reinterpret_cast<ElementA*>(outputs);
+
+ int32_t* atomic_buffer = sycl::malloc_device<int32_t>(1, *q);
+ if (atomic_buffer == nullptr) {
+ throw std::runtime_error(
+ "moe_prefill_int_dpas(per-group): failed to allocate atomic buffer");
+ }
+
+ MoEGEMMLauncher_int<'R', 'C', Policy, ScaleMode::kPerGroup>(
+ *q, activations_ca, weights_NK, scales_ca,
+ static_cast<const ElementA*>(nullptr), outputs_ca, N, K,
+ num_tokens_per_expert, E, group_size, atomic_buffer);
+
+ sycl::free(atomic_buffer, *q);
+}
+
template <typename ScalarT>
void moe_prefill_int_dpas_per_group_dispatch(
sycl::queue* q, const ScalarT* activations, const int8_t* weights_NK,
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_tensor_helpers.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_tensor_helpers.hpp
new file mode 100644
index 000000000..8d0e2026a
--- /dev/null
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_tensor_helpers.hpp
@@ -0,0 +1,20 @@
+#pragma once
+
+#include "sycl_tla_common.hpp"
+
+#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)
+
+namespace ark {
+namespace moe_prefill_int_tensor_detail {
+
+void dispatch_f16_small(const MoePrefillParams& params);
+void dispatch_f16_mid(const MoePrefillParams& params);
+void dispatch_f16_large(const MoePrefillParams& params);
+void dispatch_bf16_small(const MoePrefillParams& params);
+void dispatch_bf16_mid(const MoePrefillParams& params);
+void dispatch_bf16_large(const MoePrefillParams& params);
+
+} // namespace moe_prefill_int_tensor_detail
+} // namespace ark
+
+#endif
\ No newline at end of file
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_dpas.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_dpas.hpp
index fad8d87eb..0a2ea3d70 100644
--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_dpas.hpp
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_dpas.hpp
@@ -603,6 +603,46 @@ void MoEGEMMLauncher_s4(sycl::queue& stream, const ElementA* activations,
// Scales `[E, N, K/group_size]` in act dtype (half / bfloat16).
// ---------------------------------------------------------------------------
+template <typename ScalarT, class Policy>
+void moe_prefill_s4_dpas_per_group_dispatch_policy(
+ sycl::queue* q, const ScalarT* activations, const uint8_t* weights_NKp,
+ const ScalarT* scales, ScalarT* outputs,
+ const int* num_tokens_per_expert, int E, int N, int K, int group_size,
+ int total_tokens) {
+ if (E == 0 || N == 0 || K == 0 || total_tokens == 0) return;
+ if (K % group_size != 0) {
+ throw std::invalid_argument(
+ "moe_prefill_s4_dpas(per-group): K must be a multiple of group_size");
+ }
+ if ((K & 1) != 0) {
+ throw std::invalid_argument(
+ "moe_prefill_s4_dpas(per-group): K must be even (packed nibbles)");
+ }
+
+ compat::set_default_queue(*q);
+
+ using ElementA = cute_scalar_t<ScalarT>;
+ const auto* activations_ca =
+ reinterpret_cast<const ElementA*>(activations);
+ const auto* scales_ca = reinterpret_cast<const ElementA*>(scales);
+ auto* outputs_ca = reinterpret_cast<ElementA*>(outputs);
+ const auto* weights_i4 =
+ reinterpret_cast<const cutlass::int4b_t*>(weights_NKp);
+
+ int32_t* atomic_buffer = sycl::malloc_device<int32_t>(1, *q);
+ if (atomic_buffer == nullptr) {
+ throw std::runtime_error(
+ "moe_prefill_s4_dpas(per-group): failed to allocate atomic buffer");
+ }
+
+ MoEGEMMLauncher_s4<'R', 'C', Policy>(
+ *q, activations_ca, weights_i4, scales_ca,
+ static_cast<const ElementA*>(nullptr), outputs_ca, N, K,
+ num_tokens_per_expert, E, group_size, atomic_buffer);
+
+ sycl::free(atomic_buffer, *q);
+}
+
template <typename ScalarT>
void moe_prefill_s4_dpas_per_group_dispatch(
sycl::queue* q, const ScalarT* activations, const uint8_t* weights_NKp,
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_helpers.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_helpers.hpp
new file mode 100644
index 000000000..6010579c1
--- /dev/null
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_helpers.hpp
@@ -0,0 +1,22 @@
+#pragma once
+
+#include "sycl_tla_common.hpp"
+
+#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)
+
+namespace ark {
+namespace moe_prefill_s4_detail {
+
+bool enabled();
+bool shape_ok(int N, int K, int group_size);
+void dispatch_f16_small(const MoePrefillParams& params);
+void dispatch_f16_mid(const MoePrefillParams& params);
+void dispatch_f16_large(const MoePrefillParams& params);
+void dispatch_bf16_small(const MoePrefillParams& params);
+void dispatch_bf16_mid(const MoePrefillParams& params);
+void dispatch_bf16_large(const MoePrefillParams& params);
+
+} // namespace moe_prefill_s4_detail
+} // namespace ark
+
+#endif
\ No newline at end of file
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_upcast.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_upcast.hpp
new file mode 100644
index 000000000..1e2ff7bdb
--- /dev/null
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_upcast.hpp
@@ -0,0 +1,159 @@
+// SYCL MoE prefill low-bit to INT8 upcast kernels
+
+#pragma once
+
+#include <cstddef>
+#include <cstdint>
+#include <stdexcept>
+
+#ifdef ARK_XPU
+#include <sycl/sycl.hpp>
+#endif
+
+#include "sycl_tla_moe_dequant.hpp"
+
+#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)
+
+namespace ark {
+namespace moe_mixed_detail {
+
+class MoEUpcastInt4SymToInt8Kernel;
+class MoEUpcastInt2SymToInt8Kernel;
+class MoEUpcastInt4SymToInt8KernelFast;
+class MoEUpcastInt2SymToInt8KernelFast;
+
+constexpr int UPCAST_WG_N = 32;
+constexpr int PACK_K_INT4 = 2;
+constexpr int PACK_K_INT2 = 4;
+constexpr int PACK_K_INT4_FAST = 8;
+constexpr int PACK_K_INT2_FAST = 8;
+
+inline void launch_upcast_int4_sym_to_int8(sycl::queue* q, const uint8_t* weights_NKp, int8_t* weights_i8_NK,
+ int E, int N, int K,
+ const int* num_tokens_per_expert = nullptr) {
+ if (E == 0 || N == 0 || K == 0) return;
+ if ((K & 1) != 0) {
+ throw std::invalid_argument("moe_gemm_prefill(int4->int8 upcast): K must be even");
+ }
+ const int k_packed = K / 2;
+
+ if ((K % PACK_K_INT4_FAST) == 0) {
+ const int k_words = K / PACK_K_INT4_FAST;
+ sycl::range<3> global_fast{static_cast<size_t>(E), static_cast<size_t>(k_words),
+ static_cast<size_t>((N + UPCAST_WG_N - 1) / UPCAST_WG_N) * UPCAST_WG_N};
+ sycl::range<3> local_fast{1, 1, static_cast<size_t>(UPCAST_WG_N)};
+
+ q->parallel_for<MoEUpcastInt4SymToInt8KernelFast>(
+ sycl::nd_range<3>(global_fast, local_fast), [=](sycl::nd_item<3> it) {
+ const int e = static_cast<int>(it.get_global_id(0));
+ if (num_tokens_per_expert != nullptr && num_tokens_per_expert[e] == 0) return;
+ const int kw = static_cast<int>(it.get_global_id(1));
+ const int n = static_cast<int>(it.get_global_id(2));
+ if (n >= N) return;
+ const int k_base = kw * PACK_K_INT4_FAST;
+ const size_t row_kp_base = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * k_packed +
+ static_cast<size_t>(kw) * 4;
+ const uint32_t packed =
+ static_cast<uint32_t>(weights_NKp[row_kp_base + 0]) |
+ (static_cast<uint32_t>(weights_NKp[row_kp_base + 1]) << 8) |
+ (static_cast<uint32_t>(weights_NKp[row_kp_base + 2]) << 16) |
+ (static_cast<uint32_t>(weights_NKp[row_kp_base + 3]) << 24);
+ int qv[8];
+ moe_dequant::decode_int4_octet<false>(packed, qv);
+ const size_t out_row = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * K;
+#pragma unroll
+ for (int j = 0; j < PACK_K_INT4_FAST; ++j) {
+ weights_i8_NK[out_row + static_cast<size_t>(k_base + j)] = static_cast<int8_t>(qv[j]);
+ }
+ });
+ return;
+ }
+
+ sycl::range<3> global{static_cast<size_t>(E), static_cast<size_t>(k_packed),
+ static_cast<size_t>((N + UPCAST_WG_N - 1) / UPCAST_WG_N) * UPCAST_WG_N};
+ sycl::range<3> local{1, 1, static_cast<size_t>(UPCAST_WG_N)};
+
+ q->parallel_for<MoEUpcastInt4SymToInt8Kernel>(
+ sycl::nd_range<3>(global, local), [=](sycl::nd_item<3> it) {
+ const int e = static_cast<int>(it.get_global_id(0));
+ if (num_tokens_per_expert != nullptr && num_tokens_per_expert[e] == 0) return;
+ const int kp = static_cast<int>(it.get_global_id(1));
+ const int n = static_cast<int>(it.get_global_id(2));
+ if (n >= N) return;
+ const int k_base = kp * PACK_K_INT4;
+ const uint8_t packed =
+ weights_NKp[(static_cast<size_t>(e) * N + static_cast<size_t>(n)) * k_packed + static_cast<size_t>(kp)];
+ int q_lo, q_hi;
+ moe_dequant::decode_int4_pair<false>(packed, q_lo, q_hi);
+ const size_t out_row = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * K;
+ weights_i8_NK[out_row + static_cast<size_t>(k_base)] = static_cast<int8_t>(q_lo);
+ weights_i8_NK[out_row + static_cast<size_t>(k_base + 1)] = static_cast<int8_t>(q_hi);
+ });
+}
+
+inline void launch_upcast_int2_sym_to_int8(sycl::queue* q, const uint8_t* weights_NKp, int8_t* weights_i8_NK,
+ int E, int N, int K,
+ const int* num_tokens_per_expert = nullptr) {
+ if (E == 0 || N == 0 || K == 0) return;
+ if ((K & 3) != 0) {
+ throw std::invalid_argument("moe_gemm_prefill(int2->int8 upcast): K must be a multiple of 4");
+ }
+ const int k_packed = K / 4;
+
+ if ((K % PACK_K_INT2_FAST) == 0) {
+ const int k_words = K / PACK_K_INT2_FAST;
+ sycl::range<3> global_fast{static_cast<size_t>(E), static_cast<size_t>(k_words),
+ static_cast<size_t>((N + UPCAST_WG_N - 1) / UPCAST_WG_N) * UPCAST_WG_N};
+ sycl::range<3> local_fast{1, 1, static_cast<size_t>(UPCAST_WG_N)};
+
+ q->parallel_for<MoEUpcastInt2SymToInt8KernelFast>(
+ sycl::nd_range<3>(global_fast, local_fast), [=](sycl::nd_item<3> it) {
+ const int e = static_cast<int>(it.get_global_id(0));
+ if (num_tokens_per_expert != nullptr && num_tokens_per_expert[e] == 0) return;
+ const int kw = static_cast<int>(it.get_global_id(1));
+ const int n = static_cast<int>(it.get_global_id(2));
+ if (n >= N) return;
+ const int k_base = kw * PACK_K_INT2_FAST;
+ const size_t row_kp_base = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * k_packed +
+ static_cast<size_t>(kw) * 2;
+ const uint16_t packed = static_cast<uint16_t>(static_cast<uint32_t>(weights_NKp[row_kp_base + 0]) |
+ (static_cast<uint32_t>(weights_NKp[row_kp_base + 1]) << 8));
+ int qv[8];
+ moe_dequant::decode_int2_octet<false>(packed, qv);
+ const size_t out_row = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * K;
+#pragma unroll
+ for (int j = 0; j < PACK_K_INT2_FAST; ++j) {
+ weights_i8_NK[out_row + static_cast<size_t>(k_base + j)] = static_cast<int8_t>(qv[j]);
+ }
+ });
+ return;
+ }
+
+ sycl::range<3> global{static_cast<size_t>(E), static_cast<size_t>(k_packed),
+ static_cast<size_t>((N + UPCAST_WG_N - 1) / UPCAST_WG_N) * UPCAST_WG_N};
+ sycl::range<3> local{1, 1, static_cast<size_t>(UPCAST_WG_N)};
+
+ q->parallel_for<MoEUpcastInt2SymToInt8Kernel>(
+ sycl::nd_range<3>(global, local), [=](sycl::nd_item<3> it) {
+ const int e = static_cast<int>(it.get_global_id(0));
+ if (num_tokens_per_expert != nullptr && num_tokens_per_expert[e] == 0) return;
+ const int kp = static_cast<int>(it.get_global_id(1));
+ const int n = static_cast<int>(it.get_global_id(2));
+ if (n >= N) return;
+ const int k_base = kp * PACK_K_INT2;
+ const uint8_t packed =
+ weights_NKp[(static_cast<size_t>(e) * N + static_cast<size_t>(n)) * k_packed + static_cast<size_t>(kp)];
+ int qv[4];
+ moe_dequant::decode_int2_quad<false>(packed, qv);
+ const size_t out_row = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * K;
+#pragma unroll
+ for (int j = 0; j < PACK_K_INT2; ++j) {
+ weights_i8_NK[out_row + static_cast<size_t>(k_base + j)] = static_cast<int8_t>(qv[j]);
+ }
+ });
+}
+
+} // namespace moe_mixed_detail
+} // namespace ark
+
+#endif
\ No newline at end of file
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm.hpp
index 5a6dec850..b339ccd23 100644
--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm.hpp
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm.hpp
@@ -19,6 +19,7 @@
#endif
#include "utils.hpp"
+#include "sycl_tla_common.hpp"
namespace ark {
@@ -444,46 +445,24 @@ void run_typed(sycl::queue* q, int m, int n, int k, const int8_t* a, const int8_
launch_igemm_kblock<ElementOut>(q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize, blks);
}
-} // namespace sycl_tla_s8_detail
-
-inline void sycl_tla_igemm_s8s8_dequant(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,
- BTLA_DTYPE ct, const void* scale_a, const void* scale_b, const void* bias,
- int blocksize) {
- if (!q) throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: queue must not be null");
- if (!a || !b || !c || !scale_a || !scale_b) {
- throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: input pointers must not be null");
- }
- if (m <= 0 || n <= 0 || k <= 0) return;
-
- bool k_block = !(blocksize == k || blocksize == -1);
+template <class ElementOut, int TileM, int TileN, class SGLayout>
+void run_igemm_normal_fixed_tile(sycl::queue* q, int m, int n, int k, const int8_t* a, const int8_t* b,
+ ElementOut* c, const ElementOut* scale_a, const ElementOut* scale_b,
+ const ElementOut* bias) {
+ launch_igemm_tile<false, ElementOut, TileM, TileN, SGLayout>(
+ q, m, n, k, k, k, a, b, c, nullptr, scale_a, scale_b, bias, 0, 1);
+}
- switch (ct) {
- case BTLA_DTYPE::F32:
- sycl_tla_s8_detail::run_typed(q, m, n, k, static_cast<const int8_t*>(a), static_cast<const int8_t*>(b),
- static_cast<float*>(c), static_cast<const float*>(scale_a),
- static_cast<const float*>(scale_b), static_cast<const float*>(bias), blocksize);
- return;
- case BTLA_DTYPE::F16:
- sycl_tla_s8_detail::run_typed(q, m, n, k, static_cast<const int8_t*>(a), static_cast<const int8_t*>(b),
- static_cast<cute::half_t*>(c), static_cast<const cute::half_t*>(scale_a),
- static_cast<const cute::half_t*>(scale_b), static_cast<const cute::half_t*>(bias),
- blocksize);
- return;
- case BTLA_DTYPE::BF16:
- if (k_block) {
- throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: k-block path supports only F32/F16 output");
- }
- sycl_tla_s8_detail::run_typed(q, m, n, k, static_cast<const int8_t*>(a), static_cast<const int8_t*>(b),
- static_cast<cute::bfloat16_t*>(c),
- static_cast<const cute::bfloat16_t*>(scale_a),
- static_cast<const cute::bfloat16_t*>(scale_b),
- static_cast<const cute::bfloat16_t*>(bias), blocksize);
- return;
- default:
- throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: unsupported output dtype");
- }
+template <class ElementOut, int TileM, int TileN, class SGLayout>
+void run_igemm_kblock_fixed_tile(sycl::queue* q, int m, int n, int k, const int8_t* a, const int8_t* b,
+ ElementOut* c, const ElementOut* scale_a, const ElementOut* scale_b,
+ const ElementOut* bias, int blocksize) {
+ launch_igemm_kblock_tile_dispatch<ElementOut, TileM, TileN, SGLayout>(
+ q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize, k / blocksize);
}
+} // namespace sycl_tla_s8_detail
+
#endif // ARK_XPU && ARK_SYCL_TLA
} // namespace ark
\ No newline at end of file
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm_fixed.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm_fixed.hpp
new file mode 100644
index 000000000..f30e58a75
--- /dev/null
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm_fixed.hpp
@@ -0,0 +1,32 @@
+#pragma once
+
+#include "sycl_tla_s8_gemm.hpp"
+
+#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)
+
+#define ARK_DEFINE_S8_NORMAL_ROUTE(name, element, tile_m, tile_n, layout) \
+ namespace ark { \
+ namespace sycl_tla_s8_detail { \
+ void name(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, const void* scale_a, \
+ const void* scale_b, const void* bias) { \
+ run_igemm_normal_fixed_tile<element, tile_m, tile_n, layout>( \
+ q, m, n, k, static_cast<const int8_t*>(a), static_cast<const int8_t*>(b), static_cast<element*>(c), \
+ static_cast<const element*>(scale_a), static_cast<const element*>(scale_b), static_cast<const element*>(bias)); \
+ } \
+ } \
+ }
+
+#define ARK_DEFINE_S8_KBLOCK_ROUTE(name, element, tile_m, tile_n, layout) \
+ namespace ark { \
+ namespace sycl_tla_s8_detail { \
+ void name(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, const void* scale_a, \
+ const void* scale_b, const void* bias, int blocksize) { \
+ run_igemm_kblock_fixed_tile<element, tile_m, tile_n, layout>( \
+ q, m, n, k, static_cast<const int8_t*>(a), static_cast<const int8_t*>(b), static_cast<element*>(c), \
+ static_cast<const element*>(scale_a), static_cast<const element*>(scale_b), static_cast<const element*>(bias), \
+ blocksize); \
+ } \
+ } \
+ }
+
+#endif
\ No newline at end of file
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm_helpers.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm_helpers.hpp
new file mode 100644
index 000000000..580995406
--- /dev/null
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm_helpers.hpp
@@ -0,0 +1,53 @@
+#pragma once
+
+#include "sycl_tla_common.hpp"
+
+#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)
+
+#include <stdexcept>
+
+namespace ark {
+namespace sycl_tla_s8_detail {
+
+#define ARK_DECLARE_S8_NORMAL_ROUTES(prefix) \
+ void prefix##_normal_small(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \
+ const void* scale_a, const void* scale_b, const void* bias); \
+ void prefix##_normal_mid(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \
+ const void* scale_a, const void* scale_b, const void* bias); \
+ void prefix##_normal_medium(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \
+ const void* scale_a, const void* scale_b, const void* bias); \
+ void prefix##_normal_large(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \
+ const void* scale_a, const void* scale_b, const void* bias)
+
+#define ARK_DECLARE_S8_KBLOCK_ROUTES(prefix) \
+ void prefix##_kblock_small(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \
+ const void* scale_a, const void* scale_b, const void* bias, int blocksize); \
+ void prefix##_kblock_mid(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \
+ const void* scale_a, const void* scale_b, const void* bias, int blocksize); \
+ void prefix##_kblock_medium(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \
+ const void* scale_a, const void* scale_b, const void* bias, int blocksize); \
+ void prefix##_kblock_large(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \
+ const void* scale_a, const void* scale_b, const void* bias, int blocksize)
+
+ARK_DECLARE_S8_NORMAL_ROUTES(run_f16);
+ARK_DECLARE_S8_NORMAL_ROUTES(run_bf16);
+ARK_DECLARE_S8_NORMAL_ROUTES(run_f32);
+ARK_DECLARE_S8_KBLOCK_ROUTES(run_f16);
+ARK_DECLARE_S8_KBLOCK_ROUTES(run_f32);
+
+#undef ARK_DECLARE_S8_NORMAL_ROUTES
+#undef ARK_DECLARE_S8_KBLOCK_ROUTES
+
+inline void validate_kblock_args(int k, int blocksize) {
+ if (blocksize <= 0 || k % blocksize != 0) {
+ throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: blocksize must divide k");
+ }
+ if (blocksize % 64 != 0) {
+ throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: k-block blocksize must be a multiple of 64");
+ }
+}
+
+} // namespace sycl_tla_s8_detail
+} // namespace ark
+
+#endif
\ No newline at end of file
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_sdpa.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_sdpa.hpp
index aabe8bff0..ef91b238b 100644
--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_sdpa.hpp
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_sdpa.hpp
@@ -364,7 +364,7 @@ struct FMHAConfig {
cute::conditional_t<is_void_v<SubgroupLayoutPV_>,
decltype(cutlass::fmha::collective::get_sg_layout_pv(SubgroupLayoutQK{})), SubgroupLayoutPV_>;
- template <bool isVarLen, bool CachedKV, bool PagedKV, class Scheduler>
+ template <bool HasMask, bool isVarLen, bool CachedKV, bool PagedKV, class Scheduler>
static int run(const Options& options) {
//
// Run examples
@@ -420,7 +420,7 @@ struct FMHAConfig {
CUTLASS_CHECK(runner.run(options, hw_info));
} else {
- if (options.mask) {
+ if constexpr (HasMask) {
// Mainloop
using CollectiveMainloop =
cutlass::fmha::collective::SDPAFwdMainloop<MainloopDispatchPolicy, Causal, true, CachedKV, PagedKV,
@@ -472,8 +472,8 @@ struct FMHAConfig {
return 0;
}
+ template <bool HasMask, bool IsVarLen>
static int run(const Options& options) {
- bool cached_kv = options.seq_len_kv_cache > 0;
if constexpr (persistent) {
if (options.use_paged_kv || options.seq_len_kv_cache > 0) {
std::cerr
@@ -481,23 +481,13 @@ struct FMHAConfig {
<< std::endl;
return -1;
}
- return run<false, false, false, cutlass::fmha::kernel::XeFHMAIndividualPersistentTileScheduler>(options);
- } else if (options.use_paged_kv && !options.varlen) {
- throw std::runtime_error("Paged KV without varlen is not supported yet");
- // return run<false, true, true, cutlass::fmha::kernel::XeFHMAIndividualTileScheduler>(options);
- } else if (!options.use_paged_kv && options.varlen && !cached_kv) {
- return run<true, false, false, cutlass::fmha::kernel::XeFHMAIndividualTileScheduler>(options);
- } else if (!options.use_paged_kv && !options.varlen && !cached_kv) {
- return run<false, false, false, cutlass::fmha::kernel::XeFHMAIndividualTileScheduler>(options);
- } else if (!options.use_paged_kv && options.varlen && cached_kv) {
- throw std::runtime_error("Varlen with cached KV but without paged KV is not supported yet");
- // return run<true, true, false, cutlass::fmha::kernel::XeFHMAIndividualTileScheduler>(options);
- } else if (!options.use_paged_kv && !options.varlen && cached_kv) {
- throw std::runtime_error("Cached KV without varlen is not supported yet");
- // return run<false, true, false, cutlass::fmha::kernel::XeFHMAIndividualTileScheduler>(options);
+ return run<HasMask, false, false, false, cutlass::fmha::kernel::XeFHMAIndividualPersistentTileScheduler>(
+ options);
} else {
- throw std::runtime_error("The combination of options is not supported yet");
- // return run<true, true, true, cutlass::fmha::kernel::XeFHMAIndividualTileScheduler>(options);
+ if (options.use_paged_kv || options.seq_len_kv_cache > 0) {
+ throw std::runtime_error("Paged or cached KV is not supported yet");
+ }
+ return run<HasMask, IsVarLen, false, false, cutlass::fmha::kernel::XeFHMAIndividualTileScheduler>(options);
}
}
};
@@ -859,7 +849,8 @@ struct SageConfig {
// Prefill Kernel Launch
// ========================================================================
-template <typename ElementQ, typename ElementK, typename ElementV, bool persistent = false>
+template <bool Causal, bool HasMask, bool IsVarLen, typename ElementQ, typename ElementK, typename ElementV,
+ bool persistent = false>
inline int launch_prefill_kernel_128(Options const& options) {
constexpr int PipelineStages = 2;
constexpr int PipelineStages1 = 2;
@@ -871,10 +862,14 @@ inline int launch_prefill_kernel_128(Options const& options) {
using ShapePV1 = Shape<_256, _32, _32>;
using ShapeOut1 = Shape<_256, _128>;
using SubgroupLayoutQK1 = Layout<Shape<_16, _1, _1>>;
- return options.is_causal ? FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
- /*persistent=*/false, ElementQ, ElementK, ElementV>::run(options)
- : FMHAConfig<false, ShapeQK1, ShapePV1, ShapeOut1, SubgroupLayoutQK1, void, PipelineStages1,
- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options);
+ if constexpr (Causal) {
+ return FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
+ /*persistent=*/false, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(options);
+ } else {
+ return FMHAConfig<false, ShapeQK1, ShapePV1, ShapeOut1, SubgroupLayoutQK1, void, PipelineStages1,
+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(
+ options);
+ }
}
template <typename ElementQ, typename ElementK, typename ElementV, typename ElementO = ElementV, bool UseInt8PV = false,
@@ -917,7 +912,8 @@ inline int launch_sage_prefill_kernel_64(Options const& options) {
/*persistent=*/false, ElementQ, ElementK, ElementV, ElementO>::run(options);
}
-template <typename ElementQ, typename ElementK, typename ElementV, bool persistent = false>
+template <bool Causal, bool HasMask, bool IsVarLen, typename ElementQ, typename ElementK, typename ElementV,
+ bool persistent = false>
inline int launch_prefill_kernel_64(Options const& options) {
constexpr int PipelineStages = 1;
using ShapeQK = Shape<_256, _64, _32>;
@@ -928,39 +924,54 @@ inline int launch_prefill_kernel_64(Options const& options) {
using ShapePV1 = Shape<_128, _32, _64>;
using ShapeOut1 = Shape<_128, _64>;
using SubgroupLayoutQK1 = Layout<Shape<_8, _1, _1>>;
- return options.is_causal ? FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
- /*persistent=*/false, ElementQ, ElementK, ElementV>::run(options)
- : FMHAConfig<false, ShapeQK1, ShapePV1, ShapeOut1, SubgroupLayoutQK1, void, PipelineStages,
- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options);
+ if constexpr (Causal) {
+ return FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
+ /*persistent=*/false, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(options);
+ } else {
+ return FMHAConfig<false, ShapeQK1, ShapePV1, ShapeOut1, SubgroupLayoutQK1, void, PipelineStages,
+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(
+ options);
+ }
}
-template <typename ElementQ, typename ElementK, typename ElementV, bool persistent = false>
+template <bool Causal, bool HasMask, bool IsVarLen, typename ElementQ, typename ElementK, typename ElementV,
+ bool persistent = false>
inline int launch_prefill_kernel_192(Options const& options) {
constexpr int PipelineStages = 2;
using ShapeQK = Shape<_256, _64, _32>;
using ShapePV = Shape<_256, _32, _64>;
using ShapeOut = Shape<_256, _192>;
using SubgroupLayoutQK = Layout<Shape<_32, _1, _1>>;
- return options.is_causal ? FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
- /*persistent=*/false, ElementQ, ElementK, ElementV>::run(options)
- : FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options);
+ if constexpr (Causal) {
+ return FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
+ /*persistent=*/false, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(options);
+ } else {
+ return FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(
+ options);
+ }
}
-template <typename ElementQ, typename ElementK, typename ElementV, bool persistent = false>
+template <bool Causal, bool HasMask, bool IsVarLen, typename ElementQ, typename ElementK, typename ElementV,
+ bool persistent = false>
inline int launch_prefill_kernel_96(Options const& options) {
constexpr int PipelineStages = 2;
using ShapeQK = Shape<_128, _64, _32>;
using ShapePV = Shape<_128, _32, _64>;
using ShapeOut = Shape<_128, _96>;
using SubgroupLayoutQK = Layout<Shape<_8, _1, _1>>;
- return options.is_causal ? FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
- /*persistent=*/false, ElementQ, ElementK, ElementV>::run(options)
- : FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options);
+ if constexpr (Causal) {
+ return FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
+ /*persistent=*/false, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(options);
+ } else {
+ return FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(
+ options);
+ }
}
-template <typename ElementQ, typename ElementK, typename ElementV, bool persistent = false>
+template <bool Causal, bool HasMask, bool IsVarLen, typename ElementQ, typename ElementK, typename ElementV,
+ bool persistent = false>
inline int launch_decode_kernel_128(Options const& options) {
constexpr int PipelineStages = 1;
using NUM_SG = std::conditional_t<persistent, _16, _8>;
@@ -969,13 +980,19 @@ inline int launch_decode_kernel_128(Options const& options) {
using ShapePV = Shape<_1, _32, KV_TILE_SIZE>;
using ShapeOut = Shape<_1, _128>;
using SubgroupLayoutQK = Layout<Shape<_1, NUM_SG, _1>>;
- return options.is_causal ? FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options)
- : FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options);
+ if constexpr (Causal) {
+ return FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(
+ options);
+ } else {
+ return FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(
+ options);
+ }
}
-template <typename ElementQ, typename ElementK, typename ElementV, bool persistent = false>
+template <bool Causal, bool HasMask, bool IsVarLen, typename ElementQ, typename ElementK, typename ElementV,
+ bool persistent = false>
inline int launch_decode_kernel_64(Options const& options) {
constexpr int PipelineStages = 1;
using NUM_SG = std::conditional_t<persistent, _16, _8>;
@@ -984,13 +1001,19 @@ inline int launch_decode_kernel_64(Options const& options) {
using ShapePV = Shape<_1, _32, KV_TILE_SIZE>;
using ShapeOut = Shape<_1, _64>;
using SubgroupLayoutQK = Layout<Shape<_1, NUM_SG, _1>>;
- return options.is_causal ? FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options)
- : FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options);
+ if constexpr (Causal) {
+ return FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(
+ options);
+ } else {
+ return FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(
+ options);
+ }
}
-template <typename ElementQ, typename ElementK, typename ElementV, bool persistent = false>
+template <bool Causal, bool HasMask, bool IsVarLen, typename ElementQ, typename ElementK, typename ElementV,
+ bool persistent = false>
inline int launch_decode_kernel_96(Options const& options) {
constexpr int PipelineStages = 1;
using NUM_SG = std::conditional_t<persistent, _16, _8>;
@@ -999,13 +1022,19 @@ inline int launch_decode_kernel_96(Options const& options) {
using ShapePV = Shape<_1, _32, KV_TILE_SIZE>;
using ShapeOut = Shape<_1, _96>;
using SubgroupLayoutQK = Layout<Shape<_1, NUM_SG, _1>>;
- return options.is_causal ? FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options)
- : FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options);
+ if constexpr (Causal) {
+ return FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(
+ options);
+ } else {
+ return FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(
+ options);
+ }
}
-template <typename ElementQ, typename ElementK, typename ElementV, bool persistent = false>
+template <bool Causal, bool HasMask, bool IsVarLen, typename ElementQ, typename ElementK, typename ElementV,
+ bool persistent = false>
inline int launch_decode_kernel_192(Options const& options) {
constexpr int PipelineStages = 1;
using NUM_SG = std::conditional_t<persistent, _16, _8>;
@@ -1014,10 +1043,15 @@ inline int launch_decode_kernel_192(Options const& options) {
using ShapePV = Shape<_1, _32, KV_TILE_SIZE>;
using ShapeOut = Shape<_1, _192>;
using SubgroupLayoutQK = Layout<Shape<_1, NUM_SG, _1>>;
- return options.is_causal ? FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options)
- : FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options);
+ if constexpr (Causal) {
+ return FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(
+ options);
+ } else {
+ return FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,
+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(
+ options);
+ }
}
} // namespace detail
diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/xpu_wrapper.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/xpu_wrapper.hpp
index 804198ba3..7bfbc40bc 100644
--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/xpu_wrapper.hpp
+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/xpu_wrapper.hpp
@@ -22,7 +22,7 @@
#endif
#if ARK_SYCL_TLA
-#include "sycl_tla_dense_gemm.hpp"
+#include "sycl_tla_common.hpp"
#endif
#if ARK_XPU
diff --git a/auto_round_extension/ark/setup.py b/auto_round_extension/ark/setup.py
index 84d6e7b13..13c030f47 100644
--- a/auto_round_extension/ark/setup.py
+++ b/auto_round_extension/ark/setup.py
@@ -12,6 +12,11 @@ from setuptools import Extension, find_packages, setup
from setuptools.command.build_ext import build_ext
from setuptools.command.build_py import build_py
+
+def _build_log(message):
+ print(f"[ARK setup] {message}", file=sys.stderr, flush=True)
+
+
build_mode = os.environ.get("BUILD_MODE", "dev").lower()
try:
file_path = "./auto_round_kernel/version.py"
@@ -174,6 +179,26 @@ sycl_target = detect_sycl_target() if enable_sycl_tla else None
def get_system_memory_gb():
+ cgroup_memory_paths = (
+ "/sys/fs/cgroup/memory.max",
+ "/sys/fs/cgroup/memory/memory.limit_in_bytes",
+ )
+ for memory_path in cgroup_memory_paths:
+ try:
+ memory_limit = Path(memory_path).read_text().strip()
+ except OSError:
+ continue
+ if memory_limit == "max":
+ continue
+ try:
+ memory_bytes = int(memory_limit)
+ except ValueError:
+ continue
+ if 0 < memory_bytes < (1 << 60):
+ memory_gb = memory_bytes / (1024**3)
+ _build_log(f"System memory from {memory_path}: {memory_gb:.2f} GiB ({memory_bytes} bytes)")
+ return memory_gb
+
if hasattr(os, "sysconf"):
page_size_names = ("SC_PAGE_SIZE", "SC_PAGESIZE")
page_size = None
@@ -184,7 +209,9 @@ def get_system_memory_gb():
if page_size is not None and "SC_PHYS_PAGES" in os.sysconf_names:
phys_pages = os.sysconf("SC_PHYS_PAGES")
if phys_pages > 0:
- return (page_size * phys_pages) / (1024**3)
+ memory_gb = (page_size * phys_pages) / (1024**3)
+ _build_log(f"System memory from sysconf: {memory_gb:.2f} GiB")
+ return memory_gb
if sys.platform == "win32":
@@ -204,17 +231,46 @@ def get_system_memory_gb():
memory_status = MEMORYSTATUSEX()
memory_status.dwLength = ctypes.sizeof(MEMORYSTATUSEX)
if ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(memory_status)):
- return memory_status.ullTotalPhys / (1024**3)
+ memory_gb = memory_status.ullTotalPhys / (1024**3)
+ _build_log(f"System memory from Windows: {memory_gb:.2f} GiB")
+ return memory_gb
+ _build_log("System memory detection failed; using fallback: 64 GiB")
return 64
+def get_cpu_count():
+ if hasattr(os, "sched_getaffinity"):
+ affinity = os.sched_getaffinity(0)
+ cpu_count = max(1, len(affinity))
+ _build_log(f"CPU count from sched_getaffinity(0): {cpu_count}; allowed CPUs: {sorted(affinity)}")
+ return cpu_count
+ cpu_count = os.cpu_count() or 1
+ _build_log(f"CPU count from os.cpu_count(): {cpu_count}")
+ return cpu_count
+
+
def get_sycl_tla_job_count(cpu_job_count):
+ override = os.environ.get("ARK_SYCL_TLA_JOBS")
+ if override is not None:
+ try:
+ jobs = int(override)
+ except ValueError as error:
+ raise ValueError("ARK_SYCL_TLA_JOBS must be a positive integer") from error
+ if jobs < 1:
+ raise ValueError("ARK_SYCL_TLA_JOBS must be a positive integer")
+ final_jobs = min(cpu_job_count, jobs)
+ _build_log(f"SYCL TLA jobs: min(cpu_job_count={cpu_job_count}, override={jobs}) = {final_jobs}")
+ return final_jobs
+
memory_gb = get_system_memory_gb()
- memory_based_jobs = max(
- 1, int(memory_gb // 16)
- ) # about 5GB/job for SYCL TLA build, use at most 5/16 of total memory to avoid OOM
- return min(cpu_job_count, memory_based_jobs)
+ memory_based_jobs = max(1, int(memory_gb // 3)) # reserve about 3GB per SYCL TLA compiler job
+ final_jobs = min(cpu_job_count, memory_based_jobs)
+ _build_log(
+ f"SYCL TLA jobs: memory={memory_gb:.2f} GiB, memory_based_jobs={memory_based_jobs}, "
+ f"cpu_job_count={cpu_job_count}, final={final_jobs}"
+ )
+ return final_jobs
ROOT = Path(__file__).resolve().parent
@@ -225,6 +281,7 @@ XBUILD_DIR = ROOT / "xbuild"
class CMakeBuild(build_ext):
def run(self):
+ _build_log(f"CMakeBuild.run using setup.py={Path(__file__).resolve()}, python={sys.executable}")
cmake_cmd = [
"cmake",
"-S",
@@ -237,8 +294,9 @@ class CMakeBuild(build_ext):
cmake_cmd.append("-GNinja")
subprocess.check_call(cmake_cmd)
- n_job = os.cpu_count() or 2
- n_job = n_job // 2
+ cpu_count = get_cpu_count()
+ n_job = max(1, cpu_count)
+ _build_log(f"CPU build jobs: max(1, {cpu_count}) = {n_job}")
subprocess.check_call(["cmake", "--build", str(BUILD_DIR), "-j", str(n_job)])
ext = "pyd" if sys.platform == "win32" else "so"
@@ -271,6 +329,7 @@ class CMakeBuild(build_ext):
if sys.platform == "win32":
cmake_cmd.append("-GNinja")
xpu_n_job = get_sycl_tla_job_count(n_job) if enable_sycl_tla else n_job
+ _build_log(f"Building XPU extension with {xpu_n_job} parallel job(s)")
subprocess.check_call(cmake_cmd)
subprocess.check_call(["cmake", "--build", str(XBUILD_DIR), "-j", str(xpu_n_job)])
diff --git a/auto_round_extension/ark/test/test_ark_cpu_mixed_bestla_sdpa.py b/auto_round_extension/ark/test/test_ark_cpu_mixed_bestla_sdpa.py
index 300f51ab8..7749a326a 100644
--- a/auto_round_extension/ark/test/test_ark_cpu_mixed_bestla_sdpa.py
+++ b/auto_round_extension/ark/test/test_ark_cpu_mixed_bestla_sdpa.py
@@ -13,6 +13,11 @@ import math
import pytest
import torch
+cpuinfo = pytest.importorskip("cpuinfo")
+CPU_FLAGS = set(cpuinfo.get_cpu_info().get("flags", []))
+if "amx_bf16" not in CPU_FLAGS:
+ pytest.skip("ARK mixed BestLA SDPA tests require CPU flag amx_bf16", allow_module_level=True)
+
auto_round_kernel = pytest.importorskip(
"auto_round_kernel", reason="compiled ARK extension not built in this environment"
)
diff --git a/auto_round_extension/ark/tools/analyze_sycl_tla_templates.py b/auto_round_extension/ark/tools/analyze_sycl_tla_templates.py
new file mode 100644
index 000000000..f5a407afa
--- /dev/null
+++ b/auto_round_extension/ark/tools/analyze_sycl_tla_templates.py
@@ -0,0 +1,193 @@
+#!/usr/bin/env python3
+
+# # Copyright (C) 2026 Intel Corporation
+# # SPDX-License-Identifier: Apache-2.0
+
+import argparse
+import json
+import re
+from collections import defaultdict, deque
+from pathlib import Path
+
+LOCAL_INCLUDE = re.compile(r'^\s*#\s*include\s+"([^"]+)"')
+TEMPLATE_DECLARATION = re.compile(r"^\s*template\s*<")
+
+CALL_PATTERNS = {
+ "dense_gemm": re.compile(r"\b(?:gemm_cute_store_tile\s*<|run_dense_gemm\s*\()"),
+ "s8_gemm": re.compile(r"\b(?:launch_igemm\s*<|launch_igemm_kblock\s*<|run_typed\s*\()"),
+ "moe_gemm": re.compile(r"\bmoe_gemm_launcher\s*<"),
+ "moe_decode": re.compile(r"\blaunch_(?:fp|int4|int8|int2|fp8)\s*<"),
+ "moe_prefill": re.compile(r"\b(?:moe_prefill_[a-z0-9_]+_dispatch|dequant_to_KN|launch_dequant_fp8_slm)\s*<"),
+ "sdpa": re.compile(r"\blaunch_(?:sage|prefill|sparse)[a-z0-9_]*\s*<"),
+}
+
+ESTIMATED_KERNELS = {
+ "dense_gemm": 24,
+ "s8_gemm": 96,
+ "moe_gemm": 2,
+ "moe_decode": 22,
+ "moe_prefill": 44,
+ "sdpa": 14,
+}
+
+
+def resolve_include(include_name, including_file, files_by_name):
+ candidate = including_file.parent / include_name
+ if candidate.is_file():
+ return candidate.resolve()
+ for parent in including_file.parents:
+ candidate = parent / "wrapper" / "include" / include_name
+ if candidate.is_file():
+ return candidate.resolve()
+ matches = [path for path in files_by_name.get(Path(include_name).name, []) if path.name == Path(include_name).name]
+ return matches[0].resolve() if len(matches) == 1 else None
+
+
+def reachable_files(source, files_by_name):
+ result = []
+ pending = deque([source.resolve()])
+ visited = set()
+ while pending:
+ current = pending.popleft()
+ if current in visited or not current.is_file():
+ continue
+ visited.add(current)
+ result.append(current)
+ for line in current.read_text(encoding="utf-8", errors="ignore").splitlines():
+ match = LOCAL_INCLUDE.match(line)
+ if match:
+ included = resolve_include(match.group(1), current, files_by_name)
+ if included is not None:
+ pending.append(included)
+ return result
+
+
+def analyze_file(path):
+ declarations = 0
+ calls = defaultdict(int)
+ for line in path.read_text(encoding="utf-8", errors="ignore").splitlines():
+ if TEMPLATE_DECLARATION.match(line):
+ declarations += 1
+ continue
+ for family, pattern in CALL_PATTERNS.items():
+ calls[family] += len(pattern.findall(line))
+ return declarations, calls
+
+
+def compile_command_sources(build_dir):
+ if build_dir is None:
+ return []
+ compile_commands = build_dir / "compile_commands.json"
+ if not compile_commands.is_file():
+ return []
+ entries = json.loads(compile_commands.read_text(encoding="utf-8"))
+ sources = []
+ for entry in entries:
+ source = Path(entry["file"])
+ if not source.is_absolute():
+ source = Path(entry.get("directory", build_dir)) / source
+ source = source.resolve()
+ if source.is_file() and source.suffix in {".c", ".cc", ".cpp", ".cxx"}:
+ sources.append(source)
+ return sorted(set(sources))
+
+
+def source_files(root, build_dir=None):
+ sources = set(root.glob("*.cpp"))
+ sources.update(compile_command_sources(build_dir))
+ return sorted(path.resolve() for path in sources if path.is_file())
+
+
+def source_label(source, root):
+ try:
+ return str(source.relative_to(root))
+ except ValueError:
+ return str(source)
+
+
+def analyze_sources(root, build_dir=None):
+ files_by_name = defaultdict(list)
+ indexed_roots = [root]
+ if build_dir is not None and (build_dir / "generated").is_dir():
+ indexed_roots.append(build_dir / "generated")
+ for indexed_root in indexed_roots:
+ for path in indexed_root.rglob("*"):
+ if path.is_file():
+ files_by_name[path.name].append(path)
+
+ results = []
+ for source in source_files(root, build_dir):
+ reachable = reachable_files(source, files_by_name)
+ declarations = 0
+ reachable_calls = defaultdict(int)
+ for included in reachable:
+ file_declarations, file_calls = analyze_file(included)
+ declarations += file_declarations
+ for family, count in file_calls.items():
+ reachable_calls[family] += count
+ direct_declarations, direct_calls = analyze_file(source)
+ results.append(
+ {
+ "source": str(source),
+ "label": source_label(source, root),
+ "reachable_files": len(reachable),
+ "template_declarations": declarations,
+ "direct_template_declarations": direct_declarations,
+ "reachable_dispatch_calls": dict(sorted(reachable_calls.items())),
+ "direct_dispatch_calls": dict(sorted(direct_calls.items())),
+ }
+ )
+ return results
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Statically count SYCL-TLA template dispatch sites.")
+ parser.add_argument(
+ "--root",
+ type=Path,
+ default=Path(__file__).resolve().parents[1] / "auto_round_kernel",
+ help="ARK auto_round_kernel source directory",
+ )
+ parser.add_argument(
+ "--build-dir",
+ type=Path,
+ help="CMake build directory; compile_commands.json adds generated translation units",
+ )
+ parser.add_argument("--json", type=Path, help="Write machine-readable per-translation-unit results")
+ args = parser.parse_args()
+ root = args.root.resolve()
+ build_dir = args.build_dir.resolve() if args.build_dir else None
+ results = analyze_sources(root, build_dir)
+
+ print(f"root: {root}")
+ print("source | reachable files | declarations | direct dispatch | reachable dispatch")
+ print("-------|------------------|--------------|-----------------|-------------------")
+ for result in results:
+ direct = sum(result["direct_dispatch_calls"].values())
+ reachable = sum(result["reachable_dispatch_calls"].values())
+ print(
+ f"{result['label']} | {result['reachable_files']:16d} | {result['template_declarations']:12d} | "
+ f"{direct:15d} | {reachable:19d}"
+ )
+
+ print("\nestimated concrete kernel combinations")
+ estimated_total = sum(ESTIMATED_KERNELS.values())
+ for family, count in ESTIMATED_KERNELS.items():
+ print(f"{family}: {count}")
+ print(f"estimated total: {estimated_total}")
+ max_result = max(results, key=lambda result: sum(result["direct_dispatch_calls"].values()), default=None)
+ if max_result:
+ print(
+ f"max direct-dispatch source: {max_result['label']} "
+ f"({sum(max_result['direct_dispatch_calls'].values())} call sites)"
+ )
+ print(f"translation units analyzed: {len(results)}")
+ print(
+ "\nReachable declarations include header definitions parsed by each translation unit; direct dispatch counts only calls written in the .cpp file."
+ )
+ if args.json:
+ args.json.write_text(json.dumps(results, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/auto_round_extension/ark/tools/measure_sycl_tla_compile_memory.py b/auto_round_extension/ark/tools/measure_sycl_tla_compile_memory.py
new file mode 100644
index 000000000..c2d2a75b8
--- /dev/null
+++ b/auto_round_extension/ark/tools/measure_sycl_tla_compile_memory.py
@@ -0,0 +1,142 @@
+#!/usr/bin/env python3
+
+# # Copyright (C) 2026 Intel Corporation
+# # SPDX-License-Identifier: Apache-2.0
+
+import argparse
+import json
+import re
+import shlex
+import subprocess
+import sys
+from pathlib import Path
+
+RSS_PATTERN = re.compile(r"Maximum resident set size \(kbytes\):\s*(\d+)")
+
+
+def load_compile_commands(path):
+ entries = json.loads(path.read_text(encoding="utf-8"))
+ commands = []
+ for entry in entries:
+ if "arguments" in entry:
+ argv = list(entry["arguments"])
+ elif "command" in entry:
+ argv = shlex.split(entry["command"])
+ else:
+ raise ValueError("compile_commands entry has neither 'arguments' nor 'command'")
+ source = Path(entry["file"])
+ if not source.is_absolute():
+ source = Path(entry["directory"]) / source
+ commands.append(
+ {
+ "source": source.resolve(),
+ "directory": Path(entry["directory"]).resolve(),
+ "argv": argv,
+ }
+ )
+ return commands
+
+
+def measure_command(command, time_binary):
+ completed = subprocess.run(
+ [time_binary, "-v", *command["argv"]],
+ cwd=command["directory"],
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ text=True,
+ check=False,
+ )
+ match = RSS_PATTERN.search(completed.stderr)
+ result = {
+ "source": str(command["source"]),
+ "returncode": completed.returncode,
+ "max_rss_kib": int(match.group(1)) if match else None,
+ "max_rss_mib": int(match.group(1)) / 1024 if match else None,
+ "stdout_tail": completed.stdout[-4000:],
+ "stderr_tail": completed.stderr[-4000:],
+ "command": command["argv"],
+ }
+ return result
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Measure peak RSS for each compiler invocation in compile_commands.json."
+ )
+ parser.add_argument("--build-dir", type=Path, required=True, help="CMake build directory")
+ parser.add_argument(
+ "--compile-commands",
+ type=Path,
+ help="compile_commands.json path; defaults to <build-dir>/compile_commands.json",
+ )
+ parser.add_argument(
+ "--root",
+ type=Path,
+ default=Path(__file__).resolve().parents[1] / "auto_round_kernel",
+ help="ARK auto_round_kernel source directory",
+ )
+ parser.add_argument("--source-regex", help="Only measure translation units whose path matches this regex")
+ parser.add_argument("--time-binary", default="/usr/bin/time", help="GNU time executable")
+ parser.add_argument("--output", type=Path, help="Write detailed JSON results")
+ args = parser.parse_args()
+
+ compile_commands = (args.compile_commands or args.build_dir / "compile_commands.json").resolve()
+ if not compile_commands.is_file():
+ raise SystemExit(f"compile commands not found: {compile_commands}")
+ if not Path(args.time_binary).is_file():
+ raise SystemExit(f"time executable not found: {args.time_binary}")
+
+ commands = load_compile_commands(compile_commands)
+ if args.source_regex:
+ source_pattern = re.compile(args.source_regex)
+ commands = [command for command in commands if source_pattern.search(str(command["source"]))]
+ if not commands:
+ raise SystemExit("no compile commands selected")
+
+ sys.path.insert(0, str(Path(__file__).resolve().parent))
+ from analyze_sycl_tla_templates import analyze_sources
+
+ analysis = analyze_sources(args.root.resolve(), args.build_dir.resolve())
+ analysis_by_source = {str(Path(item["source"]).resolve()): item for item in analysis}
+
+ results = []
+ for index, command in enumerate(commands, start=1):
+ print(f"[{index}/{len(commands)}] {command['source']}", flush=True)
+ result = measure_command(command, args.time_binary)
+ result.update(analysis_by_source.get(str(command["source"]), {}))
+ direct_dispatches = sum(result.get("direct_dispatch_calls", {}).values())
+ result["rss_per_direct_dispatch_mib"] = (
+ result["max_rss_mib"] / direct_dispatches
+ if result["max_rss_mib"] is not None and direct_dispatches
+ else None
+ )
+ results.append(result)
+ rss = result["max_rss_mib"]
+ rss_text = f"{rss:.1f} MiB" if rss is not None else "unknown RSS"
+ print(f" returncode={result['returncode']} peak RSS={rss_text}", flush=True)
+
+ measured = [result for result in results if result["max_rss_kib"] is not None]
+ measured.sort(key=lambda result: result["max_rss_kib"], reverse=True)
+ print("source | peak RSS MiB | RSS/direct MiB | return code | direct dispatch | declarations")
+ print("-------|---------------|----------------|-------------|-----------------|-------------")
+ for result in measured:
+ direct = sum(result.get("direct_dispatch_calls", {}).values())
+ per_dispatch = result["rss_per_direct_dispatch_mib"]
+ per_dispatch_text = f"{per_dispatch:14.1f}" if per_dispatch is not None else " n/a"
+ print(
+ f"{result.get('label', result['source'])} | {result['max_rss_mib']:13.1f} | {per_dispatch_text} | "
+ f"{result['returncode']:11d} | {direct:15d} | {result.get('template_declarations', 'n/a'):11}"
+ )
+ if measured:
+ print(
+ f"\npeak compiler RSS: {measured[0]['max_rss_mib']:.1f} MiB ({measured[0].get('label', measured[0]['source'])})"
+ )
+ else:
+ print("\nNo GNU time RSS measurements were parsed.")
+
+ if args.output:
+ args.output.write_text(json.dumps(results, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+
+
+if __name__ == "__main__":
+ main()