PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
Reduce SYCL-TLA Compiler Memory (#2193)
Luo Yuff2fd25c644d
diff --git a/auto_round_extension/ark/README.md b/auto_round_extension/ark/README.mdindex 6e24fd355..47dd20b50 100644--- a/auto_round_extension/ark/README.md+++ b/auto_round_extension/ark/README.md@@ -225,6 +225,28 @@ python setup.py bdist_wheel; pip install dist/*Build with MoE / SageAttention support requires `ARK_SYCL_TLA=ON`.+For oneAPI/SYCL-TLA builds, template compilation can require several gigabytes+per compiler process. If the build host runs out of memory, limit concurrent+XPU compilations with `ARK_SYCL_TLA_JOBS`:++```bash+ARK_SYCL_TLA_JOBS=1 pip install . --no-build-isolation+```++On a oneAPI-equipped build host, configure an XPU build with compile commands+enabled and measure each translation unit before choosing a higher job count:++```bash+python tools/measure_sycl_tla_compile_memory.py \+ --build-dir /path/to/ark/xbuild \+ --output /tmp/ark-sycl-tla-rss.json+```++The measurement tool runs compiler commands serially and reports peak RSS,+direct dispatch count, and reachable template declarations. The current+environment must provide `icx` and a generated `compile_commands.json` for+these measurements to represent the SYCL-TLA build.+---## Testsdiff --git a/auto_round_extension/ark/README_CN.md b/auto_round_extension/ark/README_CN.mdnew file mode 100644index 000000000..8c631e56d--- /dev/null+++ b/auto_round_extension/ark/README_CN.md@@ -0,0 +1,231 @@+## 什么是 AutoRound Kernel(ARK)?++AutoRound Kernel(ARK)是面向 Intel 平台的低比特加速库,为大语言模型推理提供三类优化算子。++| 算子类别 | CPU | XPU(Battlemage) |+|:--|:--:|:--:|+| **仅权重量化线性层**(INT4/INT8/FP8/FP4) | ✅ | ✅ |+| **MoE 分组 GEMM** | ❌ | ✅ |+| **SageAttention**(SDPA / SAGE) | ❌ | ✅ |++**已验证 CPU:** Intel Xeon Scalable(Sapphire Rapids / Emerald Rapids)、Intel Xeon 6(Sierra Forest / Granite Rapids)<br>+**已验证 GPU:** Intel Arc B 系列 / Arc Pro B 系列(Battlemage)++### 生态集成++ARK 算子已集成到以下项目:++| 项目 | 集成 | 说明 |+|--|--|--|+| [vllm](https://github.com/vllm-project/vllm) | [`inc_wna16_linear.py`](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/inc/schemes/inc_wna16_linear.py#L379-L470) | `INCXPUARKLinearMethod` 通过 `auto_round_kernel.qlinear.QuantLinear` 在 XPU 上提供仅权重量化线性层。 |+| [vllm-omni](https://github.com/vllm-project/vllm-omni) | [`sage_attn.py`](https://github.com/vllm-project/vllm-omni/blob/f8340d078e4e9c3b793bd92d55d891b29703f0a8/vllm_omni/diffusion/attention/backends/sage_attn.py#L27) | `SageAttentionBackend` 通过 `ARK.sagev1` 在 XPU 上加速扩散模型注意力。 |+| [Transformers](https://github.com/huggingface/transformers)(通过 [auto-round](https://github.com/intel/auto-round)) | [`backend.py`](https://github.com/intel/auto-round/blob/main/auto_round/inference/backend.py#L531) | AutoRound 量化的模型默认在 CPU/XPU 上使用 ARK,无需额外配置。 |++---++## 1. 线性层(仅权重量化 GEMM)++面向大语言模型推理的低比特仅权重量化线性层,支持 CPU 和 XPU。++### API++| API | 说明 | 平台 |+|--|--|--|+| `QuantLinear` | 统一的 PyTorch 模块(GPTQ/AWQ/原始量化检查点) | CPU / XPU |+| `QuantLinearGPTQ` | GPTQ 格式检查点加载器 | CPU / XPU |+| `QuantLinearAWQ` | AWQ 格式检查点加载器 | CPU / XPU |+| `QuantLinearFP8` | FP8 仅权重量化线性层 | CPU / XPU |+| `woqgemm` | 低层仅权重量化 GEMM(打包格式) | CPU / XPU |+| `woqgemm_s8` | 带 scale 的低层 INT8 权重 GEMM | CPU / XPU |+| `_repack_quantized_weight` | 将原始 qweight/qzero/scale 重新打包为 ARK 格式 | CPU / XPU |+| `_unpack_weight` | 将 ARK 格式权重解包回全精度 | CPU / XPU |++### 关键特性++> **W4A8 / W2A8 重缩放(QQQ 风格)**:XPU 上支持将低比特权重(INT2/INT4)重缩放为 INT8,并通过 INT8 GEMM 计算,避免 FP16 反量化。该功能通过环境变量 `ARK_AUTO_S8` 自动启用,详见 [xpu_wrapper.hpp](auto_round_kernel/wrapper/include/xpu_wrapper.hpp)。++### 支持的数据类型++#### CPU++| 权重类型 | 计算类型 | Scale 类型 | 算法 |+|--|:--:|:--:|:--:|+| INT1–INT8 | INT8<sup>[1]</sup> / BF16 / FP32 | BF16 / FP32 | 对称 / 非对称 |+| FP8(E4M3、E5M2) | BF16 / FP32 | FP32 / FP8(E8M0) | 不适用 |+| FP4(E2M1) | BF16 / FP32 | BF16 / FP32 | 不适用 |++#### XPU++| 权重类型 | 计算类型 | Scale 类型 | 算法 |+|--|:--:|:--:|:--:|+| INT4、INT8 | INT8 / FP16 | FP16 | 对称 |+| FP8(E4M3、E5M2) | FP16 | FP16 / FP8(E8M0) | 不适用 |++<sup>[1]</sup> INT8 计算包含动态激活量化,结果会反量化为浮点类型。++### 示例++```python+import auto_round_kernel as ark++# 准备量化权重:qweight [K, N] int4/int2,scale [K/G, N] fp16/fp32,zp [K/G, N] int4/int2+packw = ark.repack_quantized_weight(+ qweight,+ scale,+ zp,+ blocksize=128,+ compute_type="fp16",+ weight_type="int4",+ scale_type="fp16",+ asym=False,+)++# 运行仅权重量化 GEMM:activation [M, K] -> output [M, N]+output = ark.woqgemm(+ activation,+ packw,+ bias,+ n,+ k,+ groupsize=128,+ compute_type="fp16",+ weight_type="int4",+ scale_type="fp16",+ asym=False,+)++# 解包回全精度以便验证+_decompressed = ark.unpack_weight(+ packw,+ dtype=torch.float16,+ n=n,+ k=k,+ groupsize=128,+ compute_type="fp16",+ weight_type="int4",+ scale_type="fp16",+ asym=False,+)+```++详见 [test_weightonly.py](test/test_weightonly.py),其中包含 CPU 和 XPU 上的权重重新打包、验证和 `woqgemm` 运行示例。++---++## 2. MoE(混合专家分组 GEMM)++面向 MoE 层的分组 GEMM,不同专家可以处理不同数量的 token。++### API++| 函数 | 说明 | 平台 | 激活类型 | 权重类型 |+|--|--|--|:--:|:--:|+| `ark.moe_gemm(...)` | 专家分组 GEMM | XPU | FP16 / BF16 | FP16 / BF16 |+| `ark.moe_gemm(...)`(开发中) | INT4 权重分组 GEMM | XPU | FP16 / BF16 | INT4 🚧 |+| `ark.moe_gemm(...)`(开发中) | INT2 权重分组 GEMM | XPU | FP16 / BF16 | INT2 🚧 |+| `ark.moe_gemm(...)`(开发中) | INT8 权重分组 GEMM | XPU | FP16 / BF16 | INT8 🚧 |++> 🚧 INT2 / INT4 / INT8 权重支持仍在开发中,详见 [#PR](https://github.com/intel/auto-round/pull)。++### 细节++| 参数 | 形状 | 类型 |+|--|--|--|+| activations | `[total_tokens, K]` | FP16 / BF16 |+| weights | `[num_experts, K, N]`(行主序) | FP16 / BF16 |+| num_tokens_per_expert | `[num_experts]` | INT32 |+| scales(可选) | `[num_experts, N]` | FP16 / BF16 |+| **output** | `[total_tokens, N]` | 与激活相同 |++### 示例++```python+# FP16/BF16 MoE+output = ark.moe_gemm(activations, weights, num_tokens_per_expert)++# INT4 MoE(即将支持)+# output = ark.moe_gemm(activations, q4_weights, num_tokens_per_expert, scales=scales)+```++构建要求:`ARK_SYCL_TLA=ON`。详见 [test_moe.py](test/test_moe.py)。++---++## 3. SageAttention(XPU SDPA 加速)++ARK 在 XPU 上提供完整的缩放点积注意力内核族,从原生 FP16 SDPA 到 INT8 量化 SageAttention 变体均可使用。++### API 概览++| 函数 | 说明 | Q/K/V 输入 | PV 精度 | Head Dim |+|--|--|--|:--:|:--:|+| `ark.sdpa` | FP16/BF16 SDPA(flash attention) | FP16 / BF16 | FP16 | 64、96、128、192 |+| `ark.sage` | 低层 INT8 SAGE(预量化 Q/K) | INT8(Q/K)、FP16(V) | FP16 | 64、128 |+| `ark.sage_pvi8` | 低层 INT8 SAGE(预量化 Q/K/V) | INT8 | INT8 | 64、128 |+| `ark.sagev1` | 高层 FP16 -> 内部 Q/K 量化 -> SAGE | FP16 / BF16 | FP16 | 64、128 |+| `ark.sagev1_pvi8` | 高层 FP16 -> 内部 Q/K/V 量化 -> SAGE PV INT8 | FP16 / BF16 | INT8 | 64、128 |+| `ark.sageattn` | SageAttention 兼容 API 的分发器 | FP16 / BF16 | FP16 / INT8 | 64、128 |+| `ark.sage_dynquant` | 动态 INT8 分块 Q/K 量化 -> SAGE(可替换 SDPA) | FP16 / BF16 | FP16 | 64、128 |++### 约束++| 约束 | `sdpa` | `sagev1` / `sagev1_pvi8` / `sage_dynquant` |+|--|:--:|:--:|+| Q/K/V 类型 | FP16、BF16 | FP16、BF16 |+| Head dim | 64、96、128、192 | 64、128 |+| `dropout_p` | 必须为 0.0 | 必须为 0.0 |+| 布尔 mask | 回退到 PyTorch | 回退到 PyTorch |+| 加性 mask 形状 | `[B, 1, Sq, Skv]` FP32 | `[B, 1, Sq, Skv]` FP32 |+| `quant_block_size` | 不适用 | 1(逐 token)或 >=32 |++---++## 安装++### 通过 pip 安装++```bash+pip install auto-round-lib+```++### 从源码安装++```bash+pip install . --no-build-isolation+# 或+python setup.py bdist_wheel; pip install dist/*+```++启用 MoE / SageAttention 支持需要 `ARK_SYCL_TLA=ON`。++对于 oneAPI/SYCL-TLA 构建,每个编译进程的模板编译可能需要数 GB 内存。如果构建主机内存不足,可以通过 `ARK_SYCL_TLA_JOBS` 限制并发的 XPU 编译数量:++```bash+ARK_SYCL_TLA_JOBS=1 pip install . --no-build-isolation+```++在配备 oneAPI 的构建主机上,建议启用编译命令并测量每个翻译单元,再选择更高的并发数:++```bash+python tools/measure_sycl_tla_compile_memory.py \+ --build-dir /path/to/ark/xbuild \+ --output /tmp/ark-sycl-tla-rss.json+```++该工具会串行运行编译器命令,并报告峰值 RSS、直接分发数量和可达模板声明数量。要使测量结果反映 SYCL-TLA 构建,当前环境必须提供 `icx` 以及生成的 `compile_commands.json`。++---++## 测试++| 测试 | 说明 |+|--|--|+| [test_weightonly.py](test/test_weightonly.py) | WOQ GEMM 打包、解包和运行(CPU/XPU) |+| [test_moe.py](test/test_moe.py) | MoE 分组 GEMM |+| [test_flash_attn.py](test/test_flash_attn.py) | SDPA(flash attention)prefill |+| [test_sdpa.py](test/test_sdpa.py) | SDPA 基准测试 |+| [test_sdpa_parity.py](test/test_sdpa_parity.py) | SDPA 与 PyTorch 的一致性测试 |+| [test_sage_dynquant.py](test/test_sage_dynquant.py) | SageAttention 动态 INT8 量化基准测试 |+| [test_bench_bmg.py](test/test_bench_bmg.py) | BMG SDPA / SageAttention 基准测试 |+| [test_matmul.py](test/test_matmul.py) | 低层 matmul |+| [test_packq.py](test/test_packq.py) | 权重打包工具 |diff --git a/auto_round_extension/ark/auto_round_kernel/CMakeLists.txt b/auto_round_extension/ark/auto_round_kernel/CMakeLists.txtindex 0d915d7f3..718ea46dd 100755--- a/auto_round_extension/ark/auto_round_kernel/CMakeLists.txt+++ b/auto_round_extension/ark/auto_round_kernel/CMakeLists.txt@@ -101,6 +101,7 @@ if(NOT ARK_XPU)endif()set(SDPA_GENERATED_SRCS)set(SDPA_KERNEL_DECLARATIONS)+set(SYCL_TLA_GENERATED_SRCS)include_directories(wrapper/include)include_directories(${CMAKE_CURRENT_SOURCE_DIR})@@ -161,9 +162,10 @@ if(ARK_XPU AND ARK_SYCL_TLA)${CUTLASS_DIR}/examples/12_xe20_moe_gemm_cute_interface)include(${CMAKE_CURRENT_LIST_DIR}/sdpa_generation.cmake)+ include(${CMAKE_CURRENT_LIST_DIR}/sycl_tla_generation.cmake)endif()-pybind11_add_module(${PY_NAME} ${SRCS} ${HEADERS} ${SDPA_GENERATED_SRCS})+pybind11_add_module(${PY_NAME} ${SRCS} ${HEADERS} ${SDPA_GENERATED_SRCS} ${SYCL_TLA_GENERATED_SRCS})target_compile_features(${PY_NAME} PRIVATE cxx_std_17)target_compile_definitions(${PY_NAME} PRIVATE PY_NAME=${PY_NAME} ${ARK_TYPE}=1)if(ARK_DNNL)diff --git a/auto_round_extension/ark/auto_round_kernel/ark.cpp b/auto_round_extension/ark/auto_round_kernel/ark.cppindex aa843020d..94510ce8d 100755--- a/auto_round_extension/ark/auto_round_kernel/ark.cpp+++ b/auto_round_extension/ark/auto_round_kernel/ark.cpp@@ -28,13 +28,7 @@ typedef uintptr_t torch_ptr;#include "xpu_wrapper.hpp"#include "sycl_s8_wrapper.hpp"#if ARK_SYCL_TLA-// Only include declarations, implementations are in separate .cpp files#include "sycl_tla_common.hpp"-#include "sycl_tla_moe.hpp"-#include "sycl_tla_moe_decode.hpp"-#include "sycl_tla_moe_mixed.hpp"-#include "sycl_tla_sdpa.hpp"-#include "sycl_tla_dense_gemm.hpp"#endif#else#include "ark/cpu/sdpa.h"diff --git a/auto_round_extension/ark/auto_round_kernel/sdpa.cpp b/auto_round_extension/ark/auto_round_kernel/sdpa.cppindex 2d9b5f424..6220d0cfa 100644--- a/auto_round_extension/ark/auto_round_kernel/sdpa.cpp+++ b/auto_round_extension/ark/auto_round_kernel/sdpa.cpp@@ -114,31 +114,103 @@ KernelLauncher select_sage_prefill_launcher(BTLA_DTYPE dtype, int head_dim) {}-KernelLauncher select_prefill_launcher(BTLA_DTYPE dtype, int head_dim) {+KernelLauncher select_prefill_launcher(BTLA_DTYPE dtype, int head_dim, bool is_causal, bool has_mask, bool is_varlen) {switch (dtype) {case BTLA_DTYPE::F16:switch (head_dim) {case 128:- return detail::launch_prefill_kernel_f16_128;+ if (is_causal) {+ return is_varlen ? detail::launch_prefill_kernel_f16_128_causal_varlen+ : detail::launch_prefill_kernel_f16_128_causal;+ }+ if (has_mask) {+ return is_varlen ? detail::launch_prefill_kernel_f16_128_noncausal_masked_varlen+ : detail::launch_prefill_kernel_f16_128_noncausal_masked;+ }+ return is_varlen ? detail::launch_prefill_kernel_f16_128_noncausal_unmasked_varlen+ : detail::launch_prefill_kernel_f16_128_noncausal_unmasked;case 64:- return detail::launch_prefill_kernel_f16_64;+ if (is_causal) {+ return is_varlen ? detail::launch_prefill_kernel_f16_64_causal_varlen+ : detail::launch_prefill_kernel_f16_64_causal;+ }+ if (has_mask) {+ return is_varlen ? detail::launch_prefill_kernel_f16_64_noncausal_masked_varlen+ : detail::launch_prefill_kernel_f16_64_noncausal_masked;+ }+ return is_varlen ? detail::launch_prefill_kernel_f16_64_noncausal_unmasked_varlen+ : detail::launch_prefill_kernel_f16_64_noncausal_unmasked;case 96:- return detail::launch_prefill_kernel_f16_96;+ if (is_causal) {+ return is_varlen ? detail::launch_prefill_kernel_f16_96_causal_varlen+ : detail::launch_prefill_kernel_f16_96_causal;+ }+ if (has_mask) {+ return is_varlen ? detail::launch_prefill_kernel_f16_96_noncausal_masked_varlen+ : detail::launch_prefill_kernel_f16_96_noncausal_masked;+ }+ return is_varlen ? detail::launch_prefill_kernel_f16_96_noncausal_unmasked_varlen+ : detail::launch_prefill_kernel_f16_96_noncausal_unmasked;case 192:- return detail::launch_prefill_kernel_f16_192;+ if (is_causal) {+ return is_varlen ? detail::launch_prefill_kernel_f16_192_causal_varlen+ : detail::launch_prefill_kernel_f16_192_causal;+ }+ if (has_mask) {+ return is_varlen ? detail::launch_prefill_kernel_f16_192_noncausal_masked_varlen+ : detail::launch_prefill_kernel_f16_192_noncausal_masked;+ }+ return is_varlen ? detail::launch_prefill_kernel_f16_192_noncausal_unmasked_varlen+ : detail::launch_prefill_kernel_f16_192_noncausal_unmasked;default:return nullptr;}case BTLA_DTYPE::BF16:switch (head_dim) {case 64:- return detail::launch_prefill_kernel_bf16_64;+ if (is_causal) {+ return is_varlen ? detail::launch_prefill_kernel_bf16_64_causal_varlen+ : detail::launch_prefill_kernel_bf16_64_causal;+ }+ if (has_mask) {+ return is_varlen ? detail::launch_prefill_kernel_bf16_64_noncausal_masked_varlen+ : detail::launch_prefill_kernel_bf16_64_noncausal_masked;+ }+ return is_varlen ? detail::launch_prefill_kernel_bf16_64_noncausal_unmasked_varlen+ : detail::launch_prefill_kernel_bf16_64_noncausal_unmasked;case 96:- return detail::launch_prefill_kernel_bf16_96;+ if (is_causal) {+ return is_varlen ? detail::launch_prefill_kernel_bf16_96_causal_varlen+ : detail::launch_prefill_kernel_bf16_96_causal;+ }+ if (has_mask) {+ return is_varlen ? detail::launch_prefill_kernel_bf16_96_noncausal_masked_varlen+ : detail::launch_prefill_kernel_bf16_96_noncausal_masked;+ }+ return is_varlen ? detail::launch_prefill_kernel_bf16_96_noncausal_unmasked_varlen+ : detail::launch_prefill_kernel_bf16_96_noncausal_unmasked;case 128:- return detail::launch_prefill_kernel_bf16_128;+ if (is_causal) {+ return is_varlen ? detail::launch_prefill_kernel_bf16_128_causal_varlen+ : detail::launch_prefill_kernel_bf16_128_causal;+ }+ if (has_mask) {+ return is_varlen ? detail::launch_prefill_kernel_bf16_128_noncausal_masked_varlen+ : detail::launch_prefill_kernel_bf16_128_noncausal_masked;+ }+ return is_varlen ? detail::launch_prefill_kernel_bf16_128_noncausal_unmasked_varlen+ : detail::launch_prefill_kernel_bf16_128_noncausal_unmasked;case 192:- return detail::launch_prefill_kernel_bf16_192;+ if (is_causal) {+ return is_varlen ? detail::launch_prefill_kernel_bf16_192_causal_varlen+ : detail::launch_prefill_kernel_bf16_192_causal;+ }+ if (has_mask) {+ return is_varlen ? detail::launch_prefill_kernel_bf16_192_noncausal_masked_varlen+ : detail::launch_prefill_kernel_bf16_192_noncausal_masked;+ }+ return is_varlen ? detail::launch_prefill_kernel_bf16_192_noncausal_unmasked_varlen+ : detail::launch_prefill_kernel_bf16_192_noncausal_unmasked;default:return nullptr;}@@ -147,31 +219,47 @@ KernelLauncher select_prefill_launcher(BTLA_DTYPE dtype, int head_dim) {}}-KernelLauncher select_decode_launcher(BTLA_DTYPE dtype, int head_dim) {+KernelLauncher select_decode_launcher(BTLA_DTYPE dtype, int head_dim, bool is_causal, bool has_mask) {switch (dtype) {case BTLA_DTYPE::F16:switch (head_dim) {case 128:- return detail::launch_decode_kernel_f16_128;+ if (is_causal) return detail::launch_decode_kernel_f16_128_causal;+ if (has_mask) return detail::launch_decode_kernel_f16_128_noncausal_masked;+ return detail::launch_decode_kernel_f16_128_noncausal_unmasked;case 64:- return detail::launch_decode_kernel_f16_64;+ if (is_causal) return detail::launch_decode_kernel_f16_64_causal;+ if (has_mask) return detail::launch_decode_kernel_f16_64_noncausal_masked;+ return detail::launch_decode_kernel_f16_64_noncausal_unmasked;case 96:- return detail::launch_decode_kernel_f16_96;+ if (is_causal) return detail::launch_decode_kernel_f16_96_causal;+ if (has_mask) return detail::launch_decode_kernel_f16_96_noncausal_masked;+ return detail::launch_decode_kernel_f16_96_noncausal_unmasked;case 192:- return detail::launch_decode_kernel_f16_192;+ if (is_causal) return detail::launch_decode_kernel_f16_192_causal;+ if (has_mask) return detail::launch_decode_kernel_f16_192_noncausal_masked;+ return detail::launch_decode_kernel_f16_192_noncausal_unmasked;default:return nullptr;}case BTLA_DTYPE::BF16:switch (head_dim) {case 64:- return detail::launch_decode_kernel_bf16_64;+ if (is_causal) return detail::launch_decode_kernel_bf16_64_causal;+ if (has_mask) return detail::launch_decode_kernel_bf16_64_noncausal_masked;+ return detail::launch_decode_kernel_bf16_64_noncausal_unmasked;case 96:- return detail::launch_decode_kernel_bf16_96;+ if (is_causal) return detail::launch_decode_kernel_bf16_96_causal;+ if (has_mask) return detail::launch_decode_kernel_bf16_96_noncausal_masked;+ return detail::launch_decode_kernel_bf16_96_noncausal_unmasked;case 128:- return detail::launch_decode_kernel_bf16_128;+ if (is_causal) return detail::launch_decode_kernel_bf16_128_causal;+ if (has_mask) return detail::launch_decode_kernel_bf16_128_noncausal_masked;+ return detail::launch_decode_kernel_bf16_128_noncausal_unmasked;case 192:- return detail::launch_decode_kernel_bf16_192;+ if (is_causal) return detail::launch_decode_kernel_bf16_192_causal;+ if (has_mask) return detail::launch_decode_kernel_bf16_192_noncausal_masked;+ return detail::launch_decode_kernel_bf16_192_noncausal_unmasked;default:return nullptr;}@@ -269,7 +357,7 @@ void flash_attn_prefill(sycl::queue* q, void* Q_ptr, void* K_ptr, void* V_ptr, voptions.lse = lse;compat::set_default_queue(*q);- KernelLauncher launcher = select_prefill_launcher(q_dtype, head_dim);+ KernelLauncher launcher = select_prefill_launcher(q_dtype, head_dim, is_causal, mask != nullptr, false);if (launcher == nullptr) {throw std::runtime_error("Unsupported dtype or head dimension for flash_attn_prefill (only F16/BF16 and 64/96/128/192 are supported)");@@ -293,7 +381,7 @@ void flash_attn_decode(sycl::queue* q, void* Q_ptr, void* K_ptr, void* V_ptr, vooptions.lse = lse;compat::set_default_queue(*q);- KernelLauncher launcher = select_decode_launcher(q_dtype, head_dim);+ KernelLauncher launcher = select_decode_launcher(q_dtype, head_dim, is_causal, mask != nullptr);if (launcher == nullptr) {throw std::runtime_error("Unsupported dtype or head dimension for flash_attn_decode (only F16/BF16 and 64/96/128/192 are supported)");@@ -520,7 +608,7 @@ void sdpa_varlen_impl(sycl::queue* q, void* Q_ptr, void* K_ptr, void* V_ptr, voioptions.cu_seqlens_kv_cache = zero_cu_buf;options.use_tensor_strides = true;- KernelLauncher launcher = select_prefill_launcher(q_dtype, head_dim);+ KernelLauncher launcher = select_prefill_launcher(q_dtype, head_dim, is_causal, mask != nullptr, true);if (launcher == nullptr) {throw std::runtime_error("Unsupported dtype or head dimension for sdpa_varlen_impl (only F16/BF16 and 64/96/128/192 are supported)");diff --git a/auto_round_extension/ark/auto_round_kernel/sdpa_generation.cmake b/auto_round_extension/ark/auto_round_kernel/sdpa_generation.cmakeindex 051de0c68..aebbbf8f8 100644--- a/auto_round_extension/ark/auto_round_kernel/sdpa_generation.cmake+++ b/auto_round_extension/ark/auto_round_kernel/sdpa_generation.cmake@@ -1,14 +1,19 @@file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/generated/sdpa)set(SDPA_DECLARATIONS_FILE ${CMAKE_CURRENT_BINARY_DIR}/generated/sdpa/sdpa_kernel_declarations.hpp)-function(generate_sdpa_instantiation mode dtype_name dtype_type head_dim template_name function_name)+function(generate_sdpa_instantiation mode dtype_name dtype_type head_dim variant_name causal_value has_mask_value+ is_varlen_value)+ set(function_name launch_${mode}_kernel_${dtype_name}_${head_dim}_${variant_name})set(SDPA_FUNCTION_NAME ${function_name})- set(SDPA_TEMPLATE_NAME ${template_name})+ set(SDPA_TEMPLATE_NAME launch_${mode}_kernel_${head_dim})+ set(SDPA_CAUSAL ${causal_value})+ set(SDPA_HAS_MASK ${has_mask_value})+ set(SDPA_IS_VARLEN ${is_varlen_value})set(SDPA_ELEMENT_TYPE ${dtype_type})string(APPEND SDPA_KERNEL_DECLARATIONS "int ${function_name}(Options const& options);\n")set(SDPA_KERNEL_DECLARATIONS ${SDPA_KERNEL_DECLARATIONS} PARENT_SCOPE)set(_generated_src- ${CMAKE_CURRENT_BINARY_DIR}/generated/sdpa/${mode}_${dtype_name}_${head_dim}.cpp)+ ${CMAKE_CURRENT_BINARY_DIR}/generated/sdpa/${mode}_${dtype_name}_${head_dim}_${variant_name}.cpp)configure_file(${CMAKE_CURRENT_LIST_DIR}/sdpa_kernel_instantiation.cpp.in ${_generated_src} @ONLY)list(APPEND SDPA_GENERATED_SRCS ${_generated_src})set(SDPA_GENERATED_SRCS ${SDPA_GENERATED_SRCS} PARENT_SCOPE)@@ -23,9 +28,44 @@ foreach(mode IN ITEMS prefill decode)endif()foreach(head_dim IN ITEMS 128 64 96 192)- set(template_name launch_${mode}_kernel_${head_dim})- set(function_name launch_${mode}_kernel_${dtype_name}_${head_dim})- generate_sdpa_instantiation(${mode} ${dtype_name} "${dtype_type}" ${head_dim} ${template_name} ${function_name})+ foreach(varlen_name IN ITEMS normal varlen)+ if(varlen_name STREQUAL "varlen")+ if(mode STREQUAL "decode")+ continue()+ endif()+ set(is_varlen_value true)+ set(varlen_suffix "_varlen")+ else()+ set(is_varlen_value false)+ set(varlen_suffix "")+ endif()++ foreach(causal_name IN ITEMS causal noncausal)+ if(causal_name STREQUAL "causal")+ set(causal_value true)+ set(mask_names unmasked)+ else()+ set(causal_value false)+ set(mask_names masked unmasked)+ endif()++ foreach(mask_name IN LISTS mask_names)+ if(mask_name STREQUAL "masked")+ set(has_mask_value true)+ else()+ set(has_mask_value false)+ endif()++ set(variant_name ${causal_name})+ if(causal_name STREQUAL "noncausal")+ set(variant_name noncausal_${mask_name})+ endif()+ string(APPEND variant_name ${varlen_suffix})+ generate_sdpa_instantiation(${mode} ${dtype_name} "${dtype_type}" ${head_dim} ${variant_name}+ ${causal_value} ${has_mask_value} ${is_varlen_value})+ endforeach()+ endforeach()+ endforeach()endforeach()endforeach()endforeach()diff --git a/auto_round_extension/ark/auto_round_kernel/sdpa_kernel_instantiation.cpp.in b/auto_round_extension/ark/auto_round_kernel/sdpa_kernel_instantiation.cpp.inindex b9f7f004e..7305a70f8 100644--- a/auto_round_extension/ark/auto_round_kernel/sdpa_kernel_instantiation.cpp.in+++ b/auto_round_extension/ark/auto_round_kernel/sdpa_kernel_instantiation.cpp.in@@ -9,7 +9,8 @@ namespace ark {namespace detail {int @SDPA_FUNCTION_NAME@(Options const& options) {- return @SDPA_TEMPLATE_NAME@<@SDPA_ELEMENT_TYPE@, @SDPA_ELEMENT_TYPE@, @SDPA_ELEMENT_TYPE@>(options);+ return @SDPA_TEMPLATE_NAME@<@SDPA_CAUSAL@, @SDPA_HAS_MASK@, @SDPA_IS_VARLEN@, @SDPA_ELEMENT_TYPE@,+ @SDPA_ELEMENT_TYPE@, @SDPA_ELEMENT_TYPE@>(options);}} // namespace detaildiff --git a/auto_round_extension/ark/auto_round_kernel/sycl_tla_gemm.cpp.in b/auto_round_extension/ark/auto_round_kernel/sycl_tla_gemm.cpp.innew file mode 100644index 000000000..60a99e768--- /dev/null+++ b/auto_round_extension/ark/auto_round_kernel/sycl_tla_gemm.cpp.in@@ -0,0 +1,167 @@+// Generated by CMake. Do not edit directly.++#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)++#if @DENSE_SOURCE_MODE@ > 0++#if @DENSE_SOURCE_MODE@ == 1++#include "sycl_tla_dense_gemm.hpp"+#include "sycl_tla_dense_gemm_helpers.hpp"++namespace ark {+namespace dense_gemm_detail {++void @DENSE_FUNCTION_NAME@(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c@DENSE_BIAS_PARAMETER@) {+ run_dense_gemm_fixed_tile<@DENSE_HAS_BIAS@, @DENSE_TILE_M@, @DENSE_TILE_N@, @DENSE_LAYOUT@>(+ q, m, n, k, static_cast<const @DENSE_ELEMENT_TYPE@*>(a), static_cast<const @DENSE_ELEMENT_TYPE@*>(b),+ static_cast<@DENSE_ELEMENT_TYPE@*>(c), @DENSE_BIAS_ARGUMENT@);+}++} // namespace dense_gemm_detail+} // namespace ark++#elif @DENSE_SOURCE_MODE@ == 2++#include "sycl_tla_common.hpp"++namespace ark {++void sycl_tla_dense_gemm(sycl::queue* q, int m, int n, int k, const void* a, BTLA_DTYPE at, const void* b,+ BTLA_DTYPE bt, void* c, BTLA_DTYPE ct, const void* bias, bool BT) {+ if (!q) {+ throw std::invalid_argument("sycl_tla_dense_gemm: stream must be a valid SYCL queue");+ }+ if (!BT) {+ throw std::invalid_argument("sycl_tla_dense_gemm: only the A @ B.T contract is supported");+ }+ if (!a || !b || !c) {+ throw std::invalid_argument("sycl_tla_dense_gemm: input and output pointers must not be null");+ }+ if (m <= 0 || n <= 0 || k <= 0) return;+ if (at != bt) {+ throw std::invalid_argument("sycl_tla_dense_gemm: A and B must use the same dtype");+ }+ if (ct != at) {+ throw std::invalid_argument("sycl_tla_dense_gemm: output dtype must match input dtype");+ }++ switch (at) {+ case BTLA_DTYPE::F32:+ sycl_tla_dense_gemm_f32(q, m, n, k, a, b, c, bias);+ return;+ case BTLA_DTYPE::F16:+ sycl_tla_dense_gemm_f16(q, m, n, k, a, b, c, bias);+ return;+ case BTLA_DTYPE::BF16:+ sycl_tla_dense_gemm_bf16(q, m, n, k, a, b, c, bias);+ return;+ default:+ throw std::invalid_argument("sycl_tla_dense_gemm: only FP32, FP16 and BF16 are supported");+ }+}++} // namespace ark++#elif @DENSE_SOURCE_MODE@ == 3++#include "sycl_tla_dense_gemm_helpers.hpp"++namespace ark {++void sycl_tla_dense_gemm_@DENSE_DISPATCH_DTYPE@(sycl::queue* q, int m, int n, int k, const void* a, const void* b,+ void* c, const void* bias) {+ if (bias != nullptr) {+ if (m < 32) dense_gemm_detail::run_@DENSE_DISPATCH_DTYPE@_with_bias_small(q, m, n, k, a, b, c, bias);+ else if (m < 128) dense_gemm_detail::run_@DENSE_DISPATCH_DTYPE@_with_bias_mid(q, m, n, k, a, b, c, bias);+ else if (m <= 1024) dense_gemm_detail::run_@DENSE_DISPATCH_DTYPE@_with_bias_medium(q, m, n, k, a, b, c, bias);+ else dense_gemm_detail::run_@DENSE_DISPATCH_DTYPE@_with_bias_large(q, m, n, k, a, b, c, bias);+ } else {+ if (m < 32) dense_gemm_detail::run_@DENSE_DISPATCH_DTYPE@_without_bias_small(q, m, n, k, a, b, c);+ else if (m < 128) dense_gemm_detail::run_@DENSE_DISPATCH_DTYPE@_without_bias_mid(q, m, n, k, a, b, c);+ else if (m <= 1024) dense_gemm_detail::run_@DENSE_DISPATCH_DTYPE@_without_bias_medium(q, m, n, k, a, b, c);+ else dense_gemm_detail::run_@DENSE_DISPATCH_DTYPE@_without_bias_large(q, m, n, k, a, b, c);+ }+}++} // namespace ark++#endif++#elif @S8_SOURCE_MODE@ > 0++#if @S8_SOURCE_MODE@ == 1++#include "sycl_tla_s8_gemm_fixed.hpp"++@S8_ROUTE_MACRO@(@S8_FUNCTION_NAME@, @S8_ELEMENT_TYPE@, @S8_TILE_M@, @S8_TILE_N@, @S8_LAYOUT@)++#elif @S8_SOURCE_MODE@ == 2++#include "sycl_tla_common.hpp"++namespace ark {++void sycl_tla_igemm_s8s8_dequant(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,+ BTLA_DTYPE ct, const void* scale_a, const void* scale_b, const void* bias,+ int blocksize) {+ if (!q) throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: queue must not be null");+ if (!a || !b || !c || !scale_a || !scale_b) {+ throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: input pointers must not be null");+ }+ if (m <= 0 || n <= 0 || k <= 0) return;+ if (ct == BTLA_DTYPE::BF16 && !(blocksize == k || blocksize == -1)) {+ throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: k-block path supports only F32/F16 output");+ }++ switch (ct) {+ case BTLA_DTYPE::F32:+ sycl_tla_igemm_s8s8_dequant_f32(q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize);+ return;+ case BTLA_DTYPE::F16:+ sycl_tla_igemm_s8s8_dequant_f16(q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize);+ return;+ case BTLA_DTYPE::BF16:+ sycl_tla_igemm_s8s8_dequant_bf16(q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize);+ return;+ default:+ throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: unsupported output dtype");+ }+}++} // namespace ark++#elif @S8_SOURCE_MODE@ == 3++#include "sycl_tla_s8_gemm_helpers.hpp"++namespace ark {++void sycl_tla_igemm_s8s8_dequant_@S8_DISPATCH_DTYPE@(sycl::queue* q, int m, int n, int k, const void* a,+ const void* b, void* c, const void* scale_a, const void* scale_b,+ const void* bias, int blocksize) {+ if (blocksize == k || blocksize == -1) {+ if (m < 16) sycl_tla_s8_detail::run_@S8_DISPATCH_DTYPE@_normal_small(q, m, n, k, a, b, c, scale_a, scale_b, bias);+ else if (m < 128) sycl_tla_s8_detail::run_@S8_DISPATCH_DTYPE@_normal_mid(q, m, n, k, a, b, c, scale_a, scale_b, bias);+ else if (m <= 1024) sycl_tla_s8_detail::run_@S8_DISPATCH_DTYPE@_normal_medium(q, m, n, k, a, b, c, scale_a, scale_b, bias);+ else sycl_tla_s8_detail::run_@S8_DISPATCH_DTYPE@_normal_large(q, m, n, k, a, b, c, scale_a, scale_b, bias);+ return;+ }+#if @S8_KBLOCK_SUPPORTED@+ sycl_tla_s8_detail::validate_kblock_args(k, blocksize);+ if (m < 16) sycl_tla_s8_detail::run_@S8_DISPATCH_DTYPE@_kblock_small(q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize);+ else if (m < 128) sycl_tla_s8_detail::run_@S8_DISPATCH_DTYPE@_kblock_mid(q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize);+ else if (m <= 1024) sycl_tla_s8_detail::run_@S8_DISPATCH_DTYPE@_kblock_medium(q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize);+ else sycl_tla_s8_detail::run_@S8_DISPATCH_DTYPE@_kblock_large(q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize);+#else+ throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: k-block path supports only F32/F16 output");+#endif+}++} // namespace ark++#endif++#endif++#endif // ARK_XPU && ARK_SYCL_TLA\ No newline at end of filediff --git a/auto_round_extension/ark/auto_round_kernel/sycl_tla_generation.cmake b/auto_round_extension/ark/auto_round_kernel/sycl_tla_generation.cmakenew file mode 100644index 000000000..811f781a0--- /dev/null+++ b/auto_round_extension/ark/auto_round_kernel/sycl_tla_generation.cmake@@ -0,0 +1,401 @@+set(SYCL_TLA_GENERATED_DIR ${CMAKE_CURRENT_BINARY_DIR}/generated/sycl_tla)+set(SYCL_TLA_TEMPLATE_DIR ${CMAKE_CURRENT_LIST_DIR})+file(MAKE_DIRECTORY ${SYCL_TLA_GENERATED_DIR})++function(generate_sycl_tla_source template_name output_name)+ set(_generated_src ${SYCL_TLA_GENERATED_DIR}/${output_name})+ configure_file(${SYCL_TLA_TEMPLATE_DIR}/${template_name} ${_generated_src} @ONLY)+ list(APPEND SYCL_TLA_GENERATED_SRCS ${_generated_src})+ set(SYCL_TLA_GENERATED_SRCS ${SYCL_TLA_GENERATED_SRCS} PARENT_SCOPE)+endfunction()++set(DENSE_SOURCE_MODE 1)+set(S8_SOURCE_MODE 0)+set(_dense_dtypes f16 bf16 f32)+foreach(_dense_dtype IN LISTS _dense_dtypes)+ if(_dense_dtype STREQUAL "f16")+ set(DENSE_ELEMENT_TYPE "cute::half_t")+ elseif(_dense_dtype STREQUAL "bf16")+ set(DENSE_ELEMENT_TYPE "cute::bfloat16_t")+ else()+ set(DENSE_ELEMENT_TYPE "float")+ endif()++ foreach(_dense_bias IN ITEMS bias no_bias)+ if(_dense_bias STREQUAL "bias")+ set(DENSE_HAS_BIAS true)+ set(DENSE_FUNCTION_BIAS with_bias)+ set(DENSE_BIAS_PARAMETER ", const void* bias")+ set(DENSE_BIAS_ARGUMENT "static_cast<const ${DENSE_ELEMENT_TYPE}*>(bias)")+ else()+ set(DENSE_HAS_BIAS false)+ set(DENSE_FUNCTION_BIAS without_bias)+ set(DENSE_BIAS_PARAMETER "")+ set(DENSE_BIAS_ARGUMENT "static_cast<const ${DENSE_ELEMENT_TYPE}*>(nullptr)")+ endif()++ foreach(_dense_tile IN ITEMS small mid medium large)+ if(_dense_tile STREQUAL "small")+ set(DENSE_TILE_M 8)+ set(DENSE_TILE_N 128)+ set(DENSE_LAYOUT DenseSmallTileSG)+ set(_dense_output sycl_tla_dense_gemm_${_dense_dtype}_${_dense_bias}.cpp)+ elseif(_dense_tile STREQUAL "mid")+ set(DENSE_TILE_M 64)+ set(DENSE_TILE_N 128)+ set(DENSE_LAYOUT DenseSmallMidTileSG)+ set(_dense_output sycl_tla_dense_gemm_${_dense_dtype}_${_dense_bias}_mid.cpp)+ elseif(_dense_tile STREQUAL "medium")+ set(DENSE_TILE_M 128)+ set(DENSE_TILE_N 128)+ set(DENSE_LAYOUT DenseMediumTileSG)+ set(_dense_output sycl_tla_dense_gemm_${_dense_dtype}_${_dense_bias}_medium.cpp)+ else()+ set(DENSE_TILE_M 256)+ set(DENSE_TILE_N 256)+ set(DENSE_LAYOUT DenseLargeTileSG)+ set(_dense_output sycl_tla_dense_gemm_${_dense_dtype}_${_dense_bias}_large.cpp)+ endif()++ set(DENSE_FUNCTION_NAME run_${_dense_dtype}_${DENSE_FUNCTION_BIAS}_${_dense_tile})+ generate_sycl_tla_source(sycl_tla_gemm.cpp.in ${_dense_output})+ endforeach()+ endforeach()+endforeach()++set(DENSE_SOURCE_MODE 0)+set(S8_SOURCE_MODE 1)+set(S8_KBLOCK_SUPPORTED 0)+set(_s8_dtypes f16 bf16 f32)+foreach(_s8_dtype IN LISTS _s8_dtypes)+ if(_s8_dtype STREQUAL "f16")+ set(S8_ELEMENT_TYPE "cute::half_t")+ elseif(_s8_dtype STREQUAL "bf16")+ set(S8_ELEMENT_TYPE "cute::bfloat16_t")+ else()+ set(S8_ELEMENT_TYPE "float")+ endif()++ set(_s8_paths normal)+ if(NOT _s8_dtype STREQUAL "bf16")+ list(APPEND _s8_paths kblock)+ endif()++ foreach(_s8_path IN LISTS _s8_paths)+ if(_s8_path STREQUAL "normal")+ set(S8_ROUTE_MACRO ARK_DEFINE_S8_NORMAL_ROUTE)+ else()+ set(S8_ROUTE_MACRO ARK_DEFINE_S8_KBLOCK_ROUTE)+ endif()++ foreach(_s8_tile IN ITEMS small mid medium large)+ if(_s8_tile STREQUAL "small")+ set(S8_TILE_M 8)+ set(S8_TILE_N 128)+ elseif(_s8_tile STREQUAL "mid")+ set(S8_TILE_M 64)+ set(S8_TILE_N 128)+ elseif(_s8_tile STREQUAL "medium")+ set(S8_TILE_M 128)+ set(S8_TILE_N 128)+ elseif(_s8_path STREQUAL "normal")+ set(S8_TILE_M 256)+ set(S8_TILE_N 256)+ else()+ set(S8_TILE_M 256)+ set(S8_TILE_N 128)+ endif()++ if(_s8_tile STREQUAL "small")+ set(_s8_output_suffix small)+ set(S8_LAYOUT SmallTileSG)+ elseif(_s8_tile STREQUAL "mid")+ set(_s8_output_suffix mid)+ set(S8_LAYOUT SmallMidTileSG)+ elseif(_s8_tile STREQUAL "medium")+ set(_s8_output_suffix medium)+ set(S8_LAYOUT MediumTileSG)+ else()+ set(_s8_output_suffix large)+ set(S8_LAYOUT LargeTileSG)+ endif()++ set(S8_FUNCTION_NAME run_${_s8_dtype}_${_s8_path}_${_s8_tile})+ set(_s8_output sycl_tla_s8_gemm_${_s8_dtype}_${_s8_path}_${_s8_output_suffix}.cpp)+ generate_sycl_tla_source(sycl_tla_gemm.cpp.in ${_s8_output})+ endforeach()+ endforeach()+endforeach()++set(MOE_SOURCE_MODE 8)+set(MOE_DEFINE_FP8_HELPERS 0)+set(MOE_DEFINE_INT8_HELPERS 0)+set(MOE_DEFINE_S4_HELPERS 0)+set(_moe_tiles small mid large)+foreach(_moe_tile IN LISTS _moe_tiles)+ if(_moe_tile STREQUAL "small")+ set(MOE_POLICY_SUFFIX small)+ set(MOE_POLICY_NAME dpas_w8a16_policy_m_16)+ elseif(_moe_tile STREQUAL "mid")+ set(MOE_POLICY_SUFFIX mid)+ set(MOE_POLICY_NAME dpas_w8a16_policy_m_32)+ else()+ set(MOE_POLICY_SUFFIX large)+ set(MOE_POLICY_NAME dpas_w8a16_policy)+ endif()++ foreach(_moe_dtype IN ITEMS f16 bf16)+ if(_moe_dtype STREQUAL "f16")+ set(MOE_SCALAR_TYPE sycl::half)+ else()+ set(MOE_SCALAR_TYPE sycl::ext::oneapi::bfloat16)+ endif()++ foreach(_moe_format IN ITEMS e4m3 e5m2)+ if(_moe_format STREQUAL "e4m3")+ set(MOE_FORMAT_ARGUMENT true)+ else()+ set(MOE_FORMAT_ARGUMENT false)+ endif()+ set(MOE_POLICY_ARGS "${MOE_FORMAT_ARGUMENT}, moe_dpas_fp8::${MOE_POLICY_NAME}")+ set(MOE_FUNCTION_NAME dispatch_${_moe_dtype}_${_moe_format}_${MOE_POLICY_SUFFIX})+ if(_moe_tile STREQUAL "small")+ set(_moe_output_suffix "")+ else()+ set(_moe_output_suffix _${MOE_POLICY_SUFFIX})+ endif()+ set(_moe_output sycl_tla_moe_prefill_fp8_${_moe_dtype}_${_moe_format}${_moe_output_suffix}.cpp)+ set(MOE_HELPER_HEADER sycl_tla_moe_prefill_fp8_helpers.hpp)+ set(MOE_DPAS_HEADER sycl_tla_moe_prefill_fp8_dpas.hpp)+ set(MOE_NAMESPACE moe_prefill_fp8_detail)+ set(MOE_DPAS_NAMESPACE moe_dpas_fp8)+ set(MOE_DISPATCH_FUNCTION moe_prefill_fp8_dpas_per_group_dispatch_policy)+ set(MOE_WEIGHT_TYPE uint8_t)+ set(MOE_DEFINE_FP8_HELPERS 0)+ set(MOE_DEFINE_INT8_HELPERS 0)+ set(MOE_DEFINE_S4_HELPERS 0)+ if(_moe_tile STREQUAL "small" AND _moe_dtype STREQUAL "f16" AND _moe_format STREQUAL "e4m3")+ set(MOE_DEFINE_FP8_HELPERS 1)+ endif()+ generate_sycl_tla_source(sycl_tla_moe.cpp.in ${_moe_output})+ endforeach()++ set(MOE_POLICY_ARGS "moe_dpas_int::${MOE_POLICY_NAME}")+ set(MOE_FUNCTION_NAME dispatch_${_moe_dtype}_${MOE_POLICY_SUFFIX})+ if(_moe_tile STREQUAL "small")+ set(_moe_output_suffix "")+ else()+ set(_moe_output_suffix _${MOE_POLICY_SUFFIX})+ endif()+ set(_moe_output sycl_tla_moe_prefill_int8_${_moe_dtype}${_moe_output_suffix}.cpp)+ set(MOE_HELPER_HEADER sycl_tla_moe_prefill_int8_helpers.hpp)+ set(MOE_DPAS_HEADER sycl_tla_moe_prefill_int_dpas.hpp)+ set(MOE_NAMESPACE moe_prefill_int8_detail)+ set(MOE_DPAS_NAMESPACE moe_dpas_int)+ set(MOE_DISPATCH_FUNCTION moe_prefill_int_dpas_per_group_dispatch_policy)+ set(MOE_WEIGHT_TYPE int8_t)+ set(MOE_DEFINE_FP8_HELPERS 0)+ set(MOE_DEFINE_INT8_HELPERS 0)+ set(MOE_DEFINE_S4_HELPERS 0)+ if(_moe_tile STREQUAL "small" AND _moe_dtype STREQUAL "f16")+ set(MOE_DEFINE_INT8_HELPERS 1)+ endif()+ generate_sycl_tla_source(sycl_tla_moe.cpp.in ${_moe_output})++ set(MOE_POLICY_ARGS "moe_dpas_s4::${MOE_POLICY_NAME}")+ set(MOE_FUNCTION_NAME dispatch_${_moe_dtype}_${MOE_POLICY_SUFFIX})+ set(_moe_output sycl_tla_moe_prefill_s4_${_moe_dtype}${_moe_output_suffix}.cpp)+ set(MOE_HELPER_HEADER sycl_tla_moe_prefill_s4_helpers.hpp)+ set(MOE_DPAS_HEADER sycl_tla_moe_prefill_s4_dpas.hpp)+ set(MOE_NAMESPACE moe_prefill_s4_detail)+ set(MOE_DPAS_NAMESPACE moe_dpas_s4)+ set(MOE_DISPATCH_FUNCTION moe_prefill_s4_dpas_per_group_dispatch_policy)+ set(MOE_WEIGHT_TYPE uint8_t)+ set(MOE_DEFINE_FP8_HELPERS 0)+ set(MOE_DEFINE_INT8_HELPERS 0)+ set(MOE_DEFINE_S4_HELPERS 0)+ if(_moe_tile STREQUAL "small" AND _moe_dtype STREQUAL "f16")+ set(MOE_DEFINE_S4_HELPERS 1)+ endif()+ generate_sycl_tla_source(sycl_tla_moe.cpp.in ${_moe_output})+ endforeach()+endforeach()++set(MOE_SOURCE_MODE 9)+foreach(_moe_dtype IN ITEMS f16 bf16)+ if(_moe_dtype STREQUAL "f16")+ set(MOE_SCALAR_TYPE sycl::half)+ else()+ set(MOE_SCALAR_TYPE sycl::ext::oneapi::bfloat16)+ endif()+ foreach(_moe_tile IN LISTS _moe_tiles)+ if(_moe_tile STREQUAL "small")+ set(MOE_POLICY_SUFFIX small)+ set(MOE_POLICY_NAME dpas_w8a16_policy_m_16)+ elseif(_moe_tile STREQUAL "mid")+ set(MOE_POLICY_SUFFIX mid)+ set(MOE_POLICY_NAME dpas_w8a16_policy_m_32)+ else()+ set(MOE_POLICY_SUFFIX large)+ set(MOE_POLICY_NAME dpas_w8a16_policy)+ endif()+ set(MOE_POLICY_ARGS "moe_dpas_int::${MOE_POLICY_NAME}")+ set(MOE_FUNCTION_NAME dispatch_${_moe_dtype}_${MOE_POLICY_SUFFIX})+ set(_moe_output sycl_tla_moe_prefill_int_tensor_${_moe_dtype}_${MOE_POLICY_SUFFIX}.cpp)+ set(MOE_HELPER_HEADER sycl_tla_moe_prefill_int_tensor_helpers.hpp)+ set(MOE_DPAS_HEADER sycl_tla_moe_prefill_int_dpas.hpp)+ set(MOE_NAMESPACE moe_prefill_int_tensor_detail)+ set(MOE_DPAS_NAMESPACE moe_dpas_int)+ set(MOE_DISPATCH_FUNCTION moe_prefill_int_dpas_per_tensor_dispatch_policy)+ set(MOE_WEIGHT_TYPE int8_t)+ set(MOE_SCALE_TYPE float)+ generate_sycl_tla_source(sycl_tla_moe.cpp.in ${_moe_output})+ endforeach()+endforeach()++set(DENSE_SOURCE_MODE 2)+set(S8_SOURCE_MODE 0)+generate_sycl_tla_source(sycl_tla_gemm.cpp.in sycl_tla_dense_gemm.cpp)+foreach(_dense_dispatch_dtype IN ITEMS f16 bf16 f32)+ set(DENSE_SOURCE_MODE 3)+ set(DENSE_DISPATCH_DTYPE ${_dense_dispatch_dtype})+ generate_sycl_tla_source(sycl_tla_gemm.cpp.in sycl_tla_dense_gemm_${_dense_dispatch_dtype}.cpp)+endforeach()++set(DENSE_SOURCE_MODE 0)+set(S8_SOURCE_MODE 2)+generate_sycl_tla_source(sycl_tla_gemm.cpp.in sycl_tla_s8_gemm.cpp)+foreach(_s8_dispatch_dtype IN ITEMS f16 bf16 f32)+ set(S8_SOURCE_MODE 3)+ set(S8_DISPATCH_DTYPE ${_s8_dispatch_dtype})+ if(_s8_dispatch_dtype STREQUAL "bf16")+ set(S8_KBLOCK_SUPPORTED 0)+ else()+ set(S8_KBLOCK_SUPPORTED 1)+ endif()+ generate_sycl_tla_source(sycl_tla_gemm.cpp.in sycl_tla_s8_gemm_${_s8_dispatch_dtype}.cpp)+endforeach()++set(MOE_SOURCE_MODE 1)+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe.cpp)+foreach(_moe_dispatch_dtype IN ITEMS f16 bf16)+ set(MOE_DISPATCH_DTYPE ${_moe_dispatch_dtype})+ if(_moe_dispatch_dtype STREQUAL "f16")+ set(MOE_DISPATCH_ELEMENT cutlass::half_t)+ else()+ set(MOE_DISPATCH_ELEMENT cutlass::bfloat16_t)+ endif()+ set(MOE_SOURCE_MODE 2)+ generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_${_moe_dispatch_dtype}.cpp)+endforeach()++set(MOE_SOURCE_MODE 3)+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_decode.cpp)+set(MOE_SOURCE_MODE 4)+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_decode_fp.cpp)+set(MOE_SOURCE_MODE 5)+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_decode_fp8.cpp)+foreach(_decode_int_kind IN ITEMS int2 int4 int8)+ set(DECODE_INT_KIND ${_decode_int_kind})+ set(MOE_SOURCE_MODE 6)+ generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_decode_${_decode_int_kind}.cpp)+endforeach()++set(MOE_SOURCE_MODE 7)+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_mixed.cpp)+set(MOE_SOURCE_MODE 11)+foreach(_moe_dequant_dtype IN ITEMS f16 bf16)+ set(MOE_DEQUANT_DTYPE ${_moe_dequant_dtype})+ if(_moe_dequant_dtype STREQUAL "f16")+ set(MOE_DEQUANT_SCALAR sycl::half)+ else()+ set(MOE_DEQUANT_SCALAR sycl::ext::oneapi::bfloat16)+ endif()+ generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_prefill_dequant_${_moe_dequant_dtype}.cpp)+endforeach()+set(MOE_SOURCE_MODE 12)+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_prefill_fp8.cpp)+set(MOE_SOURCE_MODE 13)+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_prefill_fp8_tensor.cpp)+set(MOE_SOURCE_MODE 14)+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_prefill_int8.cpp)+set(MOE_SOURCE_MODE 15)+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_prefill_int_tensor.cpp)+set(MOE_SOURCE_MODE 16)+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_prefill_s4.cpp)+set(MOE_SOURCE_MODE 17)+generate_sycl_tla_source(sycl_tla_moe.cpp.in sycl_tla_moe_prefill_native_fp8.cpp)++foreach(_moe_dtype IN ITEMS f16 bf16)+ if(_moe_dtype STREQUAL "f16")+ set(MOE_SCALAR_TYPE sycl::half)+ else()+ set(MOE_SCALAR_TYPE sycl::ext::oneapi::bfloat16)+ endif()+ foreach(_moe_format IN ITEMS e4m3 e5m2)+ if(_moe_format STREQUAL "e4m3")+ set(MOE_FORMAT_ARGUMENT true)+ else()+ set(MOE_FORMAT_ARGUMENT false)+ endif()+ foreach(_moe_tile IN LISTS _moe_tiles)+ if(_moe_tile STREQUAL "small")+ set(MOE_POLICY_SUFFIX small)+ set(MOE_POLICY_NAME dpas_w8a16_policy_m_16)+ elseif(_moe_tile STREQUAL "mid")+ set(MOE_POLICY_SUFFIX mid)+ set(MOE_POLICY_NAME dpas_w8a16_policy_m_32)+ else()+ set(MOE_POLICY_SUFFIX large)+ set(MOE_POLICY_NAME dpas_w8a16_policy)+ endif()+ set(MOE_POLICY_ARGS "${MOE_FORMAT_ARGUMENT}, moe_dpas_fp8::${MOE_POLICY_NAME}")+ set(MOE_FUNCTION_NAME dispatch_${_moe_dtype}_${_moe_format}_${MOE_POLICY_SUFFIX})+ set(_moe_output sycl_tla_moe_prefill_fp8_tensor_${_moe_dtype}_${_moe_format}_${MOE_POLICY_SUFFIX}.cpp)+ set(MOE_HELPER_HEADER sycl_tla_moe_prefill_fp8_tensor_helpers.hpp)+ set(MOE_DPAS_HEADER sycl_tla_moe_prefill_fp8_dpas.hpp)+ set(MOE_NAMESPACE moe_prefill_fp8_tensor_detail)+ set(MOE_DPAS_NAMESPACE moe_dpas_fp8)+ set(MOE_DISPATCH_FUNCTION moe_prefill_fp8_dpas_per_tensor_dispatch_policy)+ set(MOE_WEIGHT_TYPE uint8_t)+ set(MOE_SCALE_TYPE float)+ set(MOE_SOURCE_MODE 9)+ generate_sycl_tla_source(sycl_tla_moe.cpp.in ${_moe_output})+ endforeach()+ endforeach()+endforeach()++foreach(_moe_dtype IN ITEMS f16 bf16)+ if(_moe_dtype STREQUAL "f16")+ set(MOE_SCALAR_TYPE sycl::half)+ else()+ set(MOE_SCALAR_TYPE sycl::ext::oneapi::bfloat16)+ endif()+ foreach(_moe_tile IN LISTS _moe_tiles)+ if(_moe_tile STREQUAL "small")+ set(MOE_POLICY_SUFFIX small)+ set(MOE_POLICY_NAME dpas_w8a16_policy_m_16)+ elseif(_moe_tile STREQUAL "mid")+ set(MOE_POLICY_SUFFIX mid)+ set(MOE_POLICY_NAME dpas_w8a16_policy_m_32)+ else()+ set(MOE_POLICY_SUFFIX large)+ set(MOE_POLICY_NAME dpas_w8a16_policy)+ endif()+ set(MOE_POLICY_ARGS "moe_dpas_int::${MOE_POLICY_NAME}")+ set(MOE_FUNCTION_NAME dispatch_lowbit_${_moe_dtype}_${MOE_POLICY_SUFFIX})+ if(_moe_tile STREQUAL "small")+ set(_moe_output_suffix "")+ else()+ set(_moe_output_suffix _${MOE_POLICY_SUFFIX})+ endif()+ set(_moe_output sycl_tla_moe_prefill_int8_lowbit_${_moe_dtype}${_moe_output_suffix}.cpp)+ set(MOE_HELPER_HEADER sycl_tla_moe_prefill_int8_helpers.hpp)+ set(MOE_DPAS_HEADER sycl_tla_moe_prefill_int_dpas.hpp)+ set(MOE_NAMESPACE moe_prefill_int8_detail)+ set(MOE_DPAS_NAMESPACE moe_dpas_int)+ set(MOE_DISPATCH_FUNCTION moe_prefill_int_dpas_per_group_dispatch_policy)+ set(MOE_SOURCE_MODE 10)+ generate_sycl_tla_source(sycl_tla_moe.cpp.in ${_moe_output})+ endforeach()+endforeach()\ No newline at end of filediff --git a/auto_round_extension/ark/auto_round_kernel/sycl_tla_moe.cpp.in b/auto_round_extension/ark/auto_round_kernel/sycl_tla_moe.cpp.innew file mode 100644index 000000000..bc81a9714--- /dev/null+++ b/auto_round_extension/ark/auto_round_kernel/sycl_tla_moe.cpp.in@@ -0,0 +1,701 @@+// Generated by CMake. Do not edit directly.++#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)++#if @MOE_SOURCE_MODE@ == 1++#include "sycl_tla_common.hpp"++namespace ark {++void moe_gemm(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs, BTLA_DTYPE dtype,+ int N, int K, int* num_tokens_per_expert, int num_experts) {+ switch (dtype) {+ case BTLA_DTYPE::F16:+ sycl_tla_moe_gemm_f16(q, activations, weights, scales, outputs, N, K, num_tokens_per_expert, num_experts);+ return;+ case BTLA_DTYPE::BF16:+ sycl_tla_moe_gemm_bf16(q, activations, weights, scales, outputs, N, K, num_tokens_per_expert, num_experts);+ return;+ default:+ throw std::runtime_error("moe_gemm: unsupported dtype, only BF16/FP16 supported");+ }+}++} // namespace ark++#elif @MOE_SOURCE_MODE@ == 2++#include "sycl_tla_common.hpp"+#include "sycl_tla_moe.hpp"++namespace ark {++void sycl_tla_moe_gemm_@MOE_DISPATCH_DTYPE@(sycl::queue* q, void* activations, void* weights, void* scales,+ void* outputs, int N, int K, int* num_tokens_per_expert,+ int num_experts) {+ using Element = @MOE_DISPATCH_ELEMENT@;+ moe_detail::moe_gemm_launcher<'R', 'R', Element, Element, Element, Element>(+ q, static_cast<const Element*>(activations), static_cast<const Element*>(weights),+ static_cast<const Element*>(scales), static_cast<Element*>(outputs), N, K, num_tokens_per_expert, num_experts);+}++} // namespace ark++#elif @MOE_SOURCE_MODE@ == 3++#include "sycl_tla_common.hpp"++namespace ark {++void sycl_tla_moe_decode_fill_expert_id(sycl::queue* q, int* expert_id_per_token,+ const int* num_tokens_per_expert, int num_experts, int total_tokens) {+ if (total_tokens == 0) return;+ q->parallel_for(sycl::range<1>(static_cast<size_t>(total_tokens)), [=](sycl::id<1> idx) {+ const int token = static_cast<int>(idx[0]);+ int offset = 0;+ int expert = num_experts - 1;+ for (int expert_id = 0; expert_id < num_experts; ++expert_id) {+ const int token_count = num_tokens_per_expert[expert_id];+ if (token < offset + token_count) {+ expert = expert_id;+ break;+ }+ offset += token_count;+ }+ expert_id_per_token[token] = expert;+ });+}++void moe_gemm_decode(sycl::queue* q, void* activations, void* weights, void* scales, void* zeros, void* outputs,+ int* expert_id_per_token_buf, BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,+ int group_size, int* num_tokens_per_expert, int num_experts, int total_tokens, bool asym) {+ sycl_tla_moe_decode_fill_expert_id(q, expert_id_per_token_buf, num_tokens_per_expert, num_experts, total_tokens);++ MoeDecodeParams params{q, activations, weights, scales, zeros, outputs, expert_id_per_token_buf, act_dtype,+ weight_dtype, N, K, group_size, total_tokens, asym};++ if (weight_dtype == BTLA_DTYPE::F16 || weight_dtype == BTLA_DTYPE::BF16) {+ if (weight_dtype != act_dtype) {+ throw std::invalid_argument("moe_gemm_decode: unquantized weight_dtype must match act_dtype");+ }+ sycl_tla_moe_decode_fp(params);+ return;+ }+ if (weight_dtype == BTLA_DTYPE::S4_CLIP) {+ sycl_tla_moe_decode_int4(params);+ return;+ }+ if (weight_dtype == BTLA_DTYPE::S8) {+ sycl_tla_moe_decode_int8(params);+ return;+ }+ if (weight_dtype == BTLA_DTYPE::S2_CLIP) {+ sycl_tla_moe_decode_int2(params);+ return;+ }+ if (weight_dtype == BTLA_DTYPE::F8_E4M3 || weight_dtype == BTLA_DTYPE::F8_E5M2) {+ if (asym) {+ throw std::invalid_argument("moe_gemm_decode(fp8): asym mode is not supported");+ }+ sycl_tla_moe_decode_fp8(params);+ return;+ }+ throw std::invalid_argument(+ "moe_gemm_decode: unsupported weight_dtype (supported: F16, BF16, S8, S4_CLIP, S2_CLIP, F8_E4M3, F8_E5M2)");+}++} // namespace ark++#elif @MOE_SOURCE_MODE@ == 4++#include "sycl_tla_common.hpp"+#include "sycl_tla_moe_decode.hpp"++namespace ark {++void sycl_tla_moe_decode_fp(const MoeDecodeParams& params) {+ if (params.act_dtype == BTLA_DTYPE::F16) {+ moe_decode_detail::launch_fp<sycl::half>(+ params.q, static_cast<const sycl::half*>(params.activations), static_cast<const sycl::half*>(params.weights),+ static_cast<sycl::half*>(params.outputs), params.expert_id_per_token, params.total_tokens, params.N,+ params.K);+ return;+ }+ if (params.act_dtype == BTLA_DTYPE::BF16) {+ using BF = sycl::ext::oneapi::bfloat16;+ moe_decode_detail::launch_fp<BF>(+ params.q, static_cast<const BF*>(params.activations), static_cast<const BF*>(params.weights),+ static_cast<BF*>(params.outputs), params.expert_id_per_token, params.total_tokens, params.N, params.K);+ return;+ }+ throw std::invalid_argument("moe_gemm_decode(fp): act_dtype must be FP16 or BF16");+}++} // namespace ark++#elif @MOE_SOURCE_MODE@ == 5++#include "sycl_tla_common.hpp"+#include "sycl_tla_moe_decode.hpp"++namespace ark {++void sycl_tla_moe_decode_fp8(const MoeDecodeParams& params) {+ const bool is_e4m3 = params.weight_dtype == BTLA_DTYPE::F8_E4M3;+ const bool use_lut = moe_decode_detail::fp8_decode_use_lut();+ if (params.act_dtype == BTLA_DTYPE::F16) {+ if (is_e4m3) {+ if (use_lut) {+ moe_decode_detail::launch_fp8<sycl::half, true, true>(+ params.q, static_cast<const sycl::half*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const sycl::half*>(params.scales), static_cast<sycl::half*>(params.outputs),+ params.expert_id_per_token, params.total_tokens, params.N, params.K, params.group_size);+ } else {+ moe_decode_detail::launch_fp8<sycl::half, true, false>(+ params.q, static_cast<const sycl::half*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const sycl::half*>(params.scales), static_cast<sycl::half*>(params.outputs),+ params.expert_id_per_token, params.total_tokens, params.N, params.K, params.group_size);+ }+ } else if (use_lut) {+ moe_decode_detail::launch_fp8<sycl::half, false, true>(+ params.q, static_cast<const sycl::half*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const sycl::half*>(params.scales), static_cast<sycl::half*>(params.outputs),+ params.expert_id_per_token, params.total_tokens, params.N, params.K, params.group_size);+ } else {+ moe_decode_detail::launch_fp8<sycl::half, false, false>(+ params.q, static_cast<const sycl::half*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const sycl::half*>(params.scales), static_cast<sycl::half*>(params.outputs),+ params.expert_id_per_token, params.total_tokens, params.N, params.K, params.group_size);+ }+ return;+ }+ if (params.act_dtype == BTLA_DTYPE::BF16) {+ using BF = sycl::ext::oneapi::bfloat16;+ if (is_e4m3) {+ if (use_lut) {+ moe_decode_detail::launch_fp8<BF, true, true>(+ params.q, static_cast<const BF*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const BF*>(params.scales), static_cast<BF*>(params.outputs), params.expert_id_per_token,+ params.total_tokens, params.N, params.K, params.group_size);+ } else {+ moe_decode_detail::launch_fp8<BF, true, false>(+ params.q, static_cast<const BF*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const BF*>(params.scales), static_cast<BF*>(params.outputs), params.expert_id_per_token,+ params.total_tokens, params.N, params.K, params.group_size);+ }+ } else if (use_lut) {+ moe_decode_detail::launch_fp8<BF, false, true>(+ params.q, static_cast<const BF*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const BF*>(params.scales), static_cast<BF*>(params.outputs), params.expert_id_per_token,+ params.total_tokens, params.N, params.K, params.group_size);+ } else {+ moe_decode_detail::launch_fp8<BF, false, false>(+ params.q, static_cast<const BF*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const BF*>(params.scales), static_cast<BF*>(params.outputs), params.expert_id_per_token,+ params.total_tokens, params.N, params.K, params.group_size);+ }+ return;+ }+ throw std::invalid_argument("moe_gemm_decode(fp8): act_dtype must be FP16 or BF16");+}++} // namespace ark++#elif @MOE_SOURCE_MODE@ == 6++#include "sycl_tla_common.hpp"+#include "sycl_tla_moe_decode.hpp"++namespace ark {++void sycl_tla_moe_decode_@DECODE_INT_KIND@(const MoeDecodeParams& params) {+ if (params.act_dtype == BTLA_DTYPE::F16) {+ if (params.asym) {+ moe_decode_detail::launch_@DECODE_INT_KIND@<sycl::half, true>(+ params.q, static_cast<const sycl::half*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const sycl::half*>(params.scales), static_cast<const sycl::half*>(params.zeros),+ static_cast<sycl::half*>(params.outputs), params.expert_id_per_token, params.total_tokens, params.N,+ params.K, params.group_size);+ } else {+ moe_decode_detail::launch_@DECODE_INT_KIND@<sycl::half, false>(+ params.q, static_cast<const sycl::half*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const sycl::half*>(params.scales), static_cast<const sycl::half*>(params.zeros),+ static_cast<sycl::half*>(params.outputs), params.expert_id_per_token, params.total_tokens, params.N,+ params.K, params.group_size);+ }+ return;+ }+ if (params.act_dtype == BTLA_DTYPE::BF16) {+ using BF = sycl::ext::oneapi::bfloat16;+ if (params.asym) {+ moe_decode_detail::launch_@DECODE_INT_KIND@<BF, true>(+ params.q, static_cast<const BF*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const BF*>(params.scales), static_cast<const BF*>(params.zeros), static_cast<BF*>(params.outputs),+ params.expert_id_per_token, params.total_tokens, params.N, params.K, params.group_size);+ } else {+ moe_decode_detail::launch_@DECODE_INT_KIND@<BF, false>(+ params.q, static_cast<const BF*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const BF*>(params.scales), static_cast<const BF*>(params.zeros), static_cast<BF*>(params.outputs),+ params.expert_id_per_token, params.total_tokens, params.N, params.K, params.group_size);+ }+ return;+ }+ throw std::invalid_argument("moe_gemm_decode(@DECODE_INT_KIND@): act_dtype must be FP16 or BF16");+}++} // namespace ark++#elif @MOE_SOURCE_MODE@ == 7++#include "sycl_tla_common.hpp"++namespace ark {++void moe_gemm_prefill(sycl::queue* q, void* activations, void* weights, void* scales, void* zeros, void* outputs,+ void* dequant_workspace, BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,+ int group_size, int* num_tokens_per_expert, int num_experts, int total_tokens, bool asym) {+ if (total_tokens == 0) return;++ if (weight_dtype == act_dtype && weight_dtype == BTLA_DTYPE::F16) {+ sycl_tla_moe_gemm_f16(q, activations, weights, scales, outputs, N, K, num_tokens_per_expert, num_experts);+ return;+ }+ if (weight_dtype == act_dtype && weight_dtype == BTLA_DTYPE::BF16) {+ sycl_tla_moe_gemm_bf16(q, activations, weights, scales, outputs, N, K, num_tokens_per_expert, num_experts);+ return;+ }++ MoePrefillParams params{q, activations, weights, scales, zeros, outputs, dequant_workspace, act_dtype, weight_dtype,+ N, K, group_size, num_tokens_per_expert, num_experts, total_tokens, asym};++ if (sycl_tla_moe_prefill_int8_dpas(params)) return;+ if (sycl_tla_moe_prefill_s4_dpas(params)) return;+ if (sycl_tla_moe_prefill_lowbit_int8_dpas(params)) return;+ if (sycl_tla_moe_prefill_fp8_dpas(params)) return;+ if (sycl_tla_moe_prefill_native_fp8(params)) return;++ if (dequant_workspace == nullptr) {+ throw std::invalid_argument("moe_gemm_prefill: dequant_workspace must be non-null for quantized paths");+ }+ if (act_dtype == BTLA_DTYPE::F16) {+ sycl_tla_moe_prefill_dequant_f16(params);+ return;+ }+ if (act_dtype == BTLA_DTYPE::BF16) {+ sycl_tla_moe_prefill_dequant_bf16(params);+ return;+ }+ throw std::invalid_argument("moe_gemm_prefill: act_dtype must be F16 or BF16");+}++} // namespace ark++#elif @MOE_SOURCE_MODE@ == 8++#include "@MOE_HELPER_HEADER@"+#include "@MOE_DPAS_HEADER@"++namespace ark {+namespace @MOE_NAMESPACE@ {++#if @MOE_DEFINE_FP8_HELPERS@+bool enabled() { return moe_dpas_fp8::moe_prefill_dpas_fp8_enabled(); }+bool shape_ok(int N, int K, int group_size) {+ return moe_dpas_fp8::moe_prefill_dpas_fp8_pergroup_shape_ok(N, K, group_size);+}+#endif+#if @MOE_DEFINE_INT8_HELPERS@+bool enabled() { return moe_dpas_int::moe_prefill_dpas_int_enabled(); }+bool shape_ok(int N, int K, int group_size) {+ return moe_dpas_int::moe_prefill_dpas_int_pergroup_shape_ok(N, K, group_size);+}+#endif+#if @MOE_DEFINE_S4_HELPERS@+bool enabled() { return moe_dpas_s4::moe_prefill_dpas_s4_enabled(); }+bool shape_ok(int N, int K, int group_size) {+ return moe_dpas_s4::moe_prefill_dpas_s4_pergroup_shape_ok(N, K, group_size);+}+#endif++void @MOE_FUNCTION_NAME@(const MoePrefillParams& params) {+ using ScalarT = @MOE_SCALAR_TYPE@;+ @MOE_DPAS_NAMESPACE@::@MOE_DISPATCH_FUNCTION@<ScalarT, @MOE_POLICY_ARGS@>(+ params.q, static_cast<const ScalarT*>(params.activations),+ static_cast<const @MOE_WEIGHT_TYPE@*>(params.weights), static_cast<const ScalarT*>(params.scales),+ static_cast<ScalarT*>(params.outputs), params.num_tokens_per_expert, params.num_experts, params.N, params.K,+ params.group_size, params.total_tokens);+}++} // namespace @MOE_NAMESPACE@+} // namespace ark++#elif @MOE_SOURCE_MODE@ == 9++#include "@MOE_HELPER_HEADER@"+#include "@MOE_DPAS_HEADER@"++namespace ark {+namespace @MOE_NAMESPACE@ {++void @MOE_FUNCTION_NAME@(const MoePrefillParams& params) {+ using ScalarT = @MOE_SCALAR_TYPE@;+ @MOE_DPAS_NAMESPACE@::@MOE_DISPATCH_FUNCTION@<ScalarT, @MOE_POLICY_ARGS@>(+ params.q, static_cast<const ScalarT*>(params.activations),+ static_cast<const @MOE_WEIGHT_TYPE@*>(params.weights), static_cast<const @MOE_SCALE_TYPE@*>(params.scales),+ static_cast<ScalarT*>(params.outputs), params.num_tokens_per_expert, params.num_experts, params.N, params.K,+ params.total_tokens);+}++} // namespace @MOE_NAMESPACE@+} // namespace ark++#elif @MOE_SOURCE_MODE@ == 10++#include "@MOE_HELPER_HEADER@"+#include "@MOE_DPAS_HEADER@"+#include "sycl_tla_moe_prefill_upcast.hpp"++namespace ark {+namespace @MOE_NAMESPACE@ {++void @MOE_FUNCTION_NAME@(const MoePrefillParams& params) {+ using ScalarT = @MOE_SCALAR_TYPE@;+ auto* upcast_i8 = static_cast<int8_t*>(params.dequant_workspace);+ if (params.weight_dtype == BTLA_DTYPE::S4_CLIP) {+ moe_mixed_detail::launch_upcast_int4_sym_to_int8(+ params.q, static_cast<const uint8_t*>(params.weights), upcast_i8, params.num_experts, params.N, params.K,+ params.num_tokens_per_expert);+ } else {+ moe_mixed_detail::launch_upcast_int2_sym_to_int8(+ params.q, static_cast<const uint8_t*>(params.weights), upcast_i8, params.num_experts, params.N, params.K,+ params.num_tokens_per_expert);+ }+ @MOE_DPAS_NAMESPACE@::@MOE_DISPATCH_FUNCTION@<ScalarT, @MOE_POLICY_ARGS@>(+ params.q, static_cast<const ScalarT*>(params.activations), upcast_i8, static_cast<const ScalarT*>(params.scales),+ static_cast<ScalarT*>(params.outputs), params.num_tokens_per_expert, params.num_experts, params.N, params.K,+ params.group_size, params.total_tokens);+}++} // namespace @MOE_NAMESPACE@+} // namespace ark++#elif @MOE_SOURCE_MODE@ == 11++#include "sycl_tla_common.hpp"+#include "sycl_tla_moe_mixed.hpp"++namespace ark {++void sycl_tla_moe_prefill_dequant_@MOE_DEQUANT_DTYPE@(const MoePrefillParams& params) {+ using ScalarT = @MOE_DEQUANT_SCALAR@;+ auto* weights_kn = static_cast<ScalarT*>(params.dequant_workspace);+ moe_mixed_detail::dequant_to_KN<ScalarT>(+ params.q, params.weights, params.scales, params.zeros, weights_kn, params.weight_dtype, params.num_experts,+ params.N, params.K, params.group_size, params.asym, params.num_tokens_per_expert);+ moe_gemm(params.q, params.activations, weights_kn, nullptr, params.outputs, params.act_dtype, params.N, params.K,+ params.num_tokens_per_expert, params.num_experts);+}++} // namespace ark++#elif @MOE_SOURCE_MODE@ == 12++#include "sycl_tla_common.hpp"+#include "sycl_tla_moe_prefill_fp8_helpers.hpp"++namespace ark {++bool sycl_tla_moe_prefill_fp8_dpas(const MoePrefillParams& params) {+ if ((params.weight_dtype != BTLA_DTYPE::F8_E4M3 && params.weight_dtype != BTLA_DTYPE::F8_E5M2) || params.asym ||+ !moe_prefill_fp8_detail::enabled() ||+ !moe_prefill_fp8_detail::shape_ok(params.N, params.K, params.group_size)) {+ return false;+ }+ const bool is_e4m3 = params.weight_dtype == BTLA_DTYPE::F8_E4M3;+ const int average_tokens = params.num_experts > 0 ? params.total_tokens / params.num_experts : 0;+ if (params.act_dtype == BTLA_DTYPE::F16) {+ if (is_e4m3) {+ if (average_tokens <= 8) moe_prefill_fp8_detail::dispatch_f16_e4m3_small(params);+ else if (average_tokens <= 32) moe_prefill_fp8_detail::dispatch_f16_e4m3_mid(params);+ else moe_prefill_fp8_detail::dispatch_f16_e4m3_large(params);+ } else {+ if (average_tokens <= 8) moe_prefill_fp8_detail::dispatch_f16_e5m2_small(params);+ else if (average_tokens <= 32) moe_prefill_fp8_detail::dispatch_f16_e5m2_mid(params);+ else moe_prefill_fp8_detail::dispatch_f16_e5m2_large(params);+ }+ return true;+ }+ if (params.act_dtype == BTLA_DTYPE::BF16) {+ if (is_e4m3) {+ if (average_tokens <= 8) moe_prefill_fp8_detail::dispatch_bf16_e4m3_small(params);+ else if (average_tokens <= 32) moe_prefill_fp8_detail::dispatch_bf16_e4m3_mid(params);+ else moe_prefill_fp8_detail::dispatch_bf16_e4m3_large(params);+ } else {+ if (average_tokens <= 8) moe_prefill_fp8_detail::dispatch_bf16_e5m2_small(params);+ else if (average_tokens <= 32) moe_prefill_fp8_detail::dispatch_bf16_e5m2_mid(params);+ else moe_prefill_fp8_detail::dispatch_bf16_e5m2_large(params);+ }+ return true;+ }+ return false;+}++} // namespace ark++#elif @MOE_SOURCE_MODE@ == 13++#include "sycl_tla_common.hpp"+#include "sycl_tla_moe_prefill_fp8_tensor_helpers.hpp"++namespace ark {++void moe_gemm_prefill_fp8_dpas(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs,+ BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,+ int* num_tokens_per_expert, int num_experts, int total_tokens) {+ if (total_tokens == 0) return;+ if (weight_dtype != BTLA_DTYPE::F8_E4M3 && weight_dtype != BTLA_DTYPE::F8_E5M2) {+ throw std::invalid_argument("moe_gemm_prefill_fp8_dpas: weight_dtype must be F8_E4M3 or F8_E5M2");+ }+ MoePrefillParams params{q, activations, weights, scales, nullptr, outputs, nullptr, act_dtype, weight_dtype, N, K,+ 0, num_tokens_per_expert, num_experts, total_tokens, false};+ const int average_tokens = num_experts > 0 ? total_tokens / num_experts : 0;+ const bool is_e4m3 = weight_dtype == BTLA_DTYPE::F8_E4M3;+ if (act_dtype == BTLA_DTYPE::F16) {+ if (is_e4m3) {+ if (average_tokens <= 8) moe_prefill_fp8_tensor_detail::dispatch_f16_e4m3_small(params);+ else if (average_tokens <= 32) moe_prefill_fp8_tensor_detail::dispatch_f16_e4m3_mid(params);+ else moe_prefill_fp8_tensor_detail::dispatch_f16_e4m3_large(params);+ } else {+ if (average_tokens <= 8) moe_prefill_fp8_tensor_detail::dispatch_f16_e5m2_small(params);+ else if (average_tokens <= 32) moe_prefill_fp8_tensor_detail::dispatch_f16_e5m2_mid(params);+ else moe_prefill_fp8_tensor_detail::dispatch_f16_e5m2_large(params);+ }+ return;+ }+ if (act_dtype == BTLA_DTYPE::BF16) {+ if (is_e4m3) {+ if (average_tokens <= 8) moe_prefill_fp8_tensor_detail::dispatch_bf16_e4m3_small(params);+ else if (average_tokens <= 32) moe_prefill_fp8_tensor_detail::dispatch_bf16_e4m3_mid(params);+ else moe_prefill_fp8_tensor_detail::dispatch_bf16_e4m3_large(params);+ } else {+ if (average_tokens <= 8) moe_prefill_fp8_tensor_detail::dispatch_bf16_e5m2_small(params);+ else if (average_tokens <= 32) moe_prefill_fp8_tensor_detail::dispatch_bf16_e5m2_mid(params);+ else moe_prefill_fp8_tensor_detail::dispatch_bf16_e5m2_large(params);+ }+ return;+ }+ throw std::invalid_argument("moe_gemm_prefill_fp8_dpas: act_dtype must be F16 or BF16");+}++} // namespace ark++#elif @MOE_SOURCE_MODE@ == 14++#include "sycl_tla_common.hpp"+#include "sycl_tla_moe_prefill_int8_helpers.hpp"++namespace ark {++bool sycl_tla_moe_prefill_int8_dpas(const MoePrefillParams& params) {+ if (params.weight_dtype == BTLA_DTYPE::S8 && !params.asym &&+ moe_prefill_int8_detail::enabled() &&+ moe_prefill_int8_detail::shape_ok(params.N, params.K, params.group_size)) {+ const int average_tokens = params.num_experts > 0 ? params.total_tokens / params.num_experts : 0;+ if (params.act_dtype == BTLA_DTYPE::F16) {+ if (average_tokens <= 8) moe_prefill_int8_detail::dispatch_f16_small(params);+ else if (average_tokens <= 32) moe_prefill_int8_detail::dispatch_f16_mid(params);+ else moe_prefill_int8_detail::dispatch_f16_large(params);+ return true;+ }+ if (params.act_dtype == BTLA_DTYPE::BF16) {+ if (average_tokens <= 8) moe_prefill_int8_detail::dispatch_bf16_small(params);+ else if (average_tokens <= 32) moe_prefill_int8_detail::dispatch_bf16_mid(params);+ else moe_prefill_int8_detail::dispatch_bf16_large(params);+ return true;+ }+ }+ return false;+}++bool sycl_tla_moe_prefill_lowbit_int8_dpas(const MoePrefillParams& params) {+ if ((params.weight_dtype != BTLA_DTYPE::S4_CLIP && params.weight_dtype != BTLA_DTYPE::S2_CLIP) || params.asym ||+ !moe_prefill_int8_detail::enabled() ||+ !moe_prefill_int8_detail::shape_ok(params.N, params.K, params.group_size) ||+ params.dequant_workspace == nullptr ||+ (params.act_dtype != BTLA_DTYPE::F16 && params.act_dtype != BTLA_DTYPE::BF16)) {+ return false;+ }+ if (params.act_dtype == BTLA_DTYPE::F16) {+ const int average_tokens = params.num_experts > 0 ? params.total_tokens / params.num_experts : 0;+ if (average_tokens <= 8) moe_prefill_int8_detail::dispatch_lowbit_f16_small(params);+ else if (average_tokens <= 32) moe_prefill_int8_detail::dispatch_lowbit_f16_mid(params);+ else moe_prefill_int8_detail::dispatch_lowbit_f16_large(params);+ } else {+ const int average_tokens = params.num_experts > 0 ? params.total_tokens / params.num_experts : 0;+ if (average_tokens <= 8) moe_prefill_int8_detail::dispatch_lowbit_bf16_small(params);+ else if (average_tokens <= 32) moe_prefill_int8_detail::dispatch_lowbit_bf16_mid(params);+ else moe_prefill_int8_detail::dispatch_lowbit_bf16_large(params);+ }+ return true;+}++} // namespace ark++#elif @MOE_SOURCE_MODE@ == 15++#include "sycl_tla_common.hpp"+#include "sycl_tla_moe_prefill_int_tensor_helpers.hpp"++namespace ark {++void moe_gemm_prefill_int_dpas(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs,+ BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,+ int* num_tokens_per_expert, int num_experts, int total_tokens) {+ if (total_tokens == 0) return;+ if (weight_dtype != BTLA_DTYPE::S8) {+ throw std::invalid_argument("moe_gemm_prefill_int_dpas: weight_dtype must be S8");+ }+ MoePrefillParams params{q, activations, weights, scales, nullptr, outputs, nullptr, act_dtype, weight_dtype, N, K,+ 0, num_tokens_per_expert, num_experts, total_tokens, false};+ const int average_tokens = num_experts > 0 ? total_tokens / num_experts : 0;+ if (act_dtype == BTLA_DTYPE::F16) {+ if (average_tokens <= 8) moe_prefill_int_tensor_detail::dispatch_f16_small(params);+ else if (average_tokens <= 32) moe_prefill_int_tensor_detail::dispatch_f16_mid(params);+ else moe_prefill_int_tensor_detail::dispatch_f16_large(params);+ return;+ }+ if (act_dtype == BTLA_DTYPE::BF16) {+ if (average_tokens <= 8) moe_prefill_int_tensor_detail::dispatch_bf16_small(params);+ else if (average_tokens <= 32) moe_prefill_int_tensor_detail::dispatch_bf16_mid(params);+ else moe_prefill_int_tensor_detail::dispatch_bf16_large(params);+ return;+ }+ throw std::invalid_argument("moe_gemm_prefill_int_dpas: act_dtype must be F16 or BF16");+}++} // namespace ark++#elif @MOE_SOURCE_MODE@ == 16++#include "sycl_tla_common.hpp"+#include "sycl_tla_moe_prefill_s4_helpers.hpp"++namespace ark {++bool sycl_tla_moe_prefill_s4_dpas(const MoePrefillParams& params) {+ if (params.weight_dtype != BTLA_DTYPE::S4_CLIP || params.asym ||+ !moe_prefill_s4_detail::enabled() ||+ !moe_prefill_s4_detail::shape_ok(params.N, params.K, params.group_size) ||+ (params.act_dtype != BTLA_DTYPE::F16 && params.act_dtype != BTLA_DTYPE::BF16)) {+ return false;+ }+ if (params.act_dtype == BTLA_DTYPE::F16) {+ const int average_tokens = params.num_experts > 0 ? params.total_tokens / params.num_experts : 0;+ if (average_tokens <= 8) {+ moe_prefill_s4_detail::dispatch_f16_small(params);+ } else if (average_tokens <= 32) {+ moe_prefill_s4_detail::dispatch_f16_mid(params);+ } else {+ moe_prefill_s4_detail::dispatch_f16_large(params);+ }+ } else {+ const int average_tokens = params.num_experts > 0 ? params.total_tokens / params.num_experts : 0;+ if (average_tokens <= 8) {+ moe_prefill_s4_detail::dispatch_bf16_small(params);+ } else if (average_tokens <= 32) {+ moe_prefill_s4_detail::dispatch_bf16_mid(params);+ } else {+ moe_prefill_s4_detail::dispatch_bf16_large(params);+ }+ }+ return true;+}++} // namespace ark++#elif @MOE_SOURCE_MODE@ == 17++#include "sycl_tla_common.hpp"+#include "sycl_tla_moe_prefill_fp8_native.hpp"++namespace ark {++bool sycl_tla_moe_prefill_native_fp8(const MoePrefillParams& params) {+ if ((params.weight_dtype != BTLA_DTYPE::F8_E4M3 && params.weight_dtype != BTLA_DTYPE::F8_E5M2) || params.asym ||+ !moe_prefill_native_fp8_detail::moe_prefill_native_fp8_enabled() ||+ !moe_prefill_native_fp8_detail::moe_prefill_native_fp8_shape_ok(params.N, params.K, params.group_size)) {+ return false;+ }+ const bool is_e4m3 = params.weight_dtype == BTLA_DTYPE::F8_E4M3;+ const bool use_lut = moe_dequant::fp8_decode_use_lut();+ if (params.act_dtype == BTLA_DTYPE::F16) {+ using ScalarT = sycl::half;+ if (is_e4m3) {+ if (use_lut) {+ moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, true, true>(+ params.q, static_cast<const ScalarT*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const ScalarT*>(params.scales), static_cast<ScalarT*>(params.outputs),+ params.num_tokens_per_expert, params.num_experts, params.N, params.K, params.group_size,+ params.total_tokens);+ } else {+ moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, true, false>(+ params.q, static_cast<const ScalarT*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const ScalarT*>(params.scales), static_cast<ScalarT*>(params.outputs),+ params.num_tokens_per_expert, params.num_experts, params.N, params.K, params.group_size,+ params.total_tokens);+ }+ } else if (use_lut) {+ moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, false, true>(+ params.q, static_cast<const ScalarT*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const ScalarT*>(params.scales), static_cast<ScalarT*>(params.outputs),+ params.num_tokens_per_expert, params.num_experts, params.N, params.K, params.group_size,+ params.total_tokens);+ } else {+ moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, false, false>(+ params.q, static_cast<const ScalarT*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const ScalarT*>(params.scales), static_cast<ScalarT*>(params.outputs),+ params.num_tokens_per_expert, params.num_experts, params.N, params.K, params.group_size,+ params.total_tokens);+ }+ return true;+ }+ if (params.act_dtype == BTLA_DTYPE::BF16) {+ using ScalarT = sycl::ext::oneapi::bfloat16;+ if (is_e4m3) {+ if (use_lut) {+ moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, true, true>(+ params.q, static_cast<const ScalarT*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const ScalarT*>(params.scales), static_cast<ScalarT*>(params.outputs),+ params.num_tokens_per_expert, params.num_experts, params.N, params.K, params.group_size,+ params.total_tokens);+ } else {+ moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, true, false>(+ params.q, static_cast<const ScalarT*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const ScalarT*>(params.scales), static_cast<ScalarT*>(params.outputs),+ params.num_tokens_per_expert, params.num_experts, params.N, params.K, params.group_size,+ params.total_tokens);+ }+ } else if (use_lut) {+ moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, false, true>(+ params.q, static_cast<const ScalarT*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const ScalarT*>(params.scales), static_cast<ScalarT*>(params.outputs),+ params.num_tokens_per_expert, params.num_experts, params.N, params.K, params.group_size,+ params.total_tokens);+ } else {+ moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, false, false>(+ params.q, static_cast<const ScalarT*>(params.activations), static_cast<const uint8_t*>(params.weights),+ static_cast<const ScalarT*>(params.scales), static_cast<ScalarT*>(params.outputs),+ params.num_tokens_per_expert, params.num_experts, params.N, params.K, params.group_size,+ params.total_tokens);+ }+ return true;+ }+ return false;+}++} // namespace ark++#endif++#endif // ARK_XPU && ARK_SYCL_TLA\ No newline at end of filediff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_s8_wrapper.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_s8_wrapper.hppindex a77dcf2c7..31435a661 100644--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_s8_wrapper.hpp+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_s8_wrapper.hpp@@ -16,7 +16,7 @@#if ARK_XPU#if ARK_SYCL_TLA-#include "sycl_tla_s8_gemm.hpp"+#include "sycl_tla_common.hpp"#endifnamespace ark {diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_common.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_common.hppindex 824053d1b..390b5f890 100644--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_common.hpp+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_common.hpp@@ -34,13 +34,17 @@ namespace ark {*/void moe_gemm(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs, BTLA_DTYPE dtype, int N,int K, int* num_tokens_per_expert, int num_experts);+void sycl_tla_moe_gemm_f16(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs, int N, int K,+ int* num_tokens_per_expert, int num_experts);+void sycl_tla_moe_gemm_bf16(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs, int N, int K,+ int* num_tokens_per_expert, int num_experts);/*** @brief MoE GEMV optimized for the decode phase (M per expert is typically* 1-2 tokens). Supports unquantized FP16/BF16 weights and int4 (S4_CLIP)* weights with group-wise scales and optional zero-points.*- * Implementation is header-only in `sycl_tla_moe_decode.hpp`.+ * Heavy template implementations are split across the decode family source files.** @param q SYCL queue* @param activations [total_tokens, K] in `act_dtype`@@ -69,6 +73,31 @@ void moe_gemm_decode(sycl::queue* q, void* activations, void* weights, void* scaint* expert_id_per_token_buf, BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,int group_size, int* num_tokens_per_expert, int num_experts, int total_tokens, bool asym);+struct MoeDecodeParams {+ sycl::queue* q;+ void* activations;+ void* weights;+ void* scales;+ void* zeros;+ void* outputs;+ int* expert_id_per_token;+ BTLA_DTYPE act_dtype;+ BTLA_DTYPE weight_dtype;+ int N;+ int K;+ int group_size;+ int total_tokens;+ bool asym;+};++void sycl_tla_moe_decode_fill_expert_id(sycl::queue* q, int* expert_id_per_token,+ const int* num_tokens_per_expert, int num_experts, int total_tokens);+void sycl_tla_moe_decode_fp(const MoeDecodeParams& params);+void sycl_tla_moe_decode_int4(const MoeDecodeParams& params);+void sycl_tla_moe_decode_int8(const MoeDecodeParams& params);+void sycl_tla_moe_decode_int2(const MoeDecodeParams& params);+void sycl_tla_moe_decode_fp8(const MoeDecodeParams& params);+/*** @brief MoE Grouped GEMM optimized for the prefill phase, supporting the* same set of weight encodings as `moe_gemm_decode` (FP16/BF16, INT8 sym/asym,@@ -80,7 +109,7 @@ void moe_gemm_decode(sycl::queue* q, void* activations, void* weights, void* sca* existing `moe_gemm` baseline. This guarantees numerical parity with the* decode path. Mainloop fusion is the follow-up perf-tuning step.*- * Implementation is header-only in `sycl_tla_moe_mixed.hpp`.+ * Heavy template implementations are split across the prefill family source files.** Layout convention (matches `moe_gemm_decode`):* - activations: [total_tokens, K] in act_dtype@@ -96,6 +125,33 @@ void moe_gemm_prefill(sycl::queue* q, void* activations, void* weights, void* scvoid* dequant_workspace, BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,int group_size, int* num_tokens_per_expert, int num_experts, int total_tokens, bool asym);+struct MoePrefillParams {+ sycl::queue* q;+ void* activations;+ void* weights;+ void* scales;+ void* zeros;+ void* outputs;+ void* dequant_workspace;+ BTLA_DTYPE act_dtype;+ BTLA_DTYPE weight_dtype;+ int N;+ int K;+ int group_size;+ int* num_tokens_per_expert;+ int num_experts;+ int total_tokens;+ bool asym;+};++bool sycl_tla_moe_prefill_int8_dpas(const MoePrefillParams& params);+bool sycl_tla_moe_prefill_s4_dpas(const MoePrefillParams& params);+bool sycl_tla_moe_prefill_lowbit_int8_dpas(const MoePrefillParams& params);+bool sycl_tla_moe_prefill_fp8_dpas(const MoePrefillParams& params);+bool sycl_tla_moe_prefill_native_fp8(const MoePrefillParams& params);+void sycl_tla_moe_prefill_dequant_f16(const MoePrefillParams& params);+void sycl_tla_moe_prefill_dequant_bf16(const MoePrefillParams& params);+/*** @brief MoE prefill Grouped GEMM -- FP8 per-tensor mixed-input DPAS* (Variant A of the vllm-xpu-kernels FP8 port).@@ -109,7 +165,7 @@ void moe_gemm_prefill(sycl::queue* q, void* activations, void* weights, void* sc* `sycl_tla_moe_prefill_fp8_dpas.hpp` for the port's provenance & the* on-hardware TODOs.*- * Implementation is header-only in `sycl_tla_moe_prefill_fp8_dpas.hpp`.+ * The public entry point is implemented in `sycl_tla_moe_prefill_fp8_tensor.cpp`.*/void moe_gemm_prefill_fp8_dpas(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs,BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,@@ -133,12 +189,31 @@ void moe_gemm_prefill_fp8_dpas(sycl::queue* q, void* activations, void* weights,* `sycl_tla_moe_prefill_int_dpas.hpp` for the port's provenance & the* on-hardware TODOs.*- * Implementation is header-only in `sycl_tla_moe_prefill_int_dpas.hpp`.+ * The public entry point is implemented in `sycl_tla_moe_prefill_int_tensor.cpp`.*/void moe_gemm_prefill_int_dpas(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs,BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,int* num_tokens_per_expert, int num_experts, int total_tokens);+void sycl_tla_dense_gemm(sycl::queue* q, int m, int n, int k, const void* a, BTLA_DTYPE at, const void* b,+ BTLA_DTYPE bt, void* c, BTLA_DTYPE ct, const void* bias, bool BT);+void sycl_tla_dense_gemm_f32(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,+ const void* bias);+void sycl_tla_dense_gemm_f16(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,+ const void* bias);+void sycl_tla_dense_gemm_bf16(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,+ const void* bias);++void sycl_tla_igemm_s8s8_dequant(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,+ BTLA_DTYPE ct, const void* scale_a, const void* scale_b, const void* bias,+ int blocksize);+void sycl_tla_igemm_s8s8_dequant_f32(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,+ const void* scale_a, const void* scale_b, const void* bias, int blocksize);+void sycl_tla_igemm_s8s8_dequant_f16(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,+ const void* scale_a, const void* scale_b, const void* bias, int blocksize);+void sycl_tla_igemm_s8s8_dequant_bf16(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,+ const void* scale_a, const void* scale_b, const void* bias, int blocksize);+// ========================================================================// Public API// ========================================================================diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm.hppindex 8db797453..70450879d 100644--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm.hpp+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm.hpp@@ -30,6 +30,7 @@#endif#include "utils.hpp"+#include "sycl_tla_common.hpp"namespace ark {@@ -210,6 +211,21 @@ void gemm_cute_store_tile(sycl::queue* q, ATensor const& A, BTensor const& B, CT}+using DenseSmallTileSG = Layout<Shape<_1, _4, _1>, Stride<_0, _1, _0>>;+using DenseSmallMidTileSG = Layout<Shape<_2, _4, _1>, Stride<_4, _1, _0>>;+using DenseMediumTileSG = Layout<Shape<_4, _4, _1>, Stride<_4, _1, _0>>;+using DenseLargeTileSG = Layout<Shape<_8, _4, _1>, Stride<_4, _1, _0>>;++template <bool HasBias, int TileM, int TileN, class SGLayout, typename Element>+void run_dense_gemm_fixed_tile(sycl::queue* q, int m, int n, int k, const Element* a_ptr, const Element* b_ptr,+ Element* c_ptr, const Element* bias_ptr) {+ auto A = make_tensor(make_gmem_ptr(const_cast<Element*>(a_ptr)), make_shape(m, k), make_stride(k, _1{}));+ auto B = make_tensor(make_gmem_ptr(const_cast<Element*>(b_ptr)), make_shape(n, k), make_stride(k, _1{}));+ auto C = make_tensor(make_gmem_ptr(c_ptr), make_shape(m, n), make_stride(n, _1{}));+ gemm_cute_store_tile<HasBias, TileM, TileN, SGLayout, decltype(A), decltype(B), decltype(C), Element, Element, Element,+ 'R', 'R'>(q, A, B, C, bias_ptr);+}+template <bool HasBias, typename Element>void run_dense_gemm_tuned_tile(sycl::queue* q, int m, int n, int k, const Element* a_ptr, const Element* b_ptr,@@ -250,52 +266,6 @@ void run_dense_gemm(sycl::queue* q, int m, int n, int k, const Element* a_ptr, c} // namespace dense_gemm_detail-inline void sycl_tla_dense_gemm(sycl::queue* q, int m, int n, int k, const void* a, BTLA_DTYPE at,- const void* b, BTLA_DTYPE bt, void* c, BTLA_DTYPE ct, const void* bias,- bool BT) {- if (!q) {- throw std::invalid_argument("sycl_tla_dense_gemm: stream must be a valid SYCL queue");- }- if (!BT) {- throw std::invalid_argument("sycl_tla_dense_gemm: only the A @ B.T contract is supported");- }- if (!a || !b || !c) {- throw std::invalid_argument("sycl_tla_dense_gemm: input and output pointers must not be null");- }- if (m <= 0 || n <= 0 || k <= 0) {- return;- }- if (at != bt) {- throw std::invalid_argument("sycl_tla_dense_gemm: A and B must use the same dtype");- }- if (ct != at) {- throw std::invalid_argument("sycl_tla_dense_gemm: output dtype must match input dtype");- }-- switch (at) {- case BTLA_DTYPE::F32:- dense_gemm_detail::run_dense_gemm(- q, m, n, k, static_cast<const float*>(a), static_cast<const float*>(b),- static_cast<float*>(c), static_cast<const float*>(bias));- return;-- case BTLA_DTYPE::F16:- dense_gemm_detail::run_dense_gemm(- q, m, n, k, static_cast<const cute::half_t*>(a), static_cast<const cute::half_t*>(b),- static_cast<cute::half_t*>(c), static_cast<const cute::half_t*>(bias));- return;-- case BTLA_DTYPE::BF16:- dense_gemm_detail::run_dense_gemm(- q, m, n, k, static_cast<const cute::bfloat16_t*>(a), static_cast<const cute::bfloat16_t*>(b),- static_cast<cute::bfloat16_t*>(c), static_cast<const cute::bfloat16_t*>(bias));- return;-- default:- throw std::invalid_argument("sycl_tla_dense_gemm: only FP32, FP16 and BF16 are supported");- }-}-#endif // ARK_XPU && ARK_SYCL_TLA} // namespace ark\ No newline at end of filediff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm_helpers.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm_helpers.hppnew file mode 100644index 000000000..086a284d0--- /dev/null+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_dense_gemm_helpers.hpp@@ -0,0 +1,33 @@+#pragma once++#include "sycl_tla_common.hpp"++#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)++namespace ark {+namespace dense_gemm_detail {++#define ARK_DECLARE_DENSE_GEMM_ROUTES(prefix) \+ void prefix##_with_bias_small(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \+ const void* bias); \+ void prefix##_with_bias_mid(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \+ const void* bias); \+ void prefix##_with_bias_medium(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \+ const void* bias); \+ void prefix##_with_bias_large(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \+ const void* bias); \+ void prefix##_without_bias_small(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c); \+ void prefix##_without_bias_mid(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c); \+ void prefix##_without_bias_medium(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c); \+ void prefix##_without_bias_large(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c)++ARK_DECLARE_DENSE_GEMM_ROUTES(run_f16);+ARK_DECLARE_DENSE_GEMM_ROUTES(run_bf16);+ARK_DECLARE_DENSE_GEMM_ROUTES(run_f32);++#undef ARK_DECLARE_DENSE_GEMM_ROUTES++} // namespace dense_gemm_detail+} // namespace ark++#endif\ No newline at end of filediff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe.hppindex cff5626de..d2a0b0ae5 100644--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe.hpp+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe.hpp@@ -123,31 +123,6 @@ void moe_gemm_launcher(sycl::queue* q, const ElementA* activations, const Elemen} // namespace moe_detail-// Public MOE GEMM API-inline void moe_gemm(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs, BTLA_DTYPE dtype,- int N, int K, int* num_tokens_per_expert, int num_experts) {- switch (dtype) {- case BTLA_DTYPE::BF16: {- using Element = cutlass::bfloat16_t;- moe_detail::moe_gemm_launcher<'R', 'R', Element, Element, Element, Element>(- q, static_cast<const Element*>(activations), static_cast<const Element*>(weights),- static_cast<const Element*>(scales), static_cast<Element*>(outputs), N, K, num_tokens_per_expert,- num_experts);- break;- }- case BTLA_DTYPE::F16: {- using Element = cutlass::half_t;- moe_detail::moe_gemm_launcher<'R', 'R', Element, Element, Element, Element>(- q, static_cast<const Element*>(activations), static_cast<const Element*>(weights),- static_cast<const Element*>(scales), static_cast<Element*>(outputs), N, K, num_tokens_per_expert,- num_experts);- break;- }- default:- throw std::runtime_error("moe_gemm: unsupported dtype, only BF16/FP16 supported");- }-}-#endif // ARK_SYCL_TLA} // namespace arkdiff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_decode.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_decode.hppindex 1b8a4feb0..4a79c5034 100644--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_decode.hpp+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_decode.hpp@@ -39,6 +39,7 @@#include "bestla/bestla.h"#include "sycl_tla_moe_dequant.hpp"+#include "sycl_tla_common.hpp"#ifdef ARK_XPU#include <sycl/sycl.hpp>@@ -637,196 +638,6 @@ void launch_fp8(sycl::queue* q, const ScalarT* activations, const uint8_t* weigh// group-wise scales, no zero-points// act_dtype: F16 or BF16 (must match scales/outputs dtype)// -----------------------------------------------------------------------------inline void moe_gemm_decode(sycl::queue* q, void* activations, void* weights, void* scales, void* zeros,- void* outputs, int* expert_id_per_token_buf, BTLA_DTYPE act_dtype,- BTLA_DTYPE weight_dtype, int N, int K, int group_size, int* num_tokens_per_expert,- int num_experts, int total_tokens, bool asym) {- moe_decode_detail::fill_expert_id_per_token(q, expert_id_per_token_buf, num_tokens_per_expert, num_experts,- total_tokens);-- if (weight_dtype == BTLA_DTYPE::F16 || weight_dtype == BTLA_DTYPE::BF16) {- if (weight_dtype != act_dtype) {- throw std::invalid_argument("moe_gemm_decode: unquantized weight_dtype must match act_dtype");- }- if (act_dtype == BTLA_DTYPE::F16) {- moe_decode_detail::launch_fp<sycl::half>(q, static_cast<const sycl::half*>(activations),- static_cast<const sycl::half*>(weights),- static_cast<sycl::half*>(outputs), expert_id_per_token_buf,- total_tokens, N, K);- } else {- moe_decode_detail::launch_fp<sycl::ext::oneapi::bfloat16>(- q, static_cast<const sycl::ext::oneapi::bfloat16*>(activations),- static_cast<const sycl::ext::oneapi::bfloat16*>(weights),- static_cast<sycl::ext::oneapi::bfloat16*>(outputs), expert_id_per_token_buf, total_tokens, N, K);- }- return;- }-- if (weight_dtype == BTLA_DTYPE::S4_CLIP) {- if (act_dtype == BTLA_DTYPE::F16) {- if (asym) {- moe_decode_detail::launch_int4<sycl::half, true>(- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const sycl::half*>(scales), static_cast<const sycl::half*>(zeros),- static_cast<sycl::half*>(outputs), expert_id_per_token_buf, total_tokens, N, K, group_size);- } else {- moe_decode_detail::launch_int4<sycl::half, false>(- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const sycl::half*>(scales), static_cast<const sycl::half*>(zeros),- static_cast<sycl::half*>(outputs), expert_id_per_token_buf, total_tokens, N, K, group_size);- }- } else if (act_dtype == BTLA_DTYPE::BF16) {- using BF = sycl::ext::oneapi::bfloat16;- if (asym) {- moe_decode_detail::launch_int4<BF, true>(- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const BF*>(scales), static_cast<const BF*>(zeros), static_cast<BF*>(outputs),- expert_id_per_token_buf, total_tokens, N, K, group_size);- } else {- moe_decode_detail::launch_int4<BF, false>(- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const BF*>(scales), static_cast<const BF*>(zeros), static_cast<BF*>(outputs),- expert_id_per_token_buf, total_tokens, N, K, group_size);- }- } else {- throw std::invalid_argument("moe_gemm_decode(int4): act_dtype must be FP16 or BF16");- }- return;- }-- if (weight_dtype == BTLA_DTYPE::S8) {- if (act_dtype == BTLA_DTYPE::F16) {- if (asym) {- moe_decode_detail::launch_int8<sycl::half, true>(- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const sycl::half*>(scales), static_cast<const sycl::half*>(zeros),- static_cast<sycl::half*>(outputs), expert_id_per_token_buf, total_tokens, N, K, group_size);- } else {- moe_decode_detail::launch_int8<sycl::half, false>(- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const sycl::half*>(scales), static_cast<const sycl::half*>(zeros),- static_cast<sycl::half*>(outputs), expert_id_per_token_buf, total_tokens, N, K, group_size);- }- } else if (act_dtype == BTLA_DTYPE::BF16) {- using BF = sycl::ext::oneapi::bfloat16;- if (asym) {- moe_decode_detail::launch_int8<BF, true>(- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const BF*>(scales), static_cast<const BF*>(zeros), static_cast<BF*>(outputs),- expert_id_per_token_buf, total_tokens, N, K, group_size);- } else {- moe_decode_detail::launch_int8<BF, false>(- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const BF*>(scales), static_cast<const BF*>(zeros), static_cast<BF*>(outputs),- expert_id_per_token_buf, total_tokens, N, K, group_size);- }- } else {- throw std::invalid_argument("moe_gemm_decode(int8): act_dtype must be FP16 or BF16");- }- return;- }-- if (weight_dtype == BTLA_DTYPE::S2_CLIP) {- if (act_dtype == BTLA_DTYPE::F16) {- if (asym) {- moe_decode_detail::launch_int2<sycl::half, true>(- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const sycl::half*>(scales), static_cast<const sycl::half*>(zeros),- static_cast<sycl::half*>(outputs), expert_id_per_token_buf, total_tokens, N, K, group_size);- } else {- moe_decode_detail::launch_int2<sycl::half, false>(- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const sycl::half*>(scales), static_cast<const sycl::half*>(zeros),- static_cast<sycl::half*>(outputs), expert_id_per_token_buf, total_tokens, N, K, group_size);- }- } else if (act_dtype == BTLA_DTYPE::BF16) {- using BF = sycl::ext::oneapi::bfloat16;- if (asym) {- moe_decode_detail::launch_int2<BF, true>(- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const BF*>(scales), static_cast<const BF*>(zeros), static_cast<BF*>(outputs),- expert_id_per_token_buf, total_tokens, N, K, group_size);- } else {- moe_decode_detail::launch_int2<BF, false>(- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const BF*>(scales), static_cast<const BF*>(zeros), static_cast<BF*>(outputs),- expert_id_per_token_buf, total_tokens, N, K, group_size);- }- } else {- throw std::invalid_argument("moe_gemm_decode(int2): act_dtype must be FP16 or BF16");- }- return;- }-- if (weight_dtype == BTLA_DTYPE::F8_E4M3 || weight_dtype == BTLA_DTYPE::F8_E5M2) {- if (asym) {- throw std::invalid_argument("moe_gemm_decode(fp8): asym mode is not supported");- }- const bool is_e4m3 = (weight_dtype == BTLA_DTYPE::F8_E4M3);- const bool use_lut = moe_decode_detail::fp8_decode_use_lut();- if (act_dtype == BTLA_DTYPE::F16) {- if (is_e4m3) {- if (use_lut) {- moe_decode_detail::launch_fp8<sycl::half, true, true>(- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const sycl::half*>(scales), static_cast<sycl::half*>(outputs), expert_id_per_token_buf,- total_tokens, N, K, group_size);- } else {- moe_decode_detail::launch_fp8<sycl::half, true, false>(- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const sycl::half*>(scales), static_cast<sycl::half*>(outputs), expert_id_per_token_buf,- total_tokens, N, K, group_size);- }- } else {- if (use_lut) {- moe_decode_detail::launch_fp8<sycl::half, false, true>(- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const sycl::half*>(scales), static_cast<sycl::half*>(outputs), expert_id_per_token_buf,- total_tokens, N, K, group_size);- } else {- moe_decode_detail::launch_fp8<sycl::half, false, false>(- q, static_cast<const sycl::half*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const sycl::half*>(scales), static_cast<sycl::half*>(outputs), expert_id_per_token_buf,- total_tokens, N, K, group_size);- }- }- } else if (act_dtype == BTLA_DTYPE::BF16) {- using BF = sycl::ext::oneapi::bfloat16;- if (is_e4m3) {- if (use_lut) {- moe_decode_detail::launch_fp8<BF, true, true>(- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const BF*>(scales), static_cast<BF*>(outputs), expert_id_per_token_buf, total_tokens, N, K,- group_size);- } else {- moe_decode_detail::launch_fp8<BF, true, false>(- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const BF*>(scales), static_cast<BF*>(outputs), expert_id_per_token_buf, total_tokens, N, K,- group_size);- }- } else {- if (use_lut) {- moe_decode_detail::launch_fp8<BF, false, true>(- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const BF*>(scales), static_cast<BF*>(outputs), expert_id_per_token_buf, total_tokens, N, K,- group_size);- } else {- moe_decode_detail::launch_fp8<BF, false, false>(- q, static_cast<const BF*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const BF*>(scales), static_cast<BF*>(outputs), expert_id_per_token_buf, total_tokens, N, K,- group_size);- }- }- } else {- throw std::invalid_argument("moe_gemm_decode(fp8): act_dtype must be FP16 or BF16");- }- return;- }-- throw std::invalid_argument(- "moe_gemm_decode: unsupported weight_dtype (supported: F16, BF16, S8, S4_CLIP, S2_CLIP, F8_E4M3, F8_E5M2)");-}-} // namespace ark#endif // ARK_XPU && ARK_SYCL_TLAdiff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_mixed.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_mixed.hppindex e6a3ececc..51d218a13 100644--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_mixed.hpp+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_mixed.hpp@@ -40,13 +40,10 @@#include <stdexcept>#include "bestla/bestla.h"-#include "sycl_tla_moe.hpp"#include "sycl_tla_moe_dequant.hpp"-#include "sycl_tla_moe_prefill_fp8_dpas.hpp"-#include "sycl_tla_moe_prefill_int_dpas.hpp"-#include "sycl_tla_moe_prefill_s4_dpas.hpp"-#include "sycl_tla_moe_prefill_fp8_native.hpp"#include "sycl_tla_moe_prefill_fused.hpp"+#include "sycl_tla_moe_prefill_upcast.hpp"+#include "sycl_tla_common.hpp"#ifdef ARK_XPU#include <sycl/sycl.hpp>@@ -88,21 +85,6 @@ class MoEDequantKernelInt2Fast;template <typename ScalarT, bool IsE4M3, bool UseLut>class MoEDequantKernelFP8;-// Kernel name tags for the low-bit-width -> int8 sym upcast helpers used-// by the INT DPAS grouped-GEMM path. Output layout is `[E, N, K]` int8-// row-major (auto-round decode convention) so the caller can hand the-// resulting buffer to `moe_prefill_int_dpas_per_group_dispatch` unchanged.-class MoEUpcastInt4SymToInt8Kernel;-class MoEUpcastInt2SymToInt8Kernel;--// PR-1 fast-path counterparts of the sym->int8 upcast kernels: one-// work-item processes 4 packed bytes (INT4) / 2 packed bytes (INT2) = 8-// consecutive int8 outputs. Same bit-level decode as the scalar path so-// the downstream `moe_prefill_int_dpas_per_group_dispatch` sees identical-// bytes.-class MoEUpcastInt4SymToInt8KernelFast;-class MoEUpcastInt2SymToInt8KernelFast;-// Tile sizes for the dequant kernels.//// Each work-group covers a (PACK_K x WG_N) tile in (k, n) and writes PACK_K@@ -131,8 +113,6 @@ class MoEUpcastInt2SymToInt8KernelFast;constexpr int WG_N = 32;constexpr int PACK_K_FP = 4;constexpr int PACK_K_INT8 = 4;-constexpr int PACK_K_INT4 = 2;-constexpr int PACK_K_INT2 = 4;constexpr int PACK_K_FP8 = 4;// PR-1 fast-path PACK_K: one work-item handles 4 packed bytes for INT4@@ -146,9 +126,6 @@ constexpr int PACK_K_FP8 = 4;// suite (group_size is 32/64/128 and K is a multiple of group_size), but// the kernels fall back to the byte-per-work-item path when either// constraint is violated so short-K unit tests keep working.-constexpr int PACK_K_INT4_FAST = 8;-constexpr int PACK_K_INT2_FAST = 8;-// ----------------------------------------------------------------------------// FP16 / BF16 weight reshape: in-place transpose [E, N, K] -> [E, K, N].// Implemented as a generic dequant pass with identity scale; used so the@@ -479,168 +456,6 @@ void launch_dequant_int2(sycl::queue* q, const uint8_t* weights_NKp, const Scala});}-// -----------------------------------------------------------------------------// INT4 sym -> INT8 sym upcast: [E, N, K/2] uint8 packed -> [E, N, K] int8.-//-// Sign-extends each 4-bit signed field to a full `int8_t` (range [-8, 7])-// preserving the auto-round `[E, N, K]` decode-style layout so the result-// can be fed directly to `moe_prefill_int_dpas_per_group_dispatch`, which-// treats its `int8_t` weight input as sym-packed values that the per-group-// scale (already `[E, N, K/group_size]` in act dtype) rescales at each-// group boundary. Algebraically:-// dequant(w_int4)[e, n, k] = int4_signed(w_int4[e, n, k]) * scale[e, n, g]-// = int8_signed(upcast[e, n, k]) * scale[e, n, g]-// so the same `scale` tensor packs unmodified into the DPAS mainloop.-// -----------------------------------------------------------------------------inline void launch_upcast_int4_sym_to_int8(sycl::queue* q, const uint8_t* weights_NKp, int8_t* weights_i8_NK,- int E, int N, int K,- const int* num_tokens_per_expert = nullptr) {- if (E == 0 || N == 0 || K == 0) return;- if ((K & 1) != 0) {- throw std::invalid_argument("moe_gemm_prefill(int4->int8 upcast): K must be even");- }- const int k_packed = K / 2;-- // PR-1 fast path: one work-item processes a 4-byte packed word = 8 int8- // outputs, matching the launch geometry of the int4 dequant fast path so- // the two kernels share cache-hit patterns for the packed weight buffer.- // Output writes here are contiguous along K (buffer layout `[E, N, K]`),- // so 8 consecutive int8 stores form an aligned 8-byte burst.- if ((K % PACK_K_INT4_FAST) == 0) {- const int k_words = K / PACK_K_INT4_FAST;- sycl::range<3> global_fast{static_cast<size_t>(E), static_cast<size_t>(k_words),- static_cast<size_t>((N + WG_N - 1) / WG_N) * WG_N};- sycl::range<3> local_fast{1, 1, static_cast<size_t>(WG_N)};-- q->parallel_for<MoEUpcastInt4SymToInt8KernelFast>(- sycl::nd_range<3>(global_fast, local_fast), [=](sycl::nd_item<3> it) {- const int e = static_cast<int>(it.get_global_id(0));- if (num_tokens_per_expert != nullptr && num_tokens_per_expert[e] == 0) return;- const int kw = static_cast<int>(it.get_global_id(1));- const int n = static_cast<int>(it.get_global_id(2));- if (n >= N) return;- const int k_base = kw * PACK_K_INT4_FAST;- const size_t row_kp_base = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * k_packed +- static_cast<size_t>(kw) * 4;- const uint32_t packed =- static_cast<uint32_t>(weights_NKp[row_kp_base + 0]) |- (static_cast<uint32_t>(weights_NKp[row_kp_base + 1]) << 8) |- (static_cast<uint32_t>(weights_NKp[row_kp_base + 2]) << 16) |- (static_cast<uint32_t>(weights_NKp[row_kp_base + 3]) << 24);- int qv[8];- moe_dequant::decode_int4_octet</*Asym=*/false>(packed, qv);- const size_t out_row = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * K;-#pragma unroll- for (int j = 0; j < PACK_K_INT4_FAST; ++j) {- weights_i8_NK[out_row + static_cast<size_t>(k_base + j)] = static_cast<int8_t>(qv[j]);- }- });- return;- }-- // Fallback byte-per-work-item path (short-K configurations).- // One work-item per packed byte writes PACK_K_INT4 (=2) consecutive int8- // outputs for a single (e, n). Matches the launch geometry of- // `launch_dequant_int4` so the two paths share cache-hit patterns for- // the packed-weight buffer.- sycl::range<3> global{static_cast<size_t>(E), static_cast<size_t>(k_packed),- static_cast<size_t>((N + WG_N - 1) / WG_N) * WG_N};- sycl::range<3> local{1, 1, static_cast<size_t>(WG_N)};-- q->parallel_for<MoEUpcastInt4SymToInt8Kernel>(- sycl::nd_range<3>(global, local), [=](sycl::nd_item<3> it) {- const int e = static_cast<int>(it.get_global_id(0));- if (num_tokens_per_expert != nullptr && num_tokens_per_expert[e] == 0) return;- const int kp = static_cast<int>(it.get_global_id(1));- const int n = static_cast<int>(it.get_global_id(2));- if (n >= N) return;- const int k_base = kp * PACK_K_INT4;- const uint8_t packed =- weights_NKp[(static_cast<size_t>(e) * N + static_cast<size_t>(n)) * k_packed +- static_cast<size_t>(kp)];- int q_lo, q_hi;- moe_dequant::decode_int4_pair</*Asym=*/false>(packed, q_lo, q_hi);- const size_t out_row = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * K;- weights_i8_NK[out_row + static_cast<size_t>(k_base)] = static_cast<int8_t>(q_lo);- weights_i8_NK[out_row + static_cast<size_t>(k_base + 1)] = static_cast<int8_t>(q_hi);- });-}--// -----------------------------------------------------------------------------// INT2 sym -> INT8 sym upcast: [E, N, K/4] uint8 packed -> [E, N, K] int8.-//-// Sign-extends each 2-bit signed field to a full `int8_t` (range [-2, 1]).-// Same rationale as `launch_upcast_int4_sym_to_int8` -- the DPAS per-group-// mainloop consumes the resulting `[E, N, K]` int8 buffer with the packed-// int2 scale tensor unmodified.-// -----------------------------------------------------------------------------inline void launch_upcast_int2_sym_to_int8(sycl::queue* q, const uint8_t* weights_NKp, int8_t* weights_i8_NK,- int E, int N, int K,- const int* num_tokens_per_expert = nullptr) {- if (E == 0 || N == 0 || K == 0) return;- if ((K & 3) != 0) {- throw std::invalid_argument("moe_gemm_prefill(int2->int8 upcast): K must be a multiple of 4");- }- const int k_packed = K / 4;-- // PR-1 fast path: one work-item processes a 2-byte packed word = 8 int8- // outputs. Same rationale as the int4 upcast fast path above.- if ((K % PACK_K_INT2_FAST) == 0) {- const int k_words = K / PACK_K_INT2_FAST;- sycl::range<3> global_fast{static_cast<size_t>(E), static_cast<size_t>(k_words),- static_cast<size_t>((N + WG_N - 1) / WG_N) * WG_N};- sycl::range<3> local_fast{1, 1, static_cast<size_t>(WG_N)};-- q->parallel_for<MoEUpcastInt2SymToInt8KernelFast>(- sycl::nd_range<3>(global_fast, local_fast), [=](sycl::nd_item<3> it) {- const int e = static_cast<int>(it.get_global_id(0));- if (num_tokens_per_expert != nullptr && num_tokens_per_expert[e] == 0) return;- const int kw = static_cast<int>(it.get_global_id(1));- const int n = static_cast<int>(it.get_global_id(2));- if (n >= N) return;- const int k_base = kw * PACK_K_INT2_FAST;- const size_t row_kp_base = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * k_packed +- static_cast<size_t>(kw) * 2;- const uint16_t packed =- static_cast<uint16_t>(static_cast<uint32_t>(weights_NKp[row_kp_base + 0]) |- (static_cast<uint32_t>(weights_NKp[row_kp_base + 1]) << 8));- int qv[8];- moe_dequant::decode_int2_octet</*Asym=*/false>(packed, qv);- const size_t out_row = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * K;-#pragma unroll- for (int j = 0; j < PACK_K_INT2_FAST; ++j) {- weights_i8_NK[out_row + static_cast<size_t>(k_base + j)] = static_cast<int8_t>(qv[j]);- }- });- return;- }-- // Fallback byte-per-work-item path (short-K configurations).- sycl::range<3> global{static_cast<size_t>(E), static_cast<size_t>(k_packed),- static_cast<size_t>((N + WG_N - 1) / WG_N) * WG_N};- sycl::range<3> local{1, 1, static_cast<size_t>(WG_N)};-- q->parallel_for<MoEUpcastInt2SymToInt8Kernel>(- sycl::nd_range<3>(global, local), [=](sycl::nd_item<3> it) {- const int e = static_cast<int>(it.get_global_id(0));- if (num_tokens_per_expert != nullptr && num_tokens_per_expert[e] == 0) return;- const int kp = static_cast<int>(it.get_global_id(1));- const int n = static_cast<int>(it.get_global_id(2));- if (n >= N) return;- const int k_base = kp * PACK_K_INT2;- const uint8_t packed =- weights_NKp[(static_cast<size_t>(e) * N + static_cast<size_t>(n)) * k_packed +- static_cast<size_t>(kp)];- int qv[4];- moe_dequant::decode_int2_quad</*Asym=*/false>(packed, qv);- const size_t out_row = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * K;-#pragma unroll- for (int j = 0; j < PACK_K_INT2; ++j) {- weights_i8_NK[out_row + static_cast<size_t>(k_base + j)] = static_cast<int8_t>(qv[j]);- }- });-}-// ----------------------------------------------------------------------------// FP8 (E4M3 / E5M2) dequant: [E, N, K] uint8 -> [E, K, N] ScalarT.// Per-group scale applied; no zero-points (caller must enforce sym).@@ -811,290 +626,6 @@ void dequant_to_KN(sycl::queue* q, const void* weights, const void* scales, cons// under the same shape contract as `moe_gemm` and the quantized paths// under the same shape contract as `moe_gemm_decode`.// -----------------------------------------------------------------------------inline void moe_gemm_prefill(sycl::queue* q, void* activations, void* weights, void* scales, void* zeros,- void* outputs, void* dequant_workspace, BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype,- int N, int K, int group_size, int* num_tokens_per_expert, int num_experts,- int total_tokens, bool asym) {- if (total_tokens == 0) return;-- // Unquantized fast path: forward directly to the existing prefill GEMM.- // The caller is responsible for passing weights in `[E, K, N]` layout- // (matching the existing `moe_gemm` contract).- if (weight_dtype == act_dtype && (weight_dtype == BTLA_DTYPE::F16 || weight_dtype == BTLA_DTYPE::BF16)) {- moe_gemm(q, activations, weights, scales, outputs, act_dtype, N, K, num_tokens_per_expert, num_experts);- return;- }-- // Native FP8 fused GEMM (Variant A: fp8 weight x bf16/fp16 activation,- // in-register upcast, group-boundary scale fold, no `[E, K, N]` workspace).- // Opt-in via `ARK_MOE_PREFILL_NATIVE_FP8=1`; silent fallback if the shape- // doesn't satisfy the tile preconditions or if `asym=true` is requested.- // Precedence (highest first): dpas -> native -> fused-dequant -> v1 dequant.- // See `sycl_tla_moe_prefill_fp8_native.hpp` for the tile / SLM design.-- // INT8 mixed-input DPAS grouped GEMM (Variant B, sym only): per-K-group- // scale, in-register int8->act upcast, XMX MMA. Opt-in default via- // `ARK_MOE_PREFILL_DPAS_INT8` (default ON, matching the FP8 DPAS gate).- // Shape gate mirrors the FP8 per-group predicate exactly. Only `asym=false`- // is accelerated here -- the asym port used a per-M-row per-K-group- // activation-sum precompute and regressed below the dequant fallback on- // hardware, so asym S8 falls through to the dequant + `moe_gemm` path- // below.- //- // Scales are already the activation dtype (`[E, N, K/group_size]` half /- // bfloat16) in auto-round's INT8 per-group layout, so no conversion is- // needed relative to the dequant fallback -- the kernel just consumes the- // same scale tensor.- //- // STATUS: NEEDS-HARDWARE-VALIDATION. See- // `sycl_tla_moe_prefill_int_dpas.hpp` for the port's provenance & the- // on-hardware TODOs.- if (weight_dtype == BTLA_DTYPE::S8 && !asym &&- moe_dpas_int::moe_prefill_dpas_int_enabled() &&- moe_dpas_int::moe_prefill_dpas_int_pergroup_shape_ok(N, K, group_size)) {- const auto* weights_i8 = reinterpret_cast<const int8_t*>(weights);- if (act_dtype == BTLA_DTYPE::F16) {- using ScalarT = sycl::half;- moe_dpas_int::moe_prefill_int_dpas_per_group_dispatch<ScalarT>(- q, static_cast<const ScalarT*>(activations), weights_i8,- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs),- num_tokens_per_expert, num_experts, N, K, group_size, total_tokens);- return;- } else if (act_dtype == BTLA_DTYPE::BF16) {- using ScalarT = sycl::ext::oneapi::bfloat16;- moe_dpas_int::moe_prefill_int_dpas_per_group_dispatch<ScalarT>(- q, static_cast<const ScalarT*>(activations), weights_i8,- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs),- num_tokens_per_expert, num_experts, N, K, group_size, total_tokens);- return;- }- // Unsupported act_dtype falls through to the dequant branch below.- }-- // S4-sym single-pass DPAS grouped GEMM (Variant B: per-K-group scale,- // in-register nibble->act upcast, XMX MMA). This is the newer path that- // reads packed `[E, N, K/2]` uint8_t nibbles directly and folds the- // upcast into the DPAS mainloop via CuTe's `reorder(tBrB, tCrB)`, so- // the B-side global traffic is halved vs. the S4->S8 upcast branch- // below. Opt-in default via `ARK_MOE_PREFILL_DPAS_S4` (default ON);- // silent fallback to the S4->S8 upcast branch (which is itself gated- // by `ARK_MOE_PREFILL_DPAS_INT8`) or to the generic dequant path if- // the shape gate rejects the tile geometry.- //- // STATUS: NEEDS-HARDWARE-VALIDATION. See- // `sycl_tla_moe_prefill_s4_dpas.hpp` for the port's provenance & the- // on-hardware TODOs (chief among them: `NumericArrayConverter- // <ElementA, cutlass::int4b_t, N>` availability in the pinned- // cutlass-sycl).- if (weight_dtype == BTLA_DTYPE::S4_CLIP && !asym &&- moe_dpas_s4::moe_prefill_dpas_s4_enabled() &&- moe_dpas_s4::moe_prefill_dpas_s4_pergroup_shape_ok(N, K, group_size) &&- (act_dtype == BTLA_DTYPE::F16 || act_dtype == BTLA_DTYPE::BF16)) {- if (act_dtype == BTLA_DTYPE::F16) {- using ScalarT = sycl::half;- moe_dpas_s4::moe_prefill_s4_dpas_per_group_dispatch<ScalarT>(- q, static_cast<const ScalarT*>(activations),- static_cast<const uint8_t*>(weights),- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs),- num_tokens_per_expert, num_experts, N, K, group_size, total_tokens);- } else {- using ScalarT = sycl::ext::oneapi::bfloat16;- moe_dpas_s4::moe_prefill_s4_dpas_per_group_dispatch<ScalarT>(- q, static_cast<const ScalarT*>(activations),- static_cast<const uint8_t*>(weights),- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs),- num_tokens_per_expert, num_experts, N, K, group_size, total_tokens);- }- return;- }-- // INT4-sym / INT2-sym via INT8 DPAS. Rather than dequantise packed- // nibbles/crumbs into a bf16/fp16 `[E, K, N]` workspace and then run a- // bf16 x bf16 GEMM, we upcast the low-bit-width weights to `int8_t` in- // a `[E, N, K]` layout and hand the resulting buffer to the same- // per-group INT8 DPAS mainloop the S8 branch above uses. The upcast- // writes 1 byte per element (vs. 2 bytes for the bf16/fp16 dequant),- // and the DPAS mainloop then folds the per-K-group scale exactly the- // same way as the S8-sym path -- reusing the packed scale tensor- // unmodified. Silent fallback to the generic dequant path if the shape- // predicate rejects the tile geometry, if `asym=true`, or if the caller- // opted out via `ARK_MOE_PREFILL_DPAS_INT8=0`.- //- // For S4-sym specifically this branch is the *fallback* for the- // single-pass S4 DPAS path above -- callers who disable- // `ARK_MOE_PREFILL_DPAS_S4` land here instead of on the generic- // dequant path, so the two-pass INT4->INT8 pipeline stays available- // as a runtime kill-switch until the single-pass mainloop is- // hardware-validated.- //- // The dequant workspace pointer we reinterpret as `int8_t*` is the same- // caller-owned buffer used by the bf16/fp16 dequant fallback: since it- // is sized to `E * K * N * sizeof(act_dtype)` (>= 2 bytes/element) and- // the int8 view needs exactly `E * K * N` bytes, the reinterpretation is- // safe and does not require a separate allocation.- if ((weight_dtype == BTLA_DTYPE::S4_CLIP || weight_dtype == BTLA_DTYPE::S2_CLIP) && !asym &&- moe_dpas_int::moe_prefill_dpas_int_enabled() &&- moe_dpas_int::moe_prefill_dpas_int_pergroup_shape_ok(N, K, group_size) &&- dequant_workspace != nullptr &&- (act_dtype == BTLA_DTYPE::F16 || act_dtype == BTLA_DTYPE::BF16)) {- auto* upcast_i8 = static_cast<int8_t*>(dequant_workspace);- if (weight_dtype == BTLA_DTYPE::S4_CLIP) {- moe_mixed_detail::launch_upcast_int4_sym_to_int8(- q, static_cast<const uint8_t*>(weights), upcast_i8, num_experts, N, K,- num_tokens_per_expert);- } else {- moe_mixed_detail::launch_upcast_int2_sym_to_int8(- q, static_cast<const uint8_t*>(weights), upcast_i8, num_experts, N, K,- num_tokens_per_expert);- }- if (act_dtype == BTLA_DTYPE::F16) {- using ScalarT = sycl::half;- moe_dpas_int::moe_prefill_int_dpas_per_group_dispatch<ScalarT>(- q, static_cast<const ScalarT*>(activations), upcast_i8,- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs),- num_tokens_per_expert, num_experts, N, K, group_size, total_tokens);- } else {- using ScalarT = sycl::ext::oneapi::bfloat16;- moe_dpas_int::moe_prefill_int_dpas_per_group_dispatch<ScalarT>(- q, static_cast<const ScalarT*>(activations), upcast_i8,- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs),- num_tokens_per_expert, num_experts, N, K, group_size, total_tokens);- }- return;- }-- // FP8 mixed-input DPAS grouped GEMM (Variant B: per-K-group scale,- // in-register fp8->act upcast, XMX MMA). Opt-in default via- // `ARK_MOE_PREFILL_DPAS_FP8` (default ON); shape gate is stricter than- // the native path -- silent fallback to the native/dequant paths for any- // shape that doesn't satisfy the tile preconditions. Drop-in for- // existing auto-round FP8 checkpoints -- no re-calibration needed.- //- // STATUS: NEEDS-HARDWARE-VALIDATION. The port is untested; on-hardware- // parity against `test_moe_prefill_accuracy.py::test_accuracy_fp8` must- // be verified before this branch is trusted. See the header preamble- // in `sycl_tla_moe_prefill_fp8_dpas.hpp` for the open questions.- if ((weight_dtype == BTLA_DTYPE::F8_E4M3 || weight_dtype == BTLA_DTYPE::F8_E5M2) && !asym &&- moe_dpas_fp8::moe_prefill_dpas_fp8_enabled() &&- moe_dpas_fp8::moe_prefill_dpas_fp8_pergroup_shape_ok(N, K, group_size)) {- const bool is_e4m3 = (weight_dtype == BTLA_DTYPE::F8_E4M3);- if (act_dtype == BTLA_DTYPE::F16) {- using ScalarT = sycl::half;- if (is_e4m3) {- moe_dpas_fp8::moe_prefill_fp8_dpas_per_group_dispatch<ScalarT, true>(- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,- N, K, group_size, total_tokens);- } else {- moe_dpas_fp8::moe_prefill_fp8_dpas_per_group_dispatch<ScalarT, false>(- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,- N, K, group_size, total_tokens);- }- return;- } else if (act_dtype == BTLA_DTYPE::BF16) {- using ScalarT = sycl::ext::oneapi::bfloat16;- if (is_e4m3) {- moe_dpas_fp8::moe_prefill_fp8_dpas_per_group_dispatch<ScalarT, true>(- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,- N, K, group_size, total_tokens);- } else {- moe_dpas_fp8::moe_prefill_fp8_dpas_per_group_dispatch<ScalarT, false>(- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,- N, K, group_size, total_tokens);- }- return;- }- // Unsupported act_dtype falls through to the native / dequant branches.- }-- if ((weight_dtype == BTLA_DTYPE::F8_E4M3 || weight_dtype == BTLA_DTYPE::F8_E5M2) && !asym &&- moe_prefill_native_fp8_detail::moe_prefill_native_fp8_enabled() &&- moe_prefill_native_fp8_detail::moe_prefill_native_fp8_shape_ok(N, K, group_size)) {- const bool is_e4m3 = (weight_dtype == BTLA_DTYPE::F8_E4M3);- const bool use_lut = moe_dequant::fp8_decode_use_lut();- if (act_dtype == BTLA_DTYPE::F16) {- using ScalarT = sycl::half;- if (is_e4m3) {- if (use_lut) {- moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, true, true>(- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,- N, K, group_size, total_tokens);- } else {- moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, true, false>(- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,- N, K, group_size, total_tokens);- }- } else {- if (use_lut) {- moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, false, true>(- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,- N, K, group_size, total_tokens);- } else {- moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, false, false>(- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,- N, K, group_size, total_tokens);- }- }- return;- } else if (act_dtype == BTLA_DTYPE::BF16) {- using ScalarT = sycl::ext::oneapi::bfloat16;- if (is_e4m3) {- if (use_lut) {- moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, true, true>(- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,- N, K, group_size, total_tokens);- } else {- moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, true, false>(- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,- N, K, group_size, total_tokens);- }- } else {- if (use_lut) {- moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, false, true>(- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,- N, K, group_size, total_tokens);- } else {- moe_prefill_native_fp8_detail::moe_prefill_fp8_native_dispatch<ScalarT, false, false>(- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const ScalarT*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts,- N, K, group_size, total_tokens);- }- }- return;- }- // Unsupported act_dtype falls through to the generic dequant check below.- }-- if (dequant_workspace == nullptr) {- throw std::invalid_argument("moe_gemm_prefill: dequant_workspace must be non-null for quantized paths");- }-- if (act_dtype == BTLA_DTYPE::F16) {- auto* w_kn = static_cast<sycl::half*>(dequant_workspace);- moe_mixed_detail::dequant_to_KN<sycl::half>(q, weights, scales, zeros, w_kn, weight_dtype, num_experts, N, K,- group_size, asym, num_tokens_per_expert);- moe_gemm(q, activations, w_kn, /*scales=*/nullptr, outputs, act_dtype, N, K, num_tokens_per_expert, num_experts);- } else if (act_dtype == BTLA_DTYPE::BF16) {- using BF = sycl::ext::oneapi::bfloat16;- auto* w_kn = static_cast<BF*>(dequant_workspace);- moe_mixed_detail::dequant_to_KN<BF>(q, weights, scales, zeros, w_kn, weight_dtype, num_experts, N, K, group_size,- asym, num_tokens_per_expert);- moe_gemm(q, activations, w_kn, /*scales=*/nullptr, outputs, act_dtype, N, K, num_tokens_per_expert, num_experts);- } else {- throw std::invalid_argument("moe_gemm_prefill: act_dtype must be F16 or BF16");- }-}-// ----------------------------------------------------------------------------// MoE prefill Grouped GEMM -- FP8 per-tensor DPAS (Variant A).//@@ -1107,46 +638,6 @@ inline void moe_gemm_prefill(sycl::queue* q, void* activations, void* weights, v// `sycl_tla_moe_prefill_fp8_dpas.hpp` for the port's provenance & the// on-hardware TODOs.// -----------------------------------------------------------------------------inline void moe_gemm_prefill_fp8_dpas(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs,- BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,- int* num_tokens_per_expert, int num_experts, int total_tokens) {- if (total_tokens == 0) return;- if (weight_dtype != BTLA_DTYPE::F8_E4M3 && weight_dtype != BTLA_DTYPE::F8_E5M2) {- throw std::invalid_argument(- "moe_gemm_prefill_fp8_dpas: weight_dtype must be F8_E4M3 or F8_E5M2");- }- const bool is_e4m3 = (weight_dtype == BTLA_DTYPE::F8_E4M3);- if (act_dtype == BTLA_DTYPE::F16) {- using ScalarT = sycl::half;- if (is_e4m3) {- moe_dpas_fp8::moe_prefill_fp8_dpas_per_tensor_dispatch<ScalarT, true>(- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const float*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts, N,- K, total_tokens);- } else {- moe_dpas_fp8::moe_prefill_fp8_dpas_per_tensor_dispatch<ScalarT, false>(- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const float*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts, N,- K, total_tokens);- }- } else if (act_dtype == BTLA_DTYPE::BF16) {- using ScalarT = sycl::ext::oneapi::bfloat16;- if (is_e4m3) {- moe_dpas_fp8::moe_prefill_fp8_dpas_per_tensor_dispatch<ScalarT, true>(- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const float*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts, N,- K, total_tokens);- } else {- moe_dpas_fp8::moe_prefill_fp8_dpas_per_tensor_dispatch<ScalarT, false>(- q, static_cast<const ScalarT*>(activations), static_cast<const uint8_t*>(weights),- static_cast<const float*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts, N,- K, total_tokens);- }- } else {- throw std::invalid_argument("moe_gemm_prefill_fp8_dpas: act_dtype must be F16 or BF16");- }-}-// ----------------------------------------------------------------------------// MoE prefill Grouped GEMM -- INT8 per-tensor DPAS (Variant A).//@@ -1160,31 +651,6 @@ inline void moe_gemm_prefill_fp8_dpas(sycl::queue* q, void* activations, void* w// `sycl_tla_moe_prefill_int_dpas.hpp` for the port's provenance & the// on-hardware TODOs.// -----------------------------------------------------------------------------inline void moe_gemm_prefill_int_dpas(sycl::queue* q, void* activations, void* weights, void* scales, void* outputs,- BTLA_DTYPE act_dtype, BTLA_DTYPE weight_dtype, int N, int K,- int* num_tokens_per_expert, int num_experts, int total_tokens) {- if (total_tokens == 0) return;- if (weight_dtype != BTLA_DTYPE::S8) {- throw std::invalid_argument(- "moe_gemm_prefill_int_dpas: weight_dtype must be S8");- }- if (act_dtype == BTLA_DTYPE::F16) {- using ScalarT = sycl::half;- moe_dpas_int::moe_prefill_int_dpas_per_tensor_dispatch<ScalarT>(- q, static_cast<const ScalarT*>(activations), static_cast<const int8_t*>(weights),- static_cast<const float*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts, N,- K, total_tokens);- } else if (act_dtype == BTLA_DTYPE::BF16) {- using ScalarT = sycl::ext::oneapi::bfloat16;- moe_dpas_int::moe_prefill_int_dpas_per_tensor_dispatch<ScalarT>(- q, static_cast<const ScalarT*>(activations), static_cast<const int8_t*>(weights),- static_cast<const float*>(scales), static_cast<ScalarT*>(outputs), num_tokens_per_expert, num_experts, N,- K, total_tokens);- } else {- throw std::invalid_argument("moe_gemm_prefill_int_dpas: act_dtype must be F16 or BF16");- }-}-} // namespace ark#endif // ARK_XPU && ARK_SYCL_TLAdiff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_dpas.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_dpas.hppindex 1638bf7af..5946c9f1c 100644--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_dpas.hpp+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_dpas.hpp@@ -679,11 +679,12 @@ CUTE_DEVICE void xe_gemm_fp8_pergroup(template <typename T, char LayoutKind>CUTE_DEVICE auto make_moe_tensor(T* ptr, int r, int c) {auto shape = make_shape(r, c);+ auto gmem_ptr = make_gmem_ptr(ptr);if constexpr (LayoutKind == 'C')- return make_tensor(make_gmem_ptr(ptr),+ return make_tensor(gmem_ptr,make_layout(shape, make_stride(_1{}, r)));else- return make_tensor(make_gmem_ptr(ptr),+ return make_tensor(gmem_ptr,make_layout(shape, make_stride(c, _1{})));}@@ -910,6 +911,36 @@ void MoEGEMMLauncher(sycl::queue& stream, const ElementA* activations,// pointer `[E]` FP32.// ---------------------------------------------------------------------------+template <typename ScalarT, bool IsE4M3, class Policy>+void moe_prefill_fp8_dpas_per_tensor_dispatch_policy(+ sycl::queue* q, const ScalarT* activations, const uint8_t* weights_KN,+ const float* scales_e, ScalarT* outputs, const int* num_tokens_per_expert,+ int E, int N, int K, int total_tokens) {+ if (E == 0 || N == 0 || K == 0 || total_tokens == 0) return;++ compat::set_default_queue(*q);++ using ElementB = std::conditional_t<IsE4M3, cutlass::float_e4m3_t,+ cutlass::float_e5m2_t>;+ using ElementA = cute_scalar_t<ScalarT>;+ const auto* activations_ca =+ reinterpret_cast<const ElementA*>(activations);+ auto* outputs_ca = reinterpret_cast<ElementA*>(outputs);++ int32_t* atomic_buffer = sycl::malloc_device<int32_t>(1, *q);+ if (atomic_buffer == nullptr) {+ throw std::runtime_error(+ "moe_prefill_fp8_dpas(per-tensor): failed to allocate atomic buffer");+ }++ MoEGEMMLauncher<'R', 'R', Policy, ScaleMode::kPerTensor>(+ *q, activations_ca, reinterpret_cast<const ElementB*>(weights_KN),+ scales_e, static_cast<const ElementA*>(nullptr), outputs_ca, N, K,+ num_tokens_per_expert, E, /*group_size=*/0, atomic_buffer);++ sycl::free(atomic_buffer, *q);+}+template <typename ScalarT, bool IsE4M3>void moe_prefill_fp8_dpas_per_tensor_dispatch(sycl::queue* q, const ScalarT* activations, const uint8_t* weights_KN,@@ -963,6 +994,41 @@ void moe_prefill_fp8_dpas_per_tensor_dispatch(// Scales `[E, N, K/group_size]` in act dtype.// ---------------------------------------------------------------------------+template <typename ScalarT, bool IsE4M3, class Policy>+void moe_prefill_fp8_dpas_per_group_dispatch_policy(+ sycl::queue* q, const ScalarT* activations, const uint8_t* weights_NK,+ const ScalarT* scales, ScalarT* outputs, const int* num_tokens_per_expert,+ int E, int N, int K, int group_size, int total_tokens) {+ if (E == 0 || N == 0 || K == 0 || total_tokens == 0) return;+ if (K % group_size != 0) {+ throw std::invalid_argument(+ "moe_prefill_fp8_dpas(per-group): K must be a multiple of group_size");+ }++ compat::set_default_queue(*q);++ using ElementB = std::conditional_t<IsE4M3, cutlass::float_e4m3_t,+ cutlass::float_e5m2_t>;+ using ElementA = cute_scalar_t<ScalarT>;+ const auto* activations_ca =+ reinterpret_cast<const ElementA*>(activations);+ const auto* scales_ca = reinterpret_cast<const ElementA*>(scales);+ auto* outputs_ca = reinterpret_cast<ElementA*>(outputs);++ int32_t* atomic_buffer = sycl::malloc_device<int32_t>(1, *q);+ if (atomic_buffer == nullptr) {+ throw std::runtime_error(+ "moe_prefill_fp8_dpas(per-group): failed to allocate atomic buffer");+ }++ MoEGEMMLauncher<'R', 'C', Policy, ScaleMode::kPerGroup>(+ *q, activations_ca, reinterpret_cast<const ElementB*>(weights_NK),+ scales_ca, static_cast<const ElementA*>(nullptr), outputs_ca, N, K,+ num_tokens_per_expert, E, group_size, atomic_buffer);++ sycl::free(atomic_buffer, *q);+}+template <typename ScalarT, bool IsE4M3>void moe_prefill_fp8_dpas_per_group_dispatch(sycl::queue* q, const ScalarT* activations, const uint8_t* weights_NK,diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_helpers.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_helpers.hppnew file mode 100644index 000000000..eda62655c--- /dev/null+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_helpers.hpp@@ -0,0 +1,28 @@+#pragma once++#include "sycl_tla_common.hpp"++#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)++namespace ark {+namespace moe_prefill_fp8_detail {++bool enabled();+bool shape_ok(int N, int K, int group_size);+void dispatch_f16_e4m3_small(const MoePrefillParams& params);+void dispatch_f16_e4m3_mid(const MoePrefillParams& params);+void dispatch_f16_e4m3_large(const MoePrefillParams& params);+void dispatch_f16_e5m2_small(const MoePrefillParams& params);+void dispatch_f16_e5m2_mid(const MoePrefillParams& params);+void dispatch_f16_e5m2_large(const MoePrefillParams& params);+void dispatch_bf16_e4m3_small(const MoePrefillParams& params);+void dispatch_bf16_e4m3_mid(const MoePrefillParams& params);+void dispatch_bf16_e4m3_large(const MoePrefillParams& params);+void dispatch_bf16_e5m2_small(const MoePrefillParams& params);+void dispatch_bf16_e5m2_mid(const MoePrefillParams& params);+void dispatch_bf16_e5m2_large(const MoePrefillParams& params);++} // namespace moe_prefill_fp8_detail+} // namespace ark++#endif\ No newline at end of filediff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_tensor_helpers.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_tensor_helpers.hppnew file mode 100644index 000000000..2dd584970--- /dev/null+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_fp8_tensor_helpers.hpp@@ -0,0 +1,26 @@+#pragma once++#include "sycl_tla_common.hpp"++#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)++namespace ark {+namespace moe_prefill_fp8_tensor_detail {++void dispatch_f16_e4m3_small(const MoePrefillParams& params);+void dispatch_f16_e4m3_mid(const MoePrefillParams& params);+void dispatch_f16_e4m3_large(const MoePrefillParams& params);+void dispatch_f16_e5m2_small(const MoePrefillParams& params);+void dispatch_f16_e5m2_mid(const MoePrefillParams& params);+void dispatch_f16_e5m2_large(const MoePrefillParams& params);+void dispatch_bf16_e4m3_small(const MoePrefillParams& params);+void dispatch_bf16_e4m3_mid(const MoePrefillParams& params);+void dispatch_bf16_e4m3_large(const MoePrefillParams& params);+void dispatch_bf16_e5m2_small(const MoePrefillParams& params);+void dispatch_bf16_e5m2_mid(const MoePrefillParams& params);+void dispatch_bf16_e5m2_large(const MoePrefillParams& params);++} // namespace moe_prefill_fp8_tensor_detail+} // namespace ark++#endif\ No newline at end of filediff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int8_helpers.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int8_helpers.hppnew file mode 100644index 000000000..67d75928d--- /dev/null+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int8_helpers.hpp@@ -0,0 +1,28 @@+#pragma once++#include "sycl_tla_common.hpp"++#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)++namespace ark {+namespace moe_prefill_int8_detail {++bool enabled();+bool shape_ok(int N, int K, int group_size);+void dispatch_f16_small(const MoePrefillParams& params);+void dispatch_f16_mid(const MoePrefillParams& params);+void dispatch_f16_large(const MoePrefillParams& params);+void dispatch_bf16_small(const MoePrefillParams& params);+void dispatch_bf16_mid(const MoePrefillParams& params);+void dispatch_bf16_large(const MoePrefillParams& params);+void dispatch_lowbit_f16_small(const MoePrefillParams& params);+void dispatch_lowbit_f16_mid(const MoePrefillParams& params);+void dispatch_lowbit_f16_large(const MoePrefillParams& params);+void dispatch_lowbit_bf16_small(const MoePrefillParams& params);+void dispatch_lowbit_bf16_mid(const MoePrefillParams& params);+void dispatch_lowbit_bf16_large(const MoePrefillParams& params);++} // namespace moe_prefill_int8_detail+} // namespace ark++#endif\ No newline at end of filediff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_dpas.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_dpas.hppindex 2e05dfa44..d7f976011 100644--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_dpas.hpp+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_dpas.hpp@@ -761,6 +761,34 @@ void MoEGEMMLauncher_int(sycl::queue& stream, const ElementA* activations,// Scale pointer `[E]` FP32 (one per-expert scalar).// ---------------------------------------------------------------------------+template <typename ScalarT, class Policy>+void moe_prefill_int_dpas_per_tensor_dispatch_policy(+ sycl::queue* q, const ScalarT* activations, const int8_t* weights_KN,+ const float* scales_e, ScalarT* outputs, const int* num_tokens_per_expert,+ int E, int N, int K, int total_tokens) {+ if (E == 0 || N == 0 || K == 0 || total_tokens == 0) return;++ compat::set_default_queue(*q);++ using ElementA = cute_scalar_t<ScalarT>;+ const auto* activations_ca =+ reinterpret_cast<const ElementA*>(activations);+ auto* outputs_ca = reinterpret_cast<ElementA*>(outputs);++ int32_t* atomic_buffer = sycl::malloc_device<int32_t>(1, *q);+ if (atomic_buffer == nullptr) {+ throw std::runtime_error(+ "moe_prefill_int_dpas(per-tensor): failed to allocate atomic buffer");+ }++ MoEGEMMLauncher_int<'R', 'R', Policy, ScaleMode::kPerTensor>(+ *q, activations_ca, weights_KN, scales_e,+ static_cast<const ElementA*>(nullptr), outputs_ca, N, K,+ num_tokens_per_expert, E, /*group_size=*/0, atomic_buffer);++ sycl::free(atomic_buffer, *q);+}+template <typename ScalarT>void moe_prefill_int_dpas_per_tensor_dispatch(sycl::queue* q, const ScalarT* activations, const int8_t* weights_KN,@@ -819,6 +847,40 @@ void moe_prefill_int_dpas_per_tensor_dispatch(// as the FP8 per-group path.// ---------------------------------------------------------------------------+template <typename ScalarT, class Policy>+void moe_prefill_int_dpas_per_group_dispatch_policy(+ sycl::queue* q, const ScalarT* activations, const int8_t* weights_NK,+ const ScalarT* scales, ScalarT* outputs,+ const int* num_tokens_per_expert, int E, int N, int K, int group_size,+ int total_tokens) {+ if (E == 0 || N == 0 || K == 0 || total_tokens == 0) return;+ if (K % group_size != 0) {+ throw std::invalid_argument(+ "moe_prefill_int_dpas(per-group): K must be a multiple of group_size");+ }++ compat::set_default_queue(*q);++ using ElementA = cute_scalar_t<ScalarT>;+ const auto* activations_ca =+ reinterpret_cast<const ElementA*>(activations);+ const auto* scales_ca = reinterpret_cast<const ElementA*>(scales);+ auto* outputs_ca = reinterpret_cast<ElementA*>(outputs);++ int32_t* atomic_buffer = sycl::malloc_device<int32_t>(1, *q);+ if (atomic_buffer == nullptr) {+ throw std::runtime_error(+ "moe_prefill_int_dpas(per-group): failed to allocate atomic buffer");+ }++ MoEGEMMLauncher_int<'R', 'C', Policy, ScaleMode::kPerGroup>(+ *q, activations_ca, weights_NK, scales_ca,+ static_cast<const ElementA*>(nullptr), outputs_ca, N, K,+ num_tokens_per_expert, E, group_size, atomic_buffer);++ sycl::free(atomic_buffer, *q);+}+template <typename ScalarT>void moe_prefill_int_dpas_per_group_dispatch(sycl::queue* q, const ScalarT* activations, const int8_t* weights_NK,diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_tensor_helpers.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_tensor_helpers.hppnew file mode 100644index 000000000..8d0e2026a--- /dev/null+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_int_tensor_helpers.hpp@@ -0,0 +1,20 @@+#pragma once++#include "sycl_tla_common.hpp"++#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)++namespace ark {+namespace moe_prefill_int_tensor_detail {++void dispatch_f16_small(const MoePrefillParams& params);+void dispatch_f16_mid(const MoePrefillParams& params);+void dispatch_f16_large(const MoePrefillParams& params);+void dispatch_bf16_small(const MoePrefillParams& params);+void dispatch_bf16_mid(const MoePrefillParams& params);+void dispatch_bf16_large(const MoePrefillParams& params);++} // namespace moe_prefill_int_tensor_detail+} // namespace ark++#endif\ No newline at end of filediff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_dpas.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_dpas.hppindex fad8d87eb..0a2ea3d70 100644--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_dpas.hpp+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_dpas.hpp@@ -603,6 +603,46 @@ void MoEGEMMLauncher_s4(sycl::queue& stream, const ElementA* activations,// Scales `[E, N, K/group_size]` in act dtype (half / bfloat16).// ---------------------------------------------------------------------------+template <typename ScalarT, class Policy>+void moe_prefill_s4_dpas_per_group_dispatch_policy(+ sycl::queue* q, const ScalarT* activations, const uint8_t* weights_NKp,+ const ScalarT* scales, ScalarT* outputs,+ const int* num_tokens_per_expert, int E, int N, int K, int group_size,+ int total_tokens) {+ if (E == 0 || N == 0 || K == 0 || total_tokens == 0) return;+ if (K % group_size != 0) {+ throw std::invalid_argument(+ "moe_prefill_s4_dpas(per-group): K must be a multiple of group_size");+ }+ if ((K & 1) != 0) {+ throw std::invalid_argument(+ "moe_prefill_s4_dpas(per-group): K must be even (packed nibbles)");+ }++ compat::set_default_queue(*q);++ using ElementA = cute_scalar_t<ScalarT>;+ const auto* activations_ca =+ reinterpret_cast<const ElementA*>(activations);+ const auto* scales_ca = reinterpret_cast<const ElementA*>(scales);+ auto* outputs_ca = reinterpret_cast<ElementA*>(outputs);+ const auto* weights_i4 =+ reinterpret_cast<const cutlass::int4b_t*>(weights_NKp);++ int32_t* atomic_buffer = sycl::malloc_device<int32_t>(1, *q);+ if (atomic_buffer == nullptr) {+ throw std::runtime_error(+ "moe_prefill_s4_dpas(per-group): failed to allocate atomic buffer");+ }++ MoEGEMMLauncher_s4<'R', 'C', Policy>(+ *q, activations_ca, weights_i4, scales_ca,+ static_cast<const ElementA*>(nullptr), outputs_ca, N, K,+ num_tokens_per_expert, E, group_size, atomic_buffer);++ sycl::free(atomic_buffer, *q);+}+template <typename ScalarT>void moe_prefill_s4_dpas_per_group_dispatch(sycl::queue* q, const ScalarT* activations, const uint8_t* weights_NKp,diff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_helpers.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_helpers.hppnew file mode 100644index 000000000..6010579c1--- /dev/null+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_s4_helpers.hpp@@ -0,0 +1,22 @@+#pragma once++#include "sycl_tla_common.hpp"++#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)++namespace ark {+namespace moe_prefill_s4_detail {++bool enabled();+bool shape_ok(int N, int K, int group_size);+void dispatch_f16_small(const MoePrefillParams& params);+void dispatch_f16_mid(const MoePrefillParams& params);+void dispatch_f16_large(const MoePrefillParams& params);+void dispatch_bf16_small(const MoePrefillParams& params);+void dispatch_bf16_mid(const MoePrefillParams& params);+void dispatch_bf16_large(const MoePrefillParams& params);++} // namespace moe_prefill_s4_detail+} // namespace ark++#endif\ No newline at end of filediff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_upcast.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_upcast.hppnew file mode 100644index 000000000..1e2ff7bdb--- /dev/null+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_moe_prefill_upcast.hpp@@ -0,0 +1,159 @@+// SYCL MoE prefill low-bit to INT8 upcast kernels++#pragma once++#include <cstddef>+#include <cstdint>+#include <stdexcept>++#ifdef ARK_XPU+#include <sycl/sycl.hpp>+#endif++#include "sycl_tla_moe_dequant.hpp"++#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)++namespace ark {+namespace moe_mixed_detail {++class MoEUpcastInt4SymToInt8Kernel;+class MoEUpcastInt2SymToInt8Kernel;+class MoEUpcastInt4SymToInt8KernelFast;+class MoEUpcastInt2SymToInt8KernelFast;++constexpr int UPCAST_WG_N = 32;+constexpr int PACK_K_INT4 = 2;+constexpr int PACK_K_INT2 = 4;+constexpr int PACK_K_INT4_FAST = 8;+constexpr int PACK_K_INT2_FAST = 8;++inline void launch_upcast_int4_sym_to_int8(sycl::queue* q, const uint8_t* weights_NKp, int8_t* weights_i8_NK,+ int E, int N, int K,+ const int* num_tokens_per_expert = nullptr) {+ if (E == 0 || N == 0 || K == 0) return;+ if ((K & 1) != 0) {+ throw std::invalid_argument("moe_gemm_prefill(int4->int8 upcast): K must be even");+ }+ const int k_packed = K / 2;++ if ((K % PACK_K_INT4_FAST) == 0) {+ const int k_words = K / PACK_K_INT4_FAST;+ sycl::range<3> global_fast{static_cast<size_t>(E), static_cast<size_t>(k_words),+ static_cast<size_t>((N + UPCAST_WG_N - 1) / UPCAST_WG_N) * UPCAST_WG_N};+ sycl::range<3> local_fast{1, 1, static_cast<size_t>(UPCAST_WG_N)};++ q->parallel_for<MoEUpcastInt4SymToInt8KernelFast>(+ sycl::nd_range<3>(global_fast, local_fast), [=](sycl::nd_item<3> it) {+ const int e = static_cast<int>(it.get_global_id(0));+ if (num_tokens_per_expert != nullptr && num_tokens_per_expert[e] == 0) return;+ const int kw = static_cast<int>(it.get_global_id(1));+ const int n = static_cast<int>(it.get_global_id(2));+ if (n >= N) return;+ const int k_base = kw * PACK_K_INT4_FAST;+ const size_t row_kp_base = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * k_packed ++ static_cast<size_t>(kw) * 4;+ const uint32_t packed =+ static_cast<uint32_t>(weights_NKp[row_kp_base + 0]) |+ (static_cast<uint32_t>(weights_NKp[row_kp_base + 1]) << 8) |+ (static_cast<uint32_t>(weights_NKp[row_kp_base + 2]) << 16) |+ (static_cast<uint32_t>(weights_NKp[row_kp_base + 3]) << 24);+ int qv[8];+ moe_dequant::decode_int4_octet<false>(packed, qv);+ const size_t out_row = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * K;+#pragma unroll+ for (int j = 0; j < PACK_K_INT4_FAST; ++j) {+ weights_i8_NK[out_row + static_cast<size_t>(k_base + j)] = static_cast<int8_t>(qv[j]);+ }+ });+ return;+ }++ sycl::range<3> global{static_cast<size_t>(E), static_cast<size_t>(k_packed),+ static_cast<size_t>((N + UPCAST_WG_N - 1) / UPCAST_WG_N) * UPCAST_WG_N};+ sycl::range<3> local{1, 1, static_cast<size_t>(UPCAST_WG_N)};++ q->parallel_for<MoEUpcastInt4SymToInt8Kernel>(+ sycl::nd_range<3>(global, local), [=](sycl::nd_item<3> it) {+ const int e = static_cast<int>(it.get_global_id(0));+ if (num_tokens_per_expert != nullptr && num_tokens_per_expert[e] == 0) return;+ const int kp = static_cast<int>(it.get_global_id(1));+ const int n = static_cast<int>(it.get_global_id(2));+ if (n >= N) return;+ const int k_base = kp * PACK_K_INT4;+ const uint8_t packed =+ weights_NKp[(static_cast<size_t>(e) * N + static_cast<size_t>(n)) * k_packed + static_cast<size_t>(kp)];+ int q_lo, q_hi;+ moe_dequant::decode_int4_pair<false>(packed, q_lo, q_hi);+ const size_t out_row = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * K;+ weights_i8_NK[out_row + static_cast<size_t>(k_base)] = static_cast<int8_t>(q_lo);+ weights_i8_NK[out_row + static_cast<size_t>(k_base + 1)] = static_cast<int8_t>(q_hi);+ });+}++inline void launch_upcast_int2_sym_to_int8(sycl::queue* q, const uint8_t* weights_NKp, int8_t* weights_i8_NK,+ int E, int N, int K,+ const int* num_tokens_per_expert = nullptr) {+ if (E == 0 || N == 0 || K == 0) return;+ if ((K & 3) != 0) {+ throw std::invalid_argument("moe_gemm_prefill(int2->int8 upcast): K must be a multiple of 4");+ }+ const int k_packed = K / 4;++ if ((K % PACK_K_INT2_FAST) == 0) {+ const int k_words = K / PACK_K_INT2_FAST;+ sycl::range<3> global_fast{static_cast<size_t>(E), static_cast<size_t>(k_words),+ static_cast<size_t>((N + UPCAST_WG_N - 1) / UPCAST_WG_N) * UPCAST_WG_N};+ sycl::range<3> local_fast{1, 1, static_cast<size_t>(UPCAST_WG_N)};++ q->parallel_for<MoEUpcastInt2SymToInt8KernelFast>(+ sycl::nd_range<3>(global_fast, local_fast), [=](sycl::nd_item<3> it) {+ const int e = static_cast<int>(it.get_global_id(0));+ if (num_tokens_per_expert != nullptr && num_tokens_per_expert[e] == 0) return;+ const int kw = static_cast<int>(it.get_global_id(1));+ const int n = static_cast<int>(it.get_global_id(2));+ if (n >= N) return;+ const int k_base = kw * PACK_K_INT2_FAST;+ const size_t row_kp_base = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * k_packed ++ static_cast<size_t>(kw) * 2;+ const uint16_t packed = static_cast<uint16_t>(static_cast<uint32_t>(weights_NKp[row_kp_base + 0]) |+ (static_cast<uint32_t>(weights_NKp[row_kp_base + 1]) << 8));+ int qv[8];+ moe_dequant::decode_int2_octet<false>(packed, qv);+ const size_t out_row = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * K;+#pragma unroll+ for (int j = 0; j < PACK_K_INT2_FAST; ++j) {+ weights_i8_NK[out_row + static_cast<size_t>(k_base + j)] = static_cast<int8_t>(qv[j]);+ }+ });+ return;+ }++ sycl::range<3> global{static_cast<size_t>(E), static_cast<size_t>(k_packed),+ static_cast<size_t>((N + UPCAST_WG_N - 1) / UPCAST_WG_N) * UPCAST_WG_N};+ sycl::range<3> local{1, 1, static_cast<size_t>(UPCAST_WG_N)};++ q->parallel_for<MoEUpcastInt2SymToInt8Kernel>(+ sycl::nd_range<3>(global, local), [=](sycl::nd_item<3> it) {+ const int e = static_cast<int>(it.get_global_id(0));+ if (num_tokens_per_expert != nullptr && num_tokens_per_expert[e] == 0) return;+ const int kp = static_cast<int>(it.get_global_id(1));+ const int n = static_cast<int>(it.get_global_id(2));+ if (n >= N) return;+ const int k_base = kp * PACK_K_INT2;+ const uint8_t packed =+ weights_NKp[(static_cast<size_t>(e) * N + static_cast<size_t>(n)) * k_packed + static_cast<size_t>(kp)];+ int qv[4];+ moe_dequant::decode_int2_quad<false>(packed, qv);+ const size_t out_row = (static_cast<size_t>(e) * N + static_cast<size_t>(n)) * K;+#pragma unroll+ for (int j = 0; j < PACK_K_INT2; ++j) {+ weights_i8_NK[out_row + static_cast<size_t>(k_base + j)] = static_cast<int8_t>(qv[j]);+ }+ });+}++} // namespace moe_mixed_detail+} // namespace ark++#endif\ No newline at end of filediff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm.hppindex 5a6dec850..b339ccd23 100644--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm.hpp+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm.hpp@@ -19,6 +19,7 @@#endif#include "utils.hpp"+#include "sycl_tla_common.hpp"namespace ark {@@ -444,46 +445,24 @@ void run_typed(sycl::queue* q, int m, int n, int k, const int8_t* a, const int8_launch_igemm_kblock<ElementOut>(q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize, blks);}-} // namespace sycl_tla_s8_detail--inline void sycl_tla_igemm_s8s8_dequant(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c,- BTLA_DTYPE ct, const void* scale_a, const void* scale_b, const void* bias,- int blocksize) {- if (!q) throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: queue must not be null");- if (!a || !b || !c || !scale_a || !scale_b) {- throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: input pointers must not be null");- }- if (m <= 0 || n <= 0 || k <= 0) return;-- bool k_block = !(blocksize == k || blocksize == -1);+template <class ElementOut, int TileM, int TileN, class SGLayout>+void run_igemm_normal_fixed_tile(sycl::queue* q, int m, int n, int k, const int8_t* a, const int8_t* b,+ ElementOut* c, const ElementOut* scale_a, const ElementOut* scale_b,+ const ElementOut* bias) {+ launch_igemm_tile<false, ElementOut, TileM, TileN, SGLayout>(+ q, m, n, k, k, k, a, b, c, nullptr, scale_a, scale_b, bias, 0, 1);+}- switch (ct) {- case BTLA_DTYPE::F32:- sycl_tla_s8_detail::run_typed(q, m, n, k, static_cast<const int8_t*>(a), static_cast<const int8_t*>(b),- static_cast<float*>(c), static_cast<const float*>(scale_a),- static_cast<const float*>(scale_b), static_cast<const float*>(bias), blocksize);- return;- case BTLA_DTYPE::F16:- sycl_tla_s8_detail::run_typed(q, m, n, k, static_cast<const int8_t*>(a), static_cast<const int8_t*>(b),- static_cast<cute::half_t*>(c), static_cast<const cute::half_t*>(scale_a),- static_cast<const cute::half_t*>(scale_b), static_cast<const cute::half_t*>(bias),- blocksize);- return;- case BTLA_DTYPE::BF16:- if (k_block) {- throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: k-block path supports only F32/F16 output");- }- sycl_tla_s8_detail::run_typed(q, m, n, k, static_cast<const int8_t*>(a), static_cast<const int8_t*>(b),- static_cast<cute::bfloat16_t*>(c),- static_cast<const cute::bfloat16_t*>(scale_a),- static_cast<const cute::bfloat16_t*>(scale_b),- static_cast<const cute::bfloat16_t*>(bias), blocksize);- return;- default:- throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: unsupported output dtype");- }+template <class ElementOut, int TileM, int TileN, class SGLayout>+void run_igemm_kblock_fixed_tile(sycl::queue* q, int m, int n, int k, const int8_t* a, const int8_t* b,+ ElementOut* c, const ElementOut* scale_a, const ElementOut* scale_b,+ const ElementOut* bias, int blocksize) {+ launch_igemm_kblock_tile_dispatch<ElementOut, TileM, TileN, SGLayout>(+ q, m, n, k, a, b, c, scale_a, scale_b, bias, blocksize, k / blocksize);}+} // namespace sycl_tla_s8_detail+#endif // ARK_XPU && ARK_SYCL_TLA} // namespace ark\ No newline at end of filediff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm_fixed.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm_fixed.hppnew file mode 100644index 000000000..f30e58a75--- /dev/null+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm_fixed.hpp@@ -0,0 +1,32 @@+#pragma once++#include "sycl_tla_s8_gemm.hpp"++#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)++#define ARK_DEFINE_S8_NORMAL_ROUTE(name, element, tile_m, tile_n, layout) \+ namespace ark { \+ namespace sycl_tla_s8_detail { \+ void name(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, const void* scale_a, \+ const void* scale_b, const void* bias) { \+ run_igemm_normal_fixed_tile<element, tile_m, tile_n, layout>( \+ q, m, n, k, static_cast<const int8_t*>(a), static_cast<const int8_t*>(b), static_cast<element*>(c), \+ static_cast<const element*>(scale_a), static_cast<const element*>(scale_b), static_cast<const element*>(bias)); \+ } \+ } \+ }++#define ARK_DEFINE_S8_KBLOCK_ROUTE(name, element, tile_m, tile_n, layout) \+ namespace ark { \+ namespace sycl_tla_s8_detail { \+ void name(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, const void* scale_a, \+ const void* scale_b, const void* bias, int blocksize) { \+ run_igemm_kblock_fixed_tile<element, tile_m, tile_n, layout>( \+ q, m, n, k, static_cast<const int8_t*>(a), static_cast<const int8_t*>(b), static_cast<element*>(c), \+ static_cast<const element*>(scale_a), static_cast<const element*>(scale_b), static_cast<const element*>(bias), \+ blocksize); \+ } \+ } \+ }++#endif\ No newline at end of filediff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm_helpers.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm_helpers.hppnew file mode 100644index 000000000..580995406--- /dev/null+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_s8_gemm_helpers.hpp@@ -0,0 +1,53 @@+#pragma once++#include "sycl_tla_common.hpp"++#if defined(ARK_XPU) && defined(ARK_SYCL_TLA)++#include <stdexcept>++namespace ark {+namespace sycl_tla_s8_detail {++#define ARK_DECLARE_S8_NORMAL_ROUTES(prefix) \+ void prefix##_normal_small(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \+ const void* scale_a, const void* scale_b, const void* bias); \+ void prefix##_normal_mid(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \+ const void* scale_a, const void* scale_b, const void* bias); \+ void prefix##_normal_medium(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \+ const void* scale_a, const void* scale_b, const void* bias); \+ void prefix##_normal_large(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \+ const void* scale_a, const void* scale_b, const void* bias)++#define ARK_DECLARE_S8_KBLOCK_ROUTES(prefix) \+ void prefix##_kblock_small(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \+ const void* scale_a, const void* scale_b, const void* bias, int blocksize); \+ void prefix##_kblock_mid(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \+ const void* scale_a, const void* scale_b, const void* bias, int blocksize); \+ void prefix##_kblock_medium(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \+ const void* scale_a, const void* scale_b, const void* bias, int blocksize); \+ void prefix##_kblock_large(sycl::queue* q, int m, int n, int k, const void* a, const void* b, void* c, \+ const void* scale_a, const void* scale_b, const void* bias, int blocksize)++ARK_DECLARE_S8_NORMAL_ROUTES(run_f16);+ARK_DECLARE_S8_NORMAL_ROUTES(run_bf16);+ARK_DECLARE_S8_NORMAL_ROUTES(run_f32);+ARK_DECLARE_S8_KBLOCK_ROUTES(run_f16);+ARK_DECLARE_S8_KBLOCK_ROUTES(run_f32);++#undef ARK_DECLARE_S8_NORMAL_ROUTES+#undef ARK_DECLARE_S8_KBLOCK_ROUTES++inline void validate_kblock_args(int k, int blocksize) {+ if (blocksize <= 0 || k % blocksize != 0) {+ throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: blocksize must divide k");+ }+ if (blocksize % 64 != 0) {+ throw std::invalid_argument("sycl_tla_igemm_s8s8_dequant: k-block blocksize must be a multiple of 64");+ }+}++} // namespace sycl_tla_s8_detail+} // namespace ark++#endif\ No newline at end of filediff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_sdpa.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_sdpa.hppindex aabe8bff0..ef91b238b 100644--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_sdpa.hpp+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/sycl_tla_sdpa.hpp@@ -364,7 +364,7 @@ struct FMHAConfig {cute::conditional_t<is_void_v<SubgroupLayoutPV_>,decltype(cutlass::fmha::collective::get_sg_layout_pv(SubgroupLayoutQK{})), SubgroupLayoutPV_>;- template <bool isVarLen, bool CachedKV, bool PagedKV, class Scheduler>+ template <bool HasMask, bool isVarLen, bool CachedKV, bool PagedKV, class Scheduler>static int run(const Options& options) {//// Run examples@@ -420,7 +420,7 @@ struct FMHAConfig {CUTLASS_CHECK(runner.run(options, hw_info));} else {- if (options.mask) {+ if constexpr (HasMask) {// Mainloopusing CollectiveMainloop =cutlass::fmha::collective::SDPAFwdMainloop<MainloopDispatchPolicy, Causal, true, CachedKV, PagedKV,@@ -472,8 +472,8 @@ struct FMHAConfig {return 0;}+ template <bool HasMask, bool IsVarLen>static int run(const Options& options) {- bool cached_kv = options.seq_len_kv_cache > 0;if constexpr (persistent) {if (options.use_paged_kv || options.seq_len_kv_cache > 0) {std::cerr@@ -481,23 +481,13 @@ struct FMHAConfig {<< std::endl;return -1;}- return run<false, false, false, cutlass::fmha::kernel::XeFHMAIndividualPersistentTileScheduler>(options);- } else if (options.use_paged_kv && !options.varlen) {- throw std::runtime_error("Paged KV without varlen is not supported yet");- // return run<false, true, true, cutlass::fmha::kernel::XeFHMAIndividualTileScheduler>(options);- } else if (!options.use_paged_kv && options.varlen && !cached_kv) {- return run<true, false, false, cutlass::fmha::kernel::XeFHMAIndividualTileScheduler>(options);- } else if (!options.use_paged_kv && !options.varlen && !cached_kv) {- return run<false, false, false, cutlass::fmha::kernel::XeFHMAIndividualTileScheduler>(options);- } else if (!options.use_paged_kv && options.varlen && cached_kv) {- throw std::runtime_error("Varlen with cached KV but without paged KV is not supported yet");- // return run<true, true, false, cutlass::fmha::kernel::XeFHMAIndividualTileScheduler>(options);- } else if (!options.use_paged_kv && !options.varlen && cached_kv) {- throw std::runtime_error("Cached KV without varlen is not supported yet");- // return run<false, true, false, cutlass::fmha::kernel::XeFHMAIndividualTileScheduler>(options);+ return run<HasMask, false, false, false, cutlass::fmha::kernel::XeFHMAIndividualPersistentTileScheduler>(+ options);} else {- throw std::runtime_error("The combination of options is not supported yet");- // return run<true, true, true, cutlass::fmha::kernel::XeFHMAIndividualTileScheduler>(options);+ if (options.use_paged_kv || options.seq_len_kv_cache > 0) {+ throw std::runtime_error("Paged or cached KV is not supported yet");+ }+ return run<HasMask, IsVarLen, false, false, cutlass::fmha::kernel::XeFHMAIndividualTileScheduler>(options);}}};@@ -859,7 +849,8 @@ struct SageConfig {// Prefill Kernel Launch// ========================================================================-template <typename ElementQ, typename ElementK, typename ElementV, bool persistent = false>+template <bool Causal, bool HasMask, bool IsVarLen, typename ElementQ, typename ElementK, typename ElementV,+ bool persistent = false>inline int launch_prefill_kernel_128(Options const& options) {constexpr int PipelineStages = 2;constexpr int PipelineStages1 = 2;@@ -871,10 +862,14 @@ inline int launch_prefill_kernel_128(Options const& options) {using ShapePV1 = Shape<_256, _32, _32>;using ShapeOut1 = Shape<_256, _128>;using SubgroupLayoutQK1 = Layout<Shape<_16, _1, _1>>;- return options.is_causal ? FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,- /*persistent=*/false, ElementQ, ElementK, ElementV>::run(options)- : FMHAConfig<false, ShapeQK1, ShapePV1, ShapeOut1, SubgroupLayoutQK1, void, PipelineStages1,- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options);+ if constexpr (Causal) {+ return FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,+ /*persistent=*/false, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(options);+ } else {+ return FMHAConfig<false, ShapeQK1, ShapePV1, ShapeOut1, SubgroupLayoutQK1, void, PipelineStages1,+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(+ options);+ }}template <typename ElementQ, typename ElementK, typename ElementV, typename ElementO = ElementV, bool UseInt8PV = false,@@ -917,7 +912,8 @@ inline int launch_sage_prefill_kernel_64(Options const& options) {/*persistent=*/false, ElementQ, ElementK, ElementV, ElementO>::run(options);}-template <typename ElementQ, typename ElementK, typename ElementV, bool persistent = false>+template <bool Causal, bool HasMask, bool IsVarLen, typename ElementQ, typename ElementK, typename ElementV,+ bool persistent = false>inline int launch_prefill_kernel_64(Options const& options) {constexpr int PipelineStages = 1;using ShapeQK = Shape<_256, _64, _32>;@@ -928,39 +924,54 @@ inline int launch_prefill_kernel_64(Options const& options) {using ShapePV1 = Shape<_128, _32, _64>;using ShapeOut1 = Shape<_128, _64>;using SubgroupLayoutQK1 = Layout<Shape<_8, _1, _1>>;- return options.is_causal ? FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,- /*persistent=*/false, ElementQ, ElementK, ElementV>::run(options)- : FMHAConfig<false, ShapeQK1, ShapePV1, ShapeOut1, SubgroupLayoutQK1, void, PipelineStages,- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options);+ if constexpr (Causal) {+ return FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,+ /*persistent=*/false, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(options);+ } else {+ return FMHAConfig<false, ShapeQK1, ShapePV1, ShapeOut1, SubgroupLayoutQK1, void, PipelineStages,+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(+ options);+ }}-template <typename ElementQ, typename ElementK, typename ElementV, bool persistent = false>+template <bool Causal, bool HasMask, bool IsVarLen, typename ElementQ, typename ElementK, typename ElementV,+ bool persistent = false>inline int launch_prefill_kernel_192(Options const& options) {constexpr int PipelineStages = 2;using ShapeQK = Shape<_256, _64, _32>;using ShapePV = Shape<_256, _32, _64>;using ShapeOut = Shape<_256, _192>;using SubgroupLayoutQK = Layout<Shape<_32, _1, _1>>;- return options.is_causal ? FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,- /*persistent=*/false, ElementQ, ElementK, ElementV>::run(options)- : FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options);+ if constexpr (Causal) {+ return FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,+ /*persistent=*/false, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(options);+ } else {+ return FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(+ options);+ }}-template <typename ElementQ, typename ElementK, typename ElementV, bool persistent = false>+template <bool Causal, bool HasMask, bool IsVarLen, typename ElementQ, typename ElementK, typename ElementV,+ bool persistent = false>inline int launch_prefill_kernel_96(Options const& options) {constexpr int PipelineStages = 2;using ShapeQK = Shape<_128, _64, _32>;using ShapePV = Shape<_128, _32, _64>;using ShapeOut = Shape<_128, _96>;using SubgroupLayoutQK = Layout<Shape<_8, _1, _1>>;- return options.is_causal ? FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,- /*persistent=*/false, ElementQ, ElementK, ElementV>::run(options)- : FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options);+ if constexpr (Causal) {+ return FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,+ /*persistent=*/false, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(options);+ } else {+ return FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(+ options);+ }}-template <typename ElementQ, typename ElementK, typename ElementV, bool persistent = false>+template <bool Causal, bool HasMask, bool IsVarLen, typename ElementQ, typename ElementK, typename ElementV,+ bool persistent = false>inline int launch_decode_kernel_128(Options const& options) {constexpr int PipelineStages = 1;using NUM_SG = std::conditional_t<persistent, _16, _8>;@@ -969,13 +980,19 @@ inline int launch_decode_kernel_128(Options const& options) {using ShapePV = Shape<_1, _32, KV_TILE_SIZE>;using ShapeOut = Shape<_1, _128>;using SubgroupLayoutQK = Layout<Shape<_1, NUM_SG, _1>>;- return options.is_causal ? FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options)- : FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options);+ if constexpr (Causal) {+ return FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(+ options);+ } else {+ return FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(+ options);+ }}-template <typename ElementQ, typename ElementK, typename ElementV, bool persistent = false>+template <bool Causal, bool HasMask, bool IsVarLen, typename ElementQ, typename ElementK, typename ElementV,+ bool persistent = false>inline int launch_decode_kernel_64(Options const& options) {constexpr int PipelineStages = 1;using NUM_SG = std::conditional_t<persistent, _16, _8>;@@ -984,13 +1001,19 @@ inline int launch_decode_kernel_64(Options const& options) {using ShapePV = Shape<_1, _32, KV_TILE_SIZE>;using ShapeOut = Shape<_1, _64>;using SubgroupLayoutQK = Layout<Shape<_1, NUM_SG, _1>>;- return options.is_causal ? FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options)- : FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options);+ if constexpr (Causal) {+ return FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(+ options);+ } else {+ return FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(+ options);+ }}-template <typename ElementQ, typename ElementK, typename ElementV, bool persistent = false>+template <bool Causal, bool HasMask, bool IsVarLen, typename ElementQ, typename ElementK, typename ElementV,+ bool persistent = false>inline int launch_decode_kernel_96(Options const& options) {constexpr int PipelineStages = 1;using NUM_SG = std::conditional_t<persistent, _16, _8>;@@ -999,13 +1022,19 @@ inline int launch_decode_kernel_96(Options const& options) {using ShapePV = Shape<_1, _32, KV_TILE_SIZE>;using ShapeOut = Shape<_1, _96>;using SubgroupLayoutQK = Layout<Shape<_1, NUM_SG, _1>>;- return options.is_causal ? FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options)- : FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options);+ if constexpr (Causal) {+ return FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(+ options);+ } else {+ return FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(+ options);+ }}-template <typename ElementQ, typename ElementK, typename ElementV, bool persistent = false>+template <bool Causal, bool HasMask, bool IsVarLen, typename ElementQ, typename ElementK, typename ElementV,+ bool persistent = false>inline int launch_decode_kernel_192(Options const& options) {constexpr int PipelineStages = 1;using NUM_SG = std::conditional_t<persistent, _16, _8>;@@ -1014,10 +1043,15 @@ inline int launch_decode_kernel_192(Options const& options) {using ShapePV = Shape<_1, _32, KV_TILE_SIZE>;using ShapeOut = Shape<_1, _192>;using SubgroupLayoutQK = Layout<Shape<_1, NUM_SG, _1>>;- return options.is_causal ? FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options)- : FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,- /*persistent=*/persistent, ElementQ, ElementK, ElementV>::run(options);+ if constexpr (Causal) {+ return FMHAConfig<true, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(+ options);+ } else {+ return FMHAConfig<false, ShapeQK, ShapePV, ShapeOut, SubgroupLayoutQK, void, PipelineStages,+ /*persistent=*/persistent, ElementQ, ElementK, ElementV>::template run<HasMask, IsVarLen>(+ options);+ }}} // namespace detaildiff --git a/auto_round_extension/ark/auto_round_kernel/wrapper/include/xpu_wrapper.hpp b/auto_round_extension/ark/auto_round_kernel/wrapper/include/xpu_wrapper.hppindex 804198ba3..7bfbc40bc 100644--- a/auto_round_extension/ark/auto_round_kernel/wrapper/include/xpu_wrapper.hpp+++ b/auto_round_extension/ark/auto_round_kernel/wrapper/include/xpu_wrapper.hpp@@ -22,7 +22,7 @@#endif#if ARK_SYCL_TLA-#include "sycl_tla_dense_gemm.hpp"+#include "sycl_tla_common.hpp"#endif#if ARK_XPUdiff --git a/auto_round_extension/ark/setup.py b/auto_round_extension/ark/setup.pyindex 84d6e7b13..13c030f47 100644--- a/auto_round_extension/ark/setup.py+++ b/auto_round_extension/ark/setup.py@@ -12,6 +12,11 @@ from setuptools import Extension, find_packages, setupfrom setuptools.command.build_ext import build_extfrom setuptools.command.build_py import build_py++def _build_log(message):+ print(f"[ARK setup] {message}", file=sys.stderr, flush=True)++build_mode = os.environ.get("BUILD_MODE", "dev").lower()try:file_path = "./auto_round_kernel/version.py"@@ -174,6 +179,26 @@ sycl_target = detect_sycl_target() if enable_sycl_tla else Nonedef get_system_memory_gb():+ cgroup_memory_paths = (+ "/sys/fs/cgroup/memory.max",+ "/sys/fs/cgroup/memory/memory.limit_in_bytes",+ )+ for memory_path in cgroup_memory_paths:+ try:+ memory_limit = Path(memory_path).read_text().strip()+ except OSError:+ continue+ if memory_limit == "max":+ continue+ try:+ memory_bytes = int(memory_limit)+ except ValueError:+ continue+ if 0 < memory_bytes < (1 << 60):+ memory_gb = memory_bytes / (1024**3)+ _build_log(f"System memory from {memory_path}: {memory_gb:.2f} GiB ({memory_bytes} bytes)")+ return memory_gb+if hasattr(os, "sysconf"):page_size_names = ("SC_PAGE_SIZE", "SC_PAGESIZE")page_size = None@@ -184,7 +209,9 @@ def get_system_memory_gb():if page_size is not None and "SC_PHYS_PAGES" in os.sysconf_names:phys_pages = os.sysconf("SC_PHYS_PAGES")if phys_pages > 0:- return (page_size * phys_pages) / (1024**3)+ memory_gb = (page_size * phys_pages) / (1024**3)+ _build_log(f"System memory from sysconf: {memory_gb:.2f} GiB")+ return memory_gbif sys.platform == "win32":@@ -204,17 +231,46 @@ def get_system_memory_gb():memory_status = MEMORYSTATUSEX()memory_status.dwLength = ctypes.sizeof(MEMORYSTATUSEX)if ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(memory_status)):- return memory_status.ullTotalPhys / (1024**3)+ memory_gb = memory_status.ullTotalPhys / (1024**3)+ _build_log(f"System memory from Windows: {memory_gb:.2f} GiB")+ return memory_gb+ _build_log("System memory detection failed; using fallback: 64 GiB")return 64+def get_cpu_count():+ if hasattr(os, "sched_getaffinity"):+ affinity = os.sched_getaffinity(0)+ cpu_count = max(1, len(affinity))+ _build_log(f"CPU count from sched_getaffinity(0): {cpu_count}; allowed CPUs: {sorted(affinity)}")+ return cpu_count+ cpu_count = os.cpu_count() or 1+ _build_log(f"CPU count from os.cpu_count(): {cpu_count}")+ return cpu_count++def get_sycl_tla_job_count(cpu_job_count):+ override = os.environ.get("ARK_SYCL_TLA_JOBS")+ if override is not None:+ try:+ jobs = int(override)+ except ValueError as error:+ raise ValueError("ARK_SYCL_TLA_JOBS must be a positive integer") from error+ if jobs < 1:+ raise ValueError("ARK_SYCL_TLA_JOBS must be a positive integer")+ final_jobs = min(cpu_job_count, jobs)+ _build_log(f"SYCL TLA jobs: min(cpu_job_count={cpu_job_count}, override={jobs}) = {final_jobs}")+ return final_jobs+memory_gb = get_system_memory_gb()- memory_based_jobs = max(- 1, int(memory_gb // 16)- ) # about 5GB/job for SYCL TLA build, use at most 5/16 of total memory to avoid OOM- return min(cpu_job_count, memory_based_jobs)+ memory_based_jobs = max(1, int(memory_gb // 3)) # reserve about 3GB per SYCL TLA compiler job+ final_jobs = min(cpu_job_count, memory_based_jobs)+ _build_log(+ f"SYCL TLA jobs: memory={memory_gb:.2f} GiB, memory_based_jobs={memory_based_jobs}, "+ f"cpu_job_count={cpu_job_count}, final={final_jobs}"+ )+ return final_jobsROOT = Path(__file__).resolve().parent@@ -225,6 +281,7 @@ XBUILD_DIR = ROOT / "xbuild"class CMakeBuild(build_ext):def run(self):+ _build_log(f"CMakeBuild.run using setup.py={Path(__file__).resolve()}, python={sys.executable}")cmake_cmd = ["cmake","-S",@@ -237,8 +294,9 @@ class CMakeBuild(build_ext):cmake_cmd.append("-GNinja")subprocess.check_call(cmake_cmd)- n_job = os.cpu_count() or 2- n_job = n_job // 2+ cpu_count = get_cpu_count()+ n_job = max(1, cpu_count)+ _build_log(f"CPU build jobs: max(1, {cpu_count}) = {n_job}")subprocess.check_call(["cmake", "--build", str(BUILD_DIR), "-j", str(n_job)])ext = "pyd" if sys.platform == "win32" else "so"@@ -271,6 +329,7 @@ class CMakeBuild(build_ext):if sys.platform == "win32":cmake_cmd.append("-GNinja")xpu_n_job = get_sycl_tla_job_count(n_job) if enable_sycl_tla else n_job+ _build_log(f"Building XPU extension with {xpu_n_job} parallel job(s)")subprocess.check_call(cmake_cmd)subprocess.check_call(["cmake", "--build", str(XBUILD_DIR), "-j", str(xpu_n_job)])diff --git a/auto_round_extension/ark/test/test_ark_cpu_mixed_bestla_sdpa.py b/auto_round_extension/ark/test/test_ark_cpu_mixed_bestla_sdpa.pyindex 300f51ab8..7749a326a 100644--- a/auto_round_extension/ark/test/test_ark_cpu_mixed_bestla_sdpa.py+++ b/auto_round_extension/ark/test/test_ark_cpu_mixed_bestla_sdpa.py@@ -13,6 +13,11 @@ import mathimport pytestimport torch+cpuinfo = pytest.importorskip("cpuinfo")+CPU_FLAGS = set(cpuinfo.get_cpu_info().get("flags", []))+if "amx_bf16" not in CPU_FLAGS:+ pytest.skip("ARK mixed BestLA SDPA tests require CPU flag amx_bf16", allow_module_level=True)+auto_round_kernel = pytest.importorskip("auto_round_kernel", reason="compiled ARK extension not built in this environment")diff --git a/auto_round_extension/ark/tools/analyze_sycl_tla_templates.py b/auto_round_extension/ark/tools/analyze_sycl_tla_templates.pynew file mode 100644index 000000000..f5a407afa--- /dev/null+++ b/auto_round_extension/ark/tools/analyze_sycl_tla_templates.py@@ -0,0 +1,193 @@+#!/usr/bin/env python3++# # Copyright (C) 2026 Intel Corporation+# # SPDX-License-Identifier: Apache-2.0++import argparse+import json+import re+from collections import defaultdict, deque+from pathlib import Path++LOCAL_INCLUDE = re.compile(r'^\s*#\s*include\s+"([^"]+)"')+TEMPLATE_DECLARATION = re.compile(r"^\s*template\s*<")++CALL_PATTERNS = {+ "dense_gemm": re.compile(r"\b(?:gemm_cute_store_tile\s*<|run_dense_gemm\s*\()"),+ "s8_gemm": re.compile(r"\b(?:launch_igemm\s*<|launch_igemm_kblock\s*<|run_typed\s*\()"),+ "moe_gemm": re.compile(r"\bmoe_gemm_launcher\s*<"),+ "moe_decode": re.compile(r"\blaunch_(?:fp|int4|int8|int2|fp8)\s*<"),+ "moe_prefill": re.compile(r"\b(?:moe_prefill_[a-z0-9_]+_dispatch|dequant_to_KN|launch_dequant_fp8_slm)\s*<"),+ "sdpa": re.compile(r"\blaunch_(?:sage|prefill|sparse)[a-z0-9_]*\s*<"),+}++ESTIMATED_KERNELS = {+ "dense_gemm": 24,+ "s8_gemm": 96,+ "moe_gemm": 2,+ "moe_decode": 22,+ "moe_prefill": 44,+ "sdpa": 14,+}+++def resolve_include(include_name, including_file, files_by_name):+ candidate = including_file.parent / include_name+ if candidate.is_file():+ return candidate.resolve()+ for parent in including_file.parents:+ candidate = parent / "wrapper" / "include" / include_name+ if candidate.is_file():+ return candidate.resolve()+ matches = [path for path in files_by_name.get(Path(include_name).name, []) if path.name == Path(include_name).name]+ return matches[0].resolve() if len(matches) == 1 else None+++def reachable_files(source, files_by_name):+ result = []+ pending = deque([source.resolve()])+ visited = set()+ while pending:+ current = pending.popleft()+ if current in visited or not current.is_file():+ continue+ visited.add(current)+ result.append(current)+ for line in current.read_text(encoding="utf-8", errors="ignore").splitlines():+ match = LOCAL_INCLUDE.match(line)+ if match:+ included = resolve_include(match.group(1), current, files_by_name)+ if included is not None:+ pending.append(included)+ return result+++def analyze_file(path):+ declarations = 0+ calls = defaultdict(int)+ for line in path.read_text(encoding="utf-8", errors="ignore").splitlines():+ if TEMPLATE_DECLARATION.match(line):+ declarations += 1+ continue+ for family, pattern in CALL_PATTERNS.items():+ calls[family] += len(pattern.findall(line))+ return declarations, calls+++def compile_command_sources(build_dir):+ if build_dir is None:+ return []+ compile_commands = build_dir / "compile_commands.json"+ if not compile_commands.is_file():+ return []+ entries = json.loads(compile_commands.read_text(encoding="utf-8"))+ sources = []+ for entry in entries:+ source = Path(entry["file"])+ if not source.is_absolute():+ source = Path(entry.get("directory", build_dir)) / source+ source = source.resolve()+ if source.is_file() and source.suffix in {".c", ".cc", ".cpp", ".cxx"}:+ sources.append(source)+ return sorted(set(sources))+++def source_files(root, build_dir=None):+ sources = set(root.glob("*.cpp"))+ sources.update(compile_command_sources(build_dir))+ return sorted(path.resolve() for path in sources if path.is_file())+++def source_label(source, root):+ try:+ return str(source.relative_to(root))+ except ValueError:+ return str(source)+++def analyze_sources(root, build_dir=None):+ files_by_name = defaultdict(list)+ indexed_roots = [root]+ if build_dir is not None and (build_dir / "generated").is_dir():+ indexed_roots.append(build_dir / "generated")+ for indexed_root in indexed_roots:+ for path in indexed_root.rglob("*"):+ if path.is_file():+ files_by_name[path.name].append(path)++ results = []+ for source in source_files(root, build_dir):+ reachable = reachable_files(source, files_by_name)+ declarations = 0+ reachable_calls = defaultdict(int)+ for included in reachable:+ file_declarations, file_calls = analyze_file(included)+ declarations += file_declarations+ for family, count in file_calls.items():+ reachable_calls[family] += count+ direct_declarations, direct_calls = analyze_file(source)+ results.append(+ {+ "source": str(source),+ "label": source_label(source, root),+ "reachable_files": len(reachable),+ "template_declarations": declarations,+ "direct_template_declarations": direct_declarations,+ "reachable_dispatch_calls": dict(sorted(reachable_calls.items())),+ "direct_dispatch_calls": dict(sorted(direct_calls.items())),+ }+ )+ return results+++def main():+ parser = argparse.ArgumentParser(description="Statically count SYCL-TLA template dispatch sites.")+ parser.add_argument(+ "--root",+ type=Path,+ default=Path(__file__).resolve().parents[1] / "auto_round_kernel",+ help="ARK auto_round_kernel source directory",+ )+ parser.add_argument(+ "--build-dir",+ type=Path,+ help="CMake build directory; compile_commands.json adds generated translation units",+ )+ parser.add_argument("--json", type=Path, help="Write machine-readable per-translation-unit results")+ args = parser.parse_args()+ root = args.root.resolve()+ build_dir = args.build_dir.resolve() if args.build_dir else None+ results = analyze_sources(root, build_dir)++ print(f"root: {root}")+ print("source | reachable files | declarations | direct dispatch | reachable dispatch")+ print("-------|------------------|--------------|-----------------|-------------------")+ for result in results:+ direct = sum(result["direct_dispatch_calls"].values())+ reachable = sum(result["reachable_dispatch_calls"].values())+ print(+ f"{result['label']} | {result['reachable_files']:16d} | {result['template_declarations']:12d} | "+ f"{direct:15d} | {reachable:19d}"+ )++ print("\nestimated concrete kernel combinations")+ estimated_total = sum(ESTIMATED_KERNELS.values())+ for family, count in ESTIMATED_KERNELS.items():+ print(f"{family}: {count}")+ print(f"estimated total: {estimated_total}")+ max_result = max(results, key=lambda result: sum(result["direct_dispatch_calls"].values()), default=None)+ if max_result:+ print(+ f"max direct-dispatch source: {max_result['label']} "+ f"({sum(max_result['direct_dispatch_calls'].values())} call sites)"+ )+ print(f"translation units analyzed: {len(results)}")+ print(+ "\nReachable declarations include header definitions parsed by each translation unit; direct dispatch counts only calls written in the .cpp file."+ )+ if args.json:+ args.json.write_text(json.dumps(results, indent=2, sort_keys=True) + "\n", encoding="utf-8")+++if __name__ == "__main__":+ main()diff --git a/auto_round_extension/ark/tools/measure_sycl_tla_compile_memory.py b/auto_round_extension/ark/tools/measure_sycl_tla_compile_memory.pynew file mode 100644index 000000000..c2d2a75b8--- /dev/null+++ b/auto_round_extension/ark/tools/measure_sycl_tla_compile_memory.py@@ -0,0 +1,142 @@+#!/usr/bin/env python3++# # Copyright (C) 2026 Intel Corporation+# # SPDX-License-Identifier: Apache-2.0++import argparse+import json+import re+import shlex+import subprocess+import sys+from pathlib import Path++RSS_PATTERN = re.compile(r"Maximum resident set size \(kbytes\):\s*(\d+)")+++def load_compile_commands(path):+ entries = json.loads(path.read_text(encoding="utf-8"))+ commands = []+ for entry in entries:+ if "arguments" in entry:+ argv = list(entry["arguments"])+ elif "command" in entry:+ argv = shlex.split(entry["command"])+ else:+ raise ValueError("compile_commands entry has neither 'arguments' nor 'command'")+ source = Path(entry["file"])+ if not source.is_absolute():+ source = Path(entry["directory"]) / source+ commands.append(+ {+ "source": source.resolve(),+ "directory": Path(entry["directory"]).resolve(),+ "argv": argv,+ }+ )+ return commands+++def measure_command(command, time_binary):+ completed = subprocess.run(+ [time_binary, "-v", *command["argv"]],+ cwd=command["directory"],+ stdout=subprocess.PIPE,+ stderr=subprocess.PIPE,+ text=True,+ check=False,+ )+ match = RSS_PATTERN.search(completed.stderr)+ result = {+ "source": str(command["source"]),+ "returncode": completed.returncode,+ "max_rss_kib": int(match.group(1)) if match else None,+ "max_rss_mib": int(match.group(1)) / 1024 if match else None,+ "stdout_tail": completed.stdout[-4000:],+ "stderr_tail": completed.stderr[-4000:],+ "command": command["argv"],+ }+ return result+++def main():+ parser = argparse.ArgumentParser(+ description="Measure peak RSS for each compiler invocation in compile_commands.json."+ )+ parser.add_argument("--build-dir", type=Path, required=True, help="CMake build directory")+ parser.add_argument(+ "--compile-commands",+ type=Path,+ help="compile_commands.json path; defaults to <build-dir>/compile_commands.json",+ )+ parser.add_argument(+ "--root",+ type=Path,+ default=Path(__file__).resolve().parents[1] / "auto_round_kernel",+ help="ARK auto_round_kernel source directory",+ )+ parser.add_argument("--source-regex", help="Only measure translation units whose path matches this regex")+ parser.add_argument("--time-binary", default="/usr/bin/time", help="GNU time executable")+ parser.add_argument("--output", type=Path, help="Write detailed JSON results")+ args = parser.parse_args()++ compile_commands = (args.compile_commands or args.build_dir / "compile_commands.json").resolve()+ if not compile_commands.is_file():+ raise SystemExit(f"compile commands not found: {compile_commands}")+ if not Path(args.time_binary).is_file():+ raise SystemExit(f"time executable not found: {args.time_binary}")++ commands = load_compile_commands(compile_commands)+ if args.source_regex:+ source_pattern = re.compile(args.source_regex)+ commands = [command for command in commands if source_pattern.search(str(command["source"]))]+ if not commands:+ raise SystemExit("no compile commands selected")++ sys.path.insert(0, str(Path(__file__).resolve().parent))+ from analyze_sycl_tla_templates import analyze_sources++ analysis = analyze_sources(args.root.resolve(), args.build_dir.resolve())+ analysis_by_source = {str(Path(item["source"]).resolve()): item for item in analysis}++ results = []+ for index, command in enumerate(commands, start=1):+ print(f"[{index}/{len(commands)}] {command['source']}", flush=True)+ result = measure_command(command, args.time_binary)+ result.update(analysis_by_source.get(str(command["source"]), {}))+ direct_dispatches = sum(result.get("direct_dispatch_calls", {}).values())+ result["rss_per_direct_dispatch_mib"] = (+ result["max_rss_mib"] / direct_dispatches+ if result["max_rss_mib"] is not None and direct_dispatches+ else None+ )+ results.append(result)+ rss = result["max_rss_mib"]+ rss_text = f"{rss:.1f} MiB" if rss is not None else "unknown RSS"+ print(f" returncode={result['returncode']} peak RSS={rss_text}", flush=True)++ measured = [result for result in results if result["max_rss_kib"] is not None]+ measured.sort(key=lambda result: result["max_rss_kib"], reverse=True)+ print("source | peak RSS MiB | RSS/direct MiB | return code | direct dispatch | declarations")+ print("-------|---------------|----------------|-------------|-----------------|-------------")+ for result in measured:+ direct = sum(result.get("direct_dispatch_calls", {}).values())+ per_dispatch = result["rss_per_direct_dispatch_mib"]+ per_dispatch_text = f"{per_dispatch:14.1f}" if per_dispatch is not None else " n/a"+ print(+ f"{result.get('label', result['source'])} | {result['max_rss_mib']:13.1f} | {per_dispatch_text} | "+ f"{result['returncode']:11d} | {direct:15d} | {result.get('template_declarations', 'n/a'):11}"+ )+ if measured:+ print(+ f"\npeak compiler RSS: {measured[0]['max_rss_mib']:.1f} MiB ({measured[0].get('label', measured[0]['source'])})"+ )+ else:+ print("\nNo GNU time RSS measurements were parsed.")++ if args.output:+ args.output.write_text(json.dumps(results, indent=2, sort_keys=True) + "\n", encoding="utf-8")+++if __name__ == "__main__":+ main()