LLM inference in C/C++
C++ ggml
sycl : support limit max alloc memory within 2GB for host-pinned memory (#27559)
Neo Zhang5eec3ad01762
diff --git a/docs/backend/SYCL.md b/docs/backend/SYCL.mdindex 5554c1d185..b30e109ad4 100644--- a/docs/backend/SYCL.md+++ b/docs/backend/SYCL.md@@ -795,7 +795,8 @@ User can use the device management in [docs/multi-gpu.md](https://github.com/ggm| GGML_SYCL_ENABLE_FLASH_ATTN | 1 (default) or 0| Enable Flash-Attention. It can reduce memory usage. The performance impact depends on the LLM.|| GGML_SYCL_ENABLE_OPT | 0 or 1 (default)| Enable optimize features for Intel GPUs. (Recommended to 0 for Intel devices older than Gen 10) || GGML_SYCL_ENABLE_GRAPH | 0 (default) or 1 | Enable running computations through SYCL Graphs feature. Disabled by default because SYCL Graph is still on development, no better performance. |-| GGML_SYCL_ENABLE_HOST_PINNED_MEM | 0 or 1 (default) | Enable host pinned memory to speed up copy data from host to device. When disable it, host memory will common malloc() on CPU.|+| GGML_SYCL_ENABLE_HOST_PINNED_MEM | 0 or 1 (default) | Enable host pinned memory to speed up copy data from host to device. When disable it, host memory will common malloc() on CPU. Disable it when use `--load-model mlock`.|+| GGML_SYCL_HOST_PINNED_MEM_2G | 0 (default) or 1 | Limit the max memory allocation to be no more than 2GB when enable host pinned memory. USM allocations above 2 GiB take the relaxed/large-allocation path, which serializes H2D copies with compute and prevents copy/compute overlap. It will impact the startup time. Need more test. Depend on `GGML_SYCL_ENABLE_HOST_PINNED_MEM=1`.|| GGML_SYCL_GET_MEM_API | 0 (default) or 1 | Set to get memory info (free, total) by Level Zero or SYCL API:<br>0 - Level Zero API: support more GPUs, only run on Level Zero running time. When there is an error, fallback to call SYCL API. Depend on GGML_SYCL_SUPPORT_LEVEL_ZERO_API.<br>1 - SYCL API: legacy, support more running time, it can't get the free size of some GPUs (like Arc770). In such case, return total size for free size.|| GGML_SYCL_USE_LEVEL_ZERO_API | 1 (default) or 0 | Use Level Zero API for device memory allocation instead of SYCL. Reduces system RAM usage on Intel dGPUs by avoiding DMA-buf/TTM host memory staging. Requires GGML_SYCL_SUPPORT_LEVEL_ZERO_API=ON at build time. SYCL backend always runs on Level Zero running time even if it's set as OFF (The SYCL api will be usage for memory allocation).|| GGML_SYCL_ENABLE_DNN | 0 or 1 (default)| Enable running computations through oneDNN and always use oneMKL. |diff --git a/ggml/src/ggml-sycl/ggml-sycl.cpp b/ggml/src/ggml-sycl/ggml-sycl.cppindex 290fb46767..626f0f3bff 100644--- a/ggml/src/ggml-sycl/ggml-sycl.cpp+++ b/ggml/src/ggml-sycl/ggml-sycl.cpp@@ -107,6 +107,7 @@ int g_ggml_sycl_enable_flash_attention = 1;int g_ggml_sycl_dev2dev_memcpy = DEV2DEV_MEMCPY_SYCL;int g_ggml_sycl_usm_system = 0;int g_ggml_sycl_enable_host_pinned_mem = 1;+int g_ggml_sycl_host_pinned_mem_2g = 0;int g_ggml_sycl_get_mem_api = MEMORY_API_TYPE_LEVEL_ZERO;@@ -355,6 +356,8 @@ static void ggml_check_sycl() try {g_ggml_sycl_enable_host_pinned_mem =ggml_sycl_get_env("GGML_SYCL_ENABLE_HOST_PINNED_MEM", 1);+ g_ggml_sycl_host_pinned_mem_2g =+ ggml_sycl_get_env("GGML_SYCL_HOST_PINNED_MEM_2G", 0) & g_ggml_sycl_enable_host_pinned_mem;GGML_SYCL_DEBUG("[SYCL] call ggml_check_sycl\n");@@ -457,6 +460,7 @@ static void ggml_check_sycl() try {GGML_LOG_INFO(" GGML_SYCL_USM_SYSTEM: %d\n", g_ggml_sycl_usm_system);GGML_LOG_INFO(" GGML_SYCL_ENABLE_HOST_PINNED_MEM: %d\n", g_ggml_sycl_enable_host_pinned_mem);+ GGML_LOG_INFO(" GGML_SYCL_HOST_PINNED_MEM_2G: %d\n", g_ggml_sycl_host_pinned_mem_2g);/* NOT REMOVE, keep it for next optimize for XMX.#if defined(SYCL_USE_XMX)@@ -977,8 +981,12 @@ static size_t ggml_backend_sycl_buffer_type_get_alignment(ggml_backend_buffer_ty}static size_t ggml_backend_sycl_buffer_type_get_max_size(ggml_backend_buffer_type_t buft) {- return dpct::get_current_device().get_max_mem_alloc_size();-+ size_t max_alloc_size = dpct::get_current_device().get_max_mem_alloc_size();+ if (g_ggml_sycl_host_pinned_mem_2g) {+ return std::min(max_alloc_size, (size_t) 2LL*1024*1024*1024);+ } else {+ return max_alloc_size;+ }GGML_UNUSED(buft);}@@ -1551,7 +1559,12 @@ static size_t ggml_backend_sycl_host_buffer_type_get_max_size(ggml_backend_buffeif (g_ggml_sycl_enable_host_pinned_mem) {ggml_backend_sycl_device_context * dev_ctx = (ggml_backend_sycl_device_context *) buft->device->context;- return dpct::dev_mgr::instance().get_device(dev_ctx->device).get_max_mem_alloc_size();+ size_t max_alloc_size = dpct::dev_mgr::instance().get_device(dev_ctx->device).get_max_mem_alloc_size();+ if (g_ggml_sycl_host_pinned_mem_2g) {+ return std::min(max_alloc_size, (size_t) 2LL*1024*1024*1024);+ } else {+ return max_alloc_size;+ }} else {return SIZE_MAX;}