LLM inference in C/C++
C++ ggml
common: rename --tensor-read-lazy to --lazy-mode, add -lzm shorthand (#27969)
Georgi Gerganovbebc9350ecc4
diff --git a/common/arg.cpp b/common/arg.cppindex 4469612cd5..79405b59e0 100644--- a/common/arg.cpp+++ b/common/arg.cpp@@ -2729,7 +2729,7 @@ common_params_context common_params_parser_init(common_params & params, llama_ex}).set_env("LLAMA_ARG_LOAD_MODE"));add_opt(common_arg(- {"--tensor-read-lazy"}, "MODE",+ {"-lzm", "--lazy-mode"}, "MODE","on-demand reading of certain tensors, for example per-layer embeddings (default: auto)\n""- on: read the rows of such tensors from disk on demand instead of keeping them resident (requires mmap)\n""- auto: on, but only for tensors larger than 4 GiB\n"@@ -2740,7 +2740,7 @@ common_params_context common_params_parser_init(common_params & params, llama_exelse if (value == "off") { params.lazy_mode = LLAMA_LAZY_MODE_OFF; }else { throw std::invalid_argument("invalid value"); }}- ).set_env("LLAMA_ARG_TENSOR_READ_LAZY"));+ ).set_env("LLAMA_ARG_LAZY_MODE"));add_opt(common_arg({"--numa"}, "TYPE","attempt optimizations that help on some NUMA systems\n"diff --git a/tools/cli/README.md b/tools/cli/README.mdindex 163ee4fbaf..b874d02073 100644--- a/tools/cli/README.md+++ b/tools/cli/README.md@@ -59,7 +59,7 @@| `--mmap, --no-mmap` | DEPRECATED in favor of `--load-mode`: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>(env: LLAMA_ARG_MMAP) || `-dio, --direct-io, -ndio, --no-direct-io` | DEPRECATED in favor of `--load-mode`: use DirectIO if available<br/>(env: LLAMA_ARG_DIO) || `-lm, --load-mode MODE` | model loading mode (default: auto)<br/>- auto: mmap, unless a device does not support it<br/>- none: no special loading mode<br/>- mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>- mlock: force system to keep model in RAM rather than swapping or compressing<br/>- mmap+mlock: mmap + force system to keep model in RAM rather than swapping or compressing<br/>- dio: use DirectIO if available<br/><br/>(env: LLAMA_ARG_LOAD_MODE) |-| `--tensor-read-lazy MODE` | on-demand reading of certain tensors, for example per-layer embeddings (default: auto)<br/>- on: read the rows of such tensors from disk on demand instead of keeping them resident (requires mmap)<br/>- auto: on, but only for tensors larger than 4 GiB<br/>- off: always keep them resident<br/>(env: LLAMA_ARG_TENSOR_READ_LAZY) |+| `-lzm, --lazy-mode MODE` | on-demand reading of certain tensors, for example per-layer embeddings (default: auto)<br/>- on: read the rows of such tensors from disk on demand instead of keeping them resident (requires mmap)<br/>- auto: on, but only for tensors larger than 4 GiB<br/>- off: always keep them resident<br/>(env: LLAMA_ARG_LAZY_MODE) || `--numa TYPE` | attempt optimizations that help on some NUMA systems<br/>- distribute: spread execution evenly over all nodes<br/>- isolate: only spawn threads on CPUs on the node that execution started on<br/>- numactl: use the CPU map provided by numactl<br/>if run without this previously, it is recommended to drop the system page cache before using this<br/>see https://github.com/ggml-org/llama.cpp/issues/1437<br/>(env: LLAMA_ARG_NUMA) || `-dev, --device <dev1,dev2,..>` | comma-separated list of devices to use for offloading (none = don't offload)<br/>use --list-devices to see a list of available devices<br/>(env: LLAMA_ARG_DEVICE) || `--list-devices` | print list of available devices and exit |diff --git a/tools/completion/README.md b/tools/completion/README.mdindex 0cd86bac70..145be77e31 100644--- a/tools/completion/README.md+++ b/tools/completion/README.md@@ -142,7 +142,7 @@ llama-completion.exe -m models\gemma-1.1-7b-it.Q4_K_M.gguf --ignore-eos -n -1| `--mmap, --no-mmap` | DEPRECATED in favor of `--load-mode`: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>(env: LLAMA_ARG_MMAP) || `-dio, --direct-io, -ndio, --no-direct-io` | DEPRECATED in favor of `--load-mode`: use DirectIO if available<br/>(env: LLAMA_ARG_DIO) || `-lm, --load-mode MODE` | model loading mode (default: auto)<br/>- auto: mmap, unless a device does not support it<br/>- none: no special loading mode<br/>- mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>- mlock: force system to keep model in RAM rather than swapping or compressing<br/>- mmap+mlock: mmap + force system to keep model in RAM rather than swapping or compressing<br/>- dio: use DirectIO if available<br/><br/>(env: LLAMA_ARG_LOAD_MODE) |-| `--tensor-read-lazy MODE` | on-demand reading of certain tensors, for example per-layer embeddings (default: auto)<br/>- on: read the rows of such tensors from disk on demand instead of keeping them resident (requires mmap)<br/>- auto: on, but only for tensors larger than 4 GiB<br/>- off: always keep them resident<br/>(env: LLAMA_ARG_TENSOR_READ_LAZY) |+| `-lzm, --lazy-mode MODE` | on-demand reading of certain tensors, for example per-layer embeddings (default: auto)<br/>- on: read the rows of such tensors from disk on demand instead of keeping them resident (requires mmap)<br/>- auto: on, but only for tensors larger than 4 GiB<br/>- off: always keep them resident<br/>(env: LLAMA_ARG_LAZY_MODE) || `--numa TYPE` | attempt optimizations that help on some NUMA systems<br/>- distribute: spread execution evenly over all nodes<br/>- isolate: only spawn threads on CPUs on the node that execution started on<br/>- numactl: use the CPU map provided by numactl<br/>if run without this previously, it is recommended to drop the system page cache before using this<br/>see https://github.com/ggml-org/llama.cpp/issues/1437<br/>(env: LLAMA_ARG_NUMA) || `-dev, --device <dev1,dev2,..>` | comma-separated list of devices to use for offloading (none = don't offload)<br/>use --list-devices to see a list of available devices<br/>(env: LLAMA_ARG_DEVICE) || `--list-devices` | print list of available devices and exit |diff --git a/tools/llama-bench/README.md b/tools/llama-bench/README.mdindex a1404d2e33..8adc56514d 100644--- a/tools/llama-bench/README.md+++ b/tools/llama-bench/README.md@@ -67,7 +67,7 @@ test parameters:-nkvo, --no-kv-offload <0|1> (default: 0)-fa, --flash-attn <on|off|auto> (default: auto)-dev, --device <dev0/dev1/...> (default: auto)- --tensor-read-lazy <on|auto|off> (default: auto)+ -lzm, --lazy-mode <on|auto|off> (default: auto)-mmp, --mmap <0|1> (DEPRECATED IN FAVOUR OF --load-mode)-dio, --direct-io <0|1> (DEPRECATED IN FAVOUR OF --load-mode)-embd, --embeddings <0|1> (default: 0)diff --git a/tools/llama-bench/llama-bench.cpp b/tools/llama-bench/llama-bench.cppindex 1b4bbde4d4..1fff21f701 100644--- a/tools/llama-bench/llama-bench.cpp+++ b/tools/llama-bench/llama-bench.cpp@@ -280,7 +280,7 @@ static const char * lazy_mode_str(llama_lazy_mode mode) {case LLAMA_LAZY_MODE_ON:return "on";default:- GGML_ABORT("invalid tensor read lazy mode");+ GGML_ABORT("invalid lazy mode");}}@@ -475,7 +475,7 @@ static void print_usage(int /* argc */, char ** argv) {printf(" -fa, --flash-attn <on|off|auto> (default: %s)\n", join(transform_to_str(cmd_params_defaults.flash_attn, llama_flash_attn_type_name), ",").c_str());printf(" -dev, --device <dev0/dev1/...> (default: auto)\n");printf(" -lm, --load-mode <auto|none|mmap|mlock|mmap+mlock|dio> (default: %s)\n", join(transform_to_str(cmd_params_defaults.load_mode, llama_load_mode_name), ",").c_str());- printf(" --tensor-read-lazy <on|auto|off> (default: %s)\n", join(transform_to_str(cmd_params_defaults.lazy_mode, lazy_mode_str), ",").c_str());+ printf(" -lzm, --lazy-mode <on|auto|off> (default: %s)\n", join(transform_to_str(cmd_params_defaults.lazy_mode, lazy_mode_str), ",").c_str());printf(" -mmp, --mmap <0|1> (DEPRECATED IN FAVOUR OF --load-mode)\n");printf(" -dio, --direct-io <0|1> (DEPRECATED IN FAVOUR OF --load-mode)\n");printf(" -embd, --embeddings <0|1> (default: %s)\n", join(cmd_params_defaults.embeddings, ",").c_str());@@ -802,7 +802,7 @@ static cmd_params parse_cmd_params(int argc, char ** argv) {break;}params.load_mode.insert(params.load_mode.end(), modes.begin(), modes.end());- } else if (arg == "--tensor-read-lazy") {+ } else if (arg == "-lzm" || arg == "--lazy-mode") {if (++i >= argc) {invalid_param = true;break;diff --git a/tools/server/README.md b/tools/server/README.mdindex 3c2228f343..c6e907ba91 100644--- a/tools/server/README.md+++ b/tools/server/README.md@@ -76,7 +76,7 @@ For the full list of features, please refer to [server's changelog](https://gith| `--mmap, --no-mmap` | DEPRECATED in favor of `--load-mode`: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>(env: LLAMA_ARG_MMAP) || `-dio, --direct-io, -ndio, --no-direct-io` | DEPRECATED in favor of `--load-mode`: use DirectIO if available<br/>(env: LLAMA_ARG_DIO) || `-lm, --load-mode MODE` | model loading mode (default: auto)<br/>- auto: mmap, unless a device does not support it<br/>- none: no special loading mode<br/>- mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>- mlock: force system to keep model in RAM rather than swapping or compressing<br/>- mmap+mlock: mmap + force system to keep model in RAM rather than swapping or compressing<br/>- dio: use DirectIO if available<br/><br/>(env: LLAMA_ARG_LOAD_MODE) |-| `--tensor-read-lazy MODE` | on-demand reading of certain tensors, for example per-layer embeddings (default: auto)<br/>- on: read the rows of such tensors from disk on demand instead of keeping them resident (requires mmap)<br/>- auto: on, but only for tensors larger than 4 GiB<br/>- off: always keep them resident<br/>(env: LLAMA_ARG_TENSOR_READ_LAZY) |+| `-lzm, --lazy-mode MODE` | on-demand reading of certain tensors, for example per-layer embeddings (default: auto)<br/>- on: read the rows of such tensors from disk on demand instead of keeping them resident (requires mmap)<br/>- auto: on, but only for tensors larger than 4 GiB<br/>- off: always keep them resident<br/>(env: LLAMA_ARG_LAZY_MODE) || `--numa TYPE` | attempt optimizations that help on some NUMA systems<br/>- distribute: spread execution evenly over all nodes<br/>- isolate: only spawn threads on CPUs on the node that execution started on<br/>- numactl: use the CPU map provided by numactl<br/>if run without this previously, it is recommended to drop the system page cache before using this<br/>see https://github.com/ggml-org/llama.cpp/issues/1437<br/>(env: LLAMA_ARG_NUMA) || `-dev, --device <dev1,dev2,..>` | comma-separated list of devices to use for offloading (none = don't offload)<br/>use --list-devices to see a list of available devices<br/>(env: LLAMA_ARG_DEVICE) || `--list-devices` | print list of available devices and exit |