LLM inference in C/C++
C++ ggml
args: officially deprecate --mmap|mlock|dio (#28334)
Aaron Teo14a9d09f7568
diff --git a/common/arg.cpp b/common/arg.cppindex 74241f9312..43052d58d1 100644--- a/common/arg.cpp+++ b/common/arg.cpp@@ -872,17 +872,6 @@ static bool common_params_parse_ex(int argc, char ** argv, common_params_contextarg.c_str(), e.what(), opt.to_string().c_str()));}}-- // TODO: remove this check after deprecating --mmap|mlock|dio- auto has_arg = [&](std::initializer_list<const char *> names) {- return std::any_of(names.begin(), names.end(), [&](const char * name) {- return seen_args.count(name);- });- };- if (has_arg({"-lm", "--load-mode"}) &&- has_arg({"--mlock", "--mmap", "--no-mmap", "-dio", "--direct-io", "-ndio", "--no-direct-io"})) {- LOG_WRN("DEPRECATED: `--load-mode` and `--mlock`/`--mmap`/`--direct-io` should not be combined; only the last flag on the command line will take effect\n");- }};// parse all CLI args now, so that -hf is available below for remote preset resolution@@ -2694,32 +2683,6 @@ common_params_context common_params_parser_init(common_params & params, llama_ex}).set_env("LLAMA_ARG_RPC"));}- add_opt(common_arg(- {"--mlock"},- "DEPRECATED in favor of `--load-mode`: force system to keep model in RAM rather than swapping or compressing",- [](common_params & params) {- LOG_WRN("DEPRECATED: --mlock is deprecated. use --load-mode mlock instead\n");- params.load_mode = LLAMA_LOAD_MODE_MLOCK;- }- ).set_env("LLAMA_ARG_MLOCK"));- add_opt(common_arg(- {"--mmap"},- {"--no-mmap"},- "DEPRECATED in favor of `--load-mode`: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock)",- [](common_params & params, bool value) {- LOG_WRN("DEPRECATED: --mmap and --no-mmap are deprecated. use --load-mode mmap instead\n");- params.load_mode = value ? LLAMA_LOAD_MODE_MMAP : LLAMA_LOAD_MODE_NONE;- }- ).set_env("LLAMA_ARG_MMAP"));- add_opt(common_arg(- {"-dio", "--direct-io"},- {"-ndio", "--no-direct-io"},- "DEPRECATED in favor of `--load-mode`: use DirectIO if available",- [](common_params & params, bool value) {- LOG_WRN("DEPRECATED: --direct-io and --no-direct-io are deprecated. use --load-mode dio instead\n");- params.load_mode = value ? LLAMA_LOAD_MODE_DIRECT_IO : LLAMA_LOAD_MODE_NONE;- }- ).set_env("LLAMA_ARG_DIO"));add_opt(common_arg({"-lm", "--load-mode"}, "MODE","model loading mode (default: auto)\n"diff --git a/tools/cli/README.md b/tools/cli/README.mdindex efe653494d..77a5e6fe32 100644--- a/tools/cli/README.md+++ b/tools/cli/README.md@@ -55,9 +55,6 @@| `-dt, --defrag-thold N` | KV cache defragmentation threshold (DEPRECATED)<br/>(env: LLAMA_ARG_DEFRAG_THOLD) || `-np, --parallel N` | number of parallel sequences to decode (default: 1)<br/>(env: LLAMA_ARG_N_PARALLEL) || `--rpc SERVERS` | comma-separated list of RPC servers (host:port)<br/>(env: LLAMA_ARG_RPC) |-| `--mlock` | DEPRECATED in favor of `--load-mode`: force system to keep model in RAM rather than swapping or compressing<br/>(env: LLAMA_ARG_MLOCK) |-| `--mmap, --no-mmap` | DEPRECATED in favor of `--load-mode`: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>(env: LLAMA_ARG_MMAP) |-| `-dio, --direct-io, -ndio, --no-direct-io` | DEPRECATED in favor of `--load-mode`: use DirectIO if available<br/>(env: LLAMA_ARG_DIO) || `-lm, --load-mode MODE` | model loading mode (default: auto)<br/>- auto: mmap, unless a device does not support it<br/>- none: no special loading mode<br/>- mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>- mlock: force system to keep model in RAM rather than swapping or compressing<br/>- mmap+mlock: mmap + force system to keep model in RAM rather than swapping or compressing<br/>- dio: use DirectIO if available<br/><br/>(env: LLAMA_ARG_LOAD_MODE) || `-lzm, --lazy-mode MODE` | on-demand reading of certain tensors, for example per-layer embeddings (default: auto)<br/>- on: read the rows of such tensors from disk on demand instead of keeping them resident (requires mmap)<br/>- auto: on, but only for tensors larger than 4 GiB<br/>- off: always keep them resident<br/>(env: LLAMA_ARG_LAZY_MODE) || `--numa TYPE` | attempt optimizations that help on some NUMA systems<br/>- distribute: spread execution evenly over all nodes<br/>- isolate: only spawn threads on CPUs on the node that execution started on<br/>- numactl: use the CPU map provided by numactl<br/>if run without this previously, it is recommended to drop the system page cache before using this<br/>see https://github.com/ggml-org/llama.cpp/issues/1437<br/>(env: LLAMA_ARG_NUMA) |diff --git a/tools/completion/README.md b/tools/completion/README.mdindex 702a1c4c29..08485a95f5 100644--- a/tools/completion/README.md+++ b/tools/completion/README.md@@ -138,9 +138,6 @@ llama-completion.exe -m models\gemma-1.1-7b-it.Q4_K_M.gguf --ignore-eos -n -1| `-dt, --defrag-thold N` | KV cache defragmentation threshold (DEPRECATED)<br/>(env: LLAMA_ARG_DEFRAG_THOLD) || `-np, --parallel N` | number of parallel sequences to decode (default: 1)<br/>(env: LLAMA_ARG_N_PARALLEL) || `--rpc SERVERS` | comma-separated list of RPC servers (host:port)<br/>(env: LLAMA_ARG_RPC) |-| `--mlock` | DEPRECATED in favor of `--load-mode`: force system to keep model in RAM rather than swapping or compressing<br/>(env: LLAMA_ARG_MLOCK) |-| `--mmap, --no-mmap` | DEPRECATED in favor of `--load-mode`: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>(env: LLAMA_ARG_MMAP) |-| `-dio, --direct-io, -ndio, --no-direct-io` | DEPRECATED in favor of `--load-mode`: use DirectIO if available<br/>(env: LLAMA_ARG_DIO) || `-lm, --load-mode MODE` | model loading mode (default: auto)<br/>- auto: mmap, unless a device does not support it<br/>- none: no special loading mode<br/>- mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>- mlock: force system to keep model in RAM rather than swapping or compressing<br/>- mmap+mlock: mmap + force system to keep model in RAM rather than swapping or compressing<br/>- dio: use DirectIO if available<br/><br/>(env: LLAMA_ARG_LOAD_MODE) || `-lzm, --lazy-mode MODE` | on-demand reading of certain tensors, for example per-layer embeddings (default: auto)<br/>- on: read the rows of such tensors from disk on demand instead of keeping them resident (requires mmap)<br/>- auto: on, but only for tensors larger than 4 GiB<br/>- off: always keep them resident<br/>(env: LLAMA_ARG_LAZY_MODE) || `--numa TYPE` | attempt optimizations that help on some NUMA systems<br/>- distribute: spread execution evenly over all nodes<br/>- isolate: only spawn threads on CPUs on the node that execution started on<br/>- numactl: use the CPU map provided by numactl<br/>if run without this previously, it is recommended to drop the system page cache before using this<br/>see https://github.com/ggml-org/llama.cpp/issues/1437<br/>(env: LLAMA_ARG_NUMA) |diff --git a/tools/llama-bench/llama-bench.cpp b/tools/llama-bench/llama-bench.cppindex 1fff21f701..17adda3809 100644--- a/tools/llama-bench/llama-bench.cpp+++ b/tools/llama-bench/llama-bench.cpp@@ -476,8 +476,6 @@ static void print_usage(int /* argc */, char ** argv) {printf(" -dev, --device <dev0/dev1/...> (default: auto)\n");printf(" -lm, --load-mode <auto|none|mmap|mlock|mmap+mlock|dio> (default: %s)\n", join(transform_to_str(cmd_params_defaults.load_mode, llama_load_mode_name), ",").c_str());printf(" -lzm, --lazy-mode <on|auto|off> (default: %s)\n", join(transform_to_str(cmd_params_defaults.lazy_mode, lazy_mode_str), ",").c_str());- printf(" -mmp, --mmap <0|1> (DEPRECATED IN FAVOUR OF --load-mode)\n");- printf(" -dio, --direct-io <0|1> (DEPRECATED IN FAVOUR OF --load-mode)\n");printf(" -embd, --embeddings <0|1> (default: %s)\n", join(cmd_params_defaults.embeddings, ",").c_str());printf(" -ts, --tensor-split <ts0/ts1/..> (default: 0)\n");printf(" -ot --override-tensor <tensor name pattern>=<buffer type>;...\n");@@ -883,44 +881,6 @@ static cmd_params parse_cmd_params(int argc, char ** argv) {break;}params.flash_attn.insert(params.flash_attn.end(), types.begin(), types.end());- } else if (arg == "-mmp" || arg == "--mmap") {- if (++i >= argc) {- invalid_param = true;- break;- }- LOG_WRN("DEPRECATED: -mmp and --mmap are deprecated in favour of --load-mode. Please use --load-mode mmap instead.\n");- auto p = string_split<bool>(argv[i], split_delim);-- std::vector<llama_load_mode> modes;- for (const auto & m : p) {- llama_load_mode mode;- if (m) {- mode = LLAMA_LOAD_MODE_MMAP;- } else {- mode = LLAMA_LOAD_MODE_NONE;- }- modes.push_back(mode);- }- params.load_mode.insert(params.load_mode.end(), modes.begin(), modes.end());- } else if (arg == "-dio" || arg == "--direct-io") {- if (++i >= argc) {- invalid_param = true;- break;- }- LOG_WRN("DEPRECATED: -dio and --direct-io are deprecated in favour of --load-mode. Please use --load-mode dio instead.\n");- auto p = string_split<bool>(argv[i], split_delim);-- std::vector<llama_load_mode> modes;- for (const auto & m : p) {- llama_load_mode mode;- if (m) {- mode = LLAMA_LOAD_MODE_DIRECT_IO;- } else {- mode = LLAMA_LOAD_MODE_NONE;- }- modes.push_back(mode);- }- params.load_mode.insert(params.load_mode.end(), modes.begin(), modes.end());} else if (arg == "-embd" || arg == "--embeddings") {if (++i >= argc) {invalid_param = true;diff --git a/tools/server/README.md b/tools/server/README.mdindex 71ebb95434..1909076328 100644--- a/tools/server/README.md+++ b/tools/server/README.md@@ -72,9 +72,6 @@ For the full list of features, please refer to [server's changelog](https://gith| `-ctv, --cache-type-v TYPE` | KV cache data type for V<br/>allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1<br/>(default: f16)<br/>(env: LLAMA_ARG_CACHE_TYPE_V) || `-dt, --defrag-thold N` | KV cache defragmentation threshold (DEPRECATED)<br/>(env: LLAMA_ARG_DEFRAG_THOLD) || `--rpc SERVERS` | comma-separated list of RPC servers (host:port)<br/>(env: LLAMA_ARG_RPC) |-| `--mlock` | DEPRECATED in favor of `--load-mode`: force system to keep model in RAM rather than swapping or compressing<br/>(env: LLAMA_ARG_MLOCK) |-| `--mmap, --no-mmap` | DEPRECATED in favor of `--load-mode`: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>(env: LLAMA_ARG_MMAP) |-| `-dio, --direct-io, -ndio, --no-direct-io` | DEPRECATED in favor of `--load-mode`: use DirectIO if available<br/>(env: LLAMA_ARG_DIO) || `-lm, --load-mode MODE` | model loading mode (default: auto)<br/>- auto: mmap, unless a device does not support it<br/>- none: no special loading mode<br/>- mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>- mlock: force system to keep model in RAM rather than swapping or compressing<br/>- mmap+mlock: mmap + force system to keep model in RAM rather than swapping or compressing<br/>- dio: use DirectIO if available<br/><br/>(env: LLAMA_ARG_LOAD_MODE) || `-lzm, --lazy-mode MODE` | on-demand reading of certain tensors, for example per-layer embeddings (default: auto)<br/>- on: read the rows of such tensors from disk on demand instead of keeping them resident (requires mmap)<br/>- auto: on, but only for tensors larger than 4 GiB<br/>- off: always keep them resident<br/>(env: LLAMA_ARG_LAZY_MODE) || `--numa TYPE` | attempt optimizations that help on some NUMA systems<br/>- distribute: spread execution evenly over all nodes<br/>- isolate: only spawn threads on CPUs on the node that execution started on<br/>- numactl: use the CPU map provided by numactl<br/>if run without this previously, it is recommended to drop the system page cache before using this<br/>see https://github.com/ggml-org/llama.cpp/issues/1437<br/>(env: LLAMA_ARG_NUMA) |