Distribute and run LLMs with a single file.
C++ cross-platformggufllama-cpplocal-ailocal-inferencelocal-llmopen-source-aisingle-file-executablespeech-to-text
docs: Add docs for cli args and tools (#1037)
angpt5e9b21948d3c
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.mdindex fdb53a7b..e77848ca 100644--- a/docs/SUMMARY.md+++ b/docs/SUMMARY.md@@ -16,6 +16,7 @@## Reference+* [CLI Arguments and Flags](cli_arguments.md)* [Technical details](technical_details.md)* [Security](security.md)* [Supported Systems](support.md)diff --git a/docs/cli_arguments.md b/docs/cli_arguments.mdnew file mode 100644index 00000000..55e2aa2a--- /dev/null+++ b/docs/cli_arguments.md@@ -0,0 +1,189 @@+# CLI Arguments and Flags++llamafile accepts two layers of command-line options:++1. Wrapper flags added by llamafile itself, such as `--server`, `--chat`, `--cli`, and `--gpu`.+2. The bundled `llama.cpp` flags that are passed through to chat, CLI, and server mode.++The examples on this page use `llamafile` as the executable name. This is the+name of the standalone runtime distributed in the releases. A pre-built,+model-bundled llamafile normally has a model-specific filename instead; use+that filename in the same commands. For example, these commands display the+same kind of help:++```sh+llamafile --help+./Qwen3.5-0.8B-Q8_0.llamafile --help+```++The `./` prefix runs a downloaded file from the current directory on macOS,+Linux, and BSD. On Windows, use the filename after adding the `.exe` suffix.++## Discovering Available Options++Command-line options are scoped by execution mode rather than collected in a+single mode-independent list. The complete user-facing interface is the union+of the general and mode-specific help views:++```sh+llamafile --help+llamafile --server --help+llamafile --chat --help+llamafile --cli --help+```++The top-level command is the reference for general llamafile wrapper flags.+Each mode-specific command selects the matching `llama.cpp` parser and adds+that mode's wrapper flags:++| Command | Options shown |+| --- | --- |+| `llamafile --help` | Default combined-mode options and general llamafile wrapper flags. |+| `llamafile --server --help` | HTTP server, API, Web UI, slot, and server sandbox options. |+| `llamafile --chat --help` | Interactive chat, conversation, and multimodal options. |+| `llamafile --cli --help` | Single-prompt, output, sampling, and multimodal options. |++For fuller descriptions and upstream examples, see the llama.cpp+[CLI argument reference](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md#usage)+and [server argument reference](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#usage).+Those pages track the latest llama.cpp development version, so an option may+differ from the version bundled in a particular llamafile. The help output+from that llamafile is authoritative for its accepted options.++An option listed for one mode is not necessarily available in another. For+example, `--host`, `--port`, and `--slot-save-path` configure the server,+while `--nothink` is a CLI-mode option.++Since v0.10.4, each `llama.cpp` option list is generated by the same parser+that accepts arguments for that mode. This keeps accepted options and their+help entries in sync. Llamafile-specific wrapper options appear in the+top-level help or in the relevant mode introduction.++Releases before v0.10.4 printed abbreviated hand-written help for the default,+chat, and CLI modes. In those versions, `--server --help` provides the broadest+option list, although it also contains server-only options.++### Additional Accepted Parameter Spellings++The `--gpu MODE` help uses the canonical backend names but does not spell out+all accepted compatibility aliases. Values are case-insensitive:++| Backend | Accepted `MODE` values |+| --- | --- |+| Automatic selection | `auto` |+| NVIDIA CUDA | `nvidia`, `cublas` |+| AMD ROCm/HIP | `amd`, `rocm`, `rocblas`, `hip` |+| Apple Metal | `apple`, `metal` |+| Vulkan | `vulkan`, `vk` |+| CPU only | `disable`, `disabled` |++Prefer the first value in each row in scripts. The aliases are retained for+compatibility.++## Commonly Used Flags++| Flag | What it does |+| --- | --- |+| `-t, --threads N` | Number of CPU threads to use during generation. |+| `-tb, --threads-batch N` | Number of CPU threads to use during prompt and batch processing. |+| `-c, --ctx-size N` | Context window size. `0` means to use the model default. |+| `-b, --batch-size N` | Logical maximum batch size. |+| `-ub, --ubatch-size N` | Physical maximum batch size. |+| `--mlock` | Keep the model in RAM instead of letting the OS swap or compress it. |+| `--repeat-penalty N` | Penalize repeating tokens during sampling. `1.0` disables the penalty. |+| `-ngl, --gpu-layers, --n-gpu-layers N` | Number of layers to offload to GPU. |+| `--host HOST` | Server bind address. |+| `--port PORT` | Server listen port. |++## Wrapper and Mode Flags++- Mode selection: `--server`, `--chat`, `--cli`+- General wrapper flags: `--gpu MODE`, `--unsecure`, `--version`, `--help`+- Chat-only wrapper flags: `--nologo`, `--ascii`+- CLI-only wrapper flags: `--nothink`+- Server-only wrapper flag: `--confine-reads`++`--unsecure` disables the pledge/SECCOMP sandbox in modes where that sandbox+is active. `--confine-reads` adds filesystem read confinement and only applies+to `--server` mode.++Model, prompt, GPU-layer, and logging options such as `-m`, `-p`, `-ngl`, and+`--verbose` come from the bundled `llama.cpp` parser rather than the llamafile+wrapper.++## Shared Flags Accepted by Chat, CLI, and Server++### General, CPU, Context, and Batching++- General help and shell integration: `-h`, `--help`, `--usage`, `--version`, `-cl`, `--cache-list`, `--completion-bash`, `--verbose-prompt`+- CPU scheduling and affinity: `-t`, `--threads`, `-tb`, `--threads-batch`, `-C`, `--cpu-mask`, `-Cr`, `--cpu-range`, `--cpu-strict`, `--prio`, `--poll`, `-Cb`, `--cpu-mask-batch`, `-Crb`, `--cpu-range-batch`, `--cpu-strict-batch`, `--prio-batch`, `--poll-batch`+- Context and batching: `-c`, `--ctx-size`, `-n`, `--predict`, `--n-predict`, `-b`, `--batch-size`, `-ub`, `--ubatch-size`, `--keep`, `--swa-full`, `-fa`, `--flash-attn`+- Prompt input: `-p`, `--prompt`, `-f`, `--file`, `-bf`, `--binary-file`, `-e`, `--escape`, `--no-escape`+- RoPE and YaRN: `--rope-scaling`, `--rope-scale`, `--rope-freq-base`, `--rope-freq-scale`, `--yarn-orig-ctx`, `--yarn-ext-factor`, `--yarn-attn-factor`, `--yarn-beta-slow`, `--yarn-beta-fast`++### Memory, KV Cache, and Offload++- KV cache and host memory: `-kvo`, `--kv-offload`, `-nkvo`, `--no-kv-offload`, `--repack`, `-nr`, `--no-repack`, `--no-host`+- Cache storage and defragmentation: `-ctk`, `--cache-type-k`, `-ctv`, `--cache-type-v`, `-dt`, `--defrag-thold`+- Memory mapping and residency: `--mlock`, `--mmap`, `--no-mmap`, `--numa`, `--check-tensors`, `--op-offload`, `--no-op-offload`+- Draft-model cache types: `-ctkd`, `--cache-type-k-draft`, `-ctvd`, `--cache-type-v-draft`++### Devices, GPU, Tensors, and Adapters++- Devices and tensor placement: `-dev`, `--device`, `--list-devices`, `-ot`, `--override-tensor`+- CPU MoE controls: `-cmoe`, `--cpu-moe`, `-ncmoe`, `--n-cpu-moe`+- GPU offload controls: `-ngl`, `--gpu-layers`, `--n-gpu-layers`, `-sm`, `--split-mode`, `-ts`, `--tensor-split`, `-mg`, `--main-gpu`+- Automatic fitting: `-fit`, `--fit`, `-fitt`, `--fit-target`, `-fitc`, `--fit-ctx`+- Adapters and metadata overrides: `--lora`, `--lora-scaled`, `--control-vector`, `--control-vector-scaled`, `--control-vector-layer-range`, `--override-kv`++### Model Selection, Downloads, and Logging++- Model path and downloads: `-m`, `--model`, `-mu`, `--model-url`, `-dr`, `--docker-repo`+- Hugging Face and related model selectors: `-hf`, `-hfr`, `--hf-repo`, `-hfd`, `-hfrd`, `--hf-repo-draft`, `-hff`, `--hf-file`, `-hfv`, `-hfrv`, `--hf-repo-v`, `-hffv`, `--hf-file-v`, `-hft`, `--hf-token`+- Logging and diagnostics: `--log-disable`, `--log-file`, `--log-colors`, `-v`, `--verbose`, `--log-verbose`, `--offline`, `-lv`, `--verbosity`, `--log-verbosity`, `--log-prefix`, `--log-timestamps`, `--perf`, `--no-perf`++## Sampling Flags++- Sampling order and randomness: `--samplers`, `-s`, `--seed`, `--sampler-seq`, `--sampling-seq`, `--ignore-eos`+- Core sampling controls: `--temp`, `--top-k`, `--top-p`, `--min-p`, `--top-nsigma`, `--xtc-probability`, `--xtc-threshold`, `--typical`+- Repetition controls: `--repeat-last-n`, `--repeat-penalty`, `--presence-penalty`, `--frequency-penalty`+- DRY and dynamic temperature: `--dry-multiplier`, `--dry-base`, `--dry-allowed-length`, `--dry-penalty-last-n`, `--dry-sequence-breaker`, `--dynatemp-range`, `--dynatemp-exp`+- Mirostat and constrained output: `--mirostat`, `--mirostat-lr`, `--mirostat-ent`, `-l`, `--logit-bias`, `--grammar`, `--grammar-file`, `-j`, `--json-schema`, `-jf`, `--json-schema-file`++## CLI and Chat-Specific Flags++- Output and interaction: `--display-prompt`, `--no-display-prompt`, `-co`, `--color`, `--show-timings`, `--no-show-timings`, `-cnv`, `--conversation`, `-no-cnv`, `--no-conversation`, `-st`, `--single-turn`, `-mli`, `--multiline-input`, `--simple-io`+- System prompt and stopping: `-sys`, `--system-prompt`, `-sysf`, `--system-prompt-file`, `-r`, `--reverse-prompt`, `-sp`, `--special`+- Context and cache management: `--ctx-checkpoints`, `--swa-checkpoints`, `-cram`, `--cache-ram`, `--context-shift`, `--no-context-shift`, `--warmup`, `--no-warmup`+- Parallel decode and multimodal input: `-np`, `--parallel`, `-mm`, `--mmproj`, `-mmu`, `--mmproj-url`, `--mmproj-auto`, `--no-mmproj`, `--no-mmproj-auto`, `--mmproj-offload`, `--no-mmproj-offload`, `--image`, `--audio`, `--image-min-tokens`, `--image-max-tokens`+- Draft and speculative decoding: `-otd`, `--override-tensor-draft`, `-cmoed`, `--cpu-moe-draft`, `-ncmoed`, `--n-cpu-moe-draft`, `--draft`, `--draft-n`, `--draft-max`, `--draft-min`, `--draft-n-min`, `--draft-p-min`, `-cd`, `--ctx-size-draft`, `-devd`, `--device-draft`, `-ngld`, `--gpu-layers-draft`, `--n-gpu-layers-draft`, `-md`, `--model-draft`, `--spec-replace`+- Chat templates and reasoning controls: `--chat-template-kwargs`, `--jinja`, `--no-jinja`, `--reasoning-format`, `--reasoning-budget`, `--chat-template`, `--chat-template-file`+- Built-in defaults: `--gpt-oss-20b-default`, `--gpt-oss-120b-default`, `--vision-gemma-4b-default`, `--vision-gemma-12b-default`++`-np, --parallel` in CLI and chat mode controls the number of parallel sequences to decode.++## Server-Only Flags++- Server runtime and batching: `-kvu`, `--kv-unified`, `--spm-infill`, `--pooling`, `-np`, `--parallel`, `-cb`, `--cont-batching`, `-nocb`, `--no-cont-batching`, `--threads-http`, `--cache-reuse`+- Network and API configuration: `-a`, `--alias`, `--host`, `--port`, `--path`, `--api-prefix`, `--api-key`, `--api-key-file`, `-to`, `--timeout`+- Web UI and monitoring: `--webui-config`, `--webui-config-file`, `--webui`, `--no-webui`, `--metrics`, `--props`, `--slots`, `--no-slots`, `--slot-save-path`+- Built-in tools and agent mode: `--tools`, `-ag`, `--agent`, `-no-ag`, `--no-agent`, `--ui-mcp-proxy`, `--webui-mcp-proxy`, `--no-ui-mcp-proxy`, `--no-webui-mcp-proxy`+- Router and media: `--media-path`, `--models-dir`, `--models-preset`, `--models-max`, `--models-autoload`, `--no-models-autoload`+- Embeddings, reranking, and TLS: `--embedding`, `--embeddings`, `--rerank`, `--reranking`, `--ssl-key-file`, `--ssl-cert-file`+- Chat templating and slot behavior: `--chat-template-kwargs`, `--jinja`, `--no-jinja`, `--reasoning-format`, `--reasoning-budget`, `--chat-template`, `--chat-template-file`, `--prefill-assistant`, `--no-prefill-assistant`, `-sps`, `--slot-prompt-similarity`, `--lora-init-without-apply`, `--sleep-idle-seconds`+- Draft, speculative decoding, and TTS: `-td`, `--threads-draft`, `-tbd`, `--threads-batch-draft`, `--draft`, `--draft-n`, `--draft-max`, `--draft-min`, `--draft-n-min`, `--draft-p-min`, `-cd`, `--ctx-size-draft`, `-devd`, `--device-draft`, `-ngld`, `--gpu-layers-draft`, `--n-gpu-layers-draft`, `-md`, `--model-draft`, `--spec-replace`, `-mv`, `--model-vocoder`, `--tts-use-guide-tokens`+- Built-in server defaults: `--embd-gemma-default`, `--fim-qwen-1.5b-default`, `--fim-qwen-3b-default`, `--fim-qwen-7b-default`, `--fim-qwen-7b-spec`, `--fim-qwen-14b-spec`, `--fim-qwen-30b-default`, `--gpt-oss-20b-default`, `--gpt-oss-120b-default`, `--vision-gemma-4b-default`, `--vision-gemma-12b-default`++`-np, --parallel` in server mode controls the number of server slots rather than the number of parallel decode sequences.++The Web UI is inherited from llama.cpp and may refer to its upstream server+executable as `llama-server`. In those instructions, use the name of the+llamafile executable instead. For example, `llama-server --tools all` becomes+`./ModelName.llamafile --server --tools all` for a model-bundled llamafile.++## Notes++- The lists above are a readable index, not a second source of truth. The help+ output from your executable reflects the exact bundled `llama.cpp` version.+- If you are unsure whether a flag is shared, CLI-only, or server-only, check+ the help for that mode rather than relying on another mode's option list.diff --git a/docs/running_llamafile.md b/docs/running_llamafile.mdindex 4631316d..96b463d2 100644--- a/docs/running_llamafile.md+++ b/docs/running_llamafile.md@@ -14,6 +14,9 @@ llamafile -m Apertus-8B-Instruct-2509.gguf --temp ..../Apertus-8B-Instruct-2509.llamafile --temp ...```+For a full reference of wrapper flags and pass-through `llama.cpp` options, see+[CLI Arguments and Flags](cli_arguments.md).+## Running llamafile in CLI mode@@ -95,6 +98,31 @@ large enough (but still fits your memory). For instance:--ctx-size 64000```+### Built-in tools++The server Web UI is inherited from llama.cpp, so some messages in the UI use+the upstream executable name `llama-server`. When following those messages,+replace `llama-server` with the standalone runtime or the filename of the+model-bundled llamafile you are running. For example, these commands enable+the same built-in tools:++```sh+llamafile -m model.gguf --server --tools all+./ModelName.llamafile --server --tools all+```++`--tools all` enables every tool provided by the bundled server. A+comma-separated list enables only selected tools, such as+`--tools read_file,file_glob_search,grep_search`. The exact tool names for a+build are listed by `--server --help`.++Built-in tools may access local files with the+permissions of the llamafile process. Enable only the tools you need, do not+enable them in an untrusted environment, and review the+[sandbox limitations](security.md#when-the-sandbox-is-relaxed-or-skipped).+For security, the server restricts allowed browser origins to localhost by+default when built-in tools are enabled.+## Running llamafile in combined modeCombined mode is the default for the last generation of llamafiles: when you@@ -154,4 +182,3 @@ llamafile -ngl 9999 --temp 0 \sed -e's/ /_/g' -e's/$/.jpg/'a_baby_monkey_on_the_back_of_a_mother.jpg```-