Skip to content

Distribute and run LLMs with a single file.

docs: Add docs for cli args and tools (#1037)

angpt5e9b21948d3c
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
index fdb53a7b..e77848ca 100644
--- a/docs/SUMMARY.md
+++ b/docs/SUMMARY.md
@@ -16,6 +16,7 @@
## Reference
+* [CLI Arguments and Flags](cli_arguments.md)
* [Technical details](technical_details.md)
* [Security](security.md)
* [Supported Systems](support.md)
diff --git a/docs/cli_arguments.md b/docs/cli_arguments.md
new file mode 100644
index 00000000..55e2aa2a
--- /dev/null
+++ b/docs/cli_arguments.md
@@ -0,0 +1,189 @@
+# CLI Arguments and Flags
+
+llamafile accepts two layers of command-line options:
+
+1. Wrapper flags added by llamafile itself, such as `--server`, `--chat`, `--cli`, and `--gpu`.
+2. The bundled `llama.cpp` flags that are passed through to chat, CLI, and server mode.
+
+The examples on this page use `llamafile` as the executable name. This is the
+name of the standalone runtime distributed in the releases. A pre-built,
+model-bundled llamafile normally has a model-specific filename instead; use
+that filename in the same commands. For example, these commands display the
+same kind of help:
+
+```sh
+llamafile --help
+./Qwen3.5-0.8B-Q8_0.llamafile --help
+```
+
+The `./` prefix runs a downloaded file from the current directory on macOS,
+Linux, and BSD. On Windows, use the filename after adding the `.exe` suffix.
+
+## Discovering Available Options
+
+Command-line options are scoped by execution mode rather than collected in a
+single mode-independent list. The complete user-facing interface is the union
+of the general and mode-specific help views:
+
+```sh
+llamafile --help
+llamafile --server --help
+llamafile --chat --help
+llamafile --cli --help
+```
+
+The top-level command is the reference for general llamafile wrapper flags.
+Each mode-specific command selects the matching `llama.cpp` parser and adds
+that mode's wrapper flags:
+
+| Command | Options shown |
+| --- | --- |
+| `llamafile --help` | Default combined-mode options and general llamafile wrapper flags. |
+| `llamafile --server --help` | HTTP server, API, Web UI, slot, and server sandbox options. |
+| `llamafile --chat --help` | Interactive chat, conversation, and multimodal options. |
+| `llamafile --cli --help` | Single-prompt, output, sampling, and multimodal options. |
+
+For fuller descriptions and upstream examples, see the llama.cpp
+[CLI argument reference](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md#usage)
+and [server argument reference](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#usage).
+Those pages track the latest llama.cpp development version, so an option may
+differ from the version bundled in a particular llamafile. The help output
+from that llamafile is authoritative for its accepted options.
+
+An option listed for one mode is not necessarily available in another. For
+example, `--host`, `--port`, and `--slot-save-path` configure the server,
+while `--nothink` is a CLI-mode option.
+
+Since v0.10.4, each `llama.cpp` option list is generated by the same parser
+that accepts arguments for that mode. This keeps accepted options and their
+help entries in sync. Llamafile-specific wrapper options appear in the
+top-level help or in the relevant mode introduction.
+
+Releases before v0.10.4 printed abbreviated hand-written help for the default,
+chat, and CLI modes. In those versions, `--server --help` provides the broadest
+option list, although it also contains server-only options.
+
+### Additional Accepted Parameter Spellings
+
+The `--gpu MODE` help uses the canonical backend names but does not spell out
+all accepted compatibility aliases. Values are case-insensitive:
+
+| Backend | Accepted `MODE` values |
+| --- | --- |
+| Automatic selection | `auto` |
+| NVIDIA CUDA | `nvidia`, `cublas` |
+| AMD ROCm/HIP | `amd`, `rocm`, `rocblas`, `hip` |
+| Apple Metal | `apple`, `metal` |
+| Vulkan | `vulkan`, `vk` |
+| CPU only | `disable`, `disabled` |
+
+Prefer the first value in each row in scripts. The aliases are retained for
+compatibility.
+
+## Commonly Used Flags
+
+| Flag | What it does |
+| --- | --- |
+| `-t, --threads N` | Number of CPU threads to use during generation. |
+| `-tb, --threads-batch N` | Number of CPU threads to use during prompt and batch processing. |
+| `-c, --ctx-size N` | Context window size. `0` means to use the model default. |
+| `-b, --batch-size N` | Logical maximum batch size. |
+| `-ub, --ubatch-size N` | Physical maximum batch size. |
+| `--mlock` | Keep the model in RAM instead of letting the OS swap or compress it. |
+| `--repeat-penalty N` | Penalize repeating tokens during sampling. `1.0` disables the penalty. |
+| `-ngl, --gpu-layers, --n-gpu-layers N` | Number of layers to offload to GPU. |
+| `--host HOST` | Server bind address. |
+| `--port PORT` | Server listen port. |
+
+## Wrapper and Mode Flags
+
+- Mode selection: `--server`, `--chat`, `--cli`
+- General wrapper flags: `--gpu MODE`, `--unsecure`, `--version`, `--help`
+- Chat-only wrapper flags: `--nologo`, `--ascii`
+- CLI-only wrapper flags: `--nothink`
+- Server-only wrapper flag: `--confine-reads`
+
+`--unsecure` disables the pledge/SECCOMP sandbox in modes where that sandbox
+is active. `--confine-reads` adds filesystem read confinement and only applies
+to `--server` mode.
+
+Model, prompt, GPU-layer, and logging options such as `-m`, `-p`, `-ngl`, and
+`--verbose` come from the bundled `llama.cpp` parser rather than the llamafile
+wrapper.
+
+## Shared Flags Accepted by Chat, CLI, and Server
+
+### General, CPU, Context, and Batching
+
+- General help and shell integration: `-h`, `--help`, `--usage`, `--version`, `-cl`, `--cache-list`, `--completion-bash`, `--verbose-prompt`
+- CPU scheduling and affinity: `-t`, `--threads`, `-tb`, `--threads-batch`, `-C`, `--cpu-mask`, `-Cr`, `--cpu-range`, `--cpu-strict`, `--prio`, `--poll`, `-Cb`, `--cpu-mask-batch`, `-Crb`, `--cpu-range-batch`, `--cpu-strict-batch`, `--prio-batch`, `--poll-batch`
+- Context and batching: `-c`, `--ctx-size`, `-n`, `--predict`, `--n-predict`, `-b`, `--batch-size`, `-ub`, `--ubatch-size`, `--keep`, `--swa-full`, `-fa`, `--flash-attn`
+- Prompt input: `-p`, `--prompt`, `-f`, `--file`, `-bf`, `--binary-file`, `-e`, `--escape`, `--no-escape`
+- RoPE and YaRN: `--rope-scaling`, `--rope-scale`, `--rope-freq-base`, `--rope-freq-scale`, `--yarn-orig-ctx`, `--yarn-ext-factor`, `--yarn-attn-factor`, `--yarn-beta-slow`, `--yarn-beta-fast`
+
+### Memory, KV Cache, and Offload
+
+- KV cache and host memory: `-kvo`, `--kv-offload`, `-nkvo`, `--no-kv-offload`, `--repack`, `-nr`, `--no-repack`, `--no-host`
+- Cache storage and defragmentation: `-ctk`, `--cache-type-k`, `-ctv`, `--cache-type-v`, `-dt`, `--defrag-thold`
+- Memory mapping and residency: `--mlock`, `--mmap`, `--no-mmap`, `--numa`, `--check-tensors`, `--op-offload`, `--no-op-offload`
+- Draft-model cache types: `-ctkd`, `--cache-type-k-draft`, `-ctvd`, `--cache-type-v-draft`
+
+### Devices, GPU, Tensors, and Adapters
+
+- Devices and tensor placement: `-dev`, `--device`, `--list-devices`, `-ot`, `--override-tensor`
+- CPU MoE controls: `-cmoe`, `--cpu-moe`, `-ncmoe`, `--n-cpu-moe`
+- GPU offload controls: `-ngl`, `--gpu-layers`, `--n-gpu-layers`, `-sm`, `--split-mode`, `-ts`, `--tensor-split`, `-mg`, `--main-gpu`
+- Automatic fitting: `-fit`, `--fit`, `-fitt`, `--fit-target`, `-fitc`, `--fit-ctx`
+- Adapters and metadata overrides: `--lora`, `--lora-scaled`, `--control-vector`, `--control-vector-scaled`, `--control-vector-layer-range`, `--override-kv`
+
+### Model Selection, Downloads, and Logging
+
+- Model path and downloads: `-m`, `--model`, `-mu`, `--model-url`, `-dr`, `--docker-repo`
+- Hugging Face and related model selectors: `-hf`, `-hfr`, `--hf-repo`, `-hfd`, `-hfrd`, `--hf-repo-draft`, `-hff`, `--hf-file`, `-hfv`, `-hfrv`, `--hf-repo-v`, `-hffv`, `--hf-file-v`, `-hft`, `--hf-token`
+- Logging and diagnostics: `--log-disable`, `--log-file`, `--log-colors`, `-v`, `--verbose`, `--log-verbose`, `--offline`, `-lv`, `--verbosity`, `--log-verbosity`, `--log-prefix`, `--log-timestamps`, `--perf`, `--no-perf`
+
+## Sampling Flags
+
+- Sampling order and randomness: `--samplers`, `-s`, `--seed`, `--sampler-seq`, `--sampling-seq`, `--ignore-eos`
+- Core sampling controls: `--temp`, `--top-k`, `--top-p`, `--min-p`, `--top-nsigma`, `--xtc-probability`, `--xtc-threshold`, `--typical`
+- Repetition controls: `--repeat-last-n`, `--repeat-penalty`, `--presence-penalty`, `--frequency-penalty`
+- DRY and dynamic temperature: `--dry-multiplier`, `--dry-base`, `--dry-allowed-length`, `--dry-penalty-last-n`, `--dry-sequence-breaker`, `--dynatemp-range`, `--dynatemp-exp`
+- Mirostat and constrained output: `--mirostat`, `--mirostat-lr`, `--mirostat-ent`, `-l`, `--logit-bias`, `--grammar`, `--grammar-file`, `-j`, `--json-schema`, `-jf`, `--json-schema-file`
+
+## CLI and Chat-Specific Flags
+
+- Output and interaction: `--display-prompt`, `--no-display-prompt`, `-co`, `--color`, `--show-timings`, `--no-show-timings`, `-cnv`, `--conversation`, `-no-cnv`, `--no-conversation`, `-st`, `--single-turn`, `-mli`, `--multiline-input`, `--simple-io`
+- System prompt and stopping: `-sys`, `--system-prompt`, `-sysf`, `--system-prompt-file`, `-r`, `--reverse-prompt`, `-sp`, `--special`
+- Context and cache management: `--ctx-checkpoints`, `--swa-checkpoints`, `-cram`, `--cache-ram`, `--context-shift`, `--no-context-shift`, `--warmup`, `--no-warmup`
+- Parallel decode and multimodal input: `-np`, `--parallel`, `-mm`, `--mmproj`, `-mmu`, `--mmproj-url`, `--mmproj-auto`, `--no-mmproj`, `--no-mmproj-auto`, `--mmproj-offload`, `--no-mmproj-offload`, `--image`, `--audio`, `--image-min-tokens`, `--image-max-tokens`
+- Draft and speculative decoding: `-otd`, `--override-tensor-draft`, `-cmoed`, `--cpu-moe-draft`, `-ncmoed`, `--n-cpu-moe-draft`, `--draft`, `--draft-n`, `--draft-max`, `--draft-min`, `--draft-n-min`, `--draft-p-min`, `-cd`, `--ctx-size-draft`, `-devd`, `--device-draft`, `-ngld`, `--gpu-layers-draft`, `--n-gpu-layers-draft`, `-md`, `--model-draft`, `--spec-replace`
+- Chat templates and reasoning controls: `--chat-template-kwargs`, `--jinja`, `--no-jinja`, `--reasoning-format`, `--reasoning-budget`, `--chat-template`, `--chat-template-file`
+- Built-in defaults: `--gpt-oss-20b-default`, `--gpt-oss-120b-default`, `--vision-gemma-4b-default`, `--vision-gemma-12b-default`
+
+`-np, --parallel` in CLI and chat mode controls the number of parallel sequences to decode.
+
+## Server-Only Flags
+
+- Server runtime and batching: `-kvu`, `--kv-unified`, `--spm-infill`, `--pooling`, `-np`, `--parallel`, `-cb`, `--cont-batching`, `-nocb`, `--no-cont-batching`, `--threads-http`, `--cache-reuse`
+- Network and API configuration: `-a`, `--alias`, `--host`, `--port`, `--path`, `--api-prefix`, `--api-key`, `--api-key-file`, `-to`, `--timeout`
+- Web UI and monitoring: `--webui-config`, `--webui-config-file`, `--webui`, `--no-webui`, `--metrics`, `--props`, `--slots`, `--no-slots`, `--slot-save-path`
+- Built-in tools and agent mode: `--tools`, `-ag`, `--agent`, `-no-ag`, `--no-agent`, `--ui-mcp-proxy`, `--webui-mcp-proxy`, `--no-ui-mcp-proxy`, `--no-webui-mcp-proxy`
+- Router and media: `--media-path`, `--models-dir`, `--models-preset`, `--models-max`, `--models-autoload`, `--no-models-autoload`
+- Embeddings, reranking, and TLS: `--embedding`, `--embeddings`, `--rerank`, `--reranking`, `--ssl-key-file`, `--ssl-cert-file`
+- Chat templating and slot behavior: `--chat-template-kwargs`, `--jinja`, `--no-jinja`, `--reasoning-format`, `--reasoning-budget`, `--chat-template`, `--chat-template-file`, `--prefill-assistant`, `--no-prefill-assistant`, `-sps`, `--slot-prompt-similarity`, `--lora-init-without-apply`, `--sleep-idle-seconds`
+- Draft, speculative decoding, and TTS: `-td`, `--threads-draft`, `-tbd`, `--threads-batch-draft`, `--draft`, `--draft-n`, `--draft-max`, `--draft-min`, `--draft-n-min`, `--draft-p-min`, `-cd`, `--ctx-size-draft`, `-devd`, `--device-draft`, `-ngld`, `--gpu-layers-draft`, `--n-gpu-layers-draft`, `-md`, `--model-draft`, `--spec-replace`, `-mv`, `--model-vocoder`, `--tts-use-guide-tokens`
+- Built-in server defaults: `--embd-gemma-default`, `--fim-qwen-1.5b-default`, `--fim-qwen-3b-default`, `--fim-qwen-7b-default`, `--fim-qwen-7b-spec`, `--fim-qwen-14b-spec`, `--fim-qwen-30b-default`, `--gpt-oss-20b-default`, `--gpt-oss-120b-default`, `--vision-gemma-4b-default`, `--vision-gemma-12b-default`
+
+`-np, --parallel` in server mode controls the number of server slots rather than the number of parallel decode sequences.
+
+The Web UI is inherited from llama.cpp and may refer to its upstream server
+executable as `llama-server`. In those instructions, use the name of the
+llamafile executable instead. For example, `llama-server --tools all` becomes
+`./ModelName.llamafile --server --tools all` for a model-bundled llamafile.
+
+## Notes
+
+- The lists above are a readable index, not a second source of truth. The help
+ output from your executable reflects the exact bundled `llama.cpp` version.
+- If you are unsure whether a flag is shared, CLI-only, or server-only, check
+ the help for that mode rather than relying on another mode's option list.
diff --git a/docs/running_llamafile.md b/docs/running_llamafile.md
index 4631316d..96b463d2 100644
--- a/docs/running_llamafile.md
+++ b/docs/running_llamafile.md
@@ -14,6 +14,9 @@ llamafile -m Apertus-8B-Instruct-2509.gguf --temp ...
./Apertus-8B-Instruct-2509.llamafile --temp ...
```
+For a full reference of wrapper flags and pass-through `llama.cpp` options, see
+[CLI Arguments and Flags](cli_arguments.md).
+
## Running llamafile in CLI mode
@@ -95,6 +98,31 @@ large enough (but still fits your memory). For instance:
--ctx-size 64000
```
+### Built-in tools
+
+The server Web UI is inherited from llama.cpp, so some messages in the UI use
+the upstream executable name `llama-server`. When following those messages,
+replace `llama-server` with the standalone runtime or the filename of the
+model-bundled llamafile you are running. For example, these commands enable
+the same built-in tools:
+
+```sh
+llamafile -m model.gguf --server --tools all
+./ModelName.llamafile --server --tools all
+```
+
+`--tools all` enables every tool provided by the bundled server. A
+comma-separated list enables only selected tools, such as
+`--tools read_file,file_glob_search,grep_search`. The exact tool names for a
+build are listed by `--server --help`.
+
+Built-in tools may access local files with the
+permissions of the llamafile process. Enable only the tools you need, do not
+enable them in an untrusted environment, and review the
+[sandbox limitations](security.md#when-the-sandbox-is-relaxed-or-skipped).
+For security, the server restricts allowed browser origins to localhost by
+default when built-in tools are enabled.
+
## Running llamafile in combined mode
Combined mode is the default for the last generation of llamafiles: when you
@@ -154,4 +182,3 @@ llamafile -ngl 9999 --temp 0 \
sed -e's/ /_/g' -e's/$/.jpg/'
a_baby_monkey_on_the_back_of_a_mother.jpg
```
-