Skip to content

Distribute and run LLMs with a single file.

Bring back pledge/SECCOMP sandboxing (#930) (#1015)

Davide Eynard8b7081179a07
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
index 091b3386..fdb53a7b 100644
--- a/docs/SUMMARY.md
+++ b/docs/SUMMARY.md
@@ -17,6 +17,7 @@
## Reference
* [Technical details](technical_details.md)
+* [Security](security.md)
* [Supported Systems](support.md)
* [Troubleshooting](troubleshooting.md)
diff --git a/docs/security.md b/docs/security.md
new file mode 100644
index 00000000..e7516405
--- /dev/null
+++ b/docs/security.md
@@ -0,0 +1,133 @@
+# Security
+
+llamafile can sandbox itself with two [Cosmopolitan
+Libc](https://github.com/jart/cosmopolitan) primitives:
+
+- **[pledge()](https://man.openbsd.org/pledge.2)** restricts which system
+ calls the process may make. On Linux it installs a SECCOMP BPF filter;
+ on OpenBSD it calls the native `pledge(2)`. **On by default.**
+- **[unveil()](https://man.openbsd.org/unveil.2)** restricts which parts of
+ the filesystem the process can see. On Linux it uses the
+ [Landlock](https://docs.kernel.org/userspace-api/landlock.html) LSM
+ (kernel 5.13+); on OpenBSD it calls the native `unveil(2)`. **Opt-in**,
+ via `--confine-reads`.
+
+Neither needs any kernel configuration or privileges. SECCOMP filtering
+requires Linux 3.5+ on x86-64 and 5.13+ (for Landlock) on either
+architecture; on older kernels, and on macOS, Windows, and other BSDs, the
+calls are no-ops and llamafile logs that sandboxing is unavailable and keeps
+running. The pledge() sandbox can be turned off entirely with `--unsecure`.
+
+## The default: pledge()
+
+The `llamafile --server` process runs under `stdio anet rpath` on Linux
+(`stdio inet rpath` on OpenBSD). After startup this means:
+
+- **No outbound network.** `anet` allows `accept()` but not `connect()`, so
+ the only networking the server can do is answer connections it received.
+ If the server is ever compromised (say, through a bug in the GGUF parser
+ or an HTTP handler), this is the single most valuable restriction: it cuts
+ the attacker's ability to exfiltrate anything over the network.
+- **No writing, creating, deleting, executing, or forking.** A compromised
+ server can't modify your files, drop a payload, or launch a program.
+ (`--slot-save-path` and a prompt cache add write access to *their*
+ directories only.)
+- **Reads are allowed** (`rpath`) — anywhere the process could already read.
+ This is deliberate: a server routinely opens files whose paths it only
+ learns at request time (multimodal media, static assets), which cannot be
+ known when a path allow-list would have to be locked. If you want to
+ restrict *which* files are readable, see `--confine-reads` below.
+
+`--cli` runs under `stdio rpath tty` and `--chat` under
+`stdio rpath wpath cpath tty` — neither has any network access at all.
+
+## Opt-in read confinement: `--confine-reads`
+
+Passing `--confine-reads` to the server additionally applies `unveil()`,
+confining reads to the executable and the directories holding the weights
+(the model and its shards, `--mmproj`, LoRA adapters, a draft model, control
+vectors, `--media-path`, a static `--path` web root) plus the two
+name-resolution files a non-numeric `--host` needs. The rest of the
+filesystem — `/etc/passwd`, your SSH keys, other users' files — becomes
+invisible to the server even though it is world-readable.
+
+Two things to know before relying on it:
+
+- **It confines directories, not individual files.** The model's *parent
+ directory* is unveiled (so multi-part GGUF shards beside it load), which
+ means anything else in that directory is readable too. Put weights in a
+ dedicated directory if you don't want their neighbours exposed.
+- **It's incompatible with reading files by arbitrary path at request time.**
+ Because the readable set is locked at startup, a request that references a
+ file outside the unveiled directories will be denied. The built-in flows
+ (media under `--media-path`, static files under `--path`) are covered;
+ bespoke setups that read elsewhere are not.
+- **It requires a filesystem Landlock can govern.** On the handful it cannot
+ (some network mounts, virtiofs, 9p) llamafile keeps the pledge() sandbox
+ but skips confinement rather than refusing to load the model, and logs
+ that it did so.
+
+## When the sandbox is relaxed or skipped
+
+llamafile logs a notice in each case:
+
+- **Outbound features** (`--rpc` for distributed inference, server-side
+ tools, the MCP proxy) legitimately need to `connect()` out, so the
+ networking promise is relaxed from `anet` to `inet` for them. Writes and
+ exec stay blocked.
+- **GPU mode gets no sandbox.** GPU backends are loaded dynamically and their
+ drivers need device access (`ioctl`, `/dev/*`) that no promise set covers,
+ so the sandbox is skipped whenever a GPU backend is loaded. **This is the
+ common production case** — a GPU server is not sandboxed. Pass
+ `--gpu disable` to force CPU inference with the sandbox.
+- **Combined mode** (the default `llamafile -m model.gguf`, TUI + server in
+ one process) hosts an in-process HTTP client that must `connect()` to the
+ server, so it's skipped. Run `llamafile --server` for the sandboxed server.
+- Model downloads (`-hf`, `--model-url`) happen *before* the sandbox is
+ installed; once the download completes the process is sandboxed normally.
+
+## A note on the penalty mode
+
+On Linux, sandbox violations are configured to return `EPERM` (permission
+denied) rather than killing the process, so a blocked syscall surfaces as an
+ordinary I/O error. If a server refuses to start with a `pledge failed`
+error (an exotic kernel or an outer seccomp policy), `--unsecure` disables
+the sandbox. OpenBSD's native `pledge(2)` always terminates a violating
+process with `SIGABRT` instead; that behavior is not configurable there.
+
+## Verifying the sandbox
+
+Unit tests exercise every promise set llamafile uses, assert that allowed
+operations work while blocked ones fail, and check that `unveil()` confines
+reads to the unveiled directory (run on Linux):
+
+```sh
+.cosmocc/4.0.2/bin/make o//tests/sandbox_test && o//tests/sandbox_test
+```
+
+Integration tests verify the running server end-to-end — every thread of the
+server process carries the SECCOMP filter, completions still work inside the
+sandbox, a bundled `/zip/` llamafile loads under it, `--confine-reads`
+confines while the default does not, and `--unsecure` really disables it:
+
+```sh
+cd tests/integration
+./run_tests.sh --executable ../../o/llamafile/llamafile \
+ --model ../../models/TinyLLama-v0.1-5M-F16.gguf -m sandbox
+```
+
+You can also check a running server by hand:
+
+```sh
+llamafile --server -m model.gguf &
+grep Seccomp /proc/$!/status # "Seccomp: 2" means the filter is active
+```
+
+## Caveats
+
+Your llamafile is able to protect itself against the outside world, but that
+doesn't mean you're protected from llamafile. Sandboxing is self-imposed. If
+you obtained your llamafile from an untrusted source then its author could
+have simply modified it to not do that. In that case, you can run the
+untrusted llamafile inside another sandbox, such as a virtual machine, to
+make sure it behaves how you expect.
diff --git a/llama.cpp.patches/README.md b/llama.cpp.patches/README.md
index 1f96e6e5..d439b9d7 100644
--- a/llama.cpp.patches/README.md
+++ b/llama.cpp.patches/README.md
@@ -156,7 +156,7 @@ These patches integrate llamafile's file handling APIs for loading models from b
| Patch | Description |
|-------|-------------|
-| `tools_server_server.cpp.patch` | Renames upstream's `llama_server()` to `server_main()` and adds `on_ready`/`on_shutdown_available` callbacks for combined TUI+server mode; adds Metal/GPU backend trigger before `common_init()`; adds Cosmopolitan-specific standalone `main()` with `cosmo_args`, verbose flag handling, and GPU pre-initialization; handles `LLAMAFILE_TUI` exit to avoid Metal cleanup crashes |
+| `tools_server_server.cpp.patch` | Renames upstream's `llama_server()` to `server_main()` and adds `on_ready`/`on_shutdown_available` callbacks for combined TUI+server mode; adds Metal/GPU backend trigger before `common_init()`; installs the sandbox (`llamafile_sandbox_server()`, issue #930) — the mechanism lives in `llamafile/sandbox.c`; the patch fills a `llamafile_sandbox_spec` with the on-disk paths the loader opens after the lock (model, mmproj, media dir, LoRA, draft model, control vectors, public path as read; slot-save and prompt cache as read-write), detects outbound-needing features (`--rpc` via argv/env, MCP proxy, server tools) to relax `anet`→`inet`, passes `FLAG_confine_reads` for opt-in unveil(), quiesces the log worker (`common_log_pause`/`resume`, so no thread escapes the per-thread filter) around the call, before the HTTP listener spawns and before model load, and skips it in combined/GPU modes; adds Cosmopolitan-specific standalone `main()` with `cosmo_args`, verbose flag handling, `--unsecure` consumption (`llamafile_consume_flag`), and GPU pre-initialization; handles `LLAMAFILE_TUI` exit to avoid Metal cleanup crashes |
The web UI moved upstream from prebuilt `tools/server/public/*` assets to
a Svelte/PWA project under `tools/ui/`, embedded at CMake time via
diff --git a/llama.cpp.patches/llamafile-files/BUILD.mk b/llama.cpp.patches/llamafile-files/BUILD.mk
index 0a23aca5..04c7f787 100644
--- a/llama.cpp.patches/llamafile-files/BUILD.mk
+++ b/llama.cpp.patches/llamafile-files/BUILD.mk
@@ -436,6 +436,7 @@ UI_GEN_OBJ := $(UI_CPP_GEN:%.cpp=%.cpp.o)
TOOL_LLAMAFILE_OBJS := \
o/$(MODE)/llamafile/llamafile.o \
o/$(MODE)/llamafile/gpu.a \
+ o/$(MODE)/llamafile/sandbox.o \
o/$(MODE)/llamafile/zip.o
# Server objects depend on the llamafile bridge header and on the
diff --git a/llama.cpp.patches/patches/tools_server_server.cpp.patch b/llama.cpp.patches/patches/tools_server_server.cpp.patch
index f9c8402b..f1e2918f 100644
--- a/llama.cpp.patches/patches/tools_server_server.cpp.patch
+++ b/llama.cpp.patches/patches/tools_server_server.cpp.patch
@@ -9,11 +9,13 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp
#include <signal.h>
#include <thread> // for std::thread::hardware_concurrency
-@@ -21,6 +22,11 @@
+@@ -21,6 +22,13 @@
#include <windows.h>
#endif
+#ifdef COSMOCC
++#include <cerrno>
++#include <cstring>
+#include <cosmo.h>
+#include "llamafile.h"
+#endif
@@ -21,7 +23,7 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp
static std::function<void(int)> shutdown_handler;
static std::atomic_flag is_terminating = ATOMIC_FLAG_INIT;
-@@ -71,15 +77,26 @@ static server_http_context::handler_t ex_wrapper(server_http_context::handler_t
+@@ -71,15 +79,26 @@ static server_http_context::handler_t ex_wrapper(server_http_context::handler_t
};
}
@@ -52,7 +54,94 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp
common_init();
if (!common_params_parse(argc, argv, params, LLAMA_EXAMPLE_SERVER)) {
-@@ -332,6 +349,13 @@ int llama_server(int argc, char ** argv) {
+@@ -297,6 +316,86 @@ int llama_server(int argc, char ** argv) {
+ llama_backend_free();
+ };
+
++#ifdef COSMOCC
++ // Sandbox (pledge()/unveil(), llamafile issue #930). The whole
++ // derivation lives in llamafile/sandbox.c so it stays unit-testable
++ // and off the vendored diff. Placement is constrained on every side:
++ // - after common_params_parse(): -hf downloads need connect()
++ // - after ctx_http.init(): the TLS certificate/key are read there
++ // - before ctx_http.start(): filters only cover threads created
++ // after installation, so the HTTP listener/pool must not exist
++ // - before load_model(): untrusted GGUF data is parsed sandboxed
++ //
++ // Combined mode hosts an in-process HTTP client (the TUI) that needs
++ // connect(), which the accept-only sandbox forbids, so it's skipped.
++ const bool combined_mode = on_ready || on_shutdown_available;
++ if (combined_mode) {
++ SRV_WRN("%s", "sandbox: disabled in combined mode; use --server for a sandboxed server\n");
++ } else {
++ // pledge() (no outbound network / writes / exec) is applied by
++ // default. unveil() path confinement is opt-in (--confine-reads),
++ // because a server opens files by path at request time (multimodal
++ // media, etc.) that can't be known when the ruleset locks.
++ //
++ // cosmo installs the SECCOMP/Landlock filters on the calling thread
++ // only (no TSYNC), so quiesce the llama.cpp log worker across the
++ // call: pause() joins it, resume() respawns it under the filter.
++ //
++ // When --confine-reads is on, unveil() confines reads to these
++ // directories, so collect every on-disk path the loader opens
++ // after the lock. Add to this list when llama.cpp gains a new file
++ // parameter (else it'd be denied under confinement). Paths read
++ // before the sandbox (ssl_file_key/cert in ctx_http.init(), -hf
++ // downloads in parse) need no entry.
++ std::vector<const char *> read_paths, rw_paths;
++ read_paths.push_back(params.model.path.c_str());
++ read_paths.push_back(params.mmproj.path.c_str());
++ read_paths.push_back(params.media_path.c_str()); // request-time media
++ read_paths.push_back(params.speculative.draft.mparams.path.c_str());
++ read_paths.push_back(params.speculative.ngram_cache.lookup_cache_static.c_str());
++ read_paths.push_back(params.public_path.c_str());
++ for (const auto & la : params.lora_adapters)
++ read_paths.push_back(la.path.c_str());
++ for (const auto & cv : params.control_vectors)
++ read_paths.push_back(cv.fname.c_str());
++ rw_paths.push_back(params.slot_save_path.c_str());
++ rw_paths.push_back(params.path_prompt_cache.c_str());
++ rw_paths.push_back(params.speculative.ngram_cache.lookup_cache_dynamic.c_str());
++
++ // Features that dial out at runtime need connect(), which the
++ // accept-only default forbids; relax to full sockets for them
++ // (still no writes/exec). --rpc has no params field -- its handler
++ // registers backends directly -- so detect it on argv/env.
++ const bool needs_outbound =
++ params.ui_mcp_proxy ||
++ !params.server_tools.empty() ||
++ llamafile_has(argv, "--rpc") ||
++ (std::getenv("LLAMA_ARG_RPC") != nullptr);
++
++ llamafile_sandbox_spec spec = {
++ read_paths.data(), (int) read_paths.size(),
++ rw_paths.data(), (int) rw_paths.size(),
++ FLAG_confine_reads, needs_outbound,
++ };
++ char promises[64];
++ common_log_pause(common_log_main());
++ int sandbox = llamafile_sandbox_server(&spec, promises, sizeof(promises));
++ int sandbox_errno = errno; // capture before resume() spawns a thread
++ common_log_resume(common_log_main());
++ if (sandbox == LLAMAFILE_SANDBOX_FAILED) {
++ SRV_ERR("sandbox: pledge failed: %s; retry with --unsecure to disable sandboxing\n",
++ strerror(sandbox_errno));
++ clean_up();
++ return 1;
++ } else if (llamafile_sandbox_is_active(sandbox)) {
++ SRV_INF("sandbox: pledge(\"%s\") %s\n", promises,
++ llamafile_sandbox_describe(sandbox));
++ } else {
++ SRV_WRN("sandbox: %s\n", llamafile_sandbox_describe(sandbox));
++ }
++ }
++#endif
++
+ // start the HTTP server before loading the model to be able to serve /health requests
+ if (!ctx_http.start()) {
+ clean_up();
+@@ -332,6 +431,13 @@ int llama_server(int argc, char ** argv) {
// this will unblock start_loop()
ctx_server.terminate();
};
@@ -66,7 +155,7 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp
}
// TODO: refactor in common/console
-@@ -368,6 +392,11 @@ int llama_server(int argc, char ** argv) {
+@@ -368,6 +474,11 @@ int llama_server(int argc, char ** argv) {
} else {
SRV_INF("server is listening on %s\n", ctx_http.listening_address.c_str());
@@ -78,7 +167,7 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp
// optionally, notify router server that this instance is ready
std::thread monitor_thread;
if (child.is_child()) {
-@@ -392,5 +421,48 @@ int llama_server(int argc, char ** argv) {
+@@ -392,5 +503,56 @@ int llama_server(int argc, char ** argv) {
}
}
@@ -111,6 +200,14 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp
+ bool verbose = llamafile_has(argv, "--verbose");
+ FLAG_verbose = verbose ? 1 : 0;
+
++ // --unsecure is llamafile's flag, not llama.cpp's: consume it here so
++ // common_params_parse() never sees it (llamafile's own main strips it
++ // via parse_llamafile_args instead)
++ if (llamafile_consume_flag(&argc, argv, "--unsecure"))
++ FLAG_unsecure = true;
++ if (llamafile_consume_flag(&argc, argv, "--confine-reads"))
++ FLAG_confine_reads = true;
++
+ // Initialize GPU support early (must happen BEFORE llama_backend_init())
+ // This triggers dynamic loading of GPU backends (CUDA, ROCm, Metal)
+ // The llamafile_has_* functions use lazy initialization via cosmo_once()
diff --git a/llamafile/BUILD.mk b/llamafile/BUILD.mk
index bafde492..470afcf0 100644
--- a/llamafile/BUILD.mk
+++ b/llamafile/BUILD.mk
@@ -138,6 +138,7 @@ LLAMAFILE_SRCS_C := \
llamafile/gpu_backend.c \
llamafile/llamafile.c \
llamafile/metal.c \
+ llamafile/sandbox.c \
llamafile/vulkan.c \
llamafile/zip.c
diff --git a/llamafile/args.cpp b/llamafile/args.cpp
index 8f78367a..73c610f9 100644
--- a/llamafile/args.cpp
+++ b/llamafile/args.cpp
@@ -36,6 +36,8 @@ static bool is_llamafile_flag(const char* arg) {
strcmp(arg, "--ascii") == 0 ||
strcmp(arg, "--nologo") == 0 ||
strcmp(arg, "--nothink") == 0 ||
+ strcmp(arg, "--unsecure") == 0 ||
+ strcmp(arg, "--confine-reads") == 0 ||
strcmp(arg, "--version") == 0;
}
@@ -87,6 +89,12 @@ LlamafileArgs parse_llamafile_args(int argc, char** argv) {
FLAG_nologo = llamafile_has(argv, "--nologo");
FLAG_ascii = llamafile_has(argv, "--ascii");
+ // Check --unsecure flag (disables pledge() sandboxing, see sandbox.c)
+ FLAG_unsecure = llamafile_has(argv, "--unsecure");
+
+ // Check --confine-reads flag (opt-in unveil() path confinement, server mode)
+ FLAG_confine_reads = llamafile_has(argv, "--confine-reads");
+
// Filter out llamafile-specific arguments
// These are not recognized by llama.cpp and would cause errors
g_filtered_argv.clear();
diff --git a/llamafile/chatbot_cli.cpp b/llamafile/chatbot_cli.cpp
index 3c312635..9f4b72ec 100644
--- a/llamafile/chatbot_cli.cpp
+++ b/llamafile/chatbot_cli.cpp
@@ -168,6 +168,17 @@ int cli_main(int argc, char **argv) {
params.n_gpu_layers = INT_MAX;
}
+ // Drop network and filesystem-write access before the untrusted GGUF
+ // file is parsed. Must happen after common_params_parse() (-hf model
+ // downloads need the network) and before threads/weights come up.
+ // Quiesce the log worker across the pledge so it doesn't outlive the
+ // per-thread filter unsandboxed (see llamafile/sandbox.c rule 1).
+ common_log_pause(common_log_main());
+ int sandbox = llamafile_sandbox_enter("stdio rpath tty", FLAG_verbose);
+ common_log_resume(common_log_main());
+ if (sandbox == LLAMAFILE_SANDBOX_FAILED)
+ return 1;
+
// Load model
llama_model_params model_params = common_model_params_to_llama(params);
llama_model *model = llama_model_load_from_file(params.model.path.c_str(), model_params);
diff --git a/llamafile/chatbot_main.cpp b/llamafile/chatbot_main.cpp
index ccf415f6..d061e2ad 100644
--- a/llamafile/chatbot_main.cpp
+++ b/llamafile/chatbot_main.cpp
@@ -161,6 +161,18 @@ int main(int argc, char **argv) {
}
clear_ephemeral();
+ // Drop network access before the untrusted GGUF file is parsed. Must
+ // happen after common_params_parse() (-hf model downloads need the
+ // network) and before threads/weights come up. Unlike --cli, the chat
+ // TUI keeps wpath+cpath so /dump and /push can still write files.
+ // Quiesce the log worker across the pledge so it doesn't outlive the
+ // per-thread filter unsandboxed (see llamafile/sandbox.c rule 1).
+ common_log_pause(common_log_main());
+ int sandbox = llamafile_sandbox_enter("stdio rpath wpath cpath tty", verbose);
+ common_log_resume(common_log_main());
+ if (sandbox == LLAMAFILE_SANDBOX_FAILED)
+ exit(1);
+
// Suppress logging for model loading unless --verbose was specified
// We must set this AFTER common_init() since it overwrites the log callback
// and BEFORE model loading to suppress those logs
diff --git a/llamafile/llamafile.c b/llamafile/llamafile.c
index bcb04791..88e80d25 100644
--- a/llamafile/llamafile.c
+++ b/llamafile/llamafile.c
@@ -285,9 +285,17 @@ struct llamafile *llamafile_open_gguf(const char *fname, const char *mode) {
// open from file or from our own executable if it doesn't exist
struct llamafile *file;
if (!(file = llamafile_open_file(fname, mode))) {
- if (errno == ENOENT) {
+ // ENOENT: the path isn't a real file (e.g. a bare basename
+ // embedded by llamafile-convert). EACCES/EPERM: the sandbox
+ // (unveil/pledge) denies opening it directly; the embedded copy
+ // in our own /zip/ store is still reachable via the executable.
+ int open_errno = errno;
+ if (open_errno == ENOENT || open_errno == EACCES || open_errno == EPERM) {
if (!(file = llamafile_open_zip(GetProgramExecutableName(), fname, mode))) {
- errno = ENOENT;
+ // no embedded copy either: report why the real open failed
+ // (e.g. "Permission denied" for a sandbox/mode denial, not a
+ // misleading "No such file or directory")
+ errno = open_errno;
return 0;
}
return file;
@@ -451,6 +459,21 @@ bool llamafile_has(char **a, const char *x) {
return false;
}
+bool llamafile_consume_flag(int *argc, char **argv, const char *flag) {
+ bool found = false;
+ int w = 1;
+ for (int r = 1; r < *argc; ++r) {
+ if (!strcmp(argv[r], flag)) {
+ found = true;
+ } else {
+ argv[w++] = argv[r];
+ }
+ }
+ argv[w] = 0;
+ *argc = w;
+ return found;
+}
+
static const char *llamafile_get_home_dir(void) {
const char *homedir;
if (!(homedir = getenv("HOME")) || !*homedir)
diff --git a/llamafile/llamafile.h b/llamafile/llamafile.h
index ae7052fc..81063ec5 100644
--- a/llamafile/llamafile.h
+++ b/llamafile/llamafile.h
@@ -35,6 +35,7 @@ extern bool FLAG_nologo; // Suppresses logo display (chatbot_main.cpp)
extern bool FLAG_nothink; // Filters thinking/reasoning content (chatbot_cli.cpp)
extern bool FLAG_precise; // Forces precise math in tinyblas (tinyblas_cpu.h)
extern bool FLAG_recompile; // Forces GPU library recompilation (metal.c)
+extern bool FLAG_unsecure; // Disables pledge() sandboxing (sandbox.c)
extern int FLAG_gpu; // GPU backend selection (llamafile.c, metal.c, cuda.c)
extern int FLAG_verbose; // Verbose output (chatbot_main.cpp, metal.c, cuda.c)
@@ -114,6 +115,62 @@ int llamafile_gpu_parse(const char *); // Defined in llamafile.c
const char *llamafile_describe_gpu(void); // Defined in llamafile.c
void llamafile_early_gpu_init(char **); // Defined in llamafile.c
+// =============================================================================
+// Sandboxing - pledge()/SECCOMP + unveil()/Landlock (defined in sandbox.c)
+// =============================================================================
+
+// The pledge() syscall sandbox (no outbound network, no writes, no exec) is
+// applied by default. unveil() path confinement is opt-in (--confine-reads),
+// because locking the readable paths at startup is incompatible with files a
+// server opens by path at request time (multimodal media, etc.).
+
+#define LLAMAFILE_SANDBOX_FAILED -1 // pledge() failed, errno is set
+#define LLAMAFILE_SANDBOX_ACTIVE 0 // pledge() active
+#define LLAMAFILE_SANDBOX_UNSECURE 1 // skipped: --unsecure flag
+#define LLAMAFILE_SANDBOX_GPU 2 // skipped: GPU backend loaded
+#define LLAMAFILE_SANDBOX_UNSUPPORTED 3 // skipped: OS can't enforce pledge()
+#define LLAMAFILE_SANDBOX_ACTIVE_CONFINED 4 // pledge() active + unveil() applied
+#define LLAMAFILE_SANDBOX_ACTIVE_UNCONFINED 5 // pledge() active; unveil() asked
+ // for but filesystem can't enforce it
+
+extern bool FLAG_confine_reads; // opt-in unveil() path confinement (sandbox.c)
+
+bool llamafile_sandbox_supported(void); // Probe only, installs nothing
+int llamafile_sandbox_apply(const char *); // Unconditional pledge()
+int llamafile_sandbox(const char *); // Honors --unsecure and GPU mode
+int llamafile_sandbox_enter(const char *, bool);// sandbox() + perror/verbose report
+const char *llamafile_sandbox_describe(int); // Status code -> human string
+bool llamafile_sandbox_is_active(int); // true for the ACTIVE* statuses
+
+// Inputs to the server sandbox. read_paths are opened read-only (model,
+// mmproj, LoRA, draft model, control vectors, media dir, static web root);
+// rw_paths get write+create (slot-save dir, prompt cache). confine requests
+// unveil() path confinement; needs_outbound relaxes accept()-only networking
+// to full sockets when the server must dial out (--rpc, tools, MCP proxy).
+struct llamafile_sandbox_spec {
+ const char *const *read_paths;
+ int n_read;
+ const char *const *rw_paths;
+ int n_rw;
+ bool confine;
+ bool needs_outbound;
+};
+
+// Applies the server sandbox: pledge() always, plus unveil() when
+// spec->confine. Fills promises_out with the pledge string for logging.
+// Returns an ACTIVE* status (see describe()) or a skip/FAILED code.
+int llamafile_sandbox_server(const struct llamafile_sandbox_spec *spec,
+ char *promises_out, size_t promises_len);
+
+// Pure promise-string derivation, exposed for unit testing.
+void llamafile_sandbox_server_promises(char *out, size_t len, bool is_openbsd,
+ bool has_rw, bool needs_outbound);
+
+// Removes every occurrence of flag from argv in place, updating *argc, and
+// returns true if it was present. Used to consume llamafile-only flags
+// (e.g. --unsecure) before handing argv to llama.cpp's parser. In llamafile.c.
+bool llamafile_consume_flag(int *argc, char **argv, const char *flag);
+
// Log callback type for Metal backend (matches ggml_log_callback)
typedef void (*llamafile_log_callback)(int level, const char *text, void *user_data);
diff --git a/llamafile/main.cpp b/llamafile/main.cpp
index 699c90d5..33963f3e 100644
--- a/llamafile/main.cpp
+++ b/llamafile/main.cpp
@@ -172,6 +172,8 @@ static void print_general_help(bool show_full_list_hint) {
" --gpu MODE GPU backend (auto, nvidia, amd, apple, disable)\n"
" --nologo suppress the startup logo\n"
" --ascii use ASCII art instead of Unicode for logo\n"
+ " --unsecure disable pledge() sandboxing (Linux/OpenBSD)\n"
+ " --confine-reads restrict server file reads to the weights dirs\n"
" --verbose enable verbose logging\n"
" --version show version information\n"
" --help show this help\n"
@@ -205,6 +207,8 @@ static void print_server_help(bool show_full_list_hint) {
" --host ADDR ip address to listen on (default 127.0.0.1)\n"
" --port N tcp port to listen on (default 8080)\n"
" --api-key KEY require this API key for authentication\n"
+ " --confine-reads restrict file reads to the weights directories\n"
+ " --unsecure disable pledge() sandboxing (Linux/OpenBSD)\n"
"\n");
if (show_full_list_hint) {
printf("all other llama.cpp options are also accepted.\n"
@@ -395,7 +399,11 @@ int main(int argc, char **argv) {
if (llamafile_has(argv, "--help") || llamafile_has(argv, "-h")) {
switch (args.mode) {
case lf::ProgramMode::SERVER:
- break; // fall through to llama.cpp's server help
+ // Show llamafile's own server flags (which are stripped before
+ // llama.cpp sees them, so its parser can't list them) before
+ // falling through to llama.cpp's server help.
+ print_server_help(false);
+ break;
case lf::ProgramMode::AUTO:
// Combined mode: args go to the HTTP server, so show the
// llamafile intro followed by llama.cpp's server option list.
@@ -467,6 +475,12 @@ int main(int argc, char **argv) {
args.llama_argv = quiet_argv.data();
}
+ // --confine-reads only affects the --server sandbox. In the other modes
+ // (including the default combined mode, whose sandbox is skipped) it does
+ // nothing, so say so rather than silently ignoring it.
+ if (FLAG_confine_reads && args.mode != lf::ProgramMode::SERVER)
+ fprintf(stderr, "warning: --confine-reads only applies to --server mode; ignoring\n");
+
// Initialize GPU support (triggers dynamic loading of GPU backends)
llamafile_has_gpu();
diff --git a/llamafile/sandbox.c b/llamafile/sandbox.c
new file mode 100644
index 00000000..624df03a
--- /dev/null
+++ b/llamafile/sandbox.c
@@ -0,0 +1,421 @@
+// -*- mode:c;indent-tabs-mode:nil;c-basic-offset:4;coding:utf-8 -*-
+// vi: set et ft=c ts=4 sts=4 sw=4 fenc=utf-8 :vi
+//
+// Copyright 2026 Mozilla.ai
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//
+// llamafile sandboxing (issue #930)
+//
+// This module wraps two Cosmopolitan Libc primitives:
+//
+// pledge() OpenBSD-style syscall sandboxing. On OpenBSD it calls the
+// native pledge(2); on Linux it installs a SECCOMP BPF filter
+// emulating the requested promise set. No kernel config or
+// privileges are needed.
+// unveil() restricts which filesystem paths are reachable. On OpenBSD
+// it calls the native unveil(2); on Linux it uses the Landlock
+// LSM (kernel 5.13+). Together with pledge's "rpath" this turns
+// "read-only, anywhere" into "read-only, within these dirs".
+//
+// On every other platform (macOS, Windows, other BSDs) both are no-ops:
+// the sandbox reports that and callers keep running unsandboxed, matching
+// the pre-0.10 behavior.
+//
+// Promise cheat sheet (see Cosmopolitan libc/calls/pledge.c for the
+// full syscall lists):
+//
+// stdio read/write on already-open fds, mmap w/o PROT_EXEC,
+// futexes, clocks, poll/epoll, and thread creation --
+// enough for ggml inference on memory-mapped weights
+// rpath open files read-only (model weights, executable zip store)
+// wpath open existing files for writing
+// cpath create/rename/unlink files
+// tty terminal ioctls (raw mode, window size)
+// inet IPv4/IPv6 sockets incl. connect()
+// anet like inet but connect() is forbidden: the process can
+// accept() connections yet never initiate them (Linux only;
+// on OpenBSD the closest promise is inet)
+//
+// Three rules of engagement:
+//
+// 1. SECCOMP/Landlock filters attach to the calling thread and are
+// inherited only by threads created *afterward* (cosmo does not use
+// SECCOMP_FILTER_FLAG_TSYNC). Callers must sandbox before spawning
+// worker threads, and must ensure no pre-existing background thread
+// (e.g. the llama.cpp log worker) survives across the call --- see
+// the common_log_pause()/resume() dance at the server call site.
+//
+// 2. GPU support is loaded with cosmo_dlopen() and drivers keep talking
+// to /dev nodes at inference time, which no reasonable promise set
+// covers. When a GPU is in use the sandbox is skipped and we say so.
+//
+// 3. unveil() must run *before* pledge(): it needs Landlock syscalls
+// that the pledge filter would otherwise deny.
+//
+// On Linux, violations return EPERM instead of killing the process
+// (PLEDGE_PENALTY_RETURN_EPERM), so a blocked syscall surfaces as an
+// ordinary I/O error. OpenBSD's native pledge(2) always kills with
+// SIGABRT instead; that penalty mode is Linux-only.
+//
+
+#define _COSMO_SOURCE // exposes pledge(), unveil(), __pledge_mode
+
+#include "llamafile.h"
+
+#include <errno.h>
+#include <limits.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/wait.h>
+#include <unistd.h>
+
+#ifdef COSMOCC
+#include <cosmo.h>
+#include <fcntl.h>
+#include <libc/calls/calls.h>
+#include <libc/calls/pledge.h>
+#include <libc/dce.h> // IsOpenbsd
+#endif
+
+bool FLAG_unsecure = false;
+bool FLAG_confine_reads = false;
+
+// Reports whether the host OS can enforce pledge() promises
+// (Linux with SECCOMP_MODE_FILTER, or OpenBSD). Does not install anything.
+bool llamafile_sandbox_supported(void) {
+#ifdef COSMOCC
+ return !pledge(0, 0);
+#else
+ return false;
+#endif
+}
+
+// Shared policy gate: LLAMAFILE_SANDBOX_ACTIVE means "install it", otherwise a
+// skip code. Both entry points consult this so the gate lives in one place.
+static int sandbox_skip_status(void) {
+ if (FLAG_unsecure)
+ return LLAMAFILE_SANDBOX_UNSECURE;
+ if (llamafile_has_gpu())
+ return LLAMAFILE_SANDBOX_GPU;
+ if (!llamafile_sandbox_supported())
+ return LLAMAFILE_SANDBOX_UNSUPPORTED;
+ return LLAMAFILE_SANDBOX_ACTIVE;
+}
+
+// Shared pledge() tail: EPERM penalty mode + install the filter.
+static int sandbox_install_pledge(const char *promises) {
+#ifdef COSMOCC
+ __pledge_mode = PLEDGE_PENALTY_RETURN_EPERM;
+ return pledge(promises, 0) ? LLAMAFILE_SANDBOX_FAILED
+ : LLAMAFILE_SANDBOX_ACTIVE;
+#else
+ (void)promises;
+ return LLAMAFILE_SANDBOX_UNSUPPORTED;
+#endif
+}
+
+// Installs the pledge() sandbox unconditionally, skipping the --unsecure/GPU
+// policy gate (the gated entry points share sandbox_install_pledge() directly
+// instead). Exposed for the unit test, which drives the promise sets in a
+// forked child without the gate.
+int llamafile_sandbox_apply(const char *promises) {
+ if (!llamafile_sandbox_supported())
+ return LLAMAFILE_SANDBOX_UNSUPPORTED;
+ return sandbox_install_pledge(promises);
+}
+
+// Policy entry point for the CLI and chat modes: honors --unsecure and GPU
+// mode, then pledges. These modes never confine paths -- they have no network
+// at all and legitimately read user-chosen files (e.g. --image).
+int llamafile_sandbox(const char *promises) {
+ int skip = sandbox_skip_status();
+ if (skip != LLAMAFILE_SANDBOX_ACTIVE)
+ return skip;
+ return sandbox_install_pledge(promises);
+}
+
+// Applies the CLI/chat sandbox and reports the outcome, collapsing the
+// apply-perror-log boilerplate the two modes used to duplicate. On
+// failure the caller decides whether to exit or return; every other
+// status is informational and printed only when verbose.
+int llamafile_sandbox_enter(const char *promises, bool verbose) {
+ int status = llamafile_sandbox(promises);
+ if (status == LLAMAFILE_SANDBOX_FAILED) {
+ perror("pledge");
+ } else if (verbose) {
+ fprintf(stderr, "sandbox: %s\n", llamafile_sandbox_describe(status));
+ }
+ return status;
+}
+
+// Canary files the governability probe reads back after locking: if one is
+// still readable, unveil() installed nothing (no Landlock, or an ungoverned
+// filesystem) and we must not claim confinement. Requirements:
+// - world-readable, so the ONLY thing that can deny it is Landlock -- a
+// root-only file would be denied by ordinary permissions too, masking a
+// no-op unveil() and yielding a false "confined";
+// - outside every unveiled weights dir;
+// - benign and commonly read by ordinary software (distro/hostname
+// detection), so the probe isn't mistaken for credential recon the way
+// reading /etc/passwd would be.
+// The probe uses the first one that exists; if none do, it conservatively
+// reports "not confined" rather than guessing.
+static const char *const kSandboxCanaries[] = {
+ "/etc/os-release",
+ "/etc/hostname",
+};
+
+// Derives the server pledge() promise string. Pure and side-effect-free so
+// it can be unit-tested. rpath is unconditional: the model loader always
+// performs a real open() -- of the on-disk weights, or of the executable
+// itself to read its embedded /zip/ store (llamafile_open_zip) -- so a server
+// can never load a model without it.
+void llamafile_sandbox_server_promises(char *out, size_t len, bool is_openbsd,
+ bool has_rw, bool needs_outbound) {
+ // "anet" is accept()-only networking (no connect) -- the default, so a
+ // compromised server can't dial out. When the server itself must make
+ // outbound connections (--rpc, server-side tools, the MCP proxy) we relax
+ // to "inet", which still blocks writes and exec. OpenBSD has no "anet", so
+ // it always uses "inet".
+ const char *net = (is_openbsd || needs_outbound) ? "inet" : "anet";
+ if (has_rw) // slot save/restore and the prompt cache read+write on disk
+ snprintf(out, len, "stdio %s rpath wpath cpath", net);
+ else
+ snprintf(out, len, "stdio %s rpath", net);
+}
+
+#ifdef COSMOCC
+
+// A path is "on disk" (needs its own unveil rule) when it exists on the
+// real filesystem and isn't served from the executable's /zip/ store.
+// Embedded weights -- a /zip/ path, or a bare basename that resolves to
+// the executable via llamafile_open_zip -- are covered by unveiling the
+// executable itself.
+static bool path_on_disk(const char *path) {
+ return path && *path && strncmp(path, "/zip/", 5) && !access(path, F_OK);
+}
+
+// Writes into `dir` the path that must be reachable to open `path`: the
+// path itself if it's a directory (a web root, a slot-save dir), else its
+// parent directory, so multi-part GGUF shards beside a weights file come
+// along -- except a file at the filesystem root, whose parent "/" would
+// expose everything, so the file itself is used. Returns false for paths
+// not on the real filesystem (empty, /zip/, or absent). Single source of
+// truth for both unveil_container() (apply) and the governability probe.
+static bool container_of(const char *path, char *dir, size_t len) {
+ if (!path_on_disk(path))
+ return false;
+ if (isdirectory(path)) {
+ strlcpy(dir, path, len);
+ return true;
+ }
+ strlcpy(dir, path, len);
+ char *slash = strrchr(dir, '/');
+ if (!slash)
+ strlcpy(dir, ".", len); // bare "file" -> cwd
+ else if (slash == dir)
+ strlcpy(dir, path, len); // "/file" -> the file, never "/"
+ else
+ *slash = '\0'; // "dir/file" -> "dir"
+ return true;
+}
+
+// unveil()s the container of `path` with the given perms, if it's on disk.
+static void unveil_container(const char *path, const char *perms) {
+ char dir[PATH_MAX];
+ if (container_of(path, dir, sizeof(dir)))
+ unveil(dir, perms);
+}
+
+// Installs the unveil() read/read-write rules (does NOT lock; caller locks
+// with unveil(0,0)). The executable is always readable -- llamafile_open_zip
+// reopens it for embedded weights and the bundled web UI -- as are the name
+// resolution files a non-numeric --host needs. rw paths (which may not exist
+// yet, e.g. a prompt cache created on first save) get their container dir.
+static void unveil_apply(const char *const *read_paths, int n_read,
+ const char *const *rw_paths, int n_rw) {
+ unveil(GetProgramExecutableName(), "r");
+ if (path_on_disk("/etc/hosts"))
+ unveil("/etc/hosts", "r");
+ if (path_on_disk("/etc/resolv.conf"))
+ unveil("/etc/resolv.conf", "r");
+ for (int i = 0; i < n_read; ++i)
+ unveil_container(read_paths[i], "r");
+ for (int i = 0; i < n_rw; ++i)
+ unveil_container(rw_paths[i], "rwc");
+}
+
+// True if `path` opens read-only under the (now locked) ruleset.
+static bool can_read(const char *path) {
+ int fd = open(path, O_RDONLY);
+ if (fd < 0)
+ return false;
+ close(fd);
+ return true;
+}
+
+// True only if opening `path` is *actively denied* (EACCES/EPERM), not
+// merely failing for some other reason such as ENOENT.
+static bool is_denied(const char *path) {
+ errno = 0;
+ int fd = open(path, O_RDONLY);
+ if (fd >= 0) {
+ close(fd);
+ return false;
+ }
+ return errno == EACCES || errno == EPERM;
+}
+
+// Probes whether unveil() confinement can be both enforced and survived on
+// this filesystem, in a throwaway child so a "no" doesn't lock down the
+// real process. Two failure modes to rule out:
+// - it installed nothing (no Landlock kernel, or cosmo's silent no-op):
+// the canary outside every rule stays readable -> not confined;
+// - it over-denies (virtiofs/9p/NFS accept rules then block everything):
+// a governed weights path can't be opened -> would break model load.
+// Every path_on_disk()/container check runs before the lock, since access()
+// is itself governed once the ruleset is locked.
+#define SANDBOX_PROBE_MAX 64
+static bool unveil_is_governable(const char *const *read_paths, int n_read,
+ const char *const *rw_paths, int n_rw) {
+ pid_t pid = fork();
+ if (pid < 0)
+ return false;
+ if (!pid) {
+ // Snapshot the paths the locked ruleset must still let us read,
+ // computed while access() is unrestricted.
+ char verify[SANDBOX_PROBE_MAX][PATH_MAX];
+ int nv = 0;
+ strlcpy(verify[nv++], GetProgramExecutableName(), PATH_MAX);
+ for (int i = 0; i < n_read && nv < SANDBOX_PROBE_MAX; ++i)
+ if (path_on_disk(read_paths[i]))
+ strlcpy(verify[nv++], read_paths[i], PATH_MAX);
+ for (int i = 0; i < n_rw && nv < SANDBOX_PROBE_MAX; ++i)
+ if (container_of(rw_paths[i], verify[nv], PATH_MAX))
+ ++nv;
+
+ // Pick a canary that exists now, before the ruleset can hide it.
+ const char *canary = 0;
+ for (size_t i = 0; i < sizeof(kSandboxCanaries) / sizeof(*kSandboxCanaries); ++i)
+ if (!access(kSandboxCanaries[i], F_OK)) {
+ canary = kSandboxCanaries[i];
+ break;
+ }
+
+ unveil_apply(read_paths, n_read, rw_paths, n_rw);
+ unveil(0, 0); // lock
+
+ int ok = 1;
+ for (int i = 0; ok && i < nv; ++i)
+ ok = can_read(verify[i]);
+ // Confinement must demonstrably bite: the canary, outside every
+ // rule, must now be *denied* (EACCES/EPERM). No canary, or one that
+ // fails for another reason, is inconclusive -- treat as "not
+ // confined" rather than claiming a guarantee unveil() didn't install.
+ if (ok && (!canary || !is_denied(canary)))
+ ok = 0;
+ _exit(ok ? 0 : 1);
+ }
+ int ws;
+ while (waitpid(pid, &ws, 0) < 0 && errno == EINTR) {
+ }
+ return WIFEXITED(ws) && WEXITSTATUS(ws) == 0;
+}
+
+#endif // COSMOCC
+
+// Applies the server sandbox: pledge() always (accept()-only networking, no
+// writes/exec unless configured), plus unveil() path confinement when
+// spec->confine and the filesystem can enforce it. Fills promises_out with
+// the pledge string for logging; returns an ACTIVE* status or a skip/FAILED
+// code. The caller must quiesce background threads around this call (the
+// filters are per-thread; see rule 1 above).
+int llamafile_sandbox_server(const struct llamafile_sandbox_spec *spec,
+ char *promises_out, size_t promises_len) {
+ if (promises_out && promises_len)
+ promises_out[0] = '\0';
+ int skip = sandbox_skip_status();
+ if (skip != LLAMAFILE_SANDBOX_ACTIVE)
+ return skip;
+#ifdef COSMOCC
+ // wpath/cpath only when a write target is actually configured; callers
+ // pass fixed-position entries that are often empty strings.
+ bool has_rw = false;
+ for (int i = 0; i < spec->n_rw; ++i)
+ if (spec->rw_paths[i] && *spec->rw_paths[i])
+ has_rw = true;
+ char promises[64];
+ llamafile_sandbox_server_promises(promises, sizeof(promises), IsOpenbsd(),
+ has_rw, spec->needs_outbound);
+ if (promises_out)
+ strlcpy(promises_out, promises, promises_len);
+
+ // Opt-in unveil() path confinement, run before pledge() (it needs
+ // Landlock syscalls the pledge filter denies). Skip it when the
+ // filesystem can't enforce it, rather than locking down a server that can
+ // no longer read its weights. 0 = not requested, 1 = confined, 2 = asked
+ // for but unavailable.
+ int confine_state = 0;
+ if (spec->confine) {
+ if (unveil_is_governable(spec->read_paths, spec->n_read,
+ spec->rw_paths, spec->n_rw)) {
+ unveil_apply(spec->read_paths, spec->n_read,
+ spec->rw_paths, spec->n_rw);
+ unveil(0, 0); // lock the ruleset
+ confine_state = 1;
+ } else {
+ confine_state = 2;
+ }
+ }
+
+ if (sandbox_install_pledge(promises) == LLAMAFILE_SANDBOX_FAILED)
+ return LLAMAFILE_SANDBOX_FAILED;
+ return confine_state == 1 ? LLAMAFILE_SANDBOX_ACTIVE_CONFINED
+ : confine_state == 2 ? LLAMAFILE_SANDBOX_ACTIVE_UNCONFINED
+ : LLAMAFILE_SANDBOX_ACTIVE;
+#else
+ (void)spec;
+ return LLAMAFILE_SANDBOX_UNSUPPORTED;
+#endif
+}
+
+// True for the statuses where the pledge() filter is installed (whether or
+// not path confinement also applied), as opposed to a skip or failure.
+bool llamafile_sandbox_is_active(int status) {
+ return status == LLAMAFILE_SANDBOX_ACTIVE ||
+ status == LLAMAFILE_SANDBOX_ACTIVE_CONFINED ||
+ status == LLAMAFILE_SANDBOX_ACTIVE_UNCONFINED;
+}
+
+const char *llamafile_sandbox_describe(int status) {
+ switch (status) {
+ case LLAMAFILE_SANDBOX_ACTIVE:
+ return "active";
+ case LLAMAFILE_SANDBOX_ACTIVE_CONFINED:
+ return "active, reads confined to weights dirs";
+ case LLAMAFILE_SANDBOX_ACTIVE_UNCONFINED:
+ return "active (path confinement requested but unavailable on this filesystem)";
+ case LLAMAFILE_SANDBOX_UNSECURE:
+ return "disabled by --unsecure";
+ case LLAMAFILE_SANDBOX_GPU:
+ return "disabled in GPU mode";
+ case LLAMAFILE_SANDBOX_UNSUPPORTED:
+ return "not supported on this OS";
+ case LLAMAFILE_SANDBOX_FAILED:
+ return "failed";
+ default:
+ return "unknown";
+ }
+}
diff --git a/tests/BUILD.mk b/tests/BUILD.mk
index 9ec82eff..b6ce417f 100644
--- a/tests/BUILD.mk
+++ b/tests/BUILD.mk
@@ -120,6 +120,28 @@ o/$(MODE)/tests/gpu_backend_test: \
@mkdir -p $(@D)
$(CXX) $(LDFLAGS) -o $@ $^ $(LDLIBS)
+# ==============================================================================
+# Test: sandbox_test (issue #930 pledge/SECCOMP sandboxing)
+# ==============================================================================
+#
+# Forks children that install the promise sets llamafile uses (server:
+# stdio+anet, cli: stdio+rpath+tty) and asserts allowed syscalls still
+# work while blocked ones fail with EPERM. Self-skips on OSes where
+# pledge() can't be enforced (anything but Linux/OpenBSD). Links against
+# sandbox.o alone: it stubs llamafile_has_gpu() to stay off the GPU stack.
+
+SANDBOX_TEST_DEPS := \
+ o/$(MODE)/llamafile/sandbox.o
+
+# sandbox_test.o is built by the generic %.o: %.c rule (-iquote. resolves
+# the llamafile/llamafile.h include)
+
+o/$(MODE)/tests/sandbox_test: \
+ o/$(MODE)/tests/sandbox_test.o \
+ $(SANDBOX_TEST_DEPS)
+ @mkdir -p $(@D)
+ $(CC) $(LDFLAGS) -o $@ $^ $(LDLIBS)
+
# ==============================================================================
# Test: backend_ops_test (upstream ggml test-backend-ops under llamafile)
# ==============================================================================
@@ -183,4 +205,5 @@ o/$(MODE)/tests: \
o/$(MODE)/tests/extract_data_uris_test.runs \
o/$(MODE)/tests/fa_helpers_test.runs \
o/$(MODE)/tests/gpu_backend_test.runs \
+ o/$(MODE)/tests/sandbox_test.runs \
o/$(MODE)/tests/transcribefile_smoke.runs
diff --git a/tests/integration/README.md b/tests/integration/README.md
index 0f7f9489..2ea61732 100644
--- a/tests/integration/README.md
+++ b/tests/integration/README.md
@@ -9,6 +9,8 @@ Integration tests for llamafile covering CLI, TUI, and server modes.
## Running Tests
+### Single model
+
```bash
cd tests/integration
@@ -22,6 +24,22 @@ cd tests/integration
./run_tests.sh --executable ~/model.llamafile -v
```
+### Full suite across all models
+
+`tests/run_integration_tests.sh` runs the full suite across a configurable list
+of models, then runs the `help` and `ssl` tests once against the bare executable.
+Edit the `TESTS` array and path variables at the top of the script to match your
+local model collection.
+
+```bash
+# From the repo root — runs every model in the TESTS list, then help + ssl
+./tests/run_integration_tests.sh
+```
+
+> **Network access**: the script includes the `ssl online` tests, which download
+> a small model from Hugging Face. Make sure you have internet access before
+> running it, or remove the `ssl and online` invocation from the script.
+
## Test Categories
Use `-m` to select test categories:
@@ -37,8 +55,11 @@ Use `-m` to select test categories:
| `thinking` | Thinking model tests (QwQ, DeepSeek-R1, etc.) |
| `gpu` | GPU acceleration tests |
| `cpu` | CPU-only tests |
-| `ssl` | HTTPS/TLS tests: serving with `--ssl-cert-file`, model download over HTTPS, untrusted-cert rejection. Needs `openssl` in PATH to generate a test certificate |
-| `online` | Tests that need network access (currently the HTTPS model download). Skip with `-m "not online"` |
+| `help` | `--help` output tests; no model needed |
+| `bare_executable` | Tests that run once against the bare binary (no embedded model); automatically skipped when the executable is a `.llamafile` bundle. Includes `help` and `ssl`. |
+| `ssl` | HTTPS/TLS tests: serving with `--ssl-cert-file` and model download over HTTPS. Needs `openssl` in PATH. Run against the bare executable with `--model`. |
+| `determinism` | Tests requiring reproducible output at temperature=0. Unreliable on gpt-oss models (see `tests/run_integration_tests.sh`); deselect with `-m "not determinism"`. |
+| `online` | Tests that need network access (the HTTPS model download from Hugging Face). Skip with `-m "not online"` if offline. |
Examples:
@@ -52,11 +73,9 @@ Examples:
# Skip multimodal and tool_calling tests
./run_tests.sh --executable ~/model.llamafile -m "not multimodal and not tool_calling"
-# Run only the HTTPS/TLS tests (or skip them with -m "not ssl")
-./run_tests.sh --executable ./o/llamafile/llamafile --model model.gguf -m ssl
-
-# Run the HTTPS/TLS tests without the network-dependent ones
+# Run the ssl tests (bare executable required; --model for the serving tests)
./run_tests.sh --executable ./o/llamafile/llamafile --model model.gguf -m "ssl and not online"
+./run_tests.sh --executable ./o/llamafile/llamafile -m "ssl and online"
```
## Options
@@ -91,20 +110,25 @@ Use `--log-cli-level` to see what the model returns:
## Test Structure
```
-tests/integration/
-├── run_tests.sh # Test runner script
-├── conftest.py # Pytest fixtures
-├── pyproject.toml # Dependencies and pytest config
-├── utils/
-│ └── llamafile.py # LlamafileRunner utility class
-├── fixtures/
-│ └── test_image.png # Test image for multimodal tests
-└── tests/
- ├── test_cli.py # CLI mode tests
- ├── test_tui.py # TUI mode tests
- ├── test_server.py # Server mode tests
- ├── test_combined.py # TUI+Server simultaneous mode
- ├── test_multimodal.py # Image description tests
- ├── test_tool_calling.py # Tool use tests
- └── test_gpu.py # GPU/CPU execution tests
+tests/
+├── run_integration_tests.sh # Full-suite runner across all models (edit to customise)
+└── integration/
+ ├── run_tests.sh # Single-model test runner
+ ├── conftest.py # Pytest fixtures
+ ├── pyproject.toml # Dependencies and pytest config
+ ├── utils/
+ │ └── llamafile.py # LlamafileRunner utility class and shared helpers
+ ├── fixtures/
+ │ └── test_image.png # Test image for multimodal tests
+ └── tests/
+ ├── test_cli.py # CLI mode tests
+ ├── test_tui.py # TUI/chat mode tests
+ ├── test_server.py # Server mode tests
+ ├── test_combined.py # TUI+Server simultaneous mode
+ ├── test_multimodal.py # Image description tests
+ ├── test_tool_calling.py # Tool use tests
+ ├── test_gpu.py # GPU/CPU execution tests
+ ├── test_help.py # --help output tests (bare executable, no model)
+ ├── test_ssl.py # HTTPS/TLS serving and model download tests
+ └── test_sandbox.py # pledge/SECCOMP sandboxing tests
```
diff --git a/tests/integration/pyproject.toml b/tests/integration/pyproject.toml
index 8b47aa39..0d19c3f8 100644
--- a/tests/integration/pyproject.toml
+++ b/tests/integration/pyproject.toml
@@ -29,4 +29,7 @@ markers = [
"help: tests --help output (no model required)",
"ssl: tests HTTPS/TLS support (serving and model download)",
"online: tests that need network access (deselect with '-m \"not online\"')",
+ "sandbox: tests pledge()/SECCOMP sandboxing (enforcement checks need Linux)",
+ "bare_executable: requires a bare llamafile binary with no embedded model (skip for pre-built .llamafile bundles)",
+ "determinism: requires reproducible output at temperature=0 (unreliable on gpt-oss models, whose long forced reasoning trace and MXFP4 quantization give a threaded matmul rounding difference more opportunities to flip a near-tied token; see tests/run_integration_tests.sh)",
]
diff --git a/tests/integration/tests/test_cli.py b/tests/integration/tests/test_cli.py
index 51113e27..d636e5fc 100644
--- a/tests/integration/tests/test_cli.py
+++ b/tests/integration/tests/test_cli.py
@@ -2,6 +2,8 @@
import pytest
+from utils.prompts import ADD_2_2, GREETING_PROMPT
+
@pytest.mark.cli
class TestCLIBasic:
@@ -9,19 +11,19 @@ class TestCLIBasic:
def test_cli_responds(self, llamafile, timeouts):
"""Test that CLI mode starts and generates a response."""
- result = llamafile.run_cli("Say hello in exactly one word.", timeout=timeouts.cli)
+ result = llamafile.run_cli(GREETING_PROMPT, timeout=timeouts.cli)
assert result.returncode == 0, f"CLI failed: {result.stderr}"
assert len(result.stdout.strip()) > 0, "No output generated"
def test_cli_math_question(self, llamafile, timeouts):
"""Test that CLI can answer a simple math question."""
- result = llamafile.run_cli(
- "What is 2+2? Answer with just the number.", timeout=timeouts.cli
- )
+ result = llamafile.run_cli(ADD_2_2.prompt, timeout=timeouts.cli)
assert result.returncode == 0, f"CLI failed: {result.stderr}"
- assert "4" in result.stdout, f"Expected '4' in output: {result.stdout}"
+ assert ADD_2_2.check(result.stdout), (
+ f"Expected {ADD_2_2.describe()} in output: {result.stdout}"
+ )
def test_cli_exits_cleanly(self, llamafile, timeouts):
"""Test that CLI exits with code 0 after completion."""
diff --git a/tests/integration/tests/test_combined.py b/tests/integration/tests/test_combined.py
index ccb7823c..5ffff04f 100644
--- a/tests/integration/tests/test_combined.py
+++ b/tests/integration/tests/test_combined.py
@@ -3,6 +3,7 @@
import pytest
from utils.llamafile import LlamafileRunner, read_until_idle, stop_tui
+from utils.prompts import ADD_1_1, ADD_2_2, ADD_3_3, GREETING_PROMPT
@pytest.mark.tui
@@ -17,13 +18,13 @@ class TestCombinedMode:
try:
ready = LlamafileRunner.wait_for_server(
- server_port, timeout=timeouts.server_ready
+ server_port, timeout=timeouts.server_ready, proc=proc
)
assert ready, "Server did not become ready in combined mode"
response = LlamafileRunner.chat_completion(
port=server_port,
- messages=[{"role": "user", "content": "Say hello"}],
+ messages=[{"role": "user", "content": GREETING_PROMPT}],
timeout=timeouts.http_request,
)
@@ -39,7 +40,7 @@ class TestCombinedMode:
try:
ready = LlamafileRunner.wait_for_server(
- server_port, timeout=timeouts.server_ready
+ server_port, timeout=timeouts.server_ready, proc=proc
)
assert ready, "Server did not become ready"
@@ -49,13 +50,14 @@ class TestCombinedMode:
# Test 1: Send a request via server API
response1 = LlamafileRunner.chat_completion(
port=server_port,
- messages=[{"role": "user", "content": "What is 1+1?"}],
+ messages=[{"role": "user", "content": ADD_1_1.prompt}],
timeout=timeouts.http_request,
)
- assert "2" in response1["choices"][0]["message"]["content"]
+ content1 = response1["choices"][0]["message"]["content"]
+ assert ADD_1_1.check(content1), f"Expected {ADD_1_1.describe()} in content: {content1}"
# Test 2: Send TUI input and verify response
- proc.stdin.write("What is 2+2?\n")
+ proc.stdin.write(f"{ADD_2_2.prompt}\n")
proc.stdin.flush()
# Read TUI output until model stops generating
@@ -65,15 +67,18 @@ class TestCombinedMode:
max_timeout=timeouts.cli,
)
assert len(tui_output) > 0, "TUI produced no output"
- assert "4" in tui_output, f"Expected '4' in TUI output: {tui_output}"
+ assert ADD_2_2.check(tui_output), (
+ f"Expected {ADD_2_2.describe()} in TUI output: {tui_output}"
+ )
# Test 3: Server should still work after TUI interaction
response2 = LlamafileRunner.chat_completion(
port=server_port,
- messages=[{"role": "user", "content": "What is 3+3?"}],
+ messages=[{"role": "user", "content": ADD_3_3.prompt}],
timeout=timeouts.http_request,
)
- assert "6" in response2["choices"][0]["message"]["content"]
+ content2 = response2["choices"][0]["message"]["content"]
+ assert ADD_3_3.check(content2), f"Expected {ADD_3_3.describe()} in content: {content2}"
finally:
stop_tui(proc)
diff --git a/tests/integration/tests/test_gpu.py b/tests/integration/tests/test_gpu.py
index 2d97f783..08a072e3 100644
--- a/tests/integration/tests/test_gpu.py
+++ b/tests/integration/tests/test_gpu.py
@@ -8,6 +8,7 @@ import tempfile
import pytest
from utils.llamafile import LlamafileRunner
+from utils.prompts import GREETING_PROMPT
def get_available_gpu() -> str | None:
@@ -139,7 +140,7 @@ class TestGPUAcceleration:
try:
result = runner.run_cli(
- "Say hello",
+ GREETING_PROMPT,
timeout=timeouts.cli,
log_file=log_path,
extra_args=["--verbose"],
@@ -176,14 +177,14 @@ class TestGPUAcceleration:
try:
ready = LlamafileRunner.wait_for_server(
- server_port, timeout=timeouts.server_ready
+ server_port, timeout=timeouts.server_ready, proc=proc
)
assert ready
# Verify response works
response = LlamafileRunner.chat_completion(
port=server_port,
- messages=[{"role": "user", "content": "Say hello"}],
+ messages=[{"role": "user", "content": GREETING_PROMPT}],
timeout=timeouts.http_request,
)
assert len(response["choices"][0]["message"]["content"]) > 0
@@ -222,7 +223,7 @@ class TestCPUExecution:
try:
result = runner.run_cli(
- "Say hello",
+ GREETING_PROMPT,
timeout=timeouts.cli,
log_file=log_path,
extra_args=["--verbose"],
@@ -259,14 +260,14 @@ class TestCPUExecution:
try:
ready = LlamafileRunner.wait_for_server(
- server_port, timeout=timeouts.server_ready
+ server_port, timeout=timeouts.server_ready, proc=proc
)
assert ready
# Verify response works
response = LlamafileRunner.chat_completion(
port=server_port,
- messages=[{"role": "user", "content": "Say hello"}],
+ messages=[{"role": "user", "content": GREETING_PROMPT}],
timeout=timeouts.http_request,
)
assert len(response["choices"][0]["message"]["content"]) > 0
diff --git a/tests/integration/tests/test_help.py b/tests/integration/tests/test_help.py
index 5f540634..acc5284e 100644
--- a/tests/integration/tests/test_help.py
+++ b/tests/integration/tests/test_help.py
@@ -15,6 +15,8 @@ import subprocess
import pytest
+from utils.llamafile import skip_if_bundled
+
HELP_TIMEOUT = 30.0
@@ -135,9 +137,23 @@ class TestHelpListsLlamaCppArgs:
@pytest.mark.help
class TestHelpMissingModel:
"""The missing-model error path should still print a helpful intro and
- point the user at --help, without requiring the full option list."""
+ point the user at --help, without requiring the full option list.
+
+ These tests require a bare llamafile binary (no embedded model). When run
+ against a pre-built .llamafile that already contains a model the binary
+ starts normally instead of printing an error, so the tests are skipped
+ automatically in that case. To exercise this code path, run against the
+ build output directly, e.g.::
+
+ ./run_tests.sh --executable ~/llamafile/o/llamafile/llamafile
+ """
+
+ @pytest.fixture(autouse=True)
+ def _require_bare_executable(self, executable):
+ skip_if_bundled(executable)
def _run_no_model(self, executable):
if platform.system() == "Windows":
diff --git a/tests/integration/tests/test_multimodal.py b/tests/integration/tests/test_multimodal.py
index 693afb54..55416660 100644
--- a/tests/integration/tests/test_multimodal.py
+++ b/tests/integration/tests/test_multimodal.py
@@ -128,7 +128,7 @@ class TestMultimodalServer:
try:
ready = LlamafileRunner.wait_for_server(
- server_port, timeout=timeouts.server_ready
+ server_port, timeout=timeouts.server_ready, proc=proc
)
assert ready, "Server did not become ready"
@@ -153,7 +153,7 @@ class TestMultimodalServer:
try:
ready = LlamafileRunner.wait_for_server(
- server_port, timeout=timeouts.server_ready
+ server_port, timeout=timeouts.server_ready, proc=proc
)
assert ready
diff --git a/tests/integration/tests/test_sandbox.py b/tests/integration/tests/test_sandbox.py
new file mode 100644
index 00000000..6b5f86a4
--- /dev/null
+++ b/tests/integration/tests/test_sandbox.py
@@ -0,0 +1,465 @@
+"""Sandbox (pledge/SECCOMP/unveil) integration tests.
+
+llamafile sandboxes itself with Cosmopolitan's pledge() and unveil() (issue
+#930). On Linux pledge() installs a SECCOMP BPF filter and unveil() uses
+Landlock. These tests verify externally that the sandbox is really in place:
+
+ - a pledged process carries one more SECCOMP filter than its parent
+ (/proc/<pid>/status ``Seccomp_filters``); the comparison is relative to
+ the pytest process because container runtimes (e.g. Docker's default
+ profile) already wrap everything in a filter of their own;
+ - every thread of the server carries that filter, not just the main one
+ (guards against a pre-existing thread escaping the per-thread filter);
+ - inference keeps working while sandboxed, including for a bundled
+ ``/zip/`` llamafile whose weights load through the executable;
+ - ``--unsecure``, combined mode, and GPU mode skip the sandbox and say so.
+
+pledge() is on by default; unveil() path confinement is opt-in via
+``--confine-reads`` (the default must NOT confine, so files opened by path at
+request time keep working). The confinement tests below check that the default
+does not confine and that ``--confine-reads`` does; the syscall-level proof
+that confinement denies reads outside the weights dirs is in sandbox_test.c.
+
+Enforcement is only observable on Linux; on other hosts the enforcement
+tests skip and we only check that llamafile reports its status honestly.
+tests/sandbox_test.c covers the syscall-level allow/deny behavior and the
+unveil() path confinement directly.
+"""
+
+import os
+import platform
+import shutil
+import subprocess
+import time
+from pathlib import Path
+
+import pytest
+
+from utils.llamafile import LlamafileRunner
+from utils.prompts import GREETING_PROMPT
+
+IS_LINUX = platform.system() == "Linux"
+
+requires_linux = pytest.mark.skipif(
+ not IS_LINUX, reason="SECCOMP enforcement is only observable on Linux"
+)
+
+
+def _status_field(pid_or_tid_path: str, field: str) -> int | None:
+ try:
+ with open(pid_or_tid_path) as f:
+ for line in f:
+ if line.startswith(field + ":"):
+ return int(line.split()[1])
+ except OSError:
+ pass
+ return None
+
+
+def _seccomp_filters(pid: int) -> int | None:
+ """Exact count of SECCOMP filters on <pid>, or None on kernels older
+ than 5.9 that don't expose ``Seccomp_filters``. Callers skip the strict
+ comparison when this is None rather than guessing from the coarse
+ ``Seccomp`` mode field (which can't tell a container filter apart from
+ our own)."""
+ return _status_field(f"/proc/{pid}/status", "Seccomp_filters")
+
+
+def _seccomp_mode(pid: int) -> int | None:
+ """The coarse ``Seccomp`` mode field: 2 (SECCOMP_MODE_FILTER) means at
+ least one filter is installed, 0 means none. Available on every kernel
+ with SECCOMP. Can't distinguish our filter from a container's, so it's
+ only a positive 'a filter exists' check, used as a fallback when the
+ exact count is unavailable so a fully-unsandboxed build still fails."""
+ return _status_field(f"/proc/{pid}/status", "Seccomp")
+
+
+def _assert_sandbox_installed(pid: int):
+ """A sandboxed process gains at least one filter beyond the runner's
+ baseline. Prefer the exact count; fall back to the mode field on old
+ kernels so a totally unsandboxed regression can't skip through."""
+ filters, base = _seccomp_filters(pid), _baseline_filters()
+ if filters is not None and base is not None:
+ assert filters > base, (
+ "expected the process to gain a SECCOMP filter; the pledge() "
+ "sandbox is not installed"
+ )
+ else:
+ assert _seccomp_mode(pid) == 2, (
+ "expected SECCOMP_MODE_FILTER; the pledge() sandbox is not installed"
+ )
+
+
+def _assert_no_added_filter(pid: int):
+ """The process carries no filter beyond the runner's baseline (the
+ sandbox was skipped). Needs the exact count to tell our filter from a
+ container's; skip the check where it's unavailable."""
+ filters, base = _seccomp_filters(pid), _baseline_filters()
+ if filters is None or base is None:
+ pytest.skip("kernel lacks Seccomp_filters; cannot count filters")
+ assert filters == base, "expected no added SECCOMP filter"
+
+
+def _thread_filter_counts(pid: int) -> dict[int, int]:
+ """Per-thread SECCOMP filter counts for every thread of <pid>."""
+ counts = {}
+ for tid in os.listdir(f"/proc/{pid}/task"):
+ n = _status_field(f"/proc/{pid}/task/{tid}/status", "Seccomp_filters")
+ if n is not None:
+ counts[int(tid)] = n
+ return counts
+
+
+def _baseline_filters() -> int | None:
+ """SECCOMP filters every spawned process inherits from this test runner
+ (0 on bare metal, typically 1 under Docker's default profile)."""
+ return _seccomp_filters(os.getpid())
+
+
+def _resolve_server_pid(pid: int) -> int:
+ """Follow the sh -> APE-loader chain down to the llamafile process.
+
+ The runner launches ``sh <executable>``. The APE shell header normally
+ exec()s in place so the PID stays the same, but if any stage forked we
+ walk single-child links downward.
+ """
+ for _ in range(5):
+ children = []
+ try:
+ for tid in os.listdir(f"/proc/{pid}/task"):
+ with open(f"/proc/{pid}/task/{tid}/children") as f:
+ children.extend(int(p) for p in f.read().split())
+ except OSError:
+ break
+ if not children:
+ break
+ pid = children[0]
+ return pid
+
+
+def _stop_hard(proc):
+ """Terminate a process without ever hanging the test: SIGTERM first
+ (llamafile shuts down gracefully), SIGKILL if it doesn't oblige."""
+ proc.terminate()
+ try:
+ proc.wait(timeout=15)
+ except subprocess.TimeoutExpired:
+ proc.kill()
+ proc.wait()
+
+
+def cpu_runner(executable, model) -> LlamafileRunner:
+ """Runner pinned to CPU, with --no-warmup.
+
+ --gpu disable: a loaded GPU backend disables the sandbox (drivers need
+ syscalls pledge forbids), so CPU is forced to make sandbox state
+ deterministic on any host.
+
+ --no-warmup: large models (e.g. 26B MXFP4 MoE) can take longer than the
+ 120s server-ready timeout when the kernel page cache is cold, because
+ pledge may restrict the prefetch hints (posix_fadvise / MADV_WILLNEED)
+ that speed up the initial memory-mapped read. Sandbox tests care about
+ whether the pledge filter is installed and whether inference works, not
+ about the warmup pass itself.
+ """
+ return LlamafileRunner(executable=executable, model=model, gpu="disable", no_warmup=True)
+
+
+class TestServerSandbox:
+ @requires_linux
+ def test_server_sandboxed_by_default(self, cpu_runner, server_port, timeouts, tmp_path):
+ """--server installs a SECCOMP filter on every thread and still
+ serves completions inside the sandbox."""
+ log_file = str(tmp_path / "server.log")
+ proc = cpu_runner.start_server(port=server_port, log_file=log_file)
+ try:
+ assert LlamafileRunner.wait_for_server(
+ server_port, timeout=timeouts.server_ready, proc=proc
+ ), "Server did not become ready"
+
+ pid = _resolve_server_pid(proc.pid)
+ # The sandbox adds at least the pledge filter (and, when unveil
+ # confinement applies, cosmo's extra truncate/setxattr filter);
+ # the exact count is cosmo's business, so assert presence not a
+ # specific delta. The negative cases below pin the baseline.
+ _assert_sandbox_installed(pid)
+
+ # Every thread must carry the same filter count, not just the
+ # main thread: a pre-existing background thread (e.g. the log
+ # worker) would otherwise escape the per-thread SECCOMP filter
+ # (issue #930). Needs the exact per-thread count (5.9+); where
+ # unavailable the presence check above still holds.
+ counts = _thread_filter_counts(pid)
+ if counts:
+ base = _baseline_filters()
+ assert set(counts.values()) == {max(counts.values())}, (
+ f"threads have mismatched SECCOMP filter counts {counts}; "
+ f"a thread escaped the sandbox"
+ )
+ if base is not None:
+ assert max(counts.values()) > base, "no thread is sandboxed"
+
+ # inference must keep working inside the sandbox
+ response = LlamafileRunner.chat_completion(
+ port=server_port,
+ messages=[{"role": "user", "content": GREETING_PROMPT}],
+ timeout=timeouts.http_request,
+ )
+ assert response["choices"][0]["message"]["content"].strip()
+ finally:
+ _stop_hard(proc)
+
+ log = LlamafileRunner.read_log_file(log_file)
+ assert 'sandbox: pledge("' in log, "server should log the active promise set"
+
+ @requires_linux
+ def test_unsecure_flag_disables_sandbox(self, cpu_runner, server_port, timeouts, tmp_path):
+ """--unsecure opts out: no SECCOMP filter, and the log says why."""
+ log_file = str(tmp_path / "server.log")
+ proc = cpu_runner.start_server(
+ port=server_port, log_file=log_file, extra_args=["--unsecure"]
+ )
+ try:
+ assert LlamafileRunner.wait_for_server(
+ server_port, timeout=timeouts.server_ready, proc=proc
+ ), "Server did not become ready"
+ _assert_no_added_filter(_resolve_server_pid(proc.pid))
+ finally:
+ _stop_hard(proc)
+
+ log = LlamafileRunner.read_log_file(log_file)
+ assert "disabled by --unsecure" in log
+
+ def test_sandbox_status_always_logged(self, cpu_runner, server_port, timeouts, tmp_path):
+ """On every platform the server reports its sandbox state honestly
+ (active on Linux/OpenBSD, an explicit skip reason elsewhere)."""
+ log_file = str(tmp_path / "server.log")
+ proc = cpu_runner.start_server(port=server_port, log_file=log_file)
+ try:
+ assert LlamafileRunner.wait_for_server(
+ server_port, timeout=timeouts.server_ready, proc=proc
+ ), "Server did not become ready"
+ finally:
+ _stop_hard(proc)
+
+ log = LlamafileRunner.read_log_file(log_file)
+ assert "sandbox:" in log, "server must log its sandbox status"
+ if not IS_LINUX:
+ assert "not supported on this OS" in log
+
+
+class TestConfineReads:
+ """The pledge()/unveil() split: pledge() is on by default, unveil() path
+ confinement is opt-in via --confine-reads."""
+
+ @requires_linux
+ def test_default_does_not_confine(self, cpu_runner, server_port, timeouts, tmp_path):
+ """By default the server is pledged but NOT path-confined, so files
+ opened by path at request time (multimodal media, etc.) keep working.
+ The log must say active without claiming confinement."""
+ log_file = str(tmp_path / "server.log")
+ proc = cpu_runner.start_server(port=server_port, log_file=log_file)
+ try:
+ assert LlamafileRunner.wait_for_server(
+ server_port, timeout=timeouts.server_ready, proc=proc
+ ), "Server did not become ready"
+ finally:
+ _stop_hard(proc)
+ log = LlamafileRunner.read_log_file(log_file)
+ assert 'pledge("stdio anet rpath")' in log, "default should pledge, no confinement"
+ assert "confined to weights dirs" not in log, "default must not confine reads"
+
+ @requires_linux
+ def test_confine_reads_confines_and_serves(self, cpu_runner, server_port, timeouts, tmp_path):
+ """--confine-reads adds unveil() confinement and still serves."""
+ log_file = str(tmp_path / "server.log")
+ proc = cpu_runner.start_server(
+ port=server_port, log_file=log_file, extra_args=["--confine-reads"]
+ )
+ try:
+ assert LlamafileRunner.wait_for_server(
+ server_port, timeout=timeouts.server_ready, proc=proc
+ ), "Server did not become ready"
+ _assert_sandbox_installed(_resolve_server_pid(proc.pid))
+ resp = LlamafileRunner.chat_completion(
+ port=server_port,
+ messages=[{"role": "user", "content": GREETING_PROMPT}],
+ timeout=timeouts.http_request,
+ )
+ assert resp["choices"][0]["message"]["content"].strip()
+ finally:
+ _stop_hard(proc)
+ log = LlamafileRunner.read_log_file(log_file)
+ # This log line is emitted only after the governability probe confirms
+ # a canary outside the rules is actually denied (see sandbox.c), so it
+ # means confinement is installed and biting -- not merely requested.
+ assert "reads confined to weights dirs" in log
+
+ # Note: there is no integration test that drives a *live* server into
+ # reading a file outside the weights dirs, because the HTTP API exposes no
+ # arbitrary-read primitive to exploit -- static serving (httplib) and media
+ # loading (fs_validate_filename) both reject traversal/symlinks before a
+ # read is attempted, so unveil() never gets a chance to be the thing that
+ # denies. unveil() is the defense-in-depth backstop *behind* those checks;
+ # its enforcement (allowed inside, EACCES outside) is proven directly at
+ # the syscall level in tests/sandbox_test.c (child_unveil).
+
+
+def _find_zipalign(executable: str) -> str | None:
+ """Locate the zipalign tool for building a bundled llamafile: env
+ override, then the sibling build tree of a direct build, then PATH."""
+ env = os.environ.get("LLAMAFILE_ZIPALIGN")
+ if env and os.path.exists(env):
+ return env
+ for parent in Path(executable).resolve().parents:
+ cand = parent / "third_party" / "zipalign" / "zipalign"
+ if cand.exists():
+ return str(cand)
+ return shutil.which("zipalign")
+
+
+class TestBundledLlamafileSandbox:
+ @requires_linux
+ def test_embedded_zip_model_loads_under_sandbox(
+ self, executable, model, server_port, timeouts, tmp_path
+ ):
+ """A bundled llamafile references its weights as ``/zip/model.gguf``;
+ the loader reads them by reopening the executable's zip store, which
+ needs the rpath promise. This is the exact configuration issue #930's
+ rpath fix targets — a naive 'embedded => no filesystem' policy makes
+ the model fail to load. Build a real bundle and assert it serves."""
+ if not model:
+ pytest.skip("needs --model to embed into a bundle")
+ zipalign = _find_zipalign(executable)
+ if not zipalign:
+ pytest.skip("zipalign not found (set LLAMAFILE_ZIPALIGN)")
+
+ bundle = tmp_path / "bundled.llamafile"
+ shutil.copy(executable, bundle)
+ os.chmod(bundle, 0o755)
+ model_name = os.path.basename(model)
+ args = tmp_path / ".args"
+ args.write_text(f"-m\n/zip/{model_name}\n--server\n--gpu\ndisable\n")
+ # zipalign is an APE binary; run it through sh like the llamafile
+ # itself, so hosts without binfmt_misc for APE still work.
+ za = ["sh", zipalign] if os.name != "nt" else [zipalign]
+ subprocess.run(za + ["-j0", str(bundle), model, str(args)], check=True)
+
+ log_file = str(tmp_path / "bundled.log")
+ proc = subprocess.Popen(
+ ["sh", str(bundle), "--port", str(server_port), "--log-file", log_file],
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
+ )
+ try:
+ assert LlamafileRunner.wait_for_server(
+ server_port, timeout=timeouts.server_ready, proc=proc
+ ), "bundled /zip/ server failed to load under the sandbox"
+ finally:
+ _stop_hard(proc)
+
+ log = LlamafileRunner.read_log_file(log_file)
+ assert 'sandbox: pledge("' in log
+ assert "rpath" in log, "embedded weights still need the rpath promise"
+
+
+class TestCliSandbox:
+ @requires_linux
+ def test_cli_sandboxed_and_functional(self, cpu_runner, timeouts):
+ """--cli pledges "stdio rpath tty" after arg parsing and still
+ generates a response. --verbose surfaces the status line on
+ stderr (CLI keeps stdout clean for the model output)."""
+ result = cpu_runner.run_cli(
+ GREETING_PROMPT, extra_args=["--verbose"], timeout=timeouts.cli
+ )
+ assert result.returncode == 0
+ assert "sandbox: active" in result.stderr
+ assert result.stdout.strip()
+
+
+class TestChatSandbox:
+ @requires_linux
+ def test_chat_all_threads_sandboxed(self, cpu_runner, timeouts, tmp_path):
+ """Chat mode wraps the pledge in log pause/resume like the server,
+ so no pre-existing thread (the log worker) escapes the per-thread
+ filter. Launch headless with stdin held open so the REPL blocks
+ after sandboxing, then confirm every thread carries the filter
+ (issue #930 — the chat/cli path the round-1 tests missed)."""
+ # The "sandbox:" status is written to stderr (not the common-log
+ # --log-file), so capture stderr to detect when the pledge is in.
+ err_file = tmp_path / "chat.err"
+ args = cpu_runner._base_args() + ["--chat", "--verbose", "--nologo"]
+ with open(err_file, "wb") as err:
+ proc = subprocess.Popen(
+ args, stdin=subprocess.PIPE,
+ stdout=subprocess.DEVNULL, stderr=err,
+ )
+ try:
+ deadline = time.time() + timeouts.server_ready
+ ready = False
+ while time.time() < deadline:
+ if proc.poll() is not None:
+ pytest.fail("chat exited before reaching the sandbox step")
+ if "sandbox:" in err_file.read_text(errors="replace"):
+ ready = True
+ break
+ time.sleep(0.2)
+ assert ready, "chat did not reach the sandbox step"
+
+ pid = _resolve_server_pid(proc.pid)
+ _assert_sandbox_installed(pid)
+ counts = _thread_filter_counts(pid)
+ if counts:
+ assert set(counts.values()) == {max(counts.values())}, (
+ f"chat threads have mismatched SECCOMP filter counts "
+ f"{counts}; a thread (e.g. the log worker) escaped"
+ )
+ finally:
+ _stop_hard(proc)
+
+ assert "sandbox: active" in err_file.read_text(errors="replace")
+
+
+class TestCombinedModeSandbox:
+ @requires_linux
+ def test_combined_mode_skips_sandbox(self, cpu_runner, server_port, timeouts, tmp_path):
+ """Combined mode hosts a TUI HTTP client in-process, which needs
+ connect(); the accept-only sandbox is skipped and logged.
+
+ --verbose is required: without it main.cpp injects --log-verbosity 1
+ (ERROR-only threshold), suppressing the WARN-level skip notice even
+ in the log file. The log file (not a pipe) is used for output so
+ --verbose does not cause a pipe-buffer overflow.
+ """
+ log_file = str(tmp_path / "combined.log")
+ proc = cpu_runner.start_combined(
+ port=server_port, log_file=log_file, extra_args=["--verbose"],
+ )
+ try:
+ assert LlamafileRunner.wait_for_server(
+ server_port, timeout=timeouts.server_ready, proc=proc
+ ), "Combined-mode server did not become ready"
+ _assert_no_added_filter(_resolve_server_pid(proc.pid))
+ finally:
+ _stop_hard(proc)
+
+ log = LlamafileRunner.read_log_file(log_file)
+ assert "disabled in combined mode" in log
diff --git a/tests/integration/tests/test_server.py b/tests/integration/tests/test_server.py
index 1215e4f1..ec07406e 100644
--- a/tests/integration/tests/test_server.py
+++ b/tests/integration/tests/test_server.py
@@ -3,6 +3,7 @@
import pytest
from utils.llamafile import LlamafileRunner
+from utils.prompts import ADD_2_2, GREETING_PROMPT
@pytest.mark.server
@@ -15,7 +16,7 @@ class TestServerBasic:
try:
ready = LlamafileRunner.wait_for_server(
- server_port, timeout=timeouts.server_ready
+ server_port, timeout=timeouts.server_ready, proc=proc
)
assert ready, "Server did not become ready in time"
finally:
@@ -28,13 +29,13 @@ class TestServerBasic:
try:
ready = LlamafileRunner.wait_for_server(
- server_port, timeout=timeouts.server_ready
+ server_port, timeout=timeouts.server_ready, proc=proc
)
assert ready, "Server did not become ready"
response = LlamafileRunner.chat_completion(
port=server_port,
- messages=[{"role": "user", "content": "Say hello in one word."}],
+ messages=[{"role": "user", "content": GREETING_PROMPT}],
timeout=timeouts.http_request,
)
@@ -53,20 +54,18 @@ class TestServerBasic:
try:
ready = LlamafileRunner.wait_for_server(
- server_port, timeout=timeouts.server_ready
+ server_port, timeout=timeouts.server_ready, proc=proc
)
assert ready, "Server did not become ready"
response = LlamafileRunner.chat_completion(
port=server_port,
- messages=[
- {"role": "user", "content": "What is 2+2? Answer with just the number."}
- ],
+ messages=[{"role": "user", "content": ADD_2_2.prompt}],
timeout=timeouts.http_request,
)
content = response["choices"][0]["message"]["content"]
- assert "4" in content
+ assert ADD_2_2.check(content), f"Expected {ADD_2_2.describe()} in content: {content}"
finally:
proc.terminate()
@@ -78,13 +77,14 @@ class TestServerBasic:
class TestServerParameters:
"""Test server with various parameters."""
+ @pytest.mark.determinism
def test_server_with_temperature_zero(self, llamafile, server_port, timeouts):
"""Test that temperature=0 produces consistent output."""
proc = llamafile.start_server(port=server_port)
try:
ready = LlamafileRunner.wait_for_server(
- server_port, timeout=timeouts.server_ready
+ server_port, timeout=timeouts.server_ready, proc=proc
)
assert ready
@@ -130,7 +130,7 @@ class TestServerParameters:
try:
ready = LlamafileRunner.wait_for_server(
- server_port, timeout=timeouts.server_ready
+ server_port, timeout=timeouts.server_ready, proc=proc
)
assert ready
diff --git a/tests/integration/tests/test_ssl.py b/tests/integration/tests/test_ssl.py
index ddc9a5c8..63d5a44f 100644
--- a/tests/integration/tests/test_ssl.py
+++ b/tests/integration/tests/test_ssl.py
@@ -1,8 +1,22 @@
"""HTTPS/TLS integration tests.
-Covers the TLS support added via cpp-httplib's Mbed TLS backend:
-serving over HTTPS (--ssl-cert-file/--ssl-key-file), downloading models
-over HTTPS (-hf/--model-url), and rejection of untrusted certificates.
+Covers two distinct uses of TLS in llamafile:
+
+1. **Serving over HTTPS** (--ssl-cert-file / --ssl-key-file): the server
+ presents a certificate to connecting clients.
+
+2. **Downloading models over HTTPS** (-hf / --model-url): llamafile acts
+ as an HTTP client to fetch a model before serving it.
+
+Note on client-side certificate verification
+--------------------------------------------
+llamafile does **not** verify TLS certificates when downloading models.
+This matches upstream llama.cpp behaviour: ``common/http.h`` was introduced
+in ggml-org/llama.cpp#16373 without verification, and PR #18828 (Jan 2026)
+removed libcurl — which used the system CA store by default — replacing it
+with a bare cpp-httplib client that calls neither ``set_ca_cert_path()``
+nor any equivalent. No open issue or PR in the upstream repo tracks this
+gap, so there is no expectation of a fix in the near term.
Toggle with the ``ssl`` marker: ``-m ssl`` / ``-m "not ssl"``.
``test_download_model_over_https`` needs network access (huggingface.co)
@@ -18,7 +32,7 @@ import subprocess
import pytest
import requests
-from utils.llamafile import LlamafileRunner
+from utils.llamafile import LlamafileRunner, skip_if_bundled
# Tiny model used for the network download test (~1 MiB)
HF_TEST_REPO = "ggml-org/models"
@@ -96,10 +110,17 @@ def fresh_cache_env(cache_dir):
@pytest.mark.ssl
@pytest.mark.server
class TestHttpsServing:
"""Serving over TLS with --ssl-cert-file/--ssl-key-file."""
+ @pytest.fixture(autouse=True)
+ def _require_bare_executable(self, executable, model):
+ skip_if_bundled(executable)
+ if model is None:
+ pytest.skip("ssl serving tests need --model; pass a .gguf with the bare executable")
+
def test_server_serves_https(self, llamafile, server_port, timeouts, ssl_cert):
"""Server comes up over HTTPS and a verifying client can connect."""
cert, key = ssl_cert
@@ -146,9 +167,14 @@ class TestHttpsServing:
@pytest.mark.ssl
class TestHttpsDownload:
"""Model downloads over HTTPS (the client side of TLS support)."""
+ @pytest.fixture(autouse=True)
+ def _require_bare_executable(self, executable):
+ skip_if_bundled(executable)
+
@pytest.mark.online
def test_download_model_over_https(
self, executable, server_port, timeouts, tmp_path
@@ -175,7 +201,7 @@ class TestHttpsDownload:
)
try:
assert LlamafileRunner.wait_for_server(
- server_port, timeout=timeouts.server_ready
+ server_port, timeout=timeouts.server_ready, proc=proc
), "Server did not become ready (HTTPS download failed?)"
if platform.system() != "Windows":
@@ -190,48 +216,3 @@ class TestHttpsDownload:
proc.terminate()
proc.wait()
- def test_download_rejects_untrusted_cert(
- self, llamafile, executable, server_port, timeouts, ssl_cert, tmp_path
- ):
- """Downloading from a server with an untrusted (self-signed) cert fails.
-
- Runs fully offline: our own HTTPS server plays the untrusted host.
- Its /health endpoint would happily serve a body, so if the client's
- certificate verification were broken the download would 'succeed'
- and leave a file in the cache; instead the TLS handshake must fail
- and the cache must stay empty.
- """
- cert, key = ssl_cert
- rogue_port = server_port + 1
- server = llamafile.start_server(
- port=rogue_port,
- extra_args=["--ssl-cert-file", str(cert), "--ssl-key-file", str(key)],
- )
- try:
- assert wait_for_https_server(
- rogue_port, cert, timeout=timeouts.server_ready
- ), "TLS server did not become ready"
-
- cache = tmp_path / "cache"
- cache.mkdir()
- args = base_exe_args(executable) + [
- "--server",
- "--port", str(server_port),
- "--model-url", f"https://127.0.0.1:{rogue_port}/health",
- ]
- client = subprocess.run(
- args,
- capture_output=True,
- text=True,
- env=fresh_cache_env(cache),
- timeout=timeouts.server_ready,
- )
- assert client.returncode != 0, "Download from untrusted cert host succeeded"
- leftovers = [p for p in cache.rglob("*") if p.is_file()]
- assert not leftovers, (
- "Client fetched data from a server with an untrusted certificate: "
- f"{leftovers}"
- )
- finally:
- server.terminate()
- server.wait()
diff --git a/tests/integration/tests/test_tool_calling.py b/tests/integration/tests/test_tool_calling.py
index ce4669d0..452c0f40 100644
--- a/tests/integration/tests/test_tool_calling.py
+++ b/tests/integration/tests/test_tool_calling.py
@@ -56,7 +56,7 @@ class TestToolCalling:
try:
ready = LlamafileRunner.wait_for_server(
- server_port, timeout=timeouts.server_ready
+ server_port, timeout=timeouts.server_ready, proc=proc
)
assert ready
@@ -86,12 +86,12 @@ class TestToolCalling:
proc.wait()
def test_tool_call_correct_function(self, llamafile, server_port, timeouts):
- """Test that model calls the correct tool."""
+ """Test that model calls the correct tool when two are available."""
proc = llamafile.start_server(port=server_port)
try:
ready = LlamafileRunner.wait_for_server(
- server_port, timeout=timeouts.server_ready
+ server_port, timeout=timeouts.server_ready, proc=proc
)
assert ready
@@ -107,10 +107,13 @@ class TestToolCalling:
message = response["choices"][0]["message"]
- if "tool_calls" in message and len(message["tool_calls"]) > 0:
- tool_call = message["tool_calls"][0]
- # Should call weather, not calculator
- assert tool_call["function"]["name"] == "get_weather"
+ assert "tool_calls" in message and len(message["tool_calls"]) > 0, (
+ f"Expected a tool call. Got: {message}"
+ )
+ # Should call weather, not calculator
+ assert message["tool_calls"][0]["function"]["name"] == "get_weather", (
+ f"Expected get_weather, got: {message['tool_calls'][0]['function']['name']}"
+ )
finally:
proc.terminate()
@@ -122,7 +125,7 @@ class TestToolCalling:
try:
ready = LlamafileRunner.wait_for_server(
- server_port, timeout=timeouts.server_ready
+ server_port, timeout=timeouts.server_ready, proc=proc
)
assert ready
diff --git a/tests/integration/tests/test_tui.py b/tests/integration/tests/test_tui.py
index 1281e9a5..f1d7de6c 100644
--- a/tests/integration/tests/test_tui.py
+++ b/tests/integration/tests/test_tui.py
@@ -2,6 +2,8 @@
import pytest
+from utils.prompts import ADD_2_2
+
@pytest.mark.tui
class TestTUIBasic:
@@ -21,13 +23,14 @@ class TestTUIBasic:
def test_tui_math_question(self, llamafile, tmp_path, timeouts):
"""Test TUI with a simple math question."""
input_file = tmp_path / "input.txt"
- input_file.write_text("What is 2+2?\n/exit\n")
+ input_file.write_text(f"{ADD_2_2.prompt}\n/exit\n")
result = llamafile.run_tui(str(input_file), timeout=timeouts.tui)
assert result.returncode == 0
- # The response should contain "4" somewhere
- assert "4" in result.stdout
+ assert ADD_2_2.check(result.stdout), (
+ f"Expected {ADD_2_2.describe()} in output: {result.stdout}"
+ )
def test_tui_multi_turn(self, llamafile, tmp_path, timeouts):
"""Test TUI with multiple turns of conversation."""
diff --git a/tests/integration/utils/llamafile.py b/tests/integration/utils/llamafile.py
index dc8fe2c2..d4508882 100644
--- a/tests/integration/utils/llamafile.py
+++ b/tests/integration/utils/llamafile.py
@@ -11,11 +11,28 @@ import time
from pathlib import Path
from typing import Any
+import pytest
import requests
logger = logging.getLogger(__name__)
+def skip_if_bundled(executable: str) -> None:
+ """Skip the current test when *executable* is a pre-built .llamafile bundle.
+
+ Call from an autouse fixture in any class marked ``bare_executable``::
+
+ @pytest.fixture(autouse=True)
+ def _require_bare_executable(self, executable):
+ skip_if_bundled(executable)
+ """
+ if executable.endswith(".llamafile"):
+ pytest.skip(
+ "bare_executable tests run once against the bare llamafile binary; "
+ "skip for bundled .llamafile models"
+ )
+
+
def read_until_idle(fd, idle_timeout=1.0, max_timeout=60.0):
"""Read from file descriptor until output stops (model finished generating).
@@ -116,6 +133,7 @@ class LlamafileRunner:
model: str | None = None,
gpu: str | None = None,
default_thinking: bool = False,
+ no_warmup: bool = False,
):
"""Initialize the runner.
@@ -127,11 +145,16 @@ class LlamafileRunner:
one. Set to True for tests that exercise thinking
behavior; False otherwise so small reasoning
models don't derail unrelated assertions.
+ no_warmup: Pass --no-warmup to every server/combined start. Use
+ for tests that care about the server being up, not about
+ warmup speed -- prevents cold-cache timeouts for large
+ models without affecting inference correctness.
"""
self.executable = os.path.abspath(executable)
self.model = os.path.abspath(model) if model else None
self.gpu = gpu
self.default_thinking = default_thinking
+ self.no_warmup = no_warmup
if not os.path.exists(self.executable):
raise FileNotFoundError(f"Executable not found: {executable}")
@@ -311,6 +334,9 @@ class LlamafileRunner:
if log_file:
args.extend(["--log-file", log_file])
+ if self.no_warmup:
+ args.append("--no-warmup")
+
if extra_args:
args.extend(extra_args)
@@ -337,8 +363,13 @@ class LlamafileRunner:
None, falls back to the runner's default_thinking.
When False, adds --reasoning off.
extra_args: Additional command-line arguments
- log_file: If provided, adds --log-file flag. Caller should read
- the file after terminating the process.
+ log_file: If provided, adds --log-file flag and redirects
+ stdout/stderr to DEVNULL. When a log file captures
+ all diagnostics there is no need to pipe stdout/stderr,
+ and leaving them as PIPE without a reader causes the
+ OS pipe buffer (~64 KiB) to fill and the process to
+ block indefinitely. Omit log_file when you need to
+ interact with the TUI via proc.stdout / proc.stdin.
Returns:
Popen process handle (caller must terminate)
@@ -352,15 +383,28 @@ class LlamafileRunner:
if log_file:
args.extend(["--log-file", log_file])
+ if self.no_warmup:
+ args.append("--no-warmup")
+
if extra_args:
args.extend(extra_args)
+ # When a log file is used, discard stdout/stderr via DEVNULL to avoid
+ # blocking when the pipe buffer fills up (nobody is reading the pipe).
+ # When no log file is set, use PIPE so callers can read TUI output.
+ if log_file:
+ stdout = subprocess.DEVNULL
+ stderr = subprocess.DEVNULL
+ else:
+ stdout = subprocess.PIPE
+ stderr = subprocess.PIPE
+
logger.info("Starting combined mode: %s", " ".join(args))
return subprocess.Popen(
args,
stdin=subprocess.PIPE,
- stdout=subprocess.PIPE,
- stderr=subprocess.PIPE,
+ stdout=stdout,
+ stderr=stderr,
text=True,
)
@@ -384,6 +428,7 @@ class LlamafileRunner:
@staticmethod
def wait_for_server(
port: int,
+ proc: subprocess.Popen,
host: str = "127.0.0.1",
timeout: float = TIMEOUT_SERVER_READY,
poll_interval: float = POLL_INTERVAL,
@@ -392,18 +437,30 @@ class LlamafileRunner:
Args:
port: Server port
+ proc: The server process. When it exits before the server
+ becomes ready (e.g. a bare build launched without
+ --model errors out at startup), give up immediately
+ instead of polling a dead port for the full timeout.
host: Server host
timeout: Maximum time to wait in seconds
poll_interval: Time between health checks
Returns:
- True if server is ready, False if timeout
+ True if server is ready, False if it exited early or timed out
"""
url = f"http://{host}:{port}/health"
start_time = time.time()
logger.info("Waiting for server at %s (timeout=%.1fs)", url, timeout)
while time.time() - start_time < timeout:
+ if proc.poll() is not None:
+ logger.warning(
+ "Server process exited before becoming ready (rc=%s). "
+ "If this is a bare llamafile build, did you pass "
+ "--model / LLAMAFILE_MODEL?",
+ proc.returncode,
+ )
+ return False
try:
response = requests.get(url, timeout=2)
if response.status_code == 200:
diff --git a/tests/integration/utils/prompts.py b/tests/integration/utils/prompts.py
new file mode 100644
index 00000000..2b3d8614
--- /dev/null
+++ b/tests/integration/utils/prompts.py
@@ -0,0 +1,64 @@
+"""Shared prompts and answer-checkers for liveness-style integration tests.
+
+These prompts don't test model capability (math, instruction-following) --
+they exist to confirm a mode is up and producing a coherent, on-topic
+response. Keeping them here means updating a prompt or its accepted answers
+happens in one place instead of drifting across test files.
+"""
+
+import re
+
+NUMBER_WORDS = {
+ 0: "zero", 1: "one", 2: "two", 3: "three", 4: "four",
+ 5: "five", 6: "six", 7: "seven", 8: "eight", 9: "nine", 10: "ten",
+}
+
+
+def contains_number(text: str, n: int) -> bool:
+ """Check whether *text* contains *n* as a digit or spelled-out word.
+
+ Matches on word boundaries so checking for 6 doesn't match "sixteen" and
+ checking for 2 doesn't match "24".
+ """
+ forms = [str(n)]
+ if n in NUMBER_WORDS:
+ forms.append(NUMBER_WORDS[n])
+ pattern = r"\b(?:" + "|".join(re.escape(f) for f in forms) + r")\b"
+ return re.search(pattern, text, re.IGNORECASE) is not None
+
+
+class MathPrompt:
+ """A simple addition fact used as a liveness/coherence check.
+
+ Not a math-capability test: models may answer in digits or words, so
+ the check accepts either.
+ """
+
+ def __init__(self, prompt: str, answer: int):
+ self.prompt = prompt
+ self.answer = answer
+
+ def check(self, text: str) -> bool:
+ return contains_number(text, self.answer)
+
+ def describe(self) -> str:
+ return f"{self.answer} (digit or word form)"
+
+
+# Canonical single-shot liveness prompt -- use this one everywhere a test
+# just needs to confirm the model produced a sane, on-topic answer.
+ADD_2_2 = MathPrompt("What is 2+2?", 4)
+
+# Reserved for test_combined's multi-step test, which makes three sequential
+# calls (server, TUI, server) in one test run. Using distinct facts there is
+# deliberate: it's how that test catches cross-talk (e.g. the TUI reading the
+# server's response, or a stale/cached answer) that reusing ADD_2_2 three
+# times would mask, since a "4" would look correct regardless of which call
+# it actually came from.
+ADD_1_1 = MathPrompt("What is 1+1?", 2)
+ADD_3_3 = MathPrompt("What is 3+3?", 6)
+
+# Canonical greeting liveness prompt -- confirms a mode responds with some
+# non-empty, coherent output. The exact wording doesn't matter (these tests
+# only check the response is non-empty), so one wording is reused everywhere.
+GREETING_PROMPT = "Say hello in one word."
diff --git a/tests/run_integration_tests.sh b/tests/run_integration_tests.sh
new file mode 100755
index 00000000..8ff4601f
--- /dev/null
+++ b/tests/run_integration_tests.sh
@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+
+# Each entry: "<llamafile-name>;<-m filter expression>"
+# Leave the filter empty to run without -m.
+TESTS=(
+ "Apertus-8B-Instruct-2509.llamafile;not thinking and not multimodal"
+ "Bonsai-1.7B.llamafile;not thinking and not multimodal"
+ "Bonsai-4B.llamafile;not thinking and not multimodal and not tool_calling" # upstream llama.cpp peg-native parser fails to convert <tool_call> output to tool_calls for this model
+ "Bonsai-8B.llamafile;not thinking and not multimodal"
+ "LFM2-24B-A2B-Q5_K_M.llamafile;not thinking and not multimodal"
+ "Ministral-3-3B-Instruct-2512-BF16.llamafile;not thinking"
+ "Ministral-3-3B-Instruct-2512-Q4_K_M.llamafile;not thinking"
+ "Qwen3.5-0.8B-Q8_0.llamafile;not thinking"
+ "Qwen3.5-2B-Q8_0.llamafile;not thinking"
+ "Qwen3.5-4B-Q5_K_S.llamafile;not thinking"
+ "Qwen3.5-9B-Q5_K_S.llamafile;"
+ "Qwen3.6-27B-Q4_K_M.llamafile;not cpu"
+ "gemma-4-E2B-it-Q5_K_M.llamafile;"
+ "gemma-4-E4B-it-Q5_K_M.llamafile;"
+ "gemma-4-26B-A4B-it-MXFP4_MOE.llamafile;not cpu"
+ "gemma-4-31B-it-Q5_K_M.llamafile;not cpu"
+ # gpt-oss models do not really disable thinking, they just put it inline so some tests fail
+ # they also occasionally fail the temperature=0 determinism check: their long forced
+ # reasoning trace + MXFP4 quantization give a threaded matmul rounding difference more
+ # opportunities to flip a near-tied token over a long generation
+ "gpt-oss-20b-Q5_K_S.llamafile;not cpu and not multimodal and not thinking and not determinism"
+ "gpt-oss-20b-mxfp4.llamafile;not cpu and not multimodal and not thinking and not determinism"
+ "llava-v1.6-mistral-7b-Q4_K_M.llamafile;not thinking and not tool_calling"
+ "llava-v1.6-mistral-7b-Q8_0.llamafile;not thinking and not tool_calling"
+)
+
+LLAMAFILE_EXE=~/llamafile/o/llamafile/llamafile
+RUNNER=~/llamafile/tests/integration/run_tests.sh
+MODELS_DIR=~/llamafiles
+
+# Model used for the ssl serving tests (any small .gguf works; content is irrelevant).
+# Override via SSL_TEST_MODEL env var if you have a different one available.
+SSL_TEST_MODEL="${SSL_TEST_MODEL:-${MODELS_DIR}/Qwen3.5-0.8B-Q8_0.gguf}"
+
+for entry in "${TESTS[@]}"; do
+ model="${entry%%;*}"
+ filter="${entry#*;}"
+
+ echo "=== Testing ${model} ==="
+ if [ -n "${filter}" ]; then
+ "${RUNNER}" --executable "${MODELS_DIR}/${model}" -m "${filter}"
+ else
+ "${RUNNER}" --executable "${MODELS_DIR}/${model}"
+ fi
+done
+
+echo "=== Running help tests (on bare executable: ${LLAMAFILE_EXE} ) ==="
+"${RUNNER}" --executable "${LLAMAFILE_EXE}" -m help
+
+echo "=== Running ssl tests (bare executable, network download) ==="
+${RUNNER} --executable "${LLAMAFILE_EXE}" -m "ssl and online"
+
+echo "=== Running ssl tests (on bare executable: ${LLAMAFILE_EXE}, model: ${SSL_TEST_MODEL}) ==="
+"${RUNNER}" --executable "${LLAMAFILE_EXE}" --model "${SSL_TEST_MODEL}" -m "ssl and not online"
diff --git a/tests/sandbox_test.c b/tests/sandbox_test.c
new file mode 100644
index 00000000..fec0f2c2
--- /dev/null
+++ b/tests/sandbox_test.c
@@ -0,0 +1,275 @@
+// -*- mode:c;indent-tabs-mode:nil;c-basic-offset:4;coding:utf-8 -*-
+// vi: set et ft=c ts=4 sts=4 sw=4 fenc=utf-8 :vi
+//
+// Copyright 2026 Mozilla.ai
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//
+// Verifies pledge()/SECCOMP sandboxing (llamafile/sandbox.c, issue #930).
+//
+// Each case forks a child that installs one of the promise sets llamafile
+// actually uses, then checks from inside the sandbox that:
+// - what must keep working still works (read-only opens under rpath,
+// accept() under anet, thread creation under stdio), and
+// - what must be blocked fails with EPERM instead of killing the
+// process (PLEDGE_PENALTY_RETURN_EPERM), exactly what the server
+// and CLI rely on.
+//
+// pledge() can only be enforced on Linux (SECCOMP) and OpenBSD. On other
+// platforms this test reports SKIP and exits 0.
+//
+
+#include "llamafile/llamafile.h"
+
+#include <arpa/inet.h>
+#include <cosmo.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <libc/calls/calls.h> // unveil
+#include <libc/dce.h> // IsLinux
+#include <limits.h>
+#include <netinet/in.h>
+#include <pthread.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/socket.h>
+#include <sys/wait.h>
+#include <unistd.h>
+
+// The policy wrapper llamafile_sandbox() consults llamafile_has_gpu(),
+// which lives in llamafile.c and drags in the GPU loader stack. This
+// test only exercises the enforcement primitives, so a stub keeps the
+// link small.
+bool llamafile_has_gpu(void) {
+ return false;
+}
+
+#define CHECK(ok, what) \
+ do { \
+ if (!(ok)) { \
+ fprintf(stderr, "FAIL(%s:%d): %s (errno=%s)\n", __FILE__, \
+ __LINE__, what, strerror(errno)); \
+ _exit(1); \
+ } \
+ } while (0)
+
+static void *thread_fn(void *arg) {
+ return arg;
+}
+
+// what the --cli / --chat modes rely on
+static void child_rpath_promises(void) {
+ CHECK(llamafile_sandbox_apply("stdio rpath") == LLAMAFILE_SANDBOX_ACTIVE,
+ "apply stdio rpath");
+
+ // reads must keep working: model weights are opened after pledge
+ int fd = open("/dev/null", O_RDONLY);
+ CHECK(fd != -1, "open O_RDONLY allowed under rpath");
+ close(fd);
+
+ // threads must keep working: ggml spawns its compute pool after pledge
+ pthread_t th;
+ CHECK(!pthread_create(&th, 0, thread_fn, 0), "pthread_create under stdio");
+ pthread_join(th, 0);
+
+ // no filesystem writes
+ errno = 0;
+ CHECK(open("sandbox_test.tmp", O_WRONLY | O_CREAT | O_TRUNC, 0644) == -1,
+ "open O_WRONLY|O_CREAT blocked");
+ CHECK(errno == EPERM, "write open fails with EPERM, not a kill");
+
+ // no network at all
+ errno = 0;
+ CHECK(socket(AF_INET, SOCK_STREAM, 0) == -1, "socket() blocked");
+ CHECK(errno == EPERM, "socket() fails with EPERM");
+
+ _exit(0);
+}
+
+// without rpath even read-only opens must be denied (embedded-weights
+// server: filesystem access disabled entirely)
+static void child_stdio_only(void) {
+ CHECK(llamafile_sandbox_apply("stdio") == LLAMAFILE_SANDBOX_ACTIVE,
+ "apply stdio");
+ errno = 0;
+ CHECK(open("/dev/null", O_RDONLY) == -1, "open O_RDONLY blocked");
+ CHECK(errno == EPERM, "read open fails with EPERM");
+ _exit(0);
+}
+
+// what --server relies on: accept incoming connections, never initiate
+static void child_anet_promises(void) {
+ // pre-pledge: a second socket to attempt the outbound connect() with
+ int censored = socket(AF_INET, SOCK_STREAM, 0);
+ CHECK(censored != -1, "pre-pledge socket");
+
+ CHECK(llamafile_sandbox_apply("stdio anet rpath") == LLAMAFILE_SANDBOX_ACTIVE,
+ "apply stdio anet rpath");
+
+ // the listening side of the HTTP server must keep working
+ int server = socket(AF_INET, SOCK_STREAM, 0);
+ CHECK(server != -1, "socket() allowed under anet");
+
+ struct sockaddr_in addr;
+ memset(&addr, 0, sizeof(addr));
+ addr.sin_family = AF_INET;
+ addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
+ addr.sin_port = 0; // ephemeral
+ CHECK(!bind(server, (struct sockaddr *)&addr, sizeof(addr)),
+ "bind() allowed under anet");
+ CHECK(!listen(server, 1), "listen() allowed under anet");
+
+ socklen_t addrlen = sizeof(addr);
+ CHECK(!getsockname(server, (struct sockaddr *)&addr, &addrlen),
+ "getsockname()");
+
+ // accept() is permitted: with no pending client it must report
+ // EAGAIN (not EPERM, which would mean the filter rejected it)
+ int flags = fcntl(server, F_GETFL);
+ CHECK(flags != -1 && fcntl(server, F_SETFL, flags | O_NONBLOCK) != -1,
+ "set O_NONBLOCK");
+ errno = 0;
+ CHECK(accept(server, 0, 0) == -1, "accept with no client");
+ CHECK(errno == EAGAIN || errno == EWOULDBLOCK,
+ "accept() allowed by filter (EAGAIN, not EPERM)");
+
+ // outbound connections must be denied, even to ourselves
+ errno = 0;
+ CHECK(connect(censored, (struct sockaddr *)&addr, sizeof(addr)) == -1,
+ "connect() blocked under anet");
+ CHECK(errno == EPERM, "connect() fails with EPERM");
+
+ _exit(0);
+}
+
+// unveil() confinement: reads under an unveiled directory succeed while
+// the rest of the filesystem is denied. Landlock accepts rules on any
+// filesystem but enforces on only some (see llamafile_sandbox_server's
+// governability probe), so if a read inside the unveiled dir is itself
+// denied we treat the filesystem as ungovernable and skip -- matching the
+// production fallback -- rather than failing.
+static void child_unveil(void) {
+ // Two files we own: one inside a directory we will unveil (must stay
+ // readable), one outside it (a self-made canary that must become denied
+ // -- a sibling under /tmp, never a system file).
+ char dir[] = "/tmp/lf_uv_XXXXXX";
+ CHECK(mkdtemp(dir), "mkdtemp");
+ char inside[PATH_MAX];
+ snprintf(inside, sizeof(inside), "%s/weights.bin", dir);
+ int fd = open(inside, O_WRONLY | O_CREAT | O_TRUNC, 0644);
+ CHECK(fd != -1, "create file inside temp dir");
+ close(fd);
+
+ char canary[] = "/tmp/lf_canary_XXXXXX";
+ fd = mkstemp(canary); // outside `dir`, so confinement must hide it
+ CHECK(fd != -1, "mkstemp canary");
+ close(fd);
+
+ unveil(dir, "r");
+ unveil(0, 0); // lock
+
+ int in = open(inside, O_RDONLY);
+ if (in == -1) {
+ // filesystem accepts unveil() but denies everything: over-denies
+ fprintf(stderr, "sandbox_test: unveil: SKIP (filesystem not "
+ "Landlock-governable)\n");
+ _exit(0);
+ }
+ close(in);
+
+ // If the canary outside the unveiled dir is still readable, unveil()
+ // installed nothing (kernel without Landlock, or cosmo's silent no-op):
+ // there is no confinement to assert. Same check the production probe
+ // makes, on a file we created rather than a system file.
+ errno = 0;
+ int c = open(canary, O_RDONLY);
+ if (c != -1) {
+ close(c);
+ fprintf(stderr, "sandbox_test: unveil: SKIP (Landlock unavailable; "
+ "confinement not installed)\n");
+ _exit(0);
+ }
+ CHECK(errno == EACCES || errno == EPERM,
+ "unveil denial is EACCES/EPERM, not a kill");
+ _exit(0);
+}
+
+static int run_case(const char *name, void (*fn)(void)) {
+ pid_t pid = fork();
+ if (!pid)
+ fn(); // never returns
+ int ws;
+ if (waitpid(pid, &ws, 0) != pid || !WIFEXITED(ws) || WEXITSTATUS(ws)) {
+ fprintf(stderr, "sandbox_test: %s: FAILED%s\n", name,
+ WIFSIGNALED(ws) ? " (killed by signal)" : "");
+ return 1;
+ }
+ printf("sandbox_test: %s: OK\n", name);
+ return 0;
+}
+
+// Pure promise-string derivation: keep the server policy unit-testable
+// rather than buried in the vendored server.cpp patch. Covers the
+// accept-only default, the wpath/cpath (has_rw) and inet (needs_outbound)
+// relaxations, and OpenBSD's lack of "anet".
+static int test_promises(void) {
+ struct {
+ bool openbsd, has_rw, outbound;
+ const char *want;
+ } cases[] = {
+ {false, false, false, "stdio anet rpath"},
+ {false, true, false, "stdio anet rpath wpath cpath"},
+ {false, false, true, "stdio inet rpath"}, // --rpc etc.
+ {false, true, true, "stdio inet rpath wpath cpath"},
+ {true, false, false, "stdio inet rpath"}, // OpenBSD: no anet
+ {true, true, false, "stdio inet rpath wpath cpath"},
+ };
+ int rc = 0;
+ for (size_t i = 0; i < sizeof(cases) / sizeof(*cases); ++i) {
+ char got[64];
+ llamafile_sandbox_server_promises(got, sizeof(got), cases[i].openbsd,
+ cases[i].has_rw, cases[i].outbound);
+ if (strcmp(got, cases[i].want)) {
+ fprintf(stderr, "sandbox_test: promises: FAILED want=\"%s\" got=\"%s\"\n",
+ cases[i].want, got);
+ rc = 1;
+ }
+ }
+ if (!rc)
+ printf("sandbox_test: server promise derivation: OK\n");
+ return rc;
+}
+
+int main(void) {
+ int rc = 0;
+
+ // Pure logic, runs on every platform.
+ rc |= test_promises();
+
+ // The enforcement cases assert Linux SECCOMP semantics: the "anet"
+ // promise is a cosmo Linux extension OpenBSD's pledge(2) rejects, and
+ // the EPERM assertions assume PLEDGE_PENALTY_RETURN_EPERM (Linux only;
+ // OpenBSD kills violators). So gate them to Linux.
+ if (!IsLinux() || !llamafile_sandbox_supported()) {
+ printf("sandbox_test: SKIP enforcement: needs Linux with SECCOMP "
+ "(anet/EPERM semantics are Linux-specific)\n");
+ return rc;
+ }
+ rc |= run_case("stdio+rpath (cli promises)", child_rpath_promises);
+ rc |= run_case("stdio only (embedded weights)", child_stdio_only);
+ rc |= run_case("stdio+anet+rpath (server promises)", child_anet_promises);
+ rc |= run_case("unveil confinement", child_unveil);
+ return rc;
+}