Distribute and run LLMs with a single file.
C++ cross-platformggufllama-cpplocal-ailocal-inferencelocal-llmopen-source-aisingle-file-executablespeech-to-text
Bring back pledge/SECCOMP sandboxing (#930) (#1015)
Davide Eynard8b7081179a07
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.mdindex 091b3386..fdb53a7b 100644--- a/docs/SUMMARY.md+++ b/docs/SUMMARY.md@@ -17,6 +17,7 @@## Reference* [Technical details](technical_details.md)+* [Security](security.md)* [Supported Systems](support.md)* [Troubleshooting](troubleshooting.md)diff --git a/docs/security.md b/docs/security.mdnew file mode 100644index 00000000..e7516405--- /dev/null+++ b/docs/security.md@@ -0,0 +1,133 @@+# Security++llamafile can sandbox itself with two [Cosmopolitan+Libc](https://github.com/jart/cosmopolitan) primitives:++- **[pledge()](https://man.openbsd.org/pledge.2)** restricts which system+ calls the process may make. On Linux it installs a SECCOMP BPF filter;+ on OpenBSD it calls the native `pledge(2)`. **On by default.**+- **[unveil()](https://man.openbsd.org/unveil.2)** restricts which parts of+ the filesystem the process can see. On Linux it uses the+ [Landlock](https://docs.kernel.org/userspace-api/landlock.html) LSM+ (kernel 5.13+); on OpenBSD it calls the native `unveil(2)`. **Opt-in**,+ via `--confine-reads`.++Neither needs any kernel configuration or privileges. SECCOMP filtering+requires Linux 3.5+ on x86-64 and 5.13+ (for Landlock) on either+architecture; on older kernels, and on macOS, Windows, and other BSDs, the+calls are no-ops and llamafile logs that sandboxing is unavailable and keeps+running. The pledge() sandbox can be turned off entirely with `--unsecure`.++## The default: pledge()++The `llamafile --server` process runs under `stdio anet rpath` on Linux+(`stdio inet rpath` on OpenBSD). After startup this means:++- **No outbound network.** `anet` allows `accept()` but not `connect()`, so+ the only networking the server can do is answer connections it received.+ If the server is ever compromised (say, through a bug in the GGUF parser+ or an HTTP handler), this is the single most valuable restriction: it cuts+ the attacker's ability to exfiltrate anything over the network.+- **No writing, creating, deleting, executing, or forking.** A compromised+ server can't modify your files, drop a payload, or launch a program.+ (`--slot-save-path` and a prompt cache add write access to *their*+ directories only.)+- **Reads are allowed** (`rpath`) — anywhere the process could already read.+ This is deliberate: a server routinely opens files whose paths it only+ learns at request time (multimodal media, static assets), which cannot be+ known when a path allow-list would have to be locked. If you want to+ restrict *which* files are readable, see `--confine-reads` below.++`--cli` runs under `stdio rpath tty` and `--chat` under+`stdio rpath wpath cpath tty` — neither has any network access at all.++## Opt-in read confinement: `--confine-reads`++Passing `--confine-reads` to the server additionally applies `unveil()`,+confining reads to the executable and the directories holding the weights+(the model and its shards, `--mmproj`, LoRA adapters, a draft model, control+vectors, `--media-path`, a static `--path` web root) plus the two+name-resolution files a non-numeric `--host` needs. The rest of the+filesystem — `/etc/passwd`, your SSH keys, other users' files — becomes+invisible to the server even though it is world-readable.++Two things to know before relying on it:++- **It confines directories, not individual files.** The model's *parent+ directory* is unveiled (so multi-part GGUF shards beside it load), which+ means anything else in that directory is readable too. Put weights in a+ dedicated directory if you don't want their neighbours exposed.+- **It's incompatible with reading files by arbitrary path at request time.**+ Because the readable set is locked at startup, a request that references a+ file outside the unveiled directories will be denied. The built-in flows+ (media under `--media-path`, static files under `--path`) are covered;+ bespoke setups that read elsewhere are not.+- **It requires a filesystem Landlock can govern.** On the handful it cannot+ (some network mounts, virtiofs, 9p) llamafile keeps the pledge() sandbox+ but skips confinement rather than refusing to load the model, and logs+ that it did so.++## When the sandbox is relaxed or skipped++llamafile logs a notice in each case:++- **Outbound features** (`--rpc` for distributed inference, server-side+ tools, the MCP proxy) legitimately need to `connect()` out, so the+ networking promise is relaxed from `anet` to `inet` for them. Writes and+ exec stay blocked.+- **GPU mode gets no sandbox.** GPU backends are loaded dynamically and their+ drivers need device access (`ioctl`, `/dev/*`) that no promise set covers,+ so the sandbox is skipped whenever a GPU backend is loaded. **This is the+ common production case** — a GPU server is not sandboxed. Pass+ `--gpu disable` to force CPU inference with the sandbox.+- **Combined mode** (the default `llamafile -m model.gguf`, TUI + server in+ one process) hosts an in-process HTTP client that must `connect()` to the+ server, so it's skipped. Run `llamafile --server` for the sandboxed server.+- Model downloads (`-hf`, `--model-url`) happen *before* the sandbox is+ installed; once the download completes the process is sandboxed normally.++## A note on the penalty mode++On Linux, sandbox violations are configured to return `EPERM` (permission+denied) rather than killing the process, so a blocked syscall surfaces as an+ordinary I/O error. If a server refuses to start with a `pledge failed`+error (an exotic kernel or an outer seccomp policy), `--unsecure` disables+the sandbox. OpenBSD's native `pledge(2)` always terminates a violating+process with `SIGABRT` instead; that behavior is not configurable there.++## Verifying the sandbox++Unit tests exercise every promise set llamafile uses, assert that allowed+operations work while blocked ones fail, and check that `unveil()` confines+reads to the unveiled directory (run on Linux):++```sh+.cosmocc/4.0.2/bin/make o//tests/sandbox_test && o//tests/sandbox_test+```++Integration tests verify the running server end-to-end — every thread of the+server process carries the SECCOMP filter, completions still work inside the+sandbox, a bundled `/zip/` llamafile loads under it, `--confine-reads`+confines while the default does not, and `--unsecure` really disables it:++```sh+cd tests/integration+./run_tests.sh --executable ../../o/llamafile/llamafile \+ --model ../../models/TinyLLama-v0.1-5M-F16.gguf -m sandbox+```++You can also check a running server by hand:++```sh+llamafile --server -m model.gguf &+grep Seccomp /proc/$!/status # "Seccomp: 2" means the filter is active+```++## Caveats++Your llamafile is able to protect itself against the outside world, but that+doesn't mean you're protected from llamafile. Sandboxing is self-imposed. If+you obtained your llamafile from an untrusted source then its author could+have simply modified it to not do that. In that case, you can run the+untrusted llamafile inside another sandbox, such as a virtual machine, to+make sure it behaves how you expect.diff --git a/llama.cpp.patches/README.md b/llama.cpp.patches/README.mdindex 1f96e6e5..d439b9d7 100644--- a/llama.cpp.patches/README.md+++ b/llama.cpp.patches/README.md@@ -156,7 +156,7 @@ These patches integrate llamafile's file handling APIs for loading models from b| Patch | Description ||-------|-------------|-| `tools_server_server.cpp.patch` | Renames upstream's `llama_server()` to `server_main()` and adds `on_ready`/`on_shutdown_available` callbacks for combined TUI+server mode; adds Metal/GPU backend trigger before `common_init()`; adds Cosmopolitan-specific standalone `main()` with `cosmo_args`, verbose flag handling, and GPU pre-initialization; handles `LLAMAFILE_TUI` exit to avoid Metal cleanup crashes |+| `tools_server_server.cpp.patch` | Renames upstream's `llama_server()` to `server_main()` and adds `on_ready`/`on_shutdown_available` callbacks for combined TUI+server mode; adds Metal/GPU backend trigger before `common_init()`; installs the sandbox (`llamafile_sandbox_server()`, issue #930) — the mechanism lives in `llamafile/sandbox.c`; the patch fills a `llamafile_sandbox_spec` with the on-disk paths the loader opens after the lock (model, mmproj, media dir, LoRA, draft model, control vectors, public path as read; slot-save and prompt cache as read-write), detects outbound-needing features (`--rpc` via argv/env, MCP proxy, server tools) to relax `anet`→`inet`, passes `FLAG_confine_reads` for opt-in unveil(), quiesces the log worker (`common_log_pause`/`resume`, so no thread escapes the per-thread filter) around the call, before the HTTP listener spawns and before model load, and skips it in combined/GPU modes; adds Cosmopolitan-specific standalone `main()` with `cosmo_args`, verbose flag handling, `--unsecure` consumption (`llamafile_consume_flag`), and GPU pre-initialization; handles `LLAMAFILE_TUI` exit to avoid Metal cleanup crashes |The web UI moved upstream from prebuilt `tools/server/public/*` assets toa Svelte/PWA project under `tools/ui/`, embedded at CMake time viadiff --git a/llama.cpp.patches/llamafile-files/BUILD.mk b/llama.cpp.patches/llamafile-files/BUILD.mkindex 0a23aca5..04c7f787 100644--- a/llama.cpp.patches/llamafile-files/BUILD.mk+++ b/llama.cpp.patches/llamafile-files/BUILD.mk@@ -436,6 +436,7 @@ UI_GEN_OBJ := $(UI_CPP_GEN:%.cpp=%.cpp.o)TOOL_LLAMAFILE_OBJS := \o/$(MODE)/llamafile/llamafile.o \o/$(MODE)/llamafile/gpu.a \+ o/$(MODE)/llamafile/sandbox.o \o/$(MODE)/llamafile/zip.o# Server objects depend on the llamafile bridge header and on thediff --git a/llama.cpp.patches/patches/tools_server_server.cpp.patch b/llama.cpp.patches/patches/tools_server_server.cpp.patchindex f9c8402b..f1e2918f 100644--- a/llama.cpp.patches/patches/tools_server_server.cpp.patch+++ b/llama.cpp.patches/patches/tools_server_server.cpp.patch@@ -9,11 +9,13 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp#include <signal.h>#include <thread> // for std::thread::hardware_concurrency-@@ -21,6 +22,11 @@+@@ -21,6 +22,13 @@#include <windows.h>#endif+#ifdef COSMOCC++#include <cerrno>++#include <cstring>+#include <cosmo.h>+#include "llamafile.h"+#endif@@ -21,7 +23,7 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cppstatic std::function<void(int)> shutdown_handler;static std::atomic_flag is_terminating = ATOMIC_FLAG_INIT;-@@ -71,15 +77,26 @@ static server_http_context::handler_t ex_wrapper(server_http_context::handler_t+@@ -71,15 +79,26 @@ static server_http_context::handler_t ex_wrapper(server_http_context::handler_t};}@@ -52,7 +54,94 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cppcommon_init();if (!common_params_parse(argc, argv, params, LLAMA_EXAMPLE_SERVER)) {-@@ -332,6 +349,13 @@ int llama_server(int argc, char ** argv) {+@@ -297,6 +316,86 @@ int llama_server(int argc, char ** argv) {+ llama_backend_free();+ };+++#ifdef COSMOCC++ // Sandbox (pledge()/unveil(), llamafile issue #930). The whole++ // derivation lives in llamafile/sandbox.c so it stays unit-testable++ // and off the vendored diff. Placement is constrained on every side:++ // - after common_params_parse(): -hf downloads need connect()++ // - after ctx_http.init(): the TLS certificate/key are read there++ // - before ctx_http.start(): filters only cover threads created++ // after installation, so the HTTP listener/pool must not exist++ // - before load_model(): untrusted GGUF data is parsed sandboxed++ //++ // Combined mode hosts an in-process HTTP client (the TUI) that needs++ // connect(), which the accept-only sandbox forbids, so it's skipped.++ const bool combined_mode = on_ready || on_shutdown_available;++ if (combined_mode) {++ SRV_WRN("%s", "sandbox: disabled in combined mode; use --server for a sandboxed server\n");++ } else {++ // pledge() (no outbound network / writes / exec) is applied by++ // default. unveil() path confinement is opt-in (--confine-reads),++ // because a server opens files by path at request time (multimodal++ // media, etc.) that can't be known when the ruleset locks.++ //++ // cosmo installs the SECCOMP/Landlock filters on the calling thread++ // only (no TSYNC), so quiesce the llama.cpp log worker across the++ // call: pause() joins it, resume() respawns it under the filter.++ //++ // When --confine-reads is on, unveil() confines reads to these++ // directories, so collect every on-disk path the loader opens++ // after the lock. Add to this list when llama.cpp gains a new file++ // parameter (else it'd be denied under confinement). Paths read++ // before the sandbox (ssl_file_key/cert in ctx_http.init(), -hf++ // downloads in parse) need no entry.++ std::vector<const char *> read_paths, rw_paths;++ read_paths.push_back(params.model.path.c_str());++ read_paths.push_back(params.mmproj.path.c_str());++ read_paths.push_back(params.media_path.c_str()); // request-time media++ read_paths.push_back(params.speculative.draft.mparams.path.c_str());++ read_paths.push_back(params.speculative.ngram_cache.lookup_cache_static.c_str());++ read_paths.push_back(params.public_path.c_str());++ for (const auto & la : params.lora_adapters)++ read_paths.push_back(la.path.c_str());++ for (const auto & cv : params.control_vectors)++ read_paths.push_back(cv.fname.c_str());++ rw_paths.push_back(params.slot_save_path.c_str());++ rw_paths.push_back(params.path_prompt_cache.c_str());++ rw_paths.push_back(params.speculative.ngram_cache.lookup_cache_dynamic.c_str());++++ // Features that dial out at runtime need connect(), which the++ // accept-only default forbids; relax to full sockets for them++ // (still no writes/exec). --rpc has no params field -- its handler++ // registers backends directly -- so detect it on argv/env.++ const bool needs_outbound =++ params.ui_mcp_proxy ||++ !params.server_tools.empty() ||++ llamafile_has(argv, "--rpc") ||++ (std::getenv("LLAMA_ARG_RPC") != nullptr);++++ llamafile_sandbox_spec spec = {++ read_paths.data(), (int) read_paths.size(),++ rw_paths.data(), (int) rw_paths.size(),++ FLAG_confine_reads, needs_outbound,++ };++ char promises[64];++ common_log_pause(common_log_main());++ int sandbox = llamafile_sandbox_server(&spec, promises, sizeof(promises));++ int sandbox_errno = errno; // capture before resume() spawns a thread++ common_log_resume(common_log_main());++ if (sandbox == LLAMAFILE_SANDBOX_FAILED) {++ SRV_ERR("sandbox: pledge failed: %s; retry with --unsecure to disable sandboxing\n",++ strerror(sandbox_errno));++ clean_up();++ return 1;++ } else if (llamafile_sandbox_is_active(sandbox)) {++ SRV_INF("sandbox: pledge(\"%s\") %s\n", promises,++ llamafile_sandbox_describe(sandbox));++ } else {++ SRV_WRN("sandbox: %s\n", llamafile_sandbox_describe(sandbox));++ }++ }++#endif+++ // start the HTTP server before loading the model to be able to serve /health requests+ if (!ctx_http.start()) {+ clean_up();+@@ -332,6 +431,13 @@ int llama_server(int argc, char ** argv) {// this will unblock start_loop()ctx_server.terminate();};@@ -66,7 +155,7 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp}// TODO: refactor in common/console-@@ -368,6 +392,11 @@ int llama_server(int argc, char ** argv) {+@@ -368,6 +474,11 @@ int llama_server(int argc, char ** argv) {} else {SRV_INF("server is listening on %s\n", ctx_http.listening_address.c_str());@@ -78,7 +167,7 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp// optionally, notify router server that this instance is readystd::thread monitor_thread;if (child.is_child()) {-@@ -392,5 +421,48 @@ int llama_server(int argc, char ** argv) {+@@ -392,5 +503,56 @@ int llama_server(int argc, char ** argv) {}}@@ -111,6 +200,14 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp+ bool verbose = llamafile_has(argv, "--verbose");+ FLAG_verbose = verbose ? 1 : 0;+++ // --unsecure is llamafile's flag, not llama.cpp's: consume it here so++ // common_params_parse() never sees it (llamafile's own main strips it++ // via parse_llamafile_args instead)++ if (llamafile_consume_flag(&argc, argv, "--unsecure"))++ FLAG_unsecure = true;++ if (llamafile_consume_flag(&argc, argv, "--confine-reads"))++ FLAG_confine_reads = true;+++ // Initialize GPU support early (must happen BEFORE llama_backend_init())+ // This triggers dynamic loading of GPU backends (CUDA, ROCm, Metal)+ // The llamafile_has_* functions use lazy initialization via cosmo_once()diff --git a/llamafile/BUILD.mk b/llamafile/BUILD.mkindex bafde492..470afcf0 100644--- a/llamafile/BUILD.mk+++ b/llamafile/BUILD.mk@@ -138,6 +138,7 @@ LLAMAFILE_SRCS_C := \llamafile/gpu_backend.c \llamafile/llamafile.c \llamafile/metal.c \+ llamafile/sandbox.c \llamafile/vulkan.c \llamafile/zip.cdiff --git a/llamafile/args.cpp b/llamafile/args.cppindex 8f78367a..73c610f9 100644--- a/llamafile/args.cpp+++ b/llamafile/args.cpp@@ -36,6 +36,8 @@ static bool is_llamafile_flag(const char* arg) {strcmp(arg, "--ascii") == 0 ||strcmp(arg, "--nologo") == 0 ||strcmp(arg, "--nothink") == 0 ||+ strcmp(arg, "--unsecure") == 0 ||+ strcmp(arg, "--confine-reads") == 0 ||strcmp(arg, "--version") == 0;}@@ -87,6 +89,12 @@ LlamafileArgs parse_llamafile_args(int argc, char** argv) {FLAG_nologo = llamafile_has(argv, "--nologo");FLAG_ascii = llamafile_has(argv, "--ascii");+ // Check --unsecure flag (disables pledge() sandboxing, see sandbox.c)+ FLAG_unsecure = llamafile_has(argv, "--unsecure");++ // Check --confine-reads flag (opt-in unveil() path confinement, server mode)+ FLAG_confine_reads = llamafile_has(argv, "--confine-reads");+// Filter out llamafile-specific arguments// These are not recognized by llama.cpp and would cause errorsg_filtered_argv.clear();diff --git a/llamafile/chatbot_cli.cpp b/llamafile/chatbot_cli.cppindex 3c312635..9f4b72ec 100644--- a/llamafile/chatbot_cli.cpp+++ b/llamafile/chatbot_cli.cpp@@ -168,6 +168,17 @@ int cli_main(int argc, char **argv) {params.n_gpu_layers = INT_MAX;}+ // Drop network and filesystem-write access before the untrusted GGUF+ // file is parsed. Must happen after common_params_parse() (-hf model+ // downloads need the network) and before threads/weights come up.+ // Quiesce the log worker across the pledge so it doesn't outlive the+ // per-thread filter unsandboxed (see llamafile/sandbox.c rule 1).+ common_log_pause(common_log_main());+ int sandbox = llamafile_sandbox_enter("stdio rpath tty", FLAG_verbose);+ common_log_resume(common_log_main());+ if (sandbox == LLAMAFILE_SANDBOX_FAILED)+ return 1;+// Load modelllama_model_params model_params = common_model_params_to_llama(params);llama_model *model = llama_model_load_from_file(params.model.path.c_str(), model_params);diff --git a/llamafile/chatbot_main.cpp b/llamafile/chatbot_main.cppindex ccf415f6..d061e2ad 100644--- a/llamafile/chatbot_main.cpp+++ b/llamafile/chatbot_main.cpp@@ -161,6 +161,18 @@ int main(int argc, char **argv) {}clear_ephemeral();+ // Drop network access before the untrusted GGUF file is parsed. Must+ // happen after common_params_parse() (-hf model downloads need the+ // network) and before threads/weights come up. Unlike --cli, the chat+ // TUI keeps wpath+cpath so /dump and /push can still write files.+ // Quiesce the log worker across the pledge so it doesn't outlive the+ // per-thread filter unsandboxed (see llamafile/sandbox.c rule 1).+ common_log_pause(common_log_main());+ int sandbox = llamafile_sandbox_enter("stdio rpath wpath cpath tty", verbose);+ common_log_resume(common_log_main());+ if (sandbox == LLAMAFILE_SANDBOX_FAILED)+ exit(1);+// Suppress logging for model loading unless --verbose was specified// We must set this AFTER common_init() since it overwrites the log callback// and BEFORE model loading to suppress those logsdiff --git a/llamafile/llamafile.c b/llamafile/llamafile.cindex bcb04791..88e80d25 100644--- a/llamafile/llamafile.c+++ b/llamafile/llamafile.c@@ -285,9 +285,17 @@ struct llamafile *llamafile_open_gguf(const char *fname, const char *mode) {// open from file or from our own executable if it doesn't existstruct llamafile *file;if (!(file = llamafile_open_file(fname, mode))) {- if (errno == ENOENT) {+ // ENOENT: the path isn't a real file (e.g. a bare basename+ // embedded by llamafile-convert). EACCES/EPERM: the sandbox+ // (unveil/pledge) denies opening it directly; the embedded copy+ // in our own /zip/ store is still reachable via the executable.+ int open_errno = errno;+ if (open_errno == ENOENT || open_errno == EACCES || open_errno == EPERM) {if (!(file = llamafile_open_zip(GetProgramExecutableName(), fname, mode))) {- errno = ENOENT;+ // no embedded copy either: report why the real open failed+ // (e.g. "Permission denied" for a sandbox/mode denial, not a+ // misleading "No such file or directory")+ errno = open_errno;return 0;}return file;@@ -451,6 +459,21 @@ bool llamafile_has(char **a, const char *x) {return false;}+bool llamafile_consume_flag(int *argc, char **argv, const char *flag) {+ bool found = false;+ int w = 1;+ for (int r = 1; r < *argc; ++r) {+ if (!strcmp(argv[r], flag)) {+ found = true;+ } else {+ argv[w++] = argv[r];+ }+ }+ argv[w] = 0;+ *argc = w;+ return found;+}+static const char *llamafile_get_home_dir(void) {const char *homedir;if (!(homedir = getenv("HOME")) || !*homedir)diff --git a/llamafile/llamafile.h b/llamafile/llamafile.hindex ae7052fc..81063ec5 100644--- a/llamafile/llamafile.h+++ b/llamafile/llamafile.h@@ -35,6 +35,7 @@ extern bool FLAG_nologo; // Suppresses logo display (chatbot_main.cpp)extern bool FLAG_nothink; // Filters thinking/reasoning content (chatbot_cli.cpp)extern bool FLAG_precise; // Forces precise math in tinyblas (tinyblas_cpu.h)extern bool FLAG_recompile; // Forces GPU library recompilation (metal.c)+extern bool FLAG_unsecure; // Disables pledge() sandboxing (sandbox.c)extern int FLAG_gpu; // GPU backend selection (llamafile.c, metal.c, cuda.c)extern int FLAG_verbose; // Verbose output (chatbot_main.cpp, metal.c, cuda.c)@@ -114,6 +115,62 @@ int llamafile_gpu_parse(const char *); // Defined in llamafile.cconst char *llamafile_describe_gpu(void); // Defined in llamafile.cvoid llamafile_early_gpu_init(char **); // Defined in llamafile.c+// =============================================================================+// Sandboxing - pledge()/SECCOMP + unveil()/Landlock (defined in sandbox.c)+// =============================================================================++// The pledge() syscall sandbox (no outbound network, no writes, no exec) is+// applied by default. unveil() path confinement is opt-in (--confine-reads),+// because locking the readable paths at startup is incompatible with files a+// server opens by path at request time (multimodal media, etc.).++#define LLAMAFILE_SANDBOX_FAILED -1 // pledge() failed, errno is set+#define LLAMAFILE_SANDBOX_ACTIVE 0 // pledge() active+#define LLAMAFILE_SANDBOX_UNSECURE 1 // skipped: --unsecure flag+#define LLAMAFILE_SANDBOX_GPU 2 // skipped: GPU backend loaded+#define LLAMAFILE_SANDBOX_UNSUPPORTED 3 // skipped: OS can't enforce pledge()+#define LLAMAFILE_SANDBOX_ACTIVE_CONFINED 4 // pledge() active + unveil() applied+#define LLAMAFILE_SANDBOX_ACTIVE_UNCONFINED 5 // pledge() active; unveil() asked+ // for but filesystem can't enforce it++extern bool FLAG_confine_reads; // opt-in unveil() path confinement (sandbox.c)++bool llamafile_sandbox_supported(void); // Probe only, installs nothing+int llamafile_sandbox_apply(const char *); // Unconditional pledge()+int llamafile_sandbox(const char *); // Honors --unsecure and GPU mode+int llamafile_sandbox_enter(const char *, bool);// sandbox() + perror/verbose report+const char *llamafile_sandbox_describe(int); // Status code -> human string+bool llamafile_sandbox_is_active(int); // true for the ACTIVE* statuses++// Inputs to the server sandbox. read_paths are opened read-only (model,+// mmproj, LoRA, draft model, control vectors, media dir, static web root);+// rw_paths get write+create (slot-save dir, prompt cache). confine requests+// unveil() path confinement; needs_outbound relaxes accept()-only networking+// to full sockets when the server must dial out (--rpc, tools, MCP proxy).+struct llamafile_sandbox_spec {+ const char *const *read_paths;+ int n_read;+ const char *const *rw_paths;+ int n_rw;+ bool confine;+ bool needs_outbound;+};++// Applies the server sandbox: pledge() always, plus unveil() when+// spec->confine. Fills promises_out with the pledge string for logging.+// Returns an ACTIVE* status (see describe()) or a skip/FAILED code.+int llamafile_sandbox_server(const struct llamafile_sandbox_spec *spec,+ char *promises_out, size_t promises_len);++// Pure promise-string derivation, exposed for unit testing.+void llamafile_sandbox_server_promises(char *out, size_t len, bool is_openbsd,+ bool has_rw, bool needs_outbound);++// Removes every occurrence of flag from argv in place, updating *argc, and+// returns true if it was present. Used to consume llamafile-only flags+// (e.g. --unsecure) before handing argv to llama.cpp's parser. In llamafile.c.+bool llamafile_consume_flag(int *argc, char **argv, const char *flag);+// Log callback type for Metal backend (matches ggml_log_callback)typedef void (*llamafile_log_callback)(int level, const char *text, void *user_data);diff --git a/llamafile/main.cpp b/llamafile/main.cppindex 699c90d5..33963f3e 100644--- a/llamafile/main.cpp+++ b/llamafile/main.cpp@@ -172,6 +172,8 @@ static void print_general_help(bool show_full_list_hint) {" --gpu MODE GPU backend (auto, nvidia, amd, apple, disable)\n"" --nologo suppress the startup logo\n"" --ascii use ASCII art instead of Unicode for logo\n"+ " --unsecure disable pledge() sandboxing (Linux/OpenBSD)\n"+ " --confine-reads restrict server file reads to the weights dirs\n"" --verbose enable verbose logging\n"" --version show version information\n"" --help show this help\n"@@ -205,6 +207,8 @@ static void print_server_help(bool show_full_list_hint) {" --host ADDR ip address to listen on (default 127.0.0.1)\n"" --port N tcp port to listen on (default 8080)\n"" --api-key KEY require this API key for authentication\n"+ " --confine-reads restrict file reads to the weights directories\n"+ " --unsecure disable pledge() sandboxing (Linux/OpenBSD)\n""\n");if (show_full_list_hint) {printf("all other llama.cpp options are also accepted.\n"@@ -395,7 +399,11 @@ int main(int argc, char **argv) {if (llamafile_has(argv, "--help") || llamafile_has(argv, "-h")) {switch (args.mode) {case lf::ProgramMode::SERVER:- break; // fall through to llama.cpp's server help+ // Show llamafile's own server flags (which are stripped before+ // llama.cpp sees them, so its parser can't list them) before+ // falling through to llama.cpp's server help.+ print_server_help(false);+ break;case lf::ProgramMode::AUTO:// Combined mode: args go to the HTTP server, so show the// llamafile intro followed by llama.cpp's server option list.@@ -467,6 +475,12 @@ int main(int argc, char **argv) {args.llama_argv = quiet_argv.data();}+ // --confine-reads only affects the --server sandbox. In the other modes+ // (including the default combined mode, whose sandbox is skipped) it does+ // nothing, so say so rather than silently ignoring it.+ if (FLAG_confine_reads && args.mode != lf::ProgramMode::SERVER)+ fprintf(stderr, "warning: --confine-reads only applies to --server mode; ignoring\n");+// Initialize GPU support (triggers dynamic loading of GPU backends)llamafile_has_gpu();diff --git a/llamafile/sandbox.c b/llamafile/sandbox.cnew file mode 100644index 00000000..624df03a--- /dev/null+++ b/llamafile/sandbox.c@@ -0,0 +1,421 @@+// -*- mode:c;indent-tabs-mode:nil;c-basic-offset:4;coding:utf-8 -*-+// vi: set et ft=c ts=4 sts=4 sw=4 fenc=utf-8 :vi+//+// Copyright 2026 Mozilla.ai+//+// Licensed under the Apache License, Version 2.0 (the "License");+// you may not use this file except in compliance with the License.+// You may obtain a copy of the License at+//+// http://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing, software+// distributed under the License is distributed on an "AS IS" BASIS,+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+// See the License for the specific language governing permissions and+// limitations under the License.++//+// llamafile sandboxing (issue #930)+//+// This module wraps two Cosmopolitan Libc primitives:+//+// pledge() OpenBSD-style syscall sandboxing. On OpenBSD it calls the+// native pledge(2); on Linux it installs a SECCOMP BPF filter+// emulating the requested promise set. No kernel config or+// privileges are needed.+// unveil() restricts which filesystem paths are reachable. On OpenBSD+// it calls the native unveil(2); on Linux it uses the Landlock+// LSM (kernel 5.13+). Together with pledge's "rpath" this turns+// "read-only, anywhere" into "read-only, within these dirs".+//+// On every other platform (macOS, Windows, other BSDs) both are no-ops:+// the sandbox reports that and callers keep running unsandboxed, matching+// the pre-0.10 behavior.+//+// Promise cheat sheet (see Cosmopolitan libc/calls/pledge.c for the+// full syscall lists):+//+// stdio read/write on already-open fds, mmap w/o PROT_EXEC,+// futexes, clocks, poll/epoll, and thread creation --+// enough for ggml inference on memory-mapped weights+// rpath open files read-only (model weights, executable zip store)+// wpath open existing files for writing+// cpath create/rename/unlink files+// tty terminal ioctls (raw mode, window size)+// inet IPv4/IPv6 sockets incl. connect()+// anet like inet but connect() is forbidden: the process can+// accept() connections yet never initiate them (Linux only;+// on OpenBSD the closest promise is inet)+//+// Three rules of engagement:+//+// 1. SECCOMP/Landlock filters attach to the calling thread and are+// inherited only by threads created *afterward* (cosmo does not use+// SECCOMP_FILTER_FLAG_TSYNC). Callers must sandbox before spawning+// worker threads, and must ensure no pre-existing background thread+// (e.g. the llama.cpp log worker) survives across the call --- see+// the common_log_pause()/resume() dance at the server call site.+//+// 2. GPU support is loaded with cosmo_dlopen() and drivers keep talking+// to /dev nodes at inference time, which no reasonable promise set+// covers. When a GPU is in use the sandbox is skipped and we say so.+//+// 3. unveil() must run *before* pledge(): it needs Landlock syscalls+// that the pledge filter would otherwise deny.+//+// On Linux, violations return EPERM instead of killing the process+// (PLEDGE_PENALTY_RETURN_EPERM), so a blocked syscall surfaces as an+// ordinary I/O error. OpenBSD's native pledge(2) always kills with+// SIGABRT instead; that penalty mode is Linux-only.+//++#define _COSMO_SOURCE // exposes pledge(), unveil(), __pledge_mode++#include "llamafile.h"++#include <errno.h>+#include <limits.h>+#include <stdio.h>+#include <string.h>+#include <sys/wait.h>+#include <unistd.h>++#ifdef COSMOCC+#include <cosmo.h>+#include <fcntl.h>+#include <libc/calls/calls.h>+#include <libc/calls/pledge.h>+#include <libc/dce.h> // IsOpenbsd+#endif++bool FLAG_unsecure = false;+bool FLAG_confine_reads = false;++// Reports whether the host OS can enforce pledge() promises+// (Linux with SECCOMP_MODE_FILTER, or OpenBSD). Does not install anything.+bool llamafile_sandbox_supported(void) {+#ifdef COSMOCC+ return !pledge(0, 0);+#else+ return false;+#endif+}++// Shared policy gate: LLAMAFILE_SANDBOX_ACTIVE means "install it", otherwise a+// skip code. Both entry points consult this so the gate lives in one place.+static int sandbox_skip_status(void) {+ if (FLAG_unsecure)+ return LLAMAFILE_SANDBOX_UNSECURE;+ if (llamafile_has_gpu())+ return LLAMAFILE_SANDBOX_GPU;+ if (!llamafile_sandbox_supported())+ return LLAMAFILE_SANDBOX_UNSUPPORTED;+ return LLAMAFILE_SANDBOX_ACTIVE;+}++// Shared pledge() tail: EPERM penalty mode + install the filter.+static int sandbox_install_pledge(const char *promises) {+#ifdef COSMOCC+ __pledge_mode = PLEDGE_PENALTY_RETURN_EPERM;+ return pledge(promises, 0) ? LLAMAFILE_SANDBOX_FAILED+ : LLAMAFILE_SANDBOX_ACTIVE;+#else+ (void)promises;+ return LLAMAFILE_SANDBOX_UNSUPPORTED;+#endif+}++// Installs the pledge() sandbox unconditionally, skipping the --unsecure/GPU+// policy gate (the gated entry points share sandbox_install_pledge() directly+// instead). Exposed for the unit test, which drives the promise sets in a+// forked child without the gate.+int llamafile_sandbox_apply(const char *promises) {+ if (!llamafile_sandbox_supported())+ return LLAMAFILE_SANDBOX_UNSUPPORTED;+ return sandbox_install_pledge(promises);+}++// Policy entry point for the CLI and chat modes: honors --unsecure and GPU+// mode, then pledges. These modes never confine paths -- they have no network+// at all and legitimately read user-chosen files (e.g. --image).+int llamafile_sandbox(const char *promises) {+ int skip = sandbox_skip_status();+ if (skip != LLAMAFILE_SANDBOX_ACTIVE)+ return skip;+ return sandbox_install_pledge(promises);+}++// Applies the CLI/chat sandbox and reports the outcome, collapsing the+// apply-perror-log boilerplate the two modes used to duplicate. On+// failure the caller decides whether to exit or return; every other+// status is informational and printed only when verbose.+int llamafile_sandbox_enter(const char *promises, bool verbose) {+ int status = llamafile_sandbox(promises);+ if (status == LLAMAFILE_SANDBOX_FAILED) {+ perror("pledge");+ } else if (verbose) {+ fprintf(stderr, "sandbox: %s\n", llamafile_sandbox_describe(status));+ }+ return status;+}++// Canary files the governability probe reads back after locking: if one is+// still readable, unveil() installed nothing (no Landlock, or an ungoverned+// filesystem) and we must not claim confinement. Requirements:+// - world-readable, so the ONLY thing that can deny it is Landlock -- a+// root-only file would be denied by ordinary permissions too, masking a+// no-op unveil() and yielding a false "confined";+// - outside every unveiled weights dir;+// - benign and commonly read by ordinary software (distro/hostname+// detection), so the probe isn't mistaken for credential recon the way+// reading /etc/passwd would be.+// The probe uses the first one that exists; if none do, it conservatively+// reports "not confined" rather than guessing.+static const char *const kSandboxCanaries[] = {+ "/etc/os-release",+ "/etc/hostname",+};++// Derives the server pledge() promise string. Pure and side-effect-free so+// it can be unit-tested. rpath is unconditional: the model loader always+// performs a real open() -- of the on-disk weights, or of the executable+// itself to read its embedded /zip/ store (llamafile_open_zip) -- so a server+// can never load a model without it.+void llamafile_sandbox_server_promises(char *out, size_t len, bool is_openbsd,+ bool has_rw, bool needs_outbound) {+ // "anet" is accept()-only networking (no connect) -- the default, so a+ // compromised server can't dial out. When the server itself must make+ // outbound connections (--rpc, server-side tools, the MCP proxy) we relax+ // to "inet", which still blocks writes and exec. OpenBSD has no "anet", so+ // it always uses "inet".+ const char *net = (is_openbsd || needs_outbound) ? "inet" : "anet";+ if (has_rw) // slot save/restore and the prompt cache read+write on disk+ snprintf(out, len, "stdio %s rpath wpath cpath", net);+ else+ snprintf(out, len, "stdio %s rpath", net);+}++#ifdef COSMOCC++// A path is "on disk" (needs its own unveil rule) when it exists on the+// real filesystem and isn't served from the executable's /zip/ store.+// Embedded weights -- a /zip/ path, or a bare basename that resolves to+// the executable via llamafile_open_zip -- are covered by unveiling the+// executable itself.+static bool path_on_disk(const char *path) {+ return path && *path && strncmp(path, "/zip/", 5) && !access(path, F_OK);+}++// Writes into `dir` the path that must be reachable to open `path`: the+// path itself if it's a directory (a web root, a slot-save dir), else its+// parent directory, so multi-part GGUF shards beside a weights file come+// along -- except a file at the filesystem root, whose parent "/" would+// expose everything, so the file itself is used. Returns false for paths+// not on the real filesystem (empty, /zip/, or absent). Single source of+// truth for both unveil_container() (apply) and the governability probe.+static bool container_of(const char *path, char *dir, size_t len) {+ if (!path_on_disk(path))+ return false;+ if (isdirectory(path)) {+ strlcpy(dir, path, len);+ return true;+ }+ strlcpy(dir, path, len);+ char *slash = strrchr(dir, '/');+ if (!slash)+ strlcpy(dir, ".", len); // bare "file" -> cwd+ else if (slash == dir)+ strlcpy(dir, path, len); // "/file" -> the file, never "/"+ else+ *slash = '\0'; // "dir/file" -> "dir"+ return true;+}++// unveil()s the container of `path` with the given perms, if it's on disk.+static void unveil_container(const char *path, const char *perms) {+ char dir[PATH_MAX];+ if (container_of(path, dir, sizeof(dir)))+ unveil(dir, perms);+}++// Installs the unveil() read/read-write rules (does NOT lock; caller locks+// with unveil(0,0)). The executable is always readable -- llamafile_open_zip+// reopens it for embedded weights and the bundled web UI -- as are the name+// resolution files a non-numeric --host needs. rw paths (which may not exist+// yet, e.g. a prompt cache created on first save) get their container dir.+static void unveil_apply(const char *const *read_paths, int n_read,+ const char *const *rw_paths, int n_rw) {+ unveil(GetProgramExecutableName(), "r");+ if (path_on_disk("/etc/hosts"))+ unveil("/etc/hosts", "r");+ if (path_on_disk("/etc/resolv.conf"))+ unveil("/etc/resolv.conf", "r");+ for (int i = 0; i < n_read; ++i)+ unveil_container(read_paths[i], "r");+ for (int i = 0; i < n_rw; ++i)+ unveil_container(rw_paths[i], "rwc");+}++// True if `path` opens read-only under the (now locked) ruleset.+static bool can_read(const char *path) {+ int fd = open(path, O_RDONLY);+ if (fd < 0)+ return false;+ close(fd);+ return true;+}++// True only if opening `path` is *actively denied* (EACCES/EPERM), not+// merely failing for some other reason such as ENOENT.+static bool is_denied(const char *path) {+ errno = 0;+ int fd = open(path, O_RDONLY);+ if (fd >= 0) {+ close(fd);+ return false;+ }+ return errno == EACCES || errno == EPERM;+}++// Probes whether unveil() confinement can be both enforced and survived on+// this filesystem, in a throwaway child so a "no" doesn't lock down the+// real process. Two failure modes to rule out:+// - it installed nothing (no Landlock kernel, or cosmo's silent no-op):+// the canary outside every rule stays readable -> not confined;+// - it over-denies (virtiofs/9p/NFS accept rules then block everything):+// a governed weights path can't be opened -> would break model load.+// Every path_on_disk()/container check runs before the lock, since access()+// is itself governed once the ruleset is locked.+#define SANDBOX_PROBE_MAX 64+static bool unveil_is_governable(const char *const *read_paths, int n_read,+ const char *const *rw_paths, int n_rw) {+ pid_t pid = fork();+ if (pid < 0)+ return false;+ if (!pid) {+ // Snapshot the paths the locked ruleset must still let us read,+ // computed while access() is unrestricted.+ char verify[SANDBOX_PROBE_MAX][PATH_MAX];+ int nv = 0;+ strlcpy(verify[nv++], GetProgramExecutableName(), PATH_MAX);+ for (int i = 0; i < n_read && nv < SANDBOX_PROBE_MAX; ++i)+ if (path_on_disk(read_paths[i]))+ strlcpy(verify[nv++], read_paths[i], PATH_MAX);+ for (int i = 0; i < n_rw && nv < SANDBOX_PROBE_MAX; ++i)+ if (container_of(rw_paths[i], verify[nv], PATH_MAX))+ ++nv;++ // Pick a canary that exists now, before the ruleset can hide it.+ const char *canary = 0;+ for (size_t i = 0; i < sizeof(kSandboxCanaries) / sizeof(*kSandboxCanaries); ++i)+ if (!access(kSandboxCanaries[i], F_OK)) {+ canary = kSandboxCanaries[i];+ break;+ }++ unveil_apply(read_paths, n_read, rw_paths, n_rw);+ unveil(0, 0); // lock++ int ok = 1;+ for (int i = 0; ok && i < nv; ++i)+ ok = can_read(verify[i]);+ // Confinement must demonstrably bite: the canary, outside every+ // rule, must now be *denied* (EACCES/EPERM). No canary, or one that+ // fails for another reason, is inconclusive -- treat as "not+ // confined" rather than claiming a guarantee unveil() didn't install.+ if (ok && (!canary || !is_denied(canary)))+ ok = 0;+ _exit(ok ? 0 : 1);+ }+ int ws;+ while (waitpid(pid, &ws, 0) < 0 && errno == EINTR) {+ }+ return WIFEXITED(ws) && WEXITSTATUS(ws) == 0;+}++#endif // COSMOCC++// Applies the server sandbox: pledge() always (accept()-only networking, no+// writes/exec unless configured), plus unveil() path confinement when+// spec->confine and the filesystem can enforce it. Fills promises_out with+// the pledge string for logging; returns an ACTIVE* status or a skip/FAILED+// code. The caller must quiesce background threads around this call (the+// filters are per-thread; see rule 1 above).+int llamafile_sandbox_server(const struct llamafile_sandbox_spec *spec,+ char *promises_out, size_t promises_len) {+ if (promises_out && promises_len)+ promises_out[0] = '\0';+ int skip = sandbox_skip_status();+ if (skip != LLAMAFILE_SANDBOX_ACTIVE)+ return skip;+#ifdef COSMOCC+ // wpath/cpath only when a write target is actually configured; callers+ // pass fixed-position entries that are often empty strings.+ bool has_rw = false;+ for (int i = 0; i < spec->n_rw; ++i)+ if (spec->rw_paths[i] && *spec->rw_paths[i])+ has_rw = true;+ char promises[64];+ llamafile_sandbox_server_promises(promises, sizeof(promises), IsOpenbsd(),+ has_rw, spec->needs_outbound);+ if (promises_out)+ strlcpy(promises_out, promises, promises_len);++ // Opt-in unveil() path confinement, run before pledge() (it needs+ // Landlock syscalls the pledge filter denies). Skip it when the+ // filesystem can't enforce it, rather than locking down a server that can+ // no longer read its weights. 0 = not requested, 1 = confined, 2 = asked+ // for but unavailable.+ int confine_state = 0;+ if (spec->confine) {+ if (unveil_is_governable(spec->read_paths, spec->n_read,+ spec->rw_paths, spec->n_rw)) {+ unveil_apply(spec->read_paths, spec->n_read,+ spec->rw_paths, spec->n_rw);+ unveil(0, 0); // lock the ruleset+ confine_state = 1;+ } else {+ confine_state = 2;+ }+ }++ if (sandbox_install_pledge(promises) == LLAMAFILE_SANDBOX_FAILED)+ return LLAMAFILE_SANDBOX_FAILED;+ return confine_state == 1 ? LLAMAFILE_SANDBOX_ACTIVE_CONFINED+ : confine_state == 2 ? LLAMAFILE_SANDBOX_ACTIVE_UNCONFINED+ : LLAMAFILE_SANDBOX_ACTIVE;+#else+ (void)spec;+ return LLAMAFILE_SANDBOX_UNSUPPORTED;+#endif+}++// True for the statuses where the pledge() filter is installed (whether or+// not path confinement also applied), as opposed to a skip or failure.+bool llamafile_sandbox_is_active(int status) {+ return status == LLAMAFILE_SANDBOX_ACTIVE ||+ status == LLAMAFILE_SANDBOX_ACTIVE_CONFINED ||+ status == LLAMAFILE_SANDBOX_ACTIVE_UNCONFINED;+}++const char *llamafile_sandbox_describe(int status) {+ switch (status) {+ case LLAMAFILE_SANDBOX_ACTIVE:+ return "active";+ case LLAMAFILE_SANDBOX_ACTIVE_CONFINED:+ return "active, reads confined to weights dirs";+ case LLAMAFILE_SANDBOX_ACTIVE_UNCONFINED:+ return "active (path confinement requested but unavailable on this filesystem)";+ case LLAMAFILE_SANDBOX_UNSECURE:+ return "disabled by --unsecure";+ case LLAMAFILE_SANDBOX_GPU:+ return "disabled in GPU mode";+ case LLAMAFILE_SANDBOX_UNSUPPORTED:+ return "not supported on this OS";+ case LLAMAFILE_SANDBOX_FAILED:+ return "failed";+ default:+ return "unknown";+ }+}diff --git a/tests/BUILD.mk b/tests/BUILD.mkindex 9ec82eff..b6ce417f 100644--- a/tests/BUILD.mk+++ b/tests/BUILD.mk@@ -120,6 +120,28 @@ o/$(MODE)/tests/gpu_backend_test: \@mkdir -p $(@D)$(CXX) $(LDFLAGS) -o $@ $^ $(LDLIBS)+# ==============================================================================+# Test: sandbox_test (issue #930 pledge/SECCOMP sandboxing)+# ==============================================================================+#+# Forks children that install the promise sets llamafile uses (server:+# stdio+anet, cli: stdio+rpath+tty) and asserts allowed syscalls still+# work while blocked ones fail with EPERM. Self-skips on OSes where+# pledge() can't be enforced (anything but Linux/OpenBSD). Links against+# sandbox.o alone: it stubs llamafile_has_gpu() to stay off the GPU stack.++SANDBOX_TEST_DEPS := \+ o/$(MODE)/llamafile/sandbox.o++# sandbox_test.o is built by the generic %.o: %.c rule (-iquote. resolves+# the llamafile/llamafile.h include)++o/$(MODE)/tests/sandbox_test: \+ o/$(MODE)/tests/sandbox_test.o \+ $(SANDBOX_TEST_DEPS)+ @mkdir -p $(@D)+ $(CC) $(LDFLAGS) -o $@ $^ $(LDLIBS)+# ==============================================================================# Test: backend_ops_test (upstream ggml test-backend-ops under llamafile)# ==============================================================================@@ -183,4 +205,5 @@ o/$(MODE)/tests: \o/$(MODE)/tests/extract_data_uris_test.runs \o/$(MODE)/tests/fa_helpers_test.runs \o/$(MODE)/tests/gpu_backend_test.runs \+ o/$(MODE)/tests/sandbox_test.runs \o/$(MODE)/tests/transcribefile_smoke.runsdiff --git a/tests/integration/README.md b/tests/integration/README.mdindex 0f7f9489..2ea61732 100644--- a/tests/integration/README.md+++ b/tests/integration/README.md@@ -9,6 +9,8 @@ Integration tests for llamafile covering CLI, TUI, and server modes.## Running Tests+### Single model+```bashcd tests/integration@@ -22,6 +24,22 @@ cd tests/integration./run_tests.sh --executable ~/model.llamafile -v```+### Full suite across all models++`tests/run_integration_tests.sh` runs the full suite across a configurable list+of models, then runs the `help` and `ssl` tests once against the bare executable.+Edit the `TESTS` array and path variables at the top of the script to match your+local model collection.++```bash+# From the repo root — runs every model in the TESTS list, then help + ssl+./tests/run_integration_tests.sh+```++> **Network access**: the script includes the `ssl online` tests, which download+> a small model from Hugging Face. Make sure you have internet access before+> running it, or remove the `ssl and online` invocation from the script.+## Test CategoriesUse `-m` to select test categories:@@ -37,8 +55,11 @@ Use `-m` to select test categories:| `thinking` | Thinking model tests (QwQ, DeepSeek-R1, etc.) || `gpu` | GPU acceleration tests || `cpu` | CPU-only tests |-| `ssl` | HTTPS/TLS tests: serving with `--ssl-cert-file`, model download over HTTPS, untrusted-cert rejection. Needs `openssl` in PATH to generate a test certificate |-| `online` | Tests that need network access (currently the HTTPS model download). Skip with `-m "not online"` |+| `help` | `--help` output tests; no model needed |+| `bare_executable` | Tests that run once against the bare binary (no embedded model); automatically skipped when the executable is a `.llamafile` bundle. Includes `help` and `ssl`. |+| `ssl` | HTTPS/TLS tests: serving with `--ssl-cert-file` and model download over HTTPS. Needs `openssl` in PATH. Run against the bare executable with `--model`. |+| `determinism` | Tests requiring reproducible output at temperature=0. Unreliable on gpt-oss models (see `tests/run_integration_tests.sh`); deselect with `-m "not determinism"`. |+| `online` | Tests that need network access (the HTTPS model download from Hugging Face). Skip with `-m "not online"` if offline. |Examples:@@ -52,11 +73,9 @@ Examples:# Skip multimodal and tool_calling tests./run_tests.sh --executable ~/model.llamafile -m "not multimodal and not tool_calling"-# Run only the HTTPS/TLS tests (or skip them with -m "not ssl")-./run_tests.sh --executable ./o/llamafile/llamafile --model model.gguf -m ssl--# Run the HTTPS/TLS tests without the network-dependent ones+# Run the ssl tests (bare executable required; --model for the serving tests)./run_tests.sh --executable ./o/llamafile/llamafile --model model.gguf -m "ssl and not online"+./run_tests.sh --executable ./o/llamafile/llamafile -m "ssl and online"```## Options@@ -91,20 +110,25 @@ Use `--log-cli-level` to see what the model returns:## Test Structure```-tests/integration/-├── run_tests.sh # Test runner script-├── conftest.py # Pytest fixtures-├── pyproject.toml # Dependencies and pytest config-├── utils/-│ └── llamafile.py # LlamafileRunner utility class-├── fixtures/-│ └── test_image.png # Test image for multimodal tests-└── tests/- ├── test_cli.py # CLI mode tests- ├── test_tui.py # TUI mode tests- ├── test_server.py # Server mode tests- ├── test_combined.py # TUI+Server simultaneous mode- ├── test_multimodal.py # Image description tests- ├── test_tool_calling.py # Tool use tests- └── test_gpu.py # GPU/CPU execution tests+tests/+├── run_integration_tests.sh # Full-suite runner across all models (edit to customise)+└── integration/+ ├── run_tests.sh # Single-model test runner+ ├── conftest.py # Pytest fixtures+ ├── pyproject.toml # Dependencies and pytest config+ ├── utils/+ │ └── llamafile.py # LlamafileRunner utility class and shared helpers+ ├── fixtures/+ │ └── test_image.png # Test image for multimodal tests+ └── tests/+ ├── test_cli.py # CLI mode tests+ ├── test_tui.py # TUI/chat mode tests+ ├── test_server.py # Server mode tests+ ├── test_combined.py # TUI+Server simultaneous mode+ ├── test_multimodal.py # Image description tests+ ├── test_tool_calling.py # Tool use tests+ ├── test_gpu.py # GPU/CPU execution tests+ ├── test_help.py # --help output tests (bare executable, no model)+ ├── test_ssl.py # HTTPS/TLS serving and model download tests+ └── test_sandbox.py # pledge/SECCOMP sandboxing tests```diff --git a/tests/integration/pyproject.toml b/tests/integration/pyproject.tomlindex 8b47aa39..0d19c3f8 100644--- a/tests/integration/pyproject.toml+++ b/tests/integration/pyproject.toml@@ -29,4 +29,7 @@ markers = ["help: tests --help output (no model required)","ssl: tests HTTPS/TLS support (serving and model download)","online: tests that need network access (deselect with '-m \"not online\"')",+ "sandbox: tests pledge()/SECCOMP sandboxing (enforcement checks need Linux)",+ "bare_executable: requires a bare llamafile binary with no embedded model (skip for pre-built .llamafile bundles)",+ "determinism: requires reproducible output at temperature=0 (unreliable on gpt-oss models, whose long forced reasoning trace and MXFP4 quantization give a threaded matmul rounding difference more opportunities to flip a near-tied token; see tests/run_integration_tests.sh)",]diff --git a/tests/integration/tests/test_cli.py b/tests/integration/tests/test_cli.pyindex 51113e27..d636e5fc 100644--- a/tests/integration/tests/test_cli.py+++ b/tests/integration/tests/test_cli.py@@ -2,6 +2,8 @@import pytest+from utils.prompts import ADD_2_2, GREETING_PROMPT+@pytest.mark.cliclass TestCLIBasic:@@ -9,19 +11,19 @@ class TestCLIBasic:def test_cli_responds(self, llamafile, timeouts):"""Test that CLI mode starts and generates a response."""- result = llamafile.run_cli("Say hello in exactly one word.", timeout=timeouts.cli)+ result = llamafile.run_cli(GREETING_PROMPT, timeout=timeouts.cli)assert result.returncode == 0, f"CLI failed: {result.stderr}"assert len(result.stdout.strip()) > 0, "No output generated"def test_cli_math_question(self, llamafile, timeouts):"""Test that CLI can answer a simple math question."""- result = llamafile.run_cli(- "What is 2+2? Answer with just the number.", timeout=timeouts.cli- )+ result = llamafile.run_cli(ADD_2_2.prompt, timeout=timeouts.cli)assert result.returncode == 0, f"CLI failed: {result.stderr}"- assert "4" in result.stdout, f"Expected '4' in output: {result.stdout}"+ assert ADD_2_2.check(result.stdout), (+ f"Expected {ADD_2_2.describe()} in output: {result.stdout}"+ )def test_cli_exits_cleanly(self, llamafile, timeouts):"""Test that CLI exits with code 0 after completion."""diff --git a/tests/integration/tests/test_combined.py b/tests/integration/tests/test_combined.pyindex ccb7823c..5ffff04f 100644--- a/tests/integration/tests/test_combined.py+++ b/tests/integration/tests/test_combined.py@@ -3,6 +3,7 @@import pytestfrom utils.llamafile import LlamafileRunner, read_until_idle, stop_tui+from utils.prompts import ADD_1_1, ADD_2_2, ADD_3_3, GREETING_PROMPT@pytest.mark.tui@@ -17,13 +18,13 @@ class TestCombinedMode:try:ready = LlamafileRunner.wait_for_server(- server_port, timeout=timeouts.server_ready+ server_port, timeout=timeouts.server_ready, proc=proc)assert ready, "Server did not become ready in combined mode"response = LlamafileRunner.chat_completion(port=server_port,- messages=[{"role": "user", "content": "Say hello"}],+ messages=[{"role": "user", "content": GREETING_PROMPT}],timeout=timeouts.http_request,)@@ -39,7 +40,7 @@ class TestCombinedMode:try:ready = LlamafileRunner.wait_for_server(- server_port, timeout=timeouts.server_ready+ server_port, timeout=timeouts.server_ready, proc=proc)assert ready, "Server did not become ready"@@ -49,13 +50,14 @@ class TestCombinedMode:# Test 1: Send a request via server APIresponse1 = LlamafileRunner.chat_completion(port=server_port,- messages=[{"role": "user", "content": "What is 1+1?"}],+ messages=[{"role": "user", "content": ADD_1_1.prompt}],timeout=timeouts.http_request,)- assert "2" in response1["choices"][0]["message"]["content"]+ content1 = response1["choices"][0]["message"]["content"]+ assert ADD_1_1.check(content1), f"Expected {ADD_1_1.describe()} in content: {content1}"# Test 2: Send TUI input and verify response- proc.stdin.write("What is 2+2?\n")+ proc.stdin.write(f"{ADD_2_2.prompt}\n")proc.stdin.flush()# Read TUI output until model stops generating@@ -65,15 +67,18 @@ class TestCombinedMode:max_timeout=timeouts.cli,)assert len(tui_output) > 0, "TUI produced no output"- assert "4" in tui_output, f"Expected '4' in TUI output: {tui_output}"+ assert ADD_2_2.check(tui_output), (+ f"Expected {ADD_2_2.describe()} in TUI output: {tui_output}"+ )# Test 3: Server should still work after TUI interactionresponse2 = LlamafileRunner.chat_completion(port=server_port,- messages=[{"role": "user", "content": "What is 3+3?"}],+ messages=[{"role": "user", "content": ADD_3_3.prompt}],timeout=timeouts.http_request,)- assert "6" in response2["choices"][0]["message"]["content"]+ content2 = response2["choices"][0]["message"]["content"]+ assert ADD_3_3.check(content2), f"Expected {ADD_3_3.describe()} in content: {content2}"finally:stop_tui(proc)diff --git a/tests/integration/tests/test_gpu.py b/tests/integration/tests/test_gpu.pyindex 2d97f783..08a072e3 100644--- a/tests/integration/tests/test_gpu.py+++ b/tests/integration/tests/test_gpu.py@@ -8,6 +8,7 @@ import tempfileimport pytestfrom utils.llamafile import LlamafileRunner+from utils.prompts import GREETING_PROMPTdef get_available_gpu() -> str | None:@@ -139,7 +140,7 @@ class TestGPUAcceleration:try:result = runner.run_cli(- "Say hello",+ GREETING_PROMPT,timeout=timeouts.cli,log_file=log_path,extra_args=["--verbose"],@@ -176,14 +177,14 @@ class TestGPUAcceleration:try:ready = LlamafileRunner.wait_for_server(- server_port, timeout=timeouts.server_ready+ server_port, timeout=timeouts.server_ready, proc=proc)assert ready# Verify response worksresponse = LlamafileRunner.chat_completion(port=server_port,- messages=[{"role": "user", "content": "Say hello"}],+ messages=[{"role": "user", "content": GREETING_PROMPT}],timeout=timeouts.http_request,)assert len(response["choices"][0]["message"]["content"]) > 0@@ -222,7 +223,7 @@ class TestCPUExecution:try:result = runner.run_cli(- "Say hello",+ GREETING_PROMPT,timeout=timeouts.cli,log_file=log_path,extra_args=["--verbose"],@@ -259,14 +260,14 @@ class TestCPUExecution:try:ready = LlamafileRunner.wait_for_server(- server_port, timeout=timeouts.server_ready+ server_port, timeout=timeouts.server_ready, proc=proc)assert ready# Verify response worksresponse = LlamafileRunner.chat_completion(port=server_port,- messages=[{"role": "user", "content": "Say hello"}],+ messages=[{"role": "user", "content": GREETING_PROMPT}],timeout=timeouts.http_request,)assert len(response["choices"][0]["message"]["content"]) > 0diff --git a/tests/integration/tests/test_help.py b/tests/integration/tests/test_help.pyindex 5f540634..acc5284e 100644--- a/tests/integration/tests/test_help.py+++ b/tests/integration/tests/test_help.py@@ -15,6 +15,8 @@ import subprocessimport pytest+from utils.llamafile import skip_if_bundled+HELP_TIMEOUT = 30.0@@ -135,9 +137,23 @@ class TestHelpListsLlamaCppArgs:@pytest.mark.help[email protected]_executableclass TestHelpMissingModel:"""The missing-model error path should still print a helpful intro and- point the user at --help, without requiring the full option list."""+ point the user at --help, without requiring the full option list.++ These tests require a bare llamafile binary (no embedded model). When run+ against a pre-built .llamafile that already contains a model the binary+ starts normally instead of printing an error, so the tests are skipped+ automatically in that case. To exercise this code path, run against the+ build output directly, e.g.::++ ./run_tests.sh --executable ~/llamafile/o/llamafile/llamafile+ """++ @pytest.fixture(autouse=True)+ def _require_bare_executable(self, executable):+ skip_if_bundled(executable)def _run_no_model(self, executable):if platform.system() == "Windows":diff --git a/tests/integration/tests/test_multimodal.py b/tests/integration/tests/test_multimodal.pyindex 693afb54..55416660 100644--- a/tests/integration/tests/test_multimodal.py+++ b/tests/integration/tests/test_multimodal.py@@ -128,7 +128,7 @@ class TestMultimodalServer:try:ready = LlamafileRunner.wait_for_server(- server_port, timeout=timeouts.server_ready+ server_port, timeout=timeouts.server_ready, proc=proc)assert ready, "Server did not become ready"@@ -153,7 +153,7 @@ class TestMultimodalServer:try:ready = LlamafileRunner.wait_for_server(- server_port, timeout=timeouts.server_ready+ server_port, timeout=timeouts.server_ready, proc=proc)assert readydiff --git a/tests/integration/tests/test_sandbox.py b/tests/integration/tests/test_sandbox.pynew file mode 100644index 00000000..6b5f86a4--- /dev/null+++ b/tests/integration/tests/test_sandbox.py@@ -0,0 +1,465 @@+"""Sandbox (pledge/SECCOMP/unveil) integration tests.++llamafile sandboxes itself with Cosmopolitan's pledge() and unveil() (issue+#930). On Linux pledge() installs a SECCOMP BPF filter and unveil() uses+Landlock. These tests verify externally that the sandbox is really in place:++ - a pledged process carries one more SECCOMP filter than its parent+ (/proc/<pid>/status ``Seccomp_filters``); the comparison is relative to+ the pytest process because container runtimes (e.g. Docker's default+ profile) already wrap everything in a filter of their own;+ - every thread of the server carries that filter, not just the main one+ (guards against a pre-existing thread escaping the per-thread filter);+ - inference keeps working while sandboxed, including for a bundled+ ``/zip/`` llamafile whose weights load through the executable;+ - ``--unsecure``, combined mode, and GPU mode skip the sandbox and say so.++pledge() is on by default; unveil() path confinement is opt-in via+``--confine-reads`` (the default must NOT confine, so files opened by path at+request time keep working). The confinement tests below check that the default+does not confine and that ``--confine-reads`` does; the syscall-level proof+that confinement denies reads outside the weights dirs is in sandbox_test.c.++Enforcement is only observable on Linux; on other hosts the enforcement+tests skip and we only check that llamafile reports its status honestly.+tests/sandbox_test.c covers the syscall-level allow/deny behavior and the+unveil() path confinement directly.+"""++import os+import platform+import shutil+import subprocess+import time+from pathlib import Path++import pytest++from utils.llamafile import LlamafileRunner+from utils.prompts import GREETING_PROMPT++IS_LINUX = platform.system() == "Linux"++requires_linux = pytest.mark.skipif(+ not IS_LINUX, reason="SECCOMP enforcement is only observable on Linux"+)+++def _status_field(pid_or_tid_path: str, field: str) -> int | None:+ try:+ with open(pid_or_tid_path) as f:+ for line in f:+ if line.startswith(field + ":"):+ return int(line.split()[1])+ except OSError:+ pass+ return None+++def _seccomp_filters(pid: int) -> int | None:+ """Exact count of SECCOMP filters on <pid>, or None on kernels older+ than 5.9 that don't expose ``Seccomp_filters``. Callers skip the strict+ comparison when this is None rather than guessing from the coarse+ ``Seccomp`` mode field (which can't tell a container filter apart from+ our own)."""+ return _status_field(f"/proc/{pid}/status", "Seccomp_filters")+++def _seccomp_mode(pid: int) -> int | None:+ """The coarse ``Seccomp`` mode field: 2 (SECCOMP_MODE_FILTER) means at+ least one filter is installed, 0 means none. Available on every kernel+ with SECCOMP. Can't distinguish our filter from a container's, so it's+ only a positive 'a filter exists' check, used as a fallback when the+ exact count is unavailable so a fully-unsandboxed build still fails."""+ return _status_field(f"/proc/{pid}/status", "Seccomp")+++def _assert_sandbox_installed(pid: int):+ """A sandboxed process gains at least one filter beyond the runner's+ baseline. Prefer the exact count; fall back to the mode field on old+ kernels so a totally unsandboxed regression can't skip through."""+ filters, base = _seccomp_filters(pid), _baseline_filters()+ if filters is not None and base is not None:+ assert filters > base, (+ "expected the process to gain a SECCOMP filter; the pledge() "+ "sandbox is not installed"+ )+ else:+ assert _seccomp_mode(pid) == 2, (+ "expected SECCOMP_MODE_FILTER; the pledge() sandbox is not installed"+ )+++def _assert_no_added_filter(pid: int):+ """The process carries no filter beyond the runner's baseline (the+ sandbox was skipped). Needs the exact count to tell our filter from a+ container's; skip the check where it's unavailable."""+ filters, base = _seccomp_filters(pid), _baseline_filters()+ if filters is None or base is None:+ pytest.skip("kernel lacks Seccomp_filters; cannot count filters")+ assert filters == base, "expected no added SECCOMP filter"+++def _thread_filter_counts(pid: int) -> dict[int, int]:+ """Per-thread SECCOMP filter counts for every thread of <pid>."""+ counts = {}+ for tid in os.listdir(f"/proc/{pid}/task"):+ n = _status_field(f"/proc/{pid}/task/{tid}/status", "Seccomp_filters")+ if n is not None:+ counts[int(tid)] = n+ return counts+++def _baseline_filters() -> int | None:+ """SECCOMP filters every spawned process inherits from this test runner+ (0 on bare metal, typically 1 under Docker's default profile)."""+ return _seccomp_filters(os.getpid())+++def _resolve_server_pid(pid: int) -> int:+ """Follow the sh -> APE-loader chain down to the llamafile process.++ The runner launches ``sh <executable>``. The APE shell header normally+ exec()s in place so the PID stays the same, but if any stage forked we+ walk single-child links downward.+ """+ for _ in range(5):+ children = []+ try:+ for tid in os.listdir(f"/proc/{pid}/task"):+ with open(f"/proc/{pid}/task/{tid}/children") as f:+ children.extend(int(p) for p in f.read().split())+ except OSError:+ break+ if not children:+ break+ pid = children[0]+ return pid+++def _stop_hard(proc):+ """Terminate a process without ever hanging the test: SIGTERM first+ (llamafile shuts down gracefully), SIGKILL if it doesn't oblige."""+ proc.terminate()+ try:+ proc.wait(timeout=15)+ except subprocess.TimeoutExpired:+ proc.kill()+ proc.wait()+++def cpu_runner(executable, model) -> LlamafileRunner:+ """Runner pinned to CPU, with --no-warmup.++ --gpu disable: a loaded GPU backend disables the sandbox (drivers need+ syscalls pledge forbids), so CPU is forced to make sandbox state+ deterministic on any host.++ --no-warmup: large models (e.g. 26B MXFP4 MoE) can take longer than the+ 120s server-ready timeout when the kernel page cache is cold, because+ pledge may restrict the prefetch hints (posix_fadvise / MADV_WILLNEED)+ that speed up the initial memory-mapped read. Sandbox tests care about+ whether the pledge filter is installed and whether inference works, not+ about the warmup pass itself.+ """+ return LlamafileRunner(executable=executable, model=model, gpu="disable", no_warmup=True)+++class TestServerSandbox:+ @requires_linux+ def test_server_sandboxed_by_default(self, cpu_runner, server_port, timeouts, tmp_path):+ """--server installs a SECCOMP filter on every thread and still+ serves completions inside the sandbox."""+ log_file = str(tmp_path / "server.log")+ proc = cpu_runner.start_server(port=server_port, log_file=log_file)+ try:+ assert LlamafileRunner.wait_for_server(+ server_port, timeout=timeouts.server_ready, proc=proc+ ), "Server did not become ready"++ pid = _resolve_server_pid(proc.pid)+ # The sandbox adds at least the pledge filter (and, when unveil+ # confinement applies, cosmo's extra truncate/setxattr filter);+ # the exact count is cosmo's business, so assert presence not a+ # specific delta. The negative cases below pin the baseline.+ _assert_sandbox_installed(pid)++ # Every thread must carry the same filter count, not just the+ # main thread: a pre-existing background thread (e.g. the log+ # worker) would otherwise escape the per-thread SECCOMP filter+ # (issue #930). Needs the exact per-thread count (5.9+); where+ # unavailable the presence check above still holds.+ counts = _thread_filter_counts(pid)+ if counts:+ base = _baseline_filters()+ assert set(counts.values()) == {max(counts.values())}, (+ f"threads have mismatched SECCOMP filter counts {counts}; "+ f"a thread escaped the sandbox"+ )+ if base is not None:+ assert max(counts.values()) > base, "no thread is sandboxed"++ # inference must keep working inside the sandbox+ response = LlamafileRunner.chat_completion(+ port=server_port,+ messages=[{"role": "user", "content": GREETING_PROMPT}],+ timeout=timeouts.http_request,+ )+ assert response["choices"][0]["message"]["content"].strip()+ finally:+ _stop_hard(proc)++ log = LlamafileRunner.read_log_file(log_file)+ assert 'sandbox: pledge("' in log, "server should log the active promise set"++ @requires_linux+ def test_unsecure_flag_disables_sandbox(self, cpu_runner, server_port, timeouts, tmp_path):+ """--unsecure opts out: no SECCOMP filter, and the log says why."""+ log_file = str(tmp_path / "server.log")+ proc = cpu_runner.start_server(+ port=server_port, log_file=log_file, extra_args=["--unsecure"]+ )+ try:+ assert LlamafileRunner.wait_for_server(+ server_port, timeout=timeouts.server_ready, proc=proc+ ), "Server did not become ready"+ _assert_no_added_filter(_resolve_server_pid(proc.pid))+ finally:+ _stop_hard(proc)++ log = LlamafileRunner.read_log_file(log_file)+ assert "disabled by --unsecure" in log++ def test_sandbox_status_always_logged(self, cpu_runner, server_port, timeouts, tmp_path):+ """On every platform the server reports its sandbox state honestly+ (active on Linux/OpenBSD, an explicit skip reason elsewhere)."""+ log_file = str(tmp_path / "server.log")+ proc = cpu_runner.start_server(port=server_port, log_file=log_file)+ try:+ assert LlamafileRunner.wait_for_server(+ server_port, timeout=timeouts.server_ready, proc=proc+ ), "Server did not become ready"+ finally:+ _stop_hard(proc)++ log = LlamafileRunner.read_log_file(log_file)+ assert "sandbox:" in log, "server must log its sandbox status"+ if not IS_LINUX:+ assert "not supported on this OS" in log+++class TestConfineReads:+ """The pledge()/unveil() split: pledge() is on by default, unveil() path+ confinement is opt-in via --confine-reads."""++ @requires_linux+ def test_default_does_not_confine(self, cpu_runner, server_port, timeouts, tmp_path):+ """By default the server is pledged but NOT path-confined, so files+ opened by path at request time (multimodal media, etc.) keep working.+ The log must say active without claiming confinement."""+ log_file = str(tmp_path / "server.log")+ proc = cpu_runner.start_server(port=server_port, log_file=log_file)+ try:+ assert LlamafileRunner.wait_for_server(+ server_port, timeout=timeouts.server_ready, proc=proc+ ), "Server did not become ready"+ finally:+ _stop_hard(proc)+ log = LlamafileRunner.read_log_file(log_file)+ assert 'pledge("stdio anet rpath")' in log, "default should pledge, no confinement"+ assert "confined to weights dirs" not in log, "default must not confine reads"++ @requires_linux+ def test_confine_reads_confines_and_serves(self, cpu_runner, server_port, timeouts, tmp_path):+ """--confine-reads adds unveil() confinement and still serves."""+ log_file = str(tmp_path / "server.log")+ proc = cpu_runner.start_server(+ port=server_port, log_file=log_file, extra_args=["--confine-reads"]+ )+ try:+ assert LlamafileRunner.wait_for_server(+ server_port, timeout=timeouts.server_ready, proc=proc+ ), "Server did not become ready"+ _assert_sandbox_installed(_resolve_server_pid(proc.pid))+ resp = LlamafileRunner.chat_completion(+ port=server_port,+ messages=[{"role": "user", "content": GREETING_PROMPT}],+ timeout=timeouts.http_request,+ )+ assert resp["choices"][0]["message"]["content"].strip()+ finally:+ _stop_hard(proc)+ log = LlamafileRunner.read_log_file(log_file)+ # This log line is emitted only after the governability probe confirms+ # a canary outside the rules is actually denied (see sandbox.c), so it+ # means confinement is installed and biting -- not merely requested.+ assert "reads confined to weights dirs" in log++ # Note: there is no integration test that drives a *live* server into+ # reading a file outside the weights dirs, because the HTTP API exposes no+ # arbitrary-read primitive to exploit -- static serving (httplib) and media+ # loading (fs_validate_filename) both reject traversal/symlinks before a+ # read is attempted, so unveil() never gets a chance to be the thing that+ # denies. unveil() is the defense-in-depth backstop *behind* those checks;+ # its enforcement (allowed inside, EACCES outside) is proven directly at+ # the syscall level in tests/sandbox_test.c (child_unveil).+++def _find_zipalign(executable: str) -> str | None:+ """Locate the zipalign tool for building a bundled llamafile: env+ override, then the sibling build tree of a direct build, then PATH."""+ env = os.environ.get("LLAMAFILE_ZIPALIGN")+ if env and os.path.exists(env):+ return env+ for parent in Path(executable).resolve().parents:+ cand = parent / "third_party" / "zipalign" / "zipalign"+ if cand.exists():+ return str(cand)+ return shutil.which("zipalign")+++class TestBundledLlamafileSandbox:+ @requires_linux+ def test_embedded_zip_model_loads_under_sandbox(+ self, executable, model, server_port, timeouts, tmp_path+ ):+ """A bundled llamafile references its weights as ``/zip/model.gguf``;+ the loader reads them by reopening the executable's zip store, which+ needs the rpath promise. This is the exact configuration issue #930's+ rpath fix targets — a naive 'embedded => no filesystem' policy makes+ the model fail to load. Build a real bundle and assert it serves."""+ if not model:+ pytest.skip("needs --model to embed into a bundle")+ zipalign = _find_zipalign(executable)+ if not zipalign:+ pytest.skip("zipalign not found (set LLAMAFILE_ZIPALIGN)")++ bundle = tmp_path / "bundled.llamafile"+ shutil.copy(executable, bundle)+ os.chmod(bundle, 0o755)+ model_name = os.path.basename(model)+ args = tmp_path / ".args"+ args.write_text(f"-m\n/zip/{model_name}\n--server\n--gpu\ndisable\n")+ # zipalign is an APE binary; run it through sh like the llamafile+ # itself, so hosts without binfmt_misc for APE still work.+ za = ["sh", zipalign] if os.name != "nt" else [zipalign]+ subprocess.run(za + ["-j0", str(bundle), model, str(args)], check=True)++ log_file = str(tmp_path / "bundled.log")+ proc = subprocess.Popen(+ ["sh", str(bundle), "--port", str(server_port), "--log-file", log_file],+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,+ )+ try:+ assert LlamafileRunner.wait_for_server(+ server_port, timeout=timeouts.server_ready, proc=proc+ ), "bundled /zip/ server failed to load under the sandbox"+ finally:+ _stop_hard(proc)++ log = LlamafileRunner.read_log_file(log_file)+ assert 'sandbox: pledge("' in log+ assert "rpath" in log, "embedded weights still need the rpath promise"+++class TestCliSandbox:+ @requires_linux+ def test_cli_sandboxed_and_functional(self, cpu_runner, timeouts):+ """--cli pledges "stdio rpath tty" after arg parsing and still+ generates a response. --verbose surfaces the status line on+ stderr (CLI keeps stdout clean for the model output)."""+ result = cpu_runner.run_cli(+ GREETING_PROMPT, extra_args=["--verbose"], timeout=timeouts.cli+ )+ assert result.returncode == 0+ assert "sandbox: active" in result.stderr+ assert result.stdout.strip()+++class TestChatSandbox:+ @requires_linux+ def test_chat_all_threads_sandboxed(self, cpu_runner, timeouts, tmp_path):+ """Chat mode wraps the pledge in log pause/resume like the server,+ so no pre-existing thread (the log worker) escapes the per-thread+ filter. Launch headless with stdin held open so the REPL blocks+ after sandboxing, then confirm every thread carries the filter+ (issue #930 — the chat/cli path the round-1 tests missed)."""+ # The "sandbox:" status is written to stderr (not the common-log+ # --log-file), so capture stderr to detect when the pledge is in.+ err_file = tmp_path / "chat.err"+ args = cpu_runner._base_args() + ["--chat", "--verbose", "--nologo"]+ with open(err_file, "wb") as err:+ proc = subprocess.Popen(+ args, stdin=subprocess.PIPE,+ stdout=subprocess.DEVNULL, stderr=err,+ )+ try:+ deadline = time.time() + timeouts.server_ready+ ready = False+ while time.time() < deadline:+ if proc.poll() is not None:+ pytest.fail("chat exited before reaching the sandbox step")+ if "sandbox:" in err_file.read_text(errors="replace"):+ ready = True+ break+ time.sleep(0.2)+ assert ready, "chat did not reach the sandbox step"++ pid = _resolve_server_pid(proc.pid)+ _assert_sandbox_installed(pid)+ counts = _thread_filter_counts(pid)+ if counts:+ assert set(counts.values()) == {max(counts.values())}, (+ f"chat threads have mismatched SECCOMP filter counts "+ f"{counts}; a thread (e.g. the log worker) escaped"+ )+ finally:+ _stop_hard(proc)++ assert "sandbox: active" in err_file.read_text(errors="replace")+++class TestCombinedModeSandbox:+ @requires_linux+ def test_combined_mode_skips_sandbox(self, cpu_runner, server_port, timeouts, tmp_path):+ """Combined mode hosts a TUI HTTP client in-process, which needs+ connect(); the accept-only sandbox is skipped and logged.++ --verbose is required: without it main.cpp injects --log-verbosity 1+ (ERROR-only threshold), suppressing the WARN-level skip notice even+ in the log file. The log file (not a pipe) is used for output so+ --verbose does not cause a pipe-buffer overflow.+ """+ log_file = str(tmp_path / "combined.log")+ proc = cpu_runner.start_combined(+ port=server_port, log_file=log_file, extra_args=["--verbose"],+ )+ try:+ assert LlamafileRunner.wait_for_server(+ server_port, timeout=timeouts.server_ready, proc=proc+ ), "Combined-mode server did not become ready"+ _assert_no_added_filter(_resolve_server_pid(proc.pid))+ finally:+ _stop_hard(proc)++ log = LlamafileRunner.read_log_file(log_file)+ assert "disabled in combined mode" in logdiff --git a/tests/integration/tests/test_server.py b/tests/integration/tests/test_server.pyindex 1215e4f1..ec07406e 100644--- a/tests/integration/tests/test_server.py+++ b/tests/integration/tests/test_server.py@@ -3,6 +3,7 @@import pytestfrom utils.llamafile import LlamafileRunner+from utils.prompts import ADD_2_2, GREETING_PROMPT@pytest.mark.server@@ -15,7 +16,7 @@ class TestServerBasic:try:ready = LlamafileRunner.wait_for_server(- server_port, timeout=timeouts.server_ready+ server_port, timeout=timeouts.server_ready, proc=proc)assert ready, "Server did not become ready in time"finally:@@ -28,13 +29,13 @@ class TestServerBasic:try:ready = LlamafileRunner.wait_for_server(- server_port, timeout=timeouts.server_ready+ server_port, timeout=timeouts.server_ready, proc=proc)assert ready, "Server did not become ready"response = LlamafileRunner.chat_completion(port=server_port,- messages=[{"role": "user", "content": "Say hello in one word."}],+ messages=[{"role": "user", "content": GREETING_PROMPT}],timeout=timeouts.http_request,)@@ -53,20 +54,18 @@ class TestServerBasic:try:ready = LlamafileRunner.wait_for_server(- server_port, timeout=timeouts.server_ready+ server_port, timeout=timeouts.server_ready, proc=proc)assert ready, "Server did not become ready"response = LlamafileRunner.chat_completion(port=server_port,- messages=[- {"role": "user", "content": "What is 2+2? Answer with just the number."}- ],+ messages=[{"role": "user", "content": ADD_2_2.prompt}],timeout=timeouts.http_request,)content = response["choices"][0]["message"]["content"]- assert "4" in content+ assert ADD_2_2.check(content), f"Expected {ADD_2_2.describe()} in content: {content}"finally:proc.terminate()@@ -78,13 +77,14 @@ class TestServerBasic:class TestServerParameters:"""Test server with various parameters."""+ @pytest.mark.determinismdef test_server_with_temperature_zero(self, llamafile, server_port, timeouts):"""Test that temperature=0 produces consistent output."""proc = llamafile.start_server(port=server_port)try:ready = LlamafileRunner.wait_for_server(- server_port, timeout=timeouts.server_ready+ server_port, timeout=timeouts.server_ready, proc=proc)assert ready@@ -130,7 +130,7 @@ class TestServerParameters:try:ready = LlamafileRunner.wait_for_server(- server_port, timeout=timeouts.server_ready+ server_port, timeout=timeouts.server_ready, proc=proc)assert readydiff --git a/tests/integration/tests/test_ssl.py b/tests/integration/tests/test_ssl.pyindex ddc9a5c8..63d5a44f 100644--- a/tests/integration/tests/test_ssl.py+++ b/tests/integration/tests/test_ssl.py@@ -1,8 +1,22 @@"""HTTPS/TLS integration tests.-Covers the TLS support added via cpp-httplib's Mbed TLS backend:-serving over HTTPS (--ssl-cert-file/--ssl-key-file), downloading models-over HTTPS (-hf/--model-url), and rejection of untrusted certificates.+Covers two distinct uses of TLS in llamafile:++1. **Serving over HTTPS** (--ssl-cert-file / --ssl-key-file): the server+ presents a certificate to connecting clients.++2. **Downloading models over HTTPS** (-hf / --model-url): llamafile acts+ as an HTTP client to fetch a model before serving it.++Note on client-side certificate verification+--------------------------------------------+llamafile does **not** verify TLS certificates when downloading models.+This matches upstream llama.cpp behaviour: ``common/http.h`` was introduced+in ggml-org/llama.cpp#16373 without verification, and PR #18828 (Jan 2026)+removed libcurl — which used the system CA store by default — replacing it+with a bare cpp-httplib client that calls neither ``set_ca_cert_path()``+nor any equivalent. No open issue or PR in the upstream repo tracks this+gap, so there is no expectation of a fix in the near term.Toggle with the ``ssl`` marker: ``-m ssl`` / ``-m "not ssl"``.``test_download_model_over_https`` needs network access (huggingface.co)@@ -18,7 +32,7 @@ import subprocessimport pytestimport requests-from utils.llamafile import LlamafileRunner+from utils.llamafile import LlamafileRunner, skip_if_bundled# Tiny model used for the network download test (~1 MiB)HF_TEST_REPO = "ggml-org/models"@@ -96,10 +110,17 @@ def fresh_cache_env(cache_dir):@pytest.mark.ssl[email protected]_executable@pytest.mark.serverclass TestHttpsServing:"""Serving over TLS with --ssl-cert-file/--ssl-key-file."""+ @pytest.fixture(autouse=True)+ def _require_bare_executable(self, executable, model):+ skip_if_bundled(executable)+ if model is None:+ pytest.skip("ssl serving tests need --model; pass a .gguf with the bare executable")+def test_server_serves_https(self, llamafile, server_port, timeouts, ssl_cert):"""Server comes up over HTTPS and a verifying client can connect."""cert, key = ssl_cert@@ -146,9 +167,14 @@ class TestHttpsServing:@pytest.mark.ssl[email protected]_executableclass TestHttpsDownload:"""Model downloads over HTTPS (the client side of TLS support)."""+ @pytest.fixture(autouse=True)+ def _require_bare_executable(self, executable):+ skip_if_bundled(executable)+@pytest.mark.onlinedef test_download_model_over_https(self, executable, server_port, timeouts, tmp_path@@ -175,7 +201,7 @@ class TestHttpsDownload:)try:assert LlamafileRunner.wait_for_server(- server_port, timeout=timeouts.server_ready+ server_port, timeout=timeouts.server_ready, proc=proc), "Server did not become ready (HTTPS download failed?)"if platform.system() != "Windows":@@ -190,48 +216,3 @@ class TestHttpsDownload:proc.terminate()proc.wait()- def test_download_rejects_untrusted_cert(- self, llamafile, executable, server_port, timeouts, ssl_cert, tmp_path- ):- """Downloading from a server with an untrusted (self-signed) cert fails.-- Runs fully offline: our own HTTPS server plays the untrusted host.- Its /health endpoint would happily serve a body, so if the client's- certificate verification were broken the download would 'succeed'- and leave a file in the cache; instead the TLS handshake must fail- and the cache must stay empty.- """- cert, key = ssl_cert- rogue_port = server_port + 1- server = llamafile.start_server(- port=rogue_port,- extra_args=["--ssl-cert-file", str(cert), "--ssl-key-file", str(key)],- )- try:- assert wait_for_https_server(- rogue_port, cert, timeout=timeouts.server_ready- ), "TLS server did not become ready"-- cache = tmp_path / "cache"- cache.mkdir()- args = base_exe_args(executable) + [- "--server",- "--port", str(server_port),- "--model-url", f"https://127.0.0.1:{rogue_port}/health",- ]- client = subprocess.run(- args,- capture_output=True,- text=True,- env=fresh_cache_env(cache),- timeout=timeouts.server_ready,- )- assert client.returncode != 0, "Download from untrusted cert host succeeded"- leftovers = [p for p in cache.rglob("*") if p.is_file()]- assert not leftovers, (- "Client fetched data from a server with an untrusted certificate: "- f"{leftovers}"- )- finally:- server.terminate()- server.wait()diff --git a/tests/integration/tests/test_tool_calling.py b/tests/integration/tests/test_tool_calling.pyindex ce4669d0..452c0f40 100644--- a/tests/integration/tests/test_tool_calling.py+++ b/tests/integration/tests/test_tool_calling.py@@ -56,7 +56,7 @@ class TestToolCalling:try:ready = LlamafileRunner.wait_for_server(- server_port, timeout=timeouts.server_ready+ server_port, timeout=timeouts.server_ready, proc=proc)assert ready@@ -86,12 +86,12 @@ class TestToolCalling:proc.wait()def test_tool_call_correct_function(self, llamafile, server_port, timeouts):- """Test that model calls the correct tool."""+ """Test that model calls the correct tool when two are available."""proc = llamafile.start_server(port=server_port)try:ready = LlamafileRunner.wait_for_server(- server_port, timeout=timeouts.server_ready+ server_port, timeout=timeouts.server_ready, proc=proc)assert ready@@ -107,10 +107,13 @@ class TestToolCalling:message = response["choices"][0]["message"]- if "tool_calls" in message and len(message["tool_calls"]) > 0:- tool_call = message["tool_calls"][0]- # Should call weather, not calculator- assert tool_call["function"]["name"] == "get_weather"+ assert "tool_calls" in message and len(message["tool_calls"]) > 0, (+ f"Expected a tool call. Got: {message}"+ )+ # Should call weather, not calculator+ assert message["tool_calls"][0]["function"]["name"] == "get_weather", (+ f"Expected get_weather, got: {message['tool_calls'][0]['function']['name']}"+ )finally:proc.terminate()@@ -122,7 +125,7 @@ class TestToolCalling:try:ready = LlamafileRunner.wait_for_server(- server_port, timeout=timeouts.server_ready+ server_port, timeout=timeouts.server_ready, proc=proc)assert readydiff --git a/tests/integration/tests/test_tui.py b/tests/integration/tests/test_tui.pyindex 1281e9a5..f1d7de6c 100644--- a/tests/integration/tests/test_tui.py+++ b/tests/integration/tests/test_tui.py@@ -2,6 +2,8 @@import pytest+from utils.prompts import ADD_2_2+@pytest.mark.tuiclass TestTUIBasic:@@ -21,13 +23,14 @@ class TestTUIBasic:def test_tui_math_question(self, llamafile, tmp_path, timeouts):"""Test TUI with a simple math question."""input_file = tmp_path / "input.txt"- input_file.write_text("What is 2+2?\n/exit\n")+ input_file.write_text(f"{ADD_2_2.prompt}\n/exit\n")result = llamafile.run_tui(str(input_file), timeout=timeouts.tui)assert result.returncode == 0- # The response should contain "4" somewhere- assert "4" in result.stdout+ assert ADD_2_2.check(result.stdout), (+ f"Expected {ADD_2_2.describe()} in output: {result.stdout}"+ )def test_tui_multi_turn(self, llamafile, tmp_path, timeouts):"""Test TUI with multiple turns of conversation."""diff --git a/tests/integration/utils/llamafile.py b/tests/integration/utils/llamafile.pyindex dc8fe2c2..d4508882 100644--- a/tests/integration/utils/llamafile.py+++ b/tests/integration/utils/llamafile.py@@ -11,11 +11,28 @@ import timefrom pathlib import Pathfrom typing import Any+import pytestimport requestslogger = logging.getLogger(__name__)+def skip_if_bundled(executable: str) -> None:+ """Skip the current test when *executable* is a pre-built .llamafile bundle.++ Call from an autouse fixture in any class marked ``bare_executable``::++ @pytest.fixture(autouse=True)+ def _require_bare_executable(self, executable):+ skip_if_bundled(executable)+ """+ if executable.endswith(".llamafile"):+ pytest.skip(+ "bare_executable tests run once against the bare llamafile binary; "+ "skip for bundled .llamafile models"+ )++def read_until_idle(fd, idle_timeout=1.0, max_timeout=60.0):"""Read from file descriptor until output stops (model finished generating).@@ -116,6 +133,7 @@ class LlamafileRunner:model: str | None = None,gpu: str | None = None,default_thinking: bool = False,+ no_warmup: bool = False,):"""Initialize the runner.@@ -127,11 +145,16 @@ class LlamafileRunner:one. Set to True for tests that exercise thinkingbehavior; False otherwise so small reasoningmodels don't derail unrelated assertions.+ no_warmup: Pass --no-warmup to every server/combined start. Use+ for tests that care about the server being up, not about+ warmup speed -- prevents cold-cache timeouts for large+ models without affecting inference correctness."""self.executable = os.path.abspath(executable)self.model = os.path.abspath(model) if model else Noneself.gpu = gpuself.default_thinking = default_thinking+ self.no_warmup = no_warmupif not os.path.exists(self.executable):raise FileNotFoundError(f"Executable not found: {executable}")@@ -311,6 +334,9 @@ class LlamafileRunner:if log_file:args.extend(["--log-file", log_file])+ if self.no_warmup:+ args.append("--no-warmup")+if extra_args:args.extend(extra_args)@@ -337,8 +363,13 @@ class LlamafileRunner:None, falls back to the runner's default_thinking.When False, adds --reasoning off.extra_args: Additional command-line arguments- log_file: If provided, adds --log-file flag. Caller should read- the file after terminating the process.+ log_file: If provided, adds --log-file flag and redirects+ stdout/stderr to DEVNULL. When a log file captures+ all diagnostics there is no need to pipe stdout/stderr,+ and leaving them as PIPE without a reader causes the+ OS pipe buffer (~64 KiB) to fill and the process to+ block indefinitely. Omit log_file when you need to+ interact with the TUI via proc.stdout / proc.stdin.Returns:Popen process handle (caller must terminate)@@ -352,15 +383,28 @@ class LlamafileRunner:if log_file:args.extend(["--log-file", log_file])+ if self.no_warmup:+ args.append("--no-warmup")+if extra_args:args.extend(extra_args)+ # When a log file is used, discard stdout/stderr via DEVNULL to avoid+ # blocking when the pipe buffer fills up (nobody is reading the pipe).+ # When no log file is set, use PIPE so callers can read TUI output.+ if log_file:+ stdout = subprocess.DEVNULL+ stderr = subprocess.DEVNULL+ else:+ stdout = subprocess.PIPE+ stderr = subprocess.PIPE+logger.info("Starting combined mode: %s", " ".join(args))return subprocess.Popen(args,stdin=subprocess.PIPE,- stdout=subprocess.PIPE,- stderr=subprocess.PIPE,+ stdout=stdout,+ stderr=stderr,text=True,)@@ -384,6 +428,7 @@ class LlamafileRunner:@staticmethoddef wait_for_server(port: int,+ proc: subprocess.Popen,host: str = "127.0.0.1",timeout: float = TIMEOUT_SERVER_READY,poll_interval: float = POLL_INTERVAL,@@ -392,18 +437,30 @@ class LlamafileRunner:Args:port: Server port+ proc: The server process. When it exits before the server+ becomes ready (e.g. a bare build launched without+ --model errors out at startup), give up immediately+ instead of polling a dead port for the full timeout.host: Server hosttimeout: Maximum time to wait in secondspoll_interval: Time between health checksReturns:- True if server is ready, False if timeout+ True if server is ready, False if it exited early or timed out"""url = f"http://{host}:{port}/health"start_time = time.time()logger.info("Waiting for server at %s (timeout=%.1fs)", url, timeout)while time.time() - start_time < timeout:+ if proc.poll() is not None:+ logger.warning(+ "Server process exited before becoming ready (rc=%s). "+ "If this is a bare llamafile build, did you pass "+ "--model / LLAMAFILE_MODEL?",+ proc.returncode,+ )+ return Falsetry:response = requests.get(url, timeout=2)if response.status_code == 200:diff --git a/tests/integration/utils/prompts.py b/tests/integration/utils/prompts.pynew file mode 100644index 00000000..2b3d8614--- /dev/null+++ b/tests/integration/utils/prompts.py@@ -0,0 +1,64 @@+"""Shared prompts and answer-checkers for liveness-style integration tests.++These prompts don't test model capability (math, instruction-following) --+they exist to confirm a mode is up and producing a coherent, on-topic+response. Keeping them here means updating a prompt or its accepted answers+happens in one place instead of drifting across test files.+"""++import re++NUMBER_WORDS = {+ 0: "zero", 1: "one", 2: "two", 3: "three", 4: "four",+ 5: "five", 6: "six", 7: "seven", 8: "eight", 9: "nine", 10: "ten",+}+++def contains_number(text: str, n: int) -> bool:+ """Check whether *text* contains *n* as a digit or spelled-out word.++ Matches on word boundaries so checking for 6 doesn't match "sixteen" and+ checking for 2 doesn't match "24".+ """+ forms = [str(n)]+ if n in NUMBER_WORDS:+ forms.append(NUMBER_WORDS[n])+ pattern = r"\b(?:" + "|".join(re.escape(f) for f in forms) + r")\b"+ return re.search(pattern, text, re.IGNORECASE) is not None+++class MathPrompt:+ """A simple addition fact used as a liveness/coherence check.++ Not a math-capability test: models may answer in digits or words, so+ the check accepts either.+ """++ def __init__(self, prompt: str, answer: int):+ self.prompt = prompt+ self.answer = answer++ def check(self, text: str) -> bool:+ return contains_number(text, self.answer)++ def describe(self) -> str:+ return f"{self.answer} (digit or word form)"+++# Canonical single-shot liveness prompt -- use this one everywhere a test+# just needs to confirm the model produced a sane, on-topic answer.+ADD_2_2 = MathPrompt("What is 2+2?", 4)++# Reserved for test_combined's multi-step test, which makes three sequential+# calls (server, TUI, server) in one test run. Using distinct facts there is+# deliberate: it's how that test catches cross-talk (e.g. the TUI reading the+# server's response, or a stale/cached answer) that reusing ADD_2_2 three+# times would mask, since a "4" would look correct regardless of which call+# it actually came from.+ADD_1_1 = MathPrompt("What is 1+1?", 2)+ADD_3_3 = MathPrompt("What is 3+3?", 6)++# Canonical greeting liveness prompt -- confirms a mode responds with some+# non-empty, coherent output. The exact wording doesn't matter (these tests+# only check the response is non-empty), so one wording is reused everywhere.+GREETING_PROMPT = "Say hello in one word."diff --git a/tests/run_integration_tests.sh b/tests/run_integration_tests.shnew file mode 100755index 00000000..8ff4601f--- /dev/null+++ b/tests/run_integration_tests.sh@@ -0,0 +1,59 @@+#!/usr/bin/env bash++# Each entry: "<llamafile-name>;<-m filter expression>"+# Leave the filter empty to run without -m.+TESTS=(+ "Apertus-8B-Instruct-2509.llamafile;not thinking and not multimodal"+ "Bonsai-1.7B.llamafile;not thinking and not multimodal"+ "Bonsai-4B.llamafile;not thinking and not multimodal and not tool_calling" # upstream llama.cpp peg-native parser fails to convert <tool_call> output to tool_calls for this model+ "Bonsai-8B.llamafile;not thinking and not multimodal"+ "LFM2-24B-A2B-Q5_K_M.llamafile;not thinking and not multimodal"+ "Ministral-3-3B-Instruct-2512-BF16.llamafile;not thinking"+ "Ministral-3-3B-Instruct-2512-Q4_K_M.llamafile;not thinking"+ "Qwen3.5-0.8B-Q8_0.llamafile;not thinking"+ "Qwen3.5-2B-Q8_0.llamafile;not thinking"+ "Qwen3.5-4B-Q5_K_S.llamafile;not thinking"+ "Qwen3.5-9B-Q5_K_S.llamafile;"+ "Qwen3.6-27B-Q4_K_M.llamafile;not cpu"+ "gemma-4-E2B-it-Q5_K_M.llamafile;"+ "gemma-4-E4B-it-Q5_K_M.llamafile;"+ "gemma-4-26B-A4B-it-MXFP4_MOE.llamafile;not cpu"+ "gemma-4-31B-it-Q5_K_M.llamafile;not cpu"+ # gpt-oss models do not really disable thinking, they just put it inline so some tests fail+ # they also occasionally fail the temperature=0 determinism check: their long forced+ # reasoning trace + MXFP4 quantization give a threaded matmul rounding difference more+ # opportunities to flip a near-tied token over a long generation+ "gpt-oss-20b-Q5_K_S.llamafile;not cpu and not multimodal and not thinking and not determinism"+ "gpt-oss-20b-mxfp4.llamafile;not cpu and not multimodal and not thinking and not determinism"+ "llava-v1.6-mistral-7b-Q4_K_M.llamafile;not thinking and not tool_calling"+ "llava-v1.6-mistral-7b-Q8_0.llamafile;not thinking and not tool_calling"+)++LLAMAFILE_EXE=~/llamafile/o/llamafile/llamafile+RUNNER=~/llamafile/tests/integration/run_tests.sh+MODELS_DIR=~/llamafiles++# Model used for the ssl serving tests (any small .gguf works; content is irrelevant).+# Override via SSL_TEST_MODEL env var if you have a different one available.+SSL_TEST_MODEL="${SSL_TEST_MODEL:-${MODELS_DIR}/Qwen3.5-0.8B-Q8_0.gguf}"++for entry in "${TESTS[@]}"; do+ model="${entry%%;*}"+ filter="${entry#*;}"++ echo "=== Testing ${model} ==="+ if [ -n "${filter}" ]; then+ "${RUNNER}" --executable "${MODELS_DIR}/${model}" -m "${filter}"+ else+ "${RUNNER}" --executable "${MODELS_DIR}/${model}"+ fi+done++echo "=== Running help tests (on bare executable: ${LLAMAFILE_EXE} ) ==="+"${RUNNER}" --executable "${LLAMAFILE_EXE}" -m help++echo "=== Running ssl tests (bare executable, network download) ==="+${RUNNER} --executable "${LLAMAFILE_EXE}" -m "ssl and online"++echo "=== Running ssl tests (on bare executable: ${LLAMAFILE_EXE}, model: ${SSL_TEST_MODEL}) ==="+"${RUNNER}" --executable "${LLAMAFILE_EXE}" --model "${SSL_TEST_MODEL}" -m "ssl and not online"diff --git a/tests/sandbox_test.c b/tests/sandbox_test.cnew file mode 100644index 00000000..fec0f2c2--- /dev/null+++ b/tests/sandbox_test.c@@ -0,0 +1,275 @@+// -*- mode:c;indent-tabs-mode:nil;c-basic-offset:4;coding:utf-8 -*-+// vi: set et ft=c ts=4 sts=4 sw=4 fenc=utf-8 :vi+//+// Copyright 2026 Mozilla.ai+//+// Licensed under the Apache License, Version 2.0 (the "License");+// you may not use this file except in compliance with the License.+// You may obtain a copy of the License at+//+// http://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing, software+// distributed under the License is distributed on an "AS IS" BASIS,+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+// See the License for the specific language governing permissions and+// limitations under the License.++//+// Verifies pledge()/SECCOMP sandboxing (llamafile/sandbox.c, issue #930).+//+// Each case forks a child that installs one of the promise sets llamafile+// actually uses, then checks from inside the sandbox that:+// - what must keep working still works (read-only opens under rpath,+// accept() under anet, thread creation under stdio), and+// - what must be blocked fails with EPERM instead of killing the+// process (PLEDGE_PENALTY_RETURN_EPERM), exactly what the server+// and CLI rely on.+//+// pledge() can only be enforced on Linux (SECCOMP) and OpenBSD. On other+// platforms this test reports SKIP and exits 0.+//++#include "llamafile/llamafile.h"++#include <arpa/inet.h>+#include <cosmo.h>+#include <errno.h>+#include <fcntl.h>+#include <libc/calls/calls.h> // unveil+#include <libc/dce.h> // IsLinux+#include <limits.h>+#include <netinet/in.h>+#include <pthread.h>+#include <stdio.h>+#include <stdlib.h>+#include <string.h>+#include <sys/socket.h>+#include <sys/wait.h>+#include <unistd.h>++// The policy wrapper llamafile_sandbox() consults llamafile_has_gpu(),+// which lives in llamafile.c and drags in the GPU loader stack. This+// test only exercises the enforcement primitives, so a stub keeps the+// link small.+bool llamafile_has_gpu(void) {+ return false;+}++#define CHECK(ok, what) \+ do { \+ if (!(ok)) { \+ fprintf(stderr, "FAIL(%s:%d): %s (errno=%s)\n", __FILE__, \+ __LINE__, what, strerror(errno)); \+ _exit(1); \+ } \+ } while (0)++static void *thread_fn(void *arg) {+ return arg;+}++// what the --cli / --chat modes rely on+static void child_rpath_promises(void) {+ CHECK(llamafile_sandbox_apply("stdio rpath") == LLAMAFILE_SANDBOX_ACTIVE,+ "apply stdio rpath");++ // reads must keep working: model weights are opened after pledge+ int fd = open("/dev/null", O_RDONLY);+ CHECK(fd != -1, "open O_RDONLY allowed under rpath");+ close(fd);++ // threads must keep working: ggml spawns its compute pool after pledge+ pthread_t th;+ CHECK(!pthread_create(&th, 0, thread_fn, 0), "pthread_create under stdio");+ pthread_join(th, 0);++ // no filesystem writes+ errno = 0;+ CHECK(open("sandbox_test.tmp", O_WRONLY | O_CREAT | O_TRUNC, 0644) == -1,+ "open O_WRONLY|O_CREAT blocked");+ CHECK(errno == EPERM, "write open fails with EPERM, not a kill");++ // no network at all+ errno = 0;+ CHECK(socket(AF_INET, SOCK_STREAM, 0) == -1, "socket() blocked");+ CHECK(errno == EPERM, "socket() fails with EPERM");++ _exit(0);+}++// without rpath even read-only opens must be denied (embedded-weights+// server: filesystem access disabled entirely)+static void child_stdio_only(void) {+ CHECK(llamafile_sandbox_apply("stdio") == LLAMAFILE_SANDBOX_ACTIVE,+ "apply stdio");+ errno = 0;+ CHECK(open("/dev/null", O_RDONLY) == -1, "open O_RDONLY blocked");+ CHECK(errno == EPERM, "read open fails with EPERM");+ _exit(0);+}++// what --server relies on: accept incoming connections, never initiate+static void child_anet_promises(void) {+ // pre-pledge: a second socket to attempt the outbound connect() with+ int censored = socket(AF_INET, SOCK_STREAM, 0);+ CHECK(censored != -1, "pre-pledge socket");++ CHECK(llamafile_sandbox_apply("stdio anet rpath") == LLAMAFILE_SANDBOX_ACTIVE,+ "apply stdio anet rpath");++ // the listening side of the HTTP server must keep working+ int server = socket(AF_INET, SOCK_STREAM, 0);+ CHECK(server != -1, "socket() allowed under anet");++ struct sockaddr_in addr;+ memset(&addr, 0, sizeof(addr));+ addr.sin_family = AF_INET;+ addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);+ addr.sin_port = 0; // ephemeral+ CHECK(!bind(server, (struct sockaddr *)&addr, sizeof(addr)),+ "bind() allowed under anet");+ CHECK(!listen(server, 1), "listen() allowed under anet");++ socklen_t addrlen = sizeof(addr);+ CHECK(!getsockname(server, (struct sockaddr *)&addr, &addrlen),+ "getsockname()");++ // accept() is permitted: with no pending client it must report+ // EAGAIN (not EPERM, which would mean the filter rejected it)+ int flags = fcntl(server, F_GETFL);+ CHECK(flags != -1 && fcntl(server, F_SETFL, flags | O_NONBLOCK) != -1,+ "set O_NONBLOCK");+ errno = 0;+ CHECK(accept(server, 0, 0) == -1, "accept with no client");+ CHECK(errno == EAGAIN || errno == EWOULDBLOCK,+ "accept() allowed by filter (EAGAIN, not EPERM)");++ // outbound connections must be denied, even to ourselves+ errno = 0;+ CHECK(connect(censored, (struct sockaddr *)&addr, sizeof(addr)) == -1,+ "connect() blocked under anet");+ CHECK(errno == EPERM, "connect() fails with EPERM");++ _exit(0);+}++// unveil() confinement: reads under an unveiled directory succeed while+// the rest of the filesystem is denied. Landlock accepts rules on any+// filesystem but enforces on only some (see llamafile_sandbox_server's+// governability probe), so if a read inside the unveiled dir is itself+// denied we treat the filesystem as ungovernable and skip -- matching the+// production fallback -- rather than failing.+static void child_unveil(void) {+ // Two files we own: one inside a directory we will unveil (must stay+ // readable), one outside it (a self-made canary that must become denied+ // -- a sibling under /tmp, never a system file).+ char dir[] = "/tmp/lf_uv_XXXXXX";+ CHECK(mkdtemp(dir), "mkdtemp");+ char inside[PATH_MAX];+ snprintf(inside, sizeof(inside), "%s/weights.bin", dir);+ int fd = open(inside, O_WRONLY | O_CREAT | O_TRUNC, 0644);+ CHECK(fd != -1, "create file inside temp dir");+ close(fd);++ char canary[] = "/tmp/lf_canary_XXXXXX";+ fd = mkstemp(canary); // outside `dir`, so confinement must hide it+ CHECK(fd != -1, "mkstemp canary");+ close(fd);++ unveil(dir, "r");+ unveil(0, 0); // lock++ int in = open(inside, O_RDONLY);+ if (in == -1) {+ // filesystem accepts unveil() but denies everything: over-denies+ fprintf(stderr, "sandbox_test: unveil: SKIP (filesystem not "+ "Landlock-governable)\n");+ _exit(0);+ }+ close(in);++ // If the canary outside the unveiled dir is still readable, unveil()+ // installed nothing (kernel without Landlock, or cosmo's silent no-op):+ // there is no confinement to assert. Same check the production probe+ // makes, on a file we created rather than a system file.+ errno = 0;+ int c = open(canary, O_RDONLY);+ if (c != -1) {+ close(c);+ fprintf(stderr, "sandbox_test: unveil: SKIP (Landlock unavailable; "+ "confinement not installed)\n");+ _exit(0);+ }+ CHECK(errno == EACCES || errno == EPERM,+ "unveil denial is EACCES/EPERM, not a kill");+ _exit(0);+}++static int run_case(const char *name, void (*fn)(void)) {+ pid_t pid = fork();+ if (!pid)+ fn(); // never returns+ int ws;+ if (waitpid(pid, &ws, 0) != pid || !WIFEXITED(ws) || WEXITSTATUS(ws)) {+ fprintf(stderr, "sandbox_test: %s: FAILED%s\n", name,+ WIFSIGNALED(ws) ? " (killed by signal)" : "");+ return 1;+ }+ printf("sandbox_test: %s: OK\n", name);+ return 0;+}++// Pure promise-string derivation: keep the server policy unit-testable+// rather than buried in the vendored server.cpp patch. Covers the+// accept-only default, the wpath/cpath (has_rw) and inet (needs_outbound)+// relaxations, and OpenBSD's lack of "anet".+static int test_promises(void) {+ struct {+ bool openbsd, has_rw, outbound;+ const char *want;+ } cases[] = {+ {false, false, false, "stdio anet rpath"},+ {false, true, false, "stdio anet rpath wpath cpath"},+ {false, false, true, "stdio inet rpath"}, // --rpc etc.+ {false, true, true, "stdio inet rpath wpath cpath"},+ {true, false, false, "stdio inet rpath"}, // OpenBSD: no anet+ {true, true, false, "stdio inet rpath wpath cpath"},+ };+ int rc = 0;+ for (size_t i = 0; i < sizeof(cases) / sizeof(*cases); ++i) {+ char got[64];+ llamafile_sandbox_server_promises(got, sizeof(got), cases[i].openbsd,+ cases[i].has_rw, cases[i].outbound);+ if (strcmp(got, cases[i].want)) {+ fprintf(stderr, "sandbox_test: promises: FAILED want=\"%s\" got=\"%s\"\n",+ cases[i].want, got);+ rc = 1;+ }+ }+ if (!rc)+ printf("sandbox_test: server promise derivation: OK\n");+ return rc;+}++int main(void) {+ int rc = 0;++ // Pure logic, runs on every platform.+ rc |= test_promises();++ // The enforcement cases assert Linux SECCOMP semantics: the "anet"+ // promise is a cosmo Linux extension OpenBSD's pledge(2) rejects, and+ // the EPERM assertions assume PLEDGE_PENALTY_RETURN_EPERM (Linux only;+ // OpenBSD kills violators). So gate them to Linux.+ if (!IsLinux() || !llamafile_sandbox_supported()) {+ printf("sandbox_test: SKIP enforcement: needs Linux with SECCOMP "+ "(anet/EPERM semantics are Linux-specific)\n");+ return rc;+ }+ rc |= run_case("stdio+rpath (cli promises)", child_rpath_promises);+ rc |= run_case("stdio only (embedded weights)", child_stdio_only);+ rc |= run_case("stdio+anet+rpath (server promises)", child_anet_promises);+ rc |= run_case("unveil confinement", child_unveil);+ return rc;+}