Skip to content

Distribute and run LLMs with a single file.

36.1 KB
Raw Blame

llama.cpp Patches for Llamafile

This directory contains patches that adapt llama.cpp for use with Llamafile and Cosmopolitan libc. These patches enable llama.cpp to run as a portable, single-file executable across Windows, macOS, Linux, and BSD without installation.

Directory Structure

llama.cpp.patches/
├── README.md              # This file
├── apply-patches.sh       # Script to apply all patches to llama.cpp submodule
├── fetch-ui-assets.sh     # Downloads + validates the prebuilt web UI (see Server Integration)
├── ui-embed.sh            # Renders upstream's ui.{cpp,h}.in templates (see Server Integration)
├── renames.sh             # Script for file renames/moves (if any)
├── llamafile-files/       # Additional files to copy into llama.cpp
│   ├── BUILD.mk           # Makefile for building llama.cpp with cosmocc
│   ├── README.llamafile   # License and modification notes
│   └── common/
│       └── license.cpp    # Llama.cpp's license file (cmake creates this at build time)
└── patches/               # Patch files for upstream sources

Applying Patches

To apply all patches to the llama.cpp submodule:

./llama.cpp.patches/apply-patches.sh

To reset the submodule to its clean state:

cd llama.cpp && git reset --hard && git clean -fdx

Patch Index

Windows/macOS ABI Compatibility (GGML_CALL)

GPU backends (CUDA, Vulkan, Metal) are compiled as shared libraries (.dll/.so/.dylib) using native compilers, but the llamafile host binary is built with Cosmopolitan libc which uses System V AMD64 ABI everywhere — including on Windows. When the host calls function pointers inside backend interface structs, the calling convention must match.

The GGML_CALL macro (defined as __attribute__((__ms_abi__)) when GGML_MULTIPLATFORM is set) annotates all function pointers in the backend interface structs and their implementations, so the correct calling convention is used on every platform.

Patch Description
ggml_include_ggml-backend.h.patch Defines the GGML_CALL macro; adds it to the five get_proc_address return typedefs (ggml_backend_split_buffer_type_t, ggml_backend_set_n_threads_t, ggml_backend_dev_get_extra_bufts_t, ggml_backend_set_abort_callback_t, ggml_backend_get_features_t)
ggml_include_ggml-cpu.h.patch Adds GGML_CALL to declarations of ggml_backend_cpu_set_n_threads and ggml_backend_cpu_set_abort_callback (returned via get_proc_address)
ggml_include_ggml-cuda.h.patch Adds GGML_CALL to the declaration of ggml_backend_cuda_register_host_buffer (upstream removed the row-split multi-GPU buffer in b10052, so the former ggml_backend_cuda_split_buffer_type annotation is gone)
ggml_src_ggml-backend-impl.h.patch Adds GGML_CALL to all 49+ function pointers across the five interface structs (ggml_backend_buffer_type_i, ggml_backend_buffer_i, ggml_backend_i, ggml_backend_device_i, ggml_backend_reg_i), including the optional alloc_buffer_n/get_alloc_size_n buffer-type members added in v0.8.0; also adds free_struct callback (see Cross-Module Memory below)
ggml_src_ggml-backend.cpp.patch Adds GGML_CALL to CPU buffer, buffer type, and multi-buffer callback implementations; also adds free_struct support (see Cross-Module Memory below), and carries upstream PR #25584 (see Upstream fixes carried ahead of a release)
ggml_src_ggml-cpu_ggml-cpu.cpp.patch Adds GGML_CALL to all CPU backend, device, and registry callback implementations, plus get_proc_address-returned functions (set_n_threads, set_abort_callback, get_extra_buffers_type, get_features)
ggml_src_ggml-cpu_amx_amx.cpp.patch Adds GGML_CALL to all AMX buffer and buffer type callback implementations (10 functions)
ggml_src_ggml-cpu_repack.cpp.patch Adds GGML_CALL to CPU repack buffer and buffer type callback implementations (5 functions)
ggml_src_ggml-cuda_ggml-cuda.cu.patch Adds GGML_CALL to all CUDA backend callback implementations (60+ functions, including the pinned host buffer type's get_max_size, new in v0.8.0); also adds free_struct and TinyBLAS BF16 guard (see below)
ggml_src_ggml-metal_ggml-metal.cpp.patch Adds GGML_CALL to all Metal backend callback implementations (62 functions); also adds free_struct (see below)
ggml_src_ggml-vulkan_ggml-vulkan.cpp.patch Adds GGML_CALL to all Vulkan backend callback implementations; also adds free_struct (see below)
ggml_src_ggml-vulkan_ggml-vulkan-common.h.patch Adds GGML_CALL to the backend callback declarations. b11100 split the Vulkan backend into several translation units and moved the interface structs into ggml-vulkan-buffers.cpp, so the callbacks it holds became non-static and are declared here; declaration and definition must agree on the convention
ggml_src_ggml-vulkan_ggml-vulkan-buffers.cpp.patch Adds the free_struct entry to ggml_backend_vk_buffer_interface, which b11100 moved into this new file
ggml_src_ggml-metal_ggml-metal-fusion.h.patch / .cpp.patch Adds GGML_CALL to the add_alloc_dep function pointer these pass through (see Graph-optimize allocation dependencies below), and includes ggml-backend.h for the macro
ggml_src_ggml-backend-meta.cpp.patch Adds GGML_CALL to all meta-device, meta-buffer-type (including its alloc_buffer_n, new in v0.8.0), meta-buffer, and meta-backend callback implementations (the meta backend aggregates several simple backends behind one interface, so its callbacks are reached through the same function-pointer structs)

Cross-Module Memory Management

When GPU backends (CUDA, Vulkan, Metal) are loaded as dynamic libraries, memory allocated by the DSO must be freed by the DSO's allocator, not the main executable's.

Patch Description
ggml_src_ggml-backend-impl.h.patch Adds free_struct callback to ggml_backend_buffer_i interface for cross-module buffer cleanup
ggml_src_ggml-backend.cpp.patch Implements free_struct callback support in ggml_backend_buffer_free() — calls DSO's free_struct instead of delete when set
ggml_src_ggml-cuda_ggml-cuda.cu.patch Adds free_struct implementation for CUDA buffers (regular and host; upstream removed the split buffer in b10052); sets it on fallback CPU buffers allocated within the DSO
ggml_src_ggml-metal_ggml-metal.cpp.patch Adds free_struct implementation for Metal shared and private buffers
ggml_src_ggml-vulkan_ggml-vulkan.cpp.patch Adds free_struct implementation for Vulkan buffers and host buffer fallback path
ggml_src_ggml-vulkan_ggml-vulkan-buffers.cpp.patch Wires that free_struct into ggml_backend_vk_buffer_interface, which lives in this file since b11100

Graph-optimize allocation dependencies

b11100 gave graph_optimize a ggml_backend_graph_optimize_params *, through which a backend tells the scheduler to keep a tensor alive until a later node (params->add_alloc_dep(...)). The callback is the host's, and the CUDA, Vulkan and Metal backends call it from inside their DSO, so it crosses the ABI boundary in the opposite direction to the backend interface structs and needs GGML_CALL just the same.

Patch Description
ggml_src_ggml-backend-impl.h.patch Adds GGML_CALL to the add_alloc_dep member of ggml_backend_graph_optimize_params
ggml_src_ggml-backend.cpp.patch Replaces upstream's captureless lambda for add_alloc_dep with a named ggml_backend_sched_add_alloc_dep() — a lambda cannot carry the attribute
ggml_src_ggml-metal_ggml-metal-fusion.h.patch / .cpp.patch Annotates the same pointer where ggml_metal_fusion_add_alloc_deps() passes it on, and includes ggml-backend.h so the macro is in scope

Generated version headers

Since b11100 ggml/src/ggml.c includes ggml-version.h and src/llama.cpp includes llama-version.h; both are produced by CMake's configure_file() from the .in templates next to them, and hold the GGML_VERSION/GGML_COMMIT and LLAMA_VERSION/LLAMA_COMMIT macros that used to arrive as -D flags. There is no CMake step in the cosmocc build, so apply-patches.sh generates both headers (with the same values build/config.mk computes) straight into the source tree rather than into o/. That single location covers every consumer: the make build, llamafile/*.sh and *.bat, which compile ggml.c with -I ggml/src, and llamafile/metal.c, which extracts ggml-version.h alongside ggml.c for the runtime Metal build.

They are generated, not patched: keep them out of llamafile-files/ (delete both before running generate-patches, which would otherwise pick them up as new files and freeze one bump's commit hash into the repo).

Cosmopolitan Libc Compatibility

These patches address compatibility issues when building with Cosmopolitan libc (cosmocc).

Patch Description
common_arg.cpp.patch Adds COSMOCC platform detection for PATH_MAX (includes linux/limits.h)
common_common.cpp.patch Adds platform-aware cache directory detection for Cosmopolitan (fs_get_cache_directory: checks LOCALAPPDATA, XDG_CACHE_HOME, falls back to ~/.cache/) and the same for the config directory (fs_get_config_directory: APPDATA, XDG_CONFIG_HOME, ~/.config/). cosmocc defines none of __linux__/__APPLE__/_WIN32; until v0.8.0 that hit #error Unknown architecture, and since then it falls to upstream's generic XDG branch, which compiles but would move a Windows host's cache out of %LOCALAPPDATA% and orphan models already downloaded there, so the COSMOCC branch stays. (Until v0.8.0 it also added the mmproj file size to every device's fit target. Upstream's server reserves the projector itself since #21489, per device, and llamafile's --cli/--chat load models without fit, so that hunk only double-counted: on a 24 GB GPU, Gemma-4-31B + mmproj lost 3 offloaded layers. Dropped.)
common_download.cpp.patch Adds COSMOCC platform detection for PATH_MAX
common_common.h.patch Keeps the server's default --port at 8080, llamafile's documented default (help text, docs, integration tests, users' scripts). Upstream moved it to 9931 in v0.8.0 (#30159); this is a deliberate product choice, not a portability fix
vendor_sheredom_subprocess.h.patch Under __COSMOPOLITAN__, rewrites subprocess_error_from_errno()'s switch over E* values as an if/else chain. Cosmopolitan resolves errno at runtime (one binary, every host OS), so those are not constant expressions and cannot be case labels — upstream's version fails with case value is not a constant expression. Same mapping, no behavior change. Needed since b10441, which added this function; the header only enters the build when LLAMA_SUBPROCESS is set (see Server Integration). Also decodes the child's exit status on Windows, where Cosmopolitan's waitpid() returns the NT exit code as the wait status: Cosmopolitan children exit with code << 8, native ones (cmd.exe) with the plain code, so upstream's WIFEXITED/WEXITSTATUS turned exit 3 into 1 and exit 128 into 0. A status with its low byte set is now taken as a native exit code (#1066)
tools_server_server-tools.cpp.patch Under __COSMOPOLITAN__, picks the shell for the built-in exec_shell_command and get_info tools at run time with IsWindows(): cmd.exe /c on Windows, sh -c / uname -a elsewhere (and always inside a --tools-runtime isolate), the same choice upstream makes at compile time with _WIN32, which the cosmocc build never defines. The name is spelled cmd.exe because Cosmopolitan's posix_spawnp() does not try the .exe suffix when it searches PATH. Without it both tools fail on Windows with "failed to spawn process" (#1066)

Threading and Signal Handling

Cosmopolitan libc has specific behaviors with condition variables and signals that require workarounds.

Patch Description
common_log.cpp.patch Adds #include <csignal>; blocks SIGINT/SIGTERM on logger thread via pthread_sigmask to prevent EINTR exceptions; replaces the untimed cv.wait() calls with wait_for(30s) loops (the worker's cv_new wait, plus the queue-full cv_full waits in add() and in add_json(), new upstream in b11100) to work around XNU futex timeout bug (~72 minute expiry). The add*() waits run on the logging caller's thread, so the signal mask does not cover them
tools_server_server-models.cpp.patch Adds #include <csignal>; blocks signals via pthread_sigmask on server_monitor's thread (b11100 replaced the per-model stopping_thread this used to cover with one monitor thread watching every child); replaces untimed cv.wait() with wait_for(30s) loops on every model-lifecycle wait (the reload-drain wait, unload_all, the is_reloading guard in load, and the generic wait() predicate helper) to work around the XNU futex timeout bug
tools_server_server-models-sched.cpp.patch Same wait_for(30s) treatment for the two waits in server_models_sched::make_room() (the memory-measurement wait and the LRU-unload wait). v0.8.0 moved the LRU eviction out of server_models::unload_lru() into this new scheduler
tools_server_server-queue.cpp.patch Adds missing includes (<cerrno>, <system_error>, <csignal>); blocks SIGINT/SIGTERM on queue thread (the yield_to_queue worker thread is spawned after this, so it inherits the mask); replaces wait() with wait_for(30s) loops in five locations (wait_until_no_sleep, main loop, recv, plus worker_loop and yield_to_queue, both new upstream in b10441); runs yield_to_queue() inline when a GPU backend is loaded (see "GPU decode must stay on the main thread" below)
tools_server_server-stream.cpp.patch Adds #include <csignal>; blocks signals on the stream-session GC thread via pthread_sigmask. Without it, Ctrl+C on llama-server aborts with "libc++abi: terminating due to uncaught exception ... condition_variable timed_wait failed: Interrupted system call" instead of shutting down: SIGINT is delivered to that thread while it sits in gc_wake_cv.wait_for(60s), pthread_cond_timedwait() returns EINTR, and libcxx rethrows it as std::system_error that nothing catches. This thread was the last one in the server with a condition-variable wait and no signal mask (the log worker, task queue, model-stopping thread and httplib pool were already covered). Pre-existing, not introduced by b10441; combined TUI mode is unaffected
vendor_cpp-httplib_httplib.cpp.patch Fixes httplib thread pool with wait_for() instead of wait() for XNU futex compatibility; passes the full sockaddr_un for a pathname Unix socket, since Cosmopolitan's bind()/connect() reject addresses shorter than struct sockaddr and a --host socket path under 14 bytes failed with EINVAL; also see HTTPS / TLS Support below

HTTPS / TLS Support

Upstream llama.cpp gets TLS from cpp-httplib's OpenSSL backend (CPPHTTPLIB_OPENSSL_SUPPORT, satisfied by system OpenSSL or vendored BoringSSL/LibreSSL at cmake time). None of those is available in the cosmocc make build, so llamafile instead enables cpp-httplib's Mbed TLS backend (CPPHTTPLIB_MBEDTLS_SUPPORT) against the mbedtls fork already vendored in third_party/mbedtls — the same TLS stack llamafile <= 0.9.3 used. third_party/mbedtls/include/ maps the canonical <mbedtls/*.h> include paths onto the fork's headers, and BUILD.mk sets the macro on every object that can reach httplib.h (the macro changes httplib class layouts, so all TUs must agree) and links mbedtls.a into llama-server. This enables HTTPS model downloads (-hf, --model-url), https clients in server-models, and TLS serving via --ssl-cert-file/--ssl-key-file.

Keep-in-sync on a bump: third_party/mbedtls/include/mbedtls/ holds one forwarding header per <mbedtls/*.h> that the vendored httplib.h includes. When upstream refreshes cpp-httplib and it reaches for a new one, the build fails with fatal error: 'mbedtls/<x>.h' file not found — add the matching one-line forwarder (b10441 needed version.h, which httplib now uses to gate its 2.x/3.x/4.x API branches; the vendored fork is 2.26, so MBEDTLS_VERSION_MAJOR is 2 and the 2.x path stays selected).

Patch Description
common_http.h.patch #ifndef CPPHTTPLIB_OPENSSL_SUPPORT -> #ifndef CPPHTTPLIB_SSL_ENABLED for the "HTTPS is not supported" guard, so any cpp-httplib TLS backend counts (candidate for upstreaming)
tools_server_server-http.cpp.patch Same macro fix for the httplib::SSLServer (--ssl-cert-file/--ssl-key-file) guard (candidate for upstreaming)
tools_server_server-models.cpp.patch Same macro fix for the direct httplib::SSLClient construction in server_http_proxy (candidate for upstreaming)
vendor_cpp-httplib_httplib.cpp.patch Under __COSMOPOLITAN__: appends /zip/third_party/mbedtls/sslroot to system_ca_dirs() as the trust-store fallback (essential on Windows hosts, where the _WIN32 cert-store branches are not compiled into an APE), and __static_yoink("ssl_root_support") so the Mozilla root PEMs bundled by third_party/mbedtls/BUILD.mk are pulled into the executable's zip

TinyBLAS Integration

Llamafile uses TinyBLAS as a lightweight replacement for cuBLAS, enabling GPU support without CUDA SDK dependencies.

Patch Description
ggml_src_ggml-cuda_vendors_cuda.h.patch Includes TinyBLAS headers (tinyblas.h, tinyblas-compat.h) instead of cublas_v2.h when GGML_USE_TINYBLAS is defined; guards backward-compat CUBLAS_* defines so they don't conflict with TinyBLAS's own definitions
ggml_src_ggml-cuda_common.cuh.patch Disables BF16 MMA when using TinyBLAS (TinyBLAS would incorrectly interpret BF16 as FP16)
ggml_src_ggml-cuda_ggml-cuda.cu.patch Disables BF16 in ggml_cuda_op_mul_mat_cublas when using TinyBLAS

Optional IQ-Quant Exclusion (CUDA)

The IQ ("importance") quantization formats (IQ1_S, IQ2_XXS/XS/S, IQ3_S/XXS, IQ4_NL/XS) pull in a large amount of CUDA template instantiation that inflates compile time and binary size. These patches gate the IQ code paths behind #ifndef GGML_CUDA_NO_IQ_QUANTS — the MMQ/MMVQ matmul kernels, the f32 → IQ4_NL copy, and the IQ dequant cases in ggml_get_to_bf16_cuda/ggml_get_to_fp16_cuda — so a build can compile them out by defining GGML_CUDA_NO_IQ_QUANTS. (ggml_get_to_fp32_cuda's IQ cases are not guarded, so the float dequant-template instantiations still compile — a minor size cost, harmless because ggml_backend_cuda_device_supports_op gates the same IQ ops, so those tensors fall back to CPU in a minimized build regardless.) When the macro is undefined (the default), behavior is unchanged.

Patch Description
ggml_src_ggml-cuda_convert.cu.patch Guards IQ dequantization cases in ggml_get_to_bf16_cuda and ggml_get_to_fp16_cuda
ggml_src_ggml-cuda_cpy.cu.patch Guards the f32 → IQ4_NL copy helper and its dispatch case
ggml_src_ggml-cuda_mmq.cu.patch Guards IQ cases in ggml_cuda_mul_mat_q_switch_type and in the ggml_cuda_should_use_mmq support/heuristic switches
ggml_src_ggml-cuda_mmq.cuh.patch Guards the extern DECL_MMQ_CASE(...) declarations for IQ types
ggml_src_ggml-cuda_mmvq.cu.patch Guards IQ cases in get_vec_dot_q_cuda and get_vdr_mmvq

CPU Performance Optimizations (llamafile #975)

These patches restore llamafile's optimized CPU kernels (TinyBLAS matmul, AVX-512 flash-attention helpers) on top of upstream's CPU backend, and tune CPU-only defaults. The hooks call into symbols exported from llamafile/sgemm.cpp and are compiled only when GGML_USE_LLAMAFILE is defined.

Patch Description
ggml_src_ggml-cpu_ggml-cpu.c.patch Routes MoE matmul (ggml_compute_forward_mul_mat_id) through llamafile_mixmul / llamafile_mixmul_iqk, mirroring the dense-matmul llamafile_sgemm hook; reserves work-buffer space for the MoE kernel in ggml_graph_plan via llamafile_mixmul_needs (in the GGML_OP_MUL_MAT_ID case, after upstream's tiled workspace; on the v0.8.0 bump the hunk's context drifted and patch silently placed it in GGML_OP_OUT_PROD, so check its position on each bump). v0.8.0 replaced the IQ panel path (iqp.cpp) with a tiled k-quant matmul (ggml-cpu/tiled/) that runs before these hooks, but it is compiled only under __AVX__/__AVX2__/__AVX512VNNI__ and the cosmocc build compiles ggml-cpu at baseline ISA, so in llamafile it always declines and the TinyBLAS hooks keep handling those matmuls
ggml_src_ggml-cpu_ops.cpp.patch Routes flash-attention inner loops through llamafile's AVX-512 helpers (llamafile_fa_vec_dot_f16, llamafile_fa_fp16_to_fp32_row, llamafile_fa_simd_gemm) in both the one-chunk and tiled FA paths; also accumulates VKQ in f32 on CPUs lacking native f16 FMA (avoiding costly f16↔f32 round-trips per KV step)
src_llama-context.cpp.patch Defaults -fa auto to off on CPU-only setups (no GPU devices), since the CPU flash-attention path is slower than the non-FA path on x86; users can still force -fa on for memory savings on long contexts

Llamafile File Handling

These patches integrate llamafile's file handling APIs for loading models from bundled zip archives and .llamafile containers.

Patch Description
src_llama-mmap.h.patch Adds has_premapped_content(), premapped_content(), and get_llamafile() methods to llama_file class
src_llama-mmap.cpp.patch Under COSMOCC, redirects file open/read/seek/tell/close through llamafile API (llamafile_open_gguf, llamafile_read, etc.); adds premapped content support to llama_mmap using llamafile reference counting (llamafile_ref/llamafile_unref); skips munmap for premapped content; on Windows makes unmap_fragment() a no-op, as upstream's _WIN32 branch does (cosmocc never defines _WIN32, so llamafile runs the POSIX branch everywhere, and Windows can only unmap a whole view: cosmo's munmap fails with ENOTSUP/EINVAL on a sub-range, which printed "munmap failed" warnings and left the view mapped at exit). Memory use then matches the official llama.cpp Windows build: the whole file stays mapped after GPU offload in both
ggml_src_gguf.cpp.patch Adds tell()/seek() to gguf_reader; under COSMOCC, adds gguf_llamafile_reader that reads via llamafile API; templatizes gguf_init_from_reader_impl so both readers work; redirects gguf_init_from_file through llamafile_open_gguf (supports /zip/ paths, .llamafile containers)

Server Integration

Patch Description
tools_server_server.cpp.patch Makes server_main() the real entry point (upstream's llama_server(argc, argv) becomes a thin wrapper around it) and adds on_ready/on_shutdown_available callbacks for combined TUI+server mode. v0.8.0 (#31113) split the server into server_run_router() / server_run_instance() / server_run(); the callbacks (and argv, for --rpc detection) now travel in upstream's server_run_opts, and on_ready, on_shutdown_available, and the sandbox all live in server_run_instance() (the router stays unsandboxed, as before). The sandbox is also skipped under --connect (new in v0.8.0), which spawns llama-connect after the lock and would otherwise fail to start; adds Metal/GPU backend trigger before common_init(); installs the sandbox (llamafile_sandbox_server(), issue #930) — the mechanism lives in llamafile/sandbox.c; the patch fills a llamafile_sandbox_spec with the on-disk paths the loader opens after the lock (model, mmproj, media dir, LoRA, draft model, control vectors, public path as read; slot-save and prompt cache as read-write), detects outbound-needing features (--rpc via argv/env, MCP proxy, server tools) to relax anet→inet, passes FLAG_confine_reads for opt-in unveil(), binds the listening sockets first (ctx_http.bind_socket(), so a --host /path.sock Unix socket exists before anet forbids creating one, issue #971), quiesces the log worker (common_log_pause/resume, so no thread escapes the per-thread filter) around the call, before the HTTP listener spawns and before model load, and skips it in combined/GPU modes; adds Cosmopolitan-specific standalone main() with cosmo_args, verbose flag handling, --unsecure consumption (llamafile_consume_flag), and GPU pre-initialization; handles LLAMAFILE_TUI exit to avoid Metal cleanup crashes
tools_server_server-http.h.patch / tools_server_server-http.cpp.patch Splits the bind out of server_http_context::start() into bind_socket(), which start() calls if it hasn't run yet, so the server can bind before installing the sandbox (issue #971). Since v0.8.0 start() binds every listener (one per --host) before spawning the HTTP thread pool, and bind_socket() does exactly that first part

GPU decode must stay on the main thread (tools_server_server-queue.cpp.patch). b10441 moved llama_decode() off the main thread: update_slots() now hands each batch to queue_tasks.yield_to_queue(), which runs it on a worker thread created in start_loop(). Under Cosmopolitan that is fatal for every GPU backend, because they are host ELFs loaded with cosmo_dlopen() and their libc reaches thread-local state through %fs:

thread fs_base gs_base *(fs+0x10) (glibc TCB self)
main (created by the host) 0x7ffff7e54740 0x7ffff7ff8000 valid, points to itself
cosmo-spawned worker 0x7fffe83ca880 same as fs 0

Cosmo gives its own threads cosmo TLS in both %fs and %gs; there is no glibc TCB. cosmo_dlopen's foreign_tramp does swap in a host TLS block (__foreign, set up once via cosmo_once), but only for pointers obtained through cosmo_dlsym — ggml's ggml_backend_i vtable holds raw DSO pointers that ggml core calls directly. So the first CUDA decode on the worker lands in host pthread_getspecific, which reads %fs:0x10 as the TCB self-pointer, gets 0, and segfaults at 0x328 (verified on an L40S: deterministic, any model size, dmesg shows segfault at 328 ... in libc.so.6).

The patch therefore runs the work inline on the calling thread whenever llamafile_has_gpu() — restoring exactly where decode ran before b10441. CPU-only runs keep upstream's worker. The cost is narrow: the yield wraps a single llama_decode(batch_view), and only SERVER_TASK_TYPE_METRICS is serviced during it, so the sole effect is that /metrics and /slots wait for the in-flight batch (~29 ms while generating; up to a few seconds for a full prompt batch). This will recur whenever upstream moves work onto a new thread.

Bitmap hashing needs vendor/hash (added to BUILD.mk, not a patch). b11100 made mtmd-helper.cpp derive a bitmap's ID from hash_sha256_hex() in the vendored vendor/hash, which upstream links into mtmd as vendor::hash. BUILD.mk builds hash.cpp and sha256/sha256.c into MTMD_OBJS (with -iquote llama.cpp/vendor/hash, since sha256.c reaches for "rotate-bits/rotate-bits.h"). Upstream's target also builds xxhash and sha1; nothing here calls either, and sha1 would have to be compiled as C++ despite its .c extension (its declarations sit in a namespace, and upstream forces LANGUAGE CXX on it), so neither is built. A future caller shows up as an undefined reference at link.

Subprocess support needs -DLLAMA_SUBPROCESS (set in BUILD.mk, not a patch). b10441 moved child-process spawning into common_subproc (common/subproc.*, wrapping the vendored sheredom/subprocess.h) and put it behind a build gate; before that server-models.cpp included subprocess.h unconditionally, so the cosmocc build got it for free. Upstream's CMake defaults the option ON except on iOS/Android/WASM. It gates three server features, all reachable from llamafile since args.cpp strips only llamafile's own flags and passes the rest to common_params_parse:

  • MCP stdio servers (--mcp-servers-config, --mcp-servers-json) — server-mcp.cpp spawns them and, unlike the other two, does not check common_subproc::is_supported(), so without the define they fail silently rather than reporting an error.
  • --tools — server-tools.cpp throws "subprocess is not enabled on this build", caught into a clean return 1.
  • Router mode — llama-server with no model; init_routes() throws the same message, and the server exits at startup.

The macro also switches subprocess_s between its real and dummy definition in subproc.h, so every object that can reach that header must be compiled with it. Enabling it pulls subprocess.h into the build, which needs the Cosmopolitan fix in the patch table above.

The web UI moved upstream from prebuilt tools/server/public/* assets to a Svelte/PWA project under tools/ui/, embedded into the binary at build time. cosmocc has no JS toolchain, so fetch-ui-assets.sh (run by apply-patches.sh / make setup) downloads the prebuilt site tarball dist.tar.gz (plus its .sha256) from the ggml-org/llama-ui Hugging Face bucket — picking the newest bNNNN tag <= our pinned build — verifies it, and extracts the whole static site into llama.cpp/tools/ui/dist/. The modern site is no longer four flat files (bundle.css/bundle.js/index.html/loading.html) but a hashed tree (_app/immutable/bundle.HASH.js, a service worker, manifest, icons, splash screens). To keep the embedded payload small, the script also builds a dist/_gzip/ mirror with every file gzip-compressed under its original name.

b11100 deleted upstream's standalone tools/ui/embed.cpp and replaced it with scripts/ui-assets.cmake, which provisions the assets and renders tools/ui/ui.{cpp,h}.in into ui.cpp/ui.h inside a CMake build. There is no CMake step in the cosmocc build, so ui-embed.sh does that rendering — a translation of that script's emit_files(), in the same spirit as BUILD.mk being a translation of llama.cpp's CMake build.

The important property is that it renders upstream's own templates, so the generated interface (llama_ui_find_asset / llama_ui_get_assets / llama_ui_use_gzip and LLAMA_UI_HAS_ASSETS — all that the unpatched server-http.cpp consumes) tracks upstream automatically and cannot drift. Only the substitutions live on our side: @ASSET_ARRAYS@, @ASSET_TABLE@, @N_ASSETS@, @USE_GZIP@ and the #cmakedefine. ETags are the SHA-256 of the embedded bytes, as upstream computes them; the MIME table mirrors mime_from_ext(), and an extension missing from it degrades to application/octet-stream rather than breaking anything.

At build time llama.cpp/BUILD.mk runs ui-embed.sh <out_cpp> <out_h> [dist], which walks dist/ (using the dist/_gzip/ mirror when present, so the embedded bytes are gzip-compressed) and writes o/$(MODE)/llama.cpp/tools/ui/ui.{cpp,h}, compiled like any other C++ source and linked into llama-server and llamafile. server-http.cpp registers a route per embedded asset (index.html at /) and serves them with Content-Encoding: gzip. If the download fails (offline, version not yet on HF) the fetch script leaves dist/ empty; ui-embed.sh then emits the no-asset stub and the LLAMA_UI_HAS_ASSETS guard keeps the UI routes unregistered, so the REST API still works. A zero-length asset takes the same path, with a warning — upstream hard-errors there, but aborting a whole build over a corrupt UI tarball is the wrong trade for us.

ui-embed.sh embeds whatever it is given without validating the set, so the one required-asset check lives in fetch-ui-assets.sh (ui_missing_assets: index.html, manifest.webmanifest, sw.js, build.json, version.json and the hashed bundle*.js/bundle*.css/workbox*.js). When a downloaded tarball is partial or has drifted it clears dist/ and takes the UI-less path. That list mirrors ui_validate_assets() in scripts/ui-assets.cmake, which is upstream's definition of a complete tree and the only remaining sync point in the UI path — when upstream changes it on a bump, update ui_missing_assets to match, or we will accept a tree upstream considers incomplete (silently broken UI) or reject one it considers fine (UI-less build).

Upstream fixes carried ahead of a release

Patches that are not llamafile's own: an upstream fix we need before it lands in a tagged llama.cpp. Drop each one at the bump that first includes it, rather than reconciling it — check_patches.sh will flag it as conflicting once upstream has the same change.

Patch Description
ggml_src_ggml-backend.cpp.patch (part) PR #25584, still open at v0.8.0. When a buffer type has a max_size (1 GiB on Vulkan), the allocation of a context's tensors is split across several buffers; when a tensor larger than max_size is followed only by views, the zero-sized tail is never planned as a buffer range and those views never get ggml_backend_view_init(). The persistent KV stream views (layer.k_stream/v_stream) then keep data == NULL, and the server's state save/restore hits GGML_ASSERT(tensor->data != NULL && "tensor not allocated") in ggml_backend_tensor_get(). The fix allocates only parent tensors per range and initializes every view in one final pass. Up to b11100 the splitting lived in ggml-alloc.c (where the PR applies, as ggml_src_ggml-alloc.c.patch); v0.8.0 (#23671) moved it, bug included, into ggml_backend_buft_alloc_buffer_n_default() in ggml-backend.cpp, so the same fix is now carried there. Symptom without it: llama-server on Vulkan dies on the second request at the default --parallel 4 (upstream #29221; --parallel 1 is the workaround). Reproduced on an L40S with vanilla b11100, so it is not llamafile-specific; verified fixed on both vanilla and llamafile at b11100. Upstream also files it against #19839, #23737 and #21762.

Bug Fixes

Patch Description
ggml_src_ggml-backend-reg.cpp.patch Suppresses debug log noise for non-existent backend search paths (irrelevant for llamafile's DSO loading approach). Since v0.8.0 (#28367) a missing path surfaces as a directory_iterator ENOENT, so only that error is silenced; other enumeration failures are still logged
src_models_t5.cpp.patch Forward-declares the graph<false>/graph<true> explicit specializations before build_arch_graph so clang's -std=gnu++23 doesn't reject them as specializations after implicit instantiation
ggml_src_ggml.c.patch Makes the ggml_time_ms()/ggml_time_us() call ggml_time_init() lazily. The Windows dylibs link their own copy of ggml.c, so the DLL's timer_freq is never set by the executable's ggml_init() and any ggml_time_*() call made from inside the DLL divides by zero (0xC0000094, surfaced by Cosmopolitan as SIGFPE). First hit when #1051 enabled -DGGML_CUDA_USE_GRAPHS, calling ggml_time_us() in ggml-cuda. Linux/macOS are unaffected (clock_gettime, no static divisor). v0.8.0 (#30969) made the division floating-point, so it no longer traps, but inf cast to int64_t is undefined (INT64_MIN on x86) and every DLL-side timing would be garbage, so the lazy init is still needed.

The same clang -std=gnu++23 problem used to need patches for eagle3.cpp and dflash.cpp; b11100 defines build_arch_graph at the end of both files, after the specializations, so those two patches were dropped as obsolete. A new model file that defines build_arch_graph before its graph<...> specializations will reintroduce the error, and wants the same treatment.

Creating New Patches

Files in llama.cpp are usually modified in-place for development and testing. Once they are ready to be committed, you can update all files in the llama.cpp.patches directory by running the following:

# echo y answers the prompt; the subshell restores the cwd even on failure
( cd llama.cpp && echo y | ../tools/generate_patches.sh --output-dir ../llama.cpp.patches )

Patch filenames will automatically reflect the file path with underscores replacing slashes (e.g., common_arg.cpp.patch for common/arg.cpp).