Distribute and run LLMs with a single file.
C++ cross-platformggufllama-cpplocal-ailocal-inferencelocal-llmopen-source-aisingle-file-executablespeech-to-text
[draft] Update llama.cpp to b11100 (#1061)
Davide Eynardfef6e44e1b71
diff --git a/docs/cli_arguments.md b/docs/cli_arguments.mdindex beda957c..5be2866d 100644--- a/docs/cli_arguments.md+++ b/docs/cli_arguments.md@@ -89,12 +89,39 @@ compatibility.| `-c, --ctx-size N` | Context window size. `0` means to use the model default. || `-b, --batch-size N` | Logical maximum batch size. || `-ub, --ubatch-size N` | Physical maximum batch size. |-| `--mlock` | Keep the model in RAM instead of letting the OS swap or compress it. |+| `-lm, --load-mode MODE` | How the weights are read: `auto` (default), `none`, `mmap`, `mlock`, `mmap+mlock`, `dio`. Replaces `--mmap`, `--no-mmap`, `--mlock` and `-dio`; see [Model Loading Flags](#model-loading-flags). || `--repeat-penalty N` | Penalize repeating tokens during sampling. `1.0` disables the penalty. || `-ngl, --gpu-layers, --n-gpu-layers N` | Number of layers to offload to GPU. || `--host HOST` | Server bind address. || `--port PORT` | Server listen port. |+## Model Loading Flags++llama.cpp b11100 replaced the separate `--mmap`, `--no-mmap`, `--mlock`,+`-dio`/`--direct-io` and `-ndio`/`--no-direct-io` flags with a single+`-lm, --load-mode MODE`, and no longer accepts the old flags. Use+`--load-mode` in new command lines and `.args` files, as llama.cpp does.++So that existing llamafiles and `.args` files keep working, llamafile still+accepts the old flags. It prints a warning and replaces them with the+equivalent `--load-mode`, keeping their original meaning: memory mapping is on+unless `--no-mmap` is given, `--mlock` locks the model in RAM on top of that,+and direct I/O takes precedence over memory mapping.++| Old flags | Equivalent |+| --- | --- |+| `--mmap` | `--load-mode mmap` |+| `--no-mmap` | `--load-mode none` |+| `--mlock` | `--load-mode mmap+mlock` |+| `--no-mmap --mlock` | `--load-mode mlock` |+| `-dio`, `--direct-io` | `--load-mode dio` |+| `-ndio`, `--no-direct-io` | `--load-mode mmap` |++Several old flags combine into one mode, applied in command-line order. If+you mix them with `--load-mode`, whichever comes last wins. The old+`LLAMA_ARG_MMAP`, `LLAMA_ARG_MLOCK` and `LLAMA_ARG_DIO` environment variables+are not translated; use `LLAMA_ARG_LOAD_MODE` instead.+## Wrapper and Mode Flags- Mode selection: `--server`, `--chat`, `--cli`@@ -125,7 +152,7 @@ wrapper.- KV cache and host memory: `-kvo`, `--kv-offload`, `-nkvo`, `--no-kv-offload`, `--repack`, `-nr`, `--no-repack`, `--no-host`- Cache storage and defragmentation: `-ctk`, `--cache-type-k`, `-ctv`, `--cache-type-v`, `-dt`, `--defrag-thold`-- Memory mapping and residency: `--mlock`, `--mmap`, `--no-mmap`, `--numa`, `--check-tensors`, `--op-offload`, `--no-op-offload`+- Memory mapping and residency: `-lm`, `--load-mode`, `-lzm`, `--lazy-mode`, `--numa`, `--check-tensors`, `--op-offload`, `--no-op-offload`- Draft-model cache types: `-ctkd`, `--cache-type-k-draft`, `-ctvd`, `--cache-type-v-draft`### Devices, GPU, Tensors, and Adaptersdiff --git a/docs/creating_llamafiles.md b/docs/creating_llamafiles.mdindex 1707fb8a..90e5ffd5 100644--- a/docs/creating_llamafiles.md+++ b/docs/creating_llamafiles.md@@ -70,7 +70,8 @@ on-n32768--no-context-shift---no-mmap+--load-mode+none...```@@ -81,6 +82,12 @@ on> prefix is required whenever referencing a file packaged inside the llamafile.> The `...` token is replaced with any additional CLI arguments the user passes> at runtime.+>+> `--load-mode none` disables memory mapping. It replaces `--no-mmap`, which+> llama.cpp removed in b11100. Existing `.args` files that still use+> `--no-mmap`, `--mmap` or `--mlock` keep working, because llamafile+> translates them (see [Model Loading Flags](cli_arguments.md#model-loading-flags)).+> New ones should use `--load-mode`.- Copy the llamafile executable and run zipalign to embed the weights and args:@@ -128,7 +135,8 @@ embed both the weights and the following `.args` file0.0.0.0-ngl9999---no-mmap+--load-mode+none...```diff --git a/llama.cpp b/llama.cppindex 0177dcc7..7ab4ee7b 160000--- a/llama.cpp+++ b/llama.cpp@@ -1 +1 @@-Subproject commit 0177dcc7300bad8914bb838baabce87899812491+Subproject commit 7ab4ee7baad2d920464cbacfad4f4b07cf111fd2diff --git a/llama.cpp.patches/README.md b/llama.cpp.patches/README.mdindex 67c7f7b5..a64f50a8 100644--- a/llama.cpp.patches/README.md+++ b/llama.cpp.patches/README.md@@ -9,6 +9,7 @@ llama.cpp.patches/├── README.md # This file├── apply-patches.sh # Script to apply all patches to llama.cpp submodule├── fetch-ui-assets.sh # Downloads + validates the prebuilt web UI (see Server Integration)+├── ui-embed.sh # Renders upstream's ui.{cpp,h}.in templates (see Server Integration)├── renames.sh # Script for file renames/moves (if any)├── llamafile-files/ # Additional files to copy into llama.cpp│ ├── BUILD.mk # Makefile for building llama.cpp with cosmocc@@ -53,6 +54,9 @@ The `GGML_CALL` macro (defined as `__attribute__((__ms_abi__))` when `GGML_MULTI| `ggml_src_ggml-cuda_ggml-cuda.cu.patch` | Adds `GGML_CALL` to all CUDA backend callback implementations (60+ functions); also adds `free_struct` and TinyBLAS BF16 guard (see below) || `ggml_src_ggml-metal_ggml-metal.cpp.patch` | Adds `GGML_CALL` to all Metal backend callback implementations (62 functions); also adds `free_struct` (see below) || `ggml_src_ggml-vulkan_ggml-vulkan.cpp.patch` | Adds `GGML_CALL` to all Vulkan backend callback implementations; also adds `free_struct` and a heap memory underflow fix (see below) |+| `ggml_src_ggml-vulkan_ggml-vulkan-common.h.patch` | Adds `GGML_CALL` to the backend callback declarations. b11100 split the Vulkan backend into several translation units and moved the interface structs into `ggml-vulkan-buffers.cpp`, so the callbacks it holds became non-static and are declared here; declaration and definition must agree on the convention |+| `ggml_src_ggml-vulkan_ggml-vulkan-buffers.cpp.patch` | Adds the `free_struct` entry to `ggml_backend_vk_buffer_interface`, which b11100 moved into this new file |+| `ggml_src_ggml-metal_ggml-metal-fusion.h.patch` / `.cpp.patch` | Adds `GGML_CALL` to the `add_alloc_dep` function pointer these pass through (see Graph-optimize allocation dependencies below), and includes `ggml-backend.h` for the macro || `ggml_src_ggml-backend-meta.cpp.patch` | Adds `GGML_CALL` to all meta-device, meta-buffer-type, meta-buffer, and meta-backend callback implementations (the meta backend aggregates several simple backends behind one interface, so its callbacks are reached through the same function-pointer structs) |### Cross-Module Memory Management@@ -66,6 +70,39 @@ When GPU backends (CUDA, Vulkan, Metal) are loaded as dynamic libraries, memory| `ggml_src_ggml-cuda_ggml-cuda.cu.patch` | Adds `free_struct` implementation for CUDA buffers (regular and host; upstream removed the split buffer in b10052); sets it on fallback CPU buffers allocated within the DSO || `ggml_src_ggml-metal_ggml-metal.cpp.patch` | Adds `free_struct` implementation for Metal shared and private buffers || `ggml_src_ggml-vulkan_ggml-vulkan.cpp.patch` | Adds `free_struct` implementation for Vulkan buffers and host buffer fallback path |+| `ggml_src_ggml-vulkan_ggml-vulkan-buffers.cpp.patch` | Wires that `free_struct` into `ggml_backend_vk_buffer_interface`, which lives in this file since b11100 |++### Graph-optimize allocation dependencies++b11100 gave `graph_optimize` a `ggml_backend_graph_optimize_params *`, through+which a backend tells the scheduler to keep a tensor alive until a later node+(`params->add_alloc_dep(...)`). The callback is *the host's*, and the CUDA,+Vulkan and Metal backends call it from inside their DSO, so it crosses the ABI+boundary in the opposite direction to the backend interface structs and needs+`GGML_CALL` just the same.++| Patch | Description |+|-------|-------------|+| `ggml_src_ggml-backend-impl.h.patch` | Adds `GGML_CALL` to the `add_alloc_dep` member of `ggml_backend_graph_optimize_params` |+| `ggml_src_ggml-backend.cpp.patch` | Replaces upstream's captureless lambda for `add_alloc_dep` with a named `ggml_backend_sched_add_alloc_dep()` — a lambda cannot carry the attribute |+| `ggml_src_ggml-metal_ggml-metal-fusion.h.patch` / `.cpp.patch` | Annotates the same pointer where `ggml_metal_fusion_add_alloc_deps()` passes it on, and includes `ggml-backend.h` so the macro is in scope |++### Generated version headers++Since b11100 `ggml/src/ggml.c` includes `ggml-version.h` and `src/llama.cpp`+includes `llama-version.h`; both are produced by CMake's `configure_file()`+from the `.in` templates next to them, and hold the `GGML_VERSION`/`GGML_COMMIT`+and `LLAMA_VERSION`/`LLAMA_COMMIT` macros that used to arrive as `-D` flags.+There is no CMake step in the cosmocc build, so **`apply-patches.sh` generates+both headers** (with the same values `build/config.mk` computes) straight into+the source tree rather than into `o/`. That single location covers every+consumer: the make build, `llamafile/*.sh` and `*.bat`, which compile `ggml.c`+with `-I ggml/src`, and `llamafile/metal.c`, which extracts `ggml-version.h`+alongside `ggml.c` for the runtime Metal build.++They are generated, not patched: keep them out of `llamafile-files/` (delete+both before running `generate-patches`, which would otherwise pick them up as+new files and freeze one bump's commit hash into the repo).### Cosmopolitan Libc Compatibility@@ -84,8 +121,8 @@ Cosmopolitan libc has specific behaviors with condition variables and signals th| Patch | Description ||-------|-------------|-| `common_log.cpp.patch` | Adds `#include <csignal>`; blocks `SIGINT`/`SIGTERM` on logger thread via `pthread_sigmask` to prevent `EINTR` exceptions; replaces `cv.wait()` with `wait_for(30s)` loop to work around XNU futex timeout bug (~72 minute expiry) |-| `tools_server_server-models.cpp.patch` | Adds `#include <csignal>`; blocks signals on the stopping thread via `pthread_sigmask`; replaces untimed `cv.wait()` with `wait_for(30s)` loops on every model-lifecycle wait (`unload_lru`, the reload-drain wait, `stopping_thread`, the `is_reloading` guard in `load`, and the generic `wait()` predicate helper) to work around the XNU futex timeout bug |+| `common_log.cpp.patch` | Adds `#include <csignal>`; blocks `SIGINT`/`SIGTERM` on logger thread via `pthread_sigmask` to prevent `EINTR` exceptions; replaces the untimed `cv.wait()` calls with `wait_for(30s)` loops (the worker's `cv_new` wait, plus the queue-full `cv_full` waits in `add()` and in `add_json()`, new upstream in b11100) to work around XNU futex timeout bug (~72 minute expiry). The `add*()` waits run on the logging caller's thread, so the signal mask does not cover them |+| `tools_server_server-models.cpp.patch` | Adds `#include <csignal>`; blocks signals via `pthread_sigmask` on `server_monitor`'s thread (b11100 replaced the per-model `stopping_thread` this used to cover with one monitor thread watching every child); replaces untimed `cv.wait()` with `wait_for(30s)` loops on every model-lifecycle wait (`unload_lru`, the reload-drain wait, `unload_all`, the `is_reloading` guard in `load`, and the generic `wait()` predicate helper) to work around the XNU futex timeout bug || `tools_server_server-queue.cpp.patch` | Adds missing includes (`<cerrno>`, `<system_error>`, `<csignal>`); blocks `SIGINT`/`SIGTERM` on queue thread (the `yield_to_queue` worker thread is spawned after this, so it inherits the mask); replaces `wait()` with `wait_for(30s)` loops in five locations (`wait_until_no_sleep`, main loop, `recv`, plus `worker_loop` and `yield_to_queue`, both new upstream in b10441); runs `yield_to_queue()` inline when a GPU backend is loaded (see "GPU decode must stay on the main thread" below) || `tools_server_server-stream.cpp.patch` | Adds `#include <csignal>`; blocks signals on the stream-session GC thread via `pthread_sigmask`. Without it, Ctrl+C on `llama-server` aborts with *"libc++abi: terminating due to uncaught exception ... condition_variable timed_wait failed: Interrupted system call"* instead of shutting down: `SIGINT` is delivered to that thread while it sits in `gc_wake_cv.wait_for(60s)`, `pthread_cond_timedwait()` returns `EINTR`, and libcxx rethrows it as `std::system_error` that nothing catches. This thread was the last one in the server with a condition-variable wait and no signal mask (the log worker, task queue, model-stopping thread and httplib pool were already covered). Pre-existing, not introduced by b10441; combined TUI mode is unaffected || `vendor_cpp-httplib_httplib.cpp.patch` | Fixes httplib thread pool with `wait_for()` instead of `wait()` for XNU futex compatibility; also see HTTPS / TLS Support below |@@ -197,6 +234,17 @@ serviced during it, so the sole effect is that `/metrics` and `/slots` wait forthe in-flight batch (~29 ms while generating; up to a few seconds for a fullprompt batch). This will recur whenever upstream moves work onto a new thread.+**Bitmap hashing needs `vendor/hash`** (added to `BUILD.mk`, not a patch).+b11100 made `mtmd-helper.cpp` derive a bitmap's ID from `hash_sha256_hex()` in+the vendored `vendor/hash`, which upstream links into mtmd as `vendor::hash`.+`BUILD.mk` builds `hash.cpp` and `sha256/sha256.c` into `MTMD_OBJS` (with+`-iquote llama.cpp/vendor/hash`, since `sha256.c` reaches for+`"rotate-bits/rotate-bits.h"`). Upstream's target also builds `xxhash` and+`sha1`; nothing here calls either, and `sha1` would have to be compiled as C+++despite its `.c` extension (its declarations sit in a namespace, and upstream+forces `LANGUAGE CXX` on it), so neither is built. A future caller shows up as+an undefined reference at link.+**Subprocess support needs `-DLLAMA_SUBPROCESS`** (set in `BUILD.mk`, not apatch). b10441 moved child-process spawning into `common_subproc`(`common/subproc.*`, wrapping the vendored `sheredom/subprocess.h`) and put it@@ -210,7 +258,7 @@ llamafile's own flags and passes the rest to `common_params_parse`:`server-mcp.cpp` spawns them and, unlike the other two, does **not** check`common_subproc::is_supported()`, so without the define they fail silentlyrather than reporting an error.-- **`--server-tools`** — `server-tools.cpp` throws *"subprocess is not enabled+- **`--tools`** — `server-tools.cpp` throws *"subprocess is not enabledon this build"*, caught into a clean `return 1`.- **Router mode** — `llama-server` with no model; `init_routes()` throws thesame message, and the server exits at startup.@@ -221,8 +269,8 @@ with it. Enabling it pulls `subprocess.h` into the build, which needs theCosmopolitan fix in the patch table above.The web UI moved upstream from prebuilt `tools/server/public/*` assets to-a Svelte/PWA project under `tools/ui/`, embedded at CMake time via-`tools/ui/embed.cpp`. cosmocc has no JS toolchain, so `fetch-ui-assets.sh`+a Svelte/PWA project under `tools/ui/`, embedded into the binary at build+time. cosmocc has no JS toolchain, so `fetch-ui-assets.sh`(run by `apply-patches.sh` / `make setup`) downloads the prebuilt site**tarball** `dist.tar.gz` (plus its `.sha256`) from the `ggml-org/llama-ui`Hugging Face bucket — picking the newest `bNNNN` tag `<=` our pinned build —@@ -233,31 +281,57 @@ verifies it, and extracts the whole static site intoscreens). To keep the embedded payload small, the script also builds a`dist/_gzip/` mirror with every file gzip-compressed under its original name.-At build time, `llama.cpp/BUILD.mk` compiles `tools/ui/embed.cpp` with-`cosmoc++` (its APE output runs on the host) and runs it against the `dist/`-**directory** (the new `embed <out_cpp> <out_h> [<asset_dir>]` interface):-`embed.cpp` recursively bakes every file — auto-detecting `dist/_gzip/` and-emitting gzip-encoded assets keyed by relative path — into-`o/$(MODE)/llama.cpp/tools/ui/ui.{cpp,h}`, which is compiled like any other-C++ source and linked into `llama-server` and `llamafile`. Upstream's-`server-http.cpp` (unpatched) registers a route per embedded asset-(`index.html` at `/`) and serves gzip assets with `Content-Encoding: gzip`.-If the download fails (offline, version not yet on HF) the script leaves-`dist/` empty — `embed.cpp` then emits a no-op `llama_ui_find_asset` and the-`LLAMA_UI_HAS_ASSETS` guard keeps the UI routes unregistered, so the REST-API still works.--Note that `embed.cpp` only takes that graceful no-asset path when `dist/` is-*empty*: once the directory is non-empty it enforces a required-asset set-(its `required_check[]` table — `index.html`, `manifest.webmanifest`, `sw.js`,-`build.json`, `version.json`, and the hashed-`bundle*.js`/`bundle*.css`/`workbox*.js`) and returns a hard error if any is-missing. So `fetch-ui-assets.sh` validates the *same* set after extracting-(`ui_missing_assets`); if a downloaded tarball is partial or has drifted, it-clears `dist/` and falls back to a UI-less build rather than letting the embed-step abort the whole build. Because `embed.cpp` is an upstream file, that list-can change on a llama.cpp bump — when it does, the asset list in-`ui_missing_assets` must be updated to match.+b11100 deleted upstream's standalone `tools/ui/embed.cpp` and replaced it with+`scripts/ui-assets.cmake`, which provisions the assets and renders+`tools/ui/ui.{cpp,h}.in` into `ui.cpp`/`ui.h` inside a CMake build. There is no+CMake step in the cosmocc build, so **`ui-embed.sh` does that rendering** — a+translation of that script's `emit_files()`, in the same spirit as `BUILD.mk`+being a translation of llama.cpp's CMake build.++The important property is that it renders **upstream's own templates**, so the+generated interface (`llama_ui_find_asset` / `llama_ui_get_assets` /+`llama_ui_use_gzip` and `LLAMA_UI_HAS_ASSETS` — all that the unpatched+`server-http.cpp` consumes) tracks upstream automatically and cannot drift.+Only the substitutions live on our side: `@ASSET_ARRAYS@`, `@ASSET_TABLE@`,+`@N_ASSETS@`, `@USE_GZIP@` and the `#cmakedefine`. ETags are the SHA-256 of the+embedded bytes, as upstream computes them; the MIME table mirrors+`mime_from_ext()`, and an extension missing from it degrades to+`application/octet-stream` rather than breaking anything.++At build time `llama.cpp/BUILD.mk` runs `ui-embed.sh <out_cpp> <out_h> [dist]`,+which walks `dist/` (using the `dist/_gzip/` mirror when present, so the+embedded bytes are gzip-compressed) and writes+`o/$(MODE)/llama.cpp/tools/ui/ui.{cpp,h}`, compiled like any other C++ source+and linked into `llama-server` and `llamafile`. `server-http.cpp` registers a+route per embedded asset (`index.html` at `/`) and serves them with+`Content-Encoding: gzip`. If the download fails (offline, version not yet on+HF) the fetch script leaves `dist/` empty; `ui-embed.sh` then emits the+no-asset stub and the `LLAMA_UI_HAS_ASSETS` guard keeps the UI routes+unregistered, so the REST API still works. A zero-length asset takes the same+path, with a warning — upstream hard-errors there, but aborting a whole build+over a corrupt UI tarball is the wrong trade for us.++`ui-embed.sh` embeds whatever it is given without validating the set, so the+**one** required-asset check lives in `fetch-ui-assets.sh` (`ui_missing_assets`:+`index.html`, `manifest.webmanifest`, `sw.js`, `build.json`, `version.json` and+the hashed `bundle*.js`/`bundle*.css`/`workbox*.js`). When a downloaded tarball+is partial or has drifted it clears `dist/` and takes the UI-less path. That+list mirrors `ui_validate_assets()` in `scripts/ui-assets.cmake`, which is+upstream's definition of a complete tree and the only remaining sync point in+the UI path — when upstream changes it on a bump, update `ui_missing_assets` to+match, or we will accept a tree upstream considers incomplete (silently broken+UI) or reject one it considers fine (UI-less build).++### Upstream fixes carried ahead of a release++Patches that are **not** llamafile's own: an upstream fix we need before it+lands in a tagged llama.cpp. Drop each one at the bump that first includes it,+rather than reconciling it — `check_patches.sh` will flag it as conflicting+once upstream has the same change.++| Patch | Description |+|-------|-------------|+| `ggml_src_ggml-alloc.c.patch` | [PR #25584](https://github.com/ggml-org/llama.cpp/pull/25584), verbatim minus its test. `ggml_backend_alloc_ctx_tensors_from_buft()` splits a context across buffers when a tensor exceeds the backend's `max_size` (1 GiB on Vulkan); when the tail of the context holds only views, the final `alloc_tensor_range()` is skipped and those views never get `ggml_backend_view_init()`. The persistent KV stream views (`layer.k_stream`/`v_stream`) then keep `data == NULL`, and the server's state save/restore hits `GGML_ASSERT(tensor->data != NULL && "tensor not allocated")` in `ggml_backend_tensor_get()`. The fix allocates only parent tensors per split range and initializes every view in one final pass. **Symptom without it:** `llama-server` on Vulkan dies on the *second* request at the default `--parallel 4` (upstream [#29221](https://github.com/ggml-org/llama.cpp/issues/29221); `--parallel 1` is the workaround). Reproduced on an L40S with vanilla b11100, so it is not llamafile-specific; verified fixed on both vanilla and llamafile. Upstream also files it against #19839, #23737 and #21762. |### Bug Fixes@@ -266,10 +340,14 @@ can change on a llama.cpp bump — when it does, the asset list in| `ggml_src_ggml-backend-reg.cpp.patch` | Suppresses debug log noise for non-existent backend search paths (irrelevant for llamafile's DSO loading approach) || `ggml_src_ggml-vulkan_ggml-vulkan.cpp.patch` | Fixes unsigned integer underflow in `ggml_backend_vk_get_device_memory` where Vulkan's `heapUsage` can exceed `heapBudget` (clamps to zero instead of wrapping) || `src_models_t5.cpp.patch` | Forward-declares the `graph<false>`/`graph<true>` explicit specializations before `build_arch_graph` so clang's `-std=gnu++23` doesn't reject them as specializations after implicit instantiation |-| `src_models_eagle3.cpp.patch` | Moves `build_arch_graph` to the end of the file, after the `graph<true>`/`graph<false>` constructor specializations, so clang's `-std=gnu++23` doesn't reject them as explicit specializations appearing after the `make_unique<graph<...>>` implicit instantiation point |-| `src_models_dflash.cpp.patch` | Same fix as `eagle3` for the DFlash model (new in b10052): moves `build_arch_graph` to the end of the file, after the `graph<true>`/`graph<false>`/`graph_dsv4` specializations, so clang's `-std=gnu++23` doesn't reject them as explicit specializations after the `make_unique<graph<...>>` implicit instantiation point || `ggml_src_ggml.c.patch` | Makes the `ggml_time_ms()`/`ggml_time_us()` call `ggml_time_init()` lazily. The Windows dylibs link their own copy of `ggml.c`, so the DLL's `timer_freq` is never set by the executable's `ggml_init()` and any `ggml_time_*()` call made from inside the DLL divides by zero (`0xC0000094`, surfaced by Cosmopolitan as SIGFPE). First hit when #1051 enabled `-DGGML_CUDA_USE_GRAPHS`, calling `ggml_time_us()` in `ggml-cuda`. Linux/macOS are unaffected (`clock_gettime`, no static divisor). |+The same clang `-std=gnu++23` problem used to need patches for `eagle3.cpp` and+`dflash.cpp`; b11100 defines `build_arch_graph` at the end of both files, after+the specializations, so those two patches were dropped as obsolete. A new model+file that defines `build_arch_graph` before its `graph<...>` specializations+will reintroduce the error, and wants the same treatment.+## Creating New PatchesFiles in `llama.cpp` are usually modified in-place for development and testing.diff --git a/llama.cpp.patches/apply-patches.sh b/llama.cpp.patches/apply-patches.shindex 2ddcd43a..95ca9e1d 100755--- a/llama.cpp.patches/apply-patches.sh+++ b/llama.cpp.patches/apply-patches.sh@@ -37,6 +37,27 @@ cp -r "$LLAMAFILE_FILES_DIR"/* .../llama.cpp.patches/renames.sh+# Generate the version headers that CMake's configure_file() produces. Since+# b11100 ggml/src/ggml.c #includes "ggml-version.h" and src/llama.cpp #includes+# "llama-version.h"; before that the same macros came from -D flags. They are+# written next to their sources (rather than into o/) so that every consumer+# finds them: the cosmocc make build, and llamafile/*.sh + *.bat, which compile+# ggml.c with -I ggml/src.+echo "Generating version headers..."+ggml_version_major=$(grep -E 'GGML_VERSION_MAJOR [0-9]+' ggml/CMakeLists.txt | sed 's/[^0-9]*//g')+ggml_version_minor=$(grep -E 'GGML_VERSION_MINOR [0-9]+' ggml/CMakeLists.txt | sed 's/[^0-9]*//g')+ggml_version_patch=$(grep -E 'GGML_VERSION_PATCH [0-9]+' ggml/CMakeLists.txt | sed 's/[^0-9]*//g')+ggml_commit=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown")+llama_version=$(git describe --tags --always 2>/dev/null || echo "unknown")++sed -e "s/@GGML_VERSION@/${ggml_version_major}.${ggml_version_minor}.${ggml_version_patch}/g" \+ -e "s/@GGML_BUILD_COMMIT@/${ggml_commit}/g" \+ ggml/src/ggml-version.h.in > ggml/src/ggml-version.h++sed -e "s/@LLAMA_VERSION@/${llama_version}/g" \+ -e "s/@LLAMA_BUILD_COMMIT@/${ggml_commit}/g" \+ src/llama-version.h.in > src/llama-version.h+echo "Removing unnecessary files and directories..."# If you want to clean up the original code, add your `rm` commands here.# For example:diff --git a/llama.cpp.patches/fetch-ui-assets.sh b/llama.cpp.patches/fetch-ui-assets.shindex 6180f173..c81257e2 100755--- a/llama.cpp.patches/fetch-ui-assets.sh+++ b/llama.cpp.patches/fetch-ui-assets.sh@@ -70,23 +70,27 @@ build_gzip_mirror() {( cd "$UI_DIST" && find . -type f ! -path './_gzip/*' -print0 \| while IFS= read -r -d '' f; domkdir -p "_gzip/$(dirname "$f")"- gzip -9 -c "$f" > "_gzip/$f"+ # -n: omit the timestamp, so identical assets give identical bytes+ # (and therefore stable ETags) on every machine and every fetch.+ gzip -9 -n -c "$f" > "_gzip/$f"done )}-# Echo (one per line) any asset embed.cpp requires that is absent from the tree.+# Echo (one per line) any asset the web UI needs at runtime that is absent from+# the tree.#-# !!! KEEP IN SYNC WITH UPSTREAM tools/ui/embed.cpp !!!-# The list below mirrors the required_check[] table in llama.cpp/tools/ui/embed.cpp.-# Once dist/ is non-empty, embed.cpp hard-fails the build (return 1) if any of-# those assets is missing. If we only checked index.html here, a partial/drifted-# tarball would pass our check but then abort the *entire* build at the embed-# step instead of falling back to a UI-less build. So we validate the same set-# embed.cpp does and, when it's incomplete, clear dist/ to take the UI-less path.+# !!! KEEP IN SYNC WITH UPSTREAM scripts/ui-assets.cmake !!!+# The list below mirrors ui_validate_assets() in llama.cpp/scripts/ui-assets.cmake,+# which is upstream's definition of a complete asset tree. This is the only copy+# of it on our side: ui-embed.sh embeds whatever it is given without validating,+# so a partial or drifted tarball that slipped past this check would silently+# produce a broken UI rather than a failed build. When the set is incomplete we+# clear dist/ and take the UI-less path instead.#-# embed.cpp is an UPSTREAM file, so it can change on a llama.cpp bump. When it-# does (a new required asset, a renamed one), this list must be updated to match-# -- otherwise the two checks disagree again and the build-failure bug returns.+# ui-assets.cmake is an UPSTREAM file, so its list can change on a llama.cpp+# bump. When it does (a new required asset, a renamed one), update this list to+# match -- otherwise we accept a tree upstream considers incomplete, or reject+# one it considers fine.ui_missing_assets() {local root="$1" f blocal -a bases=()diff --git a/llama.cpp.patches/llamafile-files/BUILD.mk b/llama.cpp.patches/llamafile-files/BUILD.mkindex 1c680c60..9e670287 100644--- a/llama.cpp.patches/llamafile-files/BUILD.mk+++ b/llama.cpp.patches/llamafile-files/BUILD.mk@@ -34,6 +34,7 @@ GGML_SRCS_CPP := \llama.cpp/ggml/src/ggml-cpu/binary-ops.cpp \llama.cpp/ggml/src/ggml-cpu/ggml-cpu.cpp \llama.cpp/ggml/src/ggml-cpu/hbm.cpp \+ llama.cpp/ggml/src/ggml-cpu/iqp.cpp \llama.cpp/ggml/src/ggml-cpu/ops.cpp \llama.cpp/ggml/src/ggml-cpu/repack.cpp \llama.cpp/ggml/src/ggml-cpu/traits.cpp \@@ -60,6 +61,7 @@ LLAMA_SRCS_CPP := \llama.cpp/src/models/baichuan.cpp \llama.cpp/src/models/bailingmoe.cpp \llama.cpp/src/models/bailingmoe2.cpp \+ llama.cpp/src/models/bailingmoe3.cpp \llama.cpp/src/models/bert.cpp \llama.cpp/src/models/bitnet.cpp \llama.cpp/src/models/bloom.cpp \@@ -81,6 +83,7 @@ LLAMA_SRCS_CPP := \llama.cpp/src/models/delta-net-base.cpp \llama.cpp/src/models/dflash.cpp \llama.cpp/src/models/dots1.cpp \+ llama.cpp/src/models/dots3note.cpp \llama.cpp/src/models/dream.cpp \llama.cpp/src/models/eagle3.cpp \llama.cpp/src/models/ernie4-5-moe.cpp \@@ -107,6 +110,7 @@ LLAMA_SRCS_CPP := \llama.cpp/src/models/granite-moe.cpp \llama.cpp/src/models/granite-switch.cpp \llama.cpp/src/models/granite.cpp \+ llama.cpp/src/models/granite-swa.cpp \llama.cpp/src/models/mamba-base.cpp \llama.cpp/src/models/grok.cpp \llama.cpp/src/models/grovemoe.cpp \@@ -114,6 +118,8 @@ LLAMA_SRCS_CPP := \llama.cpp/src/models/hunyuan-moe.cpp \llama.cpp/src/models/hunyuan-vl.cpp \llama.cpp/src/models/hy-v3.cpp \+ llama.cpp/src/models/hy-v4.cpp \+ llama.cpp/src/models/hrm-text.cpp \llama.cpp/src/models/internlm2.cpp \llama.cpp/src/models/jais.cpp \llama.cpp/src/models/jais2.cpp \@@ -121,6 +127,7 @@ LLAMA_SRCS_CPP := \llama.cpp/src/models/jina-bert-v2.cpp \llama.cpp/src/models/jina-bert-v3.cpp \llama.cpp/src/models/kimi-linear.cpp \+ llama.cpp/src/models/kimi-k3.cpp \llama.cpp/src/models/laguna.cpp \llama.cpp/src/models/lfm2.cpp \llama.cpp/src/models/lfm2moe.cpp \@@ -132,6 +139,7 @@ LLAMA_SRCS_CPP := \llama.cpp/src/models/maincoder.cpp \llama.cpp/src/models/mamba.cpp \llama.cpp/src/models/mamba2.cpp \+ llama.cpp/src/models/maple.cpp \llama.cpp/src/models/mellum.cpp \llama.cpp/src/models/mimo2.cpp \llama.cpp/src/models/minicpm.cpp \@@ -179,6 +187,7 @@ LLAMA_SRCS_CPP := \llama.cpp/src/models/qwen35moe.cpp \llama.cpp/src/models/qwen3vl.cpp \llama.cpp/src/models/qwen3vlmoe.cpp \+ llama.cpp/src/models/qwen4exp.cpp \llama.cpp/src/models/refact.cpp \llama.cpp/src/models/rnd1.cpp \llama.cpp/src/models/rwkv6-base.cpp \@@ -189,6 +198,7 @@ LLAMA_SRCS_CPP := \llama.cpp/src/models/seed-oss.cpp \llama.cpp/src/models/smallthinker.cpp \llama.cpp/src/models/smollm3.cpp \+ llama.cpp/src/models/spark2-5.cpp \llama.cpp/src/models/stablelm.cpp \llama.cpp/src/models/starcoder.cpp \llama.cpp/src/models/starcoder2.cpp \@@ -210,11 +220,13 @@ LLAMA_SRCS_CPP := \llama.cpp/src/llama-impl.cpp \llama.cpp/src/llama-io.cpp \llama.cpp/src/llama-kv-cache-dsa.cpp \+ llama.cpp/src/llama-kv-cache-dsa-iswa.cpp \llama.cpp/src/llama-kv-cache-dsv4.cpp \llama.cpp/src/llama-kv-cache-iswa.cpp \llama.cpp/src/llama-kv-cache-msa.cpp \llama.cpp/src/llama-kv-cache.cpp \llama.cpp/src/llama-memory-hybrid.cpp \+ llama.cpp/src/llama-memory-hybrid-idx.cpp \llama.cpp/src/llama-memory-hybrid-iswa.cpp \llama.cpp/src/llama-memory-recurrent.cpp \llama.cpp/src/llama-memory.cpp \@@ -255,12 +267,30 @@ COMMON_SRCS_CPP := \llama.cpp/common/jinja/string.cpp \llama.cpp/common/jinja/value.cpp \llama.cpp/common/json-schema-to-grammar.cpp \+ llama.cpp/common/json-schema.cpp \+ llama.cpp/common/json.cpp \llama.cpp/common/license.cpp \llama.cpp/common/llguidance.cpp \llama.cpp/common/log.cpp \llama.cpp/common/ngram-cache.cpp \llama.cpp/common/ngram-map.cpp \llama.cpp/common/ngram-mod.cpp \+ llama.cpp/common/parsers/parsers.cpp \+ llama.cpp/common/parsers/cohere2moe.cpp \+ llama.cpp/common/parsers/deepseek.cpp \+ llama.cpp/common/parsers/functionary-v3-2.cpp \+ llama.cpp/common/parsers/gemma4.cpp \+ llama.cpp/common/parsers/gigachat-v3.cpp \+ llama.cpp/common/parsers/gpt-oss.cpp \+ llama.cpp/common/parsers/kimi-k2.cpp \+ llama.cpp/common/parsers/kimi-k3.cpp \+ llama.cpp/common/parsers/lfm2.cpp \+ llama.cpp/common/parsers/ling3.cpp \+ llama.cpp/common/parsers/minicpm5.cpp \+ llama.cpp/common/parsers/minimax-m3.cpp \+ llama.cpp/common/parsers/ministral3.cpp \+ llama.cpp/common/parsers/muse-glimmer.cpp \+ llama.cpp/common/parsers/qwen3-coder.cpp \llama.cpp/common/peg-parser.cpp \llama.cpp/common/preset.cpp \llama.cpp/common/reasoning-budget.cpp \@@ -325,8 +355,10 @@ MTMD_SRCS_CPP := \llama.cpp/tools/mtmd/models/cogvlm.cpp \llama.cpp/tools/mtmd/models/deepseekocr.cpp \llama.cpp/tools/mtmd/models/deepseekocr2.cpp \+ llama.cpp/tools/mtmd/models/deepseek4v.cpp \llama.cpp/tools/mtmd/models/conformer.cpp \llama.cpp/tools/mtmd/models/dotsocr.cpp \+ llama.cpp/tools/mtmd/models/dots3note.cpp \llama.cpp/tools/mtmd/models/exaone4_5.cpp \llama.cpp/tools/mtmd/models/gemma4a.cpp \llama.cpp/tools/mtmd/models/gemma4ua.cpp \@@ -365,7 +397,29 @@ MTMD_SRCS_CPP := \llama.cpp/tools/mtmd/models/yasa2.cpp \llama.cpp/tools/mtmd/models/youtuvl.cpp-MTMD_OBJS := $(MTMD_SRCS_CPP:%.cpp=o/$(MODE)/%.cpp.o)+# Vendored hashing, linked into mtmd by upstream as vendor::hash. Since b11100+# mtmd-helper.cpp derives bitmap IDs with hash_sha256_hex() from it, which is+# the only entry point hash.h exposes and the only one anything here calls.+# Upstream's vendor/hash also builds xxhash and sha1; both are unreferenced, and+# sha1 would need compiling as C++ despite its .c extension (its declarations+# live in a namespace; upstream forces LANGUAGE CXX on it), so neither is built+# here. Adding a caller for them shows up as an undefined reference at link.+VENDOR_HASH_SRCS_C := \+ llama.cpp/vendor/hash/sha256/sha256.c++VENDOR_HASH_SRCS_CPP := \+ llama.cpp/vendor/hash/hash.cpp++VENDOR_HASH_OBJS := \+ $(VENDOR_HASH_SRCS_C:%.c=o/$(MODE)/%.c.o) \+ $(VENDOR_HASH_SRCS_CPP:%.cpp=o/$(MODE)/%.cpp.o)++MTMD_OBJS := \+ $(MTMD_SRCS_CPP:%.cpp=o/$(MODE)/%.cpp.o) \+ $(VENDOR_HASH_OBJS)++# sha256.c reaches for "rotate-bits/rotate-bits.h" next to it+$(VENDOR_HASH_OBJS): private CPPFLAGS += -iquote llama.cpp/vendor/hash# ==============================================================================# cpp-httplib (HTTP library for server)@@ -379,25 +433,29 @@ HTTPLIB_OBJS := $(HTTPLIB_SRCS:%.cpp=o/$(MODE)/%.cpp.o)# ==============================================================================## Upstream switched from prebuilt bundles in tools/server/public/ to a-# Svelte/PWA project under tools/ui/, embedded at CMake time by-# tools/ui/embed.cpp into a generated ui.cpp + ui.h. cosmocc has no JS-# toolchain, so apply-patches.sh (run by `make setup`) downloads the-# prebuilt site tarball (dist.tar.gz) from the ggml-org/llama-ui Hugging-# Face bucket and extracts the whole static site into-# llama.cpp/tools/ui/dist/ (see fetch-ui-assets.sh). embed.cpp then-# recursively embeds every file under that directory, keyed by its-# relative path (e.g. "_app/immutable/bundle.HASH.js"). fetch-ui-assets.sh-# also builds a dist/_gzip/ mirror of gzip-compressed files; embed.cpp-# auto-detects it and emits gzip-encoded assets (keeping the embedded-# payload small), which server-http.cpp serves with Content-Encoding: gzip.+# Svelte/PWA project under tools/ui/, embedded into a generated ui.cpp + ui.h+# at build time. cosmocc has no JS toolchain, so apply-patches.sh (run by+# `make setup`) downloads the prebuilt site tarball (dist.tar.gz) from the+# ggml-org/llama-ui Hugging Face bucket and extracts the whole static site into+# llama.cpp/tools/ui/dist/, plus a dist/_gzip/ mirror of gzip-compressed files+# (see fetch-ui-assets.sh).+#+# b11100 replaced upstream's standalone tools/ui/embed.cpp with+# scripts/ui-assets.cmake, which renders tools/ui/ui.{cpp,h}.in inside a CMake+# build. There is no CMake step here, so ui-embed.sh does that rendering — a+# translation of that script's emit_files(), the same way this file is a+# translation of llama.cpp's CMake build. It renders upstream's own templates,+# so the generated interface cannot drift from what server-http.cpp expects.+## With assets present, ui.h defines LLAMA_UI_HAS_ASSETS and server-http.cpp-# registers a route per asset (index.html at /); without them, embed.cpp-# emits a no-op llama_ui_find_asset and the UI routes stay unregistered.+# registers a route per asset (index.html at /), serving the gzip-encoded bytes+# with Content-Encoding: gzip; without them, the generated llama_ui_find_asset+# is a no-op and the UI routes stay unregistered.UI_DIST := llama.cpp/tools/ui/distUI_GEN_DIR := o/$(MODE)/llama.cpp/tools/ui-UI_EMBED_SRC := llama.cpp/tools/ui/embed.cpp-UI_EMBED_TOOL := $(UI_GEN_DIR)/llama-ui-embed+UI_EMBED_SH := llama.cpp.patches/ui-embed.sh+UI_TEMPLATES := llama.cpp/tools/ui/ui.cpp.in llama.cpp/tools/ui/ui.h.inUI_CPP_GEN := $(UI_GEN_DIR)/ui.cppUI_H_GEN := $(UI_GEN_DIR)/ui.h@@ -408,21 +466,14 @@ UI_H_GEN := $(UI_GEN_DIR)/ui.h# letting the build proceed UI-less (offline / asset build not yet published).UI_ASSETS_INDEX_HTML := $(wildcard $(UI_DIST)/index.html)-# Build embed.cpp standalone (no llamafile flags, no llama.cpp includes).-# cosmoc++ produces an APE that runs on the build host, so we don't need-# a separate system compiler. Compiled with stock C++17 (embed.cpp uses-# <filesystem>) so the source isn't entangled with -DCOSMOCC or GGML defines.-$(UI_EMBED_TOOL): $(UI_EMBED_SRC) $(COSMOCC)- @mkdir -p $(@D)- $(CXX) -O2 -std=gnu++17 -o $@ $<--# Generate ui.cpp/ui.h. Re-run when the embed tool or the fetched UI changes.-# When dist/ has assets, pass the directory: embed.cpp recurses it (auto-using-# dist/_gzip when present). When dist/ is empty, pass no directory so embed-# emits its no-asset stub.-$(UI_CPP_GEN) $(UI_H_GEN) &: $(UI_EMBED_TOOL) $(UI_ASSETS_INDEX_HTML)+# Generate ui.cpp/ui.h. Re-runs when the generator, upstream's templates or the+# fetched UI change. When dist/ has assets, pass the directory (the script picks+# up dist/_gzip itself); when it is empty, pass none so it emits the no-asset+# stub. The script rewrites its outputs only when their contents change, so an+# unchanged re-fetch does not cascade a rebuild.+$(UI_CPP_GEN) $(UI_H_GEN) &: $(UI_EMBED_SH) $(UI_TEMPLATES) $(UI_ASSETS_INDEX_HTML)@mkdir -p $(UI_GEN_DIR)- $(UI_EMBED_TOOL) $(UI_CPP_GEN) $(UI_H_GEN) \+ $(UI_EMBED_SH) $(UI_CPP_GEN) $(UI_H_GEN) \$(if $(UI_ASSETS_INDEX_HTML),$(UI_DIST))# ==============================================================================@@ -521,13 +572,9 @@ $(UI_GEN_OBJ): private CPPFLAGS += -iquote $(UI_GEN_DIR)$(UI_GEN_OBJ): $(UI_H_GEN)# Version definitions-$(GGML_OBJS): private CCFLAGS += \- -DGGML_VERSION=\"$(GGML_VERSION)\" \- -DGGML_COMMIT=\"$(GGML_COMMIT)\"--$(LLAMA_OBJS): private CCFLAGS += \- -DLLAMA_VERSION=\"$(LLAMA_VERSION)\" \- -DLLAMA_COMMIT=\"$(LLAMA_COMMIT)\"+# ggml/src/ggml.c and src/llama.cpp #include "ggml-version.h" / "llama-version.h",+# which CMake generates with configure_file(). apply-patches.sh writes them next+# to their sources from the same .in templates, so no -D flags are needed here.# Base flags for all objects$(LLAMA_CPP_OBJS) $(TOOL_SERVER_OBJS): private CCFLAGS += \diff --git a/llama.cpp.patches/patches/common_arg.cpp.patch b/llama.cpp.patches/patches/common_arg.cpp.patchindex 2e371ba7..a8c55ce1 100644--- a/llama.cpp.patches/patches/common_arg.cpp.patch+++ b/llama.cpp.patches/patches/common_arg.cpp.patch@@ -1,7 +1,7 @@diff --git a/common/arg.cpp b/common/arg.cpp--- a/llama.cpp/common/arg.cpp+++ b/llama.cpp/common/arg.cpp-@@ -42,6 +42,8 @@+@@ -41,6 +41,8 @@#ifndef __EMSCRIPTEN__#ifdef __linux__#include <linux/limits.h>diff --git a/llama.cpp.patches/patches/common_common.cpp.patch b/llama.cpp.patches/patches/common_common.cpp.patchindex 226ea76b..1b0565e0 100644--- a/llama.cpp.patches/patches/common_common.cpp.patch+++ b/llama.cpp.patches/patches/common_common.cpp.patch@@ -52,7 +52,7 @@ diff --git a/common/common.cpp b/common/common.cppint32_t logical_cpus = _system_configuration.ncpus;int32_t smt_threads = _system_configuration.smt_threads;if (smt_threads > 0) {-@@ -1058,6 +1093,20 @@ std::string fs_get_cache_directory() {+@@ -1059,6 +1094,20 @@ std::string fs_get_cache_directory() {if (cache_directory.empty()) {throw std::runtime_error("Failed to find %LOCALAPPDATA% directory");}@@ -73,7 +73,7 @@ diff --git a/common/common.cpp b/common/common.cpp#elif defined(__EMSCRIPTEN__)GGML_ABORT("not implemented on this platform");#else-@@ -1103,6 +1152,20 @@ std::string fs_get_config_directory() {+@@ -1104,6 +1153,20 @@ std::string fs_get_config_directory() {if (config_directory.empty()) {throw std::runtime_error("Failed to find %APPDATA% directory");}@@ -94,11 +94,10 @@ diff --git a/common/common.cpp b/common/common.cpp#elif defined(__EMSCRIPTEN__)// caller decides what to do when there is no config directorythrow std::runtime_error("not implemented on this platform");-@@ -1294,10 +1357,31 @@ common_init_result::common_init_result(common_params & params, bool model_only)- if (params.fit_params) {+@@ -1296,6 +1359,26 @@ common_init_result::common_init_result(common_params & params, bool model_only)COM_TRC("%s", "fitting params to device memory ...\n");COM_TRC("%s", "(for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)\n");-+++ // if a multimodal projection model is specified and will be loaded on GPU,+ // add its estimated size to the fit margins so the fit leaves enough room for it+ std::vector<size_t> fit_params_target = params.fit_params_target;@@ -119,11 +118,15 @@ diff --git a/common/common.cpp b/common/common.cpp+ }+ }++ // the draft context is created from the same base params and follows the main context, fit both together+ const bool has_draft = params.speculative.has_dft();+ const bool spec_mtp = std::find(params.speculative.types.begin(), params.speculative.types.end(),+@@ -1320,7 +1403,7 @@ common_init_result::common_init_result(common_params & params, bool model_only)common_fit_params(params.model.path.c_str(), &mparams, &cparams,params.tensor_split,params.tensor_buft_overrides.data(),- params.fit_params_target.data(),+ fit_params_target.data(),params.fit_params_min_ctx,+ has_draft || spec_mtp ? &extra : nullptr,params.verbosity >= LOG_LEVEL_DEBUG ? GGML_LOG_LEVEL_DEBUG : GGML_LOG_LEVEL_ERROR);- }diff --git a/llama.cpp.patches/patches/common_download.cpp.patch b/llama.cpp.patches/patches/common_download.cpp.patchindex f50ed915..a731fcb1 100644--- a/llama.cpp.patches/patches/common_download.cpp.patch+++ b/llama.cpp.patches/patches/common_download.cpp.patch@@ -1,7 +1,7 @@diff --git a/common/download.cpp b/common/download.cpp--- a/llama.cpp/common/download.cpp+++ b/llama.cpp/common/download.cpp-@@ -26,6 +26,8 @@+@@ -24,6 +24,8 @@#ifndef __EMSCRIPTEN__#ifdef __linux__#include <linux/limits.h>diff --git a/llama.cpp.patches/patches/common_log.cpp.patch b/llama.cpp.patches/patches/common_log.cpp.patchindex 5b7215cc..e80ea789 100644--- a/llama.cpp.patches/patches/common_log.cpp.patch+++ b/llama.cpp.patches/patches/common_log.cpp.patch@@ -1,7 +1,7 @@diff --git a/common/log.cpp b/common/log.cpp--- a/llama.cpp/common/log.cpp+++ b/llama.cpp/common/log.cpp-@@ -24,6 +24,7 @@+@@ -25,6 +25,7 @@# define fileno _fileno#else# include <unistd.h>@@ -9,7 +9,37 @@ diff --git a/common/log.cpp b/common/log.cpp#endif // defined(_WIN32)int common_log_verbosity_thold = LOG_DEFAULT_LLAMA;-@@ -265,9 +266,32 @@ public:+@@ -243,7 +244,13 @@ public:+ std::unique_lock<std::mutex> lock(mtx);++ // block if the queue is full+- cv_full.wait(lock, [this]() { return !running || !is_full(); });++ // Use wait_for() instead of an untimed wait to work around a++ // Cosmopolitan libc bug where untimed futex waits on XNU (macOS)++ // expire after ~72 minutes, causing condition_variable::wait() to++ // throw ETIMEDOUT.++ while (running && is_full()) {++ cv_full.wait_for(lock, std::chrono::seconds(30));++ }++ if (!running) {+ // discard messages while the worker thread is paused+@@ -310,7 +317,13 @@ public:+ std::unique_lock<std::mutex> lock(mtx);++ // block if the queue is full+- cv_full.wait(lock, [this]() { return !running || !is_full(); });++ // Use wait_for() instead of an untimed wait to work around a++ // Cosmopolitan libc bug where untimed futex waits on XNU (macOS)++ // expire after ~72 minutes, causing condition_variable::wait() to++ // throw ETIMEDOUT.++ while (running && is_full()) {++ cv_full.wait_for(lock, std::chrono::seconds(30));++ }++ if (!running) {+ // discard messages while the worker thread is paused+@@ -345,9 +358,32 @@ public:running = true;thrd = std::thread([this]() {diff --git a/llama.cpp.patches/patches/ggml_src_ggml-alloc.c.patch b/llama.cpp.patches/patches/ggml_src_ggml-alloc.c.patchnew file mode 100644index 00000000..05ca1a25--- /dev/null+++ b/llama.cpp.patches/patches/ggml_src_ggml-alloc.c.patch@@ -0,0 +1,67 @@+diff --git a/ggml/src/ggml-alloc.c b/ggml/src/ggml-alloc.c+--- a/llama.cpp/ggml/src/ggml-alloc.c++++ b/llama.cpp/ggml/src/ggml-alloc.c+@@ -1124,6 +1124,20 @@ static void free_buffers(ggml_backend_buffer_t ** buffers, const size_t * n_buff+ free(*buffers);+ }+++static bool alloc_ctx_tensors_finalize_views(struct ggml_context * ctx) {++ for (struct ggml_tensor * t = ggml_get_first_tensor(ctx); t != NULL; t = ggml_get_next_tensor(ctx, t)) {++ if (t->view_src != NULL && t->buffer == NULL) {++ enum ggml_status status = ggml_backend_view_init(t);++ if (status != GGML_STATUS_SUCCESS) {++ GGML_LOG_ERROR("%s: failed to initialize view tensor %s\n", __func__, t->name);++ return false;++ }++ }++ }++++ return true;++}+++ static bool alloc_tensor_range(struct ggml_context * ctx,+ struct ggml_tensor * first, struct ggml_tensor * last,+ ggml_backend_buffer_type_t buft, size_t size,+@@ -1142,24 +1156,14 @@ static bool alloc_tensor_range(struct ggml_context * ctx,+ struct ggml_tallocr tallocr = ggml_tallocr_new(buffer);++ for (struct ggml_tensor * t = first; t != last; t = ggml_get_next_tensor(ctx, t)) {+- enum ggml_status status = GGML_STATUS_SUCCESS;+- if (t->data == NULL) {+- if (t->view_src == NULL) {+- status = ggml_tallocr_alloc(&tallocr, t);+- } else if (t->buffer == NULL) {+- status = ggml_backend_view_init(t);+- }+- } else {+- if (t->view_src != NULL && t->buffer == NULL) {+- // view of a pre-allocated tensor+- status = ggml_backend_view_init(t);++ if (t->view_src == NULL && t->data == NULL) {++ enum ggml_status status = ggml_tallocr_alloc(&tallocr, t);++ if (status != GGML_STATUS_SUCCESS) {++ GGML_LOG_ERROR("%s: failed to allocate tensor %s\n", __func__, t->name);++ free_buffers(buffers, n_buffers);++ return false;+ }+ }+- if (status != GGML_STATUS_SUCCESS) {+- GGML_LOG_ERROR("%s: failed to initialize tensor %s\n", __func__, t->name);+- free_buffers(buffers, n_buffers);+- return false;+- }+ }++ return true;+@@ -1217,6 +1221,11 @@ static ggml_backend_buffer_t ggml_backend_alloc_ctx_tensors_from_buft_impl(+ return NULL;+ }+++ if (!alloc_ctx_tensors_finalize_views(ctx)) {++ free_buffers(&buffers, &n_buffers);++ return NULL;++ }+++ ggml_backend_buffer_t buffer;+ if (n_buffers == 1) {+ buffer = buffers[0];diff --git a/llama.cpp.patches/patches/ggml_src_ggml-backend-impl.h.patch b/llama.cpp.patches/patches/ggml_src_ggml-backend-impl.h.patchindex 3aaec764..138340bf 100644--- a/llama.cpp.patches/patches/ggml_src_ggml-backend-impl.h.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml-backend-impl.h.patch@@ -25,7 +25,7 @@ diff --git a/ggml/src/ggml-backend-impl.h b/ggml/src/ggml-backend-impl.h};struct ggml_backend_buffer_type {-@@ -39,26 +39,30 @@ extern "C" {+@@ -44,26 +44,30 @@ extern "C" {//struct ggml_backend_buffer_i {@@ -68,8 +68,14 @@ diff --git a/ggml/src/ggml-backend-impl.h b/ggml/src/ggml-backend-impl.h};struct ggml_backend_buffer {-@@ -103,40 +107,40 @@ extern "C" {- //+@@ -114,45 +118,45 @@ extern "C" {+ struct ggml_backend_graph_optimize_params {+ // keep `tensor` allocated at least until `until` (a node of the same graph) has been computed+ // can be called multiple times for the same tensor: the longest lifetime applies+- void (*add_alloc_dep)(void * user_data, struct ggml_tensor * tensor, struct ggml_tensor * until);++ void (GGML_CALL *add_alloc_dep)(void * user_data, struct ggml_tensor * tensor, struct ggml_tensor * until);+ void * user_data;+ };struct ggml_backend_i {- const char * (*get_name)(ggml_backend_t backend);@@ -120,12 +126,12 @@ diff --git a/ggml/src/ggml-backend-impl.h b/ggml/src/ggml-backend-impl.h+ void (GGML_CALL *event_wait) (ggml_backend_t backend, ggml_backend_event_t event);// (optional) sort/optimize the nodes in the graph-- void (*graph_optimize) (ggml_backend_t backend, struct ggml_cgraph * cgraph);-+ void (GGML_CALL *graph_optimize) (ggml_backend_t backend, struct ggml_cgraph * cgraph);+- void (*graph_optimize) (ggml_backend_t backend, struct ggml_cgraph * cgraph, struct ggml_backend_graph_optimize_params * params);++ void (GGML_CALL *graph_optimize) (ggml_backend_t backend, struct ggml_cgraph * cgraph, struct ggml_backend_graph_optimize_params * params);};struct ggml_backend {-@@ -159,46 +163,46 @@ extern "C" {+@@ -175,46 +179,46 @@ extern "C" {// the current functions to obtain the properties can remain, since they are more convenient for often used propertiesstruct ggml_backend_device_i {// device name: short identifier for this device, such as "CPU" or "CUDA0"@@ -187,7 +193,7 @@ diff --git a/ggml/src/ggml-backend-impl.h b/ggml/src/ggml-backend-impl.h};struct ggml_backend_device {-@@ -212,15 +216,15 @@ extern "C" {+@@ -228,15 +232,15 @@ extern "C" {//struct ggml_backend_reg_i {diff --git a/llama.cpp.patches/patches/ggml_src_ggml-backend-meta.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-backend-meta.cpp.patchindex 85503cea..df78814f 100644--- a/llama.cpp.patches/patches/ggml_src_ggml-backend-meta.cpp.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml-backend-meta.cpp.patch@@ -141,7 +141,7 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cppGGML_ASSERT(ggml_backend_buffer_is_meta(buffer));ggml_backend_meta_buffer_context * buf_ctx = (ggml_backend_meta_buffer_context *) buffer->context;delete buf_ctx;-@@ -1123,7 +1123,7 @@ static struct ggml_backend_meta_split_state ggml_backend_meta_get_split_state(co+@@ -1172,7 +1172,7 @@ static struct ggml_backend_meta_split_state ggml_backend_meta_get_split_state(coreturn ggml_backend_meta_get_split_state(buf_ctx->get_simple_tensor_container(tensor), tensor, assume_sync);}@@ -150,7 +150,7 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cppGGML_UNUSED(buffer);return (void *) 0x1000000000000000; // FIXME}-@@ -1248,14 +1248,14 @@ static enum ggml_status ggml_backend_meta_buffer_init_tensor_impl(ggml_backend_m+@@ -1304,14 +1304,14 @@ static enum ggml_status ggml_backend_meta_buffer_init_tensor_impl(ggml_backend_mreturn GGML_STATUS_SUCCESS;}@@ -162,12 +162,21 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cppreturn ggml_backend_meta_buffer_init_tensor_impl(buf_ctx->get_simple_tensor_container(tensor), tensor);}+-static void ggml_backend_meta_buffer_memset_tensor(++static void GGML_CALL ggml_backend_meta_buffer_memset_tensor(+ ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size) {+ const size_t n_bufs = ggml_backend_meta_buffer_n_bufs(buffer);+ const ggml_backend_meta_split_state split_state =+@@ -1413,7 +1413,7 @@ static void ggml_backend_meta_buffer_memset_tensor(+ }+ }+-static void ggml_backend_meta_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) {+static void GGML_CALL ggml_backend_meta_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) {const size_t n_bufs = ggml_backend_meta_buffer_n_bufs(buffer);const ggml_backend_meta_split_state split_state = ggml_backend_meta_get_split_state(tensor, /*assume_sync =*/ false);GGML_ASSERT(ggml_is_contiguous(tensor) || split_state.axis == GGML_BACKEND_SPLIT_AXIS_MIRRORED);-@@ -1369,7 +1369,7 @@ static void ggml_backend_meta_buffer_set_tensor(ggml_backend_buffer_t buffer, gg+@@ -1541,7 +1541,7 @@ static void ggml_backend_meta_buffer_set_tensor(ggml_backend_buffer_t buffer, gg}}@@ -176,7 +185,7 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cppconst size_t n_bufs = ggml_backend_meta_buffer_n_bufs(buffer);const ggml_backend_meta_split_state split_state = ggml_backend_meta_get_split_state(tensor, /*assume_sync =*/ false);GGML_ASSERT(ggml_is_contiguous(tensor) || split_state.axis == GGML_BACKEND_SPLIT_AXIS_MIRRORED);-@@ -1469,14 +1469,14 @@ static void ggml_backend_meta_buffer_get_tensor(ggml_backend_buffer_t buffer, co+@@ -1641,14 +1641,14 @@ static void ggml_backend_meta_buffer_get_tensor(ggml_backend_buffer_t buffer, co}}@@ -193,8 +202,8 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cppGGML_ASSERT(ggml_backend_buffer_is_meta(buffer));ggml_backend_meta_buffer_context * buf_ctx = (ggml_backend_meta_buffer_context *) buffer->context;for (size_t i = 0; i < buf_ctx->bufs.size(); i++) {-@@ -1502,7 +1502,7 @@ bool ggml_backend_buffer_is_meta(ggml_backend_buffer_t buf) {- return buf != nullptr && buf->iface.free_buffer == ggml_backend_meta_buffer_iface.free_buffer;+@@ -1684,7 +1684,7 @@ void ggml_backend_meta_buffer_set_usage(ggml_backend_buffer_t buffer, enum ggml_+ }}-static ggml_backend_buffer_t ggml_backend_meta_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) {@@ -202,7 +211,7 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cppconst size_t n_simple_bufts = ggml_backend_meta_buft_n_bufts(buft);const ggml_init_params params = {-@@ -1669,20 +1669,20 @@ struct ggml_backend_meta_context {+@@ -1851,20 +1851,20 @@ struct ggml_backend_meta_context {}};@@ -226,7 +235,7 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cppconst size_t n_backends = ggml_backend_meta_n_backends(backend);GGML_ASSERT(offset == 0);GGML_ASSERT(ggml_is_contiguous(tensor));-@@ -1727,7 +1727,7 @@ static void ggml_backend_meta_set_tensor_async(ggml_backend_t backend, ggml_tens+@@ -1909,7 +1909,7 @@ static void ggml_backend_meta_set_tensor_async(ggml_backend_t backend, ggml_tens}}@@ -235,7 +244,7 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cppconst size_t n_backends = ggml_backend_meta_n_backends(backend);GGML_ASSERT(offset == 0);GGML_ASSERT(ggml_is_contiguous(tensor));-@@ -1772,14 +1772,14 @@ static void ggml_backend_meta_get_tensor_async(ggml_backend_t backend, const ggm+@@ -1954,14 +1954,14 @@ static void ggml_backend_meta_get_tensor_async(ggml_backend_t backend, const ggm}}@@ -252,7 +261,7 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cppGGML_ASSERT(cgraph->grads == nullptr);const size_t n_backends = ggml_backend_meta_n_backends(backend);ggml_backend_meta_context * backend_ctx = (ggml_backend_meta_context *) backend->context;-@@ -2249,7 +2249,7 @@ bool ggml_backend_is_meta(ggml_backend_t backend) {+@@ -2487,7 +2487,7 @@ bool ggml_backend_is_meta(ggml_backend_t backend) {return backend != nullptr && backend->iface.get_name == ggml_backend_meta_i.get_name;}diff --git a/llama.cpp.patches/patches/ggml_src_ggml-backend-reg.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-backend-reg.cpp.patchindex 6286b24e..63af2d06 100644--- a/llama.cpp.patches/patches/ggml_src_ggml-backend-reg.cpp.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml-backend-reg.cpp.patch@@ -1,7 +1,7 @@diff --git a/ggml/src/ggml-backend-reg.cpp b/ggml/src/ggml-backend-reg.cpp--- a/llama.cpp/ggml/src/ggml-backend-reg.cpp+++ b/llama.cpp/ggml/src/ggml-backend-reg.cpp-@@ -504,7 +504,7 @@ static ggml_backend_reg_t ggml_backend_load_best(const char * name, bool silent,+@@ -510,7 +510,7 @@ static ggml_backend_reg_t ggml_backend_load_best(const char * name, bool silent,if (ec) {GGML_LOG_DEBUG("%s: posix_stat(%s) failure, error-message: %s\n", __func__, path_str(search_path).c_str(), ec.message().c_str());} else {diff --git a/llama.cpp.patches/patches/ggml_src_ggml-backend.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-backend.cpp.patchindex 5b1f9bfa..4337f9fa 100644--- a/llama.cpp.patches/patches/ggml_src_ggml-backend.cpp.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml-backend.cpp.patch@@ -1,7 +1,7 @@diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp--- a/llama.cpp/ggml/src/ggml-backend.cpp+++ b/llama.cpp/ggml/src/ggml-backend.cpp-@@ -112,7 +112,14 @@ void ggml_backend_buffer_free(ggml_backend_buffer_t buffer) {+@@ -121,7 +121,14 @@ void ggml_backend_buffer_free(ggml_backend_buffer_t buffer) {if (buffer->iface.free_buffer != NULL) {buffer->iface.free_buffer(buffer);}@@ -17,7 +17,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp}size_t ggml_backend_buffer_get_size(ggml_backend_buffer_t buffer) {-@@ -671,7 +678,7 @@ struct ggml_backend_multi_buffer_context {+@@ -682,7 +689,7 @@ struct ggml_backend_multi_buffer_context {size_t n_buffers;};@@ -26,7 +26,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cppGGML_ASSERT(buffer);ggml_backend_multi_buffer_context * ctx = (ggml_backend_multi_buffer_context *) buffer->context;for (size_t i = 0; i < ctx->n_buffers; i++) {-@@ -682,7 +689,7 @@ static void ggml_backend_multi_buffer_free_buffer(ggml_backend_buffer_t buffer)+@@ -693,7 +700,7 @@ static void ggml_backend_multi_buffer_free_buffer(ggml_backend_buffer_t buffer)free(ctx);}@@ -35,7 +35,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cppGGML_ASSERT(buffer);ggml_backend_multi_buffer_context * ctx = (ggml_backend_multi_buffer_context *) buffer->context;for (size_t i = 0; i < ctx->n_buffers; i++) {-@@ -702,6 +709,7 @@ static const struct ggml_backend_buffer_i ggml_backend_multi_buffer_i = {+@@ -713,6 +720,7 @@ static const struct ggml_backend_buffer_i ggml_backend_multi_buffer_i = {/* .cpy_tensor = */ NULL,/* .clear = */ ggml_backend_multi_buffer_clear,/* .reset = */ NULL,@@ -43,7 +43,41 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp};ggml_backend_buffer_t ggml_backend_multi_buffer_alloc_buffer(ggml_backend_buffer_t * buffers, size_t n_buffers) {-@@ -2270,7 +2278,7 @@ bool ggml_backend_compare_graph_backend(ggml_backend_t backend1, ggml_backend_t+@@ -1063,6 +1071,18 @@ static void ggml_backend_sched_set_if_supported(ggml_backend_sched_t sched, stru+ }++ // assigns backends to ops and splits the graph into subgraphs that can be computed on the same backend++// [jart] The backends invoke this through ggml_backend_graph_optimize_params,++// and they live in dynamically loaded modules built with the host ABI, so the++// callback has to be GGML_CALL. A lambda cannot carry the attribute, so this is++// a named function rather than the lambda upstream uses.++static void GGML_CALL ggml_backend_sched_add_alloc_dep(void * user_data, ggml_tensor * tensor, ggml_tensor * until) {++ auto & deps = *(std::unordered_map<ggml_tensor *, std::vector<ggml_tensor *>> *) user_data;++ std::vector<ggml_tensor *> & keep = deps[until];++ if (std::find(keep.begin(), keep.end(), tensor) == keep.end()) {++ keep.push_back(tensor);++ }++}+++ void ggml_backend_sched_split_graph(ggml_backend_sched_t sched, struct ggml_cgraph * graph) {+ // reset splits+ sched->n_splits = 0;+@@ -1445,13 +1465,7 @@ void ggml_backend_sched_split_graph(ggml_backend_sched_t sched, struct ggml_cgra+ std::unordered_map<ggml_tensor *, std::vector<ggml_tensor *>> alloc_deps;++ struct ggml_backend_graph_optimize_params opt_params = {+- /* .add_alloc_dep = */ [](void * user_data, ggml_tensor * tensor, ggml_tensor * until) {+- auto & deps = *(std::unordered_map<ggml_tensor *, std::vector<ggml_tensor *>> *) user_data;+- std::vector<ggml_tensor *> & keep = deps[until];+- if (std::find(keep.begin(), keep.end(), tensor) == keep.end()) {+- keep.push_back(tensor);+- }+- },++ /* .add_alloc_dep = */ ggml_backend_sched_add_alloc_dep,+ /* .user_data = */ &alloc_deps,+ };++@@ -2352,7 +2366,7 @@ bool ggml_backend_compare_graph_backend(ggml_backend_t backend1, ggml_backend_t// CPU backend - buffer@@ -52,7 +86,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cppGGML_ASSERT(buffer);uintptr_t data = (uintptr_t)buffer->context;-@@ -2282,33 +2290,33 @@ static void * ggml_backend_cpu_buffer_get_base(ggml_backend_buffer_t buffer) {+@@ -2364,33 +2378,33 @@ static void * ggml_backend_cpu_buffer_get_base(ggml_backend_buffer_t buffer) {return (void *)data;}@@ -91,7 +125,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cppGGML_ASSERT(src);if (ggml_backend_buffer_is_host(src->buffer)) {memcpy(dst->data, src->data, ggml_nbytes(src));-@@ -2319,7 +2327,7 @@ static bool ggml_backend_cpu_buffer_cpy_tensor(ggml_backend_buffer_t buffer, con+@@ -2401,7 +2415,7 @@ static bool ggml_backend_cpu_buffer_cpy_tensor(ggml_backend_buffer_t buffer, conGGML_UNUSED(buffer);}@@ -100,7 +134,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cppGGML_ASSERT(buffer);memset(buffer->context, value, buffer->size);}-@@ -2336,6 +2344,7 @@ static const struct ggml_backend_buffer_i ggml_backend_cpu_buffer_i = {+@@ -2418,6 +2432,7 @@ static const struct ggml_backend_buffer_i ggml_backend_cpu_buffer_i = {/* .cpy_tensor = */ ggml_backend_cpu_buffer_cpy_tensor,/* .clear = */ ggml_backend_cpu_buffer_clear,/* .reset = */ NULL,@@ -108,7 +142,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp};static const struct ggml_backend_buffer_i ggml_backend_cpu_buffer_from_ptr_i = {-@@ -2350,19 +2359,20 @@ static const struct ggml_backend_buffer_i ggml_backend_cpu_buffer_from_ptr_i = {+@@ -2432,19 +2447,20 @@ static const struct ggml_backend_buffer_i ggml_backend_cpu_buffer_from_ptr_i = {/* .cpy_tensor = */ ggml_backend_cpu_buffer_cpy_tensor,/* .clear = */ ggml_backend_cpu_buffer_clear,/* .reset = */ NULL,@@ -131,7 +165,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cppvoid * data = ggml_aligned_malloc(size);if (data == NULL) {-@@ -2373,13 +2383,13 @@ static ggml_backend_buffer_t ggml_backend_cpu_buffer_type_alloc_buffer(ggml_back+@@ -2455,13 +2471,13 @@ static ggml_backend_buffer_t ggml_backend_cpu_buffer_type_alloc_buffer(ggml_backreturn ggml_backend_buffer_init(buft, ggml_backend_cpu_buffer_i, data, size);}@@ -147,7 +181,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cppreturn true;GGML_UNUSED(buft);-@@ -2402,7 +2412,7 @@ ggml_backend_buffer_type_t ggml_backend_cpu_buffer_type(void) {+@@ -2484,7 +2500,7 @@ ggml_backend_buffer_type_t ggml_backend_cpu_buffer_type(void) {return &ggml_backend_cpu_buffer_type;}diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.c.patch b/llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.c.patchindex 370aedc3..d388f878 100644--- a/llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.c.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.c.patch@@ -1,7 +1,7 @@diff --git a/ggml/src/ggml-cpu/ggml-cpu.c b/ggml/src/ggml-cpu/ggml-cpu.c--- a/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c+++ b/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c-@@ -1539,6 +1539,26 @@ static void ggml_compute_forward_mul_mat_id(+@@ -1555,6 +1555,26 @@ static void ggml_compute_forward_mul_mat_id(const struct ggml_tensor * src1 = dst->src[1];const struct ggml_tensor * ids = dst->src[2];@@ -28,7 +28,7 @@ diff --git a/ggml/src/ggml-cpu/ggml-cpu.c b/ggml/src/ggml-cpu/ggml-cpu.cGGML_TENSOR_BINARY_OP_LOCALSconst int ith = params->ith;-@@ -1658,6 +1678,24 @@ static void ggml_compute_forward_mul_mat_id(+@@ -1691,6 +1711,24 @@ static void ggml_compute_forward_mul_mat_id(const int64_t nr0 = ne01;const int64_t nr1 = cne1;@@ -53,10 +53,10 @@ diff --git a/ggml/src/ggml-cpu/ggml-cpu.c b/ggml/src/ggml-cpu/ggml-cpu.cint chunk_size = 16;if (nr0 == 1 || nr1 == 1) {chunk_size = 64;-@@ -2876,6 +2914,19 @@ struct ggml_cplan ggml_graph_plan(- cur += n_as*ids->ne[0]*ids->ne[1]*sizeof(struct mmid_row_mapping) + sizeof(int64_t);- // atomic_current_chunk- cur += CACHE_LINE_SIZE*n_as + CACHE_LINE_SIZE;+@@ -2919,6 +2957,19 @@ struct ggml_cplan ggml_graph_plan(+ if (ggml_cpu_iqp_supports_mul_mat_id(node)) {+ cur += n_tasks * ggml_cpu_iqp_scratch_size(node) + 64;+ }+#if GGML_USE_LLAMAFILE+ // [jart] reserve work-buffer space for llamafile_mixmul.+ // Without this, MixMul::allocate_shared_memory() indiff --git a/llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.cpp.patchindex 4993e680..fc7397ea 100644--- a/llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.cpp.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.cpp.patch@@ -151,7 +151,7 @@ diff --git a/ggml/src/ggml-cpu/ggml-cpu.cpp b/ggml/src/ggml-cpu/ggml-cpu.cppconst struct ggml_tensor * src0 = op->src[0];const struct ggml_tensor * src1 = op->src[1];-@@ -479,7 +479,7 @@ static bool ggml_backend_cpu_device_supports_op(ggml_backend_dev_t dev, const st+@@ -483,7 +483,7 @@ static bool ggml_backend_cpu_device_supports_op(ggml_backend_dev_t dev, const st}}@@ -160,7 +160,7 @@ diff --git a/ggml/src/ggml-cpu/ggml-cpu.cpp b/ggml/src/ggml-cpu/ggml-cpu.cppreturn ggml_backend_buft_is_host(buft) || ggml_backend_cpu_is_extra_buffer_type(buft);GGML_UNUSED(dev);}-@@ -504,19 +504,19 @@ static const struct ggml_backend_device_i ggml_backend_cpu_device_i = {+@@ -508,19 +508,19 @@ static const struct ggml_backend_device_i ggml_backend_cpu_device_i = {// CPU backend - backend (reg)@@ -183,7 +183,7 @@ diff --git a/ggml/src/ggml-cpu/ggml-cpu.cpp b/ggml/src/ggml-cpu/ggml-cpu.cppGGML_ASSERT(index == 0);static ggml_backend_cpu_device_context ctx;-@@ -531,7 +531,7 @@ static ggml_backend_dev_t ggml_backend_cpu_reg_get_device(ggml_backend_reg_t reg+@@ -535,7 +535,7 @@ static ggml_backend_dev_t ggml_backend_cpu_reg_get_device(ggml_backend_reg_t reg// This is intended to replace the the ggml_cpu_has_* functions when loading the CPU backend dynamically,// and additionally to allow other backends to expose their own list of features that applications can query using the same API@@ -192,7 +192,7 @@ diff --git a/ggml/src/ggml-cpu/ggml-cpu.cpp b/ggml/src/ggml-cpu/ggml-cpu.cppstatic std::vector<ggml_backend_feature> features = []() {ggml_cpu_init();-@@ -648,7 +648,7 @@ static ggml_backend_feature * ggml_backend_cpu_get_features(ggml_backend_reg_t r+@@ -652,7 +652,7 @@ static ggml_backend_feature * ggml_backend_cpu_get_features(ggml_backend_reg_t rGGML_UNUSED(reg);}diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cpu_ops.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-cpu_ops.cpp.patchindex 2d5db121..b9208666 100644--- a/llama.cpp.patches/patches/ggml_src_ggml-cpu_ops.cpp.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml-cpu_ops.cpp.patch@@ -20,7 +20,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cpp// ggml_compute_forward_dupstatic void ggml_compute_forward_dup_same_cont(-@@ -8547,6 +8559,15 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(+@@ -8693,6 +8705,15 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(GGML_ASSERT(( q_to_vec_dot) && "fattn: unsupported K-type");GGML_ASSERT((v->type == GGML_TYPE_F32 || v_to_float ) && "fattn: unsupported V-type");@@ -36,7 +36,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cppint ith = params->ith;for (int ir = ir0; ir < ir1; ++ir) {-@@ -8566,10 +8587,22 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(+@@ -8712,10 +8733,22 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(ggml_fp16_t * VKQ16 = (ggml_fp16_t *) (VKQ32 + 1*DV); // (temporary) FP16 VKQ accumulatorggml_fp16_t * Q_q = (ggml_fp16_t *) (VKQ32 + 2*DV); // (temporary) buffer for Q converted to quantized/FP16@@ -62,7 +62,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cpp}const ggml_fp16_t * mp = mask ? (ggml_fp16_t *)((char *) mask->data + iq1*mask->nb[1] + (iq2%mask->ne[2])*mask->nb[2] + (iq3%mask->ne[3])*mask->nb[3]) : NULL;-@@ -8598,7 +8631,13 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(+@@ -8744,7 +8777,13 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(float s; // KQ valueconst char * k_data = (const char *) k->data + ( ic*nbk1 + ik2*nbk2 + ik3*nbk3);@@ -76,7 +76,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cpps = s*scale; // scale KQ value-@@ -8615,21 +8654,36 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(+@@ -8761,21 +8800,36 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(const char * v_data = ((const char *) v->data + (ic*nbv1 + iv2*nbv2 + iv3*nbv3));@@ -116,7 +116,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cpp} else {if (s > M) {// s is new maximum, ms < 1.0f, vs == expf(s - s) == 1.0f-@@ -8637,26 +8691,20 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(+@@ -8783,26 +8837,20 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(ms = expf(Mold - M);// V = V*expf(Mold - M)@@ -146,7 +146,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cppfor (int64_t d = 0; d < DV; ++d) {VKQ32[d] = GGML_CPU_FP16_TO_FP32(VKQ16[d]);}-@@ -8877,8 +8925,24 @@ static void ggml_compute_forward_flash_attn_ext_tiled(+@@ -9023,8 +9071,24 @@ static void ggml_compute_forward_flash_attn_ext_tiled(const char * k_data = (const char *)k->data + (ic + tk)*nbk1 + ik2*nbk2 + ik3*nbk3;if (kv_type == GGML_TYPE_F16) {const ggml_fp16_t * k_f16 = (const ggml_fp16_t *)k_data;@@ -173,7 +173,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cpp}} else {const float * k_f32_src = (const float *)k_data;-@@ -8888,7 +8952,18 @@ static void ggml_compute_forward_flash_attn_ext_tiled(+@@ -9034,7 +9098,18 @@ static void ggml_compute_forward_flash_attn_ext_tiled(}}memset(KQ, 0, Q_TILE_SZ * KV_TILE_SZ * sizeof(float));@@ -192,7 +192,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cppggml_vec_scale_f32(Q_TILE_SZ * KV_TILE_SZ, KQ, scale);// Set padded KQ entries to -inf so softmax gives them zero weight-@@ -8941,7 +9016,15 @@ static void ggml_compute_forward_flash_attn_ext_tiled(+@@ -9087,7 +9162,15 @@ static void ggml_compute_forward_flash_attn_ext_tiled(for (int tk = 0; tk < kv_tile; tk++) {const char * v_data = (const char *)v->data + (ic + tk)*nbv1 + iv2*nbv2 + iv3*nbv3;if (kv_type == GGML_TYPE_F16) {@@ -208,7 +208,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cpp} else {memcpy(V32 + tk * DV, v_data, DV * sizeof(float));}-@@ -8951,7 +9034,16 @@ static void ggml_compute_forward_flash_attn_ext_tiled(+@@ -9097,7 +9180,16 @@ static void ggml_compute_forward_flash_attn_ext_tiled(memset(KQ + tq * KV_TILE_SZ, 0, KV_TILE_SZ * sizeof(float));}}diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cpu_repack.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-cpu_repack.cpp.patchindex ae8d8d00..c067ae03 100644--- a/llama.cpp.patches/patches/ggml_src_ggml-cpu_repack.cpp.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml-cpu_repack.cpp.patch@@ -1,7 +1,7 @@diff --git a/ggml/src/ggml-cpu/repack.cpp b/ggml/src/ggml-cpu/repack.cpp--- a/llama.cpp/ggml/src/ggml-cpu/repack.cpp+++ b/llama.cpp/ggml/src/ggml-cpu/repack.cpp-@@ -4723,14 +4723,14 @@ static const ggml::cpu::tensor_traits * ggml_repack_get_optimal_repack_type(cons+@@ -5140,14 +5140,14 @@ static const ggml::cpu::tensor_traits * ggml_repack_get_optimal_repack_type(consreturn nullptr;}@@ -18,7 +18,7 @@ diff --git a/ggml/src/ggml-cpu/repack.cpp b/ggml/src/ggml-cpu/repack.cppconst void * data, size_t offset, size_t size) {GGML_ASSERT(offset == 0);GGML_ASSERT(size == ggml_nbytes(tensor));-@@ -4742,13 +4742,13 @@ static void ggml_backend_cpu_repack_buffer_set_tensor(ggml_backend_buffer_t buff+@@ -5159,13 +5159,13 @@ static void ggml_backend_cpu_repack_buffer_set_tensor(ggml_backend_buffer_t buffGGML_UNUSED(buffer);}@@ -34,7 +34,7 @@ diff --git a/ggml/src/ggml-cpu/repack.cpp b/ggml/src/ggml-cpu/repack.cppggml_backend_buffer_t buffer = ggml_backend_buft_alloc_buffer(ggml_backend_cpu_buffer_type(), size);if (buffer == nullptr) {-@@ -4763,7 +4763,7 @@ static ggml_backend_buffer_t ggml_backend_cpu_repack_buffer_type_alloc_buffer(gg+@@ -5180,7 +5180,7 @@ static ggml_backend_buffer_t ggml_backend_cpu_repack_buffer_type_alloc_buffer(ggreturn buffer;}diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cuda_common.cuh.patch b/llama.cpp.patches/patches/ggml_src_ggml-cuda_common.cuh.patchindex c80a874f..8a2e8a33 100644--- a/llama.cpp.patches/patches/ggml_src_ggml-cuda_common.cuh.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml-cuda_common.cuh.patch@@ -1,7 +1,7 @@diff --git a/ggml/src/ggml-cuda/common.cuh b/ggml/src/ggml-cuda/common.cuh--- a/llama.cpp/ggml/src/ggml-cuda/common.cuh+++ b/llama.cpp/ggml/src/ggml-cuda/common.cuh-@@ -320,9 +320,15 @@ static bool fp16_mma_hardware_available(const int cc) {+@@ -336,9 +336,15 @@ static bool fast_bf16_hardware_available(const int cc) {}static bool bf16_mma_hardware_available(const int cc) {diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cuda_convert.cu.patch b/llama.cpp.patches/patches/ggml_src_ggml-cuda_convert.cu.patchindex c945f000..0a13aeef 100644--- a/llama.cpp.patches/patches/ggml_src_ggml-cuda_convert.cu.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml-cuda_convert.cu.patch@@ -1,7 +1,7 @@diff --git a/ggml/src/ggml-cuda/convert.cu b/ggml/src/ggml-cuda/convert.cu--- a/llama.cpp/ggml/src/ggml-cuda/convert.cu+++ b/llama.cpp/ggml/src/ggml-cuda/convert.cu-@@ -481,6 +481,7 @@ to_bf16_cuda_t ggml_get_to_bf16_cuda(ggml_type type) {+@@ -513,6 +513,7 @@ to_bf16_cuda_t ggml_get_to_bf16_cuda(ggml_type type) {return dequantize_row_q5_K_cuda;case GGML_TYPE_Q6_K:return dequantize_row_q6_K_cuda;@@ -9,7 +9,7 @@ diff --git a/ggml/src/ggml-cuda/convert.cu b/ggml/src/ggml-cuda/convert.cucase GGML_TYPE_IQ2_XXS:return dequantize_row_iq2_xxs_cuda;case GGML_TYPE_IQ2_XS:-@@ -499,6 +500,7 @@ to_bf16_cuda_t ggml_get_to_bf16_cuda(ggml_type type) {+@@ -531,6 +532,7 @@ to_bf16_cuda_t ggml_get_to_bf16_cuda(ggml_type type) {return dequantize_row_iq4_xs_cuda;case GGML_TYPE_IQ3_S:return dequantize_row_iq3_s_cuda;@@ -17,7 +17,7 @@ diff --git a/ggml/src/ggml-cuda/convert.cu b/ggml/src/ggml-cuda/convert.cucase GGML_TYPE_MXFP4:return dequantize_row_mxfp4_cuda;case GGML_TYPE_NVFP4:-@@ -541,6 +543,7 @@ to_fp16_cuda_t ggml_get_to_fp16_cuda(ggml_type type) {+@@ -573,6 +575,7 @@ to_fp16_cuda_t ggml_get_to_fp16_cuda(ggml_type type) {return dequantize_row_q5_K_cuda;case GGML_TYPE_Q6_K:return dequantize_row_q6_K_cuda;@@ -25,7 +25,7 @@ diff --git a/ggml/src/ggml-cuda/convert.cu b/ggml/src/ggml-cuda/convert.cucase GGML_TYPE_IQ2_XXS:return dequantize_row_iq2_xxs_cuda;case GGML_TYPE_IQ2_XS:-@@ -559,6 +562,7 @@ to_fp16_cuda_t ggml_get_to_fp16_cuda(ggml_type type) {+@@ -591,6 +594,7 @@ to_fp16_cuda_t ggml_get_to_fp16_cuda(ggml_type type) {return dequantize_row_iq4_xs_cuda;case GGML_TYPE_IQ3_S:return dequantize_row_iq3_s_cuda;diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cuda_ggml-cuda.cu.patch b/llama.cpp.patches/patches/ggml_src_ggml-cuda_ggml-cuda.cu.patchindex acef87f6..72ec44fb 100644--- a/llama.cpp.patches/patches/ggml_src_ggml-cuda_ggml-cuda.cu.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml-cuda_ggml-cuda.cu.patch@@ -1,7 +1,7 @@diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu--- a/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu+++ b/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu-@@ -736,7 +736,7 @@ struct ggml_backend_cuda_buffer_context {+@@ -738,7 +738,7 @@ struct ggml_backend_cuda_buffer_context {}};@@ -10,7 +10,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context;delete ctx;}-@@ -745,12 +745,12 @@ static bool ggml_backend_buffer_is_cuda(ggml_backend_buffer_t buffer) {+@@ -747,12 +747,12 @@ static bool ggml_backend_buffer_is_cuda(ggml_backend_buffer_t buffer) {return buffer->iface.free_buffer == ggml_backend_cuda_buffer_free_buffer;}@@ -25,7 +25,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context;if (tensor->view_src != NULL) {-@@ -771,7 +771,7 @@ static enum ggml_status ggml_backend_cuda_buffer_init_tensor(ggml_backend_buffer+@@ -773,7 +773,7 @@ static enum ggml_status ggml_backend_cuda_buffer_init_tensor(ggml_backend_bufferreturn GGML_STATUS_SUCCESS;}@@ -34,7 +34,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context;ggml_cuda_set_device(ctx->device);-@@ -779,7 +779,7 @@ static void ggml_backend_cuda_buffer_memset_tensor(ggml_backend_buffer_t buffer,+@@ -781,7 +781,7 @@ static void ggml_backend_cuda_buffer_memset_tensor(ggml_backend_buffer_t buffer,CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));}@@ -43,7 +43,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context;ggml_cuda_set_device(ctx->device);-@@ -787,7 +787,7 @@ static void ggml_backend_cuda_buffer_set_tensor(ggml_backend_buffer_t buffer, gg+@@ -789,7 +789,7 @@ static void ggml_backend_cuda_buffer_set_tensor(ggml_backend_buffer_t buffer, ggCUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));}@@ -52,7 +52,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context;ggml_cuda_set_device(ctx->device);-@@ -795,7 +795,7 @@ static void ggml_backend_cuda_buffer_get_tensor(ggml_backend_buffer_t buffer, co+@@ -797,7 +797,7 @@ static void ggml_backend_cuda_buffer_get_tensor(ggml_backend_buffer_t buffer, coCUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));}@@ -61,7 +61,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cusize_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) {ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context;-@@ -805,7 +805,7 @@ static void ggml_backend_cuda_buffer_set_tensor_2d(ggml_backend_buffer_t buffer,+@@ -807,7 +807,7 @@ static void ggml_backend_cuda_buffer_set_tensor_2d(ggml_backend_buffer_t buffer,CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));}@@ -70,7 +70,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cusize_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) {ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context;-@@ -815,7 +815,7 @@ static void ggml_backend_cuda_buffer_get_tensor_2d(ggml_backend_buffer_t buffer,+@@ -817,7 +817,7 @@ static void ggml_backend_cuda_buffer_get_tensor_2d(ggml_backend_buffer_t buffer,CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));}@@ -79,7 +79,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuif (ggml_backend_buffer_is_cuda(src->buffer)) {ggml_backend_cuda_buffer_context * src_ctx = (ggml_backend_cuda_buffer_context *)src->buffer->context;ggml_backend_cuda_buffer_context * dst_ctx = (ggml_backend_cuda_buffer_context *)dst->buffer->context;-@@ -840,7 +840,7 @@ static bool ggml_backend_cuda_buffer_cpy_tensor(ggml_backend_buffer_t buffer, co+@@ -842,7 +842,7 @@ static bool ggml_backend_cuda_buffer_cpy_tensor(ggml_backend_buffer_t buffer, coGGML_UNUSED(buffer);}@@ -88,7 +88,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context;ggml_cuda_set_device(ctx->device);-@@ -848,6 +848,14 @@ static void ggml_backend_cuda_buffer_clear(ggml_backend_buffer_t buffer, uint8_t+@@ -850,6 +850,14 @@ static void ggml_backend_cuda_buffer_clear(ggml_backend_buffer_t buffer, uint8_tCUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));}@@ -103,7 +103,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.custatic const ggml_backend_buffer_i ggml_backend_cuda_buffer_interface = {/* .free_buffer = */ ggml_backend_cuda_buffer_free_buffer,/* .get_base = */ ggml_backend_cuda_buffer_get_base,-@@ -860,6 +868,7 @@ static const ggml_backend_buffer_i ggml_backend_cuda_buffer_interface = {+@@ -862,6 +870,7 @@ static const ggml_backend_buffer_i ggml_backend_cuda_buffer_interface = {/* .cpy_tensor = */ ggml_backend_cuda_buffer_cpy_tensor,/* .clear = */ ggml_backend_cuda_buffer_clear,/* .reset = */ NULL,@@ -111,7 +111,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu};// cuda buffer type-@@ -868,7 +877,7 @@ struct ggml_backend_cuda_buffer_type_context {+@@ -870,7 +879,7 @@ struct ggml_backend_cuda_buffer_type_context {std::string name;};@@ -120,7 +120,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_buffer_type_context * ctx = (ggml_backend_cuda_buffer_type_context *)buft->context;return ctx->name.c_str();-@@ -878,7 +887,7 @@ static bool ggml_backend_buft_is_cuda(ggml_backend_buffer_type_t buft) {+@@ -880,7 +889,7 @@ static bool ggml_backend_buft_is_cuda(ggml_backend_buffer_type_t buft) {return buft->iface.get_name == ggml_backend_cuda_buffer_type_get_name;}@@ -129,7 +129,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_buffer_type_context * buft_ctx = (ggml_backend_cuda_buffer_type_context *)buft->context;ggml_cuda_set_device(buft_ctx->device);-@@ -897,13 +906,13 @@ static ggml_backend_buffer_t ggml_backend_cuda_buffer_type_alloc_buffer(ggml_bac+@@ -899,13 +908,13 @@ static ggml_backend_buffer_t ggml_backend_cuda_buffer_type_alloc_buffer(ggml_bacreturn ggml_backend_buffer_init(buft, ggml_backend_cuda_buffer_interface, ctx, size);}@@ -145,7 +145,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_buffer_type_context * buft_ctx = (ggml_backend_cuda_buffer_type_context *) buft->context;size_t size = tensor->op == GGML_OP_FLASH_ATTN_EXT-@@ -1256,7 +1265,7 @@ static bool ggml_backend_cuda_comm_allreduce_tensor(void * comm_ctx_v, struct gg+@@ -1259,7 +1268,7 @@ static bool ggml_backend_cuda_comm_allreduce_tensor(void * comm_ctx_v, struct gg// host buffer type@@ -154,7 +154,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cureturn GGML_CUDA_NAME "_Host";GGML_UNUSED(buft);-@@ -1266,7 +1275,7 @@ static bool ggml_backend_buft_is_cuda_host(ggml_backend_buffer_type_t buft) {+@@ -1269,7 +1278,7 @@ static bool ggml_backend_buft_is_cuda_host(ggml_backend_buffer_type_t buft) {return buft->iface.get_name == ggml_backend_cuda_host_buffer_type_name;}@@ -163,7 +163,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuCUDA_CHECK(cudaFreeHost(buffer->context));}-@@ -1288,17 +1297,25 @@ static void * ggml_cuda_host_malloc(size_t size) {+@@ -1291,17 +1300,25 @@ static void * ggml_cuda_host_malloc(size_t size) {return ptr;}@@ -191,7 +191,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cureturn buffer;}-@@ -1644,6 +1661,14 @@ static void ggml_cuda_mul_mat_cublas(ggml_backend_cuda_context & ctx, const ggml+@@ -1655,6 +1672,14 @@ static void ggml_cuda_mul_mat_cublas(ggml_backend_cuda_context & ctx, const ggml}}@@ -206,7 +206,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuswitch (compute_type) {case GGML_TYPE_F32:ggml_cuda_mul_mat_cublas_impl<GGML_TYPE_F32>(ctx, src0, src1, dst);-@@ -2412,20 +2437,20 @@ static bool ggml_cuda_compute_forward(ggml_backend_cuda_context & ctx, struct gg+@@ -2429,20 +2454,20 @@ static bool ggml_cuda_compute_forward(ggml_backend_cuda_context & ctx, struct gg// backend@@ -230,7 +230,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context;ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer;-@@ -2434,7 +2459,7 @@ static void ggml_backend_cuda_set_tensor_async(ggml_backend_t backend, ggml_tens+@@ -2451,7 +2476,7 @@ static void ggml_backend_cuda_set_tensor_async(ggml_backend_t backend, ggml_tensCUDA_CHECK(cudaMemcpyAsync((char *) tensor->data + offset, data, size, cudaMemcpyHostToDevice, cuda_ctx->stream()));}@@ -239,7 +239,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context;ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer;-@@ -2443,7 +2468,7 @@ static void ggml_backend_cuda_get_tensor_async(ggml_backend_t backend, const ggm+@@ -2460,7 +2485,7 @@ static void ggml_backend_cuda_get_tensor_async(ggml_backend_t backend, const ggmCUDA_CHECK(cudaMemcpyAsync(data, (const char *) tensor->data + offset, size, cudaMemcpyDeviceToHost, cuda_ctx->stream()));}@@ -248,7 +248,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cusize_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) {ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context;ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer;-@@ -2454,7 +2479,7 @@ static void ggml_backend_cuda_set_tensor_2d_async(ggml_backend_t backend, struct+@@ -2471,7 +2496,7 @@ static void ggml_backend_cuda_set_tensor_2d_async(ggml_backend_t backend, struct(char *) tensor->data + offset, stride_tensor, data, stride_data, size, n_copies, cudaMemcpyHostToDevice, cuda_ctx->stream()));}@@ -257,7 +257,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cusize_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) {ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context;ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer;-@@ -2465,7 +2490,7 @@ static void ggml_backend_cuda_get_tensor_2d_async(ggml_backend_t backend, const+@@ -2482,7 +2507,7 @@ static void ggml_backend_cuda_get_tensor_2d_async(ggml_backend_t backend, constdata, stride_data, (const char *) tensor->data + offset, stride_tensor, size, n_copies, cudaMemcpyDeviceToHost, cuda_ctx->stream()));}@@ -266,7 +266,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_buffer_t buf_src = src->view_src ? src->view_src->buffer : src->buffer;ggml_backend_buffer_t buf_dst = dst->view_src ? dst->view_src->buffer : dst->buffer;-@@ -2524,7 +2549,7 @@ static bool ggml_backend_cuda_cpy_tensor_async(ggml_backend_t backend_src, ggml_+@@ -2541,7 +2566,7 @@ static bool ggml_backend_cuda_cpy_tensor_async(ggml_backend_t backend_src, ggml_return true;}@@ -275,7 +275,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context;CUDA_CHECK(cudaStreamSynchronize(cuda_ctx->stream()));-@@ -4231,7 +4256,7 @@ static bool ggml_cuda_graph_set_enabled(ggml_backend_cuda_context * cuda_ctx, co+@@ -4417,7 +4442,7 @@ static bool ggml_cuda_graph_set_enabled(ggml_backend_cuda_context * cuda_ctx, co}#endif // USE_CUDA_GRAPH@@ -284,7 +284,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context;ggml_cuda_set_device(cuda_ctx->device);-@@ -4290,13 +4315,13 @@ static enum ggml_status ggml_backend_cuda_graph_compute(ggml_backend_t backend,+@@ -4476,13 +4501,13 @@ static enum ggml_status ggml_backend_cuda_graph_compute(ggml_backend_t backend,return GGML_STATUS_SUCCESS;}@@ -300,16 +300,16 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context;if (ggml_backend_is_cuda(backend)) {-@@ -4315,7 +4340,7 @@ static void ggml_backend_cuda_event_wait(ggml_backend_t backend, ggml_backend_ev+@@ -4501,7 +4526,7 @@ static void ggml_backend_cuda_event_wait(ggml_backend_t backend, ggml_backend_ev}}--static void ggml_backend_cuda_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph) {-+static void GGML_CALL ggml_backend_cuda_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph) {+-static void ggml_backend_cuda_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph, ggml_backend_graph_optimize_params * params) {++static void GGML_CALL ggml_backend_cuda_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph, ggml_backend_graph_optimize_params * params) {ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context;- #ifdef USE_CUDA_GRAPH-@@ -4625,7 +4650,7 @@ void ggml_backend_cuda_get_device_memory(int device, size_t * free, size_t * tot+ static const bool disable_fusion = getenv("GGML_CUDA_DISABLE_FUSION") != nullptr && std::atoi(getenv("GGML_CUDA_DISABLE_FUSION"));+@@ -4835,7 +4860,7 @@ void ggml_backend_cuda_get_device_memory(int device, size_t * free, size_t * tot*total /= share_count;}@@ -318,7 +318,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuif (getenv("GGML_CUDA_REGISTER_HOST") == nullptr) {return false;}-@@ -4648,7 +4673,7 @@ bool ggml_backend_cuda_register_host_buffer(void * buffer, size_t size) {+@@ -4858,7 +4883,7 @@ bool ggml_backend_cuda_register_host_buffer(void * buffer, size_t size) {#endif // CUDART_VERSION >= 11010 || defined(GGML_USE_MUSA)}@@ -327,7 +327,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuif (getenv("GGML_CUDA_REGISTER_HOST") == nullptr) {return;}-@@ -4671,12 +4696,12 @@ struct ggml_backend_cuda_device_context {+@@ -4881,12 +4906,12 @@ struct ggml_backend_cuda_device_context {int op_offload_min_batch_size;};@@ -342,7 +342,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context;return ctx->description.c_str();}-@@ -4757,7 +4782,7 @@ static bool ggml_backend_cuda_get_available_uma_memory(long * available_memory_k+@@ -4967,7 +4992,7 @@ static bool ggml_backend_cuda_get_available_uma_memory(long * available_memory_k}#endif // defined(__linux__)@@ -351,7 +351,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context;ggml_cuda_set_device(ctx->device);cudaError_t err = cudaMemGetInfo(free, total);-@@ -4798,7 +4823,7 @@ static void ggml_backend_cuda_device_get_memory(ggml_backend_dev_t dev, size_t *+@@ -5008,7 +5033,7 @@ static void ggml_backend_cuda_device_get_memory(ggml_backend_dev_t dev, size_t **total /= share_count;}@@ -360,7 +360,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *) dev->context;cudaDeviceProp prop;-@@ -4809,7 +4834,7 @@ static enum ggml_backend_dev_type ggml_backend_cuda_device_get_type(ggml_backend+@@ -5019,7 +5044,7 @@ static enum ggml_backend_dev_type ggml_backend_cuda_device_get_type(ggml_backend: GGML_BACKEND_DEVICE_TYPE_GPU;}@@ -369,7 +369,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context;props->name = ggml_backend_cuda_device_get_name(dev);-@@ -4834,24 +4859,24 @@ static void ggml_backend_cuda_device_get_props(ggml_backend_dev_t dev, ggml_back+@@ -5044,24 +5069,24 @@ static void ggml_backend_cuda_device_get_props(ggml_backend_dev_t dev, ggml_back};}@@ -398,7 +398,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context;// check if all the sources are allocated on this device-@@ -4951,6 +4976,13 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g+@@ -5162,6 +5187,13 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const gcase GGML_TYPE_Q5_K:case GGML_TYPE_Q6_K:case GGML_TYPE_Q8_K:@@ -412,7 +412,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cucase GGML_TYPE_IQ1_M:case GGML_TYPE_IQ1_S:case GGML_TYPE_IQ2_S:-@@ -4960,6 +4992,7 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g+@@ -5171,6 +5203,7 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const gcase GGML_TYPE_IQ3_XXS:case GGML_TYPE_IQ4_NL:case GGML_TYPE_IQ4_XS:@@ -420,7 +420,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cucase GGML_TYPE_BF16:return true;default:-@@ -5069,9 +5102,14 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g+@@ -5280,9 +5313,14 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const gif (src0_type == GGML_TYPE_Q5_1 && src1_type == GGML_TYPE_F32) {return true;}@@ -435,7 +435,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuif (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_I32) {return true;}-@@ -5303,7 +5341,7 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g+@@ -5519,7 +5557,7 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g}}@@ -444,7 +444,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context;const bool integrated = ggml_cuda_info().devices[dev_ctx->device].integrated;return (ggml_backend_buft_is_cuda(buft) && buft->device == dev) || (integrated && ggml_backend_buft_is_cuda_host(buft));-@@ -5324,13 +5362,13 @@ static int64_t get_op_batch_size(const ggml_tensor * op) {+@@ -5540,13 +5578,13 @@ static int64_t get_op_batch_size(const ggml_tensor * op) {}}@@ -460,7 +460,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu#ifdef GGML_CUDA_NO_PEER_COPYGGML_UNUSED(dev);return nullptr;-@@ -5349,14 +5387,14 @@ static ggml_backend_event_t ggml_backend_cuda_device_event_new(ggml_backend_dev_+@@ -5565,14 +5603,14 @@ static ggml_backend_event_t ggml_backend_cuda_device_event_new(ggml_backend_dev_#endif}@@ -477,7 +477,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cuGGML_UNUSED(dev);CUDA_CHECK(cudaEventSynchronize((cudaEvent_t)event->context));}-@@ -5385,23 +5423,23 @@ struct ggml_backend_cuda_reg_context {+@@ -5601,23 +5639,23 @@ struct ggml_backend_cuda_reg_context {std::vector<ggml_backend_dev_t> devices;};@@ -505,7 +505,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.custatic std::vector<ggml_backend_feature> features = []() {std::vector<ggml_backend_feature> features;#define _STRINGIFY(...) #__VA_ARGS__-@@ -5458,7 +5496,7 @@ static ggml_backend_feature * ggml_backend_cuda_get_features(ggml_backend_reg_t+@@ -5674,7 +5712,7 @@ static ggml_backend_feature * ggml_backend_cuda_get_features(ggml_backend_reg_tGGML_UNUSED(reg);}diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmq.cu.patch b/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmq.cu.patchindex baab0e7b..673e761f 100644--- a/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmq.cu.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmq.cu.patch@@ -17,7 +17,7 @@ diff --git a/ggml/src/ggml-cuda/mmq.cu b/ggml/src/ggml-cuda/mmq.cu// -----------------------------------------------------------------------case GGML_TYPE_MXFP4:mul_mat_q_case<GGML_TYPE_MXFP4>(ctx, args, stream);-@@ -278,6 +280,7 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t+@@ -285,6 +287,7 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_tcase GGML_TYPE_Q5_K:case GGML_TYPE_Q6_K:// -------------------------------------------------@@ -25,7 +25,7 @@ diff --git a/ggml/src/ggml-cuda/mmq.cu b/ggml/src/ggml-cuda/mmq.cucase GGML_TYPE_IQ1_S:case GGML_TYPE_IQ2_XXS:case GGML_TYPE_IQ2_XS:-@@ -286,6 +289,7 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t+@@ -293,6 +296,7 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_tcase GGML_TYPE_IQ3_S:case GGML_TYPE_IQ4_XS:case GGML_TYPE_IQ4_NL:@@ -33,7 +33,7 @@ diff --git a/ggml/src/ggml-cuda/mmq.cu b/ggml/src/ggml-cuda/mmq.cu// -------------------------------------------------case GGML_TYPE_MXFP4:case GGML_TYPE_NVFP4:-@@ -360,9 +364,11 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t+@@ -369,9 +373,11 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_treturn ne11 <= 128;case GGML_TYPE_Q6_K:return ne11 <= (GGML_CUDA_CC_IS_RDNA3_0(cc) ? 128 : 256);diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmq.cuh.patch b/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmq.cuh.patchindex 9bda9b59..e8b3f216 100644--- a/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmq.cuh.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmq.cuh.patch@@ -1,7 +1,7 @@diff --git a/ggml/src/ggml-cuda/mmq.cuh b/ggml/src/ggml-cuda/mmq.cuh--- a/llama.cpp/ggml/src/ggml-cuda/mmq.cuh+++ b/llama.cpp/ggml/src/ggml-cuda/mmq.cuh-@@ -1577,6 +1577,7 @@ extern DECL_MMQ_CASE(GGML_TYPE_Q4_K);+@@ -1585,6 +1585,7 @@ extern DECL_MMQ_CASE(GGML_TYPE_Q4_K);extern DECL_MMQ_CASE(GGML_TYPE_Q5_K);extern DECL_MMQ_CASE(GGML_TYPE_Q6_K);// -----------------------------------------@@ -9,7 +9,7 @@ diff --git a/ggml/src/ggml-cuda/mmq.cuh b/ggml/src/ggml-cuda/mmq.cuhextern DECL_MMQ_CASE(GGML_TYPE_IQ1_S);extern DECL_MMQ_CASE(GGML_TYPE_IQ2_XXS);extern DECL_MMQ_CASE(GGML_TYPE_IQ2_XS);-@@ -1585,6 +1586,7 @@ extern DECL_MMQ_CASE(GGML_TYPE_IQ3_XXS);+@@ -1593,6 +1594,7 @@ extern DECL_MMQ_CASE(GGML_TYPE_IQ3_XXS);extern DECL_MMQ_CASE(GGML_TYPE_IQ3_S);extern DECL_MMQ_CASE(GGML_TYPE_IQ4_NL);extern DECL_MMQ_CASE(GGML_TYPE_IQ4_XS);diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmvq.cu.patch b/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmvq.cu.patchindex a7dbc1c7..5a5ebad5 100644--- a/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmvq.cu.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmvq.cu.patch@@ -1,7 +1,7 @@diff --git a/ggml/src/ggml-cuda/mmvq.cu b/ggml/src/ggml-cuda/mmvq.cu--- a/llama.cpp/ggml/src/ggml-cuda/mmvq.cu+++ b/llama.cpp/ggml/src/ggml-cuda/mmvq.cu-@@ -23,6 +23,7 @@ static constexpr __device__ vec_dot_q_cuda_t get_vec_dot_q_cuda(ggml_type type)+@@ -53,6 +53,7 @@ static constexpr __device__ vec_dot_q_cuda_t get_vec_dot_q_cuda(ggml_type type)case GGML_TYPE_Q4_K: return vec_dot_q4_K_q8_1;case GGML_TYPE_Q5_K: return vec_dot_q5_K_q8_1;case GGML_TYPE_Q6_K: return vec_dot_q6_K_q8_1;@@ -9,7 +9,7 @@ diff --git a/ggml/src/ggml-cuda/mmvq.cu b/ggml/src/ggml-cuda/mmvq.cucase GGML_TYPE_IQ2_XXS: return vec_dot_iq2_xxs_q8_1;case GGML_TYPE_IQ2_XS: return vec_dot_iq2_xs_q8_1;case GGML_TYPE_IQ2_S: return vec_dot_iq2_s_q8_1;-@@ -32,6 +33,7 @@ static constexpr __device__ vec_dot_q_cuda_t get_vec_dot_q_cuda(ggml_type type)+@@ -62,6 +63,7 @@ static constexpr __device__ vec_dot_q_cuda_t get_vec_dot_q_cuda(ggml_type type)case GGML_TYPE_IQ4_NL: return vec_dot_iq4_nl_q8_1;case GGML_TYPE_IQ4_XS: return vec_dot_iq4_xs_q8_1;case GGML_TYPE_IQ3_S: return vec_dot_iq3_s_q8_1;@@ -17,7 +17,7 @@ diff --git a/ggml/src/ggml-cuda/mmvq.cu b/ggml/src/ggml-cuda/mmvq.cudefault: return nullptr;}}-@@ -52,6 +54,7 @@ static constexpr __host__ __device__ int get_vdr_mmvq(ggml_type type) {+@@ -82,6 +84,7 @@ static constexpr __host__ __device__ int get_vdr_mmvq(ggml_type type) {case GGML_TYPE_Q4_K: return VDR_Q4_K_Q8_1_MMVQ;case GGML_TYPE_Q5_K: return VDR_Q5_K_Q8_1_MMVQ;case GGML_TYPE_Q6_K: return VDR_Q6_K_Q8_1_MMVQ;@@ -25,7 +25,7 @@ diff --git a/ggml/src/ggml-cuda/mmvq.cu b/ggml/src/ggml-cuda/mmvq.cucase GGML_TYPE_IQ2_XXS: return VDR_IQ2_XXS_Q8_1_MMVQ;case GGML_TYPE_IQ2_XS: return VDR_IQ2_XS_Q8_1_MMVQ;case GGML_TYPE_IQ2_S: return VDR_IQ2_S_Q8_1_MMVQ;-@@ -59,6 +62,7 @@ static constexpr __host__ __device__ int get_vdr_mmvq(ggml_type type) {+@@ -89,6 +92,7 @@ static constexpr __host__ __device__ int get_vdr_mmvq(ggml_type type) {case GGML_TYPE_IQ3_S: return VDR_IQ3_S_Q8_1_MMVQ;case GGML_TYPE_IQ4_NL: return VDR_IQ4_NL_Q8_1_MMVQ;case GGML_TYPE_IQ4_XS: return VDR_IQ4_XS_Q8_1_MMVQ;diff --git a/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal-fusion.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal-fusion.cpp.patchnew file mode 100644index 00000000..6f17a51e--- /dev/null+++ b/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal-fusion.cpp.patch@@ -0,0 +1,21 @@+diff --git a/ggml/src/ggml-metal/ggml-metal-fusion.cpp b/ggml/src/ggml-metal/ggml-metal-fusion.cpp+--- a/llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.cpp++++ b/llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.cpp+@@ -717,7 +717,7 @@ static bool ggml_metal_fusion_match_raw_pattern(++ static void ggml_metal_fusion_add_pattern_alloc_deps(+ void * user_data,+- void (*add_alloc_dep)(void *, ggml_tensor *, ggml_tensor *),++ void (GGML_CALL *add_alloc_dep)(void *, ggml_tensor *, ggml_tensor *),+ ggml_cgraph * gf,+ const ggml_metal_fusion * fusion,+ int node_idx) {+@@ -739,7 +739,7 @@ static void ggml_metal_fusion_add_pattern_alloc_deps(++ void ggml_metal_fusion_add_alloc_deps(+ void * user_data,+- void (*add_alloc_dep)(void *, ggml_tensor *, ggml_tensor *),++ void (GGML_CALL *add_alloc_dep)(void *, ggml_tensor *, ggml_tensor *),+ ggml_cgraph * gf) {+ for (int i = 0; i < gf->n_nodes; ++i) {+ const ggml_metal_fusion * best = nullptr;diff --git a/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal-fusion.h.patch b/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal-fusion.h.patchnew file mode 100644index 00000000..5dbbf36a--- /dev/null+++ b/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal-fusion.h.patch@@ -0,0 +1,20 @@+diff --git a/ggml/src/ggml-metal/ggml-metal-fusion.h b/ggml/src/ggml-metal/ggml-metal-fusion.h+--- a/llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.h++++ b/llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.h+@@ -8,6 +8,7 @@+ #pragma once++ #include "ggml-impl.h"++#include "ggml-backend.h"++ #include <stdbool.h>++@@ -51,7 +52,7 @@ ggml_metal_fusion_id ggml_metal_fusion_get_id(const struct ggml_metal_fusion * f+ // apply any alloc-dependencies required by the fused kernels during graph optimize+ void ggml_metal_fusion_add_alloc_deps(+ void * user_data,+- void (*add_alloc_dep)(void *, struct ggml_tensor *, struct ggml_tensor *),++ void (GGML_CALL *add_alloc_dep)(void *, struct ggml_tensor *, struct ggml_tensor *),+ struct ggml_cgraph * gf);++ // ---- shared fusion info ---------------------------------------------------diff --git a/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal.cpp.patchindex 0473ca6b..d5b1b90c 100644--- a/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal.cpp.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal.cpp.patch@@ -1,7 +1,7 @@diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal.cpp--- a/llama.cpp/ggml/src/ggml-metal/ggml-metal.cpp+++ b/llama.cpp/ggml/src/ggml-metal/ggml-metal.cpp-@@ -26,7 +26,7 @@ static bool ggml_backend_buffer_is_metal(ggml_backend_buffer_t buffer);+@@ -28,7 +28,7 @@ static bool ggml_backend_buffer_is_metal(ggml_backend_buffer_t buffer);// shared buffer@@ -10,7 +10,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;GGML_ASSERT(ggml_metal_buffer_is_shared(ctx));-@@ -34,7 +34,7 @@ static void ggml_backend_metal_buffer_shared_free_buffer(ggml_backend_buffer_t b+@@ -36,7 +36,7 @@ static void ggml_backend_metal_buffer_shared_free_buffer(ggml_backend_buffer_t bggml_metal_buffer_free(ctx);}@@ -19,7 +19,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;GGML_ASSERT(ggml_metal_buffer_is_shared(ctx));-@@ -42,7 +42,7 @@ static void * ggml_backend_metal_buffer_shared_get_base(ggml_backend_buffer_t bu+@@ -44,7 +44,7 @@ static void * ggml_backend_metal_buffer_shared_get_base(ggml_backend_buffer_t bureturn ggml_metal_buffer_get_base(ctx);}@@ -28,7 +28,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;GGML_ASSERT(ggml_metal_buffer_is_shared(ctx));-@@ -50,7 +50,7 @@ static void ggml_backend_metal_buffer_shared_memset_tensor(ggml_backend_buffer_t+@@ -52,7 +52,7 @@ static void ggml_backend_metal_buffer_shared_memset_tensor(ggml_backend_buffer_tggml_metal_buffer_memset_tensor(ctx, tensor, value, offset, size);}@@ -37,7 +37,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;GGML_ASSERT(ggml_metal_buffer_is_shared(ctx));-@@ -58,7 +58,7 @@ static void ggml_backend_metal_buffer_shared_set_tensor(ggml_backend_buffer_t bu+@@ -60,7 +60,7 @@ static void ggml_backend_metal_buffer_shared_set_tensor(ggml_backend_buffer_t buggml_metal_buffer_set_tensor(ctx, tensor, data, offset, size);}@@ -46,7 +46,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;GGML_ASSERT(ggml_metal_buffer_is_shared(ctx));-@@ -66,7 +66,7 @@ static void ggml_backend_metal_buffer_shared_get_tensor(ggml_backend_buffer_t bu+@@ -68,7 +68,7 @@ static void ggml_backend_metal_buffer_shared_get_tensor(ggml_backend_buffer_t buggml_metal_buffer_get_tensor(ctx, tensor, data, offset, size);}@@ -55,7 +55,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;GGML_ASSERT(ggml_metal_buffer_is_shared(ctx));-@@ -78,7 +78,7 @@ static bool ggml_backend_metal_buffer_shared_cpy_tensor(ggml_backend_buffer_t bu+@@ -80,7 +80,7 @@ static bool ggml_backend_metal_buffer_shared_cpy_tensor(ggml_backend_buffer_t bureturn ggml_metal_buffer_cpy_tensor(ctx, src, dst);}@@ -64,7 +64,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;GGML_ASSERT(ggml_metal_buffer_is_shared(ctx));-@@ -86,6 +86,14 @@ static void ggml_backend_metal_buffer_shared_clear(ggml_backend_buffer_t buffer,+@@ -88,6 +88,14 @@ static void ggml_backend_metal_buffer_shared_clear(ggml_backend_buffer_t buffer,ggml_metal_buffer_clear(ctx, value);}@@ -79,7 +79,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalstatic ggml_backend_buffer_i ggml_backend_metal_buffer_shared_i = {/* .free_buffer = */ ggml_backend_metal_buffer_shared_free_buffer,/* .get_base = */ ggml_backend_metal_buffer_shared_get_base,-@@ -98,11 +106,12 @@ static ggml_backend_buffer_i ggml_backend_metal_buffer_shared_i = {+@@ -100,11 +108,12 @@ static ggml_backend_buffer_i ggml_backend_metal_buffer_shared_i = {/* .cpy_tensor = */ ggml_backend_metal_buffer_shared_cpy_tensor,/* .clear = */ ggml_backend_metal_buffer_shared_clear,/* .reset = */ NULL,@@ -93,7 +93,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;GGML_ASSERT(!ggml_metal_buffer_is_shared(ctx));-@@ -110,7 +119,7 @@ static void ggml_backend_metal_buffer_private_free_buffer(ggml_backend_buffer_t+@@ -112,7 +121,7 @@ static void ggml_backend_metal_buffer_private_free_buffer(ggml_backend_buffer_tggml_metal_buffer_free(ctx);}@@ -102,7 +102,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;GGML_ASSERT(!ggml_metal_buffer_is_shared(ctx));-@@ -118,7 +127,7 @@ static void * ggml_backend_metal_buffer_private_get_base(ggml_backend_buffer_t b+@@ -120,7 +129,7 @@ static void * ggml_backend_metal_buffer_private_get_base(ggml_backend_buffer_t breturn ggml_metal_buffer_get_base(ctx);}@@ -111,7 +111,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;GGML_ASSERT(!ggml_metal_buffer_is_shared(ctx));-@@ -126,7 +135,7 @@ static void ggml_backend_metal_buffer_private_memset_tensor(ggml_backend_buffer_+@@ -128,7 +137,7 @@ static void ggml_backend_metal_buffer_private_memset_tensor(ggml_backend_buffer_ggml_metal_buffer_memset_tensor(ctx, tensor, value, offset, size);}@@ -120,7 +120,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;GGML_ASSERT(!ggml_metal_buffer_is_shared(ctx));-@@ -134,7 +143,7 @@ static void ggml_backend_metal_buffer_private_set_tensor(ggml_backend_buffer_t b+@@ -136,7 +145,7 @@ static void ggml_backend_metal_buffer_private_set_tensor(ggml_backend_buffer_t bggml_metal_buffer_set_tensor(ctx, tensor, data, offset, size);}@@ -129,7 +129,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;GGML_ASSERT(!ggml_metal_buffer_is_shared(ctx));-@@ -142,7 +151,7 @@ static void ggml_backend_metal_buffer_private_get_tensor(ggml_backend_buffer_t b+@@ -144,7 +153,7 @@ static void ggml_backend_metal_buffer_private_get_tensor(ggml_backend_buffer_t bggml_metal_buffer_get_tensor(ctx, tensor, data, offset, size);}@@ -138,7 +138,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;GGML_ASSERT(!ggml_metal_buffer_is_shared(ctx));-@@ -154,7 +163,7 @@ static bool ggml_backend_metal_buffer_private_cpy_tensor(ggml_backend_buffer_t b+@@ -156,7 +165,7 @@ static bool ggml_backend_metal_buffer_private_cpy_tensor(ggml_backend_buffer_t breturn ggml_metal_buffer_cpy_tensor(ctx, src, dst);}@@ -147,7 +147,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;GGML_ASSERT(!ggml_metal_buffer_is_shared(ctx));-@@ -174,6 +183,7 @@ static ggml_backend_buffer_i ggml_backend_metal_buffer_private_i = {+@@ -176,6 +185,7 @@ static ggml_backend_buffer_i ggml_backend_metal_buffer_private_i = {/* .cpy_tensor = */ ggml_backend_metal_buffer_private_cpy_tensor,/* .clear = */ ggml_backend_metal_buffer_private_clear,/* .reset = */ NULL,@@ -155,7 +155,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal};static bool ggml_backend_buffer_is_metal(ggml_backend_buffer_t buffer) {-@@ -246,33 +256,33 @@ static size_t ggml_backend_metal_buffer_type_get_alloc_size(ggml_backend_buffer_+@@ -256,33 +266,33 @@ static size_t ggml_backend_metal_buffer_type_get_alloc_size(ggml_backend_buffer_// default (shared) buffer type@@ -195,7 +195,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalreturn false;GGML_UNUSED(buft);-@@ -322,33 +332,33 @@ static ggml_backend_buffer_type_t ggml_backend_metal_buffer_type_shared(int devi+@@ -332,33 +342,33 @@ static ggml_backend_buffer_type_t ggml_backend_metal_buffer_type_shared(int devi// default (private) buffer type@@ -235,7 +235,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalreturn false;GGML_UNUSED(buft);-@@ -397,34 +407,34 @@ static ggml_backend_buffer_type_t ggml_backend_metal_buffer_type_private(int dev+@@ -407,34 +417,34 @@ static ggml_backend_buffer_type_t ggml_backend_metal_buffer_type_private(int dev// mapped buffer type@@ -276,7 +276,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalreturn false;GGML_UNUSED(buft);-@@ -475,13 +485,13 @@ static ggml_backend_buffer_type_t ggml_backend_metal_buffer_type_mapped(int devi+@@ -485,13 +495,13 @@ static ggml_backend_buffer_type_t ggml_backend_metal_buffer_type_mapped(int devi// backend@@ -292,7 +292,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_t ctx = (ggml_metal_t)backend->context;// wait for any ongoing async operations to finish-@@ -492,25 +502,25 @@ static void ggml_backend_metal_free(ggml_backend_t backend) {+@@ -502,25 +512,25 @@ static void ggml_backend_metal_free(ggml_backend_t backend) {free(backend);}@@ -322,7 +322,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalif (!ggml_backend_is_metal(backend_src) || !ggml_backend_is_metal(backend_dst)) {return false;}-@@ -531,27 +541,27 @@ static bool ggml_backend_metal_cpy_tensor_async(ggml_backend_t backend_src, ggml+@@ -541,27 +551,27 @@ static bool ggml_backend_metal_cpy_tensor_async(ggml_backend_t backend_src, ggmlreturn ggml_metal_cpy_tensor_async(ctx_src, ctx_dst, src, dst);}@@ -349,12 +349,12 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_event_wait(ctx, ev);}--static void ggml_backend_metal_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph) {-+static void GGML_CALL ggml_backend_metal_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph) {- ggml_metal_t ctx = (ggml_metal_t)backend->context;+-static void ggml_backend_metal_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph, ggml_backend_graph_optimize_params * params) {++static void GGML_CALL ggml_backend_metal_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph, ggml_backend_graph_optimize_params * params) {+ GGML_ASSERT(params && params->add_alloc_dep);- ggml_metal_graph_optimize(ctx, cgraph);-@@ -643,7 +653,7 @@ void ggml_backend_metal_capture_next_compute(ggml_backend_t backend) {+ // keep the MoE weighted-reduction inputs alive until the fused output so the+@@ -659,7 +669,7 @@ void ggml_backend_metal_capture_next_compute(ggml_backend_t backend) {// backend device@@ -363,7 +363,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;const ggml_metal_device_props * props_dev = ggml_metal_device_get_props(ctx_dev);-@@ -651,25 +661,25 @@ static const char * ggml_backend_metal_device_get_name(ggml_backend_dev_t dev) {+@@ -667,25 +677,25 @@ static const char * ggml_backend_metal_device_get_name(ggml_backend_dev_t dev) {return props_dev->name;}@@ -393,7 +393,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalprops->name = ggml_backend_metal_device_get_name(dev);props->description = ggml_backend_metal_device_get_description(dev);props->type = ggml_backend_metal_device_get_type(dev);-@@ -685,7 +695,7 @@ static void ggml_backend_metal_device_get_props(ggml_backend_dev_t dev, ggml_bac+@@ -701,7 +711,7 @@ static void ggml_backend_metal_device_get_props(ggml_backend_dev_t dev, ggml_bac};}@@ -402,7 +402,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;ggml_metal_t ctx = ggml_metal_init(ctx_dev);-@@ -710,7 +720,7 @@ static ggml_backend_t ggml_backend_metal_device_init_backend(ggml_backend_dev_t+@@ -726,7 +736,7 @@ static ggml_backend_t ggml_backend_metal_device_init_backend(ggml_backend_dev_tGGML_UNUSED(params);}@@ -411,7 +411,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;const ggml_metal_device_props * props_dev = ggml_metal_device_get_props(ctx_dev);-@@ -718,7 +728,7 @@ static ggml_backend_buffer_type_t ggml_backend_metal_device_get_buffer_type(ggml+@@ -734,7 +744,7 @@ static ggml_backend_buffer_type_t ggml_backend_metal_device_get_buffer_type(ggmlreturn props_dev->use_shared_buffers ? ggml_backend_metal_buffer_type_shared(props_dev->device) : ggml_backend_metal_buffer_type_private(props_dev->device);}@@ -420,7 +420,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;ggml_metal_buffer_t res = ggml_metal_buffer_map(ctx_dev, ptr, size, max_tensor_size);-@@ -728,13 +738,13 @@ static ggml_backend_buffer_t ggml_backend_metal_device_buffer_mapped(ggml_backen+@@ -744,13 +754,13 @@ static ggml_backend_buffer_t ggml_backend_metal_device_buffer_mapped(ggml_backenreturn ggml_backend_buffer_init(ggml_backend_metal_buffer_type_mapped(props_dev->device), ggml_backend_metal_buffer_shared_i, res, size);}@@ -436,7 +436,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalreturnbuft->device == dev && (buft->iface.get_name == ggml_backend_metal_buffer_type_shared_get_name ||-@@ -755,7 +765,7 @@ static int64_t get_op_batch_size(const ggml_tensor * op) {+@@ -771,7 +781,7 @@ static int64_t get_op_batch_size(const ggml_tensor * op) {}}@@ -445,7 +445,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;return (op->op == GGML_OP_MUL_MAT ||-@@ -763,7 +773,7 @@ static bool ggml_backend_metal_device_offload_op(ggml_backend_dev_t dev, const g+@@ -779,7 +789,7 @@ static bool ggml_backend_metal_device_offload_op(ggml_backend_dev_t dev, const gget_op_batch_size(op) >= ggml_metal_device_get_props(ctx_dev)->op_offload_min_batch_size;}@@ -454,7 +454,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;ggml_metal_event_t event = ggml_metal_device_event_init(ctx_dev);-@@ -777,7 +787,7 @@ static ggml_backend_event_t ggml_backend_metal_device_event_new(ggml_backend_dev+@@ -793,7 +803,7 @@ static ggml_backend_event_t ggml_backend_metal_device_event_new(ggml_backend_devreturn ev;}@@ -463,7 +463,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;ggml_metal_event_t ev = (ggml_metal_event_t)event->context;-@@ -787,7 +797,7 @@ static void ggml_backend_metal_device_event_free(ggml_backend_dev_t dev, ggml_ba+@@ -803,7 +813,7 @@ static void ggml_backend_metal_device_event_free(ggml_backend_dev_t dev, ggml_badelete event;}@@ -472,7 +472,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;ggml_metal_event_t evt = (ggml_metal_event_t)event->context;-@@ -839,18 +849,18 @@ struct ggml_backend_metal_reg_deleter {+@@ -855,18 +865,18 @@ struct ggml_backend_metal_reg_deleter {typedef std::unique_ptr<struct ggml_backend_metal_reg, ggml_backend_metal_reg_deleter> ggml_backend_metal_reg_ptr;@@ -494,7 +494,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalggml_backend_metal_reg_t ctx = (ggml_backend_metal_reg_t)reg->context;GGML_ASSERT(index < ctx->devices.size());return ctx->devices[index];-@@ -863,13 +873,13 @@ static ggml_backend_feature g_ggml_backend_metal_features[] = {+@@ -879,7 +889,7 @@ static ggml_backend_feature g_ggml_backend_metal_features[] = {{ NULL, NULL },};@@ -503,6 +503,8 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metalreturn g_ggml_backend_metal_features;GGML_UNUSED(reg);+@@ -936,7 +946,7 @@ static void ggml_backend_metal_fusion_set_enabled(ggml_backend_fusion_t finfo, b+ ggml_metal_fusion_info_set_enabled((struct ggml_metal_fusion_info *) finfo, enabled);}-static void * ggml_backend_metal_get_proc_address(ggml_backend_reg_t reg, const char * name) {diff --git a/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan-buffers.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan-buffers.cpp.patchnew file mode 100644index 00000000..15445f90--- /dev/null+++ b/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan-buffers.cpp.patch@@ -0,0 +1,11 @@+diff --git a/ggml/src/ggml-vulkan/ggml-vulkan-buffers.cpp b/ggml/src/ggml-vulkan/ggml-vulkan-buffers.cpp+--- a/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan-buffers.cpp++++ b/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan-buffers.cpp+@@ -754,6 +754,7 @@ ggml_backend_buffer_i ggml_backend_vk_buffer_interface = {+ /* .cpy_tensor = */ ggml_backend_vk_buffer_cpy_tensor,+ /* .clear = */ ggml_backend_vk_buffer_clear,+ /* .reset = */ NULL,++ /* .free_struct = */ ggml_backend_vk_buffer_free_struct,+ };++ vk_buffer ggml_vk_buffer_from_host_ptr(vk_device & device, void * ptr, size_t size) {diff --git a/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan-common.h.patch b/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan-common.h.patchnew file mode 100644index 00000000..779fdcf5--- /dev/null+++ b/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan-common.h.patch@@ -0,0 +1,48 @@+diff --git a/ggml/src/ggml-vulkan/ggml-vulkan-common.h b/ggml/src/ggml-vulkan/ggml-vulkan-common.h+--- a/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan-common.h++++ b/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan-common.h+@@ -209,26 +209,27 @@ bool ggml_vk_can_fuse_snake(ggml_backend_vk_context * ctx, const struct ggml_cgr+ bool ggml_vk_tensors_overlap(const ggml_tensor * a, const ggml_tensor * b, bool elementwise);+ bool ggml_vk_can_fuse_rms_norm_mul_rope(ggml_backend_vk_context * ctx, const struct ggml_cgraph * cgraph, int node_idx);+ uint32_t ggml_vk_fuse_multi_add(ggml_backend_vk_context * ctx, const struct ggml_cgraph * cgraph, int node_idx);+-void ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph * graph, struct ggml_backend_graph_optimize_params * params);++void GGML_CALL ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph * graph, struct ggml_backend_graph_optimize_params * params);++ // backend+ bool ggml_backend_buffer_is_vk(ggml_backend_buffer_t buffer);+-void ggml_backend_vk_buffer_free_buffer(ggml_backend_buffer_t buffer);+-void * ggml_backend_vk_buffer_get_base(ggml_backend_buffer_t buffer);+-enum ggml_status ggml_backend_vk_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor);+-void ggml_backend_vk_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size);+-void ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size);+-void ggml_backend_vk_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data);+-void ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size);+-void ggml_backend_vk_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data);+-bool ggml_backend_vk_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * src, ggml_tensor * dst);+-void ggml_backend_vk_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value);+-const char * ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft);+-ggml_backend_buffer_t ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size);+-size_t ggml_backend_vk_buffer_type_get_alignment(ggml_backend_buffer_type_t buft);+-size_t ggml_backend_vk_buffer_type_get_max_size(ggml_backend_buffer_type_t buft);+-size_t ggml_backend_vk_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor);+-void ggml_backend_vk_free(ggml_backend_t backend);++void GGML_CALL ggml_backend_vk_buffer_free_buffer(ggml_backend_buffer_t buffer);++void GGML_CALL ggml_backend_vk_buffer_free_struct(ggml_backend_buffer_t buffer);++void * GGML_CALL ggml_backend_vk_buffer_get_base(ggml_backend_buffer_t buffer);++enum ggml_status GGML_CALL ggml_backend_vk_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor);++void GGML_CALL ggml_backend_vk_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size);++void GGML_CALL ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size);++void GGML_CALL ggml_backend_vk_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data);++void GGML_CALL ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size);++void GGML_CALL ggml_backend_vk_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data);++bool GGML_CALL ggml_backend_vk_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * src, ggml_tensor * dst);++void GGML_CALL ggml_backend_vk_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value);++const char * GGML_CALL ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft);++ggml_backend_buffer_t GGML_CALL ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size);++size_t GGML_CALL ggml_backend_vk_buffer_type_get_alignment(ggml_backend_buffer_type_t buft);++size_t GGML_CALL ggml_backend_vk_buffer_type_get_max_size(ggml_backend_buffer_type_t buft);++size_t GGML_CALL ggml_backend_vk_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor);++void GGML_CALL ggml_backend_vk_free(ggml_backend_t backend);+ ggml_backend_reg_t ggml_backend_vk_reg();++ // debugdiff --git a/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan.cpp.patchindex 05a75537..36b4784f 100644--- a/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan.cpp.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan.cpp.patch@@ -1,171 +1,139 @@diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vulkan.cpp--- a/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp+++ b/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp-@@ -367,11 +367,11 @@ struct vk_queue {- bool transfer_only;- };---static const char * ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft);--static ggml_backend_buffer_t ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size);--static size_t ggml_backend_vk_buffer_type_get_alignment(ggml_backend_buffer_type_t buft);--static size_t ggml_backend_vk_buffer_type_get_max_size(ggml_backend_buffer_type_t buft);--static size_t ggml_backend_vk_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor);-+static const char * GGML_CALL ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft);-+static ggml_backend_buffer_t GGML_CALL ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size);-+static size_t GGML_CALL ggml_backend_vk_buffer_type_get_alignment(ggml_backend_buffer_type_t buft);-+static size_t GGML_CALL ggml_backend_vk_buffer_type_get_max_size(ggml_backend_buffer_type_t buft);-+static size_t GGML_CALL ggml_backend_vk_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor);- static ggml_backend_buffer_type_i ggml_backend_vk_buffer_type_interface = {- /* .get_name = */ ggml_backend_vk_buffer_type_name,- /* .alloc_buffer = */ ggml_backend_vk_buffer_type_alloc_buffer,-@@ -2576,7 +2576,7 @@ static void ggml_vk_check_results_1(ggml_backend_vk_context * ctx, ggml_cgraph *-- typedef void (*ggml_vk_func_t)(ggml_backend_vk_context * ctx, vk_context& subctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst);---static void ggml_backend_vk_free(ggml_backend_t backend);-+static void GGML_CALL ggml_backend_vk_free(ggml_backend_t backend);-- static VkDeviceSize ggml_vk_get_max_buffer_range(const ggml_backend_vk_context * ctx, const vk_buffer &buf, const VkDeviceSize offset) {- const VkDeviceSize range = std::min(VkDeviceSize{buf->size - offset},-@@ -15864,20 +15864,28 @@ static bool ggml_backend_buffer_is_vk(ggml_backend_buffer_t buffer) {+@@ -12492,20 +12492,28 @@ bool ggml_backend_buffer_is_vk(ggml_backend_buffer_t buffer) {return buffer->buft->iface.get_name == ggml_backend_vk_buffer_type_name;}--static void ggml_backend_vk_buffer_free_buffer(ggml_backend_buffer_t buffer) {-+static void GGML_CALL ggml_backend_vk_buffer_free_buffer(ggml_backend_buffer_t buffer) {+-void ggml_backend_vk_buffer_free_buffer(ggml_backend_buffer_t buffer) {++void GGML_CALL ggml_backend_vk_buffer_free_buffer(ggml_backend_buffer_t buffer) {VK_LOG_MEMORY("ggml_backend_vk_buffer_free_buffer()");ggml_backend_vk_buffer_context * ctx = (ggml_backend_vk_buffer_context *)buffer->context;ggml_vk_destroy_buffer(ctx->dev_buffer);delete ctx;}--static void * ggml_backend_vk_buffer_get_base(ggml_backend_buffer_t buffer) {+-void * ggml_backend_vk_buffer_get_base(ggml_backend_buffer_t buffer) {+// free_struct callback for cross-module memory management+// When the Vulkan backend is loaded as a dynamic library, the buffer struct+// is allocated by the dylib's 'new' operator and must be freed by the dylib's+// 'delete' operator (not the main executable's)-+static void GGML_CALL ggml_backend_vk_buffer_free_struct(ggml_backend_buffer_t buffer) {++void GGML_CALL ggml_backend_vk_buffer_free_struct(ggml_backend_buffer_t buffer) {+ delete buffer;+}+-+static void * GGML_CALL ggml_backend_vk_buffer_get_base(ggml_backend_buffer_t buffer) {++void * GGML_CALL ggml_backend_vk_buffer_get_base(ggml_backend_buffer_t buffer) {return vk_ptr_base;UNUSED(buffer);}--static enum ggml_status ggml_backend_vk_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) {-+static enum ggml_status GGML_CALL ggml_backend_vk_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) {+-enum ggml_status ggml_backend_vk_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) {++enum ggml_status GGML_CALL ggml_backend_vk_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) {VK_LOG_DEBUG("ggml_backend_vk_buffer_init_tensor(" << buffer << " (" << buffer->context << "), " << tensor << ")");if (tensor->view_src != nullptr) {GGML_ASSERT(tensor->view_src->buffer->buft == buffer->buft);-@@ -15885,7 +15893,7 @@ static enum ggml_status ggml_backend_vk_buffer_init_tensor(ggml_backend_buffer_t+@@ -12513,7 +12521,7 @@ enum ggml_status ggml_backend_vk_buffer_init_tensor(ggml_backend_buffer_t bufferreturn GGML_STATUS_SUCCESS;}--static void ggml_backend_vk_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size) {-+static void GGML_CALL ggml_backend_vk_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size) {+-void ggml_backend_vk_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size) {++void GGML_CALL ggml_backend_vk_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size) {VK_LOG_DEBUG("ggml_backend_vk_buffer_memset_tensor(" << buffer << ", " << tensor << ", " << value << ", " << offset << ", " << size << ")");ggml_backend_vk_buffer_context * buf_ctx = (ggml_backend_vk_buffer_context *)buffer->context;vk_buffer buf = buf_ctx->dev_buffer;-@@ -15898,7 +15906,7 @@ static void ggml_backend_vk_buffer_memset_tensor(ggml_backend_buffer_t buffer, g+@@ -12526,7 +12534,7 @@ void ggml_backend_vk_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tenggml_vk_buffer_memset(buf, vk_tensor_offset(tensor) + tensor->view_offs + offset, val32, size);}--static void ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) {-+static void GGML_CALL ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) {+-void ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) {++void GGML_CALL ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) {VK_LOG_DEBUG("ggml_backend_vk_buffer_set_tensor(" << buffer << ", " << tensor << ", " << data << ", " << offset << ", " << size << ")");ggml_backend_vk_buffer_context * buf_ctx = (ggml_backend_vk_buffer_context *)buffer->context;vk_buffer buf = buf_ctx->dev_buffer;-@@ -15910,7 +15918,7 @@ static void ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml+@@ -12538,7 +12546,7 @@ void ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensorggml_vk_buffer_write(buf, vk_tensor_offset(tensor) + tensor->view_offs + offset, data, size);}--static void ggml_backend_vk_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset,-+static void GGML_CALL ggml_backend_vk_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset,+-void ggml_backend_vk_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset,++void GGML_CALL ggml_backend_vk_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset,size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) {VK_LOG_DEBUG("ggml_backend_vk_buffer_set_tensor_2d(" << buffer << ", " << tensor << ", " << data << ", " << offset << ", " << size << ", " <<n_copies << ", " << stride_tensor << ", " << stride_data << ")");-@@ -15924,7 +15932,7 @@ static void ggml_backend_vk_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, g+@@ -12552,7 +12560,7 @@ void ggml_backend_vk_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, ggml_tenggml_vk_buffer_write_2d(buf, vk_tensor_offset(tensor) + tensor->view_offs + offset, data, stride_data, stride_tensor, size, n_copies);}--static void ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) {-+static void GGML_CALL ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) {+-void ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) {++void GGML_CALL ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) {VK_LOG_DEBUG("ggml_backend_vk_buffer_get_tensor(" << buffer << ", " << tensor << ", " << data << ", " << offset << ", " << size << ")");ggml_backend_vk_buffer_context * buf_ctx = (ggml_backend_vk_buffer_context *)buffer->context;-@@ -15937,7 +15945,7 @@ static void ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer_t buffer, cons+@@ -12565,7 +12573,7 @@ void ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_ggml_vk_buffer_read(buf, vk_tensor_offset(tensor) + tensor->view_offs + offset, data, size);}--static void ggml_backend_vk_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset,-+static void GGML_CALL ggml_backend_vk_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset,+-void ggml_backend_vk_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset,++void GGML_CALL ggml_backend_vk_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset,size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) {VK_LOG_DEBUG("ggml_backend_vk_buffer_get_tensor_2d(" << buffer << ", " << tensor << ", " << data << ", " << offset << ", " << size << ", " <<n_copies << ", " << stride_tensor << ", " << stride_data << ")");-@@ -15952,7 +15960,7 @@ static void ggml_backend_vk_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, c+@@ -12580,7 +12588,7 @@ void ggml_backend_vk_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const ggggml_vk_buffer_read_2d(buf, vk_tensor_offset(tensor) + tensor->view_offs + offset, data, stride_tensor, stride_data, size, n_copies);}--static bool ggml_backend_vk_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * src, ggml_tensor * dst) {-+static bool GGML_CALL ggml_backend_vk_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * src, ggml_tensor * dst) {+-bool ggml_backend_vk_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * src, ggml_tensor * dst) {++bool GGML_CALL ggml_backend_vk_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * src, ggml_tensor * dst) {if (ggml_nbytes(src) == 0) {return true;}-@@ -15973,7 +15981,7 @@ static bool ggml_backend_vk_buffer_cpy_tensor(ggml_backend_buffer_t buffer, cons+@@ -12601,19 +12609,19 @@ bool ggml_backend_vk_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_UNUSED(buffer);}--static void ggml_backend_vk_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) {-+static void GGML_CALL ggml_backend_vk_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) {+-void ggml_backend_vk_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) {++void GGML_CALL ggml_backend_vk_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) {ggml_backend_vk_buffer_context * ctx = (ggml_backend_vk_buffer_context *)buffer->context;ggml_vk_buffer_memset(ctx->dev_buffer, 0, value, buffer->size);-@@ -15991,16 +15999,17 @@ static ggml_backend_buffer_i ggml_backend_vk_buffer_interface = {- /* .cpy_tensor = */ ggml_backend_vk_buffer_cpy_tensor,- /* .clear = */ ggml_backend_vk_buffer_clear,- /* .reset = */ NULL,-+ /* .free_struct = */ ggml_backend_vk_buffer_free_struct,- };+ }- // vk buffer type--static const char * ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft) {-+static const char * GGML_CALL ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft) {+-const char * ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft) {++const char * GGML_CALL ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft) {ggml_backend_vk_buffer_type_context * ctx = (ggml_backend_vk_buffer_type_context *)buft->context;return ctx->name.c_str();}--static ggml_backend_buffer_t ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) {-+static ggml_backend_buffer_t GGML_CALL ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) {+-ggml_backend_buffer_t ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) {++ggml_backend_buffer_t GGML_CALL ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) {VK_LOG_MEMORY("ggml_backend_vk_buffer_type_alloc_buffer(" << size << ")");ggml_backend_vk_buffer_type_context * ctx = (ggml_backend_vk_buffer_type_context *) buft->context;-@@ -16016,17 +16025,17 @@ static ggml_backend_buffer_t ggml_backend_vk_buffer_type_alloc_buffer(ggml_backe+@@ -12629,17 +12637,17 @@ ggml_backend_buffer_t ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffreturn ggml_backend_buffer_init(buft, ggml_backend_vk_buffer_interface, bufctx, size);}--static size_t ggml_backend_vk_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) {-+static size_t GGML_CALL ggml_backend_vk_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) {+-size_t ggml_backend_vk_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) {++size_t GGML_CALL ggml_backend_vk_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) {ggml_backend_vk_buffer_type_context * ctx = (ggml_backend_vk_buffer_type_context *) buft->context;return ctx->device->properties.limits.minStorageBufferOffsetAlignment;}--static size_t ggml_backend_vk_buffer_type_get_max_size(ggml_backend_buffer_type_t buft) {-+static size_t GGML_CALL ggml_backend_vk_buffer_type_get_max_size(ggml_backend_buffer_type_t buft) {+-size_t ggml_backend_vk_buffer_type_get_max_size(ggml_backend_buffer_type_t buft) {++size_t GGML_CALL ggml_backend_vk_buffer_type_get_max_size(ggml_backend_buffer_type_t buft) {ggml_backend_vk_buffer_type_context * ctx = (ggml_backend_vk_buffer_type_context *) buft->context;return ctx->device->suballocation_block_size;}--static size_t ggml_backend_vk_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) {-+static size_t GGML_CALL ggml_backend_vk_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) {+-size_t ggml_backend_vk_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) {++size_t GGML_CALL ggml_backend_vk_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) {return ggml_nbytes(tensor);UNUSED(buft);-@@ -16044,18 +16053,18 @@ ggml_backend_buffer_type_t ggml_backend_vk_buffer_type(size_t dev_num) {-- // host buffer type+@@ -12655,18 +12663,18 @@ ggml_backend_buffer_type_t ggml_backend_vk_buffer_type(size_t dev_num) {+ return &dev->buffer_type;+ }-static const char * ggml_backend_vk_host_buffer_type_name(ggml_backend_buffer_type_t buft) {+static const char * GGML_CALL ggml_backend_vk_host_buffer_type_name(ggml_backend_buffer_type_t buft) {@@ -185,7 +153,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vuVK_LOG_MEMORY("ggml_backend_vk_host_buffer_type_alloc_buffer(" << size << ")");size += 32; // Behave like the CPU buffer type-@@ -16071,19 +16080,20 @@ static ggml_backend_buffer_t ggml_backend_vk_host_buffer_type_alloc_buffer(ggml_+@@ -12682,19 +12690,20 @@ static ggml_backend_buffer_t ggml_backend_vk_host_buffer_type_alloc_buffer(ggml_ggml_backend_buffer_t buffer = ggml_backend_cpu_buffer_from_ptr(ptr, size);buffer->buft = buft;buffer->iface.free_buffer = ggml_backend_vk_host_buffer_free_buffer;@@ -208,9 +176,9 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vureturn vk_instance.devices[0]->suballocation_block_size;UNUSED(buft);-@@ -16115,13 +16125,13 @@ ggml_backend_buffer_type_t ggml_backend_vk_host_buffer_type() {-- // backend+@@ -12721,13 +12730,13 @@ ggml_backend_buffer_type_t ggml_backend_vk_host_buffer_type() {+ return &ggml_backend_vk_buffer_type_host;+ }-static const char * ggml_backend_vk_name(ggml_backend_t backend) {+static const char * GGML_CALL ggml_backend_vk_name(ggml_backend_t backend) {@@ -219,12 +187,12 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vureturn ctx->name.c_str();}--static void ggml_backend_vk_free(ggml_backend_t backend) {-+static void GGML_CALL ggml_backend_vk_free(ggml_backend_t backend) {+-void ggml_backend_vk_free(ggml_backend_t backend) {++void GGML_CALL ggml_backend_vk_free(ggml_backend_t backend) {ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;VK_LOG_DEBUG("ggml_backend_vk_free(" << ctx->name << ")");-@@ -16137,7 +16147,7 @@ static ggml_backend_buffer_type_t ggml_backend_vk_get_default_buffer_type(ggml_b+@@ -12743,7 +12752,7 @@ static ggml_backend_buffer_type_t ggml_backend_vk_get_default_buffer_type(ggml_breturn &ctx->device->buffer_type;}@@ -233,7 +201,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vusize_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) {VK_LOG_DEBUG("ggml_backend_vk_set_tensor_2d_async(" << size << ", " << n_copies << ")");ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;-@@ -16195,12 +16205,12 @@ static void ggml_backend_vk_set_tensor_2d_async(ggml_backend_t backend, ggml_ten+@@ -12801,12 +12810,12 @@ static void ggml_backend_vk_set_tensor_2d_async(ggml_backend_t backend, ggml_ten}}@@ -248,7 +216,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vusize_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) {VK_LOG_DEBUG("ggml_backend_vk_get_tensor_2d_async(" << size << ", " << n_copies << ")");ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;-@@ -16251,12 +16261,12 @@ static void ggml_backend_vk_get_tensor_2d_async(ggml_backend_t backend, const gg+@@ -12857,12 +12866,12 @@ static void ggml_backend_vk_get_tensor_2d_async(ggml_backend_t backend, const gg}}@@ -263,7 +231,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vuVK_LOG_DEBUG("ggml_backend_vk_cpy_tensor_async(" << src << " -> " << dst << ", size=" << ggml_nbytes(src) << ")");ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend_dst->context;-@@ -16383,7 +16393,7 @@ static void ggml_vk_synchronize(ggml_backend_vk_context * ctx) {+@@ -13004,7 +13013,7 @@ void ggml_vk_synchronize(ggml_backend_vk_context * ctx) {}}@@ -272,7 +240,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vuVK_LOG_DEBUG("ggml_backend_vk_synchronize()");ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;-@@ -16935,7 +16945,7 @@ static int32_t find_first_set(uint32_t x) {+@@ -13741,7 +13750,7 @@ static int32_t find_first_set(uint32_t x) {return ret;}@@ -281,16 +249,16 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vuVK_LOG_DEBUG("ggml_backend_vk_graph_compute(" << cgraph->n_nodes << " nodes)");ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;-@@ -17400,7 +17410,7 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg+@@ -14236,7 +14245,7 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg+ UNUSED(backend);}- // Sort the graph for improved parallelism.--static void ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph * graph)-+static void GGML_CALL ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph * graph)+-void ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph * graph, struct ggml_backend_graph_optimize_params * params)++void GGML_CALL ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph * graph, struct ggml_backend_graph_optimize_params * params){VK_LOG_DEBUG("ggml_vk_graph_optimize(" << graph->n_nodes << " nodes)");ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;-@@ -17661,7 +17671,7 @@ static void ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph *+@@ -14617,7 +14626,7 @@ void ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph * graph,}}@@ -299,7 +267,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vuVK_LOG_DEBUG("ggml_backend_vk_event_record(backend=" << backend << ", event=" << event << ")");ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;vk_event *vkev = (vk_event *)event->context;-@@ -17699,7 +17709,7 @@ static void ggml_backend_vk_event_record(ggml_backend_t backend, ggml_backend_ev+@@ -14655,7 +14664,7 @@ static void ggml_backend_vk_event_record(ggml_backend_t backend, ggml_backend_evctx->compute_ctx.reset();}@@ -308,7 +276,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vuVK_LOG_DEBUG("ggml_backend_vk_event_wait(backend=" << backend << ", event=" << event << ")");ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;vk_event *vkev = (vk_event *)event->context;-@@ -17801,7 +17811,10 @@ void ggml_backend_vk_get_device_memory(int device, size_t * free, size_t * total+@@ -14756,7 +14765,10 @@ void ggml_backend_vk_get_device_memory(int device, size_t * free, size_t * total*total += heap.size;if (membudget_supported && i < budgetprops.heapUsage.size()) {@@ -320,9 +288,9 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu} else {*free += heap.size;}-@@ -17869,38 +17882,38 @@ struct ggml_backend_vk_device_context {- int op_offload_min_batch_size;- };+@@ -14813,38 +14825,38 @@ static std::string ggml_backend_vk_get_device_pci_id(int device_idx) {+ return std::string(pci_bus_id);+ }-static const char * ggml_backend_vk_device_get_name(ggml_backend_dev_t dev) {+static const char * GGML_CALL ggml_backend_vk_device_get_name(ggml_backend_dev_t dev) {@@ -366,7 +334,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vuggml_backend_vk_device_context * ctx = (ggml_backend_vk_device_context *)dev->context;props->name = ggml_backend_vk_device_get_name(dev);-@@ -17917,13 +17930,13 @@ static void ggml_backend_vk_device_get_props(ggml_backend_dev_t dev, struct ggml+@@ -14861,13 +14873,13 @@ static void ggml_backend_vk_device_get_props(ggml_backend_dev_t dev, struct ggml};}@@ -382,7 +350,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vuggml_backend_vk_device_context * ctx = (ggml_backend_vk_device_context *)dev->context;const vk_device& device = ggml_vk_get_device(ctx->device);-@@ -18504,7 +18517,7 @@ static bool ggml_backend_vk_device_supports_op(ggml_backend_dev_t dev, const ggm+@@ -15522,7 +15534,7 @@ static bool ggml_backend_vk_device_supports_op(ggml_backend_dev_t dev, const ggmUNUSED(dev);}@@ -391,8 +359,8 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vuif (buft->iface.get_name != ggml_backend_vk_buffer_type_name) {return false;}-@@ -18530,13 +18543,13 @@ static int64_t ggml_vk_get_op_batch_size(const ggml_tensor * op) {- }+@@ -15533,13 +15545,13 @@ static bool ggml_backend_vk_device_supports_buft(ggml_backend_dev_t dev, ggml_ba+ return buft_ctx->device->idx == ctx->device;}-static bool ggml_backend_vk_device_offload_op(ggml_backend_dev_t dev, const ggml_tensor * op) {@@ -407,7 +375,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vuggml_backend_vk_device_context * ctx = (ggml_backend_vk_device_context *)dev->context;auto device = ggml_vk_get_device(ctx->device);-@@ -18559,7 +18572,7 @@ static ggml_backend_event_t ggml_backend_vk_device_event_new(ggml_backend_dev_t+@@ -15562,7 +15574,7 @@ static ggml_backend_event_t ggml_backend_vk_device_event_new(ggml_backend_dev_t};}@@ -416,7 +384,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vuggml_backend_vk_device_context * ctx = (ggml_backend_vk_device_context *)dev->context;auto device = ggml_vk_get_device(ctx->device);-@@ -18579,7 +18592,7 @@ static void ggml_backend_vk_device_event_free(ggml_backend_dev_t dev, ggml_backe+@@ -15582,7 +15594,7 @@ static void ggml_backend_vk_device_event_free(ggml_backend_dev_t dev, ggml_backedelete event;}@@ -425,8 +393,8 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vuVK_LOG_DEBUG("ggml_backend_vk_device_event_synchronize(backend=" << dev << ", event=" << event << ")");ggml_backend_vk_device_context * ctx = (ggml_backend_vk_device_context *)dev->context;auto device = ggml_vk_get_device(ctx->device);-@@ -18636,7 +18649,7 @@ static vk_buffer ggml_vk_buffer_from_host_ptr(vk_device & device, void * ptr, si- return buf;+@@ -15614,7 +15626,7 @@ static void ggml_backend_vk_device_event_synchronize(ggml_backend_dev_t dev, ggm+ }}-static ggml_backend_buffer_t ggml_backend_vk_device_buffer_from_host_ptr(ggml_backend_dev_t dev, void * ptr, size_t size, size_t max_tensor_size) {@@ -434,7 +402,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vuVK_LOG_DEBUG("ggml_backend_vk_device_buffer_from_host_ptr(backend=" << dev << ", ptr=" << ptr << ", size=" << size << ")");GGML_UNUSED(max_tensor_size);-@@ -18674,17 +18687,17 @@ static const struct ggml_backend_device_i ggml_backend_vk_device_i = {+@@ -15652,17 +15664,17 @@ static const struct ggml_backend_device_i ggml_backend_vk_device_i = {/* .event_synchronize = */ ggml_backend_vk_device_event_synchronize,};diff --git a/llama.cpp.patches/patches/ggml_src_ggml.c.patch b/llama.cpp.patches/patches/ggml_src_ggml.c.patchindex 6b38ed86..d74076b8 100644--- a/llama.cpp.patches/patches/ggml_src_ggml.c.patch+++ b/llama.cpp.patches/patches/ggml_src_ggml.c.patch@@ -1,7 +1,7 @@diff --git a/ggml/src/ggml.c b/ggml/src/ggml.c--- a/llama.cpp/ggml/src/ggml.c+++ b/llama.cpp/ggml/src/ggml.c-@@ -547,11 +547,13 @@ void ggml_time_init(void) {+@@ -548,11 +548,13 @@ void ggml_time_init(void) {InitOnceExecuteOnce(&once, ggml_time_init_once, NULL, NULL);}int64_t ggml_time_ms(void) {diff --git a/llama.cpp.patches/patches/ggml_src_gguf.cpp.patch b/llama.cpp.patches/patches/ggml_src_gguf.cpp.patchindex 1707e908..0036961e 100644--- a/llama.cpp.patches/patches/ggml_src_gguf.cpp.patch+++ b/llama.cpp.patches/patches/ggml_src_gguf.cpp.patch@@ -12,7 +12,7 @@ diff --git a/ggml/src/gguf.cpp b/ggml/src/gguf.cpp#include <cinttypes>#include <cstddef>#include <cstdint>-@@ -989,7 +993,54 @@ struct gguf_context * gguf_init_from_buffer(const void * data, size_t size, stru+@@ -997,7 +1001,54 @@ struct gguf_context * gguf_init_from_buffer(const void * data, size_t size, strureturn gguf_init_from_reader(gr, params);}@@ -67,7 +67,7 @@ diff --git a/ggml/src/gguf.cpp b/ggml/src/gguf.cppFILE * file = ggml_fopen(fname, "rb");if (!file) {-@@ -1000,6 +1051,7 @@ struct gguf_context * gguf_init_from_file(const char * fname, struct gguf_init_p+@@ -1008,6 +1059,7 @@ struct gguf_context * gguf_init_from_file(const char * fname, struct gguf_init_pstruct gguf_context * result = gguf_init_from_file_ptr(file, params);fclose(file);return result;diff --git a/llama.cpp.patches/patches/src_llama-context.cpp.patch b/llama.cpp.patches/patches/src_llama-context.cpp.patchindex d6906466..f99ba6a1 100644--- a/llama.cpp.patches/patches/src_llama-context.cpp.patch+++ b/llama.cpp.patches/patches/src_llama-context.cpp.patch@@ -1,7 +1,7 @@diff --git a/src/llama-context.cpp b/src/llama-context.cpp--- a/llama.cpp/src/llama-context.cpp+++ b/llama.cpp/src/llama-context.cpp-@@ -551,6 +551,16 @@ void llama_context::resolve_fused_ops(const llama_memory_context_i * mctx, uint3+@@ -553,6 +553,16 @@ void llama_context::resolve_fused_ops(const llama_memory_context_i * mctx, uint3}};diff --git a/llama.cpp.patches/patches/src_llama-mmap.cpp.patch b/llama.cpp.patches/patches/src_llama-mmap.cpp.patchindex a15e763b..f0208df5 100644--- a/llama.cpp.patches/patches/src_llama-mmap.cpp.patch+++ b/llama.cpp.patches/patches/src_llama-mmap.cpp.patch@@ -11,8 +11,8 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp+#include <cstring>#include <climits>- #include <stdexcept>-@@ -181,6 +185,15 @@ struct llama_file::impl {+ #include <cstdlib>+@@ -182,6 +186,15 @@ struct llama_file::impl {}#elseimpl(const char * fname, const char * mode, [[maybe_unused]] const bool use_direct_io = false) : fname(fname) {@@ -28,7 +28,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp#ifdef __linux__// Try unbuffered I/O for read onlyif (use_direct_io && std::strcmp(mode, "rb") == 0) {-@@ -233,6 +246,9 @@ struct llama_file::impl {+@@ -234,6 +247,9 @@ struct llama_file::impl {}size_t tell() const {@@ -38,7 +38,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cppif (fd == -1) {off_t ret = llama_mmap_ftell(fp);if (ret == -1) {-@@ -250,6 +266,11 @@ struct llama_file::impl {+@@ -251,6 +267,11 @@ struct llama_file::impl {}void seek(size_t offset, int whence) const {@@ -50,7 +50,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cppoff_t ret = 0;if (fd == -1) {ret = llama_mmap_fseek(fp, offset, whence);-@@ -259,12 +280,22 @@ struct llama_file::impl {+@@ -260,12 +281,22 @@ struct llama_file::impl {if (ret == -1) {throw std::runtime_error(format("seek error: %s", strerror(errno)));}@@ -73,7 +73,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpperrno = 0;if (fd == -1) {const size_t curr_off = tell();-@@ -314,6 +345,7 @@ struct llama_file::impl {+@@ -315,6 +346,7 @@ struct llama_file::impl {bytes_read += (size_t) ret;}}@@ -81,7 +81,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp}void read_aligned_chunk(void * dest, size_t size) {-@@ -374,13 +406,20 @@ struct llama_file::impl {+@@ -375,13 +407,20 @@ struct llama_file::impl {}~impl() {@@ -102,7 +102,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cppstd::string fname;#endif-@@ -409,6 +448,10 @@ size_t llama_file::read_alignment() const { return pimpl->read_alignment(); }+@@ -410,6 +449,10 @@ size_t llama_file::read_alignment() const { return pimpl->read_alignment(); }bool llama_file::has_direct_io() const { return pimpl->has_direct_io(); }int llama_file::file_id() const {@@ -113,7 +113,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp#ifdef _WIN32return _fileno(pimpl->fp);#else-@@ -416,9 +459,10 @@ int llama_file::file_id() const {+@@ -417,9 +460,10 @@ int llama_file::file_id() const {return pimpl->fd;}#if defined(fileno)@@ -126,7 +126,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp#endif#endif}-@@ -436,14 +480,56 @@ uint32_t llama_file::read_u32() { return pimpl->read_u32(); }+@@ -437,6 +481,31 @@ uint32_t llama_file::read_u32() { return pimpl->read_u32(); }void llama_file::write_raw(const void * ptr, size_t len) const { pimpl->write_raw(ptr, len); }void llama_file::write_u32(uint32_t val) const { pimpl->write_u32(val); }@@ -157,6 +157,8 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp+// llama_mmap+ #if defined(_POSIX_MAPPED_FILES) || defined(_WIN32)+@@ -465,9 +534,26 @@ static llama_mmap::ranges ranges_complement(llama_mmap::ranges ranges, size_t listruct llama_mmap::impl {#ifdef _POSIX_MAPPED_FILESstd::vector<std::pair<size_t, size_t>> mapped_fragments;@@ -165,7 +167,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp+ llamafile * lfile = nullptr;+#endif- impl(struct llama_file * file, size_t prefetch, bool numa) {+ impl(struct llama_file * file, size_t prefetch, bool numa, const llama_mmap::ranges & lazy_ranges) {size = file->size();++ if (file->has_premapped_content()) {@@ -183,7 +185,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cppint fd = file->file_id();int flags = MAP_SHARED;if (numa) { prefetch = 0; }-@@ -488,6 +574,10 @@ struct llama_mmap::impl {+@@ -528,6 +614,10 @@ struct llama_mmap::impl {}void unmap_fragment(size_t first, size_t last) {@@ -194,7 +196,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cppint page_size = sysconf(_SC_PAGESIZE);align_range(&first, &last, page_size);size_t len = last - first;-@@ -524,6 +614,14 @@ struct llama_mmap::impl {+@@ -564,6 +654,14 @@ struct llama_mmap::impl {}~impl() {diff --git a/llama.cpp.patches/patches/src_llama-mmap.h.patch b/llama.cpp.patches/patches/src_llama-mmap.h.patchindex 39fbf730..6b6ba5eb 100644--- a/llama.cpp.patches/patches/src_llama-mmap.h.patch+++ b/llama.cpp.patches/patches/src_llama-mmap.h.patch@@ -1,7 +1,7 @@diff --git a/src/llama-mmap.h b/src/llama-mmap.h--- a/llama.cpp/src/llama-mmap.h+++ b/llama.cpp/src/llama-mmap.h-@@ -35,6 +35,11 @@ struct llama_file {+@@ -36,6 +36,11 @@ struct llama_file {size_t read_alignment() const;bool has_direct_io() const;diff --git a/llama.cpp.patches/patches/src_models_dflash.cpp.patch b/llama.cpp.patches/patches/src_models_dflash.cpp.patchdeleted file mode 100644index b9d6e9ac..00000000--- a/llama.cpp.patches/patches/src_models_dflash.cpp.patch+++ /dev/null@@ -1,48 +0,0 @@-diff --git a/src/models/dflash.cpp b/src/models/dflash.cpp---- a/llama.cpp/src/models/dflash.cpp-+++ b/llama.cpp/src/models/dflash.cpp-@@ -177,21 +177,6 @@ void llama_model_dflash::load_arch_tensors(llama_model_loader &) {- }- }---std::unique_ptr<llm_graph_context> llama_model_dflash::build_arch_graph(const llm_graph_params & params) const {-- switch (params.gtype) {-- case LLM_GRAPH_TYPE_ENCODER:-- return std::make_unique<graph<true>>(*this, params);-- case LLM_GRAPH_TYPE_DEFAULT:-- case LLM_GRAPH_TYPE_DECODER:-- if (hparams.dsv4_hc_mult > 0) {-- return std::make_unique<graph_dsv4>(*this, params);-- }-- return std::make_unique<graph<false>>(*this, params);-- default:-- GGML_ABORT("invalid graph type");-- };--}--- template <>- ggml_tensor * llama_model_dflash::graph<true>::build_inp_embd_enc() const {- auto inp_target = std::make_unique<llm_graph_input_embd>(hparams.n_embd_inp_enc());-@@ -684,3 +669,22 @@ llama_model_dflash::graph_dsv4::graph_dsv4(const llama_model & model, const llm_- build_dspark_markov_head(*this, model, inp_tokens);- }- }-+-+// Defined after the graph<true>/graph<false>/graph_dsv4 constructor-+// specializations above: cosmocc's clang rejects an explicit specialization-+// that appears after the point where make_unique<graph<...>> would implicitly-+// instantiate it.-+std::unique_ptr<llm_graph_context> llama_model_dflash::build_arch_graph(const llm_graph_params & params) const {-+ switch (params.gtype) {-+ case LLM_GRAPH_TYPE_ENCODER:-+ return std::make_unique<graph<true>>(*this, params);-+ case LLM_GRAPH_TYPE_DEFAULT:-+ case LLM_GRAPH_TYPE_DECODER:-+ if (hparams.dsv4_hc_mult > 0) {-+ return std::make_unique<graph_dsv4>(*this, params);-+ }-+ return std::make_unique<graph<false>>(*this, params);-+ default:-+ GGML_ABORT("invalid graph type");-+ };-+}diff --git a/llama.cpp.patches/patches/src_models_eagle3.cpp.patch b/llama.cpp.patches/patches/src_models_eagle3.cpp.patchdeleted file mode 100644index 35f204e1..00000000--- a/llama.cpp.patches/patches/src_models_eagle3.cpp.patch+++ /dev/null@@ -1,41 +0,0 @@-diff --git a/src/models/eagle3.cpp b/src/models/eagle3.cpp---- a/llama.cpp/src/models/eagle3.cpp-+++ b/llama.cpp/src/models/eagle3.cpp-@@ -100,18 +100,6 @@ void llama_model_eagle3::load_arch_tensors(llama_model_loader &) {- }- }---std::unique_ptr<llm_graph_context> llama_model_eagle3::build_arch_graph(const llm_graph_params & params) const {-- switch (params.gtype) {-- case LLM_GRAPH_TYPE_ENCODER:-- return std::make_unique<graph<true>>(*this, params);-- case LLM_GRAPH_TYPE_DEFAULT:-- case LLM_GRAPH_TYPE_DECODER:-- return std::make_unique<graph<false>>(*this, params);-- default:-- GGML_ABORT("invalid graph type");-- };--}--- template <>- ggml_tensor * llama_model_eagle3::graph<true>::build_inp_embd_enc() const {- ggml_tensor * cur = nullptr;-@@ -336,3 +324,18 @@ llama_model_eagle3::graph<false>::graph(const llama_model & model, const llm_gra-- ggml_build_forward_expand(gf, cur);- }-+-+// Defined after the graph<true>/graph<false> constructor specializations above:-+// cosmocc's clang rejects an explicit specialization that appears after the-+// point where make_unique<graph<...>> would implicitly instantiate it.-+std::unique_ptr<llm_graph_context> llama_model_eagle3::build_arch_graph(const llm_graph_params & params) const {-+ switch (params.gtype) {-+ case LLM_GRAPH_TYPE_ENCODER:-+ return std::make_unique<graph<true>>(*this, params);-+ case LLM_GRAPH_TYPE_DEFAULT:-+ case LLM_GRAPH_TYPE_DECODER:-+ return std::make_unique<graph<false>>(*this, params);-+ default:-+ GGML_ABORT("invalid graph type");-+ };-+}diff --git a/llama.cpp.patches/patches/src_models_t5.cpp.patch b/llama.cpp.patches/patches/src_models_t5.cpp.patchindex 40e2a98a..6bb5ae7d 100644--- a/llama.cpp.patches/patches/src_models_t5.cpp.patch+++ b/llama.cpp.patches/patches/src_models_t5.cpp.patch@@ -1,8 +1,8 @@diff --git a/src/models/t5.cpp b/src/models/t5.cpp--- a/llama.cpp/src/models/t5.cpp+++ b/llama.cpp/src/models/t5.cpp-@@ -106,6 +106,12 @@ void llama_model_t5::load_arch_tensors(llama_model_loader &) {- }+@@ -357,6 +357,12 @@ llama_model_t5::graph<true>::graph(const llama_model & model, const llm_graph_pa+ ggml_build_forward_expand(gf, cur);}+// Forward-declare the explicit specializations so that the std::make_uniquediff --git a/llama.cpp.patches/patches/tools_server_server-models.cpp.patch b/llama.cpp.patches/patches/tools_server_server-models.cpp.patchindex f9ef1320..ab633263 100644--- a/llama.cpp.patches/patches/tools_server_server-models.cpp.patch+++ b/llama.cpp.patches/patches/tools_server_server-models.cpp.patch@@ -1,7 +1,7 @@diff --git a/tools/server/server-models.cpp b/tools/server/server-models.cpp--- a/llama.cpp/tools/server/server-models.cpp+++ b/llama.cpp/tools/server/server-models.cpp-@@ -30,6 +30,7 @@+@@ -31,6 +31,7 @@#include <cstring>#ifndef _WIN32@@ -9,7 +9,34 @@ diff --git a/tools/server/server-models.cpp b/tools/server/server-models.cppextern char **environ;#endif-@@ -675,13 +676,19 @@ void server_models::load_models() {+@@ -61,7 +62,25 @@ static void request_child_exit(server_subproc & proc) {+ // single-threaded, watching all child processes at once+ struct server_monitor {+ server_monitor(server_models & models) : models(models) {+- th = std::thread([this]() { run(); });++ th = std::thread([this]() {++ // Block every signal on this thread so its waits can't be++ // interrupted with EINTR (libcxx rethrows EINTR from a condition++ // variable as std::system_error and std::terminates the process).++ // Keep synchronous fault signals deliverable so real crashes still++ // surface. SIGINT/SIGTERM belong to the main thread.++ sigset_t block_mask;++ sigfillset(&block_mask);++ sigdelset(&block_mask, SIGSEGV);++ sigdelset(&block_mask, SIGBUS);++ sigdelset(&block_mask, SIGILL);++ sigdelset(&block_mask, SIGFPE);++ sigdelset(&block_mask, SIGABRT);++ sigdelset(&block_mask, SIGSYS);++ sigdelset(&block_mask, SIGTRAP);++ pthread_sigmask(SIG_BLOCK, &block_mask, nullptr);++++ run();++ });+ }++ ~server_monitor() {+@@ -893,13 +912,19 @@ void server_models::load_models() {}// wait for all targeted models to reach UNLOADED; cv.wait handles unlock/relock@@ -29,9 +56,9 @@ diff --git a/tools/server/server-models.cpp b/tools/server/server-models.cpp+ cv.wait_for(lk, std::chrono::seconds(30));+ }- // collect all threads to join in one pass while the lock is held:- // - monitoring threads from just-unloaded models (to_unload)-@@ -913,9 +920,12 @@ void server_models::unload_lru() {+ // erase models no longer in any source+ for (auto it = mapping.begin(); it != mapping.end(); ) {+@@ -1096,9 +1121,12 @@ void server_models::unload_lru() {// wait for unload to complete{std::unique_lock<std::mutex> lk(mutex);@@ -47,7 +74,7 @@ diff --git a/tools/server/server-models.cpp b/tools/server/server-models.cpp}}}-@@ -940,7 +950,12 @@ void server_models::load(const std::string & name, const load_options & opts) {+@@ -1123,7 +1151,12 @@ void server_models::load(const std::string & name, const load_options & opts) {std::unique_lock<std::mutex> lk(mutex);// edge case: block until any in-progress reload has finished so we always load// against the freshest preset and a consistent mapping state@@ -61,45 +88,30 @@ diff --git a/tools/server/server-models.cpp b/tools/server/server-models.cppauto meta = opts.custom_meta.has_value() ? *opts.custom_meta : mapping[name].meta;if (meta.status != SERVER_MODEL_STATUS_UNLOADED) {-@@ -1037,15 +1052,34 @@ void server_models::load(const std::string & name, const load_options & opts) {- });-- std::thread stopping_thread([&]() {-+ // Block every signal on this thread so pthread_cond_timedwait()-+ // can't be interrupted with EINTR (libcxx rethrows EINTR as-+ // std::system_error and std::terminates the process). Keep-+ // synchronous fault signals deliverable so real crashes still-+ // surface.-+ sigset_t block_mask;-+ sigfillset(&block_mask);-+ sigdelset(&block_mask, SIGSEGV);-+ sigdelset(&block_mask, SIGBUS);-+ sigdelset(&block_mask, SIGILL);-+ sigdelset(&block_mask, SIGFPE);-+ sigdelset(&block_mask, SIGABRT);-+ sigdelset(&block_mask, SIGSYS);-+ sigdelset(&block_mask, SIGTRAP);-+ pthread_sigmask(SIG_BLOCK, &block_mask, nullptr);-+- // thread to monitor explicit stop requests; child crash is signalled via child_proc->stopped- auto is_stopping = [this, &name]() {- return this->stopping_models.find(name) != this->stopping_models.end();- };- {- std::unique_lock<std::mutex> lk(this->mutex);-- this->cv_stop.wait(lk, [&]() {-- return is_stopping() || child_proc->stopped.load(std::memory_order_acquire);-- });-+ // Use wait_for() instead of wait() to work around a Cosmopolitan-+ // libc bug where untimed futex waits on XNU (macOS) expire after-+ // ~72 minutes, causing condition_variable::wait() to throw ETIMEDOUT.-+ while (!(is_stopping() || child_proc->stopped.load(std::memory_order_acquire))) {-+ this->cv_stop.wait_for(lk, std::chrono::seconds(30));-+ }+@@ -1283,14 +1316,20 @@ void server_models::unload_all() {+ }+ }+ // wait for every child to exit, the monitor force-kills the ones that ignore the exit command+- cv.wait(lk, [this]() {++ // Use wait_for() instead of wait() to work around a Cosmopolitan libc bug++ // where untimed futex waits on XNU (macOS) expire after ~72 minutes,++ // causing condition_variable::wait() to throw ETIMEDOUT.++ auto all_stopped = [this]() {+ for (const auto & [name, inst] : mapping) {+ if (inst.meta.is_running() || inst.meta.status == SERVER_MODEL_STATUS_DOWNLOADING) {+ return false;}- // child crashed or finished on its own, skip graceful shutdown sequence- if (child_proc->stopped.load(std::memory_order_acquire)) {-@@ -1310,16 +1344,22 @@ void server_models::wait(const std::string & name, std::function<bool(const serv+ }+ return true;+- });++ };++ while (!all_stopped()) {++ cv.wait_for(lk, std::chrono::seconds(30));++ }+ }++ void server_models::update_status(const std::string & name, const update_status_args & args) {+@@ -1422,16 +1461,22 @@ void server_models::wait(const std::string & name, std::function<bool(const serv}void server_models::wait(std::unique_lock<std::mutex> & lk, const std::string & name, std::function<bool(const server_model_meta &)> predicate) {@@ -129,7 +141,7 @@ diff --git a/tools/server/server-models.cpp b/tools/server/server-models.cpp}bool server_models::ensure_model_ready(const std::string & name, const std::function<bool()> & should_stop) {-@@ -2347,10 +2387,10 @@ server_http_proxy::server_http_proxy(+@@ -2448,10 +2493,10 @@ server_http_proxy::server_http_proxy(auto pipe = std::make_shared<server_pipe<msg_t>>();if (scheme == "https") {diff --git a/llama.cpp.patches/patches/tools_server_server-queue.cpp.patch b/llama.cpp.patches/patches/tools_server_server-queue.cpp.patchindex 236bd692..80e0939b 100644--- a/llama.cpp.patches/patches/tools_server_server-queue.cpp.patch+++ b/llama.cpp.patches/patches/tools_server_server-queue.cpp.patch@@ -1,8 +1,8 @@diff --git a/tools/server/server-queue.cpp b/tools/server/server-queue.cpp--- a/llama.cpp/tools/server/server-queue.cpp+++ b/llama.cpp/tools/server/server-queue.cpp-@@ -5,6 +5,13 @@-+@@ -6,6 +6,13 @@+ #include <algorithm>#include <chrono>#include <thread>+#include <cerrno>@@ -15,7 +15,7 @@ diff --git a/tools/server/server-queue.cpp b/tools/server/server-queue.cpp#define QUE_INF(fmt, ...) LOG_INF("que %12.*s: " fmt, 12, __func__, __VA_ARGS__)#define QUE_WRN(fmt, ...) LOG_WRN("que %12.*s: " fmt, 12, __func__, __VA_ARGS__)-@@ -111,9 +118,13 @@ void server_queue::wait_until_no_sleep() {+@@ -123,9 +130,13 @@ void server_queue::wait_until_no_sleep() {condition_tasks.notify_one(); // only main thread is waiting on this}QUE_DBG("%s", "waiting until no sleep\n");@@ -32,24 +32,41 @@ diff --git a/tools/server/server-queue.cpp b/tools/server/server-queue.cpp}}-@@ -153,9 +164,13 @@ void server_queue::worker_loop() {- std::function<void()> work;+@@ -165,9 +176,13 @@ void server_queue::worker_loop() {{std::unique_lock<std::mutex> lock(mutex_tasks);+ // wait on busy instead of yielding - busy stays set even when the yield already ended- worker.cv.wait(lock, [&]{-- return worker.stop || worker.work != nullptr;+- return worker.stop || worker.busy;- });+ // Use wait_for() instead of wait() to work around a+ // Cosmopolitan libc bug where untimed futex waits on XNU+ // (macOS) expire after ~72 minutes, causing+ // condition_variable::wait() to throw ETIMEDOUT.-+ while (!worker.stop && worker.work == nullptr) {++ while (!worker.stop && !worker.busy) {+ worker.cv.wait_for(lock, std::chrono::seconds(30));+ }if (worker.stop) {return;}-@@ -192,6 +207,21 @@ void server_queue::worker_stop() {+@@ -192,9 +207,13 @@ void server_queue::worker_loop() {+ if (!queue_tasks.empty()) {+ continue; // a new task arrived in the meantime+ }+- condition_tasks.wait(lock, [&]{+- return worker.stop || !running || !worker.yielding || !queue_tasks.empty();+- });++ // Use wait_for() instead of wait() to work around a++ // Cosmopolitan libc bug where untimed futex waits on XNU++ // (macOS) expire after ~72 minutes, causing++ // condition_variable::wait() to throw ETIMEDOUT.++ while (!worker.stop && running && worker.yielding && queue_tasks.empty()) {++ condition_tasks.wait_for(lock, std::chrono::seconds(30));++ }+ }++ // signal to yield_to_queue() that no more tasks will be processed+@@ -220,6 +239,21 @@ void server_queue::worker_stop() {}void server_queue::yield_to_queue(std::function<void()> && work) {@@ -71,24 +88,24 @@ diff --git a/tools/server/server-queue.cpp b/tools/server/server-queue.cppGGML_ASSERT(worker.thread.joinable() && "yield_to_queue() requires start_loop() to be running");QUE_DBG("%s", "yielding to queue\n");-@@ -210,9 +240,13 @@ void server_queue::yield_to_queue(std::function<void()> && work) {-- std::unique_lock<std::mutex> lock(mutex_tasks);- // declined tasks are moved to queue_tasks_unhandled, so a non-empty queue always has something new+@@ -246,9 +280,13 @@ void server_queue::yield_to_queue(std::function<void()> && work) {+ // the yield is over, wait for the worker to finish its current task+ worker.yielding = false;+ condition_tasks.notify_all();- condition_tasks.wait(lock, [&]{-- return !worker.busy || (running && !queue_tasks.empty());+- return !worker.busy;- });+ // Use wait_for() instead of wait() to work around a+ // Cosmopolitan libc bug where untimed futex waits on XNU+ // (macOS) expire after ~72 minutes, causing+ // condition_variable::wait() to throw ETIMEDOUT.-+ while (worker.busy && !(running && !queue_tasks.empty())) {++ while (worker.busy) {+ condition_tasks.wait_for(lock, std::chrono::seconds(30));+ }- if (!worker.busy) {- break;- }-@@ -244,6 +278,21 @@ void server_queue::yield_to_queue(std::function<void()> && work) {++ // put the declined tasks back, keeping their order+ while (!queue_tasks_unhandled.empty()) {+@@ -276,6 +314,21 @@ void server_queue::yield_to_queue(std::function<void()> && work) {}void server_queue::start_loop(int64_t idle_sleep_ms) {@@ -110,8 +127,8 @@ diff --git a/tools/server/server-queue.cpp b/tools/server/server-queue.cpprunning = true;time_last_task = ggml_time_ms();-@@ -293,9 +342,13 @@ void server_queue::start_loop(int64_t idle_sleep_ms) {- callback_sleeping_state(true);+@@ -333,9 +386,13 @@ void server_queue::start_loop(int64_t idle_sleep_ms) {+ }req_stop_sleeping = false;// wait until we are requested to exit sleeping state- condition_tasks.wait(lock, [&]{@@ -127,7 +144,7 @@ diff --git a/tools/server/server-queue.cpp b/tools/server/server-queue.cppif (!running) { // may changed during sleepbreak; // terminate}-@@ -384,13 +437,17 @@ void server_response::remove_waiting_task_ids(const std::unordered_set<int> & id+@@ -427,13 +484,17 @@ void server_response::remove_waiting_task_ids(const std::unordered_set<int> & idserver_task_result_ptr server_response::recv(const std::unordered_set<int> & id_tasks) {while (true) {std::unique_lock<std::mutex> lock(mutex_results);diff --git a/llama.cpp.patches/patches/tools_server_server.cpp.patch b/llama.cpp.patches/patches/tools_server_server.cpp.patchindex bdc752d9..e6b7e53d 100644--- a/llama.cpp.patches/patches/tools_server_server.cpp.patch+++ b/llama.cpp.patches/patches/tools_server_server.cpp.patch@@ -70,7 +70,7 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cppcommon_init();// start the stream session manager GC right after common init, before any HTTP route can-@@ -109,10 +141,12 @@ int llama_server(int argc, char ** argv) {+@@ -111,10 +143,12 @@ int llama_server(int argc, char ** argv) {llama_backend_init();llama_numa_init(params.numa);@@ -85,7 +85,7 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cppbool is_run_by_cli = (argv == nullptr);common_models_handler models_handler;-@@ -435,6 +469,92 @@ int llama_server(common_params & params, int argc, char ** argv) {+@@ -457,6 +491,92 @@ int llama_server(common_params & params, int argc, char ** argv) {llama_backend_free();};@@ -178,7 +178,7 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp// start the HTTP server before loading the model to be able to serve /health requestsif (!ctx_http.start()) {clean_up();-@@ -468,6 +588,13 @@ int llama_server(common_params & params, int argc, char ** argv) {+@@ -490,6 +610,13 @@ int llama_server(common_params & params, int argc, char ** argv) {// this will unblock start_loop()ctx_server.terminate();};@@ -192,7 +192,7 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp}// register signal handler if not running by CLI-@@ -509,6 +636,11 @@ int llama_server(common_params & params, int argc, char ** argv) {+@@ -530,6 +657,11 @@ int llama_server(common_params & params, int argc, char ** argv) {// when the HTTP server stops, clean up and exitclean_up();} else {@@ -204,7 +204,7 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp// optionally, notify router server that this instance is readystd::thread monitor_thread;if (child.is_child()) {-@@ -533,5 +665,56 @@ int llama_server(common_params & params, int argc, char ** argv) {+@@ -554,5 +686,56 @@ int llama_server(common_params & params, int argc, char ** argv) {}}diff --git a/llama.cpp.patches/patches/vendor_cpp-httplib_httplib.cpp.patch b/llama.cpp.patches/patches/vendor_cpp-httplib_httplib.cpp.patchindex e00d1b68..82f01dd9 100644--- a/llama.cpp.patches/patches/vendor_cpp-httplib_httplib.cpp.patch+++ b/llama.cpp.patches/patches/vendor_cpp-httplib_httplib.cpp.patch@@ -1,7 +1,7 @@diff --git a/vendor/cpp-httplib/httplib.cpp b/vendor/cpp-httplib/httplib.cpp--- a/llama.cpp/vendor/cpp-httplib/httplib.cpp+++ b/llama.cpp/vendor/cpp-httplib/httplib.cpp-@@ -6847,6 +6847,22 @@ void ThreadPool::cleanup_finished_threads() {+@@ -7244,6 +7244,22 @@ void ThreadPool::cleanup_finished_threads() {}void ThreadPool::worker(bool is_dynamic) {@@ -24,7 +24,7 @@ diff --git a/vendor/cpp-httplib/httplib.cpp b/vendor/cpp-httplib/httplib.cppfor (;;) {std::function<void()> fn;{-@@ -6864,7 +6880,13 @@ void ThreadPool::worker(bool is_dynamic) {+@@ -7261,7 +7277,13 @@ void ThreadPool::worker(bool is_dynamic) {break;}} else {@@ -39,7 +39,7 @@ diff --git a/vendor/cpp-httplib/httplib.cpp b/vendor/cpp-httplib/httplib.cpp}idle_thread_count_--;-@@ -13151,6 +13173,13 @@ bool enumerate_macos_keychain_certs(Callback cb) {+@@ -14003,6 +14025,13 @@ bool enumerate_macos_keychain_certs(Callback cb) {#if !defined(_WIN32) && !(defined(__APPLE__) && \defined(CPPHTTPLIB_USE_CERTS_FROM_MACOSX_KEYCHAIN))@@ -53,7 +53,7 @@ diff --git a/vendor/cpp-httplib/httplib.cpp b/vendor/cpp-httplib/httplib.cpp// Common CA certificate file paths on Linux/Unixconst char **system_ca_paths() {static const char *paths[] = {-@@ -13168,6 +13197,10 @@ const char **system_ca_dirs() {+@@ -14020,6 +14049,10 @@ const char **system_ca_dirs() {static const char *dirs[] = {"/etc/ssl/certs", // Debian/Ubuntu"/etc/pki/tls/certs", // RHEL/CentOS"/usr/share/ca-certificates", // Otherdiff --git a/llama.cpp.patches/patches/vendor_sheredom_subprocess.h.patch b/llama.cpp.patches/patches/vendor_sheredom_subprocess.h.patchindex f8083354..12965efa 100644--- a/llama.cpp.patches/patches/vendor_sheredom_subprocess.h.patch+++ b/llama.cpp.patches/patches/vendor_sheredom_subprocess.h.patch@@ -1,7 +1,7 @@diff --git a/vendor/sheredom/subprocess.h b/vendor/sheredom/subprocess.h--- a/llama.cpp/vendor/sheredom/subprocess.h+++ b/llama.cpp/vendor/sheredom/subprocess.h-@@ -562,6 +562,25 @@ int subprocess_error_from_windows_error(unsigned long error) {+@@ -620,6 +620,25 @@ int subprocess_error_from_windows_error(unsigned long error) {#elsesubprocess_weak int subprocess_error_from_errno(int error);int subprocess_error_from_errno(int error) {@@ -27,7 +27,7 @@ diff --git a/vendor/sheredom/subprocess.h b/vendor/sheredom/subprocess.hswitch (error) {case ENOENT:case ENOTDIR:-@@ -578,6 +597,7 @@ int subprocess_error_from_errno(int error) {+@@ -636,6 +655,7 @@ int subprocess_error_from_errno(int error) {default:return subprocess_error_unknown;}diff --git a/llama.cpp.patches/ui-embed.sh b/llama.cpp.patches/ui-embed.shnew file mode 100755index 00000000..f03d65a5--- /dev/null+++ b/llama.cpp.patches/ui-embed.sh@@ -0,0 +1,183 @@+#!/usr/bin/env bash+#+# Generate ui.cpp / ui.h embedding the web UI assets as C arrays.+#+# Usage:+# ui-embed.sh <out_cpp> <out_h> [<asset_dir>]+#+# This is a translation of emit_files() in llama.cpp/scripts/ui-assets.cmake,+# the same way llamafile-files/BUILD.mk is a translation of llama.cpp's CMake+# build: upstream generates ui.{cpp,h} with configure_file() at CMake time, and+# the cosmocc build has no CMake step.+#+# It renders *upstream's own* templates, tools/ui/ui.{cpp,h}.in, so the+# generated interface (llama_ui_find_asset / llama_ui_get_assets /+# llama_ui_use_gzip and LLAMA_UI_HAS_ASSETS, all consumed by the unpatched+# tools/server/server-http.cpp) tracks upstream automatically and cannot drift.+# Only the template *substitutions* live here:+#+# @ASSET_ARRAYS@ one `static const unsigned char asset_N[] = {...};` per asset+# @ASSET_TABLE@ one `{ name, asset_N, sizeof(asset_N), etag, mime },` per asset+# @N_ASSETS@ asset count+# @USE_GZIP@ whether the embedded bytes are gzip-compressed+# #cmakedefine LLAMA_UI_HAS_ASSETS+#+# Assets come from <asset_dir> (fetch-ui-assets.sh populates it). When it holds+# a _gzip/ mirror the compressed bytes are embedded instead, and server-http.cpp+# serves them with Content-Encoding: gzip. Without <asset_dir> — or with no+# index.html in it — this emits the no-asset stub and the server builds UI-less.++set -euo pipefail++if [ $# -lt 2 ] || [ $# -gt 3 ]; then+ echo "usage: $0 <out_cpp> <out_h> [<asset_dir>]" >&2+ exit 1+fi++OUT_CPP="$1"+OUT_H="$2"+ASSET_DIR="${3:-}"++SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+TEMPLATE_DIR="$SCRIPT_DIR/../llama.cpp/tools/ui"+TPL_CPP="$TEMPLATE_DIR/ui.cpp.in"+TPL_H="$TEMPLATE_DIR/ui.h.in"++for t in "$TPL_CPP" "$TPL_H"; do+ if [ ! -f "$t" ]; then+ echo "ui-embed: missing upstream template $t" >&2+ exit 1+ fi+done++# ETag source. Upstream uses SHA-256; cksum (POSIX, always present) is a+# last-resort fallback — an ETag only has to be stable and change with content.+if command -v shasum >/dev/null 2>&1; then+ hash_file() { shasum -a 256 "$1" | cut -d' ' -f1; }+elif command -v sha256sum >/dev/null 2>&1; then+ hash_file() { sha256sum "$1" | cut -d' ' -f1; }+else+ hash_file() { cksum "$1" | cut -d' ' -f1,2 | tr ' ' '-'; }+fi++# Mirrors mime_from_ext() in scripts/ui-assets.cmake. An extension missing here+# is served as application/octet-stream rather than breaking the build.+mime_from_ext() {+ case "${1##*.}" in+ html) echo "text/html; charset=utf-8" ;;+ css) echo "text/css" ;;+ js) echo "application/javascript" ;;+ json) echo "application/json" ;;+ webmanifest) echo "application/manifest+json" ;;+ svg) echo "image/svg+xml" ;;+ png) echo "image/png" ;;+ jpg|jpeg) echo "image/jpeg" ;;+ ico) echo "image/x-icon" ;;+ woff) echo "font/woff" ;;+ woff2) echo "font/woff2" ;;+ *) echo "application/octet-stream" ;;+ esac+}++work="$(mktemp -d)"+trap 'rm -rf "$work"' EXIT+arrays="$work/arrays"+table="$work/table"+: > "$arrays"+: > "$table"++n_assets=0+use_gzip=false++if [ -n "$ASSET_DIR" ] && [ -f "$ASSET_DIR/index.html" ]; then+ # Asset names are paths relative to the dist root; the gzip mirror repeats+ # that layout under _gzip/, so names come from dist/ and bytes from whichever+ # tree we embed.+ embed_dir="$ASSET_DIR"+ if [ -d "$ASSET_DIR/_gzip" ]; then+ embed_dir="$ASSET_DIR/_gzip"+ use_gzip=true+ fi++ # sort for reproducible output; directory order is unspecified+ assets=()+ while IFS= read -r f; do+ assets+=("${f#./}")+ done < <(cd "$ASSET_DIR" && find . -type f ! -path './_gzip/*' | LC_ALL=C sort)++ idx=0+ for name in "${assets[@]}"; do+ src="$embed_dir/$name"+ if [ ! -s "$src" ]; then+ # A zero-length asset would emit an invalid empty C array. Upstream+ # hard-errors; drop the whole UI instead so a corrupt tarball cannot+ # abort a build, matching fetch-ui-assets.sh's fallback.+ echo "ui-embed: empty asset '$name' - building without an embedded UI" >&2+ : > "$arrays"+ : > "$table"+ idx=0+ break+ fi++ # od is POSIX; 16 bytes per line keeps the generated source wrapped.+ {+ printf 'static const unsigned char asset_%d[] = {\n' "$idx"+ od -An -v -tx1 "$src" \+ | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' \+ -e 's/\([0-9a-f][0-9a-f]\)/0x\1,/g' -e 's/[[:space:]]//g'+ printf '};\n'+ } >> "$arrays"++ printf ' { "%s", asset_%d, sizeof(asset_%d), "\\"%s\\"", "%s" },\n' \+ "$name" "$idx" "$idx" "$(hash_file "$src")" "$(mime_from_ext "$name")" \+ >> "$table"++ idx=$((idx + 1))+ done+ n_assets=$idx+fi++if [ "$n_assets" -eq 0 ]; then+ use_gzip=false+fi++# Render a template: stream the bulk substitutions in from their own files+# (they reach tens of MB, which no sed expression should carry) and expand the+# scalar ones inline.+render() {+ local tpl="$1" out="$2"+ awk -v n_assets="$n_assets" \+ -v use_gzip="$use_gzip" \+ -v has_assets="$([ "$n_assets" -gt 0 ] && echo 1 || echo 0)" \+ -v arrays="$arrays" \+ -v table="$table" '+ /@ASSET_ARRAYS@/ { fflush(); system("cat " arrays); next }+ /@ASSET_TABLE@/ { fflush(); system("cat " table); next }+ {+ if ($0 ~ /^#cmakedefine LLAMA_UI_HAS_ASSETS/) {+ print (has_assets ? "#define LLAMA_UI_HAS_ASSETS 1" \+ : "/* #undef LLAMA_UI_HAS_ASSETS */")+ next+ }+ gsub(/@N_ASSETS@/, n_assets)+ gsub(/@USE_GZIP@/, use_gzip)+ }+ ' "$tpl" > "$out.tmp"+ # Only touch the output when it changes, so a re-fetch of identical assets+ # does not force every dependent object to rebuild.+ if [ -f "$out" ] && cmp -s "$out.tmp" "$out"; then+ rm -f "$out.tmp"+ else+ mv -f "$out.tmp" "$out"+ fi+}++render "$TPL_H" "$OUT_H"+render "$TPL_CPP" "$OUT_CPP"++if [ "$n_assets" -gt 0 ]; then+ echo "ui-embed: embedded $n_assets assets$([ "$use_gzip" = true ] && echo ' (gzip)')"+else+ echo "ui-embed: no assets - building without an embedded UI"+fidiff --git a/llamafile/BUILD.mk b/llamafile/BUILD.mkindex 5a9d584b..3a0a2ef1 100644--- a/llamafile/BUILD.mk+++ b/llamafile/BUILD.mk@@ -257,7 +257,8 @@ LLAMAFILE_SERVER_SUPPORT_OBJS := \$(UI_GEN_OBJ)# Metal source files to embed in the executable (for runtime compilation on macOS)-# These are extracted at runtime and compiled into ggml-metal.dylib+# These are extracted at runtime and compiled into ggml-metal.dylib; the+# kernels/ shaders are compiled by the Metal runtime itself (see llamafile/metal.c).LLAMAFILE_METAL_SOURCES := \o/$(MODE)/llama.cpp/ggml/src/ggml.c.zip.o \o/$(MODE)/llama.cpp/ggml/src/ggml-alloc.c.zip.o \@@ -273,13 +274,13 @@ LLAMAFILE_METAL_SOURCES := \o/$(MODE)/llama.cpp/ggml/include/ggml-cpp.h.zip.o \o/$(MODE)/llama.cpp/ggml/include/ggml-metal.h.zip.o \o/$(MODE)/llama.cpp/ggml/src/ggml-impl.h.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-version.h.zip.o \o/$(MODE)/llama.cpp/ggml/src/ggml-common.h.zip.o \o/$(MODE)/llama.cpp/ggml/src/ggml-quants.h.zip.o \o/$(MODE)/llama.cpp/ggml/src/ggml-threading.h.zip.o \o/$(MODE)/llama.cpp/ggml/src/ggml-backend-impl.h.zip.o \o/$(MODE)/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-impl.h.zip.o \o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal.cpp.zip.o \- o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal.metal.zip.o \o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-impl.h.zip.o \o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-device.h.zip.o \o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-device.m.zip.o \@@ -289,7 +290,34 @@ LLAMAFILE_METAL_SOURCES := \o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-common.h.zip.o \o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-common.cpp.zip.o \o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-ops.h.zip.o \- o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-ops.cpp.zip.o+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-ops.cpp.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.h.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.cpp.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-tuning.h.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-tuning.cpp.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/common.h.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/dequantize.h.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/quantize.h.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/argsort.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/binbcast.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/conv.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/fa.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/gated_delta_net.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/misc.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/mul_mm.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/mul_mv.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/norm.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/pool.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/quantize.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/reduce.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/rope.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/softmax.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/solve_tri.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/ssm.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/tri.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/unary.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/upscale.metal.zip.o \+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/wkv.metal.zip.o# Use deferred expansion (=) since this depends on variables from llama.cpp/BUILD.mkLLAMAFILE_DEPS = \@@ -344,15 +372,6 @@ o/$(MODE)/llamafile/llamafile: \# Pattern rules for llamafile sources# ==============================================================================-# metal.c needs GGML_VERSION and GGML_COMMIT for runtime Metal compilation-# GGML_VERSION and GGML_COMMIT are inherited from build/config.mk-o/$(MODE)/llamafile/metal.o: llamafile/metal.c- @mkdir -p $(@D)- $(CC) $(CFLAGS) $(LLAMAFILE_CPPFLAGS) \- -DGGML_VERSION=\"$(GGML_VERSION)\" \- -DGGML_COMMIT=\"$(GGML_COMMIT)\" \- -c -o $@ $<-o/$(MODE)/llamafile/%.o: llamafile/%.c@mkdir -p $(@D)$(CC) $(CFLAGS) $(LLAMAFILE_CPPFLAGS) -c -o $@ $<diff --git a/llamafile/args.cpp b/llamafile/args.cppindex 73c610f9..b8446ed5 100644--- a/llamafile/args.cpp+++ b/llamafile/args.cpp@@ -19,6 +19,7 @@#include "args.h"#include "llamafile.h"+#include <cstdio>#include <cstring>#include <vector>@@ -41,6 +42,42 @@ static bool is_llamafile_flag(const char* arg) {strcmp(arg, "--version") == 0;}+// llama.cpp b11100 replaced --mmap/--no-mmap/--mlock/-dio/-ndio with+// --load-mode and now rejects them, which would break existing llamafiles+// whose .args still use them. They are dropped here and one equivalent+// "--load-mode <mode>" is inserted where the last of them was, so an+// explicit --load-mode keeps working and the last flag wins, as upstream.+//+// The mode follows the original semantics, where the flags were independent+// settings: mmap on by default, --mlock adds locking to it (mmap+mlock, not+// plain mlock), and direct I/O takes precedence over mmap.+static const char* legacy_load_flag(const char* arg, bool* mmap, bool* mlock, bool* dio) {+ if (strcmp(arg, "--mmap") == 0) {+ *mmap = true;+ } else if (strcmp(arg, "--no-mmap") == 0) {+ *mmap = false;+ } else if (strcmp(arg, "--mlock") == 0) {+ *mlock = true;+ } else if (strcmp(arg, "-dio") == 0 || strcmp(arg, "--direct-io") == 0) {+ *dio = true;+ } else if (strcmp(arg, "-ndio") == 0 || strcmp(arg, "--no-direct-io") == 0) {+ *dio = false;+ } else {+ return nullptr;+ }+ return arg;+}++static char* legacy_load_mode(bool mmap, bool mlock, bool dio) {+ static char kNone[] = "none", kMmap[] = "mmap", kMlock[] = "mlock",+ kMmapMlock[] = "mmap+mlock", kDio[] = "dio";+ if (dio)+ return kDio;+ if (mmap)+ return mlock ? kMmapMlock : kMmap;+ return mlock ? kMlock : kNone;+}+LlamafileArgs parse_llamafile_args(int argc, char** argv) {LlamafileArgs args;@@ -99,9 +136,20 @@ LlamafileArgs parse_llamafile_args(int argc, char** argv) {// These are not recognized by llama.cpp and would cause errorsg_filtered_argv.clear();+ bool legacy_mmap = true, legacy_mlock = false, legacy_dio = false;+ const char* last_legacy = nullptr;+ size_t legacy_pos = 0;+for (int i = 0; i < argc; ++i) {const char* arg = argv[i];+ // Translate removed load flags (see legacy_load_flag)+ if (const char* flag = legacy_load_flag(arg, &legacy_mmap, &legacy_mlock, &legacy_dio)) {+ last_legacy = flag;+ legacy_pos = g_filtered_argv.size();+ continue;+ }+// Skip llamafile-specific flagsif (is_llamafile_flag(arg)) {// --gpu takes a value argument, skip it too@@ -115,6 +163,14 @@ LlamafileArgs parse_llamafile_args(int argc, char** argv) {g_filtered_argv.push_back(argv[i]);}+ if (last_legacy) {+ static char kLoadMode[] = "--load-mode";+ char* mode = legacy_load_mode(legacy_mmap, legacy_mlock, legacy_dio);+ fprintf(stderr, "warning: %s is no longer supported by llama.cpp; using --load-mode %s "+ "(see docs/cli_arguments.md)\n", last_legacy, mode);+ g_filtered_argv.insert(g_filtered_argv.begin() + legacy_pos, {kLoadMode, mode});+ }+// Null-terminate argv array (required by convention)g_filtered_argv.push_back(nullptr);diff --git a/llamafile/build-functions.sh b/llamafile/build-functions.shindex 9676e5c5..5dbbccb9 100644--- a/llamafile/build-functions.sh+++ b/llamafile/build-functions.sh@@ -104,8 +104,17 @@ setup_build_dir() {# $2 = caller-supplied sources prepended to the list (e.g., tinyblas.cu# for the default TinyBLAS build; empty for the --cublas build)# $3 = NO_IQ_QUANTS (optional, "1" to exclude IQ quant MMQ templates)-# $4 = FA_ALL_QUANTS (optional, "1" to include all fattn-vec quant combos-# instead of the 3 default ones; mirrors upstream's GGML_CUDA_FA_ALL_QUANTS)+# $4 = FA_ALL_QUANTS (optional, "1" to compile every fattn-vec quant combo+# instead of the default four; mirrors upstream's+# GGML_CUDA_FA_QUANTS=all)+# Sets: CUDA_SOURCES, NUM_SOURCES, CUDA_FA_DEFINES (append the last to the+# compiler flags -- fattn.cu does not compile without it)+# The K-V types fattn.cu knows about, and the combinations compiled by default.+# Both mirror ggml/CMakeLists.txt's GGML_CUDA_FA_QUANTS default and the FA_TYPES+# list in ggml/cmake/common.cmake -- keep in sync on a llama.cpp bump.+FA_TYPES="q4_0 q4_1 q5_0 q5_1 q8_0 bf16 f16"+FA_DEFAULT_COMBINATIONS="q4_0-q4_0 q8_0-q8_0 f16-f16 bf16-bf16"+collect_gpu_sources() {local ggml_cuda_dir="$1"local caller_sources="$2"@@ -126,23 +135,55 @@ collect_gpu_sources() {[ -f "$f" ] && CUDA_SOURCES="$CUDA_SOURCES $f"done- # 3. fattn-vec: default to the 4 common quant combos (f16-f16, q4_0-q4_0,- # q8_0-q8_0, bf16-bf16), matching upstream CMake. With FA_ALL_QUANTS=1- # include all fattn-vec instances (mirrors upstream's- # GGML_CUDA_FA_ALL_QUANTS opt-in).+ # 3. fattn-vec: since b11100 fattn.cu gates every K-V combination on a+ # GGML_CUDA_FA_<K>_<V> macro that must be defined to 0 or 1 for *all* of+ # them (upstream's ggml_cuda_fattn_vec_instances() in+ # ggml/cmake/common.cmake emits them). Picking source files is no longer+ # enough: an undefined macro is a hard `identifier is undefined` error in+ # the `if constexpr`. So derive both the sources and the macros from one+ # list of combinations.+ local fa_combinationsif [ "$fa_all_quants" = "1" ]; then- for f in "$ti_dir"/fattn-vec-instance-*.cu; do- [ -f "$f" ] && CUDA_SOURCES="$CUDA_SOURCES $f"+ fa_combinations=""+ for type_v in $FA_TYPES; do+ for type_k in $FA_TYPES; do+ fa_combinations="$fa_combinations $type_k-$type_v"+ donedoneelse- for f in "$ti_dir"/fattn-vec-instance-f16-f16.cu \- "$ti_dir"/fattn-vec-instance-q4_0-q4_0.cu \- "$ti_dir"/fattn-vec-instance-q8_0-q8_0.cu \- "$ti_dir"/fattn-vec-instance-bf16-bf16.cu; do- [ -f "$f" ] && CUDA_SOURCES="$CUDA_SOURCES $f"- done+ fa_combinations="$FA_DEFAULT_COMBINATIONS"fi+ for combination in $fa_combinations; do+ f="$ti_dir/fattn-vec-instance-$combination.cu"+ if [ ! -f "$f" ]; then+ echo "Error: FlashAttention template instance not found: $f" >&2+ return 1+ fi+ CUDA_SOURCES="$CUDA_SOURCES $f"+ done++ # One -D per combination, 0 or 1, for every type pair fattn.cu can ask about.+ # No spaces in any value, so these survive the unquoted word-splitting that+ # compile_gpu_sources_parallel does on its common_flags argument.+ CUDA_FA_DEFINES=""+ for type_v in $FA_TYPES; do+ for type_k in $FA_TYPES; do+ local compiled=0+ case " $fa_combinations " in+ *" $type_k-$type_v "*) compiled=1 ;;+ esac+ local macro+ macro=$(echo "GGML_CUDA_FA_${type_k}_${type_v}" | tr '[:lower:]' '[:upper:]')+ CUDA_FA_DEFINES="$CUDA_FA_DEFINES -D$macro=$compiled"+ done+ done+ # Reported by ggml_backend_cuda_get_features as "FA_QUANTS" (guarded by+ # #ifdef, so a build without it just omits the feature).+ local fa_quants_csv+ fa_quants_csv=$(echo $fa_combinations | tr ' ' ',')+ CUDA_FA_DEFINES="$CUDA_FA_DEFINES -DGGML_CUDA_FA_QUANTS=\"$fa_quants_csv\""+# 4. mmf instances (always included)for f in "$ti_dir"/mmf-*.cu; do[ -f "$f" ] && CUDA_SOURCES="$CUDA_SOURCES $f"diff --git a/llamafile/chatbot_cli.cpp b/llamafile/chatbot_cli.cppindex bb556229..44ec6bfc 100644--- a/llamafile/chatbot_cli.cpp+++ b/llamafile/chatbot_cli.cpp@@ -241,9 +241,11 @@ int cli_main(int argc, char **argv) {// Load image bitmapsfor (const auto &image_path : params.image) {- // init_from_file now returns a wrapper {bitmap, video_ctx}; chatbot- // is image-only, so take the bitmap (placeholder=false loads pixels).- mtmd::bitmap bmp(mtmd_helper_bitmap_init_from_file(mtmd_ctx, image_path.c_str(), false).bitmap);+ // init_from_file returns a wrapper {bitmap, video_ctx}; chatbot is+ // image-only, so take the bitmap (placeholder=false loads pixels)+ // and the default opt (its video params go unused).+ mtmd::bitmap bmp(mtmd_helper_bitmap_init_from_file(+ mtmd_ctx, image_path.c_str(), false, mtmd_helper_init_opt_default()).bitmap);if (!bmp.ptr) {fprintf(stderr, "error: failed to load image: %s\n", image_path.c_str());cleanup(mtmd_ctx, sampler, ctx, model);diff --git a/llamafile/chatbot_eval.cpp b/llamafile/chatbot_eval.cppindex 30af677d..a129af3d 100644--- a/llamafile/chatbot_eval.cpp+++ b/llamafile/chatbot_eval.cpp@@ -167,10 +167,12 @@ bool eval_string(std::string_view s, bool add_special, bool parse_special) {// Create bitmaps from decoded image datamtmd::bitmaps bitmaps;for (const auto &image : extraction.images) {- // init_from_buf now returns a wrapper {bitmap, video_ctx}; chatbot is- // image-only, so take the bitmap (placeholder=false loads real pixels).+ // init_from_buf returns a wrapper {bitmap, video_ctx}; chatbot is+ // image-only, so take the bitmap (placeholder=false loads real pixels)+ // and the default opt (its video params go unused).mtmd::bitmap bmp(mtmd_helper_bitmap_init_from_buf(- g_mtmd, (const unsigned char *)image.data(), image.size(), false).bitmap);+ g_mtmd, (const unsigned char *)image.data(), image.size(), false,+ mtmd_helper_init_opt_default()).bitmap);if (!bmp.ptr) {err("failed to load image");return false;diff --git a/llamafile/cuda.bat b/llamafile/cuda.batindex c7364109..195be5fe 100644--- a/llamafile/cuda.bat+++ b/llamafile/cuda.bat@@ -191,7 +191,27 @@ set "COMMON_FLAGS=%COMMON_FLAGS% -DNDEBUG -DGGML_BUILD=1 -DGGML_SHARED=1 -DGGML_set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_CUDA_USE_GRAPHS"set "COMMON_FLAGS=%COMMON_FLAGS% %BLAS_DEFINE%"if "%NO_IQ_QUANTS%"=="1" set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_CUDA_NO_IQ_QUANTS"-if "%FA_ALL_QUANTS%"=="1" set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_CUDA_FA_ALL_QUANTS"++:: fattn.cu gates every K-V combination on a GGML_CUDA_FA_<K>_<V> macro that+:: must be defined to 0 or 1 for all of them (since b11100; upstream emits them+:: from ggml_cuda_fattn_vec_instances() in ggml/cmake/common.cmake). Selecting+:: template-instance files is no longer enough -- an undefined macro is a hard+:: error in the if constexpr. Keep this list and the default combinations in+:: sync with FA_TYPES / FA_DEFAULT_COMBINATIONS in llamafile/build-functions.sh.+:: (The informational GGML_CUDA_FA_QUANTS string macro is #ifdef-guarded and is+:: not set here, to avoid quoting it through cmd.)+set "FA_TYPES=Q4_0 Q4_1 Q5_0 Q5_1 Q8_0 BF16 F16"+if "%FA_ALL_QUANTS%"=="1" (+ set "FA_COMBINATIONS="+ for %%v in (%FA_TYPES%) do for %%k in (%FA_TYPES%) do call set "FA_COMBINATIONS=%%FA_COMBINATIONS%% %%k-%%v"+) else (+ set "FA_COMBINATIONS=Q4_0-Q4_0 Q8_0-Q8_0 F16-F16 BF16-BF16"+)+for %%v in (%FA_TYPES%) do for %%k in (%FA_TYPES%) do (+ set "FA_COMPILED=0"+ for %%c in (!FA_COMBINATIONS!) do if /i "%%c"=="%%k-%%v" set "FA_COMPILED=1"+ set "COMMON_FLAGS=!COMMON_FLAGS! -DGGML_CUDA_FA_%%k_%%v=!FA_COMPILED!"+):: -------- extract GGML version --------set "GGML_VERSION=unknown"diff --git a/llamafile/cuda.sh b/llamafile/cuda.shindex ff35f1c5..80485096 100755--- a/llamafile/cuda.sh+++ b/llamafile/cuda.sh@@ -251,12 +251,14 @@ COMMON_FLAGS="\if [ "$NO_IQ_QUANTS" = "1" ]; thenCOMMON_FLAGS="$COMMON_FLAGS -DGGML_CUDA_NO_IQ_QUANTS"fi-if [ "$FA_ALL_QUANTS" = "1" ]; then- COMMON_FLAGS="$COMMON_FLAGS -DGGML_CUDA_FA_ALL_QUANTS"-fi# Collect sourcescollect_gpu_sources "$GGML_CUDA_DIR" "$EXTRA_SOURCES" "$NO_IQ_QUANTS" "$FA_ALL_QUANTS"++# fattn.cu gates each K-V combination on a GGML_CUDA_FA_<K>_<V> macro that must+# be defined for all of them; collect_gpu_sources derives them from the same+# combination list it picked the template instances from.+COMMON_FLAGS="$COMMON_FLAGS $CUDA_FA_DEFINES"echo " Sources: $NUM_SOURCES .cu files"echo ""diff --git a/llamafile/cuda_parallel.bat b/llamafile/cuda_parallel.batindex dbf17440..509e8f28 100644--- a/llamafile/cuda_parallel.bat+++ b/llamafile/cuda_parallel.bat@@ -214,7 +214,27 @@ set "COMMON_FLAGS=%COMMON_FLAGS% -DNDEBUG -DGGML_BUILD=1 -DGGML_SHARED=1 -DGGML_set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_CUDA_USE_GRAPHS"set "COMMON_FLAGS=%COMMON_FLAGS% %BLAS_DEFINE%"if "%NO_IQ_QUANTS%"=="1" set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_CUDA_NO_IQ_QUANTS"-if "%FA_ALL_QUANTS%"=="1" set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_CUDA_FA_ALL_QUANTS"++:: fattn.cu gates every K-V combination on a GGML_CUDA_FA_<K>_<V> macro that+:: must be defined to 0 or 1 for all of them (since b11100; upstream emits them+:: from ggml_cuda_fattn_vec_instances() in ggml/cmake/common.cmake). Selecting+:: template-instance files is no longer enough -- an undefined macro is a hard+:: error in the if constexpr. Keep this list and the default combinations in+:: sync with FA_TYPES / FA_DEFAULT_COMBINATIONS in llamafile/build-functions.sh.+:: (The informational GGML_CUDA_FA_QUANTS string macro is #ifdef-guarded and is+:: not set here, to avoid quoting it through cmd.)+set "FA_TYPES=Q4_0 Q4_1 Q5_0 Q5_1 Q8_0 BF16 F16"+if "%FA_ALL_QUANTS%"=="1" (+ set "FA_COMBINATIONS="+ for %%v in (%FA_TYPES%) do for %%k in (%FA_TYPES%) do call set "FA_COMBINATIONS=%%FA_COMBINATIONS%% %%k-%%v"+) else (+ set "FA_COMBINATIONS=Q4_0-Q4_0 Q8_0-Q8_0 F16-F16 BF16-BF16"+)+for %%v in (%FA_TYPES%) do for %%k in (%FA_TYPES%) do (+ set "FA_COMPILED=0"+ for %%c in (!FA_COMBINATIONS!) do if /i "%%c"=="%%k-%%v" set "FA_COMPILED=1"+ set "COMMON_FLAGS=!COMMON_FLAGS! -DGGML_CUDA_FA_%%k_%%v=!FA_COMPILED!"+):: -------- extract GGML version --------set "GGML_VERSION=unknown"diff --git a/llamafile/metal.c b/llamafile/metal.cindex 933ddaf0..e09800aa 100644--- a/llamafile/metal.c+++ b/llamafile/metal.c@@ -22,9 +22,14 @@// This file implements dynamic compilation and loading of Metal GPU support.// At runtime on macOS ARM64:// 1. Extract Metal source files from /zip/ to ~/.llamafile/-// 2. Preprocess ggml-metal.metal to inline header files-// 3. Compile everything into a self-contained dylib using system cc-// 4. Load the dylib with cosmo_dlopen() and register the Metal backend+// 2. Compile everything into a self-contained dylib using system cc+// 3. Load the dylib with cosmo_dlopen() and register the Metal backend+//+// The shader sources (kernels/*.metal) are compiled by the Metal runtime+// itself, not by us: the backend is built without GGML_METAL_EMBED_LIBRARY,+// so ggml_metal_library_init() reads kernels/<kind>.metal, flattens their+// #includes and hands the result to newLibraryWithSource. It looks them up+// under $GGML_METAL_PATH_RESOURCES, which we point at the app dir.//// The dylib is self-contained (includes ggml core) because cosmo_dlopen()// cannot resolve symbols from the parent process.@@ -69,6 +74,7 @@ __static_yoink("llama.cpp/ggml/include/ggml-backend.h");__static_yoink("llama.cpp/ggml/include/ggml-cpp.h");__static_yoink("llama.cpp/ggml/include/ggml-metal.h");__static_yoink("llama.cpp/ggml/src/ggml-impl.h");+__static_yoink("llama.cpp/ggml/src/ggml-version.h");__static_yoink("llama.cpp/ggml/src/ggml-common.h");__static_yoink("llama.cpp/ggml/src/ggml-quants.h");__static_yoink("llama.cpp/ggml/src/ggml-threading.h");@@ -77,7 +83,6 @@ __static_yoink("llama.cpp/ggml/src/ggml-cpu/ggml-cpu-impl.h");// Metal backend__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal.cpp");-__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal.metal");__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-impl.h");__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-device.h");__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-device.m");@@ -88,6 +93,37 @@ __static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-common.h");__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-common.cpp");__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-ops.h");__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-ops.cpp");+__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.h");+__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.cpp");+__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-tuning.h");+__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-tuning.cpp");++// Metal shader sources, compiled by the Metal runtime at first use. Since+// b11100 upstream ships one .metal per op family under kernels/ instead of a+// single ggml-metal.metal.+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/common.h");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/dequantize.h");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/quantize.h");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/argsort.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/binbcast.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/conv.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/fa.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/gated_delta_net.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/misc.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/mul_mm.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/mul_mv.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/norm.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/pool.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/quantize.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/reduce.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/rope.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/softmax.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/solve_tri.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/ssm.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/tri.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/unary.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/upscale.metal");+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/wkv.metal");// Sources to extract at runtimestatic const struct MetalSource {@@ -103,6 +139,8 @@ static const struct MetalSource {{"/zip/llama.cpp/ggml/include/ggml-cpp.h", "ggml-cpp.h"},{"/zip/llama.cpp/ggml/include/ggml-metal.h", "ggml-metal.h"},{"/zip/llama.cpp/ggml/src/ggml-impl.h", "ggml-impl.h"},+ // generated by llama.cpp.patches/apply-patches.sh; ggml.c includes it+ {"/zip/llama.cpp/ggml/src/ggml-version.h", "ggml-version.h"},{"/zip/llama.cpp/ggml/src/ggml-common.h", "ggml-common.h"},{"/zip/llama.cpp/ggml/src/ggml-quants.h", "ggml-quants.h"},{"/zip/llama.cpp/ggml/src/ggml-threading.h", "ggml-threading.h"},@@ -119,7 +157,6 @@ static const struct MetalSource {// Metal-specific files{"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal.cpp", "ggml-metal.cpp"},- {"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal.metal", "ggml-metal.metal"},{"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-impl.h", "ggml-metal-impl.h"},{"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-device.h", "ggml-metal-device.h"},{"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-device.m", "ggml-metal-device.m"},@@ -130,6 +167,37 @@ static const struct MetalSource {{"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-common.cpp", "ggml-metal-common.cpp"},{"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-ops.h", "ggml-metal-ops.h"},{"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-ops.cpp", "ggml-metal-ops.cpp"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.h", "ggml-metal-fusion.h"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.cpp", "ggml-metal-fusion.cpp"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-tuning.h", "ggml-metal-tuning.h"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-tuning.cpp", "ggml-metal-tuning.cpp"},++ // Shader sources. ggml_metal_library_flatten_source() resolves the quoted+ // includes in these against kernels/ and its parent, so ggml-common.h and+ // ggml-metal-impl.h above are reachable from them once extracted.+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/common.h", "kernels/common.h"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/dequantize.h", "kernels/dequantize.h"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/quantize.h", "kernels/quantize.h"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/argsort.metal", "kernels/argsort.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/binbcast.metal", "kernels/binbcast.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/conv.metal", "kernels/conv.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/fa.metal", "kernels/fa.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/gated_delta_net.metal", "kernels/gated_delta_net.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/misc.metal", "kernels/misc.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/mul_mm.metal", "kernels/mul_mm.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/mul_mv.metal", "kernels/mul_mv.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/norm.metal", "kernels/norm.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/pool.metal", "kernels/pool.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/quantize.metal", "kernels/quantize.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/reduce.metal", "kernels/reduce.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/rope.metal", "kernels/rope.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/softmax.metal", "kernels/softmax.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/solve_tri.metal", "kernels/solve_tri.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/ssm.metal", "kernels/ssm.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/tri.metal", "kernels/tri.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/unary.metal", "kernels/unary.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/upscale.metal", "kernels/upscale.metal"},+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/wkv.metal", "kernels/wkv.metal"},};// Forward declarations for ggml backend types@@ -164,129 +232,6 @@ static struct MetalBackend {void (*log_set)(llamafile_log_callback log_callback, void *user_data);} g_metal;-static char *read_file(const char *path, size_t *size_out) {- FILE *f = fopen(path, "r");- if (!f) return NULL;-- fseek(f, 0, SEEK_END);- long size = ftell(f);- fseek(f, 0, SEEK_SET);-- char *content = malloc(size + 1);- if (!content) {- fclose(f);- return NULL;- }-- fread(content, 1, size, f);- content[size] = '\0';- fclose(f);-- if (size_out) *size_out = size;- return content;-}--static bool write_file(const char *path, const char *content, size_t size) {- FILE *f = fopen(path, "w");- if (!f) return false;- fwrite(content, 1, size, f);- fclose(f);- return true;-}--// Preprocess ggml-metal.metal to inline headers-// Metal runtime compiler doesn't support include paths-static bool PreprocessMetalShader(const char *app_dir) {- char metal_path[PATH_MAX];- char common_path[PATH_MAX];- char impl_path[PATH_MAX];-- snprintf(metal_path, PATH_MAX, "%sggml-metal.metal", app_dir);- snprintf(common_path, PATH_MAX, "%sggml-common.h", app_dir);- snprintf(impl_path, PATH_MAX, "%sggml-metal-impl.h", app_dir);-- size_t common_size, impl_size, metal_size;- char *common_content = read_file(common_path, &common_size);- if (!common_content) {- fprintf(stderr, "metal: failed to read %s\n", common_path);- return false;- }-- char *impl_content = read_file(impl_path, &impl_size);- if (!impl_content) {- free(common_content);- fprintf(stderr, "metal: failed to read %s\n", impl_path);- return false;- }-- char *metal_content = read_file(metal_path, &metal_size);- if (!metal_content) {- free(common_content);- free(impl_content);- fprintf(stderr, "metal: failed to read %s\n", metal_path);- return false;- }-- // Find #include directives- char *include_common = strstr(metal_content, "#include \"ggml-common.h\"");- char *include_impl = strstr(metal_content, "#include \"ggml-metal-impl.h\"");-- if (!include_common && !include_impl) {- // Nothing to preprocess- free(common_content);- free(impl_content);- free(metal_content);- return true;- }-- // Create preprocessed content- FILE *fout = fopen(metal_path, "w");- if (!fout) {- free(common_content);- free(impl_content);- free(metal_content);- fprintf(stderr, "metal: failed to write to %s\n", metal_path);- return false;- }-- char *pos = metal_content;-- // Process includes in order they appear- if (include_common && (!include_impl || include_common < include_impl)) {- // Write everything before #include "ggml-common.h"- fwrite(pos, 1, include_common - pos, fout);- fputs("// ggml-common.h inlined below\n", fout);- fwrite(common_content, 1, common_size, fout);- fputs("\n// end of ggml-common.h\n", fout);- pos = strchr(include_common, '\n');- if (pos) pos++;- }-- if (include_impl && pos && include_impl >= pos) {- // Write everything between includes- fwrite(pos, 1, include_impl - pos, fout);- fputs("// ggml-metal-impl.h inlined below\n", fout);- fwrite(impl_content, 1, impl_size, fout);- fputs("\n// end of ggml-metal-impl.h\n", fout);- pos = strchr(include_impl, '\n');- if (pos) pos++;- }-- // Write rest of file- if (pos) {- fputs(pos, fout);- }-- fclose(fout);- free(common_content);- free(impl_content);- free(metal_content);-- llamafile_info("metal", "preprocessed %s", metal_path);-- return true;-}-static bool BuildMetal(const char *dso) {char app_dir[PATH_MAX];char src[PATH_MAX];@@ -338,13 +283,6 @@ static bool BuildMetal(const char *dso) {}}- // Preprocess Metal shader if rebuild needed- if (needs_rebuild) {- if (!PreprocessMetalShader(app_dir)) {- return false;- }- }-// Check if dylib needs rebuildsnprintf(src, PATH_MAX, "%sggml-metal.cpp", app_dir);if (!needs_rebuild) {@@ -391,6 +329,8 @@ static bool BuildMetal(const char *dso) {"ggml-metal-device.cpp","ggml-metal-common.cpp","ggml-metal-ops.cpp",+ "ggml-metal-fusion.cpp",+ "ggml-metal-tuning.cpp","ggml-metal-device.m","ggml-metal-context.m",NULL@@ -430,8 +370,6 @@ static bool BuildMetal(const char *dso) {args[argc++] = "-ffixed-x28"; // cosmo's TLS registerargs[argc++] = "-DTARGET_OS_OSX";args[argc++] = "-DGGML_MULTIPLATFORM";- args[argc++] = "-DGGML_VERSION=\"" GGML_VERSION "\"";- args[argc++] = "-DGGML_COMMIT=\"" GGML_COMMIT "\"";args[argc++] = "-w"; // Suppress compilation warningsargs[argc++] = "-o";args[argc++] = obj_paths[i];@@ -609,6 +547,22 @@ static bool ImportMetalImpl(void) {llamafile_get_app_dir(app_dir, PATH_MAX);snprintf(dso, PATH_MAX, "%sggml-metal.dylib", app_dir);+ // The dylib is built without GGML_METAL_EMBED_LIBRARY, so+ // ggml_metal_library_init() compiles kernels/<kind>.metal at first use.+ // It finds them through [NSBundle bundleForClass:], which for a loose+ // dylib resolves to the dylib's own directory, i.e. app_dir, where+ // BuildMetal() extracts them. GGML_METAL_PATH_RESOURCES takes precedence+ // over that lookup, and it can't be overridden from here: setenv() only+ // changes Cosmopolitan's environ, which libSystem inside the dylib never+ // reads. So if it is set, kernels come from elsewhere; say so.+ const char *res_path = getenv("GGML_METAL_PATH_RESOURCES");+ if (res_path && *res_path) {+ fprintf(stderr,+ "metal: warning: GGML_METAL_PATH_RESOURCES=%s overrides the kernels "+ "extracted to %s; unset it unless those kernels match this llamafile\n",+ res_path, app_dir);+ }+if (FLAG_nocompile) {return LinkMetal(dso);}diff --git a/llamafile/release.sh b/llamafile/release.shindex 18c651e3..b292dd34 100644--- a/llamafile/release.sh+++ b/llamafile/release.sh@@ -1,5 +1,5 @@# update LLAMAFILE_VERSION for a new release-LLAMAFILE_VERSION="0.10.6"+LLAMAFILE_VERSION="0.10.7"# this is where the precompiled GPU libraries are storedGPU_LIBS_DIR="/home/mala/gpulibs/${LLAMAFILE_VERSION}"diff --git a/llamafile/rocm.bat b/llamafile/rocm.batindex 6101c063..72b1eb24 100644--- a/llamafile/rocm.bat+++ b/llamafile/rocm.bat@@ -134,6 +134,22 @@ set "COMMON_FLAGS=%COMMON_FLAGS% -I"%HIP_PATH%\include""set "COMMON_FLAGS=%COMMON_FLAGS% -DNDEBUG -DGGML_BUILD=1 -DGGML_SHARED=1 -DGGML_BACKEND_SHARED=1 -DGGML_BACKEND_BUILD=1 -DGGML_MULTIPLATFORM"set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_HIP_GRAPHS=1"set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_USE_HIP=1 -DGGML_USE_TINYBLAS=1 -DGGML_HIP_NO_VMM=1 -D__HIP_PLATFORM_AMD__"++:: fattn.cu gates every K-V combination on a GGML_CUDA_FA_<K>_<V> macro that+:: must be defined to 0 or 1 for all of them (since b11100; upstream emits them+:: from ggml_cuda_fattn_vec_instances() in ggml/cmake/common.cmake). Selecting+:: template-instance files is no longer enough -- an undefined macro is a hard+:: error in the if constexpr. Keep this list and the default combinations in+:: sync with FA_TYPES / FA_DEFAULT_COMBINATIONS in llamafile/build-functions.sh.+:: (The informational GGML_CUDA_FA_QUANTS string macro is #ifdef-guarded and is+:: not set here, to avoid quoting it through cmd.)+set "FA_TYPES=Q4_0 Q4_1 Q5_0 Q5_1 Q8_0 BF16 F16"+set "FA_COMBINATIONS=Q4_0-Q4_0 Q8_0-Q8_0 F16-F16 BF16-BF16"+for %%v in (%FA_TYPES%) do for %%k in (%FA_TYPES%) do (+ set "FA_COMPILED=0"+ for %%c in (!FA_COMBINATIONS!) do if /i "%%c"=="%%k-%%v" set "FA_COMPILED=1"+ set "COMMON_FLAGS=!COMMON_FLAGS! -DGGML_CUDA_FA_%%k_%%v=!FA_COMPILED!"+):: --offload-compress shrinks the fat binary (mirrors rocm.sh / #995); requires a:: recent ROCm HIP SDK -- drop this flag if your clang++ rejects itset "COMMON_FLAGS=%COMMON_FLAGS% --offload-compress"diff --git a/llamafile/rocm.sh b/llamafile/rocm.shindex 77740375..2efa48cc 100755--- a/llamafile/rocm.sh+++ b/llamafile/rocm.sh@@ -130,6 +130,11 @@ COMMON_FLAGS="\# Collect sources (TinyBLAS + GGML CUDA)collect_gpu_sources "$GGML_CUDA_DIR" "$BUILD_DIR/tinyblas.cu"++# fattn.cu gates each K-V combination on a GGML_CUDA_FA_<K>_<V> macro that must+# be defined for all of them; collect_gpu_sources derives them from the same+# combination list it picked the template instances from.+COMMON_FLAGS="$COMMON_FLAGS $CUDA_FA_DEFINES"echo " Sources: $NUM_SOURCES .cu files"echo ""diff --git a/llamafile/rocm_parallel.bat b/llamafile/rocm_parallel.batindex 3c9c855b..23bd6577 100644--- a/llamafile/rocm_parallel.bat+++ b/llamafile/rocm_parallel.bat@@ -156,6 +156,22 @@ set "COMMON_FLAGS=%COMMON_FLAGS% -I"%HIP_PATH%\include""set "COMMON_FLAGS=%COMMON_FLAGS% -DNDEBUG -DGGML_BUILD=1 -DGGML_SHARED=1 -DGGML_BACKEND_SHARED=1 -DGGML_BACKEND_BUILD=1 -DGGML_MULTIPLATFORM"set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_HIP_GRAPHS=1"set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_USE_HIP=1 -DGGML_USE_TINYBLAS=1 -DGGML_HIP_NO_VMM=1 -D__HIP_PLATFORM_AMD__"++:: fattn.cu gates every K-V combination on a GGML_CUDA_FA_<K>_<V> macro that+:: must be defined to 0 or 1 for all of them (since b11100; upstream emits them+:: from ggml_cuda_fattn_vec_instances() in ggml/cmake/common.cmake). Selecting+:: template-instance files is no longer enough -- an undefined macro is a hard+:: error in the if constexpr. Keep this list and the default combinations in+:: sync with FA_TYPES / FA_DEFAULT_COMBINATIONS in llamafile/build-functions.sh.+:: (The informational GGML_CUDA_FA_QUANTS string macro is #ifdef-guarded and is+:: not set here, to avoid quoting it through cmd.)+set "FA_TYPES=Q4_0 Q4_1 Q5_0 Q5_1 Q8_0 BF16 F16"+set "FA_COMBINATIONS=Q4_0-Q4_0 Q8_0-Q8_0 F16-F16 BF16-BF16"+for %%v in (%FA_TYPES%) do for %%k in (%FA_TYPES%) do (+ set "FA_COMPILED=0"+ for %%c in (!FA_COMBINATIONS!) do if /i "%%c"=="%%k-%%v" set "FA_COMPILED=1"+ set "COMMON_FLAGS=!COMMON_FLAGS! -DGGML_CUDA_FA_%%k_%%v=!FA_COMPILED!"+):: --offload-compress shrinks the fat binary (mirrors rocm.sh / #995); requires a:: recent ROCm HIP SDK -- drop this flag if your clang++ rejects itset "COMMON_FLAGS=%COMMON_FLAGS% --offload-compress"diff --git a/llamafile/version.h b/llamafile/version.hindex ca3c9a6d..3d4db985 100644--- a/llamafile/version.h+++ b/llamafile/version.h@@ -20,7 +20,7 @@#define LLAMAFILE_MAJOR 0#define LLAMAFILE_MINOR 10-#define LLAMAFILE_PATCH 6+#define LLAMAFILE_PATCH 7#define LLAMAFILE_VERSION \(100000000 * LLAMAFILE_MAJOR + 1000000 * LLAMAFILE_MINOR + LLAMAFILE_PATCH)diff --git a/llamafile/vulkan.bat b/llamafile/vulkan.batindex 4f2a1f5d..1304d5b5 100644--- a/llamafile/vulkan.bat+++ b/llamafile/vulkan.bat@@ -189,7 +189,7 @@ set "OLD_DEFINES="if exist "%FEATURES_STAMP%" set /p OLD_DEFINES=<"%FEATURES_STAMP%"if not "!OLD_DEFINES!"=="!GLSLC_DEFINES!" (echo glslc feature set changed; clearing cached shader artifacts- del /q "%BUILD_DIR%\vulkan-shaders-gen.exe" "%BUILD_DIR%\ggml-vulkan-shaders.hpp" "%BUILD_DIR%\ggml-vulkan.obj" "%BUILD_DIR%\shader-*.obj" 2>nul+ del /q "%BUILD_DIR%\vulkan-shaders-gen.exe" "%BUILD_DIR%\ggml-vulkan-shaders.hpp" "%BUILD_DIR%\ggml-vulkan*.obj" "%BUILD_DIR%\shader-*.obj" 2>nulif exist "%SHADERS_BUILD_DIR%" rmdir /s /q "%SHADERS_BUILD_DIR%"if exist "%SPVDIR%" rmdir /s /q "%SPVDIR%"mkdir "%SHADERS_BUILD_DIR%" 2>nul@@ -308,19 +308,23 @@ if !CPP_CMD_COUNT! gtr 0 (echo.:: ========================================================================-:: Phase 5: Compile ggml-vulkan.cpp+:: Phase 5: Compile the backend sources:: ========================================================================-echo Phase 5: Compiling ggml-vulkan.cpp...--set "VULKAN_OBJ=%BUILD_DIR%\ggml-vulkan.obj"-set "VULKAN_SRC=%GGML_VULKAN_DIR%\ggml-vulkan.cpp"--if not exist "%VULKAN_OBJ%" (- echo Compiling ggml-vulkan.cpp...- cl %CXX_FLAGS% /I"%GGML_VULKAN_DIR%" /I"%VULKAN_SDK%\Include" /Fo"%VULKAN_OBJ%" "%VULKAN_SRC%"- if errorlevel 1 (echo Error compiling ggml-vulkan.cpp & exit /b 1)-) else (- echo ggml-vulkan.obj is up to date+:: Since b11100 the backend is split over several translation units+:: (ggml-vulkan.cpp plus ggml-vulkan-buffers.cpp, ggml-vulkan-debug.cpp, ...),+:: so compile every .cpp at the top of ggml-vulkan\ rather than naming one.+:: vulkan-shaders\ is a subdirectory, so the wildcard leaves it alone.+echo Phase 5: Compiling backend sources...++for %%f in ("%GGML_VULKAN_DIR%\*.cpp") do (+ set "VULKAN_OBJ=%BUILD_DIR%\%%~nf.obj"+ if not exist "!VULKAN_OBJ!" (+ echo Compiling %%~nxf...+ cl %CXX_FLAGS% /I"%GGML_VULKAN_DIR%" /I"%VULKAN_SDK%\Include" /Fo"!VULKAN_OBJ!" "%%f"+ if errorlevel 1 (echo Error compiling %%~nxf & exit /b 1)+ ) else (+ echo %%~nf.obj is up to date+ ))echo.diff --git a/llamafile/vulkan.sh b/llamafile/vulkan.shindex eab04142..baf62532 100755--- a/llamafile/vulkan.sh+++ b/llamafile/vulkan.sh@@ -9,7 +9,7 @@# Build process:# 1. Build vulkan-shaders-gen tool (C++17)# 2. Generate shader C++ files from GLSL compute shaders using glslc-# 3. Compile ggml-vulkan.cpp with generated shaders+# 3. Compile the ggml-vulkan backend sources with generated shaders# 4. Compile core GGML sources# 5. Link into ggml-vulkan.so#@@ -192,7 +192,8 @@ echo ""FEATURES_STAMP="$BUILD_DIR/glslc-features.txt"if [ ! -f "$FEATURES_STAMP" ] || [ "$(cat "$FEATURES_STAMP" 2>/dev/null)" != "$GLSLC_DEFINES" ]; thenecho "glslc feature set changed; clearing cached shader artifacts"- rm -f "$BUILD_DIR/vulkan-shaders-gen" "$BUILD_DIR/ggml-vulkan-shaders.hpp" "$BUILD_DIR/ggml-vulkan.o"+ rm -f "$BUILD_DIR/vulkan-shaders-gen" "$BUILD_DIR/ggml-vulkan-shaders.hpp"+ rm -f "$BUILD_DIR"/ggml-vulkan*.orm -rf "$SHADERS_BUILD_DIR" "$SPVDIR"mkdir -p "$SHADERS_BUILD_DIR" "$SPVDIR"rm -f "$BUILD_DIR"/shader-*.o@@ -324,23 +325,38 @@ echo "Waiting for shader C++ compilation to finish..."wait#-# Phase 5: Compile ggml-vulkan.cpp+# Phase 5: Compile the backend sources+#+# Since b11100 the backend is split over several translation units+# (ggml-vulkan.cpp plus ggml-vulkan-buffers.cpp, ggml-vulkan-debug.cpp, ...),+# so take every .cpp at the top of ggml-vulkan/ rather than naming one. The+# shader generator under vulkan-shaders/ is built in phase 3 and excluded by+# -maxdepth 1.#echo ""-echo "Phase 5: Compiling ggml-vulkan.cpp..."+echo "Phase 5: Compiling backend sources..."++VULKAN_SRCS=$(find "$GGML_VULKAN_DIR" -maxdepth 1 -name "*.cpp" -type f | sort)+if [ -z "$VULKAN_SRCS" ]; then+ echo "Error: no backend sources found in $GGML_VULKAN_DIR"+ exit 1+fi-VULKAN_OBJ="$BUILD_DIR/ggml-vulkan.o"-VULKAN_SRC="$GGML_VULKAN_DIR/ggml-vulkan.cpp"+for src in $VULKAN_SRCS; do+ base=$(basename "$src" .cpp)+ obj="$BUILD_DIR/${base}.o"-if [ ! -f "$VULKAN_OBJ" ] || [ "$VULKAN_SRC" -nt "$VULKAN_OBJ" ] || [ "$SHADERS_HPP" -nt "$VULKAN_OBJ" ]; then- echo " Compiling ggml-vulkan.cpp..."+ if [ -f "$obj" ] && [ "$obj" -nt "$src" ] && [ "$obj" -nt "$SHADERS_HPP" ]; then+ echo " Skipping: $base (up to date)"+ continue+ fi++ echo " Compiling: $base"$CXX -c $CXX_FLAGS \-I$GGML_VULKAN_DIR \$SPIRV_INCLUDE \- -o "$VULKAN_OBJ" "$VULKAN_SRC"-else- echo " ggml-vulkan.o is up to date"-fi+ -o "$obj" "$src"+done## Phase 6: Compile core GGML sources