Skip to content

Distribute and run LLMs with a single file.

[draft] Update llama.cpp to b11100 (#1061)

Davide Eynardfef6e44e1b71
diff --git a/docs/cli_arguments.md b/docs/cli_arguments.md
index beda957c..5be2866d 100644
--- a/docs/cli_arguments.md
+++ b/docs/cli_arguments.md
@@ -89,12 +89,39 @@ compatibility.
| `-c, --ctx-size N` | Context window size. `0` means to use the model default. |
| `-b, --batch-size N` | Logical maximum batch size. |
| `-ub, --ubatch-size N` | Physical maximum batch size. |
-| `--mlock` | Keep the model in RAM instead of letting the OS swap or compress it. |
+| `-lm, --load-mode MODE` | How the weights are read: `auto` (default), `none`, `mmap`, `mlock`, `mmap+mlock`, `dio`. Replaces `--mmap`, `--no-mmap`, `--mlock` and `-dio`; see [Model Loading Flags](#model-loading-flags). |
| `--repeat-penalty N` | Penalize repeating tokens during sampling. `1.0` disables the penalty. |
| `-ngl, --gpu-layers, --n-gpu-layers N` | Number of layers to offload to GPU. |
| `--host HOST` | Server bind address. |
| `--port PORT` | Server listen port. |
+## Model Loading Flags
+
+llama.cpp b11100 replaced the separate `--mmap`, `--no-mmap`, `--mlock`,
+`-dio`/`--direct-io` and `-ndio`/`--no-direct-io` flags with a single
+`-lm, --load-mode MODE`, and no longer accepts the old flags. Use
+`--load-mode` in new command lines and `.args` files, as llama.cpp does.
+
+So that existing llamafiles and `.args` files keep working, llamafile still
+accepts the old flags. It prints a warning and replaces them with the
+equivalent `--load-mode`, keeping their original meaning: memory mapping is on
+unless `--no-mmap` is given, `--mlock` locks the model in RAM on top of that,
+and direct I/O takes precedence over memory mapping.
+
+| Old flags | Equivalent |
+| --- | --- |
+| `--mmap` | `--load-mode mmap` |
+| `--no-mmap` | `--load-mode none` |
+| `--mlock` | `--load-mode mmap+mlock` |
+| `--no-mmap --mlock` | `--load-mode mlock` |
+| `-dio`, `--direct-io` | `--load-mode dio` |
+| `-ndio`, `--no-direct-io` | `--load-mode mmap` |
+
+Several old flags combine into one mode, applied in command-line order. If
+you mix them with `--load-mode`, whichever comes last wins. The old
+`LLAMA_ARG_MMAP`, `LLAMA_ARG_MLOCK` and `LLAMA_ARG_DIO` environment variables
+are not translated; use `LLAMA_ARG_LOAD_MODE` instead.
+
## Wrapper and Mode Flags
- Mode selection: `--server`, `--chat`, `--cli`
@@ -125,7 +152,7 @@ wrapper.
- KV cache and host memory: `-kvo`, `--kv-offload`, `-nkvo`, `--no-kv-offload`, `--repack`, `-nr`, `--no-repack`, `--no-host`
- Cache storage and defragmentation: `-ctk`, `--cache-type-k`, `-ctv`, `--cache-type-v`, `-dt`, `--defrag-thold`
-- Memory mapping and residency: `--mlock`, `--mmap`, `--no-mmap`, `--numa`, `--check-tensors`, `--op-offload`, `--no-op-offload`
+- Memory mapping and residency: `-lm`, `--load-mode`, `-lzm`, `--lazy-mode`, `--numa`, `--check-tensors`, `--op-offload`, `--no-op-offload`
- Draft-model cache types: `-ctkd`, `--cache-type-k-draft`, `-ctvd`, `--cache-type-v-draft`
### Devices, GPU, Tensors, and Adapters
diff --git a/docs/creating_llamafiles.md b/docs/creating_llamafiles.md
index 1707fb8a..90e5ffd5 100644
--- a/docs/creating_llamafiles.md
+++ b/docs/creating_llamafiles.md
@@ -70,7 +70,8 @@ on
-n
32768
--no-context-shift
---no-mmap
+--load-mode
+none
...
```
@@ -81,6 +82,12 @@ on
> prefix is required whenever referencing a file packaged inside the llamafile.
> The `...` token is replaced with any additional CLI arguments the user passes
> at runtime.
+>
+> `--load-mode none` disables memory mapping. It replaces `--no-mmap`, which
+> llama.cpp removed in b11100. Existing `.args` files that still use
+> `--no-mmap`, `--mmap` or `--mlock` keep working, because llamafile
+> translates them (see [Model Loading Flags](cli_arguments.md#model-loading-flags)).
+> New ones should use `--load-mode`.
- Copy the llamafile executable and run zipalign to embed the weights and args:
@@ -128,7 +135,8 @@ embed both the weights and the following `.args` file
0.0.0.0
-ngl
9999
---no-mmap
+--load-mode
+none
...
```
diff --git a/llama.cpp b/llama.cpp
index 0177dcc7..7ab4ee7b 160000
--- a/llama.cpp
+++ b/llama.cpp
@@ -1 +1 @@
-Subproject commit 0177dcc7300bad8914bb838baabce87899812491
+Subproject commit 7ab4ee7baad2d920464cbacfad4f4b07cf111fd2
diff --git a/llama.cpp.patches/README.md b/llama.cpp.patches/README.md
index 67c7f7b5..a64f50a8 100644
--- a/llama.cpp.patches/README.md
+++ b/llama.cpp.patches/README.md
@@ -9,6 +9,7 @@ llama.cpp.patches/
├── README.md # This file
├── apply-patches.sh # Script to apply all patches to llama.cpp submodule
├── fetch-ui-assets.sh # Downloads + validates the prebuilt web UI (see Server Integration)
+├── ui-embed.sh # Renders upstream's ui.{cpp,h}.in templates (see Server Integration)
├── renames.sh # Script for file renames/moves (if any)
├── llamafile-files/ # Additional files to copy into llama.cpp
│ ├── BUILD.mk # Makefile for building llama.cpp with cosmocc
@@ -53,6 +54,9 @@ The `GGML_CALL` macro (defined as `__attribute__((__ms_abi__))` when `GGML_MULTI
| `ggml_src_ggml-cuda_ggml-cuda.cu.patch` | Adds `GGML_CALL` to all CUDA backend callback implementations (60+ functions); also adds `free_struct` and TinyBLAS BF16 guard (see below) |
| `ggml_src_ggml-metal_ggml-metal.cpp.patch` | Adds `GGML_CALL` to all Metal backend callback implementations (62 functions); also adds `free_struct` (see below) |
| `ggml_src_ggml-vulkan_ggml-vulkan.cpp.patch` | Adds `GGML_CALL` to all Vulkan backend callback implementations; also adds `free_struct` and a heap memory underflow fix (see below) |
+| `ggml_src_ggml-vulkan_ggml-vulkan-common.h.patch` | Adds `GGML_CALL` to the backend callback declarations. b11100 split the Vulkan backend into several translation units and moved the interface structs into `ggml-vulkan-buffers.cpp`, so the callbacks it holds became non-static and are declared here; declaration and definition must agree on the convention |
+| `ggml_src_ggml-vulkan_ggml-vulkan-buffers.cpp.patch` | Adds the `free_struct` entry to `ggml_backend_vk_buffer_interface`, which b11100 moved into this new file |
+| `ggml_src_ggml-metal_ggml-metal-fusion.h.patch` / `.cpp.patch` | Adds `GGML_CALL` to the `add_alloc_dep` function pointer these pass through (see Graph-optimize allocation dependencies below), and includes `ggml-backend.h` for the macro |
| `ggml_src_ggml-backend-meta.cpp.patch` | Adds `GGML_CALL` to all meta-device, meta-buffer-type, meta-buffer, and meta-backend callback implementations (the meta backend aggregates several simple backends behind one interface, so its callbacks are reached through the same function-pointer structs) |
### Cross-Module Memory Management
@@ -66,6 +70,39 @@ When GPU backends (CUDA, Vulkan, Metal) are loaded as dynamic libraries, memory
| `ggml_src_ggml-cuda_ggml-cuda.cu.patch` | Adds `free_struct` implementation for CUDA buffers (regular and host; upstream removed the split buffer in b10052); sets it on fallback CPU buffers allocated within the DSO |
| `ggml_src_ggml-metal_ggml-metal.cpp.patch` | Adds `free_struct` implementation for Metal shared and private buffers |
| `ggml_src_ggml-vulkan_ggml-vulkan.cpp.patch` | Adds `free_struct` implementation for Vulkan buffers and host buffer fallback path |
+| `ggml_src_ggml-vulkan_ggml-vulkan-buffers.cpp.patch` | Wires that `free_struct` into `ggml_backend_vk_buffer_interface`, which lives in this file since b11100 |
+
+### Graph-optimize allocation dependencies
+
+b11100 gave `graph_optimize` a `ggml_backend_graph_optimize_params *`, through
+which a backend tells the scheduler to keep a tensor alive until a later node
+(`params->add_alloc_dep(...)`). The callback is *the host's*, and the CUDA,
+Vulkan and Metal backends call it from inside their DSO, so it crosses the ABI
+boundary in the opposite direction to the backend interface structs and needs
+`GGML_CALL` just the same.
+
+| Patch | Description |
+|-------|-------------|
+| `ggml_src_ggml-backend-impl.h.patch` | Adds `GGML_CALL` to the `add_alloc_dep` member of `ggml_backend_graph_optimize_params` |
+| `ggml_src_ggml-backend.cpp.patch` | Replaces upstream's captureless lambda for `add_alloc_dep` with a named `ggml_backend_sched_add_alloc_dep()` — a lambda cannot carry the attribute |
+| `ggml_src_ggml-metal_ggml-metal-fusion.h.patch` / `.cpp.patch` | Annotates the same pointer where `ggml_metal_fusion_add_alloc_deps()` passes it on, and includes `ggml-backend.h` so the macro is in scope |
+
+### Generated version headers
+
+Since b11100 `ggml/src/ggml.c` includes `ggml-version.h` and `src/llama.cpp`
+includes `llama-version.h`; both are produced by CMake's `configure_file()`
+from the `.in` templates next to them, and hold the `GGML_VERSION`/`GGML_COMMIT`
+and `LLAMA_VERSION`/`LLAMA_COMMIT` macros that used to arrive as `-D` flags.
+There is no CMake step in the cosmocc build, so **`apply-patches.sh` generates
+both headers** (with the same values `build/config.mk` computes) straight into
+the source tree rather than into `o/`. That single location covers every
+consumer: the make build, `llamafile/*.sh` and `*.bat`, which compile `ggml.c`
+with `-I ggml/src`, and `llamafile/metal.c`, which extracts `ggml-version.h`
+alongside `ggml.c` for the runtime Metal build.
+
+They are generated, not patched: keep them out of `llamafile-files/` (delete
+both before running `generate-patches`, which would otherwise pick them up as
+new files and freeze one bump's commit hash into the repo).
### Cosmopolitan Libc Compatibility
@@ -84,8 +121,8 @@ Cosmopolitan libc has specific behaviors with condition variables and signals th
| Patch | Description |
|-------|-------------|
-| `common_log.cpp.patch` | Adds `#include <csignal>`; blocks `SIGINT`/`SIGTERM` on logger thread via `pthread_sigmask` to prevent `EINTR` exceptions; replaces `cv.wait()` with `wait_for(30s)` loop to work around XNU futex timeout bug (~72 minute expiry) |
-| `tools_server_server-models.cpp.patch` | Adds `#include <csignal>`; blocks signals on the stopping thread via `pthread_sigmask`; replaces untimed `cv.wait()` with `wait_for(30s)` loops on every model-lifecycle wait (`unload_lru`, the reload-drain wait, `stopping_thread`, the `is_reloading` guard in `load`, and the generic `wait()` predicate helper) to work around the XNU futex timeout bug |
+| `common_log.cpp.patch` | Adds `#include <csignal>`; blocks `SIGINT`/`SIGTERM` on logger thread via `pthread_sigmask` to prevent `EINTR` exceptions; replaces the untimed `cv.wait()` calls with `wait_for(30s)` loops (the worker's `cv_new` wait, plus the queue-full `cv_full` waits in `add()` and in `add_json()`, new upstream in b11100) to work around XNU futex timeout bug (~72 minute expiry). The `add*()` waits run on the logging caller's thread, so the signal mask does not cover them |
+| `tools_server_server-models.cpp.patch` | Adds `#include <csignal>`; blocks signals via `pthread_sigmask` on `server_monitor`'s thread (b11100 replaced the per-model `stopping_thread` this used to cover with one monitor thread watching every child); replaces untimed `cv.wait()` with `wait_for(30s)` loops on every model-lifecycle wait (`unload_lru`, the reload-drain wait, `unload_all`, the `is_reloading` guard in `load`, and the generic `wait()` predicate helper) to work around the XNU futex timeout bug |
| `tools_server_server-queue.cpp.patch` | Adds missing includes (`<cerrno>`, `<system_error>`, `<csignal>`); blocks `SIGINT`/`SIGTERM` on queue thread (the `yield_to_queue` worker thread is spawned after this, so it inherits the mask); replaces `wait()` with `wait_for(30s)` loops in five locations (`wait_until_no_sleep`, main loop, `recv`, plus `worker_loop` and `yield_to_queue`, both new upstream in b10441); runs `yield_to_queue()` inline when a GPU backend is loaded (see "GPU decode must stay on the main thread" below) |
| `tools_server_server-stream.cpp.patch` | Adds `#include <csignal>`; blocks signals on the stream-session GC thread via `pthread_sigmask`. Without it, Ctrl+C on `llama-server` aborts with *"libc++abi: terminating due to uncaught exception ... condition_variable timed_wait failed: Interrupted system call"* instead of shutting down: `SIGINT` is delivered to that thread while it sits in `gc_wake_cv.wait_for(60s)`, `pthread_cond_timedwait()` returns `EINTR`, and libcxx rethrows it as `std::system_error` that nothing catches. This thread was the last one in the server with a condition-variable wait and no signal mask (the log worker, task queue, model-stopping thread and httplib pool were already covered). Pre-existing, not introduced by b10441; combined TUI mode is unaffected |
| `vendor_cpp-httplib_httplib.cpp.patch` | Fixes httplib thread pool with `wait_for()` instead of `wait()` for XNU futex compatibility; also see HTTPS / TLS Support below |
@@ -197,6 +234,17 @@ serviced during it, so the sole effect is that `/metrics` and `/slots` wait for
the in-flight batch (~29 ms while generating; up to a few seconds for a full
prompt batch). This will recur whenever upstream moves work onto a new thread.
+**Bitmap hashing needs `vendor/hash`** (added to `BUILD.mk`, not a patch).
+b11100 made `mtmd-helper.cpp` derive a bitmap's ID from `hash_sha256_hex()` in
+the vendored `vendor/hash`, which upstream links into mtmd as `vendor::hash`.
+`BUILD.mk` builds `hash.cpp` and `sha256/sha256.c` into `MTMD_OBJS` (with
+`-iquote llama.cpp/vendor/hash`, since `sha256.c` reaches for
+`"rotate-bits/rotate-bits.h"`). Upstream's target also builds `xxhash` and
+`sha1`; nothing here calls either, and `sha1` would have to be compiled as C++
+despite its `.c` extension (its declarations sit in a namespace, and upstream
+forces `LANGUAGE CXX` on it), so neither is built. A future caller shows up as
+an undefined reference at link.
+
**Subprocess support needs `-DLLAMA_SUBPROCESS`** (set in `BUILD.mk`, not a
patch). b10441 moved child-process spawning into `common_subproc`
(`common/subproc.*`, wrapping the vendored `sheredom/subprocess.h`) and put it
@@ -210,7 +258,7 @@ llamafile's own flags and passes the rest to `common_params_parse`:
`server-mcp.cpp` spawns them and, unlike the other two, does **not** check
`common_subproc::is_supported()`, so without the define they fail silently
rather than reporting an error.
-- **`--server-tools`** — `server-tools.cpp` throws *"subprocess is not enabled
+- **`--tools`** — `server-tools.cpp` throws *"subprocess is not enabled
on this build"*, caught into a clean `return 1`.
- **Router mode** — `llama-server` with no model; `init_routes()` throws the
same message, and the server exits at startup.
@@ -221,8 +269,8 @@ with it. Enabling it pulls `subprocess.h` into the build, which needs the
Cosmopolitan fix in the patch table above.
The web UI moved upstream from prebuilt `tools/server/public/*` assets to
-a Svelte/PWA project under `tools/ui/`, embedded at CMake time via
-`tools/ui/embed.cpp`. cosmocc has no JS toolchain, so `fetch-ui-assets.sh`
+a Svelte/PWA project under `tools/ui/`, embedded into the binary at build
+time. cosmocc has no JS toolchain, so `fetch-ui-assets.sh`
(run by `apply-patches.sh` / `make setup`) downloads the prebuilt site
**tarball** `dist.tar.gz` (plus its `.sha256`) from the `ggml-org/llama-ui`
Hugging Face bucket — picking the newest `bNNNN` tag `<=` our pinned build —
@@ -233,31 +281,57 @@ verifies it, and extracts the whole static site into
screens). To keep the embedded payload small, the script also builds a
`dist/_gzip/` mirror with every file gzip-compressed under its original name.
-At build time, `llama.cpp/BUILD.mk` compiles `tools/ui/embed.cpp` with
-`cosmoc++` (its APE output runs on the host) and runs it against the `dist/`
-**directory** (the new `embed <out_cpp> <out_h> [<asset_dir>]` interface):
-`embed.cpp` recursively bakes every file — auto-detecting `dist/_gzip/` and
-emitting gzip-encoded assets keyed by relative path — into
-`o/$(MODE)/llama.cpp/tools/ui/ui.{cpp,h}`, which is compiled like any other
-C++ source and linked into `llama-server` and `llamafile`. Upstream's
-`server-http.cpp` (unpatched) registers a route per embedded asset
-(`index.html` at `/`) and serves gzip assets with `Content-Encoding: gzip`.
-If the download fails (offline, version not yet on HF) the script leaves
-`dist/` empty — `embed.cpp` then emits a no-op `llama_ui_find_asset` and the
-`LLAMA_UI_HAS_ASSETS` guard keeps the UI routes unregistered, so the REST
-API still works.
-
-Note that `embed.cpp` only takes that graceful no-asset path when `dist/` is
-*empty*: once the directory is non-empty it enforces a required-asset set
-(its `required_check[]` table — `index.html`, `manifest.webmanifest`, `sw.js`,
-`build.json`, `version.json`, and the hashed
-`bundle*.js`/`bundle*.css`/`workbox*.js`) and returns a hard error if any is
-missing. So `fetch-ui-assets.sh` validates the *same* set after extracting
-(`ui_missing_assets`); if a downloaded tarball is partial or has drifted, it
-clears `dist/` and falls back to a UI-less build rather than letting the embed
-step abort the whole build. Because `embed.cpp` is an upstream file, that list
-can change on a llama.cpp bump — when it does, the asset list in
-`ui_missing_assets` must be updated to match.
+b11100 deleted upstream's standalone `tools/ui/embed.cpp` and replaced it with
+`scripts/ui-assets.cmake`, which provisions the assets and renders
+`tools/ui/ui.{cpp,h}.in` into `ui.cpp`/`ui.h` inside a CMake build. There is no
+CMake step in the cosmocc build, so **`ui-embed.sh` does that rendering** — a
+translation of that script's `emit_files()`, in the same spirit as `BUILD.mk`
+being a translation of llama.cpp's CMake build.
+
+The important property is that it renders **upstream's own templates**, so the
+generated interface (`llama_ui_find_asset` / `llama_ui_get_assets` /
+`llama_ui_use_gzip` and `LLAMA_UI_HAS_ASSETS` — all that the unpatched
+`server-http.cpp` consumes) tracks upstream automatically and cannot drift.
+Only the substitutions live on our side: `@ASSET_ARRAYS@`, `@ASSET_TABLE@`,
+`@N_ASSETS@`, `@USE_GZIP@` and the `#cmakedefine`. ETags are the SHA-256 of the
+embedded bytes, as upstream computes them; the MIME table mirrors
+`mime_from_ext()`, and an extension missing from it degrades to
+`application/octet-stream` rather than breaking anything.
+
+At build time `llama.cpp/BUILD.mk` runs `ui-embed.sh <out_cpp> <out_h> [dist]`,
+which walks `dist/` (using the `dist/_gzip/` mirror when present, so the
+embedded bytes are gzip-compressed) and writes
+`o/$(MODE)/llama.cpp/tools/ui/ui.{cpp,h}`, compiled like any other C++ source
+and linked into `llama-server` and `llamafile`. `server-http.cpp` registers a
+route per embedded asset (`index.html` at `/`) and serves them with
+`Content-Encoding: gzip`. If the download fails (offline, version not yet on
+HF) the fetch script leaves `dist/` empty; `ui-embed.sh` then emits the
+no-asset stub and the `LLAMA_UI_HAS_ASSETS` guard keeps the UI routes
+unregistered, so the REST API still works. A zero-length asset takes the same
+path, with a warning — upstream hard-errors there, but aborting a whole build
+over a corrupt UI tarball is the wrong trade for us.
+
+`ui-embed.sh` embeds whatever it is given without validating the set, so the
+**one** required-asset check lives in `fetch-ui-assets.sh` (`ui_missing_assets`:
+`index.html`, `manifest.webmanifest`, `sw.js`, `build.json`, `version.json` and
+the hashed `bundle*.js`/`bundle*.css`/`workbox*.js`). When a downloaded tarball
+is partial or has drifted it clears `dist/` and takes the UI-less path. That
+list mirrors `ui_validate_assets()` in `scripts/ui-assets.cmake`, which is
+upstream's definition of a complete tree and the only remaining sync point in
+the UI path — when upstream changes it on a bump, update `ui_missing_assets` to
+match, or we will accept a tree upstream considers incomplete (silently broken
+UI) or reject one it considers fine (UI-less build).
+
+### Upstream fixes carried ahead of a release
+
+Patches that are **not** llamafile's own: an upstream fix we need before it
+lands in a tagged llama.cpp. Drop each one at the bump that first includes it,
+rather than reconciling it — `check_patches.sh` will flag it as conflicting
+once upstream has the same change.
+
+| Patch | Description |
+|-------|-------------|
+| `ggml_src_ggml-alloc.c.patch` | [PR #25584](https://github.com/ggml-org/llama.cpp/pull/25584), verbatim minus its test. `ggml_backend_alloc_ctx_tensors_from_buft()` splits a context across buffers when a tensor exceeds the backend's `max_size` (1 GiB on Vulkan); when the tail of the context holds only views, the final `alloc_tensor_range()` is skipped and those views never get `ggml_backend_view_init()`. The persistent KV stream views (`layer.k_stream`/`v_stream`) then keep `data == NULL`, and the server's state save/restore hits `GGML_ASSERT(tensor->data != NULL && "tensor not allocated")` in `ggml_backend_tensor_get()`. The fix allocates only parent tensors per split range and initializes every view in one final pass. **Symptom without it:** `llama-server` on Vulkan dies on the *second* request at the default `--parallel 4` (upstream [#29221](https://github.com/ggml-org/llama.cpp/issues/29221); `--parallel 1` is the workaround). Reproduced on an L40S with vanilla b11100, so it is not llamafile-specific; verified fixed on both vanilla and llamafile. Upstream also files it against #19839, #23737 and #21762. |
### Bug Fixes
@@ -266,10 +340,14 @@ can change on a llama.cpp bump — when it does, the asset list in
| `ggml_src_ggml-backend-reg.cpp.patch` | Suppresses debug log noise for non-existent backend search paths (irrelevant for llamafile's DSO loading approach) |
| `ggml_src_ggml-vulkan_ggml-vulkan.cpp.patch` | Fixes unsigned integer underflow in `ggml_backend_vk_get_device_memory` where Vulkan's `heapUsage` can exceed `heapBudget` (clamps to zero instead of wrapping) |
| `src_models_t5.cpp.patch` | Forward-declares the `graph<false>`/`graph<true>` explicit specializations before `build_arch_graph` so clang's `-std=gnu++23` doesn't reject them as specializations after implicit instantiation |
-| `src_models_eagle3.cpp.patch` | Moves `build_arch_graph` to the end of the file, after the `graph<true>`/`graph<false>` constructor specializations, so clang's `-std=gnu++23` doesn't reject them as explicit specializations appearing after the `make_unique<graph<...>>` implicit instantiation point |
-| `src_models_dflash.cpp.patch` | Same fix as `eagle3` for the DFlash model (new in b10052): moves `build_arch_graph` to the end of the file, after the `graph<true>`/`graph<false>`/`graph_dsv4` specializations, so clang's `-std=gnu++23` doesn't reject them as explicit specializations after the `make_unique<graph<...>>` implicit instantiation point |
| `ggml_src_ggml.c.patch` | Makes the `ggml_time_ms()`/`ggml_time_us()` call `ggml_time_init()` lazily. The Windows dylibs link their own copy of `ggml.c`, so the DLL's `timer_freq` is never set by the executable's `ggml_init()` and any `ggml_time_*()` call made from inside the DLL divides by zero (`0xC0000094`, surfaced by Cosmopolitan as SIGFPE). First hit when #1051 enabled `-DGGML_CUDA_USE_GRAPHS`, calling `ggml_time_us()` in `ggml-cuda`. Linux/macOS are unaffected (`clock_gettime`, no static divisor). |
+The same clang `-std=gnu++23` problem used to need patches for `eagle3.cpp` and
+`dflash.cpp`; b11100 defines `build_arch_graph` at the end of both files, after
+the specializations, so those two patches were dropped as obsolete. A new model
+file that defines `build_arch_graph` before its `graph<...>` specializations
+will reintroduce the error, and wants the same treatment.
+
## Creating New Patches
Files in `llama.cpp` are usually modified in-place for development and testing.
diff --git a/llama.cpp.patches/apply-patches.sh b/llama.cpp.patches/apply-patches.sh
index 2ddcd43a..95ca9e1d 100755
--- a/llama.cpp.patches/apply-patches.sh
+++ b/llama.cpp.patches/apply-patches.sh
@@ -37,6 +37,27 @@ cp -r "$LLAMAFILE_FILES_DIR"/* .
../llama.cpp.patches/renames.sh
+# Generate the version headers that CMake's configure_file() produces. Since
+# b11100 ggml/src/ggml.c #includes "ggml-version.h" and src/llama.cpp #includes
+# "llama-version.h"; before that the same macros came from -D flags. They are
+# written next to their sources (rather than into o/) so that every consumer
+# finds them: the cosmocc make build, and llamafile/*.sh + *.bat, which compile
+# ggml.c with -I ggml/src.
+echo "Generating version headers..."
+ggml_version_major=$(grep -E 'GGML_VERSION_MAJOR [0-9]+' ggml/CMakeLists.txt | sed 's/[^0-9]*//g')
+ggml_version_minor=$(grep -E 'GGML_VERSION_MINOR [0-9]+' ggml/CMakeLists.txt | sed 's/[^0-9]*//g')
+ggml_version_patch=$(grep -E 'GGML_VERSION_PATCH [0-9]+' ggml/CMakeLists.txt | sed 's/[^0-9]*//g')
+ggml_commit=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown")
+llama_version=$(git describe --tags --always 2>/dev/null || echo "unknown")
+
+sed -e "s/@GGML_VERSION@/${ggml_version_major}.${ggml_version_minor}.${ggml_version_patch}/g" \
+ -e "s/@GGML_BUILD_COMMIT@/${ggml_commit}/g" \
+ ggml/src/ggml-version.h.in > ggml/src/ggml-version.h
+
+sed -e "s/@LLAMA_VERSION@/${llama_version}/g" \
+ -e "s/@LLAMA_BUILD_COMMIT@/${ggml_commit}/g" \
+ src/llama-version.h.in > src/llama-version.h
+
echo "Removing unnecessary files and directories..."
# If you want to clean up the original code, add your `rm` commands here.
# For example:
diff --git a/llama.cpp.patches/fetch-ui-assets.sh b/llama.cpp.patches/fetch-ui-assets.sh
index 6180f173..c81257e2 100755
--- a/llama.cpp.patches/fetch-ui-assets.sh
+++ b/llama.cpp.patches/fetch-ui-assets.sh
@@ -70,23 +70,27 @@ build_gzip_mirror() {
( cd "$UI_DIST" && find . -type f ! -path './_gzip/*' -print0 \
| while IFS= read -r -d '' f; do
mkdir -p "_gzip/$(dirname "$f")"
- gzip -9 -c "$f" > "_gzip/$f"
+ # -n: omit the timestamp, so identical assets give identical bytes
+ # (and therefore stable ETags) on every machine and every fetch.
+ gzip -9 -n -c "$f" > "_gzip/$f"
done )
}
-# Echo (one per line) any asset embed.cpp requires that is absent from the tree.
+# Echo (one per line) any asset the web UI needs at runtime that is absent from
+# the tree.
#
-# !!! KEEP IN SYNC WITH UPSTREAM tools/ui/embed.cpp !!!
-# The list below mirrors the required_check[] table in llama.cpp/tools/ui/embed.cpp.
-# Once dist/ is non-empty, embed.cpp hard-fails the build (return 1) if any of
-# those assets is missing. If we only checked index.html here, a partial/drifted
-# tarball would pass our check but then abort the *entire* build at the embed
-# step instead of falling back to a UI-less build. So we validate the same set
-# embed.cpp does and, when it's incomplete, clear dist/ to take the UI-less path.
+# !!! KEEP IN SYNC WITH UPSTREAM scripts/ui-assets.cmake !!!
+# The list below mirrors ui_validate_assets() in llama.cpp/scripts/ui-assets.cmake,
+# which is upstream's definition of a complete asset tree. This is the only copy
+# of it on our side: ui-embed.sh embeds whatever it is given without validating,
+# so a partial or drifted tarball that slipped past this check would silently
+# produce a broken UI rather than a failed build. When the set is incomplete we
+# clear dist/ and take the UI-less path instead.
#
-# embed.cpp is an UPSTREAM file, so it can change on a llama.cpp bump. When it
-# does (a new required asset, a renamed one), this list must be updated to match
-# -- otherwise the two checks disagree again and the build-failure bug returns.
+# ui-assets.cmake is an UPSTREAM file, so its list can change on a llama.cpp
+# bump. When it does (a new required asset, a renamed one), update this list to
+# match -- otherwise we accept a tree upstream considers incomplete, or reject
+# one it considers fine.
ui_missing_assets() {
local root="$1" f b
local -a bases=()
diff --git a/llama.cpp.patches/llamafile-files/BUILD.mk b/llama.cpp.patches/llamafile-files/BUILD.mk
index 1c680c60..9e670287 100644
--- a/llama.cpp.patches/llamafile-files/BUILD.mk
+++ b/llama.cpp.patches/llamafile-files/BUILD.mk
@@ -34,6 +34,7 @@ GGML_SRCS_CPP := \
llama.cpp/ggml/src/ggml-cpu/binary-ops.cpp \
llama.cpp/ggml/src/ggml-cpu/ggml-cpu.cpp \
llama.cpp/ggml/src/ggml-cpu/hbm.cpp \
+ llama.cpp/ggml/src/ggml-cpu/iqp.cpp \
llama.cpp/ggml/src/ggml-cpu/ops.cpp \
llama.cpp/ggml/src/ggml-cpu/repack.cpp \
llama.cpp/ggml/src/ggml-cpu/traits.cpp \
@@ -60,6 +61,7 @@ LLAMA_SRCS_CPP := \
llama.cpp/src/models/baichuan.cpp \
llama.cpp/src/models/bailingmoe.cpp \
llama.cpp/src/models/bailingmoe2.cpp \
+ llama.cpp/src/models/bailingmoe3.cpp \
llama.cpp/src/models/bert.cpp \
llama.cpp/src/models/bitnet.cpp \
llama.cpp/src/models/bloom.cpp \
@@ -81,6 +83,7 @@ LLAMA_SRCS_CPP := \
llama.cpp/src/models/delta-net-base.cpp \
llama.cpp/src/models/dflash.cpp \
llama.cpp/src/models/dots1.cpp \
+ llama.cpp/src/models/dots3note.cpp \
llama.cpp/src/models/dream.cpp \
llama.cpp/src/models/eagle3.cpp \
llama.cpp/src/models/ernie4-5-moe.cpp \
@@ -107,6 +110,7 @@ LLAMA_SRCS_CPP := \
llama.cpp/src/models/granite-moe.cpp \
llama.cpp/src/models/granite-switch.cpp \
llama.cpp/src/models/granite.cpp \
+ llama.cpp/src/models/granite-swa.cpp \
llama.cpp/src/models/mamba-base.cpp \
llama.cpp/src/models/grok.cpp \
llama.cpp/src/models/grovemoe.cpp \
@@ -114,6 +118,8 @@ LLAMA_SRCS_CPP := \
llama.cpp/src/models/hunyuan-moe.cpp \
llama.cpp/src/models/hunyuan-vl.cpp \
llama.cpp/src/models/hy-v3.cpp \
+ llama.cpp/src/models/hy-v4.cpp \
+ llama.cpp/src/models/hrm-text.cpp \
llama.cpp/src/models/internlm2.cpp \
llama.cpp/src/models/jais.cpp \
llama.cpp/src/models/jais2.cpp \
@@ -121,6 +127,7 @@ LLAMA_SRCS_CPP := \
llama.cpp/src/models/jina-bert-v2.cpp \
llama.cpp/src/models/jina-bert-v3.cpp \
llama.cpp/src/models/kimi-linear.cpp \
+ llama.cpp/src/models/kimi-k3.cpp \
llama.cpp/src/models/laguna.cpp \
llama.cpp/src/models/lfm2.cpp \
llama.cpp/src/models/lfm2moe.cpp \
@@ -132,6 +139,7 @@ LLAMA_SRCS_CPP := \
llama.cpp/src/models/maincoder.cpp \
llama.cpp/src/models/mamba.cpp \
llama.cpp/src/models/mamba2.cpp \
+ llama.cpp/src/models/maple.cpp \
llama.cpp/src/models/mellum.cpp \
llama.cpp/src/models/mimo2.cpp \
llama.cpp/src/models/minicpm.cpp \
@@ -179,6 +187,7 @@ LLAMA_SRCS_CPP := \
llama.cpp/src/models/qwen35moe.cpp \
llama.cpp/src/models/qwen3vl.cpp \
llama.cpp/src/models/qwen3vlmoe.cpp \
+ llama.cpp/src/models/qwen4exp.cpp \
llama.cpp/src/models/refact.cpp \
llama.cpp/src/models/rnd1.cpp \
llama.cpp/src/models/rwkv6-base.cpp \
@@ -189,6 +198,7 @@ LLAMA_SRCS_CPP := \
llama.cpp/src/models/seed-oss.cpp \
llama.cpp/src/models/smallthinker.cpp \
llama.cpp/src/models/smollm3.cpp \
+ llama.cpp/src/models/spark2-5.cpp \
llama.cpp/src/models/stablelm.cpp \
llama.cpp/src/models/starcoder.cpp \
llama.cpp/src/models/starcoder2.cpp \
@@ -210,11 +220,13 @@ LLAMA_SRCS_CPP := \
llama.cpp/src/llama-impl.cpp \
llama.cpp/src/llama-io.cpp \
llama.cpp/src/llama-kv-cache-dsa.cpp \
+ llama.cpp/src/llama-kv-cache-dsa-iswa.cpp \
llama.cpp/src/llama-kv-cache-dsv4.cpp \
llama.cpp/src/llama-kv-cache-iswa.cpp \
llama.cpp/src/llama-kv-cache-msa.cpp \
llama.cpp/src/llama-kv-cache.cpp \
llama.cpp/src/llama-memory-hybrid.cpp \
+ llama.cpp/src/llama-memory-hybrid-idx.cpp \
llama.cpp/src/llama-memory-hybrid-iswa.cpp \
llama.cpp/src/llama-memory-recurrent.cpp \
llama.cpp/src/llama-memory.cpp \
@@ -255,12 +267,30 @@ COMMON_SRCS_CPP := \
llama.cpp/common/jinja/string.cpp \
llama.cpp/common/jinja/value.cpp \
llama.cpp/common/json-schema-to-grammar.cpp \
+ llama.cpp/common/json-schema.cpp \
+ llama.cpp/common/json.cpp \
llama.cpp/common/license.cpp \
llama.cpp/common/llguidance.cpp \
llama.cpp/common/log.cpp \
llama.cpp/common/ngram-cache.cpp \
llama.cpp/common/ngram-map.cpp \
llama.cpp/common/ngram-mod.cpp \
+ llama.cpp/common/parsers/parsers.cpp \
+ llama.cpp/common/parsers/cohere2moe.cpp \
+ llama.cpp/common/parsers/deepseek.cpp \
+ llama.cpp/common/parsers/functionary-v3-2.cpp \
+ llama.cpp/common/parsers/gemma4.cpp \
+ llama.cpp/common/parsers/gigachat-v3.cpp \
+ llama.cpp/common/parsers/gpt-oss.cpp \
+ llama.cpp/common/parsers/kimi-k2.cpp \
+ llama.cpp/common/parsers/kimi-k3.cpp \
+ llama.cpp/common/parsers/lfm2.cpp \
+ llama.cpp/common/parsers/ling3.cpp \
+ llama.cpp/common/parsers/minicpm5.cpp \
+ llama.cpp/common/parsers/minimax-m3.cpp \
+ llama.cpp/common/parsers/ministral3.cpp \
+ llama.cpp/common/parsers/muse-glimmer.cpp \
+ llama.cpp/common/parsers/qwen3-coder.cpp \
llama.cpp/common/peg-parser.cpp \
llama.cpp/common/preset.cpp \
llama.cpp/common/reasoning-budget.cpp \
@@ -325,8 +355,10 @@ MTMD_SRCS_CPP := \
llama.cpp/tools/mtmd/models/cogvlm.cpp \
llama.cpp/tools/mtmd/models/deepseekocr.cpp \
llama.cpp/tools/mtmd/models/deepseekocr2.cpp \
+ llama.cpp/tools/mtmd/models/deepseek4v.cpp \
llama.cpp/tools/mtmd/models/conformer.cpp \
llama.cpp/tools/mtmd/models/dotsocr.cpp \
+ llama.cpp/tools/mtmd/models/dots3note.cpp \
llama.cpp/tools/mtmd/models/exaone4_5.cpp \
llama.cpp/tools/mtmd/models/gemma4a.cpp \
llama.cpp/tools/mtmd/models/gemma4ua.cpp \
@@ -365,7 +397,29 @@ MTMD_SRCS_CPP := \
llama.cpp/tools/mtmd/models/yasa2.cpp \
llama.cpp/tools/mtmd/models/youtuvl.cpp
-MTMD_OBJS := $(MTMD_SRCS_CPP:%.cpp=o/$(MODE)/%.cpp.o)
+# Vendored hashing, linked into mtmd by upstream as vendor::hash. Since b11100
+# mtmd-helper.cpp derives bitmap IDs with hash_sha256_hex() from it, which is
+# the only entry point hash.h exposes and the only one anything here calls.
+# Upstream's vendor/hash also builds xxhash and sha1; both are unreferenced, and
+# sha1 would need compiling as C++ despite its .c extension (its declarations
+# live in a namespace; upstream forces LANGUAGE CXX on it), so neither is built
+# here. Adding a caller for them shows up as an undefined reference at link.
+VENDOR_HASH_SRCS_C := \
+ llama.cpp/vendor/hash/sha256/sha256.c
+
+VENDOR_HASH_SRCS_CPP := \
+ llama.cpp/vendor/hash/hash.cpp
+
+VENDOR_HASH_OBJS := \
+ $(VENDOR_HASH_SRCS_C:%.c=o/$(MODE)/%.c.o) \
+ $(VENDOR_HASH_SRCS_CPP:%.cpp=o/$(MODE)/%.cpp.o)
+
+MTMD_OBJS := \
+ $(MTMD_SRCS_CPP:%.cpp=o/$(MODE)/%.cpp.o) \
+ $(VENDOR_HASH_OBJS)
+
+# sha256.c reaches for "rotate-bits/rotate-bits.h" next to it
+$(VENDOR_HASH_OBJS): private CPPFLAGS += -iquote llama.cpp/vendor/hash
# ==============================================================================
# cpp-httplib (HTTP library for server)
@@ -379,25 +433,29 @@ HTTPLIB_OBJS := $(HTTPLIB_SRCS:%.cpp=o/$(MODE)/%.cpp.o)
# ==============================================================================
#
# Upstream switched from prebuilt bundles in tools/server/public/ to a
-# Svelte/PWA project under tools/ui/, embedded at CMake time by
-# tools/ui/embed.cpp into a generated ui.cpp + ui.h. cosmocc has no JS
-# toolchain, so apply-patches.sh (run by `make setup`) downloads the
-# prebuilt site tarball (dist.tar.gz) from the ggml-org/llama-ui Hugging
-# Face bucket and extracts the whole static site into
-# llama.cpp/tools/ui/dist/ (see fetch-ui-assets.sh). embed.cpp then
-# recursively embeds every file under that directory, keyed by its
-# relative path (e.g. "_app/immutable/bundle.HASH.js"). fetch-ui-assets.sh
-# also builds a dist/_gzip/ mirror of gzip-compressed files; embed.cpp
-# auto-detects it and emits gzip-encoded assets (keeping the embedded
-# payload small), which server-http.cpp serves with Content-Encoding: gzip.
+# Svelte/PWA project under tools/ui/, embedded into a generated ui.cpp + ui.h
+# at build time. cosmocc has no JS toolchain, so apply-patches.sh (run by
+# `make setup`) downloads the prebuilt site tarball (dist.tar.gz) from the
+# ggml-org/llama-ui Hugging Face bucket and extracts the whole static site into
+# llama.cpp/tools/ui/dist/, plus a dist/_gzip/ mirror of gzip-compressed files
+# (see fetch-ui-assets.sh).
+#
+# b11100 replaced upstream's standalone tools/ui/embed.cpp with
+# scripts/ui-assets.cmake, which renders tools/ui/ui.{cpp,h}.in inside a CMake
+# build. There is no CMake step here, so ui-embed.sh does that rendering — a
+# translation of that script's emit_files(), the same way this file is a
+# translation of llama.cpp's CMake build. It renders upstream's own templates,
+# so the generated interface cannot drift from what server-http.cpp expects.
+#
# With assets present, ui.h defines LLAMA_UI_HAS_ASSETS and server-http.cpp
-# registers a route per asset (index.html at /); without them, embed.cpp
-# emits a no-op llama_ui_find_asset and the UI routes stay unregistered.
+# registers a route per asset (index.html at /), serving the gzip-encoded bytes
+# with Content-Encoding: gzip; without them, the generated llama_ui_find_asset
+# is a no-op and the UI routes stay unregistered.
UI_DIST := llama.cpp/tools/ui/dist
UI_GEN_DIR := o/$(MODE)/llama.cpp/tools/ui
-UI_EMBED_SRC := llama.cpp/tools/ui/embed.cpp
-UI_EMBED_TOOL := $(UI_GEN_DIR)/llama-ui-embed
+UI_EMBED_SH := llama.cpp.patches/ui-embed.sh
+UI_TEMPLATES := llama.cpp/tools/ui/ui.cpp.in llama.cpp/tools/ui/ui.h.in
UI_CPP_GEN := $(UI_GEN_DIR)/ui.cpp
UI_H_GEN := $(UI_GEN_DIR)/ui.h
@@ -408,21 +466,14 @@ UI_H_GEN := $(UI_GEN_DIR)/ui.h
# letting the build proceed UI-less (offline / asset build not yet published).
UI_ASSETS_INDEX_HTML := $(wildcard $(UI_DIST)/index.html)
-# Build embed.cpp standalone (no llamafile flags, no llama.cpp includes).
-# cosmoc++ produces an APE that runs on the build host, so we don't need
-# a separate system compiler. Compiled with stock C++17 (embed.cpp uses
-# <filesystem>) so the source isn't entangled with -DCOSMOCC or GGML defines.
-$(UI_EMBED_TOOL): $(UI_EMBED_SRC) $(COSMOCC)
- @mkdir -p $(@D)
- $(CXX) -O2 -std=gnu++17 -o $@ $<
-
-# Generate ui.cpp/ui.h. Re-run when the embed tool or the fetched UI changes.
-# When dist/ has assets, pass the directory: embed.cpp recurses it (auto-using
-# dist/_gzip when present). When dist/ is empty, pass no directory so embed
-# emits its no-asset stub.
-$(UI_CPP_GEN) $(UI_H_GEN) &: $(UI_EMBED_TOOL) $(UI_ASSETS_INDEX_HTML)
+# Generate ui.cpp/ui.h. Re-runs when the generator, upstream's templates or the
+# fetched UI change. When dist/ has assets, pass the directory (the script picks
+# up dist/_gzip itself); when it is empty, pass none so it emits the no-asset
+# stub. The script rewrites its outputs only when their contents change, so an
+# unchanged re-fetch does not cascade a rebuild.
+$(UI_CPP_GEN) $(UI_H_GEN) &: $(UI_EMBED_SH) $(UI_TEMPLATES) $(UI_ASSETS_INDEX_HTML)
@mkdir -p $(UI_GEN_DIR)
- $(UI_EMBED_TOOL) $(UI_CPP_GEN) $(UI_H_GEN) \
+ $(UI_EMBED_SH) $(UI_CPP_GEN) $(UI_H_GEN) \
$(if $(UI_ASSETS_INDEX_HTML),$(UI_DIST))
# ==============================================================================
@@ -521,13 +572,9 @@ $(UI_GEN_OBJ): private CPPFLAGS += -iquote $(UI_GEN_DIR)
$(UI_GEN_OBJ): $(UI_H_GEN)
# Version definitions
-$(GGML_OBJS): private CCFLAGS += \
- -DGGML_VERSION=\"$(GGML_VERSION)\" \
- -DGGML_COMMIT=\"$(GGML_COMMIT)\"
-
-$(LLAMA_OBJS): private CCFLAGS += \
- -DLLAMA_VERSION=\"$(LLAMA_VERSION)\" \
- -DLLAMA_COMMIT=\"$(LLAMA_COMMIT)\"
+# ggml/src/ggml.c and src/llama.cpp #include "ggml-version.h" / "llama-version.h",
+# which CMake generates with configure_file(). apply-patches.sh writes them next
+# to their sources from the same .in templates, so no -D flags are needed here.
# Base flags for all objects
$(LLAMA_CPP_OBJS) $(TOOL_SERVER_OBJS): private CCFLAGS += \
diff --git a/llama.cpp.patches/patches/common_arg.cpp.patch b/llama.cpp.patches/patches/common_arg.cpp.patch
index 2e371ba7..a8c55ce1 100644
--- a/llama.cpp.patches/patches/common_arg.cpp.patch
+++ b/llama.cpp.patches/patches/common_arg.cpp.patch
@@ -1,7 +1,7 @@
diff --git a/common/arg.cpp b/common/arg.cpp
--- a/llama.cpp/common/arg.cpp
+++ b/llama.cpp/common/arg.cpp
-@@ -42,6 +42,8 @@
+@@ -41,6 +41,8 @@
#ifndef __EMSCRIPTEN__
#ifdef __linux__
#include <linux/limits.h>
diff --git a/llama.cpp.patches/patches/common_common.cpp.patch b/llama.cpp.patches/patches/common_common.cpp.patch
index 226ea76b..1b0565e0 100644
--- a/llama.cpp.patches/patches/common_common.cpp.patch
+++ b/llama.cpp.patches/patches/common_common.cpp.patch
@@ -52,7 +52,7 @@ diff --git a/common/common.cpp b/common/common.cpp
int32_t logical_cpus = _system_configuration.ncpus;
int32_t smt_threads = _system_configuration.smt_threads;
if (smt_threads > 0) {
-@@ -1058,6 +1093,20 @@ std::string fs_get_cache_directory() {
+@@ -1059,6 +1094,20 @@ std::string fs_get_cache_directory() {
if (cache_directory.empty()) {
throw std::runtime_error("Failed to find %LOCALAPPDATA% directory");
}
@@ -73,7 +73,7 @@ diff --git a/common/common.cpp b/common/common.cpp
#elif defined(__EMSCRIPTEN__)
GGML_ABORT("not implemented on this platform");
#else
-@@ -1103,6 +1152,20 @@ std::string fs_get_config_directory() {
+@@ -1104,6 +1153,20 @@ std::string fs_get_config_directory() {
if (config_directory.empty()) {
throw std::runtime_error("Failed to find %APPDATA% directory");
}
@@ -94,11 +94,10 @@ diff --git a/common/common.cpp b/common/common.cpp
#elif defined(__EMSCRIPTEN__)
// caller decides what to do when there is no config directory
throw std::runtime_error("not implemented on this platform");
-@@ -1294,10 +1357,31 @@ common_init_result::common_init_result(common_params & params, bool model_only)
- if (params.fit_params) {
+@@ -1296,6 +1359,26 @@ common_init_result::common_init_result(common_params & params, bool model_only)
COM_TRC("%s", "fitting params to device memory ...\n");
COM_TRC("%s", "(for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)\n");
-+
+
+ // if a multimodal projection model is specified and will be loaded on GPU,
+ // add its estimated size to the fit margins so the fit leaves enough room for it
+ std::vector<size_t> fit_params_target = params.fit_params_target;
@@ -119,11 +118,15 @@ diff --git a/common/common.cpp b/common/common.cpp
+ }
+ }
+
+ // the draft context is created from the same base params and follows the main context, fit both together
+ const bool has_draft = params.speculative.has_dft();
+ const bool spec_mtp = std::find(params.speculative.types.begin(), params.speculative.types.end(),
+@@ -1320,7 +1403,7 @@ common_init_result::common_init_result(common_params & params, bool model_only)
common_fit_params(params.model.path.c_str(), &mparams, &cparams,
params.tensor_split,
params.tensor_buft_overrides.data(),
- params.fit_params_target.data(),
+ fit_params_target.data(),
params.fit_params_min_ctx,
+ has_draft || spec_mtp ? &extra : nullptr,
params.verbosity >= LOG_LEVEL_DEBUG ? GGML_LOG_LEVEL_DEBUG : GGML_LOG_LEVEL_ERROR);
- }
diff --git a/llama.cpp.patches/patches/common_download.cpp.patch b/llama.cpp.patches/patches/common_download.cpp.patch
index f50ed915..a731fcb1 100644
--- a/llama.cpp.patches/patches/common_download.cpp.patch
+++ b/llama.cpp.patches/patches/common_download.cpp.patch
@@ -1,7 +1,7 @@
diff --git a/common/download.cpp b/common/download.cpp
--- a/llama.cpp/common/download.cpp
+++ b/llama.cpp/common/download.cpp
-@@ -26,6 +26,8 @@
+@@ -24,6 +24,8 @@
#ifndef __EMSCRIPTEN__
#ifdef __linux__
#include <linux/limits.h>
diff --git a/llama.cpp.patches/patches/common_log.cpp.patch b/llama.cpp.patches/patches/common_log.cpp.patch
index 5b7215cc..e80ea789 100644
--- a/llama.cpp.patches/patches/common_log.cpp.patch
+++ b/llama.cpp.patches/patches/common_log.cpp.patch
@@ -1,7 +1,7 @@
diff --git a/common/log.cpp b/common/log.cpp
--- a/llama.cpp/common/log.cpp
+++ b/llama.cpp/common/log.cpp
-@@ -24,6 +24,7 @@
+@@ -25,6 +25,7 @@
# define fileno _fileno
#else
# include <unistd.h>
@@ -9,7 +9,37 @@ diff --git a/common/log.cpp b/common/log.cpp
#endif // defined(_WIN32)
int common_log_verbosity_thold = LOG_DEFAULT_LLAMA;
-@@ -265,9 +266,32 @@ public:
+@@ -243,7 +244,13 @@ public:
+ std::unique_lock<std::mutex> lock(mtx);
+
+ // block if the queue is full
+- cv_full.wait(lock, [this]() { return !running || !is_full(); });
++ // Use wait_for() instead of an untimed wait to work around a
++ // Cosmopolitan libc bug where untimed futex waits on XNU (macOS)
++ // expire after ~72 minutes, causing condition_variable::wait() to
++ // throw ETIMEDOUT.
++ while (running && is_full()) {
++ cv_full.wait_for(lock, std::chrono::seconds(30));
++ }
+
+ if (!running) {
+ // discard messages while the worker thread is paused
+@@ -310,7 +317,13 @@ public:
+ std::unique_lock<std::mutex> lock(mtx);
+
+ // block if the queue is full
+- cv_full.wait(lock, [this]() { return !running || !is_full(); });
++ // Use wait_for() instead of an untimed wait to work around a
++ // Cosmopolitan libc bug where untimed futex waits on XNU (macOS)
++ // expire after ~72 minutes, causing condition_variable::wait() to
++ // throw ETIMEDOUT.
++ while (running && is_full()) {
++ cv_full.wait_for(lock, std::chrono::seconds(30));
++ }
+
+ if (!running) {
+ // discard messages while the worker thread is paused
+@@ -345,9 +358,32 @@ public:
running = true;
thrd = std::thread([this]() {
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-alloc.c.patch b/llama.cpp.patches/patches/ggml_src_ggml-alloc.c.patch
new file mode 100644
index 00000000..05ca1a25
--- /dev/null
+++ b/llama.cpp.patches/patches/ggml_src_ggml-alloc.c.patch
@@ -0,0 +1,67 @@
+diff --git a/ggml/src/ggml-alloc.c b/ggml/src/ggml-alloc.c
+--- a/llama.cpp/ggml/src/ggml-alloc.c
++++ b/llama.cpp/ggml/src/ggml-alloc.c
+@@ -1124,6 +1124,20 @@ static void free_buffers(ggml_backend_buffer_t ** buffers, const size_t * n_buff
+ free(*buffers);
+ }
+
++static bool alloc_ctx_tensors_finalize_views(struct ggml_context * ctx) {
++ for (struct ggml_tensor * t = ggml_get_first_tensor(ctx); t != NULL; t = ggml_get_next_tensor(ctx, t)) {
++ if (t->view_src != NULL && t->buffer == NULL) {
++ enum ggml_status status = ggml_backend_view_init(t);
++ if (status != GGML_STATUS_SUCCESS) {
++ GGML_LOG_ERROR("%s: failed to initialize view tensor %s\n", __func__, t->name);
++ return false;
++ }
++ }
++ }
++
++ return true;
++}
++
+ static bool alloc_tensor_range(struct ggml_context * ctx,
+ struct ggml_tensor * first, struct ggml_tensor * last,
+ ggml_backend_buffer_type_t buft, size_t size,
+@@ -1142,24 +1156,14 @@ static bool alloc_tensor_range(struct ggml_context * ctx,
+ struct ggml_tallocr tallocr = ggml_tallocr_new(buffer);
+
+ for (struct ggml_tensor * t = first; t != last; t = ggml_get_next_tensor(ctx, t)) {
+- enum ggml_status status = GGML_STATUS_SUCCESS;
+- if (t->data == NULL) {
+- if (t->view_src == NULL) {
+- status = ggml_tallocr_alloc(&tallocr, t);
+- } else if (t->buffer == NULL) {
+- status = ggml_backend_view_init(t);
+- }
+- } else {
+- if (t->view_src != NULL && t->buffer == NULL) {
+- // view of a pre-allocated tensor
+- status = ggml_backend_view_init(t);
++ if (t->view_src == NULL && t->data == NULL) {
++ enum ggml_status status = ggml_tallocr_alloc(&tallocr, t);
++ if (status != GGML_STATUS_SUCCESS) {
++ GGML_LOG_ERROR("%s: failed to allocate tensor %s\n", __func__, t->name);
++ free_buffers(buffers, n_buffers);
++ return false;
+ }
+ }
+- if (status != GGML_STATUS_SUCCESS) {
+- GGML_LOG_ERROR("%s: failed to initialize tensor %s\n", __func__, t->name);
+- free_buffers(buffers, n_buffers);
+- return false;
+- }
+ }
+
+ return true;
+@@ -1217,6 +1221,11 @@ static ggml_backend_buffer_t ggml_backend_alloc_ctx_tensors_from_buft_impl(
+ return NULL;
+ }
+
++ if (!alloc_ctx_tensors_finalize_views(ctx)) {
++ free_buffers(&buffers, &n_buffers);
++ return NULL;
++ }
++
+ ggml_backend_buffer_t buffer;
+ if (n_buffers == 1) {
+ buffer = buffers[0];
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-backend-impl.h.patch b/llama.cpp.patches/patches/ggml_src_ggml-backend-impl.h.patch
index 3aaec764..138340bf 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml-backend-impl.h.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml-backend-impl.h.patch
@@ -25,7 +25,7 @@ diff --git a/ggml/src/ggml-backend-impl.h b/ggml/src/ggml-backend-impl.h
};
struct ggml_backend_buffer_type {
-@@ -39,26 +39,30 @@ extern "C" {
+@@ -44,26 +44,30 @@ extern "C" {
//
struct ggml_backend_buffer_i {
@@ -68,8 +68,14 @@ diff --git a/ggml/src/ggml-backend-impl.h b/ggml/src/ggml-backend-impl.h
};
struct ggml_backend_buffer {
-@@ -103,40 +107,40 @@ extern "C" {
- //
+@@ -114,45 +118,45 @@ extern "C" {
+ struct ggml_backend_graph_optimize_params {
+ // keep `tensor` allocated at least until `until` (a node of the same graph) has been computed
+ // can be called multiple times for the same tensor: the longest lifetime applies
+- void (*add_alloc_dep)(void * user_data, struct ggml_tensor * tensor, struct ggml_tensor * until);
++ void (GGML_CALL *add_alloc_dep)(void * user_data, struct ggml_tensor * tensor, struct ggml_tensor * until);
+ void * user_data;
+ };
struct ggml_backend_i {
- const char * (*get_name)(ggml_backend_t backend);
@@ -120,12 +126,12 @@ diff --git a/ggml/src/ggml-backend-impl.h b/ggml/src/ggml-backend-impl.h
+ void (GGML_CALL *event_wait) (ggml_backend_t backend, ggml_backend_event_t event);
// (optional) sort/optimize the nodes in the graph
-- void (*graph_optimize) (ggml_backend_t backend, struct ggml_cgraph * cgraph);
-+ void (GGML_CALL *graph_optimize) (ggml_backend_t backend, struct ggml_cgraph * cgraph);
+- void (*graph_optimize) (ggml_backend_t backend, struct ggml_cgraph * cgraph, struct ggml_backend_graph_optimize_params * params);
++ void (GGML_CALL *graph_optimize) (ggml_backend_t backend, struct ggml_cgraph * cgraph, struct ggml_backend_graph_optimize_params * params);
};
struct ggml_backend {
-@@ -159,46 +163,46 @@ extern "C" {
+@@ -175,46 +179,46 @@ extern "C" {
// the current functions to obtain the properties can remain, since they are more convenient for often used properties
struct ggml_backend_device_i {
// device name: short identifier for this device, such as "CPU" or "CUDA0"
@@ -187,7 +193,7 @@ diff --git a/ggml/src/ggml-backend-impl.h b/ggml/src/ggml-backend-impl.h
};
struct ggml_backend_device {
-@@ -212,15 +216,15 @@ extern "C" {
+@@ -228,15 +232,15 @@ extern "C" {
//
struct ggml_backend_reg_i {
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-backend-meta.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-backend-meta.cpp.patch
index 85503cea..df78814f 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml-backend-meta.cpp.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml-backend-meta.cpp.patch
@@ -141,7 +141,7 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cpp
GGML_ASSERT(ggml_backend_buffer_is_meta(buffer));
ggml_backend_meta_buffer_context * buf_ctx = (ggml_backend_meta_buffer_context *) buffer->context;
delete buf_ctx;
-@@ -1123,7 +1123,7 @@ static struct ggml_backend_meta_split_state ggml_backend_meta_get_split_state(co
+@@ -1172,7 +1172,7 @@ static struct ggml_backend_meta_split_state ggml_backend_meta_get_split_state(co
return ggml_backend_meta_get_split_state(buf_ctx->get_simple_tensor_container(tensor), tensor, assume_sync);
}
@@ -150,7 +150,7 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cpp
GGML_UNUSED(buffer);
return (void *) 0x1000000000000000; // FIXME
}
-@@ -1248,14 +1248,14 @@ static enum ggml_status ggml_backend_meta_buffer_init_tensor_impl(ggml_backend_m
+@@ -1304,14 +1304,14 @@ static enum ggml_status ggml_backend_meta_buffer_init_tensor_impl(ggml_backend_m
return GGML_STATUS_SUCCESS;
}
@@ -162,12 +162,21 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cpp
return ggml_backend_meta_buffer_init_tensor_impl(buf_ctx->get_simple_tensor_container(tensor), tensor);
}
+-static void ggml_backend_meta_buffer_memset_tensor(
++static void GGML_CALL ggml_backend_meta_buffer_memset_tensor(
+ ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size) {
+ const size_t n_bufs = ggml_backend_meta_buffer_n_bufs(buffer);
+ const ggml_backend_meta_split_state split_state =
+@@ -1413,7 +1413,7 @@ static void ggml_backend_meta_buffer_memset_tensor(
+ }
+ }
+
-static void ggml_backend_meta_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) {
+static void GGML_CALL ggml_backend_meta_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) {
const size_t n_bufs = ggml_backend_meta_buffer_n_bufs(buffer);
const ggml_backend_meta_split_state split_state = ggml_backend_meta_get_split_state(tensor, /*assume_sync =*/ false);
GGML_ASSERT(ggml_is_contiguous(tensor) || split_state.axis == GGML_BACKEND_SPLIT_AXIS_MIRRORED);
-@@ -1369,7 +1369,7 @@ static void ggml_backend_meta_buffer_set_tensor(ggml_backend_buffer_t buffer, gg
+@@ -1541,7 +1541,7 @@ static void ggml_backend_meta_buffer_set_tensor(ggml_backend_buffer_t buffer, gg
}
}
@@ -176,7 +185,7 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cpp
const size_t n_bufs = ggml_backend_meta_buffer_n_bufs(buffer);
const ggml_backend_meta_split_state split_state = ggml_backend_meta_get_split_state(tensor, /*assume_sync =*/ false);
GGML_ASSERT(ggml_is_contiguous(tensor) || split_state.axis == GGML_BACKEND_SPLIT_AXIS_MIRRORED);
-@@ -1469,14 +1469,14 @@ static void ggml_backend_meta_buffer_get_tensor(ggml_backend_buffer_t buffer, co
+@@ -1641,14 +1641,14 @@ static void ggml_backend_meta_buffer_get_tensor(ggml_backend_buffer_t buffer, co
}
}
@@ -193,8 +202,8 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cpp
GGML_ASSERT(ggml_backend_buffer_is_meta(buffer));
ggml_backend_meta_buffer_context * buf_ctx = (ggml_backend_meta_buffer_context *) buffer->context;
for (size_t i = 0; i < buf_ctx->bufs.size(); i++) {
-@@ -1502,7 +1502,7 @@ bool ggml_backend_buffer_is_meta(ggml_backend_buffer_t buf) {
- return buf != nullptr && buf->iface.free_buffer == ggml_backend_meta_buffer_iface.free_buffer;
+@@ -1684,7 +1684,7 @@ void ggml_backend_meta_buffer_set_usage(ggml_backend_buffer_t buffer, enum ggml_
+ }
}
-static ggml_backend_buffer_t ggml_backend_meta_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) {
@@ -202,7 +211,7 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cpp
const size_t n_simple_bufts = ggml_backend_meta_buft_n_bufts(buft);
const ggml_init_params params = {
-@@ -1669,20 +1669,20 @@ struct ggml_backend_meta_context {
+@@ -1851,20 +1851,20 @@ struct ggml_backend_meta_context {
}
};
@@ -226,7 +235,7 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cpp
const size_t n_backends = ggml_backend_meta_n_backends(backend);
GGML_ASSERT(offset == 0);
GGML_ASSERT(ggml_is_contiguous(tensor));
-@@ -1727,7 +1727,7 @@ static void ggml_backend_meta_set_tensor_async(ggml_backend_t backend, ggml_tens
+@@ -1909,7 +1909,7 @@ static void ggml_backend_meta_set_tensor_async(ggml_backend_t backend, ggml_tens
}
}
@@ -235,7 +244,7 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cpp
const size_t n_backends = ggml_backend_meta_n_backends(backend);
GGML_ASSERT(offset == 0);
GGML_ASSERT(ggml_is_contiguous(tensor));
-@@ -1772,14 +1772,14 @@ static void ggml_backend_meta_get_tensor_async(ggml_backend_t backend, const ggm
+@@ -1954,14 +1954,14 @@ static void ggml_backend_meta_get_tensor_async(ggml_backend_t backend, const ggm
}
}
@@ -252,7 +261,7 @@ diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cpp
GGML_ASSERT(cgraph->grads == nullptr);
const size_t n_backends = ggml_backend_meta_n_backends(backend);
ggml_backend_meta_context * backend_ctx = (ggml_backend_meta_context *) backend->context;
-@@ -2249,7 +2249,7 @@ bool ggml_backend_is_meta(ggml_backend_t backend) {
+@@ -2487,7 +2487,7 @@ bool ggml_backend_is_meta(ggml_backend_t backend) {
return backend != nullptr && backend->iface.get_name == ggml_backend_meta_i.get_name;
}
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-backend-reg.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-backend-reg.cpp.patch
index 6286b24e..63af2d06 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml-backend-reg.cpp.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml-backend-reg.cpp.patch
@@ -1,7 +1,7 @@
diff --git a/ggml/src/ggml-backend-reg.cpp b/ggml/src/ggml-backend-reg.cpp
--- a/llama.cpp/ggml/src/ggml-backend-reg.cpp
+++ b/llama.cpp/ggml/src/ggml-backend-reg.cpp
-@@ -504,7 +504,7 @@ static ggml_backend_reg_t ggml_backend_load_best(const char * name, bool silent,
+@@ -510,7 +510,7 @@ static ggml_backend_reg_t ggml_backend_load_best(const char * name, bool silent,
if (ec) {
GGML_LOG_DEBUG("%s: posix_stat(%s) failure, error-message: %s\n", __func__, path_str(search_path).c_str(), ec.message().c_str());
} else {
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-backend.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-backend.cpp.patch
index 5b1f9bfa..4337f9fa 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml-backend.cpp.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml-backend.cpp.patch
@@ -1,7 +1,7 @@
diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp
--- a/llama.cpp/ggml/src/ggml-backend.cpp
+++ b/llama.cpp/ggml/src/ggml-backend.cpp
-@@ -112,7 +112,14 @@ void ggml_backend_buffer_free(ggml_backend_buffer_t buffer) {
+@@ -121,7 +121,14 @@ void ggml_backend_buffer_free(ggml_backend_buffer_t buffer) {
if (buffer->iface.free_buffer != NULL) {
buffer->iface.free_buffer(buffer);
}
@@ -17,7 +17,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp
}
size_t ggml_backend_buffer_get_size(ggml_backend_buffer_t buffer) {
-@@ -671,7 +678,7 @@ struct ggml_backend_multi_buffer_context {
+@@ -682,7 +689,7 @@ struct ggml_backend_multi_buffer_context {
size_t n_buffers;
};
@@ -26,7 +26,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp
GGML_ASSERT(buffer);
ggml_backend_multi_buffer_context * ctx = (ggml_backend_multi_buffer_context *) buffer->context;
for (size_t i = 0; i < ctx->n_buffers; i++) {
-@@ -682,7 +689,7 @@ static void ggml_backend_multi_buffer_free_buffer(ggml_backend_buffer_t buffer)
+@@ -693,7 +700,7 @@ static void ggml_backend_multi_buffer_free_buffer(ggml_backend_buffer_t buffer)
free(ctx);
}
@@ -35,7 +35,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp
GGML_ASSERT(buffer);
ggml_backend_multi_buffer_context * ctx = (ggml_backend_multi_buffer_context *) buffer->context;
for (size_t i = 0; i < ctx->n_buffers; i++) {
-@@ -702,6 +709,7 @@ static const struct ggml_backend_buffer_i ggml_backend_multi_buffer_i = {
+@@ -713,6 +720,7 @@ static const struct ggml_backend_buffer_i ggml_backend_multi_buffer_i = {
/* .cpy_tensor = */ NULL,
/* .clear = */ ggml_backend_multi_buffer_clear,
/* .reset = */ NULL,
@@ -43,7 +43,41 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp
};
ggml_backend_buffer_t ggml_backend_multi_buffer_alloc_buffer(ggml_backend_buffer_t * buffers, size_t n_buffers) {
-@@ -2270,7 +2278,7 @@ bool ggml_backend_compare_graph_backend(ggml_backend_t backend1, ggml_backend_t
+@@ -1063,6 +1071,18 @@ static void ggml_backend_sched_set_if_supported(ggml_backend_sched_t sched, stru
+ }
+
+ // assigns backends to ops and splits the graph into subgraphs that can be computed on the same backend
++// [jart] The backends invoke this through ggml_backend_graph_optimize_params,
++// and they live in dynamically loaded modules built with the host ABI, so the
++// callback has to be GGML_CALL. A lambda cannot carry the attribute, so this is
++// a named function rather than the lambda upstream uses.
++static void GGML_CALL ggml_backend_sched_add_alloc_dep(void * user_data, ggml_tensor * tensor, ggml_tensor * until) {
++ auto & deps = *(std::unordered_map<ggml_tensor *, std::vector<ggml_tensor *>> *) user_data;
++ std::vector<ggml_tensor *> & keep = deps[until];
++ if (std::find(keep.begin(), keep.end(), tensor) == keep.end()) {
++ keep.push_back(tensor);
++ }
++}
++
+ void ggml_backend_sched_split_graph(ggml_backend_sched_t sched, struct ggml_cgraph * graph) {
+ // reset splits
+ sched->n_splits = 0;
+@@ -1445,13 +1465,7 @@ void ggml_backend_sched_split_graph(ggml_backend_sched_t sched, struct ggml_cgra
+ std::unordered_map<ggml_tensor *, std::vector<ggml_tensor *>> alloc_deps;
+
+ struct ggml_backend_graph_optimize_params opt_params = {
+- /* .add_alloc_dep = */ [](void * user_data, ggml_tensor * tensor, ggml_tensor * until) {
+- auto & deps = *(std::unordered_map<ggml_tensor *, std::vector<ggml_tensor *>> *) user_data;
+- std::vector<ggml_tensor *> & keep = deps[until];
+- if (std::find(keep.begin(), keep.end(), tensor) == keep.end()) {
+- keep.push_back(tensor);
+- }
+- },
++ /* .add_alloc_dep = */ ggml_backend_sched_add_alloc_dep,
+ /* .user_data = */ &alloc_deps,
+ };
+
+@@ -2352,7 +2366,7 @@ bool ggml_backend_compare_graph_backend(ggml_backend_t backend1, ggml_backend_t
// CPU backend - buffer
@@ -52,7 +86,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp
GGML_ASSERT(buffer);
uintptr_t data = (uintptr_t)buffer->context;
-@@ -2282,33 +2290,33 @@ static void * ggml_backend_cpu_buffer_get_base(ggml_backend_buffer_t buffer) {
+@@ -2364,33 +2378,33 @@ static void * ggml_backend_cpu_buffer_get_base(ggml_backend_buffer_t buffer) {
return (void *)data;
}
@@ -91,7 +125,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp
GGML_ASSERT(src);
if (ggml_backend_buffer_is_host(src->buffer)) {
memcpy(dst->data, src->data, ggml_nbytes(src));
-@@ -2319,7 +2327,7 @@ static bool ggml_backend_cpu_buffer_cpy_tensor(ggml_backend_buffer_t buffer, con
+@@ -2401,7 +2415,7 @@ static bool ggml_backend_cpu_buffer_cpy_tensor(ggml_backend_buffer_t buffer, con
GGML_UNUSED(buffer);
}
@@ -100,7 +134,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp
GGML_ASSERT(buffer);
memset(buffer->context, value, buffer->size);
}
-@@ -2336,6 +2344,7 @@ static const struct ggml_backend_buffer_i ggml_backend_cpu_buffer_i = {
+@@ -2418,6 +2432,7 @@ static const struct ggml_backend_buffer_i ggml_backend_cpu_buffer_i = {
/* .cpy_tensor = */ ggml_backend_cpu_buffer_cpy_tensor,
/* .clear = */ ggml_backend_cpu_buffer_clear,
/* .reset = */ NULL,
@@ -108,7 +142,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp
};
static const struct ggml_backend_buffer_i ggml_backend_cpu_buffer_from_ptr_i = {
-@@ -2350,19 +2359,20 @@ static const struct ggml_backend_buffer_i ggml_backend_cpu_buffer_from_ptr_i = {
+@@ -2432,19 +2447,20 @@ static const struct ggml_backend_buffer_i ggml_backend_cpu_buffer_from_ptr_i = {
/* .cpy_tensor = */ ggml_backend_cpu_buffer_cpy_tensor,
/* .clear = */ ggml_backend_cpu_buffer_clear,
/* .reset = */ NULL,
@@ -131,7 +165,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp
void * data = ggml_aligned_malloc(size);
if (data == NULL) {
-@@ -2373,13 +2383,13 @@ static ggml_backend_buffer_t ggml_backend_cpu_buffer_type_alloc_buffer(ggml_back
+@@ -2455,13 +2471,13 @@ static ggml_backend_buffer_t ggml_backend_cpu_buffer_type_alloc_buffer(ggml_back
return ggml_backend_buffer_init(buft, ggml_backend_cpu_buffer_i, data, size);
}
@@ -147,7 +181,7 @@ diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp
return true;
GGML_UNUSED(buft);
-@@ -2402,7 +2412,7 @@ ggml_backend_buffer_type_t ggml_backend_cpu_buffer_type(void) {
+@@ -2484,7 +2500,7 @@ ggml_backend_buffer_type_t ggml_backend_cpu_buffer_type(void) {
return &ggml_backend_cpu_buffer_type;
}
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.c.patch b/llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.c.patch
index 370aedc3..d388f878 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.c.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.c.patch
@@ -1,7 +1,7 @@
diff --git a/ggml/src/ggml-cpu/ggml-cpu.c b/ggml/src/ggml-cpu/ggml-cpu.c
--- a/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c
+++ b/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c
-@@ -1539,6 +1539,26 @@ static void ggml_compute_forward_mul_mat_id(
+@@ -1555,6 +1555,26 @@ static void ggml_compute_forward_mul_mat_id(
const struct ggml_tensor * src1 = dst->src[1];
const struct ggml_tensor * ids = dst->src[2];
@@ -28,7 +28,7 @@ diff --git a/ggml/src/ggml-cpu/ggml-cpu.c b/ggml/src/ggml-cpu/ggml-cpu.c
GGML_TENSOR_BINARY_OP_LOCALS
const int ith = params->ith;
-@@ -1658,6 +1678,24 @@ static void ggml_compute_forward_mul_mat_id(
+@@ -1691,6 +1711,24 @@ static void ggml_compute_forward_mul_mat_id(
const int64_t nr0 = ne01;
const int64_t nr1 = cne1;
@@ -53,10 +53,10 @@ diff --git a/ggml/src/ggml-cpu/ggml-cpu.c b/ggml/src/ggml-cpu/ggml-cpu.c
int chunk_size = 16;
if (nr0 == 1 || nr1 == 1) {
chunk_size = 64;
-@@ -2876,6 +2914,19 @@ struct ggml_cplan ggml_graph_plan(
- cur += n_as*ids->ne[0]*ids->ne[1]*sizeof(struct mmid_row_mapping) + sizeof(int64_t);
- // atomic_current_chunk
- cur += CACHE_LINE_SIZE*n_as + CACHE_LINE_SIZE;
+@@ -2919,6 +2957,19 @@ struct ggml_cplan ggml_graph_plan(
+ if (ggml_cpu_iqp_supports_mul_mat_id(node)) {
+ cur += n_tasks * ggml_cpu_iqp_scratch_size(node) + 64;
+ }
+#if GGML_USE_LLAMAFILE
+ // [jart] reserve work-buffer space for llamafile_mixmul.
+ // Without this, MixMul::allocate_shared_memory() in
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.cpp.patch
index 4993e680..fc7397ea 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.cpp.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.cpp.patch
@@ -151,7 +151,7 @@ diff --git a/ggml/src/ggml-cpu/ggml-cpu.cpp b/ggml/src/ggml-cpu/ggml-cpu.cpp
const struct ggml_tensor * src0 = op->src[0];
const struct ggml_tensor * src1 = op->src[1];
-@@ -479,7 +479,7 @@ static bool ggml_backend_cpu_device_supports_op(ggml_backend_dev_t dev, const st
+@@ -483,7 +483,7 @@ static bool ggml_backend_cpu_device_supports_op(ggml_backend_dev_t dev, const st
}
}
@@ -160,7 +160,7 @@ diff --git a/ggml/src/ggml-cpu/ggml-cpu.cpp b/ggml/src/ggml-cpu/ggml-cpu.cpp
return ggml_backend_buft_is_host(buft) || ggml_backend_cpu_is_extra_buffer_type(buft);
GGML_UNUSED(dev);
}
-@@ -504,19 +504,19 @@ static const struct ggml_backend_device_i ggml_backend_cpu_device_i = {
+@@ -508,19 +508,19 @@ static const struct ggml_backend_device_i ggml_backend_cpu_device_i = {
// CPU backend - backend (reg)
@@ -183,7 +183,7 @@ diff --git a/ggml/src/ggml-cpu/ggml-cpu.cpp b/ggml/src/ggml-cpu/ggml-cpu.cpp
GGML_ASSERT(index == 0);
static ggml_backend_cpu_device_context ctx;
-@@ -531,7 +531,7 @@ static ggml_backend_dev_t ggml_backend_cpu_reg_get_device(ggml_backend_reg_t reg
+@@ -535,7 +535,7 @@ static ggml_backend_dev_t ggml_backend_cpu_reg_get_device(ggml_backend_reg_t reg
// This is intended to replace the the ggml_cpu_has_* functions when loading the CPU backend dynamically,
// and additionally to allow other backends to expose their own list of features that applications can query using the same API
@@ -192,7 +192,7 @@ diff --git a/ggml/src/ggml-cpu/ggml-cpu.cpp b/ggml/src/ggml-cpu/ggml-cpu.cpp
static std::vector<ggml_backend_feature> features = []() {
ggml_cpu_init();
-@@ -648,7 +648,7 @@ static ggml_backend_feature * ggml_backend_cpu_get_features(ggml_backend_reg_t r
+@@ -652,7 +652,7 @@ static ggml_backend_feature * ggml_backend_cpu_get_features(ggml_backend_reg_t r
GGML_UNUSED(reg);
}
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cpu_ops.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-cpu_ops.cpp.patch
index 2d5db121..b9208666 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml-cpu_ops.cpp.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml-cpu_ops.cpp.patch
@@ -20,7 +20,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cpp
// ggml_compute_forward_dup
static void ggml_compute_forward_dup_same_cont(
-@@ -8547,6 +8559,15 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(
+@@ -8693,6 +8705,15 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(
GGML_ASSERT(( q_to_vec_dot) && "fattn: unsupported K-type");
GGML_ASSERT((v->type == GGML_TYPE_F32 || v_to_float ) && "fattn: unsupported V-type");
@@ -36,7 +36,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cpp
int ith = params->ith;
for (int ir = ir0; ir < ir1; ++ir) {
-@@ -8566,10 +8587,22 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(
+@@ -8712,10 +8733,22 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(
ggml_fp16_t * VKQ16 = (ggml_fp16_t *) (VKQ32 + 1*DV); // (temporary) FP16 VKQ accumulator
ggml_fp16_t * Q_q = (ggml_fp16_t *) (VKQ32 + 2*DV); // (temporary) buffer for Q converted to quantized/FP16
@@ -62,7 +62,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cpp
}
const ggml_fp16_t * mp = mask ? (ggml_fp16_t *)((char *) mask->data + iq1*mask->nb[1] + (iq2%mask->ne[2])*mask->nb[2] + (iq3%mask->ne[3])*mask->nb[3]) : NULL;
-@@ -8598,7 +8631,13 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(
+@@ -8744,7 +8777,13 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(
float s; // KQ value
const char * k_data = (const char *) k->data + ( ic*nbk1 + ik2*nbk2 + ik3*nbk3);
@@ -76,7 +76,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cpp
s = s*scale; // scale KQ value
-@@ -8615,21 +8654,36 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(
+@@ -8761,21 +8800,36 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(
const char * v_data = ((const char *) v->data + (ic*nbv1 + iv2*nbv2 + iv3*nbv3));
@@ -116,7 +116,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cpp
} else {
if (s > M) {
// s is new maximum, ms < 1.0f, vs == expf(s - s) == 1.0f
-@@ -8637,26 +8691,20 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(
+@@ -8783,26 +8837,20 @@ static void ggml_compute_forward_flash_attn_ext_f16_one_chunk(
ms = expf(Mold - M);
// V = V*expf(Mold - M)
@@ -146,7 +146,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cpp
for (int64_t d = 0; d < DV; ++d) {
VKQ32[d] = GGML_CPU_FP16_TO_FP32(VKQ16[d]);
}
-@@ -8877,8 +8925,24 @@ static void ggml_compute_forward_flash_attn_ext_tiled(
+@@ -9023,8 +9071,24 @@ static void ggml_compute_forward_flash_attn_ext_tiled(
const char * k_data = (const char *)k->data + (ic + tk)*nbk1 + ik2*nbk2 + ik3*nbk3;
if (kv_type == GGML_TYPE_F16) {
const ggml_fp16_t * k_f16 = (const ggml_fp16_t *)k_data;
@@ -173,7 +173,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cpp
}
} else {
const float * k_f32_src = (const float *)k_data;
-@@ -8888,7 +8952,18 @@ static void ggml_compute_forward_flash_attn_ext_tiled(
+@@ -9034,7 +9098,18 @@ static void ggml_compute_forward_flash_attn_ext_tiled(
}
}
memset(KQ, 0, Q_TILE_SZ * KV_TILE_SZ * sizeof(float));
@@ -192,7 +192,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cpp
ggml_vec_scale_f32(Q_TILE_SZ * KV_TILE_SZ, KQ, scale);
// Set padded KQ entries to -inf so softmax gives them zero weight
-@@ -8941,7 +9016,15 @@ static void ggml_compute_forward_flash_attn_ext_tiled(
+@@ -9087,7 +9162,15 @@ static void ggml_compute_forward_flash_attn_ext_tiled(
for (int tk = 0; tk < kv_tile; tk++) {
const char * v_data = (const char *)v->data + (ic + tk)*nbv1 + iv2*nbv2 + iv3*nbv3;
if (kv_type == GGML_TYPE_F16) {
@@ -208,7 +208,7 @@ diff --git a/ggml/src/ggml-cpu/ops.cpp b/ggml/src/ggml-cpu/ops.cpp
} else {
memcpy(V32 + tk * DV, v_data, DV * sizeof(float));
}
-@@ -8951,7 +9034,16 @@ static void ggml_compute_forward_flash_attn_ext_tiled(
+@@ -9097,7 +9180,16 @@ static void ggml_compute_forward_flash_attn_ext_tiled(
memset(KQ + tq * KV_TILE_SZ, 0, KV_TILE_SZ * sizeof(float));
}
}
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cpu_repack.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-cpu_repack.cpp.patch
index ae8d8d00..c067ae03 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml-cpu_repack.cpp.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml-cpu_repack.cpp.patch
@@ -1,7 +1,7 @@
diff --git a/ggml/src/ggml-cpu/repack.cpp b/ggml/src/ggml-cpu/repack.cpp
--- a/llama.cpp/ggml/src/ggml-cpu/repack.cpp
+++ b/llama.cpp/ggml/src/ggml-cpu/repack.cpp
-@@ -4723,14 +4723,14 @@ static const ggml::cpu::tensor_traits * ggml_repack_get_optimal_repack_type(cons
+@@ -5140,14 +5140,14 @@ static const ggml::cpu::tensor_traits * ggml_repack_get_optimal_repack_type(cons
return nullptr;
}
@@ -18,7 +18,7 @@ diff --git a/ggml/src/ggml-cpu/repack.cpp b/ggml/src/ggml-cpu/repack.cpp
const void * data, size_t offset, size_t size) {
GGML_ASSERT(offset == 0);
GGML_ASSERT(size == ggml_nbytes(tensor));
-@@ -4742,13 +4742,13 @@ static void ggml_backend_cpu_repack_buffer_set_tensor(ggml_backend_buffer_t buff
+@@ -5159,13 +5159,13 @@ static void ggml_backend_cpu_repack_buffer_set_tensor(ggml_backend_buffer_t buff
GGML_UNUSED(buffer);
}
@@ -34,7 +34,7 @@ diff --git a/ggml/src/ggml-cpu/repack.cpp b/ggml/src/ggml-cpu/repack.cpp
ggml_backend_buffer_t buffer = ggml_backend_buft_alloc_buffer(ggml_backend_cpu_buffer_type(), size);
if (buffer == nullptr) {
-@@ -4763,7 +4763,7 @@ static ggml_backend_buffer_t ggml_backend_cpu_repack_buffer_type_alloc_buffer(gg
+@@ -5180,7 +5180,7 @@ static ggml_backend_buffer_t ggml_backend_cpu_repack_buffer_type_alloc_buffer(gg
return buffer;
}
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cuda_common.cuh.patch b/llama.cpp.patches/patches/ggml_src_ggml-cuda_common.cuh.patch
index c80a874f..8a2e8a33 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml-cuda_common.cuh.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml-cuda_common.cuh.patch
@@ -1,7 +1,7 @@
diff --git a/ggml/src/ggml-cuda/common.cuh b/ggml/src/ggml-cuda/common.cuh
--- a/llama.cpp/ggml/src/ggml-cuda/common.cuh
+++ b/llama.cpp/ggml/src/ggml-cuda/common.cuh
-@@ -320,9 +320,15 @@ static bool fp16_mma_hardware_available(const int cc) {
+@@ -336,9 +336,15 @@ static bool fast_bf16_hardware_available(const int cc) {
}
static bool bf16_mma_hardware_available(const int cc) {
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cuda_convert.cu.patch b/llama.cpp.patches/patches/ggml_src_ggml-cuda_convert.cu.patch
index c945f000..0a13aeef 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml-cuda_convert.cu.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml-cuda_convert.cu.patch
@@ -1,7 +1,7 @@
diff --git a/ggml/src/ggml-cuda/convert.cu b/ggml/src/ggml-cuda/convert.cu
--- a/llama.cpp/ggml/src/ggml-cuda/convert.cu
+++ b/llama.cpp/ggml/src/ggml-cuda/convert.cu
-@@ -481,6 +481,7 @@ to_bf16_cuda_t ggml_get_to_bf16_cuda(ggml_type type) {
+@@ -513,6 +513,7 @@ to_bf16_cuda_t ggml_get_to_bf16_cuda(ggml_type type) {
return dequantize_row_q5_K_cuda;
case GGML_TYPE_Q6_K:
return dequantize_row_q6_K_cuda;
@@ -9,7 +9,7 @@ diff --git a/ggml/src/ggml-cuda/convert.cu b/ggml/src/ggml-cuda/convert.cu
case GGML_TYPE_IQ2_XXS:
return dequantize_row_iq2_xxs_cuda;
case GGML_TYPE_IQ2_XS:
-@@ -499,6 +500,7 @@ to_bf16_cuda_t ggml_get_to_bf16_cuda(ggml_type type) {
+@@ -531,6 +532,7 @@ to_bf16_cuda_t ggml_get_to_bf16_cuda(ggml_type type) {
return dequantize_row_iq4_xs_cuda;
case GGML_TYPE_IQ3_S:
return dequantize_row_iq3_s_cuda;
@@ -17,7 +17,7 @@ diff --git a/ggml/src/ggml-cuda/convert.cu b/ggml/src/ggml-cuda/convert.cu
case GGML_TYPE_MXFP4:
return dequantize_row_mxfp4_cuda;
case GGML_TYPE_NVFP4:
-@@ -541,6 +543,7 @@ to_fp16_cuda_t ggml_get_to_fp16_cuda(ggml_type type) {
+@@ -573,6 +575,7 @@ to_fp16_cuda_t ggml_get_to_fp16_cuda(ggml_type type) {
return dequantize_row_q5_K_cuda;
case GGML_TYPE_Q6_K:
return dequantize_row_q6_K_cuda;
@@ -25,7 +25,7 @@ diff --git a/ggml/src/ggml-cuda/convert.cu b/ggml/src/ggml-cuda/convert.cu
case GGML_TYPE_IQ2_XXS:
return dequantize_row_iq2_xxs_cuda;
case GGML_TYPE_IQ2_XS:
-@@ -559,6 +562,7 @@ to_fp16_cuda_t ggml_get_to_fp16_cuda(ggml_type type) {
+@@ -591,6 +594,7 @@ to_fp16_cuda_t ggml_get_to_fp16_cuda(ggml_type type) {
return dequantize_row_iq4_xs_cuda;
case GGML_TYPE_IQ3_S:
return dequantize_row_iq3_s_cuda;
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cuda_ggml-cuda.cu.patch b/llama.cpp.patches/patches/ggml_src_ggml-cuda_ggml-cuda.cu.patch
index acef87f6..72ec44fb 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml-cuda_ggml-cuda.cu.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml-cuda_ggml-cuda.cu.patch
@@ -1,7 +1,7 @@
diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
--- a/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu
+++ b/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu
-@@ -736,7 +736,7 @@ struct ggml_backend_cuda_buffer_context {
+@@ -738,7 +738,7 @@ struct ggml_backend_cuda_buffer_context {
}
};
@@ -10,7 +10,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context;
delete ctx;
}
-@@ -745,12 +745,12 @@ static bool ggml_backend_buffer_is_cuda(ggml_backend_buffer_t buffer) {
+@@ -747,12 +747,12 @@ static bool ggml_backend_buffer_is_cuda(ggml_backend_buffer_t buffer) {
return buffer->iface.free_buffer == ggml_backend_cuda_buffer_free_buffer;
}
@@ -25,7 +25,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context;
if (tensor->view_src != NULL) {
-@@ -771,7 +771,7 @@ static enum ggml_status ggml_backend_cuda_buffer_init_tensor(ggml_backend_buffer
+@@ -773,7 +773,7 @@ static enum ggml_status ggml_backend_cuda_buffer_init_tensor(ggml_backend_buffer
return GGML_STATUS_SUCCESS;
}
@@ -34,7 +34,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context;
ggml_cuda_set_device(ctx->device);
-@@ -779,7 +779,7 @@ static void ggml_backend_cuda_buffer_memset_tensor(ggml_backend_buffer_t buffer,
+@@ -781,7 +781,7 @@ static void ggml_backend_cuda_buffer_memset_tensor(ggml_backend_buffer_t buffer,
CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));
}
@@ -43,7 +43,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context;
ggml_cuda_set_device(ctx->device);
-@@ -787,7 +787,7 @@ static void ggml_backend_cuda_buffer_set_tensor(ggml_backend_buffer_t buffer, gg
+@@ -789,7 +789,7 @@ static void ggml_backend_cuda_buffer_set_tensor(ggml_backend_buffer_t buffer, gg
CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));
}
@@ -52,7 +52,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context;
ggml_cuda_set_device(ctx->device);
-@@ -795,7 +795,7 @@ static void ggml_backend_cuda_buffer_get_tensor(ggml_backend_buffer_t buffer, co
+@@ -797,7 +797,7 @@ static void ggml_backend_cuda_buffer_get_tensor(ggml_backend_buffer_t buffer, co
CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));
}
@@ -61,7 +61,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) {
ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context;
-@@ -805,7 +805,7 @@ static void ggml_backend_cuda_buffer_set_tensor_2d(ggml_backend_buffer_t buffer,
+@@ -807,7 +807,7 @@ static void ggml_backend_cuda_buffer_set_tensor_2d(ggml_backend_buffer_t buffer,
CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));
}
@@ -70,7 +70,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) {
ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context;
-@@ -815,7 +815,7 @@ static void ggml_backend_cuda_buffer_get_tensor_2d(ggml_backend_buffer_t buffer,
+@@ -817,7 +817,7 @@ static void ggml_backend_cuda_buffer_get_tensor_2d(ggml_backend_buffer_t buffer,
CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));
}
@@ -79,7 +79,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
if (ggml_backend_buffer_is_cuda(src->buffer)) {
ggml_backend_cuda_buffer_context * src_ctx = (ggml_backend_cuda_buffer_context *)src->buffer->context;
ggml_backend_cuda_buffer_context * dst_ctx = (ggml_backend_cuda_buffer_context *)dst->buffer->context;
-@@ -840,7 +840,7 @@ static bool ggml_backend_cuda_buffer_cpy_tensor(ggml_backend_buffer_t buffer, co
+@@ -842,7 +842,7 @@ static bool ggml_backend_cuda_buffer_cpy_tensor(ggml_backend_buffer_t buffer, co
GGML_UNUSED(buffer);
}
@@ -88,7 +88,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context;
ggml_cuda_set_device(ctx->device);
-@@ -848,6 +848,14 @@ static void ggml_backend_cuda_buffer_clear(ggml_backend_buffer_t buffer, uint8_t
+@@ -850,6 +850,14 @@ static void ggml_backend_cuda_buffer_clear(ggml_backend_buffer_t buffer, uint8_t
CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));
}
@@ -103,7 +103,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
static const ggml_backend_buffer_i ggml_backend_cuda_buffer_interface = {
/* .free_buffer = */ ggml_backend_cuda_buffer_free_buffer,
/* .get_base = */ ggml_backend_cuda_buffer_get_base,
-@@ -860,6 +868,7 @@ static const ggml_backend_buffer_i ggml_backend_cuda_buffer_interface = {
+@@ -862,6 +870,7 @@ static const ggml_backend_buffer_i ggml_backend_cuda_buffer_interface = {
/* .cpy_tensor = */ ggml_backend_cuda_buffer_cpy_tensor,
/* .clear = */ ggml_backend_cuda_buffer_clear,
/* .reset = */ NULL,
@@ -111,7 +111,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
};
// cuda buffer type
-@@ -868,7 +877,7 @@ struct ggml_backend_cuda_buffer_type_context {
+@@ -870,7 +879,7 @@ struct ggml_backend_cuda_buffer_type_context {
std::string name;
};
@@ -120,7 +120,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_buffer_type_context * ctx = (ggml_backend_cuda_buffer_type_context *)buft->context;
return ctx->name.c_str();
-@@ -878,7 +887,7 @@ static bool ggml_backend_buft_is_cuda(ggml_backend_buffer_type_t buft) {
+@@ -880,7 +889,7 @@ static bool ggml_backend_buft_is_cuda(ggml_backend_buffer_type_t buft) {
return buft->iface.get_name == ggml_backend_cuda_buffer_type_get_name;
}
@@ -129,7 +129,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_buffer_type_context * buft_ctx = (ggml_backend_cuda_buffer_type_context *)buft->context;
ggml_cuda_set_device(buft_ctx->device);
-@@ -897,13 +906,13 @@ static ggml_backend_buffer_t ggml_backend_cuda_buffer_type_alloc_buffer(ggml_bac
+@@ -899,13 +908,13 @@ static ggml_backend_buffer_t ggml_backend_cuda_buffer_type_alloc_buffer(ggml_bac
return ggml_backend_buffer_init(buft, ggml_backend_cuda_buffer_interface, ctx, size);
}
@@ -145,7 +145,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_buffer_type_context * buft_ctx = (ggml_backend_cuda_buffer_type_context *) buft->context;
size_t size = tensor->op == GGML_OP_FLASH_ATTN_EXT
-@@ -1256,7 +1265,7 @@ static bool ggml_backend_cuda_comm_allreduce_tensor(void * comm_ctx_v, struct gg
+@@ -1259,7 +1268,7 @@ static bool ggml_backend_cuda_comm_allreduce_tensor(void * comm_ctx_v, struct gg
// host buffer type
@@ -154,7 +154,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
return GGML_CUDA_NAME "_Host";
GGML_UNUSED(buft);
-@@ -1266,7 +1275,7 @@ static bool ggml_backend_buft_is_cuda_host(ggml_backend_buffer_type_t buft) {
+@@ -1269,7 +1278,7 @@ static bool ggml_backend_buft_is_cuda_host(ggml_backend_buffer_type_t buft) {
return buft->iface.get_name == ggml_backend_cuda_host_buffer_type_name;
}
@@ -163,7 +163,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
CUDA_CHECK(cudaFreeHost(buffer->context));
}
-@@ -1288,17 +1297,25 @@ static void * ggml_cuda_host_malloc(size_t size) {
+@@ -1291,17 +1300,25 @@ static void * ggml_cuda_host_malloc(size_t size) {
return ptr;
}
@@ -191,7 +191,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
return buffer;
}
-@@ -1644,6 +1661,14 @@ static void ggml_cuda_mul_mat_cublas(ggml_backend_cuda_context & ctx, const ggml
+@@ -1655,6 +1672,14 @@ static void ggml_cuda_mul_mat_cublas(ggml_backend_cuda_context & ctx, const ggml
}
}
@@ -206,7 +206,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
switch (compute_type) {
case GGML_TYPE_F32:
ggml_cuda_mul_mat_cublas_impl<GGML_TYPE_F32>(ctx, src0, src1, dst);
-@@ -2412,20 +2437,20 @@ static bool ggml_cuda_compute_forward(ggml_backend_cuda_context & ctx, struct gg
+@@ -2429,20 +2454,20 @@ static bool ggml_cuda_compute_forward(ggml_backend_cuda_context & ctx, struct gg
// backend
@@ -230,7 +230,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context;
ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer;
-@@ -2434,7 +2459,7 @@ static void ggml_backend_cuda_set_tensor_async(ggml_backend_t backend, ggml_tens
+@@ -2451,7 +2476,7 @@ static void ggml_backend_cuda_set_tensor_async(ggml_backend_t backend, ggml_tens
CUDA_CHECK(cudaMemcpyAsync((char *) tensor->data + offset, data, size, cudaMemcpyHostToDevice, cuda_ctx->stream()));
}
@@ -239,7 +239,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context;
ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer;
-@@ -2443,7 +2468,7 @@ static void ggml_backend_cuda_get_tensor_async(ggml_backend_t backend, const ggm
+@@ -2460,7 +2485,7 @@ static void ggml_backend_cuda_get_tensor_async(ggml_backend_t backend, const ggm
CUDA_CHECK(cudaMemcpyAsync(data, (const char *) tensor->data + offset, size, cudaMemcpyDeviceToHost, cuda_ctx->stream()));
}
@@ -248,7 +248,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) {
ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context;
ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer;
-@@ -2454,7 +2479,7 @@ static void ggml_backend_cuda_set_tensor_2d_async(ggml_backend_t backend, struct
+@@ -2471,7 +2496,7 @@ static void ggml_backend_cuda_set_tensor_2d_async(ggml_backend_t backend, struct
(char *) tensor->data + offset, stride_tensor, data, stride_data, size, n_copies, cudaMemcpyHostToDevice, cuda_ctx->stream()));
}
@@ -257,7 +257,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) {
ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context;
ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer;
-@@ -2465,7 +2490,7 @@ static void ggml_backend_cuda_get_tensor_2d_async(ggml_backend_t backend, const
+@@ -2482,7 +2507,7 @@ static void ggml_backend_cuda_get_tensor_2d_async(ggml_backend_t backend, const
data, stride_data, (const char *) tensor->data + offset, stride_tensor, size, n_copies, cudaMemcpyDeviceToHost, cuda_ctx->stream()));
}
@@ -266,7 +266,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_buffer_t buf_src = src->view_src ? src->view_src->buffer : src->buffer;
ggml_backend_buffer_t buf_dst = dst->view_src ? dst->view_src->buffer : dst->buffer;
-@@ -2524,7 +2549,7 @@ static bool ggml_backend_cuda_cpy_tensor_async(ggml_backend_t backend_src, ggml_
+@@ -2541,7 +2566,7 @@ static bool ggml_backend_cuda_cpy_tensor_async(ggml_backend_t backend_src, ggml_
return true;
}
@@ -275,7 +275,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context;
CUDA_CHECK(cudaStreamSynchronize(cuda_ctx->stream()));
-@@ -4231,7 +4256,7 @@ static bool ggml_cuda_graph_set_enabled(ggml_backend_cuda_context * cuda_ctx, co
+@@ -4417,7 +4442,7 @@ static bool ggml_cuda_graph_set_enabled(ggml_backend_cuda_context * cuda_ctx, co
}
#endif // USE_CUDA_GRAPH
@@ -284,7 +284,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context;
ggml_cuda_set_device(cuda_ctx->device);
-@@ -4290,13 +4315,13 @@ static enum ggml_status ggml_backend_cuda_graph_compute(ggml_backend_t backend,
+@@ -4476,13 +4501,13 @@ static enum ggml_status ggml_backend_cuda_graph_compute(ggml_backend_t backend,
return GGML_STATUS_SUCCESS;
}
@@ -300,16 +300,16 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context;
if (ggml_backend_is_cuda(backend)) {
-@@ -4315,7 +4340,7 @@ static void ggml_backend_cuda_event_wait(ggml_backend_t backend, ggml_backend_ev
+@@ -4501,7 +4526,7 @@ static void ggml_backend_cuda_event_wait(ggml_backend_t backend, ggml_backend_ev
}
}
--static void ggml_backend_cuda_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph) {
-+static void GGML_CALL ggml_backend_cuda_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph) {
+-static void ggml_backend_cuda_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph, ggml_backend_graph_optimize_params * params) {
++static void GGML_CALL ggml_backend_cuda_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph, ggml_backend_graph_optimize_params * params) {
ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context;
- #ifdef USE_CUDA_GRAPH
-@@ -4625,7 +4650,7 @@ void ggml_backend_cuda_get_device_memory(int device, size_t * free, size_t * tot
+ static const bool disable_fusion = getenv("GGML_CUDA_DISABLE_FUSION") != nullptr && std::atoi(getenv("GGML_CUDA_DISABLE_FUSION"));
+@@ -4835,7 +4860,7 @@ void ggml_backend_cuda_get_device_memory(int device, size_t * free, size_t * tot
*total /= share_count;
}
@@ -318,7 +318,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
if (getenv("GGML_CUDA_REGISTER_HOST") == nullptr) {
return false;
}
-@@ -4648,7 +4673,7 @@ bool ggml_backend_cuda_register_host_buffer(void * buffer, size_t size) {
+@@ -4858,7 +4883,7 @@ bool ggml_backend_cuda_register_host_buffer(void * buffer, size_t size) {
#endif // CUDART_VERSION >= 11010 || defined(GGML_USE_MUSA)
}
@@ -327,7 +327,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
if (getenv("GGML_CUDA_REGISTER_HOST") == nullptr) {
return;
}
-@@ -4671,12 +4696,12 @@ struct ggml_backend_cuda_device_context {
+@@ -4881,12 +4906,12 @@ struct ggml_backend_cuda_device_context {
int op_offload_min_batch_size;
};
@@ -342,7 +342,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context;
return ctx->description.c_str();
}
-@@ -4757,7 +4782,7 @@ static bool ggml_backend_cuda_get_available_uma_memory(long * available_memory_k
+@@ -4967,7 +4992,7 @@ static bool ggml_backend_cuda_get_available_uma_memory(long * available_memory_k
}
#endif // defined(__linux__)
@@ -351,7 +351,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context;
ggml_cuda_set_device(ctx->device);
cudaError_t err = cudaMemGetInfo(free, total);
-@@ -4798,7 +4823,7 @@ static void ggml_backend_cuda_device_get_memory(ggml_backend_dev_t dev, size_t *
+@@ -5008,7 +5033,7 @@ static void ggml_backend_cuda_device_get_memory(ggml_backend_dev_t dev, size_t *
*total /= share_count;
}
@@ -360,7 +360,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *) dev->context;
cudaDeviceProp prop;
-@@ -4809,7 +4834,7 @@ static enum ggml_backend_dev_type ggml_backend_cuda_device_get_type(ggml_backend
+@@ -5019,7 +5044,7 @@ static enum ggml_backend_dev_type ggml_backend_cuda_device_get_type(ggml_backend
: GGML_BACKEND_DEVICE_TYPE_GPU;
}
@@ -369,7 +369,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context;
props->name = ggml_backend_cuda_device_get_name(dev);
-@@ -4834,24 +4859,24 @@ static void ggml_backend_cuda_device_get_props(ggml_backend_dev_t dev, ggml_back
+@@ -5044,24 +5069,24 @@ static void ggml_backend_cuda_device_get_props(ggml_backend_dev_t dev, ggml_back
};
}
@@ -398,7 +398,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context;
// check if all the sources are allocated on this device
-@@ -4951,6 +4976,13 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
+@@ -5162,6 +5187,13 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
case GGML_TYPE_Q5_K:
case GGML_TYPE_Q6_K:
case GGML_TYPE_Q8_K:
@@ -412,7 +412,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
case GGML_TYPE_IQ1_M:
case GGML_TYPE_IQ1_S:
case GGML_TYPE_IQ2_S:
-@@ -4960,6 +4992,7 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
+@@ -5171,6 +5203,7 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
case GGML_TYPE_IQ3_XXS:
case GGML_TYPE_IQ4_NL:
case GGML_TYPE_IQ4_XS:
@@ -420,7 +420,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
case GGML_TYPE_BF16:
return true;
default:
-@@ -5069,9 +5102,14 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
+@@ -5280,9 +5313,14 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
if (src0_type == GGML_TYPE_Q5_1 && src1_type == GGML_TYPE_F32) {
return true;
}
@@ -435,7 +435,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_I32) {
return true;
}
-@@ -5303,7 +5341,7 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
+@@ -5519,7 +5557,7 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
}
}
@@ -444,7 +444,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context;
const bool integrated = ggml_cuda_info().devices[dev_ctx->device].integrated;
return (ggml_backend_buft_is_cuda(buft) && buft->device == dev) || (integrated && ggml_backend_buft_is_cuda_host(buft));
-@@ -5324,13 +5362,13 @@ static int64_t get_op_batch_size(const ggml_tensor * op) {
+@@ -5540,13 +5578,13 @@ static int64_t get_op_batch_size(const ggml_tensor * op) {
}
}
@@ -460,7 +460,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
#ifdef GGML_CUDA_NO_PEER_COPY
GGML_UNUSED(dev);
return nullptr;
-@@ -5349,14 +5387,14 @@ static ggml_backend_event_t ggml_backend_cuda_device_event_new(ggml_backend_dev_
+@@ -5565,14 +5603,14 @@ static ggml_backend_event_t ggml_backend_cuda_device_event_new(ggml_backend_dev_
#endif
}
@@ -477,7 +477,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
GGML_UNUSED(dev);
CUDA_CHECK(cudaEventSynchronize((cudaEvent_t)event->context));
}
-@@ -5385,23 +5423,23 @@ struct ggml_backend_cuda_reg_context {
+@@ -5601,23 +5639,23 @@ struct ggml_backend_cuda_reg_context {
std::vector<ggml_backend_dev_t> devices;
};
@@ -505,7 +505,7 @@ diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu
static std::vector<ggml_backend_feature> features = []() {
std::vector<ggml_backend_feature> features;
#define _STRINGIFY(...) #__VA_ARGS__
-@@ -5458,7 +5496,7 @@ static ggml_backend_feature * ggml_backend_cuda_get_features(ggml_backend_reg_t
+@@ -5674,7 +5712,7 @@ static ggml_backend_feature * ggml_backend_cuda_get_features(ggml_backend_reg_t
GGML_UNUSED(reg);
}
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmq.cu.patch b/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmq.cu.patch
index baab0e7b..673e761f 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmq.cu.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmq.cu.patch
@@ -17,7 +17,7 @@ diff --git a/ggml/src/ggml-cuda/mmq.cu b/ggml/src/ggml-cuda/mmq.cu
// -----------------------------------------------------------------------
case GGML_TYPE_MXFP4:
mul_mat_q_case<GGML_TYPE_MXFP4>(ctx, args, stream);
-@@ -278,6 +280,7 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t
+@@ -285,6 +287,7 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t
case GGML_TYPE_Q5_K:
case GGML_TYPE_Q6_K:
// -------------------------------------------------
@@ -25,7 +25,7 @@ diff --git a/ggml/src/ggml-cuda/mmq.cu b/ggml/src/ggml-cuda/mmq.cu
case GGML_TYPE_IQ1_S:
case GGML_TYPE_IQ2_XXS:
case GGML_TYPE_IQ2_XS:
-@@ -286,6 +289,7 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t
+@@ -293,6 +296,7 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t
case GGML_TYPE_IQ3_S:
case GGML_TYPE_IQ4_XS:
case GGML_TYPE_IQ4_NL:
@@ -33,7 +33,7 @@ diff --git a/ggml/src/ggml-cuda/mmq.cu b/ggml/src/ggml-cuda/mmq.cu
// -------------------------------------------------
case GGML_TYPE_MXFP4:
case GGML_TYPE_NVFP4:
-@@ -360,9 +364,11 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t
+@@ -369,9 +373,11 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t
return ne11 <= 128;
case GGML_TYPE_Q6_K:
return ne11 <= (GGML_CUDA_CC_IS_RDNA3_0(cc) ? 128 : 256);
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmq.cuh.patch b/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmq.cuh.patch
index 9bda9b59..e8b3f216 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmq.cuh.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmq.cuh.patch
@@ -1,7 +1,7 @@
diff --git a/ggml/src/ggml-cuda/mmq.cuh b/ggml/src/ggml-cuda/mmq.cuh
--- a/llama.cpp/ggml/src/ggml-cuda/mmq.cuh
+++ b/llama.cpp/ggml/src/ggml-cuda/mmq.cuh
-@@ -1577,6 +1577,7 @@ extern DECL_MMQ_CASE(GGML_TYPE_Q4_K);
+@@ -1585,6 +1585,7 @@ extern DECL_MMQ_CASE(GGML_TYPE_Q4_K);
extern DECL_MMQ_CASE(GGML_TYPE_Q5_K);
extern DECL_MMQ_CASE(GGML_TYPE_Q6_K);
// -----------------------------------------
@@ -9,7 +9,7 @@ diff --git a/ggml/src/ggml-cuda/mmq.cuh b/ggml/src/ggml-cuda/mmq.cuh
extern DECL_MMQ_CASE(GGML_TYPE_IQ1_S);
extern DECL_MMQ_CASE(GGML_TYPE_IQ2_XXS);
extern DECL_MMQ_CASE(GGML_TYPE_IQ2_XS);
-@@ -1585,6 +1586,7 @@ extern DECL_MMQ_CASE(GGML_TYPE_IQ3_XXS);
+@@ -1593,6 +1594,7 @@ extern DECL_MMQ_CASE(GGML_TYPE_IQ3_XXS);
extern DECL_MMQ_CASE(GGML_TYPE_IQ3_S);
extern DECL_MMQ_CASE(GGML_TYPE_IQ4_NL);
extern DECL_MMQ_CASE(GGML_TYPE_IQ4_XS);
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmvq.cu.patch b/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmvq.cu.patch
index a7dbc1c7..5a5ebad5 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmvq.cu.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml-cuda_mmvq.cu.patch
@@ -1,7 +1,7 @@
diff --git a/ggml/src/ggml-cuda/mmvq.cu b/ggml/src/ggml-cuda/mmvq.cu
--- a/llama.cpp/ggml/src/ggml-cuda/mmvq.cu
+++ b/llama.cpp/ggml/src/ggml-cuda/mmvq.cu
-@@ -23,6 +23,7 @@ static constexpr __device__ vec_dot_q_cuda_t get_vec_dot_q_cuda(ggml_type type)
+@@ -53,6 +53,7 @@ static constexpr __device__ vec_dot_q_cuda_t get_vec_dot_q_cuda(ggml_type type)
case GGML_TYPE_Q4_K: return vec_dot_q4_K_q8_1;
case GGML_TYPE_Q5_K: return vec_dot_q5_K_q8_1;
case GGML_TYPE_Q6_K: return vec_dot_q6_K_q8_1;
@@ -9,7 +9,7 @@ diff --git a/ggml/src/ggml-cuda/mmvq.cu b/ggml/src/ggml-cuda/mmvq.cu
case GGML_TYPE_IQ2_XXS: return vec_dot_iq2_xxs_q8_1;
case GGML_TYPE_IQ2_XS: return vec_dot_iq2_xs_q8_1;
case GGML_TYPE_IQ2_S: return vec_dot_iq2_s_q8_1;
-@@ -32,6 +33,7 @@ static constexpr __device__ vec_dot_q_cuda_t get_vec_dot_q_cuda(ggml_type type)
+@@ -62,6 +63,7 @@ static constexpr __device__ vec_dot_q_cuda_t get_vec_dot_q_cuda(ggml_type type)
case GGML_TYPE_IQ4_NL: return vec_dot_iq4_nl_q8_1;
case GGML_TYPE_IQ4_XS: return vec_dot_iq4_xs_q8_1;
case GGML_TYPE_IQ3_S: return vec_dot_iq3_s_q8_1;
@@ -17,7 +17,7 @@ diff --git a/ggml/src/ggml-cuda/mmvq.cu b/ggml/src/ggml-cuda/mmvq.cu
default: return nullptr;
}
}
-@@ -52,6 +54,7 @@ static constexpr __host__ __device__ int get_vdr_mmvq(ggml_type type) {
+@@ -82,6 +84,7 @@ static constexpr __host__ __device__ int get_vdr_mmvq(ggml_type type) {
case GGML_TYPE_Q4_K: return VDR_Q4_K_Q8_1_MMVQ;
case GGML_TYPE_Q5_K: return VDR_Q5_K_Q8_1_MMVQ;
case GGML_TYPE_Q6_K: return VDR_Q6_K_Q8_1_MMVQ;
@@ -25,7 +25,7 @@ diff --git a/ggml/src/ggml-cuda/mmvq.cu b/ggml/src/ggml-cuda/mmvq.cu
case GGML_TYPE_IQ2_XXS: return VDR_IQ2_XXS_Q8_1_MMVQ;
case GGML_TYPE_IQ2_XS: return VDR_IQ2_XS_Q8_1_MMVQ;
case GGML_TYPE_IQ2_S: return VDR_IQ2_S_Q8_1_MMVQ;
-@@ -59,6 +62,7 @@ static constexpr __host__ __device__ int get_vdr_mmvq(ggml_type type) {
+@@ -89,6 +92,7 @@ static constexpr __host__ __device__ int get_vdr_mmvq(ggml_type type) {
case GGML_TYPE_IQ3_S: return VDR_IQ3_S_Q8_1_MMVQ;
case GGML_TYPE_IQ4_NL: return VDR_IQ4_NL_Q8_1_MMVQ;
case GGML_TYPE_IQ4_XS: return VDR_IQ4_XS_Q8_1_MMVQ;
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal-fusion.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal-fusion.cpp.patch
new file mode 100644
index 00000000..6f17a51e
--- /dev/null
+++ b/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal-fusion.cpp.patch
@@ -0,0 +1,21 @@
+diff --git a/ggml/src/ggml-metal/ggml-metal-fusion.cpp b/ggml/src/ggml-metal/ggml-metal-fusion.cpp
+--- a/llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.cpp
++++ b/llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.cpp
+@@ -717,7 +717,7 @@ static bool ggml_metal_fusion_match_raw_pattern(
+
+ static void ggml_metal_fusion_add_pattern_alloc_deps(
+ void * user_data,
+- void (*add_alloc_dep)(void *, ggml_tensor *, ggml_tensor *),
++ void (GGML_CALL *add_alloc_dep)(void *, ggml_tensor *, ggml_tensor *),
+ ggml_cgraph * gf,
+ const ggml_metal_fusion * fusion,
+ int node_idx) {
+@@ -739,7 +739,7 @@ static void ggml_metal_fusion_add_pattern_alloc_deps(
+
+ void ggml_metal_fusion_add_alloc_deps(
+ void * user_data,
+- void (*add_alloc_dep)(void *, ggml_tensor *, ggml_tensor *),
++ void (GGML_CALL *add_alloc_dep)(void *, ggml_tensor *, ggml_tensor *),
+ ggml_cgraph * gf) {
+ for (int i = 0; i < gf->n_nodes; ++i) {
+ const ggml_metal_fusion * best = nullptr;
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal-fusion.h.patch b/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal-fusion.h.patch
new file mode 100644
index 00000000..5dbbf36a
--- /dev/null
+++ b/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal-fusion.h.patch
@@ -0,0 +1,20 @@
+diff --git a/ggml/src/ggml-metal/ggml-metal-fusion.h b/ggml/src/ggml-metal/ggml-metal-fusion.h
+--- a/llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.h
++++ b/llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.h
+@@ -8,6 +8,7 @@
+ #pragma once
+
+ #include "ggml-impl.h"
++#include "ggml-backend.h"
+
+ #include <stdbool.h>
+
+@@ -51,7 +52,7 @@ ggml_metal_fusion_id ggml_metal_fusion_get_id(const struct ggml_metal_fusion * f
+ // apply any alloc-dependencies required by the fused kernels during graph optimize
+ void ggml_metal_fusion_add_alloc_deps(
+ void * user_data,
+- void (*add_alloc_dep)(void *, struct ggml_tensor *, struct ggml_tensor *),
++ void (GGML_CALL *add_alloc_dep)(void *, struct ggml_tensor *, struct ggml_tensor *),
+ struct ggml_cgraph * gf);
+
+ // ---- shared fusion info ---------------------------------------------------
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal.cpp.patch
index 0473ca6b..d5b1b90c 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal.cpp.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml-metal_ggml-metal.cpp.patch
@@ -1,7 +1,7 @@
diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal.cpp
--- a/llama.cpp/ggml/src/ggml-metal/ggml-metal.cpp
+++ b/llama.cpp/ggml/src/ggml-metal/ggml-metal.cpp
-@@ -26,7 +26,7 @@ static bool ggml_backend_buffer_is_metal(ggml_backend_buffer_t buffer);
+@@ -28,7 +28,7 @@ static bool ggml_backend_buffer_is_metal(ggml_backend_buffer_t buffer);
// shared buffer
@@ -10,7 +10,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;
GGML_ASSERT(ggml_metal_buffer_is_shared(ctx));
-@@ -34,7 +34,7 @@ static void ggml_backend_metal_buffer_shared_free_buffer(ggml_backend_buffer_t b
+@@ -36,7 +36,7 @@ static void ggml_backend_metal_buffer_shared_free_buffer(ggml_backend_buffer_t b
ggml_metal_buffer_free(ctx);
}
@@ -19,7 +19,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;
GGML_ASSERT(ggml_metal_buffer_is_shared(ctx));
-@@ -42,7 +42,7 @@ static void * ggml_backend_metal_buffer_shared_get_base(ggml_backend_buffer_t bu
+@@ -44,7 +44,7 @@ static void * ggml_backend_metal_buffer_shared_get_base(ggml_backend_buffer_t bu
return ggml_metal_buffer_get_base(ctx);
}
@@ -28,7 +28,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;
GGML_ASSERT(ggml_metal_buffer_is_shared(ctx));
-@@ -50,7 +50,7 @@ static void ggml_backend_metal_buffer_shared_memset_tensor(ggml_backend_buffer_t
+@@ -52,7 +52,7 @@ static void ggml_backend_metal_buffer_shared_memset_tensor(ggml_backend_buffer_t
ggml_metal_buffer_memset_tensor(ctx, tensor, value, offset, size);
}
@@ -37,7 +37,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;
GGML_ASSERT(ggml_metal_buffer_is_shared(ctx));
-@@ -58,7 +58,7 @@ static void ggml_backend_metal_buffer_shared_set_tensor(ggml_backend_buffer_t bu
+@@ -60,7 +60,7 @@ static void ggml_backend_metal_buffer_shared_set_tensor(ggml_backend_buffer_t bu
ggml_metal_buffer_set_tensor(ctx, tensor, data, offset, size);
}
@@ -46,7 +46,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;
GGML_ASSERT(ggml_metal_buffer_is_shared(ctx));
-@@ -66,7 +66,7 @@ static void ggml_backend_metal_buffer_shared_get_tensor(ggml_backend_buffer_t bu
+@@ -68,7 +68,7 @@ static void ggml_backend_metal_buffer_shared_get_tensor(ggml_backend_buffer_t bu
ggml_metal_buffer_get_tensor(ctx, tensor, data, offset, size);
}
@@ -55,7 +55,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;
GGML_ASSERT(ggml_metal_buffer_is_shared(ctx));
-@@ -78,7 +78,7 @@ static bool ggml_backend_metal_buffer_shared_cpy_tensor(ggml_backend_buffer_t bu
+@@ -80,7 +80,7 @@ static bool ggml_backend_metal_buffer_shared_cpy_tensor(ggml_backend_buffer_t bu
return ggml_metal_buffer_cpy_tensor(ctx, src, dst);
}
@@ -64,7 +64,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;
GGML_ASSERT(ggml_metal_buffer_is_shared(ctx));
-@@ -86,6 +86,14 @@ static void ggml_backend_metal_buffer_shared_clear(ggml_backend_buffer_t buffer,
+@@ -88,6 +88,14 @@ static void ggml_backend_metal_buffer_shared_clear(ggml_backend_buffer_t buffer,
ggml_metal_buffer_clear(ctx, value);
}
@@ -79,7 +79,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
static ggml_backend_buffer_i ggml_backend_metal_buffer_shared_i = {
/* .free_buffer = */ ggml_backend_metal_buffer_shared_free_buffer,
/* .get_base = */ ggml_backend_metal_buffer_shared_get_base,
-@@ -98,11 +106,12 @@ static ggml_backend_buffer_i ggml_backend_metal_buffer_shared_i = {
+@@ -100,11 +108,12 @@ static ggml_backend_buffer_i ggml_backend_metal_buffer_shared_i = {
/* .cpy_tensor = */ ggml_backend_metal_buffer_shared_cpy_tensor,
/* .clear = */ ggml_backend_metal_buffer_shared_clear,
/* .reset = */ NULL,
@@ -93,7 +93,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;
GGML_ASSERT(!ggml_metal_buffer_is_shared(ctx));
-@@ -110,7 +119,7 @@ static void ggml_backend_metal_buffer_private_free_buffer(ggml_backend_buffer_t
+@@ -112,7 +121,7 @@ static void ggml_backend_metal_buffer_private_free_buffer(ggml_backend_buffer_t
ggml_metal_buffer_free(ctx);
}
@@ -102,7 +102,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;
GGML_ASSERT(!ggml_metal_buffer_is_shared(ctx));
-@@ -118,7 +127,7 @@ static void * ggml_backend_metal_buffer_private_get_base(ggml_backend_buffer_t b
+@@ -120,7 +129,7 @@ static void * ggml_backend_metal_buffer_private_get_base(ggml_backend_buffer_t b
return ggml_metal_buffer_get_base(ctx);
}
@@ -111,7 +111,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;
GGML_ASSERT(!ggml_metal_buffer_is_shared(ctx));
-@@ -126,7 +135,7 @@ static void ggml_backend_metal_buffer_private_memset_tensor(ggml_backend_buffer_
+@@ -128,7 +137,7 @@ static void ggml_backend_metal_buffer_private_memset_tensor(ggml_backend_buffer_
ggml_metal_buffer_memset_tensor(ctx, tensor, value, offset, size);
}
@@ -120,7 +120,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;
GGML_ASSERT(!ggml_metal_buffer_is_shared(ctx));
-@@ -134,7 +143,7 @@ static void ggml_backend_metal_buffer_private_set_tensor(ggml_backend_buffer_t b
+@@ -136,7 +145,7 @@ static void ggml_backend_metal_buffer_private_set_tensor(ggml_backend_buffer_t b
ggml_metal_buffer_set_tensor(ctx, tensor, data, offset, size);
}
@@ -129,7 +129,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;
GGML_ASSERT(!ggml_metal_buffer_is_shared(ctx));
-@@ -142,7 +151,7 @@ static void ggml_backend_metal_buffer_private_get_tensor(ggml_backend_buffer_t b
+@@ -144,7 +153,7 @@ static void ggml_backend_metal_buffer_private_get_tensor(ggml_backend_buffer_t b
ggml_metal_buffer_get_tensor(ctx, tensor, data, offset, size);
}
@@ -138,7 +138,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;
GGML_ASSERT(!ggml_metal_buffer_is_shared(ctx));
-@@ -154,7 +163,7 @@ static bool ggml_backend_metal_buffer_private_cpy_tensor(ggml_backend_buffer_t b
+@@ -156,7 +165,7 @@ static bool ggml_backend_metal_buffer_private_cpy_tensor(ggml_backend_buffer_t b
return ggml_metal_buffer_cpy_tensor(ctx, src, dst);
}
@@ -147,7 +147,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_buffer_t ctx = (ggml_metal_buffer_t)buffer->context;
GGML_ASSERT(!ggml_metal_buffer_is_shared(ctx));
-@@ -174,6 +183,7 @@ static ggml_backend_buffer_i ggml_backend_metal_buffer_private_i = {
+@@ -176,6 +185,7 @@ static ggml_backend_buffer_i ggml_backend_metal_buffer_private_i = {
/* .cpy_tensor = */ ggml_backend_metal_buffer_private_cpy_tensor,
/* .clear = */ ggml_backend_metal_buffer_private_clear,
/* .reset = */ NULL,
@@ -155,7 +155,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
};
static bool ggml_backend_buffer_is_metal(ggml_backend_buffer_t buffer) {
-@@ -246,33 +256,33 @@ static size_t ggml_backend_metal_buffer_type_get_alloc_size(ggml_backend_buffer_
+@@ -256,33 +266,33 @@ static size_t ggml_backend_metal_buffer_type_get_alloc_size(ggml_backend_buffer_
// default (shared) buffer type
@@ -195,7 +195,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
return false;
GGML_UNUSED(buft);
-@@ -322,33 +332,33 @@ static ggml_backend_buffer_type_t ggml_backend_metal_buffer_type_shared(int devi
+@@ -332,33 +342,33 @@ static ggml_backend_buffer_type_t ggml_backend_metal_buffer_type_shared(int devi
// default (private) buffer type
@@ -235,7 +235,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
return false;
GGML_UNUSED(buft);
-@@ -397,34 +407,34 @@ static ggml_backend_buffer_type_t ggml_backend_metal_buffer_type_private(int dev
+@@ -407,34 +417,34 @@ static ggml_backend_buffer_type_t ggml_backend_metal_buffer_type_private(int dev
// mapped buffer type
@@ -276,7 +276,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
return false;
GGML_UNUSED(buft);
-@@ -475,13 +485,13 @@ static ggml_backend_buffer_type_t ggml_backend_metal_buffer_type_mapped(int devi
+@@ -485,13 +495,13 @@ static ggml_backend_buffer_type_t ggml_backend_metal_buffer_type_mapped(int devi
// backend
@@ -292,7 +292,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_t ctx = (ggml_metal_t)backend->context;
// wait for any ongoing async operations to finish
-@@ -492,25 +502,25 @@ static void ggml_backend_metal_free(ggml_backend_t backend) {
+@@ -502,25 +512,25 @@ static void ggml_backend_metal_free(ggml_backend_t backend) {
free(backend);
}
@@ -322,7 +322,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
if (!ggml_backend_is_metal(backend_src) || !ggml_backend_is_metal(backend_dst)) {
return false;
}
-@@ -531,27 +541,27 @@ static bool ggml_backend_metal_cpy_tensor_async(ggml_backend_t backend_src, ggml
+@@ -541,27 +551,27 @@ static bool ggml_backend_metal_cpy_tensor_async(ggml_backend_t backend_src, ggml
return ggml_metal_cpy_tensor_async(ctx_src, ctx_dst, src, dst);
}
@@ -349,12 +349,12 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_event_wait(ctx, ev);
}
--static void ggml_backend_metal_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph) {
-+static void GGML_CALL ggml_backend_metal_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph) {
- ggml_metal_t ctx = (ggml_metal_t)backend->context;
+-static void ggml_backend_metal_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph, ggml_backend_graph_optimize_params * params) {
++static void GGML_CALL ggml_backend_metal_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph, ggml_backend_graph_optimize_params * params) {
+ GGML_ASSERT(params && params->add_alloc_dep);
- ggml_metal_graph_optimize(ctx, cgraph);
-@@ -643,7 +653,7 @@ void ggml_backend_metal_capture_next_compute(ggml_backend_t backend) {
+ // keep the MoE weighted-reduction inputs alive until the fused output so the
+@@ -659,7 +669,7 @@ void ggml_backend_metal_capture_next_compute(ggml_backend_t backend) {
// backend device
@@ -363,7 +363,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;
const ggml_metal_device_props * props_dev = ggml_metal_device_get_props(ctx_dev);
-@@ -651,25 +661,25 @@ static const char * ggml_backend_metal_device_get_name(ggml_backend_dev_t dev) {
+@@ -667,25 +677,25 @@ static const char * ggml_backend_metal_device_get_name(ggml_backend_dev_t dev) {
return props_dev->name;
}
@@ -393,7 +393,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
props->name = ggml_backend_metal_device_get_name(dev);
props->description = ggml_backend_metal_device_get_description(dev);
props->type = ggml_backend_metal_device_get_type(dev);
-@@ -685,7 +695,7 @@ static void ggml_backend_metal_device_get_props(ggml_backend_dev_t dev, ggml_bac
+@@ -701,7 +711,7 @@ static void ggml_backend_metal_device_get_props(ggml_backend_dev_t dev, ggml_bac
};
}
@@ -402,7 +402,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;
ggml_metal_t ctx = ggml_metal_init(ctx_dev);
-@@ -710,7 +720,7 @@ static ggml_backend_t ggml_backend_metal_device_init_backend(ggml_backend_dev_t
+@@ -726,7 +736,7 @@ static ggml_backend_t ggml_backend_metal_device_init_backend(ggml_backend_dev_t
GGML_UNUSED(params);
}
@@ -411,7 +411,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;
const ggml_metal_device_props * props_dev = ggml_metal_device_get_props(ctx_dev);
-@@ -718,7 +728,7 @@ static ggml_backend_buffer_type_t ggml_backend_metal_device_get_buffer_type(ggml
+@@ -734,7 +744,7 @@ static ggml_backend_buffer_type_t ggml_backend_metal_device_get_buffer_type(ggml
return props_dev->use_shared_buffers ? ggml_backend_metal_buffer_type_shared(props_dev->device) : ggml_backend_metal_buffer_type_private(props_dev->device);
}
@@ -420,7 +420,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;
ggml_metal_buffer_t res = ggml_metal_buffer_map(ctx_dev, ptr, size, max_tensor_size);
-@@ -728,13 +738,13 @@ static ggml_backend_buffer_t ggml_backend_metal_device_buffer_mapped(ggml_backen
+@@ -744,13 +754,13 @@ static ggml_backend_buffer_t ggml_backend_metal_device_buffer_mapped(ggml_backen
return ggml_backend_buffer_init(ggml_backend_metal_buffer_type_mapped(props_dev->device), ggml_backend_metal_buffer_shared_i, res, size);
}
@@ -436,7 +436,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
return
buft->device == dev && (
buft->iface.get_name == ggml_backend_metal_buffer_type_shared_get_name ||
-@@ -755,7 +765,7 @@ static int64_t get_op_batch_size(const ggml_tensor * op) {
+@@ -771,7 +781,7 @@ static int64_t get_op_batch_size(const ggml_tensor * op) {
}
}
@@ -445,7 +445,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;
return (op->op == GGML_OP_MUL_MAT ||
-@@ -763,7 +773,7 @@ static bool ggml_backend_metal_device_offload_op(ggml_backend_dev_t dev, const g
+@@ -779,7 +789,7 @@ static bool ggml_backend_metal_device_offload_op(ggml_backend_dev_t dev, const g
get_op_batch_size(op) >= ggml_metal_device_get_props(ctx_dev)->op_offload_min_batch_size;
}
@@ -454,7 +454,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;
ggml_metal_event_t event = ggml_metal_device_event_init(ctx_dev);
-@@ -777,7 +787,7 @@ static ggml_backend_event_t ggml_backend_metal_device_event_new(ggml_backend_dev
+@@ -793,7 +803,7 @@ static ggml_backend_event_t ggml_backend_metal_device_event_new(ggml_backend_dev
return ev;
}
@@ -463,7 +463,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;
ggml_metal_event_t ev = (ggml_metal_event_t)event->context;
-@@ -787,7 +797,7 @@ static void ggml_backend_metal_device_event_free(ggml_backend_dev_t dev, ggml_ba
+@@ -803,7 +813,7 @@ static void ggml_backend_metal_device_event_free(ggml_backend_dev_t dev, ggml_ba
delete event;
}
@@ -472,7 +472,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_metal_device_t ctx_dev = (ggml_metal_device_t)dev->context;
ggml_metal_event_t evt = (ggml_metal_event_t)event->context;
-@@ -839,18 +849,18 @@ struct ggml_backend_metal_reg_deleter {
+@@ -855,18 +865,18 @@ struct ggml_backend_metal_reg_deleter {
typedef std::unique_ptr<struct ggml_backend_metal_reg, ggml_backend_metal_reg_deleter> ggml_backend_metal_reg_ptr;
@@ -494,7 +494,7 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
ggml_backend_metal_reg_t ctx = (ggml_backend_metal_reg_t)reg->context;
GGML_ASSERT(index < ctx->devices.size());
return ctx->devices[index];
-@@ -863,13 +873,13 @@ static ggml_backend_feature g_ggml_backend_metal_features[] = {
+@@ -879,7 +889,7 @@ static ggml_backend_feature g_ggml_backend_metal_features[] = {
{ NULL, NULL },
};
@@ -503,6 +503,8 @@ diff --git a/ggml/src/ggml-metal/ggml-metal.cpp b/ggml/src/ggml-metal/ggml-metal
return g_ggml_backend_metal_features;
GGML_UNUSED(reg);
+@@ -936,7 +946,7 @@ static void ggml_backend_metal_fusion_set_enabled(ggml_backend_fusion_t finfo, b
+ ggml_metal_fusion_info_set_enabled((struct ggml_metal_fusion_info *) finfo, enabled);
}
-static void * ggml_backend_metal_get_proc_address(ggml_backend_reg_t reg, const char * name) {
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan-buffers.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan-buffers.cpp.patch
new file mode 100644
index 00000000..15445f90
--- /dev/null
+++ b/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan-buffers.cpp.patch
@@ -0,0 +1,11 @@
+diff --git a/ggml/src/ggml-vulkan/ggml-vulkan-buffers.cpp b/ggml/src/ggml-vulkan/ggml-vulkan-buffers.cpp
+--- a/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan-buffers.cpp
++++ b/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan-buffers.cpp
+@@ -754,6 +754,7 @@ ggml_backend_buffer_i ggml_backend_vk_buffer_interface = {
+ /* .cpy_tensor = */ ggml_backend_vk_buffer_cpy_tensor,
+ /* .clear = */ ggml_backend_vk_buffer_clear,
+ /* .reset = */ NULL,
++ /* .free_struct = */ ggml_backend_vk_buffer_free_struct,
+ };
+
+ vk_buffer ggml_vk_buffer_from_host_ptr(vk_device & device, void * ptr, size_t size) {
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan-common.h.patch b/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan-common.h.patch
new file mode 100644
index 00000000..779fdcf5
--- /dev/null
+++ b/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan-common.h.patch
@@ -0,0 +1,48 @@
+diff --git a/ggml/src/ggml-vulkan/ggml-vulkan-common.h b/ggml/src/ggml-vulkan/ggml-vulkan-common.h
+--- a/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan-common.h
++++ b/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan-common.h
+@@ -209,26 +209,27 @@ bool ggml_vk_can_fuse_snake(ggml_backend_vk_context * ctx, const struct ggml_cgr
+ bool ggml_vk_tensors_overlap(const ggml_tensor * a, const ggml_tensor * b, bool elementwise);
+ bool ggml_vk_can_fuse_rms_norm_mul_rope(ggml_backend_vk_context * ctx, const struct ggml_cgraph * cgraph, int node_idx);
+ uint32_t ggml_vk_fuse_multi_add(ggml_backend_vk_context * ctx, const struct ggml_cgraph * cgraph, int node_idx);
+-void ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph * graph, struct ggml_backend_graph_optimize_params * params);
++void GGML_CALL ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph * graph, struct ggml_backend_graph_optimize_params * params);
+
+ // backend
+ bool ggml_backend_buffer_is_vk(ggml_backend_buffer_t buffer);
+-void ggml_backend_vk_buffer_free_buffer(ggml_backend_buffer_t buffer);
+-void * ggml_backend_vk_buffer_get_base(ggml_backend_buffer_t buffer);
+-enum ggml_status ggml_backend_vk_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor);
+-void ggml_backend_vk_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size);
+-void ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size);
+-void ggml_backend_vk_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data);
+-void ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size);
+-void ggml_backend_vk_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data);
+-bool ggml_backend_vk_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * src, ggml_tensor * dst);
+-void ggml_backend_vk_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value);
+-const char * ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft);
+-ggml_backend_buffer_t ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size);
+-size_t ggml_backend_vk_buffer_type_get_alignment(ggml_backend_buffer_type_t buft);
+-size_t ggml_backend_vk_buffer_type_get_max_size(ggml_backend_buffer_type_t buft);
+-size_t ggml_backend_vk_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor);
+-void ggml_backend_vk_free(ggml_backend_t backend);
++void GGML_CALL ggml_backend_vk_buffer_free_buffer(ggml_backend_buffer_t buffer);
++void GGML_CALL ggml_backend_vk_buffer_free_struct(ggml_backend_buffer_t buffer);
++void * GGML_CALL ggml_backend_vk_buffer_get_base(ggml_backend_buffer_t buffer);
++enum ggml_status GGML_CALL ggml_backend_vk_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor);
++void GGML_CALL ggml_backend_vk_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size);
++void GGML_CALL ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size);
++void GGML_CALL ggml_backend_vk_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data);
++void GGML_CALL ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size);
++void GGML_CALL ggml_backend_vk_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data);
++bool GGML_CALL ggml_backend_vk_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * src, ggml_tensor * dst);
++void GGML_CALL ggml_backend_vk_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value);
++const char * GGML_CALL ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft);
++ggml_backend_buffer_t GGML_CALL ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size);
++size_t GGML_CALL ggml_backend_vk_buffer_type_get_alignment(ggml_backend_buffer_type_t buft);
++size_t GGML_CALL ggml_backend_vk_buffer_type_get_max_size(ggml_backend_buffer_type_t buft);
++size_t GGML_CALL ggml_backend_vk_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor);
++void GGML_CALL ggml_backend_vk_free(ggml_backend_t backend);
+ ggml_backend_reg_t ggml_backend_vk_reg();
+
+ // debug
diff --git a/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan.cpp.patch b/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan.cpp.patch
index 05a75537..36b4784f 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan.cpp.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml-vulkan_ggml-vulkan.cpp.patch
@@ -1,171 +1,139 @@
diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vulkan.cpp
--- a/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp
+++ b/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp
-@@ -367,11 +367,11 @@ struct vk_queue {
- bool transfer_only;
- };
-
--static const char * ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft);
--static ggml_backend_buffer_t ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size);
--static size_t ggml_backend_vk_buffer_type_get_alignment(ggml_backend_buffer_type_t buft);
--static size_t ggml_backend_vk_buffer_type_get_max_size(ggml_backend_buffer_type_t buft);
--static size_t ggml_backend_vk_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor);
-+static const char * GGML_CALL ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft);
-+static ggml_backend_buffer_t GGML_CALL ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size);
-+static size_t GGML_CALL ggml_backend_vk_buffer_type_get_alignment(ggml_backend_buffer_type_t buft);
-+static size_t GGML_CALL ggml_backend_vk_buffer_type_get_max_size(ggml_backend_buffer_type_t buft);
-+static size_t GGML_CALL ggml_backend_vk_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor);
- static ggml_backend_buffer_type_i ggml_backend_vk_buffer_type_interface = {
- /* .get_name = */ ggml_backend_vk_buffer_type_name,
- /* .alloc_buffer = */ ggml_backend_vk_buffer_type_alloc_buffer,
-@@ -2576,7 +2576,7 @@ static void ggml_vk_check_results_1(ggml_backend_vk_context * ctx, ggml_cgraph *
-
- typedef void (*ggml_vk_func_t)(ggml_backend_vk_context * ctx, vk_context& subctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst);
-
--static void ggml_backend_vk_free(ggml_backend_t backend);
-+static void GGML_CALL ggml_backend_vk_free(ggml_backend_t backend);
-
- static VkDeviceSize ggml_vk_get_max_buffer_range(const ggml_backend_vk_context * ctx, const vk_buffer &buf, const VkDeviceSize offset) {
- const VkDeviceSize range = std::min(VkDeviceSize{buf->size - offset},
-@@ -15864,20 +15864,28 @@ static bool ggml_backend_buffer_is_vk(ggml_backend_buffer_t buffer) {
+@@ -12492,20 +12492,28 @@ bool ggml_backend_buffer_is_vk(ggml_backend_buffer_t buffer) {
return buffer->buft->iface.get_name == ggml_backend_vk_buffer_type_name;
}
--static void ggml_backend_vk_buffer_free_buffer(ggml_backend_buffer_t buffer) {
-+static void GGML_CALL ggml_backend_vk_buffer_free_buffer(ggml_backend_buffer_t buffer) {
+-void ggml_backend_vk_buffer_free_buffer(ggml_backend_buffer_t buffer) {
++void GGML_CALL ggml_backend_vk_buffer_free_buffer(ggml_backend_buffer_t buffer) {
VK_LOG_MEMORY("ggml_backend_vk_buffer_free_buffer()");
ggml_backend_vk_buffer_context * ctx = (ggml_backend_vk_buffer_context *)buffer->context;
ggml_vk_destroy_buffer(ctx->dev_buffer);
delete ctx;
}
--static void * ggml_backend_vk_buffer_get_base(ggml_backend_buffer_t buffer) {
+-void * ggml_backend_vk_buffer_get_base(ggml_backend_buffer_t buffer) {
+// free_struct callback for cross-module memory management
+// When the Vulkan backend is loaded as a dynamic library, the buffer struct
+// is allocated by the dylib's 'new' operator and must be freed by the dylib's
+// 'delete' operator (not the main executable's)
-+static void GGML_CALL ggml_backend_vk_buffer_free_struct(ggml_backend_buffer_t buffer) {
++void GGML_CALL ggml_backend_vk_buffer_free_struct(ggml_backend_buffer_t buffer) {
+ delete buffer;
+}
+
-+static void * GGML_CALL ggml_backend_vk_buffer_get_base(ggml_backend_buffer_t buffer) {
++void * GGML_CALL ggml_backend_vk_buffer_get_base(ggml_backend_buffer_t buffer) {
return vk_ptr_base;
UNUSED(buffer);
}
--static enum ggml_status ggml_backend_vk_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) {
-+static enum ggml_status GGML_CALL ggml_backend_vk_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) {
+-enum ggml_status ggml_backend_vk_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) {
++enum ggml_status GGML_CALL ggml_backend_vk_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) {
VK_LOG_DEBUG("ggml_backend_vk_buffer_init_tensor(" << buffer << " (" << buffer->context << "), " << tensor << ")");
if (tensor->view_src != nullptr) {
GGML_ASSERT(tensor->view_src->buffer->buft == buffer->buft);
-@@ -15885,7 +15893,7 @@ static enum ggml_status ggml_backend_vk_buffer_init_tensor(ggml_backend_buffer_t
+@@ -12513,7 +12521,7 @@ enum ggml_status ggml_backend_vk_buffer_init_tensor(ggml_backend_buffer_t buffer
return GGML_STATUS_SUCCESS;
}
--static void ggml_backend_vk_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size) {
-+static void GGML_CALL ggml_backend_vk_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size) {
+-void ggml_backend_vk_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size) {
++void GGML_CALL ggml_backend_vk_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size) {
VK_LOG_DEBUG("ggml_backend_vk_buffer_memset_tensor(" << buffer << ", " << tensor << ", " << value << ", " << offset << ", " << size << ")");
ggml_backend_vk_buffer_context * buf_ctx = (ggml_backend_vk_buffer_context *)buffer->context;
vk_buffer buf = buf_ctx->dev_buffer;
-@@ -15898,7 +15906,7 @@ static void ggml_backend_vk_buffer_memset_tensor(ggml_backend_buffer_t buffer, g
+@@ -12526,7 +12534,7 @@ void ggml_backend_vk_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_ten
ggml_vk_buffer_memset(buf, vk_tensor_offset(tensor) + tensor->view_offs + offset, val32, size);
}
--static void ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) {
-+static void GGML_CALL ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) {
+-void ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) {
++void GGML_CALL ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) {
VK_LOG_DEBUG("ggml_backend_vk_buffer_set_tensor(" << buffer << ", " << tensor << ", " << data << ", " << offset << ", " << size << ")");
ggml_backend_vk_buffer_context * buf_ctx = (ggml_backend_vk_buffer_context *)buffer->context;
vk_buffer buf = buf_ctx->dev_buffer;
-@@ -15910,7 +15918,7 @@ static void ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml
+@@ -12538,7 +12546,7 @@ void ggml_backend_vk_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor
ggml_vk_buffer_write(buf, vk_tensor_offset(tensor) + tensor->view_offs + offset, data, size);
}
--static void ggml_backend_vk_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset,
-+static void GGML_CALL ggml_backend_vk_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset,
+-void ggml_backend_vk_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset,
++void GGML_CALL ggml_backend_vk_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset,
size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) {
VK_LOG_DEBUG("ggml_backend_vk_buffer_set_tensor_2d(" << buffer << ", " << tensor << ", " << data << ", " << offset << ", " << size << ", " <<
n_copies << ", " << stride_tensor << ", " << stride_data << ")");
-@@ -15924,7 +15932,7 @@ static void ggml_backend_vk_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, g
+@@ -12552,7 +12560,7 @@ void ggml_backend_vk_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, ggml_ten
ggml_vk_buffer_write_2d(buf, vk_tensor_offset(tensor) + tensor->view_offs + offset, data, stride_data, stride_tensor, size, n_copies);
}
--static void ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) {
-+static void GGML_CALL ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) {
+-void ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) {
++void GGML_CALL ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) {
VK_LOG_DEBUG("ggml_backend_vk_buffer_get_tensor(" << buffer << ", " << tensor << ", " << data << ", " << offset << ", " << size << ")");
ggml_backend_vk_buffer_context * buf_ctx = (ggml_backend_vk_buffer_context *)buffer->context;
-@@ -15937,7 +15945,7 @@ static void ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer_t buffer, cons
+@@ -12565,7 +12573,7 @@ void ggml_backend_vk_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_
ggml_vk_buffer_read(buf, vk_tensor_offset(tensor) + tensor->view_offs + offset, data, size);
}
--static void ggml_backend_vk_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset,
-+static void GGML_CALL ggml_backend_vk_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset,
+-void ggml_backend_vk_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset,
++void GGML_CALL ggml_backend_vk_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset,
size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) {
VK_LOG_DEBUG("ggml_backend_vk_buffer_get_tensor_2d(" << buffer << ", " << tensor << ", " << data << ", " << offset << ", " << size << ", " <<
n_copies << ", " << stride_tensor << ", " << stride_data << ")");
-@@ -15952,7 +15960,7 @@ static void ggml_backend_vk_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, c
+@@ -12580,7 +12588,7 @@ void ggml_backend_vk_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const gg
ggml_vk_buffer_read_2d(buf, vk_tensor_offset(tensor) + tensor->view_offs + offset, data, stride_tensor, stride_data, size, n_copies);
}
--static bool ggml_backend_vk_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * src, ggml_tensor * dst) {
-+static bool GGML_CALL ggml_backend_vk_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * src, ggml_tensor * dst) {
+-bool ggml_backend_vk_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * src, ggml_tensor * dst) {
++bool GGML_CALL ggml_backend_vk_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * src, ggml_tensor * dst) {
if (ggml_nbytes(src) == 0) {
return true;
}
-@@ -15973,7 +15981,7 @@ static bool ggml_backend_vk_buffer_cpy_tensor(ggml_backend_buffer_t buffer, cons
+@@ -12601,19 +12609,19 @@ bool ggml_backend_vk_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_
UNUSED(buffer);
}
--static void ggml_backend_vk_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) {
-+static void GGML_CALL ggml_backend_vk_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) {
+-void ggml_backend_vk_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) {
++void GGML_CALL ggml_backend_vk_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) {
ggml_backend_vk_buffer_context * ctx = (ggml_backend_vk_buffer_context *)buffer->context;
ggml_vk_buffer_memset(ctx->dev_buffer, 0, value, buffer->size);
-@@ -15991,16 +15999,17 @@ static ggml_backend_buffer_i ggml_backend_vk_buffer_interface = {
- /* .cpy_tensor = */ ggml_backend_vk_buffer_cpy_tensor,
- /* .clear = */ ggml_backend_vk_buffer_clear,
- /* .reset = */ NULL,
-+ /* .free_struct = */ ggml_backend_vk_buffer_free_struct,
- };
+ }
- // vk buffer type
--static const char * ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft) {
-+static const char * GGML_CALL ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft) {
+-const char * ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft) {
++const char * GGML_CALL ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft) {
ggml_backend_vk_buffer_type_context * ctx = (ggml_backend_vk_buffer_type_context *)buft->context;
return ctx->name.c_str();
}
--static ggml_backend_buffer_t ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) {
-+static ggml_backend_buffer_t GGML_CALL ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) {
+-ggml_backend_buffer_t ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) {
++ggml_backend_buffer_t GGML_CALL ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) {
VK_LOG_MEMORY("ggml_backend_vk_buffer_type_alloc_buffer(" << size << ")");
ggml_backend_vk_buffer_type_context * ctx = (ggml_backend_vk_buffer_type_context *) buft->context;
-@@ -16016,17 +16025,17 @@ static ggml_backend_buffer_t ggml_backend_vk_buffer_type_alloc_buffer(ggml_backe
+@@ -12629,17 +12637,17 @@ ggml_backend_buffer_t ggml_backend_vk_buffer_type_alloc_buffer(ggml_backend_buff
return ggml_backend_buffer_init(buft, ggml_backend_vk_buffer_interface, bufctx, size);
}
--static size_t ggml_backend_vk_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) {
-+static size_t GGML_CALL ggml_backend_vk_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) {
+-size_t ggml_backend_vk_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) {
++size_t GGML_CALL ggml_backend_vk_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) {
ggml_backend_vk_buffer_type_context * ctx = (ggml_backend_vk_buffer_type_context *) buft->context;
return ctx->device->properties.limits.minStorageBufferOffsetAlignment;
}
--static size_t ggml_backend_vk_buffer_type_get_max_size(ggml_backend_buffer_type_t buft) {
-+static size_t GGML_CALL ggml_backend_vk_buffer_type_get_max_size(ggml_backend_buffer_type_t buft) {
+-size_t ggml_backend_vk_buffer_type_get_max_size(ggml_backend_buffer_type_t buft) {
++size_t GGML_CALL ggml_backend_vk_buffer_type_get_max_size(ggml_backend_buffer_type_t buft) {
ggml_backend_vk_buffer_type_context * ctx = (ggml_backend_vk_buffer_type_context *) buft->context;
return ctx->device->suballocation_block_size;
}
--static size_t ggml_backend_vk_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) {
-+static size_t GGML_CALL ggml_backend_vk_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) {
+-size_t ggml_backend_vk_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) {
++size_t GGML_CALL ggml_backend_vk_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) {
return ggml_nbytes(tensor);
UNUSED(buft);
-@@ -16044,18 +16053,18 @@ ggml_backend_buffer_type_t ggml_backend_vk_buffer_type(size_t dev_num) {
-
- // host buffer type
+@@ -12655,18 +12663,18 @@ ggml_backend_buffer_type_t ggml_backend_vk_buffer_type(size_t dev_num) {
+ return &dev->buffer_type;
+ }
-static const char * ggml_backend_vk_host_buffer_type_name(ggml_backend_buffer_type_t buft) {
+static const char * GGML_CALL ggml_backend_vk_host_buffer_type_name(ggml_backend_buffer_type_t buft) {
@@ -185,7 +153,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
VK_LOG_MEMORY("ggml_backend_vk_host_buffer_type_alloc_buffer(" << size << ")");
size += 32; // Behave like the CPU buffer type
-@@ -16071,19 +16080,20 @@ static ggml_backend_buffer_t ggml_backend_vk_host_buffer_type_alloc_buffer(ggml_
+@@ -12682,19 +12690,20 @@ static ggml_backend_buffer_t ggml_backend_vk_host_buffer_type_alloc_buffer(ggml_
ggml_backend_buffer_t buffer = ggml_backend_cpu_buffer_from_ptr(ptr, size);
buffer->buft = buft;
buffer->iface.free_buffer = ggml_backend_vk_host_buffer_free_buffer;
@@ -208,9 +176,9 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
return vk_instance.devices[0]->suballocation_block_size;
UNUSED(buft);
-@@ -16115,13 +16125,13 @@ ggml_backend_buffer_type_t ggml_backend_vk_host_buffer_type() {
-
- // backend
+@@ -12721,13 +12730,13 @@ ggml_backend_buffer_type_t ggml_backend_vk_host_buffer_type() {
+ return &ggml_backend_vk_buffer_type_host;
+ }
-static const char * ggml_backend_vk_name(ggml_backend_t backend) {
+static const char * GGML_CALL ggml_backend_vk_name(ggml_backend_t backend) {
@@ -219,12 +187,12 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
return ctx->name.c_str();
}
--static void ggml_backend_vk_free(ggml_backend_t backend) {
-+static void GGML_CALL ggml_backend_vk_free(ggml_backend_t backend) {
+-void ggml_backend_vk_free(ggml_backend_t backend) {
++void GGML_CALL ggml_backend_vk_free(ggml_backend_t backend) {
ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
VK_LOG_DEBUG("ggml_backend_vk_free(" << ctx->name << ")");
-@@ -16137,7 +16147,7 @@ static ggml_backend_buffer_type_t ggml_backend_vk_get_default_buffer_type(ggml_b
+@@ -12743,7 +12752,7 @@ static ggml_backend_buffer_type_t ggml_backend_vk_get_default_buffer_type(ggml_b
return &ctx->device->buffer_type;
}
@@ -233,7 +201,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) {
VK_LOG_DEBUG("ggml_backend_vk_set_tensor_2d_async(" << size << ", " << n_copies << ")");
ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
-@@ -16195,12 +16205,12 @@ static void ggml_backend_vk_set_tensor_2d_async(ggml_backend_t backend, ggml_ten
+@@ -12801,12 +12810,12 @@ static void ggml_backend_vk_set_tensor_2d_async(ggml_backend_t backend, ggml_ten
}
}
@@ -248,7 +216,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) {
VK_LOG_DEBUG("ggml_backend_vk_get_tensor_2d_async(" << size << ", " << n_copies << ")");
ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
-@@ -16251,12 +16261,12 @@ static void ggml_backend_vk_get_tensor_2d_async(ggml_backend_t backend, const gg
+@@ -12857,12 +12866,12 @@ static void ggml_backend_vk_get_tensor_2d_async(ggml_backend_t backend, const gg
}
}
@@ -263,7 +231,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
VK_LOG_DEBUG("ggml_backend_vk_cpy_tensor_async(" << src << " -> " << dst << ", size=" << ggml_nbytes(src) << ")");
ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend_dst->context;
-@@ -16383,7 +16393,7 @@ static void ggml_vk_synchronize(ggml_backend_vk_context * ctx) {
+@@ -13004,7 +13013,7 @@ void ggml_vk_synchronize(ggml_backend_vk_context * ctx) {
}
}
@@ -272,7 +240,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
VK_LOG_DEBUG("ggml_backend_vk_synchronize()");
ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
-@@ -16935,7 +16945,7 @@ static int32_t find_first_set(uint32_t x) {
+@@ -13741,7 +13750,7 @@ static int32_t find_first_set(uint32_t x) {
return ret;
}
@@ -281,16 +249,16 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
VK_LOG_DEBUG("ggml_backend_vk_graph_compute(" << cgraph->n_nodes << " nodes)");
ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
-@@ -17400,7 +17410,7 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg
+@@ -14236,7 +14245,7 @@ static ggml_status ggml_backend_vk_graph_compute(ggml_backend_t backend, ggml_cg
+ UNUSED(backend);
}
- // Sort the graph for improved parallelism.
--static void ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph * graph)
-+static void GGML_CALL ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph * graph)
+-void ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph * graph, struct ggml_backend_graph_optimize_params * params)
++void GGML_CALL ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph * graph, struct ggml_backend_graph_optimize_params * params)
{
VK_LOG_DEBUG("ggml_vk_graph_optimize(" << graph->n_nodes << " nodes)");
ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
-@@ -17661,7 +17671,7 @@ static void ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph *
+@@ -14617,7 +14626,7 @@ void ggml_vk_graph_optimize(ggml_backend_t backend, struct ggml_cgraph * graph,
}
}
@@ -299,7 +267,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
VK_LOG_DEBUG("ggml_backend_vk_event_record(backend=" << backend << ", event=" << event << ")");
ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
vk_event *vkev = (vk_event *)event->context;
-@@ -17699,7 +17709,7 @@ static void ggml_backend_vk_event_record(ggml_backend_t backend, ggml_backend_ev
+@@ -14655,7 +14664,7 @@ static void ggml_backend_vk_event_record(ggml_backend_t backend, ggml_backend_ev
ctx->compute_ctx.reset();
}
@@ -308,7 +276,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
VK_LOG_DEBUG("ggml_backend_vk_event_wait(backend=" << backend << ", event=" << event << ")");
ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
vk_event *vkev = (vk_event *)event->context;
-@@ -17801,7 +17811,10 @@ void ggml_backend_vk_get_device_memory(int device, size_t * free, size_t * total
+@@ -14756,7 +14765,10 @@ void ggml_backend_vk_get_device_memory(int device, size_t * free, size_t * total
*total += heap.size;
if (membudget_supported && i < budgetprops.heapUsage.size()) {
@@ -320,9 +288,9 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
} else {
*free += heap.size;
}
-@@ -17869,38 +17882,38 @@ struct ggml_backend_vk_device_context {
- int op_offload_min_batch_size;
- };
+@@ -14813,38 +14825,38 @@ static std::string ggml_backend_vk_get_device_pci_id(int device_idx) {
+ return std::string(pci_bus_id);
+ }
-static const char * ggml_backend_vk_device_get_name(ggml_backend_dev_t dev) {
+static const char * GGML_CALL ggml_backend_vk_device_get_name(ggml_backend_dev_t dev) {
@@ -366,7 +334,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
ggml_backend_vk_device_context * ctx = (ggml_backend_vk_device_context *)dev->context;
props->name = ggml_backend_vk_device_get_name(dev);
-@@ -17917,13 +17930,13 @@ static void ggml_backend_vk_device_get_props(ggml_backend_dev_t dev, struct ggml
+@@ -14861,13 +14873,13 @@ static void ggml_backend_vk_device_get_props(ggml_backend_dev_t dev, struct ggml
};
}
@@ -382,7 +350,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
ggml_backend_vk_device_context * ctx = (ggml_backend_vk_device_context *)dev->context;
const vk_device& device = ggml_vk_get_device(ctx->device);
-@@ -18504,7 +18517,7 @@ static bool ggml_backend_vk_device_supports_op(ggml_backend_dev_t dev, const ggm
+@@ -15522,7 +15534,7 @@ static bool ggml_backend_vk_device_supports_op(ggml_backend_dev_t dev, const ggm
UNUSED(dev);
}
@@ -391,8 +359,8 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
if (buft->iface.get_name != ggml_backend_vk_buffer_type_name) {
return false;
}
-@@ -18530,13 +18543,13 @@ static int64_t ggml_vk_get_op_batch_size(const ggml_tensor * op) {
- }
+@@ -15533,13 +15545,13 @@ static bool ggml_backend_vk_device_supports_buft(ggml_backend_dev_t dev, ggml_ba
+ return buft_ctx->device->idx == ctx->device;
}
-static bool ggml_backend_vk_device_offload_op(ggml_backend_dev_t dev, const ggml_tensor * op) {
@@ -407,7 +375,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
ggml_backend_vk_device_context * ctx = (ggml_backend_vk_device_context *)dev->context;
auto device = ggml_vk_get_device(ctx->device);
-@@ -18559,7 +18572,7 @@ static ggml_backend_event_t ggml_backend_vk_device_event_new(ggml_backend_dev_t
+@@ -15562,7 +15574,7 @@ static ggml_backend_event_t ggml_backend_vk_device_event_new(ggml_backend_dev_t
};
}
@@ -416,7 +384,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
ggml_backend_vk_device_context * ctx = (ggml_backend_vk_device_context *)dev->context;
auto device = ggml_vk_get_device(ctx->device);
-@@ -18579,7 +18592,7 @@ static void ggml_backend_vk_device_event_free(ggml_backend_dev_t dev, ggml_backe
+@@ -15582,7 +15594,7 @@ static void ggml_backend_vk_device_event_free(ggml_backend_dev_t dev, ggml_backe
delete event;
}
@@ -425,8 +393,8 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
VK_LOG_DEBUG("ggml_backend_vk_device_event_synchronize(backend=" << dev << ", event=" << event << ")");
ggml_backend_vk_device_context * ctx = (ggml_backend_vk_device_context *)dev->context;
auto device = ggml_vk_get_device(ctx->device);
-@@ -18636,7 +18649,7 @@ static vk_buffer ggml_vk_buffer_from_host_ptr(vk_device & device, void * ptr, si
- return buf;
+@@ -15614,7 +15626,7 @@ static void ggml_backend_vk_device_event_synchronize(ggml_backend_dev_t dev, ggm
+ }
}
-static ggml_backend_buffer_t ggml_backend_vk_device_buffer_from_host_ptr(ggml_backend_dev_t dev, void * ptr, size_t size, size_t max_tensor_size) {
@@ -434,7 +402,7 @@ diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vu
VK_LOG_DEBUG("ggml_backend_vk_device_buffer_from_host_ptr(backend=" << dev << ", ptr=" << ptr << ", size=" << size << ")");
GGML_UNUSED(max_tensor_size);
-@@ -18674,17 +18687,17 @@ static const struct ggml_backend_device_i ggml_backend_vk_device_i = {
+@@ -15652,17 +15664,17 @@ static const struct ggml_backend_device_i ggml_backend_vk_device_i = {
/* .event_synchronize = */ ggml_backend_vk_device_event_synchronize,
};
diff --git a/llama.cpp.patches/patches/ggml_src_ggml.c.patch b/llama.cpp.patches/patches/ggml_src_ggml.c.patch
index 6b38ed86..d74076b8 100644
--- a/llama.cpp.patches/patches/ggml_src_ggml.c.patch
+++ b/llama.cpp.patches/patches/ggml_src_ggml.c.patch
@@ -1,7 +1,7 @@
diff --git a/ggml/src/ggml.c b/ggml/src/ggml.c
--- a/llama.cpp/ggml/src/ggml.c
+++ b/llama.cpp/ggml/src/ggml.c
-@@ -547,11 +547,13 @@ void ggml_time_init(void) {
+@@ -548,11 +548,13 @@ void ggml_time_init(void) {
InitOnceExecuteOnce(&once, ggml_time_init_once, NULL, NULL);
}
int64_t ggml_time_ms(void) {
diff --git a/llama.cpp.patches/patches/ggml_src_gguf.cpp.patch b/llama.cpp.patches/patches/ggml_src_gguf.cpp.patch
index 1707e908..0036961e 100644
--- a/llama.cpp.patches/patches/ggml_src_gguf.cpp.patch
+++ b/llama.cpp.patches/patches/ggml_src_gguf.cpp.patch
@@ -12,7 +12,7 @@ diff --git a/ggml/src/gguf.cpp b/ggml/src/gguf.cpp
#include <cinttypes>
#include <cstddef>
#include <cstdint>
-@@ -989,7 +993,54 @@ struct gguf_context * gguf_init_from_buffer(const void * data, size_t size, stru
+@@ -997,7 +1001,54 @@ struct gguf_context * gguf_init_from_buffer(const void * data, size_t size, stru
return gguf_init_from_reader(gr, params);
}
@@ -67,7 +67,7 @@ diff --git a/ggml/src/gguf.cpp b/ggml/src/gguf.cpp
FILE * file = ggml_fopen(fname, "rb");
if (!file) {
-@@ -1000,6 +1051,7 @@ struct gguf_context * gguf_init_from_file(const char * fname, struct gguf_init_p
+@@ -1008,6 +1059,7 @@ struct gguf_context * gguf_init_from_file(const char * fname, struct gguf_init_p
struct gguf_context * result = gguf_init_from_file_ptr(file, params);
fclose(file);
return result;
diff --git a/llama.cpp.patches/patches/src_llama-context.cpp.patch b/llama.cpp.patches/patches/src_llama-context.cpp.patch
index d6906466..f99ba6a1 100644
--- a/llama.cpp.patches/patches/src_llama-context.cpp.patch
+++ b/llama.cpp.patches/patches/src_llama-context.cpp.patch
@@ -1,7 +1,7 @@
diff --git a/src/llama-context.cpp b/src/llama-context.cpp
--- a/llama.cpp/src/llama-context.cpp
+++ b/llama.cpp/src/llama-context.cpp
-@@ -551,6 +551,16 @@ void llama_context::resolve_fused_ops(const llama_memory_context_i * mctx, uint3
+@@ -553,6 +553,16 @@ void llama_context::resolve_fused_ops(const llama_memory_context_i * mctx, uint3
}
};
diff --git a/llama.cpp.patches/patches/src_llama-mmap.cpp.patch b/llama.cpp.patches/patches/src_llama-mmap.cpp.patch
index a15e763b..f0208df5 100644
--- a/llama.cpp.patches/patches/src_llama-mmap.cpp.patch
+++ b/llama.cpp.patches/patches/src_llama-mmap.cpp.patch
@@ -11,8 +11,8 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp
+
#include <cstring>
#include <climits>
- #include <stdexcept>
-@@ -181,6 +185,15 @@ struct llama_file::impl {
+ #include <cstdlib>
+@@ -182,6 +186,15 @@ struct llama_file::impl {
}
#else
impl(const char * fname, const char * mode, [[maybe_unused]] const bool use_direct_io = false) : fname(fname) {
@@ -28,7 +28,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp
#ifdef __linux__
// Try unbuffered I/O for read only
if (use_direct_io && std::strcmp(mode, "rb") == 0) {
-@@ -233,6 +246,9 @@ struct llama_file::impl {
+@@ -234,6 +247,9 @@ struct llama_file::impl {
}
size_t tell() const {
@@ -38,7 +38,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp
if (fd == -1) {
off_t ret = llama_mmap_ftell(fp);
if (ret == -1) {
-@@ -250,6 +266,11 @@ struct llama_file::impl {
+@@ -251,6 +267,11 @@ struct llama_file::impl {
}
void seek(size_t offset, int whence) const {
@@ -50,7 +50,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp
off_t ret = 0;
if (fd == -1) {
ret = llama_mmap_fseek(fp, offset, whence);
-@@ -259,12 +280,22 @@ struct llama_file::impl {
+@@ -260,12 +281,22 @@ struct llama_file::impl {
if (ret == -1) {
throw std::runtime_error(format("seek error: %s", strerror(errno)));
}
@@ -73,7 +73,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp
errno = 0;
if (fd == -1) {
const size_t curr_off = tell();
-@@ -314,6 +345,7 @@ struct llama_file::impl {
+@@ -315,6 +346,7 @@ struct llama_file::impl {
bytes_read += (size_t) ret;
}
}
@@ -81,7 +81,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp
}
void read_aligned_chunk(void * dest, size_t size) {
-@@ -374,13 +406,20 @@ struct llama_file::impl {
+@@ -375,13 +407,20 @@ struct llama_file::impl {
}
~impl() {
@@ -102,7 +102,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp
std::string fname;
#endif
-@@ -409,6 +448,10 @@ size_t llama_file::read_alignment() const { return pimpl->read_alignment(); }
+@@ -410,6 +449,10 @@ size_t llama_file::read_alignment() const { return pimpl->read_alignment(); }
bool llama_file::has_direct_io() const { return pimpl->has_direct_io(); }
int llama_file::file_id() const {
@@ -113,7 +113,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp
#ifdef _WIN32
return _fileno(pimpl->fp);
#else
-@@ -416,9 +459,10 @@ int llama_file::file_id() const {
+@@ -417,9 +460,10 @@ int llama_file::file_id() const {
return pimpl->fd;
}
#if defined(fileno)
@@ -126,7 +126,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp
#endif
#endif
}
-@@ -436,14 +480,56 @@ uint32_t llama_file::read_u32() { return pimpl->read_u32(); }
+@@ -437,6 +481,31 @@ uint32_t llama_file::read_u32() { return pimpl->read_u32(); }
void llama_file::write_raw(const void * ptr, size_t len) const { pimpl->write_raw(ptr, len); }
void llama_file::write_u32(uint32_t val) const { pimpl->write_u32(val); }
@@ -157,6 +157,8 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp
+
// llama_mmap
+ #if defined(_POSIX_MAPPED_FILES) || defined(_WIN32)
+@@ -465,9 +534,26 @@ static llama_mmap::ranges ranges_complement(llama_mmap::ranges ranges, size_t li
struct llama_mmap::impl {
#ifdef _POSIX_MAPPED_FILES
std::vector<std::pair<size_t, size_t>> mapped_fragments;
@@ -165,7 +167,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp
+ llamafile * lfile = nullptr;
+#endif
- impl(struct llama_file * file, size_t prefetch, bool numa) {
+ impl(struct llama_file * file, size_t prefetch, bool numa, const llama_mmap::ranges & lazy_ranges) {
size = file->size();
+
+ if (file->has_premapped_content()) {
@@ -183,7 +185,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp
int fd = file->file_id();
int flags = MAP_SHARED;
if (numa) { prefetch = 0; }
-@@ -488,6 +574,10 @@ struct llama_mmap::impl {
+@@ -528,6 +614,10 @@ struct llama_mmap::impl {
}
void unmap_fragment(size_t first, size_t last) {
@@ -194,7 +196,7 @@ diff --git a/src/llama-mmap.cpp b/src/llama-mmap.cpp
int page_size = sysconf(_SC_PAGESIZE);
align_range(&first, &last, page_size);
size_t len = last - first;
-@@ -524,6 +614,14 @@ struct llama_mmap::impl {
+@@ -564,6 +654,14 @@ struct llama_mmap::impl {
}
~impl() {
diff --git a/llama.cpp.patches/patches/src_llama-mmap.h.patch b/llama.cpp.patches/patches/src_llama-mmap.h.patch
index 39fbf730..6b6ba5eb 100644
--- a/llama.cpp.patches/patches/src_llama-mmap.h.patch
+++ b/llama.cpp.patches/patches/src_llama-mmap.h.patch
@@ -1,7 +1,7 @@
diff --git a/src/llama-mmap.h b/src/llama-mmap.h
--- a/llama.cpp/src/llama-mmap.h
+++ b/llama.cpp/src/llama-mmap.h
-@@ -35,6 +35,11 @@ struct llama_file {
+@@ -36,6 +36,11 @@ struct llama_file {
size_t read_alignment() const;
bool has_direct_io() const;
diff --git a/llama.cpp.patches/patches/src_models_dflash.cpp.patch b/llama.cpp.patches/patches/src_models_dflash.cpp.patch
deleted file mode 100644
index b9d6e9ac..00000000
--- a/llama.cpp.patches/patches/src_models_dflash.cpp.patch
+++ /dev/null
@@ -1,48 +0,0 @@
-diff --git a/src/models/dflash.cpp b/src/models/dflash.cpp
---- a/llama.cpp/src/models/dflash.cpp
-+++ b/llama.cpp/src/models/dflash.cpp
-@@ -177,21 +177,6 @@ void llama_model_dflash::load_arch_tensors(llama_model_loader &) {
- }
- }
-
--std::unique_ptr<llm_graph_context> llama_model_dflash::build_arch_graph(const llm_graph_params & params) const {
-- switch (params.gtype) {
-- case LLM_GRAPH_TYPE_ENCODER:
-- return std::make_unique<graph<true>>(*this, params);
-- case LLM_GRAPH_TYPE_DEFAULT:
-- case LLM_GRAPH_TYPE_DECODER:
-- if (hparams.dsv4_hc_mult > 0) {
-- return std::make_unique<graph_dsv4>(*this, params);
-- }
-- return std::make_unique<graph<false>>(*this, params);
-- default:
-- GGML_ABORT("invalid graph type");
-- };
--}
--
- template <>
- ggml_tensor * llama_model_dflash::graph<true>::build_inp_embd_enc() const {
- auto inp_target = std::make_unique<llm_graph_input_embd>(hparams.n_embd_inp_enc());
-@@ -684,3 +669,22 @@ llama_model_dflash::graph_dsv4::graph_dsv4(const llama_model & model, const llm_
- build_dspark_markov_head(*this, model, inp_tokens);
- }
- }
-+
-+// Defined after the graph<true>/graph<false>/graph_dsv4 constructor
-+// specializations above: cosmocc's clang rejects an explicit specialization
-+// that appears after the point where make_unique<graph<...>> would implicitly
-+// instantiate it.
-+std::unique_ptr<llm_graph_context> llama_model_dflash::build_arch_graph(const llm_graph_params & params) const {
-+ switch (params.gtype) {
-+ case LLM_GRAPH_TYPE_ENCODER:
-+ return std::make_unique<graph<true>>(*this, params);
-+ case LLM_GRAPH_TYPE_DEFAULT:
-+ case LLM_GRAPH_TYPE_DECODER:
-+ if (hparams.dsv4_hc_mult > 0) {
-+ return std::make_unique<graph_dsv4>(*this, params);
-+ }
-+ return std::make_unique<graph<false>>(*this, params);
-+ default:
-+ GGML_ABORT("invalid graph type");
-+ };
-+}
diff --git a/llama.cpp.patches/patches/src_models_eagle3.cpp.patch b/llama.cpp.patches/patches/src_models_eagle3.cpp.patch
deleted file mode 100644
index 35f204e1..00000000
--- a/llama.cpp.patches/patches/src_models_eagle3.cpp.patch
+++ /dev/null
@@ -1,41 +0,0 @@
-diff --git a/src/models/eagle3.cpp b/src/models/eagle3.cpp
---- a/llama.cpp/src/models/eagle3.cpp
-+++ b/llama.cpp/src/models/eagle3.cpp
-@@ -100,18 +100,6 @@ void llama_model_eagle3::load_arch_tensors(llama_model_loader &) {
- }
- }
-
--std::unique_ptr<llm_graph_context> llama_model_eagle3::build_arch_graph(const llm_graph_params & params) const {
-- switch (params.gtype) {
-- case LLM_GRAPH_TYPE_ENCODER:
-- return std::make_unique<graph<true>>(*this, params);
-- case LLM_GRAPH_TYPE_DEFAULT:
-- case LLM_GRAPH_TYPE_DECODER:
-- return std::make_unique<graph<false>>(*this, params);
-- default:
-- GGML_ABORT("invalid graph type");
-- };
--}
--
- template <>
- ggml_tensor * llama_model_eagle3::graph<true>::build_inp_embd_enc() const {
- ggml_tensor * cur = nullptr;
-@@ -336,3 +324,18 @@ llama_model_eagle3::graph<false>::graph(const llama_model & model, const llm_gra
-
- ggml_build_forward_expand(gf, cur);
- }
-+
-+// Defined after the graph<true>/graph<false> constructor specializations above:
-+// cosmocc's clang rejects an explicit specialization that appears after the
-+// point where make_unique<graph<...>> would implicitly instantiate it.
-+std::unique_ptr<llm_graph_context> llama_model_eagle3::build_arch_graph(const llm_graph_params & params) const {
-+ switch (params.gtype) {
-+ case LLM_GRAPH_TYPE_ENCODER:
-+ return std::make_unique<graph<true>>(*this, params);
-+ case LLM_GRAPH_TYPE_DEFAULT:
-+ case LLM_GRAPH_TYPE_DECODER:
-+ return std::make_unique<graph<false>>(*this, params);
-+ default:
-+ GGML_ABORT("invalid graph type");
-+ };
-+}
diff --git a/llama.cpp.patches/patches/src_models_t5.cpp.patch b/llama.cpp.patches/patches/src_models_t5.cpp.patch
index 40e2a98a..6bb5ae7d 100644
--- a/llama.cpp.patches/patches/src_models_t5.cpp.patch
+++ b/llama.cpp.patches/patches/src_models_t5.cpp.patch
@@ -1,8 +1,8 @@
diff --git a/src/models/t5.cpp b/src/models/t5.cpp
--- a/llama.cpp/src/models/t5.cpp
+++ b/llama.cpp/src/models/t5.cpp
-@@ -106,6 +106,12 @@ void llama_model_t5::load_arch_tensors(llama_model_loader &) {
- }
+@@ -357,6 +357,12 @@ llama_model_t5::graph<true>::graph(const llama_model & model, const llm_graph_pa
+ ggml_build_forward_expand(gf, cur);
}
+// Forward-declare the explicit specializations so that the std::make_unique
diff --git a/llama.cpp.patches/patches/tools_server_server-models.cpp.patch b/llama.cpp.patches/patches/tools_server_server-models.cpp.patch
index f9ef1320..ab633263 100644
--- a/llama.cpp.patches/patches/tools_server_server-models.cpp.patch
+++ b/llama.cpp.patches/patches/tools_server_server-models.cpp.patch
@@ -1,7 +1,7 @@
diff --git a/tools/server/server-models.cpp b/tools/server/server-models.cpp
--- a/llama.cpp/tools/server/server-models.cpp
+++ b/llama.cpp/tools/server/server-models.cpp
-@@ -30,6 +30,7 @@
+@@ -31,6 +31,7 @@
#include <cstring>
#ifndef _WIN32
@@ -9,7 +9,34 @@ diff --git a/tools/server/server-models.cpp b/tools/server/server-models.cpp
extern char **environ;
#endif
-@@ -675,13 +676,19 @@ void server_models::load_models() {
+@@ -61,7 +62,25 @@ static void request_child_exit(server_subproc & proc) {
+ // single-threaded, watching all child processes at once
+ struct server_monitor {
+ server_monitor(server_models & models) : models(models) {
+- th = std::thread([this]() { run(); });
++ th = std::thread([this]() {
++ // Block every signal on this thread so its waits can't be
++ // interrupted with EINTR (libcxx rethrows EINTR from a condition
++ // variable as std::system_error and std::terminates the process).
++ // Keep synchronous fault signals deliverable so real crashes still
++ // surface. SIGINT/SIGTERM belong to the main thread.
++ sigset_t block_mask;
++ sigfillset(&block_mask);
++ sigdelset(&block_mask, SIGSEGV);
++ sigdelset(&block_mask, SIGBUS);
++ sigdelset(&block_mask, SIGILL);
++ sigdelset(&block_mask, SIGFPE);
++ sigdelset(&block_mask, SIGABRT);
++ sigdelset(&block_mask, SIGSYS);
++ sigdelset(&block_mask, SIGTRAP);
++ pthread_sigmask(SIG_BLOCK, &block_mask, nullptr);
++
++ run();
++ });
+ }
+
+ ~server_monitor() {
+@@ -893,13 +912,19 @@ void server_models::load_models() {
}
// wait for all targeted models to reach UNLOADED; cv.wait handles unlock/relock
@@ -29,9 +56,9 @@ diff --git a/tools/server/server-models.cpp b/tools/server/server-models.cpp
+ cv.wait_for(lk, std::chrono::seconds(30));
+ }
- // collect all threads to join in one pass while the lock is held:
- // - monitoring threads from just-unloaded models (to_unload)
-@@ -913,9 +920,12 @@ void server_models::unload_lru() {
+ // erase models no longer in any source
+ for (auto it = mapping.begin(); it != mapping.end(); ) {
+@@ -1096,9 +1121,12 @@ void server_models::unload_lru() {
// wait for unload to complete
{
std::unique_lock<std::mutex> lk(mutex);
@@ -47,7 +74,7 @@ diff --git a/tools/server/server-models.cpp b/tools/server/server-models.cpp
}
}
}
-@@ -940,7 +950,12 @@ void server_models::load(const std::string & name, const load_options & opts) {
+@@ -1123,7 +1151,12 @@ void server_models::load(const std::string & name, const load_options & opts) {
std::unique_lock<std::mutex> lk(mutex);
// edge case: block until any in-progress reload has finished so we always load
// against the freshest preset and a consistent mapping state
@@ -61,45 +88,30 @@ diff --git a/tools/server/server-models.cpp b/tools/server/server-models.cpp
auto meta = opts.custom_meta.has_value() ? *opts.custom_meta : mapping[name].meta;
if (meta.status != SERVER_MODEL_STATUS_UNLOADED) {
-@@ -1037,15 +1052,34 @@ void server_models::load(const std::string & name, const load_options & opts) {
- });
-
- std::thread stopping_thread([&]() {
-+ // Block every signal on this thread so pthread_cond_timedwait()
-+ // can't be interrupted with EINTR (libcxx rethrows EINTR as
-+ // std::system_error and std::terminates the process). Keep
-+ // synchronous fault signals deliverable so real crashes still
-+ // surface.
-+ sigset_t block_mask;
-+ sigfillset(&block_mask);
-+ sigdelset(&block_mask, SIGSEGV);
-+ sigdelset(&block_mask, SIGBUS);
-+ sigdelset(&block_mask, SIGILL);
-+ sigdelset(&block_mask, SIGFPE);
-+ sigdelset(&block_mask, SIGABRT);
-+ sigdelset(&block_mask, SIGSYS);
-+ sigdelset(&block_mask, SIGTRAP);
-+ pthread_sigmask(SIG_BLOCK, &block_mask, nullptr);
-+
- // thread to monitor explicit stop requests; child crash is signalled via child_proc->stopped
- auto is_stopping = [this, &name]() {
- return this->stopping_models.find(name) != this->stopping_models.end();
- };
- {
- std::unique_lock<std::mutex> lk(this->mutex);
-- this->cv_stop.wait(lk, [&]() {
-- return is_stopping() || child_proc->stopped.load(std::memory_order_acquire);
-- });
-+ // Use wait_for() instead of wait() to work around a Cosmopolitan
-+ // libc bug where untimed futex waits on XNU (macOS) expire after
-+ // ~72 minutes, causing condition_variable::wait() to throw ETIMEDOUT.
-+ while (!(is_stopping() || child_proc->stopped.load(std::memory_order_acquire))) {
-+ this->cv_stop.wait_for(lk, std::chrono::seconds(30));
-+ }
+@@ -1283,14 +1316,20 @@ void server_models::unload_all() {
+ }
+ }
+ // wait for every child to exit, the monitor force-kills the ones that ignore the exit command
+- cv.wait(lk, [this]() {
++ // Use wait_for() instead of wait() to work around a Cosmopolitan libc bug
++ // where untimed futex waits on XNU (macOS) expire after ~72 minutes,
++ // causing condition_variable::wait() to throw ETIMEDOUT.
++ auto all_stopped = [this]() {
+ for (const auto & [name, inst] : mapping) {
+ if (inst.meta.is_running() || inst.meta.status == SERVER_MODEL_STATUS_DOWNLOADING) {
+ return false;
}
- // child crashed or finished on its own, skip graceful shutdown sequence
- if (child_proc->stopped.load(std::memory_order_acquire)) {
-@@ -1310,16 +1344,22 @@ void server_models::wait(const std::string & name, std::function<bool(const serv
+ }
+ return true;
+- });
++ };
++ while (!all_stopped()) {
++ cv.wait_for(lk, std::chrono::seconds(30));
++ }
+ }
+
+ void server_models::update_status(const std::string & name, const update_status_args & args) {
+@@ -1422,16 +1461,22 @@ void server_models::wait(const std::string & name, std::function<bool(const serv
}
void server_models::wait(std::unique_lock<std::mutex> & lk, const std::string & name, std::function<bool(const server_model_meta &)> predicate) {
@@ -129,7 +141,7 @@ diff --git a/tools/server/server-models.cpp b/tools/server/server-models.cpp
}
bool server_models::ensure_model_ready(const std::string & name, const std::function<bool()> & should_stop) {
-@@ -2347,10 +2387,10 @@ server_http_proxy::server_http_proxy(
+@@ -2448,10 +2493,10 @@ server_http_proxy::server_http_proxy(
auto pipe = std::make_shared<server_pipe<msg_t>>();
if (scheme == "https") {
diff --git a/llama.cpp.patches/patches/tools_server_server-queue.cpp.patch b/llama.cpp.patches/patches/tools_server_server-queue.cpp.patch
index 236bd692..80e0939b 100644
--- a/llama.cpp.patches/patches/tools_server_server-queue.cpp.patch
+++ b/llama.cpp.patches/patches/tools_server_server-queue.cpp.patch
@@ -1,8 +1,8 @@
diff --git a/tools/server/server-queue.cpp b/tools/server/server-queue.cpp
--- a/llama.cpp/tools/server/server-queue.cpp
+++ b/llama.cpp/tools/server/server-queue.cpp
-@@ -5,6 +5,13 @@
-
+@@ -6,6 +6,13 @@
+ #include <algorithm>
#include <chrono>
#include <thread>
+#include <cerrno>
@@ -15,7 +15,7 @@ diff --git a/tools/server/server-queue.cpp b/tools/server/server-queue.cpp
#define QUE_INF(fmt, ...) LOG_INF("que %12.*s: " fmt, 12, __func__, __VA_ARGS__)
#define QUE_WRN(fmt, ...) LOG_WRN("que %12.*s: " fmt, 12, __func__, __VA_ARGS__)
-@@ -111,9 +118,13 @@ void server_queue::wait_until_no_sleep() {
+@@ -123,9 +130,13 @@ void server_queue::wait_until_no_sleep() {
condition_tasks.notify_one(); // only main thread is waiting on this
}
QUE_DBG("%s", "waiting until no sleep\n");
@@ -32,24 +32,41 @@ diff --git a/tools/server/server-queue.cpp b/tools/server/server-queue.cpp
}
}
-@@ -153,9 +164,13 @@ void server_queue::worker_loop() {
- std::function<void()> work;
+@@ -165,9 +176,13 @@ void server_queue::worker_loop() {
{
std::unique_lock<std::mutex> lock(mutex_tasks);
+ // wait on busy instead of yielding - busy stays set even when the yield already ended
- worker.cv.wait(lock, [&]{
-- return worker.stop || worker.work != nullptr;
+- return worker.stop || worker.busy;
- });
+ // Use wait_for() instead of wait() to work around a
+ // Cosmopolitan libc bug where untimed futex waits on XNU
+ // (macOS) expire after ~72 minutes, causing
+ // condition_variable::wait() to throw ETIMEDOUT.
-+ while (!worker.stop && worker.work == nullptr) {
++ while (!worker.stop && !worker.busy) {
+ worker.cv.wait_for(lock, std::chrono::seconds(30));
+ }
if (worker.stop) {
return;
}
-@@ -192,6 +207,21 @@ void server_queue::worker_stop() {
+@@ -192,9 +207,13 @@ void server_queue::worker_loop() {
+ if (!queue_tasks.empty()) {
+ continue; // a new task arrived in the meantime
+ }
+- condition_tasks.wait(lock, [&]{
+- return worker.stop || !running || !worker.yielding || !queue_tasks.empty();
+- });
++ // Use wait_for() instead of wait() to work around a
++ // Cosmopolitan libc bug where untimed futex waits on XNU
++ // (macOS) expire after ~72 minutes, causing
++ // condition_variable::wait() to throw ETIMEDOUT.
++ while (!worker.stop && running && worker.yielding && queue_tasks.empty()) {
++ condition_tasks.wait_for(lock, std::chrono::seconds(30));
++ }
+ }
+
+ // signal to yield_to_queue() that no more tasks will be processed
+@@ -220,6 +239,21 @@ void server_queue::worker_stop() {
}
void server_queue::yield_to_queue(std::function<void()> && work) {
@@ -71,24 +88,24 @@ diff --git a/tools/server/server-queue.cpp b/tools/server/server-queue.cpp
GGML_ASSERT(worker.thread.joinable() && "yield_to_queue() requires start_loop() to be running");
QUE_DBG("%s", "yielding to queue\n");
-@@ -210,9 +240,13 @@ void server_queue::yield_to_queue(std::function<void()> && work) {
-
- std::unique_lock<std::mutex> lock(mutex_tasks);
- // declined tasks are moved to queue_tasks_unhandled, so a non-empty queue always has something new
+@@ -246,9 +280,13 @@ void server_queue::yield_to_queue(std::function<void()> && work) {
+ // the yield is over, wait for the worker to finish its current task
+ worker.yielding = false;
+ condition_tasks.notify_all();
- condition_tasks.wait(lock, [&]{
-- return !worker.busy || (running && !queue_tasks.empty());
+- return !worker.busy;
- });
+ // Use wait_for() instead of wait() to work around a
+ // Cosmopolitan libc bug where untimed futex waits on XNU
+ // (macOS) expire after ~72 minutes, causing
+ // condition_variable::wait() to throw ETIMEDOUT.
-+ while (worker.busy && !(running && !queue_tasks.empty())) {
++ while (worker.busy) {
+ condition_tasks.wait_for(lock, std::chrono::seconds(30));
+ }
- if (!worker.busy) {
- break;
- }
-@@ -244,6 +278,21 @@ void server_queue::yield_to_queue(std::function<void()> && work) {
+
+ // put the declined tasks back, keeping their order
+ while (!queue_tasks_unhandled.empty()) {
+@@ -276,6 +314,21 @@ void server_queue::yield_to_queue(std::function<void()> && work) {
}
void server_queue::start_loop(int64_t idle_sleep_ms) {
@@ -110,8 +127,8 @@ diff --git a/tools/server/server-queue.cpp b/tools/server/server-queue.cpp
running = true;
time_last_task = ggml_time_ms();
-@@ -293,9 +342,13 @@ void server_queue::start_loop(int64_t idle_sleep_ms) {
- callback_sleeping_state(true);
+@@ -333,9 +386,13 @@ void server_queue::start_loop(int64_t idle_sleep_ms) {
+ }
req_stop_sleeping = false;
// wait until we are requested to exit sleeping state
- condition_tasks.wait(lock, [&]{
@@ -127,7 +144,7 @@ diff --git a/tools/server/server-queue.cpp b/tools/server/server-queue.cpp
if (!running) { // may changed during sleep
break; // terminate
}
-@@ -384,13 +437,17 @@ void server_response::remove_waiting_task_ids(const std::unordered_set<int> & id
+@@ -427,13 +484,17 @@ void server_response::remove_waiting_task_ids(const std::unordered_set<int> & id
server_task_result_ptr server_response::recv(const std::unordered_set<int> & id_tasks) {
while (true) {
std::unique_lock<std::mutex> lock(mutex_results);
diff --git a/llama.cpp.patches/patches/tools_server_server.cpp.patch b/llama.cpp.patches/patches/tools_server_server.cpp.patch
index bdc752d9..e6b7e53d 100644
--- a/llama.cpp.patches/patches/tools_server_server.cpp.patch
+++ b/llama.cpp.patches/patches/tools_server_server.cpp.patch
@@ -70,7 +70,7 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp
common_init();
// start the stream session manager GC right after common init, before any HTTP route can
-@@ -109,10 +141,12 @@ int llama_server(int argc, char ** argv) {
+@@ -111,10 +143,12 @@ int llama_server(int argc, char ** argv) {
llama_backend_init();
llama_numa_init(params.numa);
@@ -85,7 +85,7 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp
bool is_run_by_cli = (argv == nullptr);
common_models_handler models_handler;
-@@ -435,6 +469,92 @@ int llama_server(common_params & params, int argc, char ** argv) {
+@@ -457,6 +491,92 @@ int llama_server(common_params & params, int argc, char ** argv) {
llama_backend_free();
};
@@ -178,7 +178,7 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp
// start the HTTP server before loading the model to be able to serve /health requests
if (!ctx_http.start()) {
clean_up();
-@@ -468,6 +588,13 @@ int llama_server(common_params & params, int argc, char ** argv) {
+@@ -490,6 +610,13 @@ int llama_server(common_params & params, int argc, char ** argv) {
// this will unblock start_loop()
ctx_server.terminate();
};
@@ -192,7 +192,7 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp
}
// register signal handler if not running by CLI
-@@ -509,6 +636,11 @@ int llama_server(common_params & params, int argc, char ** argv) {
+@@ -530,6 +657,11 @@ int llama_server(common_params & params, int argc, char ** argv) {
// when the HTTP server stops, clean up and exit
clean_up();
} else {
@@ -204,7 +204,7 @@ diff --git a/tools/server/server.cpp b/tools/server/server.cpp
// optionally, notify router server that this instance is ready
std::thread monitor_thread;
if (child.is_child()) {
-@@ -533,5 +665,56 @@ int llama_server(common_params & params, int argc, char ** argv) {
+@@ -554,5 +686,56 @@ int llama_server(common_params & params, int argc, char ** argv) {
}
}
diff --git a/llama.cpp.patches/patches/vendor_cpp-httplib_httplib.cpp.patch b/llama.cpp.patches/patches/vendor_cpp-httplib_httplib.cpp.patch
index e00d1b68..82f01dd9 100644
--- a/llama.cpp.patches/patches/vendor_cpp-httplib_httplib.cpp.patch
+++ b/llama.cpp.patches/patches/vendor_cpp-httplib_httplib.cpp.patch
@@ -1,7 +1,7 @@
diff --git a/vendor/cpp-httplib/httplib.cpp b/vendor/cpp-httplib/httplib.cpp
--- a/llama.cpp/vendor/cpp-httplib/httplib.cpp
+++ b/llama.cpp/vendor/cpp-httplib/httplib.cpp
-@@ -6847,6 +6847,22 @@ void ThreadPool::cleanup_finished_threads() {
+@@ -7244,6 +7244,22 @@ void ThreadPool::cleanup_finished_threads() {
}
void ThreadPool::worker(bool is_dynamic) {
@@ -24,7 +24,7 @@ diff --git a/vendor/cpp-httplib/httplib.cpp b/vendor/cpp-httplib/httplib.cpp
for (;;) {
std::function<void()> fn;
{
-@@ -6864,7 +6880,13 @@ void ThreadPool::worker(bool is_dynamic) {
+@@ -7261,7 +7277,13 @@ void ThreadPool::worker(bool is_dynamic) {
break;
}
} else {
@@ -39,7 +39,7 @@ diff --git a/vendor/cpp-httplib/httplib.cpp b/vendor/cpp-httplib/httplib.cpp
}
idle_thread_count_--;
-@@ -13151,6 +13173,13 @@ bool enumerate_macos_keychain_certs(Callback cb) {
+@@ -14003,6 +14025,13 @@ bool enumerate_macos_keychain_certs(Callback cb) {
#if !defined(_WIN32) && !(defined(__APPLE__) && \
defined(CPPHTTPLIB_USE_CERTS_FROM_MACOSX_KEYCHAIN))
@@ -53,7 +53,7 @@ diff --git a/vendor/cpp-httplib/httplib.cpp b/vendor/cpp-httplib/httplib.cpp
// Common CA certificate file paths on Linux/Unix
const char **system_ca_paths() {
static const char *paths[] = {
-@@ -13168,6 +13197,10 @@ const char **system_ca_dirs() {
+@@ -14020,6 +14049,10 @@ const char **system_ca_dirs() {
static const char *dirs[] = {"/etc/ssl/certs", // Debian/Ubuntu
"/etc/pki/tls/certs", // RHEL/CentOS
"/usr/share/ca-certificates", // Other
diff --git a/llama.cpp.patches/patches/vendor_sheredom_subprocess.h.patch b/llama.cpp.patches/patches/vendor_sheredom_subprocess.h.patch
index f8083354..12965efa 100644
--- a/llama.cpp.patches/patches/vendor_sheredom_subprocess.h.patch
+++ b/llama.cpp.patches/patches/vendor_sheredom_subprocess.h.patch
@@ -1,7 +1,7 @@
diff --git a/vendor/sheredom/subprocess.h b/vendor/sheredom/subprocess.h
--- a/llama.cpp/vendor/sheredom/subprocess.h
+++ b/llama.cpp/vendor/sheredom/subprocess.h
-@@ -562,6 +562,25 @@ int subprocess_error_from_windows_error(unsigned long error) {
+@@ -620,6 +620,25 @@ int subprocess_error_from_windows_error(unsigned long error) {
#else
subprocess_weak int subprocess_error_from_errno(int error);
int subprocess_error_from_errno(int error) {
@@ -27,7 +27,7 @@ diff --git a/vendor/sheredom/subprocess.h b/vendor/sheredom/subprocess.h
switch (error) {
case ENOENT:
case ENOTDIR:
-@@ -578,6 +597,7 @@ int subprocess_error_from_errno(int error) {
+@@ -636,6 +655,7 @@ int subprocess_error_from_errno(int error) {
default:
return subprocess_error_unknown;
}
diff --git a/llama.cpp.patches/ui-embed.sh b/llama.cpp.patches/ui-embed.sh
new file mode 100755
index 00000000..f03d65a5
--- /dev/null
+++ b/llama.cpp.patches/ui-embed.sh
@@ -0,0 +1,183 @@
+#!/usr/bin/env bash
+#
+# Generate ui.cpp / ui.h embedding the web UI assets as C arrays.
+#
+# Usage:
+# ui-embed.sh <out_cpp> <out_h> [<asset_dir>]
+#
+# This is a translation of emit_files() in llama.cpp/scripts/ui-assets.cmake,
+# the same way llamafile-files/BUILD.mk is a translation of llama.cpp's CMake
+# build: upstream generates ui.{cpp,h} with configure_file() at CMake time, and
+# the cosmocc build has no CMake step.
+#
+# It renders *upstream's own* templates, tools/ui/ui.{cpp,h}.in, so the
+# generated interface (llama_ui_find_asset / llama_ui_get_assets /
+# llama_ui_use_gzip and LLAMA_UI_HAS_ASSETS, all consumed by the unpatched
+# tools/server/server-http.cpp) tracks upstream automatically and cannot drift.
+# Only the template *substitutions* live here:
+#
+# @ASSET_ARRAYS@ one `static const unsigned char asset_N[] = {...};` per asset
+# @ASSET_TABLE@ one `{ name, asset_N, sizeof(asset_N), etag, mime },` per asset
+# @N_ASSETS@ asset count
+# @USE_GZIP@ whether the embedded bytes are gzip-compressed
+# #cmakedefine LLAMA_UI_HAS_ASSETS
+#
+# Assets come from <asset_dir> (fetch-ui-assets.sh populates it). When it holds
+# a _gzip/ mirror the compressed bytes are embedded instead, and server-http.cpp
+# serves them with Content-Encoding: gzip. Without <asset_dir> — or with no
+# index.html in it — this emits the no-asset stub and the server builds UI-less.
+
+set -euo pipefail
+
+if [ $# -lt 2 ] || [ $# -gt 3 ]; then
+ echo "usage: $0 <out_cpp> <out_h> [<asset_dir>]" >&2
+ exit 1
+fi
+
+OUT_CPP="$1"
+OUT_H="$2"
+ASSET_DIR="${3:-}"
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+TEMPLATE_DIR="$SCRIPT_DIR/../llama.cpp/tools/ui"
+TPL_CPP="$TEMPLATE_DIR/ui.cpp.in"
+TPL_H="$TEMPLATE_DIR/ui.h.in"
+
+for t in "$TPL_CPP" "$TPL_H"; do
+ if [ ! -f "$t" ]; then
+ echo "ui-embed: missing upstream template $t" >&2
+ exit 1
+ fi
+done
+
+# ETag source. Upstream uses SHA-256; cksum (POSIX, always present) is a
+# last-resort fallback — an ETag only has to be stable and change with content.
+if command -v shasum >/dev/null 2>&1; then
+ hash_file() { shasum -a 256 "$1" | cut -d' ' -f1; }
+elif command -v sha256sum >/dev/null 2>&1; then
+ hash_file() { sha256sum "$1" | cut -d' ' -f1; }
+else
+ hash_file() { cksum "$1" | cut -d' ' -f1,2 | tr ' ' '-'; }
+fi
+
+# Mirrors mime_from_ext() in scripts/ui-assets.cmake. An extension missing here
+# is served as application/octet-stream rather than breaking the build.
+mime_from_ext() {
+ case "${1##*.}" in
+ html) echo "text/html; charset=utf-8" ;;
+ css) echo "text/css" ;;
+ js) echo "application/javascript" ;;
+ json) echo "application/json" ;;
+ webmanifest) echo "application/manifest+json" ;;
+ svg) echo "image/svg+xml" ;;
+ png) echo "image/png" ;;
+ jpg|jpeg) echo "image/jpeg" ;;
+ ico) echo "image/x-icon" ;;
+ woff) echo "font/woff" ;;
+ woff2) echo "font/woff2" ;;
+ *) echo "application/octet-stream" ;;
+ esac
+}
+
+work="$(mktemp -d)"
+trap 'rm -rf "$work"' EXIT
+arrays="$work/arrays"
+table="$work/table"
+: > "$arrays"
+: > "$table"
+
+n_assets=0
+use_gzip=false
+
+if [ -n "$ASSET_DIR" ] && [ -f "$ASSET_DIR/index.html" ]; then
+ # Asset names are paths relative to the dist root; the gzip mirror repeats
+ # that layout under _gzip/, so names come from dist/ and bytes from whichever
+ # tree we embed.
+ embed_dir="$ASSET_DIR"
+ if [ -d "$ASSET_DIR/_gzip" ]; then
+ embed_dir="$ASSET_DIR/_gzip"
+ use_gzip=true
+ fi
+
+ # sort for reproducible output; directory order is unspecified
+ assets=()
+ while IFS= read -r f; do
+ assets+=("${f#./}")
+ done < <(cd "$ASSET_DIR" && find . -type f ! -path './_gzip/*' | LC_ALL=C sort)
+
+ idx=0
+ for name in "${assets[@]}"; do
+ src="$embed_dir/$name"
+ if [ ! -s "$src" ]; then
+ # A zero-length asset would emit an invalid empty C array. Upstream
+ # hard-errors; drop the whole UI instead so a corrupt tarball cannot
+ # abort a build, matching fetch-ui-assets.sh's fallback.
+ echo "ui-embed: empty asset '$name' - building without an embedded UI" >&2
+ : > "$arrays"
+ : > "$table"
+ idx=0
+ break
+ fi
+
+ # od is POSIX; 16 bytes per line keeps the generated source wrapped.
+ {
+ printf 'static const unsigned char asset_%d[] = {\n' "$idx"
+ od -An -v -tx1 "$src" \
+ | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' \
+ -e 's/\([0-9a-f][0-9a-f]\)/0x\1,/g' -e 's/[[:space:]]//g'
+ printf '};\n'
+ } >> "$arrays"
+
+ printf ' { "%s", asset_%d, sizeof(asset_%d), "\\"%s\\"", "%s" },\n' \
+ "$name" "$idx" "$idx" "$(hash_file "$src")" "$(mime_from_ext "$name")" \
+ >> "$table"
+
+ idx=$((idx + 1))
+ done
+ n_assets=$idx
+fi
+
+if [ "$n_assets" -eq 0 ]; then
+ use_gzip=false
+fi
+
+# Render a template: stream the bulk substitutions in from their own files
+# (they reach tens of MB, which no sed expression should carry) and expand the
+# scalar ones inline.
+render() {
+ local tpl="$1" out="$2"
+ awk -v n_assets="$n_assets" \
+ -v use_gzip="$use_gzip" \
+ -v has_assets="$([ "$n_assets" -gt 0 ] && echo 1 || echo 0)" \
+ -v arrays="$arrays" \
+ -v table="$table" '
+ /@ASSET_ARRAYS@/ { fflush(); system("cat " arrays); next }
+ /@ASSET_TABLE@/ { fflush(); system("cat " table); next }
+ {
+ if ($0 ~ /^#cmakedefine LLAMA_UI_HAS_ASSETS/) {
+ print (has_assets ? "#define LLAMA_UI_HAS_ASSETS 1" \
+ : "/* #undef LLAMA_UI_HAS_ASSETS */")
+ next
+ }
+ gsub(/@N_ASSETS@/, n_assets)
+ gsub(/@USE_GZIP@/, use_gzip)
+ print
+ }
+ ' "$tpl" > "$out.tmp"
+ # Only touch the output when it changes, so a re-fetch of identical assets
+ # does not force every dependent object to rebuild.
+ if [ -f "$out" ] && cmp -s "$out.tmp" "$out"; then
+ rm -f "$out.tmp"
+ else
+ mv -f "$out.tmp" "$out"
+ fi
+}
+
+render "$TPL_H" "$OUT_H"
+render "$TPL_CPP" "$OUT_CPP"
+
+if [ "$n_assets" -gt 0 ]; then
+ echo "ui-embed: embedded $n_assets assets$([ "$use_gzip" = true ] && echo ' (gzip)')"
+else
+ echo "ui-embed: no assets - building without an embedded UI"
+fi
diff --git a/llamafile/BUILD.mk b/llamafile/BUILD.mk
index 5a9d584b..3a0a2ef1 100644
--- a/llamafile/BUILD.mk
+++ b/llamafile/BUILD.mk
@@ -257,7 +257,8 @@ LLAMAFILE_SERVER_SUPPORT_OBJS := \
$(UI_GEN_OBJ)
# Metal source files to embed in the executable (for runtime compilation on macOS)
-# These are extracted at runtime and compiled into ggml-metal.dylib
+# These are extracted at runtime and compiled into ggml-metal.dylib; the
+# kernels/ shaders are compiled by the Metal runtime itself (see llamafile/metal.c).
LLAMAFILE_METAL_SOURCES := \
o/$(MODE)/llama.cpp/ggml/src/ggml.c.zip.o \
o/$(MODE)/llama.cpp/ggml/src/ggml-alloc.c.zip.o \
@@ -273,13 +274,13 @@ LLAMAFILE_METAL_SOURCES := \
o/$(MODE)/llama.cpp/ggml/include/ggml-cpp.h.zip.o \
o/$(MODE)/llama.cpp/ggml/include/ggml-metal.h.zip.o \
o/$(MODE)/llama.cpp/ggml/src/ggml-impl.h.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-version.h.zip.o \
o/$(MODE)/llama.cpp/ggml/src/ggml-common.h.zip.o \
o/$(MODE)/llama.cpp/ggml/src/ggml-quants.h.zip.o \
o/$(MODE)/llama.cpp/ggml/src/ggml-threading.h.zip.o \
o/$(MODE)/llama.cpp/ggml/src/ggml-backend-impl.h.zip.o \
o/$(MODE)/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-impl.h.zip.o \
o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal.cpp.zip.o \
- o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal.metal.zip.o \
o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-impl.h.zip.o \
o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-device.h.zip.o \
o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-device.m.zip.o \
@@ -289,7 +290,34 @@ LLAMAFILE_METAL_SOURCES := \
o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-common.h.zip.o \
o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-common.cpp.zip.o \
o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-ops.h.zip.o \
- o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-ops.cpp.zip.o
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-ops.cpp.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.h.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.cpp.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-tuning.h.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/ggml-metal-tuning.cpp.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/common.h.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/dequantize.h.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/quantize.h.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/argsort.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/binbcast.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/conv.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/fa.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/gated_delta_net.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/misc.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/mul_mm.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/mul_mv.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/norm.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/pool.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/quantize.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/reduce.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/rope.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/softmax.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/solve_tri.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/ssm.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/tri.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/unary.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/upscale.metal.zip.o \
+ o/$(MODE)/llama.cpp/ggml/src/ggml-metal/kernels/wkv.metal.zip.o
# Use deferred expansion (=) since this depends on variables from llama.cpp/BUILD.mk
LLAMAFILE_DEPS = \
@@ -344,15 +372,6 @@ o/$(MODE)/llamafile/llamafile: \
# Pattern rules for llamafile sources
# ==============================================================================
-# metal.c needs GGML_VERSION and GGML_COMMIT for runtime Metal compilation
-# GGML_VERSION and GGML_COMMIT are inherited from build/config.mk
-o/$(MODE)/llamafile/metal.o: llamafile/metal.c
- @mkdir -p $(@D)
- $(CC) $(CFLAGS) $(LLAMAFILE_CPPFLAGS) \
- -DGGML_VERSION=\"$(GGML_VERSION)\" \
- -DGGML_COMMIT=\"$(GGML_COMMIT)\" \
- -c -o $@ $<
-
o/$(MODE)/llamafile/%.o: llamafile/%.c
@mkdir -p $(@D)
$(CC) $(CFLAGS) $(LLAMAFILE_CPPFLAGS) -c -o $@ $<
diff --git a/llamafile/args.cpp b/llamafile/args.cpp
index 73c610f9..b8446ed5 100644
--- a/llamafile/args.cpp
+++ b/llamafile/args.cpp
@@ -19,6 +19,7 @@
#include "args.h"
#include "llamafile.h"
+#include <cstdio>
#include <cstring>
#include <vector>
@@ -41,6 +42,42 @@ static bool is_llamafile_flag(const char* arg) {
strcmp(arg, "--version") == 0;
}
+// llama.cpp b11100 replaced --mmap/--no-mmap/--mlock/-dio/-ndio with
+// --load-mode and now rejects them, which would break existing llamafiles
+// whose .args still use them. They are dropped here and one equivalent
+// "--load-mode <mode>" is inserted where the last of them was, so an
+// explicit --load-mode keeps working and the last flag wins, as upstream.
+//
+// The mode follows the original semantics, where the flags were independent
+// settings: mmap on by default, --mlock adds locking to it (mmap+mlock, not
+// plain mlock), and direct I/O takes precedence over mmap.
+static const char* legacy_load_flag(const char* arg, bool* mmap, bool* mlock, bool* dio) {
+ if (strcmp(arg, "--mmap") == 0) {
+ *mmap = true;
+ } else if (strcmp(arg, "--no-mmap") == 0) {
+ *mmap = false;
+ } else if (strcmp(arg, "--mlock") == 0) {
+ *mlock = true;
+ } else if (strcmp(arg, "-dio") == 0 || strcmp(arg, "--direct-io") == 0) {
+ *dio = true;
+ } else if (strcmp(arg, "-ndio") == 0 || strcmp(arg, "--no-direct-io") == 0) {
+ *dio = false;
+ } else {
+ return nullptr;
+ }
+ return arg;
+}
+
+static char* legacy_load_mode(bool mmap, bool mlock, bool dio) {
+ static char kNone[] = "none", kMmap[] = "mmap", kMlock[] = "mlock",
+ kMmapMlock[] = "mmap+mlock", kDio[] = "dio";
+ if (dio)
+ return kDio;
+ if (mmap)
+ return mlock ? kMmapMlock : kMmap;
+ return mlock ? kMlock : kNone;
+}
+
LlamafileArgs parse_llamafile_args(int argc, char** argv) {
LlamafileArgs args;
@@ -99,9 +136,20 @@ LlamafileArgs parse_llamafile_args(int argc, char** argv) {
// These are not recognized by llama.cpp and would cause errors
g_filtered_argv.clear();
+ bool legacy_mmap = true, legacy_mlock = false, legacy_dio = false;
+ const char* last_legacy = nullptr;
+ size_t legacy_pos = 0;
+
for (int i = 0; i < argc; ++i) {
const char* arg = argv[i];
+ // Translate removed load flags (see legacy_load_flag)
+ if (const char* flag = legacy_load_flag(arg, &legacy_mmap, &legacy_mlock, &legacy_dio)) {
+ last_legacy = flag;
+ legacy_pos = g_filtered_argv.size();
+ continue;
+ }
+
// Skip llamafile-specific flags
if (is_llamafile_flag(arg)) {
// --gpu takes a value argument, skip it too
@@ -115,6 +163,14 @@ LlamafileArgs parse_llamafile_args(int argc, char** argv) {
g_filtered_argv.push_back(argv[i]);
}
+ if (last_legacy) {
+ static char kLoadMode[] = "--load-mode";
+ char* mode = legacy_load_mode(legacy_mmap, legacy_mlock, legacy_dio);
+ fprintf(stderr, "warning: %s is no longer supported by llama.cpp; using --load-mode %s "
+ "(see docs/cli_arguments.md)\n", last_legacy, mode);
+ g_filtered_argv.insert(g_filtered_argv.begin() + legacy_pos, {kLoadMode, mode});
+ }
+
// Null-terminate argv array (required by convention)
g_filtered_argv.push_back(nullptr);
diff --git a/llamafile/build-functions.sh b/llamafile/build-functions.sh
index 9676e5c5..5dbbccb9 100644
--- a/llamafile/build-functions.sh
+++ b/llamafile/build-functions.sh
@@ -104,8 +104,17 @@ setup_build_dir() {
# $2 = caller-supplied sources prepended to the list (e.g., tinyblas.cu
# for the default TinyBLAS build; empty for the --cublas build)
# $3 = NO_IQ_QUANTS (optional, "1" to exclude IQ quant MMQ templates)
-# $4 = FA_ALL_QUANTS (optional, "1" to include all fattn-vec quant combos
-# instead of the 3 default ones; mirrors upstream's GGML_CUDA_FA_ALL_QUANTS)
+# $4 = FA_ALL_QUANTS (optional, "1" to compile every fattn-vec quant combo
+# instead of the default four; mirrors upstream's
+# GGML_CUDA_FA_QUANTS=all)
+# Sets: CUDA_SOURCES, NUM_SOURCES, CUDA_FA_DEFINES (append the last to the
+# compiler flags -- fattn.cu does not compile without it)
+# The K-V types fattn.cu knows about, and the combinations compiled by default.
+# Both mirror ggml/CMakeLists.txt's GGML_CUDA_FA_QUANTS default and the FA_TYPES
+# list in ggml/cmake/common.cmake -- keep in sync on a llama.cpp bump.
+FA_TYPES="q4_0 q4_1 q5_0 q5_1 q8_0 bf16 f16"
+FA_DEFAULT_COMBINATIONS="q4_0-q4_0 q8_0-q8_0 f16-f16 bf16-bf16"
+
collect_gpu_sources() {
local ggml_cuda_dir="$1"
local caller_sources="$2"
@@ -126,23 +135,55 @@ collect_gpu_sources() {
[ -f "$f" ] && CUDA_SOURCES="$CUDA_SOURCES $f"
done
- # 3. fattn-vec: default to the 4 common quant combos (f16-f16, q4_0-q4_0,
- # q8_0-q8_0, bf16-bf16), matching upstream CMake. With FA_ALL_QUANTS=1
- # include all fattn-vec instances (mirrors upstream's
- # GGML_CUDA_FA_ALL_QUANTS opt-in).
+ # 3. fattn-vec: since b11100 fattn.cu gates every K-V combination on a
+ # GGML_CUDA_FA_<K>_<V> macro that must be defined to 0 or 1 for *all* of
+ # them (upstream's ggml_cuda_fattn_vec_instances() in
+ # ggml/cmake/common.cmake emits them). Picking source files is no longer
+ # enough: an undefined macro is a hard `identifier is undefined` error in
+ # the `if constexpr`. So derive both the sources and the macros from one
+ # list of combinations.
+ local fa_combinations
if [ "$fa_all_quants" = "1" ]; then
- for f in "$ti_dir"/fattn-vec-instance-*.cu; do
- [ -f "$f" ] && CUDA_SOURCES="$CUDA_SOURCES $f"
+ fa_combinations=""
+ for type_v in $FA_TYPES; do
+ for type_k in $FA_TYPES; do
+ fa_combinations="$fa_combinations $type_k-$type_v"
+ done
done
else
- for f in "$ti_dir"/fattn-vec-instance-f16-f16.cu \
- "$ti_dir"/fattn-vec-instance-q4_0-q4_0.cu \
- "$ti_dir"/fattn-vec-instance-q8_0-q8_0.cu \
- "$ti_dir"/fattn-vec-instance-bf16-bf16.cu; do
- [ -f "$f" ] && CUDA_SOURCES="$CUDA_SOURCES $f"
- done
+ fa_combinations="$FA_DEFAULT_COMBINATIONS"
fi
+ for combination in $fa_combinations; do
+ f="$ti_dir/fattn-vec-instance-$combination.cu"
+ if [ ! -f "$f" ]; then
+ echo "Error: FlashAttention template instance not found: $f" >&2
+ return 1
+ fi
+ CUDA_SOURCES="$CUDA_SOURCES $f"
+ done
+
+ # One -D per combination, 0 or 1, for every type pair fattn.cu can ask about.
+ # No spaces in any value, so these survive the unquoted word-splitting that
+ # compile_gpu_sources_parallel does on its common_flags argument.
+ CUDA_FA_DEFINES=""
+ for type_v in $FA_TYPES; do
+ for type_k in $FA_TYPES; do
+ local compiled=0
+ case " $fa_combinations " in
+ *" $type_k-$type_v "*) compiled=1 ;;
+ esac
+ local macro
+ macro=$(echo "GGML_CUDA_FA_${type_k}_${type_v}" | tr '[:lower:]' '[:upper:]')
+ CUDA_FA_DEFINES="$CUDA_FA_DEFINES -D$macro=$compiled"
+ done
+ done
+ # Reported by ggml_backend_cuda_get_features as "FA_QUANTS" (guarded by
+ # #ifdef, so a build without it just omits the feature).
+ local fa_quants_csv
+ fa_quants_csv=$(echo $fa_combinations | tr ' ' ',')
+ CUDA_FA_DEFINES="$CUDA_FA_DEFINES -DGGML_CUDA_FA_QUANTS=\"$fa_quants_csv\""
+
# 4. mmf instances (always included)
for f in "$ti_dir"/mmf-*.cu; do
[ -f "$f" ] && CUDA_SOURCES="$CUDA_SOURCES $f"
diff --git a/llamafile/chatbot_cli.cpp b/llamafile/chatbot_cli.cpp
index bb556229..44ec6bfc 100644
--- a/llamafile/chatbot_cli.cpp
+++ b/llamafile/chatbot_cli.cpp
@@ -241,9 +241,11 @@ int cli_main(int argc, char **argv) {
// Load image bitmaps
for (const auto &image_path : params.image) {
- // init_from_file now returns a wrapper {bitmap, video_ctx}; chatbot
- // is image-only, so take the bitmap (placeholder=false loads pixels).
- mtmd::bitmap bmp(mtmd_helper_bitmap_init_from_file(mtmd_ctx, image_path.c_str(), false).bitmap);
+ // init_from_file returns a wrapper {bitmap, video_ctx}; chatbot is
+ // image-only, so take the bitmap (placeholder=false loads pixels)
+ // and the default opt (its video params go unused).
+ mtmd::bitmap bmp(mtmd_helper_bitmap_init_from_file(
+ mtmd_ctx, image_path.c_str(), false, mtmd_helper_init_opt_default()).bitmap);
if (!bmp.ptr) {
fprintf(stderr, "error: failed to load image: %s\n", image_path.c_str());
cleanup(mtmd_ctx, sampler, ctx, model);
diff --git a/llamafile/chatbot_eval.cpp b/llamafile/chatbot_eval.cpp
index 30af677d..a129af3d 100644
--- a/llamafile/chatbot_eval.cpp
+++ b/llamafile/chatbot_eval.cpp
@@ -167,10 +167,12 @@ bool eval_string(std::string_view s, bool add_special, bool parse_special) {
// Create bitmaps from decoded image data
mtmd::bitmaps bitmaps;
for (const auto &image : extraction.images) {
- // init_from_buf now returns a wrapper {bitmap, video_ctx}; chatbot is
- // image-only, so take the bitmap (placeholder=false loads real pixels).
+ // init_from_buf returns a wrapper {bitmap, video_ctx}; chatbot is
+ // image-only, so take the bitmap (placeholder=false loads real pixels)
+ // and the default opt (its video params go unused).
mtmd::bitmap bmp(mtmd_helper_bitmap_init_from_buf(
- g_mtmd, (const unsigned char *)image.data(), image.size(), false).bitmap);
+ g_mtmd, (const unsigned char *)image.data(), image.size(), false,
+ mtmd_helper_init_opt_default()).bitmap);
if (!bmp.ptr) {
err("failed to load image");
return false;
diff --git a/llamafile/cuda.bat b/llamafile/cuda.bat
index c7364109..195be5fe 100644
--- a/llamafile/cuda.bat
+++ b/llamafile/cuda.bat
@@ -191,7 +191,27 @@ set "COMMON_FLAGS=%COMMON_FLAGS% -DNDEBUG -DGGML_BUILD=1 -DGGML_SHARED=1 -DGGML_
set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_CUDA_USE_GRAPHS"
set "COMMON_FLAGS=%COMMON_FLAGS% %BLAS_DEFINE%"
if "%NO_IQ_QUANTS%"=="1" set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_CUDA_NO_IQ_QUANTS"
-if "%FA_ALL_QUANTS%"=="1" set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_CUDA_FA_ALL_QUANTS"
+
+:: fattn.cu gates every K-V combination on a GGML_CUDA_FA_<K>_<V> macro that
+:: must be defined to 0 or 1 for all of them (since b11100; upstream emits them
+:: from ggml_cuda_fattn_vec_instances() in ggml/cmake/common.cmake). Selecting
+:: template-instance files is no longer enough -- an undefined macro is a hard
+:: error in the if constexpr. Keep this list and the default combinations in
+:: sync with FA_TYPES / FA_DEFAULT_COMBINATIONS in llamafile/build-functions.sh.
+:: (The informational GGML_CUDA_FA_QUANTS string macro is #ifdef-guarded and is
+:: not set here, to avoid quoting it through cmd.)
+set "FA_TYPES=Q4_0 Q4_1 Q5_0 Q5_1 Q8_0 BF16 F16"
+if "%FA_ALL_QUANTS%"=="1" (
+ set "FA_COMBINATIONS="
+ for %%v in (%FA_TYPES%) do for %%k in (%FA_TYPES%) do call set "FA_COMBINATIONS=%%FA_COMBINATIONS%% %%k-%%v"
+) else (
+ set "FA_COMBINATIONS=Q4_0-Q4_0 Q8_0-Q8_0 F16-F16 BF16-BF16"
+)
+for %%v in (%FA_TYPES%) do for %%k in (%FA_TYPES%) do (
+ set "FA_COMPILED=0"
+ for %%c in (!FA_COMBINATIONS!) do if /i "%%c"=="%%k-%%v" set "FA_COMPILED=1"
+ set "COMMON_FLAGS=!COMMON_FLAGS! -DGGML_CUDA_FA_%%k_%%v=!FA_COMPILED!"
+)
:: -------- extract GGML version --------
set "GGML_VERSION=unknown"
diff --git a/llamafile/cuda.sh b/llamafile/cuda.sh
index ff35f1c5..80485096 100755
--- a/llamafile/cuda.sh
+++ b/llamafile/cuda.sh
@@ -251,12 +251,14 @@ COMMON_FLAGS="\
if [ "$NO_IQ_QUANTS" = "1" ]; then
COMMON_FLAGS="$COMMON_FLAGS -DGGML_CUDA_NO_IQ_QUANTS"
fi
-if [ "$FA_ALL_QUANTS" = "1" ]; then
- COMMON_FLAGS="$COMMON_FLAGS -DGGML_CUDA_FA_ALL_QUANTS"
-fi
# Collect sources
collect_gpu_sources "$GGML_CUDA_DIR" "$EXTRA_SOURCES" "$NO_IQ_QUANTS" "$FA_ALL_QUANTS"
+
+# fattn.cu gates each K-V combination on a GGML_CUDA_FA_<K>_<V> macro that must
+# be defined for all of them; collect_gpu_sources derives them from the same
+# combination list it picked the template instances from.
+COMMON_FLAGS="$COMMON_FLAGS $CUDA_FA_DEFINES"
echo " Sources: $NUM_SOURCES .cu files"
echo ""
diff --git a/llamafile/cuda_parallel.bat b/llamafile/cuda_parallel.bat
index dbf17440..509e8f28 100644
--- a/llamafile/cuda_parallel.bat
+++ b/llamafile/cuda_parallel.bat
@@ -214,7 +214,27 @@ set "COMMON_FLAGS=%COMMON_FLAGS% -DNDEBUG -DGGML_BUILD=1 -DGGML_SHARED=1 -DGGML_
set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_CUDA_USE_GRAPHS"
set "COMMON_FLAGS=%COMMON_FLAGS% %BLAS_DEFINE%"
if "%NO_IQ_QUANTS%"=="1" set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_CUDA_NO_IQ_QUANTS"
-if "%FA_ALL_QUANTS%"=="1" set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_CUDA_FA_ALL_QUANTS"
+
+:: fattn.cu gates every K-V combination on a GGML_CUDA_FA_<K>_<V> macro that
+:: must be defined to 0 or 1 for all of them (since b11100; upstream emits them
+:: from ggml_cuda_fattn_vec_instances() in ggml/cmake/common.cmake). Selecting
+:: template-instance files is no longer enough -- an undefined macro is a hard
+:: error in the if constexpr. Keep this list and the default combinations in
+:: sync with FA_TYPES / FA_DEFAULT_COMBINATIONS in llamafile/build-functions.sh.
+:: (The informational GGML_CUDA_FA_QUANTS string macro is #ifdef-guarded and is
+:: not set here, to avoid quoting it through cmd.)
+set "FA_TYPES=Q4_0 Q4_1 Q5_0 Q5_1 Q8_0 BF16 F16"
+if "%FA_ALL_QUANTS%"=="1" (
+ set "FA_COMBINATIONS="
+ for %%v in (%FA_TYPES%) do for %%k in (%FA_TYPES%) do call set "FA_COMBINATIONS=%%FA_COMBINATIONS%% %%k-%%v"
+) else (
+ set "FA_COMBINATIONS=Q4_0-Q4_0 Q8_0-Q8_0 F16-F16 BF16-BF16"
+)
+for %%v in (%FA_TYPES%) do for %%k in (%FA_TYPES%) do (
+ set "FA_COMPILED=0"
+ for %%c in (!FA_COMBINATIONS!) do if /i "%%c"=="%%k-%%v" set "FA_COMPILED=1"
+ set "COMMON_FLAGS=!COMMON_FLAGS! -DGGML_CUDA_FA_%%k_%%v=!FA_COMPILED!"
+)
:: -------- extract GGML version --------
set "GGML_VERSION=unknown"
diff --git a/llamafile/metal.c b/llamafile/metal.c
index 933ddaf0..e09800aa 100644
--- a/llamafile/metal.c
+++ b/llamafile/metal.c
@@ -22,9 +22,14 @@
// This file implements dynamic compilation and loading of Metal GPU support.
// At runtime on macOS ARM64:
// 1. Extract Metal source files from /zip/ to ~/.llamafile/
-// 2. Preprocess ggml-metal.metal to inline header files
-// 3. Compile everything into a self-contained dylib using system cc
-// 4. Load the dylib with cosmo_dlopen() and register the Metal backend
+// 2. Compile everything into a self-contained dylib using system cc
+// 3. Load the dylib with cosmo_dlopen() and register the Metal backend
+//
+// The shader sources (kernels/*.metal) are compiled by the Metal runtime
+// itself, not by us: the backend is built without GGML_METAL_EMBED_LIBRARY,
+// so ggml_metal_library_init() reads kernels/<kind>.metal, flattens their
+// #includes and hands the result to newLibraryWithSource. It looks them up
+// under $GGML_METAL_PATH_RESOURCES, which we point at the app dir.
//
// The dylib is self-contained (includes ggml core) because cosmo_dlopen()
// cannot resolve symbols from the parent process.
@@ -69,6 +74,7 @@ __static_yoink("llama.cpp/ggml/include/ggml-backend.h");
__static_yoink("llama.cpp/ggml/include/ggml-cpp.h");
__static_yoink("llama.cpp/ggml/include/ggml-metal.h");
__static_yoink("llama.cpp/ggml/src/ggml-impl.h");
+__static_yoink("llama.cpp/ggml/src/ggml-version.h");
__static_yoink("llama.cpp/ggml/src/ggml-common.h");
__static_yoink("llama.cpp/ggml/src/ggml-quants.h");
__static_yoink("llama.cpp/ggml/src/ggml-threading.h");
@@ -77,7 +83,6 @@ __static_yoink("llama.cpp/ggml/src/ggml-cpu/ggml-cpu-impl.h");
// Metal backend
__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal.cpp");
-__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal.metal");
__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-impl.h");
__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-device.h");
__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-device.m");
@@ -88,6 +93,37 @@ __static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-common.h");
__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-common.cpp");
__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-ops.h");
__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-ops.cpp");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.h");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.cpp");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-tuning.h");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/ggml-metal-tuning.cpp");
+
+// Metal shader sources, compiled by the Metal runtime at first use. Since
+// b11100 upstream ships one .metal per op family under kernels/ instead of a
+// single ggml-metal.metal.
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/common.h");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/dequantize.h");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/quantize.h");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/argsort.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/binbcast.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/conv.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/fa.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/gated_delta_net.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/misc.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/mul_mm.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/mul_mv.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/norm.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/pool.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/quantize.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/reduce.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/rope.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/softmax.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/solve_tri.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/ssm.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/tri.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/unary.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/upscale.metal");
+__static_yoink("llama.cpp/ggml/src/ggml-metal/kernels/wkv.metal");
// Sources to extract at runtime
static const struct MetalSource {
@@ -103,6 +139,8 @@ static const struct MetalSource {
{"/zip/llama.cpp/ggml/include/ggml-cpp.h", "ggml-cpp.h"},
{"/zip/llama.cpp/ggml/include/ggml-metal.h", "ggml-metal.h"},
{"/zip/llama.cpp/ggml/src/ggml-impl.h", "ggml-impl.h"},
+ // generated by llama.cpp.patches/apply-patches.sh; ggml.c includes it
+ {"/zip/llama.cpp/ggml/src/ggml-version.h", "ggml-version.h"},
{"/zip/llama.cpp/ggml/src/ggml-common.h", "ggml-common.h"},
{"/zip/llama.cpp/ggml/src/ggml-quants.h", "ggml-quants.h"},
{"/zip/llama.cpp/ggml/src/ggml-threading.h", "ggml-threading.h"},
@@ -119,7 +157,6 @@ static const struct MetalSource {
// Metal-specific files
{"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal.cpp", "ggml-metal.cpp"},
- {"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal.metal", "ggml-metal.metal"},
{"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-impl.h", "ggml-metal-impl.h"},
{"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-device.h", "ggml-metal-device.h"},
{"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-device.m", "ggml-metal-device.m"},
@@ -130,6 +167,37 @@ static const struct MetalSource {
{"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-common.cpp", "ggml-metal-common.cpp"},
{"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-ops.h", "ggml-metal-ops.h"},
{"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-ops.cpp", "ggml-metal-ops.cpp"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.h", "ggml-metal-fusion.h"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-fusion.cpp", "ggml-metal-fusion.cpp"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-tuning.h", "ggml-metal-tuning.h"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/ggml-metal-tuning.cpp", "ggml-metal-tuning.cpp"},
+
+ // Shader sources. ggml_metal_library_flatten_source() resolves the quoted
+ // includes in these against kernels/ and its parent, so ggml-common.h and
+ // ggml-metal-impl.h above are reachable from them once extracted.
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/common.h", "kernels/common.h"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/dequantize.h", "kernels/dequantize.h"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/quantize.h", "kernels/quantize.h"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/argsort.metal", "kernels/argsort.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/binbcast.metal", "kernels/binbcast.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/conv.metal", "kernels/conv.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/fa.metal", "kernels/fa.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/gated_delta_net.metal", "kernels/gated_delta_net.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/misc.metal", "kernels/misc.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/mul_mm.metal", "kernels/mul_mm.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/mul_mv.metal", "kernels/mul_mv.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/norm.metal", "kernels/norm.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/pool.metal", "kernels/pool.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/quantize.metal", "kernels/quantize.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/reduce.metal", "kernels/reduce.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/rope.metal", "kernels/rope.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/softmax.metal", "kernels/softmax.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/solve_tri.metal", "kernels/solve_tri.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/ssm.metal", "kernels/ssm.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/tri.metal", "kernels/tri.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/unary.metal", "kernels/unary.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/upscale.metal", "kernels/upscale.metal"},
+ {"/zip/llama.cpp/ggml/src/ggml-metal/kernels/wkv.metal", "kernels/wkv.metal"},
};
// Forward declarations for ggml backend types
@@ -164,129 +232,6 @@ static struct MetalBackend {
void (*log_set)(llamafile_log_callback log_callback, void *user_data);
} g_metal;
-static char *read_file(const char *path, size_t *size_out) {
- FILE *f = fopen(path, "r");
- if (!f) return NULL;
-
- fseek(f, 0, SEEK_END);
- long size = ftell(f);
- fseek(f, 0, SEEK_SET);
-
- char *content = malloc(size + 1);
- if (!content) {
- fclose(f);
- return NULL;
- }
-
- fread(content, 1, size, f);
- content[size] = '\0';
- fclose(f);
-
- if (size_out) *size_out = size;
- return content;
-}
-
-static bool write_file(const char *path, const char *content, size_t size) {
- FILE *f = fopen(path, "w");
- if (!f) return false;
- fwrite(content, 1, size, f);
- fclose(f);
- return true;
-}
-
-// Preprocess ggml-metal.metal to inline headers
-// Metal runtime compiler doesn't support include paths
-static bool PreprocessMetalShader(const char *app_dir) {
- char metal_path[PATH_MAX];
- char common_path[PATH_MAX];
- char impl_path[PATH_MAX];
-
- snprintf(metal_path, PATH_MAX, "%sggml-metal.metal", app_dir);
- snprintf(common_path, PATH_MAX, "%sggml-common.h", app_dir);
- snprintf(impl_path, PATH_MAX, "%sggml-metal-impl.h", app_dir);
-
- size_t common_size, impl_size, metal_size;
- char *common_content = read_file(common_path, &common_size);
- if (!common_content) {
- fprintf(stderr, "metal: failed to read %s\n", common_path);
- return false;
- }
-
- char *impl_content = read_file(impl_path, &impl_size);
- if (!impl_content) {
- free(common_content);
- fprintf(stderr, "metal: failed to read %s\n", impl_path);
- return false;
- }
-
- char *metal_content = read_file(metal_path, &metal_size);
- if (!metal_content) {
- free(common_content);
- free(impl_content);
- fprintf(stderr, "metal: failed to read %s\n", metal_path);
- return false;
- }
-
- // Find #include directives
- char *include_common = strstr(metal_content, "#include \"ggml-common.h\"");
- char *include_impl = strstr(metal_content, "#include \"ggml-metal-impl.h\"");
-
- if (!include_common && !include_impl) {
- // Nothing to preprocess
- free(common_content);
- free(impl_content);
- free(metal_content);
- return true;
- }
-
- // Create preprocessed content
- FILE *fout = fopen(metal_path, "w");
- if (!fout) {
- free(common_content);
- free(impl_content);
- free(metal_content);
- fprintf(stderr, "metal: failed to write to %s\n", metal_path);
- return false;
- }
-
- char *pos = metal_content;
-
- // Process includes in order they appear
- if (include_common && (!include_impl || include_common < include_impl)) {
- // Write everything before #include "ggml-common.h"
- fwrite(pos, 1, include_common - pos, fout);
- fputs("// ggml-common.h inlined below\n", fout);
- fwrite(common_content, 1, common_size, fout);
- fputs("\n// end of ggml-common.h\n", fout);
- pos = strchr(include_common, '\n');
- if (pos) pos++;
- }
-
- if (include_impl && pos && include_impl >= pos) {
- // Write everything between includes
- fwrite(pos, 1, include_impl - pos, fout);
- fputs("// ggml-metal-impl.h inlined below\n", fout);
- fwrite(impl_content, 1, impl_size, fout);
- fputs("\n// end of ggml-metal-impl.h\n", fout);
- pos = strchr(include_impl, '\n');
- if (pos) pos++;
- }
-
- // Write rest of file
- if (pos) {
- fputs(pos, fout);
- }
-
- fclose(fout);
- free(common_content);
- free(impl_content);
- free(metal_content);
-
- llamafile_info("metal", "preprocessed %s", metal_path);
-
- return true;
-}
-
static bool BuildMetal(const char *dso) {
char app_dir[PATH_MAX];
char src[PATH_MAX];
@@ -338,13 +283,6 @@ static bool BuildMetal(const char *dso) {
}
}
- // Preprocess Metal shader if rebuild needed
- if (needs_rebuild) {
- if (!PreprocessMetalShader(app_dir)) {
- return false;
- }
- }
-
// Check if dylib needs rebuild
snprintf(src, PATH_MAX, "%sggml-metal.cpp", app_dir);
if (!needs_rebuild) {
@@ -391,6 +329,8 @@ static bool BuildMetal(const char *dso) {
"ggml-metal-device.cpp",
"ggml-metal-common.cpp",
"ggml-metal-ops.cpp",
+ "ggml-metal-fusion.cpp",
+ "ggml-metal-tuning.cpp",
"ggml-metal-device.m",
"ggml-metal-context.m",
NULL
@@ -430,8 +370,6 @@ static bool BuildMetal(const char *dso) {
args[argc++] = "-ffixed-x28"; // cosmo's TLS register
args[argc++] = "-DTARGET_OS_OSX";
args[argc++] = "-DGGML_MULTIPLATFORM";
- args[argc++] = "-DGGML_VERSION=\"" GGML_VERSION "\"";
- args[argc++] = "-DGGML_COMMIT=\"" GGML_COMMIT "\"";
args[argc++] = "-w"; // Suppress compilation warnings
args[argc++] = "-o";
args[argc++] = obj_paths[i];
@@ -609,6 +547,22 @@ static bool ImportMetalImpl(void) {
llamafile_get_app_dir(app_dir, PATH_MAX);
snprintf(dso, PATH_MAX, "%sggml-metal.dylib", app_dir);
+ // The dylib is built without GGML_METAL_EMBED_LIBRARY, so
+ // ggml_metal_library_init() compiles kernels/<kind>.metal at first use.
+ // It finds them through [NSBundle bundleForClass:], which for a loose
+ // dylib resolves to the dylib's own directory, i.e. app_dir, where
+ // BuildMetal() extracts them. GGML_METAL_PATH_RESOURCES takes precedence
+ // over that lookup, and it can't be overridden from here: setenv() only
+ // changes Cosmopolitan's environ, which libSystem inside the dylib never
+ // reads. So if it is set, kernels come from elsewhere; say so.
+ const char *res_path = getenv("GGML_METAL_PATH_RESOURCES");
+ if (res_path && *res_path) {
+ fprintf(stderr,
+ "metal: warning: GGML_METAL_PATH_RESOURCES=%s overrides the kernels "
+ "extracted to %s; unset it unless those kernels match this llamafile\n",
+ res_path, app_dir);
+ }
+
if (FLAG_nocompile) {
return LinkMetal(dso);
}
diff --git a/llamafile/release.sh b/llamafile/release.sh
index 18c651e3..b292dd34 100644
--- a/llamafile/release.sh
+++ b/llamafile/release.sh
@@ -1,5 +1,5 @@
# update LLAMAFILE_VERSION for a new release
-LLAMAFILE_VERSION="0.10.6"
+LLAMAFILE_VERSION="0.10.7"
# this is where the precompiled GPU libraries are stored
GPU_LIBS_DIR="/home/mala/gpulibs/${LLAMAFILE_VERSION}"
diff --git a/llamafile/rocm.bat b/llamafile/rocm.bat
index 6101c063..72b1eb24 100644
--- a/llamafile/rocm.bat
+++ b/llamafile/rocm.bat
@@ -134,6 +134,22 @@ set "COMMON_FLAGS=%COMMON_FLAGS% -I"%HIP_PATH%\include""
set "COMMON_FLAGS=%COMMON_FLAGS% -DNDEBUG -DGGML_BUILD=1 -DGGML_SHARED=1 -DGGML_BACKEND_SHARED=1 -DGGML_BACKEND_BUILD=1 -DGGML_MULTIPLATFORM"
set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_HIP_GRAPHS=1"
set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_USE_HIP=1 -DGGML_USE_TINYBLAS=1 -DGGML_HIP_NO_VMM=1 -D__HIP_PLATFORM_AMD__"
+
+:: fattn.cu gates every K-V combination on a GGML_CUDA_FA_<K>_<V> macro that
+:: must be defined to 0 or 1 for all of them (since b11100; upstream emits them
+:: from ggml_cuda_fattn_vec_instances() in ggml/cmake/common.cmake). Selecting
+:: template-instance files is no longer enough -- an undefined macro is a hard
+:: error in the if constexpr. Keep this list and the default combinations in
+:: sync with FA_TYPES / FA_DEFAULT_COMBINATIONS in llamafile/build-functions.sh.
+:: (The informational GGML_CUDA_FA_QUANTS string macro is #ifdef-guarded and is
+:: not set here, to avoid quoting it through cmd.)
+set "FA_TYPES=Q4_0 Q4_1 Q5_0 Q5_1 Q8_0 BF16 F16"
+set "FA_COMBINATIONS=Q4_0-Q4_0 Q8_0-Q8_0 F16-F16 BF16-BF16"
+for %%v in (%FA_TYPES%) do for %%k in (%FA_TYPES%) do (
+ set "FA_COMPILED=0"
+ for %%c in (!FA_COMBINATIONS!) do if /i "%%c"=="%%k-%%v" set "FA_COMPILED=1"
+ set "COMMON_FLAGS=!COMMON_FLAGS! -DGGML_CUDA_FA_%%k_%%v=!FA_COMPILED!"
+)
:: --offload-compress shrinks the fat binary (mirrors rocm.sh / #995); requires a
:: recent ROCm HIP SDK -- drop this flag if your clang++ rejects it
set "COMMON_FLAGS=%COMMON_FLAGS% --offload-compress"
diff --git a/llamafile/rocm.sh b/llamafile/rocm.sh
index 77740375..2efa48cc 100755
--- a/llamafile/rocm.sh
+++ b/llamafile/rocm.sh
@@ -130,6 +130,11 @@ COMMON_FLAGS="\
# Collect sources (TinyBLAS + GGML CUDA)
collect_gpu_sources "$GGML_CUDA_DIR" "$BUILD_DIR/tinyblas.cu"
+
+# fattn.cu gates each K-V combination on a GGML_CUDA_FA_<K>_<V> macro that must
+# be defined for all of them; collect_gpu_sources derives them from the same
+# combination list it picked the template instances from.
+COMMON_FLAGS="$COMMON_FLAGS $CUDA_FA_DEFINES"
echo " Sources: $NUM_SOURCES .cu files"
echo ""
diff --git a/llamafile/rocm_parallel.bat b/llamafile/rocm_parallel.bat
index 3c9c855b..23bd6577 100644
--- a/llamafile/rocm_parallel.bat
+++ b/llamafile/rocm_parallel.bat
@@ -156,6 +156,22 @@ set "COMMON_FLAGS=%COMMON_FLAGS% -I"%HIP_PATH%\include""
set "COMMON_FLAGS=%COMMON_FLAGS% -DNDEBUG -DGGML_BUILD=1 -DGGML_SHARED=1 -DGGML_BACKEND_SHARED=1 -DGGML_BACKEND_BUILD=1 -DGGML_MULTIPLATFORM"
set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_HIP_GRAPHS=1"
set "COMMON_FLAGS=%COMMON_FLAGS% -DGGML_USE_HIP=1 -DGGML_USE_TINYBLAS=1 -DGGML_HIP_NO_VMM=1 -D__HIP_PLATFORM_AMD__"
+
+:: fattn.cu gates every K-V combination on a GGML_CUDA_FA_<K>_<V> macro that
+:: must be defined to 0 or 1 for all of them (since b11100; upstream emits them
+:: from ggml_cuda_fattn_vec_instances() in ggml/cmake/common.cmake). Selecting
+:: template-instance files is no longer enough -- an undefined macro is a hard
+:: error in the if constexpr. Keep this list and the default combinations in
+:: sync with FA_TYPES / FA_DEFAULT_COMBINATIONS in llamafile/build-functions.sh.
+:: (The informational GGML_CUDA_FA_QUANTS string macro is #ifdef-guarded and is
+:: not set here, to avoid quoting it through cmd.)
+set "FA_TYPES=Q4_0 Q4_1 Q5_0 Q5_1 Q8_0 BF16 F16"
+set "FA_COMBINATIONS=Q4_0-Q4_0 Q8_0-Q8_0 F16-F16 BF16-BF16"
+for %%v in (%FA_TYPES%) do for %%k in (%FA_TYPES%) do (
+ set "FA_COMPILED=0"
+ for %%c in (!FA_COMBINATIONS!) do if /i "%%c"=="%%k-%%v" set "FA_COMPILED=1"
+ set "COMMON_FLAGS=!COMMON_FLAGS! -DGGML_CUDA_FA_%%k_%%v=!FA_COMPILED!"
+)
:: --offload-compress shrinks the fat binary (mirrors rocm.sh / #995); requires a
:: recent ROCm HIP SDK -- drop this flag if your clang++ rejects it
set "COMMON_FLAGS=%COMMON_FLAGS% --offload-compress"
diff --git a/llamafile/version.h b/llamafile/version.h
index ca3c9a6d..3d4db985 100644
--- a/llamafile/version.h
+++ b/llamafile/version.h
@@ -20,7 +20,7 @@
#define LLAMAFILE_MAJOR 0
#define LLAMAFILE_MINOR 10
-#define LLAMAFILE_PATCH 6
+#define LLAMAFILE_PATCH 7
#define LLAMAFILE_VERSION \
(100000000 * LLAMAFILE_MAJOR + 1000000 * LLAMAFILE_MINOR + LLAMAFILE_PATCH)
diff --git a/llamafile/vulkan.bat b/llamafile/vulkan.bat
index 4f2a1f5d..1304d5b5 100644
--- a/llamafile/vulkan.bat
+++ b/llamafile/vulkan.bat
@@ -189,7 +189,7 @@ set "OLD_DEFINES="
if exist "%FEATURES_STAMP%" set /p OLD_DEFINES=<"%FEATURES_STAMP%"
if not "!OLD_DEFINES!"=="!GLSLC_DEFINES!" (
echo glslc feature set changed; clearing cached shader artifacts
- del /q "%BUILD_DIR%\vulkan-shaders-gen.exe" "%BUILD_DIR%\ggml-vulkan-shaders.hpp" "%BUILD_DIR%\ggml-vulkan.obj" "%BUILD_DIR%\shader-*.obj" 2>nul
+ del /q "%BUILD_DIR%\vulkan-shaders-gen.exe" "%BUILD_DIR%\ggml-vulkan-shaders.hpp" "%BUILD_DIR%\ggml-vulkan*.obj" "%BUILD_DIR%\shader-*.obj" 2>nul
if exist "%SHADERS_BUILD_DIR%" rmdir /s /q "%SHADERS_BUILD_DIR%"
if exist "%SPVDIR%" rmdir /s /q "%SPVDIR%"
mkdir "%SHADERS_BUILD_DIR%" 2>nul
@@ -308,19 +308,23 @@ if !CPP_CMD_COUNT! gtr 0 (
echo.
:: ========================================================================
-:: Phase 5: Compile ggml-vulkan.cpp
+:: Phase 5: Compile the backend sources
:: ========================================================================
-echo Phase 5: Compiling ggml-vulkan.cpp...
-
-set "VULKAN_OBJ=%BUILD_DIR%\ggml-vulkan.obj"
-set "VULKAN_SRC=%GGML_VULKAN_DIR%\ggml-vulkan.cpp"
-
-if not exist "%VULKAN_OBJ%" (
- echo Compiling ggml-vulkan.cpp...
- cl %CXX_FLAGS% /I"%GGML_VULKAN_DIR%" /I"%VULKAN_SDK%\Include" /Fo"%VULKAN_OBJ%" "%VULKAN_SRC%"
- if errorlevel 1 (echo Error compiling ggml-vulkan.cpp & exit /b 1)
-) else (
- echo ggml-vulkan.obj is up to date
+:: Since b11100 the backend is split over several translation units
+:: (ggml-vulkan.cpp plus ggml-vulkan-buffers.cpp, ggml-vulkan-debug.cpp, ...),
+:: so compile every .cpp at the top of ggml-vulkan\ rather than naming one.
+:: vulkan-shaders\ is a subdirectory, so the wildcard leaves it alone.
+echo Phase 5: Compiling backend sources...
+
+for %%f in ("%GGML_VULKAN_DIR%\*.cpp") do (
+ set "VULKAN_OBJ=%BUILD_DIR%\%%~nf.obj"
+ if not exist "!VULKAN_OBJ!" (
+ echo Compiling %%~nxf...
+ cl %CXX_FLAGS% /I"%GGML_VULKAN_DIR%" /I"%VULKAN_SDK%\Include" /Fo"!VULKAN_OBJ!" "%%f"
+ if errorlevel 1 (echo Error compiling %%~nxf & exit /b 1)
+ ) else (
+ echo %%~nf.obj is up to date
+ )
)
echo.
diff --git a/llamafile/vulkan.sh b/llamafile/vulkan.sh
index eab04142..baf62532 100755
--- a/llamafile/vulkan.sh
+++ b/llamafile/vulkan.sh
@@ -9,7 +9,7 @@
# Build process:
# 1. Build vulkan-shaders-gen tool (C++17)
# 2. Generate shader C++ files from GLSL compute shaders using glslc
-# 3. Compile ggml-vulkan.cpp with generated shaders
+# 3. Compile the ggml-vulkan backend sources with generated shaders
# 4. Compile core GGML sources
# 5. Link into ggml-vulkan.so
#
@@ -192,7 +192,8 @@ echo ""
FEATURES_STAMP="$BUILD_DIR/glslc-features.txt"
if [ ! -f "$FEATURES_STAMP" ] || [ "$(cat "$FEATURES_STAMP" 2>/dev/null)" != "$GLSLC_DEFINES" ]; then
echo "glslc feature set changed; clearing cached shader artifacts"
- rm -f "$BUILD_DIR/vulkan-shaders-gen" "$BUILD_DIR/ggml-vulkan-shaders.hpp" "$BUILD_DIR/ggml-vulkan.o"
+ rm -f "$BUILD_DIR/vulkan-shaders-gen" "$BUILD_DIR/ggml-vulkan-shaders.hpp"
+ rm -f "$BUILD_DIR"/ggml-vulkan*.o
rm -rf "$SHADERS_BUILD_DIR" "$SPVDIR"
mkdir -p "$SHADERS_BUILD_DIR" "$SPVDIR"
rm -f "$BUILD_DIR"/shader-*.o
@@ -324,23 +325,38 @@ echo "Waiting for shader C++ compilation to finish..."
wait
#
-# Phase 5: Compile ggml-vulkan.cpp
+# Phase 5: Compile the backend sources
+#
+# Since b11100 the backend is split over several translation units
+# (ggml-vulkan.cpp plus ggml-vulkan-buffers.cpp, ggml-vulkan-debug.cpp, ...),
+# so take every .cpp at the top of ggml-vulkan/ rather than naming one. The
+# shader generator under vulkan-shaders/ is built in phase 3 and excluded by
+# -maxdepth 1.
#
echo ""
-echo "Phase 5: Compiling ggml-vulkan.cpp..."
+echo "Phase 5: Compiling backend sources..."
+
+VULKAN_SRCS=$(find "$GGML_VULKAN_DIR" -maxdepth 1 -name "*.cpp" -type f | sort)
+if [ -z "$VULKAN_SRCS" ]; then
+ echo "Error: no backend sources found in $GGML_VULKAN_DIR"
+ exit 1
+fi
-VULKAN_OBJ="$BUILD_DIR/ggml-vulkan.o"
-VULKAN_SRC="$GGML_VULKAN_DIR/ggml-vulkan.cpp"
+for src in $VULKAN_SRCS; do
+ base=$(basename "$src" .cpp)
+ obj="$BUILD_DIR/${base}.o"
-if [ ! -f "$VULKAN_OBJ" ] || [ "$VULKAN_SRC" -nt "$VULKAN_OBJ" ] || [ "$SHADERS_HPP" -nt "$VULKAN_OBJ" ]; then
- echo " Compiling ggml-vulkan.cpp..."
+ if [ -f "$obj" ] && [ "$obj" -nt "$src" ] && [ "$obj" -nt "$SHADERS_HPP" ]; then
+ echo " Skipping: $base (up to date)"
+ continue
+ fi
+
+ echo " Compiling: $base"
$CXX -c $CXX_FLAGS \
-I$GGML_VULKAN_DIR \
$SPIRV_INCLUDE \
- -o "$VULKAN_OBJ" "$VULKAN_SRC"
-else
- echo " ggml-vulkan.o is up to date"
-fi
+ -o "$obj" "$src"
+done
#
# Phase 6: Compile core GGML sources