Skip to content

armgit / janas

Public Imported from GitHub
0

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

Release 2026-10-18: Qwen3.6-35B-A3B's expert cache holds what it has room for - the in-file MTP block's larger experts get slots of their own (prompt 132-133 against 75-81 tok/s, generation 27.3 against 18.1-18.9, issue/PR #34); aligned blocks on Windows given back as they were taken (PR #31); build.sh stops on a listing that fails (PR #30); the GPU code builds on ARM64; contributions wait for the foundation

camauri139abbe15fd9
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 14d20b4..5cae8c3 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -32,19 +32,11 @@ Then open a [hardware report](https://github.com/prabanta-dev/janas/issues/new?t
- A change that touches arithmetic must keep the results it promises: kernels agree bit for bit with the scalar reference, and a pass over a block of tokens agrees bit for bit with the same tokens one at a time. The tests check this; keep them passing.
- Numbers in a commit message are welcome, and should say on what they were measured.
-## Copyright and sign-off
+## Copyright
-Janas is licensed under the **GNU General Public License, version 3 or later** (see [LICENSE](LICENSE)). Contributions are taken under the same licence.
+Janas is licensed under the **GNU General Public License, version 3 or later** (see [LICENSE](LICENSE)).
-Every commit must carry a sign-off line:
-
-```
-Signed-off-by: Your Name <your@email>
-```
-
-which `git commit -s` adds for you. It means what the [Developer Certificate of Origin](https://developercertificate.org/) says: that you wrote the change, or have the right to submit it under this licence.
-
-Please keep contributions to code you own. Code copied from elsewhere, however small, has to be declared in the pull request, with where it came from and under what licence.
+**Code from contributors cannot be accepted yet.** Janas's copyright is going to be held by a non-profit foundation, which does not exist yet; once it does, contributors will assign their rights in what they contribute to it, and their authorship will be recognised. A single holder is what keeps the code free for everyone to use, in the spirit of free and open source software, and lets the project defend its licence as a whole. Until the foundation is set up, pull requests with code stay open but are not merged; reports, measurements and ideas in the issues are as welcome as ever.
## Models
diff --git a/ChangeLog.md b/ChangeLog.md
index 3c356a2..fc90a3e 100644
--- a/ChangeLog.md
+++ b/ChangeLog.md
@@ -2,6 +2,15 @@
> Curated, user-facing summary of completed work, newest first.
+## [2026-10-18] - Qwen3.6-35B-A3B's expert cache holds what it has room for
+
+- **An MTP block inside the model's file no longer halves the expert cache.** Qwen3.6-35B-A3B's files carry the block that drafts tokens as a last layer whose experts are in Q8_0, 3,342,336 bytes each against about 2 MB for the model's own, and every slot of the cache was as large as the largest expert: the cache held little more than half the experts its memory had room for. The block's experts now have slots of their own at the end of the same memory, out of the same budget (room for one request, 128 experts, or for all of them when the budget holds every expert), and the model's experts slots of their size; the automatic cache counts the new layout, and nothing changes where the block's experts are no larger. Qwen3.6-35B-A3B Q4_K_M with a 14 GiB cache, `janas-bench --prompt 512 --gen 128 --ctx 4096`, two series of two runs each alternated with the version before: prompt 132-133 tokens/s against 75-81, generation 27.3 against 18.1-18.9, with MTP drafts 39-43 against 32-33, the cache serving 98.6% of the experts against 91-92%. The answers are the same bit for bit (janas-try's fingerprints unchanged). Found by Rendeverance (#34); the fix is written anew.
+- **Windows: no fault freeing a block that starts on a page.** `free` looked for an alignment tag before every block it was given, and read the page before one that began on a page boundary, which may not be mapped: an access violation, seen on Windows on ARM64 (found by Rendeverance, #31). Blocks from the aligned allocator now go back through `janas_aligned_free`, the only one that reads the tag; the asan build uses the tagged blocks on Linux too, so a block given back the wrong way stops the tests (`tests/test_aligned`).
+- **build.sh stops when a source listing fails**, naming the directory or the file, instead of linking a library short with no message; a library of others with no flags of its own stops it too (found by phaedrus1992, #30).
+- **The GPU code builds on ARM64 with the Vulkan headers installed**: it included the x86 intrinsics' header for one spin-wait hint.
+- **Code from contributors waits** for the non-profit foundation that will hold Janas's copyright (CONTRIBUTING.md).
+
+
## [2026-10-17] - The same answer whatever the C library and the compiler
- **The engine computes exp, log, pow, sin, cos and tanh itself** (`src/llm/jmath.c`), in double with plain IEEE-754 operations, where it called the C library's: the rotary angles, the softmax of the experts' router and of the attention's pieces, the DeltaNet's softplus, decay and gates, the soft cap of the logits, the sampler. The C library's functions differ between versions and systems - glibc 2.41 rounds several of them correctly where 2.39 did not - and one unit in the last place was enough to turn a near tie between two tokens the other way: Qwen3.6-35B-A3B's answer at temperature 0 was another text on Ubuntu 24.04 (issue #38, reported by martinambrus). The float results are the double ones rounded once; `tests/test_jmath` checks them against the C library in double over the ranges the engine uses them on (exp within 0 units in the last place, the others within one).
diff --git a/README.md b/README.md
index 2ec034a..d6c7d7f 100644
--- a/README.md
+++ b/README.md
@@ -124,7 +124,7 @@ On Qwen3-Next-80B-A3B, whose file is larger than this laptop's 30 GB, llama.cpp
- [INSTALL.md](INSTALL.md) — what the build needs, how to prepare a model, and the first run
- [ChangeLog.md](ChangeLog.md) — what changed, newest first
- [MODELS.md](MODELS.md) — what a converted model's licence is, and what may be redistributed
-- [CONTRIBUTING.md](CONTRIBUTING.md) — how to help, and the sign-off
+- [CONTRIBUTING.md](CONTRIBUTING.md) — how to help, and why code from contributors waits for now
- [src/services](src/services/README.md) — Janas's services, each with its page, in `janas-chat` and in other MCP clients
- [docs/features.md](docs/features.md) — what Janas-LLM runs today, model by model and feature by feature
- [docs/models.md](docs/models.md) — getting a model: `janas-get`, or a GGUF file converted
@@ -226,4 +226,4 @@ Janas is free software under the **GNU General Public License, version 3 or late
## Contributing
-[CONTRIBUTING.md](CONTRIBUTING.md). The short version: a measurement from a machine nobody here owns is worth more than most patches, and every commit needs a `Signed-off-by` line (`git commit -s`).
+[CONTRIBUTING.md](CONTRIBUTING.md). The short version: a measurement from a machine nobody here owns is worth more than most patches, and code from contributors cannot be merged until the non-profit foundation that will hold Janas's copyright is set up.
diff --git a/build.sh b/build.sh
index 613a418..10ffb12 100755
--- a/build.sh
+++ b/build.sh
@@ -146,7 +146,10 @@ fi
case "$VARIANT" in
release) CFLAGS+=(-O2 -DNDEBUG) ;;
debug) CFLAGS+=(-O0 -g) ;;
-asan) CFLAGS+=(-O1 -g -fsanitize=address -fno-omit-frame-pointer) ;;
+# asan: aligned blocks with a tag before them, as on Windows, so that one
+# given to the wrong release stops the program (src/common/aligned.h)
+asan) CFLAGS+=(-O1 -g -fsanitize=address -fno-omit-frame-pointer
+ -DJANAS_ALIGNED_CHECK) ;;
ubsan) CFLAGS+=(-O1 -g -fsanitize=undefined -fno-sanitize-recover=all) ;;
tsan) CFLAGS+=(-O1 -g -fsanitize=thread) ;;
*)
@@ -309,7 +312,11 @@ done
CFLAGS+=(-I"$GEN")
# one object, rebuilt when its source or any header is newer
-headers_newest=$(find src include "$SPV" "$GEN" -name '*.h' -printf '%T@\n' | sort -n | tail -1)
+if ! headers_newest=$(find src include "$SPV" "$GEN" -name '*.h' -printf '%T@\n' |
+ sort -n | tail -1) || [ -z "$headers_newest" ]; then
+ echo "build.sh: cannot list the headers under src/ and include/" >&2
+ exit 2
+fi
compile()
{ # source, object, then the flags
local src="$1" obj="$2"
@@ -321,6 +328,26 @@ compile()
fi
}
objname() { echo "$OBJ/$(echo "${1#src/}" | tr '/' '_' | sed 's/\.c$/.o/')"; }
+# The output of a listing of sources, or a stop that names what was listed:
+# read through a pipe or a process substitution its failure would be lost,
+# and a program linked with fewer objects than it needs. An empty listing
+# stops too, unless "may-be-empty" comes first.
+sources()
+{ # [may-be-empty] what was listed, then the command
+ local empty="" what out
+ [ "$1" = may-be-empty ] && empty=yes && shift
+ what="$1"
+ shift
+ if ! out=$("$@"); then
+ echo "build.sh: cannot list the sources of $what" >&2
+ exit 2
+ fi
+ if [ -z "$out" ] && [ -z "$empty" ]; then
+ echo "build.sh: no sources in $what" >&2
+ exit 2
+ fi
+ [ -z "$out" ] || printf '%s\n' "$out"
+}
# Libraries of others (src/third_party/<lib>/, see its origin.md), each with
# the flags it needs and without Janas's warnings, in an archive of their
@@ -357,6 +384,13 @@ tp_flags()
"-I$c/core -I$v/builtin/src -I$c/dispatch -I$c/extras" \
"-I$c/platform -I$c/utilities -I$v/pqcp/src" \
"-I$v/pqcp/mldsa-native/mldsa" ;;
+ yyjson) ;;
+ *)
+ # without its flags it would be compiled wrong, or not at all
+ echo "build.sh: no flags for the library of others $1" \
+ "(src/third_party/$1): add it to tp_flags" >&2
+ return 1
+ ;;
esac
}
# The sources of a library of others: all its .c files, or for Mbed TLS the
@@ -368,28 +402,45 @@ tp_sources()
case "$1" in
mbedtls)
local c="$d/tf-psa-crypto" v="$d/tf-psa-crypto/drivers"
- ls "$d"/library/*.c "$c"/core/*.c "$c"/extras/*.c "$c"/platform/*.c \
- "$c"/utilities/*.c "$v"/builtin/src/*.c "$v"/pqcp/src/*.c \
- "$v"/everest/library/x25519.c \
+ local s
+ # a directory with no .c left is its pattern unexpanded, which is
+ # no file either: both missing, both named
+ for s in "$d"/library/*.c "$c"/core/*.c "$c"/extras/*.c \
+ "$c"/platform/*.c "$c"/utilities/*.c "$v"/builtin/src/*.c \
+ "$v"/pqcp/src/*.c "$v"/everest/library/x25519.c \
"$v"/everest/library/Hacl_Curve25519_joined.c \
"$v"/p256-m/p256-m_driver_entrypoints.c \
- "$v"/p256-m/p256-m/p256-m.c 2>/dev/null | sort
+ "$v"/p256-m/p256-m/p256-m.c; do
+ if [ ! -f "$s" ]; then
+ echo "build.sh: $1: missing $s" >&2
+ return 1
+ fi
+ echo "$s"
+ done | sort
;;
*) find "$d" -name '*.c' | sort ;;
esac
}
tp_objs=()
+tp_libs=0
for d in src/third_party/*/; do
[ -d "$d" ] || continue
+ tp_libs=$((tp_libs + 1))
lib=$(basename "$d")
- # shellcheck disable=SC2207
- extra=($(tp_flags "$lib"))
+ flags=$(tp_flags "$lib") || exit 2
+ # shellcheck disable=SC2206
+ extra=($flags)
+ tp_list=$(sources "$d" tp_sources "$lib") || exit 2
while IFS= read -r src; do
obj=$(objname "$src")
tp_objs+=("$obj")
compile "$src" "$obj" "${TP_CFLAGS[@]}" "${extra[@]}"
- done < <(tp_sources "$lib")
+ done <<<"$tp_list"
done
+if [ $tp_libs = 0 ]; then
+ echo "build.sh: no libraries of others in src/third_party/" >&2
+ exit 2
+fi
rm -f "$OBJ/libjanas_third.a"
# archives made anew: ar only adds and replaces, so the objects of a source
# since removed or renamed would stay in them and be linked twice
@@ -400,12 +451,15 @@ fi
# their public headers, for the programs that use them
TP_INC=(-Isrc/third_party/libmicrohttpd/src/include -Isrc/third_party/yyjson)
# Mbed TLS's, for the TLS layer of the MCP client (src/common/mcp_tls.c)
-# shellcheck disable=SC2207
-MBEDTLS_INC=($(tp_flags mbedtls))
+flags=$(tp_flags mbedtls) || exit 2
+# shellcheck disable=SC2206
+MBEDTLS_INC=($flags)
# A directory with a main.c is a program: its other .c files are its own.
# Every other .c under src/ goes into the library.
tool_dirs=$(for m in src/*/main.c src/services/*/main.c; do dirname "$m"; done)
+lib_list=$(sources src/ find src -name '*.c' ! -name main.c ! -name win_start.c \
+ ! -path 'src/third_party/*') || exit 2
objs=()
while IFS= read -r src; do
grep -qx "$(dirname "$src")" <<<"$tool_dirs" && continue
@@ -416,8 +470,7 @@ while IFS= read -r src; do
# warnings are not Janas's to fix
[ "$src" = src/common/mcp_tls.c ] && extra=("${MBEDTLS_INC[@]/#-I/-isystem}")
compile "$src" "$obj" "${CFLAGS[@]}" "${extra[@]}"
-done < <(find src -name '*.c' ! -name main.c ! -name win_start.c \
- ! -path 'src/third_party/*' | sort)
+done < <(sort <<<"$lib_list")
rm -f "$OBJ/libjanas.a"
"$AR" rcs "$OBJ/libjanas.a" "${objs[@]}"
if [ "$VARIANT" = release ]; then
@@ -482,16 +535,24 @@ fi
# test and benchmark programs, and the C tools
for src in tests/*.c tools/*.c src/*/main.c src/services/*/main.c; do
+ # a pattern that matched nothing comes here as it is
+ if [ ! -f "$src" ]; then
+ echo "build.sh: no sources match $src" >&2
+ exit 2
+ fi
exe="$BIN/$(basename "${src%.c}")$SUFFIX$EXE"
own=()
if [ "$(basename "$src")" = main.c ]; then
dir=$(dirname "$src")
exe="$BIN/janas-$(basename "$dir")$SUFFIX$EXE"
+ own_list=$(sources may-be-empty "$dir" \
+ find "$dir" -maxdepth 1 -name '*.c' ! -name main.c) || exit 2
while IFS= read -r c; do
+ [ -n "$c" ] || continue
obj=$(objname "$c")
own+=("$obj")
compile "$c" "$obj" "${CFLAGS[@]}" "${TP_INC[@]}"
- done < <(find "$dir" -maxdepth 1 -name '*.c' ! -name main.c | sort)
+ done < <(sort <<<"$own_list")
fi
echo "LD $exe"
"$CC" "${CFLAGS[@]}" "${TP_INC[@]}" "$src" "${own[@]}" "${START[@]}" \
diff --git a/src/common/aligned.h b/src/common/aligned.h
new file mode 100644
index 0000000..47dc00f
--- /dev/null
+++ b/src/common/aligned.h
@@ -0,0 +1,86 @@
+/* SPDX-License-Identifier: GPL-3.0-or-later
+ Copyright (C) 2026 Maurizio Cammalleri */
+/*
+ * aligned.h - memory on a boundary of a power of two, and its release.
+ *
+ * A block from janas_aligned_alloc goes back through janas_aligned_free,
+ * never free(), and an ordinary block never through janas_aligned_free.
+ * On Linux and macOS they are aligned_alloc and free. Windows's C library
+ * has no aligned_alloc, and its own (_aligned_malloc) wants _aligned_free:
+ * there a block is malloc's, with a tag before the address returned that
+ * says where it began, read only by janas_aligned_free. Until 18 Oct 2026
+ * free() itself looked for the tag before any block, and read the page
+ * before one that began on a page boundary, which may not be mapped: an
+ * access violation, seen on Windows on ARM64 (PR #31).
+ *
+ * build.sh's asan variant makes Linux's the tagged ones too
+ * (JANAS_ALIGNED_CHECK): a tagged block given to free() is not one malloc
+ * returned, an ordinary one given to janas_aligned_free has its tag read
+ * in the sanitizer's red zone, and either stops the program.
+ */
+#ifndef JANAS_COMMON_ALIGNED_H
+#define JANAS_COMMON_ALIGNED_H
+
+#include <errno.h>
+#include <stdint.h>
+#include <stdlib.h>
+
+#if defined(_WIN32) || defined(JANAS_ALIGNED_CHECK)
+
+/* where malloc's block began, and that address mixed with a constant: a
+ block given to the wrong release stops the program rather than freeing
+ what the bytes before it say */
+#define JANAS_ALIGN_TAG 0x4a414e4153414c47ull
+struct janas_align_tag {
+ void *base;
+ uint64_t check;
+};
+
+static inline void *janas_aligned_alloc(size_t align, size_t size)
+{
+ if (align < sizeof(void *) || (align & (align - 1))) {
+ errno = EINVAL;
+ return NULL;
+ }
+ size_t extra = align + sizeof(struct janas_align_tag);
+ if (size > SIZE_MAX - extra) {
+ errno = ENOMEM;
+ return NULL;
+ }
+ char *base = malloc(size + extra);
+ if (!base)
+ return NULL;
+ uintptr_t a =
+ ((uintptr_t)base + sizeof(struct janas_align_tag) + align - 1) &
+ ~(uintptr_t)(align - 1);
+ struct janas_align_tag *t = (struct janas_align_tag *)a - 1;
+ t->base = base;
+ t->check = JANAS_ALIGN_TAG ^ (uint64_t)a;
+ return (void *)a;
+}
+
+static inline void janas_aligned_free(void *p)
+{
+ if (!p)
+ return;
+ struct janas_align_tag *t = (struct janas_align_tag *)p - 1;
+ if (t->check != (JANAS_ALIGN_TAG ^ (uint64_t)(uintptr_t)p))
+ abort(); /* not janas_aligned_alloc's, or freed twice */
+ t->check = 0;
+ free(t->base);
+}
+
+#else
+
+static inline void *janas_aligned_alloc(size_t align, size_t size)
+{
+ return aligned_alloc(align, size);
+}
+
+static inline void janas_aligned_free(void *p)
+{
+ free(p);
+}
+
+#endif
+#endif
diff --git a/src/common/pool.c b/src/common/pool.c
index 1acb7c3..23e25d7 100644
--- a/src/common/pool.c
+++ b/src/common/pool.c
@@ -5,6 +5,7 @@
*/
#define _GNU_SOURCE
#include "pool.h"
+#include "aligned.h"
#include <pthread.h>
#include <sched.h>
@@ -185,10 +186,8 @@ struct janas_pool *janas_pool_create(int n_threads, const int *cpus)
atomic_init(&p->ticket, 0);
atomic_init(&p->spin, SPIN_ITERATIONS);
p->threads = calloc((size_t)n_threads, sizeof(pthread_t));
- if (posix_memalign((void **)&p->done, LINE,
- (size_t)n_threads * sizeof(*p->done)) != 0)
- p->done = NULL;
- else
+ p->done = janas_aligned_alloc(LINE, (size_t)n_threads * sizeof(*p->done));
+ if (p->done)
memset(p->done, 0, (size_t)n_threads * sizeof(*p->done));
if (cpus) {
p->cpus = malloc((size_t)n_threads * sizeof(int));
@@ -198,7 +197,7 @@ struct janas_pool *janas_pool_create(int n_threads, const int *cpus)
}
if (!p->threads || !p->done || (cpus && !p->cpus)) {
free(p->threads);
- free(p->done);
+ janas_aligned_free(p->done);
free(p->cpus);
free(p);
return NULL;
@@ -370,7 +369,7 @@ void janas_pool_destroy(struct janas_pool *p)
pthread_cond_destroy(&p->wake);
pthread_cond_destroy(&p->wake_idle);
free(p->threads);
- free(p->done);
+ janas_aligned_free(p->done);
free(p->cpus);
free(p);
}
diff --git a/src/common/win_compat.c b/src/common/win_compat.c
index 506571f..e7afe87 100644
--- a/src/common/win_compat.c
+++ b/src/common/win_compat.c
@@ -29,9 +29,6 @@
#undef strcasestr
#undef memmem
#undef strndup
-#undef aligned_alloc
-#undef posix_memalign
-#undef free
#undef getline
#undef mkdtemp
#undef open_memstream
@@ -284,61 +281,6 @@ long janas_sysconf(int name)
return -1;
}
-/* the tag before an aligned block: where malloc's block began, and that
- address mixed with a constant, so that the bytes before an ordinary
- block (the heap's own header) are not taken for one */
-#define ALIGN_TAG 0x4a414e4153414c47ull
-struct align_tag {
- void *base;
- uint64_t check;
-};
-
-void *janas_aligned_alloc(size_t align, size_t size)
-{
- if (align < sizeof(void *) || (align & (align - 1))) {
- errno = EINVAL;
- return NULL;
- }
- size_t extra = align + sizeof(struct align_tag);
- if (size > SIZE_MAX - extra) {
- errno = ENOMEM;
- return NULL;
- }
- char *base = malloc(size + extra);
- if (!base)
- return NULL;
- uintptr_t a = ((uintptr_t)base + sizeof(struct align_tag) + align - 1) &
- ~(uintptr_t)(align - 1);
- struct align_tag *t = (struct align_tag *)a - 1;
- t->base = base;
- t->check = ALIGN_TAG ^ (uint64_t)a;
- return (void *)a;
-}
-
-int janas_posix_memalign(void **p, size_t align, size_t size)
-{
- void *a = janas_aligned_alloc(align, size ? size : 1);
- if (!a)
- return errno;
- *p = a;
- return 0;
-}
-
-void janas_free(void *p)
-{
- if (!p)
- return;
- struct align_tag *t = (struct align_tag *)p - 1;
- if (((uintptr_t)p & 15) == 0 &&
- t->check == (ALIGN_TAG ^ (uint64_t)(uintptr_t)p)) {
- void *base = t->base;
- t->check = 0;
- free(base);
- return;
- }
- free(p);
-}
-
ssize_t janas_getline(char **line, size_t *cap, FILE *f)
{
if (!line || !cap || !f) {
diff --git a/src/common/win_compat.h b/src/common/win_compat.h
index 50252d6..125b5da 100644
--- a/src/common/win_compat.h
+++ b/src/common/win_compat.h
@@ -114,17 +114,9 @@ int janas_flock(int fd, int op);
long janas_sysconf(int name);
#define sysconf janas_sysconf
-/* Aligned memory that free() takes back, as POSIX's: MinGW's own
- (_aligned_malloc) wants _aligned_free, and code that frees aligned and
- ordinary blocks alike would have to know which is which. A block is
- malloc's, with a tag before the address returned that says where it
- began; free() looks for the tag (win_compat.c). */
-void *janas_aligned_alloc(size_t align, size_t size);
-int janas_posix_memalign(void **p, size_t align, size_t size);
-void janas_free(void *p);
-#define aligned_alloc janas_aligned_alloc
-#define posix_memalign janas_posix_memalign
-#define free janas_free
+/* No aligned_alloc nor posix_memalign: janas_aligned_alloc and
+ janas_aligned_free (common/aligned.h), so that free() never has to tell
+ an aligned block from an ordinary one. */
/* a line of any length, as POSIX's getline */
ssize_t janas_getline(char **line, size_t *cap, FILE *f);
diff --git a/src/llm/api.c b/src/llm/api.c
index 2114698..1794614 100644
--- a/src/llm/api.c
+++ b/src/llm/api.c
@@ -258,12 +258,13 @@ int32_t janas_llm_open(const char *path, const struct janas_llm_params *pp,
next) the region is smaller than the room the same slots take in it.
A dense model is the case that cannot survive the difference - its one
expert a layer is wanted at every token, in order, so a cache one slot
- short of the layers is a cache that never hits. */
- uint64_t slot = 0;
- for (uint32_t l = 0; l < j.h.n_layer; l++)
- if (j.layers[l].slot_bytes > slot)
- slot = j.layers[l].slot_bytes;
- uint64_t experts = (uint64_t)j.h.n_layer * j.h.n_expert * slot;
+ short of the layers is a cache that never hits. An MTP block in the
+ file with larger experts has slots of its own (expert_cache.h
+ create_split), and none is asked of it when the block comes in a
+ file of its own (model_load.c). */
+ uint32_t n_main = janas_llm_main_layers(&j);
+ uint64_t experts = janas_expert_cache_room(
+ &j, p.mtp_path ? n_main : j.h.n_layer, n_main, 3);
if (experts < j.h.experts_bytes)
experts = j.h.experts_bytes;
/* what a token of context costs: keys and values of the layers that
diff --git a/src/llm/expert_cache.c b/src/llm/expert_cache.c
index a16caae..b9b41dc 100644
--- a/src/llm/expert_cache.c
+++ b/src/llm/expert_cache.c
@@ -7,6 +7,7 @@
#define _GNU_SOURCE
#include "expert_cache.h"
+#include "common/aligned.h"
#include "common/reopen.h"
#include "common/uring.h"
@@ -49,6 +50,11 @@ struct janas_expert_cache {
uint64_t slot_bytes;
size_t n_slots;
uint8_t *arena;
+ /* the layers it serves: keys of layers below n_layer, experts of those
+ from first on (an MTP block's slots apart: the block's first layer) */
+ uint32_t first, n_layer;
+ struct janas_expert_cache *wide; /* the MTP block's slots, or NULL */
+ int asked_wide; /* the outstanding request is the block's (caller only) */
int32_t *where; /* (layer, expert) -> slot, n_layer * n_expert */
int64_t *key; /* slot -> (layer, expert) or NONE */
@@ -232,7 +238,7 @@ static void touch(struct janas_expert_cache *c, int32_t s)
static int32_t victim(struct janas_expert_cache *c)
{
if (c->filled == c->n_slots && c->last_pass_asked > c->n_slots) {
- uint32_t nl = c->j->h.n_layer;
+ uint32_t nl = c->n_layer;
for (uint32_t d = 0; d < nl; d++) {
uint32_t l = (c->cur_layer + nl - d) % nl;
for (int32_t s = c->ltail[l]; s != NONE; s = c->lprev[s])
@@ -443,34 +449,96 @@ static int32_t load(struct janas_expert_cache *c, uint32_t layer, uint32_t e)
return s;
}
-struct janas_expert_cache *janas_expert_cache_create(const struct janas_jns *j,
- uint64_t budget_bytes,
- int level,
- struct janas_pool *io,
- char *err, size_t err_len)
+/* How a budget is cut in slots: n_slots of slot bytes, and n_wide of
+ wide_slot for an MTP block whose experts are larger (0: none apart). */
+struct layout {
+ uint64_t slot, wide_slot;
+ size_t n_slots, n_wide;
+};
+
+static void clamp_layers(const struct janas_jns *j, uint32_t *n_layer,
+ uint32_t *n_main)
{
- if (level < 1 || level > 3) {
- snprintf(err, err_len, "expert level %d out of range", level);
- return NULL;
- }
- uint64_t slot_bytes = 0;
- for (uint32_t l = 0; l < j->h.n_layer; l++)
- if (j->layers[l].level_bytes[level - 1] > slot_bytes)
- slot_bytes = j->layers[l].level_bytes[level - 1];
- size_t n_slots = (size_t)(budget_bytes / slot_bytes);
- size_t n_keys = (size_t)j->h.n_layer * j->h.n_expert;
- if (n_slots > n_keys)
- n_slots = n_keys;
- if (n_slots < j->h.n_expert_used) {
- snprintf(err, err_len, "budget below one layer of experts");
- return NULL;
+ if (*n_layer == 0 || *n_layer > j->h.n_layer)
+ *n_layer = j->h.n_layer;
+ if (*n_main == 0 || *n_main > *n_layer)
+ *n_main = *n_layer;
+}
+
+static void layout(const struct janas_jns *j, uint32_t n_layer, uint32_t n_main,
+ int level, uint64_t budget, struct layout *o)
+{
+ size_t ne = j->h.n_expert, keys = (size_t)n_layer * ne,
+ mkeys = (size_t)n_main * ne, wkeys = keys - mkeys;
+ uint64_t main = 0, wide = 0;
+ for (uint32_t l = 0; l < n_layer; l++) {
+ uint64_t b = j->layers[l].level_bytes[level - 1], *m = &main;
+ if (l >= n_main)
+ m = &wide;
+ if (b > *m)
+ *m = b;
}
+ /* one size for all, the largest */
+ uint64_t one = main > wide ? main : wide;
+ size_t n = (size_t)(budget / one);
+ *o = (struct layout){.slot = one, .n_slots = n < keys ? n : keys};
+ if (wide <= main || main == 0)
+ return;
+ /* or the block apart: room for one request, or for all its experts
+ where the budget holds every expert of both */
+ uint64_t ws = (wide + main - 1) / main * main;
+ size_t wn = wkeys < JANAS_EXPERT_CHUNK ? wkeys : JANAS_EXPERT_CHUNK;
+ if (budget >= (uint64_t)mkeys * main + (uint64_t)wkeys * ws)
+ wn = wkeys;
+ if (budget < (uint64_t)wn * ws)
+ return;
+ size_t mn = (size_t)((budget - (uint64_t)wn * ws) / main);
+ if (mn > mkeys)
+ mn = mkeys;
+ /* only where it holds more experts (all of them: in less memory) */
+ if (mn < j->h.n_expert_used || mn + wn < o->n_slots)
+ return;
+ *o = (struct layout){
+ .slot = main, .wide_slot = ws, .n_slots = mn, .n_wide = wn};
+}
+uint64_t janas_expert_cache_room(const struct janas_jns *j, uint32_t n_layer,
+ uint32_t n_main, int level)
+{
+ struct layout o;
+ clamp_layers(j, &n_layer, &n_main);
+ layout(j, n_layer, n_main, level, UINT64_MAX, &o);
+ return (uint64_t)o.n_slots * o.slot + (uint64_t)o.n_wide * o.wide_slot;
+}
+
+/* The bytes of every slot, the MTP block's after the model's. */
+static uint64_t arena_used(const struct janas_expert_cache *c)
+{
+ uint64_t b = (uint64_t)c->n_slots * c->slot_bytes;
+ if (c->wide)
+ b += (uint64_t)c->wide->n_slots * c->wide->slot_bytes;
+ return b;
+}
+
+/*
+ * A cache of n_slots slots for the experts of layers first to n_layer - 1,
+ * in arena (someone else's, not freed here), or in an arena of its own of
+ * used bytes and the guard.
+ */
+static struct janas_expert_cache *
+part_create(const struct janas_jns *j, uint32_t first, uint32_t n_layer,
+ uint64_t slot_bytes, size_t n_slots, int level,
+ struct janas_pool *io, uint8_t *arena, uint64_t used, char *err,
+ size_t err_len)
+{
+ size_t n_keys = (size_t)n_layer * j->h.n_expert;
struct janas_expert_cache *c = calloc(1, sizeof(*c));
if (!c)
return NULL;
c->j = j;
c->level = level;
+ c->first = first;
+ c->n_layer = n_layer;
c->io = io;
c->slot_bytes = slot_bytes;
c->n_slots = n_slots;
@@ -479,10 +547,11 @@ struct janas_expert_cache *janas_expert_cache_create(const struct janas_jns *j,
c->fd = janas_reopen_read(j->fd, 1);
/* on 2 MB boundaries, for a GPU that imports it (model_load.c
janas_m_huge_pages) */
- size_t arena_bytes =
- (n_slots * slot_bytes + JANAS_ARENA_GUARD + JANAS_HUGE_PAGE - 1) /
- JANAS_HUGE_PAGE * JANAS_HUGE_PAGE;
- c->arena = aligned_alloc(JANAS_HUGE_PAGE, arena_bytes);
+ size_t arena_bytes = (used + JANAS_ARENA_GUARD + JANAS_HUGE_PAGE - 1) /
+ JANAS_HUGE_PAGE * JANAS_HUGE_PAGE;
+ c->arena_foreign = arena != NULL;
+ c->arena =
+ arena ? arena : janas_aligned_alloc(JANAS_HUGE_PAGE, arena_bytes);
c->where = malloc(n_keys * sizeof(int32_t));
c->key = malloc(n_slots * sizeof(int64_t));
c->prev = malloc(n_slots * sizeof(int32_t));
@@ -493,8 +562,8 @@ struct janas_expert_cache *janas_expert_cache_create(const struct janas_jns *j,
more than there are layers costs nothing */
c->lprev = malloc(n_slots * sizeof(int32_t));
c->lnext = malloc(n_slots * sizeof(int32_t));
- c->lhead = malloc(((size_t)j->h.n_layer + 1) * sizeof(int32_t));
- c->ltail = malloc(((size_t)j->h.n_layer + 1) * sizeof(int32_t));
+ c->lhead = malloc(((size_t)n_layer + 1) * sizeof(int32_t));
+ c->ltail = malloc(((size_t)n_layer + 1) * sizeof(int32_t));
c->pf_flag = calloc(n_slots, 1);
if (c->fd < 0 || !c->arena || !c->where || !c->key || !c->prev ||
!c->next || !c->stamp || !c->jobs || !c->lprev || !c->lnext ||
@@ -516,7 +585,8 @@ struct janas_expert_cache *janas_expert_cache_create(const struct janas_jns *j,
* gives huge pages where they are free. JANAS_ARENA_HUGE=1 asks for
* them again.
*/
- if (getenv("JANAS_ARENA_HUGE") && atoi(getenv("JANAS_ARENA_HUGE")) > 0)
+ if (!arena && getenv("JANAS_ARENA_HUGE") &&
+ atoi(getenv("JANAS_ARENA_HUGE")) > 0)
madvise(c->arena, arena_bytes, MADV_HUGEPAGE);
pthread_mutex_init(&c->api_lock, NULL);
pthread_mutex_init(&c->lock, NULL);
@@ -560,7 +630,7 @@ struct janas_expert_cache *janas_expert_cache_create(const struct janas_jns *j,
c->has_coordinator = 1;
for (size_t k = 0; k < n_keys; k++)
c->where[k] = NONE;
- for (uint32_t l = 0; l <= j->h.n_layer; l++)
+ for (uint32_t l = 0; l <= n_layer; l++)
c->lhead[l] = c->ltail[l] = NONE;
for (size_t s = 0; s < n_slots; s++) {
c->key[s] = NONE;
@@ -570,10 +640,58 @@ struct janas_expert_cache *janas_expert_cache_create(const struct janas_jns *j,
return c;
}
+struct janas_expert_cache *
+janas_expert_cache_create_split(const struct janas_jns *j, uint32_t n_layer,
+ uint32_t n_main, uint64_t budget_bytes,
+ int level, struct janas_pool *io, char *err,
+ size_t err_len)
+{
+ if (level < 1 || level > 3) {
+ snprintf(err, err_len, "expert level %d out of range", level);
+ return NULL;
+ }
+ struct layout o;
+ clamp_layers(j, &n_layer, &n_main);
+ layout(j, n_layer, n_main, level, budget_bytes, &o);
+ if (o.n_slots < j->h.n_expert_used) {
+ snprintf(err, err_len, "budget below one layer of experts");
+ return NULL;
+ }
+ uint64_t at = (uint64_t)o.n_slots * o.slot;
+ if (!o.n_wide)
+ return part_create(j, 0, n_layer, o.slot, o.n_slots, level, io, NULL,
+ at, err, err_len);
+ /* the block's slots right after the model's, in the same arena: each a
+ whole number of the model's, they start where one of those would */
+ struct janas_expert_cache *c =
+ part_create(j, 0, n_main, o.slot, o.n_slots, level, io, NULL,
+ at + (uint64_t)o.n_wide * o.wide_slot, err, err_len);
+ if (!c)
+ return NULL;
+ c->wide = part_create(j, n_main, n_layer, o.wide_slot, o.n_wide, level, io,
+ c->arena + at, 0, err, err_len);
+ if (!c->wide) {
+ janas_expert_cache_destroy(c);
+ return NULL;
+ }
+ return c;
+}
+
+struct janas_expert_cache *janas_expert_cache_create(const struct janas_jns *j,
+ uint64_t budget_bytes,
+ int level,
+ struct janas_pool *io,
+ char *err, size_t err_len)
+{
+ return janas_expert_cache_create_split(j, 0, 0, budget_bytes, level, io,
+ err, err_len);
+}
+
void janas_expert_cache_destroy(struct janas_expert_cache *c)
{
if (!c)
return;
+ janas_expert_cache_destroy(c->wide); /* in this arena: before it goes */
if (c->has_warmer) { /* it holds the lock only for one batch */
pthread_mutex_lock(&c->api_lock);
c->warm_stop = 1;
@@ -605,7 +723,7 @@ void janas_expert_cache_destroy(struct janas_expert_cache *c)
if (c->fd >= 0)
close(c->fd);
if (!c->arena_foreign)
- free(c->arena);
+ janas_aligned_free(c->arena);
free(c->where);
free(c->key);
free(c->prev);
@@ -704,7 +822,7 @@ static void *warmer_main(void *arg)
int janas_expert_cache_warm_background(struct janas_expert_cache *c,
const uint32_t *counts)
{
- size_t n_keys = (size_t)c->j->h.n_layer * c->j->h.n_expert;
+ size_t n_keys = (size_t)c->n_layer * c->j->h.n_expert;
if (c->has_warmer)
return -1;
c->warm_counts = malloc(n_keys * sizeof(uint32_t));
@@ -741,66 +859,95 @@ int janas_expert_cache_warm_background(struct janas_expert_cache *c,
return 0;
}
-const uint8_t *janas_expert_cache_resident(struct janas_expert_cache *c,
- uint64_t *bytes, uint64_t *loads)
+/* Something is being read into the part, or about to be (under
+ api_lock). */
+static int part_busy(struct janas_expert_cache *c)
+{
+ int busy = c->outstanding || c->pf_n[0] || c->pf_n[1] ||
+ (c->has_warmer && !atomic_load(&c->warm_over));
+ pthread_mutex_lock(&c->lock);
+ busy |= c->pending;
+ pthread_mutex_unlock(&c->lock);
+ return busy;
+}
+
+/* The part holds every expert of its layers, and nothing is being read. */
+static int part_complete(struct janas_expert_cache *c, uint64_t *loads)
{
- size_t total = (size_t)c->j->h.n_layer * c->j->h.n_expert;
+ size_t total = (size_t)(c->n_layer - c->first) * c->j->h.n_expert;
/* between begin and finish the caller holds api_lock: asked there, the
answer is no rather than waiting on itself (outstanding is the
caller's own) */
if (c->n_slots < total || c->outstanding)
- return NULL;
+ return 0;
pthread_mutex_lock(&c->api_lock);
- int busy = (c->has_warmer && !atomic_load(&c->warm_over)) || c->pf_n[0] ||
- c->pf_n[1];
- pthread_mutex_lock(&c->lock);
- busy |= c->pending;
- pthread_mutex_unlock(&c->lock);
+ int busy = part_busy(c);
int full = c->filled >= total;
*loads = atomic_load(&c->loads);
pthread_mutex_unlock(&c->api_lock);
- if (busy || !full)
+ return !busy && full;
+}
+
+const uint8_t *janas_expert_cache_resident(struct janas_expert_cache *c,
+ uint64_t *bytes, uint64_t *loads)
+{
+ uint64_t wide_loads = 0;
+ if (c->asked_wide || !part_complete(c, loads) ||
+ (c->wide && !part_complete(c->wide, &wide_loads)))
return NULL;
- *bytes = (uint64_t)c->n_slots * c->slot_bytes + JANAS_ARENA_GUARD;
+ *loads += wide_loads;
+ *bytes = arena_used(c) + JANAS_ARENA_GUARD;
return c->arena;
}
const uint8_t *janas_expert_cache_arena(const struct janas_expert_cache *c,
uint64_t *bytes)
{
- *bytes = (uint64_t)c->n_slots * c->slot_bytes + JANAS_ARENA_GUARD;
+ *bytes = arena_used(c) + JANAS_ARENA_GUARD;
return c->arena;
}
int janas_expert_cache_rebase(struct janas_expert_cache *c, uint8_t *arena)
{
+ struct janas_expert_cache *w = c->wide;
pthread_mutex_lock(&c->api_lock);
- int busy = c->outstanding || c->pf_n[0] || c->pf_n[1] ||
- (c->has_warmer && !atomic_load(&c->warm_over));
- pthread_mutex_lock(&c->lock);
- busy |= c->pending;
- pthread_mutex_unlock(&c->lock);
+ if (w)
+ pthread_mutex_lock(&w->api_lock);
+ int busy = c->asked_wide || part_busy(c) || (w && part_busy(w));
/* the new memory takes no O_DIRECT reads (a GPU driver's mapping):
- the file opened again without it, for the loads that may still come */
+ the file opened again without it, for the loads that may still come
+ (the MTP block's slots read through a descriptor of their own) */
int fd = busy ? -1 : janas_reopen_read(c->j->fd, 0);
- if (fd < 0) {
+ int wfd = fd >= 0 && w ? janas_reopen_read(c->j->fd, 0) : -1;
+ if (fd < 0 || (w && wfd < 0)) {
+ if (fd >= 0)
+ close(fd);
+ if (w)
+ pthread_mutex_unlock(&w->api_lock);
pthread_mutex_unlock(&c->api_lock);
return -1;
}
- memcpy(arena, c->arena, c->n_slots * c->slot_bytes);
+ memcpy(arena, c->arena, arena_used(c));
close(c->fd);
c->fd = fd;
if (!c->arena_foreign)
- free(c->arena);
+ janas_aligned_free(c->arena);
c->arena = arena;
c->arena_foreign = 1;
+ if (w) {
+ close(w->fd);
+ w->fd = wfd;
+ w->arena = arena + (size_t)c->n_slots * c->slot_bytes;
+ pthread_mutex_unlock(&w->api_lock);
+ }
pthread_mutex_unlock(&c->api_lock);
return 0;
}
uint64_t janas_expert_cache_loads(const struct janas_expert_cache *c)
{
- return atomic_load(&((struct janas_expert_cache *)c)->loads);
+ uint64_t n = atomic_load(&((struct janas_expert_cache *)c)->loads);
+ return c->wide ? n + janas_expert_cache_loads(c->wide) : n;
}
int janas_expert_cache_warming(const struct janas_expert_cache *c)
@@ -828,7 +975,7 @@ void janas_expert_cache_warm_progress(const struct janas_expert_cache *c,
int janas_expert_cache_warm(struct janas_expert_cache *c,
const uint32_t *counts)
{
- size_t n_keys = (size_t)c->j->h.n_layer * c->j->h.n_expert;
+ size_t n_keys = (size_t)c->n_layer * c->j->h.n_expert;
uint32_t *order = malloc(n_keys * sizeof(uint32_t));
if (!order)
return -1;
@@ -890,8 +1037,11 @@ static void pf_settle(struct janas_expert_cache *c, int p)
int janas_expert_cache_prefetch(struct janas_expert_cache *c, uint32_t layer,
const uint32_t *ids, size_t n, size_t max)
{
+ if (c->wide && layer >= c->n_layer)
+ return janas_expert_cache_prefetch(c->wide, layer, ids, n, max);
int p = (int)(layer & 1);
- if (!c->pf_ring[p] || layer >= c->j->h.n_layer || c->outstanding)
+ if (!c->pf_ring[p] || layer >= c->n_layer || c->outstanding ||
+ c->asked_wide)
return 0;
pthread_mutex_lock(&c->api_lock);
pf_settle(c, p); /* a set left over (its layer never asked) */
@@ -953,7 +1103,15 @@ int janas_expert_cache_begin(struct janas_expert_cache *c, uint32_t layer,
const uint32_t *ids, size_t n,
const uint8_t **slots, uint8_t *ready)
{
- if (layer >= c->j->h.n_layer || n > c->n_slots || n > 256 || c->outstanding)
+ if (c->wide && layer >= c->n_layer) {
+ if (c->outstanding || c->asked_wide)
+ return -1;
+ int rc = janas_expert_cache_begin(c->wide, layer, ids, n, slots, ready);
+ c->asked_wide = rc >= 0;
+ return rc;
+ }
+ if (layer >= c->n_layer || n > c->n_slots || n > 256 || c->outstanding ||
+ c->asked_wide)
return -1;
/* held until finish: the background filling waits for its turn */
pthread_mutex_lock(&c->api_lock);
@@ -1030,18 +1188,26 @@ int janas_expert_cache_begin(struct janas_expert_cache *c, uint32_t layer,
void janas_expert_cache_present(struct janas_expert_cache *c, uint32_t layer,
const uint32_t *ids, size_t n, uint8_t *in)
{
+ if (c->wide && layer >= c->n_layer) {
+ janas_expert_cache_present(c->wide, layer, ids, n, in);
+ return;
+ }
uint32_t ne = c->j->h.n_expert;
size_t base = (size_t)layer * ne;
/* the background filling changes the map under this lock */
pthread_mutex_lock(&c->api_lock);
for (size_t i = 0; i < n; i++)
- in[i] = layer < c->j->h.n_layer && ids[i] < ne &&
+ in[i] = layer < c->n_layer && ids[i] < ne &&
c->where[base + ids[i]] != NONE;
pthread_mutex_unlock(&c->api_lock);
}
int janas_expert_cache_finish(struct janas_expert_cache *c)
{
+ if (c->asked_wide) {
+ c->asked_wide = 0;
+ return janas_expert_cache_finish(c->wide);
+ }
double t0 = now();
int rc = wait_jobs(c);
if (c->via_ring && c->batch_rc) /* the submission itself failed */
@@ -1070,9 +1236,27 @@ void janas_expert_cache_stats(const struct janas_expert_cache *c,
pthread_mutex_lock(&m->api_lock);
*s = c->st;
pthread_mutex_unlock(&m->api_lock);
+ if (!c->wide)
+ return;
+ /* the MTP block's lookups with the model's, as when they shared the
+ slots: the hit rate stays the one of every request */
+ struct janas_expert_cache_stats w;
+ janas_expert_cache_stats(c->wide, &w);
+ s->requests += w.requests;
+ s->hits += w.hits;
+ s->misses += w.misses;
+ s->bytes_read += w.bytes_read;
+ s->warm_bytes += w.warm_bytes;
+ s->io_seconds += w.io_seconds;
+ s->wait_seconds += w.wait_seconds;
+ s->prefetch_reads += w.prefetch_reads;
+ s->prefetch_used += w.prefetch_used;
+ s->prefetch_bytes += w.prefetch_bytes;
}
void janas_expert_cache_reset_stats(struct janas_expert_cache *c)
{
memset(&c->st, 0, sizeof(c->st));
+ if (c->wide)
+ janas_expert_cache_reset_stats(c->wide);
}
diff --git a/src/llm/expert_cache.h b/src/llm/expert_cache.h
index d31c70e..6e194bc 100644
--- a/src/llm/expert_cache.h
+++ b/src/llm/expert_cache.h
@@ -4,11 +4,12 @@
* expert_cache.h - a RAM cache of routed experts over a JNS file.
*
* A fixed number of slots, each large enough for the largest expert of the
- * model, lives in one 4 KiB-aligned arena. Replacement is LRU; the cache can
- * be warmed with the experts most used by a prompt (the prefill predicts the
- * decode working set: bozza Fase 0). The misses of a request are read in
- * parallel with O_DIRECT by a pool of I/O threads, and the experts of the
- * request itself are never evicted to make room for each other.
+ * model, lives in one 4 KiB-aligned arena (an MTP block whose experts are
+ * larger than the model's has slots of its own: see create_split). Replacement
+ * is LRU; the cache can be warmed with the experts most used by a prompt (the
+ * prefill predicts the decode working set: bozza Fase 0). The misses of a
+ * request are read in parallel with O_DIRECT by a pool of I/O threads, and the
+ * experts of the request itself are never evicted to make room for each other.
*/
#ifndef JANAS_LLM_EXPERT_CACHE_H
#define JANAS_LLM_EXPERT_CACHE_H
@@ -51,10 +52,45 @@ struct janas_expert_cache *janas_expert_cache_create(const struct janas_jns *j,
struct janas_pool *io,
char *err, size_t err_len);
-/* The bytes of an expert slot this cache holds, at the level it was made. */
+/* The most experts of one layer the engine asks for in one request
+ (model_ffn.c): the fewest slots a part of the cache may have. */
+#define JANAS_EXPERT_CHUNK 128
+
+/*
+ * The same over the file's first n_layer layers (0: all), of which the
+ * first n_main are the model's own and the others an MTP block in the same
+ * file (0: all the model's). Where the block's experts are larger than the
+ * model's - Qwen3.6-35B-A3B's in Q8_0, 3,342,336 bytes against 1,835,008 -
+ * slots all as large as theirs would hold half the experts the budget can
+ * (issue #34), so the block gets slots of its own, in the same arena and
+ * out of the same budget: as many as one request asks for, or all of its
+ * experts when the budget holds every expert of both. Its slots are whole
+ * multiples of the model's, so the arena is still cut in slots of the
+ * model's size wherever a GPU imports it. Only where that holds more
+ * experts than one size for all; otherwise, and where the block's experts
+ * are no larger, the cache is the one of janas_expert_cache_create.
+ * Requests, reads ahead and questions for the block's layers go to its
+ * slots; the counts of janas_expert_cache_stats and _loads are of both,
+ * _slots and _slot_bytes the model's, and the filling (warm) is of the
+ * model's layers alone - the block's few slots fill as it asks.
+ */
+struct janas_expert_cache *
+janas_expert_cache_create_split(const struct janas_jns *j, uint32_t n_layer,
+ uint32_t n_main, uint64_t budget_bytes,
+ int level, struct janas_pool *io, char *err,
+ size_t err_len);
+
+/* The bytes of slots create_split takes to hold every expert of those
+ layers at that level: the most a budget for them is worth. */
+uint64_t janas_expert_cache_room(const struct janas_jns *j, uint32_t n_layer,
+ uint32_t n_main, int level);
+
+/* The bytes of an expert slot this cache holds, at the level it was made
+ (of the model's layers, with an MTP block apart). */
uint64_t janas_expert_cache_slot_bytes(const struct janas_expert_cache *c);
void janas_expert_cache_destroy(struct janas_expert_cache *c);
+/* Its slots (for the model's layers, with an MTP block apart). */
size_t janas_expert_cache_slots(const struct janas_expert_cache *c);
/*
@@ -78,17 +114,18 @@ const uint8_t *janas_expert_cache_arena(const struct janas_expert_cache *c,
uint64_t *bytes);
uint64_t janas_expert_cache_loads(const struct janas_expert_cache *c);
/*
- * The arena moved to arena (n_slots x slot bytes and the guard, owned by the
- * caller, not freed here), its contents copied, the file read without O_DIRECT
- * from then on: for memory a GPU's driver allocated, which it reads faster than
- * imported pages. Only when nothing is being read (as _resident); 0, or -1 with
- * nothing changed.
+ * The arena moved to arena (the bytes _arena gives, guard included, owned by
+ * the caller, not freed here), its contents copied, the file read without
+ * O_DIRECT from then on: for memory a GPU's driver allocated, which it reads
+ * faster than imported pages. Only when nothing is being read (as _resident);
+ * 0, or -1 with nothing changed.
*/
int janas_expert_cache_rebase(struct janas_expert_cache *c, uint8_t *arena);
/*
* Loads up to the cache capacity the experts with the highest counts
- * (counts has n_layer * n_expert entries, layer-major; zero means never).
+ * (counts has n_layer * n_expert entries, layer-major; zero means never;
+ * those of an MTP block with slots apart are not read).
* The most frequent end up most recently used. Returns 0 or -1 on I/O error.
*/
int janas_expert_cache_warm(struct janas_expert_cache *c,
diff --git a/src/llm/gpu.c b/src/llm/gpu.c
index 021a534..067db6e 100644
--- a/src/llm/gpu.c
+++ b/src/llm/gpu.c
@@ -1931,7 +1931,7 @@ int janas_gpu_finish(struct janas_gpu *g, int *waited)
} else {
while ((r = g->vkGetFenceStatus(g->dev, g->fence)) == VK_NOT_READY) {
*waited = 1;
- _mm_pause();
+ gpu_relax();
}
}
g->t_waited = g->verbose ? now() - t0 : 0;
diff --git a/src/llm/gpu_impl.h b/src/llm/gpu_impl.h
index 61549ee..5e93c59 100644
--- a/src/llm/gpu_impl.h
+++ b/src/llm/gpu_impl.h
@@ -22,12 +22,21 @@
#define VK_NO_PROTOTYPES
#include <dlfcn.h>
-#include <immintrin.h>
#include <pthread.h>
#include <stdatomic.h>
#include <unistd.h>
#include <vulkan/vulkan.h>
+/* the spin-wait hint while a submission runs, as pool.c's */
+#if defined(__x86_64__)
+#include <immintrin.h>
+#define gpu_relax() _mm_pause()
+#elif defined(__aarch64__)
+#define gpu_relax() __asm__ __volatile__("yield" ::: "memory")
+#else
+#define gpu_relax() ((void)0)
+#endif
+
#define MAX_REGIONS 48
#define MAX_PENDING 64
#define MAX_XCACHE 8
diff --git a/src/llm/model.h b/src/llm/model.h
index 90219e2..c4f75f3 100644
--- a/src/llm/model.h
+++ b/src/llm/model.h
@@ -130,6 +130,10 @@ int janas_llm_model_settling(const struct janas_llm_model *m);
*/
uint32_t janas_llm_prefill_block(const struct janas_jns *j);
+/* The model's own layers in j: all of them less an MTP block in the file
+ (qwen35moe: nextn_predict_layers, the last ones). */
+uint32_t janas_llm_main_layers(const struct janas_jns *j);
+
/*
* Largest block of tokens one forward call accepts: larger than
* JANAS_LLM_MAX_BLOCK, so that a prefill reads each expert once for many
diff --git a/src/llm/model_assist.c b/src/llm/model_assist.c
index 1ce185d..7ccea68 100644
--- a/src/llm/model_assist.c
+++ b/src/llm/model_assist.c
@@ -25,6 +25,7 @@
#include <string.h>
#include <unistd.h>
+#include "common/aligned.h"
#include "jmath.h"
#include "matvec.h"
#include "vmath.h"
@@ -114,8 +115,8 @@ void janas_m_assist_free(struct janas_llm_model *m)
janas_jns_close(&a->j);
janas_attn_destroy(a->attn);
janas_attn_destroy(a->attn_swa);
- free(a->res);
- free(a->slots);
+ janas_aligned_free(a->res);
+ janas_aligned_free(a->slots);
free(a->cat);
free(a->x);
free(a->xn);
@@ -191,12 +192,12 @@ int janas_m_assist_load(struct janas_llm_model *m, const char *path,
a->ff = a->j.layers[0].m[JANAS_JNS_GATE].rows;
a->hd_max = kl > kl_swa ? kl : kl_swa;
/* the resident tensors, then each layer's feed-forward (its one slot) */
- a->res = aligned_alloc(4096, h->resident_bytes);
+ a->res = janas_aligned_alloc(4096, h->resident_bytes);
uint64_t sb = 0;
for (uint32_t l = 0; l < a->n_layer; l++)
if (a->j.layers[l].slot_bytes > sb)
sb = a->j.layers[l].slot_bytes;
- a->slots = aligned_alloc(4096, sb * a->n_layer);
+ a->slots = janas_aligned_alloc(4096, sb * a->n_layer);
if (!a->res || !a->slots)
return fail(err, err_len, path, "out of memory");
struct {
diff --git a/src/llm/model_ffn.c b/src/llm/model_ffn.c
index 50477e4..a8f94ed 100644
--- a/src/llm/model_ffn.c
+++ b/src/llm/model_ffn.c
@@ -198,8 +198,9 @@ struct expert_run {
/* Experts are asked of the cache and computed a chunk at a time: 128, a
layer of Qwen3-30B-A3B at once - in chunks of 64 the GPU got half a
layer's experts per launch, and ran them at 340 GMAC/s instead of 420
- (9 Oct 2026). */
-#define EXPERT_CHUNK 128
+ (9 Oct 2026). The cache gives an MTP block's slots of their own room for
+ one chunk. */
+#define EXPERT_CHUNK JANAS_EXPERT_CHUNK
#define MAX_RUNS (EXPERT_CHUNK + 1) /* a chunk and the shared expert */
#define EX_T(v, code) \
diff --git a/src/llm/model_load.c b/src/llm/model_load.c
index 080b43e..97916c1 100644
--- a/src/llm/model_load.c
+++ b/src/llm/model_load.c
@@ -19,6 +19,7 @@
#include <unistd.h>
#include "attention.h"
+#include "common/aligned.h"
#include "common/pool.h"
#include "common/sysinfo.h"
#include "deltanet.h"
@@ -569,7 +570,7 @@ static int load_mtp(struct janas_llm_model *m, const char *path, char *err,
snprintf(err, err_len, "%s: not an MTP block of this model", path);
return -1;
}
- m->mtp_res = aligned_alloc(4096, h->resident_bytes);
+ m->mtp_res = janas_aligned_alloc(4096, h->resident_bytes);
if (!m->mtp_res) {
snprintf(err, err_len, "out of memory for the MTP block");
return -1;
@@ -662,12 +663,24 @@ int janas_m_init_draft_head(struct janas_llm_model *m, const uint8_t *src,
return 0;
}
+uint32_t janas_llm_main_layers(const struct janas_jns *j)
+{
+ const struct janas_arch *a = janas_arch_find(j->h.arch, sizeof(j->h.arch));
+ char key[128];
+ uint32_t nextn = 0;
+ snprintf(key, sizeof(key), "%.32s.nextn_predict_layers", j->h.arch);
+ if (a && a->nextn_in_file && janas_jns_meta_u32(j, key, &nextn) == 0 &&
+ nextn < j->h.n_layer)
+ return j->h.n_layer - nextn; /* the file's last blocks predict ahead */
+ return j->h.n_layer;
+}
+
/* A buffer a product writes: whole pages, so that the GPU can be given it
(janas_gpu_import), listed for that. */
static float *out_alloc(struct janas_llm_model *m, size_t bytes)
{
bytes = (bytes + 4095) & ~(size_t)4095;
- float *p = aligned_alloc(4096, bytes ? bytes : 4096);
+ float *p = janas_aligned_alloc(4096, bytes ? bytes : 4096);
if (p && m->n_out < (int)(sizeof(m->out) / sizeof(*m->out)))
m->out[m->n_out++] = (struct janas_out_buf){p, bytes};
return p;
@@ -689,11 +702,7 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
snprintf(err, err_len, "architecture %.32s not supported", m->j.h.arch);
goto fail;
}
- m->n_layer = m->j.h.n_layer;
- uint32_t nextn = 0;
- if (m->a->nextn_in_file &&
- meta_u32(m, "nextn_predict_layers", &nextn) == 0 && nextn < m->n_layer)
- m->n_layer -= nextn; /* the file's last blocks predict ahead */
+ m->n_layer = janas_llm_main_layers(&m->j);
m->d_model = m->j.h.d_model;
m->n_expert = m->j.h.n_expert;
m->k = m->j.h.n_expert_used;
@@ -791,7 +800,7 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
}
/* on 2 MB boundaries: see janas_m_huge_pages */
if (!m->resident)
- m->resident = aligned_alloc(
+ m->resident = janas_aligned_alloc(
JANAS_HUGE_PAGE, (m->j.h.resident_bytes + JANAS_HUGE_PAGE - 1) /
JANAS_HUGE_PAGE * JANAS_HUGE_PAGE);
if (!m->resident) {
@@ -961,10 +970,10 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
(gpu_token.h) */
size_t kvb = (kv_len + 4095) / 4096 * 4096,
ksb = (ks_len * sizeof(float) + 4095) / 4096 * 4096;
- m->kcache = aligned_alloc(4096, kvb ? kvb : 4096);
- m->vcache = aligned_alloc(4096, kvb ? kvb : 4096);
- m->kscale = aligned_alloc(4096, ksb ? ksb : 4096);
- m->vscale = aligned_alloc(4096, ksb ? ksb : 4096);
+ m->kcache = janas_aligned_alloc(4096, kvb ? kvb : 4096);
+ m->vcache = janas_aligned_alloc(4096, kvb ? kvb : 4096);
+ m->kscale = janas_aligned_alloc(4096, ksb ? ksb : 4096);
+ m->vscale = janas_aligned_alloc(4096, ksb ? ksb : 4096);
m->kv_bytes = kvb;
m->ks_bytes = ksb;
/* a pair's rows: the experts', or a wider shared feed-forward's */
@@ -1206,8 +1215,12 @@ struct janas_llm_model *janas_llm_model_load(const char *path,
m->caller_pinned = janas_cpus_save(m->caller_cpus) == 0;
janas_pin_current_thread(pin);
}
- m->cache = janas_expert_cache_create(&m->j, o->cache_bytes, m->exp_level,
- m->io, err, err_len);
+ /* an MTP block in the file has its experts in this cache, in slots of
+ their own where they are larger (issue #34); one in a file of its own
+ has a cache of its own, and the file's block is never asked for */
+ m->cache = janas_expert_cache_create_split(
+ &m->j, m->mtp_own_cache ? m->n_layer : m->j.h.n_layer, m->n_layer,
+ o->cache_bytes, m->exp_level, m->io, err, err_len);
if (!m->cache)
goto fail;
/* reads ahead (model_ffn.c janas_m_prefetch): where they pay, of the
@@ -1353,7 +1366,7 @@ void janas_llm_model_free(struct janas_llm_model *m)
janas_pool_destroy(m->compute);
janas_pool_destroy(m->io);
if (!m->resident_gpu)
- free(m->resident);
+ janas_aligned_free(m->resident);
if (m->layers)
for (uint32_t l = 0; l < m->n_layer; l++) {
free(m->layers[l].sh_gate);
@@ -1363,16 +1376,19 @@ void janas_llm_model_free(struct janas_llm_model *m)
free(m->layers[l].beta_own);
}
free(m->layers);
- float *fl[] = {
- m->conv_buf, m->x, m->xn, m->q, m->kk,
- m->vv, m->att, m->router, m->rope_cos, m->rope_sin,
- m->big, m->qgate, m->ba, m->mixed, m->conv_out,
- m->beta, m->decay, m->pair_w, m->gate, m->up,
- m->h, m->dout, m->log_mixed, m->log_beta, m->log_decay};
+ float *fl[] = {m->att, m->router, m->rope_cos, m->rope_sin,
+ m->qgate, m->ba, m->mixed, m->conv_out,
+ m->beta, m->decay, m->pair_w, m->h};
for (size_t i = 0; i < sizeof(fl) / sizeof(fl[0]); i++)
free(fl[i]);
- free(m->kscale);
- free(m->vscale);
+ /* the products' buffers (out_alloc) and the KV cache: aligned */
+ void *al[] = {m->conv_buf, m->x, m->xn, m->q,
+ m->kk, m->vv, m->big, m->gate,
+ m->up, m->dout, m->log_mixed, m->log_beta,
+ m->log_decay, m->ple, m->ple_g, m->ssm_buf,
+ m->kcache, m->vcache, m->kscale, m->vscale};
+ for (size_t i = 0; i < sizeof(al) / sizeof(al[0]); i++)
+ janas_aligned_free(al[i]);
free(m->kv_off);
free(m->ks_off);
free(m->kv_cap);
@@ -1381,17 +1397,12 @@ void janas_llm_model_free(struct janas_llm_model *m)
free(m->kv_hd);
free(m->kv_heads);
free(m->rope_cos_swa);
- free(m->ple);
- free(m->ple_g);
free(m->ple_row);
free(m->ple_q);
free(m->rope_sin_swa);
free(m->xres);
free(m->xbr);
free(m->hlen);
- free(m->kcache);
- free(m->vcache);
- free(m->ssm_buf);
free(m->xq);
free(m->attq);
free(m->pair_tok);
@@ -1407,7 +1418,7 @@ void janas_llm_model_free(struct janas_llm_model *m)
janas_m_assist_free(m);
if (m->mtp_open)
janas_jns_close(&m->mtp_j);
- free(m->mtp_res);
+ janas_aligned_free(m->mtp_res);
free(m->hid);
free(m->mtp_hid);
free(m->cat);
diff --git a/tests/bench_gemm.c b/tests/bench_gemm.c
index dfc8a83..8e72ef6 100644
--- a/tests/bench_gemm.c
+++ b/tests/bench_gemm.c
@@ -28,6 +28,7 @@
#endif
#include <time.h>
+#include "common/aligned.h"
#include "common/pool.h"
#include "llm/gpu.h"
#include "llm/matvec.h"
@@ -60,7 +61,7 @@ static uint8_t *weights(size_t bytes)
{
if (mem_kind == 1)
return janas_gpu_alloc(gpu, bytes);
- return aligned_alloc(mem_kind == 2 ? (size_t)2 << 20 : 4096, bytes);
+ return janas_aligned_alloc(mem_kind == 2 ? (size_t)2 << 20 : 4096, bytes);
}
/* Written weights collapsed into 2 MB pages, then imported (pinned): the
@@ -265,9 +266,9 @@ int main(int argc, char **argv)
janas_gpu_destroy(gpu); /* the driver's memory with it */
janas_pool_destroy(pool);
if (mem_kind != 1) {
- free(w);
- free(pl[0]);
- free(pl[1]);
+ janas_aligned_free(w);
+ janas_aligned_free(pl[0]);
+ janas_aligned_free(pl[1]);
}
free(x);
free(xf);
diff --git a/tests/bench_kernels.c b/tests/bench_kernels.c
index b78555d..a2d8303 100644
--- a/tests/bench_kernels.c
+++ b/tests/bench_kernels.c
@@ -28,6 +28,7 @@
#endif
#include <time.h>
+#include "common/aligned.h"
#include "common/pool.h"
#include "llm/matvec.h"
#include "llm/gpu.h"
@@ -54,7 +55,7 @@ static unsigned char *weights(size_t bytes)
return janas_gpu_alloc(gpu, bytes);
if (mem_kind == 2) {
size_t al = (size_t)2 << 20;
- return aligned_alloc(al, (bytes + al - 1) / al * al);
+ return janas_aligned_alloc(al, (bytes + al - 1) / al * al);
}
return malloc(bytes);
}
@@ -169,8 +170,10 @@ int main(int argc, char **argv)
printf(p ? "\n" : " |");
}
fflush(stdout);
- if (mem_kind != 1)
- free(w); /* the driver's goes with the GPU */
+ if (mem_kind == 2) /* the driver's goes with the GPU */
+ janas_aligned_free(w);
+ else if (mem_kind == 0)
+ free(w);
free(y);
}
free(x);
diff --git a/tests/bench_power.c b/tests/bench_power.c
index 9874eee..83739f7 100644
--- a/tests/bench_power.c
+++ b/tests/bench_power.c
@@ -21,6 +21,7 @@
#include <time.h>
#include <unistd.h>
+#include "common/aligned.h"
#include "llm/gpu.h"
#include "llm/matvec.h"
#include "llm/quant.h"
@@ -154,7 +155,7 @@ int main(int argc, char **argv)
struct ctx c;
size_t nb = COLS / JANAS_QK, wbytes = (size_t)NMAT * ROWS * nb * 144;
wbytes = (wbytes + 4095) / 4096 * 4096;
- c.w = aligned_alloc(4096, wbytes);
+ c.w = janas_aligned_alloc(4096, wbytes);
srand(1);
for (size_t i = 0; i < wbytes; i++)
c.w[i] = (uint8_t)rand();
diff --git a/tests/gpu_check.c b/tests/gpu_check.c
index c21ed7d..3ed58d9 100644
--- a/tests/gpu_check.c
+++ b/tests/gpu_check.c
@@ -15,6 +15,7 @@
#include <string.h>
#include <time.h>
+#include "common/aligned.h"
#include "llm/gpu.h"
#include "llm/matvec.h"
#include "llm/quant.h"
@@ -38,7 +39,7 @@ int main(void)
const char *bg = getenv("GPU_CHECK_BIG_GB");
if (bg) {
size_t big = (size_t)atoi(bg) << 30;
- uint8_t *b = aligned_alloc(4096, big);
+ uint8_t *b = janas_aligned_alloc(4096, big);
if (b)
memset(b, 1, big);
}
@@ -51,7 +52,7 @@ int main(void)
enum { MAXR = 4096, MAXC = 4096, MAXV = 64 };
size_t wbytes = (size_t)MAXR * (MAXC / JANAS_QK) * 144;
wbytes = (wbytes + 4095) / 4096 * 4096;
- uint8_t *w = aligned_alloc(4096, wbytes);
+ uint8_t *w = janas_aligned_alloc(4096, wbytes);
for (size_t i = 0; i < wbytes; i++)
w[i] = (uint8_t)rnd();
struct janas_block_q4k *wb = (struct janas_block_q4k *)w;
@@ -62,7 +63,7 @@ int main(void)
/* Q6_K weights too, with small scales */
size_t w6bytes = (size_t)MAXR * (MAXC / JANAS_QK) * 210 + 4096;
w6bytes = (w6bytes + 4095) / 4096 * 4096;
- uint8_t *w6 = aligned_alloc(4096, w6bytes);
+ uint8_t *w6 = janas_aligned_alloc(4096, w6bytes);
for (size_t i = 0; i < w6bytes; i++)
w6[i] = (uint8_t)rnd();
struct janas_block_q6k *w6b = (struct janas_block_q6k *)w6;
@@ -72,7 +73,7 @@ int main(void)
ql, qh, scales, d at byte 208 */
size_t nb6 = (w6bytes - 4096) / 210;
size_t wabytes = (nb6 * JANAS_Q6KA_BLOCK + 4095) / 4096 * 4096;
- uint8_t *wa = aligned_alloc(4096, wabytes);
+ uint8_t *wa = janas_aligned_alloc(4096, wabytes);
memset(wa, 0, wabytes);
for (size_t b = 0; b < nb6; b++) {
uint8_t *d = wa + b * JANAS_Q6KA_BLOCK;
@@ -87,7 +88,7 @@ int main(void)
pbase = (pbase + 4095) / 4096 * 4096;
size_t pplane = (size_t)MAXR * (MAXC / JANAS_QK) * JANAS_Q6KP_PLANE;
size_t wpbytes = pbase + 2 * pplane;
- uint8_t *wp = aligned_alloc(4096, wpbytes);
+ uint8_t *wp = janas_aligned_alloc(4096, wpbytes);
for (size_t i = 0; i < wpbytes; i++)
wp[i] = (uint8_t)rnd();
struct janas_block_q6kp *wpb = (struct janas_block_q6kp *)wp;
@@ -97,7 +98,7 @@ int main(void)
/* Q5_K */
size_t w5bytes = (size_t)MAXR * (MAXC / JANAS_QK) * 176;
w5bytes = (w5bytes + 4095) / 4096 * 4096;
- uint8_t *w5 = aligned_alloc(4096, w5bytes);
+ uint8_t *w5 = janas_aligned_alloc(4096, w5bytes);
for (size_t i = 0; i < w5bytes; i++)
w5[i] = (uint8_t)rnd();
struct janas_block_q5k *w5b = (struct janas_block_q5k *)w5;
@@ -108,7 +109,7 @@ int main(void)
/* Q8_0: blocks of 32, eight to 256 weights */
size_t w8bytes = (size_t)MAXR * (MAXC / JANAS_QK) * 272;
w8bytes = (w8bytes + 4095) / 4096 * 4096;
- uint8_t *w8 = aligned_alloc(4096, w8bytes);
+ uint8_t *w8 = janas_aligned_alloc(4096, w8bytes);
for (size_t i = 0; i < w8bytes; i++)
w8[i] = (uint8_t)rnd();
struct janas_block_q8_0 *w8b = (struct janas_block_q8_0 *)w8;
@@ -126,7 +127,7 @@ int main(void)
size_t w3bytes = (size_t)MAXR * (MAXC / JANAS_QK) * 110 + 4096;
w3bytes = (w3bytes + 4095) / 4096 * 4096;
for (int k = 0; k < 2; k++) {
- w3[k] = aligned_alloc(4096, w3bytes);
+ w3[k] = janas_aligned_alloc(4096, w3bytes);
for (size_t i = 0; i < w3bytes; i++)
w3[k][i] = (uint8_t)rnd();
for (size_t b = 0; b + 110 <= w3bytes; b += 110) {
@@ -137,7 +138,7 @@ int main(void)
/* IQ4_XS: 136-byte blocks, d, scales_h, scales_l, nibbles */
size_t wxbytes = (size_t)MAXR * (MAXC / JANAS_QK) * 136;
wxbytes = (wxbytes + 4095) / 4096 * 4096;
- uint8_t *wx = aligned_alloc(4096, wxbytes);
+ uint8_t *wx = janas_aligned_alloc(4096, wxbytes);
for (size_t i = 0; i < wxbytes; i++)
wx[i] = (uint8_t)rnd();
for (size_t b = 0; b + 136 <= wxbytes; b += 136) {
@@ -150,7 +151,7 @@ int main(void)
for (int k = 0; k < 5; k++) {
w32bytes[k] = (size_t)MAXR * (MAXC / JANAS_QK) * 8 * s32[k];
w32bytes[k] = (w32bytes[k] + 4095) / 4096 * 4096;
- w32[k] = aligned_alloc(4096, w32bytes[k]);
+ w32[k] = janas_aligned_alloc(4096, w32bytes[k]);
for (size_t i = 0; i < w32bytes[k]; i++)
w32[k][i] = (uint8_t)rnd();
for (size_t b = 0; b + s32[k] <= w32bytes[k]; b += s32[k]) {
@@ -324,17 +325,17 @@ int main(void)
bad ? "FAILED" : "ok");
janas_gpu_destroy(g);
janas_pool_destroy(pool);
- free(w);
- free(w6);
- free(wa);
- free(wp);
- free(w5);
- free(w8);
+ janas_aligned_free(w);
+ janas_aligned_free(w6);
+ janas_aligned_free(wa);
+ janas_aligned_free(wp);
+ janas_aligned_free(w5);
+ janas_aligned_free(w8);
for (int k = 0; k < 5; k++)
- free(w32[k]);
- free(wx);
- free(w3[0]);
- free(w3[1]);
+ janas_aligned_free(w32[k]);
+ janas_aligned_free(wx);
+ janas_aligned_free(w3[0]);
+ janas_aligned_free(w3[1]);
free(x);
free(xf);
free(ya);
diff --git a/tests/gpu_ffn_check.c b/tests/gpu_ffn_check.c
index d125367..bed8b0f 100644
--- a/tests/gpu_ffn_check.c
+++ b/tests/gpu_ffn_check.c
@@ -10,6 +10,7 @@
#include <stdlib.h>
#include <string.h>
+#include "common/aligned.h"
#include "llm/gpu.h"
#include "llm/quant.h"
#include "llm/vmath.h"
@@ -67,7 +68,7 @@ int main(void)
dq6 = d * ff / JANAS_QK * 210 + 4;
size_t per = gub * 2 + (dq4 > dq6 ? dq4 : dq6);
size_t bytes = (n * ((per + 255) / 256 * 256) + 4095) / 4096 * 4096;
- uint8_t *w = aligned_alloc(4096, bytes);
+ uint8_t *w = janas_aligned_alloc(4096, bytes);
memset(w, 0, bytes);
struct janas_gpu_expert e[16];
for (size_t i = 0; i < n; i++) {
diff --git a/tests/jns_check.c b/tests/jns_check.c
index 8155652..9cff41d 100644
--- a/tests/jns_check.c
+++ b/tests/jns_check.c
@@ -15,6 +15,7 @@
#include <time.h>
#include <unistd.h>
+#include "common/aligned.h"
#include "llm/jns.h"
static double now(void)
@@ -79,7 +80,7 @@ int main(int argc, char **argv)
if (j.layers[l].slot_bytes > max_slot)
max_slot = j.layers[l].slot_bytes;
void *buf = NULL;
- if (fd < 0 || posix_memalign(&buf, JANAS_JNS_ALIGN, max_slot) != 0) {
+ if (fd < 0 || !(buf = janas_aligned_alloc(JANAS_JNS_ALIGN, max_slot))) {
fprintf(stderr, "cannot open with O_DIRECT\n");
return 1;
}
@@ -105,7 +106,7 @@ int main(int argc, char **argv)
bad, (unsigned long long)h->n_layer * h->n_expert, bytes / 1e9,
dt, bytes / 1e9 / dt);
status = bad ? 1 : 0;
- free(buf);
+ janas_aligned_free(buf);
close(fd);
}
janas_jns_close(&j);
diff --git a/tests/test_aligned.c b/tests/test_aligned.c
new file mode 100644
index 0000000..6b7aade
--- /dev/null
+++ b/tests/test_aligned.c
@@ -0,0 +1,96 @@
+/* SPDX-License-Identifier: GPL-3.0-or-later
+ Copyright (C) 2026 Maurizio Cammalleri */
+/*
+ * test_aligned.c - janas_aligned_alloc on every boundary the engine asks
+ * for, from a pointer's to a huge page's, each block written whole and
+ * given back to janas_aligned_free. In the asan variant
+ * (JANAS_ALIGNED_CHECK) also the two mismatches, each in a child of its
+ * own that must not survive: an aligned block given to free(), an
+ * ordinary one to janas_aligned_free.
+ */
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "common/aligned.h"
+
+#if defined(JANAS_ALIGNED_CHECK) && !defined(_WIN32)
+#include <fcntl.h>
+#include <sys/wait.h>
+#include <unistd.h>
+#endif
+
+static int failures;
+
+#if defined(JANAS_ALIGNED_CHECK) && !defined(_WIN32)
+/* the mismatch in a child, its report silenced: 1 if the child survived */
+static int survives(int aligned_to_free)
+{
+ fflush(stdout);
+ pid_t pid = fork();
+ if (pid < 0)
+ return 1;
+ if (pid == 0) {
+ int null = open("/dev/null", O_WRONLY);
+ if (null >= 0)
+ dup2(null, 2);
+ if (aligned_to_free) {
+ void *volatile p = janas_aligned_alloc(4096, 4096);
+ free(p);
+ } else {
+ void *volatile p = malloc(4096);
+ janas_aligned_free(p);
+ }
+ _exit(0);
+ }
+ int st = 0;
+ if (waitpid(pid, &st, 0) != pid)
+ return 1;
+ return WIFEXITED(st) && WEXITSTATUS(st) == 0;
+}
+#endif
+
+int main(void)
+{
+ static const size_t aligns[] = {sizeof(void *), 16, 64, 4096,
+ (size_t)2 << 20};
+ static const size_t sizes[] = {1, 100, 4096, 12345, (size_t)3 << 20};
+ for (size_t a = 0; a < sizeof(aligns) / sizeof(*aligns); a++)
+ for (size_t s = 0; s < sizeof(sizes) / sizeof(*sizes); s++) {
+ unsigned char *p = janas_aligned_alloc(aligns[a], sizes[s]);
+ if (!p) {
+ printf("FAIL %zu bytes on %zu: no memory\n", sizes[s],
+ aligns[a]);
+ failures++;
+ continue;
+ }
+ if ((uintptr_t)p % aligns[a]) {
+ printf("FAIL %zu bytes on %zu: at %p\n", sizes[s], aligns[a],
+ (void *)p);
+ failures++;
+ }
+ memset(p, 0xa5, sizes[s]);
+ janas_aligned_free(p);
+ }
+ janas_aligned_free(NULL);
+#if defined(JANAS_ALIGNED_CHECK) && !defined(_WIN32)
+ if (survives(1)) {
+ printf("FAIL an aligned block given to free() went unnoticed\n");
+ failures++;
+ }
+ if (survives(0)) {
+ printf("FAIL an ordinary block given to janas_aligned_free went "
+ "unnoticed\n");
+ failures++;
+ }
+#endif
+ printf("test_aligned: %s%s\n", failures ? "FAILED" : "ok",
+#if defined(JANAS_ALIGNED_CHECK) && !defined(_WIN32)
+ " (mismatches caught)"
+#else
+ ""
+#endif
+ );
+ return failures ? 1 : 0;
+}
diff --git a/tests/test_expert_cache.c b/tests/test_expert_cache.c
index fd15289..6f54f8a 100644
--- a/tests/test_expert_cache.c
+++ b/tests/test_expert_cache.c
@@ -18,6 +18,7 @@
#include <string.h>
#include <unistd.h>
+#include "common/aligned.h"
#include "common/pool.h"
#include "llm/expert_cache.h"
#include "llm/jns.h"
@@ -35,6 +36,9 @@ static int failures;
} while (0)
enum { N_LAYER = 3, N_EXPERT = 8, K = 3, SLOT = 8192, SLOTS = 5 };
+/* the last layer's experts in the second file: an MTP block's, larger than
+ the model's (issue #34), whose slots apart are two of the model's */
+enum { WIDE = 12288, WIDE_SLOT = 2 * SLOT };
/* Metadata region: {"test.block_count": u32 N_LAYER, "test.eps": f32 1e-6}. */
static size_t build_metadata(uint8_t *m)
@@ -65,13 +69,24 @@ static uint32_t rnd(void)
return (uint32_t)(rng >> 32);
}
-static void fill_slot(uint8_t *p, uint32_t l, uint32_t e)
+static void fill_n(uint8_t *p, uint32_t l, uint32_t e, int n)
{
- for (int i = 0; i < SLOT; i++)
+ for (int i = 0; i < n; i++)
p[i] = (uint8_t)(l * 31 + e * 7 + i);
}
-static int write_file(const char *path)
+static void fill_slot(uint8_t *p, uint32_t l, uint32_t e)
+{
+ fill_n(p, l, e, SLOT);
+}
+
+/* The bytes of an expert of layer l: WIDE for the last with wide set. */
+static int expert_len(int l, int wide)
+{
+ return wide && l == N_LAYER - 1 ? WIDE : SLOT;
+}
+
+static int write_file(const char *path, int wide)
{
static uint8_t img[12288];
memset(img, 0, sizeof(img));
@@ -97,30 +112,33 @@ static int write_file(const char *path)
h.resident_offset = 8192;
h.resident_bytes = 4096;
h.experts_offset = 12288;
- h.experts_bytes = (uint64_t)N_LAYER * N_EXPERT * SLOT;
+ h.experts_bytes = 0;
+ for (int l = 0; l < N_LAYER; l++)
+ h.experts_bytes += (uint64_t)N_EXPERT * expert_len(l, wide);
h.file_bytes = h.experts_offset + h.experts_bytes;
- struct janas_jns_layer ly = {.slot_bytes = SLOT,
- .level_bytes = {SLOT, SLOT, SLOT}};
- for (int m = 0; m < 3; m++)
- ly.m[m] = (struct janas_jns_matrix){.type = 12,
- .rows = 16,
- .cols = 256,
- .offset = (uint64_t)m * 2304,
- .bytes = 2304};
- static uint8_t slot[SLOT];
+ static uint8_t slot[WIDE];
+ /* reversed order on disk: the offset table must be honoured */
+ uint64_t off = h.experts_offset;
+ for (int i = N_LAYER * N_EXPERT - 1; i >= 0; i--) {
+ int l = i / N_EXPERT, len = expert_len(l, wide);
+ memcpy(img + h.expert_table_offset + 8 * i, &off, 8);
+ fill_n(slot, (uint32_t)l, (uint32_t)(i % N_EXPERT), len);
+ uint64_t crc = janas_crc32(0, slot, (size_t)len);
+ memcpy(img + h.expert_sums_offset + 8 * i, &crc, 8);
+ off += (uint64_t)len;
+ }
for (int l = 0; l < N_LAYER; l++) {
+ uint64_t len = (uint64_t)expert_len(l, wide);
+ struct janas_jns_layer ly = {.slot_bytes = len,
+ .level_bytes = {len, len, len}};
+ for (int m = 0; m < 3; m++)
+ ly.m[m] = (struct janas_jns_matrix){.type = 12,
+ .rows = 16,
+ .cols = 256,
+ .offset = (uint64_t)m * 2304,
+ .bytes = 2304};
memcpy(img + h.layer_table_offset + l * sizeof(ly), &ly, sizeof(ly));
- for (int e = 0; e < N_EXPERT; e++) {
- /* reversed order on disk: the offset table must be honoured */
- int i = l * N_EXPERT + e;
- uint64_t off = h.experts_offset +
- (uint64_t)(N_LAYER * N_EXPERT - 1 - i) * SLOT;
- memcpy(img + h.expert_table_offset + 8 * i, &off, 8);
- fill_slot(slot, (uint32_t)l, (uint32_t)e);
- uint64_t crc = janas_crc32(0, slot, SLOT);
- memcpy(img + h.expert_sums_offset + 8 * i, &crc, 8);
- }
}
uint64_t s = JANAS_FNV_INIT;
s = janas_fnv1a(img + h.layer_table_offset, layer_len, s);
@@ -135,8 +153,9 @@ static int write_file(const char *path)
return -1;
fwrite(img, 1, sizeof(img), f);
for (int i = N_LAYER * N_EXPERT - 1; i >= 0; i--) {
- fill_slot(slot, (uint32_t)(i / N_EXPERT), (uint32_t)(i % N_EXPERT));
- fwrite(slot, 1, SLOT, f);
+ int len = expert_len(i / N_EXPERT, wide);
+ fill_n(slot, (uint32_t)(i / N_EXPERT), (uint32_t)(i % N_EXPERT), len);
+ fwrite(slot, 1, (size_t)len, f);
}
return fclose(f);
}
@@ -281,7 +300,7 @@ static void check_resident(struct janas_jns *j, struct janas_pool *io)
(unsigned long long)janas_expert_cache_loads(c));
/* moved to memory of another's (as a GPU driver's): every expert
where fetch says, with its bytes, in the new arena */
- uint8_t *moved = aligned_alloc(4096, (size_t)ALL * SLOT);
+ uint8_t *moved = janas_aligned_alloc(4096, (size_t)ALL * SLOT);
CHECK(moved && janas_expert_cache_rebase(c, moved) == 0,
"rebase failed");
for (uint32_t k = 0; moved && k < ALL; k++) {
@@ -296,7 +315,7 @@ static void check_resident(struct janas_jns *j, struct janas_pool *io)
CHECK(janas_expert_cache_resident(c, &bytes, &loads) == moved,
"resident is not the new arena");
janas_expert_cache_destroy(c); /* not the arena: it is ours */
- free(moved);
+ janas_aligned_free(moved);
}
/* filled in the background: never while it runs, then resident */
@@ -447,7 +466,153 @@ static void check_prefetch(struct janas_jns *j, struct janas_pool *io)
janas_expert_cache_destroy(c);
}
-static void run(const char *path, const char *how)
+/* A request on the file with the wide last layer, checked: every slot holds
+ its expert, the block's in its own slots past the model's. */
+static void ask_split(struct janas_expert_cache *c, uint32_t layer,
+ const uint32_t *ids, int n, size_t main_slots)
+{
+ static uint8_t expect[WIDE];
+ const uint8_t *slots[K];
+ uint64_t bytes;
+ const uint8_t *a = janas_expert_cache_arena(c, &bytes);
+ int len = expert_len((int)layer, 1);
+ CHECK(janas_expert_cache_fetch(c, layer, ids, (size_t)n, slots) == 0,
+ "split: fetch of layer %u failed", layer);
+ for (int i = 0; i < n; i++) {
+ size_t at = (size_t)(slots[i] - a);
+ fill_n(expect, layer, ids[i], len);
+ CHECK(memcmp(slots[i], expect, (size_t)len) == 0,
+ "split: layer %u expert %u has the wrong bytes", layer, ids[i]);
+ /* the model's slots first, the block's after them, each starting
+ where one of the model's would (a GPU cuts the arena so) */
+ CHECK(at % SLOT == 0 && at + (size_t)len <= bytes, "split: slot at %zu",
+ at);
+ CHECK((layer == N_LAYER - 1) == (at >= main_slots * SLOT),
+ "split: layer %u in the other part's slots (%zu)", layer, at);
+ }
+}
+
+/*
+ * The last layer as an MTP block whose experts are larger than the
+ * model's: slots apart, out of the same budget, only where that holds more
+ * experts; every byte where fetch says, the counts of both together, the
+ * arena whole for a GPU.
+ */
+static void check_split(const char *path, struct janas_pool *io)
+{
+ enum { MAIN = (N_LAYER - 1) * N_EXPERT, ALL = N_LAYER * N_EXPERT };
+ const uint64_t room =
+ (uint64_t)MAIN * SLOT + (uint64_t)N_EXPERT * WIDE_SLOT;
+ struct janas_jns j;
+ char err[256];
+ CHECK(janas_jns_open(path, &j, err, sizeof(err)) == 0, "open: %s", err);
+ CHECK(janas_expert_cache_room(&j, 0, N_LAYER - 1, 3) == room,
+ "room %llu, expected %llu",
+ (unsigned long long)janas_expert_cache_room(&j, 0, N_LAYER - 1, 3),
+ (unsigned long long)room);
+ CHECK(janas_expert_cache_room(&j, 0, 0, 3) == (uint64_t)ALL * WIDE,
+ "room of one size for all");
+ CHECK(janas_expert_cache_room(&j, N_LAYER - 1, N_LAYER - 1, 3) ==
+ (uint64_t)MAIN * SLOT,
+ "room of the model's layers alone");
+
+ /* a budget where slots apart do not pay (13 experts against 14): one
+ size for all, as janas_expert_cache_create makes it */
+ uint64_t small = 5 * SLOT + (uint64_t)N_EXPERT * WIDE_SLOT;
+ struct janas_expert_cache *c = janas_expert_cache_create_split(
+ &j, 0, N_LAYER - 1, small, 3, io, err, sizeof(err));
+ CHECK(c != NULL, "create_split (small): %s", err);
+ if (c) {
+ CHECK(janas_expert_cache_slots(c) == small / WIDE &&
+ janas_expert_cache_slot_bytes(c) == WIDE,
+ "small budget: %zu slots of %llu", janas_expert_cache_slots(c),
+ (unsigned long long)janas_expert_cache_slot_bytes(c));
+ janas_expert_cache_destroy(c);
+ }
+
+ /* ten of the model's slots and the block's apart: 18 experts, not 17 */
+ size_t ms = 10;
+ uint64_t budget = ms * SLOT + (uint64_t)N_EXPERT * WIDE_SLOT;
+ c = janas_expert_cache_create_split(&j, 0, N_LAYER - 1, budget, 3, io, err,
+ sizeof(err));
+ CHECK(c != NULL, "create_split: %s", err);
+ if (!c) {
+ janas_jns_close(&j);
+ return;
+ }
+ uint64_t bytes = 0, loads = 0;
+ janas_expert_cache_arena(c, &bytes);
+ CHECK(janas_expert_cache_slots(c) == ms &&
+ janas_expert_cache_slot_bytes(c) == SLOT,
+ "split: %zu slots of %llu", janas_expert_cache_slots(c),
+ (unsigned long long)janas_expert_cache_slot_bytes(c));
+ CHECK(bytes == budget + JANAS_ARENA_GUARD, "split: arena of %llu bytes",
+ (unsigned long long)bytes);
+ uint64_t asked = 0;
+ for (int iter = 0; iter < 600; iter++) {
+ uint32_t layer = rnd() % N_LAYER, ids[K];
+ for (int i = 0; i < K; i++)
+ ids[i] = (rnd() % (N_EXPERT / K)) * K + (uint32_t)i; /* distinct */
+ ask_split(c, layer, ids, K, ms);
+ asked += K;
+ }
+ struct janas_expert_cache_stats st;
+ janas_expert_cache_stats(c, &st);
+ CHECK(st.requests == asked && st.hits + st.misses == asked,
+ "split stats: %llu requests (%llu asked), %llu hits, %llu misses",
+ (unsigned long long)st.requests, (unsigned long long)asked,
+ (unsigned long long)st.hits, (unsigned long long)st.misses);
+ /* one request at a time, whichever part it is for */
+ uint32_t e0 = 0;
+ const uint8_t *slot;
+ uint8_t ready, in[2];
+ CHECK(janas_expert_cache_begin(c, N_LAYER - 1, &e0, 1, &slot, &ready) >= 0,
+ "split: begin on the block");
+ CHECK(janas_expert_cache_begin(c, 0, &e0, 1, &slot, &ready) < 0,
+ "split: a second request while the block's is out");
+ CHECK(janas_expert_cache_finish(c) == 0, "split: finish");
+ uint32_t q[2] = {0, 1};
+ janas_expert_cache_present(c, N_LAYER - 1, q, 2, in);
+ CHECK(in[0] == 1, "split: the block's expert 0 not present");
+ CHECK(janas_expert_cache_resident(c, &bytes, &loads) == NULL,
+ "split: resident without room for every expert");
+ janas_expert_cache_reset_stats(c);
+ janas_expert_cache_stats(c, &st);
+ CHECK(st.requests == 0, "split: stats not reset");
+ janas_expert_cache_destroy(c);
+
+ /* room for every expert: resident once all are in, then moved whole */
+ c = janas_expert_cache_create_split(&j, 0, N_LAYER - 1, room, 3, io, err,
+ sizeof(err));
+ CHECK(c != NULL, "create_split (all): %s", err);
+ if (c) {
+ CHECK(janas_expert_cache_slots(c) == MAIN, "all: %zu slots",
+ janas_expert_cache_slots(c));
+ for (uint32_t k = 0; k < ALL; k++) {
+ uint32_t e = k % N_EXPERT;
+ ask_split(c, k / N_EXPERT, &e, 1, MAIN);
+ }
+ const uint8_t *a = janas_expert_cache_resident(c, &bytes, &loads);
+ CHECK(a != NULL && bytes == room + JANAS_ARENA_GUARD && loads == ALL,
+ "all: resident %p, %llu bytes, %llu loads", (const void *)a,
+ (unsigned long long)bytes, (unsigned long long)loads);
+ uint8_t *moved = janas_aligned_alloc(4096, room + JANAS_ARENA_GUARD);
+ CHECK(moved && janas_expert_cache_rebase(c, moved) == 0,
+ "all: rebase failed");
+ for (uint32_t k = 0; moved && k < ALL; k++) {
+ uint32_t e = k % N_EXPERT;
+ ask_split(c, k / N_EXPERT, &e, 1, MAIN);
+ }
+ CHECK(janas_expert_cache_resident(c, &bytes, &loads) == moved &&
+ janas_expert_cache_loads(c) == ALL,
+ "all: not resident in the new arena");
+ janas_expert_cache_destroy(c);
+ janas_aligned_free(moved);
+ }
+ janas_jns_close(&j);
+}
+
+static void run(const char *path, const char *split_path, const char *how)
{
rng = 0x9E3779B97F4A7C15ull;
struct janas_jns j;
@@ -464,6 +629,19 @@ static void run(const char *path, const char *how)
}
CHECK(janas_expert_cache_slots(c) == SLOTS, "slots %zu",
janas_expert_cache_slots(c));
+ /* experts all of one size: create_split is create */
+ {
+ struct janas_expert_cache *u = janas_expert_cache_create_split(
+ &j, 0, N_LAYER - 1, (uint64_t)SLOTS * SLOT, 3, io, err,
+ sizeof(err));
+ CHECK(u && janas_expert_cache_slots(u) == SLOTS &&
+ janas_expert_cache_slot_bytes(u) == SLOT,
+ "create_split on one size: %s", u ? "other slots" : err);
+ janas_expert_cache_destroy(u);
+ CHECK(janas_expert_cache_room(&j, 0, N_LAYER - 1, 3) ==
+ (uint64_t)N_LAYER * N_EXPERT * SLOT,
+ "room on one size");
+ }
struct model m = {.cur_layer = 0};
for (int s = 0; s < SLOTS; s++)
@@ -576,6 +754,7 @@ static void run(const char *path, const char *how)
janas_expert_cache_destroy(c);
check_prefetch(&j, io);
check_resident(&j, io);
+ check_split(split_path, io);
janas_pool_destroy(io);
janas_jns_close(&j);
printf("test_expert_cache (%s): %llu requests, %llu hits (model %llu)\n",
@@ -587,11 +766,13 @@ int main(void)
{
/* a test that deadlocks must fail, not hold the suite for ever */
alarm(600);
- char path[512];
+ char path[512], split[512];
const char *dir = getenv("TMPDIR");
snprintf(path, sizeof(path), "%s/janas_test_cache_%d.jns",
dir && *dir ? dir : "/var/tmp", (int)getpid());
- if (write_file(path) != 0) {
+ snprintf(split, sizeof(split), "%s/janas_test_cache_%d_mtp.jns",
+ dir && *dir ? dir : "/var/tmp", (int)getpid());
+ if (write_file(path, 0) != 0 || write_file(split, 1) != 0) {
fprintf(stderr,
"cannot write %s: set TMPDIR to a writable directory on a "
"disk (O_DIRECT does not work on tmpfs)\n",
@@ -602,13 +783,14 @@ int main(void)
and the pool of threads, which is what runs where it has not */
unsetenv("JANAS_URING");
unsetenv("JANAS_URING_BATCH");
- run(path, "io_uring if available");
+ run(path, split, "io_uring if available");
setenv("JANAS_URING_BATCH", "1", 1); /* some batches each way */
- run(path, "both ways");
+ run(path, split, "both ways");
unsetenv("JANAS_URING_BATCH");
setenv("JANAS_URING", "0", 1);
- run(path, "thread pool");
+ run(path, split, "thread pool");
unlink(path);
+ unlink(split);
printf("test_expert_cache: %s\n", failures ? "FAILED" : "ok");
return failures ? 1 : 0;
}
diff --git a/tests/test_gpu_output_copy.c b/tests/test_gpu_output_copy.c
index 57a397f..8f6e26e 100644
--- a/tests/test_gpu_output_copy.c
+++ b/tests/test_gpu_output_copy.c
@@ -18,6 +18,7 @@
#include <stdlib.h>
#include <string.h>
+#include "common/aligned.h"
#include "common/pool.h"
#include "llm/gpu.h"
#include "llm/matvec.h"
@@ -29,7 +30,7 @@ static int n_kept;
/* aligned, and freed only after the GPU (see above) */
static void *buffer(size_t bytes)
{
- void *p = aligned_alloc(4096, bytes);
+ void *p = janas_aligned_alloc(4096, bytes);
if (p && n_kept < 8)
kept[n_kept++] = p;
return p;
@@ -125,7 +126,7 @@ int main(void)
janas_pool_destroy(pool);
janas_gpu_destroy(g);
for (int i = 0; i < n_kept; i++)
- free(kept[i]);
+ janas_aligned_free(kept[i]);
if (!rc)
printf("test_gpu_output_copy: ok\n");
return rc;