37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Release 2026-09-26: why no GPU is used
camauri791b07d565f8
diff --git a/ChangeLog.md b/ChangeLog.mdindex 097eb6f..98c0868 100644--- a/ChangeLog.md+++ b/ChangeLog.md@@ -2,6 +2,19 @@> Curated, user-facing summary of completed work, newest first.+## [2026-09-26] - Why there is no GPU++- **"GPU: none usable" now says why**: built without GPU support+ (`glslangValidator` missing at build time), no Vulkan loader, Vulkan+ seeing no GPU, the GPU turned down with its name and what its driver+ lacks (Vulkan below 1.3, a feature, the subgroup size), or turned off+ (eco mode, `JANAS_GPU=0`). The same reason is in the model's description+ (`janas_llm_describe`). A tester's RTX 5080 was reported as "none+ usable" with nothing more to go on.+- **`janas-try` reports what Vulkan sees** (each device's name, kind and+ version), whether Janas will be built with GPU support, and whether it+ runs under WSL, before building and in the report.+## [2026-09-26] - Gemma 4, first part (not yet usable)- **Gemma 4 (`gemma4`) runs its forward pass**, on Gemma-4-12B-it: layersdiff --git a/INSTALL.md b/INSTALL.mdindex 9054b9d..7f33108 100644--- a/INSTALL.md+++ b/INSTALL.md@@ -22,7 +22,7 @@ sudo apt install libvulkan-dev glslang-tools vulkan-toolsOn Fedora: `sudo dnf install gcc make` and, for the GPU, `vulkan-loader-devel glslang vulkan-tools`. On Arch: `sudo pacman -S base-devel` and `vulkan-headers vulkan-icd-loader glslang vulkan-tools`.-The GPU also needs its Vulkan driver: Mesa (`mesa-vulkan-drivers` on Debian/Ubuntu) for Intel and AMD, the proprietary driver for NVIDIA. `vulkaninfo --summary` should list your GPU.+The GPU also needs its Vulkan driver: Mesa (`mesa-vulkan-drivers` on Debian/Ubuntu) for Intel and AMD, the proprietary driver for NVIDIA. `vulkaninfo --summary` should list your GPU with Vulkan 1.3 or later. When the engine takes none, `janas-bench` and the model's description say why: a build without GPU support (`glslangValidator` was missing: install it and build again), no Vulkan loader, a GPU whose driver lacks something (named), or the GPU turned off.Then, in the source directory:diff --git a/src/bench/main.c b/src/bench/main.cindex dcbb61d..734944f 100644--- a/src/bench/main.c+++ b/src/bench/main.c@@ -241,8 +241,11 @@ int main(int argc, char **argv)char desc[256];int32_t dl;janas_llm_describe(llm, desc, sizeof(desc), &dl);- out("GPU: %s\nmodel: %s\n", gpu ? janas_gpu_name(gpu) : "none usable",- desc);+ if (gpu)+ out("GPU: %s\n", janas_gpu_name(gpu));+ else+ out("GPU: none usable: %s\n", janas_llm_model_gpu_why(m));+ out("model: %s\n", desc);/* the prompt: the text once, repeated only when more is asked for */struct janas_tokenizer *tok = janas_llm_internal_tokenizer(llm);diff --git a/src/llm/api.c b/src/llm/api.cindex ff8c300..edc066b 100644--- a/src/llm/api.c+++ b/src/llm/api.c@@ -566,8 +566,10 @@ int32_t janas_llm_describe(const janas_llm *llm, char *buf, int32_t cap,* runs. The same goes for the attention, which the context decides.*/janas_llm *w = (janas_llm *)llm;- char gpubuf[96];- const char *gpu = ", GPU not present";+ char gpubuf[224];+ snprintf(gpubuf, sizeof(gpubuf), ", no GPU (%s)",+ janas_llm_model_gpu_why(llm->m));+ const char *gpu = gpubuf;if (janas_llm_model_gpu(llm->m)) {/* which passes it is given, in the words /mode uses for them */int use = janas_llm_model_gpu_used(llm->m);diff --git a/src/llm/gpu.c b/src/llm/gpu.cindex 76f6288..c4a98db 100644--- a/src/llm/gpu.c+++ b/src/llm/gpu.c@@ -290,6 +290,15 @@ static int pick_device(struct janas_gpu *g, uint32_t *family, char *err,/* a discrete GPU first (memory and power of its own), then anintegrated one; JANAS_GPU_DEVICE=integrated|discrete restricts */const char *want_dev = getenv("JANAS_GPU_DEVICE");+ /* the first device turned down and why, for the message: a tester+ with a GPU the engine does not take has to be able to tell why */+ char why[160] = "";+#define REJECT(what) \+ { \+ if (!why[0]) \+ snprintf(why, sizeof(why), "%.80s: %s", pp.deviceName, what); \+ continue; \+ }for (int pass = 0; pass < 2; pass++)for (uint32_t i = 0; i < n; i++) {g->pd = pds[i];@@ -298,10 +307,12 @@ static int pick_device(struct janas_gpu *g, uint32_t *family, char *err,VkPhysicalDeviceType type =pass == 0 ? VK_PHYSICAL_DEVICE_TYPE_DISCRETE_GPU: VK_PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU;- if (pp.deviceType != type || pp.apiVersion < VK_API_VERSION_1_3 ||+ if (pp.deviceType != type ||(want_dev &&strcmp(want_dev, pass == 0 ? "discrete" : "integrated") != 0))continue;+ if (pp.apiVersion < VK_API_VERSION_1_3)+ REJECT("its driver offers Vulkan below 1.3")VkPhysicalDeviceVulkan13Properties p13 = {.sType =VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_VULKAN_1_3_PROPERTIES};@@ -325,7 +336,7 @@ static int pick_device(struct janas_gpu *g, uint32_t *family, char *err,if (!f12.bufferDeviceAddress || !f13.shaderIntegerDotProduct ||!(p11.subgroupSupportedOperations &VK_SUBGROUP_FEATURE_BALLOT_BIT))- continue;+ REJECT("no buffer addresses, integer dot product or ballot")/* sharing host memory: needed by an integrated GPU (it reads theweights where they are), optional for a discrete one (it gets acopy in its own memory) */@@ -333,7 +344,7 @@ static int pick_device(struct janas_gpu *g, uint32_t *family, char *err,has_extension(g, VK_EXT_EXTERNAL_MEMORY_HOST_EXTENSION_NAME);g->discrete = type == VK_PHYSICAL_DEVICE_TYPE_DISCRETE_GPU;if (!g->discrete && !g->host_import)- continue;+ REJECT("integrated, without VK_EXT_external_memory_host")/* the subgroup size the shader is built for: 8 where the size canbe chosen (Intel: fewer registers per lane), else the device'sown if it never varies (NVIDIA: 32); JANAS_GPU_SG forces one */@@ -350,9 +361,9 @@ static int pick_device(struct janas_gpu *g, uint32_t *family, char *err,else if (p13.minSubgroupSize == p13.maxSubgroupSize)sg = p11.subgroupSize;else- continue; /* varying and not selectable */+ REJECT("subgroup size varying and not selectable")if (sg < 4 || sg > ROWS_PER_GROUP || (sg & (sg - 1)) != 0)- continue;+ REJECT("subgroup size not usable")g->sg = sg;g->sg_control = control;uint32_t nf = 0;@@ -367,8 +378,18 @@ static int pick_device(struct janas_gpu *g, uint32_t *family, char *err,snprintf(g->name, sizeof(g->name), "%s", pp.deviceName);return 0;}+ REJECT("no compute queue")}- snprintf(err, err_len, "no GPU with the needed features");+#undef REJECT+ if (why[0])+ snprintf(err, err_len, "%s", why);+ else if (n == 0)+ snprintf(err, err_len, "Vulkan sees no GPU");+ else if (want_dev)+ snprintf(err, err_len, "no %.20s GPU, as JANAS_GPU_DEVICE asks",+ want_dev);+ else+ snprintf(err, err_len, "Vulkan sees no integrated or discrete GPU");return -1;}diff --git a/src/llm/model.h b/src/llm/model.hindex 5db2973..06f298e 100644--- a/src/llm/model.h+++ b/src/llm/model.h@@ -268,6 +268,9 @@ int janas_llm_model_set_experts(struct janas_llm_model *m, uint32_t n);/* The GPU in use, or NULL. */struct janas_gpu *janas_llm_model_gpu(struct janas_llm_model *m);+/* Without one, why: the driver's or the build's reason, or that it was+ turned off ("" when there is a GPU). */+const char *janas_llm_model_gpu_why(const struct janas_llm_model *m);/* The expert cache, for statistics and warm-up. */struct janas_expert_cache *janas_llm_model_cache(struct janas_llm_model *m);diff --git a/src/llm/model_impl.h b/src/llm/model_impl.hindex fba5200..15a25ea 100644--- a/src/llm/model_impl.h+++ b/src/llm/model_impl.h@@ -74,6 +74,7 @@ struct janas_llm_model {struct janas_gpu *gpu; /* optional, for products over blocks */int resident_gpu; /* the resident region belongs to the GPU */int gpu_off; /* weights not shared with it: CPU only */+ char gpu_why[160]; /* no GPU: why (janas_llm_model_gpu_why) */struct janas_expert_cache *cache;const struct janas_arch *a; /* what this architecture has (arch.h) */diff --git a/src/llm/model_load.c b/src/llm/model_load.cindex cfaf53d..f9d7989 100644--- a/src/llm/model_load.c+++ b/src/llm/model_load.c@@ -475,9 +475,15 @@ struct janas_llm_model *janas_llm_model_load(const char *path,/* resident region into RAM; with the GPU, memory from its driver(JANAS_GPU_ALLOC) or imported at the end */const char *ge = getenv("JANAS_GPU");+ snprintf(m->gpu_why, sizeof(m->gpu_why), "%s",+ ge ? "turned off by JANAS_GPU=0" : "turned off (eco mode)");if (ge ? atoi(ge) > 0 : o->use_gpu) {- char gerr[256];+ /* the reason is kept: a tester with a GPU the engine does not take+ has to be able to tell a build without GPU support from a driver+ without Vulkan 1.3 */+ char gerr[256] = "";m->gpu = janas_gpu_create(gerr, sizeof(gerr));+ snprintf(m->gpu_why, sizeof(m->gpu_why), "%.150s", m->gpu ? "" : gerr);if (m->gpu && getenv("JANAS_GPU_ALLOC") &&(m->resident = janas_gpu_alloc(m->gpu, m->j.h.resident_bytes)))m->resident_gpu = 1;@@ -965,6 +971,11 @@ struct janas_gpu *janas_llm_model_gpu(struct janas_llm_model *m)return m->gpu;}+const char *janas_llm_model_gpu_why(const struct janas_llm_model *m)+{+ return m->gpu ? "" : m->gpu_why;+}+uint32_t janas_llm_model_n_ctx(const struct janas_llm_model *m){return m->n_ctx;diff --git a/tools/janas-try.sh b/tools/janas-try.shindex 53ce10e..040165d 100755--- a/tools/janas-try.sh+++ b/tools/janas-try.sh@@ -169,6 +169,24 @@ machine()esacKERNEL=$(uname -r)DISTRO=$( . /etc/os-release 2>/dev/null && echo "${PRETTY_NAME:-unknown}")+ grep -qi microsoft /proc/version 2>/dev/null && DISTRO="$DISTRO (WSL)"+ # the GPU: whether the build can have it (the shaders need+ # glslangValidator) and what Vulkan sees - name, kind, version+ if command -v glslangValidator >/dev/null; then+ GPU_BUILD="with GPU support"+ else+ GPU_BUILD="without GPU support (glslangValidator not installed)"+ fi+ if command -v vulkaninfo >/dev/null; then+ GPU_VK=$(vulkaninfo --summary 2>/dev/null | awk '+ /apiVersion/ { v = $3 }+ /deviceType/ { t = $3; sub("PHYSICAL_DEVICE_TYPE_", "", t) }+ /deviceName/ { sub(/^[^=]*= */, ""); printf "%s%s (%s, Vulkan %s)",+ n++ ? "; " : "", $0, tolower(t), v }')+ [ -n "$GPU_VK" ] || GPU_VK="Vulkan sees no device"+ else+ GPU_VK="vulkaninfo not installed"+ fiCC_VER=$( (${CC:-gcc} --version 2>/dev/null || clang --version 2>/dev/null) |head -1)# mains when an adapter (or a USB-C source) is plugged in, battery when@@ -204,6 +222,7 @@ preflight()note "memory: $(gib "$MEM_TOTAL") GiB, $(gib "$MEM_AVAIL") GiB free now"note "disk: $DISK, $(gib "$DISK_FREE") GiB free for the models"note "system: $DISTRO, Linux $KERNEL, $CC_VER"+ note "GPU: $GPU_VK; Janas will be built $GPU_BUILD"note "power: $POWER; load now $LOAD"case " $FLAGS " in *" avx2 "*) ;; *)note "no AVX2: Janas runs on its portable path, much slower";;@@ -450,6 +469,7 @@ report()echo "- Disk: $DISK"echo "- System: $DISTRO, Linux $KERNEL"echo "- Compiler: $CC_VER"+ echo "- GPU seen by Vulkan: $GPU_VK; Janas built $GPU_BUILD"echo "- Power: $POWER"echo "- Load when it started: $LOAD_START over $THREADS threads (the speed lines below say how idle the CPU was before each measurement)"echo