Skip to content

armgit / janas

Public Imported from GitHub
0

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

Release 2026-10-10 (2): the documentation brought up to date - the tuner's four kinds of pass, the prompt's blocks of 512, the troubleshooting variables

camaurice9e06b1db2e
diff --git a/INSTALL.md b/INSTALL.md
index 530955b..66cc9ef 100644
--- a/INSTALL.md
+++ b/INSTALL.md
@@ -67,7 +67,7 @@ bin/x86_64-linux/janas-chat qwen3-next.jns --mtp qwen3-next-mtp.jns
Type a message and press Enter; a backslash and Enter add a line to it instead of sending it, and the backslash is not part of what the model reads. Commands start with `/`: `/help` lists them, `/stats` shows the speed of every reply, `/mode` the power mode (`auto`, `eco`, `max`) and what the engine chose, `/think` turns reasoning on or off for models that reason (their reasoning is printed in grey), `/experts n` sets how many experts each token uses (fewer: faster replies, a little less accurate). Ctrl-C stops a reply, Ctrl-D quits. `janas-chat` with no arguments lists the options (context length, memory for the expert cache, sampling).
-The engine tunes itself while it runs: it tries the configurations your machine offers (how many threads, whether the GPU takes part) on real work and keeps the fastest. The first minutes of use are the learning phase; the choices are kept in `~/.cache/janas/`, per machine and model. Afterwards it still tries an alternative now and then, to follow the machine's state (heat, power source), and ever more rarely while the alternatives keep losing.
+The engine tunes itself while it runs: it tries the configurations your machine offers (how many threads, whether the GPU takes part) on real work and keeps the fastest. The first minutes of use are the learning phase; the choices are kept in `~/.cache/janas/`, per machine and model. Afterwards it still tries an alternative now and then, to follow the machine's state (heat, power source), and ever more rarely while the alternatives keep losing; the number of threads, settled by the first tries, moves when other threads win several tries in a row.
## Measuring your machine
@@ -80,9 +80,10 @@ It takes a few minutes: it measures prompt processing and generation speed for e
## Troubleshooting
- `JANAS_GPU=0` turns the GPU off; `JANAS_GPU_DEVICE=integrated` or `discrete` picks one when there are both; `JANAS_GPU_FAST_PROMPT=0` reads prompts the exact way, bit for bit as the CPU (the fast prompt, on by default, rounds them group by group on the GPU).
+- `JANAS_GPU_BLOCK=0` keeps the dense models' prompts off the whole-block path (every layer of a block in one submission to the GPU) while leaving the fast prompt's products on; `JANAS_PREFILL_BLOCK=n` sets the tokens of a prompt's block (256 by default, 512 where the GPU takes the feed-forwards whole).
- `JANAS_PREFETCH=0` never reads experts ahead (by default it turns itself on when the cache misses many experts a token).
- `JANAS_ARENA_HUGE=1` asks the kernel for huge pages for the expert cache, as before 30 September 2026 (with transparent huge pages on `always` and their defrag on `madvise` it made the first writes to the cache slow).
- `JANAS_KERNELS=avx2` uses the AVX2 kernels even where AVX-VNNI is available.
- `JANAS_TUNE=0` turns self-tuning off (fixed defaults); delete `~/.cache/janas/tuning.txt` to start tuning from scratch.
-- `JANAS_TUNE_THREADS=0` keeps every usable thread instead of letting the first passes choose between every thread, one per performance core and the performance cores' threads; the choice, once made, is kept for that machine and model.
+- `JANAS_TUNE_THREADS=0` keeps every usable thread instead of letting the first passes choose between every thread, one per performance core and the performance cores' threads; the choice is kept for that machine and model, and moves only when other threads win several tries in a row.
- The expert cache takes the free memory, leaving a fifth of the machine's memory to the rest of the system; `--cache <GiB>` in `janas-chat` sets it.
diff --git a/README.md b/README.md
index b0b12e2..8d57f82 100644
--- a/README.md
+++ b/README.md
@@ -378,7 +378,7 @@ Servers are spoken to over their standard input and output (the stdio transport)
### Watching a long prompt
-A long prompt is most of what a reply costs: on a laptop CPU, 260,000 tokens of Qwen3-Next-80B-A3B take hours to read and seconds to answer. So the reading tells how it goes. The library reads a prompt a block of 256 tokens at a time, and `janas_llm_chat_next` returns an empty piece after each block, so a program has control between them; `janas_llm_chat_stats` says the stage (reading, writing, done), the tokens read of those to read, the time to the first token, the time spent building the prompt, the processor time of the reading, the threads the engine chose, the memory of the context and the process's peak, how the prompt is made up (system message and tools, the conversation before, the last message), and an estimate of the time left. `max_input` in the chat's parameters refuses a longer prompt with its numbers, and never cuts it.
+A long prompt is most of what a reply costs: on a laptop CPU, 260,000 tokens of Qwen3-Next-80B-A3B take hours to read and seconds to answer. So the reading tells how it goes. The library reads a prompt a block of 256 tokens at a time (512 on the models whose feed-forwards the GPU takes whole with the fast prompt), and `janas_llm_chat_next` returns an empty piece after each block, so a program has control between them; `janas_llm_chat_stats` says the stage (reading, writing, done), the tokens read of those to read, the time to the first token, the time spent building the prompt, the processor time of the reading, the threads the engine chose, the memory of the context and the process's peak, how the prompt is made up (system message and tools, the conversation before, the last message), and an estimate of the time left. `max_input` in the chat's parameters refuses a longer prompt with its numbers, and never cuts it.
Any program on the library, its own code unchanged, can have the same from the environment: `JANAS_PROGRESS=30` writes a line on stderr every thirty seconds while a prompt is read and one at the end of each reply, and `JANAS_METRICS=file` appends JSON lines - events `open`, `prompt`, `progress` (every `JANAS_METRICS_EVERY` seconds, 10 by default), `first_token`, `reply` and `refused` - with counters and times only, never the text of a prompt or a reply:
@@ -585,7 +585,7 @@ A client that lets you set the address of an OpenAI-compatible API should need n
bin/x86_64-linux/janas-bench qwen3-next.jns
```
-Close what you can before running it: it measures the machine you give it, and a browser with thirty tabs is part of the machine. It tries the configurations the machine offers — how many threads, which cores, with and without the GPU — on real passes of the model, keeps the fastest, writes them into `~/.cache/janas` so the next start begins from them, and leaves a report in the current directory. On the development laptop that report says, among other things, that bringing the efficiency cores in makes decoding about a tenth faster than the performance cores alone - and that for blocks of several tokens, which is what a prompt and a draft check are, the engine often wants fewer threads than for one.
+Close what you can before running it: it measures the machine you give it, and a browser with thirty tabs is part of the machine. It tries the configurations the machine offers — how many threads, which cores, with and without the GPU — on real passes of the model, keeps the fastest, writes them into `~/.cache/janas` so the next start begins from them, and leaves a report in the current directory. On the development laptop that report says, among other things (10 October 2026, ten models), that a single token and a prompt go fastest on the performance cores' 12 threads, while a draft check - a few tokens at once - goes fastest on all the usable cores: the kinds of pass want different threads, which is why each is tuned on its own.
## Tools
diff --git a/include/janas/llm.h b/include/janas/llm.h
index 284f53f..bb383d5 100644
--- a/include/janas/llm.h
+++ b/include/janas/llm.h
@@ -177,10 +177,11 @@ JANAS_LLM_API int32_t janas_llm_default_system(const janas_llm *llm, char *buf,
/*
* The engine tunes itself while it runs: for each kind of pass (one token,
- * a few, a block) it tries the configurations the machine offers (thread
- * counts, the GPU) and keeps the fastest, remembering its choices per
- * machine and model in ~/.cache/janas/. janas_llm_set_mode changes the
- * power mode; janas_llm_tuning describes the current choices.
+ * a few, a short block, a prompt's long blocks) it tries the configurations
+ * the machine offers (thread counts, the GPU) and keeps the fastest,
+ * remembering its choices per machine and model in ~/.cache/janas/.
+ * janas_llm_set_mode changes the power mode; janas_llm_tuning describes the
+ * current choices.
*/
JANAS_LLM_API int32_t janas_llm_set_mode(janas_llm *llm, int32_t mode);
JANAS_LLM_API int32_t janas_llm_tuning(const janas_llm *llm, char *buf,
@@ -362,11 +363,12 @@ JANAS_LLM_API int32_t janas_llm_chat_send_results(janas_llm_chat *c, int32_t n,
*
* The prompt is read here, not when it is sent or loaded: those only build
* it and return at once. A long prompt is read a block of tokens at a time
- * (256), and every block returns an empty piece, so the caller has control
- * between them - to say how far the reading has gone (janas_llm_chat_stats:
- * stage, input_done of input_tokens), or to stop. On a slow machine a
- * prompt of 260,000 tokens takes hours, and this is how a program shows it
- * is working and not stuck.
+ * (256; 512 on the models whose feed-forwards the GPU takes whole with the
+ * fast prompt), and every block returns an empty piece, so the caller has
+ * control between them - to say how far the reading has gone
+ * (janas_llm_chat_stats: stage, input_done of input_tokens), or to stop. On a
+ * slow machine a prompt of 260,000 tokens takes hours, and this is how a
+ * program shows it is working and not stuck.
*/
JANAS_LLM_API int32_t janas_llm_chat_next(janas_llm_chat *c, char *buf,
int32_t cap, int32_t *len);