37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
ChangeLog
Curated, user-facing summary of completed work, newest first.
[2026-10-30] - The GPU's products the CPU's to the bit on older AMD GPUs
fma()rounded once where the driver splits it (issue #48): Mesa's RADV lowers every 32-bitfmainto a multiply and an add, two roundings, on the AMD GPUs before RDNA2 - Polaris, Vega, the Renoir and Cezanne APUs, RDNA1 - even where the shader asks for one operation, so the GPU's products came out a unit in the last place off the CPU's on about half the rows, and the self-check gave them to the CPU. When the GPU opens, a probe now computesfma()on 1,024 triples where one rounding and two differ; where the driver splits it, every shader that callsfma()is built from a variant whosefma()is rounded once through float64 (the product exact in a double, the sum rounded to odd, then to a float: Boldo and Melquiond's method; the same as the C library'sfmafon 200 million triples), provided the variant passes the same probe. On the Intel Arc of the development laptop, with the variants forced (JANAS_GPU_XFMA=1):gpu_check's 3,980 products and 96 BF16 ones the CPU's to the bit.- Float conversions held exact in the shaders: Mesa's NIR took
double(float(i))fordouble(i), the integer's rounding to a float lost; every conversion is now decorated asfma()already was (tools/spv_fma.c).
[2026-10-29] - Speech in every sound file OpenAI accepts
Word times (
timestamp_granularities[]=wordin janas-server's transcriptions,wordsin the library's parameters withjanas_llm_speech_wordsandjanas_llm_speech_word, the FreeBASIC header too), found as OpenAI's open-source Whisper finds them: the window's text decoded again keeping the attention of the model's alignment heads over the sound (from the checkpoint; OpenAI's default, the decoder's second half, without them), its softmax standardized and median-filtered, the path of least cost through it by dynamic time warping, tokens joined into words as OpenAI's tokenizer joins them (characters for Chinese, Japanese, Thai, Lao, Burmese and Cantonese), punctuation joined to its word, OpenAI's corrections at pauses and segment ends, and the next window starting at the last word's end. On John F. Kennedy's 11 seconds the words end where whisper.cpp's token times (the same method, in half precision) put them, within 20-40 ms.Who speaks when (
response_format=diarized_jsonwithknown_speaker_names[]andknown_speaker_references[], as OpenAI'sgpt-4o-transcribe-diarize;diarizeandjanas_llm_speech_turnsin the library): WeSpeaker's ResNet34 speaker embeddings, converted from its ONNX model by a new tool,wespeaker2jns, run over Kaldi's filter banks computed as Kaldi computes them (1.6e-4 from kaldi-native-fbank; the embedding at a cosine of 0.9999993 to onnxruntime's), on windows of 1.5 s of the speech; WeSpeaker's spectral clustering (a k-nearest affinity, the Laplacian's eigengap for the number of speakers, k-means; the eigenvectors by Janas's own Householder and QL); each word the speaker of the windows over it, turns of consecutive words, known speakers named by their recordings. On 50 seconds spoken by three synthetic voices: ten turns of ten, the three speakers found, the known ones named; one speaker in a recording of one.Chunking by voice activity (
chunking_strategyautoorserver_vadwith its threshold, prefix padding and silence, as OpenAI's;chunkingin the library): Silero VAD v5, converted from its ONNX model by a new tool,silero2jns(a protobuf reader of its own, the 16 kHz model's weights in single precision), run as its model computes it (the STFT's basis, four convolutions, an LSTM carried from window to window) and its stretches of speech found as Silero'sget_speech_timestampsfinds them; only those are read, Whisper's windows within each, the text before carried on. Its probabilities agree with onnxruntime's on the same model to 1.7e-6; 46 seconds of Italian transcribed in eleven segments that follow the pauses.Transcriptions streamed, with log-probabilities:
stream: truegives server-sent events as OpenAI's transcription models do (atranscript.text.deltafor each segment as soon as it is final,transcript.text.donewith the text and the usage),include[]=logprobseach text token's log-probability and bytes; in the library a callback for each final segment, a flag that stops a transcription between windows, and each token's log-probability. With theopenaiSDK: an M4A of 46 seconds streamed in ten deltas, words with their times, the logprobs of John F. Kennedy's 25 tokens.A text that repeats itself is found as OpenAI's Whisper finds it: its compression ratio, its bytes over their zlib-compressed bytes, above 2.4 has the window decoded again at a higher temperature (whisper.cpp's entropy of the tokens was used before); the compressed size from Janas's own deflate (
src/common/deflate.c), identical to zlib's at its default level on 3,300 texts against Python's zlib (random bytes, words, loops, Italian, up to 32 KiB and over several blocks);compression_ratioin verbose_json's segments and injanas_llm_speech_segment_info.Faults found by fuzzing the decoders under AddressSanitizer and UndefinedBehaviorSanitizer, fixed: an Opus packet whose frames were empty concealed a frame of no samples and wrote before the CELT decoder's buffer (now no more is concealed than the packet's TOC says, as libopus does); an MP3 frame with more main data than the bit reservoir holds; ALAC's predictor and SILK's shifts of negative values in wrapping arithmetic made explicit; a FLAC header's skip.
Sound files read by Janas's own decoders, for speech to text: FLAC (every kind RFC 9639 describes), MP3 and the other MPEG audio (MPEG-1, 2 and 2.5, Layers I, II and III: the bit reservoir, joint and intensity stereo, mixed blocks, free format, LAME's gapless trim), Ogg with Vorbis (floors 0 and 1, every residue), FLAC or Opus, and Opus itself - SILK in its exact fixed point with libopus's resampler, CELT, hybrid frames, the changes of mode between them, lost frames concealed as libopus conceals them, multistream with its channel mappings. Against FFmpeg's decoders and the reference streams: FLAC identical on 21 files; MPEG at 114 to 127 dB on 37 files and on ISO 11172-4's conformance streams, within 16-bit rounding of their reference PCM; Vorbis at 130 dB (91-99 dB on floor 0, where FFmpeg computes the line spectrum in single precision); Opus with the range coder's state identical, frame after frame, on every official test vector, SILK identical to the sample and CELT at 86 dB against Debian's libopus (139 dB once its approximate exp2 is taken into account). A Vorbis stream of a single page is now trimmed at its end only, as libvorbisfile does (FFmpeg drops its last 256 samples).
MP4, M4A, MOV, AAC and WebM, read by Janas's own decoders too, so every format OpenAI's transcriptions accept is read: AAC LC, Main (its backward-adaptive prediction rounded as the reference does) and LTP, HE-AAC's spectral band replication (also downsampled, at the core's rate) and HE-AAC v2's parametric stereo, coupling channels, pulses, noise substitution, intensity and mid/side stereo, temporal noise shaping, frames of 1024 and 960; AAC in MP4 (sample tables and movie fragments, the edit list trimming the encoder's delay and padding) and in ADTS frames; ALAC; MP3, Opus, FLAC and PCM in MP4 and MOV; Matroska and WebM with sizes unknown (as browsers record them) and every lacing, with Opus (its codec delay and discard padding), Vorbis, FLAC, AAC, MPEG audio, ALAC and PCM. Against the MPEG-4 conformance streams FFmpeg's tests use (LC, Main, pulses, coupling channels, 96 kHz, 5.1, HE-AAC v1 and v2, downsampled SBR): 127 to 138 dB against FFmpeg, which itself is within 16-bit rounding of the reference PCM; ALAC, FLAC and PCM identical; AAC from FFmpeg's encoder in M4A, ADTS and Matroska 135 dB. Where FFmpeg's command line leaves the encoder's padding at the end (the edit list's length, Matroska's codec delay), Janas gives the source's exact length. AAC's low-delay (ER AAC LD, ELD) and USAC kinds are refused with a reason. Fifteen small clips, one of each format, in
tests/test_audio.cwith FFmpeg's decoding of them, every truncation and thousands of corrupted copies read or refused, clean under the sanitizers.
[2026-10-28] - Defects found by a code review, fixed; one choice of quality; JSON schemas with bounds and formats
Speech to text, with Whisper:
/v1/audio/transcriptionsand/v1/audio/translationsin janas-server (--speech-model; json, text, srt, vtt and verbose_json, as OpenAI'swhisper-1),janas_llm_speech_*in the library and its FreeBASIC header. A new tool,hf2jns_whisper, converts OpenAI's checkpoints from Hugging Face, the weights kept in the half precision they were released in; the encoder runs on the vision encoders' tiled kernels, now in half precision too, the decoder token by token with its keys and values kept, and the transcription goes window by window as OpenAI's does - the language found, timestamps, temperatures raised where the text loops, silence skipped. Against whisper.cpp on the same weights: the logits at a cosine of 1.000000, the same spectrograms, the same segments and times on 46 seconds of Italian but for two words, a comma and one time 40 ms apart; the tokenizer (GPT-2's expression, new intokenizer.c) splits as Hugging Face's. Sound read by Janas's own WAV decoder (8 to 32 bits, float, any channels) and resampled with a windowed sinc (src/common/audio*.c, tried intests/test_audio.c); FLAC and MP3 not yet.Pictures in, with Gemma 4 12B: its vision has no blocks - each patch of 48x48 pixels layer-normed, through one dense layer, normed, plus the embeddings of its column and row, normed, then projected - converted from its 100 MB of the checkpoint and run in
vision_gemma_u.c, the same as llama.cpp's to a cosine of 1.000000; its pictures attended both ways as the 26B's. It told the shapes, read "JANAS 42" and described an abstract picture.Pictures in, with Gemma 4 26B-A4B: its larger encoder (27 blocks of 1152, standardized output) converted the same way, at a cosine of 0.999998 against llama.cpp's. In its language model a picture's tokens attend to one another both ways, as it was trained: the attention takes, token by token, the last position it sees (
janas_attn_set_last, checked against a double-precision reference with a picture across calls and chunks), a prompt is never cut inside a picture, and a model whose encoder says so reads prompts in blocks that hold a picture whole; those passes stay on the CPU's attention. It describes a picture as llama.cpp's does.Pictures in, with Gemma 4 E2B and E4B: their vision encoder converted from the original checkpoint by
hf2jns_vision(of Gemma's checkpoint only the vision tower is read: 338 MB for E2B), run as the model's own code does - the picture sized to at most 280 tokens as its processor does, RMS norms, every linear clamped to its bounds, a 2D rotary embedding of the patches' columns and rows, 3 x 3 pooling - and the pictures' tokens given the padding's per-layer inputs as the reference does. Against llama.cpp's encoder on the same pictures: a cosine of 0.999999 on average. Both models told a red circle and a blue square, a green triangle, and read "JANAS 42" off a picture of text.Pictures in, with Qwen3.5:
janas_llm_chat_imagein the library,/imagein janas-chat,image_urlandinput_imageparts (data: URLs, http(s) at public addresses only, files uploaded here) in janas-server. The vision encoder is converted from the original checkpoint by a new tool,hf2jns_vision, into a.vision.jnsbeside the model, and runs as the model's own processor and code do: pictures resized as Pillow does (to the byte), Janas's own PNG and JPEG decoders (the same pixels as Pillow on 295 pictures of every kind), the ViT's output against llama.cpp's at a cosine of 0.999997; the pictures' tokens get their row and column as positions (the interleaved multimodal RoPE). With Qwen3.5-0.8B: "a red circle on the left and a blue square on the right", in the library, janas-chat and janas-server alike, and the same answers as llama.cpp's.Code interpreter (
--code-interpreter, off by default): the model writes Python and the server runs it in a cage of its own (src/common/sandbox.c: user, mount, PID, network, IPC and UTS namespaces without privileges, the system's/usrread-only, only/mnt/datawritable, no network, threads but no new processes, a seccomp filter, limits on memory, file sizes and open files), one Python session per container that keeps its variables like a notebook, figures saved as PNG;code_interpreterin the Responses API (withcode_interpreter_callitems andcontainer_file_citationannotations) and in the Assistants (steps with logs and images as Files,file_pathannotations), and the Containers API (nine operations). With theopenaiSDK and Qwen3-4B: computations, a CSV summed, a chart downloaded, variables kept; links planted by the code to the user's files never followed; the cage tried against code that wants out intests/test_sandbox.c.Assistants (
/v1/assistants,/v1/threads, messages, runs, run steps; 23 operations): assistants with functions andfile_searchover this server's vector stores, threads and their messages (files attached forfile_searchgo into the thread's own store), runs in the background through the same code as the Responses API - function calls wait forsubmit_tool_outputs, searches become steps, answers become messages withfile_citationannotations;cancelstops the model at its next token, and a run cut by a stop goes on after the restart. Runs stream their events withstream: true, the text as the model writes it. With theopenaiSDK and Qwen3-4B, all of these, polled and streamed. Not yet:code_interpreter, images.janas-server stops at once with a request running: Ctrl+C waited for the whole reply of every request in progress, even when its client had gone (a request without a stream never looks at its connection), so a stop could take minutes. The clients' requests, running and waiting, now end at their next token (or block of the prompt read), new ones are refused, and the server waits at most 2 s for them to answer: a client still there gets "the server is stopping" (as an error, in a stream too); batches and evals go on where they resume at the next start. With Qwen3-4B and a reply of 2,000 tokens running: stopped in 0.1 to 0.2 s, client gone, waiting, or reading a stream.
JSON output held to a string's
pattern: classes, groups, alternatives and repetitions as a grammar, a quote, a backslash or a control character written escaped, the match written from the string's start (an unanchored pattern let the model write on without ever reaching it); patterns it cannot hold (anchors inside, back references, look-arounds) leave the string free as before. Qwen3-4B at temperature 1, 100 replies to a schema of six patterns (a plate, a code, a phone number, a hashtag, a quoted motto, a Windows path): 100 valid.Evals (
/v1/evals, twelve operations): an eval's graders - string checks, text similarity (fuzzy match, BLEU, GLEU, METEOR, ROUGE, cosine) and the server's own model as a judge that labels or scores, held to its answer by a JSON schema - run on rows given in the request or in a JSONL file, the samples written by the model from a template or brought by the rows; runs go on in the background behind every client's request, saved row by row, so they go on after a restart, and can be cancelled. With theopenaiSDK and Qwen3-4B, all four kinds of grader, a cancel and a restart.file_searchin the Responses API: the model searches the request's vector stores, answers from the passages and cites them as [n], which becomefile_citationannotations, with afile_search_callitem holding the query and the results. With Qwen3-4B, the right file found and cited.Vector stores (
/v1/vector_stores, sixteen operations, the file batches among them): text files cut into overlapping chunks, embedded in the background with--embedding-modeland searched by their cosine, or by their words (BM25) without one; attributes and filters, scores from 0 to 1. With theopenaiSDK and Qwen3-Embedding-0.6B, each of three questions found its own file first.HTTPS (
--tls-cert,--tls-key): the server listens itself, does the TLS handshake on the Mbed TLS it carries, and hands libmicrohttpd the connection in clear through a pair of sockets, so every request and stream is the same; the key is checked against the certificate at start. Withcurland theopenaiSDK, four requests at once, clean under AddressSanitizer. Linux and macOS.web_searchin the Responses API: the model searches Wikipedia in the language it chooses (English when that has nothing), answers from the pages' first sentences and cites them as [n], which becomeurl_citationannotations, with aweb_search_callitem listing the query and the sources;--no-webrefuses it. With Qwen3-4B in English and in Italian, the answers came from the pages and cited the right ones.Batches (
/v1/batches, four operations): a JSONL file of requests checked, run one at a time behind every client's request (a chat reply in 0.12 s while a batch ran), each through the same code as the request it stands for, answered in an output file and an error file; saved after every line, so a batch goes on after a restart; cancelled or expired, what was done is kept. Tried with theopenaiSDK.Uploads (
/v1/uploads, four operations): a file of up to 8 GiB in parts of up to 64 MiB, joined into a File on completion and checked against its size and MD5 (src/common/md5.c); an upload lasts an hour. With theopenaiSDK, 150 MiB in three parts came back the same.Files (
/v1/files, five operations): uploads as the SDKs send them, with their purpose and expiry, kept in the store's directory under ids (a client's file name never a path), listed, read back after a restart, downloaded, deleted; 16 GiB in all at most, a gigabyte of disk left free; with--no-storein a directory removed at the end. The ground for batches, uploads and vector stores. Tried with theopenaiSDK.JSON output held to a schema's numbers and formats: integers to
minimum,maximum, their exclusive forms andmultipleOf(up to 50), exactly - a grammar of their digits that never begins a number it cannot end inside the bounds; other numbers to their bounds, without an exponent and with a fraction only where it cannot cross one; the formatsdate,time,date-time,uuid,emailandipv4as their grammar, through a small reader of regular expressions (src/llm/schema_re.c, there forpatternnext); string lengths up to 4,096 instead of 64. Every integer from -1,200 to 1,200 checked against five schemas; Qwen3-4B at temperature 1, 100 replies to a schema with all of them: 100 valid.One choice for how much quality the engine may give up for speed:
--quality high|medium|low(janas-chat, janas-server;min_qualityinjanas_llm_params,qualityin the Node module,Qualityin the FreeBASIC classes). Withlow, the default, the engine decides by the machine, as before: where the experts do not all fit in memory it reads them at four bits, and with less than a fifth of them in memory it leaves out the light ones not there rather than wait for the disk (Nemotron 3.5 Bolt with a 2.7 GB cache: 391 of 400 against llama.cpp where 396, 6% faster); with memory for every expert nothing is lowered.mediumallows the four bits but leaves no expert out;highlowers nothing. In janas-chat,--quality lowasked for also takes the catalog's levers measured faster on that model (the former--profile fast;--profilestays as the older name,qualitymeaninghigh). The single levers ---bits,--head,--attention, the experts per token - stay, and win over it.janas-server no longer falls over on a text part that is not a string: a message's
"text"given as an array or an object (chat completions and responses) had its length taken for its bytes and was copied past its end, and the server crashed; it is now a 400 (invalid_value). A refusal part's text is read from itsrefusalfield, as OpenAI writes it.Web pages of other sites cannot use the server: a page you visit could make the browser send requests to
127.0.0.1, and with a name of its own resolved there (DNS rebinding) read the answers. The browser'sOriginis now checked on every request: the server's own page passes, programs (which send none) are not concerned, any other page gets a 403 (origin_not_allowed) whatever the key;--allow-origin ORIGINlets a web client of another origin in, with the CORS headers it needs.MCP servers named in a request are reached at public addresses only: every address the name resolves to is checked and the connection goes to one that passed, so loopback, private, link-local (the clouds' metadata), shared, multicast and reserved ranges are refused, IPv4 inside IPv6 judged as itself;
--mcp-allow HOST[:PORT]names an MCP server of your own on this machine or the network.Stored documents have limits: at most 64 MiB each (a conversation that would grow past it gets a 400,
conversation_too_large) and 1 GiB in all, the oldest going first; adding to a conversation joins the texts instead of reading the whole of it back and writing it again.janas-chat no longer lets a model or a tool drive the terminal: what a model writes and what MCP tools return were printed as they came, so an escape sequence in them could move the cursor, switch screens or set the clipboard (OSC 52); the control characters and the C1 controls are now taken out, and a reply's tabs are four spaces. A third party's tool call is shown whole when you are asked to allow it (it was cut at 400 bytes, and
[a]lwayssaid yes to what was not shown)./load-configfreed the settings' text that options still pointed into (/save-configthen read freed memory); resizing the window no longer counts as "no" to a question or closes a page; two MCP servers whose names make the same prefix (a.banda_b) get different ones; the input history is readable by its owner only.janas-get and the tools: a download goes to
.partand takes its name only once its SHA-256 is checked (an interrupted one could pass for the model), a file Hugging Face lists no SHA-256 for is said to be unchecked, and a redirect from HTTPS to HTTP is refused; on Windows the arguments of a.cmd/.bat(npx) are always quoted and those holding",%,!or a newline refused, as cmd.exe would act on them; hf2gguf past 1,024 tensors, janas-nsm on a file with carriage returns only, and the version and the GPU's reason in build.sh's-Dstrings escaped.CPU kernels: a row ending inside a group with a shape its type cannot end in pads with zeros instead of copying past the row (the loaders refuse such shapes); the experts' bit planes are refused for any resident tensor (they could be sized but not unpacked); Q2_0's and MXFP4's tiles no longer read minima never written; NVFP4's scale table is filled at start rather than checked once a row.
GPU: fixes from the review: the keep-alive's empty dispatch wrote a float near the end of the stage that carries a card's results back, from another thread and in no order with the copies (a card with memory of its own could see four bytes zeroed): it has a buffer of its own; on a GPU whose subgroups have fewer than 8 lanes the row shaders would step 0 blocks at a time, for ever: they are not made there, the products go the other ways and the whole pass is refused; the attention's workgroups take at most 16 query heads and, with sixteen-bit keys, 2,048 numbers of them; Qwen3.5's gated query under
JANAS_GPU_REPACKcut every head from the first;JANAS_GPU_MVIDbelow the block size left the experts' tables too short for the types that always go by rows; host memory is imported only with a coherent memory type; the rotations ofqk_rope,ix_prepandix_poolare rounded twice as on the CPU, never fused by a driver; the memory a card gives back is counted again; one command buffer freed.gpu_checkon the laptop's Arc as before (3,980 products, 0 wrong), Qwen3.5-0.8B's whole pass 245 of 256 against llama.cpp.Nemotron's context slides right, and its mixture with a small cache: sliding the context (a long chat) turned back by the dropped positions keys that had never been turned, since Nemotron's attention has no RoPE: now they only move (
llm_slide, 400 tokens, 100 dropped: the same most likely token, mean |diff| 0.21, the next 24 tokens agreeing for 22). With an expert cache under a fifth of the experts, the experts left out because they were not in memory had the others' weights brought back to a sum of one, where Nemotron's router gives five; the weights now go back to the router's own sum, and the threshold is a share of it. With 2.7 GB of cache, 400 tokens of C against llama.cpp: 391 of 400 at 6.9 tokens/s with the experts left out, 396 at 6.5 without.A cache smaller than a block's experts: with fewer than 128 slots a prompt's block asked for more experts at once than the cache holds, and failed; the requests are now cut to the slots there are (Nemotron, 100 MiB of cache, blocks of 64: the same bits as a token at a time, 64 of 64 against llama.cpp). An expert that found no slot because reads ahead held them all was handed over as a null pointer: the reads ahead are settled and the slot sought again, and failing that the request fails instead of crashing.
Smaller fixes in the engine: a saved state with a recurrent log longer than its buffers or a base outside its text is refused; a pass that fails reading Gemma 4's per-layer inputs no longer loses the recurrent state first; a slide that cut the recurrent log went on replaying it from the wrong place, now the state goes on from the log's end; the MTP layer's own router and convolution copies are freed.
A model file must be what it says it is: a JNS file's resident tensors are now refused unless their bytes are what their type and shape make (a crafted file, downloaded, could have the engine read past the region), the experts' gate, up and down matrices must all have the shapes the buffers are made for (only the gate's rows were asked), and a Nemotron layer's router may name only experts present in the file and at least as many as it chooses. A GGUF tensor with an empty dimension is refused, and
gguf2jnsrefuses rows longer than 16 MiB (they overran its buffers), more layers than the engine reads, more experts used than there are, and metadata with arrays of arrays;hf2jns_mtpreads a safetensors header with every field checked. Every JNS file here opens as before, and one converted again comes out the same to the byte.Smaller fixes in the engine's plumbing: the expert cache sorted with a variable shared between caches, divided by zero on a file whose every slot is empty, and destroyed locks it had not made yet when an allocation failed; the NPU's graph cache now keys on the device too, is not used when the driver's version cannot be read, and two NPUs of one process write it without sharing a temporary name; a failed keep-awake no longer leaks when asked again; the thread pool returns NULL instead of ending the program when a thread cannot start.
Smaller fixes on the server and its clients: a prompt of token ids with a token longer than 255 bytes read past a buffer; response ids made of zeros if the system's randomness failed; the MCP client's header lines capped at 64, so a call's own pushed the configured
Authorizationout; a redirect from HTTPS to HTTP on the same host kept the credentials (now dropped on any change of scheme, host or port).
[2026-10-28] - NVIDIA's Nemotron 3.5 Bolt, and gpu_check asks what the engine asks
NVIDIA Nemotron-3.5-Bolt-74B-A9B on the CPU (architecture
nemotron_h_moe, NVIDIA's own 24GB GGUF: experts in IQ2_XXS trained at two bits, Mamba layers in NVFP4): 88 layers, each one part with its own norm and residual - 40 Mamba-2 mixers, 40 mixtures and 8 attentions without RoPE. Against llama.cpp, 400 tokens of C, the most likely token agrees at 395, mean |diff| 0.183; the tokens the same as llama.cpp's on seven texts. On the laptop's CPU, 12 threads, a token at a time with every expert in RAM: 6.8 tokens/s. What it took: the NVFP4 type (four sub-blocks of 16 with UE4M3 scales; dequantized as ggml to the bit, an AVX2 product the same bits as the reference); Mamba-2 (model_mamba.c: the conv with its bias, the scan in float, D, the gate and the grouped norm, on DeltaNet's two buffers, replay and log, so drafts work); the mixture in a latent space (the input brought down to 1,024 before the experts, their weighted sum brought up after), experts with relu^2 and no gate, routed by sigmoids with a bias for the choice, each layer with its own number of experts; Mistral's Tekken pre-tokenizer (pixtral).gguf2jnswrites layers without experts, experts without a gate and a table as wide as the layer with the most, the others' slots absent; the reader takes them only as a tail with zero checksums.The attention's way chosen once, so that a pass gives the same result every time: the choice between the query as it is and at sixteen bits (26 October) was measured on the calls themselves, a band of positions at a time, and so changed from one pass to the next: the same tokens could come out a few hundredths apart, a block's not the same bits as a token at a time (Nemotron: 0 of 64 positions). Now it is measured once, before the first call, on made-up keys of the model's geometry at 1,024 positions and up (a token's calls to 65,536, a block's to 16,384), and kept in
~/.cache/janas/attn.txtby processor, threads and geometry: the sixteen-bit way from the shortest context where it wins there and at every longer one. Blocks are again the same bits as a token at a time (Nemotron 64 of 64); Qwen3-4B after 4,096 tokens as before (writing 22.9 tok/s, prompt 98.0).JANAS_ATTN=floatorint16fixes it.Nemotron's NVFP4 on the GPU, and the NPU's graphs kept across sessions: NVFP4 is b32_matvec.comp's eighth kind (a block of 32 is half an NVFP4 block, its two sub-blocks' dots in two fmas as on the CPU), so a prompt's blocks give the GPU Nemotron's Mamba projections too, the same bits as the CPU (blocks of 64, the fast prompt's tiles apart: those round as they do for every type, and agree with llama.cpp as the CPU does, 254 of 256); its whole pass on the GPU is not there yet. The NPU's compiled graphs are now kept in
~/.cache/janas/npuby default instead of only whereJANAS_NPU_CACHEsaid: their preparation is paid once, not at every session (Nemotron: 36 s the first time, 3.8 s after). The engine's choice of the NPU and janas-bench's rows already leave that preparation out. On Nemotron the NPU takes the Mamba, attention, latent and shared products of the prompt's blocks; on the laptop it reads them a little slower than the CPU (19.2 against 20.2 tok/s), so the engine leaves it off there.gpu_checkno longer runs on the GPU a product the engine would leave to the CPU (issue #46): its "every row on the GPU" pass called the GPU without asking whether it takes the product, and one of its shapes starts a row into the buffer, two bytes off a 4-byte boundary for the blocks of 110, 210, 82, 50, 66, 74 and 98 bytes (Q3_K, IQ3_S, Q6_K, Q6_K_P, IQ1_S, IQ2_XXS/XS/S, IQ3_XXS). The engine never gives those to the GPU (janas_gpu_canwants their weights 4-byte aligned, as model files keep them); the Arc read them right anyway, AMD's driver did not, and gpu_check reported 164 wrong products on a Radeon 890M. Now they are counted as left to the CPU, as the engine computes them: on the laptop's Arc 3,980 products, 0 wrong, 220 left to the CPU.
[2026-10-26] - The GPU's attention nearer the CPU's
The attention's scores the way this processor computes them faster: in float over the query as it is, or over the query at sixteen bits (an integer sum), measured on the calls themselves - every 4,096 positions eight calls each way, a token's and a block's apart - in place of the sixteen-bit query only past 16,384 tokens of context. The two agree with llama.cpp alike: the most likely token at 2,961 of 3,200 positions with the sixteen-bit query, 2,963 in float (Qwen3 0.6B and 4B, Qwen3.5 0.8B and 2B, prose and code or instructions), mean |diff| 0.3033 and 0.3037. Qwen3-4B on the laptop's CPU, which takes the sixteen-bit way: a token written after 4,096 tokens 22.0 to 23.4 tok/s (the attention 10.7 to 8.2 ms), after 8,192 18.2 to 19.6; the prompts 95.9 to 99.6 and 74.0 to 78.7.
JANAS_ATTN=floatorint16fixes it.A long prompt goes the way the machine reads it faster: the GPU's whole pass, or the path without it (the CPU's attention, the products where they go), measured on the blocks themselves - every eight of the engine's blocks of positions one each way, in turn, and the faster for the rest - since the attention's cost grows with the position, and on the GPU it can grow faster or slower than on the CPU. On the development laptop's integrated Arc, Qwen3-4B: even from position 0 (8.51 and 8.52 ms a token), without it from 4,096 on (18.8 against 12.3 ms a token, at 12,288 41.6 against 25.0); a prompt of 16,384 tokens at 49.7 tok/s instead of 31.8. No threshold: a GPU with fast memory of its own measures its own way;
JANAS_GPU_LONG=0keeps the whole pass,JANAS_GPU_TOKEN=1too.The GPU's norms, the same bits as the CPU's, at a fraction of the time: the sum of squares' two floats added in a tree (seven or eight steps) instead of each thread going through every thread's in turn - the norms and quantization (norm_quant.comp) 352 ms to 56 a pass of 192 tokens of Qwen3-4B, q's and k's norms with RoPE (qk_rope.comp) 225 to 52, a prompt of 2,048 tokens read with the GPU at 112 tok/s instead of 93; the same bits as before (Gemma 4 12B, 367 of 400 against llama.cpp). A tile of a block's tokens a workgroup in the attention, each key read once for them, was tried and left out: eight times slower, the GPU's attention bound by its arithmetic and shared memory, not by reading the cache.
The fast prompt's Q4_K and Q6_K tiles can sum as the CPU does, opt-in (
JANAS_GPU_TILES_EXACT=1): a whole block staged per barrier, its integer sums, then the CPU's float accumulators - the same bits as the CPU on every product of Gemma 4 12B (the tiles as they are: errors of some 1e-6 of the largest output), and the 12B's blocks of 64 back to 235 of 256 against llama.cpp from 209. Not the default: this first version is several times slower than the tiles.JANAS_GPU_COPYis read by its value (=0 was taken as asked).A dense feed-forward whose down matrix is in blocks of 32 takes the CPU's activation on the GPU too: the deterministic SiLU or GELU quantized straight into the blocks of 32 (expert_act.comp's kind 1, as the experts in those types), in place of the GPU's e^x then a quantization apart (
JANAS_GPU_ACT_FLOAT=1as before). Gemma 4 26B's dense branch (Q8_0): against llama.cpp 321 of 400 instead of 315, the CPU's own (mean |diff| 0.794 instead of 0.806); DeepNSM-1B Q8_0 within the noise (464 and 465 of 477).Gemma 4 26B-A4B on the GPU's whole pass: its mixture as the CPU computes it - the dense branch (the shared expert, weight 1, no gate) on the feed-forward's norm and normalized apart, the experts' input normalized by its own norm, the router's input the residual normalized without weights times its scales over sqrt(d_model) (norm_quant.comp multiplies the normalized value first), each expert's output scale on its weight (moe_topk.comp), the experts' sum normalized and the dense branch added before the sandwich norm - and its experts' types: Q5_1 down matrices 704 wide through b32_matvec.comp by expert (the CPU's bits), the last layer's Q8_0 ones through q8_0_matvec.comp, which takes the experts' table too now; GELU in the experts' activation. Against llama.cpp, 400 tokens, the experts through a tier of 1 GiB in the card's memory: the most likely token at 315 (the CPU 321), mean |diff| 0.806 (0.817), against the CPU 332 and 0.616; blocks of 4 the same bits as token by token; no error from the validation layer with synchronization. Qwen3-Coder-30B-A3B as before (164 of 200, 2.904). The fast prompt's tiles round group by group: Gemma 4 12B, the most sensitive, in blocks of 64 at 209 of 256 against 234 exact (E4B 251 and 252, Qwen3-4B 233 and 237): to be made exact.
Gemma 4 E2B and E4B on the GPU's whole pass: their per-layer embeddings - the inputs the CPU makes for each layer (the table is read from the file, never on the GPU) go with the block, and after each feed-forward the branch on the GPU, its f32 gate, GELU times the layer's inputs in a shader of its own with the CPU's operations, the f32 projection back, its norm, the output scale after it - and the layers that read another's keys and values (their queries alone); the feed-forwards' widths may differ layer by layer (E2B's 6,144 and 12,288). Against llama.cpp, 400 tokens: E2B Q4_K_M the most likely token at 386 (the CPU 391), mean |diff| 0.427 (0.470), against the CPU 388 and 0.428; E4B 392 (394), 0.281 (0.286), against the CPU 390 and 0.271; blocks of 4 the same bits as token by token, the same as a card with memory of its own, no error from the validation layer with synchronization in either.
Gemma 4's dense layers on the GPU's whole pass, and its norms the same bits as the CPU's: the token's shaders take a layer's own geometry (Gemma 4's sliding-window layers, heads of 256, eight KV heads, a window of 1,024 kept in a ring, a RoPE table of their own; its full layers, sixteen heads of 512 on one KV head, split among four workgroups of four, V as K before its norm), V normalized without weights, scores not scaled, keys and values at sixteen bits as the CPU keeps them (the same cache for both, q in float, the scores float products), the sandwich norms of the attention's and the feed-forward's outputs and the layer's output scale, GELU in the activation's shader with the CPU's operations, the logits' soft cap after. Gemma's prompts move with any rounding of the products' inputs, so the norms now give the CPU's bits: the sum of squares in two floats, the mean rounded once, square root, reciprocal and amax / 127 correctly rounded, quantized as janas_q8k_quantize (the q and k norms, V's, the attention's output too; the experts a card's tier leaves to the CPU quantized the same way). Gemma 4 12B Q4_K_M, 400 tokens against llama.cpp: the most likely token at 367 (336 with the GPU's own rounding; the CPU 362), mean |diff| 1.01 (2.80; the CPU 0.90); blocks of 4 the same bits as token by token, of 64 too with the exact products (235 of 256, 0.97; the fast prompt's tiles 209 and 2.71: to be decided for Gemma); the same as a card with memory of its own; no error from Vulkan's validation layer with synchronization, both ways, tokens and blocks. The common models within the noise, before and after: Qwen3-0.6B 372 and 367 of 400 (mean 0.330 and 0.326; the CPU 375, 0.320), Qwen3.5-0.8B 383 and 381 (0.126, 0.127), Qwen3-4B 362 and 364 (0.733, 0.736), DeepNSM-1B 459 and 465 of 477 (0.086, 0.092), Qwen3-Coder-30B-A3B with a tier of experts 165 and 164 of 200 (2.897, 2.904). Not yet: E2B and E4B (per-layer embeddings, layers sharing their keys and values), the 26B mixture.
The query at sixteen bits in the GPU's attention, opt-in (
JANAS_GPU_Q16=1): the GPU's whole pass put q at eight bits for its integer products with the cached keys, a relative error some 4e-3 where the CPU keeps q in float; now, if asked, at sixteen, two eight-bit halves (q = 256 hi + lo), each score two integer products, 256 hi.k + lo.k. Heads up to 256 only (longer ones stay at eight bits: the sum could pass 2^31). Against the CPU, the most likely token the same at 463 of 477 positions instead of 457 and mean |diff| 0.072 instead of 0.090 (DeepNSM-1B Q8_0, as far from the CPU as the CPU is from llama.cpp), against llama.cpp 461 instead of 459; Qwen3-0.6B 234 of 256 instead of 228, 0.158 instead of 0.191; the same with a card with memory of its own (JANAS_GPU_COPY=1); no error from Vulkan's validation layer with synchronization checked. The default unchanged until its speed is measured.
[2026-10-25] - Llama 3, and DeepNSM's explications
- Llama 3 models, and a LoRA adapter merged into one:
tools/hf2ggufreads a Llama 3 checkpoint from Hugging Face (safetensors) and, if given, a PEFT LoRA adapter, merges the adapter (W + alpha/r B A in float, rounded to bf16, as PEFT's merge does on a bf16 model; the adapter's whole token tables, grown for its padding token, in place of the model's) and writes a GGUF of llama.cpp's "llama" architecture, which llama.cpp's ownllama-quantizequantizes andgguf2jnsconverts: both engines then read the same weights.gguf2jnsputs Llama's query and key rows back in the checkpoint's order (llama.cpp's converter permutes them for its rotation of adjacent pairs). The engine runs architecturellama(no norm on the query and key heads, Llama 3's RoPE factors), its pre-tokenizer (llama-bpe: numbers in runs of up to three digits) and its chat format (<|start_header_id|>,<|eot_id|>,<|begin_of_text|>at the start, also of raw text); on the CPU for now. Checked with DeepNSM-1B (Llama-3.2-1B with the adapter of arXiv 2505.11764, which writes a word's explication in the Natural Semantic Metalanguage): the merged weights the same to the bit as an independent merge in numpy, the file from Meta's weights the same byte for byte as from unsloth's copy; the tokens the same as llama.cpp's (476 of 476, digits, spaces, accents, Japanese and special tokens); the most likely token as llama.cpp's at 466 of 477 positions in Q8_0 (469 with 16-bit keys and values, the difference flat along the positions), 456 in Q4_K_M; prompt blocks the same bits as token by token. Its 149 benchmark words, Q8_0 on the laptop's CPU, DeepNSM's sampling: 57.8 tokens/s writing (llama.cpp on the same file and machine: 37.9 on 6 threads, 39.8 on 12), 270 reading the prompts (201, 144); the explications' legality by the paper's measure 5.36 +- 0.12 (llama.cpp 5.22, the paper 5.13), one sample a word, one run each. The GPU's whole pass takes them too since 26 October (it normalized every head of q and k; Llama has no such norms): against llama.cpp the most likely token agrees at 459 of 477 positions in Q8_0 (the CPU 466) and 450 in Q4_K_M (456), the same as a card with memory of its own, no error from Vulkan's validation layer with synchronization checked. janas-nsm: explications in the Natural Semantic Metalanguage, counted, and an output head restricted to their words: the command writes a word's explication with a model trained for it (DeepNSM: one word with its examples, or a batch of JSON lines, the model card's prompt or the paper's, its sampling) and counts it as the paper does - primes, function words and the other words, legality 10 (primes - molecules) / words, circularity (a form of the word in it) - or counts explications given (--score, no model). Two lists of primes: the paper's as its scores were computed (without "i" and "said": with them its records do not come back), and Goddard and Wierzbicka's 65 with their forms and phrases; NLTK's function words; the lemma by English suffix rules, a silent e put back, irregular forms. On the paper's 1,341 published records: circularity the same as spaCy's on all of them, primes and molecules on 1,337.--strictlets the model write only the primes, the function words, a list of molecules (its own default, or--molecules FILE) and punctuation: the library'sjanas_llm_chat_vocab(FreeBASICChat.Vocab) copies those tokens' rows of the output head into one matrix and the pass computes them alone, the others -inf (janas_llm_model_set_vocab; on the CPU, the copies being unknown to the GPU and the NPU; a pass whole on the GPU computes the whole head and is cut to the vocabulary after). The vocabulary's logits the same to the bit as the whole head's, token by token and in blocks, the whole head back unchanged after (vocab_check: DeepNSM-1B Q8_0 and Q4_K_M, Gemma 4 E2B with its soft cap; also under asan). DeepNSM-1B Q8_0 on the laptop's CPU, its 149 benchmark words, five seeds each way alternated: 3,841 tokens of 128,257, writing 57.9 to 72.1 tokens/s (the prompts as before, 266), legality 5.33 to 5.48 (+0.15, 95% interval +0.08 to +0.22, paired by word), primes +0.8 points, molecules -0.7, circularity and length within the noise. But the explications tell their word from its neighbours less well: another model, Gemma 4 E4B, ranking the word among four WordNet neighbours (sister senses, common words) from the explication, puts it first for 81.9% of the free explications and 75.8% of the strict ones (-6.0 points, -10.1 to -2.7; nine words lost, none won: leg, child, chat, chair, sour, candle, cut, elbow, sank; one seed, one judge), against 88.6% for WordNet's own glosses and 20.8% for another word's explication (chance 20%). The default writes freely.test_nsmcounts explications by hand with both lists, UTF-8 punctuation, circularity and the lemma's rules. DeepNSM is CC BY-NC-SA: not shipped, converted bytools/hf2gguffor research.
[2026-10-24] - A card with memory of its own runs the whole model
- A mixture larger than the card's tier: the CPU's experts computed while the GPU computes its own, a round trip less a layer: where the tier does not hold every expert a pass goes in a submission a layer, the CPU's plan between. The plan's last copies (experts read into the tier, the CPU's rows) went in a submission of their own, waited for: now in the next segment's submission, after a barrier. And a segment is two submissions: the GPU's products of the experts, not waited for, while the CPU computes the experts left to it; then their rows' copies and the rest of the layer, from the experts' sum, waited for (
cpu_part; one submission as before where the CPU has none). Qwen3-30B-A3B UD-Q3_K_XL as a card of its own on the laptop's Arc, a tier of 1 GiB, 480 tokens one at a time: the CPU's plans from 43.8 to 12.5 ms a pass, the time between segments from 51 to 29, the second half from 9.0 to 9.9 tokens/s. The same logits to the bit; with drafts the same tokens as without, prompt blocks included, and no error of the validation layer, synchronization included.JANAS_GPU_TOKEN_PROFILE=1prints a pass's experts in the tier, read into it and computed by the CPU, and the CPU's and the cache's time. - Every pass whole on a discrete GPU: on a card with memory of its own the model's passes - a prompt's blocks, a chain of drafts to verify, single tokens - go whole to the GPU, one submission each, where before every product's rows were shared with the CPU and the activations crossed the bus at each one (an Arc B580 read a prompt at 15.8 tokens/s, issue #43). The token's state - each attention layer's KV rows and scales, the Gated DeltaNet states and their log - is copied to the card's memory and has one owner at a time: the GPU after its pass, the CPU after its own, copied over only when the passes change hands (
janas_gpu_sync), only the rows the GPU's layers use, so that an MTP block's and Gemma 4's assistant's own stay the CPU's.JANAS_GPU_WHOLE=0shares the products as before. Checked as a card of its own on the development laptop's Arc (JANAS_GPU_COPY=1), functional runs only: Qwen3.5-2B, Qwen3-4B and Qwen3.5-9B greedy, with and without drafts the same tokens, the text sensible; Qwen3.5-2B's 64 tokens 141 single tokens, 27 draft checks and 2 blocks whole on the GPU, the state sent three times and never read back; a prompt on the GPU and its tokens on the CPU (JANAS_GPU_WHOLE=0) right too. As on an integrated GPU, the whole pass's norms and attention sum in the GPU's order, so the text is not the CPU's to the bit. - The whole pass's working buffers in the card's own memory: on a card with memory of its own every intermediate of a pass is in its memory, where the shaders read and write it at its speed, and only what the host writes or reads - a block's inputs and position, the parameters once, the logits and hidden states - goes through a host-visible mirror (janas_gt_sync_rangecopies those ranges); before, every layer's activations crossed the bus. The same tokens to the bit as with the buffer in host memory (Qwen3.5-2B, Qwen3-4B, Qwen3.5-9B, as a card of its own; a prompt on the GPU and its tokens on the CPU too). Mixtures too: their routing, read and written by the host between layers, goes through the mirror as the rest; Qwen3-30B-A3B's prompt blocks whole on the GPU gave the same tokens through the mirror as without, also under the validation layer (JANAS_GPU_TOK_MIRROR=1puts the buffer in the GPU's memory behind a mirror on an integrated GPU too, to test it with the weights read in place; the verbose line says where the buffer is).JANAS_GPU_TOK_HOSTBUF=1keeps the host-visible buffer for any model. The staging buffers and a card's copies of the weights now declare the copies they take part in (transfer usage), which a driver may enforce. Not yet: mixtures larger than the card, Qwen3.8-Flash-Next on a card of its own. - A dense model's pass in one submission with its copies: the copies of a pass's inputs (the embeddings, the rotations, the position words) to the card and of its logits and hidden states back to the mirror are recorded in the pass's own commands, with their barriers, instead of four submissions of their own around it, each waited for; the same tokens to the bit (Qwen3.5-2B with and without MTP drafts, Qwen3-4B against the host-visible buffer). Under the Khronos validation layer with synchronization checks, gpu_check on every type and Qwen3.5-2B's generation with drafts, on the shared memory and as a card of its own: no error.
- A mixture larger than the card: a tier of experts in its memory, the CPU computing the others: on a card with memory of its own a mixture's whole pass no longer needs every expert copied to the card. It keeps a tier of slots in the card's memory - the room it has left, 85% of its memory less what Janas took and 1 GiB for the rest (
JANAS_GPU_TIER_MB=<n>asks for n MiB), in pieces of 1 GiB that any driver allocates - and between two layers, where the CPU already sorts the router's choice, each chosen expert is either in the tier, or read into it in place of the one used least among sixteen (never one this layer counts on), or computed by the CPU from the layer's normalized input into its own rows, which the GPU then sums with the others in the router's order (janas_m_expert_rows, the CPU's expert kernels). Only the experts not in the tier are asked of the cache, so a tier hit never reads the disk. The arena is not copied to such a card any more, so the tier has its memory (before, a cache holding every expert was copied whole, and a copy that did not fit could leave pieces behind).JANAS_GPU_TIER=0turns the tier off. Checked as a card of its own on the development laptop's Arc, functional runs only: Qwen3-30B-A3B UD-Q3_K_XL with a tier of 1 GiB (480 of its 6,144 experts) and of 300 MB (140, a slot given up at nearly every layer) - sensible text, every pass whole on the GPU, no error under the validation layer with synchronization checks. The text is not the same to the bit with every tier: the two tiers' first 15 tokens are the same, then they part, as the run with drafts parts from the one without after 8-9 tokens - an expert computed by the CPU or by the GPU rounds differently (the GPU's expert products sum in their own order), and which computes it depends on what the tier holds. - Every fma() of the shaders one operation, as on the CPU: a mixture's tier no longer changes its answer: glslang decorates a
precisevariable's multiplies and adds NoContraction but not itsfma()calls, and GLSL.std.450's Fma without it may be done as a multiply and an add, two roundings - the Arc's driver did split some: the experts' activation (expert_act.comp, its e^x and reciprocal) came out a unit in the last place off the CPU's in about 1.5% of its values, which moved a block's scale in one expert in eighty. The shaders' SPIR-V now goes throughtools/spv_fma.c, which decorates every Fma (built bybuild.shwith the build machine's compiler,HOSTCC, and writing the C array that--vnwrote). With that, and the CPU's experts quantizing the GPU's input with the same deterministic operations (janas_q8k_quantize_det), an expert computed by the CPU gives the GPU's bits: Qwen3-30B-A3B's logits on 160 tokens are the same to the bit with a tier of 300 MB, 1 GiB and every expert. Found with a probe that recomputed each GPU expert on the CPU (input, gate and up the same to the bit, the activation not). Every block's experts now go through the matvecs by default (JANAS_GPU_MVID=16as before: a prompt's blocks in*_mmid's tiles), Q6_K_P's too (q6kp_matvec.compwithID, its planes and level from the table): a prompt's GPU experts went through tiles that are not the CPU's bits and its other experts through the CPU, so the tokens with drafts and without parted after some ten (Qwen3-30B-A3B with a tier of 300 MB at the 11th, Qwen3.6-35B-A3B with one of 1 GiB at the 40th); now they are the same, 48 of 48 on both. The prompt as a card of its own on the laptop's Arc: 7% faster with every expert in the tier, 3% slower with 300 MB. - A mixture's whole pass on a card of its own 2.2 times as fast on a single token: measured as a card of its own on the laptop's Arc (Qwen3-30B-A3B UD-Q3_K_XL, every expert in the tier, 160 tokens of fixed text one at a time, the second half): from 4.7 to 10.5 tokens/s; with a tier of 300 MB, the other experts on the CPU, from 3.1 to 8.5; Qwen3.6-35B-A3B Q4_K_M with a tier of 1 GiB 7.0 tokens/s. Four causes, found with the GPU's own timestamps: (1) between two layers the CPU's plan was read and written in six or seven copies of their own, each a submission waited for - now the choice of experts (and, with a tier, the experts' input) goes to the mirror at the end of a layer's commands and the tables come back at the start of the next one's, and the experts read into the tier and the CPU's rows go in one submission together (
janas_g_copies); (2) the router's product went throughdense_mm's tiles of 64 rows by 64 vectors, three workgroups on a token - nowdense_mv.comp, a workgroup a row, on up to 8 vectors (also the DeltaNet's beta and alpha, and qwen4exp's mixes, these not tried: the whole pass refuses them on a card of its own); (3)moe_topk.comphad one invocation a token search 128 logits nine times - now a workgroup a token, each search a reduction; the router from 1.07 to 0.11 ms a layer; (4) the experts' products on a few vectors went through*_mmid's tiles of 16 vectors, nearly empty on a token (9 GB/s) - now through the matvec shaders of Q3_K, Q4_K and Q6_K with a constantIDthat reads the weights, inputs and outputs from the same table (blocks of 16 tokens or fewer,JANAS_GPU_MVID=<n>, 0 never): gate and up from 55 to 26 ms a token. Every step checked: logits the same to the bit before and after the copies were moved; the same agreement with the CPU (145-150 of 160 tokens' most likely token, as Qwen3-4B's whole pass, 154); no error under the validation layer with synchronization checks; prompts as fast or faster (512 tokens 111.9 to 112.9 tokens/s, 120.7 with every block through the matvecs). With a tier holding every expert the tokens are the same with drafts and without; with a partial one they part after some tokens, as before: the CPU's experts do not round as the GPU's (open). - The tiles' transposition lighter: its weights asked for ahead, its loops unrolled, a block's 256 tokens at once: the transposition of a chunk of eight rows waited for memory (62% of its time on its loads, perf, Q4_K on 24 vectors) while the tiles that followed left memory idle: the next chunk's bytes are now prefetched while a chunk's tiles compute (
gemm_ahead, not Q8_0's: 3-4% slower there), the transposition's loops are unrolled so its registers no longer go through the stack, and the tiles hold the sums of 256 tokens, a prompt's block on the CPU, not 64 (32 for the types in blocks of 32), so each chunk is transposed once a block instead of four to eight times. On 32 vectors (12 threads) Q4_K 8.4 to 9.5 GB/s, Q3_K 5.6 to 6.4, IQ4_XS 7.8 to 9.0, Q5_K 10.5 to 11.4; on 256 6-15%. A prompt on the CPU with the experts in memory: Qwen3-30B-A3B UD-Q3_K_XL (1024 tokens) 116.4-117.7 to 121.4-123.4 tokens/s, Qwen3-4B Q4_K_M (512) 115.2 to 121.2-121.4. The same bits (test_quant's rows to 300 vectors, past a block of 256; a mutation of the blocks' index caught).bench_kernelstakes up to 512 vectors. - Q2_K and the IQ types with grids as tiles (IQ3_S, IQ3_XXS, IQ2_XXS, IQ2_XS, IQ2_S, IQ1_S, IQ1_M): through Q6_K's tiles, a group of 16 weights a scale; the IQ types' weights as signed bytes (level times sign; with AVX-VNNI plus 128, the offset taken off with the scales' sums of activations, with AVX2 by sign and abs), their scales 1 + 2 s, the block's d with its factor; Q2_K's values 0 to 3, its mins' sums apart. The same bits (
test_quant's 11 rows to 65 vectors with all eight, AVX2 and AVX-VNNI; mutations of the offset, of abs and of Q2_K's dmin caught). From 12 vectors; on 64 (12 threads) 1.6 to 2.6 times as fast: IQ1_M 0.6 to 1.6 GB/s with AVX-VNNI, IQ3_XXS 1.2 to 2.8, Q2_K 1.5 to 2.6; with AVX2 1.5 to 2.2 times. - Q2_0 as tiles, and rows that end inside a group of 256 by rows: Q2_0's two-bit values (the weight plus 1) go through the tiles of the types in blocks of 32, a block of 64 two of them with its d in both, from 24 vectors with AVX-VNNI (on 64, 12 threads, 1.7 to 2.1 GB/s); with AVX2 its products on reordered activations stay, the tiles never faster. Rows that end inside a group of 256 - Qwen3.8-Flash-Next's down experts, 640 columns - no longer go one by one on many vectors: the whole groups of all the rows, then the tails padded 16 rows at a time, both through the rows' products and their tiles (
janas_dot_multi_rows_n); the down matrix (2560 rows of 640, one core) 1.5 times as fast on 8 vectors, 2.4 on 64 with AVX-VNNI, 1.7 with AVX2. The same bits (test_row_tailson 33 vectors and 37 rows; mutations of Q2_0's offset and of the tails' sum caught). - IQ4_XS as tiles too: its signed values from the IQ4 table transposed, a signed scale a 32-weight group; with AVX-VNNI the values plus 128 by vpdpbusd, a multiply by the scales, 128 times the scales' sums of activations taken off; with AVX2 sign and abs, the scales by madd; one fma a block as before, the same bits. From 12 vectors; on 64 with AVX-VNNI 2.4 times as fast (12 threads), with AVX2 39%. Qwen3.8-27B UD-Q4_K_XL (IQ4_XS where its rows are many), prompt of 256 tokens with every tile type, AVX-VNNI: 9.0 to 15.6 tokens/s. The tiles' scales are made with vectors (an 8 x 8 transpose of the rows' scale pairs): Q3_K's tiles 19% faster on 24 vectors.
- The types in blocks of 32 as tiles too (Q8_0, IQ4_NL, Q4_0, MXFP4, Q5_0, Q4_1, Q5_1): each block's values unpacked to bytes with an offset (Q4_0's 8 and Q5_0's 16 as stored, MXFP4's 12, Q8_0's and IQ4_NL's 128 with AVX-VNNI) taken off with the block's activation sum; with AVX2 the small values' maddubs summed in int16 (eight for Q4_0 and Q4_1, four for the others), Q8_0 and IQ4_NL by sign and abs; the floats theirs, a sum per block position added as janas_hsum8, the same bits. On 64 vectors with AVX2 (12 threads) from 28% (Q8_0, IQ4_NL) to 70% (MXFP4) faster; with AVX-VNNI, whose rows' products had no reduction tree, from 24 vectors, 8% (Q8_0) to 33% (Q5_1). Gemma-4-26B-A4B UD-Q4_K_M's prompt of 512 tokens with every tile type, AVX2: 45.3 to 58.4 tokens/s.
- Q3_K as tiles too: its three-bit values (the weight plus 4) through Q6_K's tiles, four maddubs summed in int16 before a madd (AVX2), the -4 folded as Q6_K's -32; the same bits. Qwen3-30B-A3B UD-Q3_K_XL (its experts Q3_K), prompt of 512 tokens: 57.9 to 67.5 tokens/s with AVX2, 69.0 to 78.6 with AVX-VNNI.
- Q5_K as tiles too: five-bit values unpacked to bytes and transposed, four maddubs summed in int16 and one madd by the scales, twice a 32-weight piece (AVX2), or eight vpdpbusd and one multiply (AVX-VNNI); the mins and floats Q4_K's, the same bits. Qwen3.5-9B Q4_K_M (its attention and recurrent projections Q5_K), prompt of 512 tokens with every tile type: 30.7 to 43.3 tokens/s with AVX2, 38.3 to 57.5 with AVX-VNNI.
- Q6_K and Q6_K_P as tiles too: the eight rows' six-bit values unpacked to bytes (from Q6_K, or from Q6_K_P's planes at its level) and transposed the same way, a 16-weight group four dwords: two maddubs summed in int16 and one madd by the scales (AVX2), or four vpdpbusd and one multiply (AVX-VNNI); the same bits (
test_q6k_tiles: 11 rows, 9 blocks, 1 to 65 vectors, every level, also under Wine). With Q4_K's tiles, Qwen3-4B Q4_K_M's prompt of 512 tokens: 58.7 to 79.1 tokens/s with AVX2 (1.33 times llama.cpp's on the same machine), 73.0 to 103 with AVX-VNNI; Qwen3.5-9B Q4_K_M, the model of issue #45, 30.7 to 40.1 with AVX2.JANAS_Q4K_GEMMis nowJANAS_CPU_TILES. - A prompt's Q4_K products on the CPU as tiles of eight rows: on eight or more tokens the Q4_K rows are taken eight at a time, transposed four bytes at a time into a small buffer once for 64 tokens, and multiplied by tiles of tokens with every lane a row - a 32-weight piece eight maddubs summed in int16 and one madd by the scales (AVX2), or eight vpdpbusd and one multiply (AVX-VNNI), and no horizontal sums; the same integer sums and float operations, so the results are the other kernels' to the bit (
test_quanton 1 to 65 vectors and 11 rows, also under Wine). Qwen3-4B Q4_K_M's prompt of 512 tokens on the laptop's CPU, 12 threads: 57.1 to 72.0 tokens/s with AVX2 (llama.cpp on the same machine: 56.0, 59.4 on 6 threads), 71.2 to 92.2 with AVX-VNNI; Qwen3-30B-A3B Q4_K_M 54.6 to 60.7 (AVX2) and 63.9 to 70.2 (AVX-VNNI).JANAS_CPU_TILES=0keeps the rows' products.JANAS_ROW_PIECE_KB(off by default, not measured where it may count) takes a chunk's rows in pieces that stay in the level-2 cache, for CPUs with little of it (issue #45). - A mixture whose experts the card holds all: its pass in one submission, the plan on the GPU: when the tier has a slot for every expert, every one is read into it at the first pass (layer by layer, the next one read ahead) and
moe_plan.compwrites after each router what the CPU's plan wrote between two submissions - each row's token, each pair's row, the products' tables and tiles, the dispatches' sizes - the same words in the same places; the 49 submissions a token, each waited for, become one. Qwen3-30B-A3B UD-Q3_K_XL as a card of its own on the laptop's Arc, every expert in the tier: one token at a time 10.5 to 13.3 tokens/s, blocks of 16 50.2 to 53.9; the logits the same to the bit as with the CPU's plan, token by token and in blocks; drafts and none the same tokens; no error under the validation layer. The time between segments, measured past the tier's filling, was 16 ms of 88 a token, 2 of them the CPU's plan and the rest the submissions' round trip.JANAS_GPU_GPLAN=0keeps the CPU's plan; a watched router (janas_llm_model_route_observer) keeps it too. - MXFP4 and Q2_0 experts the same bits on the CPU and the card: on a card of its own a mixture's MXFP4 and Q2_0 experts go through
b32_matvec.comp's rows reading the plan's table, as Q3_K, Q4_K and Q6_K's already did, instead ofb32_mmid.comp's tiles; Qwen3.6-35B-A3B MXFP4_MOE as a card of its own on the laptop's Arc: the logits the same to the bit with tiers of 1 and 2 GiB, drafts and none the same tokens, no error under the validation layer, one token at a time 4.1 to 4.5 tokens/s with a tier of 1 GiB. Q2_0's rows of 640 take the tail ofb32_matvec.comp; Qwen3.8-Flash-Next is not yet taken whole by a card of its own, so that path is compiled, not run. - The CPU's plan in the token profile:
JANAS_GPU_TOKEN_PROFILE=1prints the time of a mixture's plans on the CPU beside the time between two segments; on Qwen3-30B-A3B with every expert in the tier, past the tier's filling, 2.1 ms of the 16 ms a token between segments: the rest is the submissions' round trip. - The GPU's timestamps of a mixture's layers:
JANAS_GPU_TOKEN_PROFILE=1now times a mixture's segments too, with the router, the experts' sum and the time between two segments (the CPU's plan, the submission's round trip) as phases of their own. - The state read back right from a cached stage: the Y stage may be cached and not coherent (it is on the laptop's Arc): a read-back is invalidated, an upload flushed - without it the CPU read stale state and wrote only token 0.
llm_gen:LLM_GEN_DUMP=<path>writes the first run's tokens.
[2026-10-24] - The GPU checked when it opens, and gpu_check diagnoses by itself
- gpu_check finds the faulty shader by itself: every product runs twice, all its rows on the GPU and then shared with the CPU as the engine does, and nothing stops at the first error (it stopped after ten, which on an Arc B580 were all one Q6_K shape). Every wrong product is counted under the shader that computed it (
janas_gpu_last_shader: name, vectors a workgroup, subgroup size), with its worst difference in units in the last place - a few: the driver rounds a float operation differently; many: wrong values - and the first wrong row with both values in hexadecimal (the printed ones looked equal on a Radeon 890M). Then it runs each wrong product again with the other shaders the engine could take for it (janas_gpu_set_path: the subgroup-per-row shaders and the ones with the lanes on the vectors off) and with every other subgroup size the device allows, and prints a summary to paste into an issue. The device line names the driver, its version and the float controls.GPU_CHECK_TYPES=checks some types only. - The engine checks the GPU when it opens it: a product of every type the GPU takes, on the vector counts that pick each of its shaders, against the CPU's, bit for bit (small random matrices, 110 ms on the Arc of the development laptop, 270 ms as a card of its own). A shader this driver gets wrong is left out - the subgroup-per-row ones, the ones with the lanes on the vectors, q4k_mv, each family with another shader for the same products - and a type still wrong without them goes to the CPU; the status line says so (
GPU present (active, ...; the subgroup-per-row shaders (wrong on this driver))). On the Arc B580 of issue #43 a Q6_K shader gave text with no sense; with the check the answers would be right from the first run. The result is kept in~/.cache/janas/gpu_selfcheckunder the device, its driver's version, the subgroup size and Janas's version: once per driver.JANAS_GPU_SELFCHECK=0leaves the check out,=2runs it again. Tested with a Q6_K row shader broken on purpose at subgroups of 32, as a card of its own: found in 312 ms and left out, and the next opening read the decision from the cache. janas-diagpackages:vkcheckandgpu_checkalone, without the engine and with a page of their own, for Linux and Windows in every release (under 1 MB on Linux).- The sums of a row's tail kept apart from its products on any GPU: in
q8_0_matvec.compandb32_matvec.compthe accumulators and their sums areprecise, so that no compiler may fuse a sum with the product before it - the CPU rounds both - where a tail's accumulators restart from zero inside the loop (Q8_0 2560x640 one bit off on a Radeon 890M, issue #46, the likely cause). The same bits on the Arc of the development laptop, in place and as a card of its own, subgroups of 8 and 32.
[2026-10-24] - Faster kernels on the CPU: Qwen3.8-Flash-Next's Q2_0 experts, IQ2 and IQ1, Q2_K, prompts of Q4_K and of the types in blocks of 32
- A reply 13% faster, a prompt 7.5% faster, the same bits: on the CPU (Core Ultra 9 185H, 12 threads, a 12 GiB expert cache), 1,024 tokens of Italian prose and then 200 generated, two runs each alternated with the release of 24 October, a reply went from 10.7-11.0 to 12.3-12.5 tokens/s and the prompt from 24.2 to 26.1; the experts of a reply from 30 ms a token to 19.8.
- Q2_0 rows against activations reordered once for all the rows: a Q2_0 byte holds four weights one apart, so each row was unpacked into the activations' order with shuffles, and the eight sums of 32 of each group of 256 reduced with six hadd. Now the activations are laid out as the weights' bit planes once per call (
q2_0_xperm), each row's 16 bytes become two registers with two variable shifts and two ands, and two hadd are enough. One vector: 7.7 GB/s on a core and 34 on twelve threads before, 19 and 69 now; two, four and eight vectors (a prompt's experts,janas_dot_multi_rows): 1.8, 1.7 and 1.5 times as fast. Sums of integers in another order and the same float operations: equal to the scalar reference to the bit. - The IQ2 and IQ1 kernels 2.5 to 4.5 times as fast, Q6_K's on 2-8 vectors 8-12%: the loops over a group's four sub-groups (IQ2_XXS/XS/S, IQ1_S/M) and over Q6_K's two halves were left rolled by GCC, their grid entries, signs and scales kept in memory; unrolled, one vector on a core went from 1.0 to 3.0 GB/s (IQ2_XXS), 1.2 to 5.4 (IQ2_S), 0.9 to 3.0 (IQ1_S), 0.6 to 2.3 (IQ1_M), and on twelve threads from 7.0 to 14.1, 8.4 to 20.9, 5.7 to 13.4 and 4.0 to 11.1. IQ1_M's scale is converted by F16C as the others', not by a call in the loop. The same bits.
- Q2_K a third faster on one vector: a block's sixteen scales are spread to its sub-groups by one shuffle each from a register, not loaded and broadcast one by one, and a vector's two sums reduced together. One vector from 8.6 to 12.9 GB/s on a core and from 37 to 49.5 on twelve threads, 2-8 vectors 8-20% (the UD-Q2_K_XL quantizations). The same bits.
- Q4_K on 5-8 vectors 9-13% faster, Q5_K 6% (
janas_dot_multi_rows): what the products need of the vectors alone - their scales, their sums of 32 two by two laid out one int32 lane a vector - is made once for all the rows of a call, so that a row's mins on eight vectors are four madd where they were eight and six hadd, and its scales one load where they were eight. On eight vectors 5.7 to 6.45 GB/s on a core, 23.1 to 25.1 on twelve threads. A prompt of 512 tokens on the CPU against the release of 24 October, with the IQ and Q6_K changes above: Qwen3-4B Q4_K_M 61.4-62.5 to 73.2-73.4 tokens/s, Qwen3-30B-A3B 59.0-59.7 to 63.0-63.8; replies the same or 1-2% faster. The same bits. - The types in blocks of 32 about twice as fast on 2-8 vectors with AVX-VNNI (Q8_0, Q4_0, IQ4_NL, MXFP4, Q5_0, Q4_1, Q5_1: a prompt's products): each group's eight blocks transposed in registers and the vectors' activations transposed once for all the rows of a call, so that eight
vpdpbusdleave each block's sum in its own lane - no sign tricks and no tree of six hadd a vector. The signed weights go unsigned with an offset (128, or Q4_0's 8 and Q5_0's 16) and the offset times each block's sum of activations is taken off, exactly. On twelve threads and eight vectors Q8_0 from 25.5 to 52 GB/s, Q4_0 14.0 to 28.4, IQ4_NL 14.0 to 26.7, MXFP4 14.2 to 25.7, Q5_0 17.2 to 29.1, Q4_1 16.2 to 26.5, Q5_1 17.0 to 26.2; on two vectors Q4_0 40 to 68, Q8_0 78.5 to 87.6. Without AVX-VNNI, and on one vector, as before. The same bits. - IQ3_XXS 40-48% faster, IQ2_XXS and IQ2_XS 30-55%: the signs of a 7-bit index, with the parity that completes them, read as eight bytes of multipliers from a table of 128 made at start (as ggml's ksigns), four at a time, instead of worked out and spread to bytes in the loop. One vector on a core IQ3_XXS from 4.65 to 6.9 GB/s, IQ2_XXS 3.0 to 4.8, IQ2_XS 3.45 to 4.8; on twelve threads 17.5 to 24.5, 14.2 to 19.6, 14.6 to 19.0. IQ2_S, whose signs are stored as bytes with no parity, keeps its way. The same bits.
tests/bench_kernelsmeasures every type the CPU multiplies (Q5_0, Q2_0, Q2_K, IQ3_XXS, IQ2_XXS/XS/S, IQ1_S/M added), with the IQ types' block scales normal fp16 numbers as in real weights (random bytes made some subnormal, and the float products slow), and the types named alone withBENCH_TYPES=Q4_K,Q2_0.test_quantchecks the rows of Q2_0 against 1-17 vectors at once, and the rows of Q2_0, the K types and the types in blocks of 32 on 2, 48 and 130 groups of 256 (the vectors made ready a few at a time, and not at all), against the single products and the reference.
[2026-10-24] - The chat's text on a Windows console in any language, gpu_check in the packages, the NPU made ready sooner
- Fixed: janas-chat on a Windows console printed every character past ASCII as mojibake under a locale whose code page is not UTF-8 (issue #44, Ukrainian: "в–„" for "▄"). The chat asked for the user's locale, which on Windows it never used - the widths of characters are counted there without it - and msvcrt then translated what was written to the console from that locale's code page. Not asked for on Windows any more; the rest of the systems as before.
gpu_checkin the Linux and Windows packages: every kind of product the GPU takes, run on the GPU Janas would use and compared with the CPU's, its last lineokorFAILEDwith the shapes that differ above it (hardware); for the cards that answer wrong with the GPU and right withJANAS_GPU=0(issue #43).- The NPU made ready on the compute threads: each matrix it takes dequantized and quantized again to INT8 a few rows a thread, the BF16 mixes turned to FP16 the same way, its inputs and outputs moved on the threads too - each value as before, so the graphs already compiled are found in the cache again. Qwen3.8-Flash-Next's 422 matrices went from 19.3 s to 11.4 s, the wait the chat shows as "preparing the NPU".
[2026-10-23] - Qwen3.8-Flash-Next's mixes of the streams faster on the CPU, at eight bits faster than BF16
- A reply 8.7% faster, a prompt 10% faster, the same bits: on the CPU (Core Ultra 9 185H, 12 threads, a 12 GiB expert cache), 1,024 tokens of Italian prose and then 200 generated, two runs each alternated with the commit before, a reply went from 10.07 to 10.95 tokens/s and the prompt from 21.75 to 24.0; the mixes of the streams from about 30 ms a token to 19.7. Against llama.cpp, 400 tokens of C and 400 of Italian prose, every position gives the same digits as before.
- Products of four rows or four tokens with their sums in registers: GCC did not unroll the short loops over rows and tokens, kept the accumulators in memory and made every fused multiply-add a load and a store; unrolled, the Q8_0 mixes' product read from L3 at 40 GB/s ran at a quarter of its speed before. Among them
janas_bf16_dot4, the four tokens of a prompt's BF16 products. - A single token's rows four at a time (
janas_bf16_dot4r,janas_q8w_dot4r), so that their chains of fused multiply-adds overlap, and rows shorter than 2 KB with their weights prefetched 2 KB ahead, where the next rows lie: a mix's up (10240 x 320) on 12 threads from 131 to 93 us in BF16, from 107 to 56 in Q8_0; its down (320 x 10240) at 82 GB/s. - Rows claimed as a share of those left in the BF16 and Q8_0 products (4 to 256, as
matvec.cclaims them), not in fixed chunks of 16: a down of 320 rows no longer ends on threads waiting for the last chunks. - A mix's own work on the threads: each stream of each token normalized by a thread of its own, a single token's mean split in pieces of 256, the rounding to bf16 eight at a time (
janas_round_bf16_n): a single token's norm from 24.8 to 6.1 us, its mean from 8.6 to 5.2. JANAS_HC_MIX=q8now faster than BF16: a reply at 11.9 tokens/s (BF16 10.95; 7.68 before these changes), a prompt at 24.0; the scale of each Q8_0 block by F16C, exact. Its quality is as measured: the most likely token at 392 and 364 of 400 positions against BF16's 390 and 367. The default stays BF16.tests/bench_longandtests/llm_evalcount the mixes in a phase of their own,mix(with the output's mix, which went intooutputbefore).test_quantchecks the four-row products against one row at a time and the rounding to bf16 against the scalar rule, ties, infinities and NaNs among the values.
[2026-10-23] - Qwen3.8-Flash-Next's prompt whole on the GPU, its mixes of the streams at eight bits
JANAS_HC_MIX=q8in the whole token on the GPU: the mixes' down and up rewritten as Q8_0 are multiplied there too (dense_mm.comp's third kind, Q8_0 weights on float inputs rounded to bf16), so the mixes at eight bits no longer keep a prompt's blocks on the CPU. Blocks of 256 tokens against llama.cpp: on C code the most likely token agrees at 247 of 256 positions (the BF16 mixes on the GPU 247), on Italian prose at 238 (230), and the mean divergence from llama.cpp's probabilities is the CPU's with the same mixes, KL 0.0180.Qwen3.8-Flash-Next's sparse attention on the GPU too: in each attention layer the indexer's keys and queries, the blocks the run completes, every token's scores of the blocks before it and its choice of the best 512 (a threshold found a byte at a time, the blocks in their order, those tied with it the earliest, as the CPU chooses) -
ix_prep,ix_pool,ix_score,ix_select- and the attention over the positions chosen (attn_decode.compandattn_merge.compread them from a list; the dense positions are a list too). Past 2,051 positions a prompt's blocks no longer go back to the CPU. The blocks chosen are the CPU's, every one, at every token looked at. Against llama.cpp, the last 512 positions of 8,192 tokens - 512 of 2,048 blocks kept: the most likely token agrees at 481 (the CPU 482), mean logit difference 0.183 (0.191); of 2,200 tokens the 149 positions past the dense ones agree at all 149 (0.111; the CPU 0.117).Fixed: after a prompt's blocks on the GPU the CPU's indexer keys were zero (since the prompt went whole to the GPU, earlier today): every attention layer wrote its input for the indexer into the first layer's row, so past 2,051 positions the CPU chose its blocks from keys of zero, decoding too; at 8,192 tokens the most likely token agreed at 381 of 512 positions instead of 481.
Qwen3.8-Flash-Next's prompt blocks whole on the GPU (the integrated GPU here, which shares the host's memory): its four streams, their mixes in BF16 (
hc_norm,hc_act,hc_mean,hc_comb), the n-gram layer (ng_gate,ng_conv), the Q2_0 experts and the shared expert at their width of 640, a layer a submission with the CPU choosing the experts in between. A mixture whose cache holds fewer than half its experts, as here with 12 GiB, goes there withJANAS_GPU_MOE=1. Against llama.cpp, blocks of 256 tokens: on C code the most likely token agrees at 247 of 256 positions (the GPU's products beside the CPU, 248); on Italian prose at 230 (the same 235 and 238 in two runs, the CPU alone 235), and the mean divergence from llama.cpp's probabilities (KL) is the lowest of the three, 0.0172 against 0.0176 and 0.0186. Speed not measured yet.Q8_0 rows ending inside a group of 256 on the GPU (
q8_0_matvec.comp): the whole groups' sum and then the tail's, as the CPU adds them, bit for bit (gpu_checknow checks them); Qwen3.8-Flash-Next's shared experts' down matrices are such rows.More than 256 experts at once from the cache (
janas_expert_cache_fetch_many): a block of 256 tokens of prose asked the first layer for 368 of its 512; requests of 256 that count as one, so that none takes a slot an earlier one gave. WithJANAS_GPU_VERBOSEa block that goes back to the CPU says so.tests/llm_evalwrites the logits of the blocks too (bdump=F).JANAS_HC_MIX=q8: the mixes of the four streams (each layer's two, and the output's), 1.26 GB of BF16 read at every token, rewritten as Q8_0 when the model opens - half the bytes - and multiplied by a product of their own, Q8_0 weights against float inputs rounded to bf16 as the BF16 products have them (janas_q8w_dot, eight AVX2 lanes, the same bits without AVX2; on the CPU). Against llama.cpp, 400 tokens of C and 400 of Italian prose: the most likely token agrees at 392 and 364 positions, mean logit difference 0.138 and 0.173, where in BF16 at 390 and 367 (0.135, 0.166). The way the other quantized products go - inputs quantized to eight bits - cost more than the weights' eight bits did: 388 and 359 (0.185, 0.204), whatever the weights' type (Q6_K gave the same), and four-bit weights (Q4_K, Q5_0) 382 and 349 even with float inputs; neither is kept. A prompt read in blocks gives the same bits as token by token, and the reply the same tokens with MTP drafts and without. The default stays BF16 until the speed is measured.--head 4on an output head at five bits (Q5_K, Qwen3.8-Flash-Next's): requantized to Q4_K at load as a Q6_K head is, 437 MB a token to 358; against llama.cpp 387 and 351 of 400 (0.184, 0.234), so a lever of the fast kind, off by default.
[2026-10-22] - Qwen3.8-Flash-Next in janas-get, its long context checked, its memory counted
janas-get qwen3.8-flash-next: downloads ISTA-DASLab's two-bit GGUF (two parts, 66.4 GB) and converts it here, then unsloth's GGUF of the MTP block from unsloth's repository, whichhf2jns_mtpcopies; every file checked against the SHA-256 inMODELS.md. Its weights are under the Qwen Community License 1.0, not Apache 2.0, and no converted file of it is published. The catalog takes an MTP block from a GGUF of its own in another repository (JANAS_EXTRA_BLOCK,extra_repo). Checked on the files already here: the model's fingerprint, the block's GGUF and the block converted, all asMODELS.mdsays.MODELS.md: Qwen3.8-Flash-Next's sources, revisions and fingerprints, and its licence in the paragraph on what this project redistributes;docs/features.md: what runs and what was checked.- The memory estimate (
janas_llm_estimate, andjanas_llm_open's automatic cache) counts qwen4exp's buffers for a block's residual streams and n-grams, about 94 MB on Qwen3.8-Flash-Next. Measured with a cache of 4 GiB, a context of 8,192 and a prompt of 4,873 tokens: 7.79 GiB estimated, 8.02 GiB at the peak; what is left over (0.2 GiB, there from the open: the tokenizer, converted copies of some weights) is not counted for any model, and the open leaves a fifth of the memory to other programs. - A long context checked: 8,192 tokens, the last 512 positions against llama.cpp on the same file - the most likely token agrees at 482, mean logit difference 0.191; the same 512 tokens read after only 512 others: 482 again, 0.178. llama.cpp's own logits are the same read in batches of 256 and of 512. The prediction file is found above 2 GiB (
janas_llm_find_mtp): a block of 3 GiB beside the model is taken, one larger than a quarter of the model is not.
[2026-10-22] - Qwen3.8-Flash-Next in the chat: its reasoning, its tools, a prompt the same every run
- The prompt Qwen3.8's template writes, token for token: with the reasoning on, the template's words on the reasoning effort at the top of the system message ("Reasoning effort is set to xhigh. ...", read from the template, not written into Janas: Qwen3.8-27B's too), with or without a system message of the caller's, before the tools; and in a conversation loaded whole, every earlier reply of the assistant with its empty reasoning (the template's
preserve_thinking), not only those after the user's last question as on Qwen3.5 and 3.6. Six cases checked against the template rendered by Jinja - reasoning on and off, a system message, a tool, a conversation with a call, its answer and a reply - the same tokens. - A service opened in the chat is then called, not opened again: after
janas__open_servicethe chat requires the next reply to call one of that service's tools. Qwen3.8-Flash-Next (two bits a weight) opened thesystemservice again and again, until the chat stopped it - the service shares its name with its tools' prefix (system__), and the model's choice fell on "jan" at -0.27 against "system" at -1.44.janas_llm_chat_tool_choicetakes a name withJANAS_LLM_TOOLS_REQUIREDfor this: the calls only to the functions whose names begin with it (NULL or "" as before). The catalog of the services tells the model that most questions need none of them - writing, code, explanations - and the system service is "this computer itself, when the user asks about it": a question of C code opens nothing now, one about the free memory opens the service and calls its tool. An opened service's tools come one a line in<tools>, as the system message lists them. JANAS_GPU_REPEATABLE=1: every product of a mixture's experts on the CPU, exact, and in a prompt's blocks the GPU only with the fast prompt's own products, so the same request gives the same reply from one run to the next, and the same with drafts and without. By default an expert's product went to the GPU's rounded tiles when the GPU could read its slot of the cache, and to the CPU when it could not - which changes as the cache fills - and products shared with the CPU, begun as the tuner tried one configuration or another, left the GPU busy for the fast ones asked meanwhile: Qwen3.8-Flash-Next's reply to the same question changed from run to run, from the first token's probability on, with drafts or without them. Found by halves (the dense products, the tiles by weight type, rows ending inside a group), the CPU alone giving the same digits every run on the same busy machine. The three ways and their difference are in features. The chains of drafts the GPU verifies no longer take the experts' rounded products in one launch (a chain of 13 tokens or more on a mixture of ten experts a token did).- Checked on Qwen3.8-Flash-Next: its reasoning (on by the chat's default for a reasoning model) closed by the model, kept apart from the answer, the same text with and without drafts on the CPU; the conversations kept in memory and on the disk (
tests/llm_keep, which now takescache=MiBfor a machine busy with other work), the model's saved state with them.
[2026-10-22] - Qwen3.8-Flash-Next's MTP block
- Drafts from Qwen3.8-Flash-Next's own prediction block, the one unsloth publishes beside its GGUF files (
MTP/mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf, 2.8 GB):hf2jns_mtpnow takes such a file of the block alone, besides a checkpoint, and copies the block as it is, in its own types (Q4_K, Q5_0, Q6_K and Q8_0), into a file of one layer (1.9 GB, the same bytes on every run). The block works on the model's four streams, as llama.cpp's graph does: each stream of the last layer's residual normalized with weights of its own, beside the token's normalized embedding, and projected back a stream at a time - not pooled - into streams of its own; then its layer as the model's, its attention dense, its streams kept for the next draft of a chain, and a final mix of its own before the model's output head. On 158 positions of C code its guess was the model's next choice at 130 (82%), the text's next token at 116; a greedy reply of 48 tokens with chains of three drafts came out the same, token for token, as without them, 32 of its 51 drafts kept. In the chat, tojanas-bench's question on bread, 64 tokens of the reply with no reasoning: 29 of 40 drafts kept greedy (72.5%), 33 of 41 sampled as the chat samples (80.5%), each reply word for word the one written without drafts; unsloth measured 64.4% for its Q4_K_M block, greedy, with another file of the model and texts of its own. On the GPU with its products bit for bit (JANAS_GPU_FAST_PROMPT=0) the four replies are the CPU's own; with the fast prompt, the default, the GPU computes a group's experts in one dispatch, summed in another order, and the replies change, with drafts or without. With the NPU reading the model's blocks of the C code the guesses agreed at 130 of the 158 positions too. The block's own indexer for the sparse attention is copied but left unused, as llama.cpp leaves it. - An MTP file's experts are all kept in memory unless the caller asks for less (
mtp_cache_bytes): the memory the API plans for a model already counted the whole file. The old cache of 256 MiB held 74 of this block's experts of 3.6 MB, fewer than a block of tokens asks for at once (128), and the block failed; now a cache is never smaller than that. Qwen3-Next-80B-A3B's block keeps its 0.9 GB of experts whole too, where 256 MiB held 151 of its 512. - The chat finds a prediction file beside a large model up to a quarter of the model's size, and at least 2 GiB as before: the fixed 2 GiB left out the block unsloth recommends for Qwen3.8-Flash-Next, its Q8_0 (2.6 GB). Qwen3.5-2B, Qwen3.5-9B, Gemma-4-E4B, Gemma-4-26B-A4B and Qwen3.8-Flash-Next find the files they found before, Qwen3-4B none; a file above 2 GiB was not tried, there being none here.
tests/mtp_evalreads a row of hidden state as wide as the model gives it (janas_llm_model_hidden_width: Qwen3.8-Flash-Next's four streams, 10,240 numbers), takes the expert cache's size (cache=MiB), and says which step failed.
[2026-10-22] - Q2_K and the IQ1, IQ2 and IQ3 types on the GPU
- Weights in Q2_K, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ1_S and IQ1_M on the GPU, the types of Qwen3.8-Flash-Next's and Qwen3.8-27B's other GSQ-RCO files and of unsloth's 1- and 2-bit ones: their products shared with the CPU, bit for bit as the CPU computes them (
iq2_matvec.comp,iq3xxs_matvec.comp,iq1_matvec.comp,q2k_matvec.comp, with ggml's grids copied as data), a prompt's tiles (iqk_mmq.comp) and a mixture's experts in one dispatch (iqk_mmid.comp), which also takes experts in IQ3_S and IQ4_XS, until now left to the CPU.tests/gpu_checkfinds 980 more shapes the CPU's bits,tests/gpu_fast_check60 more products within 1e-4 of the CPU's. No file of the catalog has these types: none of its models changes.
[2026-10-22] - Qwen3.8-Flash-Next
- Qwen3.8-Flash-Next (
qwen4exp), a mixture of 512 experts of 177 billion parameters, from ISTA-DASLab's GSQ-RCO files: its residual in four streams, mixed into the input of each layer's token mixer and feed-forward by gates of low rank in place of the norms, each sublayer's output added back to every stream with a weight of its own (hyper-connections); its table of n-gram embeddings at the second layer, 51 billion numbers read a row at a time from the file, sixteen rows of 160 a token; its DeltaNet layers' output gated by a sigmoid; and its attention's sparse selection, which past 2,051 positions attends only to the 512 blocks of four positions its indexer scores highest and to the query's own incomplete block. Against llama.cpp (build2b18470, the same Q2_0 file), the most likely token agreed at 390 of 400 positions of C code (mean absolute difference of the logits 0.135), at 367 of 400 of Italian prose (0.166), and at 2,171 of 2,200 positions of a longer text of code - all 148 past 2,051, where the attention is sparse (0.117). A pass over a block gives the same logits, bit for bit, as token by token, also when drafts are taken back. - Weights in Q2_0 (ggml's type 42: two bits in blocks of 64) and Q5_0, and IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS and Q2_K: the types of ISTA-DASLab's four GSQ-RCO files of Qwen3.8-Flash-Next and of Qwen3.8-27B, and of unsloth's 1- and 2-bit files. Unpacked to the same numbers as ggml's, bit for bit; their products on AVX2 and AVX-VNNI, on one to eight vectors, the same bits as the scalar reference.
- Rows that end inside a group of 256 weights: Qwen3.8-Flash-Next's experts are 640 wide, which only the types in blocks of 32 or 64 can hold.
gguf2jnskeeps that width (widened with zeros to 768, a fifth more bytes would be read at every token), and the products take the last group as one padded with zero blocks. - A saved state of
qwen4exp(the chat's conversations kept) holds what its n-gram embeddings and its indexer read back, and when the context fills up and the oldest exchanges are forgotten, the history of those embeddings and the indexer's keys move along with the rest. The memory the engine sets aside for a context counts those keys too, 4.5 KB a position, out of the expert cache's share. - Qwen3.8-Flash-Next's prompts on the GPU and on the NPU. Blocks of 256 tokens against llama.cpp, the most likely token: on the GPU at 248 of 256 positions of C code and 232 of 256 of Italian, on the NPU at 252 and 236, where the CPU token by token gives 246 and 235. The GPU takes Q2_0 and Q5_0 in its products shared with the CPU, bit for bit as the CPU computes them, rows of 640 among them; in the fast prompt's tiles, and the experts of a block in one dispatch (
b32_mmid.comp: Q2_0, Q5_0, IQ4_NL); and the bf16 mixes of the streams, 1.25 GB of weights read at every token. The NPU takes every type the engine unpacks, Q2_0, Q5_0 and the 1- to 3-bit kinds among them, rows of 640, a shared expert 640 wide in one run, and the bf16 mixes with their weights in FP16.tests/gpu_checkcovers the new types and rows,tests/gpu_fast_checkthe fast prompt's tiles of the blocks of 32 and 64 against the CPU. Not yet for it: the MTP block. - A reply's tokens on a card with memory of its own take the bf16 mixes too (
bf16_matvec.comp), bit for bit as the CPU computes them: its eight lanes of fused multiply-adds (src/llm/bf16.c), summed across a subgroup in the same order, the inputs and outputs through the GPU's stages. With the GPU taking single tokens (--reply-on gpu), Qwen3.8-Flash-Next gives the same logits over 32 tokens, byte for byte, as on the CPU alone - 219 bf16 products a token on the GPU - also with its weights copied into the GPU's memory as on a card (JANAS_GPU_COPY=1).tests/gpu_checkcompares 96 of them bit for bit, in place and through the stages. An integrated GPU, which does not take single tokens by default, is left as it was. - The NPU also takes the products whose outputs lie further apart than their rows - a recurrent layer's q|k|v and z side by side in one buffer, on Qwen3.5, Qwen3.6 and Qwen3.8 - which stayed on the CPU: on Qwen3.6-35B-A3B its blocks of 256 give the CPU's most likely token at 251 of 256 positions. The memory set aside for the NPU's copies counts every type it takes (only Q4_K, Q5_K, Q6_K and Q8_0 were counted), so that a model of other types does not have them on top of its expert cache.
- A per-layer table in memory where the machine holds it - Gemma 4's per-layer embeddings, Qwen3.8-Flash-Next's n-grams (28.8 GB) - read into memory at load when what is free holds it beside a cache of every expert, read a row at a time from the file only where memory is short;
JANAS_PLE_RAM=1asks for it,0never. Gemma 4 E2B gives the same logits, byte for byte, either way. The GPU is never given such a table, which it does not read: an integrated GPU's driver pins the pages of what it is given, the table's 28.8 GB among them.
[2026-10-22] - Janas's classes for FreeBASIC
include/janas/janas_oop.bi: Janas-LLM and Janas-MCP with objects, for FreeBASIC -Janas.Model,Janas.Chat, their options, a reply's counters, a conversation kept by the program, a handler that receives a reply as it is written through its overridden methods, and an MCP server's tools; objects that close what they hold at the end of their scope and cannot be copied, a Chat that knows when its Model closed; methods that return False and say why.include/janas/janas.bitakesllm.biandmcp.biin one include.tests/test_fb, in the suite where FreeBASIC is found, tries every class without a model;tests/fb_check, on a model, finds the classes' replies the library's own in C, byte for byte.- The chat in FreeBASIC moved from
tools/toexamples/fb-chat/, beside its version with classes,chat_oop.bas; its counters' line no longer overflows past 99.9 tokens a second. llm.bi: its last note gives the parameter blocks' sizes as they are, 64, 80 and 192 bytes (it said 56, 72 and 176).
[2026-10-22] - The module for Node.js in the Linux package
- The Linux package carries
janas.nodeand its types for TypeScript,janas.d.ts, innode/: a Node.js program opens a model with the package alone, without building Janas. The release workflow checks that the module, as packed, loads, where its runner has Node.js. The Windows package has no module yet.
[2026-10-21] - Janas-LLM in Node.js
janas.node, the library as a module for Node.js (Linux), made by./build.shbeside the programs, with its types for TypeScript (janas.d.ts):open()a model,model.chat(), thenchat.send(text)orchat.load(messages)and the reply read withfor await, piece by piece (the answer, the reasoning, how much of a long prompt is read), or whole withreply.text(), its counters withreply.stats(). Leaving the loop,reply.stop()or anAbortSignalstops it. Each model runs on a thread of its own, so JavaScript's thread never waits for it; a model's replies come one at a time, in the order asked; an idle model does not keep Node running, and one left open is closed when Node leaves. Written in C against Node-API (version 8), whose functions it declares itself: building it needs nothing of Node's (checked against Node's headers where they are installed). Tried with Node 20:tests/test_nodewithout a model, in the suite, andtests/node_checkon Qwen3-0.6B, where the replies through Node are the library's own in C, byte for byte; both also under AddressSanitizer and ThreadSanitizer.docs/library.md, "From Node.js".examples/node-chat, a chat in TypeScript on the module: a server onnode:httpthat opens the model in its own process and streams each reply to the browser as it is written (server-sent events), a page that keeps the conversation and shows the reasoning apart and what each reply cost, and the types of what goes between the two shared by both sides. A client that goes away stops its reply; requests from several pages wait in line, told how many are before them.
[2026-10-21] - Intel's NPU reads the prompts
- The NPU of Intel's Core Ultra processors reads a prompt's blocks (Linux; tried on the first generation's, Meteor Lake, "Intel AI Boost"). Janas-LLM writes the NPU's graphs itself, in OpenVINO's IR format, and opens Intel's Level Zero driver when a model opens: nothing is linked against it, and without it nothing changes. The weights it takes are copied in eight bits with a scale for each row and computed in sixteen-bit floats, the norms in 32. A dense model whose layers fit in the NPU's memory, nine in ten at least, goes to it layer by layer, whole - Qwen3, Qwen3.5 and Gemma 4's dense models - each block of the prompt in two halves, the NPU computing a layer on one while the CPU computes the attention, or Qwen3.5's recurrence, on the other; a larger model, or a mixture of experts, gives it the products of the weights that stay in memory. On the development laptop,
janas-bench --prompt 1024 --gen 16 --rounds 2, the best configuration with the NPU against the best without it (the GPU's fast prompt): Qwen3-0.6B 1,421 tok/s against 974, Qwen3.5-0.8B 1,403 against 950, Qwen3.5-2B 729 against 477, Gemma-4-E2B 508 against 196, Qwen3-4B 333 against 201, Gemma-4-E4B 245 against 129. A reply is written by the CPU as before. - On the same NPU, llama.cpp's OpenVINO backend (build
2b18470, OpenVINO 2026.4,GGML_OPENVINO_DEVICE=NPU): a prompt of 512 tokens of Qwen3-4B read at 120 tok/s, where Janas-LLM read it at 364, and Qwen3-0.6B at 419 against 1,542; it wrote at 5.0 and 12.0 tok/s, all on the NPU, where Janas-LLM wrote on the CPU at 26.6 and 136. The Qwen3.5 models and Gemma 4's did not run on it ("failed to decode prompt batch"), and from 768 tokens of prompt on it stopped on a segmentation fault. - The engine takes the NPU by itself only where it reads faster and its answers were measured as close to llama.cpp's as the CPU's: against llama.cpp's scores on two texts, Italian prose and code, the most likely token agreed at as many positions with the NPU as with the CPU, give or take 7 of 512, on eight models (Qwen3.5-0.8B and 2B, Qwen3-4B, Gemma-4-E2B and E4B, Qwen3-30B-A3B, Qwen3.6-35B-A3B, Qwen3-Next-80B-A3B). On Gemma 4's larger models, product by product, it agreed less (Gemma-4-26B-A4B 363 of 512 against 398): there it reads only when asked. On the mixtures and the dense models too large for it the GPU's fast prompt read faster on this laptop (Qwen3-30B-A3B 133 tok/s against 90-110 with the NPU), and the engine kept the GPU.
- The NPU is left off where its copies would take the expert cache's memory: on the mixtures whose experts do not all fit in memory (Qwen3-Next-80B-A3B, Qwen3-Coder-Next, Qwen3.6-35B-A3B) and on Qwen3.8-27B, its eight-bit copies came out of the cache, and Qwen3.8-27B wrote at 1.7 tok/s instead of 4.4. There it opens only when
--prompt-onnames it. - The chat shows the NPU being made ready. The first blocks it reads in a session wait while its weights are copied and its graphs compiled - 7.5 s for Qwen3-0.6B, 17 for Qwen3.5-2B, 24 for Gemma-4-E2B, about 50 for Qwen3-4B and Gemma-4-E4B - and the status line says how far it is ("preparing the NPU 12/25, 8 s"), wherever it happens: while the system message is read ahead, or in a reply; after the reply a line says how long it took.
janas_llm_npu_preparing(new inllm.handllm.bi) gives the same to any program, from another thread while the engine works. A block read while the expert cache still fills goes without the NPU, rather than make graphs that the whole layers drop a moment later (5 s of Qwen3.5-2B's first block). - What runs where, by hand:
--prompt-on,--drafts-onand--reply-oninjanas-chat,janas-serverandjanas-bench(andJANAS_PROMPT_ON,JANAS_DRAFTS_ON,JANAS_REPLY_ON) takeauto(the default),cpu,gpu,npu, two joined by+to work in the same pass (npu+gpu), or a list apart by commas the engine measures and chooses from (npu,gpu); the CPU takes part in every pass, the NPU in prompts only./save-configkeeps them.--npu-int8reads with eight-bit activations on the dense Qwen3 models, an experiment: Qwen3-4B 387 tok/s against 333, its scores further from the CPU's (a mean difference of 0.60 against 0.33, measured on 20 October). janas-benchmeasures the NPU's configurations, "+ NPU" and "+ GPU + NPU", prepares it in a warm-up of its own, outside the measures, and says when the engine would not choose it by itself on that model. Its lineNPU:names the NPU, or says why there is none to use.- A mixture's Q3_K experts go to the GPU in the fast prompt (
q3k_mmid.comp, one launch for a layer's experts, as Q4_K's, Q6_K's and MXFP4's), and with a whole token on the GPU (--reply-on gpu). Unsloth's 3-bit Qwen3-30B-A3B read a prompt of 1,024 tokens at 119.5 tok/s, where it read 98.5 with its experts on the CPU and llama.cpp reads 114.3: the one model of the README's table on which llama.cpp read faster no longer is. Given the CPU's quantized activations the new shader agrees with the CPU's products to their rounding: against llama.cpp's scores of Qwen's Q8_0 the most likely token agreed at 220 and 249 of 256 positions (Italian prose, code), against the exact way's 221 and 245. - An expert cache reaches the GPU in regions below 4 GiB: Mesa's Intel driver passes the size of an imported allocation on to the kernel in 32 bits, and a region of 4 GiB exactly - the second of unsloth's 3-bit Qwen3-30B-A3B, whose slots of 2,236,416 bytes fit 1,920 times 1 MB short of it - went as 0 bytes and was refused: the experts stayed on the CPU, with nothing said. None of the other mixtures measured had such a region.
JANAS_GPU_VERBOSE=1now says why an import is refused, and where. - The fast prompt's distance from the exact way, measured again, on Qwen3-30B-A3B against llama.cpp's scores of Qwen's Q8_0, Italian prose and code: the most likely token within 5 of 256 positions either way (unsloth's 3-bit file 216 and 246 against the exact way's 221 and 245, Qwen's Q4_K_M 210 and 239 against 207 and 244), the mean difference of the scores within 3%.
docs/features.mdsaid within 2-3 positions and 2%, from the models and texts measured on 10 October; on these two texts 8 to 27 of the 256 positions change their most likely token with the rounding, and they fell as far as 5 either side. - The drafts in the chat as in
janas-bench. The planner that chooses how many of the MTP block's drafts to verify learned what a pass costs from every pass, and in a new conversation the tuner tries each configuration in turn on the passes of 2 to 16 tokens, those that verify drafts: from the slow ones it tried (279-357 ms a pass where its choice takes 50) the planner learned that the drafts did not pay, and drafted one pass in sixteen. It now learns only from the passes run on the tuner's choice, without what they waited for the disk, and learns the costs again at each turn. Qwen3-Next-80B-A3B in the chat, a reply of 64 tokens with the cache warm: 31.2 tok/s with 32 drafts, where it came at 24.7 with 4; the first reply of a session 19.3 against 16.0; over five seeds 30.9-35.3 tok/s, the median 31.4. - The thread that keeps the NPU awake sleeps while it is not needed: it woke every half millisecond for as long as the program ran, 2,000 times a second beside the cores writing a reply.
janas_aligned_allocasks for a size the alignment divides, as C11 wants: ThreadSanitizer refusedtest_aligned's one byte on eight.- Two research probes, merged though not final, both off unless their variables are set: Ana-Net (the experts' down matrices by neuron,
JANAS_Q6_K_PNfromtools/jns_neurons.c, each token's neurons read at 6, 4 or 2 bits or not at all,JANAS_NEURON_LEVELS; records for the predictor's tests,JANAS_ANA_*), and Janet's detectors in place of a model's feed-forward, layer by layer (model_janet.c,JANAS_JANET_*). - The README: the banner back at the top; one table of the speeds, each engine at its best and the configuration that gave Janas-LLM its figure; the NPU, how it works, and llama.cpp's OpenVINO backend on it; the options that choose what runs where; the measurement notes of 21 October, and the conditions of the drafts' figures of 10 October beside the chat's of 21 October.
[2026-10-18] - Qwen3.6-35B-A3B's expert cache holds what it has room for
- An MTP block inside the model's file no longer halves the expert cache. Qwen3.6-35B-A3B's files carry the block that drafts tokens as a last layer whose experts are in Q8_0, 3,342,336 bytes each against about 2 MB for the model's own, and every slot of the cache was as large as the largest expert: the cache held little more than half the experts its memory had room for. The block's experts now have slots of their own at the end of the same memory, out of the same budget (room for one request, 128 experts, or for all of them when the budget holds every expert), and the model's experts slots of their size; the automatic cache,
janas_llm_estimateandjanas_llm_memorycount the new layout, and nothing changes where the block's experts are no larger. Qwen3.6-35B-A3B Q4_K_M with a 14 GiB cache,janas-bench --prompt 512 --gen 128 --ctx 4096, two series of two runs each alternated with the version before: prompt 132-133 tokens/s against 75-81, generation 27.3 against 18.1-18.9, with MTP drafts 39-43 against 32-33, the cache serving 98.6% of the experts against 91-92%. The answers are the same bit for bit (janas-try's fingerprints unchanged). Found by Rendeverance (#34); the fix is written anew. - Windows: no fault freeing a block that starts on a page.
freelooked for an alignment tag before every block it was given, and read the page before one that began on a page boundary, which may not be mapped: an access violation, seen on Windows on ARM64 (found by Rendeverance, #31). Blocks from the aligned allocator now go back throughjanas_aligned_free, the only one that reads the tag; the asan build uses the tagged blocks on Linux too, so a block given back the wrong way stops the tests (tests/test_aligned). - build.sh stops when a source listing fails, naming the directory or the file, instead of linking a library short with no message; a library of others with no flags of its own stops it too (found by phaedrus1992, #30).
- The GPU code builds on ARM64 with the Vulkan headers installed: it included the x86 intrinsics' header for one spin-wait hint.
- Code from contributors waits for the non-profit foundation that will hold Janas's copyright (CONTRIBUTING.md).
[2026-10-17] - The same answer whatever the C library and the compiler
- The engine computes exp, log, pow, sin, cos and tanh itself (
src/llm/jmath.c), in double with plain IEEE-754 operations, where it called the C library's: the rotary angles, the softmax of the experts' router and of the attention's pieces, the DeltaNet's softplus, decay and gates, the soft cap of the logits, the sampler. The C library's functions differ between versions and systems - glibc 2.41 rounds several of them correctly where 2.39 did not - and one unit in the last place was enough to turn a near tie between two tokens the other way: Qwen3.6-35B-A3B's answer at temperature 0 was another text on Ubuntu 24.04 (issue #38, reported by martinambrus). The float results are the double ones rounded once;tests/test_jmathchecks them against the C library in double over the ranges the engine uses them on (exp within 0 units in the last place, the others within one). - No compiler fuses a multiply and an add on its own (
-ffp-contract=off): gcc did in the kernels' functions compiled for FMA, each version where it saw fit - eight places, all in the scalar ends of vector loops; the fused operations the kernels write themselves stay. - janas-try's answer fingerprints: medium's is
13e2f198c270b826and gemma'sbf1626afffa7eeabnow, quick's and full's unchanged, each the same with the GPU, without it and with the AVX2 kernels forced. Against llama.cpp, on eleven texts of six models (Qwen3-0.6B, Qwen3.5-0.8B, 2B and 9B, Qwen3-4B against its own file and against the Q8_0's), the most likely token agrees at 4,111 of 4,400 positions, 4,108 before. - test_quant says whether it compared the scalar SiLU with the AVX2 one: on a CPU without AVX2 and FMA it passed as if a million values had agreed, comparing none (issue #26, reported by phaedrus1992); it prints how many it compared, or that there was nothing to compare with.
[2026-10-14] - A model's memory, before it opens and while it is open
- A model's memory, before it opens and while it is open:
janas_llm_estimategives what a model takes whatever its expert cache and what a cache holding all its experts takes, the reckoningjanas_llm_openmakes for its automatic cache;janas_llm_memorywhat an open model holds (resident weights, expert cache, keys and values).
[2026-10-14] - A mixture's experts on the GPU beside a shared expert of another type
- A layer's experts go to the GPU even where its shared expert is of another type. Where the expert cache holds less than half a mixture's experts, a prompt's experts go to the GPU a layer's group at a time, and a group opens with the layer's shared expert; when its type differed from the routed experts' - Q8_0 beside Qwen3.6-35B-A3B's Q4_K or MXFP4 experts, a plain Q6_K down beside Qwen3-Next's - the whole group stayed on the CPU. Now the CPU computes the shared expert beside the GPU's share. Qwen3.6-35B-A3B Q4_K_M with a 14 GiB cache: the prompt's rows with the GPU 3.7% faster (69.9 tokens/s against 67.3, four runs alternated), the most likely token agreeing with llama.cpp at 197 of 200 positions against 196.
[2026-10-14] - MXFP4 on the GPU, and the experts of a large cache back on it
- A mixture's experts reach the GPU again when the expert cache holds more than 4 GiB of them. Since the release of 13 October the memory the GPU reads is given to it in regions that start on 2 MB boundaries, each overlapping the one before, and the overlap was not counted in the 4 GiB a region may take: past the first, a region could come to a little more, which the GPU refused, and the expert cache was then not given to it at all. On an integrated GPU a mixture's experts stayed on the CPU, in the fast prompt and in the block read whole on the GPU, whenever the cache held more than about 4 GiB of experts, depending on the size of an expert: Qwen3.6-35B-A3B's MXFP4_MOE file beyond 4 GiB, its Q4_K_M at six bits beyond 8 GiB. Each region counts its overlap now, and
tests/test_import_regionschecks them. - MXFP4 on the GPU, which only the CPU read since this morning's MXFP4 products: the products of one vector to a few, to the same bits as the CPU; the fast prompt's tiles; and a mixture's MXFP4 experts in one launch per layer, the layer read whole on the GPU. noctrex's Qwen3.6-35B-A3B-MXFP4_MOE read in blocks of 100 with the GPU, against llama.cpp on the same file: 200 of 200 most likely tokens agree, mean logit difference 0.106 (on the CPU, token by token: 199, 0.105).
[2026-10-14] - An integrated GPU no longer kept awake for single tokens
- The GPU is kept awake only for the passes it takes. On an integrated GPU a reply's single tokens stay on the CPU, but a configuration with the GPU kept it awake for them too - an empty dispatch every half millisecond - and on a laptop the awake GPU takes the power the cores need: janas-bench's "+ GPU" rows wrote a reply slower than the CPU alone, 10.5 tokens/s against 17.3 on a Radeon 780M (issue #16, reported by dlahoza), 2-5% slower on the development laptop's Arc. Now they write as fast (Qwen3-4B, 20, 12 and 6 threads: 28.5/28.1, 28.0/28.0 and 26.3/26.3 tokens/s with "+ GPU", 28.5/28.3, 28.1/28.0 and 26.3/26.3 without, two alternated runs). A chat's own choices were not affected: the tuner gives single tokens to the GPU only where it takes them, a discrete GPU. janas-bench's note says that the single tokens are the CPU's in every row where so.
[2026-10-14] - MXFP4 experts, and output heads in BF16
- The "MXFP4_MOE" files run. Their routed experts are MXFP4 (OCP's microscaling format: 32 four-bit values and a power of two a block), which gguf2jns refused and the engine could not read. The engine has MXFP4 products now - IQ4_NL's kernel with the block's power of two for a scale, the same bits on AVX2 and AVX-VNNI, for one vector or eight, and the same bits as the exact Q8_0 rewrite of the same blocks - and gguf2jns keeps such experts as they are. noctrex's Qwen3.6-35B-A3B-MXFP4_MOE against llama.cpp on the same file: 199 of 200 most likely tokens agree, mean logit difference 0.105 (on Italian prose 63 of 64, 0.085), closer than bartowski's Q4_K_M to its own llama.cpp (196, 0.146; 60, 0.135). The file itself, against llama.cpp's Q8_0 of the same model, is further than that Q4_K_M (378 against 387 of 400 most likely tokens, 364 against 370 on Italian prose) and 9% smaller (20.3 GB against 22.3). It is also slower on the development laptop (janas-bench, two runs each in alternated order): a prompt at 85 tokens/s against 96, a reply at 25.3 against 27.0, and without drafts, where the Q4_K_M has its MTP block (38 tokens/s): its weights outside the experts are Q8_0 and the head BF16, 3.1 GB in the file where the Q4_K_M's are 1.9. On 12 threads of the development laptop the MXFP4 product reads 66 GB/s where IQ4_NL reads 73: 4% fewer weights a second, a block of it being 17 bytes to IQ4_NL's 18 (its scales come from a table: computed in the kernel, they took registers the unpacking needs and it ran 14% slower).
- An output head in BF16 or F16 is read as Q8_0. The "MXFP4_MOE" files keep theirs in BF16 (1,017 MB on Qwen3.6-35B-A3B), which the engine refused. It is rewritten once, as the model opens, the way llama.cpp's quantization makes Q8_0, into 540 MB - the head is read whole at every token - and the model's description says so; the file keeps the BF16. A token embedding table in BF16 or F16 is rewritten the same way.
[2026-10-14] - janas-try's answers, and the GPU's subgroup operations
- janas-try's answers match again, on every machine (issue #23, reported by clethbridgedaniel): since 1 October janas-chat starts Janas's own services and writes their catalog into the system message, even with
--system "", and janas-try asked its fixed question that way, while the expected fingerprints date from 25 September: no level could match, on any machine. The engine was right to the bit. Built from clethbridgedaniel's commit, here on an Intel Core Ultra 9 185H with AVX-VNNI and GCC 14, CPU only, Qwen3-Next-80B answers withf393d4bfb674402f, the fingerprint the report gave from a Ryzen 9 5900HX with AVX2 and GCC 16 - the same without the MTP drafts and with the AVX2 kernels forced - and with4aae064b918a0e75, the expected one, without the services; Qwen3-4B likewise gives its expectedc08f0f936eeba5cbwithout them. janas-try now asks without MCP servers and without the machine's measure (--no-mcp --no-tune), in configuration, cache and data directories of its own, so that neither settings saved with/save-confignor a system message kept from an earlier chat can change the answer. The medium level expectse49b0016cedf6c50now: Qwen3.6-35B-A3B's Q8_0 tensors take the activations with a scale per 32 since 27 September (the fix that brought Gemma-4-26B from 147 to 156 of 200 against an exact float64 pass), and janas-try still expected the answer from before. All four levels give their fingerprints here, with the GPU and without it. - A GPU without the subgroup operations the shaders use is turned down, and the reason names them (
no clustered, shuffle subgroup operations in compute shaders): besides ballot, 13 shaders use subgroup arithmetic, clustered reductions or shuffles, and the engine checked only ballot, so a driver without them would have been handed shaders it cannot build.vkcheckgives the same decision, where it used to warn that the engine did not check them yet.
[2026-10-13] - A ceiling on the reasoning
- The reasoning can be given a ceiling:
max_reasoningin the library's chat parameters,--reasoning-budgetandreasoning_budget_tokens(orthinking_budget_tokens, the names llama.cpp takes) on the server,--think Nand/think Nin the chat. A model still reasoning there has its reasoning closed for it, a sentence ("I have thought enough; now I give the answer.") and the closing mark written as if it had written them, and goes on with its answer; with no ceiling nothing changes, the replies identical to the bit. The sentence is what makes a small model answer: cut at 32 tokens with janas-chat's system message, Qwen3.5-0.8B, Qwen3.5-2B and Qwen3-4B ended 20 of 36 replies themselves given the mark alone, the small ones going on reasoning in their answer and mostly looping to its end, and 35 of 36 given the sentence, the replies a third as long (96 tokens on average against 265); Qwen3-4B answered either way.
[2026-10-13] - A build without the GPU, and a request that never ended
- A build without GPU support links again (issue #22, reported by mashirotamigi): two of the GPU's functions had no stand-in in such a build, and without
glslangValidatorthe link failed. - A glslang too old for the shaders builds without the GPU, and says so, where the build stopped at the first shader: the shaders need
GL_EXT_integer_dot_product, which glslang knows from 15.2 on, and Ubuntu 24.04's is 15.1.build.shtriesglslangValidatoron a probe first; the reason reaches the engine, which gives it when asked for the GPU.INSTALL.mdsays where to find a newer glslang on Ubuntu 24.04. - Issue #21, a server request that never ended on an RTX 5060 Ti with the tuner on, was issue #17's: on a discrete card the tuner tries the GPU for single tokens too, whose results never reached the CPU, and the model's reasoning, garbled, never closed; in a chat
max_tokenscaps the answer, not the reasoning, so the reply ran towards the end of the context. Reproduced here on the path a discrete card takes: with the code before the fix the request had not ended after 180 s, the server at 57 minutes of CPU; with it, the right answer in 7.5 s.
[2026-10-13] - vkcheck: can this GPU run Janas-LLM?
vkchecktells whether a GPU can run Janas-LLM's GPU backend, and why not: every requirement with the driver's answer (Vulkan 1.3, buffer addresses, the integer dot product, the subgroup operations the shaders use, the size of a subgroup, host memory imported in place), the decision the engine would take, then three tests on the device: zero-copy, the int8 dot product (exact, and a rough rate that shows whether the driver emulates it) and the float operations that keep the GPU's results the CPU's to the bit. On the development laptop's Arc everything passes, the dot product at 1,711 billion multiply-adds a second. It is a single C file that builds where Janas does not build yet, macOS with MoltenVK included: the first step of the arm64 and Apple Silicon port proposed in issue #20.
[2026-10-13] - Qwen3-Coder-Next as llama.cpp writes it now
- qwen3next models converted by llama.cpp since early 2026 load (issue #18, reported by nica-f with unsloth's Qwen3-Coder-Next-Q3_K_M): the recurrent layers' input projection comes as two tensors,
attn_qkvandattn_gate, where the engine wanted the singlessm_in, its rows interleaved per key head. Both layouts are read; Qwen3-Coder-Next rewritten in the new one gives the very logits of the old, to the bit, token by token and in blocks. - MXFP4 tensors written as Q8_0, exactly: the same file has its shared experts in MXFP4 (ggml's type 39), which the engine did not know. An MXFP4 weight is an integer of -12 to 12 times its block's power of two, and Q8_0 holds both, so
gguf2jnsrewrites such a tensor with every weight unchanged, at twice the bytes (80 MB more for Qwen3-Coder-Next's shared experts). Checked on a Qwen3-4B with four layers' attention in MXFP4: the converted tensors give the GGUF's weights to the bit. Routed experts in MXFP4 are refused with a message for now: twice their bytes on every token would undo what the format is for.
[2026-10-13] - janas-bench says what the GPU does
janas-benchsays that the GPU reads a prompt in every configuration (issue #16, a Radeon 780M): with the fast prompt a prompt's blocks of 64 tokens or more go to the GPU whether a row says "+ GPU" or not, so that a prompt gives the same results in each, and the rows without it had been read as the CPU alone, the GPU as of no help where it read Qwen3-4B's prompt 5.2 times faster (477 against 92 tokens/s). A note under the table now says so; "+ GPU" is the GPU's part in the shorter passes.JANAS_GPU_VERBOSEputs the calls the GPU does whole on lines of their own ("gpu alone"): the CPU waits there with nothing to do, and they had been counted as a GPU late, with a share that does not apply to them. Reporting only: the share is learned as before.- The hardware page tells what testers found: on the 780M subgroups of 32 (
JANAS_GPU_SG=32) cost 14% against the 64 the engine chooses, and on discrete cards the defect of issue #17.
[2026-10-13] - A discrete GPU's results reach the CPU again
- On a GPU with memory of its own the products' results reach the CPU again (issue #17, reported and diagnosed by xpil on a Quadro RTX 5000, with the fix's first form and its test): since the release of 10 October the engine registered its output buffers with the GPU, so that one sharing the host's memory writes them in place, but on a discrete card a registration is a copy in the card's memory, and the CPU went on reading what its buffers held before. Results are written in place only where the GPU shares the host's memory; elsewhere they come back through the staging buffer, as before 10 October.
test_gpu_output_copychecks it on any GPU (JANAS_GPU_COPY=1gives an integrated one the copy path): -12345 instead of 256 before the fix, 256 with it. On that path Qwen3-4B read a prompt in blocks of 64 tokens with the most likely token as read token by token at 0 of 64 positions before the fix (mean difference of the logits 5.73), and at 61 with it (0.098, the fast prompt's usual rounding).
[2026-10-13] - The GPU's passes of a few tokens
- Qwen3.8-27B's drafts verified on the GPU a little faster, 7.7 tokens/s in two runs against 7.6 and 7.0 (
JANAS_GPU_SHORT=1, 64 tokens of code): no chain longer than the GPU's shaders of a few vectors take (a ninth token doubled the pass: 854 ms against 437 for 5), and the recurrent layers' betas and decays read their inputs a quarter as often (3.7 ms a pass of 8 tokens where they took 13.5), with the same results to the bit. - Qwen3.8-27B writes 7.6 tokens/s with its drafts verified on the GPU (
JANAS_GPU_SHORT=1), against 6.3 on the CPU alone (64 tokens of code, in a series with each build alternated; the figures depend on the length and on the chains the planner chooses): the GPU's products of 2 to 8 vectors on Q5_K and IQ4_XS weights - two thirds of what the model reads - now take the lanes on the vectors too (q5k_mvd,iq4xs_mvd), a pass of 8 tokens 482.9 ms where it took 509.6, with the same results as the CPU to the bit. On 17408 x 5120, 2 to 8 vectors: Q5_K 1.23-1.38 ms against 1.48-1.54, IQ4_XS 1.85-1.96 against 2.36-2.42. - Qwen3.8-27B reads a prompt 19% faster on the GPU, 22.7 tokens/s against 19.0 in blocks of 100 tokens (
janas-bench, 1,024 tokens: 30.6 against 26.4; llama.cpp 25.6), and withJANAS_GPU_SHORT=1writes 8.7 tokens/s with its drafts against 7.0 on the CPU (128 tokens of code): two of its down matrices are IQ4_NL, in blocks of 32, which had kept every pass of the model off the GPU's single submission; such a layer now takes its activation in both layouts there, as a mixture's shared expert already did. Its logits as close to the reference as before (top-1 197/200, mean |diff| 0.095; the CPU 195/200, 0.098). - Qwen3.5-9B with its MTP block writes 24.2 tokens/s with its chains of drafts verified whole on the GPU, against 20.0 on the CPU (
JANAS_GPU_SHORT=1, still asked for only): the Gated DeltaNet layers' state is replayed and logged on the GPU as on the CPU, in the same log, so that a pass of drafts goes to the GPU in one submission and the next pass replays there the tokens kept. A short block's betas and decays have a shader of their own, a workgroup a row, where one workgroup took the whole block. A pass of 7.7 tokens: Qwen3.5-0.8B 49.6 to 38.8 ms, 2B 53.7 to 42.4, 9B 161.3 to 132.0. The 9B's logits as close to the reference as the CPU's (top-1 243/256 in blocks of 8, mean |diff| 0.098; the CPU 373/400, 0.099); the text not the CPU's, as the GPU's norms and sums go in their own order. - The GPU's products of 2 to 8 vectors on Q4_K weights take 12-25% longer than one vector's, where they took 21-98% longer (
q4k_mvd; 17408 x 5120: 0.83, 0.85 and 0.92 ms for 2, 4 and 8 vectors against 0.74 for one): the lanes of a subgroup take the vectors instead of the rows, so that a row's block is read 128 contiguous bytes at a time and its words reach every vector as scalar operands. The shaders before put a row on a lane, and each of their loads touched eight rows: the development laptop's GPU read the weights so at some 50 GB/s against 68. Qwen3-4B with its whole pass on the GPU, ms a token in blocks of 2 / 4 / 8 tokens (a draft chain's verification): 30.0 / 17.2 / 11.1 against 30.9 / 23.0 / 12.7, the same results bit for bit; at 8 tokens the GPU now takes 28% less than the CPU (15.5). On where a subgroup is 8 lanes (Intel);JANAS_GPU_MVD=0leaves it out. - The same for Q6_K weights (
q6k_mvd), whose 210-byte blocks start on a word or two bytes past one: each lane loads the words around its part of the block and shifts them into place. Qwen3-4B's output (151936 x 2560): 7.1 ms for 2 vectors and 7.6 for 8, where it took 10.1 and 16.9, and 9.6 for a single vector. Qwen3-4B with its whole pass on the GPU, now 28.3 / 16.0 / 9.5 ms a token in blocks of 2 / 4 / 8: from 4 tokens up the GPU takes less than the CPU (17.1 at 4, 15.5 at 8), the same results bit for bit. - Products of 9 to 63 vectors dispatched with the vectors' groups first: the groups of the same rows go out together and those after the first find the weights in the L3. 16 vectors of a 17408 x 5120 Q4_K matrix: 1.53 ms instead of 2.06, bit for bit as before.
q4k_mv, a Q4_K product of a row an invocation with a third fewer instructions, behindJANAS_GPU_MV=1while it is measured: no faster up to 8 vectors, faster beyond (16 vectors 1.22 ms against 1.53).bench_gemmputs the weights where the engine has them (BENCH_MEM=huge2m: 2 MB pages;driver: the GPU driver's memory). A correction to earlier figures: on a GPU sharing the host's memory the products of a single vector go to the CPU (JANAS_GPU_MIN_N), sobench_gemm's "GPU" times for one vector were the CPU's; the GPU's own for 17408 x 5120 is 0.74 ms.- Fixed on the way: the server's store, reading its conversations back from disk with none there, gave
qsorta null list (harmless in practice, undefined in C; found by the UndefinedBehaviorSanitizer suite).
[2026-10-12] - janas-bench takes a model by name
- The GPU's products of a few vectors on IQ4_XS, IQ3_S and IQ4_NL weights a fifth to a third faster: their table of levels (the IQ3_S grid) read from the workgroup's shared memory instead of a constant array; on 8 vectors of a 17408 x 5120 matrix, IQ4_XS 3.73 to 2.65 ms, IQ3_S 3.57 to 2.67, IQ4_NL 4.35 to 3.46, the results the CPU's to the bit.
- The memory a GPU sharing the host's imports starts on 2 MB boundaries, collapsed into huge pages before it is pinned: on the development laptop's GPU such memory reads at 71.6 GB/s against 59.6, and the CPU reads it as fast as before. No change yet on prompts or replies, where the GPU is bound by its clock and the CPU writes the tokens; it is for the passes the GPU would read whole.
- Qwen3.8-27B in the catalog (
qwen3.8-27b): unsloth's UD-Q4_K_XL, dense, its multi-token prediction block inside the file; converted already on Hugging Face (prabanta-dev/Qwen3.8-27B-JNS), its fingerprints in MODELS.md, a row in the README's table. - IQ4_XS, IQ3_S and IQ4_NL weights in a prompt's products on the GPU (the fast prompt's tiles, as Q4_K, Q5_K, Q6_K, Q3_K and Q8_0 already were): unsloth's UD files mix these types in, and they went through the slow path one vector at a time.
bench_gemm2048 x 6144: 260-600 GMAC/s, every output as the CPU's to within 7 parts in 10 million of the largest (bench_gemmnow checks the GPU against the CPU output by output). Qwen3.8-27B UD-Q4_K_XL reads a prompt of 1,024 tokens at 26.4 tok/s where it read 18.7, against 25.6 for llama.cpp with Vulkan; it writes at 4.4 tok/s against 2.1 (12 October 2026, the development laptop). Its answers against llama.cpp's: the most likely token the same at 194 of 200 positions through the GPU's blocks, 195 token by token. janas-benchtakes the model asjanas-chatdoes: a name of the models Janas knows (janas-bench qwen3-4b), downloaded after asking if it is not there yet; with no model, the list to choose from; a.jnsfile as before. The model's prediction file comes with it as in the chat - the one the catalog names, or for another file the one beside it - so the drafts are measured as the chat uses them;--no-mtpleaves it out,--modelssays where the models are. Both programs share one way of finding a model (src/common/model_pick.c).
[2026-10-12] - Janas on Windows
- The same sources build for Windows x86-64, with MinGW-w64 from Linux (
JANAS_OS=windows ./build.sh): every program, the two libraries as DLLs, Janas's services (janas-system with Windows's own calls), the GPU through Vulkan; nothing needed but Windows's own DLLs. Run on Windows 11 (build 26300) on the development laptop on 12 October 2026: all 31 tests pass, and with the same files and commands as on Linux it writes as fast (Qwen3-4B 25.6 tok/s against 27, Qwen3-30B-A3B 34.2 against 32) and reads a prompt of 1,024 tokens as fast with the mixture (Qwen3-30B-A3B 132 against 135) and 1.8 times as fast with the dense model (Qwen3-4B 346 against 193), whose whole blocks go to the GPU, there through Intel's own driver. - What differs between the systems has one place each: the reading of the experts (unbuffered overlapped reads on Windows, io_uring on Linux), the CPU's cores, the programs started (a job object holds what an MCP server starts in turn), sockets, the chat's terminal, time zones (Windows's ICU), the folders (
%LOCALAPPDATA%\janas,%APPDATA%\janas). - The cores on Windows: the low-power efficiency cores are told from the others by the level-3 cache they do not share; the clocks Windows reports had kept them and left out the eight ordinary efficiency cores.
- Fixed on the way, on Linux too:
libjanas_mcp.solacked functions it called, now an error when it is linked (--no-undefined); janas-system's list of errors could come in a different order when two came in the same second.
[2026-10-11] - A binary package, and the models already converted
- The documentation in pages, and a wiki. The README keeps the speeds, the tables and the machine they were measured on; the rest is in
docs/(what works, getting a model, the chat, the library, serving the model, the tools, your hardware), and the README's old sections point there, so the links already given still work. Each release writes the project's wiki from the same pages, so the two never part. - Q3_K weights in a prompt's products on the GPU (the fast prompt's tiles): as the CPU's to seven digits, 420-635 GMAC/s against 267 on twelve threads. A mixture's Q3_K experts stay on the CPU, where they are faster: on unsloth's 3-bit Qwen3-30B-A3B, which still reads a prompt slower than llama.cpp (102 tok/s against 112).
- Measured on 11 October 2026: Qwen3-Coder-Next reads 66 tok/s and writes 22 where llama.cpp manages 14 and 2.2; Qwen3.6-35B-A3B reads at 104 tok/s against 79. The README's tables give the operating system of the test machine (Debian 13, kernel 6.12.107, Mesa 26.1.6), and the engine is called Janas-LLM where the engine is meant.
- A binary package for Linux x86-64, built by GitHub Actions on Ubuntu 22.04 and attached to each release:
janas-linux-x86_64.tar.gz, at a link that always gives the newest (releases/latest/download/janas-linux-x86_64.tar.gz). The programs and the converters, Janas's services inservices/with their pages, the libraries and their headers, the SHA-256 of every file; no script. It runs with the C library of Ubuntu 22.04, Debian 12 and anything newer, the GPU included.janas-chatfinds the services inservices/beside it. - The converted models are a promise: the
.jnsfiles on Hugging Face (format version 2) will be read by every later version of Janas. janas-getdownloads the.jnsfiles themselves, from the project's repositories on Hugging Face (prabanta-dev/<model>-JNS): no conversion on your machine, half the room on disk, every file checked against the same fingerprints of MODELS.md.--from-ggufkeeps the old way (the original GGUF, converted here), andjanas-gettakes it by itself where a converted file is not on Hugging Face;janas-get hf:<owner>/<repo>/<file.jns>takes any.jns.janas-get listshows the size of the converted files.
[2026-10-10] - The tuner leaves cores another program has taken
- The threads move as soon as they turn slow. With another program busy on the efficiency cores, Qwen3.5-2B's tokens on 20 threads took 680-850 ms each against 22 on 6 or 12, and the tuner moved them only after 645 tokens: the 6 and 12 took the tries in turn and each counted only its own wins, and the first try came 128 passes in. Now a pass of the choice twice as slow as its mean brings a try at once, a try twice as fast moves the threads, wins by any other threads count together, and the next try comes 16 passes after one that won. A reply of 3,000 tokens under that load: 584 s before, 57 s now; without it, nothing moves (38 s, the same threads).
JANAS_TUNE_TRACE=1prints the tuner's tries and moves.
[2026-10-10] - A mixture's prompt layer by layer on the GPU
- A mixture's prompt blocks read on the GPU alone, a layer a submission (qwen3moe, qwen35moe, integrated GPUs). Each layer is one submission - attention or Gated DeltaNet, norms, shared expert, router and its choice of experts - and between two of them the CPU only sorts the router's choice by expert, asks the expert cache for their slots and has the next layer's experts read ahead while the GPU computes; the experts' products, their activation and each token's weighted sum stay on the GPU. Reading 1,024 tokens on the development laptop: Qwen3.6-35B-A3B at four bits, 59% of its experts in the cache, 85.6 -> 101.9 tok/s, the cache finding 96% of the experts it is asked for instead of 87%. Against Qwen's Q8_0 the most likely token agrees as often (250 against 249 of 256 on code, 234 against 238 and 234 against 234 on Italian), the mean difference equal or smaller. Qwen3-30B-A3B, its experts all in memory, comes out even (130.6 against 130.0 over three runs each), with the CPU free during the prompt.
- On by itself where the memory allows it: where the expert cache holds half a model's experts or more - the 30B and the 35B on a 32 GB machine, larger models on larger ones. A smaller cache keeps the way the experts were shared between CPU and GPU, not measured below that yet; so do experts whose down matrices are plain Q6_K (Qwen3-Coder-30B-A3B read 116.9 tok/s that way instead of 131.5). bartowski's Qwen3-30B-A3B came out even (127.0 against 128.4).
JANAS_GPU_MOE=1takes any mixture,0none. The CPU-only engine is unchanged. - The experts' shader on the GPU reads Q6_K in planes at four and two bits too, not only at six: a model whose experts are read at fewer bits (
--bits 4, Qwen3.6-35B-A3B on a 32 GB machine) gets its experts' down products on the GPU. janas-bench --settle <seconds>waits after the warm-up for the expert cache's background filling to end before measuring.
[2026-10-10] - Prompts read on the GPU, faster than llama.cpp on the same GPU
- The fast prompt, on by default. A prompt's blocks go to the GPU's tiled products - Q4_K, Q5_K, Q6_K (plain and in planes), Q8_0, f32 and bf16 weights, a mixture's experts in one launch per layer, the GPU taking 60% of their tokens while the CPU computes the rest - and on the dense Qwen3 and Qwen3.5 models the whole block, every layer, goes to the GPU in one submission while the CPU rests. They round the sums group by group, so a prompt is no longer read to the bit as the CPU reads it; against the references the most likely token agrees as often as the exact way (within 2-3 of 256 on every model measured). A reply's tokens are as before.
JANAS_GPU_FAST_PROMPT=0reads prompts the exact way. - Qwen3.5's recurrent layers on the GPU. The Gated DeltaNet layers - convolution, recurrence (each row of the state kept in registers for the whole block, at sixteen bits between tokens as the CPU keeps it), gated norm - have shaders of their own, so Qwen3.5's prompts go to the GPU whole too; with an MTP block the hidden states come back for it (drafts kept 78-86% as before).
- Faster than llama.cpp's Vulkan backend on the same integrated GPU, reading 1,024 tokens: Qwen3.5-2B 441 tok/s against 323, Qwen3-4B 193 against 164, Qwen3.5-9B 111 against 78, Qwen3-30B-A3B 135 against 122, the four Gemma 4 models 65-191 against 46-150; writing 1.3 to 2.1 times as fast. The README's tables have the figures.
- On a laptop, the CPU rests while the GPU computes. CPU and GPU share one power limit: the CPU's threads spinning beside the GPU held it at half its clock. The pool's idle workers now sleep and the GPU's fence is waited on, with the fast prompt: Qwen3.5-2B +18%, Qwen3.5-9B +22%, Gemma 4 E4B +12%.
- The tuner chooses better. With the fast prompt a prompt starts on the performance cores' threads; the default competes from its first measurement (an alternative tried three times used to win even when slower: Qwen3.5-2B's prompts went to 20 threads, 272 tok/s, against 318 on 12); a chat's short blocks (17-63 tokens) are tuned apart from long prompts; and settled threads move when other threads win three tries in a row, since a laptop that warms up turns the first verdict round.
- Larger prompt blocks where they pay: 512 tokens on the Qwen3, Qwen3 MoE and Qwen3.5 models whose feed-forwards the GPU takes whole (Qwen3-30B-A3B reading 1,024 tokens: 108 -> 123 tok/s from that alone).
- Fixed: the last layer's down matrix went to the CPU without a word on several models: the GPU's Q6_K shaders read 4 bytes past a matrix and the expert arena's last slot had none. A guard page now follows it.
- Fixed: a crash now and then while reading a mixture's first prompts on the GPU, when the table of products' shapes filled up.
[2026-10-09] - Qwen3-4B and Qwen3-30B-A3B also with an importance matrix
qwen3-4b-imatrixandqwen3-30b-a3b-imatrix: bartowski's Q4_K_M beside Qwen's own, which stay the default. Qwen's Q4_K_M files are quantized without an importance matrix, bartowski's with one. Against Qwen's own Q8_0 (400 tokens of llama.cpp's logits, most likely token agreeing): Qwen3-4B 381 against 364 on code and 351 against 329 on Italian, at the same size and speed; Qwen3-30B-A3B 388 against 376 and 367 against 335, the nearest of the three versions now in the catalog, 8-10% slower held in memory since it keeps more matrices in Q6_K. Each has its fast profile measured:--head 4 --attention fast(+5.7%) and--head 4(+3.2%). The three-bit Qwen3-30B-A3B also has one now:--head 4(+3.8%).
[2026-10-09] - Qwen3-30B-A3B in three bits
qwen3-30b-a3b-q3: the same model in unsloth's dynamic three-bit mix, smaller and closer to the original. Measured against Qwen's own Q8_0 (400 tokens of llama.cpp's logits), unsloth's UD-Q3_K_XL agrees on the most likely token more often than Qwen's Q4_K_M that the catalog already had: 386 against 376 on code, 360 against 335 on Italian prose, with 27% fewer bytes of experts (12.8 GB against 17.6). With little memory it is also faster: with the expert cache held at 2 GiB, 19.8 tokens/s against 13.6. Held whole in memory it is as fast: 32.0-32.2 tokens/s against 32.1-32.2, once the Q3_K product was made faster (below).qwen3-30b-a3bis unchanged.- The Q3_K product is a quarter faster, with the same results to the bit. It unpacked the eight registers of weights of a block and their eight registers of scales before using any: sixteen, all that AVX2 has, so they went through the stack (perf counted more loads than Q4_K for half the bytes). Made where each is used, one core reads 16.4 GB/s instead of 11.9, twelve threads 60.8 instead of 48.9, and the three-bit Qwen3-30B-A3B went from 29.3 tokens/s to 32.1 held in memory. Every unsloth or bartowski file with Q3_K in it gains as much. The same trade does not pay on Qwen3.6-35B-A3B, whose Q4_K_M is already made with an importance matrix: bartowski's Q3_K_M lost 26 positions of 400 on Italian for 2-12% more speed, so it is not in the catalog.
[2026-10-08] - Light experts not in memory left out, not the lightest of all
- With a small expert cache, only what would cost a read is given up. When the cache holds less than a fifth of a model's experts, the engine used to drop the lightest experts of every token, six of ten or eight, whether they were in memory or not. Now it keeps every expert in memory and leaves out only the light ones it would have to read from disk (below 0.8/k of a token's weight, 0.95/k under a tenth), the other experts taking their share. Qwen3-Next-80B against llama.cpp, 400 tokens: with a 2 GiB cache 26.4 tokens/s instead of 20.9 at about the same agreement (380 against 383 most likely tokens); with 4 GiB the same speed (27.7) with a mean logit difference of 0.36 instead of 0.53; on Italian prose with 8 GiB 372 tokens agreeing instead of 361, and faster (23.7 against 21.4 tokens/s). On Qwen3-30B-A3B and Qwen3.6-35B-A3B with 1.5-3.5 GiB of cache it was 9-37% faster than the old way on code and on Italian prose, as close to llama.cpp or closer in most cases.
/experts nasks for an exact number again and turns it off;JANAS_MISS_SKIPsets the weight for measures.
[2026-10-08] - The machine's measure without a head start
- The measure on this machine no longer favours whichever way goes second. It warmed the model on the prose prompt alone and then measured drafting first, so the code and the list ran on cold experts with the drafts and on warm ones without: Qwen3-30B-A3B looked 10% slower drafting from the conversation (31.4 tokens/s against 34.8) and was set to
--no-spec; with the order swapped it was 9% faster. Now every prompt warms the model, and drafting and not drafting are measured prompt by prompt, taking turns at going first: 33.5 and 34.3, twice alike, under the 3% a change needs, so the 30B keeps its drafts. Runjanas-chat <model> --tuneto measure again a model tuned before.
[2026-10-08] - Qwen3.5-0.8B is a test model
- The catalog says what each small model is for. Qwen3.5-0.8B is a test model of the Qwen3.5 architecture, not for chatting: asked for twenty actions, it called the right tool 7 times out of 15 and made facts up ("Bologna founded in 1177"). janas-chat leaves it out of its list and opens it by name with a warning;
janas-get listsays so. Qwen3.5-2B, which called the right tool 14 times out of 15, is described as the small one that calls tools well; Qwen3-0.6B stays the draft of Qwen3-4B.
[2026-10-08] - Cities by their Italian names; each model's own layouts
- Roma, Torino, Londra, Parigi are cities to the flights and weather services. The tables name cities in English, and an Italian name went astray: "Roma" was the airport of Roma in Queensland (0 aircraft within 50 km of it, for a question about Rome), Torino, Londra and Parigi nothing at all. The Italian names of the cities most asked for, Italy's and abroad, now lead to theirs, each in its country (Siracusa stays in Sicily); an airport found by a city's name has to be within 100 km of the largest city so called ("Florence" is Peretola, not Florence Regional in South Carolina). Asked again: Roma, 17 aircraft within 50 km of Ciampino.
test_geochecks them. - Each model translates the services' layouts for itself, and is not asked again after it failed. A layout (the text a service's answer is shown in) was translated once and kept for every model, so Qwen3.5-0.8B's poor translation ("misurato dal punto") came with every other model's answers once it had written it first; and a translation that failed was asked for again at the next question, two or three minutes a reply for Qwen3-4B. Now the file carries the model's name, and a failure is kept beside it: Qwen3-4B 291 s for its first weather question, 42 s for the next. Layouts translated before are translated again, once, by each model.
[2026-10-08] - The weather of a region
- A region is a place the weather service knows. Asked for "Liguria", the geocoding it uses has no such place, only a peak of that name in Antarctica, and that is what it took: -26 °C in Liguria, found by Qwen3-0.6B asking for Liguria's warnings. A region is now taken as its largest city with the region kept for the warnings (Liguria: Genova, and MeteoAlarm's warnings for Liguria): an Italian one by its Italian or English name before the geocoding (so "Molise" is Campobasso, not the hamlet of 150 people called so), any other only where the geocoding knows no town of that name ("Washington" stays the capital). Of the geocoding's answers, a place people live in now always comes before a peak, a range or a lake, and anything in Antarctica after everything else.
test_geochecks the regions.
[2026-10-08] - Every model's fast profile, measured
- Every lever that trades a little accuracy for speed was tried on every model of the catalog, alone and in every combination:
--head 4,--attention fast, and--bits 4where the experts are in planes. A lever is in a model'sfastprofile when it adds 2% or more: Qwen3-0.6B--head 4 --attention fast(+14.3%), Qwen3-4B the same (+6.8%), Qwen3.5-0.8B--attention fast(+2.0%), Qwen3.5-2B and 9B--head 4(+10.2%, +5.1%), Qwen3-30B-A3B all three (+10.0%), Qwen3.6-35B-A3B, Qwen3-Next-80B-A3B and Qwen3-Coder-Next--head 4 --bits 4(+6.5%, +8.6%, +10.5%). The agreement with llama.cpp of each is in the README. Gemma 4 has none: sixteen-bit keys, heads not six-bit, experts not in planes. Oracles of llama.cpp for the four Gemma 4 models were made for it (micro-batches of 8). - Measured on the way: for a model larger than its cache, a benchmark that generates its own text cannot compare levers, since each lever changes the greedy text and with it the experts read from disk (Qwen3.6-35B-A3B: 92.5-96% of the experts from memory, each lever slower alone and all three 7.9% faster); those three models were measured on a fixed text instead.
- The measure on this machine uses three replies of different kinds, not one: on Gemma 4 E2B a single reply drafted at 38.6 tokens/s in one run and 46.9 in the next.
- A model of the catalog takes its prediction file from the catalog. One merely found beside it may belong to another model of the same architecture: Qwen3-Coder-Next took Qwen3-Next-80B-A3B's MTP block and drafted at 25.9 tokens/s where the conversation's drafts gave 28.8.
[2026-10-08] - The chat measures a model on the machine it runs on
- The first time a model starts in a terminal, janas-chat offers to measure it: the model opened each way that can differ on this machine - drafts from its prediction file or draft, from the conversation, or none; in the
fastprofile also the head as in the file - one reply to warm it and the same reply measured at temperature 0. The fastest way, if it beats the starting one by 3% or more, is kept in~/.cache/janas/tune-<name>-<profile>.conf, a layer between the catalog and the user's files, used only for levers the user did not set; a "no" is kept as well.--tunemeasures again,--no-tuneleaves it out. Development laptop: Qwen3-4B 33.1 tokens/s with its draft against 29.2-29.5 otherwise, Qwen3.5-2B 77.4 with its MTP block against 41.9 without drafts and 60.6 from the conversation. - A model freed gives its caller back the CPUs it had. Opening a model pins the calling thread to one CPU; a second model opened afterwards in the same process saw that one CPU and ran on a single thread ("up to 1 compute threads"): the measure found it, opening the model three times. The caller's CPUs are now kept at load and restored when the model is freed.
- Without drafting (
--no-spec), the chat no longer loads a prediction file or draft it would not use.
[2026-10-08] - The chat takes a model by name
janas-chat qwen3-30b-a3b: a name of the catalog instead of a path, looked for in~/.local/share/janas/models(orJANAS_MODELS, or the new--models <dir>); a model not there yet is downloaded and converted throughjanas-get, with its prediction file or draft, once the user says yes. With no model, the chat lists the catalog, marks what is there, and asks which one. The first download proposes the folder and takes another typed one, saying how the chat will find it again. A path works as before, and a file of the catalog given by path gets its profile too.- Two profiles per model, and settings in layers.
--profile quality(the default: the answers the file gives) or--profile fast(the catalog's measured levers on top, today--head 4on six models). Each layer over the one before: the catalog's profile,chat.conf, the model's own~/.config/janas/models/<name>.conf, the command line./save-config,/load-configand/del-configwork on the model's file, or withglobalonchat.conf; nothing writes the catalog. A draft named by the catalog is taken by itself (Qwen3-4B: "drafts from qwen3-0.6b").
[2026-10-08] - One catalog of models, and a home for them
- The catalog of models moved from
janas-getto the code every program shares (src/common/catalog.c), so that the chat can use it too. Each model may now name a draft (Qwen3-4B: Qwen3-0.6B, new in the catalog and in MODELS.md, from unsloth's repository since the Qwen team's no longer has a Q4_K_M) and a "fast" profile of chat options measured to make it faster at a small cost to its answers (--head 4on Qwen3-0.6B, Qwen3-4B, Qwen3.5-2B, 9B, Qwen3-30B-A3B and Qwen3-Next-80B-A3B; not on Qwen3.5-0.8B, which with it answered "3" to "2+3" in a chat).janas-get qwen3-4bbrings its draft with it, unless--no-mtp. - The models now go to
~/.local/share/janas/models($XDG_DATA_HOME/janas/models), or toJANAS_MODELS, instead of amodelsfolder under whatever directoryjanas-getwas started from;--dirstill chooses any other.
[2026-10-08] - The output head at four bits, on request
janas-chat --head 4(andhead_bitsinjanas_llm_params,JANAS_HEAD_BITS=4) requantizes a six-bit output head to four bits when the model opens. The head is the largest matrix read whole for every token - a seventh of Qwen3.5-9B's - and most files keep it in Q6_K. 128 tokens after a prompt of 1,900: Qwen3.5-9B 13.35-13.36 tokens/s where 12.76-12.77 (+4.6%), Qwen3-Next-80B-A3B with a 16 GiB cache 25.0-25.3 where 24.4-24.5 (+2.8%); prompts unchanged. The answers move a little away from the original weights: against llama.cpp on 400 tokens, the most likely token agreed 372 times where 383 on Qwen3.5-9B (Italian prose), 372 where 382 on Qwen3-Next-80B-A3B, 387 where 392 on Qwen3-30B-A3B (C source). The default is unchanged: the head as the file has it.- On the small Qwen, whose head is their token embeddings, the table is requantized for both uses: three runs each with nothing else running: Qwen3-0.6B 107.0-107.7 tokens/s where 100.6-101.9 (+6%), Qwen3.5-0.8B 113.5-114.7 where 104.2-105.8 (+9%), Qwen3.5-2B 54.4-54.9 where 50.0-50.4 (+9%), Qwen3-4B 24.0-24.1 where 23.2-23.3 (+3.4%); the answers move more than on the larger models (Qwen3.5-2B, most likely token 352 where 376 against llama.cpp).
- Only the head: requantizing every six-bit matrix (attention, shared experts too) cost more agreement (361 where 383 on Qwen3.5-9B) for next to no more speed there. Opening costs a second or two more (Qwen3.5-9B: 2.5 s where 0.7), with a new
janas_q4k_quantize_fastthat takes each sub-block's range as it is instead of searching 49 shrunken ones: nine times faster, and on this head the mean logit difference 0.157 where 0.154.include/janas/llm.bihas the new field, checked against the C layout with fbc.
[2026-10-07] - Attention merges its pieces in the same round
- The thread that finishes the last chunk of a group of query heads merges that group at once, instead of the pool starting a third round for the merge after everybody has finished. The other threads go on with their chunks meanwhile, and a token's attention costs one wait for all the threads less. Only where a group's merge is small - decoding and the check of a few drafts, or blocks at a short context; a block of 64 tokens past about 8,000 positions keeps the separate round, spread over all the threads, which there was faster. Each output is computed exactly as before, so the results are the same to the bit (
test_attention's checksum unchanged with the merge always in the round, never, and as chosen).tests/bench_attn, a block of 64 tokens at 8,000 positions: 26.0-26.3 ms per layer where 28.0-28.5 (Qwen3-30B-A3B's geometry), 26.1-26.7 where 27.1-28.0 (Qwen3-Next's). Qwen3-30B-A3B in memory, 128 tokens after a prompt of 1,900: attention 6.58-6.62 ms a token where 6.90-6.92, 27.4-27.6 tokens/s where 27.3-27.4.JANAS_ATTN_FUSE=0keeps the separate round.
[2026-10-07] - The chat redraws only what changed
- While a reply streams in,
janas-chatsends the terminal only the rows that changed. It redrew the whole conversation window for every piece of text, 8 KB a token on a 160 x 40 terminal; now each row is laid out into the bytes it is drawn with, a row the terminal already shows is not sent again, and when the text wraps the terminal moves the rows up itself (a newline on the last row of the scrolling region) and only the new ones are drawn. A clear of the screen, a change of its size and every redraw other than a reply's piece still draw everything. In a pseudo-terminal, Qwen3-0.6B answering the same question: 2.0 KB a token where 8.1, and 102 tokens/s where 80 (three rounds each, 96-110 against 78-83), the chat's own drawing no longer holding the model up. Checked by feeding the window 3,000 pieces of text at 50 x 24, 80 x 40 and 34 x 20 and rebuilding both screens with a terminal emulator (pyte): the same characters and colours after every piece. - Found on the way: a row could come out one column too wide - the space it was broken after left at its end - and the terminal, wrapping it, moved the whole window up one row until the next redraw. A row is now drawn no further than the window's edge.
[2026-10-07] - A row costs one call
- The products over one vector loop over their rows beside the kernels. Each row of a matrix-vector product went through
janas_dot, which chose the kernel by the weights' type, then a wrapper and a function pointer, and the caller worked out the task's sizes again for every row; on the short rows of a mixture of experts (2,048 columns, eight blocks) that was 6% of a token's cycles in a profile. Now the sizes are worked out once for each run of rows a thread claims, the type is chosen once, and each row is one direct call to the same kernel as before - the logits of Qwen3-30B-A3B are the same to the bit. Qwen3-30B-A3B in memory, 128 tokens after a prompt of 1,900: 29.4-29.6 tokens/s where 28.4 (+3.8%), the experts' phase 15.9 ms a token where 16.9, prompts unchanged.
[2026-10-07] - The fast attention weighs the values with integers too
- On its sixteen-bit path, attention now weighs the values with sixteen-bit integers as well as computing the scores so. A chunk's weights go to 0-32767 with a scale of their own, two positions a word, and the products with the eight-bit values are summed exactly in integers, sixteen to an instruction where a float multiply-add does eight. The path is the one the engine takes past 16,384 tokens of context or on request (
JANAS_ATTN=int16); the exact path, the default below that, is unchanged.tests/bench_attn, Qwen3-30B-A3B's geometry on 12 threads, a token's attention per layer: 2,048 positions 0.096 ms where 0.107, 8,000 0.325 where 0.354, 32,000 1.34 where 1.42 (6-11%). - The weights' rounding adds a little error: against a double-precision reference on random scores, the worst output is now 2-6e-4 off where it was below 1e-4, still under the 4e-3 the eight-bit keys and values carry, and
test_attention's threshold for this path is 1e-3. On the model it does not show: Qwen3-30B-A3B against llama.cpp, 400 tokens of C, mean logit difference 0.2865 (0.2893 with float weights, 0.2871 exact), top token 390/400 (388, 392). The scalar, AVX2 and AVX-VNNI versions give the same bits, and blocks the same bits as token by token.
[2026-10-07] - Attention's scores two heads at a time
- The scores of attention are computed for two query heads at a time, the keys four at a time: one key read serves two products, and the eight running sums, two queries and a key fit in the processor's sixteen vector registers. Each sum adds the same products in the same order, so the scores are the same to the bit (
test_attention's checksum unchanged on AVX2 and AVX-VNNI).tests/bench_attn, Qwen3-30B-A3B's geometry on 12 threads: a token's attention 4-5% faster per layer at 512, 2,048 and 8,000 positions, a block of 64 tokens 5-8%. In the engine, Qwen3-30B-A3B in memory after a prompt of 1,900 tokens: attention 6.28-6.31 ms a token where 6.50-6.53, the prompt read at 70.9-71.0 tokens/s where 70.0-70.1; the tokens written move within the noise of the rest. - Measured on the way: with the exact arithmetic, attention is bound by the multiply-adds, not by the memory - at 8,000 positions a layer reads its keys and values at 20 GB/s, and twelve threads go four times as fast as one, about as many as the six performance cores they run on.
[2026-10-07] - Weights asked for ahead of the product
- Every kernel asks for its weights 2 KB ahead, one prefetch per cache line, when it multiplies one or two vectors. The hardware prefetcher alone kept too few reads in flight for the types that compute more per byte: on 12 threads of the development laptop Q4_K read 69 GB/s where Q8_0, which asks for its bytes faster, read 84. Now, measured by
tests/bench_kernelson one vector, Q4_K reads 83 GB/s, Q5_K 80 (68), Q6_K 85 (73), Q8_0 88 (84), Q4_1 74 (67), Q4_0 and IQ4_NL 70 (65), IQ4_XS 64 (59); on two vectors, the check of one draft, Q6_K 70 (56) and Q8_0 78 (69). Q3_K and IQ3_S, bound by their arithmetic, are unchanged, and so are products of three vectors or more, where the prefetches would only cost instructions. - Inside the engine the gain is smaller, because its matrices were already read at 68-74 GB/s, nearer the memory's speed than one huge matrix in the benchmark: the output head, a single large Q6_K matrix, takes 11% less; Qwen3.6-35B-A3B writes 25.1-25.6 tokens/s where it wrote 24.3-24.5 (+3.4%), Qwen3-30B-A3B 30.9-31.0 where 30.6-30.9 (+0.8%), prompts unchanged. Results the same to the bit (
test_quanton AVX2 and AVX-VNNI).
[2026-10-07] - Small gains, kept: blocks of tokens and long contexts
- The activations' sums of 32 are made once. The Q4_K and Q5_K products over several vectors - a prompt read in blocks, the drafts checked - weigh the sums of each 32 activations by the block's mins, and added them up again for every row of every matrix; the activation's quantization now leaves them beside the sums of 16 (16 bytes more a block of 256). 5-6% on those products.
- Three or four vectors reduced in four lanes, where they were reduced in eight, half of them empty: the size of the check of a few drafts. 2% on 12 threads, 5% on one core.
- Attention's scores eight keys at a time. The sum of each key's products was reduced on its own, with as many instructions as the products and on the same ports; eight are reduced together now, in the same order of additions, so the scores are the same to the bit. 5-8% off the attention of a layer at 8,000 positions.
- Together, on Qwen3-30B-A3B in memory, a prompt of 1,900 tokens: read at 70.1-70.5 tokens/s where it was 67.4-67.5 (+4%), the 128 tokens after it 25.8-26.0 where 25.4-25.5 (+1.5%), attention 6.5 ms a token where 6.9. Every result the same to the bit (
test_quanton AVX2 and AVX-VNNI,test_attention's checksum unchanged).
[2026-10-06] - Q3_K, IQ4_XS and IQ3_S weights read at full speed
- The products of Q3_K, IQ4_XS and IQ3_S weights were six to thirteen times slower than they should have been. Their kernels converted each block's 16-bit scale by calling a function, inside the vector loop, where the others use the processor's own conversion; now they do too, with the same result to the bit. On one core, a 4096 x 2048 matrix: Q3_K 3.38 to 0.43 ms, IQ4_XS 3.26 to 0.25, IQ3_S 3.45 to 0.56. Unsloth's UD files mix these types in: Qwen3.8-27B UD-Q4_K_XL, 3.3 GB of whose 17.5 are of them, writes 3.6 tokens/s where it wrote 1.7-1.8, and reads a prompt at 8.2 tokens/s where it read 6.4.
- Q3_K a half faster again: its 16 scales unpacked and spread in vector registers rather than one by one, and each weight's high bit put in place in one step fewer. One vector on 12 threads, 33 to 48 GB/s; the same sums, the same result to the bit.
tests/bench_kernels: every weight type's product on one core and on many, over 1 to 8 vectors, in GB/s of weights, to set beside the memory's bandwidth. On the development machine, one vector on 12 threads: Q8_0 84 GB/s, Q6_K 73, Q4_K 69, the 4- and 5-bit types 59-68, Q3_K 48 and IQ3_S 25 - IQ3_S still bound by its table lookups.- Where the time of a token goes, against the bandwidth (bytes each phase reads, over its time): on Qwen3-Next-80B-A3B, Qwen3.6-35B-A3B, Qwen3-30B-A3B and the 27B, every phase that reads weights runs at 60-70 GB/s, 59-62 on average - three quarters of the 81 GB/s the memory gives, nine tenths of what the kernels do alone.
[2026-10-06] - A long prompt reads the next layer's experts ahead
- The prefill reads ahead too. Block by block, the next layer's router is applied to the whole block before the current layer's experts, and the experts most of its tokens will want are read from the disk while the current layer computes; until now each layer waited for the disk chunk by chunk. On after a block whose layers missed more than 32 experts each, off where the experts fit. Qwen3-Next-80B-A3B, 4,096 tokens, every bit: 36.8 to 42.0 tokens/s with a 4 GiB cache (time waited for the disk 40.8 to 20.7 s), 48.5 to 50.0 with 16 GiB, the 256 tokens written after it 21.8 to 21.5 tokens/s; Qwen3-30B-A3B in a cache that holds it, even. The cost is the router applied once more, about 1.1 ms a token.
JANAS_PREFETCH_PREFILL=0never,=2always. The idea is Strata's (github.com/Niko1221/Strata), which streams the next layer's experts to its graphics card. - The experts read ahead are kept until their layer is done. A prefill block asks the cache for its experts 64 at a time, and the first request let go of every expert read ahead for the layer, so those read for the later chunks were thrown out before they were asked for: 42% of the reads ahead were used, 82-89% now.
tests/bench_longcounts the experts read ahead as read from the disk, as the chat's statistics do; it counted them as found in memory.- Tried and left out: drafts copied from the context weighed against the MTP block's, as Strata does. On Qwen3.5-9B the MTP drafts already kept 95-99% where a reply copies its prompt (a function renamed, a table), and the copies, kept 20% of the time, only took their place.
[2026-10-03] - Strict settings
- A word not known is refused, with the one nearest to it.
janas-chat(command line andchat.conf),janas-server,janas-mcpandjanas-benchrefuse an option mistyped ("--tmep is not an option (did you mean --temp?)") and a value that is not one: numbers are read whole and within their limits (--ctx 32kwas 32 tokens,--temp 0,7was 0,--port 70000wrapped), choices are one of theirs (--think ofturned reasoning on,--mode eccowas auto).janas-benchleft an unknown option and a last one without its value aside, and measured something else than asked. The chat's file said "a word it does not know" without saying which, and went on. - mcp.json: a key a letter or two from one Janas reads (
arg,evn) is refused with the one meant,disabledmust betrueorfalse; the keys of the other clients are left to them. - The services' settings files (
flights.conf,prices.conf): a line naming no setting is told at the start and in the answer that says the key is missing;aviationstack_key_old = ...is no longer read as the key. - The variables of the environment: a
JANAS_one Janas does not read is told when a program starts, with the name meant;tests/test_words.cchecks the table of names against every one the sources read.
[2026-10-03] - The services' pages beside their code
- src/services/README.md sums up the services and links each one's page, now
src/services/<service>/README.mdbeside its code (they were indocs/services/); the main README links the summary.
[2026-10-03] - janas-quakes: the earthquakes
- janas-quakes, an MCP service of the earthquakes: the latest near a place, where the user is, or within a sea of the built-in tables (by its polygons), the strongest in the world with the USGS's expected impact and its tsunami flag (told as a flag, not as a tsunami), one in detail (how many felt it, the strongest shaking estimated, the link to the source's page). INGV (CC BY 4.0) for Italy and the seas that reach into it, the USGS (public domain) elsewhere, each the other's fallback; the answers give the model each event's id, to name the one meant. Run over the protocol and in janas-chat;
tests/test_quakes_parse.cchecks the reading of both sources and the layouts.
[2026-10-02] - janas-system: this computer
- janas-system, an MCP service that only reads and reaches no network: how the computer is (the processor and the kernel's share of it, the memory and the swap, on Linux the pressure, every disk, the battery and its wear, the temperatures a device), what fills a directory (8 seconds and 64 levels at most, never across filesystems or links; "at least" when the time runs out), the programs using the most processor or memory or those of a name, the errors of the logs (the same message but for its numbers counted once, the latest read first) and the services that failed, the updates by the package manager's last list (apt, dnf, pacman; nothing fetched), the network interfaces, the Wi-Fi signal, the gateway. Run over the protocol on the development laptop;
tests/test_system_parse.cchecks the reading of Linux's texts and the layouts. - A Windows part for it, on the Win32 API (memory, processor, drives, power, processes, event log, services, IP Helper, WLAN): compiled and linked with MinGW-w64 and run under Wine; not yet on Windows, and Janas itself is not built for Windows yet.
- A layout that comes back untranslated is asked once more in
janas-chat, the model told so: Qwen3.6-35B-A3B gave janas-system's status back as it was the first time; run again, it came in Italian. - The services run a program the same way:
janas-git's runner (no shell, no terminal, a time limit) moves tosrc/services/common/run.c.
[2026-10-02] - The services under one directory
- The MCP services in
src/services/: each insrc/services/<service>/(flights, git, github, maps, prices, weather, wiki), what they share insrc/services/common/(the geography's tables, the GET over HTTPS, where the user is, the layouts of the answers). The programs keep their names.
[2026-10-02] - janas-prices: prices and finance, for information
- janas-prices, an MCP service for information only: an amount in other currencies at the ECB's reference rates (Frankfurter; the ECB's file when it does not answer), a crypto-asset's price (CoinGecko's keyless API; Coinbase when it does not answer), a share's (Alpha Vantage, with a free key of the user's: the only tool that needs one), inflation (Eurostat monthly for the EU, Iceland and Norway, the World Bank yearly for every other country), the rates of the ECB, the Fed, the Bank of England and the SNB, the wholesale price of electricity hour by hour (energy-charts, only the bidding zones it gives under CC BY: northern Italy yes, Sicily no), the cheapest fuel near a place (Italy's MIMIT, France's live flow, Spain's ministry, Austria's E-Control; another country is said not to be covered; a price not told for over eight days is left out and counted: a station's of two months before had come first in Palermo). Every answer says its source, its date, and "for information only, not advice". Layouts for every answer;
tests/test_prices_parse.con answers of 2 October. - The system message read ahead is kept on disk again: since the services' tools are given when needed it is under a thousand tokens (767 with seven services), and copies under 1024 tokens were never written, so every start read it anew; a recurrent model's (Qwen3.6-35B-A3B) cannot be cut back from a longer conversation either. A read-ahead copy is now written from 64 tokens. Two starts in a row of
janas-chatwith a "Ciao", on a machine busy with other work: Qwen3.6-35B-A3B 21.6 s then 4.4 s, the second reading nothing again; Qwen3-4B 11.6 then 2.1 s; Qwen3.5-2B 6.3 then 1.4 s.tests/llm_keep.cchecks a short system message too, and no longer leaves its cache directory behind. - Spain's fuel at the ministry's newer address: the older one offers only TLS 1.2 with RSA's key exchange, which Mbed TLS 4 no longer has.
- Places by Nominatim: Open-Meteo's geocoding matches one language only ("Wien" was a place in the United States, "Milano" Milanówek in Poland).
[2026-10-02] - janas-github and janas-git
- janas-github, GitHub as an MCP service that only reads: a repository's issues and pull requests (
new: those opened since the last time asked, kept in~/.config/janas/github-seen.txt), its latest releases (its tags when it has none), a branch's recent commits, what it is. A name alone is looked for among the user's repositories, else the one with the most stars, and the answer says which. The token ofGH_TOKEN,GITHUB_TOKENorgh auth token, kept in memory only; without one, 60 requests an hour. Layouts for every answer;tests/test_github_parse.con answers of 2 October. - janas-git, the computer's git repositories:
git_status,git_log,git_diff,git_branchesread;git_commit,git_pull(fast-forward only),git_push(never by force),git_switchchange something and say so (readOnlyHintfalse). Nothing that throws work away; git without a shell, without a terminal, with a time limit; no argument of the model's taken as an option.tests/test_git_parse.c. - janas-chat asks every time for a tool that changes something, with
--mcp-autotoo and never "always", showing the call whole; on a pipe it is not run (mcpc_writes). - Translations of layouts: a text given back untranslated is refused and asked again the next time; git's and GitHub's layouts get the terms of the craft from the glossary (commit, branch, stage...), the others keep theirs; the states of an issue ("aperta", not the "aperte" the model wrote for one) from the glossary in Italian, French, Spanish and German.
- A redirect to another host loses the
Authorizationheader (janas_https_get), as curl does.
[2026-10-02] - janas-maps: roads and places
- janas-maps, roads and places as an MCP service, from OpenStreetMap's data through free services with no key: a route by car, bike or on foot between two places or through a third, avoiding tolls, motorways or ferries when asked, with its length, time without traffic, main roads, up to two other ways, the way step by step in the computer's language and a link to the map (
maps_route: Valhalla on FOSSGIS' servers, OSRM's route without steps when Valhalla does not answer); journeys by public transport leaving or arriving at a time, leg by leg (maps_transit: Transitous, over the operators' open timetables); the places of a kind around a place, nearest first, from a list of twenty kinds (maps_nearby: Overpass); where a place or an address is (maps_find: Nominatim, Photon when it does not answer). Every answer is data with a layout: the user sees the steps and the journeys, the model reads a brief.janas-chatstarts it by itself; src/services/maps. - The sources' rules are kept: a User-Agent naming the program, at most one request a second to each server (
janas-mapswaits between two), answers kept a while, every server's address changeable (JANAS_VALHALLA_URLand the like), OpenStreetMap's attribution in every answer. - What could go wrong, found and mended on the way: a town named alone ("Trapani") is the municipality's boundary in Nominatim, whose point lies in the fields 25 km from the town, and a journey by bus from there took thirteen hours; the town's own point is taken now. OSRM's server knows Europe only and put New York on the nearest European road, a route of 3,556 km; a place that far from a road is now said to be unreachable.
- Tried in
janas-chatwith Qwen3.6-35B-A3B: a route, the train, a pharmacy near the user. Mended after the first run: a translated layout keeps the English lines after its tags (the model had added blank lines between the legs of a journey:src/chat/layout.c, with a test); an arrival reads "06:26 get off at MARSALA" ("at" before both a time and a place came out wrong in Italian); "near me" is never a place only talked of. Overpass: a second instance (Private.coffee) when the first is busy, 25 s for each (janas_https_get_ms; the other sources keep 15 s), and both failures in the error.
[2026-10-02] - Layouts: the services' data in the user's language, a brief to the model
- The longest answers of the services come as data with a layout: the aircraft over a sea and near a place, the weather now, the forecast and the sea.
janas-chatfills the layout in the user's language and shows it, and the model reads only a brief of the figures. With Qwen3.6-35B-A3B, "how many aircraft over the Tyrrhenian now?" read 101 tokens and wrote 16, 3 s, against 3,011 and 334, 64 s; the weather in Trapani 162 tokens instead of about 400. - A layout is translated once, by the model, on the side (
janas_llm_chat_aside: the conversation saved and put back), checked, and kept in~/.config/janas/layouts. The model never sees the tags: asked to keep them, the 35B changed a key and a field in two layouts of the weather. It gets numbered markers and numbered words with what each word is (a Beaufort force, a Douglas sea state), and the tags and keys are put back. The scales' terms come from a glossary instead, for Italian, French, Spanish and German: the Beaufort forces and the Douglas sea states as the national weather services name them ("bava di vento", "poco mosso"; the model had written "brezza leggera", "liscio" and "burrasna"), never asked of the model, with the running text's terms of the sea ("mare lungo" for the swell) given to it in the request. The sea's states are now numbered as Douglas degrees, 1 to 9: they were one below. - In the libraries:
src/common/template.c(fields, words, lists, parts, a translation's shape checked);janas_tpl_resultfor the services;janas_mcp_result_json, a call's whole result;janas_llm_chat_aside. The MCP server merges a tool's_metawith its own, which two_metakeys had hidden.
[2026-10-01] - janas-wiki, and the services together
The services' tools are given when a question needs them. The system message holds a catalog of Janas's services, a line each, and one tool,
janas__open_service: the model opens a service, gets its instructions and tools as the call's answer, and calls them from then on; the next reply is held to be a call, since Qwen3.5-0.8B, told to call one, made the weather up instead. In the library,janas_llm_chat_tools_add(also inllm.bi): tools the model may call that are not written into the system message. With Qwen3.6-35B-A3B the system message went from 4,057 tokens to 616 (11 s the first time instead of 74), and opening the weather costs 1,048 tokens once in a conversation; one session asked the weather, a page of Wikipedia and the aircraft over the Tyrrhenian, opening each service when it came to it.--all-toolswrites them all as before.Shorter tools and instructions. The services' descriptions, parameters and instructions say the same in fewer words: flights 40% shorter, weather 33%, Wikipedia 16%.
The system message is read ahead, and kept on the disk. With the three services' tools it is about 4,000 tokens, and on Qwen3.6-35B-A3B the first reply sat on "reading" for 72-84 s.
janas-chatnow reads it as soon as it starts, while the user writes, on a thread of its own, and shows it: a line in the conversation, and the share read in the status line, in the part kept longest; a message sent earlier waits in plain sight. Once read it goes to~/.cache/janas, and the next chat on the model finds it there and starts at once. In the library,janas_llm_chat_prepare(and inllm.bi): checked on Qwen3-0.6B and Qwen3.5-0.8B (the same reply with it and without, the first message finding the 2,000 tokens computed, a new chat finding them on the disk, a new system message dropping them) and injanas-chatwith the 35B, twice: the second chat found its 4,057 tokens on the disk.janas-wiki, Wikipedia as an MCP service, with no key, used when the user asks for it: its search, best first, a title and a line each (
wiki_search); a page shown to the user as it is, a short one whole, a long one as its introduction and then the section asked for, while the model gets a note of it - its address and its sections - and not its text (wiki_page); and the pages about what lies around a place, nearest first, with how far and which way (wiki_nearby; with no place, where the user is). Titles that lead elsewhere are followed and disambiguation pages listed. Otherwise the model answers from what it knows.janas-chatstarts it by itself; src/services/wiki.A tool's answer can be for the user alone. MCP marks the parts of an answer with their audience;
libjanas_mcpnow keeps those meant for the user alone apart from the model's text (janas_mcp_result_user, also inmcp.bi), andjanas-chatshows them as they are. A page of Wikipedia read by the model cost thousands of tokens and over a minute of a large model, and a few filled its context: Qwen3-Next-80B, asked about Franco Zeffirelli, read pages and then asked for the weather and the flights of Florence, and reached 13,429 tokens without answering. With the 35B, "show me his page" now costs a note of 174 tokens, 3 s.The services do not call each other's tools unasked. Each one's instructions say so: no weather or flights for a person's home town unless the user asks.
A place named in English finds the right town. "Milan" gave the weather of Milan, Tennessee, and the model told it as Milan's; "Florence" and "Venice" went to Alabama and to Dayton, Ohio. Of the places the geocoding finds, the one taken is now the most people weighed by the place's rank (a capital, a region's seat): eighteen names out of eighteen tried, Italian and foreign.
A web search was tried and left out. Of eight engines, on 1 October: DuckDuckGo's HTML page answered well and blocked the fifth request a second apart, for minutes; Brave's API has had no free plan since February; Marginalia's shared key was always spent; Stract's API is gone; Mojeek and Qwant answer with a captcha; Mwmbl's index is small and old.
[2026-10-01] - janas-weather, and "here" for the services
- janas-weather, the weather as an MCP service, from free sources with no key: the weather now (
weather_now: Open-Meteo's model, and what the nearest airport's station measured, with when and how far; when the two disagree, the answer says so), the forecast day by day and hour by hour (weather_forecast), the sea at points across it or off a place, with the waves' names (weather_sea), and the warnings in force from MeteoAlarm, with the Civil Protection's levels in Italy (weather_alerts). MET Norway answers when Open-Meteo does not.janas-chatstarts it by itself; src/services/weather. - "Here", when a question names no place:
JANAS_LOCATION(a city,lat,lon, oroff), else the internet connection's position (GeoJS, ipwho.is), else the city of the computer's time zone; the answer says which, and the model tells the user it is a guess. For the weather and forflights_nearby. - Tool calls in Qwen's XML form take their parameters in any order. The grammar held them to the schema's order, and a model that began with a later one could no longer write an earlier one: Qwen3.6-35B-A3B asked the forecast with the number of days first and the place lost, sixteen times over.
janas-chatdoes not run again a call that failed with the same arguments in the same exchange: the model is told to answer or to ask, and at the second time the exchange stops.
[2026-10-01] - Code completion in the editor
janas-serverfills in code at an editor's cursor, with a model for code such as Qwen3-Coder-30B-A3B-Instruct, through OpenAI's/v1/completionswith asuffixand through llama.cpp's/infill, which the extensions made for llama.cpp ask (llama.vscode): the line up to the cursor,n_predict(0: the prompt only),n_indentandt_max_predict_msread as llama.cpp reads them, the answer as llama.cpp's. Tried in VSCodium with llama.vscode and with Continue; how to set them up in docs/code-completion.md.- The window sent at every keystroke is kept from moving. The extensions send "the last N lines before the cursor", whose start moves at every new line, and the whole window was read again: 17 s for 1,330 tokens at every request. The start of each window is now remembered and kept, put ahead when the editor slides its window, the text before the cursor cut at a line within
--fim-prefixtokens (2048), the lines after it within--fim-suffix(8), the text written within--fim-max(256): 1.5-2.2 s a proposal after the first request of a window, on a laptop with Qwen3-Coder-30B-A3B in memory. - Sixteen stop strings a request, where OpenAI takes four: Continue sends more, and every one of its requests was refused.
[2026-10-01] - janas-flights, the first of Janas's services
- janas-flights, flights as an MCP server of its own, for
janas-chator any MCP client: where a flight is and what it is doing (flight_status), the flights of a route (flights_between), what flies now over a sea, by its name in English or Italian (flights_over), and near an airport, a city or a point (flights_nearby). Live positions come from the open ADS-B networks (adsb.lol, adsb.fi as the fallback) and need no key; schedules, gates and delays come from AviationStack with a free key of the user's own (100 calls a month), asked only about one flight or one route - lists and areas use open data only. Times are given in each airport's local time and in UTC, places in words, and a live position is given to a scheduled flight only when the aircraft fits its route. On 1 October: AZ1709 from Fiumicino to Catania seen over the Tyrrhenian Sea with two arrival estimates, the 21 flights from Fiumicino to Catania, 45 aircraft over the Tyrrhenian Sea. janas-chatstarts Janas's services by itself: those it finds beside it, with no configuration. Their tools run without asking (they read public data and change nothing), their calls show as dimmed lines, and the model is told to use them whenever a question is on their subject, without naming them.--no-servicesstarts none.- The server side of MCP moved to
src/common/mcp_server.c, shared byjanas-mcpandjanas-flights. - docs/services/: the services documented apart from the README, with how to use them in Claude Code and the clients that read an
mcpServersfile.
[2026-10-01] - Experts read ahead, steadier drafts, every weight type on the GPU
- The drafts no longer switch themselves off for good. The planner of the multi-token prediction drafts judges each number of drafts by the cost of a pass of that many tokens, as last timed. One pass of two tokens timed while the disk was busy made it refuse every draft, and with no drafts that cost was never timed again: in some sessions the drafts stopped after the first hundred tokens and never came back, which is why the same chat ran at 22 tokens/s one time and 26 the next. It now drafts one pass in sixteen even when it chose none, as the drafts copied from the conversation already did. Qwen3-Next-80B in
janas-chat, three replies of 600 tokens, two rounds alternated with the release of 28 September: 26.0 tokens/s (25.5-26.5) against 24.3 (22.5-26.1), +7%. - The same seed gives the same conversation, drafts or not. A reply cut short - by
--max, or stopped - after a pass that had accepted drafts past its end dropped those tokens but not the random numbers drawn for them, so every later reply of the conversation came out different from the one without drafts. The numbers are given back now: with the same seed the replies are identical with and without drafts, cut or not. - A reply's statistics count the experts read ahead; left out, they made the cache look warmer and the disk idler than they were.
- Experts read ahead, where the cache misses many. A mixture whose experts do not fit in memory waited for the disk one layer at a time: Qwen3-Next-80B with every bit and a 4 GiB cache spent half of each token waiting, with the disk busy only 60% of the time. Now the next layer's router is applied ahead, and of its first ten experts up to two not in the cache are read while the current layer computes (on that model the prediction held 91% of the experts chosen). Reading ahead costs on every token and pays only when the cache misses many experts, so it turns itself on above 60 missed experts a token and off below 40. Qwen3-Next-80B, every bit, a fresh cache, three rounds alternated with it off: 4 GiB 10.9-11.1 to 11.7-11.8 tokens/s (+6%), 8 GiB 15.6-15.7 to 16.2-16.3 (+4%); with a warm 20 GiB cache it never turns on and the speed is the same; in
janas-chat, with a 4 GiB cache and every expert, 12.6 tokens/s against 12.1 of the release of 28 September (+3.6%). The results are the same bits, on or off.JANAS_PREFETCH=0turns it off. - The expert cache without transparent-huge-page advice. With the kernel's huge pages on "always" and their defrag on "madvise" (the common setting), the advice made every first write to the cache try a synchronous compaction of memory, which looked like waiting for the disk. Without it, a reply of Qwen3-Next-80B from a fresh 20 GiB cache went from 17.5-17.7 to 19.7 tokens/s, and a prompt from a warm one from 51 to 55; other sizes are even.
JANAS_ARENA_HUGE=1asks for huge pages as before. - Every quantized type on the GPU: Q8_0, Q4_0, Q4_1, Q5_1, IQ4_NL, IQ4_XS, Q3_K and IQ3_S join Q4_K, Q5_K and Q6_K, bit for bit the CPU's results (
gpu_check: 1680 shapes). Gemma-4-26B keeps its attention, its dense feed-forward and its output head in Q8_0; Qwen3.5-9B's "UD" mix keeps some layers in IQ4_XS and Q8_0. - A whole token on the GPU, as an experiment (
JANAS_GPU_TOKEN=1, dense Qwen3 models, integrated GPUs): every step of a token - norms, RoPE, attention, products, output head - in one submission, recorded once.JANAS_GPU_BLOCK=1does the same for blocks of up to 64 tokens (a prompt, the drafts a speculative pass checks), the same bits as one token at a time;JANAS_GPU_REPACK=1gives the Q6_K weights an aligned copy for it (some 0.8 GB more for Qwen3-4B). On the development laptop's integrated GPU it runs level with the CPU: Qwen3-4B writes 24.3-24.7 tokens/s against the CPU's 24.8, and in blocks reads a 2000-token prompt at 107 tokens/s where a token at a time gave 61. It is off unless asked for. - Tools for measures:
llm_gen cpus=puts the compute threads on chosen CPUs;bench_longreports how long the disk was busy beside how long the engine waited for it, prints the experts' bits and count in use (BENCH_EXACT=1keeps every bit and the model's own count), and can save the prompt and the reply; the library can hand a caller the experts each layer routes a token to, and a prediction of the next layer's.
[2026-09-28] - More of the work on the GPU
- The experts' weights go to the GPU too once every expert of every layer is in the cache: the feed-forward weights of a dense model, and a mixture small enough to fit. They were always left to the CPU. On the development laptop's integrated GPU, Qwen3-4B reads a prompt 29% faster (83 to 107 tokens/s) and the chip spends 36% less energy per token (0.49 to 0.31 J, RAPL package counter); replies are unchanged.
JANAS_GPU_EXPERTS=0keeps them on the CPU. Regions over 4 GB no longer turn the GPU off: they are imported in pieces. - Single tokens on a discrete GPU, where the tuner finds it faster. The tuner never gave a reply's single token to the GPU; on an RTX 5080 (issue #15) doing so made replies 11-16% faster, even with only attention and output in the card's memory. A discrete GPU now takes them, and the tuner tries it and keeps it only if it measures faster; integrated GPUs are left as they were.
- Two more weight types on the GPU: Q6_K in bit planes (the down matrices of every Qwen mixture and of some layers of the dense models) and Q5_K (Qwen3.5 2B and 9B). Like the others they give the CPU's results bit for bit, so a product still splits between the two.
JANAS_GPU_VERBOSE=2lists each product shape once with its weight types and whether the GPU takes them.
[2026-09-27] - No Python left
- The README names its author, Maurizio "camauri" Cammalleri, under the introduction and in a new Credits section, with the copyright line in the Licence one; the section also credits the models' authors, llama.cpp and ggml (the reference, and the IQ3_S grid), the Unicode Character Database, OpenAI's specification, the libraries compiled in, and the testers. An
AUTHORSfile says the same in the GNU way. - The last two Python scripts are C programs, built with everything else:
tools/gen_unicodewrites the tokenizer's Unicode tables from the Unicode Character Database (/usr/share/unicodeon Debian and its kin),tools/openapi_routeswritesjanas-server's route table from OpenAI's specification. Both give, byte for byte, the files the scripts gave (Unicode 15.1.0; OpenAPI 2.3.0, 345 operations), and the coverage count is the same. Nothing in Janas needs Python any more, to build or to regenerate.
[2026-09-27] - janas-get: a model in one command
janas-getdownloads a model of its catalog (the thirteen of MODELS.md: Qwen3, Qwen3.5, Qwen3.6, Qwen3-Next, Coder-Next, the embedding model, Gemma 4) from Hugging Face, checks the SHA-256 of every file against MODELS.md, runs the converters of the same build (gguf2jns, jns_planes) and makes the prediction file beside the model - hf2jns_mtp over range requests, or Gemma 4's assistant - with its fingerprint too. Downloads resume; files already there are checked, not fetched again; the GGUF goes as soon as it is converted, so the disk holds two copies at the most.janas-get hf:<owner>/<repo>/<file>takes any GGUF, checked against the SHA-256 Hugging Face lists.- Its own HTTPS download in pieces (the client of hf2jns_mtp) and its own SHA-256, on the x86 SHA extensions where the CPU has them (1.4 GB/s on the development laptop): no curl, no sha256sum.
janas-try.shgets its models through it: the catalog and the fingerprints are in one place. A test checks the catalog against MODELS.md, digit by digit.
[2026-09-27] - janas-try tries Gemma
janas-try.sh --level gemma: Gemma-4-E4B and its assistant, downloaded from unsloth and checked by SHA-256, converted in one step, with the fingerprints of both files and of the answer at temperature 0 (b2c01def5ff6bd5c: the same with the default kernels and AVX2 alone, with and without the assistant, on 4 CPUs and without the GPU). 5.1 GB to download; the model runs in about 3 GB of memory.
[2026-09-27] - The prediction file found beside the model
janas-chatandjanas-serverfind the model's multi-token prediction file by themselves: of the.jnsfiles beside the model, the one whose header fits it - an MTP block of the same architecture, width and experts, or Gemma 4's assistant as wide as the model's output and with its vocabulary - the longest shared start of the name settling a tie. Files over 2 GB (models) are not even opened.--mtpstill names one, and--no-mtpgoes without. For programs,janas_llm_find_mtp()(and its FreeBASIC declaration); opening a model never looks by itself.
[2026-09-27] - Finer activations for weights in blocks of 32
- Weights stored in blocks of 32 (Q8_0, Q4_0, Q4_1, Q5_1, IQ4_NL) now meet activations with a scale per 32 values, as llama.cpp gives them, instead of one per 256: a large activation no longer flattens the small ones beside it. The activations are quantized once into both forms; the K types keep theirs. Checked against an exact float64 forward pass of Gemma-4-26B-A4B (all Q8_0 and Q5_1 where it is most sensitive), 200 tokens: the likeliest token at 156 positions instead of 147, mean logit error 0.87 instead of 1.39 (llama.cpp: 160, 0.90). Against llama.cpp, mean differences fall by 15-40% on Qwen3.5 in IQ4_NL, Q4_0, Q5_1 and UD-Q4_K_XL (Q5_1: 376 -> 389 of 400). Models made only of K types give the same bits as before.
[2026-09-27] - The threads chosen per machine
The time left to read a long prompt no longer jumps at its start. The fitted line of issue #9 rested on two blocks when first used, and their noise could tilt it: an estimate many times too long at 1,024 tokens. It now moves from the latest speed to the fitted line over the first sixteen blocks.
The engine chooses its threads again, once per machine and model. Since 26 September it always used every usable thread, which was the fastest on the development laptop; on an i9-14900HX (issue #13) every thread replied at 9.9 tokens/s where the performance cores' 16 gave 24.7, and on a 9950X3D one thread per core was 4-15% ahead (#15). The first passes of each kind now try every thread, one per performance core and the performance cores' threads; the threads of the fastest are kept for good, saved with the machine's profile, and only whether the GPU takes part keeps being tried. Tried again later, at a longer context, the thread counts had lost unfairly to costs measured at a short one, which is why they had been fixed.
JANAS_TUNE_THREADS=0keeps every thread.
[2026-09-27] - Gemma 4's assistant drafts
- Gemma 4's assistant is taken as the MTP file (
--mtp, converted bygguf2jnsfrommtp-gemma-4-*.gguf): a small model of its own - 256 wide, four layers with queries only - reading the main model's keys and values, fed the main model's embedding of the last token and its final state, drafting at that token's position and chaining further drafts at the same one, as it was trained. A draft costs about a millisecond. - An Italian chat reply of 110 tokens, greedy, the same tokens with and without it (Core Ultra 9 185H, tokens/s): E2B 34.7 -> 51.0 (1.47x), E4B 18.3 -> 26.3 (1.44x), 12B 8.5 -> 12.9 (1.51x), 26B 16.9 -> 24.5 (1.44x); 64-83% of the drafts the planner chose to verify were accepted (on E4B llama.cpp with the same assistant accepts 50% of its drafts). The 12B's and 26B's assistants, 1024 wide, give a KV head count per layer, which is read as such.
- Its drafts are picked, as the MTP blocks' are, from a head of 32768 token ids plus those met in the conversation, instead of all 262144: 8 MB read a draft instead of 67, for as many accepted.
- E2B's and E4B's per-layer branch reads its f32 matrices (220 MB a token on E4B) once for all the tokens of a pass, with a vector dot product: passes over a few tokens 17% faster, single tokens 4%.
JANAS_DRAFT_VOCABsets the draft head's size, to experiment.
[2026-09-27] - janas-try reports only this run
- A test report carries only the speeds this run measured. The reports of issues #10 and #11 also showed the bench files earlier runs had left behind - another level, an older commit - as if they were new. They stay in the work directory, out of the report.
[2026-09-27] - Why a build has no GPU
- A build without GPU support says what was missing. The GPU needs both
glslangValidatorand the Vulkan headers when Janas is built; with the first and not the second (CachyOS withoutvulkan-headers) the build left the GPU out without a word, andjanas-try.shreported "built with GPU support". Nowbuild.shprints which piece is missing, the engine's "GPU: none usable" names it,janas-try.shchecks for the headers too, and the README gives the Arch packages.
[2026-09-27] - Gemma 4: E2B and E4B
- Gemma-4-E2B-it and E4B-it run and chat, tools included. Each token brings a few numbers for every layer (per-layer embeddings): a row of a table of their own plus a projection of the embedding (bf16), which open a branch of the residual after each feed-forward. The table, half of E2B's file, is never loaded: a token's row, a few kilobytes, is read from the file when it is needed, so E2B runs in 1.5 GB of memory from a 3.1 GB file, E4B in 3.1 GB from 5.0 GB.
- The last layers share keys and values: E2B's last 20 layers and E4B's last 18 compute queries only, and read the cache of the last earlier layer of their kind. E2B's feed-forward widens from 6,144 to 12,288 halfway; the engine takes a width per layer.
- Against llama.cpp, 200 tokens of text: E2B agrees on the most likely token at 191 positions, E4B at 195; prompt blocks give the same bits as token by token.
- Without reasoning, E2B and E4B are no longer handed an empty thought channel: their template does not write one (the 12B's and 26B's do), and given one they reasoned inside the reply. The chat template is now read whole: Gemma 4's is 19 KB, and the part after 16 KB was lost.
[2026-09-27] - Gemma 4: tools
- Gemma 4 calls tools, in its own notation: the functions declared in the system turn (
<|tool>declaration:name{...}<tool|>, parameters written as its template writes them, keys sorted), callscall:name{key:value}read into JSON, held by a grammar to a declared name and a well-formed value, and the tools' answers written inside the model's own turn, which it then continues. Checked token for token against the template rendered with Jinja: the first prompt with its declarations, a conversation loaded with a call and its answer (asjanas-servergives it), and the answer appended live. Gemma-4-12B-it called a weather function with the right argument and used its answer.
[2026-09-27] - Gemma 4: the 26B mixture, a sliding context, rings
- Gemma-4-26B-A4B-it runs and chats: 128 experts (eight a token) beside a dense feed-forward, the two as branches of their own - each normalized on the way in and out, then summed - a router reading its own input, and a scale per expert, folded into its weight.
gguf2jnssplits gate and up, which the file keeps fused, and widens with zeros the feed-forwards that are not a multiple of 256 wide (704 and 2,112 here): every weight type reads zero bytes as zero weights, so the products are unchanged. Files that need no widening convert to the same bytes as before. - The sliding-window layers keep a ring of the positions they can see
- the window and a prefill block, 1,536 for Gemma 4 - instead of the whole context, and attention skips the chunks before the window without reading them. The logits are the same to the bit with and without the ring (
JANAS_SWA_RING=0), 1,500 tokens past the window; kept conversations, on disk too, come back with the same replies.
- the window and a prefill block, 1,536 for Gemma 4 - instead of the whole context, and attention skips the chunks before the window without reading them. The logits are the same to the bit with and without the ring (
- A full context slides on Gemma 4 too: each layer's keys turned back with its own RoPE (two kinds, the full layers with their frequency factors), sixteen-bit keys, the rings. Qwen models slide as before, to the bit.
[2026-09-26] - Gemma 4 in the chat
- Gemma-4-12B-it chats: Gemma's format -
<bos>, turns opened by<|turn>and closed by<turn|>, the assistant called "model" - with its reasoning on or off: on,<|think|>heads the system turn and the model writes its thought channel, shown as reasoning; off, the channel is written empty, as the template does. The prompts were checked token for token against the model's own template rendered with Jinja and tokenized by llama.cpp, with and without reasoning. No tools yet, and a full context stops the chat instead of sliding.
[2026-09-26] - Gemma 4: keys and values at sixteen bits
- Gemma 4 keeps its keys and values at sixteen bits (integers with the same scale per position and head, some 250 times finer than eight bits); the attention reads them through kernels of their own, the scores over the float query. Qwen models stay at eight bits, bit for bit as before;
JANAS_KV=8or16overrides the choice. Against llama.cpp on 200 tokens of prose: 160 most likely tokens in agreement and a mean logit difference of 0.84, against 143 and 1.53 at eight bits. - A trap in the reference: llama.cpp's logits of Gemma 4 depend on its batch size - with batches of 64 tokens or more, the early positions see later tokens. The comparisons above use batches of 8, which agree with a prompt of two tokens run alone. The earlier figures (175 of 200) were measured against the larger batches.
[2026-09-26] - Why there is no GPU
- "GPU: none usable" now says why: built without GPU support (
glslangValidatormissing at build time), no Vulkan loader, Vulkan seeing no GPU, the GPU turned down with its name and what its driver lacks (Vulkan below 1.3, a feature, the subgroup size), or turned off (eco mode,JANAS_GPU=0). The same reason is in the model's description (janas_llm_describe). A tester's RTX 5080 was reported as "none usable" with nothing more to go on. janas-tryreports what Vulkan sees (each device's name, kind and version), whether Janas will be built with GPU support, and whether it runs under WSL, before building and in the report.
[2026-09-26] - Gemma 4, first part (not yet usable)
- Gemma 4 (
gemma4) runs its forward pass, on Gemma-4-12B-it: layers with a sliding window of 1,024 positions and every sixth a full one, each kind with its own head size (256 and 512), KV heads and RoPE (the full layers with Google's per-pair frequency factors); V taken from K where a layer has no V projection, and normalized; norms after attention and after the feed-forward, GELU, a scale per layer, soft-capped logits. The attention takes a sliding window and a score scale of its own, heads up to 512, and the KV cache is laid out per layer. - Gemma's tokenizer: BPE on UTF-8 characters with SentencePiece's spaces and byte tokens, in the same tokenizer as Qwen's. On a mixed text (Italian, English, code, CJK, emoji) it gives llama.cpp's 128 tokens.
- Not yet usable: the chat does not know Gemma's format, and with the eight-bit KV cache the logits drift from llama.cpp's (the most likely token agrees at 175 of 200 positions, mean difference 1.36; llama.cpp agrees with itself at 169 when its cache goes from f16 to eight bits). Every Qwen model gives the same logits as before, to the bit.
[2026-09-26] - Long prompts: the expert cache, the tuner, the ETA (issue #9)
- The expert cache no longer thrashes on large prefill blocks. A pass asks for experts layer after layer, and when one pass asks for more than the cache holds, least-recently-used eviction throws out exactly the expert wanted again soonest: on a 262K prompt of Qwen3-Next-80B, blocks of 256 read 2.17x the bytes of blocks of 64 (issue #9). In that case the victim is now the expert wanted again latest - the current layer's others, then the layer before, and back from there. Measured on the 80B (4,096 tokens, 12 GB of cache, four bits, blocks of 256): 238 GB read at 38.2 token/s before, 172 GB at 45.8 now; blocks of 64 are unchanged.
- The tuner tries alternatives ever more rarely while they lose: at a fixed 128 passes it made one slow block every 32,768 tokens of a long prompt. The gap now doubles up to 1,024 passes and returns to 128 as soon as a try beats the choice.
- A prompt's time left follows the cost of a token as the context grows (
input_eta_seconds, new at the end ofjanas_llm_chat_statsand inllm.bi): a line in the position, fitted on the blocks read and integrated to the end of the prompt, instead of the latest speed, which promised 2.5 times too little at 32K of a 262K prompt.
[2026-09-26] - IQ3_S weights
- IQ3_S weights, the last type unsloth's "UD" files of the Qwen family were missing: 256 weights in eight groups, each weight a level of a 512-entry grid, with its sign and a scale per group. A group's eight grid entries come in one gather, the signs go to the activations, and the integer sum is exact, one fma per 256 weights; AVX2 and AVX-VNNI give the reference's bits. The grid is ggml's, copied as data (
src/llm/iq3s_grid.h, MIT, as ggml is) since it defines the format. The weights match ggml's to the bit on random blocks; Qwen3.5-0.8B requantized to IQ3_S agrees with llama.cpp at a mean logit difference of 0.123, the disagreements near-ties.
[2026-09-26] - Q3_K weights, and unsloth's UD files of Qwen3.5-9B
- Q3_K weights: 256 three-bit values in sixteen groups with 6-bit scales. The values are summed unsigned (0-7) and the -4 of each is taken from the activations' block sums, so the integer sum of a block is exact and one fma follows, as for Q4_K; AVX2 and AVX-VNNI give the reference's bits. The weights match ggml's to the bit on random blocks; Qwen3.5-0.8B in Q3_K_M agrees with llama.cpp at a mean logit difference of 0.115.
- A DeltaNet's alpha and beta in F16 or BF16 are taken too: unsloth's "UD" files of Qwen3.5-9B keep them so. Qwen3.5-9B UD-Q4_K_XL - Q4_K, Q5_K, Q6_K, Q8_0, IQ4_XS and F16 in one file - runs and agrees with llama.cpp at 0.110, as its Q4_K_M does.
[2026-09-26] - Q4_0, Q4_1 and Q5_1 weights
- Models quantized in Q4_0, Q4_1 and Q5_1 convert and run - the types of Google's QAT Gemma files and of many older GGUFs. Q4_0 is IQ4_NL's layout with evenly spaced levels, so it runs IQ4_NL's kernel with another table; Q4_1 and Q5_1 (weights d x q + m, Q5_1 with a fifth bit) share a kernel that takes the minimum's share from the activations' block sums, exactly. On one to eight vectors, AVX2 and AVX-VNNI give the scalar reference's bits. Checked against ggml on 4,096 random blocks of each (the weights identical to the bit), and on Qwen3.5-0.8B against llama.cpp: the Q4_0 file (which holds Q4_1 tensors too) at a mean logit difference of 0.109, a Q5_1 one requantized from the Q8_0 at 0.114, the disagreements near-ties.
- A DeltaNet layer's alpha and beta are taken in any type Janas unpacks (they came in Q5_1 in the requantized file), not only f32 and Q8_0.
[2026-09-26] - IQ4_NL and IQ4_XS weights
- Models quantized in IQ4_NL and IQ4_XS convert and run: the two 4-bit types whose sixteen levels are spaced unevenly, closer near zero, that many of the most downloaded GGUF files use. Their kernels are Q8_0's with a table lookup in front (one
pshufbturns 32 indices into 32 weights), on one to eight vectors, AVX2 and AVX-VNNI giving the same bits as the scalar reference; IQ4_XS sums its eight scaled groups exactly and takes one fma per 256 weights. Checked against ggml on 4,096 random blocks of each (the unpacked weights identical to the bit) and on Qwen3.5-0.8B in both types against llama.cpp: mean logit difference 0.11, as the Q4_K_M file of the same model, and where the most likely token differs the two were within 0.46 of each other in llama.cpp's own logits. - Not yet: files that mix in IQ3_S or Q3_K (unsloth's "UD" quantizations), and the speed of these kernels, which has not been worked on - reading a prompt, Qwen3.5-0.8B went at 230 tok/s in IQ4_NL and 155 in IQ4_XS against 289 in Q4_K_M.
[2026-09-26] - janas-bench measures the drafts as a chat meets them
- The "with MTP" figures were too high, and this is why.
janas-benchmeasured the prediction block by continuing its own prompt, one paragraph repeated, greedily - and a model continuing prose copies the paragraphs it has seen, which the block guesses almost every time. On Qwen3.5-2B it kept 95% of its drafts there and reported 107 tok/s; a real chat with the same model kept 34-68% and wrote at 43-62. Every "with drafts" figure in the README until today (Qwen3-Next-80B-A3B 30, Qwen3.6-35B-A3B 29, Qwen3.5-9B 26, Qwen3.5-2B 107) came from that measurement. - Now the block is measured on a chat reply: an ordinary question in the model's chat format, the first 64 tokens of the answer, once greedy and once sampled as the chat samples (temperature 0.7, top-k 20, top-p 0.8, fixed seed), each with the share of drafts kept. The prompt for the reading speed is prose that does not repeat itself; three rounds by default, as the README always said (it was two).
- What the block is worth depends on the text: on Qwen3.5-2B, 68% of the drafts kept on an English explanation, 34% on Italian prose, where the engine drafts less and the gain all but goes.
- The chat runs on the cores
janas-benchmeasures: every usable core, the performance cores with their threads and the efficiency cores, never the low-power island. The tuner used to try one thread per performance core and their threads alone as well, and after a stretch of long context a chat could settle on 12 threads where the bench had reported 20: the same machine, two speeds. Measured on six models, from Qwen3.5-2B to Qwen3-Next-80B-A3B, all the cores were the fastest or level in every kind of pass, and the others lost 5-20% writing; so the threads are fixed now, and the tuner only decides whether the GPU helps.JANAS_TUNE_THREADS=1brings the other counts back, to experiment with. janas-benchreports what the engine will run: its result line is the configuration the engine chose for each kind of pass, as a chat or the server will use it, not the fastest figure of each column.
[2026-09-26] - Qwen3.5 dense models
- Measured on the development laptop, idle,
janas-bench: Qwen3.5-9B reads a prompt at 46 tok/s and writes at 13, 22 with the block in a chat; Qwen3.5-2B at 215, 49 and 75. (First published as 26 and 107, with the block measured on repeated text: see the entry above.) llama.cpp on the same 9B file and machine (llama-bench, build of 18 September) did best at 32 and 9.0. README and MODELS.md list the three dense Qwen3.5 models, their fingerprints and how to get their block (hf:with the checkpoint's revision). - The dense Qwen3.5 models run (
qwen35: Qwen3.5, 3.6 and 3.8 without experts, half of the GGUF downloads on Hugging Face): the layers of Qwen3.6-35B-A3B with a dense feed-forward, one row of the architecture table. The DeltaNet's alpha and beta come in Q8_0 in these files and are made f32 once, exactly, at load. Checked against llama.cpp on the same files, 400 tokens of Italian prose and 400 of Pascal: Qwen3.5-0.8B agrees on the most likely token at 387 and 386 positions, Qwen3.5-2B at 376 and 388, mean logit difference 0.10-0.13, and where they differ the two candidates were within 0.53 of each other in llama.cpp's own logits. Chat and tool calls (XML) work throughjanas-chatandjanas-server. hf2jns_mtpconverts the MTP block of a dense Qwen3.5 too, its feed-forward as the one slot, and takes the block from several shards (hf2jns_mtp <main.jns> <shard>... <out.jns>): Qwen3.5-9B spreads it over three. The number and shape of the experts come from the model and the checkpoint instead of Qwen3-Next's written in; its block converts to the same file as before, to the byte. On Qwen3.5-9B the drafts are accepted at 53% (prose) and 68% (code), the text the same as without.hf2jns_mtp <main.jns> hf:<owner>/<repo> <out.jns>reads the MTP block straight from Hugging Face, with range requests: only themtp.*tensors, 487 MB of Qwen3.5-9B's 14 GB of shards, in under a minute, with no other program (Janas's own HTTPS client, the one of its MCP transport, now following redirects). The file it writes is the same, to the byte, as from the shards on disk. A server that ignores a range stops it, never a whole file downloaded. Qwen3.5-2B's block too: drafts accepted at 78% and 64%. Neighbouring tensors are read in one request (up to 256 MB): Qwen3-Next's block, 1,500 tensors, came in 27 minutes one connection each and comes in 5 now, to the same bytes.- The memory counted for the context asks the architecture table which layers have keys and values, instead of two names written in.
[2026-09-26] - Architectures as data
- What sets one architecture apart from another is a row of a table (
src/llm/arch.c): recurrent layers and their layout, a gated query, a shared expert, which norm comes before the feed-forward, how value heads share key heads, MTP layers in the file. The flagsnextandq35are gone; a new family is a row plus the code of what it has that no other had. The logits are the same bits as before on Qwen3-4B, Qwen3-30B-A3B, Qwen3.6-35B-A3B and Qwen3-Next-80B-A3B, token by token and in blocks, and so are the MTP drafts of the 35B and the 80B. model.cdivided by what it does: it was 2,597 lines; the passes stay in it, the attention, DeltaNet and feed-forward layers, opening and closing, and the threads with the experts' profile each have a file of their own (model_attn.c,model_rec.c,model_ffn.c,model_load.c,model_tune.c). Code moved, not changed: the same bits again on the four models.build.shmakes its archives anew every time:aronly adds and replaces, so the object of a source since removed or renamed stayed inlibjanas.aand was linked twice (going back to an older commit failed to link).
[2026-09-26] - Qwen3-Coder's tool calls, run
- Qwen3-Coder-Next calls tools through
janas-server, in the XML way its template describes: asked for the weather and a list of files, it made both calls at once with valid arguments, and used their answers in the reply; the second turn read 66 new tokens and reused 496. The README no longer calls this way untried.
[2026-09-25] - Builds with clang
janas-try.shno longer skips a 16 GB machine: it compared the memory the kernel reports with 16 GiB exactly, and a machine sold with 16 GB shows less (the kernel and an integrated GPU keep some), so every one of them lost the medium and full levels, and an 8 GB one the quick level. 85% of the memory a level asks for now counts as enough.CC=clang ./build.shworks (clang 19, every variant under-Werror): two float conversions ofRAND_MAXin the benchmarks, a string concatenation clang took for a missing comma intest_grammar, and thedieof the two converters marked as not returning. The test suite passes, and the 4B's answer at temperature 0 is the same text built with gcc or clang, with AVX-VNNI or AVX2 kernels.
[2026-09-25] - tools/janas-try.sh: try Janas and report, in one command
- One command to try Janas on a machine: it builds and tests, downloads a model (quick 4B, medium 35B-A3B, full 80B-A3B with MTP), checks every file against its SHA-256 and every conversion against MODELS.md, checks the answer at temperature 0 against the text every machine should write (the same with any thread count, AVX2 or AVX-VNNI, GPU or not), measures with
janas-bench, triesjanas-server, and writes a report with no names or paths. It asks before each long step and before sending; the issue goes throughgh, or a link fills the new test report form. Nothing to install, no root. - It measures only a machine left to itself, or says it did not: before the speed test it samples how idle the CPU is for five seconds and, when something else is running, explains that the numbers would not be the machine's and offers to wait, measure all the same or skip. The report says how idle the CPU was and the load average - general figures only, never which programs ran.
- MODELS.md: the fingerprint of the MTP block is the one of the file
hf2jns_mtpwrites since issue #6.
[2026-09-25] - The converter is C: nothing but the build to run Janas
gguf2jnsis now a C program, built with everything else, in place of the Python script: trying Janas needs a compiler and nothing more. It writes the very same bytes - the SHA-256 fingerprints in MODELS.md, taken from the Python one, come out identical for Qwen3-4B, Qwen3-30B-A3B and Qwen3-Embedding-0.6B.- Its GGUF reader (
src/llm/gguf.c) takes nothing on trust: lengths and counts checked against the bytes there are, every tensor's data against the size of the file;test_gguffuzzes it.
[2026-09-25] - max_tokens is the answer's; Open WebUI tried
- In janas-server's chat completions,
max_tokenslimits the answer alone, the reasoning not counted;max_completion_tokens(and the Responses API'smax_output_tokens) still counts both, as OpenAI defines it. Open WebUI asks for a chat's title withmax_tokens1000, and with a model that reasons at length the reasoning used it all: no title. The library hasmax_answerfor it, besidemax_reply. - Open WebUI 0.11.4 tried against janas-server through its API: streamed chat, titles, tags, follow-up suggestions, a document indexed with Janas's embeddings and answered from; the conversations kept computed across its side requests. The README says how to run them together (
--think off).
[2026-09-25] - The MTP block of Qwen3-Next loads again (issue #6)
hf2jns_mtpwrote a file the engine refused since model files gained levels (version 3, bit planes): its layer's levels were left at zero, and loading it with--mtpfailed withlayer 0: bad levels. It writes them now, and a file it wrote before is read as what it meant (a slot with no planes), so nothing has to be converted again.- The error of an MTP file now names that file, not the model's.
jns_checksays when a file has such levels, and that converting it again is optional.
[2026-09-25] - MCP: a cancel between rounds, the servers' instructions
- janas-server: a
/responses/{id}/cancelarriving between two MCP rounds of a background response now stops it - the next round is no longer started, and the tool call running is stopped too. Checked under ThreadSanitizer. - janas-mcp works under Claude Code (2.1.282, which speaks revision 2026-07-28): its discover and tools/list results now carry the caching hints that revision requires (
ttlMs,cacheScope), without which Claude Code refused the tool list. generate and embed tried from it. JANAS_METRICSnaming a file that cannot be opened is now said on stderr.- Tried against servers of others: the reference filesystem and git servers over stdio (npx, uvx), DeepWiki over HTTPS from the client and from janas-server's Responses API through the
openaiSDK. - janas-chat
--mcp-instructions: what the MCP servers say of their tools goes after the system message, marked as theirs (off by default).janas_mcp_instructionsin libjanas_mcp gives it to any program.
[2026-09-25] - A long prompt says how it goes, and what it costs
Asked for in issue #5, after a 262,144-token run that read its prompt for fifteen hours without a word.
- The prompt is read a block at a time:
janas_llm_chat_nextreturns an empty piece after each block of 256 tokens, so a program has control while a long prompt is read - to show progress, or to stop (janas-server now stops a cancelled request in the middle of its prompt). Same speed as before, measured. janas_llm_chat_statssays more, appended to the struct: the stage, the tokens read so far, time to the first token, time building the prompt, processor time of the reading, the threads chosen, the memory of the context and the process's peak, and the prompt by its parts (system message and tools, conversation, last message).max_inputin the chat's parameters, and--max-inputin janas-server: a longer prompt is refused with its numbers (JANAS_LLM_ELIMIT; a 400input_limit_exceededwithinput_tokens,max_input_tokens,excess_tokens), never cut.--warn-inputtakes it and says so.JANAS_PROGRESSandJANAS_METRICSfor any program on the library: progress lines on stderr, and JSON lines of events (open, prompt, progress, first token, reply, refused), counters only, never text. janas-chat shows the progress in its status line, has/contextand--progress; janas-server has--metricsand--progress.- Documented: the prompt is read in
janas_llm_chat_next, not when it is sent or loaded;context_usedholds the token that ended the reply, hence prompt + output + 1.
[2026-09-25] - MCP servers in janas-server's Responses API
- Tools of type
mcpinPOST /v1/responses, as OpenAI's API has them: the server reaches the MCP server a request names (server_url,headers,authorization) through libjanas_mcp, starts the output with anmcp_list_toolsitem, and gives the tools to the model asserver_label__tool. A call needing approval (require_approval,alwaysby default) ends the response with anmcp_approval_requestand runs when anmcp_approval_responseanswers it; one that needs none runs at once as anmcp_callitem and the model goes on in the same response, a round at a time.allowed_tools, the approval filters,tool_choiceof typemcp, streamedresponse.mcp_*events, and the items read back from the history.--no-mcprefuses them.
[2026-09-25] - https:// MCP servers
- TLS for the MCP client, on Mbed TLS 4.2.0 compiled in (
src/third_party/mbedtls, unmodified, with TF-PSA-Crypto):https://servers from libjanas_mcp and janas-chat, TLS 1.2 and 1.3, the server's certificate checked against the system's certificate authorities (SSL_CERT_FILEandSSL_CERT_DIRoverride them) and against the host name. Mbed TLS's symbols stay hidden inside libjanas_mcp; libjanas_llm has none of it.
[2026-09-25] - MCP servers over HTTP
- libjanas_mcp reaches servers over Streamable HTTP (
janas_mcp_open_url, orurlandheadersin the configuration, so janas-chat uses them too): every message a POST, the answer a JSON object or an event stream, the body's fields mirrored intoMCP-Protocol-Version,Mcp-Method,Mcp-Nameand theMcp-Param-*headers a tool's schema asks for withx-mcp-header(base64 where a header cannot carry the value; tools whose marks break the rules are left out). Both eras: a modern server found by its answer toserver/discover, a legacy one by its refusal, theninitializeand its session, ended with DELETE.https://needs TLS, which is not built in yet.
[2026-09-25] - Janas as an MCP server
janas-mcp: the model offered to MCP clients (Claude Code, editors, agents) over stdio, as the toolsgenerate(a question to the model, its answer back) andembed(with--embedding-model). Both protocol eras: requests of revision 2026-07-28 served statelessly, and theinitializehandshake for revisions 2025-11-25 back to 2024-11-05. Pings are answered during a generation, andnotifications/cancelledstops it.
[2026-09-25] - Tools from MCP servers in the chat
- janas-chat gives the model the tools of MCP servers: those listed in
~/.config/janas/mcp.json(or--mcp-config FILE), in themcpServersformat of the other clients, started as child processes and spoken to over stdio, in both eras of the protocol - the stateless revision 2026-07-28, found by probing withserver/discover, and the handshake of 2025-11-25 for the servers that predate it. Every call is shown and waits for yes, no or always-this-tool;/mcp autoand--mcp-autorun them without asking,/mcplists servers and tools,--no-mcpstarts none. libjanas_mcp, the client as a library of its own (include/janas/mcp.h, andmcp.bifor FreeBASIC), with nothing of the engine in it.janas_llm_chat_send_resultsinlibjanas_llm: what the tools answered to the calls of the last reply, sent in the conversation the library keeps, the model's reply following as after a message.tests/mcp_fake_serverandtest_mcp: the client against a server of each era, and the readers of JSON-RPC lines and configuration files fuzzed.
[2026-09-24] - The memory left to other programs can be chosen
--reserve GIBin janas-chat, janas-server and janas-bench, andjanas_llm_params.reserve_bytesin the library: when the expert cache takes the free memory, how much of it stays free for the rest of the machine. The default is unchanged, a fifth of the memory and at least 2 GiB. More keeps a busy desktop out of swap, at the price of a smaller cache on a model larger than the memory: on the 32 GB test machine, Qwen3-Next-80B-A3B's cache went from 20.6 to 18.8 GiB with--reserve 8. The conversations kept computed size their automatic ceiling on it too.
[2026-09-24] - Prompts read in blocks of 256 tokens
- A prompt is now read 256 tokens at a pass instead of 64, so that a mixture-of-experts model reads each expert once for four times as many tokens, and gives it enough of them to run at full speed. Attention and the recurrent log still work 64 tokens at a time inside the block, and the logits are bit for bit those of reading token by token (checked on Qwen3-4B, Qwen3-30B-A3B and Qwen3-Next-80B-A3B). Reading a 1024-token prompt: Qwen3-Next-80B-A3B 68.4 → 75.1 tokens/s (+9.9%), Qwen3-30B-A3B 92.4 → 97.5 (+5.5%), the dense Qwen3-4B unchanged (87.9 → 87.3). Replies are as fast as before.
JANAS_PREFILL_BLOCK(64 to 4096) sets another size; larger blocks gained nothing more here, and 1024 was slower on Qwen3-4B and on Qwen3-Next-80B-A3B.
[2026-09-24] - The chat's cache share moves while it waits; attention's memory counted
- The share of the expert cache filled, in janas-chat's footer, was drawn only at the start and after each reply: it showed whatever it was at that moment - 60% at one start, 0% at the next - and kept it. The footer is now drawn again while the editor waits for a key, when the share changes, and the share goes when the filling is over; on Qwen3-Next-80B-A3B it climbed from 0% to 98% in nine seconds and went, with the cursor never leaving the line being written. A filling overtaken by the conversation's own requests (they filled the cache first) now ends:
janas_llm_preloadthen gives a total equal to what it loaded, instead of a share stuck below 100%. - The partial results of attention grow with the context like the keys and values, and were left out when the memory was shared out: 1.08 GB at 262,144 tokens of Qwen3-Next-80B-A3B, taken from the margin left to other programs. They are counted now (
janas_attn_part_bytes).
[2026-09-23] - Conversations kept computed, and a store that outlives the server
- Conversations kept computed. The model holds one sequence, so a server whose clients take turns - or one client that asks on the side for titles, tags and suggestions, as Open WebUI does four times a turn - had each conversation read again from the start when it came back. Now the last few are copied out of the model and back in:
janas_llm_chat_keepin the library,--keep N(8) and--keep-memory GIBinjanas-server. The copy is of the keys and values of every attention layer (the prediction block's included) and of the recurrent state of Qwen3-Next and Qwen3.5/3.6. On Qwen3-4B, two conversations taking turns reuse 133 of 160 and 119 of 143 tokens at their second turn, against 3; on Qwen3.6-35B-A3B, 133 and 119 against 0; the replies are the same text either way. - A second reply to the same prompt on models with a recurrent state (
nabove 1, a reply asked again) read the whole prompt again, since that state cannot go back. The prompt is now kept before the reply: 143 of 143 tokens found computed on Qwen3.6-35B-A3B. The system message is kept on its own too, so that a new conversation with the same one starts from it. - On disk, below the memory:
janas_llm_chat_keep_disk,--keep-disk GIB(8, never more than a quarter of the free space), in~/.cache/janas. A copy of at least 1024 tokens is written when it leaves the memory and when the server stops, never at every turn; a long system message with its tools is then read once, also across restarts. A new chat found 2012 of 2013 tokens computed on Qwen3-4B (a 153 MiB file) and 2103 of 2103 on Qwen3.6-35B-A3B. - The store outlives the server. Stored completions, responses and conversations are written one file each to
~/.local/share/janas/server(--store DIR;--no-storekeeps them in memory only), readable by their owner alone, one server at a time per directory. A response left running in the background by a killed server is found as failed. - Fixed: the memory a long context takes was counted four times on Qwen3-Next and Qwen3.5/3.6, whose files give one count of key-value heads for all layers: every layer was counted, not only the attention ones. Opened at 262,144 tokens, Qwen3-Next-80B-A3B set 13.1 GB aside for 3.3 GB of keys and values, left its expert cache 6.7 GiB, and fell back to four bits and eight experts; now 13.5 GiB and all ten experts. Reading 15,904 tokens went from 22.1 to 30.4 tokens/s and the reply from 14.2 to 16.9, with the three facts hidden in the text found either way.
[2026-09-23] - It builds with GCC 13 again
- Ubuntu 24.04's compiler stopped the build (issue #4): GCC 13 drops a
#pragma GCC unrollon a loop whose condition holds a?:and warns, and-Werrormade the warning an error. The bound of the two loops is now a constant of its own; with GCC 14 the machine code is the same, byte for byte. Checked with GCC 13.3.0 from Debian's packages, also with the options Ubuntu's GCC turns on by default.
[2026-09-23] - The first ring of OpenAI's API, whole
Every operation of the API that a local text model can serve now answers: 27 of the specification's 345, against 4.
- Tools in chat and in responses: the tools written into the system message as the model's template writes them, and every call held to a grammar of the functions and their parameters from the moment the model opens it, so its arguments are always valid JSON for the schema;
tool_choicenone, auto, required or a function, parallel calls, the calls of earlier turns and the tools' answers in the history. Tried on Qwen3-4B: two calls at once, their answers sent back, the reply using them, the second turn reusing 304 of its 351 tokens; the XML way of Qwen3.5 and 3.6 on Qwen3.6-35B-A3B the same. Qwen3-Coder's description of the tools is written from its template and not yet run. - JSON output held to a grammar token by token: a JSON object, or JSON valid against a JSON Schema (types, properties in order, required ones, items and bounds, enum, const, anyOf, $ref with recursion).
- Log-probabilities with up to 20 alternatives;
n,best_of, presence and frequency penalties,logit_bias; several prompts, token ids,echoandsuffixin completions. - Stored chat completions: kept on request, listed with metadata filters and pages, read, changed, deleted, their messages listed.
- The Responses API with its conversations: items in and out,
previous_response_id, streamed events, responses in the background and their cancelling, input items, token counts, compaction into a summary; the eight operations of/conversations. - Embeddings from a second model (
--embedding-model, e.g. Qwen3-Embedding-0.6B): against llama.cpp on the same file, a cosine of 0.977 to 0.996 between the vectors of the same text. - Moderations asked of the chat model, with the scores taken from its log-probabilities: a general model with instructions, not a trained classifier.
- In the library, for every program: a sampler of its own with a filter of the tokens that may come next, grammars with the model's special tokens as symbols, JSON Schema as a grammar, a small JSON reader, conversations built message by message with their tool calls, token counts, fill in the middle, embeddings. The text of a reply is unchanged byte for byte where none of it is asked for.
- While a request waits, its stream says where it stands in the line, and the test page shows it.
[2026-09-23] - janas-server: the model behind OpenAI's API
Asked for in issue #3: an HTTP API that clients written for OpenAI's can use, so that a model running on this machine can sit behind a chat front end, an editor or a script without anything written for Janas.
janas-serveranswersGET /v1/models,POST /v1/chat/completionsandPOST /v1/completions, whole or streamed as server-sent events, with the reasoning apart inreasoning_contentand the counts inusage. It listens on this machine only unless told otherwise, asks for a key when given one, and warns when it is opened to the network without one. Requests run one at a time, in the order they came; a client that goes away stops its reply.- The whole of OpenAI's API is in its route table, generated from the specification (2.3.0, 345 operations) by
tools/openapi_routes.py: what is not written yet answers 501 and says why, and what a request asks that is not written yet (tools, JSON output, log probabilities) is refused with a 400 rather than ignored. - A conversation is not read twice. OpenAI's API sends the whole conversation with every request; the server reads again only what is new. The replies it wrote are remembered with their reasoning, so the answer a client sends back without it still matches. Checked: three turns through the server and through
janas-chatgive the same words, and the second turn reads 18 tokens of 260 or 19 of 200. --testserves Janas-Chat Web, a chat page inside the program: the reasoning in grey, the context and the speed under each reply, and the marks a model writes read asjanas-chatreads them, plus quotes, tables, struck text, six levels of heading and links that ask before they open. Markdown sample shows every mark it knows without asking the model.- Two libraries of others, in
janas-serveronly: GNU libmicrohttpd 1.0.10 (signature checked) and yyjson 0.13.0, unchanged insrc/third_party/.build.shcompiles them apart and never intolibjanas_llm. - In the library:
janas_llm_chat_loadtakes a conversation whole,janas_llm_chat_promptcontinues raw text,janas_llm_default_systemis the system message both programs now share, and the reply statistics say why a reply ended and how much of its prompt was already computed. The FreeBASIC binding follows, checked withfbc. - In
janas-chat: the title is Janas-Chat; a mark at the very end of a reply (**bold**,`code`) was printed instead of read, and now is read; an underscore alone at the edge of a word is italic, so**_this_**works whilesnake_caseand__init__stay as they are; and the system message asks the model not to introduce itself unless asked, which on Qwen3-4B turned the whole presentation it gave to "Ciao!" into "Ciao! 😊". - Checked: the unit tests in release, asan and tsan, a test that sends twenty thousand mangled requests to the parser, the engine's logits unchanged byte for byte, and the server under asan and tsan. The only races tsan reports are two inside libmicrohttpd, formal and harmless in the mode it runs in.
[2026-09-23] - Tests: two failures on every machine but one
Reported in issue #2 from a Ryzen AI 9 HX 370: test_expert_cache could not write its file and test_tuner found its saved choices missing. Both had the same cause, a habit of the development machine leaking into the tests: they wrote under $HOME/tmp, a directory that exists there and almost nowhere else.
test_expert_cachewrites its file in$TMPDIR, else in/var/tmp- a disk, where O_DIRECT works, which it does not on the tmpfs/tmpoften is. If it cannot, it says so and says what to set.test_tunermakes a scratch directory of its own withmkdtempand removes it when done.- The engine made only the last two levels of its cache directory. With
XDG_CACHE_HOMEpointing somewhere that did not exist yet, the tuner and the expert profile were silently not saved. Every missing level is made now.
Checked the way it failed: with a HOME that has no tmp and no TMPDIR, the old binaries print exactly the two failures of the report and the new ones pass.
[2026-09-22] - Janas-LLM: reading a prompt, 8 to 12 per cent faster
Measured before touching anything: while a block of 64 prompt tokens went through the model, nineteen of twenty threads spent 9% of the time on Qwen3-4B and 13% on Qwen3-30B-A3B waiting for the calling thread, which did per-token work alone - the norms and the quantization of the activations, the normalization and rotation of each head of q and k with the eight-bit copy into the attention cache, the router's choice of experts, and the sum of the experts' outputs.
- All of that now runs on the pool, token by token or head by head. Each token takes the same operations in the same order as before, including the order in which it adds up its experts, so nothing changes in the result: the logits of Qwen3-4B, Qwen3-30B-A3B and Qwen3-Next-80B-A3B are identical byte for byte to the previous version's, over 256 tokens, both in blocks and one token at a time.
- Prefill, 1024 tokens, cache warm, 20 threads, old and new alternated: Qwen3-4B 82.4 to 88.5 tok/s (+7.6%), Qwen3-30B-A3B 82.7 to 92.6 (+12.1%), Qwen3-Next-80B-A3B 58.3 to 64.1 (+9.9%). The time the calling thread works alone falls to 0.3% and 0.9%.
- Decoding: the heads of q and k are spread over the pool there too, which saves about a third of a millisecond a token: Qwen3-30B-A3B 28.56 to 28.92 tok/s over three alternated rounds, Qwen3-4B unchanged within the noise.
[2026-09-22] - Janas-LLM: three ways the chat laid text out wrong
All three found by using it, and all three in the same place: what the chat knows about where a line may break.
- A code block inside a list never closed. A model asked for examples put a Python block inside a numbered list, so it indented the fence that ends it. The markdown machine closed a block only on a backtick in the very first column, so the fence came out as text and everything after it stayed inside the block - where nothing else is a mark, which is why the rest of the reply arrived with its stars and its bullets as written. Spaces no longer end the start of a line inside a block, which is the rule the rest of the machine already used.
- And the indent of that fence stayed on the screen. A fence that opens is swallowed whole, its line with it, but the spaces before it had already been printed and stayed behind on a line whose end had gone: the first line of code then came out with its own indent added to the fence's. The spaces a line begins with are now held until it is known whether a fence follows them.
- A list item that wraps now lines up under its own text, not under its mark, which is what a reader expects and what this looked wrong without. A line that begins with a bullet or with a number and a dot has its continuations pushed right by the columns its mark took; everything else is untouched, which is what separates this from the hanging indent on all text that was tried earlier in the month and taken out again. Below about eight columns of room it is dropped. Checked at 74, 45, 34 and 20 columns.
- The box you type in broke words in half. It wrapped by counting columns while the conversation has wrapped by word since September. It uses the same rule now, and so does the line repeated in the conversation after it is sent.
term_box_rowshad a copy of the counting and now asks the layout, so the two cannot disagree about where a word goes. One thing the change brought with it and that had to be caught: a cursor just after the space a break leaves at the end of a row would have been drawn one column outside the box - seen at sixteen columns, position fourteen, column seventeen of sixteen. It rests on the last column instead. Checked at every position of a line at 74, 40, 24, 16 and 12 columns.
[2026-09-22] - Janas-LLM: the engine was slower with its own speculation on
Four things, found by pulling one thread: what a draft is worth depends on what a second token in a pass costs, and nobody had ever measured that. It is 6% on a dense model, where the weights are read once for both tokens, and 28% on a mixture, where two tokens route to different experts. Against that, drafts copied from the conversation survive 11% of the time on ordinary prose and 64% on a page of tables, and at the temperature a chat actually uses they survive 25%. The engine asked for four of them regardless.
- The drafts copied from the context now have a planner, the one the prediction block already used, with the chance of a draft in each position measured per position rather than guessed and halved every 192 drafts so it follows the text.
spec_kgoes back to being a ceiling on what may be asked for. Before this, on ordinary prose at the chat's own temperature, speculation made the engine slower than not speculating at all: 0.97x on Qwen3-4B and 0.78x on Qwen3-30B-A3B, with the safety net parking the drafts for 137 passes out of 256. After: 24.3 and 26.3 token/s against 21.4 and 22.0, so 13 and 20 per cent recovered. - The sampler sorted thirty-four thousand tokens to use twenty. Choosing a token collected every logit within thirty times the temperature of the largest - about 34000 of a vocabulary of 151936 - sorted all of them, then kept the twenty
top_kasks for. It cost 2.4 to 2.9 ms a token on the calling thread while nineteen others waited, five to six per cent of a token of Qwen3-4B, and more with a larger vocabulary (Qwen3.6 has 248320). The twenty largest are now found in one pass with a heap of twenty. Paired against a binary differing only in this: sampling 0.51 s to 0.09 s, decoding 22.3 to 23.7 token/s over three rounds. --draftputs a small model in place of the copying. A model of the same family predicts where copying only repeats: Qwen3-0.6B drafting Qwen3-4B is accepted 78% of the time and gives 1.14x, 23.0 to 26.3 token/s. It shares the big model's compute pool, because two pools of twenty threads on twenty cores take the cores from each other - a pass of the drafting model cost 22.1 ms beside the target's pool and 7.2 ms alone. On a mixture it does not pay, and their answer stays the prediction block.- None of this changes a word of the output. A draft is never a token: the engine samples from the big model and keeps the draft only where the token it sampled is the very same one.
llm_genchecks it on every run, 256 tokens with drafts and 256 without from one seed, identical. - What was measured and refused: the planner of the prediction block is not conservative, whatever it looked like. Fixed numbers of drafts were tried against it on Qwen3-Next-80B - 26.2, 25.1, 23.6 and 21.0 token/s at one, two, three and four - and the planner gives 29.1. It wins by drafting on 79 passes out of 145 rather than by drafting more: its acceptance is 91% where a fixed single draft gets 72%. At four drafts speculation becomes a loss even there.
[2026-09-22] - Janas-LLM: the 35B's expert phase was never slow
This project's notes have carried, for two sessions, a figure described as the largest thing left open: the routed-expert phase of Qwen3.6-35B running at 40 GB/s where the same phase of Qwen3-30B runs at 62.4, with the difference unexplained after several candidates had been measured and dropped. There was nothing to explain. The three models run that phase at 65.3, 65.9 and 63.4 GB/s - within four per cent of each other - and the 40 was an accounting mistake in three parts.
- The phase must not be measured whole. It holds three things with three different rules for counting bytes: the cache, which is a wait on the disk and is 11.9% of the phase on the 35B in decoding and 25.8% on Qwen3-Next-80B; the
gateandupmatrices, plain Q4_K read in full every time; anddown, which is six-bit planes whose byte count depends on a level the engine picks from how much memory there is. The honest number comes fromgate+upalone, where nothing has to be guessed, andJANAS_EXPTRACE=1separates them. - The shared expert was not being counted.
qwen35moeandqwen3nexthave one, used by every token in every layer on top of thekrouted ones;qwen3moe, which is the 30B, does not. Comparing the two while counting only routed experts takes away from the 35B bytes it really reads. - And the 35B's shared expert is Q8_0, where Qwen3-Next's is Q4_K: 1.114 MB against 0.590 for the same 2048x512 matrix, 1.89 times the bytes. It carries 89.1 MB a token, 19% of all the
gate+upbytes, for one expert out of nine. That comes from the GGUF it was converted from, not from any choice here. - A fourth, smaller: the 35B declares 41 layers and runs 40. The last one is the multi-token-prediction block, which sits in the file and does not run when generating normally.
| Model | layers | k | routed matrix | shared | bytes/token | ms/token | GB/s |
|---|---|---|---|---|---|---|---|
| Qwen3-30B-A3B | 48 | 8 | 0.885 MB | — | 679.5 MB | 10.41 | 65.3 |
| Qwen3.6-35B-A3B | 40 | 8 | 0.590 MB | 2.228 MB | 466.6 MB | 7.08 | 65.9 |
| Qwen3-Next-80B-A3B | 48 | 10 | 0.590 MB | 1.180 MB | 622.8 MB | 9.83 | 63.4 |
- Checked by trying to break it. An arithmetic that comes out right can come out right by accident, so the count was asked to predict something it had not been fitted to: with
JANAS_EXPERTS=6on the 35B it saysgate+upshould take 5.65 ms. Measured, 5.60. Nine parts in a thousand. - What is left, and it is real: requantizing the 35B's shared expert to Q4_K would save 42 MB a token, 9% of the
gate+upbytes, about 1.7% of a token. It is not an obvious call - the shared expert is the one every token crosses in every layer, so it is the most sensitive of all to losing bits, which is very likely why whoever quantized the source left it at eight. The quality has to be measured first, against the oracles that already exist. Converter work, offline, with no line of the engine to change.
[2026-09-22] - Janas-LLM: the pool was not the bottleneck, and now there is a yardstick
This project's own notes opened the session with a plan: 21.2% of decoding sits in the thread pool's machinery, the engine makes some 340 dispatches a token and each one is a barrier, so fusing the three dispatches of the expert phase into one would take away two thirds of them. The premise does not survive being measured. What came out of measuring it is worth more than the plan was.
JANAS_POOLTRACE=1is new and stays: one line per call site - the name comes free from a macro aroundjanas_pool_run- with its runs, the threads they used, their wall time, and how much of that thread time was spent at the end of a run waiting for the slowest. What the table deliberately does not hold is the time between runs, the caller working alone; that is the difference between the wall of the pass and the sum of the column, and it is printed beside it. Off, it costs one variable read per dispatch.- The barriers cost 2.4% of the thread time, not 21%. On Qwen3-30B-A3B decoding 256 tokens, the matrix products - 89% of the time inside the pool - waste 1.4%. Work is claimed on demand there, and it shows.
- What the profile was really showing is the caller. The caller alone is 11.0% of decoding on Qwen3-30B and 2.9% on Qwen3-4B, whose experts all fit in RAM and which reads nothing from disk. The threads spinning under the pool's name in the profile are waiting for the caller, and the caller is waiting for the NVMe. The lever is the cache, not the barrier.
- So the expert phase will not be fused. It would attack 96 barriers out of 392, worth well under one per cent, against a concurrent rewrite of
run_expertswith per-expert dependencies. The two sites that do waste their threads - the activation and the attention merge, which divide the work by index rather than on demand - come to 73 ms out of 8272 together. The number does not justify the change, and saying so is the result. - The chunks of a matrix product now shrink towards the end of a run. They were a fixed
rows/(threads*4), about four per thread, so whoever claimed the last one kept everyone else waiting for as long as it took - and this machine's cores come in three speeds. A chunk is now1/(2*threads)of what is left to claim, between the same sixteen and two hundred and fifty-six rows: long at the start where the prefetcher earns its keep, short at the end where nobody should wait.
| Model, prefill at 20 threads | before | after |
|---|---|---|
| Qwen3-4B (dense) | 79.7 | 86.8 (+8.9%) |
| Qwen3-30B-A3B | 84.4-84.9 | 88.0-88.7 (+4.4%) |
| Qwen3-Next-80B-A3B | 56.1 | 59.0 (+5.2%) |
Decoding gains about one per cent at six and twelve threads and nothing at twenty; prefill gains most because there each row carries 256 vectors, so a chunk costs 256 times more and so does its tail. On bench_long the spread across three runs fell from 0.14 to 0.02 token/s - the runs became repeatable, which is worth as much as the per cent.
- The barrier was fixed anyway, on its own evidence. It ended on one counter that every thread decremented, which at twenty threads is twenty transfers of a single cache line from core to core, in series, and the last thing to happen before the caller can go on. Each thread now has a line of its own; the mutex and the broadcast are skipped when nobody is asleep, which between two dispatches a few microseconds apart is always. An empty run at twenty threads went from 2.75 us to 0.53. On the engine it is worth nothing measurable - those transfers were hiding under the caller's own share of the work - and that is precisely what makes it certain that dispatching is not where the time goes.
- A caution, since this session produced one wrong number before the right one. A first version of the probe reported "41.9% idle at the barrier" by adding the I/O pool's runs, which have eight threads and wait on the disk, to the compute pool's, and multiplying the sum by twenty. The true figure is 2.4%. Thread time has to be counted run by run, with the threads that run actually used.
[2026-09-22] - Janas-LLM: the expert phase taken apart
The entry below put the headroom at two per cent and said the engine was finished as an implementation. That was argued from the engine measured against itself, which settles nothing: it says there is no waste between a kernel and the phase around it, not that the kernel is good. Taken apart properly, the figure is about seven per cent, and it is in one place.
JANAS_EXPTRACE=1is new and stays: it splits the routed-expert phase into its parts - the two grouped products, the activation and its quantization, the final accumulate, the setup, and the cache - and prints them whenever the per-phase times are read, resetting as it goes. That last detail is the point of it. Prefill and decoding are different machines: decoding is 95.5% matrix product and its cache costs 2.7%, prefill is 67.4% and 29.5%, because prefill really does read from disk. Averaged together they say something true of neither, which is exactly the mistake this entry was written to correct.- There is nothing to take in the glue. In decoding the whole of the setup, the activation, the accumulate and the cache come to 4.5% of the phase, and the measured wait on the disk is 1% of a token. The 9% quoted earlier was prefill mixed in.
- The ceiling is the machine, not our probe - checked a second time, since it had already been wrong once today. Four independent load chains or sixteen make no difference, and software prefetch makes it worse: the hardware prefetcher is already saturating. 81 GB/s, and eight extra cores past twelve buy one per cent, so what saturates is the bus rather than the cores. Against a theoretical 102 to 119 GB/s for this machine's memory, that is 68 to 79%, and no kernel crosses it. At one vector the work is the read: a better kernel can only read fewer bytes.
- Scattered reading costs nothing. Large matrices drawn at random from eight gigabytes run at 77.6 GB/s against 78.4 for one contiguous matrix. The contiguity lever, taken literally, is not there.
- What is left sits in the expert product itself: 64.5 GB/s in the engine, against 70.2 for the same geometry in a probe and 78.4 for a contiguous read. Six of those points are the model's own shape - a single expert matrix of Qwen3-30B is 0.9 MB, and the size curve puts it there. Eight points are between the engine and its own probe at the same geometry, which is the first properly paired comparison of the day with a difference that survives it. The engine keeps its down matrix in six-bit planes where the probe uses four, and groups pairs through strides rather than separate matrices. The cause is not identified.
- Headroom, third figure of the day. This project published 1.32x this morning, then 1.02x, now about 1.07x. The first two were extrapolated from a yardstick taken somewhere else; this one is built on the phase itself, in decoding, separated from prefill. It is not promised to be the last.
[2026-09-22] - Janas-LLM: how much is left, and the answer is very little
The entry below this one closed by saying the engine reaches the metal on one phase and asking what the rest could be worth. Measured, the answer is two per cent, and the number published earlier today - a third more speed - does not survive. What follows is how that was settled, including the two guesses that were wrong.
- There are two yardsticks, not one. A bare Q4_K product over one contiguous matrix runs at 78.4 GB/s, 97% of the machine - measured on 1.2 GB, where the level-3 cache is 1% of it and cannot flatter the figure. The same product over expert-sized blocks drawn at random from eight gigabytes runs at 68.4. Reading scattered costs 12%, and that is the structural price of being a mixture engine, not a fault in the code.
- How wrong a yardstick can be: the same command on a 151 MB matrix reported 84.3 GB/s, above the machine's own read bandwidth, because a sixth of that matrix never leaves the cache between passes. 151 MB reads 84.3, 302 reads 80.8, 604 reads 79.2, 1208 reads 78.4. A benchmark that re-reads its data measures the cache unless it is far larger than one.
- Matrix size is the whole story, and it is bigger than reported this morning. Earlier today this was measured on twelve threads and over too narrow a range, and came out as 9%. On twenty threads, from one matrix of 0.9 MB up to one of 56.6, the throughput goes 44.3, 55.7, 62.5, 65.4, 66.1, 67.1, 68.4: a spread of 54%. A single expert of Qwen3-30B is 0.9 MB, at the bottom of that curve. The engine does not send them one at a time - it groups the eight a token needs into one product, which is what lifts 44.3 to 64.6. The recovery is already taken.
- Every phase is at 97 to 100% of what its own shape allows: output 77.4 against 78.4, the attention output 65.1 against about 66, the experts 62.4 against 64.6, the attention projections 57.7 against 58. There is no implementation left to win on this machine.
- Two ideas measured and dropped. Laying the experts a token needs side by side in the cache, instead of wherever replacement put them, is worth nothing - a slot holds gate, up and down back to back, so neighbouring slots still leave the gate matrices 2.65 MB apart, and neighbouring turns out not to matter at that size anyway. And the bit-plane format, which the kernels said costs 22% of the arithmetic, costs nothing on the engine: that 22% is measured where the matrix sits in the level-2 cache and the arithmetic is the limit, while in the engine it hides under the wait for memory. Both were expected to pay and neither does.
- What is left that is free: fusing the three attention projections into one matrix in the converter. Three matrices of 4.7, 1.2 and 1.2 MB become one of 7.1, which the curve puts at 65.4 GB/s against 58. On Qwen3-30B that is 0.59 ms of 32.4: two per cent. The same bits, offline, with nothing in the engine to change.
- Everything beyond that costs bytes - fewer bits for each weight, which is quality - or different hardware.
[2026-09-22] - Janas-LLM: the speed of the thing, measured again
All of it re-measured on an idle machine with the expert cache left to size itself, which is what the engine does when nobody tells it otherwise. The numbers published before were taken with the cache pinned to 20 GiB by hand - a setting the product never chooses on its own.
| Model | cache | prefill | decode | with MTP |
|---|---|---|---|---|
| Qwen3-4B (dense) | 1.6 GiB | 81.1 | 25.7 | — |
| Qwen3-30B-A3B | 17.5 GiB | 84.7-86.6 | 31.0-31.4 | — |
| Qwen3.6-35B-A3B | 20.5 GiB | 72.2 | 23.9 | 29.1 |
| Qwen3-Next-80B-A3B | 19.8 GiB | 63.4 | 24.5 | 34.2 |
- Qwen3-30B gains 4 to 6%: the cache cap fixed this session gave it 17.5 GiB where it used to get 16.3. Qwen3.6-35B gains 7% in decoding. Qwen3-Next-80B is where it was.
- Letting the cache size itself is as good as pinning it, or better. Checked in alternation rather than back to back, so that drift could not pass for an effect: on Qwen3.6-35B the automatic size gives 23.7 token/s against 23.1 and 23.2 for the old cap, on Qwen3-Next-80B 24.6 against 24.5 twice, with the speculative figure better in both.
- A first measurement on a fresh machine reads 7 to 15% low, and it is not the engine.
janas-benchfills the engine's tuning profile while it measures, so its runs are not independent of each other: from an empty profile Qwen3-Next-80B reports 58.5 / 22.6 / 29.9, and after a few runs the same binary on the same weights reports 63.4 / 24.5 / 34.2. Twice in an hour that rise was mistaken here for a regression that had been introduced, and twice a paired run said otherwise. Three runs, and take the numbers when they stop climbing.
[2026-09-22] - Janas-LLM: the ceiling we measured against was wrong
We have been quoting a number that does not hold, in this file and in the project's own notes, since the first week. It is corrected here in full, with how it was found and what it changes.
- The claim. This machine's memory reads at 83.9 GB/s, the engine's kernels reach 84% of that, and the missing 16% is the cost of unpacking 4-bit weights.
- What was wrong. The probe behind that number gave every thread an equal share of the work and waited for all of them. On a CPU with six performance cores, eight efficiency cores and two low-power ones, equal shares on unequal cores measure the slowest core, not the memory. It reported 81.4 GB/s on twelve threads, then a collapse to 69.4 on sixteen - a collapse that does not exist.
- How it surfaced. The bare Q4_K product, on a matrix far past the caches, reached 77.6 GB/s on twenty threads while the same machine's "pure read" was measuring 73.4. A kernel that also unpacks and multiplies cannot outrun a loop that only reads. One of the two was wrong, and it was not the kernel.
- The correct figure. With the work handed out dynamically - the way the engine's own thread pool has always done it - the read bandwidth is 80.5 GB/s and flat from twelve threads to twenty-two. The new probe sits beside the old one, which now carries a warning rather than being deleted.
- What it changes. The kernels are not at 84% of the machine. On twenty threads they are at 97%: 77.6 GB/s of 79.6, unpacking included. The missing 16% was never the dequantization - it was the four cores the old measurement could not use. The conclusion it had been used to support, that hand-written assembly has nothing left to win, still stands, and now for the right reason.
- Where the engine actually is. On Qwen3-30B-A3B the output projection - one large matrix, read end to end - runs at 77.4 GB/s, 97% of the machine and 100% of the bare kernel. The engine can reach the metal. A whole token averages 59.3 GB/s, because the other phases do not: the attention projections run at 57.7 and, on Qwen3.6-35B, the routed experts at 40.0. So the gap is not spread thinly across the engine - it sits in named phases. (This entry first went on to say that if every phase ran as fast as the output phase a token would take 24.8 ms and 30.5 token/s would become 40.3. That was wrong, and the entry above it says why: the other phases read a different shape and cannot get there.)
- Two things we looked for and did not find. The size of the expert matrices is not the cause: across a twenty-four-fold change in matrix size, at constant bytes and constant dispatches, the throughput moves by 9%. Nor are the cache misses: a smaller expert cache is faster - on Qwen3-30B, 14 GiB gives 28.0 token/s where 20 GiB gives 27.2, with the hit rate unchanged at 97.7%. Both were announced here as explanations before being measured, and both were wrong.
- The bit-plane format costs 22%.
q6kp3holds the same bits as plain Q6_K, reordered into three planes so that a machine short of memory can read a prefix of them. At the real expert sizes that reordering costs 57.7 GMAC/s against 73.8. Every model inMODELS.mdis kept in plane form, so on a machine with memory to spare that is 22% paid for an option not taken. - The engine is unchanged by any of this. What changed is what we know about it, and what we will say about it.
[2026-09-22] - Janas-LLM: dense models, and where the speed comes from
- Dense models run. A dense feed-forward is a mixture with one expert and no router: with a single expert the softmax of any one logit is exactly 1, so the routed path serves it as it is.
gguf2jns.pywrites the layer's feed-forward as its only slot when the source has no expert count, and the engine allocates a zero router for it - 5 KB a layer, 0.05 ms of the 42 a token costs. Nothing in the forward pass had to learn a second shape. - The automatic expert cache was one slot short of the layers, and on a dense model that is every token. The cache's slots are all as big as the largest, but the size it was allowed to take was the expert region, which is smaller wherever the layers differ - Qwen3-4B keeps
downin Q6_K in half its layers and Q4_K in the other half, so 1.63 GB of region bought 33 slots for 36 layers. A mixture survives that; a dense model asks for every layer at every token, in order, which is the one pattern LRU cannot serve: 0% hits, 3.2 token/s. The cache is now allowed the room its slots actually take: 36 slots, 100% hits, 26.1 token/s. Mixtures gain from it too - Qwen3-30B can hold all 6144 experts where it used to hold 93.5% of them. - The reasoning markers are no longer printed. The opening
<think>is written into the prompt and never generated by the model, so letting the closing one through put half a pair on the screen; the colour of the reasoning already says where it begins and ends. - The dense Qwen3 ties its output head to the embedding table, so
output.weightis no longer required: where it is absent the table is the head. - Qwen3-4B against llama.cpp on the same weights, machine idle: 24.9 token/s decoding against 17.9, 77.3 prefilling against 66.2. Both read the same 2.49 GB a token; Janas moves them at 62 GB/s, llama.cpp at 44.6 GB/s. (The first version of this line put those 62 GB/s at 74% of an 83.9 GB/s ceiling. That ceiling was wrong - see the entry below.)
- That margin is not the streaming. On a 4B nothing is read from disk - the expert cache never misses - and the ratio is the same 1.39x it has always been. On Qwen3-Next-80B, where 45 GiB have to pass through 30 GiB of memory, it becomes 2.38x (24.5 against 10.3), and 3.19x with the multi-token block (32.8). The engine's own kernels win the first factor; streaming wins the second.
- Correctness of the dense path against llama.cpp: 397/400 on C source, 379/400 on Italian, and blocks of 64 bit-identical to decoding one token at a time.
- A 4B dense model decodes slower than a 30B mixture (24.9 against 29.7), because dense means every parameter is active: 2.49 GB a token against 1.82. The project's premise, measured rather than argued.
[2026-09-21] - janas-chat: /about, and pages that can be read
/aboutsays what this is and what it is running on: the version, the author, the licence and the repository beside the mascot, then the model, the machine and the engine as they are at that moment. Nothing in it is new - it was all there, in the line at the start, in the terminal's own idea of the processor, in what the engine was told to do - but it was to be gathered by hand from three places, and a report of a problem needs exactly those three./helpand/aboutare pages now rather than a screenful printed once. The text is laid out to the window however narrow it is, the arrows and Page Up and Page Down move in it, and ESC closes it - no other key does, since a page that shuts at the first key touched shuts while you are looking for a way down it. Before this, in a window narrow enough to want a big font, the help was a text the terminal broke wherever it liked, with no way to reach the end of it.- The help itself is a command to a line, in the colour of the program, with what it does under it and a blank line between one and the next. Two columns were a column of wrapped fragments as soon as the window narrowed; this reads the same at any width, and
/aboutis built the same way, a section name and its text. - The model was named in three places at once - the banner, the bar and
/about- so the banner no longer names it. The bar says it and goes on saying it while the banner scrolls away, and/abouthas the whole of it. - The default system message says who wrote the engine: asked what it is, the assistant answers Janas, the model it thinks with, and Janas-LLM written in C by Maurizio "camauri" Cammalleri.
- A space that a line break carried to the head of the next row is dropped now, and one that hung over the end of a row is no longer part of it - it was being copied out with the text. The nothing after the last newline of a page is no longer a blank row of its own.
[2026-09-21] - janas-chat: a narrow window keeps what matters
- A big font is a narrow window, and the chat was written for a wide one. The line at the foot was built whole and cut at the edge, so the first thing lost was whatever came last in it - the speed, which is the one number a reader watches while a reply arrives. It is made of parts now, each saying how long it is to be kept: the cache goes first, then the bits and the experts, the name of the model, the context, the state, the average, and the speed last of all, still there in twenty columns. A part is in or out, never half printed, since a number cut in two is worse than none.
- The speed was also written twice while a reply came - once in the state, once in its own field, both out of the same statistics - and the average had no unit beside it. Fifteen columns back, and a number that says what it is.
- Beside the mascot in the banner there are nineteen columns less than the window has, so in a narrow one the name of the model and the line about the commands ran off the edge and came back at the first column, under the drawing. They are laid out now, and what does not fit goes on under the line above, from the column where
Janas-LLMbegins; the banner grows past the mascot when the words need it, and below twelve columns of room the mascot goes altogether, a drawing that costs the words their room not being worth it. The layout is the rule the conversation already uses, not a second one written for the banner. - Both of them used to count bytes where they should count columns, so a name with an accent in it was cut short and then padded past the edge.
[2026-09-21] - build: ./build.sh --help, and a clean that stays in its variant
- The script had no help at all, and the part nobody can guess is the variants: a name like
ubsansays nothing about signed overflow or a shift as wide as the type, and a sanitizer that is built but never run reports nothing whatever, since it speaks at run time../build.sh --helpnow says for each one what it catches, what it costs and when it is the one to reach for, with the actions, where the objects and the programs land, and the three environment variables (CC,AR,JANAS_CFLAGS). cleanon its own leaves a tree that does not build, which is seldom what the hand that typed it wanted.clean-buildandclean-testpull down and put back in one go - the first for a changed compiler or a doubt about a stale object, the second as the honest answer before a commit.cleanstays, because freeing the disk is a reason of its own.- And
cleannow removes what its variant made and nothing else. Release has no suffix to match, so it had been sweeping away the programs ofasan,ubsanandtsanas well, leaving their objects behind: the next sanitizer run found nothing to run and rebuilt in silence. An unknown action is an error now rather than a quietbuild, and a variant typed where the action goes is told where it belongs.
[2026-09-21] - llm: the model line keeps what it says about the experts
- The line that describes an open model was built in ninety-six bytes and kept in forty-eight. With the down matrix at three bits and the number of experts chosen by the tuner the sentence is sixty-two characters, so what was cut was the end: how many experts a token is given, and that it was the engine that decided, not the reader. Both buffers now fit what can be written into them;
janas_llm_describestill cuts to the buffer the caller hands it, so nothing changes for a program using the library.
[2026-09-21] - janas-chat: settings last for the conversation, unless they are saved
- What a command changes has always lasted only as long as the chat is open, which is right - a knob turned to try something should not quietly become the way things are - but there was then no way to keep a setting one had settled on.
/save-configwrites them to~/.config/janas/chat.confand/load-configreads them back, losing whatever is set at the time - which it asks about first, since what it throws away was typed on purpose - and/del-configthrows the file away, so that the next start uses the chat's own settings again. That one asks first too. - The file holds the words of the command line, one option to a line and the rest of the line its value, so one reader takes both and a setting cannot mean one thing typed and another saved. It is read before the command line, which therefore wins, and nothing is written to it unless it is asked for. The system message is not kept, and the context, the cache, the bits, the attention and the MTP block are read at the next start, since they are settled when the model is opened -
/load-configsays so when it has read one of those.
[2026-09-21] - janas-chat: the GPU can be told no, and the line says which it is
--no-gpuat the start and/gpu [on|off]during a conversation say whether the GPU may be given work at all, without going through--mode eco, which decides other things besides. The engine gainedjanas_llm_set_gpufor it, and the FreeBASIC binding has it too.- The line at the start is built when it is asked for rather than when the model was opened, so it tells the truth as the engine changes its mind: whether the GPU is given the long passes is something the tuner decides pass by pass, and it was being reported once, at the only moment nothing had been decided yet.
[2026-09-21] - janas-chat: the line at the start says what the GPU does, not that it is there
- "GPU available" told the reader nothing they could act on: the engine opens the GPU unless it is in eco, and the tuner then decides pass by pass whether to give it work - on this laptop it takes the block passes of one model and none of another. The line now says which it is - "GPU present (active, on blocks)", "GPU present (not used)", "GPU not present" - and says it in all three cases, silence being the one answer that left the reader guessing. When it is given work it names which passes it is given, since that is the part worth knowing;
/modehas the same thing class by class, with the times behind the choice./modehas the whole of it, class by class, and is live as the tuner changes its mind.
[2026-09-21] - janas-chat: the reply is laid out, not left to the terminal
- The chat printed the reply and let the terminal break the line at its edge, which it does wherever the edge falls: "per" came out as "p" with "er" on the row below. It lays the text out itself now, breaking after the last space that fits, so a word is never cut - unless it is longer than the row and has nowhere to break, where it is cut as it must be. A space at the end of a row is allowed to hang over the edge, or a word that fits exactly would be pushed down by the space in front of it.
- A window that is not full fills from the top, as a terminal does: drawing it instead of printing into it had put the banner of a conversation just begun at the foot of the screen, with the empty room above it.
- One rule does it, and it is the one that was already there for reading the conversation back and for re-wrapping it when the window narrows: what streams and what is read again are laid out by the same code, so they cannot disagree. The window is drawn rather than printed into, which costs a few kilobytes a token and buys a line that breaks where a line should.
[2026-09-21] - janas-chat: the assistant is called Janas, and says which model it thinks with
- The default system message told the model it was the model - "You are Qwen3 Next 80B A3B Instruct, running through Janas" - so asked who it was it gave the file's name and nothing else. The assistant has a name of its own now, Janas, and the model is what it thinks with: asked who or what it is, it is told to give both. Left to itself a model answers out of its training, and they very often claim to be somebody else's assistant, which is why the message exists at all.
[2026-09-21] - janas-chat: the bar keeps its own colour, and says the average
- The bar at the foot of the window came out grey while the model was writing its reasoning and white the rest of the time, because saving the cursor saves the colours with it and the bar was drawing its own dimming on top of whatever was underneath. It sets the colours it wants from nothing now, and so do the rules around the box.
token/sistok/sthere and in the counters, which is whatjanas-benchhas always written, and the bar says the average of the whole conversation beside the speed of the last reply: one long answer and one short one are two different numbers, and the second on its own says little about how the machine is doing.- A mark that closed at the end of a line - and
**at the end of a line is where a model puts most of them - was printed instead of read: the newline was looked at before the two stars being held for it, and it flushed them as text. What is held is decided first now, the newline included.
[2026-09-21] - build: harder compiler settings buy nothing here, measured
build.shtakesJANAS_CFLAGSfor trying a compiler out - nothing is added when it is unset, so the released build is the one it always was - and uses whateverARnames, since link-time optimization needs an archiver that reads its own objects. It was written to answer a question with a number rather than an opinion: is-O2leaving anything on the table?- It is not. Four builds -
-O2,-O3,-O2 -flto,-O3 -flto- on the naked Q4_K product over a matrix past the caches: 69.4, 69.2, 68.9, 68.9 GB/s. On the engine itself with Qwen3-Next-80B-A3B: 24.10, 24.14, 24.00, 24.16 token/s. Nothing outside the noise, and it is not a surprise: what the time goes into is hand-written AVX2 and AVX-VNNI, which a compiler translates almost one for one, and the rest waits for memory. The settings would have to make the machine read fewer bytes, and no setting can. - Trying them did find something, though: with the whole program in front of it the compiler could no longer see that two variables in the chat are only read where the call that fills them has succeeded. They are initialized now - it costs nothing and the reasoning is written beside them.
[2026-09-21] - janas-chat: a message of several lines is written as one
- A backslash at the end of a line and Enter used to send that line and wait for the next, so a message of three lines was three trips through the editor, each with its own rule drawn under it - and the backslash and the newline it stood for both went to the model, which had to make sense of them. Now the backslash asks for a line and is gone once it has been given, the way it would be in an editor: the message stays in the box, the box grows a row, and Enter on a line that does not end in one sends the whole thing. What the model reads is the message, with its newlines and without a backslash in sight.
- The box and the line repeated in the conversation break a row wherever a newline is, and the cursor is followed by its byte rather than by its column, because a newline takes no columns and two places either side of one would otherwise be the same place.
- Up and down move through the message while there is one to move through, keeping the column as an editor keeps it, and bring back earlier prompts once the cursor is off its first or last row. They follow the rows on the screen, so they walk a long line that has wrapped as well as one the writer broke.
[2026-09-21] - janas-chat: the marks a model writes are read, not printed
- Models mark their replies up and the marks were printed as they came:
**around a word, three backticks around a block, hashes before a heading. Printed they are noise; dropped, the reply loses what they meant - a block of code is worth telling from prose. They are read now and turned into what the terminal can do: bold is bold, emphasis is emphasis, code takes a colour of its own, a dash or a star at the start of a line becomes a bullet, and the language written after a fence is not printed. - The text arrives a piece at a time and a mark can be cut in half by the end of a piece, so the few bytes that might yet be a mark are held back until what follows says what they are; nothing is lost, the end of the reply flushes them. Checked on the same text fed one byte at a time, four at a time and seven: the same output all three ways.
- Underscores are left alone on purpose: in a conversation about code
snake_caseis everywhere, and every renderer that treats_as emphasis turns half of it into slanted nonsense. A star opens emphasis only when something follows it and closes it only when something came before, and never inside a word, so2*3stays a product.--no-markdownand/markdown offprint the marks as they come; a pipe always gets them, because a script may want them.
[2026-09-21] - janas-chat: the box grows with the message
- A message longer than the window scrolled sideways inside a box one row high, so what had been written was out of sight while it was being written. The box now takes the rows the message needs - up to a third of the window, so that a long question never leaves the conversation without room - and goes back to one when the message has been sent. Past the cap it shows the rows the cursor is on, which are the ones being written.
- Growing it moves the scrolling region, and the place the conversation writes from can end up under it, which is how a footer of this kind gets written over. Each time the region moves, the window above is drawn again at its new size and that place is put back where the window now ends.
[2026-09-21] - janas-chat: the conversation can be read again
- A terminal that pins a footer does it with a scrolling region, and the lines that leave the top of a region never reach the terminal's own history: they are gone, and Page Up found nothing to show. The chat keeps them itself now - as the lines it wrote, escape sequences and all, not broken into screen rows - and draws a window on what it kept. Page Up and Page Down move it, Ctrl-Home and Ctrl-End go to the beginning and the end. The line being written stays where it is while you read, and sending brings the end back, because asking is wanting to see the answer.
- Lines are wrapped when they are drawn, not when they are written, so making the window narrower re-wraps the conversation instead of ruining it, and a row that starts in the middle of a coloured line is drawn with the colour it inherits. A reply that arrives while you are reading further up waits below instead of throwing you to the bottom.
- The keys a terminal sends were read three bytes at a time, which was enough for the arrows and Home but not for Ctrl-Home, which sends five: the rest of the sequence used to be typed into the line. The chat reads the whole of a sequence now, whatever its length.
[2026-09-21] - janas-chat: a column is not a character
- The chat measured its lines by counting the lead bytes of the UTF-8, which counts characters, not the room they take: a tick or an ideogram fills two columns and a combining accent none. A model puts ticks and arrows in its replies all the time, so the line the user wrote came back wrapped a column short of where it should, and on Chinese or Japanese a line of eighteen characters was drawn twice as wide as the window. Widths now come from
wcwidth(), and the chat asks the system for the user's own locale, without which that function knows nothing past ASCII. Wherewchar_tis not Unicode the old count is kept: right for everything but the wide and the combining. - Checked on ticks, ideograms, combining accents and the box-drawing characters the chat itself uses, and on a window of twenty columns, where a run of ideograms now stops at nine to the line.
[2026-09-21] - janas-chat: a blank line between one block and the next
- The line the user wrote, the reply, the counters, what a command answers: each ran straight into the one after it, and the counters in particular read as though they belonged to the last paragraph of the reply. Every block now asks for the space it needs before printing (
term_gap), so there is exactly one blank line between any two and none at the top. One place decides, which is how the two the editor and the counters were each printing became one. - A reply that ends on a newline of its own no longer gets a second one, and none of this happens on a pipe: a script reads the lines, and the spacing is not for it to guess at.
[2026-09-21] - Janas-LLM: turning the drafts off and on again brings them back
/spec offdid not turn the drafts off: it turned the model's multi-token prediction block off. The session read "this conversation uses the block" and "I want drafts now" as one flag, so switching the drafts off stopped the block from following the tokens, and switching them on again found a block whose cache had holes and a conversation past its beginning - the one case the code refuses, for a good reason - so nothing drafted again until the conversation was reset. Seen on the development laptop: 25.3 token/s before the switch, 20.6 after, and not one draft in between. The two are now two things. The block keeps noting the tokens whether drafts are wanted or not, which is one pass of one layer and no vocabulary head, and the drafts come and go as asked.- The planner that decides how many tokens to draft then refused to draft anyway, about two times in three. It weighs the cost of a plain pass against the cost of a pass over a block of drafts, and it had timed the blocks before the pause and the plain pass after it - but a conversation is not where it was half an hour earlier, because the expert cache warms as it goes: on this laptop a plain pass fell from 73.7 to 43.1 ms over five turns. It now forgets those timings when the drafts come back and measures them again. Three runs of the same sequence, all three the same: the first reply after
/spec onruns at 32 token/s with every draft accepted, against 16.7 with the drafts off. Read that 32 for what it is - the best turn there is, the one whose expert cache two turns on the same subject have just warmed. Over the two replies that follow the switch it averages 27, and a conversation that moves from one subject to another gives 25 to 27: that is the figure to expect, and it is the same with the sampling greedy or at a temperature of 0.7, which costs 9% on one reply and gives 12% back on the next. - Both are the shape of fault this file had twice already: a decision taken from a measurement that has aged, which then stops the measurement being taken again. The drafts' own switch-off was the first (fixed earlier today), and it is worth saying plainly that none of the three was ever caught by a test, because none of them makes anything wrong - only slow.
[2026-09-21] - Janas-LLM: the drafts are not switched off on one unlucky pass
- The engine drafts tokens ahead and keeps the drafts the model itself would have written, and it stops drafting where it judges the drafts are not paying for themselves. That judgement was made on one measurement, and on a comparison of unlike things: the cost of a plain pass is learned only on the passes that make no draft, which are the first of a reply, where the context is shortest and the expert cache warmest, while the cost of a speculative pass is learned all the way through a reply. Read as it was, the comparison leaned towards switching off - and then switched off for 32 passes, then 64, then 128, up to 512, longer than most replies, with the way back needing the single speculative pass at the end of that count to go well. Seen on 21 Sep 2026 on the development laptop: after one such judgement a reply of 1301 tokens came out at 19.0 token/s where the drafts were giving 25.6, and no later reply in that session ever drafted again.
- Now the drafts must lose by more than 5%, must lose three judgements running, and must be judged against a plain cost measured within the last sixty-four passes - without one that fresh there is nothing honest to compare them with, and they stay on. The longest they go off for is 128 passes instead of 512. Every one of those conditions makes switching off rarer than it was, so the worst this can do is what the engine did before the judgement ever ran, which is never switch off at all.
- The counter that would have shown this - passes run without drafting although drafts were asked for - existed inside the engine and reached nobody. It is now the last field of
janas_llm_chat_stats(the struct carries its own size, so callers built against the older header are unaffected), it is in the FreeBASIC binding, andjanas-chat --statsprints it: "drafts off N passes (the engine judged)". A reply that is slower than it should be now says who decided.
[2026-09-21] - tests: a test that deadlocks fails instead of waiting
- A test that hangs is worse than one that fails: the suite stops and says nothing. A deadlock in the thread pool, introduced and caught the same afternoon, held it for twenty-two minutes before anyone looked. Every test now sets an alarm of ten minutes, so it dies, the runner counts a failure and the suite goes on.
[2026-09-21] - Janas-LLM: a run wakes only the threads it has work for
- The thread pool can be told to use only the first n of its threads, which is how the engine tries one configuration against another. Every run still broadcast on the single condition variable they all waited on, so the threads with no work woke, queued for the mutex, found the run was not for them and went back to sleep - some three hundred times a token. A pool of twenty running twelve cost 6.72 us a run where a pool of twelve cost 1.82: the engine paid 17% for threads it had decided not to use. Those left out now wait apart, on a second condition variable broadcast only when the count grows again, and the cost is gone: 1.82 against 1.81.
- On the development laptop with the twenty threads the engine uses, Qwen3-Next-80B-A3B decodes at 23.77 token/s against 22.72, and steadily: three paired runs spread 0.4 token/s where before they spread 3. Over a longer reply, 512 tokens, 24.05 against 20.94. With twelve threads, where no thread is ever left out, nothing changes either way.
[2026-09-21] - Janas-LLM: the engine learns again
- The engine tries every configuration it can use on the passes it runs anyway and keeps the fastest, and it does not learn while the expert cache is still filling in the background, because a pass that waits for the disk says nothing about the configuration. It asked "is it filling?" by comparing the experts the filling had loaded with the number it meant to load - and that comparison is never equal: the filling only reads while nobody is asking, and it stops as soon as the cache is full, so the experts the caller fetched for itself fill slots the filling then never counts. Measured on 21 Sep 2026: in a session that generates without pause the filling loaded nothing at all, 0 experts of 5596, and the engine believed it was still settling from the first token to the last. So it never learned, in any session; and
janas-bench, whose whole purpose is to leave the machine tuned, threw away every measurement it took - two full runs left the saved profile identical to the digit. - The question is now whether the filling has read anything in the last quarter of a second, which is what it was always meant to ask. The saved profile becomes truthful: for Qwen3-Next-80B-A3B it now holds 46.4, 41.2 and 40.5 ms a token for the three thread counts, against a true 41, where before it held 54.3 and 65.9. Profiles are keyed v2 instead of v1, since what earlier versions wrote there was measured under the old rule and cannot be compared with what is written now.
[2026-09-21] - Janas-LLM: the activation is quantized eight lanes at a time
- Before every product the engine turns the vector it multiplies into Q8_K, and that one step had stayed scalar: three passes over each block of 256 values, one
lrintfat a time. It runs on the thread that drives the phase while the other eleven wait for it, so on an activation of 4096 values it cost 6.83 us, and a layer has several. Eight lanes at a time it costs 1.00 us. The answer is the same to the bit - the largest absolute value is a maximum, which no order changes, the scale and its reciprocal stay the two scalar operations they were, and the vector conversion rounds to nearest with ties to even, which is whatlrintfandnearbyintfdo - so every logit that follows is unchanged.test_quantchecks it on zeros, on values that land exactly halfway, on tiny and on enormous ones, for both the plain and the deterministic quantizer, and fails if one byte differs. - On the development laptop, paired runs, machine idle: Qwen3-30B-A3B decode 27.70 to 28.53 token/s, Qwen3.6-35B-A3B 25.55 to 26.08, Qwen3-Next-80B-A3B 23.60 to 24.41. In the phases where it was worst, the projection after attention goes from 4.28 to 3.76 ms a token, which is 89% of what its own kernel does against 79% before.
bench_longtakes the number of compute threads as an optional last argument. Without it, the twelve threads on CPUs 0-11 every measure of this file has used so far; with it, the CPUs the engine itself picks. It was written to find out whether the benchmarks had been measuring a configuration the product does not use: they had not, twelve is also what the engine chooses, and it is the fastest of the three tried (12, 16 and 20 give 23.72, 23.64 and 22.52 token/s).
[2026-09-21] - Janas-LLM: the experts' reads start without waking anybody
- Reading an expert the cache does not hold was handed to a coordinator thread, which woke the I/O pool: the hand-over alone measured 271 us a time, 2.43 ms of a 46 ms token, because a thread woken by a busy one lands on its CPU and takes its turn. The engine now queues those reads itself, with io_uring - its system calls, not liburing: for reads the whole of it is three calls and two shared rings - so the kernel performs them while the engine computes on the experts it already has, and nobody is woken at all. Where there is no io_uring, an old kernel or a sandbox that forbids it, the old way is still there and still whole.
- Neither way is better than the other, and the engine picks one for each batch of reads. The ring costs nothing to start, but the kernel makes its readers one at a time, as each blocks, so a short queue is read almost one read at a time; the pool has its eight threads already there, and costs the wake-up. So: the ring for a batch of at most four reads, which is a reply being written one token at a time, where that wake-up is the whole cost; the pool for what is in between; and the ring again past four times the I/O threads, a queue deep enough that the kernel's readers are by then more numerous than the pool's and free to run on any core.
- Qwen3-Next-80B-A3B on the development laptop, three paired runs each, machine idle. With an expert cache of 20 GiB, which holds three quarters of what a prompt touches: decode 23.42 to 23.76 token/s, and the prefill unchanged, 32.73 to 32.90. With 8 GiB, the machine short of memory this engine is for: decode 19.66 to 20.38, prefill 26.07 to 27.73. The expert phase of a decode step gives up 0.61 ms of 46 in the first case and 1.37 ms of 54 in the second, and the time waited for the disk falls by a sixth to a half.
[2026-09-20] - Janas-LLM: the Q6_K kernels build their scales once
- The Q6_K kernels, plain and in bit planes, rebuilt the scale vector of every eighth of a block with two broadcasts and an insert - a third of their instructions went there, and at the low bit levels, where there is least else to do, it showed most. They now take the block's sixteen scales once, spread both halves over both lanes, and pick each pair with one shuffle. The weights, the sums and the results are what they were, to the last bit: against llama.cpp the logits agree exactly as before (393 of 400, mean 0.1714), and on plain AVX2 too.
- On a 122880 x 2048 matrix with twelve threads: the two-bit plane product 1.43x quicker, four-bit 1.12x, six-bit and plain Q6_K 1.06x. In the engine, where the experts' down matrix and the output head are the ones that use them, a decode step goes from 22.2 to 22.5 token/s with the experts at four bits and from 22.1 to 22.7 at two - the level a machine short of memory falls back to, which is the one this engine is for.
[2026-09-20] - Janas-LLM: one undefined shift out of the reference
- The scalar reference of the bit-plane dot product shifted a signed sum left, which C leaves undefined when that sum is negative, as it often is. It multiplies by the power of two instead - the same numbers, the same instruction once compiled, and
./build.sh ubsan testnow finishes clean. The vector kernels never had the problem:vpslldshifts a bit pattern and says so.
[2026-09-20] - Janas-LLM: a Q8_0 token embedding is read as Q8_0
- A model whose
token_embd.weightis Q8_0 - unsloth's "dynamic" quantizations among them - loaded, passed its checksums and then answered nonsense: the embedding was unpacked as Q4_K whatever its type really was, which made the numbers hundreds of times too large and the first layer produce NaN. Every row the engine has to unpack whole now goes through one place,janas_dequantize, which reaches the right unpacker for each of the four types a model file may carry and refuses anything else instead of guessing. Found and diagnosed on an Intel i7-1365U, with the two-line remedy already worked out (issue #1). test_quantnow checks that unpacking a row of every supported type gives what that type's own unpacker gives, and that a row read at its own offset in a table gives the same numbers: put the old behaviour back and both checks fail.
[2026-09-20] - Janas-LLM: attention scores as exact integer sums, when the context is long
- The attention scores can now be computed two ways, and the engine keeps both. The exact one reads the query as it is and sums in floating point, as before. The fast one reads it as sixteen-bit integers with a scale of its own per token and head: the dot product is then an exact integer sum, which AVX-VNNI does sixteen products at a time (
vpdpwssd) and plain AVX2 in two instructions (vpmaddwd,vpaddd) instead of eight floating-point ones. On the development laptop that is 1.22x off a decode step at 16384 positions and 1.26x off a prefill block, and the scores kernel alone runs at 71 GMAC/s against 39 on one core. - Every way of computing them agrees to the last bit: scalar, AVX2 and AVX-VNNI sum the same integers, and the eight query heads of a group are reduced together by a tree of adds without changing a single result. A block of tokens still gives what the same tokens give one at a time. The test prints a checksum of every output it produced, so the two instruction sets can be compared by running it twice.
- What the fast way costs is the query's quantization, a relative 3e-5 - against the 4e-3 the eight-bit keys and values already carry. Measured on four hundred tokens against llama.cpp: the most likely token agrees 394 times of 400 on C source (393 the exact way) and 377 of 400 on Italian prose (382). The engine therefore asks for it by itself only above 16384 tokens of context, where attention is the greater part of a token and a conversation will reach the far end;
janas-chat --attention exact|fast,janas_llm_params.attn_scoresandJANAS_ATTN=float|int16settle it either way, andjanas_llm_attn_scoressays what is in use.
[2026-09-20] - Janas: three forms, so a report carries what it needs
- Three issue forms in
.github/ISSUE_TEMPLATE. The hardware report asks for the filejanas-benchwrites and then for the four things that file cannot know - distribution and kernel, memory free at the time, the kind of disk the model sits on, which model - because a measurement missing any of them cannot be compared with another machine. The other two are for something that went wrong, which asks for the banner Janas prints before anything else, and for a model that will not convert, which asks for the GGUF's own SHA-256 so that the file in question is beyond doubt. - The README and CONTRIBUTING now link the forms where they ask for a report.
[2026-09-20] - Janas-LLM: the models are a recipe and a fingerprint, not a download
MODELS.mdnow carries the whole chain for every model this project runs: the Hugging Face repository each GGUF came from, its SHA-256 as Hugging Face itself publishes it, the two commands that convert it, and the SHA-256 of both files that come out. Nothing is uploaded, and nothing needs to be:tools/gguf2jns.pyandtools/jns_planesare deterministic — the layout follows from the sizes in the source, nothing is timed, drawn at random or read from the environment, and the weights are copied, never recomputed — so a fingerprint is worth as much as the file itself.- Measured, not assumed. Each of the four models was converted again from its GGUF and compared byte for byte with the file the numbers in this repository were measured on; all four matched. The two steps cost 47 s and 104 s for the 18.6 GB model, 153 s and 194 s for the 48.4 GB one, less than the download either way. The multi-token prediction block, which comes from the original checkpoint instead, gives the same bytes on two runs.
- The README pointed at the wrong repositories: three of the four GGUFs come from Qwen's own, not from ggml-org's or unsloth's. It also left out the bit-planes step, although that is the form the models are kept in here, and understated the room a conversion needs: three copies of the model at the worst moment, about 145 GB for the largest.
[2026-09-20] - Janas-LLM: the FreeBASIC example reads a pipe, and prints what it was given
tools/chat.basnow works in a pipe as well as at the terminal. FreeBASIC'sline inputreads the console, not the standard input: given a pipe it waits for a terminal that will never answer. The example asksisattyand, when the input is redirected, reads it through theCONSdevice; at the terminal nothing changes,line inputkeeps its line editing.- It also prints the reply as the library hands it over.
janas_llm_chat_nextreturns the length of the piece and does not close it with a NUL, so the example leaves room for one and writes it: before, a short piece carried the tail of the one before it, and the reply came out asOkay,kay the user iser asking. The two lines shown in the README had the same fault. - The counters fit their format again: four hundred tokens in a field of two digits printed
%400.
[2026-09-20] - Janas-LLM: long context, and a library for FreeBASIC
- The KV cache is kept in eight bits with a scale of its own per position and head, instead of half precision: half the memory, so twice the context in the same machine, and at a long context - where attention is bound by reading it - half the work. Nothing is lost by it: against llama.cpp on Qwen3-Next the most likely token now agrees in 397 positions of 400 on C source where half precision gave 394, and 386 of 400 on Italian prose where it gave 383, since a scale that follows one head's numbers describes a narrow range better than a fixed exponent. A decode step at 65,536 positions costs 3.6 ms a layer.
- The context stops where the model was trained (262,144 tokens for Qwen3-Next, 40,960 for Qwen3-30B) and is paid for out of the expert cache: asking for 131,072 tokens on the development laptop leaves 15.2 GiB of cache instead of 20.5, and runs.
include/janas/llm.bideclares the library for FreeBASIC, and so for BASIC MODERN, the dialect Prabanta implements;tools/chat.basis a chat in sixty lines that uses it.
[2026-09-20] - Janas-LLM: a chat that does not stop, and a window of its own
- The context slides instead of ending. A conversation that fills it now forgets its oldest exchanges - whole turns, never half a message - and keeps going at the same speed; the system message, or the first few tokens when there is none, stays where it is, since attention leans on those positions. In a hybrid model the recurrent state is left untouched: it has seen everything, and that is what carries the older text once the window has passed it. Twelve turns in a window of 1024 tokens run through without stopping, and the model still answers what the cat in its story was called.
janas-chattakes the terminal: a banner, the conversation scrolling under it, and a footer that stays at the bottom with the model, the bits and experts in use, the context filled, the speed of the last reply and the state of the expert cache. Lines are edited with the arrows, Home, End and the usual control keys, and earlier prompts come back, kept between sessions. Colours followNO_COLORandTERM. Piped in or out, it stays the plain thing it was.- How many bits the experts' down matrix is read with, and how many experts a token uses, now follow the memory the machine has: every bit when the experts all fit in the cache, four otherwise; all the experts unless the cache holds less than a fifth of the model. On a machine with a cache of 6% of the experts, that is 21.9 token/s instead of 15.0.
[2026-09-20] - Janas-LLM: the expert cache is filled before the first reply
- The engine remembers which experts this machine uses with a model (a small file in
~/.cache/janas, added to at every session) and fills the cache with them when the model is opened, instead of letting the first replies find them one token at a time. With Qwen3-Next-80B-A3B on the development laptop, preloading 11,360 experts costs 3.9 seconds once: the second reply of a chat - the one that used to finish filling the cache - goes from 17.9 to 24.5 token/s, its prompt from 23.5 to 46.7 token/s, and it reads 3.0 GB from disk instead of 14.3, waiting 0.35 seconds instead of 3.35. The first short reply goes from 10.3 to 15.3 token/s. janas-chat --no-preload,janas_llm_params.no_preloadandJANAS_WARM=0turn it off; the startup line says how many experts were preloaded and how long it took.
[2026-09-20] - Janas-LLM: experts' down matrix in bit planes
- A model file can now hold the down matrix of every routed expert as three planes of two bits instead of one block of six, which takes exactly the same room: the engine then reads as much of each expert as the machine can afford - all of it (the weights as they were quantized, bit for bit), two planes, or one - and the choice is made when the model is opened, from the same file.
tools/jns_planesrewrites an existing model file into the new form without going back to the original GGUF; files in the old form keep working. - With Qwen3-Next-80B-A3B on the development laptop, four bits per weight instead of six: 21.5 token/s instead of 19.6 with the most likely token agreeing with llama.cpp in 389 positions of 400 against 394, and in a real chat a quarter to a half fewer bytes read from disk (Italian prose 8,969 -> 3,572 MB, C source 19,895 -> 13,386 MB) with the waits for them cut by half. Two bits: 21.9 token/s and 381 of 400.
janas-chat --bits 2|4|6,janas_llm_params.expert_bitsfor the library,JANAS_EXPERT_BITSfor measurements. Six bits, the whole matrix, stays the default.
[2026-09-20] - Janas-LLM: experts per token as a dial
- How many experts a token uses is now a choice, not a constant of the model:
janas_llm_set_experts,janas_llm_experts,/experts ninjanas-chatandJANAS_EXPERTSfor measurements. The experts left out are those the router weights least, so fewer of them means fewer bytes read per token: with Qwen3-Next-80B-A3B, eight of its ten give 21.9 token/s instead of 20.6 on the development laptop, and the most likely token still agrees with llama.cpp in 387 positions of 400 against 394 for all ten. - In a real chat the dial pays more than the bare arithmetic suggests, because a smaller working set fits the expert cache better. Same question, warm cache, temperature zero, Qwen3-Next-80B-A3B on the development laptop: with all ten experts 23.6 token/s out and 30.5 in, 16,615 MB read from disk; with six, 26.4 out (+12%) and 41.9 in (+37%), 9,284 MB (-44%). The hidden cost is the multi-token draft, which predicts the whole model: its drafts are accepted 69.3% of the time instead of 77.0%.
[2026-09-20] - Janas-LLM: Qwen3.5 and Qwen3.6, reasoning models
- Qwen3.5 and Qwen3.6 mixture-of-experts models (
qwen35moe, e.g. Qwen3.6-35B-A3B): the same layers as Qwen3-Next with separate DeltaNet projections, a different key-head sharing (settled against llama.cpp's graph with a new per-layer trace) and the multi-token prediction block inside the model file. Top-1 agreement with llama.cpp 197/200 on C source; the internal MTP accepts 87% of its drafts (32.3 token/s decoding, 76 prefill on the development laptop). - Q5_K and Q8_0 weights, used by recent GGUF files, with the same exact per-block arithmetic as the other formats.
- The qwen35 pre-tokenizer (word runs take combining marks): identical to llama.cpp's tokenizer on every corpus.
- Reasoning models: the reasoning part of a reply is marked (
janas_llm_chat_thinking), can be turned off (thinking,/think) and is printed in grey byjanas-chat. tools/gguf2jns.pyreads models split into several GGUF files, so Qwen3-Coder-Next (80 billion parameters, the same architecture as Qwen3-Next) runs with no change: 194/200 top-1 agreement with llama.cpp.
[2026-09-19] - Janas-LLM: self-tuning engine and janas-bench
- The engine tunes itself on the passes it runs: for one token, 2-16 tokens and blocks it tries the configurations the machine offers (one thread per performance core, all their threads, the efficiency cores too; with and without the GPU), measures them net of disk waits, keeps the fastest and checks the others again now and then. Choices are remembered per machine and model in
~/.cache/janas/tuning.txt. - Power modes:
econever uses the GPU for speed alone,maxtakes the fastest configuration,auto(default) follows the power source;janas_llm_set_mode,janas_llm_tuning,/modeinjanas-chat. janas-bench: measures prefill and decoding speed (also with MTP drafts) for every configuration, fills the tuning profile and writes a report to share. On the development laptop the efficiency cores speed decoding up by 16%.
[2026-09-19] - Janas-LLM: shared CPU/GPU arithmetic, optional GPU, adaptive threads
- Q4_K and Q6_K products compute exact integer sums per block and one fma per block and vector; the scalar reference, the AVX2 and AVX-VNNI kernels and the GPU shader perform the same float operations and agree bit for bit (tests check equality). Faster on blocks (12288x2048, 8 vectors: 0.68 -> 0.62 ms).
- Optional integrated GPU (
src/llm/gpu.c, off by default,JANAS_GPU=1): Vulkan loaded at run time, the shader built into the library, the resident weights shared with no copy, products split by rows with an adaptive share, a keep-alive against the GPU's sleep states. Results identical to the CPU's. On the development laptop it brings no net speed-up (CPU and iGPU share the package's power and memory);tests/bench_powermeasures time and energy per product (the iGPU does 2-2.6x the work per joule). - Adaptive threads: one thread per physical core for passes over blocks, all threads for single tokens; the thread pool runs on a chosen number of threads. Qwen3-Next prefill 56.9 -> 62.5 token/s.
- The chat's automatic expert cache leaves a fifth of the memory free.
[2026-09-19] - Janas-LLM: tokenizer, sessions, C library and chat
- Own byte-level BPE tokenizer (
src/llm/tokenizer.c), built from the model file's metadata: the pre-tokenizer's expression as a hand-written matcher over generated Unicode tables (tools/gen_unicode.py), merges by rank with a heap (n log n on long words). Identical token ids to the reference on every corpus and a stress text, exact round trip;tests/tok_check.c. - Sessions (
janas_llm_session_*): a token sequence that grows by appended turns and by tokens generated on request, each computed once, with the KV cache and the recurrent state carried across turns. Sampling (temperature, top-k, top-p, min-p, seed) draws one random number per token, so the text is the same with and without speculation.tests/llm_session.c. - Public C API
include/janas/llm.handlibjanas_llm.so(only the API is exported), shaped for foreign-function interfaces: opaque handles, fixed-width types, int32 error codes, size-prefixed parameter structs, reply text returned on request as whole UTF-8 characters. Chat format and stop tokens from the model's metadata (ChatML); expert cache size and CPU split chosen from the machine. janas-chat: terminal chat with streaming output, slash commands (/help,/stats,/reset,/system,/temp,/spec,/quit) and a per-reply report of input speed, output speed and total time.- The per-reply report also gives the share of routed experts served from the cache, the bytes read from disk and the time waited for them.
janas-chatstarts with a system message naming the model (from its metadata) and the engine, since models left without one often claim to be another model.
[2026-09-19] - Janas-LLM: half-precision router and recurrent state
- The router weights are converted to half precision in place at load (no extra memory) and multiplied with an F16C kernel: 3.5 -> 2.1 ms per token on Qwen3-Next, 0.8 -> 0.6 on Qwen3-30B-A3B; agreement with llama.cpp unchanged within noise.
- The Gated DeltaNet state is stored in half precision (computed in float, rounded once per stored row,
janas_gdn_step_h): half the memory traffic, 2.6-3.0 -> 2.0-2.1 ms per token. Its effect on the logits is within the model's numerical noise; a more precise int16 format with row scales measured the same and ran slower.
[2026-09-19] - Janas-LLM: MTP drafts for Qwen3-Next
tools/hf2jns_mtp.c: converts the multi-token prediction block, absent from GGUF conversions, from the original safetensors checkpoint into a one-layer JNS file (routed experts as expert slots, the rest resident); Q4_K with Janas's own quantizer (janas_q4k_quantize: per sub-block affine fit searched over shrunken ranges and refit by least squares, block factors refit against the 6-bit scales). The checkpoint's conventions were checked on the weights (zero-centred norms, row order).janas_llm_mtp_forwardand the hidden states of the main pass: the block's input order and hidden-state choice were settled on data (tests/mtp_eval.c): it agrees with the main model's next choice 88.9% of the time.- Generation with
JANAS_SPEC_MTP: after each pass the MTP rows of the accepted tokens keep its KV cache complete, then up to k drafts are chained on its own hidden state. On Qwen3-Next: 1.17x with one draft (89% accepted), 3.12 tokens per pass with three; output identical to plain decoding. build.shalso builds the C tools intools/.- Drafts are chosen on a reduced LM head: the 32,768 lowest token ids plus every token seen in the context (95.5% coverage measured), 21% of the full head's reads (
janas_llm_mtp_draft,janas_llm_mtp_note_tokens). - Draft planner (
spec_autowith MTP): measured pass and chain-step costs and survival odds calibrated on earlier verifications choose, pass by pass, the number of drafts that maximizes expected tokens per second; the chain stops as soon as the last draft would not be verified. Without any tuning it lands near the best fixed threshold of each text: 1.28x on C source, 1.12x on Italian prose, 1.10x on BASIC. A fixed threshold (spec_min_conf) is also available: 1.34x on C source at 0.8.
[2026-09-19] - Janas-LLM: block products and verification passes
- Quantized dot products over 1 to 8 vectors from one kernel body (
src/llm/quant_dotn.h), compiled for AVX2 and for AVX-VNNI (vpdpwssd): the unpacked weights and one accumulator per vector stay in registers. 12288x2048 Q4_K over 16 vectors: 168 -> 229 GMAC/s on AVX2, 297 GMAC/s with VNNI. Every variant gives bit-identical results (integer sums are exact);JANAS_KERNELS=avx2forces the AVX2 path. - Gated DeltaNet rework: the recurrent state is double-buffered as a base and a tip, so a call that continues the previous one needs no replay; the recurrence is split by state rows (128 work items instead of 32 heads); vectorized convolution with the L2 norms, and gated RMS norm with a vectorized SiLU (
src/llm/vmath.h). 3.7 -> 2.6 ms per decoded token, 15.2 -> 6.5 ms per 5-token block. - The router reads each expert's weight row once per block.
- A 5-token verification pass on Qwen3-Next costs 2.47 single passes (was 2.87); lookup speculation now pays: 1.11x with k = 2. Prefill 54 token/s.
- Phase timing gains
ssm(DeltaNet convolution and recurrence).
[2026-09-19] - Janas-LLM: Qwen3-Next
src/llm/model.cruns qwen3next (Qwen3-Next-80B-A3B): Gated DeltaNet layers (causal convolution, L2-normalized keys and queries, gated RMS norm) alternating with gated full attention (partial NeoX RoPE on 64 of 256 dimensions), routed experts plus a sigmoid-gated shared expert. The KV cache is allocated only for the attention layers.src/llm/deltanet.c: the DeltaNet step in a single pass over each state row (decay, key product, update, query product fused);tests/test_deltanet.cchecks it against the recurrence in double precision.- The shared expert runs as one more pair in the same dispatch as the routed experts already in RAM.
- Logits against llama.cpp: 393/400 top-1 agreement on a C source prompt, mean absolute difference 0.133; on 64 Italian tokens 0.094, below llama.cpp's own difference between its two kernel paths (0.101). Blocks are bit-identical to token-by-token decoding.
- 16.1 token/s at steady state with a 20 GiB expert cache.
- Rollback of the recurrent state for speculative decoding, at no extra memory traffic: the state stays at the start of the last call and each call replays the accepted tokens of a per-token input log in place before running its own tokens on a cached copy of each head. Speculative and plain generation produce identical tokens.
src/llm/attention.c: attention by GQA group over fixed chunks of 256 positions, merged with an online softmax, so every key and value row is read once for all the query heads of its group and a long context uses every thread. Kernels on 8, 4, 2 or 1 query vectors with the accumulators in registers, a vectorized exp, dynamic work claiming. Chunks start at absolute positions, keeping blocks bit-identical to token-by-token decoding;tests/test_attention.cchecks both properties,tests/bench_attn.ctimes one layer.- The KV cache is stored in half precision (the representation llama.cpp uses too): attention at 8,000 positions went from 1.13 to 0.375 ms per layer, and the cache takes half the memory.
- The I/O threads sleep between requests instead of spinning (
janas_pool_set_spin), leaving the package power budget to the compute cores. - Qwen3-Next after an 8,000-token prompt within 24 GB (peak resident memory 21.9 GiB, 20 GiB of expert cache): 17.6 token/s decoding, 27.8 token/s prefill. At 400 tokens: 19.2 token/s; Qwen3-30B-A3B: 24.4 token/s.
- Fix: the Q4_K/Q6_K kernels are selected before
main, removing a data race when the first product ran on pool threads.
[2026-09-19] - Janas-LLM: speculative decoding
- Forward pass over blocks of up to 64 tokens (prefill, speculative verification), bit-identical to token-by-token decoding; four-vector Q4_K/Q6_K kernels.
src/llm/generate.c: greedy generation with optional speculative decoding. Optionsspec_mode(off, lookup),spec_k,spec_auto; lookup drafts copy the continuation of the longest recent n-gram match. The output is identical with and without speculation.spec_automeasures tokens per verification pass against the cost of a plain pass and pauses speculation when it does not pay, retrying with exponential back-off.- Measured on Qwen3-30B-A3B (C source prompt, 150 tokens): drafts accepted 65% (k = 4, 3.57 tokens per pass) but a 5-token verification costs 4.1 plain passes, so speculation is 0.87x;
spec_autoholds it at 0.96x. It will pay once block products are compute-efficient (register-tiled GEMM).
[2026-09-19] - Janas-LLM: first complete forward pass
src/llm/model.c: decode for the qwen3moe architecture (Qwen3-30B-A3B): RMS norms, per-head q/k norms, NeoX RoPE, grouped-query attention over a float32 KV cache, softmax router with normalized top-k weights, routed experts streamed through the expert cache with compute overlapping the reads.- Decode at 21.4 token/s on Qwen3-30B-A3B (positions 200-400, 12 compute threads): AVX2 float dot/axpy for attention and router, router spread over the thread pool, RoPE angles computed once per token. Per-phase timing in
janas_llm_model_phases. tests/llm_eval.ccompares the logits with a llama.cpp dump: on 64 positions of Qwen3-30B-A3B the most likely token agrees 59 times, with a mean absolute logit difference of 0.155; llama.cpp against itself with a different kernel path agrees 60 times with 0.140.
[2026-09-19] - Janas-LLM: model file format
- JNS, the Janas-LLM model file (
src/llm/jns.h): each routed expert is one contiguous, 4 KiB-aligned slot (gate, up, down) read with a single O_DIRECT request; an offset table leaves the on-disk order of the experts free; checksums on header, tables and every expert. - Expert cache (
src/llm/expert_cache.c): fixed slots in one aligned arena, LRU replacement that never evicts the experts of the current request, warm start from prompt frequencies, misses read in parallel with O_DIRECT by an I/O thread pool;tests/test_expert_cache.cchecks contents and hit counts against a reference model. - The expert cache reads misses in the background (
janas_expert_cache_begin/finish), so the experts already in RAM are computed while the missing ones load: +12-17% on Qwen3-Next with a 20 GiB cache, +34% with 10 GiB. No prediction involved. tests/bench_stream.c: the Phase 1 test bench. Replays a router trace against a JNS file through the expert cache, computing gate, up and down with the real weights; reports coverage, NVMe bytes, I/O wait and compute time per token.- JNS version 2: a metadata region carries the key/value section of the source GGUF byte for byte (hyperparameters, tokenizer), validated entry by entry and covered by the fuzz test.
tools/gguf2jns.py: pure-Python GGUF to JNS converter, streaming, no dependencies.- Reader with overflow-checked validation of every offset and size;
tests/test_jns.cfuzzes it with byte flips and field mutations;tests/jns_check.cvalidates a file and verifies every expert through O_DIRECT.
[2026-09-19] - Janas-LLM: first kernel
build.sh: release, debug, asan, ubsan and tsan variants;./build.sh <variant> testruns the tests.- Q4_K and Q8_K block formats, byte-compatible with GGUF k-quants; portable scalar dot product and an AVX2 kernel selected at run time.
- Thread pool (
src/common/pool.c) and a multithreaded Q4_K matrix-vector product with dynamic row distribution. tests/test_quant.c(kernels against the scalar reference) andtests/bench_matvec.c(weight bandwidth).- Q6_K format with scalar reference and AVX2 kernel (the down projection of GGUF Q4_K_M experts is often Q6_K); dequantization matches ggml bit for bit on real OLMoE tensors. Grouped products take a quantization type per task.
- Grouped matvec: several independent products (the experts of a MoE layer) in one pool dispatch;
tests/bench_moe_layer.cmeasures the routed-expert part of decode layers. - Q4_K matvec at 63.6 GB/s of weights on 12 threads (Core Ultra 9 185H, 81 GB/s read bandwidth): SIMD unpacking of scales and mins, adaptive row chunks so each thread streams long contiguous runs.
[2026-09-18] - Project start
- Repository created with the base layout:
src/,include/, documentation files.