37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
C avx2ccpu-inferenceggufinference-enginelinuxllmlocal-llmmixture-of-expertsmoenvmequantizationqwen3simdspeculative-decodingvulkan
Release 2026-10-01: code completion in the editor
camauri6476bbc69520
diff --git a/ChangeLog.md b/ChangeLog.mdindex 2b2e45f..20ad08b 100644--- a/ChangeLog.md+++ b/ChangeLog.md@@ -2,6 +2,28 @@> Curated, user-facing summary of completed work, newest first.+## [2026-10-01] - Code completion in the editor++- **`janas-server` fills in code at an editor's cursor**, with a model for+ code such as Qwen3-Coder-30B-A3B-Instruct, through OpenAI's+ `/v1/completions` with a `suffix` and through llama.cpp's **`/infill`**,+ which the extensions made for llama.cpp ask (llama.vscode): the line up+ to the cursor, `n_predict` (0: the prompt only), `n_indent` and+ `t_max_predict_ms` read as llama.cpp reads them, the answer as+ llama.cpp's. Tried in VSCodium with llama.vscode and with Continue;+ how to set them up in [docs/code-completion.md](docs/code-completion.md).+- **The window sent at every keystroke is kept from moving.** The+ extensions send "the last N lines before the cursor", whose start moves+ at every new line, and the whole window was read again: 17 s for 1,330+ tokens at every request. The start of each window is now remembered and+ kept, put ahead when the editor slides its window, the text before the+ cursor cut at a line within `--fim-prefix` tokens (2048), the lines after+ it within `--fim-suffix` (8), the text written within `--fim-max` (256):+ 1.5-2.2 s a proposal after the first request of a window, on a laptop+ with Qwen3-Coder-30B-A3B in memory.+- **Sixteen stop strings** a request, where OpenAI takes four: Continue+ sends more, and every one of its requests was refused.+## [2026-10-01] - janas-flights, the first of Janas's services- **janas-flights**, flights as an MCP server of its own, for `janas-chat`diff --git a/MODELS.md b/MODELS.mdindex a800260..7809901 100644--- a/MODELS.md+++ b/MODELS.md@@ -103,6 +103,14 @@ b46fedd33e0bfb0cae308aa3c158d0a4b2c4a1d2185a1ed6f093cdaf39064772 Qwen_Qwen3.6-30276b0697a05075fec8ab076a757205537804ff41dd7e9dadace7c6993763563 qwen3-coder-next-q4km.jns (jns_planes)```+**Qwen3-Coder-30B-A3B-Instruct**, for code completion, from+[`unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF`](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF):++```+fadc3e5f8d42bf7e894a785b05082e47daee4df26680389817e2093056f088ad Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf+bee7d25509e911a195cbf05a157e69bb05a62d2241cca2c4fdcfef269dcb2bdd qwen3-coder-30b-a3b-q4km.jns (gguf2jns)+```+**Qwen3-4B**, from [`Qwen/Qwen3-4B-GGUF`](https://huggingface.co/Qwen/Qwen3-4B-GGUF).This one is dense: it has no experts to route, so its feed-forward becomes thesingle slot of each layer, and half its layers keep a `down` matrix in Q4_K,diff --git a/README.md b/README.mdindex d5a96e3..32e5118 100644--- a/README.md+++ b/README.md@@ -130,8 +130,9 @@ report from `janas-bench` is the single most useful thing you can send back**that clients written for it can use a model running on this machine:chat and completions with tools, JSON output held to a schema andlog-probabilities, the Responses API with its conversations, embeddings- and moderations, and a chat page of its own to try it from a browser- ([below](#using-it-over-http)).+ and moderations, a chat page of its own to try it from a browser+ ([below](#using-it-over-http)), and code completion for VSCodium and VS+ Code ([docs/code-completion.md](docs/code-completion.md)).- **`janas-bench`:** measures the machine, fills its profile and writes a report.- **Faster replies** from drafts the model verifies, so the text is exactly theone it would have written: from the model's own multi-token prediction block@@ -211,6 +212,7 @@ the machine to itself.- [MODELS.md](MODELS.md) — what a converted model's licence is, and what may be redistributed- [CONTRIBUTING.md](CONTRIBUTING.md) — how to help, and the sign-off- [docs/services](docs/services/README.md) — Janas's services (flights), in `janas-chat` and in other MCP clients+- [docs/code-completion.md](docs/code-completion.md) — code completion in VSCodium and VS Code, through `janas-server`- [AUTHORS](AUTHORS) — who wrote it, and what it owes to others- [LICENSE](LICENSE) — GNU GPL, version 3 or later@@ -257,6 +259,7 @@ that is where these examples put them, and where the project keeps its own.| **Qwen3-Next-80B-A3B-Instruct** — most of this engine was measured on it | [Qwen](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct-GGUF) | 48.4 GB | 48.4 GB | 32 GB || **Qwen3.6-35B-A3B** — the easiest to start with | [bartowski](https://huggingface.co/bartowski/Qwen_Qwen3.6-35B-A3B-GGUF) | 22.3 GB | 22.3 GB | 16-32 GB || **Qwen3-Coder-Next** | [Qwen](https://huggingface.co/Qwen/Qwen3-Coder-Next-GGUF) | 48.4 GB | 48.4 GB | 32 GB |+| **Qwen3-Coder-30B-A3B-Instruct** — for code completion in the editor ([docs/code-completion.md](docs/code-completion.md)) | [unsloth](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF) | 18.6 GB | 18.6 GB | 32 GB, to keep it whole in memory || **Qwen3-30B-A3B** | [Qwen](https://huggingface.co/Qwen/Qwen3-30B-A3B-GGUF) | 18.6 GB | 18.6 GB | 16 GB || **Qwen3.5-9B** — dense; its prediction block from the checkpoint ([MODELS.md](MODELS.md#the-fingerprints)) | [unsloth](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF) | 5.7 GB | 5.7 GB | 16 GB || **Qwen3.5-2B** — dense, small and quick; a prediction block too | [unsloth](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF) | 1.3 GB | 1.3 GB | 8 GB |@@ -669,6 +672,17 @@ estimate has not yet been checked on a run that long.`janas-server` takes `--metrics`, `--progress`, `--max-input` and`--warn-input`.+## Filling in code++`janas-server` fills in code at an editor's cursor, with a model for code+such as Qwen3-Coder-30B-A3B: OpenAI's `/v1/completions` with a `suffix`,+and llama.cpp's `/infill`, which llama.vscode asks. The window sent at+every keystroke is kept from moving, so that a keystroke reads only what+it changed: 1.5-2.2 s a proposal on a laptop, where a window slid a line+was read again in 17 s. How to set it up in VSCodium and VS Code, with+llama.vscode or Continue, and what was measured:+[docs/code-completion.md](docs/code-completion.md).+## Using it from a programThe library is shaped for foreign-function interfaces from the start: opaque@@ -835,6 +849,7 @@ model running on this machine. It takes the model options `janas-chat` takes| `--test` | serve a chat page for trying it out, on `/test` (below) || `--max-input N` | the longest input a request may have, in tokens, below the context: a longer one gets a 400 (`input_limit_exceeded`) whose error says `input_tokens`, `max_input_tokens`, `excess_tokens` and the parts, and nothing is ever cut to fit || `--warn-input N` | longer inputs are taken, and said on stderr with their parts |+| `--fim-prefix N`, `--fim-suffix N`, `--fim-max N` | completions with a `suffix`, an editor filling in code at its cursor: the tokens before the cursor read at most (2048; 0: all) and the lines after it (8; 0: all); `--fim-max N` the tokens written there at most, whatever the request asks (256; 0: as asked), since an editor asks for thousands and the requests of the next keystrokes wait for them. Over the budget the text before the cursor is cut at a line to three quarters of it, and the start of the window is kept across keystrokes, also when the editor slides its window, so that a keystroke reads only what it changed (see [Filling in code](#filling-in-code)) || `--metrics FILE`, `--progress S` | JSON lines of what every reply costs, and progress lines while a long prompt is read (see [Watching a long prompt](#watching-a-long-prompt)); `--verbose` prints progress every 30 s || `--no-mcp` | refuse tools of type `mcp` in `/v1/responses`: no connection leaves the server on a request's word (below) || `--keep N` | conversations kept computed besides the one in use (8; 0: none), so that clients taking turns, or a client's requests on the side for titles and tags, do not have their conversations read again from the start |@@ -855,6 +870,7 @@ What it answers:| `POST /v1/chat/completions` | a conversation, the reply whole or streamed (`"stream": true`, server-sent events), with tools, JSON output, log-probabilities and more than one reply (below) || `GET`, `POST`, `DELETE /v1/chat/completions/{id}`, `GET /v1/chat/completions`, `.../messages` | the completions a client asked to keep (`"store": true`): read, listed with their `metadata`, changed, deleted || `POST /v1/completions` | raw text to continue, no chat format around it; several prompts, token ids, `echo`, `logprobs`, `best_of`, `suffix` |+| `POST /infill` | llama.cpp's route for filling in code, which editors' extensions made for llama.cpp ask (llama.vscode): `input_prefix`, `prompt`, `input_suffix`, `n_predict` (0: the prompt only, read for the next request), `n_indent`, `t_max_predict_ms`, the answer as llama.cpp's (`content`, `timings`); `input_extra` is not used, and the most likely token is always taken || `POST /v1/responses` and the six operations under it | OpenAI's newer API: items in and out, `previous_response_id`, conversations, streamed events, `background` and `cancel`, `input_items`, `input_tokens`, `compact` || `/v1/conversations` and its seven operations | conversations the server keeps, and their items || `POST /v1/embeddings` | with `--embedding-model FILE`: the vectors of texts (below) |diff --git a/docs/code-completion.md b/docs/code-completion.mdnew file mode 100644index 0000000..af31ed3--- /dev/null+++ b/docs/code-completion.md@@ -0,0 +1,178 @@+# Code completion in the editor++`janas-server` fills in code at the cursor of an editor, as the+completion extensions of VS Code and VSCodium ask for it: the code before+the cursor and the code after it go to the model, which writes what goes+between. It runs on the processor, with a model for code; nothing leaves+the machine.++## The model++A model trained to fill in the middle is needed, with the+`<|fim_prefix|>`, `<|fim_suffix|>` and `<|fim_middle|>` tokens of the Qwen+coders. Every Qwen3-Coder is, says its+[README](https://github.com/QwenLM/Qwen3-Coder). The one tried is+**Qwen3-Coder-30B-A3B-Instruct**, Q4_K_M (unsloth's GGUF, converted with+`gguf2jns`): 18.6 GB, a mixture of experts with 3 billion parameters+active a token, which on a 32 GB machine fits in memory whole+(`--cache 18`).++Qwen3-Coder-Next fills in too, but its recurrent layers cannot go back to+a point of a sequence: every keystroke would read the window again from+its start.++## Starting the server++```sh+janas-server models/jns/qwen3-coder-30b-a3b-q4km.jns --cache 18+```++It listens on `http://127.0.0.1:8080`. `--verbose` writes on stderr, for+each request, where its window was cut and why.++## The two routes++- **`POST /v1/completions` with a `suffix`**, OpenAI's: the `prompt` is+ the code before the cursor, the `suffix` the code after it.+- **`POST /infill`**, llama.cpp's, which the extensions made for+ llama.cpp's server ask: `input_prefix` (the lines before the cursor),+ `prompt` (the line up to it), `input_suffix` (what follows), and the+ answer as llama.cpp gives it (`content`, `timings`). It reads+ `n_predict` (0: the prompt only, read so that the next request finds it+ computed), `n_indent` (a new line indented less ends the text) and+ `t_max_predict_ms` (after a new line, past that time the text ends), as+ llama.cpp does. The code of other files (`input_extra`) is not used: it+ comes first, changes often, and every change would have the whole+ prompt read again. The most likely token is always taken, so the same+ place gives the same proposal.++## The window++An editor asks at every keystroke. The server keeps computed what the+previous request shares from its start, so what matters is that the+start of the text sent does not move; and the extensions move it, sending+"the last N lines before the cursor". So, for requests that fill in:++- the code before the cursor is read up to `--fim-prefix` tokens (2048);+ over that it is cut at a line to three quarters of it, which leaves+ room for many lines before the next cut;+- the start of each window is remembered, for a few files, and a later+ request holding it is cut there;+- a request whose text starts inside the head of a window before, past+ its start, comes from an editor that slides its window: its start is put+ a quarter of the way in, so that the lines it slides next keep it in;+- the code after the cursor, read again at every request, is cut after+ `--fim-suffix` lines (8);+- the text written is at most `--fim-max` tokens (256), whatever the+ request asks: the extensions ask for thousands, and the requests of the+ next keystrokes would wait for them.++Each cut costs one reading of the whole window; the keystrokes after it+read only what they changed and the lines after the cursor.++## In VSCodium (and VS Code): llama.vscode++[llama.vscode](https://github.com/ggml-org/llama.vscode), by the authors of+llama.cpp, is on Open VSX, where VSCodium finds it. In the settings:++```json+{+ "llama-vscode.endpoint": "http://127.0.0.1:8080",+ "llama-vscode.max_parallel_completions": 1,+ "llama-vscode.ring_n_chunks": 0,+ "llama-vscode.rag_enabled": false,+ "editor.quickSuggestions": {"other": "off", "comments": "off", "strings": "off"},+ "editor.inlineSuggest.suppressSuggestions": true+}+```++One proposal is written (`max_parallel_completions`), the chunks of other+files would not be used (`ring_n_chunks`), and the editor's own list of+suggestions is kept from opening by itself: while it is open, a proposal+that does not extend its selected entry is not shown. Its welcome page+offers to install llama.cpp: it is not needed.++Tab takes the proposal, Shift+Tab its first line, Ctrl+Right its first+word; Ctrl+L asks for one.++## Continue++[Continue](https://www.continue.dev) works too, with three things to know:+the provider must be one that sends the suffix (`siliconflow`, whose+requests go to `{apiBase}completions` with `prompt` and `suffix`: the+`openai` provider builds the prompt itself and sends it as raw text), its+default `modelTimeout` of 150 ms cuts a proposal after ten pieces, and the+snippets of other files it puts in front of the prefix change at every+request. In `~/.continue/config.yaml`:++```yaml+name: Janas+version: 1.0.0+schema: v1+models:+ - name: Janas Coder+ provider: siliconflow+ model: qwen3-coder-30b-a3b-q4km+ apiBase: http://127.0.0.1:8080/v1/ # the final slash is needed+ apiKey: none+ roles:+ - autocomplete+ autocompleteOptions:+ maxPromptTokens: 2048+ prefixPercentage: 0.85+ maxSuffixPercentage: 0.1+ debounceDelay: 500+ modelTimeout: 20000+ multilineCompletions: never+ useCache: false+ useRecentlyEdited: false+ useRecentlyOpened: false+ useImports: false+ experimental_includeRecentlyVisitedRanges: false+ experimental_includeRecentlyEditedRanges: false+ experimental_includeDiff: false+```++With Continue some proposals the server wrote were not shown, and the+editor was not seen to ask again after some keystrokes; why was not found+(Continue filters, caches and reuses proposals before showing them).+llama.vscode's behaviour was easier to follow.++## What has been run++1 October 2026, Qwen3-Coder-30B-A3B-Instruct Q4_K_M on an Intel Core+Ultra 9 185H with every expert in memory (`--cache 18`), VSCodium 1.135+with llama.vscode 0.0.67 and Continue 2.0.0.++The window, with requests as an editor makes them on the 1,445 lines of+`src/chat/main.c`, the cursor going down a line a request, time to the+first token:++| Requests | first request | the ones after |+|---|---|---|+| before the window: the 110 lines before the cursor, slid a line (1,330 tokens) | 17 s | 17 s: the whole window read again |+| before the window: the file from its start to the cursor (5,300 tokens) | 81 s | 7 s |+| now: the 200 lines before the cursor, slid a line (cut to 1,527 tokens) | 19 s | 1.5-2.2 s |+| now: the file from its start to the cursor (cut to 1,566 tokens) | 20 s | 1.6-1.8 s |++In VSCodium with llama.vscode, writing in `src/flights/schedule.c`: after+the first request of a window, each request found 680-800 of its 750-870+tokens computed and was answered in one to two seconds (about 90 tokens+read, the lines after the cursor and what had changed, and 2-30+written).++What it proposed:++- under the comment "how many answers are kept and still fresh", the body+ of a new function counting the answers kept, with the names of the file+ (`kept[]`, `N_KEPT`); with `time_t now = time(NULL);` written first, it+ added the test on their age (`now - kept[i].at < KEEP_S`) by itself;+- for a line taken out of `local_time` (`tm.tm_mon -= 1;`), the right line+ first (`tm.tm_mon--;`), and then lines of its own: a handling of time+ zones the file does not have. With 8, 16, 32 or 64 lines after the+ cursor given to it the first line was always right and the rest always+ invented: the model does not see that only one line was missing.+ Shift+Tab takes the first line only.++The first request of a new window reads all of it, at 65-80 tokens a+second: 10-20 s for the 750-1,500 tokens of a window.diff --git a/src/server/engine.c b/src/server/engine.cindex e665049..86e5971 100644--- a/src/server/engine.c+++ b/src/server/engine.c@@ -13,6 +13,7 @@* prompt, which is then read once where the model can go back over it.*/#include "server.h"+#include "fim.h"#include <stdio.h>#include <stdlib.h>@@ -31,6 +32,8 @@ struct srv_engine {int stop;int verbose;uint32_t max_input, warn_input;+ struct srv_fim *fim; /* the windows of fill-in-the-middle requests */+ uint32_t fim_max; /* the tokens they write at most; 0: as asked */char desc[1024];char system[1024]; /* the system message Janas's programs use */int64_t created;@@ -313,6 +316,36 @@ static int32_t ids_text(struct srv_engine *e, struct srv_prompt *pp)return pp->text ? JANAS_LLM_OK : JANAS_LLM_EINVAL;}+/* The tokens of a text, for the window of a fill-in-the-middle request. */+static int32_t count_tokens(void *ctx, const char *s, size_t n)+{+ int32_t t = 0;+ if (n > INT32_MAX)+ return -1;+ int32_t rc = janas_llm_tokenize(ctx, s, (int32_t)n, 0, NULL, 0, &t);+ return rc == JANAS_LLM_OK || rc == JANAS_LLM_ESMALL ? t : -1;+}++/* The prompt of a request with a suffix: the window around the cursor+ (fim.c), so that the next keystroke finds it computed. */+static int32_t load_infill(struct srv_engine *e, struct srv_job *j,+ const struct srv_prompt *pp)+{+ size_t off = 0, sl = j->n_suffix;+ const char *how = "";+ if (e->fim)+ srv_fim_window(e->fim, count_tokens, e->llm, pp->text, pp->n, j->suffix,+ j->n_suffix, &off, &sl, &how);+ if (e->verbose && (off || sl < j->n_suffix))+ fprintf(stderr,+ "janas-server: fill in the middle: prefix from byte %zu of "+ "%zu%s%s, suffix %zu of %zu bytes\n",+ off, pp->n, *how ? ", " : "", how, sl, j->n_suffix);+ return janas_llm_chat_infill(e->chat, pp->text + off,+ (int32_t)(pp->n - off), j->suffix,+ (int32_t)sl);+}+/* The request's prompt for reply k. */static int32_t load(struct srv_engine *e, struct srv_job *j, int k){@@ -324,10 +357,8 @@ static int32_t load(struct srv_engine *e, struct srv_job *j, int k)if (rc != JANAS_LLM_OK)return rc;}- return j->suffix- ? janas_llm_chat_infill(c, pp->text, (int32_t)pp->n,- j->suffix, (int32_t)j->n_suffix)- : janas_llm_chat_prompt(c, pp->text, (int32_t)pp->n);+ return j->suffix ? load_infill(e, j, pp)+ : janas_llm_chat_prompt(c, pp->text, (int32_t)pp->n);}int32_t rc = janas_llm_chat_begin(c);for (int32_t i = 0; rc == JANAS_LLM_OK && i < j->n_msg; i++) {@@ -340,6 +371,36 @@ static int32_t load(struct srv_engine *e, struct srv_job *j, int k)return rc == JANAS_LLM_OK ? janas_llm_chat_run(c) : rc;}+/* /infill's n_indent: a line of the text started past *nl indented less+ than n_indent (blank lines aside) ends it there, as llama.cpp does. 1+ when cut. */+static int indent_cut(struct srv_choice *ch, int n_indent, size_t *nl)+{+ for (;;) {+ if (*nl > 0) {+ size_t pos = *nl;+ int ind = 0;+ while (pos < ch->n_text &&+ (ch->text[pos] == ' ' || ch->text[pos] == '\t')) {+ ind++;+ pos++;+ }+ if (pos < ch->n_text && ch->text[pos] != '\n' &&+ ch->text[pos] != '\r' && ind < n_indent) {+ ch->n_text = *nl;+ ch->text[*nl] = 0;+ return 1;+ }+ }+ const char *p = *nl < ch->n_text+ ? memchr(ch->text + *nl, '\n', ch->n_text - *nl)+ : NULL;+ if (!p)+ return 0;+ *nl = (size_t)(p - ch->text) + 1;+ }+}+/* One reply, into choice k. Returns the library's code. */static int32_t run_one(struct srv_engine *e, struct srv_job *j, int k){@@ -349,6 +410,9 @@ static int32_t run_one(struct srv_engine *e, struct srv_job *j, int k)if (j->seeded)p.seed += (uint64_t)k;p.max_input = e->max_input; /* the server's limit, not the request's */+ if (j->suffix && e->fim_max &&+ (p.max_reply <= 0 || p.max_reply > (int32_t)e->fim_max))+ p.max_reply = (int32_t)e->fim_max;int32_t rc = janas_llm_chat_set_params(c, &p);if (rc == JANAS_LLM_OK)rc = load(e, j, k);@@ -357,6 +421,8 @@ static int32_t run_one(struct srv_engine *e, struct srv_job *j, int k)if (strlen(j->stop[i]) > max_stop)max_stop = strlen(j->stop[i]);int32_t seen = 0;+ size_t nl = 0; /* n_indent: where the last line started */+ struct timespec t0 = {0}; /* t_max_predict_ms: the first token */while (rc == JANAS_LLM_OK) {char buf[512];int32_t len = 0;@@ -380,6 +446,19 @@ static int32_t run_one(struct srv_engine *e, struct srv_job *j, int k)ch->text[at] = 0;ch->by_stop = 1;}+ if (!bad && j->n_indent > 0 && indent_cut(ch, j->n_indent, &nl))+ ch->by_stop = 1;+ if (!bad && j->t_max_predict_ms > 0) {+ struct timespec now;+ clock_gettime(CLOCK_MONOTONIC, &now);+ if (!t0.tv_sec && !t0.tv_nsec)+ t0 = now;+ double ms = (double)(now.tv_sec - t0.tv_sec) * 1e3 ++ (double)(now.tv_nsec - t0.tv_nsec) / 1e6;+ if (ms > j->t_max_predict_ms &&+ memchr(ch->text, '\n', ch->n_text))+ ch->by_stop = 1;+ }}int quit = j->cancel || ch->by_stop || bad || rc == JANAS_LLM_DONE;pthread_cond_broadcast(&j->cv);@@ -677,8 +756,11 @@ struct srv_engine *srv_engine_start(const struct srv_config *cfg, char *err,janas_llm_describe(e->llm, e->desc, sizeof(e->desc), &n);janas_llm_default_system(e->llm, e->system, sizeof(e->system), &n);e->created = (int64_t)time(NULL);+ e->fim = srv_fim_new(cfg->fim_prefix, cfg->fim_suffix);+ e->fim_max = cfg->fim_max;if (pthread_create(&e->th, NULL, engine_main, e) != 0) {snprintf(err, err_len, "cannot start the engine's thread");+ srv_fim_free(e->fim);janas_llm_chat_destroy(e->chat);janas_llm_close(e->llm);free(e);@@ -712,6 +794,7 @@ void srv_engine_stop(struct srv_engine *e)janas_llm_chat_destroy(e->chat);janas_llm_close(e->llm);janas_llm_close(e->emb);+ srv_fim_free(e->fim);pthread_mutex_destroy(&e->mu);pthread_cond_destroy(&e->cv);free(e);diff --git a/src/server/fim.c b/src/server/fim.cnew file mode 100644index 0000000..2bb0544--- /dev/null+++ b/src/server/fim.c@@ -0,0 +1,216 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * fim.c - the window of a fill-in-the-middle request.+ *+ * An editor asks at every keystroke, with the text before the cursor and+ * after it. The session reuses what the last request shares from the+ * start, so the start of the prefix must not move. An editor that sends+ * "the last N lines" moves it at every new line, and the whole prefix was+ * read again - 81 s for 5300 tokens, Qwen3-Coder-30B on a laptop, against+ * 7 s for one keystroke more.+ *+ * So the head of each window is remembered (a few, for a few files):+ * - a later prefix holding the start of one is cut there;+ * - a prefix starting inside the head of one, past its start, comes from+ * an editor that slides its window: it is cut a quarter of the way in,+ * so that the next lines it slides keep that start in;+ * - a prefix over the budget is cut to three quarters of it, at a line,+ * which leaves room for many lines before the next cut.+ * Each cut costs one reading of the whole window, and the keystrokes after+ * it read only what they changed.+ *+ * The suffix comes after the prefix and is read again at every request:+ * only its first lines are kept.+ */+#include "fim.h"++#include <stdlib.h>+#include <string.h>++#define N_HEADS 8 /* windows remembered: a few files */+#define HEAD_MAX 4096 /* bytes of a window's head remembered */+#define START_MAX 160 /* of them, the bytes a start is matched by */+#define START_MIN 24 /* shorter ones match too easily */++struct head {+ char s[HEAD_MAX];+ size_t n;+ uint64_t used;+};++struct srv_fim {+ uint32_t prefix_tokens, suffix_lines;+ uint64_t clock;+ struct head h[N_HEADS];+};++struct srv_fim *srv_fim_new(uint32_t prefix_tokens, uint32_t suffix_lines)+{+ struct srv_fim *f = calloc(1, sizeof(*f));+ if (f) {+ f->prefix_tokens = prefix_tokens;+ f->suffix_lines = suffix_lines;+ }+ return f;+}++void srv_fim_free(struct srv_fim *f)+{+ free(f);+}++/* The start of the line at off or the first after it; without one, off+ itself moved past the bytes that continue a UTF-8 character. */+static size_t line_from(const char *p, size_t n, size_t off)+{+ if (off == 0 || off >= n)+ return off < n ? off : n;+ if (p[off - 1] == '\n')+ return off;+ const char *nl = memchr(p + off, '\n', n - off);+ if (nl && (size_t)(nl - p) + 1 < n)+ return (size_t)(nl - p) + 1;+ while (off < n && ((unsigned char)p[off] & 0xC0) == 0x80)+ off++;+ return off;+}++/* Where the k bytes of s are at the start of a line of t (n bytes), from+ the line at or after from on: the first place, or SIZE_MAX. */+static size_t at_line(const char *t, size_t n, size_t from, const char *s,+ size_t k)+{+ for (size_t o = line_from(t, n, from); o < n;) {+ if (n - o >= k && memcmp(t + o, s, k) == 0)+ return o;+ const char *nl = memchr(t + o, '\n', n - o);+ if (!nl)+ break;+ o = (size_t)(nl - t) + 1;+ }+ return SIZE_MAX;+}++static size_t start_len(const struct head *h)+{+ return h->n < START_MAX ? h->n : START_MAX;+}++/* The prefix cut at a line so that its tokens are want at most, from off+ on: the new offset. */+static size_t cut_to(srv_fim_count count, void *ctx, const char *pre, size_t n,+ size_t off, int32_t t, uint32_t want)+{+ for (int tries = 0; tries < 16 && t > (int32_t)want; tries++) {+ size_t rest = n - off;+ /* the bytes in proportion, a little less each try */+ size_t keep = (size_t)((double)rest * want / t * (1.0 - tries / 16.0));+ size_t to = line_from(pre, n, n - keep);+ if (to <= off)+ to = line_from(pre, n, off + 1);+ off = to;+ if (off >= n)+ break;+ t = count(ctx, pre + off, n - off);+ if (t < 0)+ break;+ }+ return off;+}++void srv_fim_window(struct srv_fim *f, srv_fim_count count, void *ctx,+ const char *pre, size_t n_pre, const char *suf,+ size_t n_suf, size_t *pre_off, size_t *suf_len,+ const char **how)+{+ *how = "";+ *suf_len = n_suf;+ if (f->suffix_lines) {+ uint32_t lines = 0;+ for (size_t i = 0; i < n_suf; i++)+ if (suf[i] == '\n' && ++lines == f->suffix_lines) {+ *suf_len = i + 1;+ break;+ }+ }++ /* blank lines in front are left out: Continue joins its snippets of+ other files to the prefix with a newline, and sends one even with+ none, which put each window's start at the end of a line */+ size_t lead = 0;+ while (lead < n_pre && (pre[lead] == '\n' || pre[lead] == '\r'))+ lead++;+ pre += lead;+ n_pre -= lead;++ /* the start of a window before, the one used last if several */+ int hit = -1;+ size_t off = 0;+ for (int i = 0; i < N_HEADS; i++) {+ if (!f->h[i].n)+ continue;+ size_t o = at_line(pre, n_pre, 0, f->h[i].s, start_len(&f->h[i]));+ if (o != SIZE_MAX && (hit < 0 || f->h[i].used > f->h[hit].used)) {+ hit = i;+ off = o;+ }+ }+ if (off)+ *how = "at the start of the window before";++ /* none: does this prefix start inside the head of one, past its start?+ Then the editor slides its window, and the start is put ahead */+ size_t k = n_pre < START_MAX ? n_pre : START_MAX;+ if (hit < 0 && k >= START_MIN)+ for (int i = 0; i < N_HEADS; i++)+ if (f->h[i].n > k &&+ at_line(f->h[i].s, f->h[i].n, 1, pre, k) != SIZE_MAX) {+ off = line_from(pre, n_pre, n_pre / 4);+ *how = "ahead of a sliding window";+ break;+ }++ if (f->prefix_tokens && count) {+ int32_t t = count(ctx, pre + off, n_pre - off);+ uint32_t want = f->prefix_tokens / 4 * 3;+ if (t > (int32_t)f->prefix_tokens) {+ off = cut_to(count, ctx, pre, n_pre, off, t, want);+ *how = "over the budget";+ hit = -1;+ }+ }+ /* a start that would leave less than half of the text, and under a+ budget less than three eighths of it, is too far (the cursor went+ up): the window starts again */+ int32_t kept = f->prefix_tokens && count && hit >= 0+ ? count(ctx, pre + off, n_pre - off)+ : 0;+ if (hit >= 0 && off > n_pre / 2 &&+ (!f->prefix_tokens || kept < (int32_t)(f->prefix_tokens / 8 * 3))) {+ off = 0;+ *how = "";+ hit = -1;+ int32_t t = f->prefix_tokens && count ? count(ctx, pre, n_pre) : 0;+ if (t > (int32_t)f->prefix_tokens) {+ off =+ cut_to(count, ctx, pre, n_pre, 0, t, f->prefix_tokens / 4 * 3);+ *how = "over the budget";+ }+ }++ size_t rest = n_pre - off;+ size_t len = rest / 2 < HEAD_MAX ? rest / 2 : HEAD_MAX;+ if (hit >= 0) {+ f->h[hit].used = ++f->clock;+ } else if (len >= START_MIN) {+ int slot = 0;+ for (int i = 1; i < N_HEADS; i++)+ if (f->h[i].used < f->h[slot].used)+ slot = i;+ memcpy(f->h[slot].s, pre + off, len);+ f->h[slot].n = len;+ f->h[slot].used = ++f->clock;+ }+ *pre_off = lead + off;+}diff --git a/src/server/fim.h b/src/server/fim.hnew file mode 100644index 0000000..42768f5--- /dev/null+++ b/src/server/fim.h@@ -0,0 +1,31 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * fim.h - the window of a fill-in-the-middle request (fim.c): the part of+ * the text before the cursor and after it that the model reads, chosen so+ * that the next keystroke finds most of it already computed.+ */+#ifndef JANAS_SERVER_FIM_H+#define JANAS_SERVER_FIM_H++#include <stddef.h>+#include <stdint.h>++struct srv_fim;++/* The tokens of n bytes of s, or -1. */+typedef int32_t (*srv_fim_count)(void *ctx, const char *s, size_t n);++/* prefix_tokens: the text before the cursor read at most (0: all of it);+ suffix_lines: the lines after it (0: all of them). */+struct srv_fim *srv_fim_new(uint32_t prefix_tokens, uint32_t suffix_lines);+void srv_fim_free(struct srv_fim *f);++/* The window: the prefix from *pre_off on, the first *suf_len bytes of the+ suffix. *how says why the prefix was cut ("" when it was not). */+void srv_fim_window(struct srv_fim *f, srv_fim_count count, void *ctx,+ const char *pre, size_t n_pre, const char *suf,+ size_t n_suf, size_t *pre_off, size_t *suf_len,+ const char **how);++#endifdiff --git a/src/server/http.c b/src/server/http.cindex 3319c61..b643472 100644--- a/src/server/http.c+++ b/src/server/http.c@@ -288,6 +288,10 @@ static enum MHD_Result dispatch(struct srv_req *r)r, 401, "invalid_request_error", "invalid_api_key","Incorrect or missing API key: send it as ""\"Authorization: Bearer <key>\".");+ /* llama.cpp's route for filling in code, which editors' extensions+ made for it ask: not OpenAI's, so not in routes.inc */+ if (strcmp(r->path, "/infill") == 0 && strcmp(r->method, "POST") == 0)+ return (enum MHD_Result)srv_infill(r);const struct route *other = NULL;for (size_t i = 0; i < sizeof(routes) / sizeof(routes[0]); i++) {const struct route *rt = &routes[i];diff --git a/src/server/infill.c b/src/server/infill.cnew file mode 100644index 0000000..2f1bce6--- /dev/null+++ b/src/server/infill.c@@ -0,0 +1,197 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * infill.c - POST /infill, llama.cpp's route for filling in code, which+ * the editors' extensions made for it ask (llama.vscode, llama.vim): the+ * lines before the cursor (input_prefix), the line up to it (prompt), and+ * what follows (input_suffix), into the same window as a completion with+ * a suffix (fim.c), the answer as llama.cpp gives it (content, timings).+ *+ * Read as llama.cpp reads them: n_predict (0: the prompt only, read so+ * that the next request finds it computed), n_indent (a new line indented+ * less ends the text), t_max_predict_ms (after a new line, past that time+ * the text ends). The extra context of other files (input_extra) is not+ * used: it changes often, comes first, and every change would have the+ * whole prompt read again. The samplers are not either: the most likely+ * token is taken, so that the same place gives the same proposal.+ */+#include "request.h"++#include <stdio.h>+#include <stdlib.h>+#include <string.h>++static const char *str_of(yyjson_val *o, const char *k, size_t *n, int *bad)+{+ yyjson_val *v = yyjson_obj_get(o, k);+ if (!v || yyjson_is_null(v)) {+ *n = 0;+ return NULL;+ }+ if (!yyjson_is_str(v)) {+ *bad = 1;+ return NULL;+ }+ *n = yyjson_get_len(v);+ return yyjson_get_str(v);+}++static long num_of(yyjson_val *o, const char *k, long dflt, int *bad)+{+ yyjson_val *v = yyjson_obj_get(o, k);+ if (!v || yyjson_is_null(v))+ return dflt;+ if (!yyjson_is_num(v)) {+ *bad = 1;+ return dflt;+ }+ return (long)yyjson_get_num(v);+}++/* The answer, as llama.cpp's: the fields the extensions read. */+static char *answer(const struct srv_req *r, const struct srv_job *j, size_t *n)+{+ const struct srv_choice *ch = &j->ch[0];+ const struct janas_llm_chat_stats *st = &j->st;+ const char *text = j->prefill_only || !ch->text ? "" : ch->text;+ double in_ms = st->input_seconds * 1000.0,+ out_ms = st->output_seconds * 1000.0;+ uint32_t out_n = j->prefill_only ? 0 : st->output_tokens;+ const char *stop_type =+ ch->finish == JANAS_LLM_FINISH_LENGTH ? "limit" : "eos";+ yyjson_mut_doc *d = yyjson_mut_doc_new(NULL);+ if (!d)+ return NULL;+ yyjson_mut_val *o = yyjson_mut_obj(d);+ yyjson_mut_doc_set_root(d, o);+ yyjson_mut_obj_add_strncpy(d, o, "content", text, strlen(text));+ yyjson_mut_obj_add_strcpy(d, o, "model", r->srv->cfg.model_id);+ yyjson_mut_obj_add_bool(d, o, "stop", true);+ yyjson_mut_obj_add_strcpy(d, o, "stop_type", stop_type);+ yyjson_mut_obj_add_uint(d, o, "tokens_predicted", out_n);+ yyjson_mut_obj_add_uint(d, o, "tokens_evaluated", st->prompt_tokens);+ yyjson_mut_obj_add_uint(d, o, "tokens_cached", st->context_used);+ yyjson_mut_obj_add_bool(d, o, "truncated", false);+ yyjson_mut_val *gs = yyjson_mut_obj_add_obj(d, o, "generation_settings");+ yyjson_mut_obj_add_uint(d, gs, "n_ctx", st->context_size);+ yyjson_mut_obj_add_strcpy(d, gs, "model", r->srv->cfg.model_id);+ yyjson_mut_val *t = yyjson_mut_obj_add_obj(d, o, "timings");+ yyjson_mut_obj_add_uint(d, t, "cache_n", st->cached_tokens);+ yyjson_mut_obj_add_uint(d, t, "prompt_n", st->input_tokens);+ yyjson_mut_obj_add_real(d, t, "prompt_ms", in_ms);+ yyjson_mut_obj_add_real(d, t, "prompt_per_second",+ in_ms > 0 ? st->input_tokens * 1000.0 / in_ms : 0);+ yyjson_mut_obj_add_uint(d, t, "predicted_n", out_n);+ yyjson_mut_obj_add_real(d, t, "predicted_ms", out_ms);+ yyjson_mut_obj_add_real(d, t, "predicted_per_second",+ out_ms > 0 ? out_n * 1000.0 / out_ms : 0);+ char *json = yyjson_mut_write(d, 0, n);+ yyjson_mut_doc_free(d);+ return json;+}++int srv_infill(struct srv_req *r)+{+ yyjson_doc *doc = yyjson_read(r->body, r->n_body, 0);+ yyjson_val *root = doc ? yyjson_doc_get_root(doc) : NULL;+ if (!root || !yyjson_is_obj(root)) {+ yyjson_doc_free(doc);+ return srv_reply_error(r, 400, "invalid_request_error", NULL,+ "The body must be a JSON object.");+ }+ int bad = 0;+ size_t n_pre = 0, n_line = 0, n_suf = 0;+ const char *pre = str_of(root, "input_prefix", &n_pre, &bad);+ const char *line = str_of(root, "prompt", &n_line, &bad);+ const char *suf = str_of(root, "input_suffix", &n_suf, &bad);+ long n_predict = num_of(root, "n_predict", -1, &bad);+ long n_indent = num_of(root, "n_indent", 0, &bad);+ long t_max = num_of(root, "t_max_predict_ms", 0, &bad);+ yyjson_val *ex = yyjson_obj_get(root, "input_extra");+ yyjson_val *sm = yyjson_obj_get(root, "stream");+ const char *why = NULL;+ if (bad)+ why = "'input_prefix', 'input_suffix' and 'prompt' must be strings, "+ "'n_predict', 'n_indent' and 't_max_predict_ms' numbers.";+ else if (!pre || !suf)+ why = "'input_prefix' and 'input_suffix' are required.";+ else if (ex && !yyjson_is_null(ex) && !yyjson_is_arr(ex))+ why = "'input_extra' must be an array of {\"filename\": string, "+ "\"text\": string}.";+ else if (sm && yyjson_is_bool(sm) && yyjson_get_bool(sm))+ why = "'stream' is not supported on /infill: the answer comes whole.";+ if (why) {+ yyjson_doc_free(doc);+ return srv_reply_error(r, 400, "invalid_request_error", "invalid_value",+ "%s", why);+ }++ struct srv_job *j = srv_job_new();+ if (!j || srv_job_choices(j, 1, 1) != 0 ||+ !(j->prompts = calloc(1, sizeof(*j->prompts)))) {+ yyjson_doc_free(doc);+ srv_job_release(j);+ return srv_reply_error(r, 500, "server_error", NULL, "out of memory");+ }+ j->kind = JOB_TEXT;+ j->n_prompts = 1;+ struct srv_prompt *pp = &j->prompts[0];+ pp->n = n_pre + n_line;+ pp->text = malloc(pp->n + 1);+ j->suffix = srv_dup_n(suf, n_suf);+ j->n_suffix = n_suf;+ if (!pp->text || !j->suffix) {+ yyjson_doc_free(doc);+ srv_job_release(j);+ return srv_reply_error(r, 500, "server_error", NULL, "out of memory");+ }+ memcpy(pp->text, pre, n_pre);+ if (n_line)+ memcpy(pp->text + n_pre, line, n_line);+ pp->text[pp->n] = 0;+ if (r->srv->cfg.verbose && ex && yyjson_arr_size(ex))+ fprintf(stderr,+ "janas-server: /infill: %zu chunks of other files "+ "left out\n",+ yyjson_arr_size(ex));+ yyjson_doc_free(doc);++ j->p.temperature = 0;+ j->p.thinking = 0;+ /* n_predict 0: the prompt only, which costs one token written */+ j->prefill_only = n_predict == 0;+ j->p.max_reply = n_predict == 0 ? 1+ : n_predict > 0 ? (int32_t)n_predict+ : 0;+ j->n_indent = n_indent > 0 ? (int)n_indent : 0;+ j->t_max_predict_ms = t_max > 0 ? (int32_t)t_max : 0;++ if (srv_engine_submit(r->srv->engine, j) != 0) {+ srv_job_release(j);+ return srv_reply_error(r, 429, "rate_limit_exceeded", NULL,+ "The server is busy: %u requests are waiting. "+ "Try again shortly.",+ r->srv->cfg.max_queue);+ }+ pthread_mutex_lock(&j->mu);+ while (!j->done)+ pthread_cond_wait(&j->cv, &j->mu);+ pthread_mutex_unlock(&j->mu);+ int ret;+ if (j->err != JANAS_LLM_OK || j->ch[0].finish == JANAS_LLM_FINISH_ERROR) {+ unsigned code =+ j->err == JANAS_LLM_EFULL || j->err == JANAS_LLM_ELIMIT ||+ j->err == JANAS_LLM_EINVAL || j->err == JANAS_LLM_EMODEL+ ? 400+ : 500;+ ret = srv_reply_error(+ r, code, code == 400 ? "invalid_request_error" : "server_error",+ NULL, "%s", j->errmsg[0] ? j->errmsg : "generation failed");+ } else {+ size_t n = 0;+ char *json = answer(r, j, &n);+ ret = srv_reply_json(r, 200, json, n);+ }+ srv_job_release(j);+ return ret;+}diff --git a/src/server/main.c b/src/server/main.cindex 4e073fe..2575f0a 100644--- a/src/server/main.c+++ b/src/server/main.c@@ -67,6 +67,16 @@ static void usage(FILE *f)" below the context: longer ones get a 400 that\n"" says by how much (nothing is ever cut)\n"" --warn-input N longer inputs are taken, and said on stderr\n"+ " --fim-prefix N completions with a suffix (an editor filling in\n"+ " code): the tokens before the cursor read at\n"+ " most, cut at a line and from a start kept\n"+ " across keystrokes (default 2048; 0: all)\n"+ " --fim-suffix N the lines after the cursor read (default 8;\n"+ " 0: all)\n"+ " --fim-max N the tokens written there at most, whatever the\n"+ " request asks (default 256; 0: as asked): an\n"+ " editor asks for thousands, and the requests\n"+ " of the next keystrokes wait for them\n"" --metrics FILE JSON lines of what every reply costs: prompt,\n"" progress of a long one, first token, the end\n"" (counters and times, never text)\n"@@ -123,6 +133,9 @@ int main(int argc, char **argv)c->max_queue = 16;c->keep = 8;c->keep_disk = (uint64_t)8 << 30;+ c->fim_prefix = 2048;+ c->fim_suffix = 8;+ c->fim_max = 256;int no_store = 0;c->max_body = (size_t)32 << 20;c->max_connections = 64;@@ -195,6 +208,12 @@ int main(int argc, char **argv)c->max_input = (uint32_t)atol(arg(argc, argv, &i));} else if (!strcmp(a, "--warn-input")) {c->warn_input = (uint32_t)atol(arg(argc, argv, &i));+ } else if (!strcmp(a, "--fim-prefix")) {+ c->fim_prefix = (uint32_t)atol(arg(argc, argv, &i));+ } else if (!strcmp(a, "--fim-suffix")) {+ c->fim_suffix = (uint32_t)atol(arg(argc, argv, &i));+ } else if (!strcmp(a, "--fim-max")) {+ c->fim_max = (uint32_t)atol(arg(argc, argv, &i));} else if (!strcmp(a, "--metrics")) {setenv("JANAS_METRICS", arg(argc, argv, &i), 1);} else if (!strcmp(a, "--progress")) {diff --git a/src/server/request_opts.c b/src/server/request_opts.cindex e2aede3..56daee6 100644--- a/src/server/request_opts.c+++ b/src/server/request_opts.c@@ -53,9 +53,9 @@ static int read_stop(struct srv_perr *e, struct srv_parsed *pr, yyjson_val *st,} else if (st && yyjson_is_arr(st)) {size_t idx, max;yyjson_val *s;- if (yyjson_arr_size(st) > 4)- return srv_pfail(- pr, BAD("invalid_value", "'stop' holds at most four strings."));+ if (yyjson_arr_size(st) > SRV_MAX_STOP)+ return srv_pfail(pr, BAD("invalid_value",+ "'stop' holds at most sixteen strings."));yyjson_arr_foreach(st, idx, max, s){if (!yyjson_is_str(s))diff --git a/src/server/server.h b/src/server/server.hindex 0b5cc42..b9c84b2 100644--- a/src/server/server.h+++ b/src/server/server.h@@ -52,8 +52,16 @@ struct srv_config {no connection goes out on a request's word */uint32_t max_input; /* longest prompt taken, tokens; 0: the context */uint32_t warn_input; /* prompts longer are taken, and said on stderr */+ uint32_t fim_prefix; /* fill in the middle: the tokens before the+ cursor read at most (0: all) */+ uint32_t fim_suffix; /* the lines after it (0: all) */+ uint32_t fim_max; /* the tokens it writes at most (0: as asked) */};+/* Stop strings a request may give: OpenAI takes four, editors' extensions+ send more (Continue up to sixteen). */+#define SRV_MAX_STOP 16+/* The engine: the model, its conversations, and the queue. */struct srv_engine;@@ -122,9 +130,15 @@ struct srv_job {int n_prompts;char *suffix;size_t n_suffix;+ /* /infill, as llama.cpp: a new line indented less than n_indent ends+ the text; past t_max_predict_ms, so does a new line; prefill_only:+ the prompt is read, nothing is answered */+ int n_indent, prefill_only;+ int32_t t_max_predict_ms;struct janas_llm_chat_params p;- int seeded; /* the request gave a seed: choice k takes seed + k */- char *stop[4]; /* strings that end the reply where they appear */+ int seeded; /* the request gave a seed: choice k takes seed + k */+ char *stop[SRV_MAX_STOP]; /* strings that end the reply where they+ appear */int n_stop;/* tools (a JSON array, as the request has it), tool_choice, and theform of the answer */@@ -246,6 +260,7 @@ int srv_embeddings_create(struct srv_req *r);int srv_moderations_create(struct srv_req *r);int srv_chat_create(struct srv_req *r);int srv_completion_create(struct srv_req *r);+int srv_infill(struct srv_req *r); /* POST /infill, llama.cpp's *//* stored.c: the completions kept with store: true */int srv_stored_list(struct srv_req *r);int srv_stored_get(struct srv_req *r);diff --git a/tests/test_server_fim.c b/tests/test_server_fim.cnew file mode 100644index 0000000..9708f40--- /dev/null+++ b/tests/test_server_fim.c@@ -0,0 +1,186 @@+/* SPDX-License-Identifier: GPL-3.0-or-later+ Copyright (C) 2026 Maurizio Cammalleri */+/*+ * test_server_fim.c - the window of a fill-in-the-middle request+ * (src/server/fim.c), with a token every four bytes:+ * 1. the suffix is cut after its lines;+ * 2. an editor that sends the prefix from the start of the file keeps+ * its start;+ * 3. an editor that slides a window of 200 lines: the start is put a+ * quarter ahead at its first slide, and kept for the slides after;+ * 4. a prefix over the budget is cut at a line to three quarters of it,+ * and the start kept while the prefix grows;+ * 5. two files in turn keep a start each;+ * 6. as 3, with a newline in front of every prefix, as Continue sends.+ */+#include <stdio.h>+#include <stdlib.h>+#include <string.h>+#include <unistd.h>++#include "server/fim.c"++static int failures;++#define CHECK(c, ...) \+ do { \+ if (!(c)) { \+ printf(__VA_ARGS__); \+ printf("\n"); \+ failures++; \+ } \+ } while (0)++static int32_t by4(void *ctx, const char *s, size_t n)+{+ (void)ctx, (void)s;+ return (int32_t)((n + 3) / 4);+}++/* lines from..to-1 of a file named name, into a buffer of its own */+static char *lines(const char *name, int from, int to, size_t *n)+{+ size_t cap = (size_t)(to - from) * 64 + 1;+ char *b = malloc(cap), *p = b;+ for (int i = from; i < to; i++)+ p += snprintf(p, cap - (size_t)(p - b),+ "%s line %04d: x = f(x, %d); /* code */\n", name, i, i);+ *n = (size_t)(p - b);+ return b;+}++/* the line number a window starts at */+static int first_line(const char *s)+{+ const char *p = strstr(s, " line ");+ return p ? atoi(p + 6) : -1;+}++int main(void)+{+ alarm(60);+ size_t off, sl, n;+ const char *how;++ /* 1. */+ struct srv_fim *f = srv_fim_new(0, 8);+ char *suf = lines("s", 0, 20, &n);+ char *pre = lines("a", 0, 10, &n);+ srv_fim_window(f, by4, NULL, pre, n, suf, strlen(suf), &off, &sl, &how);+ size_t want = 0;+ for (int i = 0; i < 8; i++)+ want += strchr(suf + want, '\n') - (suf + want) + 1;+ CHECK(sl == want, "suffix %zu bytes, not %zu", sl, want);+ free(suf);+ free(pre);+ srv_fim_free(f);++ /* 2. */+ f = srv_fim_new(0, 0);+ for (int end = 100; end < 160; end += 7) {+ pre = lines("a", 0, end, &n);+ srv_fim_window(f, by4, NULL, pre, n, "", 0, &off, &sl, &how);+ CHECK(off == 0, "from the start, %d lines: cut at %zu (%s)", end, off,+ how);+ free(pre);+ }+ srv_fim_free(f);++ /* 3. */+ f = srv_fim_new(0, 0);+ int start = -1, reads = 0;+ for (int top = 0; top < 120; top++) {+ pre = lines("a", top, top + 200, &n);+ srv_fim_window(f, by4, NULL, pre, n, "", 0, &off, &sl, &how);+ CHECK(off == 0 || pre[off - 1] == '\n', "slide %d: not at a line", top);+ int at = first_line(pre + off);+ if (at != start) {+ reads++;+ start = at;+ }+ if (top == 1)+ CHECK(at > 40 && at < 60 &&+ !strcmp(how, "ahead of a sliding window"),+ "first slide: line %d (%s)", at, how);+ CHECK(at >= top, "slide %d: line %d", top, at);+ free(pre);+ }+ /* starts at 0, 51, then every 50 lines or so: four or five readings+ of the whole window in 120 lines, not 120 */+ CHECK(reads <= 5, "120 slides read the window %d times", reads);+ srv_fim_free(f);++ /* 4. lines of 42 bytes, 11 tokens: a budget of 1000 takes 90 */+ f = srv_fim_new(1000, 0);+ start = -1;+ reads = 0;+ for (int end = 300; end < 400; end++) {+ pre = lines("a", 0, end, &n);+ srv_fim_window(f, by4, NULL, pre, n, "", 0, &off, &sl, &how);+ int32_t t = by4(NULL, pre + off, n - off);+ CHECK(t <= 1000 && (off == 0 || pre[off - 1] == '\n'),+ "%d lines: %d tokens from %zu", end, t, off);+ int at = first_line(pre + off);+ if (at != start) {+ CHECK(t <= 750 && t > 650, "%d lines: cut to %d tokens", end, t);+ reads++;+ start = at;+ }+ free(pre);+ }+ CHECK(reads <= 6, "100 lines more read the window %d times", reads);+ srv_fim_free(f);++ /* 5. */+ f = srv_fim_new(0, 0);+ int sa = -1, sb = -1;+ for (int top = 0; top < 6; top++) {+ pre = lines("a", top, top + 200, &n);+ srv_fim_window(f, by4, NULL, pre, n, "", 0, &off, &sl, &how);+ int a = first_line(pre + off);+ free(pre);+ pre = lines("b", top + 1000, top + 1200, &n);+ srv_fim_window(f, by4, NULL, pre, n, "", 0, &off, &sl, &how);+ int b = first_line(pre + off);+ free(pre);+ if (top >= 2) {+ CHECK(a == sa && b == sb, "two files, slide %d: %d %d, not %d %d",+ top, a, b, sa, sb);+ }+ sa = a;+ sb = b;+ }+ srv_fim_free(f);++ /* 6. */+ f = srv_fim_new(2048, 0);+ start = -1;+ reads = 0;+ for (int top = 0; top < 120; top++) {+ char *w = lines("a", top, top + 100, &n);+ pre = malloc(n + 2);+ pre[0] = '\n';+ memcpy(pre + 1, w, n + 1);+ free(w);+ srv_fim_window(f, by4, NULL, pre, n + 1, "", 0, &off, &sl, &how);+ CHECK(off >= 1 && pre[off - 1] == '\n' && pre[off] != '\n',+ "newline in front, slide %d: at %zu", top, off);+ int at = first_line(pre + off);+ if (at != start) {+ reads++;+ start = at;+ }+ CHECK(at >= top, "newline in front, slide %d: line %d", top, at);+ free(pre);+ }+ CHECK(reads <= 6, "newline in front: 120 slides read the window %d times",+ reads);+ srv_fim_free(f);++ if (failures) {+ printf("test_server_fim: %d failures\n", failures);+ return 1;+ }+ printf("test_server_fim: ok\n");+ return 0;+}diff --git a/tests/test_server_request.c b/tests/test_server_request.cindex 53fca7c..ac6866a 100644--- a/tests/test_server_request.c+++ b/tests/test_server_request.c@@ -13,6 +13,7 @@#include <string.h>#include "server/engine.c"+#include "server/fim.c"#include "server/request.c"#include "server/request_opts.c"@@ -190,7 +191,8 @@ static void invalid(void)"\"response_format\":{\"type\":\"xml\"}}",1, "invalid_value"},{"{\"messages\":[{\"role\":\"user\",\"content\":\"a\"}],"- "\"stop\":[\"a\",\"b\",\"c\",\"d\",\"e\"]}",+ "\"stop\":[\"a\",\"b\",\"c\",\"d\",\"e\",\"f\",\"g\",\"h\",\"i\","+ "\"j\",\"k\",\"l\",\"m\",\"n\",\"o\",\"p\",\"q\"]}",1, "invalid_value"},{"{\"messages\":[{\"role\":\"user\",\"content\":\"a\"}],""\"top_logprobs\":3}",@@ -274,11 +276,46 @@ static void mangled(void)accepted, total);}+/* /infill's n_indent (engine.c), fed a token at a time */+static void indent(void)+{+ static const struct {+ const char *tokens[8];+ int n_indent;+ const char *want;+ } t[] = {+ /* the body goes on, then a line less indented: cut before it */+ {{"x = 1;\n", " y", " = 2;\n", "}\n", "int z;"},+ 4,+ "x = 1;\n y = 2;\n"},+ /* blank lines do not end it */+ {{"a;\n", "\n", " b;\n", " c;"}, 4, "a;\n\n b;\n"},+ /* nothing less indented: all of it */+ {{"if (x)\n", " y();\n", " z();"},+ 4,+ "if (x)\n y();\n z();"},+ };+ for (size_t i = 0; i < sizeof(t) / sizeof(t[0]); i++) {+ struct srv_choice ch = {0};+ size_t nl = 0;+ int cut = 0;+ for (int k = 0; k < 8 && t[i].tokens[k] && !cut; k++) {+ grow(&ch.text, &ch.n_text, &ch.cap_text, t[i].tokens[k],+ strlen(t[i].tokens[k]));+ cut = indent_cut(&ch, t[i].n_indent, &nl);+ }+ CHECK(ch.text && strcmp(ch.text, t[i].want) == 0,+ "n_indent case %zu: '%s'", i, ch.text ? ch.text : "");+ free(ch.text);+ }+}+int main(void){valid();invalid();mangled();+ indent();printf("test_server_request: %s\n", failures ? "FAILED" : "ok");return failures ? 1 : 0;}