Skip to content

armgit / janas

Public Imported from GitHub
0

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

Release 2026-10-01: code completion in the editor

camauri6476bbc69520
diff --git a/ChangeLog.md b/ChangeLog.md
index 2b2e45f..20ad08b 100644
--- a/ChangeLog.md
+++ b/ChangeLog.md
@@ -2,6 +2,28 @@
> Curated, user-facing summary of completed work, newest first.
+## [2026-10-01] - Code completion in the editor
+
+- **`janas-server` fills in code at an editor's cursor**, with a model for
+ code such as Qwen3-Coder-30B-A3B-Instruct, through OpenAI's
+ `/v1/completions` with a `suffix` and through llama.cpp's **`/infill`**,
+ which the extensions made for llama.cpp ask (llama.vscode): the line up
+ to the cursor, `n_predict` (0: the prompt only), `n_indent` and
+ `t_max_predict_ms` read as llama.cpp reads them, the answer as
+ llama.cpp's. Tried in VSCodium with llama.vscode and with Continue;
+ how to set them up in [docs/code-completion.md](docs/code-completion.md).
+- **The window sent at every keystroke is kept from moving.** The
+ extensions send "the last N lines before the cursor", whose start moves
+ at every new line, and the whole window was read again: 17 s for 1,330
+ tokens at every request. The start of each window is now remembered and
+ kept, put ahead when the editor slides its window, the text before the
+ cursor cut at a line within `--fim-prefix` tokens (2048), the lines after
+ it within `--fim-suffix` (8), the text written within `--fim-max` (256):
+ 1.5-2.2 s a proposal after the first request of a window, on a laptop
+ with Qwen3-Coder-30B-A3B in memory.
+- **Sixteen stop strings** a request, where OpenAI takes four: Continue
+ sends more, and every one of its requests was refused.
+
## [2026-10-01] - janas-flights, the first of Janas's services
- **janas-flights**, flights as an MCP server of its own, for `janas-chat`
diff --git a/MODELS.md b/MODELS.md
index a800260..7809901 100644
--- a/MODELS.md
+++ b/MODELS.md
@@ -103,6 +103,14 @@ b46fedd33e0bfb0cae308aa3c158d0a4b2c4a1d2185a1ed6f093cdaf39064772 Qwen_Qwen3.6-3
0276b0697a05075fec8ab076a757205537804ff41dd7e9dadace7c6993763563 qwen3-coder-next-q4km.jns (jns_planes)
```
+**Qwen3-Coder-30B-A3B-Instruct**, for code completion, from
+[`unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF`](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF):
+
+```
+fadc3e5f8d42bf7e894a785b05082e47daee4df26680389817e2093056f088ad Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
+bee7d25509e911a195cbf05a157e69bb05a62d2241cca2c4fdcfef269dcb2bdd qwen3-coder-30b-a3b-q4km.jns (gguf2jns)
+```
+
**Qwen3-4B**, from [`Qwen/Qwen3-4B-GGUF`](https://huggingface.co/Qwen/Qwen3-4B-GGUF).
This one is dense: it has no experts to route, so its feed-forward becomes the
single slot of each layer, and half its layers keep a `down` matrix in Q4_K,
diff --git a/README.md b/README.md
index d5a96e3..32e5118 100644
--- a/README.md
+++ b/README.md
@@ -130,8 +130,9 @@ report from `janas-bench` is the single most useful thing you can send back**
that clients written for it can use a model running on this machine:
chat and completions with tools, JSON output held to a schema and
log-probabilities, the Responses API with its conversations, embeddings
- and moderations, and a chat page of its own to try it from a browser
- ([below](#using-it-over-http)).
+ and moderations, a chat page of its own to try it from a browser
+ ([below](#using-it-over-http)), and code completion for VSCodium and VS
+ Code ([docs/code-completion.md](docs/code-completion.md)).
- **`janas-bench`:** measures the machine, fills its profile and writes a report.
- **Faster replies** from drafts the model verifies, so the text is exactly the
one it would have written: from the model's own multi-token prediction block
@@ -211,6 +212,7 @@ the machine to itself.
- [MODELS.md](MODELS.md) — what a converted model's licence is, and what may be redistributed
- [CONTRIBUTING.md](CONTRIBUTING.md) — how to help, and the sign-off
- [docs/services](docs/services/README.md) — Janas's services (flights), in `janas-chat` and in other MCP clients
+- [docs/code-completion.md](docs/code-completion.md) — code completion in VSCodium and VS Code, through `janas-server`
- [AUTHORS](AUTHORS) — who wrote it, and what it owes to others
- [LICENSE](LICENSE) — GNU GPL, version 3 or later
@@ -257,6 +259,7 @@ that is where these examples put them, and where the project keeps its own.
| **Qwen3-Next-80B-A3B-Instruct** — most of this engine was measured on it | [Qwen](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct-GGUF) | 48.4 GB | 48.4 GB | 32 GB |
| **Qwen3.6-35B-A3B** — the easiest to start with | [bartowski](https://huggingface.co/bartowski/Qwen_Qwen3.6-35B-A3B-GGUF) | 22.3 GB | 22.3 GB | 16-32 GB |
| **Qwen3-Coder-Next** | [Qwen](https://huggingface.co/Qwen/Qwen3-Coder-Next-GGUF) | 48.4 GB | 48.4 GB | 32 GB |
+| **Qwen3-Coder-30B-A3B-Instruct** — for code completion in the editor ([docs/code-completion.md](docs/code-completion.md)) | [unsloth](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF) | 18.6 GB | 18.6 GB | 32 GB, to keep it whole in memory |
| **Qwen3-30B-A3B** | [Qwen](https://huggingface.co/Qwen/Qwen3-30B-A3B-GGUF) | 18.6 GB | 18.6 GB | 16 GB |
| **Qwen3.5-9B** — dense; its prediction block from the checkpoint ([MODELS.md](MODELS.md#the-fingerprints)) | [unsloth](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF) | 5.7 GB | 5.7 GB | 16 GB |
| **Qwen3.5-2B** — dense, small and quick; a prediction block too | [unsloth](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF) | 1.3 GB | 1.3 GB | 8 GB |
@@ -669,6 +672,17 @@ estimate has not yet been checked on a run that long.
`janas-server` takes `--metrics`, `--progress`, `--max-input` and
`--warn-input`.
+## Filling in code
+
+`janas-server` fills in code at an editor's cursor, with a model for code
+such as Qwen3-Coder-30B-A3B: OpenAI's `/v1/completions` with a `suffix`,
+and llama.cpp's `/infill`, which llama.vscode asks. The window sent at
+every keystroke is kept from moving, so that a keystroke reads only what
+it changed: 1.5-2.2 s a proposal on a laptop, where a window slid a line
+was read again in 17 s. How to set it up in VSCodium and VS Code, with
+llama.vscode or Continue, and what was measured:
+[docs/code-completion.md](docs/code-completion.md).
+
## Using it from a program
The library is shaped for foreign-function interfaces from the start: opaque
@@ -835,6 +849,7 @@ model running on this machine. It takes the model options `janas-chat` takes
| `--test` | serve a chat page for trying it out, on `/test` (below) |
| `--max-input N` | the longest input a request may have, in tokens, below the context: a longer one gets a 400 (`input_limit_exceeded`) whose error says `input_tokens`, `max_input_tokens`, `excess_tokens` and the parts, and nothing is ever cut to fit |
| `--warn-input N` | longer inputs are taken, and said on stderr with their parts |
+| `--fim-prefix N`, `--fim-suffix N`, `--fim-max N` | completions with a `suffix`, an editor filling in code at its cursor: the tokens before the cursor read at most (2048; 0: all) and the lines after it (8; 0: all); `--fim-max N` the tokens written there at most, whatever the request asks (256; 0: as asked), since an editor asks for thousands and the requests of the next keystrokes wait for them. Over the budget the text before the cursor is cut at a line to three quarters of it, and the start of the window is kept across keystrokes, also when the editor slides its window, so that a keystroke reads only what it changed (see [Filling in code](#filling-in-code)) |
| `--metrics FILE`, `--progress S` | JSON lines of what every reply costs, and progress lines while a long prompt is read (see [Watching a long prompt](#watching-a-long-prompt)); `--verbose` prints progress every 30 s |
| `--no-mcp` | refuse tools of type `mcp` in `/v1/responses`: no connection leaves the server on a request's word (below) |
| `--keep N` | conversations kept computed besides the one in use (8; 0: none), so that clients taking turns, or a client's requests on the side for titles and tags, do not have their conversations read again from the start |
@@ -855,6 +870,7 @@ What it answers:
| `POST /v1/chat/completions` | a conversation, the reply whole or streamed (`"stream": true`, server-sent events), with tools, JSON output, log-probabilities and more than one reply (below) |
| `GET`, `POST`, `DELETE /v1/chat/completions/{id}`, `GET /v1/chat/completions`, `.../messages` | the completions a client asked to keep (`"store": true`): read, listed with their `metadata`, changed, deleted |
| `POST /v1/completions` | raw text to continue, no chat format around it; several prompts, token ids, `echo`, `logprobs`, `best_of`, `suffix` |
+| `POST /infill` | llama.cpp's route for filling in code, which editors' extensions made for llama.cpp ask (llama.vscode): `input_prefix`, `prompt`, `input_suffix`, `n_predict` (0: the prompt only, read for the next request), `n_indent`, `t_max_predict_ms`, the answer as llama.cpp's (`content`, `timings`); `input_extra` is not used, and the most likely token is always taken |
| `POST /v1/responses` and the six operations under it | OpenAI's newer API: items in and out, `previous_response_id`, conversations, streamed events, `background` and `cancel`, `input_items`, `input_tokens`, `compact` |
| `/v1/conversations` and its seven operations | conversations the server keeps, and their items |
| `POST /v1/embeddings` | with `--embedding-model FILE`: the vectors of texts (below) |
diff --git a/docs/code-completion.md b/docs/code-completion.md
new file mode 100644
index 0000000..af31ed3
--- /dev/null
+++ b/docs/code-completion.md
@@ -0,0 +1,178 @@
+# Code completion in the editor
+
+`janas-server` fills in code at the cursor of an editor, as the
+completion extensions of VS Code and VSCodium ask for it: the code before
+the cursor and the code after it go to the model, which writes what goes
+between. It runs on the processor, with a model for code; nothing leaves
+the machine.
+
+## The model
+
+A model trained to fill in the middle is needed, with the
+`<|fim_prefix|>`, `<|fim_suffix|>` and `<|fim_middle|>` tokens of the Qwen
+coders. Every Qwen3-Coder is, says its
+[README](https://github.com/QwenLM/Qwen3-Coder). The one tried is
+**Qwen3-Coder-30B-A3B-Instruct**, Q4_K_M (unsloth's GGUF, converted with
+`gguf2jns`): 18.6 GB, a mixture of experts with 3 billion parameters
+active a token, which on a 32 GB machine fits in memory whole
+(`--cache 18`).
+
+Qwen3-Coder-Next fills in too, but its recurrent layers cannot go back to
+a point of a sequence: every keystroke would read the window again from
+its start.
+
+## Starting the server
+
+```sh
+janas-server models/jns/qwen3-coder-30b-a3b-q4km.jns --cache 18
+```
+
+It listens on `http://127.0.0.1:8080`. `--verbose` writes on stderr, for
+each request, where its window was cut and why.
+
+## The two routes
+
+- **`POST /v1/completions` with a `suffix`**, OpenAI's: the `prompt` is
+ the code before the cursor, the `suffix` the code after it.
+- **`POST /infill`**, llama.cpp's, which the extensions made for
+ llama.cpp's server ask: `input_prefix` (the lines before the cursor),
+ `prompt` (the line up to it), `input_suffix` (what follows), and the
+ answer as llama.cpp gives it (`content`, `timings`). It reads
+ `n_predict` (0: the prompt only, read so that the next request finds it
+ computed), `n_indent` (a new line indented less ends the text) and
+ `t_max_predict_ms` (after a new line, past that time the text ends), as
+ llama.cpp does. The code of other files (`input_extra`) is not used: it
+ comes first, changes often, and every change would have the whole
+ prompt read again. The most likely token is always taken, so the same
+ place gives the same proposal.
+
+## The window
+
+An editor asks at every keystroke. The server keeps computed what the
+previous request shares from its start, so what matters is that the
+start of the text sent does not move; and the extensions move it, sending
+"the last N lines before the cursor". So, for requests that fill in:
+
+- the code before the cursor is read up to `--fim-prefix` tokens (2048);
+ over that it is cut at a line to three quarters of it, which leaves
+ room for many lines before the next cut;
+- the start of each window is remembered, for a few files, and a later
+ request holding it is cut there;
+- a request whose text starts inside the head of a window before, past
+ its start, comes from an editor that slides its window: its start is put
+ a quarter of the way in, so that the lines it slides next keep it in;
+- the code after the cursor, read again at every request, is cut after
+ `--fim-suffix` lines (8);
+- the text written is at most `--fim-max` tokens (256), whatever the
+ request asks: the extensions ask for thousands, and the requests of the
+ next keystrokes would wait for them.
+
+Each cut costs one reading of the whole window; the keystrokes after it
+read only what they changed and the lines after the cursor.
+
+## In VSCodium (and VS Code): llama.vscode
+
+[llama.vscode](https://github.com/ggml-org/llama.vscode), by the authors of
+llama.cpp, is on Open VSX, where VSCodium finds it. In the settings:
+
+```json
+{
+ "llama-vscode.endpoint": "http://127.0.0.1:8080",
+ "llama-vscode.max_parallel_completions": 1,
+ "llama-vscode.ring_n_chunks": 0,
+ "llama-vscode.rag_enabled": false,
+ "editor.quickSuggestions": {"other": "off", "comments": "off", "strings": "off"},
+ "editor.inlineSuggest.suppressSuggestions": true
+}
+```
+
+One proposal is written (`max_parallel_completions`), the chunks of other
+files would not be used (`ring_n_chunks`), and the editor's own list of
+suggestions is kept from opening by itself: while it is open, a proposal
+that does not extend its selected entry is not shown. Its welcome page
+offers to install llama.cpp: it is not needed.
+
+Tab takes the proposal, Shift+Tab its first line, Ctrl+Right its first
+word; Ctrl+L asks for one.
+
+## Continue
+
+[Continue](https://www.continue.dev) works too, with three things to know:
+the provider must be one that sends the suffix (`siliconflow`, whose
+requests go to `{apiBase}completions` with `prompt` and `suffix`: the
+`openai` provider builds the prompt itself and sends it as raw text), its
+default `modelTimeout` of 150 ms cuts a proposal after ten pieces, and the
+snippets of other files it puts in front of the prefix change at every
+request. In `~/.continue/config.yaml`:
+
+```yaml
+name: Janas
+version: 1.0.0
+schema: v1
+models:
+ - name: Janas Coder
+ provider: siliconflow
+ model: qwen3-coder-30b-a3b-q4km
+ apiBase: http://127.0.0.1:8080/v1/ # the final slash is needed
+ apiKey: none
+ roles:
+ - autocomplete
+ autocompleteOptions:
+ maxPromptTokens: 2048
+ prefixPercentage: 0.85
+ maxSuffixPercentage: 0.1
+ debounceDelay: 500
+ modelTimeout: 20000
+ multilineCompletions: never
+ useCache: false
+ useRecentlyEdited: false
+ useRecentlyOpened: false
+ useImports: false
+ experimental_includeRecentlyVisitedRanges: false
+ experimental_includeRecentlyEditedRanges: false
+ experimental_includeDiff: false
+```
+
+With Continue some proposals the server wrote were not shown, and the
+editor was not seen to ask again after some keystrokes; why was not found
+(Continue filters, caches and reuses proposals before showing them).
+llama.vscode's behaviour was easier to follow.
+
+## What has been run
+
+1 October 2026, Qwen3-Coder-30B-A3B-Instruct Q4_K_M on an Intel Core
+Ultra 9 185H with every expert in memory (`--cache 18`), VSCodium 1.135
+with llama.vscode 0.0.67 and Continue 2.0.0.
+
+The window, with requests as an editor makes them on the 1,445 lines of
+`src/chat/main.c`, the cursor going down a line a request, time to the
+first token:
+
+| Requests | first request | the ones after |
+|---|---|---|
+| before the window: the 110 lines before the cursor, slid a line (1,330 tokens) | 17 s | 17 s: the whole window read again |
+| before the window: the file from its start to the cursor (5,300 tokens) | 81 s | 7 s |
+| now: the 200 lines before the cursor, slid a line (cut to 1,527 tokens) | 19 s | 1.5-2.2 s |
+| now: the file from its start to the cursor (cut to 1,566 tokens) | 20 s | 1.6-1.8 s |
+
+In VSCodium with llama.vscode, writing in `src/flights/schedule.c`: after
+the first request of a window, each request found 680-800 of its 750-870
+tokens computed and was answered in one to two seconds (about 90 tokens
+read, the lines after the cursor and what had changed, and 2-30
+written).
+
+What it proposed:
+
+- under the comment "how many answers are kept and still fresh", the body
+ of a new function counting the answers kept, with the names of the file
+ (`kept[]`, `N_KEPT`); with `time_t now = time(NULL);` written first, it
+ added the test on their age (`now - kept[i].at < KEEP_S`) by itself;
+- for a line taken out of `local_time` (`tm.tm_mon -= 1;`), the right line
+ first (`tm.tm_mon--;`), and then lines of its own: a handling of time
+ zones the file does not have. With 8, 16, 32 or 64 lines after the
+ cursor given to it the first line was always right and the rest always
+ invented: the model does not see that only one line was missing.
+ Shift+Tab takes the first line only.
+
+The first request of a new window reads all of it, at 65-80 tokens a
+second: 10-20 s for the 750-1,500 tokens of a window.
diff --git a/src/server/engine.c b/src/server/engine.c
index e665049..86e5971 100644
--- a/src/server/engine.c
+++ b/src/server/engine.c
@@ -13,6 +13,7 @@
* prompt, which is then read once where the model can go back over it.
*/
#include "server.h"
+#include "fim.h"
#include <stdio.h>
#include <stdlib.h>
@@ -31,6 +32,8 @@ struct srv_engine {
int stop;
int verbose;
uint32_t max_input, warn_input;
+ struct srv_fim *fim; /* the windows of fill-in-the-middle requests */
+ uint32_t fim_max; /* the tokens they write at most; 0: as asked */
char desc[1024];
char system[1024]; /* the system message Janas's programs use */
int64_t created;
@@ -313,6 +316,36 @@ static int32_t ids_text(struct srv_engine *e, struct srv_prompt *pp)
return pp->text ? JANAS_LLM_OK : JANAS_LLM_EINVAL;
}
+/* The tokens of a text, for the window of a fill-in-the-middle request. */
+static int32_t count_tokens(void *ctx, const char *s, size_t n)
+{
+ int32_t t = 0;
+ if (n > INT32_MAX)
+ return -1;
+ int32_t rc = janas_llm_tokenize(ctx, s, (int32_t)n, 0, NULL, 0, &t);
+ return rc == JANAS_LLM_OK || rc == JANAS_LLM_ESMALL ? t : -1;
+}
+
+/* The prompt of a request with a suffix: the window around the cursor
+ (fim.c), so that the next keystroke finds it computed. */
+static int32_t load_infill(struct srv_engine *e, struct srv_job *j,
+ const struct srv_prompt *pp)
+{
+ size_t off = 0, sl = j->n_suffix;
+ const char *how = "";
+ if (e->fim)
+ srv_fim_window(e->fim, count_tokens, e->llm, pp->text, pp->n, j->suffix,
+ j->n_suffix, &off, &sl, &how);
+ if (e->verbose && (off || sl < j->n_suffix))
+ fprintf(stderr,
+ "janas-server: fill in the middle: prefix from byte %zu of "
+ "%zu%s%s, suffix %zu of %zu bytes\n",
+ off, pp->n, *how ? ", " : "", how, sl, j->n_suffix);
+ return janas_llm_chat_infill(e->chat, pp->text + off,
+ (int32_t)(pp->n - off), j->suffix,
+ (int32_t)sl);
+}
+
/* The request's prompt for reply k. */
static int32_t load(struct srv_engine *e, struct srv_job *j, int k)
{
@@ -324,10 +357,8 @@ static int32_t load(struct srv_engine *e, struct srv_job *j, int k)
if (rc != JANAS_LLM_OK)
return rc;
}
- return j->suffix
- ? janas_llm_chat_infill(c, pp->text, (int32_t)pp->n,
- j->suffix, (int32_t)j->n_suffix)
- : janas_llm_chat_prompt(c, pp->text, (int32_t)pp->n);
+ return j->suffix ? load_infill(e, j, pp)
+ : janas_llm_chat_prompt(c, pp->text, (int32_t)pp->n);
}
int32_t rc = janas_llm_chat_begin(c);
for (int32_t i = 0; rc == JANAS_LLM_OK && i < j->n_msg; i++) {
@@ -340,6 +371,36 @@ static int32_t load(struct srv_engine *e, struct srv_job *j, int k)
return rc == JANAS_LLM_OK ? janas_llm_chat_run(c) : rc;
}
+/* /infill's n_indent: a line of the text started past *nl indented less
+ than n_indent (blank lines aside) ends it there, as llama.cpp does. 1
+ when cut. */
+static int indent_cut(struct srv_choice *ch, int n_indent, size_t *nl)
+{
+ for (;;) {
+ if (*nl > 0) {
+ size_t pos = *nl;
+ int ind = 0;
+ while (pos < ch->n_text &&
+ (ch->text[pos] == ' ' || ch->text[pos] == '\t')) {
+ ind++;
+ pos++;
+ }
+ if (pos < ch->n_text && ch->text[pos] != '\n' &&
+ ch->text[pos] != '\r' && ind < n_indent) {
+ ch->n_text = *nl;
+ ch->text[*nl] = 0;
+ return 1;
+ }
+ }
+ const char *p = *nl < ch->n_text
+ ? memchr(ch->text + *nl, '\n', ch->n_text - *nl)
+ : NULL;
+ if (!p)
+ return 0;
+ *nl = (size_t)(p - ch->text) + 1;
+ }
+}
+
/* One reply, into choice k. Returns the library's code. */
static int32_t run_one(struct srv_engine *e, struct srv_job *j, int k)
{
@@ -349,6 +410,9 @@ static int32_t run_one(struct srv_engine *e, struct srv_job *j, int k)
if (j->seeded)
p.seed += (uint64_t)k;
p.max_input = e->max_input; /* the server's limit, not the request's */
+ if (j->suffix && e->fim_max &&
+ (p.max_reply <= 0 || p.max_reply > (int32_t)e->fim_max))
+ p.max_reply = (int32_t)e->fim_max;
int32_t rc = janas_llm_chat_set_params(c, &p);
if (rc == JANAS_LLM_OK)
rc = load(e, j, k);
@@ -357,6 +421,8 @@ static int32_t run_one(struct srv_engine *e, struct srv_job *j, int k)
if (strlen(j->stop[i]) > max_stop)
max_stop = strlen(j->stop[i]);
int32_t seen = 0;
+ size_t nl = 0; /* n_indent: where the last line started */
+ struct timespec t0 = {0}; /* t_max_predict_ms: the first token */
while (rc == JANAS_LLM_OK) {
char buf[512];
int32_t len = 0;
@@ -380,6 +446,19 @@ static int32_t run_one(struct srv_engine *e, struct srv_job *j, int k)
ch->text[at] = 0;
ch->by_stop = 1;
}
+ if (!bad && j->n_indent > 0 && indent_cut(ch, j->n_indent, &nl))
+ ch->by_stop = 1;
+ if (!bad && j->t_max_predict_ms > 0) {
+ struct timespec now;
+ clock_gettime(CLOCK_MONOTONIC, &now);
+ if (!t0.tv_sec && !t0.tv_nsec)
+ t0 = now;
+ double ms = (double)(now.tv_sec - t0.tv_sec) * 1e3 +
+ (double)(now.tv_nsec - t0.tv_nsec) / 1e6;
+ if (ms > j->t_max_predict_ms &&
+ memchr(ch->text, '\n', ch->n_text))
+ ch->by_stop = 1;
+ }
}
int quit = j->cancel || ch->by_stop || bad || rc == JANAS_LLM_DONE;
pthread_cond_broadcast(&j->cv);
@@ -677,8 +756,11 @@ struct srv_engine *srv_engine_start(const struct srv_config *cfg, char *err,
janas_llm_describe(e->llm, e->desc, sizeof(e->desc), &n);
janas_llm_default_system(e->llm, e->system, sizeof(e->system), &n);
e->created = (int64_t)time(NULL);
+ e->fim = srv_fim_new(cfg->fim_prefix, cfg->fim_suffix);
+ e->fim_max = cfg->fim_max;
if (pthread_create(&e->th, NULL, engine_main, e) != 0) {
snprintf(err, err_len, "cannot start the engine's thread");
+ srv_fim_free(e->fim);
janas_llm_chat_destroy(e->chat);
janas_llm_close(e->llm);
free(e);
@@ -712,6 +794,7 @@ void srv_engine_stop(struct srv_engine *e)
janas_llm_chat_destroy(e->chat);
janas_llm_close(e->llm);
janas_llm_close(e->emb);
+ srv_fim_free(e->fim);
pthread_mutex_destroy(&e->mu);
pthread_cond_destroy(&e->cv);
free(e);
diff --git a/src/server/fim.c b/src/server/fim.c
new file mode 100644
index 0000000..2bb0544
--- /dev/null
+++ b/src/server/fim.c
@@ -0,0 +1,216 @@
+/* SPDX-License-Identifier: GPL-3.0-or-later
+ Copyright (C) 2026 Maurizio Cammalleri */
+/*
+ * fim.c - the window of a fill-in-the-middle request.
+ *
+ * An editor asks at every keystroke, with the text before the cursor and
+ * after it. The session reuses what the last request shares from the
+ * start, so the start of the prefix must not move. An editor that sends
+ * "the last N lines" moves it at every new line, and the whole prefix was
+ * read again - 81 s for 5300 tokens, Qwen3-Coder-30B on a laptop, against
+ * 7 s for one keystroke more.
+ *
+ * So the head of each window is remembered (a few, for a few files):
+ * - a later prefix holding the start of one is cut there;
+ * - a prefix starting inside the head of one, past its start, comes from
+ * an editor that slides its window: it is cut a quarter of the way in,
+ * so that the next lines it slides keep that start in;
+ * - a prefix over the budget is cut to three quarters of it, at a line,
+ * which leaves room for many lines before the next cut.
+ * Each cut costs one reading of the whole window, and the keystrokes after
+ * it read only what they changed.
+ *
+ * The suffix comes after the prefix and is read again at every request:
+ * only its first lines are kept.
+ */
+#include "fim.h"
+
+#include <stdlib.h>
+#include <string.h>
+
+#define N_HEADS 8 /* windows remembered: a few files */
+#define HEAD_MAX 4096 /* bytes of a window's head remembered */
+#define START_MAX 160 /* of them, the bytes a start is matched by */
+#define START_MIN 24 /* shorter ones match too easily */
+
+struct head {
+ char s[HEAD_MAX];
+ size_t n;
+ uint64_t used;
+};
+
+struct srv_fim {
+ uint32_t prefix_tokens, suffix_lines;
+ uint64_t clock;
+ struct head h[N_HEADS];
+};
+
+struct srv_fim *srv_fim_new(uint32_t prefix_tokens, uint32_t suffix_lines)
+{
+ struct srv_fim *f = calloc(1, sizeof(*f));
+ if (f) {
+ f->prefix_tokens = prefix_tokens;
+ f->suffix_lines = suffix_lines;
+ }
+ return f;
+}
+
+void srv_fim_free(struct srv_fim *f)
+{
+ free(f);
+}
+
+/* The start of the line at off or the first after it; without one, off
+ itself moved past the bytes that continue a UTF-8 character. */
+static size_t line_from(const char *p, size_t n, size_t off)
+{
+ if (off == 0 || off >= n)
+ return off < n ? off : n;
+ if (p[off - 1] == '\n')
+ return off;
+ const char *nl = memchr(p + off, '\n', n - off);
+ if (nl && (size_t)(nl - p) + 1 < n)
+ return (size_t)(nl - p) + 1;
+ while (off < n && ((unsigned char)p[off] & 0xC0) == 0x80)
+ off++;
+ return off;
+}
+
+/* Where the k bytes of s are at the start of a line of t (n bytes), from
+ the line at or after from on: the first place, or SIZE_MAX. */
+static size_t at_line(const char *t, size_t n, size_t from, const char *s,
+ size_t k)
+{
+ for (size_t o = line_from(t, n, from); o < n;) {
+ if (n - o >= k && memcmp(t + o, s, k) == 0)
+ return o;
+ const char *nl = memchr(t + o, '\n', n - o);
+ if (!nl)
+ break;
+ o = (size_t)(nl - t) + 1;
+ }
+ return SIZE_MAX;
+}
+
+static size_t start_len(const struct head *h)
+{
+ return h->n < START_MAX ? h->n : START_MAX;
+}
+
+/* The prefix cut at a line so that its tokens are want at most, from off
+ on: the new offset. */
+static size_t cut_to(srv_fim_count count, void *ctx, const char *pre, size_t n,
+ size_t off, int32_t t, uint32_t want)
+{
+ for (int tries = 0; tries < 16 && t > (int32_t)want; tries++) {
+ size_t rest = n - off;
+ /* the bytes in proportion, a little less each try */
+ size_t keep = (size_t)((double)rest * want / t * (1.0 - tries / 16.0));
+ size_t to = line_from(pre, n, n - keep);
+ if (to <= off)
+ to = line_from(pre, n, off + 1);
+ off = to;
+ if (off >= n)
+ break;
+ t = count(ctx, pre + off, n - off);
+ if (t < 0)
+ break;
+ }
+ return off;
+}
+
+void srv_fim_window(struct srv_fim *f, srv_fim_count count, void *ctx,
+ const char *pre, size_t n_pre, const char *suf,
+ size_t n_suf, size_t *pre_off, size_t *suf_len,
+ const char **how)
+{
+ *how = "";
+ *suf_len = n_suf;
+ if (f->suffix_lines) {
+ uint32_t lines = 0;
+ for (size_t i = 0; i < n_suf; i++)
+ if (suf[i] == '\n' && ++lines == f->suffix_lines) {
+ *suf_len = i + 1;
+ break;
+ }
+ }
+
+ /* blank lines in front are left out: Continue joins its snippets of
+ other files to the prefix with a newline, and sends one even with
+ none, which put each window's start at the end of a line */
+ size_t lead = 0;
+ while (lead < n_pre && (pre[lead] == '\n' || pre[lead] == '\r'))
+ lead++;
+ pre += lead;
+ n_pre -= lead;
+
+ /* the start of a window before, the one used last if several */
+ int hit = -1;
+ size_t off = 0;
+ for (int i = 0; i < N_HEADS; i++) {
+ if (!f->h[i].n)
+ continue;
+ size_t o = at_line(pre, n_pre, 0, f->h[i].s, start_len(&f->h[i]));
+ if (o != SIZE_MAX && (hit < 0 || f->h[i].used > f->h[hit].used)) {
+ hit = i;
+ off = o;
+ }
+ }
+ if (off)
+ *how = "at the start of the window before";
+
+ /* none: does this prefix start inside the head of one, past its start?
+ Then the editor slides its window, and the start is put ahead */
+ size_t k = n_pre < START_MAX ? n_pre : START_MAX;
+ if (hit < 0 && k >= START_MIN)
+ for (int i = 0; i < N_HEADS; i++)
+ if (f->h[i].n > k &&
+ at_line(f->h[i].s, f->h[i].n, 1, pre, k) != SIZE_MAX) {
+ off = line_from(pre, n_pre, n_pre / 4);
+ *how = "ahead of a sliding window";
+ break;
+ }
+
+ if (f->prefix_tokens && count) {
+ int32_t t = count(ctx, pre + off, n_pre - off);
+ uint32_t want = f->prefix_tokens / 4 * 3;
+ if (t > (int32_t)f->prefix_tokens) {
+ off = cut_to(count, ctx, pre, n_pre, off, t, want);
+ *how = "over the budget";
+ hit = -1;
+ }
+ }
+ /* a start that would leave less than half of the text, and under a
+ budget less than three eighths of it, is too far (the cursor went
+ up): the window starts again */
+ int32_t kept = f->prefix_tokens && count && hit >= 0
+ ? count(ctx, pre + off, n_pre - off)
+ : 0;
+ if (hit >= 0 && off > n_pre / 2 &&
+ (!f->prefix_tokens || kept < (int32_t)(f->prefix_tokens / 8 * 3))) {
+ off = 0;
+ *how = "";
+ hit = -1;
+ int32_t t = f->prefix_tokens && count ? count(ctx, pre, n_pre) : 0;
+ if (t > (int32_t)f->prefix_tokens) {
+ off =
+ cut_to(count, ctx, pre, n_pre, 0, t, f->prefix_tokens / 4 * 3);
+ *how = "over the budget";
+ }
+ }
+
+ size_t rest = n_pre - off;
+ size_t len = rest / 2 < HEAD_MAX ? rest / 2 : HEAD_MAX;
+ if (hit >= 0) {
+ f->h[hit].used = ++f->clock;
+ } else if (len >= START_MIN) {
+ int slot = 0;
+ for (int i = 1; i < N_HEADS; i++)
+ if (f->h[i].used < f->h[slot].used)
+ slot = i;
+ memcpy(f->h[slot].s, pre + off, len);
+ f->h[slot].n = len;
+ f->h[slot].used = ++f->clock;
+ }
+ *pre_off = lead + off;
+}
diff --git a/src/server/fim.h b/src/server/fim.h
new file mode 100644
index 0000000..42768f5
--- /dev/null
+++ b/src/server/fim.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-3.0-or-later
+ Copyright (C) 2026 Maurizio Cammalleri */
+/*
+ * fim.h - the window of a fill-in-the-middle request (fim.c): the part of
+ * the text before the cursor and after it that the model reads, chosen so
+ * that the next keystroke finds most of it already computed.
+ */
+#ifndef JANAS_SERVER_FIM_H
+#define JANAS_SERVER_FIM_H
+
+#include <stddef.h>
+#include <stdint.h>
+
+struct srv_fim;
+
+/* The tokens of n bytes of s, or -1. */
+typedef int32_t (*srv_fim_count)(void *ctx, const char *s, size_t n);
+
+/* prefix_tokens: the text before the cursor read at most (0: all of it);
+ suffix_lines: the lines after it (0: all of them). */
+struct srv_fim *srv_fim_new(uint32_t prefix_tokens, uint32_t suffix_lines);
+void srv_fim_free(struct srv_fim *f);
+
+/* The window: the prefix from *pre_off on, the first *suf_len bytes of the
+ suffix. *how says why the prefix was cut ("" when it was not). */
+void srv_fim_window(struct srv_fim *f, srv_fim_count count, void *ctx,
+ const char *pre, size_t n_pre, const char *suf,
+ size_t n_suf, size_t *pre_off, size_t *suf_len,
+ const char **how);
+
+#endif
diff --git a/src/server/http.c b/src/server/http.c
index 3319c61..b643472 100644
--- a/src/server/http.c
+++ b/src/server/http.c
@@ -288,6 +288,10 @@ static enum MHD_Result dispatch(struct srv_req *r)
r, 401, "invalid_request_error", "invalid_api_key",
"Incorrect or missing API key: send it as "
"\"Authorization: Bearer <key>\".");
+ /* llama.cpp's route for filling in code, which editors' extensions
+ made for it ask: not OpenAI's, so not in routes.inc */
+ if (strcmp(r->path, "/infill") == 0 && strcmp(r->method, "POST") == 0)
+ return (enum MHD_Result)srv_infill(r);
const struct route *other = NULL;
for (size_t i = 0; i < sizeof(routes) / sizeof(routes[0]); i++) {
const struct route *rt = &routes[i];
diff --git a/src/server/infill.c b/src/server/infill.c
new file mode 100644
index 0000000..2f1bce6
--- /dev/null
+++ b/src/server/infill.c
@@ -0,0 +1,197 @@
+/* SPDX-License-Identifier: GPL-3.0-or-later
+ Copyright (C) 2026 Maurizio Cammalleri */
+/*
+ * infill.c - POST /infill, llama.cpp's route for filling in code, which
+ * the editors' extensions made for it ask (llama.vscode, llama.vim): the
+ * lines before the cursor (input_prefix), the line up to it (prompt), and
+ * what follows (input_suffix), into the same window as a completion with
+ * a suffix (fim.c), the answer as llama.cpp gives it (content, timings).
+ *
+ * Read as llama.cpp reads them: n_predict (0: the prompt only, read so
+ * that the next request finds it computed), n_indent (a new line indented
+ * less ends the text), t_max_predict_ms (after a new line, past that time
+ * the text ends). The extra context of other files (input_extra) is not
+ * used: it changes often, comes first, and every change would have the
+ * whole prompt read again. The samplers are not either: the most likely
+ * token is taken, so that the same place gives the same proposal.
+ */
+#include "request.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+static const char *str_of(yyjson_val *o, const char *k, size_t *n, int *bad)
+{
+ yyjson_val *v = yyjson_obj_get(o, k);
+ if (!v || yyjson_is_null(v)) {
+ *n = 0;
+ return NULL;
+ }
+ if (!yyjson_is_str(v)) {
+ *bad = 1;
+ return NULL;
+ }
+ *n = yyjson_get_len(v);
+ return yyjson_get_str(v);
+}
+
+static long num_of(yyjson_val *o, const char *k, long dflt, int *bad)
+{
+ yyjson_val *v = yyjson_obj_get(o, k);
+ if (!v || yyjson_is_null(v))
+ return dflt;
+ if (!yyjson_is_num(v)) {
+ *bad = 1;
+ return dflt;
+ }
+ return (long)yyjson_get_num(v);
+}
+
+/* The answer, as llama.cpp's: the fields the extensions read. */
+static char *answer(const struct srv_req *r, const struct srv_job *j, size_t *n)
+{
+ const struct srv_choice *ch = &j->ch[0];
+ const struct janas_llm_chat_stats *st = &j->st;
+ const char *text = j->prefill_only || !ch->text ? "" : ch->text;
+ double in_ms = st->input_seconds * 1000.0,
+ out_ms = st->output_seconds * 1000.0;
+ uint32_t out_n = j->prefill_only ? 0 : st->output_tokens;
+ const char *stop_type =
+ ch->finish == JANAS_LLM_FINISH_LENGTH ? "limit" : "eos";
+ yyjson_mut_doc *d = yyjson_mut_doc_new(NULL);
+ if (!d)
+ return NULL;
+ yyjson_mut_val *o = yyjson_mut_obj(d);
+ yyjson_mut_doc_set_root(d, o);
+ yyjson_mut_obj_add_strncpy(d, o, "content", text, strlen(text));
+ yyjson_mut_obj_add_strcpy(d, o, "model", r->srv->cfg.model_id);
+ yyjson_mut_obj_add_bool(d, o, "stop", true);
+ yyjson_mut_obj_add_strcpy(d, o, "stop_type", stop_type);
+ yyjson_mut_obj_add_uint(d, o, "tokens_predicted", out_n);
+ yyjson_mut_obj_add_uint(d, o, "tokens_evaluated", st->prompt_tokens);
+ yyjson_mut_obj_add_uint(d, o, "tokens_cached", st->context_used);
+ yyjson_mut_obj_add_bool(d, o, "truncated", false);
+ yyjson_mut_val *gs = yyjson_mut_obj_add_obj(d, o, "generation_settings");
+ yyjson_mut_obj_add_uint(d, gs, "n_ctx", st->context_size);
+ yyjson_mut_obj_add_strcpy(d, gs, "model", r->srv->cfg.model_id);
+ yyjson_mut_val *t = yyjson_mut_obj_add_obj(d, o, "timings");
+ yyjson_mut_obj_add_uint(d, t, "cache_n", st->cached_tokens);
+ yyjson_mut_obj_add_uint(d, t, "prompt_n", st->input_tokens);
+ yyjson_mut_obj_add_real(d, t, "prompt_ms", in_ms);
+ yyjson_mut_obj_add_real(d, t, "prompt_per_second",
+ in_ms > 0 ? st->input_tokens * 1000.0 / in_ms : 0);
+ yyjson_mut_obj_add_uint(d, t, "predicted_n", out_n);
+ yyjson_mut_obj_add_real(d, t, "predicted_ms", out_ms);
+ yyjson_mut_obj_add_real(d, t, "predicted_per_second",
+ out_ms > 0 ? out_n * 1000.0 / out_ms : 0);
+ char *json = yyjson_mut_write(d, 0, n);
+ yyjson_mut_doc_free(d);
+ return json;
+}
+
+int srv_infill(struct srv_req *r)
+{
+ yyjson_doc *doc = yyjson_read(r->body, r->n_body, 0);
+ yyjson_val *root = doc ? yyjson_doc_get_root(doc) : NULL;
+ if (!root || !yyjson_is_obj(root)) {
+ yyjson_doc_free(doc);
+ return srv_reply_error(r, 400, "invalid_request_error", NULL,
+ "The body must be a JSON object.");
+ }
+ int bad = 0;
+ size_t n_pre = 0, n_line = 0, n_suf = 0;
+ const char *pre = str_of(root, "input_prefix", &n_pre, &bad);
+ const char *line = str_of(root, "prompt", &n_line, &bad);
+ const char *suf = str_of(root, "input_suffix", &n_suf, &bad);
+ long n_predict = num_of(root, "n_predict", -1, &bad);
+ long n_indent = num_of(root, "n_indent", 0, &bad);
+ long t_max = num_of(root, "t_max_predict_ms", 0, &bad);
+ yyjson_val *ex = yyjson_obj_get(root, "input_extra");
+ yyjson_val *sm = yyjson_obj_get(root, "stream");
+ const char *why = NULL;
+ if (bad)
+ why = "'input_prefix', 'input_suffix' and 'prompt' must be strings, "
+ "'n_predict', 'n_indent' and 't_max_predict_ms' numbers.";
+ else if (!pre || !suf)
+ why = "'input_prefix' and 'input_suffix' are required.";
+ else if (ex && !yyjson_is_null(ex) && !yyjson_is_arr(ex))
+ why = "'input_extra' must be an array of {\"filename\": string, "
+ "\"text\": string}.";
+ else if (sm && yyjson_is_bool(sm) && yyjson_get_bool(sm))
+ why = "'stream' is not supported on /infill: the answer comes whole.";
+ if (why) {
+ yyjson_doc_free(doc);
+ return srv_reply_error(r, 400, "invalid_request_error", "invalid_value",
+ "%s", why);
+ }
+
+ struct srv_job *j = srv_job_new();
+ if (!j || srv_job_choices(j, 1, 1) != 0 ||
+ !(j->prompts = calloc(1, sizeof(*j->prompts)))) {
+ yyjson_doc_free(doc);
+ srv_job_release(j);
+ return srv_reply_error(r, 500, "server_error", NULL, "out of memory");
+ }
+ j->kind = JOB_TEXT;
+ j->n_prompts = 1;
+ struct srv_prompt *pp = &j->prompts[0];
+ pp->n = n_pre + n_line;
+ pp->text = malloc(pp->n + 1);
+ j->suffix = srv_dup_n(suf, n_suf);
+ j->n_suffix = n_suf;
+ if (!pp->text || !j->suffix) {
+ yyjson_doc_free(doc);
+ srv_job_release(j);
+ return srv_reply_error(r, 500, "server_error", NULL, "out of memory");
+ }
+ memcpy(pp->text, pre, n_pre);
+ if (n_line)
+ memcpy(pp->text + n_pre, line, n_line);
+ pp->text[pp->n] = 0;
+ if (r->srv->cfg.verbose && ex && yyjson_arr_size(ex))
+ fprintf(stderr,
+ "janas-server: /infill: %zu chunks of other files "
+ "left out\n",
+ yyjson_arr_size(ex));
+ yyjson_doc_free(doc);
+
+ j->p.temperature = 0;
+ j->p.thinking = 0;
+ /* n_predict 0: the prompt only, which costs one token written */
+ j->prefill_only = n_predict == 0;
+ j->p.max_reply = n_predict == 0 ? 1
+ : n_predict > 0 ? (int32_t)n_predict
+ : 0;
+ j->n_indent = n_indent > 0 ? (int)n_indent : 0;
+ j->t_max_predict_ms = t_max > 0 ? (int32_t)t_max : 0;
+
+ if (srv_engine_submit(r->srv->engine, j) != 0) {
+ srv_job_release(j);
+ return srv_reply_error(r, 429, "rate_limit_exceeded", NULL,
+ "The server is busy: %u requests are waiting. "
+ "Try again shortly.",
+ r->srv->cfg.max_queue);
+ }
+ pthread_mutex_lock(&j->mu);
+ while (!j->done)
+ pthread_cond_wait(&j->cv, &j->mu);
+ pthread_mutex_unlock(&j->mu);
+ int ret;
+ if (j->err != JANAS_LLM_OK || j->ch[0].finish == JANAS_LLM_FINISH_ERROR) {
+ unsigned code =
+ j->err == JANAS_LLM_EFULL || j->err == JANAS_LLM_ELIMIT ||
+ j->err == JANAS_LLM_EINVAL || j->err == JANAS_LLM_EMODEL
+ ? 400
+ : 500;
+ ret = srv_reply_error(
+ r, code, code == 400 ? "invalid_request_error" : "server_error",
+ NULL, "%s", j->errmsg[0] ? j->errmsg : "generation failed");
+ } else {
+ size_t n = 0;
+ char *json = answer(r, j, &n);
+ ret = srv_reply_json(r, 200, json, n);
+ }
+ srv_job_release(j);
+ return ret;
+}
diff --git a/src/server/main.c b/src/server/main.c
index 4e073fe..2575f0a 100644
--- a/src/server/main.c
+++ b/src/server/main.c
@@ -67,6 +67,16 @@ static void usage(FILE *f)
" below the context: longer ones get a 400 that\n"
" says by how much (nothing is ever cut)\n"
" --warn-input N longer inputs are taken, and said on stderr\n"
+ " --fim-prefix N completions with a suffix (an editor filling in\n"
+ " code): the tokens before the cursor read at\n"
+ " most, cut at a line and from a start kept\n"
+ " across keystrokes (default 2048; 0: all)\n"
+ " --fim-suffix N the lines after the cursor read (default 8;\n"
+ " 0: all)\n"
+ " --fim-max N the tokens written there at most, whatever the\n"
+ " request asks (default 256; 0: as asked): an\n"
+ " editor asks for thousands, and the requests\n"
+ " of the next keystrokes wait for them\n"
" --metrics FILE JSON lines of what every reply costs: prompt,\n"
" progress of a long one, first token, the end\n"
" (counters and times, never text)\n"
@@ -123,6 +133,9 @@ int main(int argc, char **argv)
c->max_queue = 16;
c->keep = 8;
c->keep_disk = (uint64_t)8 << 30;
+ c->fim_prefix = 2048;
+ c->fim_suffix = 8;
+ c->fim_max = 256;
int no_store = 0;
c->max_body = (size_t)32 << 20;
c->max_connections = 64;
@@ -195,6 +208,12 @@ int main(int argc, char **argv)
c->max_input = (uint32_t)atol(arg(argc, argv, &i));
} else if (!strcmp(a, "--warn-input")) {
c->warn_input = (uint32_t)atol(arg(argc, argv, &i));
+ } else if (!strcmp(a, "--fim-prefix")) {
+ c->fim_prefix = (uint32_t)atol(arg(argc, argv, &i));
+ } else if (!strcmp(a, "--fim-suffix")) {
+ c->fim_suffix = (uint32_t)atol(arg(argc, argv, &i));
+ } else if (!strcmp(a, "--fim-max")) {
+ c->fim_max = (uint32_t)atol(arg(argc, argv, &i));
} else if (!strcmp(a, "--metrics")) {
setenv("JANAS_METRICS", arg(argc, argv, &i), 1);
} else if (!strcmp(a, "--progress")) {
diff --git a/src/server/request_opts.c b/src/server/request_opts.c
index e2aede3..56daee6 100644
--- a/src/server/request_opts.c
+++ b/src/server/request_opts.c
@@ -53,9 +53,9 @@ static int read_stop(struct srv_perr *e, struct srv_parsed *pr, yyjson_val *st,
} else if (st && yyjson_is_arr(st)) {
size_t idx, max;
yyjson_val *s;
- if (yyjson_arr_size(st) > 4)
- return srv_pfail(
- pr, BAD("invalid_value", "'stop' holds at most four strings."));
+ if (yyjson_arr_size(st) > SRV_MAX_STOP)
+ return srv_pfail(pr, BAD("invalid_value",
+ "'stop' holds at most sixteen strings."));
yyjson_arr_foreach(st, idx, max, s)
{
if (!yyjson_is_str(s))
diff --git a/src/server/server.h b/src/server/server.h
index 0b5cc42..b9c84b2 100644
--- a/src/server/server.h
+++ b/src/server/server.h
@@ -52,8 +52,16 @@ struct srv_config {
no connection goes out on a request's word */
uint32_t max_input; /* longest prompt taken, tokens; 0: the context */
uint32_t warn_input; /* prompts longer are taken, and said on stderr */
+ uint32_t fim_prefix; /* fill in the middle: the tokens before the
+ cursor read at most (0: all) */
+ uint32_t fim_suffix; /* the lines after it (0: all) */
+ uint32_t fim_max; /* the tokens it writes at most (0: as asked) */
};
+/* Stop strings a request may give: OpenAI takes four, editors' extensions
+ send more (Continue up to sixteen). */
+#define SRV_MAX_STOP 16
+
/* The engine: the model, its conversations, and the queue. */
struct srv_engine;
@@ -122,9 +130,15 @@ struct srv_job {
int n_prompts;
char *suffix;
size_t n_suffix;
+ /* /infill, as llama.cpp: a new line indented less than n_indent ends
+ the text; past t_max_predict_ms, so does a new line; prefill_only:
+ the prompt is read, nothing is answered */
+ int n_indent, prefill_only;
+ int32_t t_max_predict_ms;
struct janas_llm_chat_params p;
- int seeded; /* the request gave a seed: choice k takes seed + k */
- char *stop[4]; /* strings that end the reply where they appear */
+ int seeded; /* the request gave a seed: choice k takes seed + k */
+ char *stop[SRV_MAX_STOP]; /* strings that end the reply where they
+ appear */
int n_stop;
/* tools (a JSON array, as the request has it), tool_choice, and the
form of the answer */
@@ -246,6 +260,7 @@ int srv_embeddings_create(struct srv_req *r);
int srv_moderations_create(struct srv_req *r);
int srv_chat_create(struct srv_req *r);
int srv_completion_create(struct srv_req *r);
+int srv_infill(struct srv_req *r); /* POST /infill, llama.cpp's */
/* stored.c: the completions kept with store: true */
int srv_stored_list(struct srv_req *r);
int srv_stored_get(struct srv_req *r);
diff --git a/tests/test_server_fim.c b/tests/test_server_fim.c
new file mode 100644
index 0000000..9708f40
--- /dev/null
+++ b/tests/test_server_fim.c
@@ -0,0 +1,186 @@
+/* SPDX-License-Identifier: GPL-3.0-or-later
+ Copyright (C) 2026 Maurizio Cammalleri */
+/*
+ * test_server_fim.c - the window of a fill-in-the-middle request
+ * (src/server/fim.c), with a token every four bytes:
+ * 1. the suffix is cut after its lines;
+ * 2. an editor that sends the prefix from the start of the file keeps
+ * its start;
+ * 3. an editor that slides a window of 200 lines: the start is put a
+ * quarter ahead at its first slide, and kept for the slides after;
+ * 4. a prefix over the budget is cut at a line to three quarters of it,
+ * and the start kept while the prefix grows;
+ * 5. two files in turn keep a start each;
+ * 6. as 3, with a newline in front of every prefix, as Continue sends.
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include "server/fim.c"
+
+static int failures;
+
+#define CHECK(c, ...) \
+ do { \
+ if (!(c)) { \
+ printf(__VA_ARGS__); \
+ printf("\n"); \
+ failures++; \
+ } \
+ } while (0)
+
+static int32_t by4(void *ctx, const char *s, size_t n)
+{
+ (void)ctx, (void)s;
+ return (int32_t)((n + 3) / 4);
+}
+
+/* lines from..to-1 of a file named name, into a buffer of its own */
+static char *lines(const char *name, int from, int to, size_t *n)
+{
+ size_t cap = (size_t)(to - from) * 64 + 1;
+ char *b = malloc(cap), *p = b;
+ for (int i = from; i < to; i++)
+ p += snprintf(p, cap - (size_t)(p - b),
+ "%s line %04d: x = f(x, %d); /* code */\n", name, i, i);
+ *n = (size_t)(p - b);
+ return b;
+}
+
+/* the line number a window starts at */
+static int first_line(const char *s)
+{
+ const char *p = strstr(s, " line ");
+ return p ? atoi(p + 6) : -1;
+}
+
+int main(void)
+{
+ alarm(60);
+ size_t off, sl, n;
+ const char *how;
+
+ /* 1. */
+ struct srv_fim *f = srv_fim_new(0, 8);
+ char *suf = lines("s", 0, 20, &n);
+ char *pre = lines("a", 0, 10, &n);
+ srv_fim_window(f, by4, NULL, pre, n, suf, strlen(suf), &off, &sl, &how);
+ size_t want = 0;
+ for (int i = 0; i < 8; i++)
+ want += strchr(suf + want, '\n') - (suf + want) + 1;
+ CHECK(sl == want, "suffix %zu bytes, not %zu", sl, want);
+ free(suf);
+ free(pre);
+ srv_fim_free(f);
+
+ /* 2. */
+ f = srv_fim_new(0, 0);
+ for (int end = 100; end < 160; end += 7) {
+ pre = lines("a", 0, end, &n);
+ srv_fim_window(f, by4, NULL, pre, n, "", 0, &off, &sl, &how);
+ CHECK(off == 0, "from the start, %d lines: cut at %zu (%s)", end, off,
+ how);
+ free(pre);
+ }
+ srv_fim_free(f);
+
+ /* 3. */
+ f = srv_fim_new(0, 0);
+ int start = -1, reads = 0;
+ for (int top = 0; top < 120; top++) {
+ pre = lines("a", top, top + 200, &n);
+ srv_fim_window(f, by4, NULL, pre, n, "", 0, &off, &sl, &how);
+ CHECK(off == 0 || pre[off - 1] == '\n', "slide %d: not at a line", top);
+ int at = first_line(pre + off);
+ if (at != start) {
+ reads++;
+ start = at;
+ }
+ if (top == 1)
+ CHECK(at > 40 && at < 60 &&
+ !strcmp(how, "ahead of a sliding window"),
+ "first slide: line %d (%s)", at, how);
+ CHECK(at >= top, "slide %d: line %d", top, at);
+ free(pre);
+ }
+ /* starts at 0, 51, then every 50 lines or so: four or five readings
+ of the whole window in 120 lines, not 120 */
+ CHECK(reads <= 5, "120 slides read the window %d times", reads);
+ srv_fim_free(f);
+
+ /* 4. lines of 42 bytes, 11 tokens: a budget of 1000 takes 90 */
+ f = srv_fim_new(1000, 0);
+ start = -1;
+ reads = 0;
+ for (int end = 300; end < 400; end++) {
+ pre = lines("a", 0, end, &n);
+ srv_fim_window(f, by4, NULL, pre, n, "", 0, &off, &sl, &how);
+ int32_t t = by4(NULL, pre + off, n - off);
+ CHECK(t <= 1000 && (off == 0 || pre[off - 1] == '\n'),
+ "%d lines: %d tokens from %zu", end, t, off);
+ int at = first_line(pre + off);
+ if (at != start) {
+ CHECK(t <= 750 && t > 650, "%d lines: cut to %d tokens", end, t);
+ reads++;
+ start = at;
+ }
+ free(pre);
+ }
+ CHECK(reads <= 6, "100 lines more read the window %d times", reads);
+ srv_fim_free(f);
+
+ /* 5. */
+ f = srv_fim_new(0, 0);
+ int sa = -1, sb = -1;
+ for (int top = 0; top < 6; top++) {
+ pre = lines("a", top, top + 200, &n);
+ srv_fim_window(f, by4, NULL, pre, n, "", 0, &off, &sl, &how);
+ int a = first_line(pre + off);
+ free(pre);
+ pre = lines("b", top + 1000, top + 1200, &n);
+ srv_fim_window(f, by4, NULL, pre, n, "", 0, &off, &sl, &how);
+ int b = first_line(pre + off);
+ free(pre);
+ if (top >= 2) {
+ CHECK(a == sa && b == sb, "two files, slide %d: %d %d, not %d %d",
+ top, a, b, sa, sb);
+ }
+ sa = a;
+ sb = b;
+ }
+ srv_fim_free(f);
+
+ /* 6. */
+ f = srv_fim_new(2048, 0);
+ start = -1;
+ reads = 0;
+ for (int top = 0; top < 120; top++) {
+ char *w = lines("a", top, top + 100, &n);
+ pre = malloc(n + 2);
+ pre[0] = '\n';
+ memcpy(pre + 1, w, n + 1);
+ free(w);
+ srv_fim_window(f, by4, NULL, pre, n + 1, "", 0, &off, &sl, &how);
+ CHECK(off >= 1 && pre[off - 1] == '\n' && pre[off] != '\n',
+ "newline in front, slide %d: at %zu", top, off);
+ int at = first_line(pre + off);
+ if (at != start) {
+ reads++;
+ start = at;
+ }
+ CHECK(at >= top, "newline in front, slide %d: line %d", top, at);
+ free(pre);
+ }
+ CHECK(reads <= 6, "newline in front: 120 slides read the window %d times",
+ reads);
+ srv_fim_free(f);
+
+ if (failures) {
+ printf("test_server_fim: %d failures\n", failures);
+ return 1;
+ }
+ printf("test_server_fim: ok\n");
+ return 0;
+}
diff --git a/tests/test_server_request.c b/tests/test_server_request.c
index 53fca7c..ac6866a 100644
--- a/tests/test_server_request.c
+++ b/tests/test_server_request.c
@@ -13,6 +13,7 @@
#include <string.h>
#include "server/engine.c"
+#include "server/fim.c"
#include "server/request.c"
#include "server/request_opts.c"
@@ -190,7 +191,8 @@ static void invalid(void)
"\"response_format\":{\"type\":\"xml\"}}",
1, "invalid_value"},
{"{\"messages\":[{\"role\":\"user\",\"content\":\"a\"}],"
- "\"stop\":[\"a\",\"b\",\"c\",\"d\",\"e\"]}",
+ "\"stop\":[\"a\",\"b\",\"c\",\"d\",\"e\",\"f\",\"g\",\"h\",\"i\","
+ "\"j\",\"k\",\"l\",\"m\",\"n\",\"o\",\"p\",\"q\"]}",
1, "invalid_value"},
{"{\"messages\":[{\"role\":\"user\",\"content\":\"a\"}],"
"\"top_logprobs\":3}",
@@ -274,11 +276,46 @@ static void mangled(void)
accepted, total);
}
+/* /infill's n_indent (engine.c), fed a token at a time */
+static void indent(void)
+{
+ static const struct {
+ const char *tokens[8];
+ int n_indent;
+ const char *want;
+ } t[] = {
+ /* the body goes on, then a line less indented: cut before it */
+ {{"x = 1;\n", " y", " = 2;\n", "}\n", "int z;"},
+ 4,
+ "x = 1;\n y = 2;\n"},
+ /* blank lines do not end it */
+ {{"a;\n", "\n", " b;\n", " c;"}, 4, "a;\n\n b;\n"},
+ /* nothing less indented: all of it */
+ {{"if (x)\n", " y();\n", " z();"},
+ 4,
+ "if (x)\n y();\n z();"},
+ };
+ for (size_t i = 0; i < sizeof(t) / sizeof(t[0]); i++) {
+ struct srv_choice ch = {0};
+ size_t nl = 0;
+ int cut = 0;
+ for (int k = 0; k < 8 && t[i].tokens[k] && !cut; k++) {
+ grow(&ch.text, &ch.n_text, &ch.cap_text, t[i].tokens[k],
+ strlen(t[i].tokens[k]));
+ cut = indent_cut(&ch, t[i].n_indent, &nl);
+ }
+ CHECK(ch.text && strcmp(ch.text, t[i].want) == 0,
+ "n_indent case %zu: '%s'", i, ch.text ? ch.text : "");
+ free(ch.text);
+ }
+}
+
int main(void)
{
valid();
invalid();
mangled();
+ indent();
printf("test_server_request: %s\n", failures ? "FAILED" : "ok");
return failures ? 1 : 0;
}