Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs(api): document logprobs, echo and top_logprobs

monotophic73dc1f5538d8
diff --git a/c/tests/golden_fixture_capture.py b/c/tests/golden_fixture_capture.py
index 9ae6424e..f966662f 100644
--- a/c/tests/golden_fixture_capture.py
+++ b/c/tests/golden_fixture_capture.py
@@ -21,9 +21,11 @@ Usage:
# 3. compare (exit 0 = byte-identical modulo normalized fields):
python3 tests/golden_fixture_capture.py diff fixtures_pre fixtures_post
-Battery: chat, chat+tools, chat streaming, completions without logprobs, and
-the error cases whose behavior must not move (seed 400, array-prompt 400,
-logprobs 400, out-of-range temperature 400) plus /v1/models.
+Battery: chat, chat+tools, chat streaming, completions without logprobs, the
+logprobs case now served on the glm engine (which a pre-change capture answered
+with a 400, so the two differ there by construction), and the error cases whose
+behavior must not move (seed 400, array-prompt 400, out-of-range temperature
+400) plus /v1/models.
Normalization: every "id"/"created" field (recursively, and per SSE event) is
replaced with a constant; nothing else is touched. Generation-bearing requests
@@ -80,7 +82,7 @@ def battery(model):
{"model": model, "prompt": "hello", "max_tokens": 1, "seed": 1234}),
("err_array_prompt", "POST", "/v1/completions",
{"model": model, "prompt": [1, 2, 3], "max_tokens": 1}),
- ("err_logprobs", "POST", "/v1/completions",
+ ("logprobs_served", "POST", "/v1/completions",
{"model": model, "prompt": "hello", "max_tokens": 1, "logprobs": 1}),
("err_bad_temperature", "POST", "/v1/chat/completions",
{"model": model, "messages": chat_messages,
diff --git a/docs/api.md b/docs/api.md
index aa337532..3b5dbc2f 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -78,10 +78,126 @@ sequences. The extension
The server serves one generation at a time: the model stays in one persistent
process, so concurrent HTTP requests queue instead of loading duplicate model
copies. Tool calling depends on the active engine; see the support matrix below.
-Images, log probabilities, and token penalties return an explicit error rather
-than being silently ignored. Audio is accepted only by Inkling checkpoints with
-audio support. The default bind address is localhost; set `COLI_API_KEY` before
-exposing the server beyond the machine.
+Images and token penalties return an explicit error rather than being silently
+ignored. The OpenAI-compatible endpoints request log probabilities only from a
+glm engine (see below); on every other engine such a request is refused with a
+named error, never silently ignored. Audio is accepted only by Inkling
+checkpoints with audio support. The default bind address is localhost; set
+`COLI_API_KEY` before exposing the server beyond the machine.
+
+### Log probabilities and prompt echo
+
+`/v1/completions` accepts the legacy integer `logprobs` (**0–32**; 0 means no log
+probabilities at all, see below; the upper bound is the engine's top-32 read-out
+interface, and anything above 32 is a named 400) and boolean `echo`;
+`/v1/chat/completions` accepts boolean `logprobs` plus integer `top_logprobs` (0–32) and
+returns `choices[].logprobs.content[]` (`{token, logprob, bytes, top_logprobs}` per
+generated token) β€” chat has no `echo` concept and rejects one with a 400. Each endpoint
+takes the OpenAI request shape for it and refuses the other's by name. A non-boolean
+`echo` is a named 400 (`invalid_value`) on both endpoints, independent of whether
+`logprobs` is requested at all. On chat, `top_logprobs` is type- and range-checked even
+when `logprobs` is false or absent, so a malformed `top_logprobs` is a named 400 whether
+or not the gate it would feed is open; a valid `top_logprobs` with `logprobs` off remains
+a documented no-op.
+
+The zero semantics are explicit, not a truthiness accident: on `/v1/completions`,
+`logprobs: 0`, `false` and `null` all mean **no log probabilities** (the request succeeds
+with `choices[].logprobs: null`, exactly as if the field were omitted), while boolean
+`true` is a named 400 β€” the legacy field is an integer count, and a boolean carries no
+count. On `/v1/chat/completions` the field is a boolean gate (`null` behaves like `false`;
+any integer is a named 400). `echo: null` normalises to absent on both endpoints, so a
+client that serialises its whole request model with nulls is never refused for a field it
+did not mean to set. `logprobs` together with `stream` is a named 400: per-delta log
+probabilities are not built.
+
+`/v1/completions` with `echo: true` returns the full legacy `logprobs` object (`tokens`,
+`token_logprobs`, `top_logprobs`, `text_offset`) covering the echoed prompt plus any
+generated tokens, and `text` itself is the reconstructed prompt followed by the completion
+(the standard OpenAI legacy behavior for `echo: true`) rather than the completion alone;
+`echo: true` without an active `logprobs` request is a named 400 (`echo` requires
+`logprobs`): the prompt echo is built out of the engine's per-token records, so without
+them there is nothing to echo, and a request that asks for one is told so rather than
+served without it. Echoing a prompt with no logprobs at all is offered as a separate
+proposal. `text_offset` is a character offset into
+that same returned `text` string, always counted from 0 β€” including when `echo` is false,
+where `text` holds only the completion and the offsets describe only that text, not a
+position within the (unreturned) prompt.
+
+`"".join(tokens)` is the returned `text`: where a `stop` sequence matches partway through
+a token, that token's `tokens` entry is truncated to the characters actually emitted (a
+`stop` of `"lo"` against a token decoding to `"Hello"` reports `"Hel"`), rather than being
+dropped or reporting characters the client did not receive. On chat, a truncated entry's
+`bytes` β€” and its `top_logprobs` entries' `bytes` β€” are truncated with it, to the UTF-8 of
+the characters that entry reports. An entry that was not truncated keeps its frame's own
+payload, which need not be the encoding of its `token`: a frame carrying only part of a
+multi-byte codepoint reports `token: ""` with that frame's bytes.
+
+The requested top-k table is **unsorted** on the wire β€” do not assume the first entry is
+the argmax. Per-token values are printed by the engine to six decimal digits of precision.
+Non-finite values (a degenerate all-`-inf` logit row, say) serialize as JSON `null`, never
+a clamped number.
+
+These endpoints request the numeric per-token channel only from a glm engine; on every
+other engine the request returns a named 400 rather than being silently ignored. That is a
+statement about what these endpoints request, and about nothing else.
+
+Known limitations, current build:
+
+- **Cost.** Requesting `logprobs` at all β€” completions or chat, `echo` or not β€” asks the
+ engine for a full read-out pass over the whole prompt, because the request carries one
+ opt-in and not a separate "echo" one. Such a request normally forfeits prefix-cache
+ reuse. There is no cap on prompt length for it, so a very long prompt pays a
+ correspondingly large one-shot cost.
+- **Waiting.** A logprobs request waits for the engine exactly as any other request waits.
+ An engine that never acknowledges the submission β€” one older than this extension, or one
+ whose parser disagrees about the header's shape β€” leaves the request waiting, as it
+ would through a cold prefill. There is no separate bounded wait for the opt-in. A request
+ whose per-token frames could not be read waits the same way: the turn belongs to the
+ engine until it sends a terminal frame, so the refusal below is delivered then rather
+ than the moment the unreadable frame arrives. The engine is sent one stop when that
+ happens, so the turn does not spend the rest of its budget on a response nobody receives.
+- **Cancellation.** A cancel may not take effect until the read-out this request asked for
+ has finished, so the window in which a disconnect goes unnoticed is wider on a long
+ prompt than it is without `logprobs`.
+- **Alternative-token labels.** `top_logprobs` entries for candidate token ids other than
+ the position's own actual token are not decoded text (no server-side tokenizer exists, by
+ design) β€” they are labeled `<token_id:N>`. Only the position's own token, identified by
+ an exact logprob match rather than by id, gets its real decoded text.
+- **The sampled token is not guaranteed to appear in its own `top_logprobs` table.** The
+ engine's numeric channel reports the top-k candidates by its own read-out; if the chosen
+ token falls outside that table, no entry represents it, and the response's
+ `token_logprobs`/`logprob` field is still the chosen token's own value read from the
+ frame directly, not looked up in the table.
+- **The arrays describe the text the client received, exactly, and nothing beyond it.**
+ Every generated token is located in the returned text by its span in the raw engine
+ stream, so the stage that consumed a character decides how much of that token's entry
+ reaches the array: a token a matched `stop` cut partway through reports the characters
+ that were emitted, and generated text that does not reach `message.content` β€” reasoning,
+ template markers, tool-call syntax, a swallowed role marker β€” is not described there.
+ Reasoning, template and tool-call tokens are therefore not described by
+ `logprobs.content`; a separate field for the full stream is proposed separately. On
+ every response that carries logprobs, `"".join(tokens)` is the returned `text` and
+ `"".join` over `logprobs.content` is `message.content`. When the engine's records cannot
+ support that β€” they stop short of the generated text, or none arrive at all β€” the
+ request is refused by name rather than answered with arrays describing a prefix.
+- **Engine-side faults are named, not generic.** When the engine's per-token records cannot
+ answer the request, the 5xx body carries a `code` saying what the engine did:
+ `engine_logprob_tail_malformed` (a numeric tail this server cannot read β€” it fails that
+ one request and no other), `engine_echo_position_malformed` (an unreadable prompt-echo
+ position, which is a different field of the same frame),
+ `engine_duplicate_logprob_candidate` (a top-k table repeating a candidate at one
+ position), `engine_logprob_records_incomplete` (records that stop
+ short of the generated text, none at all, or β€” under `echo` β€” no prompt-echo records,
+ any of which would leave the arrays describing a prefix), and
+ `engine_pinned_prefix_not_echoed` (a KV slot holding a pin snapshot, so the echoed
+ positions would not start at 0). No logprobs-bearing response is ever a truncated or
+ half-described 200.
+- **Server-side buffering.** A request that asks to see the echoed prompt (`echo: true` on
+ `/v1/completions`) has its full echo table held in memory for the request's lifetime; no
+ streaming is allowed together with `logprobs`, so that hold is bounded by one
+ non-streaming response. A logprobs request that does not ask for the echo β€” all of chat,
+ and every `echo: false` completion β€” retains nothing: the engine still sends an ECHO frame
+ for every prompt position, and they are dropped as they arrive rather than held.
### Tool-calling support