Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

v4: max_tokens is a ceiling — clamp to context instead of rejecting (#975)

JustVugg8c40fbddc74d
diff --git a/c/coli b/c/coli
index 8d968f45..1e0127d6 100755
--- a/c/coli
+++ b/c/coli
@@ -1003,7 +1003,7 @@ def chat_attached(a, base, model_id):
its KV slots reuse the conversation prefix, so a continued chat skips
re-prefill too. The engine byte-protocol stays untouched — this is plain
OpenAI SSE over localhost, stdlib only."""
- import urllib.request
+ import urllib.request, urllib.error
print(f" {C.grn}✦ attached{C.r} {C.dim}to {base} · model {model_id} · the engine stays warm after you quit{C.r}")
print(f" {C.dim}type and press Enter · Ctrl-C stops the answer · :reset starts a new conversation · :q exits{C.r}\n")
msgs=[]
@@ -1048,6 +1048,20 @@ def chat_attached(a, base, model_id):
md.feed(txt); reply.append(txt)
except KeyboardInterrupt:
interrupted=True # il server annulla la richiesta alla disconnessione
+ except urllib.error.HTTPError as e:
+ # HTTPError IS-A OSError: without this arm a 4xx/5xx fell into the
+ # branch below and read as "server unreachable" — hiding the API's
+ # own explanation of what was wrong with the request (#975: a V4
+ # CONTEXT_EXCEEDED 400 took a screenshot to diagnose instead of a
+ # sentence). The body is OpenAI-shaped; show its message.
+ sp.stop()
+ try:
+ detail=json.loads(e.read().decode("utf-8","replace")).get("error",{}).get("message","")
+ except Exception:
+ detail=""
+ print(f"\n {C.yel}[the server rejected the request: HTTP {e.code}"
+ f"{' — '+detail if detail else ''}]{C.r}")
+ msgs.pop(); continue # la richiesta non è mai partita: togli il turno e riprova
except OSError as e:
sp.stop()
print(f"\n {C.yel}[server unreachable: {e}]{C.r}"); break
diff --git a/c/deepseek_v4.c b/c/deepseek_v4.c
index fe94d5c3..d2bd6d5f 100644
--- a/c/deepseek_v4.c
+++ b/c/deepseek_v4.c
@@ -8742,7 +8742,9 @@ static void v4_serve_one(ColiV4Engine *engine, ColiV4Session *session,
session->max_prompt_tokens + 16);
int context = engine->runtime.context_tokens;
if (prompt_count < 1 || prompt_count > session->max_prompt_tokens ||
- prompt_count + request->max_tokens > context) {
+ prompt_count + 1 > context) {
+ /* The PROMPT does not fit (or leaves no room for a single generated
+ * token) -- that is the only honest CONTEXT_EXCEEDED. */
char message[256];
snprintf(message, sizeof(message),
"CONTEXT_EXCEEDED prompt_tokens=%d requested=%d capacity=%d",
@@ -8750,6 +8752,19 @@ static void v4_serve_one(ColiV4Engine *engine, ColiV4Session *session,
v4_serve_error(request->id, message);
return;
}
+ /* max_tokens is a CEILING, not a target (#260/#382): generation ends at
+ * EOS either way, so an oversized budget is clamped to what the context
+ * can hold -- the same semantics the GLM serve path has had since #260.
+ * Rejecting instead made `coli chat`'s interactive default (16384) a
+ * guaranteed 400 on every first message of a fresh V4 chat (#975). */
+ if (request->max_tokens > context - prompt_count) {
+ fprintf(stderr,
+ "[V4] max_tokens %d clamped to %d (context %d - prompt %d); "
+ "raise CTX for longer answers\n",
+ request->max_tokens, context - prompt_count, context,
+ prompt_count);
+ request->max_tokens = context - prompt_count;
+ }
printf("ACCEPT %s %d\n", request->id, prompt_count);
fflush(stdout);
diff --git a/docs/deepseek-v4.md b/docs/deepseek-v4.md
index 415f610f..bc1251b8 100644
--- a/docs/deepseek-v4.md
+++ b/docs/deepseek-v4.md
@@ -71,6 +71,10 @@ python ./coli serve --model /path/to/DeepSeek-V4-Flash --ram 32
python ./coli web --model /path/to/DeepSeek-V4-Flash --ram 32
```
+Generation length: `--ngen` is a ceiling, not a target — answers end at EOS.
+If the ceiling exceeds what the context window can hold, the engine clamps it
+and says so on stderr; raise `CTX` for genuinely longer answers.
+
V4 chat uses native model markers. Native serving currently supports greedy
generation and one active KV slot; tools and grammar are rejected. Requests
re-prefill their context, while the process, weights, dense tensors, head, and