Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

Merge pull request #1353 from monotophic/api/server-surfaces

Vincenzo Fornaro1f9a85238dfc
diff --cc c/openai_server.py
index 9a1858b8,1e95c3b2..b4e093ed
--- a/c/openai_server.py
+++ b/c/openai_server.py
@@@ -2833,48 -3041,15 +3096,49 @@@ def _split_thinking(text, enable_thinki
thinking, answer = [], []
split = ThinkingStreamSplit(thinking.append, answer.append,
initial_thinking=starts_in_reasoning(enable_thinking,
- add_generation_prompt))
+ add_generation_prompt),
+ track_spans=track_spans)
split.feed(text)
split.finish()
- return "".join(thinking), "".join(answer)
+ return "".join(thinking), "".join(answer), (split.thinking_spans, split.text_spans)
-def _anthropic_block_text(blocks, param):
- """Text out of an Anthropic content array (tool_result content is the same shape)."""
+def _anthropic_image_part(block, where):
+ """An Anthropic `image` block as the OpenAI-shaped part expand_*_images() reads.
+
+ A base64 source becomes the same data: URI a client of /v1/chat/completions would
+ send, so both endpoints hand the expander one picture in one shape. A `url` source
+ is passed through as-is: the expander already refuses to fetch remote URLs, and
+ the Anthropic client must meet that refusal, not a second one."""
+ source = block.get("source")
+ if not isinstance(source, dict):
+ raise APIError(400, "`image` blocks require a `source` object.", f"{where}.source")
+ kind = source.get("type")
+ if kind == "base64":
+ media_type, data = source.get("media_type"), source.get("data")
+ if not isinstance(media_type, str) or not media_type:
+ raise APIError(400, "`image.source.media_type` is required for base64 sources.",
+ f"{where}.source.media_type")
+ if not isinstance(data, str) or not data:
+ raise APIError(400, "`image.source.data` is required for base64 sources.",
+ f"{where}.source.data")
+ return {"type": "image_url", "image_url": {"url": f"data:{media_type};base64,{data}"}}
+ if kind == "url":
+ url = source.get("url")
+ if not isinstance(url, str) or not url:
+ raise APIError(400, "`image.source.url` is required for url sources.",
+ f"{where}.source.url")
+ return {"type": "image_url", "image_url": {"url": url}}
+ raise APIError(400, "`image.source.type` must be base64 or url.", f"{where}.source.type",
+ "unsupported_content_type")
+
+
+def _anthropic_block_text(blocks, param, images=None):
+ """Text out of an Anthropic content array (tool_result content is the same shape).
+
+ With `images`, a list, `image` blocks are allowed and their OpenAI-shaped parts are
+ appended to it instead of raising: a tool_result can carry a picture (Claude Code's
+ Read on an image file does), and the caller decides which turn it rides on."""
if isinstance(blocks, str):
return blocks
if not isinstance(blocks, list):
diff --cc docs/api.md
index 13615f00,fd7467b7..d989d742
--- a/docs/api.md
+++ b/docs/api.md
@@@ -90,21 -90,14 +90,23 @@@ too. Both values render byte for byte l
The server serves one generation at a time: the model stays in one persistent
process, so concurrent HTTP requests queue instead of loading duplicate model
copies. Tool calling depends on the active engine; see the support matrix below.
- Images, log probabilities, and token penalties return an explicit error rather
- than being silently ignored. `seed` is accepted and ignored (see below).
- Audio is accepted only by Inkling checkpoints with
- audio support. The default bind address is localhost; set `COLI_API_KEY` before
- exposing the server beyond the machine.
+ Images and token penalties return an explicit error rather than being silently
+ ignored. The OpenAI-compatible endpoints request log probabilities only from a
+ glm engine (see below); on every other engine such a request is refused with a
+ named error, never silently ignored. `seed` is accepted and ignored (see
+ below). Audio is accepted only by Inkling checkpoints with audio support. The
+ default bind address is localhost; set `COLI_API_KEY` before exposing the
+ server beyond the machine.
+The hosted-platform bookkeeping fields `store`, `metadata`, `service_tier`,
+`user`, `safety_identifier`, `parallel_tool_calls`, `prompt_cache_key`,
+`verbosity`, `web_search_options`, `moderation`, and
+`stream_options.include_obfuscation` are accepted and intentionally ignored:
+they have no local equivalent and do not affect generation. Unsupported
+result-shaping requests are refused explicitly: `best_of` values above 1, a
+non-empty `logit_bias`, `suffix` infill, and audio output requested through
+`modalities`.
+
### `seed`
`seed` is accepted (not rejected) for OpenAI-API request-shape compatibility;