Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
docs: GLiNER2.5-Decide
JustVugge481f128a316
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex b5828fc2..bddd1805 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -570,6 +570,14 @@ Read **only** by `c/laya.c`, the decision engine ([laya.md](laya.md)).| `COLI_LAYA_MAX_LEN` | the checkpoint's `max_len` | Tokens per question sequence (512 on the English checkpoint, capped at the encoder's 8192 positions). || `COLI_LAYA_HEAD_MAX_LEN` | the checkpoint's `head_max_len` | Tokens shared by a question's instructions and options; raise it for questions with many options. |+## GLiNER2.5-Decide engine (`gliner_decide`)++Read **only** by `c/gliner_decide.c`, the decision engine ([gliner_decide.md](gliner_decide.md)).++| Variable | Default | Effect |+|---|---|---|+| `COLI_GLINER_MAX_LEN` | `4096` | Tokens of the one sequence a request becomes: every question with its options, then the state. The state is cut at the last whole word that fits; questions that alone exceed it are refused with a 422. |+---## Server / CLI (`openai_server.py`, `coli`)diff --git a/docs/api.md b/docs/api.mdindex b41fa19f..1654215d 100644--- a/docs/api.md+++ b/docs/api.md@@ -25,7 +25,8 @@ Implemented endpoints are `GET /v1/models`, `GET /v1/models/{model}`,`POST /v1/chat/completions`, legacy `POST /v1/completions`, `POST /v1/brio`(closed-set scoring, [systemone.md](systemone.md)) and `POST /v1/systemone`, therequest and reply of TypeSafe's Jev API served by the same channel, or answered-natively by a decision model such as [Laya](laya.md). Chat and+natively by a decision model such as [Laya](laya.md) or+[GLiNER2.5-Decide](gliner_decide.md). Chat andcompletion requests support JSON responses, SSE streaming, usage counts,`max_tokens`/`max_completion_tokens`, `temperature`, `top_p`, and up to fourcustom `stop` sequences. `max_tokens` is a ceiling, not a target: when thediff --git a/docs/docs.json b/docs/docs.jsonindex 4590577b..004c3115 100644--- a/docs/docs.json+++ b/docs/docs.json@@ -23,7 +23,7 @@},{"group": "Model engines",- "pages": ["deepseek-v4", "kimi_k3", "qwen38", "qwen36", "inkling", "mimo", "qwen-image", "laya"]+ "pages": ["deepseek-v4", "kimi_k3", "qwen38", "qwen36", "inkling", "mimo", "qwen-image", "laya", "gliner_decide"]},{"group": "Configure and tune",diff --git a/docs/gliner_decide.md b/docs/gliner_decide.mdnew file mode 100644index 00000000..5cb4b35e--- /dev/null+++ b/docs/gliner_decide.md@@ -0,0 +1,219 @@+# GLiNER2.5-Decide on colibri++[GLiNER2.5-Decide](https://huggingface.co/fastino/GLiNER2.5-Decide), by+fastino (Apache-2.0), is a classifier for operational decisions: intent,+routing, sentiment, urgency, moderation, yes/no questions over a passage,+ordinal scores. It is a GLiNER2 checkpoint, a DeBERTa-v3-large encoder with+GLiNER2's heads, and it never generates text.++`c/gliner_decide` runs it from the official checkpoint with no conversion,+and `coli serve` exposes it on `POST /v1/systemone`, the same request and+reply as TypeSafe's Jev API and as every other model colibri serves+([systemone.md](systemone.md#decision-engines)).++## Download and run++The checkpoint is the root of the repository (1.9 GB, F32):++```sh+hf download fastino/GLiNER2.5-Decide model.safetensors config.json \+ "encoder_config/*" tokenizer.json tokenizer_config.json special_tokens_map.json \+ --local-dir ~/Models/GLiNER2.5-Decide+make -C c gliner_decide+./c/coli serve --model ~/Models/GLiNER2.5-Decide+```++`coli` recognises the checkpoint by its `config.json` (`model_type`+`extractor`, GLiNER2's) and the encoder's `encoder_config/config.json`; the+registry keys it `gliner2_span_deberta-v2`. `coli info` shows the encoder and+the token budget, `coli plan` the RAM, `coli doctor` what is missing.++```sh+curl -s localhost:8000/v1/systemone -H 'Content-Type: application/json' -d '{+ "model": "jev-latest",+ "state": "Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan.",+ "questions": {+ "department": {"type": "choice", "instructions": "Which department should handle this?",+ "criteria": {"billing": "invoices, payments, refunds",+ "technical": "bugs, outages, system errors", "other": "everything else"}},+ "urgency": {"type": "score", "instructions": "How urgent is this?",+ "criteria": ["not urgent", "soon", "blocking"]},+ "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"}}}'+```++```json+{"id": "req_...", "model": "gliner2.5-decide", "provider": "colibri",+ "answers": {+ "department": {"type": "choice", "choice": "billing",+ "probabilities": {"billing": 0.751237, "technical": 0.092047, "other": 0.156716},+ "confidence": 0.626855},+ "urgency": {"type": "score", "score": 1.218665,+ "legend": {"0": "not urgent", "1": "soon", "2": "blocking"},+ "probabilities": {"0": 0.167397, "1": 0.446541, "2": 0.386062}, "confidence": 0.169812},+ "churn_risk": {"type": "noul", "noul": 0.995977}},+ "usage": {"input_tokens": 118, "output_tokens": 0, "cost": 0}}+```++That is the reply `coli serve` gave on the release (the `id` shortened).+`input_tokens` is the length of the one sequence the encoder read. Chat,+completions and messages answer 400 with a pointer to `/v1/systemone`, and+`coli chat` says the same before starting anything.++## From System One to the model's task++GLiNER2 has no notion of choice, score or noul: it classifies a text against+named tasks, each with its labels. The engine turns a System One request into+the call the model card makes, `classify_text(state, tasks)`, one task per+question, in the order given:++| System One | GLiNER2 task |+|---|---|+| question id | the task name (`department`, `urgency`) |+| `instructions` | the task's `prompt` (`department: Which department should handle this?`) |+| `choice` criteria | the labels, in order; a non-empty description becomes a label description |+| `score` criteria | the labels `"0"` .. `"n-1"`, the way the model card writes an ordinal scale; each level's text becomes its description |+| `noul` | the labels `"yes"` then `"no"`, the order of every yes/no example of the model card; `criteria.true` / `criteria.false` describe them |++Every task is single-label, so its probabilities are the softmax over its+labels, which is what `classify_text` returns for a single-label task. The+reply follows from them as on every engine: `choice` is the argmax, `score`+the expected level, `noul` the probability of `yes`.++Two consequences of reading the request as one call:++- **All questions share one sequence.** The encoder reads every question's+ schema and the state together, so a question's probabilities depend on the+ other questions of the same request, as they do in the package when one+ call scores several tasks (the model card's email triage). One request is+ one forward pass, whatever the number of questions.+- **Missing instructions are filled by the gateway** with the text the+ language-model path asks with (`Which of the following applies?`,+ `Rate this on the scale below.`, `Is this true?`). For GLiNER2 that text+ becomes the task's prompt. Measured on eleven single-task examples of the+ model card, asked with no prompt and with the default one (`Is this true?`+ for the yes/no labels, `Rate this on the scale below.` for the 0-5 and 0-10+ scales, `Which of the following applies?` for the rest), the default prompt+ changed no decision and moved the top probability by less than 0.001 up to+ 0.12 (refund_request 0.997 -> 0.969, seat_change 0.501 -> 0.622). Send+ `instructions` when you have them.++## What the engine computes++Each step follows the `gliner2` package (2.0.0) and the transformers+DeBERTa-v2 code it runs, which are the reference:++| step | reference | in `c/gliner_decide.c` |+|---|---|---|+| the text | `processor._collate_batch` | a `.` appended when the state does not end in `.`, `!` or `?`; an empty state is `.` |+| words | `processing/word_splitter.WhitespaceTokenSplitter` | Python's regex (URLs, e-mails, @handles, `\w+(?:[-_]\w+)*`, `\S`) with Python's own `\w`, `\s` and `re.IGNORECASE`, each word lower-cased with `str.lower()` (final sigma and dotted capital I included); the tables come from Python by `tools/gen_gliner_unicode.py` |+| schema | `Schema.classification`, `processor._transform_schema` | per task `( [P] name[: prompt][ [DESCRIPTION] label: text ...] ( [L] label ... ) )`, tasks joined by `[SEP_STRUCT]`, then `[SEP_TEXT]` and the words |+| tokens | `processor._format_input_with_mapping` | every element above tokenized on its own, no `[CLS]` or `[SEP]`; the classifier reads the first token of each `[L]` |+| tokenizer | `tokenizer.json` (DebertaV2TokenizerFast) | added tokens split out of the raw text, `Replace(\s{2,}\|[\n\r\t] -> " ")`, NFC, right strip, `[UNK]` matched after normalization, Metaspace (prepend always, split), Unigram (best-scoring segmentation, unknown characters at min score - 10, adjacent unknowns fused into one `[UNK]`) |+| encoder | `DebertaV2Model` | word embeddings and LayerNorm (no absolute positions); 24 post-norm layers of disentangled attention: `(q.k + q.pos_k[r] + k.pos_q[r]) / sqrt(3 * 64)` with `r` the relative position in log buckets (256 buckets, maximum 512, clamped to the 512-row table), `pos_q` and `pos_k` the LayerNorm'd relative embeddings through the layer's own query and key projections; GELU FFN |+| head | `runtime._extract_classification_result` | the classifier (Linear 1024 -> 2048, ReLU, Linear 2048 -> 1) at each `[L]`, temperature 1, softmax over the task's labels |++The span, count and relation heads (`span_rep`, `count_pred`,+`count_embed`) serve extraction and are not loaded. The relative embeddings+are the same for every request, so their projections are computed once at+load. Weights are kept in f32 as released.++## Checked against the reference++- **Tiny oracle** (`make -C c gliner-decide-tiny-check`, in CI): a random+ checkpoint in the release's layout (`tools/make_gliner_decide_tiny.py`,+ numpy, a Unigram tokenizer counted from a small corpus, a head size of 24,+ 16 position buckets with a maximum of 64 so every sequence reaches the log+ buckets and the clamp) and what `gliner2` answers on it+ (`gliner_decide_tiny/ref.json`, written by `tools/make_gliner_decide_ref.py`):+ 13 requests with 26 questions plus one whose questions do not fit, covering+ every type, 2 to 40 options, labels with and without descriptions, missing+ instructions, long states and a conversation cut at `max_len`, JSON states+ and instructions, the GLiNER2 marker tokens inside the text, URLs, e-mails,+ handles, unicode and an empty state; plus 30 tokenizer strings and 15+ word-splitter strings. Token ids and markers identical, probabilities within+ 2.0e-6 (tolerance 2e-5), every decision the same; then the same fixture+ through `coli serve`.+- **Tokenizer and word splitter fuzzing** (during development, not in CI):+ 20,013 random strings per seed mixing ASCII, whitespace of every kind, Greek+ (final sigma), Cyrillic, CJK, combining marks, emoji, the Kelvin sign, long+ s, dotted and dotless i, URLs, e-mails and the added tokens. Four seeds+ against the release's tokenizer and the word splitter, two against the tiny+ tokenizer: 0 differences in token ids, 0 in words.+- **The real checkpoint** (`tools/compare_gliner_decide.py`): 39 requests, 79+ questions: every example of the model card written as System One questions+ (the multi-label ones as one noul per label), the Jev examples of+ systemone.md, a contract of about 960 tokens, a conversation, JSON states+ and instructions, 40 banking intents, ten questions on one ticket, unicode,+ emoji, URLs, the marker tokens in the text, an empty state. Token ids+ identical on 39/39, decisions 79/79 the same, max |dp| 1.7e-6.++## Speed++Measured on an i7-1355U laptop (10 cores, 12 threads), 6 threads for both+sides, on a machine shared with other jobs: the load average stayed between+10 and 13 during the runs below, so absolute numbers move from run to run.+Engine and package were run in turn, request by request, so both saw the+same load (median of 5 each):++| request | tokens | `coli serve`, at the client | of which engine | `gliner2` package (torch, CPU, f32) |+|---|---|---|---|---|+| one question (model card, spam or not) | 37 | 294 ms | 290 ms | 718 ms |+| the README request, 3 questions | 118 | 897 ms | 888 ms | 1217 ms |+| ten questions on one ticket | 286 | 1955 ms | 1950 ms | 2231 ms |+| a contract, 3 questions | 965 | 8401 ms | 8397 ms | 9457 ms |++The gateway's share is 4 to 10 ms. Over the 39 requests of the comparison+the engine's median was 502 ms per request (load average 5 to 8 when+sampled). A request is one forward pass whatever the number of questions,+so its time follows the length of the sequence: the ten questions above are+one pass over 286 tokens, most of them the questions' own schema.++Longer sequences cost more than in proportion, the attention growing with+the square of the length. A three-label question over a repeated support+text, same machine: 0.54 s at 81 tokens and 2.8 s at 494 with the machine+otherwise idle; with a load average of 7 to 15, 0.69 s at 81, 4.0 s at 494,+8.7 s at 970, 25.7 s at 1923 and 93 s at 3829. `COLI_GLINER_MAX_LEN` bounds+it.++RAM: 1.81 GB resident for the engine serving the release (peak 1.88 GB):+the f32 encoder and classifier, 436M parameters of the 486M in the file.++## Knobs++| variable | default | meaning |+|---|---|---|+| `COLI_GLINER_MAX_LEN` | 4096 | tokens of the one sequence a request becomes: the questions first, then as many whole words of the state as fit |+| `OMP_NUM_THREADS` | physical cores | as for the other engines |++Questions whose schema alone exceeds `COLI_GLINER_MAX_LEN` are refused with a+422 that says how many tokens they take.++## Limits++- **English.** The model card measures this checkpoint on English; fastino+ points to GLiNER2.5-multi-Decide for other languages, which this engine has+ not been run on.+- **No calibration ships with the checkpoint.** The probabilities are the+ softmax of the classifier at temperature 1, as the package computes them;+ `temperature` in the DECISION frame is `null`.+- **Single-label only.** System One has no multi-label question, so the+ package's `multi_label` mode (a sigmoid per label and a threshold) is not+ reachable; ask one noul per label instead, as the comparison above does.+- **Long states.** The package reads a whole text by default; the engine+ stops at `COLI_GLINER_MAX_LEN` tokens so that one request cannot take the+ machine, and `state_dropped` in the DECISION frame says how many state+ tokens were left out. Below the limit the two read the same tokens. The+ cost grows with the square of the length (see Speed).+- **The `noul` label order.** The engine asks `yes` before `no`; the order+ moves the probabilities a little (handoff example of the model card:+ p(yes) 0.325 asked yes-first, 0.297 asked no-first), never the decision on+ the examples measured.+- **Question ids that extend each other.** The package finds a task's answer+ by the prefix of its schema text, so with two question ids such as `risk`+ (with instructions `score it`) and `risk: score`, `classify_text` returns+ only one of them (checked on the tiny fixture). The engine answers every+ question by position.+- **Other GLiNER2 checkpoints.** The registry accepts any span GLiNER2+ checkpoint with a DeBERTa-v2/v3 encoder, and a 24 x 1024 encoder is named+ GLiNER2.5-Decide. Only `fastino/GLiNER2.5-Decide` was compared with the+ package; the boundary architecture is refused.diff --git a/docs/serve_protocol.md b/docs/serve_protocol.mdindex 2629c623..a3a852bd 100644--- a/docs/serve_protocol.md+++ b/docs/serve_protocol.md@@ -42,7 +42,7 @@ Two more keys say what kind of engine answers. `decide=1`: the engine takes`DECIDE` (below), and the gateway sends `POST /v1/systemone` to it as one recordinstead of scoring options through the logprob channel. `chat=0`: it has nothingelse, so the generating endpoints answer 400 with a pointer to `/v1/systemone`.-A decision engine (laya) says `decide=1 chat=0`; an engine that chats and also+A decision engine (laya, gliner_decide) says `decide=1 chat=0`; an engine that chats and alsodecides natively would say `decide=1` alone.## Requests (server → engine)diff --git a/docs/systemone.md b/docs/systemone.mdindex b34bd32b..fff9dd1b 100644--- a/docs/systemone.md+++ b/docs/systemone.md@@ -510,6 +510,7 @@ cannot tell which kind answered.| engine | model | doc ||---|---|---|| `c/laya` | Laya (Convai Innovations): ModernBERT encoder + decision head | [laya.md](laya.md) |+| `c/gliner_decide` | GLiNER2.5-Decide (fastino): DeBERTa-v3 encoder + GLiNER2's classification head | [gliner_decide.md](gliner_decide.md) |A decision engine serves `POST /v1/systemone` only. Chat, completions,messages and `/v1/brio` answer 400 with a pointer to `/v1/systemone`, and@@ -579,23 +580,23 @@ the engine's own record of how it got there. A record the engine refuses is**Registry.** A family with `modality="decision"` and`FamilyCapabilities(decision=True)` in `c/family_registry.py`, and a way for`resolve_model` to recognise its checkpoint (Laya: `rl_agent_config.json`-plus `encoder/config.json`, keyed `laya_<encoder model_type>`). `coli serve`+plus `encoder/config.json`, keyed `laya_<encoder model_type>`;+GLiNER2.5-Decide: a `config.json` whose `model_type` is `extractor` plus+`encoder_config/config.json`, keyed+`gliner2_<architecture>_<encoder model_type>`). `checkpoint_files` names the+files `coli doctor` checks. `coli serve`and `coli web` then serve it, `coli info` and `coli plan` describe it, and`coli chat` / `coli run` refuse it with a pointer to the endpoint.**Tests.** A tiny fixture whose reference answers come from the model's ownpackage (`tools/make_laya_tiny.py`, `tools/make_laya_ref.py`,-`tests/test_laya_tiny.py`), and the fixture behind the real gateway-(`tests/test_decision_serve.py`, `tests/test_jev_sdk.py`).+`tests/test_laya_tiny.py`; `tools/make_gliner_decide_tiny.py`,+`tools/make_gliner_decide_ref.py`, `tests/test_gliner_decide_tiny.py`), and+the fixture behind the real gateway (`tests/test_decision_serve.py`,+`tests/test_jev_sdk.py`).-### The next two+### The next one-- **GLiNER2.5-Decide** (DeBERTa-v3 encoder, GLiNER2 heads): a second encoder- engine on the same contract. It needs the DeBERTa-v3 encoder (relative- position buckets and disentangled attention, a SentencePiece tokenizer) and- its heads, its own rendering of a record into its input, and a- `resolve_model` rule for its checkpoint. `decide_serve.h`, the gateway and- the SDK tests stay as they are.- **Clef** (a joint schema head over Qwen3.8-27B, dense, with vision): thebackbone is the qwen36 engine's. It would answer `DECIDE` in that engine'sserve loop next to `SUBMIT`, announce `decide=1` without `chat=0` (the same