Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

fix(oracle): enforce GLM comparison results in strict mode

Blakebc15290cbab5
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index c9af31a3..822ae37f 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -923,10 +923,15 @@ jobs:
/tmp/tike
- name: Generate the glm_tiny fixture
run: cd c && python3 tools/make_glm_oracle.py
- - name: Token-exact oracle (teacher forcing)
- # The fixture is only trustworthy if the engine reproduces it, so assert
- # that before reading anything else out of a run.
- run: cd c && SNAP=./glm_tiny TF=1 COLI_TEMP=0 ./colibri 64 16 16
+ - name: Oracle (30–32/32 teacher forcing + exact greedy)
+ # Preserve CONTRIBUTING.md's two TF near-tie allowance. Greedy remains
+ # exact, and invalid/non-finite results cannot consume the allowance.
+ run: |
+ cd c
+ SNAP=./glm_tiny TF=1 COLI_TEMP=0 ORACLE_STRICT=1 ORACLE_TF_MAX_MISMATCHES=2 ./colibri 64 16 16
+ SNAP=./glm_tiny COLI_TEMP=0 ORACLE_STRICT=1 ./colibri 64 16 16
+ - name: Oracle allowance and failure regressions
+ run: cd c && python3 tests/test_glm_oracle.py
- name: Structural efficiency tests
# test_cpu_vs_cpu_tok_s_stability is NOT in this list: it is a tok/s
# bound (two runs within 25%) over a ~15 ms tiny replay -- it measured
@@ -1229,7 +1234,7 @@ jobs:
# This job builds every engine on arm64 (NEON compile coverage), runs the
# integer-kernel bit-exactness gate with the NEON branches live, and replays
# the glm_tiny teacher-forcing oracle against a fixture generated on THIS
- # runner (same-machine torch reference, so no cross-ISA float excuses).
+ # runner. As on x86, allow two TF near ties; greedy remains exact.
oracle-arm:
name: ARM (engines + NEON kernel exactness + tiny oracle)
runs-on: ubuntu-24.04-arm
@@ -1252,5 +1257,10 @@ jobs:
/tmp/tike
- name: Generate the glm_tiny fixture on this runner
run: cd c && python3 tools/make_glm_oracle.py
- - name: Token-exact oracle on ARM (teacher forcing)
- run: cd c && SNAP=./glm_tiny TF=1 COLI_TEMP=0 ./colibri 64 16 16
+ - name: Oracle on ARM (30–32/32 teacher forcing + exact greedy)
+ run: |
+ cd c
+ SNAP=./glm_tiny TF=1 COLI_TEMP=0 ORACLE_STRICT=1 ORACLE_TF_MAX_MISMATCHES=2 ./colibri 64 16 16
+ SNAP=./glm_tiny COLI_TEMP=0 ORACLE_STRICT=1 ./colibri 64 16 16
+ - name: Oracle allowance and failure regressions on ARM
+ run: cd c && python3 tests/test_glm_oracle.py
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 28cebe5d..2fcc4432 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -5,7 +5,7 @@ Keep changes focused and preserve Colibri's dependency-free default CPU path.
## Branches
- **`main`** is the stable branch. It's what users clone, and it stays known-good
- (engine always passes the token-exact oracle: `SNAP=./glm_tiny TF=1 ./colibri 64 16 16`,
+ (engine always passes the oracle: `SNAP=./glm_tiny TF=1 COLI_TEMP=0 ORACLE_STRICT=1 ORACLE_TF_MAX_MISMATCHES=2 ./colibri 64 16 16`,
run from `c/`). The `glm_tiny` fixture is generated, not committed --
`python3 c/tools/make_glm_oracle.py` builds it (needs torch).
- **`dev`** is the integration branch. **Open your PR against `dev`.** Reviewed PRs
@@ -13,9 +13,37 @@ Keep changes focused and preserve Colibri's dependency-free default CPU path.
it into `main`. This keeps `main` clean instead of taking every PR one at a time.
Every PR — on either branch — is reviewed for a clean build (0 warnings), the oracle
-(~30-32/32 TF depending on floating-point near-ties + 20/20 greedy), and its own
+(30–32/32 TF depending on floating-point near ties + 20/20 greedy), and its own
targeted validation before merge.
+After generating the fixture, run both enforced comparisons from `c/`:
+
+```sh
+SNAP=./glm_tiny TF=1 COLI_TEMP=0 ORACLE_STRICT=1 ORACLE_TF_MAX_MISMATCHES=2 ./colibri 64 16 16
+SNAP=./glm_tiny COLI_TEMP=0 ORACLE_STRICT=1 ./colibri 64 16 16
+python3 tests/test_glm_oracle.py
+```
+
+`ORACLE_STRICT=1` makes a failed comparison exit with status 1. Strict mode is
+token-exact by default. The teacher-forcing command above explicitly allows at
+most two mismatches, preserving the 30–32/32 acceptance range for this fixture;
+greedy comparison always requires every continuation token to match. Non-finite
+output and incomplete generation fail regardless of the mismatch allowance.
+Invalid reference arrays/JSON fail in either mode.
+Without strict mode (or with `ORACLE_STRICT=0`), a completed comparison remains
+report-only for diagnostic/benchmark callers; its exit status is not a correctness
+gate. Strict mode rejects `REPLAY`, serving, text generation, and other execution
+modes that would bypass the comparison.
+
+`ORACLE_TF_MAX_MISMATCHES` is a nonnegative integer smaller than the number of
+TF positions; it is used only for strict teacher-forcing comparisons. Unset or
+`0` requires exact TF agreement. Generate the reference on the test host and
+inspect near ties with `TF=1 DEBUG_LOGITS=1`; an allowed mismatch is still printed
+and does not establish token-exact agreement. The regression script checks the
+two-mismatch boundary, optional exact mode, and rejection of a single greedy
+mismatch. It validates the original TF fixture within the same allowance. Routine
+`make check` covers reference validation without needing torch or a generated model.
+
## Local checks
Run the lightweight checks locally:
diff --git a/c/Makefile b/c/Makefile
index 6cfa5cf7..a10ff16d 100644
--- a/c/Makefile
+++ b/c/Makefile
@@ -733,7 +733,7 @@ $(file >.build-config,$(BUILD_CONFIG))
endif
.build-config: ;
-colibri$(EXE): colibri.c cli_args.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h quant.h sample.h kv_persist.h telemetry.h route_trace.h omp_tune.h kv_fp8.h kv_tq.h abl.h backend_cuda.h backend_metal.h backend_vulkan.h decode_batch.h edge_adapters.h edge_runtime.h edge_tok_internal.h schema_gbnf.h segment_adapter_internal.h segment_adapters.h segment_runtime.h tier.h $(CUDA_OBJ) $(METAL_OBJ) $(VK_OBJ) $(VK_SPV) .build-config
+colibri$(EXE): colibri.c cli_args.h oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h quant.h sample.h kv_persist.h telemetry.h route_trace.h omp_tune.h kv_fp8.h kv_tq.h abl.h backend_cuda.h backend_metal.h backend_vulkan.h decode_batch.h edge_adapters.h edge_runtime.h edge_tok_internal.h schema_gbnf.h segment_adapter_internal.h segment_adapters.h segment_runtime.h tier.h $(CUDA_OBJ) $(METAL_OBJ) $(VK_OBJ) $(VK_SPV) .build-config
$(CC) $(CFLAGS) colibri.c $(CUDA_OBJ) $(METAL_OBJ) $(VK_OBJ) -o colibri$(EXE) $(LDFLAGS)
# Vulkan backend object (plain C + vulkan headers) and its SPIR-V shaders.
@@ -1207,7 +1207,7 @@ iobench$(EXE): iobench.c compat.h
tests/test_serve_sentinel$(EXE): tests/test_serve_sentinel.c compat.h serve_codec.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_cluster_protocol$(EXE): tests/test_cluster_protocol.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_cluster_protocol$(EXE): tests/test_cluster_protocol.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_ue8m0$(EXE): tests/test_ue8m0.c st.h json.h compat.h
@@ -1313,12 +1313,12 @@ tests/test_grammar$(EXE): tests/test_grammar.c grammar.h
# equivalence under the shipping flags is covered by the differential dump
# (old vs new rope_interleave are byte-identical); this test guards the
# source-level formula equivalence and must not depend on contraction luck.
-tests/test_rope_invfreq$(EXE): tests/test_rope_invfreq.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_rope_invfreq$(EXE): tests/test_rope_invfreq.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) -ffp-contract=off $< -o $@ $(LDFLAGS)
# schema->GBNF compile cache (#7): grammar_reset must equal a fresh setup, and
# the GrDraft.src ownership must not leak/double-free. Includes colibri.c.
-tests/test_grammar_cache$(EXE): tests/test_grammar_cache.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_grammar_cache$(EXE): tests/test_grammar_cache.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# Standalone: drives a faithful miniature of moe()'s routing+accumulate and links
@@ -1329,7 +1329,7 @@ tests/test_ablate$(EXE): tests/test_ablate.c abl.h
tests/test_schema_gbnf$(EXE): tests/test_schema_gbnf.c schema_gbnf.h grammar.h json.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_spec_decode_state$(EXE): tests/test_spec_decode_state.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_spec_decode_state$(EXE): tests/test_spec_decode_state.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_decode_batch$(EXE): tests/test_decode_batch.c decode_batch.h
@@ -1382,33 +1382,33 @@ tests/test_olmoe_cache_index$(EXE): tests/test_olmoe_cache_index.c olmoe.c st.h
tests/test_qwen36_cache_index$(EXE): tests/test_qwen36_cache_index.c qwen36.c expert_ffn.h st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h qwen36_tier.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)
$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)
-tests/test_idot$(EXE): tests/test_idot.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_idot$(EXE): tests/test_idot.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_i4_grouped$(EXE): tests/test_i4_grouped.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_i4_grouped$(EXE): tests/test_i4_grouped.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_stops$(EXE): tests/test_stops.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_stops$(EXE): tests/test_stops.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_cfg_topk$(EXE): tests/test_cfg_topk.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_cfg_topk$(EXE): tests/test_cfg_topk.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_topp$(EXE): tests/test_topp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_topp$(EXE): tests/test_topp.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_topp is a microbenchmark (old qsort vs new heap partial-select, #335), NOT a test
# gate -- intentionally absent from TEST_BINS. Build on demand: make tests/bench_topp
-tests/bench_topp$(EXE): tests/bench_topp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_topp$(EXE): tests/bench_topp.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_sample_nan$(EXE): tests/test_sample_nan.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_sample_nan$(EXE): tests/test_sample_nan.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_temp_env$(EXE): tests/test_temp_env.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_temp_env$(EXE): tests/test_temp_env.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_kv_alloc$(EXE): tests/test_kv_alloc.c colibri.c st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h kv_fp8.h kv_tq.h
+tests/test_kv_alloc$(EXE): tests/test_kv_alloc.c colibri.c oracle.h st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h kv_fp8.h kv_tq.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_kv_fp8$(EXE): tests/test_kv_fp8.c kv_fp8.h decode_batch.h
@@ -1417,7 +1417,7 @@ tests/test_kv_fp8$(EXE): tests/test_kv_fp8.c kv_fp8.h decode_batch.h
tests/test_kv_tq$(EXE): tests/test_kv_tq.c kv_tq.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_kv_disk$(EXE): tests/test_kv_disk.c colibri.c st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h kv_fp8.h kv_tq.h
+tests/test_kv_disk$(EXE): tests/test_kv_disk.c colibri.c oracle.h st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h kv_fp8.h kv_tq.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# fmt=6 kernel oracle: needs the generated grid table, and its fixture comes from
@@ -1452,10 +1452,10 @@ fuzz-rans: tests/fuzz_rans.c rans.h
tests/fuzz_rans.c -o tests/fuzz_rans -lm
./tests/fuzz_rans
-tests/test_int3$(EXE): tests/test_int3.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_int3$(EXE): tests/test_int3.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_int3_load$(EXE): tests/test_int3_load.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_int3_load$(EXE): tests/test_int3_load.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_fp8_passthrough$(EXE): tests/test_fp8_passthrough.c quant.h
@@ -1467,16 +1467,16 @@ tests/test_fp8_passthrough$(EXE): tests/test_fp8_passthrough.c quant.h
tests/test_cuda_fmt_guard$(EXE): tests/test_cuda_fmt_guard.c backend_cuda.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_fp8_load$(EXE): tests/test_fp8_load.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_fp8_load$(EXE): tests/test_fp8_load.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_qt_addrow$(EXE): tests/test_qt_addrow.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_qt_addrow$(EXE): tests/test_qt_addrow.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_logit_nan$(EXE): tests/test_logit_nan.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
+tests/test_logit_nan$(EXE): tests/test_logit_nan.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_router_nan$(EXE): tests/test_router_nan.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
+tests/test_router_nan$(EXE): tests/test_router_nan.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_i4_acc512$(EXE): tests/test_i4_acc512.c
@@ -1526,7 +1526,7 @@ SEGMENT_RUNTIME_LIB = $(SEGMENT_BUILD_DIR)/libcolibri_segment_edge.a
$(SEGMENT_BUILD_DIR):
mkdir -p $@
-$(SEGMENT_BUILD_DIR)/glm.o: colibri.c segment_runtime.h edge_runtime.h \
+$(SEGMENT_BUILD_DIR)/glm.o: colibri.c oracle.h segment_runtime.h edge_runtime.h \
segment_adapters.h edge_adapters.h segment_adapter_internal.h \
edge_adapter_internal.h edge_tok_internal.h st.h | $(SEGMENT_BUILD_DIR)
$(CC) $(SEGMENT_CPU_CFLAGS) -DCOLIBRI_NO_MAIN -c colibri.c -o $@
@@ -1749,6 +1749,9 @@ tests/test_route_trace$(EXE): tests/test_route_trace.c route_trace.h compat.h
tests/test_cli_args$(EXE): tests/test_cli_args.c cli_args.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
+tests/test_oracle$(EXE): tests/test_oracle.c oracle.h json.h
+ $(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
+
# Il tier CUDA con un backend finto: gira SENZA GPU perche' il test definisce i
# coli_cuda_* e registra cosa riceve. E' il solo modo di provare in CI che un
# esperto arriva davvero in VRAM e nel formato giusto (#1331) -- un test che si
@@ -1838,15 +1841,15 @@ tests/test_mem_available$(EXE): tests/test_mem_available.c compat.h
tests/test_798_guards$(EXE): tests/test_798_guards.c st.h json.h compat.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_dsa_select$(EXE): tests/test_dsa_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_dsa_select$(EXE): tests/test_dsa_select.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_corpus_draft$(EXE): tests/test_corpus_draft.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_corpus_draft$(EXE): tests/test_corpus_draft.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_cap_precedence$(EXE): tests/test_cap_precedence.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_cap_precedence$(EXE): tests/test_cap_precedence.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_mirror_stripe_split$(EXE): tests/test_mirror_stripe_split.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_mirror_stripe_split$(EXE): tests/test_mirror_stripe_split.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_v4_hybrid_policy$(EXE): tests/test_v4_hybrid_policy.c deepseek_v4_hybrid.h hybrid_split.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
@@ -1857,29 +1860,29 @@ tests/test_k3_fill_budget$(EXE): tests/test_k3_fill_budget.c hybrid_split.h
tests/test_v4_bank_pair$(EXE): tests/test_v4_bank_pair.c deepseek_v4_bank_pair.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_ram_clamp$(EXE): tests/test_ram_clamp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_ram_clamp$(EXE): tests/test_ram_clamp.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_cap_mixed_width$(EXE): tests/test_cap_mixed_width.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_cap_mixed_width$(EXE): tests/test_cap_mixed_width.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_eslot_inflight$(EXE): tests/test_eslot_inflight.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_eslot_inflight$(EXE): tests/test_eslot_inflight.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_glm_cache_index$(EXE): tests/test_glm_cache_index.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_glm_cache_index$(EXE): tests/test_glm_cache_index.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_ssd_probe$(EXE): tests/test_ssd_probe.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_ssd_probe$(EXE): tests/test_ssd_probe.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_dsa_select is a microbenchmark (old qsort vs new quickselect partial-select, #356),
# NOT a test gate -- intentionally absent from TEST_BINS. Build on demand: make tests/bench_dsa_select
-tests/bench_dsa_select$(EXE): tests/bench_dsa_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_dsa_select$(EXE): tests/bench_dsa_select.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_router_select is a microbenchmark (duplicate-prefix scan vs marked-score scan),
# not a test gate. Build on demand: make tests/bench_router_select.
-tests/bench_router_select$(EXE): tests/bench_router_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_router_select$(EXE): tests/bench_router_select.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_indexer_allocations: microbenchmark (DeepSeek V4 indexer malloc vs persistent arena scratch), NOT a test gate.
@@ -1889,29 +1892,29 @@ tests/bench_indexer_allocations$(EXE): tests/bench_indexer_allocations.c
# bench_idot: microbenchmark (single-acc vs independent-acc AVX-VNNI idot), NOT a test gate.
# Build on demand on an AVX-VNNI CPU: make tests/bench_idot ARCH=native
-tests/bench_idot$(EXE): tests/bench_idot.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_idot$(EXE): tests/bench_idot.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_gemv_stream: microbenchmark (decode-regime GEMV bandwidth vs the read ceiling;
# frozen-baseline + deinterleaved-x candidate A/B), NOT a test gate.
# Build on demand: make tests/bench_gemv_stream ARCH=native
-tests/bench_gemv_stream$(EXE): tests/bench_gemv_stream.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_gemv_stream$(EXE): tests/bench_gemv_stream.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_mla_simd: microbenchmark (scalar vs AVX2/NEON MLA-absorb reductions, #442),
# NOT a test gate. Build on demand: make tests/bench_mla_simd
-tests/bench_mla_simd$(EXE): tests/bench_mla_simd.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_mla_simd$(EXE): tests/bench_mla_simd.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_uring$(EXE): tests/test_uring.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_uring$(EXE): tests/test_uring.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_pipe_block$(EXE): tests/test_pipe_block.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
+tests/test_pipe_block$(EXE): tests/test_pipe_block.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_pilot_ring$(EXE): tests/test_pilot_ring.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_pilot_ring$(EXE): tests/test_pilot_ring.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_moe_gs_guard$(EXE): tests/test_moe_gs_guard.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_moe_gs_guard$(EXE): tests/test_moe_gs_guard.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_omp_tune$(EXE): tests/test_omp_tune.c omp_tune.h compat.h
@@ -1920,7 +1923,7 @@ tests/test_omp_tune$(EXE): tests/test_omp_tune.c omp_tune.h compat.h
tests/test_compat_env$(EXE): tests/test_compat_env.c compat.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_kvb_notice$(EXE): tests/test_kvb_notice.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_kvb_notice$(EXE): tests/test_kvb_notice.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
test-c: $(TEST_BINS)
diff --git a/c/colibri.c b/c/colibri.c
index 24ab3b3a..5358c236 100644
--- a/c/colibri.c
+++ b/c/colibri.c
@@ -60,6 +60,7 @@
#include <cpuid.h> /* hwinfo_emit: CPU brand string senza /proc */
#endif
#include "cli_args.h"
+#include "oracle.h"
#include "st.h"
#ifdef __linux__
#include "uring.h"
@@ -1641,7 +1642,8 @@ static void rope_interleave(float *v, int pos, const Cfg *c){
* unverified mirrors (see qt_check_fmt threat model); an unbounded ftell->malloc
* gave a hostile file a load-time OOM or, on malloc failure, a NULL deref via
* b[got]=0. Cap the size, NULL-check the alloc, require a full read. Returns a
- * malloc'd NUL-terminated buffer, or NULL on any failure. Mirrors tok.h tk_read_file. */
+ * malloc'd NUL-terminated buffer, or NULL on any failure. Embedded NUL bytes are
+ * invalid JSON and must not hide an unchecked suffix. Mirrors tok.h tk_read_file. */
#define CFG_MAX_BYTES (256ll<<20) /* config/oracle JSON is KB-MB in practice */
static char* cfg_slurp(const char *path){
FILE *f=fopen(path,"rb"); if(!f) return NULL;
@@ -1649,7 +1651,7 @@ static char* cfg_slurp(const char *path){
if(n<0 || (long long)n>CFG_MAX_BYTES){ fclose(f); return NULL; }
char *b=malloc((size_t)n+1); if(!b){ fclose(f); return NULL; }
size_t got=fread(b,1,(size_t)n,f); fclose(f);
- if((long)got!=n){ free(b); return NULL; }
+ if((long)got!=n || memchr(b,'\0',got)){ free(b); return NULL; }
b[got]=0; return b;
}
static jval* cfg_root(const char *snap, char **arena){
@@ -7787,6 +7789,15 @@ static int spec_decode(Model *m, int *all, int kv, int n_new, int eos, float *lo
/* emit callback: accumula in un array (validazione) */
typedef struct { int *dst; int n; } EmitStore;
static void emit_store(int t, const float *lo, void *ud){ (void)lo; EmitStore *e=(EmitStore*)ud; e->dst[e->n++]=t; }
+typedef struct { EmitStore tokens; int vocab, finite; } OracleEmit;
+static void emit_oracle(int t, const float *lo, void *ud){
+ OracleEmit *e=(OracleEmit*)ud;
+ if(!oracle_logits_finite(lo,e->vocab)){
+ fprintf(stderr,"[ORACLE] non-finite logits at generated token %d\n",e->tokens.n);
+ e->finite=0;
+ }
+ emit_store(t,lo,&e->tokens);
+}
/* emit callback: detokenizza e stampa in streaming (chat/run), con heartbeat */
typedef struct { Tok *T; Model *m; double t0; int count; int quiet; } EmitStream;
static void emit_stream(int t, const float *lo, void *ud){
@@ -7826,8 +7837,9 @@ static void dump_top5_logits(int pos, const float *lo, int V, int expected, int
for(int k=0;k<5&&idx[k]>=0;k++) fprintf(stderr," %d:%.5f", idx[k], (double)val[k]);
fprintf(stderr,"\n");
}
-static void forward_all(Model *m, const int *ids, int S, int *pred, const int *ref){
+static int forward_all(Model *m, const int *ids, int S, int *pred, const int *ref){
Cfg *c=&m->c; int D=c->hidden;
+ int finite=1;
int dbg = ref && getenv("DEBUG_LOGITS");
kv_alloc(m,S);
float *x=falloc((int64_t)S*D);
@@ -7838,11 +7850,16 @@ static void forward_all(Model *m, const int *ids, int S, int *pred, const int *r
for(int s=0;s<S;s++){
rmsnorm(row, x+(int64_t)s*D, m->final_norm, D, c->eps); /* heap row (#183) */
matmul_qt(lo, row, &m->lm_head, 1);
+ if(!oracle_logits_finite(lo,c->vocab)){
+ fprintf(stderr,"[ORACLE] non-finite logits at teacher-forcing position %d\n",s);
+ finite=0; pred[s]=-1; continue;
+ }
int best=0; float bv=lo[0]; for(int i=1;i<c->vocab;i++) if(lo[i]>bv){bv=lo[i];best=i;}
pred[s]=best;
if(dbg && pred[s]!=ref[s]) dump_top5_logits(s, lo, c->vocab, ref[s], pred[s]);
}
free(x); free(lo); free(row);
+ return finite;
}
/* log-prob (log-softmax) del token target dato il vettore di logit; *am=1 se e' l'argmax */
@@ -8006,12 +8023,14 @@ static void run_ablate_score(Model *m, const char *path){
free(ln); free(ids); free(x); free(lo); free(row); fclose(f);
}
-static void generate(Model *m, const int *prompt, int np, int n_new, int *out){
+static int generate(Model *m, const int *prompt, int np, int n_new, int *out, int *finite){
kv_alloc(m,np+n_new+g_draft+2);
for(int i=0;i<np;i++) out[i]=prompt[i];
float *logit=step(m,prompt,np,0);
- EmitStore es={out+np,0};
- spec_decode(m,out,np,n_new,-1,logit,emit_store,&es,NULL,NULL);
+ OracleEmit es={{out+np,0},m->c.vocab,1};
+ int emitted=spec_decode(m,out,np,n_new,-1,logit,emit_oracle,&es,NULL,NULL);
+ *finite=es.finite;
+ return emitted;
}
static void profile_print(Model *m, double elapsed){
@@ -9346,13 +9365,6 @@ static void run_serve(Model *m, const char *snap){
free(ctx); m->kv=NULL; m->Lc=m->Rc=m->Ic=NULL; m->Lc8=m->Rc8=NULL; m->Lsc=m->Rsc=NULL; m->kv_start=NULL; m->max_t=0;
}
-static int *read_arr(jval*o,const char*k,int*n){
- jval*a=json_get(o,k);
- if(!a){ *n=0; return NULL; }
- int*r=malloc(a->len*sizeof(int));
- if(!r){ fprintf(stderr,"OOM read_arr\n"); exit(1); }
- for(int i=0;i<a->len;i++) r[i]=(int)a->kids[i]->num; *n=a->len; return r; }
-
/* telemetry, stats, usage persistence — moved to telemetry.h */
#ifdef COLI_VULKAN
@@ -10753,6 +10765,19 @@ static int coli_env_on(const char *name)
#ifndef COLIBRI_NO_MAIN
int main(int argc, char **argv){
+ int strict=coli_env_on("ORACLE_STRICT");
+ if(strict){
+ const char *modes[]={"REPLAY","SERVE","SCORE","ABLATE_SCORE","EXPERT_WORKER",
+ "I4_ACC512_TEST","I3_AVX512_TEST"};
+ for(size_t i=0;i<sizeof(modes)/sizeof(modes[0]);i++) if(getenv(modes[i])){
+ fprintf(stderr,"[ORACLE] ORACLE_STRICT cannot be combined with %s\n",modes[i]);
+ return 1;
+ }
+ if(coli_user_prompt() || (getenv("COLI_ANS_PACK") && atoi(getenv("COLI_ANS_PACK")))){
+ fprintf(stderr,"[ORACLE] ORACLE_STRICT requires oracle comparison mode\n");
+ return 1;
+ }
+ }
/* ---- Permanent OpenMP hot-thread tuning. The per-expert matmul regions are
* tiny and back-to-back; with the default passive wait policy libgomp parks
* the worker team between regions and the re-wake latency dominates. Keeping
@@ -11521,13 +11546,22 @@ int main(int argc, char **argv){
}
/* altrimenti: validazione contro l'oracolo (ref_glm.json) */
+ int teacher_forcing=getenv("TF")!=NULL;
const char *refpath=getenv("REF")?getenv("REF"):"ref_glm.json";
char *b=cfg_slurp(refpath);
- if(!b){ fprintf(stderr,"%s: cannot read oracle file (missing, unreadable, short, or > %lld bytes)\n",refpath,(long long)CFG_MAX_BYTES); return 1; }
- char *ar=NULL; jval *ref=json_parse(b,&ar);
- int np=0,nfull=0; int *prompt=read_arr(ref,"prompt_ids",&np); int *full=read_arr(ref,"full_ids",&nfull);
- if(!prompt||!full||np<1||nfull<np){ fprintf(stderr,"ref file missing prompt_ids/full_ids or empty\n"); return 1; }
+ if(!b){ fprintf(stderr,"%s: cannot read oracle file (missing, unreadable, short, contains NUL, or > %lld bytes)\n",refpath,(long long)CFG_MAX_BYTES); return 1; }
+ OracleRef ref;
+ int valid=oracle_ref_parse(b,m.c.vocab,teacher_forcing,&ref);
+ free(b);
+ if(!valid) return 1;
+ int np=ref.np,nfull=ref.nfull; int *prompt=ref.prompt,*full=ref.full;
int n_new=nfull-np;
+ int tf_allowed=0;
+ if(strict && teacher_forcing &&
+ !oracle_tf_allowance(getenv("ORACLE_TF_MAX_MISMATCHES"),nfull,&tf_allowed)){
+ fprintf(stderr,"[ORACLE] ORACLE_TF_MAX_MISMATCHES must be an integer in [0,%d)\n",nfull);
+ oracle_ref_free(&ref); return 1;
+ }
/* L'oracolo (ref_glm.json in repo) e' del modello TINY: contro il 744B da' 0/20
* garantito su OGNI piattaforma (prompt-token tiny = spazzatura per il modello vero).
* Non e' un bug del motore — vedi #76. */
@@ -11544,19 +11578,22 @@ int main(int argc, char **argv){
" Nessun PROMPT: modo auto-validazione, ma ref_glm.json e' l'oracolo del modello TINY\n"
" (token max %d, il tuo vocab e' %d). Usa PROMPT=... per generare davvero (vedi sopra).\n",
maxid, m.c.vocab, maxid, m.c.vocab);
- return 1;
+ oracle_ref_free(&ref); return 1;
} }
if(getenv("REPLAY")){
run_replay(&m,full,nfull,np);
if(stats) stats_dump(&m,stats);
+ oracle_ref_free(&ref);
return 0;
}
- if(getenv("TF")){
- int *tf=read_arr(ref,"tf_pred",&(int){0});
- int *pred=malloc(nfull*sizeof(int)); double tt=now_s();
- forward_all(&m, full, nfull, pred, tf); double tdt=now_s()-tt;
+ if(teacher_forcing){
+ int *tf=ref.tf;
+ int *pred=malloc((size_t)nfull*sizeof(int));
+ if(!pred){ oracle_ref_free(&ref); return 1; }
+ double tt=now_s();
+ int finite=forward_all(&m, full, nfull, pred, tf); double tdt=now_s()-tt;
int ok=0; for(int i=0;i<nfull;i++){
if(pred[i]==tf[i]) ok++;
else fprintf(stderr,"[ORACLE] mismatch pos=%d expected=%d got=%d\n",i,tf[i],pred[i]);
@@ -11566,19 +11603,24 @@ int main(int argc, char **argv){
if(ok<nfull) fprintf(stderr,
"[ORACLE] %d/%d mismatches — run: TF=1 DEBUG_LOGITS=1 for top-5 logit dump\n",
nfull-ok,nfull);
+ if(strict) fprintf(stderr,"[ORACLE] teacher-forcing mismatch allowance: %d/%d\n",tf_allowed,nfull);
profile_print(&m,tdt);
#ifdef COLI_CUDA
if(g_cuda_enabled) cuda_stats_print();
#endif
- return 0;
+ free(pred); oracle_ref_free(&ref);
+ return strict && (!finite || nfull-ok>tf_allowed);
}
- int *out=malloc((np+n_new)*sizeof(int));
+ int *out=malloc((size_t)nfull*sizeof(int));
+ if(!out){ oracle_ref_free(&ref); return 1; }
ProfBase pb; prof_base(&m,&pb);
- double t=now_s(); generate(&m,prompt,np,n_new,out); double dt=now_s()-t;
+ int finite=1;
+ double t=now_s(); int emitted=generate(&m,prompt,np,n_new,out,&finite); double dt=now_s()-t;
int match=0;
printf("\nReference (oracle): "); for(int i=np;i<nfull;i++) printf("%d ", full[i]);
- printf("\nGLM C engine : "); for(int i=np;i<nfull;i++){ printf("%d ", out[i]); if(out[i]==full[i])match++; }
+ printf("\nGLM C engine : "); for(int i=np;i<np+emitted;i++){ printf("%d ", out[i]); if(out[i]==full[i])match++; }
printf("\nMatching tokens: %d/%d\n", match, n_new);
+ if(emitted!=n_new) fprintf(stderr,"[ORACLE] incomplete generation: %d/%d tokens\n",emitted,n_new);
double tot=m.hits+m.miss;
printf("N-gram speculation (DRAFT=%d): %.2f tokens/forward (%llu forwards per %llu tokens)\n",
g_draft, m.n_fw?(double)m.n_emit/m.n_fw:1.0, (unsigned long long)m.n_fw, (unsigned long long)m.n_emit);
@@ -11599,7 +11641,8 @@ int main(int argc, char **argv){
la_tot[i]?100.0*la_hit[i]/la_tot[i]:0.0, (long long)la_hit[i], (long long)la_tot[i]);
}
if(stats) stats_dump(&m,stats);
- return 0;
+ free(out); oracle_ref_free(&ref);
+ return strict && (match!=n_new || emitted!=n_new || !finite);
}
#endif /* COLIBRI_NO_MAIN */
diff --git a/c/json.h b/c/json.h
index 649b92ca..3a938b67 100644
--- a/c/json.h
+++ b/c/json.h
@@ -28,6 +28,7 @@ typedef struct {
size_t acap, aoff;
int depth; /* annidamento corrente: bound contro lo stack-overflow
* da JSON malevolo tipo [[[[...]]]] (discesa ricorsiva) */
+ int error; /* used by json_parse_checked; legacy parsing stays permissive */
} jparser;
/* tetto di annidamento: gli header safetensors / config sono piatti (profondita'
@@ -43,7 +44,13 @@ static char *j_dup(jparser *p, const char *b, int n) {
return d;
}
-static void j_ws(jparser *p) { while (*p->s && isspace((unsigned char)*p->s)) p->s++; }
+static void j_ws(jparser *p) {
+ while (*p->s && isspace((unsigned char)*p->s)) {
+ char c=*p->s++;
+ /* Preserve legacy consumption, but only JSON whitespace is valid. */
+ if (c!=' ' && c!='\t' && c!='\r' && c!='\n') p->error=1;
+ }
+}
static jval *j_new(jtype t) {
jval *v = (jval *)calloc(1, sizeof(jval));
@@ -52,12 +59,12 @@ static jval *j_new(jtype t) {
static jval *j_parse_val(jparser *p);
-static char *j_parse_str_raw(jparser *p) {
+static char *j_parse_str_raw(jparser *p, int is_key) {
/* SEC (GHSA-2qrj): fail closed if not actually at a quote. The old comment
* "assume *p->s == '\"'" was violated on the object-key path, and the
* unconditional p->s++ would step past the buffer's NUL terminator and scan
* adjacent heap (OOB read leaking into tensor names). */
- if (*p->s != '"') return j_dup(p, "", 0);
+ if (*p->s != '"') { p->error = 1; return j_dup(p, "", 0); }
p->s++;
/* buffer su heap che CRESCE: niente troncamento silenzioso a 64KB (le stringhe
* lunghe di tokenizer.json/config venivano tagliate) e niente 64KB di stack. */
@@ -67,6 +74,7 @@ static char *j_parse_str_raw(jparser *p) {
if (!tmp) { fprintf(stderr, "OOM parsing JSON string\n"); exit(1); } } tmp[n++] = (char)(ch); }while(0)
while (*p->s && *p->s != '"') {
char c = *p->s++;
+ if ((unsigned char)c < 0x20) p->error = 1;
if (c == '\\' && *p->s) {
char e = *p->s++;
switch (e) {
@@ -75,9 +83,12 @@ static char *j_parse_str_raw(jparser *p) {
case 'f': c = '\f'; break; case '/': c = '/'; break;
case '\\': c = '\\'; break; case '"': c = '"'; break;
case 'u': { /* \uXXXX -> codepoint UTF-8 (con coppie surrogate) */
- if (!p->s[0]||!p->s[1]||!p->s[2]||!p->s[3]) { c='?'; break; } /* \u troncato: non leggere oltre il NUL */
+ if (!p->s[0]||!p->s[1]||!p->s[2]||!p->s[3]) { p->error=1; c='?'; break; } /* \u troncato: non leggere oltre il NUL */
+ for (int i=0; i<4; i++) if (!isxdigit((unsigned char)p->s[i])) p->error=1;
unsigned cp = (unsigned)strtoul((char[]){p->s[0],p->s[1],p->s[2],p->s[3],0}, NULL, 16);
p->s += 4;
+ /* Checked keys must retain their identity in C-string lookups. */
+ if (is_key && cp==0) p->error=1;
if (cp >= 0xD800 && cp <= 0xDBFF && p->s[0]=='\\' && p->s[1]=='u'
&& p->s[2] && p->s[3] && p->s[4] && p->s[5]) {
unsigned lo = (unsigned)strtoul((char[]){p->s[2],p->s[3],p->s[4],p->s[5],0}, NULL, 16);
@@ -89,13 +100,13 @@ static char *j_parse_str_raw(jparser *p) {
else { J_PUT(0xF0|(cp>>18)); J_PUT(0x80|((cp>>12)&0x3F)); J_PUT(0x80|((cp>>6)&0x3F)); J_PUT(0x80|(cp&0x3F)); }
continue;
}
- default: c = e; break;
+ default: p->error = 1; c = e; break;
}
}
J_PUT(c);
}
#undef J_PUT
- if (*p->s == '"') p->s++;
+ if (*p->s == '"') p->s++; else p->error = 1;
char *out = j_dup(p, tmp, (int)n); free(tmp);
return out;
}
@@ -103,9 +114,9 @@ static char *j_parse_str_raw(jparser *p) {
static jval *j_parse_val(jparser *p) {
j_ws(p);
char c = *p->s;
- if (c == '"') { jval *v = j_new(J_STR); v->str = j_parse_str_raw(p); return v; }
+ if (c == '"') { jval *v = j_new(J_STR); v->str = j_parse_str_raw(p,0); return v; }
if (c == '{') {
- if (++p->depth > J_MAX_DEPTH) { p->depth--; return j_new(J_NULL); }
+ if (++p->depth > J_MAX_DEPTH) { p->error=1; p->depth--; return j_new(J_NULL); }
p->s++; jval *v = j_new(J_OBJ);
int cap = 8;
v->keys = malloc(cap * sizeof(char*));
@@ -116,9 +127,9 @@ static jval *j_parse_val(jparser *p) {
if (*p->s == '}') { p->s++; p->depth--; return v; }
for (;;) {
j_ws(p);
- if (*p->s != '"') break; /* SEC (GHSA-2qrj): object key must be a quoted string; stop on malformed input */
- char *key = j_parse_str_raw(p);
- j_ws(p); if (*p->s == ':') p->s++;
+ if (*p->s != '"') { p->error=1; break; } /* SEC (GHSA-2qrj): object key must be a quoted string; stop on malformed input */
+ char *key = j_parse_str_raw(p,1);
+ j_ws(p); if (*p->s == ':') p->s++; else p->error=1;
jval *val = j_parse_val(p);
if (v->len == cap) { cap *= 2;
char **nk = (char**)realloc(v->keys, cap*sizeof(char*));
@@ -131,13 +142,14 @@ static jval *j_parse_val(jparser *p) {
j_ws(p);
if (*p->s == ',') { p->s++; continue; }
if (*p->s == '}') { p->s++; break; }
+ p->error = 1;
break;
}
p->depth--;
return v;
}
if (c == '[') {
- if (++p->depth > J_MAX_DEPTH) { p->depth--; return j_new(J_NULL); }
+ if (++p->depth > J_MAX_DEPTH) { p->error=1; p->depth--; return j_new(J_NULL); }
p->s++; jval *v = j_new(J_ARR);
int cap = 8;
v->kids = malloc(cap * sizeof(jval*));
@@ -154,6 +166,7 @@ static jval *j_parse_val(jparser *p) {
j_ws(p);
if (*p->s == ',') { p->s++; continue; }
if (*p->s == ']') { p->s++; break; }
+ p->error = 1;
break;
}
p->depth--;
@@ -163,12 +176,28 @@ static jval *j_parse_val(jparser *p) {
if (c == 'f' && !strncmp(p->s, "false", 5)) { p->s += 5; jval *v = j_new(J_BOOL); v->boolean = 0; return v; }
if (c == 'n' && !strncmp(p->s, "null", 4)) { p->s += 4; return j_new(J_NULL); }
/* numero */
- { char *end; double d = strtod(p->s, &end); p->s = end; jval *v = j_new(J_NUM); v->num = d; return v; }
+ { const char *start=p->s, *q=start;
+ if (*q=='-') q++;
+ if (*q=='0') q++;
+ else if (*q>='1' && *q<='9') { do { q++; } while (*q>='0' && *q<='9'); }
+ else p->error=1;
+ if (*q=='.') {
+ q++; if (*q<'0' || *q>'9') p->error=1;
+ while (*q>='0' && *q<='9') q++;
+ }
+ if (*q=='e' || *q=='E') {
+ q++; if (*q=='+' || *q=='-') q++;
+ if (*q<'0' || *q>'9') p->error=1;
+ while (*q>='0' && *q<='9') q++;
+ }
+ char *end; double d = strtod(start, &end);
+ if (end==start || end!=q) p->error=1;
+ p->s = end; jval *v = j_new(J_NUM); v->num = d; return v; }
}
/* API */
static jval *json_parse(const char *text, char **arena_out) {
- jparser p = { text, NULL, 0, 0, 0 };
+ jparser p = { text, NULL, 0, 0, 0, 0 };
jval *v = j_parse_val(&p);
if (arena_out) *arena_out = p.arena; else free(p.arena);
return v;
@@ -195,4 +224,16 @@ static void json_free(jval *v) {
free(v);
}
+/* An oracle must not accept a partial tree from a truncated/malformed file.
+ * Keep the existing API's permissive behavior for other engine consumers. */
+static jval *json_parse_checked(const char *text) {
+ if (!text) return NULL;
+ jparser p = { text, NULL, 0, 0, 0, 0 };
+ jval *v = j_parse_val(&p);
+ j_ws(&p);
+ if (p.error || *p.s) { json_free(v); v=NULL; }
+ free(p.arena);
+ return v;
+}
+
#endif
diff --git a/c/oracle.h b/c/oracle.h
new file mode 100644
index 00000000..7790cb92
--- /dev/null
+++ b/c/oracle.h
@@ -0,0 +1,97 @@
+/* Validation for the GLM reference file; no model or Python dependency. */
+#ifndef COLI_ORACLE_H
+#define COLI_ORACLE_H
+#include <errno.h>
+#include <math.h>
+#include "json.h"
+
+typedef struct {
+ int *prompt, *full, *tf;
+ int np, nfull;
+} OracleRef;
+
+static void oracle_ref_free(OracleRef *r) {
+ free(r->prompt); free(r->full); free(r->tf);
+ memset(r, 0, sizeof(*r));
+}
+
+static int *oracle_read_ids(jval *root, const char *key, int vocab, int *n) {
+ jval *a=json_get(root,key);
+ *n=0;
+ if (!a || a->t!=J_ARR || a->len<1) {
+ fprintf(stderr,"[ORACLE] %s must be a nonempty token array\n",key);
+ return NULL;
+ }
+ int *ids=malloc((size_t)a->len*sizeof(*ids));
+ if (!ids) { fprintf(stderr,"[ORACLE] out of memory reading %s\n",key); return NULL; }
+ for (int i=0; i<a->len; i++) {
+ jval *v=a->kids[i];
+ if (v->t!=J_NUM || !isfinite(v->num) || v->num<0 || v->num>=vocab ||
+ v->num!=floor(v->num)) {
+ fprintf(stderr,"[ORACLE] %s[%d] must be an integer token in [0,%d)\n",key,i,vocab);
+ free(ids); return NULL;
+ }
+ ids[i]=(int)v->num;
+ }
+ *n=a->len;
+ return ids;
+}
+
+static int oracle_ref_parse(const char *text, int vocab, int teacher_forcing, OracleRef *r) {
+ memset(r,0,sizeof(*r));
+ jval *root=json_parse_checked(text);
+ if (!root || root->t!=J_OBJ || vocab<1) {
+ fprintf(stderr,"[ORACLE] invalid reference JSON or vocabulary\n");
+ json_free(root); return 0;
+ }
+ /* Duplicate keys could otherwise silently select a stale prediction array. */
+ const char *keys[]={"prompt_ids","full_ids","tf_pred"};
+ for (int k=0; k<3; k++) {
+ int count=0;
+ for (int i=0; i<root->len; i++) if (!strcmp(root->keys[i],keys[k])) count++;
+ if (count>1) {
+ fprintf(stderr,"[ORACLE] duplicate reference field %s\n",keys[k]);
+ goto fail;
+ }
+ }
+ r->prompt=oracle_read_ids(root,"prompt_ids",vocab,&r->np);
+ r->full=oracle_read_ids(root,"full_ids",vocab,&r->nfull);
+ if (!r->prompt || !r->full) goto fail;
+ if (r->nfull<r->np || (!teacher_forcing && r->nfull==r->np) ||
+ memcmp(r->prompt,r->full,(size_t)r->np*sizeof(int))) {
+ fprintf(stderr,"[ORACLE] full_ids must start with prompt_ids and include the compared tokens\n");
+ goto fail;
+ }
+ if (teacher_forcing) {
+ int ntf=0;
+ r->tf=oracle_read_ids(root,"tf_pred",vocab,&ntf);
+ if (!r->tf) goto fail;
+ if (ntf!=r->nfull) {
+ fprintf(stderr,"[ORACLE] tf_pred length %d != full_ids length %d\n",ntf,r->nfull);
+ goto fail;
+ }
+ }
+ json_free(root); return 1;
+fail:
+ json_free(root); oracle_ref_free(r); return 0;
+}
+
+static int oracle_logits_finite(const float *lo, int vocab) {
+ for (int i=0; i<vocab; i++) if (!isfinite(lo[i])) return 0;
+ return 1;
+}
+
+/* Exact by default. A configured budget must leave at least one required match. */
+static int oracle_tf_allowance(const char *text, int total, int *allowed) {
+ *allowed=0;
+ if (total<1) return 0;
+ if (!text) return 1;
+ if (*text<'0' || *text>'9') return 0;
+ char *end;
+ errno=0;
+ long value=strtol(text,&end,10);
+ if (errno==ERANGE || *end || value<0 || value>=total) return 0;
+ *allowed=(int)value;
+ return 1;
+}
+#endif
diff --git a/c/tests/test_glm_oracle.py b/c/tests/test_glm_oracle.py
new file mode 100644
index 00000000..14003ec6
--- /dev/null
+++ b/c/tests/test_glm_oracle.py
@@ -0,0 +1,271 @@
+"""Real-process GLM oracle exit-status regression; stdlib only.
+
+Discovery skips when the generated fixture is absent. Direct execution (as in
+CI) requires it, so a missing engine/fixture cannot silently pass the gate.
+"""
+import argparse
+import copy
+import json
+import os
+from pathlib import Path
+import re
+import shutil
+import struct
+import subprocess
+import tempfile
+import unittest
+
+C_DIR = Path(__file__).resolve().parents[1]
+ENGINE = C_DIR / ("colibri.exe" if os.name == "nt" else "colibri")
+SNAP = C_DIR / "glm_tiny"
+REFERENCE = C_DIR / "ref_glm.json"
+
+
+class GlmOracleTest(unittest.TestCase):
+ @classmethod
+ def setUpClass(cls):
+ if not (ENGINE.is_file() and (SNAP / "model.safetensors").is_file()
+ and REFERENCE.is_file()):
+ raise unittest.SkipTest("build colibri and generate the GLM tiny oracle first")
+ cls.reference = json.loads(REFERENCE.read_text(encoding="utf-8"))
+ cls.vocab = json.loads((SNAP / "config.json").read_text(encoding="utf-8"))["vocab_size"]
+ # Keep the independent fixture for the numerical gate. Use observed TF
+ # predictions only to place exit-status tests at exact mismatch counts,
+ # even on a host with one or two floating-point near ties.
+ cls.tf_reference = cls.observed_tf_reference(cls.reference)
+
+ @classmethod
+ def run_oracle(cls, reference=None, *, tf=True, strict="1", raw=None,
+ snapshot=None, extra_env=None):
+ # Do not inherit serving/sampling/offload knobs from an interactive shell.
+ keep = {"PATH", "SYSTEMROOT", "WINDIR", "TEMP", "TMP", "HOME", "USERPROFILE",
+ "LD_LIBRARY_PATH", "DYLD_LIBRARY_PATH", "ASAN_OPTIONS", "UBSAN_OPTIONS"}
+ env = {k: v for k, v in os.environ.items() if k in keep}
+ env.update(SNAP=str(SNAP if snapshot is None else snapshot), COLI_TEMP="0", OMP_NUM_THREADS="4")
+ if tf:
+ env["TF"] = "1"
+ if strict is not None:
+ env["ORACLE_STRICT"] = strict
+ env.update(extra_env or {})
+ with tempfile.TemporaryDirectory(prefix="colibri-oracle-") as tmp:
+ path = Path(tmp) / "ref.json"
+ path.write_text(raw if raw is not None else json.dumps(
+ cls.reference if reference is None else reference), encoding="utf-8")
+ env["REF"] = str(path)
+ return subprocess.run([str(ENGINE), "64", "16", "16"], cwd=C_DIR,
+ env=env, capture_output=True, text=True,
+ encoding="utf-8", errors="replace", timeout=60)
+
+ @classmethod
+ def observed_tf_reference(cls, reference):
+ result = cls.run_oracle(reference, strict="0")
+ if result.returncode:
+ raise AssertionError(result.stdout + result.stderr)
+ score = re.search(r"C vs oracle: (\d+)/(\d+) positions", result.stdout)
+ if score is None or int(score[2]) != len(reference["tf_pred"]):
+ raise AssertionError("missing TF score: " + result.stdout + result.stderr)
+ ref = copy.deepcopy(reference)
+ mismatches = re.findall(r"\[ORACLE\] mismatch pos=(\d+) expected=(\d+) got=(-?\d+)", result.stderr)
+ if len(mismatches) != int(score[2]) - int(score[1]):
+ raise AssertionError("incomplete TF diagnostics: " + result.stderr)
+ for pos, expected, got in mismatches:
+ pos, expected, got = int(pos), int(expected), int(got)
+ if not (0 <= pos < len(ref["tf_pred"]) and 0 <= got < cls.vocab
+ and ref["tf_pred"][pos] == expected):
+ raise AssertionError("invalid TF prediction: " + result.stderr)
+ ref["tf_pred"][pos] = got
+ return ref
+
+ def assert_failed(self, result):
+ self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
+
+ def corrupted(self, tf, count=1):
+ ref = copy.deepcopy(self.tf_reference if tf else self.reference)
+ if tf:
+ for i in range(count):
+ ref["tf_pred"][i] = (ref["tf_pred"][i] + 1) % self.vocab
+ else:
+ ref["full_ids"][-1] = (ref["full_ids"][-1] + 1) % self.vocab
+ return ref
+
+ def test_valid_reference_passes_both_modes(self):
+ for tf in (True, False):
+ with self.subTest(tf=tf):
+ result = self.run_oracle(tf=tf, extra_env={"ORACLE_TF_MAX_MISMATCHES": "2"})
+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+ total = len(self.reference["full_ids"])
+ new = total - len(self.reference["prompt_ids"])
+ if tf:
+ score = re.search(r"C vs oracle: (\d+)/(\d+) positions", result.stdout)
+ self.assertIsNotNone(score, result.stdout)
+ self.assertEqual(int(score[2]), total)
+ self.assertGreaterEqual(int(score[1]), total - 2)
+ else:
+ self.assertIn(f"Matching tokens: {new}/{new}", result.stdout)
+
+ def test_exact_mode_rejects_one_wrong_prediction(self):
+ for tf in (True, False):
+ with self.subTest(tf=tf):
+ self.assert_failed(self.run_oracle(self.corrupted(tf), tf=tf))
+
+ def test_teacher_forcing_allowance_boundary(self):
+ total = len(self.reference["tf_pred"])
+ for wrong in (0, 1, 2, 3):
+ with self.subTest(wrong=wrong):
+ result = self.run_oracle(self.corrupted(True, wrong),
+ extra_env={"ORACLE_TF_MAX_MISMATCHES": "2"})
+ self.assertEqual(result.returncode, int(wrong > 2), result.stdout + result.stderr)
+ self.assertIn(f"{total - wrong}/{total} positions", result.stdout)
+
+ def test_explicit_zero_allowance_is_exact(self):
+ for wrong in (0, 1):
+ with self.subTest(wrong=wrong):
+ result = self.run_oracle(self.corrupted(True, wrong),
+ extra_env={"ORACLE_TF_MAX_MISMATCHES": "0"})
+ self.assertEqual(result.returncode, wrong, result.stdout + result.stderr)
+
+ def test_teacher_forcing_allowance_does_not_relax_greedy(self):
+ self.assert_failed(self.run_oracle(self.corrupted(False), tf=False,
+ extra_env={"ORACLE_TF_MAX_MISMATCHES": "2"}))
+
+ def test_invalid_allowance_fails_before_comparison(self):
+ total = len(self.reference["tf_pred"])
+ for value in ("", "-1", "+2", " 2", "2 ", "2x", "2.0", str(total), str(total + 1), "9" * 40):
+ with self.subTest(value=value):
+ result = self.run_oracle(extra_env={"ORACLE_TF_MAX_MISMATCHES": value})
+ self.assert_failed(result)
+ self.assertIn("ORACLE_TF_MAX_MISMATCHES", result.stderr)
+ self.assertNotIn("C vs oracle:", result.stdout)
+
+ def test_diagnostic_mode_keeps_report_only_exit_status(self):
+ for tf in (True, False):
+ for strict in (None, "0"):
+ with self.subTest(tf=tf, strict=strict):
+ result = self.run_oracle(self.corrupted(tf, 3), tf=tf, strict=strict,
+ extra_env={"ORACLE_TF_MAX_MISMATCHES": "2"})
+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+
+ def test_missing_short_and_long_tf_predictions_fail_cleanly(self):
+ for value in (None, [], self.reference["tf_pred"][:-1],
+ self.reference["tf_pred"] + [0], "not an array"):
+ with self.subTest(value=value):
+ ref = copy.deepcopy(self.reference)
+ if value is None:
+ del ref["tf_pred"]
+ else:
+ ref["tf_pred"] = value
+ self.assert_failed(self.run_oracle(ref, extra_env={"ORACLE_TF_MAX_MISMATCHES": "2"}))
+
+ def test_invalid_token_ids_fail_before_inference(self):
+ for field in ("prompt_ids", "full_ids", "tf_pred"):
+ for value in (-1, self.vocab, 1.5, True, "1", None, 1e100):
+ with self.subTest(field=field, value=value):
+ ref = copy.deepcopy(self.reference)
+ ref[field][0] = value
+ result = self.run_oracle(ref)
+ self.assert_failed(result)
+ self.assertIn("[ORACLE]", result.stderr)
+
+ def test_truncated_json_and_mismatched_prompt_fail(self):
+ raw = json.dumps(self.reference)
+ for invalid in (raw[:-1], raw[:-2], raw + " garbage", raw + "\0garbage"):
+ with self.subTest(raw=invalid[-30:]):
+ self.assert_failed(self.run_oracle(raw=invalid))
+ ref = copy.deepcopy(self.reference)
+ ref["prompt_ids"][0] = (ref["prompt_ids"][0] + 1) % self.vocab
+ self.assert_failed(self.run_oracle(ref, tf=False))
+
+ def test_empty_continuation_fails(self):
+ ref = copy.deepcopy(self.reference)
+ ref["full_ids"] = ref["prompt_ids"][:]
+ self.assert_failed(self.run_oracle(ref, tf=False))
+
+ def test_escaped_nul_cannot_alias_reference_keys(self):
+ for field in ("prompt_ids", "full_ids", "tf_pred"):
+ ref = copy.deepcopy(self.reference)
+ ref[field + "\0ignored"] = ref.pop(field)
+ for tf in (True, False):
+ for strict in ("1", None):
+ with self.subTest(field=field, tf=tf, strict=strict):
+ self.assert_failed(self.run_oracle(ref, tf=tf, strict=strict))
+
+ def test_non_json_whitespace_fails(self):
+ raw = json.dumps(self.reference)
+ for whitespace in ("\f", "\v"):
+ for invalid in (whitespace + raw, raw.replace(":", ":" + whitespace, 1),
+ raw.replace("[", "[" + whitespace, 1), raw + whitespace):
+ for tf in (True, False):
+ for strict in ("1", None):
+ with self.subTest(raw=invalid, tf=tf, strict=strict):
+ self.assert_failed(self.run_oracle(raw=invalid, tf=tf, strict=strict))
+
+ def test_other_modes_cannot_bypass_strict_comparison(self):
+ for mode in ("REPLAY", "SERVE", "SCORE", "ABLATE_SCORE", "EXPERT_WORKER",
+ "I4_ACC512_TEST", "I3_AVX512_TEST", "COLI_ANS_PACK", "COLI_PROMPT"):
+ with self.subTest(mode=mode):
+ result = self.run_oracle(extra_env={mode: "1"})
+ self.assert_failed(result)
+ self.assertIn("ORACLE_STRICT", result.stderr)
+
+ def test_nonfinite_model_logits_fail_both_modes(self):
+ # The tiny generator stores an F32 lm_head in one safetensors shard.
+ # Poison a single output row, including when it would not win argmax.
+ with tempfile.TemporaryDirectory(prefix="colibri-oracle-weights-") as tmp:
+ snapshot = Path(tmp)
+ shutil.copy2(SNAP / "config.json", snapshot / "config.json")
+ weights = (SNAP / "model.safetensors").read_bytes()
+ header_size = struct.unpack_from("<Q", weights)[0]
+ header = json.loads(weights[8:8 + header_size])
+ tensor = header["lm_head.weight"]
+ self.assertEqual(tensor["dtype"], "F32")
+ start = 8 + header_size + tensor["data_offsets"][0]
+ row_bytes = tensor["shape"][1] * 4
+ for value in (float("nan"), float("inf"), -float("inf")):
+ poisoned = bytearray(weights)
+ poisoned[start:start + row_bytes] = struct.pack("<f", value) * tensor["shape"][1]
+ (snapshot / "model.safetensors").write_bytes(poisoned)
+ for tf in (True, False):
+ with self.subTest(value=value, tf=tf):
+ result = self.run_oracle(tf=tf, snapshot=snapshot,
+ extra_env={"ORACLE_TF_MAX_MISMATCHES": "2"})
+ self.assert_failed(result)
+ self.assertIn("non-finite logits", result.stderr)
+
+ def test_one_nonfinite_position_is_not_an_allowed_mismatch(self):
+ # Poison only the last position of a short causal sequence. Its single
+ # mismatch fits within the budget, but non-finite output must still fail.
+ ref = copy.deepcopy(self.reference)
+ ref["prompt_ids"] = ref["full_ids"][:2]
+ last = next(token for token in range(self.vocab) if token not in ref["prompt_ids"])
+ ref["full_ids"] = ref["prompt_ids"] + [last]
+ ref["tf_pred"] = ref["tf_pred"][:3]
+ ref = self.observed_tf_reference(ref)
+ with tempfile.TemporaryDirectory(prefix="colibri-oracle-position-") as tmp:
+ snapshot = Path(tmp)
+ shutil.copy2(SNAP / "config.json", snapshot / "config.json")
+ weights = bytearray((SNAP / "model.safetensors").read_bytes())
+ header_size = struct.unpack_from("<Q", weights)[0]
+ header = json.loads(weights[8:8 + header_size])
+ tensor = header["model.embed_tokens.weight"]
+ self.assertEqual(tensor["dtype"], "F32")
+ row_bytes = tensor["shape"][1] * 4
+ start = 8 + header_size + tensor["data_offsets"][0] + last * row_bytes
+ weights[start:start + row_bytes] = struct.pack("<f", float("nan")) * tensor["shape"][1]
+ (snapshot / "model.safetensors").write_bytes(weights)
+ result = self.run_oracle(ref, snapshot=snapshot,
+ extra_env={"ORACLE_TF_MAX_MISMATCHES": "2"})
+ self.assert_failed(result)
+ self.assertIn("2/3 positions", result.stdout)
+ self.assertIn("non-finite logits at teacher-forcing position 2", result.stderr)
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--engine", type=Path, default=ENGINE)
+ parser.add_argument("--snapshot", type=Path, default=SNAP)
+ parser.add_argument("--reference", type=Path, default=REFERENCE)
+ args = parser.parse_args()
+ ENGINE, SNAP, REFERENCE = (p.resolve() for p in (args.engine, args.snapshot, args.reference))
+ if not (ENGINE.is_file() and (SNAP / "model.safetensors").is_file() and REFERENCE.is_file()):
+ parser.error("build colibri and generate the GLM tiny oracle first")
+ unittest.main(argv=[__file__, "-v"])
diff --git a/c/tests/test_json.c b/c/tests/test_json.c
index 56941a90..da0a4d19 100644
--- a/c/tests/test_json.c
+++ b/c/tests/test_json.c
@@ -30,6 +30,42 @@ int main(void) {
CHECK(values->kids[1]->num == -2.5);
CHECK(values->kids[2]->num == 300.0);
CHECK(strcmp(json_get(root, "unicode")->str, "λ 🚀") == 0);
+ json_free(root);
+
+ /* Checked key lookup must not alias a name through a decoded NUL. The
+ * legacy API still returns its original permissive representation. */
+ const char *nul_key="{\"x\\u0000ignored\":1}";
+ CHECK(json_parse_checked(nul_key) == NULL);
+ root=json_parse(nul_key,NULL);
+ CHECK(root && json_get(root,"x") && json_get(root,"x")->num == 1.0);
+ json_free(root);
+
+ /* Legal escapes, including NUL in a string value, remain valid. */
+ root=json_parse_checked("{\"\\u0078\":\"a\\u0000b\\f\\u000b\"}");
+ CHECK(root && json_get(root,"x"));
+ CHECK(memcmp(json_get(root,"x")->str,"a\0b\f\v",5) == 0);
+ json_free(root);
+
+ const char whitespace[]=" \t\r\n\f\v";
+ const char *positions[]={
+ "%c{\"x\":[1]}", "{%c\"x\":[1]}", "{\"x\"%c:[1]}",
+ "{\"x\":%c[1]}", "{\"x\":[%c1]}", "{\"x\":[1%c]}",
+ "{\"x\":[1]%c}", "{\"x\":[1]}%c",
+ };
+ for (size_t i=0; i<sizeof(whitespace)-1; i++) {
+ for (size_t j=0; j<sizeof(positions)/sizeof(positions[0]); j++) {
+ char input[64];
+ snprintf(input,sizeof(input),positions[j],whitespace[i]);
+ root=json_parse_checked(input);
+ CHECK((root != NULL) == (i < 4));
+ json_free(root);
+ root=json_parse(input,NULL);
+ values=json_get(root,"x");
+ CHECK(values && values->t == J_ARR && values->len == 1);
+ CHECK(values->kids[0]->num == 1.0);
+ json_free(root);
+ }
+ }
puts("json tests: ok");
return 0;
diff --git a/c/tests/test_oracle.c b/c/tests/test_oracle.c
new file mode 100644
index 00000000..57b23cbf
--- /dev/null
+++ b/c/tests/test_oracle.c
@@ -0,0 +1,83 @@
+#include <stdio.h>
+#include <limits.h>
+#include "../oracle.h"
+
+#define CHECK(c) do { if (!(c)) { \
+ fprintf(stderr,"%s:%d: %s\n",__FILE__,__LINE__,#c); return 1; \
+} } while (0)
+
+int main(void) {
+ OracleRef r;
+ const char *good="{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[2,3]}";
+ CHECK(oracle_ref_parse(good,4,1,&r));
+ CHECK(r.np==1 && r.nfull==2 && r.prompt[0]==1 && r.full[1]==2 && r.tf[1]==3);
+ oracle_ref_free(&r);
+ CHECK(oracle_ref_parse("{\"prompt_ids\":[1],\"full_ids\":[1,2]}",4,0,&r));
+ CHECK(r.tf==NULL);
+ oracle_ref_free(&r);
+
+ /* Missing/truncated/wrongly typed predictions used to reach tf[i] unchecked. */
+ const char *bad[]={
+ "{}",
+ "[]",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2]}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[2]}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[2,3,1]}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":{\"x\":2,\"y\":3}}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[2,3]",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[2,3}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[2,3,]}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[2,3]} garbage",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[2 3]}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\" [2,3]}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[2,3],\"tf_pred\":[1,1]}",
+ "{\"prompt_ids\":[1,2],\"full_ids\":[1],\"tf_pred\":[2]}",
+ "{\"prompt_ids\":[2],\"full_ids\":[1,2],\"tf_pred\":[2,3]}",
+ "{\"prompt_ids\":[],\"full_ids\":[1,2],\"tf_pred\":[2,3]}",
+ "{\"prompt_ids\":[-1],\"full_ids\":[-1,2],\"tf_pred\":[2,3]}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,4],\"tf_pred\":[2,3]}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[true,3]}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[\"2\",3]}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[2.5,3]}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[1e999,3]}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[NaN,3]}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[+2,3]}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[02,3]}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[2.,3]}",
+ "{\"prompt_ids\":[1],\"full_ids\":[1,2],\"tf_pred\":[0x2,3]}",
+ };
+ for (size_t i=0; i<sizeof(bad)/sizeof(bad[0]); i++) {
+ CHECK(!oracle_ref_parse(bad[i],4,1,&r));
+ CHECK(!r.prompt && !r.full && !r.tf);
+ }
+ CHECK(!oracle_ref_parse(good,0,1,&r));
+ CHECK(!oracle_ref_parse(NULL,4,1,&r));
+ CHECK(!oracle_ref_parse("{\"prompt_ids\":[1],\"full_ids\":[1]}",4,0,&r));
+ CHECK(oracle_ref_parse("{\"prompt_ids\":[1],\"full_ids\":[1],\"tf_pred\":[2]}",4,1,&r));
+ oracle_ref_free(&r);
+
+ /* Even a non-winning NaN/Inf must invalidate the model output. */
+ float logits[]={2.0f,1.0f,-3.0f};
+ CHECK(oracle_logits_finite(logits,3));
+ logits[2]=NAN; CHECK(!oracle_logits_finite(logits,3));
+ logits[2]=INFINITY; CHECK(!oracle_logits_finite(logits,3));
+ logits[2]=-INFINITY; CHECK(!oracle_logits_finite(logits,3));
+
+ int allowed=-1;
+ CHECK(oracle_tf_allowance(NULL,32,&allowed) && allowed==0);
+ CHECK(oracle_tf_allowance("0",32,&allowed) && allowed==0);
+ CHECK(oracle_tf_allowance("2",32,&allowed) && allowed==2);
+ CHECK(oracle_tf_allowance("31",32,&allowed) && allowed==31);
+ CHECK(oracle_tf_allowance("0",1,&allowed) && allowed==0);
+ CHECK(!oracle_tf_allowance("2",2,&allowed));
+ CHECK(!oracle_tf_allowance(NULL,0,&allowed));
+ const char *bad_allowance[]={"", "-1", "+2", " 2", "2 ", "2x", "2.0", "32", "33",
+ "9999999999999999999999999999999999999999"};
+ for (size_t i=0; i<sizeof(bad_allowance)/sizeof(bad_allowance[0]); i++)
+ CHECK(!oracle_tf_allowance(bad_allowance[i],32,&allowed) && allowed==0);
+ char large[32];
+ snprintf(large,sizeof(large),"%d",INT_MAX-1);
+ CHECK(oracle_tf_allowance(large,INT_MAX,&allowed) && allowed==INT_MAX-1);
+ puts("oracle input and finite-logit tests: ok");
+ return 0;
+}
diff --git a/c/tests/test_spec_decode_state.c b/c/tests/test_spec_decode_state.c
index 1360628d..291e7697 100644
--- a/c/tests/test_spec_decode_state.c
+++ b/c/tests/test_spec_decode_state.c
@@ -247,6 +247,39 @@ static int test_eos_and_non_limit_stop(void) {
return 0;
}
+static int test_oracle_reports_partial_generation(void) {
+ const int next[V] = {1, 2, 3, 4, 5, 6, 7, 0};
+ Toy t;
+ toy_init(&t, next);
+ reset_decode_globals(0);
+ /* Oracle comparison must not read unwritten output after a stop/interrupt. */
+ int prompt[] = {0, 1};
+ int out[5] = {-1, -1, -1, -1, -1}, finite = 0;
+ g_stop[0] = 3; g_nstop = 1;
+ CHECK(generate(&t.m, prompt, 2, 3, out, &finite) == 1);
+ CHECK(finite && out[2] == 2 && out[3] == -1 && out[4] == -1);
+ g_intr = 1;
+ CHECK(generate(&t.m, prompt, 2, 3, out, &finite) == 0);
+ reset_decode_globals(0);
+ toy_free(&t);
+ return 0;
+}
+
+static int test_teacher_forcing_reports_one_nonfinite_position(void) {
+ const int next[V] = {1, 2, 3, 4, 5, 6, 7, 0};
+ Toy t;
+ toy_init(&t, next);
+ reset_decode_globals(0);
+ int ids[] = {0, 1, 2}, pred[3];
+ CHECK(forward_all(&t.m, ids, 3, pred, NULL));
+ CHECK(pred[0] == 1 && pred[1] == 2 && pred[2] == 3);
+ t.m.embed.qf[2 * D + 2] = NAN;
+ CHECK(!forward_all(&t.m, ids, 3, pred, NULL));
+ CHECK(pred[0] == 1 && pred[1] == 2 && pred[2] == -1);
+ toy_free(&t);
+ return 0;
+}
+
int main(void) {
CHECK(test_ngen_zero_and_one_shot() == 0);
CHECK(test_stateful_ngen_and_more() == 0);
@@ -254,6 +287,8 @@ int main(void) {
CHECK(test_emit_carries_scoring_logits_for_accepted_drafts() == 0);
CHECK(test_accepted_speculative_tokens() == 0);
CHECK(test_eos_and_non_limit_stop() == 0);
+ CHECK(test_oracle_reports_partial_generation() == 0);
+ CHECK(test_teacher_forcing_reports_one_nonfinite_position() == 0);
puts("spec_decode state tests: ok");
return 0;
}
diff --git a/c/tools/make_glm_oracle.py b/c/tools/make_glm_oracle.py
index b8c05b50..97c07568 100644
--- a/c/tools/make_glm_oracle.py
+++ b/c/tools/make_glm_oracle.py
@@ -30,7 +30,7 @@ same quantizer), so the engine's decode reproduces the reference token-exactly.
Regeneration (run from c/):
python3 tools/make_glm_oracle.py --fmt6 # -> glm_tiny_fmt6/ (model.safetensors, config.json, ref_glm.json)
python3 tools/make_glm_oracle.py --fmt4 # -> glm_tiny_fmt4/
- # verify: SNAP=./glm_tiny_fmt6 REF=./glm_tiny_fmt6/ref_glm.json TF=1 COLI_TEMP=0 ./colibri 64 16 16"""
+ # verify: SNAP=./glm_tiny_fmt6 REF=./glm_tiny_fmt6/ref_glm.json TF=1 COLI_TEMP=0 ORACLE_STRICT=1 ./colibri 64 16 16"""
import json, sys, argparse
from pathlib import Path
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
index b3d36252..ef69e1bc 100644
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@ -2,7 +2,7 @@
Reference for the environment variables read by the colibrì engine.
-**Generated from `dev @ def8419`** by scanning every `getenv()` / `getenv_utf8()` site in `c/*.c`, `c/*.h`, `c/*.cu` and `c/*.mm`. Defaults and behavior are taken from the source; see [MAINTAINING-DOCS.md](MAINTAINING-DOCS.md) to regenerate this after the code changes.
+**Baseline generated from `dev @ def8419`** by scanning every `getenv()` / `getenv_utf8()` site in `c/*.c`, `c/*.h`, `c/*.cu` and `c/*.mm`. Individual entries are also maintained with their owning source. Defaults and behavior are taken from the source; see [MAINTAINING-DOCS.md](MAINTAINING-DOCS.md) to regenerate the full inventory after the code changes.
## Which program reads these?
@@ -292,6 +292,8 @@ These are for testing, benchmarking, or internal use — not part of the everyda
| `REF` / `REF_FORCE` | `ref_glm.json` | Reference-output comparison mode. |
| `REPLAY` | unset | Replay mode. |
| `TF` | unset | Teacher-forcing mode. |
+| `ORACLE_STRICT` | unset (off) | `colibri` only, env-only. `=1` makes failed teacher-forcing (`TF`) and greedy oracle comparisons exit with status 1. Token-exact by default; only TF can use the mismatch allowance below. Non-finite logits and incomplete generation always fail strict mode; modes that bypass comparison are rejected. Unset or `0` keeps completed comparisons report-only. Invalid reference JSON/arrays fail regardless of this setting. See [CONTRIBUTING.md](../CONTRIBUTING.md) for the strict oracle commands. |
+| `ORACLE_TF_MAX_MISMATCHES` | `0` | `colibri` only, env-only. Maximum token mismatches accepted with `ORACLE_STRICT=1` and `TF` set. Must be a nonnegative decimal integer smaller than the number of TF positions. CI uses `2` for the 32-position tiny fixture (30–32 matches); unset or `0` requires exact agreement. Mismatches remain visible in diagnostics. Ignored outside strict TF mode; cannot relax greedy comparison or non-finite-output checks. |
| `CHAT_TEMPLATE` | `1` | Apply the GLM chat template (`0` = raw prompt). |
| `PPL` | off (`olmoe.c` and `qwen38.c` only) | `PPL=1` enters teacher-forced NLL/perplexity meter mode in the OLMoE and Qwen3.8 sister engines. |
| `ABLATE_SCORE` | unset | Causal-ablation sweep over `ABLATE_SCORE=<file>`, with a per-target-position final-logit read-out. Runs before `SCORE` and exits when done. |