Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

tok.h: the DeepSeek V4/V4.1 pre-tokenizer, so Turkish tokenizes like HF

JustVugg101c6a185b16
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 886d0182..ab191ac4 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -825,9 +825,12 @@ jobs:
# accepted and the verification path would never run. V41_SPEC_FORCE drafts
# the reference's own tokens instead: 1 accepts the whole block, 2 corrupts
# the last draft so a round is rejected part way and the rollback runs, 3
- # keeps the head's drafts, which is what serving does. All three have to
- # reproduce the reference token for token, at any cache capacity.
- for force in 1 2 3; do
+ # keeps the head's drafts, which is what serving does, 4 rejects the first
+ # draft of every round (all drafted rows rolled back, what serving at 10-30%
+ # acceptance does most of the time) and 5 a different one each round. All
+ # of them have to reproduce the reference token for token, at any cache
+ # capacity.
+ for force in 1 2 3 4 5; do
for cap in 1 8; do
echo "::group::V41_SPEC_FORCE=$force cap=$cap"
SNAP=dsv41_tiny V41_DSPARK=1 V41_SPEC_FORCE=$force ./deepseek_v41 "$cap" dsv41_tiny/ref.json
diff --git a/c/Makefile b/c/Makefile
index 370160e7..495b2017 100644
--- a/c/Makefile
+++ b/c/Makefile
@@ -811,7 +811,7 @@ endif
endif
.build-config: ;
-colibri$(EXE): colibri.c sse41_kernels.h exact_dot.h oracle.h pin_pool.h cli_args.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h omp_tune.h kv_fp8.h kv_tq.h abl.h backend_cuda.h backend_metal.h backend_vulkan.h backend_xdna.h decode_batch.h edge_adapters.h edge_runtime.h edge_tok_internal.h evidence_digest.h schema_gbnf.h segment_adapter_internal.h segment_adapters.h segment_runtime.h tier.h $(CUDA_OBJ) $(METAL_OBJ) $(VK_OBJ) $(VK_SPV) $(XDNA_OBJ) .build-config
+colibri$(EXE): colibri.c sse41_kernels.h exact_dot.h oracle.h pin_pool.h cli_args.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h omp_tune.h kv_fp8.h kv_tq.h abl.h backend_cuda.h backend_metal.h backend_vulkan.h backend_xdna.h decode_batch.h edge_adapters.h edge_runtime.h edge_tok_internal.h evidence_digest.h schema_gbnf.h segment_adapter_internal.h segment_adapters.h segment_runtime.h tier.h $(CUDA_OBJ) $(METAL_OBJ) $(VK_OBJ) $(VK_SPV) $(XDNA_OBJ) .build-config
$(CC) $(CFLAGS) colibri.c $(CUDA_OBJ) $(METAL_OBJ) $(VK_OBJ) $(XDNA_OBJ) -o colibri$(EXE) $(LDFLAGS)
# Vulkan backend object (plain C + vulkan headers) and its SPIR-V shaders.
@@ -1151,7 +1151,7 @@ tests/bench_cuda_resident_batch$(EXE): tests/bench_cuda_resident_batch.cu backen
# the GPU sits idle. Same guard #783 put on kimi_k3, which since gaining an
# MXFP4 expert path no longer needs it. tests/test_makefile_cuda_scope.py
# asserts this shape.
-olmoe$(EXE): olmoe.c cli_args.h st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_o200k.h omp_tune.h kv_prefix.h pin_pool.h route_trace.h serve_codec.h serve_budget.h edge_adapters.h edge_runtime.h edge_tok_internal.h fused_simd.h segment_adapter_internal.h segment_adapters.h segment_runtime.h sse41_kernels.h
+olmoe$(EXE): olmoe.c cli_args.h st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h omp_tune.h kv_prefix.h pin_pool.h route_trace.h serve_codec.h serve_budget.h edge_adapters.h edge_runtime.h edge_tok_internal.h fused_simd.h segment_adapter_internal.h segment_adapters.h segment_runtime.h sse41_kernels.h
$(CC) $(NOCUDA_CFLAGS) olmoe.c -o olmoe$(EXE) $(NOCUDA_LDFLAGS)
# Qwen3.6-35B-A3B engine (hybrid Gated Attention + Gated DeltaNet + streaming
@@ -1191,7 +1191,7 @@ qwen36$(EXE): qwen36.c sse41_kernels.h decode_batch.h serve_poll.h cli_args.h qw
# DeepSeek V4.1 Flash: one file, like every other portable engine. The fp4 experts
# stream from the official checkpoint and quant.h's mxfp4 kernel reads them as they
# are, so there is no conversion target here to go with it.
-deepseek_v41$(EXE): deepseek_v41.c sse41_kernels.h cli_args.h st.h json.h tok.h tok_unicode.h compat.h omp_tune.h kv_prefix.h pin_pool.h quant.h fp8_format.h idot.h \
+deepseek_v41$(EXE): deepseek_v41.c sse41_kernels.h cli_args.h st.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h omp_tune.h kv_prefix.h pin_pool.h quant.h fp8_format.h idot.h \
sparse_attn.h hyper_connections.h serve_codec.h serve_poll.h .build-config
$(CC) $(CFLAGS) deepseek_v41.c -o deepseek_v41$(EXE) $(LDFLAGS)
@@ -1199,7 +1199,7 @@ deepseek_v41$(EXE): deepseek_v41.c sse41_kernels.h cli_args.h st.h json.h tok.h
# loader and SERVE=1 protocol. With CUDA=1 it links the same expert tier as
# qwen36 (fp8 streaming mode: hot experts get VRAM copies, the RAM LRU stays);
# without it the tier header's inline stubs keep the build toolkit-free.
-qwen38$(EXE): qwen38.c sse41_kernels.h pin_pool.h cli_args.h qwen38_core.h kv_prefix.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h omp_tune.h quant.h fp8_format.h idot.h route_trace.h tok.h tok_unicode.h tok_unicode_o200k.h serve_codec.h edge_adapter_internal.h edge_adapters.h edge_runtime.h qwen38_vision.h segment_adapter_internal.h segment_adapters.h segment_runtime.h qwen36_tier.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)
+qwen38$(EXE): qwen38.c sse41_kernels.h pin_pool.h cli_args.h qwen38_core.h kv_prefix.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h omp_tune.h quant.h fp8_format.h idot.h route_trace.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h serve_codec.h edge_adapter_internal.h edge_adapters.h edge_runtime.h qwen38_vision.h segment_adapter_internal.h segment_adapters.h segment_runtime.h qwen36_tier.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)
$(CC) $(QWEN36_CFLAGS) qwen38.c $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o qwen38$(EXE) $(QWEN36_LDFLAGS)
.PHONY: qwen38-tiny-generate qwen38-tiny-check
@@ -1298,7 +1298,7 @@ tests/test_qwen36_tokenizer$(EXE): tests/test_qwen36_tokenizer.c gsgemv.h qgemv.
tests/bench_qwen36_dense_batch$(EXE): tests/bench_qwen36_dense_batch.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)
$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)
-inkling$(EXE): inkling.c cli_args.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h omp_tune.h route_trace.h kv_prefix.h pin_pool.h serve_codec.h serve_budget.h backend_cuda_ink.h backend_metal.h edge_adapters.h edge_runtime.h edge_tok_internal.h segment_adapter_internal.h segment_adapters.h segment_runtime.h $(INK_CUDA_OBJ) $(METAL_OBJ)
+inkling$(EXE): inkling.c cli_args.h st.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h compat.h omp_tune.h route_trace.h kv_prefix.h pin_pool.h serve_codec.h serve_budget.h backend_cuda_ink.h backend_metal.h edge_adapters.h edge_runtime.h edge_tok_internal.h segment_adapter_internal.h segment_adapters.h segment_runtime.h $(INK_CUDA_OBJ) $(METAL_OBJ)
$(CC) $(CFLAGS) inkling.c $(INK_CUDA_OBJ) $(METAL_OBJ) -o inkling$(EXE) $(LDFLAGS)
# ENGINES WITHOUT A CUDA BACKEND (#783).
@@ -1315,10 +1315,10 @@ NOCUDA_LDFLAGS = $(filter-out -lcudart -lstdc++ -lcuda -L$(CUDA_HOME)/lib64 \
# GLM-5.3-Flash: routed experts stream from the int4-gs64 container.
# METAL=1 accelerates resident matrices and routed MoE; CPU remains fallback.
-glm53$(EXE): glm53.c sse41_kernels.h decode_batch.h pin_pool.h cli_args.h st.h json.h stop_ids.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h omp_tune.h serve_poll.h route_trace.h quant.h fp8_format.h idot.h hyper_connections.h delta_attention.h sparse_index.h vision_tower.h backend_metal.h backend_vulkan.h edge_adapter_internal.h edge_adapters.h edge_runtime.h segment_adapter_internal.h segment_adapters.h segment_runtime.h $(METAL_OBJ) $(VK_OBJ) $(VK_SPV)
+glm53$(EXE): glm53.c sse41_kernels.h decode_batch.h pin_pool.h cli_args.h st.h json.h stop_ids.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h compat.h omp_tune.h serve_poll.h route_trace.h quant.h fp8_format.h idot.h hyper_connections.h delta_attention.h sparse_index.h vision_tower.h backend_metal.h backend_vulkan.h edge_adapter_internal.h edge_adapters.h edge_runtime.h segment_adapter_internal.h segment_adapters.h segment_runtime.h $(METAL_OBJ) $(VK_OBJ) $(VK_SPV)
$(CC) $(CFLAGS) glm53.c $(METAL_OBJ) $(VK_OBJ) -o glm53$(EXE) $(LDFLAGS)
-kimi_k3$(EXE): kimi_k3.c sse41_kernels.h serve_budget.h cli_args.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h fp8_format.h idot.h omp_tune.h route_trace.h kv_prefix.h pin_pool.h serve_codec.h backend_cuda.h backend_metal.h backend_vulkan.h edge_adapters.h edge_runtime.h edge_tok_internal.h hybrid_split.h segment_adapter_internal.h segment_adapters.h segment_runtime.h $(CUDA_OBJ) $(VK_OBJ) $(VK_SPV) $(METAL_OBJ)
+kimi_k3$(EXE): kimi_k3.c sse41_kernels.h serve_budget.h cli_args.h st.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h compat.h quant.h fp8_format.h idot.h omp_tune.h route_trace.h kv_prefix.h pin_pool.h serve_codec.h backend_cuda.h backend_metal.h backend_vulkan.h edge_adapters.h edge_runtime.h edge_tok_internal.h hybrid_split.h segment_adapter_internal.h segment_adapters.h segment_runtime.h $(CUDA_OBJ) $(VK_OBJ) $(VK_SPV) $(METAL_OBJ)
$(CC) $(CFLAGS) kimi_k3.c $(CUDA_OBJ) $(VK_OBJ) $(METAL_OBJ) -o kimi_k3$(EXE) $(LDFLAGS)
# Use a baseline that matches the compiler target. macOS already targets a
@@ -1346,7 +1346,7 @@ iobench$(EXE): iobench.c compat.h
tests/test_serve_sentinel$(EXE): tests/test_serve_sentinel.c compat.h serve_codec.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_cluster_protocol$(EXE): tests/test_cluster_protocol.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_cluster_protocol$(EXE): tests/test_cluster_protocol.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
tests/test_ue8m0$(EXE): tests/test_ue8m0.c st.h json.h compat.h
@@ -1354,7 +1354,7 @@ tests/test_ue8m0$(EXE): tests/test_ue8m0.c st.h json.h compat.h
# Compiles the engine with -DCOLI_XDNA so the QT side pointer, its reset helper
# and the expert-slot wiring are exercised as the engine actually builds them.
-tests/test_xdna_qt_state$(EXE): tests/test_xdna_qt_state.c colibri.c backend_xdna.c backend_xdna.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_xdna_qt_state$(EXE): tests/test_xdna_qt_state.c colibri.c backend_xdna.c backend_xdna.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) -DCOLI_XDNA $< backend_xdna.c -o $@ $(LDFLAGS)
tests/test_xdna_prepared_state$(EXE): tests/test_xdna_prepared_state.c backend_xdna.c backend_xdna.h compat.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
@@ -1381,7 +1381,7 @@ tests/test_xdna_execution$(EXE): tests/test_xdna_execution.c backend_xdna.c back
# Failure/fallback owner. Needs BOTH the engine (matmul_qt is the fallback
# target it verifies against) and the synthetic ABI-2 helpers, which is why it
# is a separate owner from test_xdna_execution.
-tests/test_xdna_failure$(EXE): tests/test_xdna_failure.c colibri.c backend_xdna.c backend_xdna.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h $(XDNA_FAKE_HELPERS)
+tests/test_xdna_failure$(EXE): tests/test_xdna_failure.c colibri.c backend_xdna.c backend_xdna.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h $(XDNA_FAKE_HELPERS)
$(CC) $(CFLAGS) -DCOLI_XDNA $< backend_xdna.c -o $@ $(LDFLAGS)
# NOT a test gate -- intentionally absent from TEST_BINS (its name does not match
@@ -1393,16 +1393,18 @@ tests/test_xdna_failure$(EXE): tests/test_xdna_failure.c colibri.c backend_xdna.
#
# M-list defaults to 1,32,64 (the M64 bucket); pass 65,130,256 to qualify
# the M256 bucket through the same owner.
-tests/xdna_physical_probe$(EXE): tests/xdna_physical_probe.c colibri.c backend_xdna.c backend_xdna.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/xdna_physical_probe$(EXE): tests/xdna_physical_probe.c colibri.c backend_xdna.c backend_xdna.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) -DCOLI_XDNA $< backend_xdna.c -o $@ $(LDFLAGS)
tests/test_json$(EXE): tests/test_json.c json.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_tok_o200k$(EXE): tests/test_tok_o200k.c tok.h tok_unicode.h tok_unicode_o200k.h json.h
+tests/test_tok_o200k$(EXE): tests/test_tok_o200k.c tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h json.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_tok_gpt2$(EXE): tests/test_tok_gpt2.c tok.h tok_unicode.h tok_unicode_o200k.h json.h
+tests/test_tok_gpt2$(EXE): tests/test_tok_gpt2.c tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h json.h
+ $(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
+tests/test_tok_deepseek$(EXE): tests/test_tok_deepseek.c tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h json.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_k3_ram_budget$(EXE): tests/test_k3_ram_budget.c sse41_kernels.h kimi_k3.c st.h tok.h quant.h fp8_format.h idot.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h $(VK_OBJ)
@@ -1411,7 +1413,7 @@ tests/test_k3_ram_budget$(EXE): tests/test_k3_ram_budget.c sse41_kernels.h kimi_
tests/test_k3_mmap$(EXE): tests/test_k3_mmap.c sse41_kernels.h kimi_k3.c st.h tok.h quant.h fp8_format.h idot.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h $(VK_OBJ)
$(CC) $(NOCUDA_CFLAGS) $< $(VK_OBJ) -o $@ $(NOCUDA_LDFLAGS)
-tests/test_tok_kimi_tiny$(EXE): tests/test_tok_kimi_tiny.c tok.h tok_unicode.h tok_unicode_o200k.h json.h
+tests/test_tok_kimi_tiny$(EXE): tests/test_tok_kimi_tiny.c tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h json.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_st_pread$(EXE): tests/test_st_pread.c st.h json.h compat.h
$(CC) $(CFLAGS) -DST_PREAD_CHUNK=7 $< -o $@ $(LDFLAGS)
@@ -1419,19 +1421,19 @@ tests/test_st_pread$(EXE): tests/test_st_pread.c st.h json.h compat.h
tests/test_st_slice$(EXE): tests/test_st_slice.c st.h json.h compat.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_qwen38_tokenizer$(EXE): tests/test_qwen38_tokenizer.c sse41_kernels.h qwen38.c qwen38_core.h kv_prefix.h qwen38_nfc.h qwen38_nfc_tables.h edge_runtime.c edge_runtime.h edge_adapters.h edge_adapter_internal.h st.h json.h compat.h quant.h fp8_format.h idot.h route_trace.h tok_unicode.h tok_unicode_o200k.h
+tests/test_qwen38_tokenizer$(EXE): tests/test_qwen38_tokenizer.c sse41_kernels.h qwen38.c qwen38_core.h kv_prefix.h qwen38_nfc.h qwen38_nfc_tables.h edge_runtime.c edge_runtime.h edge_adapters.h edge_adapter_internal.h st.h json.h compat.h quant.h fp8_format.h idot.h route_trace.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h
$(CC) $(NOCUDA_CFLAGS) $< edge_runtime.c -o $@ $(NOCUDA_LDFLAGS)
-tests/test_qwen38_config$(EXE): tests/test_qwen38_config.c sse41_kernels.h qwen38.c qwen38_core.h kv_prefix.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h fp8_format.h idot.h route_trace.h tok_unicode.h tok_unicode_o200k.h
+tests/test_qwen38_config$(EXE): tests/test_qwen38_config.c sse41_kernels.h qwen38.c qwen38_core.h kv_prefix.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h fp8_format.h idot.h route_trace.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h
$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)
-tests/test_dsv41_serve_budget$(EXE): tests/test_dsv41_serve_budget.c sse41_kernels.h deepseek_v41.c cli_args.h st.h json.h tok.h tok_unicode.h compat.h omp_tune.h kv_prefix.h pin_pool.h quant.h fp8_format.h idot.h sparse_attn.h hyper_connections.h serve_codec.h serve_poll.h
+tests/test_dsv41_serve_budget$(EXE): tests/test_dsv41_serve_budget.c sse41_kernels.h deepseek_v41.c cli_args.h st.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h omp_tune.h kv_prefix.h pin_pool.h quant.h fp8_format.h idot.h sparse_attn.h hyper_connections.h serve_codec.h serve_poll.h
$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)
-tests/test_qwen38_idot$(EXE): tests/test_qwen38_idot.c sse41_kernels.h qwen38.c qwen38_core.h kv_prefix.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h fp8_format.h idot.h route_trace.h tok_unicode.h tok_unicode_o200k.h
+tests/test_qwen38_idot$(EXE): tests/test_qwen38_idot.c sse41_kernels.h qwen38.c qwen38_core.h kv_prefix.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h fp8_format.h idot.h route_trace.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h
$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)
-tests/test_qwen38_serve_framing$(EXE): tests/test_qwen38_serve_framing.c sse41_kernels.h qwen38.c qwen38_core.h kv_prefix.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h fp8_format.h idot.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h
+tests/test_qwen38_serve_framing$(EXE): tests/test_qwen38_serve_framing.c sse41_kernels.h qwen38.c qwen38_core.h kv_prefix.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h fp8_format.h idot.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h
$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)
tests/test_qwen38_vision$(EXE): tests/test_qwen38_vision.c qwen38_vision.h st.h json.h compat.h
@@ -1453,13 +1455,13 @@ qwen38-vision-serve-check: qwen38$(EXE)
$(PYTHON) tools/make_edge_tiny_tokenizer.py --vocab-size 64 ./qwen38_mm_tiny
$(PYTHON) tests/test_qwen38_vision_serve.py --binary ./qwen38$(EXE) --fixture ./qwen38_mm_tiny
-tests/test_qwen38_prefix$(EXE): tests/test_qwen38_prefix.c sse41_kernels.h qwen38.c qwen38_core.h kv_prefix.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h fp8_format.h idot.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h
+tests/test_qwen38_prefix$(EXE): tests/test_qwen38_prefix.c sse41_kernels.h qwen38.c qwen38_core.h kv_prefix.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h fp8_format.h idot.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h
$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)
-tests/test_qwen38_metrics$(EXE): tests/test_qwen38_metrics.c sse41_kernels.h qwen38.c qwen38_core.h kv_prefix.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h fp8_format.h idot.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h
+tests/test_qwen38_metrics$(EXE): tests/test_qwen38_metrics.c sse41_kernels.h qwen38.c qwen38_core.h kv_prefix.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h fp8_format.h idot.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h
$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)
-tests/test_qwen38_native_weights$(EXE): tests/test_qwen38_native_weights.c sse41_kernels.h qwen38.c qwen38_core.h kv_prefix.h qwen38_nfc.h qwen38_nfc_tables.h segment_runtime.c segment_runtime.h segment_adapters.h segment_adapter_internal.h st.h json.h compat.h quant.h fp8_format.h idot.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h
+tests/test_qwen38_native_weights$(EXE): tests/test_qwen38_native_weights.c sse41_kernels.h qwen38.c qwen38_core.h kv_prefix.h qwen38_nfc.h qwen38_nfc_tables.h segment_runtime.c segment_runtime.h segment_adapters.h segment_adapter_internal.h st.h json.h compat.h quant.h fp8_format.h idot.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h
$(CC) $(NOCUDA_CFLAGS) $< segment_runtime.c -o $@ $(NOCUDA_LDFLAGS)
tests/test_st_map$(EXE): tests/test_st_map.c st.h json.h compat.h
@@ -1527,12 +1529,12 @@ tests/test_grammar$(EXE): tests/test_grammar.c grammar.h
# equivalence under the shipping flags is covered by the differential dump
# (old vs new rope_interleave are byte-identical); this test guards the
# source-level formula equivalence and must not depend on contraction luck.
-tests/test_rope_invfreq$(EXE): tests/test_rope_invfreq.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_rope_invfreq$(EXE): tests/test_rope_invfreq.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) -ffp-contract=off $< $(VK_OBJ) -o $@ $(LDFLAGS)
# schema->GBNF compile cache (#7): grammar_reset must equal a fresh setup, and
# the GrDraft.src ownership must not leak/double-free. Includes colibri.c.
-tests/test_grammar_cache$(EXE): tests/test_grammar_cache.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_grammar_cache$(EXE): tests/test_grammar_cache.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
# The classified logit-row reduction in sample.h and the digest in
@@ -1540,13 +1542,13 @@ tests/test_grammar_cache$(EXE): tests/test_grammar_cache.c sse41_kernels.h colib
# agreement check against the plain logprob_target the sampling path uses
# on rows a float subtraction cannot mis-round, and a double-precision
# check against a second double computation on rows that do.
-tests/test_logprob_status$(EXE): tests/test_logprob_status.c colibri.c sse41_kernels.h exact_dot.h oracle.h sample.h evidence_digest.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h kv_persist.h telemetry.h route_trace.h
+tests/test_logprob_status$(EXE): tests/test_logprob_status.c colibri.c sse41_kernels.h exact_dot.h oracle.h sample.h evidence_digest.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# The ablation mode's manifest loader, evidence writer and dispatch contract.
# The adapter build bypasses only the model computation, so the parser/writer
# path is exercised for real with no model and no weights.
-tests/test_ablate_mode$(EXE): tests/test_ablate_mode.c colibri.c sse41_kernels.h exact_dot.h oracle.h sample.h evidence_digest.h abl.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h kv_persist.h telemetry.h route_trace.h
+tests/test_ablate_mode$(EXE): tests/test_ablate_mode.c colibri.c sse41_kernels.h exact_dot.h oracle.h sample.h evidence_digest.h abl.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# Standalone: drives a faithful miniature of moe()'s routing+accumulate and links
@@ -1562,7 +1564,7 @@ tests/test_degrade_zero$(EXE): tests/test_degrade_zero.c
tests/test_schema_gbnf$(EXE): tests/test_schema_gbnf.c schema_gbnf.h grammar.h json.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_spec_decode_state$(EXE): tests/test_spec_decode_state.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_spec_decode_state$(EXE): tests/test_spec_decode_state.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
tests/test_decode_batch$(EXE): tests/test_decode_batch.c decode_batch.h
@@ -1583,81 +1585,81 @@ tests/test_segment_runtime$(EXE): tests/test_segment_runtime.c segment_runtime.c
tests/test_segment_conformance$(EXE): tests/test_segment_conformance.c tests/segment_conformance_fixtures.c tests/segment_conformance_fixtures.h segment_runtime.c segment_runtime.h
$(CC) $(CFLAGS) tests/test_segment_conformance.c tests/segment_conformance_fixtures.c segment_runtime.c -o $@ $(LDFLAGS)
-tests/test_inkling_serve_framing$(EXE): tests/test_inkling_serve_framing.c inkling.c st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h serve_budget.h $(INK_CUDA_OBJ) $(METAL_OBJ)
+tests/test_inkling_serve_framing$(EXE): tests/test_inkling_serve_framing.c inkling.c st.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h serve_budget.h $(INK_CUDA_OBJ) $(METAL_OBJ)
$(CC) $(CFLAGS) $< $(INK_CUDA_OBJ) $(METAL_OBJ) -o $@ $(LDFLAGS)
-tests/test_inkling_shared_batch$(EXE): tests/test_inkling_shared_batch.c inkling.c st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h $(INK_CUDA_OBJ) $(METAL_OBJ)
+tests/test_inkling_shared_batch$(EXE): tests/test_inkling_shared_batch.c inkling.c st.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h $(INK_CUDA_OBJ) $(METAL_OBJ)
$(CC) $(CFLAGS) $< $(INK_CUDA_OBJ) $(METAL_OBJ) -o $@ $(LDFLAGS)
# Reproducible local A/B for the shared-expert prefill path. This is timing
# evidence, not a CI gate: build and run it explicitly on the target CPU.
-tests/bench_inkling_shared_batch$(EXE): tests/bench_inkling_shared_batch.c inkling.c st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h $(INK_CUDA_OBJ) $(METAL_OBJ)
+tests/bench_inkling_shared_batch$(EXE): tests/bench_inkling_shared_batch.c inkling.c st.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h $(INK_CUDA_OBJ) $(METAL_OBJ)
$(CC) $(CFLAGS) $< $(INK_CUDA_OBJ) $(METAL_OBJ) -o $@ $(LDFLAGS)
-tests/test_inkling_cache_index$(EXE): tests/test_inkling_cache_index.c inkling.c st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h $(INK_CUDA_OBJ) $(METAL_OBJ)
+tests/test_inkling_cache_index$(EXE): tests/test_inkling_cache_index.c inkling.c st.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h $(INK_CUDA_OBJ) $(METAL_OBJ)
$(CC) $(CFLAGS) $< $(INK_CUDA_OBJ) $(METAL_OBJ) -o $@ $(LDFLAGS)
-tests/test_kimi_serve_framing$(EXE): tests/test_kimi_serve_framing.c sse41_kernels.h serve_budget.h kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h fp8_format.h idot.h omp_tune.h route_trace.h $(VK_OBJ)
+tests/test_kimi_serve_framing$(EXE): tests/test_kimi_serve_framing.c sse41_kernels.h serve_budget.h kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h compat.h quant.h fp8_format.h idot.h omp_tune.h route_trace.h $(VK_OBJ)
$(CC) $(NOCUDA_CFLAGS) $< $(VK_OBJ) -o $@ $(NOCUDA_LDFLAGS)
-tests/test_kimi_cache_index$(EXE): tests/test_kimi_cache_index.c sse41_kernels.h kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h fp8_format.h idot.h omp_tune.h route_trace.h $(VK_OBJ)
+tests/test_kimi_cache_index$(EXE): tests/test_kimi_cache_index.c sse41_kernels.h kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h compat.h quant.h fp8_format.h idot.h omp_tune.h route_trace.h $(VK_OBJ)
$(CC) $(NOCUDA_CFLAGS) $< $(VK_OBJ) -o $@ $(NOCUDA_LDFLAGS)
-tests/test_k3_chat_tools$(EXE): tests/test_k3_chat_tools.c sse41_kernels.h kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h fp8_format.h idot.h omp_tune.h route_trace.h $(VK_OBJ)
+tests/test_k3_chat_tools$(EXE): tests/test_k3_chat_tools.c sse41_kernels.h kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h compat.h quant.h fp8_format.h idot.h omp_tune.h route_trace.h $(VK_OBJ)
$(CC) $(NOCUDA_CFLAGS) $< $(VK_OBJ) -o $@ $(NOCUDA_LDFLAGS)
# olmoe's matmul_q, not colibri's: compares the IDOT path against FP32
# ACTIVATIONS rather than an integer reference. NOCUDA_* for the same reason
# the olmoe target uses it -- olmoe.c contains no COLI_CUDA code.
-tests/test_olmoe_matmul_q$(EXE): tests/test_olmoe_matmul_q.c sse41_kernels.h olmoe.c st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h
+tests/test_olmoe_matmul_q$(EXE): tests/test_olmoe_matmul_q.c sse41_kernels.h olmoe.c st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h
$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)
-tests/test_olmoe_serve_framing$(EXE): tests/test_olmoe_serve_framing.c serve_budget.h sse41_kernels.h olmoe.c st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h
+tests/test_olmoe_serve_framing$(EXE): tests/test_olmoe_serve_framing.c serve_budget.h sse41_kernels.h olmoe.c st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h
$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)
-tests/test_olmoe_cache_index$(EXE): tests/test_olmoe_cache_index.c sse41_kernels.h olmoe.c st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h
+tests/test_olmoe_cache_index$(EXE): tests/test_olmoe_cache_index.c sse41_kernels.h olmoe.c st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h
$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)
-tests/test_qwen36_cache_index$(EXE): tests/test_qwen36_cache_index.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c expert_ffn.h st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h qwen36_tier.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)
+tests/test_qwen36_cache_index$(EXE): tests/test_qwen36_cache_index.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c expert_ffn.h st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h qwen36_tier.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)
$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)
-tests/test_idot$(EXE): tests/test_idot.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_idot$(EXE): tests/test_idot.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_i4_grouped$(EXE): tests/test_i4_grouped.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_i4_grouped$(EXE): tests/test_i4_grouped.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) -DCOLI_HAVE_GROUPED_PAIR $< $(VK_OBJ) -o $@ $(LDFLAGS)
# Force the grouped-int4 oracle onto the SSE4.1 tier even on an AVX2 host.
-tests/test_i4_grouped_sse41$(EXE): tests/test_i4_grouped.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sse41_kernels.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_i4_grouped_sse41$(EXE): tests/test_i4_grouped.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h idot.h sse41_kernels.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) -O3 -DCOLI_HAVE_GROUPED_PAIR -DCOLI_I4_GROUPED_SCALAR_EXACT -msse4.1 -mno-avx2 -mno-fma $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_i4_grouped_sse41_o1$(EXE): tests/test_i4_grouped.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sse41_kernels.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_i4_grouped_sse41_o1$(EXE): tests/test_i4_grouped.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h idot.h sse41_kernels.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) -O1 -DCOLI_HAVE_GROUPED_PAIR -DCOLI_I4_GROUPED_SCALAR_EXACT -msse4.1 -mno-avx2 -mno-fma $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_i4_grouped_sse41_no_contract$(EXE): tests/test_i4_grouped.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sse41_kernels.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_i4_grouped_sse41_no_contract$(EXE): tests/test_i4_grouped.c colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h idot.h sse41_kernels.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) -O3 -ffp-contract=off -DCOLI_HAVE_GROUPED_PAIR -DCOLI_I4_GROUPED_SCALAR_EXACT -msse4.1 -mno-avx2 -mno-fma $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_stops$(EXE): tests/test_stops.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_stops$(EXE): tests/test_stops.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_cfg_topk$(EXE): tests/test_cfg_topk.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_cfg_topk$(EXE): tests/test_cfg_topk.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_topp$(EXE): tests/test_topp.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_topp$(EXE): tests/test_topp.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
# bench_topp is a microbenchmark (old qsort vs new heap partial-select, #335), NOT a test
# gate -- intentionally absent from TEST_BINS. Build on demand: make tests/bench_topp
-tests/bench_topp$(EXE): tests/bench_topp.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_topp$(EXE): tests/bench_topp.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_sample_nan$(EXE): tests/test_sample_nan.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_sample_nan$(EXE): tests/test_sample_nan.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_temp_env$(EXE): tests/test_temp_env.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_temp_env$(EXE): tests/test_temp_env.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_kv_alloc$(EXE): tests/test_kv_alloc.c sse41_kernels.h colibri.c oracle.h st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h kv_fp8.h kv_tq.h $(VK_OBJ)
+tests/test_kv_alloc$(EXE): tests/test_kv_alloc.c sse41_kernels.h colibri.c oracle.h st.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h kv_fp8.h kv_tq.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
tests/test_kv_fp8$(EXE): tests/test_kv_fp8.c kv_fp8.h decode_batch.h
@@ -1666,7 +1668,7 @@ tests/test_kv_fp8$(EXE): tests/test_kv_fp8.c kv_fp8.h decode_batch.h
tests/test_kv_tq$(EXE): tests/test_kv_tq.c kv_tq.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_kv_disk$(EXE): tests/test_kv_disk.c sse41_kernels.h colibri.c oracle.h st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h kv_fp8.h kv_tq.h $(VK_OBJ)
+tests/test_kv_disk$(EXE): tests/test_kv_disk.c sse41_kernels.h colibri.c oracle.h st.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h kv_fp8.h kv_tq.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
# fmt=6 kernel oracle: needs the generated grid table, and its fixture comes from
@@ -1701,10 +1703,10 @@ fuzz-rans: tests/fuzz_rans.c rans.h
tests/fuzz_rans.c -o tests/fuzz_rans -lm
./tests/fuzz_rans
-tests/test_int3$(EXE): tests/test_int3.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_int3$(EXE): tests/test_int3.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_int3_load$(EXE): tests/test_int3_load.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_int3_load$(EXE): tests/test_int3_load.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
tests/test_fp8_passthrough$(EXE): tests/test_fp8_passthrough.c sse41_kernels.h quant.h fp8_format.h idot.h
@@ -1723,16 +1725,16 @@ tests/test_cuda_fmt_guard$(EXE): tests/test_cuda_fmt_guard.c backend_cuda.h
tests/test_cuda_lut_gate$(EXE): tests/test_cuda_lut_gate.c backend_cuda.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_fp8_load$(EXE): tests/test_fp8_load.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
+tests/test_fp8_load$(EXE): tests/test_fp8_load.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_qt_addrow$(EXE): tests/test_qt_addrow.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
+tests/test_qt_addrow$(EXE): tests/test_qt_addrow.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_logit_nan$(EXE): tests/test_logit_nan.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h $(VK_OBJ)
+tests/test_logit_nan$(EXE): tests/test_logit_nan.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_router_nan$(EXE): tests/test_router_nan.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h $(VK_OBJ)
+tests/test_router_nan$(EXE): tests/test_router_nan.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
tests/test_i4_acc512$(EXE): tests/test_i4_acc512.c
@@ -1816,7 +1818,7 @@ $(SEGMENT_BUILD_DIR)/qwen36.o: qwen36.c gsgemv.h qgemv.h sse41_kernels.h segment
$(SEGMENT_BUILD_DIR)/qwen38.o: qwen38.c sse41_kernels.h qwen38_core.h kv_prefix.h segment_runtime.h \
segment_adapters.h segment_adapter_internal.h edge_runtime.h \
edge_adapters.h edge_adapter_internal.h st.h json.h compat.h quant.h fp8_format.h \
- qwen38_nfc.h qwen38_nfc_tables.h route_trace.h tok_unicode.h \
+ qwen38_nfc.h qwen38_nfc_tables.h route_trace.h tok_unicode.h tok_unicode_deepseek.h \
tok_unicode_o200k.h | $(SEGMENT_BUILD_DIR)
$(CC) $(SEGMENT_CPU_CFLAGS) -DQWEN38_NO_MAIN -c qwen38.c -o $@
@@ -2132,15 +2134,15 @@ tests/test_exact_dot$(EXE): tests/test_exact_dot.c exact_dot.h
tests/test_798_guards$(EXE): tests/test_798_guards.c st.h json.h compat.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_dsa_select$(EXE): tests/test_dsa_select.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_dsa_select$(EXE): tests/test_dsa_select.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_corpus_draft$(EXE): tests/test_corpus_draft.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_corpus_draft$(EXE): tests/test_corpus_draft.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_cap_precedence$(EXE): tests/test_cap_precedence.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
+tests/test_cap_precedence$(EXE): tests/test_cap_precedence.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_mirror_stripe_split$(EXE): tests/test_mirror_stripe_split.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
+tests/test_mirror_stripe_split$(EXE): tests/test_mirror_stripe_split.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
tests/test_v4_hybrid_policy$(EXE): tests/test_v4_hybrid_policy.c deepseek_v4_hybrid.h hybrid_split.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
@@ -2151,29 +2153,29 @@ tests/test_k3_fill_budget$(EXE): tests/test_k3_fill_budget.c hybrid_split.h
tests/test_v4_bank_pair$(EXE): tests/test_v4_bank_pair.c deepseek_v4_bank_pair.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_ram_clamp$(EXE): tests/test_ram_clamp.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
+tests/test_ram_clamp$(EXE): tests/test_ram_clamp.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_cap_mixed_width$(EXE): tests/test_cap_mixed_width.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_cap_mixed_width$(EXE): tests/test_cap_mixed_width.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_eslot_inflight$(EXE): tests/test_eslot_inflight.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_eslot_inflight$(EXE): tests/test_eslot_inflight.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_glm_cache_index$(EXE): tests/test_glm_cache_index.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_glm_cache_index$(EXE): tests/test_glm_cache_index.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_ssd_probe$(EXE): tests/test_ssd_probe.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
+tests/test_ssd_probe$(EXE): tests/test_ssd_probe.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
# bench_dsa_select is a microbenchmark (old qsort vs new quickselect partial-select, #356),
# NOT a test gate -- intentionally absent from TEST_BINS. Build on demand: make tests/bench_dsa_select
-tests/bench_dsa_select$(EXE): tests/bench_dsa_select.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_dsa_select$(EXE): tests/bench_dsa_select.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_router_select is a microbenchmark (duplicate-prefix scan vs marked-score scan),
# not a test gate. Build on demand: make tests/bench_router_select.
-tests/bench_router_select$(EXE): tests/bench_router_select.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_router_select$(EXE): tests/bench_router_select.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_indexer_allocations: microbenchmark (DeepSeek V4 indexer malloc vs persistent arena scratch), NOT a test gate.
@@ -2183,34 +2185,34 @@ tests/bench_indexer_allocations$(EXE): tests/bench_indexer_allocations.c
# bench_idot: microbenchmark (single-acc vs independent-acc AVX-VNNI idot), NOT a test gate.
# Build on demand on an AVX-VNNI CPU: make tests/bench_idot ARCH=native
-tests/bench_idot$(EXE): tests/bench_idot.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_idot$(EXE): tests/bench_idot.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_i4p_gidot: microbenchmark (per-row vs multi-row/AMX K1b grouped planar IDOT), NOT a test gate.
# Build on demand: make tests/bench_i4p_gidot ARCH=native
-tests/bench_i4p_gidot$(EXE): tests/bench_i4p_gidot.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_i4p_gidot$(EXE): tests/bench_i4p_gidot.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_gemv_stream: microbenchmark (decode-regime GEMV bandwidth vs the read ceiling;
# frozen-baseline + deinterleaved-x candidate A/B), NOT a test gate.
# Build on demand: make tests/bench_gemv_stream ARCH=native
-tests/bench_gemv_stream$(EXE): tests/bench_gemv_stream.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_gemv_stream$(EXE): tests/bench_gemv_stream.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_mla_simd: microbenchmark (scalar vs AVX2/NEON MLA-absorb reductions, #442),
# NOT a test gate. Build on demand: make tests/bench_mla_simd
-tests/bench_mla_simd$(EXE): tests/bench_mla_simd.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_mla_simd$(EXE): tests/bench_mla_simd.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_uring$(EXE): tests/test_uring.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
+tests/test_uring$(EXE): tests/test_uring.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_pipe_block$(EXE): tests/test_pipe_block.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h $(VK_OBJ)
+tests/test_pipe_block$(EXE): tests/test_pipe_block.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_pilot_ring$(EXE): tests/test_pilot_ring.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
+tests/test_pilot_ring$(EXE): tests/test_pilot_ring.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
-tests/test_moe_gs_guard$(EXE): tests/test_moe_gs_guard.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
+tests/test_moe_gs_guard$(EXE): tests/test_moe_gs_guard.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
tests/test_omp_tune$(EXE): tests/test_omp_tune.c omp_tune.h compat.h
@@ -2219,7 +2221,7 @@ tests/test_omp_tune$(EXE): tests/test_omp_tune.c omp_tune.h compat.h
tests/test_compat_env$(EXE): tests/test_compat_env.c compat.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_kvb_notice$(EXE): tests/test_kvb_notice.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
+tests/test_kvb_notice$(EXE): tests/test_kvb_notice.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
# Standalone: proves the group-scaled int8 GEMV keeps every row's float
@@ -2249,20 +2251,20 @@ tests/test_qgemv_sse41$(EXE): tests/test_qgemv.c qgemv.h sse41_kernels.h
# olmoe's dot_i8_16 (ARM NEON/AVX2/SSE4.1 variants): must be bit-exact against
# a scalar int8 reference -- pure integer arithmetic, no tolerance needed.
# NOCUDA_* for the same reason the other olmoe.c-including test rules use it.
-tests/test_olmoe_dot_i8_16$(EXE): tests/test_olmoe_dot_i8_16.c olmoe.c sse41_kernels.h st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h
+tests/test_olmoe_dot_i8_16$(EXE): tests/test_olmoe_dot_i8_16.c olmoe.c sse41_kernels.h st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h
$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)
# Same gate, forced onto the SSE4.1 tier: overrides the host's default -march
# so the SSE4.1 body in olmoe.c's dot_i8_16 is actually exercised even on a
# devbox that would otherwise always pick the AVX2 tier.
-tests/test_olmoe_dot_i8_16_sse41$(EXE): tests/test_olmoe_dot_i8_16.c olmoe.c sse41_kernels.h st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h
+tests/test_olmoe_dot_i8_16_sse41$(EXE): tests/test_olmoe_dot_i8_16.c olmoe.c sse41_kernels.h st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h
$(CC) $(NOCUDA_CFLAGS) -msse4.1 -mno-avx2 -mno-fma $< -o $@ $(NOCUDA_LDFLAGS)
# layer_cuda_shard_kvb's format-allowlist refusal (see the test's file header). The
# function only exists under -DCOLI_CUDA, so on a default (CPU) build the binary is a
# loud SKIP; built with CUDA=1 it links backend_cuda.o ($(CUDA_OBJ)) and exercises the
# real guard -- no GPU needed, every probed path returns before any device context.
-tests/test_shard_kvb_refuse$(EXE): tests/test_shard_kvb_refuse.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h fp8_format.h sample.h kv_persist.h telemetry.h $(CUDA_OBJ) $(VK_OBJ)
+tests/test_shard_kvb_refuse$(EXE): tests/test_shard_kvb_refuse.c colibri.c st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h sample.h kv_persist.h telemetry.h $(CUDA_OBJ) $(VK_OBJ)
$(CC) $(CFLAGS) $< $(VK_OBJ) $(CUDA_OBJ) -o $@ $(LDFLAGS)
# The e8x4g64 container loader harness. It takes a minted directory on argv and
@@ -2415,9 +2417,9 @@ bench: iobench$(EXE)
tests/test_kv_prefix$(EXE): tests/test_kv_prefix.c kv_prefix.h
$(CC) $(CFLAGS) tests/test_kv_prefix.c -o tests/test_kv_prefix$(EXE) $(LDFLAGS)
-tests/test_kimi_request_state$(EXE): tests/test_kimi_request_state.c sse41_kernels.h kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h fp8_format.h idot.h omp_tune.h route_trace.h $(VK_OBJ)
+tests/test_kimi_request_state$(EXE): tests/test_kimi_request_state.c sse41_kernels.h kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h compat.h quant.h fp8_format.h idot.h omp_tune.h route_trace.h $(VK_OBJ)
$(CC) $(NOCUDA_CFLAGS) tests/test_kimi_request_state.c $(VK_OBJ) -o tests/test_kimi_request_state$(EXE) $(NOCUDA_LDFLAGS)
# Kimi CUDA dispatch/fallback with a fake backend; no CUDA toolkit required.
-tests/test_kimi_cuda_expert$(EXE): tests/test_kimi_cuda_expert.c sse41_kernels.h kimi_k3.c backend_cuda.h kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h fp8_format.h idot.h omp_tune.h route_trace.h $(VK_OBJ)
+tests/test_kimi_cuda_expert$(EXE): tests/test_kimi_cuda_expert.c sse41_kernels.h kimi_k3.c backend_cuda.h kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h compat.h quant.h fp8_format.h idot.h omp_tune.h route_trace.h $(VK_OBJ)
$(CC) $(NOCUDA_CFLAGS) -DCOLI_CUDA $< $(VK_OBJ) -o $@ $(NOCUDA_LDFLAGS)
diff --git a/c/deepseek_v41.c b/c/deepseek_v41.c
index 93860212..d4e2df1d 100644
--- a/c/deepseek_v41.c
+++ b/c/deepseek_v41.c
@@ -3998,7 +3998,7 @@ int main(int argc, char **argv) {
int *draft = block ? xmalloc((size_t)(block + 1) * sizeof(int), "drafts") : NULL;
float *confidence = block ? xmalloc((size_t)block * sizeof(float), "draft confidence") : NULL;
int spec_failed = 0, spec_checked = 0, round = 0;
- uint64_t forced_prop = 0, forced_acc = 0;
+ uint64_t forced_prop = 0, forced_acc = 0, forced_rounds = 0;
int matched = 0;
double decode_started = now_s();
@@ -4045,10 +4045,16 @@ int main(int argc, char **argv) {
/* 3 keeps the head's own drafts, which is what serving does; 1 and 2 put
* the reference's tokens in their place so the verification path runs at
* full width even on a fixture whose draft head is random noise */
- if (force < 3) {
+ if (force != 3) {
for (int i = 0; i < drafted; i++) draft[1 + i] = expected[step + i];
- if (force >= 2 && drafted > 0)
- draft[drafted] = (draft[drafted] + 1) % c->vocab; /* one bad draft */
+ /* 2 rejects the last draft, 4 the first -- every drafted row is
+ * rolled back, which is what serving at 10-30% acceptance does most
+ * of the time -- and 5 a different one each round */
+ int bad = force == 2 ? drafted : force == 4 ? 1
+ : force == 5 ? 1 + (int)(forced_rounds % (uint64_t)drafted) : 0;
+ if (bad > 0 && drafted > 0)
+ draft[bad] = (draft[bad] + 1) % c->vocab; /* one bad draft */
+ forced_rounds++;
}
} else {
drafted = 0; /* the plain oracle decodes one at a time */
diff --git a/c/tests/test_tok_deepseek.c b/c/tests/test_tok_deepseek.c
new file mode 100644
index 00000000..25cbc55b
--- /dev/null
+++ b/c/tests/test_tok_deepseek.c
@@ -0,0 +1,67 @@
+/* DeepSeek V4 / V4.1 pre-tokenizer family (three chained Splits: digit groups, CJK runs,
+ * then a regex of its own) against HF-tokenizers-generated expectations.
+ * tests/tok_deepseek_tiny.json is a byte-level BPE trained with `tokenizers` on a few
+ * multilingual sentences (20 KB, no model download) carrying exactly the pre_tokenizer
+ * block of the real tokenizer.json, and tests/tok_deepseek_cases.txt holds the ids HF
+ * produced for each case; tools/make_deepseek_tok_fixture.py rebuilds both. Guards the
+ * rules where this family and cl100k differ, which tok.h used to get wrong for both
+ * DeepSeek engines: an apostrophe takes the ASCII letters after it ("1956'da" is
+ * 195|6|'da, not the English contraction 'd|a -- #1772), marks join letter runs (Arabic,
+ * Hebrew, Devanagari), CJK and kana runs stand alone, punctuation is \p{P}\p{S}.
+ * Round-trips every case. Measured on the real tokenizers against HF on 18,000 random
+ * strings plus a curated set: V4.1 145/4000 random strings differed before, V4 573/6000;
+ * 0 after, on both. */
+#define _GNU_SOURCE
+#include "../tok.h"
+
+int main(void) {
+ Tok T;
+ tok_load(&T, "tests/tok_deepseek_tiny.json");
+ if (!T.deepseek) { fprintf(stderr, "test_tok_deepseek: DeepSeek family not detected\n"); return 1; }
+ FILE *f = fopen("tests/tok_deepseek_cases.txt", "rb");
+ if (!f) { perror("tests/tok_deepseek_cases.txt"); return 1; }
+ /* fgets, not getline: MinGW's UCRT lacks getline and this must run on the windows job */
+ char line[8192];
+ int pass = 0, tot = 0, dpass = 0;
+ while (fgets(line, sizeof(line), f)) {
+ size_t nr = strlen(line);
+ while (nr > 0 && (line[nr-1] == '\n' || line[nr-1] == '\r')) line[--nr] = 0;
+ if (nr == 0) continue;
+ char *tab = strchr(line, '\t'); if (!tab) continue;
+ *tab = 0;
+ const char *text = line, *idstr = tab + 1;
+ char tbuf[4096]; int tn = 0;
+ for (const char *q = text; *q && tn < 4095; q++) {
+ if (q[0]=='\\' && q[1]=='n') { tbuf[tn++]='\n'; q++; }
+ else if (q[0]=='\\' && q[1]=='t') { tbuf[tn++]='\t'; q++; }
+ else if (q[0]=='\\' && q[1]=='r') { tbuf[tn++]='\r'; q++; }
+ else if (q[0]=='\\' && q[1]=='\\') { tbuf[tn++]='\\'; q++; }
+ else tbuf[tn++] = *q;
+ }
+ tbuf[tn] = 0;
+ int exp[512], ne = 0;
+ for (const char *q = idstr; *q; ) {
+ while (*q == ',' || *q == ' ') q++;
+ if (!*q) break;
+ exp[ne++] = atoi(q);
+ while (*q && *q != ',') q++;
+ }
+ int got[512]; int ng = tok_encode(&T, tbuf, tn, got, 512);
+ int ok = (ng == ne);
+ for (int i = 0; i < ng && ok; i++) ok = (got[i] == exp[i]);
+ tot++; if (ok) pass++;
+ char dec[8192]; int dn = tok_decode(&T, got, ng, dec, 8191);
+ int drt = (dn == tn) && !memcmp(dec, tbuf, tn);
+ if (drt) dpass++;
+ if (!ok || !drt) {
+ fprintf(stderr, "MISMATCH text=%s\n exp(%d):", text, ne);
+ for (int i = 0; i < ne; i++) fprintf(stderr, " %d", exp[i]);
+ fprintf(stderr, "\n got(%d):", ng);
+ for (int i = 0; i < ng; i++) fprintf(stderr, " %d", got[i]);
+ fprintf(stderr, "\n decode_ok=%d\n", drt);
+ }
+ }
+ fclose(f);
+ printf("test_tok_deepseek: ENCODE %d/%d DECODE %d/%d\n", pass, tot, dpass, tot);
+ return (pass == tot && dpass == tot) ? 0 : 2;
+}
diff --git a/c/tests/tok_deepseek_cases.txt b/c/tests/tok_deepseek_cases.txt
new file mode 100644
index 00000000..751e9e0c
--- /dev/null
+++ b/c/tests/tok_deepseek_cases.txt
@@ -0,0 +1,89 @@
+Yapay zekâ kavramı ilk kez 1956'da Dartmouth Konferansı'nda John McCarthy tarafından ortaya atıldı. 851,657,594,820,511,763,736,223,411,24,405,797,535,798,707,836,420,651,847,655,856,899,895,16
+Alan Turing'in 1950'deki makalesi, makinelerin düşünüp düşünemeyeceği sorusunu gündeme getirdi. 846,653,712,299,223,411,18,726,826,523,14,827,269,829,828,782,767,781,802,803,334,16
+1997'de IBM'in Deep Blue'su Kasparov'u yendi; 2016'da AlphaGo, Lee Sedol'e karşı kazandı. 752,25,279,759,299,796,649,532,724,799,516,434,770,29,223,775,24,405,733,498,14,650,652,480,325,737,264,821,335,16
+Türkiye'de, Ankara'dan, İstanbul'a, ABD'den, Atatürk'ün, Google'ın, 2000'li yıllarda değişiklikler oldu. 850,279,14,734,723,14,661,706,78,298,14,732,725,14,735,817,9,294,14,897,9,400,14,223,774,18,835,769,699,812,731,860,898,16
+I'd say it's John's book, we're done, you'll see, I've been, I'm here, don't, can't, O'Reilly's. 43,277,766,762,278,420,278,760,14,656,837,819,14,768,436,765,14,313,838,761,14,313,431,654,71,14,403,326,14,801,326,14,636,833,714,278,16
+L'intelligence artificielle d'aujourd'hui; dell'arte, all'estero, c'è un'altra idea. 46,751,859,894,854,276,750,730,834,29,811,839,14,800,773,729,14,380,9,595,646,749,308,764,16
+Русский язык и Ελληνικά γράμματα, ещё раз и ещё раз. 882,789,788,120,771,883,408,805,786,880,804,663,881,667,14,423,413,408,423,413,16
+بِسْمِ اللَّهِ الرَّحْمَٰنِ الرَّحِيمِ العربية שָׁלוֹם עוֹלָם עִבְרִית 792,412,743,884,831,670,830,412,742,791,607,772,665,825,822,688,254,823,870
+नमस्ते दुनिया, क्षत्रिय हिन्दी भाषा। สวัสดีครับ ภาษาไทย 816,876,738,664,824,14,740,875,739,869,741,696,708,810,785,874,251,809,697,886,98
+中文测试:你好,世界!日本語のテキスト・カタカナ「ラーメン」 한국어 테스트입니다. 795,672,625,719,890,718,794,237,717,891,871,873,692,878,709,818,710,889,711,879,618,807,892,115,806,893,690,673,100,16
+def f(x):\n return x**2 + 1 # comment\n\tif (a && b) { return a->b[i]; }\r\n 701,637,438,840,808,421,638,440,20,633,223,19,223,376,896,201,630,632,67,648,314,11,639,421,379,441,68,468,469,658,201
+https://example.com/path?q=1&r=2#frag [email protected] @handle #tag $HOME ~/.bashrc 3.14159 1,234,567 857,451,855,327,841,452,31,19,427,31,20,832,647,281,845,327,635,858,376,529,631,849,659,853,69,223,21,16,842,450,223,19,14,843,14,844
+Emoji 👍🏽 👨‍👩‍👧 ❤️ ©®™ ½ ² Ⅻ ① ٣٤٥ 012 -42 1e-10 0x1F 848,814,676,813,373,747,373,746,662,100,629,660,872,223,592,223,591,223,887,223,888,223,885,223,720,679,243,634,448,223,19,71,15,445,223,18,90,19,40
+1956'da 411,24,405
+Dartmouth'ta 38,744,535,326,67
+Türkiye'de 850,279
+Ankara'dan 35,513,723
+ABD'den 35,455,725
+IBM'in 43,456,299
+Turing'in 465,712,299
+Atatürk'ün 35,526,817,9,294
+Google'ın 329,517,9,400
+2000'li 774,18,835
+Kasparov'u 45,776,516,434
+Lee Sedol'e 461,652,480,325
+McCarthy'nin 463,847,432,269
+O'Reilly 49,833,714
+rock'n'roll 84,81,69,77,432,433,81,270
+I'd 43,277
+it's 75,86,278
+you'll 91,515,436
+we're 537,837
+I've 43,838
+I'm 43,431
+'quoted' 9,83,87,81,86,71,70,9
+‘curly’ “double” 161,225,249,69,282,78,91,161,225,250,223,161,225,253,70,515,68,268,161,225,254
+d'accord 70,298,69,303,84,70
+l'IA 78,9,43,35
+c'è 69,9,595
+بِسْمِ 792,412
+שָׁלוֹם 150,556,665,825
+क्षत्रिय 259,683,875
+สวัสดี 260,784,785,116
+é ä 71,139,226,379,139,233
+́alone 139,226,67,78,307,71
+x​zero‍width 90,161,225,236,92,729,373,89,75,70,525
+中文测试 795,672,625
+ 中文 223,795,232
+カタカナ・ラーメン 818,710,709,711,879
+テキスト 全角 267,671,692,878,616,225,164,230,104,167,103,243
+Fullwidth 272,102,174,124,246,174,124,237,174,124,237,174,124,248,174,124,234,174,124,229,174,124,245,174,124,233
+123456789 19,447,22,449,25,26,27
+12345 67 19,447,22,23,223,24,25
+a1b22c333d4444 67,19,68,20,20,69,21,21,21,70,22,22,22,22
+½¾²³ 592,129,125,591,129,114
+Ⅻ① 887,888
+٣٤٥٦ 885,152,102
+3.14 21,16,446
+(1956) 10,411,24,11
+%40 7,22,18
+ leading 223,223,268,67,70,712
+trailing 86,308,499,712,808
+a b 67,223,314
+a\tb 67,200,68
+a b 67,129,257,68
+line\nnext 78,269,71,201,80,484,86
+line\r\nnext 78,269,71,204,201,80,484,86
+\n\n\nx 201,201,201,90
+x\n \n y 90,201,223,201,317
+ \n 223,201
+end \r\n\r\n 304,70,223,204,201,204,201
+!!!word 3,3,3,89,343,70
+'''x 9,9,9,90
+... 16,16,16
+—dash— 161,225,245,70,331,74,161,225,245
+«guillemets» 129,107,73,87,499,268,79,305,85,129,122
+¿Qué? 129,126,51,87,130,105,33
+¡Hola! 129,97,42,342,67,3
+<tag attr="v"> 30,529,641,86,84,31,4,88,4,32
+a->b 67,441,68
+x**2 90,440,20
+\\path\\to 62,82,753,62,86,81
+`code` 66,303,262,66
+{"k": [1, 2]} 93,4,77,4,28,223,61,19,14,223,20,63,95
+👍🏽 296,675,676
+👨‍👩‍👧 296,104,373,747,373,746
+🇹🇷 275,232,120,275,232,118
+❤️ 615,100,629
+©®™ 593,872
diff --git a/c/tests/tok_deepseek_tiny.json b/c/tests/tok_deepseek_tiny.json
new file mode 100644
index 00000000..5ba4dd30
--- /dev/null
+++ b/c/tests/tok_deepseek_tiny.json
@@ -0,0 +1 @@
+{"version":"1.0","truncation":null,"padding":null,"added_tokens":[{"id":0,"content":"<|begin▁of▁sentence|>","single_word":false,"lstrip":false,"rstrip":false,"normalized":false,"special":true},{"id":1,"content":"<|User|>","single_word":false,"lstrip":false,"rstrip":false,"normalized":false,"special":true},{"id":2,"content":"<|Assistant|>","single_word":false,"lstrip":false,"rstrip":false,"normalized":false,"special":true}],"normalizer":{"type":"Sequence","normalizers":[]},"pre_tokenizer":{"type":"Sequence","pretokenizers":[{"type":"Split","pattern":{"Regex":"\\p{N}{1,3}"},"behavior":"Isolated","invert":false},{"type":"Split","pattern":{"Regex":"[一-龥぀-ゟ゠-ヿ]+"},"behavior":"Isolated","invert":false},{"type":"Split","pattern":{"Regex":"[!\"#$%&'()*+,\\-./:;<=>?@\\[\\\\\\]^_`{|}~][A-Za-z]+|[^\r\n\\p{L}\\p{P}\\p{S}]?[\\p{L}\\p{M}]+| ?[\\p{P}\\p{S}]+[\r\n]*|\\s*[\r\n]+|\\s+(?!\\S)|\\s+"},"behavior":"Isolated","invert":false},{"type":"ByteLevel","add_prefix_space":false,"trim_offsets":true,"use_regex":false}]},"post_processor":null,"decoder":{"type":"ByteLevel","add_prefix_space":true,"trim_offsets":true,"use_regex":true},"model":{"type":"BPE","dropout":null,"unk_token":null,"continuing_subword_prefix":null,"end_of_word_suffix":null,"fuse_unk":false,"byte_fallback":false,"ignore_merges":false,"vocab":{"<|begin▁of▁sentence|>":0,"<|User|>":1,"<|Assistant|>":2,"!":3,"\"":4,"#":5,"$":6,"%":7,"&":8,"'":9,"(":10,")":11,"*":12,"+":13,",":14,"-":15,".":16,"/":17,"0":18,"1":19,"2":20,"3":21,"4":22,"5":23,"6":24,"7":25,"8":26,"9":27,":":28,";":29,"<":30,"=":31,">":32,"?":33,"@":34,"A":35,"B":36,"C":37,"D":38,"E":39,"F":40,"G":41,"H":42,"I":43,"J":44,"K":45,"L":46,"M":47,"N":48,"O":49,"P":50,"Q":51,"R":52,"S":53,"T":54,"U":55,"V":56,"W":57,"X":58,"Y":59,"Z":60,"[":61,"\\":62,"]":63,"^":64,"_":65,"`":66,"a":67,"b":68,"c":69,"d":70,"e":71,"f":72,"g":73,"h":74,"i":75,"j":76,"k":77,"l":78,"m":79,"n":80,"o":81,"p":82,"q":83,"r":84,"s":85,"t":86,"u":87,"v":88,"w":89,"x":90,"y":91,"z":92,"{":93,"|":94,"}":95,"~":96,"¡":97,"¢":98,"£":99,"¤":100,"¥":101,"¦":102,"§":103,"¨":104,"©":105,"ª":106,"«":107,"¬":108,"®":109,"¯":110,"°":111,"±":112,"²":113,"³":114,"´":115,"µ":116,"¶":117,"·":118,"¸":119,"¹":120,"º":121,"»":122,"¼":123,"½":124,"¾":125,"¿":126,"À":127,"Á":128,"Â":129,"Ã":130,"Ä":131,"Å":132,"Æ":133,"Ç":134,"È":135,"É":136,"Ê":137,"Ë":138,"Ì":139,"Í":140,"Î":141,"Ï":142,"Ð":143,"Ñ":144,"Ò":145,"Ó":146,"Ô":147,"Õ":148,"Ö":149,"×":150,"Ø":151,"Ù":152,"Ú":153,"Û":154,"Ü":155,"Ý":156,"Þ":157,"ß":158,"à":159,"á":160,"â":161,"ã":162,"ä":163,"å":164,"æ":165,"ç":166,"è":167,"é":168,"ê":169,"ë":170,"ì":171,"í":172,"î":173,"ï":174,"ð":175,"ñ":176,"ò":177,"ó":178,"ô":179,"õ":180,"ö":181,"÷":182,"ø":183,"ù":184,"ú":185,"û":186,"ü":187,"ý":188,"þ":189,"ÿ":190,"Ā":191,"ā":192,"Ă":193,"ă":194,"Ą":195,"ą":196,"Ć":197,"ć":198,"Ĉ":199,"ĉ":200,"Ċ":201,"ċ":202,"Č":203,"č":204,"Ď":205,"ď":206,"Đ":207,"đ":208,"Ē":209,"ē":210,"Ĕ":211,"ĕ":212,"Ė":213,"ė":214,"Ę":215,"ę":216,"Ě":217,"ě":218,"Ĝ":219,"ĝ":220,"Ğ":221,"ğ":222,"Ġ":223,"ġ":224,"Ģ":225,"ģ":226,"Ĥ":227,"ĥ":228,"Ħ":229,"ħ":230,"Ĩ":231,"ĩ":232,"Ī":233,"ī":234,"Ĭ":235,"ĭ":236,"Į":237,"į":238,"İ":239,"ı":240,"IJ":241,"ij":242,"Ĵ":243,"ĵ":244,"Ķ":245,"ķ":246,"ĸ":247,"Ĺ":248,"ĺ":249,"Ļ":250,"ļ":251,"Ľ":252,"ľ":253,"Ŀ":254,"ŀ":255,"Ł":256,"ł":257,"Ń":258,"à¤":259,"à¸":260,"ar":261,"de":262,"ü":263,"ı":264,"an":265,"à¥":266,"ãĥ":267,"le":268,"in":269,"ll":270,"ÙIJ":271,"ï¼":272,"ÙĦ":273,"ãĤ":274,"ðŁ":275,"Ġd":276,"'d":277,"'s":278,"'de":279,"ee":280,"er":281,"ur":282,"§ÙĦ":283,"ÅŁ":284,"اÙĦ":285,"Ùİ":286,"ĠA":287,"Ġk":288,"ĠÐ":289,"Ġà¤":290,"ĠاÙĦ":291,"įà¤":292,"art":293,"ün":294,"à¥įà¤":295,"ðŁij":296,"ÙİÙ":297,"'a":298,"'in":299,"19":300,"at":301,"ay":302,"co":303,"en":304,"et":305,"ma":306,"on":307,"ra":308,"¿à¤":309,"з":310,"ر":311,"Ùħ":312,"ĠI":313,"Ġb":314,"Ġi":315,"Ġs":316,"Ġy":317,"ĠÑ":318,"Ġ×":319,"Ģį":320,"ा":321,"िà¤":322,"ÙİÙij":323,"com":324,"'e":325,"'t":326,".com":327,"20":328,"Go":329,"Jo":330,"as":331,"ce":332,"da":333,"di":334,"dı":335,"hn":336,"if":337,"ik":338,"ki":339,"me":340,"mo":341,"ol":342,"or":343,"ret":344,"st":345,"un":346,"us":347,"ye":348,"¦à¥":349,"ªà¸":350,"°Ð·":351,"±à¸":352,"´×":353,"µÑ":354,"¸×":355,"¹×":356,"»Î":357,"¼Î":358,"ÄŁ":359,"ά":360,"к":361,"аз":362,"Ñģ":363,"Ñij":364,"Ö´×":365,"Ö¸×":366,"Ö¹×":367,"×ķ":368,"ب":369,"ØŃ":370,"ÙĬ":371,"ÙĴ":372,"âĢį":373,"ä¸":374,"ëĭ":375,"Ġ#":376,"ĠD":377,"ĠK":378,"Ġa":379,"Ġc":380,"Ġg":381,"ĠÎ":382,"Ġí":383,"ĠĠ":384,"Ġà¸":385,"Ġde":386,"ĠðŁij":387,"Ġma":388,"ĠJo":389,"Ġret":390,"Ģаз":391,"Ħà¸":392,"īÑij":393,"न":394,"า":395,"ัà¸":396,"ara":397,"ür":398,"Ã¼ÅŁ":399,"ın":400,"ler":401,"ãĤ«":402,"Ġdon":403,"ĠdÃ¼ÅŁ":404,"'da":405,"urn":406,"Ġka":407,"Ġи":408,"ĠеÑ":409,"ĠاÙĦر":410,"195":411,"ÙħÙIJ":412,"ĠÑĢаз":413,"Ġ×¢":414,"िय":415,"ÙİÙijØŃ":416,"ÄŁi":417,"×ķÖ¹×":418,"Ġmak":419,"ĠJohn":420,"Ġreturn":421,"ĠdÃ¼ÅŁÃ¼n":422,"ĠеÑīÑij":423,"ĠاÙĦرÙİÙijØŃ":424,"#f":425,"&&":426,"&r":427,"'R":428,"'h":429,"'l":430,"'m":431,"'n":432,"'r":433,"'u":434,"'v":435,"'ll":436,"'art":437,"(x":438,"):":439,"**":440,"->":441,"/.":442,"//":443,"/p":444,"10":445,"14":446,"23":447,"42":448,"56":449,"59":450,"://":451,"?q":452,"@ma":453,"Al":454,"BD":455,"BM":456,"Bl":457,"Cart":458,"Emo":459,"HO":460,"Lee":461,"ME":462,"Mc":463,"Se":464,"Tur":465,"Tür":466,"Ya":467,"[i":468,"];":469,"ag":470,"am":471,"ale":472,"aGo":473,"bu":474,"bas":475,"ci":476,"du":477,"dan":478,"dle":479,"dol":480,"ei":481,"ek":482,"el":483,"ex":484,"ez":485,"ell":486,"eme":487,"eler":488,"fer":489,"fın":490,"gle":491,"gen":492,"hr":493,"ht":494,"hy":495,"han":496,"her":497,"haGo":498,"il":499,"ir":500,"ici":501,"igen":502,"ji":503,"jo":504,"kara":505,"lk":506,"lp":507,"lt":508,"ldı":509,"lik":510,"mı":511,"men":512,"nkara":513,"ok":514,"ou":515,"ov":516,"ogle":517,"ook":518,"ps":519,"par":520,"ple":521,"pay":522,"si":523,"sı":524,"th":525,"tat":526,"tay":527,"tara":528,"tag":529,"tell":530,"tps":531,"ue":532,"ui":533,"ujo":534,"uth":535,"vra":536,"we":537,"zan":538,"zek":539,"}č":540,"~/.":541,"¡ãĥ":542,"£Ù":543,"£à¸±à¸":544,"¤Ù":545,"¤í":546,"¤à¥":547,"¤à¥įà¤":548,"¥½":549,"¥æ":550,"§à¸±à¸":551,"¨à¥įà¤":552,"¨Ö´×":553,"©Â":554,"©ãĥ":555,"©Ö¸×":556,"©à¸²":557,"ªŀ":558,"¬è":559,"®â":560,"®à¤":561,"®ãĥ":562,"¯ķ":563,"°×":564,"°Ù":565,"°st":566,"°à¤¿à¤¯":567,"±Ï":568,"³Ï":569,"³ÙĴ":570,"µĭ":571,"µŃ":572,"µà¸":573,"·Î":574,"·à¤":575,"·à¤¾":576,"¸Ð":577,"¸ì":578,"¸ı":579,"¸à¥įà¤":580,"¹Î":581,"¹ãĥ":582,"¹Ø±":583,"¹à¤¿à¤":584,"¹Ħà¸":585,"ºÎ¬":586,"¼ãĥ":587,"½Î":588,"½ł":589,"¿ãĤ«":590,"²":591,"½":592,"©Â":593,"â":594,"è":595,"İst":596,"α":597,"λÎ":598,"μÎ":599,"Ðł":600,"иÐ":601,"Ñĥ":602,"Ñĭ":603,"Ö°×":604,"ת":605,"׾":606,"Ø©":607,"سÙĴ":608,"عر":609,"Ùĩ":610,"Ù£Ù":611,"à¹Ħà¸":612,"âħ":613,"âij":614,"âĿ":615,"ãĢ":616,"ãģ":617,"ãĢį":618,"ä½ł":619,"好":620,"æĸ":621,"æĹ":622,"æµĭ":623,"çķ":624,"è¯ķ":625,"êµŃ":626,"ìĬ":627,"ìĸ":628,"ï¸ı":629,"ĉif":630,"Ġ$":631,"Ġ(":632,"Ġ+":633,"Ġ-":634,"Ġ@":635,"ĠO":636,"Ġf":637,"Ġx":638,"Ġ{":639,"Ġart":640,"Ġat":641,"Ġcom":642,"ĠGo":643,"Ġol":644,"Ġor":645,"Ġun":646,"Ġus":647,"Ġ&&":648,"ĠBl":649,"ĠLee":650,"ĠMc":651,"ĠSe":652,"ĠTur":653,"Ġher":654,"Ġtara":655,"Ġwe":656,"Ġzek":657,"Ġ}č":658,"Ġ~/.":659,"Ġ©Â":660,"Ġİst":661,"ĠâĿ":662,"ģά":663,"ģन":664,"ģ׾":665,"Ħ¢":666,"Ħα":667,"ħĮ":668,"ħëĭ":669,"ĨÙIJ":670,"ĨãĤ":671,"ĩæµĭ":672,"Īëĭ":673,"Ĭ¸ì":674,"įðŁ":675,"ı½":676,"ıз":677,"IJï¼":678,"ijï¼":679,"ijÖ°×":680,"Ķà¸":681,"ķľ":682,"ķà¥įà¤":683,"ķλÎ":684,"ĸçķ":685,"Ĺà¸":686,"Ļת":687,"ľÖ¸×":688,"ľ¬è":689,"ŀħëĭ":690,"łà¸²":691,"ŃãĤ":692,"Ńा":693,"Ńæĸ":694,"मà¤":695,"षा":696,"ษา":697,"arÅŁ":698,"arda":699,"dea":700,"def":701,"deme":702,"üp":703,"ıll":704,"ıldı":705,"anbu":706,"ansı":707,"।":708,"ãĥ»":709,"ãĥĬ":710,"ãĥ©ãĥ":711,"ing":712,"ineler":713,"lly":714,"ÙIJÙĬ":715,"ÙIJسÙĴ":716,"ï¼ģ":717,"ï¼Į":718,"ï¼ļ":719,"ï¼IJï¼":720,"ÙĦÙİÙij":721,"ãĤ¿ãĤ«":722,"'dan":723,"'su":724,"'den":725,"'deki":726,"een":727,"eep":728,"ero":729,"urd":730,"ÅŁik":731,"ĠABD":732,"ĠAlp":733,"ĠAnkara":734,"ĠAtat":735,"Ġkez":736,"ĠkarÅŁ":737,"Ġदà¥":738,"Ġहिà¤":739,"Ġà¤ķà¥įà¤":740,"Ġà¤Ńा":741,"ĠاÙĦعر":742,"ĠاÙĦÙĦÙİÙij":743,"artmo":744,"ündeme":745,"ðŁij§":746,"ðŁij©":747,"ÙİÙ°Ù":748,"'alt":749,"'aujo":750,"'intell":751,"199":752,"ath":753,"endi":754,"etir":755,"onfer":756,"rag":757,"ÙħÙİÙ°Ù":758,"ĠIBM":759,"Ġbook":760,"Ġbeen":761,"Ġit":762,"Ġilk":763,"Ġidea":764,"Ġsee":765,"Ġsay":766,"Ġsor":767,"Ġyou":768,"Ġyıll":769,"Ġyendi":770,"ĠÑıз":771,"Ġשָ×":772,"'est":773,"200":774,"201":775,"aspar":776,"ceÄŁi":777,"ifici":778,"kiye":779,"unu":780,"usunu":781,"yeceÄŁi":782,"¦à¥Ģ":783,"ªà¸§à¸±à¸":784,"ªà¸Ķà¸":785,"»Î·Î":786,"¼Î±Ï":787,"киÐ":788,"ÑģÑģ":789,"Ö´×ijÖ°×":790,"بÙĬ":791,"بÙIJسÙĴ":792,"ÙĴÙħÙİÙ°Ù":793,"ä¸ĸçķ":794,"ä¸Ńæĸ":795,"ĠDeep":796,"ĠDartmo":797,"ĠKonfer":798,"ĠKaspar":799,"Ġall":800,"Ġcan":801,"Ġgündeme":802,"Ġgetir":803,"ĠγÏ":804,"ĠÎķλÎ":805,"ĠíħĮ":806,"Ġíķľ":807,"ĠĠĠ":808,"Ġà¸łà¸²":809,"Ġสวัà¸":810,"Ġdell":811,"ĠdeÄŁi":812,"ĠðŁij¨":813,"ĠðŁijįðŁ":814,"Ħรัà¸":815,"नमà¤":816,"ürk":817,"ãĤ«ãĤ¿ãĤ«":818,"Ġdone":819,"Ġkavra":820,"Ġkazan":821,"Ġ×¢×ķÖ¹×":822,"Ġ×¢Ö´×ijÖ°×":823,"िया":824,"×ķÖ¹×Ŀ":825,"Ġmakale":826,"Ġmakineler":827,"ĠdÃ¼ÅŁÃ¼neme":828,"ĠdÃ¼ÅŁÃ¼nüp":829,"ĠاÙĦرÙİÙijØŃÙIJÙĬ":830,"ĠاÙĦرÙİÙijØŃÙĴÙħÙİÙ°Ù":831,"#frag":832,"'Rei":833,"'hui":834,"'li":835,"'nda":836,"'re":837,"'ve":838,"'arte":839,"):Ċ":840,"/path":841,"141":842,"234":843,"567":844,"@mail":845,"Alan":846,"Carthy":847,"Emoji":848,"HOME":849,"Türkiye":850,"Yapay":851,"ample":852,"bashr":853,"elle":854,"example":855,"fından":856,"https":857,"handle":858,"igence":859,"likler":860,"ment":861,"taya":862,"¡ãĥ³":863,"¤Ù¥":864,"¤íĬ¸ì":865,"¤à¥ĩ":866,"¤à¥įरिय":867,"¥æľ¬è":868,"¨à¥įदà¥Ģ":869,"¨Ö´×Ļת":870,"ªŀãģ":871,"®âĦ¢":872,"®ãĥĨãĤ":873,"µà¸Ħรัà¸":874,"·à¤¤à¥įरिय":875,"¸à¥įतà¥ĩ":876,"¹ÎºÎ¬":877,"¹ãĥĪ":878,"¼ãĥ¡ãĥ³":879,"½Î¹ÎºÎ¬":880,"μμαÏ":881,"ÐłÑĥ":882,"Ñĭк":883,"ÙĩÙIJ":884,"٣٤٥":885,"à¹Ħà¸Ĺà¸":886,"âħ«":887,"âijł":888,"ãĢĮ":889,"ä½łå¥½":890,"æĹ¥æľ¬è":891,"êµŃìĸ":892,"ìĬ¤íĬ¸ì":893,"Ġartifici":894,"Ġatıldı":895,"Ġcomment":896,"ĠGoogle":897,"Ġoldu":898,"Ġortaya":899},"merges":[["à","¤"],["à","¸"],["a","r"],["d","e"],["Ã","¼"],["Ä","±"],["a","n"],["à","¥"],["ã","ĥ"],["l","e"],["i","n"],["l","l"],["Ù","IJ"],["ï","¼"],["Ù","Ħ"],["ã","Ĥ"],["ð","Ł"],["Ġ","d"],["'","d"],["'","s"],["'","de"],["e","e"],["e","r"],["u","r"],["§","ÙĦ"],["Å","Ł"],["Ø","§ÙĦ"],["Ù","İ"],["Ġ","A"],["Ġ","k"],["Ġ","Ð"],["Ġ","à¤"],["Ġ","اÙĦ"],["į","à¤"],["ar","t"],["ü","n"],["à¥","įà¤"],["ðŁ","ij"],["Ùİ","Ù"],["'","a"],["'","in"],["1","9"],["a","t"],["a","y"],["c","o"],["e","n"],["e","t"],["m","a"],["o","n"],["r","a"],["¿","à¤"],["Ð","·"],["Ø","±"],["Ù","ħ"],["Ġ","I"],["Ġ","b"],["Ġ","i"],["Ġ","s"],["Ġ","y"],["Ġ","Ñ"],["Ġ","×"],["Ģ","į"],["à¤","¾"],["à¤","¿à¤"],["ÙİÙ","ij"],["co","m"],["'","e"],["'","t"],[".","com"],["2","0"],["G","o"],["J","o"],["a","s"],["c","e"],["d","a"],["d","i"],["d","ı"],["h","n"],["i","f"],["i","k"],["k","i"],["m","e"],["m","o"],["o","l"],["o","r"],["r","et"],["s","t"],["u","n"],["u","s"],["y","e"],["¦","à¥"],["ª","à¸"],["°","з"],["±","à¸"],["´","×"],["µ","Ñ"],["¸","×"],["¹","×"],["»","Î"],["¼","Î"],["Ä","Ł"],["Î","¬"],["Ð","º"],["Ð","°Ð·"],["Ñ","ģ"],["Ñ","ij"],["Ö","´×"],["Ö","¸×"],["Ö","¹×"],["×","ķ"],["Ø","¨"],["Ø","Ń"],["Ù","Ĭ"],["Ù","Ĵ"],["â","Ģį"],["ä","¸"],["ë","ĭ"],["Ġ","#"],["Ġ","D"],["Ġ","K"],["Ġ","a"],["Ġ","c"],["Ġ","g"],["Ġ","Î"],["Ġ","í"],["Ġ","Ġ"],["Ġ","à¸"],["Ġ","de"],["Ġ","ðŁij"],["Ġ","ma"],["Ġ","Jo"],["Ġ","ret"],["Ģ","аз"],["Ħ","à¸"],["ī","Ñij"],["à¤","¨"],["à¸","²"],["à¸","±à¸"],["ar","a"],["ü","r"],["ü","ÅŁ"],["ı","n"],["le","r"],["ãĤ","«"],["Ġd","on"],["Ġd","Ã¼ÅŁ"],["'d","a"],["ur","n"],["Ġk","a"],["ĠÐ","¸"],["ĠÐ","µÑ"],["ĠاÙĦ","ر"],["19","5"],["Ùħ","ÙIJ"],["ĠÑ","Ģаз"],["Ġ×","¢"],["िà¤","¯"],["ÙİÙij","ØŃ"],["ÄŁ","i"],["×ķ","Ö¹×"],["Ġma","k"],["ĠJo","hn"],["Ġret","urn"],["ĠdÃ¼ÅŁ","ün"],["ĠеÑ","īÑij"],["ĠاÙĦر","ÙİÙijØŃ"],["#","f"],["&","&"],["&","r"],["'","R"],["'","h"],["'","l"],["'","m"],["'","n"],["'","r"],["'","u"],["'","v"],["'","ll"],["'","art"],["(","x"],[")",":"],["*","*"],["-",">"],["/","."],["/","/"],["/","p"],["1","0"],["1","4"],["2","3"],["4","2"],["5","6"],["5","9"],[":","//"],["?","q"],["@","ma"],["A","l"],["B","D"],["B","M"],["B","l"],["C","art"],["E","mo"],["H","O"],["L","ee"],["M","E"],["M","c"],["S","e"],["T","ur"],["T","ür"],["Y","a"],["[","i"],["]",";"],["a","g"],["a","m"],["a","le"],["a","Go"],["b","u"],["b","as"],["c","i"],["d","u"],["d","an"],["d","le"],["d","ol"],["e","i"],["e","k"],["e","l"],["e","x"],["e","z"],["e","ll"],["e","me"],["e","ler"],["f","er"],["f","ın"],["g","le"],["g","en"],["h","r"],["h","t"],["h","y"],["h","an"],["h","er"],["h","aGo"],["i","l"],["i","r"],["i","ci"],["i","gen"],["j","i"],["j","o"],["k","ara"],["l","k"],["l","p"],["l","t"],["l","dı"],["l","ik"],["m","ı"],["m","en"],["n","kara"],["o","k"],["o","u"],["o","v"],["o","gle"],["o","ok"],["p","s"],["p","ar"],["p","le"],["p","ay"],["s","i"],["s","ı"],["t","h"],["t","at"],["t","ay"],["t","ara"],["t","ag"],["t","ell"],["t","ps"],["u","e"],["u","i"],["u","jo"],["u","th"],["v","ra"],["w","e"],["z","an"],["z","ek"],["}","č"],["~","/."],["¡","ãĥ"],["£","Ù"],["£","ัà¸"],["¤","Ù"],["¤","í"],["¤","à¥"],["¤","à¥įà¤"],["¥","½"],["¥","æ"],["§","ัà¸"],["¨","à¥įà¤"],["¨","Ö´×"],["©","Â"],["©","ãĥ"],["©","Ö¸×"],["©","า"],["ª","ŀ"],["¬","è"],["®","â"],["®","à¤"],["®","ãĥ"],["¯","ķ"],["°","×"],["°","Ù"],["°","st"],["°","िय"],["±","Ï"],["³","Ï"],["³","ÙĴ"],["µ","ĭ"],["µ","Ń"],["µ","à¸"],["·","Î"],["·","à¤"],["·","ा"],["¸","Ð"],["¸","ì"],["¸","ı"],["¸","à¥įà¤"],["¹","Î"],["¹","ãĥ"],["¹","ر"],["¹","िà¤"],["¹","Ħà¸"],["º","ά"],["¼","ãĥ"],["½","Î"],["½","ł"],["¿","ãĤ«"],["Â","²"],["Â","½"],["Â","©Â"],["Ã","¢"],["Ã","¨"],["Ä","°st"],["Î","±"],["Î","»Î"],["Î","¼Î"],["Ð","ł"],["Ð","¸Ð"],["Ñ","ĥ"],["Ñ","ĭ"],["Ö","°×"],["×","ª"],["×","ľ"],["Ø","©"],["Ø","³ÙĴ"],["Ø","¹Ø±"],["Ù","ĩ"],["Ù","£Ù"],["à","¹Ħà¸"],["â","ħ"],["â","ij"],["â","Ŀ"],["ã","Ģ"],["ã","ģ"],["ã","Ģį"],["ä","½ł"],["å","¥½"],["æ","ĸ"],["æ","Ĺ"],["æ","µĭ"],["ç","ķ"],["è","¯ķ"],["ê","µŃ"],["ì","Ĭ"],["ì","ĸ"],["ï","¸ı"],["ĉ","if"],["Ġ","$"],["Ġ","("],["Ġ","+"],["Ġ","-"],["Ġ","@"],["Ġ","O"],["Ġ","f"],["Ġ","x"],["Ġ","{"],["Ġ","art"],["Ġ","at"],["Ġ","com"],["Ġ","Go"],["Ġ","ol"],["Ġ","or"],["Ġ","un"],["Ġ","us"],["Ġ","&&"],["Ġ","Bl"],["Ġ","Lee"],["Ġ","Mc"],["Ġ","Se"],["Ġ","Tur"],["Ġ","her"],["Ġ","tara"],["Ġ","we"],["Ġ","zek"],["Ġ","}č"],["Ġ","~/."],["Ġ","©Â"],["Ġ","İst"],["Ġ","âĿ"],["ģ","ά"],["ģ","न"],["ģ","׾"],["Ħ","¢"],["Ħ","α"],["ħ","Į"],["ħ","ëĭ"],["Ĩ","ÙIJ"],["Ĩ","ãĤ"],["ĩ","æµĭ"],["Ī","ëĭ"],["Ĭ","¸ì"],["į","ðŁ"],["ı","½"],["ı","з"],["IJ","ï¼"],["ij","ï¼"],["ij","Ö°×"],["Ķ","à¸"],["ķ","ľ"],["ķ","à¥įà¤"],["ķ","λÎ"],["ĸ","çķ"],["Ĺ","à¸"],["Ļ","ת"],["ľ","Ö¸×"],["ľ","¬è"],["ŀ","ħëĭ"],["ł","า"],["Ń","ãĤ"],["Ń","ा"],["Ń","æĸ"],["à¤","®à¤"],["à¤","·à¤¾"],["à¸","©à¸²"],["ar","ÅŁ"],["ar","da"],["de","a"],["de","f"],["de","me"],["ü","p"],["ı","ll"],["ı","ldı"],["an","bu"],["an","sı"],["à¥","¤"],["ãĥ","»"],["ãĥ","Ĭ"],["ãĥ","©ãĥ"],["in","g"],["in","eler"],["ll","y"],["ÙIJ","ÙĬ"],["ÙIJ","سÙĴ"],["ï¼","ģ"],["ï¼","Į"],["ï¼","ļ"],["ï¼","IJï¼"],["ÙĦ","ÙİÙij"],["ãĤ","¿ãĤ«"],["'d","an"],["'s","u"],["'de","n"],["'de","ki"],["ee","n"],["ee","p"],["er","o"],["ur","d"],["ÅŁ","ik"],["ĠA","BD"],["ĠA","lp"],["ĠA","nkara"],["ĠA","tat"],["Ġk","ez"],["Ġk","arÅŁ"],["Ġà¤","¦à¥"],["Ġà¤","¹à¤¿à¤"],["Ġà¤","ķà¥įà¤"],["Ġà¤","Ńा"],["ĠاÙĦ","عر"],["ĠاÙĦ","ÙĦÙİÙij"],["art","mo"],["ün","deme"],["ðŁij","§"],["ðŁij","©"],["ÙİÙ","°Ù"],["'a","lt"],["'a","ujo"],["'in","tell"],["19","9"],["at","h"],["en","di"],["et","ir"],["on","fer"],["ra","g"],["Ùħ","ÙİÙ°Ù"],["ĠI","BM"],["Ġb","ook"],["Ġb","een"],["Ġi","t"],["Ġi","lk"],["Ġi","dea"],["Ġs","ee"],["Ġs","ay"],["Ġs","or"],["Ġy","ou"],["Ġy","ıll"],["Ġy","endi"],["ĠÑ","ıз"],["Ġ×","©Ö¸×"],["'e","st"],["20","0"],["20","1"],["as","par"],["ce","ÄŁi"],["if","ici"],["ki","ye"],["un","u"],["us","unu"],["ye","ceÄŁi"],["¦à¥","Ģ"],["ªà¸","§à¸±à¸"],["ªà¸","Ķà¸"],["»Î","·Î"],["¼Î","±Ï"],["к","иÐ"],["Ñģ","Ñģ"],["Ö´×","ijÖ°×"],["ب","ÙĬ"],["ب","ÙIJسÙĴ"],["ÙĴ","ÙħÙİÙ°Ù"],["ä¸","ĸçķ"],["ä¸","Ńæĸ"],["ĠD","eep"],["ĠD","artmo"],["ĠK","onfer"],["ĠK","aspar"],["Ġa","ll"],["Ġc","an"],["Ġg","ündeme"],["Ġg","etir"],["ĠÎ","³Ï"],["ĠÎ","ķλÎ"],["Ġí","ħĮ"],["Ġí","ķľ"],["ĠĠ","Ġ"],["Ġà¸","łà¸²"],["Ġà¸","ªà¸§à¸±à¸"],["Ġde","ll"],["Ġde","ÄŁi"],["ĠðŁij","¨"],["ĠðŁij","įðŁ"],["Ħà¸","£à¸±à¸"],["न","मà¤"],["ür","k"],["ãĤ«","ãĤ¿ãĤ«"],["Ġdon","e"],["Ġka","vra"],["Ġka","zan"],["Ġ×¢","×ķÖ¹×"],["Ġ×¢","Ö´×ijÖ°×"],["िय","ा"],["×ķÖ¹×","Ŀ"],["Ġmak","ale"],["Ġmak","ineler"],["ĠdÃ¼ÅŁÃ¼n","eme"],["ĠdÃ¼ÅŁÃ¼n","üp"],["ĠاÙĦرÙİÙijØŃ","ÙIJÙĬ"],["ĠاÙĦرÙİÙijØŃ","ÙĴÙħÙİÙ°Ù"],["#f","rag"],["'R","ei"],["'h","ui"],["'l","i"],["'n","da"],["'r","e"],["'v","e"],["'art","e"],["):","Ċ"],["/p","ath"],["14","1"],["23","4"],["56","7"],["@ma","il"],["Al","an"],["Cart","hy"],["Emo","ji"],["HO","ME"],["Tür","kiye"],["Ya","pay"],["am","ple"],["bas","hr"],["el","le"],["ex","ample"],["fın","dan"],["ht","tps"],["han","dle"],["igen","ce"],["lik","ler"],["men","t"],["tay","a"],["¡ãĥ","³"],["¤Ù","¥"],["¤í","Ĭ¸ì"],["¤à¥","ĩ"],["¤à¥įà¤","°à¤¿à¤¯"],["¥æ","ľ¬è"],["¨à¥įà¤","¦à¥Ģ"],["¨Ö´×","Ļת"],["ªŀ","ãģ"],["®â","Ħ¢"],["®ãĥ","ĨãĤ"],["µà¸","Ħรัà¸"],["·à¤","¤à¥įरिय"],["¸à¥įà¤","¤à¥ĩ"],["¹Î","ºÎ¬"],["¹ãĥ","Ī"],["¼ãĥ","¡ãĥ³"],["½Î","¹ÎºÎ¬"],["μÎ","¼Î±Ï"],["Ðł","Ñĥ"],["Ñĭ","к"],["Ùĩ","ÙIJ"],["Ù£Ù","¤Ù¥"],["à¹Ħà¸","Ĺà¸"],["âħ","«"],["âij","ł"],["ãĢ","Į"],["ä½ł","好"],["æĹ","¥æľ¬è"],["êµŃ","ìĸ"],["ìĬ","¤íĬ¸ì"],["Ġart","ifici"],["Ġat","ıldı"],["Ġcom","ment"],["ĠGo","ogle"],["Ġol","du"],["Ġor","taya"]]}}
\ No newline at end of file
diff --git a/c/tok.h b/c/tok.h
index 83b649df..9f452f24 100644
--- a/c/tok.h
+++ b/c/tok.h
@@ -23,6 +23,7 @@
#include "json.h"
#include "tok_unicode.h"
#include "tok_unicode_o200k.h"
+#include "tok_unicode_deepseek.h"
/* ---------- hash map (chiavi binarie con lunghezza) ---------- */
typedef struct { const char *k; int klen; int v; int used; } ment;
@@ -60,6 +61,9 @@ typedef struct {
* Han excluded from the letter classes, no '/' tail in the punct rule */
int gpt2; /* 1 = GPT-2 family (OLMoE / GPT-NeoX): a bare ByteLevel pre_tokenizer
* with use_regex and no Split, so HF applies the original GPT-2 pattern */
+ int deepseek; /* 1 = DeepSeek V4 / V4.1: three chained Splits (digit groups, CJK
+ * runs, then a regex of its own) -- not cl100k, see
+ * pretok_chunk_deepseek */
int rankbpe; /* 1 = no merges list (tiktoken-derived vocab): merge the adjacent
* pair whose CONCATENATION has the lowest vocab id — exactly
* tiktoken's byte_pair_encode, no recovered merges to diverge */
@@ -233,6 +237,10 @@ static void tok_load(Tok *T, const char *path){
jval *rx=pat?json_get(pat,"Regex"):NULL;
if(rx&&rx->t==J_STR&&strstr(rx->str,"\\p{Lu}")) T->o200k=1;
if(rx&&rx->t==J_STR&&strstr(rx->str,"\\p{Han}")) T->kimi=1;
+ /* DeepSeek's third Split: ASCII punctuation takes the ASCII letters after
+ * it, and marks join letter runs -- no other family writes either */
+ if(rx&&rx->t==J_STR&&strstr(rx->str,"][A-Za-z]+|")&&strstr(rx->str,"[\\p{L}\\p{M}]+"))
+ T->deepseek=1;
}
}
T->json_root=root;
@@ -587,6 +595,115 @@ static void pretok_chunk_gpt2(Tok *T, const unsigned char *p, int a, int b, int
free(cp); free(off);
}
+/* ---------- pre-tokenizer DeepSeek V4 / V4.1 ----------
+ * tokenizer.json chains three Splits, all "Isolated", each one applied to every piece the
+ * one before produced, then a ByteLevel without its own regex:
+ * 1. \p{N}{1,3}
+ * 2. [一-龥぀-ゟ゠-ヿ]+
+ * 3. [!"#$%&'()*+,\-./:;<=>?@\[\\\]^_`{|}~][A-Za-z]+
+ * | [^\r\n\p{L}\p{P}\p{S}]?[\p{L}\p{M}]+
+ * | ?[\p{P}\p{S}]+[\r\n]*
+ * | \s*[\r\n]+ | \s+(?!\S) | \s+
+ * Isolated keeps whatever a Split does not match as a piece of its own, so a digit group
+ * never joins a letter and a CJK run never takes the space before it.
+ * It is not cl100k, which this tokenizer used to be read as. There are no English
+ * contractions: the apostrophe takes the ASCII letters after it, so "1956'da" is 195|6|'da
+ * where cl100k made it 195|6|'d|a -- every Turkish suffix after an apostrophe that starts
+ * with d, s, t or m came out as a split the model never saw (#1772). Marks join letter
+ * runs, and the punctuation rule is \p{P}\p{S} instead of "neither space, letter nor
+ * number". */
+static int ds_ascii_punct(uint32_t c){
+ return (c>=0x21&&c<=0x2F)||(c>=0x3A&&c<=0x40)||(c>=0x5B&&c<=0x60)||(c>=0x7B&&c<=0x7E);
+}
+static int ds_ascii_alpha(uint32_t c){ return (c>='A'&&c<='Z')||(c>='a'&&c<='z'); }
+static int ds_cjk(uint32_t c){
+ return (c>=0x4E00&&c<=0x9FA5)||(c>=0x3040&&c<=0x309F)||(c>=0x30A0&&c<=0x30FF);
+}
+static int ds_lm(uint32_t c){ return is_L(c)||is_Mk(c); }
+static int ds_ps(uint32_t c){ return is_Pu(c)||is_Sy(c); }
+
+/* The third Split at position i of the piece [i, e): the end of the match, -1 if none of
+ * the alternatives matches here. Alternatives in order, as the regex engine tries them. */
+static int ds_match(const uint32_t *cp, int i, int e){
+ uint32_t c=cp[i];
+ /* [ASCII punctuation][A-Za-z]+ */
+ if(ds_ascii_punct(c) && i+1<e && ds_ascii_alpha(cp[i+1])){
+ int j=i+1; while(j<e && ds_ascii_alpha(cp[j])) j++; return j;
+ }
+ /* [^\r\n\p{L}\p{P}\p{S}]?[\p{L}\p{M}]+ -- the optional prefix first, then without */
+ if(c!='\r' && c!='\n' && !is_L(c) && !ds_ps(c) && i+1<e && ds_lm(cp[i+1])){
+ int j=i+1; while(j<e && ds_lm(cp[j])) j++; return j;
+ }
+ if(ds_lm(c)){ int j=i; while(j<e && ds_lm(cp[j])) j++; return j; }
+ /* ?[\p{P}\p{S}]+[\r\n]* */
+ {
+ int j=-1;
+ if(c==' ' && i+1<e && ds_ps(cp[i+1])) j=i+1;
+ else if(ds_ps(c)) j=i;
+ if(j>=0){
+ while(j<e && ds_ps(cp[j])) j++;
+ while(j<e && (cp[j]=='\r'||cp[j]=='\n')) j++;
+ return j;
+ }
+ }
+ /* \s*[\r\n]+ reaches the last newline of the whitespace run; \s+(?!\S) stops one short
+ * of a following non-space, unless that leaves nothing; \s+ takes the run */
+ {
+ int r=i; while(r<e && is_S(cp[r])) r++;
+ if(r>i){
+ int last=-1;
+ for(int j=i;j<r;j++) if(cp[j]=='\r'||cp[j]=='\n') last=j;
+ if(last>=0) return last+1;
+ return (r<e && r-1>i) ? r-1 : r;
+ }
+ }
+ return -1;
+}
+
+/* The third Split over one piece: every match is a piece, and so is every run between
+ * matches (Isolated). */
+static void ds_split3(Tok *T, const unsigned char *p, const uint32_t *cp, const int *off,
+ int s, int e, int *out, int *no, int max){
+ int i=s, pending=s;
+ while(i<e){
+ int j=ds_match(cp,i,e);
+ if(j<0){ i++; continue; }
+ if(pending<i) bpe_piece(T,p,off[pending],off[i],out,no,max);
+ bpe_piece(T,p,off[i],off[j],out,no,max);
+ i=pending=j;
+ }
+ if(pending<e) bpe_piece(T,p,off[pending],off[e],out,no,max);
+}
+
+/* The second Split: CJK runs apart from the rest, and both through the third. */
+static void ds_split2(Tok *T, const unsigned char *p, const uint32_t *cp, const int *off,
+ int s, int e, int *out, int *no, int max){
+ int i=s;
+ while(i<e){
+ int j=i, cjk=ds_cjk(cp[i]);
+ while(j<e && ds_cjk(cp[j])==cjk) j++;
+ ds_split3(T,p,cp,off,i,j,out,no,max);
+ i=j;
+ }
+}
+
+static void pretok_chunk_deepseek(Tok *T, const unsigned char *p, int a, int b, int *out, int *no, int max){
+ int nb=b-a; if(nb<=0) return;
+ uint32_t *cp=malloc((nb+1)*sizeof(uint32_t)); int *off=malloc((nb+2)*sizeof(int)); int n=0;
+ for(int i=a;i<b;){ uint32_t c; int k=u8_next(p,b,i,&c); off[n]=i; cp[n]=c; n++; i+=k; }
+ off[n]=b;
+ /* the first Split: groups of up to three numbers, left to right, each its own piece */
+ int i=0;
+ while(i<n){
+ int j=i;
+ if(is_N(cp[i])){ while(j<n && is_N(cp[j]) && j-i<3) j++; }
+ else { while(j<n && !is_N(cp[j])) j++; }
+ ds_split2(T,p,cp,off,i,j,out,no,max);
+ i=j;
+ }
+ free(cp); free(off);
+}
+
/* ---------- encode: testo -> id (split sugli added token, poi pretok+BPE) ---------- */
static int tok_encode(Tok *T, const char *text, int len, int *out, int max){
const unsigned char *p=(const unsigned char*)text; int no=0; int i=0;
@@ -602,6 +719,7 @@ static int tok_encode(Tok *T, const char *text, int len, int *out, int max){
int chunk_end = (hitpos<0) ? len : hitpos;
if(chunk_end>i){
if(T->gpt2) pretok_chunk_gpt2(T,p,i,chunk_end,out,&no,max);
+ else if(T->deepseek) pretok_chunk_deepseek(T,p,i,chunk_end,out,&no,max);
else if(T->kimi) pretok_chunk_kimi(T,p,i,chunk_end,out,&no,max);
else if(T->o200k) pretok_chunk_o200k(T,p,i,chunk_end,out,&no,max);
else pretok_chunk(T,p,i,chunk_end,out,&no,max);
diff --git a/c/tok_unicode_deepseek.h b/c/tok_unicode_deepseek.h
new file mode 100644
index 00000000..f044f96c
--- /dev/null
+++ b/c/tok_unicode_deepseek.h
@@ -0,0 +1,145 @@
+/* GENERATED by tools/gen_unicode_deepseek.py (Unicode 15.0.0) -- do not edit by hand. */
+#ifndef TOK_UNICODE_DEEPSEEK_H
+#define TOK_UNICODE_DEEPSEEK_H
+#include <stdint.h>
+#include "tok_unicode.h"
+
+static const uint32_t uni_Mk[][2] = {
+ {0x300,0x36F},{0x483,0x489},{0x591,0x5BD},{0x5BF,0x5BF},{0x5C1,0x5C2},{0x5C4,0x5C5},
+ {0x5C7,0x5C7},{0x610,0x61A},{0x64B,0x65F},{0x670,0x670},{0x6D6,0x6DC},{0x6DF,0x6E4},
+ {0x6E7,0x6E8},{0x6EA,0x6ED},{0x711,0x711},{0x730,0x74A},{0x7A6,0x7B0},{0x7EB,0x7F3},
+ {0x7FD,0x7FD},{0x816,0x819},{0x81B,0x823},{0x825,0x827},{0x829,0x82D},{0x859,0x85B},
+ {0x898,0x89F},{0x8CA,0x8E1},{0x8E3,0x903},{0x93A,0x93C},{0x93E,0x94F},{0x951,0x957},
+ {0x962,0x963},{0x981,0x983},{0x9BC,0x9BC},{0x9BE,0x9C4},{0x9C7,0x9C8},{0x9CB,0x9CD},
+ {0x9D7,0x9D7},{0x9E2,0x9E3},{0x9FE,0x9FE},{0xA01,0xA03},{0xA3C,0xA3C},{0xA3E,0xA42},
+ {0xA47,0xA48},{0xA4B,0xA4D},{0xA51,0xA51},{0xA70,0xA71},{0xA75,0xA75},{0xA81,0xA83},
+ {0xABC,0xABC},{0xABE,0xAC5},{0xAC7,0xAC9},{0xACB,0xACD},{0xAE2,0xAE3},{0xAFA,0xAFF},
+ {0xB01,0xB03},{0xB3C,0xB3C},{0xB3E,0xB44},{0xB47,0xB48},{0xB4B,0xB4D},{0xB55,0xB57},
+ {0xB62,0xB63},{0xB82,0xB82},{0xBBE,0xBC2},{0xBC6,0xBC8},{0xBCA,0xBCD},{0xBD7,0xBD7},
+ {0xC00,0xC04},{0xC3C,0xC3C},{0xC3E,0xC44},{0xC46,0xC48},{0xC4A,0xC4D},{0xC55,0xC56},
+ {0xC62,0xC63},{0xC81,0xC83},{0xCBC,0xCBC},{0xCBE,0xCC4},{0xCC6,0xCC8},{0xCCA,0xCCD},
+ {0xCD5,0xCD6},{0xCE2,0xCE3},{0xCF3,0xCF3},{0xD00,0xD03},{0xD3B,0xD3C},{0xD3E,0xD44},
+ {0xD46,0xD48},{0xD4A,0xD4D},{0xD57,0xD57},{0xD62,0xD63},{0xD81,0xD83},{0xDCA,0xDCA},
+ {0xDCF,0xDD4},{0xDD6,0xDD6},{0xDD8,0xDDF},{0xDF2,0xDF3},{0xE31,0xE31},{0xE34,0xE3A},
+ {0xE47,0xE4E},{0xEB1,0xEB1},{0xEB4,0xEBC},{0xEC8,0xECE},{0xF18,0xF19},{0xF35,0xF35},
+ {0xF37,0xF37},{0xF39,0xF39},{0xF3E,0xF3F},{0xF71,0xF84},{0xF86,0xF87},{0xF8D,0xF97},
+ {0xF99,0xFBC},{0xFC6,0xFC6},{0x102B,0x103E},{0x1056,0x1059},{0x105E,0x1060},{0x1062,0x1064},
+ {0x1067,0x106D},{0x1071,0x1074},{0x1082,0x108D},{0x108F,0x108F},{0x109A,0x109D},{0x135D,0x135F},
+ {0x1712,0x1715},{0x1732,0x1734},{0x1752,0x1753},{0x1772,0x1773},{0x17B4,0x17D3},{0x17DD,0x17DD},
+ {0x180B,0x180D},{0x180F,0x180F},{0x1885,0x1886},{0x18A9,0x18A9},{0x1920,0x192B},{0x1930,0x193B},
+ {0x1A17,0x1A1B},{0x1A55,0x1A5E},{0x1A60,0x1A7C},{0x1A7F,0x1A7F},{0x1AB0,0x1ACE},{0x1B00,0x1B04},
+ {0x1B34,0x1B44},{0x1B6B,0x1B73},{0x1B80,0x1B82},{0x1BA1,0x1BAD},{0x1BE6,0x1BF3},{0x1C24,0x1C37},
+ {0x1CD0,0x1CD2},{0x1CD4,0x1CE8},{0x1CED,0x1CED},{0x1CF4,0x1CF4},{0x1CF7,0x1CF9},{0x1DC0,0x1DFF},
+ {0x20D0,0x20F0},{0x2CEF,0x2CF1},{0x2D7F,0x2D7F},{0x2DE0,0x2DFF},{0x302A,0x302F},{0x3099,0x309A},
+ {0xA66F,0xA672},{0xA674,0xA67D},{0xA69E,0xA69F},{0xA6F0,0xA6F1},{0xA802,0xA802},{0xA806,0xA806},
+ {0xA80B,0xA80B},{0xA823,0xA827},{0xA82C,0xA82C},{0xA880,0xA881},{0xA8B4,0xA8C5},{0xA8E0,0xA8F1},
+ {0xA8FF,0xA8FF},{0xA926,0xA92D},{0xA947,0xA953},{0xA980,0xA983},{0xA9B3,0xA9C0},{0xA9E5,0xA9E5},
+ {0xAA29,0xAA36},{0xAA43,0xAA43},{0xAA4C,0xAA4D},{0xAA7B,0xAA7D},{0xAAB0,0xAAB0},{0xAAB2,0xAAB4},
+ {0xAAB7,0xAAB8},{0xAABE,0xAABF},{0xAAC1,0xAAC1},{0xAAEB,0xAAEF},{0xAAF5,0xAAF6},{0xABE3,0xABEA},
+ {0xABEC,0xABED},{0xFB1E,0xFB1E},{0xFE00,0xFE0F},{0xFE20,0xFE2F},{0x101FD,0x101FD},{0x102E0,0x102E0},
+ {0x10376,0x1037A},{0x10A01,0x10A03},{0x10A05,0x10A06},{0x10A0C,0x10A0F},{0x10A38,0x10A3A},{0x10A3F,0x10A3F},
+ {0x10AE5,0x10AE6},{0x10D24,0x10D27},{0x10EAB,0x10EAC},{0x10EFD,0x10EFF},{0x10F46,0x10F50},{0x10F82,0x10F85},
+ {0x11000,0x11002},{0x11038,0x11046},{0x11070,0x11070},{0x11073,0x11074},{0x1107F,0x11082},{0x110B0,0x110BA},
+ {0x110C2,0x110C2},{0x11100,0x11102},{0x11127,0x11134},{0x11145,0x11146},{0x11173,0x11173},{0x11180,0x11182},
+ {0x111B3,0x111C0},{0x111C9,0x111CC},{0x111CE,0x111CF},{0x1122C,0x11237},{0x1123E,0x1123E},{0x11241,0x11241},
+ {0x112DF,0x112EA},{0x11300,0x11303},{0x1133B,0x1133C},{0x1133E,0x11344},{0x11347,0x11348},{0x1134B,0x1134D},
+ {0x11357,0x11357},{0x11362,0x11363},{0x11366,0x1136C},{0x11370,0x11374},{0x11435,0x11446},{0x1145E,0x1145E},
+ {0x114B0,0x114C3},{0x115AF,0x115B5},{0x115B8,0x115C0},{0x115DC,0x115DD},{0x11630,0x11640},{0x116AB,0x116B7},
+ {0x1171D,0x1172B},{0x1182C,0x1183A},{0x11930,0x11935},{0x11937,0x11938},{0x1193B,0x1193E},{0x11940,0x11940},
+ {0x11942,0x11943},{0x119D1,0x119D7},{0x119DA,0x119E0},{0x119E4,0x119E4},{0x11A01,0x11A0A},{0x11A33,0x11A39},
+ {0x11A3B,0x11A3E},{0x11A47,0x11A47},{0x11A51,0x11A5B},{0x11A8A,0x11A99},{0x11C2F,0x11C36},{0x11C38,0x11C3F},
+ {0x11C92,0x11CA7},{0x11CA9,0x11CB6},{0x11D31,0x11D36},{0x11D3A,0x11D3A},{0x11D3C,0x11D3D},{0x11D3F,0x11D45},
+ {0x11D47,0x11D47},{0x11D8A,0x11D8E},{0x11D90,0x11D91},{0x11D93,0x11D97},{0x11EF3,0x11EF6},{0x11F00,0x11F01},
+ {0x11F03,0x11F03},{0x11F34,0x11F3A},{0x11F3E,0x11F42},{0x13440,0x13440},{0x13447,0x13455},{0x16AF0,0x16AF4},
+ {0x16B30,0x16B36},{0x16F4F,0x16F4F},{0x16F51,0x16F87},{0x16F8F,0x16F92},{0x16FE4,0x16FE4},{0x16FF0,0x16FF1},
+ {0x1BC9D,0x1BC9E},{0x1CF00,0x1CF2D},{0x1CF30,0x1CF46},{0x1D165,0x1D169},{0x1D16D,0x1D172},{0x1D17B,0x1D182},
+ {0x1D185,0x1D18B},{0x1D1AA,0x1D1AD},{0x1D242,0x1D244},{0x1DA00,0x1DA36},{0x1DA3B,0x1DA6C},{0x1DA75,0x1DA75},
+ {0x1DA84,0x1DA84},{0x1DA9B,0x1DA9F},{0x1DAA1,0x1DAAF},{0x1E000,0x1E006},{0x1E008,0x1E018},{0x1E01B,0x1E021},
+ {0x1E023,0x1E024},{0x1E026,0x1E02A},{0x1E08F,0x1E08F},{0x1E130,0x1E136},{0x1E2AE,0x1E2AE},{0x1E2EC,0x1E2EF},
+ {0x1E4EC,0x1E4EF},{0x1E8D0,0x1E8D6},{0x1E944,0x1E94A},{0xE0100,0xE01EF},
+};
+static const int uni_Mk_n = 310;
+
+static const uint32_t uni_Pu[][2] = {
+ {0x21,0x23},{0x25,0x2A},{0x2C,0x2F},{0x3A,0x3B},{0x3F,0x40},{0x5B,0x5D},
+ {0x5F,0x5F},{0x7B,0x7B},{0x7D,0x7D},{0xA1,0xA1},{0xA7,0xA7},{0xAB,0xAB},
+ {0xB6,0xB7},{0xBB,0xBB},{0xBF,0xBF},{0x37E,0x37E},{0x387,0x387},{0x55A,0x55F},
+ {0x589,0x58A},{0x5BE,0x5BE},{0x5C0,0x5C0},{0x5C3,0x5C3},{0x5C6,0x5C6},{0x5F3,0x5F4},
+ {0x609,0x60A},{0x60C,0x60D},{0x61B,0x61B},{0x61D,0x61F},{0x66A,0x66D},{0x6D4,0x6D4},
+ {0x700,0x70D},{0x7F7,0x7F9},{0x830,0x83E},{0x85E,0x85E},{0x964,0x965},{0x970,0x970},
+ {0x9FD,0x9FD},{0xA76,0xA76},{0xAF0,0xAF0},{0xC77,0xC77},{0xC84,0xC84},{0xDF4,0xDF4},
+ {0xE4F,0xE4F},{0xE5A,0xE5B},{0xF04,0xF12},{0xF14,0xF14},{0xF3A,0xF3D},{0xF85,0xF85},
+ {0xFD0,0xFD4},{0xFD9,0xFDA},{0x104A,0x104F},{0x10FB,0x10FB},{0x1360,0x1368},{0x1400,0x1400},
+ {0x166E,0x166E},{0x169B,0x169C},{0x16EB,0x16ED},{0x1735,0x1736},{0x17D4,0x17D6},{0x17D8,0x17DA},
+ {0x1800,0x180A},{0x1944,0x1945},{0x1A1E,0x1A1F},{0x1AA0,0x1AA6},{0x1AA8,0x1AAD},{0x1B5A,0x1B60},
+ {0x1B7D,0x1B7E},{0x1BFC,0x1BFF},{0x1C3B,0x1C3F},{0x1C7E,0x1C7F},{0x1CC0,0x1CC7},{0x1CD3,0x1CD3},
+ {0x2010,0x2027},{0x2030,0x2043},{0x2045,0x2051},{0x2053,0x205E},{0x207D,0x207E},{0x208D,0x208E},
+ {0x2308,0x230B},{0x2329,0x232A},{0x2768,0x2775},{0x27C5,0x27C6},{0x27E6,0x27EF},{0x2983,0x2998},
+ {0x29D8,0x29DB},{0x29FC,0x29FD},{0x2CF9,0x2CFC},{0x2CFE,0x2CFF},{0x2D70,0x2D70},{0x2E00,0x2E2E},
+ {0x2E30,0x2E4F},{0x2E52,0x2E5D},{0x3001,0x3003},{0x3008,0x3011},{0x3014,0x301F},{0x3030,0x3030},
+ {0x303D,0x303D},{0x30A0,0x30A0},{0x30FB,0x30FB},{0xA4FE,0xA4FF},{0xA60D,0xA60F},{0xA673,0xA673},
+ {0xA67E,0xA67E},{0xA6F2,0xA6F7},{0xA874,0xA877},{0xA8CE,0xA8CF},{0xA8F8,0xA8FA},{0xA8FC,0xA8FC},
+ {0xA92E,0xA92F},{0xA95F,0xA95F},{0xA9C1,0xA9CD},{0xA9DE,0xA9DF},{0xAA5C,0xAA5F},{0xAADE,0xAADF},
+ {0xAAF0,0xAAF1},{0xABEB,0xABEB},{0xFD3E,0xFD3F},{0xFE10,0xFE19},{0xFE30,0xFE52},{0xFE54,0xFE61},
+ {0xFE63,0xFE63},{0xFE68,0xFE68},{0xFE6A,0xFE6B},{0xFF01,0xFF03},{0xFF05,0xFF0A},{0xFF0C,0xFF0F},
+ {0xFF1A,0xFF1B},{0xFF1F,0xFF20},{0xFF3B,0xFF3D},{0xFF3F,0xFF3F},{0xFF5B,0xFF5B},{0xFF5D,0xFF5D},
+ {0xFF5F,0xFF65},{0x10100,0x10102},{0x1039F,0x1039F},{0x103D0,0x103D0},{0x1056F,0x1056F},{0x10857,0x10857},
+ {0x1091F,0x1091F},{0x1093F,0x1093F},{0x10A50,0x10A58},{0x10A7F,0x10A7F},{0x10AF0,0x10AF6},{0x10B39,0x10B3F},
+ {0x10B99,0x10B9C},{0x10EAD,0x10EAD},{0x10F55,0x10F59},{0x10F86,0x10F89},{0x11047,0x1104D},{0x110BB,0x110BC},
+ {0x110BE,0x110C1},{0x11140,0x11143},{0x11174,0x11175},{0x111C5,0x111C8},{0x111CD,0x111CD},{0x111DB,0x111DB},
+ {0x111DD,0x111DF},{0x11238,0x1123D},{0x112A9,0x112A9},{0x1144B,0x1144F},{0x1145A,0x1145B},{0x1145D,0x1145D},
+ {0x114C6,0x114C6},{0x115C1,0x115D7},{0x11641,0x11643},{0x11660,0x1166C},{0x116B9,0x116B9},{0x1173C,0x1173E},
+ {0x1183B,0x1183B},{0x11944,0x11946},{0x119E2,0x119E2},{0x11A3F,0x11A46},{0x11A9A,0x11A9C},{0x11A9E,0x11AA2},
+ {0x11B00,0x11B09},{0x11C41,0x11C45},{0x11C70,0x11C71},{0x11EF7,0x11EF8},{0x11F43,0x11F4F},{0x11FFF,0x11FFF},
+ {0x12470,0x12474},{0x12FF1,0x12FF2},{0x16A6E,0x16A6F},{0x16AF5,0x16AF5},{0x16B37,0x16B3B},{0x16B44,0x16B44},
+ {0x16E97,0x16E9A},{0x16FE2,0x16FE2},{0x1BC9F,0x1BC9F},{0x1DA87,0x1DA8B},{0x1E95E,0x1E95F},
+};
+static const int uni_Pu_n = 191;
+
+static const uint32_t uni_Sy[][2] = {
+ {0x24,0x24},{0x2B,0x2B},{0x3C,0x3E},{0x5E,0x5E},{0x60,0x60},{0x7C,0x7C},
+ {0x7E,0x7E},{0xA2,0xA6},{0xA8,0xA9},{0xAC,0xAC},{0xAE,0xB1},{0xB4,0xB4},
+ {0xB8,0xB8},{0xD7,0xD7},{0xF7,0xF7},{0x2C2,0x2C5},{0x2D2,0x2DF},{0x2E5,0x2EB},
+ {0x2ED,0x2ED},{0x2EF,0x2FF},{0x375,0x375},{0x384,0x385},{0x3F6,0x3F6},{0x482,0x482},
+ {0x58D,0x58F},{0x606,0x608},{0x60B,0x60B},{0x60E,0x60F},{0x6DE,0x6DE},{0x6E9,0x6E9},
+ {0x6FD,0x6FE},{0x7F6,0x7F6},{0x7FE,0x7FF},{0x888,0x888},{0x9F2,0x9F3},{0x9FA,0x9FB},
+ {0xAF1,0xAF1},{0xB70,0xB70},{0xBF3,0xBFA},{0xC7F,0xC7F},{0xD4F,0xD4F},{0xD79,0xD79},
+ {0xE3F,0xE3F},{0xF01,0xF03},{0xF13,0xF13},{0xF15,0xF17},{0xF1A,0xF1F},{0xF34,0xF34},
+ {0xF36,0xF36},{0xF38,0xF38},{0xFBE,0xFC5},{0xFC7,0xFCC},{0xFCE,0xFCF},{0xFD5,0xFD8},
+ {0x109E,0x109F},{0x1390,0x1399},{0x166D,0x166D},{0x17DB,0x17DB},{0x1940,0x1940},{0x19DE,0x19FF},
+ {0x1B61,0x1B6A},{0x1B74,0x1B7C},{0x1FBD,0x1FBD},{0x1FBF,0x1FC1},{0x1FCD,0x1FCF},{0x1FDD,0x1FDF},
+ {0x1FED,0x1FEF},{0x1FFD,0x1FFE},{0x2044,0x2044},{0x2052,0x2052},{0x207A,0x207C},{0x208A,0x208C},
+ {0x20A0,0x20C0},{0x2100,0x2101},{0x2103,0x2106},{0x2108,0x2109},{0x2114,0x2114},{0x2116,0x2118},
+ {0x211E,0x2123},{0x2125,0x2125},{0x2127,0x2127},{0x2129,0x2129},{0x212E,0x212E},{0x213A,0x213B},
+ {0x2140,0x2144},{0x214A,0x214D},{0x214F,0x214F},{0x218A,0x218B},{0x2190,0x2307},{0x230C,0x2328},
+ {0x232B,0x2426},{0x2440,0x244A},{0x249C,0x24E9},{0x2500,0x2767},{0x2794,0x27C4},{0x27C7,0x27E5},
+ {0x27F0,0x2982},{0x2999,0x29D7},{0x29DC,0x29FB},{0x29FE,0x2B73},{0x2B76,0x2B95},{0x2B97,0x2BFF},
+ {0x2CE5,0x2CEA},{0x2E50,0x2E51},{0x2E80,0x2E99},{0x2E9B,0x2EF3},{0x2F00,0x2FD5},{0x2FF0,0x2FFB},
+ {0x3004,0x3004},{0x3012,0x3013},{0x3020,0x3020},{0x3036,0x3037},{0x303E,0x303F},{0x309B,0x309C},
+ {0x3190,0x3191},{0x3196,0x319F},{0x31C0,0x31E3},{0x3200,0x321E},{0x322A,0x3247},{0x3250,0x3250},
+ {0x3260,0x327F},{0x328A,0x32B0},{0x32C0,0x33FF},{0x4DC0,0x4DFF},{0xA490,0xA4C6},{0xA700,0xA716},
+ {0xA720,0xA721},{0xA789,0xA78A},{0xA828,0xA82B},{0xA836,0xA839},{0xAA77,0xAA79},{0xAB5B,0xAB5B},
+ {0xAB6A,0xAB6B},{0xFB29,0xFB29},{0xFBB2,0xFBC2},{0xFD40,0xFD4F},{0xFDCF,0xFDCF},{0xFDFC,0xFDFF},
+ {0xFE62,0xFE62},{0xFE64,0xFE66},{0xFE69,0xFE69},{0xFF04,0xFF04},{0xFF0B,0xFF0B},{0xFF1C,0xFF1E},
+ {0xFF3E,0xFF3E},{0xFF40,0xFF40},{0xFF5C,0xFF5C},{0xFF5E,0xFF5E},{0xFFE0,0xFFE6},{0xFFE8,0xFFEE},
+ {0xFFFC,0xFFFD},{0x10137,0x1013F},{0x10179,0x10189},{0x1018C,0x1018E},{0x10190,0x1019C},{0x101A0,0x101A0},
+ {0x101D0,0x101FC},{0x10877,0x10878},{0x10AC8,0x10AC8},{0x1173F,0x1173F},{0x11FD5,0x11FF1},{0x16B3C,0x16B3F},
+ {0x16B45,0x16B45},{0x1BC9C,0x1BC9C},{0x1CF50,0x1CFC3},{0x1D000,0x1D0F5},{0x1D100,0x1D126},{0x1D129,0x1D164},
+ {0x1D16A,0x1D16C},{0x1D183,0x1D184},{0x1D18C,0x1D1A9},{0x1D1AE,0x1D1EA},{0x1D200,0x1D241},{0x1D245,0x1D245},
+ {0x1D300,0x1D356},{0x1D6C1,0x1D6C1},{0x1D6DB,0x1D6DB},{0x1D6FB,0x1D6FB},{0x1D715,0x1D715},{0x1D735,0x1D735},
+ {0x1D74F,0x1D74F},{0x1D76F,0x1D76F},{0x1D789,0x1D789},{0x1D7A9,0x1D7A9},{0x1D7C3,0x1D7C3},{0x1D800,0x1D9FF},
+ {0x1DA37,0x1DA3A},{0x1DA6D,0x1DA74},{0x1DA76,0x1DA83},{0x1DA85,0x1DA86},{0x1E14F,0x1E14F},{0x1E2FF,0x1E2FF},
+ {0x1ECAC,0x1ECAC},{0x1ECB0,0x1ECB0},{0x1ED2E,0x1ED2E},{0x1EEF0,0x1EEF1},{0x1F000,0x1F02B},{0x1F030,0x1F093},
+ {0x1F0A0,0x1F0AE},{0x1F0B1,0x1F0BF},{0x1F0C1,0x1F0CF},{0x1F0D1,0x1F0F5},{0x1F10D,0x1F1AD},{0x1F1E6,0x1F202},
+ {0x1F210,0x1F23B},{0x1F240,0x1F248},{0x1F250,0x1F251},{0x1F260,0x1F265},{0x1F300,0x1F6D7},{0x1F6DC,0x1F6EC},
+ {0x1F6F0,0x1F6FC},{0x1F700,0x1F776},{0x1F77B,0x1F7D9},{0x1F7E0,0x1F7EB},{0x1F7F0,0x1F7F0},{0x1F800,0x1F80B},
+ {0x1F810,0x1F847},{0x1F850,0x1F859},{0x1F860,0x1F887},{0x1F890,0x1F8AD},{0x1F8B0,0x1F8B1},{0x1F900,0x1FA53},
+ {0x1FA60,0x1FA6D},{0x1FA70,0x1FA7C},{0x1FA80,0x1FA88},{0x1FA90,0x1FABD},{0x1FABF,0x1FAC5},{0x1FACE,0x1FADB},
+ {0x1FAE0,0x1FAE8},{0x1FAF0,0x1FAF8},{0x1FB00,0x1FB92},{0x1FB94,0x1FBCA},
+};
+static const int uni_Sy_n = 232;
+
+static inline int is_Mk(uint32_t c){ return uni_in(uni_Mk,uni_Mk_n,c); }
+static inline int is_Pu(uint32_t c){ return uni_in(uni_Pu,uni_Pu_n,c); }
+static inline int is_Sy(uint32_t c){ return uni_in(uni_Sy,uni_Sy_n,c); }
+#endif
diff --git a/c/tools/gen_unicode_deepseek.py b/c/tools/gen_unicode_deepseek.py
new file mode 100644
index 00000000..920ea096
--- /dev/null
+++ b/c/tools/gen_unicode_deepseek.py
@@ -0,0 +1,49 @@
+"""Generates tok_unicode_deepseek.h: the Unicode classes the DeepSeek V4 / V4.1
+pre-tokenizer needs on top of tok_unicode.h (\\p{L}, \\p{N}, \\s):
+ - \\p{M} marks (category starting with 'M')
+ - \\p{P} punctuation (category starting with 'P')
+ - \\p{S} symbols (category starting with 'S')
+Same range format and the same uni_in() lookup as tok_unicode.h. Run once:
+ python3 tools/gen_unicode_deepseek.py > tok_unicode_deepseek.h
+"""
+import unicodedata
+
+
+def ranges(pred):
+ out, lo = [], None
+ for cp in range(0x110000):
+ if 0xD800 <= cp <= 0xDFFF or not pred(cp):
+ if lo is not None:
+ out.append((lo, cp - 1))
+ lo = None
+ continue
+ if lo is None:
+ lo = cp
+ if lo is not None:
+ out.append((lo, 0x10FFFF))
+ return out
+
+
+def cat(cp):
+ return unicodedata.category(chr(cp))
+
+
+def emit(name, rs):
+ print(f"static const uint32_t {name}[][2] = {{")
+ for i in range(0, len(rs), 6):
+ print(" " + "".join(f"{{0x{lo:X},0x{hi:X}}}," for lo, hi in rs[i:i + 6]))
+ print("};")
+ print(f"static const int {name}_n = {len(rs)};\n")
+
+
+print(f"/* GENERATED by tools/gen_unicode_deepseek.py (Unicode {unicodedata.unidata_version})"
+ " -- do not edit by hand. */")
+print("#ifndef TOK_UNICODE_DEEPSEEK_H\n#define TOK_UNICODE_DEEPSEEK_H\n#include <stdint.h>\n"
+ "#include \"tok_unicode.h\"\n")
+emit("uni_Mk", ranges(lambda c: cat(c).startswith("M")))
+emit("uni_Pu", ranges(lambda c: cat(c).startswith("P")))
+emit("uni_Sy", ranges(lambda c: cat(c).startswith("S")))
+print("static inline int is_Mk(uint32_t c){ return uni_in(uni_Mk,uni_Mk_n,c); }")
+print("static inline int is_Pu(uint32_t c){ return uni_in(uni_Pu,uni_Pu_n,c); }")
+print("static inline int is_Sy(uint32_t c){ return uni_in(uni_Sy,uni_Sy_n,c); }")
+print("#endif")
diff --git a/c/tools/make_deepseek_tok_fixture.py b/c/tools/make_deepseek_tok_fixture.py
new file mode 100644
index 00000000..dc36486a
--- /dev/null
+++ b/c/tools/make_deepseek_tok_fixture.py
@@ -0,0 +1,84 @@
+"""Build tests/tok_deepseek_tiny.json and tests/tok_deepseek_cases.txt.
+
+A byte-level BPE trained with `tokenizers` on a few multilingual sentences (no model
+download), carrying exactly the pre_tokenizer block of the DeepSeek V4 / V4.1
+tokenizer.json, and for each case the ids HF produces. tests/test_tok_deepseek.c holds
+tok.h to those ids. The merges only have to be rich enough that a different piece
+boundary changes the ids: that is what the test is about.
+
+ python3 tools/make_deepseek_tok_fixture.py
+"""
+import json
+from pathlib import Path
+
+from tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers, normalizers
+
+HERE = Path(__file__).resolve().parent.parent
+# Verbatim from DeepSeek-V4.1-Flash tokenizer.json (the same block as DeepSeek-V4).
+DEEPSEEK_PRE_TOKENIZER = json.loads(r'''{"type": "Sequence", "pretokenizers": [{"type": "Split", "pattern": {"Regex": "\\p{N}{1,3}"}, "behavior": "Isolated", "invert": false}, {"type": "Split", "pattern": {"Regex": "[\u4e00-\u9fa5\u3040-\u309f\u30a0-\u30ff]+"}, "behavior": "Isolated", "invert": false}, {"type": "Split", "pattern": {"Regex": "[!\"#$%&'()*+,\\-./:;<=>?@\\[\\\\\\]^_`{|}~][A-Za-z]+|[^\r\n\\p{L}\\p{P}\\p{S}]?[\\p{L}\\p{M}]+| ?[\\p{P}\\p{S}]+[\r\n]*|\\s*[\r\n]+|\\s+(?!\\S)|\\s+"}, "behavior": "Isolated", "invert": false}, {"type": "ByteLevel", "add_prefix_space": false, "trim_offsets": true, "use_regex": false}]}''')
+
+TRAIN = [
+ "Yapay zekâ kavramı ilk kez 1956'da Dartmouth Konferansı'nda John McCarthy tarafından ortaya atıldı.",
+ "Alan Turing'in 1950'deki makalesi, makinelerin düşünüp düşünemeyeceği sorusunu gündeme getirdi.",
+ "1997'de IBM'in Deep Blue'su Kasparov'u yendi; 2016'da AlphaGo, Lee Sedol'e karşı kazandı.",
+ "Türkiye'de, Ankara'dan, İstanbul'a, ABD'den, Atatürk'ün, Google'ın, 2000'li yıllarda değişiklikler oldu.",
+ "I'd say it's John's book, we're done, you'll see, I've been, I'm here, don't, can't, O'Reilly's.",
+ "L'intelligence artificielle d'aujourd'hui; dell'arte, all'estero, c'è un'altra idea.",
+ "Русский язык и Ελληνικά γράμματα, ещё раз и ещё раз.",
+ "بِسْمِ اللَّهِ الرَّحْمَٰنِ الرَّحِيمِ العربية שָׁלוֹם עוֹלָם עִבְרִית",
+ "नमस्ते दुनिया, क्षत्रिय हिन्दी भाषा। สวัสดีครับ ภาษาไทย",
+ "中文测试:你好,世界!日本語のテキスト・カタカナ「ラーメン」 한국어 테스트입니다.",
+ "def f(x):\n return x**2 + 1 # comment\n\tif (a && b) { return a->b[i]; }\r\n",
+ "https://example.com/path?q=1&r=2#frag [email protected] @handle #tag $HOME ~/.bashrc 3.14159 1,234,567",
+ "Emoji 👍🏽 👨‍👩‍👧 ❤️ ©®™ ½ ² Ⅻ ① ٣٤٥ 012 -42 1e-10 0x1F",
+]
+
+CASES = TRAIN + [
+ "1956'da", "Dartmouth'ta", "Türkiye'de", "Ankara'dan", "ABD'den", "IBM'in", "Turing'in",
+ "Atatürk'ün", "Google'ın", "2000'li", "Kasparov'u", "Lee Sedol'e", "McCarthy'nin",
+ "O'Reilly", "rock'n'roll", "I'd", "it's", "you'll", "we're", "I've", "I'm", "'quoted'",
+ "‘curly’ “double”", "d'accord", "l'IA", "c'è",
+ "بِسْمِ", "שָׁלוֹם", "क्षत्रिय", "สวัสดี", "é ä", "́alone", "x​zero‍width",
+ "中文测试", " 中文", "カタカナ・ラーメン", "テキスト 全角", "Fullwidth",
+ "123456789", "12345 67", "a1b22c333d4444", "½¾²³", "Ⅻ①", "٣٤٥٦", "3.14", "(1956)", "%40",
+ " leading", "trailing ", "a b", "a\tb", "a b", "line\nnext", "line\r\nnext",
+ "\n\n\nx", "x\n \n y", " \n", "end \r\n\r\n",
+ "!!!word", "'''x", "...", "—dash—", "«guillemets»", "¿Qué?", "¡Hola!",
+ "<tag attr=\"v\">", "a->b", "x**2", "\\path\\to", "`code`", "{\"k\": [1, 2]}",
+ "👍🏽", "👨‍👩‍👧", "🇹🇷", "❤️", "©®™",
+]
+
+
+def escape(text):
+ return (text.replace("\\", "\\\\").replace("\n", "\\n")
+ .replace("\t", "\\t").replace("\r", "\\r"))
+
+
+def main():
+ tok = Tokenizer(models.BPE())
+ tok.normalizer = normalizers.Sequence([])
+ # built from the verbatim block, so the fixture carries it byte for byte
+ spec = json.loads(tok.to_str())
+ spec["pre_tokenizer"] = DEEPSEEK_PRE_TOKENIZER
+ tok = Tokenizer.from_str(json.dumps(spec))
+ tok.decoder = decoders.ByteLevel()
+ trainer = trainers.BpeTrainer(vocab_size=900, min_frequency=1, show_progress=False,
+ initial_alphabet=pre_tokenizers.ByteLevel.alphabet(),
+ special_tokens=["<|begin▁of▁sentence|>", "<|User|>",
+ "<|Assistant|>"])
+ tok.train_from_iterator(TRAIN * 3, trainer=trainer)
+ saved = json.loads(tok.to_str())
+ assert saved["pre_tokenizer"] == DEEPSEEK_PRE_TOKENIZER, "pre_tokenizer block drifted"
+ (HERE / "tests" / "tok_deepseek_tiny.json").write_text(
+ json.dumps(saved, ensure_ascii=False, separators=(",", ":")), encoding="utf-8")
+ lines = []
+ for case in CASES:
+ ids = tok.encode(case, add_special_tokens=False).ids
+ assert tok.decode(ids, skip_special_tokens=False) == case, case
+ lines.append(f"{escape(case)}\t{','.join(map(str, ids))}")
+ (HERE / "tests" / "tok_deepseek_cases.txt").write_text("\n".join(lines) + "\n", encoding="utf-8")
+ print(f"{len(CASES)} cases, vocab {tok.get_vocab_size()}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
index 7af67ef0..7298dfa3 100644
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@ -544,4 +544,4 @@ COLI_TEMP=0 COLI_METAL=1 DIRECT=1 COLI_NO_OMP_TUNE=1 PIPE=1 PIPE_WORKERS=6 MTP=0
| `V41_DSPARK` | on when the checkpoint carries the head | DeepSeek V4.1: `0` disables the DSpark draft head, which is then not loaded. Drafts never change what a turn produces, only how many forwards it takes: measured +17% on the real checkpoint from a cold cache (24 tokens in 99.3 s against 116.6). |
| `V41_DSPARK_MAX` | the checkpoint's `dspark_block_size` | DeepSeek V4.1: how many drafted tokens go in front of the main model per round. Fewer costs less when a round is rejected and caps the win when it is not. |
| `V41_DSPARK_MINACC` | 60 | DeepSeek V4.1: percent of drafts that must be accepted over a window of ten before drafting pauses for 64 tokens. 60 is the measured break-even. |
-| `V41_SPEC_FORCE` | unset | DeepSeek V4.1, oracle mode only: draft the reference's own tokens (`1`), corrupt the last one (`2`), or keep the head's (`3`), so the verification path runs on a fixture whose draft head is random noise. |
+| `V41_SPEC_FORCE` | unset | DeepSeek V4.1, oracle mode only: draft the reference's own tokens (`1`), corrupt the last one (`2`), keep the head's (`3`), corrupt the first one (`4`) or a different one each round (`5`), so the verification path runs on a fixture whose draft head is random noise. |
diff --git a/docs/deepseek-v41.md b/docs/deepseek-v41.md
index b81d8d94..ca1c35ce 100644
--- a/docs/deepseek-v41.md
+++ b/docs/deepseek-v41.md
@@ -104,8 +104,10 @@ be clobbered by a draft several positions ahead:
Each row saves what it displaced, and a rejected row puts it back. `V41_SPEC_FORCE` in
the oracle drafts the reference's own next tokens (`1`), or corrupts the last of them
-so a round is rejected part way (`2`), or keeps the head's own (`3`); all three have to
-reproduce the reference token for token, and CI runs them.
+so a round is rejected part way (`2`), or keeps the head's own (`3`), or corrupts the
+first (`4`: every drafted row is rolled back, the common case at low acceptance) or a
+different one each round (`5`); all of them have to reproduce the reference token for
+token, and CI runs them.
Drafting is on, and the default is a measurement. Three runs of the same
24-token turn on a 16-thread CPU server holding 68% of the experts, each from a
@@ -358,7 +360,7 @@ and costs the contention. Anything that tries again has to start from that.
| `V41_DSPARK` | on when the checkpoint carries the head | `0` disables the draft head and does not load it. Measured at +17% on the real checkpoint, cold; see above. |
| `V41_DSPARK_MAX` | the checkpoint's `dspark_block_size` | how many of the drafted tokens are put in front of the main model. Fewer means a cheaper rejected round and a lower ceiling on the win. |
| `V41_DSPARK_MINACC` | 60 | percent of drafts that must be accepted over a window of ten for drafting to continue; below it, drafts pause for 64 tokens. 60 is the measured break-even, not a guess. |
-| `V41_SPEC_FORCE` | unset | oracle mode only: draft the reference's tokens (`1`), corrupt the last one (`2`), or use the head's own (`3`), to exercise the verification path on a fixture whose draft head is random. |
+| `V41_SPEC_FORCE` | unset | oracle mode only: draft the reference's tokens (`1`), corrupt the last one (`2`), use the head's own (`3`), corrupt the first one (`4`) or a different one each round (`5`), to exercise the verification path on a fixture whose draft head is random. |
## Reusing a turn, and why it is asked for rather than assumed