Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

qwen36: integer dot products for the dense trunk and the routed experts

JustVuggdfec3a4b6637
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a9847b86..58072d29 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -58,6 +58,21 @@ is now measured on the card in front of it instead of predicted.
output. A hand-written `COLI_PLACE` stands; `COLI_TRUNK_PROBE=0` trusts
the placer.
+### Performance
+
+- **#PR**: qwen36's dense trunk and routed experts multiply with integer
+ dot products. The activation is quantized to int8 once per call and the
+ weights, int8 rows or int4 planar blocks, meet it with maddubs / vpdpbusd
+ instead of a float conversion per weight; the integer kernels move from
+ `quant.h` into `idot.h`, shared by every engine. Measured on the 35B, 8
+ threads, every expert resident: decode 6.71 to 8.23 tok/s (+22.6%), lm_head
+ 12.6 to 10.1 ms/token, the expert compute 22.7 to 15.6, for +1.3%
+ perplexity on 4 x 512 tokens. Both are the default (`COLI_DENSE_IDOT=0`,
+ `QWEN_EXPERT_ACT=f32` restore the f32 kernels). `COLI_DENSE_BITS=4` with
+ `COLI_DENSE_INT4=<components>` stores part of the trunk as int4 in blocks
+ of 64: opt-in, with the perplexity it costs per component in the docs
+ (lm_head alone +2.4%, everything +10%).
+
### Performance, from contributors
- **#1606**: the K1b grouped int4 family gets a multi-row tile and AVX-512
diff --git a/c/Makefile b/c/Makefile
index 7db43a57..55a8bfeb 100644
--- a/c/Makefile
+++ b/c/Makefile
@@ -765,7 +765,7 @@ $(file >.build-config,$(BUILD_CONFIG))
endif
.build-config: ;
-colibri$(EXE): colibri.c pin_pool.h cli_args.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h quant.h sample.h kv_persist.h telemetry.h route_trace.h omp_tune.h kv_fp8.h kv_tq.h abl.h backend_cuda.h backend_metal.h backend_vulkan.h decode_batch.h edge_adapters.h edge_runtime.h edge_tok_internal.h schema_gbnf.h segment_adapter_internal.h segment_adapters.h segment_runtime.h tier.h $(CUDA_OBJ) $(METAL_OBJ) $(VK_OBJ) $(VK_SPV) .build-config
+colibri$(EXE): colibri.c pin_pool.h cli_args.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h omp_tune.h kv_fp8.h kv_tq.h abl.h backend_cuda.h backend_metal.h backend_vulkan.h decode_batch.h edge_adapters.h edge_runtime.h edge_tok_internal.h schema_gbnf.h segment_adapter_internal.h segment_adapters.h segment_runtime.h tier.h $(CUDA_OBJ) $(METAL_OBJ) $(VK_OBJ) $(VK_SPV) .build-config
$(CC) $(CFLAGS) colibri.c $(CUDA_OBJ) $(METAL_OBJ) $(VK_OBJ) -o colibri$(EXE) $(LDFLAGS)
# Vulkan backend object (plain C + vulkan headers) and its SPIR-V shaders.
@@ -1109,13 +1109,13 @@ QWEN36_TIER_SRC =
QWEN36_CFLAGS = $(NOCUDA_CFLAGS)
QWEN36_LDFLAGS = $(NOCUDA_LDFLAGS)
endif
-qwen36$(EXE): qwen36.c decode_batch.h serve_poll.h cli_args.h qwen36_tier.h expert_ffn.h st.h json.h compat.h omp_tune.h kv_prefix.h pin_pool.h edge_adapter_internal.h edge_adapters.h edge_runtime.h segment_adapter_internal.h segment_adapters.h segment_runtime.h gsgemv.h qgemv.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)
+qwen36$(EXE): qwen36.c decode_batch.h serve_poll.h cli_args.h qwen36_tier.h expert_ffn.h idot.h st.h json.h compat.h omp_tune.h kv_prefix.h pin_pool.h edge_adapter_internal.h edge_adapters.h edge_runtime.h segment_adapter_internal.h segment_adapters.h segment_runtime.h gsgemv.h qgemv.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)
$(CC) $(QWEN36_CFLAGS) qwen36.c $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o qwen36$(EXE) $(QWEN36_LDFLAGS)
# DeepSeek V4.1 Flash: one file, like every other portable engine. The fp4 experts
# stream from the official checkpoint and quant.h's mxfp4 kernel reads them as they
# are, so there is no conversion target here to go with it.
-deepseek_v41$(EXE): deepseek_v41.c cli_args.h st.h json.h tok.h tok_unicode.h compat.h omp_tune.h kv_prefix.h pin_pool.h quant.h \
+deepseek_v41$(EXE): deepseek_v41.c cli_args.h st.h json.h tok.h tok_unicode.h compat.h omp_tune.h kv_prefix.h pin_pool.h quant.h idot.h \
sparse_attn.h hyper_connections.h serve_codec.h serve_poll.h .build-config
$(CC) $(CFLAGS) deepseek_v41.c -o deepseek_v41$(EXE) $(LDFLAGS)
@@ -1123,7 +1123,7 @@ deepseek_v41$(EXE): deepseek_v41.c cli_args.h st.h json.h tok.h tok_unicode.h co
# loader and SERVE=1 protocol. With CUDA=1 it links the same expert tier as
# qwen36 (fp8 streaming mode: hot experts get VRAM copies, the RAM LRU stays);
# without it the tier header's inline stubs keep the build toolkit-free.
-qwen38$(EXE): qwen38.c pin_pool.h cli_args.h qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h omp_tune.h quant.h route_trace.h tok.h tok_unicode.h tok_unicode_o200k.h serve_codec.h edge_adapter_internal.h edge_adapters.h edge_runtime.h qwen38_vision.h segment_adapter_internal.h segment_adapters.h segment_runtime.h qwen36_tier.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)
+qwen38$(EXE): qwen38.c pin_pool.h cli_args.h qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h omp_tune.h quant.h idot.h route_trace.h tok.h tok_unicode.h tok_unicode_o200k.h serve_codec.h edge_adapter_internal.h edge_adapters.h edge_runtime.h qwen38_vision.h segment_adapter_internal.h segment_adapters.h segment_runtime.h qwen36_tier.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)
$(CC) $(QWEN36_CFLAGS) qwen38.c $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o qwen38$(EXE) $(QWEN36_LDFLAGS)
.PHONY: qwen38-tiny-generate qwen38-tiny-check
@@ -1200,6 +1200,10 @@ tests/test_qwen36_tok_truncated$(EXE): tests/test_qwen36_tok_truncated.c qwen36.
tests/test_qwen36_encode_oom$(EXE): tests/test_qwen36_encode_oom.c qwen36.c expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)
$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)
+# the dense trunk's integer path: quantizer contract, int4 planar packing, dispatch.
+tests/test_qwen36_dense_idot$(EXE): tests/test_qwen36_dense_idot.c qwen36.c expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)
+ $(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)
+
# #1653: added tokens are split out before the regex pre-tokenizer, HF-style.
tests/test_qwen36_tokenizer$(EXE): tests/test_qwen36_tokenizer.c qwen36.c expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)
$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)
@@ -1225,10 +1229,10 @@ NOCUDA_LDFLAGS = $(filter-out -lcudart -lstdc++ -lcuda -L$(CUDA_HOME)/lib64 \
# GLM-5.3-Flash: routed experts stream from the int4-gs64 container.
# METAL=1 accelerates resident matrices and routed MoE; CPU remains fallback.
-glm53$(EXE): glm53.c decode_batch.h pin_pool.h cli_args.h st.h json.h stop_ids.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h omp_tune.h serve_poll.h route_trace.h quant.h hyper_connections.h delta_attention.h sparse_index.h vision_tower.h backend_metal.h backend_vulkan.h edge_adapter_internal.h edge_adapters.h edge_runtime.h segment_adapter_internal.h segment_adapters.h segment_runtime.h $(METAL_OBJ) $(VK_OBJ) $(VK_SPV)
+glm53$(EXE): glm53.c decode_batch.h pin_pool.h cli_args.h st.h json.h stop_ids.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h omp_tune.h serve_poll.h route_trace.h quant.h idot.h hyper_connections.h delta_attention.h sparse_index.h vision_tower.h backend_metal.h backend_vulkan.h edge_adapter_internal.h edge_adapters.h edge_runtime.h segment_adapter_internal.h segment_adapters.h segment_runtime.h $(METAL_OBJ) $(VK_OBJ) $(VK_SPV)
$(CC) $(CFLAGS) glm53.c $(METAL_OBJ) $(VK_OBJ) -o glm53$(EXE) $(LDFLAGS)
-kimi_k3$(EXE): kimi_k3.c cli_args.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h omp_tune.h route_trace.h kv_prefix.h pin_pool.h serve_codec.h backend_cuda.h backend_metal.h backend_vulkan.h edge_adapters.h edge_runtime.h edge_tok_internal.h hybrid_split.h segment_adapter_internal.h segment_adapters.h segment_runtime.h $(CUDA_OBJ) $(VK_OBJ) $(VK_SPV) $(METAL_OBJ)
+kimi_k3$(EXE): kimi_k3.c cli_args.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h idot.h omp_tune.h route_trace.h kv_prefix.h pin_pool.h serve_codec.h backend_cuda.h backend_metal.h backend_vulkan.h edge_adapters.h edge_runtime.h edge_tok_internal.h hybrid_split.h segment_adapter_internal.h segment_adapters.h segment_runtime.h $(CUDA_OBJ) $(VK_OBJ) $(VK_SPV) $(METAL_OBJ)
$(CC) $(CFLAGS) kimi_k3.c $(CUDA_OBJ) $(VK_OBJ) $(METAL_OBJ) -o kimi_k3$(EXE) $(LDFLAGS)
# Use a baseline that matches the compiler target. macOS already targets a
@@ -1256,7 +1260,7 @@ iobench$(EXE): iobench.c compat.h
tests/test_serve_sentinel$(EXE): tests/test_serve_sentinel.c compat.h serve_codec.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_cluster_protocol$(EXE): tests/test_cluster_protocol.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_cluster_protocol$(EXE): tests/test_cluster_protocol.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_ue8m0$(EXE): tests/test_ue8m0.c st.h json.h compat.h
@@ -1271,10 +1275,10 @@ tests/test_tok_o200k$(EXE): tests/test_tok_o200k.c tok.h tok_unicode.h tok_unico
tests/test_tok_gpt2$(EXE): tests/test_tok_gpt2.c tok.h tok_unicode.h tok_unicode_o200k.h json.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_k3_ram_budget$(EXE): tests/test_k3_ram_budget.c kimi_k3.c st.h tok.h quant.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h
+tests/test_k3_ram_budget$(EXE): tests/test_k3_ram_budget.c kimi_k3.c st.h tok.h quant.h idot.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_k3_mmap$(EXE): tests/test_k3_mmap.c kimi_k3.c st.h tok.h quant.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h
+tests/test_k3_mmap$(EXE): tests/test_k3_mmap.c kimi_k3.c st.h tok.h quant.h idot.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h
$(CC) $(NOCUDA_CFLAGS) $< $(VK_OBJ) -o $@ $(NOCUDA_LDFLAGS)
tests/test_tok_kimi_tiny$(EXE): tests/test_tok_kimi_tiny.c tok.h tok_unicode.h tok_unicode_o200k.h json.h
@@ -1285,13 +1289,13 @@ tests/test_st_pread$(EXE): tests/test_st_pread.c st.h json.h compat.h
tests/test_st_slice$(EXE): tests/test_st_slice.c st.h json.h compat.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_qwen38_tokenizer$(EXE): tests/test_qwen38_tokenizer.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h edge_runtime.c edge_runtime.h edge_adapters.h edge_adapter_internal.h st.h json.h compat.h quant.h route_trace.h tok_unicode.h tok_unicode_o200k.h
+tests/test_qwen38_tokenizer$(EXE): tests/test_qwen38_tokenizer.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h edge_runtime.c edge_runtime.h edge_adapters.h edge_adapter_internal.h st.h json.h compat.h quant.h idot.h route_trace.h tok_unicode.h tok_unicode_o200k.h
$(CC) $(NOCUDA_CFLAGS) $< edge_runtime.c -o $@ $(NOCUDA_LDFLAGS)
-tests/test_qwen38_config$(EXE): tests/test_qwen38_config.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h route_trace.h tok_unicode.h tok_unicode_o200k.h
+tests/test_qwen38_config$(EXE): tests/test_qwen38_config.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h idot.h route_trace.h tok_unicode.h tok_unicode_o200k.h
$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)
-tests/test_qwen38_serve_framing$(EXE): tests/test_qwen38_serve_framing.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h
+tests/test_qwen38_serve_framing$(EXE): tests/test_qwen38_serve_framing.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h idot.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h
$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)
tests/test_qwen38_vision$(EXE): tests/test_qwen38_vision.c qwen38_vision.h st.h json.h compat.h
@@ -1313,13 +1317,13 @@ qwen38-vision-serve-check: qwen38$(EXE)
$(PYTHON) tools/make_edge_tiny_tokenizer.py --vocab-size 64 ./qwen38_mm_tiny
$(PYTHON) tests/test_qwen38_vision_serve.py --binary ./qwen38$(EXE) --fixture ./qwen38_mm_tiny
-tests/test_qwen38_prefix$(EXE): tests/test_qwen38_prefix.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h
+tests/test_qwen38_prefix$(EXE): tests/test_qwen38_prefix.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h idot.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h
$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)
-tests/test_qwen38_metrics$(EXE): tests/test_qwen38_metrics.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h
+tests/test_qwen38_metrics$(EXE): tests/test_qwen38_metrics.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h idot.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h
$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)
-tests/test_qwen38_native_weights$(EXE): tests/test_qwen38_native_weights.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h segment_runtime.c segment_runtime.h segment_adapters.h segment_adapter_internal.h st.h json.h compat.h quant.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h
+tests/test_qwen38_native_weights$(EXE): tests/test_qwen38_native_weights.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h segment_runtime.c segment_runtime.h segment_adapters.h segment_adapter_internal.h st.h json.h compat.h quant.h idot.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h
$(CC) $(NOCUDA_CFLAGS) $< segment_runtime.c -o $@ $(NOCUDA_LDFLAGS)
tests/test_st_map$(EXE): tests/test_st_map.c st.h json.h compat.h
@@ -1365,12 +1369,12 @@ tests/test_grammar$(EXE): tests/test_grammar.c grammar.h
# equivalence under the shipping flags is covered by the differential dump
# (old vs new rope_interleave are byte-identical); this test guards the
# source-level formula equivalence and must not depend on contraction luck.
-tests/test_rope_invfreq$(EXE): tests/test_rope_invfreq.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_rope_invfreq$(EXE): tests/test_rope_invfreq.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) -ffp-contract=off $< -o $@ $(LDFLAGS)
# schema->GBNF compile cache (#7): grammar_reset must equal a fresh setup, and
# the GrDraft.src ownership must not leak/double-free. Includes colibri.c.
-tests/test_grammar_cache$(EXE): tests/test_grammar_cache.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_grammar_cache$(EXE): tests/test_grammar_cache.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# Standalone: drives a faithful miniature of moe()'s routing+accumulate and links
@@ -1386,7 +1390,7 @@ tests/test_degrade_zero$(EXE): tests/test_degrade_zero.c
tests/test_schema_gbnf$(EXE): tests/test_schema_gbnf.c schema_gbnf.h grammar.h json.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_spec_decode_state$(EXE): tests/test_spec_decode_state.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_spec_decode_state$(EXE): tests/test_spec_decode_state.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_decode_batch$(EXE): tests/test_decode_batch.c decode_batch.h
@@ -1418,13 +1422,13 @@ tests/bench_inkling_shared_batch$(EXE): tests/bench_inkling_shared_batch.c inkli
tests/test_inkling_cache_index$(EXE): tests/test_inkling_cache_index.c inkling.c st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h $(INK_CUDA_OBJ) $(METAL_OBJ)
$(CC) $(CFLAGS) $< $(INK_CUDA_OBJ) $(METAL_OBJ) -o $@ $(LDFLAGS)
-tests/test_kimi_serve_framing$(EXE): tests/test_kimi_serve_framing.c kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h omp_tune.h route_trace.h
+tests/test_kimi_serve_framing$(EXE): tests/test_kimi_serve_framing.c kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h idot.h omp_tune.h route_trace.h
$(CC) $(NOCUDA_CFLAGS) $< $(VK_OBJ) -o $@ $(NOCUDA_LDFLAGS)
-tests/test_kimi_cache_index$(EXE): tests/test_kimi_cache_index.c kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h omp_tune.h route_trace.h
+tests/test_kimi_cache_index$(EXE): tests/test_kimi_cache_index.c kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h idot.h omp_tune.h route_trace.h
$(CC) $(NOCUDA_CFLAGS) $< $(VK_OBJ) -o $@ $(NOCUDA_LDFLAGS)
-tests/test_k3_chat_tools$(EXE): tests/test_k3_chat_tools.c kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h omp_tune.h route_trace.h
+tests/test_k3_chat_tools$(EXE): tests/test_k3_chat_tools.c kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h idot.h omp_tune.h route_trace.h
$(CC) $(NOCUDA_CFLAGS) $< $(VK_OBJ) -o $@ $(NOCUDA_LDFLAGS)
# olmoe's matmul_q, not colibri's: compares the IDOT path against FP32
@@ -1442,33 +1446,33 @@ tests/test_olmoe_cache_index$(EXE): tests/test_olmoe_cache_index.c olmoe.c st.h
tests/test_qwen36_cache_index$(EXE): tests/test_qwen36_cache_index.c qwen36.c expert_ffn.h st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h qwen36_tier.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)
$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)
-tests/test_idot$(EXE): tests/test_idot.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_idot$(EXE): tests/test_idot.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_i4_grouped$(EXE): tests/test_i4_grouped.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_i4_grouped$(EXE): tests/test_i4_grouped.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_stops$(EXE): tests/test_stops.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_stops$(EXE): tests/test_stops.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_cfg_topk$(EXE): tests/test_cfg_topk.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_cfg_topk$(EXE): tests/test_cfg_topk.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_topp$(EXE): tests/test_topp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_topp$(EXE): tests/test_topp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_topp is a microbenchmark (old qsort vs new heap partial-select, #335), NOT a test
# gate -- intentionally absent from TEST_BINS. Build on demand: make tests/bench_topp
-tests/bench_topp$(EXE): tests/bench_topp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_topp$(EXE): tests/bench_topp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_sample_nan$(EXE): tests/test_sample_nan.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_sample_nan$(EXE): tests/test_sample_nan.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_temp_env$(EXE): tests/test_temp_env.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_temp_env$(EXE): tests/test_temp_env.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_kv_alloc$(EXE): tests/test_kv_alloc.c colibri.c st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h kv_fp8.h kv_tq.h
+tests/test_kv_alloc$(EXE): tests/test_kv_alloc.c colibri.c st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h kv_fp8.h kv_tq.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_kv_fp8$(EXE): tests/test_kv_fp8.c kv_fp8.h decode_batch.h
@@ -1477,15 +1481,15 @@ tests/test_kv_fp8$(EXE): tests/test_kv_fp8.c kv_fp8.h decode_batch.h
tests/test_kv_tq$(EXE): tests/test_kv_tq.c kv_tq.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_kv_disk$(EXE): tests/test_kv_disk.c colibri.c st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h kv_fp8.h kv_tq.h
+tests/test_kv_disk$(EXE): tests/test_kv_disk.c colibri.c st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h kv_fp8.h kv_tq.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# fmt=6 kernel oracle: needs the generated grid table, and its fixture comes from
# the reference codec (tools/make_e8_fixture.py) β€” regenerate if the layout moves.
-tests/test_e8_kernel$(EXE): tests/test_e8_kernel.c quant.h
+tests/test_e8_kernel$(EXE): tests/test_e8_kernel.c quant.h idot.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_e4m3_vector$(EXE): tests/test_e4m3_vector.c quant.h
+tests/test_e4m3_vector$(EXE): tests/test_e4m3_vector.c quant.h idot.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_stop_ids$(EXE): tests/test_stop_ids.c stop_ids.h json.h
@@ -1512,13 +1516,13 @@ fuzz-rans: tests/fuzz_rans.c rans.h
tests/fuzz_rans.c -o tests/fuzz_rans -lm
./tests/fuzz_rans
-tests/test_int3$(EXE): tests/test_int3.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_int3$(EXE): tests/test_int3.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_int3_load$(EXE): tests/test_int3_load.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_int3_load$(EXE): tests/test_int3_load.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_fp8_passthrough$(EXE): tests/test_fp8_passthrough.c quant.h
+tests/test_fp8_passthrough$(EXE): tests/test_fp8_passthrough.c quant.h idot.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# Host-only: backend_cuda.h's format predicate is plain C, so its truth table is
@@ -1527,10 +1531,10 @@ tests/test_fp8_passthrough$(EXE): tests/test_fp8_passthrough.c quant.h
tests/test_cuda_fmt_guard$(EXE): tests/test_cuda_fmt_guard.c backend_cuda.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_fp8_load$(EXE): tests/test_fp8_load.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_fp8_load$(EXE): tests/test_fp8_load.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_qt_addrow$(EXE): tests/test_qt_addrow.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_qt_addrow$(EXE): tests/test_qt_addrow.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_logit_nan$(EXE): tests/test_logit_nan.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
@@ -1548,7 +1552,7 @@ tests/test_compat_direct$(EXE): tests/test_compat_direct.c compat.h
tests/test_expert_store_ops$(EXE): tests/test_expert_store_ops.c expert_store.h tensor.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_native_quant$(EXE): tests/test_native_quant.c deepseek_v4.c native_quant.h tensor.h quant.h
+tests/test_native_quant$(EXE): tests/test_native_quant.c deepseek_v4.c native_quant.h tensor.h quant.h idot.h
$(CC) $(CFLAGS) -DCOLI_V4_UNIT_NATIVE_QUANT deepseek_v4.c $< -o $@ $(LDFLAGS)
tests/test_edge_runtime$(EXE): tests/test_edge_runtime.c edge_runtime.c edge_runtime.h
@@ -1785,7 +1789,7 @@ $(V4_OWN_DIR)/COLI_V4_UNIT_CONFIG.o: deepseek_v4.c deepseek_v4.h deepseek_v4_int
$(V4_OWN_DIR)/COLI_V4_UNIT_ST.o: deepseek_v4.c deepseek_v4.h deepseek_v4_internal.h st.h | $(V4_OWN_DIR)
$(CC) $(V4_OWN_CFLAGS) -DCOLI_V4_UNIT_ST -c deepseek_v4.c -o $@
-$(V4_OWN_DIR)/COLI_V4_UNIT_NATIVE_QUANT.o: deepseek_v4.c deepseek_v4.h deepseek_v4_internal.h quant.h | $(V4_OWN_DIR)
+$(V4_OWN_DIR)/COLI_V4_UNIT_NATIVE_QUANT.o: deepseek_v4.c deepseek_v4.h deepseek_v4_internal.h quant.h idot.h | $(V4_OWN_DIR)
$(CC) $(V4_OWN_CFLAGS) -DCOLI_V4_UNIT_NATIVE_QUANT -c deepseek_v4.c -o $@
# The engine (RUNTIME unit) opens its expert store through the pluggable
@@ -1887,7 +1891,7 @@ tests/test_qwen36_tier_int8_decode$(EXE): tests/test_qwen36_tier_int8_decode.c t
# The qwen38 engine through its own main() on the fake backend: tier start
# from the FP8 fixture, reduced CPU list, qt_note on recycled slots, oracle
# tokens unchanged. Needs the fixture (qwen38-tiny-fp8-generate).
-tests/test_qwen38_tier_engine$(EXE): tests/test_qwen38_tier_engine.c tests/qwen36_fake_cuda.h qwen38.c qwen38_core.h qwen36_tier.c qwen36_tier.h backend_cuda.h cli_args.h st.h json.h compat.h quant.h route_trace.h
+tests/test_qwen38_tier_engine$(EXE): tests/test_qwen38_tier_engine.c tests/qwen36_fake_cuda.h qwen38.c qwen38_core.h qwen36_tier.c qwen36_tier.h backend_cuda.h cli_args.h st.h json.h compat.h quant.h idot.h route_trace.h
$(CC) $(CFLAGS) -DCOLI_CUDA $< -o $@ $(LDFLAGS)
tests/test_serve_poll$(EXE): tests/test_serve_poll.c serve_poll.h
@@ -1906,15 +1910,15 @@ tests/test_mem_available$(EXE): tests/test_mem_available.c compat.h
tests/test_798_guards$(EXE): tests/test_798_guards.c st.h json.h compat.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_dsa_select$(EXE): tests/test_dsa_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_dsa_select$(EXE): tests/test_dsa_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_corpus_draft$(EXE): tests/test_corpus_draft.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_corpus_draft$(EXE): tests/test_corpus_draft.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_cap_precedence$(EXE): tests/test_cap_precedence.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_cap_precedence$(EXE): tests/test_cap_precedence.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_mirror_stripe_split$(EXE): tests/test_mirror_stripe_split.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_mirror_stripe_split$(EXE): tests/test_mirror_stripe_split.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_v4_hybrid_policy$(EXE): tests/test_v4_hybrid_policy.c deepseek_v4_hybrid.h hybrid_split.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
@@ -1925,29 +1929,29 @@ tests/test_k3_fill_budget$(EXE): tests/test_k3_fill_budget.c hybrid_split.h
tests/test_v4_bank_pair$(EXE): tests/test_v4_bank_pair.c deepseek_v4_bank_pair.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_ram_clamp$(EXE): tests/test_ram_clamp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_ram_clamp$(EXE): tests/test_ram_clamp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_cap_mixed_width$(EXE): tests/test_cap_mixed_width.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_cap_mixed_width$(EXE): tests/test_cap_mixed_width.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_eslot_inflight$(EXE): tests/test_eslot_inflight.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_eslot_inflight$(EXE): tests/test_eslot_inflight.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_glm_cache_index$(EXE): tests/test_glm_cache_index.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_glm_cache_index$(EXE): tests/test_glm_cache_index.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_ssd_probe$(EXE): tests/test_ssd_probe.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_ssd_probe$(EXE): tests/test_ssd_probe.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_dsa_select is a microbenchmark (old qsort vs new quickselect partial-select, #356),
# NOT a test gate -- intentionally absent from TEST_BINS. Build on demand: make tests/bench_dsa_select
-tests/bench_dsa_select$(EXE): tests/bench_dsa_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_dsa_select$(EXE): tests/bench_dsa_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_router_select is a microbenchmark (duplicate-prefix scan vs marked-score scan),
# not a test gate. Build on demand: make tests/bench_router_select.
-tests/bench_router_select$(EXE): tests/bench_router_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_router_select$(EXE): tests/bench_router_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_indexer_allocations: microbenchmark (DeepSeek V4 indexer malloc vs persistent arena scratch), NOT a test gate.
@@ -1957,34 +1961,34 @@ tests/bench_indexer_allocations$(EXE): tests/bench_indexer_allocations.c
# bench_idot: microbenchmark (single-acc vs independent-acc AVX-VNNI idot), NOT a test gate.
# Build on demand on an AVX-VNNI CPU: make tests/bench_idot ARCH=native
-tests/bench_idot$(EXE): tests/bench_idot.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_idot$(EXE): tests/bench_idot.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_i4p_gidot: microbenchmark (per-row vs multi-row/AMX K1b grouped planar IDOT), NOT a test gate.
# Build on demand: make tests/bench_i4p_gidot ARCH=native
-tests/bench_i4p_gidot$(EXE): tests/bench_i4p_gidot.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_i4p_gidot$(EXE): tests/bench_i4p_gidot.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_gemv_stream: microbenchmark (decode-regime GEMV bandwidth vs the read ceiling;
# frozen-baseline + deinterleaved-x candidate A/B), NOT a test gate.
# Build on demand: make tests/bench_gemv_stream ARCH=native
-tests/bench_gemv_stream$(EXE): tests/bench_gemv_stream.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_gemv_stream$(EXE): tests/bench_gemv_stream.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# bench_mla_simd: microbenchmark (scalar vs AVX2/NEON MLA-absorb reductions, #442),
# NOT a test gate. Build on demand: make tests/bench_mla_simd
-tests/bench_mla_simd$(EXE): tests/bench_mla_simd.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/bench_mla_simd$(EXE): tests/bench_mla_simd.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_uring$(EXE): tests/test_uring.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h
+tests/test_uring$(EXE): tests/test_uring.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_pipe_block$(EXE): tests/test_pipe_block.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_pilot_ring$(EXE): tests/test_pilot_ring.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_pilot_ring$(EXE): tests/test_pilot_ring.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_moe_gs_guard$(EXE): tests/test_moe_gs_guard.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_moe_gs_guard$(EXE): tests/test_moe_gs_guard.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
tests/test_omp_tune$(EXE): tests/test_omp_tune.c omp_tune.h compat.h
@@ -1993,7 +1997,7 @@ tests/test_omp_tune$(EXE): tests/test_omp_tune.c omp_tune.h compat.h
tests/test_compat_env$(EXE): tests/test_compat_env.c compat.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
-tests/test_kvb_notice$(EXE): tests/test_kvb_notice.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h
+tests/test_kvb_notice$(EXE): tests/test_kvb_notice.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
# Standalone: proves the group-scaled int8 GEMV keeps every row's float
@@ -2166,5 +2170,5 @@ bench: iobench$(EXE)
tests/test_kv_prefix$(EXE): tests/test_kv_prefix.c kv_prefix.h
$(CC) $(CFLAGS) tests/test_kv_prefix.c -o tests/test_kv_prefix$(EXE) $(LDFLAGS)
-tests/test_kimi_request_state$(EXE): tests/test_kimi_request_state.c kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h omp_tune.h route_trace.h
+tests/test_kimi_request_state$(EXE): tests/test_kimi_request_state.c kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h idot.h omp_tune.h route_trace.h
$(CC) $(NOCUDA_CFLAGS) tests/test_kimi_request_state.c $(VK_OBJ) -o tests/test_kimi_request_state$(EXE) $(NOCUDA_LDFLAGS)
diff --git a/c/idot.h b/c/idot.h
new file mode 100644
index 00000000..6738abeb
--- /dev/null
+++ b/c/idot.h
@@ -0,0 +1,978 @@
+/* idot.h -- integer dot-product kernels shared by every engine.
+ *
+ * The activation is quantized to int8 once (one scale per row, amax/127,
+ * qrow_i8) and the weights, int8 rows or int4 planar blocks with per-row or
+ * per-group scales, are multiplied with integer instructions: maddubs on
+ * AVX2, vpdpbusd on AVX-VNNI and AVX-512 VNNI, sdot/smmla on NEON, and the
+ * AMX tile kernel where the CPU has it. Exact int32 sums, scaled once per
+ * output, so every ISA path is bit-identical to the scalar reference.
+ *
+ * Lived inside quant.h; moved here so an engine that has its own dense
+ * kernels (qwen36.c uses qgemv.h/gsgemv.h, whose matmul_q would collide
+ * with quant.h's) can still take the integer path for its dense trunk.
+ * quant.h includes this file at the top, so its other consumers see the
+ * same declarations in the same place as before. */
+#ifndef COLIBRI_IDOT_H
+#define COLIBRI_IDOT_H
+#include <stdint.h>
+#include <stdlib.h>
+#include <string.h>
+#include <math.h>
+#include <stdio.h>
+/* ---- SIMD includes -------------------------------------------------------- */
+#ifdef __AVX2__
+#include <immintrin.h>
+static inline float hsum256(__m256 v){
+ __m128 lo=_mm256_castps256_ps128(v), hi=_mm256_extractf128_ps(v,1);
+ lo=_mm_add_ps(lo,hi); __m128 sh=_mm_movehl_ps(lo,lo); lo=_mm_add_ps(lo,sh);
+ sh=_mm_shuffle_ps(lo,lo,1); lo=_mm_add_ss(lo,sh); return _mm_cvtss_f32(lo);
+}
+static inline int hsum256_i32(__m256i v){
+ __m128i lo=_mm256_castsi256_si128(v), hi=_mm256_extracti128_si256(v,1);
+ lo=_mm_add_epi32(lo,hi); lo=_mm_hadd_epi32(lo,lo); lo=_mm_hadd_epi32(lo,lo);
+ return _mm_cvtsi128_si32(lo);
+}
+#endif
+#if defined(__AVXVNNI__) && defined(__AVX2__)
+static inline int hsum128_i32(__m128i v){
+ v=_mm_hadd_epi32(v,v); v=_mm_hadd_epi32(v,v); return _mm_cvtsi128_si32(v);
+}
+#endif
+#ifdef __ARM_NEON
+#include <arm_neon.h>
+#endif
+#ifdef __VSX__
+#include <altivec.h>
+#undef vector
+#undef pixel
+#undef bool
+#endif
+
+
+/* ---- IDOT: integer dot kernels (int8-quantized activations) --------------- */
+#if defined(__AVX512VNNI__) && defined(__AVX512BW__)
+#define IDOT_KERNEL "avx512-vnni"
+#elif defined(__AVXVNNI__) && defined(__AVX2__)
+#define IDOT_KERNEL "avx-vnni"
+#elif defined(__AVX2__)
+#define IDOT_KERNEL "avx2"
+#elif defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)
+#define IDOT_KERNEL "neon-i8mm"
+#elif defined(__ARM_NEON)
+#define IDOT_KERNEL "neon"
+#elif defined(__VSX__)
+#define IDOT_KERNEL "vsx"
+#else
+#define IDOT_KERNEL "scalar"
+#endif
+
+static inline float qrow_i8(const float *x, int8_t *q, int I){
+ float amax=0; for(int i=0;i<I;i++){ float a=fabsf(x[i]); if(a>amax)amax=a; }
+ float s=amax/127.f; if(s<1e-12f) s=1e-12f; float inv=1.f/s;
+ for(int i=0;i<I;i++) q[i]=(int8_t)lrintf(x[i]*inv);
+ return s;
+}
+
+/* dot int8*int8 */
+static inline int32_t dot_i8i8(const int8_t *w, const int8_t *x, int I){
+ int32_t sum=0; int i=0;
+#if defined(__AVX512VNNI__) && defined(__AVX512BW__)
+ __m512i acc=_mm512_setzero_si512();
+ for(;i+64<=I;i+=64){
+ __m512i wv=_mm512_loadu_si512((const void*)(w+i));
+ __m512i xv=_mm512_loadu_si512((const void*)(x+i));
+ __mmask64 neg=_mm512_movepi8_mask(wv);
+ __m512i xs=_mm512_mask_sub_epi8(xv,neg,_mm512_setzero_si512(),xv);
+ acc=_mm512_dpbusd_epi32(acc,_mm512_abs_epi8(wv),xs);
+ }
+ sum=_mm512_reduce_add_epi32(acc);
+#elif defined(__AVXVNNI__) && defined(__AVX2__)
+ /* 4 accumulatori indipendenti (64 byte/iter): un solo acc incatena i vpdpbusd
+ * (latenza-bound ~5c). Somme intere associative -> bit-identico. Stessa struttura
+ * dei 4 accumulatori del ramo NEON piu' sotto.
+ * EN: four independent accumulators break the serial vpdpbusd->acc chain; integer
+ * adds are associative, so the result is bit-identical (mirrors the NEON path). */
+ __m128i a0=_mm_setzero_si128(),a1=_mm_setzero_si128(),a2=_mm_setzero_si128(),a3=_mm_setzero_si128();
+ for(;i+64<=I;i+=64){
+ __m128i w0=_mm_loadu_si128((const __m128i*)(w+i)), x0=_mm_loadu_si128((const __m128i*)(x+i));
+ __m128i w1=_mm_loadu_si128((const __m128i*)(w+i+16)), x1=_mm_loadu_si128((const __m128i*)(x+i+16));
+ __m128i w2=_mm_loadu_si128((const __m128i*)(w+i+32)), x2=_mm_loadu_si128((const __m128i*)(x+i+32));
+ __m128i w3=_mm_loadu_si128((const __m128i*)(w+i+48)), x3=_mm_loadu_si128((const __m128i*)(x+i+48));
+ a0=_mm_dpbusd_epi32(a0,_mm_abs_epi8(w0),_mm_sign_epi8(x0,w0));
+ a1=_mm_dpbusd_epi32(a1,_mm_abs_epi8(w1),_mm_sign_epi8(x1,w1));
+ a2=_mm_dpbusd_epi32(a2,_mm_abs_epi8(w2),_mm_sign_epi8(x2,w2));
+ a3=_mm_dpbusd_epi32(a3,_mm_abs_epi8(w3),_mm_sign_epi8(x3,w3));
+ }
+ __m128i acc=_mm_add_epi32(_mm_add_epi32(a0,a1),_mm_add_epi32(a2,a3));
+ for(;i+16<=I;i+=16){
+ __m128i wv=_mm_loadu_si128((const __m128i*)(w+i));
+ __m128i xv=_mm_loadu_si128((const __m128i*)(x+i));
+ acc=_mm_dpbusd_epi32(acc,_mm_abs_epi8(wv),_mm_sign_epi8(xv,wv));
+ }
+ sum=hsum128_i32(acc);
+#elif defined(__AVX2__)
+ __m256i acc=_mm256_setzero_si256(); const __m256i ones=_mm256_set1_epi16(1);
+ for(;i+32<=I;i+=32){
+ __m256i wv=_mm256_loadu_si256((const __m256i*)(w+i));
+ __m256i xv=_mm256_loadu_si256((const __m256i*)(x+i));
+ __m256i p=_mm256_maddubs_epi16(_mm256_sign_epi8(wv,wv),_mm256_sign_epi8(xv,wv));
+ acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p,ones));
+ }
+ sum=hsum256_i32(acc);
+#elif defined(__ARM_NEON)
+#if defined(__ARM_FEATURE_DOTPROD)
+ int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0);
+ for(;i+64<=I;i+=64){
+ a0=vdotq_s32(a0,vld1q_s8(w+i), vld1q_s8(x+i));
+ a1=vdotq_s32(a1,vld1q_s8(w+i+16),vld1q_s8(x+i+16));
+ a2=vdotq_s32(a2,vld1q_s8(w+i+32),vld1q_s8(x+i+32));
+ a3=vdotq_s32(a3,vld1q_s8(w+i+48),vld1q_s8(x+i+48));
+ }
+ int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));
+ for(;i+16<=I;i+=16) acc=vdotq_s32(acc,vld1q_s8(w+i),vld1q_s8(x+i));
+ sum=vaddvq_s32(acc);
+#else
+ int32x4_t acc=vdupq_n_s32(0);
+ for(;i+16<=I;i+=16){
+ int8x16_t wv=vld1q_s8(w+i), xv=vld1q_s8(x+i);
+ int16x8_t p=vmull_s8(vget_low_s8(wv),vget_low_s8(xv));
+ p=vmlal_s8(p,vget_high_s8(wv),vget_high_s8(xv));
+ acc=vpadalq_s16(acc,p);
+ }
+ sum=vaddvq_s32(acc);
+#endif
+#elif defined(__VSX__)
+ __vector signed int acc=vec_splats(0);
+ const __vector signed char vz=vec_splats((signed char)0);
+ for(;i+16<=I;i+=16){
+ __vector signed char wv=vec_xl(0,(const signed char*)(w+i));
+ __vector signed char xv=vec_xl(0,(const signed char*)(x+i));
+ __vector __bool char neg=vec_cmplt(wv,vz);
+ __vector signed char xs=vec_sel(xv,vec_sub(vz,xv),neg);
+ __vector unsigned char wa=(__vector unsigned char)vec_sel(wv,vec_sub(vz,wv),neg);
+ acc=vec_msum(xs,wa,acc);
+ }
+ sum=vec_extract(acc,0)+vec_extract(acc,1)+vec_extract(acc,2)+vec_extract(acc,3);
+#endif
+ for(;i<I;i++) sum+=(int32_t)w[i]*x[i];
+ return sum;
+}
+
+/* dot int4(packed)*int8 */
+static inline int32_t dot_i4i8(const uint8_t *w4, const int8_t *x, int I){
+ int32_t sum=0; int i=0;
+#if defined(__AVX512VNNI__) && defined(__AVX512BW__)
+ const __m256i m4v=_mm256_set1_epi8(0x0F);
+ const __m512i b8v=_mm512_set1_epi8(8);
+ const __m512i xidx=_mm512_setr_epi64(0,1,4,5,2,3,6,7);
+ __m512i acc=_mm512_setzero_si512();
+ for(;i+64<=I;i+=64){
+ __m256i by=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));
+ __m256i lo=_mm256_and_si256(by,m4v), hi=_mm256_and_si256(_mm256_srli_epi16(by,4),m4v);
+ __m256i z0=_mm256_unpacklo_epi8(lo,hi), z1=_mm256_unpackhi_epi8(lo,hi);
+ __m512i wv=_mm512_sub_epi8(_mm512_inserti64x4(_mm512_castsi256_si512(z0),z1,1),b8v);
+ __m512i xv=_mm512_permutexvar_epi64(xidx,_mm512_loadu_si512((const void*)(x+i)));
+ __mmask64 neg=_mm512_movepi8_mask(wv);
+ __m512i xs=_mm512_mask_sub_epi8(xv,neg,_mm512_setzero_si512(),xv);
+ acc=_mm512_dpbusd_epi32(acc,_mm512_abs_epi8(wv),xs);
+ }
+ sum=_mm512_reduce_add_epi32(acc);
+#elif defined(__AVXVNNI__) && defined(__AVX2__)
+ /* 4 accumulatori indipendenti (64 elementi = 32 byte packed/iter): un solo acc
+ * incatena i vpdpbusd (latenza-bound ~5c). Somme intere associative -> bit-identico.
+ * Stessa struttura dei 4 accumulatori del ramo NEON piu' sotto.
+ * EN: four independent accumulators break the serial vpdpbusd->acc chain; integer
+ * adds are associative, so the result is bit-identical (mirrors the NEON path). */
+ const __m128i m4=_mm_set1_epi8(0x0F); const __m128i b8=_mm_set1_epi8(8);
+ __m128i a0=_mm_setzero_si128(),a1=_mm_setzero_si128(),a2=_mm_setzero_si128(),a3=_mm_setzero_si128();
+ for(;i+64<=I;i+=64){
+ __m128i by0=_mm_loadu_si128((const __m128i*)(w4+(i>>1))); /* elem i..i+31 */
+ __m128i by1=_mm_loadu_si128((const __m128i*)(w4+(i>>1)+16)); /* elem i+32..i+63 */
+ __m128i lo0=_mm_and_si128(by0,m4), hi0=_mm_and_si128(_mm_srli_epi16(by0,4),m4);
+ __m128i lo1=_mm_and_si128(by1,m4), hi1=_mm_and_si128(_mm_srli_epi16(by1,4),m4);
+ __m128i w0=_mm_sub_epi8(_mm_unpacklo_epi8(lo0,hi0),b8), w1=_mm_sub_epi8(_mm_unpackhi_epi8(lo0,hi0),b8);
+ __m128i w2=_mm_sub_epi8(_mm_unpacklo_epi8(lo1,hi1),b8), w3=_mm_sub_epi8(_mm_unpackhi_epi8(lo1,hi1),b8);
+ __m128i x0=_mm_loadu_si128((const __m128i*)(x+i)), x1=_mm_loadu_si128((const __m128i*)(x+i+16));
+ __m128i x2=_mm_loadu_si128((const __m128i*)(x+i+32)), x3=_mm_loadu_si128((const __m128i*)(x+i+48));
+ a0=_mm_dpbusd_epi32(a0,_mm_abs_epi8(w0),_mm_sign_epi8(x0,w0));
+ a1=_mm_dpbusd_epi32(a1,_mm_abs_epi8(w1),_mm_sign_epi8(x1,w1));
+ a2=_mm_dpbusd_epi32(a2,_mm_abs_epi8(w2),_mm_sign_epi8(x2,w2));
+ a3=_mm_dpbusd_epi32(a3,_mm_abs_epi8(w3),_mm_sign_epi8(x3,w3));
+ }
+ __m128i acc=_mm_add_epi32(_mm_add_epi32(a0,a1),_mm_add_epi32(a2,a3));
+ for(;i+32<=I;i+=32){ /* 32-nibble remainder: 2 dpbusd, same unpack */
+ __m128i by=_mm_loadu_si128((const __m128i*)(w4+(i>>1)));
+ __m128i lo=_mm_and_si128(by,m4), hi=_mm_and_si128(_mm_srli_epi16(by,4),m4);
+ __m128i w0=_mm_sub_epi8(_mm_unpacklo_epi8(lo,hi),b8), w1=_mm_sub_epi8(_mm_unpackhi_epi8(lo,hi),b8);
+ __m128i x0=_mm_loadu_si128((const __m128i*)(x+i));
+ __m128i x1=_mm_loadu_si128((const __m128i*)(x+i+16));
+ acc=_mm_dpbusd_epi32(acc,_mm_abs_epi8(w0),_mm_sign_epi8(x0,w0));
+ acc=_mm_dpbusd_epi32(acc,_mm_abs_epi8(w1),_mm_sign_epi8(x1,w1));
+ }
+ sum=hsum128_i32(acc);
+#elif defined(__AVX2__)
+ const __m128i m4=_mm_set1_epi8(0x0F); const __m256i b8=_mm256_set1_epi8(8);
+ const __m256i ones=_mm256_set1_epi16(1);
+ __m256i acc=_mm256_setzero_si256();
+ for(;i+32<=I;i+=32){
+ __m128i by=_mm_loadu_si128((const __m128i*)(w4+(i>>1)));
+ __m128i lo=_mm_and_si128(by,m4), hi=_mm_and_si128(_mm_srli_epi16(by,4),m4);
+ __m128i n0=_mm_unpacklo_epi8(lo,hi), n1=_mm_unpackhi_epi8(lo,hi);
+ __m256i wv=_mm256_sub_epi8(_mm256_set_m128i(n1,n0),b8);
+ __m256i xv=_mm256_loadu_si256((const __m256i*)(x+i));
+ __m256i p=_mm256_maddubs_epi16(_mm256_sign_epi8(wv,wv),_mm256_sign_epi8(xv,wv));
+ acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p,ones));
+ }
+ sum=hsum256_i32(acc);
+#elif defined(__ARM_NEON)
+ const uint8x16_t m4q=vdupq_n_u8(0x0F); const int8x16_t b8q=vdupq_n_s8(8);
+#if defined(__ARM_FEATURE_DOTPROD)
+ int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0);
+ for(;i+64<=I;i+=64){
+ uint8x16_t byA=vld1q_u8(w4+(i>>1)), byB=vld1q_u8(w4+(i>>1)+16);
+ uint8x16x2_t zA=vzipq_u8(vandq_u8(byA,m4q), vshrq_n_u8(byA,4));
+ uint8x16x2_t zB=vzipq_u8(vandq_u8(byB,m4q), vshrq_n_u8(byB,4));
+ a0=vdotq_s32(a0,vsubq_s8(vreinterpretq_s8_u8(zA.val[0]),b8q),vld1q_s8(x+i));
+ a1=vdotq_s32(a1,vsubq_s8(vreinterpretq_s8_u8(zA.val[1]),b8q),vld1q_s8(x+i+16));
+ a2=vdotq_s32(a2,vsubq_s8(vreinterpretq_s8_u8(zB.val[0]),b8q),vld1q_s8(x+i+32));
+ a3=vdotq_s32(a3,vsubq_s8(vreinterpretq_s8_u8(zB.val[1]),b8q),vld1q_s8(x+i+48));
+ }
+ int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));
+ for(;i+32<=I;i+=32){
+ uint8x16_t by=vld1q_u8(w4+(i>>1));
+ uint8x16x2_t z=vzipq_u8(vandq_u8(by,m4q), vshrq_n_u8(by,4));
+ acc=vdotq_s32(acc,vsubq_s8(vreinterpretq_s8_u8(z.val[0]),b8q),vld1q_s8(x+i));
+ acc=vdotq_s32(acc,vsubq_s8(vreinterpretq_s8_u8(z.val[1]),b8q),vld1q_s8(x+i+16));
+ }
+ sum=vaddvq_s32(acc);
+#else
+ int32x4_t acc=vdupq_n_s32(0);
+ for(;i+32<=I;i+=32){
+ uint8x16_t by=vld1q_u8(w4+(i>>1));
+ uint8x16x2_t z=vzipq_u8(vandq_u8(by,m4q), vshrq_n_u8(by,4));
+ int8x16_t w0=vsubq_s8(vreinterpretq_s8_u8(z.val[0]),b8q);
+ int8x16_t w1=vsubq_s8(vreinterpretq_s8_u8(z.val[1]),b8q);
+ int8x16_t x0=vld1q_s8(x+i), x1=vld1q_s8(x+i+16);
+ int16x8_t p=vmull_s8(vget_low_s8(w0),vget_low_s8(x0));
+ p=vmlal_s8(p,vget_high_s8(w0),vget_high_s8(x0));
+ acc=vpadalq_s16(acc,p);
+ p=vmull_s8(vget_low_s8(w1),vget_low_s8(x1));
+ p=vmlal_s8(p,vget_high_s8(w1),vget_high_s8(x1));
+ acc=vpadalq_s16(acc,p);
+ }
+ sum=vaddvq_s32(acc);
+#endif
+#elif defined(__VSX__)
+ const __vector unsigned char m4v=vec_splats((unsigned char)0x0F);
+ const __vector unsigned char sh4=vec_splats((unsigned char)4);
+ const __vector signed char b8v=vec_splats((signed char)8);
+ const __vector signed char vz=vec_splats((signed char)0);
+ __vector signed int acc=vec_splats(0);
+ for(;i+32<=I;i+=32){
+ __vector unsigned char by=vec_xl(0,w4+(i>>1));
+ __vector unsigned char lo=vec_and(by,m4v), hi=vec_sr(by,sh4);
+ __vector signed char w0=vec_sub((__vector signed char)vec_mergeh(lo,hi),b8v);
+ __vector signed char w1=vec_sub((__vector signed char)vec_mergel(lo,hi),b8v);
+ __vector signed char x0=vec_xl(0,(const signed char*)(x+i));
+ __vector signed char x1=vec_xl(0,(const signed char*)(x+i+16));
+ __vector __bool char n0=vec_cmplt(w0,vz), n1=vec_cmplt(w1,vz);
+ acc=vec_msum(vec_sel(x0,vec_sub(vz,x0),n0),
+ (__vector unsigned char)vec_sel(w0,vec_sub(vz,w0),n0),acc);
+ acc=vec_msum(vec_sel(x1,vec_sub(vz,x1),n1),
+ (__vector unsigned char)vec_sel(w1,vec_sub(vz,w1),n1),acc);
+ }
+ sum=vec_extract(acc,0)+vec_extract(acc,1)+vec_extract(acc,2)+vec_extract(acc,3);
+#endif
+ for(;i+1<I;i+=2){ uint8_t b=w4[i>>1]; sum+=((int)(b&0xF)-8)*x[i]+((int)(b>>4)-8)*x[i+1]; }
+ if(i<I){ uint8_t b=w4[i>>1]; sum+=((int)(b&0xF)-8)*x[i]; }
+ return sum;
+}
+
+/* ---- ARM i8mm SMMLA tiled kernels ---------------------------------------- */
+#if defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)
+static inline int32x4_t mm_tile16(int32x4_t acc, int8x16_t wo, int8x16_t wo1,
+ int8x16_t xs, int8x16_t xs1){
+ acc=vmmlaq_s32(acc, vcombine_s8(vget_low_s8(wo), vget_low_s8(wo1)),
+ vcombine_s8(vget_low_s8(xs), vget_low_s8(xs1)));
+ return vmmlaq_s32(acc, vcombine_s8(vget_high_s8(wo), vget_high_s8(wo1)),
+ vcombine_s8(vget_high_s8(xs), vget_high_s8(xs1)));
+}
+static void matmul_q_idot_mm(float *y, const int8_t *xq, const float *sx, const int8_t *q,
+ const float *scale, int S, int I, int O){
+ #pragma omp parallel for schedule(static)
+ for(int o=0;o<(O&~1);o+=2){
+ const int8_t *wo=q+(int64_t)o*I, *wo1=q+(int64_t)(o+1)*I;
+ float sc0=scale[o], sc1=scale[o+1];
+ for(int s=0;s<(S&~1);s+=2){
+ const int8_t *xs=xq+(int64_t)s*I, *xs1=xq+(int64_t)(s+1)*I;
+ int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0); int i=0;
+ for(;i+64<=I;i+=64){
+ a0=mm_tile16(a0,vld1q_s8(wo+i), vld1q_s8(wo1+i), vld1q_s8(xs+i), vld1q_s8(xs1+i));
+ a1=mm_tile16(a1,vld1q_s8(wo+i+16),vld1q_s8(wo1+i+16),vld1q_s8(xs+i+16),vld1q_s8(xs1+i+16));
+ a2=mm_tile16(a2,vld1q_s8(wo+i+32),vld1q_s8(wo1+i+32),vld1q_s8(xs+i+32),vld1q_s8(xs1+i+32));
+ a3=mm_tile16(a3,vld1q_s8(wo+i+48),vld1q_s8(wo1+i+48),vld1q_s8(xs+i+48),vld1q_s8(xs1+i+48));
+ }
+ for(;i+16<=I;i+=16)
+ a0=mm_tile16(a0,vld1q_s8(wo+i),vld1q_s8(wo1+i),vld1q_s8(xs+i),vld1q_s8(xs1+i));
+ int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));
+ int32_t d00=vgetq_lane_s32(acc,0), d01=vgetq_lane_s32(acc,1);
+ int32_t d10=vgetq_lane_s32(acc,2), d11=vgetq_lane_s32(acc,3);
+ for(;i<I;i++){ int a=wo[i],b=wo1[i],u=xs[i],v=xs1[i];
+ d00+=a*u; d01+=a*v; d10+=b*u; d11+=b*v; }
+ y[(int64_t)s*O+o] =(float)d00*sc0*sx[s];
+ y[(int64_t)s*O+(o+1)] =(float)d10*sc1*sx[s];
+ y[(int64_t)(s+1)*O+o] =(float)d01*sc0*sx[s+1];
+ y[(int64_t)(s+1)*O+(o+1)]=(float)d11*sc1*sx[s+1];
+ }
+ if(S&1){ int s=S-1; const int8_t *xs=xq+(int64_t)s*I;
+ y[(int64_t)s*O+o] =(float)dot_i8i8(wo, xs,I)*sc0*sx[s];
+ y[(int64_t)s*O+(o+1)]=(float)dot_i8i8(wo1,xs,I)*sc1*sx[s]; }
+ }
+ if(O&1){ int o=O-1; const int8_t *w=q+(int64_t)o*I; float sc=scale[o];
+ #pragma omp parallel for schedule(static)
+ for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i8i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }
+}
+static void matmul_i4_idot_mm(float *y, const int8_t *xq, const float *sx, const uint8_t *q4,
+ const float *scale, int S, int I, int O){
+ int rb=(I+1)/2;
+ #pragma omp parallel for schedule(static)
+ for(int o=0;o<(O&~1);o+=2){
+ const uint8x16_t m4q=vdupq_n_u8(0x0F); const int8x16_t b8q=vdupq_n_s8(8);
+ const uint8_t *wo=q4+(int64_t)o*rb, *wo1=q4+(int64_t)(o+1)*rb;
+ float sc0=scale[o], sc1=scale[o+1];
+ for(int s=0;s<(S&~1);s+=2){
+ const int8_t *xs=xq+(int64_t)s*I, *xs1=xq+(int64_t)(s+1)*I;
+ int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0); int i=0;
+ for(;i+64<=I;i+=64){
+ uint8x16_t byo=vld1q_u8(wo+(i>>1)), byo1=vld1q_u8(wo1+(i>>1));
+ uint8x16_t cyo=vld1q_u8(wo+(i>>1)+16), cyo1=vld1q_u8(wo1+(i>>1)+16);
+ uint8x16x2_t zo =vzipq_u8(vandq_u8(byo, m4q), vshrq_n_u8(byo, 4));
+ uint8x16x2_t zo1=vzipq_u8(vandq_u8(byo1,m4q), vshrq_n_u8(byo1,4));
+ uint8x16x2_t ko =vzipq_u8(vandq_u8(cyo, m4q), vshrq_n_u8(cyo, 4));
+ uint8x16x2_t ko1=vzipq_u8(vandq_u8(cyo1,m4q), vshrq_n_u8(cyo1,4));
+ a0=mm_tile16(a0, vsubq_s8(vreinterpretq_s8_u8(zo.val[0]),b8q),
+ vsubq_s8(vreinterpretq_s8_u8(zo1.val[0]),b8q),
+ vld1q_s8(xs+i), vld1q_s8(xs1+i));
+ a1=mm_tile16(a1, vsubq_s8(vreinterpretq_s8_u8(zo.val[1]),b8q),
+ vsubq_s8(vreinterpretq_s8_u8(zo1.val[1]),b8q),
+ vld1q_s8(xs+i+16), vld1q_s8(xs1+i+16));
+ a2=mm_tile16(a2, vsubq_s8(vreinterpretq_s8_u8(ko.val[0]),b8q),
+ vsubq_s8(vreinterpretq_s8_u8(ko1.val[0]),b8q),
+ vld1q_s8(xs+i+32), vld1q_s8(xs1+i+32));
+ a3=mm_tile16(a3, vsubq_s8(vreinterpretq_s8_u8(ko.val[1]),b8q),
+ vsubq_s8(vreinterpretq_s8_u8(ko1.val[1]),b8q),
+ vld1q_s8(xs+i+48), vld1q_s8(xs1+i+48));
+ }
+ for(;i+32<=I;i+=32){
+ uint8x16_t byo=vld1q_u8(wo+(i>>1)), byo1=vld1q_u8(wo1+(i>>1));
+ uint8x16x2_t zo =vzipq_u8(vandq_u8(byo, m4q), vshrq_n_u8(byo, 4));
+ uint8x16x2_t zo1=vzipq_u8(vandq_u8(byo1,m4q), vshrq_n_u8(byo1,4));
+ a0=mm_tile16(a0, vsubq_s8(vreinterpretq_s8_u8(zo.val[0]),b8q),
+ vsubq_s8(vreinterpretq_s8_u8(zo1.val[0]),b8q),
+ vld1q_s8(xs+i), vld1q_s8(xs1+i));
+ a1=mm_tile16(a1, vsubq_s8(vreinterpretq_s8_u8(zo.val[1]),b8q),
+ vsubq_s8(vreinterpretq_s8_u8(zo1.val[1]),b8q),
+ vld1q_s8(xs+i+16), vld1q_s8(xs1+i+16));
+ }
+ int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));
+ int32_t d00=vgetq_lane_s32(acc,0), d01=vgetq_lane_s32(acc,1);
+ int32_t d10=vgetq_lane_s32(acc,2), d11=vgetq_lane_s32(acc,3);
+ for(;i+1<I;i+=2){ uint8_t bo=wo[i>>1], bo1=wo1[i>>1];
+ int a0=(int)(bo&0xF)-8, a1=(int)(bo>>4)-8, b0=(int)(bo1&0xF)-8, b1=(int)(bo1>>4)-8;
+ int u0=xs[i],u1=xs[i+1],v0=xs1[i],v1=xs1[i+1];
+ d00+=a0*u0+a1*u1; d01+=a0*v0+a1*v1; d10+=b0*u0+b1*u1; d11+=b0*v0+b1*v1; }
+ if(i<I){ uint8_t bo=wo[i>>1], bo1=wo1[i>>1];
+ int a0=(int)(bo&0xF)-8, b0=(int)(bo1&0xF)-8;
+ d00+=a0*xs[i]; d01+=a0*xs1[i]; d10+=b0*xs[i]; d11+=b0*xs1[i]; }
+ y[(int64_t)s*O+o] =(float)d00*sc0*sx[s];
+ y[(int64_t)s*O+(o+1)] =(float)d10*sc1*sx[s];
+ y[(int64_t)(s+1)*O+o] =(float)d01*sc0*sx[s+1];
+ y[(int64_t)(s+1)*O+(o+1)]=(float)d11*sc1*sx[s+1];
+ }
+ if(S&1){ int s=S-1; const int8_t *xs=xq+(int64_t)s*I;
+ y[(int64_t)s*O+o] =(float)dot_i4i8(wo, xs,I)*sc0*sx[s];
+ y[(int64_t)s*O+(o+1)]=(float)dot_i4i8(wo1,xs,I)*sc1*sx[s]; }
+ }
+ if(O&1){ int o=O-1; const uint8_t *w=q4+(int64_t)o*rb; float sc=scale[o];
+ #pragma omp parallel for schedule(static)
+ for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i4i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }
+}
+#endif
+
+/* ---- IDOT dispatch (int8-quantized activations) --------------------------- */
+static void matmul_q_idot(float *y, const int8_t *xq, const float *sx, const int8_t *q,
+ const float *scale, int S, int I, int O){
+#if defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)
+ if(S>=2){ matmul_q_idot_mm(y,xq,sx,q,scale,S,I,O); return; }
+#endif
+ #pragma omp parallel for schedule(static)
+ for(int o=0;o<O;o++){ const int8_t *w=q+(int64_t)o*I; float sc=scale[o];
+ for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i8i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }
+}
+static void matmul_i4_idot(float *y, const int8_t *xq, const float *sx, const uint8_t *q4,
+ const float *scale, int S, int I, int O){
+ int rb=(I+1)/2;
+#if defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)
+ if(S>=2){ matmul_i4_idot_mm(y,xq,sx,q4,scale,S,I,O); return; }
+#endif
+ #pragma omp parallel for schedule(static)
+ for(int o=0;o<O;o++){ const uint8_t *w=q4+(int64_t)o*rb; float sc=scale[o];
+ for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i4i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }
+}
+
+
+/* ================= K1: layout int4 A PIANI (fmt 2/4, opzionale) ==============
+ * Layout classico ("a coppie"): byte k = elementi (2k, 2k+1). Ogni kernel paga
+ * unpacklo/unpackhi per riordinare i nibble, e l'IDOT paga sub+abs+sign per
+ * forzare l'operando con segno dentro vpdpbusd (che e' nativamente u8 x s8).
+ *
+ * Layout A PIANI, per blocco di 64 elementi consecutivi (32 byte):
+ * byte k del blocco = (elem k + 8) | ((elem k+32 + 8) << 4) k = 0..31
+ * `and 0x0F` produce gli elementi 0..31 GIA' in ordine; `srli 4 + and` produce
+ * 32..63 in ordine: l'unpack sparisce. La coda (I mod 64) resta a coppie.
+ *
+ * L'algebra dell'IDOT: i nibble memorizzati sono u = v+8 (unsigned 0..15), e
+ * dot(v, x) = dot(u, x) - 8 * sum(x)
+ * dove sum(x) e' UN int32 per riga di attivazione, condiviso da ogni riga di
+ * output. vpdpbusd consuma (u8, s8) direttamente: via sub+abs+sign (3 op
+ * vettoriali per dpbusd). Misurato su questa famiglia: 1.87x L3-resident,
+ * 25.3 GB/s DRAM (al tetto), 0 differenze di bit.
+ * NB: allargare a 256 bit SENZA cambiare layout e' stato misurato 0.79-0.81x
+ * (il fixup di lane costa piu' della larghezza): il layout e' la precondizione
+ * della larghezza, non un'ottimizzazione indipendente.
+ *
+ * I kernel f32 planari preservano ESATTAMENTE l'ordine di accumulazione dei
+ * gemelli a coppie (stesse lane, stessa sequenza 0..63): bit-identici anche
+ * loro. EN: plane-nibble int4 layout. Low nibble of block byte k = element k,
+ * high = element k+32; the unpack disappears, and the stored-unsigned nibbles
+ * feed vpdpbusd natively via dot(v,x) = dot(u,x) - 8*sum(x). The f32 planar
+ * kernels keep the pair kernels' exact accumulation order: bit-identical. */
+
+/* riordina in place una riga (o un tensore [O,rb] riga per riga) da coppie a
+ * piani; la coda I%64 resta a coppie. EN: in-place pair->planar repack. */
+static void planarize_i4_row(uint8_t *row, int I){
+ uint8_t tmp[32];
+ int nb=I/64;
+ for(int b=0;b<nb;b++){
+ uint8_t *blk=row+b*32;
+ /* pair: byte j ha (elem 2j, elem 2j+1). planar: byte k = (elem k, elem k+32) */
+ for(int k=0;k<32;k++){
+ int src_lo=k, src_hi=k+32; /* elementi voluti */
+ uint8_t nib_lo=(blk[src_lo>>1]>>((src_lo&1)*4))&0xF;
+ uint8_t nib_hi=(blk[src_hi>>1]>>((src_hi&1)*4))&0xF;
+ tmp[k]=(uint8_t)(nib_lo|(nib_hi<<4));
+ }
+ memcpy(blk,tmp,32);
+ }
+}
+static void planarize_i4(uint8_t *q4, int O, int I){
+ int rb=(I+1)/2;
+ #pragma omp parallel for schedule(static)
+ for(int o=0;o<O;o++) planarize_i4_row(q4+(int64_t)o*rb, I);
+}
+
+/* dot unsigned-nibble planare * int8: ritorna dot(u, x) SENZA la correzione
+ * -8*sum(x) (la applica il chiamante, una volta per riga di output).
+ * Coda I%64: nibble a coppie letti come unsigned, stessa identita'. */
+#if defined(__AVXVNNI__) || (defined(__AVX512VNNI__) && defined(__AVX512VL__))
+#if defined(__AVXVNNI__) && !defined(__AVX512VNNI__)
+#define coli_dpbusd256 _mm256_dpbusd_avx_epi32
+#else
+#define coli_dpbusd256 _mm256_dpbusd_epi32
+#endif
+#endif
+static inline int32_t dot_i4p_u(const uint8_t *w4, const int8_t *x, int I){
+ int32_t sum=0; int i=0;
+#if defined(coli_dpbusd256)
+ const __m256i m4=_mm256_set1_epi8(0x0F);
+ __m256i a0=_mm256_setzero_si256(),a1=_mm256_setzero_si256();
+ __m256i a2=_mm256_setzero_si256(),a3=_mm256_setzero_si256();
+ for(;i+128<=I;i+=128){
+ __m256i b0=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));
+ __m256i b1=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)+32));
+ a0=coli_dpbusd256(a0,_mm256_and_si256(b0,m4),
+ _mm256_loadu_si256((const __m256i*)(x+i)));
+ a1=coli_dpbusd256(a1,_mm256_and_si256(_mm256_srli_epi16(b0,4),m4),
+ _mm256_loadu_si256((const __m256i*)(x+i+32)));
+ a2=coli_dpbusd256(a2,_mm256_and_si256(b1,m4),
+ _mm256_loadu_si256((const __m256i*)(x+i+64)));
+ a3=coli_dpbusd256(a3,_mm256_and_si256(_mm256_srli_epi16(b1,4),m4),
+ _mm256_loadu_si256((const __m256i*)(x+i+96)));
+ }
+ __m256i acc=_mm256_add_epi32(_mm256_add_epi32(a0,a1),_mm256_add_epi32(a2,a3));
+ for(;i+64<=I;i+=64){
+ __m256i b0=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));
+ acc=coli_dpbusd256(acc,_mm256_and_si256(b0,m4),
+ _mm256_loadu_si256((const __m256i*)(x+i)));
+ acc=coli_dpbusd256(acc,_mm256_and_si256(_mm256_srli_epi16(b0,4),m4),
+ _mm256_loadu_si256((const __m256i*)(x+i+32)));
+ }
+ sum=hsum256_i32(acc);
+#elif defined(__AVX2__)
+ const __m256i m4=_mm256_set1_epi8(0x0F);
+ const __m256i ones=_mm256_set1_epi16(1);
+ __m256i acc=_mm256_setzero_si256();
+ for(;i+64<=I;i+=64){
+ __m256i b0=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));
+ /* maddubs(u8, s8): u<=15, |x|<=127 -> coppia <= 3810, int16 sicuro */
+ __m256i p0=_mm256_maddubs_epi16(_mm256_and_si256(b0,m4),
+ _mm256_loadu_si256((const __m256i*)(x+i)));
+ __m256i p1=_mm256_maddubs_epi16(_mm256_and_si256(_mm256_srli_epi16(b0,4),m4),
+ _mm256_loadu_si256((const __m256i*)(x+i+32)));
+ acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p0,ones));
+ acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p1,ones));
+ }
+ sum=hsum256_i32(acc);
+#elif defined(__ARM_NEON)
+ int32x4_t acc=vdupq_n_s32(0);
+ for(;i+64<=I;i+=64){
+ uint8x16_t b0=vld1q_u8(w4+(i>>1)), b1=vld1q_u8(w4+(i>>1)+16);
+ int8x16_t lo0=vreinterpretq_s8_u8(vandq_u8(b0,vdupq_n_u8(0x0F)));
+ int8x16_t lo1=vreinterpretq_s8_u8(vandq_u8(b1,vdupq_n_u8(0x0F)));
+ int8x16_t hi0=vreinterpretq_s8_u8(vshrq_n_u8(b0,4));
+ int8x16_t hi1=vreinterpretq_s8_u8(vshrq_n_u8(b1,4));
+#if defined(__ARM_FEATURE_DOTPROD)
+ acc=vdotq_s32(acc,lo0,vld1q_s8(x+i));
+ acc=vdotq_s32(acc,lo1,vld1q_s8(x+i+16));
+ acc=vdotq_s32(acc,hi0,vld1q_s8(x+i+32));
+ acc=vdotq_s32(acc,hi1,vld1q_s8(x+i+48));
+#else
+ int8x16_t xs0=vld1q_s8(x+i), xs1=vld1q_s8(x+i+16);
+ int8x16_t xs2=vld1q_s8(x+i+32), xs3=vld1q_s8(x+i+48);
+ int16x8_t m;
+ m=vmull_s8(vget_low_s8(lo0),vget_low_s8(xs0)); acc=vpadalq_s16(acc,m);
+ m=vmull_s8(vget_high_s8(lo0),vget_high_s8(xs0)); acc=vpadalq_s16(acc,m);
+ m=vmull_s8(vget_low_s8(lo1),vget_low_s8(xs1)); acc=vpadalq_s16(acc,m);
+ m=vmull_s8(vget_high_s8(lo1),vget_high_s8(xs1)); acc=vpadalq_s16(acc,m);
+ m=vmull_s8(vget_low_s8(hi0),vget_low_s8(xs2)); acc=vpadalq_s16(acc,m);
+ m=vmull_s8(vget_high_s8(hi0),vget_high_s8(xs2)); acc=vpadalq_s16(acc,m);
+ m=vmull_s8(vget_low_s8(hi1),vget_low_s8(xs3)); acc=vpadalq_s16(acc,m);
+ m=vmull_s8(vget_high_s8(hi1),vget_high_s8(xs3)); acc=vpadalq_s16(acc,m);
+#endif
+ }
+ sum=vaddvq_s32(acc);
+#endif
+ for(;i+64<=I;i+=64){ /* fallback scalare sui blocchi planari */
+ const uint8_t *blk=w4+(i>>1);
+ for(int k=0;k<32;k++){
+ sum+=(int32_t)(blk[k]&0xF)*x[i+k];
+ sum+=(int32_t)(blk[k]>>4)*x[i+k+32];
+ }
+ }
+ for(;i<I;i+=2){ /* coda a coppie, unsigned (u=v+8 memorizzato) */
+ uint8_t byte=w4[i>>1];
+ sum+=(int32_t)(byte&0xF)*x[i];
+ if(i+1<I) sum+=(int32_t)(byte>>4)*x[i+1];
+ }
+ return sum;
+}
+
+/* ---- K1b (OPT-IN, IDOT_GS=1): IDOT planare A GRUPPI (fmt=4, gs%64==0) -----
+ * Con gs=64 il gruppo di scala COINCIDE col blocco-piano da 64 elementi: il
+ * dot unsigned del blocco (2 dpbusd) -> int32 di gruppo, meno 8*somma(x) del
+ * gruppo, per la scala f32 del gruppo. Attivazioni int8 (stessa famiglia
+ * qrow_i8 del resto dell'IDOT): NON bit-identico al kernel f32 a gruppi --
+ * per questo e' dietro flag, in attesa dell'ablazione. xsg = somme int32
+ * per (riga, gruppo), calcolate dal chiamante in una passata esatta.
+ * EN: grouped planar IDOT, opt-in. With gs=64 the scale group IS the plane
+ * block; per-group unsigned dot minus 8*group-sum, times the group scale.
+ * int8 activations: not bit-identical to the f32 grouped kernel, hence the
+ * flag until the ablation blesses a default.
+ *
+ * Three execution shapes below, one contract: per (row, output) the float
+ * accumulation is the SAME sequence of fmaf((float)group_int, scale[g], a)
+ * in ascending g, and every group_int is an exact int32 β€” so the per-row
+ * path, the 1x4 row tile, and the AMX tile are bit-identical to each other
+ * and to the pure-C reference on every ISA. */
+
+/* one planar 64-element block, unpacked once and shared across a row tile:
+ * lo nibbles = elements base..base+31 in order, hi = base+32..base+63. */
+#if defined(coli_dpbusd256)
+static inline void i4p_blk256(const uint8_t *blk, __m256i *lo, __m256i *hi){
+ const __m256i m4=_mm256_set1_epi8(0x0F);
+ __m256i b=_mm256_loadu_si256((const __m256i*)blk);
+ *lo=_mm256_and_si256(b,m4);
+ *hi=_mm256_and_si256(_mm256_srli_epi16(b,4),m4);
+}
+#endif
+#if defined(__AVX512VNNI__) && defined(__AVX512BW__)
+/* whole block in one zmm: lanes 0..31 = lo elements, 32..63 = hi β€” matches a
+ * straight 64-byte load of the activation block, so ONE dpbusd per block. */
+static inline __m512i i4p_blk512(const uint8_t *blk){
+ const __m256i m4=_mm256_set1_epi8(0x0F);
+ __m256i b=_mm256_loadu_si256((const __m256i*)blk);
+ return _mm512_inserti64x4(_mm512_castsi256_si512(_mm256_and_si256(b,m4)),
+ _mm256_and_si256(_mm256_srli_epi16(b,4),m4),1);
+}
+#endif
+
+/* ---- K1c: AMX int8 tile kernel (Sapphire Rapids+, opt-in via the same
+ * IDOT_GS=1 family gate; AMX=0 kills it, AMX_S_MIN sets the row threshold).
+ * With gs a multiple of 64, K=64 tile-multiplies cover a scale group exactly:
+ * B tile = 16 output rows' int4 block unpacked once to SIGNED int8 (v-8, so
+ * tdpbssd returns d - 8*sum(x_group) directly β€” the same int32 the vector
+ * path computes as d_unsigned - 8*xsg), A tile = up to 16 activation rows.
+ * The unpack cost is paid once per (output tile, group) and amortized over
+ * every activation row β€” the multi-row reuse the pair-layout kernels lack.
+ * Linux-only arming: tile data needs an ARCH_REQ_XCOMP_PERM handshake. */
+#if defined(__AMX_INT8__) && defined(__AMX_TILE__) && defined(__AVX512F__)
+#define COLI_HAVE_AMX_I4P 1
+#if defined(__linux__)
+#include <unistd.h>
+#include <sys/syscall.h>
+#elif defined(_WIN32)
+#ifndef WIN32_LEAN_AND_MEAN
+#define WIN32_LEAN_AND_MEAN
+#endif
+#include <windows.h>
+#endif
+static int coli_amx_state=-1;
+static int coli_amx_s_min=8;
+static int coli_amx_ok(void){
+ if(coli_amx_state<0){
+ const char *e=getenv("AMX");
+ const char *sm=getenv("AMX_S_MIN"); if(sm&&atoi(sm)>0) coli_amx_s_min=atoi(sm);
+#if defined(__linux__)
+ /* ARCH_REQ_XCOMP_PERM(XFEATURE_XTILEDATA): kernel >= 5.16 grants tile
+ * state per-process; without it the first tile op SIGILLs. */
+ coli_amx_state=(e&&*e=='0')?0:(syscall(SYS_arch_prctl,0x1023,18)==0);
+#elif defined(_WIN32)
+ /* Windows 11: tile data is an opt-in per-process XState feature.
+ * Dynamic lookup keeps older kernels building/running (arming just
+ * fails closed there). XSTATE_AMX_TILE_DATA = 18. */
+ if(e&&*e=='0') coli_amx_state=0;
+ else {
+ HMODULE k32=GetModuleHandleA("kernel32.dll");
+ typedef BOOL (WINAPI *coli_xstate_fn)(ULONG64);
+ coli_xstate_fn f=k32?(coli_xstate_fn)(void*)GetProcAddress(k32,"EnableProcessOptionalXStateFeatures"):NULL;
+ coli_amx_state=(f && f(1ull<<18))?1:0;
+ }
+#else
+ (void)e; coli_amx_state=0;
+#endif
+ if(coli_amx_state)
+ fprintf(stderr,"[K1c] AMX int8 tile kernel armed for gs64 tensors "
+ "(AMX=0 disables, engages at S>=%d)\n",coli_amx_s_min);
+ }
+ return coli_amx_state;
+}
+/* ldtilecfg layout (palette 1): tmm0=C [rows x 16 i32], tmm1=A [rows x 64 i8],
+ * tmm2=B [16 x 64 i8 VNNI]. rows<16 reconfigures for the last partial s-tile. */
+struct coli_tilecfg { uint8_t palette,start_row,rsvd[14]; uint16_t colsb[16]; uint8_t rows[16]; };
+static void coli_amx_cfg(int arows){
+ struct coli_tilecfg c; memset(&c,0,sizeof c); c.palette=1;
+ c.rows[0]=(uint8_t)arows; c.colsb[0]=64;
+ c.rows[1]=(uint8_t)arows; c.colsb[1]=64;
+ c.rows[2]=16; c.colsb[2]=64;
+ _tile_loadconfig(&c);
+}
+static void matmul_i4p_gidot_amx(float *y, const int8_t *xq, const float *sx,
+ const uint8_t *q4, const float *scale,
+ int S, int I, int O16, int O, int gs){
+ int rb=(I+1)/2, ng=(I+gs-1)/gs, bpg=gs/64;
+ #pragma omp parallel
+ {
+ float *acc=malloc((size_t)S*16*sizeof(float));
+ if(!acc){ fprintf(stderr,"OOM: amx acc\n"); exit(1); }
+ int8_t bstage[4*1024] __attribute__((aligned(64))); /* bpg<=4 gated below */
+ int32_t cbuf[16*16] __attribute__((aligned(64)));
+ float sclT[16];
+ int cur=16; coli_amx_cfg(16);
+ #pragma omp for schedule(static)
+ for(int ot=0; ot<O16; ot+=16){
+ memset(acc,0,(size_t)S*16*sizeof(float));
+ for(int g=0; g<ng; g++){
+ for(int j=0;j<16;j++) sclT[j]=scale[(int64_t)(ot+j)*ng+g];
+ for(int b=0;b<bpg;b++){
+ /* B tile, VNNI [k/4][n][4]: element k of output row ot+n
+ * lands at row k/4, byte n*4+k%4 β€” signed (v-8) at unpack. */
+ int8_t *dst=bstage+(size_t)b*1024;
+ int64_t boff=((int64_t)g*gs+b*64)>>1;
+ for(int n=0;n<16;n++){
+ const uint8_t *blk=q4+(int64_t)(ot+n)*rb+boff;
+ for(int k=0;k<32;k++){
+ dst[(k>>2)*64+n*4+(k&3)] =(int8_t)((blk[k]&0xF)-8);
+ dst[((k+32)>>2)*64+n*4+((k+32)&3)]=(int8_t)((blk[k]>>4)-8);
+ }
+ }
+ }
+ for(int st=0; st<S; st+=16){
+ int rows=S-st<16?S-st:16;
+ if(rows!=cur){ coli_amx_cfg(rows); cur=rows; }
+ _tile_zero(0);
+ for(int b=0;b<bpg;b++){
+ _tile_loadd(1, xq+(int64_t)st*I+(int64_t)g*gs+b*64, (size_t)I);
+ _tile_loadd(2, bstage+(size_t)b*1024, 64);
+ _tile_dpbssd(0,1,2);
+ }
+ _tile_stored(0, cbuf, 64);
+ /* per-lane fmadd == the vector path's per-(s,o) scalar fmaf:
+ * same values, same ascending-g order, single rounding. */
+ for(int r=0;r<rows;r++){
+ __m512 cv=_mm512_cvtepi32_ps(_mm512_load_si512((const void*)(cbuf+(size_t)r*16)));
+ __m512 av=_mm512_loadu_ps(acc+(int64_t)(st+r)*16);
+ _mm512_storeu_ps(acc+(int64_t)(st+r)*16,
+ _mm512_fmadd_ps(cv,_mm512_loadu_ps(sclT),av));
+ }
+ }
+ }
+ for(int s=0;s<S;s++)
+ _mm512_storeu_ps(y+(int64_t)s*O+ot,
+ _mm512_mul_ps(_mm512_loadu_ps(acc+(int64_t)s*16),
+ _mm512_set1_ps(sx[s])));
+ }
+ _tile_release();
+ free(acc);
+ }
+}
+#endif
+
+/* vector body over an output range [o0,o1): the 1x4 row tile pays the weight
+ * block's load+mask once per 4 activation rows (the fmt=2 K2 tile's idea,
+ * brought to the grouped family β€” the prefill batch-union and the serve mux
+ * deliver exactly these multi-row calls). Integer group dots in any lane
+ * order are exact, and each row keeps its own ascending-g fmaf chain. */
+static void i4p_gidot_rows(float *y, const int8_t *xq, const float *sx,
+ const int32_t *xsg, const uint8_t *q4,
+ const float *scale, int S, int I, int O, int gs,
+ int o0, int o1){
+ int rb=(I+1)/2, ng=(I+gs-1)/gs, bpg=gs/64; /* blocchi-piano per gruppo */
+ #pragma omp parallel for schedule(static)
+ for(int o=o0;o<o1;o++){
+ const uint8_t *w=q4+(int64_t)o*rb;
+ const float *scl=scale+(int64_t)o*ng;
+ int s=0;
+ for(; s+4<=S; s+=4){
+ const int8_t *x0=xq +(int64_t)s*I, *x1=x0+I, *x2=x1+I, *x3=x2+I;
+ const int32_t *g0=xsg+(int64_t)s*ng, *g1=g0+ng, *g2=g1+ng, *g3=g2+ng;
+ float a0=0,a1=0,a2=0,a3=0;
+ int g=0;
+ for(; (g+1)*gs<=I; g++){
+ int32_t d0,d1,d2,d3;
+#if defined(__AVX512VNNI__) && defined(__AVX512BW__)
+ __m512i A0=_mm512_setzero_si512(),A1=_mm512_setzero_si512();
+ __m512i A2=_mm512_setzero_si512(),A3=_mm512_setzero_si512();
+ for(int b=0;b<bpg;b++){
+ int base=g*gs+b*64;
+ __m512i wz=i4p_blk512(w+(base>>1));
+ A0=_mm512_dpbusd_epi32(A0,wz,_mm512_loadu_si512((const void*)(x0+base)));
+ A1=_mm512_dpbusd_epi32(A1,wz,_mm512_loadu_si512((const void*)(x1+base)));
+ A2=_mm512_dpbusd_epi32(A2,wz,_mm512_loadu_si512((const void*)(x2+base)));
+ A3=_mm512_dpbusd_epi32(A3,wz,_mm512_loadu_si512((const void*)(x3+base)));
+ }
+ d0=_mm512_reduce_add_epi32(A0); d1=_mm512_reduce_add_epi32(A1);
+ d2=_mm512_reduce_add_epi32(A2); d3=_mm512_reduce_add_epi32(A3);
+#elif defined(coli_dpbusd256)
+ __m256i A0=_mm256_setzero_si256(),A1=_mm256_setzero_si256();
+ __m256i A2=_mm256_setzero_si256(),A3=_mm256_setzero_si256();
+ for(int b=0;b<bpg;b++){
+ int base=g*gs+b*64; __m256i lo,hi;
+ i4p_blk256(w+(base>>1),&lo,&hi);
+ A0=coli_dpbusd256(A0,lo,_mm256_loadu_si256((const __m256i*)(x0+base)));
+ A0=coli_dpbusd256(A0,hi,_mm256_loadu_si256((const __m256i*)(x0+base+32)));
+ A1=coli_dpbusd256(A1,lo,_mm256_loadu_si256((const __m256i*)(x1+base)));
+ A1=coli_dpbusd256(A1,hi,_mm256_loadu_si256((const __m256i*)(x1+base+32)));
+ A2=coli_dpbusd256(A2,lo,_mm256_loadu_si256((const __m256i*)(x2+base)));
+ A2=coli_dpbusd256(A2,hi,_mm256_loadu_si256((const __m256i*)(x2+base+32)));
+ A3=coli_dpbusd256(A3,lo,_mm256_loadu_si256((const __m256i*)(x3+base)));
+ A3=coli_dpbusd256(A3,hi,_mm256_loadu_si256((const __m256i*)(x3+base+32)));
+ }
+ d0=hsum256_i32(A0); d1=hsum256_i32(A1);
+ d2=hsum256_i32(A2); d3=hsum256_i32(A3);
+#elif defined(__AVX2__)
+ const __m256i ones=_mm256_set1_epi16(1);
+ const __m256i m4=_mm256_set1_epi8(0x0F);
+ __m256i A0=_mm256_setzero_si256(),A1=_mm256_setzero_si256();
+ __m256i A2=_mm256_setzero_si256(),A3=_mm256_setzero_si256();
+ for(int b=0;b<bpg;b++){
+ int base=g*gs+b*64;
+ __m256i bb=_mm256_loadu_si256((const __m256i*)(w+(base>>1)));
+ __m256i lo=_mm256_and_si256(bb,m4);
+ __m256i hi=_mm256_and_si256(_mm256_srli_epi16(bb,4),m4);
+ /* maddubs(u8,s8): u<=15, |x|<=127 -> pair <= 3810, int16-safe */
+ A0=_mm256_add_epi32(A0,_mm256_madd_epi16(_mm256_maddubs_epi16(lo,_mm256_loadu_si256((const __m256i*)(x0+base))),ones));
+ A0=_mm256_add_epi32(A0,_mm256_madd_epi16(_mm256_maddubs_epi16(hi,_mm256_loadu_si256((const __m256i*)(x0+base+32))),ones));
+ A1=_mm256_add_epi32(A1,_mm256_madd_epi16(_mm256_maddubs_epi16(lo,_mm256_loadu_si256((const __m256i*)(x1+base))),ones));
+ A1=_mm256_add_epi32(A1,_mm256_madd_epi16(_mm256_maddubs_epi16(hi,_mm256_loadu_si256((const __m256i*)(x1+base+32))),ones));
+ A2=_mm256_add_epi32(A2,_mm256_madd_epi16(_mm256_maddubs_epi16(lo,_mm256_loadu_si256((const __m256i*)(x2+base))),ones));
+ A2=_mm256_add_epi32(A2,_mm256_madd_epi16(_mm256_maddubs_epi16(hi,_mm256_loadu_si256((const __m256i*)(x2+base+32))),ones));
+ A3=_mm256_add_epi32(A3,_mm256_madd_epi16(_mm256_maddubs_epi16(lo,_mm256_loadu_si256((const __m256i*)(x3+base))),ones));
+ A3=_mm256_add_epi32(A3,_mm256_madd_epi16(_mm256_maddubs_epi16(hi,_mm256_loadu_si256((const __m256i*)(x3+base+32))),ones));
+ }
+ d0=hsum256_i32(A0); d1=hsum256_i32(A1);
+ d2=hsum256_i32(A2); d3=hsum256_i32(A3);
+#else
+ d0=d1=d2=d3=0;
+ for(int b=0;b<bpg;b++){
+ int base=g*gs+b*64;
+ const uint8_t *blk=w+(base>>1);
+ for(int k=0;k<32;k++){
+ int32_t ul=(int32_t)(blk[k]&0xF), uh=(int32_t)(blk[k]>>4);
+ d0+=ul*x0[base+k]+uh*x0[base+k+32];
+ d1+=ul*x1[base+k]+uh*x1[base+k+32];
+ d2+=ul*x2[base+k]+uh*x2[base+k+32];
+ d3+=ul*x3[base+k]+uh*x3[base+k+32];
+ }
+ }
+#endif
+ a0=fmaf((float)(d0-8*g0[g]),scl[g],a0);
+ a1=fmaf((float)(d1-8*g1[g]),scl[g],a1);
+ a2=fmaf((float)(d2-8*g2[g]),scl[g],a2);
+ a3=fmaf((float)(d3-8*g3[g]),scl[g],a3);
+ }
+ if(g*gs<I){ /* coda: gruppo parziale, nibble a coppie */
+ int32_t d0=0,d1=0,d2=0,d3=0;
+ for(int i=g*gs;i<I;i++){
+ uint8_t byte=w[i>>1];
+ int32_t u=(int32_t)((i&1)?(byte>>4):(byte&0xF));
+ d0+=u*x0[i]; d1+=u*x1[i]; d2+=u*x2[i]; d3+=u*x3[i];
+ }
+ a0=fmaf((float)(d0-8*g0[g]),scl[g],a0);
+ a1=fmaf((float)(d1-8*g1[g]),scl[g],a1);
+ a2=fmaf((float)(d2-8*g2[g]),scl[g],a2);
+ a3=fmaf((float)(d3-8*g3[g]),scl[g],a3);
+ }
+ y[(int64_t)s*O+o] =a0*sx[s];
+ y[(int64_t)(s+1)*O+o]=a1*sx[s+1];
+ y[(int64_t)(s+2)*O+o]=a2*sx[s+2];
+ y[(int64_t)(s+3)*O+o]=a3*sx[s+3];
+ }
+ for(; s<S; s++){
+ const int8_t *xr=xq+(int64_t)s*I;
+ const int32_t *xg=xsg+(int64_t)s*ng;
+ float a=0; int g=0;
+ for(; (g+1)*gs<=I; g++){ /* gruppi interi */
+ int32_t d=0;
+#if defined(__AVX512VNNI__) && defined(__AVX512BW__)
+ __m512i acc=_mm512_setzero_si512();
+ for(int b=0;b<bpg;b++){
+ int base=g*gs+b*64;
+ acc=_mm512_dpbusd_epi32(acc,i4p_blk512(w+(base>>1)),
+ _mm512_loadu_si512((const void*)(xr+base)));
+ }
+ d=_mm512_reduce_add_epi32(acc);
+#else
+ for(int b=0;b<bpg;b++){
+ int base=g*gs+b*64;
+ const uint8_t *blk=w+(base>>1);
+ const int8_t *xb=xr+base;
+#if defined(coli_dpbusd256)
+ __m256i lo,hi; i4p_blk256(blk,&lo,&hi);
+ __m256i acc=_mm256_setzero_si256();
+ acc=coli_dpbusd256(acc,lo,_mm256_loadu_si256((const __m256i*)xb));
+ acc=coli_dpbusd256(acc,hi,_mm256_loadu_si256((const __m256i*)(xb+32)));
+ d+=hsum256_i32(acc);
+#elif defined(__AVX2__)
+ const __m256i m4=_mm256_set1_epi8(0x0F);
+ const __m256i ones=_mm256_set1_epi16(1);
+ __m256i bb=_mm256_loadu_si256((const __m256i*)blk);
+ __m256i p0=_mm256_maddubs_epi16(_mm256_and_si256(bb,m4),
+ _mm256_loadu_si256((const __m256i*)xb));
+ __m256i p1=_mm256_maddubs_epi16(_mm256_and_si256(_mm256_srli_epi16(bb,4),m4),
+ _mm256_loadu_si256((const __m256i*)(xb+32)));
+ __m256i acc=_mm256_add_epi32(_mm256_madd_epi16(p0,ones),
+ _mm256_madd_epi16(p1,ones));
+ d+=hsum256_i32(acc);
+#else
+ for(int k=0;k<32;k++){
+ d+=(int32_t)(blk[k]&0xF)*xb[k];
+ d+=(int32_t)(blk[k]>>4)*xb[k+32];
+ }
+#endif
+ }
+#endif
+ a=fmaf((float)(d-8*xg[g]),scl[g],a);
+ }
+ if(g*gs<I){ /* coda: gruppo parziale, nibble a coppie */
+ int32_t d=0;
+ for(int i=g*gs;i<I;i++){
+ uint8_t byte=w[i>>1];
+ d+=(int32_t)((i&1)?(byte>>4):(byte&0xF))*xr[i];
+ }
+ a=fmaf((float)(d-8*xg[g]),scl[g],a);
+ }
+ y[(int64_t)s*O+o]=a*sx[s];
+ }
+ }
+}
+
+static void matmul_i4p_grouped_idot(float *y, const int8_t *xq, const float *sx,
+ const int32_t *xsg, const uint8_t *q4,
+ const float *scale, int S, int I, int O, int gs){
+#if defined(COLI_HAVE_AMX_I4P)
+ /* AMX takes the aligned bulk (full 16-output tiles, full groups only β€”
+ * the real gs64 checkpoints have I%gs==0); the vector path finishes any
+ * output remainder. Below the row threshold the B-tile unpack does not
+ * amortize and the vector tile is the better kernel. */
+ if(coli_amx_ok() && S>=coli_amx_s_min && gs%64==0 && gs<=256 && I%gs==0 && O>=16){
+ int O16=O&~15;
+ matmul_i4p_gidot_amx(y,xq,sx,q4,scale,S,I,O16,O,gs);
+ if(O16<O) i4p_gidot_rows(y,xq,sx,xsg,q4,scale,S,I,O,gs,O16,O);
+ return;
+ }
+#endif
+ i4p_gidot_rows(y,xq,sx,xsg,q4,scale,S,I,O,gs,0,O);
+}
+
+/* matmul IDOT planare (fmt=2): y = (dot_u - 8*xsum[s]) * scale[o] * sx[s].
+ * Bit-identico a matmul_i4_idot: somme intere, identita' esatta. */
+static void matmul_i4p_idot(float *y, const int8_t *xq, const float *sx, const int32_t *xsum,
+ const uint8_t *q4, const float *scale, int S, int I, int O){
+ int rb=(I+1)/2;
+ #pragma omp parallel for schedule(static)
+ for(int o=0;o<O;o++){ const uint8_t *w=q4+(int64_t)o*rb; float sc=scale[o];
+ int s=0;
+#if defined(coli_dpbusd256)
+ /* K2: tile 1x4 sulla union β€” il blocco pesi (load + 2 and + srli) si
+ * paga UNA volta per 4 righe di attivazione invece che per riga. La
+ * union del prefill consegna nr=2..16 righe per expert: e' esattamente
+ * la finestra che il per-riga serviva peggio. Somme intere ->
+ * bit-identico al path per-riga per associativita'.
+ * EN: 1x4 register tile β€” the weight block's load+mask cost is paid
+ * once per 4 activation rows. Integer sums: bit-identical to the
+ * per-row path by associativity. */
+ const __m256i m4t=_mm256_set1_epi8(0x0F);
+ for(;s+4<=S;s+=4){
+ const int8_t *x0=xq+(int64_t)s*I, *x1=x0+I, *x2=x1+I, *x3=x2+I;
+ __m256i a0=_mm256_setzero_si256(), a1=_mm256_setzero_si256();
+ __m256i a2=_mm256_setzero_si256(), a3=_mm256_setzero_si256();
+ int i=0;
+ for(;i+64<=I;i+=64){
+ __m256i b =_mm256_loadu_si256((const __m256i*)(w+(i>>1)));
+ __m256i lo=_mm256_and_si256(b,m4t);
+ __m256i hi=_mm256_and_si256(_mm256_srli_epi16(b,4),m4t);
+ a0=coli_dpbusd256(a0,lo,_mm256_loadu_si256((const __m256i*)(x0+i)));
+ a0=coli_dpbusd256(a0,hi,_mm256_loadu_si256((const __m256i*)(x0+i+32)));
+ a1=coli_dpbusd256(a1,lo,_mm256_loadu_si256((const __m256i*)(x1+i)));
+ a1=coli_dpbusd256(a1,hi,_mm256_loadu_si256((const __m256i*)(x1+i+32)));
+ a2=coli_dpbusd256(a2,lo,_mm256_loadu_si256((const __m256i*)(x2+i)));
+ a2=coli_dpbusd256(a2,hi,_mm256_loadu_si256((const __m256i*)(x2+i+32)));
+ a3=coli_dpbusd256(a3,lo,_mm256_loadu_si256((const __m256i*)(x3+i)));
+ a3=coli_dpbusd256(a3,hi,_mm256_loadu_si256((const __m256i*)(x3+i+32)));
+ }
+ int32_t d0=hsum256_i32(a0), d1=hsum256_i32(a1);
+ int32_t d2=hsum256_i32(a2), d3=hsum256_i32(a3);
+ /* coda a coppie, unsigned: stessa identita' -8*xsum del per-riga */
+ for(;i<I;i+=2){
+ uint8_t byte=w[i>>1];
+ d0+=(int32_t)(byte&0xF)*x0[i]; d1+=(int32_t)(byte&0xF)*x1[i];
+ d2+=(int32_t)(byte&0xF)*x2[i]; d3+=(int32_t)(byte&0xF)*x3[i];
+ if(i+1<I){
+ d0+=(int32_t)(byte>>4)*x0[i+1]; d1+=(int32_t)(byte>>4)*x1[i+1];
+ d2+=(int32_t)(byte>>4)*x2[i+1]; d3+=(int32_t)(byte>>4)*x3[i+1];
+ }
+ }
+ y[(int64_t)(s+0)*O+o]=(float)(d0-8*xsum[s+0])*sc*sx[s+0];
+ y[(int64_t)(s+1)*O+o]=(float)(d1-8*xsum[s+1])*sc*sx[s+1];
+ y[(int64_t)(s+2)*O+o]=(float)(d2-8*xsum[s+2])*sc*sx[s+2];
+ y[(int64_t)(s+3)*O+o]=(float)(d3-8*xsum[s+3])*sc*sx[s+3];
+ }
+#endif
+ for(;s<S;s++){
+ int32_t d=dot_i4p_u(w,xq+(int64_t)s*I,I)-8*xsum[s];
+ y[(int64_t)s*O+o]=(float)d*sc*sx[s];
+ }
+ }
+}
+
+#endif /* COLIBRI_IDOT_H */
diff --git a/c/quant.h b/c/quant.h
index e2c0b314..91cf7549 100644
--- a/c/quant.h
+++ b/c/quant.h
@@ -14,35 +14,7 @@
#include <omp.h>
#endif
-/* ---- SIMD includes -------------------------------------------------------- */
-#ifdef __AVX2__
-#include <immintrin.h>
-static inline float hsum256(__m256 v){
- __m128 lo=_mm256_castps256_ps128(v), hi=_mm256_extractf128_ps(v,1);
- lo=_mm_add_ps(lo,hi); __m128 sh=_mm_movehl_ps(lo,lo); lo=_mm_add_ps(lo,sh);
- sh=_mm_shuffle_ps(lo,lo,1); lo=_mm_add_ss(lo,sh); return _mm_cvtss_f32(lo);
-}
-static inline int hsum256_i32(__m256i v){
- __m128i lo=_mm256_castsi256_si128(v), hi=_mm256_extracti128_si256(v,1);
- lo=_mm_add_epi32(lo,hi); lo=_mm_hadd_epi32(lo,lo); lo=_mm_hadd_epi32(lo,lo);
- return _mm_cvtsi128_si32(lo);
-}
-#endif
-#if defined(__AVXVNNI__) && defined(__AVX2__)
-static inline int hsum128_i32(__m128i v){
- v=_mm_hadd_epi32(v,v); v=_mm_hadd_epi32(v,v); return _mm_cvtsi128_si32(v);
-}
-#endif
-#ifdef __ARM_NEON
-#include <arm_neon.h>
-#endif
-#ifdef __VSX__
-#include <altivec.h>
-#undef vector
-#undef pixel
-#undef bool
-#endif
-
+#include "idot.h" /* SIMD prelude + the integer dot kernels, shared with qwen36 */
/* ---- AVX-512 int4->float accumulator -------------------------------------- */
#if defined(__AVX512F__) && defined(__AVX512BW__)
static int g_i4_acc512=1;
@@ -654,931 +626,6 @@ static void matmul_fp8(float *y, const float *x, const uint8_t *q8, const float
}
#endif
-/* ---- IDOT: integer dot kernels (int8-quantized activations) --------------- */
-#if defined(__AVX512VNNI__) && defined(__AVX512BW__)
-#define IDOT_KERNEL "avx512-vnni"
-#elif defined(__AVXVNNI__) && defined(__AVX2__)
-#define IDOT_KERNEL "avx-vnni"
-#elif defined(__AVX2__)
-#define IDOT_KERNEL "avx2"
-#elif defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)
-#define IDOT_KERNEL "neon-i8mm"
-#elif defined(__ARM_NEON)
-#define IDOT_KERNEL "neon"
-#elif defined(__VSX__)
-#define IDOT_KERNEL "vsx"
-#else
-#define IDOT_KERNEL "scalar"
-#endif
-
-static inline float qrow_i8(const float *x, int8_t *q, int I){
- float amax=0; for(int i=0;i<I;i++){ float a=fabsf(x[i]); if(a>amax)amax=a; }
- float s=amax/127.f; if(s<1e-12f) s=1e-12f; float inv=1.f/s;
- for(int i=0;i<I;i++) q[i]=(int8_t)lrintf(x[i]*inv);
- return s;
-}
-
-/* dot int8*int8 */
-static inline int32_t dot_i8i8(const int8_t *w, const int8_t *x, int I){
- int32_t sum=0; int i=0;
-#if defined(__AVX512VNNI__) && defined(__AVX512BW__)
- __m512i acc=_mm512_setzero_si512();
- for(;i+64<=I;i+=64){
- __m512i wv=_mm512_loadu_si512((const void*)(w+i));
- __m512i xv=_mm512_loadu_si512((const void*)(x+i));
- __mmask64 neg=_mm512_movepi8_mask(wv);
- __m512i xs=_mm512_mask_sub_epi8(xv,neg,_mm512_setzero_si512(),xv);
- acc=_mm512_dpbusd_epi32(acc,_mm512_abs_epi8(wv),xs);
- }
- sum=_mm512_reduce_add_epi32(acc);
-#elif defined(__AVXVNNI__) && defined(__AVX2__)
- /* 4 accumulatori indipendenti (64 byte/iter): un solo acc incatena i vpdpbusd
- * (latenza-bound ~5c). Somme intere associative -> bit-identico. Stessa struttura
- * dei 4 accumulatori del ramo NEON piu' sotto.
- * EN: four independent accumulators break the serial vpdpbusd->acc chain; integer
- * adds are associative, so the result is bit-identical (mirrors the NEON path). */
- __m128i a0=_mm_setzero_si128(),a1=_mm_setzero_si128(),a2=_mm_setzero_si128(),a3=_mm_setzero_si128();
- for(;i+64<=I;i+=64){
- __m128i w0=_mm_loadu_si128((const __m128i*)(w+i)), x0=_mm_loadu_si128((const __m128i*)(x+i));
- __m128i w1=_mm_loadu_si128((const __m128i*)(w+i+16)), x1=_mm_loadu_si128((const __m128i*)(x+i+16));
- __m128i w2=_mm_loadu_si128((const __m128i*)(w+i+32)), x2=_mm_loadu_si128((const __m128i*)(x+i+32));
- __m128i w3=_mm_loadu_si128((const __m128i*)(w+i+48)), x3=_mm_loadu_si128((const __m128i*)(x+i+48));
- a0=_mm_dpbusd_epi32(a0,_mm_abs_epi8(w0),_mm_sign_epi8(x0,w0));
- a1=_mm_dpbusd_epi32(a1,_mm_abs_epi8(w1),_mm_sign_epi8(x1,w1));
- a2=_mm_dpbusd_epi32(a2,_mm_abs_epi8(w2),_mm_sign_epi8(x2,w2));
- a3=_mm_dpbusd_epi32(a3,_mm_abs_epi8(w3),_mm_sign_epi8(x3,w3));
- }
- __m128i acc=_mm_add_epi32(_mm_add_epi32(a0,a1),_mm_add_epi32(a2,a3));
- for(;i+16<=I;i+=16){
- __m128i wv=_mm_loadu_si128((const __m128i*)(w+i));
- __m128i xv=_mm_loadu_si128((const __m128i*)(x+i));
- acc=_mm_dpbusd_epi32(acc,_mm_abs_epi8(wv),_mm_sign_epi8(xv,wv));
- }
- sum=hsum128_i32(acc);
-#elif defined(__AVX2__)
- __m256i acc=_mm256_setzero_si256(); const __m256i ones=_mm256_set1_epi16(1);
- for(;i+32<=I;i+=32){
- __m256i wv=_mm256_loadu_si256((const __m256i*)(w+i));
- __m256i xv=_mm256_loadu_si256((const __m256i*)(x+i));
- __m256i p=_mm256_maddubs_epi16(_mm256_sign_epi8(wv,wv),_mm256_sign_epi8(xv,wv));
- acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p,ones));
- }
- sum=hsum256_i32(acc);
-#elif defined(__ARM_NEON)
-#if defined(__ARM_FEATURE_DOTPROD)
- int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0);
- for(;i+64<=I;i+=64){
- a0=vdotq_s32(a0,vld1q_s8(w+i), vld1q_s8(x+i));
- a1=vdotq_s32(a1,vld1q_s8(w+i+16),vld1q_s8(x+i+16));
- a2=vdotq_s32(a2,vld1q_s8(w+i+32),vld1q_s8(x+i+32));
- a3=vdotq_s32(a3,vld1q_s8(w+i+48),vld1q_s8(x+i+48));
- }
- int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));
- for(;i+16<=I;i+=16) acc=vdotq_s32(acc,vld1q_s8(w+i),vld1q_s8(x+i));
- sum=vaddvq_s32(acc);
-#else
- int32x4_t acc=vdupq_n_s32(0);
- for(;i+16<=I;i+=16){
- int8x16_t wv=vld1q_s8(w+i), xv=vld1q_s8(x+i);
- int16x8_t p=vmull_s8(vget_low_s8(wv),vget_low_s8(xv));
- p=vmlal_s8(p,vget_high_s8(wv),vget_high_s8(xv));
- acc=vpadalq_s16(acc,p);
- }
- sum=vaddvq_s32(acc);
-#endif
-#elif defined(__VSX__)
- __vector signed int acc=vec_splats(0);
- const __vector signed char vz=vec_splats((signed char)0);
- for(;i+16<=I;i+=16){
- __vector signed char wv=vec_xl(0,(const signed char*)(w+i));
- __vector signed char xv=vec_xl(0,(const signed char*)(x+i));
- __vector __bool char neg=vec_cmplt(wv,vz);
- __vector signed char xs=vec_sel(xv,vec_sub(vz,xv),neg);
- __vector unsigned char wa=(__vector unsigned char)vec_sel(wv,vec_sub(vz,wv),neg);
- acc=vec_msum(xs,wa,acc);
- }
- sum=vec_extract(acc,0)+vec_extract(acc,1)+vec_extract(acc,2)+vec_extract(acc,3);
-#endif
- for(;i<I;i++) sum+=(int32_t)w[i]*x[i];
- return sum;
-}
-
-/* dot int4(packed)*int8 */
-static inline int32_t dot_i4i8(const uint8_t *w4, const int8_t *x, int I){
- int32_t sum=0; int i=0;
-#if defined(__AVX512VNNI__) && defined(__AVX512BW__)
- const __m256i m4v=_mm256_set1_epi8(0x0F);
- const __m512i b8v=_mm512_set1_epi8(8);
- const __m512i xidx=_mm512_setr_epi64(0,1,4,5,2,3,6,7);
- __m512i acc=_mm512_setzero_si512();
- for(;i+64<=I;i+=64){
- __m256i by=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));
- __m256i lo=_mm256_and_si256(by,m4v), hi=_mm256_and_si256(_mm256_srli_epi16(by,4),m4v);
- __m256i z0=_mm256_unpacklo_epi8(lo,hi), z1=_mm256_unpackhi_epi8(lo,hi);
- __m512i wv=_mm512_sub_epi8(_mm512_inserti64x4(_mm512_castsi256_si512(z0),z1,1),b8v);
- __m512i xv=_mm512_permutexvar_epi64(xidx,_mm512_loadu_si512((const void*)(x+i)));
- __mmask64 neg=_mm512_movepi8_mask(wv);
- __m512i xs=_mm512_mask_sub_epi8(xv,neg,_mm512_setzero_si512(),xv);
- acc=_mm512_dpbusd_epi32(acc,_mm512_abs_epi8(wv),xs);
- }
- sum=_mm512_reduce_add_epi32(acc);
-#elif defined(__AVXVNNI__) && defined(__AVX2__)
- /* 4 accumulatori indipendenti (64 elementi = 32 byte packed/iter): un solo acc
- * incatena i vpdpbusd (latenza-bound ~5c). Somme intere associative -> bit-identico.
- * Stessa struttura dei 4 accumulatori del ramo NEON piu' sotto.
- * EN: four independent accumulators break the serial vpdpbusd->acc chain; integer
- * adds are associative, so the result is bit-identical (mirrors the NEON path). */
- const __m128i m4=_mm_set1_epi8(0x0F); const __m128i b8=_mm_set1_epi8(8);
- __m128i a0=_mm_setzero_si128(),a1=_mm_setzero_si128(),a2=_mm_setzero_si128(),a3=_mm_setzero_si128();
- for(;i+64<=I;i+=64){
- __m128i by0=_mm_loadu_si128((const __m128i*)(w4+(i>>1))); /* elem i..i+31 */
- __m128i by1=_mm_loadu_si128((const __m128i*)(w4+(i>>1)+16)); /* elem i+32..i+63 */
- __m128i lo0=_mm_and_si128(by0,m4), hi0=_mm_and_si128(_mm_srli_epi16(by0,4),m4);
- __m128i lo1=_mm_and_si128(by1,m4), hi1=_mm_and_si128(_mm_srli_epi16(by1,4),m4);
- __m128i w0=_mm_sub_epi8(_mm_unpacklo_epi8(lo0,hi0),b8), w1=_mm_sub_epi8(_mm_unpackhi_epi8(lo0,hi0),b8);
- __m128i w2=_mm_sub_epi8(_mm_unpacklo_epi8(lo1,hi1),b8), w3=_mm_sub_epi8(_mm_unpackhi_epi8(lo1,hi1),b8);
- __m128i x0=_mm_loadu_si128((const __m128i*)(x+i)), x1=_mm_loadu_si128((const __m128i*)(x+i+16));
- __m128i x2=_mm_loadu_si128((const __m128i*)(x+i+32)), x3=_mm_loadu_si128((const __m128i*)(x+i+48));
- a0=_mm_dpbusd_epi32(a0,_mm_abs_epi8(w0),_mm_sign_epi8(x0,w0));
- a1=_mm_dpbusd_epi32(a1,_mm_abs_epi8(w1),_mm_sign_epi8(x1,w1));
- a2=_mm_dpbusd_epi32(a2,_mm_abs_epi8(w2),_mm_sign_epi8(x2,w2));
- a3=_mm_dpbusd_epi32(a3,_mm_abs_epi8(w3),_mm_sign_epi8(x3,w3));
- }
- __m128i acc=_mm_add_epi32(_mm_add_epi32(a0,a1),_mm_add_epi32(a2,a3));
- for(;i+32<=I;i+=32){ /* 32-nibble remainder: 2 dpbusd, same unpack */
- __m128i by=_mm_loadu_si128((const __m128i*)(w4+(i>>1)));
- __m128i lo=_mm_and_si128(by,m4), hi=_mm_and_si128(_mm_srli_epi16(by,4),m4);
- __m128i w0=_mm_sub_epi8(_mm_unpacklo_epi8(lo,hi),b8), w1=_mm_sub_epi8(_mm_unpackhi_epi8(lo,hi),b8);
- __m128i x0=_mm_loadu_si128((const __m128i*)(x+i));
- __m128i x1=_mm_loadu_si128((const __m128i*)(x+i+16));
- acc=_mm_dpbusd_epi32(acc,_mm_abs_epi8(w0),_mm_sign_epi8(x0,w0));
- acc=_mm_dpbusd_epi32(acc,_mm_abs_epi8(w1),_mm_sign_epi8(x1,w1));
- }
- sum=hsum128_i32(acc);
-#elif defined(__AVX2__)
- const __m128i m4=_mm_set1_epi8(0x0F); const __m256i b8=_mm256_set1_epi8(8);
- const __m256i ones=_mm256_set1_epi16(1);
- __m256i acc=_mm256_setzero_si256();
- for(;i+32<=I;i+=32){
- __m128i by=_mm_loadu_si128((const __m128i*)(w4+(i>>1)));
- __m128i lo=_mm_and_si128(by,m4), hi=_mm_and_si128(_mm_srli_epi16(by,4),m4);
- __m128i n0=_mm_unpacklo_epi8(lo,hi), n1=_mm_unpackhi_epi8(lo,hi);
- __m256i wv=_mm256_sub_epi8(_mm256_set_m128i(n1,n0),b8);
- __m256i xv=_mm256_loadu_si256((const __m256i*)(x+i));
- __m256i p=_mm256_maddubs_epi16(_mm256_sign_epi8(wv,wv),_mm256_sign_epi8(xv,wv));
- acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p,ones));
- }
- sum=hsum256_i32(acc);
-#elif defined(__ARM_NEON)
- const uint8x16_t m4q=vdupq_n_u8(0x0F); const int8x16_t b8q=vdupq_n_s8(8);
-#if defined(__ARM_FEATURE_DOTPROD)
- int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0);
- for(;i+64<=I;i+=64){
- uint8x16_t byA=vld1q_u8(w4+(i>>1)), byB=vld1q_u8(w4+(i>>1)+16);
- uint8x16x2_t zA=vzipq_u8(vandq_u8(byA,m4q), vshrq_n_u8(byA,4));
- uint8x16x2_t zB=vzipq_u8(vandq_u8(byB,m4q), vshrq_n_u8(byB,4));
- a0=vdotq_s32(a0,vsubq_s8(vreinterpretq_s8_u8(zA.val[0]),b8q),vld1q_s8(x+i));
- a1=vdotq_s32(a1,vsubq_s8(vreinterpretq_s8_u8(zA.val[1]),b8q),vld1q_s8(x+i+16));
- a2=vdotq_s32(a2,vsubq_s8(vreinterpretq_s8_u8(zB.val[0]),b8q),vld1q_s8(x+i+32));
- a3=vdotq_s32(a3,vsubq_s8(vreinterpretq_s8_u8(zB.val[1]),b8q),vld1q_s8(x+i+48));
- }
- int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));
- for(;i+32<=I;i+=32){
- uint8x16_t by=vld1q_u8(w4+(i>>1));
- uint8x16x2_t z=vzipq_u8(vandq_u8(by,m4q), vshrq_n_u8(by,4));
- acc=vdotq_s32(acc,vsubq_s8(vreinterpretq_s8_u8(z.val[0]),b8q),vld1q_s8(x+i));
- acc=vdotq_s32(acc,vsubq_s8(vreinterpretq_s8_u8(z.val[1]),b8q),vld1q_s8(x+i+16));
- }
- sum=vaddvq_s32(acc);
-#else
- int32x4_t acc=vdupq_n_s32(0);
- for(;i+32<=I;i+=32){
- uint8x16_t by=vld1q_u8(w4+(i>>1));
- uint8x16x2_t z=vzipq_u8(vandq_u8(by,m4q), vshrq_n_u8(by,4));
- int8x16_t w0=vsubq_s8(vreinterpretq_s8_u8(z.val[0]),b8q);
- int8x16_t w1=vsubq_s8(vreinterpretq_s8_u8(z.val[1]),b8q);
- int8x16_t x0=vld1q_s8(x+i), x1=vld1q_s8(x+i+16);
- int16x8_t p=vmull_s8(vget_low_s8(w0),vget_low_s8(x0));
- p=vmlal_s8(p,vget_high_s8(w0),vget_high_s8(x0));
- acc=vpadalq_s16(acc,p);
- p=vmull_s8(vget_low_s8(w1),vget_low_s8(x1));
- p=vmlal_s8(p,vget_high_s8(w1),vget_high_s8(x1));
- acc=vpadalq_s16(acc,p);
- }
- sum=vaddvq_s32(acc);
-#endif
-#elif defined(__VSX__)
- const __vector unsigned char m4v=vec_splats((unsigned char)0x0F);
- const __vector unsigned char sh4=vec_splats((unsigned char)4);
- const __vector signed char b8v=vec_splats((signed char)8);
- const __vector signed char vz=vec_splats((signed char)0);
- __vector signed int acc=vec_splats(0);
- for(;i+32<=I;i+=32){
- __vector unsigned char by=vec_xl(0,w4+(i>>1));
- __vector unsigned char lo=vec_and(by,m4v), hi=vec_sr(by,sh4);
- __vector signed char w0=vec_sub((__vector signed char)vec_mergeh(lo,hi),b8v);
- __vector signed char w1=vec_sub((__vector signed char)vec_mergel(lo,hi),b8v);
- __vector signed char x0=vec_xl(0,(const signed char*)(x+i));
- __vector signed char x1=vec_xl(0,(const signed char*)(x+i+16));
- __vector __bool char n0=vec_cmplt(w0,vz), n1=vec_cmplt(w1,vz);
- acc=vec_msum(vec_sel(x0,vec_sub(vz,x0),n0),
- (__vector unsigned char)vec_sel(w0,vec_sub(vz,w0),n0),acc);
- acc=vec_msum(vec_sel(x1,vec_sub(vz,x1),n1),
- (__vector unsigned char)vec_sel(w1,vec_sub(vz,w1),n1),acc);
- }
- sum=vec_extract(acc,0)+vec_extract(acc,1)+vec_extract(acc,2)+vec_extract(acc,3);
-#endif
- for(;i+1<I;i+=2){ uint8_t b=w4[i>>1]; sum+=((int)(b&0xF)-8)*x[i]+((int)(b>>4)-8)*x[i+1]; }
- if(i<I){ uint8_t b=w4[i>>1]; sum+=((int)(b&0xF)-8)*x[i]; }
- return sum;
-}
-
-/* ---- ARM i8mm SMMLA tiled kernels ---------------------------------------- */
-#if defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)
-static inline int32x4_t mm_tile16(int32x4_t acc, int8x16_t wo, int8x16_t wo1,
- int8x16_t xs, int8x16_t xs1){
- acc=vmmlaq_s32(acc, vcombine_s8(vget_low_s8(wo), vget_low_s8(wo1)),
- vcombine_s8(vget_low_s8(xs), vget_low_s8(xs1)));
- return vmmlaq_s32(acc, vcombine_s8(vget_high_s8(wo), vget_high_s8(wo1)),
- vcombine_s8(vget_high_s8(xs), vget_high_s8(xs1)));
-}
-static void matmul_q_idot_mm(float *y, const int8_t *xq, const float *sx, const int8_t *q,
- const float *scale, int S, int I, int O){
- #pragma omp parallel for schedule(static)
- for(int o=0;o<(O&~1);o+=2){
- const int8_t *wo=q+(int64_t)o*I, *wo1=q+(int64_t)(o+1)*I;
- float sc0=scale[o], sc1=scale[o+1];
- for(int s=0;s<(S&~1);s+=2){
- const int8_t *xs=xq+(int64_t)s*I, *xs1=xq+(int64_t)(s+1)*I;
- int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0); int i=0;
- for(;i+64<=I;i+=64){
- a0=mm_tile16(a0,vld1q_s8(wo+i), vld1q_s8(wo1+i), vld1q_s8(xs+i), vld1q_s8(xs1+i));
- a1=mm_tile16(a1,vld1q_s8(wo+i+16),vld1q_s8(wo1+i+16),vld1q_s8(xs+i+16),vld1q_s8(xs1+i+16));
- a2=mm_tile16(a2,vld1q_s8(wo+i+32),vld1q_s8(wo1+i+32),vld1q_s8(xs+i+32),vld1q_s8(xs1+i+32));
- a3=mm_tile16(a3,vld1q_s8(wo+i+48),vld1q_s8(wo1+i+48),vld1q_s8(xs+i+48),vld1q_s8(xs1+i+48));
- }
- for(;i+16<=I;i+=16)
- a0=mm_tile16(a0,vld1q_s8(wo+i),vld1q_s8(wo1+i),vld1q_s8(xs+i),vld1q_s8(xs1+i));
- int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));
- int32_t d00=vgetq_lane_s32(acc,0), d01=vgetq_lane_s32(acc,1);
- int32_t d10=vgetq_lane_s32(acc,2), d11=vgetq_lane_s32(acc,3);
- for(;i<I;i++){ int a=wo[i],b=wo1[i],u=xs[i],v=xs1[i];
- d00+=a*u; d01+=a*v; d10+=b*u; d11+=b*v; }
- y[(int64_t)s*O+o] =(float)d00*sc0*sx[s];
- y[(int64_t)s*O+(o+1)] =(float)d10*sc1*sx[s];
- y[(int64_t)(s+1)*O+o] =(float)d01*sc0*sx[s+1];
- y[(int64_t)(s+1)*O+(o+1)]=(float)d11*sc1*sx[s+1];
- }
- if(S&1){ int s=S-1; const int8_t *xs=xq+(int64_t)s*I;
- y[(int64_t)s*O+o] =(float)dot_i8i8(wo, xs,I)*sc0*sx[s];
- y[(int64_t)s*O+(o+1)]=(float)dot_i8i8(wo1,xs,I)*sc1*sx[s]; }
- }
- if(O&1){ int o=O-1; const int8_t *w=q+(int64_t)o*I; float sc=scale[o];
- #pragma omp parallel for schedule(static)
- for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i8i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }
-}
-static void matmul_i4_idot_mm(float *y, const int8_t *xq, const float *sx, const uint8_t *q4,
- const float *scale, int S, int I, int O){
- int rb=(I+1)/2;
- #pragma omp parallel for schedule(static)
- for(int o=0;o<(O&~1);o+=2){
- const uint8x16_t m4q=vdupq_n_u8(0x0F); const int8x16_t b8q=vdupq_n_s8(8);
- const uint8_t *wo=q4+(int64_t)o*rb, *wo1=q4+(int64_t)(o+1)*rb;
- float sc0=scale[o], sc1=scale[o+1];
- for(int s=0;s<(S&~1);s+=2){
- const int8_t *xs=xq+(int64_t)s*I, *xs1=xq+(int64_t)(s+1)*I;
- int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0); int i=0;
- for(;i+64<=I;i+=64){
- uint8x16_t byo=vld1q_u8(wo+(i>>1)), byo1=vld1q_u8(wo1+(i>>1));
- uint8x16_t cyo=vld1q_u8(wo+(i>>1)+16), cyo1=vld1q_u8(wo1+(i>>1)+16);
- uint8x16x2_t zo =vzipq_u8(vandq_u8(byo, m4q), vshrq_n_u8(byo, 4));
- uint8x16x2_t zo1=vzipq_u8(vandq_u8(byo1,m4q), vshrq_n_u8(byo1,4));
- uint8x16x2_t ko =vzipq_u8(vandq_u8(cyo, m4q), vshrq_n_u8(cyo, 4));
- uint8x16x2_t ko1=vzipq_u8(vandq_u8(cyo1,m4q), vshrq_n_u8(cyo1,4));
- a0=mm_tile16(a0, vsubq_s8(vreinterpretq_s8_u8(zo.val[0]),b8q),
- vsubq_s8(vreinterpretq_s8_u8(zo1.val[0]),b8q),
- vld1q_s8(xs+i), vld1q_s8(xs1+i));
- a1=mm_tile16(a1, vsubq_s8(vreinterpretq_s8_u8(zo.val[1]),b8q),
- vsubq_s8(vreinterpretq_s8_u8(zo1.val[1]),b8q),
- vld1q_s8(xs+i+16), vld1q_s8(xs1+i+16));
- a2=mm_tile16(a2, vsubq_s8(vreinterpretq_s8_u8(ko.val[0]),b8q),
- vsubq_s8(vreinterpretq_s8_u8(ko1.val[0]),b8q),
- vld1q_s8(xs+i+32), vld1q_s8(xs1+i+32));
- a3=mm_tile16(a3, vsubq_s8(vreinterpretq_s8_u8(ko.val[1]),b8q),
- vsubq_s8(vreinterpretq_s8_u8(ko1.val[1]),b8q),
- vld1q_s8(xs+i+48), vld1q_s8(xs1+i+48));
- }
- for(;i+32<=I;i+=32){
- uint8x16_t byo=vld1q_u8(wo+(i>>1)), byo1=vld1q_u8(wo1+(i>>1));
- uint8x16x2_t zo =vzipq_u8(vandq_u8(byo, m4q), vshrq_n_u8(byo, 4));
- uint8x16x2_t zo1=vzipq_u8(vandq_u8(byo1,m4q), vshrq_n_u8(byo1,4));
- a0=mm_tile16(a0, vsubq_s8(vreinterpretq_s8_u8(zo.val[0]),b8q),
- vsubq_s8(vreinterpretq_s8_u8(zo1.val[0]),b8q),
- vld1q_s8(xs+i), vld1q_s8(xs1+i));
- a1=mm_tile16(a1, vsubq_s8(vreinterpretq_s8_u8(zo.val[1]),b8q),
- vsubq_s8(vreinterpretq_s8_u8(zo1.val[1]),b8q),
- vld1q_s8(xs+i+16), vld1q_s8(xs1+i+16));
- }
- int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));
- int32_t d00=vgetq_lane_s32(acc,0), d01=vgetq_lane_s32(acc,1);
- int32_t d10=vgetq_lane_s32(acc,2), d11=vgetq_lane_s32(acc,3);
- for(;i+1<I;i+=2){ uint8_t bo=wo[i>>1], bo1=wo1[i>>1];
- int a0=(int)(bo&0xF)-8, a1=(int)(bo>>4)-8, b0=(int)(bo1&0xF)-8, b1=(int)(bo1>>4)-8;
- int u0=xs[i],u1=xs[i+1],v0=xs1[i],v1=xs1[i+1];
- d00+=a0*u0+a1*u1; d01+=a0*v0+a1*v1; d10+=b0*u0+b1*u1; d11+=b0*v0+b1*v1; }
- if(i<I){ uint8_t bo=wo[i>>1], bo1=wo1[i>>1];
- int a0=(int)(bo&0xF)-8, b0=(int)(bo1&0xF)-8;
- d00+=a0*xs[i]; d01+=a0*xs1[i]; d10+=b0*xs[i]; d11+=b0*xs1[i]; }
- y[(int64_t)s*O+o] =(float)d00*sc0*sx[s];
- y[(int64_t)s*O+(o+1)] =(float)d10*sc1*sx[s];
- y[(int64_t)(s+1)*O+o] =(float)d01*sc0*sx[s+1];
- y[(int64_t)(s+1)*O+(o+1)]=(float)d11*sc1*sx[s+1];
- }
- if(S&1){ int s=S-1; const int8_t *xs=xq+(int64_t)s*I;
- y[(int64_t)s*O+o] =(float)dot_i4i8(wo, xs,I)*sc0*sx[s];
- y[(int64_t)s*O+(o+1)]=(float)dot_i4i8(wo1,xs,I)*sc1*sx[s]; }
- }
- if(O&1){ int o=O-1; const uint8_t *w=q4+(int64_t)o*rb; float sc=scale[o];
- #pragma omp parallel for schedule(static)
- for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i4i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }
-}
-#endif
-
-/* ---- IDOT dispatch (int8-quantized activations) --------------------------- */
-static void matmul_q_idot(float *y, const int8_t *xq, const float *sx, const int8_t *q,
- const float *scale, int S, int I, int O){
-#if defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)
- if(S>=2){ matmul_q_idot_mm(y,xq,sx,q,scale,S,I,O); return; }
-#endif
- #pragma omp parallel for schedule(static)
- for(int o=0;o<O;o++){ const int8_t *w=q+(int64_t)o*I; float sc=scale[o];
- for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i8i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }
-}
-static void matmul_i4_idot(float *y, const int8_t *xq, const float *sx, const uint8_t *q4,
- const float *scale, int S, int I, int O){
- int rb=(I+1)/2;
-#if defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)
- if(S>=2){ matmul_i4_idot_mm(y,xq,sx,q4,scale,S,I,O); return; }
-#endif
- #pragma omp parallel for schedule(static)
- for(int o=0;o<O;o++){ const uint8_t *w=q4+(int64_t)o*rb; float sc=scale[o];
- for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i4i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }
-}
-
-
-/* ================= K1: layout int4 A PIANI (fmt 2/4, opzionale) ==============
- * Layout classico ("a coppie"): byte k = elementi (2k, 2k+1). Ogni kernel paga
- * unpacklo/unpackhi per riordinare i nibble, e l'IDOT paga sub+abs+sign per
- * forzare l'operando con segno dentro vpdpbusd (che e' nativamente u8 x s8).
- *
- * Layout A PIANI, per blocco di 64 elementi consecutivi (32 byte):
- * byte k del blocco = (elem k + 8) | ((elem k+32 + 8) << 4) k = 0..31
- * `and 0x0F` produce gli elementi 0..31 GIA' in ordine; `srli 4 + and` produce
- * 32..63 in ordine: l'unpack sparisce. La coda (I mod 64) resta a coppie.
- *
- * L'algebra dell'IDOT: i nibble memorizzati sono u = v+8 (unsigned 0..15), e
- * dot(v, x) = dot(u, x) - 8 * sum(x)
- * dove sum(x) e' UN int32 per riga di attivazione, condiviso da ogni riga di
- * output. vpdpbusd consuma (u8, s8) direttamente: via sub+abs+sign (3 op
- * vettoriali per dpbusd). Misurato su questa famiglia: 1.87x L3-resident,
- * 25.3 GB/s DRAM (al tetto), 0 differenze di bit.
- * NB: allargare a 256 bit SENZA cambiare layout e' stato misurato 0.79-0.81x
- * (il fixup di lane costa piu' della larghezza): il layout e' la precondizione
- * della larghezza, non un'ottimizzazione indipendente.
- *
- * I kernel f32 planari preservano ESATTAMENTE l'ordine di accumulazione dei
- * gemelli a coppie (stesse lane, stessa sequenza 0..63): bit-identici anche
- * loro. EN: plane-nibble int4 layout. Low nibble of block byte k = element k,
- * high = element k+32; the unpack disappears, and the stored-unsigned nibbles
- * feed vpdpbusd natively via dot(v,x) = dot(u,x) - 8*sum(x). The f32 planar
- * kernels keep the pair kernels' exact accumulation order: bit-identical. */
-
-/* riordina in place una riga (o un tensore [O,rb] riga per riga) da coppie a
- * piani; la coda I%64 resta a coppie. EN: in-place pair->planar repack. */
-static void planarize_i4_row(uint8_t *row, int I){
- uint8_t tmp[32];
- int nb=I/64;
- for(int b=0;b<nb;b++){
- uint8_t *blk=row+b*32;
- /* pair: byte j ha (elem 2j, elem 2j+1). planar: byte k = (elem k, elem k+32) */
- for(int k=0;k<32;k++){
- int src_lo=k, src_hi=k+32; /* elementi voluti */
- uint8_t nib_lo=(blk[src_lo>>1]>>((src_lo&1)*4))&0xF;
- uint8_t nib_hi=(blk[src_hi>>1]>>((src_hi&1)*4))&0xF;
- tmp[k]=(uint8_t)(nib_lo|(nib_hi<<4));
- }
- memcpy(blk,tmp,32);
- }
-}
-static void planarize_i4(uint8_t *q4, int O, int I){
- int rb=(I+1)/2;
- #pragma omp parallel for schedule(static)
- for(int o=0;o<O;o++) planarize_i4_row(q4+(int64_t)o*rb, I);
-}
-
-/* dot unsigned-nibble planare * int8: ritorna dot(u, x) SENZA la correzione
- * -8*sum(x) (la applica il chiamante, una volta per riga di output).
- * Coda I%64: nibble a coppie letti come unsigned, stessa identita'. */
-#if defined(__AVXVNNI__) || (defined(__AVX512VNNI__) && defined(__AVX512VL__))
-#if defined(__AVXVNNI__) && !defined(__AVX512VNNI__)
-#define coli_dpbusd256 _mm256_dpbusd_avx_epi32
-#else
-#define coli_dpbusd256 _mm256_dpbusd_epi32
-#endif
-#endif
-static inline int32_t dot_i4p_u(const uint8_t *w4, const int8_t *x, int I){
- int32_t sum=0; int i=0;
-#if defined(coli_dpbusd256)
- const __m256i m4=_mm256_set1_epi8(0x0F);
- __m256i a0=_mm256_setzero_si256(),a1=_mm256_setzero_si256();
- __m256i a2=_mm256_setzero_si256(),a3=_mm256_setzero_si256();
- for(;i+128<=I;i+=128){
- __m256i b0=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));
- __m256i b1=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)+32));
- a0=coli_dpbusd256(a0,_mm256_and_si256(b0,m4),
- _mm256_loadu_si256((const __m256i*)(x+i)));
- a1=coli_dpbusd256(a1,_mm256_and_si256(_mm256_srli_epi16(b0,4),m4),
- _mm256_loadu_si256((const __m256i*)(x+i+32)));
- a2=coli_dpbusd256(a2,_mm256_and_si256(b1,m4),
- _mm256_loadu_si256((const __m256i*)(x+i+64)));
- a3=coli_dpbusd256(a3,_mm256_and_si256(_mm256_srli_epi16(b1,4),m4),
- _mm256_loadu_si256((const __m256i*)(x+i+96)));
- }
- __m256i acc=_mm256_add_epi32(_mm256_add_epi32(a0,a1),_mm256_add_epi32(a2,a3));
- for(;i+64<=I;i+=64){
- __m256i b0=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));
- acc=coli_dpbusd256(acc,_mm256_and_si256(b0,m4),
- _mm256_loadu_si256((const __m256i*)(x+i)));
- acc=coli_dpbusd256(acc,_mm256_and_si256(_mm256_srli_epi16(b0,4),m4),
- _mm256_loadu_si256((const __m256i*)(x+i+32)));
- }
- sum=hsum256_i32(acc);
-#elif defined(__AVX2__)
- const __m256i m4=_mm256_set1_epi8(0x0F);
- const __m256i ones=_mm256_set1_epi16(1);
- __m256i acc=_mm256_setzero_si256();
- for(;i+64<=I;i+=64){
- __m256i b0=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));
- /* maddubs(u8, s8): u<=15, |x|<=127 -> coppia <= 3810, int16 sicuro */
- __m256i p0=_mm256_maddubs_epi16(_mm256_and_si256(b0,m4),
- _mm256_loadu_si256((const __m256i*)(x+i)));
- __m256i p1=_mm256_maddubs_epi16(_mm256_and_si256(_mm256_srli_epi16(b0,4),m4),
- _mm256_loadu_si256((const __m256i*)(x+i+32)));
- acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p0,ones));
- acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p1,ones));
- }
- sum=hsum256_i32(acc);
-#elif defined(__ARM_NEON)
- int32x4_t acc=vdupq_n_s32(0);
- for(;i+64<=I;i+=64){
- uint8x16_t b0=vld1q_u8(w4+(i>>1)), b1=vld1q_u8(w4+(i>>1)+16);
- int8x16_t lo0=vreinterpretq_s8_u8(vandq_u8(b0,vdupq_n_u8(0x0F)));
- int8x16_t lo1=vreinterpretq_s8_u8(vandq_u8(b1,vdupq_n_u8(0x0F)));
- int8x16_t hi0=vreinterpretq_s8_u8(vshrq_n_u8(b0,4));
- int8x16_t hi1=vreinterpretq_s8_u8(vshrq_n_u8(b1,4));
-#if defined(__ARM_FEATURE_DOTPROD)
- acc=vdotq_s32(acc,lo0,vld1q_s8(x+i));
- acc=vdotq_s32(acc,lo1,vld1q_s8(x+i+16));
- acc=vdotq_s32(acc,hi0,vld1q_s8(x+i+32));
- acc=vdotq_s32(acc,hi1,vld1q_s8(x+i+48));
-#else
- int8x16_t xs0=vld1q_s8(x+i), xs1=vld1q_s8(x+i+16);
- int8x16_t xs2=vld1q_s8(x+i+32), xs3=vld1q_s8(x+i+48);
- int16x8_t m;
- m=vmull_s8(vget_low_s8(lo0),vget_low_s8(xs0)); acc=vpadalq_s16(acc,m);
- m=vmull_s8(vget_high_s8(lo0),vget_high_s8(xs0)); acc=vpadalq_s16(acc,m);
- m=vmull_s8(vget_low_s8(lo1),vget_low_s8(xs1)); acc=vpadalq_s16(acc,m);
- m=vmull_s8(vget_high_s8(lo1),vget_high_s8(xs1)); acc=vpadalq_s16(acc,m);
- m=vmull_s8(vget_low_s8(hi0),vget_low_s8(xs2)); acc=vpadalq_s16(acc,m);
- m=vmull_s8(vget_high_s8(hi0),vget_high_s8(xs2)); acc=vpadalq_s16(acc,m);
- m=vmull_s8(vget_low_s8(hi1),vget_low_s8(xs3)); acc=vpadalq_s16(acc,m);
- m=vmull_s8(vget_high_s8(hi1),vget_high_s8(xs3)); acc=vpadalq_s16(acc,m);
-#endif
- }
- sum=vaddvq_s32(acc);
-#endif
- for(;i+64<=I;i+=64){ /* fallback scalare sui blocchi planari */
- const uint8_t *blk=w4+(i>>1);
- for(int k=0;k<32;k++){
- sum+=(int32_t)(blk[k]&0xF)*x[i+k];
- sum+=(int32_t)(blk[k]>>4)*x[i+k+32];
- }
- }
- for(;i<I;i+=2){ /* coda a coppie, unsigned (u=v+8 memorizzato) */
- uint8_t byte=w4[i>>1];
- sum+=(int32_t)(byte&0xF)*x[i];
- if(i+1<I) sum+=(int32_t)(byte>>4)*x[i+1];
- }
- return sum;
-}
-
-/* ---- K1b (OPT-IN, IDOT_GS=1): IDOT planare A GRUPPI (fmt=4, gs%64==0) -----
- * Con gs=64 il gruppo di scala COINCIDE col blocco-piano da 64 elementi: il
- * dot unsigned del blocco (2 dpbusd) -> int32 di gruppo, meno 8*somma(x) del
- * gruppo, per la scala f32 del gruppo. Attivazioni int8 (stessa famiglia
- * qrow_i8 del resto dell'IDOT): NON bit-identico al kernel f32 a gruppi --
- * per questo e' dietro flag, in attesa dell'ablazione. xsg = somme int32
- * per (riga, gruppo), calcolate dal chiamante in una passata esatta.
- * EN: grouped planar IDOT, opt-in. With gs=64 the scale group IS the plane
- * block; per-group unsigned dot minus 8*group-sum, times the group scale.
- * int8 activations: not bit-identical to the f32 grouped kernel, hence the
- * flag until the ablation blesses a default.
- *
- * Three execution shapes below, one contract: per (row, output) the float
- * accumulation is the SAME sequence of fmaf((float)group_int, scale[g], a)
- * in ascending g, and every group_int is an exact int32 β€” so the per-row
- * path, the 1x4 row tile, and the AMX tile are bit-identical to each other
- * and to the pure-C reference on every ISA. */
-
-/* one planar 64-element block, unpacked once and shared across a row tile:
- * lo nibbles = elements base..base+31 in order, hi = base+32..base+63. */
-#if defined(coli_dpbusd256)
-static inline void i4p_blk256(const uint8_t *blk, __m256i *lo, __m256i *hi){
- const __m256i m4=_mm256_set1_epi8(0x0F);
- __m256i b=_mm256_loadu_si256((const __m256i*)blk);
- *lo=_mm256_and_si256(b,m4);
- *hi=_mm256_and_si256(_mm256_srli_epi16(b,4),m4);
-}
-#endif
-#if defined(__AVX512VNNI__) && defined(__AVX512BW__)
-/* whole block in one zmm: lanes 0..31 = lo elements, 32..63 = hi β€” matches a
- * straight 64-byte load of the activation block, so ONE dpbusd per block. */
-static inline __m512i i4p_blk512(const uint8_t *blk){
- const __m256i m4=_mm256_set1_epi8(0x0F);
- __m256i b=_mm256_loadu_si256((const __m256i*)blk);
- return _mm512_inserti64x4(_mm512_castsi256_si512(_mm256_and_si256(b,m4)),
- _mm256_and_si256(_mm256_srli_epi16(b,4),m4),1);
-}
-#endif
-
-/* ---- K1c: AMX int8 tile kernel (Sapphire Rapids+, opt-in via the same
- * IDOT_GS=1 family gate; AMX=0 kills it, AMX_S_MIN sets the row threshold).
- * With gs a multiple of 64, K=64 tile-multiplies cover a scale group exactly:
- * B tile = 16 output rows' int4 block unpacked once to SIGNED int8 (v-8, so
- * tdpbssd returns d - 8*sum(x_group) directly β€” the same int32 the vector
- * path computes as d_unsigned - 8*xsg), A tile = up to 16 activation rows.
- * The unpack cost is paid once per (output tile, group) and amortized over
- * every activation row β€” the multi-row reuse the pair-layout kernels lack.
- * Linux-only arming: tile data needs an ARCH_REQ_XCOMP_PERM handshake. */
-#if defined(__AMX_INT8__) && defined(__AMX_TILE__) && defined(__AVX512F__)
-#define COLI_HAVE_AMX_I4P 1
-#if defined(__linux__)
-#include <unistd.h>
-#include <sys/syscall.h>
-#elif defined(_WIN32)
-#ifndef WIN32_LEAN_AND_MEAN
-#define WIN32_LEAN_AND_MEAN
-#endif
-#include <windows.h>
-#endif
-static int coli_amx_state=-1;
-static int coli_amx_s_min=8;
-static int coli_amx_ok(void){
- if(coli_amx_state<0){
- const char *e=getenv("AMX");
- const char *sm=getenv("AMX_S_MIN"); if(sm&&atoi(sm)>0) coli_amx_s_min=atoi(sm);
-#if defined(__linux__)
- /* ARCH_REQ_XCOMP_PERM(XFEATURE_XTILEDATA): kernel >= 5.16 grants tile
- * state per-process; without it the first tile op SIGILLs. */
- coli_amx_state=(e&&*e=='0')?0:(syscall(SYS_arch_prctl,0x1023,18)==0);
-#elif defined(_WIN32)
- /* Windows 11: tile data is an opt-in per-process XState feature.
- * Dynamic lookup keeps older kernels building/running (arming just
- * fails closed there). XSTATE_AMX_TILE_DATA = 18. */
- if(e&&*e=='0') coli_amx_state=0;
- else {
- HMODULE k32=GetModuleHandleA("kernel32.dll");
- typedef BOOL (WINAPI *coli_xstate_fn)(ULONG64);
- coli_xstate_fn f=k32?(coli_xstate_fn)(void*)GetProcAddress(k32,"EnableProcessOptionalXStateFeatures"):NULL;
- coli_amx_state=(f && f(1ull<<18))?1:0;
- }
-#else
- (void)e; coli_amx_state=0;
-#endif
- if(coli_amx_state)
- fprintf(stderr,"[K1c] AMX int8 tile kernel armed for gs64 tensors "
- "(AMX=0 disables, engages at S>=%d)\n",coli_amx_s_min);
- }
- return coli_amx_state;
-}
-/* ldtilecfg layout (palette 1): tmm0=C [rows x 16 i32], tmm1=A [rows x 64 i8],
- * tmm2=B [16 x 64 i8 VNNI]. rows<16 reconfigures for the last partial s-tile. */
-struct coli_tilecfg { uint8_t palette,start_row,rsvd[14]; uint16_t colsb[16]; uint8_t rows[16]; };
-static void coli_amx_cfg(int arows){
- struct coli_tilecfg c; memset(&c,0,sizeof c); c.palette=1;
- c.rows[0]=(uint8_t)arows; c.colsb[0]=64;
- c.rows[1]=(uint8_t)arows; c.colsb[1]=64;
- c.rows[2]=16; c.colsb[2]=64;
- _tile_loadconfig(&c);
-}
-static void matmul_i4p_gidot_amx(float *y, const int8_t *xq, const float *sx,
- const uint8_t *q4, const float *scale,
- int S, int I, int O16, int O, int gs){
- int rb=(I+1)/2, ng=(I+gs-1)/gs, bpg=gs/64;
- #pragma omp parallel
- {
- float *acc=malloc((size_t)S*16*sizeof(float));
- if(!acc){ fprintf(stderr,"OOM: amx acc\n"); exit(1); }
- int8_t bstage[4*1024] __attribute__((aligned(64))); /* bpg<=4 gated below */
- int32_t cbuf[16*16] __attribute__((aligned(64)));
- float sclT[16];
- int cur=16; coli_amx_cfg(16);
- #pragma omp for schedule(static)
- for(int ot=0; ot<O16; ot+=16){
- memset(acc,0,(size_t)S*16*sizeof(float));
- for(int g=0; g<ng; g++){
- for(int j=0;j<16;j++) sclT[j]=scale[(int64_t)(ot+j)*ng+g];
- for(int b=0;b<bpg;b++){
- /* B tile, VNNI [k/4][n][4]: element k of output row ot+n
- * lands at row k/4, byte n*4+k%4 β€” signed (v-8) at unpack. */
- int8_t *dst=bstage+(size_t)b*1024;
- int64_t boff=((int64_t)g*gs+b*64)>>1;
- for(int n=0;n<16;n++){
- const uint8_t *blk=q4+(int64_t)(ot+n)*rb+boff;
- for(int k=0;k<32;k++){
- dst[(k>>2)*64+n*4+(k&3)] =(int8_t)((blk[k]&0xF)-8);
- dst[((k+32)>>2)*64+n*4+((k+32)&3)]=(int8_t)((blk[k]>>4)-8);
- }
- }
- }
- for(int st=0; st<S; st+=16){
- int rows=S-st<16?S-st:16;
- if(rows!=cur){ coli_amx_cfg(rows); cur=rows; }
- _tile_zero(0);
- for(int b=0;b<bpg;b++){
- _tile_loadd(1, xq+(int64_t)st*I+(int64_t)g*gs+b*64, (size_t)I);
- _tile_loadd(2, bstage+(size_t)b*1024, 64);
- _tile_dpbssd(0,1,2);
- }
- _tile_stored(0, cbuf, 64);
- /* per-lane fmadd == the vector path's per-(s,o) scalar fmaf:
- * same values, same ascending-g order, single rounding. */
- for(int r=0;r<rows;r++){
- __m512 cv=_mm512_cvtepi32_ps(_mm512_load_si512((const void*)(cbuf+(size_t)r*16)));
- __m512 av=_mm512_loadu_ps(acc+(int64_t)(st+r)*16);
- _mm512_storeu_ps(acc+(int64_t)(st+r)*16,
- _mm512_fmadd_ps(cv,_mm512_loadu_ps(sclT),av));
- }
- }
- }
- for(int s=0;s<S;s++)
- _mm512_storeu_ps(y+(int64_t)s*O+ot,
- _mm512_mul_ps(_mm512_loadu_ps(acc+(int64_t)s*16),
- _mm512_set1_ps(sx[s])));
- }
- _tile_release();
- free(acc);
- }
-}
-#endif
-
-/* vector body over an output range [o0,o1): the 1x4 row tile pays the weight
- * block's load+mask once per 4 activation rows (the fmt=2 K2 tile's idea,
- * brought to the grouped family β€” the prefill batch-union and the serve mux
- * deliver exactly these multi-row calls). Integer group dots in any lane
- * order are exact, and each row keeps its own ascending-g fmaf chain. */
-static void i4p_gidot_rows(float *y, const int8_t *xq, const float *sx,
- const int32_t *xsg, const uint8_t *q4,
- const float *scale, int S, int I, int O, int gs,
- int o0, int o1){
- int rb=(I+1)/2, ng=(I+gs-1)/gs, bpg=gs/64; /* blocchi-piano per gruppo */
- #pragma omp parallel for schedule(static)
- for(int o=o0;o<o1;o++){
- const uint8_t *w=q4+(int64_t)o*rb;
- const float *scl=scale+(int64_t)o*ng;
- int s=0;
- for(; s+4<=S; s+=4){
- const int8_t *x0=xq +(int64_t)s*I, *x1=x0+I, *x2=x1+I, *x3=x2+I;
- const int32_t *g0=xsg+(int64_t)s*ng, *g1=g0+ng, *g2=g1+ng, *g3=g2+ng;
- float a0=0,a1=0,a2=0,a3=0;
- int g=0;
- for(; (g+1)*gs<=I; g++){
- int32_t d0,d1,d2,d3;
-#if defined(__AVX512VNNI__) && defined(__AVX512BW__)
- __m512i A0=_mm512_setzero_si512(),A1=_mm512_setzero_si512();
- __m512i A2=_mm512_setzero_si512(),A3=_mm512_setzero_si512();
- for(int b=0;b<bpg;b++){
- int base=g*gs+b*64;
- __m512i wz=i4p_blk512(w+(base>>1));
- A0=_mm512_dpbusd_epi32(A0,wz,_mm512_loadu_si512((const void*)(x0+base)));
- A1=_mm512_dpbusd_epi32(A1,wz,_mm512_loadu_si512((const void*)(x1+base)));
- A2=_mm512_dpbusd_epi32(A2,wz,_mm512_loadu_si512((const void*)(x2+base)));
- A3=_mm512_dpbusd_epi32(A3,wz,_mm512_loadu_si512((const void*)(x3+base)));
- }
- d0=_mm512_reduce_add_epi32(A0); d1=_mm512_reduce_add_epi32(A1);
- d2=_mm512_reduce_add_epi32(A2); d3=_mm512_reduce_add_epi32(A3);
-#elif defined(coli_dpbusd256)
- __m256i A0=_mm256_setzero_si256(),A1=_mm256_setzero_si256();
- __m256i A2=_mm256_setzero_si256(),A3=_mm256_setzero_si256();
- for(int b=0;b<bpg;b++){
- int base=g*gs+b*64; __m256i lo,hi;
- i4p_blk256(w+(base>>1),&lo,&hi);
- A0=coli_dpbusd256(A0,lo,_mm256_loadu_si256((const __m256i*)(x0+base)));
- A0=coli_dpbusd256(A0,hi,_mm256_loadu_si256((const __m256i*)(x0+base+32)));
- A1=coli_dpbusd256(A1,lo,_mm256_loadu_si256((const __m256i*)(x1+base)));
- A1=coli_dpbusd256(A1,hi,_mm256_loadu_si256((const __m256i*)(x1+base+32)));
- A2=coli_dpbusd256(A2,lo,_mm256_loadu_si256((const __m256i*)(x2+base)));
- A2=coli_dpbusd256(A2,hi,_mm256_loadu_si256((const __m256i*)(x2+base+32)));
- A3=coli_dpbusd256(A3,lo,_mm256_loadu_si256((const __m256i*)(x3+base)));
- A3=coli_dpbusd256(A3,hi,_mm256_loadu_si256((const __m256i*)(x3+base+32)));
- }
- d0=hsum256_i32(A0); d1=hsum256_i32(A1);
- d2=hsum256_i32(A2); d3=hsum256_i32(A3);
-#elif defined(__AVX2__)
- const __m256i ones=_mm256_set1_epi16(1);
- const __m256i m4=_mm256_set1_epi8(0x0F);
- __m256i A0=_mm256_setzero_si256(),A1=_mm256_setzero_si256();
- __m256i A2=_mm256_setzero_si256(),A3=_mm256_setzero_si256();
- for(int b=0;b<bpg;b++){
- int base=g*gs+b*64;
- __m256i bb=_mm256_loadu_si256((const __m256i*)(w+(base>>1)));
- __m256i lo=_mm256_and_si256(bb,m4);
- __m256i hi=_mm256_and_si256(_mm256_srli_epi16(bb,4),m4);
- /* maddubs(u8,s8): u<=15, |x|<=127 -> pair <= 3810, int16-safe */
- A0=_mm256_add_epi32(A0,_mm256_madd_epi16(_mm256_maddubs_epi16(lo,_mm256_loadu_si256((const __m256i*)(x0+base))),ones));
- A0=_mm256_add_epi32(A0,_mm256_madd_epi16(_mm256_maddubs_epi16(hi,_mm256_loadu_si256((const __m256i*)(x0+base+32))),ones));
- A1=_mm256_add_epi32(A1,_mm256_madd_epi16(_mm256_maddubs_epi16(lo,_mm256_loadu_si256((const __m256i*)(x1+base))),ones));
- A1=_mm256_add_epi32(A1,_mm256_madd_epi16(_mm256_maddubs_epi16(hi,_mm256_loadu_si256((const __m256i*)(x1+base+32))),ones));
- A2=_mm256_add_epi32(A2,_mm256_madd_epi16(_mm256_maddubs_epi16(lo,_mm256_loadu_si256((const __m256i*)(x2+base))),ones));
- A2=_mm256_add_epi32(A2,_mm256_madd_epi16(_mm256_maddubs_epi16(hi,_mm256_loadu_si256((const __m256i*)(x2+base+32))),ones));
- A3=_mm256_add_epi32(A3,_mm256_madd_epi16(_mm256_maddubs_epi16(lo,_mm256_loadu_si256((const __m256i*)(x3+base))),ones));
- A3=_mm256_add_epi32(A3,_mm256_madd_epi16(_mm256_maddubs_epi16(hi,_mm256_loadu_si256((const __m256i*)(x3+base+32))),ones));
- }
- d0=hsum256_i32(A0); d1=hsum256_i32(A1);
- d2=hsum256_i32(A2); d3=hsum256_i32(A3);
-#else
- d0=d1=d2=d3=0;
- for(int b=0;b<bpg;b++){
- int base=g*gs+b*64;
- const uint8_t *blk=w+(base>>1);
- for(int k=0;k<32;k++){
- int32_t ul=(int32_t)(blk[k]&0xF), uh=(int32_t)(blk[k]>>4);
- d0+=ul*x0[base+k]+uh*x0[base+k+32];
- d1+=ul*x1[base+k]+uh*x1[base+k+32];
- d2+=ul*x2[base+k]+uh*x2[base+k+32];
- d3+=ul*x3[base+k]+uh*x3[base+k+32];
- }
- }
-#endif
- a0=fmaf((float)(d0-8*g0[g]),scl[g],a0);
- a1=fmaf((float)(d1-8*g1[g]),scl[g],a1);
- a2=fmaf((float)(d2-8*g2[g]),scl[g],a2);
- a3=fmaf((float)(d3-8*g3[g]),scl[g],a3);
- }
- if(g*gs<I){ /* coda: gruppo parziale, nibble a coppie */
- int32_t d0=0,d1=0,d2=0,d3=0;
- for(int i=g*gs;i<I;i++){
- uint8_t byte=w[i>>1];
- int32_t u=(int32_t)((i&1)?(byte>>4):(byte&0xF));
- d0+=u*x0[i]; d1+=u*x1[i]; d2+=u*x2[i]; d3+=u*x3[i];
- }
- a0=fmaf((float)(d0-8*g0[g]),scl[g],a0);
- a1=fmaf((float)(d1-8*g1[g]),scl[g],a1);
- a2=fmaf((float)(d2-8*g2[g]),scl[g],a2);
- a3=fmaf((float)(d3-8*g3[g]),scl[g],a3);
- }
- y[(int64_t)s*O+o] =a0*sx[s];
- y[(int64_t)(s+1)*O+o]=a1*sx[s+1];
- y[(int64_t)(s+2)*O+o]=a2*sx[s+2];
- y[(int64_t)(s+3)*O+o]=a3*sx[s+3];
- }
- for(; s<S; s++){
- const int8_t *xr=xq+(int64_t)s*I;
- const int32_t *xg=xsg+(int64_t)s*ng;
- float a=0; int g=0;
- for(; (g+1)*gs<=I; g++){ /* gruppi interi */
- int32_t d=0;
-#if defined(__AVX512VNNI__) && defined(__AVX512BW__)
- __m512i acc=_mm512_setzero_si512();
- for(int b=0;b<bpg;b++){
- int base=g*gs+b*64;
- acc=_mm512_dpbusd_epi32(acc,i4p_blk512(w+(base>>1)),
- _mm512_loadu_si512((const void*)(xr+base)));
- }
- d=_mm512_reduce_add_epi32(acc);
-#else
- for(int b=0;b<bpg;b++){
- int base=g*gs+b*64;
- const uint8_t *blk=w+(base>>1);
- const int8_t *xb=xr+base;
-#if defined(coli_dpbusd256)
- __m256i lo,hi; i4p_blk256(blk,&lo,&hi);
- __m256i acc=_mm256_setzero_si256();
- acc=coli_dpbusd256(acc,lo,_mm256_loadu_si256((const __m256i*)xb));
- acc=coli_dpbusd256(acc,hi,_mm256_loadu_si256((const __m256i*)(xb+32)));
- d+=hsum256_i32(acc);
-#elif defined(__AVX2__)
- const __m256i m4=_mm256_set1_epi8(0x0F);
- const __m256i ones=_mm256_set1_epi16(1);
- __m256i bb=_mm256_loadu_si256((const __m256i*)blk);
- __m256i p0=_mm256_maddubs_epi16(_mm256_and_si256(bb,m4),
- _mm256_loadu_si256((const __m256i*)xb));
- __m256i p1=_mm256_maddubs_epi16(_mm256_and_si256(_mm256_srli_epi16(bb,4),m4),
- _mm256_loadu_si256((const __m256i*)(xb+32)));
- __m256i acc=_mm256_add_epi32(_mm256_madd_epi16(p0,ones),
- _mm256_madd_epi16(p1,ones));
- d+=hsum256_i32(acc);
-#else
- for(int k=0;k<32;k++){
- d+=(int32_t)(blk[k]&0xF)*xb[k];
- d+=(int32_t)(blk[k]>>4)*xb[k+32];
- }
-#endif
- }
-#endif
- a=fmaf((float)(d-8*xg[g]),scl[g],a);
- }
- if(g*gs<I){ /* coda: gruppo parziale, nibble a coppie */
- int32_t d=0;
- for(int i=g*gs;i<I;i++){
- uint8_t byte=w[i>>1];
- d+=(int32_t)((i&1)?(byte>>4):(byte&0xF))*xr[i];
- }
- a=fmaf((float)(d-8*xg[g]),scl[g],a);
- }
- y[(int64_t)s*O+o]=a*sx[s];
- }
- }
-}
-
-static void matmul_i4p_grouped_idot(float *y, const int8_t *xq, const float *sx,
- const int32_t *xsg, const uint8_t *q4,
- const float *scale, int S, int I, int O, int gs){
-#if defined(COLI_HAVE_AMX_I4P)
- /* AMX takes the aligned bulk (full 16-output tiles, full groups only β€”
- * the real gs64 checkpoints have I%gs==0); the vector path finishes any
- * output remainder. Below the row threshold the B-tile unpack does not
- * amortize and the vector tile is the better kernel. */
- if(coli_amx_ok() && S>=coli_amx_s_min && gs%64==0 && gs<=256 && I%gs==0 && O>=16){
- int O16=O&~15;
- matmul_i4p_gidot_amx(y,xq,sx,q4,scale,S,I,O16,O,gs);
- if(O16<O) i4p_gidot_rows(y,xq,sx,xsg,q4,scale,S,I,O,gs,O16,O);
- return;
- }
-#endif
- i4p_gidot_rows(y,xq,sx,xsg,q4,scale,S,I,O,gs,0,O);
-}
-
-/* matmul IDOT planare (fmt=2): y = (dot_u - 8*xsum[s]) * scale[o] * sx[s].
- * Bit-identico a matmul_i4_idot: somme intere, identita' esatta. */
-static void matmul_i4p_idot(float *y, const int8_t *xq, const float *sx, const int32_t *xsum,
- const uint8_t *q4, const float *scale, int S, int I, int O){
- int rb=(I+1)/2;
- #pragma omp parallel for schedule(static)
- for(int o=0;o<O;o++){ const uint8_t *w=q4+(int64_t)o*rb; float sc=scale[o];
- int s=0;
-#if defined(coli_dpbusd256)
- /* K2: tile 1x4 sulla union β€” il blocco pesi (load + 2 and + srli) si
- * paga UNA volta per 4 righe di attivazione invece che per riga. La
- * union del prefill consegna nr=2..16 righe per expert: e' esattamente
- * la finestra che il per-riga serviva peggio. Somme intere ->
- * bit-identico al path per-riga per associativita'.
- * EN: 1x4 register tile β€” the weight block's load+mask cost is paid
- * once per 4 activation rows. Integer sums: bit-identical to the
- * per-row path by associativity. */
- const __m256i m4t=_mm256_set1_epi8(0x0F);
- for(;s+4<=S;s+=4){
- const int8_t *x0=xq+(int64_t)s*I, *x1=x0+I, *x2=x1+I, *x3=x2+I;
- __m256i a0=_mm256_setzero_si256(), a1=_mm256_setzero_si256();
- __m256i a2=_mm256_setzero_si256(), a3=_mm256_setzero_si256();
- int i=0;
- for(;i+64<=I;i+=64){
- __m256i b =_mm256_loadu_si256((const __m256i*)(w+(i>>1)));
- __m256i lo=_mm256_and_si256(b,m4t);
- __m256i hi=_mm256_and_si256(_mm256_srli_epi16(b,4),m4t);
- a0=coli_dpbusd256(a0,lo,_mm256_loadu_si256((const __m256i*)(x0+i)));
- a0=coli_dpbusd256(a0,hi,_mm256_loadu_si256((const __m256i*)(x0+i+32)));
- a1=coli_dpbusd256(a1,lo,_mm256_loadu_si256((const __m256i*)(x1+i)));
- a1=coli_dpbusd256(a1,hi,_mm256_loadu_si256((const __m256i*)(x1+i+32)));
- a2=coli_dpbusd256(a2,lo,_mm256_loadu_si256((const __m256i*)(x2+i)));
- a2=coli_dpbusd256(a2,hi,_mm256_loadu_si256((const __m256i*)(x2+i+32)));
- a3=coli_dpbusd256(a3,lo,_mm256_loadu_si256((const __m256i*)(x3+i)));
- a3=coli_dpbusd256(a3,hi,_mm256_loadu_si256((const __m256i*)(x3+i+32)));
- }
- int32_t d0=hsum256_i32(a0), d1=hsum256_i32(a1);
- int32_t d2=hsum256_i32(a2), d3=hsum256_i32(a3);
- /* coda a coppie, unsigned: stessa identita' -8*xsum del per-riga */
- for(;i<I;i+=2){
- uint8_t byte=w[i>>1];
- d0+=(int32_t)(byte&0xF)*x0[i]; d1+=(int32_t)(byte&0xF)*x1[i];
- d2+=(int32_t)(byte&0xF)*x2[i]; d3+=(int32_t)(byte&0xF)*x3[i];
- if(i+1<I){
- d0+=(int32_t)(byte>>4)*x0[i+1]; d1+=(int32_t)(byte>>4)*x1[i+1];
- d2+=(int32_t)(byte>>4)*x2[i+1]; d3+=(int32_t)(byte>>4)*x3[i+1];
- }
- }
- y[(int64_t)(s+0)*O+o]=(float)(d0-8*xsum[s+0])*sc*sx[s+0];
- y[(int64_t)(s+1)*O+o]=(float)(d1-8*xsum[s+1])*sc*sx[s+1];
- y[(int64_t)(s+2)*O+o]=(float)(d2-8*xsum[s+2])*sc*sx[s+2];
- y[(int64_t)(s+3)*O+o]=(float)(d3-8*xsum[s+3])*sc*sx[s+3];
- }
-#endif
- for(;s<S;s++){
- int32_t d=dot_i4p_u(w,xq+(int64_t)s*I,I)-8*xsum[s];
- y[(int64_t)s*O+o]=(float)d*sc*sx[s];
- }
- }
-}
/* f32 planare (fmt=2): stesso ordine di accumulazione di matmul_i4 (sequenza
* 0..63 in blocchi di 8 lane, stessa lane per elemento) -> bit-identico. */
diff --git a/c/qwen36.c b/c/qwen36.c
index 7a5de982..49ced244 100644
--- a/c/qwen36.c
+++ b/c/qwen36.c
@@ -67,6 +67,7 @@ static int qwen36_max_ctx(void) {
#include "json.h" /* tokenizer.json parsing (reuse minimal parser) */
#include "qwen36_tier.h" /* optional CUDA VRAM expert tier */
#include "expert_ffn.h" /* routed experts: planar int4 kernel + layer runner */
+#include "idot.h" /* integer dot kernels for the dense trunk (COLI_DENSE_IDOT, COLI_DENSE_BITS) */
#ifdef COLI_SEGMENT_ADAPTER
#include "segment_runtime.h"
#include "segment_adapters.h"
@@ -989,6 +990,12 @@ static int g_expert_is_int4 = 1;
* int8 copy) and with QWEN_EXPERT_KERNEL=0, which keeps the historical
* unpack-to-int8 path for A/Bs. Decided once from the container itself. */
static int container_layer_is_int4(Model *m, int layer);
+/* The routed experts take the same integer path as the dense trunk
+ * (expert_ffn.h mode 1: activation to int8 once per row, dpbusd against the
+ * planar nibbles). Measured on the 35B: expert compute 22.7 to 15.9
+ * ms/token for +0.1% perplexity. QWEN_EXPERT_ACT=f32 restores mode 0, f32
+ * activations and the bit-identical contract with the pair kernels. */
+static int xf_act_mode(void){ static int v=-1; if(v<0){ const char *e=getenv("QWEN_EXPERT_ACT"); v=(e&&!strcmp(e,"f32"))?0:1; } return v; }
static int xf_mode(Model *m) {
static int v = -1;
if (v >= 0) return v;
@@ -1037,8 +1044,131 @@ static void matmul_qe(float *y, const float *x, const int8_t *q, const float *sc
* matmul_d dispatches via pointer lookup to matmul_q; COLI_DENSE_I8=0 falls
* back to f32 (reference path for parity tests). ~4x less memory traffic. */
#define QDW_MAX 1024
-static struct { const float *w; int8_t *q; float *sc; int I, O; } g_qdw[QDW_MAX];
+/* q/sc: int8 rows with one scale per row (the classic copy, what the VRAM
+ * tier uploads). q4/sg: the same matrix as int4 planar blocks of 64 with one
+ * scale per group (COLI_DENSE_BITS=4), the layout the K1b grouped kernel
+ * reads; ng = I/64 groups per row. */
+static struct { const float *w; int8_t *q; float *sc; int I, O; uint8_t *q4; float *sg; int ng; } g_qdw[QDW_MAX];
static int g_qdw_n = 0;
+/* ---- Dense trunk, integer dot products ------------------------------------
+ *
+ * Every dense GEMV of a token (DeltaNet projections and out_proj, attention
+ * q/k/v/o, the shared expert, lm_head) used to multiply int8 weights by f32
+ * activations: each weight byte converted to f32 and fed to an FMA, eight
+ * weights per instruction. Measured on lm_head (248320 x 2048 int8, 508 MB)
+ * that runs at 29 GB/s on a 16-core AVX-512 host whose memory bus does 80:
+ * the kernel, not the bus, was the limit, and the dense part of a token is
+ * 1.9 GB of int8 on the 35B, three times the routed experts.
+ *
+ * The activation is now quantized to int8 once per call (one scale,
+ * amax/127, the qrow_i8 contract the expert IDOT already uses) and the dot
+ * is integer: 32 weights per instruction on AVX2 (maddubs), 64 on AVX-512
+ * VNNI, exact int32 sums scaled once per output. Not bit-identical to the
+ * f32 path (the activation is rounded): measured on the 35B, +1.0%
+ * perplexity on 4 x 512 tokens, lm_head 12.6 to 10.2 ms/token, decode
+ * 6.71 to 7.35 tok/s alone and 8.23 with the experts' int8 activations.
+ * COLI_DENSE_IDOT=0 restores the f32-activation kernel.
+ *
+ * COLI_DENSE_BITS=4 additionally stores the dense matrices as int4 in blocks
+ * of 64 with one scale per block, the K1b planar layout, halving the bytes
+ * the token reads; lm_head alone goes from 508 to 254 MB. It implies the
+ * integer dot (that layout has no f32 kernel). Same gate: measured. */
+static int dense_idot_on(void){ static int v=-1; if(v<0){ const char *e=getenv("COLI_DENSE_IDOT"); v=!(e&&*e=='0'); } return v; }
+static int dense_bits(void){ static int v=-1; if(v<0){ const char *e=getenv("COLI_DENSE_BITS"); v=(e&&atoi(e)==4)?4:8; } return v; }
+
+/* Activation -> int8 with one scale, plus the int32 sum of every block of 64
+ * (the K1b kernel subtracts 8*sum per group because its nibbles are unsigned).
+ * Vectorized: the scalar lrintf loop was measured at ~7 us for 4096 values,
+ * which times ~150 GEMVs per token is a millisecond thrown away. Rounding is
+ * to nearest even in both paths, so the vector path equals qrow_i8 bit for bit. */
+static float dense_act_i8(const float *x, int I, int8_t *xq, int32_t *xsg){
+ float amax = 0.f;
+ int i = 0;
+#ifdef __AVX2__
+ {
+ __m256 am = _mm256_setzero_ps();
+ const __m256 sign = _mm256_set1_ps(-0.0f);
+ for (; i + 8 <= I; i += 8) am = _mm256_max_ps(am, _mm256_andnot_ps(sign, _mm256_loadu_ps(x + i)));
+ float tmp[8]; _mm256_storeu_ps(tmp, am);
+ for (int k = 0; k < 8; k++) if (tmp[k] > amax) amax = tmp[k];
+ }
+#endif
+ for (; i < I; i++) { float a = fabsf(x[i]); if (a > amax) amax = a; }
+ float s = amax / 127.f; if (s < 1e-12f) s = 1e-12f;
+ float inv = 1.f / s;
+ i = 0;
+#ifdef __AVX2__
+ {
+ const __m256 vinv = _mm256_set1_ps(inv);
+ for (; i + 32 <= I; i += 32) {
+ __m256i a = _mm256_cvtps_epi32(_mm256_mul_ps(_mm256_loadu_ps(x + i), vinv));
+ __m256i b = _mm256_cvtps_epi32(_mm256_mul_ps(_mm256_loadu_ps(x + i + 8), vinv));
+ __m256i c = _mm256_cvtps_epi32(_mm256_mul_ps(_mm256_loadu_ps(x + i + 16), vinv));
+ __m256i d = _mm256_cvtps_epi32(_mm256_mul_ps(_mm256_loadu_ps(x + i + 24), vinv));
+ /* packs interleave 128-bit lanes: fix the order with one permute */
+ __m256i ab = _mm256_packs_epi32(a, b), cd = _mm256_packs_epi32(c, d);
+ __m256i abcd = _mm256_packs_epi16(ab, cd);
+ abcd = _mm256_permutevar8x32_epi32(abcd, _mm256_setr_epi32(0, 4, 1, 5, 2, 6, 3, 7));
+ _mm256_storeu_si256((__m256i *)(xq + i), abcd);
+ }
+ }
+#endif
+ for (; i < I; i++) xq[i] = (int8_t)lrintf(x[i] * inv);
+ if (xsg) {
+ int ng = I / 64;
+ for (int g = 0; g < ng; g++) {
+ int32_t sum = 0;
+ for (int k = 0; k < 64; k++) sum += xq[g * 64 + k];
+ xsg[g] = sum;
+ }
+ }
+ return s;
+}
+
+/* f32 rows -> int4 in blocks of 64 with one f32 scale per block, packed as the
+ * K1b planar layout (unsigned nibbles v+8, block b: lo nibbles = elements
+ * b*64..b*64+31, hi = b*64+32..b*64+63). The quantizer is the symmetric
+ * absmax/7 the expert containers use. */
+static void pack_int4_g64_planar(const float *w, uint8_t *q4, float *sg, int O, int I){
+ int rb = I / 2, ng = I / 64;
+ #pragma omp parallel for schedule(static)
+ for (int o = 0; o < O; o++) {
+ const float *wr = w + (int64_t)o * I;
+ uint8_t *row = q4 + (int64_t)o * rb;
+ float *sr = sg + (int64_t)o * ng;
+ for (int g = 0; g < ng; g++) {
+ const float *blk = wr + g * 64;
+ float amax = 0.f;
+ for (int k = 0; k < 64; k++) { float a = fabsf(blk[k]); if (a > amax) amax = a; }
+ /* absmax/7 is where the search starts; the scale is then refined
+ * by least squares against the rounded codes (s = w.q / q.q) and
+ * the candidate with the smallest squared error wins. Three
+ * rounds: measured on the 35B this recovers a third of the
+ * perplexity absmax alone loses at 4 bits. */
+ float best_s = amax / 7.f; if (best_s < 1e-8f) best_s = 1e-8f;
+ int best_q[64]; double best_err = 1e30;
+ float s = best_s;
+ for (int round = 0; round < 4; round++) {
+ int q[64]; double err = 0, wq = 0, qq = 0;
+ float inv = 1.f / s;
+ for (int k = 0; k < 64; k++) {
+ int v = (int)lrintf(blk[k] * inv); if (v > 7) v = 7; if (v < -8) v = -8;
+ q[k] = v; double d = (double)blk[k] - (double)v * s; err += d * d;
+ wq += (double)blk[k] * v; qq += (double)v * v;
+ }
+ if (err < best_err) { best_err = err; best_s = s; memcpy(best_q, q, sizeof q); }
+ if (qq <= 0) break;
+ float ns = (float)(wq / qq); /* least-squares scale for these codes */
+ if (ns <= 0.f || ns == s) break;
+ s = ns;
+ }
+ sr[g] = best_s;
+ uint8_t *dst = row + g * 32;
+ for (int k = 0; k < 32; k++)
+ dst[k] = (uint8_t)((best_q[k] + 8) | ((best_q[k + 32] + 8) << 4));
+ }
+ }
+}
#ifdef COLI_QWEN_BATCH_TEST
static uint64_t g_qwen_matmul_d_calls;
#endif
@@ -1048,7 +1178,27 @@ static int dense_i8_on(void){ static int v=-1; if(v<0){ const char *e=getenv("CO
* the time. */
static int kv_prefix_off(void){ const char *e=getenv("COLI_KV_PREFIX"); return e && *e=='0'; }
static int dense_batch_on(void){ const char *e=getenv("QWEN_DENSE_BATCH"); return !(e&&*e=='0'); }
-static void qdw_register(const float *W, int I, int O){
+/* COLI_DENSE_INT4 names which components take the int4 copy when
+ * COLI_DENSE_BITS=4: a comma list of lmhead, dnproj, dnout, attn, shexp,
+ * router; unset means all of them. The parts of the trunk pay 4 bits
+ * differently (measured: the attention projections and the DeltaNet input
+ * projections cost the most perplexity, lm_head the least per byte saved),
+ * so the default is chosen per component from the numbers, not for the
+ * whole trunk at once. */
+static int dense_int4_wanted(const char *tag){
+ if (dense_bits() != 4 || !tag) return 0;
+ const char *e = getenv("COLI_DENSE_INT4");
+ if (!e || !*e) return 1;
+ size_t n = strlen(tag);
+ for (const char *p = e; *p; ) {
+ while (*p == ',' || *p == ' ') p++;
+ const char *q = p; while (*q && *q != ',' && *q != ' ') q++;
+ if ((size_t)(q - p) == n && !strncmp(p, tag, n)) return 1;
+ p = q;
+ }
+ return 0;
+}
+static void qdw_register_as(const float *W, int I, int O, const char *tag){
if (!W || !dense_i8_on() || g_qdw_n >= QDW_MAX) return;
int8_t *q = malloc((size_t)O*I); float *sc = malloc((size_t)O*sizeof(float));
if (!q || !sc) { free(q); free(sc); return; }
@@ -1060,13 +1210,40 @@ static void qdw_register(const float *W, int I, int O){
int8_t *d = q + (int64_t)o*I;
for (int i = 0; i < I; i++) { int v = (int)lrintf(r[i]*inv); if (v>127) v=127; if (v<-127) v=-127; d[i] = (int8_t)v; }
}
- g_qdw[g_qdw_n].w=W; g_qdw[g_qdw_n].q=q; g_qdw[g_qdw_n].sc=sc; g_qdw[g_qdw_n].I=I; g_qdw[g_qdw_n].O=O; g_qdw_n++;
+ g_qdw[g_qdw_n].w=W; g_qdw[g_qdw_n].q=q; g_qdw[g_qdw_n].sc=sc; g_qdw[g_qdw_n].I=I; g_qdw[g_qdw_n].O=O;
+ g_qdw[g_qdw_n].q4=NULL; g_qdw[g_qdw_n].sg=NULL; g_qdw[g_qdw_n].ng=0;
+ if (dense_int4_wanted(tag) && I%64==0) {
+ uint8_t *q4=malloc((size_t)O*(I/2)); float *sg=malloc((size_t)O*(I/64)*sizeof(float));
+ if (q4 && sg) {
+ pack_int4_g64_planar(W, q4, sg, O, I);
+ g_qdw[g_qdw_n].q4=q4; g_qdw[g_qdw_n].sg=sg; g_qdw[g_qdw_n].ng=I/64;
+ } else { free(q4); free(sg); }
+ }
+ g_qdw_n++;
}
+static void qdw_register(const float *W, int I, int O){ qdw_register_as(W, I, O, NULL); }
static void matmul_d(float *y, const float *x, const float *W, int S, int I, int O){
#ifdef COLI_QWEN_BATCH_TEST
g_qwen_matmul_d_calls++;
#endif
for (int i = 0; i < g_qdw_n; i++) if (g_qdw[i].w == W && g_qdw[i].I == I) {
+ if (g_qdw[i].q4 || dense_idot_on()) {
+ /* integer dot: the activation rows to int8 once, then the K1b
+ * grouped kernel (int4 planar) or the per-row int8 kernel */
+ int ng = I / 64;
+ int8_t *xq = malloc((size_t)S * I);
+ float *sx = malloc((size_t)S * sizeof(float));
+ int32_t *xsg = g_qdw[i].q4 ? malloc((size_t)S * ng * sizeof(int32_t)) : NULL;
+ if (xq && sx && (!g_qdw[i].q4 || xsg)) {
+ for (int s = 0; s < S; s++)
+ sx[s] = dense_act_i8(x + (int64_t)s * I, I, xq + (int64_t)s * I, xsg ? xsg + (int64_t)s * ng : NULL);
+ if (g_qdw[i].q4) matmul_i4p_grouped_idot(y, xq, sx, xsg, g_qdw[i].q4, g_qdw[i].sg, S, I, O, 64);
+ else matmul_q_idot(y, xq, sx, g_qdw[i].q, g_qdw[i].sc, S, I, O);
+ free(xq); free(sx); free(xsg);
+ return;
+ }
+ free(xq); free(sx); free(xsg); /* out of memory: the f32 path below */
+ }
if (S > 1 && dense_batch_on())
matmul_q_batch(y, x, g_qdw[i].q, g_qdw[i].sc, S, I, O);
else
@@ -2023,9 +2200,9 @@ static void moe_xf_run(Model *m, int layer, const float *x, int S, float *out, c
exp[dst] = &ex[dst];
}
double t1 = timed ? tm_now() : 0;
- if (kper == K) xf_moe_run(out + (int64_t)s0 * D, x + (int64_t)s0 * D, per, K, D, F, ridx, rval, exp, 0, scratch);
+ if (kper == K) xf_moe_run(out + (int64_t)s0 * D, x + (int64_t)s0 * D, per, K, D, F, ridx, rval, exp, xf_act_mode(), scratch);
else {
- xf_moe_run(tmp, x + (int64_t)s0 * D, 1, 1, D, F, ridx, rval, exp, 0, scratch);
+ xf_moe_run(tmp, x + (int64_t)s0 * D, 1, 1, D, F, ridx, rval, exp, xf_act_mode(), scratch);
float *os = out + (int64_t)s0 * D; for (int d = 0; d < D; d++) os[d] += tmp[d];
}
if (timed) { double t2 = tm_now(); g_xf_load += t1 - t0; g_xf_run += t2 - t1; }
@@ -3630,16 +3807,16 @@ int main(int argc, char **argv) {
int q_out = qc->q_heads * qc->q_head_dim, kv_out = qc->kv_heads * qc->k_head_dim;
for (int i = 0; i < qc->n_layers; i++) {
Layer *l = &m.L[i];
- qdw_register(l->q, D2, q_out); qdw_register(l->k, D2, kv_out);
- qdw_register(l->v, D2, kv_out); qdw_register(l->o, qc->o_in, D2);
- qdw_register(l->gate, D2, qc->n_experts);
- qdw_register(l->sh_g, D2, qc->shared_inter); qdw_register(l->sh_u, D2, qc->shared_inter);
- qdw_register(l->sh_d, qc->shared_inter, D2);
- qdw_register(l->dn_qkv, D2, qc->dn_conv_dim);
- qdw_register(l->dn_z, D2, qc->dn_vheads * qc->dn_vdim);
- qdw_register(l->dn_out, qc->dn_vheads * qc->dn_vdim, D2);
+ qdw_register_as(l->q, D2, q_out, "attn"); qdw_register_as(l->k, D2, kv_out, "attn");
+ qdw_register_as(l->v, D2, kv_out, "attn"); qdw_register_as(l->o, qc->o_in, D2, "attn");
+ qdw_register_as(l->gate, D2, qc->n_experts, "router");
+ qdw_register_as(l->sh_g, D2, qc->shared_inter, "shexp"); qdw_register_as(l->sh_u, D2, qc->shared_inter, "shexp");
+ qdw_register_as(l->sh_d, qc->shared_inter, D2, "shexp");
+ qdw_register_as(l->dn_qkv, D2, qc->dn_conv_dim, "dnproj");
+ qdw_register_as(l->dn_z, D2, qc->dn_vheads * qc->dn_vdim, "dnproj");
+ qdw_register_as(l->dn_out, qc->dn_vheads * qc->dn_vdim, D2, "dnout");
}
- qdw_register(m.lm_head, D2, qc->vocab);
+ qdw_register_as(m.lm_head, D2, qc->vocab, "lmhead");
/* Free the f32 originals -- the pointers only serve as lookup keys in
* matmul_d from here on (never dereferenced again).
* COLI_KEEP_F32=1 keeps them (debug). */
diff --git a/c/tests/test_qwen36_dense_idot.c b/c/tests/test_qwen36_dense_idot.c
new file mode 100644
index 00000000..c9c2ef47
--- /dev/null
+++ b/c/tests/test_qwen36_dense_idot.c
@@ -0,0 +1,127 @@
+/* The dense trunk's integer path (COLI_DENSE_IDOT, COLI_DENSE_BITS=4).
+ *
+ * Pins, with no model:
+ * - dense_act_i8's vector path equals the scalar lrintf contract bit for bit
+ * (same rounding, same scale) and its block sums are exact;
+ * - pack_int4_g64_planar stores what the K1b layout expects: nibble v+8,
+ * lo nibbles = elements 0..31 of the block, hi = 32..63, one scale per 64;
+ * - matmul_d through the int4 grouped kernel and through the int8 integer
+ * kernel both agree with the f32 reference within what an int8 activation
+ * and an int4 weight allow, on every row of a random matrix, for one row
+ * (decode) and for several rows (prefill);
+ * - with neither flag the dispatch is the one before: the int8 f32-activation
+ * kernel, byte-identical to its own output. */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <math.h>
+
+#define main qwen36_main_unused
+#include "../qwen36.c"
+#undef main
+#include "../compat.h" /* setenv: MinGW has none */
+
+static int fails;
+static void ck(int ok, const char *what) { if (ok) { printf(" ok %s\n", what); return; } printf(" FAIL %s\n", what); fails++; }
+
+static unsigned g_seed = 4242;
+static float rnd(void) { g_seed = g_seed * 1103515245u + 12345u; return ((g_seed >> 8) & 0xFFFF) / 32768.f - 1.f; }
+
+static void ref_matmul(float *y, const float *x, const float *W, int S, int I, int O) {
+ for (int s = 0; s < S; s++) for (int o = 0; o < O; o++) {
+ double a = 0; for (int i = 0; i < I; i++) a += (double)x[(size_t)s * I + i] * W[(size_t)o * I + i];
+ y[(size_t)s * O + o] = (float)a;
+ }
+}
+static double rel_gap(const float *a, const float *b, int n) {
+ double worst = 0, scale = 1e-6;
+ for (int i = 0; i < n; i++) { double d = fabs((double)a[i] - b[i]); if (d > worst) worst = d; if (fabs((double)b[i]) > scale) scale = fabs((double)b[i]); }
+ return worst / scale;
+}
+
+int main(void) {
+ enum { I = 256, O = 96, S = 5 };
+ printf("activation quantizer\n");
+ {
+ float x[I]; int8_t a[I], b[I]; int32_t sums[I / 64];
+ for (int t = 0; t < 50; t++) {
+ float mag = (t % 5 == 0) ? 1e-3f : 3.f;
+ for (int i = 0; i < I; i++) x[i] = rnd() * mag;
+ float sa = dense_act_i8(x, I, a, sums);
+ float sb = qrow_i8(x, b, I);
+ if (sa != sb || memcmp(a, b, I)) { ck(0, "vector quantizer equals the scalar contract"); break; }
+ for (int g = 0; g < I / 64; g++) { int32_t s = 0; for (int k = 0; k < 64; k++) s += a[g * 64 + k]; if (s != sums[g]) { ck(0, "block sums are exact"); break; } }
+ if (t == 49) ck(1, "vector quantizer equals the scalar lrintf contract on 50 random rows, block sums exact");
+ }
+ }
+ printf("int4 planar packing\n");
+ {
+ float *W = malloc((size_t)O * I * sizeof(float));
+ for (size_t i = 0; i < (size_t)O * I; i++) W[i] = rnd();
+ uint8_t *q4 = malloc((size_t)O * I / 2); float *sg = malloc((size_t)O * (I / 64) * sizeof(float));
+ pack_int4_g64_planar(W, q4, sg, O, I);
+ int bad = 0;
+ for (int o = 0; o < O && !bad; o++) for (int g = 0; g < I / 64 && !bad; g++) {
+ const uint8_t *blk = q4 + (size_t)o * (I / 2) + g * 32;
+ float s = sg[(size_t)o * (I / 64) + g];
+ for (int k = 0; k < 64; k++) {
+ int nib = k < 32 ? (blk[k] & 0xF) : (blk[k - 32] >> 4);
+ float w = W[(size_t)o * I + g * 64 + k];
+ int want = (int)lrintf(w / s); if (want > 7) want = 7; if (want < -8) want = -8;
+ if (nib - 8 != want) { bad = 1; break; }
+ }
+ }
+ ck(!bad, "nibble v+8 in the planar order, one absmax/7 scale per block of 64");
+ free(W); free(q4); free(sg);
+ }
+ printf("matmul_d dispatch\n");
+ {
+ float *W = malloc((size_t)O * I * sizeof(float)), *x = malloc((size_t)S * I * sizeof(float));
+ float *ref = malloc((size_t)S * O * sizeof(float)), *y = malloc((size_t)S * O * sizeof(float)), *y0 = malloc((size_t)S * O * sizeof(float));
+ for (size_t i = 0; i < (size_t)O * I; i++) W[i] = rnd();
+ for (size_t i = 0; i < (size_t)S * I; i++) x[i] = rnd() * 2.f;
+ ref_matmul(ref, x, W, S, I, O);
+ /* the classic path (no flags): int8 rows, f32 activations */
+ setenv("COLI_DENSE_IDOT", "0", 1); setenv("COLI_DENSE_BITS", "8", 1);
+ qdw_register(W, I, O);
+ matmul_d(y0, x, W, S, I, O);
+ double g0 = rel_gap(y0, ref, S * O);
+ ck(g0 < 2e-2, "classic int8 path within 2% of the f32 reference (per-row int8)");
+ matmul_d(y, x, W, 1, I, O);
+ ck(!memcmp(y, y0, (size_t)O * sizeof(float)), "one row and the first of a batch agree on the classic path");
+ /* the integer path on the same int8 rows */
+ g_qdw_n = 0;
+ /* dense_idot_on caches its answer: this TU reads the env once, so set it before the first call */
+ }
+ {
+ /* fresh process-wide state for the flag readers: emulate by direct calls */
+ float *W = malloc((size_t)O * I * sizeof(float)), *x = malloc((size_t)S * I * sizeof(float));
+ float *ref = malloc((size_t)S * O * sizeof(float)), *y = malloc((size_t)S * O * sizeof(float));
+ for (size_t i = 0; i < (size_t)O * I; i++) W[i] = rnd();
+ for (size_t i = 0; i < (size_t)S * I; i++) x[i] = rnd() * 2.f;
+ ref_matmul(ref, x, W, S, I, O);
+ int8_t *q = malloc((size_t)O * I); float *sc = malloc((size_t)O * sizeof(float));
+ for (int o = 0; o < O; o++) { /* the per-row int8 the engine builds at load */
+ float am = 0.f; for (int i = 0; i < I; i++) { float a = fabsf(W[(size_t)o * I + i]); if (a > am) am = a; }
+ float s = am > 1e-12f ? am / 127.f : 1.f; sc[o] = s;
+ for (int i = 0; i < I; i++) { int v = (int)lrintf(W[(size_t)o * I + i] / s); if (v > 127) v = 127; if (v < -127) v = -127; q[(size_t)o * I + i] = (int8_t)v; }
+ }
+ int8_t *xq = malloc((size_t)S * I); float sx[S]; int32_t *xsg = malloc((size_t)S * (I / 64) * sizeof(int32_t));
+ for (int s = 0; s < S; s++) sx[s] = dense_act_i8(x + (size_t)s * I, I, xq + (size_t)s * I, xsg + (size_t)s * (I / 64));
+ matmul_q_idot(y, xq, sx, q, sc, S, I, O);
+ ck(rel_gap(y, ref, S * O) < 3e-2, "int8 weights x int8 activations within 3% of the f32 reference, 5 rows");
+ matmul_q_idot(y, xq, sx, q, sc, 1, I, O);
+ ck(rel_gap(y, ref, O) < 3e-2, "same for one row");
+ uint8_t *q4 = malloc((size_t)O * I / 2); float *sg = malloc((size_t)O * (I / 64) * sizeof(float));
+ pack_int4_g64_planar(W, q4, sg, O, I);
+ matmul_i4p_grouped_idot(y, xq, sx, xsg, q4, sg, S, I, O, 64);
+ ck(rel_gap(y, ref, S * O) < 8e-2, "int4 blocks of 64 x int8 activations within 8% of the f32 reference, 5 rows");
+ float y1[O];
+ matmul_i4p_grouped_idot(y1, xq, sx, xsg, q4, sg, 1, I, O, 64);
+ ck(!memcmp(y1, y, (size_t)O * sizeof(float)), "the one-row path and the row tile give the same bytes (the K1b contract)");
+ free(W); free(x); free(ref); free(y); free(q); free(sc); free(xq); free(xsg); free(q4); free(sg);
+ }
+ if (fails) { printf("test_qwen36_dense_idot: %d failure(s)\n", fails); return 1; }
+ printf("OK test_qwen36_dense_idot: the dense trunk's integer path\n");
+ return 0;
+}
diff --git a/c/tests/test_qwen36_trunk_dense.c b/c/tests/test_qwen36_trunk_dense.c
index 907f19a5..02a9e1a3 100644
--- a/c/tests/test_qwen36_trunk_dense.c
+++ b/c/tests/test_qwen36_trunk_dense.c
@@ -95,6 +95,10 @@ int main(void) {
setenv("COLI_PLACE", "", 1); /* "" == unset == auto */
setenv("CUDA_EXPERT_GB", "1", 1); /* room for everything */
unsetenv("COLI_DENSE_I8");
+ /* the GEMV from VRAM is compared with the CPU's f32-activation kernel, the
+ * contract the tier uploads: the integer dense path rounds the activation
+ * and is measured elsewhere (test_qwen36_dense_idot) */
+ setenv("COLI_DENSE_IDOT", "0", 1);
fake_ndev = 1; fake_uploads = 0; fake_dense_compute = 1;
Model m; build(&m);
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
index b32b2920..91969d00 100644
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@ -391,6 +391,10 @@ and the CPU/GPU execution split.
| Variable | Default | Effect |
|---|---|---|
| `COLI_DENSE_I8` | `1` (on) | Quantize resident dense matrices to per-row int8 at startup. `=0` keeps the f32 reference path for quality A/Bs. |
+| `COLI_DENSE_IDOT` | `1` (on) | The dense trunk's GEMVs (DeltaNet projections and out_proj, attention q/k/v/o, shared expert, lm_head) quantize the activation to int8 once per call and run integer dot products (maddubs on AVX2, vpdpbusd on AVX-VNNI / AVX-512 VNNI) instead of converting every int8 weight to f32. Not bit-identical to the f32 path; measured +1.0% perplexity, lm_head 12.6 to 10.2 ms/token. `=0` restores the f32-activation kernel. |
+| `QWEN_EXPERT_ACT` | `i8` | The routed experts' activation quantized to int8 once per row (expert_ffn.h mode 1). Measured +0.1% perplexity, expert compute 22.7 to 15.9 ms/token. `=f32` restores f32 activations and the bit-identical contract with the pair kernels. |
+| `COLI_DENSE_BITS` | `8` | `=4` stores the dense trunk as int4 in blocks of 64 with one scale per block (the K1b planar layout, half the bytes), served by the grouped integer kernel; implies the integer dot. Opt-in: on the 35B it costs +10% perplexity on the whole trunk, +2.4% on lm_head alone (see `COLI_DENSE_INT4`). |
+| `COLI_DENSE_INT4` | all components | With `COLI_DENSE_BITS=4`, a comma list of the components that take int4: `lmhead`, `dnproj`, `dnout`, `attn`, `shexp`, `router`. Measured on the 35B: `lmhead` +2.4% perplexity for 254 MB less per token; `lmhead,dnproj,dnout` +5.6%; everything +10%. |
| `QWEN_EXPERT_KERNEL` | `1` (on) | Routed experts run through the shared `expert_ffn.h` kernel: the int4 stays packed in RAM (planar layout, half the expert-cache RSS of the int8 unpack), gate+up are one pass, and a layer is two OpenMP regions over (expert, row-chunk) items instead of 3 x top-k GEMV regions. Takes effect on an int4 gs=64 container whose hidden and expert widths are multiples of 64, and not under the CUDA expert tier. `=0` restores the unpack-to-int8 path; the two produce the same tokens (1024-token decode on the real container byte-identical; pinned on the tiny int4 fixture in CI), only the f32 accumulation order inside a dot differs. Measured at cap 256 on the real container: 12.8 -> 15.7 tok/s, peak RSS 29 -> 17 GB. |
| `QWEN_DENSE_BATCH` | `1` (on) | On AVX2/FMA, reuse each dense-int8 weight decode across two prompt rows. `=0` restores one GEMV call per row. Decode `S=1` is unchanged. |
| `QWEN_SHARED_BATCH` | bounded by 32 MiB scratch | Batch the CPU shared expert across prompt rows. `=0` restores scalar calls; a positive integer caps rows per chunk. The CUDA-tier overlap path is unchanged. |
diff --git a/docs/qwen36.md b/docs/qwen36.md
index b449af49..c23f7871 100644
--- a/docs/qwen36.md
+++ b/docs/qwen36.md
@@ -104,6 +104,52 @@ overlap, not this kernel. `tests/test_expert_ffn` holds the numerics.
keeps its own path: it uploads the pair-layout int4 and computes misses from
the int8 copy.
+## The dense trunk: integer dot products
+
+Every dense GEMV of a token, the DeltaNet in and out projections, the
+attention q/k/v/o, the shared expert and lm_head, used to multiply int8
+weights by f32 activations: each weight byte converted to f32 and fed to an
+FMA, eight weights per instruction. On a 16-core AVX-512 host lm_head
+(248320 x 2048 int8, 508 MB) ran at 29 GB/s on a memory bus that does 80:
+the kernel was the limit, and the dense part of the token is 1.9 GB of int8
+on the 35B, three times what the routed experts read.
+
+Since 1.12.1 the activation is quantized to int8 once per call (one scale,
+amax/127, the same contract the expert integer kernels use) and the products
+are integer: 32 weights per instruction on AVX2, 64 on AVX-512 VNNI, exact
+int32 sums scaled once per output. The routed experts take the same path for
+their activations. Measured on Qwen3.6-35B-A3B, 8 threads, 300 decoded
+tokens, every expert resident, perplexity on 4 x 512 tokens of English text:
+
+| | tok/s | ms/token: DeltaNet proj / out, attention, lm_head, expert compute | perplexity |
+|---|---|---|---|
+| f32 activations (1.12.0) | 6.71 | 21.9 / 8.6, 9.5, 12.6, 22.7 | 13.79 |
+| int8 activations, dense trunk (`COLI_DENSE_IDOT=1`) | 7.35 | 17.4 / 6.3, 7.7, 10.2, 22.7 | 13.92 (+1.0%) |
+| int8 activations, routed experts (`QWEN_EXPERT_ACT=i8`) | 7.20 | 21.9 / 8.6, 9.5, 12.6, 15.9 | 13.80 (+0.1%) |
+| both (the 1.12.1 default) | **8.23** (+22.6%) | 17.2 / 6.3, 7.5, 10.1, 15.6 | 13.97 (+1.3%) |
+
+Both are the default; `COLI_DENSE_IDOT=0` and `QWEN_EXPERT_ACT=f32`
+restore the f32 kernels, which stay bit-identical to their references.
+
+**int4 for the trunk is opt-in, per component.** `COLI_DENSE_BITS=4` stores
+the dense matrices as int4 in blocks of 64 with one scale per block (the
+planar layout of the grouped expert kernel) and halves the bytes the token
+reads, but the parts of the trunk pay 4 bits very differently, so
+`COLI_DENSE_INT4` picks which ones take it:
+
+| int4 on | tok/s | perplexity |
+|---|---|---|
+| nothing (int8) | 7.35 | 13.92 |
+| `lmhead` | 8.15 (lm_head 10.1 to 8.4 ms; the total is within noise of the cache warming) | 14.13 (+2.4% vs f32) |
+| `lmhead,dnproj,dnout` | | 14.56 (+5.6%) |
+| `lmhead,dnproj,dnout,shexp` | | 14.94 (+8.3%) |
+| everything (`attn` included) | 8.09 | 15.17 (+10%) |
+
+A least-squares refinement of the block scale was tried and changes nothing
+(15.16 against 15.17): the loss is the matrices' sensitivity, not the
+quantizer. If you take one, take `lmhead`: 254 MB less per token for the
+smallest cost.
+
## Cache-aware routing (`CACHE_ROUTE`, off by default)
The residual misses above are the lever's target. `CACHE_ROUTE=1` ports the