Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
qwen36: integer dot products for the dense trunk and the routed experts
JustVuggdfec3a4b6637
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex a9847b86..58072d29 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -58,6 +58,21 @@ is now measured on the card in front of it instead of predicted.output. A hand-written `COLI_PLACE` stands; `COLI_TRUNK_PROBE=0` truststhe placer.+### Performance++- **#PR**: qwen36's dense trunk and routed experts multiply with integer+ dot products. The activation is quantized to int8 once per call and the+ weights, int8 rows or int4 planar blocks, meet it with maddubs / vpdpbusd+ instead of a float conversion per weight; the integer kernels move from+ `quant.h` into `idot.h`, shared by every engine. Measured on the 35B, 8+ threads, every expert resident: decode 6.71 to 8.23 tok/s (+22.6%), lm_head+ 12.6 to 10.1 ms/token, the expert compute 22.7 to 15.6, for +1.3%+ perplexity on 4 x 512 tokens. Both are the default (`COLI_DENSE_IDOT=0`,+ `QWEN_EXPERT_ACT=f32` restore the f32 kernels). `COLI_DENSE_BITS=4` with+ `COLI_DENSE_INT4=<components>` stores part of the trunk as int4 in blocks+ of 64: opt-in, with the perplexity it costs per component in the docs+ (lm_head alone +2.4%, everything +10%).+### Performance, from contributors- **#1606**: the K1b grouped int4 family gets a multi-row tile and AVX-512diff --git a/c/Makefile b/c/Makefileindex 7db43a57..55a8bfeb 100644--- a/c/Makefile+++ b/c/Makefile@@ -765,7 +765,7 @@ $(file >.build-config,$(BUILD_CONFIG))endif.build-config: ;-colibri$(EXE): colibri.c pin_pool.h cli_args.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h quant.h sample.h kv_persist.h telemetry.h route_trace.h omp_tune.h kv_fp8.h kv_tq.h abl.h backend_cuda.h backend_metal.h backend_vulkan.h decode_batch.h edge_adapters.h edge_runtime.h edge_tok_internal.h schema_gbnf.h segment_adapter_internal.h segment_adapters.h segment_runtime.h tier.h $(CUDA_OBJ) $(METAL_OBJ) $(VK_OBJ) $(VK_SPV) .build-config+colibri$(EXE): colibri.c pin_pool.h cli_args.h st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h omp_tune.h kv_fp8.h kv_tq.h abl.h backend_cuda.h backend_metal.h backend_vulkan.h decode_batch.h edge_adapters.h edge_runtime.h edge_tok_internal.h schema_gbnf.h segment_adapter_internal.h segment_adapters.h segment_runtime.h tier.h $(CUDA_OBJ) $(METAL_OBJ) $(VK_OBJ) $(VK_SPV) .build-config$(CC) $(CFLAGS) colibri.c $(CUDA_OBJ) $(METAL_OBJ) $(VK_OBJ) -o colibri$(EXE) $(LDFLAGS)# Vulkan backend object (plain C + vulkan headers) and its SPIR-V shaders.@@ -1109,13 +1109,13 @@ QWEN36_TIER_SRC =QWEN36_CFLAGS = $(NOCUDA_CFLAGS)QWEN36_LDFLAGS = $(NOCUDA_LDFLAGS)endif-qwen36$(EXE): qwen36.c decode_batch.h serve_poll.h cli_args.h qwen36_tier.h expert_ffn.h st.h json.h compat.h omp_tune.h kv_prefix.h pin_pool.h edge_adapter_internal.h edge_adapters.h edge_runtime.h segment_adapter_internal.h segment_adapters.h segment_runtime.h gsgemv.h qgemv.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)+qwen36$(EXE): qwen36.c decode_batch.h serve_poll.h cli_args.h qwen36_tier.h expert_ffn.h idot.h st.h json.h compat.h omp_tune.h kv_prefix.h pin_pool.h edge_adapter_internal.h edge_adapters.h edge_runtime.h segment_adapter_internal.h segment_adapters.h segment_runtime.h gsgemv.h qgemv.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(QWEN36_CFLAGS) qwen36.c $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o qwen36$(EXE) $(QWEN36_LDFLAGS)# DeepSeek V4.1 Flash: one file, like every other portable engine. The fp4 experts# stream from the official checkpoint and quant.h's mxfp4 kernel reads them as they# are, so there is no conversion target here to go with it.-deepseek_v41$(EXE): deepseek_v41.c cli_args.h st.h json.h tok.h tok_unicode.h compat.h omp_tune.h kv_prefix.h pin_pool.h quant.h \+deepseek_v41$(EXE): deepseek_v41.c cli_args.h st.h json.h tok.h tok_unicode.h compat.h omp_tune.h kv_prefix.h pin_pool.h quant.h idot.h \sparse_attn.h hyper_connections.h serve_codec.h serve_poll.h .build-config$(CC) $(CFLAGS) deepseek_v41.c -o deepseek_v41$(EXE) $(LDFLAGS)@@ -1123,7 +1123,7 @@ deepseek_v41$(EXE): deepseek_v41.c cli_args.h st.h json.h tok.h tok_unicode.h co# loader and SERVE=1 protocol. With CUDA=1 it links the same expert tier as# qwen36 (fp8 streaming mode: hot experts get VRAM copies, the RAM LRU stays);# without it the tier header's inline stubs keep the build toolkit-free.-qwen38$(EXE): qwen38.c pin_pool.h cli_args.h qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h omp_tune.h quant.h route_trace.h tok.h tok_unicode.h tok_unicode_o200k.h serve_codec.h edge_adapter_internal.h edge_adapters.h edge_runtime.h qwen38_vision.h segment_adapter_internal.h segment_adapters.h segment_runtime.h qwen36_tier.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)+qwen38$(EXE): qwen38.c pin_pool.h cli_args.h qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h omp_tune.h quant.h idot.h route_trace.h tok.h tok_unicode.h tok_unicode_o200k.h serve_codec.h edge_adapter_internal.h edge_adapters.h edge_runtime.h qwen38_vision.h segment_adapter_internal.h segment_adapters.h segment_runtime.h qwen36_tier.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(QWEN36_CFLAGS) qwen38.c $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o qwen38$(EXE) $(QWEN36_LDFLAGS).PHONY: qwen38-tiny-generate qwen38-tiny-check@@ -1200,6 +1200,10 @@ tests/test_qwen36_tok_truncated$(EXE): tests/test_qwen36_tok_truncated.c qwen36.tests/test_qwen36_encode_oom$(EXE): tests/test_qwen36_encode_oom.c qwen36.c expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)+# the dense trunk's integer path: quantizer contract, int4 planar packing, dispatch.+tests/test_qwen36_dense_idot$(EXE): tests/test_qwen36_dense_idot.c qwen36.c expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)+ $(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)+# #1653: added tokens are split out before the regex pre-tokenizer, HF-style.tests/test_qwen36_tokenizer$(EXE): tests/test_qwen36_tokenizer.c qwen36.c expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)@@ -1225,10 +1229,10 @@ NOCUDA_LDFLAGS = $(filter-out -lcudart -lstdc++ -lcuda -L$(CUDA_HOME)/lib64 \# GLM-5.3-Flash: routed experts stream from the int4-gs64 container.# METAL=1 accelerates resident matrices and routed MoE; CPU remains fallback.-glm53$(EXE): glm53.c decode_batch.h pin_pool.h cli_args.h st.h json.h stop_ids.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h omp_tune.h serve_poll.h route_trace.h quant.h hyper_connections.h delta_attention.h sparse_index.h vision_tower.h backend_metal.h backend_vulkan.h edge_adapter_internal.h edge_adapters.h edge_runtime.h segment_adapter_internal.h segment_adapters.h segment_runtime.h $(METAL_OBJ) $(VK_OBJ) $(VK_SPV)+glm53$(EXE): glm53.c decode_batch.h pin_pool.h cli_args.h st.h json.h stop_ids.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h omp_tune.h serve_poll.h route_trace.h quant.h idot.h hyper_connections.h delta_attention.h sparse_index.h vision_tower.h backend_metal.h backend_vulkan.h edge_adapter_internal.h edge_adapters.h edge_runtime.h segment_adapter_internal.h segment_adapters.h segment_runtime.h $(METAL_OBJ) $(VK_OBJ) $(VK_SPV)$(CC) $(CFLAGS) glm53.c $(METAL_OBJ) $(VK_OBJ) -o glm53$(EXE) $(LDFLAGS)-kimi_k3$(EXE): kimi_k3.c cli_args.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h omp_tune.h route_trace.h kv_prefix.h pin_pool.h serve_codec.h backend_cuda.h backend_metal.h backend_vulkan.h edge_adapters.h edge_runtime.h edge_tok_internal.h hybrid_split.h segment_adapter_internal.h segment_adapters.h segment_runtime.h $(CUDA_OBJ) $(VK_OBJ) $(VK_SPV) $(METAL_OBJ)+kimi_k3$(EXE): kimi_k3.c cli_args.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h idot.h omp_tune.h route_trace.h kv_prefix.h pin_pool.h serve_codec.h backend_cuda.h backend_metal.h backend_vulkan.h edge_adapters.h edge_runtime.h edge_tok_internal.h hybrid_split.h segment_adapter_internal.h segment_adapters.h segment_runtime.h $(CUDA_OBJ) $(VK_OBJ) $(VK_SPV) $(METAL_OBJ)$(CC) $(CFLAGS) kimi_k3.c $(CUDA_OBJ) $(VK_OBJ) $(METAL_OBJ) -o kimi_k3$(EXE) $(LDFLAGS)# Use a baseline that matches the compiler target. macOS already targets a@@ -1256,7 +1260,7 @@ iobench$(EXE): iobench.c compat.htests/test_serve_sentinel$(EXE): tests/test_serve_sentinel.c compat.h serve_codec.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_cluster_protocol$(EXE): tests/test_cluster_protocol.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_cluster_protocol$(EXE): tests/test_cluster_protocol.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)tests/test_ue8m0$(EXE): tests/test_ue8m0.c st.h json.h compat.h@@ -1271,10 +1275,10 @@ tests/test_tok_o200k$(EXE): tests/test_tok_o200k.c tok.h tok_unicode.h tok_unicotests/test_tok_gpt2$(EXE): tests/test_tok_gpt2.c tok.h tok_unicode.h tok_unicode_o200k.h json.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_k3_ram_budget$(EXE): tests/test_k3_ram_budget.c kimi_k3.c st.h tok.h quant.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h+tests/test_k3_ram_budget$(EXE): tests/test_k3_ram_budget.c kimi_k3.c st.h tok.h quant.h idot.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_k3_mmap$(EXE): tests/test_k3_mmap.c kimi_k3.c st.h tok.h quant.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h+tests/test_k3_mmap$(EXE): tests/test_k3_mmap.c kimi_k3.c st.h tok.h quant.h idot.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h$(CC) $(NOCUDA_CFLAGS) $< $(VK_OBJ) -o $@ $(NOCUDA_LDFLAGS)tests/test_tok_kimi_tiny$(EXE): tests/test_tok_kimi_tiny.c tok.h tok_unicode.h tok_unicode_o200k.h json.h@@ -1285,13 +1289,13 @@ tests/test_st_pread$(EXE): tests/test_st_pread.c st.h json.h compat.htests/test_st_slice$(EXE): tests/test_st_slice.c st.h json.h compat.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_qwen38_tokenizer$(EXE): tests/test_qwen38_tokenizer.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h edge_runtime.c edge_runtime.h edge_adapters.h edge_adapter_internal.h st.h json.h compat.h quant.h route_trace.h tok_unicode.h tok_unicode_o200k.h+tests/test_qwen38_tokenizer$(EXE): tests/test_qwen38_tokenizer.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h edge_runtime.c edge_runtime.h edge_adapters.h edge_adapter_internal.h st.h json.h compat.h quant.h idot.h route_trace.h tok_unicode.h tok_unicode_o200k.h$(CC) $(NOCUDA_CFLAGS) $< edge_runtime.c -o $@ $(NOCUDA_LDFLAGS)-tests/test_qwen38_config$(EXE): tests/test_qwen38_config.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h route_trace.h tok_unicode.h tok_unicode_o200k.h+tests/test_qwen38_config$(EXE): tests/test_qwen38_config.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h idot.h route_trace.h tok_unicode.h tok_unicode_o200k.h$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)-tests/test_qwen38_serve_framing$(EXE): tests/test_qwen38_serve_framing.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h+tests/test_qwen38_serve_framing$(EXE): tests/test_qwen38_serve_framing.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h idot.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)tests/test_qwen38_vision$(EXE): tests/test_qwen38_vision.c qwen38_vision.h st.h json.h compat.h@@ -1313,13 +1317,13 @@ qwen38-vision-serve-check: qwen38$(EXE)$(PYTHON) tools/make_edge_tiny_tokenizer.py --vocab-size 64 ./qwen38_mm_tiny$(PYTHON) tests/test_qwen38_vision_serve.py --binary ./qwen38$(EXE) --fixture ./qwen38_mm_tiny-tests/test_qwen38_prefix$(EXE): tests/test_qwen38_prefix.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h+tests/test_qwen38_prefix$(EXE): tests/test_qwen38_prefix.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h idot.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)-tests/test_qwen38_metrics$(EXE): tests/test_qwen38_metrics.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h+tests/test_qwen38_metrics$(EXE): tests/test_qwen38_metrics.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h st.h json.h compat.h quant.h idot.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)-tests/test_qwen38_native_weights$(EXE): tests/test_qwen38_native_weights.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h segment_runtime.c segment_runtime.h segment_adapters.h segment_adapter_internal.h st.h json.h compat.h quant.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h+tests/test_qwen38_native_weights$(EXE): tests/test_qwen38_native_weights.c qwen38.c qwen38_core.h qwen38_nfc.h qwen38_nfc_tables.h segment_runtime.c segment_runtime.h segment_adapters.h segment_adapter_internal.h st.h json.h compat.h quant.h idot.h route_trace.h serve_codec.h tok_unicode.h tok_unicode_o200k.h$(CC) $(NOCUDA_CFLAGS) $< segment_runtime.c -o $@ $(NOCUDA_LDFLAGS)tests/test_st_map$(EXE): tests/test_st_map.c st.h json.h compat.h@@ -1365,12 +1369,12 @@ tests/test_grammar$(EXE): tests/test_grammar.c grammar.h# equivalence under the shipping flags is covered by the differential dump# (old vs new rope_interleave are byte-identical); this test guards the# source-level formula equivalence and must not depend on contraction luck.-tests/test_rope_invfreq$(EXE): tests/test_rope_invfreq.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_rope_invfreq$(EXE): tests/test_rope_invfreq.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) -ffp-contract=off $< -o $@ $(LDFLAGS)# schema->GBNF compile cache (#7): grammar_reset must equal a fresh setup, and# the GrDraft.src ownership must not leak/double-free. Includes colibri.c.-tests/test_grammar_cache$(EXE): tests/test_grammar_cache.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_grammar_cache$(EXE): tests/test_grammar_cache.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)# Standalone: drives a faithful miniature of moe()'s routing+accumulate and links@@ -1386,7 +1390,7 @@ tests/test_degrade_zero$(EXE): tests/test_degrade_zero.ctests/test_schema_gbnf$(EXE): tests/test_schema_gbnf.c schema_gbnf.h grammar.h json.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_spec_decode_state$(EXE): tests/test_spec_decode_state.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_spec_decode_state$(EXE): tests/test_spec_decode_state.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)tests/test_decode_batch$(EXE): tests/test_decode_batch.c decode_batch.h@@ -1418,13 +1422,13 @@ tests/bench_inkling_shared_batch$(EXE): tests/bench_inkling_shared_batch.c inklitests/test_inkling_cache_index$(EXE): tests/test_inkling_cache_index.c inkling.c st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h omp_tune.h route_trace.h kv_prefix.h serve_codec.h $(INK_CUDA_OBJ) $(METAL_OBJ)$(CC) $(CFLAGS) $< $(INK_CUDA_OBJ) $(METAL_OBJ) -o $@ $(LDFLAGS)-tests/test_kimi_serve_framing$(EXE): tests/test_kimi_serve_framing.c kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h omp_tune.h route_trace.h+tests/test_kimi_serve_framing$(EXE): tests/test_kimi_serve_framing.c kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h idot.h omp_tune.h route_trace.h$(CC) $(NOCUDA_CFLAGS) $< $(VK_OBJ) -o $@ $(NOCUDA_LDFLAGS)-tests/test_kimi_cache_index$(EXE): tests/test_kimi_cache_index.c kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h omp_tune.h route_trace.h+tests/test_kimi_cache_index$(EXE): tests/test_kimi_cache_index.c kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h idot.h omp_tune.h route_trace.h$(CC) $(NOCUDA_CFLAGS) $< $(VK_OBJ) -o $@ $(NOCUDA_LDFLAGS)-tests/test_k3_chat_tools$(EXE): tests/test_k3_chat_tools.c kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h omp_tune.h route_trace.h+tests/test_k3_chat_tools$(EXE): tests/test_k3_chat_tools.c kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h idot.h omp_tune.h route_trace.h$(CC) $(NOCUDA_CFLAGS) $< $(VK_OBJ) -o $@ $(NOCUDA_LDFLAGS)# olmoe's matmul_q, not colibri's: compares the IDOT path against FP32@@ -1442,33 +1446,33 @@ tests/test_olmoe_cache_index$(EXE): tests/test_olmoe_cache_index.c olmoe.c st.htests/test_qwen36_cache_index$(EXE): tests/test_qwen36_cache_index.c qwen36.c expert_ffn.h st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h qwen36_tier.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)-tests/test_idot$(EXE): tests/test_idot.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_idot$(EXE): tests/test_idot.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_i4_grouped$(EXE): tests/test_i4_grouped.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_i4_grouped$(EXE): tests/test_i4_grouped.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_stops$(EXE): tests/test_stops.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_stops$(EXE): tests/test_stops.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_cfg_topk$(EXE): tests/test_cfg_topk.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_cfg_topk$(EXE): tests/test_cfg_topk.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_topp$(EXE): tests/test_topp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_topp$(EXE): tests/test_topp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)# bench_topp is a microbenchmark (old qsort vs new heap partial-select, #335), NOT a test# gate -- intentionally absent from TEST_BINS. Build on demand: make tests/bench_topp-tests/bench_topp$(EXE): tests/bench_topp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/bench_topp$(EXE): tests/bench_topp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_sample_nan$(EXE): tests/test_sample_nan.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_sample_nan$(EXE): tests/test_sample_nan.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_temp_env$(EXE): tests/test_temp_env.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_temp_env$(EXE): tests/test_temp_env.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_kv_alloc$(EXE): tests/test_kv_alloc.c colibri.c st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h kv_fp8.h kv_tq.h+tests/test_kv_alloc$(EXE): tests/test_kv_alloc.c colibri.c st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h kv_fp8.h kv_tq.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)tests/test_kv_fp8$(EXE): tests/test_kv_fp8.c kv_fp8.h decode_batch.h@@ -1477,15 +1481,15 @@ tests/test_kv_fp8$(EXE): tests/test_kv_fp8.c kv_fp8.h decode_batch.htests/test_kv_tq$(EXE): tests/test_kv_tq.c kv_tq.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_kv_disk$(EXE): tests/test_kv_disk.c colibri.c st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h kv_fp8.h kv_tq.h+tests/test_kv_disk$(EXE): tests/test_kv_disk.c colibri.c st.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h kv_fp8.h kv_tq.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)# fmt=6 kernel oracle: needs the generated grid table, and its fixture comes from# the reference codec (tools/make_e8_fixture.py) β regenerate if the layout moves.-tests/test_e8_kernel$(EXE): tests/test_e8_kernel.c quant.h+tests/test_e8_kernel$(EXE): tests/test_e8_kernel.c quant.h idot.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_e4m3_vector$(EXE): tests/test_e4m3_vector.c quant.h+tests/test_e4m3_vector$(EXE): tests/test_e4m3_vector.c quant.h idot.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)tests/test_stop_ids$(EXE): tests/test_stop_ids.c stop_ids.h json.h@@ -1512,13 +1516,13 @@ fuzz-rans: tests/fuzz_rans.c rans.htests/fuzz_rans.c -o tests/fuzz_rans -lm./tests/fuzz_rans-tests/test_int3$(EXE): tests/test_int3.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_int3$(EXE): tests/test_int3.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_int3_load$(EXE): tests/test_int3_load.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_int3_load$(EXE): tests/test_int3_load.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_fp8_passthrough$(EXE): tests/test_fp8_passthrough.c quant.h+tests/test_fp8_passthrough$(EXE): tests/test_fp8_passthrough.c quant.h idot.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)# Host-only: backend_cuda.h's format predicate is plain C, so its truth table is@@ -1527,10 +1531,10 @@ tests/test_fp8_passthrough$(EXE): tests/test_fp8_passthrough.c quant.htests/test_cuda_fmt_guard$(EXE): tests/test_cuda_fmt_guard.c backend_cuda.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_fp8_load$(EXE): tests/test_fp8_load.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h+tests/test_fp8_load$(EXE): tests/test_fp8_load.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_qt_addrow$(EXE): tests/test_qt_addrow.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h+tests/test_qt_addrow$(EXE): tests/test_qt_addrow.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)tests/test_logit_nan$(EXE): tests/test_logit_nan.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h@@ -1548,7 +1552,7 @@ tests/test_compat_direct$(EXE): tests/test_compat_direct.c compat.htests/test_expert_store_ops$(EXE): tests/test_expert_store_ops.c expert_store.h tensor.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_native_quant$(EXE): tests/test_native_quant.c deepseek_v4.c native_quant.h tensor.h quant.h+tests/test_native_quant$(EXE): tests/test_native_quant.c deepseek_v4.c native_quant.h tensor.h quant.h idot.h$(CC) $(CFLAGS) -DCOLI_V4_UNIT_NATIVE_QUANT deepseek_v4.c $< -o $@ $(LDFLAGS)tests/test_edge_runtime$(EXE): tests/test_edge_runtime.c edge_runtime.c edge_runtime.h@@ -1785,7 +1789,7 @@ $(V4_OWN_DIR)/COLI_V4_UNIT_CONFIG.o: deepseek_v4.c deepseek_v4.h deepseek_v4_int$(V4_OWN_DIR)/COLI_V4_UNIT_ST.o: deepseek_v4.c deepseek_v4.h deepseek_v4_internal.h st.h | $(V4_OWN_DIR)$(CC) $(V4_OWN_CFLAGS) -DCOLI_V4_UNIT_ST -c deepseek_v4.c -o $@-$(V4_OWN_DIR)/COLI_V4_UNIT_NATIVE_QUANT.o: deepseek_v4.c deepseek_v4.h deepseek_v4_internal.h quant.h | $(V4_OWN_DIR)+$(V4_OWN_DIR)/COLI_V4_UNIT_NATIVE_QUANT.o: deepseek_v4.c deepseek_v4.h deepseek_v4_internal.h quant.h idot.h | $(V4_OWN_DIR)$(CC) $(V4_OWN_CFLAGS) -DCOLI_V4_UNIT_NATIVE_QUANT -c deepseek_v4.c -o $@# The engine (RUNTIME unit) opens its expert store through the pluggable@@ -1887,7 +1891,7 @@ tests/test_qwen36_tier_int8_decode$(EXE): tests/test_qwen36_tier_int8_decode.c t# The qwen38 engine through its own main() on the fake backend: tier start# from the FP8 fixture, reduced CPU list, qt_note on recycled slots, oracle# tokens unchanged. Needs the fixture (qwen38-tiny-fp8-generate).-tests/test_qwen38_tier_engine$(EXE): tests/test_qwen38_tier_engine.c tests/qwen36_fake_cuda.h qwen38.c qwen38_core.h qwen36_tier.c qwen36_tier.h backend_cuda.h cli_args.h st.h json.h compat.h quant.h route_trace.h+tests/test_qwen38_tier_engine$(EXE): tests/test_qwen38_tier_engine.c tests/qwen36_fake_cuda.h qwen38.c qwen38_core.h qwen36_tier.c qwen36_tier.h backend_cuda.h cli_args.h st.h json.h compat.h quant.h idot.h route_trace.h$(CC) $(CFLAGS) -DCOLI_CUDA $< -o $@ $(LDFLAGS)tests/test_serve_poll$(EXE): tests/test_serve_poll.c serve_poll.h@@ -1906,15 +1910,15 @@ tests/test_mem_available$(EXE): tests/test_mem_available.c compat.htests/test_798_guards$(EXE): tests/test_798_guards.c st.h json.h compat.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_dsa_select$(EXE): tests/test_dsa_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_dsa_select$(EXE): tests/test_dsa_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_corpus_draft$(EXE): tests/test_corpus_draft.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_corpus_draft$(EXE): tests/test_corpus_draft.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_cap_precedence$(EXE): tests/test_cap_precedence.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h+tests/test_cap_precedence$(EXE): tests/test_cap_precedence.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_mirror_stripe_split$(EXE): tests/test_mirror_stripe_split.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h+tests/test_mirror_stripe_split$(EXE): tests/test_mirror_stripe_split.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)tests/test_v4_hybrid_policy$(EXE): tests/test_v4_hybrid_policy.c deepseek_v4_hybrid.h hybrid_split.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)@@ -1925,29 +1929,29 @@ tests/test_k3_fill_budget$(EXE): tests/test_k3_fill_budget.c hybrid_split.htests/test_v4_bank_pair$(EXE): tests/test_v4_bank_pair.c deepseek_v4_bank_pair.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_ram_clamp$(EXE): tests/test_ram_clamp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h+tests/test_ram_clamp$(EXE): tests/test_ram_clamp.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_cap_mixed_width$(EXE): tests/test_cap_mixed_width.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_cap_mixed_width$(EXE): tests/test_cap_mixed_width.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_eslot_inflight$(EXE): tests/test_eslot_inflight.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_eslot_inflight$(EXE): tests/test_eslot_inflight.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_glm_cache_index$(EXE): tests/test_glm_cache_index.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_glm_cache_index$(EXE): tests/test_glm_cache_index.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_ssd_probe$(EXE): tests/test_ssd_probe.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h+tests/test_ssd_probe$(EXE): tests/test_ssd_probe.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)# bench_dsa_select is a microbenchmark (old qsort vs new quickselect partial-select, #356),# NOT a test gate -- intentionally absent from TEST_BINS. Build on demand: make tests/bench_dsa_select-tests/bench_dsa_select$(EXE): tests/bench_dsa_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/bench_dsa_select$(EXE): tests/bench_dsa_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)# bench_router_select is a microbenchmark (duplicate-prefix scan vs marked-score scan),# not a test gate. Build on demand: make tests/bench_router_select.-tests/bench_router_select$(EXE): tests/bench_router_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/bench_router_select$(EXE): tests/bench_router_select.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)# bench_indexer_allocations: microbenchmark (DeepSeek V4 indexer malloc vs persistent arena scratch), NOT a test gate.@@ -1957,34 +1961,34 @@ tests/bench_indexer_allocations$(EXE): tests/bench_indexer_allocations.c# bench_idot: microbenchmark (single-acc vs independent-acc AVX-VNNI idot), NOT a test gate.# Build on demand on an AVX-VNNI CPU: make tests/bench_idot ARCH=native-tests/bench_idot$(EXE): tests/bench_idot.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/bench_idot$(EXE): tests/bench_idot.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)# bench_i4p_gidot: microbenchmark (per-row vs multi-row/AMX K1b grouped planar IDOT), NOT a test gate.# Build on demand: make tests/bench_i4p_gidot ARCH=native-tests/bench_i4p_gidot$(EXE): tests/bench_i4p_gidot.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/bench_i4p_gidot$(EXE): tests/bench_i4p_gidot.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)# bench_gemv_stream: microbenchmark (decode-regime GEMV bandwidth vs the read ceiling;# frozen-baseline + deinterleaved-x candidate A/B), NOT a test gate.# Build on demand: make tests/bench_gemv_stream ARCH=native-tests/bench_gemv_stream$(EXE): tests/bench_gemv_stream.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/bench_gemv_stream$(EXE): tests/bench_gemv_stream.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)# bench_mla_simd: microbenchmark (scalar vs AVX2/NEON MLA-absorb reductions, #442),# NOT a test gate. Build on demand: make tests/bench_mla_simd-tests/bench_mla_simd$(EXE): tests/bench_mla_simd.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/bench_mla_simd$(EXE): tests/bench_mla_simd.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_uring$(EXE): tests/test_uring.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h route_trace.h+tests/test_uring$(EXE): tests/test_uring.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h route_trace.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)tests/test_pipe_block$(EXE): tests/test_pipe_block.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_pilot_ring$(EXE): tests/test_pilot_ring.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h+tests/test_pilot_ring$(EXE): tests/test_pilot_ring.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_moe_gs_guard$(EXE): tests/test_moe_gs_guard.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h+tests/test_moe_gs_guard$(EXE): tests/test_moe_gs_guard.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)tests/test_omp_tune$(EXE): tests/test_omp_tune.c omp_tune.h compat.h@@ -1993,7 +1997,7 @@ tests/test_omp_tune$(EXE): tests/test_omp_tune.c omp_tune.h compat.htests/test_compat_env$(EXE): tests/test_compat_env.c compat.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)-tests/test_kvb_notice$(EXE): tests/test_kvb_notice.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h sample.h kv_persist.h telemetry.h+tests/test_kvb_notice$(EXE): tests/test_kvb_notice.c colibri.c st.h uring.h json.h tok.h tok_unicode.h compat.h grammar.h tier.h quant.h idot.h sample.h kv_persist.h telemetry.h$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)# Standalone: proves the group-scaled int8 GEMV keeps every row's float@@ -2166,5 +2170,5 @@ bench: iobench$(EXE)tests/test_kv_prefix$(EXE): tests/test_kv_prefix.c kv_prefix.h$(CC) $(CFLAGS) tests/test_kv_prefix.c -o tests/test_kv_prefix$(EXE) $(LDFLAGS)-tests/test_kimi_request_state$(EXE): tests/test_kimi_request_state.c kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h omp_tune.h route_trace.h+tests/test_kimi_request_state$(EXE): tests/test_kimi_request_state.c kimi_k3.c kv_prefix.h serve_codec.h st.h json.h tok.h tok_unicode.h tok_unicode_o200k.h compat.h quant.h idot.h omp_tune.h route_trace.h$(CC) $(NOCUDA_CFLAGS) tests/test_kimi_request_state.c $(VK_OBJ) -o tests/test_kimi_request_state$(EXE) $(NOCUDA_LDFLAGS)diff --git a/c/idot.h b/c/idot.hnew file mode 100644index 00000000..6738abeb--- /dev/null+++ b/c/idot.h@@ -0,0 +1,978 @@+/* idot.h -- integer dot-product kernels shared by every engine.+ *+ * The activation is quantized to int8 once (one scale per row, amax/127,+ * qrow_i8) and the weights, int8 rows or int4 planar blocks with per-row or+ * per-group scales, are multiplied with integer instructions: maddubs on+ * AVX2, vpdpbusd on AVX-VNNI and AVX-512 VNNI, sdot/smmla on NEON, and the+ * AMX tile kernel where the CPU has it. Exact int32 sums, scaled once per+ * output, so every ISA path is bit-identical to the scalar reference.+ *+ * Lived inside quant.h; moved here so an engine that has its own dense+ * kernels (qwen36.c uses qgemv.h/gsgemv.h, whose matmul_q would collide+ * with quant.h's) can still take the integer path for its dense trunk.+ * quant.h includes this file at the top, so its other consumers see the+ * same declarations in the same place as before. */+#ifndef COLIBRI_IDOT_H+#define COLIBRI_IDOT_H+#include <stdint.h>+#include <stdlib.h>+#include <string.h>+#include <math.h>+#include <stdio.h>+/* ---- SIMD includes -------------------------------------------------------- */+#ifdef __AVX2__+#include <immintrin.h>+static inline float hsum256(__m256 v){+ __m128 lo=_mm256_castps256_ps128(v), hi=_mm256_extractf128_ps(v,1);+ lo=_mm_add_ps(lo,hi); __m128 sh=_mm_movehl_ps(lo,lo); lo=_mm_add_ps(lo,sh);+ sh=_mm_shuffle_ps(lo,lo,1); lo=_mm_add_ss(lo,sh); return _mm_cvtss_f32(lo);+}+static inline int hsum256_i32(__m256i v){+ __m128i lo=_mm256_castsi256_si128(v), hi=_mm256_extracti128_si256(v,1);+ lo=_mm_add_epi32(lo,hi); lo=_mm_hadd_epi32(lo,lo); lo=_mm_hadd_epi32(lo,lo);+ return _mm_cvtsi128_si32(lo);+}+#endif+#if defined(__AVXVNNI__) && defined(__AVX2__)+static inline int hsum128_i32(__m128i v){+ v=_mm_hadd_epi32(v,v); v=_mm_hadd_epi32(v,v); return _mm_cvtsi128_si32(v);+}+#endif+#ifdef __ARM_NEON+#include <arm_neon.h>+#endif+#ifdef __VSX__+#include <altivec.h>+#undef vector+#undef pixel+#undef bool+#endif+++/* ---- IDOT: integer dot kernels (int8-quantized activations) --------------- */+#if defined(__AVX512VNNI__) && defined(__AVX512BW__)+#define IDOT_KERNEL "avx512-vnni"+#elif defined(__AVXVNNI__) && defined(__AVX2__)+#define IDOT_KERNEL "avx-vnni"+#elif defined(__AVX2__)+#define IDOT_KERNEL "avx2"+#elif defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)+#define IDOT_KERNEL "neon-i8mm"+#elif defined(__ARM_NEON)+#define IDOT_KERNEL "neon"+#elif defined(__VSX__)+#define IDOT_KERNEL "vsx"+#else+#define IDOT_KERNEL "scalar"+#endif++static inline float qrow_i8(const float *x, int8_t *q, int I){+ float amax=0; for(int i=0;i<I;i++){ float a=fabsf(x[i]); if(a>amax)amax=a; }+ float s=amax/127.f; if(s<1e-12f) s=1e-12f; float inv=1.f/s;+ for(int i=0;i<I;i++) q[i]=(int8_t)lrintf(x[i]*inv);+ return s;+}++/* dot int8*int8 */+static inline int32_t dot_i8i8(const int8_t *w, const int8_t *x, int I){+ int32_t sum=0; int i=0;+#if defined(__AVX512VNNI__) && defined(__AVX512BW__)+ __m512i acc=_mm512_setzero_si512();+ for(;i+64<=I;i+=64){+ __m512i wv=_mm512_loadu_si512((const void*)(w+i));+ __m512i xv=_mm512_loadu_si512((const void*)(x+i));+ __mmask64 neg=_mm512_movepi8_mask(wv);+ __m512i xs=_mm512_mask_sub_epi8(xv,neg,_mm512_setzero_si512(),xv);+ acc=_mm512_dpbusd_epi32(acc,_mm512_abs_epi8(wv),xs);+ }+ sum=_mm512_reduce_add_epi32(acc);+#elif defined(__AVXVNNI__) && defined(__AVX2__)+ /* 4 accumulatori indipendenti (64 byte/iter): un solo acc incatena i vpdpbusd+ * (latenza-bound ~5c). Somme intere associative -> bit-identico. Stessa struttura+ * dei 4 accumulatori del ramo NEON piu' sotto.+ * EN: four independent accumulators break the serial vpdpbusd->acc chain; integer+ * adds are associative, so the result is bit-identical (mirrors the NEON path). */+ __m128i a0=_mm_setzero_si128(),a1=_mm_setzero_si128(),a2=_mm_setzero_si128(),a3=_mm_setzero_si128();+ for(;i+64<=I;i+=64){+ __m128i w0=_mm_loadu_si128((const __m128i*)(w+i)), x0=_mm_loadu_si128((const __m128i*)(x+i));+ __m128i w1=_mm_loadu_si128((const __m128i*)(w+i+16)), x1=_mm_loadu_si128((const __m128i*)(x+i+16));+ __m128i w2=_mm_loadu_si128((const __m128i*)(w+i+32)), x2=_mm_loadu_si128((const __m128i*)(x+i+32));+ __m128i w3=_mm_loadu_si128((const __m128i*)(w+i+48)), x3=_mm_loadu_si128((const __m128i*)(x+i+48));+ a0=_mm_dpbusd_epi32(a0,_mm_abs_epi8(w0),_mm_sign_epi8(x0,w0));+ a1=_mm_dpbusd_epi32(a1,_mm_abs_epi8(w1),_mm_sign_epi8(x1,w1));+ a2=_mm_dpbusd_epi32(a2,_mm_abs_epi8(w2),_mm_sign_epi8(x2,w2));+ a3=_mm_dpbusd_epi32(a3,_mm_abs_epi8(w3),_mm_sign_epi8(x3,w3));+ }+ __m128i acc=_mm_add_epi32(_mm_add_epi32(a0,a1),_mm_add_epi32(a2,a3));+ for(;i+16<=I;i+=16){+ __m128i wv=_mm_loadu_si128((const __m128i*)(w+i));+ __m128i xv=_mm_loadu_si128((const __m128i*)(x+i));+ acc=_mm_dpbusd_epi32(acc,_mm_abs_epi8(wv),_mm_sign_epi8(xv,wv));+ }+ sum=hsum128_i32(acc);+#elif defined(__AVX2__)+ __m256i acc=_mm256_setzero_si256(); const __m256i ones=_mm256_set1_epi16(1);+ for(;i+32<=I;i+=32){+ __m256i wv=_mm256_loadu_si256((const __m256i*)(w+i));+ __m256i xv=_mm256_loadu_si256((const __m256i*)(x+i));+ __m256i p=_mm256_maddubs_epi16(_mm256_sign_epi8(wv,wv),_mm256_sign_epi8(xv,wv));+ acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p,ones));+ }+ sum=hsum256_i32(acc);+#elif defined(__ARM_NEON)+#if defined(__ARM_FEATURE_DOTPROD)+ int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0);+ for(;i+64<=I;i+=64){+ a0=vdotq_s32(a0,vld1q_s8(w+i), vld1q_s8(x+i));+ a1=vdotq_s32(a1,vld1q_s8(w+i+16),vld1q_s8(x+i+16));+ a2=vdotq_s32(a2,vld1q_s8(w+i+32),vld1q_s8(x+i+32));+ a3=vdotq_s32(a3,vld1q_s8(w+i+48),vld1q_s8(x+i+48));+ }+ int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));+ for(;i+16<=I;i+=16) acc=vdotq_s32(acc,vld1q_s8(w+i),vld1q_s8(x+i));+ sum=vaddvq_s32(acc);+#else+ int32x4_t acc=vdupq_n_s32(0);+ for(;i+16<=I;i+=16){+ int8x16_t wv=vld1q_s8(w+i), xv=vld1q_s8(x+i);+ int16x8_t p=vmull_s8(vget_low_s8(wv),vget_low_s8(xv));+ p=vmlal_s8(p,vget_high_s8(wv),vget_high_s8(xv));+ acc=vpadalq_s16(acc,p);+ }+ sum=vaddvq_s32(acc);+#endif+#elif defined(__VSX__)+ __vector signed int acc=vec_splats(0);+ const __vector signed char vz=vec_splats((signed char)0);+ for(;i+16<=I;i+=16){+ __vector signed char wv=vec_xl(0,(const signed char*)(w+i));+ __vector signed char xv=vec_xl(0,(const signed char*)(x+i));+ __vector __bool char neg=vec_cmplt(wv,vz);+ __vector signed char xs=vec_sel(xv,vec_sub(vz,xv),neg);+ __vector unsigned char wa=(__vector unsigned char)vec_sel(wv,vec_sub(vz,wv),neg);+ acc=vec_msum(xs,wa,acc);+ }+ sum=vec_extract(acc,0)+vec_extract(acc,1)+vec_extract(acc,2)+vec_extract(acc,3);+#endif+ for(;i<I;i++) sum+=(int32_t)w[i]*x[i];+ return sum;+}++/* dot int4(packed)*int8 */+static inline int32_t dot_i4i8(const uint8_t *w4, const int8_t *x, int I){+ int32_t sum=0; int i=0;+#if defined(__AVX512VNNI__) && defined(__AVX512BW__)+ const __m256i m4v=_mm256_set1_epi8(0x0F);+ const __m512i b8v=_mm512_set1_epi8(8);+ const __m512i xidx=_mm512_setr_epi64(0,1,4,5,2,3,6,7);+ __m512i acc=_mm512_setzero_si512();+ for(;i+64<=I;i+=64){+ __m256i by=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));+ __m256i lo=_mm256_and_si256(by,m4v), hi=_mm256_and_si256(_mm256_srli_epi16(by,4),m4v);+ __m256i z0=_mm256_unpacklo_epi8(lo,hi), z1=_mm256_unpackhi_epi8(lo,hi);+ __m512i wv=_mm512_sub_epi8(_mm512_inserti64x4(_mm512_castsi256_si512(z0),z1,1),b8v);+ __m512i xv=_mm512_permutexvar_epi64(xidx,_mm512_loadu_si512((const void*)(x+i)));+ __mmask64 neg=_mm512_movepi8_mask(wv);+ __m512i xs=_mm512_mask_sub_epi8(xv,neg,_mm512_setzero_si512(),xv);+ acc=_mm512_dpbusd_epi32(acc,_mm512_abs_epi8(wv),xs);+ }+ sum=_mm512_reduce_add_epi32(acc);+#elif defined(__AVXVNNI__) && defined(__AVX2__)+ /* 4 accumulatori indipendenti (64 elementi = 32 byte packed/iter): un solo acc+ * incatena i vpdpbusd (latenza-bound ~5c). Somme intere associative -> bit-identico.+ * Stessa struttura dei 4 accumulatori del ramo NEON piu' sotto.+ * EN: four independent accumulators break the serial vpdpbusd->acc chain; integer+ * adds are associative, so the result is bit-identical (mirrors the NEON path). */+ const __m128i m4=_mm_set1_epi8(0x0F); const __m128i b8=_mm_set1_epi8(8);+ __m128i a0=_mm_setzero_si128(),a1=_mm_setzero_si128(),a2=_mm_setzero_si128(),a3=_mm_setzero_si128();+ for(;i+64<=I;i+=64){+ __m128i by0=_mm_loadu_si128((const __m128i*)(w4+(i>>1))); /* elem i..i+31 */+ __m128i by1=_mm_loadu_si128((const __m128i*)(w4+(i>>1)+16)); /* elem i+32..i+63 */+ __m128i lo0=_mm_and_si128(by0,m4), hi0=_mm_and_si128(_mm_srli_epi16(by0,4),m4);+ __m128i lo1=_mm_and_si128(by1,m4), hi1=_mm_and_si128(_mm_srli_epi16(by1,4),m4);+ __m128i w0=_mm_sub_epi8(_mm_unpacklo_epi8(lo0,hi0),b8), w1=_mm_sub_epi8(_mm_unpackhi_epi8(lo0,hi0),b8);+ __m128i w2=_mm_sub_epi8(_mm_unpacklo_epi8(lo1,hi1),b8), w3=_mm_sub_epi8(_mm_unpackhi_epi8(lo1,hi1),b8);+ __m128i x0=_mm_loadu_si128((const __m128i*)(x+i)), x1=_mm_loadu_si128((const __m128i*)(x+i+16));+ __m128i x2=_mm_loadu_si128((const __m128i*)(x+i+32)), x3=_mm_loadu_si128((const __m128i*)(x+i+48));+ a0=_mm_dpbusd_epi32(a0,_mm_abs_epi8(w0),_mm_sign_epi8(x0,w0));+ a1=_mm_dpbusd_epi32(a1,_mm_abs_epi8(w1),_mm_sign_epi8(x1,w1));+ a2=_mm_dpbusd_epi32(a2,_mm_abs_epi8(w2),_mm_sign_epi8(x2,w2));+ a3=_mm_dpbusd_epi32(a3,_mm_abs_epi8(w3),_mm_sign_epi8(x3,w3));+ }+ __m128i acc=_mm_add_epi32(_mm_add_epi32(a0,a1),_mm_add_epi32(a2,a3));+ for(;i+32<=I;i+=32){ /* 32-nibble remainder: 2 dpbusd, same unpack */+ __m128i by=_mm_loadu_si128((const __m128i*)(w4+(i>>1)));+ __m128i lo=_mm_and_si128(by,m4), hi=_mm_and_si128(_mm_srli_epi16(by,4),m4);+ __m128i w0=_mm_sub_epi8(_mm_unpacklo_epi8(lo,hi),b8), w1=_mm_sub_epi8(_mm_unpackhi_epi8(lo,hi),b8);+ __m128i x0=_mm_loadu_si128((const __m128i*)(x+i));+ __m128i x1=_mm_loadu_si128((const __m128i*)(x+i+16));+ acc=_mm_dpbusd_epi32(acc,_mm_abs_epi8(w0),_mm_sign_epi8(x0,w0));+ acc=_mm_dpbusd_epi32(acc,_mm_abs_epi8(w1),_mm_sign_epi8(x1,w1));+ }+ sum=hsum128_i32(acc);+#elif defined(__AVX2__)+ const __m128i m4=_mm_set1_epi8(0x0F); const __m256i b8=_mm256_set1_epi8(8);+ const __m256i ones=_mm256_set1_epi16(1);+ __m256i acc=_mm256_setzero_si256();+ for(;i+32<=I;i+=32){+ __m128i by=_mm_loadu_si128((const __m128i*)(w4+(i>>1)));+ __m128i lo=_mm_and_si128(by,m4), hi=_mm_and_si128(_mm_srli_epi16(by,4),m4);+ __m128i n0=_mm_unpacklo_epi8(lo,hi), n1=_mm_unpackhi_epi8(lo,hi);+ __m256i wv=_mm256_sub_epi8(_mm256_set_m128i(n1,n0),b8);+ __m256i xv=_mm256_loadu_si256((const __m256i*)(x+i));+ __m256i p=_mm256_maddubs_epi16(_mm256_sign_epi8(wv,wv),_mm256_sign_epi8(xv,wv));+ acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p,ones));+ }+ sum=hsum256_i32(acc);+#elif defined(__ARM_NEON)+ const uint8x16_t m4q=vdupq_n_u8(0x0F); const int8x16_t b8q=vdupq_n_s8(8);+#if defined(__ARM_FEATURE_DOTPROD)+ int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0);+ for(;i+64<=I;i+=64){+ uint8x16_t byA=vld1q_u8(w4+(i>>1)), byB=vld1q_u8(w4+(i>>1)+16);+ uint8x16x2_t zA=vzipq_u8(vandq_u8(byA,m4q), vshrq_n_u8(byA,4));+ uint8x16x2_t zB=vzipq_u8(vandq_u8(byB,m4q), vshrq_n_u8(byB,4));+ a0=vdotq_s32(a0,vsubq_s8(vreinterpretq_s8_u8(zA.val[0]),b8q),vld1q_s8(x+i));+ a1=vdotq_s32(a1,vsubq_s8(vreinterpretq_s8_u8(zA.val[1]),b8q),vld1q_s8(x+i+16));+ a2=vdotq_s32(a2,vsubq_s8(vreinterpretq_s8_u8(zB.val[0]),b8q),vld1q_s8(x+i+32));+ a3=vdotq_s32(a3,vsubq_s8(vreinterpretq_s8_u8(zB.val[1]),b8q),vld1q_s8(x+i+48));+ }+ int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));+ for(;i+32<=I;i+=32){+ uint8x16_t by=vld1q_u8(w4+(i>>1));+ uint8x16x2_t z=vzipq_u8(vandq_u8(by,m4q), vshrq_n_u8(by,4));+ acc=vdotq_s32(acc,vsubq_s8(vreinterpretq_s8_u8(z.val[0]),b8q),vld1q_s8(x+i));+ acc=vdotq_s32(acc,vsubq_s8(vreinterpretq_s8_u8(z.val[1]),b8q),vld1q_s8(x+i+16));+ }+ sum=vaddvq_s32(acc);+#else+ int32x4_t acc=vdupq_n_s32(0);+ for(;i+32<=I;i+=32){+ uint8x16_t by=vld1q_u8(w4+(i>>1));+ uint8x16x2_t z=vzipq_u8(vandq_u8(by,m4q), vshrq_n_u8(by,4));+ int8x16_t w0=vsubq_s8(vreinterpretq_s8_u8(z.val[0]),b8q);+ int8x16_t w1=vsubq_s8(vreinterpretq_s8_u8(z.val[1]),b8q);+ int8x16_t x0=vld1q_s8(x+i), x1=vld1q_s8(x+i+16);+ int16x8_t p=vmull_s8(vget_low_s8(w0),vget_low_s8(x0));+ p=vmlal_s8(p,vget_high_s8(w0),vget_high_s8(x0));+ acc=vpadalq_s16(acc,p);+ p=vmull_s8(vget_low_s8(w1),vget_low_s8(x1));+ p=vmlal_s8(p,vget_high_s8(w1),vget_high_s8(x1));+ acc=vpadalq_s16(acc,p);+ }+ sum=vaddvq_s32(acc);+#endif+#elif defined(__VSX__)+ const __vector unsigned char m4v=vec_splats((unsigned char)0x0F);+ const __vector unsigned char sh4=vec_splats((unsigned char)4);+ const __vector signed char b8v=vec_splats((signed char)8);+ const __vector signed char vz=vec_splats((signed char)0);+ __vector signed int acc=vec_splats(0);+ for(;i+32<=I;i+=32){+ __vector unsigned char by=vec_xl(0,w4+(i>>1));+ __vector unsigned char lo=vec_and(by,m4v), hi=vec_sr(by,sh4);+ __vector signed char w0=vec_sub((__vector signed char)vec_mergeh(lo,hi),b8v);+ __vector signed char w1=vec_sub((__vector signed char)vec_mergel(lo,hi),b8v);+ __vector signed char x0=vec_xl(0,(const signed char*)(x+i));+ __vector signed char x1=vec_xl(0,(const signed char*)(x+i+16));+ __vector __bool char n0=vec_cmplt(w0,vz), n1=vec_cmplt(w1,vz);+ acc=vec_msum(vec_sel(x0,vec_sub(vz,x0),n0),+ (__vector unsigned char)vec_sel(w0,vec_sub(vz,w0),n0),acc);+ acc=vec_msum(vec_sel(x1,vec_sub(vz,x1),n1),+ (__vector unsigned char)vec_sel(w1,vec_sub(vz,w1),n1),acc);+ }+ sum=vec_extract(acc,0)+vec_extract(acc,1)+vec_extract(acc,2)+vec_extract(acc,3);+#endif+ for(;i+1<I;i+=2){ uint8_t b=w4[i>>1]; sum+=((int)(b&0xF)-8)*x[i]+((int)(b>>4)-8)*x[i+1]; }+ if(i<I){ uint8_t b=w4[i>>1]; sum+=((int)(b&0xF)-8)*x[i]; }+ return sum;+}++/* ---- ARM i8mm SMMLA tiled kernels ---------------------------------------- */+#if defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)+static inline int32x4_t mm_tile16(int32x4_t acc, int8x16_t wo, int8x16_t wo1,+ int8x16_t xs, int8x16_t xs1){+ acc=vmmlaq_s32(acc, vcombine_s8(vget_low_s8(wo), vget_low_s8(wo1)),+ vcombine_s8(vget_low_s8(xs), vget_low_s8(xs1)));+ return vmmlaq_s32(acc, vcombine_s8(vget_high_s8(wo), vget_high_s8(wo1)),+ vcombine_s8(vget_high_s8(xs), vget_high_s8(xs1)));+}+static void matmul_q_idot_mm(float *y, const int8_t *xq, const float *sx, const int8_t *q,+ const float *scale, int S, int I, int O){+ #pragma omp parallel for schedule(static)+ for(int o=0;o<(O&~1);o+=2){+ const int8_t *wo=q+(int64_t)o*I, *wo1=q+(int64_t)(o+1)*I;+ float sc0=scale[o], sc1=scale[o+1];+ for(int s=0;s<(S&~1);s+=2){+ const int8_t *xs=xq+(int64_t)s*I, *xs1=xq+(int64_t)(s+1)*I;+ int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0); int i=0;+ for(;i+64<=I;i+=64){+ a0=mm_tile16(a0,vld1q_s8(wo+i), vld1q_s8(wo1+i), vld1q_s8(xs+i), vld1q_s8(xs1+i));+ a1=mm_tile16(a1,vld1q_s8(wo+i+16),vld1q_s8(wo1+i+16),vld1q_s8(xs+i+16),vld1q_s8(xs1+i+16));+ a2=mm_tile16(a2,vld1q_s8(wo+i+32),vld1q_s8(wo1+i+32),vld1q_s8(xs+i+32),vld1q_s8(xs1+i+32));+ a3=mm_tile16(a3,vld1q_s8(wo+i+48),vld1q_s8(wo1+i+48),vld1q_s8(xs+i+48),vld1q_s8(xs1+i+48));+ }+ for(;i+16<=I;i+=16)+ a0=mm_tile16(a0,vld1q_s8(wo+i),vld1q_s8(wo1+i),vld1q_s8(xs+i),vld1q_s8(xs1+i));+ int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));+ int32_t d00=vgetq_lane_s32(acc,0), d01=vgetq_lane_s32(acc,1);+ int32_t d10=vgetq_lane_s32(acc,2), d11=vgetq_lane_s32(acc,3);+ for(;i<I;i++){ int a=wo[i],b=wo1[i],u=xs[i],v=xs1[i];+ d00+=a*u; d01+=a*v; d10+=b*u; d11+=b*v; }+ y[(int64_t)s*O+o] =(float)d00*sc0*sx[s];+ y[(int64_t)s*O+(o+1)] =(float)d10*sc1*sx[s];+ y[(int64_t)(s+1)*O+o] =(float)d01*sc0*sx[s+1];+ y[(int64_t)(s+1)*O+(o+1)]=(float)d11*sc1*sx[s+1];+ }+ if(S&1){ int s=S-1; const int8_t *xs=xq+(int64_t)s*I;+ y[(int64_t)s*O+o] =(float)dot_i8i8(wo, xs,I)*sc0*sx[s];+ y[(int64_t)s*O+(o+1)]=(float)dot_i8i8(wo1,xs,I)*sc1*sx[s]; }+ }+ if(O&1){ int o=O-1; const int8_t *w=q+(int64_t)o*I; float sc=scale[o];+ #pragma omp parallel for schedule(static)+ for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i8i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }+}+static void matmul_i4_idot_mm(float *y, const int8_t *xq, const float *sx, const uint8_t *q4,+ const float *scale, int S, int I, int O){+ int rb=(I+1)/2;+ #pragma omp parallel for schedule(static)+ for(int o=0;o<(O&~1);o+=2){+ const uint8x16_t m4q=vdupq_n_u8(0x0F); const int8x16_t b8q=vdupq_n_s8(8);+ const uint8_t *wo=q4+(int64_t)o*rb, *wo1=q4+(int64_t)(o+1)*rb;+ float sc0=scale[o], sc1=scale[o+1];+ for(int s=0;s<(S&~1);s+=2){+ const int8_t *xs=xq+(int64_t)s*I, *xs1=xq+(int64_t)(s+1)*I;+ int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0); int i=0;+ for(;i+64<=I;i+=64){+ uint8x16_t byo=vld1q_u8(wo+(i>>1)), byo1=vld1q_u8(wo1+(i>>1));+ uint8x16_t cyo=vld1q_u8(wo+(i>>1)+16), cyo1=vld1q_u8(wo1+(i>>1)+16);+ uint8x16x2_t zo =vzipq_u8(vandq_u8(byo, m4q), vshrq_n_u8(byo, 4));+ uint8x16x2_t zo1=vzipq_u8(vandq_u8(byo1,m4q), vshrq_n_u8(byo1,4));+ uint8x16x2_t ko =vzipq_u8(vandq_u8(cyo, m4q), vshrq_n_u8(cyo, 4));+ uint8x16x2_t ko1=vzipq_u8(vandq_u8(cyo1,m4q), vshrq_n_u8(cyo1,4));+ a0=mm_tile16(a0, vsubq_s8(vreinterpretq_s8_u8(zo.val[0]),b8q),+ vsubq_s8(vreinterpretq_s8_u8(zo1.val[0]),b8q),+ vld1q_s8(xs+i), vld1q_s8(xs1+i));+ a1=mm_tile16(a1, vsubq_s8(vreinterpretq_s8_u8(zo.val[1]),b8q),+ vsubq_s8(vreinterpretq_s8_u8(zo1.val[1]),b8q),+ vld1q_s8(xs+i+16), vld1q_s8(xs1+i+16));+ a2=mm_tile16(a2, vsubq_s8(vreinterpretq_s8_u8(ko.val[0]),b8q),+ vsubq_s8(vreinterpretq_s8_u8(ko1.val[0]),b8q),+ vld1q_s8(xs+i+32), vld1q_s8(xs1+i+32));+ a3=mm_tile16(a3, vsubq_s8(vreinterpretq_s8_u8(ko.val[1]),b8q),+ vsubq_s8(vreinterpretq_s8_u8(ko1.val[1]),b8q),+ vld1q_s8(xs+i+48), vld1q_s8(xs1+i+48));+ }+ for(;i+32<=I;i+=32){+ uint8x16_t byo=vld1q_u8(wo+(i>>1)), byo1=vld1q_u8(wo1+(i>>1));+ uint8x16x2_t zo =vzipq_u8(vandq_u8(byo, m4q), vshrq_n_u8(byo, 4));+ uint8x16x2_t zo1=vzipq_u8(vandq_u8(byo1,m4q), vshrq_n_u8(byo1,4));+ a0=mm_tile16(a0, vsubq_s8(vreinterpretq_s8_u8(zo.val[0]),b8q),+ vsubq_s8(vreinterpretq_s8_u8(zo1.val[0]),b8q),+ vld1q_s8(xs+i), vld1q_s8(xs1+i));+ a1=mm_tile16(a1, vsubq_s8(vreinterpretq_s8_u8(zo.val[1]),b8q),+ vsubq_s8(vreinterpretq_s8_u8(zo1.val[1]),b8q),+ vld1q_s8(xs+i+16), vld1q_s8(xs1+i+16));+ }+ int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));+ int32_t d00=vgetq_lane_s32(acc,0), d01=vgetq_lane_s32(acc,1);+ int32_t d10=vgetq_lane_s32(acc,2), d11=vgetq_lane_s32(acc,3);+ for(;i+1<I;i+=2){ uint8_t bo=wo[i>>1], bo1=wo1[i>>1];+ int a0=(int)(bo&0xF)-8, a1=(int)(bo>>4)-8, b0=(int)(bo1&0xF)-8, b1=(int)(bo1>>4)-8;+ int u0=xs[i],u1=xs[i+1],v0=xs1[i],v1=xs1[i+1];+ d00+=a0*u0+a1*u1; d01+=a0*v0+a1*v1; d10+=b0*u0+b1*u1; d11+=b0*v0+b1*v1; }+ if(i<I){ uint8_t bo=wo[i>>1], bo1=wo1[i>>1];+ int a0=(int)(bo&0xF)-8, b0=(int)(bo1&0xF)-8;+ d00+=a0*xs[i]; d01+=a0*xs1[i]; d10+=b0*xs[i]; d11+=b0*xs1[i]; }+ y[(int64_t)s*O+o] =(float)d00*sc0*sx[s];+ y[(int64_t)s*O+(o+1)] =(float)d10*sc1*sx[s];+ y[(int64_t)(s+1)*O+o] =(float)d01*sc0*sx[s+1];+ y[(int64_t)(s+1)*O+(o+1)]=(float)d11*sc1*sx[s+1];+ }+ if(S&1){ int s=S-1; const int8_t *xs=xq+(int64_t)s*I;+ y[(int64_t)s*O+o] =(float)dot_i4i8(wo, xs,I)*sc0*sx[s];+ y[(int64_t)s*O+(o+1)]=(float)dot_i4i8(wo1,xs,I)*sc1*sx[s]; }+ }+ if(O&1){ int o=O-1; const uint8_t *w=q4+(int64_t)o*rb; float sc=scale[o];+ #pragma omp parallel for schedule(static)+ for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i4i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }+}+#endif++/* ---- IDOT dispatch (int8-quantized activations) --------------------------- */+static void matmul_q_idot(float *y, const int8_t *xq, const float *sx, const int8_t *q,+ const float *scale, int S, int I, int O){+#if defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)+ if(S>=2){ matmul_q_idot_mm(y,xq,sx,q,scale,S,I,O); return; }+#endif+ #pragma omp parallel for schedule(static)+ for(int o=0;o<O;o++){ const int8_t *w=q+(int64_t)o*I; float sc=scale[o];+ for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i8i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }+}+static void matmul_i4_idot(float *y, const int8_t *xq, const float *sx, const uint8_t *q4,+ const float *scale, int S, int I, int O){+ int rb=(I+1)/2;+#if defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)+ if(S>=2){ matmul_i4_idot_mm(y,xq,sx,q4,scale,S,I,O); return; }+#endif+ #pragma omp parallel for schedule(static)+ for(int o=0;o<O;o++){ const uint8_t *w=q4+(int64_t)o*rb; float sc=scale[o];+ for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i4i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }+}+++/* ================= K1: layout int4 A PIANI (fmt 2/4, opzionale) ==============+ * Layout classico ("a coppie"): byte k = elementi (2k, 2k+1). Ogni kernel paga+ * unpacklo/unpackhi per riordinare i nibble, e l'IDOT paga sub+abs+sign per+ * forzare l'operando con segno dentro vpdpbusd (che e' nativamente u8 x s8).+ *+ * Layout A PIANI, per blocco di 64 elementi consecutivi (32 byte):+ * byte k del blocco = (elem k + 8) | ((elem k+32 + 8) << 4) k = 0..31+ * `and 0x0F` produce gli elementi 0..31 GIA' in ordine; `srli 4 + and` produce+ * 32..63 in ordine: l'unpack sparisce. La coda (I mod 64) resta a coppie.+ *+ * L'algebra dell'IDOT: i nibble memorizzati sono u = v+8 (unsigned 0..15), e+ * dot(v, x) = dot(u, x) - 8 * sum(x)+ * dove sum(x) e' UN int32 per riga di attivazione, condiviso da ogni riga di+ * output. vpdpbusd consuma (u8, s8) direttamente: via sub+abs+sign (3 op+ * vettoriali per dpbusd). Misurato su questa famiglia: 1.87x L3-resident,+ * 25.3 GB/s DRAM (al tetto), 0 differenze di bit.+ * NB: allargare a 256 bit SENZA cambiare layout e' stato misurato 0.79-0.81x+ * (il fixup di lane costa piu' della larghezza): il layout e' la precondizione+ * della larghezza, non un'ottimizzazione indipendente.+ *+ * I kernel f32 planari preservano ESATTAMENTE l'ordine di accumulazione dei+ * gemelli a coppie (stesse lane, stessa sequenza 0..63): bit-identici anche+ * loro. EN: plane-nibble int4 layout. Low nibble of block byte k = element k,+ * high = element k+32; the unpack disappears, and the stored-unsigned nibbles+ * feed vpdpbusd natively via dot(v,x) = dot(u,x) - 8*sum(x). The f32 planar+ * kernels keep the pair kernels' exact accumulation order: bit-identical. */++/* riordina in place una riga (o un tensore [O,rb] riga per riga) da coppie a+ * piani; la coda I%64 resta a coppie. EN: in-place pair->planar repack. */+static void planarize_i4_row(uint8_t *row, int I){+ uint8_t tmp[32];+ int nb=I/64;+ for(int b=0;b<nb;b++){+ uint8_t *blk=row+b*32;+ /* pair: byte j ha (elem 2j, elem 2j+1). planar: byte k = (elem k, elem k+32) */+ for(int k=0;k<32;k++){+ int src_lo=k, src_hi=k+32; /* elementi voluti */+ uint8_t nib_lo=(blk[src_lo>>1]>>((src_lo&1)*4))&0xF;+ uint8_t nib_hi=(blk[src_hi>>1]>>((src_hi&1)*4))&0xF;+ tmp[k]=(uint8_t)(nib_lo|(nib_hi<<4));+ }+ memcpy(blk,tmp,32);+ }+}+static void planarize_i4(uint8_t *q4, int O, int I){+ int rb=(I+1)/2;+ #pragma omp parallel for schedule(static)+ for(int o=0;o<O;o++) planarize_i4_row(q4+(int64_t)o*rb, I);+}++/* dot unsigned-nibble planare * int8: ritorna dot(u, x) SENZA la correzione+ * -8*sum(x) (la applica il chiamante, una volta per riga di output).+ * Coda I%64: nibble a coppie letti come unsigned, stessa identita'. */+#if defined(__AVXVNNI__) || (defined(__AVX512VNNI__) && defined(__AVX512VL__))+#if defined(__AVXVNNI__) && !defined(__AVX512VNNI__)+#define coli_dpbusd256 _mm256_dpbusd_avx_epi32+#else+#define coli_dpbusd256 _mm256_dpbusd_epi32+#endif+#endif+static inline int32_t dot_i4p_u(const uint8_t *w4, const int8_t *x, int I){+ int32_t sum=0; int i=0;+#if defined(coli_dpbusd256)+ const __m256i m4=_mm256_set1_epi8(0x0F);+ __m256i a0=_mm256_setzero_si256(),a1=_mm256_setzero_si256();+ __m256i a2=_mm256_setzero_si256(),a3=_mm256_setzero_si256();+ for(;i+128<=I;i+=128){+ __m256i b0=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));+ __m256i b1=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)+32));+ a0=coli_dpbusd256(a0,_mm256_and_si256(b0,m4),+ _mm256_loadu_si256((const __m256i*)(x+i)));+ a1=coli_dpbusd256(a1,_mm256_and_si256(_mm256_srli_epi16(b0,4),m4),+ _mm256_loadu_si256((const __m256i*)(x+i+32)));+ a2=coli_dpbusd256(a2,_mm256_and_si256(b1,m4),+ _mm256_loadu_si256((const __m256i*)(x+i+64)));+ a3=coli_dpbusd256(a3,_mm256_and_si256(_mm256_srli_epi16(b1,4),m4),+ _mm256_loadu_si256((const __m256i*)(x+i+96)));+ }+ __m256i acc=_mm256_add_epi32(_mm256_add_epi32(a0,a1),_mm256_add_epi32(a2,a3));+ for(;i+64<=I;i+=64){+ __m256i b0=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));+ acc=coli_dpbusd256(acc,_mm256_and_si256(b0,m4),+ _mm256_loadu_si256((const __m256i*)(x+i)));+ acc=coli_dpbusd256(acc,_mm256_and_si256(_mm256_srli_epi16(b0,4),m4),+ _mm256_loadu_si256((const __m256i*)(x+i+32)));+ }+ sum=hsum256_i32(acc);+#elif defined(__AVX2__)+ const __m256i m4=_mm256_set1_epi8(0x0F);+ const __m256i ones=_mm256_set1_epi16(1);+ __m256i acc=_mm256_setzero_si256();+ for(;i+64<=I;i+=64){+ __m256i b0=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));+ /* maddubs(u8, s8): u<=15, |x|<=127 -> coppia <= 3810, int16 sicuro */+ __m256i p0=_mm256_maddubs_epi16(_mm256_and_si256(b0,m4),+ _mm256_loadu_si256((const __m256i*)(x+i)));+ __m256i p1=_mm256_maddubs_epi16(_mm256_and_si256(_mm256_srli_epi16(b0,4),m4),+ _mm256_loadu_si256((const __m256i*)(x+i+32)));+ acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p0,ones));+ acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p1,ones));+ }+ sum=hsum256_i32(acc);+#elif defined(__ARM_NEON)+ int32x4_t acc=vdupq_n_s32(0);+ for(;i+64<=I;i+=64){+ uint8x16_t b0=vld1q_u8(w4+(i>>1)), b1=vld1q_u8(w4+(i>>1)+16);+ int8x16_t lo0=vreinterpretq_s8_u8(vandq_u8(b0,vdupq_n_u8(0x0F)));+ int8x16_t lo1=vreinterpretq_s8_u8(vandq_u8(b1,vdupq_n_u8(0x0F)));+ int8x16_t hi0=vreinterpretq_s8_u8(vshrq_n_u8(b0,4));+ int8x16_t hi1=vreinterpretq_s8_u8(vshrq_n_u8(b1,4));+#if defined(__ARM_FEATURE_DOTPROD)+ acc=vdotq_s32(acc,lo0,vld1q_s8(x+i));+ acc=vdotq_s32(acc,lo1,vld1q_s8(x+i+16));+ acc=vdotq_s32(acc,hi0,vld1q_s8(x+i+32));+ acc=vdotq_s32(acc,hi1,vld1q_s8(x+i+48));+#else+ int8x16_t xs0=vld1q_s8(x+i), xs1=vld1q_s8(x+i+16);+ int8x16_t xs2=vld1q_s8(x+i+32), xs3=vld1q_s8(x+i+48);+ int16x8_t m;+ m=vmull_s8(vget_low_s8(lo0),vget_low_s8(xs0)); acc=vpadalq_s16(acc,m);+ m=vmull_s8(vget_high_s8(lo0),vget_high_s8(xs0)); acc=vpadalq_s16(acc,m);+ m=vmull_s8(vget_low_s8(lo1),vget_low_s8(xs1)); acc=vpadalq_s16(acc,m);+ m=vmull_s8(vget_high_s8(lo1),vget_high_s8(xs1)); acc=vpadalq_s16(acc,m);+ m=vmull_s8(vget_low_s8(hi0),vget_low_s8(xs2)); acc=vpadalq_s16(acc,m);+ m=vmull_s8(vget_high_s8(hi0),vget_high_s8(xs2)); acc=vpadalq_s16(acc,m);+ m=vmull_s8(vget_low_s8(hi1),vget_low_s8(xs3)); acc=vpadalq_s16(acc,m);+ m=vmull_s8(vget_high_s8(hi1),vget_high_s8(xs3)); acc=vpadalq_s16(acc,m);+#endif+ }+ sum=vaddvq_s32(acc);+#endif+ for(;i+64<=I;i+=64){ /* fallback scalare sui blocchi planari */+ const uint8_t *blk=w4+(i>>1);+ for(int k=0;k<32;k++){+ sum+=(int32_t)(blk[k]&0xF)*x[i+k];+ sum+=(int32_t)(blk[k]>>4)*x[i+k+32];+ }+ }+ for(;i<I;i+=2){ /* coda a coppie, unsigned (u=v+8 memorizzato) */+ uint8_t byte=w4[i>>1];+ sum+=(int32_t)(byte&0xF)*x[i];+ if(i+1<I) sum+=(int32_t)(byte>>4)*x[i+1];+ }+ return sum;+}++/* ---- K1b (OPT-IN, IDOT_GS=1): IDOT planare A GRUPPI (fmt=4, gs%64==0) -----+ * Con gs=64 il gruppo di scala COINCIDE col blocco-piano da 64 elementi: il+ * dot unsigned del blocco (2 dpbusd) -> int32 di gruppo, meno 8*somma(x) del+ * gruppo, per la scala f32 del gruppo. Attivazioni int8 (stessa famiglia+ * qrow_i8 del resto dell'IDOT): NON bit-identico al kernel f32 a gruppi --+ * per questo e' dietro flag, in attesa dell'ablazione. xsg = somme int32+ * per (riga, gruppo), calcolate dal chiamante in una passata esatta.+ * EN: grouped planar IDOT, opt-in. With gs=64 the scale group IS the plane+ * block; per-group unsigned dot minus 8*group-sum, times the group scale.+ * int8 activations: not bit-identical to the f32 grouped kernel, hence the+ * flag until the ablation blesses a default.+ *+ * Three execution shapes below, one contract: per (row, output) the float+ * accumulation is the SAME sequence of fmaf((float)group_int, scale[g], a)+ * in ascending g, and every group_int is an exact int32 β so the per-row+ * path, the 1x4 row tile, and the AMX tile are bit-identical to each other+ * and to the pure-C reference on every ISA. */++/* one planar 64-element block, unpacked once and shared across a row tile:+ * lo nibbles = elements base..base+31 in order, hi = base+32..base+63. */+#if defined(coli_dpbusd256)+static inline void i4p_blk256(const uint8_t *blk, __m256i *lo, __m256i *hi){+ const __m256i m4=_mm256_set1_epi8(0x0F);+ __m256i b=_mm256_loadu_si256((const __m256i*)blk);+ *lo=_mm256_and_si256(b,m4);+ *hi=_mm256_and_si256(_mm256_srli_epi16(b,4),m4);+}+#endif+#if defined(__AVX512VNNI__) && defined(__AVX512BW__)+/* whole block in one zmm: lanes 0..31 = lo elements, 32..63 = hi β matches a+ * straight 64-byte load of the activation block, so ONE dpbusd per block. */+static inline __m512i i4p_blk512(const uint8_t *blk){+ const __m256i m4=_mm256_set1_epi8(0x0F);+ __m256i b=_mm256_loadu_si256((const __m256i*)blk);+ return _mm512_inserti64x4(_mm512_castsi256_si512(_mm256_and_si256(b,m4)),+ _mm256_and_si256(_mm256_srli_epi16(b,4),m4),1);+}+#endif++/* ---- K1c: AMX int8 tile kernel (Sapphire Rapids+, opt-in via the same+ * IDOT_GS=1 family gate; AMX=0 kills it, AMX_S_MIN sets the row threshold).+ * With gs a multiple of 64, K=64 tile-multiplies cover a scale group exactly:+ * B tile = 16 output rows' int4 block unpacked once to SIGNED int8 (v-8, so+ * tdpbssd returns d - 8*sum(x_group) directly β the same int32 the vector+ * path computes as d_unsigned - 8*xsg), A tile = up to 16 activation rows.+ * The unpack cost is paid once per (output tile, group) and amortized over+ * every activation row β the multi-row reuse the pair-layout kernels lack.+ * Linux-only arming: tile data needs an ARCH_REQ_XCOMP_PERM handshake. */+#if defined(__AMX_INT8__) && defined(__AMX_TILE__) && defined(__AVX512F__)+#define COLI_HAVE_AMX_I4P 1+#if defined(__linux__)+#include <unistd.h>+#include <sys/syscall.h>+#elif defined(_WIN32)+#ifndef WIN32_LEAN_AND_MEAN+#define WIN32_LEAN_AND_MEAN+#endif+#include <windows.h>+#endif+static int coli_amx_state=-1;+static int coli_amx_s_min=8;+static int coli_amx_ok(void){+ if(coli_amx_state<0){+ const char *e=getenv("AMX");+ const char *sm=getenv("AMX_S_MIN"); if(sm&&atoi(sm)>0) coli_amx_s_min=atoi(sm);+#if defined(__linux__)+ /* ARCH_REQ_XCOMP_PERM(XFEATURE_XTILEDATA): kernel >= 5.16 grants tile+ * state per-process; without it the first tile op SIGILLs. */+ coli_amx_state=(e&&*e=='0')?0:(syscall(SYS_arch_prctl,0x1023,18)==0);+#elif defined(_WIN32)+ /* Windows 11: tile data is an opt-in per-process XState feature.+ * Dynamic lookup keeps older kernels building/running (arming just+ * fails closed there). XSTATE_AMX_TILE_DATA = 18. */+ if(e&&*e=='0') coli_amx_state=0;+ else {+ HMODULE k32=GetModuleHandleA("kernel32.dll");+ typedef BOOL (WINAPI *coli_xstate_fn)(ULONG64);+ coli_xstate_fn f=k32?(coli_xstate_fn)(void*)GetProcAddress(k32,"EnableProcessOptionalXStateFeatures"):NULL;+ coli_amx_state=(f && f(1ull<<18))?1:0;+ }+#else+ (void)e; coli_amx_state=0;+#endif+ if(coli_amx_state)+ fprintf(stderr,"[K1c] AMX int8 tile kernel armed for gs64 tensors "+ "(AMX=0 disables, engages at S>=%d)\n",coli_amx_s_min);+ }+ return coli_amx_state;+}+/* ldtilecfg layout (palette 1): tmm0=C [rows x 16 i32], tmm1=A [rows x 64 i8],+ * tmm2=B [16 x 64 i8 VNNI]. rows<16 reconfigures for the last partial s-tile. */+struct coli_tilecfg { uint8_t palette,start_row,rsvd[14]; uint16_t colsb[16]; uint8_t rows[16]; };+static void coli_amx_cfg(int arows){+ struct coli_tilecfg c; memset(&c,0,sizeof c); c.palette=1;+ c.rows[0]=(uint8_t)arows; c.colsb[0]=64;+ c.rows[1]=(uint8_t)arows; c.colsb[1]=64;+ c.rows[2]=16; c.colsb[2]=64;+ _tile_loadconfig(&c);+}+static void matmul_i4p_gidot_amx(float *y, const int8_t *xq, const float *sx,+ const uint8_t *q4, const float *scale,+ int S, int I, int O16, int O, int gs){+ int rb=(I+1)/2, ng=(I+gs-1)/gs, bpg=gs/64;+ #pragma omp parallel+ {+ float *acc=malloc((size_t)S*16*sizeof(float));+ if(!acc){ fprintf(stderr,"OOM: amx acc\n"); exit(1); }+ int8_t bstage[4*1024] __attribute__((aligned(64))); /* bpg<=4 gated below */+ int32_t cbuf[16*16] __attribute__((aligned(64)));+ float sclT[16];+ int cur=16; coli_amx_cfg(16);+ #pragma omp for schedule(static)+ for(int ot=0; ot<O16; ot+=16){+ memset(acc,0,(size_t)S*16*sizeof(float));+ for(int g=0; g<ng; g++){+ for(int j=0;j<16;j++) sclT[j]=scale[(int64_t)(ot+j)*ng+g];+ for(int b=0;b<bpg;b++){+ /* B tile, VNNI [k/4][n][4]: element k of output row ot+n+ * lands at row k/4, byte n*4+k%4 β signed (v-8) at unpack. */+ int8_t *dst=bstage+(size_t)b*1024;+ int64_t boff=((int64_t)g*gs+b*64)>>1;+ for(int n=0;n<16;n++){+ const uint8_t *blk=q4+(int64_t)(ot+n)*rb+boff;+ for(int k=0;k<32;k++){+ dst[(k>>2)*64+n*4+(k&3)] =(int8_t)((blk[k]&0xF)-8);+ dst[((k+32)>>2)*64+n*4+((k+32)&3)]=(int8_t)((blk[k]>>4)-8);+ }+ }+ }+ for(int st=0; st<S; st+=16){+ int rows=S-st<16?S-st:16;+ if(rows!=cur){ coli_amx_cfg(rows); cur=rows; }+ _tile_zero(0);+ for(int b=0;b<bpg;b++){+ _tile_loadd(1, xq+(int64_t)st*I+(int64_t)g*gs+b*64, (size_t)I);+ _tile_loadd(2, bstage+(size_t)b*1024, 64);+ _tile_dpbssd(0,1,2);+ }+ _tile_stored(0, cbuf, 64);+ /* per-lane fmadd == the vector path's per-(s,o) scalar fmaf:+ * same values, same ascending-g order, single rounding. */+ for(int r=0;r<rows;r++){+ __m512 cv=_mm512_cvtepi32_ps(_mm512_load_si512((const void*)(cbuf+(size_t)r*16)));+ __m512 av=_mm512_loadu_ps(acc+(int64_t)(st+r)*16);+ _mm512_storeu_ps(acc+(int64_t)(st+r)*16,+ _mm512_fmadd_ps(cv,_mm512_loadu_ps(sclT),av));+ }+ }+ }+ for(int s=0;s<S;s++)+ _mm512_storeu_ps(y+(int64_t)s*O+ot,+ _mm512_mul_ps(_mm512_loadu_ps(acc+(int64_t)s*16),+ _mm512_set1_ps(sx[s])));+ }+ _tile_release();+ free(acc);+ }+}+#endif++/* vector body over an output range [o0,o1): the 1x4 row tile pays the weight+ * block's load+mask once per 4 activation rows (the fmt=2 K2 tile's idea,+ * brought to the grouped family β the prefill batch-union and the serve mux+ * deliver exactly these multi-row calls). Integer group dots in any lane+ * order are exact, and each row keeps its own ascending-g fmaf chain. */+static void i4p_gidot_rows(float *y, const int8_t *xq, const float *sx,+ const int32_t *xsg, const uint8_t *q4,+ const float *scale, int S, int I, int O, int gs,+ int o0, int o1){+ int rb=(I+1)/2, ng=(I+gs-1)/gs, bpg=gs/64; /* blocchi-piano per gruppo */+ #pragma omp parallel for schedule(static)+ for(int o=o0;o<o1;o++){+ const uint8_t *w=q4+(int64_t)o*rb;+ const float *scl=scale+(int64_t)o*ng;+ int s=0;+ for(; s+4<=S; s+=4){+ const int8_t *x0=xq +(int64_t)s*I, *x1=x0+I, *x2=x1+I, *x3=x2+I;+ const int32_t *g0=xsg+(int64_t)s*ng, *g1=g0+ng, *g2=g1+ng, *g3=g2+ng;+ float a0=0,a1=0,a2=0,a3=0;+ int g=0;+ for(; (g+1)*gs<=I; g++){+ int32_t d0,d1,d2,d3;+#if defined(__AVX512VNNI__) && defined(__AVX512BW__)+ __m512i A0=_mm512_setzero_si512(),A1=_mm512_setzero_si512();+ __m512i A2=_mm512_setzero_si512(),A3=_mm512_setzero_si512();+ for(int b=0;b<bpg;b++){+ int base=g*gs+b*64;+ __m512i wz=i4p_blk512(w+(base>>1));+ A0=_mm512_dpbusd_epi32(A0,wz,_mm512_loadu_si512((const void*)(x0+base)));+ A1=_mm512_dpbusd_epi32(A1,wz,_mm512_loadu_si512((const void*)(x1+base)));+ A2=_mm512_dpbusd_epi32(A2,wz,_mm512_loadu_si512((const void*)(x2+base)));+ A3=_mm512_dpbusd_epi32(A3,wz,_mm512_loadu_si512((const void*)(x3+base)));+ }+ d0=_mm512_reduce_add_epi32(A0); d1=_mm512_reduce_add_epi32(A1);+ d2=_mm512_reduce_add_epi32(A2); d3=_mm512_reduce_add_epi32(A3);+#elif defined(coli_dpbusd256)+ __m256i A0=_mm256_setzero_si256(),A1=_mm256_setzero_si256();+ __m256i A2=_mm256_setzero_si256(),A3=_mm256_setzero_si256();+ for(int b=0;b<bpg;b++){+ int base=g*gs+b*64; __m256i lo,hi;+ i4p_blk256(w+(base>>1),&lo,&hi);+ A0=coli_dpbusd256(A0,lo,_mm256_loadu_si256((const __m256i*)(x0+base)));+ A0=coli_dpbusd256(A0,hi,_mm256_loadu_si256((const __m256i*)(x0+base+32)));+ A1=coli_dpbusd256(A1,lo,_mm256_loadu_si256((const __m256i*)(x1+base)));+ A1=coli_dpbusd256(A1,hi,_mm256_loadu_si256((const __m256i*)(x1+base+32)));+ A2=coli_dpbusd256(A2,lo,_mm256_loadu_si256((const __m256i*)(x2+base)));+ A2=coli_dpbusd256(A2,hi,_mm256_loadu_si256((const __m256i*)(x2+base+32)));+ A3=coli_dpbusd256(A3,lo,_mm256_loadu_si256((const __m256i*)(x3+base)));+ A3=coli_dpbusd256(A3,hi,_mm256_loadu_si256((const __m256i*)(x3+base+32)));+ }+ d0=hsum256_i32(A0); d1=hsum256_i32(A1);+ d2=hsum256_i32(A2); d3=hsum256_i32(A3);+#elif defined(__AVX2__)+ const __m256i ones=_mm256_set1_epi16(1);+ const __m256i m4=_mm256_set1_epi8(0x0F);+ __m256i A0=_mm256_setzero_si256(),A1=_mm256_setzero_si256();+ __m256i A2=_mm256_setzero_si256(),A3=_mm256_setzero_si256();+ for(int b=0;b<bpg;b++){+ int base=g*gs+b*64;+ __m256i bb=_mm256_loadu_si256((const __m256i*)(w+(base>>1)));+ __m256i lo=_mm256_and_si256(bb,m4);+ __m256i hi=_mm256_and_si256(_mm256_srli_epi16(bb,4),m4);+ /* maddubs(u8,s8): u<=15, |x|<=127 -> pair <= 3810, int16-safe */+ A0=_mm256_add_epi32(A0,_mm256_madd_epi16(_mm256_maddubs_epi16(lo,_mm256_loadu_si256((const __m256i*)(x0+base))),ones));+ A0=_mm256_add_epi32(A0,_mm256_madd_epi16(_mm256_maddubs_epi16(hi,_mm256_loadu_si256((const __m256i*)(x0+base+32))),ones));+ A1=_mm256_add_epi32(A1,_mm256_madd_epi16(_mm256_maddubs_epi16(lo,_mm256_loadu_si256((const __m256i*)(x1+base))),ones));+ A1=_mm256_add_epi32(A1,_mm256_madd_epi16(_mm256_maddubs_epi16(hi,_mm256_loadu_si256((const __m256i*)(x1+base+32))),ones));+ A2=_mm256_add_epi32(A2,_mm256_madd_epi16(_mm256_maddubs_epi16(lo,_mm256_loadu_si256((const __m256i*)(x2+base))),ones));+ A2=_mm256_add_epi32(A2,_mm256_madd_epi16(_mm256_maddubs_epi16(hi,_mm256_loadu_si256((const __m256i*)(x2+base+32))),ones));+ A3=_mm256_add_epi32(A3,_mm256_madd_epi16(_mm256_maddubs_epi16(lo,_mm256_loadu_si256((const __m256i*)(x3+base))),ones));+ A3=_mm256_add_epi32(A3,_mm256_madd_epi16(_mm256_maddubs_epi16(hi,_mm256_loadu_si256((const __m256i*)(x3+base+32))),ones));+ }+ d0=hsum256_i32(A0); d1=hsum256_i32(A1);+ d2=hsum256_i32(A2); d3=hsum256_i32(A3);+#else+ d0=d1=d2=d3=0;+ for(int b=0;b<bpg;b++){+ int base=g*gs+b*64;+ const uint8_t *blk=w+(base>>1);+ for(int k=0;k<32;k++){+ int32_t ul=(int32_t)(blk[k]&0xF), uh=(int32_t)(blk[k]>>4);+ d0+=ul*x0[base+k]+uh*x0[base+k+32];+ d1+=ul*x1[base+k]+uh*x1[base+k+32];+ d2+=ul*x2[base+k]+uh*x2[base+k+32];+ d3+=ul*x3[base+k]+uh*x3[base+k+32];+ }+ }+#endif+ a0=fmaf((float)(d0-8*g0[g]),scl[g],a0);+ a1=fmaf((float)(d1-8*g1[g]),scl[g],a1);+ a2=fmaf((float)(d2-8*g2[g]),scl[g],a2);+ a3=fmaf((float)(d3-8*g3[g]),scl[g],a3);+ }+ if(g*gs<I){ /* coda: gruppo parziale, nibble a coppie */+ int32_t d0=0,d1=0,d2=0,d3=0;+ for(int i=g*gs;i<I;i++){+ uint8_t byte=w[i>>1];+ int32_t u=(int32_t)((i&1)?(byte>>4):(byte&0xF));+ d0+=u*x0[i]; d1+=u*x1[i]; d2+=u*x2[i]; d3+=u*x3[i];+ }+ a0=fmaf((float)(d0-8*g0[g]),scl[g],a0);+ a1=fmaf((float)(d1-8*g1[g]),scl[g],a1);+ a2=fmaf((float)(d2-8*g2[g]),scl[g],a2);+ a3=fmaf((float)(d3-8*g3[g]),scl[g],a3);+ }+ y[(int64_t)s*O+o] =a0*sx[s];+ y[(int64_t)(s+1)*O+o]=a1*sx[s+1];+ y[(int64_t)(s+2)*O+o]=a2*sx[s+2];+ y[(int64_t)(s+3)*O+o]=a3*sx[s+3];+ }+ for(; s<S; s++){+ const int8_t *xr=xq+(int64_t)s*I;+ const int32_t *xg=xsg+(int64_t)s*ng;+ float a=0; int g=0;+ for(; (g+1)*gs<=I; g++){ /* gruppi interi */+ int32_t d=0;+#if defined(__AVX512VNNI__) && defined(__AVX512BW__)+ __m512i acc=_mm512_setzero_si512();+ for(int b=0;b<bpg;b++){+ int base=g*gs+b*64;+ acc=_mm512_dpbusd_epi32(acc,i4p_blk512(w+(base>>1)),+ _mm512_loadu_si512((const void*)(xr+base)));+ }+ d=_mm512_reduce_add_epi32(acc);+#else+ for(int b=0;b<bpg;b++){+ int base=g*gs+b*64;+ const uint8_t *blk=w+(base>>1);+ const int8_t *xb=xr+base;+#if defined(coli_dpbusd256)+ __m256i lo,hi; i4p_blk256(blk,&lo,&hi);+ __m256i acc=_mm256_setzero_si256();+ acc=coli_dpbusd256(acc,lo,_mm256_loadu_si256((const __m256i*)xb));+ acc=coli_dpbusd256(acc,hi,_mm256_loadu_si256((const __m256i*)(xb+32)));+ d+=hsum256_i32(acc);+#elif defined(__AVX2__)+ const __m256i m4=_mm256_set1_epi8(0x0F);+ const __m256i ones=_mm256_set1_epi16(1);+ __m256i bb=_mm256_loadu_si256((const __m256i*)blk);+ __m256i p0=_mm256_maddubs_epi16(_mm256_and_si256(bb,m4),+ _mm256_loadu_si256((const __m256i*)xb));+ __m256i p1=_mm256_maddubs_epi16(_mm256_and_si256(_mm256_srli_epi16(bb,4),m4),+ _mm256_loadu_si256((const __m256i*)(xb+32)));+ __m256i acc=_mm256_add_epi32(_mm256_madd_epi16(p0,ones),+ _mm256_madd_epi16(p1,ones));+ d+=hsum256_i32(acc);+#else+ for(int k=0;k<32;k++){+ d+=(int32_t)(blk[k]&0xF)*xb[k];+ d+=(int32_t)(blk[k]>>4)*xb[k+32];+ }+#endif+ }+#endif+ a=fmaf((float)(d-8*xg[g]),scl[g],a);+ }+ if(g*gs<I){ /* coda: gruppo parziale, nibble a coppie */+ int32_t d=0;+ for(int i=g*gs;i<I;i++){+ uint8_t byte=w[i>>1];+ d+=(int32_t)((i&1)?(byte>>4):(byte&0xF))*xr[i];+ }+ a=fmaf((float)(d-8*xg[g]),scl[g],a);+ }+ y[(int64_t)s*O+o]=a*sx[s];+ }+ }+}++static void matmul_i4p_grouped_idot(float *y, const int8_t *xq, const float *sx,+ const int32_t *xsg, const uint8_t *q4,+ const float *scale, int S, int I, int O, int gs){+#if defined(COLI_HAVE_AMX_I4P)+ /* AMX takes the aligned bulk (full 16-output tiles, full groups only β+ * the real gs64 checkpoints have I%gs==0); the vector path finishes any+ * output remainder. Below the row threshold the B-tile unpack does not+ * amortize and the vector tile is the better kernel. */+ if(coli_amx_ok() && S>=coli_amx_s_min && gs%64==0 && gs<=256 && I%gs==0 && O>=16){+ int O16=O&~15;+ matmul_i4p_gidot_amx(y,xq,sx,q4,scale,S,I,O16,O,gs);+ if(O16<O) i4p_gidot_rows(y,xq,sx,xsg,q4,scale,S,I,O,gs,O16,O);+ return;+ }+#endif+ i4p_gidot_rows(y,xq,sx,xsg,q4,scale,S,I,O,gs,0,O);+}++/* matmul IDOT planare (fmt=2): y = (dot_u - 8*xsum[s]) * scale[o] * sx[s].+ * Bit-identico a matmul_i4_idot: somme intere, identita' esatta. */+static void matmul_i4p_idot(float *y, const int8_t *xq, const float *sx, const int32_t *xsum,+ const uint8_t *q4, const float *scale, int S, int I, int O){+ int rb=(I+1)/2;+ #pragma omp parallel for schedule(static)+ for(int o=0;o<O;o++){ const uint8_t *w=q4+(int64_t)o*rb; float sc=scale[o];+ int s=0;+#if defined(coli_dpbusd256)+ /* K2: tile 1x4 sulla union β il blocco pesi (load + 2 and + srli) si+ * paga UNA volta per 4 righe di attivazione invece che per riga. La+ * union del prefill consegna nr=2..16 righe per expert: e' esattamente+ * la finestra che il per-riga serviva peggio. Somme intere ->+ * bit-identico al path per-riga per associativita'.+ * EN: 1x4 register tile β the weight block's load+mask cost is paid+ * once per 4 activation rows. Integer sums: bit-identical to the+ * per-row path by associativity. */+ const __m256i m4t=_mm256_set1_epi8(0x0F);+ for(;s+4<=S;s+=4){+ const int8_t *x0=xq+(int64_t)s*I, *x1=x0+I, *x2=x1+I, *x3=x2+I;+ __m256i a0=_mm256_setzero_si256(), a1=_mm256_setzero_si256();+ __m256i a2=_mm256_setzero_si256(), a3=_mm256_setzero_si256();+ int i=0;+ for(;i+64<=I;i+=64){+ __m256i b =_mm256_loadu_si256((const __m256i*)(w+(i>>1)));+ __m256i lo=_mm256_and_si256(b,m4t);+ __m256i hi=_mm256_and_si256(_mm256_srli_epi16(b,4),m4t);+ a0=coli_dpbusd256(a0,lo,_mm256_loadu_si256((const __m256i*)(x0+i)));+ a0=coli_dpbusd256(a0,hi,_mm256_loadu_si256((const __m256i*)(x0+i+32)));+ a1=coli_dpbusd256(a1,lo,_mm256_loadu_si256((const __m256i*)(x1+i)));+ a1=coli_dpbusd256(a1,hi,_mm256_loadu_si256((const __m256i*)(x1+i+32)));+ a2=coli_dpbusd256(a2,lo,_mm256_loadu_si256((const __m256i*)(x2+i)));+ a2=coli_dpbusd256(a2,hi,_mm256_loadu_si256((const __m256i*)(x2+i+32)));+ a3=coli_dpbusd256(a3,lo,_mm256_loadu_si256((const __m256i*)(x3+i)));+ a3=coli_dpbusd256(a3,hi,_mm256_loadu_si256((const __m256i*)(x3+i+32)));+ }+ int32_t d0=hsum256_i32(a0), d1=hsum256_i32(a1);+ int32_t d2=hsum256_i32(a2), d3=hsum256_i32(a3);+ /* coda a coppie, unsigned: stessa identita' -8*xsum del per-riga */+ for(;i<I;i+=2){+ uint8_t byte=w[i>>1];+ d0+=(int32_t)(byte&0xF)*x0[i]; d1+=(int32_t)(byte&0xF)*x1[i];+ d2+=(int32_t)(byte&0xF)*x2[i]; d3+=(int32_t)(byte&0xF)*x3[i];+ if(i+1<I){+ d0+=(int32_t)(byte>>4)*x0[i+1]; d1+=(int32_t)(byte>>4)*x1[i+1];+ d2+=(int32_t)(byte>>4)*x2[i+1]; d3+=(int32_t)(byte>>4)*x3[i+1];+ }+ }+ y[(int64_t)(s+0)*O+o]=(float)(d0-8*xsum[s+0])*sc*sx[s+0];+ y[(int64_t)(s+1)*O+o]=(float)(d1-8*xsum[s+1])*sc*sx[s+1];+ y[(int64_t)(s+2)*O+o]=(float)(d2-8*xsum[s+2])*sc*sx[s+2];+ y[(int64_t)(s+3)*O+o]=(float)(d3-8*xsum[s+3])*sc*sx[s+3];+ }+#endif+ for(;s<S;s++){+ int32_t d=dot_i4p_u(w,xq+(int64_t)s*I,I)-8*xsum[s];+ y[(int64_t)s*O+o]=(float)d*sc*sx[s];+ }+ }+}++#endif /* COLIBRI_IDOT_H */diff --git a/c/quant.h b/c/quant.hindex e2c0b314..91cf7549 100644--- a/c/quant.h+++ b/c/quant.h@@ -14,35 +14,7 @@#include <omp.h>#endif-/* ---- SIMD includes -------------------------------------------------------- */-#ifdef __AVX2__-#include <immintrin.h>-static inline float hsum256(__m256 v){- __m128 lo=_mm256_castps256_ps128(v), hi=_mm256_extractf128_ps(v,1);- lo=_mm_add_ps(lo,hi); __m128 sh=_mm_movehl_ps(lo,lo); lo=_mm_add_ps(lo,sh);- sh=_mm_shuffle_ps(lo,lo,1); lo=_mm_add_ss(lo,sh); return _mm_cvtss_f32(lo);-}-static inline int hsum256_i32(__m256i v){- __m128i lo=_mm256_castsi256_si128(v), hi=_mm256_extracti128_si256(v,1);- lo=_mm_add_epi32(lo,hi); lo=_mm_hadd_epi32(lo,lo); lo=_mm_hadd_epi32(lo,lo);- return _mm_cvtsi128_si32(lo);-}-#endif-#if defined(__AVXVNNI__) && defined(__AVX2__)-static inline int hsum128_i32(__m128i v){- v=_mm_hadd_epi32(v,v); v=_mm_hadd_epi32(v,v); return _mm_cvtsi128_si32(v);-}-#endif-#ifdef __ARM_NEON-#include <arm_neon.h>-#endif-#ifdef __VSX__-#include <altivec.h>-#undef vector-#undef pixel-#undef bool-#endif-+#include "idot.h" /* SIMD prelude + the integer dot kernels, shared with qwen36 *//* ---- AVX-512 int4->float accumulator -------------------------------------- */#if defined(__AVX512F__) && defined(__AVX512BW__)static int g_i4_acc512=1;@@ -654,931 +626,6 @@ static void matmul_fp8(float *y, const float *x, const uint8_t *q8, const float}#endif-/* ---- IDOT: integer dot kernels (int8-quantized activations) --------------- */-#if defined(__AVX512VNNI__) && defined(__AVX512BW__)-#define IDOT_KERNEL "avx512-vnni"-#elif defined(__AVXVNNI__) && defined(__AVX2__)-#define IDOT_KERNEL "avx-vnni"-#elif defined(__AVX2__)-#define IDOT_KERNEL "avx2"-#elif defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)-#define IDOT_KERNEL "neon-i8mm"-#elif defined(__ARM_NEON)-#define IDOT_KERNEL "neon"-#elif defined(__VSX__)-#define IDOT_KERNEL "vsx"-#else-#define IDOT_KERNEL "scalar"-#endif--static inline float qrow_i8(const float *x, int8_t *q, int I){- float amax=0; for(int i=0;i<I;i++){ float a=fabsf(x[i]); if(a>amax)amax=a; }- float s=amax/127.f; if(s<1e-12f) s=1e-12f; float inv=1.f/s;- for(int i=0;i<I;i++) q[i]=(int8_t)lrintf(x[i]*inv);- return s;-}--/* dot int8*int8 */-static inline int32_t dot_i8i8(const int8_t *w, const int8_t *x, int I){- int32_t sum=0; int i=0;-#if defined(__AVX512VNNI__) && defined(__AVX512BW__)- __m512i acc=_mm512_setzero_si512();- for(;i+64<=I;i+=64){- __m512i wv=_mm512_loadu_si512((const void*)(w+i));- __m512i xv=_mm512_loadu_si512((const void*)(x+i));- __mmask64 neg=_mm512_movepi8_mask(wv);- __m512i xs=_mm512_mask_sub_epi8(xv,neg,_mm512_setzero_si512(),xv);- acc=_mm512_dpbusd_epi32(acc,_mm512_abs_epi8(wv),xs);- }- sum=_mm512_reduce_add_epi32(acc);-#elif defined(__AVXVNNI__) && defined(__AVX2__)- /* 4 accumulatori indipendenti (64 byte/iter): un solo acc incatena i vpdpbusd- * (latenza-bound ~5c). Somme intere associative -> bit-identico. Stessa struttura- * dei 4 accumulatori del ramo NEON piu' sotto.- * EN: four independent accumulators break the serial vpdpbusd->acc chain; integer- * adds are associative, so the result is bit-identical (mirrors the NEON path). */- __m128i a0=_mm_setzero_si128(),a1=_mm_setzero_si128(),a2=_mm_setzero_si128(),a3=_mm_setzero_si128();- for(;i+64<=I;i+=64){- __m128i w0=_mm_loadu_si128((const __m128i*)(w+i)), x0=_mm_loadu_si128((const __m128i*)(x+i));- __m128i w1=_mm_loadu_si128((const __m128i*)(w+i+16)), x1=_mm_loadu_si128((const __m128i*)(x+i+16));- __m128i w2=_mm_loadu_si128((const __m128i*)(w+i+32)), x2=_mm_loadu_si128((const __m128i*)(x+i+32));- __m128i w3=_mm_loadu_si128((const __m128i*)(w+i+48)), x3=_mm_loadu_si128((const __m128i*)(x+i+48));- a0=_mm_dpbusd_epi32(a0,_mm_abs_epi8(w0),_mm_sign_epi8(x0,w0));- a1=_mm_dpbusd_epi32(a1,_mm_abs_epi8(w1),_mm_sign_epi8(x1,w1));- a2=_mm_dpbusd_epi32(a2,_mm_abs_epi8(w2),_mm_sign_epi8(x2,w2));- a3=_mm_dpbusd_epi32(a3,_mm_abs_epi8(w3),_mm_sign_epi8(x3,w3));- }- __m128i acc=_mm_add_epi32(_mm_add_epi32(a0,a1),_mm_add_epi32(a2,a3));- for(;i+16<=I;i+=16){- __m128i wv=_mm_loadu_si128((const __m128i*)(w+i));- __m128i xv=_mm_loadu_si128((const __m128i*)(x+i));- acc=_mm_dpbusd_epi32(acc,_mm_abs_epi8(wv),_mm_sign_epi8(xv,wv));- }- sum=hsum128_i32(acc);-#elif defined(__AVX2__)- __m256i acc=_mm256_setzero_si256(); const __m256i ones=_mm256_set1_epi16(1);- for(;i+32<=I;i+=32){- __m256i wv=_mm256_loadu_si256((const __m256i*)(w+i));- __m256i xv=_mm256_loadu_si256((const __m256i*)(x+i));- __m256i p=_mm256_maddubs_epi16(_mm256_sign_epi8(wv,wv),_mm256_sign_epi8(xv,wv));- acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p,ones));- }- sum=hsum256_i32(acc);-#elif defined(__ARM_NEON)-#if defined(__ARM_FEATURE_DOTPROD)- int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0);- for(;i+64<=I;i+=64){- a0=vdotq_s32(a0,vld1q_s8(w+i), vld1q_s8(x+i));- a1=vdotq_s32(a1,vld1q_s8(w+i+16),vld1q_s8(x+i+16));- a2=vdotq_s32(a2,vld1q_s8(w+i+32),vld1q_s8(x+i+32));- a3=vdotq_s32(a3,vld1q_s8(w+i+48),vld1q_s8(x+i+48));- }- int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));- for(;i+16<=I;i+=16) acc=vdotq_s32(acc,vld1q_s8(w+i),vld1q_s8(x+i));- sum=vaddvq_s32(acc);-#else- int32x4_t acc=vdupq_n_s32(0);- for(;i+16<=I;i+=16){- int8x16_t wv=vld1q_s8(w+i), xv=vld1q_s8(x+i);- int16x8_t p=vmull_s8(vget_low_s8(wv),vget_low_s8(xv));- p=vmlal_s8(p,vget_high_s8(wv),vget_high_s8(xv));- acc=vpadalq_s16(acc,p);- }- sum=vaddvq_s32(acc);-#endif-#elif defined(__VSX__)- __vector signed int acc=vec_splats(0);- const __vector signed char vz=vec_splats((signed char)0);- for(;i+16<=I;i+=16){- __vector signed char wv=vec_xl(0,(const signed char*)(w+i));- __vector signed char xv=vec_xl(0,(const signed char*)(x+i));- __vector __bool char neg=vec_cmplt(wv,vz);- __vector signed char xs=vec_sel(xv,vec_sub(vz,xv),neg);- __vector unsigned char wa=(__vector unsigned char)vec_sel(wv,vec_sub(vz,wv),neg);- acc=vec_msum(xs,wa,acc);- }- sum=vec_extract(acc,0)+vec_extract(acc,1)+vec_extract(acc,2)+vec_extract(acc,3);-#endif- for(;i<I;i++) sum+=(int32_t)w[i]*x[i];- return sum;-}--/* dot int4(packed)*int8 */-static inline int32_t dot_i4i8(const uint8_t *w4, const int8_t *x, int I){- int32_t sum=0; int i=0;-#if defined(__AVX512VNNI__) && defined(__AVX512BW__)- const __m256i m4v=_mm256_set1_epi8(0x0F);- const __m512i b8v=_mm512_set1_epi8(8);- const __m512i xidx=_mm512_setr_epi64(0,1,4,5,2,3,6,7);- __m512i acc=_mm512_setzero_si512();- for(;i+64<=I;i+=64){- __m256i by=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));- __m256i lo=_mm256_and_si256(by,m4v), hi=_mm256_and_si256(_mm256_srli_epi16(by,4),m4v);- __m256i z0=_mm256_unpacklo_epi8(lo,hi), z1=_mm256_unpackhi_epi8(lo,hi);- __m512i wv=_mm512_sub_epi8(_mm512_inserti64x4(_mm512_castsi256_si512(z0),z1,1),b8v);- __m512i xv=_mm512_permutexvar_epi64(xidx,_mm512_loadu_si512((const void*)(x+i)));- __mmask64 neg=_mm512_movepi8_mask(wv);- __m512i xs=_mm512_mask_sub_epi8(xv,neg,_mm512_setzero_si512(),xv);- acc=_mm512_dpbusd_epi32(acc,_mm512_abs_epi8(wv),xs);- }- sum=_mm512_reduce_add_epi32(acc);-#elif defined(__AVXVNNI__) && defined(__AVX2__)- /* 4 accumulatori indipendenti (64 elementi = 32 byte packed/iter): un solo acc- * incatena i vpdpbusd (latenza-bound ~5c). Somme intere associative -> bit-identico.- * Stessa struttura dei 4 accumulatori del ramo NEON piu' sotto.- * EN: four independent accumulators break the serial vpdpbusd->acc chain; integer- * adds are associative, so the result is bit-identical (mirrors the NEON path). */- const __m128i m4=_mm_set1_epi8(0x0F); const __m128i b8=_mm_set1_epi8(8);- __m128i a0=_mm_setzero_si128(),a1=_mm_setzero_si128(),a2=_mm_setzero_si128(),a3=_mm_setzero_si128();- for(;i+64<=I;i+=64){- __m128i by0=_mm_loadu_si128((const __m128i*)(w4+(i>>1))); /* elem i..i+31 */- __m128i by1=_mm_loadu_si128((const __m128i*)(w4+(i>>1)+16)); /* elem i+32..i+63 */- __m128i lo0=_mm_and_si128(by0,m4), hi0=_mm_and_si128(_mm_srli_epi16(by0,4),m4);- __m128i lo1=_mm_and_si128(by1,m4), hi1=_mm_and_si128(_mm_srli_epi16(by1,4),m4);- __m128i w0=_mm_sub_epi8(_mm_unpacklo_epi8(lo0,hi0),b8), w1=_mm_sub_epi8(_mm_unpackhi_epi8(lo0,hi0),b8);- __m128i w2=_mm_sub_epi8(_mm_unpacklo_epi8(lo1,hi1),b8), w3=_mm_sub_epi8(_mm_unpackhi_epi8(lo1,hi1),b8);- __m128i x0=_mm_loadu_si128((const __m128i*)(x+i)), x1=_mm_loadu_si128((const __m128i*)(x+i+16));- __m128i x2=_mm_loadu_si128((const __m128i*)(x+i+32)), x3=_mm_loadu_si128((const __m128i*)(x+i+48));- a0=_mm_dpbusd_epi32(a0,_mm_abs_epi8(w0),_mm_sign_epi8(x0,w0));- a1=_mm_dpbusd_epi32(a1,_mm_abs_epi8(w1),_mm_sign_epi8(x1,w1));- a2=_mm_dpbusd_epi32(a2,_mm_abs_epi8(w2),_mm_sign_epi8(x2,w2));- a3=_mm_dpbusd_epi32(a3,_mm_abs_epi8(w3),_mm_sign_epi8(x3,w3));- }- __m128i acc=_mm_add_epi32(_mm_add_epi32(a0,a1),_mm_add_epi32(a2,a3));- for(;i+32<=I;i+=32){ /* 32-nibble remainder: 2 dpbusd, same unpack */- __m128i by=_mm_loadu_si128((const __m128i*)(w4+(i>>1)));- __m128i lo=_mm_and_si128(by,m4), hi=_mm_and_si128(_mm_srli_epi16(by,4),m4);- __m128i w0=_mm_sub_epi8(_mm_unpacklo_epi8(lo,hi),b8), w1=_mm_sub_epi8(_mm_unpackhi_epi8(lo,hi),b8);- __m128i x0=_mm_loadu_si128((const __m128i*)(x+i));- __m128i x1=_mm_loadu_si128((const __m128i*)(x+i+16));- acc=_mm_dpbusd_epi32(acc,_mm_abs_epi8(w0),_mm_sign_epi8(x0,w0));- acc=_mm_dpbusd_epi32(acc,_mm_abs_epi8(w1),_mm_sign_epi8(x1,w1));- }- sum=hsum128_i32(acc);-#elif defined(__AVX2__)- const __m128i m4=_mm_set1_epi8(0x0F); const __m256i b8=_mm256_set1_epi8(8);- const __m256i ones=_mm256_set1_epi16(1);- __m256i acc=_mm256_setzero_si256();- for(;i+32<=I;i+=32){- __m128i by=_mm_loadu_si128((const __m128i*)(w4+(i>>1)));- __m128i lo=_mm_and_si128(by,m4), hi=_mm_and_si128(_mm_srli_epi16(by,4),m4);- __m128i n0=_mm_unpacklo_epi8(lo,hi), n1=_mm_unpackhi_epi8(lo,hi);- __m256i wv=_mm256_sub_epi8(_mm256_set_m128i(n1,n0),b8);- __m256i xv=_mm256_loadu_si256((const __m256i*)(x+i));- __m256i p=_mm256_maddubs_epi16(_mm256_sign_epi8(wv,wv),_mm256_sign_epi8(xv,wv));- acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p,ones));- }- sum=hsum256_i32(acc);-#elif defined(__ARM_NEON)- const uint8x16_t m4q=vdupq_n_u8(0x0F); const int8x16_t b8q=vdupq_n_s8(8);-#if defined(__ARM_FEATURE_DOTPROD)- int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0);- for(;i+64<=I;i+=64){- uint8x16_t byA=vld1q_u8(w4+(i>>1)), byB=vld1q_u8(w4+(i>>1)+16);- uint8x16x2_t zA=vzipq_u8(vandq_u8(byA,m4q), vshrq_n_u8(byA,4));- uint8x16x2_t zB=vzipq_u8(vandq_u8(byB,m4q), vshrq_n_u8(byB,4));- a0=vdotq_s32(a0,vsubq_s8(vreinterpretq_s8_u8(zA.val[0]),b8q),vld1q_s8(x+i));- a1=vdotq_s32(a1,vsubq_s8(vreinterpretq_s8_u8(zA.val[1]),b8q),vld1q_s8(x+i+16));- a2=vdotq_s32(a2,vsubq_s8(vreinterpretq_s8_u8(zB.val[0]),b8q),vld1q_s8(x+i+32));- a3=vdotq_s32(a3,vsubq_s8(vreinterpretq_s8_u8(zB.val[1]),b8q),vld1q_s8(x+i+48));- }- int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));- for(;i+32<=I;i+=32){- uint8x16_t by=vld1q_u8(w4+(i>>1));- uint8x16x2_t z=vzipq_u8(vandq_u8(by,m4q), vshrq_n_u8(by,4));- acc=vdotq_s32(acc,vsubq_s8(vreinterpretq_s8_u8(z.val[0]),b8q),vld1q_s8(x+i));- acc=vdotq_s32(acc,vsubq_s8(vreinterpretq_s8_u8(z.val[1]),b8q),vld1q_s8(x+i+16));- }- sum=vaddvq_s32(acc);-#else- int32x4_t acc=vdupq_n_s32(0);- for(;i+32<=I;i+=32){- uint8x16_t by=vld1q_u8(w4+(i>>1));- uint8x16x2_t z=vzipq_u8(vandq_u8(by,m4q), vshrq_n_u8(by,4));- int8x16_t w0=vsubq_s8(vreinterpretq_s8_u8(z.val[0]),b8q);- int8x16_t w1=vsubq_s8(vreinterpretq_s8_u8(z.val[1]),b8q);- int8x16_t x0=vld1q_s8(x+i), x1=vld1q_s8(x+i+16);- int16x8_t p=vmull_s8(vget_low_s8(w0),vget_low_s8(x0));- p=vmlal_s8(p,vget_high_s8(w0),vget_high_s8(x0));- acc=vpadalq_s16(acc,p);- p=vmull_s8(vget_low_s8(w1),vget_low_s8(x1));- p=vmlal_s8(p,vget_high_s8(w1),vget_high_s8(x1));- acc=vpadalq_s16(acc,p);- }- sum=vaddvq_s32(acc);-#endif-#elif defined(__VSX__)- const __vector unsigned char m4v=vec_splats((unsigned char)0x0F);- const __vector unsigned char sh4=vec_splats((unsigned char)4);- const __vector signed char b8v=vec_splats((signed char)8);- const __vector signed char vz=vec_splats((signed char)0);- __vector signed int acc=vec_splats(0);- for(;i+32<=I;i+=32){- __vector unsigned char by=vec_xl(0,w4+(i>>1));- __vector unsigned char lo=vec_and(by,m4v), hi=vec_sr(by,sh4);- __vector signed char w0=vec_sub((__vector signed char)vec_mergeh(lo,hi),b8v);- __vector signed char w1=vec_sub((__vector signed char)vec_mergel(lo,hi),b8v);- __vector signed char x0=vec_xl(0,(const signed char*)(x+i));- __vector signed char x1=vec_xl(0,(const signed char*)(x+i+16));- __vector __bool char n0=vec_cmplt(w0,vz), n1=vec_cmplt(w1,vz);- acc=vec_msum(vec_sel(x0,vec_sub(vz,x0),n0),- (__vector unsigned char)vec_sel(w0,vec_sub(vz,w0),n0),acc);- acc=vec_msum(vec_sel(x1,vec_sub(vz,x1),n1),- (__vector unsigned char)vec_sel(w1,vec_sub(vz,w1),n1),acc);- }- sum=vec_extract(acc,0)+vec_extract(acc,1)+vec_extract(acc,2)+vec_extract(acc,3);-#endif- for(;i+1<I;i+=2){ uint8_t b=w4[i>>1]; sum+=((int)(b&0xF)-8)*x[i]+((int)(b>>4)-8)*x[i+1]; }- if(i<I){ uint8_t b=w4[i>>1]; sum+=((int)(b&0xF)-8)*x[i]; }- return sum;-}--/* ---- ARM i8mm SMMLA tiled kernels ---------------------------------------- */-#if defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)-static inline int32x4_t mm_tile16(int32x4_t acc, int8x16_t wo, int8x16_t wo1,- int8x16_t xs, int8x16_t xs1){- acc=vmmlaq_s32(acc, vcombine_s8(vget_low_s8(wo), vget_low_s8(wo1)),- vcombine_s8(vget_low_s8(xs), vget_low_s8(xs1)));- return vmmlaq_s32(acc, vcombine_s8(vget_high_s8(wo), vget_high_s8(wo1)),- vcombine_s8(vget_high_s8(xs), vget_high_s8(xs1)));-}-static void matmul_q_idot_mm(float *y, const int8_t *xq, const float *sx, const int8_t *q,- const float *scale, int S, int I, int O){- #pragma omp parallel for schedule(static)- for(int o=0;o<(O&~1);o+=2){- const int8_t *wo=q+(int64_t)o*I, *wo1=q+(int64_t)(o+1)*I;- float sc0=scale[o], sc1=scale[o+1];- for(int s=0;s<(S&~1);s+=2){- const int8_t *xs=xq+(int64_t)s*I, *xs1=xq+(int64_t)(s+1)*I;- int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0); int i=0;- for(;i+64<=I;i+=64){- a0=mm_tile16(a0,vld1q_s8(wo+i), vld1q_s8(wo1+i), vld1q_s8(xs+i), vld1q_s8(xs1+i));- a1=mm_tile16(a1,vld1q_s8(wo+i+16),vld1q_s8(wo1+i+16),vld1q_s8(xs+i+16),vld1q_s8(xs1+i+16));- a2=mm_tile16(a2,vld1q_s8(wo+i+32),vld1q_s8(wo1+i+32),vld1q_s8(xs+i+32),vld1q_s8(xs1+i+32));- a3=mm_tile16(a3,vld1q_s8(wo+i+48),vld1q_s8(wo1+i+48),vld1q_s8(xs+i+48),vld1q_s8(xs1+i+48));- }- for(;i+16<=I;i+=16)- a0=mm_tile16(a0,vld1q_s8(wo+i),vld1q_s8(wo1+i),vld1q_s8(xs+i),vld1q_s8(xs1+i));- int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));- int32_t d00=vgetq_lane_s32(acc,0), d01=vgetq_lane_s32(acc,1);- int32_t d10=vgetq_lane_s32(acc,2), d11=vgetq_lane_s32(acc,3);- for(;i<I;i++){ int a=wo[i],b=wo1[i],u=xs[i],v=xs1[i];- d00+=a*u; d01+=a*v; d10+=b*u; d11+=b*v; }- y[(int64_t)s*O+o] =(float)d00*sc0*sx[s];- y[(int64_t)s*O+(o+1)] =(float)d10*sc1*sx[s];- y[(int64_t)(s+1)*O+o] =(float)d01*sc0*sx[s+1];- y[(int64_t)(s+1)*O+(o+1)]=(float)d11*sc1*sx[s+1];- }- if(S&1){ int s=S-1; const int8_t *xs=xq+(int64_t)s*I;- y[(int64_t)s*O+o] =(float)dot_i8i8(wo, xs,I)*sc0*sx[s];- y[(int64_t)s*O+(o+1)]=(float)dot_i8i8(wo1,xs,I)*sc1*sx[s]; }- }- if(O&1){ int o=O-1; const int8_t *w=q+(int64_t)o*I; float sc=scale[o];- #pragma omp parallel for schedule(static)- for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i8i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }-}-static void matmul_i4_idot_mm(float *y, const int8_t *xq, const float *sx, const uint8_t *q4,- const float *scale, int S, int I, int O){- int rb=(I+1)/2;- #pragma omp parallel for schedule(static)- for(int o=0;o<(O&~1);o+=2){- const uint8x16_t m4q=vdupq_n_u8(0x0F); const int8x16_t b8q=vdupq_n_s8(8);- const uint8_t *wo=q4+(int64_t)o*rb, *wo1=q4+(int64_t)(o+1)*rb;- float sc0=scale[o], sc1=scale[o+1];- for(int s=0;s<(S&~1);s+=2){- const int8_t *xs=xq+(int64_t)s*I, *xs1=xq+(int64_t)(s+1)*I;- int32x4_t a0=vdupq_n_s32(0),a1=vdupq_n_s32(0),a2=vdupq_n_s32(0),a3=vdupq_n_s32(0); int i=0;- for(;i+64<=I;i+=64){- uint8x16_t byo=vld1q_u8(wo+(i>>1)), byo1=vld1q_u8(wo1+(i>>1));- uint8x16_t cyo=vld1q_u8(wo+(i>>1)+16), cyo1=vld1q_u8(wo1+(i>>1)+16);- uint8x16x2_t zo =vzipq_u8(vandq_u8(byo, m4q), vshrq_n_u8(byo, 4));- uint8x16x2_t zo1=vzipq_u8(vandq_u8(byo1,m4q), vshrq_n_u8(byo1,4));- uint8x16x2_t ko =vzipq_u8(vandq_u8(cyo, m4q), vshrq_n_u8(cyo, 4));- uint8x16x2_t ko1=vzipq_u8(vandq_u8(cyo1,m4q), vshrq_n_u8(cyo1,4));- a0=mm_tile16(a0, vsubq_s8(vreinterpretq_s8_u8(zo.val[0]),b8q),- vsubq_s8(vreinterpretq_s8_u8(zo1.val[0]),b8q),- vld1q_s8(xs+i), vld1q_s8(xs1+i));- a1=mm_tile16(a1, vsubq_s8(vreinterpretq_s8_u8(zo.val[1]),b8q),- vsubq_s8(vreinterpretq_s8_u8(zo1.val[1]),b8q),- vld1q_s8(xs+i+16), vld1q_s8(xs1+i+16));- a2=mm_tile16(a2, vsubq_s8(vreinterpretq_s8_u8(ko.val[0]),b8q),- vsubq_s8(vreinterpretq_s8_u8(ko1.val[0]),b8q),- vld1q_s8(xs+i+32), vld1q_s8(xs1+i+32));- a3=mm_tile16(a3, vsubq_s8(vreinterpretq_s8_u8(ko.val[1]),b8q),- vsubq_s8(vreinterpretq_s8_u8(ko1.val[1]),b8q),- vld1q_s8(xs+i+48), vld1q_s8(xs1+i+48));- }- for(;i+32<=I;i+=32){- uint8x16_t byo=vld1q_u8(wo+(i>>1)), byo1=vld1q_u8(wo1+(i>>1));- uint8x16x2_t zo =vzipq_u8(vandq_u8(byo, m4q), vshrq_n_u8(byo, 4));- uint8x16x2_t zo1=vzipq_u8(vandq_u8(byo1,m4q), vshrq_n_u8(byo1,4));- a0=mm_tile16(a0, vsubq_s8(vreinterpretq_s8_u8(zo.val[0]),b8q),- vsubq_s8(vreinterpretq_s8_u8(zo1.val[0]),b8q),- vld1q_s8(xs+i), vld1q_s8(xs1+i));- a1=mm_tile16(a1, vsubq_s8(vreinterpretq_s8_u8(zo.val[1]),b8q),- vsubq_s8(vreinterpretq_s8_u8(zo1.val[1]),b8q),- vld1q_s8(xs+i+16), vld1q_s8(xs1+i+16));- }- int32x4_t acc=vaddq_s32(vaddq_s32(a0,a1),vaddq_s32(a2,a3));- int32_t d00=vgetq_lane_s32(acc,0), d01=vgetq_lane_s32(acc,1);- int32_t d10=vgetq_lane_s32(acc,2), d11=vgetq_lane_s32(acc,3);- for(;i+1<I;i+=2){ uint8_t bo=wo[i>>1], bo1=wo1[i>>1];- int a0=(int)(bo&0xF)-8, a1=(int)(bo>>4)-8, b0=(int)(bo1&0xF)-8, b1=(int)(bo1>>4)-8;- int u0=xs[i],u1=xs[i+1],v0=xs1[i],v1=xs1[i+1];- d00+=a0*u0+a1*u1; d01+=a0*v0+a1*v1; d10+=b0*u0+b1*u1; d11+=b0*v0+b1*v1; }- if(i<I){ uint8_t bo=wo[i>>1], bo1=wo1[i>>1];- int a0=(int)(bo&0xF)-8, b0=(int)(bo1&0xF)-8;- d00+=a0*xs[i]; d01+=a0*xs1[i]; d10+=b0*xs[i]; d11+=b0*xs1[i]; }- y[(int64_t)s*O+o] =(float)d00*sc0*sx[s];- y[(int64_t)s*O+(o+1)] =(float)d10*sc1*sx[s];- y[(int64_t)(s+1)*O+o] =(float)d01*sc0*sx[s+1];- y[(int64_t)(s+1)*O+(o+1)]=(float)d11*sc1*sx[s+1];- }- if(S&1){ int s=S-1; const int8_t *xs=xq+(int64_t)s*I;- y[(int64_t)s*O+o] =(float)dot_i4i8(wo, xs,I)*sc0*sx[s];- y[(int64_t)s*O+(o+1)]=(float)dot_i4i8(wo1,xs,I)*sc1*sx[s]; }- }- if(O&1){ int o=O-1; const uint8_t *w=q4+(int64_t)o*rb; float sc=scale[o];- #pragma omp parallel for schedule(static)- for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i4i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }-}-#endif--/* ---- IDOT dispatch (int8-quantized activations) --------------------------- */-static void matmul_q_idot(float *y, const int8_t *xq, const float *sx, const int8_t *q,- const float *scale, int S, int I, int O){-#if defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)- if(S>=2){ matmul_q_idot_mm(y,xq,sx,q,scale,S,I,O); return; }-#endif- #pragma omp parallel for schedule(static)- for(int o=0;o<O;o++){ const int8_t *w=q+(int64_t)o*I; float sc=scale[o];- for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i8i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }-}-static void matmul_i4_idot(float *y, const int8_t *xq, const float *sx, const uint8_t *q4,- const float *scale, int S, int I, int O){- int rb=(I+1)/2;-#if defined(__ARM_NEON) && defined(__ARM_FEATURE_MATMUL_INT8)- if(S>=2){ matmul_i4_idot_mm(y,xq,sx,q4,scale,S,I,O); return; }-#endif- #pragma omp parallel for schedule(static)- for(int o=0;o<O;o++){ const uint8_t *w=q4+(int64_t)o*rb; float sc=scale[o];- for(int s=0;s<S;s++) y[(int64_t)s*O+o]=(float)dot_i4i8(w,xq+(int64_t)s*I,I)*sc*sx[s]; }-}---/* ================= K1: layout int4 A PIANI (fmt 2/4, opzionale) ==============- * Layout classico ("a coppie"): byte k = elementi (2k, 2k+1). Ogni kernel paga- * unpacklo/unpackhi per riordinare i nibble, e l'IDOT paga sub+abs+sign per- * forzare l'operando con segno dentro vpdpbusd (che e' nativamente u8 x s8).- *- * Layout A PIANI, per blocco di 64 elementi consecutivi (32 byte):- * byte k del blocco = (elem k + 8) | ((elem k+32 + 8) << 4) k = 0..31- * `and 0x0F` produce gli elementi 0..31 GIA' in ordine; `srli 4 + and` produce- * 32..63 in ordine: l'unpack sparisce. La coda (I mod 64) resta a coppie.- *- * L'algebra dell'IDOT: i nibble memorizzati sono u = v+8 (unsigned 0..15), e- * dot(v, x) = dot(u, x) - 8 * sum(x)- * dove sum(x) e' UN int32 per riga di attivazione, condiviso da ogni riga di- * output. vpdpbusd consuma (u8, s8) direttamente: via sub+abs+sign (3 op- * vettoriali per dpbusd). Misurato su questa famiglia: 1.87x L3-resident,- * 25.3 GB/s DRAM (al tetto), 0 differenze di bit.- * NB: allargare a 256 bit SENZA cambiare layout e' stato misurato 0.79-0.81x- * (il fixup di lane costa piu' della larghezza): il layout e' la precondizione- * della larghezza, non un'ottimizzazione indipendente.- *- * I kernel f32 planari preservano ESATTAMENTE l'ordine di accumulazione dei- * gemelli a coppie (stesse lane, stessa sequenza 0..63): bit-identici anche- * loro. EN: plane-nibble int4 layout. Low nibble of block byte k = element k,- * high = element k+32; the unpack disappears, and the stored-unsigned nibbles- * feed vpdpbusd natively via dot(v,x) = dot(u,x) - 8*sum(x). The f32 planar- * kernels keep the pair kernels' exact accumulation order: bit-identical. */--/* riordina in place una riga (o un tensore [O,rb] riga per riga) da coppie a- * piani; la coda I%64 resta a coppie. EN: in-place pair->planar repack. */-static void planarize_i4_row(uint8_t *row, int I){- uint8_t tmp[32];- int nb=I/64;- for(int b=0;b<nb;b++){- uint8_t *blk=row+b*32;- /* pair: byte j ha (elem 2j, elem 2j+1). planar: byte k = (elem k, elem k+32) */- for(int k=0;k<32;k++){- int src_lo=k, src_hi=k+32; /* elementi voluti */- uint8_t nib_lo=(blk[src_lo>>1]>>((src_lo&1)*4))&0xF;- uint8_t nib_hi=(blk[src_hi>>1]>>((src_hi&1)*4))&0xF;- tmp[k]=(uint8_t)(nib_lo|(nib_hi<<4));- }- memcpy(blk,tmp,32);- }-}-static void planarize_i4(uint8_t *q4, int O, int I){- int rb=(I+1)/2;- #pragma omp parallel for schedule(static)- for(int o=0;o<O;o++) planarize_i4_row(q4+(int64_t)o*rb, I);-}--/* dot unsigned-nibble planare * int8: ritorna dot(u, x) SENZA la correzione- * -8*sum(x) (la applica il chiamante, una volta per riga di output).- * Coda I%64: nibble a coppie letti come unsigned, stessa identita'. */-#if defined(__AVXVNNI__) || (defined(__AVX512VNNI__) && defined(__AVX512VL__))-#if defined(__AVXVNNI__) && !defined(__AVX512VNNI__)-#define coli_dpbusd256 _mm256_dpbusd_avx_epi32-#else-#define coli_dpbusd256 _mm256_dpbusd_epi32-#endif-#endif-static inline int32_t dot_i4p_u(const uint8_t *w4, const int8_t *x, int I){- int32_t sum=0; int i=0;-#if defined(coli_dpbusd256)- const __m256i m4=_mm256_set1_epi8(0x0F);- __m256i a0=_mm256_setzero_si256(),a1=_mm256_setzero_si256();- __m256i a2=_mm256_setzero_si256(),a3=_mm256_setzero_si256();- for(;i+128<=I;i+=128){- __m256i b0=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));- __m256i b1=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)+32));- a0=coli_dpbusd256(a0,_mm256_and_si256(b0,m4),- _mm256_loadu_si256((const __m256i*)(x+i)));- a1=coli_dpbusd256(a1,_mm256_and_si256(_mm256_srli_epi16(b0,4),m4),- _mm256_loadu_si256((const __m256i*)(x+i+32)));- a2=coli_dpbusd256(a2,_mm256_and_si256(b1,m4),- _mm256_loadu_si256((const __m256i*)(x+i+64)));- a3=coli_dpbusd256(a3,_mm256_and_si256(_mm256_srli_epi16(b1,4),m4),- _mm256_loadu_si256((const __m256i*)(x+i+96)));- }- __m256i acc=_mm256_add_epi32(_mm256_add_epi32(a0,a1),_mm256_add_epi32(a2,a3));- for(;i+64<=I;i+=64){- __m256i b0=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));- acc=coli_dpbusd256(acc,_mm256_and_si256(b0,m4),- _mm256_loadu_si256((const __m256i*)(x+i)));- acc=coli_dpbusd256(acc,_mm256_and_si256(_mm256_srli_epi16(b0,4),m4),- _mm256_loadu_si256((const __m256i*)(x+i+32)));- }- sum=hsum256_i32(acc);-#elif defined(__AVX2__)- const __m256i m4=_mm256_set1_epi8(0x0F);- const __m256i ones=_mm256_set1_epi16(1);- __m256i acc=_mm256_setzero_si256();- for(;i+64<=I;i+=64){- __m256i b0=_mm256_loadu_si256((const __m256i*)(w4+(i>>1)));- /* maddubs(u8, s8): u<=15, |x|<=127 -> coppia <= 3810, int16 sicuro */- __m256i p0=_mm256_maddubs_epi16(_mm256_and_si256(b0,m4),- _mm256_loadu_si256((const __m256i*)(x+i)));- __m256i p1=_mm256_maddubs_epi16(_mm256_and_si256(_mm256_srli_epi16(b0,4),m4),- _mm256_loadu_si256((const __m256i*)(x+i+32)));- acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p0,ones));- acc=_mm256_add_epi32(acc,_mm256_madd_epi16(p1,ones));- }- sum=hsum256_i32(acc);-#elif defined(__ARM_NEON)- int32x4_t acc=vdupq_n_s32(0);- for(;i+64<=I;i+=64){- uint8x16_t b0=vld1q_u8(w4+(i>>1)), b1=vld1q_u8(w4+(i>>1)+16);- int8x16_t lo0=vreinterpretq_s8_u8(vandq_u8(b0,vdupq_n_u8(0x0F)));- int8x16_t lo1=vreinterpretq_s8_u8(vandq_u8(b1,vdupq_n_u8(0x0F)));- int8x16_t hi0=vreinterpretq_s8_u8(vshrq_n_u8(b0,4));- int8x16_t hi1=vreinterpretq_s8_u8(vshrq_n_u8(b1,4));-#if defined(__ARM_FEATURE_DOTPROD)- acc=vdotq_s32(acc,lo0,vld1q_s8(x+i));- acc=vdotq_s32(acc,lo1,vld1q_s8(x+i+16));- acc=vdotq_s32(acc,hi0,vld1q_s8(x+i+32));- acc=vdotq_s32(acc,hi1,vld1q_s8(x+i+48));-#else- int8x16_t xs0=vld1q_s8(x+i), xs1=vld1q_s8(x+i+16);- int8x16_t xs2=vld1q_s8(x+i+32), xs3=vld1q_s8(x+i+48);- int16x8_t m;- m=vmull_s8(vget_low_s8(lo0),vget_low_s8(xs0)); acc=vpadalq_s16(acc,m);- m=vmull_s8(vget_high_s8(lo0),vget_high_s8(xs0)); acc=vpadalq_s16(acc,m);- m=vmull_s8(vget_low_s8(lo1),vget_low_s8(xs1)); acc=vpadalq_s16(acc,m);- m=vmull_s8(vget_high_s8(lo1),vget_high_s8(xs1)); acc=vpadalq_s16(acc,m);- m=vmull_s8(vget_low_s8(hi0),vget_low_s8(xs2)); acc=vpadalq_s16(acc,m);- m=vmull_s8(vget_high_s8(hi0),vget_high_s8(xs2)); acc=vpadalq_s16(acc,m);- m=vmull_s8(vget_low_s8(hi1),vget_low_s8(xs3)); acc=vpadalq_s16(acc,m);- m=vmull_s8(vget_high_s8(hi1),vget_high_s8(xs3)); acc=vpadalq_s16(acc,m);-#endif- }- sum=vaddvq_s32(acc);-#endif- for(;i+64<=I;i+=64){ /* fallback scalare sui blocchi planari */- const uint8_t *blk=w4+(i>>1);- for(int k=0;k<32;k++){- sum+=(int32_t)(blk[k]&0xF)*x[i+k];- sum+=(int32_t)(blk[k]>>4)*x[i+k+32];- }- }- for(;i<I;i+=2){ /* coda a coppie, unsigned (u=v+8 memorizzato) */- uint8_t byte=w4[i>>1];- sum+=(int32_t)(byte&0xF)*x[i];- if(i+1<I) sum+=(int32_t)(byte>>4)*x[i+1];- }- return sum;-}--/* ---- K1b (OPT-IN, IDOT_GS=1): IDOT planare A GRUPPI (fmt=4, gs%64==0) ------ * Con gs=64 il gruppo di scala COINCIDE col blocco-piano da 64 elementi: il- * dot unsigned del blocco (2 dpbusd) -> int32 di gruppo, meno 8*somma(x) del- * gruppo, per la scala f32 del gruppo. Attivazioni int8 (stessa famiglia- * qrow_i8 del resto dell'IDOT): NON bit-identico al kernel f32 a gruppi --- * per questo e' dietro flag, in attesa dell'ablazione. xsg = somme int32- * per (riga, gruppo), calcolate dal chiamante in una passata esatta.- * EN: grouped planar IDOT, opt-in. With gs=64 the scale group IS the plane- * block; per-group unsigned dot minus 8*group-sum, times the group scale.- * int8 activations: not bit-identical to the f32 grouped kernel, hence the- * flag until the ablation blesses a default.- *- * Three execution shapes below, one contract: per (row, output) the float- * accumulation is the SAME sequence of fmaf((float)group_int, scale[g], a)- * in ascending g, and every group_int is an exact int32 β so the per-row- * path, the 1x4 row tile, and the AMX tile are bit-identical to each other- * and to the pure-C reference on every ISA. */--/* one planar 64-element block, unpacked once and shared across a row tile:- * lo nibbles = elements base..base+31 in order, hi = base+32..base+63. */-#if defined(coli_dpbusd256)-static inline void i4p_blk256(const uint8_t *blk, __m256i *lo, __m256i *hi){- const __m256i m4=_mm256_set1_epi8(0x0F);- __m256i b=_mm256_loadu_si256((const __m256i*)blk);- *lo=_mm256_and_si256(b,m4);- *hi=_mm256_and_si256(_mm256_srli_epi16(b,4),m4);-}-#endif-#if defined(__AVX512VNNI__) && defined(__AVX512BW__)-/* whole block in one zmm: lanes 0..31 = lo elements, 32..63 = hi β matches a- * straight 64-byte load of the activation block, so ONE dpbusd per block. */-static inline __m512i i4p_blk512(const uint8_t *blk){- const __m256i m4=_mm256_set1_epi8(0x0F);- __m256i b=_mm256_loadu_si256((const __m256i*)blk);- return _mm512_inserti64x4(_mm512_castsi256_si512(_mm256_and_si256(b,m4)),- _mm256_and_si256(_mm256_srli_epi16(b,4),m4),1);-}-#endif--/* ---- K1c: AMX int8 tile kernel (Sapphire Rapids+, opt-in via the same- * IDOT_GS=1 family gate; AMX=0 kills it, AMX_S_MIN sets the row threshold).- * With gs a multiple of 64, K=64 tile-multiplies cover a scale group exactly:- * B tile = 16 output rows' int4 block unpacked once to SIGNED int8 (v-8, so- * tdpbssd returns d - 8*sum(x_group) directly β the same int32 the vector- * path computes as d_unsigned - 8*xsg), A tile = up to 16 activation rows.- * The unpack cost is paid once per (output tile, group) and amortized over- * every activation row β the multi-row reuse the pair-layout kernels lack.- * Linux-only arming: tile data needs an ARCH_REQ_XCOMP_PERM handshake. */-#if defined(__AMX_INT8__) && defined(__AMX_TILE__) && defined(__AVX512F__)-#define COLI_HAVE_AMX_I4P 1-#if defined(__linux__)-#include <unistd.h>-#include <sys/syscall.h>-#elif defined(_WIN32)-#ifndef WIN32_LEAN_AND_MEAN-#define WIN32_LEAN_AND_MEAN-#endif-#include <windows.h>-#endif-static int coli_amx_state=-1;-static int coli_amx_s_min=8;-static int coli_amx_ok(void){- if(coli_amx_state<0){- const char *e=getenv("AMX");- const char *sm=getenv("AMX_S_MIN"); if(sm&&atoi(sm)>0) coli_amx_s_min=atoi(sm);-#if defined(__linux__)- /* ARCH_REQ_XCOMP_PERM(XFEATURE_XTILEDATA): kernel >= 5.16 grants tile- * state per-process; without it the first tile op SIGILLs. */- coli_amx_state=(e&&*e=='0')?0:(syscall(SYS_arch_prctl,0x1023,18)==0);-#elif defined(_WIN32)- /* Windows 11: tile data is an opt-in per-process XState feature.- * Dynamic lookup keeps older kernels building/running (arming just- * fails closed there). XSTATE_AMX_TILE_DATA = 18. */- if(e&&*e=='0') coli_amx_state=0;- else {- HMODULE k32=GetModuleHandleA("kernel32.dll");- typedef BOOL (WINAPI *coli_xstate_fn)(ULONG64);- coli_xstate_fn f=k32?(coli_xstate_fn)(void*)GetProcAddress(k32,"EnableProcessOptionalXStateFeatures"):NULL;- coli_amx_state=(f && f(1ull<<18))?1:0;- }-#else- (void)e; coli_amx_state=0;-#endif- if(coli_amx_state)- fprintf(stderr,"[K1c] AMX int8 tile kernel armed for gs64 tensors "- "(AMX=0 disables, engages at S>=%d)\n",coli_amx_s_min);- }- return coli_amx_state;-}-/* ldtilecfg layout (palette 1): tmm0=C [rows x 16 i32], tmm1=A [rows x 64 i8],- * tmm2=B [16 x 64 i8 VNNI]. rows<16 reconfigures for the last partial s-tile. */-struct coli_tilecfg { uint8_t palette,start_row,rsvd[14]; uint16_t colsb[16]; uint8_t rows[16]; };-static void coli_amx_cfg(int arows){- struct coli_tilecfg c; memset(&c,0,sizeof c); c.palette=1;- c.rows[0]=(uint8_t)arows; c.colsb[0]=64;- c.rows[1]=(uint8_t)arows; c.colsb[1]=64;- c.rows[2]=16; c.colsb[2]=64;- _tile_loadconfig(&c);-}-static void matmul_i4p_gidot_amx(float *y, const int8_t *xq, const float *sx,- const uint8_t *q4, const float *scale,- int S, int I, int O16, int O, int gs){- int rb=(I+1)/2, ng=(I+gs-1)/gs, bpg=gs/64;- #pragma omp parallel- {- float *acc=malloc((size_t)S*16*sizeof(float));- if(!acc){ fprintf(stderr,"OOM: amx acc\n"); exit(1); }- int8_t bstage[4*1024] __attribute__((aligned(64))); /* bpg<=4 gated below */- int32_t cbuf[16*16] __attribute__((aligned(64)));- float sclT[16];- int cur=16; coli_amx_cfg(16);- #pragma omp for schedule(static)- for(int ot=0; ot<O16; ot+=16){- memset(acc,0,(size_t)S*16*sizeof(float));- for(int g=0; g<ng; g++){- for(int j=0;j<16;j++) sclT[j]=scale[(int64_t)(ot+j)*ng+g];- for(int b=0;b<bpg;b++){- /* B tile, VNNI [k/4][n][4]: element k of output row ot+n- * lands at row k/4, byte n*4+k%4 β signed (v-8) at unpack. */- int8_t *dst=bstage+(size_t)b*1024;- int64_t boff=((int64_t)g*gs+b*64)>>1;- for(int n=0;n<16;n++){- const uint8_t *blk=q4+(int64_t)(ot+n)*rb+boff;- for(int k=0;k<32;k++){- dst[(k>>2)*64+n*4+(k&3)] =(int8_t)((blk[k]&0xF)-8);- dst[((k+32)>>2)*64+n*4+((k+32)&3)]=(int8_t)((blk[k]>>4)-8);- }- }- }- for(int st=0; st<S; st+=16){- int rows=S-st<16?S-st:16;- if(rows!=cur){ coli_amx_cfg(rows); cur=rows; }- _tile_zero(0);- for(int b=0;b<bpg;b++){- _tile_loadd(1, xq+(int64_t)st*I+(int64_t)g*gs+b*64, (size_t)I);- _tile_loadd(2, bstage+(size_t)b*1024, 64);- _tile_dpbssd(0,1,2);- }- _tile_stored(0, cbuf, 64);- /* per-lane fmadd == the vector path's per-(s,o) scalar fmaf:- * same values, same ascending-g order, single rounding. */- for(int r=0;r<rows;r++){- __m512 cv=_mm512_cvtepi32_ps(_mm512_load_si512((const void*)(cbuf+(size_t)r*16)));- __m512 av=_mm512_loadu_ps(acc+(int64_t)(st+r)*16);- _mm512_storeu_ps(acc+(int64_t)(st+r)*16,- _mm512_fmadd_ps(cv,_mm512_loadu_ps(sclT),av));- }- }- }- for(int s=0;s<S;s++)- _mm512_storeu_ps(y+(int64_t)s*O+ot,- _mm512_mul_ps(_mm512_loadu_ps(acc+(int64_t)s*16),- _mm512_set1_ps(sx[s])));- }- _tile_release();- free(acc);- }-}-#endif--/* vector body over an output range [o0,o1): the 1x4 row tile pays the weight- * block's load+mask once per 4 activation rows (the fmt=2 K2 tile's idea,- * brought to the grouped family β the prefill batch-union and the serve mux- * deliver exactly these multi-row calls). Integer group dots in any lane- * order are exact, and each row keeps its own ascending-g fmaf chain. */-static void i4p_gidot_rows(float *y, const int8_t *xq, const float *sx,- const int32_t *xsg, const uint8_t *q4,- const float *scale, int S, int I, int O, int gs,- int o0, int o1){- int rb=(I+1)/2, ng=(I+gs-1)/gs, bpg=gs/64; /* blocchi-piano per gruppo */- #pragma omp parallel for schedule(static)- for(int o=o0;o<o1;o++){- const uint8_t *w=q4+(int64_t)o*rb;- const float *scl=scale+(int64_t)o*ng;- int s=0;- for(; s+4<=S; s+=4){- const int8_t *x0=xq +(int64_t)s*I, *x1=x0+I, *x2=x1+I, *x3=x2+I;- const int32_t *g0=xsg+(int64_t)s*ng, *g1=g0+ng, *g2=g1+ng, *g3=g2+ng;- float a0=0,a1=0,a2=0,a3=0;- int g=0;- for(; (g+1)*gs<=I; g++){- int32_t d0,d1,d2,d3;-#if defined(__AVX512VNNI__) && defined(__AVX512BW__)- __m512i A0=_mm512_setzero_si512(),A1=_mm512_setzero_si512();- __m512i A2=_mm512_setzero_si512(),A3=_mm512_setzero_si512();- for(int b=0;b<bpg;b++){- int base=g*gs+b*64;- __m512i wz=i4p_blk512(w+(base>>1));- A0=_mm512_dpbusd_epi32(A0,wz,_mm512_loadu_si512((const void*)(x0+base)));- A1=_mm512_dpbusd_epi32(A1,wz,_mm512_loadu_si512((const void*)(x1+base)));- A2=_mm512_dpbusd_epi32(A2,wz,_mm512_loadu_si512((const void*)(x2+base)));- A3=_mm512_dpbusd_epi32(A3,wz,_mm512_loadu_si512((const void*)(x3+base)));- }- d0=_mm512_reduce_add_epi32(A0); d1=_mm512_reduce_add_epi32(A1);- d2=_mm512_reduce_add_epi32(A2); d3=_mm512_reduce_add_epi32(A3);-#elif defined(coli_dpbusd256)- __m256i A0=_mm256_setzero_si256(),A1=_mm256_setzero_si256();- __m256i A2=_mm256_setzero_si256(),A3=_mm256_setzero_si256();- for(int b=0;b<bpg;b++){- int base=g*gs+b*64; __m256i lo,hi;- i4p_blk256(w+(base>>1),&lo,&hi);- A0=coli_dpbusd256(A0,lo,_mm256_loadu_si256((const __m256i*)(x0+base)));- A0=coli_dpbusd256(A0,hi,_mm256_loadu_si256((const __m256i*)(x0+base+32)));- A1=coli_dpbusd256(A1,lo,_mm256_loadu_si256((const __m256i*)(x1+base)));- A1=coli_dpbusd256(A1,hi,_mm256_loadu_si256((const __m256i*)(x1+base+32)));- A2=coli_dpbusd256(A2,lo,_mm256_loadu_si256((const __m256i*)(x2+base)));- A2=coli_dpbusd256(A2,hi,_mm256_loadu_si256((const __m256i*)(x2+base+32)));- A3=coli_dpbusd256(A3,lo,_mm256_loadu_si256((const __m256i*)(x3+base)));- A3=coli_dpbusd256(A3,hi,_mm256_loadu_si256((const __m256i*)(x3+base+32)));- }- d0=hsum256_i32(A0); d1=hsum256_i32(A1);- d2=hsum256_i32(A2); d3=hsum256_i32(A3);-#elif defined(__AVX2__)- const __m256i ones=_mm256_set1_epi16(1);- const __m256i m4=_mm256_set1_epi8(0x0F);- __m256i A0=_mm256_setzero_si256(),A1=_mm256_setzero_si256();- __m256i A2=_mm256_setzero_si256(),A3=_mm256_setzero_si256();- for(int b=0;b<bpg;b++){- int base=g*gs+b*64;- __m256i bb=_mm256_loadu_si256((const __m256i*)(w+(base>>1)));- __m256i lo=_mm256_and_si256(bb,m4);- __m256i hi=_mm256_and_si256(_mm256_srli_epi16(bb,4),m4);- /* maddubs(u8,s8): u<=15, |x|<=127 -> pair <= 3810, int16-safe */- A0=_mm256_add_epi32(A0,_mm256_madd_epi16(_mm256_maddubs_epi16(lo,_mm256_loadu_si256((const __m256i*)(x0+base))),ones));- A0=_mm256_add_epi32(A0,_mm256_madd_epi16(_mm256_maddubs_epi16(hi,_mm256_loadu_si256((const __m256i*)(x0+base+32))),ones));- A1=_mm256_add_epi32(A1,_mm256_madd_epi16(_mm256_maddubs_epi16(lo,_mm256_loadu_si256((const __m256i*)(x1+base))),ones));- A1=_mm256_add_epi32(A1,_mm256_madd_epi16(_mm256_maddubs_epi16(hi,_mm256_loadu_si256((const __m256i*)(x1+base+32))),ones));- A2=_mm256_add_epi32(A2,_mm256_madd_epi16(_mm256_maddubs_epi16(lo,_mm256_loadu_si256((const __m256i*)(x2+base))),ones));- A2=_mm256_add_epi32(A2,_mm256_madd_epi16(_mm256_maddubs_epi16(hi,_mm256_loadu_si256((const __m256i*)(x2+base+32))),ones));- A3=_mm256_add_epi32(A3,_mm256_madd_epi16(_mm256_maddubs_epi16(lo,_mm256_loadu_si256((const __m256i*)(x3+base))),ones));- A3=_mm256_add_epi32(A3,_mm256_madd_epi16(_mm256_maddubs_epi16(hi,_mm256_loadu_si256((const __m256i*)(x3+base+32))),ones));- }- d0=hsum256_i32(A0); d1=hsum256_i32(A1);- d2=hsum256_i32(A2); d3=hsum256_i32(A3);-#else- d0=d1=d2=d3=0;- for(int b=0;b<bpg;b++){- int base=g*gs+b*64;- const uint8_t *blk=w+(base>>1);- for(int k=0;k<32;k++){- int32_t ul=(int32_t)(blk[k]&0xF), uh=(int32_t)(blk[k]>>4);- d0+=ul*x0[base+k]+uh*x0[base+k+32];- d1+=ul*x1[base+k]+uh*x1[base+k+32];- d2+=ul*x2[base+k]+uh*x2[base+k+32];- d3+=ul*x3[base+k]+uh*x3[base+k+32];- }- }-#endif- a0=fmaf((float)(d0-8*g0[g]),scl[g],a0);- a1=fmaf((float)(d1-8*g1[g]),scl[g],a1);- a2=fmaf((float)(d2-8*g2[g]),scl[g],a2);- a3=fmaf((float)(d3-8*g3[g]),scl[g],a3);- }- if(g*gs<I){ /* coda: gruppo parziale, nibble a coppie */- int32_t d0=0,d1=0,d2=0,d3=0;- for(int i=g*gs;i<I;i++){- uint8_t byte=w[i>>1];- int32_t u=(int32_t)((i&1)?(byte>>4):(byte&0xF));- d0+=u*x0[i]; d1+=u*x1[i]; d2+=u*x2[i]; d3+=u*x3[i];- }- a0=fmaf((float)(d0-8*g0[g]),scl[g],a0);- a1=fmaf((float)(d1-8*g1[g]),scl[g],a1);- a2=fmaf((float)(d2-8*g2[g]),scl[g],a2);- a3=fmaf((float)(d3-8*g3[g]),scl[g],a3);- }- y[(int64_t)s*O+o] =a0*sx[s];- y[(int64_t)(s+1)*O+o]=a1*sx[s+1];- y[(int64_t)(s+2)*O+o]=a2*sx[s+2];- y[(int64_t)(s+3)*O+o]=a3*sx[s+3];- }- for(; s<S; s++){- const int8_t *xr=xq+(int64_t)s*I;- const int32_t *xg=xsg+(int64_t)s*ng;- float a=0; int g=0;- for(; (g+1)*gs<=I; g++){ /* gruppi interi */- int32_t d=0;-#if defined(__AVX512VNNI__) && defined(__AVX512BW__)- __m512i acc=_mm512_setzero_si512();- for(int b=0;b<bpg;b++){- int base=g*gs+b*64;- acc=_mm512_dpbusd_epi32(acc,i4p_blk512(w+(base>>1)),- _mm512_loadu_si512((const void*)(xr+base)));- }- d=_mm512_reduce_add_epi32(acc);-#else- for(int b=0;b<bpg;b++){- int base=g*gs+b*64;- const uint8_t *blk=w+(base>>1);- const int8_t *xb=xr+base;-#if defined(coli_dpbusd256)- __m256i lo,hi; i4p_blk256(blk,&lo,&hi);- __m256i acc=_mm256_setzero_si256();- acc=coli_dpbusd256(acc,lo,_mm256_loadu_si256((const __m256i*)xb));- acc=coli_dpbusd256(acc,hi,_mm256_loadu_si256((const __m256i*)(xb+32)));- d+=hsum256_i32(acc);-#elif defined(__AVX2__)- const __m256i m4=_mm256_set1_epi8(0x0F);- const __m256i ones=_mm256_set1_epi16(1);- __m256i bb=_mm256_loadu_si256((const __m256i*)blk);- __m256i p0=_mm256_maddubs_epi16(_mm256_and_si256(bb,m4),- _mm256_loadu_si256((const __m256i*)xb));- __m256i p1=_mm256_maddubs_epi16(_mm256_and_si256(_mm256_srli_epi16(bb,4),m4),- _mm256_loadu_si256((const __m256i*)(xb+32)));- __m256i acc=_mm256_add_epi32(_mm256_madd_epi16(p0,ones),- _mm256_madd_epi16(p1,ones));- d+=hsum256_i32(acc);-#else- for(int k=0;k<32;k++){- d+=(int32_t)(blk[k]&0xF)*xb[k];- d+=(int32_t)(blk[k]>>4)*xb[k+32];- }-#endif- }-#endif- a=fmaf((float)(d-8*xg[g]),scl[g],a);- }- if(g*gs<I){ /* coda: gruppo parziale, nibble a coppie */- int32_t d=0;- for(int i=g*gs;i<I;i++){- uint8_t byte=w[i>>1];- d+=(int32_t)((i&1)?(byte>>4):(byte&0xF))*xr[i];- }- a=fmaf((float)(d-8*xg[g]),scl[g],a);- }- y[(int64_t)s*O+o]=a*sx[s];- }- }-}--static void matmul_i4p_grouped_idot(float *y, const int8_t *xq, const float *sx,- const int32_t *xsg, const uint8_t *q4,- const float *scale, int S, int I, int O, int gs){-#if defined(COLI_HAVE_AMX_I4P)- /* AMX takes the aligned bulk (full 16-output tiles, full groups only β- * the real gs64 checkpoints have I%gs==0); the vector path finishes any- * output remainder. Below the row threshold the B-tile unpack does not- * amortize and the vector tile is the better kernel. */- if(coli_amx_ok() && S>=coli_amx_s_min && gs%64==0 && gs<=256 && I%gs==0 && O>=16){- int O16=O&~15;- matmul_i4p_gidot_amx(y,xq,sx,q4,scale,S,I,O16,O,gs);- if(O16<O) i4p_gidot_rows(y,xq,sx,xsg,q4,scale,S,I,O,gs,O16,O);- return;- }-#endif- i4p_gidot_rows(y,xq,sx,xsg,q4,scale,S,I,O,gs,0,O);-}--/* matmul IDOT planare (fmt=2): y = (dot_u - 8*xsum[s]) * scale[o] * sx[s].- * Bit-identico a matmul_i4_idot: somme intere, identita' esatta. */-static void matmul_i4p_idot(float *y, const int8_t *xq, const float *sx, const int32_t *xsum,- const uint8_t *q4, const float *scale, int S, int I, int O){- int rb=(I+1)/2;- #pragma omp parallel for schedule(static)- for(int o=0;o<O;o++){ const uint8_t *w=q4+(int64_t)o*rb; float sc=scale[o];- int s=0;-#if defined(coli_dpbusd256)- /* K2: tile 1x4 sulla union β il blocco pesi (load + 2 and + srli) si- * paga UNA volta per 4 righe di attivazione invece che per riga. La- * union del prefill consegna nr=2..16 righe per expert: e' esattamente- * la finestra che il per-riga serviva peggio. Somme intere ->- * bit-identico al path per-riga per associativita'.- * EN: 1x4 register tile β the weight block's load+mask cost is paid- * once per 4 activation rows. Integer sums: bit-identical to the- * per-row path by associativity. */- const __m256i m4t=_mm256_set1_epi8(0x0F);- for(;s+4<=S;s+=4){- const int8_t *x0=xq+(int64_t)s*I, *x1=x0+I, *x2=x1+I, *x3=x2+I;- __m256i a0=_mm256_setzero_si256(), a1=_mm256_setzero_si256();- __m256i a2=_mm256_setzero_si256(), a3=_mm256_setzero_si256();- int i=0;- for(;i+64<=I;i+=64){- __m256i b =_mm256_loadu_si256((const __m256i*)(w+(i>>1)));- __m256i lo=_mm256_and_si256(b,m4t);- __m256i hi=_mm256_and_si256(_mm256_srli_epi16(b,4),m4t);- a0=coli_dpbusd256(a0,lo,_mm256_loadu_si256((const __m256i*)(x0+i)));- a0=coli_dpbusd256(a0,hi,_mm256_loadu_si256((const __m256i*)(x0+i+32)));- a1=coli_dpbusd256(a1,lo,_mm256_loadu_si256((const __m256i*)(x1+i)));- a1=coli_dpbusd256(a1,hi,_mm256_loadu_si256((const __m256i*)(x1+i+32)));- a2=coli_dpbusd256(a2,lo,_mm256_loadu_si256((const __m256i*)(x2+i)));- a2=coli_dpbusd256(a2,hi,_mm256_loadu_si256((const __m256i*)(x2+i+32)));- a3=coli_dpbusd256(a3,lo,_mm256_loadu_si256((const __m256i*)(x3+i)));- a3=coli_dpbusd256(a3,hi,_mm256_loadu_si256((const __m256i*)(x3+i+32)));- }- int32_t d0=hsum256_i32(a0), d1=hsum256_i32(a1);- int32_t d2=hsum256_i32(a2), d3=hsum256_i32(a3);- /* coda a coppie, unsigned: stessa identita' -8*xsum del per-riga */- for(;i<I;i+=2){- uint8_t byte=w[i>>1];- d0+=(int32_t)(byte&0xF)*x0[i]; d1+=(int32_t)(byte&0xF)*x1[i];- d2+=(int32_t)(byte&0xF)*x2[i]; d3+=(int32_t)(byte&0xF)*x3[i];- if(i+1<I){- d0+=(int32_t)(byte>>4)*x0[i+1]; d1+=(int32_t)(byte>>4)*x1[i+1];- d2+=(int32_t)(byte>>4)*x2[i+1]; d3+=(int32_t)(byte>>4)*x3[i+1];- }- }- y[(int64_t)(s+0)*O+o]=(float)(d0-8*xsum[s+0])*sc*sx[s+0];- y[(int64_t)(s+1)*O+o]=(float)(d1-8*xsum[s+1])*sc*sx[s+1];- y[(int64_t)(s+2)*O+o]=(float)(d2-8*xsum[s+2])*sc*sx[s+2];- y[(int64_t)(s+3)*O+o]=(float)(d3-8*xsum[s+3])*sc*sx[s+3];- }-#endif- for(;s<S;s++){- int32_t d=dot_i4p_u(w,xq+(int64_t)s*I,I)-8*xsum[s];- y[(int64_t)s*O+o]=(float)d*sc*sx[s];- }- }-}/* f32 planare (fmt=2): stesso ordine di accumulazione di matmul_i4 (sequenza* 0..63 in blocchi di 8 lane, stessa lane per elemento) -> bit-identico. */diff --git a/c/qwen36.c b/c/qwen36.cindex 7a5de982..49ced244 100644--- a/c/qwen36.c+++ b/c/qwen36.c@@ -67,6 +67,7 @@ static int qwen36_max_ctx(void) {#include "json.h" /* tokenizer.json parsing (reuse minimal parser) */#include "qwen36_tier.h" /* optional CUDA VRAM expert tier */#include "expert_ffn.h" /* routed experts: planar int4 kernel + layer runner */+#include "idot.h" /* integer dot kernels for the dense trunk (COLI_DENSE_IDOT, COLI_DENSE_BITS) */#ifdef COLI_SEGMENT_ADAPTER#include "segment_runtime.h"#include "segment_adapters.h"@@ -989,6 +990,12 @@ static int g_expert_is_int4 = 1;* int8 copy) and with QWEN_EXPERT_KERNEL=0, which keeps the historical* unpack-to-int8 path for A/Bs. Decided once from the container itself. */static int container_layer_is_int4(Model *m, int layer);+/* The routed experts take the same integer path as the dense trunk+ * (expert_ffn.h mode 1: activation to int8 once per row, dpbusd against the+ * planar nibbles). Measured on the 35B: expert compute 22.7 to 15.9+ * ms/token for +0.1% perplexity. QWEN_EXPERT_ACT=f32 restores mode 0, f32+ * activations and the bit-identical contract with the pair kernels. */+static int xf_act_mode(void){ static int v=-1; if(v<0){ const char *e=getenv("QWEN_EXPERT_ACT"); v=(e&&!strcmp(e,"f32"))?0:1; } return v; }static int xf_mode(Model *m) {static int v = -1;if (v >= 0) return v;@@ -1037,8 +1044,131 @@ static void matmul_qe(float *y, const float *x, const int8_t *q, const float *sc* matmul_d dispatches via pointer lookup to matmul_q; COLI_DENSE_I8=0 falls* back to f32 (reference path for parity tests). ~4x less memory traffic. */#define QDW_MAX 1024-static struct { const float *w; int8_t *q; float *sc; int I, O; } g_qdw[QDW_MAX];+/* q/sc: int8 rows with one scale per row (the classic copy, what the VRAM+ * tier uploads). q4/sg: the same matrix as int4 planar blocks of 64 with one+ * scale per group (COLI_DENSE_BITS=4), the layout the K1b grouped kernel+ * reads; ng = I/64 groups per row. */+static struct { const float *w; int8_t *q; float *sc; int I, O; uint8_t *q4; float *sg; int ng; } g_qdw[QDW_MAX];static int g_qdw_n = 0;+/* ---- Dense trunk, integer dot products ------------------------------------+ *+ * Every dense GEMV of a token (DeltaNet projections and out_proj, attention+ * q/k/v/o, the shared expert, lm_head) used to multiply int8 weights by f32+ * activations: each weight byte converted to f32 and fed to an FMA, eight+ * weights per instruction. Measured on lm_head (248320 x 2048 int8, 508 MB)+ * that runs at 29 GB/s on a 16-core AVX-512 host whose memory bus does 80:+ * the kernel, not the bus, was the limit, and the dense part of a token is+ * 1.9 GB of int8 on the 35B, three times the routed experts.+ *+ * The activation is now quantized to int8 once per call (one scale,+ * amax/127, the qrow_i8 contract the expert IDOT already uses) and the dot+ * is integer: 32 weights per instruction on AVX2 (maddubs), 64 on AVX-512+ * VNNI, exact int32 sums scaled once per output. Not bit-identical to the+ * f32 path (the activation is rounded): measured on the 35B, +1.0%+ * perplexity on 4 x 512 tokens, lm_head 12.6 to 10.2 ms/token, decode+ * 6.71 to 7.35 tok/s alone and 8.23 with the experts' int8 activations.+ * COLI_DENSE_IDOT=0 restores the f32-activation kernel.+ *+ * COLI_DENSE_BITS=4 additionally stores the dense matrices as int4 in blocks+ * of 64 with one scale per block, the K1b planar layout, halving the bytes+ * the token reads; lm_head alone goes from 508 to 254 MB. It implies the+ * integer dot (that layout has no f32 kernel). Same gate: measured. */+static int dense_idot_on(void){ static int v=-1; if(v<0){ const char *e=getenv("COLI_DENSE_IDOT"); v=!(e&&*e=='0'); } return v; }+static int dense_bits(void){ static int v=-1; if(v<0){ const char *e=getenv("COLI_DENSE_BITS"); v=(e&&atoi(e)==4)?4:8; } return v; }++/* Activation -> int8 with one scale, plus the int32 sum of every block of 64+ * (the K1b kernel subtracts 8*sum per group because its nibbles are unsigned).+ * Vectorized: the scalar lrintf loop was measured at ~7 us for 4096 values,+ * which times ~150 GEMVs per token is a millisecond thrown away. Rounding is+ * to nearest even in both paths, so the vector path equals qrow_i8 bit for bit. */+static float dense_act_i8(const float *x, int I, int8_t *xq, int32_t *xsg){+ float amax = 0.f;+ int i = 0;+#ifdef __AVX2__+ {+ __m256 am = _mm256_setzero_ps();+ const __m256 sign = _mm256_set1_ps(-0.0f);+ for (; i + 8 <= I; i += 8) am = _mm256_max_ps(am, _mm256_andnot_ps(sign, _mm256_loadu_ps(x + i)));+ float tmp[8]; _mm256_storeu_ps(tmp, am);+ for (int k = 0; k < 8; k++) if (tmp[k] > amax) amax = tmp[k];+ }+#endif+ for (; i < I; i++) { float a = fabsf(x[i]); if (a > amax) amax = a; }+ float s = amax / 127.f; if (s < 1e-12f) s = 1e-12f;+ float inv = 1.f / s;+ i = 0;+#ifdef __AVX2__+ {+ const __m256 vinv = _mm256_set1_ps(inv);+ for (; i + 32 <= I; i += 32) {+ __m256i a = _mm256_cvtps_epi32(_mm256_mul_ps(_mm256_loadu_ps(x + i), vinv));+ __m256i b = _mm256_cvtps_epi32(_mm256_mul_ps(_mm256_loadu_ps(x + i + 8), vinv));+ __m256i c = _mm256_cvtps_epi32(_mm256_mul_ps(_mm256_loadu_ps(x + i + 16), vinv));+ __m256i d = _mm256_cvtps_epi32(_mm256_mul_ps(_mm256_loadu_ps(x + i + 24), vinv));+ /* packs interleave 128-bit lanes: fix the order with one permute */+ __m256i ab = _mm256_packs_epi32(a, b), cd = _mm256_packs_epi32(c, d);+ __m256i abcd = _mm256_packs_epi16(ab, cd);+ abcd = _mm256_permutevar8x32_epi32(abcd, _mm256_setr_epi32(0, 4, 1, 5, 2, 6, 3, 7));+ _mm256_storeu_si256((__m256i *)(xq + i), abcd);+ }+ }+#endif+ for (; i < I; i++) xq[i] = (int8_t)lrintf(x[i] * inv);+ if (xsg) {+ int ng = I / 64;+ for (int g = 0; g < ng; g++) {+ int32_t sum = 0;+ for (int k = 0; k < 64; k++) sum += xq[g * 64 + k];+ xsg[g] = sum;+ }+ }+ return s;+}++/* f32 rows -> int4 in blocks of 64 with one f32 scale per block, packed as the+ * K1b planar layout (unsigned nibbles v+8, block b: lo nibbles = elements+ * b*64..b*64+31, hi = b*64+32..b*64+63). The quantizer is the symmetric+ * absmax/7 the expert containers use. */+static void pack_int4_g64_planar(const float *w, uint8_t *q4, float *sg, int O, int I){+ int rb = I / 2, ng = I / 64;+ #pragma omp parallel for schedule(static)+ for (int o = 0; o < O; o++) {+ const float *wr = w + (int64_t)o * I;+ uint8_t *row = q4 + (int64_t)o * rb;+ float *sr = sg + (int64_t)o * ng;+ for (int g = 0; g < ng; g++) {+ const float *blk = wr + g * 64;+ float amax = 0.f;+ for (int k = 0; k < 64; k++) { float a = fabsf(blk[k]); if (a > amax) amax = a; }+ /* absmax/7 is where the search starts; the scale is then refined+ * by least squares against the rounded codes (s = w.q / q.q) and+ * the candidate with the smallest squared error wins. Three+ * rounds: measured on the 35B this recovers a third of the+ * perplexity absmax alone loses at 4 bits. */+ float best_s = amax / 7.f; if (best_s < 1e-8f) best_s = 1e-8f;+ int best_q[64]; double best_err = 1e30;+ float s = best_s;+ for (int round = 0; round < 4; round++) {+ int q[64]; double err = 0, wq = 0, qq = 0;+ float inv = 1.f / s;+ for (int k = 0; k < 64; k++) {+ int v = (int)lrintf(blk[k] * inv); if (v > 7) v = 7; if (v < -8) v = -8;+ q[k] = v; double d = (double)blk[k] - (double)v * s; err += d * d;+ wq += (double)blk[k] * v; qq += (double)v * v;+ }+ if (err < best_err) { best_err = err; best_s = s; memcpy(best_q, q, sizeof q); }+ if (qq <= 0) break;+ float ns = (float)(wq / qq); /* least-squares scale for these codes */+ if (ns <= 0.f || ns == s) break;+ s = ns;+ }+ sr[g] = best_s;+ uint8_t *dst = row + g * 32;+ for (int k = 0; k < 32; k++)+ dst[k] = (uint8_t)((best_q[k] + 8) | ((best_q[k + 32] + 8) << 4));+ }+ }+}#ifdef COLI_QWEN_BATCH_TESTstatic uint64_t g_qwen_matmul_d_calls;#endif@@ -1048,7 +1178,27 @@ static int dense_i8_on(void){ static int v=-1; if(v<0){ const char *e=getenv("CO* the time. */static int kv_prefix_off(void){ const char *e=getenv("COLI_KV_PREFIX"); return e && *e=='0'; }static int dense_batch_on(void){ const char *e=getenv("QWEN_DENSE_BATCH"); return !(e&&*e=='0'); }-static void qdw_register(const float *W, int I, int O){+/* COLI_DENSE_INT4 names which components take the int4 copy when+ * COLI_DENSE_BITS=4: a comma list of lmhead, dnproj, dnout, attn, shexp,+ * router; unset means all of them. The parts of the trunk pay 4 bits+ * differently (measured: the attention projections and the DeltaNet input+ * projections cost the most perplexity, lm_head the least per byte saved),+ * so the default is chosen per component from the numbers, not for the+ * whole trunk at once. */+static int dense_int4_wanted(const char *tag){+ if (dense_bits() != 4 || !tag) return 0;+ const char *e = getenv("COLI_DENSE_INT4");+ if (!e || !*e) return 1;+ size_t n = strlen(tag);+ for (const char *p = e; *p; ) {+ while (*p == ',' || *p == ' ') p++;+ const char *q = p; while (*q && *q != ',' && *q != ' ') q++;+ if ((size_t)(q - p) == n && !strncmp(p, tag, n)) return 1;+ p = q;+ }+ return 0;+}+static void qdw_register_as(const float *W, int I, int O, const char *tag){if (!W || !dense_i8_on() || g_qdw_n >= QDW_MAX) return;int8_t *q = malloc((size_t)O*I); float *sc = malloc((size_t)O*sizeof(float));if (!q || !sc) { free(q); free(sc); return; }@@ -1060,13 +1210,40 @@ static void qdw_register(const float *W, int I, int O){int8_t *d = q + (int64_t)o*I;for (int i = 0; i < I; i++) { int v = (int)lrintf(r[i]*inv); if (v>127) v=127; if (v<-127) v=-127; d[i] = (int8_t)v; }}- g_qdw[g_qdw_n].w=W; g_qdw[g_qdw_n].q=q; g_qdw[g_qdw_n].sc=sc; g_qdw[g_qdw_n].I=I; g_qdw[g_qdw_n].O=O; g_qdw_n++;+ g_qdw[g_qdw_n].w=W; g_qdw[g_qdw_n].q=q; g_qdw[g_qdw_n].sc=sc; g_qdw[g_qdw_n].I=I; g_qdw[g_qdw_n].O=O;+ g_qdw[g_qdw_n].q4=NULL; g_qdw[g_qdw_n].sg=NULL; g_qdw[g_qdw_n].ng=0;+ if (dense_int4_wanted(tag) && I%64==0) {+ uint8_t *q4=malloc((size_t)O*(I/2)); float *sg=malloc((size_t)O*(I/64)*sizeof(float));+ if (q4 && sg) {+ pack_int4_g64_planar(W, q4, sg, O, I);+ g_qdw[g_qdw_n].q4=q4; g_qdw[g_qdw_n].sg=sg; g_qdw[g_qdw_n].ng=I/64;+ } else { free(q4); free(sg); }+ }+ g_qdw_n++;}+static void qdw_register(const float *W, int I, int O){ qdw_register_as(W, I, O, NULL); }static void matmul_d(float *y, const float *x, const float *W, int S, int I, int O){#ifdef COLI_QWEN_BATCH_TESTg_qwen_matmul_d_calls++;#endiffor (int i = 0; i < g_qdw_n; i++) if (g_qdw[i].w == W && g_qdw[i].I == I) {+ if (g_qdw[i].q4 || dense_idot_on()) {+ /* integer dot: the activation rows to int8 once, then the K1b+ * grouped kernel (int4 planar) or the per-row int8 kernel */+ int ng = I / 64;+ int8_t *xq = malloc((size_t)S * I);+ float *sx = malloc((size_t)S * sizeof(float));+ int32_t *xsg = g_qdw[i].q4 ? malloc((size_t)S * ng * sizeof(int32_t)) : NULL;+ if (xq && sx && (!g_qdw[i].q4 || xsg)) {+ for (int s = 0; s < S; s++)+ sx[s] = dense_act_i8(x + (int64_t)s * I, I, xq + (int64_t)s * I, xsg ? xsg + (int64_t)s * ng : NULL);+ if (g_qdw[i].q4) matmul_i4p_grouped_idot(y, xq, sx, xsg, g_qdw[i].q4, g_qdw[i].sg, S, I, O, 64);+ else matmul_q_idot(y, xq, sx, g_qdw[i].q, g_qdw[i].sc, S, I, O);+ free(xq); free(sx); free(xsg);+ return;+ }+ free(xq); free(sx); free(xsg); /* out of memory: the f32 path below */+ }if (S > 1 && dense_batch_on())matmul_q_batch(y, x, g_qdw[i].q, g_qdw[i].sc, S, I, O);else@@ -2023,9 +2200,9 @@ static void moe_xf_run(Model *m, int layer, const float *x, int S, float *out, cexp[dst] = &ex[dst];}double t1 = timed ? tm_now() : 0;- if (kper == K) xf_moe_run(out + (int64_t)s0 * D, x + (int64_t)s0 * D, per, K, D, F, ridx, rval, exp, 0, scratch);+ if (kper == K) xf_moe_run(out + (int64_t)s0 * D, x + (int64_t)s0 * D, per, K, D, F, ridx, rval, exp, xf_act_mode(), scratch);else {- xf_moe_run(tmp, x + (int64_t)s0 * D, 1, 1, D, F, ridx, rval, exp, 0, scratch);+ xf_moe_run(tmp, x + (int64_t)s0 * D, 1, 1, D, F, ridx, rval, exp, xf_act_mode(), scratch);float *os = out + (int64_t)s0 * D; for (int d = 0; d < D; d++) os[d] += tmp[d];}if (timed) { double t2 = tm_now(); g_xf_load += t1 - t0; g_xf_run += t2 - t1; }@@ -3630,16 +3807,16 @@ int main(int argc, char **argv) {int q_out = qc->q_heads * qc->q_head_dim, kv_out = qc->kv_heads * qc->k_head_dim;for (int i = 0; i < qc->n_layers; i++) {Layer *l = &m.L[i];- qdw_register(l->q, D2, q_out); qdw_register(l->k, D2, kv_out);- qdw_register(l->v, D2, kv_out); qdw_register(l->o, qc->o_in, D2);- qdw_register(l->gate, D2, qc->n_experts);- qdw_register(l->sh_g, D2, qc->shared_inter); qdw_register(l->sh_u, D2, qc->shared_inter);- qdw_register(l->sh_d, qc->shared_inter, D2);- qdw_register(l->dn_qkv, D2, qc->dn_conv_dim);- qdw_register(l->dn_z, D2, qc->dn_vheads * qc->dn_vdim);- qdw_register(l->dn_out, qc->dn_vheads * qc->dn_vdim, D2);+ qdw_register_as(l->q, D2, q_out, "attn"); qdw_register_as(l->k, D2, kv_out, "attn");+ qdw_register_as(l->v, D2, kv_out, "attn"); qdw_register_as(l->o, qc->o_in, D2, "attn");+ qdw_register_as(l->gate, D2, qc->n_experts, "router");+ qdw_register_as(l->sh_g, D2, qc->shared_inter, "shexp"); qdw_register_as(l->sh_u, D2, qc->shared_inter, "shexp");+ qdw_register_as(l->sh_d, qc->shared_inter, D2, "shexp");+ qdw_register_as(l->dn_qkv, D2, qc->dn_conv_dim, "dnproj");+ qdw_register_as(l->dn_z, D2, qc->dn_vheads * qc->dn_vdim, "dnproj");+ qdw_register_as(l->dn_out, qc->dn_vheads * qc->dn_vdim, D2, "dnout");}- qdw_register(m.lm_head, D2, qc->vocab);+ qdw_register_as(m.lm_head, D2, qc->vocab, "lmhead");/* Free the f32 originals -- the pointers only serve as lookup keys in* matmul_d from here on (never dereferenced again).* COLI_KEEP_F32=1 keeps them (debug). */diff --git a/c/tests/test_qwen36_dense_idot.c b/c/tests/test_qwen36_dense_idot.cnew file mode 100644index 00000000..c9c2ef47--- /dev/null+++ b/c/tests/test_qwen36_dense_idot.c@@ -0,0 +1,127 @@+/* The dense trunk's integer path (COLI_DENSE_IDOT, COLI_DENSE_BITS=4).+ *+ * Pins, with no model:+ * - dense_act_i8's vector path equals the scalar lrintf contract bit for bit+ * (same rounding, same scale) and its block sums are exact;+ * - pack_int4_g64_planar stores what the K1b layout expects: nibble v+8,+ * lo nibbles = elements 0..31 of the block, hi = 32..63, one scale per 64;+ * - matmul_d through the int4 grouped kernel and through the int8 integer+ * kernel both agree with the f32 reference within what an int8 activation+ * and an int4 weight allow, on every row of a random matrix, for one row+ * (decode) and for several rows (prefill);+ * - with neither flag the dispatch is the one before: the int8 f32-activation+ * kernel, byte-identical to its own output. */+#include <stdio.h>+#include <stdlib.h>+#include <string.h>+#include <math.h>++#define main qwen36_main_unused+#include "../qwen36.c"+#undef main+#include "../compat.h" /* setenv: MinGW has none */++static int fails;+static void ck(int ok, const char *what) { if (ok) { printf(" ok %s\n", what); return; } printf(" FAIL %s\n", what); fails++; }++static unsigned g_seed = 4242;+static float rnd(void) { g_seed = g_seed * 1103515245u + 12345u; return ((g_seed >> 8) & 0xFFFF) / 32768.f - 1.f; }++static void ref_matmul(float *y, const float *x, const float *W, int S, int I, int O) {+ for (int s = 0; s < S; s++) for (int o = 0; o < O; o++) {+ double a = 0; for (int i = 0; i < I; i++) a += (double)x[(size_t)s * I + i] * W[(size_t)o * I + i];+ y[(size_t)s * O + o] = (float)a;+ }+}+static double rel_gap(const float *a, const float *b, int n) {+ double worst = 0, scale = 1e-6;+ for (int i = 0; i < n; i++) { double d = fabs((double)a[i] - b[i]); if (d > worst) worst = d; if (fabs((double)b[i]) > scale) scale = fabs((double)b[i]); }+ return worst / scale;+}++int main(void) {+ enum { I = 256, O = 96, S = 5 };+ printf("activation quantizer\n");+ {+ float x[I]; int8_t a[I], b[I]; int32_t sums[I / 64];+ for (int t = 0; t < 50; t++) {+ float mag = (t % 5 == 0) ? 1e-3f : 3.f;+ for (int i = 0; i < I; i++) x[i] = rnd() * mag;+ float sa = dense_act_i8(x, I, a, sums);+ float sb = qrow_i8(x, b, I);+ if (sa != sb || memcmp(a, b, I)) { ck(0, "vector quantizer equals the scalar contract"); break; }+ for (int g = 0; g < I / 64; g++) { int32_t s = 0; for (int k = 0; k < 64; k++) s += a[g * 64 + k]; if (s != sums[g]) { ck(0, "block sums are exact"); break; } }+ if (t == 49) ck(1, "vector quantizer equals the scalar lrintf contract on 50 random rows, block sums exact");+ }+ }+ printf("int4 planar packing\n");+ {+ float *W = malloc((size_t)O * I * sizeof(float));+ for (size_t i = 0; i < (size_t)O * I; i++) W[i] = rnd();+ uint8_t *q4 = malloc((size_t)O * I / 2); float *sg = malloc((size_t)O * (I / 64) * sizeof(float));+ pack_int4_g64_planar(W, q4, sg, O, I);+ int bad = 0;+ for (int o = 0; o < O && !bad; o++) for (int g = 0; g < I / 64 && !bad; g++) {+ const uint8_t *blk = q4 + (size_t)o * (I / 2) + g * 32;+ float s = sg[(size_t)o * (I / 64) + g];+ for (int k = 0; k < 64; k++) {+ int nib = k < 32 ? (blk[k] & 0xF) : (blk[k - 32] >> 4);+ float w = W[(size_t)o * I + g * 64 + k];+ int want = (int)lrintf(w / s); if (want > 7) want = 7; if (want < -8) want = -8;+ if (nib - 8 != want) { bad = 1; break; }+ }+ }+ ck(!bad, "nibble v+8 in the planar order, one absmax/7 scale per block of 64");+ free(W); free(q4); free(sg);+ }+ printf("matmul_d dispatch\n");+ {+ float *W = malloc((size_t)O * I * sizeof(float)), *x = malloc((size_t)S * I * sizeof(float));+ float *ref = malloc((size_t)S * O * sizeof(float)), *y = malloc((size_t)S * O * sizeof(float)), *y0 = malloc((size_t)S * O * sizeof(float));+ for (size_t i = 0; i < (size_t)O * I; i++) W[i] = rnd();+ for (size_t i = 0; i < (size_t)S * I; i++) x[i] = rnd() * 2.f;+ ref_matmul(ref, x, W, S, I, O);+ /* the classic path (no flags): int8 rows, f32 activations */+ setenv("COLI_DENSE_IDOT", "0", 1); setenv("COLI_DENSE_BITS", "8", 1);+ qdw_register(W, I, O);+ matmul_d(y0, x, W, S, I, O);+ double g0 = rel_gap(y0, ref, S * O);+ ck(g0 < 2e-2, "classic int8 path within 2% of the f32 reference (per-row int8)");+ matmul_d(y, x, W, 1, I, O);+ ck(!memcmp(y, y0, (size_t)O * sizeof(float)), "one row and the first of a batch agree on the classic path");+ /* the integer path on the same int8 rows */+ g_qdw_n = 0;+ /* dense_idot_on caches its answer: this TU reads the env once, so set it before the first call */+ }+ {+ /* fresh process-wide state for the flag readers: emulate by direct calls */+ float *W = malloc((size_t)O * I * sizeof(float)), *x = malloc((size_t)S * I * sizeof(float));+ float *ref = malloc((size_t)S * O * sizeof(float)), *y = malloc((size_t)S * O * sizeof(float));+ for (size_t i = 0; i < (size_t)O * I; i++) W[i] = rnd();+ for (size_t i = 0; i < (size_t)S * I; i++) x[i] = rnd() * 2.f;+ ref_matmul(ref, x, W, S, I, O);+ int8_t *q = malloc((size_t)O * I); float *sc = malloc((size_t)O * sizeof(float));+ for (int o = 0; o < O; o++) { /* the per-row int8 the engine builds at load */+ float am = 0.f; for (int i = 0; i < I; i++) { float a = fabsf(W[(size_t)o * I + i]); if (a > am) am = a; }+ float s = am > 1e-12f ? am / 127.f : 1.f; sc[o] = s;+ for (int i = 0; i < I; i++) { int v = (int)lrintf(W[(size_t)o * I + i] / s); if (v > 127) v = 127; if (v < -127) v = -127; q[(size_t)o * I + i] = (int8_t)v; }+ }+ int8_t *xq = malloc((size_t)S * I); float sx[S]; int32_t *xsg = malloc((size_t)S * (I / 64) * sizeof(int32_t));+ for (int s = 0; s < S; s++) sx[s] = dense_act_i8(x + (size_t)s * I, I, xq + (size_t)s * I, xsg + (size_t)s * (I / 64));+ matmul_q_idot(y, xq, sx, q, sc, S, I, O);+ ck(rel_gap(y, ref, S * O) < 3e-2, "int8 weights x int8 activations within 3% of the f32 reference, 5 rows");+ matmul_q_idot(y, xq, sx, q, sc, 1, I, O);+ ck(rel_gap(y, ref, O) < 3e-2, "same for one row");+ uint8_t *q4 = malloc((size_t)O * I / 2); float *sg = malloc((size_t)O * (I / 64) * sizeof(float));+ pack_int4_g64_planar(W, q4, sg, O, I);+ matmul_i4p_grouped_idot(y, xq, sx, xsg, q4, sg, S, I, O, 64);+ ck(rel_gap(y, ref, S * O) < 8e-2, "int4 blocks of 64 x int8 activations within 8% of the f32 reference, 5 rows");+ float y1[O];+ matmul_i4p_grouped_idot(y1, xq, sx, xsg, q4, sg, 1, I, O, 64);+ ck(!memcmp(y1, y, (size_t)O * sizeof(float)), "the one-row path and the row tile give the same bytes (the K1b contract)");+ free(W); free(x); free(ref); free(y); free(q); free(sc); free(xq); free(xsg); free(q4); free(sg);+ }+ if (fails) { printf("test_qwen36_dense_idot: %d failure(s)\n", fails); return 1; }+ printf("OK test_qwen36_dense_idot: the dense trunk's integer path\n");+ return 0;+}diff --git a/c/tests/test_qwen36_trunk_dense.c b/c/tests/test_qwen36_trunk_dense.cindex 907f19a5..02a9e1a3 100644--- a/c/tests/test_qwen36_trunk_dense.c+++ b/c/tests/test_qwen36_trunk_dense.c@@ -95,6 +95,10 @@ int main(void) {setenv("COLI_PLACE", "", 1); /* "" == unset == auto */setenv("CUDA_EXPERT_GB", "1", 1); /* room for everything */unsetenv("COLI_DENSE_I8");+ /* the GEMV from VRAM is compared with the CPU's f32-activation kernel, the+ * contract the tier uploads: the integer dense path rounds the activation+ * and is measured elsewhere (test_qwen36_dense_idot) */+ setenv("COLI_DENSE_IDOT", "0", 1);fake_ndev = 1; fake_uploads = 0; fake_dense_compute = 1;Model m; build(&m);diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex b32b2920..91969d00 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -391,6 +391,10 @@ and the CPU/GPU execution split.| Variable | Default | Effect ||---|---|---|| `COLI_DENSE_I8` | `1` (on) | Quantize resident dense matrices to per-row int8 at startup. `=0` keeps the f32 reference path for quality A/Bs. |+| `COLI_DENSE_IDOT` | `1` (on) | The dense trunk's GEMVs (DeltaNet projections and out_proj, attention q/k/v/o, shared expert, lm_head) quantize the activation to int8 once per call and run integer dot products (maddubs on AVX2, vpdpbusd on AVX-VNNI / AVX-512 VNNI) instead of converting every int8 weight to f32. Not bit-identical to the f32 path; measured +1.0% perplexity, lm_head 12.6 to 10.2 ms/token. `=0` restores the f32-activation kernel. |+| `QWEN_EXPERT_ACT` | `i8` | The routed experts' activation quantized to int8 once per row (expert_ffn.h mode 1). Measured +0.1% perplexity, expert compute 22.7 to 15.9 ms/token. `=f32` restores f32 activations and the bit-identical contract with the pair kernels. |+| `COLI_DENSE_BITS` | `8` | `=4` stores the dense trunk as int4 in blocks of 64 with one scale per block (the K1b planar layout, half the bytes), served by the grouped integer kernel; implies the integer dot. Opt-in: on the 35B it costs +10% perplexity on the whole trunk, +2.4% on lm_head alone (see `COLI_DENSE_INT4`). |+| `COLI_DENSE_INT4` | all components | With `COLI_DENSE_BITS=4`, a comma list of the components that take int4: `lmhead`, `dnproj`, `dnout`, `attn`, `shexp`, `router`. Measured on the 35B: `lmhead` +2.4% perplexity for 254 MB less per token; `lmhead,dnproj,dnout` +5.6%; everything +10%. || `QWEN_EXPERT_KERNEL` | `1` (on) | Routed experts run through the shared `expert_ffn.h` kernel: the int4 stays packed in RAM (planar layout, half the expert-cache RSS of the int8 unpack), gate+up are one pass, and a layer is two OpenMP regions over (expert, row-chunk) items instead of 3 x top-k GEMV regions. Takes effect on an int4 gs=64 container whose hidden and expert widths are multiples of 64, and not under the CUDA expert tier. `=0` restores the unpack-to-int8 path; the two produce the same tokens (1024-token decode on the real container byte-identical; pinned on the tiny int4 fixture in CI), only the f32 accumulation order inside a dot differs. Measured at cap 256 on the real container: 12.8 -> 15.7 tok/s, peak RSS 29 -> 17 GB. || `QWEN_DENSE_BATCH` | `1` (on) | On AVX2/FMA, reuse each dense-int8 weight decode across two prompt rows. `=0` restores one GEMV call per row. Decode `S=1` is unchanged. || `QWEN_SHARED_BATCH` | bounded by 32 MiB scratch | Batch the CPU shared expert across prompt rows. `=0` restores scalar calls; a positive integer caps rows per chunk. The CUDA-tier overlap path is unchanged. |diff --git a/docs/qwen36.md b/docs/qwen36.mdindex b449af49..c23f7871 100644--- a/docs/qwen36.md+++ b/docs/qwen36.md@@ -104,6 +104,52 @@ overlap, not this kernel. `tests/test_expert_ffn` holds the numerics.keeps its own path: it uploads the pair-layout int4 and computes misses fromthe int8 copy.+## The dense trunk: integer dot products++Every dense GEMV of a token, the DeltaNet in and out projections, the+attention q/k/v/o, the shared expert and lm_head, used to multiply int8+weights by f32 activations: each weight byte converted to f32 and fed to an+FMA, eight weights per instruction. On a 16-core AVX-512 host lm_head+(248320 x 2048 int8, 508 MB) ran at 29 GB/s on a memory bus that does 80:+the kernel was the limit, and the dense part of the token is 1.9 GB of int8+on the 35B, three times what the routed experts read.++Since 1.12.1 the activation is quantized to int8 once per call (one scale,+amax/127, the same contract the expert integer kernels use) and the products+are integer: 32 weights per instruction on AVX2, 64 on AVX-512 VNNI, exact+int32 sums scaled once per output. The routed experts take the same path for+their activations. Measured on Qwen3.6-35B-A3B, 8 threads, 300 decoded+tokens, every expert resident, perplexity on 4 x 512 tokens of English text:++| | tok/s | ms/token: DeltaNet proj / out, attention, lm_head, expert compute | perplexity |+|---|---|---|---|+| f32 activations (1.12.0) | 6.71 | 21.9 / 8.6, 9.5, 12.6, 22.7 | 13.79 |+| int8 activations, dense trunk (`COLI_DENSE_IDOT=1`) | 7.35 | 17.4 / 6.3, 7.7, 10.2, 22.7 | 13.92 (+1.0%) |+| int8 activations, routed experts (`QWEN_EXPERT_ACT=i8`) | 7.20 | 21.9 / 8.6, 9.5, 12.6, 15.9 | 13.80 (+0.1%) |+| both (the 1.12.1 default) | **8.23** (+22.6%) | 17.2 / 6.3, 7.5, 10.1, 15.6 | 13.97 (+1.3%) |++Both are the default; `COLI_DENSE_IDOT=0` and `QWEN_EXPERT_ACT=f32`+restore the f32 kernels, which stay bit-identical to their references.++**int4 for the trunk is opt-in, per component.** `COLI_DENSE_BITS=4` stores+the dense matrices as int4 in blocks of 64 with one scale per block (the+planar layout of the grouped expert kernel) and halves the bytes the token+reads, but the parts of the trunk pay 4 bits very differently, so+`COLI_DENSE_INT4` picks which ones take it:++| int4 on | tok/s | perplexity |+|---|---|---|+| nothing (int8) | 7.35 | 13.92 |+| `lmhead` | 8.15 (lm_head 10.1 to 8.4 ms; the total is within noise of the cache warming) | 14.13 (+2.4% vs f32) |+| `lmhead,dnproj,dnout` | | 14.56 (+5.6%) |+| `lmhead,dnproj,dnout,shexp` | | 14.94 (+8.3%) |+| everything (`attn` included) | 8.09 | 15.17 (+10%) |++A least-squares refinement of the block scale was tried and changes nothing+(15.16 against 15.17): the loss is the matrices' sensitivity, not the+quantizer. If you take one, take `lmhead`: 254 MB less per token for the+smallest cost.+## Cache-aware routing (`CACHE_ROUTE`, off by default)The residual misses above are the lever's target. `CACHE_ROUTE=1` ports the