Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C
1.9 KB
Raw Blame
/* fmt=8 (fp8-e4m3-b128) block geometry -- the single definition site for the
 * 128x128 scale-block edge, shared by the CPU side (quant.h: matmul_fp8,
 * e4m3 dequant plumbing; colibri.c: qt_addrow/qt_matvec_rows, qt_from_disk)
 * and the CUDA backend's three converted sites (backend_cuda.cu:
 * absorb_scale, quant_matmul's fmt=8 branch, the upload-time ng/scale_count
 * computation); the f8-warp and f8-group kernels there keep their own
 * 128 / >>7 literals, pinned against drift by backend_cuda.cu's
 * static_assert(FP8_BLOCK == 128). Before this header the two backends
 * agreed by IDENTICAL LITERALS restated in each file, so an edit to one side
 * could not break the other side's build -- only a full-scale CPU-vs-CUDA
 * parity run would have noticed. Kept deliberately tiny (no LUTs, no
 * functions with OpenMP pragmas, no intrinsics) so the CUDA translation unit
 * can include it without dragging in quant.h.
 *
 * The in-repo FP8 tests are NOT independent of this constant: the reference
 * decoders in test_fp8_passthrough.c, test_fp8_load.c, test_fp8_e2e_loader.c,
 * test_qwen38_native_weights.c, test_qt_addrow.c and test_shard_kvb_refuse.c
 * all index with FP8_BLOCK/fp8_nblk via quant.h, so they move in lockstep
 * with an edit here (test_backend_metal.mm's ref_fp8_nblk and
 * test_backend_cuda.cu's fmt=8 reference keep their own literals, on
 * purpose). An edit to FP8_BLOCK is a format change, not a tunable -- those
 * tests cannot catch a wrong edit on their own. */
#ifndef COLI_FP8_FORMAT_H
#define COLI_FP8_FORMAT_H

#include <stdint.h>

#define FP8_BLOCK 128

/* Blocks covering n elements: ceil(n/FP8_BLOCK). Host-side helper (device
 * code uses the FP8_BLOCK macro arithmetic directly -- this is not decorated
 * for device compilation on purpose, to keep the header plain C). */
static inline int64_t fp8_nblk(int n){ return ((int64_t)n + FP8_BLOCK - 1) / FP8_BLOCK; }

#endif /* COLI_FP8_FORMAT_H */