A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
C avx2c99cpu-inferencedeep-learningfrom-scratchinference-enginekimi-k3linear-attentionllmllm-inferencemachine-learningmemory-efficientmixture-of-expertsmoemxfp4quantizationsimdsystems-programmingtransformerzero-dependencies
Commits on main
fix(trunk): charge a prefetched layer to the bind that reaches it
7721a1b Fareed Khan committed 1 month ago
docs(contributors): credit Deobot2 for the trunk pinning work
358dfcc Fareed Khan committed 1 month ago
docs: say how much free memory a preset needs, not only its peak RSS
e5dc42f Fareed Khan committed 1 month ago
docs: catch the README up with the CLI, and record this round
1ae4d62 Fareed Khan committed 1 month ago
feat(scripts): fetch only the shards a --layers N run needs
0838cb4 Fareed Khan committed 1 month ago
fix(cli): let --preset auto start on large machines, and add --threads
64bf532 Fareed Khan committed 1 month ago
fix(cli): read available memory on macOS, cap it by commit headroom on Windows
998b489 Fareed Khan committed 1 month ago
feat(chat): reuse the previous turn's state instead of re-prefilling
fdf4505 Fareed Khan committed 1 month ago
feat(cli): checksum the state file and publish it with an atomic rename
16843c5 Fareed Khan committed 1 month ago
feat(cli): stop at a safe point on the first Ctrl-C and still write everything
5392746 Fareed Khan committed 1 month ago
test(sampler): run the chat sampler checks without a tokenizer
6bc20f4 Fareed Khan committed 1 month ago
fix(cli): refuse --ids and --layers values that do not fit an int
54e448f Fareed Khan committed 1 month ago
fix(io): say which file is missing when a tensor is not found
e52d766 Fareed Khan committed 1 month ago
fix(io): refuse safetensors headers whose spans overlap or leave gaps
003c6b1 Fareed Khan committed 1 month ago
fix(trunk): free the parsed trunk.json tree when an open is refused
d44d158 Fareed Khan committed 1 month ago
ci(windows): install libgomp explicitly, gcc no longer pulls it in
a2ad8e5 Fareed Khan committed 1 month ago
docs: record the chat template, trunk pinning, and this round's hardening
741cec9 Fareed Khan committed 1 month ago
fix(cli): refuse a malformed --ids list, an out of range --layers, and a lost --out
3fbc175 Fareed Khan committed 1 month ago
fix(core): zero a dropped expert's slots in the batched MoE prefill
037bb21 Fareed Khan committed 1 month ago
fix(security): harden safetensors, trunk, and JSON parsing against hostile input
31444e6 Fareed Khan committed 1 month ago
perf(trunk): largest-first pinning, --trunk-ring, and two prefetcher fixes
fbbfeb1 Fareed Khan committed 1 month ago
build(cmake): find Homebrew libomp on Apple and actually link it
e1ad3b5 Fareed Khan committed 1 month ago
docs: record --stop-id, --gen 0 prefill and the chunked trunk reader
ac1584a Fareed Khan committed 2 months ago