Skip to content

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

4.7 KB
Raw Blame

Measurement data

The raw output behind every number in PERFORMANCE.md and TUNING.md. Each table there cites the file it comes from.

Everything was produced in one campaign on one machine. See environment.txt for the full capture, including the measurement hygiene applied before any timing was taken. Summary: AMD EPYC 7763, 124 vCPU (2×62, no SMT), 228 GiB RAM, 3.2 TB NVMe at 3.2 GB/s O_DIRECT, Ubuntu 24.04.3, GCC 13.3, AVX2 without AVX-512.

file what it is backs
memory-ladder.tsv 12 memory budgets, each under a hard cgroup cap the ladder table, the two-cache finding
trunk-cache-split.tsv 12 splits at two fixed budgets (32 GB, 128 GB) the allocation recommendation
replication.tsv 3 runs of one identical configuration the noise floor
expert-cache-capacity.txt LRU / Belady / pinned hit rates from the recorded trace why the expert cache is weak
trunk-quantisation.txt int8 vs int4 reconstruction error on real weights why the trunk is streamed, not quantised
generations.txt what the model actually produced, decoded the output claims
environment.txt the machine everything

Read the noise floor first

run  s/token
1    14.7760
2    14.6655
3    20.1408

Same binary, same prompt, same flags, same minute. mean 16.53, sd 3.13, spread 33.1%.

A third of the measurement is noise. Every single-sample comparison in these files is subject to it, and any difference smaller than ~33% between two single runs is not an established effect. This is stated up front because it invalidated several conclusions drawn earlier in the campaign, including a "128 GB outlier" that was simply dispersion.

What the split data actually shows

This is the one place where the raw data is stronger than a naive reading and weaker than a careless one, so it is worth being exact.

128 GB, six splits (trunk GB → s/token):

12.3 → 28.38    30.8 → 25.20    49.2 → 25.69
73.8 → 18.37    98.4 → 19.46   110.0 → 16.80
  • Endpoint ratio 1.689×, which is where the 1.69× figure comes from.
  • Not monotonic. Two inversions: +1.9% at 30.8→49.2, and +5.9% at 73.8→98.4.
  • Spearman ρ = −0.886 over the six points.

32 GB, six splits, an independent replication of the same question at a quarter of the memory:

 2.7 → 31.23     6.8 → 30.58    10.8 → 31.25
16.2 → 30.88    21.6 → 29.13    25.6 → 28.06
  • Endpoint ratio 1.113×. One inversion, +2.2%.
  • Spearman ρ = −0.714.

In summary: twelve points across two independent budgets, both with a strong negative rank correlation between trunk share and time per token. Every inversion is far inside the 33% noise floor, so none of them is evidence against the trend, and none of the individual steps is evidence for it either. What carries the conclusion is the rank correlation across twelve points and the mechanism, the trunk is re-read in full every token (108.81 GB) while only ~25.8 GB of experts are touched.

What this data does not establish: that 1.69× is the effect size. It is one endpoint comparison of single samples. The direction is well supported; the magnitude is not replicated. benchmarks/split-sweep.sh re-runs the experiment with repetitions.

Reproducing

memory-ladder.tsv and trunk-cache-split.tsv are regenerated by benchmarks/memory-ladder.sh and benchmarks/split-sweep.sh respectively. Both need the 1.56 TB checkpoint, a packed trunk, and a Linux host with systemd (for the cgroup caps that make a memory budget mean anything). Both write a machine.txt alongside their results so a new run can be compared against this one.

expert-cache-capacity.txt needs neither weights nor a GPU, it replays tests/fixtures/expert_trace.bin, which ships in this repository:

python3 tools/sim_cache.py tests/fixtures/expert_trace.bin

That is the one result here anyone can re-derive in seconds, and it is the one that explains the two-cache finding.

trunk-quantisation.txt sampled the released checkpoint over HTTP range reads rather than downloading it. The script that did so is not shipped; the file records its method and its limits.

Scope

These are engine measurements: speed, memory, bytes moved, cache behaviour, and what the model emitted. They are not quality benchmarks. No perplexity, no task evaluation, no comparison against another implementation's output quality. The correctness evidence is separate and stronger: see ../TESTING.md and the logs in tests/fixtures/gates/ and tests/fixtures/golden/.