Skip to content

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

9.8 KB
Raw Blame

Performance

All figures measured on a single machine: AMD EPYC 7763, 124 vCPU (2×62, no SMT), 228 GiB RAM, NVMe measured at 3.2 GB/s O_DIRECT, Ubuntu 24.04, GCC 13.3, AVX2 without AVX-512. Background auto-updates were disabled before measurement.

Every table on this page is transcribed from raw output in data/. The full machine capture is data/environment.txt. Nothing here is quoted from memory or rounded by hand, if a table and its data file disagree, the data file is right and this page has a bug.

Read the noise floor before quoting any single number. Three runs of an identical configuration span 33%. Differences smaller than that are not effects.

The memory ladder

Twelve budgets, each enforced by a hard cgroup cap, same prompt, same binary. Raw: data/memory-ladder.tsv.

RAM pinned layers expert cache s/token vs 8 GB expert hit trunk hit GB read/tok peak RSS
8 0 0.49 32.69 1.00× 0.0% 0.0% 25.83 8.24
12 0 2.79 31.41 1.04× 0.0% 0.0% 25.83 10.53
16 3 4.39 32.21 1.01× 0.0% 2.8% 25.83 16.00
24 7 7.58 31.85 1.03× 0.0% 6.6% 25.83 23.95
32 11 10.80 31.44 1.04× 0.0% 10.3% 25.83 31.90
48 19 17.19 29.76 1.10× 0.0% 17.9% 25.83 47.80
64 27 23.59 28.60 1.14× 0.0% 25.4% 25.83 63.71
96 43 36.39 24.40 1.34× 29.9% 40.5% 18.11 95.51
128 60 49.19 29.40 1.11× 32.2% 56.5% 17.51 128.18
160 76 61.99 26.31 1.24× 33.1% 71.5% 17.28 159.98
192 90 77.00 21.32 1.53× 35.5% 84.7% 16.65 191.83
224 90 108.98 19.21 1.70× 43.8% 84.7% 14.53 223.82

Every row produced byte-identical output: 17374, 20829, 10, 427, 414, 1008, 606, 142957. 28x the memory changes the speed and nothing else.

Two caches, and only one of them matters below 96 GB

The engine has two independent caches. They do not behave alike.

The expert cache does nothing below ~36 GB of arena. From 28 slots to 1,344 slots the retention stays at 0.0% and the bytes read per token stay pinned at exactly 25.83 GB. Seven consecutive budgets, a 48× increase in cache size, zero change in bytes moved.

This is a property of the model, not a defect in the cache. K3's router is trained with Quantile Balancing, which deliberately flattens expert usage across the pool. With 16 of 896 experts selected per layer and no hot subset to exploit, there is nothing for an LRU to retain.

The trunk ring is the one that responds. Its hit rate tracks pinned layers almost exactly:

pinned predicted (n/93) measured
3 3.2% 2.8%
7 7.5% 6.6%
11 11.8% 10.3%
27 29.0% 25.4%
90 96.8% 96.8%¹

¹ steady state. The 8-token runs report 84.7% because token 0 pays the pinning cost and tokens 1 to 7 each hit 90 layers: 7 × 90 / 744 = 84.7%.

The mechanism is simply arithmetic: per token the engine reads 108.81 GB of trunk against 25.83 GB of experts, 4.2× the bytes and 2.9× the time. A gigabyte given to the trunk removes ~1.17 GB/token of guaranteed traffic. A gigabyte given to the expert cache removes, below the knee, nothing.

How much cache would be enough

The ladder can only answer this where it happened to sample. Replaying the recorded request trace answers it everywhere, for free data/expert-cache-capacity.txt, regenerated by:

python3 tools/sim_cache.py tests/fixtures/expert_trace.bin

100,096 recorded expert requests, 10,010 distinct experts (12.1% of the pool):

cache slots LRU Belady GB read/token
8 GB 455 36.2% 39.4% 16.47
32 GB 1,823 36.2% 49.0% 16.47
64 GB 3,647 36.2% 61.7% 16.47
128 GB 7,294 49.2% 84.6% 13.12
192 GB 10,941 90.0% 90.0% 2.58
1450 GB 82,633 90.0% 90.0% 2.58

LRU is flat from 8 GB to 64 GB, an 8× increase in capacity buying nothing, then saturates at 192 GB, which is where the working set finally fits. Belady's optimal policy climbs steadily over the same range, so the flatness is LRU's, not the workload's: there is exploitable locality, and LRU cannot reach it.

Two things to hold onto:

  • This x-axis is cache size; the ladder's is total budget. The ladder's 8 GB row has only 0.49 GB of cache (28 slots), which is why it shows 0.0% retention and this table shows 36.2% at 455 slots. They do not disagree.
  • These are upper bounds, not forecasts. The trace was recorded during full-recompute decode, which re-touches the same experts ~68 times. Steady-state incremental decode has far less reuse and will do worse. The file says so itself.

The practical reading: below ~192 GB of cache alone, capacity is not the lever. That is the same conclusion the fixed-budget sweep below reaches from the opposite direction.

Allocation beats capacity

At a fixed total budget, varying only the split. All twelve rows, from data/trunk-cache-split.tsv:

total trunk cache s/token total trunk cache s/token
128 12.3 110.7 28.38 32 2.7 24.3 31.23
128 30.8 92.2 25.20 32 6.8 20.2 30.58
128 49.2 73.8 25.69 32 10.8 16.2 31.25
128 73.8 49.2 18.37 32 16.2 10.8 30.88
128 98.4 24.6 19.46 32 21.6 5.4 29.13
128 110.0 13.0 16.80 32 25.6 1.4 28.06

1.69× faster at identical total memory (28.38 → 16.80 at 128 GB). Note that the fastest configuration has 0.0% expert-cache retention.

How far this goes. Each row is a single sample against a 33% noise floor, so the endpoint comparison alone would not settle it, and the trend is not strictly monotonic. There are two inversions at 128 GB (+1.9% at 30.8→49.2, +5.9% at 73.8→98.4) and one at 32 GB (+2.2%). Every one of them is far inside the noise floor, so none is evidence against the trend; equally, no individual step is evidence for it.

What carries the conclusion is the rank correlation over twelve points across two independent budgets: Spearman ρ = −0.886 at 128 GB and −0.714 at 32 GB. Two separately run sweeps, at a 4× difference in total memory, both order the splits the same way.

The magnitude is a different matter. 1.69× is one endpoint comparison of single samples and is not replicated to the three-run standard this project asks for elsewhere (see CONTRIBUTING.md). Treat the direction as established and the factor as approximate; benchmarks/split-sweep.sh re-runs it with repetitions.

128 GB allocated well (16.80 s/token) beats 224 GB allocated badly (19.21). If you take one tuning decision from this document, take that one. It is also the one the mechanism predicts independently of any measurement: the trunk is re-read in full every token and the experts are not.

The noise floor

Three runs of an identical configuration, back to back, on a quiet machine. Raw: data/replication.tsv.

run s/token
1 14.78
2 14.67
3 20.14

mean 16.53, sd 3.13, spread 33.1%.

Same binary, same prompt, same flags, same minute. A third of the measurement is noise.

What this means for every table above. Each row is a single sample. Any difference under ~33% is not an established effect. Concretely:

  • The 128 GB row (29.40) sitting above its neighbours (24.40, 26.31) is inside the band. It is dispersion, not an anomaly, and it needs no explanation.
  • Individual step-to-step differences in the middle of the ladder are not resolvable.

What survives, being larger than the floor:

  • the ladder's overall span, 8 GB → 224 GB, a 70% change;
  • trunk-first at 128 GB, a 69% change.

What is untouched by noise entirely, these are counts and byte totals, not stopwatch readings, and no amount of scheduling jitter moves them:

  • byte-identical output at all twelve budgets;
  • GB read/token pinned at exactly 25.83 across seven of them;
  • expert retention of 0.0% from 28 to 1,344 slots;
  • trunk hit ≈ pinned/93.

So the mechanism rests on counts and stands. What is soft is the precise second-count attached to any one configuration.

Likely contributors to the variance, none isolated: virtualised NVMe latency on a workload that moves ~135 GB per token; page-cache interaction at 128 GB RSS on a 228 GiB box; NUMA effects across two sockets; and cold-start amortisation, every ladder run is only 8 tokens, and token 0 pays the full pinning cost. The last is supported by the end-to-end generations at 16 to 32 tokens, which spread only 10% (10.66 to 11.79 s/token) against this run's 33%.

Longer runs are faster

Because cold start is amortised:

tokens s/token
8 (ladder, best) 19.21
16 to 32 (sustained generations) 10.66 to 11.79

The ladder understates sustained throughput. Quote it as a comparison across budgets, not as the engine's speed.

Reproducing

make -j
./scripts/k3-doctor.sh ~/k3model          # what this machine should manage
./benchmarks/memory-ladder.sh ~/k3model ~/k3trunk out/   # the full sweep

See BENCHMARKING.md, in particular, replicate before you conclude. Replication is not a formality here: on a 33% noise floor, a single-sample difference of 20% is indistinguishable from dispersion, and treating one as a finding is the easiest mistake to make with this engine.

The raw output of the run these tables come from is in data/, so a new run can be compared against it directly rather than against the numbers as transcribed.