Skip to content

Official Inference Code of EAGLE-4

5.5 KB
Raw Blame

type: reference owner: eagle-harness last_verified: 2026-10-23

EAGLE-4 0.1.2 release notes

EAGLE-4 0.1.2 updates eagle-harness on the 0.1.1 engines. On SGLang, EAGLE-4 decodes a batch without speculation wherever speculation would be slower, so the high-load recipe keeps pace with plain decoding up to 512 concurrent requests on four B300 GPUs and keeps its speedup at low load. eagle-harness calibrate --engine vllm completes on hybrid targets, and the TokenSpeed draft runs its candidate head on the engine's low-latency GEMM.

Upgrade

A new install follows the installation guide, which installs 0.1.2. To upgrade a 0.1.1 environment, run your engine's install command with the pins below, then eagle-harness-check:

Engine 0.1.2 pins
SGLang CUDA eagle-harness[sglang]==0.1.2, sglang==0.5.23.dev252+eagle.5 (unchanged)
vLLM eagle-harness[vllm]==0.1.2, vllm==0.32.0rc1.dev283+eagle.4 (unchanged)
TokenSpeed This engine build was withdrawn. Use the current TokenSpeed install command, not the historical build measured below.

The SGLang and vLLM :0.1.2 images carry eagle-harness 0.1.2 on the same engines. The TokenSpeed image from this release was withdrawn. The install guide lists the current images. Nothing in the 0.1.2 configuration changed: every 0.1.1 configuration loaded as before.

What changed

SGLang

  • A served tree, the default, decides per batch size whether to speculate at all. It prices speculation by the tokens its rounds commit over whole served intervals, measures plain decoding last, and decodes a batch without speculation where plain decoding wins. Requests decoded that way stop drafting until they finish.
  • Served chains ("intensity": "chain_intensity") pick one verify width per batch size the same way, and decode without speculation where that wins.
  • On B300 TP4 with the Qwen3.8-27B checkpoint pair, the first start serves packaged choices and searches only the batch sizes in between. The choices are cached next to the live costs, so a restart on any setup serves them from its first round. Calibration describes both policies.
  • Every KV-move program a tree round runs compiles at server startup, and rounds that stalled on one-time work or followed a switch between speculation and plain decoding stay out of the prices.
  • The switch needs the radix cache off (--disable-radix-cache), because with it on this hybrid model tracks prefix-cache state that plain decoding does not carry. A server with the radix cache on speculates at every load and says so in its log.

vLLM

  • eagle-harness calibrate --engine vllm completes on the Qwen3.8 hybrid target: its token budget fits one batch, the tables are keyed by the models the engine resolves, and the printed serve.sh loads them.

TokenSpeed

  • The draft scores its low-rank candidate head through the engine's linear layer, so TokenSpeed's GEMM dispatch serves the few draft rows with its low-latency BF16 kernel. Checkpoints load as before.

Measured on the release stack

Qwen3.8-27B with its EAGLE-4 draft on 4x B300 (TP4), BF16, temperature 0, on the release candidate build installed into clean environments and served from outside any checkout. Each speedup divides EAGLE-4 throughput by the same engine without speculation, measured in the same allocation. Every EAGLE-4 start passed its twenty-prompt greedy check against no-spec under the fp32 rule.

Engine and command Concurrency EAGLE-4 tokens/s Speedup EAGLE-4 starts
SGLang, measured recipe 1 733.8 3.35× 2
SGLang, measured recipe 8 3257.9 2.11× 2
SGLang, measured recipe 64 6488.2 1.38× 2
SGLang, high-load recipe 64 7393.3 1.01× 3
SGLang, high-load recipe 128 11114.3 1.00× 3
SGLang, high-load recipe 256 14332.3 1.00× 3
SGLang, high-load recipe 512 16197.5 0.99× 3
vLLM, measured recipe 1 678.5 2.83× 4
vLLM, measured recipe 8 2887.4 1.73× 4
vLLM, measured recipe 64 8013.4 1.19× 4
TokenSpeed, measured recipe 1 886.5 3.13× 2
TokenSpeed, measured recipe 8 3256.4 2.00× 2
TokenSpeed, measured recipe 64 6974.1 1.40× 2

The high-load rows are restarts with warm caches. A first start on a machine measured 0.95× at 64 and 0.97× at 128 while it compiles kernels and measures costs. With 0.1.1, the same SGLang setup at a running cap of 512 measured 0.74× at 256 and 0.61× at 512. At concurrency 1, SGLang and TokenSpeed serve multi-turn conversations with up to 1024 new tokens per turn and vLLM 22 first turns with up to 512. The support matrix has temperature 1, one GPU, no-spec throughput and the conditions of each row.

Limits

  • The SGLang quick-start command is slower than plain decoding above about 32 to 64 concurrent requests, on one GPU and with --tp-size 4. For high load, serve with the high-load recipe on four GPUs. A fix for the quick-start command is planned.
  • High load on one GPU is not supported yet. On one GPU, keep the radix cache on.
  • vLLM and TokenSpeed do not switch to plain decoding at high load yet. Above concurrency 64, serve them without speculation for now.
  • Known limits has the measured numbers and the rest of the limits, and the support matrix has every measured configuration.