Official Inference Code of EAGLE-4
type: reference owner: eagle-harness last_verified: 2026-10-23
EAGLE-4 0.1.2 release notes
EAGLE-4 0.1.2 updates eagle-harness on the 0.1.1 engines. On SGLang, EAGLE-4 decodes a batch without speculation
wherever speculation would be slower, so the high-load recipe keeps pace with
plain decoding up to 512 concurrent requests on four B300 GPUs and keeps its speedup at low load.
eagle-harness calibrate --engine vllm completes on hybrid targets, and the TokenSpeed draft runs its candidate head
on the engine's low-latency GEMM.
Upgrade
A new install follows the installation guide, which installs 0.1.2. To upgrade a 0.1.1
environment, run your engine's install command with the pins below, then eagle-harness-check:
| Engine | 0.1.2 pins |
|---|---|
| SGLang CUDA | eagle-harness[sglang]==0.1.2, sglang==0.5.23.dev252+eagle.5 (unchanged) |
| vLLM | eagle-harness[vllm]==0.1.2, vllm==0.32.0rc1.dev283+eagle.4 (unchanged) |
| TokenSpeed | This engine build was withdrawn. Use the current TokenSpeed install command, not the historical build measured below. |
The SGLang and vLLM :0.1.2 images carry eagle-harness 0.1.2 on the same engines. The TokenSpeed image from
this release was withdrawn. The install guide lists the current images.
Nothing in the 0.1.2 configuration changed: every 0.1.1 configuration loaded as before.
What changed
SGLang
- A served tree, the default, decides per batch size whether to speculate at all. It prices speculation by the tokens its rounds commit over whole served intervals, measures plain decoding last, and decodes a batch without speculation where plain decoding wins. Requests decoded that way stop drafting until they finish.
- Served chains (
"intensity": "chain_intensity") pick one verify width per batch size the same way, and decode without speculation where that wins. - On B300 TP4 with the Qwen3.8-27B checkpoint pair, the first start serves packaged choices and searches only the batch sizes in between. The choices are cached next to the live costs, so a restart on any setup serves them from its first round. Calibration describes both policies.
- Every KV-move program a tree round runs compiles at server startup, and rounds that stalled on one-time work or followed a switch between speculation and plain decoding stay out of the prices.
- The switch needs the radix cache off (
--disable-radix-cache), because with it on this hybrid model tracks prefix-cache state that plain decoding does not carry. A server with the radix cache on speculates at every load and says so in its log.
vLLM
eagle-harness calibrate --engine vllmcompletes on the Qwen3.8 hybrid target: its token budget fits one batch, the tables are keyed by the models the engine resolves, and the printedserve.shloads them.
TokenSpeed
- The draft scores its low-rank candidate head through the engine's linear layer, so TokenSpeed's GEMM dispatch serves the few draft rows with its low-latency BF16 kernel. Checkpoints load as before.
Measured on the release stack
Qwen3.8-27B with its EAGLE-4 draft on 4x B300 (TP4), BF16, temperature 0, on the release candidate build installed into clean environments and served from outside any checkout. Each speedup divides EAGLE-4 throughput by the same engine without speculation, measured in the same allocation. Every EAGLE-4 start passed its twenty-prompt greedy check against no-spec under the fp32 rule.
| Engine and command | Concurrency | EAGLE-4 tokens/s | Speedup | EAGLE-4 starts |
|---|---|---|---|---|
| SGLang, measured recipe | 1 | 733.8 | 3.35× | 2 |
| SGLang, measured recipe | 8 | 3257.9 | 2.11× | 2 |
| SGLang, measured recipe | 64 | 6488.2 | 1.38× | 2 |
| SGLang, high-load recipe | 64 | 7393.3 | 1.01× | 3 |
| SGLang, high-load recipe | 128 | 11114.3 | 1.00× | 3 |
| SGLang, high-load recipe | 256 | 14332.3 | 1.00× | 3 |
| SGLang, high-load recipe | 512 | 16197.5 | 0.99× | 3 |
| vLLM, measured recipe | 1 | 678.5 | 2.83× | 4 |
| vLLM, measured recipe | 8 | 2887.4 | 1.73× | 4 |
| vLLM, measured recipe | 64 | 8013.4 | 1.19× | 4 |
| TokenSpeed, measured recipe | 1 | 886.5 | 3.13× | 2 |
| TokenSpeed, measured recipe | 8 | 3256.4 | 2.00× | 2 |
| TokenSpeed, measured recipe | 64 | 6974.1 | 1.40× | 2 |
The high-load rows are restarts with warm caches. A first start on a machine measured 0.95× at 64 and 0.97× at 128 while it compiles kernels and measures costs. With 0.1.1, the same SGLang setup at a running cap of 512 measured 0.74× at 256 and 0.61× at 512. At concurrency 1, SGLang and TokenSpeed serve multi-turn conversations with up to 1024 new tokens per turn and vLLM 22 first turns with up to 512. The support matrix has temperature 1, one GPU, no-spec throughput and the conditions of each row.
Limits
- The SGLang quick-start command is slower than plain decoding above about 32 to 64 concurrent requests, on one
GPU and with
--tp-size 4. For high load, serve with the high-load recipe on four GPUs. A fix for the quick-start command is planned. - High load on one GPU is not supported yet. On one GPU, keep the radix cache on.
- vLLM and TokenSpeed do not switch to plain decoding at high load yet. Above concurrency 64, serve them without speculation for now.
- Known limits has the measured numbers and the rest of the limits, and the support matrix has every measured configuration.