Official Inference Code of EAGLE-4
EAGLE-4 0.1.4
EAGLE-4 combines a draft model with eagle-harness inside your inference engine to adapt speculative decoding to your hardware, batch and workload. Version 0.1.4 makes vLLM serving on two or more GPUs more robust. During server startup and CUDA graph capture, vLLM issues its symmetric-memory collectives from several CUDA streams, and CUDA does not order work across streams. This release orders those collectives, so each all-reduce and each candidate all-gather runs alone on its buffer on every GPU, and startup and graph capture complete with correct results.
- vLLM's symmetric-memory all-reduce (on by default,
VLLM_ALLREDUCE_USE_SYMM_MEM=1) now waits for the previous all-reduce whenever the calling stream changes. - The
eagle-harnesscandidate all-gather on vLLM does the same.
Calls from one stream were already ordered, so steady-state decoding does no extra work. One-GPU vLLM, SGLang and TokenSpeed do not use these paths.
This release ships vLLM 0.32.0rc1.dev283+eagle.5, built from eagle-hq/vllm eagle/v0.1.4. It is +eagle.4 plus the all-reduce fix, with the same native libraries. SGLang CUDA 0.5.23.dev252+eagle.9 and TokenSpeed 0.1.1+eagle.8 are unchanged.
Install
You don't need to download anything from the release page. In a Python 3.12 environment, install the vLLM packages together:
pip install "eagle-harness[vllm]==0.1.4" "vllm==0.32.0rc1.dev283+eagle.5" \
--extra-index-url https://eagle-hq.github.io/EAGLE-4/whl/cu130/ \
--extra-index-url https://download.pytorch.org/whl/cu130
To upgrade a 0.1.3 vLLM environment, run this command, then run eagle-harness-check. SGLang and TokenSpeed users can install eagle-harness==0.1.4 with their 0.1.3 engine pins: the install guide lists them. Nothing in the configuration changed.
| Engine | 0.1.4 pins |
|---|---|
| SGLang CUDA | eagle-harness[sglang]==0.1.4, sglang==0.5.23.dev252+eagle.9 (unchanged) |
| vLLM | eagle-harness[vllm]==0.1.4, vllm==0.32.0rc1.dev283+eagle.5 |
| TokenSpeed | eagle-harness[tokenspeed]==0.1.4, tokenspeed==0.1.1+eagle.8 and its 0.1.3 scheduler, kernel and engine wheels (unchanged) |
What was checked
- Stress tests on B300. Collectives alternated between two calling streams while one GPU lagged 20 ms on every seventh call, and every result was checked:
- All-reduce: 40 of 40 check windows returned the expected sums at both 8 and 2 GPUs.
- Candidate all-gather: 400 of 400 results matched the expected rows at both 8 and 2 GPUs.
- Serving check. We served Qwen3.8-27B with the EAGLE-4 draft on vLLM and compared the fixed build with 0.1.3 on four B300 GPUs and on one, with greedy decoding. That build also enabled three default-off options that are not part of this release, and they changed no output:
- Outputs: token-identical between the two builds on all 20 prompts, at both GPU counts. Both builds pass the release correctness check against vLLM without speculation.
- Logprobs: identical on all 768 checked positions.
- Speed: four interleaved server starts per build, at 1, 32, 64 and 128 concurrent requests. Every cell is within the start-to-start spread; the fixed build measures 0.99× to 1.04× the throughput of 0.1.3.
- Measured throughput: 713.6 against 712.6 output tokens per second at 1 request on four GPUs, and 7,890 against 7,858 at 64.
- Release packages. A fresh environment installed from the exact 0.1.4 wheels served the same 20 prompts on one B300 with token-identical output and identical logprobs, and the candidate all-gather GPU tests passed.
Limits
This release changes nothing else, so the 0.1.3 limits and known limits still apply. Two items that the 0.1.3 notes planned for 0.1.4 move to a later release:
- Comparing the vLLM check against plain decoding at the same batch size.
- TokenSpeed static mode with the sliding-window EAGLE-4 draft. Serve mode remains the supported TokenSpeed path.
The 0.1.3 speed tables remain the current speed measurements.
Files on the release page
Install eagle-harness from PyPI. The engine index selects the matching engine wheels from their release assets.
The source archives are for reading or building the code, not required for installation.