Official Inference Code of EAGLE-4
EAGLE-4 0.1.5
EAGLE-4 combines a draft model with eagle-harness inside your inference engine to adapt speculative decoding to your hardware, batch and workload. Version 0.1.5 improves sampled decoding and startup on SGLang. Sampled draft trees now propose tokens from the request's own temperature, top-k, top-p and min-p settings, so more drafted tokens are accepted when you sample. The server also builds more of its kernels before it serves the first request.
- Sampled trees follow the request's sampling settings. The draft tree draws its proposals with the request's temperature, top-k, top-p and min-p. It applies those filters in the engine's order.
- Draining batches keep their prices. When a batch shrinks to a size with no measured cost yet, live costs price it from the nearest measured sizes.
- More work before the first request. The server compiles the round tail's kernels and the native-decode position kernel at startup.
- Remaining requests keep their drafts. When some requests in a batch finish, the requests that remain keep their prelaunched draft trees. The server captures CUDA graphs for the round tail only after a batch size recurs.
- Draft attention splits by batch size. The draft's attention picks its key splits from the batch's size.
Every change is in the SGLang path of eagle-harness. vLLM and TokenSpeed serving are unchanged, and the binary wheel carries the same native libraries as 0.1.4.
The engine pins don't change: SGLang CUDA 0.5.23.dev252+eagle.9, vLLM 0.32.0rc1.dev283+eagle.5 and TokenSpeed 0.1.1+eagle.8.
Install
You don't need to download anything from the release page. In a Python 3.12 environment, install the SGLang packages together:
pip install "eagle-harness[sglang]==0.1.5" "sglang==0.5.23.dev252+eagle.9" \
--extra-index-url https://eagle-hq.github.io/EAGLE-4/whl/cu130/ \
--extra-index-url https://download.pytorch.org/whl/cu130 \
--extra-index-url https://pypi.nvidia.com
To upgrade a 0.1.4 environment, run pip install "eagle-harness==0.1.5" in it, then run eagle-harness-check. Nothing in the configuration changed. The install guide lists the vLLM and TokenSpeed pins, and the release images stay at :0.1.3 with eagle-harness 0.1.3 installed.
What was checked
We served Qwen3.8-27B with the EAGLE-4 draft on SGLang with the high-load recipe on four B300 GPUs. We compared 0.1.5 with 0.1.4 and with the same engine without speculation, using interleaved server starts. There were three starts each for 0.1.4 and plain decoding, and two completed starts for 0.1.5 (see Limits).
Greedy speed. 0.1.5 measured 1.005× the throughput of 0.1.4 at 1 request, 0.997× at 8 and 0.983× at 64. That is 718 against 714 output tokens per second at 1 request, 2,918 against 2,927 at 8, and 8,003 against 8,142 at 64. Each 0.1.5 mean is inside the range of the 0.1.4 starts. At 64 requests the recipe decodes without speculation in both versions.
Sampling acceptance. At temperature 1, the mean accepted length per round rose:
Sampling Requests 0.1.4 0.1.5 top-k 8, top-p 0.7 1 4.30 4.55 top-k 8, top-p 0.7 8 3.88 4.16 min-p 0.05 1 4.12 4.17 min-p 0.05 8 3.81 3.99 no filter 1 4.01 4.04 no filter 8 3.84 3.86 Sampled throughput measured 0.98× to 1.01× that of 0.1.4, and each 0.1.5 mean is inside the range of the 0.1.4 starts, apart from one cold-cache start described under Limits.
Greedy correctness. On 20 prompts of 512 tokens at 1 request, the 0.1.5 starts produced output identical to the first 0.1.4 start on 20 of 20 and 18 of 20 prompts. The two 0.1.4 repeat starts matched it on 20 and 19 of 20, and one of the 0.1.5 differences is at the same token of the same prompt as the 0.1.4 one. Every completed start of both versions passes the release correctness check against SGLang without speculation: each first difference is within the tolerance of an fp32 recomputation.
Sampling correctness. The sampled-output distribution check passes for top-k with top-p and for min-p against SGLang without speculation.
Release packages. The binary wheel's Python files are the ones served in these checks, apart from the version string, and its native libraries are byte-identical to those in the 0.1.4 wheel.
Limits
- Known issue: one server start stopped with a CUDA illegal memory access. In one of three 0.1.5 starts, the SGLang scheduler stopped with
CUDA error: an illegal memory access was encounteredon one of its first requests. That start reused the live-cost cache written by the start before it. The other two 0.1.5 starts, one of which loaded the same cache, and all three 0.1.4 starts ran without error. We have not reproduced it yet and are investigating. If a server stops this way, restart it. - Compile pauses on a cold cache. SGLang compiles some sampling kernels the first time a request needs them. One 0.1.5 start measured 592 output tokens per second at 8 requests with min-p because that compile ran during the measurement, and one 0.1.4 start measured 579 at 1 request for the same reason. Warm up the server before you time it.
- Speed was measured with the model, recipe and GPU count above. The 0.1.3 speed tables remain the current speed measurements, and the 0.1.3 limits and known limits still apply.
Files on the release page
Install eagle-harness from PyPI. The engine index selects the matching engine wheels from their release assets.
The source archives are for reading or building the code, not required for installation.