Skip to content

Official Inference Code of EAGLE-4

6.6 KB
Raw Blame

EAGLE-4 0.1.3

EAGLE-4 combines a draft model with eagle-harness inside your inference engine to adapt speculative decoding to your hardware, batch and workload. Version 0.1.3 brings native decoding at high load to the SGLang README command, including the one-GPU setup, and fills draft KV without prediction work for prompts admitted above the verify limit. It also builds engine wheels and release images from approved inputs and explains how to warm up a server before measuring it.

This release ships SGLang CUDA 0.5.23.dev252+eagle.9, keeps vLLM 0.32.0rc1.dev283+eagle.4 unchanged, and adds TokenSpeed 0.1.1+eagle.8, built from eagle-hq/tokenspeed eagle/v0.1.3. The TokenSpeed engine exposes generic engine hooks, and EAGLE-4 policy and tree verification live in the harness plugin. TokenSpeed serves EAGLE-4 in serve mode from cost tables you calibrate once on your setup. On four B300 GPUs this build passed calibration, greedy serving from the calibrated profile (twenty prompts: 3 identical to TokenSpeed without speculation, 17 first divergences within the fp32 tolerance, the same in two starts) and sampled serving at temperature 1. TokenSpeed's 0.1.2 speed measurements remain historical results, not measurements of this build.

Install

You don't need to download anything from the release page. In a Python 3.12 environment, install the SGLang packages together:

pip install "eagle-harness[sglang]==0.1.3" "sglang==0.5.23.dev252+eagle.9" \
  --extra-index-url https://eagle-hq.github.io/EAGLE-4/whl/cu130/ \
  --extra-index-url https://download.pytorch.org/whl/cu130 \
  --extra-index-url https://pypi.nvidia.com

Follow the SGLang guide, vLLM guide or TokenSpeed guide to serve an EAGLE-4 draft. Install TokenSpeed with its matching scheduler, kernel and control-plane wheels using the install guide. The SGLang, vLLM and TokenSpeed images are :0.1.3.

Measured speed

These measurements use Qwen3.8-27B with the EAGLE-4 draft on NVIDIA B300 GPUs and greedy decoding. Each cell is EAGLE-4 output throughput divided by the same engine's throughput without speculation, averaged over four starts per configuration. Client concurrency ranges from 1 to 512 for the README and high-load recipes, and from 1 to 64 for the measured recipes. Controls use the same recipe, tensor-parallel size and running-request cap. These throughput results do not imply a passing correctness check, as described below.

Engine and command GPUs 1 request 8 64 128 256 512
SGLang README 1 2.93× 1.99× 1.00× 1.00× 1.00× 0.99×
SGLang README 4 2.34× 1.64× 0.99× 1.00× 1.02× 1.03×
SGLang measured recipe 4 3.37× 2.06× 1.27× Not measured Not measured Not measured
SGLang high-load recipe 4 2.90× 1.81× 1.00× 0.98× 1.01× 0.99×
vLLM measured recipe 4 2.91× 1.74× 1.00× Not measured Not measured Not measured

The corresponding EAGLE-4 throughput is in output tokens per second:

Engine and command GPUs 1 request 8 64 128 256 512
SGLang README 1 309.5 1,356.6 3,336.8 3,975.7 3,764.4 3,714.5
SGLang README 4 577.7 2,549.3 8,083.9 12,614.1 17,079.3 18,687.9
SGLang measured recipe 4 736.5 2,959.6 5,699.3 Not measured Not measured Not measured
SGLang high-load recipe 4 729.7 2,972.0 8,321.0 12,134.7 15,447.2 17,216.5
vLLM measured recipe 4 696.5 2,632.1 7,135.1 Not measured Not measured Not measured

The SGLang high-load recipe and warm-up guidance explain the serving setup and warm-up. The one-GPU and four-GPU README commands use their default cache settings.

Contributor fixes

Four tooling fixes in this release come from @kevin9327: cuda_redist.py installs CUDA on Python 3.10, wheels without a description keep their added License-File headers, the wheel index tool writes its root page before any artifact exists, and stop_check.py reports a workload file that is too short before it sends any request. @lunar-me fixed grammar in the FAQ and known limits. Thank you both.

Limits

The release correctness check compares each EAGLE-4 output with the same engine's output without speculation on the same prompts, and judges every first difference against an fp32 recomputation of that position. Across the four SGLang configurations, 12,089 of 12,096 checked outputs (prompts over starts and concurrency levels) pass. Of the 15 outputs that first fell outside the tolerance, 8 carry a token that plain decoding on the same engine build, recipe and tensor-parallel size also emits at the exact prefix, in a replay batch no larger than the tested concurrency. Numerical thresholds stay unchanged. Seven outputs remain outside the tolerance:

  • SGLang README command on one GPU at 128 concurrent requests: one prompt in each of four starts.
  • SGLang measured recipe on four GPUs at 64 concurrent requests: one prompt in each of two starts.
  • SGLang high-load recipe on four GPUs at 512 concurrent requests: one prompt in one start.

The vLLM measured recipe passes its release correctness check. Version 0.1.4 will compare against plain decoding at the same batch size.

On the four-GPU SGLang README command at 8 concurrent requests, 0.1.3 reaches 0.97× the throughput of 0.1.2 (2,549.3 against 2,624.1 output tokens per second, four starts each), below the 0.98× release threshold. The speed measurements cover the model, commands and GPU counts above. SGLang's high-load recipe measures 0.98× plain-decoding throughput at 128 concurrent requests, and some other high-load cells are slightly below 1.00×. This release makes no claim that speculation is faster for every workload.

TokenSpeed static mode ("mode": "static") stops at server start with the released EAGLE-4 draft, whose attention layer uses a 4096-token sliding window. The TokenSpeed 0.1.3 engine has no sliding-window path for draft trees yet. Serve mode runs the draft tree in eagle-harness and is the supported TokenSpeed path. Static-mode support is planned for 0.1.4. The known limits describe the remaining engine and request-feature constraints.

Files on the release page

Install eagle-harness from PyPI. The engine index selects the matching engine wheels from their release assets. The source archives are for reading or building the code, not required for installation.