Official Inference Code of EAGLE-4
type: reference owner: eagle-harness last_verified: 2026-10-23
Known limits
Use the support matrix for measured model, hardware and workload combinations. Correctness and speed claims come from those measured runs. A loaded model, an admitted request cap or a compiled kernel is a startup check.
Models and hardware
- EAGLE-4 drafts need the trained candidate head and a supported draft structure.
- Dense and latent-MoE EAGLE-4 drafts and Mamba2 targets have adapters. Nemotron 3.5 Super runs 1.4× to 2.5× faster than plain decoding on SGLang, vLLM and TokenSpeed with 0.1.2 (measured rows). Its twenty-prompt checks have first divergences beyond the fp32 rule. At each one, the same engine without speculation emits the same token at the same prefix, so they count as the engine's own numerics. An architecture that loads is not yet qualified.
- Serving measurements cover NVIDIA B300 and SGLang on AMD MI355X. Compilation on another NVIDIA or AMD architecture is separate evidence. Multi-node deployment and pipeline parallelism are being qualified.
- Throughput depends on the model, draft, engine, precision, TP size, attention backend, context length and workload. A result at one concurrency does not establish a result at another.
High load
On SGLang with four GPUs, the high-load recipe keeps pace with plain decoding up to 512 concurrent requests. These limits apply around it:
- The SGLang command of the README quick start decodes without speculation at high load. With its default cache settings it measured 1.00× the same engine without speculation at 64, 128 and 256 concurrent requests and 0.99× at 512 on one B300, and from 0.99× to 1.03× at 64 to 512 on four (4 starts each). On four GPUs at 8 concurrent requests its throughput is 0.97× that of 0.1.2 (0.1.3 release notes).
- On one GPU, a speculative server started with
--disable-radix-cachestops during draft graph capture, so keep the radix cache on there. The high-load recipe is for four GPUs, and the server log's advice to add--disable-radix-cacheapplies to four GPUs only. - The first start of the high-load recipe on a machine compiles kernels and measures its live costs during its first requests. It measured 0.95× at 64 and 0.97× at 128 concurrent requests, and restarts with warm caches 1.01× and 1.00× (3 starts each).
- The packaged choices cover B300 TP4 with the Qwen3.8-27B checkpoint pair. On other setups the first start also searches its choices while serving: with the packaged choices withheld on that same setup, first starts measured 0.80× at concurrency 64 (4 starts, from 0.71× to 0.96×) and restarts 0.94× (2 starts). From 128 concurrent requests up they measured 0.97× or more.
- Requests decoded without speculation stop drafting until they finish, and so do requests that join their batch. After a load spike, a batch keeps decoding without speculation until it empties, even when the load drops in the meantime.
- Greedy agreement at 64 and 512 concurrent requests is checked with the fp32 rule, not claimed. At 64, one prompt
takes another token at one position in 3 of 12 EAGLE-4 starts, a prefix where the engine without speculation
emits the same token under another batch shape. At 512, the engine without speculation also differs between its
own starts and between batch sizes: 3 and 1 of 512 prompts fall outside the 2-ULP rule, up to 3.8 bf16 ULP. Its
default kernels are not batch-invariant. EAGLE-4 decodes without speculation at this load, and its differences
fall on the same prompts. With
--enable-deterministic-inferenceon both servers, restarts and three EAGLE-4 starts match token for token at 512 (512 of 512 prompts). In that mode EAGLE-4 decodes without speculation at 64 and 512, so it gives no speedup there, and the mode lowers throughput. Every twenty-prompt check passes. - vLLM and TokenSpeed do not switch to plain decoding at high load yet. Above concurrency 64 they can be slower with speculation than without it, so serve those loads without speculation for now. The switch is planned for both engines.
Performance and correctness in progress
- SGLang greedy correctness at 64 or more concurrent requests: 7 of 12,096 checked outputs in the 0.1.3 release check fall outside the fp32 tolerance and are not reproduced by plain decoding at the same prefix. They are on the one-GPU README command at 128 concurrent requests (one prompt in each of four starts), the four-GPU measured recipe at 64 (one prompt in each of two starts) and the four-GPU high-load recipe at 512 (one prompt in one start). Version 0.1.4 compares against plain decoding at the same batch size (0.1.3 release notes).
- SGLang TP4 single-request and 32K-context optimization is in progress, as are TP8 concurrency-8 prefill overhead and intermittent stalls. Repeated passes in one process are not independent server starts.
- SGLang native-OFF, including OFF forced by a cost profile, uses the engine's per-variant forward-mode
decode graphs (the
forward_modefield of the decode graph variant). The SGLang CUDA engine0.5.23.dev252+eagle.5has them. On the 0.1.0 SGLang CUDA engine and on ROCm, EAGLE-4 keeps native-OFF off. - The SGLang temperature-1 sampled tree (draft proposals drawn in one fused launch and verified against their
recorded probabilities) uses the engine's
sampling_stepverify hook, which the SGLang CUDA engine0.5.23.dev252+eagle.5has. On the 0.1.0 SGLang engines and on ROCm, temperature-1 requests use a greedy draft verified by the target's sampler. On every engine, a batch uses that path when any request has another temperature, a grammar or a penalty. - The shipped vLLM TP4 rounds schedule improves the measured concurrency-64 and concurrency-128 cases at a running cap of 64, at temperature 0 and 1; see the support matrix. The schedule has no TP1/TP8 profile. Running caps of 128 and above are being qualified. Native-OFF is off by default while its TP1 throughput is optimized to match native decoding.
- vLLM TP4 at concurrency 64: in a 64-prompt greedy comparison against no-spec, one prompt can take another token at one position, up to 5.4 bf16 ULP from the fp32 choice, beyond the 2-ULP rule. It appeared in 3 of 6 EAGLE-4 starts with 0.1.1 and in 2 of 6 with 0.1.0, and 0.1.2 shows it too (the vLLM engine is the same); the other prompts and every twenty-prompt check at concurrency 1 pass. vLLM without speculation emits the same token at that prefix when 64 sequences share the batch (up to 19 of 64 identical copies in one batch), so the difference comes from the engine's batch-dependent numerics, not from tree verification.
- AMD measurements apply to the MI355X TP1 Qwen3.8 profile. On one MI355X the speedup covers concurrency up to 32; concurrency 64 is being optimized, so serve it without speculation for now. HIP side-stream prelaunch stays off: its replay check passes and it measured no throughput gain on MI355X. Further target, draft and verify phase optimization is in progress.
Requests
- Client concurrency can exceed the server's running cap and include queued requests. Keep both values with benchmark results. A cap of 512 is an admission limit, not a claim of acceleration or correctness at that load.
- SGLang supports accepted-token and input logprobs with the required host hooks. Images are refused.
- vLLM tree verification needs an unquantized KV cache. The target's sampling restrictions still apply.
Sampling smoke tests check the path, not distribution equivalence on every workload. On TP4 at concurrency 1,
8 and 64, the vLLM tree's sampled tokens match the engine without speculation at temperature 1 and 0.7, with
top-k 8 and top-p 0.7, with a repetition penalty of 1.5, and with frequency and presence penalties of 0.5 (two
tree starts against three starts without speculation, chi-square tests at a 1 % family-wise level). vLLM
refuses
min_pandlogit_biaswith speculative decoding. - Structured output and tool calls require the model's engine parser. Raw tool syntax in text is not a structured response. In a vLLM tree batch with a grammar constraint, every request in the batch advances one token per verify round. Check finish reasons and validate the completed output. See compatibility.
TokenSpeed
TokenSpeed serves EAGLE-4 in serve mode with calibrated trees, chains and sampled trees on NVIDIA GPUs. Install the
engine, its native packages and eagle-harness together with the install command,
because the adapter needs the matching engine hooks. On four B300 GPUs, the 0.1.3 build passed TP4 calibration,
greedy serving from that profile (twenty prompts: 3 identical, 17 first divergences within the fp32 tolerance) and
sampled tree serving at temperature 1. The 0.1.2 measurements cover serve mode at TP1, TP4 and TP8, temperature 0
and 1, 32K- and 128K-token contexts and four-GPU chains on the 0.1.2 engine build
(support matrix).
- Static mode (
"mode": "static") stops at server start with the released EAGLE-4 draft, whose attention layer uses a 4096-token sliding window. The TokenSpeed 0.1.3 engine raisesdraft trees have no sliding-window or attention-sink path yetwhen it builds the draft tree. Serve mode runs the draft tree ineagle-harnessand is not affected. Static-mode support for sliding-window drafts is planned for 0.1.4. - TokenSpeed runs on CUDA only. On AMD GPUs, serve with SGLang ROCm (AMD guide).
- On one GPU, start the server with
--disable-kvstore. By default TokenSpeed adds a host KV cache tier twice the size of the GPU's KV cache, and a one-GPU server cannot pin that much host memory. The guide's commands pass the flag. - Some request features are limits of the TokenSpeed engine, with or without speculation:
- Prompt logprobs: a request with
logprob_start_len0 returns output-token logprobs only. Top-k logprobs needmax_new_tokensof 1. Output-token logprobs match the engine without speculation (see the support matrix). - Sampling filters: with FlashInfer sampling, top-k/top-p and min-p cut the distribution where the engine's own sampler cuts it, which is not where SGLang cuts it. The sampled-distribution check therefore covers temperature, unrestricted and penalty settings; for top-k/top-p and min-p only its per-token tests apply.
- Sampling stream: with FlashInfer sampling and a draft tree, TokenSpeed refuses
--sampling-stream per-request, because its tree verifier draws one random stream for the batch. Use--sampling-stream batch, or chains. - Draft trees run at most 32 requests at a time: the engine sizes its tree buffers for 32. Chains run up to 512.
- Native no-speculation rounds: TokenSpeed has no hook that skips the draft for a batch yet, so EAGLE-4 cannot fall back to plain decoding at high load on TokenSpeed (high load). A zero-draft speculative round is not a native no-speculation round.
- Live cost measurement: serve mode reads cost tables measured for TokenSpeed.
- Prompt logprobs: a request with
- Serve mode needs cost tables measured for TokenSpeed. Tree and chain cost data should match the served shape, TP size, target, draft and recurrent-state dtype. EAGLE-4 0.1.3 ships no TokenSpeed tables: calibrate once on your setup (TokenSpeed guide). The 0.1.3 build was checked at TP4. TP1, TP8 and chains were measured on 0.1.2.
- Retraction is not qualified. TokenSpeed reserves pages when it admits a request and queues the rest, so the measured servers never retract. With retractions forced by a test-only setting, one prompt in three chain starts fails the fp32 rule (support matrix).
- EAGLE-3 drafts are not checked on TokenSpeed. TP4 single-request optimization is in progress.
- On Ubuntu 26.04 / glibc 2.43, set
CUDA_HOMEto CUDA 13.2 before the first start, which compiles the tree-round CUDA sources (troubleshooting).
Calibration
Cost tables are configuration-specific. Keep the recorded settings with them and measure again after changing the model, draft, TP size, precision, attention backend or engine. Keyed tables are checked at loading, but the key does not encode every setting. A cache or table is not evidence of an optimal policy for every workload.
The vLLM public sweep measures batch-one tree forwards, not complete high-concurrency serving rounds.
The SGLang served-chain driver measures a separate path from dynamic-tree calibration. Its CLI uses the
per-variant native-decode graphs of the SGLang CUDA engine 0.5.23.dev252+eagle.5; live-cost tree serving runs on
both CUDA engines. Built-in smoke prompts check the command, not representative acceptance. An interrupted
run keeps its partial output for diagnosis and writes serve.sh only on success. See
calibration.
Distribution
The install guide describes the release path: eagle-harness packages on PyPI,
patched engine wheels on the GitHub Pages indexes, and images on GHCR. Model and draft weights come from
Hugging Face under their owners' licenses.
- Use the install guide for image versions. TokenSpeed's
:0.1.3image uses the hooks-only engine and harness plugin; earlier TokenSpeed image checks are historical, not qualification of this build. - The files under
docker/show how the release images are built. The checked CUDA images carry a prebuilteagle-harnesswheel, so the published images are the checked path. The TokenSpeed Dockerfile takes an explicit compatibleBASE_IMAGE. - On AMD, install with pip: the ROCm closure is clean-installed from the public indexes and inference-tested on an MI355X. The ROCm Dockerfile is being container-qualified, and no ROCm image is published yet.
- TokenSpeed serve mode uses cost tables measured for TokenSpeed on the served setup. EAGLE-4 does not ship them, so the TokenSpeed guide calibrates them before the first server start.
- Use the pinned artifacts: stock engines and unpinned packages do not carry the EAGLE-4 hooks.