Skip to content

Official Inference Code of EAGLE-4

7.3 KB
Raw Blame

type: reference owner: eagle-harness last_verified: 2026-10-21

EAGLE-4 0.1.1 release notes

EAGLE-4 0.1.1 updates eagle-harness and two engine builds. An SGLang server now needs only the draft path to serve EAGLE-4, starts its first requests faster and runs on a new SGLang CUDA engine with native no-speculation rounds and the temperature-1 sampled tree. vLLM measures the served tree acceptance it schedules from, and the TokenSpeed integration moves to a new engine build. The vLLM engine is unchanged.

Upgrade

A new install follows the installation guide, which installs 0.1.1. To upgrade a 0.1.0 environment, run your engine's install command with the pins below, then eagle-harness-check:

Engine 0.1.1 pins
SGLang CUDA eagle-harness[sglang]==0.1.1, sglang==0.5.23.dev252+eagle.5
vLLM eagle-harness[vllm]==0.1.1, vllm==0.32.0rc1.dev283+eagle.4 (unchanged)
TokenSpeed This engine build was withdrawn. Use the current TokenSpeed install command, not the historical build measured below.

The SGLang and vLLM :0.1.1 images carried these engines with eagle-harness 0.1.1. The TokenSpeed image from this release was withdrawn. The install guide lists the current images.

Breaking: the configuration is streamlined. A config that still sets "mode": "eval", ragged (enabled, force_uniform), cost_model.ragged_draft_relax, cost_model.ragged_verify_threshold or draft_model.moe_gate_acc stops at load with <option>: removed in eagle-harness 0.1.1. Delete the option (use "mode": "serve" instead of "eval"). The migration guide lists every removed option and environment variable. No shipped configuration uses them.

What changed

SGLang

  • sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 --speculative-algorithm EAGLE_HARNESS --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 is a complete launch: EAGLE-4 drafts a tree of up to 12 steps with 4 candidates per step, verifies up to 16 tokens, prices every round from live costs and runs on SGLang's default overlap scheduler. On one B300 (TP1) this launch measured 342 tokens/s at concurrency 1 and 1,581 tokens/s at concurrency 8, 3.35x and 2.21x the same engine without speculation (2 starts each).
  • First requests start faster. The draft's first-use kernels and softmax compile at server startup, and a slow first round no longer enters the live costs. At TP4 the longest pause in a new server's first requests fell from 5.2 s to 0.7 s.
  • Long-context first requests capture their CUDA graphs without flushing the allocator. At a 32K context (TP4, concurrency 1) EAGLE-4 measured 715 tokens/s, 2.98x the same engine without speculation (4 starts).
  • The SGLang CUDA engine 0.5.23.dev252+eagle.5 adds per-variant native decode graphs, so EAGLE-4 can run a round without speculation where the live costs select it, and the engine's sampling_step verify hook, so temperature-1 requests draw their draft tree in one fused launch and verify it against the recorded proposal probabilities. It also passes the sliding-window lengths the Triton attention backend needs.
  • 16-token top-1 chains served with cost tables size each round with a chain-specific depth rule. Narrower chains keep the tree rule.
  • The SGLang recipes use the engine's prefill CUDA graph.

vLLM

  • The plugin measures the acceptance of the trees it serves and chooses the batch size where the tree narrows from round-weighted gains. The documented TP4 recipe keeps its packaged schedule.
  • The plugin's INFO messages appear in the vLLM server log, including the active tree rounds file and its SHA256.
  • Setup-aware calibration: chain widths, cost tables and rounds/tree.json load only when their key.json matches the GPU, TP size, model pair, engine and dtype being served.

TokenSpeed

  • The now-withdrawn TokenSpeed engine build measured for 0.1.1. Its historical TP4 serve-mode speedups are in the support matrix.
  • Serve mode is the default, as on the other engines. A launch needs a data_path or a complete schedule; {"mode": "static"} selects static mode.
  • The plugin negotiates its contract with the engine by capability.

AMD ROCm

  • The SGLang ROCm path installs from public indexes with one hash-locked closure (docker/rocm/requirements.lock). See SGLang ROCm on AMD.

Correctness tools (in the repository's tools/)

  • tools/fp32_arbiter.py re-forwards a greedy first divergence above the near-tie gap in fp32. It passes when the fp32 gap is within 2 bf16 ULP, or when our token is the fp32 argmax.
  • tools/sampling_check.py names every sampling setting explicitly, judges truncated settings at fixed contexts against an fp32 re-forward, and accepts a no-speculation server as its own reference.

Ready for upcoming engines. The plugin also registers on the engine changes under upstream review: vLLM's suspend_drafting and verify_budget speculator hooks, SGLang's msgspec accept types, and TokenSpeed capability negotiation. None of this changes behaviour on the pinned engines.

Measured on the release stack

Qwen3.8-27B with its EAGLE-4 draft on 4x B300 (TP4), BF16, installed from the 0.1.1 wheels into clean environments and served from outside any checkout. Each speedup divides EAGLE-4 throughput by the same engine without speculation, measured in the same allocation. Every EAGLE-4 start passed its twenty-prompt greedy check against no-spec under the fp32 rule.

Engine Workload Concurrency EAGLE-4 tokens/s Speedup EAGLE-4 starts
SGLang Multi-turn chat, temperature 0 1 750.6 3.44× 3
SGLang Chat, temperature 0 8 3203.1 2.08× 3
SGLang Chat, temperature 0 64 6454.1 1.39× 3
SGLang Multi-turn chat, temperature 1 1 597.3 2.76× 3
SGLang Chat, temperature 1 8 2741.8 1.75× 3
SGLang Chat, temperature 1 64 5752.2 1.22× 3
SGLang 32K-token context, temperature 0 1 715.3 2.98× 4
vLLM Chat, temperature 0 1 681.0 2.83× 6
vLLM Chat, temperature 0 8 2894.8 1.72× 6
vLLM Chat, temperature 0 64 7935.3 1.14× 6

Concurrency 1 uses 22 conversations with up to 1024 new tokens per turn (SGLang) or 22 first turns with up to 512 (vLLM); concurrency 8 and 64 use 88 and 220 first turns with up to 512 new tokens. The SGLang rows use the tuned command of the SGLang guide and the vLLM rows the tuned command of the vLLM guide.

Limits

  • SGLang above concurrency 64 is slower with speculation than without at a running cap of 512 (0.74× at 256, 0.61× at 512); serve those loads without speculation for now.
  • The :0.1.1 images (eagle-4-sglang, eagle-4-vllm, eagle-4-tokenspeed) are CUDA only. There is no SGLang ROCm image: build it yourself.
  • The SGLang ROCm install keeps sglang==0.5.23.dev252+eagle.4.rocm and, through its hash lock, eagle-harness 0.1.0 until the lock moves to the published 0.1.1 wheel.
  • Known limits lists the rest, and the support matrix has every measured configuration.