Skip to content

Official Inference Code of EAGLE-4

6.8 KB
Raw Blame

type: tutorial owner: eagle-harness last_verified: 2026-10-23

SGLang guide

Serve Qwen3.8-27B and its EAGLE-4 draft with SGLang. SGLang prices the draft tree from costs it measures on the served rounds, so nothing needs calibrating first. For MI355X, use the AMD guide.

Start a server

Install the engine with the SGLang install command, then start the server. The model and the draft download from Hugging Face on the first start:

sglang serve --model-path Qwen/Qwen3.8-27B \
  --speculative-algorithm EAGLE_HARNESS \
  --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4

Or run the published image, which has the engine and EAGLE-4 installed:

docker run --gpus all --ipc host -p 127.0.0.1:30000:30000 \
  -e HF_HOME=/hf -v ~/.cache/huggingface:/hf \
  ghcr.io/eagle-hq/eagle-4-sglang:0.1.3 \
  sglang serve --model-path Qwen/Qwen3.8-27B --host 0.0.0.0 \
  --disable-overlap-schedule --speculative-algorithm EAGLE_HARNESS \
  --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 \
  --speculative-num-steps 12 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \
  --speculative-algorithm-config '{}'

Both commands serve on one GPU. Add --tp-size 4 to serve on four, as the measured recipe below does.

By default EAGLE-4 drafts a tree of up to 12 steps with 4 candidates per step, verifies up to 16 tokens, and prices every round from live costs. --speculative-num-steps, --speculative-eagle-topk, --speculative-num-draft-tokens and --speculative-algorithm-config override those (configuration). These defaults need eagle-harness 0.1.1 or newer, which the image carries. The image command spells the same settings out and adds --disable-overlap-schedule; EAGLE-4 also serves with SGLang's default overlap scheduler.

Send a request

When the server logs The server is fired up and ready to roll!, send a request from another terminal:

curl http://localhost:30000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model": "Qwen/Qwen3.8-27B", "messages": [{"role": "user", "content": "Explain how a hash table works."}], "max_tokens": 256}'

The response holds the generated message. The first requests compile kernels and seed the live costs, so send a few before you measure speed. Compare against the same engine without the --speculative-* flags.

The measured recipe

The SGLang rows of the support matrix were measured with this command. It adds the attention backends, graph settings, memory sizes and recurrent-state dtype of the benchmark:

export TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=4
export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 SGLANG_TIMEOUT_KEEP_ALIVE=3600
sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 --dtype bfloat16 \
  --trust-remote-code --attention-backend flashinfer --page-size 1 \
  --max-running-requests 32 --cuda-graph-max-bs-decode 32 \
  --disable-overlap-schedule --cuda-graph-backend-decode full \
  --mem-fraction-static 0.75 --context-length 196608 --max-total-tokens 196608 \
  --chunked-prefill-size 16384 --mamba-ssm-dtype bfloat16 --disable-radix-cache \
  --reasoning-parser qwen3 --random-seed 721 --skip-server-warmup \
  --speculative-algorithm EAGLE_HARNESS \
  --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 \
  --speculative-draft-model-quantization unquant --speculative-draft-attention-backend flashinfer \
  --speculative-attention-mode prefill --speculative-num-steps 12 \
  --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \
  --speculative-algorithm-config '{"mode":"serve"}'

Before timing, run one untimed pass of your workload at each concurrency to compile kernels and fill cost caches (benchmark warm-up). The measured command uses client warm-up with --skip-server-warmup. Keep CUDA graph padding enabled, and do not add --disable-cuda-graph-padding. --max-running-requests 32 is the tree default; set a larger cap, up to 512, for larger batches, or serve with the high-load recipe.

High load

For many concurrent requests on four GPUs, serve with the high-load recipe. It raises the running cap and the decode graph limit to 512 and turns the radix cache off, so EAGLE-4 can decode a batch without speculation wherever speculation would be slower:

export TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=4
export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 SGLANG_TIMEOUT_KEEP_ALIVE=3600
sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 --dtype bfloat16 \
  --trust-remote-code --attention-backend flashinfer --page-size 1 \
  --max-running-requests 512 --cuda-graph-max-bs-decode 512 \
  --mem-fraction-static 0.70 --context-length 32768 \
  --chunked-prefill-size 16384 --mamba-ssm-dtype bfloat16 --disable-radix-cache \
  --random-seed 721 --skip-server-warmup \
  --speculative-algorithm EAGLE_HARNESS \
  --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 \
  --speculative-draft-model-quantization unquant --speculative-draft-attention-backend flashinfer \
  --speculative-attention-mode prefill --speculative-num-steps 12 \
  --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \
  --speculative-algorithm-config '{"mode":"serve"}'

With the radix cache on, this hybrid model tracks prefix-cache state that a batch switching to plain decoding cannot carry, so the server speculates at every load. On one GPU, keep the radix cache on: high load on one GPU is not supported yet (known limits).

Each speedup divides EAGLE-4 throughput by the same command without the --speculative-* flags, in the same allocation, on 4x B300 with Qwen3.8-27B and its EAGLE-4 draft at temperature 0. Columns are concurrent requests:

Server start 1 8 64 128 256 512
First start on a machine 2.80× 1.67× 0.95× 0.97× 1.00× 0.99×
Restart with warm caches 2.76× 1.72× 1.01× 1.00× 1.00× 0.99×

The first start compiles kernels and measures its live costs during its first requests, which costs up to 5% at 64 and 128 concurrent requests. A restart reuses the kernels and the costs and choices cached under ~/.cache/eagle-harness/costs. This setup ships packaged choices; on another GPU, TP size or model pair the first start also searches them (known limits). The support matrix has the tokens/s and the correctness checks.

Live costs are cached per GPU, TP size, model pair, engine and dtype under ~/.cache/eagle-harness/costs. The calibration guide explains live costs and offline chain calibration, and the known limits cover sampled requests, logprobs and multimodal inputs.