Skip to content

Official Inference Code of EAGLE-4

docs: 0.1.1 release notes, release-stack numbers and limits

Seyfal Sultanov2ace92d51ee2
diff --git a/docs/known-limits.md b/docs/known-limits.md
index 297778f..0e7e531 100644
--- a/docs/known-limits.md
+++ b/docs/known-limits.md
@@ -1,7 +1,7 @@
---
type: reference
owner: eagle-harness
-last_verified: 2026-10-20
+last_verified: 2026-10-21
---
# Known limits
@@ -25,22 +25,28 @@ kernel is a startup check.
- SGLang TP4 single-request and 32K-context optimization is in progress, as is TP8 concurrency-8 prefill
overhead and intermittent stalls. Repeated passes in one process are not independent server starts.
-- SGLang automatic high-load policy and high-load fixed-chain improvements are being qualified and stay off by
- default, as does work on high-load and Nemotron first divergences. A release-engine cap-512 startup and
- request smoke passes; high-load acceleration and concurrent correctness are being measured.
+- SGLang above concurrency 64: with a running cap of 512 and the default tree (TP4), EAGLE-4 measured 1.03x the
+ same engine without speculation at concurrency 64, 0.73x at 256 and 0.61x at 512 (3 starts each). Serve
+ concurrency above 64 without speculation for now. The automatic high-load policy and high-load fixed-chain
+ improvements are being qualified and stay off by default. Greedy agreement with no-spec at concurrency 512 is
+ not claimed: the engine without speculation also differs between two starts there.
- SGLang native-OFF, including OFF forced by a cost profile, uses the engine's per-variant forward-mode
- decode graphs (the `forward_mode` field of the decode graph variant). They arrive with the next SGLang
- engine release; on the 0.1.0 SGLang CUDA engine EAGLE-4 keeps native-OFF off.
+ decode graphs (the `forward_mode` field of the decode graph variant). The SGLang CUDA engine
+ `0.5.23.dev252+eagle.5` has them. On the 0.1.0 SGLang CUDA engine and on ROCm, EAGLE-4 keeps native-OFF off.
- The SGLang temperature-1 sampled tree (draft proposals drawn in one fused launch and verified against their
- recorded probabilities) uses the engine's `sampling_step` verify hook, which arrives with the next SGLang
- engine release. On the 0.1.0 SGLang engines temperature-1 requests use a greedy draft verified by the
- target's sampler.
+ recorded probabilities) uses the engine's `sampling_step` verify hook, which the SGLang CUDA engine
+ `0.5.23.dev252+eagle.5` has. On the 0.1.0 SGLang engines and on ROCm, temperature-1 requests use a greedy draft
+ verified by the target's sampler.
On every engine, a batch uses that path when any request has another temperature, a grammar or a penalty.
- The shipped vLLM TP4 rounds schedule improves the measured concurrency-64 and concurrency-128 cases at a
running cap of 64, at temperature 0 and 1; see the
[support matrix](support-matrix.md#measured-high-load-tree-schedule). The schedule has no TP1/TP8 profile.
Running caps of 128 and above are being qualified. Native-OFF is off by default while its TP1 throughput is
optimized to match native decoding.
+- vLLM TP4 at concurrency 64: in a 64-prompt greedy comparison against no-spec, one prompt can take another token
+ at one position, up to 5.4 bf16 ULP from the fp32 choice, beyond the 2-ULP rule. It appeared in 3 of 6 EAGLE-4
+ starts with 0.1.1 and in 2 of 6 with 0.1.0 (the vLLM engine is the same); the other prompts and every
+ twenty-prompt check at concurrency 1 pass. The cause is being investigated.
- AMD measurements apply to the MI355X TP1 Qwen3.8 profile. On one MI355X the speedup covers concurrency up
to 32; concurrency 64 is being optimized, so serve it without speculation for now. HIP side-stream prelaunch
stays off: its replay check passes and it measured no throughput gain on MI355X. Further target, draft and
@@ -84,8 +90,8 @@ key does not encode every setting. A cache or table is not evidence of an optima
The vLLM public sweep measures batch-one tree forwards, not complete high-concurrency serving rounds.
The SGLang served-chain driver measures a separate path from dynamic-tree calibration. Its CLI uses the
-per-variant native-decode graphs that arrive with the next SGLang engine release; live-cost tree serving runs on
-the current engine. Built-in smoke prompts check the command, not representative acceptance. An interrupted
+per-variant native-decode graphs of the SGLang CUDA engine `0.5.23.dev252+eagle.5`; live-cost tree serving runs on
+both CUDA engines. Built-in smoke prompts check the command, not representative acceptance. An interrupted
run keeps its partial output for diagnosis and writes `serve.sh` only on success. See
[calibration](calibration.md).
@@ -97,7 +103,8 @@ Hugging Face under their owners' licenses.
- The published SGLang and vLLM images passed serving and correctness checks, and the published TokenSpeed image
passed a GPU check in serve mode ([support matrix](support-matrix.md#artifact-installation-checks)). They carry
- `eagle-harness` 0.1.0.
+ `eagle-harness` 0.1.0 and the 0.1.0 engines; images with 0.1.1 follow. Use the image commands in the engine guides
+ with them.
- The files under `docker/` show how the release images are built. The checked CUDA images carry a prebuilt
`eagle-harness` wheel, so the published images are the checked path. The TokenSpeed Dockerfile takes an
explicit compatible `BASE_IMAGE`.
diff --git a/docs/release-notes/0.1.1.md b/docs/release-notes/0.1.1.md
new file mode 100644
index 0000000..7b69d22
--- /dev/null
+++ b/docs/release-notes/0.1.1.md
@@ -0,0 +1,115 @@
+---
+type: reference
+owner: eagle-harness
+last_verified: 2026-10-21
+---
+
+# EAGLE-4 0.1.1 release notes
+
+EAGLE-4 0.1.1 updates `eagle-harness` and two engine builds. An SGLang server now needs only the draft path to
+serve EAGLE-4, starts its first requests faster and runs on a new SGLang CUDA engine with native no-speculation
+rounds and the temperature-1 sampled tree. vLLM measures the served tree acceptance it schedules from, and the
+experimental TokenSpeed integration moves to a new engine build. The vLLM engine is unchanged.
+
+## Upgrade
+
+A new install follows the [installation guide](../install.md), which installs 0.1.1. To upgrade a 0.1.0
+environment, run your engine's install command with the pins below, then `eagle-harness-check`:
+
+| Engine | 0.1.1 pins |
+| --- | --- |
+| SGLang CUDA | `eagle-harness[sglang]==0.1.1`, `sglang==0.5.23.dev252+eagle.5` |
+| vLLM | `eagle-harness[vllm]==0.1.1`, `vllm==0.32.0rc1.dev283+eagle.4` (unchanged) |
+| TokenSpeed, experimental | `eagle-harness[tokenspeed]==0.1.1`, `tokenspeed==0.1.1+eagle.5`, `tokenspeed-kernel==0.1.4+eagle.5`, `tokenspeed-scheduler==0.1.40+eagle.4` |
+
+The published `0.1.0` images keep working with the commands in the engine guides. Images with 0.1.1 follow.
+
+**Breaking:** the configuration is streamlined. A config that still sets `"mode": "eval"`, `ragged`
+(`enabled`, `force_uniform`), `cost_model.ragged_draft_relax`, `cost_model.ragged_verify_threshold` or
+`draft_model.moe_gate_acc` stops at load with `<option>: removed in eagle-harness 0.1.1`. Delete the option
+(use `"mode": "serve"` instead of `"eval"`). The [migration guide](../migration.md#removed-in-011) lists every
+removed option and environment variable. No shipped configuration uses them.
+
+## What changed
+
+**SGLang**
+- `sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 --speculative-algorithm EAGLE_HARNESS
+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4` is a complete launch: EAGLE-4 drafts a tree of
+ up to 12 steps with 4 candidates per step, verifies up to 16 tokens, prices every round from live costs and
+ runs on SGLang's default overlap scheduler. On one B300 (TP1) this launch measured 342 tokens/s at concurrency 1
+ and 1,581 tokens/s at concurrency 8, 3.35x and 2.21x the same engine without speculation (2 starts each).
+- First requests start faster. The draft's first-use kernels and softmax compile at server startup, and a slow
+ first round no longer enters the live costs. At TP4 the longest pause in a new server's first requests fell from
+ 5.2 s to 0.7 s.
+- Long-context first requests capture their CUDA graphs without flushing the allocator. At a 32K context (TP4,
+ concurrency 1) EAGLE-4 measured 715 tokens/s, 2.98x the same engine without speculation (4 starts).
+- The SGLang CUDA engine `0.5.23.dev252+eagle.5` adds per-variant native decode graphs, so EAGLE-4 can run a
+ round without speculation where the live costs select it, and the engine's `sampling_step` verify hook, so
+ temperature-1 requests draw their draft tree in one fused launch and verify it against the recorded proposal
+ probabilities. It also passes the sliding-window lengths the Triton attention backend needs.
+- 16-token top-1 chains served with cost tables size each round with a chain-specific depth rule. Narrower
+ chains keep the tree rule.
+- The SGLang recipes use the engine's prefill CUDA graph.
+
+**vLLM**
+- The plugin measures the acceptance of the trees it serves and chooses the batch size where the tree narrows
+ from round-weighted gains. The documented TP4 recipe keeps its packaged schedule.
+- The plugin's INFO messages appear in the vLLM server log, including the active tree rounds file and its SHA256.
+- Setup-aware calibration: chain widths, cost tables and `rounds/tree.json` load only when their `key.json`
+ matches the GPU, TP size, model pair, engine and dtype being served.
+
+**TokenSpeed (experimental)**
+- The engine build `tokenspeed==0.1.1+eagle.5` with `tokenspeed-kernel==0.1.4+eagle.5`. TokenSpeed stays
+ experimental, without a throughput claim.
+- Serve mode is the default, as on the other engines. A launch needs a `data_path` or a complete schedule;
+ `{"mode": "static"}` selects static mode.
+- The plugin negotiates its contract with the engine by capability.
+
+**AMD ROCm**
+- The SGLang ROCm path installs from public indexes with one hash-locked closure
+ (`docker/rocm/requirements.lock`). See [SGLang ROCm on AMD](../install.md#sglang-rocm-on-amd).
+
+**Correctness tools** (in the repository's `tools/`)
+- `tools/fp32_arbiter.py` re-forwards a greedy first divergence above the near-tie gap in fp32. It passes when the
+ fp32 gap is within 2 bf16 ULP, or when our token is the fp32 argmax.
+- `tools/sampling_check.py` names every sampling setting explicitly, judges truncated settings at fixed contexts
+ against an fp32 re-forward, and accepts a no-speculation server as its own reference.
+
+**Ready for upcoming engines.** The plugin also registers on the engine changes under upstream review: vLLM's
+`suspend_drafting` and `verify_budget` speculator hooks, SGLang's msgspec accept types, and TokenSpeed capability
+negotiation. None of this changes behaviour on the pinned engines.
+
+## Measured on the release stack
+
+Qwen3.8-27B with its EAGLE-4 draft on 4x B300 (TP4), BF16, installed from the 0.1.1 wheels into clean environments
+and served from outside any checkout. Each speedup divides EAGLE-4 throughput by the same engine without
+speculation, measured in the same allocation. Every EAGLE-4 start passed its twenty-prompt greedy check against
+no-spec under the fp32 rule.
+
+| Engine | Workload | Concurrency | EAGLE-4 tokens/s | Speedup | EAGLE-4 starts |
+| --- | --- | ---: | ---: | ---: | ---: |
+| SGLang | Multi-turn chat, temperature 0 | 1 | 750.6 | 3.44× | 3 |
+| SGLang | Chat, temperature 0 | 8 | 3203.1 | 2.08× | 3 |
+| SGLang | Chat, temperature 0 | 64 | 6454.1 | 1.39× | 3 |
+| SGLang | Multi-turn chat, temperature 1 | 1 | 597.3 | 2.76× | 3 |
+| SGLang | Chat, temperature 1 | 8 | 2741.8 | 1.75× | 3 |
+| SGLang | Chat, temperature 1 | 64 | 5752.2 | 1.22× | 3 |
+| SGLang | 32K-token context, temperature 0 | 1 | 715.3 | 2.98× | 4 |
+| vLLM | Chat, temperature 0 | 1 | 681.0 | 2.83× | 6 |
+| vLLM | Chat, temperature 0 | 8 | 2894.8 | 1.72× | 6 |
+| vLLM | Chat, temperature 0 | 64 | 7935.3 | 1.14× | 6 |
+
+Concurrency 1 uses 22 conversations with up to 1024 new tokens per turn (SGLang) or 22 first turns with up to 512
+(vLLM); concurrency 8 and 64 use 88 and 220 first turns with up to 512 new tokens. The SGLang rows use the tuned
+command of the [SGLang guide](../quickstart-sglang.md) and the vLLM rows the tuned command of the
+[vLLM guide](../quickstart-vllm.md).
+
+## Limits
+
+- SGLang above concurrency 64 is slower with speculation than without at a running cap of 512 (0.73× at 256,
+ 0.61× at 512); serve those loads without speculation for now.
+- The `:0.1.0` images keep the 0.1.0 engines and `eagle-harness` 0.1.0; images with 0.1.1 follow.
+- The SGLang ROCm install keeps `sglang==0.5.23.dev252+eagle.4.rocm` and, through its hash lock,
+ `eagle-harness` 0.1.0 until the lock moves to the published 0.1.1 wheel.
+- [Known limits](../known-limits.md) lists the rest, and the [support matrix](../support-matrix.md) has every
+ measured configuration.
diff --git a/docs/support-matrix.md b/docs/support-matrix.md
index 9936c32..5908a08 100644
--- a/docs/support-matrix.md
+++ b/docs/support-matrix.md
@@ -15,9 +15,9 @@ matched no-speculation control. [Reading the numbers](#reading-the-numbers) defi
|---|---:|---:|---|---:|---:|---:|
| 1 | 1 | 1 | Irrelevant at batch 1 | 103.1 | 343.1 | 3.33× |
| 1 | 8 | 8 | Disabled in measurement | 711.7 | 1503.7 | 2.11× |
-| 4 | 1 | 32 | Irrelevant at batch 1 | 216.3 | 735.7 | 3.40× |
-| 4 | 8 | 32 | Default, enabled | 1479.4 | 2820.6 | 1.91× |
-| 4 | 64 | 32 | Disabled in measurement | 3925.3 | 5281.3 | 1.35× |
+| 4 | 1 | 32 | Irrelevant at batch 1 | 218.5 | 750.6 | 3.44× |
+| 4 | 8 | 32 | Default, enabled | 1541.7 | 3203.1 | 2.08× |
+| 4 | 64 | 32 | Default, enabled | 4655.3 | 6454.1 | 1.39× |
| 8 | 1 | 1 | Irrelevant at batch 1 | 241.7 | 828.6 | 3.43× |
### B300 SGLang measurement conditions
@@ -28,13 +28,13 @@ Concurrency 1 uses 22 conversations and 31 turns with at most 1024 new tokens pe
88 first turns with at most 512 new tokens. Concurrency 64 uses 220 first turns with the same 512-token cap.
Every timed pass follows a warm pass. EAGLE-4 measures live costs. Each EAGLE-4 configuration has two starts.
-TP4 concurrency 1 measures the released stack: `sglang==0.5.23.dev252+eagle.4` from the CUDA wheel index and
-`eagle-harness==0.1.0` from PyPI, installed into a clean environment and served by the SGLang quickstart command
-from a directory outside any checkout. It has six EAGLE-4 starts (725.2 to 752.7 tokens/s) and three no-spec
-starts, interleaved in one allocation. Every EAGLE-4 start gives 5 exact / 15 near tie / 0 fp32 / 0 failed in its
-twenty-prompt check.
-
-TP4 concurrency 8 measures 2749.5 and 2891.8 tokens/s across its two starts.
+The TP4 rows measure the 0.1.1 release stack: `sglang==0.5.23.dev252+eagle.5` and `eagle-harness` 0.1.1,
+installed by the install guide's SGLang commands into a clean environment and served by the guide's tuned SGLang
+command from a directory outside any checkout. Three EAGLE-4 starts and one no-spec start ran interleaved in one
+allocation; EAGLE-4 measured 750.2 to 751.3 tokens/s at concurrency 1, 3183.1 to 3237.0 at concurrency 8 and
+6273.7 to 6591.8 at concurrency 64. Every EAGLE-4 start gives 4 exact / 16 near tie / 0 failed in its
+twenty-prompt check. At temperature 1 the same starts measure 597.3, 2741.8 and 5752.2 tokens/s at concurrency 1,
+8 and 64, 2.76×, 1.75× and 1.22× the no-spec control at temperature 1.
TP8 concurrency 1 and 8 each give 5 exact / 14 near tie / 1 fp32-tolerance pass, with zero failures
in the configuration-specific twenty-prompt checks. TP8 concurrency-8 throughput variability is being investigated.