Skip to content

Official Inference Code of EAGLE-4

docs: state the parity rule as the same engine and mark the vLLM Nemotron tree

Seyfal Sultanovd800de818448
diff --git a/docs/known-limits.md b/docs/known-limits.md
index 2231f8a..8f13b10 100644
--- a/docs/known-limits.md
+++ b/docs/known-limits.md
@@ -16,8 +16,9 @@ kernel is a startup check.
- Dense and latent-MoE EAGLE-4 drafts and Mamba2 targets have adapters. Nemotron 3.5 Super runs 1.4× to 2.5×
faster than plain decoding on SGLang, vLLM and TokenSpeed with 0.1.2
([measured rows](support-matrix.md#012-nemotron-35-super)). Its twenty-prompt checks have first divergences
- beyond the fp32 rule, and at each one an engine without speculation emits the same token at the same prefix, so
- they count as the engine's own numerics. An architecture that loads is not yet qualified.
+ beyond the fp32 rule. At all but one, the same engine without speculation emits the same token at the same prefix,
+ so they count as the engine's own numerics. The remaining one is on the vLLM tree, whose cells are marked ⚠️ until
+ vLLM without speculation settles it. An architecture that loads is not yet qualified.
- Serving measurements cover NVIDIA B300 and SGLang on AMD MI355X. Compilation on another NVIDIA or AMD
architecture is separate evidence. Multi-node deployment and pipeline parallelism are being qualified.
- Throughput depends on the model, draft, engine, precision, TP size, attention backend, context length and
diff --git a/docs/support-matrix.md b/docs/support-matrix.md
index de3112b..3fc31c5 100644
--- a/docs/support-matrix.md
+++ b/docs/support-matrix.md
@@ -20,8 +20,8 @@ share a row. Column headers are client concurrency, the number of requests the c
| — | Not reported |
At concurrency 64 and above, and in the Nemotron twenty-prompt checks, a difference beyond the fp32 rule counts as
-the engine's own numerics and passes when a server without speculation emits that token at the same prefix and the
-number of such differences stays within the range between that server's restarts.
+the engine's own numerics and passes when the same engine without speculation, in the same job, emits that token at
+the same prefix and the number of such differences stays within the range between that engine's restarts.
[How these were measured](#how-these-were-measured) defines the greedy check, the default workloads and the terms.
@@ -352,7 +352,7 @@ same server without speculation, in the same allocation, 2 starts per arm, tempe
|---|---|---|---|
| SGLang tree | ✅ 2.48× | ✅ 1.78× | ✅ 1.70× |
| SGLang chain, TRT-LLM MHA target attention, page size 64 | ✅ 2.52× | — | — |
-| vLLM tree | ✅ 2.37× | ✅ 1.84× | ✅ 1.41× |
+| vLLM tree | ⚠️ 2.37× | ⚠️ 1.84× | ⚠️ 1.41× |
| vLLM 8-token chain | — | ✅ 1.78× | ✅ 1.95× |
| TokenSpeed tree | ✅ 2.00× | ✅ 1.89× | — |
@@ -368,9 +368,11 @@ Sources:
chain check has 2.
3. TokenSpeed: running cap 8, float32 recurrent state, 11 and 22 first turns with up to 256 tokens. 619.1 against
309.6 and 2173.4 against 1149.5 tokens/s. The check has 3 failures.
-4. At every failure, an engine without speculation emits the same token at the same prefix under another batch shape
- (the same engine in 11 of 12, TokenSpeed for the remaining vLLM row). These differences count as the engines'
- own numerics under the [rule above the tables](#measured-support-matrix), so every check passes.
+4. At 11 of the 12 failures, the same engine without speculation emits the same token at the same prefix under
+ another batch shape. These differences count as the engines' own numerics under the
+ [rule above the tables](#measured-support-matrix), so those checks pass.
+5. vLLM tree: one of its 3 failures (2.04 bf16 ULP) is reproduced by TokenSpeed without speculation but not yet by
+ vLLM without speculation, so it does not meet the rule and the vLLM tree cells are marked ⚠️.
## Unreleased (measured on candidates, not in a release)