Skip to content

Official Inference Code of EAGLE-4

docs: simplify the README, install and quickstarts

Seyfal Sultanov18df0e919ebc
diff --git a/README.md b/README.md
index 5d9d17a..748ee5a 100644
--- a/README.md
+++ b/README.md
@@ -10,12 +10,12 @@
<p>
<a href="EAGLE4_Paper.pdf"><b>Paper (EAGLE-4)</b></a> ·
- <a href="https://huggingface.co/EAGLE-corp/EAGLE4-Qwen3.8-27B"><b>EAGLE-4 draft for Qwen3.8-27B</b></a> ·
+ <a href="https://huggingface.co/EAGLE-Corp/Qwen3.8-27B-EAGLE4"><b>EAGLE-4 draft for Qwen3.8-27B</b></a> ·
<a href="docs/README.md"><b>Documentation</b></a>
</p>
<p align="center">
- <a href="https://github.com/eagle-hq/EAGLE-4/releases/tag/v0.1.1"><img src="https://img.shields.io/badge/eagle--harness-v0.1.1-orange.svg" alt="eagle-harness 0.1.1"></a>
+ <a href="https://pypi.org/project/eagle-harness/"><img src="https://img.shields.io/pypi/v/eagle-harness?label=eagle-harness&color=orange" alt="eagle-harness on PyPI"></a>
<a href="https://github.com/eagle-hq/EAGLE-4/issues" target="_blank"><img src="https://img.shields.io/badge/Maintained%3F-yes-green.svg" alt="Maintenance"></a>
<a href="https://github.com/eagle-hq/EAGLE-4/pulls" target="_blank"><img src="https://img.shields.io/badge/Contributions-welcome-brightgreen.svg" alt="Contributions welcome"></a>
</p>
@@ -49,165 +49,81 @@
## EAGLE-4
-EAGLE-4 (Extrapolation Algorithm for Greater Language-model Efficiency) is the latest generation of the EAGLE family
-for accelerating large language model inference. One drafter adapts its speculative decoding strategy to the batch
-size, the hardware and the workload, and keeps the target model's output distribution.
-
-The EAGLE-4 draft uses a sliding-window draft layer, a low-rank candidate head and an append-only draft cache. Dense
-and latent-MoE draft structures are implemented. Qwen3.8-27B with its
-[EAGLE-4 draft](https://huggingface.co/EAGLE-corp/EAGLE4-Qwen3.8-27B) is the measured target pair.
-
-## Engines
-
-This repository is `eagle-harness`, a pip-installable package (`eagle_harness`) that serves EAGLE-4 drafts on
-SGLang, vLLM and TokenSpeed through engine plugins. EAGLE-4 sizes the speculative tree from measured draft and
-verification costs for the batch size, hardware and workload it is running on. It registers through hooks the engine
-exposes and does not patch the engine at runtime.
-
-The patched engines ship as pinned wheels and container images, not as source trees in this repository:
-
-| Engine | Hardware | Wheel index | Image | Source |
-| --- | --- | --- | --- | --- |
-| SGLang | NVIDIA B300, CUDA 13.0 | [cu130](https://eagle-hq.github.io/EAGLE-4/whl/cu130/) | `ghcr.io/eagle-hq/eagle-4-sglang:0.1.0` | [eagle-hq/sglang](https://github.com/eagle-hq/sglang) |
-| SGLang | AMD MI355X, ROCm 7.2 | [rocm](https://eagle-hq.github.io/EAGLE-4/whl/rocm/) | not yet published ([build it](docs/install.md#build-the-rocm-image)) | [eagle-hq/sglang](https://github.com/eagle-hq/sglang) |
-| vLLM | NVIDIA B300, CUDA 13.0 | [cu130](https://eagle-hq.github.io/EAGLE-4/whl/cu130/) | `ghcr.io/eagle-hq/eagle-4-vllm:0.1.0` | [eagle-hq/vllm](https://github.com/eagle-hq/vllm) |
-| TokenSpeed, experimental | NVIDIA CUDA only | [cu130](https://eagle-hq.github.io/EAGLE-4/whl/cu130/) | `ghcr.io/eagle-hq/eagle-4-tokenspeed:0.1.0` | [eagle-hq/tokenspeed](https://github.com/eagle-hq/tokenspeed) |
-
-Stock engine packages do not carry the hooks. [Release settings](docs/release.env) is the one place that holds
-every version pin, index URL and image tag. The wheels are also attached to the
-[v0.1.0 GitHub release](https://github.com/eagle-hq/EAGLE-4/releases/tag/v0.1.0): the engines did not change in 0.1.1.
-The 0.1.0 images carry `eagle-harness` 0.1.0.
-
-## Measured results
-
-The [support matrix](docs/support-matrix.md) reports every measurement with its conditions, separately for
-loading, greedy correctness, request features and throughput. Throughput is generated tokens per second of
-workload time, higher is better. The rows below are the shapes the quickstarts ship. Each one is the mean of
-at least two server starts on the SPEED-Bench client at temperature 0.
-
-| Engine | GPUs | Client concurrency | No-spec tokens/s | EAGLE-4 tokens/s | Speedup |
-| --- | --- | ---: | ---: | ---: | ---: |
-| SGLang | 4× B300, TP4 | 1 | 216.3 | 735.7 | 3.40× |
-| SGLang | 4× B300, TP4 | 8 | 1479.4 | 2820.6 | 1.91× |
-| SGLang | 4× B300, TP4 | 64 | 3925.3 | 5281.3 | 1.35× |
-| SGLang | 1× MI355X, TP1 | 8 | 552.7 | 1145.9 | 2.07× |
-| SGLang | 1× MI355X, TP1 | 32 | 1269.2 | 1980.8 | 1.56× |
-| vLLM | 4× B300, TP4 | 1 | 241.6 | 722.8 | 2.99× |
-| vLLM | 4× B300, TP4 | 8 | 1699.9 | 2883.9 | 1.70× |
-
-Each speedup divides EAGLE-4 throughput by the same engine's matched no-speculation control.
-TokenSpeed is experimental; its measured concurrent throughput is in the
-[support matrix](docs/support-matrix.md#tokenspeed-experimental). An unexplained greedy mismatch fails the
-correctness gate even when the speculative server is faster.
-
-## Install
-
-Install one patched engine per environment, then `eagle-harness==0.1.1` from PyPI. The
-[v0.1.0 GitHub release](https://github.com/eagle-hq/EAGLE-4/releases/tag/v0.1.0) carries the engine wheels, not `eagle-harness`.
-Use Python 3.12 for the CUDA engines and Python 3.10 for the ROCm wheel. The commands read their pins from the
-release settings file in the checkout. Run them, and start servers, from the directory that contains the checkout.
-Inside it, `python -m` would import the source tree's `eagle_harness/` instead of the installed wheel.
+EAGLE-4 (Extrapolation Algorithm for Greater Language-model Efficiency) speeds up LLM inference with speculative
+decoding. One draft model adapts its draft tree to the batch size, the hardware and the workload, and keeps the
+target model's output distribution. On four B300 GPUs it serves Qwen3.8-27B 3.40× faster than SGLang without
+speculation for one request at a time.
+
+This repository is `eagle-harness`, the package that runs EAGLE-4 drafts inside SGLang, vLLM and TokenSpeed.
+
+## Quick start
+
+You need Linux, four NVIDIA GPUs with a CUDA 13 driver (tested on B300) and about 70 GB of disk for
+[Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) and its
+[EAGLE-4 draft](https://huggingface.co/EAGLE-Corp/Qwen3.8-27B-EAGLE4), which download on the first start.
+
+Start an SGLang server with Docker:
```bash
-git clone https://github.com/eagle-hq/EAGLE-4
-. EAGLE-4/docs/release.env
+docker run --gpus all --ipc host -p 127.0.0.1:30000:30000 \
+ -e HF_HOME=/hf -v ~/.cache/huggingface:/hf \
+ ghcr.io/eagle-hq/eagle-4-sglang:0.1.0 \
+ sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 --host 0.0.0.0 \
+ --disable-overlap-schedule --speculative-algorithm EAGLE_HARNESS \
+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 \
+ --speculative-num-steps 12 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \
+ --speculative-algorithm-config '{}'
```
-SGLang on CUDA:
+Or install with pip, in Python 3.12. Pip installs also need the [CUDA toolkit and a C++ compiler](docs/install.md#requirements):
```bash
-python3.12 -m venv .venv-sglang && source .venv-sglang/bin/activate
-python -m pip install --upgrade pip
-python -m pip install --extra-index-url "$EAGLE_CUDA_INDEX" \
- --extra-index-url "$EAGLE_TORCH_CUDA_INDEX" "sglang==$EAGLE_SGLANG_VERSION" 'flashinfer-python>=0.7.2'
-python -m pip install "eagle-harness==$EAGLE_HARNESS_VERSION"
-eagle-harness-check
+python3.12 -m venv .venv && source .venv/bin/activate
+pip install "eagle-harness[sglang]" "sglang==0.5.23.dev252+eagle.5" \
+ --extra-index-url https://eagle-hq.github.io/EAGLE-4/whl/cu130/ \
+ --extra-index-url https://download.pytorch.org/whl/cu130 \
+ --extra-index-url https://pypi.nvidia.com
```
-Or open the release image, which has the engine and EAGLE-4 installed. The 0.1.0 image carries `eagle-harness`
-0.1.0; upgrade it inside the container:
-
```bash
-docker run --rm --gpus all --ipc host -it \
- -p 127.0.0.1:30000:30000 -v "$PWD/models:/models:ro" \
- --entrypoint bash "$EAGLE_SGLANG_IMAGE"
-python -m pip install -U eagle-harness==0.1.1
-eagle-harness-check
+sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 \
+ --speculative-algorithm EAGLE_HARNESS \
+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4
```
-`eagle-harness-check` exits zero when the installed engine provides every hook the plugin needs. It does not
-prove that kernels run or that generation is correct, so send the quickstart request before you measure anything.
+When the server logs `The server is fired up and ready to roll!`, send a request from another terminal:
+
+```bash
+curl http://localhost:30000/v1/chat/completions -H 'Content-Type: application/json' \
+ -d '{"model": "Qwen/Qwen3.8-27B", "messages": [{"role": "user", "content": "Explain how a hash table works."}], "max_tokens": 256}'
+```
-The [installation guide](docs/install.md) has the same steps for [vLLM](docs/install.md#vllm-cuda) and
-[TokenSpeed](docs/install.md#tokenspeed-cuda-experimental), which needs its scheduler and kernel wheels from the
-same index. SGLang on AMD MI355X installs a hash-locked closure from PyPI, the ROCm index and AMD's indexes:
-follow [SGLang ROCm](docs/install.md#sglang-rocm-on-amd), then the [AMD recipe](docs/quickstart-amd.md).
+For vLLM, TokenSpeed (experimental) and AMD MI355X, follow the [engine guides](#more).
-## Serve
+## Measured speedup
-Serve Qwen3.8-27B BF16 with its EAGLE-4 draft on four B300 GPUs. SGLang prices the tree from costs it measures
-during served rounds, so this recipe needs no offline calibration. The full command with its environment is in
-the [SGLang quickstart](docs/quickstart-sglang.md):
+Each speedup divides EAGLE-4 throughput by the same engine's throughput without speculation, on Qwen3.8-27B at
+temperature 0 on the SPEED-Bench client, with the measured recipe of each engine guide. The
+[support matrix](docs/support-matrix.md) has the tokens/s, the correctness checks and how each number was measured.
-```bash
-python -m sglang.launch_server --model-path "$MODEL" --tp-size 4 --dtype bfloat16 \
- --trust-remote-code --attention-backend flashinfer --page-size 1 \
- --max-running-requests 32 --cuda-graph-max-bs-decode 32 \
- --disable-overlap-schedule --cuda-graph-backend-decode full \
- --mem-fraction-static 0.75 --context-length 196608 --max-total-tokens 196608 \
- --chunked-prefill-size 16384 --mamba-ssm-dtype bfloat16 --disable-radix-cache \
- --reasoning-parser qwen3 --random-seed 721 --skip-server-warmup \
- --host 127.0.0.1 --port 30000 \
- --speculative-algorithm EAGLE_HARNESS --speculative-draft-model-path "$DRAFT" \
- --speculative-draft-model-quantization unquant --speculative-draft-attention-backend flashinfer \
- --speculative-attention-mode prefill --speculative-num-steps 12 \
- --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \
- --speculative-algorithm-config '{"mode":"serve"}'
-```
+| Engine | GPUs | 1 request | 8 concurrent | 64 concurrent |
+| --- | --- | ---: | ---: | ---: |
+| SGLang | 4× B300 | 3.40× | 1.91× | 1.35× |
+| vLLM | 4× B300 | 2.99× | 1.70× | 1.16× |
+| TokenSpeed, experimental | 4× B300 | not measured | 2.02× | 1.42× |
-Then send a greedy request and check that the response carries generated text:
+On one AMD MI355X, SGLang with EAGLE-4 is 2.07× faster at 8 concurrent requests and 1.56× at 32.
-```bash
-curl --fail http://127.0.0.1:30000/generate \
- -H 'Content-Type: application/json' \
- -d '{"text":"Explain how a hash table works.","sampling_params":{"temperature":0,"max_new_tokens":128}}'
-```
+## More
-The first request compiles kernels and seeds the cost tables, so repeat it before timing.
-
-The other engines have their own recipes. [vLLM](docs/quickstart-vllm.md) uses a static tree whose depth stops on
-calibrated costs, produced once with `eagle-harness calibrate --engine vllm --model "$MODEL" --draft "$DRAFT" --tp 4`
-while the GPUs are idle, plus the shipped rounds file. [SGLang on MI355X](docs/quickstart-amd.md) selects the
-packaged MI355X cost profile automatically. [TokenSpeed](docs/quickstart-tokenspeed.md) is experimental and
-CUDA-only, with a fixed-shape mode for checking generation without cost tables. The
-[calibration guide](docs/calibration.md) explains the measurement, and [new-model onboarding](docs/new-model.md)
-covers training and exporting a draft for another target.
-
-## What it supports
-
-- Drafts: EAGLE-4 drafts with the low-rank candidate head, tested with Qwen3.8-27B including its GDN layers.
- Mamba2 targets and latent-MoE drafts have adapters on SGLang and vLLM. Per-model correctness coverage is in the
- support matrix.
-- Engines: SGLang (V2 speculative worker), vLLM (Model Runner V2) and experimental TokenSpeed, each through the
- eagle-harness hooks. On SGLang and vLLM a draft top-k of 1 gives a chain and a larger top-k gives a tree.
- TokenSpeed implements static and calibrated trees, chains and sampled trees, without prompt logprobs or native
- no-speculation rounds, and its coverage is limited to the configurations in the support matrix.
-- GPUs: NVIDIA sm_80 and newer, and AMD CDNA2 to CDNA4, through Triton kernels. The kernel tests run on B300
- and the other architectures are compile-checked. MI355X is covered by the serving measurements in the support
- matrix. See [kernels](docs/kernels.md).
-- Requests: text only. vLLM tree verification handles greedy and sampled requests and needs an unquantized KV
- cache. Constrained output takes the correctness fallback in [compatibility](docs/compatibility.md).
-- Batch: SGLang and vLLM trees and chains admit up to 512 running requests within the engine's KV and
- recurrent-state capacity. The SGLang tree default is 32, so set `--max-running-requests` for larger batches.
- No pipeline parallelism. Admission is not a throughput or correctness guarantee. Check the
- [support matrix](docs/support-matrix.md) and the [known limits](docs/known-limits.md).
-
-## Documentation
-
-[Architecture](docs/architecture.md) · [Configuration](docs/configuration.md) · [Calibration](docs/calibration.md) ·
-[Kernels](docs/kernels.md) · [Compatibility](docs/compatibility.md) · [Troubleshooting](docs/troubleshooting.md) ·
-[FAQ](docs/faq.md) · [Release notes](docs/release-notes/0.1.0.md) · [Contributing](CONTRIBUTING.md) ·
-[Security](SECURITY.md)
+- [Install](docs/install.md): pins, wheel indexes, Docker images and the ROCm closure.
+- Engine guides with the measured recipes: [SGLang](docs/quickstart-sglang.md), [vLLM](docs/quickstart-vllm.md),
+ [TokenSpeed, experimental](docs/quickstart-tokenspeed.md), [AMD MI355X](docs/quickstart-amd.md).
+- [Support matrix](docs/support-matrix.md) and [known limits](docs/known-limits.md).
+- [Calibration](docs/calibration.md), [configuration](docs/configuration.md) and
+ [troubleshooting](docs/troubleshooting.md).
+- [How it works](docs/architecture.md), [kernels](docs/kernels.md), [compatibility](docs/compatibility.md),
+ [new models](docs/new-model.md) and the [FAQ](docs/faq.md).
## Citation
diff --git a/docs/README.md b/docs/README.md
index 2b784f8..6ef7382 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -1,30 +1,29 @@
---
type: explanation
owner: eagle-harness
-last_verified: 2026-10-19
+last_verified: 2026-10-21
---
-# eagle-harness documentation
+# EAGLE-4 documentation
-Install a pinned engine and check generation with live costs on SGLang or calibrated costs and the shipped
-batch-dependent tree schedule on vLLM.
-TokenSpeed is experimental. Measure the setup you will deploy.
-Start with the measured model pairs rather than treating config admission as proof of support.
+To serve a model with EAGLE-4, install one engine and follow its guide. SGLang needs no preparation, vLLM
+reaches its measured speed after a one-time calibration, and TokenSpeed is experimental.
| Task | Guide |
|---|---|
-| Install wheels or run a release Docker image | [Install](install.md) |
-| Generate text on SGLang CUDA | [SGLang quickstart](quickstart-sglang.md) |
-| Generate text on AMD MI355X | [AMD quickstart](quickstart-amd.md) |
-| Generate text on experimental TokenSpeed | [TokenSpeed quickstart](quickstart-tokenspeed.md) |
-| Generate text on vLLM | [vLLM quickstart](quickstart-vllm.md) |
-| Measure and fit serving costs | [Calibration](calibration.md) |
-| Train, export and qualify a draft for another target | [New-model onboarding](new-model.md) |
-| Check measured models, TP sizes and request features | [Support matrix](support-matrix.md) |
+| Install an engine with pip, or pick a Docker image | [Install](install.md) |
+| Serve on SGLang | [SGLang guide](quickstart-sglang.md) |
+| Serve on vLLM | [vLLM guide](quickstart-vllm.md) |
+| Serve on TokenSpeed, experimental | [TokenSpeed guide](quickstart-tokenspeed.md) |
+| Serve on AMD MI355X | [AMD guide](quickstart-amd.md) |
+| See measured speed and correctness | [Support matrix](support-matrix.md) |
| Check unsupported or unmeasured behavior | [Known limits](known-limits.md) |
+| Measure and fit serving costs | [Calibration](calibration.md) |
| Look up config keys and defaults | [Configuration](configuration.md) |
+| Diagnose install, startup and generation errors | [Troubleshooting](troubleshooting.md) |
+| Train, export and qualify a draft for another target | [New-model onboarding](new-model.md) |
| Understand plugin hooks and decode rounds | [Architecture](architecture.md) |
| Check kernel implementations and fallbacks | [Kernels](kernels.md) |
| Check host versions and release numbering | [Compatibility](compatibility.md) |
-| Diagnose startup and generation errors | [Troubleshooting](troubleshooting.md) |
+| Convert an existing EAGLE-4 configuration | [Migration](migration.md) |
| Find short answers | [FAQ](faq.md) |
diff --git a/docs/install.md b/docs/install.md
index ad6be74..0435913 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -1,93 +1,96 @@
---
type: how-to
owner: eagle-harness
-last_verified: 2026-10-20
+last_verified: 2026-10-21
---
-# Install EAGLE-4 0.1.1
+# Install EAGLE-4
-Use one patched engine per environment. Every path installs from public package indexes:
-`eagle-harness==0.1.1` from PyPI and the patched engine from the EAGLE-4 wheel index.
-Stock engine packages do not provide the required plugin hooks. Choose the pip or Docker path below,
-then follow that engine's quickstart.
+To install EAGLE-4, create a Python 3.12 environment and run the one `pip install` command for your engine.
+Each command installs `eagle-harness` from PyPI and the patched engine from the EAGLE-4 wheel index, because the
+stock engine packages do not have the hooks EAGLE-4 needs. Use one environment per engine. If you prefer
+containers, use a [published image](#docker-images) instead.
-## Release locations and availability
-
-[Release settings](release.env) is the single source for package pins, wheel indexes and container tags.
-The engine wheels belong to the GitHub release named by `EAGLE_RELEASE_ASSETS` (v0.1.0: the engines did not
-change in 0.1.1). Its PEP 503 indexes
-separate CUDA and ROCm wheels. The `eagle-harness` wheels and source distribution are published only to PyPI,
-not to the GitHub Release. Publishing the public release runs `publish.yml` with PyPI trusted publishing.
+## Requirements
-Fresh CUDA environments installed the engine wheels and served real requests, and the `eagle-harness`
-wheel and source-distribution forms passed installation checks. Those checks used a private index.
-Fresh installs of the SGLang and TokenSpeed CUDA wheels from the public index, with `eagle-harness` from PyPI,
-later served the measured results in the [support matrix](support-matrix.md). The published SGLang and vLLM
-images passed serving and correctness checks, and the published TokenSpeed image passed a GPU check in serve
-mode. The ROCm pip path below was clean-installed from the
-public indexes and inference-tested on an MI355X. No ROCm image is published, and its Docker recipe has not been
-container-qualified.
-Do not replace an unavailable artifact with the stock engine or an unpinned package.
+- Linux x86-64 and Python 3.12. On Ubuntu, `python3.12 -m venv` needs the `python3.12-venv` package.
+- NVIDIA GPUs with a CUDA 13 driver. The measured recipes use B300 GPUs.
+- For pip installs, a CUDA toolkit with `nvcc` and a C++ compiler for kernels compiled on the first start.
+ The driver alone is not enough. On Ubuntu 26.04 / glibc 2.43, use CUDA 13.2
+ ([toolkit setup](troubleshooting.md#install-problems)). Docker images include their build tools.
+- For AMD MI355X, see [SGLang ROCm on AMD](#sglang-rocm-on-amd), which uses Python 3.10.
-Clone the release checkout containing these docs and read its settings:
+## SGLang CUDA
```bash
-git clone https://github.com/eagle-hq/EAGLE-4
-. EAGLE-4/docs/release.env
+python3.12 -m venv .venv && source .venv/bin/activate
+pip install "eagle-harness[sglang]" "sglang==0.5.23.dev252+eagle.5" \
+ --extra-index-url https://eagle-hq.github.io/EAGLE-4/whl/cu130/ \
+ --extra-index-url https://download.pytorch.org/whl/cu130 \
+ --extra-index-url https://pypi.nvidia.com
```
-Run every command in this guide and in the quickstarts from the directory that contains the checkout,
-not from inside it. `python -m` puts the current directory first on the import path, so a server started
-inside the checkout imports the source tree's `eagle_harness/` instead of the installed wheel.
-`eagle-harness-check` cannot detect that, because a console script does not import from the current directory.
+Then follow the [SGLang guide](quickstart-sglang.md).
-## Requirements
+## vLLM CUDA
-- Linux x86-64. Use Python 3.12 for CUDA engines and Python 3.10 for the ROCm wheel.
-- NVIDIA wheels target CUDA 13.0 and require a compatible host driver. The serving recipes use B300 GPUs.
- Runtime compilation also needs a CUDA toolkit compatible with the host C/C++ headers.
- On the tested Ubuntu 26.04 / glibc 2.43 host, CUDA 13.0 `nvcc` fails with `rsqrt`/`rsqrtf`
- exception-specification errors. Use `export CUDA_HOME=/usr/local/cuda-13.2` for that host's JIT
- compilation, or use the release container's compatible toolchain. This does not change the wheel index.
-- SGLang ROCm requires Ubuntu 22.04, the ROCm 7.2 developer toolkit (`hipcc` and headers),
- a C++ compiler, Git, and an MI355X with a compatible host driver.
- Install the hash-locked ROCm closure below. AMD's PyTorch build for ROCm 7.2 and the
- engine wheel use the CPython 3.10 ABI. Python 3.12 is not a supported substitute.
-- Docker needs NVIDIA GPU access for CUDA, or `/dev/kfd` and `/dev/dri` access for ROCm.
-- Obtain readable target and compatible draft weights before serving. The quickstarts use
- Qwen3.8-27B BF16 and its EAGLE-4 draft. Check the [support matrix](support-matrix.md) for other pairs.
+```bash
+python3.12 -m venv .venv && source .venv/bin/activate
+pip install "eagle-harness[vllm]" "vllm==0.32.0rc1.dev283+eagle.4" \
+ --extra-index-url https://eagle-hq.github.io/EAGLE-4/whl/cu130/ \
+ --extra-index-url https://download.pytorch.org/whl/cu130
+```
-## SGLang CUDA
+Then follow the [vLLM guide](quickstart-vllm.md).
-Create a Python 3.12 environment and install the pinned CUDA engine:
+## TokenSpeed CUDA, experimental
```bash
-python3.12 -m venv .venv-sglang
-source .venv-sglang/bin/activate
-python -m pip install --upgrade pip
-python -m pip install --extra-index-url "$EAGLE_CUDA_INDEX" \
- --extra-index-url "$EAGLE_TORCH_CUDA_INDEX" --extra-index-url "$EAGLE_NVIDIA_INDEX" \
- "sglang==$EAGLE_SGLANG_VERSION" 'flashinfer-python>=0.7.2'
-python -m pip install "eagle-harness==$EAGLE_HARNESS_VERSION"
-eagle-harness-check
+python3.12 -m venv .venv && source .venv/bin/activate
+pip install "eagle-harness[tokenspeed]" "tokenspeed==0.1.1+eagle.5" \
+ "tokenspeed-scheduler==0.1.40+eagle.4" "tokenspeed-kernel==0.1.4+eagle.5" \
+ --extra-index-url https://eagle-hq.github.io/EAGLE-4/whl/cu130/ \
+ --extra-index-url https://download.pytorch.org/whl/cu130
```
-The NVIDIA index supplies the pinned `cuda-tile==1.6.0rc5` wheel. It avoids relying on a source build
-from PyPI. The private install check also succeeded with that exact dependency built from its PyPI source.
+TokenSpeed runs on NVIDIA GPUs only. Then follow the [TokenSpeed guide](quickstart-tokenspeed.md).
-Or open the published image and check the hooks. The 0.1.0 images carry `eagle-harness` 0.1.0; upgrade it
-inside the container:
+## What the indexes and pins are for
-```bash
-docker run --rm --gpus all --ipc host -it \
- -p 127.0.0.1:30000:30000 -v "$PWD/models:/models:ro" \
- --entrypoint bash "$EAGLE_SGLANG_IMAGE"
-python -m pip install -U eagle-harness==0.1.1
-eagle-harness-check
-```
+| Index or pin | Why |
+| --- | --- |
+| `https://eagle-hq.github.io/EAGLE-4/whl/cu130/` | The patched SGLang, vLLM and TokenSpeed wheels. They carry a `+eagle.N` local version, so pip picks them only when the command pins that exact version. |
+| `https://download.pytorch.org/whl/cu130` | PyTorch built for CUDA 13.0, which the engine wheels need. |
+| `https://pypi.nvidia.com` | The `cuda-tile==1.6.0rc5` wheel that the SGLang engine pins. Without it pip builds that package from source. |
+| `tokenspeed-scheduler`, `tokenspeed-kernel` | TokenSpeed's native packages, built from the same source as the engine. The stock wheels do not work with it. |
+
+The engine wheels are assets of the [v0.1.0 GitHub release](https://github.com/eagle-hq/EAGLE-4/releases/tag/v0.1.0),
+and the index pages link to them with SHA-256 hashes. Their sources are [eagle-hq/sglang](https://github.com/eagle-hq/sglang),
+[eagle-hq/vllm](https://github.com/eagle-hq/vllm) and [eagle-hq/tokenspeed](https://github.com/eagle-hq/tokenspeed).
+`eagle-harness` itself is published only on PyPI.
+[docs/release.env](release.env) holds the same pins and URLs for scripts and image builds.
+[docker/sglang/requirements.lock](../docker/sglang/requirements.lock) pins the SGLang CUDA runtime dependencies by
+wheel hash for the SGLang image build.
-Inside the container, use `/models` paths and `--host 0.0.0.0` in the
-[SGLang CUDA quickstart](quickstart-sglang.md). The host port stays bound to loopback.
+## Docker images
+
+Each image has one patched engine and `eagle-harness` 0.1.0 installed. The engine guides show the `docker run`
+command for each one.
+
+| Engine | Image |
+| --- | --- |
+| SGLang CUDA | `ghcr.io/eagle-hq/eagle-4-sglang:0.1.0` |
+| vLLM CUDA | `ghcr.io/eagle-hq/eagle-4-vllm:0.1.0` |
+| TokenSpeed CUDA, experimental | `ghcr.io/eagle-hq/eagle-4-tokenspeed:0.1.0` |
+| SGLang ROCm | not published, [build it](#build-the-rocm-image) |
+
+To run a newer `eagle-harness` in an image, run `pip install -U eagle-harness` inside the container.
+
+## Check the installation
+
+`eagle-harness-check` exits zero when the installed engine has every hook EAGLE-4 needs. It does not start a
+server, so a served request is still the real test. The [troubleshooting guide](troubleshooting.md) explains
+its output and the startup errors.
## SGLang ROCm on AMD
@@ -104,10 +107,15 @@ The wheels come from public indexes:
| Triton `3.7.0+amd.rocm7.2.0.git89002410` | AMD's Triton index, `EAGLE_AMD_TRITON_INDEX` |
| Torchvision `0.24.0+rocm7.2.0.gitb919bd0c` | AMD's ROCm 7.2 wheel directory, `EAGLE_AMD_ROCM_WHEELS` |
-Do not use `--no-deps`, a source-overlay `PYTHONPATH`, or `--system-site-packages`.
-On the ROCm host, from the directory that contains the checkout:
+The host needs Ubuntu 22.04, the ROCm 7.2 developer toolkit (`hipcc` and headers), a C++ compiler, Git, and an
+MI355X with a compatible host driver. AMD's PyTorch build for ROCm 7.2 and the engine wheel use the CPython 3.10
+ABI, so Python 3.12 is not a substitute. Do not use `--no-deps`, a source-overlay `PYTHONPATH`, or
+`--system-site-packages`. Clone the repository for the lock file and the settings, then install from the
+directory that contains the checkout:
```bash
+git clone https://github.com/eagle-hq/EAGLE-4
+. EAGLE-4/docs/release.env
unset PYTHONPATH
export PATH="/opt/rocm/bin:$PATH"
export LD_LIBRARY_PATH="/opt/rocm/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
@@ -127,7 +135,7 @@ the HIP `sglang-kernel==0.4.7` for `gfx950`. Three of these are not on their ven
tested form, so the EAGLE-4 index serves them. Each is selected by its hash, not just its version:
- Torch, build tag `1eagle`, is AMD's ROCm 7.2 wheel with one dependency corrected. AMD's metadata pins
- Triton 3.5.1, while AITER needs Triton 3.6 or newer. The rebuilt metadata pins the Triton build used
+ Triton 3.5.1 and AITER uses Triton 3.6 or newer, so the rebuilt metadata pins the Triton build used
by the upstream SGLang MI35x image instead. Only the metadata, `WHEEL` and `RECORD` files differ from
AMD's wheel. Every runtime file and its license files are unchanged.
- `sglang-kernel`, build tag `1eagle`, is the HIP build of SGLang's kernel package for `gfx950`.
@@ -137,13 +145,14 @@ tested form, so the EAGLE-4 index serves them. Each is selected by its hash, not
upstream SGLang MI35x image uses. AMD publishes no wheel of that build.
[Third-party notices](../THIRD_PARTY_NOTICES.md#redistributed-rocm-wheels) lists their licenses and changes.
+Then follow the [AMD guide](quickstart-amd.md). Do not install CUDA FlashInfer packages into this environment.
### Build the ROCm image
The Dockerfile uses the immutable `lmsysorg/sglang:v0.5.20-rocm720-mi35x` base and installs
this same closure in an isolated Python 3.10 environment. It does not reuse the base image's
-editable SGLang installation. Container construction and serving must be qualified separately
-before a release image is advertised as tested.
+editable SGLang installation. The ROCm image is being container-qualified and is not published yet
+([known limits](known-limits.md#distribution)).
Download the locked wheels with the Python 3.10 environment above, then build with Docker BuildKit:
@@ -161,106 +170,4 @@ python -m pip check
eagle-harness-check
```
-Follow the [AMD quickstart](quickstart-amd.md). Inside a container, use `/models` paths and
-`--host 0.0.0.0`. Do not install CUDA FlashInfer packages into this environment.
-
-## vLLM CUDA
-
-Use a separate environment:
-
-```bash
-python3.12 -m venv .venv-vllm
-source .venv-vllm/bin/activate
-python -m pip install --upgrade pip
-python -m pip install --extra-index-url "$EAGLE_CUDA_INDEX" \
- --extra-index-url "$EAGLE_TORCH_CUDA_INDEX" "vllm==$EAGLE_VLLM_VERSION"
-python -m pip install "eagle-harness==$EAGLE_HARNESS_VERSION"
-eagle-harness-check
-```
-
-Or use the published vLLM image, which passed serving and correctness checks. It carries `eagle-harness` 0.1.0;
-upgrade it inside the container. Mount a writable directory for the calibration output:
-
-```bash
-mkdir -p calibration
-docker run --rm --gpus all --ipc host -it \
- -p 127.0.0.1:8000:8000 -v "$PWD/models:/models:ro" \
- -v "$PWD/calibration:/calibration" --entrypoint bash "$EAGLE_VLLM_IMAGE"
-python -m pip install -U eagle-harness==0.1.1
-eagle-harness-check
-```
-
-Follow the [vLLM quickstart](quickstart-vllm.md). Inside the container, use `/models` paths,
-place calibration output under `/calibration`, and bind the server to `0.0.0.0`.
-
-## TokenSpeed CUDA, experimental
-
-TokenSpeed requires the matching engine, scheduler and kernel wheels pinned in `release.env`.
-Install all three from the CUDA release index. Stock scheduler and kernel wheels are not substitutes.
-Tree-round CUDA sources compile at first use even though the runtime wheel has a `py3-none-any` tag.
-The passing serve-mode check uses a host-compatible CUDA 13.2 JIT on Ubuntu 26.04 / glibc 2.43,
-with `CUDA_HOME=/usr/local/cuda-13.2`; the original CUDA 13.0 JIT fails on that host. This does not
-change the `cu130` wheel channel. The scheduler and kernel wheels contain native code:
-
-```bash
-python3.12 -m venv .venv-tokenspeed
-source .venv-tokenspeed/bin/activate
-python -m pip install --upgrade pip
-: "${EAGLE_TOKENSPEED_AUX_REQUIREMENTS:?release native-wheel pins are required}"
-# Intentional word splitting: this variable contains pinned package requirements.
-python -m pip install --extra-index-url "$EAGLE_CUDA_INDEX" \
- --extra-index-url "$EAGLE_TORCH_CUDA_INDEX" \
- "tokenspeed==$EAGLE_TOKENSPEED_VERSION" $EAGLE_TOKENSPEED_AUX_REQUIREMENTS
-python -m pip install "eagle-harness==$EAGLE_HARNESS_VERSION"
-eagle-harness-check
-```
-
-Or use the published TokenSpeed image, which passed a GPU check in serve mode. It carries `eagle-harness` 0.1.0;
-upgrade it inside the container:
-
-```bash
-docker run --rm --gpus all --ipc host -it \
- -p 127.0.0.1:30000:30000 -v "$PWD/models:/models:ro" \
- --entrypoint bash "$EAGLE_TOKENSPEED_IMAGE"
-python -m pip install -U eagle-harness==0.1.1
-eagle-harness-check
-```
-
-Follow the [TokenSpeed quickstart](quickstart-tokenspeed.md). Inside the container, use `/models` paths
-and bind to `0.0.0.0`. This path is NVIDIA-only, not a ROCm recipe.
-
-## Check the installation
-
-`eagle-harness-check` must exit zero and report the installed engine's hook checks. A failure means the
-engine does not provide the expected contract. Fix that before loading weights. This check does not
-prove GPU kernels work or generation is correct. The quickstart's generated response is a separate check.
-
-From the directory where you start servers, confirm that Python imports the installed package:
-
-```bash
-python -c 'import eagle_harness; print(eagle_harness.__file__)'
-```
-
-The printed path must be inside the environment's `site-packages`. A path inside the checkout means
-the server would run the source tree instead of the installed wheel.
-
-For the SGLang and vLLM calibration CLI, also run:
-
-```bash
-eagle-harness calibrate --help
-```
-
-Do not install multiple engines in one environment. Use only the trusted release indexes above.
-The [troubleshooting guide](troubleshooting.md) covers hook failures and startup errors.
-
-## Dockerfile build status
-
-The files under `docker/` describe release image construction, not proof of tested image builds.
-The private CUDA image checks used a prebuilt `eagle-harness` wheel instead of the sketches' source install,
-so they do not qualify those Dockerfiles verbatim. The corrected SGLang TP4 image passed hook, dependency,
-request and twenty-prompt correctness checks. The published vLLM image passed the same checks, the TokenSpeed
-image passed a GPU check in serve mode, and ROCm has no published image or container qualification.
-The ROCm recipe pins its upstream base by digest and takes the downloaded locked wheels as a named
-build context. The TokenSpeed sketch requires an explicit compatible `BASE_IMAGE`.
-Do not infer container qualification from a successful pip installation or syntax check.
-Final rebuilds and publication under the release tags are separate release steps.
+Inside a container, use `/models` paths and `--host 0.0.0.0`.
diff --git a/docs/known-limits.md b/docs/known-limits.md
index 4a992ee..297778f 100644
--- a/docs/known-limits.md
+++ b/docs/known-limits.md
@@ -7,44 +7,44 @@ last_verified: 2026-10-20
# Known limits
Use the [support matrix](support-matrix.md) for measured model, hardware and workload combinations.
-A loaded model, an admitted request cap or a compiled kernel does not establish serving correctness or speed.
+Correctness and speed claims come from those measured runs. A loaded model, an admitted request cap or a compiled
+kernel is a startup check.
## Models and hardware
- EAGLE-4 drafts need the trained candidate head and a supported draft structure.
- Dense and latent-MoE EAGLE-4 drafts and Mamba2 targets have adapters. Nemotron's SGLang comparison has
- unresolved first divergences under the fp32 gate. vLLM and experimental TokenSpeed do not have a complete
- release-stack Nemotron performance and correctness result. Do not treat architecture admission as qualification.
+ first divergences under the fp32 gate that are being resolved. Release-stack Nemotron performance and correctness
+ results for vLLM and experimental TokenSpeed are in progress. An architecture that loads is not yet qualified.
- Serving measurements cover NVIDIA B300 and SGLang on AMD MI355X. Compilation on another NVIDIA or AMD
- architecture is separate evidence. Multi-node deployment and pipeline parallelism are not qualified.
+ architecture is separate evidence. Multi-node deployment and pipeline parallelism are being qualified.
- Throughput depends on the model, draft, engine, precision, TP size, attention backend, context length and
workload. A result at one concurrency does not establish a result at another.
-## Performance and correctness gaps
-
-- SGLang TP4 single-request and 32K-context optimization remain open. TP8 concurrency-8 prefill
- overhead and intermittent stalls remain unresolved. Repeated passes in one process are not
- independent server starts.
-- SGLang automatic high-load policy and high-load fixed-chain improvements are not qualified release defaults.
- High-load and Nemotron first divergences remain open. A release-engine cap-512
- startup and request smoke passed, but it did not establish high-load acceleration or concurrent correctness.
-- SGLang native-OFF, including OFF forced by a cost profile, needs the engine's per-variant forward-mode
- decode graphs (the `forward_mode` field of the decode graph variant). The 0.1.0 SGLang CUDA engine lacks
- them, so EAGLE-4 refuses native-OFF on that engine.
+## Performance and correctness in progress
+
+- SGLang TP4 single-request and 32K-context optimization is in progress, as is TP8 concurrency-8 prefill
+ overhead and intermittent stalls. Repeated passes in one process are not independent server starts.
+- SGLang automatic high-load policy and high-load fixed-chain improvements are being qualified and stay off by
+ default, as does work on high-load and Nemotron first divergences. A release-engine cap-512 startup and
+ request smoke passes; high-load acceleration and concurrent correctness are being measured.
+- SGLang native-OFF, including OFF forced by a cost profile, uses the engine's per-variant forward-mode
+ decode graphs (the `forward_mode` field of the decode graph variant). They arrive with the next SGLang
+ engine release; on the 0.1.0 SGLang CUDA engine EAGLE-4 keeps native-OFF off.
- The SGLang temperature-1 sampled tree (draft proposals drawn in one fused launch and verified against their
- recorded probabilities) needs the engine's `sampling_step` verify hook. The 0.1.0 SGLang engines lack it, so
- on them temperature-1 requests keep the 0.1.0 behaviour: a greedy draft verified by the target's sampler.
+ recorded probabilities) uses the engine's `sampling_step` verify hook, which arrives with the next SGLang
+ engine release. On the 0.1.0 SGLang engines temperature-1 requests use a greedy draft verified by the
+ target's sampler.
On every engine, a batch uses that path when any request has another temperature, a grammar or a penalty.
- The shipped vLLM TP4 rounds schedule improves the measured concurrency-64 and concurrency-128 cases at a
running cap of 64, at temperature 0 and 1; see the
[support matrix](support-matrix.md#measured-high-load-tree-schedule). The schedule has no TP1/TP8 profile.
- A running cap of 128 and higher concurrency are not qualified. Native-OFF is disabled by default and does
- not reach native-decode throughput in the measured TP1 cases.
-- AMD measurements apply to the MI355X TP1 Qwen3.8 profile. At concurrency 64 on one MI355X, a short
- diagnostic on the release stack measured EAGLE-4 at 2703.6 tokens/s against 2975.4 tokens/s for no-spec
- decoding, so speculation is slower than no-spec there. HIP side-stream prelaunch stays disabled: its replay check now passes, but it
- measured no throughput gain on MI355X. Remaining target, draft and verify phase overheads are separate
- from the measured-profile throughput results.
+ Running caps of 128 and above are being qualified. Native-OFF is off by default while its TP1 throughput is
+ optimized to match native decoding.
+- AMD measurements apply to the MI355X TP1 Qwen3.8 profile. On one MI355X the speedup covers concurrency up
+ to 32; concurrency 64 is being optimized, so serve it without speculation for now. HIP side-stream prelaunch
+ stays off: its replay check passes and it measured no throughput gain on MI355X. Further target, draft and
+ verify phase optimization is in progress.
## Requests
@@ -52,29 +52,29 @@ A loaded model, an admitted request cap or a compiled kernel does not establish
benchmark results. A cap of 512 is an admission limit, not a claim of acceleration or correctness at that load.
- SGLang supports accepted-token and input logprobs with the required host hooks. Images are refused.
- vLLM tree verification needs an unquantized KV cache. The target's sampling restrictions still apply.
- Sampling smoke tests do not prove distribution equivalence on every workload. The vLLM sampled-distribution
- check passes only for temperature 1 with top-p 1 and no top-k; other temperatures, top-k/top-p and
- penalties are not qualified. vLLM refuses `min_p` and `logit_bias` with speculative decoding.
+ Sampling smoke tests check the path, not distribution equivalence on every workload. The vLLM
+ sampled-distribution check covers temperature 1 with top-p 1 and no top-k; other temperatures, top-k/top-p
+ and penalties are being qualified. vLLM refuses `min_p` and `logit_bias` with speculative decoding.
- Structured output and tool calls require the model's engine parser. Raw tool syntax in text is not a structured
- response. A vLLM tree batch containing a grammar constraint emits only the root token per verify round,
- including unconstrained requests in that batch. Check finish reasons and validate the completed output.
+ response. In a vLLM tree batch with a grammar constraint, every request in the batch advances one token per
+ verify round. Check finish reasons and validate the completed output.
See [compatibility](compatibility.md#structured-output-on-vllm).
## TokenSpeed (experimental)
TokenSpeed implements static and calibrated trees, chains and sampled trees. Its engine and adapter must be
installed together with the matching hooks. Performance and correctness claims apply only to reported
-configurations, not every implemented path. The sampled-output distribution is unqualified.
+configurations, not every implemented path. The sampled-output distribution is being qualified.
Prompt logprobs, per-request sampled-tree overrides, live cost measurement and native no-speculation rounds
-are not supported. A zero-draft speculative round is not a native no-speculation round.
+are not available on TokenSpeed yet. A zero-draft speculative round is not a native no-speculation round.
Tree and chain cost data must match the served shape, TP size, target, draft and recurrent-state dtype.
-TP4 single-request optimization remains open. Release-stack TP1/TP8, long contexts, four-GPU chains,
-retraction under serving load and original-row output-logprob checks are incomplete. The reported sampled
-cap-64 path is refused. A CUDA static-mode install smoke generated text but retained one fp32-gate failure.
-The serve-mode install smoke passed its checked greedy prompts only with the host-compatible CUDA 13.2
-JIT. The original CUDA 13.0 JIT fails on Ubuntu 26.04 / glibc 2.43. Static mode is not qualified.
-These checks do not qualify arbitrary CUDA-toolkit and host-compiler combinations.
+TP4 single-request optimization is in progress, as are release-stack TP1/TP8, long contexts, four-GPU chains,
+retraction under serving load and original-row output-logprob checks. The reported sampled cap-64 path is
+refused. The CUDA static-mode install smoke generates text with one first divergence above the fp32 gate, so
+static mode is a setup check, not a qualified path. The serve-mode install smoke passes its greedy prompts with
+the CUDA 13.2 JIT; on Ubuntu 26.04 / glibc 2.43 set `CUDA_HOME` to CUDA 13.2
+([troubleshooting](troubleshooting.md#install-problems)). These checks cover the toolkits named here.
## Calibration
@@ -83,16 +83,27 @@ the model, draft, TP size, precision, attention backend or engine. Keyed tables
key does not encode every setting. A cache or table is not evidence of an optimal policy for every workload.
The vLLM public sweep measures batch-one tree forwards, not complete high-concurrency serving rounds.
-The SGLang served-chain driver measures a separate path from dynamic-tree calibration. The pinned SGLang
-CUDA engine lacks its required per-variant native-decode graph contract, so the CLI path is unavailable
-on that engine. Live-cost tree serving does not require it. Built-in smoke prompts
-check the command, not representative acceptance. A failed run retains partial output for diagnosis and does
-not write a successful `serve.sh`. See [calibration](calibration.md).
+The SGLang served-chain driver measures a separate path from dynamic-tree calibration. Its CLI uses the
+per-variant native-decode graphs that arrive with the next SGLang engine release; live-cost tree serving runs on
+the current engine. Built-in smoke prompts check the command, not representative acceptance. An interrupted
+run keeps its partial output for diagnosis and writes `serve.sh` only on success. See
+[calibration](calibration.md).
## Distribution
The [install guide](install.md) describes the release path: `eagle-harness` packages on PyPI,
-patched engine wheels on the GitHub Pages indexes, and images on GHCR. These destinations do not establish
-publication or image qualification. Private-index CUDA installation and request checks do not test the public
-URLs. AMD has a qualified private offline pip closure, but its fresh-install commands remain deferred
-to the AMD guide update, and its Docker image is unqualified. Model and draft weights require separate access.
+patched engine wheels on the GitHub Pages indexes, and images on GHCR. Model and draft weights come from
+Hugging Face under their owners' licenses.
+
+- The published SGLang and vLLM images passed serving and correctness checks, and the published TokenSpeed image
+ passed a GPU check in serve mode ([support matrix](support-matrix.md#artifact-installation-checks)). They carry
+ `eagle-harness` 0.1.0.
+- The files under `docker/` show how the release images are built. The checked CUDA images carry a prebuilt
+ `eagle-harness` wheel, so the published images are the checked path. The TokenSpeed Dockerfile takes an
+ explicit compatible `BASE_IMAGE`.
+- On AMD, install with pip: the ROCm closure is clean-installed from the public indexes and inference-tested on
+ an MI355X. The ROCm Dockerfile is being container-qualified, and no ROCm image is published yet.
+- TokenSpeed serve mode uses cost tables measured for TokenSpeed on the served setup. EAGLE-4 does not ship them
+ or a command that measures them yet, so the [TokenSpeed guide](quickstart-tokenspeed.md) starts in static
+ mode as a setup check.
+- Use the pinned artifacts: stock engines and unpinned packages do not carry the EAGLE-4 hooks.
diff --git a/docs/migration.md b/docs/migration.md
index 04602cf..3f92c85 100644
--- a/docs/migration.md
+++ b/docs/migration.md
@@ -12,21 +12,22 @@ triples.
or diagnostic output. The defaults below describe the legacy environment-variable configuration, not an
environment parser.
-## Removed unused fields
+## Streamlined draft fields
-Remove `draft_model.moe_top_k` and `draft_model.packed_candidate_gather` from old JSON.
-Neither field affected execution; they are now rejected as unknown keys. Expert counts come
-from the checkpoint. The unused internal `CostModel.ragged_recent_verify_nums` state is also removed.
+Remove `draft_model.moe_top_k` and `draft_model.packed_candidate_gather` from old JSON; the loader rejects them
+as unknown keys. Expert counts come from the checkpoint. The internal `CostModel.ragged_recent_verify_nums`
+state is also removed.
## Removed in 0.1.1
-No engine adapter turned these options on, so they changed nothing. A config that still sets one fails at load
-with `<option>: removed in eagle-harness 0.1.1`. Delete the option; use `"mode": "serve"` instead of `"eval"`.
+Streamlined configuration: these options are removed, and serving behaves the same without them. A config that
+still sets one stops at load with `<option>: removed in eagle-harness 0.1.1`. Delete the option; use
+`"mode": "serve"` in place of `"eval"`.
- `"mode": "eval"`
- `ragged` (`enabled`, `force_uniform`)
- `cost_model.ragged_draft_relax`, `cost_model.ragged_verify_threshold`
-- `draft_model.moe_gate_acc` (it applied only to SwiGLU-MoE drafts, which every engine refuses)
+- `draft_model.moe_gate_acc` (SwiGLU-MoE drafts)
## Environment variables
@@ -56,52 +57,52 @@ with `<option>: removed in eagle-harness 0.1.1`. Delete the option; use `"mode":
| `SGLANG_EAGLE_DRAFT_EMBED_FULL` | `draft_model.embed_full` | `1` | Set the field |
| `SGLANG_EAGLE4_REPLICATED_DRAFT_EMBED` | `draft_model.embed_full` | `1 (deprecated alias)` | Set the field |
| `SGLANG_DISABLE_EAGLE_DRAFT_EMBED_TRITON` | `draft_model.embed_triton` | `0` | Invert the boolean |
-| `SGLANG_DISABLE_EAGLE_HARNESS_VERIFY_ARGMAX` | — | `0` | Dropped |
-| `SGLANG_DISABLE_EAGLE_HARNESS_SAMPLED_TREE` | — | `0` | Dropped |
-| `SGLANG_DISABLE_FLASHINFER_VERIFY_EMPTY_OUT` | — | `0` | Dropped |
-| `SGLANG_DISABLE_FLASHINFER_VERIFY_TAIL_MASK` | — | `0` | Dropped |
-| `SGLANG_DISABLE_EAGLE_HARNESS_FUSED_VERIFY_PREP` | — | `0` | Dropped |
-| `SGLANG_DISABLE_EAGLE_HARNESS_VERIFY_REPLAY_TRIMS` | — | `0` | Dropped |
-| `SGLANG_DISABLE_EAGLE_HARNESS_FOLD_COMMIT_STEPS` | — | `0` | Dropped |
-| `SGLANG_EAGLE_HARNESS_DRAFT_GRAPH_OPT` | — | `1` | Dropped |
-| `SGLANG_EAGLE_HARNESS_DRAFT_STRIDED_Q` | — | `1` | Dropped |
-| `SGLANG_EAGLE_HARNESS_DRAFT_ATTN_OUT` | — | `1` | Dropped |
-| `SGLANG_EAGLE_HARNESS_DRAFT_NORM_HALVES` | — | `1` | Dropped |
-| `SGLANG_EAGLE_HARNESS_DRAFT_GATHER_DOT_BK1` | — | `1` | Dropped |
-| `SGLANG_EAGLE_HARNESS_EXTEND_LAST_ROW_GATHER` | — | `1` | Dropped |
-| `SGLANG_EAGLE_HARNESS_EXTEND_FC_NORM_THIRDS` | — | `1` | Dropped |
-| `SGLANG_EAGLE_HARNESS_EXTEND_FC_STRIDED_INPUT` | — | `1` | Dropped |
-| `SGLANG_EAGLE_HARNESS_EXTEND_STRIDED_Q` | — | `1` | Dropped |
-| `SGLANG_EAGLE_HARNESS_FUSED_STEP_UPDATE` | — | `1` | Dropped |
-| `SGLANG_EAGLE_HARNESS_FUSED_STEP_ZERO_COPY` | — | `1` | Dropped |
-| `SGLANG_EXACT_SOFTMAX` | — | `1` | Dropped |
-| `SGLANG_EAGLE_HARNESS_SIDE_STREAM_VERIFY_SCORES` | — | `1` | Dropped |
-| `SGLANG_EAGLE_HARNESS_REUSE_EXTEND_DRAFT_TOPK` | — | `1` | Dropped |
-| `SGLANG_EAGLE_HARNESS_CHECK_EXTEND_DRAFT_TOPK` | — | `0` | Dropped |
-| `SGLANG_EAGLE_HARNESS_FUSED_POST_VERIFY` | — | `1` | Dropped |
-| `SGLANG_DISABLE_EAGLE_HARNESS_DRAFT_PRELAUNCH` | — | `0` | Dropped |
-| `SGLANG_DISABLE_EAGLE_HARNESS_SIDE_STREAM_STATE_COMMIT` | — | `0` | Dropped |
-| `SGLANG_DISABLE_EAGLE_HARNESS_LEAN_DRAFT_INIT` | — | `0` | Dropped |
-| `SGLANG_DISABLE_EAGLE_HARNESS_LEAN_DRAFT_STEP` | — | `0` | Dropped |
-| `SGLANG_DISABLE_EAGLE_HARNESS_SKIP_NOOP_MOE_CONTEXTS` | — | `0` | Dropped |
-| `SGLANG_DISABLE_EAGLE_HARNESS_DRAFT_STEP_META` | — | `0` | Dropped |
-| `SGLANG_DISABLE_EAGLE_HARNESS_DRAFT_PREP_ONE_LAUNCH` | — | `0` | Dropped |
-| `SGLANG_EAGLE_HARNESS_VERIFY_PREALLOC` | — | `1` | Dropped |
-| `SGLANG_EAGLE_HARNESS_CACHED_VERIFY_BATCH` | — | `1` | Dropped |
-| `SGLANG_EAGLE_HARNESS_DEVICE_ACCEPT` | — | `1` | Dropped |
-| `SGLANG_EAGLE_HARNESS_DEVICE_UPDATE_TREE` | — | `1` | Dropped |
-| `SGLANG_OPT_GDN_TREE_GLUON` | — | `1` | Dropped |
-| `SGLANG_OPT_GDN_TREE_GLUON_PDL` | — | `1` | Dropped |
-| `SGLANG_OPT_GDN_TREE_GLUON_V2` | — | `1` | Dropped |
-| `SGLANG_OPT_GDN_TREE_GLUON_V3` | — | `1` | Dropped |
-| `SGLANG_DISABLE_GDN_TREE_VERIFY_IN_PLACE` | — | `0` | Dropped |
-| `SGLANG_EAGLE_HARNESS_CHAIN_DEBUG` | — | `0` | Dropped |
-| `SGLANG_EAGLE_HARNESS_CHAIN_HIST` | — | `unset` | Dropped |
-| `SGLANG_EAGLE4_SWA_AUDIT` | — | `unset` | Dropped |
-| `SGLANG_EAGLE4_SWA_AUDIT_DIR` | — | `unset` | Dropped |
+| `SGLANG_DISABLE_EAGLE_HARNESS_VERIFY_ARGMAX` | — | `0` | Removed |
+| `SGLANG_DISABLE_EAGLE_HARNESS_SAMPLED_TREE` | — | `0` | Removed |
+| `SGLANG_DISABLE_FLASHINFER_VERIFY_EMPTY_OUT` | — | `0` | Removed |
+| `SGLANG_DISABLE_FLASHINFER_VERIFY_TAIL_MASK` | — | `0` | Removed |
+| `SGLANG_DISABLE_EAGLE_HARNESS_FUSED_VERIFY_PREP` | — | `0` | Removed |
+| `SGLANG_DISABLE_EAGLE_HARNESS_VERIFY_REPLAY_TRIMS` | — | `0` | Removed |
+| `SGLANG_DISABLE_EAGLE_HARNESS_FOLD_COMMIT_STEPS` | — | `0` | Removed |
+| `SGLANG_EAGLE_HARNESS_DRAFT_GRAPH_OPT` | — | `1` | Removed |
+| `SGLANG_EAGLE_HARNESS_DRAFT_STRIDED_Q` | — | `1` | Removed |
+| `SGLANG_EAGLE_HARNESS_DRAFT_ATTN_OUT` | — | `1` | Removed |
+| `SGLANG_EAGLE_HARNESS_DRAFT_NORM_HALVES` | — | `1` | Removed |
+| `SGLANG_EAGLE_HARNESS_DRAFT_GATHER_DOT_BK1` | — | `1` | Removed |
+| `SGLANG_EAGLE_HARNESS_EXTEND_LAST_ROW_GATHER` | — | `1` | Removed |
+| `SGLANG_EAGLE_HARNESS_EXTEND_FC_NORM_THIRDS` | — | `1` | Removed |
+| `SGLANG_EAGLE_HARNESS_EXTEND_FC_STRIDED_INPUT` | — | `1` | Removed |
+| `SGLANG_EAGLE_HARNESS_EXTEND_STRIDED_Q` | — | `1` | Removed |
+| `SGLANG_EAGLE_HARNESS_FUSED_STEP_UPDATE` | — | `1` | Removed |
+| `SGLANG_EAGLE_HARNESS_FUSED_STEP_ZERO_COPY` | — | `1` | Removed |
+| `SGLANG_EXACT_SOFTMAX` | — | `1` | Removed |
+| `SGLANG_EAGLE_HARNESS_SIDE_STREAM_VERIFY_SCORES` | — | `1` | Removed |
+| `SGLANG_EAGLE_HARNESS_REUSE_EXTEND_DRAFT_TOPK` | — | `1` | Removed |
+| `SGLANG_EAGLE_HARNESS_CHECK_EXTEND_DRAFT_TOPK` | — | `0` | Removed |
+| `SGLANG_EAGLE_HARNESS_FUSED_POST_VERIFY` | — | `1` | Removed |
+| `SGLANG_DISABLE_EAGLE_HARNESS_DRAFT_PRELAUNCH` | — | `0` | Removed |
+| `SGLANG_DISABLE_EAGLE_HARNESS_SIDE_STREAM_STATE_COMMIT` | — | `0` | Removed |
+| `SGLANG_DISABLE_EAGLE_HARNESS_LEAN_DRAFT_INIT` | — | `0` | Removed |
+| `SGLANG_DISABLE_EAGLE_HARNESS_LEAN_DRAFT_STEP` | — | `0` | Removed |
+| `SGLANG_DISABLE_EAGLE_HARNESS_SKIP_NOOP_MOE_CONTEXTS` | — | `0` | Removed |
+| `SGLANG_DISABLE_EAGLE_HARNESS_DRAFT_STEP_META` | — | `0` | Removed |
+| `SGLANG_DISABLE_EAGLE_HARNESS_DRAFT_PREP_ONE_LAUNCH` | — | `0` | Removed |
+| `SGLANG_EAGLE_HARNESS_VERIFY_PREALLOC` | — | `1` | Removed |
+| `SGLANG_EAGLE_HARNESS_CACHED_VERIFY_BATCH` | — | `1` | Removed |
+| `SGLANG_EAGLE_HARNESS_DEVICE_ACCEPT` | — | `1` | Removed |
+| `SGLANG_EAGLE_HARNESS_DEVICE_UPDATE_TREE` | — | `1` | Removed |
+| `SGLANG_OPT_GDN_TREE_GLUON` | — | `1` | Removed |
+| `SGLANG_OPT_GDN_TREE_GLUON_PDL` | — | `1` | Removed |
+| `SGLANG_OPT_GDN_TREE_GLUON_V2` | — | `1` | Removed |
+| `SGLANG_OPT_GDN_TREE_GLUON_V3` | — | `1` | Removed |
+| `SGLANG_DISABLE_GDN_TREE_VERIFY_IN_PLACE` | — | `0` | Removed |
+| `SGLANG_EAGLE_HARNESS_CHAIN_DEBUG` | — | `0` | Removed |
+| `SGLANG_EAGLE_HARNESS_CHAIN_HIST` | — | `unset` | Removed |
+| `SGLANG_EAGLE4_SWA_AUDIT` | — | `unset` | Removed |
+| `SGLANG_EAGLE4_SWA_AUDIT_DIR` | — | `unset` | Removed |
| `DRAFT_TREE_PATH` | `data_path` | `--speculative-data-path` | Set the field |
| `COMPUTE_INTENSITY` | `intensity` | `--speculative-compute-intensity` | Set the field |
-| `DEBUG_STEP` | — | `0 (never read)` | Dropped |
+| `DEBUG_STEP` | — | `0` | Removed |
## Command-line arguments
diff --git a/docs/quickstart-sglang.md b/docs/quickstart-sglang.md
index 5e7f5a8..e97df72 100644
--- a/docs/quickstart-sglang.md
+++ b/docs/quickstart-sglang.md
@@ -1,74 +1,85 @@
---
type: tutorial
owner: eagle-harness
-last_verified: 2026-10-20
+last_verified: 2026-10-21
---
-# SGLang CUDA quickstart
+# SGLang guide
-Serve Qwen3.8-27B BF16 with its EAGLE-4 draft on four B300 GPUs. SGLang measures the draft and
-verification costs during served rounds, so this recipe needs no offline calibration directory.
-For MI355X, use the separate [AMD quickstart](quickstart-amd.md).
+Serve Qwen3.8-27B with its EAGLE-4 draft on four NVIDIA GPUs. SGLang prices the draft tree from costs it
+measures on the served rounds, so nothing needs calibrating first. For MI355X, use the [AMD guide](quickstart-amd.md).
-## Before starting
+## Start a server
-Complete the [SGLang CUDA installation](install.md#sglang-cuda) and run `eagle-harness-check`.
-Set `MODEL` and `DRAFT` to readable local target and compatible draft directories.
-The release recipe below uses TP4 and a running-request cap of 32. See the
-[support matrix](support-matrix.md) before changing the model, hardware or tensor-parallel size.
-Only enable `--trust-remote-code` for model code you have reviewed.
+Install the engine with the [SGLang install command](install.md#sglang-cuda), then start the server. The model and
+the draft download from Hugging Face on the first start:
-## Start the server
+```bash
+sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 \
+ --speculative-algorithm EAGLE_HARNESS \
+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4
+```
+
+Or run the published image, which has the engine and EAGLE-4 installed:
+
+```bash
+docker run --gpus all --ipc host -p 127.0.0.1:30000:30000 \
+ -e HF_HOME=/hf -v ~/.cache/huggingface:/hf \
+ ghcr.io/eagle-hq/eagle-4-sglang:0.1.0 \
+ sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 --host 0.0.0.0 \
+ --disable-overlap-schedule --speculative-algorithm EAGLE_HARNESS \
+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 \
+ --speculative-num-steps 12 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \
+ --speculative-algorithm-config '{}'
+```
+
+By default EAGLE-4 drafts a tree of up to 12 steps with 4 candidates per step, verifies up to 16 tokens, and
+prices every round from live costs. `--speculative-num-steps`, `--speculative-eagle-topk`,
+`--speculative-num-draft-tokens` and `--speculative-algorithm-config` override those
+([configuration](configuration.md)). These defaults need `eagle-harness` 0.1.1 or newer. The image carries 0.1.0,
+so its command spells them out and adds `--disable-overlap-schedule`. From 0.1.1, EAGLE-4 serves with SGLang's
+default overlap scheduler.
+
+## Send a request
+
+When the server logs `The server is fired up and ready to roll!`, send a request from another terminal:
+
+```bash
+curl http://localhost:30000/v1/chat/completions -H 'Content-Type: application/json' \
+ -d '{"model": "Qwen/Qwen3.8-27B", "messages": [{"role": "user", "content": "Explain how a hash table works."}], "max_tokens": 256}'
+```
+
+The response holds the generated message. The first requests compile kernels and seed the live costs, so send a
+few before you measure speed. Compare against the same engine without the `--speculative-*` flags.
+
+## The measured recipe
-Use the release tree shape and keep CUDA graph padding enabled:
+The SGLang rows of the [support matrix](support-matrix.md#sglang-on-nvidia-b300) were measured with this command.
+It adds the attention backends, graph settings, memory sizes and recurrent-state dtype of the benchmark:
```bash
-: "${MODEL:?set the local Qwen3.8-27B BF16 target directory}"
-: "${DRAFT:?set the local compatible EAGLE-4 draft directory}"
-export HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=4
+export TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=4
export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 SGLANG_TIMEOUT_KEEP_ALIVE=3600
-python -m sglang.launch_server --model-path "$MODEL" --tp-size 4 --dtype bfloat16 \
+sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 --dtype bfloat16 \
--trust-remote-code --attention-backend flashinfer --page-size 1 \
--max-running-requests 32 --cuda-graph-max-bs-decode 32 \
--disable-overlap-schedule --cuda-graph-backend-decode full \
--mem-fraction-static 0.75 --context-length 196608 --max-total-tokens 196608 \
--chunked-prefill-size 16384 --mamba-ssm-dtype bfloat16 --disable-radix-cache \
--reasoning-parser qwen3 --random-seed 721 --skip-server-warmup \
- --host 127.0.0.1 --port 30000 \
- --speculative-algorithm EAGLE_HARNESS --speculative-draft-model-path "$DRAFT" \
+ --speculative-algorithm EAGLE_HARNESS \
+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 \
--speculative-draft-model-quantization unquant --speculative-draft-attention-backend flashinfer \
--speculative-attention-mode prefill --speculative-num-steps 12 \
--speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \
--speculative-algorithm-config '{"mode":"serve"}'
```
-Do not add `--disable-cuda-graph-padding` to this CUDA recipe. The release benchmark uses a separate
-client warmup, hence `--skip-server-warmup`. With this exact command, the first generation below performs
-cold compilation and cost seeding. Remove that flag if you want the engine's startup warmup instead,
-but do not call the modified startup sequence an identical benchmark reproduction.
-
-The shape flags bound the tree. Initial served rounds price captured verification widths and later
-rounds refine their costs. The cache is keyed by GPU, TP size, model configurations, engine and dtype
-under `~/.cache/eagle-harness/costs`. No private cost-table path is required.
-
-## Generate text
-
-In another terminal, wait until the model listing succeeds, then send a greedy request:
-
-```bash
-curl --fail http://127.0.0.1:30000/v1/models
-curl --fail http://127.0.0.1:30000/generate \
- -H 'Content-Type: application/json' \
- -d '{"text":"Explain how a hash table works.","sampling_params":{"temperature":0,"max_new_tokens":128}}'
-```
-
-The response must contain generated text and token metadata. Repeat the generation request after
-compilation before measuring latency. A successful model listing alone does not check decoding.
-For Docker, bind the server to `0.0.0.0` inside the container and keep the published host port on loopback.
-
-## Next step
+The benchmark warmed the server from its client, hence `--skip-server-warmup`: the first request after this start
+compiles kernels and seeds the costs. Keep CUDA graph padding enabled, and do not add
+`--disable-cuda-graph-padding`. `--max-running-requests 32` is the tree default; set a larger cap, up to 512, for
+larger batches.
-Use representative prompts and compare against the same engine without speculation before deployment.
-The [calibration guide](calibration.md) explains live costs and offline chain calibration.
-The [known limits](known-limits.md) distinguish greedy checks, sampled requests, logprobs and multimodal support.
-Stop the server with Ctrl-C before starting another server or calibration on the same GPUs.
+Live costs are cached per GPU, TP size, model pair, engine and dtype under `~/.cache/eagle-harness/costs`.
+The [calibration guide](calibration.md) explains live costs and offline chain calibration, and the
+[known limits](known-limits.md) cover sampled requests, logprobs and multimodal inputs.
diff --git a/docs/quickstart-tokenspeed.md b/docs/quickstart-tokenspeed.md
index 9da76bc..7e9a794 100644
--- a/docs/quickstart-tokenspeed.md
+++ b/docs/quickstart-tokenspeed.md
@@ -1,85 +1,66 @@
---
type: tutorial
owner: eagle-harness
-last_verified: 2026-10-20
+last_verified: 2026-10-21
---
-# TokenSpeed quickstart, experimental
+# TokenSpeed guide, experimental
-Run the experimental TokenSpeed adapter on NVIDIA CUDA hardware. This is not a ROCm path.
-The release serving recipe uses Qwen3.8-27B BF16 and its EAGLE-4 draft on four B300 GPUs.
+Serve Qwen3.8-27B with its EAGLE-4 draft on four NVIDIA GPUs with TokenSpeed. TokenSpeed support is experimental
+and runs on NVIDIA GPUs only. Its measured cost-model mode needs cost tables that EAGLE-4 does not ship yet, so the
+first server below uses a static tree to check that generation works.
-## Before starting
+## Start a server
-Complete the [TokenSpeed installation](install.md#tokenspeed-cuda-experimental), including its scheduler
-and kernel wheels, and run `eagle-harness-check`. Tree-round sources need `nvcc` at first use.
-On the tested Ubuntu 26.04 / glibc 2.43 host, the CUDA 13.0 JIT fails. The passing serve-mode
-check uses a host-compatible CUDA 13.2 JIT with `CUDA_HOME=/usr/local/cuda-13.2`. Set that path
-to the installed compatible toolkit before launching. The engine wheels still come from `cu130`.
-Set `MODEL` and `DRAFT` to readable local directories.
-The measured cost-model recipe also requires matching calibrated target and draft tables under
-`COST_ROOT/cost/`. EAGLE-4 does not include a TokenSpeed calibration CLI or automatically select
-an installed TokenSpeed profile. Do not substitute vLLM or ROCm tables.
-
-Use `mode=serve` with matching costs for the checked path. Its twenty-prompt install check passed
-with 4 exact and 16 fp32-tolerance results. Static mode is not a qualified substitute when cost files
-are unavailable: its check has one fp32 gap of 0.178637, above the 0.125 tolerance.
-These are first-divergence checks, not bitwise output equality or sampled-distribution qualification.
-
-## Start the server
-
-With matching costs, construct the configuration using JSON encoding:
+Install the engine with the [TokenSpeed install command](install.md#tokenspeed-cuda-experimental), then start a
+server with a static tree:
```bash
-: "${COST_ROOT:?set the matching TokenSpeed TP4 calibrated cost directory}"
-export COST_ROOT
-STEPS=12
-test -d "$COST_ROOT/cost"
-CONFIG=$(python -c 'import json, os; print(json.dumps({"mode":"serve","data_path":os.environ["COST_ROOT"]}))')
+tokenspeed serve Qwen/Qwen3.8-27B --tensor-parallel-size 4 \
+ --attention-backend trtllm --max-model-len 65536 --sampling-backend greedy \
+ --speculative-algorithm EAGLE_HARNESS \
+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 \
+ --speculative-num-steps 5 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \
+ --speculative-algorithm-config '{"mode": "static"}'
```
-Use the release launch flags with the serve-mode configuration:
+Static mode drafts the full tree every round, without a cost model. Its twenty-prompt greedy check has one
+first divergence above the fp32 gate, so it checks setup rather than correctness
+([known limits](known-limits.md#tokenspeed-experimental)). `--sampling-backend greedy` is required, because
+draft trees are verified by the greedy and triton sampling backends.
+
+## Send a request
```bash
-: "${MODEL:?set the local Qwen3.8-27B BF16 target directory}"
-: "${DRAFT:?set the local compatible EAGLE-4 draft directory}"
-: "${CONFIG:?construct the serve-mode configuration above}"
-: "${STEPS:?set the serve-mode step bound above}"
-export HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=4
-python -m tokenspeed.cli serve "$MODEL" --tensor-parallel-size 4 \
- --attention-backend trtllm --max-model-len 65536 --max-num-seqs 4 \
- --disable-kvstore --seed 721 --gpu-memory-utilization 0.6 --disable-prefix-caching \
- --host 127.0.0.1 --port 30000 --reasoning-parser qwen3 \
- --speculative-draft-model-path "$DRAFT" --speculative-num-steps "$STEPS" \
- --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \
- --speculative-algorithm EAGLE_HARNESS --speculative-algorithm-config "$CONFIG" \
- --sampling-backend greedy
+curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
+ -d '{"model": "Qwen/Qwen3.8-27B", "messages": [{"role": "user", "content": "Explain how a hash table works."}], "max_tokens": 256}'
```
-The running-request cap is four in the single-client release recipe. The concurrent release recipe
-uses `--max-num-seqs 32` with the same TP4 and context settings. Neither setting promises a speedup
-for your workload. The published TokenSpeed image passed a GPU check in serve mode.
-Inside the image, bind the server to `0.0.0.0` and keep the published host port on loopback.
-If using cost files in Docker, mount their directory read-only and set `COST_ROOT` to that container path.
+The response holds the generated message. The first start compiles tree-round CUDA sources, which needs `nvcc`.
+On Ubuntu 26.04 / glibc 2.43, set `CUDA_HOME=/usr/local/cuda-13.2` first ([troubleshooting](troubleshooting.md#install-problems)).
-## Generate text
+## The measured recipe
-In another terminal, read the served model identifier and send a greedy chat request:
+The TokenSpeed rows of the [support matrix](support-matrix.md#tokenspeed-experimental) use serve mode, where the
+cost model sizes every round from cost tables measured for TokenSpeed on B300 at TP4. Set `COST_ROOT` to a
+directory whose `cost/` folder holds such tables. vLLM and ROCm tables are not substitutes.
```bash
-MODEL=$(curl --fail --silent http://127.0.0.1:30000/v1/models | python -c 'import json,sys; print(json.load(sys.stdin)["data"][0]["id"])')
-export MODEL
-python -c 'import json, os; print(json.dumps({"model":os.environ["MODEL"],"messages":[{"role":"user","content":"Explain how a hash table works."}],"temperature":0,"max_tokens":128}))' | \
- curl --fail http://127.0.0.1:30000/v1/chat/completions \
- -H 'Content-Type: application/json' --data-binary @-
+export COST_ROOT=/path/to/tokenspeed-tp4-costs
+tokenspeed serve Qwen/Qwen3.8-27B --tensor-parallel-size 4 \
+ --attention-backend trtllm --max-model-len 65536 --max-num-seqs 32 \
+ --disable-kvstore --seed 721 --gpu-memory-utilization 0.6 --disable-prefix-caching \
+ --reasoning-parser qwen3 --sampling-backend greedy \
+ --speculative-algorithm EAGLE_HARNESS \
+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 \
+ --speculative-num-steps 12 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \
+ --speculative-algorithm-config "{\"mode\": \"serve\", \"data_path\": \"$COST_ROOT\"}"
```
-The response must contain a generated choice. A reasoning model may place its output in the reasoning
-field rather than `message.content`. Repeat after warmup before timing.
-
-## Experimental limits
+The single-client recipe uses `--max-num-seqs 4` and the concurrent recipe `--max-num-seqs 32`. Neither setting
+promises a speedup for your workload. The serve-mode check passed twenty greedy prompts with 4 exact and 16
+fp32-tolerance results. These are first-divergence checks, not bitwise output equality or sampled-distribution
+qualification.
-Prompt logprobs, per-request sampled trees and native-OFF are not supported by this adapter.
-This greedy quickstart does not enable sampled decoding.
-See the [support matrix](support-matrix.md) for model-specific correctness and performance qualifications.
-Stop the server with Ctrl-C before starting another engine on the same GPUs.
+Prompt logprobs, per-request sampled trees, live cost measurement and native no-speculation rounds are not
+supported. The [known limits](known-limits.md#tokenspeed-experimental) list the rest.
diff --git a/docs/quickstart-vllm.md b/docs/quickstart-vllm.md
index d589350..26606b0 100644
--- a/docs/quickstart-vllm.md
+++ b/docs/quickstart-vllm.md
@@ -1,92 +1,83 @@
---
type: tutorial
owner: eagle-harness
-last_verified: 2026-10-20
+last_verified: 2026-10-21
---
-# vLLM quickstart
+# vLLM guide
-Serve Qwen3.8-27B BF16 with its EAGLE-4 draft on four B300 GPUs using the release tree configuration.
-Load the shipped rounds file into matching calibrated costs to enable its batch-dependent verification width.
-The rounds file is packaged, but the target and draft `cost/` tables must be measured separately.
+Serve Qwen3.8-27B with its EAGLE-4 draft on four NVIDIA GPUs with vLLM. The first server uses a fixed tree and
+needs no preparation. The measured recipe adds cost tables you calibrate once and a packaged schedule that
+narrows the tree at high load.
-## Before starting
+## Start a server
-Complete the [vLLM installation](install.md#vllm-cuda) and run `eagle-harness-check`.
-Set `MODEL` and `DRAFT` to readable local directories. This recipe is specific to the Qwen3.8-27B / EAGLE-4
-v3 pair at TP4. Do not reuse its rounds file for another GPU, tensor-parallel size or model pair.
-See the [support matrix](support-matrix.md) for other configurations.
+Install the engine with the [vLLM install command](install.md#vllm-cuda), then start the server. The model and the
+draft download from Hugging Face on the first start:
-## Prepare the costs and shipped rounds
+```bash
+VLLM_USE_V2_MODEL_RUNNER=1 vllm serve Qwen/Qwen3.8-27B --tensor-parallel-size 4 \
+ --attention-backend CUSTOM --linear-backend flashinfer_cutedsl \
+ --speculative-config '{"method": "eagle_harness", "model": "EAGLE-Corp/Qwen3.8-27B-EAGLE4", "method_config": {"mode": "static", "node_top_k": 4, "tree_depth": 14, "max_verify_width": 16}}'
+```
-If you already have matching calibrated TP4 costs, set `COST_ROOT` to that directory and skip calibration.
-Otherwise, with the GPUs idle, create them with the installed CLI:
+Or run the published image, which has the engine and EAGLE-4 installed and starts `vllm serve` itself:
```bash
-: "${MODEL:?set the local Qwen3.8-27B BF16 target directory}"
-: "${DRAFT:?set the local compatible EAGLE-4 draft directory}"
-export MODEL DRAFT
-export COST_ROOT="$PWD/calibration-vllm-tp4"
-eagle-harness calibrate --engine vllm --model "$MODEL" --draft "$DRAFT" --tp 4 \
- --engine-args '{"linear_backend":"flashinfer_cutedsl"}' --output "$COST_ROOT"
+docker run --gpus all --ipc host -p 127.0.0.1:8000:8000 \
+ -e HF_HOME=/hf -v ~/.cache/huggingface:/hf -e VLLM_USE_V2_MODEL_RUNNER=1 \
+ ghcr.io/eagle-hq/eagle-4-vllm:0.1.0 \
+ Qwen/Qwen3.8-27B --tensor-parallel-size 4 \
+ --attention-backend CUSTOM --linear-backend flashinfer_cutedsl \
+ --speculative-config '{"method": "eagle_harness", "model": "EAGLE-Corp/Qwen3.8-27B-EAGLE4", "method_config": {"mode": "static", "node_top_k": 4, "tree_depth": 14, "max_verify_width": 16}}'
```
-Calibration refuses an existing output directory. It measures batch-one costs, not full high-concurrency
-serving. New measurements are not the exact table bytes used by the release benchmark. In Docker, choose
-a fresh directory beneath the writable `/calibration` mount instead of `$PWD`.
+The tree drafts 14 steps with 4 candidates per step and verifies 16 tokens. Tree verification needs vLLM's Model
+Runner V2 (`VLLM_USE_V2_MODEL_RUNNER=1`) and the `CUSTOM` attention backend, and an unquantized KV cache.
-Copy the shipped measured rounds without replacing the calibrated `cost/` tables:
+## Send a request
+
+When the server logs `Application startup complete`, send a request from another terminal:
```bash
-: "${COST_ROOT:?set the matching calibrated TP4 cost directory}"
-export COST_ROOT
-TREE_ROUNDS=$(python -c 'from importlib.resources import files; print(files("eagle_harness.core").joinpath("profiles/vllm-qwen38-tp4/rounds/tree.json"))')
-test -d "$COST_ROOT/cost"
-mkdir -p "$COST_ROOT/rounds"
-cp "$TREE_ROUNDS" "$COST_ROOT/rounds/tree.json"
+curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
+ -d '{"model": "Qwen/Qwen3.8-27B", "messages": [{"role": "user", "content": "Explain how a hash table works."}], "max_tokens": 256}'
```
-This file keeps width 16 below batch size 20 and selects width 4 at larger batches. It does not enable
-native-OFF. The [profile notes](../eagle_harness/core/profiles/vllm-qwen38-tp4/README.md) describe its scope.
+The response holds the generated message. Send a few requests before you measure speed.
+
+## The measured recipe
-## Start the server
+The TP4 vLLM rows of the [support matrix](support-matrix.md#vllm-on-nvidia-b300) stop the tree early when another
+draft step costs more than it gains, from cost tables measured on your GPUs. At 20 or more running requests the
+packaged rounds file narrows the verified tokens from 16 to 4
+([high-load rows](support-matrix.md#measured-high-load-tree-schedule)). Both live in one directory.
-Build the speculative configuration with JSON encoding so paths with spaces or quotes remain valid:
+Stop any server on these GPUs, then calibrate once and add the rounds file:
```bash
-: "${MODEL:?set the target directory}"
-: "${DRAFT:?set the draft directory}"
-: "${COST_ROOT:?set the prepared cost directory}"
-export MODEL DRAFT COST_ROOT
-SPEC=$(python -c 'import json, os; print(json.dumps({"method":"eagle_harness","model":os.environ["DRAFT"],"method_config":{"mode":"static","node_top_k":4,"tree_depth":14,"max_verify_width":16,"data_path":os.environ["COST_ROOT"]}}))')
-export VLLM_USE_V2_MODEL_RUNNER=1 HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=4
-python -m vllm.entrypoints.cli.main serve "$MODEL" --tensor-parallel-size 4 \
- --max-model-len 32768 --gpu-memory-utilization 0.85 --max-num-seqs 64 \
- --seed 721 --host 127.0.0.1 --port 8000 --no-enable-prefix-caching \
- --linear-backend flashinfer_cutedsl --reasoning-parser qwen3 \
- --attention-backend CUSTOM --speculative-config "$SPEC"
+eagle-harness calibrate --engine vllm --model Qwen/Qwen3.8-27B --draft EAGLE-Corp/Qwen3.8-27B-EAGLE4 --tp 4 \
+ --engine-args '{"linear_backend":"flashinfer_cutedsl"}' --output calibration-vllm-tp4
+mkdir -p calibration-vllm-tp4/rounds
+python -c 'import shutil; from importlib.resources import files; shutil.copy(files("eagle_harness.core") / "profiles/vllm-qwen38-tp4/rounds/tree.json", "calibration-vllm-tp4/rounds/")'
```
-Model Runner V2 and `CUSTOM` attention are required for this tree. The static shape supplies the bounds,
-the calibrated costs supply depth stopping, and the rounds file selects width by batch size.
-Registration-time INFO messages can be dropped before vLLM configures logging, so an absent schedule-load
-line does not prove the schedule is missing. Without `rounds/tree.json`, the configured width stays fixed.
-Do not point `data_path` at the packaged
-rounds-only directory because it lacks the depth-stop cost tables.
-
-## Generate text
+Calibration refuses an existing output directory. It measures batch-one costs, not full high-concurrency serving,
+and new measurements are not the exact table bytes of the release benchmark. The rounds file is measured for this
+model pair on B300 at TP4 only ([profile notes](../eagle_harness/core/profiles/vllm-qwen38-tp4/README.md)).
-In another terminal, read the served model identifier and send a completion:
+Then serve with the measured settings, from the directory that holds `calibration-vllm-tp4`:
```bash
-MODEL=$(curl --fail --silent http://127.0.0.1:8000/v1/models | python -c 'import json,sys; print(json.load(sys.stdin)["data"][0]["id"])')
-export MODEL
-python -c 'import json, os; print(json.dumps({"model":os.environ["MODEL"],"prompt":"Explain how a hash table works.","temperature":0,"max_tokens":128}))' | \
- curl --fail http://127.0.0.1:8000/v1/completions \
- -H 'Content-Type: application/json' --data-binary @-
+export VLLM_USE_V2_MODEL_RUNNER=1 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=4
+vllm serve Qwen/Qwen3.8-27B --tensor-parallel-size 4 \
+ --max-model-len 32768 --gpu-memory-utilization 0.85 --max-num-seqs 64 \
+ --seed 721 --no-enable-prefix-caching \
+ --linear-backend flashinfer_cutedsl --reasoning-parser qwen3 --attention-backend CUSTOM \
+ --speculative-config '{"method": "eagle_harness", "model": "EAGLE-Corp/Qwen3.8-27B-EAGLE4", "method_config": {"mode": "static", "node_top_k": 4, "tree_depth": 14, "max_verify_width": 16, "data_path": "calibration-vllm-tp4"}}'
```
-The response must contain a generated choice. Repeat after warmup before timing. In Docker, bind the server
-to `0.0.0.0` inside the container while publishing the host port on loopback.
-Sampling and logprobs have separate checks in the support matrix. This plain-text example does not qualify
-multimodal requests or quantized KV caches. Stop the server with Ctrl-C before another calibration.
+With INFO logging the server prints `eagle_harness: tree verify width per batch size from <path>` when it loads the
+rounds file. Without `rounds/tree.json` the verify width stays 16 at every batch size. Do not point `data_path` at
+the packaged rounds-only directory, because it has no cost tables. Sampling, logprobs and structured output have
+their own limits in the [known limits](known-limits.md#requests).
diff --git a/docs/release.env b/docs/release.env
index 7677402..4690b1d 100644
--- a/docs/release.env
+++ b/docs/release.env
@@ -1,10 +1,10 @@
# Copyright 2026 Eagle AI Corp.
# SPDX-License-Identifier: LicenseRef-Eagle-4-Source-Available
# Source it from the directory that contains the checkout: . EAGLE-4/docs/release.env
-# Distribution locations for eagle-harness 0.1.1. Publication is a separate release gate.
-# The engines did not change in 0.1.1: their wheels stay on the v0.1.0 GitHub release (EAGLE_RELEASE_ASSETS), which
-# the wheel indexes serve, and the published images are the 0.1.0 images, which carry eagle-harness 0.1.0.
-export EAGLE_HARNESS_VERSION='0.1.1'
+# Distribution locations for eagle-harness 0.1.1. Publication is a separate release gate. x-release-please-version
+# Engine wheel assets and image tags have their own release versions.
+# Advance their pins only after the corresponding artifacts pass release qualification.
+export EAGLE_HARNESS_VERSION='0.1.1' # x-release-please-version
export EAGLE_IMAGE_VERSION='0.1.0'
export EAGLE_WHEEL_ROOT='https://eagle-hq.github.io/EAGLE-4/whl'
export EAGLE_CUDA_INDEX="$EAGLE_WHEEL_ROOT/cu130/"
@@ -19,12 +19,12 @@ export EAGLE_SGLANG_IMAGE="$EAGLE_IMAGE_ROOT-sglang:$EAGLE_IMAGE_VERSION"
export EAGLE_ROCM_IMAGE="$EAGLE_IMAGE_ROOT-sglang-rocm:$EAGLE_IMAGE_VERSION"
export EAGLE_VLLM_IMAGE="$EAGLE_IMAGE_ROOT-vllm:$EAGLE_IMAGE_VERSION"
export EAGLE_TOKENSPEED_IMAGE="$EAGLE_IMAGE_ROOT-tokenspeed:$EAGLE_IMAGE_VERSION"
-export EAGLE_SGLANG_VERSION='0.5.23.dev252+eagle.4'
+export EAGLE_SGLANG_VERSION='0.5.23.dev252+eagle.5'
export EAGLE_ROCM_VERSION='0.5.23.dev252+eagle.4.rocm'
export EAGLE_VLLM_VERSION='0.32.0rc1.dev283+eagle.4'
-export EAGLE_TOKENSPEED_VERSION='0.1.1+eagle.4'
+export EAGLE_TOKENSPEED_VERSION='0.1.1+eagle.5'
export EAGLE_TORCH_CUDA_INDEX='https://download.pytorch.org/whl/cu130'
# The pinned SGLang dependency cuda-tile==1.6.0rc5 has vendor wheels here.
export EAGLE_NVIDIA_INDEX='https://pypi.nvidia.com'
# These are coupled native packages from the same TokenSpeed release source.
-export EAGLE_TOKENSPEED_AUX_REQUIREMENTS='tokenspeed-scheduler==0.1.40+eagle.4 tokenspeed-kernel==0.1.4+eagle.4'
+export EAGLE_TOKENSPEED_AUX_REQUIREMENTS='tokenspeed-scheduler==0.1.40+eagle.4 tokenspeed-kernel==0.1.4+eagle.5'
diff --git a/docs/support-matrix.md b/docs/support-matrix.md
index 748ddd0..9936c32 100644
--- a/docs/support-matrix.md
+++ b/docs/support-matrix.md
@@ -6,26 +6,11 @@ last_verified: 2026-10-20
# Measured support matrix
-EAGLE-4 sizes speculative trees from measured serving costs. The EAGLE-4 tables report Qwen3.8-27B with an
-EAGLE-4 draft on NVIDIA B300 and AMD MI355X. TokenSpeed is experimental.
-
-Throughput is generated tokens divided by elapsed workload time, in tokens/s. Higher is better.
-An engine's speedup divides EAGLE-4 throughput by its own matched no-speculation control.
-Client concurrency is distinct from the server's running-request cap. A client above that cap queues.
-
-These are configuration-specific measurements, not a claim of a speedup at every batch size. Rows from
-short replays, full multi-turn workloads and synthetic contexts are not interchangeable. In particular, these
-replays do not replace EAGLE-4's full SPEED-Bench qualitative result or its HumanEval closed-loop results.
-Raw launch manifests and request traces are retained by the maintainers, not shipped in this source package.
+Every row is Qwen3.8-27B with its EAGLE-4 draft, and each speedup divides EAGLE-4 throughput by the same engine's
+matched no-speculation control. [Reading the numbers](#reading-the-numbers) defines the terms.
## SGLang on NVIDIA B300
-The target and EAGLE-4 v3 draft use BF16. The dynamic tree has maximum depth 12, top-k 4 and verify width 16.
-Target and draft attention use FlashInfer with page size 1. The SPEED-Bench HTTP client uses temperature 0.
-Concurrency 1 uses 22 conversations and 31 turns with at most 1024 new tokens per turn. Concurrency 8 uses
-88 first turns with at most 512 new tokens. Concurrency 64 uses 220 first turns with the same 512-token cap.
-Every timed pass follows a warm pass. EAGLE-4 measures live costs. Each EAGLE-4 configuration has two starts.
-
| GPUs / TP | Client concurrency | Running cap | Graph padding | No-spec tokens/s | EAGLE-4 tokens/s | Speedup |
|---|---:|---:|---|---:|---:|---:|
| 1 | 1 | 1 | Irrelevant at batch 1 | 103.1 | 343.1 | 3.33× |
@@ -35,6 +20,14 @@ Every timed pass follows a warm pass. EAGLE-4 measures live costs. Each EAGLE-4
| 4 | 64 | 32 | Disabled in measurement | 3925.3 | 5281.3 | 1.35× |
| 8 | 1 | 1 | Irrelevant at batch 1 | 241.7 | 828.6 | 3.43× |
+### B300 SGLang measurement conditions
+
+The target and EAGLE-4 v3 draft use BF16. The dynamic tree has maximum depth 12, top-k 4 and verify width 16.
+Target and draft attention use FlashInfer with page size 1. The SPEED-Bench HTTP client uses temperature 0.
+Concurrency 1 uses 22 conversations and 31 turns with at most 1024 new tokens per turn. Concurrency 8 uses
+88 first turns with at most 512 new tokens. Concurrency 64 uses 220 first turns with the same 512-token cap.
+Every timed pass follows a warm pass. EAGLE-4 measures live costs. Each EAGLE-4 configuration has two starts.
+
TP4 concurrency 1 measures the released stack: `sglang==0.5.23.dev252+eagle.4` from the CUDA wheel index and
`eagle-harness==0.1.0` from PyPI, installed into a clean environment and served by the SGLang quickstart command
from a directory outside any checkout. It has six EAGLE-4 starts (725.2 to 752.7 tokens/s) and three no-spec
@@ -43,16 +36,26 @@ twenty-prompt check.
TP4 concurrency 8 measures 2749.5 and 2891.8 tokens/s across its two starts.
TP8 concurrency 1 and 8 each give 5 exact / 14 near tie / 1 fp32-tolerance pass, with zero failures
-in the configuration-specific twenty-prompt checks. This does not resolve TP8 concurrency-8 throughput variability.
+in the configuration-specific twenty-prompt checks. TP8 concurrency-8 throughput variability is being investigated.
Concurrency 64 queues into 32 running slots. The measured padding-disabled settings identify those results,
not a recommendation to disable padding. Concurrent serving commands keep SGLang's default padding enabled.
## SGLang on AMD MI355X
+| Configuration | Client concurrency | No-spec tokens/s | EAGLE-4 tokens/s | Speedup | EAGLE-4 starts |
+|---|---:|---:|---:|---:|---:|
+| Earlier one-request profile | 1 | — | 289.4 | — | 2 |
+| Measured profile, explicit path | 8 | 552.7 | 1145.9 | 2.07× | 2 |
+| Measured profile, explicit path | 32 | 1269.2 | 1980.8 | 1.56× | 2 |
+| Packaged profile, automatic selection | 8 | — | 1209.2 | — | 1 |
+
+Concurrency 64 on one MI355X is a [known limit](known-limits.md#performance-and-correctness-in-progress).
+
+### MI355X measurement conditions
+
Qwen3.8-27B and its EAGLE-4 draft run on one MI355X at TP1 with BF16 target, KV cache and recurrent state.
-AMD ROCm is included at all supported batch sizes. Concurrency 64 on one MI355X is a
-[known limit](known-limits.md#performance-and-correctness-gaps).
+AMD ROCm is included at all supported batch sizes.
The measured-profile workload uses the original SPEED-Bench client, 44 conversations and 60 turns,
temperature 0, at most 1024 new tokens per turn and a full matching warm pass. The running cap matches client
concurrency. The tree has maximum depth 12, top-k 4 and verify width 32. It uses measured MI355X costs,
@@ -60,13 +63,6 @@ AITER target attention, rounded recurrent-state updates and disabled draft prela
disable graph padding but capture every integer batch size up to the running cap. These are recorded
measurement conditions, not a recommendation to disable padding.
-| Configuration | Client concurrency | No-spec tokens/s | EAGLE-4 tokens/s | Speedup | EAGLE-4 starts |
-|---|---:|---:|---:|---:|---:|
-| Earlier one-request profile | 1 | — | 289.4 | — | 2 |
-| Measured profile, explicit path | 8 | 552.7 | 1145.9 | 2.07× | 2 |
-| Measured profile, explicit path | 32 | 1269.2 | 1980.8 | 1.56× | 2 |
-| Packaged profile, automatic selection | 8 | — | 1209.2 | — | 1 |
-
The earlier one-request profile uses 22 conversations and 31 turns with the same temperature and token cap,
a running cap of 1 and two starts at 291.8 and 287.0 tokens/s. A separate packaged-profile check at concurrency 1
reaches 290.9 tokens/s on the 60-turn workload in one start. No matched no-speculation speedup is reported for
@@ -84,11 +80,6 @@ separate from a tolerance pass and does not imply equality to the native BF16 to
## vLLM on NVIDIA B300
-These speedups use vLLM's own matched no-speculation control.
-Qwen3.8-27B BF16 uses the EAGLE-4 v3 draft and the SPEED-Bench HTTP client that preserves reasoning text between
-turns. Temperature is 0. Concurrency 1 uses 22 conversations and 31 turns with at most 1024 new tokens per turn.
-Concurrency 8 uses 88 first turns with at most 512 new tokens. Each timed pass follows a full matching warm pass.
-
| GPUs / TP | Client concurrency | Running cap | Mode | No-spec tokens/s | EAGLE-4 tokens/s | Speedup | EAGLE-4 starts |
|---|---:|---:|---|---:|---:|---:|---:|
| 1 | 1 | 8 | Static tree | 104.0 | 279.9 | 2.69× | 2 |
@@ -98,18 +89,18 @@ Concurrency 8 uses 88 first turns with at most 512 new tokens. Each timed pass f
| 8 | 1 | 8 | Static tree | 317.2 | 745.4 | 2.35× | 2 |
| 8 | 8 | 8 | Static tree | 2063.0 | 2890.0 | 1.40× | 2 |
+### B300 vLLM measurement conditions
+
+Qwen3.8-27B BF16 uses the EAGLE-4 v3 draft and the SPEED-Bench HTTP client that preserves reasoning text between
+turns. Temperature is 0. Concurrency 1 uses 22 conversations and 31 turns with at most 1024 new tokens per turn.
+Concurrency 8 uses 88 first turns with at most 512 new tokens. Each timed pass follows a full matching warm pass.
+
The tree uses top-k 4, depth 14 and verify width 16 with `CUSTOM` attention. TP4 uses the derived depth stop.
TP1 and TP8 use fixed depth. No-spec uses the engine's default FlashInfer attention with the same cap.
The two TP8 concurrency-8 starts differ by 11%, and both are included in the mean.
### Measured high-load tree schedule
-These rows measure the release vLLM engine with the `eagle-harness` 0.1.0 vLLM code on B300 TP4, served by
-the vLLM quickstart recipe: running cap 64, calibrated TP4 costs and the packaged rounds file. The workload is
-SPEED-Bench first turns with at most 512 new tokens: 88 requests at concurrency 8, 220 at 64 and 440 at 128.
-Each timed pass follows a matching warm pass. Every arm, including the matched no-spec control, has two
-independent server starts, interleaved in one allocation. Temperature 1 uses top-p 1, no top-k and a fixed seed.
-
| Client concurrency | Temperature | No-spec tokens/s | EAGLE-4 tokens/s | Speedup | Verify width |
|---:|---:|---:|---:|---:|---:|
| 8 | 0 | 1695.2 | 2932.7 | 1.73× | 16 |
@@ -117,9 +108,17 @@ independent server starts, interleaved in one allocation. Temperature 1 uses top
| 128 | 0 | 7063.0 | 8552.5 | 1.21× | 4 |
| 64 | 1 | 7034.3 | 7267.9 | 1.03× | 4 |
-Concurrency 128 queues into the 64 running slots. Without the rounds file the tree keeps width 16 at every
-batch size, and the same concurrency-64 cells give 5349.4 tokens/s (0.76×) at temperature 0 and 4537.2 tokens/s
-(0.65×) at temperature 1.
+Without the rounds file the tree keeps width 16 at every batch size, and the same concurrency-64 cells give
+5349.4 tokens/s (0.76×) at temperature 0 and 4537.2 tokens/s (0.65×) at temperature 1.
+
+#### How the high-load rows were measured
+
+These rows measure the release vLLM engine with the `eagle-harness` 0.1.0 vLLM code on B300 TP4, served by
+the vLLM quickstart recipe: running cap 64, calibrated TP4 costs and the packaged rounds file. The workload is
+SPEED-Bench first turns with at most 512 new tokens: 88 requests at concurrency 8, 220 at 64 and 440 at 128.
+Each timed pass follows a matching warm pass. Every arm, including the matched no-spec control, has two
+independent server starts, interleaved in one allocation. Temperature 1 uses top-p 1, no top-k and a fixed seed.
+Concurrency 128 queues into the 64 running slots.
Greedy checks against no-speculation decoding pass on both verify widths. Twenty sequential prompts (width 16)
give 5 exact / 14 near tie / 1 fp32-tolerance pass, and 64 concurrent prompts at concurrency 64 (width 4) give
@@ -137,23 +136,27 @@ No rounds file means no width change.
## Short-workload C8 checks
-These B300 TP4 checks use two independent starts, 16 requests with a 256-token cap and a matching
-warm pass. They report absolute rates for the final runtime source on its pinned release engine.
-They do not replace the longer-workload tables or establish a speedup over no-spec decoding.
-
| Engine | Tokens/s | Running cap | Policy |
|---|---:|---:|---|
| SGLang | 2586.7 | 32 | Live costs, graph padding enabled |
| vLLM | 2708.8 | 64 | Width 16 at this batch size |
+### Short-workload measurement conditions
+
+These B300 TP4 checks use two independent starts, 16 requests with a 256-token cap and a matching
+warm pass. They report absolute rates for the final runtime source on its pinned release engine.
+They do not replace the longer-workload tables or establish a speedup over no-spec decoding.
Each measured pass lasts roughly 1.5 seconds with substantial start-to-start spread. No matching
no-spec mean is reported for these short checks.
## TokenSpeed (experimental)
-The adapter implements static and calibrated trees, chains and sampled trees. This is implementation coverage,
-not a performance or distribution-quality claim for every configuration.
-Prompt logprobs, live cost measurement and native no-speculation rounds are not supported.
+| GPUs / TP | Client concurrency | Running cap | No-spec tokens/s | EAGLE-4 tokens/s | Speedup |
+|---|---:|---:|---:|---:|---:|
+| 4 | 8 | 32 | 1555.7 | 3150.2 | 2.02× |
+| 4 | 64 | 32 | 4894.4 | 6950.2 | 1.42× |
+
+### TokenSpeed measurement conditions
These rows measure the released wheels (`tokenspeed==0.1.1+eagle.4` with its scheduler and kernel wheels from
the CUDA index, `eagle-harness==0.1.0` from PyPI), installed by the install guide's TokenSpeed block and served
@@ -162,15 +165,13 @@ turns at temperature 0 with at most 256 new tokens: 22 requests at concurrency 8
timed pass follows a matching warm pass. EAGLE-4 and the matched no-spec control each have two starts,
interleaved in one allocation.
-| GPUs / TP | Client concurrency | Running cap | No-spec tokens/s | EAGLE-4 tokens/s | Speedup |
-|---|---:|---:|---:|---:|---:|
-| 4 | 8 | 32 | 1555.7 | 3150.2 | 2.02× |
-| 4 | 64 | 32 | 4894.4 | 6950.2 | 1.42× |
-
Each pass is one short wave of requests. The EAGLE-4 starts are 3062.5 and 3238.0 tokens/s at concurrency 8,
and 6876.4 and 7023.9 tokens/s at concurrency 64. A twenty-prompt greedy check of the same installation
gives 4 exact / 16 fp32-tolerance passes and no failures.
+The adapter implements static and calibrated trees, chains and sampled trees. This is implementation coverage,
+not a performance or distribution-quality claim for every configuration.
+Prompt logprobs, live cost measurement and native no-speculation rounds are not available on TokenSpeed yet.
No TokenSpeed throughput number is claimed for an unmeasured or non-matching workload. The CUDA install
checks completed real static and cost-model requests, but a static-mode first divergence remains above the
fp32 gate. Successful installation or generation is not blanket correctness qualification. TP1/TP8, sampled
@@ -178,11 +179,6 @@ distributions, long contexts and release-stack Nemotron need separate qualificat
## Artifact installation checks
-These checks use one server start on B300 TP4 and twenty prompts with up to 256 generated tokens. The rate is
-total generated tokens divided by summed request latency, not a multi-start concurrent throughput benchmark.
-The image rows test the published `0.1.0` images, each of which is pullable without credentials. The wheel
-rows predate publication and used a private index.
-
| Artifact and mode | Correctness result | Tokens/s | Qualification limit |
|---|---|---:|---|
| SGLang image `eagle-4-sglang:0.1.0` | 4 exact / 16 near ties | 537.40 | Hook, dependency, quickstart request and correctness checks pass |
@@ -191,17 +187,17 @@ rows predate publication and used a private index.
| TokenSpeed wheel, serve | 4 exact / 16 fp32-tolerance passes | 786.33 | Host-compatible CUDA 13.2 JIT; CUDA 13.0 JIT fails on Ubuntu 26.04 / glibc 2.43 |
| TokenSpeed wheel, static | 5 exact / 14 fp32 passes / 1 failure | 706.36 | Not qualified: failing gap 0.178637 exceeds 0.125 |
-The SGLang and vLLM images contain the `eagle-harness` 0.1.0 wheel published on PyPI.
-A near-tie or fp32-tolerance pass is not an exact-token match. TokenSpeed lacks full-decode native
-top-logprobs, so each divergent prefix in these install checks goes directly to fp32 arbitration.
+### Installation-check conditions
-## Correctness and request features
+These checks use one server start on B300 TP4 and twenty prompts with up to 256 generated tokens. The rate is
+total generated tokens divided by summed request latency, not a multi-start concurrent throughput benchmark.
+The image rows test the published `0.1.0` images, each of which is pullable without credentials. The wheel
+rows predate publication and used a private index. The SGLang and vLLM images contain the `eagle-harness` 0.1.0
+wheel published on PyPI. A near-tie or fp32-tolerance pass is not an exact-token match. Full-decode native
+top-logprobs are not available on TokenSpeed yet, so each divergent prefix in these install checks goes directly
+to fp32 arbitration.
-The greedy gate reports exact, near-tie, fp32-tolerance, candidate-is-fp32-argmax and failed outputs separately.
-A first-divergence reference-logprob gap of at most 0.13 is a near tie. Larger gaps require an fp32 forward
-at the exact common prefix. A pair within two BF16 steps at that logit magnitude passes the tolerance check.
-EAGLE-4 token equal to the fp32 argmax also passes, reported in its own category. Passing either rule
-is not bitwise equality to the native BF16 token stream.
+## Correctness and request features
| Configuration or feature | Evidence |
|---|---|
@@ -217,5 +213,24 @@ is not bitwise equality to the native BF16 token stream.
| SGLang logprobs and sampling | Accepted-token and input logprobs and target-side speculative sampling are implemented |
| Nemotron / Mamba2 | Adapters are implemented on SGLang and vLLM, but a general correctness-qualified speed claim is not established |
+### How these were checked
+
+The greedy gate reports exact, near-tie, fp32-tolerance, candidate-is-fp32-argmax and failed outputs separately.
+A first-divergence reference-logprob gap of at most 0.13 is a near tie. Larger gaps require an fp32 forward
+at the exact common prefix. A pair within two BF16 steps at that logit magnitude passes the tolerance check.
+EAGLE-4 token equal to the fp32 argmax also passes, reported in its own category. Passing either rule
+is not bitwise equality to the native BF16 token stream.
+
See [known limits](known-limits.md) for configuration restrictions and [compatibility](compatibility.md) for
required engine hooks. Kernel compilation on another GPU does not establish serving correctness or speed there.
+
+## Reading the numbers
+
+Throughput is generated tokens divided by elapsed workload time, in tokens/s, and higher is better. Client
+concurrency is the number of requests the client keeps in flight. It is distinct from the server's running-request
+cap, and a client above that cap queues. TokenSpeed is experimental.
+
+These are configuration-specific measurements, not a claim of a speedup at every batch size. Rows from short
+replays, full multi-turn workloads and synthetic contexts are not interchangeable. In particular, these replays do
+not replace EAGLE-4's full SPEED-Bench qualitative result or its HumanEval closed-loop results. Raw launch manifests
+and request traces are retained by the maintainers, not shipped in this source package.
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
index 165062d..0be6cba 100644
--- a/docs/troubleshooting.md
+++ b/docs/troubleshooting.md
@@ -39,6 +39,16 @@ tokenspeed: not installed
no supported host installed (one of: sglang, vllm, tokenspeed)
```
+## Install problems
+
+| symptom | cause | fix |
+|---|---|---|
+| `The virtual environment was not created successfully because ensurepip is not available` | Ubuntu ships `venv` without `ensurepip` | `apt install python3.12-venv`, then create the environment again |
+| pip builds `cuda-tile` from source or cannot find `cuda-tile==1.6.0rc5` | the SGLang command ran without `--extra-index-url https://pypi.nvidia.com` | use the full command from [Install](install.md#sglang-cuda) |
+| pip installs a stock `sglang`, `vllm` or `tokenspeed` | the engine pin or the EAGLE-4 index is missing, and stock packages have no EAGLE-4 hooks | pin the exact `+eagle.N` version with the EAGLE-4 index, as in [Install](install.md) |
+| JIT compilation fails with `rsqrt`/`rsqrtf` exception-specification errors | CUDA 13.0 `nvcc` with the C headers of Ubuntu 26.04 / glibc 2.43 | `export CUDA_HOME=/usr/local/cuda-13.2` (a CUDA toolkit compatible with the host headers), or use a release image. The wheel index stays `cu130`. |
+| `eagle_harness.__file__` points into a checkout of this repository | `python -m` puts the current directory first on the import path, so a server started inside the checkout runs the source tree | start servers from another directory, or with the `sglang serve` and `vllm serve` commands, which do not import from the current directory |
+
## Server errors at startup
| message | cause | fix |
@@ -60,13 +70,14 @@ no supported host installed (one of: sglang, vllm, tokenspeed)
| `FlashInferSamplingBackend cannot verify draft trees; use --sampling-backend greedy or triton` (TokenSpeed) | TokenSpeed's default sampling backend | pass `--sampling-backend greedy` |
| `--speculative-num-draft-tokens N verifies more nodes than a tree of depth ... holds` (TokenSpeed) | the tree cannot fill the verify width | raise `--speculative-num-steps` or `--speculative-eagle-topk`, or lower the width |
| `EAGLE-4 drafts with [...] are not ported to eagle-harness yet (dense drafts only).` | a draft structure the engine does not build (see the FAQ, "Which models work?") | use a supported single-layer EAGLE-4 draft |
+| `TypeError: int() argument must be ... not 'NoneType'` in `average_context`, on the first request (sglang) | `eagle-harness` 0.1.0 with SGLang's default overlap scheduler | upgrade to `eagle-harness` 0.1.1 or newer, which serves with the overlap scheduler, or pass `--disable-overlap-schedule` |
## Requests refused
sglang refuses requests with multimodal inputs (`Speculative algorithm EAGLE_HARNESS does not support multimodal
inputs.`). Serve them from a server without the method.
-## Slower than expected
+## Throughput below the support matrix
- Check that the cost tables match your GPU, TP size, tree shape and engine build ([Calibration](calibration.md)).
A mismatched `key.json` is refused. Unkeyed legacy tables can load with a warning, and the key does not
diff --git a/tests/test_install_docs.py b/tests/test_install_docs.py
index cc4b044..5a7c55b 100644
--- a/tests/test_install_docs.py
+++ b/tests/test_install_docs.py
@@ -1,7 +1,7 @@
# Copyright 2026 Eagle AI Corp.
# SPDX-License-Identifier: LicenseRef-Eagle-4-Source-Available
-"""The install guides run from outside the checkout and install ROCm from public indexes only."""
+"""The install guides use the release pins, run from outside the checkout and install ROCm from public indexes only."""
from pathlib import Path
import re
@@ -10,6 +10,7 @@ import pytest
REPO = Path(__file__).resolve().parents[1]
GUIDES = [REPO / "README.md", *sorted((REPO / "docs").glob("*.md"))]
+QUICKSTARTS = [REPO / "README.md", REPO / "docs/install.md", *sorted((REPO / "docs").glob("quickstart-*.md"))]
def bash_blocks(path: Path) -> list[str]:
@@ -25,6 +26,17 @@ def release_settings() -> dict[str, str]:
return dict(re.findall(r"^export (EAGLE_[A-Z0-9_]+)=(.*)$", text, re.M))
+def resolved_settings() -> dict[str, str]:
+ """release.env values without quotes and with earlier variables substituted, as the shell reads them."""
+ values: dict[str, str] = {}
+ for name, raw in release_settings().items():
+ value = raw[1:-1] if raw[:1] in "'\"" else raw
+ if raw.startswith('"'):
+ value = re.sub(r"\$\{?(\w+)\}?", lambda m: values[m[1]], value)
+ values[name] = value
+ return values
+
+
@pytest.mark.parametrize("guide", GUIDES, ids=lambda p: str(p.relative_to(REPO)))
def test_guides_never_enter_the_checkout(guide):
# `python -m` imports from the current directory, so a server started inside the checkout
@@ -42,6 +54,50 @@ def test_guide_variables_come_from_release_settings(guide):
assert name in settings, f"{guide.name} uses {name}, which docs/release.env does not export"
[email protected]("guide", QUICKSTARTS, ids=lambda p: str(p.relative_to(REPO)))
+def test_copy_paste_commands_use_the_release_pins(guide):
+ # The quick starts spell the pins out so they paste without release.env; they must match it exactly.
+ s = resolved_settings()
+ pins = {
+ "sglang": s["EAGLE_SGLANG_VERSION"],
+ "vllm": s["EAGLE_VLLM_VERSION"],
+ "tokenspeed": s["EAGLE_TOKENSPEED_VERSION"],
+ **dict(re.findall(r"(tokenspeed-\w+)==(\S+)", s["EAGLE_TOKENSPEED_AUX_REQUIREMENTS"])),
+ }
+ indexes = {s[k] for k in ("EAGLE_CUDA_INDEX", "EAGLE_TORCH_CUDA_INDEX", "EAGLE_NVIDIA_INDEX")}
+ images = {s[k] for k in ("EAGLE_SGLANG_IMAGE", "EAGLE_VLLM_IMAGE", "EAGLE_TOKENSPEED_IMAGE")}
+ for block in bash_blocks(guide):
+ for package, version in re.findall(r'"(sglang|vllm|tokenspeed(?:-\w+)?)==([^"\s]+)"', block):
+ assert version == pins[package], f"{guide.name}: {package}=={version}, release.env pins {pins[package]}"
+ for url in re.findall(r"--extra-index-url (https://\S+)", block):
+ assert url in indexes, f"{guide.name}: {url} is not a release index"
+ for image in re.findall(r"ghcr\.io/eagle-hq/\S+", block):
+ assert image in images, f"{guide.name}: {image} is not a published release image"
+ if re.search(r'"(sglang|vllm|tokenspeed)==[^"]*\+eagle\.', block):
+ # A +eagle.N pin resolves only with the EAGLE-4 wheel index in the same command.
+ assert s["EAGLE_CUDA_INDEX"] in block, block
+
+
+def test_readme_quick_start_is_one_install_one_serve_one_request():
+ quick = section(REPO / "README.md", "## Quick start")
+ docker, install, serve, request = re.findall(r"```bash\n(.*?)```", quick, re.S)
+ assert "docker run" in docker and "--speculative-algorithm EAGLE_HARNESS" in docker
+ assert install.count("pip install") == 1 and '"eagle-harness[sglang]"' in install
+ assert serve.startswith("sglang serve ") and re.search(r"--speculative-draft-model-path [\w-]+/Qwen3\.8-27B-EAGLE4$", serve.strip())
+ assert request.startswith("curl http://localhost:30000/")
+ # The README, the install guide and the SGLang guide show the same commands.
+ assert install in bash_blocks(REPO / "docs/install.md")
+ assert serve in bash_blocks(REPO / "docs/quickstart-sglang.md")
+ assert docker in bash_blocks(REPO / "docs/quickstart-sglang.md")
+
+
+def test_copy_paste_commands_never_pin_the_harness():
+ # An unpinned harness cannot name an unpublished or old version.
+ for guide in QUICKSTARTS:
+ for block in bash_blocks(guide):
+ assert not re.search(r"eagle-harness(\[\w+\])?==\d", block), f"{guide.name}: {block}"
+
+
def test_rocm_install_uses_public_indexes_and_the_shipped_lock():
install = section(REPO / "docs/install.md", "## SGLang ROCm on AMD")
block = re.findall(r"```bash\n(.*?)```", install, re.S)[0]