Official Inference Code of EAGLE-4
docs: simplify the README, install and quickstarts
Seyfal Sultanov18df0e919ebc
diff --git a/README.md b/README.mdindex 5d9d17a..748ee5a 100644--- a/README.md+++ b/README.md@@ -10,12 +10,12 @@<p><a href="EAGLE4_Paper.pdf"><b>Paper (EAGLE-4)</b></a> ·- <a href="https://huggingface.co/EAGLE-corp/EAGLE4-Qwen3.8-27B"><b>EAGLE-4 draft for Qwen3.8-27B</b></a> ·+ <a href="https://huggingface.co/EAGLE-Corp/Qwen3.8-27B-EAGLE4"><b>EAGLE-4 draft for Qwen3.8-27B</b></a> ·<a href="docs/README.md"><b>Documentation</b></a></p><p align="center">- <a href="https://github.com/eagle-hq/EAGLE-4/releases/tag/v0.1.1"><img src="https://img.shields.io/badge/eagle--harness-v0.1.1-orange.svg" alt="eagle-harness 0.1.1"></a>+ <a href="https://pypi.org/project/eagle-harness/"><img src="https://img.shields.io/pypi/v/eagle-harness?label=eagle-harness&color=orange" alt="eagle-harness on PyPI"></a><a href="https://github.com/eagle-hq/EAGLE-4/issues" target="_blank"><img src="https://img.shields.io/badge/Maintained%3F-yes-green.svg" alt="Maintenance"></a><a href="https://github.com/eagle-hq/EAGLE-4/pulls" target="_blank"><img src="https://img.shields.io/badge/Contributions-welcome-brightgreen.svg" alt="Contributions welcome"></a></p>@@ -49,165 +49,81 @@## EAGLE-4-EAGLE-4 (Extrapolation Algorithm for Greater Language-model Efficiency) is the latest generation of the EAGLE family-for accelerating large language model inference. One drafter adapts its speculative decoding strategy to the batch-size, the hardware and the workload, and keeps the target model's output distribution.--The EAGLE-4 draft uses a sliding-window draft layer, a low-rank candidate head and an append-only draft cache. Dense-and latent-MoE draft structures are implemented. Qwen3.8-27B with its-[EAGLE-4 draft](https://huggingface.co/EAGLE-corp/EAGLE4-Qwen3.8-27B) is the measured target pair.--## Engines--This repository is `eagle-harness`, a pip-installable package (`eagle_harness`) that serves EAGLE-4 drafts on-SGLang, vLLM and TokenSpeed through engine plugins. EAGLE-4 sizes the speculative tree from measured draft and-verification costs for the batch size, hardware and workload it is running on. It registers through hooks the engine-exposes and does not patch the engine at runtime.--The patched engines ship as pinned wheels and container images, not as source trees in this repository:--| Engine | Hardware | Wheel index | Image | Source |-| --- | --- | --- | --- | --- |-| SGLang | NVIDIA B300, CUDA 13.0 | [cu130](https://eagle-hq.github.io/EAGLE-4/whl/cu130/) | `ghcr.io/eagle-hq/eagle-4-sglang:0.1.0` | [eagle-hq/sglang](https://github.com/eagle-hq/sglang) |-| SGLang | AMD MI355X, ROCm 7.2 | [rocm](https://eagle-hq.github.io/EAGLE-4/whl/rocm/) | not yet published ([build it](docs/install.md#build-the-rocm-image)) | [eagle-hq/sglang](https://github.com/eagle-hq/sglang) |-| vLLM | NVIDIA B300, CUDA 13.0 | [cu130](https://eagle-hq.github.io/EAGLE-4/whl/cu130/) | `ghcr.io/eagle-hq/eagle-4-vllm:0.1.0` | [eagle-hq/vllm](https://github.com/eagle-hq/vllm) |-| TokenSpeed, experimental | NVIDIA CUDA only | [cu130](https://eagle-hq.github.io/EAGLE-4/whl/cu130/) | `ghcr.io/eagle-hq/eagle-4-tokenspeed:0.1.0` | [eagle-hq/tokenspeed](https://github.com/eagle-hq/tokenspeed) |--Stock engine packages do not carry the hooks. [Release settings](docs/release.env) is the one place that holds-every version pin, index URL and image tag. The wheels are also attached to the-[v0.1.0 GitHub release](https://github.com/eagle-hq/EAGLE-4/releases/tag/v0.1.0): the engines did not change in 0.1.1.-The 0.1.0 images carry `eagle-harness` 0.1.0.--## Measured results--The [support matrix](docs/support-matrix.md) reports every measurement with its conditions, separately for-loading, greedy correctness, request features and throughput. Throughput is generated tokens per second of-workload time, higher is better. The rows below are the shapes the quickstarts ship. Each one is the mean of-at least two server starts on the SPEED-Bench client at temperature 0.--| Engine | GPUs | Client concurrency | No-spec tokens/s | EAGLE-4 tokens/s | Speedup |-| --- | --- | ---: | ---: | ---: | ---: |-| SGLang | 4× B300, TP4 | 1 | 216.3 | 735.7 | 3.40× |-| SGLang | 4× B300, TP4 | 8 | 1479.4 | 2820.6 | 1.91× |-| SGLang | 4× B300, TP4 | 64 | 3925.3 | 5281.3 | 1.35× |-| SGLang | 1× MI355X, TP1 | 8 | 552.7 | 1145.9 | 2.07× |-| SGLang | 1× MI355X, TP1 | 32 | 1269.2 | 1980.8 | 1.56× |-| vLLM | 4× B300, TP4 | 1 | 241.6 | 722.8 | 2.99× |-| vLLM | 4× B300, TP4 | 8 | 1699.9 | 2883.9 | 1.70× |--Each speedup divides EAGLE-4 throughput by the same engine's matched no-speculation control.-TokenSpeed is experimental; its measured concurrent throughput is in the-[support matrix](docs/support-matrix.md#tokenspeed-experimental). An unexplained greedy mismatch fails the-correctness gate even when the speculative server is faster.--## Install--Install one patched engine per environment, then `eagle-harness==0.1.1` from PyPI. The-[v0.1.0 GitHub release](https://github.com/eagle-hq/EAGLE-4/releases/tag/v0.1.0) carries the engine wheels, not `eagle-harness`.-Use Python 3.12 for the CUDA engines and Python 3.10 for the ROCm wheel. The commands read their pins from the-release settings file in the checkout. Run them, and start servers, from the directory that contains the checkout.-Inside it, `python -m` would import the source tree's `eagle_harness/` instead of the installed wheel.+EAGLE-4 (Extrapolation Algorithm for Greater Language-model Efficiency) speeds up LLM inference with speculative+decoding. One draft model adapts its draft tree to the batch size, the hardware and the workload, and keeps the+target model's output distribution. On four B300 GPUs it serves Qwen3.8-27B 3.40× faster than SGLang without+speculation for one request at a time.++This repository is `eagle-harness`, the package that runs EAGLE-4 drafts inside SGLang, vLLM and TokenSpeed.++## Quick start++You need Linux, four NVIDIA GPUs with a CUDA 13 driver (tested on B300) and about 70 GB of disk for+[Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) and its+[EAGLE-4 draft](https://huggingface.co/EAGLE-Corp/Qwen3.8-27B-EAGLE4), which download on the first start.++Start an SGLang server with Docker:```bash-git clone https://github.com/eagle-hq/EAGLE-4-. EAGLE-4/docs/release.env+docker run --gpus all --ipc host -p 127.0.0.1:30000:30000 \+ -e HF_HOME=/hf -v ~/.cache/huggingface:/hf \+ ghcr.io/eagle-hq/eagle-4-sglang:0.1.0 \+ sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 --host 0.0.0.0 \+ --disable-overlap-schedule --speculative-algorithm EAGLE_HARNESS \+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 \+ --speculative-num-steps 12 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \+ --speculative-algorithm-config '{}'```-SGLang on CUDA:+Or install with pip, in Python 3.12. Pip installs also need the [CUDA toolkit and a C++ compiler](docs/install.md#requirements):```bash-python3.12 -m venv .venv-sglang && source .venv-sglang/bin/activate-python -m pip install --upgrade pip-python -m pip install --extra-index-url "$EAGLE_CUDA_INDEX" \- --extra-index-url "$EAGLE_TORCH_CUDA_INDEX" "sglang==$EAGLE_SGLANG_VERSION" 'flashinfer-python>=0.7.2'-python -m pip install "eagle-harness==$EAGLE_HARNESS_VERSION"-eagle-harness-check+python3.12 -m venv .venv && source .venv/bin/activate+pip install "eagle-harness[sglang]" "sglang==0.5.23.dev252+eagle.5" \+ --extra-index-url https://eagle-hq.github.io/EAGLE-4/whl/cu130/ \+ --extra-index-url https://download.pytorch.org/whl/cu130 \+ --extra-index-url https://pypi.nvidia.com```-Or open the release image, which has the engine and EAGLE-4 installed. The 0.1.0 image carries `eagle-harness`-0.1.0; upgrade it inside the container:-```bash-docker run --rm --gpus all --ipc host -it \- -p 127.0.0.1:30000:30000 -v "$PWD/models:/models:ro" \- --entrypoint bash "$EAGLE_SGLANG_IMAGE"-python -m pip install -U eagle-harness==0.1.1-eagle-harness-check+sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 \+ --speculative-algorithm EAGLE_HARNESS \+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4```-`eagle-harness-check` exits zero when the installed engine provides every hook the plugin needs. It does not-prove that kernels run or that generation is correct, so send the quickstart request before you measure anything.+When the server logs `The server is fired up and ready to roll!`, send a request from another terminal:++```bash+curl http://localhost:30000/v1/chat/completions -H 'Content-Type: application/json' \+ -d '{"model": "Qwen/Qwen3.8-27B", "messages": [{"role": "user", "content": "Explain how a hash table works."}], "max_tokens": 256}'+```-The [installation guide](docs/install.md) has the same steps for [vLLM](docs/install.md#vllm-cuda) and-[TokenSpeed](docs/install.md#tokenspeed-cuda-experimental), which needs its scheduler and kernel wheels from the-same index. SGLang on AMD MI355X installs a hash-locked closure from PyPI, the ROCm index and AMD's indexes:-follow [SGLang ROCm](docs/install.md#sglang-rocm-on-amd), then the [AMD recipe](docs/quickstart-amd.md).+For vLLM, TokenSpeed (experimental) and AMD MI355X, follow the [engine guides](#more).-## Serve+## Measured speedup-Serve Qwen3.8-27B BF16 with its EAGLE-4 draft on four B300 GPUs. SGLang prices the tree from costs it measures-during served rounds, so this recipe needs no offline calibration. The full command with its environment is in-the [SGLang quickstart](docs/quickstart-sglang.md):+Each speedup divides EAGLE-4 throughput by the same engine's throughput without speculation, on Qwen3.8-27B at+temperature 0 on the SPEED-Bench client, with the measured recipe of each engine guide. The+[support matrix](docs/support-matrix.md) has the tokens/s, the correctness checks and how each number was measured.-```bash-python -m sglang.launch_server --model-path "$MODEL" --tp-size 4 --dtype bfloat16 \- --trust-remote-code --attention-backend flashinfer --page-size 1 \- --max-running-requests 32 --cuda-graph-max-bs-decode 32 \- --disable-overlap-schedule --cuda-graph-backend-decode full \- --mem-fraction-static 0.75 --context-length 196608 --max-total-tokens 196608 \- --chunked-prefill-size 16384 --mamba-ssm-dtype bfloat16 --disable-radix-cache \- --reasoning-parser qwen3 --random-seed 721 --skip-server-warmup \- --host 127.0.0.1 --port 30000 \- --speculative-algorithm EAGLE_HARNESS --speculative-draft-model-path "$DRAFT" \- --speculative-draft-model-quantization unquant --speculative-draft-attention-backend flashinfer \- --speculative-attention-mode prefill --speculative-num-steps 12 \- --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \- --speculative-algorithm-config '{"mode":"serve"}'-```+| Engine | GPUs | 1 request | 8 concurrent | 64 concurrent |+| --- | --- | ---: | ---: | ---: |+| SGLang | 4× B300 | 3.40× | 1.91× | 1.35× |+| vLLM | 4× B300 | 2.99× | 1.70× | 1.16× |+| TokenSpeed, experimental | 4× B300 | not measured | 2.02× | 1.42× |-Then send a greedy request and check that the response carries generated text:+On one AMD MI355X, SGLang with EAGLE-4 is 2.07× faster at 8 concurrent requests and 1.56× at 32.-```bash-curl --fail http://127.0.0.1:30000/generate \- -H 'Content-Type: application/json' \- -d '{"text":"Explain how a hash table works.","sampling_params":{"temperature":0,"max_new_tokens":128}}'-```+## More-The first request compiles kernels and seeds the cost tables, so repeat it before timing.--The other engines have their own recipes. [vLLM](docs/quickstart-vllm.md) uses a static tree whose depth stops on-calibrated costs, produced once with `eagle-harness calibrate --engine vllm --model "$MODEL" --draft "$DRAFT" --tp 4`-while the GPUs are idle, plus the shipped rounds file. [SGLang on MI355X](docs/quickstart-amd.md) selects the-packaged MI355X cost profile automatically. [TokenSpeed](docs/quickstart-tokenspeed.md) is experimental and-CUDA-only, with a fixed-shape mode for checking generation without cost tables. The-[calibration guide](docs/calibration.md) explains the measurement, and [new-model onboarding](docs/new-model.md)-covers training and exporting a draft for another target.--## What it supports--- Drafts: EAGLE-4 drafts with the low-rank candidate head, tested with Qwen3.8-27B including its GDN layers.- Mamba2 targets and latent-MoE drafts have adapters on SGLang and vLLM. Per-model correctness coverage is in the- support matrix.-- Engines: SGLang (V2 speculative worker), vLLM (Model Runner V2) and experimental TokenSpeed, each through the- eagle-harness hooks. On SGLang and vLLM a draft top-k of 1 gives a chain and a larger top-k gives a tree.- TokenSpeed implements static and calibrated trees, chains and sampled trees, without prompt logprobs or native- no-speculation rounds, and its coverage is limited to the configurations in the support matrix.-- GPUs: NVIDIA sm_80 and newer, and AMD CDNA2 to CDNA4, through Triton kernels. The kernel tests run on B300- and the other architectures are compile-checked. MI355X is covered by the serving measurements in the support- matrix. See [kernels](docs/kernels.md).-- Requests: text only. vLLM tree verification handles greedy and sampled requests and needs an unquantized KV- cache. Constrained output takes the correctness fallback in [compatibility](docs/compatibility.md).-- Batch: SGLang and vLLM trees and chains admit up to 512 running requests within the engine's KV and- recurrent-state capacity. The SGLang tree default is 32, so set `--max-running-requests` for larger batches.- No pipeline parallelism. Admission is not a throughput or correctness guarantee. Check the- [support matrix](docs/support-matrix.md) and the [known limits](docs/known-limits.md).--## Documentation--[Architecture](docs/architecture.md) · [Configuration](docs/configuration.md) · [Calibration](docs/calibration.md) ·-[Kernels](docs/kernels.md) · [Compatibility](docs/compatibility.md) · [Troubleshooting](docs/troubleshooting.md) ·-[FAQ](docs/faq.md) · [Release notes](docs/release-notes/0.1.0.md) · [Contributing](CONTRIBUTING.md) ·-[Security](SECURITY.md)+- [Install](docs/install.md): pins, wheel indexes, Docker images and the ROCm closure.+- Engine guides with the measured recipes: [SGLang](docs/quickstart-sglang.md), [vLLM](docs/quickstart-vllm.md),+ [TokenSpeed, experimental](docs/quickstart-tokenspeed.md), [AMD MI355X](docs/quickstart-amd.md).+- [Support matrix](docs/support-matrix.md) and [known limits](docs/known-limits.md).+- [Calibration](docs/calibration.md), [configuration](docs/configuration.md) and+ [troubleshooting](docs/troubleshooting.md).+- [How it works](docs/architecture.md), [kernels](docs/kernels.md), [compatibility](docs/compatibility.md),+ [new models](docs/new-model.md) and the [FAQ](docs/faq.md).## Citationdiff --git a/docs/README.md b/docs/README.mdindex 2b784f8..6ef7382 100644--- a/docs/README.md+++ b/docs/README.md@@ -1,30 +1,29 @@---type: explanationowner: eagle-harness-last_verified: 2026-10-19+last_verified: 2026-10-21----# eagle-harness documentation+# EAGLE-4 documentation-Install a pinned engine and check generation with live costs on SGLang or calibrated costs and the shipped-batch-dependent tree schedule on vLLM.-TokenSpeed is experimental. Measure the setup you will deploy.-Start with the measured model pairs rather than treating config admission as proof of support.+To serve a model with EAGLE-4, install one engine and follow its guide. SGLang needs no preparation, vLLM+reaches its measured speed after a one-time calibration, and TokenSpeed is experimental.| Task | Guide ||---|---|-| Install wheels or run a release Docker image | [Install](install.md) |-| Generate text on SGLang CUDA | [SGLang quickstart](quickstart-sglang.md) |-| Generate text on AMD MI355X | [AMD quickstart](quickstart-amd.md) |-| Generate text on experimental TokenSpeed | [TokenSpeed quickstart](quickstart-tokenspeed.md) |-| Generate text on vLLM | [vLLM quickstart](quickstart-vllm.md) |-| Measure and fit serving costs | [Calibration](calibration.md) |-| Train, export and qualify a draft for another target | [New-model onboarding](new-model.md) |-| Check measured models, TP sizes and request features | [Support matrix](support-matrix.md) |+| Install an engine with pip, or pick a Docker image | [Install](install.md) |+| Serve on SGLang | [SGLang guide](quickstart-sglang.md) |+| Serve on vLLM | [vLLM guide](quickstart-vllm.md) |+| Serve on TokenSpeed, experimental | [TokenSpeed guide](quickstart-tokenspeed.md) |+| Serve on AMD MI355X | [AMD guide](quickstart-amd.md) |+| See measured speed and correctness | [Support matrix](support-matrix.md) || Check unsupported or unmeasured behavior | [Known limits](known-limits.md) |+| Measure and fit serving costs | [Calibration](calibration.md) || Look up config keys and defaults | [Configuration](configuration.md) |+| Diagnose install, startup and generation errors | [Troubleshooting](troubleshooting.md) |+| Train, export and qualify a draft for another target | [New-model onboarding](new-model.md) || Understand plugin hooks and decode rounds | [Architecture](architecture.md) || Check kernel implementations and fallbacks | [Kernels](kernels.md) || Check host versions and release numbering | [Compatibility](compatibility.md) |-| Diagnose startup and generation errors | [Troubleshooting](troubleshooting.md) |+| Convert an existing EAGLE-4 configuration | [Migration](migration.md) || Find short answers | [FAQ](faq.md) |diff --git a/docs/install.md b/docs/install.mdindex ad6be74..0435913 100644--- a/docs/install.md+++ b/docs/install.md@@ -1,93 +1,96 @@---type: how-toowner: eagle-harness-last_verified: 2026-10-20+last_verified: 2026-10-21----# Install EAGLE-4 0.1.1+# Install EAGLE-4-Use one patched engine per environment. Every path installs from public package indexes:-`eagle-harness==0.1.1` from PyPI and the patched engine from the EAGLE-4 wheel index.-Stock engine packages do not provide the required plugin hooks. Choose the pip or Docker path below,-then follow that engine's quickstart.+To install EAGLE-4, create a Python 3.12 environment and run the one `pip install` command for your engine.+Each command installs `eagle-harness` from PyPI and the patched engine from the EAGLE-4 wheel index, because the+stock engine packages do not have the hooks EAGLE-4 needs. Use one environment per engine. If you prefer+containers, use a [published image](#docker-images) instead.-## Release locations and availability--[Release settings](release.env) is the single source for package pins, wheel indexes and container tags.-The engine wheels belong to the GitHub release named by `EAGLE_RELEASE_ASSETS` (v0.1.0: the engines did not-change in 0.1.1). Its PEP 503 indexes-separate CUDA and ROCm wheels. The `eagle-harness` wheels and source distribution are published only to PyPI,-not to the GitHub Release. Publishing the public release runs `publish.yml` with PyPI trusted publishing.+## Requirements-Fresh CUDA environments installed the engine wheels and served real requests, and the `eagle-harness`-wheel and source-distribution forms passed installation checks. Those checks used a private index.-Fresh installs of the SGLang and TokenSpeed CUDA wheels from the public index, with `eagle-harness` from PyPI,-later served the measured results in the [support matrix](support-matrix.md). The published SGLang and vLLM-images passed serving and correctness checks, and the published TokenSpeed image passed a GPU check in serve-mode. The ROCm pip path below was clean-installed from the-public indexes and inference-tested on an MI355X. No ROCm image is published, and its Docker recipe has not been-container-qualified.-Do not replace an unavailable artifact with the stock engine or an unpinned package.+- Linux x86-64 and Python 3.12. On Ubuntu, `python3.12 -m venv` needs the `python3.12-venv` package.+- NVIDIA GPUs with a CUDA 13 driver. The measured recipes use B300 GPUs.+- For pip installs, a CUDA toolkit with `nvcc` and a C++ compiler for kernels compiled on the first start.+ The driver alone is not enough. On Ubuntu 26.04 / glibc 2.43, use CUDA 13.2+ ([toolkit setup](troubleshooting.md#install-problems)). Docker images include their build tools.+- For AMD MI355X, see [SGLang ROCm on AMD](#sglang-rocm-on-amd), which uses Python 3.10.-Clone the release checkout containing these docs and read its settings:+## SGLang CUDA```bash-git clone https://github.com/eagle-hq/EAGLE-4-. EAGLE-4/docs/release.env+python3.12 -m venv .venv && source .venv/bin/activate+pip install "eagle-harness[sglang]" "sglang==0.5.23.dev252+eagle.5" \+ --extra-index-url https://eagle-hq.github.io/EAGLE-4/whl/cu130/ \+ --extra-index-url https://download.pytorch.org/whl/cu130 \+ --extra-index-url https://pypi.nvidia.com```-Run every command in this guide and in the quickstarts from the directory that contains the checkout,-not from inside it. `python -m` puts the current directory first on the import path, so a server started-inside the checkout imports the source tree's `eagle_harness/` instead of the installed wheel.-`eagle-harness-check` cannot detect that, because a console script does not import from the current directory.+Then follow the [SGLang guide](quickstart-sglang.md).-## Requirements+## vLLM CUDA-- Linux x86-64. Use Python 3.12 for CUDA engines and Python 3.10 for the ROCm wheel.-- NVIDIA wheels target CUDA 13.0 and require a compatible host driver. The serving recipes use B300 GPUs.- Runtime compilation also needs a CUDA toolkit compatible with the host C/C++ headers.- On the tested Ubuntu 26.04 / glibc 2.43 host, CUDA 13.0 `nvcc` fails with `rsqrt`/`rsqrtf`- exception-specification errors. Use `export CUDA_HOME=/usr/local/cuda-13.2` for that host's JIT- compilation, or use the release container's compatible toolchain. This does not change the wheel index.-- SGLang ROCm requires Ubuntu 22.04, the ROCm 7.2 developer toolkit (`hipcc` and headers),- a C++ compiler, Git, and an MI355X with a compatible host driver.- Install the hash-locked ROCm closure below. AMD's PyTorch build for ROCm 7.2 and the- engine wheel use the CPython 3.10 ABI. Python 3.12 is not a supported substitute.-- Docker needs NVIDIA GPU access for CUDA, or `/dev/kfd` and `/dev/dri` access for ROCm.-- Obtain readable target and compatible draft weights before serving. The quickstarts use- Qwen3.8-27B BF16 and its EAGLE-4 draft. Check the [support matrix](support-matrix.md) for other pairs.+```bash+python3.12 -m venv .venv && source .venv/bin/activate+pip install "eagle-harness[vllm]" "vllm==0.32.0rc1.dev283+eagle.4" \+ --extra-index-url https://eagle-hq.github.io/EAGLE-4/whl/cu130/ \+ --extra-index-url https://download.pytorch.org/whl/cu130+```-## SGLang CUDA+Then follow the [vLLM guide](quickstart-vllm.md).-Create a Python 3.12 environment and install the pinned CUDA engine:+## TokenSpeed CUDA, experimental```bash-python3.12 -m venv .venv-sglang-source .venv-sglang/bin/activate-python -m pip install --upgrade pip-python -m pip install --extra-index-url "$EAGLE_CUDA_INDEX" \- --extra-index-url "$EAGLE_TORCH_CUDA_INDEX" --extra-index-url "$EAGLE_NVIDIA_INDEX" \- "sglang==$EAGLE_SGLANG_VERSION" 'flashinfer-python>=0.7.2'-python -m pip install "eagle-harness==$EAGLE_HARNESS_VERSION"-eagle-harness-check+python3.12 -m venv .venv && source .venv/bin/activate+pip install "eagle-harness[tokenspeed]" "tokenspeed==0.1.1+eagle.5" \+ "tokenspeed-scheduler==0.1.40+eagle.4" "tokenspeed-kernel==0.1.4+eagle.5" \+ --extra-index-url https://eagle-hq.github.io/EAGLE-4/whl/cu130/ \+ --extra-index-url https://download.pytorch.org/whl/cu130```-The NVIDIA index supplies the pinned `cuda-tile==1.6.0rc5` wheel. It avoids relying on a source build-from PyPI. The private install check also succeeded with that exact dependency built from its PyPI source.+TokenSpeed runs on NVIDIA GPUs only. Then follow the [TokenSpeed guide](quickstart-tokenspeed.md).-Or open the published image and check the hooks. The 0.1.0 images carry `eagle-harness` 0.1.0; upgrade it-inside the container:+## What the indexes and pins are for-```bash-docker run --rm --gpus all --ipc host -it \- -p 127.0.0.1:30000:30000 -v "$PWD/models:/models:ro" \- --entrypoint bash "$EAGLE_SGLANG_IMAGE"-python -m pip install -U eagle-harness==0.1.1-eagle-harness-check-```+| Index or pin | Why |+| --- | --- |+| `https://eagle-hq.github.io/EAGLE-4/whl/cu130/` | The patched SGLang, vLLM and TokenSpeed wheels. They carry a `+eagle.N` local version, so pip picks them only when the command pins that exact version. |+| `https://download.pytorch.org/whl/cu130` | PyTorch built for CUDA 13.0, which the engine wheels need. |+| `https://pypi.nvidia.com` | The `cuda-tile==1.6.0rc5` wheel that the SGLang engine pins. Without it pip builds that package from source. |+| `tokenspeed-scheduler`, `tokenspeed-kernel` | TokenSpeed's native packages, built from the same source as the engine. The stock wheels do not work with it. |++The engine wheels are assets of the [v0.1.0 GitHub release](https://github.com/eagle-hq/EAGLE-4/releases/tag/v0.1.0),+and the index pages link to them with SHA-256 hashes. Their sources are [eagle-hq/sglang](https://github.com/eagle-hq/sglang),+[eagle-hq/vllm](https://github.com/eagle-hq/vllm) and [eagle-hq/tokenspeed](https://github.com/eagle-hq/tokenspeed).+`eagle-harness` itself is published only on PyPI.+[docs/release.env](release.env) holds the same pins and URLs for scripts and image builds.+[docker/sglang/requirements.lock](../docker/sglang/requirements.lock) pins the SGLang CUDA runtime dependencies by+wheel hash for the SGLang image build.-Inside the container, use `/models` paths and `--host 0.0.0.0` in the-[SGLang CUDA quickstart](quickstart-sglang.md). The host port stays bound to loopback.+## Docker images++Each image has one patched engine and `eagle-harness` 0.1.0 installed. The engine guides show the `docker run`+command for each one.++| Engine | Image |+| --- | --- |+| SGLang CUDA | `ghcr.io/eagle-hq/eagle-4-sglang:0.1.0` |+| vLLM CUDA | `ghcr.io/eagle-hq/eagle-4-vllm:0.1.0` |+| TokenSpeed CUDA, experimental | `ghcr.io/eagle-hq/eagle-4-tokenspeed:0.1.0` |+| SGLang ROCm | not published, [build it](#build-the-rocm-image) |++To run a newer `eagle-harness` in an image, run `pip install -U eagle-harness` inside the container.++## Check the installation++`eagle-harness-check` exits zero when the installed engine has every hook EAGLE-4 needs. It does not start a+server, so a served request is still the real test. The [troubleshooting guide](troubleshooting.md) explains+its output and the startup errors.## SGLang ROCm on AMD@@ -104,10 +107,15 @@ The wheels come from public indexes:| Triton `3.7.0+amd.rocm7.2.0.git89002410` | AMD's Triton index, `EAGLE_AMD_TRITON_INDEX` || Torchvision `0.24.0+rocm7.2.0.gitb919bd0c` | AMD's ROCm 7.2 wheel directory, `EAGLE_AMD_ROCM_WHEELS` |-Do not use `--no-deps`, a source-overlay `PYTHONPATH`, or `--system-site-packages`.-On the ROCm host, from the directory that contains the checkout:+The host needs Ubuntu 22.04, the ROCm 7.2 developer toolkit (`hipcc` and headers), a C++ compiler, Git, and an+MI355X with a compatible host driver. AMD's PyTorch build for ROCm 7.2 and the engine wheel use the CPython 3.10+ABI, so Python 3.12 is not a substitute. Do not use `--no-deps`, a source-overlay `PYTHONPATH`, or+`--system-site-packages`. Clone the repository for the lock file and the settings, then install from the+directory that contains the checkout:```bash+git clone https://github.com/eagle-hq/EAGLE-4+. EAGLE-4/docs/release.envunset PYTHONPATHexport PATH="/opt/rocm/bin:$PATH"export LD_LIBRARY_PATH="/opt/rocm/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"@@ -127,7 +135,7 @@ the HIP `sglang-kernel==0.4.7` for `gfx950`. Three of these are not on their ventested form, so the EAGLE-4 index serves them. Each is selected by its hash, not just its version:- Torch, build tag `1eagle`, is AMD's ROCm 7.2 wheel with one dependency corrected. AMD's metadata pins- Triton 3.5.1, while AITER needs Triton 3.6 or newer. The rebuilt metadata pins the Triton build used+ Triton 3.5.1 and AITER uses Triton 3.6 or newer, so the rebuilt metadata pins the Triton build usedby the upstream SGLang MI35x image instead. Only the metadata, `WHEEL` and `RECORD` files differ fromAMD's wheel. Every runtime file and its license files are unchanged.- `sglang-kernel`, build tag `1eagle`, is the HIP build of SGLang's kernel package for `gfx950`.@@ -137,13 +145,14 @@ tested form, so the EAGLE-4 index serves them. Each is selected by its hash, notupstream SGLang MI35x image uses. AMD publishes no wheel of that build.[Third-party notices](../THIRD_PARTY_NOTICES.md#redistributed-rocm-wheels) lists their licenses and changes.+Then follow the [AMD guide](quickstart-amd.md). Do not install CUDA FlashInfer packages into this environment.### Build the ROCm imageThe Dockerfile uses the immutable `lmsysorg/sglang:v0.5.20-rocm720-mi35x` base and installsthis same closure in an isolated Python 3.10 environment. It does not reuse the base image's-editable SGLang installation. Container construction and serving must be qualified separately-before a release image is advertised as tested.+editable SGLang installation. The ROCm image is being container-qualified and is not published yet+([known limits](known-limits.md#distribution)).Download the locked wheels with the Python 3.10 environment above, then build with Docker BuildKit:@@ -161,106 +170,4 @@ python -m pip checkeagle-harness-check```-Follow the [AMD quickstart](quickstart-amd.md). Inside a container, use `/models` paths and-`--host 0.0.0.0`. Do not install CUDA FlashInfer packages into this environment.--## vLLM CUDA--Use a separate environment:--```bash-python3.12 -m venv .venv-vllm-source .venv-vllm/bin/activate-python -m pip install --upgrade pip-python -m pip install --extra-index-url "$EAGLE_CUDA_INDEX" \- --extra-index-url "$EAGLE_TORCH_CUDA_INDEX" "vllm==$EAGLE_VLLM_VERSION"-python -m pip install "eagle-harness==$EAGLE_HARNESS_VERSION"-eagle-harness-check-```--Or use the published vLLM image, which passed serving and correctness checks. It carries `eagle-harness` 0.1.0;-upgrade it inside the container. Mount a writable directory for the calibration output:--```bash-mkdir -p calibration-docker run --rm --gpus all --ipc host -it \- -p 127.0.0.1:8000:8000 -v "$PWD/models:/models:ro" \- -v "$PWD/calibration:/calibration" --entrypoint bash "$EAGLE_VLLM_IMAGE"-python -m pip install -U eagle-harness==0.1.1-eagle-harness-check-```--Follow the [vLLM quickstart](quickstart-vllm.md). Inside the container, use `/models` paths,-place calibration output under `/calibration`, and bind the server to `0.0.0.0`.--## TokenSpeed CUDA, experimental--TokenSpeed requires the matching engine, scheduler and kernel wheels pinned in `release.env`.-Install all three from the CUDA release index. Stock scheduler and kernel wheels are not substitutes.-Tree-round CUDA sources compile at first use even though the runtime wheel has a `py3-none-any` tag.-The passing serve-mode check uses a host-compatible CUDA 13.2 JIT on Ubuntu 26.04 / glibc 2.43,-with `CUDA_HOME=/usr/local/cuda-13.2`; the original CUDA 13.0 JIT fails on that host. This does not-change the `cu130` wheel channel. The scheduler and kernel wheels contain native code:--```bash-python3.12 -m venv .venv-tokenspeed-source .venv-tokenspeed/bin/activate-python -m pip install --upgrade pip-: "${EAGLE_TOKENSPEED_AUX_REQUIREMENTS:?release native-wheel pins are required}"-# Intentional word splitting: this variable contains pinned package requirements.-python -m pip install --extra-index-url "$EAGLE_CUDA_INDEX" \- --extra-index-url "$EAGLE_TORCH_CUDA_INDEX" \- "tokenspeed==$EAGLE_TOKENSPEED_VERSION" $EAGLE_TOKENSPEED_AUX_REQUIREMENTS-python -m pip install "eagle-harness==$EAGLE_HARNESS_VERSION"-eagle-harness-check-```--Or use the published TokenSpeed image, which passed a GPU check in serve mode. It carries `eagle-harness` 0.1.0;-upgrade it inside the container:--```bash-docker run --rm --gpus all --ipc host -it \- -p 127.0.0.1:30000:30000 -v "$PWD/models:/models:ro" \- --entrypoint bash "$EAGLE_TOKENSPEED_IMAGE"-python -m pip install -U eagle-harness==0.1.1-eagle-harness-check-```--Follow the [TokenSpeed quickstart](quickstart-tokenspeed.md). Inside the container, use `/models` paths-and bind to `0.0.0.0`. This path is NVIDIA-only, not a ROCm recipe.--## Check the installation--`eagle-harness-check` must exit zero and report the installed engine's hook checks. A failure means the-engine does not provide the expected contract. Fix that before loading weights. This check does not-prove GPU kernels work or generation is correct. The quickstart's generated response is a separate check.--From the directory where you start servers, confirm that Python imports the installed package:--```bash-python -c 'import eagle_harness; print(eagle_harness.__file__)'-```--The printed path must be inside the environment's `site-packages`. A path inside the checkout means-the server would run the source tree instead of the installed wheel.--For the SGLang and vLLM calibration CLI, also run:--```bash-eagle-harness calibrate --help-```--Do not install multiple engines in one environment. Use only the trusted release indexes above.-The [troubleshooting guide](troubleshooting.md) covers hook failures and startup errors.--## Dockerfile build status--The files under `docker/` describe release image construction, not proof of tested image builds.-The private CUDA image checks used a prebuilt `eagle-harness` wheel instead of the sketches' source install,-so they do not qualify those Dockerfiles verbatim. The corrected SGLang TP4 image passed hook, dependency,-request and twenty-prompt correctness checks. The published vLLM image passed the same checks, the TokenSpeed-image passed a GPU check in serve mode, and ROCm has no published image or container qualification.-The ROCm recipe pins its upstream base by digest and takes the downloaded locked wheels as a named-build context. The TokenSpeed sketch requires an explicit compatible `BASE_IMAGE`.-Do not infer container qualification from a successful pip installation or syntax check.-Final rebuilds and publication under the release tags are separate release steps.+Inside a container, use `/models` paths and `--host 0.0.0.0`.diff --git a/docs/known-limits.md b/docs/known-limits.mdindex 4a992ee..297778f 100644--- a/docs/known-limits.md+++ b/docs/known-limits.md@@ -7,44 +7,44 @@ last_verified: 2026-10-20# Known limitsUse the [support matrix](support-matrix.md) for measured model, hardware and workload combinations.-A loaded model, an admitted request cap or a compiled kernel does not establish serving correctness or speed.+Correctness and speed claims come from those measured runs. A loaded model, an admitted request cap or a compiled+kernel is a startup check.## Models and hardware- EAGLE-4 drafts need the trained candidate head and a supported draft structure.- Dense and latent-MoE EAGLE-4 drafts and Mamba2 targets have adapters. Nemotron's SGLang comparison has- unresolved first divergences under the fp32 gate. vLLM and experimental TokenSpeed do not have a complete- release-stack Nemotron performance and correctness result. Do not treat architecture admission as qualification.+ first divergences under the fp32 gate that are being resolved. Release-stack Nemotron performance and correctness+ results for vLLM and experimental TokenSpeed are in progress. An architecture that loads is not yet qualified.- Serving measurements cover NVIDIA B300 and SGLang on AMD MI355X. Compilation on another NVIDIA or AMD- architecture is separate evidence. Multi-node deployment and pipeline parallelism are not qualified.+ architecture is separate evidence. Multi-node deployment and pipeline parallelism are being qualified.- Throughput depends on the model, draft, engine, precision, TP size, attention backend, context length andworkload. A result at one concurrency does not establish a result at another.-## Performance and correctness gaps--- SGLang TP4 single-request and 32K-context optimization remain open. TP8 concurrency-8 prefill- overhead and intermittent stalls remain unresolved. Repeated passes in one process are not- independent server starts.-- SGLang automatic high-load policy and high-load fixed-chain improvements are not qualified release defaults.- High-load and Nemotron first divergences remain open. A release-engine cap-512- startup and request smoke passed, but it did not establish high-load acceleration or concurrent correctness.-- SGLang native-OFF, including OFF forced by a cost profile, needs the engine's per-variant forward-mode- decode graphs (the `forward_mode` field of the decode graph variant). The 0.1.0 SGLang CUDA engine lacks- them, so EAGLE-4 refuses native-OFF on that engine.+## Performance and correctness in progress++- SGLang TP4 single-request and 32K-context optimization is in progress, as is TP8 concurrency-8 prefill+ overhead and intermittent stalls. Repeated passes in one process are not independent server starts.+- SGLang automatic high-load policy and high-load fixed-chain improvements are being qualified and stay off by+ default, as does work on high-load and Nemotron first divergences. A release-engine cap-512 startup and+ request smoke passes; high-load acceleration and concurrent correctness are being measured.+- SGLang native-OFF, including OFF forced by a cost profile, uses the engine's per-variant forward-mode+ decode graphs (the `forward_mode` field of the decode graph variant). They arrive with the next SGLang+ engine release; on the 0.1.0 SGLang CUDA engine EAGLE-4 keeps native-OFF off.- The SGLang temperature-1 sampled tree (draft proposals drawn in one fused launch and verified against their- recorded probabilities) needs the engine's `sampling_step` verify hook. The 0.1.0 SGLang engines lack it, so- on them temperature-1 requests keep the 0.1.0 behaviour: a greedy draft verified by the target's sampler.+ recorded probabilities) uses the engine's `sampling_step` verify hook, which arrives with the next SGLang+ engine release. On the 0.1.0 SGLang engines temperature-1 requests use a greedy draft verified by the+ target's sampler.On every engine, a batch uses that path when any request has another temperature, a grammar or a penalty.- The shipped vLLM TP4 rounds schedule improves the measured concurrency-64 and concurrency-128 cases at arunning cap of 64, at temperature 0 and 1; see the[support matrix](support-matrix.md#measured-high-load-tree-schedule). The schedule has no TP1/TP8 profile.- A running cap of 128 and higher concurrency are not qualified. Native-OFF is disabled by default and does- not reach native-decode throughput in the measured TP1 cases.-- AMD measurements apply to the MI355X TP1 Qwen3.8 profile. At concurrency 64 on one MI355X, a short- diagnostic on the release stack measured EAGLE-4 at 2703.6 tokens/s against 2975.4 tokens/s for no-spec- decoding, so speculation is slower than no-spec there. HIP side-stream prelaunch stays disabled: its replay check now passes, but it- measured no throughput gain on MI355X. Remaining target, draft and verify phase overheads are separate- from the measured-profile throughput results.+ Running caps of 128 and above are being qualified. Native-OFF is off by default while its TP1 throughput is+ optimized to match native decoding.+- AMD measurements apply to the MI355X TP1 Qwen3.8 profile. On one MI355X the speedup covers concurrency up+ to 32; concurrency 64 is being optimized, so serve it without speculation for now. HIP side-stream prelaunch+ stays off: its replay check passes and it measured no throughput gain on MI355X. Further target, draft and+ verify phase optimization is in progress.## Requests@@ -52,29 +52,29 @@ A loaded model, an admitted request cap or a compiled kernel does not establishbenchmark results. A cap of 512 is an admission limit, not a claim of acceleration or correctness at that load.- SGLang supports accepted-token and input logprobs with the required host hooks. Images are refused.- vLLM tree verification needs an unquantized KV cache. The target's sampling restrictions still apply.- Sampling smoke tests do not prove distribution equivalence on every workload. The vLLM sampled-distribution- check passes only for temperature 1 with top-p 1 and no top-k; other temperatures, top-k/top-p and- penalties are not qualified. vLLM refuses `min_p` and `logit_bias` with speculative decoding.+ Sampling smoke tests check the path, not distribution equivalence on every workload. The vLLM+ sampled-distribution check covers temperature 1 with top-p 1 and no top-k; other temperatures, top-k/top-p+ and penalties are being qualified. vLLM refuses `min_p` and `logit_bias` with speculative decoding.- Structured output and tool calls require the model's engine parser. Raw tool syntax in text is not a structured- response. A vLLM tree batch containing a grammar constraint emits only the root token per verify round,- including unconstrained requests in that batch. Check finish reasons and validate the completed output.+ response. In a vLLM tree batch with a grammar constraint, every request in the batch advances one token per+ verify round. Check finish reasons and validate the completed output.See [compatibility](compatibility.md#structured-output-on-vllm).## TokenSpeed (experimental)TokenSpeed implements static and calibrated trees, chains and sampled trees. Its engine and adapter must beinstalled together with the matching hooks. Performance and correctness claims apply only to reported-configurations, not every implemented path. The sampled-output distribution is unqualified.+configurations, not every implemented path. The sampled-output distribution is being qualified.Prompt logprobs, per-request sampled-tree overrides, live cost measurement and native no-speculation rounds-are not supported. A zero-draft speculative round is not a native no-speculation round.+are not available on TokenSpeed yet. A zero-draft speculative round is not a native no-speculation round.Tree and chain cost data must match the served shape, TP size, target, draft and recurrent-state dtype.-TP4 single-request optimization remains open. Release-stack TP1/TP8, long contexts, four-GPU chains,-retraction under serving load and original-row output-logprob checks are incomplete. The reported sampled-cap-64 path is refused. A CUDA static-mode install smoke generated text but retained one fp32-gate failure.-The serve-mode install smoke passed its checked greedy prompts only with the host-compatible CUDA 13.2-JIT. The original CUDA 13.0 JIT fails on Ubuntu 26.04 / glibc 2.43. Static mode is not qualified.-These checks do not qualify arbitrary CUDA-toolkit and host-compiler combinations.+TP4 single-request optimization is in progress, as are release-stack TP1/TP8, long contexts, four-GPU chains,+retraction under serving load and original-row output-logprob checks. The reported sampled cap-64 path is+refused. The CUDA static-mode install smoke generates text with one first divergence above the fp32 gate, so+static mode is a setup check, not a qualified path. The serve-mode install smoke passes its greedy prompts with+the CUDA 13.2 JIT; on Ubuntu 26.04 / glibc 2.43 set `CUDA_HOME` to CUDA 13.2+([troubleshooting](troubleshooting.md#install-problems)). These checks cover the toolkits named here.## Calibration@@ -83,16 +83,27 @@ the model, draft, TP size, precision, attention backend or engine. Keyed tableskey does not encode every setting. A cache or table is not evidence of an optimal policy for every workload.The vLLM public sweep measures batch-one tree forwards, not complete high-concurrency serving rounds.-The SGLang served-chain driver measures a separate path from dynamic-tree calibration. The pinned SGLang-CUDA engine lacks its required per-variant native-decode graph contract, so the CLI path is unavailable-on that engine. Live-cost tree serving does not require it. Built-in smoke prompts-check the command, not representative acceptance. A failed run retains partial output for diagnosis and does-not write a successful `serve.sh`. See [calibration](calibration.md).+The SGLang served-chain driver measures a separate path from dynamic-tree calibration. Its CLI uses the+per-variant native-decode graphs that arrive with the next SGLang engine release; live-cost tree serving runs on+the current engine. Built-in smoke prompts check the command, not representative acceptance. An interrupted+run keeps its partial output for diagnosis and writes `serve.sh` only on success. See+[calibration](calibration.md).## DistributionThe [install guide](install.md) describes the release path: `eagle-harness` packages on PyPI,-patched engine wheels on the GitHub Pages indexes, and images on GHCR. These destinations do not establish-publication or image qualification. Private-index CUDA installation and request checks do not test the public-URLs. AMD has a qualified private offline pip closure, but its fresh-install commands remain deferred-to the AMD guide update, and its Docker image is unqualified. Model and draft weights require separate access.+patched engine wheels on the GitHub Pages indexes, and images on GHCR. Model and draft weights come from+Hugging Face under their owners' licenses.++- The published SGLang and vLLM images passed serving and correctness checks, and the published TokenSpeed image+ passed a GPU check in serve mode ([support matrix](support-matrix.md#artifact-installation-checks)). They carry+ `eagle-harness` 0.1.0.+- The files under `docker/` show how the release images are built. The checked CUDA images carry a prebuilt+ `eagle-harness` wheel, so the published images are the checked path. The TokenSpeed Dockerfile takes an+ explicit compatible `BASE_IMAGE`.+- On AMD, install with pip: the ROCm closure is clean-installed from the public indexes and inference-tested on+ an MI355X. The ROCm Dockerfile is being container-qualified, and no ROCm image is published yet.+- TokenSpeed serve mode uses cost tables measured for TokenSpeed on the served setup. EAGLE-4 does not ship them+ or a command that measures them yet, so the [TokenSpeed guide](quickstart-tokenspeed.md) starts in static+ mode as a setup check.+- Use the pinned artifacts: stock engines and unpinned packages do not carry the EAGLE-4 hooks.diff --git a/docs/migration.md b/docs/migration.mdindex 04602cf..3f92c85 100644--- a/docs/migration.md+++ b/docs/migration.md@@ -12,21 +12,22 @@ triples.or diagnostic output. The defaults below describe the legacy environment-variable configuration, not anenvironment parser.-## Removed unused fields+## Streamlined draft fields-Remove `draft_model.moe_top_k` and `draft_model.packed_candidate_gather` from old JSON.-Neither field affected execution; they are now rejected as unknown keys. Expert counts come-from the checkpoint. The unused internal `CostModel.ragged_recent_verify_nums` state is also removed.+Remove `draft_model.moe_top_k` and `draft_model.packed_candidate_gather` from old JSON; the loader rejects them+as unknown keys. Expert counts come from the checkpoint. The internal `CostModel.ragged_recent_verify_nums`+state is also removed.## Removed in 0.1.1-No engine adapter turned these options on, so they changed nothing. A config that still sets one fails at load-with `<option>: removed in eagle-harness 0.1.1`. Delete the option; use `"mode": "serve"` instead of `"eval"`.+Streamlined configuration: these options are removed, and serving behaves the same without them. A config that+still sets one stops at load with `<option>: removed in eagle-harness 0.1.1`. Delete the option; use+`"mode": "serve"` in place of `"eval"`.- `"mode": "eval"`- `ragged` (`enabled`, `force_uniform`)- `cost_model.ragged_draft_relax`, `cost_model.ragged_verify_threshold`-- `draft_model.moe_gate_acc` (it applied only to SwiGLU-MoE drafts, which every engine refuses)+- `draft_model.moe_gate_acc` (SwiGLU-MoE drafts)## Environment variables@@ -56,52 +57,52 @@ with `<option>: removed in eagle-harness 0.1.1`. Delete the option; use `"mode":| `SGLANG_EAGLE_DRAFT_EMBED_FULL` | `draft_model.embed_full` | `1` | Set the field || `SGLANG_EAGLE4_REPLICATED_DRAFT_EMBED` | `draft_model.embed_full` | `1 (deprecated alias)` | Set the field || `SGLANG_DISABLE_EAGLE_DRAFT_EMBED_TRITON` | `draft_model.embed_triton` | `0` | Invert the boolean |-| `SGLANG_DISABLE_EAGLE_HARNESS_VERIFY_ARGMAX` | — | `0` | Dropped |-| `SGLANG_DISABLE_EAGLE_HARNESS_SAMPLED_TREE` | — | `0` | Dropped |-| `SGLANG_DISABLE_FLASHINFER_VERIFY_EMPTY_OUT` | — | `0` | Dropped |-| `SGLANG_DISABLE_FLASHINFER_VERIFY_TAIL_MASK` | — | `0` | Dropped |-| `SGLANG_DISABLE_EAGLE_HARNESS_FUSED_VERIFY_PREP` | — | `0` | Dropped |-| `SGLANG_DISABLE_EAGLE_HARNESS_VERIFY_REPLAY_TRIMS` | — | `0` | Dropped |-| `SGLANG_DISABLE_EAGLE_HARNESS_FOLD_COMMIT_STEPS` | — | `0` | Dropped |-| `SGLANG_EAGLE_HARNESS_DRAFT_GRAPH_OPT` | — | `1` | Dropped |-| `SGLANG_EAGLE_HARNESS_DRAFT_STRIDED_Q` | — | `1` | Dropped |-| `SGLANG_EAGLE_HARNESS_DRAFT_ATTN_OUT` | — | `1` | Dropped |-| `SGLANG_EAGLE_HARNESS_DRAFT_NORM_HALVES` | — | `1` | Dropped |-| `SGLANG_EAGLE_HARNESS_DRAFT_GATHER_DOT_BK1` | — | `1` | Dropped |-| `SGLANG_EAGLE_HARNESS_EXTEND_LAST_ROW_GATHER` | — | `1` | Dropped |-| `SGLANG_EAGLE_HARNESS_EXTEND_FC_NORM_THIRDS` | — | `1` | Dropped |-| `SGLANG_EAGLE_HARNESS_EXTEND_FC_STRIDED_INPUT` | — | `1` | Dropped |-| `SGLANG_EAGLE_HARNESS_EXTEND_STRIDED_Q` | — | `1` | Dropped |-| `SGLANG_EAGLE_HARNESS_FUSED_STEP_UPDATE` | — | `1` | Dropped |-| `SGLANG_EAGLE_HARNESS_FUSED_STEP_ZERO_COPY` | — | `1` | Dropped |-| `SGLANG_EXACT_SOFTMAX` | — | `1` | Dropped |-| `SGLANG_EAGLE_HARNESS_SIDE_STREAM_VERIFY_SCORES` | — | `1` | Dropped |-| `SGLANG_EAGLE_HARNESS_REUSE_EXTEND_DRAFT_TOPK` | — | `1` | Dropped |-| `SGLANG_EAGLE_HARNESS_CHECK_EXTEND_DRAFT_TOPK` | — | `0` | Dropped |-| `SGLANG_EAGLE_HARNESS_FUSED_POST_VERIFY` | — | `1` | Dropped |-| `SGLANG_DISABLE_EAGLE_HARNESS_DRAFT_PRELAUNCH` | — | `0` | Dropped |-| `SGLANG_DISABLE_EAGLE_HARNESS_SIDE_STREAM_STATE_COMMIT` | — | `0` | Dropped |-| `SGLANG_DISABLE_EAGLE_HARNESS_LEAN_DRAFT_INIT` | — | `0` | Dropped |-| `SGLANG_DISABLE_EAGLE_HARNESS_LEAN_DRAFT_STEP` | — | `0` | Dropped |-| `SGLANG_DISABLE_EAGLE_HARNESS_SKIP_NOOP_MOE_CONTEXTS` | — | `0` | Dropped |-| `SGLANG_DISABLE_EAGLE_HARNESS_DRAFT_STEP_META` | — | `0` | Dropped |-| `SGLANG_DISABLE_EAGLE_HARNESS_DRAFT_PREP_ONE_LAUNCH` | — | `0` | Dropped |-| `SGLANG_EAGLE_HARNESS_VERIFY_PREALLOC` | — | `1` | Dropped |-| `SGLANG_EAGLE_HARNESS_CACHED_VERIFY_BATCH` | — | `1` | Dropped |-| `SGLANG_EAGLE_HARNESS_DEVICE_ACCEPT` | — | `1` | Dropped |-| `SGLANG_EAGLE_HARNESS_DEVICE_UPDATE_TREE` | — | `1` | Dropped |-| `SGLANG_OPT_GDN_TREE_GLUON` | — | `1` | Dropped |-| `SGLANG_OPT_GDN_TREE_GLUON_PDL` | — | `1` | Dropped |-| `SGLANG_OPT_GDN_TREE_GLUON_V2` | — | `1` | Dropped |-| `SGLANG_OPT_GDN_TREE_GLUON_V3` | — | `1` | Dropped |-| `SGLANG_DISABLE_GDN_TREE_VERIFY_IN_PLACE` | — | `0` | Dropped |-| `SGLANG_EAGLE_HARNESS_CHAIN_DEBUG` | — | `0` | Dropped |-| `SGLANG_EAGLE_HARNESS_CHAIN_HIST` | — | `unset` | Dropped |-| `SGLANG_EAGLE4_SWA_AUDIT` | — | `unset` | Dropped |-| `SGLANG_EAGLE4_SWA_AUDIT_DIR` | — | `unset` | Dropped |+| `SGLANG_DISABLE_EAGLE_HARNESS_VERIFY_ARGMAX` | — | `0` | Removed |+| `SGLANG_DISABLE_EAGLE_HARNESS_SAMPLED_TREE` | — | `0` | Removed |+| `SGLANG_DISABLE_FLASHINFER_VERIFY_EMPTY_OUT` | — | `0` | Removed |+| `SGLANG_DISABLE_FLASHINFER_VERIFY_TAIL_MASK` | — | `0` | Removed |+| `SGLANG_DISABLE_EAGLE_HARNESS_FUSED_VERIFY_PREP` | — | `0` | Removed |+| `SGLANG_DISABLE_EAGLE_HARNESS_VERIFY_REPLAY_TRIMS` | — | `0` | Removed |+| `SGLANG_DISABLE_EAGLE_HARNESS_FOLD_COMMIT_STEPS` | — | `0` | Removed |+| `SGLANG_EAGLE_HARNESS_DRAFT_GRAPH_OPT` | — | `1` | Removed |+| `SGLANG_EAGLE_HARNESS_DRAFT_STRIDED_Q` | — | `1` | Removed |+| `SGLANG_EAGLE_HARNESS_DRAFT_ATTN_OUT` | — | `1` | Removed |+| `SGLANG_EAGLE_HARNESS_DRAFT_NORM_HALVES` | — | `1` | Removed |+| `SGLANG_EAGLE_HARNESS_DRAFT_GATHER_DOT_BK1` | — | `1` | Removed |+| `SGLANG_EAGLE_HARNESS_EXTEND_LAST_ROW_GATHER` | — | `1` | Removed |+| `SGLANG_EAGLE_HARNESS_EXTEND_FC_NORM_THIRDS` | — | `1` | Removed |+| `SGLANG_EAGLE_HARNESS_EXTEND_FC_STRIDED_INPUT` | — | `1` | Removed |+| `SGLANG_EAGLE_HARNESS_EXTEND_STRIDED_Q` | — | `1` | Removed |+| `SGLANG_EAGLE_HARNESS_FUSED_STEP_UPDATE` | — | `1` | Removed |+| `SGLANG_EAGLE_HARNESS_FUSED_STEP_ZERO_COPY` | — | `1` | Removed |+| `SGLANG_EXACT_SOFTMAX` | — | `1` | Removed |+| `SGLANG_EAGLE_HARNESS_SIDE_STREAM_VERIFY_SCORES` | — | `1` | Removed |+| `SGLANG_EAGLE_HARNESS_REUSE_EXTEND_DRAFT_TOPK` | — | `1` | Removed |+| `SGLANG_EAGLE_HARNESS_CHECK_EXTEND_DRAFT_TOPK` | — | `0` | Removed |+| `SGLANG_EAGLE_HARNESS_FUSED_POST_VERIFY` | — | `1` | Removed |+| `SGLANG_DISABLE_EAGLE_HARNESS_DRAFT_PRELAUNCH` | — | `0` | Removed |+| `SGLANG_DISABLE_EAGLE_HARNESS_SIDE_STREAM_STATE_COMMIT` | — | `0` | Removed |+| `SGLANG_DISABLE_EAGLE_HARNESS_LEAN_DRAFT_INIT` | — | `0` | Removed |+| `SGLANG_DISABLE_EAGLE_HARNESS_LEAN_DRAFT_STEP` | — | `0` | Removed |+| `SGLANG_DISABLE_EAGLE_HARNESS_SKIP_NOOP_MOE_CONTEXTS` | — | `0` | Removed |+| `SGLANG_DISABLE_EAGLE_HARNESS_DRAFT_STEP_META` | — | `0` | Removed |+| `SGLANG_DISABLE_EAGLE_HARNESS_DRAFT_PREP_ONE_LAUNCH` | — | `0` | Removed |+| `SGLANG_EAGLE_HARNESS_VERIFY_PREALLOC` | — | `1` | Removed |+| `SGLANG_EAGLE_HARNESS_CACHED_VERIFY_BATCH` | — | `1` | Removed |+| `SGLANG_EAGLE_HARNESS_DEVICE_ACCEPT` | — | `1` | Removed |+| `SGLANG_EAGLE_HARNESS_DEVICE_UPDATE_TREE` | — | `1` | Removed |+| `SGLANG_OPT_GDN_TREE_GLUON` | — | `1` | Removed |+| `SGLANG_OPT_GDN_TREE_GLUON_PDL` | — | `1` | Removed |+| `SGLANG_OPT_GDN_TREE_GLUON_V2` | — | `1` | Removed |+| `SGLANG_OPT_GDN_TREE_GLUON_V3` | — | `1` | Removed |+| `SGLANG_DISABLE_GDN_TREE_VERIFY_IN_PLACE` | — | `0` | Removed |+| `SGLANG_EAGLE_HARNESS_CHAIN_DEBUG` | — | `0` | Removed |+| `SGLANG_EAGLE_HARNESS_CHAIN_HIST` | — | `unset` | Removed |+| `SGLANG_EAGLE4_SWA_AUDIT` | — | `unset` | Removed |+| `SGLANG_EAGLE4_SWA_AUDIT_DIR` | — | `unset` | Removed || `DRAFT_TREE_PATH` | `data_path` | `--speculative-data-path` | Set the field || `COMPUTE_INTENSITY` | `intensity` | `--speculative-compute-intensity` | Set the field |-| `DEBUG_STEP` | — | `0 (never read)` | Dropped |+| `DEBUG_STEP` | — | `0` | Removed |## Command-line argumentsdiff --git a/docs/quickstart-sglang.md b/docs/quickstart-sglang.mdindex 5e7f5a8..e97df72 100644--- a/docs/quickstart-sglang.md+++ b/docs/quickstart-sglang.md@@ -1,74 +1,85 @@---type: tutorialowner: eagle-harness-last_verified: 2026-10-20+last_verified: 2026-10-21----# SGLang CUDA quickstart+# SGLang guide-Serve Qwen3.8-27B BF16 with its EAGLE-4 draft on four B300 GPUs. SGLang measures the draft and-verification costs during served rounds, so this recipe needs no offline calibration directory.-For MI355X, use the separate [AMD quickstart](quickstart-amd.md).+Serve Qwen3.8-27B with its EAGLE-4 draft on four NVIDIA GPUs. SGLang prices the draft tree from costs it+measures on the served rounds, so nothing needs calibrating first. For MI355X, use the [AMD guide](quickstart-amd.md).-## Before starting+## Start a server-Complete the [SGLang CUDA installation](install.md#sglang-cuda) and run `eagle-harness-check`.-Set `MODEL` and `DRAFT` to readable local target and compatible draft directories.-The release recipe below uses TP4 and a running-request cap of 32. See the-[support matrix](support-matrix.md) before changing the model, hardware or tensor-parallel size.-Only enable `--trust-remote-code` for model code you have reviewed.+Install the engine with the [SGLang install command](install.md#sglang-cuda), then start the server. The model and+the draft download from Hugging Face on the first start:-## Start the server+```bash+sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 \+ --speculative-algorithm EAGLE_HARNESS \+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4+```++Or run the published image, which has the engine and EAGLE-4 installed:++```bash+docker run --gpus all --ipc host -p 127.0.0.1:30000:30000 \+ -e HF_HOME=/hf -v ~/.cache/huggingface:/hf \+ ghcr.io/eagle-hq/eagle-4-sglang:0.1.0 \+ sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 --host 0.0.0.0 \+ --disable-overlap-schedule --speculative-algorithm EAGLE_HARNESS \+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 \+ --speculative-num-steps 12 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \+ --speculative-algorithm-config '{}'+```++By default EAGLE-4 drafts a tree of up to 12 steps with 4 candidates per step, verifies up to 16 tokens, and+prices every round from live costs. `--speculative-num-steps`, `--speculative-eagle-topk`,+`--speculative-num-draft-tokens` and `--speculative-algorithm-config` override those+([configuration](configuration.md)). These defaults need `eagle-harness` 0.1.1 or newer. The image carries 0.1.0,+so its command spells them out and adds `--disable-overlap-schedule`. From 0.1.1, EAGLE-4 serves with SGLang's+default overlap scheduler.++## Send a request++When the server logs `The server is fired up and ready to roll!`, send a request from another terminal:++```bash+curl http://localhost:30000/v1/chat/completions -H 'Content-Type: application/json' \+ -d '{"model": "Qwen/Qwen3.8-27B", "messages": [{"role": "user", "content": "Explain how a hash table works."}], "max_tokens": 256}'+```++The response holds the generated message. The first requests compile kernels and seed the live costs, so send a+few before you measure speed. Compare against the same engine without the `--speculative-*` flags.++## The measured recipe-Use the release tree shape and keep CUDA graph padding enabled:+The SGLang rows of the [support matrix](support-matrix.md#sglang-on-nvidia-b300) were measured with this command.+It adds the attention backends, graph settings, memory sizes and recurrent-state dtype of the benchmark:```bash-: "${MODEL:?set the local Qwen3.8-27B BF16 target directory}"-: "${DRAFT:?set the local compatible EAGLE-4 draft directory}"-export HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=4+export TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=4export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 SGLANG_TIMEOUT_KEEP_ALIVE=3600-python -m sglang.launch_server --model-path "$MODEL" --tp-size 4 --dtype bfloat16 \+sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 --dtype bfloat16 \--trust-remote-code --attention-backend flashinfer --page-size 1 \--max-running-requests 32 --cuda-graph-max-bs-decode 32 \--disable-overlap-schedule --cuda-graph-backend-decode full \--mem-fraction-static 0.75 --context-length 196608 --max-total-tokens 196608 \--chunked-prefill-size 16384 --mamba-ssm-dtype bfloat16 --disable-radix-cache \--reasoning-parser qwen3 --random-seed 721 --skip-server-warmup \- --host 127.0.0.1 --port 30000 \- --speculative-algorithm EAGLE_HARNESS --speculative-draft-model-path "$DRAFT" \+ --speculative-algorithm EAGLE_HARNESS \+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 \--speculative-draft-model-quantization unquant --speculative-draft-attention-backend flashinfer \--speculative-attention-mode prefill --speculative-num-steps 12 \--speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \--speculative-algorithm-config '{"mode":"serve"}'```-Do not add `--disable-cuda-graph-padding` to this CUDA recipe. The release benchmark uses a separate-client warmup, hence `--skip-server-warmup`. With this exact command, the first generation below performs-cold compilation and cost seeding. Remove that flag if you want the engine's startup warmup instead,-but do not call the modified startup sequence an identical benchmark reproduction.--The shape flags bound the tree. Initial served rounds price captured verification widths and later-rounds refine their costs. The cache is keyed by GPU, TP size, model configurations, engine and dtype-under `~/.cache/eagle-harness/costs`. No private cost-table path is required.--## Generate text--In another terminal, wait until the model listing succeeds, then send a greedy request:--```bash-curl --fail http://127.0.0.1:30000/v1/models-curl --fail http://127.0.0.1:30000/generate \- -H 'Content-Type: application/json' \- -d '{"text":"Explain how a hash table works.","sampling_params":{"temperature":0,"max_new_tokens":128}}'-```--The response must contain generated text and token metadata. Repeat the generation request after-compilation before measuring latency. A successful model listing alone does not check decoding.-For Docker, bind the server to `0.0.0.0` inside the container and keep the published host port on loopback.--## Next step+The benchmark warmed the server from its client, hence `--skip-server-warmup`: the first request after this start+compiles kernels and seeds the costs. Keep CUDA graph padding enabled, and do not add+`--disable-cuda-graph-padding`. `--max-running-requests 32` is the tree default; set a larger cap, up to 512, for+larger batches.-Use representative prompts and compare against the same engine without speculation before deployment.-The [calibration guide](calibration.md) explains live costs and offline chain calibration.-The [known limits](known-limits.md) distinguish greedy checks, sampled requests, logprobs and multimodal support.-Stop the server with Ctrl-C before starting another server or calibration on the same GPUs.+Live costs are cached per GPU, TP size, model pair, engine and dtype under `~/.cache/eagle-harness/costs`.+The [calibration guide](calibration.md) explains live costs and offline chain calibration, and the+[known limits](known-limits.md) cover sampled requests, logprobs and multimodal inputs.diff --git a/docs/quickstart-tokenspeed.md b/docs/quickstart-tokenspeed.mdindex 9da76bc..7e9a794 100644--- a/docs/quickstart-tokenspeed.md+++ b/docs/quickstart-tokenspeed.md@@ -1,85 +1,66 @@---type: tutorialowner: eagle-harness-last_verified: 2026-10-20+last_verified: 2026-10-21----# TokenSpeed quickstart, experimental+# TokenSpeed guide, experimental-Run the experimental TokenSpeed adapter on NVIDIA CUDA hardware. This is not a ROCm path.-The release serving recipe uses Qwen3.8-27B BF16 and its EAGLE-4 draft on four B300 GPUs.+Serve Qwen3.8-27B with its EAGLE-4 draft on four NVIDIA GPUs with TokenSpeed. TokenSpeed support is experimental+and runs on NVIDIA GPUs only. Its measured cost-model mode needs cost tables that EAGLE-4 does not ship yet, so the+first server below uses a static tree to check that generation works.-## Before starting+## Start a server-Complete the [TokenSpeed installation](install.md#tokenspeed-cuda-experimental), including its scheduler-and kernel wheels, and run `eagle-harness-check`. Tree-round sources need `nvcc` at first use.-On the tested Ubuntu 26.04 / glibc 2.43 host, the CUDA 13.0 JIT fails. The passing serve-mode-check uses a host-compatible CUDA 13.2 JIT with `CUDA_HOME=/usr/local/cuda-13.2`. Set that path-to the installed compatible toolkit before launching. The engine wheels still come from `cu130`.-Set `MODEL` and `DRAFT` to readable local directories.-The measured cost-model recipe also requires matching calibrated target and draft tables under-`COST_ROOT/cost/`. EAGLE-4 does not include a TokenSpeed calibration CLI or automatically select-an installed TokenSpeed profile. Do not substitute vLLM or ROCm tables.--Use `mode=serve` with matching costs for the checked path. Its twenty-prompt install check passed-with 4 exact and 16 fp32-tolerance results. Static mode is not a qualified substitute when cost files-are unavailable: its check has one fp32 gap of 0.178637, above the 0.125 tolerance.-These are first-divergence checks, not bitwise output equality or sampled-distribution qualification.--## Start the server--With matching costs, construct the configuration using JSON encoding:+Install the engine with the [TokenSpeed install command](install.md#tokenspeed-cuda-experimental), then start a+server with a static tree:```bash-: "${COST_ROOT:?set the matching TokenSpeed TP4 calibrated cost directory}"-export COST_ROOT-STEPS=12-test -d "$COST_ROOT/cost"-CONFIG=$(python -c 'import json, os; print(json.dumps({"mode":"serve","data_path":os.environ["COST_ROOT"]}))')+tokenspeed serve Qwen/Qwen3.8-27B --tensor-parallel-size 4 \+ --attention-backend trtllm --max-model-len 65536 --sampling-backend greedy \+ --speculative-algorithm EAGLE_HARNESS \+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 \+ --speculative-num-steps 5 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \+ --speculative-algorithm-config '{"mode": "static"}'```-Use the release launch flags with the serve-mode configuration:+Static mode drafts the full tree every round, without a cost model. Its twenty-prompt greedy check has one+first divergence above the fp32 gate, so it checks setup rather than correctness+([known limits](known-limits.md#tokenspeed-experimental)). `--sampling-backend greedy` is required, because+draft trees are verified by the greedy and triton sampling backends.++## Send a request```bash-: "${MODEL:?set the local Qwen3.8-27B BF16 target directory}"-: "${DRAFT:?set the local compatible EAGLE-4 draft directory}"-: "${CONFIG:?construct the serve-mode configuration above}"-: "${STEPS:?set the serve-mode step bound above}"-export HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=4-python -m tokenspeed.cli serve "$MODEL" --tensor-parallel-size 4 \- --attention-backend trtllm --max-model-len 65536 --max-num-seqs 4 \- --disable-kvstore --seed 721 --gpu-memory-utilization 0.6 --disable-prefix-caching \- --host 127.0.0.1 --port 30000 --reasoning-parser qwen3 \- --speculative-draft-model-path "$DRAFT" --speculative-num-steps "$STEPS" \- --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \- --speculative-algorithm EAGLE_HARNESS --speculative-algorithm-config "$CONFIG" \- --sampling-backend greedy+curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \+ -d '{"model": "Qwen/Qwen3.8-27B", "messages": [{"role": "user", "content": "Explain how a hash table works."}], "max_tokens": 256}'```-The running-request cap is four in the single-client release recipe. The concurrent release recipe-uses `--max-num-seqs 32` with the same TP4 and context settings. Neither setting promises a speedup-for your workload. The published TokenSpeed image passed a GPU check in serve mode.-Inside the image, bind the server to `0.0.0.0` and keep the published host port on loopback.-If using cost files in Docker, mount their directory read-only and set `COST_ROOT` to that container path.+The response holds the generated message. The first start compiles tree-round CUDA sources, which needs `nvcc`.+On Ubuntu 26.04 / glibc 2.43, set `CUDA_HOME=/usr/local/cuda-13.2` first ([troubleshooting](troubleshooting.md#install-problems)).-## Generate text+## The measured recipe-In another terminal, read the served model identifier and send a greedy chat request:+The TokenSpeed rows of the [support matrix](support-matrix.md#tokenspeed-experimental) use serve mode, where the+cost model sizes every round from cost tables measured for TokenSpeed on B300 at TP4. Set `COST_ROOT` to a+directory whose `cost/` folder holds such tables. vLLM and ROCm tables are not substitutes.```bash-MODEL=$(curl --fail --silent http://127.0.0.1:30000/v1/models | python -c 'import json,sys; print(json.load(sys.stdin)["data"][0]["id"])')-export MODEL-python -c 'import json, os; print(json.dumps({"model":os.environ["MODEL"],"messages":[{"role":"user","content":"Explain how a hash table works."}],"temperature":0,"max_tokens":128}))' | \- curl --fail http://127.0.0.1:30000/v1/chat/completions \- -H 'Content-Type: application/json' --data-binary @-+export COST_ROOT=/path/to/tokenspeed-tp4-costs+tokenspeed serve Qwen/Qwen3.8-27B --tensor-parallel-size 4 \+ --attention-backend trtllm --max-model-len 65536 --max-num-seqs 32 \+ --disable-kvstore --seed 721 --gpu-memory-utilization 0.6 --disable-prefix-caching \+ --reasoning-parser qwen3 --sampling-backend greedy \+ --speculative-algorithm EAGLE_HARNESS \+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 \+ --speculative-num-steps 12 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \+ --speculative-algorithm-config "{\"mode\": \"serve\", \"data_path\": \"$COST_ROOT\"}"```-The response must contain a generated choice. A reasoning model may place its output in the reasoning-field rather than `message.content`. Repeat after warmup before timing.--## Experimental limits+The single-client recipe uses `--max-num-seqs 4` and the concurrent recipe `--max-num-seqs 32`. Neither setting+promises a speedup for your workload. The serve-mode check passed twenty greedy prompts with 4 exact and 16+fp32-tolerance results. These are first-divergence checks, not bitwise output equality or sampled-distribution+qualification.-Prompt logprobs, per-request sampled trees and native-OFF are not supported by this adapter.-This greedy quickstart does not enable sampled decoding.-See the [support matrix](support-matrix.md) for model-specific correctness and performance qualifications.-Stop the server with Ctrl-C before starting another engine on the same GPUs.+Prompt logprobs, per-request sampled trees, live cost measurement and native no-speculation rounds are not+supported. The [known limits](known-limits.md#tokenspeed-experimental) list the rest.diff --git a/docs/quickstart-vllm.md b/docs/quickstart-vllm.mdindex d589350..26606b0 100644--- a/docs/quickstart-vllm.md+++ b/docs/quickstart-vllm.md@@ -1,92 +1,83 @@---type: tutorialowner: eagle-harness-last_verified: 2026-10-20+last_verified: 2026-10-21----# vLLM quickstart+# vLLM guide-Serve Qwen3.8-27B BF16 with its EAGLE-4 draft on four B300 GPUs using the release tree configuration.-Load the shipped rounds file into matching calibrated costs to enable its batch-dependent verification width.-The rounds file is packaged, but the target and draft `cost/` tables must be measured separately.+Serve Qwen3.8-27B with its EAGLE-4 draft on four NVIDIA GPUs with vLLM. The first server uses a fixed tree and+needs no preparation. The measured recipe adds cost tables you calibrate once and a packaged schedule that+narrows the tree at high load.-## Before starting+## Start a server-Complete the [vLLM installation](install.md#vllm-cuda) and run `eagle-harness-check`.-Set `MODEL` and `DRAFT` to readable local directories. This recipe is specific to the Qwen3.8-27B / EAGLE-4-v3 pair at TP4. Do not reuse its rounds file for another GPU, tensor-parallel size or model pair.-See the [support matrix](support-matrix.md) for other configurations.+Install the engine with the [vLLM install command](install.md#vllm-cuda), then start the server. The model and the+draft download from Hugging Face on the first start:-## Prepare the costs and shipped rounds+```bash+VLLM_USE_V2_MODEL_RUNNER=1 vllm serve Qwen/Qwen3.8-27B --tensor-parallel-size 4 \+ --attention-backend CUSTOM --linear-backend flashinfer_cutedsl \+ --speculative-config '{"method": "eagle_harness", "model": "EAGLE-Corp/Qwen3.8-27B-EAGLE4", "method_config": {"mode": "static", "node_top_k": 4, "tree_depth": 14, "max_verify_width": 16}}'+```-If you already have matching calibrated TP4 costs, set `COST_ROOT` to that directory and skip calibration.-Otherwise, with the GPUs idle, create them with the installed CLI:+Or run the published image, which has the engine and EAGLE-4 installed and starts `vllm serve` itself:```bash-: "${MODEL:?set the local Qwen3.8-27B BF16 target directory}"-: "${DRAFT:?set the local compatible EAGLE-4 draft directory}"-export MODEL DRAFT-export COST_ROOT="$PWD/calibration-vllm-tp4"-eagle-harness calibrate --engine vllm --model "$MODEL" --draft "$DRAFT" --tp 4 \- --engine-args '{"linear_backend":"flashinfer_cutedsl"}' --output "$COST_ROOT"+docker run --gpus all --ipc host -p 127.0.0.1:8000:8000 \+ -e HF_HOME=/hf -v ~/.cache/huggingface:/hf -e VLLM_USE_V2_MODEL_RUNNER=1 \+ ghcr.io/eagle-hq/eagle-4-vllm:0.1.0 \+ Qwen/Qwen3.8-27B --tensor-parallel-size 4 \+ --attention-backend CUSTOM --linear-backend flashinfer_cutedsl \+ --speculative-config '{"method": "eagle_harness", "model": "EAGLE-Corp/Qwen3.8-27B-EAGLE4", "method_config": {"mode": "static", "node_top_k": 4, "tree_depth": 14, "max_verify_width": 16}}'```-Calibration refuses an existing output directory. It measures batch-one costs, not full high-concurrency-serving. New measurements are not the exact table bytes used by the release benchmark. In Docker, choose-a fresh directory beneath the writable `/calibration` mount instead of `$PWD`.+The tree drafts 14 steps with 4 candidates per step and verifies 16 tokens. Tree verification needs vLLM's Model+Runner V2 (`VLLM_USE_V2_MODEL_RUNNER=1`) and the `CUSTOM` attention backend, and an unquantized KV cache.-Copy the shipped measured rounds without replacing the calibrated `cost/` tables:+## Send a request++When the server logs `Application startup complete`, send a request from another terminal:```bash-: "${COST_ROOT:?set the matching calibrated TP4 cost directory}"-export COST_ROOT-TREE_ROUNDS=$(python -c 'from importlib.resources import files; print(files("eagle_harness.core").joinpath("profiles/vllm-qwen38-tp4/rounds/tree.json"))')-test -d "$COST_ROOT/cost"-mkdir -p "$COST_ROOT/rounds"-cp "$TREE_ROUNDS" "$COST_ROOT/rounds/tree.json"+curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \+ -d '{"model": "Qwen/Qwen3.8-27B", "messages": [{"role": "user", "content": "Explain how a hash table works."}], "max_tokens": 256}'```-This file keeps width 16 below batch size 20 and selects width 4 at larger batches. It does not enable-native-OFF. The [profile notes](../eagle_harness/core/profiles/vllm-qwen38-tp4/README.md) describe its scope.+The response holds the generated message. Send a few requests before you measure speed.++## The measured recipe-## Start the server+The TP4 vLLM rows of the [support matrix](support-matrix.md#vllm-on-nvidia-b300) stop the tree early when another+draft step costs more than it gains, from cost tables measured on your GPUs. At 20 or more running requests the+packaged rounds file narrows the verified tokens from 16 to 4+([high-load rows](support-matrix.md#measured-high-load-tree-schedule)). Both live in one directory.-Build the speculative configuration with JSON encoding so paths with spaces or quotes remain valid:+Stop any server on these GPUs, then calibrate once and add the rounds file:```bash-: "${MODEL:?set the target directory}"-: "${DRAFT:?set the draft directory}"-: "${COST_ROOT:?set the prepared cost directory}"-export MODEL DRAFT COST_ROOT-SPEC=$(python -c 'import json, os; print(json.dumps({"method":"eagle_harness","model":os.environ["DRAFT"],"method_config":{"mode":"static","node_top_k":4,"tree_depth":14,"max_verify_width":16,"data_path":os.environ["COST_ROOT"]}}))')-export VLLM_USE_V2_MODEL_RUNNER=1 HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=4-python -m vllm.entrypoints.cli.main serve "$MODEL" --tensor-parallel-size 4 \- --max-model-len 32768 --gpu-memory-utilization 0.85 --max-num-seqs 64 \- --seed 721 --host 127.0.0.1 --port 8000 --no-enable-prefix-caching \- --linear-backend flashinfer_cutedsl --reasoning-parser qwen3 \- --attention-backend CUSTOM --speculative-config "$SPEC"+eagle-harness calibrate --engine vllm --model Qwen/Qwen3.8-27B --draft EAGLE-Corp/Qwen3.8-27B-EAGLE4 --tp 4 \+ --engine-args '{"linear_backend":"flashinfer_cutedsl"}' --output calibration-vllm-tp4+mkdir -p calibration-vllm-tp4/rounds+python -c 'import shutil; from importlib.resources import files; shutil.copy(files("eagle_harness.core") / "profiles/vllm-qwen38-tp4/rounds/tree.json", "calibration-vllm-tp4/rounds/")'```-Model Runner V2 and `CUSTOM` attention are required for this tree. The static shape supplies the bounds,-the calibrated costs supply depth stopping, and the rounds file selects width by batch size.-Registration-time INFO messages can be dropped before vLLM configures logging, so an absent schedule-load-line does not prove the schedule is missing. Without `rounds/tree.json`, the configured width stays fixed.-Do not point `data_path` at the packaged-rounds-only directory because it lacks the depth-stop cost tables.--## Generate text+Calibration refuses an existing output directory. It measures batch-one costs, not full high-concurrency serving,+and new measurements are not the exact table bytes of the release benchmark. The rounds file is measured for this+model pair on B300 at TP4 only ([profile notes](../eagle_harness/core/profiles/vllm-qwen38-tp4/README.md)).-In another terminal, read the served model identifier and send a completion:+Then serve with the measured settings, from the directory that holds `calibration-vllm-tp4`:```bash-MODEL=$(curl --fail --silent http://127.0.0.1:8000/v1/models | python -c 'import json,sys; print(json.load(sys.stdin)["data"][0]["id"])')-export MODEL-python -c 'import json, os; print(json.dumps({"model":os.environ["MODEL"],"prompt":"Explain how a hash table works.","temperature":0,"max_tokens":128}))' | \- curl --fail http://127.0.0.1:8000/v1/completions \- -H 'Content-Type: application/json' --data-binary @-+export VLLM_USE_V2_MODEL_RUNNER=1 TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=4+vllm serve Qwen/Qwen3.8-27B --tensor-parallel-size 4 \+ --max-model-len 32768 --gpu-memory-utilization 0.85 --max-num-seqs 64 \+ --seed 721 --no-enable-prefix-caching \+ --linear-backend flashinfer_cutedsl --reasoning-parser qwen3 --attention-backend CUSTOM \+ --speculative-config '{"method": "eagle_harness", "model": "EAGLE-Corp/Qwen3.8-27B-EAGLE4", "method_config": {"mode": "static", "node_top_k": 4, "tree_depth": 14, "max_verify_width": 16, "data_path": "calibration-vllm-tp4"}}'```-The response must contain a generated choice. Repeat after warmup before timing. In Docker, bind the server-to `0.0.0.0` inside the container while publishing the host port on loopback.-Sampling and logprobs have separate checks in the support matrix. This plain-text example does not qualify-multimodal requests or quantized KV caches. Stop the server with Ctrl-C before another calibration.+With INFO logging the server prints `eagle_harness: tree verify width per batch size from <path>` when it loads the+rounds file. Without `rounds/tree.json` the verify width stays 16 at every batch size. Do not point `data_path` at+the packaged rounds-only directory, because it has no cost tables. Sampling, logprobs and structured output have+their own limits in the [known limits](known-limits.md#requests).diff --git a/docs/release.env b/docs/release.envindex 7677402..4690b1d 100644--- a/docs/release.env+++ b/docs/release.env@@ -1,10 +1,10 @@# Copyright 2026 Eagle AI Corp.# SPDX-License-Identifier: LicenseRef-Eagle-4-Source-Available# Source it from the directory that contains the checkout: . EAGLE-4/docs/release.env-# Distribution locations for eagle-harness 0.1.1. Publication is a separate release gate.-# The engines did not change in 0.1.1: their wheels stay on the v0.1.0 GitHub release (EAGLE_RELEASE_ASSETS), which-# the wheel indexes serve, and the published images are the 0.1.0 images, which carry eagle-harness 0.1.0.-export EAGLE_HARNESS_VERSION='0.1.1'+# Distribution locations for eagle-harness 0.1.1. Publication is a separate release gate. x-release-please-version+# Engine wheel assets and image tags have their own release versions.+# Advance their pins only after the corresponding artifacts pass release qualification.+export EAGLE_HARNESS_VERSION='0.1.1' # x-release-please-versionexport EAGLE_IMAGE_VERSION='0.1.0'export EAGLE_WHEEL_ROOT='https://eagle-hq.github.io/EAGLE-4/whl'export EAGLE_CUDA_INDEX="$EAGLE_WHEEL_ROOT/cu130/"@@ -19,12 +19,12 @@ export EAGLE_SGLANG_IMAGE="$EAGLE_IMAGE_ROOT-sglang:$EAGLE_IMAGE_VERSION"export EAGLE_ROCM_IMAGE="$EAGLE_IMAGE_ROOT-sglang-rocm:$EAGLE_IMAGE_VERSION"export EAGLE_VLLM_IMAGE="$EAGLE_IMAGE_ROOT-vllm:$EAGLE_IMAGE_VERSION"export EAGLE_TOKENSPEED_IMAGE="$EAGLE_IMAGE_ROOT-tokenspeed:$EAGLE_IMAGE_VERSION"-export EAGLE_SGLANG_VERSION='0.5.23.dev252+eagle.4'+export EAGLE_SGLANG_VERSION='0.5.23.dev252+eagle.5'export EAGLE_ROCM_VERSION='0.5.23.dev252+eagle.4.rocm'export EAGLE_VLLM_VERSION='0.32.0rc1.dev283+eagle.4'-export EAGLE_TOKENSPEED_VERSION='0.1.1+eagle.4'+export EAGLE_TOKENSPEED_VERSION='0.1.1+eagle.5'export EAGLE_TORCH_CUDA_INDEX='https://download.pytorch.org/whl/cu130'# The pinned SGLang dependency cuda-tile==1.6.0rc5 has vendor wheels here.export EAGLE_NVIDIA_INDEX='https://pypi.nvidia.com'# These are coupled native packages from the same TokenSpeed release source.-export EAGLE_TOKENSPEED_AUX_REQUIREMENTS='tokenspeed-scheduler==0.1.40+eagle.4 tokenspeed-kernel==0.1.4+eagle.4'+export EAGLE_TOKENSPEED_AUX_REQUIREMENTS='tokenspeed-scheduler==0.1.40+eagle.4 tokenspeed-kernel==0.1.4+eagle.5'diff --git a/docs/support-matrix.md b/docs/support-matrix.mdindex 748ddd0..9936c32 100644--- a/docs/support-matrix.md+++ b/docs/support-matrix.md@@ -6,26 +6,11 @@ last_verified: 2026-10-20# Measured support matrix-EAGLE-4 sizes speculative trees from measured serving costs. The EAGLE-4 tables report Qwen3.8-27B with an-EAGLE-4 draft on NVIDIA B300 and AMD MI355X. TokenSpeed is experimental.--Throughput is generated tokens divided by elapsed workload time, in tokens/s. Higher is better.-An engine's speedup divides EAGLE-4 throughput by its own matched no-speculation control.-Client concurrency is distinct from the server's running-request cap. A client above that cap queues.--These are configuration-specific measurements, not a claim of a speedup at every batch size. Rows from-short replays, full multi-turn workloads and synthetic contexts are not interchangeable. In particular, these-replays do not replace EAGLE-4's full SPEED-Bench qualitative result or its HumanEval closed-loop results.-Raw launch manifests and request traces are retained by the maintainers, not shipped in this source package.+Every row is Qwen3.8-27B with its EAGLE-4 draft, and each speedup divides EAGLE-4 throughput by the same engine's+matched no-speculation control. [Reading the numbers](#reading-the-numbers) defines the terms.## SGLang on NVIDIA B300-The target and EAGLE-4 v3 draft use BF16. The dynamic tree has maximum depth 12, top-k 4 and verify width 16.-Target and draft attention use FlashInfer with page size 1. The SPEED-Bench HTTP client uses temperature 0.-Concurrency 1 uses 22 conversations and 31 turns with at most 1024 new tokens per turn. Concurrency 8 uses-88 first turns with at most 512 new tokens. Concurrency 64 uses 220 first turns with the same 512-token cap.-Every timed pass follows a warm pass. EAGLE-4 measures live costs. Each EAGLE-4 configuration has two starts.-| GPUs / TP | Client concurrency | Running cap | Graph padding | No-spec tokens/s | EAGLE-4 tokens/s | Speedup ||---|---:|---:|---|---:|---:|---:|| 1 | 1 | 1 | Irrelevant at batch 1 | 103.1 | 343.1 | 3.33× |@@ -35,6 +20,14 @@ Every timed pass follows a warm pass. EAGLE-4 measures live costs. Each EAGLE-4| 4 | 64 | 32 | Disabled in measurement | 3925.3 | 5281.3 | 1.35× || 8 | 1 | 1 | Irrelevant at batch 1 | 241.7 | 828.6 | 3.43× |+### B300 SGLang measurement conditions++The target and EAGLE-4 v3 draft use BF16. The dynamic tree has maximum depth 12, top-k 4 and verify width 16.+Target and draft attention use FlashInfer with page size 1. The SPEED-Bench HTTP client uses temperature 0.+Concurrency 1 uses 22 conversations and 31 turns with at most 1024 new tokens per turn. Concurrency 8 uses+88 first turns with at most 512 new tokens. Concurrency 64 uses 220 first turns with the same 512-token cap.+Every timed pass follows a warm pass. EAGLE-4 measures live costs. Each EAGLE-4 configuration has two starts.+TP4 concurrency 1 measures the released stack: `sglang==0.5.23.dev252+eagle.4` from the CUDA wheel index and`eagle-harness==0.1.0` from PyPI, installed into a clean environment and served by the SGLang quickstart commandfrom a directory outside any checkout. It has six EAGLE-4 starts (725.2 to 752.7 tokens/s) and three no-spec@@ -43,16 +36,26 @@ twenty-prompt check.TP4 concurrency 8 measures 2749.5 and 2891.8 tokens/s across its two starts.TP8 concurrency 1 and 8 each give 5 exact / 14 near tie / 1 fp32-tolerance pass, with zero failures-in the configuration-specific twenty-prompt checks. This does not resolve TP8 concurrency-8 throughput variability.+in the configuration-specific twenty-prompt checks. TP8 concurrency-8 throughput variability is being investigated.Concurrency 64 queues into 32 running slots. The measured padding-disabled settings identify those results,not a recommendation to disable padding. Concurrent serving commands keep SGLang's default padding enabled.## SGLang on AMD MI355X+| Configuration | Client concurrency | No-spec tokens/s | EAGLE-4 tokens/s | Speedup | EAGLE-4 starts |+|---|---:|---:|---:|---:|---:|+| Earlier one-request profile | 1 | — | 289.4 | — | 2 |+| Measured profile, explicit path | 8 | 552.7 | 1145.9 | 2.07× | 2 |+| Measured profile, explicit path | 32 | 1269.2 | 1980.8 | 1.56× | 2 |+| Packaged profile, automatic selection | 8 | — | 1209.2 | — | 1 |++Concurrency 64 on one MI355X is a [known limit](known-limits.md#performance-and-correctness-in-progress).++### MI355X measurement conditions+Qwen3.8-27B and its EAGLE-4 draft run on one MI355X at TP1 with BF16 target, KV cache and recurrent state.-AMD ROCm is included at all supported batch sizes. Concurrency 64 on one MI355X is a-[known limit](known-limits.md#performance-and-correctness-gaps).+AMD ROCm is included at all supported batch sizes.The measured-profile workload uses the original SPEED-Bench client, 44 conversations and 60 turns,temperature 0, at most 1024 new tokens per turn and a full matching warm pass. The running cap matches clientconcurrency. The tree has maximum depth 12, top-k 4 and verify width 32. It uses measured MI355X costs,@@ -60,13 +63,6 @@ AITER target attention, rounded recurrent-state updates and disabled draft preladisable graph padding but capture every integer batch size up to the running cap. These are recordedmeasurement conditions, not a recommendation to disable padding.-| Configuration | Client concurrency | No-spec tokens/s | EAGLE-4 tokens/s | Speedup | EAGLE-4 starts |-|---|---:|---:|---:|---:|---:|-| Earlier one-request profile | 1 | — | 289.4 | — | 2 |-| Measured profile, explicit path | 8 | 552.7 | 1145.9 | 2.07× | 2 |-| Measured profile, explicit path | 32 | 1269.2 | 1980.8 | 1.56× | 2 |-| Packaged profile, automatic selection | 8 | — | 1209.2 | — | 1 |-The earlier one-request profile uses 22 conversations and 31 turns with the same temperature and token cap,a running cap of 1 and two starts at 291.8 and 287.0 tokens/s. A separate packaged-profile check at concurrency 1reaches 290.9 tokens/s on the 60-turn workload in one start. No matched no-speculation speedup is reported for@@ -84,11 +80,6 @@ separate from a tolerance pass and does not imply equality to the native BF16 to## vLLM on NVIDIA B300-These speedups use vLLM's own matched no-speculation control.-Qwen3.8-27B BF16 uses the EAGLE-4 v3 draft and the SPEED-Bench HTTP client that preserves reasoning text between-turns. Temperature is 0. Concurrency 1 uses 22 conversations and 31 turns with at most 1024 new tokens per turn.-Concurrency 8 uses 88 first turns with at most 512 new tokens. Each timed pass follows a full matching warm pass.-| GPUs / TP | Client concurrency | Running cap | Mode | No-spec tokens/s | EAGLE-4 tokens/s | Speedup | EAGLE-4 starts ||---|---:|---:|---|---:|---:|---:|---:|| 1 | 1 | 8 | Static tree | 104.0 | 279.9 | 2.69× | 2 |@@ -98,18 +89,18 @@ Concurrency 8 uses 88 first turns with at most 512 new tokens. Each timed pass f| 8 | 1 | 8 | Static tree | 317.2 | 745.4 | 2.35× | 2 || 8 | 8 | 8 | Static tree | 2063.0 | 2890.0 | 1.40× | 2 |+### B300 vLLM measurement conditions++Qwen3.8-27B BF16 uses the EAGLE-4 v3 draft and the SPEED-Bench HTTP client that preserves reasoning text between+turns. Temperature is 0. Concurrency 1 uses 22 conversations and 31 turns with at most 1024 new tokens per turn.+Concurrency 8 uses 88 first turns with at most 512 new tokens. Each timed pass follows a full matching warm pass.+The tree uses top-k 4, depth 14 and verify width 16 with `CUSTOM` attention. TP4 uses the derived depth stop.TP1 and TP8 use fixed depth. No-spec uses the engine's default FlashInfer attention with the same cap.The two TP8 concurrency-8 starts differ by 11%, and both are included in the mean.### Measured high-load tree schedule-These rows measure the release vLLM engine with the `eagle-harness` 0.1.0 vLLM code on B300 TP4, served by-the vLLM quickstart recipe: running cap 64, calibrated TP4 costs and the packaged rounds file. The workload is-SPEED-Bench first turns with at most 512 new tokens: 88 requests at concurrency 8, 220 at 64 and 440 at 128.-Each timed pass follows a matching warm pass. Every arm, including the matched no-spec control, has two-independent server starts, interleaved in one allocation. Temperature 1 uses top-p 1, no top-k and a fixed seed.-| Client concurrency | Temperature | No-spec tokens/s | EAGLE-4 tokens/s | Speedup | Verify width ||---:|---:|---:|---:|---:|---:|| 8 | 0 | 1695.2 | 2932.7 | 1.73× | 16 |@@ -117,9 +108,17 @@ independent server starts, interleaved in one allocation. Temperature 1 uses top| 128 | 0 | 7063.0 | 8552.5 | 1.21× | 4 || 64 | 1 | 7034.3 | 7267.9 | 1.03× | 4 |-Concurrency 128 queues into the 64 running slots. Without the rounds file the tree keeps width 16 at every-batch size, and the same concurrency-64 cells give 5349.4 tokens/s (0.76×) at temperature 0 and 4537.2 tokens/s-(0.65×) at temperature 1.+Without the rounds file the tree keeps width 16 at every batch size, and the same concurrency-64 cells give+5349.4 tokens/s (0.76×) at temperature 0 and 4537.2 tokens/s (0.65×) at temperature 1.++#### How the high-load rows were measured++These rows measure the release vLLM engine with the `eagle-harness` 0.1.0 vLLM code on B300 TP4, served by+the vLLM quickstart recipe: running cap 64, calibrated TP4 costs and the packaged rounds file. The workload is+SPEED-Bench first turns with at most 512 new tokens: 88 requests at concurrency 8, 220 at 64 and 440 at 128.+Each timed pass follows a matching warm pass. Every arm, including the matched no-spec control, has two+independent server starts, interleaved in one allocation. Temperature 1 uses top-p 1, no top-k and a fixed seed.+Concurrency 128 queues into the 64 running slots.Greedy checks against no-speculation decoding pass on both verify widths. Twenty sequential prompts (width 16)give 5 exact / 14 near tie / 1 fp32-tolerance pass, and 64 concurrent prompts at concurrency 64 (width 4) give@@ -137,23 +136,27 @@ No rounds file means no width change.## Short-workload C8 checks-These B300 TP4 checks use two independent starts, 16 requests with a 256-token cap and a matching-warm pass. They report absolute rates for the final runtime source on its pinned release engine.-They do not replace the longer-workload tables or establish a speedup over no-spec decoding.-| Engine | Tokens/s | Running cap | Policy ||---|---:|---:|---|| SGLang | 2586.7 | 32 | Live costs, graph padding enabled || vLLM | 2708.8 | 64 | Width 16 at this batch size |+### Short-workload measurement conditions++These B300 TP4 checks use two independent starts, 16 requests with a 256-token cap and a matching+warm pass. They report absolute rates for the final runtime source on its pinned release engine.+They do not replace the longer-workload tables or establish a speedup over no-spec decoding.Each measured pass lasts roughly 1.5 seconds with substantial start-to-start spread. No matchingno-spec mean is reported for these short checks.## TokenSpeed (experimental)-The adapter implements static and calibrated trees, chains and sampled trees. This is implementation coverage,-not a performance or distribution-quality claim for every configuration.-Prompt logprobs, live cost measurement and native no-speculation rounds are not supported.+| GPUs / TP | Client concurrency | Running cap | No-spec tokens/s | EAGLE-4 tokens/s | Speedup |+|---|---:|---:|---:|---:|---:|+| 4 | 8 | 32 | 1555.7 | 3150.2 | 2.02× |+| 4 | 64 | 32 | 4894.4 | 6950.2 | 1.42× |++### TokenSpeed measurement conditionsThese rows measure the released wheels (`tokenspeed==0.1.1+eagle.4` with its scheduler and kernel wheels fromthe CUDA index, `eagle-harness==0.1.0` from PyPI), installed by the install guide's TokenSpeed block and served@@ -162,15 +165,13 @@ turns at temperature 0 with at most 256 new tokens: 22 requests at concurrency 8timed pass follows a matching warm pass. EAGLE-4 and the matched no-spec control each have two starts,interleaved in one allocation.-| GPUs / TP | Client concurrency | Running cap | No-spec tokens/s | EAGLE-4 tokens/s | Speedup |-|---|---:|---:|---:|---:|---:|-| 4 | 8 | 32 | 1555.7 | 3150.2 | 2.02× |-| 4 | 64 | 32 | 4894.4 | 6950.2 | 1.42× |-Each pass is one short wave of requests. The EAGLE-4 starts are 3062.5 and 3238.0 tokens/s at concurrency 8,and 6876.4 and 7023.9 tokens/s at concurrency 64. A twenty-prompt greedy check of the same installationgives 4 exact / 16 fp32-tolerance passes and no failures.+The adapter implements static and calibrated trees, chains and sampled trees. This is implementation coverage,+not a performance or distribution-quality claim for every configuration.+Prompt logprobs, live cost measurement and native no-speculation rounds are not available on TokenSpeed yet.No TokenSpeed throughput number is claimed for an unmeasured or non-matching workload. The CUDA installchecks completed real static and cost-model requests, but a static-mode first divergence remains above thefp32 gate. Successful installation or generation is not blanket correctness qualification. TP1/TP8, sampled@@ -178,11 +179,6 @@ distributions, long contexts and release-stack Nemotron need separate qualificat## Artifact installation checks-These checks use one server start on B300 TP4 and twenty prompts with up to 256 generated tokens. The rate is-total generated tokens divided by summed request latency, not a multi-start concurrent throughput benchmark.-The image rows test the published `0.1.0` images, each of which is pullable without credentials. The wheel-rows predate publication and used a private index.-| Artifact and mode | Correctness result | Tokens/s | Qualification limit ||---|---|---:|---|| SGLang image `eagle-4-sglang:0.1.0` | 4 exact / 16 near ties | 537.40 | Hook, dependency, quickstart request and correctness checks pass |@@ -191,17 +187,17 @@ rows predate publication and used a private index.| TokenSpeed wheel, serve | 4 exact / 16 fp32-tolerance passes | 786.33 | Host-compatible CUDA 13.2 JIT; CUDA 13.0 JIT fails on Ubuntu 26.04 / glibc 2.43 || TokenSpeed wheel, static | 5 exact / 14 fp32 passes / 1 failure | 706.36 | Not qualified: failing gap 0.178637 exceeds 0.125 |-The SGLang and vLLM images contain the `eagle-harness` 0.1.0 wheel published on PyPI.-A near-tie or fp32-tolerance pass is not an exact-token match. TokenSpeed lacks full-decode native-top-logprobs, so each divergent prefix in these install checks goes directly to fp32 arbitration.+### Installation-check conditions-## Correctness and request features+These checks use one server start on B300 TP4 and twenty prompts with up to 256 generated tokens. The rate is+total generated tokens divided by summed request latency, not a multi-start concurrent throughput benchmark.+The image rows test the published `0.1.0` images, each of which is pullable without credentials. The wheel+rows predate publication and used a private index. The SGLang and vLLM images contain the `eagle-harness` 0.1.0+wheel published on PyPI. A near-tie or fp32-tolerance pass is not an exact-token match. Full-decode native+top-logprobs are not available on TokenSpeed yet, so each divergent prefix in these install checks goes directly+to fp32 arbitration.-The greedy gate reports exact, near-tie, fp32-tolerance, candidate-is-fp32-argmax and failed outputs separately.-A first-divergence reference-logprob gap of at most 0.13 is a near tie. Larger gaps require an fp32 forward-at the exact common prefix. A pair within two BF16 steps at that logit magnitude passes the tolerance check.-EAGLE-4 token equal to the fp32 argmax also passes, reported in its own category. Passing either rule-is not bitwise equality to the native BF16 token stream.+## Correctness and request features| Configuration or feature | Evidence ||---|---|@@ -217,5 +213,24 @@ is not bitwise equality to the native BF16 token stream.| SGLang logprobs and sampling | Accepted-token and input logprobs and target-side speculative sampling are implemented || Nemotron / Mamba2 | Adapters are implemented on SGLang and vLLM, but a general correctness-qualified speed claim is not established |+### How these were checked++The greedy gate reports exact, near-tie, fp32-tolerance, candidate-is-fp32-argmax and failed outputs separately.+A first-divergence reference-logprob gap of at most 0.13 is a near tie. Larger gaps require an fp32 forward+at the exact common prefix. A pair within two BF16 steps at that logit magnitude passes the tolerance check.+EAGLE-4 token equal to the fp32 argmax also passes, reported in its own category. Passing either rule+is not bitwise equality to the native BF16 token stream.+See [known limits](known-limits.md) for configuration restrictions and [compatibility](compatibility.md) forrequired engine hooks. Kernel compilation on another GPU does not establish serving correctness or speed there.++## Reading the numbers++Throughput is generated tokens divided by elapsed workload time, in tokens/s, and higher is better. Client+concurrency is the number of requests the client keeps in flight. It is distinct from the server's running-request+cap, and a client above that cap queues. TokenSpeed is experimental.++These are configuration-specific measurements, not a claim of a speedup at every batch size. Rows from short+replays, full multi-turn workloads and synthetic contexts are not interchangeable. In particular, these replays do+not replace EAGLE-4's full SPEED-Bench qualitative result or its HumanEval closed-loop results. Raw launch manifests+and request traces are retained by the maintainers, not shipped in this source package.diff --git a/docs/troubleshooting.md b/docs/troubleshooting.mdindex 165062d..0be6cba 100644--- a/docs/troubleshooting.md+++ b/docs/troubleshooting.md@@ -39,6 +39,16 @@ tokenspeed: not installedno supported host installed (one of: sglang, vllm, tokenspeed)```+## Install problems++| symptom | cause | fix |+|---|---|---|+| `The virtual environment was not created successfully because ensurepip is not available` | Ubuntu ships `venv` without `ensurepip` | `apt install python3.12-venv`, then create the environment again |+| pip builds `cuda-tile` from source or cannot find `cuda-tile==1.6.0rc5` | the SGLang command ran without `--extra-index-url https://pypi.nvidia.com` | use the full command from [Install](install.md#sglang-cuda) |+| pip installs a stock `sglang`, `vllm` or `tokenspeed` | the engine pin or the EAGLE-4 index is missing, and stock packages have no EAGLE-4 hooks | pin the exact `+eagle.N` version with the EAGLE-4 index, as in [Install](install.md) |+| JIT compilation fails with `rsqrt`/`rsqrtf` exception-specification errors | CUDA 13.0 `nvcc` with the C headers of Ubuntu 26.04 / glibc 2.43 | `export CUDA_HOME=/usr/local/cuda-13.2` (a CUDA toolkit compatible with the host headers), or use a release image. The wheel index stays `cu130`. |+| `eagle_harness.__file__` points into a checkout of this repository | `python -m` puts the current directory first on the import path, so a server started inside the checkout runs the source tree | start servers from another directory, or with the `sglang serve` and `vllm serve` commands, which do not import from the current directory |+## Server errors at startup| message | cause | fix |@@ -60,13 +70,14 @@ no supported host installed (one of: sglang, vllm, tokenspeed)| `FlashInferSamplingBackend cannot verify draft trees; use --sampling-backend greedy or triton` (TokenSpeed) | TokenSpeed's default sampling backend | pass `--sampling-backend greedy` || `--speculative-num-draft-tokens N verifies more nodes than a tree of depth ... holds` (TokenSpeed) | the tree cannot fill the verify width | raise `--speculative-num-steps` or `--speculative-eagle-topk`, or lower the width || `EAGLE-4 drafts with [...] are not ported to eagle-harness yet (dense drafts only).` | a draft structure the engine does not build (see the FAQ, "Which models work?") | use a supported single-layer EAGLE-4 draft |+| `TypeError: int() argument must be ... not 'NoneType'` in `average_context`, on the first request (sglang) | `eagle-harness` 0.1.0 with SGLang's default overlap scheduler | upgrade to `eagle-harness` 0.1.1 or newer, which serves with the overlap scheduler, or pass `--disable-overlap-schedule` |## Requests refusedsglang refuses requests with multimodal inputs (`Speculative algorithm EAGLE_HARNESS does not support multimodalinputs.`). Serve them from a server without the method.-## Slower than expected+## Throughput below the support matrix- Check that the cost tables match your GPU, TP size, tree shape and engine build ([Calibration](calibration.md)).A mismatched `key.json` is refused. Unkeyed legacy tables can load with a warning, and the key does notdiff --git a/tests/test_install_docs.py b/tests/test_install_docs.pyindex cc4b044..5a7c55b 100644--- a/tests/test_install_docs.py+++ b/tests/test_install_docs.py@@ -1,7 +1,7 @@# Copyright 2026 Eagle AI Corp.# SPDX-License-Identifier: LicenseRef-Eagle-4-Source-Available-"""The install guides run from outside the checkout and install ROCm from public indexes only."""+"""The install guides use the release pins, run from outside the checkout and install ROCm from public indexes only."""from pathlib import Pathimport re@@ -10,6 +10,7 @@ import pytestREPO = Path(__file__).resolve().parents[1]GUIDES = [REPO / "README.md", *sorted((REPO / "docs").glob("*.md"))]+QUICKSTARTS = [REPO / "README.md", REPO / "docs/install.md", *sorted((REPO / "docs").glob("quickstart-*.md"))]def bash_blocks(path: Path) -> list[str]:@@ -25,6 +26,17 @@ def release_settings() -> dict[str, str]:return dict(re.findall(r"^export (EAGLE_[A-Z0-9_]+)=(.*)$", text, re.M))+def resolved_settings() -> dict[str, str]:+ """release.env values without quotes and with earlier variables substituted, as the shell reads them."""+ values: dict[str, str] = {}+ for name, raw in release_settings().items():+ value = raw[1:-1] if raw[:1] in "'\"" else raw+ if raw.startswith('"'):+ value = re.sub(r"\$\{?(\w+)\}?", lambda m: values[m[1]], value)+ values[name] = value+ return values++@pytest.mark.parametrize("guide", GUIDES, ids=lambda p: str(p.relative_to(REPO)))def test_guides_never_enter_the_checkout(guide):# `python -m` imports from the current directory, so a server started inside the checkout@@ -42,6 +54,50 @@ def test_guide_variables_come_from_release_settings(guide):assert name in settings, f"{guide.name} uses {name}, which docs/release.env does not export"[email protected]("guide", QUICKSTARTS, ids=lambda p: str(p.relative_to(REPO)))+def test_copy_paste_commands_use_the_release_pins(guide):+ # The quick starts spell the pins out so they paste without release.env; they must match it exactly.+ s = resolved_settings()+ pins = {+ "sglang": s["EAGLE_SGLANG_VERSION"],+ "vllm": s["EAGLE_VLLM_VERSION"],+ "tokenspeed": s["EAGLE_TOKENSPEED_VERSION"],+ **dict(re.findall(r"(tokenspeed-\w+)==(\S+)", s["EAGLE_TOKENSPEED_AUX_REQUIREMENTS"])),+ }+ indexes = {s[k] for k in ("EAGLE_CUDA_INDEX", "EAGLE_TORCH_CUDA_INDEX", "EAGLE_NVIDIA_INDEX")}+ images = {s[k] for k in ("EAGLE_SGLANG_IMAGE", "EAGLE_VLLM_IMAGE", "EAGLE_TOKENSPEED_IMAGE")}+ for block in bash_blocks(guide):+ for package, version in re.findall(r'"(sglang|vllm|tokenspeed(?:-\w+)?)==([^"\s]+)"', block):+ assert version == pins[package], f"{guide.name}: {package}=={version}, release.env pins {pins[package]}"+ for url in re.findall(r"--extra-index-url (https://\S+)", block):+ assert url in indexes, f"{guide.name}: {url} is not a release index"+ for image in re.findall(r"ghcr\.io/eagle-hq/\S+", block):+ assert image in images, f"{guide.name}: {image} is not a published release image"+ if re.search(r'"(sglang|vllm|tokenspeed)==[^"]*\+eagle\.', block):+ # A +eagle.N pin resolves only with the EAGLE-4 wheel index in the same command.+ assert s["EAGLE_CUDA_INDEX"] in block, block+++def test_readme_quick_start_is_one_install_one_serve_one_request():+ quick = section(REPO / "README.md", "## Quick start")+ docker, install, serve, request = re.findall(r"```bash\n(.*?)```", quick, re.S)+ assert "docker run" in docker and "--speculative-algorithm EAGLE_HARNESS" in docker+ assert install.count("pip install") == 1 and '"eagle-harness[sglang]"' in install+ assert serve.startswith("sglang serve ") and re.search(r"--speculative-draft-model-path [\w-]+/Qwen3\.8-27B-EAGLE4$", serve.strip())+ assert request.startswith("curl http://localhost:30000/")+ # The README, the install guide and the SGLang guide show the same commands.+ assert install in bash_blocks(REPO / "docs/install.md")+ assert serve in bash_blocks(REPO / "docs/quickstart-sglang.md")+ assert docker in bash_blocks(REPO / "docs/quickstart-sglang.md")+++def test_copy_paste_commands_never_pin_the_harness():+ # An unpinned harness cannot name an unpublished or old version.+ for guide in QUICKSTARTS:+ for block in bash_blocks(guide):+ assert not re.search(r"eagle-harness(\[\w+\])?==\d", block), f"{guide.name}: {block}"++def test_rocm_install_uses_public_indexes_and_the_shipped_lock():install = section(REPO / "docs/install.md", "## SGLang ROCm on AMD")block = re.findall(r"```bash\n(.*?)```", install, re.S)[0]