Skip to content

Official Inference Code of EAGLE-4

fix(sglang): decode without speculation wherever speculation would be slower

Seyfal Sultanovdf5a23ad573f
diff --git a/README.md b/README.md
index 8d6408a..205a979 100644
--- a/README.md
+++ b/README.md
@@ -51,6 +51,9 @@
**Leave speculation on. EAGLE-4 adapts to your hardware, batch and workload.**
+For high load, serve SGLang on four B300 GPUs with the [high-load recipe](docs/quickstart-sglang.md#high-load), which
+keeps pace with plain decoding up to 512 concurrent requests.
+
EAGLE-4 (Extrapolation Algorithm for Greater Language-model Efficiency) speeds up LLM inference with
speculative decoding. EAGLE-4 draft models are built to run with `eagle-harness`, which decides how far to
speculate from measured costs on your setup, so every draft tree fits the batch size, the hardware and the
diff --git a/docs/calibration.md b/docs/calibration.md
index 8d4ab3c..070882c 100644
--- a/docs/calibration.md
+++ b/docs/calibration.md
@@ -36,6 +36,28 @@ optimal acceptance or no-loss behavior at high concurrency; compare sustained wo
After seeding, a draft prefix is timed once per table-refresh period; target timings continue on served verifies.
Prelaunched draft samples use that batch's bucket and context, not the preceding batch's seed key.
+An SGLang chain (`"intensity": "chain_intensity"`, `--speculative-eagle-topk 1`) without `data_path` does not use
+these cost tables. It picks one verify width per batch size (keyed to the nearest calibration batch size) from its
+own served rounds: committed tokens per wall-clock time over whole served intervals, prefill steps included, so the
+draft work a prefill or a batch change throws away is priced. Two widths are compared at a time on alternating
+blocks of rounds, widening while the wider one wins. When the engine captures native decode graphs, native decode
+is measured last at a batch size and, where it wins, serves that batch size and every larger one: its requests stop
+drafting until they finish. The choices are cached next to the live costs (`<key>-chain.json`), so a restart on the
+same setup serves them from its first round. A first start on a B300 TP4 with the Qwen3.8-27B checkpoint pair serves
+packaged choices instead (width 4 at batch 16 and 64, native decode from 256;
+[profile](../eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/README.md)) and searches only the other batch
+sizes; on other setups the first start pays the search.
+
+A serve-mode tree (the default) keeps the cost tables for the shape of each round and decides per batch size only
+whether to speculate at all, on the same served intervals: speculation is priced by the tokens its rounds committed,
+rounds that seed the live costs are not counted, and native decode is measured last and must lose by two standard
+errors. Requests decoded natively stop drafting until they finish, and so do requests that join their batch, so
+native decode is not measured where speculation beats its estimate from the live costs (the one-token verify) by
+1.5x. Native decode at a batch size serves every larger one and speculation every smaller one. The choices are cached
+as `<key>-tree.json`; a first start on the B300 TP4 setup above serves packaged choices (speculation up to batch 8,
+native decode from 64) and searches the batch sizes in between. Without native decode graphs in the engine, or with
+Mamba prefix-cache tracking or DP attention, the tree speculates at every batch size, as before.
+
Offline calibration below remains the precise mode: it times isolated steps on dummy batches over a fine context
grid. Its tables carry the same key in `key.json`; a server refuses tables keyed for another setup, and accepts
tables without a key (measured before keys existed) with a warning.
@@ -151,6 +173,8 @@ or a private proxy before exposing the API.
The vLLM sweep takes `context_stop` from the resolved `max_model_len` (including engine overrides),
records it in each raw file, and fits that same range. The default engine limit remains 4608.
+The engine and `serve.sh` keep vLLM's default `max_num_batched_tokens`, which fits one batch of
+`max_num_seqs` requests at `max_model_len`. A value set through `--engine-args` must stay within that product.
The limit must be a multiple of the context step, with room for the widest verify after the
last sample; unsupported grids fail rather than silently measuring a shorter context.
Raw files without `context_stop` must be regenerated. Context-indexed cost tables log once
diff --git a/docs/known-limits.md b/docs/known-limits.md
index 20dbe4d..356d4b0 100644
--- a/docs/known-limits.md
+++ b/docs/known-limits.md
@@ -1,7 +1,7 @@
---
type: reference
owner: eagle-harness
-last_verified: 2026-10-21
+last_verified: 2026-10-23
---
# Known limits
@@ -21,15 +21,45 @@ kernel is a startup check.
- Throughput depends on the model, draft, engine, precision, TP size, attention backend, context length and
workload. A result at one concurrency does not establish a result at another.
+## High load
+
+On SGLang with four GPUs, the [high-load recipe](quickstart-sglang.md#high-load) keeps pace with plain decoding up
+to 512 concurrent requests. These limits apply around it:
+
+- The SGLang command of the README quick start is slower than plain decoding above about 32 to 64 concurrent
+ requests. On one B300 it measured 0.73× the same engine without speculation at concurrency 64 and 0.59× at 256.
+ With `--tp-size 4` it measured 0.80× at 64, 0.48× at 128, 0.37× at 256 and 0.32× at 512 (2 starts each). Three
+ settings cause it. The radix cache, on by default, makes this hybrid model track prefix-cache state that a batch
+ switching to plain decoding cannot carry yet, so the server speculates at every load. The default tree caps a
+ four-GPU server at 32 running requests, while plain decoding admits several hundred. On one GPU, the engine
+ reserves verify state for every admitted request, so a speculative server admits about a quarter of the
+ requests plain decoding does. For high load on four GPUs, use the high-load recipe. A fix for the default
+ command is planned.
+- High load on one GPU is not supported yet. On one GPU, a speculative server started with
+ `--disable-radix-cache` stops during draft graph capture, so keep the radix cache on there. The high-load recipe
+ is for four GPUs, and the server log's advice to add `--disable-radix-cache` applies to four GPUs only.
+- The first start of the high-load recipe on a machine compiles kernels and measures its live costs during its
+ first requests. It measured 0.95× at 64 and 0.97× at 128 concurrent requests, and restarts with warm caches
+ 1.01× and 1.00× (3 starts each).
+- The packaged choices cover B300 TP4 with the Qwen3.8-27B checkpoint pair. On other setups the first start also
+ searches its choices while serving: with the packaged choices withheld on that same setup, first starts measured
+ 0.80× at concurrency 64 (4 starts, from 0.71× to 0.96×) and restarts 0.94× (2 starts). From 128 concurrent
+ requests up they measured 0.97× or more.
+- Requests decoded without speculation stop drafting until they finish, and so do requests that join their
+ batch. After a load spike, a batch keeps decoding without speculation until it empties, even when the load
+ drops in the meantime.
+- Greedy agreement at 64 and 512 concurrent requests is checked with the fp32 rule, not claimed. At 64, one prompt
+ takes another token at one position in 3 of 12 EAGLE-4 starts, a prefix where the engine without speculation
+ emits the same token under another batch shape. At 512, the engine without speculation also differs between its
+ own starts. Every twenty-prompt check passes.
+- vLLM and TokenSpeed do not switch to plain decoding at high load yet. Above concurrency 64 they can be slower
+ with speculation than without it, so serve those loads without speculation for now. The switch is planned for
+ both engines.
+
## Performance and correctness in progress
- SGLang TP4 single-request and 32K-context optimization is in progress, as is TP8 concurrency-8 prefill
overhead and intermittent stalls. Repeated passes in one process are not independent server starts.
-- SGLang above concurrency 64: with a running cap of 512 and the default tree (TP4), EAGLE-4 measured 1.03x the
- same engine without speculation at concurrency 64, 0.74x at 256 and 0.61x at 512 (3 starts each). Serve
- concurrency above 64 without speculation for now. The automatic high-load policy and high-load fixed-chain
- improvements are being qualified and stay off by default. Greedy agreement with no-spec at concurrency 512 is
- not claimed: the engine without speculation also differs between two starts there.
- SGLang native-OFF, including OFF forced by a cost profile, uses the engine's per-variant forward-mode
decode graphs (the `forward_mode` field of the decode graph variant). The SGLang CUDA engine
`0.5.23.dev252+eagle.5` has them. On the 0.1.0 SGLang CUDA engine and on ROCm, EAGLE-4 keeps native-OFF off.
@@ -45,8 +75,8 @@ kernel is a startup check.
optimized to match native decoding.
- vLLM TP4 at concurrency 64: in a 64-prompt greedy comparison against no-spec, one prompt can take another token
at one position, up to 5.4 bf16 ULP from the fp32 choice, beyond the 2-ULP rule. It appeared in 3 of 6 EAGLE-4
- starts with 0.1.1 and in 2 of 6 with 0.1.0 (the vLLM engine is the same); the other prompts and every
- twenty-prompt check at concurrency 1 pass. The cause is being investigated.
+ starts with 0.1.1 and in 2 of 6 with 0.1.0, and 0.1.2 shows it too (the vLLM engine is the same); the other
+ prompts and every twenty-prompt check at concurrency 1 pass. The cause is being investigated.
- AMD measurements apply to the MI355X TP1 Qwen3.8 profile. On one MI355X the speedup covers concurrency up
to 32; concurrency 64 is being optimized, so serve it without speculation for now. HIP side-stream prelaunch
stays off: its replay check passes and it measured no throughput gain on MI355X. Further target, draft and
diff --git a/docs/quickstart-sglang.md b/docs/quickstart-sglang.md
index d39d428..69daed9 100644
--- a/docs/quickstart-sglang.md
+++ b/docs/quickstart-sglang.md
@@ -1,7 +1,7 @@
---
type: tutorial
owner: eagle-harness
-last_verified: 2026-10-21
+last_verified: 2026-10-23
---
# SGLang guide
@@ -80,7 +80,48 @@ sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 --dtype bfloat16 \
The benchmark warmed the server from its client, hence `--skip-server-warmup`: the first request after this start
compiles kernels and seeds the costs. Keep CUDA graph padding enabled, and do not add
`--disable-cuda-graph-padding`. `--max-running-requests 32` is the tree default; set a larger cap, up to 512, for
-larger batches.
+larger batches, or serve with the [high-load recipe](#high-load).
+
+## High load
+
+For many concurrent requests on four GPUs, serve with the high-load recipe. It raises the running cap and the decode
+graph limit to 512 and turns the radix cache off, so EAGLE-4 can decode a batch without speculation wherever
+speculation would be slower:
+
+```bash
+export TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=4
+export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 SGLANG_TIMEOUT_KEEP_ALIVE=3600
+sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 --dtype bfloat16 \
+ --trust-remote-code --attention-backend flashinfer --page-size 1 \
+ --max-running-requests 512 --cuda-graph-max-bs-decode 512 \
+ --mem-fraction-static 0.70 --context-length 32768 \
+ --chunked-prefill-size 16384 --mamba-ssm-dtype bfloat16 --disable-radix-cache \
+ --random-seed 721 --skip-server-warmup \
+ --speculative-algorithm EAGLE_HARNESS \
+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 \
+ --speculative-draft-model-quantization unquant --speculative-draft-attention-backend flashinfer \
+ --speculative-attention-mode prefill --speculative-num-steps 12 \
+ --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \
+ --speculative-algorithm-config '{"mode":"serve"}'
+```
+
+With the radix cache on, this hybrid model tracks prefix-cache state that a batch switching to plain decoding
+cannot carry, so the server speculates at every load. On one GPU, keep the radix cache on: high load on one GPU is
+not supported yet ([known limits](known-limits.md#high-load)).
+
+Each speedup divides EAGLE-4 throughput by the same command without the `--speculative-*` flags, in the same
+allocation, on 4x B300 with Qwen3.8-27B and its EAGLE-4 draft at temperature 0. Columns are concurrent requests:
+
+| Server start | 1 | 8 | 64 | 128 | 256 | 512 |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| First start on a machine | 2.80× | 1.67× | 0.95× | 0.97× | 1.00× | 0.99× |
+| Restart with warm caches | 2.76× | 1.72× | 1.01× | 1.00× | 1.00× | 0.99× |
+
+The first start compiles kernels and measures its live costs during its first requests, which costs up to 5% at 64
+and 128 concurrent requests. A restart reuses the kernels and the costs and choices cached under
+`~/.cache/eagle-harness/costs`. This setup ships packaged choices; on another GPU, TP size or model pair the first
+start also searches them ([known limits](known-limits.md#high-load)). The
+[support matrix](support-matrix.md#012-high-load) has the tokens/s and the correctness checks.
Live costs are cached per GPU, TP size, model pair, engine and dtype under `~/.cache/eagle-harness/costs`.
The [calibration guide](calibration.md) explains live costs and offline chain calibration, and the
diff --git a/docs/release-notes/0.1.2.md b/docs/release-notes/0.1.2.md
new file mode 100644
index 0000000..ca915d9
--- /dev/null
+++ b/docs/release-notes/0.1.2.md
@@ -0,0 +1,93 @@
+---
+type: reference
+owner: eagle-harness
+last_verified: 2026-10-23
+---
+
+# EAGLE-4 0.1.2 release notes
+
+EAGLE-4 0.1.2 updates `eagle-harness` on the 0.1.1 engines. On SGLang, EAGLE-4 decodes a batch without speculation
+wherever speculation would be slower, so the [high-load recipe](../quickstart-sglang.md#high-load) keeps pace with
+plain decoding up to 512 concurrent requests on four B300 GPUs and keeps its speedup at low load.
+`eagle-harness calibrate --engine vllm` completes on hybrid targets, and the TokenSpeed draft runs its candidate head
+on the engine's low-latency GEMM.
+
+## Upgrade
+
+A new install follows the [installation guide](../install.md), which installs 0.1.2. To upgrade a 0.1.1
+environment, run your engine's install command with the pins below, then `eagle-harness-check`:
+
+| Engine | 0.1.2 pins |
+| --- | --- |
+| SGLang CUDA | `eagle-harness[sglang]==0.1.2`, `sglang==0.5.23.dev252+eagle.5` (unchanged) |
+| vLLM | `eagle-harness[vllm]==0.1.2`, `vllm==0.32.0rc1.dev283+eagle.4` (unchanged) |
+| TokenSpeed | `eagle-harness[tokenspeed]==0.1.2`, `tokenspeed==0.1.1+eagle.5`, `tokenspeed-kernel==0.1.4+eagle.5`, `tokenspeed-scheduler==0.1.40+eagle.4` (unchanged) |
+
+The `:0.1.1` images carry `eagle-harness` 0.1.1. The [install guide](../install.md#docker-images) lists the current
+images. Nothing in the configuration changes: every 0.1.1 configuration loads as before.
+
+## What changed
+
+**SGLang**
+- A served tree, the default, decides per batch size whether to speculate at all. It prices speculation by the
+ tokens its rounds commit over whole served intervals, measures plain decoding last, and decodes a batch without
+ speculation where plain decoding wins. Requests decoded that way stop drafting until they finish.
+- Served chains (`"intensity": "chain_intensity"`) pick one verify width per batch size the same way, and decode
+ without speculation where that wins.
+- On B300 TP4 with the Qwen3.8-27B checkpoint pair, the first start serves packaged choices and searches only the
+ batch sizes in between. The choices are cached next to the live costs, so a restart on any setup serves them from
+ its first round. [Calibration](../calibration.md) describes both policies.
+- Every KV-move program a tree round runs compiles at server startup, and rounds that stalled on one-time work or
+ followed a switch between speculation and plain decoding stay out of the prices.
+- The switch needs the radix cache off (`--disable-radix-cache`), because with it on this hybrid model tracks
+ prefix-cache state that plain decoding does not carry. A server with the radix cache on speculates at every load
+ and says so in its log.
+
+**vLLM**
+- `eagle-harness calibrate --engine vllm` completes on the Qwen3.8 hybrid target: its token budget fits one batch,
+ the tables are keyed by the models the engine resolves, and the printed `serve.sh` loads them.
+
+**TokenSpeed**
+- The draft scores its low-rank candidate head through the engine's linear layer, so TokenSpeed's GEMM dispatch
+ serves the few draft rows with its low-latency BF16 kernel. Checkpoints load as before.
+
+## Measured on the release stack
+
+Qwen3.8-27B with its EAGLE-4 draft on 4x B300 (TP4), BF16, temperature 0, on the release candidate build installed
+into clean environments and served from outside any checkout. Each speedup divides EAGLE-4 throughput by the same
+engine without speculation, measured in the same allocation. Every EAGLE-4 start passed its twenty-prompt greedy
+check against no-spec under the fp32 rule.
+
+| Engine and command | Concurrency | EAGLE-4 tokens/s | Speedup | EAGLE-4 starts |
+| --- | ---: | ---: | ---: | ---: |
+| SGLang, measured recipe | 1 | 733.8 | 3.35× | 2 |
+| SGLang, measured recipe | 8 | 3257.9 | 2.11× | 2 |
+| SGLang, measured recipe | 64 | 6488.2 | 1.38× | 2 |
+| SGLang, high-load recipe | 64 | 7393.3 | 1.01× | 3 |
+| SGLang, high-load recipe | 128 | 11114.3 | 1.00× | 3 |
+| SGLang, high-load recipe | 256 | 14332.3 | 1.00× | 3 |
+| SGLang, high-load recipe | 512 | 16197.5 | 0.99× | 3 |
+| vLLM, measured recipe | 1 | 678.5 | 2.83× | 4 |
+| vLLM, measured recipe | 8 | 2887.4 | 1.73× | 4 |
+| vLLM, measured recipe | 64 | 8013.4 | 1.19× | 4 |
+| TokenSpeed, measured recipe | 1 | 886.5 | 3.13× | 2 |
+| TokenSpeed, measured recipe | 8 | 3256.4 | 2.00× | 2 |
+| TokenSpeed, measured recipe | 64 | 6974.1 | 1.40× | 2 |
+
+The high-load rows are restarts with warm caches. A first start on a machine measured 0.95× at 64 and 0.97× at
+128 while it compiles kernels and measures costs. With 0.1.1, the same SGLang setup at a running cap of 512
+measured 0.74× at 256 and 0.61× at 512. At concurrency 1, SGLang and TokenSpeed serve multi-turn conversations
+with up to 1024 new tokens per turn and vLLM 22 first turns with up to 512. The
+[support matrix](../support-matrix.md#012-released-current) has temperature 1, one GPU, no-spec throughput and the
+conditions of each row.
+
+## Limits
+
+- The SGLang quick-start command is slower than plain decoding above about 32 to 64 concurrent requests, on one
+ GPU and with `--tp-size 4`. For high load, serve with the high-load recipe on four GPUs. A fix for the
+ quick-start command is planned.
+- High load on one GPU is not supported yet. On one GPU, keep the radix cache on.
+- vLLM and TokenSpeed do not switch to plain decoding at high load yet. Above concurrency 64, serve them without
+ speculation for now.
+- [Known limits](../known-limits.md#high-load) has the measured numbers and the rest of the limits, and the
+ [support matrix](../support-matrix.md) has every measured configuration.
diff --git a/docs/support-matrix.md b/docs/support-matrix.md
index 30b0e22..8c7bcd7 100644
--- a/docs/support-matrix.md
+++ b/docs/support-matrix.md
@@ -1,14 +1,14 @@
---
type: reference
owner: eagle-harness
-last_verified: 2026-10-21
+last_verified: 2026-10-23
---
# Measured support matrix
Each cell divides EAGLE-4 throughput by the same engine's matched no-speculation control, measured in the same
-allocation, on Qwen3.8-27B with its EAGLE-4 draft unless the table names another model. The page has three parts:
-[0.1.0](#010-released), [0.1.1](#011-released-current) and
+allocation, on Qwen3.8-27B with its EAGLE-4 draft unless the table names another model. The page has four parts:
+[0.1.0](#010-released), [0.1.1](#011-released), [0.1.2](#012-released-current) and
[unreleased candidates](#unreleased-measured-on-candidates-not-in-a-release). Shipped and candidate numbers never
share a row. Column headers are client concurrency, the number of requests the client keeps in flight.
@@ -130,7 +130,7 @@ test the published `0.1.0` images, each pullable without credentials, and the SG
`eagle-harness` 0.1.0 wheel from PyPI. TokenSpeed has no full-decode native top-logprobs yet, so each divergent
prefix in its checks goes directly to fp32 arbitration.
-## 0.1.1 (released, current)
+## 0.1.1 (released)
`eagle-harness` 0.1.1 with `sglang==0.5.23.dev252+eagle.5`, the unchanged `vllm==0.32.0rc1.dev283+eagle.4`, and
`tokenspeed==0.1.1+eagle.5` with `tokenspeed-kernel==0.1.4+eagle.5` and `tokenspeed-scheduler==0.1.40+eagle.4`.
@@ -215,6 +215,88 @@ The published 0.1.1 images (`eagle-4-sglang:0.1.1`, `eagle-4-vllm:0.1.1`, `eagle
checks as the 0.1.0 images: SGLang 5 exact / 15 near ties, vLLM 7 exact / 12 near ties / 1 fp32-tolerance pass, and
TokenSpeed in serve mode 6 exact / 8 fp32-tolerance passes / 6 fp32-argmax.
+## 0.1.2 (released, current)
+
+`eagle-harness` 0.1.2 with the 0.1.1 engines: `sglang==0.5.23.dev252+eagle.5`, `vllm==0.32.0rc1.dev283+eagle.4`,
+and `tokenspeed==0.1.1+eagle.5` with `tokenspeed-kernel==0.1.4+eagle.5` and `tokenspeed-scheduler==0.1.40+eagle.4`.
+Every row runs on NVIDIA B300, on the release candidate build of `eagle-harness` 0.1.2 installed into clean
+environments with the published engine wheels and served from outside any checkout. In the same allocations, 0.1.2
+measured between 0.98× and 1.07× the 0.1.1 packages in every cell where both ran. The SGLang ROCm install keeps its
+0.1.0 packages, so the 0.1.0 MI355X row applies.
+
+### 0.1.2 speed by engine and concurrency
+
+| Engine and setup | 1 | 8 | 64 |
+|---|---|---|---|
+| SGLang TP4, temperature 0 | ✅ 3.35× | ✅ 2.11× | ✅ 1.38× |
+| SGLang TP4, temperature 1 | ✅ 2.74× | ✅ 1.78× | ✅ 1.25× |
+| SGLang TP1, quick-start command | ✅ 2.87× | ✅ 1.87× | ⚠️ 0.73× |
+| vLLM TP4 | ✅ 2.83× | ✅ 1.73× | ⚠️ 1.19× |
+| TokenSpeed TP4 | ✅ 3.13× | ✅ 2.00× | ✅ 1.40× |
+
+Sources:
+
+1. SGLang TP4, temperature 0: the measured recipe of the [SGLang guide](quickstart-sglang.md#the-measured-recipe)
+ (running cap 32). EAGLE-4 733.8, 3257.9 and 6488.2 tokens/s, no-spec 219.0, 1545.2 and 4718.6, 2 starts per
+ arm. The 0.1.1 packages in the same allocation measured 3.36×, 2.11× and 1.40×. Every start gives 5 exact /
+ 15 near tie.
+2. SGLang TP4, temperature 1, same starts: EAGLE-4 595.6, 2739.8 and 5827.9 tokens/s, no-spec at temperature 1
+ 217.4, 1539.4 and 4675.2.
+3. SGLang TP1: the quick-start command on one GPU, 20, 64 and 384 SPEED-Bench first turns with up to 256 new
+ tokens. EAGLE-4 302.7, 1271.8 and 2096.3 tokens/s, no-spec 105.5, 679.9 and 2870.2, 2 starts per arm. The 0.1.1
+ packages measured 2.90×, 1.91× and 0.75×. The twenty-prompt and concurrency-64 checks pass. Concurrency 64 is a
+ [known limit](known-limits.md#high-load).
+4. vLLM TP4: the measured command of the [vLLM guide](quickstart-vllm.md) (running cap 64, packaged rounds file),
+ two allocations with 2 starts per arm each. Concurrency 1 is 22 first turns with up to 512 new tokens. EAGLE-4
+ 678.5, 2887.4 and 8013.4 tokens/s, no-spec 239.4, 1668.9 and 6725.4. Every twenty-prompt check passes. The
+ 64-prompt check at concurrency 64 shows the one-prompt divergence of 0.1.1, which the 0.1.1 packages also show in
+ the same allocation ([known limits](known-limits.md#performance-and-correctness-in-progress)).
+5. TokenSpeed TP4: serve mode with TokenSpeed cost tables, running cap 4 at 1 and 32 at 8 and 64, 2 starts per
+ arm. Concurrency 1 is 22 conversations and 31 turns with up to 1024 new tokens: 886.5 against 283.5 tokens/s.
+ Eleven first turns with up to 256 tokens give 819.7 against 280.7 (2.92×). Concurrency 8 and 64 use 22 and 88
+ first turns with up to 256 tokens: 3256.4 against 1628.2 and 6974.1 against 4969.8. Every start gives 6 exact /
+ 14 fp32 pass.
+
+### 0.1.2 high load
+
+SGLang TP4 with a running cap of 512: the [high-load recipe](quickstart-sglang.md#high-load), and the quick-start
+command with `--tp-size 4`. Concurrency 64 serves 384 SPEED-Bench first turns and higher concurrencies serve four
+times the concurrency, each with up to 256 new tokens.
+
+| Engine and setup | 64 | 128 | 256 | 512 |
+|---|---|---|---|---|
+| SGLang TP4, high-load recipe, first start | ⚠️ 0.95× | ✅ 0.97× | ✅ 1.00× | ⚠️ 0.99× |
+| SGLang TP4, high-load recipe, restart with warm caches | ⚠️ 1.01× | ✅ 1.00× | ✅ 1.00× | ⚠️ 0.99× |
+| SGLang TP4, quick-start command with `--tp-size 4` | ⚠️ 0.80× | ⚠️ 0.48× | ⚠️ 0.37× | ⚠️ 0.32× |
+| SGLang TP1, quick-start command | ⚠️ 0.73× | — | ⚠️ 0.59× | — |
+
+Sources:
+
+1. High-load recipe, first start on a machine (empty kernel and cost caches), 3 starts over three allocations,
+ each against the mean of 3 no-spec starts in its own allocation: EAGLE-4 6935.2, 10737.0, 14253.5 and
+ 16152.6 tokens/s, no-spec 7313.7, 11086.4, 14264.9 and 16340.7. At 1 and 8 concurrent requests it measured 2.80×
+ and 1.67× (704.3 and 2732.3 against 251.7 and 1634.7). Concurrency 64 ranges from 0.935× to 0.969× over the
+ three starts.
+2. Restart with the first start's caches, 3 starts in the same allocations: 7393.3, 11114.3, 14332.3 and
+ 16197.5 tokens/s, and 2.76× and 1.72× at 1 and 8. Every twenty-prompt check passes for both rows. The 384-prompt
+ check at 64 has one prompt outside the fp32 rule in 2 of 6 starts, at a prefix where the engine without
+ speculation emits the same token under another batch shape. The 2048-prompt check at 512 has one to three such
+ prompts in 3 of 6 starts, and one to three between two no-spec starts in every allocation.
+3. Quick-start command with `--tp-size 4`: 4377.7, 4723.1, 4490.4 and 4503.0 tokens/s against 5486.2, 9782.2,
+ 12034.3 and 14159.3, 2 starts per arm. The server admits 32 running requests (the default tree), and the radix
+ cache keeps the batch on speculation. The 0.1.1 packages measured 0.75×, 0.48×, 0.38× and 0.32×.
+4. TP1 quick-start command, the starts of the speed table: 2183.0 against 3691.8 tokens/s at 256. The 0.1.1
+ packages measured 0.75× and 0.60×. See [known limits](known-limits.md#high-load).
+
+### 0.1.2 correctness and request features
+
+| Change | Result |
+|---|---|
+| `eagle-harness calibrate --engine vllm` completes on the Qwen3.8 hybrid target, and its `serve.sh` loads the tables it wrote | ✅ 20/20 |
+
+The calibration ran on B300 TP4 with the 0.1.1 vLLM engine. Both the printed `serve.sh` and the quickstart command
+pass the twenty-prompt check (4 exact / 15 near tie / 1 fp32 pass), 1 start each. No speed is claimed for it.
+
## Unreleased (measured on candidates, not in a release)
These rows measure changes that are not in a published release. Each row names the change and its engine build.
@@ -299,15 +381,6 @@ Sources:
3. TokenSpeed: running cap 8, 11 and 22 first turns with up to 256 tokens, 2 starts per arm. 597.3 against 312.2 and
2122.4 against 1167.0 tokens/s. The check has 1 failure, reproduced by the engine without speculation.
-### Candidate correctness checks
-
-| Change | Result |
-|---|---|
-| `eagle-harness calibrate --engine vllm` completes on the Qwen3.8 hybrid target, and its `serve.sh` loads the tables it wrote | ✅ 20/20 |
-
-The calibration ran on B300 TP4 with the 0.1.1 vLLM engine. Both the printed `serve.sh` and the quickstart command
-pass the twenty-prompt check (4 exact / 15 near tie / 1 fp32 pass), 1 start each. No speed is claimed for it.
-
### In progress
1. vLLM sampled-distribution checks for settings other than temperature 1 with top-p 1.
diff --git a/eagle_harness/cli.py b/eagle_harness/cli.py
index d07ca55..4af76e7 100644
--- a/eagle_harness/cli.py
+++ b/eagle_harness/cli.py
@@ -114,6 +114,7 @@ def serving_command(args: argparse.Namespace) -> list[str]:
"data_path": str(args.output),
},
}
+ # max_num_batched_tokens stays vLLM's default, which fits one full batch (see eagle_harness.vllm.calibrate.engine).
options = {
"tensor_parallel_size": args.tp,
"host": "127.0.0.1",
@@ -122,7 +123,6 @@ def serving_command(args: argparse.Namespace) -> list[str]:
"max_model_len": 4608,
"gpu_memory_utilization": 0.85,
"max_num_seqs": 1,
- "max_num_batched_tokens": 8192,
"seed": 0,
}
options.update(args.engine_args)
diff --git a/eagle_harness/core/cost_profiles.py b/eagle_harness/core/cost_profiles.py
index 2c94850..754a042 100644
--- a/eagle_harness/core/cost_profiles.py
+++ b/eagle_harness/core/cost_profiles.py
@@ -1,17 +1,21 @@
# Copyright 2026 Eagle AI Corp.
# SPDX-License-Identifier: LicenseRef-Eagle-4-Source-Available
-"""Packaged serving costs for measured device and checkpoint configurations."""
+"""Packaged serving costs and choices for measured device and checkpoint configurations."""
from __future__ import annotations
import json
from pathlib import Path
-from typing import Optional
+from typing import Dict, Optional, Sequence
MI355X_PROFILE = Path(__file__).parent / "profiles" / "mi355x-qwen3.8-27b-tp1"
+B300_SERVED_PROFILE = Path(__file__).parent / "profiles" / "b300-qwen3.8-27b-tp4-chain"
-def packaged_cost_profile(
+def _measured_on(
+ profile: Path,
+ devices: Sequence[str],
+ tp: int,
*,
device_name: str,
model_path: str,
@@ -19,27 +23,85 @@ def packaged_cost_profile(
tp_size: int,
dtype: str,
quantization: Optional[str],
-) -> Optional[str]:
- """Select the MI355X BF16 TP1 profile only for its measured checkpoint configs.
-
- Model directory names do not participate in identity. Unknown devices,
- unresolved Hub paths, different configs, TP sizes or dtypes keep the caller's
- ordinary calibration path; a profile for one model must not price another.
- """
- if device_name not in ("AMD Instinct MI355X", "AMD Instinct MI355X VF"):
- return None
- if tp_size != 1 or dtype not in ("auto", "bfloat16", "torch.bfloat16") or quantization is not None:
- return None
+) -> bool:
+ """Whether a setup is the one ``profile`` was measured on: one of ``devices``, ``tp`` ranks, BF16 without
+ quantization, and the target and draft ``config.json`` of the profile (parsed: directory names do not matter)."""
+ if device_name not in devices or tp_size != tp:
+ return False
+ if dtype not in ("auto", "bfloat16", "torch.bfloat16") or quantization is not None:
+ return False
for model, name in ((model_path, "target"), (draft_model_path, "draft")):
if not model:
- return None
+ return False
config_path = Path(model) / "config.json"
if not config_path.is_file():
- return None
+ return False
with config_path.open(encoding="utf-8") as file:
config = json.load(file)
- with (MI355X_PROFILE / f"{name}_config.json").open(encoding="utf-8") as file:
+ with (profile / f"{name}_config.json").open(encoding="utf-8") as file:
measured = json.load(file)
if config != measured:
- return None
- return str(MI355X_PROFILE)
+ return False
+ return True
+
+
+def packaged_cost_profile(
+ *,
+ device_name: str,
+ model_path: str,
+ draft_model_path: str,
+ tp_size: int,
+ dtype: str,
+ quantization: Optional[str],
+) -> Optional[str]:
+ """Select the MI355X BF16 TP1 profile only for its measured checkpoint configs.
+
+ Unknown devices, unresolved Hub paths, different configs, TP sizes or dtypes keep the caller's
+ ordinary calibration path; a profile for one model must not price another.
+ """
+ if _measured_on(
+ MI355X_PROFILE,
+ ("AMD Instinct MI355X", "AMD Instinct MI355X VF"),
+ 1,
+ device_name=device_name,
+ model_path=model_path,
+ draft_model_path=draft_model_path,
+ tp_size=tp_size,
+ dtype=dtype,
+ quantization=quantization,
+ ):
+ return str(MI355X_PROFILE)
+ return None
+
+
+def packaged_served_choices(
+ kind: str,
+ *,
+ device_name: str,
+ model_path: str,
+ draft_model_path: str,
+ tp_size: int,
+ dtype: str,
+ quantization: Optional[str],
+) -> Optional[Dict[int, int]]:
+ """The served choice per batch key measured on this setup, or None: for a ``"chain"`` the verify width, for a
+ ``"tree"`` 1 to speculate (the cost model's tree) or 0 to decode natively.
+
+ Only the B300 BF16 TP4 setup of the profile's checkpoint pair has them. A served chain or tree
+ (:class:`~eagle_harness.sglang.chain_rounds.ServedChainPolicy`, ``ServedTreePolicy``) starts from them where it
+ has no choices of its own.
+ """
+ if not _measured_on(
+ B300_SERVED_PROFILE,
+ ("NVIDIA B300 SXM6 AC",),
+ 4,
+ device_name=device_name,
+ model_path=model_path,
+ draft_model_path=draft_model_path,
+ tp_size=tp_size,
+ dtype=dtype,
+ quantization=quantization,
+ ):
+ return None
+ with (B300_SERVED_PROFILE / f"{kind}.json").open(encoding="utf-8") as file:
+ return {int(key): int(choice) for key, choice in json.load(file)[kind].items()}
diff --git a/eagle_harness/core/live_costs.py b/eagle_harness/core/live_costs.py
index e8dd41d..6ff8b00 100644
--- a/eagle_harness/core/live_costs.py
+++ b/eagle_harness/core/live_costs.py
@@ -189,6 +189,10 @@ class LiveCosts:
# Seeding counts samples across intervals; crossing a boundary must still discard a first-use stall.
return min((c.us for (b, w, _), c in self.cells.items() if b == bucket and w == width and c.samples), default=None)
+ def verify_us(self, bucket: int, width: int) -> Optional[float]:
+ """The priced verify of ``width`` tokens per request at ``bucket`` (shortest context first), None untimed."""
+ return self._value(bucket, width, 0)
+
def tables(self) -> Dict[int, BatchCostTable]:
"""One table per ready bucket (the cost model's input), from the current cells."""
tables: Dict[int, BatchCostTable] = {}
diff --git a/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/README.md b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/README.md
new file mode 100644
index 0000000..1ee7c7e
--- /dev/null
+++ b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/README.md
@@ -0,0 +1,23 @@
+# B300 Qwen3.8-27B TP4 served chain widths and tree choices
+
+An SGLang chain without `data_path` (`"intensity": "chain_intensity"`, `--speculative-eagle-topk 1`) starts from
+these widths on an NVIDIA B300 SXM6 AC, BF16, TP4, with the measured target and draft checkpoint configurations.
+Selection compares parsed local `config.json` content with `target_config.json` and `draft_config.json`; directory
+names do not matter. Other devices, TP sizes, dtypes, quantization or model configurations search on their served
+rounds from the first round.
+
+`chain.json` maps a batch key (the nearest calibration batch size) to the verify width served there; 0 is native
+decode, used only when the engine captures native decode graphs (otherwise the server searches). The batch keys
+come from forced-width runs on SPEED-Bench prompts with 256-token outputs, matched against no-spec in the same job:
+width 4 at batch 16 and 64, native decode at 256. Native decode at a batch key also serves every larger one. The
+remaining batch keys are searched on the served rounds, never wider than a smaller key's width.
+
+`tree.json` does the same for a serve-mode tree (the default shape): 1 speculates (the cost model's tree), 0 decodes
+natively. Two cold starts with the default tree at a 512-request cap (SPEED-Bench prompts, 256-token outputs) chose
+speculation up to batch 8 and native decode at 64; native decode at 64 also serves every larger batch key, and
+speculation at 8 every smaller one. Batch keys 12 to 48 are searched on the served rounds.
+
+Choices the server makes itself (`<cost_cache>/<key>-chain.json`, `<key>-tree.json`) take precedence. A workload with very different
+acceptance or output lengths may favor other widths; calibrate it (`"mode": "calibrate"`) and serve the resulting
+`data_path`.
+This directory contains no model weights.
diff --git a/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/chain.json b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/chain.json
new file mode 100644
index 0000000..127d10e
--- /dev/null
+++ b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/chain.json
@@ -0,0 +1,7 @@
+{
+ "chain": {
+ "16": 4,
+ "64": 4,
+ "256": 0
+ }
+}
diff --git a/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/draft_config.json b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/draft_config.json
new file mode 100644
index 0000000..29dab0b
--- /dev/null
+++ b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/draft_config.json
@@ -0,0 +1,57 @@
+{
+ "_torchspec_version": "0.1.0",
+ "architectures": [
+ "LlamaForCausalLMEagle4"
+ ],
+ "attention_bias": false,
+ "attention_dropout": 0.0,
+ "bos_token_id": 248044,
+ "draft_vocab_size": 248320,
+ "dtype": "bfloat16",
+ "eagle_config": {
+ "eagle_aux_hidden_state_layer_ids": [
+ 3,
+ 31,
+ 59
+ ],
+ "use_aux_hidden_state": true
+ },
+ "efficient_lm_head_config": {
+ "num_candidates": 128,
+ "rank": 128,
+ "type": "low_rank"
+ },
+ "eos_token_id": 248044,
+ "fc_norm": true,
+ "head_dim": 128,
+ "hidden_act": "silu",
+ "hidden_size": 5120,
+ "initializer_range": 0.02,
+ "intermediate_size": 16384,
+ "layer_types": [
+ "sliding_attention"
+ ],
+ "max_position_embeddings": 262144,
+ "mlp_bias": false,
+ "model_type": "llama",
+ "norm_output": true,
+ "num_attention_heads": 40,
+ "num_aux_hidden_states": 3,
+ "num_hidden_layers": 1,
+ "num_key_value_heads": 40,
+ "pad_token_id": 248044,
+ "pretraining_tp": 1,
+ "rms_norm_eps": 1e-06,
+ "rope_parameters": {
+ "rope_theta": 10000000.0,
+ "rope_type": "default"
+ },
+ "sliding_window": 4096,
+ "target_hidden_size": 5120,
+ "tie_word_embeddings": false,
+ "transformers_version": "5.8.1",
+ "use_cache": true,
+ "use_sliding_window": true,
+ "vocab_size": 248320,
+ "torch_dtype": "bfloat16"
+}
\ No newline at end of file
diff --git a/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/target_config.json b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/target_config.json
new file mode 100644
index 0000000..706cebd
--- /dev/null
+++ b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/target_config.json
@@ -0,0 +1,140 @@
+{
+ "architectures": [
+ "Qwen3_5ForConditionalGeneration"
+ ],
+ "image_token_id": 248056,
+ "language_model_only": false,
+ "model_type": "qwen3_5",
+ "text_config": {
+ "attention_bias": false,
+ "attention_dropout": 0.0,
+ "attn_output_gate": true,
+ "bos_token_id": 248044,
+ "dtype": "bfloat16",
+ "eos_token_id": 248044,
+ "full_attention_interval": 4,
+ "head_dim": 256,
+ "hidden_act": "silu",
+ "hidden_size": 5120,
+ "initializer_range": 0.02,
+ "intermediate_size": 17408,
+ "layer_types": [
+ "linear_attention",
+ "linear_attention",
+ "linear_attention",
+ "full_attention",
+ "linear_attention",
+ "linear_attention",
+ "linear_attention",
+ "full_attention",
+ "linear_attention",
+ "linear_attention",
+ "linear_attention",
+ "full_attention",
+ "linear_attention",
+ "linear_attention",
+ "linear_attention",
+ "full_attention",
+ "linear_attention",
+ "linear_attention",
+ "linear_attention",
+ "full_attention",
+ "linear_attention",
+ "linear_attention",
+ "linear_attention",
+ "full_attention",
+ "linear_attention",
+ "linear_attention",
+ "linear_attention",
+ "full_attention",
+ "linear_attention",
+ "linear_attention",
+ "linear_attention",
+ "full_attention",
+ "linear_attention",
+ "linear_attention",
+ "linear_attention",
+ "full_attention",
+ "linear_attention",
+ "linear_attention",
+ "linear_attention",
+ "full_attention",
+ "linear_attention",
+ "linear_attention",
+ "linear_attention",
+ "full_attention",
+ "linear_attention",
+ "linear_attention",
+ "linear_attention",
+ "full_attention",
+ "linear_attention",
+ "linear_attention",
+ "linear_attention",
+ "full_attention",
+ "linear_attention",
+ "linear_attention",
+ "linear_attention",
+ "full_attention",
+ "linear_attention",
+ "linear_attention",
+ "linear_attention",
+ "full_attention",
+ "linear_attention",
+ "linear_attention",
+ "linear_attention",
+ "full_attention"
+ ],
+ "linear_conv_kernel_dim": 4,
+ "linear_key_head_dim": 128,
+ "linear_num_key_heads": 16,
+ "linear_num_value_heads": 48,
+ "linear_value_head_dim": 128,
+ "mamba_ssm_dtype": "float32",
+ "max_position_embeddings": 262144,
+ "model_type": "qwen3_5_text",
+ "mtp_num_hidden_layers": 1,
+ "mtp_use_dedicated_embeddings": false,
+ "num_attention_heads": 24,
+ "num_hidden_layers": 64,
+ "num_key_value_heads": 4,
+ "output_gate_type": "swish",
+ "pad_token_id": null,
+ "partial_rotary_factor": 0.25,
+ "rms_norm_eps": 1e-06,
+ "rope_parameters": {
+ "mrope_interleaved": true,
+ "mrope_section": [
+ 11,
+ 11,
+ 10
+ ],
+ "partial_rotary_factor": 0.25,
+ "rope_theta": 10000000,
+ "rope_type": "default"
+ },
+ "tie_word_embeddings": false,
+ "use_cache": true,
+ "vocab_size": 248320
+ },
+ "tie_word_embeddings": false,
+ "transformers_version": "5.8.0.dev0",
+ "video_token_id": 248057,
+ "vision_config": {
+ "deepstack_visual_indexes": [],
+ "depth": 27,
+ "hidden_act": "gelu_pytorch_tanh",
+ "hidden_size": 1152,
+ "in_channels": 3,
+ "initializer_range": 0.02,
+ "intermediate_size": 4304,
+ "model_type": "qwen3_5",
+ "num_heads": 16,
+ "num_position_embeddings": 2304,
+ "out_hidden_size": 5120,
+ "patch_size": 16,
+ "spatial_merge_size": 2,
+ "temporal_patch_size": 2
+ },
+ "vision_end_token_id": 248054,
+ "vision_start_token_id": 248053
+}
\ No newline at end of file
diff --git a/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/tree.json b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/tree.json
new file mode 100644
index 0000000..e97f609
--- /dev/null
+++ b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/tree.json
@@ -0,0 +1,13 @@
+{
+ "tree": {
+ "1": 1,
+ "2": 1,
+ "3": 1,
+ "4": 1,
+ "5": 1,
+ "6": 1,
+ "7": 1,
+ "8": 1,
+ "64": 0
+ }
+}
diff --git a/eagle_harness/models/low_rank_head.py b/eagle_harness/models/low_rank_head.py
index 361e666..fbe59d7 100644
--- a/eagle_harness/models/low_rank_head.py
+++ b/eagle_harness/models/low_rank_head.py
@@ -34,7 +34,11 @@ def low_rank_scores(
) -> torch.Tensor:
"""``[rows, shard vocab]`` rank-r scores of one vocab shard (``down_proj`` [rank, H], ``score_head`` [shard vocab,
rank]), ``-inf`` past the shard's ``vocab_rows`` real rows."""
- low_rank = hidden @ down_proj.t()
+ return shard_scores(hidden @ down_proj.t(), score_head, vocab_rows)
+
+
+def shard_scores(low_rank: torch.Tensor, score_head: torch.Tensor, vocab_rows: int) -> torch.Tensor:
+ """:func:`low_rank_scores` from the rank-r projection ``low_rank`` [rows, rank] the caller computed."""
scores = low_rank.to(score_head.dtype) @ score_head.t()
if scores.shape[-1] > vocab_rows:
scores[:, vocab_rows:] = float("-inf")
diff --git a/eagle_harness/sglang/algorithm.py b/eagle_harness/sglang/algorithm.py
index c1fb8b5..d874191 100644
--- a/eagle_harness/sglang/algorithm.py
+++ b/eagle_harness/sglang/algorithm.py
@@ -138,9 +138,14 @@ class EagleHarnessSpecAlgo(CustomSpecAlgo):
load_round_times(self.config.data_path)
if self.config.mode is Mode.SERVE and self.config.data_path else None
)
- plain = get_spec().speculative_eagle_topk == 1 and (
- self.config.mode is Mode.CALIBRATE or any(0 in row for row in (rounds or {}).values())
- )
+ from eagle_harness.sglang.chain_rounds import served_chain, served_tree
+ from eagle_harness.sglang.plain_decode import native_decode_available
+
+ topk = get_spec().speculative_eagle_topk
+ plain = (
+ topk == 1
+ and (self.config.mode is Mode.CALIBRATE or any(0 in row for row in (rounds or {}).values()))
+ ) or ((served_chain(self.config, topk) or served_tree(self.config, topk)) and native_decode_available())
self.target_verify_graphs = VerifyWidthGraphs(
self.config.intensity, max_width, TreeTailMask(info.device), plain_decode=plain
)
diff --git a/eagle_harness/sglang/chain_rounds.py b/eagle_harness/sglang/chain_rounds.py
index 662aa4b..f3e5359 100644
--- a/eagle_harness/sglang/chain_rounds.py
+++ b/eagle_harness/sglang/chain_rounds.py
@@ -25,6 +25,8 @@ serve-mode chain server with the same ``data_path`` reads.
from __future__ import annotations
import json
+import logging
+import math
import os
import statistics
import time
@@ -34,14 +36,41 @@ from typing import TYPE_CHECKING, Dict, List, Sequence, Tuple
import torch
-from eagle_harness.core.chain_width import save_accept_profile, save_round_times, width_at
+from eagle_harness.core.chain_width import AcceptProfile, save_accept_profile, save_round_times, width_at
+from eagle_harness.core.config import HarnessConfig, Mode
+from eagle_harness.core.limits import ComputeIntensity, cost_batch_sizes, nearest_cost_batch_size
if TYPE_CHECKING:
from sglang.srt.managers.schedule_batch import ScheduleBatch
+logger = logging.getLogger(__name__)
+
BLOCK_ROUNDS = 8
FLUSH_ROUNDS = 256
MIN_ROUNDS = 8
+# Whole-interval periods of one width before two widths are compared. Under sustained load the periods that hold
+# a prefill are several times the others (coefficient of variation ~0.5), so the mean of 32 is within ~9%: enough
+# to separate native decode from speculation where the choice matters, short enough to settle in a first pass.
+EXPLORE_ROUNDS = 32
+# A period in which more than 1/BURST_SHARE of the batch joined holds a burst of arrivals (the load changed), not the
+# refills of sustained load (tokens per round / output length: under 1/8 for outputs over 64 tokens at width <= 8).
+# Its prefills are the same work under every width and, landing in one width's few periods, would decide the choice.
+BURST_SHARE = 8
+# Speculation serves a batch key only when it beats native decode by this many standard errors of the difference of
+# the two measured rates: a wrong native choice costs only the speculative gain, a wrong speculative one runs slower
+# than plain decoding.
+NATIVE_MARGIN_SE = 2.0
+# Speculation that beats native decode's estimate (the batch's one-token verify in the live costs) by this factor,
+# beyond NATIVE_MARGIN_SE standard errors, is not measured against native decode. Requests decoded natively cannot
+# resume drafting and a batch holding one decodes natively, so measuring native decode keeps the batch native until it
+# empties: under sustained load, for good. The estimate is a model of native decode, not a measurement (served native
+# rounds ran 1.1x it at batch key 64 on B300), hence the wide factor.
+NATIVE_SKIP_RATIO = 1.5
+# A period longer than STALL_FACTOR times its cell's median holds one-time work, not the served load: a kernel compile
+# or graph capture on a cold cache (1-3 s on B300 TP4, against 5-15 ms rounds). The refill prefills of sustained load
+# stay within a few medians. One such period in a 32-period cell moves its rate several-fold (native decode at batch
+# key 32 read 0.28 tokens/ms against 5.6 served), so it is counted, not priced.
+STALL_FACTOR = 10
RAW_FILE = os.path.join("raw", "sglang_rounds.json")
@@ -157,12 +186,439 @@ class ChainCalibration:
save_accept_profile(self._data_path, drafts, per_position)
+def served_chain(config: HarnessConfig, topk: int) -> bool:
+ """Whether a server chooses its chain widths on its served rounds (:class:`ServedChainPolicy`): a ``serve``-mode
+ chain without ``data_path``."""
+ return (
+ config.mode is Mode.SERVE and not config.data_path and config.intensity is ComputeIntensity.CHAIN and topk == 1
+ )
+
+
+def served_tree(config: HarnessConfig, topk: int) -> bool:
+ """Whether a tree chooses between speculation and native decode on its served rounds (:class:`ServedTreePolicy`):
+ every ``serve``-mode tree (top-k > 1), the default configuration; the engine must also have the native route
+ (:func:`~eagle_harness.sglang.plain_decode.native_decode_available`)."""
+ return config.mode is Mode.SERVE and topk > 1
+
+
+def _ratio_rate(samples: Sequence[Tuple[float, float]]) -> Tuple[float, float]:
+ """Tokens per microsecond over periods ``(us, tokens)`` (the ratio of the sums) and its variance (ratio
+ estimator; infinite below two periods)."""
+ n = len(samples)
+ us = sum(u for u, _ in samples)
+ if n == 0 or us <= 0:
+ return 0.0, math.inf
+ rate = sum(t for _, t in samples) / us
+ if n < 2:
+ return rate, math.inf
+ mean_us = us / n
+ residual = sum((t - rate * u) ** 2 for u, t in samples)
+ return rate, residual / (n * (n - 1) * mean_us**2)
+
+
+def _settled(samples: Sequence[tuple]) -> List[tuple]:
+ """The periods ``(us, ...)`` of a cell without its stalls (:data:`STALL_FACTOR`)."""
+ if len(samples) < 2:
+ return list(samples)
+ limit = STALL_FACTOR * statistics.median(sample[0] for sample in samples)
+ return [sample for sample in samples if sample[0] <= limit]
+
+
+class ServedChainPolicy:
+ """The chain width per batch size of a ``serve``-mode chain without ``data_path``, chosen on its served rounds.
+
+ Price: tokens per wall-clock time over whole served intervals. A round's period runs from its start to the next
+ decode round's start, so the prefill steps between them are in it, and so are the costs a chain pays around them
+ (a prelaunched draft dropped at a prefill or a batch change, the next round drafting from scratch). Common prefill
+ work costs every action the same per committed token, so it does not move the choice; dropping those periods
+ (as :class:`ChainCalibration` does) prices chains on the rounds where they are cheapest. A period is cut where
+ the batch turned over completely (the server may have idled in between), and one that holds a burst of arrivals
+ (:data:`BURST_SHARE`), a stall
+ (:data:`STALL_FACTOR`) or the first round after a switch of width (it also finishes the previous width's work)
+ counts toward the comparison without being priced.
+
+ Search, per batch key (the nearest calibration batch size, :data:`~eagle_harness.core.limits.COST_BATCH_SIZES`):
+ width 1 against the next wider captured width, then wider while the wider one wins (tokens per time is unimodal in
+ the width: acceptance grows concavely, the verify cost convexly), never wider than a smaller batch key chose.
+ Each comparison interleaves the two widths in
+ blocks of :data:`BLOCK_ROUNDS` until each has :data:`EXPLORE_ROUNDS` periods, so both see the same load. With
+ ``native_decode``, the search starts at width 2 (width 1 commits the same token as native decode with more work)
+ and native decode (width 0) is measured last: its requests cannot resume drafting, so it runs right after a fresh
+ block of the best width (both again after a complete turnover, so they see the same load), and the batch stays
+ native until those requests finish (and the ones that join it meanwhile). So with ``native_us`` (native decode's
+ microseconds per round at a batch size, from the live costs), speculation that beats that estimate by
+ :data:`NATIVE_SKIP_RATIO` is served without measuring native decode. Speculation loses more as the
+ batch grows (the verify becomes compute-bound), so where native decode wins every larger batch key takes it
+ unmeasured, and where speculation wins no smaller batch key measures native decode.
+
+ Decisions are made on TP rank 0 and broadcast (``sync``) at the round that completes a comparison, the same
+ round on every rank. Rank 0 keeps them in ``cache_path`` (keyed like the live costs), so a restart on the same
+ setup serves them from its first round. Without that file, a first start serves the ``packaged`` choices of its
+ setup (:func:`~eagle_harness.core.cost_profiles.packaged_served_choices`) and searches only the other batch keys.
+ shortcut: a decision is not revisited when the workload's acceptance changes; re-explore (delete the file) after
+ changing the served workload class.
+ """
+
+ def __init__(
+ self,
+ widths: Sequence[int],
+ device: torch.device,
+ *,
+ max_batch_size: int,
+ native_decode: bool,
+ writer: bool,
+ sync: Callable[[object], object],
+ cache_path: str | None = None,
+ key: dict | None = None,
+ packaged: Dict[int, int] | None = None,
+ native_us: Callable[[int], float | None] | None = None,
+ ):
+ self.widths = sorted(widths)
+ self._native = native_decode
+ self._native_us = native_us
+ self._native_checked: set = set() # batch keys whose speculation was checked against native decode's estimate
+ self._writer = writer
+ self._sync = sync
+ self._path = cache_path
+ self._identity = {"key": key, "widths": self.widths, "native_decode": native_decode}
+ self._grid = cost_batch_sizes(max_batch_size) or [max_batch_size]
+ self._key = [0] + [nearest_cost_batch_size(bs, self._grid) for bs in range(1, max_batch_size + 1)]
+ self.decided: Dict[int, int] = {} # batch key -> served width (0: native decode)
+ self._pair: Dict[int, Tuple[int, int]] = {} # batch key -> (best width so far, challenger)
+ self._started: Dict[int, int] = defaultdict(int) # rounds of the comparison in progress, per batch key
+ self._cells: Dict[Tuple[int, int], List[float]] = {} # (batch key, width) -> [period us, requests, periods]
+ # (batch key, width) -> the priced periods: (us, requests, tokens committed: a device scalar, trees only)
+ self._samples: Dict[Tuple[int, int], List[Tuple[float, int, object]]] = defaultdict(list)
+ # key, width, start ns, bs, request ids, whether the round is priced (not a live-cost seed round), whether it
+ # switched action (its period also holds the previous action's tail: a dropped prelaunched draft, the switch)
+ self._previous: Tuple[int, int, int, int, frozenset, bool, bool] | None = None
+ self._last_width: int | None = None # the width of the last round seen
+ self._accept = torch.zeros(self.widths[-1] + 1, self.widths[-1] + 1, dtype=torch.int64, device=device)
+ if not self._load() and packaged:
+ self._start_from(packaged, "the packaged choices of this setup")
+
+ # ---- the served width
+ def width(self, bs: int) -> int:
+ """The width of a round of ``bs`` requests (0: native decode)."""
+ key = self._key[min(bs, len(self._key) - 1)]
+ if key in self.decided:
+ return self.decided[key]
+ # With native decode, width 1 (the same tokens with more work) never wins: the search starts above it.
+ first = self.widths[1] if self._native and len(self.widths) > 2 else self.widths[0]
+ best, challenger = self._pair.setdefault(key, (first, self._wider(first)))
+ if challenger == 0: # the best width's periods, then native decode's right after them
+ return 0 if self._cells.setdefault((key, best), [0.0, 0, 0])[2] >= EXPLORE_ROUNDS else best
+ return challenger if (self._started[key] // BLOCK_ROUNDS) % 2 else best
+
+ def _wider(self, width: int) -> int:
+ """The next wider width; past the widest, native decode (0)."""
+ return next((w for w in self.widths if w > width), 0)
+
+ def exploring(self, bs: int) -> bool:
+ return self._key[min(bs, len(self._key) - 1)] not in self.decided
+
+ # ---- recording
+ def round_start(self, bs: int, width: int, requests: Sequence, settled: bool = True) -> None:
+ """A decode round of ``bs`` requests at ``width`` starts: the previous round's whole interval ends.
+ ``settled``: False for a round the drafter forces to seed its live costs, whose period is not counted."""
+ now = time.perf_counter_ns()
+ key = self._key[min(bs, len(self._key) - 1)]
+ previous, self._previous = self._previous, None
+ switched, self._last_width = self._last_width not in (None, width), width
+ if not self._pair and key in self.decided:
+ return
+ ids = frozenset(req.rid for req in requests)
+ self._previous = (key, width, now, bs, ids, settled, switched)
+ if previous is not None and previous[4].isdisjoint(ids):
+ # The batch turned over (the server may have idled): native decode and the block of the best width it is
+ # compared with are measured again together, under the load that follows.
+ for k, pair in self._pair.items():
+ if pair[1] == 0:
+ self._clear(k, pair[0])
+ self._clear(k, 0)
+ elif previous is not None and previous[5] and previous[1] in self._pair.get(previous[0], ()):
+ cell = self._cells.setdefault(previous[:2], [0.0, 0, 0])
+ cell[2] += 1
+ # A burst of arrivals, or the first round after a switch of action, is counted, not priced.
+ if len(ids - previous[4]) * BURST_SHARE <= previous[3] and not previous[6]:
+ us = (now - previous[2]) / 1000.0
+ cell[0] += us
+ cell[1] += previous[3]
+ self._samples[previous[:2]].append((us, previous[3], self._period_tokens(previous[1])))
+ if key in self.decided:
+ return
+ self._started[key] += 1
+ pair = self._pair.get(key)
+ if pair is None:
+ return
+ periods = [self._cells.setdefault((key, w), [0.0, 0, 0])[2] for w in pair]
+ if all(n >= EXPLORE_ROUNDS for n in periods):
+ self._compare(key, pair)
+ elif (
+ self._native_us is not None
+ and pair[1] == 0
+ and periods[0] >= EXPLORE_ROUNDS
+ and not periods[1]
+ and key not in self._native_checked
+ ):
+ self._check_native(key, pair[0])
+
+ def accepted(self, width: int, accept_lens: torch.Tensor) -> None:
+ """Count a round's accept lengths (tokens committed per request) under its width (rank 0 decides)."""
+ if width >= 1 and self._writer: # index_add_, not bincount: bincount sizes its output from a device read
+ lens = accept_lens.to(torch.int64)
+ self._accept[width].index_add_(0, lens, torch.ones_like(lens))
+
+ def _period_tokens(self, width: int) -> object:
+ """The tokens a period's round committed, when the policy measures them (trees); None: priced from the
+ acceptance profile."""
+ return None
+
+ def _clear(self, key: int, width: int) -> None:
+ self._cells[(key, width)] = [0.0, 0, 0]
+ self._samples.pop((key, width), None)
+ if width == 0:
+ self._native_checked.discard(key)
+
+ # ---- decisions
+ def profile(self) -> AcceptProfile | None:
+ """Acceptance by drafted position from every served width: a round of width ``w`` drafted positions
+ ``1 .. w - 1``, and accepted position ``d`` when its accept length exceeds ``d``."""
+ hist = self._accept.tolist()
+ rates = []
+ for d in range(1, len(hist) - 1):
+ drafted = sum(sum(row) for row in hist[d + 1 :])
+ if not drafted:
+ break
+ rates.append(sum(sum(row[d + 1 :]) for row in hist[d + 1 :]) / drafted)
+ return AcceptProfile(tuple(rates)) if rates else None
+
+ def _rates(self, key: int, pair: Tuple[int, int]) -> Dict[int, Tuple[float, float]]:
+ """Committed tokens per microsecond of the pair's widths and the variance of each (native decode and width
+ 1 commit one token per request; wider chains the profile's expected tokens)."""
+ profile = self.profile()
+ covered = len(profile.rates) + 1 if profile else 1
+ rates = {}
+ for width in pair:
+ if width <= covered and self._cells[(key, width)][0] > 0:
+ tokens = profile.expected_tokens(width) if width > 1 else 1.0
+ rates[width] = _ratio_rate([(us, tokens * n) for us, n, _ in _settled(self._samples[(key, width)])])
+ return rates
+
+ @staticmethod
+ def _better(rates: Dict[int, Tuple[float, float]], best: int, challenger: int) -> int:
+ """The winner of a comparison. Against native decode, speculation must win by :data:`NATIVE_MARGIN_SE`
+ standard errors."""
+ (best_rate, best_var), (rate, var) = rates.get(best, (0.0, math.inf)), rates.get(challenger, (0.0, math.inf))
+ if challenger != 0:
+ return challenger if rate > best_rate else best
+ margin = NATIVE_MARGIN_SE * math.sqrt(best_var + var)
+ return best if best_rate - rate > margin else 0
+
+ def _compare(self, key: int, pair: Tuple[int, int]) -> None:
+ """Both widths of the pair have their periods: rank 0 decides the next step, every rank applies it."""
+ step = None
+ if self._writer:
+ rates = self._rates(key, pair)
+ best, challenger = pair
+ best = self._better(rates, best, challenger)
+ # A larger batch never verifies wider than a smaller one chose, and native decode where speculation won
+ # at a larger batch would only lose (and stick).
+ widest = min((w for k, w in self.decided.items() if k < key and w >= 1), default=self.widths[-1])
+ spec_won_above = any(w >= 1 and k >= key for k, w in self.decided.items())
+ if best != 0 and challenger != 0 and best == challenger and best < widest:
+ step = ("compare", best, self._wider(best))
+ elif best != 0 and challenger != 0 and self._native and not spec_won_above:
+ step = ("compare", best, 0)
+ else:
+ step = ("decide", best, None)
+ logger.info(
+ "Served chain rates at batch key %d (tokens per ms): %s -> %s",
+ key,
+ {w: (round(1000 * r, 3), round(1000 * math.sqrt(v), 3)) for w, (r, v) in sorted(rates.items())},
+ step,
+ )
+ kind, width, challenger = self._sync(step)
+ self._started[key] = 0
+ for w in pair:
+ self._clear(key, w)
+ if kind == "compare":
+ self._pair[key] = (width, challenger)
+ return
+ self._decide(key, width)
+
+ def _decide(self, key: int, width: int) -> None:
+ self._pair.pop(key, None)
+ self.decided[key] = width
+ self._infer()
+ self._save()
+
+ def _native_estimate(self, key: int, width: int) -> float | None:
+ """Native decode's tokens per microsecond over the batches of ``width``'s periods at ``key``, priced by
+ ``native_us``; None where it has no price."""
+ if self._native_us is None:
+ return None
+ us = requests = 0.0
+ for _, n, _ in _settled(self._samples[(key, width)]):
+ price = self._native_us(n)
+ if price is None or price <= 0:
+ return None
+ us += price
+ requests += n
+ return requests / us if us else None
+
+ def _check_native(self, key: int, best: int) -> None:
+ """``best`` has its periods before native decode's: rank 0 decides whether it beats native decode's estimate
+ so clearly (:data:`NATIVE_SKIP_RATIO`) that native decode is not measured, and every rank applies it."""
+ self._native_checked.add(key)
+ step = None
+ if self._writer:
+ rate, var = self._rates(key, (best,)).get(best, (0.0, math.inf))
+ native = self._native_estimate(key, best)
+ won = native is not None and rate - NATIVE_MARGIN_SE * math.sqrt(var) > NATIVE_SKIP_RATIO * native
+ step = ("decide", best, None) if won else ("compare", best, 0)
+ logger.info(
+ "Served chain rate at batch key %d (tokens per ms): %s, native decode estimate %s -> %s",
+ key,
+ (round(1000 * rate, 3), round(1000 * math.sqrt(var), 3)),
+ None if native is None else round(1000 * native, 3),
+ step,
+ )
+ kind, width, _ = self._sync(step)
+ if kind == "decide":
+ self._started[key] = 0
+ self._clear(key, width)
+ self._clear(key, 0)
+ self._decide(key, width)
+
+ def _infer(self) -> None:
+ """Native decode at a batch key serves every larger one unmeasured."""
+ for key in self._grid:
+ if key not in self.decided and any(w == 0 for k, w in self.decided.items() if k < key):
+ self.decided[key] = 0
+ self._pair.pop(key, None)
+
+ # ---- persistence
+ def _start_from(self, decided: Dict[int, int], source: str) -> bool:
+ """Serve ``decided`` (batch key -> width) if every width is an action of this server."""
+ decided = {k: w for k, w in decided.items() if k in self._grid}
+ if any(w not in (0, *self.widths) or (w == 0 and not self._native) for w in decided.values()):
+ logger.info("Served chain widths %s from %s are not actions of this server; exploring.", decided, source)
+ return False
+ self.decided = dict(decided)
+ self._infer()
+ logger.info("Served chain widths per batch key from %s: %s", source, self.decided)
+ return True
+
+ def _load(self) -> bool:
+ if self._path is None:
+ return False
+ try:
+ with open(self._path, encoding="utf-8") as f:
+ data = json.load(f)
+ if {k: data.get(k) for k in self._identity} != self._identity:
+ logger.info("Served chain widths in %s are for another setup; exploring again.", self._path)
+ return False
+ decided = {int(k): int(w) for k, w in data["chain"].items()}
+ except (OSError, ValueError, TypeError, KeyError, AttributeError) as e:
+ logger.info("Served chain widths %s unavailable or invalid (%s); exploring.", self._path, e)
+ return False
+ return self._start_from(decided, self._path)
+
+ def _save(self) -> None:
+ if not (self._writer and self._path):
+ return
+ try:
+ os.makedirs(os.path.dirname(self._path) or ".", exist_ok=True)
+ with open(f"{self._path}.tmp", "w", encoding="utf-8") as f:
+ json.dump({**self._identity, "chain": {str(k): w for k, w in sorted(self.decided.items())}}, f)
+ os.replace(f"{self._path}.tmp", self._path)
+ except OSError as e:
+ logger.warning("Could not write the served chain widths %s: %s", self._path, e)
+
+
+class ServedTreePolicy(ServedChainPolicy):
+ """Speculation or native decode per batch size for a ``serve``-mode tree, chosen on its served rounds.
+
+ The cost model sizes every speculative round (its verify width included); this policy decides only whether a batch
+ key speculates at all. Speculation is the one action ``spec_width`` (the tree's maximum verify width, whatever
+ width each round verifies), compared with native decode like the best chain width of :class:`ServedChainPolicy`:
+ the same whole served intervals, the same order (native decode last, sticky), the same persistence. Its price is
+ the tokens its rounds committed (the sum of their accept lengths, kept on the GPU until the comparison), since a
+ tree's acceptance has no per-position profile. A round that seeds the live costs (``settled=False``) forces a
+ width the cost model would not choose and is not counted.
+
+ Speculation gains less as the batch grows, so native decode at a batch key serves every larger key and
+ speculation at a batch key every smaller one, both unmeasured. ``packaged`` choices map a batch key to 1
+ (speculate) or 0 (native decode).
+ """
+
+ def __init__(
+ self,
+ spec_width: int,
+ device: torch.device,
+ *,
+ max_batch_size: int,
+ writer: bool,
+ sync: Callable[[object], object],
+ cache_path: str | None = None,
+ key: dict | None = None,
+ packaged: Dict[int, int] | None = None,
+ native_us: Callable[[int], float | None] | None = None,
+ ):
+ self._tokens: torch.Tensor | None = None # the committed tokens of the last speculative round
+ super().__init__(
+ [spec_width],
+ device,
+ max_batch_size=max_batch_size,
+ native_decode=True,
+ writer=writer,
+ sync=sync,
+ cache_path=cache_path,
+ key=key,
+ packaged=None if packaged is None else {k: spec_width if c else 0 for k, c in packaged.items()},
+ native_us=native_us,
+ )
+
+ def round_start(self, bs: int, width: int, requests: Sequence, settled: bool = True) -> None:
+ super().round_start(bs, self.widths[0] if width else 0, requests, settled)
+
+ def accepted(self, width: int, accept_lens: torch.Tensor) -> None:
+ if width >= 1 and self._writer:
+ self._tokens = accept_lens.sum()
+
+ def _period_tokens(self, width: int) -> object:
+ return self._tokens if width and self._writer else None
+
+ def _rates(self, key: int, pair: Tuple[int, int]) -> Dict[int, Tuple[float, float]]:
+ rates = {}
+ for width in pair:
+ samples = _settled(self._samples[(key, width)])
+ if not samples:
+ continue
+ if width:
+ tokens = torch.stack([t for _, _, t in samples]).tolist() # the comparison's one device read
+ rates[width] = _ratio_rate([(us, t) for (us, _, _), t in zip(samples, tokens)])
+ else:
+ rates[width] = _ratio_rate([(us, n) for us, n, _ in samples])
+ return rates
+
+ def _infer(self) -> None:
+ super()._infer()
+ spec = self.widths[0]
+ for key in self._grid:
+ if key not in self.decided and any(w == spec for k, w in self.decided.items() if k > key):
+ self.decided[key] = spec
+ self._pair.pop(key, None)
+
+
class ChainRounds:
"""The served/calibrated chain width and optional recording of completed rounds.
Drafting asks for ``draft_width`` once when it queues a round. Native decode
reads ``plain_width`` without advancing that plan, including a prelaunched
choice of OFF. Context is read only for calibration, without a device sync.
+ A ``policy`` (:class:`ServedChainPolicy`) chooses the width of each batch on the served rounds instead.
"""
def __init__(
@@ -170,26 +626,38 @@ class ChainRounds:
schedule: Sequence[tuple[int, int, int]] | None,
calibration: ChainCalibration | None,
average_context: Callable[[ScheduleBatch], int],
+ policy: ServedChainPolicy | ServedTreePolicy | None = None,
):
self._schedule = schedule
self._calibration = calibration
self._average_context = average_context
+ self._policy = policy
def draft_width(self, bs: int) -> int:
- width = self._calibration.width(bs) if self._calibration is not None else width_at(self._schedule, bs)
+ if self._policy is not None:
+ width = self._policy.width(bs)
+ else:
+ width = self._calibration.width(bs) if self._calibration is not None else width_at(self._schedule, bs)
return max(1, width)
def plain_width(self, bs: int) -> int:
+ if self._policy is not None:
+ return self._policy.width(bs) # this batch's own choice: a prelaunch planned the previous batch
return self._calibration.last_width if self._calibration is not None else width_at(self._schedule, bs)
def interrupt(self) -> None:
if self._calibration is not None:
self._calibration.interrupt()
- def round_start(self, bs: int, width: int, batch: ScheduleBatch) -> None:
+ def round_start(self, bs: int, width: int, batch: ScheduleBatch, settled: bool = True) -> None:
+ """A decode round starts; ``settled``: False for a round forced to seed the live costs."""
if self._calibration is not None:
self._calibration.round_start(bs, width, self._average_context(batch))
+ if self._policy is not None:
+ self._policy.round_start(bs, width, batch.reqs, settled)
def accepted(self, width: int, accept_lens: torch.Tensor) -> None:
if self._calibration is not None:
self._calibration.accepted(width, accept_lens)
+ if self._policy is not None and self._policy.exploring(len(accept_lens)):
+ self._policy.accepted(width, accept_lens)
diff --git a/eagle_harness/sglang/drafter.py b/eagle_harness/sglang/drafter.py
index e177bbb..0531a6a 100644
--- a/eagle_harness/sglang/drafter.py
+++ b/eagle_harness/sglang/drafter.py
@@ -267,6 +267,11 @@ class DynamicDrafter:
self._sampled_depth = tree.shape.max_depth
# ----------------------------------------------------------------- policy
+ @property
+ def seeding(self) -> bool:
+ """Whether the last drafted round was forced to seed the live costs of its batch bucket."""
+ return self._seeding is not None
+
def bucket(self, bs: int) -> int:
"""The cost bucket (captured batch size) a batch of ``bs`` requests runs at."""
return self._bucket(bs)
diff --git a/eagle_harness/sglang/plain_decode.py b/eagle_harness/sglang/plain_decode.py
index d0a1b34..a4ca5a1 100644
--- a/eagle_harness/sglang/plain_decode.py
+++ b/eagle_harness/sglang/plain_decode.py
@@ -23,6 +23,24 @@ if TYPE_CHECKING:
from eagle_harness.sglang.chain_rounds import ChainRounds
+def native_decode_available() -> bool:
+ """Whether native decode can serve requests that stop drafting: the engine captures native DECODE graphs among
+ the speculative graph variants, and neither Mamba prefix-cache tracking nor DP attention needs the draft's state."""
+ from sglang.srt.runtime_context import get_exec, get_parallel
+
+ from eagle_harness.sglang.verify_graphs import native_decode_graphs
+
+ return native_decode_graphs() and not get_exec().mamba.enable_mamba_extra_buffer and get_parallel().dp_size == 1
+
+
+def native_decode_blocked() -> bool:
+ """Whether the engine has native DECODE graphs but this server's settings (Mamba prefix-cache tracking, DP
+ attention) keep requests from switching to them."""
+ from eagle_harness.sglang.verify_graphs import native_decode_graphs
+
+ return native_decode_graphs() and not native_decode_available()
+
+
class NoSpecRequests:
"""Keep a request off until it finishes; skipped draft KV is never reused."""
diff --git a/eagle_harness/sglang/verify_graphs.py b/eagle_harness/sglang/verify_graphs.py
index 58f9924..ad64d95 100644
--- a/eagle_harness/sglang/verify_graphs.py
+++ b/eagle_harness/sglang/verify_graphs.py
@@ -28,6 +28,11 @@ from eagle_harness.sglang.verify_input import HarnessVerifyInput
from eagle_harness.sglang.verify_plan import TreeTailMask
+def native_decode_graphs() -> bool:
+ """Whether the engine captures a speculative graph variant in its own forward mode (native DECODE graphs)."""
+ return hasattr(DecodeGraphVariant, "forward_mode")
+
+
def verify_widths(batch_size: int, intensity: ComputeIntensity, max_width: int) -> list:
"""Verify widths with a graph at ``batch_size``, widest first (``max_width`` and 1 always included)."""
widths = {w for w in verify_buckets(batch_size, intensity) if w <= max_width}
@@ -60,7 +65,7 @@ class VerifyWidthGraphs(DecodeGraphVariants):
self.max_width = max_width
self.tail_mask = tail_mask
self.greedy = greedy
- if plain_decode and not hasattr(DecodeGraphVariant, "forward_mode"):
+ if plain_decode and not native_decode_graphs():
raise RuntimeError("Native no-spec fallback requires SGLang's per-variant forward-mode graph contract.")
self._plain = DecodeGraphVariant("decode", 1, ForwardMode.DECODE) if plain_decode else None
self._variants: Dict[int, DecodeGraphVariant] = {}
diff --git a/eagle_harness/sglang/verify_step.py b/eagle_harness/sglang/verify_step.py
index 2ea0b06..eb68a0b 100644
--- a/eagle_harness/sglang/verify_step.py
+++ b/eagle_harness/sglang/verify_step.py
@@ -205,3 +205,19 @@ def fused_accept_step(
)
return accept_step
+
+
+# sglang's tiled KV move compiles one Triton program per next_power_of_2(moved slots), up to the pool's chunk size
+# (128 or 256 slots), and per slot count specialization (1, a multiple of 16, any other); it compiles only the
+# chunk-sized one at startup. A tree round moves the KV of its accepted tokens, so the smaller programs compiled while
+# serving: ~2 s each on B300, in the first rounds of each new load.
+KV_MOVE_MAX_SLOTS = 256
+
+
+def warm_up_kv_moves(kv_cache: Any, device: torch.device) -> None:
+ """Compile, before serving, the KV move of every slot count a tree round can move: slot 0 (the pool's padding
+ slot) onto itself, for each power of two up to :data:`KV_MOVE_MAX_SLOTS` and the count below it."""
+ counts = sorted({1} | {n - d for n in (2**i for i in range(1, KV_MOVE_MAX_SLOTS.bit_length())) for d in (0, 1)})
+ for count in counts:
+ slots = torch.zeros(count, dtype=torch.int64, device=device)
+ kv_cache.move_kv_cache(slots, slots)
diff --git a/eagle_harness/sglang/worker.py b/eagle_harness/sglang/worker.py
index f422c93..8b9163d 100644
--- a/eagle_harness/sglang/worker.py
+++ b/eagle_harness/sglang/worker.py
@@ -23,8 +23,9 @@ One decode round:
resumes them if the batch holds the same requests, else drafts from scratch.
A chain (top-k 1, ``chain_intensity``) with served round times under ``data_path`` verifies a fixed width per
-batch size instead (:mod:`eagle_harness.sglang.chain_rounds`), and ``"mode": "calibrate"`` forces and times every
-chain width while it serves.
+batch size instead (:mod:`eagle_harness.sglang.chain_rounds`), a chain without ``data_path`` chooses that width on
+its served rounds (:class:`~eagle_harness.sglang.chain_rounds.ServedChainPolicy`), and ``"mode": "calibrate"`` forces
+and times every chain width while it serves.
Prefill (target extend with aux hidden capture, then the draft extend) is
sglang's EAGLE V2 path unchanged. With overlap scheduling, the worker publishes
@@ -39,7 +40,7 @@ import contextlib
from functools import partial
import logging
import os
-from typing import TYPE_CHECKING, Optional, Union
+from typing import TYPE_CHECKING, Dict, Optional, Union
import torch
@@ -60,19 +61,32 @@ from eagle_harness.core.chain_width import chain_widths
from eagle_harness.kernels.accept_tail import warm_up as warm_up_accept_tail
from eagle_harness.core.config import HarnessConfig, Mode, build_cost_model, parse_harness_config
from eagle_harness.core.cost_model import CostModel, FixedShapePolicy
+from eagle_harness.core.cost_profiles import packaged_served_choices
from eagle_harness.core.limits import ComputeIntensity
from eagle_harness.core.draft_tree import DraftTree, TreeShape
from eagle_harness.core.models import target_is_moe
-from eagle_harness.sglang.chain_rounds import ChainCalibration, ChainRounds
+from eagle_harness.sglang.chain_rounds import (
+ ChainCalibration,
+ ChainRounds,
+ ServedChainPolicy,
+ ServedTreePolicy,
+ served_chain,
+ served_tree,
+)
from eagle_harness.core.live_costs import CostKey, LiveCosts, config_digest
from eagle_harness.core.gemm_solutions import load_gemm_solutions
from eagle_harness.sglang.draft_extend import DraftExtend
from eagle_harness.sglang.draft_steps import DraftSteps
from eagle_harness.sglang.drafter import DynamicDrafter, PrelaunchResult, spin_wait
from eagle_harness.sglang.l2_warm import MlpL2WarmAhead
-from eagle_harness.sglang.plain_decode import NativeDecode
+from eagle_harness.sglang.plain_decode import NativeDecode, native_decode_available, native_decode_blocked
from eagle_harness.sglang.row_attention import DraftRowAttention
-from eagle_harness.sglang.verify_step import AcceptTailGraphs, fused_accept_step, fused_accept_supported
+from eagle_harness.sglang.verify_step import (
+ AcceptTailGraphs,
+ fused_accept_step,
+ fused_accept_supported,
+ warm_up_kv_moves,
+)
from eagle_harness.sglang.verify_graphs import verify_widths
from eagle_harness.sglang.hooks import parameter_names
from eagle_harness.sglang.verify_input import HarnessVerifyInput
@@ -100,6 +114,20 @@ def cost_key(runner) -> CostKey:
)
+def packaged_choices(kind: str, key: CostKey, runner) -> Optional[Dict[int, int]]:
+ """The packaged served ``kind`` ("chain" or "tree") choices of this setup, if it was measured
+ (:func:`~eagle_harness.core.cost_profiles.packaged_served_choices`)."""
+ return packaged_served_choices(
+ kind,
+ device_name=key.gpu,
+ model_path=get_model().model_path,
+ draft_model_path=get_spec().speculative_draft_model_path,
+ tp_size=key.gpu_count,
+ dtype=str(runner.dtype),
+ quantization=get_model().quantization,
+ )
+
+
def _sampled_tree(worker):
"""The draft's recorded-q sampled tree, or None: engines without sglang's verify sampler hook build none, and a
worker built without sglang's __init__ (or serving natively, OFF) may have no draft steps to ask."""
@@ -171,12 +199,15 @@ class HarnessDraftWorker(EagleDraftWorker):
def _warm_up_kernels(self) -> None:
"""Compile, before serving, the kernels whose first use would otherwise stall a served round: the fused
- accept step (batch-size-1 trees) and the softmax of sglang's draft extend after a prefill (a JIT module).
+ accept step (batch-size-1 trees), the KV move of a tree's accepted tokens at every slot count, and the
+ softmax of sglang's draft extend after a prefill (a JIT module). A stall in a served round also skews the
+ served tree policy's comparison of speculation with native decode.
"""
if torch.device(self.device).type != "cuda":
return
if self.topk > 1:
warm_up_accept_tail(self.extend.hidden_rows(2), self.target_worker.model_runner.dtype)
+ warm_up_kv_moves(self.target_worker.model_runner.token_to_kv_pool_allocator.get_kvcache(), self.device)
softmax = getattr(spec_utils, "sampling_softmax", None)
if softmax is not None:
# The engine dispatches by vocabulary size; a tiny tensor can warm a different backend.
@@ -221,17 +252,28 @@ class EagleHarnessWorker(EAGLEWorkerV2):
verify_num=spec.speculative_num_draft_tokens,
)
self._chain_schedule = self._fixed_chain_widths(spec)
+ # A chain without served round times picks its width per batch size on the served rounds.
+ self._served_chain = served_chain(self.config, spec.speculative_eagle_topk)
+ # A tree speculates or decodes natively per batch size, chosen on its served rounds, where the engine can.
+ self._served_tree = served_tree(self.config, spec.speculative_eagle_topk) and native_decode_available()
+ if served_tree(self.config, spec.speculative_eagle_topk) and native_decode_blocked():
+ logger.warning(
+ "EAGLE_HARNESS: this server cannot switch to native decode where speculation loses (Mamba prefix-cache "
+ "tracking or DP attention), so it speculates at every load; --disable-radix-cache enables the switch."
+ )
native_decode = self.config.mode is Mode.CALIBRATE or any(
width == 0 for _, _, width in (self._chain_schedule or ())
)
if native_decode and (get_exec().mamba.enable_mamba_extra_buffer or get_parallel().dp_size != 1):
raise ValueError("Native no-spec fallback requires DP1 and disabled Mamba prefix-cache tracking.")
- self._native_decode_enabled = native_decode
+ self._native_decode_enabled = (
+ native_decode or self._served_tree or (self._served_chain and native_decode_available())
+ )
self.fixed = FixedShapePolicy(self.config.intensity, **shape)
# Built before sglang's __init__, which creates the draft worker through create_draft_worker.
self.cost_model: Optional[CostModel] = None
self.live: Optional[LiveCosts] = None
- if self.config.mode.uses_cost_model and self._chain_schedule is None:
+ if self.config.mode.uses_cost_model and self._chain_schedule is None and not self._served_chain:
runner = target_worker.model_runner
key = cost_key(runner)
if self.config.data_path:
@@ -322,9 +364,43 @@ class EagleHarnessWorker(EAGLEWorkerV2):
writer=writer,
native_decode=True,
)
- self._chain_rounds = ChainRounds(self._chain_schedule, calibration, DynamicDrafter.average_context)
capture_bs, _ = get_batch_sizes_to_capture(self.target_worker.model_runner)
+
+ def bucket(bs: int) -> int:
+ return min((b for b in capture_bs if b >= bs), default=max(capture_bs, default=bs))
+
tp_group = get_tp_group()
+ policy = None
+ if self._served_chain:
+ runner = self.target_worker.model_runner
+ key = cost_key(runner)
+ policy = ServedChainPolicy(
+ verify_widths(1, intensity, full),
+ torch.device(self.device),
+ max_batch_size=get_schedule().max_running_requests,
+ native_decode=self._native_decode_enabled,
+ writer=tp_group.rank_in_group == 0,
+ sync=lambda step: tp_group.broadcast_object(step, src=0),
+ cache_path=os.path.join(self.config.cost_cache_dir, f"{key.digest}-chain.json"),
+ key=key.to_dict(),
+ packaged=packaged_choices("chain", key, runner),
+ )
+ elif self._served_tree:
+ runner = self.target_worker.model_runner
+ key = cost_key(runner)
+ policy = ServedTreePolicy(
+ full,
+ torch.device(self.device),
+ max_batch_size=get_schedule().max_running_requests,
+ writer=tp_group.rank_in_group == 0,
+ sync=lambda step: tp_group.broadcast_object(step, src=0),
+ cache_path=os.path.join(self.config.cost_cache_dir, f"{key.digest}-tree.json"),
+ key=key.to_dict(),
+ packaged=packaged_choices("tree", key, runner),
+ # Native decode priced as the target's one-token verify at the batch's bucket.
+ native_us=None if self.live is None else lambda bs: self.live.verify_us(bucket(bs), 1),
+ )
+ self._chain_rounds = ChainRounds(self._chain_schedule, calibration, DynamicDrafter.average_context, policy)
self._drafter = DynamicDrafter(
self._draft_worker.steps,
self.cost_model,
@@ -336,11 +412,13 @@ class EagleHarnessWorker(EAGLEWorkerV2):
tree_mask_mode=self._draft_worker.tree_mask_mode,
device=self.device,
live=self.live,
- bucket=lambda bs: min((b for b in capture_bs if b >= bs), default=max(capture_bs, default=bs)),
+ bucket=bucket,
sync_cells=lambda cells: tp_group.broadcast_object(cells, src=0),
save_cells=self._save_live_costs if tp_group.rank_in_group == 0 else None,
chain_width=(
- self._chain_rounds.draft_width if calibration is not None or self._chain_schedule is not None else None
+ self._chain_rounds.draft_width
+ if calibration is not None or self._chain_schedule is not None or self._served_chain
+ else None
),
)
self._native_decode = NativeDecode(
@@ -403,7 +481,7 @@ class EagleHarnessWorker(EAGLEWorkerV2):
with self._draft_context(), spec_stage_span("draft"):
batch.spec_info = self._drafter.draft(batch)
width = self._drafter.last_width
- rounds.round_start(len(batch.seq_lens), width, batch)
+ rounds.round_start(len(batch.seq_lens), width, batch, settled=not self._drafter.seeding)
with spec_stage_span("verify"):
started = None if timer is None else timer.start()
result, staged = self._verify(batch, width, grammar_barrier)
diff --git a/eagle_harness/tokenspeed/models.py b/eagle_harness/tokenspeed/models.py
index e18af6f..8c758f2 100644
--- a/eagle_harness/tokenspeed/models.py
+++ b/eagle_harness/tokenspeed/models.py
@@ -25,14 +25,14 @@ import torch
from tokenspeed.runtime.configs.model_config import configure_mha_attention
from tokenspeed.runtime.configs.model_profile import ModelProfile
from tokenspeed.runtime.distributed.comm_ops import all_gather
+from tokenspeed.runtime.layers.linear import ReplicatedLinear
from tokenspeed.runtime.layers.logits_processor import LogitsProcessorOutput
from tokenspeed.runtime.layers.vocab_parallel_embedding import ParallelLMHead
from tokenspeed.runtime.models.llama_eagle3 import Eagle3DecoderLayer, LlamaForCausalLMEagle3
-from torch import nn
from eagle_harness.kernels.candidate_ops import candidate_head as fused_candidate_head
from eagle_harness.models.eagle4_config import check_ported
-from eagle_harness.models.low_rank_head import low_rank_scores
+from eagle_harness.models.low_rank_head import shard_scores
@dataclass
@@ -72,7 +72,13 @@ class LlamaForCausalLMEagle4(LlamaForCausalLMEagle3):
tp_group=lm_head.tp_group,
prefix=f"{prefix}.efficient_lm_head" if prefix else "efficient_lm_head",
)
- self.efficient_lm_head_down_proj = nn.Linear(config.hidden_size, rank, bias=False)
+ # The engine's linear: its GEMM dispatch serves the few draft rows with a low-latency kernel.
+ self.efficient_lm_head_down_proj = ReplicatedLinear(
+ config.hidden_size,
+ rank,
+ bias=False,
+ prefix=f"{prefix}.efficient_lm_head_down_proj" if prefix else "efficient_lm_head_down_proj",
+ )
self.tp_size = lm_head.tp_size
self.tp_group = lm_head.tp_group
self.cuda_candidate_kernels = True # HarnessConfig.draft_model.cuda_candidate_kernels, set by the drafter
@@ -119,11 +125,9 @@ class LlamaForCausalLMEagle4(LlamaForCausalLMEagle3):
def candidate_head(self, normed: torch.Tensor, k: int):
"""Candidate logits, target ids and top-k for recorded-q proposals."""
shard = self.lm_head.shard_indices
- scores = low_rank_scores(
- normed,
- self.efficient_lm_head_down_proj.weight,
- self.efficient_lm_head.weight,
- shard.org_vocab_end_index - shard.org_vocab_start_index,
+ low_rank, _ = self.efficient_lm_head_down_proj(normed)
+ scores = shard_scores(
+ low_rank, self.efficient_lm_head.weight, shard.org_vocab_end_index - shard.org_vocab_start_index
)
return fused_candidate_head(
scores,
diff --git a/eagle_harness/vllm/calibrate.py b/eagle_harness/vllm/calibrate.py
index 71d4f39..a802253 100644
--- a/eagle_harness/vllm/calibrate.py
+++ b/eagle_harness/vllm/calibrate.py
@@ -230,13 +230,14 @@ def engine(model: str, draft: str, tp: int, width: int, chain: bool, max_num_seq
else:
method_config = {"mode": "static", "node_top_k": 8, "tree_depth": -(-width // 8) + 1, "max_verify_width": width}
backend = "CUSTOM"
+ # vLLM's default token budget fits one full batch; a larger explicit one makes its warmup run a request longer
+ # than max_model_len, past the end of the block table.
args = {
"model": model,
"tensor_parallel_size": tp,
"max_model_len": 4608,
"gpu_memory_utilization": 0.85,
"max_num_seqs": max_num_seqs,
- "max_num_batched_tokens": max(8192, max_num_seqs * width),
"speculative_config": {
"method": "eagle_harness",
"model": draft,
@@ -294,7 +295,14 @@ def measure(
"engine_args": engine_args or {},
"target_shape": asdict(target_shape),
"draft_shape": asdict(draft_shape),
- "key": cost_key(model, draft, tp, config.model_config.dtype).to_dict(),
+ # Key by the engine's resolved models, as the server does when it loads the tables (speculator.load_model):
+ # vLLM may resolve a Hub id to a local snapshot directory, which config_digest names differently.
+ "key": cost_key(
+ config.model_config.model,
+ config.speculative_config.draft_model_config.model,
+ tp,
+ config.model_config.dtype,
+ ).to_dict(),
}
raw.update(context_step=context_step, context_stop=context_stop, target=target_rows, draft=draft_rows)
with open(raw_path, "w", encoding="utf-8") as f:
diff --git a/tests/test_cli.py b/tests/test_cli.py
index cd70dc6..5b669da 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -243,6 +243,20 @@ def test_vllm_overrides_survive_command_rendering(options, expected, capsys):
assert command[index : index + len(expected)] == expected
[email protected]("engine_args", [{}, {"max_model_len": 2048}])
+def test_vllm_serve_command_token_budget_fits_one_batch(engine_args, capsys):
+ # A budget above max_num_seqs * max_model_len makes vLLM's warmup run one request past the block table (#128).
+ argv = ["calibrate", "--model", "m", "--draft", "d", "--engine", "vllm", "--dry-run"]
+ assert cli.main([*argv, "--engine-args", json.dumps(engine_args)]) == 0
+ command = shlex.split(capsys.readouterr().out.splitlines()[-1])
+
+ def flag(name):
+ return int(command[command.index(name) + 1]) if name in command else None
+
+ budget = flag("--max-num-batched-tokens")
+ assert budget is None or budget <= flag("--max-num-seqs") * flag("--max-model-len")
+
+
def test_empty_cost_tables_do_not_publish_serve_command(monkeypatch, tmp_path):
monkeypatch.setattr(cli, "_engine", lambda name: "vllm")
diff --git a/tests/test_cost_profiles.py b/tests/test_cost_profiles.py
index c9fd16e..ddc99b5 100644
--- a/tests/test_cost_profiles.py
+++ b/tests/test_cost_profiles.py
@@ -8,8 +8,14 @@ from pathlib import Path
import pytest
from eagle_harness.core.config import parse_harness_config
-from eagle_harness.core.cost_profiles import MI355X_PROFILE, packaged_cost_profile
+from eagle_harness.core.cost_profiles import (
+ B300_SERVED_PROFILE,
+ MI355X_PROFILE,
+ packaged_cost_profile,
+ packaged_served_choices,
+)
from eagle_harness.core.cost_table import load_cost_tables
+from eagle_harness.core.limits import ComputeIntensity, cost_batch_sizes, verify_buckets
@pytest.fixture
@@ -78,3 +84,24 @@ def test_changed_model_config_does_not_select_profile(model_paths, index):
def test_profile_does_not_depend_on_model_directory_name(model_paths):
assert select(model_paths, dtype="auto") == str(MI355X_PROFILE)
+
+
+def test_b300_served_choices_are_selected_for_their_measured_setup_only(tmp_path):
+ paths = []
+ for name in ("target", "draft"):
+ path = tmp_path / name
+ path.mkdir()
+ (path / "config.json").write_bytes((B300_SERVED_PROFILE / f"{name}_config.json").read_bytes())
+ paths.append(str(path))
+ args = dict(device_name="NVIDIA B300 SXM6 AC", model_path=paths[0], draft_model_path=paths[1], tp_size=4,
+ dtype="torch.bfloat16", quantization=None)
+ widths = packaged_served_choices("chain", **args)
+ assert widths and set(widths) <= set(cost_batch_sizes(512))
+ assert set(widths.values()) <= {0, *verify_buckets(1, ComputeIntensity.CHAIN)}
+ trees = packaged_served_choices("tree", **args)
+ assert trees and set(trees) <= set(cost_batch_sizes(512)) and set(trees.values()) == {0, 1}
+ for override in ({"tp_size": 8}, {"device_name": "NVIDIA H200"}, {"quantization": "fp8"},
+ {"draft_model_path": paths[0]}):
+ assert packaged_served_choices("chain", **{**args, **override}) is None
+ assert packaged_served_choices("tree", **{**args, **override}) is None
+ assert packaged_cost_profile(**args) is None
diff --git a/tests/test_live_costs.py b/tests/test_live_costs.py
index b7cdad3..c38eb98 100644
--- a/tests/test_live_costs.py
+++ b/tests/test_live_costs.py
@@ -37,6 +37,15 @@ def test_seed_sweep_orders_width_one_first_then_widest_and_ends_when_sampled():
assert not live.ready(4) and live.seed_width(4) == 1
+def test_a_verify_is_priced_once_timed_and_before_its_bucket_is_seeded():
+ live = _live()
+ assert live.verify_us(1, 1) is None
+ live.record(1, 1, 500, 4400.0)
+ assert live.verify_us(1, 1) == 4400.0 and not live.ready(1)
+ live.record(1, 1, 500, 4000.0)
+ assert live.verify_us(1, 1) == 4000.0 and live.verify_us(4, 1) is None
+
+
def test_tables_hold_the_served_curve_and_fill_unmeasured_intervals():
live = _live()
_seed(live, 1, {1: 4000.0, 64: 7200.0, 32: 5700.0, 16: 4800.0, 8: 4500.0}, context=3000)
diff --git a/tests/test_served_chain.py b/tests/test_served_chain.py
new file mode 100644
index 0000000..2a50309
--- /dev/null
+++ b/tests/test_served_chain.py
@@ -0,0 +1,455 @@
+# Copyright 2026 Eagle AI Corp.
+# SPDX-License-Identifier: LicenseRef-Eagle-4-Source-Available
+
+"""The served-round chain policy of a serve-mode chain without data_path (:class:`ServedChainPolicy`), on CPU."""
+
+from __future__ import annotations
+
+import json
+from types import SimpleNamespace
+
+import pytest
+import torch
+
+from eagle_harness.core.config import HarnessConfig, Mode
+from eagle_harness.core.limits import ComputeIntensity
+from eagle_harness.sglang import chain_rounds
+from eagle_harness.sglang.chain_rounds import ChainRounds, ServedChainPolicy, ServedTreePolicy, served_chain, served_tree
+
+
+class Clock:
+ """``time.perf_counter_ns`` under test control."""
+
+ def __init__(self):
+ self.ns = 0
+
+ def __call__(self):
+ return self.ns
+
+ def advance(self, ms: float) -> None:
+ self.ns += int(ms * 1e6)
+
+
+def clock(monkeypatch):
+ clock = Clock()
+ monkeypatch.setattr(chain_rounds.time, "perf_counter_ns", clock)
+ return clock
+
+
+def _policy(
+ tmp_path=None, *, widths=(1, 2, 4, 8), native=False, writer=True, sync=lambda step: step, key=None, packaged=None
+):
+ return ServedChainPolicy(
+ widths,
+ torch.device("cpu"),
+ max_batch_size=512,
+ native_decode=native,
+ writer=writer,
+ sync=sync,
+ cache_path=None if tmp_path is None else str(tmp_path / "chain.json"),
+ key=key,
+ packaged=packaged,
+ )
+
+
+def _reqs(*ids):
+ return [SimpleNamespace(rid=i) for i in ids]
+
+
+def _serve(policy, clock, bs, period_ms, accept, rounds=10_000):
+ """Serve rounds of ``bs`` requests until their batch key is decided; ``period_ms[w]`` is the whole period of a
+ width-``w`` round (0: native decode) and ``accept[w]`` its accept length. Returns the widths served."""
+ requests = _reqs(*range(bs))
+ served = []
+ for _ in range(rounds):
+ width = policy.width(bs)
+ policy.round_start(bs, width, requests)
+ if not policy.exploring(bs):
+ return served
+ served.append(width)
+ if width:
+ policy.accepted(width, torch.full((bs,), accept[width]))
+ clock.advance(period_ms[width])
+ raise AssertionError("no decision")
+
+
+def test_a_prefill_between_two_rounds_is_part_of_the_first_rounds_period(clock):
+ policy = _policy()
+ width = policy.width(64)
+ policy.round_start(64, width, _reqs(1, 2))
+ clock.advance(10)
+ # A prefill admits request 3 (no decode round): its time stays in the previous round's period.
+ clock.advance(25)
+ policy.round_start(64, policy.width(64), _reqs(1, 2, 3))
+ assert policy._cells[(64, width)] == [35_000.0, 64, 1] # us, requests (the batch size), periods
+
+
+def test_a_complete_turnover_does_not_count_as_a_served_interval(clock):
+ policy = _policy()
+ width = policy.width(8)
+ policy.round_start(8, width, _reqs(1, 2))
+ clock.advance(5_000) # the batch drained: the server may have idled before the next requests came
+ policy.round_start(8, policy.width(8), _reqs(3, 4))
+ assert policy._cells.get((8, width), [0.0, 0, 0])[2] == 0
+
+
+def test_a_burst_of_arrivals_counts_toward_the_comparison_without_being_priced(clock):
+ policy = _policy()
+ width = policy.width(64)
+ policy.round_start(64, width, _reqs(*range(64)))
+ clock.advance(200) # a large prefill admits 9 requests (more than 1/8 of the batch)
+ policy.round_start(64, policy.width(64), _reqs(*range(9, 73)))
+ assert policy._cells[(64, width)] == [0.0, 0, 1]
+ clock.advance(10) # one refill: priced
+ policy.round_start(64, policy.width(64), _reqs(*range(10, 74)))
+ assert policy._cells[(64, width)] == [10_000.0, 64, 2]
+
+
+def test_a_turnover_measures_native_decode_and_its_best_width_again_together(clock):
+ policy = _policy(native=True)
+ policy._pair[64] = (4, 0)
+ requests = _reqs(*range(64))
+ while policy.width(64) == 4:
+ policy.round_start(64, 4, requests)
+ clock.advance(10)
+ policy.round_start(64, 0, requests) # native decode's first round: the best width has its block
+ clock.advance(5_000)
+ policy.round_start(64, policy.width(64), _reqs(*range(100, 164))) # a new load after the batch drained
+ assert policy._cells[(64, 4)] == [0.0, 0, 0] and policy._cells[(64, 0)] == [0.0, 0, 0]
+ assert policy.width(64) == 4
+
+
+def test_the_search_widens_while_the_wider_chain_wins(clock):
+ policy = _policy()
+ # Accept lengths 1, 2, 3, 3 at widths 1, 2, 4, 8: tokens per ms 0.1, 0.167, 0.2, 0.1 per request.
+ served = _serve(policy, clock, 64, {1: 10, 2: 12, 4: 15, 8: 30}, {1: 1, 2: 2, 4: 3, 8: 3})
+ assert policy.decided == {64: 4} and set(served) == {1, 2, 4, 8}
+ # Another batch key searches on its own: compute-bound rounds keep the root-only verify.
+ _serve(policy, clock, 256, {1: 20, 2: 45, 4: 80, 8: 160}, {1: 1, 2: 2, 4: 3, 8: 3})
+ assert policy.decided == {64: 4, 256: 1}
+
+
+def test_a_larger_batch_never_searches_wider_than_a_smaller_one_chose(clock):
+ policy = _policy()
+ _serve(policy, clock, 16, {1: 10, 2: 12, 4: 15, 8: 30}, {1: 1, 2: 2, 4: 3, 8: 3})
+ # At batch 64 width 8 would win, but width 4 won at batch 16: the search stops there.
+ served = _serve(policy, clock, 64, {1: 10, 2: 12, 4: 15, 8: 16}, {1: 1, 2: 2, 4: 3, 8: 5})
+ assert 8 not in served and policy.decided == {16: 4, 64: 4}
+
+
+def test_native_decode_is_measured_last_and_spreads_to_larger_batches(clock):
+ policy = _policy(native=True)
+ # Width 2 beats width 1 (2/15 > 1/10) and width 4 (2/18); native decode (1/7) beats width 2.
+ served = _serve(policy, clock, 64, {0: 7, 1: 10, 2: 15, 4: 18, 8: 25}, {1: 1, 2: 2, 4: 2, 8: 2})
+ first_native = served.index(0)
+ assert 0 not in served[:first_native] and set(served[first_native:]) == {0} # once native, the batch stays
+ assert policy.decided[64] == 0
+ assert all(policy.decided[bs] == 0 for bs in (96, 128, 192, 256, 384, 512)) # larger batch keys: unmeasured
+ assert policy.exploring(48)
+
+
+def test_a_losing_native_decode_leaves_the_best_width(clock):
+ policy = _policy(native=True)
+ _serve(policy, clock, 16, {0: 9, 1: 10, 2: 12, 4: 15, 8: 30}, {1: 1, 2: 2, 4: 3, 8: 3})
+ assert policy.decided == {16: 4}
+
+
+def test_no_smaller_batch_measures_native_decode_once_speculation_won(clock):
+ policy = _policy(native=True)
+ periods, accept = {0: 9, 1: 10, 2: 12, 4: 15, 8: 30}, {1: 1, 2: 2, 4: 3, 8: 3}
+ assert 0 in _serve(policy, clock, 64, periods, accept) and policy.decided == {64: 4}
+ assert 0 not in _serve(policy, clock, 16, periods, accept) and policy.decided[16] == 4
+
+
+def test_native_decode_is_not_chosen_without_the_engine_route(clock):
+ policy = _policy(native=False)
+ served = _serve(policy, clock, 64, {0: 1, 1: 10, 2: 15, 4: 18, 8: 25}, {1: 1, 2: 2, 4: 2, 8: 2})
+ assert 0 not in served and policy.decided == {64: 2}
+
+
+def test_comparisons_interleave_the_two_widths_in_blocks(clock):
+ policy = _policy()
+ plan = []
+ for _ in range(4 * chain_rounds.BLOCK_ROUNDS):
+ width = policy.width(64)
+ plan.append(width)
+ policy.round_start(64, width, _reqs(1))
+ block = chain_rounds.BLOCK_ROUNDS
+ assert plan == ([1] * block + [2] * block) * 2
+
+
+def test_decisions_survive_a_restart_on_the_same_setup_only(tmp_path, clock):
+ key = {"gpu": "NVIDIA B300", "gpu_count": 4}
+ policy = _policy(tmp_path, key=key)
+ _serve(policy, clock, 64, {1: 10, 2: 12, 4: 15, 8: 30}, {1: 1, 2: 2, 4: 3, 8: 3})
+ restarted = _policy(tmp_path, key=key, writer=False)
+ assert restarted.decided == {64: 4} and restarted.width(60) == 4 and not restarted.exploring(64)
+ assert _policy(tmp_path, key={**key, "gpu_count": 8}).decided == {} # another setup explores again
+ assert _policy(tmp_path, key=key, native=True).decided == {} # another action set too
+ (tmp_path / "chain.json").write_text(
+ json.dumps({**json.loads((tmp_path / "chain.json").read_text()), "chain": {"64": 3}})
+ )
+ assert _policy(tmp_path, key=key).decided == {} # a width without a graph is refused
+
+
+def test_a_first_start_serves_the_packaged_choices_of_its_setup(clock):
+ policy = _policy(native=True, packaged={16: 4, 64: 4, 256: 0})
+ assert {k: w for k, w in policy.decided.items() if w} == {16: 4, 64: 4}
+ assert all(policy.decided[k] == 0 for k in (256, 384, 512)) # native decode serves every larger batch key
+ assert policy.width(60) == 4 and policy.width(300) == 0 and not policy.exploring(64)
+ assert policy.exploring(128) and policy.exploring(8) # unmeasured keys are searched on the served rounds
+ # A search between packaged keys stays within the width a smaller key chose.
+ served = _serve(policy, clock, 128, {0: 9, 1: 10, 2: 12, 4: 15, 8: 30}, {1: 1, 2: 2, 4: 3, 8: 3})
+ assert 8 not in served and policy.decided[128] == 4
+
+
+def test_the_servers_own_choices_take_precedence_over_the_packaged_ones(tmp_path, clock):
+ key = {"gpu": "NVIDIA B300 SXM6 AC", "gpu_count": 4}
+ periods, accept = {0: 9, 1: 10, 2: 12, 4: 30, 8: 60}, {1: 1, 2: 2, 4: 3, 8: 3}
+ _serve(_policy(tmp_path, key=key, native=True), clock, 64, periods, accept)
+ assert _policy(tmp_path, key=key, native=True, packaged={16: 4, 64: 4, 256: 0}).decided == {64: 2}
+
+
+def test_packaged_choices_that_are_not_actions_of_the_server_are_not_served():
+ assert _policy(native=False, packaged={16: 4, 64: 4, 256: 0}).decided == {} # no native decode route
+ assert _policy(native=True, widths=(1, 2, 8), packaged={16: 4}).decided == {} # no graph for width 4
+
+
+def test_other_ranks_apply_rank_zeros_step_without_reading_the_gpu(clock):
+ class NoDeviceCounts:
+ def __getitem__(self, width):
+ raise AssertionError("a non-writer rank counted accept lengths")
+
+ def tolist(self):
+ raise AssertionError("a non-writer rank read the accept counts")
+
+ received = []
+ policy = _policy(writer=False, sync=lambda step: received.append(step) or ("decide", 2, None))
+ policy._accept = NoDeviceCounts()
+ rounds = ChainRounds(None, None, lambda batch: 0, policy)
+ requests = _reqs(1)
+ while policy.exploring(8):
+ width = policy.width(8)
+ policy.round_start(8, width, requests)
+ rounds.accepted(width, torch.ones(8, dtype=torch.int64))
+ clock.advance(1)
+ assert received == [None] and policy.decided == {8: 2}
+
+
+def test_batch_sizes_share_the_nearest_calibration_batch_size():
+ policy = _policy()
+ policy.decided = {64: 4, 256: 0}
+ rounds = ChainRounds(None, None, lambda batch: 0, policy)
+ assert rounds.draft_width(57) == rounds.draft_width(70) == 4
+ assert rounds.plain_width(256) == 0 and rounds.draft_width(256) == 1 # a draft never asks for width 0
+ assert policy.exploring(8) and not policy.exploring(250)
+
+
+def test_only_a_serve_mode_chain_without_data_path_is_served_by_the_policy():
+ chain = HarnessConfig(intensity=ComputeIntensity.CHAIN)
+ assert served_chain(chain, 1)
+ assert not served_chain(chain, 4)
+ assert not served_chain(HarnessConfig(intensity=ComputeIntensity.CHAIN, data_path="/cost"), 1)
+ assert not served_chain(HarnessConfig(intensity=ComputeIntensity.CHAIN, mode=Mode.STATIC), 1)
+ assert not served_chain(HarnessConfig(), 1)
+
+
+def _decide_width_4_or_native(clock, period_ms):
+ """Width 4 (1.05 tokens per request and round) against native decode at batch 64; ``period_ms(i)`` is the i-th
+ period of either."""
+ policy = _policy(native=True)
+ policy._pair[64] = (4, 0)
+ policy.profile = lambda: chain_rounds.AcceptProfile((0.05, 0.0, 0.0))
+ requests = _reqs(*range(64))
+ i = 0
+ while policy.exploring(64):
+ policy.round_start(64, policy.width(64), requests)
+ clock.advance(period_ms(i))
+ i += 1
+ return policy.decided[64]
+
+
+def test_speculation_beats_native_decode_only_beyond_the_noise_of_the_two_rates(clock):
+ # Width 4 commits 5% more per period, but every other period of both takes twice as long: 32 periods each cannot
+ # tell them apart, and native decode serves.
+ assert _decide_width_4_or_native(clock, lambda i: 20 if i % 2 else 10) == 0
+ # Without the noise the same 5% decides for speculation.
+ assert _decide_width_4_or_native(clock, lambda i: 10) == 4
+
+
+def _tree(tmp_path=None, key=None, writer=True, sync=lambda step: step, native_us=None):
+ return ServedTreePolicy(
+ 16,
+ torch.device("cpu"),
+ max_batch_size=512,
+ writer=writer,
+ sync=sync,
+ cache_path=None if tmp_path is None else str(tmp_path / "tree.json"),
+ key=key,
+ native_us=native_us,
+ )
+
+
+def _serve_tree(policy, clock, bs, spec_ms, native_ms, accept, rounds=10_000, seed_rounds=0, widths=(16, 7, 3)):
+ """Serve ``bs`` requests until their batch key is decided: speculative rounds (the cost model's widths, cycling
+ through ``widths``) commit ``accept`` tokens per request in ``spec_ms``, native decode one in ``native_ms``. The
+ first ``seed_rounds`` speculative rounds seed the live costs. Returns the actions served (16: speculation)."""
+ requests = _reqs(*range(bs))
+ served = []
+ for i in range(rounds):
+ action = policy.width(bs)
+ width = widths[i % len(widths)] if action else 0
+ settled = not (action and i < seed_rounds)
+ policy.round_start(bs, width, requests, settled=settled)
+ if not policy.exploring(bs):
+ return served
+ served.append(action)
+ if width:
+ policy.accepted(width, torch.full((bs,), accept, dtype=torch.int32))
+ clock.advance(spec_ms if action and settled else (100 * spec_ms if action else native_ms))
+ raise AssertionError("no decision")
+
+
+def test_a_tree_decodes_natively_where_its_rounds_commit_fewer_tokens_per_time(clock):
+ policy = _tree()
+ # 2.5 tokens per request in 30 ms (0.083 per ms) against native decode's 1 in 10 ms (0.1 per ms).
+ served = _serve_tree(policy, clock, 256, spec_ms=30, native_ms=10, accept=2.5)
+ first_native = served.index(0)
+ assert set(served[:first_native]) == {16} and set(served[first_native:]) == {0} # native decode last, sticky
+ assert all(policy.decided[k] == 0 for k in (256, 384, 512)) and policy.exploring(192)
+
+
[email protected]("stalled, spec_ms, winner", [(0, 30, 0), (16, 15, 16)])
+def test_a_one_time_stall_does_not_decide_the_comparison(clock, stalled, spec_ms, winner):
+ """A kernel compile or graph capture in one period (3 s against 10-30 ms rounds) is counted, not priced: the
+ action that commits more tokens per time (2.5 in ``spec_ms`` against 1 in 10 ms) wins although it stalled."""
+ policy, requests, rounds = _tree(), _reqs(*range(256)), {0: 0, 16: 0}
+ while policy.exploring(256):
+ action = policy.width(256)
+ policy.round_start(256, action, requests)
+ if action:
+ policy.accepted(action, torch.full((256,), 2.5, dtype=torch.int32))
+ rounds[action] += 1
+ clock.advance(3000 if action == stalled and rounds[action] == 2 else spec_ms if action else 10)
+ assert rounds[stalled] > 2 and policy.decided[256] == winner
+ assert chain_rounds.STALL_FACTOR * 30 < 3000
+
+
+def test_the_first_round_after_a_switch_of_action_is_counted_not_priced(clock):
+ """Native decode's first round after speculation also finishes speculation's work (a prelaunched draft): its
+ period (95 ms here, under the stall factor) is counted toward the comparison but not priced."""
+ policy, requests, compared = _tree(), _reqs(*range(256)), []
+ policy._compare = lambda key, pair: compared.append(policy._rates(key, pair)) or policy._decide(key, 0)
+ native = 0
+ while policy.exploring(256):
+ action = policy.width(256)
+ policy.round_start(256, action, requests)
+ if action:
+ policy.accepted(action, torch.full((256,), 2.5, dtype=torch.int32))
+ native += not action
+ clock.advance(30 if action else 95 if native == 1 else 10)
+ (rates,) = compared
+ assert rates[0][0] == pytest.approx(256 / 10_000) and native == chain_rounds.EXPLORE_ROUNDS + 1
+
+
+def test_a_tree_keeps_speculating_where_it_wins_and_so_does_every_smaller_batch(clock):
+ policy = _tree()
+ _serve_tree(policy, clock, 64, spec_ms=15, native_ms=10, accept=2.5)
+ assert policy.decided[64] == 16
+ assert all(policy.decided[k] == 16 for k in (1, 2, 8, 16, 48)) and policy.exploring(96)
+ rounds = ChainRounds(None, None, lambda batch: 0, policy)
+ assert rounds.plain_width(1) == 16 and rounds.plain_width(64) == 16 # never native below a speculating key
+
+
+def test_rounds_that_seed_the_live_costs_are_not_counted(clock):
+ policy = _tree()
+ # The seed rounds force widths the cost model would not choose (100x the settled period here): were they counted,
+ # native decode would win.
+ served = _serve_tree(policy, clock, 64, spec_ms=15, native_ms=10, accept=2.5, seed_rounds=40)
+ assert policy.decided[64] == 16
+ assert served[:40] == [16] * 40 and served.count(0) == chain_rounds.EXPLORE_ROUNDS
+
+
+def test_speculation_far_ahead_of_native_decodes_estimate_never_decodes_natively(clock):
+ # 4 tokens per request in 10 ms (0.4 per ms) against an estimated 1 in 10 ms: native decode would hold the batch.
+ priced = []
+ policy = _tree(native_us=lambda bs: priced.append(bs) or 10_000.0)
+ served = _serve_tree(policy, clock, 8, spec_ms=10, native_ms=10, accept=4)
+ assert 0 not in served and len(served) == chain_rounds.EXPLORE_ROUNDS
+ assert set(priced) == {8} and all(policy.decided[k] == 16 for k in (1, 2, 8))
+
+
[email protected]("native_us", [lambda bs: 10_000.0, lambda bs: None])
+def test_speculation_near_native_decodes_estimate_or_without_one_measures_native_decode(clock, native_us):
+ # 2 tokens per request in 15 ms against an estimated 1 in 10 ms (1.33x, inside the skip ratio), or no estimate.
+ policy = _tree(native_us=native_us)
+ served = _serve_tree(policy, clock, 8, spec_ms=15, native_ms=10, accept=2)
+ assert served.count(0) == chain_rounds.EXPLORE_ROUNDS and policy.decided[8] == 16
+
+
+def test_noisy_speculation_measures_native_decode_even_ahead_of_its_estimate(clock):
+ # Periods of 1 and 49 ms: 1.6x the estimated 0.8 tokens per ms on average, but not beyond the noise.
+ policy = _tree(native_us=lambda bs: 10_000.0)
+ requests, served = _reqs(*range(8)), []
+ while policy.exploring(8):
+ action = policy.width(8)
+ policy.round_start(8, action, requests)
+ served.append(action)
+ if action:
+ policy.accepted(action, torch.full((8,), 4, dtype=torch.int32))
+ clock.advance((1, 49)[len(served) % 2] if action else 10)
+ assert 0 in served
+
+
+def test_a_chain_without_a_native_estimate_never_checks_one(clock):
+ policy = _policy(native=True)
+ policy._check_native = lambda key, best: pytest.fail("checked native decode without an estimate")
+ _serve(policy, clock, 64, {0: 9, 1: 10, 2: 12, 4: 15, 8: 30}, {1: 1, 2: 2, 4: 3, 8: 3})
+ assert 64 in policy.decided
+
+
+def test_other_ranks_take_rank_zeros_native_check_without_pricing(clock):
+ received = []
+
+ def sync(step):
+ received.append(step)
+ return ("decide", 16, None)
+
+ policy = _tree(writer=False, sync=sync, native_us=lambda bs: pytest.fail("a non-writer rank priced native decode"))
+ served = _serve_tree(policy, clock, 8, spec_ms=10, native_ms=10, accept=4)
+ assert received == [None] and 0 not in served and policy.decided[8] == 16
+
+
+def test_a_first_tree_start_serves_the_packaged_choices_of_its_setup():
+ policy = ServedTreePolicy(
+ 16, torch.device("cpu"), max_batch_size=512, writer=True, sync=lambda step: step, packaged={8: 1, 64: 0}
+ )
+ assert all(policy.decided[k] == 16 for k in (1, 2, 8)) and all(policy.decided[k] == 0 for k in (64, 96, 512))
+ assert policy.exploring(16) and policy.exploring(48) # unmeasured keys are searched on the served rounds
+
+
+def test_tree_decisions_survive_a_restart_on_the_same_setup(tmp_path, clock):
+ key = {"gpu": "NVIDIA B300 SXM6 AC", "gpu_count": 4}
+ _serve_tree(_tree(tmp_path, key=key), clock, 128, spec_ms=30, native_ms=10, accept=2.5)
+ restarted = _tree(tmp_path, key=key, writer=False)
+ assert restarted.width(130) == 0 and restarted.width(512) == 0 and restarted.exploring(64)
+ assert _tree(tmp_path, key={**key, "gpu_count": 8}).decided == {}
+
+
+def test_other_ranks_never_read_the_trees_tokens(clock):
+ policy = _tree(writer=False, sync=lambda step: ("decide", 0, None))
+ requests = _reqs(1)
+ while policy.exploring(256):
+ width = policy.width(256)
+ policy.round_start(256, width, requests)
+ policy.accepted(width, torch.ones(256, dtype=torch.int32))
+ clock.advance(1)
+ assert policy._tokens is None and policy.decided[256] == 0
+
+
+def test_every_serve_mode_tree_chooses_native_decode_on_its_served_rounds():
+ assert served_tree(HarnessConfig(), 4)
+ assert served_tree(HarnessConfig(data_path="/cost"), 4)
+ assert not served_tree(HarnessConfig(), 1)
+ assert not served_tree(HarnessConfig(mode=Mode.STATIC), 4)
diff --git a/tests/test_sglang_overlap.py b/tests/test_sglang_overlap.py
index 3076bf1..393326d 100644
--- a/tests/test_sglang_overlap.py
+++ b/tests/test_sglang_overlap.py
@@ -33,6 +33,7 @@ def test_decode_handoff_uses_the_scheduler_stream_contract(monkeypatch, overlap)
drafter = SimpleNamespace(
draft=lambda b: calls.append("draft"),
last_width=4,
+ seeding=False, # the round served the cost model's tree, not a live-cost seed width
timer=None,
prelaunch=Mock(side_effect=lambda b, **kwargs: (calls.append("prelaunch"), "fence")[1]),
)
diff --git a/tests/test_sglang_plugin.py b/tests/test_sglang_plugin.py
index 27f4a56..a3fb3de 100644
--- a/tests/test_sglang_plugin.py
+++ b/tests/test_sglang_plugin.py
@@ -349,6 +349,85 @@ def test_live_chain_does_not_read_offline_round_files(registered):
assert isinstance(provider, VerifyWidthGraphs)
[email protected]("native", [True, False])
+def test_a_serve_mode_tree_captures_the_native_decode_graph_where_the_engine_has_the_route(registered, monkeypatch, native):
+ from sglang.srt.model_executor.runner.decode_graph_variants import DecodeGraphVariant
+
+ from eagle_harness.core.config import HarnessConfig
+ from eagle_harness.sglang import plain_decode
+
+ if native and not hasattr(DecodeGraphVariant, "forward_mode"):
+ pytest.skip("host has no per-variant forward mode")
+ monkeypatch.setattr(plain_decode, "native_decode_available", lambda: native)
+ registered.__dict__["config"] = HarnessConfig()
+ with get_context().override_server_args(
+ speculative_num_draft_tokens=16, speculative_eagle_topk=4, max_running_requests=512
+ ):
+ provider = registered.create_decode_graph_variants(_runner_info(False))
+ assert (provider._plain is not None) is native
+
+
[email protected]("draft_tokens", [4, 8, 16])
+def test_served_chain_never_builds_the_cost_model(monkeypatch, draft_tokens):
+ """A serve-mode chain without data_path chooses its widths on the served rounds, at every chain width (#145:
+ 0.1.1 sized chains under 16 tokens by the tree rule). The cost model's chain-depth rule needs every width 1..max
+ in its tables, which live costs (captured widths only) never have: it failed at the first seeded batch
+ (``Invalid chain cost index: [0, 1, 3, 5, 7]``)."""
+ from sglang.srt.speculative.eagle_worker_v2 import EAGLEWorkerV2
+
+ from eagle_harness.sglang import worker as worker_mod
+
+ def refuse(*args, **kwargs):
+ raise AssertionError("the served chain built the cost model")
+
+ monkeypatch.setattr(MlpL2WarmAhead, "install", classmethod(lambda cls, model, plan, device: None))
+ monkeypatch.setattr(worker_mod, "build_cost_model", refuse)
+ monkeypatch.setattr(worker_mod, "LiveCosts", refuse)
+ monkeypatch.setattr(worker_mod, "cost_key", lambda runner: SimpleNamespace(digest="setup"))
+ monkeypatch.setattr(EAGLEWorkerV2, "__init__", lambda self, *a, **k: None)
+ monkeypatch.setattr(torch.cuda, "Event", lambda *a, **k: None)
+ target = SimpleNamespace(model_runner=SimpleNamespace(model="target", device="cpu"))
+ with get_context().override_server_args(
+ speculative_algorithm_config=json.dumps({"intensity": "chain_intensity"}),
+ speculative_eagle_topk=1,
+ speculative_num_steps=draft_tokens - 1,
+ speculative_num_draft_tokens=draft_tokens,
+ max_running_requests=512,
+ ):
+ worker = EagleHarnessWorker(None, 0, 0, target)
+ assert worker._served_chain and worker.cost_model is None and worker.live is None
+
+
[email protected]("native", [True, False])
+def test_a_serve_mode_tree_keeps_its_cost_model_and_decodes_natively_where_the_engine_can(monkeypatch, native):
+ """The default tree sizes its rounds with the cost model on live costs and, where the engine has the native
+ route, chooses native decode per batch size on its served rounds (ServedTreePolicy)."""
+ from sglang.srt.speculative.eagle_worker_v2 import EAGLEWorkerV2
+
+ from eagle_harness.sglang import worker as worker_mod
+
+ built = []
+ monkeypatch.setattr(MlpL2WarmAhead, "install", classmethod(lambda cls, model, plan, device: None))
+ monkeypatch.setattr(worker_mod, "build_cost_model", lambda config, **kw: built.append(kw) or SimpleNamespace(tables={}))
+ monkeypatch.setattr(worker_mod, "LiveCosts", lambda **kw: SimpleNamespace(load=lambda path: False))
+ monkeypatch.setattr(worker_mod, "cost_key", lambda runner: SimpleNamespace(digest="setup"))
+ monkeypatch.setattr(worker_mod, "target_is_moe", lambda hf_config: False)
+ monkeypatch.setattr(worker_mod, "native_decode_available", lambda: native)
+ monkeypatch.setattr(EAGLEWorkerV2, "__init__", lambda self, *a, **k: None)
+ monkeypatch.setattr(torch.cuda, "Event", lambda *a, **k: None)
+ runner = SimpleNamespace(model="target", device="cpu", model_config=SimpleNamespace(hf_config=None))
+ with get_context().override_server_args(
+ speculative_algorithm_config="{}",
+ speculative_eagle_topk=4,
+ speculative_num_steps=12,
+ speculative_num_draft_tokens=16,
+ max_running_requests=512,
+ ):
+ worker = EagleHarnessWorker(None, 0, 0, SimpleNamespace(model_runner=runner))
+ assert len(built) == 1 and worker.live is not None and not worker._served_chain
+ assert worker._served_tree is native and worker._native_decode_enabled is native
+
+
def test_verify_widths_follow_the_cost_buckets():
from eagle_harness.core.limits import ComputeIntensity
diff --git a/tests/test_sglang_sampled_worker.py b/tests/test_sglang_sampled_worker.py
index d1c4800..31d0abf 100644
--- a/tests/test_sglang_sampled_worker.py
+++ b/tests/test_sglang_sampled_worker.py
@@ -96,6 +96,44 @@ def test_startup_softmax_uses_the_served_vocabulary(monkeypatch, vocab_size):
assert calls == [((1, vocab_size), torch.float32)]
+def test_tree_warm_up_moves_the_kv_of_the_target_pool(monkeypatch):
+ """A tree's verify moves the accepted tokens' KV in the target's pool: its move kernels compile before serving."""
+ pool = object()
+ moved, accept_tail = [], []
+ monkeypatch.setattr(worker_mod, "warm_up_accept_tail", lambda *args: accept_tail.append(args))
+ monkeypatch.setattr(worker_mod, "warm_up_kv_moves", lambda kv_cache, device: moved.append((kv_cache, device)))
+ monkeypatch.setattr(worker_mod.spec_utils, "sampling_softmax", None)
+ allocator = SimpleNamespace(get_kvcache=lambda: pool)
+ worker = SimpleNamespace(
+ device="cuda", topk=4, extend=SimpleNamespace(hidden_rows=lambda rows: rows),
+ target_worker=SimpleNamespace(model_runner=SimpleNamespace(dtype=torch.bfloat16,
+ token_to_kv_pool_allocator=allocator)),
+ )
+ worker_mod.HarnessDraftWorker._warm_up_kernels(worker)
+ assert moved == [(pool, "cuda")] and len(accept_tail) == 1
+ worker.topk = 1 # a chain's accepted tokens are already in place: nothing moves
+ moved.clear()
+ worker_mod.HarnessDraftWorker._warm_up_kernels(worker)
+ assert moved == []
+
+
+def test_kv_move_warm_up_covers_every_program_a_round_can_compile():
+ """sglang's tiled KV move compiles one program per next power of two of the moved slots (up to its chunk) and
+ per slot count specialization (1, a multiple of 16, any other): the warm-up runs each, on the padding slot."""
+ from eagle_harness.sglang.verify_step import KV_MOVE_MAX_SLOTS, warm_up_kv_moves
+
+ moves = []
+ warm_up_kv_moves(SimpleNamespace(move_kv_cache=lambda tgt, src: moves.append((tgt.clone(), src.clone()))), "cpu")
+ assert all(tgt.dtype == torch.int64 and not tgt.any() and not src.any() for tgt, src in moves)
+
+ def program(count):
+ return 1 << (count - 1).bit_length(), "one" if count == 1 else "x16" if count % 16 == 0 else "other"
+
+ warmed = {program(len(tgt)) for tgt, _ in moves}
+ assert warmed == {program(count) for count in range(1, KV_MOVE_MAX_SLOTS + 1)}
+ assert KV_MOVE_MAX_SLOTS >= 256 # sglang's largest chunk (smaller KV strides)
+
+
@pytest.mark.parametrize("engine_verify, active", [(_verify_without_hook, False), (_verify_with_hook, False), (_verify_with_hook, True)])
def test_verify_passes_the_tree_sampler_only_for_an_active_sampled_tree(monkeypatch, engine_verify, active):
monkeypatch.setattr(worker_mod, "run_eagle_verify", engine_verify)
diff --git a/tests/test_tokenspeed_plugin.py b/tests/test_tokenspeed_plugin.py
index 923c96b..324c8e9 100755
--- a/tests/test_tokenspeed_plugin.py
+++ b/tests/test_tokenspeed_plugin.py
@@ -148,3 +148,44 @@ def test_candidate_gather_rides_the_last_dim_as_16_bit_words(monkeypatch):
assert calls == [(torch.int16, (rows, 2 * n), "tp", -1)]
assert out.shape == (tp, rows, n) and out.dtype == torch.int32
assert all(torch.equal(out[r], shards[r]) for r in range(tp))
+
+
+def test_candidate_head_projects_through_the_engine_linear(monkeypatch):
+ """The rank-r projection runs through the engine's linear (its GEMM dispatch picks the low-latency kernel for the
+ few draft rows), and the shard scores are that projection times the score head, ``-inf`` past the real rows."""
+ import torch
+
+ from eagle_harness.tokenspeed import models
+
+ rows, hidden, rank, shard_vocab, real_rows = 3, 16, 4, 10, 8
+ g = torch.Generator().manual_seed(11)
+ normed = torch.randn(rows, hidden, generator=g)
+ down = torch.randn(rank, hidden, generator=g)
+ score_head = torch.randn(shard_vocab, rank, generator=g)
+ projected = []
+
+ def project(x):
+ projected.append(x)
+ return x @ down.t(), None
+
+ head = models.LlamaForCausalLMEagle4.__new__(models.LlamaForCausalLMEagle4)
+ torch.nn.Module.__init__(head)
+ head.efficient_lm_head_down_proj = project
+ head.efficient_lm_head = torch.nn.Module()
+ head.efficient_lm_head.weight = torch.nn.Parameter(score_head)
+ head.lm_head = torch.nn.Module()
+ head.lm_head.shard_indices = type("Shard", (), {"org_vocab_start_index": 0, "org_vocab_end_index": real_rows})()
+ head.lm_head.weight = torch.nn.Parameter(torch.zeros(shard_vocab, hidden))
+ head.num_candidates, head.tp_size, head.cuda_candidate_kernels = 4, 1, False
+ seen = {}
+
+ def candidate_head(scores, *args, **kwargs):
+ seen["scores"] = scores
+ return None, None, None
+
+ monkeypatch.setattr(models, "fused_candidate_head", candidate_head)
+ head.candidate_topk(normed, 2)
+ assert len(projected) == 1 and projected[0] is normed
+ expected = (normed @ down.t()) @ score_head.t()
+ assert torch.allclose(seen["scores"][:, :real_rows], expected[:, :real_rows])
+ assert torch.isinf(seen["scores"][:, real_rows:]).all()
diff --git a/tests/test_vllm_calibrate.py b/tests/test_vllm_calibrate.py
index beac155..71e25f1 100644
--- a/tests/test_vllm_calibrate.py
+++ b/tests/test_vllm_calibrate.py
@@ -4,6 +4,7 @@
"""Config-derived calibration geometry and real CPU fitting."""
import json
+import sys
from dataclasses import asdict
from types import SimpleNamespace
@@ -124,8 +125,8 @@ def test_measure_records_resolved_model_limit(monkeypatch, tmp_path):
geometry = SimpleNamespace(num_hidden_layers=2)
config = SimpleNamespace(
- model_config=SimpleNamespace(hf_config=geometry, max_model_len=8192, dtype="bfloat16"),
- speculative_config=SimpleNamespace(draft_model_config=SimpleNamespace(hf_config=geometry)),
+ model_config=SimpleNamespace(model="target", hf_config=geometry, max_model_len=8192, dtype="bfloat16"),
+ speculative_config=SimpleNamespace(draft_model_config=SimpleNamespace(model="draft", hf_config=geometry)),
)
measured = []
@@ -145,6 +146,52 @@ def test_measure_records_resolved_model_limit(monkeypatch, tmp_path):
assert max(measured) > 4096 and max(measured) + 64 <= 8192
+def test_measure_keys_tables_like_the_server(monkeypatch, tmp_path):
+ # The server keys the tables it loads by the engine's resolved models (speculator.load_model). vLLM can resolve
+ # a Hub id to a local snapshot directory, so keying by the command-line id makes serve.sh refuse its own tables.
+ from eagle_harness.vllm import calibrate
+
+ snapshots = {}
+ for name in ("target", "draft"):
+ snapshots[name] = tmp_path / "hub" / name / "snapshots" / "0123abcd"
+ snapshots[name].mkdir(parents=True)
+ (snapshots[name] / "config.json").write_text(json.dumps({"name": name}))
+ geometry = SimpleNamespace(num_hidden_layers=2)
+ config = SimpleNamespace(
+ model_config=SimpleNamespace(
+ model=str(snapshots["target"]), hf_config=geometry, max_model_len=4608, dtype="torch.bfloat16"
+ ),
+ speculative_config=SimpleNamespace(
+ draft_model_config=SimpleNamespace(model=str(snapshots["draft"]), hf_config=geometry)
+ ),
+ )
+ llm = SimpleNamespace(llm_engine=SimpleNamespace(vllm_config=config), collective_rpc=lambda name, args: [[]])
+ monkeypatch.setattr(calibrate, "engine", lambda *args, **kwargs: llm)
+ monkeypatch.setattr(calibrate.torch.cuda, "get_device_name", lambda *args: "fixture GPU")
+ monkeypatch.setitem(sys.modules, "vllm", SimpleNamespace(__version__="fixture"))
+ output = tmp_path / "raw.json"
+ calibrate.measure(str(output), 8, False, [1], 256, 1, model="Org/target", draft="Org/draft", tp=4)
+ served = calibrate.cost_key(str(snapshots["target"]), str(snapshots["draft"]), 4, "torch.bfloat16")
+ assert json.loads(output.read_text())["key"] == served.to_dict()
+
+
+ "chain,width,max_num_seqs,overrides",
+ [(False, 8, 1, {}), (False, 64, 1, {}), (True, 16, 64, {}), (False, 8, 1, {"max_model_len": 2048})],
+)
+def test_engine_token_budget_fits_one_batch(monkeypatch, chain, width, max_num_seqs, overrides):
+ # vLLM's warmup splits its token budget over max_num_seqs requests; a request longer than max_model_len indexes
+ # past the block table (device-side assert in the GDN checkpoint gather, issue #128).
+ from eagle_harness.vllm import calibrate
+
+ captured = {}
+ monkeypatch.setitem(sys.modules, "vllm", SimpleNamespace(LLM=lambda **kwargs: captured.update(kwargs)))
+ monkeypatch.setitem(sys.modules, "vllm.plugins", SimpleNamespace(load_general_plugins=lambda: None))
+ calibrate.engine("target", "draft", 4, width, chain, max_num_seqs, **overrides)
+ budget = captured.get("max_num_batched_tokens")
+ assert budget is None or budget <= captured["max_num_seqs"] * captured["max_model_len"]
+
+
def test_merge_rejects_different_tp(tmp_path):
paths = []
for tp in (1, 2):
diff --git a/tests/test_worker_rounds.py b/tests/test_worker_rounds.py
index 7e64312..7ff5aa8 100644
--- a/tests/test_worker_rounds.py
+++ b/tests/test_worker_rounds.py
@@ -104,7 +104,7 @@ def modules(monkeypatch):
"draft_steps": ("DraftSteps",),
"l2_warm": ("MlpL2WarmAhead",),
"row_attention": ("DraftRowAttention",),
- "verify_step": ("AcceptTailGraphs", "fused_accept_step", "fused_accept_supported"),
+ "verify_step": ("AcceptTailGraphs", "fused_accept_step", "fused_accept_supported", "warm_up_kv_moves"),
"verify_graphs": ("verify_widths", "round_up_width"),
"verify_input": ("HarnessVerifyInput", "build_attn_args"),
"verify_tail": ("VerifyTail",),
@@ -164,6 +164,7 @@ def make_worker(modules, monkeypatch, *, calibration=None, schedule=None, native
draft=draft,
last_width=4,
timer=None,
+ seeding=False,
prelaunch=prelaunch,
)
worker._drafter = drafter
@@ -383,7 +384,7 @@ def test_result_copies_precede_scheduler_prelaunch_fence(modules, monkeypatch):
assert modules.copies == [("copy", True, False), ("fence", fence)]
[email protected]("kind", ["tree", "schedule", "calibrate"])
[email protected]("kind", ["tree", "schedule", "calibrate", "served", "served-tree"])
def test_attention_initialization_wires_the_same_width_plan_into_both_routes(modules, monkeypatch, tmp_path, kind):
class Draft:
average_context = staticmethod(lambda batch: 5)
@@ -396,12 +397,18 @@ def test_attention_initialization_wires_the_same_width_plan_into_both_routes(mod
monkeypatch.setattr(module, "DynamicDrafter", Draft)
monkeypatch.setattr(module, "verify_widths", lambda *args: [4, 1])
worker = module.EagleHarnessWorker.__new__(module.EagleHarnessWorker)
+ monkeypatch.setattr(module, "get_schedule", lambda: SimpleNamespace(max_running_requests=8))
+ monkeypatch.setattr(module, "cost_key", lambda runner: SimpleNamespace(digest="setup", to_dict=lambda: {}))
+ monkeypatch.setattr(module, "packaged_choices", lambda kind, key, runner: None) # an unmeasured setup
worker.config = SimpleNamespace(
mode=module.Mode.CALIBRATE if kind == "calibrate" else module.Mode.SERVE,
intensity=module.ComputeIntensity.CHAIN,
- data_path=str(tmp_path),
+ data_path=None if kind == "served" else str(tmp_path),
+ cost_cache_dir=str(tmp_path),
)
worker._chain_schedule = [(1, 1, 4), (2, 8, 0)] if kind == "schedule" else None
+ worker._served_chain = kind == "served"
+ worker._served_tree = kind == "served-tree"
worker._native_decode_enabled = kind != "tree"
worker._target_worker = SimpleNamespace(model_runner=object())
worker.fixed = object()
@@ -420,6 +427,14 @@ def test_attention_initialization_wires_the_same_width_plan_into_both_routes(mod
elif kind == "schedule":
assert width(1) == 4 and width(2) == 1
assert worker._native_decode.select(two)
+ elif kind == "served":
+ # Every batch size starts its search at the root-only width, and native decode is never explored.
+ assert width(1) == width(2) == 1
+ assert not worker._native_decode.select(two)
+ elif kind == "served-tree":
+ # The cost model sizes every round; a batch size speculates before native decode is measured against it.
+ assert width is None and not worker._native_decode.select(two)
+ assert isinstance(worker._chain_rounds._policy, module.ServedTreePolicy)
else:
block = BLOCK_ROUNDS + MIN_ROUNDS
widths = [width(1) for _ in range(2 * block + 1)]
@@ -427,6 +442,18 @@ def test_attention_initialization_wires_the_same_width_plan_into_both_routes(mod
assert worker._native_decode.select(one)
+def test_packaged_choices_look_up_this_servers_setup(modules, monkeypatch):
+ module = modules.worker
+ monkeypatch.setattr(module, "get_model", lambda: SimpleNamespace(model_path="/target", quantization=None))
+ monkeypatch.setattr(module, "get_spec", lambda: SimpleNamespace(speculative_draft_model_path="/draft"))
+ seen = {}
+ monkeypatch.setattr(module, "packaged_served_choices", lambda kind, **setup: seen.update(setup, kind=kind) or {16: 4})
+ key = SimpleNamespace(gpu="NVIDIA B300 SXM6 AC", gpu_count=4)
+ assert module.packaged_choices("tree", key, SimpleNamespace(dtype=torch.bfloat16)) == {16: 4}
+ assert seen == dict(kind="tree", device_name="NVIDIA B300 SXM6 AC", model_path="/target", draft_model_path="/draft", tp_size=4,
+ dtype="torch.bfloat16", quantization=None)
+
+
@pytest.mark.parametrize("dp_size,tracking", [(2, False), (1, True), (2, True)])
def test_native_decode_still_rejects_dp_and_mamba_tracking(modules, monkeypatch, dp_size, tracking):
module = modules.worker