Official Inference Code of EAGLE-4
fix(sglang): decode without speculation wherever speculation would be slower
Seyfal Sultanovdf5a23ad573f
diff --git a/README.md b/README.mdindex 8d6408a..205a979 100644--- a/README.md+++ b/README.md@@ -51,6 +51,9 @@**Leave speculation on. EAGLE-4 adapts to your hardware, batch and workload.**+For high load, serve SGLang on four B300 GPUs with the [high-load recipe](docs/quickstart-sglang.md#high-load), which+keeps pace with plain decoding up to 512 concurrent requests.+EAGLE-4 (Extrapolation Algorithm for Greater Language-model Efficiency) speeds up LLM inference withspeculative decoding. EAGLE-4 draft models are built to run with `eagle-harness`, which decides how far tospeculate from measured costs on your setup, so every draft tree fits the batch size, the hardware and thediff --git a/docs/calibration.md b/docs/calibration.mdindex 8d4ab3c..070882c 100644--- a/docs/calibration.md+++ b/docs/calibration.md@@ -36,6 +36,28 @@ optimal acceptance or no-loss behavior at high concurrency; compare sustained woAfter seeding, a draft prefix is timed once per table-refresh period; target timings continue on served verifies.Prelaunched draft samples use that batch's bucket and context, not the preceding batch's seed key.+An SGLang chain (`"intensity": "chain_intensity"`, `--speculative-eagle-topk 1`) without `data_path` does not use+these cost tables. It picks one verify width per batch size (keyed to the nearest calibration batch size) from its+own served rounds: committed tokens per wall-clock time over whole served intervals, prefill steps included, so the+draft work a prefill or a batch change throws away is priced. Two widths are compared at a time on alternating+blocks of rounds, widening while the wider one wins. When the engine captures native decode graphs, native decode+is measured last at a batch size and, where it wins, serves that batch size and every larger one: its requests stop+drafting until they finish. The choices are cached next to the live costs (`<key>-chain.json`), so a restart on the+same setup serves them from its first round. A first start on a B300 TP4 with the Qwen3.8-27B checkpoint pair serves+packaged choices instead (width 4 at batch 16 and 64, native decode from 256;+[profile](../eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/README.md)) and searches only the other batch+sizes; on other setups the first start pays the search.++A serve-mode tree (the default) keeps the cost tables for the shape of each round and decides per batch size only+whether to speculate at all, on the same served intervals: speculation is priced by the tokens its rounds committed,+rounds that seed the live costs are not counted, and native decode is measured last and must lose by two standard+errors. Requests decoded natively stop drafting until they finish, and so do requests that join their batch, so+native decode is not measured where speculation beats its estimate from the live costs (the one-token verify) by+1.5x. Native decode at a batch size serves every larger one and speculation every smaller one. The choices are cached+as `<key>-tree.json`; a first start on the B300 TP4 setup above serves packaged choices (speculation up to batch 8,+native decode from 64) and searches the batch sizes in between. Without native decode graphs in the engine, or with+Mamba prefix-cache tracking or DP attention, the tree speculates at every batch size, as before.+Offline calibration below remains the precise mode: it times isolated steps on dummy batches over a fine contextgrid. Its tables carry the same key in `key.json`; a server refuses tables keyed for another setup, and acceptstables without a key (measured before keys existed) with a warning.@@ -151,6 +173,8 @@ or a private proxy before exposing the API.The vLLM sweep takes `context_stop` from the resolved `max_model_len` (including engine overrides),records it in each raw file, and fits that same range. The default engine limit remains 4608.+The engine and `serve.sh` keep vLLM's default `max_num_batched_tokens`, which fits one batch of+`max_num_seqs` requests at `max_model_len`. A value set through `--engine-args` must stay within that product.The limit must be a multiple of the context step, with room for the widest verify after thelast sample; unsupported grids fail rather than silently measuring a shorter context.Raw files without `context_stop` must be regenerated. Context-indexed cost tables log oncediff --git a/docs/known-limits.md b/docs/known-limits.mdindex 20dbe4d..356d4b0 100644--- a/docs/known-limits.md+++ b/docs/known-limits.md@@ -1,7 +1,7 @@---type: referenceowner: eagle-harness-last_verified: 2026-10-21+last_verified: 2026-10-23---# Known limits@@ -21,15 +21,45 @@ kernel is a startup check.- Throughput depends on the model, draft, engine, precision, TP size, attention backend, context length andworkload. A result at one concurrency does not establish a result at another.+## High load++On SGLang with four GPUs, the [high-load recipe](quickstart-sglang.md#high-load) keeps pace with plain decoding up+to 512 concurrent requests. These limits apply around it:++- The SGLang command of the README quick start is slower than plain decoding above about 32 to 64 concurrent+ requests. On one B300 it measured 0.73× the same engine without speculation at concurrency 64 and 0.59× at 256.+ With `--tp-size 4` it measured 0.80× at 64, 0.48× at 128, 0.37× at 256 and 0.32× at 512 (2 starts each). Three+ settings cause it. The radix cache, on by default, makes this hybrid model track prefix-cache state that a batch+ switching to plain decoding cannot carry yet, so the server speculates at every load. The default tree caps a+ four-GPU server at 32 running requests, while plain decoding admits several hundred. On one GPU, the engine+ reserves verify state for every admitted request, so a speculative server admits about a quarter of the+ requests plain decoding does. For high load on four GPUs, use the high-load recipe. A fix for the default+ command is planned.+- High load on one GPU is not supported yet. On one GPU, a speculative server started with+ `--disable-radix-cache` stops during draft graph capture, so keep the radix cache on there. The high-load recipe+ is for four GPUs, and the server log's advice to add `--disable-radix-cache` applies to four GPUs only.+- The first start of the high-load recipe on a machine compiles kernels and measures its live costs during its+ first requests. It measured 0.95× at 64 and 0.97× at 128 concurrent requests, and restarts with warm caches+ 1.01× and 1.00× (3 starts each).+- The packaged choices cover B300 TP4 with the Qwen3.8-27B checkpoint pair. On other setups the first start also+ searches its choices while serving: with the packaged choices withheld on that same setup, first starts measured+ 0.80× at concurrency 64 (4 starts, from 0.71× to 0.96×) and restarts 0.94× (2 starts). From 128 concurrent+ requests up they measured 0.97× or more.+- Requests decoded without speculation stop drafting until they finish, and so do requests that join their+ batch. After a load spike, a batch keeps decoding without speculation until it empties, even when the load+ drops in the meantime.+- Greedy agreement at 64 and 512 concurrent requests is checked with the fp32 rule, not claimed. At 64, one prompt+ takes another token at one position in 3 of 12 EAGLE-4 starts, a prefix where the engine without speculation+ emits the same token under another batch shape. At 512, the engine without speculation also differs between its+ own starts. Every twenty-prompt check passes.+- vLLM and TokenSpeed do not switch to plain decoding at high load yet. Above concurrency 64 they can be slower+ with speculation than without it, so serve those loads without speculation for now. The switch is planned for+ both engines.+## Performance and correctness in progress- SGLang TP4 single-request and 32K-context optimization is in progress, as is TP8 concurrency-8 prefilloverhead and intermittent stalls. Repeated passes in one process are not independent server starts.-- SGLang above concurrency 64: with a running cap of 512 and the default tree (TP4), EAGLE-4 measured 1.03x the- same engine without speculation at concurrency 64, 0.74x at 256 and 0.61x at 512 (3 starts each). Serve- concurrency above 64 without speculation for now. The automatic high-load policy and high-load fixed-chain- improvements are being qualified and stay off by default. Greedy agreement with no-spec at concurrency 512 is- not claimed: the engine without speculation also differs between two starts there.- SGLang native-OFF, including OFF forced by a cost profile, uses the engine's per-variant forward-modedecode graphs (the `forward_mode` field of the decode graph variant). The SGLang CUDA engine`0.5.23.dev252+eagle.5` has them. On the 0.1.0 SGLang CUDA engine and on ROCm, EAGLE-4 keeps native-OFF off.@@ -45,8 +75,8 @@ kernel is a startup check.optimized to match native decoding.- vLLM TP4 at concurrency 64: in a 64-prompt greedy comparison against no-spec, one prompt can take another tokenat one position, up to 5.4 bf16 ULP from the fp32 choice, beyond the 2-ULP rule. It appeared in 3 of 6 EAGLE-4- starts with 0.1.1 and in 2 of 6 with 0.1.0 (the vLLM engine is the same); the other prompts and every- twenty-prompt check at concurrency 1 pass. The cause is being investigated.+ starts with 0.1.1 and in 2 of 6 with 0.1.0, and 0.1.2 shows it too (the vLLM engine is the same); the other+ prompts and every twenty-prompt check at concurrency 1 pass. The cause is being investigated.- AMD measurements apply to the MI355X TP1 Qwen3.8 profile. On one MI355X the speedup covers concurrency upto 32; concurrency 64 is being optimized, so serve it without speculation for now. HIP side-stream prelaunchstays off: its replay check passes and it measured no throughput gain on MI355X. Further target, draft anddiff --git a/docs/quickstart-sglang.md b/docs/quickstart-sglang.mdindex d39d428..69daed9 100644--- a/docs/quickstart-sglang.md+++ b/docs/quickstart-sglang.md@@ -1,7 +1,7 @@---type: tutorialowner: eagle-harness-last_verified: 2026-10-21+last_verified: 2026-10-23---# SGLang guide@@ -80,7 +80,48 @@ sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 --dtype bfloat16 \The benchmark warmed the server from its client, hence `--skip-server-warmup`: the first request after this startcompiles kernels and seeds the costs. Keep CUDA graph padding enabled, and do not add`--disable-cuda-graph-padding`. `--max-running-requests 32` is the tree default; set a larger cap, up to 512, for-larger batches.+larger batches, or serve with the [high-load recipe](#high-load).++## High load++For many concurrent requests on four GPUs, serve with the high-load recipe. It raises the running cap and the decode+graph limit to 512 and turns the radix cache off, so EAGLE-4 can decode a batch without speculation wherever+speculation would be slower:++```bash+export TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=4+export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 SGLANG_TIMEOUT_KEEP_ALIVE=3600+sglang serve --model-path Qwen/Qwen3.8-27B --tp-size 4 --dtype bfloat16 \+ --trust-remote-code --attention-backend flashinfer --page-size 1 \+ --max-running-requests 512 --cuda-graph-max-bs-decode 512 \+ --mem-fraction-static 0.70 --context-length 32768 \+ --chunked-prefill-size 16384 --mamba-ssm-dtype bfloat16 --disable-radix-cache \+ --random-seed 721 --skip-server-warmup \+ --speculative-algorithm EAGLE_HARNESS \+ --speculative-draft-model-path EAGLE-Corp/Qwen3.8-27B-EAGLE4 \+ --speculative-draft-model-quantization unquant --speculative-draft-attention-backend flashinfer \+ --speculative-attention-mode prefill --speculative-num-steps 12 \+ --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 \+ --speculative-algorithm-config '{"mode":"serve"}'+```++With the radix cache on, this hybrid model tracks prefix-cache state that a batch switching to plain decoding+cannot carry, so the server speculates at every load. On one GPU, keep the radix cache on: high load on one GPU is+not supported yet ([known limits](known-limits.md#high-load)).++Each speedup divides EAGLE-4 throughput by the same command without the `--speculative-*` flags, in the same+allocation, on 4x B300 with Qwen3.8-27B and its EAGLE-4 draft at temperature 0. Columns are concurrent requests:++| Server start | 1 | 8 | 64 | 128 | 256 | 512 |+| --- | ---: | ---: | ---: | ---: | ---: | ---: |+| First start on a machine | 2.80× | 1.67× | 0.95× | 0.97× | 1.00× | 0.99× |+| Restart with warm caches | 2.76× | 1.72× | 1.01× | 1.00× | 1.00× | 0.99× |++The first start compiles kernels and measures its live costs during its first requests, which costs up to 5% at 64+and 128 concurrent requests. A restart reuses the kernels and the costs and choices cached under+`~/.cache/eagle-harness/costs`. This setup ships packaged choices; on another GPU, TP size or model pair the first+start also searches them ([known limits](known-limits.md#high-load)). The+[support matrix](support-matrix.md#012-high-load) has the tokens/s and the correctness checks.Live costs are cached per GPU, TP size, model pair, engine and dtype under `~/.cache/eagle-harness/costs`.The [calibration guide](calibration.md) explains live costs and offline chain calibration, and thediff --git a/docs/release-notes/0.1.2.md b/docs/release-notes/0.1.2.mdnew file mode 100644index 0000000..ca915d9--- /dev/null+++ b/docs/release-notes/0.1.2.md@@ -0,0 +1,93 @@+---+type: reference+owner: eagle-harness+last_verified: 2026-10-23+---++# EAGLE-4 0.1.2 release notes++EAGLE-4 0.1.2 updates `eagle-harness` on the 0.1.1 engines. On SGLang, EAGLE-4 decodes a batch without speculation+wherever speculation would be slower, so the [high-load recipe](../quickstart-sglang.md#high-load) keeps pace with+plain decoding up to 512 concurrent requests on four B300 GPUs and keeps its speedup at low load.+`eagle-harness calibrate --engine vllm` completes on hybrid targets, and the TokenSpeed draft runs its candidate head+on the engine's low-latency GEMM.++## Upgrade++A new install follows the [installation guide](../install.md), which installs 0.1.2. To upgrade a 0.1.1+environment, run your engine's install command with the pins below, then `eagle-harness-check`:++| Engine | 0.1.2 pins |+| --- | --- |+| SGLang CUDA | `eagle-harness[sglang]==0.1.2`, `sglang==0.5.23.dev252+eagle.5` (unchanged) |+| vLLM | `eagle-harness[vllm]==0.1.2`, `vllm==0.32.0rc1.dev283+eagle.4` (unchanged) |+| TokenSpeed | `eagle-harness[tokenspeed]==0.1.2`, `tokenspeed==0.1.1+eagle.5`, `tokenspeed-kernel==0.1.4+eagle.5`, `tokenspeed-scheduler==0.1.40+eagle.4` (unchanged) |++The `:0.1.1` images carry `eagle-harness` 0.1.1. The [install guide](../install.md#docker-images) lists the current+images. Nothing in the configuration changes: every 0.1.1 configuration loads as before.++## What changed++**SGLang**+- A served tree, the default, decides per batch size whether to speculate at all. It prices speculation by the+ tokens its rounds commit over whole served intervals, measures plain decoding last, and decodes a batch without+ speculation where plain decoding wins. Requests decoded that way stop drafting until they finish.+- Served chains (`"intensity": "chain_intensity"`) pick one verify width per batch size the same way, and decode+ without speculation where that wins.+- On B300 TP4 with the Qwen3.8-27B checkpoint pair, the first start serves packaged choices and searches only the+ batch sizes in between. The choices are cached next to the live costs, so a restart on any setup serves them from+ its first round. [Calibration](../calibration.md) describes both policies.+- Every KV-move program a tree round runs compiles at server startup, and rounds that stalled on one-time work or+ followed a switch between speculation and plain decoding stay out of the prices.+- The switch needs the radix cache off (`--disable-radix-cache`), because with it on this hybrid model tracks+ prefix-cache state that plain decoding does not carry. A server with the radix cache on speculates at every load+ and says so in its log.++**vLLM**+- `eagle-harness calibrate --engine vllm` completes on the Qwen3.8 hybrid target: its token budget fits one batch,+ the tables are keyed by the models the engine resolves, and the printed `serve.sh` loads them.++**TokenSpeed**+- The draft scores its low-rank candidate head through the engine's linear layer, so TokenSpeed's GEMM dispatch+ serves the few draft rows with its low-latency BF16 kernel. Checkpoints load as before.++## Measured on the release stack++Qwen3.8-27B with its EAGLE-4 draft on 4x B300 (TP4), BF16, temperature 0, on the release candidate build installed+into clean environments and served from outside any checkout. Each speedup divides EAGLE-4 throughput by the same+engine without speculation, measured in the same allocation. Every EAGLE-4 start passed its twenty-prompt greedy+check against no-spec under the fp32 rule.++| Engine and command | Concurrency | EAGLE-4 tokens/s | Speedup | EAGLE-4 starts |+| --- | ---: | ---: | ---: | ---: |+| SGLang, measured recipe | 1 | 733.8 | 3.35× | 2 |+| SGLang, measured recipe | 8 | 3257.9 | 2.11× | 2 |+| SGLang, measured recipe | 64 | 6488.2 | 1.38× | 2 |+| SGLang, high-load recipe | 64 | 7393.3 | 1.01× | 3 |+| SGLang, high-load recipe | 128 | 11114.3 | 1.00× | 3 |+| SGLang, high-load recipe | 256 | 14332.3 | 1.00× | 3 |+| SGLang, high-load recipe | 512 | 16197.5 | 0.99× | 3 |+| vLLM, measured recipe | 1 | 678.5 | 2.83× | 4 |+| vLLM, measured recipe | 8 | 2887.4 | 1.73× | 4 |+| vLLM, measured recipe | 64 | 8013.4 | 1.19× | 4 |+| TokenSpeed, measured recipe | 1 | 886.5 | 3.13× | 2 |+| TokenSpeed, measured recipe | 8 | 3256.4 | 2.00× | 2 |+| TokenSpeed, measured recipe | 64 | 6974.1 | 1.40× | 2 |++The high-load rows are restarts with warm caches. A first start on a machine measured 0.95× at 64 and 0.97× at+128 while it compiles kernels and measures costs. With 0.1.1, the same SGLang setup at a running cap of 512+measured 0.74× at 256 and 0.61× at 512. At concurrency 1, SGLang and TokenSpeed serve multi-turn conversations+with up to 1024 new tokens per turn and vLLM 22 first turns with up to 512. The+[support matrix](../support-matrix.md#012-released-current) has temperature 1, one GPU, no-spec throughput and the+conditions of each row.++## Limits++- The SGLang quick-start command is slower than plain decoding above about 32 to 64 concurrent requests, on one+ GPU and with `--tp-size 4`. For high load, serve with the high-load recipe on four GPUs. A fix for the+ quick-start command is planned.+- High load on one GPU is not supported yet. On one GPU, keep the radix cache on.+- vLLM and TokenSpeed do not switch to plain decoding at high load yet. Above concurrency 64, serve them without+ speculation for now.+- [Known limits](../known-limits.md#high-load) has the measured numbers and the rest of the limits, and the+ [support matrix](../support-matrix.md) has every measured configuration.diff --git a/docs/support-matrix.md b/docs/support-matrix.mdindex 30b0e22..8c7bcd7 100644--- a/docs/support-matrix.md+++ b/docs/support-matrix.md@@ -1,14 +1,14 @@---type: referenceowner: eagle-harness-last_verified: 2026-10-21+last_verified: 2026-10-23---# Measured support matrixEach cell divides EAGLE-4 throughput by the same engine's matched no-speculation control, measured in the same-allocation, on Qwen3.8-27B with its EAGLE-4 draft unless the table names another model. The page has three parts:-[0.1.0](#010-released), [0.1.1](#011-released-current) and+allocation, on Qwen3.8-27B with its EAGLE-4 draft unless the table names another model. The page has four parts:+[0.1.0](#010-released), [0.1.1](#011-released), [0.1.2](#012-released-current) and[unreleased candidates](#unreleased-measured-on-candidates-not-in-a-release). Shipped and candidate numbers nevershare a row. Column headers are client concurrency, the number of requests the client keeps in flight.@@ -130,7 +130,7 @@ test the published `0.1.0` images, each pullable without credentials, and the SG`eagle-harness` 0.1.0 wheel from PyPI. TokenSpeed has no full-decode native top-logprobs yet, so each divergentprefix in its checks goes directly to fp32 arbitration.-## 0.1.1 (released, current)+## 0.1.1 (released)`eagle-harness` 0.1.1 with `sglang==0.5.23.dev252+eagle.5`, the unchanged `vllm==0.32.0rc1.dev283+eagle.4`, and`tokenspeed==0.1.1+eagle.5` with `tokenspeed-kernel==0.1.4+eagle.5` and `tokenspeed-scheduler==0.1.40+eagle.4`.@@ -215,6 +215,88 @@ The published 0.1.1 images (`eagle-4-sglang:0.1.1`, `eagle-4-vllm:0.1.1`, `eaglechecks as the 0.1.0 images: SGLang 5 exact / 15 near ties, vLLM 7 exact / 12 near ties / 1 fp32-tolerance pass, andTokenSpeed in serve mode 6 exact / 8 fp32-tolerance passes / 6 fp32-argmax.+## 0.1.2 (released, current)++`eagle-harness` 0.1.2 with the 0.1.1 engines: `sglang==0.5.23.dev252+eagle.5`, `vllm==0.32.0rc1.dev283+eagle.4`,+and `tokenspeed==0.1.1+eagle.5` with `tokenspeed-kernel==0.1.4+eagle.5` and `tokenspeed-scheduler==0.1.40+eagle.4`.+Every row runs on NVIDIA B300, on the release candidate build of `eagle-harness` 0.1.2 installed into clean+environments with the published engine wheels and served from outside any checkout. In the same allocations, 0.1.2+measured between 0.98× and 1.07× the 0.1.1 packages in every cell where both ran. The SGLang ROCm install keeps its+0.1.0 packages, so the 0.1.0 MI355X row applies.++### 0.1.2 speed by engine and concurrency++| Engine and setup | 1 | 8 | 64 |+|---|---|---|---|+| SGLang TP4, temperature 0 | ✅ 3.35× | ✅ 2.11× | ✅ 1.38× |+| SGLang TP4, temperature 1 | ✅ 2.74× | ✅ 1.78× | ✅ 1.25× |+| SGLang TP1, quick-start command | ✅ 2.87× | ✅ 1.87× | ⚠️ 0.73× |+| vLLM TP4 | ✅ 2.83× | ✅ 1.73× | ⚠️ 1.19× |+| TokenSpeed TP4 | ✅ 3.13× | ✅ 2.00× | ✅ 1.40× |++Sources:++1. SGLang TP4, temperature 0: the measured recipe of the [SGLang guide](quickstart-sglang.md#the-measured-recipe)+ (running cap 32). EAGLE-4 733.8, 3257.9 and 6488.2 tokens/s, no-spec 219.0, 1545.2 and 4718.6, 2 starts per+ arm. The 0.1.1 packages in the same allocation measured 3.36×, 2.11× and 1.40×. Every start gives 5 exact /+ 15 near tie.+2. SGLang TP4, temperature 1, same starts: EAGLE-4 595.6, 2739.8 and 5827.9 tokens/s, no-spec at temperature 1+ 217.4, 1539.4 and 4675.2.+3. SGLang TP1: the quick-start command on one GPU, 20, 64 and 384 SPEED-Bench first turns with up to 256 new+ tokens. EAGLE-4 302.7, 1271.8 and 2096.3 tokens/s, no-spec 105.5, 679.9 and 2870.2, 2 starts per arm. The 0.1.1+ packages measured 2.90×, 1.91× and 0.75×. The twenty-prompt and concurrency-64 checks pass. Concurrency 64 is a+ [known limit](known-limits.md#high-load).+4. vLLM TP4: the measured command of the [vLLM guide](quickstart-vllm.md) (running cap 64, packaged rounds file),+ two allocations with 2 starts per arm each. Concurrency 1 is 22 first turns with up to 512 new tokens. EAGLE-4+ 678.5, 2887.4 and 8013.4 tokens/s, no-spec 239.4, 1668.9 and 6725.4. Every twenty-prompt check passes. The+ 64-prompt check at concurrency 64 shows the one-prompt divergence of 0.1.1, which the 0.1.1 packages also show in+ the same allocation ([known limits](known-limits.md#performance-and-correctness-in-progress)).+5. TokenSpeed TP4: serve mode with TokenSpeed cost tables, running cap 4 at 1 and 32 at 8 and 64, 2 starts per+ arm. Concurrency 1 is 22 conversations and 31 turns with up to 1024 new tokens: 886.5 against 283.5 tokens/s.+ Eleven first turns with up to 256 tokens give 819.7 against 280.7 (2.92×). Concurrency 8 and 64 use 22 and 88+ first turns with up to 256 tokens: 3256.4 against 1628.2 and 6974.1 against 4969.8. Every start gives 6 exact /+ 14 fp32 pass.++### 0.1.2 high load++SGLang TP4 with a running cap of 512: the [high-load recipe](quickstart-sglang.md#high-load), and the quick-start+command with `--tp-size 4`. Concurrency 64 serves 384 SPEED-Bench first turns and higher concurrencies serve four+times the concurrency, each with up to 256 new tokens.++| Engine and setup | 64 | 128 | 256 | 512 |+|---|---|---|---|---|+| SGLang TP4, high-load recipe, first start | ⚠️ 0.95× | ✅ 0.97× | ✅ 1.00× | ⚠️ 0.99× |+| SGLang TP4, high-load recipe, restart with warm caches | ⚠️ 1.01× | ✅ 1.00× | ✅ 1.00× | ⚠️ 0.99× |+| SGLang TP4, quick-start command with `--tp-size 4` | ⚠️ 0.80× | ⚠️ 0.48× | ⚠️ 0.37× | ⚠️ 0.32× |+| SGLang TP1, quick-start command | ⚠️ 0.73× | — | ⚠️ 0.59× | — |++Sources:++1. High-load recipe, first start on a machine (empty kernel and cost caches), 3 starts over three allocations,+ each against the mean of 3 no-spec starts in its own allocation: EAGLE-4 6935.2, 10737.0, 14253.5 and+ 16152.6 tokens/s, no-spec 7313.7, 11086.4, 14264.9 and 16340.7. At 1 and 8 concurrent requests it measured 2.80×+ and 1.67× (704.3 and 2732.3 against 251.7 and 1634.7). Concurrency 64 ranges from 0.935× to 0.969× over the+ three starts.+2. Restart with the first start's caches, 3 starts in the same allocations: 7393.3, 11114.3, 14332.3 and+ 16197.5 tokens/s, and 2.76× and 1.72× at 1 and 8. Every twenty-prompt check passes for both rows. The 384-prompt+ check at 64 has one prompt outside the fp32 rule in 2 of 6 starts, at a prefix where the engine without+ speculation emits the same token under another batch shape. The 2048-prompt check at 512 has one to three such+ prompts in 3 of 6 starts, and one to three between two no-spec starts in every allocation.+3. Quick-start command with `--tp-size 4`: 4377.7, 4723.1, 4490.4 and 4503.0 tokens/s against 5486.2, 9782.2,+ 12034.3 and 14159.3, 2 starts per arm. The server admits 32 running requests (the default tree), and the radix+ cache keeps the batch on speculation. The 0.1.1 packages measured 0.75×, 0.48×, 0.38× and 0.32×.+4. TP1 quick-start command, the starts of the speed table: 2183.0 against 3691.8 tokens/s at 256. The 0.1.1+ packages measured 0.75× and 0.60×. See [known limits](known-limits.md#high-load).++### 0.1.2 correctness and request features++| Change | Result |+|---|---|+| `eagle-harness calibrate --engine vllm` completes on the Qwen3.8 hybrid target, and its `serve.sh` loads the tables it wrote | ✅ 20/20 |++The calibration ran on B300 TP4 with the 0.1.1 vLLM engine. Both the printed `serve.sh` and the quickstart command+pass the twenty-prompt check (4 exact / 15 near tie / 1 fp32 pass), 1 start each. No speed is claimed for it.+## Unreleased (measured on candidates, not in a release)These rows measure changes that are not in a published release. Each row names the change and its engine build.@@ -299,15 +381,6 @@ Sources:3. TokenSpeed: running cap 8, 11 and 22 first turns with up to 256 tokens, 2 starts per arm. 597.3 against 312.2 and2122.4 against 1167.0 tokens/s. The check has 1 failure, reproduced by the engine without speculation.-### Candidate correctness checks--| Change | Result |-|---|---|-| `eagle-harness calibrate --engine vllm` completes on the Qwen3.8 hybrid target, and its `serve.sh` loads the tables it wrote | ✅ 20/20 |--The calibration ran on B300 TP4 with the 0.1.1 vLLM engine. Both the printed `serve.sh` and the quickstart command-pass the twenty-prompt check (4 exact / 15 near tie / 1 fp32 pass), 1 start each. No speed is claimed for it.-### In progress1. vLLM sampled-distribution checks for settings other than temperature 1 with top-p 1.diff --git a/eagle_harness/cli.py b/eagle_harness/cli.pyindex d07ca55..4af76e7 100644--- a/eagle_harness/cli.py+++ b/eagle_harness/cli.py@@ -114,6 +114,7 @@ def serving_command(args: argparse.Namespace) -> list[str]:"data_path": str(args.output),},}+ # max_num_batched_tokens stays vLLM's default, which fits one full batch (see eagle_harness.vllm.calibrate.engine).options = {"tensor_parallel_size": args.tp,"host": "127.0.0.1",@@ -122,7 +123,6 @@ def serving_command(args: argparse.Namespace) -> list[str]:"max_model_len": 4608,"gpu_memory_utilization": 0.85,"max_num_seqs": 1,- "max_num_batched_tokens": 8192,"seed": 0,}options.update(args.engine_args)diff --git a/eagle_harness/core/cost_profiles.py b/eagle_harness/core/cost_profiles.pyindex 2c94850..754a042 100644--- a/eagle_harness/core/cost_profiles.py+++ b/eagle_harness/core/cost_profiles.py@@ -1,17 +1,21 @@# Copyright 2026 Eagle AI Corp.# SPDX-License-Identifier: LicenseRef-Eagle-4-Source-Available-"""Packaged serving costs for measured device and checkpoint configurations."""+"""Packaged serving costs and choices for measured device and checkpoint configurations."""from __future__ import annotationsimport jsonfrom pathlib import Path-from typing import Optional+from typing import Dict, Optional, SequenceMI355X_PROFILE = Path(__file__).parent / "profiles" / "mi355x-qwen3.8-27b-tp1"+B300_SERVED_PROFILE = Path(__file__).parent / "profiles" / "b300-qwen3.8-27b-tp4-chain"-def packaged_cost_profile(+def _measured_on(+ profile: Path,+ devices: Sequence[str],+ tp: int,*,device_name: str,model_path: str,@@ -19,27 +23,85 @@ def packaged_cost_profile(tp_size: int,dtype: str,quantization: Optional[str],-) -> Optional[str]:- """Select the MI355X BF16 TP1 profile only for its measured checkpoint configs.-- Model directory names do not participate in identity. Unknown devices,- unresolved Hub paths, different configs, TP sizes or dtypes keep the caller's- ordinary calibration path; a profile for one model must not price another.- """- if device_name not in ("AMD Instinct MI355X", "AMD Instinct MI355X VF"):- return None- if tp_size != 1 or dtype not in ("auto", "bfloat16", "torch.bfloat16") or quantization is not None:- return None+) -> bool:+ """Whether a setup is the one ``profile`` was measured on: one of ``devices``, ``tp`` ranks, BF16 without+ quantization, and the target and draft ``config.json`` of the profile (parsed: directory names do not matter)."""+ if device_name not in devices or tp_size != tp:+ return False+ if dtype not in ("auto", "bfloat16", "torch.bfloat16") or quantization is not None:+ return Falsefor model, name in ((model_path, "target"), (draft_model_path, "draft")):if not model:- return None+ return Falseconfig_path = Path(model) / "config.json"if not config_path.is_file():- return None+ return Falsewith config_path.open(encoding="utf-8") as file:config = json.load(file)- with (MI355X_PROFILE / f"{name}_config.json").open(encoding="utf-8") as file:+ with (profile / f"{name}_config.json").open(encoding="utf-8") as file:measured = json.load(file)if config != measured:- return None- return str(MI355X_PROFILE)+ return False+ return True+++def packaged_cost_profile(+ *,+ device_name: str,+ model_path: str,+ draft_model_path: str,+ tp_size: int,+ dtype: str,+ quantization: Optional[str],+) -> Optional[str]:+ """Select the MI355X BF16 TP1 profile only for its measured checkpoint configs.++ Unknown devices, unresolved Hub paths, different configs, TP sizes or dtypes keep the caller's+ ordinary calibration path; a profile for one model must not price another.+ """+ if _measured_on(+ MI355X_PROFILE,+ ("AMD Instinct MI355X", "AMD Instinct MI355X VF"),+ 1,+ device_name=device_name,+ model_path=model_path,+ draft_model_path=draft_model_path,+ tp_size=tp_size,+ dtype=dtype,+ quantization=quantization,+ ):+ return str(MI355X_PROFILE)+ return None+++def packaged_served_choices(+ kind: str,+ *,+ device_name: str,+ model_path: str,+ draft_model_path: str,+ tp_size: int,+ dtype: str,+ quantization: Optional[str],+) -> Optional[Dict[int, int]]:+ """The served choice per batch key measured on this setup, or None: for a ``"chain"`` the verify width, for a+ ``"tree"`` 1 to speculate (the cost model's tree) or 0 to decode natively.++ Only the B300 BF16 TP4 setup of the profile's checkpoint pair has them. A served chain or tree+ (:class:`~eagle_harness.sglang.chain_rounds.ServedChainPolicy`, ``ServedTreePolicy``) starts from them where it+ has no choices of its own.+ """+ if not _measured_on(+ B300_SERVED_PROFILE,+ ("NVIDIA B300 SXM6 AC",),+ 4,+ device_name=device_name,+ model_path=model_path,+ draft_model_path=draft_model_path,+ tp_size=tp_size,+ dtype=dtype,+ quantization=quantization,+ ):+ return None+ with (B300_SERVED_PROFILE / f"{kind}.json").open(encoding="utf-8") as file:+ return {int(key): int(choice) for key, choice in json.load(file)[kind].items()}diff --git a/eagle_harness/core/live_costs.py b/eagle_harness/core/live_costs.pyindex e8dd41d..6ff8b00 100644--- a/eagle_harness/core/live_costs.py+++ b/eagle_harness/core/live_costs.py@@ -189,6 +189,10 @@ class LiveCosts:# Seeding counts samples across intervals; crossing a boundary must still discard a first-use stall.return min((c.us for (b, w, _), c in self.cells.items() if b == bucket and w == width and c.samples), default=None)+ def verify_us(self, bucket: int, width: int) -> Optional[float]:+ """The priced verify of ``width`` tokens per request at ``bucket`` (shortest context first), None untimed."""+ return self._value(bucket, width, 0)+def tables(self) -> Dict[int, BatchCostTable]:"""One table per ready bucket (the cost model's input), from the current cells."""tables: Dict[int, BatchCostTable] = {}diff --git a/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/README.md b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/README.mdnew file mode 100644index 0000000..1ee7c7e--- /dev/null+++ b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/README.md@@ -0,0 +1,23 @@+# B300 Qwen3.8-27B TP4 served chain widths and tree choices++An SGLang chain without `data_path` (`"intensity": "chain_intensity"`, `--speculative-eagle-topk 1`) starts from+these widths on an NVIDIA B300 SXM6 AC, BF16, TP4, with the measured target and draft checkpoint configurations.+Selection compares parsed local `config.json` content with `target_config.json` and `draft_config.json`; directory+names do not matter. Other devices, TP sizes, dtypes, quantization or model configurations search on their served+rounds from the first round.++`chain.json` maps a batch key (the nearest calibration batch size) to the verify width served there; 0 is native+decode, used only when the engine captures native decode graphs (otherwise the server searches). The batch keys+come from forced-width runs on SPEED-Bench prompts with 256-token outputs, matched against no-spec in the same job:+width 4 at batch 16 and 64, native decode at 256. Native decode at a batch key also serves every larger one. The+remaining batch keys are searched on the served rounds, never wider than a smaller key's width.++`tree.json` does the same for a serve-mode tree (the default shape): 1 speculates (the cost model's tree), 0 decodes+natively. Two cold starts with the default tree at a 512-request cap (SPEED-Bench prompts, 256-token outputs) chose+speculation up to batch 8 and native decode at 64; native decode at 64 also serves every larger batch key, and+speculation at 8 every smaller one. Batch keys 12 to 48 are searched on the served rounds.++Choices the server makes itself (`<cost_cache>/<key>-chain.json`, `<key>-tree.json`) take precedence. A workload with very different+acceptance or output lengths may favor other widths; calibrate it (`"mode": "calibrate"`) and serve the resulting+`data_path`.+This directory contains no model weights.diff --git a/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/chain.json b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/chain.jsonnew file mode 100644index 0000000..127d10e--- /dev/null+++ b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/chain.json@@ -0,0 +1,7 @@+{+ "chain": {+ "16": 4,+ "64": 4,+ "256": 0+ }+}diff --git a/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/draft_config.json b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/draft_config.jsonnew file mode 100644index 0000000..29dab0b--- /dev/null+++ b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/draft_config.json@@ -0,0 +1,57 @@+{+ "_torchspec_version": "0.1.0",+ "architectures": [+ "LlamaForCausalLMEagle4"+ ],+ "attention_bias": false,+ "attention_dropout": 0.0,+ "bos_token_id": 248044,+ "draft_vocab_size": 248320,+ "dtype": "bfloat16",+ "eagle_config": {+ "eagle_aux_hidden_state_layer_ids": [+ 3,+ 31,+ 59+ ],+ "use_aux_hidden_state": true+ },+ "efficient_lm_head_config": {+ "num_candidates": 128,+ "rank": 128,+ "type": "low_rank"+ },+ "eos_token_id": 248044,+ "fc_norm": true,+ "head_dim": 128,+ "hidden_act": "silu",+ "hidden_size": 5120,+ "initializer_range": 0.02,+ "intermediate_size": 16384,+ "layer_types": [+ "sliding_attention"+ ],+ "max_position_embeddings": 262144,+ "mlp_bias": false,+ "model_type": "llama",+ "norm_output": true,+ "num_attention_heads": 40,+ "num_aux_hidden_states": 3,+ "num_hidden_layers": 1,+ "num_key_value_heads": 40,+ "pad_token_id": 248044,+ "pretraining_tp": 1,+ "rms_norm_eps": 1e-06,+ "rope_parameters": {+ "rope_theta": 10000000.0,+ "rope_type": "default"+ },+ "sliding_window": 4096,+ "target_hidden_size": 5120,+ "tie_word_embeddings": false,+ "transformers_version": "5.8.1",+ "use_cache": true,+ "use_sliding_window": true,+ "vocab_size": 248320,+ "torch_dtype": "bfloat16"+}\ No newline at end of filediff --git a/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/target_config.json b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/target_config.jsonnew file mode 100644index 0000000..706cebd--- /dev/null+++ b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/target_config.json@@ -0,0 +1,140 @@+{+ "architectures": [+ "Qwen3_5ForConditionalGeneration"+ ],+ "image_token_id": 248056,+ "language_model_only": false,+ "model_type": "qwen3_5",+ "text_config": {+ "attention_bias": false,+ "attention_dropout": 0.0,+ "attn_output_gate": true,+ "bos_token_id": 248044,+ "dtype": "bfloat16",+ "eos_token_id": 248044,+ "full_attention_interval": 4,+ "head_dim": 256,+ "hidden_act": "silu",+ "hidden_size": 5120,+ "initializer_range": 0.02,+ "intermediate_size": 17408,+ "layer_types": [+ "linear_attention",+ "linear_attention",+ "linear_attention",+ "full_attention",+ "linear_attention",+ "linear_attention",+ "linear_attention",+ "full_attention",+ "linear_attention",+ "linear_attention",+ "linear_attention",+ "full_attention",+ "linear_attention",+ "linear_attention",+ "linear_attention",+ "full_attention",+ "linear_attention",+ "linear_attention",+ "linear_attention",+ "full_attention",+ "linear_attention",+ "linear_attention",+ "linear_attention",+ "full_attention",+ "linear_attention",+ "linear_attention",+ "linear_attention",+ "full_attention",+ "linear_attention",+ "linear_attention",+ "linear_attention",+ "full_attention",+ "linear_attention",+ "linear_attention",+ "linear_attention",+ "full_attention",+ "linear_attention",+ "linear_attention",+ "linear_attention",+ "full_attention",+ "linear_attention",+ "linear_attention",+ "linear_attention",+ "full_attention",+ "linear_attention",+ "linear_attention",+ "linear_attention",+ "full_attention",+ "linear_attention",+ "linear_attention",+ "linear_attention",+ "full_attention",+ "linear_attention",+ "linear_attention",+ "linear_attention",+ "full_attention",+ "linear_attention",+ "linear_attention",+ "linear_attention",+ "full_attention",+ "linear_attention",+ "linear_attention",+ "linear_attention",+ "full_attention"+ ],+ "linear_conv_kernel_dim": 4,+ "linear_key_head_dim": 128,+ "linear_num_key_heads": 16,+ "linear_num_value_heads": 48,+ "linear_value_head_dim": 128,+ "mamba_ssm_dtype": "float32",+ "max_position_embeddings": 262144,+ "model_type": "qwen3_5_text",+ "mtp_num_hidden_layers": 1,+ "mtp_use_dedicated_embeddings": false,+ "num_attention_heads": 24,+ "num_hidden_layers": 64,+ "num_key_value_heads": 4,+ "output_gate_type": "swish",+ "pad_token_id": null,+ "partial_rotary_factor": 0.25,+ "rms_norm_eps": 1e-06,+ "rope_parameters": {+ "mrope_interleaved": true,+ "mrope_section": [+ 11,+ 11,+ 10+ ],+ "partial_rotary_factor": 0.25,+ "rope_theta": 10000000,+ "rope_type": "default"+ },+ "tie_word_embeddings": false,+ "use_cache": true,+ "vocab_size": 248320+ },+ "tie_word_embeddings": false,+ "transformers_version": "5.8.0.dev0",+ "video_token_id": 248057,+ "vision_config": {+ "deepstack_visual_indexes": [],+ "depth": 27,+ "hidden_act": "gelu_pytorch_tanh",+ "hidden_size": 1152,+ "in_channels": 3,+ "initializer_range": 0.02,+ "intermediate_size": 4304,+ "model_type": "qwen3_5",+ "num_heads": 16,+ "num_position_embeddings": 2304,+ "out_hidden_size": 5120,+ "patch_size": 16,+ "spatial_merge_size": 2,+ "temporal_patch_size": 2+ },+ "vision_end_token_id": 248054,+ "vision_start_token_id": 248053+}\ No newline at end of filediff --git a/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/tree.json b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/tree.jsonnew file mode 100644index 0000000..e97f609--- /dev/null+++ b/eagle_harness/core/profiles/b300-qwen3.8-27b-tp4-chain/tree.json@@ -0,0 +1,13 @@+{+ "tree": {+ "1": 1,+ "2": 1,+ "3": 1,+ "4": 1,+ "5": 1,+ "6": 1,+ "7": 1,+ "8": 1,+ "64": 0+ }+}diff --git a/eagle_harness/models/low_rank_head.py b/eagle_harness/models/low_rank_head.pyindex 361e666..fbe59d7 100644--- a/eagle_harness/models/low_rank_head.py+++ b/eagle_harness/models/low_rank_head.py@@ -34,7 +34,11 @@ def low_rank_scores() -> torch.Tensor:"""``[rows, shard vocab]`` rank-r scores of one vocab shard (``down_proj`` [rank, H], ``score_head`` [shard vocab,rank]), ``-inf`` past the shard's ``vocab_rows`` real rows."""- low_rank = hidden @ down_proj.t()+ return shard_scores(hidden @ down_proj.t(), score_head, vocab_rows)+++def shard_scores(low_rank: torch.Tensor, score_head: torch.Tensor, vocab_rows: int) -> torch.Tensor:+ """:func:`low_rank_scores` from the rank-r projection ``low_rank`` [rows, rank] the caller computed."""scores = low_rank.to(score_head.dtype) @ score_head.t()if scores.shape[-1] > vocab_rows:scores[:, vocab_rows:] = float("-inf")diff --git a/eagle_harness/sglang/algorithm.py b/eagle_harness/sglang/algorithm.pyindex c1fb8b5..d874191 100644--- a/eagle_harness/sglang/algorithm.py+++ b/eagle_harness/sglang/algorithm.py@@ -138,9 +138,14 @@ class EagleHarnessSpecAlgo(CustomSpecAlgo):load_round_times(self.config.data_path)if self.config.mode is Mode.SERVE and self.config.data_path else None)- plain = get_spec().speculative_eagle_topk == 1 and (- self.config.mode is Mode.CALIBRATE or any(0 in row for row in (rounds or {}).values())- )+ from eagle_harness.sglang.chain_rounds import served_chain, served_tree+ from eagle_harness.sglang.plain_decode import native_decode_available++ topk = get_spec().speculative_eagle_topk+ plain = (+ topk == 1+ and (self.config.mode is Mode.CALIBRATE or any(0 in row for row in (rounds or {}).values()))+ ) or ((served_chain(self.config, topk) or served_tree(self.config, topk)) and native_decode_available())self.target_verify_graphs = VerifyWidthGraphs(self.config.intensity, max_width, TreeTailMask(info.device), plain_decode=plain)diff --git a/eagle_harness/sglang/chain_rounds.py b/eagle_harness/sglang/chain_rounds.pyindex 662aa4b..f3e5359 100644--- a/eagle_harness/sglang/chain_rounds.py+++ b/eagle_harness/sglang/chain_rounds.py@@ -25,6 +25,8 @@ serve-mode chain server with the same ``data_path`` reads.from __future__ import annotationsimport json+import logging+import mathimport osimport statisticsimport time@@ -34,14 +36,41 @@ from typing import TYPE_CHECKING, Dict, List, Sequence, Tupleimport torch-from eagle_harness.core.chain_width import save_accept_profile, save_round_times, width_at+from eagle_harness.core.chain_width import AcceptProfile, save_accept_profile, save_round_times, width_at+from eagle_harness.core.config import HarnessConfig, Mode+from eagle_harness.core.limits import ComputeIntensity, cost_batch_sizes, nearest_cost_batch_sizeif TYPE_CHECKING:from sglang.srt.managers.schedule_batch import ScheduleBatch+logger = logging.getLogger(__name__)+BLOCK_ROUNDS = 8FLUSH_ROUNDS = 256MIN_ROUNDS = 8+# Whole-interval periods of one width before two widths are compared. Under sustained load the periods that hold+# a prefill are several times the others (coefficient of variation ~0.5), so the mean of 32 is within ~9%: enough+# to separate native decode from speculation where the choice matters, short enough to settle in a first pass.+EXPLORE_ROUNDS = 32+# A period in which more than 1/BURST_SHARE of the batch joined holds a burst of arrivals (the load changed), not the+# refills of sustained load (tokens per round / output length: under 1/8 for outputs over 64 tokens at width <= 8).+# Its prefills are the same work under every width and, landing in one width's few periods, would decide the choice.+BURST_SHARE = 8+# Speculation serves a batch key only when it beats native decode by this many standard errors of the difference of+# the two measured rates: a wrong native choice costs only the speculative gain, a wrong speculative one runs slower+# than plain decoding.+NATIVE_MARGIN_SE = 2.0+# Speculation that beats native decode's estimate (the batch's one-token verify in the live costs) by this factor,+# beyond NATIVE_MARGIN_SE standard errors, is not measured against native decode. Requests decoded natively cannot+# resume drafting and a batch holding one decodes natively, so measuring native decode keeps the batch native until it+# empties: under sustained load, for good. The estimate is a model of native decode, not a measurement (served native+# rounds ran 1.1x it at batch key 64 on B300), hence the wide factor.+NATIVE_SKIP_RATIO = 1.5+# A period longer than STALL_FACTOR times its cell's median holds one-time work, not the served load: a kernel compile+# or graph capture on a cold cache (1-3 s on B300 TP4, against 5-15 ms rounds). The refill prefills of sustained load+# stay within a few medians. One such period in a 32-period cell moves its rate several-fold (native decode at batch+# key 32 read 0.28 tokens/ms against 5.6 served), so it is counted, not priced.+STALL_FACTOR = 10RAW_FILE = os.path.join("raw", "sglang_rounds.json")@@ -157,12 +186,439 @@ class ChainCalibration:save_accept_profile(self._data_path, drafts, per_position)+def served_chain(config: HarnessConfig, topk: int) -> bool:+ """Whether a server chooses its chain widths on its served rounds (:class:`ServedChainPolicy`): a ``serve``-mode+ chain without ``data_path``."""+ return (+ config.mode is Mode.SERVE and not config.data_path and config.intensity is ComputeIntensity.CHAIN and topk == 1+ )+++def served_tree(config: HarnessConfig, topk: int) -> bool:+ """Whether a tree chooses between speculation and native decode on its served rounds (:class:`ServedTreePolicy`):+ every ``serve``-mode tree (top-k > 1), the default configuration; the engine must also have the native route+ (:func:`~eagle_harness.sglang.plain_decode.native_decode_available`)."""+ return config.mode is Mode.SERVE and topk > 1+++def _ratio_rate(samples: Sequence[Tuple[float, float]]) -> Tuple[float, float]:+ """Tokens per microsecond over periods ``(us, tokens)`` (the ratio of the sums) and its variance (ratio+ estimator; infinite below two periods)."""+ n = len(samples)+ us = sum(u for u, _ in samples)+ if n == 0 or us <= 0:+ return 0.0, math.inf+ rate = sum(t for _, t in samples) / us+ if n < 2:+ return rate, math.inf+ mean_us = us / n+ residual = sum((t - rate * u) ** 2 for u, t in samples)+ return rate, residual / (n * (n - 1) * mean_us**2)+++def _settled(samples: Sequence[tuple]) -> List[tuple]:+ """The periods ``(us, ...)`` of a cell without its stalls (:data:`STALL_FACTOR`)."""+ if len(samples) < 2:+ return list(samples)+ limit = STALL_FACTOR * statistics.median(sample[0] for sample in samples)+ return [sample for sample in samples if sample[0] <= limit]+++class ServedChainPolicy:+ """The chain width per batch size of a ``serve``-mode chain without ``data_path``, chosen on its served rounds.++ Price: tokens per wall-clock time over whole served intervals. A round's period runs from its start to the next+ decode round's start, so the prefill steps between them are in it, and so are the costs a chain pays around them+ (a prelaunched draft dropped at a prefill or a batch change, the next round drafting from scratch). Common prefill+ work costs every action the same per committed token, so it does not move the choice; dropping those periods+ (as :class:`ChainCalibration` does) prices chains on the rounds where they are cheapest. A period is cut where+ the batch turned over completely (the server may have idled in between), and one that holds a burst of arrivals+ (:data:`BURST_SHARE`), a stall+ (:data:`STALL_FACTOR`) or the first round after a switch of width (it also finishes the previous width's work)+ counts toward the comparison without being priced.++ Search, per batch key (the nearest calibration batch size, :data:`~eagle_harness.core.limits.COST_BATCH_SIZES`):+ width 1 against the next wider captured width, then wider while the wider one wins (tokens per time is unimodal in+ the width: acceptance grows concavely, the verify cost convexly), never wider than a smaller batch key chose.+ Each comparison interleaves the two widths in+ blocks of :data:`BLOCK_ROUNDS` until each has :data:`EXPLORE_ROUNDS` periods, so both see the same load. With+ ``native_decode``, the search starts at width 2 (width 1 commits the same token as native decode with more work)+ and native decode (width 0) is measured last: its requests cannot resume drafting, so it runs right after a fresh+ block of the best width (both again after a complete turnover, so they see the same load), and the batch stays+ native until those requests finish (and the ones that join it meanwhile). So with ``native_us`` (native decode's+ microseconds per round at a batch size, from the live costs), speculation that beats that estimate by+ :data:`NATIVE_SKIP_RATIO` is served without measuring native decode. Speculation loses more as the+ batch grows (the verify becomes compute-bound), so where native decode wins every larger batch key takes it+ unmeasured, and where speculation wins no smaller batch key measures native decode.++ Decisions are made on TP rank 0 and broadcast (``sync``) at the round that completes a comparison, the same+ round on every rank. Rank 0 keeps them in ``cache_path`` (keyed like the live costs), so a restart on the same+ setup serves them from its first round. Without that file, a first start serves the ``packaged`` choices of its+ setup (:func:`~eagle_harness.core.cost_profiles.packaged_served_choices`) and searches only the other batch keys.+ shortcut: a decision is not revisited when the workload's acceptance changes; re-explore (delete the file) after+ changing the served workload class.+ """++ def __init__(+ self,+ widths: Sequence[int],+ device: torch.device,+ *,+ max_batch_size: int,+ native_decode: bool,+ writer: bool,+ sync: Callable[[object], object],+ cache_path: str | None = None,+ key: dict | None = None,+ packaged: Dict[int, int] | None = None,+ native_us: Callable[[int], float | None] | None = None,+ ):+ self.widths = sorted(widths)+ self._native = native_decode+ self._native_us = native_us+ self._native_checked: set = set() # batch keys whose speculation was checked against native decode's estimate+ self._writer = writer+ self._sync = sync+ self._path = cache_path+ self._identity = {"key": key, "widths": self.widths, "native_decode": native_decode}+ self._grid = cost_batch_sizes(max_batch_size) or [max_batch_size]+ self._key = [0] + [nearest_cost_batch_size(bs, self._grid) for bs in range(1, max_batch_size + 1)]+ self.decided: Dict[int, int] = {} # batch key -> served width (0: native decode)+ self._pair: Dict[int, Tuple[int, int]] = {} # batch key -> (best width so far, challenger)+ self._started: Dict[int, int] = defaultdict(int) # rounds of the comparison in progress, per batch key+ self._cells: Dict[Tuple[int, int], List[float]] = {} # (batch key, width) -> [period us, requests, periods]+ # (batch key, width) -> the priced periods: (us, requests, tokens committed: a device scalar, trees only)+ self._samples: Dict[Tuple[int, int], List[Tuple[float, int, object]]] = defaultdict(list)+ # key, width, start ns, bs, request ids, whether the round is priced (not a live-cost seed round), whether it+ # switched action (its period also holds the previous action's tail: a dropped prelaunched draft, the switch)+ self._previous: Tuple[int, int, int, int, frozenset, bool, bool] | None = None+ self._last_width: int | None = None # the width of the last round seen+ self._accept = torch.zeros(self.widths[-1] + 1, self.widths[-1] + 1, dtype=torch.int64, device=device)+ if not self._load() and packaged:+ self._start_from(packaged, "the packaged choices of this setup")++ # ---- the served width+ def width(self, bs: int) -> int:+ """The width of a round of ``bs`` requests (0: native decode)."""+ key = self._key[min(bs, len(self._key) - 1)]+ if key in self.decided:+ return self.decided[key]+ # With native decode, width 1 (the same tokens with more work) never wins: the search starts above it.+ first = self.widths[1] if self._native and len(self.widths) > 2 else self.widths[0]+ best, challenger = self._pair.setdefault(key, (first, self._wider(first)))+ if challenger == 0: # the best width's periods, then native decode's right after them+ return 0 if self._cells.setdefault((key, best), [0.0, 0, 0])[2] >= EXPLORE_ROUNDS else best+ return challenger if (self._started[key] // BLOCK_ROUNDS) % 2 else best++ def _wider(self, width: int) -> int:+ """The next wider width; past the widest, native decode (0)."""+ return next((w for w in self.widths if w > width), 0)++ def exploring(self, bs: int) -> bool:+ return self._key[min(bs, len(self._key) - 1)] not in self.decided++ # ---- recording+ def round_start(self, bs: int, width: int, requests: Sequence, settled: bool = True) -> None:+ """A decode round of ``bs`` requests at ``width`` starts: the previous round's whole interval ends.+ ``settled``: False for a round the drafter forces to seed its live costs, whose period is not counted."""+ now = time.perf_counter_ns()+ key = self._key[min(bs, len(self._key) - 1)]+ previous, self._previous = self._previous, None+ switched, self._last_width = self._last_width not in (None, width), width+ if not self._pair and key in self.decided:+ return+ ids = frozenset(req.rid for req in requests)+ self._previous = (key, width, now, bs, ids, settled, switched)+ if previous is not None and previous[4].isdisjoint(ids):+ # The batch turned over (the server may have idled): native decode and the block of the best width it is+ # compared with are measured again together, under the load that follows.+ for k, pair in self._pair.items():+ if pair[1] == 0:+ self._clear(k, pair[0])+ self._clear(k, 0)+ elif previous is not None and previous[5] and previous[1] in self._pair.get(previous[0], ()):+ cell = self._cells.setdefault(previous[:2], [0.0, 0, 0])+ cell[2] += 1+ # A burst of arrivals, or the first round after a switch of action, is counted, not priced.+ if len(ids - previous[4]) * BURST_SHARE <= previous[3] and not previous[6]:+ us = (now - previous[2]) / 1000.0+ cell[0] += us+ cell[1] += previous[3]+ self._samples[previous[:2]].append((us, previous[3], self._period_tokens(previous[1])))+ if key in self.decided:+ return+ self._started[key] += 1+ pair = self._pair.get(key)+ if pair is None:+ return+ periods = [self._cells.setdefault((key, w), [0.0, 0, 0])[2] for w in pair]+ if all(n >= EXPLORE_ROUNDS for n in periods):+ self._compare(key, pair)+ elif (+ self._native_us is not None+ and pair[1] == 0+ and periods[0] >= EXPLORE_ROUNDS+ and not periods[1]+ and key not in self._native_checked+ ):+ self._check_native(key, pair[0])++ def accepted(self, width: int, accept_lens: torch.Tensor) -> None:+ """Count a round's accept lengths (tokens committed per request) under its width (rank 0 decides)."""+ if width >= 1 and self._writer: # index_add_, not bincount: bincount sizes its output from a device read+ lens = accept_lens.to(torch.int64)+ self._accept[width].index_add_(0, lens, torch.ones_like(lens))++ def _period_tokens(self, width: int) -> object:+ """The tokens a period's round committed, when the policy measures them (trees); None: priced from the+ acceptance profile."""+ return None++ def _clear(self, key: int, width: int) -> None:+ self._cells[(key, width)] = [0.0, 0, 0]+ self._samples.pop((key, width), None)+ if width == 0:+ self._native_checked.discard(key)++ # ---- decisions+ def profile(self) -> AcceptProfile | None:+ """Acceptance by drafted position from every served width: a round of width ``w`` drafted positions+ ``1 .. w - 1``, and accepted position ``d`` when its accept length exceeds ``d``."""+ hist = self._accept.tolist()+ rates = []+ for d in range(1, len(hist) - 1):+ drafted = sum(sum(row) for row in hist[d + 1 :])+ if not drafted:+ break+ rates.append(sum(sum(row[d + 1 :]) for row in hist[d + 1 :]) / drafted)+ return AcceptProfile(tuple(rates)) if rates else None++ def _rates(self, key: int, pair: Tuple[int, int]) -> Dict[int, Tuple[float, float]]:+ """Committed tokens per microsecond of the pair's widths and the variance of each (native decode and width+ 1 commit one token per request; wider chains the profile's expected tokens)."""+ profile = self.profile()+ covered = len(profile.rates) + 1 if profile else 1+ rates = {}+ for width in pair:+ if width <= covered and self._cells[(key, width)][0] > 0:+ tokens = profile.expected_tokens(width) if width > 1 else 1.0+ rates[width] = _ratio_rate([(us, tokens * n) for us, n, _ in _settled(self._samples[(key, width)])])+ return rates++ @staticmethod+ def _better(rates: Dict[int, Tuple[float, float]], best: int, challenger: int) -> int:+ """The winner of a comparison. Against native decode, speculation must win by :data:`NATIVE_MARGIN_SE`+ standard errors."""+ (best_rate, best_var), (rate, var) = rates.get(best, (0.0, math.inf)), rates.get(challenger, (0.0, math.inf))+ if challenger != 0:+ return challenger if rate > best_rate else best+ margin = NATIVE_MARGIN_SE * math.sqrt(best_var + var)+ return best if best_rate - rate > margin else 0++ def _compare(self, key: int, pair: Tuple[int, int]) -> None:+ """Both widths of the pair have their periods: rank 0 decides the next step, every rank applies it."""+ step = None+ if self._writer:+ rates = self._rates(key, pair)+ best, challenger = pair+ best = self._better(rates, best, challenger)+ # A larger batch never verifies wider than a smaller one chose, and native decode where speculation won+ # at a larger batch would only lose (and stick).+ widest = min((w for k, w in self.decided.items() if k < key and w >= 1), default=self.widths[-1])+ spec_won_above = any(w >= 1 and k >= key for k, w in self.decided.items())+ if best != 0 and challenger != 0 and best == challenger and best < widest:+ step = ("compare", best, self._wider(best))+ elif best != 0 and challenger != 0 and self._native and not spec_won_above:+ step = ("compare", best, 0)+ else:+ step = ("decide", best, None)+ logger.info(+ "Served chain rates at batch key %d (tokens per ms): %s -> %s",+ key,+ {w: (round(1000 * r, 3), round(1000 * math.sqrt(v), 3)) for w, (r, v) in sorted(rates.items())},+ step,+ )+ kind, width, challenger = self._sync(step)+ self._started[key] = 0+ for w in pair:+ self._clear(key, w)+ if kind == "compare":+ self._pair[key] = (width, challenger)+ return+ self._decide(key, width)++ def _decide(self, key: int, width: int) -> None:+ self._pair.pop(key, None)+ self.decided[key] = width+ self._infer()+ self._save()++ def _native_estimate(self, key: int, width: int) -> float | None:+ """Native decode's tokens per microsecond over the batches of ``width``'s periods at ``key``, priced by+ ``native_us``; None where it has no price."""+ if self._native_us is None:+ return None+ us = requests = 0.0+ for _, n, _ in _settled(self._samples[(key, width)]):+ price = self._native_us(n)+ if price is None or price <= 0:+ return None+ us += price+ requests += n+ return requests / us if us else None++ def _check_native(self, key: int, best: int) -> None:+ """``best`` has its periods before native decode's: rank 0 decides whether it beats native decode's estimate+ so clearly (:data:`NATIVE_SKIP_RATIO`) that native decode is not measured, and every rank applies it."""+ self._native_checked.add(key)+ step = None+ if self._writer:+ rate, var = self._rates(key, (best,)).get(best, (0.0, math.inf))+ native = self._native_estimate(key, best)+ won = native is not None and rate - NATIVE_MARGIN_SE * math.sqrt(var) > NATIVE_SKIP_RATIO * native+ step = ("decide", best, None) if won else ("compare", best, 0)+ logger.info(+ "Served chain rate at batch key %d (tokens per ms): %s, native decode estimate %s -> %s",+ key,+ (round(1000 * rate, 3), round(1000 * math.sqrt(var), 3)),+ None if native is None else round(1000 * native, 3),+ step,+ )+ kind, width, _ = self._sync(step)+ if kind == "decide":+ self._started[key] = 0+ self._clear(key, width)+ self._clear(key, 0)+ self._decide(key, width)++ def _infer(self) -> None:+ """Native decode at a batch key serves every larger one unmeasured."""+ for key in self._grid:+ if key not in self.decided and any(w == 0 for k, w in self.decided.items() if k < key):+ self.decided[key] = 0+ self._pair.pop(key, None)++ # ---- persistence+ def _start_from(self, decided: Dict[int, int], source: str) -> bool:+ """Serve ``decided`` (batch key -> width) if every width is an action of this server."""+ decided = {k: w for k, w in decided.items() if k in self._grid}+ if any(w not in (0, *self.widths) or (w == 0 and not self._native) for w in decided.values()):+ logger.info("Served chain widths %s from %s are not actions of this server; exploring.", decided, source)+ return False+ self.decided = dict(decided)+ self._infer()+ logger.info("Served chain widths per batch key from %s: %s", source, self.decided)+ return True++ def _load(self) -> bool:+ if self._path is None:+ return False+ try:+ with open(self._path, encoding="utf-8") as f:+ data = json.load(f)+ if {k: data.get(k) for k in self._identity} != self._identity:+ logger.info("Served chain widths in %s are for another setup; exploring again.", self._path)+ return False+ decided = {int(k): int(w) for k, w in data["chain"].items()}+ except (OSError, ValueError, TypeError, KeyError, AttributeError) as e:+ logger.info("Served chain widths %s unavailable or invalid (%s); exploring.", self._path, e)+ return False+ return self._start_from(decided, self._path)++ def _save(self) -> None:+ if not (self._writer and self._path):+ return+ try:+ os.makedirs(os.path.dirname(self._path) or ".", exist_ok=True)+ with open(f"{self._path}.tmp", "w", encoding="utf-8") as f:+ json.dump({**self._identity, "chain": {str(k): w for k, w in sorted(self.decided.items())}}, f)+ os.replace(f"{self._path}.tmp", self._path)+ except OSError as e:+ logger.warning("Could not write the served chain widths %s: %s", self._path, e)+++class ServedTreePolicy(ServedChainPolicy):+ """Speculation or native decode per batch size for a ``serve``-mode tree, chosen on its served rounds.++ The cost model sizes every speculative round (its verify width included); this policy decides only whether a batch+ key speculates at all. Speculation is the one action ``spec_width`` (the tree's maximum verify width, whatever+ width each round verifies), compared with native decode like the best chain width of :class:`ServedChainPolicy`:+ the same whole served intervals, the same order (native decode last, sticky), the same persistence. Its price is+ the tokens its rounds committed (the sum of their accept lengths, kept on the GPU until the comparison), since a+ tree's acceptance has no per-position profile. A round that seeds the live costs (``settled=False``) forces a+ width the cost model would not choose and is not counted.++ Speculation gains less as the batch grows, so native decode at a batch key serves every larger key and+ speculation at a batch key every smaller one, both unmeasured. ``packaged`` choices map a batch key to 1+ (speculate) or 0 (native decode).+ """++ def __init__(+ self,+ spec_width: int,+ device: torch.device,+ *,+ max_batch_size: int,+ writer: bool,+ sync: Callable[[object], object],+ cache_path: str | None = None,+ key: dict | None = None,+ packaged: Dict[int, int] | None = None,+ native_us: Callable[[int], float | None] | None = None,+ ):+ self._tokens: torch.Tensor | None = None # the committed tokens of the last speculative round+ super().__init__(+ [spec_width],+ device,+ max_batch_size=max_batch_size,+ native_decode=True,+ writer=writer,+ sync=sync,+ cache_path=cache_path,+ key=key,+ packaged=None if packaged is None else {k: spec_width if c else 0 for k, c in packaged.items()},+ native_us=native_us,+ )++ def round_start(self, bs: int, width: int, requests: Sequence, settled: bool = True) -> None:+ super().round_start(bs, self.widths[0] if width else 0, requests, settled)++ def accepted(self, width: int, accept_lens: torch.Tensor) -> None:+ if width >= 1 and self._writer:+ self._tokens = accept_lens.sum()++ def _period_tokens(self, width: int) -> object:+ return self._tokens if width and self._writer else None++ def _rates(self, key: int, pair: Tuple[int, int]) -> Dict[int, Tuple[float, float]]:+ rates = {}+ for width in pair:+ samples = _settled(self._samples[(key, width)])+ if not samples:+ continue+ if width:+ tokens = torch.stack([t for _, _, t in samples]).tolist() # the comparison's one device read+ rates[width] = _ratio_rate([(us, t) for (us, _, _), t in zip(samples, tokens)])+ else:+ rates[width] = _ratio_rate([(us, n) for us, n, _ in samples])+ return rates++ def _infer(self) -> None:+ super()._infer()+ spec = self.widths[0]+ for key in self._grid:+ if key not in self.decided and any(w == spec for k, w in self.decided.items() if k > key):+ self.decided[key] = spec+ self._pair.pop(key, None)++class ChainRounds:"""The served/calibrated chain width and optional recording of completed rounds.Drafting asks for ``draft_width`` once when it queues a round. Native decodereads ``plain_width`` without advancing that plan, including a prelaunchedchoice of OFF. Context is read only for calibration, without a device sync.+ A ``policy`` (:class:`ServedChainPolicy`) chooses the width of each batch on the served rounds instead."""def __init__(@@ -170,26 +626,38 @@ class ChainRounds:schedule: Sequence[tuple[int, int, int]] | None,calibration: ChainCalibration | None,average_context: Callable[[ScheduleBatch], int],+ policy: ServedChainPolicy | ServedTreePolicy | None = None,):self._schedule = scheduleself._calibration = calibrationself._average_context = average_context+ self._policy = policydef draft_width(self, bs: int) -> int:- width = self._calibration.width(bs) if self._calibration is not None else width_at(self._schedule, bs)+ if self._policy is not None:+ width = self._policy.width(bs)+ else:+ width = self._calibration.width(bs) if self._calibration is not None else width_at(self._schedule, bs)return max(1, width)def plain_width(self, bs: int) -> int:+ if self._policy is not None:+ return self._policy.width(bs) # this batch's own choice: a prelaunch planned the previous batchreturn self._calibration.last_width if self._calibration is not None else width_at(self._schedule, bs)def interrupt(self) -> None:if self._calibration is not None:self._calibration.interrupt()- def round_start(self, bs: int, width: int, batch: ScheduleBatch) -> None:+ def round_start(self, bs: int, width: int, batch: ScheduleBatch, settled: bool = True) -> None:+ """A decode round starts; ``settled``: False for a round forced to seed the live costs."""if self._calibration is not None:self._calibration.round_start(bs, width, self._average_context(batch))+ if self._policy is not None:+ self._policy.round_start(bs, width, batch.reqs, settled)def accepted(self, width: int, accept_lens: torch.Tensor) -> None:if self._calibration is not None:self._calibration.accepted(width, accept_lens)+ if self._policy is not None and self._policy.exploring(len(accept_lens)):+ self._policy.accepted(width, accept_lens)diff --git a/eagle_harness/sglang/drafter.py b/eagle_harness/sglang/drafter.pyindex e177bbb..0531a6a 100644--- a/eagle_harness/sglang/drafter.py+++ b/eagle_harness/sglang/drafter.py@@ -267,6 +267,11 @@ class DynamicDrafter:self._sampled_depth = tree.shape.max_depth# ----------------------------------------------------------------- policy+ @property+ def seeding(self) -> bool:+ """Whether the last drafted round was forced to seed the live costs of its batch bucket."""+ return self._seeding is not None+def bucket(self, bs: int) -> int:"""The cost bucket (captured batch size) a batch of ``bs`` requests runs at."""return self._bucket(bs)diff --git a/eagle_harness/sglang/plain_decode.py b/eagle_harness/sglang/plain_decode.pyindex d0a1b34..a4ca5a1 100644--- a/eagle_harness/sglang/plain_decode.py+++ b/eagle_harness/sglang/plain_decode.py@@ -23,6 +23,24 @@ if TYPE_CHECKING:from eagle_harness.sglang.chain_rounds import ChainRounds+def native_decode_available() -> bool:+ """Whether native decode can serve requests that stop drafting: the engine captures native DECODE graphs among+ the speculative graph variants, and neither Mamba prefix-cache tracking nor DP attention needs the draft's state."""+ from sglang.srt.runtime_context import get_exec, get_parallel++ from eagle_harness.sglang.verify_graphs import native_decode_graphs++ return native_decode_graphs() and not get_exec().mamba.enable_mamba_extra_buffer and get_parallel().dp_size == 1+++def native_decode_blocked() -> bool:+ """Whether the engine has native DECODE graphs but this server's settings (Mamba prefix-cache tracking, DP+ attention) keep requests from switching to them."""+ from eagle_harness.sglang.verify_graphs import native_decode_graphs++ return native_decode_graphs() and not native_decode_available()++class NoSpecRequests:"""Keep a request off until it finishes; skipped draft KV is never reused."""diff --git a/eagle_harness/sglang/verify_graphs.py b/eagle_harness/sglang/verify_graphs.pyindex 58f9924..ad64d95 100644--- a/eagle_harness/sglang/verify_graphs.py+++ b/eagle_harness/sglang/verify_graphs.py@@ -28,6 +28,11 @@ from eagle_harness.sglang.verify_input import HarnessVerifyInputfrom eagle_harness.sglang.verify_plan import TreeTailMask+def native_decode_graphs() -> bool:+ """Whether the engine captures a speculative graph variant in its own forward mode (native DECODE graphs)."""+ return hasattr(DecodeGraphVariant, "forward_mode")++def verify_widths(batch_size: int, intensity: ComputeIntensity, max_width: int) -> list:"""Verify widths with a graph at ``batch_size``, widest first (``max_width`` and 1 always included)."""widths = {w for w in verify_buckets(batch_size, intensity) if w <= max_width}@@ -60,7 +65,7 @@ class VerifyWidthGraphs(DecodeGraphVariants):self.max_width = max_widthself.tail_mask = tail_maskself.greedy = greedy- if plain_decode and not hasattr(DecodeGraphVariant, "forward_mode"):+ if plain_decode and not native_decode_graphs():raise RuntimeError("Native no-spec fallback requires SGLang's per-variant forward-mode graph contract.")self._plain = DecodeGraphVariant("decode", 1, ForwardMode.DECODE) if plain_decode else Noneself._variants: Dict[int, DecodeGraphVariant] = {}diff --git a/eagle_harness/sglang/verify_step.py b/eagle_harness/sglang/verify_step.pyindex 2ea0b06..eb68a0b 100644--- a/eagle_harness/sglang/verify_step.py+++ b/eagle_harness/sglang/verify_step.py@@ -205,3 +205,19 @@ def fused_accept_step()return accept_step+++# sglang's tiled KV move compiles one Triton program per next_power_of_2(moved slots), up to the pool's chunk size+# (128 or 256 slots), and per slot count specialization (1, a multiple of 16, any other); it compiles only the+# chunk-sized one at startup. A tree round moves the KV of its accepted tokens, so the smaller programs compiled while+# serving: ~2 s each on B300, in the first rounds of each new load.+KV_MOVE_MAX_SLOTS = 256+++def warm_up_kv_moves(kv_cache: Any, device: torch.device) -> None:+ """Compile, before serving, the KV move of every slot count a tree round can move: slot 0 (the pool's padding+ slot) onto itself, for each power of two up to :data:`KV_MOVE_MAX_SLOTS` and the count below it."""+ counts = sorted({1} | {n - d for n in (2**i for i in range(1, KV_MOVE_MAX_SLOTS.bit_length())) for d in (0, 1)})+ for count in counts:+ slots = torch.zeros(count, dtype=torch.int64, device=device)+ kv_cache.move_kv_cache(slots, slots)diff --git a/eagle_harness/sglang/worker.py b/eagle_harness/sglang/worker.pyindex f422c93..8b9163d 100644--- a/eagle_harness/sglang/worker.py+++ b/eagle_harness/sglang/worker.py@@ -23,8 +23,9 @@ One decode round:resumes them if the batch holds the same requests, else drafts from scratch.A chain (top-k 1, ``chain_intensity``) with served round times under ``data_path`` verifies a fixed width per-batch size instead (:mod:`eagle_harness.sglang.chain_rounds`), and ``"mode": "calibrate"`` forces and times every-chain width while it serves.+batch size instead (:mod:`eagle_harness.sglang.chain_rounds`), a chain without ``data_path`` chooses that width on+its served rounds (:class:`~eagle_harness.sglang.chain_rounds.ServedChainPolicy`), and ``"mode": "calibrate"`` forces+and times every chain width while it serves.Prefill (target extend with aux hidden capture, then the draft extend) issglang's EAGLE V2 path unchanged. With overlap scheduling, the worker publishes@@ -39,7 +40,7 @@ import contextlibfrom functools import partialimport loggingimport os-from typing import TYPE_CHECKING, Optional, Union+from typing import TYPE_CHECKING, Dict, Optional, Unionimport torch@@ -60,19 +61,32 @@ from eagle_harness.core.chain_width import chain_widthsfrom eagle_harness.kernels.accept_tail import warm_up as warm_up_accept_tailfrom eagle_harness.core.config import HarnessConfig, Mode, build_cost_model, parse_harness_configfrom eagle_harness.core.cost_model import CostModel, FixedShapePolicy+from eagle_harness.core.cost_profiles import packaged_served_choicesfrom eagle_harness.core.limits import ComputeIntensityfrom eagle_harness.core.draft_tree import DraftTree, TreeShapefrom eagle_harness.core.models import target_is_moe-from eagle_harness.sglang.chain_rounds import ChainCalibration, ChainRounds+from eagle_harness.sglang.chain_rounds import (+ ChainCalibration,+ ChainRounds,+ ServedChainPolicy,+ ServedTreePolicy,+ served_chain,+ served_tree,+)from eagle_harness.core.live_costs import CostKey, LiveCosts, config_digestfrom eagle_harness.core.gemm_solutions import load_gemm_solutionsfrom eagle_harness.sglang.draft_extend import DraftExtendfrom eagle_harness.sglang.draft_steps import DraftStepsfrom eagle_harness.sglang.drafter import DynamicDrafter, PrelaunchResult, spin_waitfrom eagle_harness.sglang.l2_warm import MlpL2WarmAhead-from eagle_harness.sglang.plain_decode import NativeDecode+from eagle_harness.sglang.plain_decode import NativeDecode, native_decode_available, native_decode_blockedfrom eagle_harness.sglang.row_attention import DraftRowAttention-from eagle_harness.sglang.verify_step import AcceptTailGraphs, fused_accept_step, fused_accept_supported+from eagle_harness.sglang.verify_step import (+ AcceptTailGraphs,+ fused_accept_step,+ fused_accept_supported,+ warm_up_kv_moves,+)from eagle_harness.sglang.verify_graphs import verify_widthsfrom eagle_harness.sglang.hooks import parameter_namesfrom eagle_harness.sglang.verify_input import HarnessVerifyInput@@ -100,6 +114,20 @@ def cost_key(runner) -> CostKey:)+def packaged_choices(kind: str, key: CostKey, runner) -> Optional[Dict[int, int]]:+ """The packaged served ``kind`` ("chain" or "tree") choices of this setup, if it was measured+ (:func:`~eagle_harness.core.cost_profiles.packaged_served_choices`)."""+ return packaged_served_choices(+ kind,+ device_name=key.gpu,+ model_path=get_model().model_path,+ draft_model_path=get_spec().speculative_draft_model_path,+ tp_size=key.gpu_count,+ dtype=str(runner.dtype),+ quantization=get_model().quantization,+ )++def _sampled_tree(worker):"""The draft's recorded-q sampled tree, or None: engines without sglang's verify sampler hook build none, and aworker built without sglang's __init__ (or serving natively, OFF) may have no draft steps to ask."""@@ -171,12 +199,15 @@ class HarnessDraftWorker(EagleDraftWorker):def _warm_up_kernels(self) -> None:"""Compile, before serving, the kernels whose first use would otherwise stall a served round: the fused- accept step (batch-size-1 trees) and the softmax of sglang's draft extend after a prefill (a JIT module).+ accept step (batch-size-1 trees), the KV move of a tree's accepted tokens at every slot count, and the+ softmax of sglang's draft extend after a prefill (a JIT module). A stall in a served round also skews the+ served tree policy's comparison of speculation with native decode."""if torch.device(self.device).type != "cuda":returnif self.topk > 1:warm_up_accept_tail(self.extend.hidden_rows(2), self.target_worker.model_runner.dtype)+ warm_up_kv_moves(self.target_worker.model_runner.token_to_kv_pool_allocator.get_kvcache(), self.device)softmax = getattr(spec_utils, "sampling_softmax", None)if softmax is not None:# The engine dispatches by vocabulary size; a tiny tensor can warm a different backend.@@ -221,17 +252,28 @@ class EagleHarnessWorker(EAGLEWorkerV2):verify_num=spec.speculative_num_draft_tokens,)self._chain_schedule = self._fixed_chain_widths(spec)+ # A chain without served round times picks its width per batch size on the served rounds.+ self._served_chain = served_chain(self.config, spec.speculative_eagle_topk)+ # A tree speculates or decodes natively per batch size, chosen on its served rounds, where the engine can.+ self._served_tree = served_tree(self.config, spec.speculative_eagle_topk) and native_decode_available()+ if served_tree(self.config, spec.speculative_eagle_topk) and native_decode_blocked():+ logger.warning(+ "EAGLE_HARNESS: this server cannot switch to native decode where speculation loses (Mamba prefix-cache "+ "tracking or DP attention), so it speculates at every load; --disable-radix-cache enables the switch."+ )native_decode = self.config.mode is Mode.CALIBRATE or any(width == 0 for _, _, width in (self._chain_schedule or ()))if native_decode and (get_exec().mamba.enable_mamba_extra_buffer or get_parallel().dp_size != 1):raise ValueError("Native no-spec fallback requires DP1 and disabled Mamba prefix-cache tracking.")- self._native_decode_enabled = native_decode+ self._native_decode_enabled = (+ native_decode or self._served_tree or (self._served_chain and native_decode_available())+ )self.fixed = FixedShapePolicy(self.config.intensity, **shape)# Built before sglang's __init__, which creates the draft worker through create_draft_worker.self.cost_model: Optional[CostModel] = Noneself.live: Optional[LiveCosts] = None- if self.config.mode.uses_cost_model and self._chain_schedule is None:+ if self.config.mode.uses_cost_model and self._chain_schedule is None and not self._served_chain:runner = target_worker.model_runnerkey = cost_key(runner)if self.config.data_path:@@ -322,9 +364,43 @@ class EagleHarnessWorker(EAGLEWorkerV2):writer=writer,native_decode=True,)- self._chain_rounds = ChainRounds(self._chain_schedule, calibration, DynamicDrafter.average_context)capture_bs, _ = get_batch_sizes_to_capture(self.target_worker.model_runner)++ def bucket(bs: int) -> int:+ return min((b for b in capture_bs if b >= bs), default=max(capture_bs, default=bs))+tp_group = get_tp_group()+ policy = None+ if self._served_chain:+ runner = self.target_worker.model_runner+ key = cost_key(runner)+ policy = ServedChainPolicy(+ verify_widths(1, intensity, full),+ torch.device(self.device),+ max_batch_size=get_schedule().max_running_requests,+ native_decode=self._native_decode_enabled,+ writer=tp_group.rank_in_group == 0,+ sync=lambda step: tp_group.broadcast_object(step, src=0),+ cache_path=os.path.join(self.config.cost_cache_dir, f"{key.digest}-chain.json"),+ key=key.to_dict(),+ packaged=packaged_choices("chain", key, runner),+ )+ elif self._served_tree:+ runner = self.target_worker.model_runner+ key = cost_key(runner)+ policy = ServedTreePolicy(+ full,+ torch.device(self.device),+ max_batch_size=get_schedule().max_running_requests,+ writer=tp_group.rank_in_group == 0,+ sync=lambda step: tp_group.broadcast_object(step, src=0),+ cache_path=os.path.join(self.config.cost_cache_dir, f"{key.digest}-tree.json"),+ key=key.to_dict(),+ packaged=packaged_choices("tree", key, runner),+ # Native decode priced as the target's one-token verify at the batch's bucket.+ native_us=None if self.live is None else lambda bs: self.live.verify_us(bucket(bs), 1),+ )+ self._chain_rounds = ChainRounds(self._chain_schedule, calibration, DynamicDrafter.average_context, policy)self._drafter = DynamicDrafter(self._draft_worker.steps,self.cost_model,@@ -336,11 +412,13 @@ class EagleHarnessWorker(EAGLEWorkerV2):tree_mask_mode=self._draft_worker.tree_mask_mode,device=self.device,live=self.live,- bucket=lambda bs: min((b for b in capture_bs if b >= bs), default=max(capture_bs, default=bs)),+ bucket=bucket,sync_cells=lambda cells: tp_group.broadcast_object(cells, src=0),save_cells=self._save_live_costs if tp_group.rank_in_group == 0 else None,chain_width=(- self._chain_rounds.draft_width if calibration is not None or self._chain_schedule is not None else None+ self._chain_rounds.draft_width+ if calibration is not None or self._chain_schedule is not None or self._served_chain+ else None),)self._native_decode = NativeDecode(@@ -403,7 +481,7 @@ class EagleHarnessWorker(EAGLEWorkerV2):with self._draft_context(), spec_stage_span("draft"):batch.spec_info = self._drafter.draft(batch)width = self._drafter.last_width- rounds.round_start(len(batch.seq_lens), width, batch)+ rounds.round_start(len(batch.seq_lens), width, batch, settled=not self._drafter.seeding)with spec_stage_span("verify"):started = None if timer is None else timer.start()result, staged = self._verify(batch, width, grammar_barrier)diff --git a/eagle_harness/tokenspeed/models.py b/eagle_harness/tokenspeed/models.pyindex e18af6f..8c758f2 100644--- a/eagle_harness/tokenspeed/models.py+++ b/eagle_harness/tokenspeed/models.py@@ -25,14 +25,14 @@ import torchfrom tokenspeed.runtime.configs.model_config import configure_mha_attentionfrom tokenspeed.runtime.configs.model_profile import ModelProfilefrom tokenspeed.runtime.distributed.comm_ops import all_gather+from tokenspeed.runtime.layers.linear import ReplicatedLinearfrom tokenspeed.runtime.layers.logits_processor import LogitsProcessorOutputfrom tokenspeed.runtime.layers.vocab_parallel_embedding import ParallelLMHeadfrom tokenspeed.runtime.models.llama_eagle3 import Eagle3DecoderLayer, LlamaForCausalLMEagle3-from torch import nnfrom eagle_harness.kernels.candidate_ops import candidate_head as fused_candidate_headfrom eagle_harness.models.eagle4_config import check_ported-from eagle_harness.models.low_rank_head import low_rank_scores+from eagle_harness.models.low_rank_head import shard_scores@dataclass@@ -72,7 +72,13 @@ class LlamaForCausalLMEagle4(LlamaForCausalLMEagle3):tp_group=lm_head.tp_group,prefix=f"{prefix}.efficient_lm_head" if prefix else "efficient_lm_head",)- self.efficient_lm_head_down_proj = nn.Linear(config.hidden_size, rank, bias=False)+ # The engine's linear: its GEMM dispatch serves the few draft rows with a low-latency kernel.+ self.efficient_lm_head_down_proj = ReplicatedLinear(+ config.hidden_size,+ rank,+ bias=False,+ prefix=f"{prefix}.efficient_lm_head_down_proj" if prefix else "efficient_lm_head_down_proj",+ )self.tp_size = lm_head.tp_sizeself.tp_group = lm_head.tp_groupself.cuda_candidate_kernels = True # HarnessConfig.draft_model.cuda_candidate_kernels, set by the drafter@@ -119,11 +125,9 @@ class LlamaForCausalLMEagle4(LlamaForCausalLMEagle3):def candidate_head(self, normed: torch.Tensor, k: int):"""Candidate logits, target ids and top-k for recorded-q proposals."""shard = self.lm_head.shard_indices- scores = low_rank_scores(- normed,- self.efficient_lm_head_down_proj.weight,- self.efficient_lm_head.weight,- shard.org_vocab_end_index - shard.org_vocab_start_index,+ low_rank, _ = self.efficient_lm_head_down_proj(normed)+ scores = shard_scores(+ low_rank, self.efficient_lm_head.weight, shard.org_vocab_end_index - shard.org_vocab_start_index)return fused_candidate_head(scores,diff --git a/eagle_harness/vllm/calibrate.py b/eagle_harness/vllm/calibrate.pyindex 71d4f39..a802253 100644--- a/eagle_harness/vllm/calibrate.py+++ b/eagle_harness/vllm/calibrate.py@@ -230,13 +230,14 @@ def engine(model: str, draft: str, tp: int, width: int, chain: bool, max_num_seqelse:method_config = {"mode": "static", "node_top_k": 8, "tree_depth": -(-width // 8) + 1, "max_verify_width": width}backend = "CUSTOM"+ # vLLM's default token budget fits one full batch; a larger explicit one makes its warmup run a request longer+ # than max_model_len, past the end of the block table.args = {"model": model,"tensor_parallel_size": tp,"max_model_len": 4608,"gpu_memory_utilization": 0.85,"max_num_seqs": max_num_seqs,- "max_num_batched_tokens": max(8192, max_num_seqs * width),"speculative_config": {"method": "eagle_harness","model": draft,@@ -294,7 +295,14 @@ def measure("engine_args": engine_args or {},"target_shape": asdict(target_shape),"draft_shape": asdict(draft_shape),- "key": cost_key(model, draft, tp, config.model_config.dtype).to_dict(),+ # Key by the engine's resolved models, as the server does when it loads the tables (speculator.load_model):+ # vLLM may resolve a Hub id to a local snapshot directory, which config_digest names differently.+ "key": cost_key(+ config.model_config.model,+ config.speculative_config.draft_model_config.model,+ tp,+ config.model_config.dtype,+ ).to_dict(),}raw.update(context_step=context_step, context_stop=context_stop, target=target_rows, draft=draft_rows)with open(raw_path, "w", encoding="utf-8") as f:diff --git a/tests/test_cli.py b/tests/test_cli.pyindex cd70dc6..5b669da 100644--- a/tests/test_cli.py+++ b/tests/test_cli.py@@ -243,6 +243,20 @@ def test_vllm_overrides_survive_command_rendering(options, expected, capsys):assert command[index : index + len(expected)] == expected[email protected]("engine_args", [{}, {"max_model_len": 2048}])+def test_vllm_serve_command_token_budget_fits_one_batch(engine_args, capsys):+ # A budget above max_num_seqs * max_model_len makes vLLM's warmup run one request past the block table (#128).+ argv = ["calibrate", "--model", "m", "--draft", "d", "--engine", "vllm", "--dry-run"]+ assert cli.main([*argv, "--engine-args", json.dumps(engine_args)]) == 0+ command = shlex.split(capsys.readouterr().out.splitlines()[-1])++ def flag(name):+ return int(command[command.index(name) + 1]) if name in command else None++ budget = flag("--max-num-batched-tokens")+ assert budget is None or budget <= flag("--max-num-seqs") * flag("--max-model-len")++def test_empty_cost_tables_do_not_publish_serve_command(monkeypatch, tmp_path):monkeypatch.setattr(cli, "_engine", lambda name: "vllm")diff --git a/tests/test_cost_profiles.py b/tests/test_cost_profiles.pyindex c9fd16e..ddc99b5 100644--- a/tests/test_cost_profiles.py+++ b/tests/test_cost_profiles.py@@ -8,8 +8,14 @@ from pathlib import Pathimport pytestfrom eagle_harness.core.config import parse_harness_config-from eagle_harness.core.cost_profiles import MI355X_PROFILE, packaged_cost_profile+from eagle_harness.core.cost_profiles import (+ B300_SERVED_PROFILE,+ MI355X_PROFILE,+ packaged_cost_profile,+ packaged_served_choices,+)from eagle_harness.core.cost_table import load_cost_tables+from eagle_harness.core.limits import ComputeIntensity, cost_batch_sizes, verify_buckets@pytest.fixture@@ -78,3 +84,24 @@ def test_changed_model_config_does_not_select_profile(model_paths, index):def test_profile_does_not_depend_on_model_directory_name(model_paths):assert select(model_paths, dtype="auto") == str(MI355X_PROFILE)+++def test_b300_served_choices_are_selected_for_their_measured_setup_only(tmp_path):+ paths = []+ for name in ("target", "draft"):+ path = tmp_path / name+ path.mkdir()+ (path / "config.json").write_bytes((B300_SERVED_PROFILE / f"{name}_config.json").read_bytes())+ paths.append(str(path))+ args = dict(device_name="NVIDIA B300 SXM6 AC", model_path=paths[0], draft_model_path=paths[1], tp_size=4,+ dtype="torch.bfloat16", quantization=None)+ widths = packaged_served_choices("chain", **args)+ assert widths and set(widths) <= set(cost_batch_sizes(512))+ assert set(widths.values()) <= {0, *verify_buckets(1, ComputeIntensity.CHAIN)}+ trees = packaged_served_choices("tree", **args)+ assert trees and set(trees) <= set(cost_batch_sizes(512)) and set(trees.values()) == {0, 1}+ for override in ({"tp_size": 8}, {"device_name": "NVIDIA H200"}, {"quantization": "fp8"},+ {"draft_model_path": paths[0]}):+ assert packaged_served_choices("chain", **{**args, **override}) is None+ assert packaged_served_choices("tree", **{**args, **override}) is None+ assert packaged_cost_profile(**args) is Nonediff --git a/tests/test_live_costs.py b/tests/test_live_costs.pyindex b7cdad3..c38eb98 100644--- a/tests/test_live_costs.py+++ b/tests/test_live_costs.py@@ -37,6 +37,15 @@ def test_seed_sweep_orders_width_one_first_then_widest_and_ends_when_sampled():assert not live.ready(4) and live.seed_width(4) == 1+def test_a_verify_is_priced_once_timed_and_before_its_bucket_is_seeded():+ live = _live()+ assert live.verify_us(1, 1) is None+ live.record(1, 1, 500, 4400.0)+ assert live.verify_us(1, 1) == 4400.0 and not live.ready(1)+ live.record(1, 1, 500, 4000.0)+ assert live.verify_us(1, 1) == 4000.0 and live.verify_us(4, 1) is None++def test_tables_hold_the_served_curve_and_fill_unmeasured_intervals():live = _live()_seed(live, 1, {1: 4000.0, 64: 7200.0, 32: 5700.0, 16: 4800.0, 8: 4500.0}, context=3000)diff --git a/tests/test_served_chain.py b/tests/test_served_chain.pynew file mode 100644index 0000000..2a50309--- /dev/null+++ b/tests/test_served_chain.py@@ -0,0 +1,455 @@+# Copyright 2026 Eagle AI Corp.+# SPDX-License-Identifier: LicenseRef-Eagle-4-Source-Available++"""The served-round chain policy of a serve-mode chain without data_path (:class:`ServedChainPolicy`), on CPU."""++from __future__ import annotations++import json+from types import SimpleNamespace++import pytest+import torch++from eagle_harness.core.config import HarnessConfig, Mode+from eagle_harness.core.limits import ComputeIntensity+from eagle_harness.sglang import chain_rounds+from eagle_harness.sglang.chain_rounds import ChainRounds, ServedChainPolicy, ServedTreePolicy, served_chain, served_tree+++class Clock:+ """``time.perf_counter_ns`` under test control."""++ def __init__(self):+ self.ns = 0++ def __call__(self):+ return self.ns++ def advance(self, ms: float) -> None:+ self.ns += int(ms * 1e6)+++def clock(monkeypatch):+ clock = Clock()+ monkeypatch.setattr(chain_rounds.time, "perf_counter_ns", clock)+ return clock+++def _policy(+ tmp_path=None, *, widths=(1, 2, 4, 8), native=False, writer=True, sync=lambda step: step, key=None, packaged=None+):+ return ServedChainPolicy(+ widths,+ torch.device("cpu"),+ max_batch_size=512,+ native_decode=native,+ writer=writer,+ sync=sync,+ cache_path=None if tmp_path is None else str(tmp_path / "chain.json"),+ key=key,+ packaged=packaged,+ )+++def _reqs(*ids):+ return [SimpleNamespace(rid=i) for i in ids]+++def _serve(policy, clock, bs, period_ms, accept, rounds=10_000):+ """Serve rounds of ``bs`` requests until their batch key is decided; ``period_ms[w]`` is the whole period of a+ width-``w`` round (0: native decode) and ``accept[w]`` its accept length. Returns the widths served."""+ requests = _reqs(*range(bs))+ served = []+ for _ in range(rounds):+ width = policy.width(bs)+ policy.round_start(bs, width, requests)+ if not policy.exploring(bs):+ return served+ served.append(width)+ if width:+ policy.accepted(width, torch.full((bs,), accept[width]))+ clock.advance(period_ms[width])+ raise AssertionError("no decision")+++def test_a_prefill_between_two_rounds_is_part_of_the_first_rounds_period(clock):+ policy = _policy()+ width = policy.width(64)+ policy.round_start(64, width, _reqs(1, 2))+ clock.advance(10)+ # A prefill admits request 3 (no decode round): its time stays in the previous round's period.+ clock.advance(25)+ policy.round_start(64, policy.width(64), _reqs(1, 2, 3))+ assert policy._cells[(64, width)] == [35_000.0, 64, 1] # us, requests (the batch size), periods+++def test_a_complete_turnover_does_not_count_as_a_served_interval(clock):+ policy = _policy()+ width = policy.width(8)+ policy.round_start(8, width, _reqs(1, 2))+ clock.advance(5_000) # the batch drained: the server may have idled before the next requests came+ policy.round_start(8, policy.width(8), _reqs(3, 4))+ assert policy._cells.get((8, width), [0.0, 0, 0])[2] == 0+++def test_a_burst_of_arrivals_counts_toward_the_comparison_without_being_priced(clock):+ policy = _policy()+ width = policy.width(64)+ policy.round_start(64, width, _reqs(*range(64)))+ clock.advance(200) # a large prefill admits 9 requests (more than 1/8 of the batch)+ policy.round_start(64, policy.width(64), _reqs(*range(9, 73)))+ assert policy._cells[(64, width)] == [0.0, 0, 1]+ clock.advance(10) # one refill: priced+ policy.round_start(64, policy.width(64), _reqs(*range(10, 74)))+ assert policy._cells[(64, width)] == [10_000.0, 64, 2]+++def test_a_turnover_measures_native_decode_and_its_best_width_again_together(clock):+ policy = _policy(native=True)+ policy._pair[64] = (4, 0)+ requests = _reqs(*range(64))+ while policy.width(64) == 4:+ policy.round_start(64, 4, requests)+ clock.advance(10)+ policy.round_start(64, 0, requests) # native decode's first round: the best width has its block+ clock.advance(5_000)+ policy.round_start(64, policy.width(64), _reqs(*range(100, 164))) # a new load after the batch drained+ assert policy._cells[(64, 4)] == [0.0, 0, 0] and policy._cells[(64, 0)] == [0.0, 0, 0]+ assert policy.width(64) == 4+++def test_the_search_widens_while_the_wider_chain_wins(clock):+ policy = _policy()+ # Accept lengths 1, 2, 3, 3 at widths 1, 2, 4, 8: tokens per ms 0.1, 0.167, 0.2, 0.1 per request.+ served = _serve(policy, clock, 64, {1: 10, 2: 12, 4: 15, 8: 30}, {1: 1, 2: 2, 4: 3, 8: 3})+ assert policy.decided == {64: 4} and set(served) == {1, 2, 4, 8}+ # Another batch key searches on its own: compute-bound rounds keep the root-only verify.+ _serve(policy, clock, 256, {1: 20, 2: 45, 4: 80, 8: 160}, {1: 1, 2: 2, 4: 3, 8: 3})+ assert policy.decided == {64: 4, 256: 1}+++def test_a_larger_batch_never_searches_wider_than_a_smaller_one_chose(clock):+ policy = _policy()+ _serve(policy, clock, 16, {1: 10, 2: 12, 4: 15, 8: 30}, {1: 1, 2: 2, 4: 3, 8: 3})+ # At batch 64 width 8 would win, but width 4 won at batch 16: the search stops there.+ served = _serve(policy, clock, 64, {1: 10, 2: 12, 4: 15, 8: 16}, {1: 1, 2: 2, 4: 3, 8: 5})+ assert 8 not in served and policy.decided == {16: 4, 64: 4}+++def test_native_decode_is_measured_last_and_spreads_to_larger_batches(clock):+ policy = _policy(native=True)+ # Width 2 beats width 1 (2/15 > 1/10) and width 4 (2/18); native decode (1/7) beats width 2.+ served = _serve(policy, clock, 64, {0: 7, 1: 10, 2: 15, 4: 18, 8: 25}, {1: 1, 2: 2, 4: 2, 8: 2})+ first_native = served.index(0)+ assert 0 not in served[:first_native] and set(served[first_native:]) == {0} # once native, the batch stays+ assert policy.decided[64] == 0+ assert all(policy.decided[bs] == 0 for bs in (96, 128, 192, 256, 384, 512)) # larger batch keys: unmeasured+ assert policy.exploring(48)+++def test_a_losing_native_decode_leaves_the_best_width(clock):+ policy = _policy(native=True)+ _serve(policy, clock, 16, {0: 9, 1: 10, 2: 12, 4: 15, 8: 30}, {1: 1, 2: 2, 4: 3, 8: 3})+ assert policy.decided == {16: 4}+++def test_no_smaller_batch_measures_native_decode_once_speculation_won(clock):+ policy = _policy(native=True)+ periods, accept = {0: 9, 1: 10, 2: 12, 4: 15, 8: 30}, {1: 1, 2: 2, 4: 3, 8: 3}+ assert 0 in _serve(policy, clock, 64, periods, accept) and policy.decided == {64: 4}+ assert 0 not in _serve(policy, clock, 16, periods, accept) and policy.decided[16] == 4+++def test_native_decode_is_not_chosen_without_the_engine_route(clock):+ policy = _policy(native=False)+ served = _serve(policy, clock, 64, {0: 1, 1: 10, 2: 15, 4: 18, 8: 25}, {1: 1, 2: 2, 4: 2, 8: 2})+ assert 0 not in served and policy.decided == {64: 2}+++def test_comparisons_interleave_the_two_widths_in_blocks(clock):+ policy = _policy()+ plan = []+ for _ in range(4 * chain_rounds.BLOCK_ROUNDS):+ width = policy.width(64)+ plan.append(width)+ policy.round_start(64, width, _reqs(1))+ block = chain_rounds.BLOCK_ROUNDS+ assert plan == ([1] * block + [2] * block) * 2+++def test_decisions_survive_a_restart_on_the_same_setup_only(tmp_path, clock):+ key = {"gpu": "NVIDIA B300", "gpu_count": 4}+ policy = _policy(tmp_path, key=key)+ _serve(policy, clock, 64, {1: 10, 2: 12, 4: 15, 8: 30}, {1: 1, 2: 2, 4: 3, 8: 3})+ restarted = _policy(tmp_path, key=key, writer=False)+ assert restarted.decided == {64: 4} and restarted.width(60) == 4 and not restarted.exploring(64)+ assert _policy(tmp_path, key={**key, "gpu_count": 8}).decided == {} # another setup explores again+ assert _policy(tmp_path, key=key, native=True).decided == {} # another action set too+ (tmp_path / "chain.json").write_text(+ json.dumps({**json.loads((tmp_path / "chain.json").read_text()), "chain": {"64": 3}})+ )+ assert _policy(tmp_path, key=key).decided == {} # a width without a graph is refused+++def test_a_first_start_serves_the_packaged_choices_of_its_setup(clock):+ policy = _policy(native=True, packaged={16: 4, 64: 4, 256: 0})+ assert {k: w for k, w in policy.decided.items() if w} == {16: 4, 64: 4}+ assert all(policy.decided[k] == 0 for k in (256, 384, 512)) # native decode serves every larger batch key+ assert policy.width(60) == 4 and policy.width(300) == 0 and not policy.exploring(64)+ assert policy.exploring(128) and policy.exploring(8) # unmeasured keys are searched on the served rounds+ # A search between packaged keys stays within the width a smaller key chose.+ served = _serve(policy, clock, 128, {0: 9, 1: 10, 2: 12, 4: 15, 8: 30}, {1: 1, 2: 2, 4: 3, 8: 3})+ assert 8 not in served and policy.decided[128] == 4+++def test_the_servers_own_choices_take_precedence_over_the_packaged_ones(tmp_path, clock):+ key = {"gpu": "NVIDIA B300 SXM6 AC", "gpu_count": 4}+ periods, accept = {0: 9, 1: 10, 2: 12, 4: 30, 8: 60}, {1: 1, 2: 2, 4: 3, 8: 3}+ _serve(_policy(tmp_path, key=key, native=True), clock, 64, periods, accept)+ assert _policy(tmp_path, key=key, native=True, packaged={16: 4, 64: 4, 256: 0}).decided == {64: 2}+++def test_packaged_choices_that_are_not_actions_of_the_server_are_not_served():+ assert _policy(native=False, packaged={16: 4, 64: 4, 256: 0}).decided == {} # no native decode route+ assert _policy(native=True, widths=(1, 2, 8), packaged={16: 4}).decided == {} # no graph for width 4+++def test_other_ranks_apply_rank_zeros_step_without_reading_the_gpu(clock):+ class NoDeviceCounts:+ def __getitem__(self, width):+ raise AssertionError("a non-writer rank counted accept lengths")++ def tolist(self):+ raise AssertionError("a non-writer rank read the accept counts")++ received = []+ policy = _policy(writer=False, sync=lambda step: received.append(step) or ("decide", 2, None))+ policy._accept = NoDeviceCounts()+ rounds = ChainRounds(None, None, lambda batch: 0, policy)+ requests = _reqs(1)+ while policy.exploring(8):+ width = policy.width(8)+ policy.round_start(8, width, requests)+ rounds.accepted(width, torch.ones(8, dtype=torch.int64))+ clock.advance(1)+ assert received == [None] and policy.decided == {8: 2}+++def test_batch_sizes_share_the_nearest_calibration_batch_size():+ policy = _policy()+ policy.decided = {64: 4, 256: 0}+ rounds = ChainRounds(None, None, lambda batch: 0, policy)+ assert rounds.draft_width(57) == rounds.draft_width(70) == 4+ assert rounds.plain_width(256) == 0 and rounds.draft_width(256) == 1 # a draft never asks for width 0+ assert policy.exploring(8) and not policy.exploring(250)+++def test_only_a_serve_mode_chain_without_data_path_is_served_by_the_policy():+ chain = HarnessConfig(intensity=ComputeIntensity.CHAIN)+ assert served_chain(chain, 1)+ assert not served_chain(chain, 4)+ assert not served_chain(HarnessConfig(intensity=ComputeIntensity.CHAIN, data_path="/cost"), 1)+ assert not served_chain(HarnessConfig(intensity=ComputeIntensity.CHAIN, mode=Mode.STATIC), 1)+ assert not served_chain(HarnessConfig(), 1)+++def _decide_width_4_or_native(clock, period_ms):+ """Width 4 (1.05 tokens per request and round) against native decode at batch 64; ``period_ms(i)`` is the i-th+ period of either."""+ policy = _policy(native=True)+ policy._pair[64] = (4, 0)+ policy.profile = lambda: chain_rounds.AcceptProfile((0.05, 0.0, 0.0))+ requests = _reqs(*range(64))+ i = 0+ while policy.exploring(64):+ policy.round_start(64, policy.width(64), requests)+ clock.advance(period_ms(i))+ i += 1+ return policy.decided[64]+++def test_speculation_beats_native_decode_only_beyond_the_noise_of_the_two_rates(clock):+ # Width 4 commits 5% more per period, but every other period of both takes twice as long: 32 periods each cannot+ # tell them apart, and native decode serves.+ assert _decide_width_4_or_native(clock, lambda i: 20 if i % 2 else 10) == 0+ # Without the noise the same 5% decides for speculation.+ assert _decide_width_4_or_native(clock, lambda i: 10) == 4+++def _tree(tmp_path=None, key=None, writer=True, sync=lambda step: step, native_us=None):+ return ServedTreePolicy(+ 16,+ torch.device("cpu"),+ max_batch_size=512,+ writer=writer,+ sync=sync,+ cache_path=None if tmp_path is None else str(tmp_path / "tree.json"),+ key=key,+ native_us=native_us,+ )+++def _serve_tree(policy, clock, bs, spec_ms, native_ms, accept, rounds=10_000, seed_rounds=0, widths=(16, 7, 3)):+ """Serve ``bs`` requests until their batch key is decided: speculative rounds (the cost model's widths, cycling+ through ``widths``) commit ``accept`` tokens per request in ``spec_ms``, native decode one in ``native_ms``. The+ first ``seed_rounds`` speculative rounds seed the live costs. Returns the actions served (16: speculation)."""+ requests = _reqs(*range(bs))+ served = []+ for i in range(rounds):+ action = policy.width(bs)+ width = widths[i % len(widths)] if action else 0+ settled = not (action and i < seed_rounds)+ policy.round_start(bs, width, requests, settled=settled)+ if not policy.exploring(bs):+ return served+ served.append(action)+ if width:+ policy.accepted(width, torch.full((bs,), accept, dtype=torch.int32))+ clock.advance(spec_ms if action and settled else (100 * spec_ms if action else native_ms))+ raise AssertionError("no decision")+++def test_a_tree_decodes_natively_where_its_rounds_commit_fewer_tokens_per_time(clock):+ policy = _tree()+ # 2.5 tokens per request in 30 ms (0.083 per ms) against native decode's 1 in 10 ms (0.1 per ms).+ served = _serve_tree(policy, clock, 256, spec_ms=30, native_ms=10, accept=2.5)+ first_native = served.index(0)+ assert set(served[:first_native]) == {16} and set(served[first_native:]) == {0} # native decode last, sticky+ assert all(policy.decided[k] == 0 for k in (256, 384, 512)) and policy.exploring(192)++[email protected]("stalled, spec_ms, winner", [(0, 30, 0), (16, 15, 16)])+def test_a_one_time_stall_does_not_decide_the_comparison(clock, stalled, spec_ms, winner):+ """A kernel compile or graph capture in one period (3 s against 10-30 ms rounds) is counted, not priced: the+ action that commits more tokens per time (2.5 in ``spec_ms`` against 1 in 10 ms) wins although it stalled."""+ policy, requests, rounds = _tree(), _reqs(*range(256)), {0: 0, 16: 0}+ while policy.exploring(256):+ action = policy.width(256)+ policy.round_start(256, action, requests)+ if action:+ policy.accepted(action, torch.full((256,), 2.5, dtype=torch.int32))+ rounds[action] += 1+ clock.advance(3000 if action == stalled and rounds[action] == 2 else spec_ms if action else 10)+ assert rounds[stalled] > 2 and policy.decided[256] == winner+ assert chain_rounds.STALL_FACTOR * 30 < 3000+++def test_the_first_round_after_a_switch_of_action_is_counted_not_priced(clock):+ """Native decode's first round after speculation also finishes speculation's work (a prelaunched draft): its+ period (95 ms here, under the stall factor) is counted toward the comparison but not priced."""+ policy, requests, compared = _tree(), _reqs(*range(256)), []+ policy._compare = lambda key, pair: compared.append(policy._rates(key, pair)) or policy._decide(key, 0)+ native = 0+ while policy.exploring(256):+ action = policy.width(256)+ policy.round_start(256, action, requests)+ if action:+ policy.accepted(action, torch.full((256,), 2.5, dtype=torch.int32))+ native += not action+ clock.advance(30 if action else 95 if native == 1 else 10)+ (rates,) = compared+ assert rates[0][0] == pytest.approx(256 / 10_000) and native == chain_rounds.EXPLORE_ROUNDS + 1+++def test_a_tree_keeps_speculating_where_it_wins_and_so_does_every_smaller_batch(clock):+ policy = _tree()+ _serve_tree(policy, clock, 64, spec_ms=15, native_ms=10, accept=2.5)+ assert policy.decided[64] == 16+ assert all(policy.decided[k] == 16 for k in (1, 2, 8, 16, 48)) and policy.exploring(96)+ rounds = ChainRounds(None, None, lambda batch: 0, policy)+ assert rounds.plain_width(1) == 16 and rounds.plain_width(64) == 16 # never native below a speculating key+++def test_rounds_that_seed_the_live_costs_are_not_counted(clock):+ policy = _tree()+ # The seed rounds force widths the cost model would not choose (100x the settled period here): were they counted,+ # native decode would win.+ served = _serve_tree(policy, clock, 64, spec_ms=15, native_ms=10, accept=2.5, seed_rounds=40)+ assert policy.decided[64] == 16+ assert served[:40] == [16] * 40 and served.count(0) == chain_rounds.EXPLORE_ROUNDS+++def test_speculation_far_ahead_of_native_decodes_estimate_never_decodes_natively(clock):+ # 4 tokens per request in 10 ms (0.4 per ms) against an estimated 1 in 10 ms: native decode would hold the batch.+ priced = []+ policy = _tree(native_us=lambda bs: priced.append(bs) or 10_000.0)+ served = _serve_tree(policy, clock, 8, spec_ms=10, native_ms=10, accept=4)+ assert 0 not in served and len(served) == chain_rounds.EXPLORE_ROUNDS+ assert set(priced) == {8} and all(policy.decided[k] == 16 for k in (1, 2, 8))++[email protected]("native_us", [lambda bs: 10_000.0, lambda bs: None])+def test_speculation_near_native_decodes_estimate_or_without_one_measures_native_decode(clock, native_us):+ # 2 tokens per request in 15 ms against an estimated 1 in 10 ms (1.33x, inside the skip ratio), or no estimate.+ policy = _tree(native_us=native_us)+ served = _serve_tree(policy, clock, 8, spec_ms=15, native_ms=10, accept=2)+ assert served.count(0) == chain_rounds.EXPLORE_ROUNDS and policy.decided[8] == 16+++def test_noisy_speculation_measures_native_decode_even_ahead_of_its_estimate(clock):+ # Periods of 1 and 49 ms: 1.6x the estimated 0.8 tokens per ms on average, but not beyond the noise.+ policy = _tree(native_us=lambda bs: 10_000.0)+ requests, served = _reqs(*range(8)), []+ while policy.exploring(8):+ action = policy.width(8)+ policy.round_start(8, action, requests)+ served.append(action)+ if action:+ policy.accepted(action, torch.full((8,), 4, dtype=torch.int32))+ clock.advance((1, 49)[len(served) % 2] if action else 10)+ assert 0 in served+++def test_a_chain_without_a_native_estimate_never_checks_one(clock):+ policy = _policy(native=True)+ policy._check_native = lambda key, best: pytest.fail("checked native decode without an estimate")+ _serve(policy, clock, 64, {0: 9, 1: 10, 2: 12, 4: 15, 8: 30}, {1: 1, 2: 2, 4: 3, 8: 3})+ assert 64 in policy.decided+++def test_other_ranks_take_rank_zeros_native_check_without_pricing(clock):+ received = []++ def sync(step):+ received.append(step)+ return ("decide", 16, None)++ policy = _tree(writer=False, sync=sync, native_us=lambda bs: pytest.fail("a non-writer rank priced native decode"))+ served = _serve_tree(policy, clock, 8, spec_ms=10, native_ms=10, accept=4)+ assert received == [None] and 0 not in served and policy.decided[8] == 16+++def test_a_first_tree_start_serves_the_packaged_choices_of_its_setup():+ policy = ServedTreePolicy(+ 16, torch.device("cpu"), max_batch_size=512, writer=True, sync=lambda step: step, packaged={8: 1, 64: 0}+ )+ assert all(policy.decided[k] == 16 for k in (1, 2, 8)) and all(policy.decided[k] == 0 for k in (64, 96, 512))+ assert policy.exploring(16) and policy.exploring(48) # unmeasured keys are searched on the served rounds+++def test_tree_decisions_survive_a_restart_on_the_same_setup(tmp_path, clock):+ key = {"gpu": "NVIDIA B300 SXM6 AC", "gpu_count": 4}+ _serve_tree(_tree(tmp_path, key=key), clock, 128, spec_ms=30, native_ms=10, accept=2.5)+ restarted = _tree(tmp_path, key=key, writer=False)+ assert restarted.width(130) == 0 and restarted.width(512) == 0 and restarted.exploring(64)+ assert _tree(tmp_path, key={**key, "gpu_count": 8}).decided == {}+++def test_other_ranks_never_read_the_trees_tokens(clock):+ policy = _tree(writer=False, sync=lambda step: ("decide", 0, None))+ requests = _reqs(1)+ while policy.exploring(256):+ width = policy.width(256)+ policy.round_start(256, width, requests)+ policy.accepted(width, torch.ones(256, dtype=torch.int32))+ clock.advance(1)+ assert policy._tokens is None and policy.decided[256] == 0+++def test_every_serve_mode_tree_chooses_native_decode_on_its_served_rounds():+ assert served_tree(HarnessConfig(), 4)+ assert served_tree(HarnessConfig(data_path="/cost"), 4)+ assert not served_tree(HarnessConfig(), 1)+ assert not served_tree(HarnessConfig(mode=Mode.STATIC), 4)diff --git a/tests/test_sglang_overlap.py b/tests/test_sglang_overlap.pyindex 3076bf1..393326d 100644--- a/tests/test_sglang_overlap.py+++ b/tests/test_sglang_overlap.py@@ -33,6 +33,7 @@ def test_decode_handoff_uses_the_scheduler_stream_contract(monkeypatch, overlap)drafter = SimpleNamespace(draft=lambda b: calls.append("draft"),last_width=4,+ seeding=False, # the round served the cost model's tree, not a live-cost seed widthtimer=None,prelaunch=Mock(side_effect=lambda b, **kwargs: (calls.append("prelaunch"), "fence")[1]),)diff --git a/tests/test_sglang_plugin.py b/tests/test_sglang_plugin.pyindex 27f4a56..a3fb3de 100644--- a/tests/test_sglang_plugin.py+++ b/tests/test_sglang_plugin.py@@ -349,6 +349,85 @@ def test_live_chain_does_not_read_offline_round_files(registered):assert isinstance(provider, VerifyWidthGraphs)[email protected]("native", [True, False])+def test_a_serve_mode_tree_captures_the_native_decode_graph_where_the_engine_has_the_route(registered, monkeypatch, native):+ from sglang.srt.model_executor.runner.decode_graph_variants import DecodeGraphVariant++ from eagle_harness.core.config import HarnessConfig+ from eagle_harness.sglang import plain_decode++ if native and not hasattr(DecodeGraphVariant, "forward_mode"):+ pytest.skip("host has no per-variant forward mode")+ monkeypatch.setattr(plain_decode, "native_decode_available", lambda: native)+ registered.__dict__["config"] = HarnessConfig()+ with get_context().override_server_args(+ speculative_num_draft_tokens=16, speculative_eagle_topk=4, max_running_requests=512+ ):+ provider = registered.create_decode_graph_variants(_runner_info(False))+ assert (provider._plain is not None) is native++[email protected]("draft_tokens", [4, 8, 16])+def test_served_chain_never_builds_the_cost_model(monkeypatch, draft_tokens):+ """A serve-mode chain without data_path chooses its widths on the served rounds, at every chain width (#145:+ 0.1.1 sized chains under 16 tokens by the tree rule). The cost model's chain-depth rule needs every width 1..max+ in its tables, which live costs (captured widths only) never have: it failed at the first seeded batch+ (``Invalid chain cost index: [0, 1, 3, 5, 7]``)."""+ from sglang.srt.speculative.eagle_worker_v2 import EAGLEWorkerV2++ from eagle_harness.sglang import worker as worker_mod++ def refuse(*args, **kwargs):+ raise AssertionError("the served chain built the cost model")++ monkeypatch.setattr(MlpL2WarmAhead, "install", classmethod(lambda cls, model, plan, device: None))+ monkeypatch.setattr(worker_mod, "build_cost_model", refuse)+ monkeypatch.setattr(worker_mod, "LiveCosts", refuse)+ monkeypatch.setattr(worker_mod, "cost_key", lambda runner: SimpleNamespace(digest="setup"))+ monkeypatch.setattr(EAGLEWorkerV2, "__init__", lambda self, *a, **k: None)+ monkeypatch.setattr(torch.cuda, "Event", lambda *a, **k: None)+ target = SimpleNamespace(model_runner=SimpleNamespace(model="target", device="cpu"))+ with get_context().override_server_args(+ speculative_algorithm_config=json.dumps({"intensity": "chain_intensity"}),+ speculative_eagle_topk=1,+ speculative_num_steps=draft_tokens - 1,+ speculative_num_draft_tokens=draft_tokens,+ max_running_requests=512,+ ):+ worker = EagleHarnessWorker(None, 0, 0, target)+ assert worker._served_chain and worker.cost_model is None and worker.live is None++[email protected]("native", [True, False])+def test_a_serve_mode_tree_keeps_its_cost_model_and_decodes_natively_where_the_engine_can(monkeypatch, native):+ """The default tree sizes its rounds with the cost model on live costs and, where the engine has the native+ route, chooses native decode per batch size on its served rounds (ServedTreePolicy)."""+ from sglang.srt.speculative.eagle_worker_v2 import EAGLEWorkerV2++ from eagle_harness.sglang import worker as worker_mod++ built = []+ monkeypatch.setattr(MlpL2WarmAhead, "install", classmethod(lambda cls, model, plan, device: None))+ monkeypatch.setattr(worker_mod, "build_cost_model", lambda config, **kw: built.append(kw) or SimpleNamespace(tables={}))+ monkeypatch.setattr(worker_mod, "LiveCosts", lambda **kw: SimpleNamespace(load=lambda path: False))+ monkeypatch.setattr(worker_mod, "cost_key", lambda runner: SimpleNamespace(digest="setup"))+ monkeypatch.setattr(worker_mod, "target_is_moe", lambda hf_config: False)+ monkeypatch.setattr(worker_mod, "native_decode_available", lambda: native)+ monkeypatch.setattr(EAGLEWorkerV2, "__init__", lambda self, *a, **k: None)+ monkeypatch.setattr(torch.cuda, "Event", lambda *a, **k: None)+ runner = SimpleNamespace(model="target", device="cpu", model_config=SimpleNamespace(hf_config=None))+ with get_context().override_server_args(+ speculative_algorithm_config="{}",+ speculative_eagle_topk=4,+ speculative_num_steps=12,+ speculative_num_draft_tokens=16,+ max_running_requests=512,+ ):+ worker = EagleHarnessWorker(None, 0, 0, SimpleNamespace(model_runner=runner))+ assert len(built) == 1 and worker.live is not None and not worker._served_chain+ assert worker._served_tree is native and worker._native_decode_enabled is native++def test_verify_widths_follow_the_cost_buckets():from eagle_harness.core.limits import ComputeIntensitydiff --git a/tests/test_sglang_sampled_worker.py b/tests/test_sglang_sampled_worker.pyindex d1c4800..31d0abf 100644--- a/tests/test_sglang_sampled_worker.py+++ b/tests/test_sglang_sampled_worker.py@@ -96,6 +96,44 @@ def test_startup_softmax_uses_the_served_vocabulary(monkeypatch, vocab_size):assert calls == [((1, vocab_size), torch.float32)]+def test_tree_warm_up_moves_the_kv_of_the_target_pool(monkeypatch):+ """A tree's verify moves the accepted tokens' KV in the target's pool: its move kernels compile before serving."""+ pool = object()+ moved, accept_tail = [], []+ monkeypatch.setattr(worker_mod, "warm_up_accept_tail", lambda *args: accept_tail.append(args))+ monkeypatch.setattr(worker_mod, "warm_up_kv_moves", lambda kv_cache, device: moved.append((kv_cache, device)))+ monkeypatch.setattr(worker_mod.spec_utils, "sampling_softmax", None)+ allocator = SimpleNamespace(get_kvcache=lambda: pool)+ worker = SimpleNamespace(+ device="cuda", topk=4, extend=SimpleNamespace(hidden_rows=lambda rows: rows),+ target_worker=SimpleNamespace(model_runner=SimpleNamespace(dtype=torch.bfloat16,+ token_to_kv_pool_allocator=allocator)),+ )+ worker_mod.HarnessDraftWorker._warm_up_kernels(worker)+ assert moved == [(pool, "cuda")] and len(accept_tail) == 1+ worker.topk = 1 # a chain's accepted tokens are already in place: nothing moves+ moved.clear()+ worker_mod.HarnessDraftWorker._warm_up_kernels(worker)+ assert moved == []+++def test_kv_move_warm_up_covers_every_program_a_round_can_compile():+ """sglang's tiled KV move compiles one program per next power of two of the moved slots (up to its chunk) and+ per slot count specialization (1, a multiple of 16, any other): the warm-up runs each, on the padding slot."""+ from eagle_harness.sglang.verify_step import KV_MOVE_MAX_SLOTS, warm_up_kv_moves++ moves = []+ warm_up_kv_moves(SimpleNamespace(move_kv_cache=lambda tgt, src: moves.append((tgt.clone(), src.clone()))), "cpu")+ assert all(tgt.dtype == torch.int64 and not tgt.any() and not src.any() for tgt, src in moves)++ def program(count):+ return 1 << (count - 1).bit_length(), "one" if count == 1 else "x16" if count % 16 == 0 else "other"++ warmed = {program(len(tgt)) for tgt, _ in moves}+ assert warmed == {program(count) for count in range(1, KV_MOVE_MAX_SLOTS + 1)}+ assert KV_MOVE_MAX_SLOTS >= 256 # sglang's largest chunk (smaller KV strides)++@pytest.mark.parametrize("engine_verify, active", [(_verify_without_hook, False), (_verify_with_hook, False), (_verify_with_hook, True)])def test_verify_passes_the_tree_sampler_only_for_an_active_sampled_tree(monkeypatch, engine_verify, active):monkeypatch.setattr(worker_mod, "run_eagle_verify", engine_verify)diff --git a/tests/test_tokenspeed_plugin.py b/tests/test_tokenspeed_plugin.pyindex 923c96b..324c8e9 100755--- a/tests/test_tokenspeed_plugin.py+++ b/tests/test_tokenspeed_plugin.py@@ -148,3 +148,44 @@ def test_candidate_gather_rides_the_last_dim_as_16_bit_words(monkeypatch):assert calls == [(torch.int16, (rows, 2 * n), "tp", -1)]assert out.shape == (tp, rows, n) and out.dtype == torch.int32assert all(torch.equal(out[r], shards[r]) for r in range(tp))+++def test_candidate_head_projects_through_the_engine_linear(monkeypatch):+ """The rank-r projection runs through the engine's linear (its GEMM dispatch picks the low-latency kernel for the+ few draft rows), and the shard scores are that projection times the score head, ``-inf`` past the real rows."""+ import torch++ from eagle_harness.tokenspeed import models++ rows, hidden, rank, shard_vocab, real_rows = 3, 16, 4, 10, 8+ g = torch.Generator().manual_seed(11)+ normed = torch.randn(rows, hidden, generator=g)+ down = torch.randn(rank, hidden, generator=g)+ score_head = torch.randn(shard_vocab, rank, generator=g)+ projected = []++ def project(x):+ projected.append(x)+ return x @ down.t(), None++ head = models.LlamaForCausalLMEagle4.__new__(models.LlamaForCausalLMEagle4)+ torch.nn.Module.__init__(head)+ head.efficient_lm_head_down_proj = project+ head.efficient_lm_head = torch.nn.Module()+ head.efficient_lm_head.weight = torch.nn.Parameter(score_head)+ head.lm_head = torch.nn.Module()+ head.lm_head.shard_indices = type("Shard", (), {"org_vocab_start_index": 0, "org_vocab_end_index": real_rows})()+ head.lm_head.weight = torch.nn.Parameter(torch.zeros(shard_vocab, hidden))+ head.num_candidates, head.tp_size, head.cuda_candidate_kernels = 4, 1, False+ seen = {}++ def candidate_head(scores, *args, **kwargs):+ seen["scores"] = scores+ return None, None, None++ monkeypatch.setattr(models, "fused_candidate_head", candidate_head)+ head.candidate_topk(normed, 2)+ assert len(projected) == 1 and projected[0] is normed+ expected = (normed @ down.t()) @ score_head.t()+ assert torch.allclose(seen["scores"][:, :real_rows], expected[:, :real_rows])+ assert torch.isinf(seen["scores"][:, real_rows:]).all()diff --git a/tests/test_vllm_calibrate.py b/tests/test_vllm_calibrate.pyindex beac155..71e25f1 100644--- a/tests/test_vllm_calibrate.py+++ b/tests/test_vllm_calibrate.py@@ -4,6 +4,7 @@"""Config-derived calibration geometry and real CPU fitting."""import json+import sysfrom dataclasses import asdictfrom types import SimpleNamespace@@ -124,8 +125,8 @@ def test_measure_records_resolved_model_limit(monkeypatch, tmp_path):geometry = SimpleNamespace(num_hidden_layers=2)config = SimpleNamespace(- model_config=SimpleNamespace(hf_config=geometry, max_model_len=8192, dtype="bfloat16"),- speculative_config=SimpleNamespace(draft_model_config=SimpleNamespace(hf_config=geometry)),+ model_config=SimpleNamespace(model="target", hf_config=geometry, max_model_len=8192, dtype="bfloat16"),+ speculative_config=SimpleNamespace(draft_model_config=SimpleNamespace(model="draft", hf_config=geometry)),)measured = []@@ -145,6 +146,52 @@ def test_measure_records_resolved_model_limit(monkeypatch, tmp_path):assert max(measured) > 4096 and max(measured) + 64 <= 8192+def test_measure_keys_tables_like_the_server(monkeypatch, tmp_path):+ # The server keys the tables it loads by the engine's resolved models (speculator.load_model). vLLM can resolve+ # a Hub id to a local snapshot directory, so keying by the command-line id makes serve.sh refuse its own tables.+ from eagle_harness.vllm import calibrate++ snapshots = {}+ for name in ("target", "draft"):+ snapshots[name] = tmp_path / "hub" / name / "snapshots" / "0123abcd"+ snapshots[name].mkdir(parents=True)+ (snapshots[name] / "config.json").write_text(json.dumps({"name": name}))+ geometry = SimpleNamespace(num_hidden_layers=2)+ config = SimpleNamespace(+ model_config=SimpleNamespace(+ model=str(snapshots["target"]), hf_config=geometry, max_model_len=4608, dtype="torch.bfloat16"+ ),+ speculative_config=SimpleNamespace(+ draft_model_config=SimpleNamespace(model=str(snapshots["draft"]), hf_config=geometry)+ ),+ )+ llm = SimpleNamespace(llm_engine=SimpleNamespace(vllm_config=config), collective_rpc=lambda name, args: [[]])+ monkeypatch.setattr(calibrate, "engine", lambda *args, **kwargs: llm)+ monkeypatch.setattr(calibrate.torch.cuda, "get_device_name", lambda *args: "fixture GPU")+ monkeypatch.setitem(sys.modules, "vllm", SimpleNamespace(__version__="fixture"))+ output = tmp_path / "raw.json"+ calibrate.measure(str(output), 8, False, [1], 256, 1, model="Org/target", draft="Org/draft", tp=4)+ served = calibrate.cost_key(str(snapshots["target"]), str(snapshots["draft"]), 4, "torch.bfloat16")+ assert json.loads(output.read_text())["key"] == served.to_dict()+++ "chain,width,max_num_seqs,overrides",+ [(False, 8, 1, {}), (False, 64, 1, {}), (True, 16, 64, {}), (False, 8, 1, {"max_model_len": 2048})],+)+def test_engine_token_budget_fits_one_batch(monkeypatch, chain, width, max_num_seqs, overrides):+ # vLLM's warmup splits its token budget over max_num_seqs requests; a request longer than max_model_len indexes+ # past the block table (device-side assert in the GDN checkpoint gather, issue #128).+ from eagle_harness.vllm import calibrate++ captured = {}+ monkeypatch.setitem(sys.modules, "vllm", SimpleNamespace(LLM=lambda **kwargs: captured.update(kwargs)))+ monkeypatch.setitem(sys.modules, "vllm.plugins", SimpleNamespace(load_general_plugins=lambda: None))+ calibrate.engine("target", "draft", 4, width, chain, max_num_seqs, **overrides)+ budget = captured.get("max_num_batched_tokens")+ assert budget is None or budget <= captured["max_num_seqs"] * captured["max_model_len"]++def test_merge_rejects_different_tp(tmp_path):paths = []for tp in (1, 2):diff --git a/tests/test_worker_rounds.py b/tests/test_worker_rounds.pyindex 7e64312..7ff5aa8 100644--- a/tests/test_worker_rounds.py+++ b/tests/test_worker_rounds.py@@ -104,7 +104,7 @@ def modules(monkeypatch):"draft_steps": ("DraftSteps",),"l2_warm": ("MlpL2WarmAhead",),"row_attention": ("DraftRowAttention",),- "verify_step": ("AcceptTailGraphs", "fused_accept_step", "fused_accept_supported"),+ "verify_step": ("AcceptTailGraphs", "fused_accept_step", "fused_accept_supported", "warm_up_kv_moves"),"verify_graphs": ("verify_widths", "round_up_width"),"verify_input": ("HarnessVerifyInput", "build_attn_args"),"verify_tail": ("VerifyTail",),@@ -164,6 +164,7 @@ def make_worker(modules, monkeypatch, *, calibration=None, schedule=None, nativedraft=draft,last_width=4,timer=None,+ seeding=False,prelaunch=prelaunch,)worker._drafter = drafter@@ -383,7 +384,7 @@ def test_result_copies_precede_scheduler_prelaunch_fence(modules, monkeypatch):assert modules.copies == [("copy", True, False), ("fence", fence)][email protected]("kind", ["tree", "schedule", "calibrate"])[email protected]("kind", ["tree", "schedule", "calibrate", "served", "served-tree"])def test_attention_initialization_wires_the_same_width_plan_into_both_routes(modules, monkeypatch, tmp_path, kind):class Draft:average_context = staticmethod(lambda batch: 5)@@ -396,12 +397,18 @@ def test_attention_initialization_wires_the_same_width_plan_into_both_routes(modmonkeypatch.setattr(module, "DynamicDrafter", Draft)monkeypatch.setattr(module, "verify_widths", lambda *args: [4, 1])worker = module.EagleHarnessWorker.__new__(module.EagleHarnessWorker)+ monkeypatch.setattr(module, "get_schedule", lambda: SimpleNamespace(max_running_requests=8))+ monkeypatch.setattr(module, "cost_key", lambda runner: SimpleNamespace(digest="setup", to_dict=lambda: {}))+ monkeypatch.setattr(module, "packaged_choices", lambda kind, key, runner: None) # an unmeasured setupworker.config = SimpleNamespace(mode=module.Mode.CALIBRATE if kind == "calibrate" else module.Mode.SERVE,intensity=module.ComputeIntensity.CHAIN,- data_path=str(tmp_path),+ data_path=None if kind == "served" else str(tmp_path),+ cost_cache_dir=str(tmp_path),)worker._chain_schedule = [(1, 1, 4), (2, 8, 0)] if kind == "schedule" else None+ worker._served_chain = kind == "served"+ worker._served_tree = kind == "served-tree"worker._native_decode_enabled = kind != "tree"worker._target_worker = SimpleNamespace(model_runner=object())worker.fixed = object()@@ -420,6 +427,14 @@ def test_attention_initialization_wires_the_same_width_plan_into_both_routes(modelif kind == "schedule":assert width(1) == 4 and width(2) == 1assert worker._native_decode.select(two)+ elif kind == "served":+ # Every batch size starts its search at the root-only width, and native decode is never explored.+ assert width(1) == width(2) == 1+ assert not worker._native_decode.select(two)+ elif kind == "served-tree":+ # The cost model sizes every round; a batch size speculates before native decode is measured against it.+ assert width is None and not worker._native_decode.select(two)+ assert isinstance(worker._chain_rounds._policy, module.ServedTreePolicy)else:block = BLOCK_ROUNDS + MIN_ROUNDSwidths = [width(1) for _ in range(2 * block + 1)]@@ -427,6 +442,18 @@ def test_attention_initialization_wires_the_same_width_plan_into_both_routes(modassert worker._native_decode.select(one)+def test_packaged_choices_look_up_this_servers_setup(modules, monkeypatch):+ module = modules.worker+ monkeypatch.setattr(module, "get_model", lambda: SimpleNamespace(model_path="/target", quantization=None))+ monkeypatch.setattr(module, "get_spec", lambda: SimpleNamespace(speculative_draft_model_path="/draft"))+ seen = {}+ monkeypatch.setattr(module, "packaged_served_choices", lambda kind, **setup: seen.update(setup, kind=kind) or {16: 4})+ key = SimpleNamespace(gpu="NVIDIA B300 SXM6 AC", gpu_count=4)+ assert module.packaged_choices("tree", key, SimpleNamespace(dtype=torch.bfloat16)) == {16: 4}+ assert seen == dict(kind="tree", device_name="NVIDIA B300 SXM6 AC", model_path="/target", draft_model_path="/draft", tp_size=4,+ dtype="torch.bfloat16", quantization=None)++@pytest.mark.parametrize("dp_size,tracking", [(2, False), (1, True), (2, True)])def test_native_decode_still_rejects_dp_and_mamba_tracking(modules, monkeypatch, dp_size, tracking):module = modules.worker