Skip to content

Official Inference Code of EAGLE-4

10.1 KB
Raw Blame

type: how-to owner: eagle-harness last_verified: 2026-10-19

Onboard a target model

Train a draft whose structure the serving adapter implements, export its trained candidate head, then check correctness before calibrating. Changing an architecture name or adding a candidate-head config to untrained weights does not make a compatible draft.

The export steps below follow TorchSpec's tools/convert_to_hf.py and a real EAGLE4 safetensors export. A complete training-to-serving run for an arbitrary new target is not measured. Treat the admission and greedy checks as required stop points, not as proof that every target is supported.

Check the target and draft contract

First serve the target without speculation in your pinned engine. Confirm that its tokenizer, chat template, precision and attention backend work. Keep those settings for the speculative comparison.

For the measured EAGLE4 ports, train a dense single-layer draft with rotary positions, full target vocabulary, three auxiliary hidden-state inputs and a low-rank candidate head. Grouped-query draft attention is implemented, but the query and KV head counts must be compatible with the selected adapter and tensor-parallel layout. The target itself can use a hybrid architecture if the engine and EAGLE-4 support its verify and state-commit paths.

Use the target's hidden size, vocabulary and token IDs when building your draft configuration. Keep its trained RoPE parameters, auxiliary layer IDs, sliding window and normalization settings in the export. Only choose structures admitted by eagle_harness/models/eagle4_config.py and the selected engine's draft loader. See the support matrix for measured pairs.

Train with TorchSpec

The upstream training project is TorchSpec. Obtain a source bundle with the efficient-head training and export support below from the training maintainer. A reproducible repository revision for the inspected bundle is not available with this release. The commands below are an external recipe checked against its exporter source, not a verified end-to-end training path. Do not substitute an arbitrary TorchSpec revision. Install its training backend in a separate environment from serving, following that bundle's setup instructions. Its docs/draft_training_guide.md and configs/recipes/ contain the training recipes.

Prepare a target-specific YAML, train.yaml, with these fields set rather than copied from another target:

Field What to supply
model.target_model_path Target weights and tokenizer
model.draft_model_config Compatible draft architecture JSON
model.embedding_key, model.lm_head_key, model.norm_key Exact target checkpoint tensor names
inference.aux_hidden_states_layers Three training layer-output indices
dataset.train_data_path, dataset.chat_template, dataset.prompt_key Training conversations in the target's template
inference.inference_num_gpus, training.training_num_gpus_per_node Allocated inference and training resources
training.num_train_steps, training.save_interval, output_dir Run length and saved checkpoint locations

Train the dense drafter first and retain a saved checkpoint. From the TorchSpec checkout:

python -m torchspec.train_entry --config train.yaml --print-config-only
python -m torchspec.train_entry --config train.yaml

The resolved config must name the intended target, layers and output directory. Do not use a smoke recipe with checkpoint saving disabled for this step.

Next train the low-rank candidate factors from the trained dense drafter. In a second YAML, train-candidates.yaml, retain the same target and draft architecture and set:

model:
  initial_draft_model_path: ./dense-export
training:
  efficient_lm: true
  efficient_lm_rank: 128
  efficient_lm_top_k: 128
  loss_type: forward_kl

dense-export is the dense checkpoint exported by the command in the next section. Merge these fields into the complete target-specific training YAML, not a standalone four-field recipe. Rank 128 and 128 candidates are an explicit example, not a tuned choice for your target. TorchSpec requires a pretrained drafter through model.initial_draft_model_path or training.load_path, and checks that its output head is dense. Choose a fresh output_dir and a nonzero checkpoint interval, then run:

python -m torchspec.train_entry --config train-candidates.yaml --print-config-only
python -m torchspec.train_entry --config train-candidates.yaml

Check held-out candidate recall and served acceptance. A low training loss alone does not establish a useful speculative draft.

Export the checkpoint

Set CHECKPOINT to the saved iteration directory and EXPORT to a fresh output directory. Run this from the TorchSpec checkout, once for the dense initialization and again for the trained candidate-head checkpoint:

python tools/convert_to_hf.py --input-dir "$CHECKPOINT" --output-dir "$EXPORT"

The converter uses the checkpoint's config.json and discovers its model/ distributed-checkpoint directory. If the config is stored elsewhere, pass --config with the exact training config JSON. Avoid regenerating a config from the target when doing so would lose trained draft settings. If embeddings must be supplied from the target, use --target-model-path and the correct --embedding-key prefix.

Do not pass --prune-vocab. The efficient-head export requires the full target vocabulary. Do not pass --vllm for this export either, because TorchSpec's efficient-head converter refuses that option. EAGLE-4 vLLM adapter consumes the EAGLE4 native export and translates its auxiliary layer indices itself. Keep output-layer indices in eagle_config.eagle_aux_hidden_state_layer_ids, without manually adding one.

The serving directory contains:

export/
  config.json
  model.safetensors

A real inspected export also contained conversion_validation.json. That report is useful provenance, but it is not required by the loader. The weight header carries torchspec_version, and the config carries _torchspec_version and the output dtype.

Check these candidate-head tensors before serving. V is the full target vocabulary, H the draft hidden size and R the trained candidate rank:

Tensor Expected shape
lm_head.weight [V, H]
efficient_lm_head_down_proj.weight [R, H]
efficient_lm_head.weight [V, R]
embed_tokens.weight Target-compatible embedding shape
fc.weight Projection from the three auxiliary hidden states
fc_norm.0.weight, .1.weight, .2.weight Present when the trained draft uses feature normalization

The config must contain efficient_lm_head_config with type: low_rank, the same rank and the candidate count. TorchSpec refuses missing candidate factors or incompatible shapes. A safetensors header can be inspected without loading the full weights onto a GPU:

export EXPORT
python - <<'PY'
import json
import os
from pathlib import Path
from safetensors import safe_open

root = Path(os.environ["EXPORT"])
config = json.loads((root / "config.json").read_text())
head = config["efficient_lm_head_config"]
assert head["type"] == "low_rank"
vocab, hidden, rank = config["vocab_size"], config["hidden_size"], head["rank"]
expected = {
    "lm_head.weight": [vocab, hidden],
    "efficient_lm_head_down_proj.weight": [rank, hidden],
    "efficient_lm_head.weight": [vocab, rank],
}
with safe_open(root / "model.safetensors", framework="pt", device="cpu") as weights:
    for name, shape in expected.items():
        actual = weights.get_slice(name).get_shape()
        assert actual == shape, (name, actual, shape)
        print(name, actual)
PY

The inspected Nemotron draft used this format, including a rank-128 candidate head, latent MoE, attention normalization and no RoPE. SGLang and vLLM implement those structures. Preserve them in the export rather than removing fields to pass validation. Structural admission does not establish Nemotron release-stack performance or correctness qualification. See known limits.

Check generation before calibration

In your serving environment, run eagle-harness-check, then use the matching SGLang or vLLM launch recipe with DRAFT pointing to the export. These are release-specific recipes, not generic fixed-tree examples. SGLang seeds live costs, while vLLM needs matching calibrated costs and rounds. For vLLM, prepare those inputs with the calibration guide before launching if they are absent. Do not reuse another model/draft pair's calibration. Start with one request. Refused config fields require an adapter implementation or a compatible trained draft, not a renamed architecture.

Compare greedy token IDs on held-out prompts against the same engine without speculation. For each first mismatch, inspect the no-spec token logprobs and separate a near-tie from a genuine error. Record acceptance length as well. Acceptance near one can indicate a wrong export, wrong hidden layers or an incomplete head mapping even when the target verifies every output token.

Then test your required stops, concurrency, logprobs and sampling behavior. Keep every unsupported request feature out of the deployment contract.

Calibrate and serve

Stop the test servers and measure the admitted pair:

eagle-harness calibrate --engine vllm --model "$MODEL" --draft "$EXPORT" --tp "$TP" --output calibration-new-model &&
  sh calibration-new-model/serve.sh

This offline command is for vLLM. The pinned SGLang engine lacks the native-decode graph contract needed by the served-chain calibration CLI. Use its live-cost serving path instead. On a host exposing the required contract, SGLang chain calibration should use a representative --data file. The calibration guide explains the different per-engine measurements. Benchmark against a matched no-spec server at your deployment concurrency. Save the model and draft revisions, engine build, GPU type, TP, precision and calibration settings alongside the result before calling the new pair supported.