Skip to content

DeepInfer: High-performance, large-scale inference framework for DeepSeek models

Python

docs: update V4.1 Flash Q&A to v0.2

hkeeea679bb86ce36
diff --git a/doc/inference_deployment_qa.md b/doc/inference_deployment_qa.md
index 0995f29..29d5b13 100644
--- a/doc/inference_deployment_qa.md
+++ b/doc/inference_deployment_qa.md
@@ -14,7 +14,7 @@ The DeepSeek V4.1 Flash inference system was built around three goals from the s
Around these goals we continuously optimize computation, communication, scheduling, and storage to raise the system's effective throughput while meeting latency requirements — serving more requests with less compute and energy, so that intelligent services reach more users at a lower price.
-We have previously open-sourced [TileLang and a series of inference kernel libraries](https://mp.weixin.qq.com/s/X41mKH4Ds-VXUAnK6M8Eww). **Today we are open-sourcing [DeepInfer](https://github.com/deepseek-ai/DeepInfer), our inference framework for large-scale cloud services.** It powers the high-throughput, low-latency large-scale inference service behind DeepSeek V4.1 Flash. A single cluster currently serves 155 trillion tokens per day, with overall revenue of US$38 per GPU-hour (at valley pricing).
+We have previously open-sourced [TileLang and a series of inference kernel libraries](https://mp.weixin.qq.com/s/X41mKH4Ds-VXUAnK6M8Eww). **Today we are open-sourcing [DeepInfer](https://github.com/deepseek-ai/DeepInfer), our inference framework for large-scale cloud services.** It powers the high-throughput, low-latency large-scale inference service behind DeepSeek V4.1 Flash. A single cluster currently serves 155 trillion tokens per day, with overall revenue of US$38 per GPU-hour (at the off-peak price).
DeepInfer combines high-performance kernels with a deeply optimized inference engine. It supports distributed inference, speculative decoding, SWA bounded replay, structured outputs, and more, and runs on both NVIDIA and Huawei Ascend platforms. Alongside it we are open-sourcing [**deepseek-recipe**](https://github.com/deepseek-ai/deepseek-recipe), a production-grade token encoding and decoding component that converts efficiently and accurately between mainstream LLM API protocols and model token sequences. Its processing logic is fully aligned with the official DeepSeek API, so third-party deployments get the same API behavior as the official one. Inference code for future DeepSeek models, along with subsequent performance work, will continue to be open-sourced in DeepInfer.
@@ -52,16 +52,40 @@ Reference architecture:
- TP and CP are not used for now.
+- Why use the smallest possible EP/DP size for prefill?
+
+ - Unlike decode, prefill already has a sufficiently large batch size per expert. Smaller DP/EP groups improve both data-parallel and expert-parallel load balance and reduce NVLink traffic.
+
- What does Wide-EP buy for decode?
- Wide-EP substantially reduces per-step memory traffic in the MoE expert part and, with good EPLB support, effectively raises TPS. The chart below shows EP8 vs. EP64 benchmark data (offline benchmark script, configuration as shown).
![EP8 vs. EP64 decode throughput](images/wide_ep_decode_pareto.png)
+- Why are CP and TP not used?
+
+ - Context parallelism (CP) splits long sequences across multiple GPUs, reducing tail TTFT. In our measurements, it is up to 5% slower than DP end to end. It also removes the need for the original DP load-balancing scheme, making it a reasonable tradeoff. The main reason we do not currently use it is that V4.1 is fast enough to keep tail TTFT acceptable even on a complete cache miss. Although CP improves data-parallel load balance, our experience is that it tends to worsen expert-parallel load balance. It also adds kernel, communication, and CPU overhead, which we are still optimizing.
+
+ - Tensor parallelism (TP) has high communication overhead. We do not use it when the model fits in memory and the latency SLA can be met without it.
+
+- How do you choose a parallelism strategy?
+
+ - Principle: maximize utilization while meeting reasonable TTFT and TPS targets.
+
+ - Process: analyze how each strategy affects kernel performance, account for its impact on load balance, and validate the choice with offline benchmarks.
+
- How is multimodal deployed?
- EPD disaggregation, with the vision encoder deployed on a single GPU.
+- How is Engram deployed?
+
+ - Engram uses a separate parallel group (ENP), sharded along the vocabulary dimension. Prefill uses ENP8; decode uses an ENP size equal to the DP/EP size.
+
+ - On GPUs, the vocabulary is stored in local host memory, with each rank holding only a `1/ENP` shard. This memory is registered for RDMA reads. At the start of the model's forward pass, a kernel issues one-sided RDMA reads, and the NIC transfers the data from host memory into GPU memory in the background. The scales are small enough to remain in GPU memory and are transferred over NVLink.
+
+ - On NPUs, CPU offload for Engram is not yet supported, so all Engram data remains in NPU memory.
+
- How large does the smallest deployment unit have to be?
- 1P + 1D is deployable. In practice the GPU ratio between P and D is roughly 1:1. Since the recommended decode configuration is EP64 or above, the smallest efficient deployment unit is 64 GPUs of prefill plus 64 GPUs of decode.
@@ -90,6 +114,10 @@ Reference architecture:
- A GC Worker component reclaims expired cache: one instance is deployed per storage node and periodically evicts the least recently used KV cache under an LRU policy, keeping disk usage below a watermark.
+- Can hardware be reallocated between training and inference clusters as traffic changes?
+
+ - Training and inference clusters are managed together, and the resource allocation between them can be adjusted dynamically.
+
## Expert-parallel load balancing
- What is the EPLB strategy for prefill?
@@ -126,6 +154,12 @@ Reference architecture:
- It can be compared against the offline benchmark: we reach roughly 80% of a perfectly balanced offline benchmark.
+- Why do prefill engines pull requests, while the Decode Load Balancer dispatches them?
+
+ - A prefill engine uses the remaining time in its current step to decide when to pull new requests. It pulls a batch when there is just enough time left to prepare it, overlapping request fetching and preparation with the current step's computation. The batch is then ready when that step finishes, minimizing TTFT while keeping the engine fully utilized.
+
+ - Once scheduled, a request stays on its decode engine until generation finishes. Prefill engines balance load locally within a machine. As request volume grows, the load should increase evenly across all decode engines. The Decode Load Balancer therefore collects the sequence count and total KV cache length from each decode engine, computes the averages, and routes requests to bring each engine toward those averages. Because this decision depends on a global view, requests are dispatched by the Decode Load Balancer.
+
## KV cache storage
| Content | Medium | Replication | Managed by |
@@ -156,6 +190,26 @@ Reference architecture:
- **Matching** is done by the Entrypoint against the local kvcache-store in the CHECK role, as a hash-chain prefix match over metadata only; but the hit length has to be computed before routing, so it cannot be hidden. **Fetching KV** happens asynchronously in a separate process, moving the KV cache from 3FS into host shared memory. **Swap-in** from shared memory to HBM runs in a thread inside the engine process, with the goal of overlapping per-layer H2D (host-to-device) transfers with forward computation. Because the CUDA copy engine performs poorly on the fine-grained copies of KV cache blocks, swap-in is implemented with GDR: an RDMA NIC saturates PCIe bandwidth without consuming SMs.
+- Does persisting the KV cache during decode use a write-back policy? How are write failures handled?
+
+ - Yes, it uses a write-back policy. Write failures are not reported to the engine, but the engine strictly validates data when reading it to ensure correctness.
+
+- How is KV cache version compatibility handled? Which updates prevent reuse, and have you migrated cache formats?
+
+ - Compatibility is identified by the tuple `(deployment configuration, model weights, user group)`. Changes to model weights, the block byte layout, or the tokenizer, as well as framework changes that alter KV cache values, prevent reuse of the old cache.
+
+ - We have not performed cache format migrations in production.
+
+- Would a centralized L2 cache based on CXL provide substantial benefits?
+
+ - The expected benefit would come from lower read latency on a cache hit. For V4.1, the KV cache is small, and reads already overlap with request queuing while the engine is computing, so we expect limited improvement in TTFT. SSDs also cost much less than DRAM.
+
+ - The main question is whether a CXL-based L2 cache could reduce SSD writes. For example, keeping hot or single-use cache entries in memory could reduce SSD wear and extend drive lifespan. We do not currently use this approach.
+
+- Do you aim for the maximum theoretically achievable cache hit rate? How do you balance storage capacity and hit rate?
+
+ - We use inexpensive SSD capacity to raise the hit rate as much as possible, and base the overall design on prefix reuse intervals measured in production. In practice, the current 3FS implementation comes very close to the maximum achievable hit rate: empirically, a TTL of 72 hours for agent traffic and 7 days for chatbot traffic captures almost all reusable cache entries. The 72-hour TTL for agent traffic also spans a weekend, allowing requests on subsequent workdays to reuse the cache.
+
### SWA KV cache
- How is it stored?
@@ -236,6 +290,12 @@ Reference architecture:
- End-to-end performance is close — at most 5% apart.
+- Why might Mega MoE performance stop improving as EP grows beyond 32 during decode?
+
+ - Increasing EP raises arithmetic intensity to avoid being memory-bound. Once memory bandwidth is no longer the bottleneck, a larger EP size introduces additional communication and synchronization overhead.
+
+ - Different jobs sharing NVLink within a supernode can interfere with one another and slow down. Using the whole supernode for a single EP group avoids that interference.
+
## Speculative decoding
- How is it implemented?
@@ -244,7 +304,7 @@ Reference architecture:
- How much throughput does dynamic-length verification gain over fixed-5?
- - Dynamic-length verification cuts the number of draft tokens verified per step by about 30% (5.00 → 3.2–4.4) while leaving the average accepted length essentially unchanged (3.29 vs. 3.20). Computed from EP64/DP64 benchmark data, that works out to +4–7% throughput at the production operating point (~60 concurrent sequences per rank).
+ - As shown in the table below, on the Open-PerfectBlend dataset, dynamic-length verification cuts the number of draft tokens verified per step by about 30% (5.00 → 3.2–4.4) while leaving the average accepted length essentially unchanged (3.29 vs. 3.20). Computed from EP64/DP64 benchmark data, that works out to +4–7% throughput at the production operating point (~60 concurrent sequences per rank).
| conc (seqs/rank) | fixed-5 avg verified length | dynamic-length avg verified length | fixed-5 accepted length | dynamic-length accepted length | fixed-5 estimated throughput (tok/s/rank) | dynamic-length estimated throughput (tok/s/rank) | dynamic vs. fixed-5 gain |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |