Skip to content

Accurate, fast FP4 KV-cache decoding through algorithm-system co-design.

873 B
Raw Blame

Benchmark

Kernel benchmark

Compares BF16, TRT-LLM NVFP4, and BitDecoding NVFP4 decode latency using synthetic inputs at 128K context:

REPO=/path/to/BitDecoding-FP4
CUDA_VISIBLE_DEVICES=0 "$REPO/_local/envs/bd_fp4/bin/python" "$REPO/benchmark/bench_kernel_sm10x.py"

Defaults: Qwen3-4B and Qwen3-30B-A3B shapes at batch sizes 1, 16, 32, and 64; override with --models and --batches.

E2E serving benchmark

Compares BF16, FP8, NVFP4, and BitDecoding NVFP4 decode throughput in vLLM at 1, 8, 16, 32, and 64 concurrent requests:

CUDA_VISIBLE_DEVICES=0 "$REPO/_local/envs/bd_fp4/bin/python" "$REPO/benchmark/bench_e2e_throughput.py" \
  --model Qwen/Qwen3-4B-Thinking-2507  # or Qwen/Qwen3-30B-A3B-Thinking-2507

Defaults: 64K-token prompts on one GPU; for 128K on two GPUs, add --workload 128k and set CUDA_VISIBLE_DEVICES=0,1.