FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
AMD ROCm installation (WIP)
AMD support is experimental and remains a work in progress. See the AMD Support roadmap for the current integration and qualification status.
Requirements
- Linux x86_64
- AMD RDNA3/RDNA4 GPU (
gfx1100-gfx1103,gfx1200, orgfx1201) - ROCm 7.14
- Python >= 3.10
Install from source
Use an official ROCm PyTorch image whose PyTorch version satisfies the project's
torch>=2.11,<2.12 constraint. For RDNA4, the matching ROCm 7.14 image is:
VIDEO_GID="$(getent group video | cut -d: -f3)"
RENDER_GID="$(getent group render | cut -d: -f3)"
docker run --rm -it \
--device=/dev/kfd --device=/dev/dri \
--group-add="$VIDEO_GID" --group-add="$RENDER_GID" --ipc=host \
--cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
-e PYTORCH_ROCM_ARCH=gfx1201 -e FREETOKEN_ROCM_ARCH=gfx1201 \
-v "$PWD:/workspace/FreeToken" -w /workspace/FreeToken \
rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0 bash
Inside the container, preserve the ROCm-enabled PyTorch already supplied by the image and disable build isolation so it is also used to compile the extensions:
python -m pip install --no-build-isolation -e .
Set both architecture variables to gfx1200 for RX 9060 family GPUs, or to the
actual target reported by rocminfo.
Optional kernel backends
FreeToken's FlashInfer, sgl_kernel, and vLLM kernel integrations are CUDA-only.
On ROCm, these paths use the built-in fallbacks; do not install the [accel],
[fi], or [sgl] extras. Forcing --attention-backend fi, fa, or trtllm,
or NVFP4 Marlin/b12x (e.g. --quant-backend moe.nvfp4=marlin or
--quant-backend moe.nvfp4=b12x), fails with a ROCm-specific error.
CPU and hybrid MoE
--moe-strategy cpu, hybrid and --moe-cpu-layers sync the GPU with the CPU
expert pool through HIP stream memory operations. ROCm 10 records them into CUDA
graphs itself; on ROCm 7.14, which does not, FreeToken adds the graph nodes
explicitly. When the startup probe of that path fails, CUDA graph capture stops
with an error; run those strategies with --cuda-graph-max-bs 0.