Skip to content

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

2.1 KB
Raw Blame

AMD ROCm installation (WIP)

AMD support is experimental and remains a work in progress. See the AMD Support roadmap for the current integration and qualification status.

Requirements

  • Linux x86_64
  • AMD RDNA3/RDNA4 GPU (gfx1100-gfx1103, gfx1200, or gfx1201)
  • ROCm 7.14
  • Python >= 3.10

Install from source

Use an official ROCm PyTorch image whose PyTorch version satisfies the project's torch>=2.11,<2.12 constraint. For RDNA4, the matching ROCm 7.14 image is:

VIDEO_GID="$(getent group video | cut -d: -f3)"
RENDER_GID="$(getent group render | cut -d: -f3)"
docker run --rm -it \
  --device=/dev/kfd --device=/dev/dri \
  --group-add="$VIDEO_GID" --group-add="$RENDER_GID" --ipc=host \
  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
  -e PYTORCH_ROCM_ARCH=gfx1201 -e FREETOKEN_ROCM_ARCH=gfx1201 \
  -v "$PWD:/workspace/FreeToken" -w /workspace/FreeToken \
  rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0 bash

Inside the container, preserve the ROCm-enabled PyTorch already supplied by the image and disable build isolation so it is also used to compile the extensions:

python -m pip install --no-build-isolation -e .

Set both architecture variables to gfx1200 for RX 9060 family GPUs, or to the actual target reported by rocminfo.

Optional kernel backends

FreeToken's FlashInfer, sgl_kernel, and vLLM kernel integrations are CUDA-only. On ROCm, these paths use the built-in fallbacks; do not install the [accel], [fi], or [sgl] extras. Forcing --attention-backend fi, fa, or trtllm, or NVFP4 Marlin/b12x (e.g. --quant-backend moe.nvfp4=marlin or --quant-backend moe.nvfp4=b12x), fails with a ROCm-specific error.

CPU and hybrid MoE

--moe-strategy cpu, hybrid and --moe-cpu-layers sync the GPU with the CPU expert pool through HIP stream memory operations. ROCm 10 records them into CUDA graphs itself; on ROCm 7.14, which does not, FreeToken adds the graph nodes explicitly. When the startup probe of that path fails, CUDA graph capture stops with an error; run those strategies with --cuda-graph-max-bs 0.