Skip to content

Local UI to run and train LLMs and diffusion models. Supports GGUF, MLX, Qwen3.8, DeepSeek-V4.1, MiniMax-H3, Gemma 4, FLUX and more.

Docker ROCm image: reach the GPU through WSL2's DXG bridge, not only /dev/kfd (#11212)

Leo Borcherding2223ec9c64ed
diff --git a/README.md b/README.md
index 5766c2e71ef..14cd2c40d1a 100644
--- a/README.md
+++ b/README.md
@@ -173,7 +173,7 @@ docker run -d --name unsloth --gpus all --ipc=host \
```
The log ends with your links and a generated JupyterLab password, plus a generated Unsloth Studio password on the first run against a new `unsloth-studio` volume; change that one on first sign-in or Unsloth Studio stops after an hour. A reused volume keeps the password already stored on it, and `docker exec unsloth unsloth studio reset-password --username unsloth` mints a new one and prints it. Ctrl-C stops following the log, not the container; `docker rm -f unsloth` deletes it. The Hugging Face cache keeps your models and the `unsloth-studio` volume keeps your accounts, chats and trained models, both across `docker rm`; a volume from an older image is migrated on first start, its old code kept under `.unsloth-studio-legacy/`. Those ports publish on every interface: on a cloud host add `-e UNSLOTH_STUDIO_SECURE=1`, drop `-p 8000:8000` and bind JupyterLab to `-p 127.0.0.1:8888:8888`, or bind both to `127.0.0.1` and use an SSH tunnel. Tags (`unsloth/unsloth:core` for notebooks only), GPU support and options: [Docker Hub](https://hub.docker.com/r/unsloth/unsloth).
-On AMD there is a separate image, [`unsloth/unsloth-rocm`](https://hub.docker.com/r/unsloth/unsloth-rocm), with the run command and the supported cards on its [Docker Hub page](https://hub.docker.com/r/unsloth/unsloth-rocm). It carries the training stack only, so there is no Unsloth Studio or JupyterLab in it, and it needs native Linux: WSL exposes `/dev/dxg` rather than the `/dev/kfd` that ROCm needs.
+On AMD there is a separate image, [`unsloth/unsloth-rocm`](https://hub.docker.com/r/unsloth/unsloth-rocm), with the run command and the supported cards on its [Docker Hub page](https://hub.docker.com/r/unsloth/unsloth-rocm). It carries the training stack only, so there is no Unsloth Studio or JupyterLab in it. On Linux it reaches the card through `/dev/kfd`; on Windows it goes over WSL2's `/dev/dxg` bridge instead, which needs ROCm for WSL on the host (`scripts/install_rocm_wsl_strixhalo.sh`), a docker engine running inside the WSL distribution rather than Docker Desktop's, and a per-architecture build (`ROCM_GFX=<your gfx> bash docker/build.sh --rocm`), since the generic image's torch aborts on the bridge. That build exists for the Strix APUs and RDNA4 (`gfx1150`, `gfx1151`, `gfx1152`, `gfx1200`, `gfx1201`); RDNA3 cards have no bridge path yet. `bash docker/run.sh --rocm` works out the flags on both; on WSL add `UNSLOTH_IMAGE=unsloth-rocm:latest` so it runs that build rather than the published image.
#### Remote HTTPS & LAN Access
Server-side tools are on by default - so **be careful**! Keep your password safe, or use `--disable-tools` when exposing Unsloth.
diff --git a/docker/DOCKERHUB-ROCM.md b/docker/DOCKERHUB-ROCM.md
index 0d9339b8881..fa295f1778d 100644
--- a/docker/DOCKERHUB-ROCM.md
+++ b/docker/DOCKERHUB-ROCM.md
@@ -6,13 +6,13 @@ This is the AMD counterpart to [`unsloth/unsloth`](https://hub.docker.com/r/unsl
## Quick start
-Needs Docker and a working amdgpu driver. Without Docker, start here:
+Needs Docker and, on Linux, a working amdgpu driver. Windows goes through WSL2 and is covered below. Without Docker, start here:
```bash
curl -fsSL https://get.docker.com -o get-docker.sh && sh get-docker.sh
```
-AMD GPUs are reached through the kernel driver's device nodes, not through a container toolkit, so the run command differs from the NVIDIA one:
+AMD GPUs are reached through the kernel driver's device nodes, not through a container toolkit, so the run command differs from the NVIDIA one. On Linux:
```bash
GPU_FLAGS="--device /dev/kfd"
@@ -31,16 +31,33 @@ docker run --rm -it \
Those few lines build the flags the way `docker/run.sh` does, and each part earns its place. `--group-add` needs the numeric group ids, because a name is resolved inside the container, where the host's `video` and `render` groups do not exist, so passing the names can add the wrong groups and leave `/dev/kfd` unreadable. A minimal host may have no `render` group at all, and a plain `--group-add "$(getent group render | cut -d: -f3)"` would then expand to an empty argument. And Docker refuses to start over a device that does not exist, so `/dev/dri` is named only when it is there, leaving the entrypoint to explain an incomplete driver rather than failing in the daemon.
+On Windows there is no `/dev/kfd`. WSL2 reaches the card over its DXG bridge, so the flags are the `/dev/dxg` node, the runtime's opt-in, and the host's `librocdxg` together with the `libdxcore` it loads from WSL's own lib directory (the image cannot ship `librocdxg`: its build needs Windows SDK headers). This needs ROCm for WSL on the host (`scripts/install_rocm_wsl_strixhalo.sh` installs it), a docker engine running inside the WSL distribution (Docker Desktop's own engine exposes neither node), and a per-architecture image: the generic image's torch bundles `librocprofiler-sdk`, which aborts on the bridge, and the entrypoint refuses before it gets that far. Until the per-architecture tags are published, build one with `ROCM_GFX=<your gfx> bash docker/build.sh --rocm`, which tags it `unsloth-rocm:latest` locally. That build exists for the Strix APUs and RDNA4 (`gfx1150`, `gfx1151`, `gfx1152`, `gfx1200`, `gfx1201`); RDNA3 cards (`gfx1100` to `gfx1103`) have no bridge path yet, because the Dockerfile does not map them to AMD's `gfx110X-all` wheels. Then:
+
+```bash
+GPU_FLAGS="--device /dev/dxg -e HSA_ENABLE_DXG_DETECTION=1 \
+ -v /opt/rocm/lib/librocdxg.so.1:/usr/lib/x86_64-linux-gnu/librocdxg.so:ro \
+ -v /usr/lib/wsl/lib:/usr/lib/wsl/lib:ro -e LD_LIBRARY_PATH=/usr/lib/wsl/lib"
+
+docker run --rm -it \
+ $GPU_FLAGS \
+ --ipc=host \
+ -v "$HOME/.cache/huggingface":/workspace/.cache/huggingface \
+ unsloth-rocm:latest
+```
+
+No `--group-add` here: WSL exposes `/dev/dxg` to everyone and has no `render` group.
+
The Hugging Face mount is not optional if you care about your downloads: `HF_HOME` inside the container is `/workspace/.cache/huggingface`, which lives in the container's writable layer, so without it every model is fetched again after `docker rm`.
-Or let the launcher work out the device nodes, group ids and mounts for you:
+Or let the launcher work out the device nodes, group ids and mounts for you, on Linux and on WSL alike. It defaults to the published image, so on WSL name the build from above:
```bash
curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/main/docker/run.sh -o run.sh
-bash run.sh --rocm
+bash run.sh --rocm # Linux
+UNSLOTH_IMAGE=unsloth-rocm:latest bash run.sh --rocm # WSL
```
-Check the GPU is visible before anything else, with `GPU_FLAGS` set as above:
+Check the GPU is visible before anything else, with `GPU_FLAGS` set as above (on WSL, `unsloth-rocm:latest` in place of `unsloth/unsloth-rocm`):
```bash
docker run --rm $GPU_FLAGS \
diff --git a/docker/entrypoint-rocm.sh b/docker/entrypoint-rocm.sh
index bcdfd5432fe..48e51c919ed 100644
--- a/docker/entrypoint-rocm.sh
+++ b/docker/entrypoint-rocm.sh
@@ -15,6 +15,7 @@ set -euo pipefail
err() { printf "\033[1;31mERROR:\033[0m %s\n" "$*" >&2; }
warn() { printf "\033[1;33mWARN:\033[0m %s\n" "$*" >&2; }
+note() { printf "\033[1;36mNOTE:\033[0m %s\n" "$*" >&2; }
# UNSLOTH_DEV_ROOT prefixes the /dev probes (DESTDIR idiom) so the regression
# tests can stage a device tree; leave it unset in normal use.
@@ -44,10 +45,57 @@ if [[ "${UNSLOTH_SKIP_GPU_CHECK:-0}" == "1" ]]; then
exec "$@"
fi
-# --- Check 1: /dev/kfd is accessible ----------------------------------------
-# /dev/kfd is the AMD Kernel Fusion Driver node. It must exist AND be readable
-# by the container user before any HIP / torch.cuda call can succeed.
-if [[ ! -e "$DEV_ROOT/dev/kfd" ]]; then
+# --- Check 1: a device node the HSA runtime can open ------------------------
+# Two ways in, and only one of them is /dev/kfd:
+# KFD the AMD Kernel Fusion Driver node, from the Linux amdgpu driver. Must
+# exist AND be readable before any HIP call can succeed.
+# DXG WSL2, where amdgpu is not loaded and there is no /dev/kfd at all. The
+# standard hsa-rocr runtime reaches the card through librocdxg over
+# /dev/dxg when HSA_ENABLE_DXG_DETECTION=1. Same evidence install.sh
+# gates a WSL host on (_infer_linux_amd_gfx_arch).
+# The DXG path takes the rest of the checks unchanged: check 3 asks torch, which
+# is the only gate that has ever mattered.
+UNSLOTH_ROCM_DEV_PATH=kfd
+if [[ ! -e "$DEV_ROOT/dev/kfd" && -e "$DEV_ROOT/dev/dxg" ]]; then
+ _dxg_lib=""
+ # UNSLOTH_ROCM_DXG_LIBDIRS pins the search for the regression tests, which must
+ # not answer for whatever ROCm the machine running them happens to have.
+ for _d in ${UNSLOTH_ROCM_DXG_LIBDIRS:-/opt/rocm/lib /opt/rocm/lib64 /opt/rocm-*/lib \
+ /opt/rocm-*/lib64 /usr/lib/x86_64-linux-gnu}; do
+ if [[ -e "$_d/librocdxg.so" || -e "$_d/librocdxg.so.1" ]]; then
+ _dxg_lib="$_d"
+ break
+ fi
+ done
+ if [[ -n "$_dxg_lib" ]]; then
+ UNSLOTH_ROCM_DEV_PATH=dxg
+ # Only the runtime reads this, and only when asked; exporting it here means
+ # a plain `docker run --device /dev/dxg` works without the caller knowing.
+ export HSA_ENABLE_DXG_DETECTION="${HSA_ENABLE_DXG_DETECTION:-1}"
+ note "no /dev/kfd, but /dev/dxg and librocdxg are present: using the WSL2 DXG bridge."
+ else
+ err "/dev/dxg is present but librocdxg is not, so the DXG bridge cannot load."
+ cat >&2 <<'MSG'
+
+This looks like WSL2, where there is no /dev/kfd and the card is reached through
+librocdxg instead. The image cannot ship that library (its build needs Windows
+SDK headers), so it comes off the host, along with the libdxcore it dlopens from
+WSL's own lib directory. The bundled wrapper mounts both; name this image, since
+it defaults to the published one, which cannot use the bridge:
+ UNSLOTH_IMAGE=unsloth-rocm:latest bash docker/run.sh --rocm <cmd>
+Or by hand:
+ docker run --device /dev/dxg \
+ -v /opt/rocm/lib/librocdxg.so.1:/usr/lib/x86_64-linux-gnu/librocdxg.so:ro \
+ -v /usr/lib/wsl/lib:/usr/lib/wsl/lib:ro -e LD_LIBRARY_PATH=/usr/lib/wsl/lib \
+ <other-flags> unsloth-rocm:latest <cmd>
+
+To bypass this check (e.g. offline tooling), set UNSLOTH_SKIP_GPU_CHECK=1.
+MSG
+ exit 1
+ fi
+fi
+
+if [[ "$UNSLOTH_ROCM_DEV_PATH" == "kfd" && ! -e "$DEV_ROOT/dev/kfd" ]]; then
err "/dev/kfd not found inside the container."
cat >&2 <<'MSG'
@@ -62,17 +110,20 @@ The AMD GPU device node is missing. Likely causes:
--group-add $(getent group render | cut -d: -f3) \
<other-flags> unsloth/unsloth-rocm:latest <cmd>
- 2. The host has no /dev/kfd to pass through. Docker Desktop on Windows and
- macOS never has one (ROCm needs the Linux amdgpu driver; WSL exposes
- /dev/dxg instead), so this image cannot reach a GPU there. On Linux, check
- the driver: lsmod | grep amdgpu
+ 2. The host has no /dev/kfd to pass through. On Linux, check the driver:
+ lsmod | grep amdgpu
+ On Windows there is never one: the card is reached over WSL2's /dev/dxg,
+ which only a docker engine running INSIDE your WSL distribution can pass
+ through, and only into a per-arch build of this image:
+ UNSLOTH_IMAGE=unsloth-rocm:latest bash docker/run.sh --rocm <cmd>
+ Docker Desktop's own engine exposes neither node.
To bypass this check (e.g. offline tooling), set UNSLOTH_SKIP_GPU_CHECK=1.
MSG
exit 1
fi
-if [[ ! -r "$DEV_ROOT/dev/kfd" ]]; then
+if [[ "$UNSLOTH_ROCM_DEV_PATH" == "kfd" && ! -r "$DEV_ROOT/dev/kfd" ]]; then
err "/dev/kfd exists but is not readable by the container user."
cat >&2 <<'MSG'
Add the host's video and render group ids to the container (NUMERIC: the
@@ -85,8 +136,13 @@ fi
# --- Check 2: what rocm-smi sees (advisory) --------------------------------
# rocm-smi does not list every APU (measured on gfx1151: no GPU[..] line inside
-# the container), so it cannot be the gate; check 3 asks torch itself.
-if ! command -v rocm-smi >/dev/null 2>&1; then
+# the container), so it cannot be the gate; check 3 asks torch itself. It reads
+# the amdgpu sysfs, which the DXG bridge has no equivalent of (measured: "Driver
+# not initialized"), and its advice below is about /dev/dri and group ids, which
+# do not exist on WSL: skip it there.
+if [[ "$UNSLOTH_ROCM_DEV_PATH" == "dxg" ]]; then
+ note "skipping rocm-smi: it reads the amdgpu sysfs, which the DXG bridge does not expose."
+elif ! command -v rocm-smi >/dev/null 2>&1; then
warn "rocm-smi not found inside the container; skipping its listing."
elif ! rocm-smi --showid 2>/dev/null | grep -q 'GPU\['; then
warn "rocm-smi lists no GPU from inside the container. That is normal for some APUs;"
@@ -100,7 +156,7 @@ fi
# --- Check 3: a HIP torch that can see the device ---------------------------
# ROCm maps the CUDA Python API, so torch.cuda.is_available() is the test;
# torch.version.hip first, so a CUDA or CPU torch is named, not the host driver.
-IMAGE_ROCM="$IMAGE_ROCM" python - >&2 <<'PY' || exit 1
+IMAGE_ROCM="$IMAGE_ROCM" UNSLOTH_ROCM_DEV_PATH="$UNSLOTH_ROCM_DEV_PATH" python - >&2 <<'PY' || exit 1
import os
import sys
@@ -114,12 +170,45 @@ if hip_ver is None:
print("rebuild it with bash docker/build.sh --rocm.")
sys.exit(1)
+# BEFORE the first torch.cuda call, which is where this would otherwise abort with a
+# glog FATAL and no explanation. librocprofiler-sdk enumerates GPUs from the KFD sysfs
+# topology; on the DXG bridge there is none, so it finds zero agents, disagrees with the
+# HSA agents that are there, and calls abort(). Match that library alone: the
+# librocprofiler-register.so beside it is harmless, and torch 2.11+rocm7.2 ships it and
+# runs here, so matching "rocprof" refuses a build that works.
+if os.environ.get("UNSLOTH_ROCM_DEV_PATH") == "dxg":
+ _lib = os.path.join(os.path.dirname(torch.__file__), "lib")
+ _prof = sorted(f for f in os.listdir(_lib) if f.startswith("librocprofiler-sdk")) if os.path.isdir(_lib) else []
+ if _prof:
+ print("ERROR: this image cannot use the WSL2 DXG bridge.")
+ print()
+ print(f"Its torch bundles {', '.join(_prof)}, which enumerates GPUs through the")
+ print("KFD sysfs topology that WSL does not have; it aborts rather than falling back.")
+ print("Two builds do work here: AMD's per-arch wheels, which bundle no rocprofiler")
+ print("(Strix APUs and RDNA4: gfx1150/1151/1152/1200/1201, no RDNA3 yet),")
+ print(" ROCM_GFX=<your gfx, e.g. gfx1201> bash docker/build.sh --rocm")
+ print(" UNSLOTH_IMAGE=unsloth-rocm:latest bash docker/run.sh --rocm <cmd>")
+ print("or a torch before 2.12 from the same index, which ships only -register.so.")
+ print("UNSLOTH_SKIP_GPU_CHECK=1 reaches the abort itself rather than avoiding it.")
+ sys.exit(1)
+
if torch.cuda.is_available():
sys.exit(0)
image_rocm = os.environ.get("IMAGE_ROCM") or ".".join(hip_ver.split(".")[:2])
print("ERROR: torch is a ROCm build but torch.cuda.is_available() is False: HIP could")
print("not open the device the container was given.")
print()
+if os.environ.get("UNSLOTH_ROCM_DEV_PATH") == "dxg":
+ # WSL has no host amdgpu stack: the Linux driver advice below is wrong here.
+ print("This container is on the WSL2 DXG bridge, so the host amdgpu driver is not")
+ print("involved. Run rocminfo inside the container:")
+ print(" 'Failed to load libdxcore.so': /usr/lib/wsl/lib is not mounted or not on")
+ print(" LD_LIBRARY_PATH (measured). Both come from")
+ print(" UNSLOTH_IMAGE=unsloth-rocm:latest bash docker/run.sh --rocm <cmd>")
+ print(" no GPU agent at all: update the Windows AMD driver (a current Adrenalin with")
+ print(" ROCm-on-WSL support, as scripts/install_rocm_wsl_strixhalo.sh documents).")
+ print("If HSA_OVERRIDE_GFX_VERSION is set, a wrong value also produces this.")
+ sys.exit(1)
print(f"This image was built against ROCm {image_rocm} (HIP {hip_ver}). The host's")
print("amdgpu driver has to be at least as new. Check the host (NOT the container):")
print(" rocm-smi --version or cat /opt/rocm/.info/version or dkms status")
diff --git a/docker/run.sh b/docker/run.sh
index 9ce174e603e..d27e89cfd59 100755
--- a/docker/run.sh
+++ b/docker/run.sh
@@ -67,6 +67,41 @@ fi
# regression tests can stage a fake device tree; leave it unset in normal use.
DEV_ROOT="${UNSLOTH_DEV_ROOT:-}"
+# WSL2 has no /dev/kfd: the amdgpu kernel driver is not loaded there and the card
+# is reached over the DXG bridge instead. /dev/dxg plus librocdxg is the same GPU
+# evidence install.sh gates on for a WSL host. The bridge is userspace, so this
+# needs a device and an env var, not group ids: WSL exposes /dev/dxg to everyone
+# and has no render group.
+wsl_dxg_host() {
+ [[ ! -e "$DEV_ROOT/dev/kfd" && -e "$DEV_ROOT/dev/dxg" ]]
+}
+amd_dxg_flags() {
+ printf '%s\n' --device /dev/dxg
+ # librocdxg is NOT in the image and cannot be: its cmake build needs the Windows
+ # 11 SDK 'shared' headers off the host (see scripts/install_rocm_wsl_strixhalo.sh),
+ # which no Linux build runner has. Mount the host's, which that helper installs.
+ local _so=""
+ for _c in "$DEV_ROOT"/opt/rocm/lib/librocdxg.so.1* "$DEV_ROOT"/opt/rocm-*/lib/librocdxg.so.1* \
+ "$DEV_ROOT"/opt/rocm/lib64/librocdxg.so.1* ; do
+ [[ -e "$_c" ]] && { _so="$_c"; break; }
+ done
+ if [[ -z "$_so" ]]; then
+ printf "\033[1;33mWARN:\033[0m /dev/dxg is present but no librocdxg was found under /opt/rocm.\n" >&2
+ printf " Install ROCm for WSL first: bash scripts/install_rocm_wsl_strixhalo.sh\n" >&2
+ return 0
+ fi
+ printf '%s\n' -v "${_so}:/usr/lib/x86_64-linux-gnu/librocdxg.so:ro"
+ # librocdxg dlopens libdxcore from WSL's own lib directory, which the image
+ # does not have on its search path.
+ if [[ -d "$DEV_ROOT/usr/lib/wsl/lib" ]]; then
+ printf '%s\n' -v /usr/lib/wsl/lib:/usr/lib/wsl/lib:ro -e LD_LIBRARY_PATH=/usr/lib/wsl/lib
+ else
+ printf "\033[1;33mWARN:\033[0m /usr/lib/wsl/lib is missing, so librocdxg cannot load libdxcore.\n" >&2
+ fi
+ # The standard HSA runtime only looks for the bridge when this is set.
+ printf '%s\n' -e HSA_ENABLE_DXG_DETECTION=1
+ return 0
+}
# Named once: the NVIDIA toolkit installer, both as the fallback download below
# and in the message that tells you to run it yourself.
TOOLKIT_URL="${UNSLOTH_TOOLKIT_URL:-https://raw.githubusercontent.com/unslothai/unsloth/main/docker/install_nvidia_toolkit.sh}"
@@ -100,18 +135,28 @@ collect_amd_device_flags() {
GPU_FLAG+=("$_flag")
done < <(amd_device_flags)
}
+collect_amd_dxg_flags() {
+ local _flag
+ GPU_FLAG=()
+ while IFS= read -r _flag; do
+ GPU_FLAG+=("$_flag")
+ done < <(amd_dxg_flags)
+}
if [[ $ROCM -eq 1 ]]; then
IMAGE="${UNSLOTH_IMAGE:-unsloth/unsloth-rocm:latest}"
GPUS=none
if [[ -e "$DEV_ROOT/dev/kfd" ]]; then
collect_amd_device_flags
+ elif wsl_dxg_host; then
+ collect_amd_dxg_flags
+ printf "\033[1;33mNOTE:\033[0m no /dev/kfd; passing /dev/dxg, the WSL2 bridge to the card.\n" >&2
else
GPU_FLAG=()
printf "\033[1;33mWARN:\033[0m /dev/kfd is not present, so no AMD GPU can be passed through.\n" >&2
printf " On Linux install the amdgpu driver and add yourself to the video/render\n" >&2
- printf " groups. Docker Desktop on Windows and macOS has no /dev/kfd at all: the\n" >&2
- printf " ROCm image cannot reach a GPU there, whatever the host card is.\n\n" >&2
+ printf " groups. On Windows the card is reached over WSL2's /dev/dxg, which only a\n" >&2
+ printf " docker engine running INSIDE your WSL distribution can pass through.\n\n" >&2
fi
else
IMAGE="${UNSLOTH_IMAGE:-unsloth/unsloth:latest}"
diff --git a/tests/python/test_docker_rocm.py b/tests/python/test_docker_rocm.py
index dbfad3d6587..a38f7c9b78b 100644
--- a/tests/python/test_docker_rocm.py
+++ b/tests/python/test_docker_rocm.py
@@ -25,6 +25,8 @@ _ENTRYPOINT = os.path.join(_DOCKER, "entrypoint-rocm.sh")
_DOCKERFILE = os.path.join(_DOCKER, "Dockerfile.rocm")
_SMOKE = os.path.join(_DOCKER, "smoke_test_rocm.py")
_WORKFLOW = os.path.join(_REPO, ".github", "workflows", "docker-publish-rocm.yml")
+_HUB_PAGE = os.path.join(_DOCKER, "DOCKERHUB-ROCM.md")
+_README = os.path.join(_REPO, "README.md")
_posix_shell = pytest.mark.skipif(
os.name != "posix" or shutil.which("bash") is None,
@@ -47,6 +49,9 @@ def _run_sh(
*,
kfd = True,
dri = True,
+ dxg = False,
+ librocdxg = False,
+ wsl_lib = True,
nvidia = False,
groups = "both",
extra_env = None,
@@ -83,6 +88,14 @@ def _run_sh(
(dev_root / "dev" / "kfd").write_text("")
if dri:
(dev_root / "dev" / "dri").mkdir()
+ if dxg:
+ (dev_root / "dev" / "dxg").write_text("")
+ if librocdxg:
+ lib = dev_root / "opt" / "rocm" / "lib"
+ lib.mkdir(parents = True)
+ (lib / "librocdxg.so.1.2.1").write_text("")
+ if wsl_lib:
+ (dev_root / "usr" / "lib" / "wsl" / "lib").mkdir(parents = True)
env = dict(os.environ)
env["PATH"] = str(bindir) + ":/usr/bin:/bin"
env["UNSLOTH_DEV_ROOT"] = str(dev_root)
@@ -177,12 +190,54 @@ class TestRunShRocm:
assert image == "unsloth/unsloth-rocm:latest"
assert "/dev/kfd" in argv
- def test_no_kfd_warns_and_starts_without_devices(self, tmp_path):
- """Docker Desktop has no /dev/kfd; the entrypoint then explains, so the
- container must still start rather than docker failing on a missing node."""
+ def test_no_kfd_and_no_dxg_warns_and_starts_without_devices(self, tmp_path):
+ """A host with neither node has no GPU to pass: the container must still start
+ rather than docker failing on a missing device."""
argv, stderr = _run_sh(tmp_path, ["--rocm", "true"], kfd = False)
assert "--device" not in argv and "--gpus" not in argv, argv
- assert "/dev/kfd is not present" in stderr and "Docker Desktop" in stderr, stderr
+ assert "/dev/kfd is not present" in stderr, stderr
+ assert "/dev/dxg" in stderr, stderr
+
+ def test_wsl_passes_dxg_instead_of_kfd(self, tmp_path):
+ """WSL2 has no /dev/kfd: the card is reached over the DXG bridge, so the flags
+ are the device plus the runtime's opt-in, and librocdxg off the host (its cmake
+ build needs Windows SDK headers, so no Linux image build can carry it)."""
+ argv, stderr = _run_sh(
+ tmp_path,
+ ["--rocm", "true"],
+ kfd = False,
+ dxg = True,
+ librocdxg = True,
+ )
+ assert "/dev/dxg" in argv, argv
+ assert "/dev/kfd" not in argv, argv
+ assert "HSA_ENABLE_DXG_DETECTION=1" in argv, argv
+ assert any("librocdxg.so" in a for a in argv), argv
+ # librocdxg dlopens libdxcore from here; without the mount hsa_init fails
+ # (measured on an R9700: "Failed to load libdxcore.so")
+ assert "/usr/lib/wsl/lib:/usr/lib/wsl/lib:ro" in argv, argv
+ assert "LD_LIBRARY_PATH=/usr/lib/wsl/lib" in argv, argv
+ assert "--gpus" not in argv, argv
+ assert "DXG" in stderr or "dxg" in stderr, stderr
+
+ def test_a_missing_wsl_lib_dir_is_named_not_silently_dropped(self, tmp_path):
+ argv, stderr = _run_sh(
+ tmp_path,
+ ["--rocm", "true"],
+ kfd = False,
+ dxg = True,
+ librocdxg = True,
+ wsl_lib = False,
+ )
+ assert "LD_LIBRARY_PATH=/usr/lib/wsl/lib" not in argv, argv
+ assert "/usr/lib/wsl/lib is missing" in stderr, stderr
+
+ def test_dxg_without_librocdxg_warns_and_points_at_the_helper(self, tmp_path):
+ """/dev/dxg alone is not enough: without the bridge library the runtime cannot
+ reach the card, and the fix is the WSL ROCm helper, not a docker flag."""
+ argv, stderr = _run_sh(tmp_path, ["--rocm", "true"], kfd = False, dxg = True)
+ assert "librocdxg" in stderr, stderr
+ assert "install_rocm_wsl_strixhalo.sh" in stderr, stderr
def test_a_mixed_host_is_not_offered_the_nvidia_toolkit(self, tmp_path):
"""An NVIDIA + AMD box under --rocm runs the ROCm image through the AMD nodes;
@@ -369,6 +424,37 @@ class TestBuildShRocm:
assert "org.opencontainers.image.licenses=Apache-2.0 AND AGPL-3.0-only" in body
+class TestTheUserFacingDocsCoverWsl:
+ """docker/DOCKERHUB-ROCM.md is synced to the Docker Hub page and README.md is the
+ first thing a Windows user reads. Both said the image needs native Linux, and the
+ Hub quick start passed --device /dev/kfd unconditionally, which the daemon rejects
+ on WSL before the entrypoint runs."""
+
+ def test_the_hub_page_gives_the_same_wsl_flags_as_run_sh(self):
+ text = open(_HUB_PAGE, encoding = "utf-8").read()
+ for needle in (
+ "--device /dev/dxg",
+ "HSA_ENABLE_DXG_DETECTION=1",
+ "librocdxg.so.1:/usr/lib/x86_64-linux-gnu/librocdxg.so:ro",
+ "-v /usr/lib/wsl/lib:/usr/lib/wsl/lib:ro",
+ "LD_LIBRARY_PATH=/usr/lib/wsl/lib",
+ "ROCM_GFX=<your gfx> bash docker/build.sh --rocm",
+ # run.sh defaults to the published image, which is refused on DXG
+ "UNSLOTH_IMAGE=unsloth-rocm:latest bash run.sh --rocm",
+ # Dockerfile.rocm maps no RDNA3 arch to a per-arch index, so the page must not
+ # promise one
+ "RDNA3 cards (`gfx1100` to `gfx1103`) have no bridge path yet",
+ ):
+ assert needle in text, needle
+
+ def test_the_readme_no_longer_says_native_linux_only(self):
+ text = open(_README, encoding = "utf-8").read()
+ assert "needs native Linux" not in text
+ assert "/dev/dxg" in text and "docker/run.sh --rocm" in text
+ assert "UNSLOTH_IMAGE=unsloth-rocm:latest" in text
+ assert "RDNA3 cards have no bridge path yet" in text
+
+
# ── entrypoint-rocm.sh ───────────────────────────────────────────────────────
@@ -377,6 +463,7 @@ def _entrypoint(
*,
kfd = True,
readable = True,
+ dxg = False,
smi_sees_gpu = True,
python_body = None,
env_extra = None,
@@ -391,6 +478,8 @@ def _entrypoint(
(dev_root / "dev" / "kfd").write_text("")
if not readable:
os.chmod(dev_root / "dev" / "kfd", 0)
+ if dxg:
+ (dev_root / "dev" / "dxg").write_text("")
_stub(
str(bindir / "rocm-smi"),
'echo "GPU[0] : GPU ID: 0x1586"\n' if smi_sees_gpu else "echo 'No AMD GPUs specified'\n",
@@ -418,6 +507,33 @@ def _entrypoint(
@_posix_shell
+def _fake_rocm_torch(
+ tmp_path,
+ libnames,
+ available = True,
+):
+ """A ROCm torch on a supported arch whose lib/ holds exactly `libnames`."""
+ fake = tmp_path / "fake"
+ (fake / "torch" / "cuda").mkdir(parents = True)
+ (fake / "torch" / "lib").mkdir()
+ for name in libnames:
+ (fake / "torch" / "lib" / name).write_text("")
+ (fake / "torch" / "__init__.py").write_text(
+ "__version__ = '2.11.0+rocm7.2'\n"
+ "class version:\n hip = '7.2.53211'\n"
+ "from . import cuda\n"
+ )
+ (fake / "torch" / "cuda" / "__init__.py").write_text(
+ "class _P:\n gcnArchName = 'gfx1201'\n"
+ f"def is_available(): return {available}\n"
+ "def device_count(): return 1\n"
+ "def get_device_name(i): return 'AMD Radeon AI PRO R9700'\n"
+ "def get_device_properties(i): return _P()\n"
+ "def is_bf16_supported(): return True\n"
+ )
+ return f'PYTHONPATH="{fake}" exec python3 "$@"\n'
+
+
class TestRocmEntrypoint:
def test_no_kfd_refuses_and_names_docker_desktop(self, tmp_path):
rc, ran, err = _entrypoint(tmp_path, kfd = False)
@@ -448,6 +564,103 @@ class TestRocmEntrypoint:
rc, ran, err = _entrypoint(tmp_path)
assert rc == 0 and ran, err
+ def test_dxg_is_accepted_when_kfd_is_absent(self, tmp_path):
+ """WSL2 never has /dev/kfd. /dev/dxg plus librocdxg is the same GPU evidence
+ install.sh gates a WSL host on, so the run must proceed, not refuse."""
+ lib = tmp_path / "rocmlib"
+ lib.mkdir()
+ (lib / "librocdxg.so.1").write_text("")
+ rc, ran, err = _entrypoint(
+ tmp_path,
+ kfd = False,
+ dxg = True,
+ env_extra = {"UNSLOTH_ROCM_DXG_LIBDIRS": str(lib)},
+ )
+ assert rc == 0 and ran, err
+ assert "DXG bridge" in err, err
+
+ def test_dxg_skips_the_rocm_smi_advice(self, tmp_path):
+ """rocm-smi reads the amdgpu sysfs, which the bridge has none of (measured in the
+ container: "Driver not initialized"), and its advice is /dev/dri and group ids,
+ neither of which exists on WSL."""
+ lib = tmp_path / "rocmlib"
+ lib.mkdir()
+ (lib / "librocdxg.so.1").write_text("")
+ rc, ran, err = _entrypoint(
+ tmp_path,
+ kfd = False,
+ dxg = True,
+ smi_sees_gpu = False,
+ env_extra = {"UNSLOTH_ROCM_DXG_LIBDIRS": str(lib)},
+ )
+ assert rc == 0 and ran, err
+ assert "--group-add" not in err and "/dev/dri" not in err, err
+
+ def test_dxg_without_the_bridge_library_refuses(self, tmp_path):
+ """/dev/dxg alone cannot reach the card: the HSA runtime needs librocdxg."""
+ rc, ran, err = _entrypoint(
+ tmp_path,
+ kfd = False,
+ dxg = True,
+ env_extra = {"UNSLOTH_ROCM_DXG_LIBDIRS": str(tmp_path / "empty")},
+ )
+ assert rc == 1 and not ran
+ assert "librocdxg" in err, err
+ # librocdxg dlopens libdxcore from WSL's lib dir: the hand-run recovery must
+ # mount it and put it on the search path, or hsa_init fails after this check.
+ assert "-v /usr/lib/wsl/lib:/usr/lib/wsl/lib:ro" in err, err
+ assert "LD_LIBRARY_PATH=/usr/lib/wsl/lib" in err, err
+ # run.sh defaults to the published image, which the bridge refuses: the
+ # recovery has to name the per-arch build, or it sends the user there
+ assert "UNSLOTH_IMAGE=unsloth-rocm:latest bash docker/run.sh --rocm" in err, err
+ assert "unsloth/unsloth-rocm:latest" not in err, err
+
+ def _dxg_with_torch(
+ self,
+ tmp_path,
+ libnames,
+ available = True,
+ ):
+ lib = tmp_path / "dxglib"
+ lib.mkdir()
+ (lib / "librocdxg.so.1").write_text("")
+ return _entrypoint(
+ tmp_path,
+ kfd = False,
+ dxg = True,
+ python_body = _fake_rocm_torch(tmp_path, libnames, available),
+ env_extra = {"UNSLOTH_ROCM_DXG_LIBDIRS": str(lib)},
+ )
+
+ def test_dxg_torch_failure_gives_bridge_advice_not_amdgpu_advice(self, tmp_path):
+ """WSL has no host amdgpu stack, so the rocm-smi / dkms / rebuild-against-the-host
+ advice is wrong there; the bridge has its own two causes (measured: the libdxcore
+ mount, and the Windows driver)."""
+ rc, ran, err = self._dxg_with_torch(
+ tmp_path,
+ ["librocprofiler-register.so"],
+ available = False,
+ )
+ assert rc == 1 and not ran, err
+ assert "libdxcore" in err and "Windows AMD driver" in err, err
+ assert "dkms" not in err and "amdgpu driver has to be" not in err, err
+
+ def test_dxg_refuses_a_torch_bundling_librocprofiler_sdk(self, tmp_path):
+ """That library enumerates GPUs from a KFD topology WSL does not have, and aborts."""
+ rc, ran, err = self._dxg_with_torch(
+ tmp_path,
+ ["librocprofiler-register.so", "librocprofiler-sdk.so"],
+ )
+ assert rc == 1 and not ran, err
+ assert "librocprofiler-sdk.so" in err, err
+ assert "UNSLOTH_IMAGE=unsloth-rocm:latest bash docker/run.sh --rocm" in err, err
+
+ def test_dxg_accepts_a_torch_carrying_only_librocprofiler_register(self, tmp_path):
+ """torch 2.11+rocm7.2 ships -register.so and runs on the bridge (measured on an
+ R9700), so matching every "rocprof" name refused a build that works."""
+ rc, ran, err = self._dxg_with_torch(tmp_path, ["librocprofiler-register.so"])
+ assert rc == 0 and ran, err
+
def test_a_failing_torch_check_stops_before_the_command(self, tmp_path):
rc, ran, _ = _entrypoint(tmp_path, python_body = "cat > /dev/null\nexit 1\n")
assert rc == 1 and not ran