OpenRAG is a comprehensive, single package Retrieval-Augmented Generation platform built on Langflow, Docling, and Opensearch.
Python
ollama fedora gpu troubleshooting
April Ma0495681ce77
diff --git a/docs/docs/support/troubleshoot.mdx b/docs/docs/support/troubleshoot.mdxindex 5b284b92a..9510b2321 100644--- a/docs/docs/support/troubleshoot.mdx+++ b/docs/docs/support/troubleshoot.mdx@@ -268,4 +268,117 @@ For more information, see [Model availability and performance issues](#model-ava### Responses are good but could be better-If your model is compatible with OpenRAG, and it isn't exhibiting any obvious errors, then you can use RAG best practices to refine the response quality, such as [knowledge filters](/knowledge-filters) and prompt engineering.\ No newline at end of file+If your model is compatible with OpenRAG, and it isn't exhibiting any obvious errors, then you can use RAG best practices to refine the response quality, such as [knowledge filters](/knowledge-filters) and prompt engineering.++## Cannot detect GPU with Fedora or local Ollama++If your machine has GPU support, but OpenRAG is having problems detecting or using the GPU, you might see errors like the following:++```console+Auto-detected mode as 'legacy'+nvidia-container-cli: ldcache error+error running createRuntime hook+unrecognized runtime 'crun'+OCI runtime error+```++To troubleshoot these issues, do the following:++* Make sure your environment has a compatible version of the NVIDIA Container Toolkit.++* If you are using a local Ollama deployment, Ollama must run in a container with GPU support.++The following steps explain how to troubleshoot these issues with Fedora 43, Podman, local Ollama, and a machine that has an NVIDIA GPU:++1. Make sure your NVIDIA driver is updated to the latest version.++2. Remove old toolkit packages:++ ```bash+ sudo dnf5 remove -y nvidia-container-toolkit nvidia-container-toolkit-base libnvidia-container*+ ```++ This command doesn't remove the NVIDIA driver.+++3. Get the NVIDIA Container Toolkit package repository:++ ```bash+ sudo curl -s -o /etc/yum.repos.d/nvidia-container-toolkit.repo \+ https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo+ ```++ Version 1.14 or later is required.++4. Refresh the package cache to recognize the new repository:++ ```bash+ sudo dnf5 clean expire-cache+ sudo dnf5 update+ ```++5. Install the new toolkit:++ ```bash+ sudo dnf5 install -y nvidia-container-toolkit+ ```++ This command installs the following:++ * `libnvidia-container` version 1.14 or later+ * `nvidia-container-toolkit-base` version 1.14 or later+ * `nvidia-container-toolkit` version 1.14 or later++ These versions support Fedora 43, Podman, and `crun`.++6. Configure the runtime for Podman (`crun`):++ ```bash+ sudo nvidia-ctk runtime configure --runtime=crun+ ```++ If this command fails, the make sure you installed version 1.14 or later of the NVIDIA Container Toolkit, which supports `crun`.++7. Fix the dynamic loader cache to prevent `ldconfig` errors:++ ```bash+ sudo ldconfig+ ```++8. Restart the Podman socket:++ ```bash+ systemctl --user enable --now podman.socket+ ```++9. Test the NVIDIA hook, and make sure the output doesn't include `Auto-detected mode as 'legacy'`. The output must be empty.++ ```bash+ /usr/bin/nvidia-container-runtime-hook prestart+ ```+++10. Test the GPU with Podman, and make sure `nvidia-smi` prints your GPU information instead of an error:++ ```bash+ podman run --rm \+ --hooks-dir=/usr/share/containers/oci/hooks.d \+ --device nvidia.com/gpu=all \+ nvidia/cuda:12.3.0-base-ubuntu22.04 nvidia-smi+ ```++11. Run Ollama in a GPU-enabled container:++ ```bash+ podman run -d \+ --name ollama \+ --security-opt=label=disable \+ --hooks-dir=/usr/share/containers/oci/hooks.d \+ --device nvidia.com/gpu=all \+ -p 11434:11434 \+ -v $HOME/ollama:/root/.ollama \+ docker.io/ollama/ollama:latest+ ```++12. Start the TUI, and then start OpenRAG with GPU mode enabled.+Or, for self-managed deployments, deploy OpenRAG with the `docker-compose-gpu.yml` file.\ No newline at end of file