Personal AI, On Personal Devices
Python
docs: add macOS installation guide for llama.cpp
krypticmouse0ce10bd97b41
diff --git a/docs/getting-started/macos.md b/docs/getting-started/macos.mdnew file mode 100644index 00000000..67a6fe1b--- /dev/null+++ b/docs/getting-started/macos.md@@ -0,0 +1,421 @@+---+title: macOS Installation Guide+description: Complete step-by-step guide to installing OpenJarvis on macOS with llama.cpp, including common pitfalls and fixes+search:+ boost: 2+---++# macOS Installation Guide++This guide walks through a complete OpenJarvis installation on macOS using **llama.cpp** as+the inference engine. It covers every step from scratch — including pitfalls not documented+elsewhere — and is suitable for both Apple Silicon and Intel Macs.++!!! tip "Prefer Ollama?"+ If you want the fastest possible setup, use [Ollama](installation.md#ollama-recommended)+ instead. This guide is for users who want to run GGUF models directly with llama.cpp,+ or who want a deeper understanding of the full stack.++---++## What You'll Install++| Tool | Purpose |+|------|---------|+| Homebrew | macOS package manager — installs everything else |+| uv | Python version and dependency manager |+| Git | Clones the OpenJarvis repo |+| Node.js | Required for the browser UI |+| Rust | Compiles the OpenJarvis security and memory extension |+| llama.cpp | Local inference engine that runs GGUF model files |+| OpenJarvis | The framework itself |+| A GGUF model | The actual AI model (downloaded separately) |++---++## Step-by-Step Installation++### Step 1 — Install Homebrew++Homebrew is the standard macOS package manager. Everything else in this guide is installed+through it.++```bash+/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"+```++If you already have Homebrew, skip this step.++---++### Step 2 — Install uv++`uv` replaces pip, virtualenv, and pyenv in one tool. OpenJarvis uses it to manage Python+versions, virtual environments, and project dependencies.++```bash+brew install uv+```++---++### Step 3 — Install Git++Git is used to clone the OpenJarvis source code. It may already be present if you have+Xcode Command Line Tools installed.++```bash+brew install git+```++---++### Step 4 — Install Node.js++Node.js is required to build and run the browser frontend. Without it you can still use+the CLI, but not the web UI.++```bash+brew install node+```++---++### Step 5 — Install Rust++OpenJarvis includes a Rust extension that provides security scanning, memory indexing,+rate limiting, and tool execution. It must be compiled from source.++```bash+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh+```++After the installer finishes, reload your shell so `rustc` is available:++```bash+source "$HOME/.cargo/env"+```++Verify:++```bash+rustc --version+```++---++### Step 6 — Install llama.cpp++llama.cpp is the inference engine that loads and runs GGUF model files. It is not a model+itself — think of it as a media player and the `.gguf` file as the content.++```bash+brew install llama.cpp+```++---++### Step 7 — Clone the OpenJarvis repo++Run this from your home directory or any neutral parent folder.++```bash+cd ~+git clone https://github.com/open-jarvis/OpenJarvis.git+cd OpenJarvis+```++!!! warning "Do not clone from inside an existing OpenJarvis folder"+ A common mistake is running `git clone` while already inside the repo, creating deeply+ nested duplicates (`OpenJarvis/OpenJarvis/OpenJarvis`). Always clone from `~` or a+ neutral parent directory.++---++### Step 8 — Pin Python to 3.12++!!! warning "Critical step — do not skip"+ OpenJarvis requires Python 3.10–3.13. Its Rust extension uses PyO3, which does not yet+ support Python 3.14. If `uv` has Python 3.14 available, it will use it by default,+ causing the Rust extension build to fail silently and resulting in ~250 test failures+ with `ModuleNotFoundError: No module named 'openjarvis_rust'`.++Pin the project to Python 3.12:++```bash+echo "3.12" > .python-version+uv python install 3.12+rm -rf .venv+uv venv+```++**Restart your terminal**, then verify:++```bash+uv run python --version+# Must show: Python 3.12.x+```++!!! tip "Why restart the terminal?"+ Without restarting, the shell may still reference the old virtual environment. This is+ the most common reason the version pin appears not to work.++---++### Step 9 — Install Python dependencies++```bash+uv sync --extra dev --extra server+```++The `--extra server` flag adds the FastAPI backend required for the browser UI.++---++### Step 10 — Build the Rust extension++This compiles the Rust extension and installs it into the virtual environment. It provides+security scanning, memory indexing, MCP tool execution, and rate limiting. This step takes+a few minutes on first run.++```bash+uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml+```++Verify it built correctly:++```bash+uv run python -c "import openjarvis_rust; print('Rust extension OK')"+```++---++### Step 11 — Install frontend dependencies++```bash+cd frontend && npm install && cd ..+```++---++### Step 12 — Download a model++OpenJarvis needs a GGUF model file to run inference. First install the Hugging Face CLI,+then download your chosen model.++```bash+uv tool install huggingface_hub+```++!!! note "The CLI command is `hf`, not `huggingface-cli`"+ When installed via `uv tool`, the Hugging Face CLI is invoked as `hf`.++=== "Qwen3 4B (~2.5 GB)"++ Faster, lower RAM requirement. Good for most everyday tasks.++ ```bash+ hf download bartowski/Qwen_Qwen3-4B-GGUF \+ --include "Qwen_Qwen3-4B-Q4_K_M.gguf" \+ --local-dir ~/models+ ```++=== "Qwen3 8B (~4.7 GB)"++ Better reasoning and instruction following. Requires more RAM.++ ```bash+ hf download bartowski/Qwen_Qwen3-8B-GGUF \+ --include "Qwen_Qwen3-8B-Q4_K_M.gguf" \+ --local-dir ~/models+ ```++!!! warning "Use the `Qwen_` prefix"+ bartowski's Qwen3 repos use the `Qwen_` prefix (e.g. `Qwen_Qwen3-4B-GGUF`). Using+ the shorter name without the prefix returns a "repository not found" error.++!!! tip "Apple Silicon vs Intel"+ On Apple Silicon, both models benefit from Metal GPU acceleration when using the MLX+ engine. On Intel, inference runs on CPU — the 4B model is recommended for speed.++---++### Step 13 — Configure OpenJarvis++Run the init command to detect your hardware and generate a config file:++```bash+uv run jarvis init+```++Then open the config and set the default model to match the filename you downloaded:++```bash+nano ~/.openjarvis/config.toml+```++Find the `default_model` line and update it, for example:++```toml+default_model = "Qwen_Qwen3-4B-Q4_K_M.gguf"+```++---++### Step 14 — Verify the installation++```bash+uv run jarvis doctor+```++A healthy setup looks like this:++```+✓ Python version 3.12.x+✓ Config file ~/.openjarvis/config.toml+✓ Config parsing Config loaded successfully+✓ Engine: llamacpp Reachable+✓ Models: llamacpp Qwen_Qwen3-4B-Q4_K_M.gguf+✓ Default model Qwen_Qwen3-4B-Q4_K_M.gguf (on llamacpp)+```++!!! note "Warnings for other engines are normal"+ The `!` warnings for engines like `ollama`, `vllm`, and `lmstudio` simply mean those+ backends are not running. You only need `llamacpp` to be reachable.++---++## Running OpenJarvis++### CLI++Start llama-server in one terminal, then run queries in another:++```bash+# Terminal 1 — start the inference engine+llama-server -m ~/models/Qwen_Qwen3-4B-Q4_K_M.gguf -c 4096 -t 8++# Terminal 2 — ask a question+cd ~/OpenJarvis+uv run jarvis ask "What is the capital of France?"+```++### Browser UI++```bash+# Terminal 1 — inference engine+llama-server -m ~/models/Qwen_Qwen3-4B-Q4_K_M.gguf -c 4096 -t 8++# Terminal 2 — backend+cd ~/OpenJarvis && uv run jarvis serve --port 8000++# Terminal 3 — frontend+cd ~/OpenJarvis/frontend && npm run dev+```++Then open [http://localhost:5173](http://localhost:5173).++### Skip typing `uv run` every time++Activate the virtual environment for your current terminal session:++```bash+source ~/OpenJarvis/.venv/bin/activate+```++Your prompt will show `(openjarvis)` when active, and you can type `jarvis ask "..."` directly.++---++## Performance Tips++These tips apply when using llama.cpp for CPU inference.++| Flag | Effect |+|------|--------|+| `-c 4096` | Reduces context window from the 32,768 default, freeing RAM for faster inference |+| `-t 8` | Uses all available CPU threads (default is only 4) — adjust to your machine's thread count |+| `Q4_K_M` quantization | Best balance of size, speed, and quality for CPU inference |++On Apple Silicon, switching to the [MLX engine](../architecture/engine.md) gives+significantly better performance than llama.cpp for most models.++---++## Common Errors++### `No such file or directory` when loading model++The path `path/to/model.gguf` in examples is a placeholder. Replace it with your actual+model path, e.g.:++```bash+llama-server -m ~/models/Qwen_Qwen3-4B-Q4_K_M.gguf+```++---++### `No module named 'openjarvis_rust'`++The Rust extension did not build correctly, or was built against the wrong Python version.++1. Confirm Python 3.12 is active: `uv run python --version`+2. Rebuild: `uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml`++If the version shows 3.14, go back to [Step 8](#step-8--pin-python-to-312).++---++### `PyO3 version error — Python 3.14 too new`++```+error: the configured Python interpreter version (3.14) is newer than+PyO3's maximum supported version (3.13)+```++PyO3 0.23.5 supports Python up to 3.13. Follow [Step 8](#step-8--pin-python-to-312) to+pin to 3.12, then delete `.venv`, recreate it, and restart your terminal before retrying.++---++### `Repository not found` when downloading model++bartowski's Qwen3 repos use the `Qwen_` prefix. Use:++```+bartowski/Qwen_Qwen3-4B-GGUF ✓+bartowski/Qwen3-4B-GGUF ✗+```++---++### `No inference engine available`++llama-server is not running. Start it in a separate terminal before running any `jarvis`+commands, and wait until you see `model loaded` in the output.++---++### Python version still shows 3.14 after recreating the venv++Close the terminal completely and reopen it. The old venv path is cached in the shell+environment and persists across commands until the session ends.++---++### `zsh: command not found: huggingface-cli`++When installed via `uv tool`, the CLI is invoked as `hf`, not `huggingface-cli`:++```bash+hf download ... # ✓+huggingface-cli download ... # ✗+```++---++## Next Steps++- [Quick Start](quickstart.md) — Run your first query and explore agents and tools+- [Configuration](configuration.md) — Customize engine hosts, model routing, memory, and more+- [Architecture](../architecture/overview.md) — Understand how OpenJarvis is structureddiff --git a/mkdocs.yml b/mkdocs.ymlindex 73869262..c8b179f9 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -143,6 +143,7 @@ nav:- Home: index.md- Getting Started:- Installation: getting-started/installation.md+ - macOS Guide: getting-started/macos.md- Quick Start: getting-started/quickstart.md- Code Snippets: getting-started/snippets.md- Configuration: getting-started/configuration.md