Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
AGENTS, AI_SETUP, INSTALL: the app and the Terminal first, install from source as the fallback
Niko122126613a44f62a
diff --git a/AGENTS.md b/AGENTS.mdindex c4726159f..c32a038e1 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -1,35 +1,82 @@# AGENTS.md-Strata runs the Qwen3.8-Flash-Next mixture-of-experts model (and its Coder, Swift 1.5 and Unsloth variants) on a-normal PC: one NVIDIA or AMD graphics card plus system RAM, on Windows or Linux. It has a C++/CUDA/HIP engine-(`src/`, `include/`), a Python server with an OpenAI- and Anthropic-compatible API and a web app (`serve/`), and a-one-click installer (`setup.py`, started by `START-HERE.bat` / `setup.sh`).+Strata runs the Qwen3.8-Flash-Next mixture-of-experts model (and its Coder, Swift 1.5, Unsloth, Gyro and NVFP4+variants, and Qwen3.6-35B-A3B) on a normal PC: one NVIDIA or AMD graphics card plus system RAM on Windows or Linux,+or an Apple Silicon Mac. Version 0.2.0-beta, engine 0.1.45.3. It has three parts:++- **The app** (`apps/desktop`, the Tauri shell; `apps/host`, the Node host and the web UI). It is for people who do+ not want a terminal. The installer has the engine inside.+- **The Terminal** (`strata.py`, `strata_cli/`, `strata_control/`): the `strata` command. It sets up a model, serves+ it and watches it. Agents use this part.+- **The engine and the server** (`src/`, `include/`, `metal/`, `sycl/` for the C++/CUDA/HIP/SYCL/Metal engine;+ `serve/` for the Python server with an OpenAI- and Anthropic-compatible API and a web page). `setup.py`, started by+ `START-HERE.bat` or `setup.sh`, is the older one-click installer from source.## Installing Strata for a user-Follow **[docs/AI_SETUP.md](docs/AI_SETUP.md)**: check the PC, pick the model by RAM, run setup non-interactively,-start and verify the server, and connect the user's apps. Never expose the server beyond `127.0.0.1` without-`--api-key`. As an alternative to shell commands, Strata's MCP server ([docs/MCP_SERVER.md](docs/MCP_SERVER.md))-offers the same steps as tools.+Follow **[docs/AI_SETUP.md](docs/AI_SETUP.md)**. The short version, for an agent that has a shell:++1. Install the Terminal. It needs no administrator rights.++ ```sh+ curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh | sh # Linux, Mac+ irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex # Windows PowerShell+ ```++ On Windows, open a new terminal afterwards so that `strata` is on the PATH.+2. Check what would happen. Nothing is changed:++ ```sh+ strata launch --plan --model q2_0+ ```++3. Set up and serve without questions. It downloads the engine for the card and the model (tens of GB), starts the+ server, prints the addresses and returns. The server keeps running.++ ```sh+ strata launch --model q2_0 --yes --no-tui+ ```++ Useful flags (all of them: `strata launch --help`): `--family`, `--model-path` (files the user already has),+ `--data-dir`, `--context 32k|64k|128k`, `--vision off|on|gpu|cpu`, `--parallel N`, `--gpu 0,1`,+ `--access local|lan|cloudflare|tailscale`, `--port`, `--api-key-env NAME`, `--new-key`, `--save-preset NAME`,+ `--output json`. `strata launch --status` shows what is serving, `strata launch --stop` stops it, and+ `strata launch --show-key` prints the API key.+4. Connect the user's apps: `strata connect` lists them, `strata connect <app> --check` sends one prompt and one tool+ call the way that app does.++Never expose the server beyond `127.0.0.1` without an API key. `--access lan` makes a key for you. The old route (clone+the repository and run `START-HERE.bat --yes` or `./setup.sh --yes`) still works and is the fallback when the+Terminal cannot run. Strata's MCP server ([docs/MCP_SERVER.md](docs/MCP_SERVER.md)) offers the same steps as tools.## Working on the code- How the engine works, every measured number, the API and all settings: [docs/DETAILS.md](docs/DETAILS.md) andthe [paper](docs/paper/Strata-Paper.pdf).-- AMD (HIP) build and validation: [docs/AMD_HIP.md](docs/AMD_HIP.md); multi-GPU: [docs/MULTI_GPU.md](docs/MULTI_GPU.md).-- Setup's own tests run without a GPU or downloads: `python tools/test_setup_<name>.py` (for example- `tools/test_setup_amd.py`, `tools/test_setup_choices.py`).-- Keep the docs' style: plain words, measured numbers with what they were measured on, no claims without a- measurement.+- The Terminal: [docs/TERMINAL.md](docs/TERMINAL.md) and [docs/TERMINAL_LAUNCH.md](docs/TERMINAL_LAUNCH.md). The app,+ the packages and the update feed: [packaging/README.md](packaging/README.md) and [packaging/UPDATES.md](packaging/UPDATES.md).+- AMD (HIP) build and validation: [docs/AMD_HIP.md](docs/AMD_HIP.md); multi-GPU: [docs/MULTI_GPU.md](docs/MULTI_GPU.md);+ Mac (Metal): [docs/MAC.md](docs/MAC.md).+- Tests: the Terminal and Control tests are in `tests/terminal` (`python -m pytest tests/terminal`). The app's UI+ tests run with `npm test` in `apps/host`. Setup's own tests run without a GPU or downloads:+ `python tools/test_setup_<name>.py` (for example `tools/test_setup_amd.py`, `tools/test_setup_choices.py`).+- Keep the docs' style: plain words, short sentences, written for non-technical users first. Give measured numbers+ with what they were measured on. Make no claim without a measurement.## Contributing a change or report-- Search the open issues and pull requests first, and add to a thread that already covers your point.-- Open an issue with the form that fits (bug report, feature request or question).-- One change per pull request. Say what it changes and what it leaves alone.+The rules are in **[CONTRIBUTING.md](CONTRIBUTING.md)**. Read it before you open an issue or a pull request, and+follow it whether a person or an agent writes the text. The short version:++- Search the open and closed issues and pull requests first, and add to a thread that already covers your point.+- Agree the scope in an issue before a large feature, a refactor or a change to a default.+- One problem per pull request. Lead with the problem and the result, then the approach, then the validation+ (passed, failed and not run, kept apart) and the limits.- A new feature is opt-in, and the default path stays byte-identical to the last release. Say how you checked.- Build every backend a file touches (CUDA, HIP, SYCL) before asking for review.- Change a default only where you measured it faster, and show the numbers with what they were measured on.+- Never submit measurements, logs or reproduction steps you did not produce. Mark a finding from reading code as+ unconfirmed static analysis.- A report from hardware the maintainers do not have is welcome. Follow[docs/COMMUNITY_BENCHMARKS.md](docs/COMMUNITY_BENCHMARKS.md), compare against a same-day run of the build you aretesting, and say what you did not test.diff --git a/docs/AI_SETUP.md b/docs/AI_SETUP.mdindex 236d7fb5d..3ce89c5c1 100644--- a/docs/AI_SETUP.md+++ b/docs/AI_SETUP.md@@ -2,147 +2,156 @@This page is written for an AI coding assistant (Claude Code, Cursor, Codex, GitHub Copilot, ...) that a user asked toset up Strata on their PC. Strata runs the Qwen3.8-Flash-Next model locally on one NVIDIA or AMD graphics card plus-system RAM, and serves an OpenAI- and Anthropic-compatible API on `http://127.0.0.1:8080`. The human-oriented-version of everything below is [INSTALL.md](INSTALL.md).+system RAM (or on an Apple Silicon Mac), and serves an OpenAI- and Anthropic-compatible API on `http://127.0.0.1:8080`.+The human-oriented version of everything below is [INSTALL.md](INSTALL.md).++You use the **Terminal**: the `strata` command. Install it with one line, then `strata launch` downloads the engine+for the card and the model, and starts the server. If the Terminal cannot run on this PC, step 9 has the older route+from source.Work through the steps in order. Tell the user what you are doing in plain words; they may not be technical.## 0. Ground rules-- **Never expose the server without an API key.** Keep the default `--host 127.0.0.1`. Only if the user asks for- access from other devices, use `--host 0.0.0.0` **together with** `--api-key <a long random secret>`, and give the- user the key. Never put a tunnel or port forward in front of a server without a key.-- Do not change the user's system beyond what setup does (setup installs Python for the user account if needed,- and everything else inside the Strata folder and its `Strata-data` folder). Ask before installing drivers.-- The model download is ~70 GB (94 GB for Unsloth's UD-IQ4_XS, 111 GB for UD-Q4_K_XL). Confirm the user is fine with that before you- start, especially on a metered connection.-- Setup is long-running (an hour or more on a slow connection). Run it in the background or with a long timeout and- poll its output; do not kill it because it is quiet for a while. It is resumable: running the same command again- continues where it stopped.+- **Never expose the server without an API key.** Keep the default (`--access local`, the server listens on+ `127.0.0.1` only). Only if the user asks for access from other devices, use `--access lan`. Strata makes a long+ random key for you then and will not start on a network without one. Give the user the key+ (`strata launch --show-key`). Never put a tunnel or port forward in front of a server without a key.+- Do not change the user's system beyond what Strata does. The Terminal installs into the user's home folder and needs+ no administrator rights. It does not install drivers, Python packages or compilers. Ask before installing drivers.+- The model download is ~70 GB (94 GB for Unsloth's UD-IQ4_XS, 111 GB for UD-Q4_K_XL). Confirm the user is fine with+ that before you start, especially on a metered connection.+- The first `strata launch` is long-running (an hour or more on a slow connection). Run it in the background or with a+ long timeout and poll its output; do not kill it because it is quiet for a while. It is resumable: running the same+ command again continues where it stopped.+- Piping a script from the internet into a shell runs code you have not read. Tell the user what the line does (step 2).+ If they prefer, download the script first, show it, and run it only after they agree.## 1. Check the PC-The quickest way is setup's own check, which prints the GPU(s), driver, RAM, CPU, free disk and whether this PC can run-Strata, and installs nothing except Python and the `.venv` it needs to run (step 2 gets the repository first):--```-Windows: START-HERE.bat --check-Linux: ./setup.sh --check-```--To check by hand:--| What | Windows (PowerShell) | Linux |-| --- | --- | --- |-| OS | `[Environment]::OSVersion` | `cat /etc/os-release` |-| NVIDIA GPU, VRAM, driver | `nvidia-smi --query-gpu=index,name,memory.total,driver_version --format=csv` | the same |-| AMD GPU | `Get-CimInstance Win32_VideoController \| Select-Object Name, AdapterRAM` (AdapterRAM is capped at 4 GB; trust the model name) | `lspci \| grep -i -E 'vga\|display'`; VRAM: `cat /sys/class/drm/card*/device/mem_info_vram_total` (bytes); `rocm-smi` only if ROCm is installed |-| RAM | `(Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory / 1GB` | `free -g` |-| Free disk | `Get-PSDrive -PSProvider FileSystem` | `df -h .` |-Requirements (details: [INSTALL.md](INSTALL.md#what-you-need)):- **GPU:** NVIDIA RTX 20, 30, 40 or 50 series, or AMD Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT,RX 9070 / 9070 XT, Radeon AI PRO R9700, RX 6800 / 6900 series; 12 GB of VRAM or more (an NVIDIA card with 8 GB runs,slowly). GTX 10 series and older, and integrated GPUs, are not supported; Pascal / Volta cards (P40, V100) and some- older AMD cards have experimental paths the user opts into ([OLDER_GPUS.md](OLDER_GPUS.md)).+ older AMD cards have experimental paths the user opts into ([OLDER_GPUS.md](OLDER_GPUS.md)). A **Mac** needs Apple+ Silicon (M1 or newer), macOS 15 or newer and 16 GB of memory or more ([MAC.md](MAC.md)).- **Driver:** NVIDIA 580 or newer. AMD on Linux: the kernel's amdgpu driver; on Windows: a current AMD Adrenalindriver. If the driver is missing or too old, tell the user to update it (NVIDIA App / nvidia.com/drivers, or AMDSoftware) and restart; do not install drivers yourself unless they ask.-- **RAM:** 32 GB or more (see step 3). **Disk:** ~80 GB free, ideally on an NVMe SSD. **CPU:** x86-64 with AVX2. Without AVX2 (Xeon E5 v1/v2 and older) setup still installs, as an experimental and slow build it compiles on the PC (10-20 minutes): tell the user that before starting ([INSTALL.md](INSTALL.md#older-cpus-experimental)).-- **OS:** Windows 10/11 or Linux (Ubuntu 22.04/24.04 and Debian 12/13 are fully automatic; Debian 13 gets the oldest CUDA 13 its NVIDIA repo has, currently 13.1).+- **RAM:** 32 GB or more (see step 3). **Disk:** ~80 GB free, ideally on an NVMe SSD. **CPU:** x86-64 with AVX2.+ Without AVX2 (Xeon E5 v1/v2 and older) the ready-made engine refuses to start; the route from source builds an+ experimental and slow engine on the PC (10-20 minutes): tell the user that before starting+ ([INSTALL.md](INSTALL.md#older-cpus-experimental)).+- **OS:** Windows 10/11, Linux x86-64 (Ubuntu 22.04/24.04 and Debian 12/13 are fully automatic), or macOS on Apple Silicon.++To check by hand:++| What | Windows (PowerShell) | Linux | Mac |+| --- | --- | --- | --- |+| OS | `[Environment]::OSVersion` | `cat /etc/os-release` | `sw_vers` |+| NVIDIA GPU, VRAM, driver | `nvidia-smi --query-gpu=index,name,memory.total,driver_version --format=csv` | the same | - |+| AMD GPU | `Get-CimInstance Win32_VideoController \| Select-Object Name, AdapterRAM` (AdapterRAM is capped at 4 GB; trust the model name) | `lspci \| grep -i -E 'vga\|display'`; VRAM: `cat /sys/class/drm/card*/device/mem_info_vram_total` (bytes); `rocm-smi` only if ROCm is installed | - |+| RAM | `(Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory / 1GB` | `free -g` | `sysctl -n hw.memsize` (bytes) |+| Free disk | `Get-PSDrive -PSProvider FileSystem` | `df -h .` | `df -h ~` |If the PC does not meet them, say which part is missing and stop.-## 2. Get Strata+## 2. Install the Terminal++One line. It needs no administrator rights. It checks what kind of PC this is, downloads the Terminal archive, checks+its SHA-256, unpacks it into the user's home folder, checks the release's signed update feed, and puts `strata` on the+PATH.```-git clone https://github.com/Niko1221/Strata.git+Linux, Mac: curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh | sh+Windows (PowerShell): irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex```-Without git: download https://github.com/Niko1221/Strata/archive/refs/heads/main.zip and unzip it. Pick a drive with-enough free space: the model files go to a `Strata-data` folder **next to** the Strata folder (or `--data-dir <path>`-on another drive).+On Windows, open a new terminal window afterwards (the PATH changed). Check it: `strata --version`.## 3. Pick the model-By the PC's RAM (ask the user whether they mainly want it for code - then the Coder is a good pick at any RAM size):+`strata launch --plan` shows what Strata would do and changes nothing. With no `--model` it picks the model for this+PC's RAM and prints the exact command line (`same as strata launch --model ...`). Use it as the default, or pick+yourself by the PC's RAM (ask the user whether they mainly want it for code - then the Coder is a good pick at any RAM+size):-| RAM | `--family` / `--model` | Notes |+| RAM | `--model` (and `--family`) | Notes || --- | --- | --- |-| 32 GB | `--family coder` (size IQ1_M) | for code; with a 24 GB card Q2_0 and IQ2_XS also run (setup picks the low-RAM mode) |-| 48 GB | `--family qwen --model IQ2_XS` | or `Q2_0` (fastest) |-| 64 GB | `--family qwen --model IQ2_XS` (recommended) | `IQ3_XXS` / `IQ3_S` are slower and a bit better |-| 96 GB+ | `--family qwen --model IQ3_S` | `--family unsloth --model UD-IQ4_XS` (~4-bit, 94 GB, part of the experts read from the SSD under ~80 GB of RAM); `--model UD-Q4_K_XL` is experimental: NVIDIA only, NVMe SSD, 7-8.5 tokens/s on 64 GB |--`--family swift` (Swift 1.5, a fine-tune that thinks shorter; sizes Q2_0, IQ2_XS, IQ3_XXS) is the alternative to-`qwen`. With `--yes` and no `--model`, setup picks the recommended size for the RAM itself. More: [MODELS.md](MODELS.md).+| 32 GB | `--family coder --model iq1_m` | for code; with a 24 GB card Q2_0 and IQ2_XS also run (Strata picks the low-RAM mode) |+| 48 GB | `--model iq2_xs` | or `q2_0` (fastest) |+| 64 GB | `--model iq2_xs` (recommended) | `iq3_xxs` / `iq3_s` are slower and a bit better |+| 96 GB+ | `--model iq3_s` | `--family unsloth --model ud-iq4_xs` (~4-bit, 94 GB, part of the experts read from the SSD under ~80 GB of RAM); `ud-q4_k_xl` is experimental: NVIDIA only, NVMe SSD, 7-8.5 tokens/s on 64 GB |-## 4. Run setup without questions+`--family swift` (Swift 1.5, a fine-tune that thinks shorter; sizes IQ2_XS, IQ3_XXS, IQ3_S) is the+alternative to `qwen`. A model name is not case sensitive. More: [MODELS.md](MODELS.md).-From the Strata folder (Windows: in `cmd`, or `cmd /c START-HERE.bat ...` from PowerShell):+## 4. Set up and serve, without questions```-Windows: START-HERE.bat --yes --family qwen --model IQ2_XS --no-start-Linux: ./setup.sh --yes --family qwen --model IQ2_XS --no-start+strata launch --plan --model iq2_xs # what would happen; nothing is changed+strata launch --model iq2_xs --yes --no-tui # do it: download the engine and the model, start the server```-The flags (all of them: `START-HERE.bat --help`):+With `--no-tui` and `--yes` it asks nothing, prints the addresses when the server is ready, and returns. **The server+keeps running** in the background. `strata launch` with no `--model`, `--preset` or other choice needs a terminal for+its screens, so an agent always names the model. Add `--output json` for output a script can read.++The flags (all of them, with examples: `strata launch --help`):| Flag | Meaning || --- | --- |-| `--yes` | take the recommended answer to every question (no prompts) |-| `--family qwen\|swift\|coder\|unsloth` | the model version |-| `--model Q2_0\|IQ2_XS\|IQ3_XXS\|IQ3_S\|IQ1_M\|UD-IQ4_XS\|UD-Q4_K_XL` | the size (the Coder is IQ1_M, Unsloth UD-IQ4_XS or UD-Q4_K_XL) |-| `--context N` | context in tokens; default by VRAM: 32768 under 14 GB, 65536 under 20 GB, else 131072 |-| `--vision yes\|no\|gpu\|cpu` | read pictures; `--yes` leaves images off. AMD cards: `cpu` |-| `--gpu N` / `--gpus 0,1` / `--gpus all` | one card, or several sharing the model (default: the card with the most VRAM) |-| `--backend cuda\|hip` | NVIDIA or AMD engine; chosen by itself on a PC with only one kind of card |-| `--data-dir PATH` | where the 70-120 GB of model files go |+| `--model NAME` | the size: `q2_0`, `iq2_xs`, `iq3_xxs`, `iq3_s`, `iq1_m`, `ud-iq4_xs`, `ud-q4_k_xl`, ... (the error message for a wrong name lists them) |+| `--family NAME` | the model version when several have that size: `qwen`, `swift`, `coder`, `unsloth` |+| `--model-path PATH` | files the user already has: a folder of shards or one `.gguf` file |+| `--data-dir DIR` | where the 70-120 GB of model files go |+| `--context SIZE` | `32k`, `64k`, `128k`, `256k` or a number of tokens |+| `--vision off\|on\|gpu\|cpu` | read pictures. On by default when the model can read them: the reader runs on the CPU for a card under 24 GB and on the GPU from 24 GB, on every make of card. `off` turns it off |+| `--parallel N` | requests answered at the same time, 1 to 8 (default: fit to the card) |+| `--gpu LIST` | card numbers, e.g. `0` or `0,1` (several cards share the model) |+| `--tune quick\|thorough\|skip` | find the fastest settings first (default here: `skip`) |+| `--access local\|lan\|cloudflare\|tailscale` | who can connect; beyond `local` a key is made for you || `--port N` | the server port (default 8080) |-| `--host H --api-key K` | listen beyond this PC; **only together with a key** |-| `--no-start` | install only, do not start the server |-| `--setup` | install another model or change settings of an installed one |-| `--check` | only check the PC |--`--no-start` is recommended for agents: the server runs in the foreground until its window is closed, which would-block your shell. Start it separately in step 6.+| `--api-key KEY`, `--api-key-env NAME`, `--new-key` | the key clients must send |+| `--save-preset NAME` | keep these choices; `strata launch --preset NAME` starts them again |+| `--plan` | show what would happen and exit |+| `--yes`, `--no-tui` | no questions, no screens |+| `--status`, `--stop`, `--show-key` | what is serving, stop it, print the key |Notes:-- **Linux:** setup uses `sudo apt` (or dnf/pacman) to install Python with venv if it is missing, and on AMD may need- `build-essential` and `git`. You cannot type the user's password: if a `sudo` step is needed, ask the user to run it- (e.g. `sudo apt install python3-venv build-essential git`) and then rerun setup.-- **AMD on Linux:** setup installs ROCm into `.venv` (~10 GB, no sudo) unless a system ROCm 7 exists, and compiles the- engine for the card (10-20 minutes, once).-- **AMD on Windows (new in 0.1.34):** setup downloads the ready-made AMD engine (~550 MB, ROCm included) - nothing- is compiled and only the AMD driver is needed. Before the model download it runs `engine\strata-device.exe- --list-devices`; if that does not list the card, the driver is the problem (tell the user to update AMD Software).- One card per model and no images on Windows for now. It is new: ask the user to report how it runs (docs/AMD_HIP.md).-- **NVIDIA with no ready-made engine for the card:** setup offers to install build tools and compile (20-40 minutes);- `--yes` accepts.+- **No engine for this PC.** `strata launch` does not refuse. It says "Strata couldn't find an engine for this computer",+ goes on to download the model (with `--yes` it does not ask), and ends without starting anything (exit code 0; with `--output json` the answer+ has `"serving": false` and an `engine_problem`). Tell the user which card was not recognised and stop. Updating the+ engine later with `strata update` can fix it.+- **Exit codes:** 0 done, 2 a flag or name is wrong, 5 the Terminal or the engine cannot do it, 6 an install or the+ server start failed (the engine log is named in the message), 130 stopped.+- **Linux:** nothing needs `sudo`: the Terminal does not install packages. **Windows:** the first start may ask once+ whether to allow Strata on the network if you used `--access lan`.+- **AMD on Windows:** only the AMD driver is needed. The AMD engine is a ready-made download. It has not been run with a+ model on a real AMD card on Windows by us: ask the user to report how it runs ([AMD_HIP.md](AMD_HIP.md)).+- **Mac:** the Metal engine is fetched like any other. There is no picture reader on the Mac yet.## 5. While it downloads, tell the user- It downloads about 70 GB from Hugging Face; how long depends on their connection (at 100 Mbit/s roughly 1.5-2hours). It can be stopped and continues where it left off.-- Then it prepares the model for their PC (a few minutes) and, on AMD on Linux, compiles the engine.+- Then it prepares the model for their PC (a few minutes).- **When the model starts, the PC can be slow or stop responding for 1-3 minutes** (longest the first time): Strataloads 35-55 GB into RAM and locks part of it for the graphics card. That is normal; they should not close the window.- Nothing leaves their PC: the model runs locally.-## 6. Start the server+## 6. Check that it is serving-Setup prints the start script it wrote (`start script: run-<model>.bat`). The name is the family tag plus the size,-lower case: `run-iq2_xs`, `run-swift-iq2_xs`, `run-coder-iq1_m`, `run-unsloth-ud-iq4_xs`.+`strata launch` prints `Strata is serving.` with the addresses when it is done. Later:```-Windows (PowerShell): Start-Process -FilePath ".\run-iq2_xs.bat" (opens its own window)-Linux: nohup ./run-iq2_xs.sh > strata-server.out 2>&1 &+strata launch --status # lists what is serving, with its address+strata launch --stop # stops it and frees the graphics memory+strata launch --preset NAME # starts a saved preset again (after --save-preset NAME)```-Or `START-HERE.bat` / `./setup.sh` without flags, which starts the installed model (it asks which one when several are-installed; add `--yes` to take the first). The server opens the user's browser on `http://127.0.0.1:8080` when it is-ready. Closing its window (or stopping the process) stops the model.+No browser opens by itself. Add `--open` to `strata launch` if the user wants the web page.## 7. Verify@@ -160,38 +169,115 @@ curl http://127.0.0.1:8080/v1/modelscurl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\": \"strata\", \"messages\": [{\"role\": \"user\", \"content\": \"Say hello in five words.\"}], \"max_tokens\": 64, \"reasoning_effort\": \"none\"}"```-With an API key, add `-H "Authorization: Bearer <key>"`. The server window prints a `ready: http://127.0.0.1:8080/v1`-line when it is ready; the engine log is `strata-<model>.log` in the Strata folder.+With an API key, add `-H "Authorization: Bearer <key>"`.## 8. Connect the user's apps- **Browser:** `http://127.0.0.1:8080` - Chat, a live Monitor, and About (settings and addresses).- **Any OpenAI-compatible app or agent:** base URL `http://127.0.0.1:8080/v1`, any API key (or the configured one),any model name.-- **Anthropic-compatible apps:** `http://127.0.0.1:8080/v1/messages`.+- **Anthropic-compatible apps:** base URL `http://127.0.0.1:8080`, **without `/v1`**. Those apps add `/v1/messages`+ themselves.- **Codex CLI and other Responses API apps:** `http://127.0.0.1:8080/v1/responses` (stateless; Codex's`config.toml`: [DETAILS.md](DETAILS.md#the-responses-api-and-codex-cli)).- **Claude Code:** `ANTHROPIC_BASE_URL=http://127.0.0.1:8080`, `ANTHROPIC_MODEL` set to a Claude model name it knows(Strata ignores the name), and any `ANTHROPIC_AUTH_TOKEN` (or the configured key).+- **The easy way for a coding app:** `strata connect` lists them (OpenCode, Codex CLI, Claude Code, oh-my-pi, pi,+ Hermes Agent, OpenClaw). `strata connect <app>` prints the file and environment for the running server,+ `strata connect <app> --write` writes it after asking (and keeps a backup), `strata connect <app> --check` sends+ one prompt and one tool call the way that app does and says whether each worked.- **Thinking level:** `"reasoning_effort": "none" | "low" | "medium" | "high"` (default high).-- Strata answers one request at a time; `"parallel": N` in the model's config (or setup `--parallel N`) decodes up to- N together ([BATCHING.md](BATCHING.md)). API details: [DETAILS.md](DETAILS.md#using-it).+- Strata answers one request at a time; `--parallel N` in `strata launch` decodes up to N together+ ([BATCHING.md](BATCHING.md)). API details: [DETAILS.md](DETAILS.md#using-it).++## 9. If the Terminal cannot run: install from source++Use this only when step 2 does not work on the PC (for example a CPU without AVX2, or no internet access to GitHub+Releases). It needs git or the zip, and on Linux `sudo` for Python and build tools.++```+git clone https://github.com/Niko1221/Strata.git+```++Without git: download https://github.com/Niko1221/Strata/archive/refs/heads/main.zip and unzip it. Pick a drive with+enough free space: the model files go to a `Strata-data` folder **next to** the Strata folder (or `--data-dir <path>`+on another drive). Setup's own check prints the GPU(s), driver, RAM, CPU, free disk and whether this PC can run+Strata, and installs nothing except Python and the `.venv` it needs to run:++```+Windows: START-HERE.bat --check+Linux: ./setup.sh --check+```++Run setup without questions, from the Strata folder (Windows: in `cmd`, or `cmd /c START-HERE.bat ...` from PowerShell).+The model is picked as in step 3, with setup's flag names (`--family qwen|swift|coder|unsloth --model Q2_0|IQ2_XS|...`):++```+Windows: START-HERE.bat --yes --family qwen --model IQ2_XS --vision yes --no-start+Linux: ./setup.sh --yes --family qwen --model IQ2_XS --vision yes --no-start+```++The flags (all of them: `START-HERE.bat --help`):++| Flag | Meaning |+| --- | --- |+| `--yes` | take the recommended answer to every question (no prompts) |+| `--family qwen\|swift\|coder\|unsloth` | the model version |+| `--model Q2_0\|IQ2_XS\|IQ3_XXS\|IQ3_S\|IQ1_M\|UD-IQ4_XS\|UD-Q4_K_XL` | the size (the Coder is IQ1_M, Unsloth UD-IQ4_XS or UD-Q4_K_XL) |+| `--context N` | context in tokens; default by VRAM: 32768 under 14 GB, 65536 under 20 GB, else 131072 |+| `--vision yes\|no\|gpu\|cpu` | read pictures. Pass `yes` to get the best reader for the card (the CPU under 24 GB, the GPU from 24 GB, any make); `no` turns them off |+| `--gpu N` / `--gpus 0,1` / `--gpus all` | one card, or several sharing the model (default: the card with the most VRAM) |+| `--backend cuda\|hip` | NVIDIA or AMD engine; chosen by itself on a PC with only one kind of card |+| `--data-dir PATH` | where the 70-120 GB of model files go |+| `--port N` | the server port (default 8080) |+| `--host H --api-key K` | listen beyond this PC; **only together with a key** |+| `--no-start` | install only, do not start the server |+| `--setup` | install another model or change settings of an installed one |+| `--check` | only check the PC |++`--no-start` is recommended for agents: the server runs in the foreground until its window is closed, which would+block your shell. Start it separately:++```+Windows (PowerShell): Start-Process -FilePath ".\run-iq2_xs.bat" (opens its own window)+Linux: nohup ./run-iq2_xs.sh > strata-server.out 2>&1 &+```++Setup prints the start script it wrote (`start script: run-<model>.bat`). The name is the family tag plus the size,+lower case: `run-iq2_xs`, `run-swift-iq2_xs`, `run-coder-iq1_m`, `run-unsloth-ud-iq4_xs`. The engine log is+`strata-<model>.log` in the Strata folder. Then continue with step 7.++Notes:+- **Linux:** setup uses `sudo apt` (or dnf/pacman) to install Python with venv if it is missing, and on AMD may need+ `build-essential` and `git`. You cannot type the user's password: if a `sudo` step is needed, ask the user to run it+ (e.g. `sudo apt install python3-venv build-essential git`) and then rerun setup.+- **AMD on Linux:** setup installs ROCm into `.venv` (~10 GB, no sudo) unless a system ROCm 7 exists, and compiles the+ engine for the card (10-20 minutes, once).+- **AMD on Windows:** setup downloads the ready-made AMD engine (~550 MB, ROCm included) - nothing is compiled and only+ the AMD driver is needed. Before the model download it runs `engine\strata-device.exe --list-devices`; if that does+ not list the card, the driver is the problem (tell the user to update AMD Software). One card per model. Pictures use+ the 24 GB rule like everywhere else.+- **NVIDIA with no ready-made engine for the card:** setup offers to install build tools and compile (20-40 minutes);+ `--yes` accepts.+- **Mac:** `./setup.sh` builds the Metal engine on the Mac (needs Apple's Command Line Tools, `xcode-select --install`).-## 9. When something fails+## 10. When something fails| Symptom | What to do || --- | --- |-| `the NVIDIA driver is too old` | The user updates the driver (NVIDIA App or nvidia.com/drivers) and restarts; rerun setup. |-| Download or install stopped | Rerun the same setup command; it continues. |-| `port 8080 is already in use` | Strata is already running (check `/health`), or use `--port 8081`. |-| Very slow, disk busy, or "the engine stopped unexpectedly" | Not enough free RAM: close programs, or set up a smaller size (`--setup --model Q2_0` or IQ2_XS). |+| `the NVIDIA driver is too old` | The user updates the driver (NVIDIA App or nvidia.com/drivers) and restarts; run the command again. |+| Download or install stopped | Run the same command again; it continues. |+| `Strata couldn't find an engine for this computer` | The card is not one the engine packages cover, or the update feed was not reachable. Check the card against the list in step 1, check the internet, run `strata doctor`, then `strata update`. |+| `port 8080 is already in use` | Strata is already running (check `/health` and `strata launch --status`), or use `--port 8081`. |+| Very slow, disk busy, or "the engine stopped unexpectedly" | Not enough free RAM: close programs, or choose a smaller size (`q2_0` or `iq2_xs`). || `ExpertCache: cudaMalloc(...) failed: out of memory` with free VRAM (Windows) | The page file is off or tiny: set a fixed size (Custom size, the same initial and maximum size, e.g. 65536 MB) and restart; a page file Windows grows on demand ("System managed") may not grow in time (#60). |-| `prompt ... exceeds the context` | Rerun setup with `--setup --context <bigger>`. |+| `prompt ... exceeds the context` | Start again with a bigger `--context`. || No AMD GPU found (Linux) | The amdgpu driver lists no GPU; check the card and driver. Integrated GPUs are not supported. |-| Python or build tools could not be installed | Install what the message names (links are printed), then rerun. |+| A Mac says the app is "damaged" (the app, not the Terminal) | The file is fine: [MAC.md](MAC.md) has the one command. |More: [TROUBLESHOOTING.md](TROUBLESHOOTING.md) and the [full table](DETAILS.md#troubleshooting). If it still fails,-collect `strata-<model>.log` and the setup output, and suggest an issue at https://github.com/Niko1221/Strata/issues.+run `strata doctor`, collect its output and the engine log named in the error message, and suggest an issue at+https://github.com/Niko1221/Strata/issues.## Alternative: the MCP serverdiff --git a/docs/INSTALL.md b/docs/INSTALL.mdindex c9c6c888a..58cfe8ecc 100644--- a/docs/INSTALL.md+++ b/docs/INSTALL.md@@ -1,10 +1,18 @@# Installing Strata-Everything about installing, starting, updating and configuring Strata, on Windows and Linux, with an NVIDIA or an-AMD graphics card. The short version is in the [README](../README.md#install); an AI coding assistant can do all of-this for you with [AI_SETUP.md](AI_SETUP.md).+Everything about installing, starting, updating and configuring Strata, on Windows, Linux and Mac, with an NVIDIA+or an AMD graphics card or an Apple Silicon Mac. The short version is in the [README](../README.md#install). An AI+coding assistant can do all of this for you with [AI_SETUP.md](AI_SETUP.md).-> **On this page:** [What you need](#what-you-need) · [Windows](#windows) · [Linux](#linux) ·+There are three ways to install it. Take the first one unless you have a reason not to:++1. **[The app](#the-app):** a desktop program with the engine inside. Install it, pick a model, chat.+2. **[The Terminal](#the-terminal):** one line in a terminal, then `strata launch`. For scripts, servers and AI agents.+3. **[From source](#from-source-windows):** `START-HERE.bat` on Windows, `./setup.sh` on Linux and Mac. It builds or+ fetches the engine itself.++> **On this page:** [What you need](#what-you-need) · [The app](#the-app) · [The Terminal](#the-terminal) ·+> [From source: Windows](#from-source-windows) · [Linux](#from-source-linux) · [Mac](#from-source-mac) ·> [AMD cards](#amd-cards) · [Several cards](#two-or-three-cards) · [Docker](#docker-linux) ·> [Older CPUs](#older-cpus-experimental) · [Updating](#updating) · [Where things are stored](#where-things-are-stored) ·> [Setup's questions](#setups-questions) · [Tuning](#tuning-for-your-pc) · [All options](#options-without-questions)@@ -16,13 +24,15 @@ this for you with [AI_SETUP.md](AI_SETUP.md).| **GPU** | **NVIDIA** RTX 20, 30, 40 or 50 series, **12 GB VRAM or more** (8 GB runs, slowly). Measured on an RTX 5070 and an RTX 3090; RTX 20 (Turing, since 0.1.27) was tested by a contributor on an RTX 2070. **AMD** Radeon RX 7900 XT / XTX, RX 9070 / 9070 XT and Radeon AI PRO R9700 (validated), RX 7800 XT / 7700 XT and RX 9060 XT (validated by their owners), RX 6800 / 6900 series (community-reported), with 12 GB of VRAM or more. See [AMD cards](#amd-cards). || **RAM** | Enough for the size you pick ([which model](MODELS.md#pick-by-ram)); **64 GB** runs every size. A big GPU makes up for less RAM - the [low-RAM mode](MODELS.md#a-big-graphics-card-and-little-ram). || **CPU** | x86-64 with AVX2 (any Intel/AMD desktop CPU from the last ~8 years). AVX-512 (Ryzen 7000/9000) is a bit faster. Older CPUs without AVX2 are experimental and slow: [Older CPUs](#older-cpus-experimental). |-| **Disk** | ~70-80 GB free for the model, ~6 GB for the MTP layer (+1 GB with images). **Q2_0 on an AVX-512 CPU** also writes a one-time ~40 GB copy of its experts for the fast CPU kernel. On Linux with an AMD card, ROCm takes ~10 GB more when setup installs it. An NVMe SSD is strongly recommended: it makes the first start much faster. |-| **OS** | Windows 10/11, or Linux (Ubuntu 22.04/24.04 and Debian 12/13 get everything installed automatically). |+| **Disk** | The app and its engine take about 1-3 GB. ~70-80 GB free for the model, ~6 GB for the MTP layer (+1 GB with images). **Q2_0 on an AVX-512 CPU** also writes a one-time ~40 GB copy of its experts for the fast CPU kernel. On Linux with an AMD card, ROCm takes ~10 GB more when setup installs it. An NVMe SSD is strongly recommended: it makes the first start much faster. |+| **OS** | Windows 10/11, Linux (Ubuntu 22.04/24.04 and Debian 12/13 get everything installed automatically; the app also runs on Fedora 44), or a Mac with Apple Silicon (M1 or newer) and macOS 15 or newer ([Mac](MAC.md)). || **Driver** | **NVIDIA:** a current driver, version 580 or newer ([nvidia.com/drivers](https://www.nvidia.com/drivers) or the NVIDIA App). **AMD:** on Linux the kernel's amdgpu driver (no ROCm install needed); on Windows a current AMD Software: Adrenalin Edition driver ([amd.com/support](https://www.amd.com/en/support)). |-The driver is the only thing you install yourself. Everything else - Python, the engine, the model - is set up for-you the first time: Python 3.12 if you have none (for your user account, no admin), a private Python environment in-`.venv/`, the Strata engine, the model and the MTP draft layer. On NVIDIA it uses the ready-made engine for RTX+The driver is the only thing you install yourself. With the app, the engine is inside the installer. With the+Terminal, `strata launch` fetches the engine for your card from the signed update feed, and nothing is compiled.+Installing from source sets up the rest as well, the first time: Python 3.12 if you have none (for your user+account, no admin), a private Python environment in `.venv/`, the Strata engine, the model and the MTP draft layer.+On NVIDIA it uses the ready-made engine for RTX20/30/40/50 and NVIDIA's CUDA libraries from pip (~0.4 GB); if no ready-made engine fits your PC, it offers to installthe build tools (Visual Studio Build Tools + CUDA Toolkit on Windows, `build-essential` + CUDA on Ubuntu) and compilesthe engine for your GPU (asks first; 20-40 minutes once). More: [details](DETAILS.md#before-you-start).@@ -32,7 +42,79 @@ engine or image encoder, it uses existing CMake/Ninja tools when available and i`requirements-build.txt` into `.venv`. An older `.venv` keeps any build helpers it already has; setup does not removepackages during an update.-## Windows+## The app++The app is the easiest way. It has the engine inside, so there is nothing to build and no setup window.++1. Download the installer for your system from the [releases page](https://github.com/Niko1221/Strata/releases/latest).+ The page says which installer is for which graphics card (NVIDIA or AMD). Take the one that matches yours.+ - **Windows 10 / 11:** run the `.exe`. It installs for your user and needs no administrator rights. Windows may+ say "Windows protected your PC", because the installer is not code-signed yet. Choose **More info**, then+ **Run anyway**.+ - **Linux:** the `.AppImage`. Make it runnable (right-click, Properties, Permissions, "Allow executing file as+ program", or `chmod +x`), then double-click it. It needs FUSE 2: `sudo apt install libfuse2` on Ubuntu 22.04,+ `sudo apt install libfuse2t64` on Ubuntu 24.04 and newer, `sudo dnf install fuse fuse-libs` on Fedora. Without+ FUSE you can still start it with `--appimage-extract-and-run` after the file name. The NVIDIA build needs+ driver 580 or newer.+ - **Mac:** the `.dmg`. Open it, drag Strata onto Applications, then open Strata from there. It needs Apple+ Silicon and macOS 15 or newer. If macOS says the app is "damaged", the file is fine: see [MAC.md](MAC.md).+2. Start Strata. The first start is a little slower on Linux and Mac, because the engine inside the app is copied+ into your home folder.+3. The first screen reads your PC and recommends a model that fits. Press the install button. The download is+ big (40-70 GB). If it stops, it continues where it stopped.+4. Chat. **Closing the window stops Strata and the engine.**++**Update:** Settings, then Updates, then **Check for updates**. The buttons **Update Strata** and **Update engine**+appear only when something is new. Your models and saved settings are kept. Unload the model first: an engine that+is running cannot be replaced. **Roll back** in the same place brings the previous engine back.++**Where things are:** the app's settings and chats are in the `.strata` folder in your user folder, the engine+in `%LOCALAPPDATA%\Strata\components` (Linux and Mac: `~/.local/share/strata/components`), and the models in the+`Strata` folder in your Documents (you can change the model folder in the app). Removing the app does not remove+them. **For a bug report:** Settings, Engine, Diagnostics, View logs, **Save logs**.++The app has no picture reader on the Mac yet. Everything else about pictures is in [Setup's questions](#setups-questions).++## The Terminal++The Terminal is the `strata` command. It uses the same engine, models and saved settings as the app, and it is the+way for scripts, servers and AI agents. Linux (x86-64), Mac (Apple Silicon) and Windows (x64) are supported.++Linux and Mac:++```sh+curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh | sh+```++Windows (PowerShell):++```powershell+irm https://github.com/Niko1221/Strata/releases/latest/download/install.ps1 | iex+```++The installer finds out which computer this is, downloads the Terminal archive, checks its SHA-256, unpacks it into+your home folder (`~/.local/share/strata/terminal`, or `%LOCALAPPDATA%\Strata\terminal` on Windows; no+administrator rights), checks the release's signed update feed with the Terminal it just unpacked (if that check+fails it removes what it unpacked), and puts `strata` on your PATH. On Windows, open a new terminal window+afterwards. Then:++```sh+strata launch # guided screens: model, context, pictures, who may connect, then serve+strata launch --model q2_0 --yes --no-tui # the same without questions (scripts, agents)+```++`strata launch` downloads the engine for your graphics card (signed, and test-started before it is used) and the+model, starts serving on `http://127.0.0.1:8080` and shows the monitor. It never opens a browser unless you add+`--open`. Its flags and screens: [TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md).++**Update:** `strata update --check` says what would change, `strata update` installs the new engine (it is test-started+first and switched only if that works; the previous engine is kept), `strata components rollback` goes back. It+refuses while a model is being served. **Check your PC and the installed engine:** `strata doctor`.++To read the install script before you run it, download it first (`curl -fsSL <address> -o install.sh`), read it,+then run `sh install.sh`.++## From source: Windows1. [Download this project](https://github.com/Niko1221/Strata/archive/refs/heads/main.zip) and unzip it (or`git clone` it).@@ -51,7 +133,7 @@ downloaded twice. Close its window to stop the model. `SETUP.bat` (the same as `another model or changes the settings. Starting Strata from Task Scheduler at logon needs two task settings, or thestart is 24x slower: [Running it at startup](DETAILS.md#running-it-at-startup-task-scheduler).-## Linux+## From source: Linux```bash./setup.sh@@ -68,9 +150,16 @@ On Nix/NixOS, you can use the flake:nix develop -c ./setup.sh```+## From source: Mac++On an Apple Silicon Mac with macOS 15 or newer, install Apple's Command Line Tools once (`xcode-select --install`),+then run `./setup.sh` in the Strata folder. It builds the Mac (Metal) engine on your Mac and asks the same+questions. The app and the Terminal do not build anything. All of it, and what fits which memory size, is in+[MAC.md](MAC.md).+## AMD cards-The steps are the same as with NVIDIA: `START-HERE.bat` on Windows, `./setup.sh` on Linux. Setup finds the Radeon+The app and the Terminal pick the AMD engine for a Radeon card by themselves. From source the steps are the same as with NVIDIA: `START-HERE.bat` on Windows, `./setup.sh` on Linux. Setup finds the Radeoncard and chooses the AMD (HIP) engine by itself on a PC with no NVIDIA card Strata can use; `--backend hip` choosesit on a PC that has both. Supported cards: RX 7900 XT / XTX (gfx1100), RX 7800 XT / 7700 XT (gfx1101), RX 9060 XT(gfx1200), RX 9070 / 9070 XT and Radeon AI PRO R9700 (gfx1201), and the RX 6800 / 6900 series (gfx1030). Integrated@@ -134,7 +223,13 @@ any PC (to try it). Problems and results on real hardware are welcome as GitHub## Updating-**`UPDATE.bat`** (Linux: `./update.sh`) updates Strata without starting the model - for when the GPU is busy with+**The app:** Settings, Updates, **Check for updates**, then **Update Strata** or **Update engine** (see [the app](#the-app)).++**The Terminal:** `strata update --check`, then `strata update` (see [the Terminal](#the-terminal)). Add `--app` to+download the app's installer too. Models and saved tuning presets are never downloaded, replaced or deleted by an+update. How the signed update feed works: [packaging/UPDATES.md](../packaging/UPDATES.md).++**From source: `UPDATE.bat`** (Linux: `./update.sh`) updates Strata without starting the model - for when the GPU is busy withsomething else, or you just want the new version ready. In a `git clone` it runs `git pull`, then does what`START-HERE.bat` does before a start: the engine (a new ready-made one when the new version needs it; on Linux acompiled engine is compiled again when its source changed), the Python packages, and each installed model's settings@@ -172,15 +267,21 @@ says so and starts the engine you had.## Setup's questions+These are the questions of the installer from source (`START-HERE.bat`, `./setup.sh`). The app's first screen and+`strata launch` ask the same things in their own look.+- **Which model and size?** The original, Swift 1.5, the Coder or Unsloth's 4-bit, and Q2_0, IQ2_XS, IQ3_XXS orIQ3_S - see [which model](MODELS.md).- **How much context?** How much text it can keep in mind at once (it suggests one for your card). 384K and512K (experimental) extend the model past its trained 262K by rope scaling - the setup turns it on itself (yarn and acovering factor; `--rope-scaling`/`--rope-scale` override) ([details](DETAILS.md): "Context extension past262K").-- **Images?** Whether it should also read pictures. The image encoder (0.9 GB) runs on the GPU (0.1-0.5 s per- picture, ~1.4 GB of VRAM kept free for it) or on the CPU (about 3-13 s per picture, nothing on the GPU); with AMD cards- on the CPU for now. [Details](DETAILS.md#images-vision).+- **Images?** Whether it should also read pictures. Pictures are on by default for models that can read them. The+ image encoder (0.9 GB) runs on the CPU when your card has less than 24 GB (about 3-13 s per picture, nothing taken+ from the GPU) and on the GPU from 24 GB (0.1-0.5 s per picture, ~1.4 GB of VRAM kept free for it). The rule is the+ same for NVIDIA and AMD cards; `--vision gpu` and `--vision cpu` choose the reader yourself, `--vision no` turns+ pictures off. On a Radeon AI PRO R9700 the GPU reader took 57 ms for a 260-token picture and the CPU reader 3.1 s.+ The Mac has no picture reader yet. [Details](DETAILS.md#images-vision).- **Experimental speed projection?** Off unless you say yes - [read what it does](DETAILS.md#experimental-speed-projection-experimental-off-by-default)first. It changes how the model answers, and only the original model offers it.@@ -198,6 +299,8 @@ NVIDIA cards for now. [What it measures](DETAILS.md#double-click-start-herebat).## Options without questions+For the Terminal, `strata launch --help` lists every flag; the most used are in [TERMINAL_LAUNCH.md](TERMINAL_LAUNCH.md). These are the options of the installer from source:+```START-HERE.bat --setup install another model, or change context / imagesSETUP.bat the same (double-click it)