Skip to content

7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.

TypeScript

Split README into focused docs pages (#627)

Tashfeen90d291d95658
diff --git a/README.md b/README.md
index 84eca752..9d4c86ad 100644
--- a/README.md
+++ b/README.md
@@ -31,24 +31,20 @@ Your router updates its own model catalog from a signed feed: new free models, q
- [Supported providers](#supported-providers)
- [How it compares](#how-it-compares)
- [Features](#features)
-- [Not yet supported](#not-yet-supported)
- [Quick start](#quick-start)
-- [Works with OpenAI-compatible clients](#works-with-openai-compatible-clients)
-- [Docker](#docker)
- [Desktop app](#desktop-app)
+- [Works with OpenAI-compatible clients](#works-with-openai-compatible-clients)
- [Languages](#languages)
- [Premium (live catalog)](#premium-live-catalog)
- [Using the API](#using-the-api)
- [Screenshots](#screenshots)
- [How it works](#how-it-works)
-- [Context Handoff](#context-handoff)
- [Limitations](#limitations)
-- [Documentation index](docs/README.md)
-- [Contributor guide](CONTRIBUTING.md)
- [Contributing](#contributing)
-- [Terms of Service review](#terms-of-service-review)
- [Disclaimer](#disclaimer)
+**Guides:** [Install & deploy](docs/install.md) · [API reference](docs/api.md) · [Clients & coding agents](docs/clients.md) · [Architecture & internals](docs/architecture.md) · [Documentation index](docs/README.md) · [Contributor guide](CONTRIBUTING.md)
+
## Why this exists
Every serious AI lab now offers a free tier, a few million tokens a month, a few thousand requests a day. On its own each tier is a toy. Stacked together, they add up to roughly **4 billion tokens per month** of working inference capacity, across **235 model families / 339 provider endpoints** from small-and-fast to reasonably capable.
@@ -100,46 +96,23 @@ Based on public documentation, July 2026 — corrections welcome.
![Feature overview](repo-assets/features.png)
-- **OpenAI-compatible** — `POST /v1/chat/completions` and `GET /v1/models` work with the official OpenAI SDKs and any OpenAI-compatible client (LangChain, LlamaIndex, Continue, Hermes, etc.). Just change `base_url`.
-- **Responses API** — `POST /v1/responses` (the wire format current Codex CLI versions require) is implemented as a translating shim over the same router, with full streaming events and tool calls.
-- **Editor autocomplete** — `POST /v1/completions` translates legacy prompt/suffix requests into the same router, so VS Code ghost-text clients such as Continue can use FreeLLMAPI for inline suggestions.
-- **Anthropic Messages API** — `POST /v1/messages` (plus `/v1/messages/count_tokens`) speaks Anthropic's wire format over the same router, so **Claude Code** and the official Anthropic SDKs run against your free pool. `GET /v1/models` is content-negotiated (Anthropic shape when the client sends `anthropic-version`, OpenAI shape otherwise), and Claude families (`opus` / `sonnet` / `haiku` / `default`) map to `auto` or a pinned model on the Keys page. See [Anthropic / Claude clients](#anthropic--claude-clients).
-- **Fusion (multi-model synthesis)** — request the virtual `fusion` model and the router fans your prompt out to a panel of diverse free models in parallel, then a judge model synthesizes one answer from the drafts. Panel, judge, and strategy are configurable on the dashboard's **Fusion** page or per request via the `fusion` field; each sub-call goes through normal routing, quotas, and analytics.
-- **Image generation & text-to-speech** — `POST /v1/images/generations` and `POST /v1/audio/speech` route across the providers that serve media models, including custom OpenAI-compatible media endpoints. Browse and toggle them on the dashboard's **Models → Image / Audio** tabs.
-- **Self-updating model catalog** — the router syncs a signed catalog from freellmapi.co twice a day: new models, quota changes, and provider quirk fixes land in your install automatically. See [Premium (live catalog)](#premium-live-catalog).
-- **Streaming and non-streaming** — Server-Sent Events for `stream: true`, JSON response otherwise. Every provider adapter implements both.
-- **Tool calling** — OpenAI-style `tools` / `tool_choice` requests are passed through, and assistant `tool_calls` + `tool` role follow-up messages round-trip across providers. Models that emit tool calls as plain text instead of structured JSON are rescued into real `tool_calls` automatically, and tool requests only route to models that actually support them.
-- **Structured outputs & full sampling passthrough** — `response_format` (`json_object` / `json_schema`, translated to Gemini's native `responseSchema`), plus `seed`, `top_k`, `min_p`, presence/frequency/repetition penalties, `logit_bias`, `logprobs`, and the `max_completion_tokens` alias. Params a provider is known to reject are dropped per platform (Mistral's strict API, Groq's logprobs family…), and every model advertises its honest list in `/v1/models` `supported_parameters`.
-- **Embeddings** — `/v1/embeddings` with family-based routing, including custom OpenAI-compatible embedding endpoints: failover only ever happens between providers serving the *same* model (vectors from different models are incompatible), never across models. See [Embeddings](#embeddings).
-- **Automatic fallover** — If the chosen provider returns a 429, 5xx, or times out, the router skips it, puts the key on a short cooldown, and retries on the next model in your fallback chain (up to 20 attempts, bounded by a wall-clock retry budget). A dead key rotates to its siblings instead of failing the request, and exhaustion errors carry the full attempt trail so you can see exactly what was tried.
-- **Smart routing, six strategies** — the chain is ranked by a selectable strategy: `priority` (your manual order), `balanced`, `smartest`, `fastest`, `reliable`, or `custom` with your own weight mix. Scores come from live per-model measurements (speed, capability, rate-limit headroom, recent errors) with a Thompson-sampling bandit under the hood; one-click sort presets reorder the chain from the dashboard.
-- **Unified models** — the same logical model served by several providers (say, GLM-4.7 on Cloudflare and Z.ai) collapses into one entry: one name in `/v1/models`, strict in-group failover between its providers, and merge/split overrides when the grouping guesses wrong.
-- **Model profiles** — save named fallback-chain configurations (a coding chain, a long-context chain, a vision chain) and switch the active one from the dashboard.
-- **Per-key rate tracking** — RPM, RPD, TPM, and TPD counters per `(platform, model, key)` so the router always picks a key that's under its caps. The ledger also learns: ceilings a provider reports in error bodies or quota headers (a Groq 413 naming its TPM limit) tighten the router's own limits automatically.
-- **Sticky sessions** — Multi-turn conversations keep talking to the same model for 30 minutes to avoid the hallucination spike that comes from mid-conversation model switches.
-- **Response cache (opt-in)** — an exact-match in-memory LRU for identical non-streaming requests: canonical SHA-256 keys over the full request, TTL and temperature gates, per-request `X-FreeLLM-Cache: on|off` override, and saved-token stats on the dashboard. Off by default; cache hits consume zero provider quota.
-- **Encrypted key storage** — API keys are encrypted with AES-256-GCM before hitting SQLite; decryption happens in-memory just before a request.
-- **Key import & export** — bulk-import keys by pasting a `.env` file (with preview and per-key selection), export back out as JSON, `.env`, or CSV.
-- **Unified API key** — Clients authenticate to your proxy with a single `freellmapi-…` bearer token. You never expose upstream provider keys to your apps.
-- **Dashboard login** — The admin UI and all `/api/*` routes are gated behind an email + password account (scrypt-hashed, session-token auth), set on first run. The `/v1` proxy keeps its own unified-key auth for apps.
-- **Health checks** — Periodic probes mark keys as `healthy`, `rate_limited`, `invalid`, or `error` so the router skips dead ones automatically.
-- **Admin dashboard** — React + Vite UI to manage keys, reorder the fallback chain, inspect analytics, and run prompts in a playground. Dark/light/system theme and [60 languages](#languages) included.
-- **Analytics** — Per-request logging with latency (p50 / p95 and time-to-first-token for streams), token counts, success rate, estimated cost savings, and per-provider / per-model / per-key breakdowns over 24h to 90d windows.
-- **Interactive API docs** — `GET /v1/docs` serves a dependency-free OpenAPI viewer covering every proxy endpoint; the spec itself lives at `GET /v1/openapi.json`.
-- **MCP server** — `POST /mcp` speaks the Model Context Protocol (Streamable HTTP), so MCP-capable agents can ask the router which free models are usable right now (with per-model `supported_parameters`), check provider/key health and cooldowns, read usage and cache stats, and switch the routing strategy mid-session. See [Coding agents](#coding-agents).
-- **Encrypted DB backups** — optional periodic encrypted snapshots of the SQLite database to a local path or HTTP target, restored automatically on a fresh boot (`FREEAPI_DB_BACKUP_*` env vars).
-- **Context handoff on model switch** — Optional. When a session falls over to a different model, injects one compact system message so the new model knows it is continuing an existing task. Disabled by default; enable with `FREELLMAPI_CONTEXT_HANDOFF=on_model_switch`. See [Context Handoff](#context-handoff).
+- **Every OpenAI surface** — `/v1/chat/completions`, `/v1/responses` (what Codex CLI needs), `/v1/completions` (editor ghost-text autocomplete), `/v1/images/generations`, `/v1/audio/speech`, `/v1/embeddings`, and `/v1/models` — streaming and non-streaming, from the official SDKs or any OpenAI-compatible client. [API reference →](docs/api.md)
+- **Anthropic Messages API** — `/v1/messages` speaks Anthropic's wire format over the same router, so **Claude Code** and the official Anthropic SDKs run against your free pool. [Details →](docs/api.md#anthropic--claude-clients)
+- **Fusion (multi-model synthesis)** — request the virtual `fusion` model and the router fans your prompt out to a panel of diverse free models in parallel, then a judge model synthesizes one answer from the drafts. [Details →](docs/api.md#fusion-multi-model-synthesis)
+- **Image generation & text-to-speech** — `/v1/images/generations` and `/v1/audio/speech` route across the providers that serve media models, including custom OpenAI-compatible media endpoints.
+- **Tool calling & structured outputs** — OpenAI-style `tools` round-trip across providers (plain-text tool calls are rescued into real `tool_calls`), plus `response_format`, `seed`, `logprobs`, penalties, and the rest of the sampling params passed through per provider.
+- **Smart routing, six strategies** — live per-model speed/capability/reliability scores rank your chain; automatic fallover retries the next model on 429/5xx with cooldowns and key rotation. [Routing in detail →](docs/architecture.md#routing-in-detail)
+- **Unified models & profiles** — the same model on several providers collapses into one entry with strict in-group failover; named fallback-chain profiles (a coding chain, a vision chain) switch from the dashboard or per request via `auto:<profile>`.
+- **Per-key rate tracking** — RPM/RPD/TPM/TPD counters per `(platform, model, key)` that learn providers' reported ceilings, so routing always stays under every cap.
+- **Self-updating model catalog** — the router syncs a signed catalog from freellmapi.co twice a day: new models, quota changes, and provider quirk fixes land automatically. [Premium →](#premium-live-catalog)
+- **Sticky sessions & context handoff** — conversations stay on one model for 30 minutes; an optional compact handoff note keeps the thread coherent when a mid-chat switch does happen. [Details →](docs/clients.md#context-handoff)
+- **Encrypted keys, one token out** — provider keys are AES-256-GCM encrypted in SQLite and decrypted in-memory per request; your apps only ever see a single unified `freellmapi-…` bearer token.
+- **Admin dashboard & analytics** — React UI to manage keys, reorder the chain, run a playground, and read p50/p95/TTFT analytics over 24h–90d windows; login-gated, dark/light themes, [60 languages](#languages).
+- **MCP server & interactive docs** — agents can introspect usable models, provider health, and routing strategy over `/mcp`; a dependency-free OpenAPI viewer lives at `/v1/docs`. [Coding agents →](docs/clients.md)
+- **Ops niceties** — opt-in response cache, encrypted DB backups, periodic key health checks, bulk key import/export, declarative startup config. [Install & deploy →](docs/install.md)
- **Runs anywhere Node 20+ runs** — Windows, macOS, Linux servers, or a small ARM SBC (Raspberry Pi included). ~40 MB RSS at idle behind PM2 / systemd / whatever supervisor you prefer.
-## Not yet supported
-
-The scope is deliberately narrow. If a feature isn't on this list and isn't below, assume it isn't there yet.
-
-- **Moderation** (`/v1/moderations`)
-- **`n > 1`** (multiple completions per request)
-- **Per-user billing / multi-tenant auth** — single-user by design
-
-PRs that add any of these are very welcome. See [Contributing](#contributing).
+The scope is deliberately narrow — see [what's not supported yet](docs/architecture.md#not-yet-supported).
## Quick start
@@ -149,227 +122,27 @@ PRs that add any of these are very welcome. See [Contributing](#contributing).
curl -fsSL https://freellmapi.co/install.sh | bash
```
-Prefer to read before you pipe to bash? [The script is here](https://freellmapi.co/install.sh). Re-running it is safe: your `.env` (and encryption key) is preserved and the container updates to `:latest`. Override the defaults with `FREELLMAPI_DIR`, `PORT`, or `HOST_BIND` env vars.
-
-On Windows, the easiest path is the desktop **[`.exe` installer from Releases](https://github.com/tashfeenahmed/freellmapi/releases/latest)** (below); the Docker steps work in WSL or any bash shell.
-
-On Android, see the experimental [Termux installation guide](docs/install/android-termux.md). It uses Node's built-in SQLite driver and does not require the Android NDK.
-
-**Or manually with Docker Compose.** It runs the API and dashboard together on port 3001 and persists SQLite in a named volume.
-
-**Prerequisites:** Docker, Docker Compose, OpenSSL.
-
-*On macOS / Linux (Bash):*
-```bash
-git clone https://github.com/tashfeenahmed/freellmapi.git
-cd freellmapi
-
-# Generate an encryption key for at-rest key storage
-ENCRYPTION_KEY="$(openssl rand -hex 32)"
-printf "ENCRYPTION_KEY=%s\nPORT=3001\n" "$ENCRYPTION_KEY" > .env
-
-docker compose up -d
-```
-
-*On Windows (PowerShell):*
-```powershell
-git clone https://github.com/tashfeenahmed/freellmapi.git
-cd freellmapi
-
-$Bytes = New-Object Byte[] 32
-[Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($Bytes)
-$ENCRYPTION_KEY = -join ($Bytes | ForEach-Object { "{0:x2}" -f $_ })
-"ENCRYPTION_KEY=$ENCRYPTION_KEY`nPORT=3001" | Out-File -Encoding utf8 .env
-docker compose up -d
-```
+Prefer to read before you pipe to bash? [The script is here](https://freellmapi.co/install.sh). Re-running it is safe: your `.env` (and encryption key) is preserved and the container updates to `:latest`.
Open http://localhost:3001, add your provider keys on the **Keys** page, reorder the **Fallback Chain** to taste, and grab your unified API key from the **Keys** page header. That unified key is what you point your OpenAI SDK at.
-Your install keeps itself updated from the signed catalog feed. The current full catalog is listed at [freellmapi.co/models](https://freellmapi.co/models.html).
-
-> **Reaching it from another machine?** By default the container is published only on `127.0.0.1`, so `http://<server-ip>:3001` won't load from another device (the page just hangs). To expose it on your LAN — e.g. a Raspberry Pi at `http://192.168.1.x:3001` — start it with `HOST_BIND=0.0.0.0`:
->
-> ```bash
-> HOST_BIND=0.0.0.0 docker compose up -d
-> ```
->
-> Only do this on a trusted network: the proxy is single-user and guarded only by the unified API key.
-
-### Local development
-
-**Prerequisites:** Node.js 20+, npm.
-
-*On macOS / Linux (Bash):*
-```bash
-git clone https://github.com/tashfeenahmed/freellmapi.git
-cd freellmapi
-npm install
-ENCRYPTION_KEY="$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')"
-printf "ENCRYPTION_KEY=%s\nPORT=3001\n" "$ENCRYPTION_KEY" > .env
-npm run dev
-```
-
-*On Windows (PowerShell):*
-```powershell
-git clone https://github.com/tashfeenahmed/freellmapi.git
-cd freellmapi
-npm install
-$ENCRYPTION_KEY = node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
-"ENCRYPTION_KEY=$ENCRYPTION_KEY`nPORT=3001" | Out-File -Encoding utf8 .env
-npm run dev
-```
-
-`ENCRYPTION_KEY` is required for startup. When `NODE_ENV` is not `production`
-and it is unset, the server auto-generates a development key and saves it to a
-`.encryption-key` file (0600) next to the SQLite database, not inside it. Older
-installs that kept the key in the database are migrated to this file on first
-boot. Do not rely on that fallback with real provider keys; set `ENCRYPTION_KEY`.
-
-Request analytics are retained for 90 days or 100000 request rows by default,
-whichever limit prunes first. Set `REQUEST_ANALYTICS_RETENTION_DAYS=0` or
-`REQUEST_ANALYTICS_MAX_ROWS=0` in `.env` to disable either retention limit.
-
-Open http://localhost:5173 (the Vite dev UI), add your provider keys on the **Keys** page, reorder the **Fallback Chain** to taste, and grab your unified API key from the **Keys** page header. That unified key is what you point your OpenAI SDK at.
-
-### Declarative startup config
-
-For repeatable Docker/server installs, FreeLLMAPI can apply a JSON config on
-every boot. Set `FREEAPI_CONFIG_PATH=/path/to/freellmapi.config.json` or put the
-same JSON in `FREEAPI_CONFIG_JSON`. The config is idempotent: existing keys,
-custom providers, model edits, fallback rows, and routing settings are updated
-instead of duplicated.
-
-```json
-{
- "keys": [
- { "platform": "groq", "key": "gsk_...", "label": "main" },
- { "platform": "google", "key": "AIza...", "enabled": true }
- ],
- "customProviders": [
- {
- "baseUrl": "http://host.docker.internal:11434/v1",
- "label": "Ollama",
- "models": [
- { "model": "llama3.1:8b", "displayName": "Local Llama", "supportsTools": true }
- ]
- }
- ],
- "models": [
- {
- "platform": "groq",
- "modelId": "llama-3.3-70b-versatile",
- "displayName": "Llama 3.3 70B",
- "supportsTools": true,
- "fallbackEnabled": true
- }
- ],
- "routing": { "strategy": "balanced" }
-}
-```
-
-> **Reaching the dev UI from another device on your LAN?** Use `npm run dev:lan` — it passes `--host` through to Vite, which then prints a `Network: http://<your-ip>:5173` URL you can open from a phone or another machine. (Plain `npm run dev -- --host` does *not* work here: the root `dev` script is a `concurrently` wrapper, so the flag never reaches Vite.) API calls go through Vite's dev proxy, so no extra server config is needed.
-
-For a production build without Docker:
-
-```bash
-npm run build
-node server/dist/index.js # server + dashboard both served on :3001
-```
-
-## Docker
-
-FreeLLMAPI publishes a single production image that contains the Express server and the built React dashboard:
-
-```bash
-docker pull ghcr.io/tashfeenahmed/freellmapi:latest # or pin a release, e.g. :v1.2.3
-```
-
-The image is multi-arch (`linux/amd64` + `linux/arm64`, so it runs on a Raspberry Pi). Published tags: `latest` (default branch), `v*.*.*` (git release tags), and `sha-<commit>`.
-
-The included `docker-compose.yml` is the recommended install path:
-
-```bash
-docker compose up -d
-docker compose logs -f freellmapi
-```
-
-By default the container's port is bound to `127.0.0.1` (localhost only). To reach the dashboard/API from another machine on your network, publish it on all interfaces with `HOST_BIND=0.0.0.0 docker compose up -d` — only on a trusted LAN, since the proxy is single-user.
-
-SQLite data is stored in the `freellmapi-data` volume at `/app/server/data`.
-Keep the same `.env` `ENCRYPTION_KEY` and volume when upgrading, because
-provider keys are encrypted at rest. If your host only persists a specific
-directory, set `FREEAPI_DB_PATH=/that/path/freellmapi.db`.
-
-On hosts with ephemeral disks, configure an encrypted backup target:
+On Windows, the easiest path is the desktop **[`.exe` installer from Releases](https://github.com/tashfeenahmed/freellmapi/releases/latest)** (below). On Android, see the experimental [Termux guide](docs/install/android-termux.md).
-```env
-FREEAPI_DB_BACKUP_PATH=/app/server/data/freellmapi.db.backup
-# or:
-FREEAPI_DB_BACKUP_URL=https://example.com/freellmapi.db.backup
-FREEAPI_DB_BACKUP_TOKEN=optional-bearer-token
-FREEAPI_DB_BACKUP_KEY=64-char-hex-backup-key
-FREEAPI_DB_BACKUP_INTERVAL_MS=300000
-```
-
-When the database file is missing at startup, FreeLLMAPI restores the backup
-before migrations run. While the server is running it uploads a fresh encrypted
-backup periodically. If `FREEAPI_DB_BACKUP_KEY` is omitted, the app uses
-`ENCRYPTION_KEY` for the backup envelope too.
-
-More Docker operations and examples live in [docker/README.md](./docker/README.md).
+Everything else — Docker Compose, local development, declarative startup config, production builds, LAN access, and backups — is in **[docs/install.md](docs/install.md)**.
## Desktop app
-A native menu-bar app lives in [`desktop/`](./desktop): the entire router +
-dashboard running locally from your tray, with a glass popover showing live
-request stats.
+A native menu-bar app lives in [`desktop/`](./desktop): the entire router + dashboard running locally from your tray, with a glass popover showing live request stats.
![FreeLLMAPI desktop app](repo-assets/desktop.png)
-**[Download from Releases](https://github.com/tashfeenahmed/freellmapi/releases/latest)** — the macOS `.dmg` and the Windows `.exe` installer are built and attached to every release by the [`desktop-release`](.github/workflows/desktop-release.yml) workflow. Or build it from this repo in a few minutes:
-
-> **Note for Windows users building from source:** Building the desktop app requires compiling native SQLite modules for Electron. You must have [Visual Studio Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) installed (specifically the "Desktop development with C++" workload) and Python installed before running `npm install`.
-
-```bash
-npm install
-npm install --prefix desktop # install desktop dependencies
-npm run desktop:dist # macOS → desktop/dist-electron/FreeLLMAPI-…-arm64.dmg
-npm run desktop:dist:win # Windows → "desktop/dist-electron/FreeLLMAPI Setup ….exe"
-```
-
-> Locally built apps are unsigned, so Windows SmartScreen may warn on first run
-> ("More info" → "Run anyway"); the macOS build launches without Gatekeeper prompts.
-> Full instructions in [desktop/README.md](./desktop/README.md).
-
-### Credentials and where your data lives
-
-The desktop app has **no username or password to set up**. Unlike the server
-(which gates its dashboard behind an email + password account), the desktop
-build signs the dashboard in automatically with a hidden local account, so
-you're never prompted for credentials and never need one.
-
-The only credential you need is your **unified API key** — the
-`freellmapi-…` token your OpenAI/Anthropic client points at. Get it from:
-
-- the tray popover — click the tray icon, then **Copy Key**, or
-- the dashboard **Keys** page header (tray → **Open Dashboard**).
+**[Download from Releases](https://github.com/tashfeenahmed/freellmapi/releases/latest)** — the macOS `.dmg` and the Windows `.exe` installer are attached to every release. No account or password to set up: the only credential you need is the unified API key from the tray popover. Build-from-source steps and where your data lives: [docs/install.md](docs/install.md#desktop-app).
-You do not need to open or edit `freeapi.db` by hand.
-
-Your settings and data live in one folder per OS (copy it to migrate to
-another machine or into a container):
+## Works with OpenAI-compatible clients
-| OS | Location |
-|----|----------|
-| Windows | `%APPDATA%\FreeLLMAPI\` (e.g. `C:\Users\<you>\AppData\Roaming\FreeLLMAPI\`) |
-| macOS | `~/Library/Application Support/FreeLLMAPI/` |
-| Linux | `~/.config/FreeLLMAPI/` |
+Anything that can target an OpenAI-compatible base URL works: set it to `http://localhost:3001/v1` with the unified key from the dashboard. **Claude Code**, **Codex CLI**, **Cline / Roo Code**, **Continue** (including inline autocomplete), **Aider**, **opencode**, and **Cursor** each have a short recipe in **[docs/clients.md](docs/clients.md)** — and the router doubles as an MCP server your agents can introspect mid-session.
-That folder holds `freeapi.db` (all keys, models, settings, encrypted at rest)
-and `config.json` (window/theme/port/LAN preferences). Copy both to move an
-install. For the server (non-desktop) deployment, the equivalent state is the
-`.env` file and the SQLite DB at `server/data/freeapi.db` (or wherever
-`FREEAPI_DB_PATH` points).
+FreeLLMAPI is local-first and single-user by design. Your provider keys stay in your SQLite database, encrypted at rest, and requests go from your machine to the upstream providers you enabled.
## Languages
@@ -395,47 +168,6 @@ add a language, copy `en.json`, translate the values, and register the locale in
strings); `npm test` checks every locale for key/placeholder parity — PRs
welcome.
-## Works with OpenAI-compatible clients
-
-Any client that can target an OpenAI-compatible base URL can use FreeLLMAPI:
-
-- **LangChain, LlamaIndex, official OpenAI SDKs**: set `base_url` to
- `http://localhost:3001/v1` and use the unified key from the dashboard.
-- **Local GPU boxes**: add custom OpenAI-compatible endpoints for Ollama,
- llama.cpp, LM Studio, vLLM, or an internal gateway.
-
-### Coding agents
-
-Every recipe below is the same three facts in a different config file: base URL
-`http://localhost:3001/v1`, the unified key from the dashboard's Keys page, and
-a model (`auto` lets the router pick).
-
-| Agent | Setup |
-| --- | --- |
-| **Claude Code** | `ANTHROPIC_BASE_URL=http://localhost:3001` + `ANTHROPIC_AUTH_TOKEN=<unified key>` — full walkthrough in [Anthropic / Claude clients](#anthropic--claude-clients) |
-| **Codex CLI** | add a provider in `~/.codex/config.toml` with `base_url = "http://localhost:3001/v1"` and its `env_key` pointing at the unified key — the `/v1/responses` surface it needs is implemented |
-| **Cline / Roo Code** | provider type "OpenAI Compatible", base URL `http://localhost:3001/v1`, unified key, model `auto` (or any id from `/v1/models`) |
-| **Continue** | `apiBase: http://localhost:3001/v1` in its config; inline autocomplete works too via the legacy `/v1/completions` surface |
-| **Aider** | `OPENAI_API_BASE=http://localhost:3001/v1` + `OPENAI_API_KEY=<unified key>`, then `aider --model openai/auto` |
-| **opencode** | OpenAI-compatible provider with the same base URL and key |
-| **Cursor** | paste the unified key under a custom OpenAI base URL — but note Cursor verifies and calls the API **from its servers**, so your router must be reachable from the internet (a tunnel or a host with a public address), not just `localhost` |
-
-On top of inference, the router is an **MCP server**: agents can introspect it mid-session
-(usable models and the params each one honors, provider health, usage and cache stats,
-routing strategy). For Claude Code:
-
-```bash
-claude mcp add --transport http freellmapi http://localhost:3001/mcp \
- --header "Authorization: Bearer freellmapi-your-unified-key"
-```
-
-Any MCP client that speaks Streamable HTTP works the same way: point it at `/mcp` with the
-unified key as a Bearer token.
-
-FreeLLMAPI is local-first and single-user by design. Your provider keys stay in
-your SQLite database, encrypted at rest, and requests go from your machine to the
-upstream providers you enabled.
-
## Premium (live catalog)
The router keeps its model catalog fresh on its own: it pulls a signed catalog
@@ -468,10 +200,6 @@ router stays fully self-hosted either way.
## Using the API
-Any OpenAI-compatible client works (Anthropic / Claude clients too — see [Anthropic / Claude clients](#anthropic--claude-clients)). Examples:
-
-**Python**
-
```python
from openai import OpenAI
@@ -481,237 +209,14 @@ client = OpenAI(
)
resp = client.chat.completions.create(
- model="auto", # let the router pick; or specify e.g. "gemini-2.5-flash"
+ model="auto", # let the router pick; or "auto:fast", "auto:smart", a profile, or a model id
messages=[{"role": "user", "content": "Summarise the fall of Rome in one sentence."}],
)
print(resp.choices[0].message.content)
print("Routed via:", resp.headers.get("x-routed-via"))
```
-**curl**
-
-```bash
-curl http://localhost:3001/v1/chat/completions \
- -H "Authorization: Bearer freellmapi-your-unified-key" \
- -H "Content-Type: application/json" \
- -d '{
- "model": "auto",
- "messages": [{"role": "user", "content": "hi"}]
- }'
-```
-
-**Routing strategies (`auto:*`)**
-
-Plain `auto` follows your active fallback chain. Add a suffix to steer a single request instead — no dashboard changes needed:
-
-- `auto:smart` — favor the highest-intelligence models
-- `auto:fast` — favor measured speed (throughput and time-to-first-byte)
-- `auto:cheap` — budget-leaning; currently the same blend as `balanced` (everything in the pool is already free)
-- `auto:reliable` — favor recent success rate
-- `auto:balanced` — the default blend (reliability first, speed and intelligence split the rest)
-
-These rank **every enabled model**, ignoring your chain order. Common synonyms resolve too (`auto:fastest`, `auto:speed`, `auto:smartest`, `auto:cheapest`, `auto:budget`, …), and the whole model string is case-insensitive.
-
-```bash
-curl http://localhost:3001/v1/chat/completions \
- -H "Authorization: Bearer freellmapi-your-unified-key" \
- -H "Content-Type: application/json" \
- -d '{
- "model": "auto:fast",
- "messages": [{"role": "user", "content": "hi"}]
- }'
-```
-
-`auto:<profile-name>` routes through a named profile's chain instead of the active one, so different tools can use different chains through the same key:
-
-```bash
-curl http://localhost:3001/v1/chat/completions \
- -H "Authorization: Bearer freellmapi-your-unified-key" \
- -H "Content-Type: application/json" \
- -d '{
- "model": "auto:coding",
- "messages": [{"role": "user", "content": "Write a binary search in Rust."}]
- }'
-```
-
-An unknown profile name returns a clear `400` rather than silently falling back. Profiles are the named fallback chains from **Model profiles** (see [Features](#features)) — create and switch them from the dashboard; whichever is active is what plain `auto` uses.
-
-**Streaming**
-
-```python
-stream = client.chat.completions.create(
- model="auto",
- messages=[{"role": "user", "content": "Stream me a haiku about SQLite."}],
- stream=True,
-)
-for chunk in stream:
- print(chunk.choices[0].delta.content or "", end="", flush=True)
-```
-
-**VS Code ghost-text autocomplete (Continue)**
-
-FreeLLMAPI exposes `/v1/completions` for editor autocomplete clients that send legacy OpenAI prompt/suffix requests. Example Continue config:
-
-```yaml
-models:
- - name: FreeLLMAPI Autocomplete
- provider: openai
- model: auto
- apiBase: http://localhost:3001/v1
- apiKey: freellmapi-your-unified-key
- useLegacyCompletionsEndpoint: true
- roles:
- - autocomplete
-```
-
-**Tool calling**
-
-Pass OpenAI-style `tools` and `tool_choice`; the assistant response round-trips back through the proxy exactly like the OpenAI API. Multi-step flows (assistant `tool_calls` → `tool` role follow-up → final answer) work across every provider the router can reach.
-
-```python
-tools = [{
- "type": "function",
- "function": {
- "name": "get_weather",
- "description": "Get current weather for a city.",
- "parameters": {
- "type": "object",
- "properties": {"city": {"type": "string"}},
- "required": ["city"],
- },
- },
-}]
-
-# 1. Model asks for a tool call
-first = client.chat.completions.create(
- model="auto",
- messages=[{"role": "user", "content": "What's the weather in Karachi?"}],
- tools=tools,
- tool_choice="required",
-)
-call = first.choices[0].message.tool_calls[0]
-
-# 2. You execute the tool, feed the result back
-final = client.chat.completions.create(
- model="auto",
- messages=[
- {"role": "user", "content": "What's the weather in Karachi?"},
- first.choices[0].message,
- {"role": "tool", "tool_call_id": call.id, "content": '{"temp_c": 32, "cond": "sunny"}'},
- ],
- tools=tools,
-)
-print(final.choices[0].message.content)
-```
-
-**Gemini Google Search grounding**
-
-Google's models can ground their answers in live Google Search results. Since the OpenAI wire format has no way to express that, request a tool named `google_search` and the Google provider translates it into Gemini's native grounding tool. It can be sent on its own or alongside your normal function tools.
-
-```python
-resp = client.chat.completions.create(
- model="gemini-2.5-flash", # pin a Google model so the request routes there
- messages=[{"role": "user", "content": "Who won the F1 race this weekend?"}],
- tools=[{"type": "function", "function": {"name": "google_search", "parameters": {}}}],
-)
-print(resp.choices[0].message.content)
-```
-
-**Vision / image input**
-
-Send images with the standard OpenAI `image_url` content blocks (base64 `data:` URLs or `http(s)` URLs). When a request contains an image, the router restricts itself to **vision-capable models** and ignores text-only ones. Vision models are tagged with a **Vision** badge on the Fallback Chain page; the current set includes Gemini (2.5 / 3.x), Llama 4 Scout/Maverick (Groq, NVIDIA), GLM-4.6V Flash (Z.ai), Nemotron Nano 12B VL (OpenRouter), and GitHub's GPT-4o / GPT-4.1.
-
-```python
-resp = client.chat.completions.create(
- model="auto", # auto-routes to a vision model
- messages=[{
- "role": "user",
- "content": [
- {"type": "text", "text": "What's in this image?"},
- {"type": "image_url", "image_url": {"url": "data:image/png;base64,<...>"}},
- ],
- }],
-)
-print(resp.choices[0].message.content)
-```
-
-If no vision-capable model is enabled in your Fallback Chain, an image request returns a clear `422` (`code: "no_vision_model"`) rather than silently dropping the image. (Image input on `/v1/responses` isn't supported yet — use `/v1/chat/completions`.)
-
-Works with `stream=True` as well — you'll get `delta.tool_calls` chunks followed by a `finish_reason: "tool_calls"` close. Under the hood, OpenAI-compatible providers (Groq, Cerebras, Mistral, OpenRouter, GitHub Models, HuggingFace, Cloudflare, Cohere compat) get the request passed through; Gemini requests get translated into Google's `functionDeclarations` / `functionResponse` shape and the response is translated back.
-
-Every response carries an `X-Routed-Via: <platform>/<model>` header so you can see which provider actually served each call. If a request fell over between providers, you'll also see `X-Fallback-Attempts: N`.
-
-### Embeddings
-
-`/v1/embeddings` is OpenAI-compatible, with one deliberate difference from chat routing: **failover never crosses models.** Vectors from different models live in incompatible spaces — silently switching models would corrupt any vector store built on top of the proxy. So embeddings route by **family** (one model identity + dimension), and failover only walks the providers serving that same family.
-
-```python
-resp = client.embeddings.create(
- model="auto", # default family; or a family name like "bge-m3"
- input=["the quick brown fox", "pack my box with five dozen liquor jugs"],
-)
-print(len(resp.data), "vectors of", len(resp.data[0].embedding), "dims")
-```
-
-```bash
-curl http://localhost:3001/v1/embeddings \
- -H "Authorization: Bearer freellmapi-your-unified-key" \
- -H "Content-Type: application/json" \
- -d '{"model": "auto", "input": "hello world"}'
-```
-
-`model` accepts `auto` (the configured default family), a family name, or a provider-specific model id (which resolves to its family). Available families:
-
-| Family (`model`) | Dims | Providers (failover order) |
-| --- | --- | --- |
-| `gemini-embedding-001` *(default)* | 3072 | Google |
-| `text-embedding-3-large` | 3072 | GitHub Models |
-| `text-embedding-3-small` | 1536 | GitHub Models |
-| `embed-v4.0` | 1536 | Cohere |
-| `bge-m3` | 1024 | Cloudflare → Hugging Face |
-| `qwen3-embedding-0.6b` | 1024 | Cloudflare |
-| `nv-embedqa-e5-v5` | 1024 | NVIDIA |
-| `llama-nemotron-embed-1b-v2` | 2048 | NVIDIA |
-| `llama-nemotron-embed-vl-1b-v2` | 2048 | NVIDIA → OpenRouter |
-| `embeddinggemma-300m` | 768 | Cloudflare |
-
-The default family, per-provider toggles, and priorities live on the dashboard's **Models → Embeddings** page. Pick your family once and stick with it for a given vector store — that's the whole point of the family model.
-
-### Anthropic / Claude clients
-
-FreeLLMAPI also speaks Anthropic's Messages API, so anything built for Claude — including **Claude Code** and the official Anthropic SDKs — can run against your free pool. Point the client at your server's **origin** (Anthropic clients append `/v1/messages` themselves) and authenticate with your unified key. Both `x-api-key` and `Authorization: Bearer` are accepted.
-
-```bash
-curl http://localhost:3001/v1/messages \
- -H "x-api-key: freellmapi-your-unified-key" \
- -H "anthropic-version: 2023-06-01" \
- -H "Content-Type: application/json" \
- -d '{
- "model": "claude-sonnet-4-5",
- "max_tokens": 256,
- "messages": [{"role": "user", "content": "hi"}]
- }'
-```
-
-Claude model names map to your free pool on the **Keys → Anthropic** tab: each family (`default`, `opus`, `sonnet`, `haiku`) routes to `auto` (the router picks a free model) or a model you pin. `POST /v1/messages/count_tokens` and a content-negotiated `GET /v1/models` (Anthropic shape when `anthropic-version` is sent) are implemented too. Streaming, system prompts, tool use, and image input all translate across the same router as the OpenAI endpoints.
-
-**Claude Code** — point it at your server and start it:
-
-*On macOS / Linux (Bash):*
-```bash
-export ANTHROPIC_BASE_URL=http://localhost:3001
-export ANTHROPIC_AUTH_TOKEN=freellmapi-your-unified-key # NOT ANTHROPIC_API_KEY
-claude
-```
-
-*On Windows (PowerShell):*
-```powershell
-$env:ANTHROPIC_BASE_URL="http://localhost:3001"
-$env:ANTHROPIC_AUTH_TOKEN="freellmapi-your-unified-key"
-claude
-```
-
-> Use `ANTHROPIC_AUTH_TOKEN` (sent as a Bearer token), **not** `ANTHROPIC_API_KEY` — Claude Code treats a set `ANTHROPIC_API_KEY` as a conflicting first-party credential and refuses to start.
+Streaming, the `auto:*` routing strategies, tool calling, vision input, Gemini Google Search grounding, embeddings, and the Anthropic Messages surface — with curl and Python examples for each — are all in **[docs/api.md](docs/api.md)**. Every response carries an `X-Routed-Via: <platform>/<model>` header so you can see which provider actually served it.
## Screenshots
@@ -743,76 +248,11 @@ Request volume, success rate, tokens in and out, average latency, and per-provid
![One request in, the best free model out — the fallback chain with live scores, cooldowns, and quota tracking](repo-assets/router-flow.png)
-```
-┌──────────────────┐ Bearer freellmapi-… ┌─────────────────────────┐
-│ OpenAI SDK / │ ──────────────────────▶ │ Express proxy (:3001) │
-│ curl / any │ ◀────────────────────── │ /v1/chat/completions │
-│ OpenAI client │ streamed tokens └────────────┬────────────┘
-└──────────────────┘ │
- ▼
- ┌────────────────────────────────────────────────┐
- │ Router │
- │ 1. Pick highest-priority model that │
- │ (a) has a healthy key and │
- │ (b) is under all its rate limits. │
- │ 2. Decrypt key, call provider SDK. │
- │ 3. On 429/5xx → cooldown + retry next model. │
- └────────────────────────────────────────────────┘
- │
- ┌──────────────┬────────────┬──────────┴─────────┬─────────────┬──────────┐
- ▼ ▼ ▼ ▼ ▼ ▼
- Google Groq Cerebras OpenRouter HF …22 more
-```
-
-- **Router** (`server/src/services/router.ts`) — picks a model per request.
-- **Rate-limit ledger** (`server/src/services/ratelimit.ts`) — in-memory RPM/RPD/TPM/TPD counters backed by SQLite, with cooldowns on 429s.
-- **Provider adapters** (`server/src/providers/*.ts`) — one file per provider, implementing the `Provider` base class: `chatCompletion()` and `streamChatCompletion()`.
-- **Health service** (`server/src/services/health.ts`) — periodic probe keeps key status fresh.
-- **Dashboard** (`client/`) — React + Vite + shadcn/ui admin surface.
-- **Storage** — SQLite (`better-sqlite3`) with AES-256-GCM envelope encryption for keys.
-
-## Context Handoff
-
-When FreeLLMAPI falls over to a different model mid-conversation (quota, rate limit, cooldown), the new model has no idea it is picking up someone else's task. **Context handoff** adds a single compact `system` message to the outbound request that tells the new model exactly that:
-
-```
-FreeLLMAPI context handoff:
-You are taking over an ongoing conversation from another model (groq:llama-3 → google:gemini-flash).
-Continue the user's task using the conversation context already provided in this request.
-Do not restart the task, re-ask already answered setup questions, or discard prior tool results.
-Respect the user's latest message as the highest-priority instruction.
-
-Recent session summary:
-User: …
-Assistant: …
-```
-
-**Enable it in `.env`:**
-
-```env
-FREELLMAPI_CONTEXT_HANDOFF=on_model_switch
-```
-
-**How it works:**
-
-- Messages per session are stored in memory (TTL: 3 hours).
-- Only injected when the selected model changes for a given session key.
-- Not injected on the first request, on same-model continuations, or if a handoff message is already present.
-- Session key: `X-Session-Id` header if present, otherwise SHA-1 of the first user message (same as sticky sessions).
-- Storage is in-memory only. Nothing is written to disk or logged.
-
-> **Important:** Context Handoff improves continuity for conversations routed through FreeLLMAPI. It cannot recover provider-internal hidden state or messages that were never sent to the proxy.
+One request in, the best free model out: the router picks the highest-priority model with a healthy key that's under all its rate limits, decrypts the key in memory, and calls the provider — on a 429/5xx it cools that key down and retries the next model in your chain. The component walkthrough, routing internals, and operational details live in **[docs/architecture.md](docs/architecture.md)**.
## Limitations
-Stacking free tiers has real trade-offs. Be honest with yourself about them:
-
-- **No frontier models.** The free-tier catalog tops out around Llama 3.3 70B, GLM-4.5, Qwen 3 Coder, and Gemini 2.5 Pro. You will not get GPT-5 or Claude Opus class reasoning through this. For hard problems, pay for a real API.
-- **Intelligence degrades as the day progresses.** Your top-ranked models (usually Gemini 2.5 Pro, GPT-4o via GitHub Models) have the lowest daily caps. Once they hit their limits, the router falls down your priority chain to smaller/weaker models. Expect the effective intelligence of the endpoint to drop in the late hours of each day — then reset at UTC midnight.
-- **Latency is highly variable.** Cerebras and Groq are extremely fast; others are not. You get whichever one is available.
-- **Free tiers can change without notice.** Providers regularly tighten, loosen, or remove free tiers. When that happens you'll see 429s or auth errors until the catalog update reaches you — live-feed installs get those fixes within days, free installs on the 30-day trail. Re-seed scripts live in `server/src/scripts/`.
-- **No SLA, by definition.** If you need reliability, use a paid provider with a contract.
-- **Local-first.** There's no multi-tenant auth. Run this for yourself; don't expose it to the internet.
+Stacking free tiers has real trade-offs: no frontier models, variable latency, no SLA — and the effective intelligence of the endpoint dips late in the day as top models hit their daily caps, then resets at UTC midnight. Read the honest list in **[docs/architecture.md#limitations](docs/architecture.md#limitations)** before building anything real on this.
## Contributing
@@ -824,26 +264,7 @@ Contributors very welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for the dev lo
- **Dashboard polish** — charts on the Analytics page, key rotation UX, batch import of keys from `.env`.
- **Docs** — more examples, client library snippets for Go/Rust/etc., a deployment recipe for Docker or Fly.
-**Development loop:**
-
-```bash
-npm install
-npm run dev # server on :3001, dashboard on :5173, both with HMR
-npm test # server vitest; also runs client tests if the workspace adds them
-npm run build # compile server and dashboard
-```
-
-PRs should include a test, keep the existing test suite green, and match the `.editorconfig` / tsconfig defaults already in the repo. See [CONTRIBUTING.md](./CONTRIBUTING.md) for the full contributor workflow.
-
-### Database Migrations
-
-In local development, apply pending migrations with:
-
-```bash
-NODE_ENV=development npm run db:migration:up
-```
-
-See [CONTRIBUTING.md](./CONTRIBUTING.md) for the full migration CLI and workflow.
+`npm install && npm run dev` gets you the server on :3001 and the dashboard on :5173, both with HMR. PRs should include a test, keep the existing suite green (`npm test`), and match the `.editorconfig` / tsconfig defaults already in the repo. Database migration workflow and the full contributor loop are in [CONTRIBUTING.md](./CONTRIBUTING.md).
### Contributors
@@ -909,7 +330,6 @@ See [CONTRIBUTING.md](./CONTRIBUTING.md) for the full migration CLI and workflow
<a href="https://github.com/SeanPedersen"><img src="https://images.weserv.nl/?url=github.com/SeanPedersen.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@SeanPedersen" /></a>
<a href="https://github.com/andersmmg"><img src="https://images.weserv.nl/?url=github.com/andersmmg.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@andersmmg" /></a>
<a href="https://github.com/chirag127"><img src="https://images.weserv.nl/?url=github.com/chirag127.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@chirag127" /></a>
-<a href="https://github.com/jasnoorgill"><img src="https://images.weserv.nl/?url=github.com/jasnoorgill.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@jasnoorgill" /></a>
<a href="https://github.com/allababbot"><img src="https://images.weserv.nl/?url=github.com/allababbot.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@allababbot" /></a>
<a href="https://github.com/johan-droid"><img src="https://images.weserv.nl/?url=github.com/johan-droid.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@johan-droid" /></a>
<a href="https://github.com/redenfire"><img src="https://images.weserv.nl/?url=github.com/redenfire.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@redenfire" /></a>
@@ -922,13 +342,11 @@ See [CONTRIBUTING.md](./CONTRIBUTING.md) for the full migration CLI and workflow
<a href="https://github.com/AmiroKD"><img src="https://images.weserv.nl/?url=github.com/AmiroKD.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@AmiroKD" /></a>
<a href="https://github.com/ecryptomillionaire-dev"><img src="https://images.weserv.nl/?url=github.com/ecryptomillionaire-dev.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@ecryptomillionaire-dev" /></a>
<a href="https://github.com/4riful"><img src="https://images.weserv.nl/?url=github.com/4riful.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@4riful" /></a>
-<a href="https://github.com/cagedbird043"><img src="https://images.weserv.nl/?url=github.com/cagedbird043.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@cagedbird043" /></a>
<a href="https://github.com/fix2015"><img src="https://images.weserv.nl/?url=github.com/fix2015.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@fix2015" /></a>
<a href="https://github.com/iisyw"><img src="https://images.weserv.nl/?url=github.com/iisyw.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@iisyw" /></a>
<a href="https://github.com/xsfhacg"><img src="https://images.weserv.nl/?url=github.com/xsfhacg.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@xsfhacg" /></a>
<a href="https://github.com/noobix"><img src="https://images.weserv.nl/?url=github.com/noobix.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@noobix" /></a>
<a href="https://github.com/nandukmelath"><img src="https://images.weserv.nl/?url=github.com/nandukmelath.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@nandukmelath" /></a>
-<a href="https://github.com/coffcoe"><img src="https://images.weserv.nl/?url=github.com/coffcoe.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@coffcoe" /></a>
<a href="https://github.com/NirvanaCh7"><img src="https://images.weserv.nl/?url=github.com/NirvanaCh7.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@NirvanaCh7" /></a>
<a href="https://github.com/Mohamed3nan"><img src="https://images.weserv.nl/?url=github.com/Mohamed3nan.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@Mohamed3nan" /></a>
<a href="https://github.com/Arman-Espiar"><img src="https://images.weserv.nl/?url=github.com/Arman-Espiar.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@Arman-Espiar" /></a>
@@ -938,35 +356,12 @@ See [CONTRIBUTING.md](./CONTRIBUTING.md) for the full migration CLI and workflow
<a href="https://github.com/emv33"><img src="https://images.weserv.nl/?url=github.com/emv33.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@emv33" /></a>
<a href="https://github.com/ousamabenyounes"><img src="https://images.weserv.nl/?url=github.com/ousamabenyounes.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@ousamabenyounes" /></a>
-## Terms of Service review
-
-A self-hosted, single-user, personal-use setup was re-reviewed against each provider's ToS (May 2026). Summary:
-
-| Provider | Verdict | Notes |
-|---|---|---|
-| Google Gemini | ⚠️ Caution | March 2026 ToS narrows scope to *"professional or business purposes, not for consumer use"* — a self-hosted developer proxy is still defensible, but the clause is new. |
-| Groq | ✅ Likely OK | GroqCloud Services Agreement permits Customer Application integration. |
-| Cerebras | ✅ Likely OK | Permitted; explicitly forbids selling/transferring API keys. |
-| Mistral | ✅ Likely OK | APIs allowed for personal/internal business use. |
-| OpenRouter | ✅ Likely OK | April 2026 ToS sharpens the no-resale / no-competing-service clause; private single-user proxy still fine. |
-| Cloudflare Workers AI | ⚠️ Ambiguous | No anti-proxy clause; covered by general Self-Serve Subscription Agreement. |
-| NVIDIA NIM | ⚠️ Caution | Trial ToS §1.2 / §1.4: *"evaluation only, not production."* Free access is a recurring 40 RPM rate limit (the 2025 credit system was discontinued), but the evaluation-only scope stands. |
-| GitHub Models | ⚠️ Caution | Free tier explicitly scoped to *"experimentation"* and *"prototyping."* |
-| Cohere | ❌ Avoid | Terms §14 still forbids *"personal, family or household purposes."* |
-| Zhipu (open.bigmodel.cn) | ✅ Likely OK | Personal/non-commercial research carve-out still in the platform docs. |
-| Z.ai (api.z.ai) | ⚠️ Caution | New row — Singapore entity (distinct from Zhipu CN). §III.3(l) anti-traffic-redirect clause could plausibly be read against a proxy; no explicit personal-use carve-out. |
-| Ollama Cloud | ✅ Likely OK | New row — Free plan permits cloud-model access (1 concurrent, 5-hour session caps). No anti-proxy / anti-resale clauses found. *(Integration tracked in #14.)* |
-| OVH AI Endpoints | ✅ Likely OK | New row (June 2026) — anonymous access is officially documented (2 req/min per IP per model). OVH reserves the right to introduce token/consumption caps. |
-| AI Horde | ✅ Likely OK | New row (June 2026) — a free, community-powered commons run by the Haidra non-profit; anonymous use is officially supported (key `0000000000`, lowest queue priority). No anti-proxy / anti-resale clause. The OpenAI proxy is a pilot and may be restricted by usage. *(Integration #345.)* |
-
-Rules of thumb that keep most providers happy: **one account per provider**, **no reselling**, **no sharing your endpoint with other humans**, **don't hammer a free tier as a paid production backend**. This is informational, not legal advice — read each provider's ToS and make your own call.
-
-Removed since the April 2026 review: Hugging Face, Moonshot, and MiniMax direct integrations were dropped from the catalog (HF — tool-call format issues; Moonshot — moved to paid only; MiniMax — superseded by the OpenRouter `minimax/minimax-m2.5:free` route).
-
## Disclaimer
**This project is for personal experimentation and learning, not production.** Free tiers exist so developers can prototype against them; they aren't a stable, supported inference substrate and shouldn't be treated as one. If you build something real on top of FreeLLMAPI, swap in a paid API before you ship. Your relationship with each upstream provider is governed by the terms you accepted when you created your account — those terms still apply when the traffic is proxied through this project, and you're responsible for complying with them.
+How each provider's ToS views a personal, single-user proxy — reviewed provider by provider in May 2026 — is in [docs/architecture.md#terms-of-service-review](docs/architecture.md#terms-of-service-review).
+
## License
[MIT](./LICENSE)
diff --git a/docs/README.md b/docs/README.md
index 375c316f..9a970923 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -1,34 +1,21 @@
# FreeLLMAPI documentation
-This index points to the focused guides in the repository. The root [README](../README.md) remains the product overview and complete API reference.
+The root [README](../README.md) is the product overview; the detailed guides live here.
-## Install and deploy
+## Guides
-- [Quick start](../README.md#quick-start) — Docker one-liner, manual Docker Compose, and local development.
-- [Docker deployment](../docker/README.md) — container configuration and persistent storage.
-- [Desktop app](../desktop/README.md) — build and package the Electron application.
-- [Android with Termux](install/android-termux.md) — experimental local installation using Node's built-in SQLite driver.
-
-## Configure and operate
-
-- [Declarative startup configuration](../README.md#declarative-startup-config) — configure keys and custom providers from environment variables.
-- [Credentials and local data](../README.md#credentials-and-where-your-data-lives) — desktop paths and credential storage.
-- [Premium live catalog](../README.md#premium-live-catalog) — catalog update behavior and licensing.
-- [Database migrations](../server/src/db/README.md) — create, apply, inspect, and roll back schema migrations.
+- **[Install & deploy](install.md)** — quick start, Docker Compose, local development, declarative startup config, the Docker image, backups, the desktop app, and where your data lives.
+- **[API reference](api.md)** — chat completions, `auto:*` routing strategies, streaming, tool calling, vision, Gemini Google Search grounding, embeddings, response headers, and the Anthropic Messages surface.
+- **[Clients & coding agents](clients.md)** — OpenAI-compatible clients, recipes for Claude Code / Codex CLI / Cline / Continue / Aider / opencode / Cursor, the MCP server, editor autocomplete, and Context Handoff.
+- **[Architecture & internals](architecture.md)** — how the router works, routing and operational details, what's not supported, honest limitations, and the provider Terms-of-Service review.
-## Integrate clients
-
-- [OpenAI-compatible clients](../README.md#works-with-openai-compatible-clients) — SDK and tool configuration.
-- [Coding agents](../README.md#coding-agents) — Codex CLI, Claude Code, OpenCode, and MCP clients.
-- [Using the API](../README.md#using-the-api) — request examples for chat, streaming, tools, images, and audio.
-- [Embeddings](../README.md#embeddings) — embedding-model routing and custom endpoints.
-- [Anthropic and Claude clients](../README.md#anthropic--claude-clients) — Messages API compatibility.
-
-## Develop and contribute
+## More
+- [Android with Termux](install/android-termux.md) — experimental local installation using Node's built-in SQLite driver.
+- [Docker deployment](../docker/README.md) — container configuration and persistent storage.
+- [Desktop app](../desktop/README.md) — build and package the Electron application.
- [Contributor guide](../CONTRIBUTING.md) — development loop, testing expectations, and contribution policy.
-- [How the router works](../README.md#how-it-works) — architecture and fallback behavior.
-- [Database migration guide](../server/src/db/README.md) — migration CLI and conventions.
+- [Database migrations](../server/src/db/README.md) — create, apply, inspect, and roll back schema migrations.
## Website assets in this directory
diff --git a/docs/api.md b/docs/api.md
new file mode 100644
index 00000000..9ad3df93
--- /dev/null
+++ b/docs/api.md
@@ -0,0 +1,258 @@
+# API reference
+
+[← Back to README](../README.md) · [Documentation index](README.md)
+
+Any OpenAI-compatible client works (Anthropic / Claude clients too — see [Anthropic / Claude clients](#anthropic--claude-clients)). Base URL `http://localhost:3001/v1`, unified key from the dashboard's Keys page. An interactive OpenAPI viewer covering every proxy endpoint is served at `GET /v1/docs`; the spec itself lives at `GET /v1/openapi.json`.
+
+- [Chat completions](#chat-completions)
+- [Routing strategies (`auto:*`)](#routing-strategies-auto)
+- [Streaming](#streaming)
+- [Tool calling](#tool-calling)
+- [Gemini Google Search grounding](#gemini-google-search-grounding)
+- [Vision / image input](#vision--image-input)
+- [Images & text-to-speech](#images--text-to-speech)
+- [Fusion (multi-model synthesis)](#fusion-multi-model-synthesis)
+- [Response headers](#response-headers)
+- [Embeddings](#embeddings)
+- [Anthropic / Claude clients](#anthropic--claude-clients)
+
+## Chat completions
+
+**Python**
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+ base_url="http://localhost:3001/v1",
+ api_key="freellmapi-your-unified-key",
+)
+
+resp = client.chat.completions.create(
+ model="auto", # let the router pick; or specify e.g. "gemini-2.5-flash"
+ messages=[{"role": "user", "content": "Summarise the fall of Rome in one sentence."}],
+)
+print(resp.choices[0].message.content)
+print("Routed via:", resp.headers.get("x-routed-via"))
+```
+
+**curl**
+
+```bash
+curl http://localhost:3001/v1/chat/completions \
+ -H "Authorization: Bearer freellmapi-your-unified-key" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "auto",
+ "messages": [{"role": "user", "content": "hi"}]
+ }'
+```
+
+## Routing strategies (`auto:*`)
+
+Plain `auto` follows your active fallback chain. Add a suffix to steer a single request instead — no dashboard changes needed:
+
+- `auto:smart` — favor the highest-intelligence models
+- `auto:fast` — favor measured speed (throughput and time-to-first-byte)
+- `auto:cheap` — budget-leaning; currently the same blend as `balanced` (everything in the pool is already free)
+- `auto:reliable` — favor recent success rate
+- `auto:balanced` — the default blend (reliability first, speed and intelligence split the rest)
+
+These rank **every enabled model**, ignoring your chain order. Common synonyms resolve too (`auto:fastest`, `auto:speed`, `auto:smartest`, `auto:cheapest`, `auto:budget`, …), and the whole model string is case-insensitive.
+
+```bash
+curl http://localhost:3001/v1/chat/completions \
+ -H "Authorization: Bearer freellmapi-your-unified-key" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "auto:fast",
+ "messages": [{"role": "user", "content": "hi"}]
+ }'
+```
+
+`auto:<profile-name>` routes through a named profile's chain instead of the active one, so different tools can use different chains through the same key:
+
+```bash
+curl http://localhost:3001/v1/chat/completions \
+ -H "Authorization: Bearer freellmapi-your-unified-key" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "auto:coding",
+ "messages": [{"role": "user", "content": "Write a binary search in Rust."}]
+ }'
+```
+
+An unknown profile name returns a clear `400` rather than silently falling back. Profiles are named fallback chains (see [Features](../README.md#features)) — create and switch them from the dashboard; whichever is active is what plain `auto` uses.
+
+## Streaming
+
+```python
+stream = client.chat.completions.create(
+ model="auto",
+ messages=[{"role": "user", "content": "Stream me a haiku about SQLite."}],
+ stream=True,
+)
+for chunk in stream:
+ print(chunk.choices[0].delta.content or "", end="", flush=True)
+```
+
+## Tool calling
+
+Pass OpenAI-style `tools` and `tool_choice`; the assistant response round-trips back through the proxy exactly like the OpenAI API. Multi-step flows (assistant `tool_calls` → `tool` role follow-up → final answer) work across every provider the router can reach.
+
+```python
+tools = [{
+ "type": "function",
+ "function": {
+ "name": "get_weather",
+ "description": "Get current weather for a city.",
+ "parameters": {
+ "type": "object",
+ "properties": {"city": {"type": "string"}},
+ "required": ["city"],
+ },
+ },
+}]
+
+# 1. Model asks for a tool call
+first = client.chat.completions.create(
+ model="auto",
+ messages=[{"role": "user", "content": "What's the weather in Karachi?"}],
+ tools=tools,
+ tool_choice="required",
+)
+call = first.choices[0].message.tool_calls[0]
+
+# 2. You execute the tool, feed the result back
+final = client.chat.completions.create(
+ model="auto",
+ messages=[
+ {"role": "user", "content": "What's the weather in Karachi?"},
+ first.choices[0].message,
+ {"role": "tool", "tool_call_id": call.id, "content": '{"temp_c": 32, "cond": "sunny"}'},
+ ],
+ tools=tools,
+)
+print(final.choices[0].message.content)
+```
+
+Works with `stream=True` as well — you'll get `delta.tool_calls` chunks followed by a `finish_reason: "tool_calls"` close. Under the hood, OpenAI-compatible providers (Groq, Cerebras, Mistral, OpenRouter, GitHub Models, HuggingFace, Cloudflare, Cohere compat) get the request passed through; Gemini requests get translated into Google's `functionDeclarations` / `functionResponse` shape and the response is translated back.
+
+## Gemini Google Search grounding
+
+Google's models can ground their answers in live Google Search results. Since the OpenAI wire format has no way to express that, request a tool named `google_search` and the Google provider translates it into Gemini's native grounding tool. It can be sent on its own or alongside your normal function tools.
+
+```python
+resp = client.chat.completions.create(
+ model="gemini-2.5-flash", # pin a Google model so the request routes there
+ messages=[{"role": "user", "content": "Who won the F1 race this weekend?"}],
+ tools=[{"type": "function", "function": {"name": "google_search", "parameters": {}}}],
+)
+print(resp.choices[0].message.content)
+```
+
+## Vision / image input
+
+Send images with the standard OpenAI `image_url` content blocks (base64 `data:` URLs or `http(s)` URLs). When a request contains an image, the router restricts itself to **vision-capable models** and ignores text-only ones. Vision models are tagged with a **Vision** badge on the Fallback Chain page; the current set includes Gemini (2.5 / 3.x), Llama 4 Scout/Maverick (Groq, NVIDIA), GLM-4.6V Flash (Z.ai), Nemotron Nano 12B VL (OpenRouter), and GitHub's GPT-4o / GPT-4.1.
+
+```python
+resp = client.chat.completions.create(
+ model="auto", # auto-routes to a vision model
+ messages=[{
+ "role": "user",
+ "content": [
+ {"type": "text", "text": "What's in this image?"},
+ {"type": "image_url", "image_url": {"url": "data:image/png;base64,<...>"}},
+ ],
+ }],
+)
+print(resp.choices[0].message.content)
+```
+
+If no vision-capable model is enabled in your Fallback Chain, an image request returns a clear `422` (`code: "no_vision_model"`) rather than silently dropping the image. (Image input on `/v1/responses` isn't supported yet — use `/v1/chat/completions`.)
+
+## Images & text-to-speech
+
+`POST /v1/images/generations` and `POST /v1/audio/speech` route across the providers that serve media models, including custom OpenAI-compatible media endpoints. Browse and toggle them on the dashboard's **Models → Image / Audio** tabs.
+
+## Fusion (multi-model synthesis)
+
+Request the virtual `fusion` model and the router fans your prompt out to a panel of diverse free models in parallel, then a judge model synthesizes one answer from the drafts. Panel, judge, and strategy are configurable on the dashboard's **Fusion** page or per request via the `fusion` field; each sub-call goes through normal routing, quotas, and analytics.
+
+## Response headers
+
+Every response carries an `X-Routed-Via: <platform>/<model>` header so you can see which provider actually served each call. If a request fell over between providers, you'll also see `X-Fallback-Attempts: N`.
+
+The opt-in response cache can be toggled per request with `X-FreeLLM-Cache: on|off` — an exact-match in-memory LRU for identical non-streaming requests (canonical SHA-256 keys over the full request, TTL and temperature gates, saved-token stats on the dashboard). Off by default; cache hits consume zero provider quota.
+
+## Embeddings
+
+`/v1/embeddings` is OpenAI-compatible, with one deliberate difference from chat routing: **failover never crosses models.** Vectors from different models live in incompatible spaces — silently switching models would corrupt any vector store built on top of the proxy. So embeddings route by **family** (one model identity + dimension), and failover only walks the providers serving that same family.
+
+```python
+resp = client.embeddings.create(
+ model="auto", # default family; or a family name like "bge-m3"
+ input=["the quick brown fox", "pack my box with five dozen liquor jugs"],
+)
+print(len(resp.data), "vectors of", len(resp.data[0].embedding), "dims")
+```
+
+```bash
+curl http://localhost:3001/v1/embeddings \
+ -H "Authorization: Bearer freellmapi-your-unified-key" \
+ -H "Content-Type: application/json" \
+ -d '{"model": "auto", "input": "hello world"}'
+```
+
+`model` accepts `auto` (the configured default family), a family name, or a provider-specific model id (which resolves to its family). Available families:
+
+| Family (`model`) | Dims | Providers (failover order) |
+| --- | --- | --- |
+| `gemini-embedding-001` *(default)* | 3072 | Google |
+| `text-embedding-3-large` | 3072 | GitHub Models |
+| `text-embedding-3-small` | 1536 | GitHub Models |
+| `embed-v4.0` | 1536 | Cohere |
+| `bge-m3` | 1024 | Cloudflare → Hugging Face |
+| `qwen3-embedding-0.6b` | 1024 | Cloudflare |
+| `nv-embedqa-e5-v5` | 1024 | NVIDIA |
+| `llama-nemotron-embed-1b-v2` | 2048 | NVIDIA |
+| `llama-nemotron-embed-vl-1b-v2` | 2048 | NVIDIA → OpenRouter |
+| `embeddinggemma-300m` | 768 | Cloudflare |
+
+The default family, per-provider toggles, and priorities live on the dashboard's **Models → Embeddings** page. Pick your family once and stick with it for a given vector store — that's the whole point of the family model.
+
+## Anthropic / Claude clients
+
+FreeLLMAPI also speaks Anthropic's Messages API, so anything built for Claude — including **Claude Code** and the official Anthropic SDKs — can run against your free pool. Point the client at your server's **origin** (Anthropic clients append `/v1/messages` themselves) and authenticate with your unified key. Both `x-api-key` and `Authorization: Bearer` are accepted.
+
+```bash
+curl http://localhost:3001/v1/messages \
+ -H "x-api-key: freellmapi-your-unified-key" \
+ -H "anthropic-version: 2023-06-01" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "claude-sonnet-4-5",
+ "max_tokens": 256,
+ "messages": [{"role": "user", "content": "hi"}]
+ }'
+```
+
+Claude model names map to your free pool on the **Keys → Anthropic** tab: each family (`default`, `opus`, `sonnet`, `haiku`) routes to `auto` (the router picks a free model) or a model you pin. `POST /v1/messages/count_tokens` and a content-negotiated `GET /v1/models` (Anthropic shape when `anthropic-version` is sent) are implemented too. Streaming, system prompts, tool use, and image input all translate across the same router as the OpenAI endpoints.
+
+**Claude Code** — point it at your server and start it:
+
+*On macOS / Linux (Bash):*
+```bash
+export ANTHROPIC_BASE_URL=http://localhost:3001
+export ANTHROPIC_AUTH_TOKEN=freellmapi-your-unified-key # NOT ANTHROPIC_API_KEY
+claude
+```
+
+*On Windows (PowerShell):*
+```powershell
+$env:ANTHROPIC_BASE_URL="http://localhost:3001"
+$env:ANTHROPIC_AUTH_TOKEN="freellmapi-your-unified-key"
+claude
+```
+
+> Use `ANTHROPIC_AUTH_TOKEN` (sent as a Bearer token), **not** `ANTHROPIC_API_KEY` — Claude Code treats a set `ANTHROPIC_API_KEY` as a conflicting first-party credential and refuses to start.
diff --git a/docs/architecture.md b/docs/architecture.md
new file mode 100644
index 00000000..b4376310
--- /dev/null
+++ b/docs/architecture.md
@@ -0,0 +1,112 @@
+# Architecture & internals
+
+[← Back to README](../README.md) · [Documentation index](README.md)
+
+How the router decides, what to expect from stacked free tiers, and where the boundaries are.
+
+- [How it works](#how-it-works)
+- [Routing in detail](#routing-in-detail)
+- [Operational details](#operational-details)
+- [Not yet supported](#not-yet-supported)
+- [Limitations](#limitations)
+- [Terms of Service review](#terms-of-service-review)
+
+## How it works
+
+![One request in, the best free model out — the fallback chain with live scores, cooldowns, and quota tracking](../repo-assets/router-flow.png)
+
+```
+┌──────────────────┐ Bearer freellmapi-… ┌─────────────────────────┐
+│ OpenAI SDK / │ ──────────────────────▶ │ Express proxy (:3001) │
+│ curl / any │ ◀────────────────────── │ /v1/chat/completions │
+│ OpenAI client │ streamed tokens └────────────┬────────────┘
+└──────────────────┘ │
+ ▼
+ ┌────────────────────────────────────────────────┐
+ │ Router │
+ │ 1. Pick highest-priority model that │
+ │ (a) has a healthy key and │
+ │ (b) is under all its rate limits. │
+ │ 2. Decrypt key, call provider SDK. │
+ │ 3. On 429/5xx → cooldown + retry next model. │
+ └────────────────────────────────────────────────┘
+ │
+ ┌──────────────┬────────────┬──────────┴─────────┬─────────────┬──────────┐
+ ▼ ▼ ▼ ▼ ▼ ▼
+ Google Groq Cerebras OpenRouter HF …22 more
+```
+
+- **Router** (`server/src/services/router.ts`) — picks a model per request.
+- **Rate-limit ledger** (`server/src/services/ratelimit.ts`) — in-memory RPM/RPD/TPM/TPD counters backed by SQLite, with cooldowns on 429s.
+- **Provider adapters** (`server/src/providers/*.ts`) — one file per provider, implementing the `Provider` base class: `chatCompletion()` and `streamChatCompletion()`.
+- **Health service** (`server/src/services/health.ts`) — periodic probe keeps key status fresh.
+- **Dashboard** (`client/`) — React + Vite + shadcn/ui admin surface.
+- **Storage** — SQLite (`better-sqlite3`) with AES-256-GCM envelope encryption for keys.
+
+## Routing in detail
+
+- **Automatic fallover** — If the chosen provider returns a 429, 5xx, or times out, the router skips it, puts the key on a short cooldown, and retries on the next model in your fallback chain (up to 20 attempts, bounded by a wall-clock retry budget). A dead key rotates to its siblings instead of failing the request, and exhaustion errors carry the full attempt trail so you can see exactly what was tried.
+- **Smart routing, six strategies** — the chain is ranked by a selectable strategy: `priority` (your manual order), `balanced`, `smartest`, `fastest`, `reliable`, or `custom` with your own weight mix. Scores come from live per-model measurements (speed, capability, rate-limit headroom, recent errors) with a Thompson-sampling bandit under the hood; one-click sort presets reorder the chain from the dashboard.
+- **Unified models** — the same logical model served by several providers (say, GLM-4.7 on Cloudflare and Z.ai) collapses into one entry: one name in `/v1/models`, strict in-group failover between its providers, and merge/split overrides when the grouping guesses wrong.
+- **Model profiles** — save named fallback-chain configurations (a coding chain, a long-context chain, a vision chain) and switch the active one from the dashboard.
+- **Per-key rate tracking** — RPM, RPD, TPM, and TPD counters per `(platform, model, key)` so the router always picks a key that's under its caps. The ledger also learns: ceilings a provider reports in error bodies or quota headers (a Groq 413 naming its TPM limit) tighten the router's own limits automatically.
+- **Sticky sessions** — Multi-turn conversations keep talking to the same model for 30 minutes to avoid the hallucination spike that comes from mid-conversation model switches. See also [Context Handoff](clients.md#context-handoff) for what happens when a switch is unavoidable.
+- **Structured outputs & full sampling passthrough** — `response_format` (`json_object` / `json_schema`, translated to Gemini's native `responseSchema`), plus `seed`, `top_k`, `min_p`, presence/frequency/repetition penalties, `logit_bias`, `logprobs`, and the `max_completion_tokens` alias. Params a provider is known to reject are dropped per platform (Mistral's strict API, Groq's logprobs family…), and every model advertises its honest list in `/v1/models` `supported_parameters`.
+- **Tool-call rescue** — models that emit tool calls as plain text instead of structured JSON are rescued into real `tool_calls` automatically, and tool requests only route to models that actually support them.
+
+## Operational details
+
+- **Encrypted key storage** — API keys are encrypted with AES-256-GCM before hitting SQLite; decryption happens in-memory just before a request.
+- **Unified API key** — Clients authenticate to your proxy with a single `freellmapi-…` bearer token. You never expose upstream provider keys to your apps.
+- **Dashboard login** — The admin UI and all `/api/*` routes are gated behind an email + password account (scrypt-hashed, session-token auth), set on first run. The `/v1` proxy keeps its own unified-key auth for apps.
+- **Health checks** — Periodic probes mark keys as `healthy`, `rate_limited`, `invalid`, or `error` so the router skips dead ones automatically.
+- **Response cache (opt-in)** — an exact-match in-memory LRU for identical non-streaming requests: canonical SHA-256 keys over the full request, TTL and temperature gates, per-request `X-FreeLLM-Cache: on|off` override, and saved-token stats on the dashboard. Off by default; cache hits consume zero provider quota.
+- **Key import & export** — bulk-import keys by pasting a `.env` file (with preview and per-key selection), export back out as JSON, `.env`, or CSV.
+- **Analytics** — Per-request logging with latency (p50 / p95 and time-to-first-token for streams), token counts, success rate, estimated cost savings, and per-provider / per-model / per-key breakdowns over 24h to 90d windows.
+- **Encrypted DB backups** — optional periodic encrypted snapshots of the SQLite database to a local path or HTTP target, restored automatically on a fresh boot (see [Install & deploy](install.md#docker-image--operations)).
+
+## Not yet supported
+
+The scope is deliberately narrow. If a feature isn't in the README's feature list and isn't below, assume it isn't there yet.
+
+- **Moderation** (`/v1/moderations`)
+- **`n > 1`** (multiple completions per request)
+- **Per-user billing / multi-tenant auth** — single-user by design
+
+PRs that add any of these are very welcome. See [Contributing](../README.md#contributing).
+
+## Limitations
+
+Stacking free tiers has real trade-offs. Be honest with yourself about them:
+
+- **No frontier models.** The free-tier catalog tops out around Llama 3.3 70B, GLM-4.5, Qwen 3 Coder, and Gemini 2.5 Pro. You will not get GPT-5 or Claude Opus class reasoning through this. For hard problems, pay for a real API.
+- **Intelligence degrades as the day progresses.** Your top-ranked models (usually Gemini 2.5 Pro, GPT-4o via GitHub Models) have the lowest daily caps. Once they hit their limits, the router falls down your priority chain to smaller/weaker models. Expect the effective intelligence of the endpoint to drop in the late hours of each day — then reset at UTC midnight.
+- **Latency is highly variable.** Cerebras and Groq are extremely fast; others are not. You get whichever one is available.
+- **Free tiers can change without notice.** Providers regularly tighten, loosen, or remove free tiers. When that happens you'll see 429s or auth errors until the catalog update reaches you — live-feed installs get those fixes within days, free installs on the 30-day trail. Re-seed scripts live in `server/src/scripts/`.
+- **No SLA, by definition.** If you need reliability, use a paid provider with a contract.
+- **Local-first.** There's no multi-tenant auth. Run this for yourself; don't expose it to the internet.
+
+## Terms of Service review
+
+A self-hosted, single-user, personal-use setup was re-reviewed against each provider's ToS (May 2026). Summary:
+
+| Provider | Verdict | Notes |
+|---|---|---|
+| Google Gemini | ⚠️ Caution | March 2026 ToS narrows scope to *"professional or business purposes, not for consumer use"* — a self-hosted developer proxy is still defensible, but the clause is new. |
+| Groq | ✅ Likely OK | GroqCloud Services Agreement permits Customer Application integration. |
+| Cerebras | ✅ Likely OK | Permitted; explicitly forbids selling/transferring API keys. |
+| Mistral | ✅ Likely OK | APIs allowed for personal/internal business use. |
+| OpenRouter | ✅ Likely OK | April 2026 ToS sharpens the no-resale / no-competing-service clause; private single-user proxy still fine. |
+| Cloudflare Workers AI | ⚠️ Ambiguous | No anti-proxy clause; covered by general Self-Serve Subscription Agreement. |
+| NVIDIA NIM | ⚠️ Caution | Trial ToS §1.2 / §1.4: *"evaluation only, not production."* Free access is a recurring 40 RPM rate limit (the 2025 credit system was discontinued), but the evaluation-only scope stands. |
+| GitHub Models | ⚠️ Caution | Free tier explicitly scoped to *"experimentation"* and *"prototyping."* |
+| Cohere | ❌ Avoid | Terms §14 still forbids *"personal, family or household purposes."* |
+| Zhipu (open.bigmodel.cn) | ✅ Likely OK | Personal/non-commercial research carve-out still in the platform docs. |
+| Z.ai (api.z.ai) | ⚠️ Caution | New row — Singapore entity (distinct from Zhipu CN). §III.3(l) anti-traffic-redirect clause could plausibly be read against a proxy; no explicit personal-use carve-out. |
+| Ollama Cloud | ✅ Likely OK | New row — Free plan permits cloud-model access (1 concurrent, 5-hour session caps). No anti-proxy / anti-resale clauses found. *(Integration tracked in #14.)* |
+| OVH AI Endpoints | ✅ Likely OK | New row (June 2026) — anonymous access is officially documented (2 req/min per IP per model). OVH reserves the right to introduce token/consumption caps. |
+| AI Horde | ✅ Likely OK | New row (June 2026) — a free, community-powered commons run by the Haidra non-profit; anonymous use is officially supported (key `0000000000`, lowest queue priority). No anti-proxy / anti-resale clause. The OpenAI proxy is a pilot and may be restricted by usage. *(Integration #345.)* |
+
+Rules of thumb that keep most providers happy: **one account per provider**, **no reselling**, **no sharing your endpoint with other humans**, **don't hammer a free tier as a paid production backend**. This is informational, not legal advice — read each provider's ToS and make your own call.
+
+Removed since the April 2026 review: Hugging Face, Moonshot, and MiniMax direct integrations were dropped from the catalog (HF — tool-call format issues; Moonshot — moved to paid only; MiniMax — superseded by the OpenRouter `minimax/minimax-m2.5:free` route).
diff --git a/docs/clients.md b/docs/clients.md
new file mode 100644
index 00000000..7df1e8c9
--- /dev/null
+++ b/docs/clients.md
@@ -0,0 +1,100 @@
+# Clients & coding agents
+
+[← Back to README](../README.md) · [Documentation index](README.md)
+
+- [OpenAI-compatible clients](#openai-compatible-clients)
+- [Coding agents](#coding-agents)
+- [MCP server](#mcp-server)
+- [VS Code ghost-text autocomplete (Continue)](#vs-code-ghost-text-autocomplete-continue)
+- [Context Handoff](#context-handoff)
+
+## OpenAI-compatible clients
+
+Any client that can target an OpenAI-compatible base URL can use FreeLLMAPI:
+
+- **LangChain, LlamaIndex, official OpenAI SDKs**: set `base_url` to
+ `http://localhost:3001/v1` and use the unified key from the dashboard.
+- **Local GPU boxes**: add custom OpenAI-compatible endpoints for Ollama,
+ llama.cpp, LM Studio, vLLM, or an internal gateway.
+
+## Coding agents
+
+Every recipe below is the same three facts in a different config file: base URL
+`http://localhost:3001/v1`, the unified key from the dashboard's Keys page, and
+a model (`auto` lets the router pick).
+
+| Agent | Setup |
+| --- | --- |
+| **Claude Code** | `ANTHROPIC_BASE_URL=http://localhost:3001` + `ANTHROPIC_AUTH_TOKEN=<unified key>` — full walkthrough in [Anthropic / Claude clients](api.md#anthropic--claude-clients) |
+| **Codex CLI** | add a provider in `~/.codex/config.toml` with `base_url = "http://localhost:3001/v1"` and its `env_key` pointing at the unified key — the `/v1/responses` surface it needs is implemented |
+| **Cline / Roo Code** | provider type "OpenAI Compatible", base URL `http://localhost:3001/v1`, unified key, model `auto` (or any id from `/v1/models`) |
+| **Continue** | `apiBase: http://localhost:3001/v1` in its config; inline autocomplete works too via the legacy `/v1/completions` surface |
+| **Aider** | `OPENAI_API_BASE=http://localhost:3001/v1` + `OPENAI_API_KEY=<unified key>`, then `aider --model openai/auto` |
+| **opencode** | OpenAI-compatible provider with the same base URL and key |
+| **Cursor** | paste the unified key under a custom OpenAI base URL — but note Cursor verifies and calls the API **from its servers**, so your router must be reachable from the internet (a tunnel or a host with a public address), not just `localhost` |
+
+## MCP server
+
+On top of inference, the router is an **MCP server**: agents can introspect it mid-session
+(usable models and the params each one honors, provider health, usage and cache stats,
+routing strategy). For Claude Code:
+
+```bash
+claude mcp add --transport http freellmapi http://localhost:3001/mcp \
+ --header "Authorization: Bearer freellmapi-your-unified-key"
+```
+
+Any MCP client that speaks Streamable HTTP works the same way: point it at `/mcp` with the
+unified key as a Bearer token.
+
+FreeLLMAPI is local-first and single-user by design. Your provider keys stay in
+your SQLite database, encrypted at rest, and requests go from your machine to the
+upstream providers you enabled.
+
+## VS Code ghost-text autocomplete (Continue)
+
+FreeLLMAPI exposes `/v1/completions` for editor autocomplete clients that send legacy OpenAI prompt/suffix requests. Example Continue config:
+
+```yaml
+models:
+ - name: FreeLLMAPI Autocomplete
+ provider: openai
+ model: auto
+ apiBase: http://localhost:3001/v1
+ apiKey: freellmapi-your-unified-key
+ useLegacyCompletionsEndpoint: true
+ roles:
+ - autocomplete
+```
+
+## Context Handoff
+
+When FreeLLMAPI falls over to a different model mid-conversation (quota, rate limit, cooldown), the new model has no idea it is picking up someone else's task. **Context handoff** adds a single compact `system` message to the outbound request that tells the new model exactly that:
+
+```
+FreeLLMAPI context handoff:
+You are taking over an ongoing conversation from another model (groq:llama-3 → google:gemini-flash).
+Continue the user's task using the conversation context already provided in this request.
+Do not restart the task, re-ask already answered setup questions, or discard prior tool results.
+Respect the user's latest message as the highest-priority instruction.
+
+Recent session summary:
+User: …
+Assistant: …
+```
+
+**Enable it in `.env`:**
+
+```env
+FREELLMAPI_CONTEXT_HANDOFF=on_model_switch
+```
+
+**How it works:**
+
+- Messages per session are stored in memory (TTL: 3 hours).
+- Only injected when the selected model changes for a given session key.
+- Not injected on the first request, on same-model continuations, or if a handoff message is already present.
+- Session key: `X-Session-Id` header if present, otherwise SHA-1 of the first user message (same as sticky sessions).
+- Storage is in-memory only. Nothing is written to disk or logged.
+
+> **Important:** Context Handoff improves continuity for conversations routed through FreeLLMAPI. It cannot recover provider-internal hidden state or messages that were never sent to the proxy.
diff --git a/docs/install.md b/docs/install.md
new file mode 100644
index 00000000..a20c553b
--- /dev/null
+++ b/docs/install.md
@@ -0,0 +1,245 @@
+# Install & deploy
+
+[← Back to README](../README.md) · [Documentation index](README.md)
+
+Everything about getting FreeLLMAPI running: the one-liner, Docker Compose, local development, declarative config, production builds, the desktop app, and where your data lives.
+
+- [Quick start (one-liner)](#quick-start-one-liner)
+- [Docker Compose](#docker-compose)
+- [Local development](#local-development)
+- [Declarative startup config](#declarative-startup-config)
+- [Docker image & operations](#docker-image--operations)
+- [Desktop app](#desktop-app)
+- [Credentials and where your data lives](#credentials-and-where-your-data-lives)
+
+## Quick start (one-liner)
+
+Docker required — sets up `~/freellmapi`, generates an encryption key, pulls the image, and starts the container:
+
+```bash
+curl -fsSL https://freellmapi.co/install.sh | bash
+```
+
+Prefer to read before you pipe to bash? [The script is here](https://freellmapi.co/install.sh). Re-running it is safe: your `.env` (and encryption key) is preserved and the container updates to `:latest`. Override the defaults with `FREELLMAPI_DIR`, `PORT`, or `HOST_BIND` env vars.
+
+On Windows, the easiest path is the desktop **[`.exe` installer from Releases](https://github.com/tashfeenahmed/freellmapi/releases/latest)** ([below](#desktop-app)); the Docker steps work in WSL or any bash shell.
+
+On Android, see the experimental [Termux installation guide](install/android-termux.md). It uses Node's built-in SQLite driver and does not require the Android NDK.
+
+Open http://localhost:3001, add your provider keys on the **Keys** page, reorder the **Fallback Chain** to taste, and grab your unified API key from the **Keys** page header. That unified key is what you point your OpenAI SDK at.
+
+Your install keeps itself updated from the signed catalog feed. The current full catalog is listed at [freellmapi.co/models](https://freellmapi.co/models.html).
+
+## Docker Compose
+
+Runs the API and dashboard together on port 3001 and persists SQLite in a named volume.
+
+**Prerequisites:** Docker, Docker Compose, OpenSSL.
+
+*On macOS / Linux (Bash):*
+```bash
+git clone https://github.com/tashfeenahmed/freellmapi.git
+cd freellmapi
+
+# Generate an encryption key for at-rest key storage
+ENCRYPTION_KEY="$(openssl rand -hex 32)"
+printf "ENCRYPTION_KEY=%s\nPORT=3001\n" "$ENCRYPTION_KEY" > .env
+
+docker compose up -d
+```
+
+*On Windows (PowerShell):*
+```powershell
+git clone https://github.com/tashfeenahmed/freellmapi.git
+cd freellmapi
+
+$Bytes = New-Object Byte[] 32
+[Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($Bytes)
+$ENCRYPTION_KEY = -join ($Bytes | ForEach-Object { "{0:x2}" -f $_ })
+"ENCRYPTION_KEY=$ENCRYPTION_KEY`nPORT=3001" | Out-File -Encoding utf8 .env
+docker compose up -d
+```
+
+> **Reaching it from another machine?** By default the container is published only on `127.0.0.1`, so `http://<server-ip>:3001` won't load from another device (the page just hangs). To expose it on your LAN — e.g. a Raspberry Pi at `http://192.168.1.x:3001` — start it with `HOST_BIND=0.0.0.0`:
+>
+> ```bash
+> HOST_BIND=0.0.0.0 docker compose up -d
+> ```
+>
+> Only do this on a trusted network: the proxy is single-user and guarded only by the unified API key.
+
+## Local development
+
+**Prerequisites:** Node.js 20+, npm.
+
+*On macOS / Linux (Bash):*
+```bash
+git clone https://github.com/tashfeenahmed/freellmapi.git
+cd freellmapi
+npm install
+ENCRYPTION_KEY="$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')"
+printf "ENCRYPTION_KEY=%s\nPORT=3001\n" "$ENCRYPTION_KEY" > .env
+npm run dev
+```
+
+*On Windows (PowerShell):*
+```powershell
+git clone https://github.com/tashfeenahmed/freellmapi.git
+cd freellmapi
+npm install
+$ENCRYPTION_KEY = node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
+"ENCRYPTION_KEY=$ENCRYPTION_KEY`nPORT=3001" | Out-File -Encoding utf8 .env
+npm run dev
+```
+
+`ENCRYPTION_KEY` is required for startup. When `NODE_ENV` is not `production`
+and it is unset, the server auto-generates a development key and saves it to a
+`.encryption-key` file (0600) next to the SQLite database, not inside it. Older
+installs that kept the key in the database are migrated to this file on first
+boot. Do not rely on that fallback with real provider keys; set `ENCRYPTION_KEY`.
+
+Request analytics are retained for 90 days or 100000 request rows by default,
+whichever limit prunes first. Set `REQUEST_ANALYTICS_RETENTION_DAYS=0` or
+`REQUEST_ANALYTICS_MAX_ROWS=0` in `.env` to disable either retention limit.
+
+Open http://localhost:5173 (the Vite dev UI), add your provider keys on the **Keys** page, reorder the **Fallback Chain** to taste, and grab your unified API key from the **Keys** page header. That unified key is what you point your OpenAI SDK at.
+
+> **Reaching the dev UI from another device on your LAN?** Use `npm run dev:lan` — it passes `--host` through to Vite, which then prints a `Network: http://<your-ip>:5173` URL you can open from a phone or another machine. (Plain `npm run dev -- --host` does *not* work here: the root `dev` script is a `concurrently` wrapper, so the flag never reaches Vite.) API calls go through Vite's dev proxy, so no extra server config is needed.
+
+For a production build without Docker:
+
+```bash
+npm run build
+node server/dist/index.js # server + dashboard both served on :3001
+```
+
+## Declarative startup config
+
+For repeatable Docker/server installs, FreeLLMAPI can apply a JSON config on
+every boot. Set `FREEAPI_CONFIG_PATH=/path/to/freellmapi.config.json` or put the
+same JSON in `FREEAPI_CONFIG_JSON`. The config is idempotent: existing keys,
+custom providers, model edits, fallback rows, and routing settings are updated
+instead of duplicated.
+
+```json
+{
+ "keys": [
+ { "platform": "groq", "key": "gsk_...", "label": "main" },
+ { "platform": "google", "key": "AIza...", "enabled": true }
+ ],
+ "customProviders": [
+ {
+ "baseUrl": "http://host.docker.internal:11434/v1",
+ "label": "Ollama",
+ "models": [
+ { "model": "llama3.1:8b", "displayName": "Local Llama", "supportsTools": true }
+ ]
+ }
+ ],
+ "models": [
+ {
+ "platform": "groq",
+ "modelId": "llama-3.3-70b-versatile",
+ "displayName": "Llama 3.3 70B",
+ "supportsTools": true,
+ "fallbackEnabled": true
+ }
+ ],
+ "routing": { "strategy": "balanced" }
+}
+```
+
+## Docker image & operations
+
+FreeLLMAPI publishes a single production image that contains the Express server and the built React dashboard:
+
+```bash
+docker pull ghcr.io/tashfeenahmed/freellmapi:latest # or pin a release, e.g. :v1.2.3
+```
+
+The image is multi-arch (`linux/amd64` + `linux/arm64`, so it runs on a Raspberry Pi). Published tags: `latest` (default branch), `v*.*.*` (git release tags), and `sha-<commit>`.
+
+The included `docker-compose.yml` is the recommended install path:
+
+```bash
+docker compose up -d
+docker compose logs -f freellmapi
+```
+
+By default the container's port is bound to `127.0.0.1` (localhost only). To reach the dashboard/API from another machine on your network, publish it on all interfaces with `HOST_BIND=0.0.0.0 docker compose up -d` — only on a trusted LAN, since the proxy is single-user.
+
+SQLite data is stored in the `freellmapi-data` volume at `/app/server/data`.
+Keep the same `.env` `ENCRYPTION_KEY` and volume when upgrading, because
+provider keys are encrypted at rest. If your host only persists a specific
+directory, set `FREEAPI_DB_PATH=/that/path/freellmapi.db`.
+
+On hosts with ephemeral disks, configure an encrypted backup target:
+
+```env
+FREEAPI_DB_BACKUP_PATH=/app/server/data/freellmapi.db.backup
+# or:
+FREEAPI_DB_BACKUP_URL=https://example.com/freellmapi.db.backup
+FREEAPI_DB_BACKUP_TOKEN=optional-bearer-token
+FREEAPI_DB_BACKUP_KEY=64-char-hex-backup-key
+FREEAPI_DB_BACKUP_INTERVAL_MS=300000
+```
+
+When the database file is missing at startup, FreeLLMAPI restores the backup
+before migrations run. While the server is running it uploads a fresh encrypted
+backup periodically. If `FREEAPI_DB_BACKUP_KEY` is omitted, the app uses
+`ENCRYPTION_KEY` for the backup envelope too.
+
+More Docker operations and examples live in [docker/README.md](../docker/README.md).
+
+## Desktop app
+
+A native menu-bar app lives in [`desktop/`](../desktop): the entire router +
+dashboard running locally from your tray, with a glass popover showing live
+request stats.
+
+![FreeLLMAPI desktop app](../repo-assets/desktop.png)
+
+**[Download from Releases](https://github.com/tashfeenahmed/freellmapi/releases/latest)** — the macOS `.dmg` and the Windows `.exe` installer are built and attached to every release by the [`desktop-release`](../.github/workflows/desktop-release.yml) workflow. Or build it from this repo in a few minutes:
+
+> **Note for Windows users building from source:** Building the desktop app requires compiling native SQLite modules for Electron. You must have [Visual Studio Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) installed (specifically the "Desktop development with C++" workload) and Python installed before running `npm install`.
+
+```bash
+npm install
+npm install --prefix desktop # install desktop dependencies
+npm run desktop:dist # macOS → desktop/dist-electron/FreeLLMAPI-…-arm64.dmg
+npm run desktop:dist:win # Windows → "desktop/dist-electron/FreeLLMAPI Setup ….exe"
+```
+
+> Locally built apps are unsigned, so Windows SmartScreen may warn on first run
+> ("More info" → "Run anyway"); the macOS build launches without Gatekeeper prompts.
+> Full instructions in [desktop/README.md](../desktop/README.md).
+
+## Credentials and where your data lives
+
+The desktop app has **no username or password to set up**. Unlike the server
+(which gates its dashboard behind an email + password account), the desktop
+build signs the dashboard in automatically with a hidden local account, so
+you're never prompted for credentials and never need one.
+
+The only credential you need is your **unified API key** — the
+`freellmapi-…` token your OpenAI/Anthropic client points at. Get it from:
+
+- the tray popover — click the tray icon, then **Copy Key**, or
+- the dashboard **Keys** page header (tray → **Open Dashboard**).
+
+You do not need to open or edit `freeapi.db` by hand.
+
+Your settings and data live in one folder per OS (copy it to migrate to
+another machine or into a container):
+
+| OS | Location |
+|----|----------|
+| Windows | `%APPDATA%\FreeLLMAPI\` (e.g. `C:\Users\<you>\AppData\Roaming\FreeLLMAPI\`) |
+| macOS | `~/Library/Application Support/FreeLLMAPI/` |
+| Linux | `~/.config/FreeLLMAPI/` |
+
+That folder holds `freeapi.db` (all keys, models, settings, encrypted at rest)
+and `config.json` (window/theme/port/LAN preferences). Copy both to move an
+install. For the server (non-desktop) deployment, the equivalent state is the
+`.env` file and the SQLite DB at `server/data/freeapi.db` (or wherever
+`FREEAPI_DB_PATH` points).