7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.
TypeScript
Split README into focused docs pages (#627)
Tashfeen90d291d95658
diff --git a/README.md b/README.mdindex 84eca752..9d4c86ad 100644--- a/README.md+++ b/README.md@@ -31,24 +31,20 @@ Your router updates its own model catalog from a signed feed: new free models, q- [Supported providers](#supported-providers)- [How it compares](#how-it-compares)- [Features](#features)-- [Not yet supported](#not-yet-supported)- [Quick start](#quick-start)-- [Works with OpenAI-compatible clients](#works-with-openai-compatible-clients)-- [Docker](#docker)- [Desktop app](#desktop-app)+- [Works with OpenAI-compatible clients](#works-with-openai-compatible-clients)- [Languages](#languages)- [Premium (live catalog)](#premium-live-catalog)- [Using the API](#using-the-api)- [Screenshots](#screenshots)- [How it works](#how-it-works)-- [Context Handoff](#context-handoff)- [Limitations](#limitations)-- [Documentation index](docs/README.md)-- [Contributor guide](CONTRIBUTING.md)- [Contributing](#contributing)-- [Terms of Service review](#terms-of-service-review)- [Disclaimer](#disclaimer)+**Guides:** [Install & deploy](docs/install.md) · [API reference](docs/api.md) · [Clients & coding agents](docs/clients.md) · [Architecture & internals](docs/architecture.md) · [Documentation index](docs/README.md) · [Contributor guide](CONTRIBUTING.md)+## Why this existsEvery serious AI lab now offers a free tier, a few million tokens a month, a few thousand requests a day. On its own each tier is a toy. Stacked together, they add up to roughly **4 billion tokens per month** of working inference capacity, across **235 model families / 339 provider endpoints** from small-and-fast to reasonably capable.@@ -100,46 +96,23 @@ Based on public documentation, July 2026 — corrections welcome.-- **OpenAI-compatible** — `POST /v1/chat/completions` and `GET /v1/models` work with the official OpenAI SDKs and any OpenAI-compatible client (LangChain, LlamaIndex, Continue, Hermes, etc.). Just change `base_url`.-- **Responses API** — `POST /v1/responses` (the wire format current Codex CLI versions require) is implemented as a translating shim over the same router, with full streaming events and tool calls.-- **Editor autocomplete** — `POST /v1/completions` translates legacy prompt/suffix requests into the same router, so VS Code ghost-text clients such as Continue can use FreeLLMAPI for inline suggestions.-- **Anthropic Messages API** — `POST /v1/messages` (plus `/v1/messages/count_tokens`) speaks Anthropic's wire format over the same router, so **Claude Code** and the official Anthropic SDKs run against your free pool. `GET /v1/models` is content-negotiated (Anthropic shape when the client sends `anthropic-version`, OpenAI shape otherwise), and Claude families (`opus` / `sonnet` / `haiku` / `default`) map to `auto` or a pinned model on the Keys page. See [Anthropic / Claude clients](#anthropic--claude-clients).-- **Fusion (multi-model synthesis)** — request the virtual `fusion` model and the router fans your prompt out to a panel of diverse free models in parallel, then a judge model synthesizes one answer from the drafts. Panel, judge, and strategy are configurable on the dashboard's **Fusion** page or per request via the `fusion` field; each sub-call goes through normal routing, quotas, and analytics.-- **Image generation & text-to-speech** — `POST /v1/images/generations` and `POST /v1/audio/speech` route across the providers that serve media models, including custom OpenAI-compatible media endpoints. Browse and toggle them on the dashboard's **Models → Image / Audio** tabs.-- **Self-updating model catalog** — the router syncs a signed catalog from freellmapi.co twice a day: new models, quota changes, and provider quirk fixes land in your install automatically. See [Premium (live catalog)](#premium-live-catalog).-- **Streaming and non-streaming** — Server-Sent Events for `stream: true`, JSON response otherwise. Every provider adapter implements both.-- **Tool calling** — OpenAI-style `tools` / `tool_choice` requests are passed through, and assistant `tool_calls` + `tool` role follow-up messages round-trip across providers. Models that emit tool calls as plain text instead of structured JSON are rescued into real `tool_calls` automatically, and tool requests only route to models that actually support them.-- **Structured outputs & full sampling passthrough** — `response_format` (`json_object` / `json_schema`, translated to Gemini's native `responseSchema`), plus `seed`, `top_k`, `min_p`, presence/frequency/repetition penalties, `logit_bias`, `logprobs`, and the `max_completion_tokens` alias. Params a provider is known to reject are dropped per platform (Mistral's strict API, Groq's logprobs family…), and every model advertises its honest list in `/v1/models` `supported_parameters`.-- **Embeddings** — `/v1/embeddings` with family-based routing, including custom OpenAI-compatible embedding endpoints: failover only ever happens between providers serving the *same* model (vectors from different models are incompatible), never across models. See [Embeddings](#embeddings).-- **Automatic fallover** — If the chosen provider returns a 429, 5xx, or times out, the router skips it, puts the key on a short cooldown, and retries on the next model in your fallback chain (up to 20 attempts, bounded by a wall-clock retry budget). A dead key rotates to its siblings instead of failing the request, and exhaustion errors carry the full attempt trail so you can see exactly what was tried.-- **Smart routing, six strategies** — the chain is ranked by a selectable strategy: `priority` (your manual order), `balanced`, `smartest`, `fastest`, `reliable`, or `custom` with your own weight mix. Scores come from live per-model measurements (speed, capability, rate-limit headroom, recent errors) with a Thompson-sampling bandit under the hood; one-click sort presets reorder the chain from the dashboard.-- **Unified models** — the same logical model served by several providers (say, GLM-4.7 on Cloudflare and Z.ai) collapses into one entry: one name in `/v1/models`, strict in-group failover between its providers, and merge/split overrides when the grouping guesses wrong.-- **Model profiles** — save named fallback-chain configurations (a coding chain, a long-context chain, a vision chain) and switch the active one from the dashboard.-- **Per-key rate tracking** — RPM, RPD, TPM, and TPD counters per `(platform, model, key)` so the router always picks a key that's under its caps. The ledger also learns: ceilings a provider reports in error bodies or quota headers (a Groq 413 naming its TPM limit) tighten the router's own limits automatically.-- **Sticky sessions** — Multi-turn conversations keep talking to the same model for 30 minutes to avoid the hallucination spike that comes from mid-conversation model switches.-- **Response cache (opt-in)** — an exact-match in-memory LRU for identical non-streaming requests: canonical SHA-256 keys over the full request, TTL and temperature gates, per-request `X-FreeLLM-Cache: on|off` override, and saved-token stats on the dashboard. Off by default; cache hits consume zero provider quota.-- **Encrypted key storage** — API keys are encrypted with AES-256-GCM before hitting SQLite; decryption happens in-memory just before a request.-- **Key import & export** — bulk-import keys by pasting a `.env` file (with preview and per-key selection), export back out as JSON, `.env`, or CSV.-- **Unified API key** — Clients authenticate to your proxy with a single `freellmapi-…` bearer token. You never expose upstream provider keys to your apps.-- **Dashboard login** — The admin UI and all `/api/*` routes are gated behind an email + password account (scrypt-hashed, session-token auth), set on first run. The `/v1` proxy keeps its own unified-key auth for apps.-- **Health checks** — Periodic probes mark keys as `healthy`, `rate_limited`, `invalid`, or `error` so the router skips dead ones automatically.-- **Admin dashboard** — React + Vite UI to manage keys, reorder the fallback chain, inspect analytics, and run prompts in a playground. Dark/light/system theme and [60 languages](#languages) included.-- **Analytics** — Per-request logging with latency (p50 / p95 and time-to-first-token for streams), token counts, success rate, estimated cost savings, and per-provider / per-model / per-key breakdowns over 24h to 90d windows.-- **Interactive API docs** — `GET /v1/docs` serves a dependency-free OpenAPI viewer covering every proxy endpoint; the spec itself lives at `GET /v1/openapi.json`.-- **MCP server** — `POST /mcp` speaks the Model Context Protocol (Streamable HTTP), so MCP-capable agents can ask the router which free models are usable right now (with per-model `supported_parameters`), check provider/key health and cooldowns, read usage and cache stats, and switch the routing strategy mid-session. See [Coding agents](#coding-agents).-- **Encrypted DB backups** — optional periodic encrypted snapshots of the SQLite database to a local path or HTTP target, restored automatically on a fresh boot (`FREEAPI_DB_BACKUP_*` env vars).-- **Context handoff on model switch** — Optional. When a session falls over to a different model, injects one compact system message so the new model knows it is continuing an existing task. Disabled by default; enable with `FREELLMAPI_CONTEXT_HANDOFF=on_model_switch`. See [Context Handoff](#context-handoff).+- **Every OpenAI surface** — `/v1/chat/completions`, `/v1/responses` (what Codex CLI needs), `/v1/completions` (editor ghost-text autocomplete), `/v1/images/generations`, `/v1/audio/speech`, `/v1/embeddings`, and `/v1/models` — streaming and non-streaming, from the official SDKs or any OpenAI-compatible client. [API reference →](docs/api.md)+- **Anthropic Messages API** — `/v1/messages` speaks Anthropic's wire format over the same router, so **Claude Code** and the official Anthropic SDKs run against your free pool. [Details →](docs/api.md#anthropic--claude-clients)+- **Fusion (multi-model synthesis)** — request the virtual `fusion` model and the router fans your prompt out to a panel of diverse free models in parallel, then a judge model synthesizes one answer from the drafts. [Details →](docs/api.md#fusion-multi-model-synthesis)+- **Image generation & text-to-speech** — `/v1/images/generations` and `/v1/audio/speech` route across the providers that serve media models, including custom OpenAI-compatible media endpoints.+- **Tool calling & structured outputs** — OpenAI-style `tools` round-trip across providers (plain-text tool calls are rescued into real `tool_calls`), plus `response_format`, `seed`, `logprobs`, penalties, and the rest of the sampling params passed through per provider.+- **Smart routing, six strategies** — live per-model speed/capability/reliability scores rank your chain; automatic fallover retries the next model on 429/5xx with cooldowns and key rotation. [Routing in detail →](docs/architecture.md#routing-in-detail)+- **Unified models & profiles** — the same model on several providers collapses into one entry with strict in-group failover; named fallback-chain profiles (a coding chain, a vision chain) switch from the dashboard or per request via `auto:<profile>`.+- **Per-key rate tracking** — RPM/RPD/TPM/TPD counters per `(platform, model, key)` that learn providers' reported ceilings, so routing always stays under every cap.+- **Self-updating model catalog** — the router syncs a signed catalog from freellmapi.co twice a day: new models, quota changes, and provider quirk fixes land automatically. [Premium →](#premium-live-catalog)+- **Sticky sessions & context handoff** — conversations stay on one model for 30 minutes; an optional compact handoff note keeps the thread coherent when a mid-chat switch does happen. [Details →](docs/clients.md#context-handoff)+- **Encrypted keys, one token out** — provider keys are AES-256-GCM encrypted in SQLite and decrypted in-memory per request; your apps only ever see a single unified `freellmapi-…` bearer token.+- **Admin dashboard & analytics** — React UI to manage keys, reorder the chain, run a playground, and read p50/p95/TTFT analytics over 24h–90d windows; login-gated, dark/light themes, [60 languages](#languages).+- **MCP server & interactive docs** — agents can introspect usable models, provider health, and routing strategy over `/mcp`; a dependency-free OpenAPI viewer lives at `/v1/docs`. [Coding agents →](docs/clients.md)+- **Ops niceties** — opt-in response cache, encrypted DB backups, periodic key health checks, bulk key import/export, declarative startup config. [Install & deploy →](docs/install.md)- **Runs anywhere Node 20+ runs** — Windows, macOS, Linux servers, or a small ARM SBC (Raspberry Pi included). ~40 MB RSS at idle behind PM2 / systemd / whatever supervisor you prefer.-## Not yet supported--The scope is deliberately narrow. If a feature isn't on this list and isn't below, assume it isn't there yet.--- **Moderation** (`/v1/moderations`)-- **`n > 1`** (multiple completions per request)-- **Per-user billing / multi-tenant auth** — single-user by design--PRs that add any of these are very welcome. See [Contributing](#contributing).+The scope is deliberately narrow — see [what's not supported yet](docs/architecture.md#not-yet-supported).## Quick start@@ -149,227 +122,27 @@ PRs that add any of these are very welcome. See [Contributing](#contributing).curl -fsSL https://freellmapi.co/install.sh | bash```-Prefer to read before you pipe to bash? [The script is here](https://freellmapi.co/install.sh). Re-running it is safe: your `.env` (and encryption key) is preserved and the container updates to `:latest`. Override the defaults with `FREELLMAPI_DIR`, `PORT`, or `HOST_BIND` env vars.--On Windows, the easiest path is the desktop **[`.exe` installer from Releases](https://github.com/tashfeenahmed/freellmapi/releases/latest)** (below); the Docker steps work in WSL or any bash shell.--On Android, see the experimental [Termux installation guide](docs/install/android-termux.md). It uses Node's built-in SQLite driver and does not require the Android NDK.--**Or manually with Docker Compose.** It runs the API and dashboard together on port 3001 and persists SQLite in a named volume.--**Prerequisites:** Docker, Docker Compose, OpenSSL.--*On macOS / Linux (Bash):*-```bash-git clone https://github.com/tashfeenahmed/freellmapi.git-cd freellmapi--# Generate an encryption key for at-rest key storage-ENCRYPTION_KEY="$(openssl rand -hex 32)"-printf "ENCRYPTION_KEY=%s\nPORT=3001\n" "$ENCRYPTION_KEY" > .env--docker compose up -d-```--*On Windows (PowerShell):*-```powershell-git clone https://github.com/tashfeenahmed/freellmapi.git-cd freellmapi--$Bytes = New-Object Byte[] 32-[Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($Bytes)-$ENCRYPTION_KEY = -join ($Bytes | ForEach-Object { "{0:x2}" -f $_ })-"ENCRYPTION_KEY=$ENCRYPTION_KEY`nPORT=3001" | Out-File -Encoding utf8 .env-docker compose up -d-```+Prefer to read before you pipe to bash? [The script is here](https://freellmapi.co/install.sh). Re-running it is safe: your `.env` (and encryption key) is preserved and the container updates to `:latest`.Open http://localhost:3001, add your provider keys on the **Keys** page, reorder the **Fallback Chain** to taste, and grab your unified API key from the **Keys** page header. That unified key is what you point your OpenAI SDK at.-Your install keeps itself updated from the signed catalog feed. The current full catalog is listed at [freellmapi.co/models](https://freellmapi.co/models.html).--> **Reaching it from another machine?** By default the container is published only on `127.0.0.1`, so `http://<server-ip>:3001` won't load from another device (the page just hangs). To expose it on your LAN — e.g. a Raspberry Pi at `http://192.168.1.x:3001` — start it with `HOST_BIND=0.0.0.0`:->-> ```bash-> HOST_BIND=0.0.0.0 docker compose up -d-> ```->-> Only do this on a trusted network: the proxy is single-user and guarded only by the unified API key.--### Local development--**Prerequisites:** Node.js 20+, npm.--*On macOS / Linux (Bash):*-```bash-git clone https://github.com/tashfeenahmed/freellmapi.git-cd freellmapi-npm install-ENCRYPTION_KEY="$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')"-printf "ENCRYPTION_KEY=%s\nPORT=3001\n" "$ENCRYPTION_KEY" > .env-npm run dev-```--*On Windows (PowerShell):*-```powershell-git clone https://github.com/tashfeenahmed/freellmapi.git-cd freellmapi-npm install-$ENCRYPTION_KEY = node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"-"ENCRYPTION_KEY=$ENCRYPTION_KEY`nPORT=3001" | Out-File -Encoding utf8 .env-npm run dev-```--`ENCRYPTION_KEY` is required for startup. When `NODE_ENV` is not `production`-and it is unset, the server auto-generates a development key and saves it to a-`.encryption-key` file (0600) next to the SQLite database, not inside it. Older-installs that kept the key in the database are migrated to this file on first-boot. Do not rely on that fallback with real provider keys; set `ENCRYPTION_KEY`.--Request analytics are retained for 90 days or 100000 request rows by default,-whichever limit prunes first. Set `REQUEST_ANALYTICS_RETENTION_DAYS=0` or-`REQUEST_ANALYTICS_MAX_ROWS=0` in `.env` to disable either retention limit.--Open http://localhost:5173 (the Vite dev UI), add your provider keys on the **Keys** page, reorder the **Fallback Chain** to taste, and grab your unified API key from the **Keys** page header. That unified key is what you point your OpenAI SDK at.--### Declarative startup config--For repeatable Docker/server installs, FreeLLMAPI can apply a JSON config on-every boot. Set `FREEAPI_CONFIG_PATH=/path/to/freellmapi.config.json` or put the-same JSON in `FREEAPI_CONFIG_JSON`. The config is idempotent: existing keys,-custom providers, model edits, fallback rows, and routing settings are updated-instead of duplicated.--```json-{- "keys": [- { "platform": "groq", "key": "gsk_...", "label": "main" },- { "platform": "google", "key": "AIza...", "enabled": true }- ],- "customProviders": [- {- "baseUrl": "http://host.docker.internal:11434/v1",- "label": "Ollama",- "models": [- { "model": "llama3.1:8b", "displayName": "Local Llama", "supportsTools": true }- ]- }- ],- "models": [- {- "platform": "groq",- "modelId": "llama-3.3-70b-versatile",- "displayName": "Llama 3.3 70B",- "supportsTools": true,- "fallbackEnabled": true- }- ],- "routing": { "strategy": "balanced" }-}-```--> **Reaching the dev UI from another device on your LAN?** Use `npm run dev:lan` — it passes `--host` through to Vite, which then prints a `Network: http://<your-ip>:5173` URL you can open from a phone or another machine. (Plain `npm run dev -- --host` does *not* work here: the root `dev` script is a `concurrently` wrapper, so the flag never reaches Vite.) API calls go through Vite's dev proxy, so no extra server config is needed.--For a production build without Docker:--```bash-npm run build-node server/dist/index.js # server + dashboard both served on :3001-```--## Docker--FreeLLMAPI publishes a single production image that contains the Express server and the built React dashboard:--```bash-docker pull ghcr.io/tashfeenahmed/freellmapi:latest # or pin a release, e.g. :v1.2.3-```--The image is multi-arch (`linux/amd64` + `linux/arm64`, so it runs on a Raspberry Pi). Published tags: `latest` (default branch), `v*.*.*` (git release tags), and `sha-<commit>`.--The included `docker-compose.yml` is the recommended install path:--```bash-docker compose up -d-docker compose logs -f freellmapi-```--By default the container's port is bound to `127.0.0.1` (localhost only). To reach the dashboard/API from another machine on your network, publish it on all interfaces with `HOST_BIND=0.0.0.0 docker compose up -d` — only on a trusted LAN, since the proxy is single-user.--SQLite data is stored in the `freellmapi-data` volume at `/app/server/data`.-Keep the same `.env` `ENCRYPTION_KEY` and volume when upgrading, because-provider keys are encrypted at rest. If your host only persists a specific-directory, set `FREEAPI_DB_PATH=/that/path/freellmapi.db`.--On hosts with ephemeral disks, configure an encrypted backup target:+On Windows, the easiest path is the desktop **[`.exe` installer from Releases](https://github.com/tashfeenahmed/freellmapi/releases/latest)** (below). On Android, see the experimental [Termux guide](docs/install/android-termux.md).-```env-FREEAPI_DB_BACKUP_PATH=/app/server/data/freellmapi.db.backup-# or:-FREEAPI_DB_BACKUP_URL=https://example.com/freellmapi.db.backup-FREEAPI_DB_BACKUP_TOKEN=optional-bearer-token-FREEAPI_DB_BACKUP_KEY=64-char-hex-backup-key-FREEAPI_DB_BACKUP_INTERVAL_MS=300000-```--When the database file is missing at startup, FreeLLMAPI restores the backup-before migrations run. While the server is running it uploads a fresh encrypted-backup periodically. If `FREEAPI_DB_BACKUP_KEY` is omitted, the app uses-`ENCRYPTION_KEY` for the backup envelope too.--More Docker operations and examples live in [docker/README.md](./docker/README.md).+Everything else — Docker Compose, local development, declarative startup config, production builds, LAN access, and backups — is in **[docs/install.md](docs/install.md)**.## Desktop app-A native menu-bar app lives in [`desktop/`](./desktop): the entire router +-dashboard running locally from your tray, with a glass popover showing live-request stats.+A native menu-bar app lives in [`desktop/`](./desktop): the entire router + dashboard running locally from your tray, with a glass popover showing live request stats.-**[Download from Releases](https://github.com/tashfeenahmed/freellmapi/releases/latest)** — the macOS `.dmg` and the Windows `.exe` installer are built and attached to every release by the [`desktop-release`](.github/workflows/desktop-release.yml) workflow. Or build it from this repo in a few minutes:--> **Note for Windows users building from source:** Building the desktop app requires compiling native SQLite modules for Electron. You must have [Visual Studio Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) installed (specifically the "Desktop development with C++" workload) and Python installed before running `npm install`.--```bash-npm install-npm install --prefix desktop # install desktop dependencies-npm run desktop:dist # macOS → desktop/dist-electron/FreeLLMAPI-…-arm64.dmg-npm run desktop:dist:win # Windows → "desktop/dist-electron/FreeLLMAPI Setup ….exe"-```--> Locally built apps are unsigned, so Windows SmartScreen may warn on first run-> ("More info" → "Run anyway"); the macOS build launches without Gatekeeper prompts.-> Full instructions in [desktop/README.md](./desktop/README.md).--### Credentials and where your data lives--The desktop app has **no username or password to set up**. Unlike the server-(which gates its dashboard behind an email + password account), the desktop-build signs the dashboard in automatically with a hidden local account, so-you're never prompted for credentials and never need one.--The only credential you need is your **unified API key** — the-`freellmapi-…` token your OpenAI/Anthropic client points at. Get it from:--- the tray popover — click the tray icon, then **Copy Key**, or-- the dashboard **Keys** page header (tray → **Open Dashboard**).+**[Download from Releases](https://github.com/tashfeenahmed/freellmapi/releases/latest)** — the macOS `.dmg` and the Windows `.exe` installer are attached to every release. No account or password to set up: the only credential you need is the unified API key from the tray popover. Build-from-source steps and where your data lives: [docs/install.md](docs/install.md#desktop-app).-You do not need to open or edit `freeapi.db` by hand.--Your settings and data live in one folder per OS (copy it to migrate to-another machine or into a container):+## Works with OpenAI-compatible clients-| OS | Location |-|----|----------|-| Windows | `%APPDATA%\FreeLLMAPI\` (e.g. `C:\Users\<you>\AppData\Roaming\FreeLLMAPI\`) |-| macOS | `~/Library/Application Support/FreeLLMAPI/` |-| Linux | `~/.config/FreeLLMAPI/` |+Anything that can target an OpenAI-compatible base URL works: set it to `http://localhost:3001/v1` with the unified key from the dashboard. **Claude Code**, **Codex CLI**, **Cline / Roo Code**, **Continue** (including inline autocomplete), **Aider**, **opencode**, and **Cursor** each have a short recipe in **[docs/clients.md](docs/clients.md)** — and the router doubles as an MCP server your agents can introspect mid-session.-That folder holds `freeapi.db` (all keys, models, settings, encrypted at rest)-and `config.json` (window/theme/port/LAN preferences). Copy both to move an-install. For the server (non-desktop) deployment, the equivalent state is the-`.env` file and the SQLite DB at `server/data/freeapi.db` (or wherever-`FREEAPI_DB_PATH` points).+FreeLLMAPI is local-first and single-user by design. Your provider keys stay in your SQLite database, encrypted at rest, and requests go from your machine to the upstream providers you enabled.## Languages@@ -395,47 +168,6 @@ add a language, copy `en.json`, translate the values, and register the locale instrings); `npm test` checks every locale for key/placeholder parity — PRswelcome.-## Works with OpenAI-compatible clients--Any client that can target an OpenAI-compatible base URL can use FreeLLMAPI:--- **LangChain, LlamaIndex, official OpenAI SDKs**: set `base_url` to- `http://localhost:3001/v1` and use the unified key from the dashboard.-- **Local GPU boxes**: add custom OpenAI-compatible endpoints for Ollama,- llama.cpp, LM Studio, vLLM, or an internal gateway.--### Coding agents--Every recipe below is the same three facts in a different config file: base URL-`http://localhost:3001/v1`, the unified key from the dashboard's Keys page, and-a model (`auto` lets the router pick).--| Agent | Setup |-| --- | --- |-| **Claude Code** | `ANTHROPIC_BASE_URL=http://localhost:3001` + `ANTHROPIC_AUTH_TOKEN=<unified key>` — full walkthrough in [Anthropic / Claude clients](#anthropic--claude-clients) |-| **Codex CLI** | add a provider in `~/.codex/config.toml` with `base_url = "http://localhost:3001/v1"` and its `env_key` pointing at the unified key — the `/v1/responses` surface it needs is implemented |-| **Cline / Roo Code** | provider type "OpenAI Compatible", base URL `http://localhost:3001/v1`, unified key, model `auto` (or any id from `/v1/models`) |-| **Continue** | `apiBase: http://localhost:3001/v1` in its config; inline autocomplete works too via the legacy `/v1/completions` surface |-| **Aider** | `OPENAI_API_BASE=http://localhost:3001/v1` + `OPENAI_API_KEY=<unified key>`, then `aider --model openai/auto` |-| **opencode** | OpenAI-compatible provider with the same base URL and key |-| **Cursor** | paste the unified key under a custom OpenAI base URL — but note Cursor verifies and calls the API **from its servers**, so your router must be reachable from the internet (a tunnel or a host with a public address), not just `localhost` |--On top of inference, the router is an **MCP server**: agents can introspect it mid-session-(usable models and the params each one honors, provider health, usage and cache stats,-routing strategy). For Claude Code:--```bash-claude mcp add --transport http freellmapi http://localhost:3001/mcp \- --header "Authorization: Bearer freellmapi-your-unified-key"-```--Any MCP client that speaks Streamable HTTP works the same way: point it at `/mcp` with the-unified key as a Bearer token.--FreeLLMAPI is local-first and single-user by design. Your provider keys stay in-your SQLite database, encrypted at rest, and requests go from your machine to the-upstream providers you enabled.-## Premium (live catalog)The router keeps its model catalog fresh on its own: it pulls a signed catalog@@ -468,10 +200,6 @@ router stays fully self-hosted either way.## Using the API-Any OpenAI-compatible client works (Anthropic / Claude clients too — see [Anthropic / Claude clients](#anthropic--claude-clients)). Examples:--**Python**-```pythonfrom openai import OpenAI@@ -481,237 +209,14 @@ client = OpenAI()resp = client.chat.completions.create(- model="auto", # let the router pick; or specify e.g. "gemini-2.5-flash"+ model="auto", # let the router pick; or "auto:fast", "auto:smart", a profile, or a model idmessages=[{"role": "user", "content": "Summarise the fall of Rome in one sentence."}],)print(resp.choices[0].message.content)print("Routed via:", resp.headers.get("x-routed-via"))```-**curl**--```bash-curl http://localhost:3001/v1/chat/completions \- -H "Authorization: Bearer freellmapi-your-unified-key" \- -H "Content-Type: application/json" \- -d '{- "model": "auto",- "messages": [{"role": "user", "content": "hi"}]- }'-```--**Routing strategies (`auto:*`)**--Plain `auto` follows your active fallback chain. Add a suffix to steer a single request instead — no dashboard changes needed:--- `auto:smart` — favor the highest-intelligence models-- `auto:fast` — favor measured speed (throughput and time-to-first-byte)-- `auto:cheap` — budget-leaning; currently the same blend as `balanced` (everything in the pool is already free)-- `auto:reliable` — favor recent success rate-- `auto:balanced` — the default blend (reliability first, speed and intelligence split the rest)--These rank **every enabled model**, ignoring your chain order. Common synonyms resolve too (`auto:fastest`, `auto:speed`, `auto:smartest`, `auto:cheapest`, `auto:budget`, …), and the whole model string is case-insensitive.--```bash-curl http://localhost:3001/v1/chat/completions \- -H "Authorization: Bearer freellmapi-your-unified-key" \- -H "Content-Type: application/json" \- -d '{- "model": "auto:fast",- "messages": [{"role": "user", "content": "hi"}]- }'-```--`auto:<profile-name>` routes through a named profile's chain instead of the active one, so different tools can use different chains through the same key:--```bash-curl http://localhost:3001/v1/chat/completions \- -H "Authorization: Bearer freellmapi-your-unified-key" \- -H "Content-Type: application/json" \- -d '{- "model": "auto:coding",- "messages": [{"role": "user", "content": "Write a binary search in Rust."}]- }'-```--An unknown profile name returns a clear `400` rather than silently falling back. Profiles are the named fallback chains from **Model profiles** (see [Features](#features)) — create and switch them from the dashboard; whichever is active is what plain `auto` uses.--**Streaming**--```python-stream = client.chat.completions.create(- model="auto",- messages=[{"role": "user", "content": "Stream me a haiku about SQLite."}],- stream=True,-)-for chunk in stream:- print(chunk.choices[0].delta.content or "", end="", flush=True)-```--**VS Code ghost-text autocomplete (Continue)**--FreeLLMAPI exposes `/v1/completions` for editor autocomplete clients that send legacy OpenAI prompt/suffix requests. Example Continue config:--```yaml-models:- - name: FreeLLMAPI Autocomplete- provider: openai- model: auto- apiBase: http://localhost:3001/v1- apiKey: freellmapi-your-unified-key- useLegacyCompletionsEndpoint: true- roles:- - autocomplete-```--**Tool calling**--Pass OpenAI-style `tools` and `tool_choice`; the assistant response round-trips back through the proxy exactly like the OpenAI API. Multi-step flows (assistant `tool_calls` → `tool` role follow-up → final answer) work across every provider the router can reach.--```python-tools = [{- "type": "function",- "function": {- "name": "get_weather",- "description": "Get current weather for a city.",- "parameters": {- "type": "object",- "properties": {"city": {"type": "string"}},- "required": ["city"],- },- },-}]--# 1. Model asks for a tool call-first = client.chat.completions.create(- model="auto",- messages=[{"role": "user", "content": "What's the weather in Karachi?"}],- tools=tools,- tool_choice="required",-)-call = first.choices[0].message.tool_calls[0]--# 2. You execute the tool, feed the result back-final = client.chat.completions.create(- model="auto",- messages=[- {"role": "user", "content": "What's the weather in Karachi?"},- first.choices[0].message,- {"role": "tool", "tool_call_id": call.id, "content": '{"temp_c": 32, "cond": "sunny"}'},- ],- tools=tools,-)-print(final.choices[0].message.content)-```--**Gemini Google Search grounding**--Google's models can ground their answers in live Google Search results. Since the OpenAI wire format has no way to express that, request a tool named `google_search` and the Google provider translates it into Gemini's native grounding tool. It can be sent on its own or alongside your normal function tools.--```python-resp = client.chat.completions.create(- model="gemini-2.5-flash", # pin a Google model so the request routes there- messages=[{"role": "user", "content": "Who won the F1 race this weekend?"}],- tools=[{"type": "function", "function": {"name": "google_search", "parameters": {}}}],-)-print(resp.choices[0].message.content)-```--**Vision / image input**--Send images with the standard OpenAI `image_url` content blocks (base64 `data:` URLs or `http(s)` URLs). When a request contains an image, the router restricts itself to **vision-capable models** and ignores text-only ones. Vision models are tagged with a **Vision** badge on the Fallback Chain page; the current set includes Gemini (2.5 / 3.x), Llama 4 Scout/Maverick (Groq, NVIDIA), GLM-4.6V Flash (Z.ai), Nemotron Nano 12B VL (OpenRouter), and GitHub's GPT-4o / GPT-4.1.--```python-resp = client.chat.completions.create(- model="auto", # auto-routes to a vision model- messages=[{- "role": "user",- "content": [- {"type": "text", "text": "What's in this image?"},- {"type": "image_url", "image_url": {"url": "data:image/png;base64,<...>"}},- ],- }],-)-print(resp.choices[0].message.content)-```--If no vision-capable model is enabled in your Fallback Chain, an image request returns a clear `422` (`code: "no_vision_model"`) rather than silently dropping the image. (Image input on `/v1/responses` isn't supported yet — use `/v1/chat/completions`.)--Works with `stream=True` as well — you'll get `delta.tool_calls` chunks followed by a `finish_reason: "tool_calls"` close. Under the hood, OpenAI-compatible providers (Groq, Cerebras, Mistral, OpenRouter, GitHub Models, HuggingFace, Cloudflare, Cohere compat) get the request passed through; Gemini requests get translated into Google's `functionDeclarations` / `functionResponse` shape and the response is translated back.--Every response carries an `X-Routed-Via: <platform>/<model>` header so you can see which provider actually served each call. If a request fell over between providers, you'll also see `X-Fallback-Attempts: N`.--### Embeddings--`/v1/embeddings` is OpenAI-compatible, with one deliberate difference from chat routing: **failover never crosses models.** Vectors from different models live in incompatible spaces — silently switching models would corrupt any vector store built on top of the proxy. So embeddings route by **family** (one model identity + dimension), and failover only walks the providers serving that same family.--```python-resp = client.embeddings.create(- model="auto", # default family; or a family name like "bge-m3"- input=["the quick brown fox", "pack my box with five dozen liquor jugs"],-)-print(len(resp.data), "vectors of", len(resp.data[0].embedding), "dims")-```--```bash-curl http://localhost:3001/v1/embeddings \- -H "Authorization: Bearer freellmapi-your-unified-key" \- -H "Content-Type: application/json" \- -d '{"model": "auto", "input": "hello world"}'-```--`model` accepts `auto` (the configured default family), a family name, or a provider-specific model id (which resolves to its family). Available families:--| Family (`model`) | Dims | Providers (failover order) |-| --- | --- | --- |-| `gemini-embedding-001` *(default)* | 3072 | Google |-| `text-embedding-3-large` | 3072 | GitHub Models |-| `text-embedding-3-small` | 1536 | GitHub Models |-| `embed-v4.0` | 1536 | Cohere |-| `bge-m3` | 1024 | Cloudflare → Hugging Face |-| `qwen3-embedding-0.6b` | 1024 | Cloudflare |-| `nv-embedqa-e5-v5` | 1024 | NVIDIA |-| `llama-nemotron-embed-1b-v2` | 2048 | NVIDIA |-| `llama-nemotron-embed-vl-1b-v2` | 2048 | NVIDIA → OpenRouter |-| `embeddinggemma-300m` | 768 | Cloudflare |--The default family, per-provider toggles, and priorities live on the dashboard's **Models → Embeddings** page. Pick your family once and stick with it for a given vector store — that's the whole point of the family model.--### Anthropic / Claude clients--FreeLLMAPI also speaks Anthropic's Messages API, so anything built for Claude — including **Claude Code** and the official Anthropic SDKs — can run against your free pool. Point the client at your server's **origin** (Anthropic clients append `/v1/messages` themselves) and authenticate with your unified key. Both `x-api-key` and `Authorization: Bearer` are accepted.--```bash-curl http://localhost:3001/v1/messages \- -H "x-api-key: freellmapi-your-unified-key" \- -H "anthropic-version: 2023-06-01" \- -H "Content-Type: application/json" \- -d '{- "model": "claude-sonnet-4-5",- "max_tokens": 256,- "messages": [{"role": "user", "content": "hi"}]- }'-```--Claude model names map to your free pool on the **Keys → Anthropic** tab: each family (`default`, `opus`, `sonnet`, `haiku`) routes to `auto` (the router picks a free model) or a model you pin. `POST /v1/messages/count_tokens` and a content-negotiated `GET /v1/models` (Anthropic shape when `anthropic-version` is sent) are implemented too. Streaming, system prompts, tool use, and image input all translate across the same router as the OpenAI endpoints.--**Claude Code** — point it at your server and start it:--*On macOS / Linux (Bash):*-```bash-export ANTHROPIC_BASE_URL=http://localhost:3001-export ANTHROPIC_AUTH_TOKEN=freellmapi-your-unified-key # NOT ANTHROPIC_API_KEY-claude-```--*On Windows (PowerShell):*-```powershell-$env:ANTHROPIC_BASE_URL="http://localhost:3001"-$env:ANTHROPIC_AUTH_TOKEN="freellmapi-your-unified-key"-claude-```--> Use `ANTHROPIC_AUTH_TOKEN` (sent as a Bearer token), **not** `ANTHROPIC_API_KEY` — Claude Code treats a set `ANTHROPIC_API_KEY` as a conflicting first-party credential and refuses to start.+Streaming, the `auto:*` routing strategies, tool calling, vision input, Gemini Google Search grounding, embeddings, and the Anthropic Messages surface — with curl and Python examples for each — are all in **[docs/api.md](docs/api.md)**. Every response carries an `X-Routed-Via: <platform>/<model>` header so you can see which provider actually served it.## Screenshots@@ -743,76 +248,11 @@ Request volume, success rate, tokens in and out, average latency, and per-provid-```-┌──────────────────┐ Bearer freellmapi-… ┌─────────────────────────┐-│ OpenAI SDK / │ ──────────────────────▶ │ Express proxy (:3001) │-│ curl / any │ ◀────────────────────── │ /v1/chat/completions │-│ OpenAI client │ streamed tokens └────────────┬────────────┘-└──────────────────┘ │- ▼- ┌────────────────────────────────────────────────┐- │ Router │- │ 1. Pick highest-priority model that │- │ (a) has a healthy key and │- │ (b) is under all its rate limits. │- │ 2. Decrypt key, call provider SDK. │- │ 3. On 429/5xx → cooldown + retry next model. │- └────────────────────────────────────────────────┘- │- ┌──────────────┬────────────┬──────────┴─────────┬─────────────┬──────────┐- ▼ ▼ ▼ ▼ ▼ ▼- Google Groq Cerebras OpenRouter HF …22 more-```--- **Router** (`server/src/services/router.ts`) — picks a model per request.-- **Rate-limit ledger** (`server/src/services/ratelimit.ts`) — in-memory RPM/RPD/TPM/TPD counters backed by SQLite, with cooldowns on 429s.-- **Provider adapters** (`server/src/providers/*.ts`) — one file per provider, implementing the `Provider` base class: `chatCompletion()` and `streamChatCompletion()`.-- **Health service** (`server/src/services/health.ts`) — periodic probe keeps key status fresh.-- **Dashboard** (`client/`) — React + Vite + shadcn/ui admin surface.-- **Storage** — SQLite (`better-sqlite3`) with AES-256-GCM envelope encryption for keys.--## Context Handoff--When FreeLLMAPI falls over to a different model mid-conversation (quota, rate limit, cooldown), the new model has no idea it is picking up someone else's task. **Context handoff** adds a single compact `system` message to the outbound request that tells the new model exactly that:--```-FreeLLMAPI context handoff:-You are taking over an ongoing conversation from another model (groq:llama-3 → google:gemini-flash).-Continue the user's task using the conversation context already provided in this request.-Do not restart the task, re-ask already answered setup questions, or discard prior tool results.-Respect the user's latest message as the highest-priority instruction.--Recent session summary:-User: …-Assistant: …-```--**Enable it in `.env`:**--```env-FREELLMAPI_CONTEXT_HANDOFF=on_model_switch-```--**How it works:**--- Messages per session are stored in memory (TTL: 3 hours).-- Only injected when the selected model changes for a given session key.-- Not injected on the first request, on same-model continuations, or if a handoff message is already present.-- Session key: `X-Session-Id` header if present, otherwise SHA-1 of the first user message (same as sticky sessions).-- Storage is in-memory only. Nothing is written to disk or logged.--> **Important:** Context Handoff improves continuity for conversations routed through FreeLLMAPI. It cannot recover provider-internal hidden state or messages that were never sent to the proxy.+One request in, the best free model out: the router picks the highest-priority model with a healthy key that's under all its rate limits, decrypts the key in memory, and calls the provider — on a 429/5xx it cools that key down and retries the next model in your chain. The component walkthrough, routing internals, and operational details live in **[docs/architecture.md](docs/architecture.md)**.## Limitations-Stacking free tiers has real trade-offs. Be honest with yourself about them:--- **No frontier models.** The free-tier catalog tops out around Llama 3.3 70B, GLM-4.5, Qwen 3 Coder, and Gemini 2.5 Pro. You will not get GPT-5 or Claude Opus class reasoning through this. For hard problems, pay for a real API.-- **Intelligence degrades as the day progresses.** Your top-ranked models (usually Gemini 2.5 Pro, GPT-4o via GitHub Models) have the lowest daily caps. Once they hit their limits, the router falls down your priority chain to smaller/weaker models. Expect the effective intelligence of the endpoint to drop in the late hours of each day — then reset at UTC midnight.-- **Latency is highly variable.** Cerebras and Groq are extremely fast; others are not. You get whichever one is available.-- **Free tiers can change without notice.** Providers regularly tighten, loosen, or remove free tiers. When that happens you'll see 429s or auth errors until the catalog update reaches you — live-feed installs get those fixes within days, free installs on the 30-day trail. Re-seed scripts live in `server/src/scripts/`.-- **No SLA, by definition.** If you need reliability, use a paid provider with a contract.-- **Local-first.** There's no multi-tenant auth. Run this for yourself; don't expose it to the internet.+Stacking free tiers has real trade-offs: no frontier models, variable latency, no SLA — and the effective intelligence of the endpoint dips late in the day as top models hit their daily caps, then resets at UTC midnight. Read the honest list in **[docs/architecture.md#limitations](docs/architecture.md#limitations)** before building anything real on this.## Contributing@@ -824,26 +264,7 @@ Contributors very welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for the dev lo- **Dashboard polish** — charts on the Analytics page, key rotation UX, batch import of keys from `.env`.- **Docs** — more examples, client library snippets for Go/Rust/etc., a deployment recipe for Docker or Fly.-**Development loop:**--```bash-npm install-npm run dev # server on :3001, dashboard on :5173, both with HMR-npm test # server vitest; also runs client tests if the workspace adds them-npm run build # compile server and dashboard-```--PRs should include a test, keep the existing test suite green, and match the `.editorconfig` / tsconfig defaults already in the repo. See [CONTRIBUTING.md](./CONTRIBUTING.md) for the full contributor workflow.--### Database Migrations--In local development, apply pending migrations with:--```bash-NODE_ENV=development npm run db:migration:up-```--See [CONTRIBUTING.md](./CONTRIBUTING.md) for the full migration CLI and workflow.+`npm install && npm run dev` gets you the server on :3001 and the dashboard on :5173, both with HMR. PRs should include a test, keep the existing suite green (`npm test`), and match the `.editorconfig` / tsconfig defaults already in the repo. Database migration workflow and the full contributor loop are in [CONTRIBUTING.md](./CONTRIBUTING.md).### Contributors@@ -909,7 +330,6 @@ See [CONTRIBUTING.md](./CONTRIBUTING.md) for the full migration CLI and workflow<a href="https://github.com/SeanPedersen"><img src="https://images.weserv.nl/?url=github.com/SeanPedersen.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@SeanPedersen" /></a><a href="https://github.com/andersmmg"><img src="https://images.weserv.nl/?url=github.com/andersmmg.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@andersmmg" /></a><a href="https://github.com/chirag127"><img src="https://images.weserv.nl/?url=github.com/chirag127.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@chirag127" /></a>-<a href="https://github.com/jasnoorgill"><img src="https://images.weserv.nl/?url=github.com/jasnoorgill.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@jasnoorgill" /></a><a href="https://github.com/allababbot"><img src="https://images.weserv.nl/?url=github.com/allababbot.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@allababbot" /></a><a href="https://github.com/johan-droid"><img src="https://images.weserv.nl/?url=github.com/johan-droid.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@johan-droid" /></a><a href="https://github.com/redenfire"><img src="https://images.weserv.nl/?url=github.com/redenfire.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@redenfire" /></a>@@ -922,13 +342,11 @@ See [CONTRIBUTING.md](./CONTRIBUTING.md) for the full migration CLI and workflow<a href="https://github.com/AmiroKD"><img src="https://images.weserv.nl/?url=github.com/AmiroKD.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@AmiroKD" /></a><a href="https://github.com/ecryptomillionaire-dev"><img src="https://images.weserv.nl/?url=github.com/ecryptomillionaire-dev.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@ecryptomillionaire-dev" /></a><a href="https://github.com/4riful"><img src="https://images.weserv.nl/?url=github.com/4riful.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@4riful" /></a>-<a href="https://github.com/cagedbird043"><img src="https://images.weserv.nl/?url=github.com/cagedbird043.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@cagedbird043" /></a><a href="https://github.com/fix2015"><img src="https://images.weserv.nl/?url=github.com/fix2015.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@fix2015" /></a><a href="https://github.com/iisyw"><img src="https://images.weserv.nl/?url=github.com/iisyw.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@iisyw" /></a><a href="https://github.com/xsfhacg"><img src="https://images.weserv.nl/?url=github.com/xsfhacg.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@xsfhacg" /></a><a href="https://github.com/noobix"><img src="https://images.weserv.nl/?url=github.com/noobix.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@noobix" /></a><a href="https://github.com/nandukmelath"><img src="https://images.weserv.nl/?url=github.com/nandukmelath.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@nandukmelath" /></a>-<a href="https://github.com/coffcoe"><img src="https://images.weserv.nl/?url=github.com/coffcoe.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@coffcoe" /></a><a href="https://github.com/NirvanaCh7"><img src="https://images.weserv.nl/?url=github.com/NirvanaCh7.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@NirvanaCh7" /></a><a href="https://github.com/Mohamed3nan"><img src="https://images.weserv.nl/?url=github.com/Mohamed3nan.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@Mohamed3nan" /></a><a href="https://github.com/Arman-Espiar"><img src="https://images.weserv.nl/?url=github.com/Arman-Espiar.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@Arman-Espiar" /></a>@@ -938,35 +356,12 @@ See [CONTRIBUTING.md](./CONTRIBUTING.md) for the full migration CLI and workflow<a href="https://github.com/emv33"><img src="https://images.weserv.nl/?url=github.com/emv33.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@emv33" /></a><a href="https://github.com/ousamabenyounes"><img src="https://images.weserv.nl/?url=github.com/ousamabenyounes.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@ousamabenyounes" /></a>-## Terms of Service review--A self-hosted, single-user, personal-use setup was re-reviewed against each provider's ToS (May 2026). Summary:--| Provider | Verdict | Notes |-|---|---|---|-| Google Gemini | ⚠️ Caution | March 2026 ToS narrows scope to *"professional or business purposes, not for consumer use"* — a self-hosted developer proxy is still defensible, but the clause is new. |-| Groq | ✅ Likely OK | GroqCloud Services Agreement permits Customer Application integration. |-| Cerebras | ✅ Likely OK | Permitted; explicitly forbids selling/transferring API keys. |-| Mistral | ✅ Likely OK | APIs allowed for personal/internal business use. |-| OpenRouter | ✅ Likely OK | April 2026 ToS sharpens the no-resale / no-competing-service clause; private single-user proxy still fine. |-| Cloudflare Workers AI | ⚠️ Ambiguous | No anti-proxy clause; covered by general Self-Serve Subscription Agreement. |-| NVIDIA NIM | ⚠️ Caution | Trial ToS §1.2 / §1.4: *"evaluation only, not production."* Free access is a recurring 40 RPM rate limit (the 2025 credit system was discontinued), but the evaluation-only scope stands. |-| GitHub Models | ⚠️ Caution | Free tier explicitly scoped to *"experimentation"* and *"prototyping."* |-| Cohere | ❌ Avoid | Terms §14 still forbids *"personal, family or household purposes."* |-| Zhipu (open.bigmodel.cn) | ✅ Likely OK | Personal/non-commercial research carve-out still in the platform docs. |-| Z.ai (api.z.ai) | ⚠️ Caution | New row — Singapore entity (distinct from Zhipu CN). §III.3(l) anti-traffic-redirect clause could plausibly be read against a proxy; no explicit personal-use carve-out. |-| Ollama Cloud | ✅ Likely OK | New row — Free plan permits cloud-model access (1 concurrent, 5-hour session caps). No anti-proxy / anti-resale clauses found. *(Integration tracked in #14.)* |-| OVH AI Endpoints | ✅ Likely OK | New row (June 2026) — anonymous access is officially documented (2 req/min per IP per model). OVH reserves the right to introduce token/consumption caps. |-| AI Horde | ✅ Likely OK | New row (June 2026) — a free, community-powered commons run by the Haidra non-profit; anonymous use is officially supported (key `0000000000`, lowest queue priority). No anti-proxy / anti-resale clause. The OpenAI proxy is a pilot and may be restricted by usage. *(Integration #345.)* |--Rules of thumb that keep most providers happy: **one account per provider**, **no reselling**, **no sharing your endpoint with other humans**, **don't hammer a free tier as a paid production backend**. This is informational, not legal advice — read each provider's ToS and make your own call.--Removed since the April 2026 review: Hugging Face, Moonshot, and MiniMax direct integrations were dropped from the catalog (HF — tool-call format issues; Moonshot — moved to paid only; MiniMax — superseded by the OpenRouter `minimax/minimax-m2.5:free` route).-## Disclaimer**This project is for personal experimentation and learning, not production.** Free tiers exist so developers can prototype against them; they aren't a stable, supported inference substrate and shouldn't be treated as one. If you build something real on top of FreeLLMAPI, swap in a paid API before you ship. Your relationship with each upstream provider is governed by the terms you accepted when you created your account — those terms still apply when the traffic is proxied through this project, and you're responsible for complying with them.+How each provider's ToS views a personal, single-user proxy — reviewed provider by provider in May 2026 — is in [docs/architecture.md#terms-of-service-review](docs/architecture.md#terms-of-service-review).+## License[MIT](./LICENSE)diff --git a/docs/README.md b/docs/README.mdindex 375c316f..9a970923 100644--- a/docs/README.md+++ b/docs/README.md@@ -1,34 +1,21 @@# FreeLLMAPI documentation-This index points to the focused guides in the repository. The root [README](../README.md) remains the product overview and complete API reference.+The root [README](../README.md) is the product overview; the detailed guides live here.-## Install and deploy+## Guides-- [Quick start](../README.md#quick-start) — Docker one-liner, manual Docker Compose, and local development.-- [Docker deployment](../docker/README.md) — container configuration and persistent storage.-- [Desktop app](../desktop/README.md) — build and package the Electron application.-- [Android with Termux](install/android-termux.md) — experimental local installation using Node's built-in SQLite driver.--## Configure and operate--- [Declarative startup configuration](../README.md#declarative-startup-config) — configure keys and custom providers from environment variables.-- [Credentials and local data](../README.md#credentials-and-where-your-data-lives) — desktop paths and credential storage.-- [Premium live catalog](../README.md#premium-live-catalog) — catalog update behavior and licensing.-- [Database migrations](../server/src/db/README.md) — create, apply, inspect, and roll back schema migrations.+- **[Install & deploy](install.md)** — quick start, Docker Compose, local development, declarative startup config, the Docker image, backups, the desktop app, and where your data lives.+- **[API reference](api.md)** — chat completions, `auto:*` routing strategies, streaming, tool calling, vision, Gemini Google Search grounding, embeddings, response headers, and the Anthropic Messages surface.+- **[Clients & coding agents](clients.md)** — OpenAI-compatible clients, recipes for Claude Code / Codex CLI / Cline / Continue / Aider / opencode / Cursor, the MCP server, editor autocomplete, and Context Handoff.+- **[Architecture & internals](architecture.md)** — how the router works, routing and operational details, what's not supported, honest limitations, and the provider Terms-of-Service review.-## Integrate clients--- [OpenAI-compatible clients](../README.md#works-with-openai-compatible-clients) — SDK and tool configuration.-- [Coding agents](../README.md#coding-agents) — Codex CLI, Claude Code, OpenCode, and MCP clients.-- [Using the API](../README.md#using-the-api) — request examples for chat, streaming, tools, images, and audio.-- [Embeddings](../README.md#embeddings) — embedding-model routing and custom endpoints.-- [Anthropic and Claude clients](../README.md#anthropic--claude-clients) — Messages API compatibility.--## Develop and contribute+## More+- [Android with Termux](install/android-termux.md) — experimental local installation using Node's built-in SQLite driver.+- [Docker deployment](../docker/README.md) — container configuration and persistent storage.+- [Desktop app](../desktop/README.md) — build and package the Electron application.- [Contributor guide](../CONTRIBUTING.md) — development loop, testing expectations, and contribution policy.-- [How the router works](../README.md#how-it-works) — architecture and fallback behavior.-- [Database migration guide](../server/src/db/README.md) — migration CLI and conventions.+- [Database migrations](../server/src/db/README.md) — create, apply, inspect, and roll back schema migrations.## Website assets in this directorydiff --git a/docs/api.md b/docs/api.mdnew file mode 100644index 00000000..9ad3df93--- /dev/null+++ b/docs/api.md@@ -0,0 +1,258 @@+# API reference++[← Back to README](../README.md) · [Documentation index](README.md)++Any OpenAI-compatible client works (Anthropic / Claude clients too — see [Anthropic / Claude clients](#anthropic--claude-clients)). Base URL `http://localhost:3001/v1`, unified key from the dashboard's Keys page. An interactive OpenAPI viewer covering every proxy endpoint is served at `GET /v1/docs`; the spec itself lives at `GET /v1/openapi.json`.++- [Chat completions](#chat-completions)+- [Routing strategies (`auto:*`)](#routing-strategies-auto)+- [Streaming](#streaming)+- [Tool calling](#tool-calling)+- [Gemini Google Search grounding](#gemini-google-search-grounding)+- [Vision / image input](#vision--image-input)+- [Images & text-to-speech](#images--text-to-speech)+- [Fusion (multi-model synthesis)](#fusion-multi-model-synthesis)+- [Response headers](#response-headers)+- [Embeddings](#embeddings)+- [Anthropic / Claude clients](#anthropic--claude-clients)++## Chat completions++**Python**++```python+from openai import OpenAI++client = OpenAI(+ base_url="http://localhost:3001/v1",+ api_key="freellmapi-your-unified-key",+)++resp = client.chat.completions.create(+ model="auto", # let the router pick; or specify e.g. "gemini-2.5-flash"+ messages=[{"role": "user", "content": "Summarise the fall of Rome in one sentence."}],+)+print(resp.choices[0].message.content)+print("Routed via:", resp.headers.get("x-routed-via"))+```++**curl**++```bash+curl http://localhost:3001/v1/chat/completions \+ -H "Authorization: Bearer freellmapi-your-unified-key" \+ -H "Content-Type: application/json" \+ -d '{+ "model": "auto",+ "messages": [{"role": "user", "content": "hi"}]+ }'+```++## Routing strategies (`auto:*`)++Plain `auto` follows your active fallback chain. Add a suffix to steer a single request instead — no dashboard changes needed:++- `auto:smart` — favor the highest-intelligence models+- `auto:fast` — favor measured speed (throughput and time-to-first-byte)+- `auto:cheap` — budget-leaning; currently the same blend as `balanced` (everything in the pool is already free)+- `auto:reliable` — favor recent success rate+- `auto:balanced` — the default blend (reliability first, speed and intelligence split the rest)++These rank **every enabled model**, ignoring your chain order. Common synonyms resolve too (`auto:fastest`, `auto:speed`, `auto:smartest`, `auto:cheapest`, `auto:budget`, …), and the whole model string is case-insensitive.++```bash+curl http://localhost:3001/v1/chat/completions \+ -H "Authorization: Bearer freellmapi-your-unified-key" \+ -H "Content-Type: application/json" \+ -d '{+ "model": "auto:fast",+ "messages": [{"role": "user", "content": "hi"}]+ }'+```++`auto:<profile-name>` routes through a named profile's chain instead of the active one, so different tools can use different chains through the same key:++```bash+curl http://localhost:3001/v1/chat/completions \+ -H "Authorization: Bearer freellmapi-your-unified-key" \+ -H "Content-Type: application/json" \+ -d '{+ "model": "auto:coding",+ "messages": [{"role": "user", "content": "Write a binary search in Rust."}]+ }'+```++An unknown profile name returns a clear `400` rather than silently falling back. Profiles are named fallback chains (see [Features](../README.md#features)) — create and switch them from the dashboard; whichever is active is what plain `auto` uses.++## Streaming++```python+stream = client.chat.completions.create(+ model="auto",+ messages=[{"role": "user", "content": "Stream me a haiku about SQLite."}],+ stream=True,+)+for chunk in stream:+ print(chunk.choices[0].delta.content or "", end="", flush=True)+```++## Tool calling++Pass OpenAI-style `tools` and `tool_choice`; the assistant response round-trips back through the proxy exactly like the OpenAI API. Multi-step flows (assistant `tool_calls` → `tool` role follow-up → final answer) work across every provider the router can reach.++```python+tools = [{+ "type": "function",+ "function": {+ "name": "get_weather",+ "description": "Get current weather for a city.",+ "parameters": {+ "type": "object",+ "properties": {"city": {"type": "string"}},+ "required": ["city"],+ },+ },+}]++# 1. Model asks for a tool call+first = client.chat.completions.create(+ model="auto",+ messages=[{"role": "user", "content": "What's the weather in Karachi?"}],+ tools=tools,+ tool_choice="required",+)+call = first.choices[0].message.tool_calls[0]++# 2. You execute the tool, feed the result back+final = client.chat.completions.create(+ model="auto",+ messages=[+ {"role": "user", "content": "What's the weather in Karachi?"},+ first.choices[0].message,+ {"role": "tool", "tool_call_id": call.id, "content": '{"temp_c": 32, "cond": "sunny"}'},+ ],+ tools=tools,+)+print(final.choices[0].message.content)+```++Works with `stream=True` as well — you'll get `delta.tool_calls` chunks followed by a `finish_reason: "tool_calls"` close. Under the hood, OpenAI-compatible providers (Groq, Cerebras, Mistral, OpenRouter, GitHub Models, HuggingFace, Cloudflare, Cohere compat) get the request passed through; Gemini requests get translated into Google's `functionDeclarations` / `functionResponse` shape and the response is translated back.++## Gemini Google Search grounding++Google's models can ground their answers in live Google Search results. Since the OpenAI wire format has no way to express that, request a tool named `google_search` and the Google provider translates it into Gemini's native grounding tool. It can be sent on its own or alongside your normal function tools.++```python+resp = client.chat.completions.create(+ model="gemini-2.5-flash", # pin a Google model so the request routes there+ messages=[{"role": "user", "content": "Who won the F1 race this weekend?"}],+ tools=[{"type": "function", "function": {"name": "google_search", "parameters": {}}}],+)+print(resp.choices[0].message.content)+```++## Vision / image input++Send images with the standard OpenAI `image_url` content blocks (base64 `data:` URLs or `http(s)` URLs). When a request contains an image, the router restricts itself to **vision-capable models** and ignores text-only ones. Vision models are tagged with a **Vision** badge on the Fallback Chain page; the current set includes Gemini (2.5 / 3.x), Llama 4 Scout/Maverick (Groq, NVIDIA), GLM-4.6V Flash (Z.ai), Nemotron Nano 12B VL (OpenRouter), and GitHub's GPT-4o / GPT-4.1.++```python+resp = client.chat.completions.create(+ model="auto", # auto-routes to a vision model+ messages=[{+ "role": "user",+ "content": [+ {"type": "text", "text": "What's in this image?"},+ {"type": "image_url", "image_url": {"url": "data:image/png;base64,<...>"}},+ ],+ }],+)+print(resp.choices[0].message.content)+```++If no vision-capable model is enabled in your Fallback Chain, an image request returns a clear `422` (`code: "no_vision_model"`) rather than silently dropping the image. (Image input on `/v1/responses` isn't supported yet — use `/v1/chat/completions`.)++## Images & text-to-speech++`POST /v1/images/generations` and `POST /v1/audio/speech` route across the providers that serve media models, including custom OpenAI-compatible media endpoints. Browse and toggle them on the dashboard's **Models → Image / Audio** tabs.++## Fusion (multi-model synthesis)++Request the virtual `fusion` model and the router fans your prompt out to a panel of diverse free models in parallel, then a judge model synthesizes one answer from the drafts. Panel, judge, and strategy are configurable on the dashboard's **Fusion** page or per request via the `fusion` field; each sub-call goes through normal routing, quotas, and analytics.++## Response headers++Every response carries an `X-Routed-Via: <platform>/<model>` header so you can see which provider actually served each call. If a request fell over between providers, you'll also see `X-Fallback-Attempts: N`.++The opt-in response cache can be toggled per request with `X-FreeLLM-Cache: on|off` — an exact-match in-memory LRU for identical non-streaming requests (canonical SHA-256 keys over the full request, TTL and temperature gates, saved-token stats on the dashboard). Off by default; cache hits consume zero provider quota.++## Embeddings++`/v1/embeddings` is OpenAI-compatible, with one deliberate difference from chat routing: **failover never crosses models.** Vectors from different models live in incompatible spaces — silently switching models would corrupt any vector store built on top of the proxy. So embeddings route by **family** (one model identity + dimension), and failover only walks the providers serving that same family.++```python+resp = client.embeddings.create(+ model="auto", # default family; or a family name like "bge-m3"+ input=["the quick brown fox", "pack my box with five dozen liquor jugs"],+)+print(len(resp.data), "vectors of", len(resp.data[0].embedding), "dims")+```++```bash+curl http://localhost:3001/v1/embeddings \+ -H "Authorization: Bearer freellmapi-your-unified-key" \+ -H "Content-Type: application/json" \+ -d '{"model": "auto", "input": "hello world"}'+```++`model` accepts `auto` (the configured default family), a family name, or a provider-specific model id (which resolves to its family). Available families:++| Family (`model`) | Dims | Providers (failover order) |+| --- | --- | --- |+| `gemini-embedding-001` *(default)* | 3072 | Google |+| `text-embedding-3-large` | 3072 | GitHub Models |+| `text-embedding-3-small` | 1536 | GitHub Models |+| `embed-v4.0` | 1536 | Cohere |+| `bge-m3` | 1024 | Cloudflare → Hugging Face |+| `qwen3-embedding-0.6b` | 1024 | Cloudflare |+| `nv-embedqa-e5-v5` | 1024 | NVIDIA |+| `llama-nemotron-embed-1b-v2` | 2048 | NVIDIA |+| `llama-nemotron-embed-vl-1b-v2` | 2048 | NVIDIA → OpenRouter |+| `embeddinggemma-300m` | 768 | Cloudflare |++The default family, per-provider toggles, and priorities live on the dashboard's **Models → Embeddings** page. Pick your family once and stick with it for a given vector store — that's the whole point of the family model.++## Anthropic / Claude clients++FreeLLMAPI also speaks Anthropic's Messages API, so anything built for Claude — including **Claude Code** and the official Anthropic SDKs — can run against your free pool. Point the client at your server's **origin** (Anthropic clients append `/v1/messages` themselves) and authenticate with your unified key. Both `x-api-key` and `Authorization: Bearer` are accepted.++```bash+curl http://localhost:3001/v1/messages \+ -H "x-api-key: freellmapi-your-unified-key" \+ -H "anthropic-version: 2023-06-01" \+ -H "Content-Type: application/json" \+ -d '{+ "model": "claude-sonnet-4-5",+ "max_tokens": 256,+ "messages": [{"role": "user", "content": "hi"}]+ }'+```++Claude model names map to your free pool on the **Keys → Anthropic** tab: each family (`default`, `opus`, `sonnet`, `haiku`) routes to `auto` (the router picks a free model) or a model you pin. `POST /v1/messages/count_tokens` and a content-negotiated `GET /v1/models` (Anthropic shape when `anthropic-version` is sent) are implemented too. Streaming, system prompts, tool use, and image input all translate across the same router as the OpenAI endpoints.++**Claude Code** — point it at your server and start it:++*On macOS / Linux (Bash):*+```bash+export ANTHROPIC_BASE_URL=http://localhost:3001+export ANTHROPIC_AUTH_TOKEN=freellmapi-your-unified-key # NOT ANTHROPIC_API_KEY+claude+```++*On Windows (PowerShell):*+```powershell+$env:ANTHROPIC_BASE_URL="http://localhost:3001"+$env:ANTHROPIC_AUTH_TOKEN="freellmapi-your-unified-key"+claude+```++> Use `ANTHROPIC_AUTH_TOKEN` (sent as a Bearer token), **not** `ANTHROPIC_API_KEY` — Claude Code treats a set `ANTHROPIC_API_KEY` as a conflicting first-party credential and refuses to start.diff --git a/docs/architecture.md b/docs/architecture.mdnew file mode 100644index 00000000..b4376310--- /dev/null+++ b/docs/architecture.md@@ -0,0 +1,112 @@+# Architecture & internals++[← Back to README](../README.md) · [Documentation index](README.md)++How the router decides, what to expect from stacked free tiers, and where the boundaries are.++- [How it works](#how-it-works)+- [Routing in detail](#routing-in-detail)+- [Operational details](#operational-details)+- [Not yet supported](#not-yet-supported)+- [Limitations](#limitations)+- [Terms of Service review](#terms-of-service-review)++## How it works++++```+┌──────────────────┐ Bearer freellmapi-… ┌─────────────────────────┐+│ OpenAI SDK / │ ──────────────────────▶ │ Express proxy (:3001) │+│ curl / any │ ◀────────────────────── │ /v1/chat/completions │+│ OpenAI client │ streamed tokens └────────────┬────────────┘+└──────────────────┘ │+ ▼+ ┌────────────────────────────────────────────────┐+ │ Router │+ │ 1. Pick highest-priority model that │+ │ (a) has a healthy key and │+ │ (b) is under all its rate limits. │+ │ 2. Decrypt key, call provider SDK. │+ │ 3. On 429/5xx → cooldown + retry next model. │+ └────────────────────────────────────────────────┘+ │+ ┌──────────────┬────────────┬──────────┴─────────┬─────────────┬──────────┐+ ▼ ▼ ▼ ▼ ▼ ▼+ Google Groq Cerebras OpenRouter HF …22 more+```++- **Router** (`server/src/services/router.ts`) — picks a model per request.+- **Rate-limit ledger** (`server/src/services/ratelimit.ts`) — in-memory RPM/RPD/TPM/TPD counters backed by SQLite, with cooldowns on 429s.+- **Provider adapters** (`server/src/providers/*.ts`) — one file per provider, implementing the `Provider` base class: `chatCompletion()` and `streamChatCompletion()`.+- **Health service** (`server/src/services/health.ts`) — periodic probe keeps key status fresh.+- **Dashboard** (`client/`) — React + Vite + shadcn/ui admin surface.+- **Storage** — SQLite (`better-sqlite3`) with AES-256-GCM envelope encryption for keys.++## Routing in detail++- **Automatic fallover** — If the chosen provider returns a 429, 5xx, or times out, the router skips it, puts the key on a short cooldown, and retries on the next model in your fallback chain (up to 20 attempts, bounded by a wall-clock retry budget). A dead key rotates to its siblings instead of failing the request, and exhaustion errors carry the full attempt trail so you can see exactly what was tried.+- **Smart routing, six strategies** — the chain is ranked by a selectable strategy: `priority` (your manual order), `balanced`, `smartest`, `fastest`, `reliable`, or `custom` with your own weight mix. Scores come from live per-model measurements (speed, capability, rate-limit headroom, recent errors) with a Thompson-sampling bandit under the hood; one-click sort presets reorder the chain from the dashboard.+- **Unified models** — the same logical model served by several providers (say, GLM-4.7 on Cloudflare and Z.ai) collapses into one entry: one name in `/v1/models`, strict in-group failover between its providers, and merge/split overrides when the grouping guesses wrong.+- **Model profiles** — save named fallback-chain configurations (a coding chain, a long-context chain, a vision chain) and switch the active one from the dashboard.+- **Per-key rate tracking** — RPM, RPD, TPM, and TPD counters per `(platform, model, key)` so the router always picks a key that's under its caps. The ledger also learns: ceilings a provider reports in error bodies or quota headers (a Groq 413 naming its TPM limit) tighten the router's own limits automatically.+- **Sticky sessions** — Multi-turn conversations keep talking to the same model for 30 minutes to avoid the hallucination spike that comes from mid-conversation model switches. See also [Context Handoff](clients.md#context-handoff) for what happens when a switch is unavoidable.+- **Structured outputs & full sampling passthrough** — `response_format` (`json_object` / `json_schema`, translated to Gemini's native `responseSchema`), plus `seed`, `top_k`, `min_p`, presence/frequency/repetition penalties, `logit_bias`, `logprobs`, and the `max_completion_tokens` alias. Params a provider is known to reject are dropped per platform (Mistral's strict API, Groq's logprobs family…), and every model advertises its honest list in `/v1/models` `supported_parameters`.+- **Tool-call rescue** — models that emit tool calls as plain text instead of structured JSON are rescued into real `tool_calls` automatically, and tool requests only route to models that actually support them.++## Operational details++- **Encrypted key storage** — API keys are encrypted with AES-256-GCM before hitting SQLite; decryption happens in-memory just before a request.+- **Unified API key** — Clients authenticate to your proxy with a single `freellmapi-…` bearer token. You never expose upstream provider keys to your apps.+- **Dashboard login** — The admin UI and all `/api/*` routes are gated behind an email + password account (scrypt-hashed, session-token auth), set on first run. The `/v1` proxy keeps its own unified-key auth for apps.+- **Health checks** — Periodic probes mark keys as `healthy`, `rate_limited`, `invalid`, or `error` so the router skips dead ones automatically.+- **Response cache (opt-in)** — an exact-match in-memory LRU for identical non-streaming requests: canonical SHA-256 keys over the full request, TTL and temperature gates, per-request `X-FreeLLM-Cache: on|off` override, and saved-token stats on the dashboard. Off by default; cache hits consume zero provider quota.+- **Key import & export** — bulk-import keys by pasting a `.env` file (with preview and per-key selection), export back out as JSON, `.env`, or CSV.+- **Analytics** — Per-request logging with latency (p50 / p95 and time-to-first-token for streams), token counts, success rate, estimated cost savings, and per-provider / per-model / per-key breakdowns over 24h to 90d windows.+- **Encrypted DB backups** — optional periodic encrypted snapshots of the SQLite database to a local path or HTTP target, restored automatically on a fresh boot (see [Install & deploy](install.md#docker-image--operations)).++## Not yet supported++The scope is deliberately narrow. If a feature isn't in the README's feature list and isn't below, assume it isn't there yet.++- **Moderation** (`/v1/moderations`)+- **`n > 1`** (multiple completions per request)+- **Per-user billing / multi-tenant auth** — single-user by design++PRs that add any of these are very welcome. See [Contributing](../README.md#contributing).++## Limitations++Stacking free tiers has real trade-offs. Be honest with yourself about them:++- **No frontier models.** The free-tier catalog tops out around Llama 3.3 70B, GLM-4.5, Qwen 3 Coder, and Gemini 2.5 Pro. You will not get GPT-5 or Claude Opus class reasoning through this. For hard problems, pay for a real API.+- **Intelligence degrades as the day progresses.** Your top-ranked models (usually Gemini 2.5 Pro, GPT-4o via GitHub Models) have the lowest daily caps. Once they hit their limits, the router falls down your priority chain to smaller/weaker models. Expect the effective intelligence of the endpoint to drop in the late hours of each day — then reset at UTC midnight.+- **Latency is highly variable.** Cerebras and Groq are extremely fast; others are not. You get whichever one is available.+- **Free tiers can change without notice.** Providers regularly tighten, loosen, or remove free tiers. When that happens you'll see 429s or auth errors until the catalog update reaches you — live-feed installs get those fixes within days, free installs on the 30-day trail. Re-seed scripts live in `server/src/scripts/`.+- **No SLA, by definition.** If you need reliability, use a paid provider with a contract.+- **Local-first.** There's no multi-tenant auth. Run this for yourself; don't expose it to the internet.++## Terms of Service review++A self-hosted, single-user, personal-use setup was re-reviewed against each provider's ToS (May 2026). Summary:++| Provider | Verdict | Notes |+|---|---|---|+| Google Gemini | ⚠️ Caution | March 2026 ToS narrows scope to *"professional or business purposes, not for consumer use"* — a self-hosted developer proxy is still defensible, but the clause is new. |+| Groq | ✅ Likely OK | GroqCloud Services Agreement permits Customer Application integration. |+| Cerebras | ✅ Likely OK | Permitted; explicitly forbids selling/transferring API keys. |+| Mistral | ✅ Likely OK | APIs allowed for personal/internal business use. |+| OpenRouter | ✅ Likely OK | April 2026 ToS sharpens the no-resale / no-competing-service clause; private single-user proxy still fine. |+| Cloudflare Workers AI | ⚠️ Ambiguous | No anti-proxy clause; covered by general Self-Serve Subscription Agreement. |+| NVIDIA NIM | ⚠️ Caution | Trial ToS §1.2 / §1.4: *"evaluation only, not production."* Free access is a recurring 40 RPM rate limit (the 2025 credit system was discontinued), but the evaluation-only scope stands. |+| GitHub Models | ⚠️ Caution | Free tier explicitly scoped to *"experimentation"* and *"prototyping."* |+| Cohere | ❌ Avoid | Terms §14 still forbids *"personal, family or household purposes."* |+| Zhipu (open.bigmodel.cn) | ✅ Likely OK | Personal/non-commercial research carve-out still in the platform docs. |+| Z.ai (api.z.ai) | ⚠️ Caution | New row — Singapore entity (distinct from Zhipu CN). §III.3(l) anti-traffic-redirect clause could plausibly be read against a proxy; no explicit personal-use carve-out. |+| Ollama Cloud | ✅ Likely OK | New row — Free plan permits cloud-model access (1 concurrent, 5-hour session caps). No anti-proxy / anti-resale clauses found. *(Integration tracked in #14.)* |+| OVH AI Endpoints | ✅ Likely OK | New row (June 2026) — anonymous access is officially documented (2 req/min per IP per model). OVH reserves the right to introduce token/consumption caps. |+| AI Horde | ✅ Likely OK | New row (June 2026) — a free, community-powered commons run by the Haidra non-profit; anonymous use is officially supported (key `0000000000`, lowest queue priority). No anti-proxy / anti-resale clause. The OpenAI proxy is a pilot and may be restricted by usage. *(Integration #345.)* |++Rules of thumb that keep most providers happy: **one account per provider**, **no reselling**, **no sharing your endpoint with other humans**, **don't hammer a free tier as a paid production backend**. This is informational, not legal advice — read each provider's ToS and make your own call.++Removed since the April 2026 review: Hugging Face, Moonshot, and MiniMax direct integrations were dropped from the catalog (HF — tool-call format issues; Moonshot — moved to paid only; MiniMax — superseded by the OpenRouter `minimax/minimax-m2.5:free` route).diff --git a/docs/clients.md b/docs/clients.mdnew file mode 100644index 00000000..7df1e8c9--- /dev/null+++ b/docs/clients.md@@ -0,0 +1,100 @@+# Clients & coding agents++[← Back to README](../README.md) · [Documentation index](README.md)++- [OpenAI-compatible clients](#openai-compatible-clients)+- [Coding agents](#coding-agents)+- [MCP server](#mcp-server)+- [VS Code ghost-text autocomplete (Continue)](#vs-code-ghost-text-autocomplete-continue)+- [Context Handoff](#context-handoff)++## OpenAI-compatible clients++Any client that can target an OpenAI-compatible base URL can use FreeLLMAPI:++- **LangChain, LlamaIndex, official OpenAI SDKs**: set `base_url` to+ `http://localhost:3001/v1` and use the unified key from the dashboard.+- **Local GPU boxes**: add custom OpenAI-compatible endpoints for Ollama,+ llama.cpp, LM Studio, vLLM, or an internal gateway.++## Coding agents++Every recipe below is the same three facts in a different config file: base URL+`http://localhost:3001/v1`, the unified key from the dashboard's Keys page, and+a model (`auto` lets the router pick).++| Agent | Setup |+| --- | --- |+| **Claude Code** | `ANTHROPIC_BASE_URL=http://localhost:3001` + `ANTHROPIC_AUTH_TOKEN=<unified key>` — full walkthrough in [Anthropic / Claude clients](api.md#anthropic--claude-clients) |+| **Codex CLI** | add a provider in `~/.codex/config.toml` with `base_url = "http://localhost:3001/v1"` and its `env_key` pointing at the unified key — the `/v1/responses` surface it needs is implemented |+| **Cline / Roo Code** | provider type "OpenAI Compatible", base URL `http://localhost:3001/v1`, unified key, model `auto` (or any id from `/v1/models`) |+| **Continue** | `apiBase: http://localhost:3001/v1` in its config; inline autocomplete works too via the legacy `/v1/completions` surface |+| **Aider** | `OPENAI_API_BASE=http://localhost:3001/v1` + `OPENAI_API_KEY=<unified key>`, then `aider --model openai/auto` |+| **opencode** | OpenAI-compatible provider with the same base URL and key |+| **Cursor** | paste the unified key under a custom OpenAI base URL — but note Cursor verifies and calls the API **from its servers**, so your router must be reachable from the internet (a tunnel or a host with a public address), not just `localhost` |++## MCP server++On top of inference, the router is an **MCP server**: agents can introspect it mid-session+(usable models and the params each one honors, provider health, usage and cache stats,+routing strategy). For Claude Code:++```bash+claude mcp add --transport http freellmapi http://localhost:3001/mcp \+ --header "Authorization: Bearer freellmapi-your-unified-key"+```++Any MCP client that speaks Streamable HTTP works the same way: point it at `/mcp` with the+unified key as a Bearer token.++FreeLLMAPI is local-first and single-user by design. Your provider keys stay in+your SQLite database, encrypted at rest, and requests go from your machine to the+upstream providers you enabled.++## VS Code ghost-text autocomplete (Continue)++FreeLLMAPI exposes `/v1/completions` for editor autocomplete clients that send legacy OpenAI prompt/suffix requests. Example Continue config:++```yaml+models:+ - name: FreeLLMAPI Autocomplete+ provider: openai+ model: auto+ apiBase: http://localhost:3001/v1+ apiKey: freellmapi-your-unified-key+ useLegacyCompletionsEndpoint: true+ roles:+ - autocomplete+```++## Context Handoff++When FreeLLMAPI falls over to a different model mid-conversation (quota, rate limit, cooldown), the new model has no idea it is picking up someone else's task. **Context handoff** adds a single compact `system` message to the outbound request that tells the new model exactly that:++```+FreeLLMAPI context handoff:+You are taking over an ongoing conversation from another model (groq:llama-3 → google:gemini-flash).+Continue the user's task using the conversation context already provided in this request.+Do not restart the task, re-ask already answered setup questions, or discard prior tool results.+Respect the user's latest message as the highest-priority instruction.++Recent session summary:+User: …+Assistant: …+```++**Enable it in `.env`:**++```env+FREELLMAPI_CONTEXT_HANDOFF=on_model_switch+```++**How it works:**++- Messages per session are stored in memory (TTL: 3 hours).+- Only injected when the selected model changes for a given session key.+- Not injected on the first request, on same-model continuations, or if a handoff message is already present.+- Session key: `X-Session-Id` header if present, otherwise SHA-1 of the first user message (same as sticky sessions).+- Storage is in-memory only. Nothing is written to disk or logged.++> **Important:** Context Handoff improves continuity for conversations routed through FreeLLMAPI. It cannot recover provider-internal hidden state or messages that were never sent to the proxy.diff --git a/docs/install.md b/docs/install.mdnew file mode 100644index 00000000..a20c553b--- /dev/null+++ b/docs/install.md@@ -0,0 +1,245 @@+# Install & deploy++[← Back to README](../README.md) · [Documentation index](README.md)++Everything about getting FreeLLMAPI running: the one-liner, Docker Compose, local development, declarative config, production builds, the desktop app, and where your data lives.++- [Quick start (one-liner)](#quick-start-one-liner)+- [Docker Compose](#docker-compose)+- [Local development](#local-development)+- [Declarative startup config](#declarative-startup-config)+- [Docker image & operations](#docker-image--operations)+- [Desktop app](#desktop-app)+- [Credentials and where your data lives](#credentials-and-where-your-data-lives)++## Quick start (one-liner)++Docker required — sets up `~/freellmapi`, generates an encryption key, pulls the image, and starts the container:++```bash+curl -fsSL https://freellmapi.co/install.sh | bash+```++Prefer to read before you pipe to bash? [The script is here](https://freellmapi.co/install.sh). Re-running it is safe: your `.env` (and encryption key) is preserved and the container updates to `:latest`. Override the defaults with `FREELLMAPI_DIR`, `PORT`, or `HOST_BIND` env vars.++On Windows, the easiest path is the desktop **[`.exe` installer from Releases](https://github.com/tashfeenahmed/freellmapi/releases/latest)** ([below](#desktop-app)); the Docker steps work in WSL or any bash shell.++On Android, see the experimental [Termux installation guide](install/android-termux.md). It uses Node's built-in SQLite driver and does not require the Android NDK.++Open http://localhost:3001, add your provider keys on the **Keys** page, reorder the **Fallback Chain** to taste, and grab your unified API key from the **Keys** page header. That unified key is what you point your OpenAI SDK at.++Your install keeps itself updated from the signed catalog feed. The current full catalog is listed at [freellmapi.co/models](https://freellmapi.co/models.html).++## Docker Compose++Runs the API and dashboard together on port 3001 and persists SQLite in a named volume.++**Prerequisites:** Docker, Docker Compose, OpenSSL.++*On macOS / Linux (Bash):*+```bash+git clone https://github.com/tashfeenahmed/freellmapi.git+cd freellmapi++# Generate an encryption key for at-rest key storage+ENCRYPTION_KEY="$(openssl rand -hex 32)"+printf "ENCRYPTION_KEY=%s\nPORT=3001\n" "$ENCRYPTION_KEY" > .env++docker compose up -d+```++*On Windows (PowerShell):*+```powershell+git clone https://github.com/tashfeenahmed/freellmapi.git+cd freellmapi++$Bytes = New-Object Byte[] 32+[Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($Bytes)+$ENCRYPTION_KEY = -join ($Bytes | ForEach-Object { "{0:x2}" -f $_ })+"ENCRYPTION_KEY=$ENCRYPTION_KEY`nPORT=3001" | Out-File -Encoding utf8 .env+docker compose up -d+```++> **Reaching it from another machine?** By default the container is published only on `127.0.0.1`, so `http://<server-ip>:3001` won't load from another device (the page just hangs). To expose it on your LAN — e.g. a Raspberry Pi at `http://192.168.1.x:3001` — start it with `HOST_BIND=0.0.0.0`:+>+> ```bash+> HOST_BIND=0.0.0.0 docker compose up -d+> ```+>+> Only do this on a trusted network: the proxy is single-user and guarded only by the unified API key.++## Local development++**Prerequisites:** Node.js 20+, npm.++*On macOS / Linux (Bash):*+```bash+git clone https://github.com/tashfeenahmed/freellmapi.git+cd freellmapi+npm install+ENCRYPTION_KEY="$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))')"+printf "ENCRYPTION_KEY=%s\nPORT=3001\n" "$ENCRYPTION_KEY" > .env+npm run dev+```++*On Windows (PowerShell):*+```powershell+git clone https://github.com/tashfeenahmed/freellmapi.git+cd freellmapi+npm install+$ENCRYPTION_KEY = node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"+"ENCRYPTION_KEY=$ENCRYPTION_KEY`nPORT=3001" | Out-File -Encoding utf8 .env+npm run dev+```++`ENCRYPTION_KEY` is required for startup. When `NODE_ENV` is not `production`+and it is unset, the server auto-generates a development key and saves it to a+`.encryption-key` file (0600) next to the SQLite database, not inside it. Older+installs that kept the key in the database are migrated to this file on first+boot. Do not rely on that fallback with real provider keys; set `ENCRYPTION_KEY`.++Request analytics are retained for 90 days or 100000 request rows by default,+whichever limit prunes first. Set `REQUEST_ANALYTICS_RETENTION_DAYS=0` or+`REQUEST_ANALYTICS_MAX_ROWS=0` in `.env` to disable either retention limit.++Open http://localhost:5173 (the Vite dev UI), add your provider keys on the **Keys** page, reorder the **Fallback Chain** to taste, and grab your unified API key from the **Keys** page header. That unified key is what you point your OpenAI SDK at.++> **Reaching the dev UI from another device on your LAN?** Use `npm run dev:lan` — it passes `--host` through to Vite, which then prints a `Network: http://<your-ip>:5173` URL you can open from a phone or another machine. (Plain `npm run dev -- --host` does *not* work here: the root `dev` script is a `concurrently` wrapper, so the flag never reaches Vite.) API calls go through Vite's dev proxy, so no extra server config is needed.++For a production build without Docker:++```bash+npm run build+node server/dist/index.js # server + dashboard both served on :3001+```++## Declarative startup config++For repeatable Docker/server installs, FreeLLMAPI can apply a JSON config on+every boot. Set `FREEAPI_CONFIG_PATH=/path/to/freellmapi.config.json` or put the+same JSON in `FREEAPI_CONFIG_JSON`. The config is idempotent: existing keys,+custom providers, model edits, fallback rows, and routing settings are updated+instead of duplicated.++```json+{+ "keys": [+ { "platform": "groq", "key": "gsk_...", "label": "main" },+ { "platform": "google", "key": "AIza...", "enabled": true }+ ],+ "customProviders": [+ {+ "baseUrl": "http://host.docker.internal:11434/v1",+ "label": "Ollama",+ "models": [+ { "model": "llama3.1:8b", "displayName": "Local Llama", "supportsTools": true }+ ]+ }+ ],+ "models": [+ {+ "platform": "groq",+ "modelId": "llama-3.3-70b-versatile",+ "displayName": "Llama 3.3 70B",+ "supportsTools": true,+ "fallbackEnabled": true+ }+ ],+ "routing": { "strategy": "balanced" }+}+```++## Docker image & operations++FreeLLMAPI publishes a single production image that contains the Express server and the built React dashboard:++```bash+docker pull ghcr.io/tashfeenahmed/freellmapi:latest # or pin a release, e.g. :v1.2.3+```++The image is multi-arch (`linux/amd64` + `linux/arm64`, so it runs on a Raspberry Pi). Published tags: `latest` (default branch), `v*.*.*` (git release tags), and `sha-<commit>`.++The included `docker-compose.yml` is the recommended install path:++```bash+docker compose up -d+docker compose logs -f freellmapi+```++By default the container's port is bound to `127.0.0.1` (localhost only). To reach the dashboard/API from another machine on your network, publish it on all interfaces with `HOST_BIND=0.0.0.0 docker compose up -d` — only on a trusted LAN, since the proxy is single-user.++SQLite data is stored in the `freellmapi-data` volume at `/app/server/data`.+Keep the same `.env` `ENCRYPTION_KEY` and volume when upgrading, because+provider keys are encrypted at rest. If your host only persists a specific+directory, set `FREEAPI_DB_PATH=/that/path/freellmapi.db`.++On hosts with ephemeral disks, configure an encrypted backup target:++```env+FREEAPI_DB_BACKUP_PATH=/app/server/data/freellmapi.db.backup+# or:+FREEAPI_DB_BACKUP_URL=https://example.com/freellmapi.db.backup+FREEAPI_DB_BACKUP_TOKEN=optional-bearer-token+FREEAPI_DB_BACKUP_KEY=64-char-hex-backup-key+FREEAPI_DB_BACKUP_INTERVAL_MS=300000+```++When the database file is missing at startup, FreeLLMAPI restores the backup+before migrations run. While the server is running it uploads a fresh encrypted+backup periodically. If `FREEAPI_DB_BACKUP_KEY` is omitted, the app uses+`ENCRYPTION_KEY` for the backup envelope too.++More Docker operations and examples live in [docker/README.md](../docker/README.md).++## Desktop app++A native menu-bar app lives in [`desktop/`](../desktop): the entire router ++dashboard running locally from your tray, with a glass popover showing live+request stats.++++**[Download from Releases](https://github.com/tashfeenahmed/freellmapi/releases/latest)** — the macOS `.dmg` and the Windows `.exe` installer are built and attached to every release by the [`desktop-release`](../.github/workflows/desktop-release.yml) workflow. Or build it from this repo in a few minutes:++> **Note for Windows users building from source:** Building the desktop app requires compiling native SQLite modules for Electron. You must have [Visual Studio Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) installed (specifically the "Desktop development with C++" workload) and Python installed before running `npm install`.++```bash+npm install+npm install --prefix desktop # install desktop dependencies+npm run desktop:dist # macOS → desktop/dist-electron/FreeLLMAPI-…-arm64.dmg+npm run desktop:dist:win # Windows → "desktop/dist-electron/FreeLLMAPI Setup ….exe"+```++> Locally built apps are unsigned, so Windows SmartScreen may warn on first run+> ("More info" → "Run anyway"); the macOS build launches without Gatekeeper prompts.+> Full instructions in [desktop/README.md](../desktop/README.md).++## Credentials and where your data lives++The desktop app has **no username or password to set up**. Unlike the server+(which gates its dashboard behind an email + password account), the desktop+build signs the dashboard in automatically with a hidden local account, so+you're never prompted for credentials and never need one.++The only credential you need is your **unified API key** — the+`freellmapi-…` token your OpenAI/Anthropic client points at. Get it from:++- the tray popover — click the tray icon, then **Copy Key**, or+- the dashboard **Keys** page header (tray → **Open Dashboard**).++You do not need to open or edit `freeapi.db` by hand.++Your settings and data live in one folder per OS (copy it to migrate to+another machine or into a container):++| OS | Location |+|----|----------|+| Windows | `%APPDATA%\FreeLLMAPI\` (e.g. `C:\Users\<you>\AppData\Roaming\FreeLLMAPI\`) |+| macOS | `~/Library/Application Support/FreeLLMAPI/` |+| Linux | `~/.config/FreeLLMAPI/` |++That folder holds `freeapi.db` (all keys, models, settings, encrypted at rest)+and `config.json` (window/theme/port/LAN preferences). Copy both to move an+install. For the server (non-desktop) deployment, the equivalent state is the+`.env` file and the SQLite DB at `server/data/freeapi.db` (or wherever+`FREEAPI_DB_PATH` points).