7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.
TypeScript
docs: embeddings API section — it shipped, but the README said 'not yet supported'
Tashfeen243ddb24d5f1
diff --git a/README.md b/README.mdindex 83b717ed..7dc18254 100644--- a/README.md+++ b/README.md@@ -77,6 +77,7 @@ Plus a **custom** provider — point at any OpenAI-compatible endpoint (llama.cp- **Responses API** — `POST /v1/responses` (the wire format current Codex CLI versions require) is implemented as a translating shim over the same router, with full streaming events and tool calls.- **Streaming and non-streaming** — Server-Sent Events for `stream: true`, JSON response otherwise. Every provider adapter implements both.- **Tool calling** — OpenAI-style `tools` / `tool_choice` requests are passed through, and assistant `tool_calls` + `tool` role follow-up messages round-trip across providers.+- **Embeddings** — `/v1/embeddings` with family-based routing: failover only ever happens between providers serving the *same* model (vectors from different models are incompatible), never across models. See [Embeddings](#embeddings).- **Automatic fallover** — If the chosen provider returns a 429, 5xx, or times out, the router skips it, puts the key on a short cooldown, and retries on the next model in your fallback chain (up to 20 attempts).- **Per-key rate tracking** — RPM, RPD, TPM, and TPD counters per `(platform, model, key)` so the router always picks a key that's under its caps.- **Sticky sessions** — Multi-turn conversations keep talking to the same model for 30 minutes to avoid the hallucination spike that comes from mid-conversation model switches.@@ -92,7 +93,6 @@ Plus a **custom** provider — point at any OpenAI-compatible endpoint (llama.cpThe scope is deliberately narrow. If a feature isn't on this list and isn't below, assume it isn't there yet.-- **Embeddings** (`/v1/embeddings`)- **Image generation** (`/v1/images/*`)- **Audio / speech** (`/v1/audio/*`)- **Legacy completions** (`/v1/completions`) — only the chat endpoint is implemented@@ -315,6 +315,42 @@ Works with `stream=True` as well — you'll get `delta.tool_calls` chunks followEvery response carries an `X-Routed-Via: <platform>/<model>` header so you can see which provider actually served each call. If a request fell over between providers, you'll also see `X-Fallback-Attempts: N`.+### Embeddings++`/v1/embeddings` is OpenAI-compatible, with one deliberate difference from chat routing: **failover never crosses models.** Vectors from different models live in incompatible spaces — silently switching models would corrupt any vector store built on top of the proxy. So embeddings route by **family** (one model identity + dimension), and failover only walks the providers serving that same family.++```python+resp = client.embeddings.create(+ model="auto", # default family; or a family name like "bge-m3"+ input=["the quick brown fox", "pack my box with five dozen liquor jugs"],+)+print(len(resp.data), "vectors of", len(resp.data[0].embedding), "dims")+```++```bash+curl http://localhost:3001/v1/embeddings \+ -H "Authorization: Bearer freellmapi-your-unified-key" \+ -H "Content-Type: application/json" \+ -d '{"model": "auto", "input": "hello world"}'+```++`model` accepts `auto` (the configured default family), a family name, or a provider-specific model id (which resolves to its family). Available families:++| Family (`model`) | Dims | Providers (failover order) |+| --- | --- | --- |+| `gemini-embedding-001` *(default)* | 3072 | Google |+| `text-embedding-3-large` | 3072 | GitHub Models |+| `text-embedding-3-small` | 1536 | GitHub Models |+| `embed-v4.0` | 1536 | Cohere |+| `bge-m3` | 1024 | Cloudflare → Hugging Face |+| `qwen3-embedding-0.6b` | 1024 | Cloudflare |+| `nv-embedqa-e5-v5` | 1024 | NVIDIA |+| `llama-nemotron-embed-1b-v2` | 2048 | NVIDIA |+| `llama-nemotron-embed-vl-1b-v2` | 2048 | NVIDIA → OpenRouter |+| `embeddinggemma-300m` | 768 | Cloudflare |++The default family, per-provider toggles, and priorities live on the dashboard's **Models → Embeddings** page. Pick your family once and stick with it for a given vector store — that's the whole point of the family model.+## Screenshots### Keys@@ -381,7 +417,7 @@ Stacking free tiers has real trade-offs. Be honest with yourself about them:Contributors very welcome! Good first PRs:- **Add a provider** — copy `server/src/providers/openai-compat.ts` as a template, wire it into `server/src/providers/index.ts`, seed its models in `server/src/db/index.ts`, add a test in `server/src/__tests__/providers/`.-- **Add an endpoint** — embeddings, images, moderations. The provider base class can grow new methods; adapters declare which they support.+- **Add an endpoint** — images, moderations, audio. The provider base class can grow new methods; adapters declare which they support.- **Improve the router** — cost-aware routing (cheapest-healthy-fastest tradeoffs), better latency-weighted priority, regional pinning.- **Dashboard polish** — charts on the Analytics page, key rotation UX, batch import of keys from `.env`.- **Docs** — more examples, client library snippets for Go/Rust/etc., a deployment recipe for Docker or Fly.