Skip to content

7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.

TypeScript

docs: embeddings API section — it shipped, but the README said 'not yet supported'

Tashfeen243ddb24d5f1
diff --git a/README.md b/README.md
index 83b717ed..7dc18254 100644
--- a/README.md
+++ b/README.md
@@ -77,6 +77,7 @@ Plus a **custom** provider — point at any OpenAI-compatible endpoint (llama.cp
- **Responses API** — `POST /v1/responses` (the wire format current Codex CLI versions require) is implemented as a translating shim over the same router, with full streaming events and tool calls.
- **Streaming and non-streaming** — Server-Sent Events for `stream: true`, JSON response otherwise. Every provider adapter implements both.
- **Tool calling** — OpenAI-style `tools` / `tool_choice` requests are passed through, and assistant `tool_calls` + `tool` role follow-up messages round-trip across providers.
+- **Embeddings** — `/v1/embeddings` with family-based routing: failover only ever happens between providers serving the *same* model (vectors from different models are incompatible), never across models. See [Embeddings](#embeddings).
- **Automatic fallover** — If the chosen provider returns a 429, 5xx, or times out, the router skips it, puts the key on a short cooldown, and retries on the next model in your fallback chain (up to 20 attempts).
- **Per-key rate tracking** — RPM, RPD, TPM, and TPD counters per `(platform, model, key)` so the router always picks a key that's under its caps.
- **Sticky sessions** — Multi-turn conversations keep talking to the same model for 30 minutes to avoid the hallucination spike that comes from mid-conversation model switches.
@@ -92,7 +93,6 @@ Plus a **custom** provider — point at any OpenAI-compatible endpoint (llama.cp
The scope is deliberately narrow. If a feature isn't on this list and isn't below, assume it isn't there yet.
-- **Embeddings** (`/v1/embeddings`)
- **Image generation** (`/v1/images/*`)
- **Audio / speech** (`/v1/audio/*`)
- **Legacy completions** (`/v1/completions`) — only the chat endpoint is implemented
@@ -315,6 +315,42 @@ Works with `stream=True` as well — you'll get `delta.tool_calls` chunks follow
Every response carries an `X-Routed-Via: <platform>/<model>` header so you can see which provider actually served each call. If a request fell over between providers, you'll also see `X-Fallback-Attempts: N`.
+### Embeddings
+
+`/v1/embeddings` is OpenAI-compatible, with one deliberate difference from chat routing: **failover never crosses models.** Vectors from different models live in incompatible spaces — silently switching models would corrupt any vector store built on top of the proxy. So embeddings route by **family** (one model identity + dimension), and failover only walks the providers serving that same family.
+
+```python
+resp = client.embeddings.create(
+ model="auto", # default family; or a family name like "bge-m3"
+ input=["the quick brown fox", "pack my box with five dozen liquor jugs"],
+)
+print(len(resp.data), "vectors of", len(resp.data[0].embedding), "dims")
+```
+
+```bash
+curl http://localhost:3001/v1/embeddings \
+ -H "Authorization: Bearer freellmapi-your-unified-key" \
+ -H "Content-Type: application/json" \
+ -d '{"model": "auto", "input": "hello world"}'
+```
+
+`model` accepts `auto` (the configured default family), a family name, or a provider-specific model id (which resolves to its family). Available families:
+
+| Family (`model`) | Dims | Providers (failover order) |
+| --- | --- | --- |
+| `gemini-embedding-001` *(default)* | 3072 | Google |
+| `text-embedding-3-large` | 3072 | GitHub Models |
+| `text-embedding-3-small` | 1536 | GitHub Models |
+| `embed-v4.0` | 1536 | Cohere |
+| `bge-m3` | 1024 | Cloudflare → Hugging Face |
+| `qwen3-embedding-0.6b` | 1024 | Cloudflare |
+| `nv-embedqa-e5-v5` | 1024 | NVIDIA |
+| `llama-nemotron-embed-1b-v2` | 2048 | NVIDIA |
+| `llama-nemotron-embed-vl-1b-v2` | 2048 | NVIDIA → OpenRouter |
+| `embeddinggemma-300m` | 768 | Cloudflare |
+
+The default family, per-provider toggles, and priorities live on the dashboard's **Models → Embeddings** page. Pick your family once and stick with it for a given vector store — that's the whole point of the family model.
+
## Screenshots
### Keys
@@ -381,7 +417,7 @@ Stacking free tiers has real trade-offs. Be honest with yourself about them:
Contributors very welcome! Good first PRs:
- **Add a provider** — copy `server/src/providers/openai-compat.ts` as a template, wire it into `server/src/providers/index.ts`, seed its models in `server/src/db/index.ts`, add a test in `server/src/__tests__/providers/`.
-- **Add an endpoint** — embeddings, images, moderations. The provider base class can grow new methods; adapters declare which they support.
+- **Add an endpoint** — images, moderations, audio. The provider base class can grow new methods; adapters declare which they support.
- **Improve the router** — cost-aware routing (cheapest-healthy-fastest tradeoffs), better latency-weighted priority, regional pinning.
- **Dashboard polish** — charts on the Analytics page, key rotation UX, batch import of keys from `.env`.
- **Docs** — more examples, client library snippets for Go/Rust/etc., a deployment recipe for Docker or Fly.