Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

routing: Free first spends the models that cost nothing before the ones that spend (#3054)

keeuu75e769a9f9b4
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index 6f0e54bf1..6ef978de9 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -32,7 +32,7 @@ backfilled; see [Gateway Session Aggregation](gateway-sessions.md).
| Model resolution | `serve` resolves the model the agent asked for: Claude tier stand-ins, a routing group (`provider.GroupFor`: its id, the model's id however a vendor spells it, else the one group whose name it is, unless a provider serves a model by that id; the user's own groups only while the `groups` plugin is on, [the plugin runtime](plugin-runtime.md#gateway-extension-points-p3-router-strategy)), auto stand-ins, the agent's own stand-in (`StandIn`: Codex's config.toml `model` when magpie is its provider, for a name it sends that magpie doesn't serve, such as its auto-review's `codex-auto-review`), itself taken through `GroupFor` and auto stand-ins so a config naming a group bare reaches it, then `provider.Resolve`. A Codex memory call or auto-review (`x-openai-subagent: guardian`, or `codex-auto-review`) that still names no served model goes to the model Codex's last turn was answered on (`codexMemoryStandIn`, per agent and gateway key), so a Codex on magpie through another provider table, a profile or another computer isn't answered 404 (Adam on Discord, `TestCodexAutoReviewReachesItsTurnsModel`). A bare model id several providers serve (MOMO on Discord: `glm-5.3` under opencode-go and a6api) is never refused: with found groups on (the default) `GroupFor` takes it to the group magpie found for it (`auto-<slug>`), its members in the Providers order, routed and failed over as any group, a conversation staying with the member that answered it while its cache is worth it (affinity); with them off, `Resolve` takes the first provider in the Providers order that exposes it, with no fallback. magpie writes only `provider/model` ids into agents' configs (`magpieModels`), so a bare id is one the user wrote. `/v1/models` lists bare ids only when the user asks for them ([Model ids without the provider](#model-ids-without-the-provider)). Every reply says who answered in `X-Magpie-Provider` and `X-Magpie-Model` (`noteMember`), and the usage log and the Routing page keep it. A model id with `~` and no slash that resolves as nothing else is the one served id (a provider's model or a group) whose slashes it spells as `~` (`unflat`, `provider.Unflat`; MiniMax Code's spelling, #1387), taken back in `validateModel` before key limits, middleware and usage see it; two ids spelled alike are neither. A Claude model id whose version is dashed (`relay/claude-opus-4-6-us`, with or without `:level` and `[1m]`) that is not served as spelled is the one served id it dashes (`provider.ClaudeUndashed`; the spelling magpie gives Claude Code, which reads a dotted version as Claude Opus 4), taken back in `validateModel` and `unprefixed` the same way. An unknown model gets a 404 naming why: a switched-off provider, a disabled group or an empty group. Otherwise it names up to three ids nearest to what was asked (`provider.Closest`: words of a model's display name count before words of its id, and a provider's id or name only alongside them, so "Zen-Space-Bunny" is told `opencode-zen-free/space-bunny-free`, MOMO on Discord), then the ways a model can be named, then every id (`unknownModel`; chat and embeddings). | [`gateway.go`](../../internal/gateway/gateway.go), [`internal/provider/group.go`](../../internal/provider/group.go), [`internal/provider/flat.go`](../../internal/provider/flat.go), [`internal/provider/claudedash.go`](../../internal/provider/claudedash.go) |
| A found group's two ids | A group magpie found is listed (`/v1/models`, the catalog, the lists written into agents, the pickers, the CLI and TUI) as `auto/<model>`: the model's id as a vendor spells it, dots kept and with no vendor prefix (`auto/claude-opus-5.5`; `provider.Group.Ref`, `spelledIn`, preferring a spelling models.dev lists), so a client that looks the part after the slash up on models.dev finds it (ZeroClover on X). Its old id `group/auto-<slug>` is served forever. Inside magpie the group keeps that one id: `validateModel` takes `auto/<model>` (with `:level` or `[1m]`) back to it with `provider.CanonicalRef` before key limits, routing, affinity and the usage log see the request, and the request body is rewritten to it, so both ids count as one group. A gateway key naming either id, `group/*` or `auto/*` passes for both (`allows` in `key_models.go`). `SaveGroup` stores a member, rule or classifier picked as `auto/<model>` by the old id. An agent field magpie set to `group/auto-<slug>` moves to `auto/<model>` on the next `SyncCatalog` (`agent.renameFoundGroups`), but only while its value is still what applied.json says magpie wrote; a value the user set or changed stays. A pick of the group by its old id written now (`Agent.Apply`) is written as `auto/<model>`. An agent's hidden, picked and ordered models saved by the old id hold for the new one (`PublicRef` in `visible.go`), and a bare model id still reaches its found group (`bareTarget`). Claude Code is given a dotted Claude version dashed, so it may send `auto/claude-sonnet-4-5`: that is the same group, and `Reseat` still moves an agent off it once the group is gone (`samePick`). `auto` is no provider id: `Save` and `Rename` refuse it for a new provider, and a plugin's `auto` is `auto-plugin` (`PluginID`). A provider of the user's saved as `auto` before keeps its models: while it exists, `auto/<x>` is its model and found groups list as `group/auto-<slug>` (`autoRefsOn`). Hand-made groups stay `group/<id>`. | [`internal/provider/autoref.go`](../../internal/provider/autoref.go), [`key_models.go`](../../internal/gateway/key_models.go), [`internal/agent/autoref.go`](../../internal/agent/autoref.go) |
| Candidates | `plan` lists one provider's candidates: one per key or account on, then each `Fallback` model, with resting ones last (`restLast`). `planGroup` and `planLevel` weigh a group's members together. A nested group plans by its own routing (#576). So does a member whose provider routes other than Smart (`ownRouting`): weighed as one by its first not resting (`firstAwake`), tried whole in its own order; `leastHeld` leaves such a group's order alone too. A provider on Smart, or routed as the group is, is pooled with the group's other keys and accounts. `group_own_routing_test.go`. A member's own fallbacks are not the group's. | [`fallback.go`](../../internal/gateway/fallback.go) |
-| Order of keys and accounts | `Routing`: smart (""), `order`, `rotate`, `usage` (least used, with an `usageHalfLife` of 1h), `pace` or `weight`. `weighRouted` reads each candidate into a `routes.Seat` and asks the strategy serving its routing (`routes.Find`, the router strategy point of [the plugin runtime](plugin-runtime.md#gateway-extension-points-p3-router-strategy)) for the order: Smart is `routes.Smart`, the core's; the others are `routes.Builtin`, registered by the built-in plugin `rotation`. With it off a routing no strategy serves is tried in its listed order, and `routingOf` treats it as `order` wherever the core asks about a routing (a turn's mark, `goesRound`, `sinks`, a rotate group's sessions, a group's heads). Smart, `usage` and `pace` weigh a subscription by its allowance (`allowances`), and a key by its own usage windows when magpie reads them (`keyAllowance`: a sub2api key given a 5-hour, day or 7-day limit, or a key bought as a plan, such as Kimi Code's or a GLM Coding Plan's); any other key has none. A group routed `weight` (#1910) takes its members, not their keys, by smooth weighted round robin: `planLevel` makes each model member (its keys ordered by its own provider's routing) and each nested group one unit, and `weightedUnits` puts the round's pick (`routes.Round` over `group/<id>`, keyed by the member's spelling, `Group.Weights`, 1 if unset) first and the rest in member order for failover; a unit whose head rests leaves the round, and a kept conversation moves the round's charge to the member that kept it (`wrrSeat`, `routes.MoveRound`). A lone key or account is weighed against nobody, but what it has left is still read into the trace (`weighRouted`), so the Routing page shows it as it does for several (#1016); all but used up, it keeps its conversation rather than being said to move (`affine`). A failure rests a candidate (`failure`, `failureOf`). | [`routing.go`](../../internal/gateway/routing.go), [`weighted.go`](../../internal/gateway/weighted.go), [`internal/routes/`](../../internal/routes), [`internal/provider/routing.go`](../../internal/provider/routing.go) |
+| Order of keys and accounts | `Routing`: smart (""), `order`, `rotate`, `usage` (least used, with an `usageHalfLife` of 1h), `pace`, `weight` or `free` (free first). `weighRouted` reads each candidate into a `routes.Seat` and asks the strategy serving its routing (`routes.Find`, the router strategy point of [the plugin runtime](plugin-runtime.md#gateway-extension-points-p3-router-strategy)) for the order: Smart is `routes.Smart`, the core's; the others are `routes.Builtin`, registered by the built-in plugin `rotation`. With it off a routing no strategy serves is tried in its listed order, and `routingOf` treats it as `order` wherever the core asks about a routing (a turn's mark, `goesRound`, `sinks`, a rotate group's sessions, a group's heads). Smart, `usage` and `pace` weigh a subscription by its allowance (`allowances`), and a key by its own usage windows when magpie reads them (`keyAllowance`: a sub2api key given a 5-hour, day or 7-day limit, or a key bought as a plan, such as Kimi Code's or a GLM Coding Plan's); any other key has none. `free` weighs by what a model costs there and nothing else: only a strategy with `routes.Strategy.ByFree` set (`routes.FreeFirst` is the one) has `weighRouted` fill `routes.Seat.Free`, from `provider.FreeModel` — the vendor's own `catalog.Model.Free` word, else a list price of $0 on every count — so a pool routed any other way never reads a model list for it, and one routed `free` spends the members whose model costs nothing before those that spend the allowance or credits, each tier in its listed order (nothing free is left behind a paid one, and a pool with no free model behaves as `order`). A group routed `free` weighs its members whole, each over its own accounts or keys as its provider routes them. A group routed `weight` (#1910) takes its members, not their keys, by smooth weighted round robin: `planLevel` makes each model member (its keys ordered by its own provider's routing) and each nested group one unit, and `weightedUnits` puts the round's pick (`routes.Round` over `group/<id>`, keyed by the member's spelling, `Group.Weights`, 1 if unset) first and the rest in member order for failover; a unit whose head rests leaves the round, and a kept conversation moves the round's charge to the member that kept it (`wrrSeat`, `routes.MoveRound`). A lone key or account is weighed against nobody, but what it has left is still read into the trace (`weighRouted`), so the Routing page shows it as it does for several (#1016); all but used up, it keeps its conversation rather than being said to move (`affine`). A failure rests a candidate (`failure`, `failureOf`). | [`routing.go`](../../internal/gateway/routing.go), [`weighted.go`](../../internal/gateway/weighted.go), [`internal/routes/`](../../internal/routes), [`internal/provider/routing.go`](../../internal/provider/routing.go) |
| Limits | Each key or account (a candidate's `who()`) has at most `MaxConcurrency` requests out at once (its own `AccountConcurrency` over it); one more waits in its lane's queue, bounded by `QueueLimit` and `QueueWait` and turned away with a 429 past either (`lanes.take`). `MaxRPM` (coeo91 on Discord: OpenRouter's free models take 20 a minute) bounds how many requests it sends the vendor in any rolling 60 seconds (`rpms`), counting each for a second more (`rpmSlack`), since the vendor counts a request when it arrives and the first on a new connection arrives later by its dial: one more waits for room in the minute, in the order they came, after it has its slot and right before it is sent. A wait that would be longer than 2 minutes (or the provider's `QueueWait`, when shorter: `rpmWait`) is turned away at once with a 429 whose `Retry-After` is when the minute has room; nothing rests for it. An agent gone while it waits gives its place back. Every request that reaches the vendor counts: each try of the candidate loop (a retry, the next key, a fallback) waits in `serve`, and the requests a try makes on the way and the gateway's own asks (count_tokens, System One's decisions, embeddings, drawing, Kimi's search) are counted by `rpmTransport` on `s.client` (`metered`, `paidFor`). count_tokens never waits: a candidate with no room is skipped, and the local estimate answers. A full lane or a minute with no room gives way to a ready key or account of the provider or the group (`laneMate`, `ready`); a fallback is never asked for want of either. A plugin's account sends through its own transport, so only its tries are counted, not a plugin's own extra requests. Set in the provider editor (Concurrency, Requests per minute, Queue size, Queue wait) or with `magpie provider account-concurrency`, `queue` and `rpm`; a subscription's are kept with its picks, and every account of it keeps them, those on beside the first as well (`withPicks`, called by `All` and `AlsoOn`; #1899: a second WorkBuddy AI account had none, and queued 62s with the wait set to 1s). | [`concurrency.go`](../../internal/gateway/concurrency.go), [`rpm.go`](../../internal/gateway/rpm.go), [`internal/provider/concurrency.go`](../../internal/provider/concurrency.go) |
| Sink | With `Sink` on, a key or account that answers 429 while it still has quota goes to the back, behind every one not rate limited since. Held in memory until restart. A group in a group sinks by its own Sink, within its own order (`sinkOwn`, #2255), and the group asked for then sinks the whole by its own; `TestNestedGroupSinksItsOwnOrder`. | [`sink.go`](../../internal/gateway/sink.go) |
| Image generation | `/v1/images/generations` and `/v1/images/edits` are translated to the selected model's image API. OpenRouter image-only models use its `/images` endpoint and send reference images as `input_references`; OpenRouter models that return images through chat stay on Chat Completions. Volcengine Ark (`isArk`: an `ark.*.volces.com` base, the preset's plans or one added by hand) is asked at `/images/generations` only, as its doc (https://www.volcengine.com/docs/82379/1541523) takes it (`drawArk`): `watermark: false`, since it marks every image otherwise; reference images as JSON `image` (a URL or data URL, an array for several), never a multipart edit; no `n`, so one request per image (`eachDrawing`); no ratio as `size`, so a ratio its doc lists is sent as that ratio's 2K size and any other is a 400 that says to put it in the prompt; no mask, no quality (JL on Discord, `TestArkDrawsWithoutWatermarkOncePerImage`). A routing group named for a drawing or a video (`drawingGroup`: `group/<id>`, or a bare id the group goes by that resolved to one of its members) is tried member by member in the order its routing plans for a chat (`drawingTries` over `planGroup`), each member over its own keys or accounts (`drawOnAccounts`, `startVideo`); the next member is asked while one fails as a chat's would move on (`retryable`: its vendor 5xx, out of quota, the model not served), never for a refusal of the request itself, and the call's `Fallback` names those tried first (#2005, tianxia3111: a group of two relays answered the first one's 502). Pause rules and a gateway key's group hold apply as for a chat. Nothing rests for a failed drawing. `TestImageGroupMovesOnToTheNextMember`, `TestVideoGroupMovesOnToTheNextMember`. An account's own list of models (#474) never bars a ChatGPT account's GPT Image, a Grok subscription's Grok Imagine or a WorkBuddy plan's image models (`listsCantName`): the list is picked from the editor's chat models, which they are never among, so a list kept for some chat models had every drawing refused as "set not to be served by any account" (heyiwei5 on X). A key's list, picked from models that include its image ones, still bars them. `TestCodexDrawsPastAccountModelLists`. A drawing is given Settings → Images → Image generation timeout (`imageTimeout`, 1 to 60 minutes, 5 when unset; neighbads on X: a slow image endpoint needed more) as its context's deadline (`drawTimeout`, `drawWithin`), and so is downloading an image answered as a link (`imageLongest`, never under `ImageLongest`). A drawing's request goes through a transport with no `ResponseHeaderTimeout` (`drawsApart`), since an images API sends its headers only once it has drawn, and every other request keeps `upstreamHeaderWait` (10 minutes). A video's submit, status and download calls keep 5 minutes each (`videoTimeout`). The `magpie mcp image` process, which may not read these settings (WSL), waits on a call up to `callLongest` (two hours and a minute), so it never cuts one the gateway still waits on. `TestLongImageTimeoutLetsASlowVendorFinish`, `TestImageTimeoutSettingReachesTheVendor`, `TestOnlyDrawingsWaitPastTheHeaderWait`. | [`draw.go`](../../internal/gateway/draw.go), [`draw_volcengine.go`](../../internal/gateway/draw_volcengine.go), [`video.go`](../../internal/gateway/video.go), [`catalog.go`](../../internal/catalog/catalog.go) |
diff --git a/docs/subsystems/plugin-runtime.md b/docs/subsystems/plugin-runtime.md
index 68ad1bf45..e126f85b1 100644
--- a/docs/subsystems/plugin-runtime.md
+++ b/docs/subsystems/plugin-runtime.md
@@ -186,7 +186,7 @@ The router strategy point says the order a pool of routes is tried in: a
provider's accounts or keys, or the heads of a group's members. The gateway
still lists the pool, and keeps rests, affinity, rules, pins and the walk
over the order itself; a strategy only orders. Smart, the default, stays in
-the core. The five other routings are the built-in plugin `rotation`'s.
+the core. The six other routings are the built-in plugin `rotation`'s.
| Part | What it does | Source |
| --- | --- | --- |
diff --git a/internal/cli/groups_cli.go b/internal/cli/groups_cli.go
index d09c21d41..7bf419fa0 100644
--- a/internal/cli/groups_cli.go
+++ b/internal/cli/groups_cli.go
@@ -132,6 +132,7 @@ var routingNames = []struct {
{provider.LeastUsed, "usage", []string{"least-used"}},
{provider.Pace, "pace", []string{"weekly-pace"}},
{provider.Weighted, "weight", []string{"weighted", "by-weight", "weighted-round-robin", "wrr"}},
+ {provider.FreeFirst, "free", []string{"free-first", "freest", "no-cost"}},
{provider.Manual, "manual", []string{"pick", "picked", "pinned"}},
}
diff --git a/internal/gateway/freefirst_group_test.go b/internal/gateway/freefirst_group_test.go
new file mode 100644
index 000000000..efa94846c
--- /dev/null
+++ b/internal/gateway/freefirst_group_test.go
@@ -0,0 +1,108 @@
+package gateway
+
+import (
+ "net/http"
+ "os"
+ "path/filepath"
+ "testing"
+
+ "github.com/yetone/magpie/internal/catalog"
+ "github.com/yetone/magpie/internal/provider"
+)
+
+// liveModels writes the model list a provider's vendor would have answered,
+// where catalog.Live reads it: the truth provider.Available goes by, and so
+// what FreeModel judges a model free from.
+func liveModels(t *testing.T, id, json string) {
+ t.Helper()
+ if err := os.MkdirAll(filepath.Dir(catalog.LivePath(id)), 0o755); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(catalog.LivePath(id), []byte(json), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ catalog.Reset()
+}
+
+// A group set to Free first spends what costs nothing before what spends
+// the subscription: the member whose model its vendor serves at no cost
+// answers first, and the paid one only when that can't. magpie learns which
+// is which from the vendor's own list (catalog.Model's Free), through
+// provider.FreeModel, filled into routes.Seat.Free only for a routing that
+// weighs by it (Strategy.ByFree).
+func TestFreeFirstGroupSpendsTheFreeMemberFirst(t *testing.T) {
+ fresh(t)
+ vendor := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Header().Set("Content-Type", "application/json")
+ w.Write([]byte(`{"id":"ok","choices":[{"message":{"role":"assistant","content":"hi"}}]}`))
+ })
+ serveOn(t, "freev", "sk-free", []string{"m"}, vendor)
+ serveOn(t, "paidv", "sk-paid", []string{"m"}, vendor)
+ // the vendor's own word on one, none on the other: the second is paid
+ liveModels(t, "freev", `{"models":[{"id":"m","free":true}]}`)
+ liveModels(t, "paidv", `{"models":[{"id":"other"},{"id":"m"}]}`)
+
+ // paid is listed first, so In order would spend it; Free first must not
+ if err := provider.SaveGroup(provider.Group{
+ Name: "G", Members: []string{"paidv/m", "freev/m"}, Routing: provider.FreeFirst,
+ }); err != nil {
+ t.Fatal(err)
+ }
+
+ s := New()
+ code, body := postAs(t, s, "", `{"model":"group/g","messages":[{"role":"user","content":"x"}]}`)
+ if code != 200 {
+ t.Fatalf("code %d: %s", code, body)
+ }
+ last := s.trace.routes[len(s.trace.routes)-1]
+ if len(last.Order) < 2 {
+ t.Fatalf("the trace should weigh both members: %+v", last.Order)
+ }
+ if last.Order[0].Provider != "freev" {
+ t.Fatalf("Free first went to %q first, want freev: %+v", last.Order[0].Provider, last.Order)
+ }
+ if !last.Order[0].Free {
+ t.Fatal("the free member's trace row doesn't say Free")
+ }
+ if last.Order[len(last.Order)-1].Provider != "paidv" {
+ t.Fatalf("the paid member isn't last: %+v", last.Order)
+ }
+ if last.Order[len(last.Order)-1].Free {
+ t.Fatal("the paid member's trace row says Free")
+ }
+}
+
+// The same group routed In order spends the paid member, which is listed
+// first: nothing about the change moves a pool that doesn't ask for it.
+func TestInOrderStillGoesByTheListOverTheFreeOne(t *testing.T) {
+ fresh(t)
+ vendor := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Header().Set("Content-Type", "application/json")
+ w.Write([]byte(`{"id":"ok","choices":[{"message":{"role":"assistant","content":"hi"}}]}`))
+ })
+ serveOn(t, "freev", "sk-free", []string{"m"}, vendor)
+ serveOn(t, "paidv", "sk-paid", []string{"m"}, vendor)
+ // the vendor's own word on one, none on the other: the second is paid
+ liveModels(t, "freev", `{"models":[{"id":"m","free":true}]}`)
+ liveModels(t, "paidv", `{"models":[{"id":"other"},{"id":"m"}]}`)
+ if err := provider.SaveGroup(provider.Group{
+ Name: "G", Members: []string{"paidv/m", "freev/m"}, Routing: provider.Ordered,
+ }); err != nil {
+ t.Fatal(err)
+ }
+ s := New()
+ code, body := postAs(t, s, "", `{"model":"group/g","messages":[{"role":"user","content":"x"}]}`)
+ if code != 200 {
+ t.Fatalf("code %d: %s", code, body)
+ }
+ last := s.trace.routes[len(s.trace.routes)-1]
+ if last.Order[0].Provider != "paidv" {
+ t.Fatalf("In order went to %q first, want paidv: %+v", last.Order[0].Provider, last.Order)
+ }
+ // In order doesn't weigh by Free: no row says it
+ for _, w := range last.Order {
+ if w.Free {
+ t.Fatalf("a row of an In order trace says Free: %+v", w)
+ }
+ }
+}
diff --git a/internal/gateway/routing.go b/internal/gateway/routing.go
index 6929fdd83..23cc0ce34 100644
--- a/internal/gateway/routing.go
+++ b/internal/gateway/routing.go
@@ -1196,6 +1196,10 @@ type weighing struct {
lefts map[allowanceKey]left // the allowance of each account and model
tokens map[string]float64 // least used: tokens each served lately
+
+ // free is each seat's model costing nothing there, by rest key: filled
+ // only when the routing weighs by it (routes.Strategy.ByFree)
+ free map[string]bool
}
type allowanceKey struct {
@@ -1326,6 +1330,15 @@ func weighRouted(p provider.Provider, cs []candidate, model string, from provide
wg.tokens[c.rest] = seats[i].Tokens
}
}
+ if st.ByFree {
+ // only a routing that weighs by it pays for working it out: it
+ // reads each candidate's provider list (provider.FreeModel)
+ wg.free = map[string]bool{}
+ for i, c := range cs {
+ free := c.p.FreeModel(c.model)
+ seats[i].Free, wg.free[c.rest] = free, free
+ }
+ }
order := routes.Apply(st, p.ID, seats)
out := make([]candidate, 0, len(cs))
for _, i := range order {
diff --git a/internal/gateway/trace.go b/internal/gateway/trace.go
index b3d695d6f..52f37e40c 100644
--- a/internal/gateway/trace.go
+++ b/internal/gateway/trace.go
@@ -249,6 +249,7 @@ type Weighed struct {
Pace float64 `json:"pace,omitempty"` // remaining allowance per hour until its window resets
Due *time.Time `json:"due,omitempty"` // when the allowance window used for pace resets
Tokens float64 `json:"tokens,omitempty"` // least used: tokens it served lately
+ Free bool `json:"free,omitempty"` // its model costs nothing there (free first)
Turn bool `json:"turn,omitempty"` // in turn: it was this one's turn
Fit int `json:"fit,omitempty"` // keyFit
Speaks provider.Protocol `json:"speaks,omitempty"` // a key made for one protocol only
@@ -444,6 +445,9 @@ func weighed(c candidate, p provider.Provider, wg weighing, fallback bool, from
if wg.tokens != nil {
w.Tokens = wg.tokens[c.rest]
}
+ if wg.free != nil {
+ w.Free = wg.free[c.rest]
+ }
if sinks(p.Sink, p.Routing) {
markSunk(&w, c)
}
diff --git a/internal/gui/assets/app.js b/internal/gui/assets/app.js
index 7db524b27..943e95cb7 100644
--- a/internal/gui/assets/app.js
+++ b/internal/gui/assets/app.js
@@ -11542,11 +11542,12 @@ const ROUTINGS = [
["usage", "Least used first", "Each request goes to the one used least: a subscription by the share of its allowance used, a key by the tokens it served in the last hours."],
["pace", "Weekly pace", "Each request goes to the subscription with the most of its week left per hour until it renews — the one with the most to lose at its reset — so less of each week is lost at its reset; one at 90% or more waits until the others can't answer. A key goes by the tokens it served in the last hours."],
["weight", "By weight", "Requests spread over the keys by the weight set beside each: a key weighing 3 takes three requests for every one a key weighing 1 takes, evenly over a few requests. One that fails is passed over while it rests, and the others share its requests; a conversation stays with its key as Stays says."],
+ ["free", "Free first", "Requests go to what costs nothing first: the models their vendor serves at no cost to this subscription's allowance or credits, in their listed order, and only when those can't answer to one that spends them. Nothing that cost nothing is left unused."],
];
// the routings besides Smart are the built-in plugin rotation's
// (plugins/rotation): off, their options leave the page, as anything marked
// data-feature does, and Smart is left — one in use keeps it on
-const ROTATION = new Set(["order", "rotate", "usage", "pace", "weight"]);
+const ROTATION = new Set(["order", "rotate", "usage", "pace", "weight", "free"]);
// jsRoutings: the routings JS gateway plugins serve now ("<plugin>/<name>",
// settings' jsRoutings, plugins/middleware's), and saved, a routing none
// serves now, shown as it is: its pool is tried in its listed order
diff --git a/internal/gui/assets/i18n.js b/internal/gui/assets/i18n.js
index 91e56fbad..a579d460c 100644
--- a/internal/gui/assets/i18n.js
+++ b/internal/gui/assets/i18n.js
@@ -785,7 +785,7 @@ const I18N = {
"Send each request's traces and metrics to an OpenTelemetry collector, such as Langfuse": "把每个请求的追踪和指标发送到 OpenTelemetry 收集器,比如 Langfuse",
"Sending to a collector": "正在发送到收集器",
"Routing strategies": "路由策略",
- "The routings besides Smart: In order, In turn, Least used first, Weekly pace and By weight": "智能以外的路由方式:按顺序、轮流、用量少的优先、重置前用完和按权重",
+ "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight and Free first": "智能以外的路由方式:按顺序、轮流、用量少的优先、重置前用完、按权重和免费优先",
"1 provider or group routed so": "1 个提供商或路由组在用",
"{n} providers and groups routed so": "{n} 个提供商和路由组在用",
"Run the JavaScript gateway plugins: middleware on agents' requests and replies, routings and usage observers": "运行 JavaScript 网关插件:Agent 请求和回复上的中间件、路由方式和用量观察者",
@@ -1554,6 +1554,12 @@ const I18N = {
"In turn": "轮流",
"Least used first": "用量少的优先",
"By weight": "按权重",
+ "it costs nothing there, so it's spent first.": "它在这里不花钱,所以先用它。",
+ "Free first: {model} costs nothing on {who}, so it's tried before any that spends the allowance.": "免费优先:{model} 在 {who} 上不花额度,所以先于任何花额度的尝试。",
+ "Free first: every one that costs nothing is spent or resting, so {model} on {who}, which spends the allowance, is next.": "免费优先:不花钱的都用了或在休息,接着用会花额度的 {who} 上的 {model}。",
+ "Free first": "免费优先",
+ "Requests go to what costs nothing first: the models their vendor serves at no cost to this subscription's allowance or credits, in their listed order, and only when those can't answer to one that spends them. Nothing that cost nothing is left unused.": "先把不花钱的用完:供应商对本订阅的额度或积分不收费的模型排在前面,按它们原来的顺序;这些都不能响应时,才用要花额度的。不花额度的一个都不剩下。",
+ "Free first: the members whose model its vendor serves at no cost to the subscription's allowance or credits go first, then the rest — each whole, over its own accounts or keys as its provider routes them. The free ones are tried before any that spends, so nothing that costs nothing is left unused.": "免费优先:成员中那个模型不花订阅额度或积分的排在前面,其余在后;每个成员整体尝试,各自按其供应商的路由在自己的账号或 Key 间分配。免费的先于任何花额度的,所以不花钱的一个都不剩下。",
"Requests spread over the keys by the weight set beside each: a key weighing 3 takes three requests for every one a key weighing 1 takes, evenly over a few requests. One that fails is passed over while it rests, and the others share its requests; a conversation stays with its key as Stays says.": "请求按每个密钥旁设置的权重分配:权重 3 的密钥每接 3 个请求,权重 1 的接 1 个,几个请求之内就均匀分开。出错的在休息期间被跳过,由其他密钥按各自权重分担;会话按「会话保持」的设置留在原来的密钥上。",
"Weight {n} · {share}%": "权重 {n} · {share}%",
"Its share of the requests: a key weighing 3 takes three for every one a key weighing 1 takes": "它在请求中的份额:权重 3 的密钥每接 3 个请求,权重 1 的接 1 个",
@@ -4535,7 +4541,7 @@ const I18N = {
"Send each request's traces and metrics to an OpenTelemetry collector, such as Langfuse": "把每個請求的追蹤和指標傳送到 OpenTelemetry 收集器,例如 Langfuse",
"Sending to a collector": "正在傳送到收集器",
"Routing strategies": "路由策略",
- "The routings besides Smart: In order, In turn, Least used first, Weekly pace and By weight": "智慧以外的路由方式:按順序、輪流、用量少的優先、重置前用完和按權重",
+ "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight and Free first": "智慧以外的路由方式:按順序、輪流、用量少的優先、重置前用完、按權重和免費優先",
"1 provider or group routed so": "1 個提供商或路由組在用",
"{n} providers and groups routed so": "{n} 個提供商和路由組在用",
"Run the JavaScript gateway plugins: middleware on agents' requests and replies, routings and usage observers": "執行 JavaScript 閘道外掛:Agent 請求和回覆上的中介軟體、路由方式和用量觀察者",
@@ -5293,6 +5299,12 @@ const I18N = {
"In turn": "輪流",
"Least used first": "用量少的優先",
"By weight": "按權重",
+ "it costs nothing there, so it's spent first.": "它在這裡不花錢,所以先用它。",
+ "Free first: {model} costs nothing on {who}, so it's tried before any that spends the allowance.": "免費優先:{model} 在 {who} 上不花額度,所以先於任何花額度的嘗試。",
+ "Free first: every one that costs nothing is spent or resting, so {model} on {who}, which spends the allowance, is next.": "免費優先:不花錢的都用了或在休息,接著用會花額度的 {who} 上的 {model}。",
+ "Free first": "免費優先",
+ "Requests go to what costs nothing first: the models their vendor serves at no cost to this subscription's allowance or credits, in their listed order, and only when those can't answer to one that spends them. Nothing that cost nothing is left unused.": "先把不花錢的用完:供應商對本訂閱的額度或點數不收費的模型排在前面,按它們原本的順序;這些都不能回應時,才用要花額度的。不花額度的一個都不剩下。",
+ "Free first: the members whose model its vendor serves at no cost to the subscription's allowance or credits go first, then the rest — each whole, over its own accounts or keys as its provider routes them. The free ones are tried before any that spends, so nothing that costs nothing is left unused.": "免費優先:成員中那個模型不花訂閱額度或點數的排在前面,其餘在後;每個成員整體嘗試,各自依其供應商的路由在自己的帳號或 Key 間分配。免費的先於任何花額度的,所以不花錢的一個都不剩下。",
"Requests spread over the keys by the weight set beside each: a key weighing 3 takes three requests for every one a key weighing 1 takes, evenly over a few requests. One that fails is passed over while it rests, and the others share its requests; a conversation stays with its key as Stays says.": "請求按每個金鑰旁設定的權重分配:權重 3 的金鑰每接 3 個請求,權重 1 的接 1 個,幾個請求之內就均勻分開。出錯的在休息期間被跳過,由其他金鑰按各自權重分擔;會話按「會話保持」的設定留在原來的金鑰上。",
"Weight {n} · {share}%": "權重 {n} · {share}%",
"Its share of the requests: a key weighing 3 takes three for every one a key weighing 1 takes": "它在請求中的份額:權重 3 的金鑰每接 3 個請求,權重 1 的接 1 個",
@@ -8257,7 +8269,7 @@ const I18N = {
"Send each request's traces and metrics to an OpenTelemetry collector, such as Langfuse": "各リクエストのトレースとメトリクスを Langfuse などの OpenTelemetry コレクターに送ります",
"Sending to a collector": "コレクターに送信中",
"Routing strategies": "ルーティング戦略",
- "The routings besides Smart: In order, In turn, Least used first, Weekly pace and By weight": "スマート以外のルーティング:順番、交互、使用量が最少のものを優先、週ペース、重み付け",
+ "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight and Free first": "スマート以外のルーティング:順番、交互、使用量が最少のものを優先、週ペース、重み付け、無料優先",
"1 provider or group routed so": "1 個のプロバイダーまたはグループが使用中",
"{n} providers and groups routed so": "{n} 個のプロバイダーとグループが使用中",
"Run the JavaScript gateway plugins: middleware on agents' requests and replies, routings and usage observers": "JavaScript ゲートウェイプラグインを実行します:エージェントのリクエストと応答のミドルウェア、ルーティング、使用量オブザーバー",
@@ -9009,6 +9021,12 @@ const I18N = {
"In turn": "交互",
"Least used first": "使用量が最少のものを優先",
"By weight": "重み付け",
+ "it costs nothing there, so it's spent first.": "ここでは無料なので、先に使います。",
+ "Free first: {model} costs nothing on {who}, so it's tried before any that spends the allowance.": "無料優先:{model} は {who} では利用枠を消費しないため、消費するものより先に試します。",
+ "Free first: every one that costs nothing is spent or resting, so {model} on {who}, which spends the allowance, is next.": "無料優先:無料のものはすべて使用済みか休止中なので、次は利用枠を消費する {who} の {model} です。",
+ "Free first": "無料優先",
+ "Requests go to what costs nothing first: the models their vendor serves at no cost to this subscription's allowance or credits, in their listed order, and only when those can't answer to one that spends them. Nothing that cost nothing is left unused.": "無料のものを先に使い切ります:提供元がこのサブスクリプションの利用枠やクレジットを消費しないモデルを、一覧の順に先に試し、それらが応答できないときだけ消費するものへ進みます。無料のものが残ることはありません。",
+ "Free first: the members whose model its vendor serves at no cost to the subscription's allowance or credits go first, then the rest — each whole, over its own accounts or keys as its provider routes them. The free ones are tried before any that spends, so nothing that costs nothing is left unused.": "無料優先:提供元がサブスクリプションの利用枠やクレジットを消費しないモデルのメンバーを先に、残りを後に置きます。各メンバーはまとめて試され、それぞれの提供元のルーティングに従って自分のアカウントや Key に配分されます。無料のものは消費するものより先なので、無料のものは残りません。",
"Requests spread over the keys by the weight set beside each: a key weighing 3 takes three requests for every one a key weighing 1 takes, evenly over a few requests. One that fails is passed over while it rests, and the others share its requests; a conversation stays with its key as Stays says.": "各キーの横で設定した重みでリクエストを振り分けます。重み 3 のキーは、重み 1 のキーが 1 件受けるごとに 3 件受け、数件のうちに均等に分かれます。失敗したキーは休止中は飛ばされ、他のキーがそれぞれの重みで分担します。会話は「維持」の設定どおりそのキーにとどまります。",
"Weight {n} · {share}%": "重み {n} · {share}%",
"Its share of the requests: a key weighing 3 takes three for every one a key weighing 1 takes": "リクエストの取り分:重み 3 のキーは、重み 1 のキーが 1 件受けるごとに 3 件受けます",
@@ -11993,7 +12011,7 @@ const I18N = {
"Send each request's traces and metrics to an OpenTelemetry collector, such as Langfuse": "Traces und Metriken jeder Anfrage an einen OpenTelemetry-Collector senden, etwa Langfuse",
"Sending to a collector": "Sendet an einen Collector",
"Routing strategies": "Routing-Strategien",
- "The routings besides Smart: In order, In turn, Least used first, Weekly pace and By weight": "Die Routings neben Intelligent: Der Reihe nach, Abwechselnd, Am wenigsten genutzte zuerst, Wochentempo und Nach Gewicht",
+ "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight and Free first": "Die Routings neben Intelligent: Der Reihe nach, Abwechselnd, Am wenigsten genutzte zuerst, Wochentempo, Nach Gewicht und Kostenlos zuerst",
"1 provider or group routed so": "1 Anbieter oder Gruppe so geroutet",
"{n} providers and groups routed so": "{n} Anbieter und Gruppen so geroutet",
"Run the JavaScript gateway plugins: middleware on agents' requests and replies, routings and usage observers": "Die JavaScript-Gateway-Plugins ausführen: Middleware auf Anfragen und Antworten der Agenten, Routings und Nutzungsbeobachter",
@@ -12755,6 +12773,12 @@ const I18N = {
"In turn": "Abwechselnd",
"Least used first": "Am wenigsten genutzte zuerst",
"By weight": "Nach Gewicht",
+ "it costs nothing there, so it's spent first.": "Dort kostet es nichts, also wird es zuerst verbraucht.",
+ "Free first: {model} costs nothing on {who}, so it's tried before any that spends the allowance.": "Kostenlos zuerst: {model} kostet auf {who} nichts und wird daher vor jedem verbrauchenden Versuch probiert.",
+ "Free first: every one that costs nothing is spent or resting, so {model} on {who}, which spends the allowance, is next.": "Kostenlos zuerst: Alles Kostenlose ist verbraucht oder ruht, daher folgt {model} auf {who}, das Kontingent verbraucht.",
+ "Free first": "Kostenlos zuerst",
+ "Requests go to what costs nothing first: the models their vendor serves at no cost to this subscription's allowance or credits, in their listed order, and only when those can't answer to one that spends them. Nothing that cost nothing is left unused.": "Zuerst kommt, was nichts kostet: Modelle, die ihr Anbieter diesem Abonnement ohne Verbrauch von Kontingent oder Guthaben bereitstellt, in ihrer gelisteten Reihenfolge; erst wenn diese nicht antworten können, eines, das verbraucht. Nichts Kostenloses bleibt ungenutzt.",
+ "Free first: the members whose model its vendor serves at no cost to the subscription's allowance or credits go first, then the rest — each whole, over its own accounts or keys as its provider routes them. The free ones are tried before any that spends, so nothing that costs nothing is left unused.": "Kostenlos zuerst: die Mitglieder, deren Modell ihr Anbieter dem Abonnement ohne Verbrauch von Kontingent oder Guthaben bereitstellt, kommen zuerst, dann die übrigen — jedes als Ganzes, über seine eigenen Konten oder Keys, wie es sein Anbieter routet. Die kostenlosen werden vor jedem verbrauchenden versucht, sodass nichts Kostenloses ungenutzt bleibt.",
"Requests spread over the keys by the weight set beside each: a key weighing 3 takes three requests for every one a key weighing 1 takes, evenly over a few requests. One that fails is passed over while it rests, and the others share its requests; a conversation stays with its key as Stays says.": "Anfragen verteilen sich nach dem Gewicht neben jedem Schlüssel: Einer mit Gewicht 3 bekommt drei Anfragen für jede, die einer mit Gewicht 1 bekommt, schon über wenige Anfragen gleichmäßig. Ein fehlschlagender wird übersprungen, solange er pausiert, und die anderen teilen sich seine Anfragen; ein Gespräch bleibt bei seinem Schlüssel, wie „Bleibt“ es sagt.",
"Weight {n} · {share}%": "Gewicht {n} · {share}%",
"Its share of the requests: a key weighing 3 takes three for every one a key weighing 1 takes": "Sein Anteil an den Anfragen: Ein Schlüssel mit Gewicht 3 bekommt drei für jede, die einer mit Gewicht 1 bekommt",
diff --git a/internal/gui/assets/plugins/routing/client.js b/internal/gui/assets/plugins/routing/client.js
index c44562ee0..e213da321 100644
--- a/internal/gui/assets/plugins/routing/client.js
+++ b/internal/gui/assets/plugins/routing/client.js
@@ -450,6 +450,7 @@ function routingPage(part) {
// editor's routing options say them (app.js ROUTINGS), so every
// language is already spoken
weight: ["By weight", "Requests spread over the keys by the weight set beside each: a key weighing 3 takes three requests for every one a key weighing 1 takes, evenly over a few requests. One that fails is passed over while it rests, and the others share its requests; a conversation stays with its key as Stays says."],
+ free: ["Free first", "Requests go to what costs nothing first: the models their vendor serves at no cost to this subscription's allowance or credits, in their listed order, and only when those can't answer to one that spends them. Nothing that cost nothing is left unused."],
};
// modeOf: a routing's name and what it does, a JS gateway plugin's
// ("<plugin>/<name>", app.js jsRoutings) as it is named, served or not
@@ -585,6 +586,10 @@ function routingPage(part) {
: t("In order: {who} is first, and answers everything while it can.", { who: w });
case "rotate": return t("In turn: it's {who}'s turn — each request starts one further along.", { who: w });
case "weight": return t("By weight: it's {who}'s share — each key takes requests as its weight says.", { who: w });
+ case "free":
+ if (f.free) return restedTo(t("it costs nothing there, so it's spent first."))
+ || t("Free first: {model} costs nothing on {who}, so it's tried before any that spends the allowance.", { model: f.model, who: w });
+ return t("Free first: every one that costs nothing is spent or resting, so {model} on {who}, which spends the allowance, is next.", { model: f.model, who: w });
case "usage":
if (f.known) return t("Least used first: {who} has the most of its allowance left — {n} used.", { who: w, n: pct(f.used) });
if (f.kind === "key" && !someKnown) return t("Least used first: {who} served the fewest tokens lately — {n}.", { who: w, n: tokens(f.tokens || 0) });
@@ -3584,7 +3589,7 @@ function routingPage(part) {
// after the requests: a request picked in the list plays on the stage,
// so the list sits right under it
more.append(gsec);
- const ROUTE_OPTS = [["", "Smart"], ["order", "In order"], ["rotate", "In turn"], ["usage", "Least used"], ["pace", "Weekly pace"]];
+ const ROUTE_OPTS = [["", "Smart"], ["order", "In order"], ["rotate", "In turn"], ["usage", "Least used"], ["pace", "Weekly pace"], ["free", "Free first"]];
// a group may also be routed by hand: every request to the member the
// user picks on its card (provider.Manual, #317) — a provider's keys can't
// and by its members' weights, as a provider's keys are (#1910)
@@ -3629,6 +3634,7 @@ function routingPage(part) {
usage: "Least used first: the account or key with the most of its allowance left goes first.",
pace: "Weekly pace: the account with the most of its week left per hour until it renews goes first, so less of each member's week is lost at its reset — an account with five hours and no week (Claude Enterprise) by what its five hours have left per hour until they renew, so almost always first; a key by the tokens magpie sent it lately.",
weight: GROUP_WEIGHT,
+ free: "Free first: the members whose model its vendor serves at no cost to the subscription's allowance or credits go first, then the rest — each whole, over its own accounts or keys as its provider routes them. The free ones are tried before any that spends, so nothing that costs nothing is left unused.",
manual: "Manual: every request goes to the model you pick on the group's card, over its own accounts or keys; the others, and the rules, wait until you pick another — none takes over when it fails.",
};
// a group in the group, and a provider set to route other than Smart, is
diff --git a/internal/gui/core_budget_test.go b/internal/gui/core_budget_test.go
index 26823042d..e59b8d94e 100644
--- a/internal/gui/core_budget_test.go
+++ b/internal/gui/core_budget_test.go
@@ -15,7 +15,7 @@ import (
// so a page can't quietly grow back into the core; a change that needs more
// raises them and says why in its commit.
const (
- appScriptBudget = 1_269_000 // app.js: 1,198,239 after Usage left (P2); 1,218,098 with the 10-15 fixes to pages still in the core; 1,232,466 with the provider editor's Advanced fold and every dialog holding its place, the core's own (10-16); 1,250,533 with the quota refresh kept through redraws, the core's own since the tray panel draws it too, plus 1.5% (#2397)
+ appScriptBudget = 1_271_000 // app.js: 1,198,239 after Usage left (P2); 1,218,098 with the 10-15 fixes to pages still in the core; 1,232,466 with the provider editor's Advanced fold and every dialog holding its place, the core's own (10-16); 1,250,533 with the quota refresh kept through redraws, the core's own since the tray panel draws it too, plus 1.5% (#2397); 1,269,083 with Free first's option in the provider editor's ROUTINGS, which the Routing page reads too, plus 1.5%
coreScriptBudget = 1_469_000 // every core script: 1,358,034 after Usage left; 1,380,003 with them; 1,405,937 with the fold; 1,426,102 with 10-16's core work (the GitHub list loading, the move to plugins, Settings › Long conversations), plus 1.5% (#2612); 1,447,368 with loadBack, the window's one state read per return, which every page needs and so is the core's own (1,446,845 before it, 155 bytes under), plus 1.5%; OpenCode Zen's card saying OpenCode tells no key its balance adds ~390 bytes inside it
)
diff --git a/internal/gui/tests/builtins.test.cjs b/internal/gui/tests/builtins.test.cjs
index 423b60a23..3f89c1abb 100644
--- a/internal/gui/tests/builtins.test.cjs
+++ b/internal/gui/tests/builtins.test.cjs
@@ -26,7 +26,7 @@ const PLUGINS = [
{ id: "middleware", name: "JS gateway plugins", sub: "Run the JavaScript gateway plugins: middleware on agents' requests and replies, routings and usage observers", one: "1 gateway plugin on", many: "{n} gateway plugins on" },
{ id: "otel", name: "Observability", sub: "Send each request's traces and metrics to an OpenTelemetry collector, such as Langfuse", one: "Sending to a collector", many: "Sending to a collector" },
{ id: "redact", name: "Privacy masking", sub: "Secrets, personal data and your own words go to vendors as placeholders, and come back as they were", one: "Masking is on", many: "Masking is on" },
- { id: "rotation", name: "Routing strategies", sub: "The routings besides Smart: In order, In turn, Least used first, Weekly pace and By weight", one: "1 provider or group routed so", many: "{n} providers and groups routed so" },
+ { id: "rotation", name: "Routing strategies", sub: "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight and Free first", one: "1 provider or group routed so", many: "{n} providers and groups routed so" },
{ id: "routing", name: "Routing", sub: "Each request as the gateway routes it, the accounts it weighed and what each answered, and your routing groups", one: "1 group in use", many: "{n} groups in use" },
{ id: "search", name: "Web search", sub: "Models that can't search the web search with a provider that can, or a search API", one: "1 search source set up", many: "{n} search sources set up" },
{ id: "sessions", name: "Sessions", sub: "Your agents' past conversations: read, search, resume and export them", one: "Recording gateway conversations", many: "Recording gateway conversations" },
diff --git a/internal/gui/tests/routing-save.test.cjs b/internal/gui/tests/routing-save.test.cjs
index 90ab2e4fc..2ea11c858 100644
--- a/internal/gui/tests/routing-save.test.cjs
+++ b/internal/gui/tests/routing-save.test.cjs
@@ -157,7 +157,7 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",
// on again, every routing is back
await page.evaluate(() => applyFeatures([{ id: "rotation", on: true }]));
const back = await field.locator(".segs").first().evaluate((box) => [...box.querySelectorAll(".opt")].filter((b) => getComputedStyle(b).display !== "none").length);
- assert.equal(back, p.account ? 5 : 6, `${where}: every routing back`);
+ assert.equal(back, p.account ? 6 : 7, `${where}: every routing back`);
assert.deepEqual(errors, [], where);
await page.context().close();
}
diff --git a/internal/provider/freemodel_test.go b/internal/provider/freemodel_test.go
new file mode 100644
index 000000000..24f8a3776
--- /dev/null
+++ b/internal/provider/freemodel_test.go
@@ -0,0 +1,76 @@
+package provider
+
+import (
+ "os"
+ "path/filepath"
+ "testing"
+
+ "github.com/yetone/magpie/internal/catalog"
+)
+
+// liveFree writes the list a vendor's /models answered, where
+// catalog.Live reads it: what Available goes by, and so FreeModel.
+func liveFree(t *testing.T, id, json string) {
+ t.Helper()
+ if err := os.MkdirAll(filepath.Dir(catalog.LivePath(id)), 0o755); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(catalog.LivePath(id), []byte(json), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ catalog.Reset()
+}
+
+// FreeModel is what Free first goes by: a model its vendor lists at no cost
+// is free, one the vendor prices is not, and a model the list doesn't name
+// is not either — nothing says it costs nothing, so Free first puts it with
+// the paid ones rather than inventing a saving.
+func TestFreeModelGoesByTheVendorsOwnList(t *testing.T) {
+ prefsHome(t)
+ p := Provider{ID: "fv", Name: "FV", Key: "sk-fv", Chat: "http://127.0.0.1:9/v1"}
+ if err := Save(p); err != nil {
+ t.Fatal(err)
+ }
+ liveFree(t, "fv", `{"models":[
+ {"id":"free-one","free":true},
+ {"id":"paid-one","cost":{"input":3,"output":15}},
+ {"id":"zero-priced"}
+ ]}`)
+ got, err := Find("fv")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !got.FreeModel("free-one") {
+ t.Error("a model its vendor lists free isn't free")
+ }
+ if got.FreeModel("paid-one") {
+ t.Error("a model its vendor prices as paid is free")
+ }
+ if got.FreeModel("nobody-lists-this") {
+ t.Error("a model the list doesn't name is free")
+ }
+}
+
+// The vendor's own word is what decides it, however many models it so
+// marks: a pool where some are free and some are not puts only the marked
+// ones first, and one where none is marked behaves as In order.
+func TestFreeModelIsOnlyWhereTheVendorSaysSo(t *testing.T) {
+ prefsHome(t)
+ p := Provider{ID: "mv", Name: "MV", Key: "sk-mv", Chat: "http://127.0.0.1:9/v1"}
+ if err := Save(p); err != nil {
+ t.Fatal(err)
+ }
+ liveFree(t, "mv", `{"models":[{"id":"marked","free":true},{"id":"plain"},{"id":"also-marked","free":true}]}`)
+ got, err := Find("mv")
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, model := range []string{"marked", "also-marked"} {
+ if !got.FreeModel(model) {
+ t.Errorf("%s isn't free", model)
+ }
+ }
+ if got.FreeModel("plain") {
+ t.Error("plain is free")
+ }
+}
diff --git a/internal/provider/group.go b/internal/provider/group.go
index 241d3955d..e8edea936 100644
--- a/internal/provider/group.go
+++ b/internal/provider/group.go
@@ -1266,7 +1266,7 @@ func SaveGroup(g Group) error {
if err := cleanDecisionGroup(&g, groupsIn(entries)); err != nil {
return err
}
- if g.Routing != Ordered && g.Routing != Rotate && g.Routing != LeastUsed && g.Routing != Pace && g.Routing != Weighted && g.Routing != Manual && !PluginRouting(g.Routing) {
+ if g.Routing != Ordered && g.Routing != Rotate && g.Routing != LeastUsed && g.Routing != Pace && g.Routing != Weighted && g.Routing != FreeFirst && g.Routing != Manual && !PluginRouting(g.Routing) {
g.Routing = ""
}
if !slices.Contains(Affinities, g.Affinity) {
diff --git a/internal/provider/models.go b/internal/provider/models.go
index f195280a8..3f816b5d3 100644
--- a/internal/provider/models.go
+++ b/internal/provider/models.go
@@ -1640,6 +1640,30 @@ func (p Provider) PaidBesideFree(model string) bool {
return listed && paid && free
}
+// FreeModel says model costs nothing on p: its vendor serves it at no cost
+// to the subscription's allowance or credits (catalog.Model's Free:
+// WorkBuddy's "x0.00", Qoder's is_free, Cline's free models, an OpenCode Zen
+// free one), or, with no such word from the vendor, it is listed at $0 on
+// every count — a vendor's free model a catalog prices at 0, as
+// mimo-v2.6-flash-free is, and a key's free model, which has no allowance
+// to draw on and so is judged by its price alone.
+//
+// A model neither listed free nor priced is not free: nothing says it costs
+// nothing, and the routes that go by this put it with the paid ones, so a
+// pool without free models behaves as In order.
+func (p Provider) FreeModel(model string) bool {
+ for _, m := range p.Available() {
+ if strings.EqualFold(m.ID, model) {
+ if m.Free {
+ return true
+ }
+ break
+ }
+ }
+ pr, ok := p.ListPrice(model)
+ return ok && pr.Input == 0 && pr.Output == 0 && pr.CacheRead == 0 && pr.CacheWrite == 0
+}
+
// Known are the model's own reasoning levels, when known.
func (p Provider) Known(model string) []string {
for _, m := range p.Available() {
diff --git a/internal/provider/provider.go b/internal/provider/provider.go
index dc4241bf3..6e8a7beec 100644
--- a/internal/provider/provider.go
+++ b/internal/provider/provider.go
@@ -1281,7 +1281,7 @@ func normalize(p Provider) Provider {
if (p.Preset == OMLXPreset || p.Preset == MLXServePreset || p.Preset == "ollama" || p.Preset == "ollama-cloud") && p.Decide == "" && p.Chat != "" {
p.Decide = p.Chat
}
- if p.Routing != Ordered && p.Routing != Rotate && p.Routing != LeastUsed && p.Routing != Pace && p.Routing != Weighted && !PluginRouting(p.Routing) {
+ if p.Routing != Ordered && p.Routing != Rotate && p.Routing != LeastUsed && p.Routing != Pace && p.Routing != Weighted && p.Routing != FreeFirst && !PluginRouting(p.Routing) {
p.Routing = ""
}
if !slices.Contains(Affinities, p.Affinity) {
diff --git a/internal/provider/routing.go b/internal/provider/routing.go
index b5be1394c..7d19e817b 100644
--- a/internal/provider/routing.go
+++ b/internal/provider/routing.go
@@ -28,6 +28,10 @@ const (
// Weighted spreads a provider's requests over its keys by each key's
// weight (KeyAccount.Weight, #841), smoothly: 3 and 1 go a, a, b, a…
Weighted = "weight"
+ // FreeFirst spends what costs nothing first: the models its vendor
+ // serves at no cost to the subscription's allowance or credits, before
+ // the ones that spend it (routes.FreeFirst).
+ FreeFirst = "free"
)
// pluginRouting is a routing a plugin serves: "<plugin>/<name>".
diff --git a/internal/routes/freefirst_test.go b/internal/routes/freefirst_test.go
new file mode 100644
index 000000000..e2dcb525f
--- /dev/null
+++ b/internal/routes/freefirst_test.go
@@ -0,0 +1,80 @@
+package routes
+
+import (
+ "slices"
+ "testing"
+)
+
+// FreeFirst spends what costs nothing before what spends a subscription's
+// allowance: the free seats in their listed order, then the paid ones in
+// theirs — nothing free left behind a paid one, and failover inside a tier
+// staying in the order the user set.
+func TestFreeFirstPutsFreeAheadOfPaid(t *testing.T) {
+ cases := []struct {
+ name string
+ seats []Seat
+ want []int
+ }{
+ {
+ // b and d cost nothing: they go first, each where it stood
+ name: "free among paid",
+ seats: []Seat{{Key: "a"}, {Key: "b", Free: true}, {Key: "c"}, {Key: "d", Free: true}},
+ want: []int{1, 3, 0, 2},
+ },
+ {
+ // every seat free: their order alone
+ name: "all free",
+ seats: []Seat{{Key: "a", Free: true}, {Key: "b", Free: true}},
+ want: []int{0, 1},
+ },
+ {
+ // none free: In order, so a routing set on a pool without free
+ // models changes nothing
+ name: "none free",
+ seats: []Seat{{Key: "a"}, {Key: "b"}, {Key: "c"}},
+ want: []int{0, 1, 2},
+ },
+ {
+ name: "the last alone free",
+ seats: []Seat{{Key: "a"}, {Key: "b"}, {Key: "c", Free: true}},
+ want: []int{2, 0, 1},
+ },
+ }
+ for _, c := range cases {
+ t.Run(c.name, func(t *testing.T) {
+ if got := Apply(FreeFirst, "group/g", c.seats); !slices.Equal(got, c.want) {
+ t.Fatalf("order = %v, want %v", got, c.want)
+ }
+ })
+ }
+}
+
+// FreeFirst asks the gateway for Free (Strategy.ByFree), since nothing else
+// fills it; a routing that doesn't is not to pay for reading each model
+// list.
+func TestFreeFirstIsByFree(t *testing.T) {
+ if !FreeFirst.ByFree {
+ t.Fatal("FreeFirst doesn't ask the gateway for Free")
+ }
+ for name, s := range Builtin {
+ if name == "free" {
+ continue
+ }
+ if s.ByFree {
+ t.Fatalf("%s weighs by Free too, and would read every model list", name)
+ }
+ }
+}
+
+// The routing a provider or group is saved with is the one routes.Find
+// knows: "free" is in Builtin, so plugins/rotation serves it and a saved
+// one isn't read as an unknown routing (tried in its listed order).
+func TestFreeFirstIsABuiltin(t *testing.T) {
+ s, ok := Builtin["free"]
+ if !ok {
+ t.Fatal("\"free\" is not among Builtin")
+ }
+ if s.Order == nil {
+ t.Fatal("\"free\" has no order")
+ }
+}
diff --git a/internal/routes/routes.go b/internal/routes/routes.go
index e14fd6aee..b501f1171 100644
--- a/internal/routes/routes.go
+++ b/internal/routes/routes.go
@@ -42,6 +42,10 @@ type Seat struct {
Resting bool
// Tokens is what magpie sent it lately, decayed by an hour's half-life
Tokens float64
+ // Free: the model this seat would be asked for costs nothing on it —
+ // its vendor serves that model at no cost to the subscription's
+ // allowance or credits (catalog.Model's Free). FreeFirst goes by it.
+ Free bool
}
// Strategy orders a pool.
@@ -57,6 +61,10 @@ type Strategy struct {
Exact bool
// ByUse: it weighs by Tokens, which the routing trace then shows
ByUse bool
+ // ByFree: it weighs by Free, so the gateway works out which seat's
+ // model costs nothing there, and the routing trace then shows it.
+ // Only when set: working it out reads each provider's model list.
+ ByFree bool
}
var reg = struct {
diff --git a/internal/routes/strategies.go b/internal/routes/strategies.go
index bfca6ae18..c5312e480 100644
--- a/internal/routes/strategies.go
+++ b/internal/routes/strategies.go
@@ -211,6 +211,36 @@ func First(best, n int) []int {
return append(append([]int{best}, idx[:best]...), idx[best+1:]...)
}
+// FreeFirst spends what costs nothing before what is paid for: the seats
+// whose model its vendor serves at no cost to the subscription's allowance
+// or credits go first, in their listed order, and the rest after them, also
+// in their listed order. So a request whose model is free somewhere on the
+// pool uses that first, and only when those can't answer does it go to one
+// that spends credits or quota — nothing is left unused that cost nothing.
+//
+// It orders one tier only, unlike Smart: a free seat is not pushed behind a
+// paid one for running low, since the free one spends nothing by running
+// out of. Among the paid, the listed order stands rather than Smart's
+// renewal-weighted one, because what this routing is for is the free tier;
+// set a provider's own routing (Smart, in order, weekly pace) if the paid
+// seats should be spent smartly among themselves — a provider or group in
+// this group keeps its own routing either way.
+//
+// A seat whose model costs nothing nowhere is not free: it goes with the
+// paid ones, so a routing set on a pool without free models behaves as In
+// order.
+var FreeFirst = Strategy{ByFree: true, Order: func(_ string, seats []Seat) []int {
+ free, paid := make([]int, 0, len(seats)), make([]int, 0, len(seats))
+ for i, s := range seats {
+ if s.Free {
+ free = append(free, i)
+ } else {
+ paid = append(paid, i)
+ }
+ }
+ return append(free, paid...)
+}}
+
// Builtin are the routings besides Smart and their strategies, as
// plugins/rotation registers them.
var Builtin = map[string]Strategy{
@@ -219,4 +249,5 @@ var Builtin = map[string]Strategy{
provider.LeastUsed: LeastUsed,
provider.Pace: Pace,
provider.Weighted: Weighted,
+ provider.FreeFirst: FreeFirst,
}
diff --git a/internal/tui/routing.go b/internal/tui/routing.go
index 268e1c3ff..e455a1b29 100644
--- a/internal/tui/routing.go
+++ b/internal/tui/routing.go
@@ -34,7 +34,7 @@ func (m *model) reloadGroups() {
m.grow = clamp(m.grow, len(m.groups))
}
-var routings = []string{"", provider.Ordered, provider.Rotate, provider.LeastUsed, provider.Pace, provider.Weighted, provider.Manual}
+var routings = []string{"", provider.Ordered, provider.Rotate, provider.LeastUsed, provider.Pace, provider.Weighted, provider.FreeFirst, provider.Manual}
func routingName(v string) string {
switch v {
@@ -48,6 +48,8 @@ func routingName(v string) string {
return "weekly pace"
case provider.Weighted:
return "by weight"
+ case provider.FreeFirst:
+ return "free first"
case provider.Manual:
return "manual"
}
diff --git a/plugins/rotation/rotation.go b/plugins/rotation/rotation.go
index 160e29b67..85a7f2538 100644
--- a/plugins/rotation/rotation.go
+++ b/plugins/rotation/rotation.go
@@ -1,6 +1,7 @@
// Package rotation is the routings a provider or group can be set to
-// besides Smart: In order, In turn, Least used first, Weekly pace and By
-// weight. While a provider or group is routed by one, it stays on.
+// besides Smart: In order, In turn, Least used first, Weekly pace, By
+// weight and Free first. While a provider or group is routed by one, it
+// stays on.
//
// It is the first consumer of the gateway's router strategy point (P3 of
// docs/subsystems/plugin-runtime.md): Apply registers internal/routes'
@@ -17,7 +18,7 @@ import (
func init() {
pluginrt.Register(pluginrt.Manifest{
ID: "rotation", API: "^1", Requires: []string{"providers", "gateway"}, Essential: true,
- Name: "Routing strategies", Sub: "The routings besides Smart: In order, In turn, Least used first, Weekly pace and By weight",
+ Name: "Routing strategies", Sub: "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight and Free first",
One: "1 provider or group routed so", Many: "{n} providers and groups routed so",
InUse: inUse,
Parts: []pluginrt.Part{{Slot: "part", Key: "rotation"}},