Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

routing: Speed first sends a request to the provider or key that answered the model fastest lately

yetoned9a8148d7c1e
diff --git a/docs/reference.md b/docs/reference.md
index 082a4213b..94113cd30 100644
--- a/docs/reference.md
+++ b/docs/reference.md
@@ -1460,7 +1460,11 @@ is lost at its reset) or `weight` (each model takes its share of the
requests by its weight, smoothly interleaved: `weight=claude/claude-opus-5-5=3`
gives it three requests for every one a model left at weight 1 takes; a
weight is 1 to 1000, `weight=` alone clears them all, and a group in the
-group counts as one member, `weight=group/fast=2`). A key has no allowance to weigh, so it goes by its
+group counts as one member, `weight=group/fast=2`). `fast` (speed first; also `fastest`, `speed`) sends each request to
+the member that answered fastest lately, by the time to its first token
+and how fast it wrote: compare the same model on several API channels and
+the quickest goes first. `magpie provider set <id> routing=fast` does the
+same for one provider's keys. A key has no allowance to weigh, so it goes by its
order or by what magpie sent it lately — except a sub2api key its owner gave
a 5-hour, day or 7-day limit: with the provider's Balance URL set to the
relay's `/v1/usage`, its card shows those windows and routing weighs it by
@@ -1648,6 +1652,18 @@ each of these groups ordered by the share used.
among them the one magpie sent the fewest tokens lately goes first. A
Claude account not known goes first once, as in Smart; another unknown
account counts as a fresh week.
+- **Speed first**: the account, key or group member that answered this
+ model fastest lately first. magpie keeps, in memory, how long each one's
+ streamed answers took to their first token and how many tokens a second
+ they wrote after it, newer answers counting more (a 20-minute
+ half-life), and ranks them by the time a 200-token reply would take.
+ Those within a tenth of each other count as alike and keep their order.
+ One with no answer in the last hour isn't measured: one request in five
+ goes to such a one, so a channel that was slow once is tried again. One
+ resting after a failure goes behind the others, as in any routing.
+ Replies that aren't streamed tell no speed, and a restart starts the
+ measuring over. The Routing page says why the first went first:
+ "answered fastest lately — first token in 0.8s, 62 tokens a second".
The routings besides Smart are the built-in plugin `rotation` (Routing
strategies in Plugins › Built-in). It stays on while a provider or group is
@@ -1670,6 +1686,7 @@ depends on the routing:
| Smart | Only between accounts whose windows all renew in the same hours, or whose shares used are equal. |
| Least used first | Only between accounts with the same share used and the same tokens sent lately. |
| Weekly pace | Only between accounts in the same pace band that magpie sent the same tokens lately, in practice none since magpie started. |
+| Speed first | Only between those within a tenth of each other's speed, and among those not measured lately. |
For Claude Code and Codex, *Make first* also signs the agent in to that
account, because the account the agent is signed in to is first. With
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index d0dc68011..f15b5f04a 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -33,7 +33,7 @@ backfilled; see [Gateway Session Aggregation](gateway-sessions.md).
| Model resolution | `serve` resolves the model the agent asked for: Claude tier stand-ins, a routing group (`provider.GroupFor`: its id, the model's id however a vendor spells it, else the one group whose name it is, unless a provider serves a model by that id; the user's own groups only while the `groups` plugin is on, [the plugin runtime](plugin-runtime.md#gateway-extension-points-p3-router-strategy)), auto stand-ins, the agent's own stand-in (`StandIn`: Codex's config.toml `model` when magpie is its provider, for a name it sends that magpie doesn't serve, such as its auto-review's `codex-auto-review`), itself taken through `GroupFor` and auto stand-ins so a config naming a group bare reaches it, then `provider.Resolve`. A Codex memory call or auto-review (`x-openai-subagent: guardian`, or `codex-auto-review`) that still names no served model goes to the model Codex's last turn was answered on (`codexMemoryStandIn`, per agent and gateway key), so a Codex on magpie through another provider table, a profile or another computer isn't answered 404 (Adam on Discord, `TestCodexAutoReviewReachesItsTurnsModel`). A bare model id several providers serve (MOMO on Discord: `glm-5.3` under opencode-go and a6api) is never refused: with found groups on (the default) `GroupFor` takes it to the group magpie found for it (`auto-<slug>`), its members in the Providers order, routed and failed over as any group, a conversation staying with the member that answered it while its cache is worth it (affinity); with them off, `Resolve` takes the first provider in the Providers order that exposes it, with no fallback. magpie writes only `provider/model` ids into agents' configs (`magpieModels`), so a bare id is one the user wrote. `/v1/models` lists bare ids only when the user asks for them ([Model ids without the provider](#model-ids-without-the-provider)). Every reply says who answered in `X-Magpie-Provider` and `X-Magpie-Model` (`noteMember`), and the usage log and the Routing page keep it. A model id with `~` and no slash that resolves as nothing else is the one served id (a provider's model or a group) whose slashes it spells as `~` (`unflat`, `provider.Unflat`; MiniMax Code's spelling, #1387), taken back in `validateModel` before key limits, middleware and usage see it; two ids spelled alike are neither. A Claude model id whose version is dashed (`relay/claude-opus-4-6-us`, with or without `:level` and `[1m]`) that is not served as spelled is the one served id it dashes (`provider.ClaudeUndashed`; the spelling magpie gives Claude Code, which reads a dotted version as Claude Opus 4), taken back in `validateModel` and `unprefixed` the same way. An unknown model gets a 404 naming why: a switched-off provider, a disabled group or an empty group. Otherwise it names up to three ids nearest to what was asked (`provider.Closest`: words of a model's display name count before words of its id, and a provider's id or name only alongside them, so "Zen-Space-Bunny" is told `opencode-zen-free/space-bunny-free`, MOMO on Discord), then the ways a model can be named, then every id (`unknownModel`; chat and embeddings). | [`gateway.go`](../../internal/gateway/gateway.go), [`internal/provider/group.go`](../../internal/provider/group.go), [`internal/provider/flat.go`](../../internal/provider/flat.go), [`internal/provider/claudedash.go`](../../internal/provider/claudedash.go) |
| A found group's two ids | A group magpie found is listed (`/v1/models`, the catalog, the lists written into agents, the pickers, the CLI and TUI) as `auto/<model>`: the model's id as a vendor spells it, dots kept and with no vendor prefix (`auto/claude-opus-5.5`; `provider.Group.Ref`, `spelledIn`, preferring a spelling models.dev lists), so a client that looks the part after the slash up on models.dev finds it (ZeroClover on X). Its old id `group/auto-<slug>` is served forever. Inside magpie the group keeps that one id: `validateModel` takes `auto/<model>` (with `:level` or `[1m]`) back to it with `provider.CanonicalRef` before key limits, routing, affinity and the usage log see the request, and the request body is rewritten to it, so both ids count as one group. A gateway key naming either id, `group/*` or `auto/*` passes for both (`allows` in `key_models.go`). `SaveGroup` stores a member, rule or classifier picked as `auto/<model>` by the old id. An agent field magpie set to `group/auto-<slug>` moves to `auto/<model>` on the next `SyncCatalog` (`agent.renameFoundGroups`), but only while its value is still what applied.json says magpie wrote; a value the user set or changed stays. A pick of the group by its old id written now (`Agent.Apply`) is written as `auto/<model>`. An agent's hidden, picked and ordered models saved by the old id hold for the new one (`PublicRef` in `visible.go`), and a bare model id still reaches its found group (`bareTarget`). Claude Code is given a dotted Claude version dashed, so it may send `auto/claude-sonnet-4-5`: that is the same group, and `Reseat` still moves an agent off it once the group is gone (`samePick`). `auto` is no provider id: `Save` and `Rename` refuse it for a new provider, and a plugin's `auto` is `auto-plugin` (`PluginID`). A provider of the user's saved as `auto` before keeps its models: while it exists, `auto/<x>` is its model and found groups list as `group/auto-<slug>` (`autoRefsOn`). Hand-made groups stay `group/<id>`. | [`internal/provider/autoref.go`](../../internal/provider/autoref.go), [`key_models.go`](../../internal/gateway/key_models.go), [`internal/agent/autoref.go`](../../internal/agent/autoref.go) |
| Candidates | `plan` lists one provider's candidates: one per key or account on, then each `Fallback` model, with resting ones last (`restLast`). `planGroup` and `planLevel` weigh a group's members together. A nested group plans by its own routing (#576). So does a member whose provider routes other than Smart (`ownRouting`): weighed as one by its first not resting (`firstAwake`), tried whole in its own order; `leastHeld` leaves such a group's order alone too. A provider on Smart, or routed as the group is, is pooled with the group's other keys and accounts. `group_own_routing_test.go`. A member's own fallbacks are not the group's. | [`fallback.go`](../../internal/gateway/fallback.go) |
-| Order of keys and accounts | `Routing`: smart (""), `order`, `rotate`, `usage` (least used, with an `usageHalfLife` of 1h), `pace`, `weight` or `free` (free first). `weighRouted` reads each candidate into a `routes.Seat` and asks the strategy serving its routing (`routes.Find`, the router strategy point of [the plugin runtime](plugin-runtime.md#gateway-extension-points-p3-router-strategy)) for the order: Smart is `routes.Smart`, the core's; the others are `routes.Builtin`, registered by the built-in plugin `rotation`. With it off a routing no strategy serves is tried in its listed order, and `routingOf` treats it as `order` wherever the core asks about a routing (a turn's mark, `goesRound`, `sinks`, a rotate group's sessions, a group's heads). Smart, `usage` and `pace` weigh a subscription by its allowance (`allowances`), and a key by its own usage windows when magpie reads them (`keyAllowance`: a sub2api key given a 5-hour, day or 7-day limit, or a key bought as a plan, such as Kimi Code's or a GLM Coding Plan's); any other key has none. `free` weighs by what a model costs there and nothing else: only a strategy with `routes.Strategy.ByFree` set (`routes.FreeFirst` is the one) has `weighRouted` fill `routes.Seat.Free`, from `provider.FreeModel` — the vendor's own `catalog.Model.Free` word, else a list price of $0 on every count — so a pool routed any other way never reads a model list for it, and one routed `free` spends the members whose model costs nothing before those that spend the allowance or credits, each tier in its listed order (nothing free is left behind a paid one, and a pool with no free model behaves as `order`). A group routed `free` weighs its members whole, each over its own accounts or keys as its provider routes them. A group routed `weight` (#1910) takes its members, not their keys, by smooth weighted round robin: `planLevel` makes each model member (its keys ordered by its own provider's routing) and each nested group one unit, and `weightedUnits` puts the round's pick (`routes.Round` over `group/<id>`, keyed by the member's spelling, `Group.Weights`, 1 if unset) first and the rest in member order for failover; a unit whose head rests leaves the round, while one whose accounts are only near their limit (Smart's 90% band, not resting) keeps its share, the band ordering only its own accounts (`TestWeightedGroupGivesALowGroupItsShare`; Creat.wu on Discord read the Routing page's "kept for when the others can't" as the member passed over, so the page now tells a member behind another by a group's weights or order as such, `memberSplit` in the routing plugin's `client.js`), and a kept conversation moves the round's charge to the member that kept it (`wrrSeat`, `routes.MoveRound`). A lone key or account is weighed against nobody, but what it has left is still read into the trace (`weighRouted`), so the Routing page shows it as it does for several (#1016); all but used up, it keeps its conversation rather than being said to move (`affine`). A failure rests a candidate (`failure`, `failureOf`). | [`routing.go`](../../internal/gateway/routing.go), [`weighted.go`](../../internal/gateway/weighted.go), [`internal/routes/`](../../internal/routes), [`internal/provider/routing.go`](../../internal/provider/routing.go) |
+| Order of keys and accounts | `Routing`: smart (""), `order`, `rotate`, `usage` (least used, with an `usageHalfLife` of 1h), `pace`, `weight`, `free` (free first) or `fast` (speed first). `weighRouted` reads each candidate into a `routes.Seat` and asks the strategy serving its routing (`routes.Find`, the router strategy point of [the plugin runtime](plugin-runtime.md#gateway-extension-points-p3-router-strategy)) for the order: Smart is `routes.Smart`, the core's; the others are `routes.Builtin`, registered by the built-in plugin `rotation`. With it off a routing no strategy serves is tried in its listed order, and `routingOf` treats it as `order` wherever the core asks about a routing (a turn's mark, `goesRound`, `sinks`, a rotate group's sessions, a group's heads). Smart, `usage` and `pace` weigh a subscription by its allowance (`allowances`), and a key by its own usage windows when magpie reads them (`keyAllowance`: a sub2api key given a 5-hour, day or 7-day limit, or a key bought as a plan, such as Kimi Code's or a GLM Coding Plan's); any other key has none. `free` weighs by what a model costs there and nothing else: only a strategy with `routes.Strategy.ByFree` set (`routes.FreeFirst` is the one) has `weighRouted` fill `routes.Seat.Free`, from `provider.FreeModel` — the vendor's own `catalog.Model.Free` word, else a list price of $0 on every count — so a pool routed any other way never reads a model list for it, and one routed `free` spends the members whose model costs nothing before those that spend the allowance or credits, each tier in its listed order (nothing free is left behind a paid one, and a pool with no free model behaves as `order`). A group routed `free` weighs its members whole, each over its own accounts or keys as its provider routes them. `fast` weighs by speed lately and nothing else: only a strategy with `routes.Strategy.BySpeed` set (`routes.Fastest`) has `weighRouted` fill `routes.Seat.TTFT` and `TPS`, from `speedOf` ([`speed.go`](../../internal/gateway/speed.go)): per `restID` (account or key, and model), a decayed average (half-life 20m) of a streamed answer's ms to its first content and of its output tokens a second as `usage.DecodeOf` counts them, written once as an answer ends (`noteTrySpeed`, beside `servedCandidate`) and read with one map lookup per candidate, never from the usage history; past an hour since its last answer a seat counts as not measured, the records are capped at 4096, and a restart forgets them. `routes.Fastest` ranks the awake by their first token plus 200 tokens at their pace (a seat with no pace at the measured seats' middle pace), keeps seats within a tenth of their band's fastest in listed order, and of a pool's requests with seats both measured and not, gives every fifth to one not measured (round the unmeasured, `probes`, so one is never starved); resting seats go after the awake, used-up ones (`UsedShare`) after those, and rests and failover are as for any routing. The trace's rows carry `lateTtft`/`lateTps` for the Routing page's story ("answered fastest lately — first token in 0.8s"). A non-streamed reply tells nothing (no first token), so a pool that never streams stays in listed order bar the probes. A group routed `weight` (#1910) takes its members, not their keys, by smooth weighted round robin: `planLevel` makes each model member (its keys ordered by its own provider's routing) and each nested group one unit, and `weightedUnits` puts the round's pick (`routes.Round` over `group/<id>`, keyed by the member's spelling, `Group.Weights`, 1 if unset) first and the rest in member order for failover; a unit whose head rests leaves the round, while one whose accounts are only near their limit (Smart's 90% band, not resting) keeps its share, the band ordering only its own accounts (`TestWeightedGroupGivesALowGroupItsShare`; Creat.wu on Discord read the Routing page's "kept for when the others can't" as the member passed over, so the page now tells a member behind another by a group's weights or order as such, `memberSplit` in the routing plugin's `client.js`), and a kept conversation moves the round's charge to the member that kept it (`wrrSeat`, `routes.MoveRound`). A lone key or account is weighed against nobody, but what it has left is still read into the trace (`weighRouted`), so the Routing page shows it as it does for several (#1016); all but used up, it keeps its conversation rather than being said to move (`affine`). A failure rests a candidate (`failure`, `failureOf`). | [`routing.go`](../../internal/gateway/routing.go), [`weighted.go`](../../internal/gateway/weighted.go), [`internal/routes/`](../../internal/routes), [`internal/provider/routing.go`](../../internal/provider/routing.go) |
| Limits | Each key or account (a candidate's `who()`) has at most `MaxConcurrency` requests out at once (its own `AccountConcurrency` over it); one more waits in its lane's queue, bounded by `QueueLimit` and `QueueWait` and turned away with a 429 past either (`lanes.take`). `MaxRPM` (coeo91 on Discord: OpenRouter's free models take 20 a minute) bounds how many requests it sends the vendor in any rolling 60 seconds (`rpms`), counting each for a second more (`rpmSlack`), since the vendor counts a request when it arrives and the first on a new connection arrives later by its dial: one more waits for room in the minute, in the order they came, after it has its slot and right before it is sent. A wait that would be longer than 2 minutes (or the provider's `QueueWait`, when shorter: `rpmWait`) is turned away at once with a 429 whose `Retry-After` is when the minute has room; nothing rests for it. An agent gone while it waits gives its place back. Every request that reaches the vendor counts: each try of the candidate loop (a retry, the next key, a fallback) waits in `serve`, and the requests a try makes on the way and the gateway's own asks (count_tokens, System One's decisions, embeddings, drawing, Kimi's search) are counted by `rpmTransport` on `s.client` (`metered`, `paidFor`). count_tokens never waits: a candidate with no room is skipped, and the local estimate answers. A full lane or a minute with no room gives way to a ready key or account of the provider or the group (`laneMate`, `ready`); a fallback is never asked for want of either. A plugin's account sends through its own transport, so only its tries are counted, not a plugin's own extra requests. Set in the provider editor (Concurrency, Requests per minute, Queue size, Queue wait) or with `magpie provider account-concurrency`, `queue` and `rpm`; a subscription's are kept with its picks, and every account of it keeps them, those on beside the first as well (`withPicks`, called by `All` and `AlsoOn`; #1899: a second WorkBuddy AI account had none, and queued 62s with the wait set to 1s). | [`concurrency.go`](../../internal/gateway/concurrency.go), [`rpm.go`](../../internal/gateway/rpm.go), [`internal/provider/concurrency.go`](../../internal/provider/concurrency.go) |
| Sink | With `Sink` on, a key or account that answers 429 while it still has quota goes to the back, behind every one not rate limited since. Held in memory until restart, or until its rest is lifted by hand (Try now, `Unrest`/`UnrestAll` call `unsink`), which puts it back in its place (`TestTryNowPutsASunkAccountBack`). A group in a group sinks by its own Sink, within its own order (`sinkOwn`, #2255), and the group asked for then sinks the whole by its own; `TestNestedGroupSinksItsOwnOrder`. | [`sink.go`](../../internal/gateway/sink.go) |
| Image generation | `/v1/images/generations` and `/v1/images/edits` are translated to the selected model's image API. OpenRouter image-only models use its `/images` endpoint and send reference images as `input_references`; OpenRouter models that return images through chat stay on Chat Completions. Volcengine Ark (`isArk`: an `ark.*.volces.com` base, the preset's plans or one added by hand) is asked at `/images/generations` only, as its doc (https://www.volcengine.com/docs/82379/1541523) takes it (`drawArk`): `watermark: false`, since it marks every image otherwise; reference images as JSON `image` (a URL or data URL, an array for several), never a multipart edit; no `n`, so one request per image (`eachDrawing`); no ratio as `size`, so a ratio its doc lists is sent as that ratio's 2K size and any other is a 400 that says to put it in the prompt; no mask, no quality (JL on Discord, `TestArkDrawsWithoutWatermarkOncePerImage`). A routing group named for a drawing or a video (`drawingGroup`: `group/<id>`, or a bare id the group goes by that resolved to one of its members) is tried member by member in the order its routing plans for a chat (`drawingTries` over `planGroup`), each member over its own keys or accounts (`drawOnAccounts`, `startVideo`); the next member is asked while one fails as a chat's would move on (`retryable`: its vendor 5xx, out of quota, the model not served), never for a refusal of the request itself, and the call's `Fallback` names those tried first (#2005, tianxia3111: a group of two relays answered the first one's 502). Pause rules and a gateway key's group hold apply as for a chat. A failed drawing rests no member and no key. An account out of quota for the image model (`failure` reads `failQuota`) rests that model on it alone (`restDrawer`, keyed by `restID`), until the reset its refusal names (`resetsAt`: Antigravity's "Your quota will reset after 3h19m30s.") or `quotaRest`, so the next drawing asks the accounts that still have it first (`restLast`), and the account's text models stay in. Antigravity also tells a model's quota used up as a 200 whose stream is the error; `drawCodeAssist` gives such an error the status its words stand for (`saidStatus`: quota or rate 429, credit 402, else 502), as the images API's 200 with no image does, so a spent account hands the drawing to the next one as its 429 does (01huadalang on Discord: a 502 ended the drawing on the spent account; `TestAntigravityDrawingOutOfQuotaInsideTheReplyGoesOn`, `TestAntigravityDrawingQuotaRestsTheImageModel`). `TestImageGroupMovesOnToTheNextMember`, `TestVideoGroupMovesOnToTheNextMember`. An account's own list of models (#474) never bars a ChatGPT account's GPT Image, a Grok subscription's Grok Imagine or a WorkBuddy plan's image models (`listsCantName`): the list is picked from the editor's chat models, which they are never among, so a list kept for some chat models had every drawing refused as "set not to be served by any account" (heyiwei5 on X). A key's list, picked from models that include its image ones, still bars them. `TestCodexDrawsPastAccountModelLists`. A drawing is given Settings → Images → Image generation timeout (`imageTimeout`, 1 to 60 minutes, 5 when unset; neighbads on X: a slow image endpoint needed more) as its context's deadline (`drawTimeout`, `drawWithin`), and so is downloading an image answered as a link (`imageLongest`, never under `ImageLongest`). A drawing's request goes through a transport with no `ResponseHeaderTimeout` (`drawsApart`), since an images API sends its headers only once it has drawn, and every other request keeps `upstreamHeaderWait` (10 minutes). A video's submit, status and download calls keep 5 minutes each (`videoTimeout`). The `magpie mcp image` process, which may not read these settings (WSL), waits on a call up to `callLongest` (two hours and a minute), so it never cuts one the gateway still waits on. `TestLongImageTimeoutLetsASlowVendorFinish`, `TestImageTimeoutSettingReachesTheVendor`, `TestOnlyDrawingsWaitPastTheHeaderWait`. | [`draw.go`](../../internal/gateway/draw.go), [`draw_volcengine.go`](../../internal/gateway/draw_volcengine.go), [`video.go`](../../internal/gateway/video.go), [`catalog.go`](../../internal/catalog/catalog.go) |
diff --git a/docs/subsystems/plugin-runtime.md b/docs/subsystems/plugin-runtime.md
index 024805d1d..babf1906e 100644
--- a/docs/subsystems/plugin-runtime.md
+++ b/docs/subsystems/plugin-runtime.md
@@ -189,14 +189,14 @@ The router strategy point says the order a pool of routes is tried in: a
provider's accounts or keys, or the heads of a group's members. The gateway
still lists the pool, and keeps rests, affinity, rules, pins and the walk
over the order itself; a strategy only orders. Smart, the default, stays in
-the core. The six other routings are the built-in plugin `rotation`'s.
+the core. The seven other routings are the built-in plugin `rotation`'s.
| Part | What it does | Source |
| --- | --- | --- |
-| `routes.Strategy` | `Order(pool, seats) []int`: a permutation of the seats' indexes. `pool` is a provider's id or `group/<id>`, for a strategy that keeps a round per pool. `Exact` keeps the order as given; otherwise a provider's API keys that fit the request better go first, each in its order. `ByUse` puts each seat's recent tokens in the trace. | [`routes.go`](../../internal/routes/routes.go) |
-| `routes.Seat` | What a strategy weighs, read by `weighRouted` under the routing state's lock: its rest key, whether it is an account, whether its allowance is known or it learns it as it answers, the share used, when its windows renew, its weekly pace, its key's weight, whether it rests now and its tokens of the last hours. | [`routes.go`](../../internal/routes/routes.go), [`routing.go`](../../internal/gateway/routing.go) |
+| `routes.Strategy` | `Order(pool, seats) []int`: a permutation of the seats' indexes. `pool` is a provider's id or `group/<id>`, for a strategy that keeps a round per pool. `Exact` keeps the order as given; otherwise a provider's API keys that fit the request better go first, each in its order. `ByUse` puts each seat's recent tokens in the trace; `ByFree` has the gateway fill `Seat.Free`, and `BySpeed` `Seat.TTFT` and `TPS` (its speed lately, gateway `speed.go`), each only for a strategy that asks. | [`routes.go`](../../internal/routes/routes.go) |
+| `routes.Seat` | What a strategy weighs, read by `weighRouted` under the routing state's lock: its rest key, whether it is an account, whether its allowance is known or it learns it as it answers, the share used, when its windows renew, its weekly pace, its key's weight, whether it rests now, its tokens of the last hours, and, when the strategy asks, whether its model is free and its ms to the first token and tokens a second lately. | [`routes.go`](../../internal/routes/routes.go), [`routing.go`](../../internal/gateway/routing.go) |
| `Register`, `Find`, `Apply` | `Register(routing, s)` serves a routing until its remove; registering again stacks, and the last one serves. `Find("")` is `Smart` always. `Apply` fails open: a strategy that panics, or answers what isn't a permutation, leaves the pool in its listed order, logged. Every route is still tried. | [`routes.go`](../../internal/routes/routes.go) |
-| Strategies | `Smart` (core), and `Ordered`, `Rotate`, `LeastUsed`, `Pace` and `Weighted` in `routes.Builtin`. `Rotate` keeps a turn per pool and `Weighted` a smooth weighted round (`Round`, `MoveRound`), both moved out of the gateway; `Forget` clears them. | [`strategies.go`](../../internal/routes/strategies.go), [`round.go`](../../internal/routes/round.go) |
+| Strategies | `Smart` (core), and `Ordered`, `Rotate`, `LeastUsed`, `Pace`, `Weighted`, `FreeFirst` and `Fastest` in `routes.Builtin`. `Rotate` keeps a turn per pool and `Weighted` a smooth weighted round (`Round`, `MoveRound`), both moved out of the gateway, and `Fastest` a probe count per pool, so a seat not measured lately gets one request in `ProbeEvery`; `Forget` clears them. | [`strategies.go`](../../internal/routes/strategies.go), [`round.go`](../../internal/routes/round.go) |
| Off | A routing no strategy serves is its listed order. `routingOf` in the gateway reads it as `order` wherever the core asks about a routing (a turn's mark, `sinks`, a rotate group's sessions, a group's heads), so nothing of the routing is left half on. | [`routing.go`](../../internal/gateway/routing.go) |
| `plugins/rotation` | Requires `providers` and `gateway`, `Essential`. Apply registers every `routes.Builtin` and removes them when turned off. `InUse` counts the providers and groups providers.json routes by one of them (`provider.Routings`, the file alone, not the catalog); an unreadable providers.json is an error, so it stays on. The gateway's `KeepFresh` watches providers.json too, so a routing set from the CLI or the TUI turns it on within 2s. Off, the window's routing pickers (the provider editor, the Routing page's pools and groups) offer Smart alone: their options are marked `data-feature="rotation"` (`routingSegs`). | [`rotation.go`](../../plugins/rotation/rotation.go), [`provider.go`](../../internal/provider/provider.go), [`app.js`](../../internal/gui/assets/app.js) |
diff --git a/internal/cli/groups_cli.go b/internal/cli/groups_cli.go
index e6e5b0201..b2863cf8c 100644
--- a/internal/cli/groups_cli.go
+++ b/internal/cli/groups_cli.go
@@ -118,6 +118,7 @@ const groupUsage = `usage:
magpie group set fast models+=gcloud/gemini-3.8-flash:medium
magpie group add my-free-pool models='openrouter/*:free,opencode-zen/*-free' routing=order
magpie group set opus-anywhere routing=weight weight=claude/claude-opus-5-5=3
+ magpie group set deepseek-anywhere routing=fast # the provider that answered it fastest lately first
magpie group pick opus-anywhere copilot/claude-opus-5.5
magpie claude group/opus-anywhere`
@@ -134,6 +135,7 @@ var routingNames = []struct {
{provider.Pace, "pace", []string{"weekly-pace"}},
{provider.Weighted, "weight", []string{"weighted", "by-weight", "weighted-round-robin", "wrr"}},
{provider.FreeFirst, "free", []string{"free-first", "freest", "no-cost"}},
+ {provider.Fastest, "fast", []string{"fastest", "speed", "speed-first"}},
{provider.Manual, "manual", []string{"pick", "picked", "pinned"}},
}
diff --git a/internal/cli/groups_cli_test.go b/internal/cli/groups_cli_test.go
index 1c974b571..c01d5bd8a 100644
--- a/internal/cli/groups_cli_test.go
+++ b/internal/cli/groups_cli_test.go
@@ -76,12 +76,12 @@ func storedGroups(t *testing.T) []map[string]any {
}
func TestParseRoutingStays(t *testing.T) {
- for in, want := range map[string]string{"smart": "", "": "", "order": "order", "In-Order": "order", "rotate": "rotate", "usage": "usage", "least-used": "usage"} {
+ for in, want := range map[string]string{"smart": "", "": "", "order": "order", "In-Order": "order", "rotate": "rotate", "usage": "usage", "least-used": "usage", "fast": "fast", "Fastest": "fast", "speed": "fast", "speed-first": "fast"} {
if got, err := parseRouting(in); err != nil || got != want {
t.Errorf("routing %q: %q %v, want %q", in, got, err, want)
}
}
- if _, err := parseRouting("fastest"); err == nil || !strings.Contains(err.Error(), "smart, order, rotate, usage") || !strings.Contains(err.Error(), "<plugin>/<name>") {
+ if _, err := parseRouting("cheapest"); err == nil || !strings.Contains(err.Error(), "smart, order, rotate, usage") || !strings.Contains(err.Error(), "fast") || !strings.Contains(err.Error(), "<plugin>/<name>") {
t.Errorf("unknown routing: %v", err)
}
// a gateway plugin's routing is taken as it is written, and named so
@@ -144,7 +144,7 @@ func TestApplyGroupPairs(t *testing.T) {
if err := applyGroupPairs(&g, []string{"listed=YES"}, resolve, false); err != nil || g.Unlists {
t.Fatalf("listed=yes: %v %v", g.Unlists, err)
}
- for _, bad := range [][]string{{"models"}, {"colour=red"}, {"id=x"}, {"models=nope"}, {"routing=fast"}, {"models-=a/m"}, {"context=lots"}, {"listed=maybe"}} {
+ for _, bad := range [][]string{{"models"}, {"colour=red"}, {"id=x"}, {"models=nope"}, {"routing=cheapest"}, {"models-=a/m"}, {"context=lots"}, {"listed=maybe"}} {
h := g
if err := applyGroupPairs(&h, bad, resolve, false); err == nil {
t.Errorf("%v: no error", bad)
diff --git a/internal/cli/main.go b/internal/cli/main.go
index c5a18a686..0df9d0c04 100644
--- a/internal/cli/main.go
+++ b/internal/cli/main.go
@@ -85,7 +85,7 @@ const usage = `magpie — one place to pick every agent's model
or only the models ticked for it, a new one off until it is ticked
magpie search [add <api> <key>|rm <api>] Tavily, Brave, Exa, Firecrawl, You.com, DeepSeek or SearXNG for web search when no provider can search
magpie groups routing groups: several models agents pick as one, group/<id>
- magpie group add <name> models=<m1>,<m2> [routing=smart|order|rotate|usage|pace] [stays=auto|session|turn|off]
+ magpie group add <name> models=<m1>,<m2> [routing=smart|order|rotate|usage|pace|weight|free|fast] [stays=auto|session|turn|off]
magpie group <id> | set <id> k=v… | rm <id> show, change or remove one (magpie group help for more)
magpie accounts [agent] [--json] every subscription magpie knows, with each one's allowance used and when it resets
magpie accounts add <agent> sign in to one more Claude, ChatGPT or Google (Gemini CLI, Antigravity) subscription
diff --git a/internal/cli/provider_routing_cli_test.go b/internal/cli/provider_routing_cli_test.go
new file mode 100644
index 000000000..48f1a5c45
--- /dev/null
+++ b/internal/cli/provider_routing_cli_test.go
@@ -0,0 +1,25 @@
+package cli
+
+import (
+ "testing"
+
+ "github.com/yetone/magpie/internal/provider"
+)
+
+// routing= on a provider spreads requests over its keys or accounts as
+// the editor's Routing does: fast (fastest, speed) is Speed first; manual
+// is a group's alone, and a name magpie doesn't know is refused.
+func TestProviderRoutingPair(t *testing.T) {
+ var p provider.Provider
+ for in, want := range map[string]string{"fast": provider.Fastest, "fastest": provider.Fastest, "Speed": provider.Fastest, "order": provider.Ordered, "smart": ""} {
+ if err := applyPairs(&p, []string{"routing=" + in}); err != nil || p.Routing != want {
+ t.Errorf("routing=%s: %q %v, want %q", in, p.Routing, err, want)
+ }
+ }
+ p.Routing = provider.Fastest
+ for _, bad := range []string{"manual", "cheapest"} {
+ if err := applyPairs(&p, []string{"routing=" + bad}); err == nil || p.Routing != provider.Fastest {
+ t.Errorf("routing=%s taken: %q", bad, p.Routing)
+ }
+ }
+}
diff --git a/internal/cli/providers_cli.go b/internal/cli/providers_cli.go
index e8fb34a2c..4ae01511d 100644
--- a/internal/cli/providers_cli.go
+++ b/internal/cli/providers_cli.go
@@ -42,6 +42,8 @@ const providerUsage = `usage:
compaction=native asks the provider Codex's own compaction and hands
Codex its sealed item as it came (a Responses API, a ChatGPT account,
Copilot), compaction=magpie, the default, has magpie summarise
+ routing=smart|order|rotate|usage|pace|weight|free|fast spreads requests
+ over its keys or accounts (fast: the one that answered fastest lately first);
additional-tools=on sends a tool Claude Code's tool search loads to its
Responses API where it was loaded (an additional_tools item, which
keeps the cache); additional-tools=default puts it with the other tools
@@ -1152,6 +1154,18 @@ func applyPairs(p *provider.Provider, pairs []string) error {
}
case "family", "tag":
p.Family = strings.TrimSpace(v)
+ case "routing", "strategy":
+ // how requests spread over its keys or accounts, as the
+ // provider editor's Routing says: a group's names, but a
+ // provider has no member to pick by hand
+ r, err := parseRouting(v)
+ if err != nil {
+ return err
+ }
+ if r == provider.Manual {
+ return fmt.Errorf("routing=manual is a group's: a provider's keys or accounts are routed smart, order, rotate, usage, pace, weight, free or fast")
+ }
+ p.Routing = r
case "icon":
// a picture on disk is kept by magpie; anything else is one of
// the built-in icons' names
diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.go
index d05131a11..bdbce20e3 100644
--- a/internal/gateway/gateway.go
+++ b/internal/gateway/gateway.go
@@ -3692,6 +3692,8 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro
}
} else if call.Status < 400 {
servedCandidate(c, call.Usage.Input+call.Usage.Output+call.Usage.CacheRead+call.Usage.CacheWrite)
+ // how fast it answered, for Speed first (speed.go)
+ noteTrySpeed(c, try, call.Usage.Output, call.Usage.Reasoning, ended)
// a compaction a rule sent to a model of its own leaves the
// conversation, and the turn's size, where they were
if hit == nil || !hit.Compact {
diff --git a/internal/gateway/routing.go b/internal/gateway/routing.go
index ac58fb611..7aee70698 100644
--- a/internal/gateway/routing.go
+++ b/internal/gateway/routing.go
@@ -1461,6 +1461,11 @@ type weighing struct {
// free is each seat's model costing nothing there, by rest key: filled
// only when the routing weighs by it (routes.Strategy.ByFree)
free map[string]bool
+
+ // speed is each seat's speed lately, by restID (a group's heads can
+ // be two models on one key): filled only when the routing weighs by
+ // it (routes.Strategy.BySpeed)
+ speed map[string][2]float64
}
type allowanceKey struct {
@@ -1603,6 +1608,15 @@ func weighRouted(p provider.Provider, cs []candidate, model string, from provide
seats[i].Free, wg.free[c.rest] = free, free
}
}
+ if st.BySpeed {
+ // a map lookup each (speed.go): what was measured as answers
+ // ended, never the usage history
+ wg.speed = map[string][2]float64{}
+ for i, c := range cs {
+ seats[i].TTFT, seats[i].TPS = speedOf(c, now)
+ wg.speed[c.restID()] = [2]float64{seats[i].TTFT, seats[i].TPS}
+ }
+ }
order := routes.Apply(st, p.ID, seats)
out := make([]candidate, 0, len(cs))
for _, i := range order {
diff --git a/internal/gateway/speed.go b/internal/gateway/speed.go
new file mode 100644
index 000000000..c1f1b1fae
--- /dev/null
+++ b/internal/gateway/speed.go
@@ -0,0 +1,116 @@
+package gateway
+
+import (
+ "math"
+ "sync"
+ "time"
+
+ "github.com/yetone/magpie/internal/usage"
+)
+
+// Speed lately, for Speed first (routes.Fastest): of each account or key
+// and model (restID), a decayed average of its streamed answers' time to
+// first token and of how fast they wrote. Kept in memory, a few numbers
+// each, written once as an answer ends (noteSpeed) and read with a map
+// lookup per candidate as a pool routed by speed is weighed: the usage
+// history is never read for it. A restart forgets them, and each is
+// measured again as Fastest's probes give it a turn.
+
+// speedHalfLife: a sample counts half after this, against those since.
+// speedStale: this long after its last answer a seat counts as not
+// measured lately, and Fastest gives it a turn now and then to be
+// measured again, so one that was slow once isn't kept last for good.
+const (
+ speedHalfLife = 20 * time.Minute
+ speedStale = time.Hour
+)
+
+// speedMost bounds the records: past it, those not measured lately go.
+const speedMost = 4096
+
+type speedRec struct {
+ ttft, ttftW float64 // the average ms to the first token, and its samples' weight
+ tps, tpsW float64 // the average output tokens a second, and its samples' weight
+ at, tpsAt time.Time // the last sample of each
+}
+
+var speeds = struct {
+ sync.Mutex
+ m map[string]speedRec // candidate restID → its speed lately
+}{m: map[string]speedRec{}}
+
+func decayed(w float64, since time.Duration) float64 {
+ if since <= 0 {
+ return w
+ }
+ return w * math.Exp2(-since.Seconds()/speedHalfLife.Seconds())
+}
+
+// fold is avg of weight w, decayed by since, with x added at weight 1.
+func fold(avg, w float64, since time.Duration, x float64) (float64, float64) {
+ w = decayed(w, since)
+ return (avg*w + x) / (w + 1), w + 1
+}
+
+// noteSpeed records an answer of c's: ttft ms from its try's start to its
+// first content, and, when its pace was told, tokens written in w ms
+// after it (usage.DecodeOf). A reply that wasn't streamed has no ttft and
+// tells nothing.
+func noteSpeed(c candidate, ttft int64, tokens int, w int64, now time.Time) {
+ if ttft <= 0 {
+ return
+ }
+ id := c.restID()
+ speeds.Lock()
+ defer speeds.Unlock()
+ r := speeds.m[id]
+ r.ttft, r.ttftW = fold(r.ttft, r.ttftW, now.Sub(r.at), float64(ttft))
+ r.at = now
+ if tokens > 0 && w > 0 {
+ r.tps, r.tpsW = fold(r.tps, r.tpsW, now.Sub(r.tpsAt), float64(tokens)*1000/float64(w))
+ r.tpsAt = now
+ }
+ if _, had := speeds.m[id]; !had && len(speeds.m) >= speedMost {
+ for k, o := range speeds.m {
+ if now.Sub(o.at) > speedStale {
+ delete(speeds.m, k)
+ }
+ }
+ }
+ speeds.m[id] = r
+}
+
+// noteTrySpeed is noteSpeed of a try that answered, its pace as
+// usage.DecodeOf counts it for the Requests and Usage pages.
+func noteTrySpeed(c candidate, t Try, out, reasoning int, now time.Time) {
+ tokens, w := 0, int64(0)
+ if !t.ReasoningHidden || t.FirstText > 0 {
+ tokens, w = usage.DecodeOf(out, reasoning, t.Millis, t.TTFT, t.FirstText, t.Flow, t.FirstReasoning)
+ }
+ noteSpeed(c, t.TTFT, tokens, w, now)
+}
+
+// speedOf is c's speed lately: its average ms to the first token and
+// tokens a second, 0 for what wasn't measured lately.
+func speedOf(c candidate, now time.Time) (ttft, tps float64) {
+ speeds.Lock()
+ r, ok := speeds.m[c.restID()]
+ speeds.Unlock()
+ if !ok {
+ return 0, 0
+ }
+ if now.Sub(r.at) > speedStale {
+ return 0, 0
+ }
+ if !r.tpsAt.IsZero() && now.Sub(r.tpsAt) <= speedStale {
+ tps = r.tps
+ }
+ return r.ttft, tps
+}
+
+// forgetSpeeds drops every speed: a test starts anew.
+func forgetSpeeds() {
+ speeds.Lock()
+ speeds.m = map[string]speedRec{}
+ speeds.Unlock()
+}
diff --git a/internal/gateway/speed_test.go b/internal/gateway/speed_test.go
new file mode 100644
index 000000000..2876f4ca8
--- /dev/null
+++ b/internal/gateway/speed_test.go
@@ -0,0 +1,129 @@
+package gateway
+
+import (
+ "io"
+ "math"
+ "net/http"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/yetone/magpie/internal/provider"
+)
+
+// Speed lately is a decayed average: a sample counts half after
+// speedHalfLife against those since, and a seat not measured for
+// speedStale counts as not measured at all, so Speed first gives it a
+// probe's turn instead of holding an old number against it.
+func TestSpeedLatelyDecaysAndGoesStale(t *testing.T) {
+ forgetSpeeds()
+ t.Cleanup(forgetSpeeds)
+ c := candidate{p: provider.Provider{ID: "v"}, rest: "v", model: "m"}
+ t0 := time.Date(2026, 10, 26, 12, 0, 0, 0, time.UTC)
+ noteSpeed(c, 1000, 0, 0, t0)
+ if ttft, tps := speedOf(c, t0); ttft != 1000 || tps != 0 {
+ t.Fatalf("one sample without a pace: %v %v", ttft, tps)
+ }
+ // a sample a half-life later counts double the first: (1000*0.5 + 4000)/1.5
+ noteSpeed(c, 4000, 100, 1000, t0.Add(speedHalfLife))
+ ttft, tps := speedOf(c, t0.Add(speedHalfLife))
+ if math.Abs(ttft-3000) > 0.01 || tps != 100 {
+ t.Fatalf("after a half-life: ttft %v tps %v, want 3000 and 100", ttft, tps)
+ }
+ // a reply that wasn't streamed tells nothing
+ noteSpeed(c, 0, 50, 1000, t0.Add(speedHalfLife))
+ if again, _ := speedOf(c, t0.Add(speedHalfLife)); again != ttft {
+ t.Fatalf("an unstreamed reply moved the average: %v", again)
+ }
+ // stale: not measured lately
+ if ttft, tps := speedOf(c, t0.Add(speedHalfLife+speedStale+time.Second)); ttft != 0 || tps != 0 {
+ t.Fatalf("an hour and more on, still measured: %v %v", ttft, tps)
+ }
+ // another model of the same key is its own
+ if ttft, _ := speedOf(candidate{p: c.p, rest: "v", model: "n"}, t0); ttft != 0 {
+ t.Fatalf("another model shares the speed: %v", ttft)
+ }
+}
+
+// streamed is a vendor that streams one short chat answer, its first
+// content a few ms after the request so its time to first token is
+// measured (in whole ms).
+func streamed(w http.ResponseWriter, r *http.Request) {
+ io.Copy(io.Discard, r.Body)
+ time.Sleep(3 * time.Millisecond)
+ w.Header().Set("Content-Type", "text/event-stream")
+ io.WriteString(w, "data: {\"id\":\"c\",\"object\":\"chat.completion.chunk\",\"model\":\"m\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"hi\"}}]}\n\n")
+ io.WriteString(w, "data: {\"id\":\"c\",\"object\":\"chat.completion.chunk\",\"model\":\"m\",\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":3,\"completion_tokens\":1,\"total_tokens\":4}}\n\n")
+ io.WriteString(w, "data: [DONE]\n\n")
+}
+
+// A group routed Speed first sends the same model to the provider that
+// answered it fastest lately, though it is listed second, and says so in
+// the trace; In order on the same speeds keeps the listed one. The speeds
+// are those the gateway measured on the group's own answers.
+func TestSpeedFirstGroupSendsTheFastestLately(t *testing.T) {
+ fresh(t)
+ forgetSpeeds()
+ t.Cleanup(forgetSpeeds)
+ serveOn(t, "slowv", "sk-slow", []string{"m"}, http.HandlerFunc(streamed))
+ serveOn(t, "fastv", "sk-fast", []string{"m"}, http.HandlerFunc(streamed))
+ if err := provider.SaveGroup(provider.Group{
+ Name: "G", Members: []string{"slowv/m", "fastv/m"}, Routing: provider.Fastest,
+ }); err != nil {
+ t.Fatal(err)
+ }
+ s := New()
+ ask := func() *Route {
+ t.Helper()
+ code, body := postAs(t, s, "", `{"model":"group/g","stream":true,"messages":[{"role":"user","content":"x"}]}`)
+ if code != 200 {
+ t.Fatalf("code %d: %s", code, body)
+ }
+ return s.trace.routes[len(s.trace.routes)-1]
+ }
+ // none measured: listed order, and that answer is measured
+ if first := ask(); first.Order[0].Provider != "slowv" {
+ t.Fatalf("with nothing measured Speed first left the list: %+v", first.Order)
+ }
+ var slowID string
+ speeds.Lock()
+ for id, r := range speeds.m {
+ if strings.HasPrefix(id, "slowv") && r.ttft > 0 {
+ slowID = id
+ }
+ }
+ speeds.Unlock()
+ if slowID == "" {
+ t.Fatal("the streamed answer wasn't measured")
+ }
+ // what each has answered at lately, as many answers would leave it
+ now := time.Now()
+ speeds.Lock()
+ speeds.m[slowID] = speedRec{ttft: 3000, ttftW: 5, at: now, tps: 40, tpsW: 5, tpsAt: now}
+ speeds.m[strings.Replace(slowID, "slowv", "fastv", 1)] = speedRec{ttft: 800, ttftW: 5, at: now, tps: 60, tpsW: 5, tpsAt: now}
+ speeds.Unlock()
+
+ last := ask()
+ if last.Order[0].Provider != "fastv" {
+ t.Fatalf("Speed first went to %q first, want fastv: %+v", last.Order[0].Provider, last.Order)
+ }
+ if last.Order[0].LateTTFT < 700 || last.Order[0].LateTTFT > 800 || last.Order[0].LateTPS == 0 {
+ t.Fatalf("the trace row doesn't tell fastv's speed: %+v", last.Order[0])
+ }
+
+ // In order, on the same speeds: the list, and no row tells a speed
+ if err := provider.SaveGroup(provider.Group{
+ Name: "G", Members: []string{"slowv/m", "fastv/m"}, Routing: provider.Ordered,
+ }); err != nil {
+ t.Fatal(err)
+ }
+ inOrder := ask()
+ if inOrder.Order[0].Provider != "slowv" {
+ t.Fatalf("In order went to %q first: %+v", inOrder.Order[0].Provider, inOrder.Order)
+ }
+ for _, w := range inOrder.Order {
+ if w.LateTTFT != 0 {
+ t.Fatalf("an In order row tells a speed: %+v", w)
+ }
+ }
+}
diff --git a/internal/gateway/trace.go b/internal/gateway/trace.go
index e166e63b3..e97545899 100644
--- a/internal/gateway/trace.go
+++ b/internal/gateway/trace.go
@@ -11,6 +11,7 @@ package gateway
import (
"context"
"io"
+ "math"
"net"
"net/http"
"net/url"
@@ -255,15 +256,17 @@ type Weighed struct {
Amount float64 `json:"amount,omitempty"`
Limit float64 `json:"limit,omitempty"`
Unit string `json:"unit,omitempty"`
- Renews []time.Time `json:"renews,omitempty"` // when those windows renew, the biggest first
- Pace float64 `json:"pace,omitempty"` // remaining allowance per hour until its window resets
- Due *time.Time `json:"due,omitempty"` // when the allowance window used for pace resets
- Tokens float64 `json:"tokens,omitempty"` // least used: tokens it served lately
- Free bool `json:"free,omitempty"` // its model costs nothing there (free first)
- Turn bool `json:"turn,omitempty"` // in turn: it was this one's turn
- Fit int `json:"fit,omitempty"` // keyFit
- Speaks provider.Protocol `json:"speaks,omitempty"` // a key made for one protocol only
- Rest *Rest `json:"rest,omitempty"` // resting after a failure, when the request came
+ Renews []time.Time `json:"renews,omitempty"` // when those windows renew, the biggest first
+ Pace float64 `json:"pace,omitempty"` // remaining allowance per hour until its window resets
+ Due *time.Time `json:"due,omitempty"` // when the allowance window used for pace resets
+ Tokens float64 `json:"tokens,omitempty"` // least used: tokens it served lately
+ Free bool `json:"free,omitempty"` // its model costs nothing there (free first)
+ LateTTFT float64 `json:"lateTtft,omitempty"` // speed first: its ms to the first token lately, 0 if not measured (speed.go)
+ LateTPS float64 `json:"lateTps,omitempty"` // speed first: its tokens a second lately, 0 if not told
+ Turn bool `json:"turn,omitempty"` // in turn: it was this one's turn
+ Fit int `json:"fit,omitempty"` // keyFit
+ Speaks provider.Protocol `json:"speaks,omitempty"` // a key made for one protocol only
+ Rest *Rest `json:"rest,omitempty"` // resting after a failure, when the request came
Unlisted bool `json:"unlisted,omitempty"`
// DueBy: "reset" when Due is when an auto-used Codex reset about to
// run out starts the windows again, sooner than they renew (#717)
@@ -469,6 +472,9 @@ func weighed(c candidate, p provider.Provider, wg weighing, fallback bool, from
if wg.free != nil {
w.Free = wg.free[c.rest]
}
+ if sp, ok := wg.speed[c.restID()]; ok {
+ w.LateTTFT, w.LateTPS = math.Round(sp[0]), math.Round(sp[1]*10)/10
+ }
if sinks(p.Sink, p.Routing) {
markSunk(&w, c)
}
diff --git a/internal/gui/assets/app.js b/internal/gui/assets/app.js
index 39f999fb0..ad1f29329 100644
--- a/internal/gui/assets/app.js
+++ b/internal/gui/assets/app.js
@@ -11610,11 +11610,12 @@ const ROUTINGS = [
["pace", "Weekly pace", "Each request goes to the subscription with the most of its week left per hour until it renews — the one with the most to lose at its reset — so less of each week is lost at its reset; one at 90% or more waits until the others can't answer. A key goes by the tokens it served in the last hours."],
["weight", "By weight", "Requests spread over the keys by the weight set beside each: a key weighing 3 takes three requests for every one a key weighing 1 takes, evenly over a few requests. One that fails is passed over while it rests, and the others share its requests; a conversation stays with its key as Stays says."],
["free", "Free first", "Requests go to what costs nothing first: the models their vendor serves at no cost to this subscription's allowance or credits, in their listed order, and only when those can't answer to one that spends them. Nothing that cost nothing is left unused."],
+ ["fast", "Speed first", "Requests go to the one that has answered this model fastest lately, by its time to the first token and how fast it writes; those within a tenth of each other keep their listed order. One not measured lately gets one request in five, so it is measured again, and one that fails is passed over while it rests."],
];
// the routings besides Smart are the built-in plugin rotation's
// (plugins/rotation): off, their options leave the page, as anything marked
// data-feature does, and Smart is left — one in use keeps it on
-const ROTATION = new Set(["order", "rotate", "usage", "pace", "weight", "free"]);
+const ROTATION = new Set(["order", "rotate", "usage", "pace", "weight", "free", "fast"]);
// jsRoutings: the routings JS gateway plugins serve now ("<plugin>/<name>",
// settings' jsRoutings, plugins/middleware's), and saved, a routing none
// serves now, shown as it is: its pool is tried in its listed order
diff --git a/internal/gui/assets/i18n.js b/internal/gui/assets/i18n.js
index bd901db32..4f2f98ad0 100644
--- a/internal/gui/assets/i18n.js
+++ b/internal/gui/assets/i18n.js
@@ -791,7 +791,7 @@ const I18N = {
"Send each request's traces and metrics to an OpenTelemetry collector, such as Langfuse": "把每个请求的追踪和指标发送到 OpenTelemetry 收集器,比如 Langfuse",
"Sending to a collector": "正在发送到收集器",
"Routing strategies": "路由策略",
- "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight and Free first": "智能以外的路由方式:按顺序、轮流、用量少的优先、重置前用完、按权重和免费优先",
+ "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight, Free first and Speed first": "智能以外的路由方式:按顺序、轮流、用量少的优先、重置前用完、按权重、免费优先和速度优先",
"1 provider or group routed so": "1 个提供商或路由组在用",
"{n} providers and groups routed so": "{n} 个提供商和路由组在用",
"Run the JavaScript gateway plugins: middleware on agents' requests and replies, routings and usage observers": "运行 JavaScript 网关插件:Agent 请求和回复上的中间件、路由方式和用量观察者",
@@ -1600,6 +1600,15 @@ const I18N = {
"Free first: {model} costs nothing on {who}, so it's tried before any that spends the allowance.": "免费优先:{model} 在 {who} 上不花额度,所以先于任何花额度的尝试。",
"Free first: every one that costs nothing is spent or resting, so {model} on {who}, which spends the allowance, is next.": "免费优先:不花钱的都用了或在休息,接着用会花额度的 {who} 上的 {model}。",
"Free first": "免费优先",
+ "Speed first": "速度优先",
+ "Requests go to the one that has answered this model fastest lately, by its time to the first token and how fast it writes; those within a tenth of each other keep their listed order. One not measured lately gets one request in five, so it is measured again, and one that fails is passed over while it rests.": "请求先发给最近回答这个模型最快的那个,按首字时间和输出速度算;相差不到一成的保持列出的顺序。最近没测过的每五个请求分到一个,好重新测一次;失败的在休息期间会被跳过。",
+ "Speed first: the member that answered fastest lately goes first, by its time to the first token and how fast it writes, each of its accounts or keys weighed on its own; those within a tenth keep their listed order, and one not measured lately gets one request in five, so it is measured again.": "速度优先:最近回答最快的成员先上,按首字时间和输出速度算,它的每个账号或密钥各算各的;相差不到一成的保持列出的顺序,最近没测过的每五个请求分到一个,好重新测一次。",
+ "first token in {s}s, {n} tokens a second": "首字 {s} 秒,每秒 {n} 个 token",
+ "first token in {s}s": "首字 {s} 秒",
+ "it answered fastest lately \u2014 {pace}.": "它最近回答最快——{pace}。",
+ "Speed first: {who} answered {model} fastest lately \u2014 {pace}.": "速度优先:最近 {who} 回答 {model} 最快——{pace}。",
+ "Speed first: {who} hasn't answered {model} lately, so it takes this request to be measured \u2014 one request in five goes to one not measured.": "速度优先:{who} 最近没回答过 {model},所以这个请求给它,好测一下速度——每五个请求有一个给没测过的。",
+ "Speed first: none has answered {model} lately, so {who}, first in the list, goes first and its answer measures it.": "速度优先:最近都没回答过 {model},所以按列出的顺序先用 {who},它的回答会测出速度。",
"Requests go to what costs nothing first: the models their vendor serves at no cost to this subscription's allowance or credits, in their listed order, and only when those can't answer to one that spends them. Nothing that cost nothing is left unused.": "先把不花钱的用完:供应商对本订阅的额度或积分不收费的模型排在前面,按它们原来的顺序;这些都不能响应时,才用要花额度的。不花额度的一个都不剩下。",
"Free first: the members whose model its vendor serves at no cost to the subscription's allowance or credits go first, then the rest — each whole, over its own accounts or keys as its provider routes them. The free ones are tried before any that spends, so nothing that costs nothing is left unused.": "免费优先:成员中那个模型不花订阅额度或积分的排在前面,其余在后;每个成员整体尝试,各自按其供应商的路由在自己的账号或 Key 间分配。免费的先于任何花额度的,所以不花钱的一个都不剩下。",
"Requests spread over the keys by the weight set beside each: a key weighing 3 takes three requests for every one a key weighing 1 takes, evenly over a few requests. One that fails is passed over while it rests, and the others share its requests; a conversation stays with its key as Stays says.": "请求按每个密钥旁设置的权重分配:权重 3 的密钥每接 3 个请求,权重 1 的接 1 个,几个请求之内就均匀分开。出错的在休息期间被跳过,由其他密钥按各自权重分担;会话按「会话保持」的设置留在原来的密钥上。",
@@ -4776,7 +4785,7 @@ const I18N = {
"Send each request's traces and metrics to an OpenTelemetry collector, such as Langfuse": "把每個請求的追蹤和指標傳送到 OpenTelemetry 收集器,例如 Langfuse",
"Sending to a collector": "正在傳送到收集器",
"Routing strategies": "路由策略",
- "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight and Free first": "智慧以外的路由方式:按順序、輪流、用量少的優先、重置前用完、按權重和免費優先",
+ "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight, Free first and Speed first": "智慧以外的路由方式:按順序、輪流、用量少的優先、重置前用完、按權重、免費優先和速度優先",
"1 provider or group routed so": "1 個提供商或路由組在用",
"{n} providers and groups routed so": "{n} 個提供商和路由組在用",
"Run the JavaScript gateway plugins: middleware on agents' requests and replies, routings and usage observers": "執行 JavaScript 閘道外掛:Agent 請求和回覆上的中介軟體、路由方式和用量觀察者",
@@ -5574,6 +5583,15 @@ const I18N = {
"Free first: {model} costs nothing on {who}, so it's tried before any that spends the allowance.": "免費優先:{model} 在 {who} 上不花額度,所以先於任何花額度的嘗試。",
"Free first: every one that costs nothing is spent or resting, so {model} on {who}, which spends the allowance, is next.": "免費優先:不花錢的都用了或在休息,接著用會花額度的 {who} 上的 {model}。",
"Free first": "免費優先",
+ "Speed first": "速度優先",
+ "Requests go to the one that has answered this model fastest lately, by its time to the first token and how fast it writes; those within a tenth of each other keep their listed order. One not measured lately gets one request in five, so it is measured again, and one that fails is passed over while it rests.": "請求先送給最近回答這個模型最快的那個,按首字時間和輸出速度算;相差不到一成的保持列出的順序。最近沒測過的每五個請求分到一個,好重新測一次;失敗的在休息期間會被跳過。",
+ "Speed first: the member that answered fastest lately goes first, by its time to the first token and how fast it writes, each of its accounts or keys weighed on its own; those within a tenth keep their listed order, and one not measured lately gets one request in five, so it is measured again.": "速度優先:最近回答最快的成員先上,按首字時間和輸出速度算,它的每個帳號或金鑰各算各的;相差不到一成的保持列出的順序,最近沒測過的每五個請求分到一個,好重新測一次。",
+ "first token in {s}s, {n} tokens a second": "首字 {s} 秒,每秒 {n} 個 token",
+ "first token in {s}s": "首字 {s} 秒",
+ "it answered fastest lately \u2014 {pace}.": "它最近回答最快——{pace}。",
+ "Speed first: {who} answered {model} fastest lately \u2014 {pace}.": "速度優先:最近 {who} 回答 {model} 最快——{pace}。",
+ "Speed first: {who} hasn't answered {model} lately, so it takes this request to be measured \u2014 one request in five goes to one not measured.": "速度優先:{who} 最近沒回答過 {model},所以這個請求給它,好測一下速度——每五個請求有一個給沒測過的。",
+ "Speed first: none has answered {model} lately, so {who}, first in the list, goes first and its answer measures it.": "速度優先:最近都沒回答過 {model},所以按列出的順序先用 {who},它的回答會測出速度。",
"Requests go to what costs nothing first: the models their vendor serves at no cost to this subscription's allowance or credits, in their listed order, and only when those can't answer to one that spends them. Nothing that cost nothing is left unused.": "先把不花錢的用完:供應商對本訂閱的額度或點數不收費的模型排在前面,按它們原本的順序;這些都不能回應時,才用要花額度的。不花額度的一個都不剩下。",
"Free first: the members whose model its vendor serves at no cost to the subscription's allowance or credits go first, then the rest — each whole, over its own accounts or keys as its provider routes them. The free ones are tried before any that spends, so nothing that costs nothing is left unused.": "免費優先:成員中那個模型不花訂閱額度或點數的排在前面,其餘在後;每個成員整體嘗試,各自依其供應商的路由在自己的帳號或 Key 間分配。免費的先於任何花額度的,所以不花錢的一個都不剩下。",
"Requests spread over the keys by the weight set beside each: a key weighing 3 takes three requests for every one a key weighing 1 takes, evenly over a few requests. One that fails is passed over while it rests, and the others share its requests; a conversation stays with its key as Stays says.": "請求按每個金鑰旁設定的權重分配:權重 3 的金鑰每接 3 個請求,權重 1 的接 1 個,幾個請求之內就均勻分開。出錯的在休息期間被跳過,由其他金鑰按各自權重分擔;會話按「會話保持」的設定留在原來的金鑰上。",
@@ -8733,7 +8751,7 @@ const I18N = {
"Send each request's traces and metrics to an OpenTelemetry collector, such as Langfuse": "各リクエストのトレースとメトリクスを Langfuse などの OpenTelemetry コレクターに送ります",
"Sending to a collector": "コレクターに送信中",
"Routing strategies": "ルーティング戦略",
- "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight and Free first": "スマート以外のルーティング:順番、交互、使用量が最少のものを優先、週ペース、重み付け、無料優先",
+ "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight, Free first and Speed first": "スマート以外のルーティング:順番、交互、使用量が最少のものを優先、週ペース、重み付け、無料優先、速度優先",
"1 provider or group routed so": "1 個のプロバイダーまたはグループが使用中",
"{n} providers and groups routed so": "{n} 個のプロバイダーとグループが使用中",
"Run the JavaScript gateway plugins: middleware on agents' requests and replies, routings and usage observers": "JavaScript ゲートウェイプラグインを実行します:エージェントのリクエストと応答のミドルウェア、ルーティング、使用量オブザーバー",
@@ -9525,6 +9543,15 @@ const I18N = {
"Free first: {model} costs nothing on {who}, so it's tried before any that spends the allowance.": "無料優先:{model} は {who} では利用枠を消費しないため、消費するものより先に試します。",
"Free first: every one that costs nothing is spent or resting, so {model} on {who}, which spends the allowance, is next.": "無料優先:無料のものはすべて使用済みか休止中なので、次は利用枠を消費する {who} の {model} です。",
"Free first": "無料優先",
+ "Speed first": "速度優先",
+ "Requests go to the one that has answered this model fastest lately, by its time to the first token and how fast it writes; those within a tenth of each other keep their listed order. One not measured lately gets one request in five, so it is measured again, and one that fails is passed over while it rests.": "このモデルに最近いちばん速く応答したものにリクエストを送ります。最初のトークンまでの時間と出力の速さで比べ、差が 1 割以内のものは並び順のままです。最近測っていないものには 5 件に 1 件を回して測り直し、失敗したものは休止中は飛ばします。",
+ "Speed first: the member that answered fastest lately goes first, by its time to the first token and how fast it writes, each of its accounts or keys weighed on its own; those within a tenth keep their listed order, and one not measured lately gets one request in five, so it is measured again.": "速度優先:最近いちばん速く応答したメンバーが先です。最初のトークンまでの時間と出力の速さで比べ、アカウントやキーはそれぞれ個別に測ります。差が 1 割以内のものは並び順のままで、最近測っていないものには 5 件に 1 件を回して測り直します。",
+ "first token in {s}s, {n} tokens a second": "最初のトークンまで {s} 秒、毎秒 {n} トークン",
+ "first token in {s}s": "最初のトークンまで {s} 秒",
+ "it answered fastest lately \u2014 {pace}.": "最近いちばん速く応答しました({pace})。",
+ "Speed first: {who} answered {model} fastest lately \u2014 {pace}.": "速度優先:最近 {model} にいちばん速く応答したのは {who} です({pace})。",
+ "Speed first: {who} hasn't answered {model} lately, so it takes this request to be measured \u2014 one request in five goes to one not measured.": "速度優先:{who} は最近 {model} に応答していないため、測るためにこのリクエストを受けます。5 件に 1 件は測っていないものに回ります。",
+ "Speed first: none has answered {model} lately, so {who}, first in the list, goes first and its answer measures it.": "速度優先:最近 {model} に応答したものがないため、並び順で最初の {who} が先に使われ、その応答で速さを測ります。",
"Requests go to what costs nothing first: the models their vendor serves at no cost to this subscription's allowance or credits, in their listed order, and only when those can't answer to one that spends them. Nothing that cost nothing is left unused.": "無料のものを先に使い切ります:提供元がこのサブスクリプションの利用枠やクレジットを消費しないモデルを、一覧の順に先に試し、それらが応答できないときだけ消費するものへ進みます。無料のものが残ることはありません。",
"Free first: the members whose model its vendor serves at no cost to the subscription's allowance or credits go first, then the rest — each whole, over its own accounts or keys as its provider routes them. The free ones are tried before any that spends, so nothing that costs nothing is left unused.": "無料優先:提供元がサブスクリプションの利用枠やクレジットを消費しないモデルのメンバーを先に、残りを後に置きます。各メンバーはまとめて試され、それぞれの提供元のルーティングに従って自分のアカウントや Key に配分されます。無料のものは消費するものより先なので、無料のものは残りません。",
"Requests spread over the keys by the weight set beside each: a key weighing 3 takes three requests for every one a key weighing 1 takes, evenly over a few requests. One that fails is passed over while it rests, and the others share its requests; a conversation stays with its key as Stays says.": "各キーの横で設定した重みでリクエストを振り分けます。重み 3 のキーは、重み 1 のキーが 1 件受けるごとに 3 件受け、数件のうちに均等に分かれます。失敗したキーは休止中は飛ばされ、他のキーがそれぞれの重みで分担します。会話は「維持」の設定どおりそのキーにとどまります。",
@@ -12704,7 +12731,7 @@ const I18N = {
"Send each request's traces and metrics to an OpenTelemetry collector, such as Langfuse": "Traces und Metriken jeder Anfrage an einen OpenTelemetry-Collector senden, etwa Langfuse",
"Sending to a collector": "Sendet an einen Collector",
"Routing strategies": "Routing-Strategien",
- "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight and Free first": "Die Routings neben Intelligent: Der Reihe nach, Abwechselnd, Am wenigsten genutzte zuerst, Wochentempo, Nach Gewicht und Kostenlos zuerst",
+ "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight, Free first and Speed first": "Die Routings neben Intelligent: Der Reihe nach, Abwechselnd, Am wenigsten genutzte zuerst, Wochentempo, Nach Gewicht, Kostenlos zuerst und Schnellste zuerst",
"1 provider or group routed so": "1 Anbieter oder Gruppe so geroutet",
"{n} providers and groups routed so": "{n} Anbieter und Gruppen so geroutet",
"Run the JavaScript gateway plugins: middleware on agents' requests and replies, routings and usage observers": "Die JavaScript-Gateway-Plugins ausführen: Middleware auf Anfragen und Antworten der Agenten, Routings und Nutzungsbeobachter",
@@ -13506,6 +13533,15 @@ const I18N = {
"Free first: {model} costs nothing on {who}, so it's tried before any that spends the allowance.": "Kostenlos zuerst: {model} kostet auf {who} nichts und wird daher vor jedem verbrauchenden Versuch probiert.",
"Free first: every one that costs nothing is spent or resting, so {model} on {who}, which spends the allowance, is next.": "Kostenlos zuerst: Alles Kostenlose ist verbraucht oder ruht, daher folgt {model} auf {who}, das Kontingent verbraucht.",
"Free first": "Kostenlos zuerst",
+ "Speed first": "Schnellste zuerst",
+ "Requests go to the one that has answered this model fastest lately, by its time to the first token and how fast it writes; those within a tenth of each other keep their listed order. One not measured lately gets one request in five, so it is measured again, and one that fails is passed over while it rests.": "Anfragen gehen an den, der dieses Modell zuletzt am schnellsten beantwortet hat, nach der Zeit bis zum ersten Token und dem Schreibtempo; wer weniger als ein Zehntel auseinanderliegt, behält die gelistete Reihenfolge. Wer zuletzt nicht gemessen wurde, bekommt eine von fünf Anfragen und wird so neu gemessen, und wer fehlschlägt, wird übersprungen, solange er ruht.",
+ "Speed first: the member that answered fastest lately goes first, by its time to the first token and how fast it writes, each of its accounts or keys weighed on its own; those within a tenth keep their listed order, and one not measured lately gets one request in five, so it is measured again.": "Schnellste zuerst: Das Mitglied, das zuletzt am schnellsten geantwortet hat, kommt zuerst, nach der Zeit bis zum ersten Token und dem Schreibtempo, jedes seiner Konten oder Schlüssel für sich gemessen; wer weniger als ein Zehntel auseinanderliegt, behält die gelistete Reihenfolge, und wer zuletzt nicht gemessen wurde, bekommt eine von fünf Anfragen und wird so neu gemessen.",
+ "first token in {s}s, {n} tokens a second": "erstes Token nach {s} s, {n} Tokens pro Sekunde",
+ "first token in {s}s": "erstes Token nach {s} s",
+ "it answered fastest lately \u2014 {pace}.": "er hat zuletzt am schnellsten geantwortet — {pace}.",
+ "Speed first: {who} answered {model} fastest lately \u2014 {pace}.": "Schnellste zuerst: {who} hat {model} zuletzt am schnellsten beantwortet — {pace}.",
+ "Speed first: {who} hasn't answered {model} lately, so it takes this request to be measured \u2014 one request in five goes to one not measured.": "Schnellste zuerst: {who} hat {model} zuletzt nicht beantwortet und bekommt diese Anfrage, um gemessen zu werden — eine von fünf Anfragen geht an einen nicht Gemessenen.",
+ "Speed first: none has answered {model} lately, so {who}, first in the list, goes first and its answer measures it.": "Schnellste zuerst: Keiner hat {model} zuletzt beantwortet, daher geht {who}, der Erste in der Liste, zuerst, und seine Antwort misst ihn.",
"Requests go to what costs nothing first: the models their vendor serves at no cost to this subscription's allowance or credits, in their listed order, and only when those can't answer to one that spends them. Nothing that cost nothing is left unused.": "Zuerst kommt, was nichts kostet: Modelle, die ihr Anbieter diesem Abonnement ohne Verbrauch von Kontingent oder Guthaben bereitstellt, in ihrer gelisteten Reihenfolge; erst wenn diese nicht antworten können, eines, das verbraucht. Nichts Kostenloses bleibt ungenutzt.",
"Free first: the members whose model its vendor serves at no cost to the subscription's allowance or credits go first, then the rest — each whole, over its own accounts or keys as its provider routes them. The free ones are tried before any that spends, so nothing that costs nothing is left unused.": "Kostenlos zuerst: die Mitglieder, deren Modell ihr Anbieter dem Abonnement ohne Verbrauch von Kontingent oder Guthaben bereitstellt, kommen zuerst, dann die übrigen — jedes als Ganzes, über seine eigenen Konten oder Keys, wie es sein Anbieter routet. Die kostenlosen werden vor jedem verbrauchenden versucht, sodass nichts Kostenloses ungenutzt bleibt.",
"Requests spread over the keys by the weight set beside each: a key weighing 3 takes three requests for every one a key weighing 1 takes, evenly over a few requests. One that fails is passed over while it rests, and the others share its requests; a conversation stays with its key as Stays says.": "Anfragen verteilen sich nach dem Gewicht neben jedem Schlüssel: Einer mit Gewicht 3 bekommt drei Anfragen für jede, die einer mit Gewicht 1 bekommt, schon über wenige Anfragen gleichmäßig. Ein fehlschlagender wird übersprungen, solange er pausiert, und die anderen teilen sich seine Anfragen; ein Gespräch bleibt bei seinem Schlüssel, wie „Bleibt“ es sagt.",
diff --git a/internal/gui/assets/plugins/routing/client.js b/internal/gui/assets/plugins/routing/client.js
index d2267128e..b82996d23 100644
--- a/internal/gui/assets/plugins/routing/client.js
+++ b/internal/gui/assets/plugins/routing/client.js
@@ -493,6 +493,7 @@ function routingPage(part) {
// language is already spoken
weight: ["By weight", "Requests spread over the keys by the weight set beside each: a key weighing 3 takes three requests for every one a key weighing 1 takes, evenly over a few requests. One that fails is passed over while it rests, and the others share its requests; a conversation stays with its key as Stays says."],
free: ["Free first", "Requests go to what costs nothing first: the models their vendor serves at no cost to this subscription's allowance or credits, in their listed order, and only when those can't answer to one that spends them. Nothing that cost nothing is left unused."],
+ fast: ["Speed first", "Requests go to the one that has answered this model fastest lately, by its time to the first token and how fast it writes; those within a tenth of each other keep their listed order. One not measured lately gets one request in five, so it is measured again, and one that fails is passed over while it rests."],
};
// modeOf: a routing's name and what it does, a JS gateway plugin's
// ("<plugin>/<name>", app.js jsRoutings) as it is named, served or not
@@ -680,6 +681,20 @@ function routingPage(part) {
if (f.free) return restedTo(t("it costs nothing there, so it's spent first."))
|| t("Free first: {model} costs nothing on {who}, so it's tried before any that spends the allowance.", { model: f.model, who: w });
return t("Free first: every one that costs nothing is spent or resting, so {model} on {who}, which spends the allowance, is next.", { model: f.model, who: w });
+ case "fast": {
+ // Speed first (routes.Fastest): its time to the first token and
+ // pace lately, as the gateway measured them (Weighed.lateTtft)
+ if (f.lateTtft) {
+ const secs = (f.lateTtft / 1000).toFixed(1);
+ const pace = f.lateTps ? t("first token in {s}s, {n} tokens a second", { s: secs, n: Math.round(f.lateTps) })
+ : t("first token in {s}s", { s: secs });
+ return restedTo(t("it answered fastest lately — {pace}.", { pace }))
+ || t("Speed first: {who} answered {model} fastest lately — {pace}.", { who: w, model: f.model, pace });
+ }
+ if (r.order.some((x) => x !== f && !x.fallback && !x.rest && x.lateTtft))
+ return t("Speed first: {who} hasn't answered {model} lately, so it takes this request to be measured — one request in five goes to one not measured.", { who: w, model: f.model });
+ return t("Speed first: none has answered {model} lately, so {who}, first in the list, goes first and its answer measures it.", { who: w, model: f.model });
+ }
case "usage":
if (f.known) return t("Least used first: {who} has the most of its allowance left — {n} used.", { who: w, n: pct(f.used) });
if (f.kind === "key" && !someKnown) return t("Least used first: {who} served the fewest tokens lately — {n}.", { who: w, n: tokens(f.tokens || 0) });
@@ -4428,7 +4443,7 @@ function routingPage(part) {
// after the requests: a request picked in the list plays on the stage,
// so the list sits right under it
more.append(gsec);
- const ROUTE_OPTS = [["", "Smart"], ["order", "In order"], ["rotate", "In turn"], ["usage", "Least used"], ["pace", "Weekly pace"], ["free", "Free first"]];
+ const ROUTE_OPTS = [["", "Smart"], ["order", "In order"], ["rotate", "In turn"], ["usage", "Least used"], ["pace", "Weekly pace"], ["free", "Free first"], ["fast", "Speed first"]];
// a group may also be routed by hand: every request to the member the
// user picks on its card (provider.Manual, #317) — a provider's keys can't
// and by its members' weights, as a provider's keys are (#1910)
@@ -4474,6 +4489,7 @@ function routingPage(part) {
pace: "Weekly pace: the account with the most of its week left per hour until it renews goes first, so less of each member's week is lost at its reset — an account with five hours and no week (Claude Enterprise) by what its five hours have left per hour until they renew, so almost always first; a key by the tokens magpie sent it lately.",
weight: GROUP_WEIGHT,
free: "Free first: the members whose model its vendor serves at no cost to the subscription's allowance or credits go first, then the rest — each whole, over its own accounts or keys as its provider routes them. The free ones are tried before any that spends, so nothing that costs nothing is left unused.",
+ fast: "Speed first: the member that answered fastest lately goes first, by its time to the first token and how fast it writes, each of its accounts or keys weighed on its own; those within a tenth keep their listed order, and one not measured lately gets one request in five, so it is measured again.",
manual: "Manual: every request goes to the model you pick on the group's card, over its own accounts or keys; the others, and the rules, wait until you pick another — none takes over when it fails.",
};
// a group in the group, and a provider set to route other than Smart, is
diff --git a/internal/gui/tests/builtins.test.cjs b/internal/gui/tests/builtins.test.cjs
index 76dbd25ca..b150c8f16 100644
--- a/internal/gui/tests/builtins.test.cjs
+++ b/internal/gui/tests/builtins.test.cjs
@@ -26,7 +26,7 @@ const PLUGINS = [
{ id: "middleware", name: "JS gateway plugins", sub: "Run the JavaScript gateway plugins: middleware on agents' requests and replies, routings and usage observers", one: "1 gateway plugin on", many: "{n} gateway plugins on" },
{ id: "otel", name: "Observability", sub: "Send each request's traces and metrics to an OpenTelemetry collector, such as Langfuse", one: "Sending to a collector", many: "Sending to a collector" },
{ id: "redact", name: "Privacy masking", sub: "Secrets, personal data and your own words go to vendors as placeholders, and come back as they were", one: "Masking is on", many: "Masking is on" },
- { id: "rotation", name: "Routing strategies", sub: "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight and Free first", one: "1 provider or group routed so", many: "{n} providers and groups routed so" },
+ { id: "rotation", name: "Routing strategies", sub: "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight, Free first and Speed first", one: "1 provider or group routed so", many: "{n} providers and groups routed so" },
{ id: "routing", name: "Routing", sub: "Each request as the gateway routes it, the accounts it weighed and what each answered, and your routing groups", one: "1 group in use", many: "{n} groups in use" },
{ id: "search", name: "Web search", sub: "Models that can't search the web search with a provider that can, or a search API", one: "1 search source set up", many: "{n} search sources set up" },
{ id: "sessions", name: "Sessions", sub: "Your agents' past conversations: read, search, resume and export them", one: "Recording gateway conversations", many: "Recording gateway conversations" },
diff --git a/internal/gui/tests/routing-group-strategy.test.cjs b/internal/gui/tests/routing-group-strategy.test.cjs
index a56485c2d..5f6b65a8e 100644
--- a/internal/gui/tests/routing-group-strategy.test.cjs
+++ b/internal/gui/tests/routing-group-strategy.test.cjs
@@ -26,7 +26,7 @@ const base = { members: ["a/sol", "b/flash", "c/luna"], ready: true, memberInfo:
const words = {
en: { order: "In order", manual: "Manual", weight: "By weight", smart: "Smart", head: "Routing", edit: "Edit",
- names: ["Smart", "In order", "In turn", "Least used", "Weekly pace", "Free first", "By weight", "Manual"],
+ names: ["Smart", "In order", "In turn", "Least used", "Weekly pace", "Free first", "Speed first", "By weight", "Manual"],
tip: "How Auto routes: click to change it", saved: "Auto: Manual" },
zh: { order: "按顺序", manual: "手动", weight: "按权重", smart: "智能", head: "路由", edit: "编辑",
tip: "Auto 的路由方式:点击更改", saved: "Auto:手动" },
@@ -139,7 +139,7 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",
assert.equal(await menu.locator(".pm-head").textContent(), w.head);
const items = menu.locator(".pm-item");
if (w.names) assert.deepEqual(await items.locator(".pm-name").allTextContents(), w.names);
- assert.equal(await items.count(), 8);
+ assert.equal(await items.count(), 9);
assert.deepEqual(await menu.locator('.pm-item[aria-checked="true"] .pm-name').allTextContents(), [w.order]);
assert.equal(await page.locator(".rt-gedit").count(), 0, "opening the menu opens no editor");
// the menu sits inside the window
@@ -228,7 +228,7 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",
assert.equal(await btn.getAttribute("title"), words[lang].tip);
await btn.click();
await page.locator(".proto-menu").waitFor();
- assert.equal(await page.locator(".proto-menu .pm-item").count(), 8);
+ assert.equal(await page.locator(".proto-menu .pm-item").count(), 9);
assert.deepEqual(errors, []);
});
}
diff --git a/internal/gui/tests/routing-save.test.cjs b/internal/gui/tests/routing-save.test.cjs
index 2ea11c858..5a8af4085 100644
--- a/internal/gui/tests/routing-save.test.cjs
+++ b/internal/gui/tests/routing-save.test.cjs
@@ -157,7 +157,7 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",
// on again, every routing is back
await page.evaluate(() => applyFeatures([{ id: "rotation", on: true }]));
const back = await field.locator(".segs").first().evaluate((box) => [...box.querySelectorAll(".opt")].filter((b) => getComputedStyle(b).display !== "none").length);
- assert.equal(back, p.account ? 6 : 7, `${where}: every routing back`);
+ assert.equal(back, p.account ? 7 : 8, `${where}: every routing back`);
assert.deepEqual(errors, [], where);
await page.context().close();
}
diff --git a/internal/gui/tests/routing-speed-first.test.cjs b/internal/gui/tests/routing-speed-first.test.cjs
new file mode 100644
index 000000000..3aaf65096
--- /dev/null
+++ b/internal/gui/tests/routing-speed-first.test.cjs
@@ -0,0 +1,110 @@
+// Run with Node's test runner and Playwright on the module path; see README.md.
+// Speed first (routes.Fastest) on the Routing page: a request's story says
+// why the first went first in the words of its speed lately, as the
+// gateway's trace gives it (Weighed.lateTtft, lateTps): "answered fastest
+// lately — first token in 0.8s, 62 tokens a second". One not measured
+// lately that took a probe's turn says so, and with none measured the
+// listed first says it is measured by its answer. The mode line says what
+// Speed first does. In both engines and every language.
+const assert = require("node:assert/strict");
+const fs = require("node:fs/promises");
+const path = require("node:path");
+const { test } = require("node:test");
+const { chromium, webkit } = require("playwright");
+
+const assets = path.resolve(__dirname, "../assets");
+const now = new Date();
+const ago = (minutes) => new Date(now.getTime() - minutes * 60e3).toISOString();
+const key = (who, more) => ({ id: "relay#" + who, provider: "relay", name: "Relay", who, kind: "key", model: "deepseek-v4", routing: "fast", ...more });
+
+const scenes = {
+ // b answered fastest lately, so it went first though a is listed first
+ measured: { order: [key("relay-b", { lateTtft: 820, lateTps: 61.6 }), key("relay-a", { lateTtft: 2400, lateTps: 40 })] },
+ // a told no pace lately: its first token alone
+ noPace: { order: [key("relay-a", { lateTtft: 1450 }), key("relay-b", { lateTtft: 3000 })] },
+ // c not measured lately took the probe's turn ahead of the measured a
+ probe: { order: [key("relay-c"), key("relay-a", { lateTtft: 900 })] },
+ // none measured: the list's first
+ none: { order: [key("relay-a"), key("relay-b")] },
+};
+
+function serve(lang, scene) {
+ const s = scenes[scene];
+ const route = {
+ id: 1, seq: 1, time: ago(1), agent: "codex", model: "deepseek-v4", provider: "relay", done: true, status: 200, ms: 900, tokens: 1200, order: s.order,
+ tries: [{ id: s.order[0].id, model: "deepseek-v4", start: ago(1), done: true, status: 200, ms: 900 }],
+ };
+ return async (request) => {
+ const url = new URL(request.request().url());
+ const json = (data) => request.fulfill({ json: data });
+ if (url.pathname === "/boot.js") return request.fulfill({ contentType: "text/javascript", body: `window.bootPrefs = {lang:"${lang}",theme:"light",web:true};` });
+ if (url.pathname === "/wails/runtime.js") return request.fulfill({ contentType: "text/javascript", body: "export const Window = {};" });
+ if (url.pathname === "/api/state") return json({ agents: [{ id: "codex", name: "Codex", path: "/test/config.toml", fields: [] }], profiles: [], settings: { lang, theme: "light" } });
+ if (url.pathname === "/api/gateway/trace") {
+ if (url.searchParams.get("wait")) return new Promise(() => {}); // nothing more comes
+ return json({ mine: true, now: now.toISOString(), seq: 1, totals: { requests: 1 }, routes: [route] });
+ }
+ if (url.pathname === "/api/gateway/history") return json({ cut: false, days: [], routes: [] });
+ if (url.pathname === "/api/groups") return json({ groups: [] });
+ if (url.pathname === "/api/providers") return json({ providers: [], excluded: [], gateway: { running: true, window: true } });
+ if (url.pathname.startsWith("/api/")) return json({});
+ const file = path.join(assets, url.pathname === "/" ? "index.html" : url.pathname);
+ const contentType = { ".html": "text/html", ".js": "text/javascript", ".css": "text/css", ".svg": "image/svg+xml", ".png": "image/png" }[path.extname(file)];
+ await request.fulfill({ body: await fs.readFile(file), contentType });
+ };
+}
+
+async function open(t, engine, lang, scene, width = 1100) {
+ const browser = await (engine === "webkit" ? webkit.launch() : chromium.launch({ channel: "chromium" }));
+ t.after(() => browser.close());
+ const page = await browser.newPage({ viewport: { width, height: 900 }, reducedMotion: "reduce" });
+ page.setDefaultTimeout(5000);
+ const errors = [];
+ page.on("pageerror", (error) => errors.push(error.message));
+ await page.route("**/*", serve(lang, scene));
+ await page.goto("http://magpie.test/?view=routing");
+ const req = page.locator(".rt-req").first();
+ await req.waitFor();
+ if (await req.getAttribute("aria-pressed") !== "true") await req.click();
+ await page.locator(".rt-steps li.why").first().waitFor();
+ // what a string says in lang, its {names} filled in; a language without
+ // it would say it in English, which the i18n-keys test catches
+ const say = (key, vars = {}) => page.evaluate(([lang, key, vars]) => {
+ const s = (lang !== "en" && I18N[lang]?.[key]) || key;
+ return s.replace(/\{(\w+)\}/g, (_, k) => vars[k] ?? `{${k}}`);
+ }, [lang, key, vars]);
+ return { page, errors, say, why: async () => (await page.locator(".rt-steps li.why").first().textContent()).trim() };
+}
+
+for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium", "webkit"])) {
+ for (const lang of ["en", "zh", "zh-TW", "ja", "de"]) {
+ test(`${engine} ${lang}: the fastest lately says its first token and pace`, async (t) => {
+ const { page, errors, say, why } = await open(t, engine, lang, "measured");
+ const pace = await say("first token in {s}s, {n} tokens a second", { s: "0.8", n: "62" });
+ assert.equal(await why(), await say("Speed first: {who} answered {model} fastest lately — {pace}.", { who: "relay-b", model: "deepseek-v4", pace }));
+ assert.equal((await page.locator(".rt-mode").textContent()).trim(),
+ await say("Requests go to the one that has answered this model fastest lately, by its time to the first token and how fast it writes; those within a tenth of each other keep their listed order. One not measured lately gets one request in five, so it is measured again, and one that fails is passed over while it rests."));
+ if (lang !== "en") assert.notEqual(await say("Speed first"), "Speed first", `${lang} names Speed first`);
+ assert.deepEqual(errors, []);
+ });
+
+ test(`${engine} ${lang}: no pace lately, its first token alone`, async (t) => {
+ const { errors, say, why } = await open(t, engine, lang, "noPace");
+ const pace = await say("first token in {s}s", { s: "1.4" });
+ assert.equal(await why(), await say("Speed first: {who} answered {model} fastest lately — {pace}.", { who: "relay-a", model: "deepseek-v4", pace }));
+ assert.deepEqual(errors, []);
+ });
+
+ test(`${engine} ${lang}: one not measured lately took its turn to be measured`, async (t) => {
+ const { errors, say, why } = await open(t, engine, lang, "probe");
+ assert.equal(await why(), await say("Speed first: {who} hasn't answered {model} lately, so it takes this request to be measured — one request in five goes to one not measured.", { who: "relay-c", model: "deepseek-v4" }));
+ assert.deepEqual(errors, []);
+ });
+
+ test(`${engine} ${lang}: none measured, the list's first`, async (t) => {
+ const { errors, say, why } = await open(t, engine, lang, "none");
+ assert.equal(await why(), await say("Speed first: none has answered {model} lately, so {who}, first in the list, goes first and its answer measures it.", { who: "relay-a", model: "deepseek-v4" }));
+ assert.deepEqual(errors, []);
+ });
+ }
+}
diff --git a/internal/provider/group.go b/internal/provider/group.go
index 60ebdd06f..8f49601a6 100644
--- a/internal/provider/group.go
+++ b/internal/provider/group.go
@@ -1267,7 +1267,7 @@ func SaveGroup(g Group) error {
if err := cleanDecisionGroup(&g, groupsIn(entries)); err != nil {
return err
}
- if g.Routing != Ordered && g.Routing != Rotate && g.Routing != LeastUsed && g.Routing != Pace && g.Routing != Weighted && g.Routing != FreeFirst && g.Routing != Manual && !PluginRouting(g.Routing) {
+ if g.Routing != Ordered && g.Routing != Rotate && g.Routing != LeastUsed && g.Routing != Pace && g.Routing != Weighted && g.Routing != FreeFirst && g.Routing != Fastest && g.Routing != Manual && !PluginRouting(g.Routing) {
g.Routing = ""
}
if !slices.Contains(Affinities, g.Affinity) {
diff --git a/internal/provider/provider.go b/internal/provider/provider.go
index 995594684..2e1810883 100644
--- a/internal/provider/provider.go
+++ b/internal/provider/provider.go
@@ -1329,7 +1329,7 @@ func normalize(p Provider) Provider {
if (p.Preset == OMLXPreset || p.Preset == MLXServePreset || p.Preset == "ollama" || p.Preset == "ollama-cloud") && p.Decide == "" && p.Chat != "" {
p.Decide = p.Chat
}
- if p.Routing != Ordered && p.Routing != Rotate && p.Routing != LeastUsed && p.Routing != Pace && p.Routing != Weighted && p.Routing != FreeFirst && !PluginRouting(p.Routing) {
+ if p.Routing != Ordered && p.Routing != Rotate && p.Routing != LeastUsed && p.Routing != Pace && p.Routing != Weighted && p.Routing != FreeFirst && p.Routing != Fastest && !PluginRouting(p.Routing) {
p.Routing = ""
}
if !slices.Contains(Affinities, p.Affinity) {
diff --git a/internal/provider/routing.go b/internal/provider/routing.go
index eb547eb56..ef3fc0fcd 100644
--- a/internal/provider/routing.go
+++ b/internal/provider/routing.go
@@ -32,6 +32,10 @@ const (
// serves at no cost to the subscription's allowance or credits, before
// the ones that spend it (routes.FreeFirst).
FreeFirst = "free"
+ // Fastest goes by speed: the one that has answered the model fastest
+ // lately — first token, then how fast it wrote — first; one not
+ // measured lately gets a turn now and then to be (routes.Fastest).
+ Fastest = "fast"
)
// pluginRouting is a routing a plugin serves: "<plugin>/<name>".
diff --git a/internal/routes/fastest.go b/internal/routes/fastest.go
new file mode 100644
index 000000000..852601f3b
--- /dev/null
+++ b/internal/routes/fastest.go
@@ -0,0 +1,148 @@
+package routes
+
+import (
+ "sort"
+ "sync"
+)
+
+// Speed first: of the same model on several providers, or of a
+// provider's keys, the one that has answered fastest lately goes first
+// (a user comparing API channels by hand asked for it, X joeliu96872540).
+//
+// What "fast" is: the gateway keeps, for each account or key and model, a
+// decayed average of how long its streamed answers took to their first
+// token and how fast they wrote after it (gateway speed.go), and hands
+// the strategy those of the last hour or so as Seat.TTFT and Seat.TPS. A
+// seat is ranked by the time it would take to a short reply of
+// RefTokens: its first token, then RefTokens at its pace. Those within a
+// tenth of the fastest of their band count as alike and keep their
+// listed order, so two channels a few milliseconds apart don't trade
+// places on every request.
+//
+// One not measured lately isn't starved: kept behind those measured, it
+// would never answer and never be measured, and one that got slower an
+// hour ago would never be found faster again. So one request in
+// ProbeEvery, of those the pool has with both kinds awake, puts one not
+// measured lately first: the probe's turn goes round them, so a seat
+// that answers without telling its speed (a reply that isn't streamed)
+// doesn't hold the turn from the others. When none is measured, they go
+// in their order, and the first's answer measures it.
+//
+// Speed only orders the awake: one resting after a failure goes behind
+// them, in its listed order, and one all but used up (UsedShare) behind
+// those, as the other routings that read an allowance keep it; the
+// gateway's rests and failover are as for any routing.
+
+// RefTokens is the reply a seat's speed is weighed at: its first token,
+// then this many at its pace. Short, since an agent's turns are mostly
+// tool calls and short answers, where the first token is most of the wait.
+const RefTokens = 200
+
+// ProbeEvery: of a pool's requests with a seat measured and one not, one
+// in this many gives the turn to one not measured lately.
+const ProbeEvery = 5
+
+// alike is the band within which seats count as alike: a tenth slower than
+// the band's fastest.
+const alike = 1.1
+
+type probe struct {
+ n int // requests counted toward the next probe
+ turns int // probes given: whose turn the next is
+}
+
+var probes = struct {
+ sync.Mutex
+ m map[string]probe // pool → its probes
+}{m: map[string]probe{}}
+
+// Fastest orders a pool by speed lately, as above.
+var Fastest = Strategy{BySpeed: true, Order: func(pool string, seats []Seat) []int {
+ var fast, unmeasured, resting, spent []int
+ for i, s := range seats {
+ switch {
+ case s.Resting:
+ resting = append(resting, i)
+ case tier(s.Used) == 2:
+ spent = append(spent, i)
+ case s.TTFT > 0:
+ fast = append(fast, i)
+ default:
+ unmeasured = append(unmeasured, i)
+ }
+ }
+ // a seat that told no pace counts at the middle pace of those that
+ // did, so lacking it neither helps nor hurts; none told, the first
+ // token alone decides
+ var paces []float64
+ for _, i := range fast {
+ if seats[i].TPS > 0 {
+ paces = append(paces, seats[i].TPS)
+ }
+ }
+ mid := 0.0
+ if len(paces) > 0 {
+ sort.Float64s(paces)
+ mid = paces[len(paces)/2]
+ }
+ est := make(map[int]float64, len(fast))
+ for _, i := range fast {
+ est[i] = Estimate(seats[i].TTFT, seats[i].TPS, mid)
+ }
+ sort.SliceStable(fast, func(a, b int) bool { return est[fast[a]] < est[fast[b]] })
+ // bands drawn from the fastest down, then each band in its listed
+ // order: sorting by "within a tenth" alone isn't consistent
+ band := make(map[int]int, len(fast))
+ top := 0.0
+ for k, i := range fast {
+ if k == 0 {
+ top = est[i]
+ } else if band[i] = band[fast[k-1]]; est[i] > alike*top {
+ band[i]++
+ top = est[i]
+ }
+ }
+ sort.SliceStable(fast, func(a, b int) bool {
+ if band[fast[a]] != band[fast[b]] {
+ return band[fast[a]] < band[fast[b]]
+ }
+ return fast[a] < fast[b]
+ })
+ awake := append(fast, unmeasured...)
+ if len(fast) > 0 && len(unmeasured) > 0 {
+ probes.Lock()
+ p := probes.m[pool]
+ p.n++
+ if p.n >= ProbeEvery {
+ k := unmeasured[p.turns%len(unmeasured)]
+ p.n, p.turns = 0, p.turns+1
+ awake = append([]int{k}, deleted(awake, k)...)
+ }
+ probes.m[pool] = p
+ probes.Unlock()
+ }
+ return append(append(awake, resting...), spent...)
+}}
+
+// Estimate is the ms to a reply of RefTokens at ttft and tps; a tps of 0
+// counts as mid, and with mid 0 too the first token alone counts.
+func Estimate(ttft, tps, mid float64) float64 {
+ if tps <= 0 {
+ tps = mid
+ }
+ if tps <= 0 {
+ return ttft
+ }
+ return ttft + RefTokens*1000/tps
+}
+
+// deleted is l without x, in a new slice.
+func deleted(l []int, x int) []int {
+ out := make([]int, 0, len(l))
+ for _, i := range l {
+ if i != x {
+ out = append(out, i)
+ }
+ }
+ return out
+}
diff --git a/internal/routes/fastest_test.go b/internal/routes/fastest_test.go
new file mode 100644
index 000000000..3a06c027c
--- /dev/null
+++ b/internal/routes/fastest_test.go
@@ -0,0 +1,106 @@
+package routes
+
+import (
+ "slices"
+ "testing"
+)
+
+// Speed first: the seat that answered fastest lately goes first, those
+// within a tenth of each other keep their listed order, one resting or
+// used up goes behind the awake whatever its speed, and a seat that told
+// no pace is weighed at the middle pace of those that did.
+func TestFastestOrdersBySpeedLately(t *testing.T) {
+ cases := []struct {
+ name string
+ seats []Seat
+ want []int
+ }{
+ {
+ name: "the fastest first token first",
+ seats: []Seat{{Key: "a", TTFT: 2000}, {Key: "b", TTFT: 600}, {Key: "c", TTFT: 1200}},
+ want: []int{1, 2, 0},
+ },
+ {
+ // a: 500 + 200 tok at 20/s = 10.5s; b: 900 + 200 at 100/s = 2.9s
+ name: "a quick first token that writes slowly loses",
+ seats: []Seat{{Key: "a", TTFT: 500, TPS: 20}, {Key: "b", TTFT: 900, TPS: 100}},
+ want: []int{1, 0},
+ },
+ {
+ // within a tenth: listed order, so they don't trade on every request
+ name: "alike keep their order",
+ seats: []Seat{{Key: "a", TTFT: 1050}, {Key: "b", TTFT: 1000}, {Key: "c", TTFT: 3000}},
+ want: []int{0, 1, 2},
+ },
+ {
+ name: "resting goes behind the awake, used up behind those",
+ seats: []Seat{{Key: "a", TTFT: 100, Resting: true}, {Key: "b", TTFT: 100, Used: 99}, {Key: "c", TTFT: 5000}},
+ want: []int{2, 0, 1},
+ },
+ {
+ // b told no pace: it counts at the middle pace of a, c and d
+ // (100), so 1000+2000 = 3000, behind a's 2500+200 and ahead
+ // of c's 3000+2000; at its first token alone it would lead
+ name: "a seat without a pace counts at the middle pace",
+ seats: []Seat{{Key: "a", TTFT: 2500, TPS: 1000}, {Key: "b", TTFT: 1000},
+ {Key: "c", TTFT: 3000, TPS: 100}, {Key: "d", TTFT: 3000, TPS: 50}},
+ want: []int{0, 1, 2, 3},
+ },
+ {
+ name: "none measured: their order",
+ seats: []Seat{{Key: "a"}, {Key: "b"}, {Key: "c"}},
+ want: []int{0, 1, 2},
+ },
+ }
+ for _, c := range cases {
+ t.Run(c.name, func(t *testing.T) {
+ got := Fastest.Order("order:"+c.name, c.seats)
+ if !slices.Equal(got, c.want) {
+ t.Fatalf("got %v, want %v", got, c.want)
+ }
+ })
+ }
+}
+
+// One not measured lately isn't kept behind the measured for good: one
+// request in ProbeEvery puts it first, so its answer measures it, and the
+// turn goes round those not measured.
+func TestFastestGivesTheUnmeasuredATurn(t *testing.T) {
+ seats := []Seat{{Key: "a", TTFT: 500}, {Key: "b"}, {Key: "c"}}
+ firsts := map[int]int{}
+ var probesAt []int
+ for n := 1; n <= 4*ProbeEvery; n++ {
+ got := Fastest.Order("probe-pool", seats)
+ if len(got) != 3 {
+ t.Fatalf("request %d: %v", n, got)
+ }
+ firsts[got[0]]++
+ if got[0] != 0 {
+ probesAt = append(probesAt, n)
+ }
+ }
+ if want := []int{ProbeEvery, 2 * ProbeEvery, 3 * ProbeEvery, 4 * ProbeEvery}; !slices.Equal(probesAt, want) {
+ t.Fatalf("the unmeasured went first at requests %v, want %v", probesAt, want)
+ }
+ if firsts[1] != 2 || firsts[2] != 2 {
+ t.Fatalf("the probe's turn didn't go round b and c: %v", firsts)
+ }
+ // another pool counts its own requests
+ if got := Fastest.Order("probe-other", seats); got[0] != 0 {
+ t.Fatalf("a fresh pool probed on its first request: %v", got)
+ }
+ // a resting seat never gets a probe's turn
+ rest := []Seat{{Key: "a", TTFT: 500}, {Key: "b", Resting: true}}
+ for n := 0; n < 2*ProbeEvery; n++ {
+ if got := Fastest.Order("probe-rest", rest); got[0] != 0 {
+ t.Fatalf("request %d put the resting seat first: %v", n, got)
+ }
+ }
+}
+
+func TestFastestIsBuiltinFast(t *testing.T) {
+ st, ok := Builtin["fast"]
+ if !ok || !st.BySpeed || st.Exact {
+ t.Fatalf("fast: %+v %v", st, ok)
+ }
+}
diff --git a/internal/routes/round.go b/internal/routes/round.go
index b92eb41eb..b7d700bd4 100644
--- a/internal/routes/round.go
+++ b/internal/routes/round.go
@@ -84,13 +84,16 @@ func MoveRound(pool, picked, kept string) {
func Forget(pools ...string) {
rounds.Lock()
turns.Lock()
+ probes.Lock()
defer rounds.Unlock()
defer turns.Unlock()
+ defer probes.Unlock()
if len(pools) == 0 {
- rounds.m, turns.m = map[string]*round{}, map[string]int{}
+ rounds.m, turns.m, probes.m = map[string]*round{}, map[string]int{}, map[string]probe{}
}
for _, p := range pools {
delete(rounds.m, p)
delete(turns.m, p)
+ delete(probes.m, p)
}
}
diff --git a/internal/routes/routes.go b/internal/routes/routes.go
index 9cc414e1c..f3d9fae00 100644
--- a/internal/routes/routes.go
+++ b/internal/routes/routes.go
@@ -5,7 +5,7 @@
// failover itself; a Strategy only says the order.
//
// Smart, the default, is the core's. The others a provider or group can be
-// routed by (order, rotate, usage, pace, weight) are here for a plugin to
+// routed by (order, rotate, usage, pace, weight, free, fast) are here for a plugin to
// register: plugins/rotation does, and off, a pool routed by one of them is
// tried in its listed order, as an unknown routing always was.
package routes
@@ -50,6 +50,12 @@ type Seat struct {
// credits, and is let spend them. All but used up, it goes before
// those used up with nothing to go on with.
Credits bool
+ // TTFT is how long it took lately to the first of its reply, in ms,
+ // for the model this seat would be asked for: a decayed average of its
+ // recent streamed answers, 0 when it hasn't answered one lately.
+ // TPS is how fast it wrote them, output tokens a second, 0 when not
+ // told (a reply let go in a burst tells none). Fastest goes by them.
+ TTFT, TPS float64
}
// Strategy orders a pool.
@@ -69,6 +75,9 @@ type Strategy struct {
// model costs nothing there, and the routing trace then shows it.
// Only when set: working it out reads each provider's model list.
ByFree bool
+ // BySpeed: it weighs by TTFT and TPS, so the gateway reads each seat's
+ // recent speed, and the routing trace then shows it.
+ BySpeed bool
}
var reg = struct {
diff --git a/internal/routes/strategies.go b/internal/routes/strategies.go
index bc7d0ea30..121be9d63 100644
--- a/internal/routes/strategies.go
+++ b/internal/routes/strategies.go
@@ -267,4 +267,5 @@ var Builtin = map[string]Strategy{
provider.Pace: Pace,
provider.Weighted: Weighted,
provider.FreeFirst: FreeFirst,
+ provider.Fastest: Fastest,
}
diff --git a/internal/tui/routing.go b/internal/tui/routing.go
index 6907d9212..c31fdbf52 100644
--- a/internal/tui/routing.go
+++ b/internal/tui/routing.go
@@ -34,7 +34,7 @@ func (m *model) reloadGroups() {
m.grow = clamp(m.grow, len(m.groups))
}
-var routings = []string{"", provider.Ordered, provider.Rotate, provider.LeastUsed, provider.Pace, provider.Weighted, provider.FreeFirst, provider.Manual}
+var routings = []string{"", provider.Ordered, provider.Rotate, provider.LeastUsed, provider.Pace, provider.Weighted, provider.FreeFirst, provider.Fastest, provider.Manual}
func routingName(v string) string {
switch v {
@@ -50,6 +50,8 @@ func routingName(v string) string {
return "by weight"
case provider.FreeFirst:
return "free first"
+ case provider.Fastest:
+ return "speed first"
case provider.Manual:
return "manual"
}
diff --git a/plugins/rotation/rotation.go b/plugins/rotation/rotation.go
index 85a7f2538..31d7a1eee 100644
--- a/plugins/rotation/rotation.go
+++ b/plugins/rotation/rotation.go
@@ -1,6 +1,6 @@
// Package rotation is the routings a provider or group can be set to
// besides Smart: In order, In turn, Least used first, Weekly pace, By
-// weight and Free first. While a provider or group is routed by one, it
+// weight, Free first and Speed first. While a provider or group is routed by one, it
// stays on.
//
// It is the first consumer of the gateway's router strategy point (P3 of
@@ -18,7 +18,7 @@ import (
func init() {
pluginrt.Register(pluginrt.Manifest{
ID: "rotation", API: "^1", Requires: []string{"providers", "gateway"}, Essential: true,
- Name: "Routing strategies", Sub: "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight and Free first",
+ Name: "Routing strategies", Sub: "The routings besides Smart: In order, In turn, Least used first, Weekly pace, By weight, Free first and Speed first",
One: "1 provider or group routed so", Many: "{n} providers and groups routed so",
InUse: inUse,
Parts: []pluginrt.Part{{Slot: "part", Key: "rotation"}},
diff --git a/site/public/docs/ja/start.html b/site/public/docs/ja/start.html
index fa4043b7e..a11509d3b 100644
--- a/site/public/docs/ja/start.html
+++ b/site/public/docs/ja/start.html
@@ -165,7 +165,7 @@
</figure>
<h3>1 つのプロバイダに複数のアカウントやキー</h3>
- <p>プロバイダの行を開き、使うアカウントやキーをすべてチェックします。<span class="ui">Routing</span> はリクエストをそれらにどう振り分けるかを決めます:<span class="ui">Smart</span>(デフォルト)、<span class="ui">In order</span>、<span class="ui">In turn</span>、<span class="ui">Least used first</span>、<span class="ui">Weekly pace</span>。<span class="ui">Fallback</span> には、プロバイダのクォータ切れ、レート制限、ダウン時に代わりに使うモデルを並べます。</p>
+ <p>プロバイダの行を開き、使うアカウントやキーをすべてチェックします。<span class="ui">Routing</span> はリクエストをそれらにどう振り分けるかを決めます:<span class="ui">Smart</span>(デフォルト)、<span class="ui">In order</span>、<span class="ui">In turn</span>、<span class="ui">Least used first</span>、<span class="ui">Weekly pace</span>、<span class="ui">Speed first</span>(最近いちばん速く応答したキーが先)。<span class="ui">Fallback</span> には、プロバイダのクォータ切れ、レート制限、ダウン時に代わりに使うモデルを並べます。</p>
<h3>1 つのモデルを複数のプロバイダで</h3>
<p>グループとは、1 つまたは複数のプロバイダのモデルを、エージェントが 1 つとして選べるようにまとめたものです。ピッカーの <em>Routing groups</em> の下に <code>group/&lt;id&gt;</code> として表示されます。2 つのプロバイダが同じ名前で提供しているモデルは、それだけでグループになります。<span class="ui">Routing</span> タブの <span class="ui">New group</span> では、好きなモデルを組み合わせてグループを作れます。グループで何ができるかは、最初のモデルが基準になります。</p>
@@ -178,6 +178,7 @@
<tr><td>使用量の少ない順 <code>usage</code></td><td>利用枠の残りが最も多いアカウントやキー。</td></tr>
<tr><td>週のペース <code>pace</code></td><td>更新までの 1 時間あたりの週の枠の残りが最も多いアカウント、つまりリセット時に失う分が最も多いもの。これにより各メンバーの週の枠がリセットで無駄になりにくくなります。週の枠がなく 5 時間枠だけのアカウント(Claude Enterprise)は、更新までの 1 時間あたりの 5 時間枠の残りで判断されるので、ほぼ常に最初になります。使用率が 90% 以上のものは、他が応答できなくなるまで待ちます。キーは最近送ったトークン数で判断します。</td></tr>
<tr><td>重み順 <code>weight</code></td><td>グループ編集画面で各モデルの横に設定した重み(1〜1000、未設定は 1)に応じてリクエストを分けます。重み 3 のモデルは重み 1 のモデルが 1 件受けるごとに 3 件受け、まとめてではなく均等に散らします。各モデルは自分のプロバイダーのルーティングで自分のアカウントやキーを使い、グループ内のグループは 1 つとして数えます。失敗して休んでいるものはその回から外れ、失敗したときは次のモデルが代わりに答えます。</td></tr>
+ <tr><td>速度優先 <code>fast</code></td><td>このモデルに最近いちばん速く応答したメンバー、アカウント、キーが先です。最初のトークンまでの時間と出力の速さで比べ、新しい応答ほど重く数えます。差が 1 割以内のものは並び順のままです。1 時間測っていないものには 5 件に 1 件を回すので、一度遅かったチャネルもまた試されます。失敗して休んでいるものは最後です。測るのはストリーミングの応答だけで、再起動すると測り直します。</td></tr>
</table></div>
<div class="table-wrap"><table>
<tr><th>維持</th><th>会話が、応答したアカウントやキーに留まる期間</th></tr>
diff --git a/site/public/docs/start.html b/site/public/docs/start.html
index 9f81c430b..de4f821fe 100644
--- a/site/public/docs/start.html
+++ b/site/public/docs/start.html
@@ -165,7 +165,7 @@
</figure>
<h3>Several accounts or keys in one provider</h3>
- <p>Open the provider’s row and tick every account or key to use. <span class="ui">Routing</span> says how requests spread over them: <span class="ui">Smart</span> (the default), <span class="ui">In order</span>, <span class="ui">In turn</span>, <span class="ui">Least used first</span> or <span class="ui">Weekly pace</span>. <span class="ui">Fallback</span> lists models to use instead when the provider is out of quota, rate limited or down.</p>
+ <p>Open the provider’s row and tick every account or key to use. <span class="ui">Routing</span> says how requests spread over them: <span class="ui">Smart</span> (the default), <span class="ui">In order</span>, <span class="ui">In turn</span>, <span class="ui">Least used first</span>, <span class="ui">Weekly pace</span> or <span class="ui">Speed first</span>, the key that answered fastest lately first. <span class="ui">Fallback</span> lists models to use instead when the provider is out of quota, rate limited or down.</p>
<h3>One model over several providers</h3>
<p>A group is several models, from one provider or many, that an agent picks as one: <code>group/&lt;id&gt;</code>, under <em>Routing groups</em> in the picker. A model two of your providers serve under the same name becomes a group on its own; <span class="ui">New group</span> in the <span class="ui">Routing</span> tab makes one of any models you like. The first model answers for what the group can do.</p>
@@ -178,6 +178,7 @@
<tr><td>Least used <code>usage</code></td><td>The account or key with the most of its allowance left.</td></tr>
<tr><td>Weekly pace <code>pace</code></td><td>The account with the most of its week left per hour until it renews — the one with the most to lose at its reset — so less of each member’s week is lost at its reset; an account with five hours and no week (Claude Enterprise) by what its five hours have left per hour until they renew, so almost always first. One at 90% or more waits until the others can’t answer. A key by the tokens sent it lately.</td></tr>
<tr><td>By weight <code>weight</code></td><td>Each model takes its share of the requests by the weight set beside it in the group editor (1 to 1000, 1 if unset): one weighing 3 takes three for every one a model weighing 1 takes, spread out rather than in bursts. Each model is tried whole over its own accounts or keys, as its provider routes them, and a group in the group counts as one. One resting after a failure leaves the round; the next model is the fallback when one fails.</td></tr>
+ <tr><td>Speed first <code>fast</code></td><td>The member, account or key that answered this model fastest lately: the time to its first token and how fast it writes, newer answers counting more. Those within a tenth of each other keep their order. One not measured in the last hour gets one request in five, so a channel that was slow once is tried again; one resting after a failure goes last. Only streamed replies are measured, and a restart measures again from scratch.</td></tr>
</table></div>
<div class="table-wrap"><table>
<tr><th>Stays</th><th>How long a conversation stays with the account or key that answered it</th></tr>
diff --git a/site/public/docs/zh/start.html b/site/public/docs/zh/start.html
index 6be0068c4..f73bb4224 100644
--- a/site/public/docs/zh/start.html
+++ b/site/public/docs/zh/start.html
@@ -165,7 +165,7 @@
</figure>
<h3>一个供应商里的多个账号或密钥</h3>
- <p>展开供应商那一行,勾选要用的每个账号或密钥。<span class="ui">路由</span> 决定请求怎么分给它们:<span class="ui">智能</span>(默认)、<span class="ui">按顺序</span>、<span class="ui">轮流</span>、<span class="ui">用量少的优先</span> 或 <span class="ui">重置前用完</span>。<span class="ui">备用</span> 里列出这个供应商额度用完、被限流或不可用时改用的模型。</p>
+ <p>展开供应商那一行,勾选要用的每个账号或密钥。<span class="ui">路由</span> 决定请求怎么分给它们:<span class="ui">智能</span>(默认)、<span class="ui">按顺序</span>、<span class="ui">轮流</span>、<span class="ui">用量少的优先</span>、<span class="ui">重置前用完</span> 或 <span class="ui">速度优先</span>(最近回答最快的密钥先用)。<span class="ui">备用</span> 里列出这个供应商额度用完、被限流或不可用时改用的模型。</p>
<h3>跨供应商的同一个模型</h3>
<p>路由组是几个模型(来自一个或多个供应商),Agent 把它当作一个来选:<code>group/&lt;id&gt;</code>,在选择列表的 <em>路由组</em> 下面。两个供应商用同一个名字提供的模型会自动成组;在 <span class="ui">路由</span> 标签页点 <span class="ui">新建组</span> 可以把任意模型组到一起。排第一的模型决定这个组对外的能力。</p>
@@ -178,6 +178,7 @@
<tr><td>最少使用 <code>usage</code></td><td>剩余额度最多的账号或 Key 排第一。</td></tr>
<tr><td>重置前用完 <code>pace</code></td><td>每周剩余额度除以距重置的小时数,最高的账号排第一——重置时损失最多的那个——少让成员的周额度到重置时作废;只有 5 小时额度、没有周额度的账号(如 Claude Enterprise)按 5 小时窗口的剩余额度除以距它重置的小时数算,所以几乎总排第一。用到 90% 以上的等其他账号都答不了时才用。Key 按最近发给它的 token 数排。</td></tr>
<tr><td>按权重 <code>weight</code></td><td>每个模型按群组编辑器里它旁边设的权重(1 到 1000,不设为 1)分得请求:权重 3 的模型每拿三个,权重 1 的拿一个,均匀穿插而不是成批。每个模型按它自己提供方的路由用自己的账号或 Key,群组里的群组算一个成员。失败后歇着的那个退出这一轮;一个失败时,下一个模型接上。</td></tr>
+ <tr><td>速度优先 <code>fast</code></td><td>最近回答这个模型最快的成员、账号或 Key 先用:看首字时间和输出速度,越新的回答算得越重。相差不到一成的保持原来的顺序。一小时内没测过的每五个请求分到一个,慢过一次的渠道还会再试;失败后歇着的排最后。只测流式回复,重启后重新测。</td></tr>
</table></div>
<div class="table-wrap"><table>
<tr><th>会话保持</th><th>会话在回答过它的账号或 Key 上留多久</th></tr>