Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

gateway, cli: magpie goroutines writes the running magpie's goroutine stacks to a file (#3362)

JoeyWu-techb18a6a080b81
diff --git a/docs/reference.md b/docs/reference.md
index 675a2b246..d73813c16 100644
--- a/docs/reference.md
+++ b/docs/reference.md
@@ -1936,6 +1936,21 @@ profile at a time, so a second `magpie cpu` while one records is refused
409; one whose caller hangs up stops recording. The route answers the same
callers as `/v1/magpie/heap`, and refuses the same ones 403.
+`magpie goroutines [<file>]` is for a report of magpie stuck at something
+while it goes on running, such as an app that kept an old partner list for
+days (#3362): killing it to look would end what was stuck. It asks the same
+running magpie for `GET /v1/magpie/goroutines`, every goroutine's stack as
+text (`runtime/pprof`'s `goroutine` profile at debug 2, the form a crash
+prints): each goroutine's state and how long it has waited
+(`[select, 4320 minutes]`), its frames with source lines, and who started
+it. Arguments are printed as numbers, never what they point at, so it holds
+no keys or messages. Nothing is stopped. It writes the dump to `<file>`,
+else `magpie-goroutines-<date>-<time>.txt` in the working folder, never
+over a file there, and prints the file and the process the gateway names in
+`X-Magpie-Goroutine-Stats` (its id, program, version, uptime and goroutine
+count). The route answers the same callers as `/v1/magpie/heap`, and
+refuses the same ones 403.
+
A gateway key held to a budget, to some models or to some accounts is told
none of this magpie's accounts (#1981): `/v1/magpie/quotas`,
`/v1/magpie/quotas/history`, `/v1/magpie/quotas/cards`,
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index 5ad40832b..638ae70ca 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -28,7 +28,7 @@ backfilled; see [Gateway Session Aggregation](gateway-sessions.md).
| --- | --- | --- |
| Address and caller | `DefaultAddr`. The local token `magpie` and the per-agent `magpie-<agent>` (`TokenFor`). `agentOf` tells which agent asked, first by token, then by User-Agent. Claude Science, whose User-Agent is the Anthropic SDK's, is told by its client's `X-Anthropic-Surface: operon` (`isClaudeScience`). It is listed every model as `claude-magpie-<number>` (`scienceModels`; its picker keeps only a `claude-` id whose name isn't a lower-case slug), and a Claude id magpie doesn't serve (its default before a pick, its haiku side tasks) goes to the model its last turn with tools was on, else the first it is listed (`scienceTurn`, kept in `claude-science.model`). | [`gateway.go`](../../internal/gateway/gateway.go) |
| TLS | `--tls-cert`/`--tls-key` (`MAGPIE_TLS_CERT`, `MAGPIE_TLS_KEY`) make `ListenAndServe`, `Relisten` (both through `Server.listen`) and `magpie web`'s page (`ServeTLS`) wrap the listener in `tlsListener`. A connection from another computer is `tls.Server` at once, so its plain HTTP gets net/http's 400. One from this computer (`fromThisComputer`: loopback, or a peer address equal to the local one) is sniffed on its own goroutine, 10s at most (`sniffWait`), and 0x16 makes it TLS, so `URL()`, the agents, MCP servers, `ServedBy` and the healthcheck keep `http://127.0.0.1`. `certFiles.get` reloads the pair when either file's mtime or size changes; a pair that doesn't load keeps the one served (logged once per change). `tlsConfig` refuses half a pair or files that don't load before binding. HTTP/1.1 only over TLS (hijack, Codex's websocket). `Scheme()` gives the scheme of `LANURLs`, a scheme-less `MAGPIE_PUBLIC_URL` and the page's network links. Host and Origin checks (`foreignPage`, `rebound`) don't depend on the scheme. | [`tls.go`](../../internal/gateway/tls.go), [`lan.go`](../../internal/gateway/lan.go) |
-| HTTP surface | `Handler` routes `/v1/chat/completions`, `/v1/responses`, `/v1/messages` (each also without `/v1`), `/v1/models`, Gemini's `/v1beta/models/…`, the Codex backend path, `/v1/codex/models` (Codex's catalog shape for a Codex elsewhere with a gateway key, `codexCatalog`, kept to the key's models by `keyAllowed`; #1281), Ollama's `GET /api/version`, `GET /api/tags` and `POST /api/show` (`ollama.go`, for a client that takes a custom endpoint only as an Ollama server: Visual Studio's and VS Code's Copilot, wani on Discord; the list is `catalogFor`'s, so a key's models alone, `/api/show` tells `capabilities` (`completion`, `tools`, `vision` from `Images`, `thinking` from `Thinks`) and `model_info` with `general.architecture` `magpie`, `general.basename`, `magpie.context_length` (32768, VS Code's own default, when the window isn't known, since VS Code would take a 0) and the `general.parameter_count` and `magpie.embedding_length` the ollama CLI before 0.7 reads unchecked; a name ending `:latest`, as the ollama CLI asks, is the bare id unless a model is really called that (`ollamaName`); the client then chats on `/v1/chat/completions`, or on Ollama's own `POST /api/chat` and `POST /api/generate` (`ollama_chat.go`, Zed's Ollama provider, super_lalalalala on Discord), which `ollamaChatBody` turns into a `/v1/chat/completions` body handed to the same `handle(provider.Chat)` on a clone of the request, so they are routed, held to a key's models and counted as it is: `messages` with `images` (base64, its type sniffed, as data-URL parts; a tool message's in a user message after it), an assistant's `thinking` as `reasoning_content` and its `tool_calls` given ids where Ollama sent none, a tool result matched to its call by `tool_call_id`, else `tool_name`, else in order (one with no call is a 400), `tools`, `format` (`json` or a schema), `think` (a level is `reasoning_effort`), and `options`' `temperature`, `top_p`, `seed`, `stop`, the two penalties and a positive `num_predict` as `max_tokens` (`num_ctx`, `top_k` and the rest of Ollama's own are dropped); `ollamaWriter` turns the reply into Ollama's NDJSON lines (streamed unless `stream:false`, then one object), tool calls with their `arguments` as an object and an `id`, and a last `done` line with `done_reason` and the token counts, and an error into `{"error":"…"}` with the status `/v1` would give; a request with no messages or prompt is answered `done_reason:"load"`, as Ollama does to warm a model), `/mcp/{name}`, `/mcp/magpie/web-search` and `/mcp/magpie/vision` (magpie's own web search and image recognition as MCP servers, below) and the `/v1/magpie/*` status endpoints. `/v1/magpie/heap` (`heapProfile`, [`heap.go`](../../internal/gateway/heap.go), for `magpie heap`) is held tighter than the rest by `heapRefused` in the handler itself, so it holds where `Handler` is served without `lanGuard` too: only a loopback caller that isn't proxied (`local`), carries no browser mark (`fromBrowser`) and names a loopback `Host` (`loopbackHost`) gets the profile; a gateway key never lets anything else in. `/v1/magpie/cpu` (`cpuProfile`, [`cpu.go`](../../internal/gateway/cpu.go), for `magpie cpu`) records a CPU profile for `?seconds=` (30, from 5 to 120) and is held to the same callers by `cpuRefused`, which shares `profileRefused` with `heapRefused`; one profile records at a time (`cpuRecording`), a second is answered 409, and a caller that hangs up stops it. `/v1/alpha/search` (also without `/v1`, and on the Codex backend path for a magpie model) is Codex's web.run search (`alphaSearch`, `codex_search.go`): a ChatGPT account's model goes to the ChatGPT backend's `/alpha/search` on the account's sign-in, the next account asked on 401, 403, 429 or 5xx; any other model's `search_query` is searched by `webSearch`, and when nothing answers the reason is a 200 with `output`, since Codex ends the turn on any failed web.run (anton on Discord). A GET on `/v1/responses` (a client opening Responses over a WebSocket) is answered 426, a plain one 405: Responses is relayed over HTTP only. Every route is wrapped in `counted`, `corsGuard`, `callerGuard`, `withCaller` and `keyLimited`; `/v1/messages`, `/messages` and `/v1/messages/count_tokens` in `keepsSignIn` too. `corsGuard` (#1051) serves CORS only to the origins listed in Settings › Network › Web pages (`settings.CORSOrigins`, cleaned to `scheme://host[:port]` by `settings.CleanOrigins`, which refuses a wildcard). A browser extension's own pages, popup and service worker are listed the same way by their origin, which is the browser's scheme and the extension's id: `chrome-extension://<id>` (Chromium, the id 32 letters a–p), `moz-extension://<uuid>` (Firefox, the Internal UUID about:debugging shows, not the add-on id) and `safari-web-extension://<uuid>` (Safari, which picks a new UUID on each launch, so a listing lasts until Safari restarts); the id must have that browser's shape, cased any way, and nothing else is taken — no bare scheme, no wildcard, no path (SkyAerope on Discord: an extension without host permissions is held to CORS like a page). A content script's fetch carries the page's origin, not the extension's. A listed extension is treated exactly as a listed page, below; one not listed passes as before (other schemes, below) with no CORS headers. For a listed origin it answers the preflight (204) and adds `Access-Control-Allow-Origin`, and every other request from it needs an enabled gateway key (`access.Authenticate`), even on loopback, or gets a 401. It takes the key from the caller `lanGuard` recorded too, since on the running server `lanGuard` has already put magpie's own token in the key's place. Any other web page is refused 403 on every route (`foreignPage`): on loopback a call needs no key and the body's Content-Type isn't checked, so a page the user visits could otherwise send a "simple" `text/plain` POST and spend their subscriptions blind. Only an `http`/`https` Origin is judged, and one on this computer (`localhost`, `*.localhost`, `127.0.0.0/8`, `[::1]`, any port) or the request's own origin (its host and port equal to `Host`, such as a page on the gateway's LAN address) passes, as does usemagpie.ai on `/v1/magpie/omarchy`. Other schemes (`app://`, `file://`, `tauri://`, `vscode-webview://`, `chrome-extension://`) come from desktop apps, webviews and extensions that don't enforce CORS, and pass keyless as before; `null` is not one of them, since any web page sends it from a sandboxed frame and an opened file sends it too, so a call with it needs an enabled gateway key as a listed origin's does, and is answered 403 without one (no CORS headers either way, so a frame or a local page still reads no reply). A page at a subdomain of localhost (`localhostSubdomain`, #2256) needs a key the same way: Chrome and Firefox resolve `*.localhost` to loopback themselves, but Safari and other WebKit views ask the system resolver, which before macOS 26 asks the network's DNS (WebKit bug 160504), so a resolver that answers serves such a page from anywhere. `tauri.localhost` and `wails.localhost` stay keyless on Windows only, where Tauri's and Wails' webviews serve themselves at those names; on macOS and Linux those apps use `tauri://` and `wails://`. A request with no Origin (every agent's) passes through with no CORS headers, as before. A page whose own name resolves to 127.0.0.1 (DNS rebinding) would count as same-origin here, so `lanGuard` refuses it first (`rebound`, 403): a loopback request carrying a browser's marks (`Origin`, `Sec-Fetch-Site`/`Sec-Fetch-Mode`, a `Mozilla/` user agent) whose `Host` isn't this computer's (an IP, `localhost` but not a subdomain of it, which WebKit asks the DNS for (#2256), a container runtime's name for the host such as `host.docker.internal`, `MAGPIE_PUBLIC_URL`'s or `MAGPIE_ADDR`'s host, the machine's own name) needs an enabled gateway key. Agents send none of those marks and pass under any name. Before any of that, `lanGuard` lets in one request without a key from anywhere: `POST /_magpie/claude-mcp/<token>` with the token of a Claude subscription run now live (`ownCallback`, see [the Claude subscription bridge](claude-subscription-bridge.md)), the run's MCP helper calling back to a `MAGPIE_ADDR` that isn't loopback (#1546). Next, before the key or `rebound` is looked at, the allow list (lc on Discord, magpie in Docker on a public host): with `MAGPIE_ALLOW_IPS` or Settings › Network › Allowed addresses (`settings.AllowIPs`, this computer's own, never brought by a sync or restore: see [sync and backup](sync-backup.md#what-stays-on-each-computer); the variable wins and the page then only shows it) naming IPs and CIDR ranges, a request whose connection (`RemoteAddr`, IPv4-mapped IPv6 unmapped, zone dropped) is outside them is refused 403 naming its address, even with an enabled gateway key (`refuseNotAllowed`). Loopback and this computer's own interface addresses always pass. A forwarding header is never taken for the address, so a proxy or tunnel on this computer passes (it arrives from loopback) and its clients are limited at the proxy. An entry that doesn't parse lets nothing in and is logged once; the list stays on. Settings' list is read with `settings.LoadChecked`: a settings.json that can't be read, or doesn't parse with no last good generation, is unknown, never "no list", so the list read last in this process stays in force (`lastAllow`), and with none read yet only this computer passes, the 403 saying settings.json couldn't be read; logged once. No file, or an empty one, is no list. The Settings page's saves read with `LoadChecked` too and refuse such a file, as `settings.Save` does. `magpie web`'s server is wrapped in the same check (`AllowGuard`). Settings refuses a list that would leave out the address the page saving it is open from (`KeepsOut`); the page names that address, which behind Docker or a proxy is the proxy's, and offers to add it to the refused list (`allowSays`, Mathresearcher on Discord). | [`gateway.go`](../../internal/gateway/gateway.go), [`allow.go`](../../internal/gateway/allow.go), [`internal/settings/allowips.go`](../../internal/settings/allowips.go), [`cors.go`](../../internal/gateway/cors.go), [`codex_catalog.go`](../../internal/gateway/codex_catalog.go), [`ollama.go`](../../internal/gateway/ollama.go), [`ollama_chat.go`](../../internal/gateway/ollama_chat.go), [`internal/settings/origins.go`](../../internal/settings/origins.go) |
+| HTTP surface | `Handler` routes `/v1/chat/completions`, `/v1/responses`, `/v1/messages` (each also without `/v1`), `/v1/models`, Gemini's `/v1beta/models/…`, the Codex backend path, `/v1/codex/models` (Codex's catalog shape for a Codex elsewhere with a gateway key, `codexCatalog`, kept to the key's models by `keyAllowed`; #1281), Ollama's `GET /api/version`, `GET /api/tags` and `POST /api/show` (`ollama.go`, for a client that takes a custom endpoint only as an Ollama server: Visual Studio's and VS Code's Copilot, wani on Discord; the list is `catalogFor`'s, so a key's models alone, `/api/show` tells `capabilities` (`completion`, `tools`, `vision` from `Images`, `thinking` from `Thinks`) and `model_info` with `general.architecture` `magpie`, `general.basename`, `magpie.context_length` (32768, VS Code's own default, when the window isn't known, since VS Code would take a 0) and the `general.parameter_count` and `magpie.embedding_length` the ollama CLI before 0.7 reads unchecked; a name ending `:latest`, as the ollama CLI asks, is the bare id unless a model is really called that (`ollamaName`); the client then chats on `/v1/chat/completions`, or on Ollama's own `POST /api/chat` and `POST /api/generate` (`ollama_chat.go`, Zed's Ollama provider, super_lalalalala on Discord), which `ollamaChatBody` turns into a `/v1/chat/completions` body handed to the same `handle(provider.Chat)` on a clone of the request, so they are routed, held to a key's models and counted as it is: `messages` with `images` (base64, its type sniffed, as data-URL parts; a tool message's in a user message after it), an assistant's `thinking` as `reasoning_content` and its `tool_calls` given ids where Ollama sent none, a tool result matched to its call by `tool_call_id`, else `tool_name`, else in order (one with no call is a 400), `tools`, `format` (`json` or a schema), `think` (a level is `reasoning_effort`), and `options`' `temperature`, `top_p`, `seed`, `stop`, the two penalties and a positive `num_predict` as `max_tokens` (`num_ctx`, `top_k` and the rest of Ollama's own are dropped); `ollamaWriter` turns the reply into Ollama's NDJSON lines (streamed unless `stream:false`, then one object), tool calls with their `arguments` as an object and an `id`, and a last `done` line with `done_reason` and the token counts, and an error into `{"error":"…"}` with the status `/v1` would give; a request with no messages or prompt is answered `done_reason:"load"`, as Ollama does to warm a model), `/mcp/{name}`, `/mcp/magpie/web-search` and `/mcp/magpie/vision` (magpie's own web search and image recognition as MCP servers, below) and the `/v1/magpie/*` status endpoints. `/v1/magpie/heap` (`heapProfile`, [`heap.go`](../../internal/gateway/heap.go), for `magpie heap`) is held tighter than the rest by `heapRefused` in the handler itself, so it holds where `Handler` is served without `lanGuard` too: only a loopback caller that isn't proxied (`local`), carries no browser mark (`fromBrowser`) and names a loopback `Host` (`loopbackHost`) gets the profile; a gateway key never lets anything else in. `/v1/magpie/cpu` (`cpuProfile`, [`cpu.go`](../../internal/gateway/cpu.go), for `magpie cpu`) records a CPU profile for `?seconds=` (30, from 5 to 120) and is held to the same callers by `cpuRefused`, which shares `profileRefused` with `heapRefused`; one profile records at a time (`cpuRecording`), a second is answered 409, and a caller that hangs up stops it. `/v1/magpie/goroutines` (`goroutineDump`, [`goroutines.go`](../../internal/gateway/goroutines.go), for `magpie goroutines`, #3362) answers every goroutine's stack and wait as text (`runtime/pprof`'s `goroutine` profile at debug 2) to the same callers, through `goroutinesRefused`. `/v1/alpha/search` (also without `/v1`, and on the Codex backend path for a magpie model) is Codex's web.run search (`alphaSearch`, `codex_search.go`): a ChatGPT account's model goes to the ChatGPT backend's `/alpha/search` on the account's sign-in, the next account asked on 401, 403, 429 or 5xx; any other model's `search_query` is searched by `webSearch`, and when nothing answers the reason is a 200 with `output`, since Codex ends the turn on any failed web.run (anton on Discord). A GET on `/v1/responses` (a client opening Responses over a WebSocket) is answered 426, a plain one 405: Responses is relayed over HTTP only. Every route is wrapped in `counted`, `corsGuard`, `callerGuard`, `withCaller` and `keyLimited`; `/v1/messages`, `/messages` and `/v1/messages/count_tokens` in `keepsSignIn` too. `corsGuard` (#1051) serves CORS only to the origins listed in Settings › Network › Web pages (`settings.CORSOrigins`, cleaned to `scheme://host[:port]` by `settings.CleanOrigins`, which refuses a wildcard). A browser extension's own pages, popup and service worker are listed the same way by their origin, which is the browser's scheme and the extension's id: `chrome-extension://<id>` (Chromium, the id 32 letters a–p), `moz-extension://<uuid>` (Firefox, the Internal UUID about:debugging shows, not the add-on id) and `safari-web-extension://<uuid>` (Safari, which picks a new UUID on each launch, so a listing lasts until Safari restarts); the id must have that browser's shape, cased any way, and nothing else is taken — no bare scheme, no wildcard, no path (SkyAerope on Discord: an extension without host permissions is held to CORS like a page). A content script's fetch carries the page's origin, not the extension's. A listed extension is treated exactly as a listed page, below; one not listed passes as before (other schemes, below) with no CORS headers. For a listed origin it answers the preflight (204) and adds `Access-Control-Allow-Origin`, and every other request from it needs an enabled gateway key (`access.Authenticate`), even on loopback, or gets a 401. It takes the key from the caller `lanGuard` recorded too, since on the running server `lanGuard` has already put magpie's own token in the key's place. Any other web page is refused 403 on every route (`foreignPage`): on loopback a call needs no key and the body's Content-Type isn't checked, so a page the user visits could otherwise send a "simple" `text/plain` POST and spend their subscriptions blind. Only an `http`/`https` Origin is judged, and one on this computer (`localhost`, `*.localhost`, `127.0.0.0/8`, `[::1]`, any port) or the request's own origin (its host and port equal to `Host`, such as a page on the gateway's LAN address) passes, as does usemagpie.ai on `/v1/magpie/omarchy`. Other schemes (`app://`, `file://`, `tauri://`, `vscode-webview://`, `chrome-extension://`) come from desktop apps, webviews and extensions that don't enforce CORS, and pass keyless as before; `null` is not one of them, since any web page sends it from a sandboxed frame and an opened file sends it too, so a call with it needs an enabled gateway key as a listed origin's does, and is answered 403 without one (no CORS headers either way, so a frame or a local page still reads no reply). A page at a subdomain of localhost (`localhostSubdomain`, #2256) needs a key the same way: Chrome and Firefox resolve `*.localhost` to loopback themselves, but Safari and other WebKit views ask the system resolver, which before macOS 26 asks the network's DNS (WebKit bug 160504), so a resolver that answers serves such a page from anywhere. `tauri.localhost` and `wails.localhost` stay keyless on Windows only, where Tauri's and Wails' webviews serve themselves at those names; on macOS and Linux those apps use `tauri://` and `wails://`. A request with no Origin (every agent's) passes through with no CORS headers, as before. A page whose own name resolves to 127.0.0.1 (DNS rebinding) would count as same-origin here, so `lanGuard` refuses it first (`rebound`, 403): a loopback request carrying a browser's marks (`Origin`, `Sec-Fetch-Site`/`Sec-Fetch-Mode`, a `Mozilla/` user agent) whose `Host` isn't this computer's (an IP, `localhost` but not a subdomain of it, which WebKit asks the DNS for (#2256), a container runtime's name for the host such as `host.docker.internal`, `MAGPIE_PUBLIC_URL`'s or `MAGPIE_ADDR`'s host, the machine's own name) needs an enabled gateway key. Agents send none of those marks and pass under any name. Before any of that, `lanGuard` lets in one request without a key from anywhere: `POST /_magpie/claude-mcp/<token>` with the token of a Claude subscription run now live (`ownCallback`, see [the Claude subscription bridge](claude-subscription-bridge.md)), the run's MCP helper calling back to a `MAGPIE_ADDR` that isn't loopback (#1546). Next, before the key or `rebound` is looked at, the allow list (lc on Discord, magpie in Docker on a public host): with `MAGPIE_ALLOW_IPS` or Settings › Network › Allowed addresses (`settings.AllowIPs`, this computer's own, never brought by a sync or restore: see [sync and backup](sync-backup.md#what-stays-on-each-computer); the variable wins and the page then only shows it) naming IPs and CIDR ranges, a request whose connection (`RemoteAddr`, IPv4-mapped IPv6 unmapped, zone dropped) is outside them is refused 403 naming its address, even with an enabled gateway key (`refuseNotAllowed`). Loopback and this computer's own interface addresses always pass. A forwarding header is never taken for the address, so a proxy or tunnel on this computer passes (it arrives from loopback) and its clients are limited at the proxy. An entry that doesn't parse lets nothing in and is logged once; the list stays on. Settings' list is read with `settings.LoadChecked`: a settings.json that can't be read, or doesn't parse with no last good generation, is unknown, never "no list", so the list read last in this process stays in force (`lastAllow`), and with none read yet only this computer passes, the 403 saying settings.json couldn't be read; logged once. No file, or an empty one, is no list. The Settings page's saves read with `LoadChecked` too and refuse such a file, as `settings.Save` does. `magpie web`'s server is wrapped in the same check (`AllowGuard`). Settings refuses a list that would leave out the address the page saving it is open from (`KeepsOut`); the page names that address, which behind Docker or a proxy is the proxy's, and offers to add it to the refused list (`allowSays`, Mathresearcher on Discord). | [`gateway.go`](../../internal/gateway/gateway.go), [`allow.go`](../../internal/gateway/allow.go), [`internal/settings/allowips.go`](../../internal/settings/allowips.go), [`cors.go`](../../internal/gateway/cors.go), [`codex_catalog.go`](../../internal/gateway/codex_catalog.go), [`ollama.go`](../../internal/gateway/ollama.go), [`ollama_chat.go`](../../internal/gateway/ollama_chat.go), [`internal/settings/origins.go`](../../internal/settings/origins.go) |
| Model resolution | `serve` resolves the model the agent asked for: Claude tier stand-ins, a routing group (`provider.GroupFor`: its id, the model's id however a vendor spells it, else the one group whose name it is, unless a provider serves a model by that id; the user's own groups only while the `groups` plugin is on, [the plugin runtime](plugin-runtime.md#gateway-extension-points-p3-router-strategy)), auto stand-ins, the agent's own stand-in (`StandIn`: Codex's config.toml `model` when magpie is its provider, for a name it sends that magpie doesn't serve, such as its auto-review's `codex-auto-review`), itself taken through `GroupFor` and auto stand-ins so a config naming a group bare reaches it, then `provider.Resolve`. A Codex memory call or auto-review (`x-openai-subagent: guardian`, or `codex-auto-review`) that still names no served model goes to the model Codex's last turn was answered on (`codexMemoryStandIn`, per agent and gateway key), so a Codex on magpie through another provider table, a profile or another computer isn't answered 404 (Adam on Discord, `TestCodexAutoReviewReachesItsTurnsModel`). A bare model id several providers serve (MOMO on Discord: `glm-5.3` under opencode-go and a6api) is never refused: with found groups on (the default) `GroupFor` takes it to the group magpie found for it (`auto-<slug>`), its members in the Providers order, routed and failed over as any group, a conversation staying with the member that answered it while its cache is worth it (affinity); with them off, `Resolve` takes the first provider in the Providers order that exposes it, with no fallback. magpie writes only `provider/model` ids into agents' configs (`magpieModels`), so a bare id is one the user wrote. `/v1/models` lists bare ids only when the user asks for them ([Model ids without the provider](#model-ids-without-the-provider)). Every reply says who answered in `X-Magpie-Provider` and `X-Magpie-Model` (`noteMember`), and the usage log and the Routing page keep it. A model id with `~` and no slash that resolves as nothing else is the one served id (a provider's model or a group) whose slashes it spells as `~` (`unflat`, `provider.Unflat`; MiniMax Code's spelling, #1387), taken back in `validateModel` before key limits, middleware and usage see it; two ids spelled alike are neither. A Claude model id whose version is dashed (`relay/claude-opus-4-6-us`, with or without `:level` and `[1m]`) that is not served as spelled is the one served id it dashes (`provider.ClaudeUndashed`; the spelling magpie gives Claude Code, which reads a dotted version as Claude Opus 4), taken back in `validateModel` and `unprefixed` the same way. An unknown model gets a 404 naming why: a switched-off provider, a disabled group or an empty group. Otherwise it names up to three ids nearest to what was asked (`provider.Closest`: words of a model's display name count before words of its id, and a provider's id or name only alongside them, so "Zen-Space-Bunny" is told `opencode-zen-free/space-bunny-free`, MOMO on Discord), then the ways a model can be named, then every id (`unknownModel`; chat and embeddings). | [`gateway.go`](../../internal/gateway/gateway.go), [`internal/provider/group.go`](../../internal/provider/group.go), [`internal/provider/flat.go`](../../internal/provider/flat.go), [`internal/provider/claudedash.go`](../../internal/provider/claudedash.go) |
| A found group's two ids | A group magpie found is listed (`/v1/models`, the catalog, the lists written into agents, the pickers, the CLI and TUI) as `auto/<model>`: the model's id as a vendor spells it, dots kept and with no vendor prefix (`auto/claude-opus-5.5`; `provider.Group.Ref`, `spelledIn`, preferring a spelling models.dev lists), so a client that looks the part after the slash up on models.dev finds it (ZeroClover on X). Its old id `group/auto-<slug>` is served forever. Inside magpie the group keeps that one id: `validateModel` takes `auto/<model>` (with `:level` or `[1m]`) back to it with `provider.CanonicalRef` before key limits, routing, affinity and the usage log see the request, and the request body is rewritten to it, so both ids count as one group. A gateway key naming either id, `group/*` or `auto/*` passes for both (`allows` in `key_models.go`). `SaveGroup` stores a member, rule or classifier picked as `auto/<model>` by the old id. An agent field magpie set to `group/auto-<slug>` moves to `auto/<model>` on the next `SyncCatalog` (`agent.renameFoundGroups`), but only while its value is still what applied.json says magpie wrote; a value the user set or changed stays. A pick of the group by its old id written now (`Agent.Apply`) is written as `auto/<model>`. An agent's hidden, picked and ordered models saved by the old id hold for the new one (`PublicRef` in `visible.go`), and a bare model id still reaches its found group (`bareTarget`). Claude Code is given a dotted Claude version dashed, so it may send `auto/claude-sonnet-4-5`: that is the same group, and `Reseat` still moves an agent off it once the group is gone (`samePick`). `auto` is no provider id: `Save` and `Rename` refuse it for a new provider, and a plugin's `auto` is `auto-plugin` (`PluginID`). A provider of the user's saved as `auto` before keeps its models: while it exists, `auto/<x>` is its model and found groups list as `group/auto-<slug>` (`autoRefsOn`). Hand-made groups stay `group/<id>`. | [`internal/provider/autoref.go`](../../internal/provider/autoref.go), [`key_models.go`](../../internal/gateway/key_models.go), [`internal/agent/autoref.go`](../../internal/agent/autoref.go) |
| Candidates | `plan` lists one provider's candidates: one per key or account on, then each `Fallback` model, with resting ones last (`restLast`). `planGroup` and `planLevel` weigh a group's members together. A nested group plans by its own routing (#576). So does a member whose provider routes other than Smart (`ownRouting`): weighed as one by its first not resting (`firstAwake`), tried whole in its own order; `leastHeld` leaves such a group's order alone too. A provider on Smart, or routed as the group is, is pooled with the group's other keys and accounts. `group_own_routing_test.go`. A member's own fallbacks are not the group's. | [`fallback.go`](../../internal/gateway/fallback.go) |
diff --git a/internal/cli/goroutines_cli.go b/internal/cli/goroutines_cli.go
new file mode 100644
index 000000000..fe4bafcf6
--- /dev/null
+++ b/internal/cli/goroutines_cli.go
@@ -0,0 +1,135 @@
+package cli
+
+// magpie goroutines: every goroutine of the running magpie, with its stack
+// and how long it has waited, for a report of magpie stuck at something
+// while it goes on running (#3362). Killing the app to see would end what
+// was stuck, so the dump is asked of the process itself, through the
+// gateway it serves on this computer (gateway.GoroutinesPath, loopback
+// only, as magpie heap's).
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "net/http"
+ "os"
+ "path/filepath"
+ "strings"
+ "time"
+
+ "github.com/yetone/magpie/internal/gateway"
+)
+
+const goroutinesUsage = `usage: magpie goroutines [<file>]
+ every goroutine of the magpie serving the gateway on this computer (the app, the menu
+ bar, magpie serve or magpie web), with its stack and how long it has waited, for a
+ report of magpie stuck at something while it runs. magpie keeps running; nothing is
+ stopped. It is written to <file>, else to magpie-goroutines-<date>-<time>.txt in this
+ folder, and never over a file that is there. magpie then prints the file and which
+ process it is of: send both. The dump holds function names, source lines and the path of
+ the magpie program, with arguments as numbers only, not keys or messages.
+ It asks the gateway at the port in Settings' Gateway address (or MAGPIE_ADDR) over loopback.`
+
+// goroutinesWait bounds the ask: a dump of thousands of goroutines is
+// written in well under a second, so a magpie that doesn't answer in this
+// is stuck where its gateway answers too.
+const goroutinesWait = 30 * time.Second
+
+// goroutinesCmd: magpie goroutines [<file>]
+func goroutinesCmd(args []string) error {
+ file := ""
+ for _, a := range args {
+ switch {
+ case strings.HasPrefix(a, "-") && a != "-":
+ return fmt.Errorf("unknown flag %s: nothing was written (magpie goroutines [<file>])", a)
+ case a == "-" || strings.TrimSpace(a) == "":
+ return fmt.Errorf("%q is not a file name: nothing was written (magpie goroutines [<file>])", a)
+ case file != "":
+ return errors.New("magpie goroutines takes one file: nothing was written (magpie goroutines [<file>])")
+ default:
+ file = a
+ }
+ }
+ if file == "" {
+ file = "magpie-goroutines-" + time.Now().Format("20060102-150405") + ".txt"
+ }
+ dump, stats, err := fetchGoroutines(gateway.URL())
+ if err != nil {
+ return err
+ }
+ f, err := os.OpenFile(file, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600)
+ if err != nil {
+ if errors.Is(err, os.ErrExist) {
+ return fmt.Errorf("%s is there already: name another file (magpie goroutines <file>)", file)
+ }
+ return err
+ }
+ if _, err := f.Write(dump); err != nil {
+ f.Close()
+ os.Remove(file)
+ return err
+ }
+ if err := f.Close(); err != nil {
+ os.Remove(file)
+ return err
+ }
+ if abs, err := filepath.Abs(file); err == nil {
+ file = abs
+ }
+ fmt.Printf("%s %s (%s)\n", green.Render("saved"), file, bytesSize(uint64(len(dump))))
+ fmt.Print(goroutinesSummary(stats))
+ fmt.Println(muted.Render("Send the file and these lines with the report."))
+ return nil
+}
+
+// fetchGoroutines asks the gateway at base for the dump: its text, checked
+// to be one, and which process it is of.
+func fetchGoroutines(base string) ([]byte, gateway.GoroutineStats, error) {
+ var st gateway.GoroutineStats
+ c := &http.Client{Timeout: goroutinesWait}
+ res, err := c.Get(base + gateway.GoroutinesPath)
+ if err != nil {
+ return nil, st, fmt.Errorf("no magpie answers at %s, so there is nothing to dump: is magpie running? (%v)", base, err)
+ }
+ defer res.Body.Close()
+ body, err := io.ReadAll(res.Body)
+ if err != nil {
+ return nil, st, fmt.Errorf("the dump from %s was cut off: %w", base, err)
+ }
+ if res.StatusCode == http.StatusNotFound {
+ return nil, st, fmt.Errorf("the magpie at %s is older than this one and has no goroutine dump: update it, then run magpie goroutines again", base)
+ }
+ if res.StatusCode != http.StatusOK {
+ var e struct {
+ Error struct {
+ Message string `json:"message"`
+ } `json:"error"`
+ }
+ msg := strings.TrimSpace(string(body))
+ if json.Unmarshal(body, &e) == nil && e.Error.Message != "" {
+ msg = e.Error.Message
+ }
+ return nil, st, fmt.Errorf("the magpie at %s answered %s: %s", base, res.Status, msg)
+ }
+ if !bytes.HasPrefix(body, []byte("goroutine ")) {
+ return nil, st, fmt.Errorf("what %s answered is not a goroutine dump (%d bytes): is it magpie?", base, len(body))
+ }
+ if err := json.Unmarshal([]byte(res.Header.Get(gateway.GoroutineStatsHeader)), &st); err != nil {
+ return nil, st, fmt.Errorf("the magpie at %s sent no process numbers with its dump: %v", base, err)
+ }
+ return body, st, nil
+}
+
+// goroutinesSummary is the lines magpie goroutines prints of st.
+func goroutinesSummary(st gateway.GoroutineStats) string {
+ var b strings.Builder
+ who := st.Program
+ if st.Window {
+ who += ", with a window"
+ }
+ fmt.Fprintf(&b, " process %d (%s) magpie %s, up %s\n", st.PID, who, st.Version, st.Uptime)
+ fmt.Fprintf(&b, " goroutines %d\n", st.Goroutines)
+ return b.String()
+}
diff --git a/internal/cli/goroutines_cli_test.go b/internal/cli/goroutines_cli_test.go
new file mode 100644
index 000000000..634fe9755
--- /dev/null
+++ b/internal/cli/goroutines_cli_test.go
@@ -0,0 +1,128 @@
+package cli
+
+import (
+ "bytes"
+ "net"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "path/filepath"
+ "reflect"
+ "strings"
+ "testing"
+
+ "github.com/yetone/magpie/internal/gateway"
+)
+
+// --help and its other forms print magpie goroutines' usage, ask nothing
+// and write nothing; a flag it doesn't take, "-", or a second file are
+// refused the same way, before the gateway is asked.
+func TestGoroutinesHelpAndBadArgsWriteNothing(t *testing.T) {
+ root, _ := heapDir(t)
+ asked := false
+ gw := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { asked = true }))
+ defer gw.Close()
+ t.Setenv("MAGPIE_ADDR", strings.TrimPrefix(gw.URL, "http://"))
+ before := tree(t, root)
+ for _, args := range [][]string{{"goroutines", "--help"}, {"goroutines", "-h"}, {"goroutines", "help"}, {"goroutines", "out.txt", "--help"}} {
+ out, err := runOut(t, args...)
+ if err != nil || !strings.Contains(out, "usage: magpie goroutines [<file>]") {
+ t.Errorf("magpie %s: %v %q", strings.Join(args, " "), err, out)
+ }
+ }
+ for _, args := range [][]string{{"goroutines", "--out", "x"}, {"goroutines", "-o"}, {"goroutines", "-"}, {"goroutines", "a.txt", "b.txt"}, {"goroutines", " "}} {
+ if _, err := runOut(t, args...); err == nil || !strings.Contains(err.Error(), "nothing was written") {
+ t.Errorf("magpie %s: %v", strings.Join(args, " "), err)
+ }
+ }
+ if asked {
+ t.Error("the gateway was asked")
+ }
+ if got := tree(t, root); !reflect.DeepEqual(got, before) {
+ t.Fatalf("written:\nbefore %v\nafter %v", before, got)
+ }
+}
+
+// Against a running gateway (its Handler, on loopback), magpie goroutines
+// writes the dump into the working folder, prints its path and which
+// process it is of, and never writes over a file there.
+func TestGoroutinesWritesTheRunningGatewaysDump(t *testing.T) {
+ _, dir := heapDir(t)
+ gw := httptest.NewServer(gateway.New().Handler())
+ defer gw.Close()
+ t.Setenv("MAGPIE_ADDR", strings.TrimPrefix(gw.URL, "http://"))
+
+ out, err := runOut(t, "goroutines", "stuck.txt")
+ if err != nil {
+ t.Fatal(err)
+ }
+ file := filepath.Join(dir, "stuck.txt")
+ b, err := os.ReadFile(file)
+ if err != nil {
+ t.Fatal(err)
+ }
+ // this test's own goroutine is in it, waiting on the command
+ if !bytes.HasPrefix(b, []byte("goroutine ")) || !bytes.Contains(b, []byte("TestGoroutinesWritesTheRunningGatewaysDump")) {
+ t.Fatalf("not this process's goroutine dump: %.300q", b)
+ }
+ if info, _ := os.Stat(file); info.Mode().Perm() != 0o600 && os.PathSeparator == '/' {
+ t.Errorf("mode %v", info.Mode())
+ }
+ for _, want := range []string{"stuck.txt", "process", "goroutines"} {
+ if !strings.Contains(out, want) {
+ t.Errorf("printed no %s: %q", want, out)
+ }
+ }
+
+ // the same name again: refused, the first file kept
+ if _, err := runOut(t, "goroutines", "stuck.txt"); err == nil || !strings.Contains(err.Error(), "there already") {
+ t.Fatalf("written over: %v", err)
+ }
+ if again, _ := os.ReadFile(file); !bytes.Equal(again, b) {
+ t.Fatal("the first dump changed")
+ }
+
+ // no name: magpie-goroutines-<date>-<time>.txt here
+ if _, err := runOut(t, "goroutines"); err != nil {
+ t.Fatal(err)
+ }
+ if m, _ := filepath.Glob(filepath.Join(dir, "magpie-goroutines-*.txt")); len(m) != 1 {
+ t.Fatalf("no default file: %v", m)
+ }
+}
+
+// A magpie older than this one answers 404: magpie goroutines says to
+// update it and writes nothing. With no magpie answering, it says so.
+func TestGoroutinesWithAnOlderOrNoGateway(t *testing.T) {
+ _, dir := heapDir(t)
+ old := httptest.NewServer(http.NotFoundHandler())
+ defer old.Close()
+ t.Setenv("MAGPIE_ADDR", strings.TrimPrefix(old.URL, "http://"))
+ if _, err := runOut(t, "goroutines", "x.txt"); err == nil || !strings.Contains(err.Error(), "older than this one") {
+ t.Fatalf("an older magpie: %v", err)
+ }
+
+ // a port kept by the test that hangs up on every call: one freed
+ // could be handed to anything next
+ ln, err := net.Listen("tcp", "127.0.0.1:0")
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer ln.Close()
+ go func() {
+ for {
+ c, err := ln.Accept()
+ if err != nil {
+ return
+ }
+ c.Close()
+ }
+ }()
+ t.Setenv("MAGPIE_ADDR", ln.Addr().String())
+ if _, err := runOut(t, "goroutines", "x.txt"); err == nil || !strings.Contains(err.Error(), "is magpie running") {
+ t.Fatalf("no magpie: %v", err)
+ }
+ if ents, _ := os.ReadDir(dir); len(ents) != 0 {
+ t.Fatalf("written: %v", ents)
+ }
+}
diff --git a/internal/cli/help_cli.go b/internal/cli/help_cli.go
index 20a88cc33..ed4b97c3d 100644
--- a/internal/cli/help_cli.go
+++ b/internal/cli/help_cli.go
@@ -58,6 +58,8 @@ func helpFor(args []string) string {
return heapUsage
case "cpu":
return cpuUsage
+ case "goroutines":
+ return goroutinesUsage
case "sessions":
return sessionsUsage
case "search":
diff --git a/internal/cli/help_cli_test.go b/internal/cli/help_cli_test.go
index 9817617a7..8162eddf6 100644
--- a/internal/cli/help_cli_test.go
+++ b/internal/cli/help_cli_test.go
@@ -129,7 +129,7 @@ func runCommands(t *testing.T) []string {
if len(cmds) < 40 {
t.Fatalf("read %d commands from run's switch: %v", len(cmds), cmds)
}
- return append(cmds, "healthcheck", "heap", "cpu")
+ return append(cmds, "healthcheck", "heap", "cpu", "goroutines")
}
func types(ix *ast.IndexExpr) string {
diff --git a/internal/cli/main.go b/internal/cli/main.go
index 1836268a1..68d929ec7 100644
--- a/internal/cli/main.go
+++ b/internal/cli/main.go
@@ -115,6 +115,7 @@ const usage = `magpie — one place to pick every agent's model
magpie healthcheck exit 0 when the gateway answers (a container's HEALTHCHECK)
magpie heap [<file>] a Go heap profile of the running magpie, with its memory numbers, for a report of it using too much memory
magpie cpu [<file>] [--seconds N] a Go CPU profile of the running magpie over 30s (5..120) while you use it, for a report of it using too much CPU
+ magpie goroutines [<file>] every goroutine of the running magpie, its stack and how long it waited, for a report of it stuck at something
magpie gateway-key list|add <name>|rotate <id>|remove <id> manage the keys clients use to call a shared gateway
magpie gateway-key limit <id> [list|off|reset [n]|remove <n>|[add] day|week|month|<N>d|<N>h --tokens N --cost USD --cache-reads] a key's own limits, and what it used
magpie gateway-key models <id> [all|<provider>/<model>|<provider>/* ...] the models a key may use, every one unless it names some
@@ -231,6 +232,10 @@ func run(args []string) error {
// the running magpie's CPU profile: asks it, and moves nothing here
return cpuCmd(args[1:])
}
+ if len(args) > 0 && args[0] == "goroutines" {
+ // the running magpie's goroutines: asks it, and moves nothing here
+ return goroutinesCmd(args[1:])
+ }
if len(args) == 2 && args[0] == agent.DryRunArg {
// an agent disconnected on a copy of its files under a temporary
// home, for the Agents page to show what disconnecting changes
diff --git a/internal/gateway/cpu_test.go b/internal/gateway/cpu_test.go
index df832c257..ffdeda02c 100644
--- a/internal/gateway/cpu_test.go
+++ b/internal/gateway/cpu_test.go
@@ -36,7 +36,6 @@ func TestCPUProfileOnlyToThisComputer(t *testing.T) {
cpuSecond = time.Millisecond
t.Cleanup(func() { cpuSecond = old })
srv := New()
- chrome := []string{"User-Agent", "Mozilla/5.0 (Macintosh) AppleWebKit/605.1.15 Version/27.2 Safari/605.1.15", "Sec-Fetch-Site", "cross-site", "Sec-Fetch-Mode", "cors"}
call := func(guarded bool, remote, host string, hdr ...string) *httptest.ResponseRecorder {
t.Helper()
r := httptest.NewRequest("GET", CPUPath+"?seconds=5", nil)
@@ -53,26 +52,7 @@ func TestCPUProfileOnlyToThisComputer(t *testing.T) {
h.ServeHTTP(w, r)
return w
}
- refused := []struct {
- name string
- remote, host string
- hdr []string
- }{
- {"a foreign page", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "https://evil.example"}, chrome...)},
- {"Origin null", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "null"}, chrome...)},
- {"Origin null with a key", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "null", "Authorization", "Bearer " + key}, chrome...)},
- {"a page on localhost", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "http://localhost:5173"}, chrome...)},
- {"a localhost subdomain", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "http://evil.localhost"}, chrome...)},
- {"a listed page with its key", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "https://listed.example", "Authorization", "Bearer " + key}, chrome...)},
- {"a rebound page's GET", "127.0.0.1:50123", "rebind.attacker.example:3425", chrome},
- {"a rebound page with a key", "127.0.0.1:50123", "rebind.attacker.example:3425", append([]string{"Authorization", "Bearer " + key}, chrome...)},
- {"a call to another name", "127.0.0.1:50123", "rebind.attacker.example:3425", []string{"User-Agent", "curl/8"}},
- {"a localhost subdomain as Host", "127.0.0.1:50123", "wails.localhost:3425", []string{"User-Agent", "curl/8"}},
- {"another machine", "192.168.1.20:50123", "192.168.1.10:3425", nil},
- {"another machine with a key", "192.168.1.20:50123", "192.168.1.10:3425", []string{"Authorization", "Bearer " + key}},
- {"a proxy's client", "127.0.0.1:50123", "127.0.0.1:3425", []string{"X-Forwarded-For", "203.0.113.9"}},
- {"a proxy's client with a key", "127.0.0.1:50123", "127.0.0.1:3425", []string{"Cf-Connecting-IP", "203.0.113.9", "Authorization", "Bearer " + key}},
- }
+ refused := profileCallersRefused(key)
for _, c := range refused {
for _, guarded := range []bool{true, false} {
w := call(guarded, c.remote, c.host, c.hdr...)
diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.go
index 7a1e426d4..9668dc59f 100644
--- a/internal/gateway/gateway.go
+++ b/internal/gateway/gateway.go
@@ -699,6 +699,9 @@ func (s *Server) Handler() http.Handler {
// a CPU profile of this process over some seconds, for magpie cpu
// (loopback only)
mux.HandleFunc("GET "+CPUPath, s.cpuProfile)
+ // every goroutine's stack and wait, for magpie goroutines (loopback
+ // only)
+ mux.HandleFunc("GET "+GoroutinesPath, s.goroutineDump)
mux.HandleFunc("GET /v1/magpie/omarchy", s.omarchyTheme)
mux.HandleFunc("OPTIONS /v1/magpie/omarchy", s.omarchyTheme)
mux.HandleFunc("POST /v1/chat/completions", s.handle(provider.Chat))
diff --git a/internal/gateway/goroutines.go b/internal/gateway/goroutines.go
new file mode 100644
index 000000000..5887b25ee
--- /dev/null
+++ b/internal/gateway/goroutines.go
@@ -0,0 +1,82 @@
+package gateway
+
+// GET /v1/magpie/goroutines: every goroutine of the process serving the
+// gateway, with its stack and how long it has waited, for a report of
+// magpie stuck at something while it goes on running. An app that kept
+// the partner list it had for three days could only be read by killing
+// it, which ends what was stuck (#3362); `magpie goroutines` asks the
+// running process instead and writes the file the user sends.
+//
+// The dump is runtime/pprof's "goroutine" profile at debug=2, the form a
+// crash prints: each goroutine's state, its wait ("[select, 4320
+// minutes]"), its frames with source lines, and who started it. A frame's
+// arguments are words in hex, never what they point at, so it holds no
+// keys, prompts or messages; it names the source files magpie was built
+// from and the program's path.
+
+import (
+ "bytes"
+ "encoding/json"
+ "log"
+ "net/http"
+ "os"
+ "path/filepath"
+ "runtime"
+ "runtime/pprof"
+ "time"
+
+ "github.com/yetone/magpie/internal/provider"
+)
+
+// GoroutinesPath is where the gateway answers a goroutine dump.
+const GoroutinesPath = "/v1/magpie/goroutines"
+
+// GoroutineStatsHeader carries GoroutineStats, as JSON, beside the dump.
+const GoroutineStatsHeader = "X-Magpie-Goroutine-Stats"
+
+// GoroutineStats says which process the dump is of.
+type GoroutineStats struct {
+ PID int `json:"pid"`
+ Program string `json:"program"`
+ Version string `json:"version"`
+ Window bool `json:"window"` // the app's window or magpie web serves the gateway
+ Uptime string `json:"uptime"`
+ Goroutines int `json:"goroutines"`
+}
+
+// goroutinesRefused says why r may not have the goroutine dump, "" when it
+// may: the same callers as the heap profile (heapRefused).
+func goroutinesRefused(r *http.Request) string {
+ return profileRefused(r, "goroutine dump", "magpie goroutines")
+}
+
+// goroutineDump answers GoroutinesPath: the dump as text, with
+// GoroutineStats in GoroutineStatsHeader.
+func (s *Server) goroutineDump(w http.ResponseWriter, r *http.Request) {
+ if why := goroutinesRefused(r); why != "" {
+ log.Printf("refused %s %s from %s: %s", r.Method, r.URL.Path, r.RemoteAddr, why)
+ writeError(w, provider.Chat, http.StatusForbidden, why)
+ return
+ }
+ goroutines := runtime.NumGoroutine()
+ var buf bytes.Buffer
+ if err := pprof.Lookup("goroutine").WriteTo(&buf, 2); err != nil {
+ writeError(w, provider.Chat, http.StatusInternalServerError, "the goroutine dump couldn't be written: "+err.Error())
+ return
+ }
+ exe, _ := os.Executable()
+ stats, _ := json.Marshal(GoroutineStats{
+ PID: os.Getpid(),
+ Program: filepath.Base(exe),
+ Version: Version,
+ Window: Window,
+ Uptime: time.Since(processStarted).Round(time.Second).String(),
+ Goroutines: goroutines,
+ })
+ h := w.Header()
+ h.Set("Content-Type", "text/plain; charset=utf-8")
+ h.Set("Content-Disposition", `attachment; filename="magpie-goroutines.txt"`)
+ h.Set("Cache-Control", "no-store")
+ h.Set(GoroutineStatsHeader, string(stats))
+ w.Write(buf.Bytes())
+}
diff --git a/internal/gateway/goroutines_test.go b/internal/gateway/goroutines_test.go
new file mode 100644
index 000000000..0e5607ff7
--- /dev/null
+++ b/internal/gateway/goroutines_test.go
@@ -0,0 +1,102 @@
+package gateway
+
+import (
+ "bytes"
+ "encoding/json"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "strings"
+ "testing"
+
+ "github.com/yetone/magpie/internal/settings"
+)
+
+// parkedForTheDump waits on ch, as a goroutine stuck in magpie would: the
+// dump must name it and what it waits on.
+func parkedForTheDump(ch <-chan struct{}, parked chan<- struct{}) {
+ close(parked)
+ <-ch
+}
+
+// The goroutine dump goes to the same callers as the heap profile (a
+// program on this computer over loopback, nothing a browser sends, no
+// other Host) and is refused to the rest. What it answers is every
+// goroutine's stack, a goroutine parked in magpie's code among them,
+// named with what it waits on, and which process it is of (#3362: an app
+// stuck for days could only be read by killing it).
+func TestGoroutineDumpOnlyToThisComputer(t *testing.T) {
+ fresh(t)
+ t.Setenv("MAGPIE_ADDR", "")
+ t.Setenv("MAGPIE_PUBLIC_URL", "")
+ _, secrets := newCaller(t, "Page")
+ s := settings.Load()
+ s.CORSOrigins = []string{"https://listed.example"}
+ if err := settings.Save(s); err != nil {
+ t.Fatal(err)
+ }
+ key := secrets[0]
+ srv := New()
+ call := func(guarded bool, remote, host string, hdr ...string) *httptest.ResponseRecorder {
+ t.Helper()
+ r := httptest.NewRequest("GET", GoroutinesPath, nil)
+ r.RemoteAddr = remote
+ r.Host = host
+ for i := 0; i+1 < len(hdr); i += 2 {
+ r.Header.Set(hdr[i], hdr[i+1])
+ }
+ w := httptest.NewRecorder()
+ h := srv.Handler()
+ if guarded {
+ h = lanGuard(h)
+ }
+ h.ServeHTTP(w, r)
+ return w
+ }
+ for _, c := range profileCallersRefused(key) {
+ for _, guarded := range []bool{true, false} {
+ w := call(guarded, c.remote, c.host, c.hdr...)
+ // lanGuard asks another machine without a key for one (401)
+ if w.Code != http.StatusForbidden && w.Code != http.StatusUnauthorized || w.Header().Get(GoroutineStatsHeader) != "" || bytes.HasPrefix(w.Body.Bytes(), []byte("goroutine ")) {
+ t.Errorf("%s (lanGuard %v): %d %q", c.name, guarded, w.Code, w.Body.String())
+ }
+ }
+ }
+
+ release, parked := make(chan struct{}), make(chan struct{})
+ defer close(release)
+ go parkedForTheDump(release, parked)
+ <-parked
+ for _, host := range []string{"127.0.0.1:3425", "localhost:3425", "[::1]:3425"} {
+ w := call(true, "127.0.0.1:50123", host, "User-Agent", "Go-http-client/1.1")
+ if w.Code != http.StatusOK {
+ t.Fatalf("magpie goroutines at %s: %d %q", host, w.Code, w.Body.String())
+ }
+ body := w.Body.String()
+ if !strings.HasPrefix(body, "goroutine ") {
+ t.Fatalf("at %s: not a goroutine dump: %.200q", host, body)
+ }
+ // the parked one, with its state and its frame in this file
+ i := strings.Index(body, "gateway.parkedForTheDump(")
+ if i < 0 {
+ t.Fatalf("at %s: the parked goroutine isn't in the dump", host)
+ }
+ head := body[strings.LastIndex(body[:i], "goroutine "):i]
+ if !strings.Contains(head, "[chan receive") {
+ t.Errorf("at %s: the parked goroutine's state isn't its wait: %q", host, head)
+ }
+ if !strings.Contains(body[i:], "goroutines_test.go:") {
+ t.Errorf("at %s: no source line for the parked goroutine", host)
+ }
+ var st GoroutineStats
+ if err := json.Unmarshal([]byte(w.Header().Get(GoroutineStatsHeader)), &st); err != nil {
+ t.Fatalf("at %s: stats %q: %v", host, w.Header().Get(GoroutineStatsHeader), err)
+ }
+ if st.PID != os.Getpid() || st.Goroutines < 2 || st.Uptime == "" {
+ t.Errorf("at %s: stats %+v", host, st)
+ }
+ if !strings.Contains(w.Header().Get("Cache-Control"), "no-store") || !strings.HasPrefix(w.Header().Get("Content-Type"), "text/plain") {
+ t.Errorf("at %s: headers %v", host, w.Header())
+ }
+ }
+}
diff --git a/internal/gateway/heap_test.go b/internal/gateway/heap_test.go
index 1bdf41c57..59838466c 100644
--- a/internal/gateway/heap_test.go
+++ b/internal/gateway/heap_test.go
@@ -33,7 +33,6 @@ func TestHeapProfileOnlyToThisComputer(t *testing.T) {
}
key := secrets[0]
srv := New()
- chrome := []string{"User-Agent", "Mozilla/5.0 (Macintosh) AppleWebKit/605.1.15 Version/27.2 Safari/605.1.15", "Sec-Fetch-Site", "cross-site", "Sec-Fetch-Mode", "cors"}
call := func(guarded bool, remote, host string, hdr ...string) *httptest.ResponseRecorder {
t.Helper()
r := httptest.NewRequest("GET", HeapPath, nil)
@@ -50,26 +49,7 @@ func TestHeapProfileOnlyToThisComputer(t *testing.T) {
h.ServeHTTP(w, r)
return w
}
- refused := []struct {
- name string
- remote, host string
- hdr []string
- }{
- {"a foreign page", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "https://evil.example"}, chrome...)},
- {"Origin null", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "null"}, chrome...)},
- {"Origin null with a key", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "null", "Authorization", "Bearer " + key}, chrome...)},
- {"a page on localhost", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "http://localhost:5173"}, chrome...)},
- {"a localhost subdomain", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "http://evil.localhost"}, chrome...)},
- {"a listed page with its key", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "https://listed.example", "Authorization", "Bearer " + key}, chrome...)},
- {"a rebound page's GET", "127.0.0.1:50123", "rebind.attacker.example:3425", chrome},
- {"a rebound page with a key", "127.0.0.1:50123", "rebind.attacker.example:3425", append([]string{"Authorization", "Bearer " + key}, chrome...)},
- {"a call to another name", "127.0.0.1:50123", "rebind.attacker.example:3425", []string{"User-Agent", "curl/8"}},
- {"a localhost subdomain as Host", "127.0.0.1:50123", "wails.localhost:3425", []string{"User-Agent", "curl/8"}},
- {"another machine", "192.168.1.20:50123", "192.168.1.10:3425", nil},
- {"another machine with a key", "192.168.1.20:50123", "192.168.1.10:3425", []string{"Authorization", "Bearer " + key}},
- {"a proxy's client", "127.0.0.1:50123", "127.0.0.1:3425", []string{"X-Forwarded-For", "203.0.113.9"}},
- {"a proxy's client with a key", "127.0.0.1:50123", "127.0.0.1:3425", []string{"Cf-Connecting-IP", "203.0.113.9", "Authorization", "Bearer " + key}},
- }
+ refused := profileCallersRefused(key)
for _, c := range refused {
for _, guarded := range []bool{true, false} {
w := call(guarded, c.remote, c.host, c.hdr...)
@@ -111,3 +91,32 @@ func TestHeapProfileOnlyToThisComputer(t *testing.T) {
}
}
}
+
+// profileRefusal is a caller a profile of this process is refused to.
+type profileRefusal struct {
+ name string
+ remote, host string
+ hdr []string
+}
+
+// profileCallersRefused are the callers every profile of this process
+// (heap, CPU, goroutines) is refused to, key a gateway key they may carry.
+func profileCallersRefused(key string) []profileRefusal {
+ chrome := []string{"User-Agent", "Mozilla/5.0 (Macintosh) AppleWebKit/605.1.15 Version/27.2 Safari/605.1.15", "Sec-Fetch-Site", "cross-site", "Sec-Fetch-Mode", "cors"}
+ return []profileRefusal{
+ {"a foreign page", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "https://evil.example"}, chrome...)},
+ {"Origin null", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "null"}, chrome...)},
+ {"Origin null with a key", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "null", "Authorization", "Bearer " + key}, chrome...)},
+ {"a page on localhost", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "http://localhost:5173"}, chrome...)},
+ {"a localhost subdomain", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "http://evil.localhost"}, chrome...)},
+ {"a listed page with its key", "127.0.0.1:50123", "127.0.0.1:3425", append([]string{"Origin", "https://listed.example", "Authorization", "Bearer " + key}, chrome...)},
+ {"a rebound page's GET", "127.0.0.1:50123", "rebind.attacker.example:3425", chrome},
+ {"a rebound page with a key", "127.0.0.1:50123", "rebind.attacker.example:3425", append([]string{"Authorization", "Bearer " + key}, chrome...)},
+ {"a call to another name", "127.0.0.1:50123", "rebind.attacker.example:3425", []string{"User-Agent", "curl/8"}},
+ {"a localhost subdomain as Host", "127.0.0.1:50123", "wails.localhost:3425", []string{"User-Agent", "curl/8"}},
+ {"another machine", "192.168.1.20:50123", "192.168.1.10:3425", nil},
+ {"another machine with a key", "192.168.1.20:50123", "192.168.1.10:3425", []string{"Authorization", "Bearer " + key}},
+ {"a proxy's client", "127.0.0.1:50123", "127.0.0.1:3425", []string{"X-Forwarded-For", "203.0.113.9"}},
+ {"a proxy's client with a key", "127.0.0.1:50123", "127.0.0.1:3425", []string{"Cf-Connecting-IP", "203.0.113.9", "Authorization", "Bearer " + key}},
+ }
+}