Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

gateway: a request to an unmasked route isn't masked first (#3973)

yetoneea53eb1262ed
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index 88bdcbc5a..24a0bb063 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -40,7 +40,7 @@ backfilled; see [Gateway Session Aggregation](gateway-sessions.md).
| Affinity | A conversation stays with the key or account that answered it, so the vendor's prompt cache is read again. Modes: auto (""), `session`, `turn`, `off`, `strict`. Stored in `affinity.json` next to `providers.json`. A conversation is kept by its id (`conversationID`), and a Claude Code subagent's apart from its lead's by `x-claude-code-agent-id` (`stickKey`): Claude Code sends the session's id with its subagents' requests too, and a subagent's answer took the lead's stick over, so the lead's next turn was judged by the subagent's cache and moved to another account, where its whole conversation was written again (slkassinthehood on X, `TestClaudeConversationKeepsItsAccountsCache`). Behind a remote magpie, the key or account is the remote's key and the provider there that serves the model, its id's first part (`remoteUpstream`, `candidate.vendor`, `sameVendor`): the remote is one key here, but each of its providers keeps its own cache and seals, so a conversation is kept, held strictly, told its sealed reasoning was refused and given its native compaction's reader by that pair, as the same group on the remote does (fremontdango on Discord: a group of one model from two of the remote's providers kept a conversation on the second as the same account when the first was gone, and a strict one failed over to it; `TestRemoteMagpieStrictKeepsTheRemotesProvider`, `TestRemoteMagpieOtherProviderIsNotTheSameAccount`). A remote's group is one vendor by its whole id, its own routing deciding inside it; a model named without a provider is the key alone. Lanes and requests a minute still go by the key here, the limits the user set on it. | [`affinity.go`](../../internal/gateway/affinity.go) |
| One try | `attempt` sends one candidate the request. It goes through the proxy of the account or key it uses, else the provider's (`p.Via`, `ProxyChoice`: `AccountProxies` holds an account's by its name, a key's by its `KeyID`). A request magpie makes for one provider while serving another's (a web search, an image description, a classifier call) takes its own provider's proxy or the global one, never the outer one: `netproxy.With` replaces a choice the context already names, "" too, and `webSearch` starts from none, so the search APIs follow the global proxy (#1522). A provider of the user's set not to check its certificate (`InsecureTLS`, #2383) has it skipped for its own base URLs' hosts alone, through `p.Via` ([Providers and accounts](providers-accounts.md)). Every request to one vendor's host shares one HTTP/2 connection (`upstreamTransport`); one with no frame for 20s is pinged, and dropped when no answer comes within 10s (`netproxy.HTTP2Health`, on `http.DefaultTransport` too), so a connection gone silent without closing (a proxy's way out died, a NAT forgot the flow) fails the requests on it, which move on as a 502 does, and the next request dials a new one. A live vendor answers a ping whatever its streams are doing, so a long wait for a first byte is not cut by it (#2843: a group's requests hung with no first byte until a restart; `TestGroupMovesOnFromSilentHTTP2Connection`). A subscription with its own client is served specially, such as Claude through the genuine Claude Code. The request is relayed as is (`passthrough`) when the model is usable on the agent's API, except that a Claude model where the provider has an Anthropic URL is translated to Messages from Chat or Responses too (`OnMessages`: OpenCode speaks Chat for every model, and a relay drops cache_control there), unless the user set the model's API or the vendor's list names its APIs. An account that only streams (`Account.Stream`: a plugin's, and most subscriptions magpie signs in to) has a request that asked for one body translated, which magpie always asks streamed, except a plugin's on Responses (`gathers`): that is relayed all the same, asked with `stream:true`, and the client given the `response` of the stream's `response.completed` (or `response.incomplete`) as one JSON body (`gathered` in [`gathered.go`](../../internal/gateway/gathered.go); its `output` filled from the stream's `response.output_item.done` items, or else its text, when that event's is empty). So the tools the vendor runs itself, and its response's `tools` and usage details, reach the client (plugins#112: a translation dropped Grok's `x_search`, and Grok answered without searching; `TestGrokPluginNonStreamKeepsHostedTools`). A stream that fails or ends before that event is a 502 (`TestGrokPluginNonStreamEndedShort`), never an empty 200. A Messages request is still translated, its stream having no event that holds the whole reply, and so is a subscription magpie signs in to itself (the ChatGPT backend's, Qoder's): its own code fits its requests. A request offering web search to a provider that doesn't search by itself is translated too, so magpie can search for it (`search.go`; a provider that searches for some of its models only, Command Code's Provider API, lists them in `searchModels` by API, and none count under its `x-cmd-zdr` header, #1590), but on the agent's own API where the provider serves it and no `ResponsesFirst`/`MessagesFirst` rule prefers another: Codex offers web_search on every turn, and each went out as Chat (#997). Otherwise it is translated (`translate`) to the first API `usable` gives: Chat, except Responses for OpenAI's models where their maker serves them (`ResponsesFirst`), Responses for every model at ModelScope, whose Chat API reports neither the tokens its cache answered nor reasoning while its Responses API reports both (`ResponsesTellsCache`, #1578; a 404 there sends the model to Chat, and only there), and Anthropic's Messages for a Claude model where the provider has an Anthropic URL (`MessagesFirst`, which keeps its cache breakpoints, #997). An API that answers the model is not served there (`wrongEndpoint`, `catalog.WrongEndpoint`: the English wordings and AgentRouter's Chinese one, #3766, read escaped too; a model's test reads them as well, see [Providers and accounts](providers-accounts.md)) is remembered in `unfit` until magpie restarts. A Chat API that refuses function tools with reasoning, with no other API to take them to, is asked once more with `reasoning_effort` `none` (#176), unless `none` is known refused (`turnedAway`); `none` refused naming the levels, the tools refusal is the answer, not a retry at the lowest level (#2670). When every API has turned the model away, the API the user picked for it (`ModelAPI`) is asked again, and without a pick each API the provider has a URL for (`Speaks`), Chat first. `Provider.Native`, the API agents with a per-model API (Pi, omp, droid) are wired to ask on, follows the same order. A custom provider with a Gemini URL (#1346) has Gemini last among the APIs it speaks: a Gemini client's request is relayed there, and any other is translated to it when it has no other, or when the user set Gemini as the model's API (`SetModelAPI` takes chat, responses, anthropic or gemini, each only where the provider has its URL; a model set to Gemini has no `Native`, so those agents ask it on their own API and it is translated). Upstream it is always asked `models/{model}:streamGenerateContent?alt=sse` under the base (`upstreamPath`), the model only in the path (`forwardOnce` drops the body's), so a Gemini client that asked for one JSON body is translated, as for Factory's generate route. | [`gateway.go`](../../internal/gateway/gateway.go) |
| Trace | The Routing view's record of each decision, written where the decision is made: candidates, rests, which try answered. `Route.Folder` is the folder the agent says it works in (`workDir`), its last two levels only (`shortFolder`, e.g. `workspace/magpie`), never the full path; a request that says none (a title, a topic check) takes its session's as said last (`trace.sessionFolder`), one without a session id has none (#1068). It is kept in the routing history on disk for as long as the route is (`history.go`: 30 days, 200 MB). The GUI shows it in By session's headings, a request row's tooltip, the stage's agent working in more than one folder, and the tray panel's agent tooltip. | [`trace.go`](../../internal/gateway/trace.go) |
-| Prompt | What each request's prompt held, part by part: the agent's system prompt, tool definitions (an MCP server's tools as one item), instruction files and skills, files its tools read, their other output, and the conversation. `inspectPrompt` reads it from the body as the agent sent it, before translation; `calibrated` scales the estimate to the prompt the vendor counted and sets the window of the model that answered. The route keeps names and sizes only, never the text, so the routing history on disk holds none. For the card's text (masaka on Discord), the trace keeps the last `promptBodies` (10) request bodies in memory only (at most 32 MiB each, 64 MiB together), masked as the vendor got them; for a model let have it unmasked, the masked copy is kept all the same. `PromptText` reads a part's items and their text from that body again (`prompt_text.go`), served to the GUI as `/api/context/text`; once a body is dropped, or magpie quits, its text is gone. It goes on the trace as `Route.Prompt`, with `Route.Conv` naming a conversation that has no session id. The GUI's `/api/context` scores each agent from it (`internal/gui/context.go`). A session's line and its compactions are its own turns': a call with a `Kind`, a Claude Code subagent's (`Route.Sub`, from `x-claude-code-agent-id`), and, in a session whose turns carry tool definitions, a request without any (auto mode's classifier, the title, the quota check) are left out, so a classifier's 40K after a 100K turn is no compaction (`toolSessions`, `TestContextSideCallsAreNoCompaction`). | [`prompt.go`](../../internal/gateway/prompt.go) |
+| Prompt | What each request's prompt held, part by part: the agent's system prompt, tool definitions (an MCP server's tools as one item), instruction files and skills, files its tools read, their other output, and the conversation. `inspectPrompt` reads it from the body as the agent sent it, before translation; `calibrated` scales the estimate to the prompt the vendor counted and sets the window of the model that answered. The route keeps names and sizes only, never the text, so the routing history on disk holds none. For the card's text (masaka on Discord), the trace keeps the last `promptBodies` (10) request bodies in memory only (at most 32 MiB each, 64 MiB together), masked as the vendor got them; for a model let have it unmasked, the body is kept as it went and masked when `PromptText` first reads it, so the request itself never waits on masking (#3973). `PromptText` reads a part's items and their text from that body again (`prompt_text.go`), served to the GUI as `/api/context/text`; once a body is dropped, or magpie quits, its text is gone. It goes on the trace as `Route.Prompt`, with `Route.Conv` naming a conversation that has no session id. The GUI's `/api/context` scores each agent from it (`internal/gui/context.go`). A session's line and its compactions are its own turns': a call with a `Kind`, a Claude Code subagent's (`Route.Sub`, from `x-claude-code-agent-id`), and, in a session whose turns carry tool definitions, a request without any (auto mode's classifier, the title, the quota check) are left out, so a classifier's 40K after a 100K turn is no compaction (`toolSessions`, `TestContextSideCallsAreNoCompaction`). | [`prompt.go`](../../internal/gateway/prompt.go) |
| Vertex AI | A [Google Vertex AI provider](providers-accounts.md#responsibilities-and-sources-of-truth) speaks Gemini alone. A request goes to `<base>/publishers/google/models/<model>:streamGenerateContent?alt=sse` (`provider.VertexPath`, `upstreamPath`), the model named as the user's upstream name for it says (`UpstreamNameIn`, which the usage log's endpoint names too), with no `model` or `stream` field in its body (`passthrough`, `forwardOnce`, `geminiRequest`). A Gemini client's request is relayed as it came, its own `thinkingConfig`, `safetySettings` and the rest kept: streamed there, asked whole to `:generateContent` (`provider.VertexWholePath`), which answers one JSON body (Factory's rule of translating a whole request, as its route only streams, isn't Vertex AI's). Every other API's is translated (`buildFor`, `buildVertex`) and its stream read by `codeAssistDecoder`, which gives thought parts as reasoning, never as the answer's text. Effort: `vertexEffort` fits it to the model's levels, reasoning turned off going as `minimal` fitted the same way, and holds a model whose levels aren't known to `low`…`high`. `vertexThinking` sends Gemini 3 a `thinkingLevel` (`includeThoughts` false with reasoning off) and 2.5 a `thinkingBudget` in the model's range (`vertexBudgets`): turned off, the budget that stops its thinking, or Pro's least, as Pro can't stop. Thought signatures are every Gemini path's (#1445): a call's signature reaches the client in its id (`signedID`) and goes back on the call it came on, while a call with none in a step Vertex AI signed goes without; a step with nothing to give back has every call sent `skip_thought_signature_validator`. A 400 `Invalid thought signature.` is asked again with every call sent so (`withoutSignatures`), for that request only. A 400 refusing `minimal` (`geminiRefusedLevel`) is asked again at the model's lowest other level (`onEffort`), and `markUnfit(offRefused)` keeps that level until magpie restarts. Both retries are the translated path's. Vertex AI's own Google Search grounding isn't used: `searchHosts` names only `generativelanguage.googleapis.com` for Gemini, so magpie searches for it (`search.go`). | [`vertex.go`](../../internal/gateway/vertex.go), [`gateway.go`](../../internal/gateway/gateway.go), [`codeassist.go`](../../internal/gateway/codeassist.go), [`gemini.go`](../../internal/gateway/gemini.go), [`gemini_signature.go`](../../internal/gateway/gemini_signature.go), [`search.go`](../../internal/gateway/search.go) |
| Bedrock Mantle | A request on chat completions or Responses to an [Amazon Bedrock Mantle provider](providers-accounts.md#responsibilities-and-sources-of-truth) goes to the prefix that serves its model (`mantleRoutes`, `provider.MantleBase` in `forwardOnce`): `/openai/v1` for OpenAI's closed GPTs and xAI's Grok, `/v1` for the rest. A model Mantle refuses there with "model `x` isn't supported on this route" (a maker the rule doesn't know yet) is asked once more at the other prefix, and when it is taken there that prefix is remembered for the model in `unfit` (`mantleRouteKey`) until the gateway restarts. A model refused at both gets the first refusal and nothing is remembered (`mantleReroute`). Messages go to `/anthropic` alone. | [`mantle.go`](../../internal/gateway/mantle.go), [`gateway.go`](../../internal/gateway/gateway.go) |
@@ -137,7 +137,7 @@ recorded as a prediction failure (`noPrediction`).
## Runtime path
1. `handle(proto)` reads the body and the model (`requestBody`, `requestModel`), then calls `serveAgent`. Every route that reads an agent's body (each API, count_tokens, Gemini's, embeddings, images, videos, System One, the Codex backend path) reads it through `readBoundedRequestBody` in [`request_bounds.go`](../../internal/gateway/request_bounds.go). It decodes a `Content-Encoding` of `gzip` or `zstd` (`decodedBody`; Codex sends zstd to a custom provider's `/v1/responses` too, #1223) and drops the header, so the provider is sent plain JSON, with one exception: a ChatGPT account's turn (POST `CodexBase`/responses, from the accounts loop in `forwardOnce` or relayed by `codexUpstreamOn`) is compressed again with zstd and a window of at most 2 MiB, as Codex's own client sends it (`provider.CodexCompressed`; openai/codex `responses_request_compression` and `http-client/src/request.rs`). It is compressed once the body is final, and again on each try. `/responses/compact`, an API key's requests and every other provider stay plain. Sent plain, a 1.1 MB text-only turn was answered 503 (ylorn on X). The plain body is what magpie keeps and shows, and `codexDropped` judges a reset by the bytes that went on the wire. `codexDropped` also reads Envoy's 507 "exceeded request buffer limit while retrying upstream" (the backend failed and the front couldn't retry a body that large) as that reset, at any size, told as a 503 so magpie, Codex and pi ask again and nobody rests (#2034). The size limit (`defaultBodyLimit`, 512 MB) holds for the decoded bytes (413), and any other encoding is refused 415. It was 128 MB until a long Codex session that had read many images went past it decoded, its inline screenshots resent each turn, and got the 413 on all five of Codex's retries (#1422, `TestImageHeavyCodexTurnIsRead`). magpie's own 413 says it is magpie's gateway limit, with the size, and that the provider was not asked (`overLimit`); `writeError` adds to any 413 it isn't holding that this is a byte limit, not the model's context, and to `/compact` or start a new conversation (`TestOverLimitSaysWhoseAndWhatToDo`). Every shape but Gemini's types a 413 `request_too_large`, Anthropic's own type, which pi's overflow table knows: pi compacts and tries once more rather than stopping on an OpenAI-shaped `api_error` (@wilsonxuuuu on X, `TestPayloadTooLargeSaysRequestTooLargeToPi`). `keyLimited` reads a limited key's compressed body the same way, so its reservation is the decoded size. `serveAgent` runs the [middleware](gateway-middleware.md) chain, then `serve`.
-2. `serve` redacts secrets in the request (`redacted`; `redact.MaskJSON` leaves ids alone, and in any text a backend's id (a UUID, or 32 hex digits or more), or an element token cua-driver hands the model for computer use (`s0000002a:14`, #3782), goes as written: no rule, model label or value known from before masks a piece of one (a phone number at the end of `…-a13800138000`, a card number across a UUID's groups), and a whole one is masked only by the user's own words and rules or as a credential, by the password, API key and token rules or a secret rule whose field is named for one (`api_key=`, `refresh_token`), never one named for an id, a cursor, a request or a digest (`secret_id`, `next_page_token`, `Idempotency-Key`, `secret_hash`), nor cua-driver's own `element_token` (any other name with `element` in it, `element_api_key`, is a credential's as before), a bare `key`, a serial number, or what the model takes a UUID or an element token for, nor an element token known from before (one the model labelled in prose a turn earlier) (`noIDs`, [`internal/redact/ids.go`](../../internal/redact/ids.go)); and a tool's name and its MCP server's (`namespace`, `server_label`, `server_name`, #3143); the request's tool definitions and output schemas, found by their place in each protocol (top-level `tools`, `tool_choice`, `response_format`, `text`, `generationConfig`, `stream_options`, the client's ids in `client_metadata` and the like, and the tools of a Responses `additional_tools` or `tool_search_output` item), go as written save their descriptions, so a word the user masks can't break a grammar or an enum (#1911); and what the vendor sealed with a signature, a thinking block, Gemini's thoughts or Responses' encrypted reasoning, goes back as the vendor wrote it: only the values masked elsewhere in the request, and secrets masked before, are put back as their placeholders; a value masked once, at least 8 characters long, is masked again wherever a request has it, as an exact match whatever words are around it, so what the vendor wrote about a placeholder doesn't go back with the value in it, `redact/known.go`; a text masked for a vendor before, such as a system prompt or a turn already sent, is masked as it was then, so a value a rule first finds later in a conversation doesn't change the prompt the vendor has cached and rewrite all of it to the cache, `sentKnownSpans`), resolves the model to a provider or a group, and refuses what can't be served. The reply has the values put back on its way to the agent (`redact.Writer`, a stream event by event, a whole body at its end): a placeholder the model wrote as it was given, or as models rewrite one (spaces inside the braces, a brace or both dropped, another case), is the value again when what is left names one magpie made, and one split across events is held until it is whole (`redact.Restore`, `partialTail`, [`internal/redact/redact.go`](../../internal/redact/redact.go)). A request whose provider and fallbacks, or whose group's models, all skip redaction (`unredactedRoute`, `provider.SkipsRedaction`: set *Send requests unmasked*, `Unredacted`, with every address on this machine or the local network) goes to them as the agent wrote it; the log and recent calls keep the masked body. count_tokens and embeddings follow the same rule. What a routing group's Jev is told of the turn (the message, the user's message before it and the reply to it) is masked all the same unless Jev's own provider skips redaction too (`maskedState` in `askJev`, [`decide.go`](../../internal/gateway/decide.go)): a remote Jev is none of the group's members. The masking is the built-in plugin `redact`'s (`gwmw.Masker`, [Plugin runtime](plugin-runtime.md)); when the settings ask for masking and no masker serves, or it fails, the request is a 503 and nothing is sent (`redacted`, `gwmw.Masking`).
+2. `serve` redacts secrets in the request (`redacted`; `redact.MaskJSON` leaves ids alone, and in any text a backend's id (a UUID, or 32 hex digits or more), or an element token cua-driver hands the model for computer use (`s0000002a:14`, #3782), goes as written: no rule, model label or value known from before masks a piece of one (a phone number at the end of `…-a13800138000`, a card number across a UUID's groups), and a whole one is masked only by the user's own words and rules or as a credential, by the password, API key and token rules or a secret rule whose field is named for one (`api_key=`, `refresh_token`), never one named for an id, a cursor, a request or a digest (`secret_id`, `next_page_token`, `Idempotency-Key`, `secret_hash`), nor cua-driver's own `element_token` (any other name with `element` in it, `element_api_key`, is a credential's as before), a bare `key`, a serial number, or what the model takes a UUID or an element token for, nor an element token known from before (one the model labelled in prose a turn earlier) (`noIDs`, [`internal/redact/ids.go`](../../internal/redact/ids.go)); and a tool's name and its MCP server's (`namespace`, `server_label`, `server_name`, #3143); the request's tool definitions and output schemas, found by their place in each protocol (top-level `tools`, `tool_choice`, `response_format`, `text`, `generationConfig`, `stream_options`, the client's ids in `client_metadata` and the like, and the tools of a Responses `additional_tools` or `tool_search_output` item), go as written save their descriptions, so a word the user masks can't break a grammar or an enum (#1911); and what the vendor sealed with a signature, a thinking block, Gemini's thoughts or Responses' encrypted reasoning, goes back as the vendor wrote it: only the values masked elsewhere in the request, and secrets masked before, are put back as their placeholders; a value masked once, at least 8 characters long, is masked again wherever a request has it, as an exact match whatever words are around it, so what the vendor wrote about a placeholder doesn't go back with the value in it, `redact/known.go`; a text masked for a vendor before, such as a system prompt or a turn already sent, is masked as it was then, so a value a rule first finds later in a conversation doesn't change the prompt the vendor has cached and rewrite all of it to the cache, `sentKnownSpans`), resolves the model to a provider or a group, and refuses what can't be served. The reply has the values put back on its way to the agent (`redact.Writer`, a stream event by event, a whole body at its end): a placeholder the model wrote as it was given, or as models rewrite one (spaces inside the braces, a brace or both dropped, another case), is the value again when what is left names one magpie made, and one split across events is held until it is whole (`redact.Restore`, `partialTail`, [`internal/redact/redact.go`](../../internal/redact/redact.go)). A request whose provider and fallbacks, or whose group's models, all skip redaction (`unredactedRoute`, `provider.SkipsRedaction`: set *Send requests unmasked*, `Unredacted`, with every address on this machine or the local network) goes to them as the agent wrote it, and is masked only for a body an observer wants or the request archive keeps, which may go elsewhere (`bodiesKeptAway`); without those it isn't masked at all, and the log and recent calls keep it as it went, since a request is masked only once its route is known (#3973: masking a body whose masked copy was thrown away took up to minutes of magpie's own time). A request turned away before it is routed is logged masked (`maskedCopy`). count_tokens and embeddings follow the same rule. What a routing group's Jev is told of the turn (the message, the user's message before it and the reply to it) is masked all the same unless Jev's own provider skips redaction too (`maskedState` in `askJev`, [`decide.go`](../../internal/gateway/decide.go)): a remote Jev is none of the group's members. The masking is the built-in plugin `redact`'s (`gwmw.Masker`, [Plugin runtime](plugin-runtime.md)); when the settings ask for masking and no masker serves, or it fails, the request is a 503 and nothing is sent (`redacted`, `gwmw.Masking`).
- With *Mask personal data* on, each kind in `redact.Categories` ([`internal/redact/more.go`](../../internal/redact/more.go)) is masked unless the user turned it off (`redactKinds` in settings, `Options.Kinds`). On by default: US SSNs, passport numbers, IBANs that pass mod 97, birthdays next to a word that says so (a date over a year ago: one in the last year or ahead is the day an agent runs on, `dateOK`); the word must be right before the date, so a log line's or a file name's date is none, and a time beside the date doesn't make it a log's: a date of birth stored as a time (`1990-05-11T16:00:00.000Z`) is one (`TestBirthdayLogDates`; #3878), MAC addresses, and serial numbers next to S/N or the like. Off by default: the user name in a home folder, user@host in a prompt or ssh command, public IPv4 and IPv6 addresses (never private, loopback, link-local, CGNAT or documentation ranges) and cloud bucket names, since agents send them in nearly every request and the model must copy the placeholder exactly inside a path or URL. Context rules run only near their words (`rule.matches`). What reads as an address but ends in a file's extension that no domain has (`…@2x.jpg`, `[email protected]`; `fileExt` in `realEmail`), or a scale before an extension that is a domain too (`[email protected]`, `[email protected]`; `scaleExt`), is a file's name, for the rule and the local model alike (#3029). `[email protected]` is still an address. The Settings rows come from `REDACT_KINDS` in app.js, which `TestRedactKindsAreRedacts` holds to the same ids and defaults.
- With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken, nor a snake_case word as a name (Codex's `sequential_cutoff`, #3144), nor as a secret what names a thing: a UUID, a digest with its algorithm's name, a file's path (`namesThing`; the model takes all three for secrets). The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with it on a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name; 和 or 与 says so only after a person, 吴敏和郑浩 but not 判断和边界, 发 or 打 only as the verb, 发了个 but not 关键发现, and a sentence starts after punctuation, not after a space after code, `gh pr view` 易触发; xuanwo on X), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left. The model is the user's pick (`settings.RedactModelName`, `pii.Choose`; the app's menu beside the switch, `pmPick`): AstrLink Guard (QuantumNous/astrlink-guard at a pinned revision, Apache-2.0, a 39 MB int8 Multilingual-MiniLM token classifier for Chinese and English), run in magpie's own process on the CPU by [`internal/pii/guard`](../../internal/pii/guard) (an ONNX reader, the XLM-R tokenizer and int8 kernels in Go, NEON/SDOT on arm64 and SSE2 on amd64; its labels named as Privacy Filter's, `guardLabels`), and on a Mac through CoreML on the Neural Engine or the GPU ([`coreml.go`](../../internal/pii/guard/coreml.go), [`internal/pii/guard/coreml`](../../internal/pii/guard/coreml); yyhhyyyyyy on X): the same network from the same int8 weights in float16, written from the downloaded model into a program made by `coreml_gen.py` and compiled once into the model's folder, 512 tokens in about 12 ms where the CPU takes 400, its probabilities within .07 of the publisher's fp32 model where the CPU's are within .13 (the CPU quantizes activations as the int8 model does, CoreML doesn't), so a span near the threshold can fall the other side (a go.mod module path is a secret at .81 on CoreML, .79 on the CPU, .785 in fp32); a window under 16 tokens (`coremlMin`), a Mac before macOS 13 or any failure goes to the CPU, loaded on the first request and let go after 10 minutes with none; or OpenAI's Privacy Filter. With no pick it is AstrLink Guard, the default; a user who had Privacy Filter on before there was a choice keeps it only while AstrLink Guard downloads, which the first masked request starts and which then saves AstrLink Guard as the pick (`masking.picked`, `toGuard`; an hour between tries after a failure). Measured for that default (#3578): on 234 segments of real code and config it found 2 spans where Privacy Filter found 100 (38 of them go.sum hashes after redact's own checks), and with the rules it found 42 of 42 test values to Privacy Filter's 41. Each model has its own download, cache file and turn (`backend`). Both find at a floor of 0.5 (`pii.Floor`) and give each span its score (AstrLink Guard: the least of its tokens' probabilities, rounded to a float32 so a cached span reads back the same; Privacy Filter: pf-serve's own), and masking keeps those at the model's threshold or over (`pii.Threshold` in `masking.model`): the user's for that model (`settings.RedactModelScore`, this computer's own; Settings → Privacy's Confidence threshold, `pmScore`, 0.5 to 0.95), else the model's own (`MinScore`: Privacy Filter 0.5, AstrLink Guard 0.8, its publisher's). The defaults stay because on the eval a higher one mostly loses values the checks after the model would have kept (Privacy Filter at 0.85 lost a Chinese name; AstrLink Guard at 0.9 a phone in a tool result), while the false alarms it would drop are mostly dropped by those checks already (Yayoi_no_yume on X). Measured again on 2000 segments of real agent traffic (Codex and Claude Code sessions: prompts, tool arguments and results, a quarter with Chinese) and 30 more labelled cases: AstrLink Guard at 0.88 rather than 0.8 drops 3 false alarms (a model id, a sandbox path word, a token counter) and lets a colleague's name and a signed name through; Privacy Filter at 0.88 rather than 0.5 drops 11 (timestamps in paths, log lines, a plugin id) and lets 6 real names and a phone through (4 of the names in the real traffic). Lowering Guard to 0.5 finds 2 more names in the real traffic and 3 more labelled values, for 10 more false alarms (9 of them Claude Code's `<total_tokens>` counter). Those false alarms are shapes a check can turn away; a threshold can't without losing real values. Privacy Filter is a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags; on a GPU it holds the model once, in the device's memory, #3590; every pf-serve keeps the engine's own MoE chunk, `PF_MOE_CHUNK` at the window's 4096, which an RX 470 through Vulkan read 6-9% faster than 2048 in interleaved runs and Metal ~6% faster, and inherits a value the user set): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, through magpie's proxy (`netproxy.Func`, as its other requests; hh0592821 on Discord, whose download went direct before), from hf-mirror first in China (`source.chinaFirst`) and from hf-mirror after Hugging Face everywhere else, the switch off too (`source.modelMirror`), so a poisoned or blocked huggingface.co doesn't stop it; hf-mirror serves AstrLink Guard's files itself and sends Privacy Filter's back to Hugging Face) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU, doesn't load there in 40s (`gpuLoadTimeout`, well inside the gateway's 2 minutes), or doesn't answer in time there twice in a row, is started on the CPU, and Settings says why; a request it didn't answer in time once is tried on the GPU again, since a slow request on a weak GPU is no lost GPU (#3682). The GPU is tried again 10 minutes later, doubling to 2 hours while it keeps failing, with no restart (`gpuRetryAfter`), and one that answers is the model's again), joins short texts into 12 KB chunks and maps the spans back, cuts a text over 12 KB into pieces of 12 KB at most that overlap by 1 KB (`pieces`: after a line's end, else a space, else at a rune's start), each kept by its own hash, their spans put back at the text's offsets and merged where pieces overlap (`merge`: one found twice is one, one cut by a piece's end is whole), and asks pf-serve a chunk at a time (#3682: a 480 KB text whole took an RX 470/580 90s and kept nothing when cut off), keeps the last 20000 answers by hash, on disk too (`found-<model>.bin` beside the model: hashes, offsets, labels and each span's score, never text; a `pfound01` file from before scores is read with every span sure, so it masks as it did; written through temp-and-rename, a bad read taken as empty), so a restart doesn't ask the model about every history again (#3682), and ends it after 10 minutes with no request. Its requests take turns on its one pipe (`turn`), and a request reads the cache again when its turn comes; a request waits for its turn only while its context lives (`pii.FindContext`, through `gwmw.ContextTeller`), so one whose client went away leaves the queue (the gateway's 2 minutes leave masking's time out, #3682); the chunk pf-serve is on is finished and kept, so a request sent again asks only about what is left, and between chunks the turn is given up and taken again, so a short request goes before the rest of a long one. The status (`pii.Now`), `pii.Stop` and the idle timer never wait for a load or a request: `srv.mu` is held only to read or set the state, and `Stop` ends a load or a request under way at once. A request held before routing while it waits for the model is answered 504 saying so, and how to send without it (`unroutedMsg`); one of an older tag (`oldTags`, magpie updated since the download) runs until the pinned tag's is fetched (`upgrade`, on its start), which is then started on the next request and the old deleted ([`internal/pii`](../../internal/pii)). A failure names what ended `pf-serve`: magpie's own end of a request it didn't answer in time, else its exit status or signal and its last stderr lines past ggml's startup chatter (`died`, `lastSaid`). After one, requests are held back at once with that failure, not after a load each, until the model is tried again 30s later, doubling to 10 minutes while it keeps failing (`heldBack`, `retryAfter`; lazyeo on Discord, an M2 whose every request waited out the 504). A model that is on and can't answer (not on disk, didn't start, hung, held back) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).
- With *Mask secrets* on, the user's own masking rules (`redactRules`, `redact.Rule`: a placeholder name and a prefix or an RE2 pattern, [`internal/redact/rules.go`](../../internal/redact/rules.go)) run before magpie's. Settings → Privacy adds them, removes them, and edits one in place (`POST /api/settings/redact-rule` with the rule's index and its current value, `editRedactRule` in [`internal/gui/api.go`](../../internal/gui/api.go)). Every rule is checked by `redact.CheckRules` when settings are saved; an edit that doesn't compile, matches the empty string, repeats another rule or names a rule no longer at that index is refused with the rules left as they were. The masker reads the rules from settings on each call (`options` in [`internal/masking`](../../internal/masking/masking.go)), so a change applies to the next request.
diff --git a/internal/gateway/codex_backend.go b/internal/gateway/codex_backend.go
index a0f48a441..95a66641c 100644
--- a/internal/gateway/codex_backend.go
+++ b/internal/gateway/codex_backend.go
@@ -978,7 +978,7 @@ func (s *Server) codexUpstreamOn(w http.ResponseWriter, r *http.Request, rest st
if m := noteMasked(r, "chat", maskedHits, string(body), tr.ID, "openai", model); m != nil {
s.trace.update(tr, func(t *Route) { t.Masked = m })
}
- promptRead := s.inspectPrompt(tr, provider.Responses, body, body)
+ promptRead := s.inspectPrompt(tr, provider.Responses, body, body, false)
end = func(status int, msg string, tokens, out int) {
ms := time.Since(start).Milliseconds()
ttft, text := first.ms()
diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.go
index cd7a0a410..5ea2f9dc7 100644
--- a/internal/gateway/gateway.go
+++ b/internal/gateway/gateway.go
@@ -1363,14 +1363,6 @@ func (s *Server) countTokens(w http.ResponseWriter, r *http.Request) {
writeError(w, provider.Anthropic, 400, err.Error())
return
}
- // Count the same masked prompt that generation sends to the vendor.
- plain := body
- w, body, unmask, err := redacted(r.Context(), w, body)
- defer unmask()
- if err != nil {
- writeError(w, provider.Anthropic, maskStatus(err), err.Error())
- return
- }
named := unprefixed(model)
if m := claudeTierStandIn(r, agentOf(r), model); m != "" {
model = m // counted on the model the request will go to
@@ -1394,9 +1386,17 @@ func (s *Server) countTokens(w http.ResponseWriter, r *http.Request) {
if countOff(w, provider.Anthropic, id) {
return
}
- // and counts the unmasked one where generation sends that
- if _, ms, isGroup := provider.FindGroup(id); ok && unredactedRoute(p, isGroup, ms) {
- body = plain
+ // Count the same masked prompt that generation sends to the vendor,
+ // and the one as written where generation sends that, masked only
+ // then (#3973)
+ if _, ms, isGroup := provider.FindGroup(id); !ok || !unredactedRoute(p, isGroup, ms) {
+ var unmask func()
+ w, body, unmask, err = redacted(r.Context(), w, body)
+ defer unmask()
+ if err != nil {
+ writeError(w, provider.Anthropic, maskStatus(err), err.Error())
+ return
+ }
}
if level := desktopSubagentEffort(r, agentOf(r), body, pick); level != "" && ok && !provider.Reasonless(p.ID, model) {
body = withFixedEffort(provider.Anthropic, body, fitLevel(level, p.Efforts(model)))
@@ -1702,27 +1702,22 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro
// what the vendor saw and said. plain is the body as the agent wrote
// it, for a request that goes only where the user said it may go
// unmasked (unredactedRoute)
+ // unmasked (unredactedRoute). The body is masked once the route is
+ // known (below): one that goes as written isn't masked at all, where
+ // masking it took up to minutes for nothing (#3973)
plain := body
var masked []byte // the masked body, when plain goes instead
- w, body, unmask, maskedHits, err := redactedHits(r.Context(), w, body)
- defer unmask()
- if err != nil { // fails closed: never sent unmasked
- log.Printf("gateway: %v", err)
- writeError(w, from, maskStatus(err), err.Error())
- return
- }
+ var maskedHits maskHits
+ maskDone, sentWhole := false, false
+ unmask := func() {}
+ defer func() { unmask() }()
// the reply's model the member that answered, when asked for (#822)
w, r, named := withMemberNames(w, r)
defer named()
- requestBody, requestTruncated := captureRequestBody(body)
// an observer may want the bodies uncut (the OTLP export, #538): the
// request is already whole in memory, so keep it as it came, and the
// reply goes to a spool as it streams
wholeBodies := usage.ObservedBodies() == usage.WholeBodies
- var wholeIn []byte
- if wholeBodies {
- wholeIn = body
- }
// a model asked for at an effort of its own (a Claude Code tier, #536)
// is the model, every try of it asked for that effort
askedFor := modelOf(body) // as it reaches serve, which an error in its stream names (keptAlive)
@@ -1750,7 +1745,18 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro
metadata := requestSessionMetadata(r.Header, body)
var clientResponseID string // only a reply released to the client owns this ID
call := Call{Time: start, From: from, Model: unprefixed(modelOf(body)), Agent: who.agent, Via: who.via, Kind: requestCallKind(r.Header, metadata),
- RequestBody: requestBody, RequestTruncated: requestTruncated, wholeIn: wholeIn, wire: archiving(r, capture, start, body)}
+ wire: archiving(r, capture, start, nil)}
+ // the log has the body as it went: masked, or as written where it
+ // went so
+ logBody := func(b []byte) {
+ call.RequestBody, call.RequestTruncated = captureRequestBody(b)
+ if wholeBodies {
+ call.wholeIn = b
+ }
+ if call.wire != nil {
+ call.wire.body = b
+ }
+ }
defer discardArchive(capture)
r, watched := observeRequest(r, call.Kind, body)
defer func() { watched.end(call, time.Since(start).Milliseconds()) }()
@@ -1786,6 +1792,10 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro
// a request turned away before any provider was asked is in the log
// as the failure it was, with the reason
turnedAway := func() {
+ if !maskDone {
+ // sent nowhere: the log has it masked, or not at all
+ logBody(maskedCopy(r.Context(), body))
+ }
finishCapture()
call.Millis = time.Since(start).Milliseconds()
s.record(call)
@@ -1921,10 +1931,27 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro
g, ms, isGroup := provider.FindGroup(asked)
g = g.Live() // a manual group's rules wait
// a model on this machine or the local network that the user set to
- // go unmasked gets the request as written (lc on Discord); the log
- // keeps the masked one
- if unredactedRoute(p, isGroup, ms) {
- masked, body = body, plain
+ // go unmasked gets the request as written (lc on Discord), and it is
+ // masked only for an observer or the archive, which may keep it
+ // elsewhere
+ if sentWhole = unredactedRoute(p, isGroup, ms) && !bodiesKeptAway(); !sentWhole {
+ rw, b, done, hits, err := redactedHits(r.Context(), capture.ResponseWriter, body)
+ if err != nil { // fails closed: never sent unmasked
+ log.Printf("gateway: %v", err)
+ call.Status, call.Error = maskStatus(err), err.Error()
+ writeError(w, from, call.Status, err.Error())
+ return
+ }
+ capture.ResponseWriter, body, unmask, maskedHits = rw, b, done, hits
+ if unredactedRoute(p, isGroup, ms) {
+ masked, body = body, plain
+ }
+ }
+ maskDone = true
+ if masked != nil {
+ logBody(masked)
+ } else {
+ logBody(body)
}
// a gateway key held to some models (#882) is refused another, or a
// group it doesn't name with one it may not use in it
@@ -2426,7 +2453,7 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro
// request went as written. The record names the provider and model
// the last try went to, not the group's first member, which a
// failover may never have reached
- if masked == nil {
+ if masked == nil && !sentWhole {
m, done := noteMaskedUntil(r, "chat", maskedHits, string(body), tr.ID, p.ID, call.Model)
if m != nil {
s.trace.update(tr, func(t *Route) { t.Masked = m })
@@ -2434,12 +2461,12 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro
defer func() { done(call.Provider, model) }()
}
// the card's text is the masked body, even where the request went
- // unmasked
+ // unmasked: masked by PromptText when it went whole (#3973)
textBody := body
if masked != nil {
textBody = masked
}
- promptRead := s.inspectPrompt(tr, from, body, textBody)
+ promptRead := s.inspectPrompt(tr, from, body, textBody, sentWhole)
var lastTried provider.Provider // the last try's, for its model's context window and the usage's endpoint
var skipped []string
sent := "" // the reasoning the last try's model was asked for
diff --git a/internal/gateway/prompt.go b/internal/gateway/prompt.go
index 3a2069fcd..1a205623e 100644
--- a/internal/gateway/prompt.go
+++ b/internal/gateway/prompt.go
@@ -1226,13 +1226,14 @@ func windowFor(id, model string) int {
// inspectPrompt reads the request's prompt beside it, putting the estimate
// in the trace as soon as it is read; wait returns it, for the route's
// end to scale to what was counted. masked is the body as the vendor would
-// get it with masking on, kept a while for the card's text (PromptText).
-func (s *Server) inspectPrompt(tr *Route, from provider.Protocol, body, masked []byte) (wait func() *Prompt) {
+// get it with masking on, kept a while for the card's text (PromptText);
+// plain when it went unmasked, so is masked only when the card asks.
+func (s *Server) inspectPrompt(tr *Route, from provider.Protocol, body, masked []byte, plain bool) (wait func() *Prompt) {
ch := make(chan *Prompt, 1)
go func() {
p := promptOf(from, body)
if p != nil {
- s.trace.keepBody(tr.ID, from, masked)
+ s.trace.keepBody(tr.ID, from, masked, plain)
s.trace.update(tr, func(t *Route) {
if t.Prompt == nil {
t.Prompt = p
diff --git a/internal/gateway/prompt_text.go b/internal/gateway/prompt_text.go
index 08005829a..9f56aa473 100644
--- a/internal/gateway/prompt_text.go
+++ b/internal/gateway/prompt_text.go
@@ -13,6 +13,7 @@ package gateway
import (
"bytes"
+ "context"
"encoding/json"
"slices"
"strconv"
@@ -37,17 +38,19 @@ type keptBody struct {
id int64
from provider.Protocol
body []byte
+ // plain: body went unmasked, and is masked when its text is read
+ plain bool
}
// keepBody keeps a request's masked body for its prompt's text, dropping
// the oldest past promptBodies or promptBodiesMax.
-func (t *trace) keepBody(id int64, from provider.Protocol, body []byte) {
+func (t *trace) keepBody(id int64, from provider.Protocol, body []byte, plain bool) {
if len(body) == 0 || len(body) > promptBodyMax {
return
}
t.mu.Lock()
defer t.mu.Unlock()
- t.bodies = append(t.bodies, keptBody{id: id, from: from, body: body})
+ t.bodies = append(t.bodies, keptBody{id: id, from: from, body: body, plain: plain})
total := 0
for _, b := range t.bodies {
total += len(b.body)
@@ -59,6 +62,17 @@ func (t *trace) keepBody(id int64, from provider.Protocol, body []byte) {
}
}
+// maskedBody puts body, masked, in place of route id's plain kept body.
+func (t *trace) maskedBody(id int64, body []byte) {
+ t.mu.Lock()
+ defer t.mu.Unlock()
+ for i := range t.bodies {
+ if t.bodies[i].id == id && t.bodies[i].plain {
+ t.bodies[i].body, t.bodies[i].plain = body, false
+ }
+ }
+}
+
// keptBodyOf is a route's kept body; false when it isn't kept.
func (t *trace) keptBodyOf(id int64) (keptBody, bool) {
t.mu.Lock()
@@ -104,6 +118,14 @@ func (s *Server) PromptText(id int64, kind string, withText bool) (PromptText, b
if !ok || !slices.Contains(partOrder, kind) {
return PromptText{}, false
}
+ if kb.plain {
+ // masked here, not on the request's way (#3973); kept masked for
+ // the next read
+ if kb.body = maskedCopy(context.Background(), kb.body); kb.body == nil {
+ return PromptText{}, false
+ }
+ s.trace.maskedBody(id, kb.body)
+ }
b, ok := promptPieces(kb.from, kb.body)
if !ok {
return PromptText{}, false
diff --git a/internal/gateway/prompt_text_test.go b/internal/gateway/prompt_text_test.go
index 43fcb373b..404f805b9 100644
--- a/internal/gateway/prompt_text_test.go
+++ b/internal/gateway/prompt_text_test.go
@@ -101,20 +101,20 @@ func TestPromptTextKeepsOnlyTheLastBodies(t *testing.T) {
var tr trace
body := []byte(`{"system":"s","messages":[{"role":"user","content":"hi"}]}`)
for id := int64(1); id <= promptBodies+2; id++ {
- tr.keepBody(id, provider.Anthropic, body)
+ tr.keepBody(id, provider.Anthropic, body, false)
}
for id := int64(1); id <= promptBodies+2; id++ {
if _, ok := tr.keptBodyOf(id); ok != (id > 2) {
t.Fatalf("body %d kept %v", id, ok)
}
}
- tr.keepBody(100, provider.Anthropic, make([]byte, promptBodyMax+1))
+ tr.keepBody(100, provider.Anthropic, make([]byte, promptBodyMax+1), false)
if _, ok := tr.keptBodyOf(100); ok {
t.Fatal("kept a body over promptBodyMax")
}
big := make([]byte, promptBodiesMax/3)
for id := int64(200); id < 204; id++ {
- tr.keepBody(id, provider.Anthropic, big)
+ tr.keepBody(id, provider.Anthropic, big, false)
}
total := 0
for _, b := range tr.bodies {
diff --git a/internal/gateway/redact.go b/internal/gateway/redact.go
index a91d8aaed..200e5697b 100644
--- a/internal/gateway/redact.go
+++ b/internal/gateway/redact.go
@@ -12,6 +12,7 @@ import (
"github.com/yetone/magpie/internal/maskrec"
"github.com/yetone/magpie/internal/provider"
"github.com/yetone/magpie/internal/settings"
+ "github.com/yetone/magpie/internal/usage"
)
// redacted masks a request's secrets, as the settings say, before it goes
@@ -240,6 +241,42 @@ func maskedQuery(ctx context.Context, query string, p *provider.Provider) (strin
return masked, err
}
+// maskedCopy is body masked for a record of a request sent nowhere, nil
+// when masking fails: nothing unmasked is kept in its place.
+func maskedCopy(ctx context.Context, body []byte) []byte {
+ m, err := gwmw.Masking(redactionWanted())
+ if err != nil {
+ return nil
+ }
+ if m == nil {
+ return body
+ }
+ b, err := gwmw.Safely(func() []byte {
+ if t, ok := m.(gwmw.ContextTeller); ok {
+ b, _ := t.JSONHitsContext(ctx, body)
+ return b
+ }
+ b, _ := m.JSON(body)
+ return b
+ })
+ if err != nil {
+ return nil
+ }
+ return b
+}
+
+// bodiesKeptAway says a request's body may be kept off this computer: an
+// observer wants it (the OTLP export, a JS plugin's observer) or the
+// request archive is on. A request sent unmasked is masked for them all
+// the same.
+func bodiesKeptAway() bool {
+ if usage.ObservedBodies() != usage.NoBodies {
+ return true
+ }
+ _, on := archiveDest()
+ return on
+}
+
// unredactedRoute says a request resolved to p, or to the group whose
// models are ms, goes only where the user set requests to go unmasked
// (provider.SkipsRedaction): p and each of its fallbacks, or every model of
diff --git a/internal/gateway/unmasked_route_test.go b/internal/gateway/unmasked_route_test.go
new file mode 100644
index 000000000..c93e2867d
--- /dev/null
+++ b/internal/gateway/unmasked_route_test.go
@@ -0,0 +1,123 @@
+package gateway
+
+import (
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "sync/atomic"
+ "testing"
+
+ "github.com/yetone/magpie/internal/gwmw"
+ "github.com/yetone/magpie/internal/masking"
+ "github.com/yetone/magpie/internal/provider"
+ "github.com/yetone/magpie/internal/settings"
+ "github.com/yetone/magpie/internal/usage"
+)
+
+// countedMasker is the built-in masker, counting the bodies it is asked
+// to mask.
+type countedMasker struct {
+ m masking.Masker
+ n *atomic.Int64
+}
+
+func (c countedMasker) JSON(b []byte) ([]byte, int) { c.n.Add(1); return c.m.JSON(b) }
+func (c countedMasker) Text(s string) (string, int) { c.n.Add(1); return c.m.Text(s) }
+func (c countedMasker) Restore(s string, escaped bool) string {
+ return c.m.Restore(s, escaped)
+}
+func (c countedMasker) Writer(w http.ResponseWriter) (http.ResponseWriter, func()) {
+ return c.m.Writer(w)
+}
+
+// wangbinyq (#3973): a request that goes only to a local model set to go
+// unmasked was masked all the same before it was routed, and the masked
+// body thrown away, up to minutes of magpie's own time for each. Now it
+// isn't masked at all, a chat or a token count, and the log has it as it
+// went; one that goes to a vendor, an unknown model's, or one an observer
+// keeps the bodies of is masked as before.
+func TestUnmaskedRouteIsNotMasked(t *testing.T) {
+ const key = "sk-proj-abcdEFGH1234ijklMNOP5678qrst"
+ chat := `{"model":"fake/m1","max_tokens":10,"messages":[{"role":"user","content":"my key is ` + key + `"}]}`
+ count := `{"model":"fake/m1","messages":[{"role":"user","content":"my key is ` + key + `"}]}`
+ cases := []struct {
+ name string
+ set bool
+ observed bool
+ masks bool // the masker is asked
+ whole bool // the provider is sent the key
+ }{
+ {name: "set to go unmasked", set: true, whole: true},
+ {name: "unset", masks: true},
+ {name: "an observer keeps the bodies", set: true, observed: true, masks: true, whole: true},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ f := &fake{ctype: "application/json", reply: `{"id":"msg_1","type":"message","role":"assistant","model":"m1","content":[{"type":"text","text":"ok"}],"stop_reason":"end_turn","usage":{"input_tokens":3,"output_tokens":1}}`}
+ setup(t, provider.Anthropic, f)
+ p, err := provider.Find("fake")
+ if err != nil {
+ t.Fatal(err)
+ }
+ p.Unredacted = tc.set
+ if err := provider.Save(*p); err != nil {
+ t.Fatal(err)
+ }
+ if err := settings.Save(settings.Settings{Redact: true}); err != nil {
+ t.Fatal(err)
+ }
+ if tc.observed {
+ t.Cleanup(usage.Observe(&fakeObserver{bodies: usage.CapturedBodies}))
+ }
+ withoutMasker(t)
+ n := new(atomic.Int64)
+ t.Cleanup(gwmw.Mask(countedMasker{n: n}))
+ s := New()
+ rec := httptest.NewRecorder()
+ s.Handler().ServeHTTP(rec, httptest.NewRequest("POST", "/v1/messages", strings.NewReader(chat)))
+ if rec.Code != 200 {
+ t.Fatalf("%d %s", rec.Code, rec.Body)
+ }
+ if whole := strings.Contains(string(f.got), key); whole != tc.whole {
+ t.Fatalf("sent whole %v, want %v: %s", whole, tc.whole, f.got)
+ }
+ if masks := n.Load() > 0; masks != tc.masks {
+ t.Fatalf("masked %d times, want masking %v", n.Load(), tc.masks)
+ }
+ c := s.Recent()[0]
+ if logged := strings.Contains(c.RequestBody, key); logged != !tc.masks {
+ t.Fatalf("the log has the key %v, want %v: %s", logged, !tc.masks, c.RequestBody)
+ }
+ if c.RequestBody == "" {
+ t.Fatal("the log has no body")
+ }
+ n.Store(0)
+ f.reply = `{"input_tokens":17}`
+ if code, reply := post(t, "/v1/messages/count_tokens", count); code != 200 {
+ t.Fatalf("count: %d %s", code, reply)
+ }
+ if whole := strings.Contains(string(f.got), key); whole != tc.whole {
+ t.Fatalf("counted whole %v, want %v: %s", whole, tc.whole, f.got)
+ }
+ if masks := n.Load() > 0; masks != (tc.masks && !tc.set) {
+ t.Fatalf("count masked %d times, want masking %v", n.Load(), tc.masks && !tc.set)
+ }
+ })
+ }
+ t.Run("an unknown model's log", func(t *testing.T) {
+ f := &fake{ctype: "application/json"}
+ setup(t, provider.Anthropic, f)
+ if err := settings.Save(settings.Settings{Redact: true}); err != nil {
+ t.Fatal(err)
+ }
+ s := New()
+ rec := httptest.NewRecorder()
+ s.Handler().ServeHTTP(rec, httptest.NewRequest("POST", "/v1/messages", strings.NewReader(strings.Replace(chat, "fake/m1", "nobody/m1", 1))))
+ if rec.Code != 404 {
+ t.Fatalf("%d %s", rec.Code, rec.Body)
+ }
+ if c := s.Recent()[0]; c.RequestBody == "" || strings.Contains(c.RequestBody, key) {
+ t.Fatalf("the log has %q", c.RequestBody)
+ }
+ })
+}