Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
masking: each kind is masked, sent with a warning or held back; a Never mask list and Try it (#1977)
yetoneb00c82acfa4c
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.mdindex 0cb21d653..26a0d0c71 100644--- a/docs/subsystems/gateway-routing.md+++ b/docs/subsystems/gateway-routing.md@@ -79,7 +79,8 @@ recorded as a prediction failure (`noPrediction`).- With *Mask personal data* on, each kind in `redact.Categories` ([`internal/redact/more.go`](../../internal/redact/more.go)) is masked unless the user turned it off (`redactKinds` in settings, `Options.Kinds`). On by default: US SSNs, passport numbers, IBANs that pass mod 97, birthdays next to a word that says so, MAC addresses, and serial numbers next to S/N or the like. Off by default: the user name in a home folder, user@host in a prompt or ssh command, public IPv4 and IPv6 addresses (never private, loopback, link-local, CGNAT or documentation ranges) and cloud bucket names, since agents send them in nearly every request and the model must copy the placeholder exactly inside a path or URL. Context rules run only near their words (`rule.matches`). What reads as an address but ends in a file's extension that no domain has (`…@2x.jpg`, `[email protected]`; `fileExt` in `realEmail`), or a scale before an extension that is a domain too (`[email protected]`, `[email protected]`; `scaleExt`), is a file's name, for the rule and the local model alike (#3029). `[email protected]` is still an address. The Settings rows come from `REDACT_KINDS` in app.js, which `TestRedactKindsAreRedacts` holds to the same ids and defaults.- With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken, nor a snake_case word as a name (Codex's `sequential_cutoff`, #3144), nor as a secret what names a thing: a UUID, a digest with its algorithm's name, a file's path (`namesThing`; the model takes all three for secrets). The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with it on a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left. The model is OpenAI's Privacy Filter as a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, from hf-mirror first in China, `source.chinaFirst`) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU is started on the CPU for the rest of the run), joins short texts into 12 KB chunks and maps the spans back, keeps the last 20000 answers by hash, and ends it after 10 minutes with no request ([`internal/pii`](../../internal/pii)). A model that is on and can't answer (not on disk, didn't start, hung) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).- With *Mask secrets* on, the user's own masking rules (`redactRules`, `redact.Rule`: a placeholder name and a prefix or an RE2 pattern, [`internal/redact/rules.go`](../../internal/redact/rules.go)) run before magpie's. Settings → Privacy adds them, removes them, and edits one in place (`POST /api/settings/redact-rule` with the rule's index and its current value, `editRedactRule` in [`internal/gui/api.go`](../../internal/gui/api.go)). Every rule is checked by `redact.CheckRules` when settings are saved; an edit that doesn't compile, matches the empty string, repeats another rule or names a rule no longer at that index is refused with the rules left as they were. The masker reads the rules from settings on each call (`options` in [`internal/masking`](../../internal/masking/masking.go)), so a change applies to the next request.- - Each request that had anything masked is recorded as it goes to the vendor (`noteMasked` in [`masked.go`](../../internal/gateway/masked.go), from chat, the Codex backend, search, images, video, System One, embeddings and audio; count_tokens is masked and not recorded). The masker says what it swapped through `gwmw.Teller` (`JSONHits`, `TextHits`: each placeholder's kind, its finder, `redact.ByRule`, `ByYours`, `ByWord`, `ByModel` or `ByKnown`, and a hint of its first characters and its length), and [`internal/maskrec`](../../internal/maskrec/maskrec.go) keeps one line per request in a file per day under `~/.config/magpie/masking`, 30 days and 100 MB in all: the agent, session, provider, model, and each placeholder with its kind, finder and hint, never the value. The text around each placeholder as the vendor saw it is kept in memory only, for the last 400 records, and what a placeholder stood for only in the masker's own memory (`Teller.Value`); after a restart neither is there. A recent call (`Call.Masked`) and the route trace (`Route.Masked`) carry the counts and the record's key, which the Gateway and Routing pages' badge opens in Settings → Privacy (`/api/masking/*`, [`internal/gui/masking.go`](../../internal/gui/masking.go)). A masker without `Teller`, a plugin's, masks as before and records nothing. Under a test binary testenv isolated nothing is written (`maskrec.inTest`).+ - What is done with a value found is the user's per kind (#1977, `redactActions` in settings, `Options.Actions`, set by `POST /api/settings/redact-actions`; a kind left out is masked). `redact.Warn` sends it as written: the masker tells it, with a placeholder no one remembers, and it isn't counted as masked. `redact.Block` masks it as any other, and the gateway holds the request back with a 400 that names the kind and the value's hint, never the value (`blocked`, `maskBlocked` in [`redact.go`](../../internal/gateway/redact.go)); nothing is sent and the record is kept as `Blocked` (`noteBlocked`). A value on the *Never mask* list (`redactAllow`, `Options.Allow`, `POST /api/settings/redact-allow`) goes as written when it is the whole value found, in any case, and isn't told; a longer value it is part of is masked. Settings → Privacy's *Try it* posts a text to `POST /api/masking/preview` ([`masking.go`](../../internal/gui/masking.go)), which masks it as a request would be (`redact.Preview`) and remembers nothing: its placeholders stand for nothing afterwards, its values aren't masked later as seen before, and it returns only the masked text and each hit's kind, finder, action, placeholder and hint. The page keeps the text in no storage.+ - Each request that had anything masked is recorded as it goes to the vendor (`noteMasked` in [`masked.go`](../../internal/gateway/masked.go), from chat, the Codex backend, search, images, video, System One, embeddings and audio; count_tokens is masked and not recorded). The masker says what it swapped through `gwmw.Teller` (`JSONHits`, `TextHits`: each placeholder's kind, its finder, `redact.ByRule`, `ByYours`, `ByWord`, `ByModel` or `ByKnown`, and a hint of its first characters and its length), and [`internal/maskrec`](../../internal/maskrec/maskrec.go) keeps one line per request in a file per day under `~/.config/magpie/masking`, 30 days and 100 MB in all: the agent, session, provider, model, and each placeholder with its kind, finder, action and hint, never the value. The text around each placeholder as the vendor saw it is kept in memory only, for the last 400 records, and what a placeholder stood for only in the masker's own memory (`Teller.Value`); after a restart neither is there. A recent call (`Call.Masked`) and the route trace (`Route.Masked`) carry the counts and the record's key, which the Gateway and Routing pages' badge opens in Settings → Privacy (`/api/masking/*`, [`internal/gui/masking.go`](../../internal/gui/masking.go)). A masker without `Teller`, a plugin's, masks as before and records nothing. Under a test binary testenv isolated nothing is written (`maskrec.inTest`).- `DecidesModel` gives a 400.- A gateway key held to some models (`keyHolds`, #882) gets a 403 for any other model.- A model the user disabled (#1867) gets a 403 for every caller (`idOff`), before anything is sent; see [Disabled models](#disabled-models).diff --git a/docs/subsystems/gui-shell.md b/docs/subsystems/gui-shell.mdindex 51b5d3114..4906162e4 100644--- a/docs/subsystems/gui-shell.md+++ b/docs/subsystems/gui-shell.md@@ -17,7 +17,7 @@ three.| Browser mode | `StartWeb` serves the page and the gateway for `magpie web`. The browser signs in with a run key in a link. Gateway mode shows Providers, Gateway, Routing, Usage, Sessions, Plugins and Settings on a machine that is only a gateway. `webHandler` gzips the page's text files and the API's JSON for a browser that takes gzip (`webCompressed`, with a `-gzip` ETag that still answers 304); the app's own windows read the page in-process and get it uncompressed. | [`web.go`](../../internal/gui/web.go), [`gatewaymode.go`](../../internal/gui/gatewaymode.go) || The page | `app.js` keeps one state object per view and renders it into a list, with no framework. The views have their own files: `plugins.js` and `context.js`, and those a plugin adds under `assets/plugins/<id>/` (the Gateway, Sessions, Library, Routing and Usage pages, see [plugin runtime](plugin-runtime.md)) (Usage's Context tab, and the context window card the Routing page draws under a request's story, fed by `/api/context` in `context.go`, which scores and tags each agent; clicking a part, a cell or an item opens its text from `/api/context/text`, the last 10 requests' masked bodies in memory (`openCtxText`), rendered 100 items and 20000 characters at a time; its "Tuned for you" cards are fed by `/api/tune` in `tune.go`, see [agent wiring](agent-wiring.md)). Strings go through `t()` with translations in `i18n.js`, and a plugin's own in its `i18n.js` (zh, zh-TW, ja and de; the English text is the key and the fallback). zh-TW is Traditional Chinese with Taiwan's words; a system language of zh-TW, zh-HK, zh-MO or any zh-Hant picks it, and the tray menu and notifications follow (`trayLang`). Dropdowns use `openProtoMenu`. Every app menu (`openProtoMenu`, `openRowMenu`, `openSessCombo`) is placed by `placeMenu`: under its anchor, or over it when the window runs out below; one that fits neither way takes the larger room, its height held to it, and scrolls inside itself, and a scroll inside a menu doesn't close it (#1437, `removed-groups-menu-scroll.test.cjs`). The highest any popover goes is `popTop`: 8px under `header.top` on a Mac window, where the traffic lights are drawn over the page, and 8px from the top elsewhere; `placeMenu`, the field picker's `placePop` (which also holds `#pop`'s height to that room) and an agent's model list (`openAgentModels`) all use it (#1573, `popover-mac-chrome.test.cjs`). Scrolling is done only by `scrollOnPurpose(e)` with the reader's event. | [`assets/app.js`](../../internal/gui/assets/app.js), [`assets/i18n.js`](../../internal/gui/assets/i18n.js) || Window navigation | In the window from 1024 points wide (`SIDE_AT`, the same width as app.css's `side-nav` rules), `layoutNav` moves the brand, the pages (`#nav`) and Settings into the sidebar (`#side`): the pages in four groups (Work, Models, Insights, Extend; a group none of whose pages shows, as in gateway mode, leaves out its name), the gateway's address at its foot (`sideGwRead`, read from `/api/providers` only while there is a sidebar), and the header a breadcrumb (`renderCrumb`). A page whose head holds only its tools (Library, Plugins, Settings) has them in the title bar after the breadcrumb, `tools` setting the room between it and the window's own buttons. When they don't fit there the tools show their icons alone (`tools-tight`; each keeps its name as `aria-label` and in its tooltip), then the breadcrumb drops its leading parts, and last the page's name is cut at its end, so no tool lies over the title (#2625, `title-bar-tools.test.cjs`). The open page's own tabs (`NAV_SUBS`: Usage, Library, Plugins, Settings) are listed under it as `.nav-s` rows (`role="tab"`, `data-of` the page tab's id) and the page's strip is hidden; a row's click clicks that tab at once, the selection (`.nav-hl`) having set off as the row was pressed (`navSubGlide`; back if the press leaves the row), and is the list's only shape: a row hovered or pressed has no ground of its own, and a `MutationObserver` draws the list again when the strip changes. A page's tabs share its one scroll, so a row that picks another tab opens it at its top (`scrollOnPurpose` with the row's click); the tab already open, picked again, leaves the page where it is (`sidebar-tab-top.test.cjs`). The list opens and folds with an iOS-like spring (`navSubEnter`, `navSubLeave`; none under reduced motion), and a folding list is out of reach at once. Narrower, the page beside a sidebar would be narrower than Usage's requests table needs (768 points), so they go back into the header's row, as a Mac sidebar gives way in a narrow window. At every width the reader can pick the sidebar or the header's tabs from the title bar's button (`#sideToggle`, ⌃⌘S or Ctrl+Shift+S; `toggleSideNav`), kept in localStorage `magpie.navLayout` (`side` or `top`) and so laid out from the first frame; none picked, the sidebar comes from `SIDE_AT`. Picked in a narrow window the sidebar is narrower (184 points under 860), and magpie web on a phone keeps its own tab bar and has no button. The two layouts trade places in a view transition (`navMorph`, app.css `.nav-morph`): the sidebar slides, each page's tab flies to its place in the other, and the page slides as one (its old image dropped, so no line prints twice); without view transitions, under reduced motion, or in the Linux app the change is at once: WebKitGTK with compositing off (`WEBKIT_DISABLE_COMPOSITING_MODE`, which `webkit_linux.go` sets under Wayland) crashes in a view transition. Elsewhere a transition still going after `MORPH_MAX` (1.5s) is skipped, so a webview that never ends one can't leave the window on the old layout's picture. See `side-toggle.test.cjs`. A window made wider or narrower drops what a click held (`heldWidth`): the page laid out again isn't the click's doing. The reader's own place is kept instead for a second while the page is laid out again (`relaid`, a `ResizeObserver` on the open page's parts calling `backToReader`): crossing `SIDE_AT` the browser clamps the scroll to each layout it passes through, and the page ended short of its foot (#1672, `text-size.test.cjs`). See `sidebar.test.cjs`. A sidebar longer than a short window keeps no scroll bar beside its pages: the native bars of `.side-scroll` and `.side-foot` are hidden, and `sideBars` lays a thin bar over each box's edge that shows while it scrolls, fades about a second after it stops (kept while hovered or dragged), can be dragged or clicked to page, and is gone when the box fits, the same in WebKit, WebView2 and WebKitGTK. See `sidebar-overlay-scrollbar.test.cjs`. The window opens on the page the reader last left, as a Mac app comes back as it was (molvqingtai and lc on Discord): `show` keeps the page in localStorage `magpie.lastView` (which the app's webview keeps across restarts, as it keeps `magpie.navLayout`), and `keptView` opens it at start. A page asked for in the address (`?view=`, the tray's links, a reload) wins, Agents too; a kept name that is no page opens the first page, and one gateway mode leaves out opens Providers (`show`). Opening on it scrolls nothing. See `last-page.test.cjs`. | [`assets/app.js`](../../internal/gui/assets/app.js), [`assets/app.css`](../../internal/gui/assets/app.css) |-| Settings › Privacy | The page opens on what masking kept from vendors (`drawMaskObs` in app.js, fed by `/api/masking/summary`): the values masked over 7 or 30 days with a chart of the days, the kinds, what only the local model found, agents to providers, and the values masked most, each shown only while pointed at (`maskTip`, `/api/masking/value`, which says it is gone after a restart). Its records (`drawMaskRecs`, `/api/masking/records` and `/api/masking/record`) take the page's place, filtered by kind, finder, agent or provider, a day at a time, each opening on the text the vendor saw with its placeholders marked. `maskedBadge`, on the Gateway page's recent calls and the Routing page's log, opens a request's own record (`openMaskRecord`). Below the card, *What to mask*, *How it's found* (the rules, always on while anything is masked, and the local model on top of them) and *Sharing*. See [gateway routing](gateway-routing.md) for what is recorded. | [`assets/app.js`](../../internal/gui/assets/app.js), [`masking.go`](../../internal/gui/masking.go) |+| Settings › Privacy | The page opens on what masking kept from vendors (`drawMaskObs` in app.js, fed by `/api/masking/summary`): the values masked over 7 or 30 days with a chart of the days, the kinds, what only the local model found, agents to providers, and the values masked most, each shown only while pointed at (`maskTip`, `/api/masking/value`, which says it is gone after a restart). Its records (`drawMaskRecs`, `/api/masking/records` and `/api/masking/record`) take the page's place, filtered by kind, finder, agent or provider, a day at a time, each opening on the text the vendor saw with its placeholders marked. `maskedBadge`, on the Gateway page's recent calls and the Routing page's log, opens a request's own record (`openMaskRecord`), in amber with the count sent as written when every value found was on Warn. Below the card, *What to mask*, *How it's found* (the rules, always on while anything is masked, and the local model on top of them), *When a value is found* (Mask, Warn or Block per kind, the secret kinds first and the personal ones behind *Show all kinds*, and the *Never mask* list; `renderMaskActions`), *Try it* (`renderMaskTry`, the text as the vendor would get it from `/api/masking/preview`, kept in no storage) and *Sharing*. See [gateway routing](gateway-routing.md) for what is recorded. | [`assets/app.js`](../../internal/gui/assets/app.js), [`masking.go`](../../internal/gui/masking.go) || Built-in plugins | Plugins › Built-in, the page's first tab (`drawBuiltin` in `plugins.js`, rows by `renderBuiltins` in `app.js`), lists magpie's built-in plugins, each with a name, a line on what it is and a switch: Routing groups, Library, Observability, Sessions, Warm-up and check-in, and the Gateway and Routing pages (Essential). A plugin on with settings has their rows under its row, drawn from its schema by the kernel's `ui.form`, and a change posts `{id, config}`. The Gateway page is drawn by its plugin's own scripts through the window's kernel (`kernel.js`, `assets/plugins/gateway/`); see the [plugin runtime](plugin-runtime.md), "The window's kernel". They are `plugins/*` registered with the [plugin runtime](plugin-runtime.md); `GET /api/plugins/builtin` says each one (`builtinJSON`: on is active, the user's own choice, what of it is in use) and `POST {id, on}` (or `{id, config}`) writes plugins.yml. A page marks what goes with a plugin `data-feature="<id>"`. Off, `featureStyle` hides everything so marked (rows drawn later too), `navHides` leaves its page's tab out of the sidebar and the header, `show` opens Agents for an address kept for it, and `setSetTab`, `partShows` and `findSettings` leave its Settings tab and rows out, of the search too. One in use stays on: its switch is disabled and the row says what uses it, and the POST refuses it off. A plugins.yml that doesn't parse is said above the list and no switch moves. `boot.js` says which are off (`featuresOff`) and their settings (`pluginConfig`) before the first paint, and `settings.builtins` carries the list on every state read. The TUI doesn't follow them. See `builtins.test.cjs`, `builtins_test.go`. | [`builtins.go`](../../internal/gui/builtins.go), [`assets/plugins.js`](../../internal/gui/assets/plugins.js), [`assets/app.js`](../../internal/gui/assets/app.js) || No GUI build | `-tags nogui` builds magpie without the GUI: `hasGUI` is false and `runGUI` says to use `magpie tui`. The files of `internal/gui` that need Wails build under `!nogui`. | [`gui_off.go`](../../internal/cli/gui_off.go) |diff --git a/internal/gateway/codex_backend.go b/internal/gateway/codex_backend.goindex a8031a91a..fb8f8c38a 100644--- a/internal/gateway/codex_backend.go+++ b/internal/gateway/codex_backend.go@@ -854,7 +854,7 @@ func (s *Server) codexUpstreamOn(w http.ResponseWriter, r *http.Request, rest stw, body, unmask, maskedHits, err = redactedHits(w, body)defer unmask()if err != nil {- writeError(w, provider.Responses, http.StatusServiceUnavailable, err.Error())+ writeError(w, provider.Responses, maskStatus(err), err.Error())return}}diff --git a/internal/gateway/draw.go b/internal/gateway/draw.goindex eeb03824c..d806b75dc 100644--- a/internal/gateway/draw.go+++ b/internal/gateway/draw.go@@ -472,7 +472,7 @@ func (s *Server) images(edit bool) http.HandlerFunc {w, d.Prompt, unmask, hits, err = redactedPromptHits(w, d.Prompt)defer unmask()if err != nil {- fail(http.StatusServiceUnavailable, err.Error())+ fail(maskStatus(err), err.Error())return}noteMasked(r, "image", hits, d.Prompt, 0, p.ID, call.Model)diff --git a/internal/gateway/embeddings.go b/internal/gateway/embeddings.goindex 13fc06dc1..9d4e6881c 100644--- a/internal/gateway/embeddings.go+++ b/internal/gateway/embeddings.go@@ -183,8 +183,8 @@ func (s *Server) serveRetrieval(w http.ResponseWriter, r *http.Request, path, opw, done, hits, sent, err = req.mask(w)defer done()if err != nil { // fails closed- call.Status, call.Error = http.StatusServiceUnavailable, err.Error()- writeError(w, provider.Chat, http.StatusServiceUnavailable, err.Error())+ call.Status, call.Error = maskStatus(err), err.Error()+ writeError(w, provider.Chat, call.Status, err.Error())s.record(call)return}diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.goindex a9a868a2c..ae256889d 100644--- a/internal/gateway/gateway.go+++ b/internal/gateway/gateway.go@@ -1279,7 +1279,7 @@ func (s *Server) countTokens(w http.ResponseWriter, r *http.Request) {w, body, unmask, err := redacted(w, body)defer unmask()if err != nil {- writeError(w, provider.Anthropic, http.StatusServiceUnavailable, err.Error())+ writeError(w, provider.Anthropic, maskStatus(err), err.Error())return}named := unprefixed(model)@@ -1610,7 +1610,7 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Prodefer unmask()if err != nil { // fails closed: never sent unmaskedlog.Printf("gateway: %v", err)- writeError(w, from, http.StatusServiceUnavailable, err.Error())+ writeError(w, from, maskStatus(err), err.Error())return}// the reply's model the member that answered, when asked for (#822)diff --git a/internal/gateway/mask_actions_test.go b/internal/gateway/mask_actions_test.gonew file mode 100644index 000000000..41f5fb5f8--- /dev/null+++ b/internal/gateway/mask_actions_test.go@@ -0,0 +1,98 @@+package gateway++import (+ "io"+ "net/http"+ "net/http/httptest"+ "strings"+ "sync/atomic"+ "testing"++ "github.com/yetone/magpie/internal/maskrec"+ "github.com/yetone/magpie/internal/provider"+ "github.com/yetone/magpie/internal/settings"+)++// A request with a value of a kind the user set to block goes nowhere,+// on each protocol: 400, a message that names the kind and the value's+// hint but never the value, and a record of it held back (#1977).+func TestBlockedKindHoldsTheRequestBack(t *testing.T) {+ const key = "sk-proj-abcdEFGH1234ijklMNOP5678qrst"+ var hit atomic.Int32+ up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ hit.Add(1)+ w.WriteHeader(500)+ }))+ defer up.Close()+ t.Setenv("XDG_CONFIG_HOME", t.TempDir())+ t.Setenv("XDG_CACHE_HOME", t.TempDir())+ if err := provider.Save(provider.Provider{ID: "fake", Name: "Fake", Key: "k", Models: []string{"m1", "embed-1"}, Chat: up.URL + "/v1"}); err != nil {+ t.Fatal(err)+ }+ if err := settings.Save(settings.Settings{Redact: true, RedactActions: map[string]string{"API_KEY": "block"}}); err != nil {+ t.Fatal(err)+ }+ s := New()+ for _, tc := range []struct{ path, body string }{+ {"/v1/chat/completions", `{"model":"fake/m1","messages":[{"role":"user","content":"key ` + key + `"}]}`},+ {"/v1/messages", `{"model":"fake/m1","max_tokens":10,"messages":[{"role":"user","content":"key ` + key + `"}]}`},+ {"/v1/responses", `{"model":"fake/m1","input":"key ` + key + `"}`},+ {"/v1/embeddings", `{"model":"fake/embed-1","input":"key ` + key + `"}`},+ } {+ rec := httptest.NewRecorder()+ s.Handler().ServeHTTP(rec, httptest.NewRequest("POST", tc.path, strings.NewReader(tc.body)))+ raw := rec.Body.String()+ if rec.Code != 400 || strings.Contains(raw, key[3:]) || !strings.Contains(raw, "API_KEY") || !strings.Contains(raw, "sk-") {+ t.Fatalf("%s: %d %s", tc.path, rec.Code, raw)+ }+ rs, _ := maskrec.List(0, 1, maskrec.Filter{})+ if len(rs) != 1 || !rs[0].Blocked || len(rs[0].Values) != 1 || rs[0].Values[0].Action != "block" {+ t.Fatalf("%s: recorded %+v", tc.path, rs)+ }+ }+ if n := hit.Load(); n != 0 {+ t.Fatalf("the vendor was sent %d requests", n)+ }+}++// A kind the user set to warn of goes to the vendor as written, and the+// record tells it, with no snip of it; the rest is masked.+func TestWarnedKindGoesAsWritten(t *testing.T) {+ const key, mail = "sk-proj-abcdEFGH1234ijklMNOP5678qrst", "[email protected]"+ var got []byte+ up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ got, _ = io.ReadAll(r.Body)+ w.Header().Set("Content-Type", "application/json")+ io.WriteString(w, `{"id":"c1","model":"m1","choices":[{"index":0,"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}],"usage":{"prompt_tokens":3,"completion_tokens":1}}`)+ }))+ defer up.Close()+ t.Setenv("XDG_CONFIG_HOME", t.TempDir())+ t.Setenv("XDG_CACHE_HOME", t.TempDir())+ if err := provider.Save(provider.Provider{ID: "fake", Name: "Fake", Key: "k", Models: []string{"m1"}, Chat: up.URL + "/v1"}); err != nil {+ t.Fatal(err)+ }+ if err := settings.Save(settings.Settings{Redact: true, RedactPersonal: true, RedactActions: map[string]string{"EMAIL": "warn"}}); err != nil {+ t.Fatal(err)+ }+ s := New()+ rec := httptest.NewRecorder()+ s.Handler().ServeHTTP(rec, httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(`{"model":"fake/m1","messages":[{"role":"user","content":"mail `+mail+` key `+key+`"}]}`)))+ if rec.Code != 200 || !strings.Contains(string(got), mail) || strings.Contains(string(got), key) {+ t.Fatalf("%d, sent %s", rec.Code, got)+ }+ rs, _ := maskrec.List(0, 1, maskrec.Filter{})+ if len(rs) != 1 || rs[0].N != 1 || len(rs[0].Values) != 2 {+ t.Fatalf("recorded %+v", rs)+ }+ for _, v := range rs[0].Values {+ if v.Kind == "EMAIL" && (v.Action != "warn" || rs[0].Snips[v.Placeholder] != "") {+ t.Fatalf("warned value %+v, snip %q", v, rs[0].Snips[v.Placeholder])+ }+ }+ ts := s.Trace(t.Context(), 0, 0)+ for _, rt := range ts.Routes {+ if rt.ID == rs[0].Route && (rt.Masked == nil || rt.Masked.Warned != 1 || rt.Masked.Distinct != 1) {+ t.Fatalf("route's masked: %+v", rt.Masked)+ }+ }+}diff --git a/internal/gateway/masked.go b/internal/gateway/masked.goindex d6799245c..ca4b7199f 100644--- a/internal/gateway/masked.go+++ b/internal/gateway/masked.go@@ -17,8 +17,9 @@ import (type Masked struct {N int `json:"n"`Distinct int `json:"distinct"`- Kinds map[string]int `json:"kinds,omitempty"` // different values by kind- By map[string]int `json:"by,omitempty"` // different values by finder+ Warned int `json:"warned,omitempty"` // places a value went as written, the user told+ Kinds map[string]int `json:"kinds,omitempty"` // different values by kind+ By map[string]int `json:"by,omitempty"` // different values by finderKey int64 `json:"key,omitempty"`}@@ -42,17 +43,20 @@ func noteMasked(r *http.Request, what string, h maskHits, sent string, route intif h.n == 0 && len(h.hits) == 0 {return nil}- rec := maskrec.Record{What: what, Route: route, Provider: prov, Model: model, N: max(h.n, len(h.hits))}+ rec := maskrec.Record{What: what, Route: route, Provider: prov, Model: model, N: h.n}if r != nil {rec.Agent, rec.Session = agentOf(r), sessionOf(r.Header)}- for _, x := range h.hits {- rec.Values = append(rec.Values, maskrec.Value{Placeholder: x.Placeholder, Kind: x.Kind, By: x.By, Head: x.Head, Len: x.Len})- }+ rec.Values = valuesOf(h.hits)rec.Snips = snipsOf(sent, h.hits)rec = maskrec.Add(rec)- m := &Masked{N: rec.N, Distinct: len(rec.Values), Key: rec.Key}+ m := &Masked{N: rec.N, Key: rec.Key}for _, v := range rec.Values {+ if v.Action == "warn" {+ m.Warned += v.N+ continue+ }+ m.Distinct++if m.Kinds == nil {m.Kinds, m.By = map[string]int{}, map[string]int{}}@@ -62,6 +66,20 @@ func noteMasked(r *http.Request, what string, h maskHits, sent string, route intreturn m}+// noteBlocked records a request held back for a value the user set to+// block: what it had, nothing of what it would have sent.+func noteBlocked(h maskHits) {+ maskrec.Add(maskrec.Record{What: "blocked", N: h.n, Values: valuesOf(h.hits), Blocked: true})+}++func valuesOf(hits []gwmw.Hit) []maskrec.Value {+ var out []maskrec.Value+ for _, x := range hits {+ out = append(out, maskrec.Value{Placeholder: x.Placeholder, Kind: x.Kind, By: x.By, Head: x.Head, Len: x.Len, Action: x.Action})+ }+ return out+}+// snipRunes is how much of the text on each side of a value its snip has.const snipRunes = 56diff --git a/internal/gateway/redact.go b/internal/gateway/redact.goindex d1d8c8c16..32031bbf8 100644--- a/internal/gateway/redact.go+++ b/internal/gateway/redact.go@@ -1,6 +1,8 @@package gatewayimport (+ "errors"+ "fmt""net/http""github.com/yetone/magpie/internal/gwmw"@@ -33,14 +35,20 @@ func redactedHits(w http.ResponseWriter, body []byte) (http.ResponseWriter, []byo, err := gwmw.Safely(func() out {if t, ok := m.(gwmw.Teller); ok {b, hits := t.JSONHits(body)- return out{b, maskHits{len(hits), hits}}+ return out{b, hitsOf(hits)}}b, n := m.JSON(body)return out{b, maskHits{n: n}}})- if err != nil || o.h.n == 0 {+ if err == nil {+ err = blocked(o.h)+ }+ if err != nil {return w, body, func() {}, maskHits{}, err}+ if o.h.n == 0 { // only values sent as written: told, nothing to put back+ return w, body, func() {}, o.h, nil+ }rw, done := m.Writer(w)return rw, o.b, done, o.h, nil}@@ -56,9 +64,12 @@ func redactedPrompt(w http.ResponseWriter, prompt string) (http.ResponseWriter,// redactedPromptHits is redactedPrompt with what it swapped.func redactedPromptHits(w http.ResponseWriter, prompt string) (http.ResponseWriter, string, func(), maskHits, error) {m, masked, h, err := maskTextHits(prompt)- if err != nil || h.n == 0 {+ if err != nil {return w, prompt, func() {}, maskHits{}, err}+ if h.n == 0 {+ return w, prompt, func() {}, h, nil+ }rw, done := m.Writer(w)return rw, masked, done, h, nil}@@ -84,17 +95,67 @@ func maskTextHits(s string) (gwmw.Masker, string, maskHits, error) {o, err := gwmw.Safely(func() out {if t, ok := m.(gwmw.Teller); ok {got, hits := t.TextHits(s)- return out{got, maskHits{len(hits), hits}}+ return out{got, hitsOf(hits)}}got, n := m.Text(s)return out{got, maskHits{n: n}}})+ if err == nil {+ err = blocked(o.h)+ }if err != nil {return nil, s, maskHits{}, err}return m, o.s, o.h, nil}+// hitsOf are a Teller's hits, counting the places masked: a value the+// user set to warn about goes as written, so it is told, not counted.+func hitsOf(hits []gwmw.Hit) maskHits {+ h := maskHits{hits: hits}+ for _, x := range hits {+ if x.Action != "warn" {+ h.n+++ }+ }+ return h+}++// maskBlocked is a request held back: it has a value of a kind the user+// set to block (Settings › Privacy). It names the kind and the value's+// hint, never the value.+type maskBlocked struct{ hit gwmw.Hit }++func (e maskBlocked) Error() string {+ one := fmt.Sprintf("%d characters", e.hit.Len)+ if e.hit.Head != "" {+ one = e.hit.Head + "…, " + one+ }+ return fmt.Sprintf("magpie held this request back: Settings › Privacy blocks %s, and it has one (%s). Nothing was sent.", e.hit.Kind, one)+}++// blocked is the error for hits with a value to hold the request back+// for, recorded as a request held back; nil without one.+func blocked(h maskHits) error {+ for _, x := range h.hits {+ if x.Action == "block" {+ noteBlocked(h)+ return maskBlocked{x}+ }+ }+ return nil+}++// maskStatus is the status a request masking failed for gets: 400 for+// one held back, which sent again is held back again, 503 for a masker+// that couldn't mask it.+func maskStatus(err error) int {+ if errors.As(err, new(maskBlocked)) {+ return http.StatusBadRequest+ }+ return http.StatusServiceUnavailable+}+// restored is s with what a masker masked before put back; s when none// serves.func restored(s string, escaped bool) string {diff --git a/internal/gateway/video.go b/internal/gateway/video.goindex 803b077d5..6c13012fc 100644--- a/internal/gateway/video.go+++ b/internal/gateway/video.go@@ -741,7 +741,7 @@ func (s *Server) videosCreate(w http.ResponseWriter, r *http.Request) {err = maskStrings(f.Extra, &extra)}if err != nil { // fails closed- fail(http.StatusServiceUnavailable, err.Error())+ fail(maskStatus(err), err.Error())return}n := extra.ndiff --git a/internal/gui/api.go b/internal/gui/api.goindex b0cd2bc82..e3e61cbe0 100644--- a/internal/gui/api.go+++ b/internal/gui/api.go@@ -1218,6 +1218,8 @@ func Handler(w Windows, gw *gateway.Server) http.Handler {in.GatewayConversations = cur.GatewayConversations // Sessions' explicit recording consentin.RedactRules = cur.RedactRules // the masking rules, set on their ownin.RedactModel = cur.RedactModel // set on its own (privacy-model)+ in.RedactActions = cur.RedactActions // set on their own (redact-actions below)+ in.RedactAllow = cur.RedactAllow // set on its own (redact-allow below)in.DisabledModels = cur.DisabledModels // set on their own (disabled-models below, #1867)// used or left is the Usage page's toggle as much as Settings', set on its ownin.QuotaLeft = cur.QuotaLeft@@ -2093,6 +2095,41 @@ func Handler(w Windows, gw *gateway.Server) http.Handler {}writeJSON(rw, settingsState())})+ // what is done with a value of a kind found, other than masking it+ // (warn, block), all of them each time (#1977)+ mux.HandleFunc("POST /api/settings/redact-actions", func(rw http.ResponseWriter, r *http.Request) {+ var in struct {+ Actions map[string]string `json:"actions"`+ }+ if err := json.NewDecoder(r.Body).Decode(&in); err != nil {+ fail(rw, err)+ return+ }+ s := settings.Load()+ s.RedactActions = in.Actions+ if err := settings.Save(s); err != nil {+ fail(rw, err)+ return+ }+ writeJSON(rw, settingsState())+ })+ // the values masking lets go as written, all of them each time (#1977)+ mux.HandleFunc("POST /api/settings/redact-allow", func(rw http.ResponseWriter, r *http.Request) {+ var in struct {+ Allow []string `json:"allow"`+ }+ if err := json.NewDecoder(r.Body).Decode(&in); err != nil {+ fail(rw, err)+ return+ }+ s := settings.Load()+ s.RedactAllow = in.Allow+ if err := settings.Save(s); err != nil {+ fail(rw, err)+ return+ }+ writeJSON(rw, settingsState())+ })// the GitHub token the library's requests to GitHub carry: set, or// taken away with ""mux.HandleFunc("POST /api/settings/github-token", func(rw http.ResponseWriter, r *http.Request) {diff --git a/internal/gui/assets/app.css b/internal/gui/assets/app.cssindex cef51c57f..e2a6f7b01 100644--- a/internal/gui/assets/app.css+++ b/internal/gui/assets/app.css@@ -3336,6 +3336,25 @@ body:not(.side-nav) #view-settings .set-head.stuck::before { box-shadow: 0 1px 0.mr-shape.known:focus-visible { background: var(--pill); }.mr-by { display: inline-flex; align-items: center; height: 18px; padding: 0 6px; border-radius: 5px; font-size: 10.5px; color: var(--muted); background: var(--pill); }.mr-by.model { color: var(--c6); background: color-mix(in srgb, var(--c6) 14%, transparent); }+.mr-act { display: inline-flex; align-items: center; height: 18px; padding: 0 6px; border-radius: 5px; font-size: 10.5px; color: var(--muted); background: var(--pill); }+.mr-act.warn { color: var(--amber); background: var(--amber-soft); }+.mr-act.block { color: var(--red); background: var(--red-soft); }+.mr-route .what.held { color: var(--red); }+/* what is done with a kind of value: mask, warn or block (#1977) */+.mask-act .name { display: flex; align-items: center; gap: 8px; }+#maskActList .mask-act .who .sub.act-warn { color: var(--amber); }+#maskActList .mask-act .who .sub.act-block { color: var(--red); }+.mask-allow { width: min(320px, 100%); }+.mask-try { flex-wrap: wrap; row-gap: 10px; }+.mask-try > .who { flex: 1 1 100%; }+.mask-try > .who .sub { white-space: normal; }+.mask-try-in { flex: 1 1 100%; width: 100%; box-sizing: border-box; min-height: 68px; padding: 8px 10px; border-radius: 8px; border: 1px solid var(--line); background: var(--card); color: var(--fg); outline: none; font-family: var(--code); font-size: 12px; line-height: 1.5; resize: vertical; transition: border-color .12s, box-shadow .12s; }+.mask-try-in:focus { border-color: var(--accent); box-shadow: 0 0 0 3px color-mix(in srgb, var(--accent) 18%, transparent); }+.mask-try-out { flex: 1 1 100%; min-width: 0; display: flex; flex-direction: column; gap: 8px; }+.mask-try-text { padding: 9px 11px; border-radius: 8px; border: 1px solid var(--line); font-family: var(--code); font-size: 12px; line-height: 1.6; white-space: pre-wrap; overflow-wrap: anywhere; color: var(--fg-2); }+.mask-try-hits { display: flex; flex-direction: column; }+.mask-try-note { font-size: 11.5px; color: var(--muted); }+.mask-try-note.block { color: var(--red); font-weight: 500; }.mr-x { color: var(--muted); font-variant-numeric: tabular-nums; }.mr-snip { margin-top: 8px; padding: 9px 11px; border-radius: 8px; border: 1px solid var(--line); background: var(--card-2); font-family: var(--code); font-size: 11.5px; line-height: 1.65; color: var(--fg-2); white-space: pre-wrap; overflow-wrap: anywhere; }.mr-ph { --k: var(--accent); padding: 1px 4px; border-radius: 4px; color: var(--fg); background: color-mix(in srgb, var(--k) 16%, transparent); box-shadow: inset 0 0 0 1px color-mix(in srgb, var(--k) 30%, transparent); outline: none; cursor: default; }@@ -3354,6 +3373,8 @@ body:not(.side-nav) #view-settings .set-head.stuck::before { box-shadow: 0 1px 0.mask-badge { flex: none; display: inline-flex; align-items: center; gap: 3px; height: 18px; padding: 0 6px 0 5px; border: 0; border-radius: 99px; background: var(--accent-soft); color: var(--accent); font: inherit; font-size: 11px; font-weight: 600; line-height: 1; font-variant-numeric: tabular-nums; cursor: default; }.mask-badge:hover { background: color-mix(in srgb, var(--accent) 22%, transparent); }.mask-badge:focus-visible { outline: none; box-shadow: 0 0 0 2px var(--accent-soft), inset 0 0 0 1px var(--accent); }+.mask-badge.warn { background: var(--amber-soft); color: var(--amber); }+.mask-badge.warn:hover { background: color-mix(in srgb, var(--amber) 22%, transparent); }/* a plugin's settings, under its row (kernel.js ui.form) */.row.pref.feat-set { min-height: 44px; padding-left: 30px; }.row.pref.feat-set .name { font-size: 12.5px; }diff --git a/internal/gui/assets/app.js b/internal/gui/assets/app.jsindex cc80de97a..1d6fdbddb 100644--- a/internal/gui/assets/app.js+++ b/internal/gui/assets/app.js@@ -21786,6 +21786,8 @@ function renderRedact(s, keep) {i.onblur = save;row(t("Masked words"), t("Your own words to keep from vendors, separated by commas"), i);renderRedactRules(s, row);+ renderMaskActions(s);+ renderMaskTry(s);// how it is found: the rules on every request that masks anything, and// the local model on top of them, for what no pattern can tellconst masking = s.redact || s.redactPersonal;@@ -22067,7 +22069,9 @@ function drawMaskObs() {const num = el("span", "mo-num", fmtN(sum.n));big.append(num, el("span", "mo-unit", t(sum.n === 1 ? "value masked" : "values masked")));count.append(big, el("div", "mo-meta", sum.requests- ? t(sum.distinct === 1 ? "1 different value" : "{n} different values", { n: fmtN(sum.distinct) }) + " · " + t(sum.requests === 1 ? "1 request" : "{n} requests", { n: fmtN(sum.requests) })+ ? [t(sum.distinct === 1 ? "1 different value" : "{n} different values", { n: fmtN(sum.distinct) }), t(sum.requests === 1 ? "1 request" : "{n} requests", { n: fmtN(sum.requests) }),+ sum.blocked ? t(sum.blocked === 1 ? "1 request held back" : "{n} requests held back", { n: fmtN(sum.blocked) }) : "",+ sum.warned ? t(sum.warned === 1 ? "1 value sent as written" : "{n} values sent as written", { n: fmtN(sum.warned) }) : ""].filter(Boolean).join(" · "): t("Nothing yet in these days")));const chart = el("div", "mo-days");hero.append(count, chart);@@ -22497,7 +22501,8 @@ function maskRecordRow(r, v) {if (p.icon && r.provider) route.append(icon(p.icon));route.append(el("span", "who", r.provider ? p.name : t("A vendor")));if (r.model) route.append(el("span", "model", "· " + r.model));- if (r.what && r.what !== "chat") route.append(el("span", "what", MASK_WHAT[r.what] ? t(MASK_WHAT[r.what]) : r.what));+ if (r.blocked) route.append(el("span", "what held", t("Held back")));+ else if (r.what && r.what !== "chat") route.append(el("span", "what", MASK_WHAT[r.what] ? t(MASK_WHAT[r.what]) : r.what));const kinds = el("span", "mr-kinds");const by = {};for (const x of r.values || []) by[x.k] = (by[x.k] || 0) + (x.n || 1);@@ -22548,6 +22553,8 @@ function maskRecordBody(r) {}line.append(maskDot(x.k), el("span", "mr-k", maskKindName(x.k)), shape, el("span", "mr-by" + (x.b === "model" ? " model" : ""), maskByName(x.b)));if (x.n > 1) line.append(el("span", "mr-x", "×" + fmtN(x.n)));+ if (x.a === "warn") line.append(el("span", "mr-act warn", t("Sent as written")));+ else if (x.a === "block") line.append(el("span", "mr-act block", t("Held back")));vals.append(line);}body.append(vals);@@ -22571,11 +22578,15 @@ function maskRecordBody(r) {// maskedBadge: what masking swapped in one request (gateway's Masked), for// Recent calls and the Routing page; it opens that request's recordfunction maskedBadge(m) {- const b = el("button", "mask-badge");+ // a request whose values only went as written, which the user set a kind+ // to do, is told in amber with how many+ const warned = !m.n && m.warned > 0;+ const b = el("button", "mask-badge" + (warned ? " warn" : ""));b.type = "button";- b.append(svg(SHIELD, 11, 1.7), el("span", "", fmtN(m.n)));+ b.append(svg(SHIELD, 11, 1.7), el("span", "", fmtN(warned ? m.warned : m.n)));const kinds = Object.entries(m.kinds || {}).sort((x, y) => y[1] - x[1]).map(([k, n]) => `${maskKindName(k)} ${n}`).join(", ");- b.title = t(m.n === 1 ? "1 value masked before it went to the vendor: {kinds}" : "{n} values masked before it went to the vendor: {kinds}", { n: fmtN(m.n), kinds });+ b.title = warned ? t(m.warned === 1 ? "1 value went to the vendor as written, as you set" : "{n} values went to the vendor as written, as you set", { n: fmtN(m.warned) })+ : t(m.n === 1 ? "1 value masked before it went to the vendor: {kinds}" : "{n} values masked before it went to the vendor: {kinds}", { n: fmtN(m.n), kinds });b.setAttribute("aria-label", b.title);b.onclick = (e) => { e.stopPropagation(); openMaskRecord(m.key); };b.onkeydown = (e) => e.stopPropagation(); // Enter on it isn't the row's@@ -22721,6 +22732,179 @@ function renderRedactRules(s, row) {});}+// ---------- what is done with a value found, and trying it (#1977) ----------+// Each kind masking finds is masked unless the user chose otherwise: Warn+// sends it as written and records it, Block holds the whole request back.+// The secrets, the user's own rules' kinds and any kind already set are+// shown; the rest wait behind "Show all kinds".+const MASK_SECRET_KINDS = ["API_KEY", "PRIVATE_KEY", "TOKEN", "PASSWORD", "SECRET"];+const MASK_PERSONAL_KINDS = ["EMAIL", "PHONE", "ID_CARD", "BANK_CARD", "PERSON", "ADDRESS", "ACCOUNT", "SSN", "PASSPORT", "IBAN", "BIRTHDAY", "MAC", "SERIAL", "USERNAME", "HOSTNAME", "IP", "BUCKET"];+const MASK_ACT_SUB = {+ mask: "Goes to the vendor as a placeholder, and comes back as it was",+ warn: "Goes to the vendor as written, and is recorded below",+ block: "The request is held back: nothing is sent, and the agent is told why",+};+let maskActAll = false;+// a rule's kind as its placeholders have it (redact.RuleKind)+function maskRuleKind(k) {+ const x = String(k || "").trim().toUpperCase().replace(/[ .-]/g, "_").replace(/[^A-Z0-9_]/g, "").replace(/^_+|_+$/g, "");+ return x || "CUSTOM";+}+function maskActKinds(s) {+ const acts = s.redactActions || {}, first = [], rest = [];+ const add = (k, front) => {+ if (first.includes(k) || rest.includes(k)) return;+ (front || acts[k] ? first : rest).push(k);+ };+ if (s.redact) {+ for (const k of MASK_SECRET_KINDS) add(k, true);+ for (const r of s.redactRules || []) add(maskRuleKind(r.kind), true);+ }+ if (s.redactWords?.length) add("TERM", true);+ if (s.redactPersonal) for (const k of MASK_PERSONAL_KINDS) add(k, false);+ // a choice kept for a kind not masked now stays in view, to be undone+ for (const k of Object.keys(acts)) add(k, true);+ return { first, rest };+}+function renderMaskActions(s) {+ const box = $("#maskActList");+ if (!box) return;+ box.replaceChildren();+ const acts = s.redactActions || {};+ const set = (next) => writingPrefs(api("settings/redact-actions", { actions: next }))+ .then((ns) => { prefs = ns; renderSettings(); status(t("Saved"), "ok", 1500); })+ .catch((e) => { status(t(e.message), "err"); renderSettings(); });+ const row = (name, sub, ...tools) => {+ const r = el("div", "row pref");+ const who = el("div", "who"), val = el("div", "val");+ who.append(el("div", "name", name), el("div", "sub", sub));+ val.append(...tools);+ r.append(who, val);+ box.append(r);+ return r;+ };+ const { first, rest } = maskActKinds(s);+ if (!first.length && !rest.length) {+ row(t("Mask everything found"), t("Turn on Mask secrets or Mask personal data first"));+ }+ for (const k of maskActAll ? [...first, ...rest] : first) {+ const now = acts[k] || "mask";+ const r = row(maskKindName(k), t(MASK_ACT_SUB[now]),+ segs([["mask", t("Mask")], ["warn", t("Warn")], ["block", t("Block")]], now, (v) => {+ const next = { ...acts };+ if (v === "mask") delete next[k]; else next[k] = v;+ set(next);+ }));+ r.classList.add("mask-act");+ r.dataset.kind = k;+ r.querySelector(".name").prepend(maskDot(k));+ if (now !== "mask") r.querySelector(".sub").classList.add(now === "block" ? "act-block" : "act-warn");+ }+ if (rest.length) {+ const more = el("button", "text", maskActAll ? t("Show fewer") : t("Show all kinds ({n} more)", { n: rest.length }));+ more.onclick = () => { maskActAll = !maskActAll; renderSettings(); };+ row(t("Personal data"), t("Emails, phone numbers, names and the rest, masked unless set here"), more).classList.add("mask-act-more");+ }+ // the allow list: values sent as written whichever finder finds them+ const allow = (s.redactAllow || []).join(", ");+ const i = input(allow, t("[email protected], build-01"));+ i.className = "words mask-allow";+ i.setAttribute("aria-label", t("Never mask"));+ const save = () => {+ const v = i.value.split(/[,,\n]/).map((w) => w.trim()).filter(Boolean);+ if (v.join(", ") === allow) return;+ writingPrefs(api("settings/redact-allow", { allow: v }))+ .then((ns) => { prefs = ns; renderSettings(); status(t("Saved"), "ok", 1500); })+ .catch((e) => { status(t(e.message), "err"); renderSettings(); });+ };+ i.onkeydown = (e) => { e.stopPropagation(); if (e.key === "Enter") save(); else if (e.key === "Escape") { i.value = allow; i.blur(); } };+ i.onblur = save;+ row(t("Never mask"), t("Values that always go as written, each whole and in any case, separated by commas: a team address, a test key"), i);+}++// renderMaskTry: text typed here is masked as a request would be now, and+// shown as the vendor would get it, with what was found and what would be+// done. It is sent to this magpie only, kept nowhere, and nothing of it+// is remembered (/api/masking/preview).+let maskTry = { text: "", res: null, seq: 0, timer: 0 };+function renderMaskTry(s) {+ const box = $("#maskTryList");+ if (!box) return;+ box.replaceChildren();+ const r = el("div", "row pref mask-try");+ const who = el("div", "who");+ // the section head names it; the row says only what it does+ who.append(el("div", "sub", t("Paste text an agent might send: it is masked here as a request would be, and kept nowhere")));+ const area = el("textarea", "mask-try-in");+ area.value = maskTry.text;+ area.rows = 3;+ area.spellcheck = false;+ area.placeholder = t("Deploy with key sk-proj-… and mail [email protected]");+ area.setAttribute("aria-label", t("Try it"));+ const out = el("div", "mask-try-out");+ out.setAttribute("aria-live", "polite");+ area.onkeydown = (e) => e.stopPropagation();+ area.oninput = () => {+ maskTry.text = area.value;+ clearTimeout(maskTry.timer);+ maskTry.timer = setTimeout(() => maskTryRun(out), 250);+ };+ r.append(who, area, out);+ box.append(r);+ maskTryDraw(out);+}+function maskTryRun(out) {+ const seq = ++maskTry.seq, text = maskTry.text;+ if (!text.trim()) { maskTry.res = null; maskTryDraw(out); return; }+ api("masking/preview", { text }).then((res) => {+ if (seq !== maskTry.seq) return;+ maskTry.res = res;+ maskTryDraw(out);+ }).catch((e) => { if (seq === maskTry.seq) status(t(e.message), "err"); });+}+function maskTryDraw(out) {+ out.replaceChildren();+ const res = maskTry.res;+ if (!res || !maskTry.text.trim()) { out.hidden = true; return; }+ out.hidden = false;+ const hits = res.hits || [];+ const info = new Map(hits.map((h) => [h.placeholder, h]));+ const head = el("div", "mo-lh");+ head.append(el("span", "label", t("What the vendor would get")));+ out.append(head);+ if (hits.some((h) => h.action === "block")) {+ out.append(el("div", "mask-try-note block", t("This request would be held back: nothing is sent")));+ }+ const text = el("div", "mask-try-text");+ let at = 0;+ for (const m of res.text.matchAll(PLACEHOLDER)) {+ text.append(document.createTextNode(res.text.slice(at, m.index)));+ const h = info.get(m[0]);+ const kind = h?.kind || m[0].slice(2, -2).replace(/_[^_]*$/, "");+ const mark = el("mark", "mo-ph", m[0]);+ mark.style.setProperty("--k", maskColor(kind));+ mark.title = maskKindName(kind);+ text.append(mark);+ at = m.index + m[0].length;+ }+ text.append(document.createTextNode(res.text.slice(at)));+ out.append(text);+ if (!hits.length) out.append(el("div", "mask-try-note", t("Nothing here would be masked")));+ // each value once, by its placeholder, with what is done with it+ const seen = new Set(), list = el("div", "mask-try-hits");+ for (const h of hits) {+ if (seen.has(h.placeholder)) continue;+ seen.add(h.placeholder);+ const line = el("div", "mr-val");+ const act = h.action === "block" ? t("Held back") : h.action === "warn" ? t("Sent as written") : t("Masked");+ line.append(maskDot(h.kind), el("span", "mr-k", maskKindName(h.kind)), el("span", "mr-shape", maskShape(h.head, h.len)),+ el("span", "mr-by" + (h.by === "model" ? " model" : ""), maskByName(h.by)), el("span", "mr-act " + (h.action || "mask"), act));+ list.append(line);+ }+ if (seen.size) out.append(list);+ if (res.off) out.append(el("div", "mask-try-note", t("Masking is off: this is what Mask secrets and Mask personal data would do")));+}+// editRedactRule: the rule at n, open to change in its place (nf1qaz2w on// Discord: rules could only be removed) — its name, prefix or regex and// what to match, laid out as the row that adds one. Save sends it alone,diff --git a/internal/gui/assets/i18n.js b/internal/gui/assets/i18n.jsindex 5ec6214d5..19e6dab84 100644--- a/internal/gui/assets/i18n.js+++ b/internal/gui/assets/i18n.js@@ -3765,6 +3765,36 @@ const I18N = {"Yesterday": "昨天","Not in memory: magpie has restarted since. The value itself is never written to disk": "已不在内存中:magpie 之后重启过。原值从不写入磁盘","Kept from vendors": "没让厂商看到的",+ "When a value is found": "发现敏感值时",+ "Mask": "脱敏",+ "Warn": "提醒",+ "Block": "拦截",+ "Goes to the vendor as a placeholder, and comes back as it was": "以占位符发给厂商,回复里再还原",+ "Goes to the vendor as written, and is recorded below": "按原样发给厂商,并记录在下面",+ "The request is held back: nothing is sent, and the agent is told why": "请求被拦下:什么都不发,并告诉智能体原因",+ "Mask everything found": "对找到的都脱敏",+ "Personal data": "个人信息",+ "Emails, phone numbers, names and the rest, masked unless set here": "邮箱、手机号、姓名等,除非在这里另设,都会脱敏",+ "Show all kinds ({n} more)": "显示全部类别(还有 {n} 个)",+ "Never mask": "从不脱敏",+ "[email protected], build-01": "[email protected], build-01",+ "Values that always go as written, each whole and in any case, separated by commas: a team address, a test key": "总是按原样发送的值,整段匹配、不分大小写,用逗号分隔:比如团队邮箱、测试密钥",+ "Try it": "试一试",+ "Paste text an agent might send: it is masked here as a request would be, and kept nowhere": "粘贴一段智能体可能发送的文本:会像真实请求一样在本机脱敏,且不保存",+ "Deploy with key sk-proj-… and mail [email protected]": "用密钥 sk-proj-… 部署,发邮件给 [email protected]",+ "What the vendor would get": "厂商会收到的",+ "This request would be held back: nothing is sent": "这个请求会被拦下:什么都不发",+ "Nothing here would be masked": "这里没有需要脱敏的内容",+ "Held back": "已拦截",+ "Sent as written": "按原样发送",+ "Masked": "已脱敏",+ "Masking is off: this is what Mask secrets and Mask personal data would do": "脱敏已关闭:下面是开启“脱敏密钥”和“脱敏个人信息”后的效果",+ "1 request held back": "1 个请求被拦下",+ "{n} requests held back": "{n} 个请求被拦下",+ "1 value sent as written": "1 个值按原样发送",+ "{n} values sent as written": "{n} 个值按原样发送",+ "1 value went to the vendor as written, as you set": "1 个值按你的设置原样发给了厂商",+ "{n} values went to the vendor as written, as you set": "{n} 个值按你的设置原样发给了厂商","Masked on this computer before each request left it, and put back in each answer": "每个请求离开这台电脑前在本机脱敏,回复里再换回原样","Masking is off: requests go to vendors as they are": "脱敏已关闭:请求原样发给厂商","Turn on Mask secrets or Mask personal data below, and what is kept from vendors shows here, request by request": "打开下方的「脱敏密钥」或「脱敏个人信息」,没让厂商看到的内容会逐个请求显示在这里",@@ -7587,6 +7617,36 @@ const I18N = {"Yesterday": "昨天","Not in memory: magpie has restarted since. The value itself is never written to disk": "已不在記憶體中:magpie 之後重新啟動過。原值從不寫入磁碟","Kept from vendors": "沒讓廠商看到的",+ "When a value is found": "發現敏感值時",+ "Mask": "脫敏",+ "Warn": "提醒",+ "Block": "攔截",+ "Goes to the vendor as a placeholder, and comes back as it was": "以佔位符送給廠商,回覆裡再還原",+ "Goes to the vendor as written, and is recorded below": "照原樣送給廠商,並記錄在下面",+ "The request is held back: nothing is sent, and the agent is told why": "請求被攔下:什麼都不送,並告訴智慧體原因",+ "Mask everything found": "對找到的都脫敏",+ "Personal data": "個人資訊",+ "Emails, phone numbers, names and the rest, masked unless set here": "Email、手機號碼、姓名等,除非在這裡另設,都會脫敏",+ "Show all kinds ({n} more)": "顯示全部類別(還有 {n} 個)",+ "Never mask": "從不脫敏",+ "[email protected], build-01": "[email protected], build-01",+ "Values that always go as written, each whole and in any case, separated by commas: a team address, a test key": "一律照原樣送出的值,整段比對、不分大小寫,用逗號分隔:例如團隊信箱、測試金鑰",+ "Try it": "試試看",+ "Paste text an agent might send: it is masked here as a request would be, and kept nowhere": "貼上一段智慧體可能送出的文字:會像真實請求一樣在本機脫敏,且不保存",+ "Deploy with key sk-proj-… and mail [email protected]": "用金鑰 sk-proj-… 部署,寄信給 [email protected]",+ "What the vendor would get": "廠商會收到的",+ "This request would be held back: nothing is sent": "這個請求會被攔下:什麼都不送",+ "Nothing here would be masked": "這裡沒有需要脫敏的內容",+ "Held back": "已攔截",+ "Sent as written": "照原樣送出",+ "Masked": "已脫敏",+ "Masking is off: this is what Mask secrets and Mask personal data would do": "脫敏已關閉:下面是開啟「脫敏金鑰」和「脫敏個人資訊」後的效果",+ "1 request held back": "1 個請求被攔下",+ "{n} requests held back": "{n} 個請求被攔下",+ "1 value sent as written": "1 個值照原樣送出",+ "{n} values sent as written": "{n} 個值照原樣送出",+ "1 value went to the vendor as written, as you set": "1 個值依你的設定照原樣送給了廠商",+ "{n} values went to the vendor as written, as you set": "{n} 個值依你的設定照原樣送給了廠商","Masked on this computer before each request left it, and put back in each answer": "每個請求離開這台電腦前在本機脫敏,回覆裡再換回原樣","Masking is off: requests go to vendors as they are": "脫敏已關閉:請求原樣發給廠商","Turn on Mask secrets or Mask personal data below, and what is kept from vendors shows here, request by request": "打開下方的「脫敏金鑰」或「脫敏個人資訊」,沒讓廠商看到的內容會逐個請求顯示在這裡",@@ -11411,6 +11471,36 @@ const I18N = {"Yesterday": "昨日","Not in memory: magpie has restarted since. The value itself is never written to disk": "メモリにありません:その後 magpie が再起動しました。元の値はディスクに書き込まれません","Kept from vendors": "ベンダーに渡さなかったもの",+ "When a value is found": "値が見つかったとき",+ "Mask": "マスク",+ "Warn": "警告",+ "Block": "ブロック",+ "Goes to the vendor as a placeholder, and comes back as it was": "プレースホルダーとしてベンダーへ送り、返答で元に戻します",+ "Goes to the vendor as written, and is recorded below": "そのままベンダーへ送り、下に記録します",+ "The request is held back: nothing is sent, and the agent is told why": "リクエストを止めます:何も送らず、エージェントに理由を伝えます",+ "Mask everything found": "見つかったものはすべてマスク",+ "Personal data": "個人情報",+ "Emails, phone numbers, names and the rest, masked unless set here": "メール、電話番号、氏名など。ここで設定しない限りマスクします",+ "Show all kinds ({n} more)": "すべての種類を表示(ほか {n} 件)",+ "Never mask": "マスクしない",+ "[email protected], build-01": "[email protected], build-01",+ "Values that always go as written, each whole and in any case, separated by commas: a team address, a test key": "常にそのまま送る値。全体一致、大文字小文字を区別せず、カンマ区切り:チームのアドレス、テスト用キーなど",+ "Try it": "試す",+ "Paste text an agent might send: it is masked here as a request would be, and kept nowhere": "エージェントが送りそうなテキストを貼り付けてください。実際のリクエストと同じくこのコンピューターでマスクし、どこにも保存しません",+ "Deploy with key sk-proj-… and mail [email protected]": "キー sk-proj-… でデプロイし、[email protected] にメール",+ "What the vendor would get": "ベンダーが受け取る内容",+ "This request would be held back: nothing is sent": "このリクエストは止められます:何も送りません",+ "Nothing here would be masked": "マスクされるものはありません",+ "Held back": "ブロック済み",+ "Sent as written": "そのまま送信",+ "Masked": "マスク済み",+ "Masking is off: this is what Mask secrets and Mask personal data would do": "マスクはオフです:「シークレットをマスク」と「個人情報をマスク」をオンにした場合の結果です",+ "1 request held back": "1 件のリクエストを止めました",+ "{n} requests held back": "{n} 件のリクエストを止めました",+ "1 value sent as written": "1 個の値をそのまま送信",+ "{n} values sent as written": "{n} 個の値をそのまま送信",+ "1 value went to the vendor as written, as you set": "設定どおり、1 個の値をそのままベンダーに送りました",+ "{n} values went to the vendor as written, as you set": "設定どおり、{n} 個の値をそのままベンダーに送りました","Masked on this computer before each request left it, and put back in each answer": "各リクエストがこのコンピュータを出る前にここでマスクし、応答で元に戻します","Masking is off: requests go to vendors as they are": "マスクはオフです:リクエストはそのままベンダーに送られます","Turn on Mask secrets or Mask personal data below, and what is kept from vendors shows here, request by request": "下の「シークレットをマスク」か「個人情報をマスク」をオンにすると、ベンダーに渡さなかったものがリクエストごとにここに表示されます",@@ -15235,6 +15325,36 @@ const I18N = {"Yesterday": "Gestern","Not in memory: magpie has restarted since. The value itself is never written to disk": "Nicht mehr im Speicher: magpie wurde seitdem neu gestartet. Der Wert selbst wird nie auf die Festplatte geschrieben","Kept from vendors": "Vor Anbietern verborgen",+ "When a value is found": "Wenn ein Wert gefunden wird",+ "Mask": "Maskieren",+ "Warn": "Warnen",+ "Block": "Blockieren",+ "Goes to the vendor as a placeholder, and comes back as it was": "Geht als Platzhalter an den Anbieter und kommt wie zuvor zurück",+ "Goes to the vendor as written, and is recorded below": "Geht unverändert an den Anbieter und wird unten festgehalten",+ "The request is held back: nothing is sent, and the agent is told why": "Die Anfrage wird zurückgehalten: nichts wird gesendet, der Agent erfährt warum",+ "Mask everything found": "Alles Gefundene maskieren",+ "Personal data": "Personenbezogene Daten",+ "Emails, phone numbers, names and the rest, masked unless set here": "E-Mails, Telefonnummern, Namen und mehr, maskiert, sofern hier nicht anders eingestellt",+ "Show all kinds ({n} more)": "Alle Arten zeigen ({n} weitere)",+ "Never mask": "Nie maskieren",+ "[email protected], build-01": "[email protected], build-01",+ "Values that always go as written, each whole and in any case, separated by commas: a team address, a test key": "Werte, die immer unverändert gehen, jeweils ganz und ohne Groß-/Kleinschreibung, durch Kommas getrennt: eine Team-Adresse, ein Testschlüssel",+ "Try it": "Ausprobieren",+ "Paste text an agent might send: it is masked here as a request would be, and kept nowhere": "Text einfügen, den ein Agent senden könnte: er wird hier wie eine Anfrage maskiert und nirgends gespeichert",+ "Deploy with key sk-proj-… and mail [email protected]": "Mit Schlüssel sk-proj-… deployen und an [email protected] mailen",+ "What the vendor would get": "Was der Anbieter bekäme",+ "This request would be held back: nothing is sent": "Diese Anfrage würde zurückgehalten: nichts wird gesendet",+ "Nothing here would be masked": "Hier würde nichts maskiert",+ "Held back": "Zurückgehalten",+ "Sent as written": "Unverändert gesendet",+ "Masked": "Maskiert",+ "Masking is off: this is what Mask secrets and Mask personal data would do": "Maskieren ist aus: So würden „Geheimnisse maskieren“ und „Personenbezogene Daten maskieren“ wirken",+ "1 request held back": "1 Anfrage zurückgehalten",+ "{n} requests held back": "{n} Anfragen zurückgehalten",+ "1 value sent as written": "1 Wert unverändert gesendet",+ "{n} values sent as written": "{n} Werte unverändert gesendet",+ "1 value went to the vendor as written, as you set": "1 Wert ging wie eingestellt unverändert an den Anbieter",+ "{n} values went to the vendor as written, as you set": "{n} Werte gingen wie eingestellt unverändert an den Anbieter","Masked on this computer before each request left it, and put back in each answer": "Auf diesem Computer maskiert, bevor jede Anfrage ihn verlässt, und in jeder Antwort zurückgesetzt","Masking is off: requests go to vendors as they are": "Maskierung ist aus: Anfragen gehen unverändert an die Anbieter","Turn on Mask secrets or Mask personal data below, and what is kept from vendors shows here, request by request": "Schalte unten „Geheimnisse maskieren“ oder „Personenbezogene Daten maskieren“ ein, dann zeigt sich hier pro Anfrage, was vor Anbietern verborgen blieb",diff --git a/internal/gui/assets/index.html b/internal/gui/assets/index.htmlindex 72738f7cd..d00e4e08e 100644--- a/internal/gui/assets/index.html+++ b/internal/gui/assets/index.html@@ -378,6 +378,10 @@<div class="mask-recs" id="maskRecs" hidden></div><div class="row-head"><span class="label" data-t>What to mask</span></div><div class="list prefs" id="redactList"></div>+ <div class="row-head"><span class="label" data-t>When a value is found</span></div>+ <div class="list prefs" id="maskActList"></div>+ <div class="row-head"><span class="label" data-t>Try it</span></div>+ <div class="list prefs" id="maskTryList"></div><div class="row-head"><span class="label" data-t>How it's found</span></div><div class="list prefs" id="findList"></div><div class="row-head"><span class="label" data-t>Sharing</span></div>diff --git a/internal/gui/assets/plugins/gateway/client.js b/internal/gui/assets/plugins/gateway/client.jsindex 41d26cfa9..748fb9a1a 100644--- a/internal/gui/assets/plugins/gateway/client.js+++ b/internal/gui/assets/plugins/gateway/client.js@@ -593,7 +593,7 @@ function renderActivity() {r.append(el("span", "m", c.model));r.append(el("span", "p", c.from === c.to ? c.from : `${c.from} → ${c.to}`));// what masking kept from the vendor, its record a click away- if (c.masked?.n) r.append(maskedBadge(c.masked));+ if (c.masked?.n || c.masked?.warned) r.append(maskedBadge(c.masked));r.append(el("span", "grow"));r.append(el("span", "st", c.error ? `${c.status} ${c.error}` : `${c.status} · ${ledTook(c.ms)}` + (c.ttft ? " · " + t("TTFT {ms}", { ms: ledTook(c.ttft) }) : "")));r.title = open ? t("Hide request and response bodies") : t("Show request and response bodies");diff --git a/internal/gui/assets/plugins/routing/client.js b/internal/gui/assets/plugins/routing/client.jsindex 9b3c02a08..4e6e9d13b 100644--- a/internal/gui/assets/plugins/routing/client.js+++ b/internal/gui/assets/plugins/routing/client.js@@ -1902,7 +1902,7 @@ function routingPage(part) {el("span", "", rp ? t("How the request at {time} was routed", { time: clock(r.time) }): pinned ? t("How the request at {time} was routed", { time: clock(r.time) }) : t("How the last request was routed")));// what masking kept from the vendor, its record a click away- if (r.masked?.n && !rp) logHead.append(maskedBadge(r.masked));+ if ((r.masked?.n || r.masked?.warned) && !rp) logHead.append(maskedBadge(r.masked));logHead.append(el("span", "grow"));if (!rp && r.done) {const usage = el("button", "text", t("View usage"));diff --git a/internal/gui/masking.go b/internal/gui/masking.goindex 943cc58f4..55b20ee1f 100644--- a/internal/gui/masking.go+++ b/internal/gui/masking.go@@ -1,13 +1,20 @@package guiimport (+ "encoding/json"+ "io""net/http""strconv"+ "strings""github.com/yetone/magpie/internal/gwmw"+ "github.com/yetone/magpie/internal/masking""github.com/yetone/magpie/internal/maskrec")+// previewMax is how much text the preview masks.+const previewMax = 64 << 10+// maskedRecord is a record as the page shows it: each value says whether// the masker still knows what it stood for, so the page offers to show it// only where it can.@@ -51,8 +58,37 @@ func shownRecord(r maskrec.Record, t gwmw.Teller) maskedRecord {// - GET /api/masking/record?key= one record;// - GET /api/masking/value?p= what a placeholder stood for, while this// magpie's masker remembers it (404 after a restart: it is never on-// disk).+// disk);+// - POST /api/masking/preview {text} the text as a vendor would get it+// under the settings now, and what was found, each told by its kind+// and hint, never its value; nothing of it is remembered (#1977).func maskingRoutes(mux *http.ServeMux) {+ mux.HandleFunc("POST /api/masking/preview", func(rw http.ResponseWriter, r *http.Request) {+ var in struct {+ Text string `json:"text"`+ }+ if err := json.NewDecoder(io.LimitReader(r.Body, 4*previewMax)).Decode(&in); err != nil {+ fail(rw, err)+ return+ }+ if len(in.Text) > previewMax {+ in.Text = strings.ToValidUTF8(in.Text[:previewMax], "")+ }+ text, hits, off := masking.Preview(in.Text)+ type hit struct {+ Kind string `json:"kind"`+ By string `json:"by"`+ Action string `json:"action,omitempty"`+ Placeholder string `json:"placeholder"`+ Head string `json:"head,omitempty"`+ Len int `json:"len"`+ }+ out := []hit{}+ for _, h := range hits {+ out = append(out, hit{h.Kind, h.By, h.Action, h.Placeholder, h.Head, h.Len})+ }+ writeJSON(rw, map[string]any{"text": text, "hits": out, "off": off})+ })mux.HandleFunc("GET /api/masking/summary", func(rw http.ResponseWriter, r *http.Request) {days, _ := strconv.Atoi(r.URL.Query().Get("days"))writeJSON(rw, maskrec.Sum(min(max(days, 1), 30)))diff --git a/internal/gui/settings_keep_test.go b/internal/gui/settings_keep_test.goindex 93b1cfc28..a6921252d 100644--- a/internal/gui/settings_keep_test.go+++ b/internal/gui/settings_keep_test.go@@ -135,6 +135,8 @@ func TestSettingsSaveKeepsWhatItDoesNotSend(t *testing.T) {ModelSameAs: map[string]string{"p/m": "deepseek-v4.1-flash"},ModelKinds: map[string]string{"p/m": "video"},RedactRules: []redact.Rule{{Kind: "prefix", Prefix: "oc_sk_"}},+ RedactActions: map[string]string{"API_KEY": "block", "EMAIL": "warn"},+ RedactAllow: []string{"[email protected]"},LAN: true,LANKey: "sk-lan",LANKeyID: "lan-key-id",diff --git a/internal/gui/tests/account-discovery.test.cjs b/internal/gui/tests/account-discovery.test.cjsindex 1fc111c9e..e7c187766 100644--- a/internal/gui/tests/account-discovery.test.cjs+++ b/internal/gui/tests/account-discovery.test.cjs@@ -115,11 +115,14 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",const row = box.locator(".row");// scrolled to with the wheel, as a reader does: Playwright's own// scroll into view is code, which the click guard puts back- await page.mouse.move(width / 2, 300);- for (let i = 0; i < 20 && await row.evaluate((r) => r.getBoundingClientRect().bottom > (document.querySelector("footer.foot")?.getBoundingClientRect().top ?? innerHeight) - 4); i++) {- await page.mouse.wheel(0, 200);- await page.waitForTimeout(80);- }+ const toRow = async () => {+ await page.mouse.move(width / 2, 300);+ for (let i = 0; i < 60 && await row.evaluate((r) => r.getBoundingClientRect().bottom > (document.querySelector("footer.foot")?.getBoundingClientRect().top ?? innerHeight) - 4); i++) {+ await page.mouse.wheel(0, 200);+ await page.waitForTimeout(80);+ }+ };+ await toRow();await page.waitForTimeout(300);const fit = await row.evaluate((r) => {const b = r.getBoundingClientRect(), segs = r.querySelector(".val").getBoundingClientRect();@@ -163,6 +166,8 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",assert.equal(await page.locator("#discoveryList .opt").nth(0).evaluate((b) => b.classList.contains("on")), true,"a save of another setting turned discovery back on");+ // Mask secrets on draws what is done with each kind above it+ await toRow();await page.locator("#discoveryList .opt").nth(1).click();for (let i = 0; i < 50 && posts.length < 2; i++) await page.waitForTimeout(50);assert.deepEqual(posts.at(-1), { off: false });diff --git a/internal/gui/tests/mask-actions.test.cjs b/internal/gui/tests/mask-actions.test.cjsnew file mode 100644index 000000000..8323fdb73--- /dev/null+++ b/internal/gui/tests/mask-actions.test.cjs@@ -0,0 +1,145 @@+// Run with Node's test runner and Playwright on the module path; see README.md.+// Settings › Privacy says what is done with each kind of value found+// (#1977): Mask, Warn or Block, saved as the kind's choice, Mask saved as+// no choice at all; personal kinds behind Show all kinds; a Never mask+// list saved whole; and Try it, which shows text as the vendor would get+// it, what was found and what would be done, says a block holds the+// request back, keeps the text nowhere in the browser and shows no value+// the preview didn't hand back. Every language, Chromium and WebKit,+// 1100 and 640px; the API is faked.+const assert = require("node:assert/strict");+const fs = require("node:fs/promises");+const path = require("node:path");+const { test } = require("node:test");+const { chromium, webkit } = require("playwright");+const { reach, watchSettingsReads, settingsDrawn } = require("./reader.cjs");++const assets = path.resolve(__dirname, "../assets");+const SECRET = "sk-proj-Tr1eDq8Wm3Rt6YxN2vLp";+const PH = "{{API_KEY_ab12cd34}}";++function serve(lang, posted) {+ let settings = { theme: "light", lang, tray: "panel", redact: true, redactPersonal: true };+ return async (route) => {+ const req = route.request(), url = new URL(req.url());+ const json = (data) => route.fulfill({ json: data });+ if (url.pathname === "/boot.js") return route.fulfill({ contentType: "text/javascript", body: `window.bootPrefs = {lang:"${lang}",theme:"light",web:true};` });+ if (url.pathname === "/wails/runtime.js") return route.fulfill({ contentType: "text/javascript", body: "export const Window = {};" });+ if (url.pathname === "/api/state") return json({ agents: [], profiles: [], settings });+ if (url.pathname === "/api/settings") return json(settings);+ if (url.pathname === "/api/providers") return json({ providers: [], presets: [], excluded: [], gateway: { running: true, window: true, url: "http://127.0.0.1:3999" } });+ if (url.pathname === "/api/groups") return json({ groups: [], models: [] });+ if (url.pathname === "/api/settings/redact-actions") {+ const b = req.postDataJSON();+ posted.push(["actions", b]);+ settings = { ...settings, redactActions: Object.keys(b.actions).length ? b.actions : undefined };+ return json(settings);+ }+ if (url.pathname === "/api/settings/redact-allow") {+ const b = req.postDataJSON();+ posted.push(["allow", b]);+ settings = { ...settings, redactAllow: b.allow };+ return json(settings);+ }+ if (url.pathname === "/api/masking/preview") {+ const { text } = req.postDataJSON();+ posted.push(["preview", text]);+ const block = settings.redactActions?.API_KEY === "block";+ return json({ text: text.replace(SECRET, PH), hits: text.includes(SECRET) ? [{ kind: "API_KEY", by: "rule", action: block ? "block" : "", placeholder: PH, head: "sk-p", len: SECRET.length }] : [] });+ }+ if (url.pathname.startsWith("/api/")) return json({});+ const file = path.join(assets, url.pathname === "/" ? "index.html" : url.pathname);+ const contentType = { ".html": "text/html", ".js": "text/javascript", ".css": "text/css", ".svg": "image/svg+xml", ".png": "image/png" }[path.extname(file)];+ try { await route.fulfill({ body: await fs.readFile(file), contentType }); } catch { await route.fulfill({ status: 404, body: "" }); }+ };+}++// English a translated page must not show+const english = ["When a value is found", "Show all kinds", "Never mask", "Try it", "What the vendor would get", "Held back", "Goes to the vendor"];+const wide = (page) => page.evaluate(() => {+ const over = [...document.querySelectorAll("#maskActList *, #maskTryList *")].filter((n) => n.getClientRects().length && n.getBoundingClientRect().right > innerWidth + 1).map((n) => n.className?.baseVal ?? n.className);+ return { page: document.documentElement.scrollWidth > document.documentElement.clientWidth, over };+});++for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium", "webkit"])) {+ for (const lang of ["en", "zh", "zh-TW", "ja", "de"]) {+ for (const width of [1100, 640]) {+ test(`${engine} ${lang} ${width}px: each kind's action, the allow list and Try it`, async (t) => {+ const browser = await (engine === "webkit" ? webkit.launch() : chromium.launch({ channel: "chromium" }));+ t.after(() => browser.close());+ const page = await (await browser.newContext({ viewport: { width, height: 900 }, reducedMotion: "reduce" })).newPage();+ page.setDefaultTimeout(8000);+ const errors = [], posted = [];+ page.on("pageerror", (e) => errors.push(e.message));+ await page.route("**/*", serve(lang, posted));+ await watchSettingsReads(page);+ await page.goto("http://magpie.test/?view=settings&tab=privacy");+ await settingsDrawn(page);++ // the secret kinds first, each on Mask; personal ones behind a button+ const acts = page.locator("#maskActList");+ assert.deepEqual(await acts.locator(".mask-act").evaluateAll((rs) => rs.map((r) => r.dataset.kind)), ["API_KEY", "PRIVATE_KEY", "TOKEN", "PASSWORD", "SECRET"]);+ assert.equal(await acts.locator(".mask-act .segs .opt.on").count(), 5);+ const more = acts.locator(".mask-act-more button");+ await reach(page, more);+ await more.click();+ await acts.locator('.mask-act[data-kind="EMAIL"]').waitFor();+ assert.equal(await acts.locator(".mask-act").count(), 22, "every personal kind shown");++ // Block on API_KEY is saved as its choice and said under it+ const key = acts.locator('.mask-act[data-kind="API_KEY"]');+ await reach(page, key.locator(".segs .opt").nth(2));+ await key.locator(".segs .opt").nth(2).click();+ await acts.locator('.mask-act[data-kind="API_KEY"] .sub.act-block').waitFor();+ assert.deepEqual(posted.at(-1), ["actions", { actions: { API_KEY: "block" } }]);+ // Warn on EMAIL is added beside it+ const mail = acts.locator('.mask-act[data-kind="EMAIL"]');+ await reach(page, mail.locator(".segs .opt").nth(1));+ await mail.locator(".segs .opt").nth(1).click();+ await acts.locator('.mask-act[data-kind="EMAIL"] .sub.act-warn').waitFor();+ assert.deepEqual(posted.at(-1), ["actions", { actions: { API_KEY: "block", EMAIL: "warn" } }]);+ // and Mask again is no choice at all+ await reach(page, acts.locator('.mask-act[data-kind="EMAIL"] .segs .opt').first());+ await acts.locator('.mask-act[data-kind="EMAIL"] .segs .opt').first().click();+ await page.waitForFunction(() => !document.querySelector('#maskActList .mask-act[data-kind="EMAIL"] .sub.act-warn'));+ assert.deepEqual(posted.at(-1), ["actions", { actions: { API_KEY: "block" } }]);++ // the allow list, split on commas and saved whole+ const allow = acts.locator("input.mask-allow");+ await reach(page, allow);+ await allow.fill(" [email protected],build-01 ,, ");+ await allow.press("Enter");+ for (let i = 0; i < 100 && !posted.some(([k]) => k === "allow"); i++) await page.waitForTimeout(40);+ assert.deepEqual(posted.filter(([k]) => k === "allow").at(-1), ["allow", { allow: ["[email protected]", "build-01"] }]);+ await page.waitForFunction(() => document.querySelector("#maskActList input.mask-allow")?.value === "[email protected], build-01");++ // Try it: the vendor's text, what was found, and that it is held back+ const box = page.locator("#maskTryList textarea.mask-try-in");+ await reach(page, box);+ await box.fill(`deploy with ${SECRET} now`);+ const out = page.locator("#maskTryList .mask-try-out");+ await out.locator("mark.mo-ph").waitFor();+ assert.equal(await out.locator(".mask-try-text").textContent(), `deploy with ${PH} now`);+ assert.equal(await out.locator(".mask-try-note.block").count(), 1, "a block says the request is held back");+ assert.equal(await out.locator(".mr-act.block").count(), 1);+ assert.equal((await out.textContent()).includes(SECRET), false, "the value is not shown back");+ assert.equal(posted.filter(([k]) => k === "preview").length, 1, "typing asks once, after a pause");+ // kept nowhere in the browser+ const kept = await page.evaluate(() => { const all = []; for (const s of [localStorage, sessionStorage]) for (let i = 0; i < s.length; i++) all.push(s.getItem(s.key(i))); return all.join("\n"); });+ assert.equal(kept.includes(SECRET), false, "the text is stored nowhere");+ // nothing found: said so+ await box.fill("nothing secret here");+ await out.locator(".mask-try-note").first().waitFor();+ await page.waitForFunction(() => !document.querySelector("#maskTryList mark.mo-ph"));+ assert.equal(await out.locator(".mask-try-hits").count(), 0);++ if (lang !== "en") {+ const said = await page.locator("#setPage-privacy").innerText();+ for (const w of english) assert(!said.includes(w), `${w} left in English`);+ }+ assert.deepEqual(await wide(page), { page: false, over: [] });+ assert.deepEqual(errors, []);+ });+ }+ }+}diff --git a/internal/gui/tests/masking-records.test.cjs b/internal/gui/tests/masking-records.test.cjsindex 968834fce..5a3d1a14b 100644--- a/internal/gui/tests/masking-records.test.cjs+++ b/internal/gui/tests/masking-records.test.cjs@@ -62,7 +62,8 @@ function serve(lang, sum, asked) {if (url.pathname === "/api/settings") return json(settings);if (url.pathname === "/api/providers") return json({ providers: [{ id: "anthropic", name: "Anthropic", icon: "anthropic", agents: [], models: [] }, { id: "openai", name: "OpenAI", icon: "openai", agents: [], models: [] }], presets: [], excluded: [],gateway: { running: true, window: true, mine: true, url: "http://127.0.0.1:3999", models: 0, groups: [],- calls: [{ id: 7, time: ago(120e3), agent: "claude", model: "claude-opus-4-1", from: "anthropic", to: "anthropic", status: 200, ms: 900, masked: { n: 3, distinct: 2, kinds: { API_KEY: 2, EMAIL: 1 }, by: { rule: 2 }, key: 1002 } }] } });+ calls: [{ id: 7, time: ago(120e3), agent: "claude", model: "claude-opus-4-1", from: "anthropic", to: "anthropic", status: 200, ms: 900, masked: { n: 3, distinct: 2, kinds: { API_KEY: 2, EMAIL: 1 }, by: { rule: 2 }, key: 1002 } },+ { id: 8, time: ago(60e3), agent: "claude", model: "claude-opus-4-1", from: "anthropic", to: "anthropic", status: 200, ms: 700, masked: { n: 0, distinct: 0, warned: 2, key: 1003 } }] } });if (url.pathname === "/api/groups") return json({ groups: [], models: [] });if (url.pathname === "/api/plugins") return json({ plugins: [] });if (url.pathname === "/api/privacy-model") return json({ supported: true, installed: false, size: 1.6e9 });@@ -206,8 +207,12 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",test(`${engine}: a masked call on the Gateway page opens its record`, async (t) => {const { browser, page, errors } = await open(engine, "en", 1100, summary, "?view=gateway");t.after(() => browser.close());- const badge = page.locator("#activity .call .mask-badge");+ const badge = page.locator("#activity .call .mask-badge:not(.warn)");await badge.waitFor();+ // a call whose values only went as written, as the user set (#1977)+ const warned = page.locator("#activity .call .mask-badge.warn");+ assert.equal(await warned.textContent(), "2");+ assert.equal(await warned.getAttribute("title"), "2 values went to the vendor as written, as you set");assert.equal(await badge.textContent(), "3");assert.match(await badge.getAttribute("title"), /^3 values masked before it went to the vendor: API keys 2, Emails 1$/);await badge.click();diff --git a/internal/gui/tests/privacy-usage-stats.test.cjs b/internal/gui/tests/privacy-usage-stats.test.cjsindex a90d0add1..5fef0750e 100644--- a/internal/gui/tests/privacy-usage-stats.test.cjs+++ b/internal/gui/tests/privacy-usage-stats.test.cjs@@ -67,8 +67,11 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",// below the card, scrolled to with the wheel as a reader doesconst reveal = async (name) => {await page.mouse.move(450, 300);- for (let i = 0; i < 20 && (await row(name).boundingBox()).y > 600; i++) {- await page.mouse.wheel(0, 200);+ await row(name).waitFor(); // a save draws the rows again+ for (let i = 0; i < 40; i++) {+ const y = (await row(name).boundingBox())?.y ?? 9999;+ if (y > 60 && y < 600) break;+ await page.mouse.wheel(0, y > 600 ? 200 : -200);await page.waitForTimeout(80);}};@@ -79,6 +82,7 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",assert.ok(!saved.at(-1).noStats);// another setting saved keeps it off+ await reveal(w.personal);await row(w.personal).getByRole("button", { name: w.on, exact: true }).click();await isOn(w.personal);assert.equal(saved.at(-1).noUsageStats, true, JSON.stringify(saved.at(-1)));diff --git a/internal/gwmw/gwmw.go b/internal/gwmw/gwmw.goindex 94d775437..aea7d501e 100644--- a/internal/gwmw/gwmw.go+++ b/internal/gwmw/gwmw.go@@ -87,13 +87,26 @@ type Masker interface {Writer(w http.ResponseWriter) (http.ResponseWriter, func())}-// Hit is one value a masker swapped, told without the value: its kind,+// Hit is one value a masker found, told without the value: its kind,// the finder that found it (rule, yours, word, model or known), its-// placeholder, and as much of it as may be kept where it may not (Head,-// its first character or a key's prefix, and Len, how many it has).+// placeholder, what was done with it (Action: "" masked, "warn" sent as+// written, "block" masked and the request to be held back), and as much+// of it as may be kept where it may not (Head, its first character or a+// key's prefix, and Len, how many it has).type Hit struct {- Kind, By, Placeholder, Head string- Len int+ Kind, By, Placeholder, Action, Head string+ Len int+}++// Blocks says hits have a value the user asked to hold a request back+// for.+func Blocks(hits []Hit) bool {+ for _, h := range hits {+ if h.Action == "block" {+ return true+ }+ }+ return false}// Teller is a Masker that says what it masked too, for the record ofdiff --git a/internal/masking/masking.go b/internal/masking/masking.goindex 27d2021da..210310399 100644--- a/internal/masking/masking.go+++ b/internal/masking/masking.go@@ -23,7 +23,8 @@ func Wanted(st settings.Settings) bool {func options() redact.Options {st := settings.Shared()- o := redact.Options{Secrets: st.Redact, Personal: st.RedactPersonal, Kinds: st.RedactKinds, Words: st.RedactWords, Rules: st.RedactRules}+ o := redact.Options{Secrets: st.Redact, Personal: st.RedactPersonal, Kinds: st.RedactKinds, Words: st.RedactWords, Rules: st.RedactRules,+ Actions: st.RedactActions, Allow: st.RedactAllow}if st.RedactModel {o.Model = model}@@ -74,16 +75,29 @@ func (Masker) TextHits(s string) (string, []gwmw.Hit) {func (Masker) Value(p string) (string, bool) { return redact.Value(p) }-// told adds each value masked to hits, with its hint: the value itself-// stays in redact's memory.+// told adds each value found to hits, with its hint: the value itself+// stays in redact's memory, or, sent as written, nowhere.func told(hits *[]gwmw.Hit) func(redact.Hit) {return func(h redact.Hit) {- v, _ := redact.Value(h.Placeholder)- head, n := redact.Hint(h.Kind, v)- *hits = append(*hits, gwmw.Hit{Kind: h.Kind, By: h.By, Placeholder: h.Placeholder, Head: head, Len: n})+ *hits = append(*hits, gwmw.Hit{Kind: h.Kind, By: h.By, Placeholder: h.Placeholder, Action: h.Action, Head: h.Head, Len: h.Len})}}+// Preview is s as the settings would send it, and what was found in it,+// with nothing remembered: no placeholder is kept for the replies, and+// none of its values is masked again later as one seen before. With+// masking off it is as masking would be turned on (Off says so), so the+// user can try it first.+func Preview(s string) (out string, hits []gwmw.Hit, off bool) {+ o := options()+ if !o.Secrets && !o.Personal && len(o.Words) == 0 {+ o.Secrets, o.Personal, off = true, true, true+ }+ o.Seen = told(&hits)+ out = redact.Preview(s, o)+ return out, hits, off+}+func (Masker) Restore(s string, escaped bool) string { return redact.Restore(s, escaped) }func (Masker) Writer(w http.ResponseWriter) (http.ResponseWriter, func()) {diff --git a/internal/maskrec/maskrec.go b/internal/maskrec/maskrec.goindex b79193cb6..4eec65f20 100644--- a/internal/maskrec/maskrec.go+++ b/internal/maskrec/maskrec.go@@ -44,6 +44,9 @@ type Value struct {Head string `json:"h,omitempty"` // its first character, or a key's prefixLen int `json:"l"` // its length in charactersN int `json:"n"` // the places the request had it+ // Action: "" masked, "warn" sent as written (the user asked to be+ // told only), "block" masked and the request held back+ Action string `json:"a,omitempty"`}// Record is one request's masking.@@ -58,6 +61,9 @@ type Record struct {Model string `json:"model,omitempty"`N int `json:"n"` // values swapped, each place countedValues []Value `json:"values,omitempty"`+ // Blocked: the request was held back, sent to no one, for a value+ // of a kind the user set to block+ Blocked bool `json:"blocked,omitempty"`// Snips: by placeholder, the text around its first place as the// vendor saw it; in memory onlySnips map[string]string `json:"snips,omitempty"`@@ -156,6 +162,9 @@ func merged(vs []Value) []Value {if out[i].By == "known" && v.By != "known" {out[i].By = v.By}+ if v.Action == "block" {+ out[i].Action = v.Action+ }continue}at[v.Placeholder] = len(out)diff --git a/internal/maskrec/maskrec_test.go b/internal/maskrec/maskrec_test.goindex 752e2f4f2..5f1dd7678 100644--- a/internal/maskrec/maskrec_test.go+++ b/internal/maskrec/maskrec_test.go@@ -119,3 +119,28 @@ func TestPrune(t *testing.T) {t.Fatalf("old day kept: %v", err)}}++// A value sent as written is counted as warned, not as one masked, and a+// request held back is counted, read from disk and kept up alike (#1977).+func TestSumWarnedAndBlocked(t *testing.T) {+ now := time.Date(2026, 10, 20, 15, 0, 0, 0, time.Local)+ sandbox(t, now)+ e, k := "{{EMAIL_aaaabbbb}}", "{{API_KEY_ccccdddd}}"+ Add(Record{Time: now.Add(-time.Hour), N: 1, Values: []Value{{Placeholder: e, Kind: "EMAIL", By: "rule", Len: 9, N: 2, Action: "warn"}, {Placeholder: k, Kind: "API_KEY", By: "rule", Len: 40}}})+ for _, fresh := range []bool{true, false} {+ if fresh {+ store.mu.Lock()+ store.days = nil+ store.mu.Unlock()+ }+ _ = Sum(7)+ Add(Record{Time: now, What: "blocked", Blocked: true, N: 1, Values: []Value{{Placeholder: k, Kind: "API_KEY", By: "rule", Len: 40, Action: "block"}}})+ }+ s := Sum(7)+ if s.Warned != 2 || s.Blocked != 2 || s.Distinct != 1 || s.N != 3 || len(s.Kinds) != 1 || s.Kinds[0].Name != "API_KEY" {+ t.Fatalf("summary: %+v", s)+ }+ if got, _ := List(0, 1, Filter{}); len(got) != 1 || !got[0].Blocked || got[0].Values[0].Action != "block" {+ t.Fatalf("record: %+v", got)+ }+}diff --git a/internal/maskrec/summary.go b/internal/maskrec/summary.goindex 74b5da09b..772ae0b89 100644--- a/internal/maskrec/summary.go+++ b/internal/maskrec/summary.go@@ -13,6 +13,8 @@ type dayAgg struct {size int64requests intn int+ warned int // places a value went as written, the user told+ blocked int // requests held backhours [24]intkinds map[string]int // each place countedvals map[string]*valAgg@@ -35,7 +37,14 @@ func (a *dayAgg) add(r Record) {a.requests++a.n += r.Na.hours[r.Time.Local().Hour()] += r.N+ if r.Blocked {+ a.blocked+++ }for _, v := range r.Values {+ if v.Action == "warn" { // not masked: told, not kept as one+ a.warned += v.N+ continue+ }a.kinds[v.Kind] += v.Nx := a.vals[v.Placeholder]if x == nil {@@ -115,7 +124,9 @@ type Summary struct {Requests int `json:"requests"`N int `json:"n"`Distinct int `json:"distinct"`- Hours [24]int `json:"hours"` // places, by the hour of the day+ Warned int `json:"warned,omitempty"` // places a value went as written, the user told+ Blocked int `json:"blocked,omitempty"` // requests held back+ Hours [24]int `json:"hours"` // places, by the hour of the dayKinds []Count `json:"kinds"`By map[string]int `json:"by"` // different values by the finder that found eachAgents []Count `json:"agents"`@@ -155,6 +166,8 @@ func Sum(days int) Summary {out.Days = append(out.Days, Day{Day: d, Requests: a.requests, N: a.n, Distinct: len(a.vals)})out.Requests += a.requestsout.N += a.n+ out.Warned += a.warned+ out.Blocked += a.blockedfor h, n := range a.hours {out.Hours[h] += n}diff --git a/internal/redact/actions_test.go b/internal/redact/actions_test.gonew file mode 100644index 000000000..5a89a10c2--- /dev/null+++ b/internal/redact/actions_test.go@@ -0,0 +1,94 @@+package redact++import (+ "strings"+ "testing"+)++// A kind the user set to warn of goes as written: told, with a+// placeholder no one remembers, and not counted as masked. The rest is+// masked as before (#1977).+func TestActionWarnSendsAsWritten(t *testing.T) {+ const key, mail = "sk-proj-Wn4rT7yQ2mLx9VbK3cZp", "[email protected]"+ var hits []Hit+ o := Options{Secrets: true, Personal: true, Actions: map[string]string{"EMAIL": Warn},+ Seen: func(h Hit) { hits = append(hits, h) }}+ out, n := Mask("key "+key+" mail "+mail, o)+ if !strings.Contains(out, mail) || strings.Contains(out, key) || n != 1 {+ t.Fatalf("out %q n %d", out, n)+ }+ var warned *Hit+ for i, h := range hits {+ if h.Kind == "EMAIL" {+ warned = &hits[i]+ }+ }+ if len(hits) != 2 || warned == nil || warned.Action != Warn || warned.Head != "w" || warned.Len != len(mail) {+ t.Fatalf("hits %+v", hits)+ }+ if _, ok := Value(warned.Placeholder); ok || strings.Contains(out, warned.Placeholder) {+ t.Fatalf("a value sent as written is remembered or placed: %+v in %q", warned, out)+ }++ // in a JSON body the same+ hits = nil+ b, n := MaskJSON([]byte(`{"messages":[{"role":"user","content":"mail `+mail+`"}]}`), o)+ if !strings.Contains(string(b), mail) || n != 0 || len(hits) != 1 || hits[0].Action != Warn {+ t.Fatalf("json %s n %d hits %+v", b, n, hits)+ }+}++// A kind the user set to block is masked all the same, and told as+// blocked, so the gateway holds the request back; a request that goes+// anyway goes masked.+func TestActionBlockMasks(t *testing.T) {+ const key = "sk-proj-Bl0cKq8Wm3Rt6YxN2vLp"+ var hits []Hit+ out, n := Mask("key "+key, Options{Secrets: true, Actions: map[string]string{"API_KEY": Block},+ Seen: func(h Hit) { hits = append(hits, h) }})+ if strings.Contains(out, key) || n != 1 || len(hits) != 1 || hits[0].Action != Block || hits[0].Kind != "API_KEY" {+ t.Fatalf("out %q n %d hits %+v", out, n, hits)+ }+ if v, ok := Value(hits[0].Placeholder); !ok || v != key {+ t.Fatalf("a blocked value isn't masked as any other: %q %v", v, ok)+ }+}++// A value on the allow list goes as written, whole, in any case, and+// isn't told; a value it is only part of is masked.+func TestAllowLetsAValueGo(t *testing.T) {+ const mail, key = "[email protected]", "sk-proj-Al1oWq8Wm3Rt6YxN2vLp"+ var hits []Hit+ o := Options{Secrets: true, Personal: true, Allow: []string{" [email protected] ", key},+ Seen: func(h Hit) { hits = append(hits, h) }}+ out, n := Mask("write "+mail+" with "+key+" or [email protected]", o)+ if !strings.Contains(out, mail) || !strings.Contains(out, key) || strings.Contains(out, "other.person") || n != 1 || len(hits) != 1 {+ t.Fatalf("out %q n %d hits %+v", out, n, hits)+ }+ out, _ = Mask("cc x"+mail, o)+ if strings.Contains(out, "x"+mail) {+ t.Fatalf("a value the allowed one is part of went as written: %q", out)+ }+}++// The preview masks as Mask does and remembers nothing: its placeholders+// stand for nothing afterwards, and its values aren't masked later as+// values seen before.+func TestPreviewRemembersNothing(t *testing.T) {+ const key = "sk-proj-Pr3vIewQ8m3Rt6YxN2vL"+ var hits []Hit+ out := Preview("key "+key, Options{Secrets: true, Seen: func(h Hit) { hits = append(hits, h) }})+ if strings.Contains(out, key) || len(hits) != 1 || !strings.Contains(out, hits[0].Placeholder) {+ t.Fatalf("out %q hits %+v", out, hits)+ }+ if _, ok := Value(hits[0].Placeholder); ok {+ t.Fatal("the preview's placeholder is remembered")+ }+ if spans := knownSpans("again "+key, Options{Secrets: true}); len(spans) != 0 {+ t.Fatalf("the preview's value is known: %v", spans)+ }+ // masked for real, the same value goes as the same placeholder+ if real, _ := Mask("key "+key, Options{Secrets: true}); real != out {+ t.Fatalf("preview %q, sent %q", out, real)+ }+}diff --git a/internal/redact/json.go b/internal/redact/json.goindex df00352ed..d3634438b 100644--- a/internal/redact/json.go+++ b/internal/redact/json.go@@ -427,7 +427,7 @@ func MaskJSON(body []byte, o Options) ([]byte, int) {// number or an address the vendor wrote of its own can be one// masked in another conversation (13800138000 is in many), and// masked, its text would no longer match its signature- t, n := apply(t, knownSpans(t, Options{Secrets: o.Secrets}), placeholder, o.Seen)+ t, n := apply(t, knownSpans(t, Options{Secrets: o.Secrets}), placeholder, o)total += nreturn t})diff --git a/internal/redact/redact.go b/internal/redact/redact.goindex 9beb7191c..3e89502e4 100644--- a/internal/redact/redact.go+++ b/internal/redact/redact.go@@ -44,7 +44,16 @@ type Options struct {// Model, where set, finds more in the texts to mask (model.go)Model Model- // Seen, where set, is told of each value masked+ // Actions is what is done with a value of a kind (a Hit's Kind) other+ // than masking it, where the user chose: Warn sends it as written,+ // Block masks it and the request is held back (the gateway's). A kind+ // not in it is masked.+ Actions map[string]string+ // Allow are values the user lets go as written, whichever finder finds+ // them: each a whole value, in any case.+ Allow []string++ // Seen, where set, is told of each value masked, warned of or blockedSeen func(Hit)// found are the spans Model found, by text, for one request's texts@@ -346,10 +355,17 @@ func loadKey() []byte {var b32 = base32.StdEncoding.WithPadding(base32.NoPadding)-func placeholder(kind, v string) string {+// tag is the placeholder v of kind goes as, made and not remembered: a+// value sent as written (Warn) is named by it in the record of what was+// found, and a preview names each value by it.+func tag(kind, v string) string {h := hmac.New(sha256.New, key())h.Write([]byte(kind + "\x00" + v))- p := "{{" + kind + "_" + strings.ToLower(b32.EncodeToString(h.Sum(nil))[:8]) + "}}"+ return "{{" + kind + "_" + strings.ToLower(b32.EncodeToString(h.Sum(nil))[:8]) + "}}"+}++func placeholder(kind, v string) string {+ p := tag(kind, v)mu.Lock()if _, ok := values[p]; !ok {if len(values) >= maxValues {@@ -389,10 +405,21 @@ const (ByKnown = "known")-// Hit is one value masked: its kind, the finder that found it, and the-// placeholder it went as. It never holds the value itself.+// What is done with a value of a kind (Options.Actions) other than+// masking it.+const (+ Warn = "warn" // sent as written, and recorded+ Block = "block" // masked, and the request held back+)++// Hit is one value found: its kind, the finder that found it, the+// placeholder it went as (or would have, for one sent as written), what+// was done with it ("" masked, Warn, Block) and its Hint. It never holds+// the value itself.type Hit struct {- Kind, By, Placeholder string+ Kind, By, Placeholder, Action string+ Head string+ Len int}// Value is the value behind placeholder p, while this run of magpie@@ -436,6 +463,14 @@ func Mask(s string, o Options) (string, int) {return mask(s, withModel(o, []string{s}), placeholder)}+// Preview is Mask with nothing remembered: each value goes as the+// placeholder it would have, which isn't kept for Restore or as a value+// seen before (known.go).+func Preview(s string, o Options) string {+ out, _ := mask(s, withModel(o, []string{s}), tag)+ return out+}+// mask swaps what o covers in s for what put makes of each value.func mask(s string, o Options, put func(kind, v string) string) (string, int) {if len(s) < 3 {@@ -527,7 +562,7 @@ func mask(s string, o Options, put func(kind, v string) string) (string, int) {// kind it has always hadfound = append(found, modelSpans(s, o)...)found = append(found, knownSpans(s, o)...)- return apply(s, found, put, o.Seen)+ return apply(s, found, put, o)}// by is the finder r is, for Hit.By.@@ -618,9 +653,10 @@ func (r rule) takes(s string, a, b int, ok func(string) bool) bool {}// apply swaps each of found in s for what put makes of its value, the-// first and longest where they overlap, nothing inside a placeholder.-// seen, where set, is told of each.-func apply(s string, found []span, put func(kind, v string) string, seen func(Hit)) (string, int) {+// first and longest where they overlap, nothing inside a placeholder. A+// value o allows is left as it is, and one of a kind o warns of is left+// and told. o.Seen, where set, is told of each.+func apply(s string, found []span, put func(kind, v string) string, o Options) (string, int) {if len(found) == 0 {return s, 0}@@ -643,16 +679,34 @@ func apply(s string, found []span, put func(kind, v string) string, seen func(Hireturn found[i].end > found[j].end})var b strings.Builder- last, n := 0, 0+ last, next, n := 0, 0, 0 // next: where the next value may startfor _, f := range found {- if f.start < last {+ if f.start < next {+ continue+ }+ next = f.end+ v := s[f.start:f.end]+ if o.allows(v) {+ continue+ }+ act := o.Actions[f.kind]+ if act != Warn && act != Block {+ act = ""+ }+ h := Hit{Kind: f.kind, By: f.by, Action: act}+ h.Head, h.Len = Hint(f.kind, v)+ if act == Warn {+ if o.Seen != nil {+ h.Placeholder = tag(f.kind, v)+ o.Seen(h)+ }continue}b.WriteString(s[last:f.start])- p := put(f.kind, s[f.start:f.end])- b.WriteString(p)- if seen != nil {- seen(Hit{f.kind, f.by, p})+ h.Placeholder = put(f.kind, v)+ b.WriteString(h.Placeholder)+ if o.Seen != nil {+ o.Seen(h)}last, n = f.end, n+1}@@ -660,6 +714,16 @@ func apply(s string, found []span, put func(kind, v string) string, seen func(Hireturn b.String(), n}+// allows says the user lets v go as written (Options.Allow).+func (o Options) allows(v string) bool {+ for _, a := range o.Allow {+ if strings.EqualFold(strings.TrimSpace(a), v) {+ return true+ }+ }+ return false+}+func containsAny(s string, subs []string) bool {for _, x := range subs {if strings.Contains(s, x) {diff --git a/internal/redact/rules.go b/internal/redact/rules.goindex f9e044f91..59dbdfef4 100644--- a/internal/redact/rules.go+++ b/internal/redact/rules.go@@ -57,6 +57,10 @@ func RuleKind(kind string) string {return k}+// KindName says k is written as a placeholder names a kind (API_KEY, a+// rule's own), as a setting keyed by kind must name it.+func KindName(k string) bool { return k != "" && RuleKind(k) == k }+// CheckRules tidies the user's rules (the kind as a placeholder has it,// spaces trimmed, the empty and the repeated dropped) and says what is// wrong with one that can't be used: a pattern that doesn't compile, ordiff --git a/internal/settings/masking.go b/internal/settings/masking.gonew file mode 100644index 000000000..4494035c4--- /dev/null+++ b/internal/settings/masking.go@@ -0,0 +1,60 @@+package settings++import (+ "fmt"+ "strings"++ "github.com/yetone/magpie/internal/redact"+)++// How many values the masking allowlist may hold, and how long each.+const (+ maxAllow = 200+ maxAllowLen = 256+)++// redactActions is m with only the actions redact knows, each for a kind+// named as a placeholder names one (API_KEY, a rule's own); masking, the+// default, is left out.+func redactActions(m map[string]string) (map[string]string, error) {+ out := map[string]string{}+ for k, a := range m {+ k, a = strings.TrimSpace(k), strings.ToLower(strings.TrimSpace(a))+ switch a {+ case "", "mask":+ continue+ case redact.Warn, redact.Block:+ default:+ return nil, fmt.Errorf("what is done with a kind of value found must be mask, %s or %s, not %q", redact.Warn, redact.Block, a)+ }+ if !redact.KindName(k) {+ return nil, fmt.Errorf("%q isn't a kind of value masking finds", k)+ }+ out[k] = a+ }+ if len(out) == 0 {+ return nil, nil+ }+ return out, nil+}++// redactAllow is the allowlist trimmed, each value once whatever its case.+func redactAllow(vs []string) ([]string, error) {+ var out []string+ seen := map[string]bool{}+ for _, v := range vs {+ v = strings.TrimSpace(v)+ if v == "" || seen[strings.ToLower(v)] {+ continue+ }+ if len(v) > maxAllowLen {+ return nil, fmt.Errorf("a value the masking lets go is at most %d characters", maxAllowLen)+ }+ seen[strings.ToLower(v)] = true+ out = append(out, v)+ }+ if len(out) > maxAllow {+ return nil, fmt.Errorf("the masking lets at most %d values go", maxAllow)+ }+ return out, nil+}diff --git a/internal/settings/masking_test.go b/internal/settings/masking_test.gonew file mode 100644index 000000000..30f0fd7ca--- /dev/null+++ b/internal/settings/masking_test.go@@ -0,0 +1,44 @@+package settings++import (+ "slices"+ "strings"+ "testing"+)++// Only warn and block are kept, for a kind masking names; mask, the+// default, is left out, and anything else is refused (#1977).+func TestRedactActions(t *testing.T) {+ got, err := redactActions(map[string]string{"API_KEY": "Block", " EMAIL ": "warn", "PHONE": "mask", "SSN": ""})+ if err != nil || len(got) != 2 || got["API_KEY"] != "block" || got["EMAIL"] != "warn" {+ t.Fatalf("%v %v", got, err)+ }+ if got, err := redactActions(map[string]string{"PHONE": "mask"}); err != nil || got != nil {+ t.Fatalf("all masked: %v %v", got, err)+ }+ if _, err := redactActions(map[string]string{"API_KEY": "drop"}); err == nil {+ t.Fatal("an unknown action was kept")+ }+ if _, err := redactActions(map[string]string{"api key": "warn"}); err == nil {+ t.Fatal("a kind masking can't name was kept")+ }+}++// The allow list is trimmed, each value once whatever its case, and+// bounded.+func TestRedactAllow(t *testing.T) {+ got, err := redactAllow([]string{" [email protected] ", "[email protected]", "", "build-01"})+ if err != nil || !slices.Equal(got, []string{"[email protected]", "build-01"}) {+ t.Fatalf("%q %v", got, err)+ }+ if _, err := redactAllow([]string{strings.Repeat("x", maxAllowLen+1)}); err == nil {+ t.Fatal("a value too long was kept")+ }+ many := make([]string, maxAllow+1)+ for i := range many {+ many[i] = strings.Repeat("v", i+1)+ }+ if _, err := redactAllow(many); err == nil {+ t.Fatal("too many values were kept")+ }+}diff --git a/internal/settings/settings.go b/internal/settings/settings.goindex 9fd09e1b8..18d721cb8 100644--- a/internal/settings/settings.go+++ b/internal/settings/settings.go@@ -97,6 +97,13 @@ type Settings struct {RedactKinds map[string]bool `json:"redactKinds,omitempty"`RedactWords []string `json:"redactWords,omitempty"`RedactRules []redact.Rule `json:"redactRules,omitempty"`+ // RedactActions is what is done with a value of a kind (redact.Hit's+ // Kind: API_KEY, EMAIL, a rule's own…) other than masking it, where+ // the user chose: redact.Warn sends it as written and records it,+ // redact.Block holds the request back. RedactAllow are values sent+ // as written whatever finds them, each whole, in any case (#1977).+ RedactActions map[string]string `json:"redactActions,omitempty"`+ RedactAllow []string `json:"redactAllow,omitempty"`// LAN shares the gateway on the local network; remote callers must use// named caller keys. LANKey is retained for older Magpie versions.// LAN is this computer's own (KeepOwn; Lullaby on Discord: sharing@@ -1457,6 +1464,12 @@ func save(s Settings, recording bool) error {return err}s.RedactRules = rules+ if s.RedactActions, err = redactActions(s.RedactActions); err != nil {+ return err+ }+ if s.RedactAllow, err = redactAllow(s.RedactAllow); err != nil {+ return err+ }s.AgentOrder, s.AgentsHidden, s.AgentsShown = ids(s.AgentOrder), ids(s.AgentsHidden), ids(s.AgentsShown)s.UsageOrder = ids(s.UsageOrder)s.PanelUsageHidden = ids(s.PanelUsageHidden)