Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
privacy: every masking is recorded, and Settings › Privacy shows it
yetonef21dd99f5afa
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.mdindex 15ecf4377..93cba585e 100644--- a/docs/subsystems/gateway-routing.md+++ b/docs/subsystems/gateway-routing.md@@ -78,7 +78,8 @@ recorded as a prediction failure (`noPrediction`).2. `serve` redacts secrets in the request (`redacted`; `redact.MaskJSON` leaves ids alone; the request's tool definitions and output schemas, found by their place in each protocol (top-level `tools`, `tool_choice`, `response_format`, `text`, `generationConfig` and the like, and the tools of a Responses `additional_tools` or `tool_search_output` item), go as written save their descriptions, so a word the user masks can't break a grammar or an enum (#1911); and what the vendor sealed with a signature, a thinking block, Gemini's thoughts or Responses' encrypted reasoning, goes back as the vendor wrote it: only the values masked elsewhere in the request, and secrets masked before, are put back as their placeholders; a value masked once, at least 8 characters long, is masked again wherever a request has it, as an exact match whatever words are around it, so what the vendor wrote about a placeholder doesn't go back with the value in it, `redact/known.go`), resolves the model to a provider or a group, and refuses what can't be served. The reply has the values put back on its way to the agent (`redact.Writer`, a stream event by event, a whole body at its end): a placeholder the model wrote as it was given, or as models rewrite one (spaces inside the braces, a brace or both dropped, another case), is the value again when what is left names one magpie made, and one split across events is held until it is whole (`redact.Restore`, `partialTail`, [`internal/redact/redact.go`](../../internal/redact/redact.go)). A request whose provider and fallbacks, or whose group's models, all skip redaction (`unredactedRoute`, `provider.SkipsRedaction`: set *Send requests unmasked*, `Unredacted`, with every address on this machine or the local network) goes to them as the agent wrote it; the log and recent calls keep the masked body. count_tokens and embeddings follow the same rule. What a routing group's Jev is told of the turn (the message, the user's message before it and the reply to it) is masked all the same unless Jev's own provider skips redaction too (`maskedState` in `askJev`, [`decide.go`](../../internal/gateway/decide.go)): a remote Jev is none of the group's members. The masking is the built-in plugin `redact`'s (`gwmw.Masker`, [Plugin runtime](plugin-runtime.md)); when the settings ask for masking and no masker serves, or it fails, the request is a 503 and nothing is sent (`redacted`, `gwmw.Masking`).- With *Mask personal data* on, each kind in `redact.Categories` ([`internal/redact/more.go`](../../internal/redact/more.go)) is masked unless the user turned it off (`redactKinds` in settings, `Options.Kinds`). On by default: US SSNs, passport numbers, IBANs that pass mod 97, birthdays next to a word that says so, MAC addresses, and serial numbers next to S/N or the like. Off by default: the user name in a home folder, user@host in a prompt or ssh command, public IPv4 and IPv6 addresses (never private, loopback, link-local, CGNAT or documentation ranges) and cloud bucket names, since agents send them in nearly every request and the model must copy the placeholder exactly inside a path or URL. Context rules run only near their words (`rule.matches`). The Settings rows come from `REDACT_KINDS` in app.js, which `TestRedactKindsAreRedacts` holds to the same ids and defaults.- With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken. The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with it on a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left. The model is OpenAI's Privacy Filter as a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, from hf-mirror first in China, `source.chinaFirst`) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU is started on the CPU for the rest of the run), joins short texts into 12 KB chunks and maps the spans back, keeps the last 20000 answers by hash, and ends it after 10 minutes with no request ([`internal/pii`](../../internal/pii)). A model that is on and can't answer (not on disk, didn't start, hung) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).- - With *Mask secrets* on, the user's own masking rules (`redactRules`, `redact.Rule`: a placeholder name and a prefix or an RE2 pattern, [`internal/redact/rules.go`](../../internal/redact/rules.go)) run before magpie's. Settings → Privacy adds them, removes them, and edits one in place (`POST /api/settings/redact-rule` with the rule's index and its current value, `editRedactRule` in [`internal/gui/api.go`](../../internal/gui/api.go)). Every rule is checked by `redact.CheckRules` when settings are saved; an edit that doesn't compile, matches the empty string, repeats another rule or names a rule no longer at that index is refused with the rules left as they were. The gateway reads the rules from settings on each request (`redactionOptions`), so a change applies to the next request.+ - With *Mask secrets* on, the user's own masking rules (`redactRules`, `redact.Rule`: a placeholder name and a prefix or an RE2 pattern, [`internal/redact/rules.go`](../../internal/redact/rules.go)) run before magpie's. Settings → Privacy adds them, removes them, and edits one in place (`POST /api/settings/redact-rule` with the rule's index and its current value, `editRedactRule` in [`internal/gui/api.go`](../../internal/gui/api.go)). Every rule is checked by `redact.CheckRules` when settings are saved; an edit that doesn't compile, matches the empty string, repeats another rule or names a rule no longer at that index is refused with the rules left as they were. The masker reads the rules from settings on each call (`options` in [`internal/masking`](../../internal/masking/masking.go)), so a change applies to the next request.+ - Each request that had anything masked is recorded as it goes to the vendor (`noteMasked` in [`masked.go`](../../internal/gateway/masked.go), from chat, the Codex backend, search, images, video, System One, embeddings and audio; count_tokens is masked and not recorded). The masker says what it swapped through `gwmw.Teller` (`JSONHits`, `TextHits`: each placeholder's kind, its finder, `redact.ByRule`, `ByYours`, `ByWord`, `ByModel` or `ByKnown`, and a hint of its first characters and its length), and [`internal/maskrec`](../../internal/maskrec/maskrec.go) keeps one line per request in a file per day under `~/.config/magpie/masking`, 30 days and 100 MB in all: the agent, session, provider, model, and each placeholder with its kind, finder and hint, never the value. The text around each placeholder as the vendor saw it is kept in memory only, for the last 400 records, and what a placeholder stood for only in the masker's own memory (`Teller.Value`); after a restart neither is there. A recent call (`Call.Masked`) and the route trace (`Route.Masked`) carry the counts and the record's key, which the Gateway and Routing pages' badge opens in Settings → Privacy (`/api/masking/*`, [`internal/gui/masking.go`](../../internal/gui/masking.go)). A masker without `Teller`, a plugin's, masks as before and records nothing. Under a test binary testenv isolated nothing is written (`maskrec.inTest`).- `DecidesModel` gives a 400.- A gateway key held to some models (`keyHolds`, #882) gets a 403 for any other model.- A model the user disabled (#1867) gets a 403 for every caller (`idOff`), before anything is sent; see [Disabled models](#disabled-models).diff --git a/docs/subsystems/gui-shell.md b/docs/subsystems/gui-shell.mdindex 4ab2efccf..51b5d3114 100644--- a/docs/subsystems/gui-shell.md+++ b/docs/subsystems/gui-shell.md@@ -17,6 +17,7 @@ three.| Browser mode | `StartWeb` serves the page and the gateway for `magpie web`. The browser signs in with a run key in a link. Gateway mode shows Providers, Gateway, Routing, Usage, Sessions, Plugins and Settings on a machine that is only a gateway. `webHandler` gzips the page's text files and the API's JSON for a browser that takes gzip (`webCompressed`, with a `-gzip` ETag that still answers 304); the app's own windows read the page in-process and get it uncompressed. | [`web.go`](../../internal/gui/web.go), [`gatewaymode.go`](../../internal/gui/gatewaymode.go) || The page | `app.js` keeps one state object per view and renders it into a list, with no framework. The views have their own files: `plugins.js` and `context.js`, and those a plugin adds under `assets/plugins/<id>/` (the Gateway, Sessions, Library, Routing and Usage pages, see [plugin runtime](plugin-runtime.md)) (Usage's Context tab, and the context window card the Routing page draws under a request's story, fed by `/api/context` in `context.go`, which scores and tags each agent; clicking a part, a cell or an item opens its text from `/api/context/text`, the last 10 requests' masked bodies in memory (`openCtxText`), rendered 100 items and 20000 characters at a time; its "Tuned for you" cards are fed by `/api/tune` in `tune.go`, see [agent wiring](agent-wiring.md)). Strings go through `t()` with translations in `i18n.js`, and a plugin's own in its `i18n.js` (zh, zh-TW, ja and de; the English text is the key and the fallback). zh-TW is Traditional Chinese with Taiwan's words; a system language of zh-TW, zh-HK, zh-MO or any zh-Hant picks it, and the tray menu and notifications follow (`trayLang`). Dropdowns use `openProtoMenu`. Every app menu (`openProtoMenu`, `openRowMenu`, `openSessCombo`) is placed by `placeMenu`: under its anchor, or over it when the window runs out below; one that fits neither way takes the larger room, its height held to it, and scrolls inside itself, and a scroll inside a menu doesn't close it (#1437, `removed-groups-menu-scroll.test.cjs`). The highest any popover goes is `popTop`: 8px under `header.top` on a Mac window, where the traffic lights are drawn over the page, and 8px from the top elsewhere; `placeMenu`, the field picker's `placePop` (which also holds `#pop`'s height to that room) and an agent's model list (`openAgentModels`) all use it (#1573, `popover-mac-chrome.test.cjs`). Scrolling is done only by `scrollOnPurpose(e)` with the reader's event. | [`assets/app.js`](../../internal/gui/assets/app.js), [`assets/i18n.js`](../../internal/gui/assets/i18n.js) || Window navigation | In the window from 1024 points wide (`SIDE_AT`, the same width as app.css's `side-nav` rules), `layoutNav` moves the brand, the pages (`#nav`) and Settings into the sidebar (`#side`): the pages in four groups (Work, Models, Insights, Extend; a group none of whose pages shows, as in gateway mode, leaves out its name), the gateway's address at its foot (`sideGwRead`, read from `/api/providers` only while there is a sidebar), and the header a breadcrumb (`renderCrumb`). A page whose head holds only its tools (Library, Plugins, Settings) has them in the title bar after the breadcrumb, `tools` setting the room between it and the window's own buttons. When they don't fit there the tools show their icons alone (`tools-tight`; each keeps its name as `aria-label` and in its tooltip), then the breadcrumb drops its leading parts, and last the page's name is cut at its end, so no tool lies over the title (#2625, `title-bar-tools.test.cjs`). The open page's own tabs (`NAV_SUBS`: Usage, Library, Plugins, Settings) are listed under it as `.nav-s` rows (`role="tab"`, `data-of` the page tab's id) and the page's strip is hidden; a row's click clicks that tab at once, the selection (`.nav-hl`) having set off as the row was pressed (`navSubGlide`; back if the press leaves the row), and is the list's only shape: a row hovered or pressed has no ground of its own, and a `MutationObserver` draws the list again when the strip changes. A page's tabs share its one scroll, so a row that picks another tab opens it at its top (`scrollOnPurpose` with the row's click); the tab already open, picked again, leaves the page where it is (`sidebar-tab-top.test.cjs`). The list opens and folds with an iOS-like spring (`navSubEnter`, `navSubLeave`; none under reduced motion), and a folding list is out of reach at once. Narrower, the page beside a sidebar would be narrower than Usage's requests table needs (768 points), so they go back into the header's row, as a Mac sidebar gives way in a narrow window. At every width the reader can pick the sidebar or the header's tabs from the title bar's button (`#sideToggle`, ⌃⌘S or Ctrl+Shift+S; `toggleSideNav`), kept in localStorage `magpie.navLayout` (`side` or `top`) and so laid out from the first frame; none picked, the sidebar comes from `SIDE_AT`. Picked in a narrow window the sidebar is narrower (184 points under 860), and magpie web on a phone keeps its own tab bar and has no button. The two layouts trade places in a view transition (`navMorph`, app.css `.nav-morph`): the sidebar slides, each page's tab flies to its place in the other, and the page slides as one (its old image dropped, so no line prints twice); without view transitions, under reduced motion, or in the Linux app the change is at once: WebKitGTK with compositing off (`WEBKIT_DISABLE_COMPOSITING_MODE`, which `webkit_linux.go` sets under Wayland) crashes in a view transition. Elsewhere a transition still going after `MORPH_MAX` (1.5s) is skipped, so a webview that never ends one can't leave the window on the old layout's picture. See `side-toggle.test.cjs`. A window made wider or narrower drops what a click held (`heldWidth`): the page laid out again isn't the click's doing. The reader's own place is kept instead for a second while the page is laid out again (`relaid`, a `ResizeObserver` on the open page's parts calling `backToReader`): crossing `SIDE_AT` the browser clamps the scroll to each layout it passes through, and the page ended short of its foot (#1672, `text-size.test.cjs`). See `sidebar.test.cjs`. A sidebar longer than a short window keeps no scroll bar beside its pages: the native bars of `.side-scroll` and `.side-foot` are hidden, and `sideBars` lays a thin bar over each box's edge that shows while it scrolls, fades about a second after it stops (kept while hovered or dragged), can be dragged or clicked to page, and is gone when the box fits, the same in WebKit, WebView2 and WebKitGTK. See `sidebar-overlay-scrollbar.test.cjs`. The window opens on the page the reader last left, as a Mac app comes back as it was (molvqingtai and lc on Discord): `show` keeps the page in localStorage `magpie.lastView` (which the app's webview keeps across restarts, as it keeps `magpie.navLayout`), and `keptView` opens it at start. A page asked for in the address (`?view=`, the tray's links, a reload) wins, Agents too; a kept name that is no page opens the first page, and one gateway mode leaves out opens Providers (`show`). Opening on it scrolls nothing. See `last-page.test.cjs`. | [`assets/app.js`](../../internal/gui/assets/app.js), [`assets/app.css`](../../internal/gui/assets/app.css) |+| Settings › Privacy | The page opens on what masking kept from vendors (`drawMaskObs` in app.js, fed by `/api/masking/summary`): the values masked over 7 or 30 days with a chart of the days, the kinds, what only the local model found, agents to providers, and the values masked most, each shown only while pointed at (`maskTip`, `/api/masking/value`, which says it is gone after a restart). Its records (`drawMaskRecs`, `/api/masking/records` and `/api/masking/record`) take the page's place, filtered by kind, finder, agent or provider, a day at a time, each opening on the text the vendor saw with its placeholders marked. `maskedBadge`, on the Gateway page's recent calls and the Routing page's log, opens a request's own record (`openMaskRecord`). Below the card, *What to mask*, *How it's found* (the rules, always on while anything is masked, and the local model on top of them) and *Sharing*. See [gateway routing](gateway-routing.md) for what is recorded. | [`assets/app.js`](../../internal/gui/assets/app.js), [`masking.go`](../../internal/gui/masking.go) || Built-in plugins | Plugins › Built-in, the page's first tab (`drawBuiltin` in `plugins.js`, rows by `renderBuiltins` in `app.js`), lists magpie's built-in plugins, each with a name, a line on what it is and a switch: Routing groups, Library, Observability, Sessions, Warm-up and check-in, and the Gateway and Routing pages (Essential). A plugin on with settings has their rows under its row, drawn from its schema by the kernel's `ui.form`, and a change posts `{id, config}`. The Gateway page is drawn by its plugin's own scripts through the window's kernel (`kernel.js`, `assets/plugins/gateway/`); see the [plugin runtime](plugin-runtime.md), "The window's kernel". They are `plugins/*` registered with the [plugin runtime](plugin-runtime.md); `GET /api/plugins/builtin` says each one (`builtinJSON`: on is active, the user's own choice, what of it is in use) and `POST {id, on}` (or `{id, config}`) writes plugins.yml. A page marks what goes with a plugin `data-feature="<id>"`. Off, `featureStyle` hides everything so marked (rows drawn later too), `navHides` leaves its page's tab out of the sidebar and the header, `show` opens Agents for an address kept for it, and `setSetTab`, `partShows` and `findSettings` leave its Settings tab and rows out, of the search too. One in use stays on: its switch is disabled and the row says what uses it, and the POST refuses it off. A plugins.yml that doesn't parse is said above the list and no switch moves. `boot.js` says which are off (`featuresOff`) and their settings (`pluginConfig`) before the first paint, and `settings.builtins` carries the list on every state read. The TUI doesn't follow them. See `builtins.test.cjs`, `builtins_test.go`. | [`builtins.go`](../../internal/gui/builtins.go), [`assets/plugins.js`](../../internal/gui/assets/plugins.js), [`assets/app.js`](../../internal/gui/assets/app.js) || No GUI build | `-tags nogui` builds magpie without the GUI: `hasGUI` is false and `runGUI` says to use `magpie tui`. The files of `internal/gui` that need Wails build under `!nogui`. | [`gui_off.go`](../../internal/cli/gui_off.go) |diff --git a/docs/subsystems/plugin-runtime.md b/docs/subsystems/plugin-runtime.mdindex e126f85b1..0bf8fa57d 100644--- a/docs/subsystems/plugin-runtime.md+++ b/docs/subsystems/plugin-runtime.md@@ -221,6 +221,7 @@ failure policy, in [`internal/gwmw`](../../internal/gwmw/gwmw.go):| `gwmw.Agent` | A middleware on a request an agent sent, in the agent's own API, before the gateway routes it, and on its reply as the agent gets it: `Begin(Info) Run`, and the run's `Request` (a `Reject` ends the request in the agent's error shape), `Wrap` and `End`. `Use(a)` registers one until its remove; `Agents` lists them in order. `serveAgent` runs each in turn; with none it calls `serve` directly. It fails open, as the JS middleware always did. | [`gwmw.go`](../../internal/gwmw/gwmw.go), [`middleware.go`](../../internal/gateway/middleware.go) || `plugins/middleware` | Requires `gateway`, Essential. Apply registers `middleware.Chain`, the JS gateway middleware ([Gateway middleware](gateway-middleware.md)), whose place, API, limits and fail-open policy are unchanged. `InUse` is `middleware.On()`, the middleware plugins.json has on. Off, agents' requests go as they came and no middleware is called, while plugins.json is left as it is. The gateway's `KeepFresh` watches plugins.json too, so a middleware added from the CLI or the window turns it on within 2s, and a request before then asks the gate's refresh itself (`gwmw.AgentsWanted`: with no agent middleware registered and `middleware.On()` above 0, at most once a second), so the middleware runs on that request, as it did within a second before it was a plugin. | [`middleware.go`](../../plugins/middleware/middleware.go), [`run.go`](../../internal/middleware/run.go) || `gwmw.Masker` | Masks what magpie sends a vendor and puts the values back in what comes back: `JSON`, `Text`, `Restore` and `Writer`. `Mask(m)` serves one until its remove (the last registered serves). It fails closed: `Masking(wanted)` is the masker for a request, or `ErrUnmasked` when the settings ask for masking and none serves, and `Safely` makes a masker that panics an error. The gateway's call sites (`serve`, count_tokens, embeddings and audio, transcriptions, images, video, the search stand-ins and their queries, the Codex backend) answer either with 503 "…nothing was sent", and the vendor gets nothing. `Served`, with none, asks the gate's refresh (the gateway's is `pluginrt.Refresh`, set by `gwmw.Refresh` in `ListenAndServe`) at most once a second and never from inside one, then looks again, so masking turned on a moment ago turns the plugin on for the request that needs it. | [`gwmw.go`](../../internal/gwmw/gwmw.go), [`redact.go`](../../internal/gateway/redact.go), [`gateway.go`](../../internal/gateway/gateway.go) |+| `gwmw.Teller` | What a masker can say besides masking: `JSONHits` and `TextHits` mask as `JSON` and `Text` do and also return each placeholder made or reused (`Hit`: placeholder, kind, finder, head, length), and `Value` is what a placeholder stood for while this masker remembers it. Optional: the gateway records a request's masking only when the served masker is one (`masking.Masker` is), and Settings → Privacy's records and values need it; a masker without it masks the same and leaves no record. | [`gwmw.go`](../../internal/gwmw/gwmw.go), [`masked.go`](../../internal/gateway/masked.go), [`masking.go`](../../internal/masking/masking.go) || `plugins/redact` | Requires `settings` and `gateway`, Essential. Apply serves `masking.Masker`, which reads settings.json on each call and masks with `internal/redact` as before (it lives in `internal/masking` because `settings` imports `redact`). `InUse` is `masking.Wanted`: *Mask secrets*, *Mask personal data* or a word of the user's own; settings.json that can't be read is an error, so it stays on. Since it is in use whenever masking is asked for, it can only be off while nothing is; the 503 is for a refresh not yet seen or a plugins.yml edited by hand. | [`redact.go`](../../plugins/redact/redact.go), [`masking.go`](../../internal/masking/masking.go) |## Gateway extension points (P3): tooldiff --git a/internal/gateway/codex_backend.go b/internal/gateway/codex_backend.goindex 54e2a82d6..a8031a91a 100644--- a/internal/gateway/codex_backend.go+++ b/internal/gateway/codex_backend.go@@ -847,10 +847,11 @@ func (s *Server) codexUpstreamOn(w http.ResponseWriter, r *http.Request, rest ststart := time.Now()usage.Saw(agentOf(r))w0, body0 := w, body // as they came, for the fallbacks (codexFallBack)+ var maskedHits maskHitsif r.Method == http.MethodPost {var unmask func()var err error- w, body, unmask, err = redacted(w, body)+ w, body, unmask, maskedHits, err = redactedHits(w, body)defer unmask()if err != nil {writeError(w, provider.Responses, http.StatusServiceUnavailable, err.Error())@@ -895,6 +896,10 @@ func (s *Server) codexUpstreamOn(w http.ResponseWriter, r *http.Request, rest steffort := "" // the reasoning Codex asked for, which goes on as it isend := func(status int, msg string, tokens, out int) {}var tools *toolScan // whether the reply calls a tool, for a turn's end+ if rest != "/responses" {+ // no route of its own (a compaction, a memory): recorded alone+ noteMasked(r, "chat", maskedHits, string(body), 0, "openai", modelOf(body))+ }if rest == "/responses" {imageProvider := ""if !apiKey(r.Header) {@@ -913,6 +918,9 @@ func (s *Server) codexUpstreamOn(w http.ResponseWriter, r *http.Request, rest sttools = scantr = s.trace.beginFor(r, Route{turn: turn, dirOf: dirOf(provider.Responses, body), tools: tools, imageTurn: drawingTurnID(metadata.Turn), imageCaller: codexTurnKey(r, callerOf(r).agent), imageProvider: imageProvider, TitleLink: link, Time: start, Agent: agentOf(r), Session: sessionOf(r.Header), Conv: convOf(r.Header, body), ParentSession: titleParentSession(r.Header, metadata, kind), Kind: kind, Model: model, Effort: effort, Tier: bodyTier(provider.Responses, body), Provider: "openai", Subagent: subagentPickOf(r),Order: []Weighed{seat}, Tries: []Try{{ID: seat.ID, Model: model, Effort: effort, Start: start}}})+ if m := noteMasked(r, "chat", maskedHits, string(body), tr.ID, "openai", model); m != nil {+ s.trace.update(tr, func(t *Route) { t.Masked = m })+ }promptRead := s.inspectPrompt(tr, provider.Responses, body, body)end = func(status int, msg string, tokens, out int) {ms := time.Since(start).Milliseconds()@@ -1294,6 +1302,7 @@ func (s *Server) codexUpstreamOn(w http.ResponseWriter, r *http.Request, rest st}end(call.Status, call.Error, uu.Input+uu.Output+uu.CacheRead+uu.CacheWrite, uu.Output)call.Route = tr.ID+ call.Masked = s.trace.maskedOf(tr)s.record(call)servedProvider, servedAccount := codexServed(base, r.Header, on)rec := usage.Record{RouteID: tr.ID, Time: start, Agent: call.Agent,diff --git a/internal/gateway/codex_search.go b/internal/gateway/codex_search.goindex 59a3900d5..555444959 100644--- a/internal/gateway/codex_search.go+++ b/internal/gateway/codex_search.go@@ -123,11 +123,12 @@ func chatGPTSearchers(r *http.Request, model string) ([]provider.Provider, strin// with why returned.func (s *Server) alphaSearchOn(w http.ResponseWriter, r *http.Request, body []byte, model string, ons []provider.Provider) (bool, string) {start := time.Now()- w, body, unmask, err := redacted(w, body)+ w, body, unmask, hits, err := redactedHits(w, body)defer unmask()if err != nil {return false, err.Error()}+ noteMasked(r, "search", hits, string(body), 0, "openai", model)var errs []stringfor _, on := range ons {who := "OpenAI (" + on.Account.User + "): "diff --git a/internal/gateway/decide.go b/internal/gateway/decide.goindex ab5cf659c..1290d3af1 100644--- a/internal/gateway/decide.go+++ b/internal/gateway/decide.go@@ -384,16 +384,19 @@ func (s *Server) askJev(p provider.Provider, model string, intents []string, pre// maskedState masks what Jev is told of a turn, each as maskedText does.func maskedState(text, user, reply string) (string, string, string, error) {out := [3]string{text, user, reply}+ var all maskHitsfor i, v := range out {if v == "" {continue}- m, _, err := maskedText(v)+ _, m, h, err := maskTextHits(v)if err != nil {return "", "", "", err}out[i] = m+ all.add(h)}+ noteMasked(nil, "decide", all, strings.Join(out[:], "\n"), 0, "", "")return out[0], out[1], out[2], nil}diff --git a/internal/gateway/draw.go b/internal/gateway/draw.goindex 80aa9fb98..eeb03824c 100644--- a/internal/gateway/draw.go+++ b/internal/gateway/draw.go@@ -468,12 +468,14 @@ func (s *Server) images(edit bool) http.HandlerFunc {}}var unmask func()- w, d.Prompt, unmask, err = redactedPrompt(w, d.Prompt)+ var hits maskHits+ w, d.Prompt, unmask, hits, err = redactedPromptHits(w, d.Prompt)defer unmask()if err != nil {fail(http.StatusServiceUnavailable, err.Error())return}+ noteMasked(r, "image", hits, d.Prompt, 0, p.ID, call.Model)call.Provider, call.To = p.ID, provider.Chatctx, cancel := drawWithin(r.Context(), drawTimeout())defer cancel()diff --git a/internal/gateway/embeddings.go b/internal/gateway/embeddings.goindex 58fc831a1..13fc06dc1 100644--- a/internal/gateway/embeddings.go+++ b/internal/gateway/embeddings.go@@ -51,8 +51,9 @@ func (s *Server) retrieve(path, operation string) http.HandlerFunc {// model it may be asked of.type retrieval interface {// mask masks what goes to the vendors, and gives the writer that puts- // the masked values back in their answer- mask(w http.ResponseWriter) (http.ResponseWriter, func(), error)+ // the masked values back in their answer, what it swapped, and the+ // text it sent+ mask(w http.ResponseWriter) (http.ResponseWriter, func(), maskHits, string, error)// forModel is the body and its Content-Type for the provider's own id// of the modelforModel(model string) ([]byte, string, error)@@ -64,13 +65,13 @@ type retrieval interface {// jsonRetrieval is an embeddings or rerank body: a JSON object.type jsonRetrieval struct{ req map[string]json.RawMessage }-func (j *jsonRetrieval) mask(w http.ResponseWriter) (http.ResponseWriter, func(), error) {+func (j *jsonRetrieval) mask(w http.ResponseWriter) (http.ResponseWriter, func(), maskHits, string, error) {// masked once, as the settings say: a document a reranker gives back// has its secrets againall, _ := json.Marshal(j.req)- w, all, done, err := redacted(w, all)+ w, all, done, hits, err := redactedHits(w, all)json.Unmarshal(all, &j.req)- return w, done, err+ return w, done, hits, string(all), err}// answerType is JSON, whatever a vendor labels it.@@ -177,7 +178,9 @@ func (s *Server) serveRetrieval(w http.ResponseWriter, r *http.Request, path, opif slices.ContainsFunc(tries, func(c candidate) bool { return !c.p.SkipsRedaction() }) {var done func()var err error- w, done, err = req.mask(w)+ var hits maskHits+ var sent string+ w, done, hits, sent, err = req.mask(w)defer done()if err != nil { // fails closedcall.Status, call.Error = http.StatusServiceUnavailable, err.Error()@@ -185,6 +188,11 @@ func (s *Server) serveRetrieval(w http.ResponseWriter, r *http.Request, path, ops.record(call)return}+ what := "embeddings"+ if _, audio := req.(*audioForm); audio {+ what = "audio"+ }+ noteMasked(r, what, hits, sent, 0, tries[0].p.ID, asked)}for i, c := range tries {last := i == len(tries)-1diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.goindex e8ab620a5..2a7311465 100644--- a/internal/gateway/gateway.go+++ b/internal/gateway/gateway.go@@ -339,7 +339,9 @@ type Call struct {// a Codex subagent's (callKind) — "" for a turnKind string `json:"kind,omitempty"`// For: the request a call of magpie's own (a web search) was made for- For *CallFor `json:"for,omitempty"`+ For *CallFor `json:"for,omitempty"`+ // Masked: what masking swapped in it, nil for nothing+ Masked *Masked `json:"masked,omitempty"`Model string `json:"model"`Provider string `json:"provider"`From provider.Protocol `json:"from"`@@ -1586,7 +1588,7 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro// unmasked (unredactedRoute)plain := bodyvar masked []byte // the masked body, when plain goes instead- w, body, unmask, err := redacted(w, body)+ w, body, unmask, maskedHits, err := redactedHits(w, body)defer unmask()if err != nil { // fails closed: never sent unmaskedlog.Printf("gateway: %v", err)@@ -2280,6 +2282,13 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Proif watched != nil {watched.routeID = tr.ID}+ // what masking swapped, recorded for Settings → Privacy, unless the+ // request went as written+ if masked == nil {+ if m := noteMasked(r, "chat", maskedHits, string(body), tr.ID, p.ID, call.Model); m != nil {+ s.trace.update(tr, func(t *Route) { t.Masked = m })+ }+ }// the card's text is the masked body, even where the request went// unmaskedtextBody := body@@ -3580,6 +3589,7 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Prot.Archive = call.archiveName()})call.Route = tr.ID+ call.Masked = s.trace.maskedOf(tr)s.record(call)if call.To != "" {rec := usage.Record{RouteID: tr.ID, Time: start, Agent: call.Agent, Via: call.Via, Provider: call.Provider, Host: where, Model: model,diff --git a/internal/gateway/masked.go b/internal/gateway/masked.gonew file mode 100644index 000000000..d6799245c--- /dev/null+++ b/internal/gateway/masked.go@@ -0,0 +1,180 @@+package gateway++import (+ "net/http"+ "slices"+ "strconv"+ "strings"+ "unicode/utf8"++ "github.com/yetone/magpie/internal/gwmw"+ "github.com/yetone/magpie/internal/maskrec"+)++// Masked is what masking swapped in a request, for the Routing page and+// Recent calls to show: how many places, how many different values, of+// which kinds and found by which finder; Key is its record (maskrec).+type Masked struct {+ N int `json:"n"`+ Distinct int `json:"distinct"`+ Kinds map[string]int `json:"kinds,omitempty"` // different values by kind+ By map[string]int `json:"by,omitempty"` // different values by finder+ Key int64 `json:"key,omitempty"`+}++// maskHits are what a masker swapped: each value from a gwmw.Teller, and+// how many places in all.+type maskHits struct {+ n int+ hits []gwmw.Hit+}++func (h *maskHits) add(o maskHits) {+ h.n += o.n+ h.hits = append(h.hits, o.hits...)+}++// noteMasked records what masking swapped in a request that went to a+// vendor as sent: what it was (chat, search, image…), who sent it, the+// route it is in the trace (0 for none), and the provider and model it+// went to. It returns its summary, nil when nothing was masked.+func noteMasked(r *http.Request, what string, h maskHits, sent string, route int64, prov, model string) *Masked {+ if h.n == 0 && len(h.hits) == 0 {+ return nil+ }+ rec := maskrec.Record{What: what, Route: route, Provider: prov, Model: model, N: max(h.n, len(h.hits))}+ if r != nil {+ rec.Agent, rec.Session = agentOf(r), sessionOf(r.Header)+ }+ for _, x := range h.hits {+ rec.Values = append(rec.Values, maskrec.Value{Placeholder: x.Placeholder, Kind: x.Kind, By: x.By, Head: x.Head, Len: x.Len})+ }+ rec.Snips = snipsOf(sent, h.hits)+ rec = maskrec.Add(rec)+ m := &Masked{N: rec.N, Distinct: len(rec.Values), Key: rec.Key}+ for _, v := range rec.Values {+ if m.Kinds == nil {+ m.Kinds, m.By = map[string]int{}, map[string]int{}+ }+ m.Kinds[v.Kind]+++ m.By[v.By]+++ }+ return m+}++// snipRunes is how much of the text on each side of a value its snip has.+const snipRunes = 56++// snipsOf are, by placeholder, the text around the first place sent has+// each of hits, as the vendor saw it: what the record shows of a value,+// in memory only.+func snipsOf(sent string, hits []gwmw.Hit) map[string]string {+ if sent == "" || len(hits) == 0 {+ return nil+ }+ // in a JSON body a snip stays inside the string the value is in,+ // so it reads as the text, not as the body around it+ t := strings.TrimSpace(sent)+ inJSON := t != "" && (t[0] == '{' || t[0] == '[')+ out := map[string]string{}+ for _, x := range hits {+ if _, ok := out[x.Placeholder]; ok || len(out) >= 200 {+ continue+ }+ i := strings.Index(sent, x.Placeholder)+ if i < 0 {+ continue+ }+ // a value an earlier snip shows already has that one: the+ // record shows a line once, not once per value in it+ if s, ok := snipWith(out, x.Placeholder); ok {+ out[x.Placeholder] = s+ continue+ }+ a, b := i, i+len(x.Placeholder)+ start, end := 0, len(sent)+ for k := 0; k < snipRunes && a > start; k++ {+ if inJSON && sent[a-1] == '"' && !quoteEscaped(sent, a-1) {+ start = a+ break+ }+ _, n := utf8.DecodeLastRuneInString(sent[:a])+ a -= n+ }+ for k := 0; k < snipRunes && b < end; k++ {+ if inJSON && sent[b] == '"' && !quoteEscaped(sent, b) {+ end = b+ break+ }+ _, n := utf8.DecodeRuneInString(sent[b:])+ b += n+ }+ s := unescapeSnip(sent[a:b])+ if a > start {+ s = "…" + s+ }+ if b < end {+ s += "…"+ }+ out[x.Placeholder] = s+ }+ return out+}++// snipWith is a snip in out that shows p, the first one in sorted order.+func snipWith(out map[string]string, p string) (string, bool) {+ var found []string+ for _, s := range out {+ if strings.Contains(s, p) {+ found = append(found, s)+ }+ }+ if len(found) == 0 {+ return "", false+ }+ slices.Sort(found)+ return found[0], true+}++// quoteEscaped says whether the byte at i is behind an odd run of+// backslashes.+func quoteEscaped(s string, i int) bool {+ n := 0+ for j := i - 1; j >= 0 && s[j] == '\\'; j-- {+ n+++ }+ return n%2 == 1+}++// unescapeSnip is a piece of a JSON body read as text: its escapes+// undone where they are whole, a line break a space.+func unescapeSnip(s string) string {+ if !strings.Contains(s, `\`) {+ return s+ }+ var b strings.Builder+ for i := 0; i < len(s); i++ {+ c := s[i]+ if c != '\\' || i+1 >= len(s) {+ b.WriteByte(c)+ continue+ }+ i+++ switch s[i] {+ case 'n', 'r', 't':+ b.WriteByte(' ')+ case 'u':+ if i+4 < len(s) {+ if r, err := strconv.ParseUint(s[i+1:i+5], 16, 32); err == nil {+ b.WriteRune(rune(r))+ i += 4+ continue+ }+ }+ b.WriteString(`\u`)+ default:+ b.WriteByte(s[i])+ }+ }+ return b.String()+}diff --git a/internal/gateway/masked_test.go b/internal/gateway/masked_test.gonew file mode 100644index 000000000..013db7cf4--- /dev/null+++ b/internal/gateway/masked_test.go@@ -0,0 +1,119 @@+package gateway++import (+ "context"+ "io"+ "net/http"+ "net/http/httptest"+ "strings"+ "testing"++ "github.com/yetone/magpie/internal/gwmw"+ "github.com/yetone/magpie/internal/maskrec"+ "github.com/yetone/magpie/internal/provider"+ "github.com/yetone/magpie/internal/settings"+)++// A masked request is recorded for Settings → Privacy: each value by its+// placeholder, kind and the finder that found it, the text around it as+// the vendor saw it, and the route and call say how much was masked. The+// record never holds a value.+func TestMaskedRequestIsRecorded(t *testing.T) {+ const key, word = "sk-proj-abcdEFGH1234ijklMNOP5678qrst", "Bluefin"+ var got []byte+ up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ got, _ = io.ReadAll(r.Body)+ w.Header().Set("Content-Type", "application/json")+ io.WriteString(w, `{"id":"c1","model":"m1","choices":[{"index":0,"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}],"usage":{"prompt_tokens":3,"completion_tokens":1}}`)+ }))+ defer up.Close()+ t.Setenv("XDG_CONFIG_HOME", t.TempDir())+ t.Setenv("XDG_CACHE_HOME", t.TempDir())+ if err := provider.Save(provider.Provider{ID: "fake", Name: "Fake", Key: "k", Models: []string{"m1"}, Chat: up.URL + "/v1"}); err != nil {+ t.Fatal(err)+ }+ if err := settings.Save(settings.Settings{Redact: true, RedactWords: []string{word}}); err != nil {+ t.Fatal(err)+ }+ before, _ := maskrec.List(0, 1, maskrec.Filter{})+ s := New()+ rec := httptest.NewRecorder()+ req := httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(`{"model":"fake/m1","messages":[{"role":"user","content":"project `+word+`, key `+key+`"},{"role":"user","content":"again `+key+`"}]}`))+ req.Header.Set("User-Agent", "claude-cli/2.1.0")+ s.Handler().ServeHTTP(rec, req)+ if rec.Code != 200 || strings.Contains(string(got), key) {+ t.Fatalf("%d, sent %s", rec.Code, got)+ }+ rs, _ := maskrec.List(0, 1, maskrec.Filter{})+ if len(rs) != 1 || len(before) == 1 && rs[0].Key == before[0].Key {+ t.Fatalf("nothing recorded: %+v", rs)+ }+ r := rs[0]+ if r.What != "chat" || r.Provider != "fake" || r.Route == 0 || r.N != 3 || len(r.Values) != 2 {+ t.Fatalf("record: %+v", r)+ }+ by := map[string]maskrec.Value{}+ for _, v := range r.Values {+ by[v.By] = v+ if strings.Contains(v.Placeholder+v.Head, word) || strings.Contains(v.Placeholder+v.Head, key[3:]) {+ t.Fatalf("value in the record: %+v", v)+ }+ if snip := r.Snips[v.Placeholder]; !strings.Contains(snip, v.Placeholder) || strings.Contains(snip, key) || strings.Contains(snip, word) {+ t.Fatalf("snip of %s: %q", v.Placeholder, snip)+ }+ }+ if by["rule"].Kind != "API_KEY" || by["rule"].N != 2 || by["rule"].Head != "sk-" || by["word"].Kind != "TERM" || by["word"].Len != len(word) {+ t.Fatalf("values: %+v", r.Values)+ }+ ts := s.Trace(context.Background(), 0, 0)+ found := false+ for _, rt := range ts.Routes {+ if rt.ID == r.Route {+ found = true+ if rt.Masked == nil || rt.Masked.N != 3 || rt.Masked.Distinct != 2 || rt.Masked.By["word"] != 1 || rt.Masked.Key != r.Key {+ t.Fatalf("route's masked: %+v", rt.Masked)+ }+ }+ }+ if !found {+ t.Fatalf("route %d not in the trace", r.Route)+ }+ if c, ok := s.RouteCall(r.Route); !ok || c.Masked == nil || c.Masked.Distinct != 2 {+ t.Fatalf("call's masked: %+v %v", c.Masked, ok)+ }++ // a request with nothing to mask records nothing+ rec = httptest.NewRecorder()+ s.Handler().ServeHTTP(rec, httptest.NewRequest("POST", "/v1/chat/completions", strings.NewReader(`{"model":"fake/m1","messages":[{"role":"user","content":"hello there"}]}`)))+ if again, _ := maskrec.List(0, 1, maskrec.Filter{}); rec.Code != 200 || len(again) != 1 || again[0].Key != r.Key {+ t.Fatalf("%d, recorded %+v", rec.Code, again)+ }+}++// A snip of a JSON body is the text of the string the value is in, its+// escapes undone, never the body's keys around it; plain text is cut+// by length alone.+func TestSnipStaysInItsString(t *testing.T) {+ const p = "{{API_KEY_abcd1234}}"+ hits := []gwmw.Hit{{Placeholder: p}}+ sent := `{"model":"m1","messages":[{"role":"user","content":"say \"hi\" to ` + p + ` now\nplease"}]}`+ if got, want := snipsOf(sent, hits)[p], `say "hi" to `+p+` now please`; got != want {+ t.Fatalf("JSON snip = %q, want %q", got, want)+ }+ long := strings.Repeat("a", 80) + ` "` + p + `" ` + strings.Repeat("b", 80)+ got := snipsOf(long, hits)[p]+ if !strings.HasPrefix(got, "…") || !strings.HasSuffix(got, "…") || !strings.Contains(got, `"`+p+`"`) {+ t.Fatalf("text snip = %q", got)+ }+}++// Two values a few words apart share one snip, even when the text runs+// past the window on both sides: the record shows the line once.+func TestSnipSharedByNearValues(t *testing.T) {+ const p, q = "{{PERSON_aaaa1111}}", "{{PERSON_bbbb2222}}"+ sent := strings.Repeat("x ", 40) + "Ask " + p + " whether the results for " + q + " came back " + strings.Repeat("y ", 40)+ got := snipsOf(sent, []gwmw.Hit{{Placeholder: p}, {Placeholder: q}})+ if got[p] == "" || got[p] != got[q] || !strings.Contains(got[p], q) {+ t.Fatalf("snips = %q", got)+ }+}diff --git a/internal/gateway/middleware_off_test.go b/internal/gateway/middleware_off_test.goindex 6b3e9aa4e..bb6a872ac 100644--- a/internal/gateway/middleware_off_test.go+++ b/internal/gateway/middleware_off_test.go@@ -54,10 +54,14 @@ func (e *echoing) last() string {return e.sent[len(e.sent)-1]}-// panicky is a masker that fails.+// panicky is a masker that fails, on each way the gateway asks it to+// mask (a Teller is asked through JSONHits and TextHits).type panicky struct{ masking.Masker }-func (panicky) JSON([]byte) ([]byte, int) { panic("masker broke") }+func (panicky) JSON([]byte) ([]byte, int) { panic("masker broke") }+func (panicky) JSONHits([]byte) ([]byte, []gwmw.Hit) { panic("masker broke") }+func (panicky) Text(string) (string, int) { panic("masker broke") }+func (panicky) TextHits(string) (string, []gwmw.Hit) { panic("masker broke") }// The middleware point's masker with plugins/redact on and off. On, a// secret goes to the vendor as a placeholder and comes back as it was.diff --git a/internal/gateway/redact.go b/internal/gateway/redact.goindex 9c228189f..8d3ba70e9 100644--- a/internal/gateway/redact.go+++ b/internal/gateway/redact.go@@ -15,56 +15,84 @@ import (// fails closed: masking wanted with no masker serving, or a masker that// fails, is an error, and the request goes nowhere (gwmw.Masking).func redacted(w http.ResponseWriter, body []byte) (http.ResponseWriter, []byte, func(), error) {+ w, body, done, _, err := redactedHits(w, body)+ return w, body, done, err+}++// redactedHits is redacted with what it swapped, for the record of it+// (noteMasked).+func redactedHits(w http.ResponseWriter, body []byte) (http.ResponseWriter, []byte, func(), maskHits, error) {m, err := gwmw.Masking(redactionWanted())if m == nil {- return w, body, func() {}, err+ return w, body, func() {}, maskHits{}, err}type out struct {b []byte- n int+ h maskHits}- o, err := gwmw.Safely(func() out { b, n := m.JSON(body); return out{b, n} })- if err != nil || o.n == 0 {- return w, body, func() {}, err+ o, err := gwmw.Safely(func() out {+ if t, ok := m.(gwmw.Teller); ok {+ b, hits := t.JSONHits(body)+ return out{b, maskHits{len(hits), hits}}+ }+ b, n := m.JSON(body)+ return out{b, maskHits{n: n}}+ })+ if err != nil || o.h.n == 0 {+ return w, body, func() {}, maskHits{}, err}rw, done := m.Writer(w)- return rw, o.b, done, nil+ return rw, o.b, done, o.h, nil}// redactedPrompt masks a parsed image or video prompt before any vendor// body is built, including multipart forms, and restores the reply's text.// It fails closed as redacted does.func redactedPrompt(w http.ResponseWriter, prompt string) (http.ResponseWriter, string, func(), error) {- m, masked, n, err := maskText(prompt)- if err != nil || n == 0 {- return w, prompt, func() {}, err+ w, masked, done, _, err := redactedPromptHits(w, prompt)+ return w, masked, done, err+}++// redactedPromptHits is redactedPrompt with what it swapped.+func redactedPromptHits(w http.ResponseWriter, prompt string) (http.ResponseWriter, string, func(), maskHits, error) {+ m, masked, h, err := maskTextHits(prompt)+ if err != nil || h.n == 0 {+ return w, prompt, func() {}, maskHits{}, err}rw, done := m.Writer(w)- return rw, masked, done, nil+ return rw, masked, done, h, nil}// maskedText is s masked as the settings say, and how many values it// swapped; an error when it can't be (fail closed).func maskedText(s string) (string, int, error) {- _, masked, n, err := maskText(s)- return masked, n, err+ _, masked, h, err := maskTextHits(s)+ return masked, h.n, err}-// maskText is maskedText with the masker that masked it.-func maskText(s string) (gwmw.Masker, string, int, error) {+// maskTextHits is maskedText with the masker that masked it and what it+// swapped.+func maskTextHits(s string) (gwmw.Masker, string, maskHits, error) {m, err := gwmw.Masking(redactionWanted())if m == nil {- return nil, s, 0, err+ return nil, s, maskHits{}, err}type out struct {s string- n int+ h maskHits}- o, err := gwmw.Safely(func() out { t, n := m.Text(s); return out{t, n} })+ o, err := gwmw.Safely(func() out {+ if t, ok := m.(gwmw.Teller); ok {+ got, hits := t.TextHits(s)+ return out{got, maskHits{len(hits), hits}}+ }+ got, n := m.Text(s)+ return out{got, maskHits{n: n}}+ })if err != nil {- return nil, s, 0, err+ return nil, s, maskHits{}, err}- return m, o.s, o.n, nil+ return m, o.s, o.h, nil}// restored is s with what a masker masked before put back; s when none@@ -104,7 +132,14 @@ func maskedQuery(query string, p *provider.Provider) (string, error) {if p != nil && p.SkipsRedaction() {return query, nil}- masked, _, err := maskedText(query)+ _, masked, h, err := maskTextHits(query)+ if err == nil {+ id := ""+ if p != nil {+ id = p.ID+ }+ noteMasked(nil, "search", h, masked, 0, id, "")+ }return masked, err}diff --git a/internal/gateway/trace.go b/internal/gateway/trace.goindex 52f37e40c..b923dcfc5 100644--- a/internal/gateway/trace.go+++ b/internal/gateway/trace.go@@ -89,12 +89,15 @@ type Route struct {Order []Weighed `json:"order"` // who was to try it, first firstLeft []Weighed `json:"left,omitempty"`Tries []Try `json:"tries"`- Done bool `json:"done"`- Status int `json:"status,omitempty"`- Error string `json:"error,omitempty"`- Millis int64 `json:"ms,omitempty"`- Tokens int `json:"tokens,omitempty"`- Output int `json:"out,omitempty"` // of Tokens, the reply's+ // Masked: what masking swapped in the request before it went, nil+ // for nothing (noteMasked)+ Masked *Masked `json:"masked,omitempty"`+ Done bool `json:"done"`+ Status int `json:"status,omitempty"`+ Error string `json:"error,omitempty"`+ Millis int64 `json:"ms,omitempty"`+ Tokens int `json:"tokens,omitempty"`+ Output int `json:"out,omitempty"` // of Tokens, the reply's// Reasoning: of Output, the reply's reasoning, which its speed leaves// out (usage.DecodeOf)Reasoning int `json:"reasoning,omitempty"`@@ -624,6 +627,16 @@ func (t *trace) trim() {}// update changes a route under the lock.+// maskedOf is r's Masked, read under the lock.+func (t *trace) maskedOf(r *Route) *Masked {+ if r == nil {+ return nil+ }+ t.mu.Lock()+ defer t.mu.Unlock()+ return r.Masked+}+func (t *trace) update(r *Route, f func(r *Route)) {if r == nil {returndiff --git a/internal/gateway/transcribe.go b/internal/gateway/transcribe.goindex 08bd92370..1f0db4cf3 100644--- a/internal/gateway/transcribe.go+++ b/internal/gateway/transcribe.go@@ -93,15 +93,15 @@ func readAudioForm(ctype string, body []byte) (*audioForm, error) {// mask masks the prompt, the form's only text a vendor is sent; the// audio can't be, and the transcript gets the masked values back.-func (a *audioForm) mask(w http.ResponseWriter) (http.ResponseWriter, func(), error) {+func (a *audioForm) mask(w http.ResponseWriter) (http.ResponseWriter, func(), maskHits, string, error) {if a.prompt == "" {- return w, func() {}, nil+ return w, func() {}, maskHits{}, "", nil}- w, masked, done, err := redactedPrompt(w, a.prompt)+ w, masked, done, hits, err := redactedPromptHits(w, a.prompt)if masked != a.prompt {a.masked = &masked}- return w, done, err+ return w, done, hits, masked, err}// forModel is the form as sent when nothing in it changes; otherwise it isdiff --git a/internal/gateway/video.go b/internal/gateway/video.goindex d5ba3f332..803b077d5 100644--- a/internal/gateway/video.go+++ b/internal/gateway/video.go@@ -731,18 +731,26 @@ func (s *Server) videosCreate(w http.ResponseWriter, r *http.Request) {call.To = provider.Chatvar unmask func()prompt := f.Prompt- w, f.Prompt, unmask, err = redactedPrompt(w, f.Prompt)+ var hits maskHits+ w, f.Prompt, unmask, hits, err = redactedPromptHits(w, f.Prompt)defer unmask()// the fields passed on as they came are masked as the prompt is: a// negative_prompt is the user's text all the same- n := 0+ var extra maskHitsif err == nil {- n, err = maskStrings(f.Extra)+ err = maskStrings(f.Extra, &extra)}if err != nil { // fails closedfail(http.StatusServiceUnavailable, err.Error())return}+ n := extra.n+ hits.add(extra)+ sent := f.Prompt+ if b, e := json.Marshal(f.Extra); e == nil && extra.n > 0 {+ sent += " " + string(b)+ }+ noteMasked(r, "video", hits, sent, 0, call.Provider, call.Model)if n > 0 && f.Prompt == prompt { // the prompt's mask wraps w alreadyvar done func()w, done = restoring(w)@@ -1115,13 +1123,12 @@ func (s *Server) remoteVideoContent(ctx context.Context, w http.ResponseWriter,}// maskStrings masks every string in v (a decoded JSON value) in place as a-// prompt is masked, and says how many values it swapped; an error when it-// can't (fail closed).-func maskStrings(v any) (int, error) {- n := 0+// prompt is masked, adding what it swapped to h; an error when it can't+// (fail closed).+func maskStrings(v any, h *maskHits) error {one := func(str string) (string, error) {- m, c, err := maskedText(str)- n += c+ _, m, got, err := maskTextHits(str)+ h.add(got)return m, err}var err error@@ -1131,12 +1138,10 @@ func maskStrings(v any) (int, error) {if str, ok := e.(string); ok {x[k], err = one(str)} else {- var c int- c, err = maskStrings(e)- n += c+ err = maskStrings(e, h)}if err != nil {- return n, err+ return err}}case []any:@@ -1144,14 +1149,12 @@ func maskStrings(v any) (int, error) {if str, ok := e.(string); ok {x[i], err = one(str)} else {- var c int- c, err = maskStrings(e)- n += c+ err = maskStrings(e, h)}if err != nil {- return n, err+ return err}}}- return n, nil+ return nil}diff --git a/internal/gui/api.go b/internal/gui/api.goindex c2b109805..b0cd2bc82 100644--- a/internal/gui/api.go+++ b/internal/gui/api.go@@ -1136,6 +1136,7 @@ func Handler(w Windows, gw *gateway.Server) http.Handler {gatewayFixRoutes(mux)gatewayModeRoutes(mux)privacyModelRoutes(mux)+ maskingRoutes(mux)builtinRoutes(mux)mux.HandleFunc("GET /api/settings", func(rw http.ResponseWriter, r *http.Request) {access.MigrateLegacyLANKeyBestEffort()diff --git a/internal/gui/assets/app.css b/internal/gui/assets/app.cssindex 38872924e..19ce00848 100644--- a/internal/gui/assets/app.css+++ b/internal/gui/assets/app.css@@ -3198,6 +3198,151 @@ body:not(.side-nav) #view-settings .set-head.stuck::before { box-shadow: 0 1px 0.row.pref .sub.feat-why { color: var(--muted); font-size: 11px; }/* the local model's row says a download, a device or an error in full */#privacyModelRow .sub { white-space: normal; overflow-wrap: anywhere; }+/* Privacy › how it's found: the rules, always on while anything is masked,+ and the local model, which says why it can't be used while nothing is */+.static-pill { display: inline-flex; align-items: center; height: 24px; padding: 0 10px; border-radius: 7px; font-size: 12px; color: var(--muted); background: var(--pill); white-space: nowrap; }+.static-pill.on { color: var(--green); background: color-mix(in srgb, var(--green) 12%, transparent); }+#privacyModelRow.pm-wait .name { color: var(--muted); }++/* Privacy › what masking kept from vendors (renderMaskObs): a card over the+ switches, its colours the charts' own, a glow of the accent behind */+.mask-obs { position: relative; isolation: isolate; overflow: hidden; margin-bottom: 6px; padding: 14px 16px 12px; border-radius: 14px; border: 1px solid var(--line); background: var(--card); }+.mask-obs:empty { display: none; }+.mask-obs::before { content: ""; position: absolute; inset: 0; z-index: -1; pointer-events: none; background: radial-gradient(110% 90% at 100% 0%, color-mix(in srgb, var(--accent) 11%, transparent), transparent 62%), radial-gradient(70% 80% at 0% 100%, color-mix(in srgb, var(--c2) 6%, transparent), transparent 70%); }+.mask-obs.off::before { opacity: .35; }+.mo-head { display: flex; align-items: center; gap: 10px; min-height: 30px; }+.mo-head .grow { flex: 1; }+.mo-shield { flex: none; width: 30px; height: 30px; border-radius: 9px; display: inline-flex; align-items: center; justify-content: center; color: var(--accent); background: var(--accent-soft); box-shadow: inset 0 0 0 1px color-mix(in srgb, var(--accent) 22%, transparent); }+.mask-obs.off .mo-shield { color: var(--muted); background: var(--pill); box-shadow: none; }+.mo-title { min-width: 0; }+.mo-name { font-size: 13px; font-weight: 600; color: var(--fg); }+.mo-sub { font-size: 11.5px; line-height: 1.4; color: var(--muted); }+.mo-empty { margin-top: 10px; font-size: 12px; line-height: 1.5; color: var(--muted); }+.mo-hero { display: grid; grid-template-columns: auto minmax(0, 1fr); gap: 20px; align-items: end; margin-top: 14px; }+.mo-count { min-width: 0; padding-bottom: 14px; }+.mo-big { display: flex; align-items: baseline; gap: 8px; }+.mo-num { font-size: 40px; line-height: 1; font-weight: 650; letter-spacing: -.025em; font-variant-numeric: tabular-nums; color: var(--fg); }+@supports (-webkit-background-clip: text) {+ .mo-num { background: linear-gradient(180deg, var(--fg) 30%, color-mix(in srgb, var(--fg) 55%, var(--accent))); -webkit-background-clip: text; background-clip: text; -webkit-text-fill-color: transparent; }+}+.mo-unit { font-size: 12px; color: var(--muted); white-space: nowrap; }+.mo-meta { margin-top: 7px; font-size: 12px; color: var(--fg-2); white-space: nowrap; font-variant-numeric: tabular-nums; }+.mo-chart { position: relative; min-width: 0; height: 78px; }+.mo-svg { display: block; overflow: visible; }+.mo-line { fill: none; stroke: var(--accent); stroke-width: 1.75; stroke-linecap: round; stroke-linejoin: round; }+.mo-base { stroke: var(--line); stroke-width: 1; }+.mo-axis { font-size: 10px; fill: var(--faint); font-variant-numeric: tabular-nums; }+.mo-dot { fill: var(--accent); stroke: var(--card); stroke-width: 2; }+.mo-hover line { stroke: var(--line-2); stroke-dasharray: 2 3; }+.mo-ctip { position: absolute; top: 0; transform: translate(-50%, -100%); padding: 4px 8px; border-radius: 7px; border: 1px solid var(--line-2); background: var(--card); box-shadow: 0 6px 20px rgba(0, 0, 0, .12); font-size: 11px; white-space: nowrap; color: var(--fg); font-variant-numeric: tabular-nums; pointer-events: none; }+.mo-ctip[hidden] { display: none; }+.mo-sk-num { width: 120px; height: 40px; border-radius: 8px; }+.mo-sk-chart { height: 60px; border-radius: 8px; }+.mo-sec { margin-top: 16px; }+.mo-sec > .label, .mo-lh { display: flex; align-items: baseline; gap: 8px; margin-bottom: 8px; }+.mo-hint { font-size: 11px; color: var(--faint); }+.mo-bar { display: flex; gap: 2px; height: 10px; border-radius: 99px; overflow: hidden; }+.mo-seg { flex: 1 1 0; min-width: 4px; }+.mo-legend { display: flex; flex-wrap: wrap; gap: 5px 6px; margin-top: 9px; }+.mo-chip { display: inline-flex; align-items: center; gap: 6px; height: 24px; padding: 0 9px 0 8px; border-radius: 99px; border: 1px solid var(--line); background: transparent; color: var(--fg-2); font: inherit; font-size: 12px; cursor: default; }+.mo-chip:not(:disabled):hover { background: var(--pill); }+.mo-chip b { font-weight: 600; color: var(--fg); font-variant-numeric: tabular-nums; }+.mo-dotc { flex: none; width: 7px; height: 7px; border-radius: 50%; }+.mo-model { display: flex; align-items: center; gap: 10px; margin-top: 14px; padding: 8px 8px 8px 12px; border-radius: 10px; border: 1px solid color-mix(in srgb, var(--c6) 24%, transparent); background: linear-gradient(90deg, color-mix(in srgb, var(--c6) 12%, transparent), color-mix(in srgb, var(--c2) 7%, transparent)); font-size: 12.5px; line-height: 1.4; color: var(--fg); }+.mo-model .grow { flex: 1; }+.mo-model .text { flex: none; }+.mo-spark { flex: none; display: inline-flex; color: var(--c6); }+.mo-flow { display: grid; grid-template-columns: minmax(0, 1fr) minmax(56px, 1fr) minmax(0, 1fr); }+.mo-col { display: flex; flex-direction: column; justify-content: center; gap: 4px; min-width: 0; }+.mo-node { display: flex; align-items: center; gap: 7px; height: 30px; min-width: 0; padding: 0 9px; border-radius: 8px; border: 1px solid var(--line); background: var(--card-2); color: var(--fg); font: inherit; font-size: 12px; cursor: default; }+.mo-node:not(:disabled):hover { background: var(--pill); }+.mo-node:disabled { color: var(--muted); }+.mo-node .nm { flex: 1; min-width: 0; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; text-align: left; }+.mo-node .ct { flex: none; color: var(--muted); font-variant-numeric: tabular-nums; }+.mo-mid { position: relative; min-width: 0; }+.mo-mid svg { position: absolute; inset: 0; width: 100%; height: 100%; overflow: visible; }+.mo-rib { fill: none; stroke-linecap: butt; }+.mo-rib.out { stroke: var(--accent); stroke-opacity: .42; }+.mo-hub { position: absolute; left: 50%; top: 50%; width: 32px; height: 32px; margin: -16px 0 0 -16px; border-radius: 50%; display: flex; align-items: center; justify-content: center; color: var(--accent-fg); background: var(--accent); box-shadow: 0 0 0 5px color-mix(in srgb, var(--accent) 16%, transparent), 0 4px 14px color-mix(in srgb, var(--accent) 32%, transparent); }+.mo-values { display: flex; flex-wrap: wrap; gap: 6px; }+.mo-val { display: inline-flex; align-items: center; gap: 6px; max-width: 100%; height: 26px; padding: 0 9px; border-radius: 8px; border: 1px solid var(--line); background: var(--card-2); color: var(--fg); font: inherit; font-size: 12px; cursor: default; outline: none; }+.mo-val:hover { border-color: var(--line-2); background: var(--pill); }+.mo-val:focus-visible { border-color: var(--line-2); background: var(--pill); }+.mo-val .sh { min-width: 0; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; font-family: var(--code); font-size: 11.5px; letter-spacing: .02em; }+.mo-val .x { flex: none; color: var(--muted); font-variant-numeric: tabular-nums; }+.mo-foot { display: flex; align-items: center; gap: 10px; margin-top: 14px; padding-top: 10px; border-top: 1px solid var(--line); }+.mo-note { flex: 1; min-width: 0; font-size: 11.5px; line-height: 1.4; color: var(--muted); }+.mo-open { flex: none; display: inline-flex; align-items: center; gap: 2px; }+@media (max-width: 560px) {+ .mo-hero { grid-template-columns: minmax(0, 1fr); gap: 4px; }+ .mo-count { padding-bottom: 0; }+ .mo-flow { grid-template-columns: minmax(0, 1fr) 44px minmax(0, 1fr); }+ .mo-node .ic { display: none; }+}++/* what a masked value was, beside it while it is pointed at */+.mask-tip { position: fixed; z-index: 1000; max-width: min(360px, calc(100vw - 16px)); padding: 8px 11px 9px; border-radius: 10px; border: 1px solid var(--line-2); background: var(--card); box-shadow: 0 12px 32px rgba(0, 0, 0, .18); font-size: 12px; color: var(--fg); pointer-events: none; animation: mask-tip .16s var(--ease); }+.mask-tip .k { display: flex; align-items: center; gap: 6px; font-size: 11px; color: var(--muted); }+.mask-tip .v { margin-top: 5px; font-family: var(--code); font-size: 12.5px; overflow-wrap: anywhere; }+.mask-tip .v.gone { font-family: inherit; font-size: 12px; line-height: 1.45; color: var(--muted); }+.mask-tip .m { margin-top: 4px; font-size: 11px; color: var(--faint); }+@keyframes mask-tip { from { opacity: 0; transform: translateY(3px); } }+@media (prefers-reduced-motion: reduce) { .mask-tip { animation: none; } }++/* the records, in the Privacy page's place while they are open */+#view-settings:not(.set-searching) #setPage-privacy.recs > :not(.mask-recs) { display: none; }+.set-searching .mask-obs, .set-searching .mask-recs { display: none; }+.mask-recs[hidden] { display: none; }+.mr-head { display: flex; align-items: center; gap: 10px; min-height: 28px; }+.mr-back { display: inline-flex; align-items: center; gap: 1px; margin-left: -4px; }+.mr-title { font-size: 13px; font-weight: 600; color: var(--fg); }+.mr-filters { display: flex; flex-wrap: wrap; gap: 6px; margin: 6px 0 2px; }+.mr-filters:empty { display: none; }+.mask-recs .row-head { margin-top: 8px; }+.mr-group { overflow: hidden; }+.mr-item + .mr-item { border-top: 1px solid var(--line); }+.mr-row { display: flex; align-items: center; gap: 10px; width: 100%; min-height: 42px; padding: 6px 12px 6px 10px; border: 0; background: transparent; color: var(--fg); font: inherit; font-size: 12.5px; text-align: left; cursor: default; }+.mr-row:hover { background: var(--pill); }+.mr-row:focus-visible { outline: none; box-shadow: inset 0 0 0 2px var(--accent-soft); }+.mr-chev { flex: none; display: inline-flex; color: var(--faint); transition: transform .2s var(--ease); }+.mr-item.open .mr-chev { transform: rotate(90deg); }+.mr-time { flex: none; color: var(--muted); font-variant-numeric: tabular-nums; }+.mr-route { flex: 1 1 auto; display: flex; align-items: center; gap: 5px; min-width: 0; overflow: hidden; white-space: nowrap; }+.mr-route .who { flex: none; max-width: 40%; overflow: hidden; text-overflow: ellipsis; }+.mr-route .arrow { flex: none; color: var(--faint); }+.mr-route .model { min-width: 0; overflow: hidden; text-overflow: ellipsis; color: var(--muted); }+.mr-route .what { flex: none; height: 18px; padding: 0 6px; border-radius: 5px; display: inline-flex; align-items: center; font-size: 10.5px; color: var(--muted); background: var(--pill); }+.mr-kinds { flex: none; display: flex; gap: 4px; }+.mr-kind { display: inline-flex; align-items: center; gap: 5px; height: 20px; padding: 0 7px; border-radius: 99px; border: 1px solid var(--line); background: var(--card-2); font-size: 11px; color: var(--fg-2); white-space: nowrap; }+.mr-kind b { font-weight: 600; color: var(--muted); font-variant-numeric: tabular-nums; }+.mr-n { flex: none; min-width: 20px; text-align: right; font-weight: 600; font-variant-numeric: tabular-nums; }+.mr-body { padding: 0 14px 12px 34px; }+.mr-val { display: flex; align-items: center; gap: 8px; min-height: 26px; font-size: 12px; }+.mr-k { color: var(--fg-2); }+.mr-shape { font-family: var(--code); font-size: 11.5px; color: var(--fg); border-radius: 4px; padding: 1px 3px; outline: none; }+.mr-shape.known { cursor: default; text-decoration: underline dotted var(--faint); text-underline-offset: 3px; }+.mr-shape.known:hover { background: var(--pill); }+.mr-shape.known:focus-visible { background: var(--pill); }+.mr-by { display: inline-flex; align-items: center; height: 18px; padding: 0 6px; border-radius: 5px; font-size: 10.5px; color: var(--muted); background: var(--pill); }+.mr-by.model { color: var(--c6); background: color-mix(in srgb, var(--c6) 14%, transparent); }+.mr-x { color: var(--muted); font-variant-numeric: tabular-nums; }+.mr-snip { margin-top: 8px; padding: 9px 11px; border-radius: 8px; border: 1px solid var(--line); background: var(--card-2); font-family: var(--code); font-size: 11.5px; line-height: 1.65; color: var(--fg-2); white-space: pre-wrap; overflow-wrap: anywhere; }+.mr-ph { --k: var(--accent); padding: 1px 4px; border-radius: 4px; color: var(--fg); background: color-mix(in srgb, var(--k) 16%, transparent); box-shadow: inset 0 0 0 1px color-mix(in srgb, var(--k) 30%, transparent); outline: none; cursor: default; }+.mr-ph:hover { background: color-mix(in srgb, var(--k) 26%, transparent); }+.mr-ph:focus-visible { background: color-mix(in srgb, var(--k) 26%, transparent); }+.mr-note { margin-top: 8px; font-size: 11.5px; color: var(--muted); }+.mr-none { padding: 26px 12px; text-align: center; font-size: 12.5px; color: var(--muted); }+.mr-none.err { color: var(--red); }+.mr-more { display: block; margin: 10px auto 0; }+@media (max-width: 560px) {+ .mr-kind:not(:first-child) { display: none; }+ .mr-route .model { display: none; }+}++/* a request's masked values, on Recent calls and the Routing page */+.mask-badge { flex: none; display: inline-flex; align-items: center; gap: 3px; height: 18px; padding: 0 6px 0 5px; border: 0; border-radius: 99px; background: var(--accent-soft); color: var(--accent); font: inherit; font-size: 11px; font-weight: 600; line-height: 1; font-variant-numeric: tabular-nums; cursor: default; }+.mask-badge:hover { background: color-mix(in srgb, var(--accent) 22%, transparent); }+.mask-badge:focus-visible { outline: none; box-shadow: 0 0 0 2px var(--accent-soft), inset 0 0 0 1px var(--accent); }/* a plugin's settings, under its row (kernel.js ui.form) */.row.pref.feat-set { min-height: 44px; padding-left: 30px; }.row.pref.feat-set .name { font-size: 12.5px; }diff --git a/internal/gui/assets/app.js b/internal/gui/assets/app.jsindex ad0c025a4..0ebd1561b 100644--- a/internal/gui/assets/app.js+++ b/internal/gui/assets/app.js@@ -21711,19 +21711,26 @@ async function discoveryOffWarning() {// renderRedact: what the gateway masks before a request goes to a vendor —// secrets, personal data, the user's own words — and puts back in what the-// vendor answers.+// vendor answers. The page says it in three parts under what masking did+// (renderMaskObs): what to mask, how it is found (the rules always, the+// local model beside them: layered, not one or the other) and what is+// shared.function renderRedact(s, keep) {- const box = $("#redactList");+ const box = $("#redactList"), find = $("#findList"), share = $("#shareList");box.replaceChildren();- const row = (name, sub, ...tools) => {+ find.replaceChildren();+ share.replaceChildren();+ const rowIn = (into, name, sub, ...tools) => {const r = el("div", "row pref");const who = el("div", "who");who.append(el("div", "name", name), el("div", "sub", sub));const val = el("div", "val");val.append(...tools);r.append(who, val);- box.append(r);+ into.append(r);+ return r;};+ const row = (...a) => rowIn(box, ...a);const onOff = (on, fn) => segs([["off", t("Off")], ["on", t("On")]], on ? "on" : "off", (v) => fn(v === "on"));row(t("Mask secrets"), t("API keys, private keys, tokens and passwords go to vendors as placeholders, and come back as they were"),onOff(s.redact, (redact) => savePrefs({ ...keep, redact })));@@ -21746,11 +21753,6 @@ function renderRedact(s, keep) {r.append(who, val);box.append(r);}- // the local model finds more of what the switches above mask- if (s.redact || s.redactPersonal) privacyModelRow(box);- // Routing's and Usage's Hide accounts, here too, where privacy is looked for- if (window.hideAccounts) row(t("Hide accounts"), t("Email addresses and account names on Usage and Routing are blurred, for a screenshot to share"),- onOff(window.hideAccounts.on(), (on) => { window.hideAccounts.set(on); renderSettings(); }));const words = (s.redactWords || []).join(", ");const i = input(words, t("names, codenames, hosts"));i.className = "words";@@ -21763,11 +21765,22 @@ function renderRedact(s, keep) {i.onblur = save;row(t("Masked words"), t("Your own words to keep from vendors, separated by commas"), i);renderRedactRules(s, row);- row(t("Count me as a user"), t("Once a day, a random id for this computer with magpie's version and system"),+ // how it is found: the rules on every request that masks anything, and+ // the local model on top of them, for what no pattern can tell+ const masking = s.redact || s.redactPersonal;+ const why = t("Turn on Mask secrets or Mask personal data first");+ const always = el("span", "static-pill" + (masking ? " on" : ""), masking ? t("Always on") : t("Off"));+ rowIn(find, t("Rules"), masking ? t("Keys, tokens, emails and numbers whose checksum adds up: patterns that are certain, checked on every request") : why, always).id = "maskRulesRow";+ privacyModelRow(find, masking ? "" : why);+ // Routing's and Usage's Hide accounts, here too, where privacy is looked for+ if (window.hideAccounts) rowIn(share, t("Hide accounts"), t("Email addresses and account names on Usage and Routing are blurred, for a screenshot to share"),+ onOff(window.hideAccounts.on(), (on) => { window.hideAccounts.set(on); renderSettings(); }));+ rowIn(share, t("Count me as a user"), t("Once a day, a random id for this computer with magpie's version and system"),onOff(!s.noStats, (on) => savePrefs({ ...keep, noStats: !on })));// rides on that event: nothing goes without it- if (!s.noStats) row(t("Share the agents, providers and models I use"), t("Sent with that event, by magpie's own ids, with how many of each, and how often each partner was shown, opened and added; a provider you added yourself is only “custom”. No names, addresses, accounts, keys or usage"),+ if (!s.noStats) rowIn(share, t("Share the agents, providers and models I use"), t("Sent with that event, by magpie's own ids, with how many of each, and how often each partner was shown, opened and added; a provider you added yourself is only “custom”. No names, addresses, accounts, keys or usage"),onOff(!s.noUsageStats, (on) => savePrefs({ ...keep, noUsageStats: !on })));+ renderMaskObs(s);}// privacyModelRow: the local masking model (internal/pii), downloaded when@@ -21775,10 +21788,18 @@ function renderRedact(s, keep) {// (privacy-model): read when the row is drawn, and each second while it// downloads or starts.let pmState = null, pmTimer = 0;-function privacyModelRow(box) {- const r = el("div", "row pref");+function privacyModelRow(box, why) {+ const r = el("div", "row pref" + (why ? " pm-wait" : ""));r.id = "privacyModelRow";box.append(r);+ // nothing is masked: the model has nothing to find more of+ if (why) {+ clearTimeout(pmTimer);+ const who = el("div", "who");+ who.append(el("div", "name", t("Find with a local model")), el("div", "sub", why));+ r.append(who);+ return;+ }let tries = 0;const draw = (m) => {if (!r.isConnected) return;@@ -21825,6 +21846,699 @@ function pmSub(m) {return t("Names, addresses and passwords no rule can find, found by a model that runs on this computer. Turning it on downloads {size} once", { size: fmtBytes(m.size) });}+// ---------- what masking swapped (maskrec) ----------+// Settings › Privacy opens on what masking kept from vendors: how much, of+// which kinds, found how, from which agents to which providers, over 7 or+// 30 days (/api/masking/summary), and each request's record under it+// (/api/masking/records). What a placeholder stood for is asked only while+// the reader points at it (/api/masking/value): it is in this magpie's+// memory alone, never on disk, so after a restart it is gone.+const MASK_KINDS = {+ API_KEY: "API keys", PRIVATE_KEY: "Private keys", TOKEN: "Tokens", PASSWORD: "Passwords", SECRET: "Secrets",+ EMAIL: "Emails", PHONE: "Phone numbers", ID_CARD: "ID numbers", BANK_CARD: "Bank cards",+ PERSON: "Names", ADDRESS: "Addresses", ACCOUNT: "Account numbers",+ SSN: "SSNs", PASSPORT: "Passports", IBAN: "IBANs", BIRTHDAY: "Birthdays", MAC: "MAC addresses", SERIAL: "Serial numbers",+ USERNAME: "User names", HOSTNAME: "Computer names", IP: "IP addresses", BUCKET: "Buckets", TERM: "Your words",+};+// who found a value (redact.By*)+const MASK_BY = { rule: "Rules", yours: "Your rules", word: "Your words", model: "Local model", known: "Seen before" };+// what a request was, beside a chat (gateway's noteMasked)+const MASK_WHAT = { search: "Web search", image: "Image", video: "Video", decide: "System One", embeddings: "Embeddings", audio: "Audio" };+const SHIELD = "M8 1.8 13 3.6v4.1c0 3-2.1 5.4-5 6.5-2.9-1.1-5-3.5-5-6.5V3.6Z";+const SPARK = "M7 2.2c.5 2.9 1.4 3.8 4.3 4.3-2.9.5-3.8 1.4-4.3 4.3-.5-2.9-1.4-3.8-4.3-4.3 2.9-.5 3.8-1.4 4.3-4.3Z M12.2 10.2c.2 1.1.6 1.5 1.7 1.7-1.1.2-1.5.6-1.7 1.7-.2-1.1-.6-1.5-1.7-1.7 1.1-.2 1.5-.6 1.7-1.7Z";+const PLACEHOLDER = /\{\{[^{}\s]{1,64}\}\}/g;+let maskS = null; // the settings the card was drawn with+let maskSum = null, maskSumDays = 0, maskAt = 0, maskBusy = null, maskBusyDays = 0;+let maskDays = 7; // the card's span, kept per machine+try { if (localStorage.getItem("magpie.maskDays") === "30") maskDays = 30; } catch {}+let maskDrawn = ""; // what the card says, so a redraw that changes nothing is skipped+let maskView = null; // the records, while they are open+let maskGrad = 0;+const maskValues = new Map(); // placeholder → what it stood for, as this magpie told it+const maskKindName = (k) => MASK_KINDS[k] ? t(MASK_KINDS[k]) : k;+const maskByName = (b) => MASK_BY[b] ? t(MASK_BY[b]) : b;+const stillMotion = () => matchMedia("(prefers-reduced-motion: reduce)").matches;+// a kind's colour is its place in the summary's ranking, so the bar, the+// legend, the chips and the records say a kind in one colour+function maskColor(k) {+ let i = (maskSum?.kinds || []).findIndex((x) => x.name === k);+ if (i < 0) { i = 0; for (const c of k || "") i = (i * 31 + c.charCodeAt(0)) >>> 0; }+ return `var(--c${(i % 7) + 1})`;+}+// what a value looks like masked: what it starts with, the rest dots+function maskShape(head, len) {+ head = head || "";+ const rest = Math.max(1, Math.min((len || 0) - [...head].length, 8));+ return head + "•".repeat(rest);+}+// the providers' names, read once when Settings opened before any page+// that loads them+let maskProv = null, maskProvAsked = false;+function maskWho(id, kind) {+ const x = kind === "agent" ? state.agents?.find((a) => a.id === id) : (providers || maskProv)?.providers?.find((p) => p.id === id);+ return { name: x?.name || id || t("Unknown"), icon: x?.icon };+}+function maskDot(k) {+ const d = el("span", "mo-dotc");+ d.style.background = maskColor(k);+ return d;+}+function maskDayName(when) {+ const d = new Date(when), now = new Date();+ const days = Math.round((new Date(now.toDateString()) - new Date(d.toDateString())) / 86400000);+ if (days === 0) return t("Today");+ if (days === 1) return t("Yesterday");+ return d.toLocaleDateString(intlLang(), { weekday: "short", month: "short", day: "numeric" });+}++// maskValue: what placeholder p stood for, or null where this magpie no+// longer knows. A value it doesn't know isn't kept as such: the same value+// masked again after a restart is known again.+function maskValue(p) {+ if (maskValues.has(p)) return Promise.resolve(maskValues.get(p));+ return api("masking/value?p=" + encodeURIComponent(p)).then((r) => {+ if (typeof r?.value !== "string") return null;+ maskValues.set(p, r.value);+ return r.value;+ }).catch(() => null);+}+// maskTip shows what a masked value was beside it, while it is pointed at+// or focused: its kind, who found it, and the value itself, asked for then+let maskTipEl = null;+function maskTip(anchor, v) {+ maskTipOff();+ const tip = maskTipEl = el("div", "mask-tip");+ tip.setAttribute("role", "tooltip");+ const k = el("div", "k");+ k.append(maskDot(v.kind), el("span", "", maskKindName(v.kind) + " · " + maskByName(v.by)));+ const val = el("div", "v", "…");+ tip.append(k, val);+ if (v.meta) tip.append(el("div", "m", v.meta));+ document.body.append(tip);+ const place = () => {+ const a = anchor.getBoundingClientRect(), b = tip.getBoundingClientRect();+ const left = Math.max(8, Math.min(a.left + a.width / 2 - b.width / 2, innerWidth - b.width - 8));+ const above = a.top - b.height - 8;+ tip.style.left = left + "px";+ tip.style.top = (above > 8 ? above : a.bottom + 8) + "px";+ };+ place();+ maskValue(v.p).then((s) => {+ if (maskTipEl !== tip) return;+ if (s === null) {+ val.className = "v gone";+ val.textContent = t("Not in memory: magpie has restarted since. The value itself is never written to disk");+ } else val.textContent = s;+ place();+ });+}+function maskTipOff() {+ maskTipEl?.remove();+ maskTipEl = null;+}+function maskTipOn(anchor, v) {+ anchor.onmouseenter = anchor.onfocus = () => maskTip(anchor, v);+ anchor.onmouseleave = anchor.onblur = maskTipOff;+}++// maskLoad reads the summary for the days shown, when what is kept is+// older than 10s or for other days; forced, it reads again whatever is kept+function maskLoad(force) {+ const days = maskDays;+ if (maskBusy && maskBusyDays === days) return maskBusy;+ if (!force && maskSum && maskSumDays === days && Date.now() - maskAt < 10000) return null;+ maskBusyDays = days;+ const p = maskBusy = api("masking/summary?days=" + days).then((sum) => {+ if (days !== maskDays) return;+ maskSum = { days: [], kinds: [], agents: [], providers: [], values: [], by: {}, n: 0, distinct: 0, requests: 0, ...(sum || {}) };+ maskSumDays = days;+ maskAt = Date.now();+ }).catch(() => {}).finally(() => { if (maskBusy === p) maskBusy = null; });+ return p;+}++// renderMaskObs draws the card with the summary kept, and again when a+// fresh one is in+function renderMaskObs(s) {+ maskS = s;+ if (!providers && !maskProvAsked) {+ maskProvAsked = true;+ api("providers").then((r) => { maskProv = r; drawMaskObs(); if (maskView) drawMaskRecs(); }).catch(() => { maskProvAsked = false; });+ }+ if (maskView && maskView.lang !== document.documentElement.lang) drawMaskRecs();+ drawMaskObs();+ maskLoad()?.then(drawMaskObs);+}+// while Privacy is on screen the card keeps up with requests as they go+setInterval(() => {+ if (view !== "settings" || setShown !== "privacy" || document.hidden || !maskS) return;+ maskLoad(true)?.then(drawMaskObs);+ if (maskView && !maskView.busy) maskRefreshRecs();+}, 15000);++function drawMaskObs() {+ const box = $("#maskObs");+ if (!box || !maskS) return;+ const s = maskS;+ const sum = maskSumDays === maskDays ? maskSum : null;+ const on = !!(s.redact || s.redactPersonal);+ const key = JSON.stringify([document.documentElement.lang, maskDays, on, sum, state.agents?.length, (providers || maskProv)?.providers?.length]);+ if (key === maskDrawn && box.firstChild) return;+ maskDrawn = key;+ // it moves when it first has numbers to say, and for other days: not+ // on every poll+ const move = !!sum && box.dataset.drawn !== String(maskDays) && !stillMotion();+ if (sum) box.dataset.drawn = String(maskDays);+ maskTipOff();+ box.replaceChildren();+ box.classList.toggle("off", !on);++ const head = el("div", "mo-head");+ const shield = el("span", "mo-shield");+ shield.append(svg(SHIELD, 17, 1.5));+ const words = el("div", "mo-title");+ words.append(el("div", "mo-name", t("Kept from vendors")),+ el("div", "mo-sub", on ? t("Masked on this computer before each request left it, and put back in each answer") : t("Masking is off: requests go to vendors as they are")));+ head.append(shield, words, el("span", "grow"));+ if (sum?.requests || maskDays === 30) head.append(segs([["7", t("7 days")], ["30", t("30 days")]], String(maskDays), (v) => {+ maskDays = +v;+ try { localStorage.setItem("magpie.maskDays", v); } catch {}+ maskLoad(true)?.then(drawMaskObs);+ }));+ box.append(head);+ if (!sum) {+ const sk = el("div", "mo-hero");+ sk.append(el("span", "skeleton mo-sk-num"), el("span", "skeleton mo-sk-chart"));+ box.append(sk);+ return;+ }+ if (!sum.requests && !on) {+ box.append(el("div", "mo-empty", t("Turn on Mask secrets or Mask personal data below, and what is kept from vendors shows here, request by request")));+ return;+ }++ // the number, and the days under it+ const hero = el("div", "mo-hero");+ const count = el("div", "mo-count");+ const big = el("div", "mo-big");+ const num = el("span", "mo-num", fmtN(sum.n));+ big.append(num, el("span", "mo-unit", t(sum.n === 1 ? "value masked" : "values masked")));+ count.append(big, el("div", "mo-meta", sum.requests+ ? t(sum.distinct === 1 ? "1 different value" : "{n} different values", { n: fmtN(sum.distinct) }) + " · " + t(sum.requests === 1 ? "1 request" : "{n} requests", { n: fmtN(sum.requests) })+ : t("Nothing yet in these days")));+ const chart = el("div", "mo-chart");+ hero.append(count, chart);+ box.append(hero);+ maskChart(chart, sum.days, move);+ if (move && sum.n) maskCountUp(num, sum.n);+ if (!sum.requests) {+ box.append(el("div", "mo-empty", t("A request that carries a key, a password or personal data shows here as it goes")));+ return;+ }++ // what was kept, by kind+ if (sum.kinds.length) {+ const sec = el("div", "mo-sec");+ sec.append(el("div", "label", t("What was kept")));+ const bar = el("div", "mo-bar");+ const legend = el("div", "mo-legend");+ const top = sum.kinds.slice(0, 6), rest = sum.kinds.slice(6);+ const parts = top.map((k) => ({ ...k, c: maskColor(k.name), label: maskKindName(k.name) }));+ if (rest.length) parts.push({ name: "", n: rest.reduce((a, k) => a + k.n, 0), c: "var(--faint)", label: t("Other") });+ for (const p of parts) {+ const seg = el("span", "mo-seg");+ seg.style.flexGrow = String(p.n);+ seg.style.background = p.c;+ seg.title = `${p.label} · ${fmtN(p.n)}`;+ bar.append(seg);+ const chip = el("button", "mo-chip");+ chip.type = "button";+ const dot = el("span", "mo-dotc");+ dot.style.background = p.c;+ chip.append(dot, el("span", "", p.label), el("b", "", fmtN(p.n)));+ if (p.name) {+ chip.title = t("Show the requests with {kind}", { kind: p.label });+ chip.onclick = () => maskOpenRecords({ kind: p.name });+ } else chip.disabled = true;+ legend.append(chip);+ }+ sec.append(bar, legend);+ box.append(sec);+ if (move) bar.animate([{ clipPath: "inset(0 100% 0 0 round 99px)" }, { clipPath: "inset(0 0 0 0 round 99px)" }], { duration: 900, delay: 120, easing: "cubic-bezier(.32, .72, 0, 1)", fill: "backwards" });+ }++ // what only the model found+ if (sum.by?.model) {+ const m = el("div", "mo-model");+ const sp = el("span", "mo-spark");+ sp.append(svg(SPARK, 15, 1.3));+ const b = el("button", "text", t("Show them"));+ b.onclick = () => maskOpenRecords({ by: "model" });+ m.append(sp, el("span", "", t(sum.by.model === 1 ? "1 value found by the local model, which no rule recognises" : "{n} values found by the local model, which no rule recognises", { n: fmtN(sum.by.model) })), el("span", "grow"), b);+ box.append(m);+ }++ // from which agents, to which providers+ if (sum.agents.length && sum.providers.length) {+ const sec = el("div", "mo-sec");+ sec.append(el("div", "label", t("From your agents, masked, to vendors")));+ sec.append(maskFlow(sum, move));+ box.append(sec);+ }++ // the values masked most, shown while pointed at+ if (sum.values.length) {+ const sec = el("div", "mo-sec");+ const lh = el("div", "mo-lh");+ lh.append(el("span", "label", t("Masked most")), el("span", "mo-hint", t("Point at one to see what it was")));+ sec.append(lh);+ const wrap = el("div", "mo-values");+ for (const v of sum.values.slice(0, 12)) {+ const c = el("button", "mo-val");+ c.type = "button";+ c.setAttribute("aria-label", maskKindName(v.kind) + " " + maskShape(v.head, v.len));+ c.append(maskDot(v.kind), el("span", "sh", maskShape(v.head, v.len)), el("span", "x", "×" + fmtN(v.n)));+ maskTipOn(c, { p: v.p, kind: v.kind, by: v.by, meta: t(v.requests === 1 ? "In 1 request, {when}" : "In {n} requests, most recently {when}", { n: fmtN(v.requests), when: ago(v.last) }) });+ c.onclick = () => maskOpenRecords({ kind: v.kind });+ wrap.append(c);+ }+ sec.append(wrap);+ box.append(sec);+ }++ const foot = el("div", "mo-foot");+ foot.append(el("span", "mo-note", t("Kept on this computer for 30 days, without the values themselves")));+ const open = el("button", "text mo-open");+ open.append(el("span", "", t("View records")), svg(CHEV_R, 11, 1.7));+ open.onclick = () => maskOpenRecords({});+ foot.append(open);+ box.append(foot);+ if (move) for (const [i, n] of [...box.children].slice(2).entries()) n.animate([{ opacity: 0, transform: "translateY(6px)" }, { opacity: 1, transform: "none" }], { duration: 520, delay: 160 + i * 70, easing: "cubic-bezier(.32, .72, 0, 1)", fill: "backwards" });+}++function maskCountUp(node, n) {+ const start = performance.now(), ms = 1100;+ const step = (now) => {+ if (!node.isConnected) return;+ const p = Math.min(1, (now - start) / ms);+ const e = p === 1 ? 1 : 1 - Math.pow(2, -10 * p);+ node.textContent = fmtN(Math.round(n * e));+ if (p < 1) requestAnimationFrame(step);+ };+ requestAnimationFrame(step);+}++// a path through the points that never overshoots them (Fritsch–Butland)+function maskCurve(xs, ys) {+ const n = xs.length;+ if (n === 1) return `M${xs[0] - 1},${ys[0]}L${xs[0] + 1},${ys[0]}`;+ const d = [], m = [];+ for (let i = 0; i < n - 1; i++) d.push((ys[i + 1] - ys[i]) / (xs[i + 1] - xs[i]));+ m[0] = d[0];+ m[n - 1] = d[n - 2];+ for (let i = 1; i < n - 1; i++) m[i] = d[i - 1] * d[i] <= 0 ? 0 : 2 / (1 / d[i - 1] + 1 / d[i]);+ let p = `M${xs[0]},${ys[0]}`;+ for (let i = 0; i < n - 1; i++) {+ const h = (xs[i + 1] - xs[i]) / 3;+ p += `C${xs[i] + h},${ys[i] + m[i] * h} ${xs[i + 1] - h},${ys[i + 1] - m[i + 1] * h} ${xs[i + 1]},${ys[i + 1]}`;+ }+ return p;+}++// maskChart: the values masked each day, an area under a line, the day+// pointed at said over it. Drawn again at its width as the window resizes.+function maskChart(box, days, move) {+ const H = 78, top = 10, base = H - 16;+ const s = sv("svg", { class: "mo-svg", height: H, "aria-hidden": "true" });+ const tip = el("div", "mo-ctip");+ tip.hidden = true;+ box.append(s, tip);+ const total = days.reduce((a, d) => a + d.n, 0);+ box.setAttribute("role", "img");+ box.setAttribute("aria-label", t("{n} values masked over {days} days", { n: fmtN(total), days: days.length }));+ let xs = [], first = move;+ const draw = () => {+ const W = box.clientWidth;+ if (!W || !days.length) return;+ s.setAttribute("viewBox", `0 0 ${W} ${H}`);+ s.setAttribute("width", W);+ s.replaceChildren();+ const max = Math.max(1, ...days.map((d) => d.n));+ const pad = 4;+ xs = days.map((_, i) => days.length < 2 ? W / 2 : pad + (i * (W - 2 * pad)) / (days.length - 1));+ const ys = days.map((d) => base - (d.n / max) * (base - top));+ const line = maskCurve(xs, ys);+ const id = "moGrad" + ++maskGrad;+ const defs = sv("defs");+ const g = sv("linearGradient", { id, x1: 0, y1: 0, x2: 0, y2: 1 });+ g.append(sv("stop", { offset: "0%", "stop-color": "var(--accent)", "stop-opacity": ".34" }), sv("stop", { offset: "100%", "stop-color": "var(--accent)", "stop-opacity": "0" }));+ defs.append(g);+ const area = sv("path", { d: `${line}L${xs.at(-1)},${base}L${xs[0]},${base}Z`, fill: `url(#${id})` });+ const stroke = sv("path", { d: line, class: "mo-line", pathLength: 1 });+ const end = sv("circle", { cx: xs.at(-1), cy: ys.at(-1), r: 3.2, class: "mo-dot" });+ const fmt = (d) => d === days.at(-1).day ? t("Today") : new Date(d + "T00:00").toLocaleDateString(intlLang(), { month: "short", day: "numeric" });+ const a0 = sv("text", { x: xs[0], y: H - 2, class: "mo-axis" });+ a0.textContent = fmt(days[0].day);+ const a1 = sv("text", { x: xs.at(-1), y: H - 2, class: "mo-axis", "text-anchor": "end" });+ a1.textContent = fmt(days.at(-1).day);+ const hover = sv("g", { class: "mo-hover", visibility: "hidden" });+ const vl = sv("line", { y1: top - 4, y2: base });+ const hd = sv("circle", { r: 3.2, class: "mo-dot" });+ hover.append(vl, hd);+ s.append(defs, sv("line", { x1: 0, x2: W, y1: base, y2: base, class: "mo-base" }), area, stroke, end, a0, a1, hover);+ if (first) {+ first = false;+ stroke.style.strokeDasharray = "1";+ stroke.animate([{ strokeDashoffset: 1 }, { strokeDashoffset: 0 }], { duration: 1200, easing: "cubic-bezier(.65, 0, .35, 1)", fill: "backwards" });+ area.animate([{ opacity: 0 }, { opacity: 1 }], { duration: 700, delay: 500, easing: "ease-out", fill: "backwards" });+ end.animate([{ opacity: 0, transform: "scale(.2)" }, { opacity: 1, transform: "none" }], { duration: 420, delay: 1050, easing: "cubic-bezier(.34, 1.56, .64, 1)", fill: "backwards" });+ end.style.transformBox = "fill-box";+ end.style.transformOrigin = "center";+ }+ s.onpointermove = (e) => {+ const x = e.clientX - s.getBoundingClientRect().left;+ let i = 0;+ for (let k = 1; k < xs.length; k++) if (Math.abs(xs[k] - x) < Math.abs(xs[i] - x)) i = k;+ const d = days[i];+ vl.setAttribute("x1", xs[i]);+ vl.setAttribute("x2", xs[i]);+ hd.setAttribute("cx", xs[i]);+ hd.setAttribute("cy", ys[i]);+ hover.setAttribute("visibility", "visible");+ tip.hidden = false;+ tip.textContent = `${fmt(d.day)} · ${t(d.n === 1 ? "1 value masked" : "{n} values masked", { n: fmtN(d.n) })} · ${t(d.requests === 1 ? "1 request" : "{n} requests", { n: fmtN(d.requests) })}`;+ const w = tip.offsetWidth;+ tip.style.left = Math.max(w / 2, Math.min(xs[i], W - w / 2)) + "px";+ };+ s.onpointerleave = () => { hover.setAttribute("visibility", "hidden"); tip.hidden = true; };+ };+ const ro = new ResizeObserver(() => { if (!box.isConnected) return ro.disconnect(); draw(); });+ ro.observe(box);+}++// maskFlow: the agents that sent what was masked, each a ribbon as wide+// as its share into the shield, and from it to the providers it went to+function maskFlow(sum, move) {+ const flow = el("div", "mo-flow");+ const side = (list, kind) => {+ const shown = list.slice(0, 4), rest = list.slice(4);+ const rows = shown.map((x) => ({ id: x.name, n: x.n, ...maskWho(x.name, kind) }));+ if (rest.length) rows.push({ id: "", n: rest.reduce((a, x) => a + x.n, 0), name: t("{n} more", { n: rest.length }) });+ const col = el("div", "mo-col " + kind);+ for (const r of rows) {+ const b = el("button", "mo-node");+ b.type = "button";+ if (r.icon) b.append(icon(r.icon));+ b.append(el("span", "nm", r.name), el("span", "ct", fmtN(r.n)));+ if (r.id) {+ b.title = t("Show what {name} sent masked", { name: r.name });+ b.onclick = () => maskOpenRecords({ [kind]: r.id });+ } else b.disabled = true;+ col.append(b);+ }+ return [col, rows];+ };+ const [from, fromRows] = side(sum.agents, "agent");+ const [to, toRows] = side(sum.providers, "provider");+ const mid = el("div", "mo-mid");+ const rowH = 34;+ const H = Math.max(fromRows.length, toRows.length) * rowH - 4;+ const s = sv("svg", { viewBox: `0 0 100 ${H}`, preserveAspectRatio: "none", "aria-hidden": "true" });+ const max = Math.max(...fromRows.map((r) => r.n), ...toRows.map((r) => r.n), 1);+ const defs = sv("defs");+ s.append(defs);+ const ys = (rows) => {+ const off = (H - (rows.length * rowH - 4)) / 2;+ return rows.map((_, i) => off + i * rowH + 15);+ };+ const c = H / 2;+ const ribs = [];+ ys(fromRows).forEach((y, i) => {+ const id = "moRib" + ++maskGrad;+ const g = sv("linearGradient", { id, gradientUnits: "userSpaceOnUse", x1: 0, y1: 0, x2: 50, y2: 0 });+ g.append(sv("stop", { offset: "0%", "stop-color": `var(--c${(i % 7) + 1})`, "stop-opacity": ".55" }), sv("stop", { offset: "100%", "stop-color": "var(--accent)", "stop-opacity": ".55" }));+ defs.append(g);+ ribs.push(sv("path", { d: `M0,${y}C28,${y} 26,${c} 50,${c}`, class: "mo-rib", stroke: `url(#${id})`, "stroke-width": 2 + (10 * fromRows[i].n) / max, "vector-effect": "non-scaling-stroke", pathLength: 1 }));+ });+ ys(toRows).forEach((y, i) => {+ ribs.push(sv("path", { d: `M50,${c}C74,${c} 72,${y} 100,${y}`, class: "mo-rib out", "stroke-width": 2 + (10 * toRows[i].n) / max, "vector-effect": "non-scaling-stroke", pathLength: 1 }));+ });+ s.append(...ribs);+ const hub = el("span", "mo-hub");+ hub.append(svg(SHIELD, 16, 1.6));+ hub.title = t("Masked here, on this computer");+ mid.append(s, hub);+ flow.style.height = H + "px";+ flow.append(from, mid, to);+ if (move) {+ ribs.forEach((r, i) => {+ r.style.strokeDasharray = "1";+ const out = r.classList.contains("out");+ r.animate([{ strokeDashoffset: 1 }, { strokeDashoffset: 0 }], { duration: 760, delay: 380 + (out ? 520 : 0) + i * 40, easing: "cubic-bezier(.65, 0, .35, 1)", fill: "backwards" });+ });+ hub.animate([{ transform: "scale(.4)", opacity: 0 }, { transform: "none", opacity: 1 }], { duration: 520, delay: 760, easing: "cubic-bezier(.34, 1.56, .64, 1)", fill: "backwards" });+ }+ return flow;+}++// ---- the records: each request masking changed, newest first ----+function maskOpenRecords(filter, focus) {+ maskTipOff();+ maskView = { filter: { kind: "", by: "", agent: "", provider: "", ...filter }, records: [], more: false, busy: true, open: new Set(), focus: focus || 0, one: null, lang: "" };+ $("#setPage-privacy").classList.add("recs");+ drawMaskRecs();+ maskFetchRecs();+ if (focus) api("masking/record?key=" + focus)+ .then((r) => { if (maskView?.focus === focus) { maskView.one = r; maskView.open.add(focus); drawMaskRecs(); } })+ .catch(() => { if (maskView?.focus === focus) { maskView.one = false; drawMaskRecs(); } });+}+function maskCloseRecords() {+ maskTipOff();+ maskView = null;+ $("#setPage-privacy").classList.remove("recs");+ const box = $("#maskRecs");+ box.hidden = true;+ box.replaceChildren();+}+function maskQuery(before) {+ const q = new URLSearchParams({ limit: "40" });+ if (before) q.set("before", before);+ for (const [k, v] of Object.entries(maskView.filter)) if (v) q.set(k, v);+ return "masking/records?" + q;+}+function maskFetchRecs(more) {+ const v = maskView;+ v.busy = true;+ v.error = "";+ if (more) drawMaskRecs();+ api(maskQuery(more ? v.records.at(-1)?.key : 0))+ .then((r) => {+ if (maskView !== v) return;+ const got = r?.records || [];+ v.records = more ? [...v.records, ...got] : got;+ v.more = !!r?.more;+ })+ .catch((e) => { if (maskView === v) v.error = e.message; })+ .finally(() => { if (maskView === v) { v.busy = false; drawMaskRecs(); } });+}+// new records come in over the ones shown, which keep their place+function maskRefreshRecs() {+ const v = maskView;+ if (!v.records.length) return maskFetchRecs();+ api(maskQuery(0)).then((r) => {+ if (maskView !== v || !v.records.length) return;+ const fresh = (r?.records || []).filter((x) => x.key > v.records[0].key);+ if (fresh.length) { v.records = [...fresh, ...v.records]; drawMaskRecs(); }+ }).catch(() => {});+}++function drawMaskRecs() {+ const v = maskView, box = $("#maskRecs");+ if (!v || !box) return;+ v.lang = document.documentElement.lang;+ maskTipOff();+ box.hidden = false;+ box.replaceChildren();+ const head = el("div", "mr-head");+ const back = el("button", "text mr-back");+ back.type = "button";+ back.append(svg("m9.5 4.5-3 3.5 3 3.5", 12, 1.7), el("span", "", t("Privacy")));+ back.onclick = maskCloseRecords;+ head.append(back, el("span", "mr-title", t("What was masked")));+ box.append(head);++ const sum = maskSum || {};+ const bar = el("div", "mr-filters");+ const pick = (key, all, title, opts) => {+ const b = el("button", "sess-pick");+ b.type = "button";+ if (v.filter[key] && !opts.some((o) => o.v === v.filter[key])) opts.unshift({ v: v.filter[key], name: v.filter[key] });+ sessPick(b, all, v.filter[key], opts, title, (val) => {+ v.filter[key] = val || "";+ v.focus = 0;+ v.one = null;+ maskFetchRecs();+ });+ if (v.filter[key]) b.firstChild.textContent = opts.find((o) => o.v === v.filter[key])?.name || v.filter[key];+ bar.append(b);+ };+ pick("kind", "Every kind", "Kind", (sum.kinds || []).map((k) => ({ v: k.name, name: maskKindName(k.name) })));+ pick("by", "Found by anything", "Found by", Object.keys(sum.by || {}).filter((b) => sum.by[b]).map((b) => ({ v: b, name: maskByName(b) })));+ pick("agent", "Every agent", "Agent", (sum.agents || []).map((a) => ({ v: a.name, name: maskWho(a.name, "agent").name })));+ pick("provider", "Every provider", "Provider", (sum.providers || []).map((p) => ({ v: p.name, name: maskWho(p.name, "provider").name })));+ box.append(bar);++ // the request a badge was clicked on, first and open+ if (v.focus) {+ const h = el("div", "row-head");+ h.append(el("span", "label", t("This request")));+ const list = el("div", "list mr-group");+ if (v.one) list.append(maskRecordRow(v.one, v));+ else list.append(el("div", "mr-none", v.one === false ? t("This request's record is gone: records are kept for 30 days") : t("Loading…")));+ box.append(h, list);+ }+ if (v.error) box.append(el("div", "mr-none err", t(v.error)));+ else if (!v.records.length) {+ if (!v.busy) box.append(el("div", "mr-none", Object.values(v.filter).some(Boolean) ? t("Nothing masked matches these filters") : t("Nothing masked yet")));+ else if (!v.focus) box.append(el("div", "mr-none", t("Loading…")));+ }+ let day = "", list = null;+ for (const r of v.records) {+ if (r.key === v.focus && v.one) continue;+ const d = new Date(r.t).toDateString();+ if (d !== day) {+ day = d;+ const h = el("div", "row-head");+ h.append(el("span", "label", maskDayName(r.t)));+ list = el("div", "list mr-group");+ box.append(h, list);+ }+ list.append(maskRecordRow(r, v));+ }+ if (v.more) {+ const more = el("button", "text mr-more", v.busy ? t("Loading…") : t("Load more"));+ more.disabled = v.busy;+ more.onclick = () => maskFetchRecs(true);+ box.append(more);+ }+}++function maskRecordRow(r, v) {+ const item = el("div", "mr-item");+ const row = el("button", "mr-row");+ row.type = "button";+ const chev = el("span", "mr-chev");+ chev.append(svg(CHEV_R, 11, 1.6));+ const when = el("span", "mr-time", new Date(r.t).toLocaleTimeString(intlLang(), { hour: "2-digit", minute: "2-digit" }));+ const route = el("span", "mr-route");+ const a = maskWho(r.agent, "agent"), p = maskWho(r.provider, "provider");+ if (r.agent) {+ if (a.icon) route.append(icon(a.icon));+ route.append(el("span", "who", a.name), el("span", "arrow", "→"));+ }+ if (p.icon && r.provider) route.append(icon(p.icon));+ route.append(el("span", "who", r.provider ? p.name : t("A vendor")));+ if (r.model) route.append(el("span", "model", "· " + r.model));+ if (r.what && r.what !== "chat") route.append(el("span", "what", MASK_WHAT[r.what] ? t(MASK_WHAT[r.what]) : r.what));+ const kinds = el("span", "mr-kinds");+ const by = {};+ for (const x of r.values || []) by[x.k] = (by[x.k] || 0) + (x.n || 1);+ const ranked = Object.entries(by).sort((x, y) => y[1] - x[1]);+ for (const [k, n] of ranked.slice(0, 3)) {+ const c = el("span", "mr-kind");+ c.append(maskDot(k), el("span", "", maskKindName(k)), el("b", "", "×" + fmtN(n)));+ kinds.append(c);+ }+ if (ranked.length > 3) kinds.append(el("span", "mr-kind more", "+" + (ranked.length - 3)));+ const n = el("span", "mr-n", fmtN(r.n));+ n.title = t(r.n === 1 ? "1 value masked" : "{n} values masked", { n: fmtN(r.n) });+ row.append(chev, when, route, kinds, n);+ item.append(row);+ const set = (open) => {+ item.classList.toggle("open", open);+ row.setAttribute("aria-expanded", String(open));+ };+ set(v.open.has(r.key));+ if (v.open.has(r.key)) item.append(maskRecordBody(r));+ row.onclick = () => {+ const open = !v.open.has(r.key);+ if (open) v.open.add(r.key); else v.open.delete(r.key);+ set(open);+ maskTipOff();+ const had = item.querySelector(".mr-body");+ if (!open) { if (had) sheet(had, false, () => had.remove()); return; }+ had?.remove();+ const body = maskRecordBody(r);+ item.append(body);+ sheet(body, true);+ };+ return item;+}++// maskRecordBody: each value the request had masked, and the text around+// it as the vendor saw it, each placeholder showing what it was while it is+// pointed at+function maskRecordBody(r) {+ const body = el("div", "mr-body");+ const vals = el("div", "mr-vals");+ for (const x of r.values || []) {+ const line = el("div", "mr-val");+ const shape = el("span", "mr-shape" + (x.known ? " known" : ""), maskShape(x.h, x.l));+ if (x.known) {+ shape.tabIndex = 0;+ maskTipOn(shape, { p: x.p, kind: x.k, by: x.b });+ }+ line.append(maskDot(x.k), el("span", "mr-k", maskKindName(x.k)), shape, el("span", "mr-by" + (x.b === "model" ? " model" : ""), maskByName(x.b)));+ if (x.n > 1) line.append(el("span", "mr-x", "×" + fmtN(x.n)));+ vals.append(line);+ }+ body.append(vals);+ // values side by side share their text: each piece of it is shown once+ const core = (x) => x.replace(/^…|…$/g, "");+ const all = [...new Set(Object.values(r.snips || {}))];+ const snips = all.filter((x, i) => !all.some((y, j) => j != i && (y.length > x.length || y.length == x.length && j < i) && y.includes(core(x)))).map((x) => [, x]);+ if (!snips.length) body.append(el("div", "mr-note", t("The text around them was kept in memory only, and magpie has restarted since")));+ const info = new Map((r.values || []).map((x) => [x.p, x]));+ for (const [, text] of snips) {+ const s = el("div", "mr-snip");+ let at = 0;+ for (const m of text.matchAll(PLACEHOLDER)) {+ s.append(document.createTextNode(text.slice(at, m.index)));+ const mark = el("mark", "mr-ph", m[0]);+ const x = info.get(m[0]);+ const kind = x?.k || m[0].slice(2, -2).replace(/_[^_]*$/, "");+ mark.style.setProperty("--k", maskColor(kind));+ mark.tabIndex = 0;+ maskTipOn(mark, { p: m[0], kind, by: x?.b || "known" });+ s.append(mark);+ at = m.index + m[0].length;+ }+ s.append(document.createTextNode(text.slice(at)));+ body.append(s);+ }+ return body;+}++// maskedBadge: what masking swapped in one request (gateway's Masked), for+// Recent calls and the Routing page; it opens that request's record+function maskedBadge(m) {+ const b = el("button", "mask-badge");+ b.type = "button";+ b.append(svg(SHIELD, 11, 1.7), el("span", "", fmtN(m.n)));+ const kinds = Object.entries(m.kinds || {}).sort((x, y) => y[1] - x[1]).map(([k, n]) => `${maskKindName(k)} ${n}`).join(", ");+ b.title = t(m.n === 1 ? "1 value masked before it went to the vendor: {kinds}" : "{n} values masked before it went to the vendor: {kinds}", { n: fmtN(m.n), kinds });+ b.setAttribute("aria-label", b.title);+ b.onclick = (e) => { e.stopPropagation(); openMaskRecord(m.key); };+ b.onkeydown = (e) => e.stopPropagation(); // Enter on it isn't the row's+ return b;+}+function openMaskRecord(key) {+ openSettings();+ if (mode !== "window") return;+ setSetTab("privacy", true);+ maskOpenRecords({}, key);+}+function renderOTel(s, keep) {const page = $("#otelList");page.replaceChildren();diff --git a/internal/gui/assets/i18n.js b/internal/gui/assets/i18n.jsindex 9018dfe49..17e3752e5 100644--- a/internal/gui/assets/i18n.js+++ b/internal/gui/assets/i18n.js@@ -3759,6 +3759,83 @@ const I18N = {"Vendor (first byte)": "供应商(首字节)","Streaming": "流式输出","Reading the reply": "读取回复",+ "Unknown": "未知",+ "Yesterday": "昨天",+ "Not in memory: magpie has restarted since. The value itself is never written to disk": "已不在内存中:magpie 之后重启过。原值从不写入磁盘",+ "Kept from vendors": "没让厂商看到的",+ "Masked on this computer before each request left it, and put back in each answer": "每个请求离开这台电脑前在本机脱敏,回复里再换回原样",+ "Masking is off: requests go to vendors as they are": "脱敏已关闭:请求原样发给厂商",+ "Turn on Mask secrets or Mask personal data below, and what is kept from vendors shows here, request by request": "打开下方的「脱敏密钥」或「脱敏个人信息」,没让厂商看到的内容会逐个请求显示在这里",+ "value masked": "处已脱敏",+ "values masked": "处已脱敏",+ "1 different value": "1 个不同的值",+ "{n} different values": "{n} 个不同的值",+ "Nothing yet in these days": "这几天还没有",+ "A request that carries a key, a password or personal data shows here as it goes": "带着密钥、密码或个人信息的请求,一发出就会出现在这里",+ "What was kept": "遮住了什么",+ "Show the requests with {kind}": "查看含{kind}的请求",+ "Show them": "查看",+ "1 value found by the local model, which no rule recognises": "本地模型找到 1 个规则认不出的值",+ "{n} values found by the local model, which no rule recognises": "本地模型找到 {n} 个规则认不出的值",+ "From your agents, masked, to vendors": "从你的智能体,经过脱敏,到厂商",+ "Masked most": "脱敏最多的",+ "Point at one to see what it was": "指向一个即可看到原值",+ "In 1 request, {when}": "出现在 1 个请求中,{when}",+ "In {n} requests, most recently {when}": "出现在 {n} 个请求中,最近一次 {when}",+ "Kept on this computer for 30 days, without the values themselves": "记录只存在这台电脑上,保留 30 天,不含原值",+ "View records": "查看记录",+ "{n} values masked over {days} days": "{days} 天内脱敏 {n} 处",+ "1 value masked": "脱敏 1 处",+ "{n} values masked": "脱敏 {n} 处",+ "Show what {name} sent masked": "查看 {name} 发出的脱敏内容",+ "Masked here, on this computer": "在这台电脑上脱敏",+ "What was masked": "脱敏记录",+ "This request": "这个请求",+ "This request's record is gone: records are kept for 30 days": "这个请求的记录已不在:记录保留 30 天",+ "Nothing masked matches these filters": "没有符合筛选条件的脱敏记录",+ "Nothing masked yet": "还没有脱敏过任何内容",+ "Load more": "加载更多",+ "A vendor": "厂商",+ "The text around them was kept in memory only, and magpie has restarted since": "周围的文字只存在内存里,而 magpie 之后重启过",+ "1 value masked before it went to the vendor: {kinds}": "发给厂商前脱敏了 1 处:{kinds}",+ "{n} values masked before it went to the vendor: {kinds}": "发给厂商前脱敏了 {n} 处:{kinds}",+ "API keys": "API 密钥",+ "Private keys": "私钥",+ "Passwords": "密码",+ "Secrets": "机密",+ "Emails": "邮箱",+ "Phone numbers": "手机号",+ "ID numbers": "身份证号",+ "Bank cards": "银行卡号",+ "Names": "姓名",+ "Addresses": "地址",+ "Account numbers": "账号",+ "SSNs": "社会安全号",+ "Passports": "护照号",+ "User names": "用户名",+ "Computer names": "电脑名",+ "IP addresses": "IP 地址",+ "Buckets": "存储桶",+ "Your words": "你的词",+ "Your rules": "你的规则",+ "Local model": "本地模型",+ "Seen before": "之前见过",+ "Every kind": "所有类型",+ "Kind": "类型",+ "Found by anything": "任意方式",+ "Found by": "查找方式",+ "Every agent": "所有智能体",+ "Every provider": "所有提供方",+ "What to mask": "要遮什么",+ "How it's found": "怎么找",+ "Sharing": "分享",+ "Always on": "始终开着",+ "Turn on Mask secrets or Mask personal data first": "请先打开「脱敏密钥」或「脱敏个人信息」",+ "Keys, tokens, emails and numbers whose checksum adds up: patterns that are certain, checked on every request": "密钥、令牌、邮箱和校验位对得上的号码:确定无误的模式,每个请求都会检查",+ "Video": "视频",+ "System One": "System One",+ "Embeddings": "向量嵌入",+ "Audio": "音频",},"zh-TW": {"None": "無",@@ -7501,6 +7578,83 @@ const I18N = {"Vendor (first byte)": "供應商(首位元組)","Streaming": "串流輸出","Reading the reply": "讀取回覆",+ "Unknown": "未知",+ "Yesterday": "昨天",+ "Not in memory: magpie has restarted since. The value itself is never written to disk": "已不在記憶體中:magpie 之後重新啟動過。原值從不寫入磁碟",+ "Kept from vendors": "沒讓廠商看到的",+ "Masked on this computer before each request left it, and put back in each answer": "每個請求離開這台電腦前在本機脫敏,回覆裡再換回原樣",+ "Masking is off: requests go to vendors as they are": "脫敏已關閉:請求原樣發給廠商",+ "Turn on Mask secrets or Mask personal data below, and what is kept from vendors shows here, request by request": "打開下方的「脫敏金鑰」或「脫敏個人資訊」,沒讓廠商看到的內容會逐個請求顯示在這裡",+ "value masked": "處已脫敏",+ "values masked": "處已脫敏",+ "1 different value": "1 個不同的值",+ "{n} different values": "{n} 個不同的值",+ "Nothing yet in these days": "這幾天還沒有",+ "A request that carries a key, a password or personal data shows here as it goes": "帶著金鑰、密碼或個人資訊的請求,一發出就會出現在這裡",+ "What was kept": "遮住了什麼",+ "Show the requests with {kind}": "查看含{kind}的請求",+ "Show them": "查看",+ "1 value found by the local model, which no rule recognises": "本機模型找到 1 個規則認不出的值",+ "{n} values found by the local model, which no rule recognises": "本機模型找到 {n} 個規則認不出的值",+ "From your agents, masked, to vendors": "從你的智慧體,經過脫敏,到廠商",+ "Masked most": "脫敏最多的",+ "Point at one to see what it was": "指向一個即可看到原值",+ "In 1 request, {when}": "出現在 1 個請求中,{when}",+ "In {n} requests, most recently {when}": "出現在 {n} 個請求中,最近一次 {when}",+ "Kept on this computer for 30 days, without the values themselves": "記錄只存在這台電腦上,保留 30 天,不含原值",+ "View records": "查看記錄",+ "{n} values masked over {days} days": "{days} 天內脫敏 {n} 處",+ "1 value masked": "脫敏 1 處",+ "{n} values masked": "脫敏 {n} 處",+ "Show what {name} sent masked": "查看 {name} 發出的脫敏內容",+ "Masked here, on this computer": "在這台電腦上脫敏",+ "What was masked": "脫敏記錄",+ "This request": "這個請求",+ "This request's record is gone: records are kept for 30 days": "這個請求的記錄已不在:記錄保留 30 天",+ "Nothing masked matches these filters": "沒有符合篩選條件的脫敏記錄",+ "Nothing masked yet": "還沒有脫敏過任何內容",+ "Load more": "載入更多",+ "A vendor": "廠商",+ "The text around them was kept in memory only, and magpie has restarted since": "周圍的文字只存在記憶體裡,而 magpie 之後重新啟動過",+ "1 value masked before it went to the vendor: {kinds}": "發給廠商前脫敏了 1 處:{kinds}",+ "{n} values masked before it went to the vendor: {kinds}": "發給廠商前脫敏了 {n} 處:{kinds}",+ "API keys": "API 金鑰",+ "Private keys": "私鑰",+ "Passwords": "密碼",+ "Secrets": "機密",+ "Emails": "電子郵件",+ "Phone numbers": "電話號碼",+ "ID numbers": "身分證號",+ "Bank cards": "銀行卡號",+ "Names": "姓名",+ "Addresses": "地址",+ "Account numbers": "帳號",+ "SSNs": "社會安全號碼",+ "Passports": "護照號碼",+ "User names": "使用者名稱",+ "Computer names": "電腦名稱",+ "IP addresses": "IP 位址",+ "Buckets": "儲存桶",+ "Your words": "你的詞",+ "Your rules": "你的規則",+ "Local model": "本機模型",+ "Seen before": "之前見過",+ "Every kind": "所有類型",+ "Kind": "類型",+ "Found by anything": "任意方式",+ "Found by": "查找方式",+ "Every agent": "所有智慧體",+ "Every provider": "所有提供方",+ "What to mask": "要遮什麼",+ "How it's found": "怎麼找",+ "Sharing": "分享",+ "Always on": "始終開著",+ "Turn on Mask secrets or Mask personal data first": "請先打開「脫敏金鑰」或「脫敏個人資訊」",+ "Keys, tokens, emails and numbers whose checksum adds up: patterns that are certain, checked on every request": "金鑰、權杖、電子郵件和檢查碼對得上的號碼:確定無誤的模式,每個請求都會檢查",+ "Video": "影片",+ "System One": "System One",+ "Embeddings": "向量嵌入",+ "Audio": "音訊",},ja: {"None": "なし",@@ -11245,6 +11399,83 @@ const I18N = {"Vendor (first byte)": "ベンダー(最初のバイト)","Streaming": "ストリーミング","Reading the reply": "応答の読み取り",+ "Unknown": "不明",+ "Yesterday": "昨日",+ "Not in memory: magpie has restarted since. The value itself is never written to disk": "メモリにありません:その後 magpie が再起動しました。元の値はディスクに書き込まれません",+ "Kept from vendors": "ベンダーに渡さなかったもの",+ "Masked on this computer before each request left it, and put back in each answer": "各リクエストがこのコンピュータを出る前にここでマスクし、応答で元に戻します",+ "Masking is off: requests go to vendors as they are": "マスクはオフです:リクエストはそのままベンダーに送られます",+ "Turn on Mask secrets or Mask personal data below, and what is kept from vendors shows here, request by request": "下の「シークレットをマスク」か「個人情報をマスク」をオンにすると、ベンダーに渡さなかったものがリクエストごとにここに表示されます",+ "value masked": "か所をマスク",+ "values masked": "か所をマスク",+ "1 different value": "異なる値 1 個",+ "{n} different values": "異なる値 {n} 個",+ "Nothing yet in these days": "この期間はまだありません",+ "A request that carries a key, a password or personal data shows here as it goes": "キー、パスワード、個人情報を含むリクエストは送信と同時にここに表示されます",+ "What was kept": "隠したもの",+ "Show the requests with {kind}": "{kind}を含むリクエストを表示",+ "Show them": "表示",+ "1 value found by the local model, which no rule recognises": "ルールでは見つからない値をローカルモデルが 1 個検出",+ "{n} values found by the local model, which no rule recognises": "ルールでは見つからない値をローカルモデルが {n} 個検出",+ "From your agents, masked, to vendors": "エージェントから、マスクを経て、ベンダーへ",+ "Masked most": "マスクの多い値",+ "Point at one to see what it was": "ポイントすると元の値が見えます",+ "In 1 request, {when}": "1 件のリクエスト、{when}",+ "In {n} requests, most recently {when}": "{n} 件のリクエスト、最後は{when}",+ "Kept on this computer for 30 days, without the values themselves": "記録はこのコンピュータに 30 日間保存され、元の値は含みません",+ "View records": "記録を見る",+ "{n} values masked over {days} days": "{days} 日間で {n} か所をマスク",+ "1 value masked": "1 か所をマスク",+ "{n} values masked": "{n} か所をマスク",+ "Show what {name} sent masked": "{name} が送ったマスク済みの内容を表示",+ "Masked here, on this computer": "このコンピュータでマスク",+ "What was masked": "マスクの記録",+ "This request": "このリクエスト",+ "This request's record is gone: records are kept for 30 days": "このリクエストの記録はありません:記録は 30 日間保存されます",+ "Nothing masked matches these filters": "この条件に合うマスクの記録はありません",+ "Nothing masked yet": "まだ何もマスクしていません",+ "Load more": "さらに読み込む",+ "A vendor": "ベンダー",+ "The text around them was kept in memory only, and magpie has restarted since": "前後のテキストはメモリにのみ保存され、その後 magpie が再起動しました",+ "1 value masked before it went to the vendor: {kinds}": "ベンダーに送る前に 1 か所をマスク:{kinds}",+ "{n} values masked before it went to the vendor: {kinds}": "ベンダーに送る前に {n} か所をマスク:{kinds}",+ "API keys": "API キー",+ "Private keys": "秘密鍵",+ "Passwords": "パスワード",+ "Secrets": "シークレット",+ "Emails": "メール",+ "Phone numbers": "電話番号",+ "ID numbers": "身分証番号",+ "Bank cards": "銀行カード",+ "Names": "氏名",+ "Addresses": "住所",+ "Account numbers": "口座番号",+ "SSNs": "社会保障番号",+ "Passports": "パスポート番号",+ "User names": "ユーザー名",+ "Computer names": "コンピュータ名",+ "IP addresses": "IP アドレス",+ "Buckets": "バケット",+ "Your words": "あなたの語",+ "Your rules": "あなたのルール",+ "Local model": "ローカルモデル",+ "Seen before": "既出",+ "Every kind": "すべての種類",+ "Kind": "種類",+ "Found by anything": "すべての検出方法",+ "Found by": "検出方法",+ "Every agent": "すべてのエージェント",+ "Every provider": "すべてのプロバイダ",+ "What to mask": "マスクするもの",+ "How it's found": "検出方法",+ "Sharing": "共有",+ "Always on": "常にオン",+ "Turn on Mask secrets or Mask personal data first": "先に「シークレットをマスク」か「個人情報をマスク」をオンにしてください",+ "Keys, tokens, emails and numbers whose checksum adds up: patterns that are certain, checked on every request": "キー、トークン、メール、チェックサムが合う番号:確実なパターンを毎回のリクエストで確認",+ "Video": "動画",+ "System One": "System One",+ "Embeddings": "埋め込み",+ "Audio": "音声",},de: {"None": "Keine",@@ -14989,6 +15220,83 @@ const I18N = {"Vendor (first byte)": "Anbieter (erstes Byte)","Streaming": "Streaming","Reading the reply": "Antwort lesen",+ "Unknown": "Unbekannt",+ "Yesterday": "Gestern",+ "Not in memory: magpie has restarted since. The value itself is never written to disk": "Nicht mehr im Speicher: magpie wurde seitdem neu gestartet. Der Wert selbst wird nie auf die Festplatte geschrieben",+ "Kept from vendors": "Vor Anbietern verborgen",+ "Masked on this computer before each request left it, and put back in each answer": "Auf diesem Computer maskiert, bevor jede Anfrage ihn verlässt, und in jeder Antwort zurückgesetzt",+ "Masking is off: requests go to vendors as they are": "Maskierung ist aus: Anfragen gehen unverändert an die Anbieter",+ "Turn on Mask secrets or Mask personal data below, and what is kept from vendors shows here, request by request": "Schalte unten „Geheimnisse maskieren“ oder „Personenbezogene Daten maskieren“ ein, dann zeigt sich hier pro Anfrage, was vor Anbietern verborgen blieb",+ "value masked": "Wert maskiert",+ "values masked": "Werte maskiert",+ "1 different value": "1 verschiedener Wert",+ "{n} different values": "{n} verschiedene Werte",+ "Nothing yet in these days": "In diesen Tagen noch nichts",+ "A request that carries a key, a password or personal data shows here as it goes": "Eine Anfrage mit einem Schlüssel, Passwort oder personenbezogenen Daten erscheint hier, sobald sie rausgeht",+ "What was kept": "Was verborgen wurde",+ "Show the requests with {kind}": "Anfragen mit {kind} zeigen",+ "Show them": "Anzeigen",+ "1 value found by the local model, which no rule recognises": "1 Wert vom lokalen Modell gefunden, den keine Regel erkennt",+ "{n} values found by the local model, which no rule recognises": "{n} Werte vom lokalen Modell gefunden, die keine Regel erkennt",+ "From your agents, masked, to vendors": "Von deinen Agenten, maskiert, zu den Anbietern",+ "Masked most": "Am häufigsten maskiert",+ "Point at one to see what it was": "Zeig auf einen, um ihn zu sehen",+ "In 1 request, {when}": "In 1 Anfrage, {when}",+ "In {n} requests, most recently {when}": "In {n} Anfragen, zuletzt {when}",+ "Kept on this computer for 30 days, without the values themselves": "30 Tage auf diesem Computer gespeichert, ohne die Werte selbst",+ "View records": "Einträge ansehen",+ "{n} values masked over {days} days": "{n} Werte in {days} Tagen maskiert",+ "1 value masked": "1 Wert maskiert",+ "{n} values masked": "{n} Werte maskiert",+ "Show what {name} sent masked": "Zeigen, was {name} maskiert gesendet hat",+ "Masked here, on this computer": "Hier auf diesem Computer maskiert",+ "What was masked": "Was maskiert wurde",+ "This request": "Diese Anfrage",+ "This request's record is gone: records are kept for 30 days": "Der Eintrag dieser Anfrage ist weg: Einträge bleiben 30 Tage",+ "Nothing masked matches these filters": "Nichts Maskiertes passt zu diesen Filtern",+ "Nothing masked yet": "Noch nichts maskiert",+ "Load more": "Mehr laden",+ "A vendor": "Ein Anbieter",+ "The text around them was kept in memory only, and magpie has restarted since": "Der Text um sie herum lag nur im Speicher, und magpie wurde seitdem neu gestartet",+ "1 value masked before it went to the vendor: {kinds}": "1 Wert maskiert, bevor es an den Anbieter ging: {kinds}",+ "{n} values masked before it went to the vendor: {kinds}": "{n} Werte maskiert, bevor es an den Anbieter ging: {kinds}",+ "API keys": "API-Schlüssel",+ "Private keys": "Private Schlüssel",+ "Passwords": "Passwörter",+ "Secrets": "Geheimnisse",+ "Emails": "E-Mails",+ "Phone numbers": "Telefonnummern",+ "ID numbers": "Ausweisnummern",+ "Bank cards": "Bankkarten",+ "Names": "Namen",+ "Addresses": "Adressen",+ "Account numbers": "Kontonummern",+ "SSNs": "SSNs",+ "Passports": "Reisepässe",+ "User names": "Benutzernamen",+ "Computer names": "Computernamen",+ "IP addresses": "IP-Adressen",+ "Buckets": "Buckets",+ "Your words": "Deine Wörter",+ "Your rules": "Deine Regeln",+ "Local model": "Lokales Modell",+ "Seen before": "Schon gesehen",+ "Every kind": "Jede Art",+ "Kind": "Art",+ "Found by anything": "Beliebig gefunden",+ "Found by": "Gefunden von",+ "Every agent": "Jeder Agent",+ "Every provider": "Jeder Anbieter",+ "What to mask": "Was maskiert wird",+ "How it's found": "Wie es gefunden wird",+ "Sharing": "Teilen",+ "Always on": "Immer an",+ "Turn on Mask secrets or Mask personal data first": "Schalte zuerst „Geheimnisse maskieren“ oder „Personenbezogene Daten maskieren“ ein",+ "Keys, tokens, emails and numbers whose checksum adds up: patterns that are certain, checked on every request": "Schlüssel, Tokens, E-Mails und Nummern mit stimmiger Prüfsumme: sichere Muster, bei jeder Anfrage geprüft",+ "Video": "Video",+ "System One": "System One",+ "Embeddings": "Embeddings",+ "Audio": "Audio",},};diff --git a/internal/gui/assets/index.html b/internal/gui/assets/index.htmlindex 46541fc24..72738f7cd 100644--- a/internal/gui/assets/index.html+++ b/internal/gui/assets/index.html@@ -374,7 +374,14 @@<div class="list prefs" id="disabledModelsList"></div></section><section class="set-page" id="setPage-privacy" role="tabpanel" aria-labelledby="setTab-privacy">+ <div class="mask-obs" id="maskObs"></div>+ <div class="mask-recs" id="maskRecs" hidden></div>+ <div class="row-head"><span class="label" data-t>What to mask</span></div><div class="list prefs" id="redactList"></div>+ <div class="row-head"><span class="label" data-t>How it's found</span></div>+ <div class="list prefs" id="findList"></div>+ <div class="row-head"><span class="label" data-t>Sharing</span></div>+ <div class="list prefs" id="shareList"></div><div class="row-head"><span class="label" data-t>Agents' sign-ins</span></div><div class="list prefs" id="discoveryList"></div></section>diff --git a/internal/gui/assets/plugins/gateway/client.js b/internal/gui/assets/plugins/gateway/client.jsindex 986a0ed4f..41d26cfa9 100644--- a/internal/gui/assets/plugins/gateway/client.js+++ b/internal/gui/assets/plugins/gateway/client.js@@ -592,6 +592,8 @@ function renderActivity() {r.append(el("span", "a", c.agent || "—"));r.append(el("span", "m", c.model));r.append(el("span", "p", c.from === c.to ? c.from : `${c.from} → ${c.to}`));+ // what masking kept from the vendor, its record a click away+ if (c.masked?.n) r.append(maskedBadge(c.masked));r.append(el("span", "grow"));r.append(el("span", "st", c.error ? `${c.status} ${c.error}` : `${c.status} · ${ledTook(c.ms)}` + (c.ttft ? " · " + t("TTFT {ms}", { ms: ledTook(c.ttft) }) : "")));r.title = open ? t("Hide request and response bodies") : t("Show request and response bodies");diff --git a/internal/gui/assets/plugins/routing/client.js b/internal/gui/assets/plugins/routing/client.jsindex f71b9764f..82a7b68d4 100644--- a/internal/gui/assets/plugins/routing/client.js+++ b/internal/gui/assets/plugins/routing/client.js@@ -1890,13 +1890,15 @@ function routingPage(part) {log.hidden = !r;if (!r) return;const on = () => listed().filter((x) => x.id >= logR.id);- const head = JSON.stringify([document.documentElement.lang, !!rp, pinned?.id, r.id, r.time, r.done, !!pinned && on().length > 1, bodiesFor === r.id]);+ const head = JSON.stringify([document.documentElement.lang, !!rp, pinned?.id, r.id, r.time, r.done, !!pinned && on().length > 1, bodiesFor === r.id, r.masked?.key, r.masked?.n]);if (headKey !== head) {headKey = head;logHead.replaceChildren(el("span", "", rp ? t("How the request at {time} was routed", { time: clock(r.time) })- : pinned ? t("How the request at {time} was routed", { time: clock(r.time) }) : t("How the last request was routed")),- el("span", "grow"));+ : pinned ? t("How the request at {time} was routed", { time: clock(r.time) }) : t("How the last request was routed")));+ // what masking kept from the vendor, its record a click away+ if (r.masked?.n && !rp) logHead.append(maskedBadge(r.masked));+ logHead.append(el("span", "grow"));if (!rp && r.done) {const usage = el("button", "text", t("View usage"));usage.dataset.feature = "usage"; // it opens on the Usage pagediff --git a/internal/gui/assets/plugins/routing/routing.css b/internal/gui/assets/plugins/routing/routing.cssindex d107f2721..c234d4aae 100644--- a/internal/gui/assets/plugins/routing/routing.css+++ b/internal/gui/assets/plugins/routing/routing.css@@ -864,3 +864,4 @@ button.rt-session { cursor: pointer; }.rt-gtpl-card.off b { color: var(--muted); }.rt-gtpl-card .acts { display: flex; align-items: center; gap: 8px; margin-top: auto; padding-top: 4px; }.rt-gtplbtn { display: inline-flex; align-items: center; gap: 4px; }+.rt-log-head .mask-badge { margin-left: 8px; }diff --git a/internal/gui/assets/plugins/usage/i18n.js b/internal/gui/assets/plugins/usage/i18n.jsindex a13bfd86f..ec5e5a4b9 100644--- a/internal/gui/assets/plugins/usage/i18n.js+++ b/internal/gui/assets/plugins/usage/i18n.js@@ -18,7 +18,6 @@ magpie.strings({"With cache": "含缓存","Cache read and written counted too": "同时计入缓存读取与写入","Token totals": "token 汇总",- "Yesterday": "昨天","This month": "本月","Last month": "上个月","Pick days": "选择日期",@@ -209,7 +208,6 @@ magpie.strings({"With cache": "含快取","Cache read and written counted too": "一併計入快取讀取與寫入","Token totals": "token 總計",- "Yesterday": "昨天","This month": "本月","Last month": "上個月","Pick days": "選擇日期",@@ -400,7 +398,6 @@ magpie.strings({"With cache": "キャッシュ込み","Cache read and written counted too": "キャッシュの読み取りと書き込みも含める","Token totals": "トークンの合計",- "Yesterday": "昨日","This month": "今月","Last month": "先月","Pick days": "日付を選ぶ",@@ -591,7 +588,6 @@ magpie.strings({"With cache": "Mit Cache","Cache read and written counted too": "Cache-Lesen und -Schreiben mitgezählt","Token totals": "Token-Summen",- "Yesterday": "Gestern","This month": "Dieser Monat","Last month": "Letzter Monat","Pick days": "Tage wählen",diff --git a/internal/gui/core_budget_test.go b/internal/gui/core_budget_test.goindex afe1b9594..307abed66 100644--- a/internal/gui/core_budget_test.go+++ b/internal/gui/core_budget_test.go@@ -15,8 +15,8 @@ import (// so a page can't quietly grow back into the core; a change that needs more// raises them and says why in its commit.const (- appScriptBudget = 1_292_000 // app.js: 1,198,239 after Usage left (P2); 1,218,098 with the 10-15 fixes to pages still in the core; 1,232,466 with the provider editor's Advanced fold and every dialog holding its place, the core's own (10-16); 1,250,533 with the quota refresh kept through redraws, the core's own since the tray panel draws it too, plus 1.5% (#2397); 1,269,083 with Free first's option in the provider editor's ROUTINGS, which the Routing page reads too, plus 1.5%; 1,272,726 with the Providers page drawing its last list at once (#1519), the core's page, and the Routing trace's account names read without Object.entries (#1481), plus 1.5%; a dialog growing out of the row that opened it and folding back into it, the core's own since every page's dialogs open through it (10-20); Codex's and Claude Code's own settings moved from Settings into the agent's row on the Agents page, the core's own since the tray panel and the window both draw them (10-20)- coreScriptBudget = 1_495_000 // every core script: 1,358,034 after Usage left; 1,380,003 with them; 1,405,937 with the fold; 1,426,102 with 10-16's core work (the GitHub list loading, the move to plugins, Settings › Long conversations), plus 1.5% (#2612); 1,447,368 with loadBack, the window's one state read per return, which every page needs and so is the core's own (1,446,845 before it, 155 bytes under), plus 1.5%; OpenCode Zen's card saying OpenCode tells no key its balance adds ~390 bytes inside it; 1,472,589 with the dialog that grows out of its row and the agents' own settings in their rows, on main's 10-20 core work (the Providers page's last list, the Routing trace's names), plus 1.5% (10-20)+ appScriptBudget = 1_335_000 // app.js: 1,198,239 after Usage left (P2); 1,218,098 with the 10-15 fixes to pages still in the core; 1,232,466 with the provider editor's Advanced fold and every dialog holding its place, the core's own (10-16); 1,250,533 with the quota refresh kept through redraws, the core's own since the tray panel draws it too, plus 1.5% (#2397); 1,269,083 with Free first's option in the provider editor's ROUTINGS, which the Routing page reads too, plus 1.5%; 1,272,726 with the Providers page drawing its last list at once (#1519), the core's page, and the Routing trace's account names read without Object.entries (#1481), plus 1.5%; a dialog growing out of the row that opened it and folding back into it, the core's own since every page's dialogs open through it (10-20); Codex's and Claude Code's own settings moved from Settings into the agent's row on the Agents page, the core's own since the tray panel and the window both draw them (10-20); 1,314,432 with Settings › Privacy's record of what was masked (its summary, the records view and the local model's controls), which belongs to Settings, the core's own, plus 1.5%+ coreScriptBudget = 1_528_800 // every core script: 1,358,034 after Usage left; 1,380,003 with them; 1,405,937 with the fold; 1,426,102 with 10-16's core work (the GitHub list loading, the move to plugins, Settings › Long conversations), plus 1.5% (#2612); 1,447,368 with loadBack, the window's one state read per return, which every page needs and so is the core's own (1,446,845 before it, 155 bytes under), plus 1.5%; OpenCode Zen's card saying OpenCode tells no key its balance adds ~390 bytes inside it; 1,472,589 with the dialog that grows out of its row and the agents' own settings in their rows, on main's 10-20 core work (the Providers page's last list, the Routing trace's names), plus 1.5% (10-20); 1,506,192 with Settings › Privacy's record of what was masked, plus 1.5%)func TestCoreScriptBudget(t *testing.T) {diff --git a/internal/gui/masking.go b/internal/gui/masking.gonew file mode 100644index 000000000..943cc58f4--- /dev/null+++ b/internal/gui/masking.go@@ -0,0 +1,96 @@+package gui++import (+ "net/http"+ "strconv"++ "github.com/yetone/magpie/internal/gwmw"+ "github.com/yetone/magpie/internal/maskrec"+)++// maskedRecord is a record as the page shows it: each value says whether+// the masker still knows what it stood for, so the page offers to show it+// only where it can.+type maskedRecord struct {+ maskrec.Record+ Values []maskedValue `json:"values,omitempty"`+}++type maskedValue struct {+ maskrec.Value+ Known bool `json:"known,omitempty"`+}++// teller is the served masker, where it can say what a placeholder stood+// for.+func teller() (gwmw.Teller, bool) {+ m, ok := gwmw.Served()+ if !ok {+ return nil, false+ }+ t, ok := m.(gwmw.Teller)+ return t, ok+}++func shownRecord(r maskrec.Record, t gwmw.Teller) maskedRecord {+ out := maskedRecord{Record: r}+ for _, v := range r.Values {+ mv := maskedValue{Value: v}+ if t != nil {+ _, mv.Known = t.Value(v.Placeholder)+ }+ out.Values = append(out.Values, mv)+ }+ return out+}++// maskingRoutes are Settings → Privacy's record of what masking swapped:+// - GET /api/masking/summary?days= is maskrec.Sum;+// - GET /api/masking/records?before=&limit=&kind=&by=&agent=&provider=+// a page of records, newest first;+// - GET /api/masking/record?key= one record;+// - GET /api/masking/value?p= what a placeholder stood for, while this+// magpie's masker remembers it (404 after a restart: it is never on+// disk).+func maskingRoutes(mux *http.ServeMux) {+ mux.HandleFunc("GET /api/masking/summary", func(rw http.ResponseWriter, r *http.Request) {+ days, _ := strconv.Atoi(r.URL.Query().Get("days"))+ writeJSON(rw, maskrec.Sum(min(max(days, 1), 30)))+ })+ mux.HandleFunc("GET /api/masking/records", func(rw http.ResponseWriter, r *http.Request) {+ q := r.URL.Query()+ before, _ := strconv.ParseInt(q.Get("before"), 10, 64)+ limit, _ := strconv.Atoi(q.Get("limit"))+ rs, more := maskrec.List(before, min(max(limit, 1), 200), maskrec.Filter{Kind: q.Get("kind"), By: q.Get("by"), Agent: q.Get("agent"), Provider: q.Get("provider")})+ t, _ := teller()+ out := []maskedRecord{}+ for _, rec := range rs {+ out = append(out, shownRecord(rec, t))+ }+ writeJSON(rw, map[string]any{"records": out, "more": more})+ })+ mux.HandleFunc("GET /api/masking/record", func(rw http.ResponseWriter, r *http.Request) {+ key, _ := strconv.ParseInt(r.URL.Query().Get("key"), 10, 64)+ rec, ok := maskrec.Get(key)+ if !ok {+ http.NotFound(rw, r)+ return+ }+ t, _ := teller()+ writeJSON(rw, shownRecord(rec, t))+ })+ mux.HandleFunc("GET /api/masking/value", func(rw http.ResponseWriter, r *http.Request) {+ rw.Header().Set("Cache-Control", "no-store")+ t, ok := teller()+ if !ok {+ http.NotFound(rw, r)+ return+ }+ v, ok := t.Value(r.URL.Query().Get("p"))+ if !ok {+ http.NotFound(rw, r)+ return+ }+ writeJSON(rw, map[string]string{"value": v})+ })+}diff --git a/internal/gui/masking_test.go b/internal/gui/masking_test.gonew file mode 100644index 000000000..94e7d71ed--- /dev/null+++ b/internal/gui/masking_test.go@@ -0,0 +1,90 @@+package gui++import (+ "encoding/json"+ "net/http"+ "net/http/httptest"+ "net/url"+ "path/filepath"+ "strconv"+ "strings"+ "testing"++ "github.com/yetone/magpie/internal/gwmw"+ "github.com/yetone/magpie/internal/masking"+ "github.com/yetone/magpie/internal/maskrec"+ "github.com/yetone/magpie/internal/redact"+ "github.com/yetone/magpie/internal/settings"+)++// Settings → Privacy reads what masking swapped: the summary, the records+// newest first, one record, and a value only while the masker remembers+// it; a placeholder it never made says nothing.+func TestMaskingRecordsAPI(t *testing.T) {+ const key = "sk-proj-abcdEFGH1234ijklMNOP5678qrst"+ h := t.TempDir()+ t.Setenv("HOME", h)+ t.Setenv("USERPROFILE", h)+ t.Setenv("XDG_CONFIG_HOME", filepath.Join(h, ".config"))+ t.Setenv("XDG_CACHE_HOME", filepath.Join(h, ".cache"))+ if err := settings.Save(settings.Settings{Redact: true}); err != nil {+ t.Fatal(err)+ }+ defer gwmw.Mask(masking.Masker{})()+ m, _ := gwmw.Served()+ sent, hits := m.(gwmw.Teller).TextHits("my key is " + key)+ if len(hits) != 1 || strings.Contains(sent, key) {+ t.Fatalf("masked %q %+v", sent, hits)+ }+ p := hits[0].Placeholder+ rec := maskrec.Add(maskrec.Record{What: "chat", Agent: "claude", Provider: "anthropic", N: 1,+ Values: []maskrec.Value{{Placeholder: p, Kind: hits[0].Kind, By: hits[0].By, Head: hits[0].Head, Len: hits[0].Len}},+ Snips: map[string]string{p: sent}})+ other := maskrec.Add(maskrec.Record{What: "search", N: 1, Values: []maskrec.Value{{Placeholder: "{{EMAIL_00000000}}", Kind: "EMAIL", By: redact.ByRule}}})++ srv := Handler(webHost{func() {}}, nil)+ get := func(path string, into any) int {+ rw := httptest.NewRecorder()+ srv.ServeHTTP(rw, httptest.NewRequest("GET", path, nil))+ if into != nil && rw.Code == http.StatusOK {+ if err := json.Unmarshal(rw.Body.Bytes(), into); err != nil {+ t.Fatalf("%s: %v %s", path, err, rw.Body)+ }+ }+ return rw.Code+ }+ var sum maskrec.Summary+ if get("/api/masking/summary?days=7", &sum); sum.Requests < 2 || len(sum.Days) != 7 {+ t.Fatalf("summary %+v", sum)+ }+ var page struct {+ Records []maskedRecord `json:"records"`+ More bool `json:"more"`+ }+ page.Records, page.More = nil, false+ if get("/api/masking/records?limit=1", &page); len(page.Records) != 1 || !page.More || page.Records[0].Key != other.Key {+ t.Fatalf("first page %+v", page)+ }+ page.Records, page.More = nil, false+ if get("/api/masking/records?limit=5&before="+strconv.FormatInt(other.Key, 10), &page); len(page.Records) == 0 || page.Records[0].Key != rec.Key || !page.Records[0].Values[0].Known {+ t.Fatalf("next page %+v", page)+ }+ page.Records, page.More = nil, false+ if get("/api/masking/records?kind=EMAIL", &page); len(page.Records) == 0 || page.Records[0].Key != other.Key || page.Records[0].Values[0].Known {+ t.Fatalf("by kind %+v", page)+ }+ var one maskedRecord+ if get("/api/masking/record?key="+strconv.FormatInt(rec.Key, 10), &one); one.Snips[p] != sent || !one.Values[0].Known {+ t.Fatalf("record %+v", one)+ }+ if code := get("/api/masking/record?key=1", nil); code != http.StatusNotFound {+ t.Fatalf("no such record: %d", code)+ }+ var v struct{ Value string }+ if get("/api/masking/value?p="+url.QueryEscape(p), &v); v.Value != key {+ t.Fatalf("value %q", v.Value)+ }+ if code := get("/api/masking/value?p="+url.QueryEscape("{{EMAIL_00000000}}"), nil); code != http.StatusNotFound {+ t.Fatalf("a placeholder it never made: %d", code)+ }+}diff --git a/internal/gui/tests/account-discovery.test.cjs b/internal/gui/tests/account-discovery.test.cjsindex deb49c6b1..1fc111c9e 100644--- a/internal/gui/tests/account-discovery.test.cjs+++ b/internal/gui/tests/account-discovery.test.cjs@@ -113,7 +113,13 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",assert.equal(await opts.nth(1).evaluate((b) => b.classList.contains("on")), true, "On by default");const row = box.locator(".row");- await row.scrollIntoViewIfNeeded();+ // scrolled to with the wheel, as a reader does: Playwright's own+ // scroll into view is code, which the click guard puts back+ await page.mouse.move(width / 2, 300);+ for (let i = 0; i < 20 && await row.evaluate((r) => r.getBoundingClientRect().bottom > (document.querySelector("footer.foot")?.getBoundingClientRect().top ?? innerHeight) - 4); i++) {+ await page.mouse.wheel(0, 200);+ await page.waitForTimeout(80);+ }await page.waitForTimeout(300);const fit = await row.evaluate((r) => {const b = r.getBoundingClientRect(), segs = r.querySelector(".val").getBoundingClientRect();diff --git a/internal/gui/tests/masking-records.test.cjs b/internal/gui/tests/masking-records.test.cjsnew file mode 100644index 000000000..0550919b6--- /dev/null+++ b/internal/gui/tests/masking-records.test.cjs@@ -0,0 +1,209 @@+// Run with Node's test runner and Playwright on the module path; see README.md.+// Settings › Privacy opens on what masking kept from vendors: the number,+// the days, the kinds, agents to providers and the values masked most, a+// value shown only while pointed at (and said gone when this magpie no+// longer knows it). Its records open from the card, filtered by what was+// clicked, each one opening on the text the vendor saw. A masked call on+// the Gateway page opens its own record. Nothing in the card or the+// records is left in English, nothing is wider than the window, and a+// click scrolls nothing.+// Every language, Chromium and WebKit, 1100 and 640px; the API is faked.+const assert = require("node:assert/strict");+const fs = require("node:fs/promises");+const path = require("node:path");+const { test } = require("node:test");+const { chromium, webkit } = require("playwright");+const { watchSettingsReads, settingsDrawn } = require("./reader.cjs");++const assets = path.resolve(__dirname, "../assets");+const KEY = "{{API_KEY_aaaa1111}}", NAME = "{{PERSON_bbbb2222}}", MAIL = "{{EMAIL_cccc3333}}";+const SECRET = "sk-proj-abcdEFGH1234ijkl";++function day(back) {+ const d = new Date();+ d.setDate(d.getDate() - back);+ return `${d.getFullYear()}-${String(d.getMonth() + 1).padStart(2, "0")}-${String(d.getDate()).padStart(2, "0")}`;+}+const ago = (ms) => new Date(Date.now() - ms).toISOString();++function summary(days) {+ const per = [0, 3, 0, 7, 5, 12, 15];+ return {+ days: Array.from({ length: days }, (_, i) => { const n = per[(i + 7 - days % 7) % 7]; return { day: day(days - 1 - i), requests: n ? 2 : 0, n, distinct: n ? 1 : 0 }; }),+ requests: 9, n: 42, distinct: 11, hours: [],+ kinds: [{ name: "API_KEY", n: 20, distinct: 3 }, { name: "EMAIL", n: 12, distinct: 4 }, { name: "PERSON", n: 10, distinct: 4 }],+ by: { rule: 7, model: 4 },+ agents: [{ name: "claude", n: 30, distinct: 8 }, { name: "codex", n: 12, distinct: 3 }],+ providers: [{ name: "anthropic", n: 30, distinct: 8 }, { name: "openai", n: 12, distinct: 3 }],+ values: [+ { p: KEY, kind: "API_KEY", by: "rule", head: "sk-p", len: 24, n: 14, requests: 5, last: ago(60e3) },+ { p: NAME, kind: "PERSON", by: "model", head: "", len: 3, n: 6, requests: 3, last: ago(3600e3) },+ ],+ since: ago(7 * 86400e3),+ };+}+const records = [+ { key: 1002, t: ago(120e3), route: "messages", what: "chat", agent: "claude", provider: "anthropic", model: "claude-opus-4-1", n: 3,+ values: [{ p: KEY, k: "API_KEY", b: "rule", h: "sk-p", l: 24, n: 2, known: true }, { p: MAIL, k: "EMAIL", b: "rule", h: "a", l: 17, n: 1 }],+ // two values in one line: their text is shown once+ snips: { [KEY]: `export OPENAI_API_KEY=${KEY} # mail ${MAIL}`, [MAIL]: `…_KEY=${KEY} # mail ${MAIL}` } },+ { key: 1001, t: ago(300e3), route: "search", what: "search", agent: "codex", provider: "openai", model: "gpt-5", n: 1,+ values: [{ p: NAME, k: "PERSON", b: "model", h: "", l: 3, n: 1 }] },+];++function serve(lang, sum, asked) {+ const settings = { theme: "light", lang, tray: "panel", currency: "usd", redact: true, redactPersonal: true };+ return async (route) => {+ const req = route.request(), url = new URL(req.url());+ const json = (data) => route.fulfill({ json: data });+ if (url.pathname === "/boot.js") return route.fulfill({ contentType: "text/javascript", body: `window.bootPrefs = {lang:"${lang}",theme:"light",web:true};` });+ if (url.pathname === "/wails/runtime.js") return route.fulfill({ contentType: "text/javascript", body: "export const Window = {};" });+ if (url.pathname === "/api/state") return json({ agents: [{ id: "claude", name: "Claude Code", icon: "claude" }, { id: "codex", name: "Codex", icon: "codex" }], profiles: [], settings });+ if (url.pathname === "/api/settings") return json(settings);+ if (url.pathname === "/api/providers") return json({ providers: [{ id: "anthropic", name: "Anthropic", icon: "anthropic", agents: [], models: [] }, { id: "openai", name: "OpenAI", icon: "openai", agents: [], models: [] }], presets: [], excluded: [],+ gateway: { running: true, window: true, mine: true, url: "http://127.0.0.1:3999", models: 0, groups: [],+ calls: [{ id: 7, time: ago(120e3), agent: "claude", model: "claude-opus-4-1", from: "anthropic", to: "anthropic", status: 200, ms: 900, masked: { n: 3, distinct: 2, kinds: { API_KEY: 2, EMAIL: 1 }, by: { rule: 2 }, key: 1002 } }] } });+ if (url.pathname === "/api/groups") return json({ groups: [], models: [] });+ if (url.pathname === "/api/plugins") return json({ plugins: [] });+ if (url.pathname === "/api/privacy-model") return json({ supported: true, installed: false, size: 1.6e9 });+ if (url.pathname === "/api/masking/summary") return json(sum(+url.searchParams.get("days")));+ if (url.pathname === "/api/masking/records") {+ asked.push(url.search);+ const q = url.searchParams;+ const rs = records.filter((r) => (!q.get("kind") || r.values.some((v) => v.k === q.get("kind"))) && (!q.get("by") || r.values.some((v) => v.b === q.get("by"))));+ return json({ records: rs, more: false });+ }+ if (url.pathname === "/api/masking/record") {+ const r = records.find((r) => r.key === +url.searchParams.get("key"));+ return r ? json(r) : route.fulfill({ status: 404, body: "404 page not found" });+ }+ if (url.pathname === "/api/masking/value") {+ asked.push("value " + url.searchParams.get("p"));+ return url.searchParams.get("p") === KEY ? json({ value: SECRET }) : route.fulfill({ status: 404, body: "404 page not found" });+ }+ if (url.pathname.startsWith("/api/")) return json({});+ const file = path.join(assets, url.pathname === "/" ? "index.html" : url.pathname);+ const contentType = { ".html": "text/html", ".js": "text/javascript", ".css": "text/css", ".svg": "image/svg+xml", ".png": "image/png" }[path.extname(file)];+ try { await route.fulfill({ body: await fs.readFile(file), contentType }); } catch { await route.fulfill({ status: 404, body: "" }); }+ };+}++async function open(engine, lang, width, sum, where) {+ const browser = await (engine === "webkit" ? webkit.launch() : chromium.launch({ channel: "chromium" }));+ const page = await (await browser.newContext({ viewport: { width, height: 900 }, reducedMotion: "reduce" })).newPage();+ page.setDefaultTimeout(8000);+ const errors = [], asked = [];+ page.on("pageerror", (e) => errors.push(e.message));+ await page.route("**/*", serve(lang, sum, asked));+ await watchSettingsReads(page);+ await page.goto("http://magpie.test/" + (where || "?view=settings&tab=privacy"));+ if (!where) await settingsDrawn(page);+ return { browser, page, errors, asked };+}++// English that a translated page must not show+const english = ["Kept from vendors", "values masked", "What was kept", "Masked most", "View records", "What was masked", "Every kind", "Found by", "Local model", "Not in memory", "This request"];+const wide = (page) => page.evaluate(() => {+ const over = [...document.querySelectorAll("#maskObs *, #maskRecs *")].filter((n) => n.getClientRects().length && n.getBoundingClientRect().right > innerWidth + 1).map((n) => n.className?.baseVal ?? n.className);+ return { page: document.documentElement.scrollWidth > document.documentElement.clientWidth, over };+});+const scrolled = (page) => page.evaluate(() => [scrollX, scrollY, document.querySelector("#view-settings").scrollTop]);++for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium", "webkit"])) {+ for (const lang of ["en", "zh", "zh-TW", "ja", "de"]) {+ for (const width of [1100, 640]) {+ test(`${engine} ${lang} ${width}px: the card says what was kept, and its records open on what the vendor saw`, async (t) => {+ const { browser, page, errors, asked } = await open(engine, lang, width, summary);+ t.after(() => browser.close());+ const card = page.locator("#maskObs");+ await card.locator(".mo-num").waitFor();+ assert.equal(await card.locator(".mo-num").textContent(), "42");+ assert.equal(await card.locator(".mo-svg path.mo-line").count(), 1, "the days' line");+ assert.equal(await card.locator(".mo-bar .mo-seg").count(), 3);+ assert.equal(await card.locator(".mo-legend .mo-chip:enabled").count(), 3);+ assert.equal(await card.locator(".mo-model").count(), 1, "what only the model found");+ assert.deepEqual(await card.locator(".mo-col.agent .nm").allTextContents(), ["Claude Code", "Codex"]);+ assert.deepEqual(await card.locator(".mo-col.provider .nm").allTextContents(), ["Anthropic", "OpenAI"]);+ assert.deepEqual(await card.locator(".mo-val .sh").allTextContents(), ["sk-p••••••••", "•••"]);+ assert.equal((await card.textContent()).includes(SECRET), false, "no value until pointed at");+ assert.equal(asked.filter((a) => a.startsWith("value")).length, 0, "nothing asked until pointed at");+ if (lang !== "en") {+ const said = await page.locator("#maskObs").innerText();+ for (const w of english) assert(!said.includes(w), `${w} left in English: ${said}`);+ }+ assert.deepEqual(await wide(page), { page: false, over: [] });++ // pointed at, a value says what it was; one this magpie no longer+ // knows says so+ const before = await scrolled(page);+ await card.locator(".mo-val").first().hover();+ await page.waitForFunction((s) => document.querySelector(".mask-tip .v")?.textContent === s, SECRET);+ await card.locator(".mo-val").nth(1).hover();+ await page.waitForFunction(() => document.querySelector(".mask-tip .v.gone"));+ await page.mouse.move(1, 1);+ await page.locator(".mask-tip").waitFor({ state: "detached" });++ // a kind's chip opens its records, filtered by it+ await card.locator(".mo-chip").nth(1).click();+ const recs = page.locator("#maskRecs");+ await recs.locator(".mr-item").first().waitFor();+ assert.match(asked.filter((a) => a.startsWith("?")).at(-1), /kind=EMAIL/);+ assert.equal(await page.locator("#redactList").isVisible(), false, "the records stand in for the page");+ assert.equal(await recs.locator(".mr-item").count(), 1);+ if (lang !== "en") {+ const said = await recs.innerText();+ for (const w of english) assert(!said.includes(w), `${w} left in English: ${said}`);+ }+ // opened: each value, and the text the vendor saw with its+ // placeholders marked+ await recs.locator(".mr-row").first().click();+ await recs.locator(".mr-body .mr-snip").first().waitFor();+ assert.equal(await recs.locator(".mr-val").count(), 2);+ assert.equal(await recs.locator(".mr-body .mr-snip").count(), 1, "a shared line once");+ assert.deepEqual(await recs.locator(".mr-snip mark.mr-ph").allTextContents(), [KEY, MAIL]);+ assert.equal(await recs.locator(".mr-shape.known").count(), 1, "only a value it still knows offers to show");+ await recs.locator(".mr-snip mark.mr-ph").first().hover();+ await page.waitForFunction((s) => document.querySelector(".mask-tip .v")?.textContent === s, SECRET);+ assert.deepEqual(await wide(page), { page: false, over: [] });+ assert.deepEqual(await scrolled(page), before, "no click moved the page");++ // back to Privacy, as it was+ await recs.locator(".mr-back").click();+ await page.locator("#redactList").waitFor();+ assert.equal(await recs.isHidden(), true);+ assert.deepEqual(errors, []);+ });+ }+ }++ test(`${engine}: a summary of nothing, an empty one and records filtered to nothing each say so`, async (t) => {+ let shape = {};+ const { browser, page, errors } = await open(engine, "en", 1100, () => shape);+ t.after(() => browser.close());+ // an old magpie's {}: the card waits on requests, without failing+ await page.locator("#maskObs .mo-empty").waitFor();+ assert.match(await page.locator("#maskObs").innerText(), /A request that carries a key/);+ await page.locator("#maskObs .mo-open").count().then((n) => assert.equal(n, 0));+ shape = { ...summary(7), requests: 0, n: 0, distinct: 0, kinds: [], agents: [], providers: [], values: [], by: {} };+ // the records with a filter matching nothing+ await page.evaluate(() => maskOpenRecords({ by: "yours" }));+ await page.locator("#maskRecs .mr-none", { hasText: "Nothing masked matches these filters" }).waitFor();+ assert.deepEqual(errors, []);+ });++ test(`${engine}: a masked call on the Gateway page opens its record`, async (t) => {+ const { browser, page, errors } = await open(engine, "en", 1100, summary, "?view=gateway");+ t.after(() => browser.close());+ const badge = page.locator("#activity .call .mask-badge");+ await badge.waitFor();+ assert.equal(await badge.textContent(), "3");+ assert.match(await badge.getAttribute("title"), /^3 values masked before it went to the vendor: API keys 2, Emails 1$/);+ await badge.click();+ const recs = page.locator("#maskRecs");+ await recs.locator(".row-head .label", { hasText: "This request" }).waitFor();+ // first, and open on what the vendor saw+ await recs.locator(".mr-item.open .mr-snip").first().waitFor();+ assert.equal(await page.locator("#activity .call-item.open").count(), 0, "the click wasn't the row's");+ assert.deepEqual(errors, []);+ });+}diff --git a/internal/gui/tests/privacy-hides-accounts.test.cjs b/internal/gui/tests/privacy-hides-accounts.test.cjsindex 361566aef..53fcd29f2 100644--- a/internal/gui/tests/privacy-hides-accounts.test.cjs+++ b/internal/gui/tests/privacy-hides-accounts.test.cjs@@ -94,7 +94,7 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",// personal data on turns it onconst settings = { ...base(), redactPersonal: false }, saved = [];page = await open(settings, null, "http://magpie.test/?view=settings&tab=privacy", saved);- const row = (name) => page.locator("#redactList .row", { has: page.locator(".name", { hasText: new RegExp("^" + name + "$") }) });+ const row = (name) => page.locator("#setPage-privacy .list .row", { has: page.locator(".name", { hasText: new RegExp("^" + name + "$") }) });await row(w.hide).waitFor();const top = () => page.evaluate(() => document.querySelector("#view-settings").scrollTop);const y0 = await top();@@ -103,7 +103,7 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",await page.waitForFunction(() => document.querySelector("#usageMask")?.getAttribute("aria-pressed") === "true");assert.ok(saved.some((b) => b.redactPersonal === true), JSON.stringify(saved));// the switch here says so, and turns it off again- await page.waitForFunction((on) => [...document.querySelectorAll("#redactList .row")].some((r) => r.querySelector(".name")?.textContent === on.hide && r.querySelector(".opt.on")?.textContent === on.on), w);+ await page.waitForFunction((on) => [...document.querySelectorAll("#setPage-privacy .list .row")].some((r) => r.querySelector(".name")?.textContent === on.hide && r.querySelector(".opt.on")?.textContent === on.on), w);assert.equal(await top(), y0, "a click never scrolls");// since Settings' search box (51042630e) the row sits under the// window's foot at this size, and the reader scrolls to it with thediff --git a/internal/gui/tests/privacy-model.test.cjs b/internal/gui/tests/privacy-model.test.cjsindex 389e56547..d80d4e645 100644--- a/internal/gui/tests/privacy-model.test.cjs+++ b/internal/gui/tests/privacy-model.test.cjs@@ -75,7 +75,7 @@ async function open(engine, lang, width, masking, model, posts) {// what the row shows: its words, whether they are cut, and that it is// inside the listconst look = (page) => page.evaluate(() => {- const r = document.querySelector("#privacyModelRow"), list = document.querySelector("#redactList").getBoundingClientRect();+ const r = document.querySelector("#privacyModelRow"), list = r.closest(".list").getBoundingClientRect();const sub = r.querySelector(".sub"), b = r.getBoundingClientRect(), v = r.querySelector(".val").getBoundingClientRect();return {name: r.querySelector(".name").textContent, sub: sub.textContent,@@ -149,11 +149,18 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",assert.deepEqual(errors, []);});- test(`${engine}: no model row while nothing is masked, and none to turn on where it can't run`, async (t) => {+ test(`${engine}: while nothing is masked the model row waits under Rules, and none to turn on where it can't run`, async (t) => {const { browser, page, errors } = await open(engine, "en", 1100, false, { supported: true, size }, []);t.after(() => browser.close());await page.locator("#redactList .row", { hasText: "Mask personal data" }).waitFor();- assert.equal(await page.locator("#privacyModelRow").count(), 0);+ // How it's found: the rules first, then the model, both waiting on+ // something to mask, with nothing to press+ const found = await page.$$eval("#findList > .row", (rs) => rs.map((r) => [r.id, r.querySelector(".name").textContent, r.querySelector(".sub").textContent, r.querySelectorAll("button").length]));+ assert.deepEqual(found, [+ ["maskRulesRow", "Rules", "Turn on Mask secrets or Mask personal data first", 0],+ ["privacyModelRow", "Find with a local model", "Turn on Mask secrets or Mask personal data first", 0],+ ]);+ assert.equal(await page.locator("#privacyModelRow.pm-wait").count(), 1);await browser.close();const two = await open(engine, "en", 1100, true, { supported: false, size }, []);diff --git a/internal/gui/tests/privacy-usage-stats.test.cjs b/internal/gui/tests/privacy-usage-stats.test.cjsindex 1d9a69fdb..a90d0add1 100644--- a/internal/gui/tests/privacy-usage-stats.test.cjs+++ b/internal/gui/tests/privacy-usage-stats.test.cjs@@ -55,15 +55,24 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",await watchSettingsReads(page);await page.goto("http://magpie.test/?view=settings&tab=privacy");await settingsDrawn(page); // rows found are the ones that stay (#1659)- const row = (name) => page.locator("#redactList .row", { has: page.locator(".name", { hasText: new RegExp("^" + name + "$") }) });- const isOn = (name) => page.waitForFunction(([name, on]) => [...document.querySelectorAll("#redactList .row")].some((r) => r.querySelector(".name")?.textContent === name && r.querySelector(".opt.on")?.textContent === on), [name, w.on]);- const isOff = (name) => page.waitForFunction(([name, off]) => [...document.querySelectorAll("#redactList .row")].some((r) => r.querySelector(".name")?.textContent === name && r.querySelector(".opt.on")?.textContent === off), [name, w.off]);+ const row = (name) => page.locator("#setPage-privacy .list .row", { has: page.locator(".name", { hasText: new RegExp("^" + name + "$") }) });+ const isOn = (name) => page.waitForFunction(([name, on]) => [...document.querySelectorAll("#setPage-privacy .list .row")].some((r) => r.querySelector(".name")?.textContent === name && r.querySelector(".opt.on")?.textContent === on), [name, w.on]);+ const isOff = (name) => page.waitForFunction(([name, off]) => [...document.querySelectorAll("#setPage-privacy .list .row")].some((r) => r.querySelector(".name")?.textContent === name && r.querySelector(".opt.on")?.textContent === off), [name, w.off]);// on by default, right below Count me as a userawait isOn(w.share);- const names = await page.$$eval("#redactList .row .name", (ns) => ns.map((n) => n.textContent));+ const names = await page.$$eval("#setPage-privacy .list .row .name", (ns) => ns.map((n) => n.textContent));assert.equal(names.indexOf(w.share), names.indexOf(w.count) + 1, names.join(" | "));+ // below the card, scrolled to with the wheel as a reader does+ const reveal = async (name) => {+ await page.mouse.move(450, 300);+ for (let i = 0; i < 20 && (await row(name).boundingBox()).y > 600; i++) {+ await page.mouse.wheel(0, 200);+ await page.waitForTimeout(80);+ }+ };+ await reveal(w.share);await row(w.share).getByRole("button", { name: w.off, exact: true }).click();await isOff(w.share);assert.equal(saved.at(-1).noUsageStats, true, JSON.stringify(saved));@@ -77,11 +86,7 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",// not counted at all: nothing to share, no switch. With personal data// masked, its kinds' rows put this one below the window: scrolled to// as a reader does, since a script doesn't scroll Settings in WebKit- await page.mouse.move(450, 300);- for (let i = 0; i < 20 && (await row(w.count).boundingBox()).y > 600; i++) {- await page.mouse.wheel(0, 200);- await page.waitForTimeout(80);- }+ await reveal(w.count);await row(w.count).getByRole("button", { name: w.off, exact: true }).click();await isOff(w.count);await row(w.share).waitFor({ state: "detached" });diff --git a/internal/gui/tests/redact-rule-edit.test.cjs b/internal/gui/tests/redact-rule-edit.test.cjsindex c7536a81c..4be444453 100644--- a/internal/gui/tests/redact-rule-edit.test.cjs+++ b/internal/gui/tests/redact-rule-edit.test.cjs@@ -79,7 +79,15 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",const rule = (name) => page.locator("#redactList .row").filter({ has: page.locator(".name", { hasText: new RegExp(`^${name}$`) }) });const scroll = () => page.evaluate(() => [scrollX, scrollY, ...[...document.querySelectorAll("*")].filter((e) => e.scrollTop).map((e) => e.scrollTop)]);const click = async (loc) => {- await loc.scrollIntoViewIfNeeded();+ // the app undoes a script's scroll, so scroll as a reader does+ // until the target clears the foot+ await page.mouse.move(200, 300);+ for (let i = 0; i < 20; i++) {+ const [b, top] = await Promise.all([loc.boundingBox(), page.evaluate(() => document.querySelector("footer.foot")?.getBoundingClientRect().top ?? innerHeight)]);+ if (b.y >= 0 && b.y + b.height <= top - 4) break;+ await page.mouse.wheel(0, b.y < 0 ? -200 : 200);+ await page.waitForTimeout(80);+ }const before = await scroll();const b = await loc.boundingBox();await page.mouse.click(b.x + b.width / 2, b.y + b.height / 2);diff --git a/internal/gui/tests/routing-bodies.test.cjs b/internal/gui/tests/routing-bodies.test.cjsindex a8b87a464..55016f102 100644--- a/internal/gui/tests/routing-bodies.test.cjs+++ b/internal/gui/tests/routing-bodies.test.cjs@@ -4,7 +4,8 @@// and what came back). Its story's head has "Request and response": the// gateway's recent calls' copy by the route's id, read when asked; once// they have let it go, the request archive's copy when it was on, and-// otherwise how to keep them. Another request picked closes it. Both+// otherwise how to keep them. A masked request's head has its count of+// masked values. Another request picked closes it. Both// engines, a narrow window, every language; nothing runs out of the story,// no left-border accent. The API is faked.const assert = require("node:assert/strict");@@ -23,7 +24,8 @@ const done = (id, i, extra) => ({ id, seq: id, time: at(i), agent: "codex", kindtries: [{ id: key.id, model: "deepseek-chat", start: at(i), done: true, status: 200, ms: 900 }],done: true, status: 200, ms: 900, tokens: 300, served: "deepseek-chat", ...extra });// 101 still in the recent calls, 100 only in the archive, 99 in neither-const routes = [done(101, 0), done(100, 1, { archive: day + "/abc100" }), done(99, 2)];+// and 101 had three values masked: its story's head says so+const routes = [done(101, 0, { masked: { n: 3, distinct: 2, kinds: { API_KEY: 2, EMAIL: 1 }, by: { rule: 2 }, key: 5005 } }), done(100, 1, { archive: day + "/abc100" }), done(99, 2)];let seenCalls = [];function serve(lang) {@@ -127,6 +129,8 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",const b = page.locator(".rt-log-head .rt-bodies-b");await b.waitFor();assert.equal(await b.textContent(), w.open);+ assert.equal(await page.locator(".rt-log-head .mask-badge").textContent(), "3", "the masked count in the head");+ assert.match(await page.locator(".rt-log-head .mask-badge").getAttribute("aria-label"), /3/);assert.equal(await b.getAttribute("aria-expanded"), "false");assert.equal(await page.locator(".rt-bodies").isVisible(), false);assert.deepEqual(seenCalls, [], "bodies read before they were asked for");@@ -153,6 +157,7 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",await page.locator(".rt-req").nth(1).and(page.locator('[aria-pressed="true"]')).waitFor();await page.locator(".rt-bodies").waitFor({ state: "hidden" });assert.equal(await page.locator(".rt-log-head .rt-bodies-b").textContent(), w.open);+ assert.equal(await page.locator(".rt-log-head .mask-badge").count(), 0, "nothing masked, no badge");// gone from the recent calls, kept by the archive: read back from itawait tap(page.locator(".rt-log-head .rt-bodies-b"));diff --git a/internal/gwmw/gwmw.go b/internal/gwmw/gwmw.goindex 9342455a0..94d775437 100644--- a/internal/gwmw/gwmw.go+++ b/internal/gwmw/gwmw.go@@ -87,6 +87,27 @@ type Masker interface {Writer(w http.ResponseWriter) (http.ResponseWriter, func())}+// Hit is one value a masker swapped, told without the value: its kind,+// the finder that found it (rule, yours, word, model or known), its+// placeholder, and as much of it as may be kept where it may not (Head,+// its first character or a key's prefix, and Len, how many it has).+type Hit struct {+ Kind, By, Placeholder, Head string+ Len int+}++// Teller is a Masker that says what it masked too, for the record of+// what was masked; one that doesn't is counted only.+type Teller interface {+ // JSONHits is JSON, with each value it swapped, once for each place+ JSONHits(body []byte) ([]byte, []Hit)+ // TextHits is Text, with each value it swapped+ TextHits(s string) (string, []Hit)+ // Value is the value behind a placeholder, while the masker remembers+ // it+ Value(placeholder string) (string, bool)+}+var (mu sync.Mutexagents []*Agentdiff --git a/internal/masking/masking.go b/internal/masking/masking.goindex 6368b9f0e..1cacf52b1 100644--- a/internal/masking/masking.go+++ b/internal/masking/masking.go@@ -7,6 +7,7 @@ package maskingimport ("net/http"+ "github.com/yetone/magpie/internal/gwmw""github.com/yetone/magpie/internal/pii""github.com/yetone/magpie/internal/redact""github.com/yetone/magpie/internal/settings"@@ -50,9 +51,44 @@ func (Masker) JSON(body []byte) ([]byte, int) {func (Masker) Text(s string) (string, int) { return redact.Mask(s, options()) }+// JSONHits is JSON, with what it masked (gwmw.Teller).+func (Masker) JSONHits(body []byte) ([]byte, []gwmw.Hit) {+ o := options()+ if !o.Secrets && !o.Personal && len(o.Words) == 0 {+ return body, nil+ }+ var hits []gwmw.Hit+ o.Seen = told(&hits)+ out, _ := redact.MaskJSON(body, o)+ return out, hits+}++// TextHits is Text, with what it masked.+func (Masker) TextHits(s string) (string, []gwmw.Hit) {+ o := options()+ var hits []gwmw.Hit+ o.Seen = told(&hits)+ out, _ := redact.Mask(s, o)+ return out, hits+}++func (Masker) Value(p string) (string, bool) { return redact.Value(p) }++// told adds each value masked to hits, with its hint: the value itself+// stays in redact's memory.+func told(hits *[]gwmw.Hit) func(redact.Hit) {+ return func(h redact.Hit) {+ v, _ := redact.Value(h.Placeholder)+ head, n := redact.Hint(h.Kind, v)+ *hits = append(*hits, gwmw.Hit{Kind: h.Kind, By: h.By, Placeholder: h.Placeholder, Head: head, Len: n})+ }+}+func (Masker) Restore(s string, escaped bool) string { return redact.Restore(s, escaped) }func (Masker) Writer(w http.ResponseWriter) (http.ResponseWriter, func()) {rw := redact.NewWriter(w)return rw, rw.Finish}++var _ gwmw.Teller = Masker{}diff --git a/internal/maskrec/maskrec.go b/internal/maskrec/maskrec.gonew file mode 100644index 000000000..b79193cb6--- /dev/null+++ b/internal/maskrec/maskrec.go@@ -0,0 +1,334 @@+// Package maskrec keeps the record of what masking swapped for+// placeholders before a request went to a vendor, for Settings → Privacy+// to show: which kinds, found by which finder, going to whom.+//+// A value itself is never written down. A record holds each value's+// placeholder, its kind, its finder, and a hint (its first character or a+// key's prefix, and its length). The value behind a placeholder is known+// only to the masker's memory while magpie runs (gwmw.Teller.Value), and+// the text around it as the vendor saw it is kept in memory only, for the+// latest records.+//+// A day is a file of its own, one request a line, under+// ~/.config/magpie/masking, kept keepDays days and keepBytes in all.+package maskrec++import (+ "bufio"+ "bytes"+ "encoding/json"+ "os"+ "path/filepath"+ "sort"+ "strings"+ "sync"+ "time"++ "github.com/yetone/magpie/internal/appdir"+)++const (+ keepDays = 30+ keepBytes = 100 << 20+ // keepSnips is how many of the latest records memory keeps, with the+ // text around their values+ keepSnips = 400+ dayForm = "2006-01-02"+)++// Value is one value a request had masked, as often as it had it.+type Value struct {+ Placeholder string `json:"p"`+ Kind string `json:"k"`+ By string `json:"b"` // the finder: rule, yours, word, model or known+ Head string `json:"h,omitempty"` // its first character, or a key's prefix+ Len int `json:"l"` // its length in characters+ N int `json:"n"` // the places the request had it+}++// Record is one request's masking.+type Record struct {+ Key int64 `json:"key"` // unique, later records larger+ Time time.Time `json:"t"`+ Route int64 `json:"route,omitempty"` // its route in the trace, where it has one+ What string `json:"what"` // chat, search, image, video, embeddings, audio, count+ Agent string `json:"agent,omitempty"`+ Session string `json:"session,omitempty"`+ Provider string `json:"provider,omitempty"`+ Model string `json:"model,omitempty"`+ N int `json:"n"` // values swapped, each place counted+ Values []Value `json:"values,omitempty"`+ // Snips: by placeholder, the text around its first place as the+ // vendor saw it; in memory only+ Snips map[string]string `json:"snips,omitempty"`+}++// Dir is where the days are kept.+var Dir = func() string { return filepath.Join(appdir.Config(), "masking") }++// clock is what day it is; a variable for the tests.+var clock = time.Now++var store struct {+ mu sync.Mutex+ last int64+ pruned time.Time+ days map[string]*dayAgg // a day's file → what it holds, as of its size+ snips []Record // the latest records, with their snips+}++// inTest says Add writes nothing: in a test binary testenv isolated, a+// write still going as a test ends would put masking/ back into its+// temporary HOME (see the gateway's keepsRoutes).+func inTest() bool { return os.Getenv("MAGPIE_TEST_SANDBOX") != "" && !ownTest }++// ownTest is set by this package's tests, which read what they write.+var ownTest = false++// Add records r, its Values made from hits as they came: one for each+// placeholder, counted. It returns r as recorded. Errors are swallowed:+// keeping the record must never break a request.+func Add(r Record) Record {+ if len(r.Values) == 0 && r.N == 0 {+ return r+ }+ r.Values = merged(r.Values)+ if r.Time.IsZero() {+ r.Time = clock()+ }+ store.mu.Lock()+ defer store.mu.Unlock()+ r.Key = r.Time.UnixNano()+ if r.Key <= store.last {+ r.Key = store.last + 1+ }+ store.last = r.Key+ store.snips = append(store.snips, r)+ if len(store.snips) > keepSnips {+ store.snips = append([]Record(nil), store.snips[len(store.snips)-keepSnips:]...)+ }+ if inTest() {+ return r+ }+ disk := r+ disk.Snips = nil+ b, err := json.Marshal(disk)+ if err != nil {+ return r+ }+ dir := Dir()+ if os.MkdirAll(dir, 0o700) != nil {+ return r+ }+ path := filepath.Join(dir, r.Time.Local().Format(dayForm)+".jsonl")+ f, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o600)+ if err != nil {+ return r+ }+ before, _ := f.Seek(0, 2)+ _, err = f.Write(append(b, '\n'))+ f.Close()+ // a day already read takes the record as it is written, rather than+ // being read again+ if a := store.days[path]; err == nil && a != nil && a.size == before {+ a.add(disk)+ a.size += int64(len(b) + 1)+ }+ if now := clock(); now.Sub(store.pruned) > time.Hour {+ store.pruned = now+ prune(dir, now)+ }+ return r+}++// merged is vs with one Value for each placeholder, its N the sum, in the+// order they first came. A value found by a finder and later only known+// (masked before) keeps the finder that found it.+func merged(vs []Value) []Value {+ at := map[string]int{}+ var out []Value+ for _, v := range vs {+ if v.N == 0 {+ v.N = 1+ }+ if i, ok := at[v.Placeholder]; ok {+ out[i].N += v.N+ if out[i].By == "known" && v.By != "known" {+ out[i].By = v.By+ }+ continue+ }+ at[v.Placeholder] = len(out)+ out = append(out, v)+ }+ return out+}++type dayFile struct{ day, path string }++// files are the days kept, oldest first.+func files(dir string) []dayFile {+ es, _ := os.ReadDir(dir)+ var out []dayFile+ for _, e := range es {+ n := e.Name()+ if day, ok := strings.CutSuffix(n, ".jsonl"); ok {+ if _, err := time.Parse(dayForm, day); err == nil {+ out = append(out, dayFile{day, filepath.Join(dir, n)})+ }+ }+ }+ sort.Slice(out, func(i, j int) bool { return out[i].day < out[j].day })+ return out+}++// prune drops the days older than keepDays, then the oldest until what is+// left fits keepBytes.+func prune(dir string, now time.Time) {+ oldest := now.Local().AddDate(0, 0, -keepDays).Format(dayForm)+ fs := files(dir)+ var total int64+ sizes := make([]int64, len(fs))+ for i, d := range fs {+ if fi, err := os.Stat(d.path); err == nil {+ sizes[i] = fi.Size()+ total += sizes[i]+ }+ }+ for i, d := range fs {+ if d.day >= oldest && total <= keepBytes {+ break+ }+ if os.Remove(d.path) == nil {+ total -= sizes[i]+ delete(store.days, d.path)+ }+ }+}++// read calls each with the records of path, in the order written; a line+// that doesn't parse, one half-written by another magpie, is skipped.+func read(path string, each func(Record)) int64 {+ f, err := os.Open(path)+ if err != nil {+ return 0+ }+ defer f.Close()+ var size int64+ sc := bufio.NewScanner(f)+ sc.Buffer(make([]byte, 64<<10), 16<<20)+ for sc.Scan() {+ line := sc.Bytes()+ size += int64(len(line) + 1)+ if len(bytes.TrimSpace(line)) == 0 {+ continue+ }+ var r Record+ if json.Unmarshal(line, &r) == nil {+ each(r)+ }+ }+ return size+}++// withSnips is r with the snips memory has for it.+func withSnips(r Record) Record {+ i := sort.Search(len(store.snips), func(i int) bool { return store.snips[i].Key >= r.Key })+ if i < len(store.snips) && store.snips[i].Key == r.Key {+ r.Snips = store.snips[i].Snips+ }+ return r+}++// Filter narrows List: by kind, by finder, by agent or provider; "" is+// any.+type Filter struct {+ Kind, By, Agent, Provider string+}++func (f Filter) takes(r Record) bool {+ if f.Agent != "" && r.Agent != f.Agent || f.Provider != "" && r.Provider != f.Provider {+ return false+ }+ if f.Kind == "" && f.By == "" {+ return true+ }+ for _, v := range r.Values {+ if (f.Kind == "" || v.Kind == f.Kind) && (f.By == "" || v.By == f.By) {+ return true+ }+ }+ return false+}++// List is up to limit records before key (0: the latest), newest first,+// that f takes, and whether there are more.+func List(before int64, limit int, f Filter) ([]Record, bool) {+ store.mu.Lock()+ defer store.mu.Unlock()+ if limit <= 0 {+ limit = 50+ }+ var out []Record+ fs := files(Dir())+ for i := len(fs) - 1; i >= 0; i-- {+ if before > 0 && time.Unix(0, before).Local().Format(dayForm) < fs[i].day {+ continue+ }+ var day []Record+ read(fs[i].path, func(r Record) {+ if (before == 0 || r.Key < before) && f.takes(r) {+ day = append(day, r)+ }+ })+ for j := len(day) - 1; j >= 0; j-- {+ if len(out) == limit {+ return out, true+ }+ out = append(out, withSnips(day[j]))+ }+ }+ if inTest() {+ // nothing on disk: the records in memory+ for j := len(store.snips) - 1; j >= 0; j-- {+ r := store.snips[j]+ if (before == 0 || r.Key < before) && f.takes(r) {+ if len(out) == limit {+ return out, true+ }+ out = append(out, r)+ }+ }+ }+ return out, false+}++// Get is the record of key, with its snips while memory has them.+func Get(key int64) (Record, bool) {+ store.mu.Lock()+ defer store.mu.Unlock()+ if i := sort.Search(len(store.snips), func(i int) bool { return store.snips[i].Key >= key }); i < len(store.snips) && store.snips[i].Key == key {+ return store.snips[i], true+ }+ day := time.Unix(0, key).Local().Format(dayForm)+ var got Record+ ok := false+ read(filepath.Join(Dir(), day+".jsonl"), func(r Record) {+ if r.Key == key {+ got, ok = r, true+ }+ })+ return got, ok+}++// OfRoute is the record of the trace's route id, while memory has it.+func OfRoute(id int64) (Record, bool) {+ store.mu.Lock()+ defer store.mu.Unlock()+ for j := len(store.snips) - 1; j >= 0; j-- {+ if store.snips[j].Route == id {+ return store.snips[j], true+ }+ }+ return Record{}, false+}diff --git a/internal/maskrec/maskrec_test.go b/internal/maskrec/maskrec_test.gonew file mode 100644index 000000000..752e2f4f2--- /dev/null+++ b/internal/maskrec/maskrec_test.go@@ -0,0 +1,121 @@+package maskrec++import (+ "os"+ "path/filepath"+ "strings"+ "testing"+ "time"+)++func sandbox(t *testing.T, now time.Time) string {+ t.Helper()+ dir := t.TempDir()+ oldDir, oldClock, oldOwn := Dir, clock, ownTest+ Dir, clock, ownTest = func() string { return dir }, func() time.Time { return now }, true+ store.mu.Lock()+ store.days, store.snips, store.last, store.pruned = nil, nil, 0, time.Time{}+ store.mu.Unlock()+ t.Cleanup(func() { Dir, clock, ownTest = oldDir, oldClock, oldOwn })+ return dir+}++// What is written to disk is each value's placeholder, kind, finder and+// hint: never the value, and never the text around it, which memory keeps.+func TestNoValueOnDisk(t *testing.T) {+ now := time.Date(2026, 10, 20, 15, 0, 0, 0, time.Local)+ dir := sandbox(t, now)+ const name, around = "张伟", "请联系{{PERSON_aaaabbbb}}老师确认合同"+ r := Add(Record{Time: now, Route: 7, What: "chat", Agent: "claude", Provider: "anthropic", Model: "sonnet", N: 3,+ Values: []Value{+ {Placeholder: "{{PERSON_aaaabbbb}}", Kind: "PERSON", By: "model", Head: "张", Len: 2},+ {Placeholder: "{{PERSON_aaaabbbb}}", Kind: "PERSON", By: "known", Head: "张", Len: 2},+ {Placeholder: "{{API_KEY_ccccdddd}}", Kind: "API_KEY", By: "rule", Head: "sk-", Len: 40},+ },+ Snips: map[string]string{"{{PERSON_aaaabbbb}}": around}})+ if len(r.Values) != 2 || r.Values[0].N != 2 || r.Values[0].By != "model" {+ t.Fatalf("values not merged by placeholder: %+v", r.Values)+ }+ b, err := os.ReadFile(filepath.Join(dir, "2026-10-20.jsonl"))+ if err != nil {+ t.Fatal(err)+ }+ for _, secret := range []string{name, "老师", "合同"} {+ if strings.Contains(string(b), secret) {+ t.Fatalf("disk has %q: %s", secret, b)+ }+ }+ if got, ok := Get(r.Key); !ok || got.Snips["{{PERSON_aaaabbbb}}"] != around {+ t.Fatalf("memory lost the snip: %+v", got)+ }+ if got, ok := OfRoute(7); !ok || got.Key != r.Key {+ t.Fatalf("OfRoute: %+v %v", got, ok)+ }+ // after a restart: the record, without its snips+ store.mu.Lock()+ store.snips = nil+ store.mu.Unlock()+ got, ok := Get(r.Key)+ if !ok || got.Snips != nil || len(got.Values) != 2 || got.Values[1].Head != "sk-" {+ t.Fatalf("after restart: %+v %v", got, ok)+ }+}++// The summary counts requests, places and different values, by day, kind,+// finder, agent and provider; a value found by the model and only known+// after is the model's.+func TestSum(t *testing.T) {+ now := time.Date(2026, 10, 20, 15, 0, 0, 0, time.Local)+ sandbox(t, now)+ p, k := "{{PERSON_aaaabbbb}}", "{{API_KEY_ccccdddd}}"+ Add(Record{Time: now.AddDate(0, 0, -2), Agent: "codex", Provider: "openai", N: 1, Values: []Value{{Placeholder: p, Kind: "PERSON", By: "model", Len: 2}}})+ _ = Sum(7) // read, then kept up as Add writes+ Add(Record{Time: now.Add(-time.Hour), Agent: "claude", Provider: "anthropic", N: 3, Values: []Value{{Placeholder: p, Kind: "PERSON", By: "known", Len: 2, N: 2}, {Placeholder: k, Kind: "API_KEY", By: "rule", Len: 40}}})+ Add(Record{Time: now, Agent: "claude", Provider: "anthropic", N: 1, Values: []Value{{Placeholder: k, Kind: "API_KEY", By: "rule", Len: 40}}})+ s := Sum(7)+ if s.Requests != 3 || s.N != 5 || s.Distinct != 2 || len(s.Days) != 7 {+ t.Fatalf("totals: %+v", s)+ }+ if s.Days[6].Requests != 2 || s.Days[6].Distinct != 2 || s.Days[4].Requests != 1 {+ t.Fatalf("days: %+v", s.Days)+ }+ if s.By["model"] != 1 || s.By["rule"] != 1 || s.By["known"] != 0 {+ t.Fatalf("by: %v", s.By)+ }+ if len(s.Kinds) != 2 || s.Kinds[0].Name != "PERSON" || s.Kinds[0].N != 3 || s.Kinds[0].Distinct != 1 {+ t.Fatalf("kinds: %+v", s.Kinds)+ }+ if len(s.Agents) != 2 || s.Agents[0].Name != "claude" || s.Agents[0].Distinct != 2 || s.Providers[0].Name != "anthropic" {+ t.Fatalf("agents %+v providers %+v", s.Agents, s.Providers)+ }+ if s.Values[0].Placeholder != p || s.Values[0].Requests != 2 || s.Values[0].By != "model" {+ t.Fatalf("values: %+v", s.Values)+ }+ if s.Hours[14] != 3 || s.Hours[15] != 2 {+ t.Fatalf("hours: %v", s.Hours)+ }+ // only one kind or finder+ if rs, more := List(0, 10, Filter{By: "model"}); len(rs) != 1 || more || rs[0].Agent != "codex" {+ t.Fatalf("filtered: %+v", rs)+ }+ rs, more := List(0, 2, Filter{})+ if len(rs) != 2 || !more || !rs[0].Time.Equal(now) {+ t.Fatalf("page: %+v %v", rs, more)+ }+ if rest, more := List(rs[1].Key, 2, Filter{}); len(rest) != 1 || more || rest[0].Agent != "codex" {+ t.Fatalf("next page: %+v %v", rest, more)+ }+}++// Days older than keepDays go.+func TestPrune(t *testing.T) {+ now := time.Date(2026, 10, 20, 15, 0, 0, 0, time.Local)+ dir := sandbox(t, now)+ old := filepath.Join(dir, "2026-09-01.jsonl")+ os.MkdirAll(dir, 0o700)+ os.WriteFile(old, []byte("{}\n"), 0o600)+ Add(Record{Time: now, N: 1, Values: []Value{{Placeholder: "{{X_aaaaaaaa}}", Kind: "X", By: "rule"}}})+ if _, err := os.Stat(old); !os.IsNotExist(err) {+ t.Fatalf("old day kept: %v", err)+ }+}diff --git a/internal/maskrec/summary.go b/internal/maskrec/summary.gonew file mode 100644index 000000000..74b5da09b--- /dev/null+++ b/internal/maskrec/summary.go@@ -0,0 +1,258 @@+package maskrec++import (+ "os"+ "path/filepath"+ "sort"+ "time"+)++// dayAgg is what a day's file holds, summed: read once, then kept up as+// Add writes to it.+type dayAgg struct {+ size int64+ requests int+ n int+ hours [24]int+ kinds map[string]int // each place counted+ vals map[string]*valAgg+ agents map[string]map[string]int // agent → placeholder → places+ providers map[string]map[string]int+}++type valAgg struct {+ kind, by, head string+ len int+ n, requests int+ last time.Time+}++func newDay() *dayAgg {+ return &dayAgg{kinds: map[string]int{}, vals: map[string]*valAgg{}, agents: map[string]map[string]int{}, providers: map[string]map[string]int{}}+}++func (a *dayAgg) add(r Record) {+ a.requests+++ a.n += r.N+ a.hours[r.Time.Local().Hour()] += r.N+ for _, v := range r.Values {+ a.kinds[v.Kind] += v.N+ x := a.vals[v.Placeholder]+ if x == nil {+ x = &valAgg{kind: v.Kind, by: v.By, head: v.Head, len: v.Len}+ a.vals[v.Placeholder] = x+ } else if x.by == "known" && v.By != "known" {+ x.by = v.By+ }+ x.n += v.N+ x.requests+++ if r.Time.After(x.last) {+ x.last = r.Time+ }+ addTo(a.agents, r.Agent, v)+ addTo(a.providers, r.Provider, v)+ }+}++func addTo(m map[string]map[string]int, name string, v Value) {+ if name == "" {+ return+ }+ if m[name] == nil {+ m[name] = map[string]int{}+ }+ m[name][v.Placeholder] += v.N+}++// day is the sum of path, read again only when it has changed size.+func day(path string) *dayAgg {+ fi, err := os.Stat(path)+ if err != nil {+ return newDay()+ }+ if store.days == nil {+ store.days = map[string]*dayAgg{}+ }+ if a := store.days[path]; a != nil && a.size == fi.Size() {+ return a+ }+ a := newDay()+ a.size = read(path, a.add)+ store.days[path] = a+ return a+}++// Day is one day's count.+type Day struct {+ Day string `json:"day"`+ Requests int `json:"requests"`+ N int `json:"n"` // values swapped, each place counted+ Distinct int `json:"distinct"` // different values+}++// Count is a kind's, an agent's or a provider's share.+type Count struct {+ Name string `json:"name"`+ N int `json:"n"`+ Distinct int `json:"distinct"`+}++// Seen is one different value, as the record has it.+type Seen struct {+ Placeholder string `json:"p"`+ Kind string `json:"kind"`+ By string `json:"by"`+ Head string `json:"head,omitempty"`+ Len int `json:"len"`+ N int `json:"n"`+ Requests int `json:"requests"`+ Last time.Time `json:"last"`+}++// Summary is what masking did over some days.+type Summary struct {+ Days []Day `json:"days"` // oldest first, each day of them+ Requests int `json:"requests"`+ N int `json:"n"`+ Distinct int `json:"distinct"`+ Hours [24]int `json:"hours"` // places, by the hour of the day+ Kinds []Count `json:"kinds"`+ By map[string]int `json:"by"` // different values by the finder that found each+ Agents []Count `json:"agents"`+ Providers []Count `json:"providers"`+ Values []Seen `json:"values"` // the most masked first, at most topValues+ Since string `json:"since,omitempty"`+}++const topValues = 60++// Sum is the summary of the days days up to today.+func Sum(days int) Summary {+ store.mu.Lock()+ defer store.mu.Unlock()+ if days <= 0 {+ days = 7+ }+ now := clock().Local()+ dir := Dir()+ out := Summary{By: map[string]int{}}+ if fs := files(dir); len(fs) > 0 {+ out.Since = fs[0].day+ }+ vals := map[string]*Seen{}+ kinds := map[string]*Count{}+ kindVals := map[string]map[string]bool{}+ who := map[string]map[string]int{}+ whom := map[string]map[string]int{}+ for i := days - 1; i >= 0; i-- {+ d := now.AddDate(0, 0, -i).Format(dayForm)+ var a *dayAgg+ if inTest() {+ a = memDay(d)+ } else {+ a = day(filepath.Join(dir, d+".jsonl"))+ }+ out.Days = append(out.Days, Day{Day: d, Requests: a.requests, N: a.n, Distinct: len(a.vals)})+ out.Requests += a.requests+ out.N += a.n+ for h, n := range a.hours {+ out.Hours[h] += n+ }+ for k, n := range a.kinds {+ if kinds[k] == nil {+ kinds[k] = &Count{Name: k}+ kindVals[k] = map[string]bool{}+ }+ kinds[k].N += n+ }+ for p, v := range a.vals {+ s := vals[p]+ if s == nil {+ s = &Seen{Placeholder: p, Kind: v.kind, By: v.by, Head: v.head, Len: v.len}+ vals[p] = s+ } else if s.By == "known" && v.by != "known" {+ s.By = v.by+ }+ s.N += v.n+ s.Requests += v.requests+ if v.last.After(s.Last) {+ s.Last = v.last+ }+ kindVals[v.kind][p] = true+ }+ merge(who, a.agents)+ merge(whom, a.providers)+ }+ out.Distinct = len(vals)+ for _, s := range vals {+ out.By[s.By]+++ out.Values = append(out.Values, *s)+ }+ sort.Slice(out.Values, func(i, j int) bool {+ a, b := out.Values[i], out.Values[j]+ if a.N != b.N {+ return a.N > b.N+ }+ return a.Placeholder < b.Placeholder+ })+ if len(out.Values) > topValues {+ out.Values = out.Values[:topValues]+ }+ for k, c := range kinds {+ c.Distinct = len(kindVals[k])+ out.Kinds = append(out.Kinds, *c)+ }+ out.Kinds = ranked(out.Kinds)+ out.Agents = counts(who)+ out.Providers = counts(whom)+ return out+}++func merge(into, from map[string]map[string]int) {+ for name, ps := range from {+ if into[name] == nil {+ into[name] = map[string]int{}+ }+ for p, n := range ps {+ into[name][p] += n+ }+ }+}++func counts(m map[string]map[string]int) []Count {+ var out []Count+ for name, ps := range m {+ c := Count{Name: name, Distinct: len(ps)}+ for _, n := range ps {+ c.N += n+ }+ out = append(out, c)+ }+ return ranked(out)+}++func ranked(cs []Count) []Count {+ sort.Slice(cs, func(i, j int) bool {+ if cs[i].N != cs[j].N {+ return cs[i].N > cs[j].N+ }+ return cs[i].Name < cs[j].Name+ })+ if cs == nil {+ cs = []Count{}+ }+ return cs+}++// memDay is day d of the records in memory, for a test binary that writes+// none to disk.+func memDay(d string) *dayAgg {+ a := newDay()+ for _, r := range store.snips {+ if r.Time.Local().Format(dayForm) == d {+ r.Snips = nil+ a.add(r)+ }+ }+ return a+}diff --git a/internal/redact/hits_test.go b/internal/redact/hits_test.gonew file mode 100644index 000000000..74ecb049a--- /dev/null+++ b/internal/redact/hits_test.go@@ -0,0 +1,80 @@+package redact++import (+ "strings"+ "testing"+)++// Each value masked is told to Seen with the finder that found it: the+// user's word, the user's rule, magpie's rules, the model (with the+// Chinese names it brings), and, in a later request, a value masked+// before. The hit holds the placeholder, never the value.+func TestSeenSaysWhoFoundIt(t *testing.T) {+ const word, mine, key, name, zh = "Bluefin", "ACME-7731-QXZ9", "sk-proj-Hq3vX9tLm2Wc8RzK4nYp", "Rina Kowalczyk", "张伟"+ f := &fakeModel{values: map[string]string{name: "private_person"}}+ s := "Project " + word + " ref " + mine + " key " + key + " ask " + name + ",并联系" + zh + "老师。"+ var hits []Hit+ o := Options{Secrets: true, Personal: true, Words: []string{word},+ Rules: []Rule{{Kind: "ticket", Prefix: "ACME-"}}, Model: f.find,+ Seen: func(h Hit) { hits = append(hits, h) }}+ out, n := Mask(s, o)+ if n != len(hits) {+ t.Fatalf("masked %d, told of %d: %v", n, len(hits), hits)+ }+ by := map[string]string{}+ for _, h := range hits {+ v, ok := Value(h.Placeholder)+ if !ok || !strings.Contains(out, h.Placeholder) {+ t.Fatalf("hit %v: no value or not in %q", h, out)+ }+ if strings.Contains(h.Placeholder, v) {+ t.Fatalf("hit %v holds its value", h)+ }+ by[v] = h.By+ }+ want := map[string]string{word: ByWord, mine: ByYours, key: ByRule, name: ByModel, zh: ByModel}+ for v, b := range want {+ if by[v] != b {+ t.Errorf("%q found by %q, want %q (all: %v)", v, by[v], b, by)+ }+ }++ // a later request without the model: the name it found before is+ // masked as a value known before+ hits = nil+ if out, _ := Mask("Thanks, "+name+"!", Options{Personal: true, Seen: o.Seen}); strings.Contains(out, name) || len(hits) != 1 || hits[0].By != ByKnown {+ t.Errorf("a value known before: %q %v", out, hits)+ }+}++// What JSON bodies mask is told the same way, once per place it is.+func TestSeenInJSON(t *testing.T) {+ const key = "sk-proj-Hq3vX9tLm2Wc8RzK4nYp"+ var hits []Hit+ body := `{"messages":[{"role":"user","content":"key ` + key + `"},{"role":"user","content":"again ` + key + `"}]}`+ _, n := MaskJSON([]byte(body), Options{Secrets: true, Seen: func(h Hit) { hits = append(hits, h) }})+ if n != 2 || len(hits) != 2 || hits[0].Placeholder != hits[1].Placeholder || hits[0].By != ByRule {+ t.Fatalf("n=%d hits=%v", n, hits)+ }+}++// A hint keeps a key's prefix and a name's first character, and how long+// each is, nothing more.+func TestHint(t *testing.T) {+ for _, c := range []struct {+ kind, v, head string+ n int+ }{+ {"API_KEY", "sk-proj-Hq3vX9tLm2Wc8RzK4nYp", "sk-", 28},+ {"API_KEY", "ghp_abcdefghijklmnopqrstu", "ghp_", 25},+ {"PERSON", "张伟", "张", 2},+ {"PERSON", "Rina Kowalczyk", "R", 14},+ {"EMAIL", "[email protected]", "r", 16},+ {"TERM", "Bluefin-codename-x", "B", 18},+ {"PHONE", "1", "", 1},+ } {+ if head, n := Hint(c.kind, c.v); head != c.head || n != c.n {+ t.Errorf("Hint(%s, %q) = %q, %d; want %q, %d", c.kind, c.v, head, n, c.head, c.n)+ }+ }+}diff --git a/internal/redact/json.go b/internal/redact/json.goindex 25b647dde..51f284327 100644--- a/internal/redact/json.go+++ b/internal/redact/json.go@@ -417,7 +417,7 @@ func MaskJSON(body []byte, o Options) ([]byte, int) {// number or an address the vendor wrote of its own can be one// masked in another conversation (13800138000 is in many), and// masked, its text would no longer match its signature- t, n := apply(t, knownSpans(t, Options{Secrets: o.Secrets}), placeholder)+ t, n := apply(t, knownSpans(t, Options{Secrets: o.Secrets}), placeholder, o.Seen)total += nreturn t})diff --git a/internal/redact/known.go b/internal/redact/known.goindex edee768e5..8e84d88e3 100644--- a/internal/redact/known.go+++ b/internal/redact/known.go@@ -157,7 +157,7 @@ func knownSpans(s string, o Options) []span {best = &known[s[i:i+minKnown]][j]}if best != nil {- out = append(out, span{i, i + len(best.v), best.kind})+ out = append(out, span{i, i + len(best.v), best.kind, ByKnown})i += len(best.v) - 1}}diff --git a/internal/redact/model.go b/internal/redact/model.goindex 81f5b8396..42ecd11f0 100644--- a/internal/redact/model.go+++ b/internal/redact/model.go@@ -68,7 +68,7 @@ func modelSpans(s string, o Options) []span {a, b = zhTrim(s, a, b)}if a < b && utf8.ValidString(s[a:b]) && modelTakes(kind, s[a:b]) {- out = append(out, span{a, b, kind})+ out = append(out, span{a, b, kind, ByModel})}}if o.covers(catPersonal) {diff --git a/internal/redact/redact.go b/internal/redact/redact.goindex d21c256ea..9d217b86c 100644--- a/internal/redact/redact.go+++ b/internal/redact/redact.go@@ -28,6 +28,7 @@ import ("sort""strings""sync"+ "unicode/utf8")// Options says what to mask.@@ -43,6 +44,9 @@ type Options struct {// Model, where set, finds more in the texts to mask (model.go)Model Model+ // Seen, where set, is told of each value masked+ Seen func(Hit)+// found are the spans Model found, by text, for one request's textsfound map[string][]Found}@@ -90,6 +94,8 @@ type rule struct {back, ahead int// cat is what turns the rule on (Options.covers): "" the secretscat string+ // yours is a rule the user wrote (customRules)+ yours bool// anyGroup takes the first group that matched, for a pattern with a// group in each of its alternativesanyGroup bool@@ -331,6 +337,60 @@ func placeholder(kind, v string) string {type span struct {start, end intkind string+ by string // which finder found it: one of the By constants+}++// Which finder found a value (Hit.By): magpie's rules, the user's own+// rules, the user's words, the local model (with the Chinese names it+// brings, which run only with it), or a value masked before (known.go).+const (+ ByRule = "rule"+ ByYours = "yours"+ ByWord = "word"+ ByModel = "model"+ ByKnown = "known"+)++// Hit is one value masked: its kind, the finder that found it, and the+// placeholder it went as. It never holds the value itself.+type Hit struct {+ Kind, By, Placeholder string+}++// Value is the value behind placeholder p, while this run of magpie+// remembers it.+func Value(p string) (string, bool) {+ mu.RLock()+ defer mu.RUnlock()+ v, ok := values[p]+ return v, ok+}++// Hint is as much of v as may be kept where v itself may not: its first+// character (the first few of a key's, which name its kind: sk-, ghp_)+// and how many characters it has.+func Hint(kind, v string) (head string, n int) {+ n = utf8.RuneCountInString(v)+ if n < 2 {+ return "", n+ }+ k := 1+ if _, personal := kindCats[kind]; !personal && kind != "TERM" && n >= 16 {+ // a key's prefix, up to where its random part starts+ k = 0+ for _, c := range v {+ if k >= 4 || !strings.ContainsRune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ-_", c) {+ break+ }+ k+++ if c == '-' || c == '_' {+ break+ }+ }+ k = max(k, 1)+ }+ r := []rune(v)+ return string(r[:k]), n}// Mask swaps what o covers in s for placeholders, and says how many.@@ -353,7 +413,7 @@ func mask(s string, o Options, put func(kind, v string) string) (string, int) {if j < 0 {break}- found = append(found, span{i + j, i + j + len(w), "TERM"})+ found = append(found, span{i + j, i + j + len(w), "TERM", ByWord})i += j + len(w)}}@@ -388,12 +448,12 @@ func mask(s string, o Options, put func(kind, v string) string) (string, int) {took := falsefor g, kind := range r.groups {if a, b := m[2*g+2], m[2*g+3]; a >= 0 && r.takes(s, a, b, r.groupOK[g]) {- found = append(found, span{a, b, kind})+ found = append(found, span{a, b, kind, r.by()})took = true}}if took {- split = append(split, span{m[2], m[1], ""})+ split = append(split, span{m[2], m[1], "", ""})}continue}@@ -416,7 +476,7 @@ func mask(s string, o Options, put func(kind, v string) string) (string, int) {ok = nil}if r.takes(s, a, b, ok) {- found = append(found, span{a, b, r.kind})+ found = append(found, span{a, b, r.kind, r.by()})}}}@@ -429,7 +489,15 @@ func mask(s string, o Options, put func(kind, v string) string) (string, int) {// kind it has always hadfound = append(found, modelSpans(s, o)...)found = append(found, knownSpans(s, o)...)- return apply(s, found, put)+ return apply(s, found, put, o.Seen)+}++// by is the finder r is, for Hit.By.+func (r rule) by() string {+ if r.yours {+ return ByYours+ }+ return ByRule}// matches are r's matches in s, lower being lowerASCII(s), as@@ -513,7 +581,8 @@ func (r rule) takes(s string, a, b int, ok func(string) bool) bool {// apply swaps each of found in s for what put makes of its value, the// first and longest where they overlap, nothing inside a placeholder.-func apply(s string, found []span, put func(kind, v string) string) (string, int) {+// seen, where set, is told of each.+func apply(s string, found []span, put func(kind, v string) string, seen func(Hit)) (string, int) {if len(found) == 0 {return s, 0}@@ -542,7 +611,11 @@ func apply(s string, found []span, put func(kind, v string) string) (string, intcontinue}b.WriteString(s[last:f.start])- b.WriteString(put(f.kind, s[f.start:f.end]))+ p := put(f.kind, s[f.start:f.end])+ b.WriteString(p)+ if seen != nil {+ seen(Hit{f.kind, f.by, p})+ }last, n = f.end, n+1}b.WriteString(s[last:])diff --git a/internal/redact/rules.go b/internal/redact/rules.goindex 9ad3dea03..f9e044f91 100644--- a/internal/redact/rules.go+++ b/internal/redact/rules.go@@ -130,7 +130,7 @@ func customRules(in []Rule) []rule {if re == nil {continue}- c := rule{kind: RuleKind(r.Kind), re: re, ok: func(v string) bool { return len(v) >= minCustomLen }}+ c := rule{kind: RuleKind(r.Kind), yours: true, re: re, ok: func(v string) bool { return len(v) >= minCustomLen }}if r.Prefix != "" {c.markers = []string{r.Prefix}}diff --git a/internal/redact/zhname.go b/internal/redact/zhname.goindex b3c7ce0bc..8087e0fe4 100644--- a/internal/redact/zhname.go+++ b/internal/redact/zhname.go@@ -87,7 +87,7 @@ func zhNameSpans(s string, person []span) []span {if ok {kind = "PERSON"}- cs = append(cs, span{a, b, kind})+ cs = append(cs, span{a, b, kind, ByModel})i = b}// a name joined by 和, 与, 跟, 及 or 、 to one that is a name is one