Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
privacy: masked words can match as whole words only, by a switch under Masked words (acmilan_)
yetone7aeae5967099
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.mdindex 8b854452b..57d2ee4d0 100644--- a/docs/subsystems/gateway-routing.md+++ b/docs/subsystems/gateway-routing.md@@ -141,6 +141,7 @@ recorded as a prediction failure (`noPrediction`).- With *Mask personal data* on, each kind in `redact.Categories` ([`internal/redact/more.go`](../../internal/redact/more.go)) is masked unless the user turned it off (`redactKinds` in settings, `Options.Kinds`). On by default: US SSNs, passport numbers, IBANs that pass mod 97, birthdays next to a word that says so (a date over a year ago: one in the last year or ahead is the day an agent runs on, `dateOK`), MAC addresses, and serial numbers next to S/N or the like. Off by default: the user name in a home folder, user@host in a prompt or ssh command, public IPv4 and IPv6 addresses (never private, loopback, link-local, CGNAT or documentation ranges) and cloud bucket names, since agents send them in nearly every request and the model must copy the placeholder exactly inside a path or URL. Context rules run only near their words (`rule.matches`). What reads as an address but ends in a file's extension that no domain has (`…@2x.jpg`, `[email protected]`; `fileExt` in `realEmail`), or a scale before an extension that is a domain too (`[email protected]`, `[email protected]`; `scaleExt`), is a file's name, for the rule and the local model alike (#3029). `[email protected]` is still an address. The Settings rows come from `REDACT_KINDS` in app.js, which `TestRedactKindsAreRedacts` holds to the same ids and defaults.- With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken, nor a snake_case word as a name (Codex's `sequential_cutoff`, #3144), nor as a secret what names a thing: a UUID, a digest with its algorithm's name, a file's path (`namesThing`; the model takes all three for secrets). The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with it on a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left. The model is the user's pick (`settings.RedactModelName`, `pii.Choose`; the app's menu beside the switch, `pmPick`): AstrLink Guard (QuantumNous/astrlink-guard at a pinned revision, Apache-2.0, a 39 MB int8 Multilingual-MiniLM token classifier for Chinese and English), run in magpie's own process on the CPU by [`internal/pii/guard`](../../internal/pii/guard) (an ONNX reader, the XLM-R tokenizer and int8 kernels in Go, NEON/SDOT on arm64 and SSE2 on amd64; its labels named as Privacy Filter's, `guardLabels`), and on a Mac through CoreML on the Neural Engine or the GPU ([`coreml.go`](../../internal/pii/guard/coreml.go), [`internal/pii/guard/coreml`](../../internal/pii/guard/coreml); yyhhyyyyyy on X): the same network from the same int8 weights in float16, written from the downloaded model into a program made by `coreml_gen.py` and compiled once into the model's folder, 512 tokens in about 12 ms where the CPU takes 400, its probabilities within .07 of the publisher's fp32 model where the CPU's are within .13 (the CPU quantizes activations as the int8 model does, CoreML doesn't), so a span near the threshold can fall the other side (a go.mod module path is a secret at .81 on CoreML, .79 on the CPU, .785 in fp32); a window under 16 tokens (`coremlMin`), a Mac before macOS 13 or any failure goes to the CPU, loaded on the first request and let go after 10 minutes with none; or OpenAI's Privacy Filter. With no pick, a user who had Privacy Filter on or downloaded keeps it and everyone else gets AstrLink Guard (#3578): on 234 segments of real code and config it found 2 spans where Privacy Filter found 100 (38 of them go.sum hashes after redact's own checks), and with the rules it found 42 of 42 test values to Privacy Filter's 41. Each model has its own download, cache file and turn (`backend`). Both find at a floor of 0.5 (`pii.Floor`) and give each span its score (AstrLink Guard: the least of its tokens' probabilities, rounded to a float32 so a cached span reads back the same; Privacy Filter: pf-serve's own), and masking keeps those at the model's threshold or over (`pii.Threshold` in `masking.model`): the user's for that model (`settings.RedactModelScore`, this computer's own; Settings → Privacy's Confidence threshold, `pmScore`, 0.5 to 0.95), else the model's own (`MinScore`: Privacy Filter 0.5, AstrLink Guard 0.8, its publisher's). The defaults stay because on the eval a higher one mostly loses values the checks after the model would have kept (Privacy Filter at 0.85 lost a Chinese name; AstrLink Guard at 0.9 a phone in a tool result), while the false alarms it would drop are mostly dropped by those checks already (Yayoi_no_yume on X). Privacy Filter is a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags; on a GPU it holds the model once, in the device's memory, #3590; `pf-serve-vulkan` runs its MoE in 2048-token chunks, `PF_MOE_CHUNK`, 1.77× faster than its default 4096 on an RX 470/580-class GPU and a little slower on Metal, so Vulkan's alone, and a value the user set is kept, `moeChunk`): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, through magpie's proxy (`netproxy.Func`, as its other requests; hh0592821 on Discord, whose download went direct before), from hf-mirror first in China (`source.chinaFirst`) and from hf-mirror after Hugging Face everywhere else, the switch off too (`source.modelMirror`), so a poisoned or blocked huggingface.co doesn't stop it; hf-mirror serves AstrLink Guard's files itself and sends Privacy Filter's back to Hugging Face) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU, doesn't load there in 40s (`gpuLoadTimeout`, well inside the gateway's 2 minutes), or doesn't answer in time there twice in a row, is started on the CPU, and Settings says why; a request it didn't answer in time once is tried on the GPU again, since a slow request on a weak GPU is no lost GPU (#3682). The GPU is tried again 10 minutes later, doubling to 2 hours while it keeps failing, with no restart (`gpuRetryAfter`), and one that answers is the model's again), joins short texts into 12 KB chunks and maps the spans back, cuts a text over 12 KB into pieces of 12 KB at most that overlap by 1 KB (`pieces`: after a line's end, else a space, else at a rune's start), each kept by its own hash, their spans put back at the text's offsets and merged where pieces overlap (`merge`: one found twice is one, one cut by a piece's end is whole), and asks pf-serve a chunk at a time (#3682: a 480 KB text whole took an RX 470/580 90s and kept nothing when cut off), keeps the last 20000 answers by hash, on disk too (`found-<model>.bin` beside the model: hashes, offsets, labels and each span's score, never text; a `pfound01` file from before scores is read with every span sure, so it masks as it did; written through temp-and-rename, a bad read taken as empty), so a restart doesn't ask the model about every history again (#3682), and ends it after 10 minutes with no request. Its requests take turns on its one pipe (`turn`), and a request reads the cache again when its turn comes; a request waits for its turn only while its context lives (`pii.FindContext`, through `gwmw.ContextTeller`), so one whose client went away leaves the queue (the gateway's 2 minutes leave masking's time out, #3682); the chunk pf-serve is on is finished and kept, so a request sent again asks only about what is left, and between chunks the turn is given up and taken again, so a short request goes before the rest of a long one. The status (`pii.Now`), `pii.Stop` and the idle timer never wait for a load or a request: `srv.mu` is held only to read or set the state, and `Stop` ends a load or a request under way at once. A request held before routing while it waits for the model is answered 504 saying so, and how to send without it (`unroutedMsg`); one of an older tag (`oldTags`, magpie updated since the download) runs until the pinned tag's is fetched (`upgrade`, on its start), which is then started on the next request and the old deleted ([`internal/pii`](../../internal/pii)). A failure names what ended `pf-serve`: magpie's own end of a request it didn't answer in time, else its exit status or signal and its last stderr lines past ggml's startup chatter (`died`, `lastSaid`). After one, requests are held back at once with that failure, not after a load each, until the model is tried again 30s later, doubling to 10 minutes while it keeps failing (`heldBack`, `retryAfter`; lazyeo on Discord, an M2 whose every request waited out the 504). A model that is on and can't answer (not on disk, didn't start, hung, held back) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).- With *Mask secrets* on, the user's own masking rules (`redactRules`, `redact.Rule`: a placeholder name and a prefix or an RE2 pattern, [`internal/redact/rules.go`](../../internal/redact/rules.go)) run before magpie's. Settings → Privacy adds them, removes them, and edits one in place (`POST /api/settings/redact-rule` with the rule's index and its current value, `editRedactRule` in [`internal/gui/api.go`](../../internal/gui/api.go)). Every rule is checked by `redact.CheckRules` when settings are saved; an edit that doesn't compile, matches the empty string, repeats another rule or names a rule no longer at that index is refused with the rules left as they were. The masker reads the rules from settings on each call (`options` in [`internal/masking`](../../internal/masking/masking.go)), so a change applies to the next request.+ - The user's own masked words (`redactWords`, `Options.Words`; Settings → Privacy's *Masked words*) go as `{{TERM_…}}` whatever else is on, and come back as written. They match as written, inside longer words too, unless the user asks: *Masked words in any case* (`redactWordsAnyCase`, `WordsAnyCase`) matches in any capitals by Unicode simple folding, and *Whole words only* (`redactWordsWhole`, `WordsWhole`; acmilan_ on Discord) skips a match with a letter, digit or `_` beside it, and for a number a `.` and digit beside it, so `ABC` masks `name='ABC'` but not `XABCD`, and `123456` not `91234567`, `0.123456` or `123456.78` (`wholeWord` in [`redact.go`](../../internal/redact/redact.go)). Chinese and Japanese letters count as no letter there (`wordRune`): a word of theirs is found inside their text, and `ABC` in 用ABC做 is whole. Both are off by default; a page from before them that saves without the field keeps the user's choice (`api.go`). `TestWordsWhole`, `TestWordsWholeJSON`, `TestCountTokensRedacted`.- With *Keep a key's form* on (`settings.RedactShape`, `Options.Shape`, under *Mask secrets*; Charles Cao on X), a secret of 16 characters or more, all letters, digits, `-`, `_` or `.`, goes as a made-up key of its form instead of a placeholder: its prefix kept (`sk-proj-`, `ghp_`, `AKIA`, up to four letters and their separator; `keyHead`), the same length and separators, and each other character a random one of its kind (a hex key's any hex digit of its case), drawn from an HMAC of the value under the placeholder key, so one key goes as the same made-up one every turn and a vendor's cache still matches (`shapeOf`, `shaped` in [`internal/redact/shape.go`](../../internal/redact/shape.go)). Personal data, the user's words and anything shorter or with other characters go as placeholders still. A made-up key is kept in the masker's memory with the placeholders, so it is recorded, re-masked in what a vendor sealed, and put back wherever the reply has it, before the placeholders (`restoreShapes`); a stream holds the end of an event that may be the start of one (`shapeTail`, after the last whole one), and a made-up key the vendor sends back unrestored isn't masked again (`outsideShapes`). `TestShapeKeepsForm`, `TestShapeJSON`, `TestShapeStream`. Insights › Masking and Settings' *Try it* mark a made-up key where they mark a placeholder, matching the text against the request's own values only, so a key of the same form the user wrote isn't marked (`maskMatches` in `plugins/redact/page.js`; #3599, masking-records and mask-actions GUI tests).- What is done with a value found is the user's per kind (#1977, `redactActions` in settings, `Options.Actions`, set by `POST /api/settings/redact-actions`; a kind left out is masked). `redact.Warn` sends it as written: the masker tells it, with a placeholder no one remembers, and it isn't counted as masked. `redact.Block` masks it as any other, and the gateway holds the request back with a 400 that names the kind and the value's hint, never the value (`blocked`, `maskBlocked` in [`redact.go`](../../internal/gateway/redact.go)); nothing is sent and the record is kept as `Blocked` (`noteBlocked`). A value on the *Never mask* list (`redactAllow`, `Options.Allow`, `POST /api/settings/redact-allow`) goes as written when it is the whole value found, in any case, and isn't told; a longer value it is part of is masked. Settings → Privacy's *Try it* posts a text to `POST /api/masking/preview` ([`masking.go`](../../internal/gui/masking.go)), which masks it as a request would be (`redact.Preview`) and remembers nothing: its placeholders stand for nothing afterwards, its values aren't masked later as seen before, and it returns only the masked text and each hit's kind, finder, action, placeholder and hint. The page keeps the text in no storage.- Each request that had anything masked is recorded as it goes to the vendor (`noteMasked` in [`masked.go`](../../internal/gateway/masked.go), from chat, the Codex backend, search, images, video, System One, embeddings and audio; count_tokens is masked and not recorded). The masker says what it swapped through `gwmw.Teller` (`JSONHits`, `TextHits`: each placeholder's kind, its finder, `redact.ByRule`, `ByYours`, `ByWord`, `ByModel` or `ByKnown`, and a hint of its first characters and its length), and [`internal/maskrec`](../../internal/maskrec/maskrec.go) keeps one line per request in a file per day under `~/.config/magpie/masking`, 30 days and 100 MB in all: the agent, session, provider, model, and each placeholder with its kind, finder, action and hint, never the value. A chat, an embedding or a video whose group can pass it to another member is held in memory until the request is over (`noteMaskedUntil`, `maskrec.Hold`), and its line names the provider and model the last try went to, not the first planned, which a failover may never have reached; the list sorts each day's lines by key, since a long request's line is written after later ones. The text around each placeholder as the vendor saw it is kept in memory only, for the last 400 records, and what a placeholder stood for only in the masker's own memory (`Teller.Value`); after a restart neither is there. A recent call (`Call.Masked`) and the route trace (`Route.Masked`) carry the counts and the record's key, which the Gateway and Routing pages' badge opens in Settings → Privacy (`/api/masking/*`, [`internal/gui/masking.go`](../../internal/gui/masking.go)). A masker without `Teller`, a plugin's, masks as before and records nothing. Under a test binary testenv isolated nothing is written (`maskrec.inTest`). The page's Clear records… (after a confirm) posts `/api/masking/clear`, and `maskrec.Clear` removes every day's file whole under the store's lock, with the in-memory text around placeholders; a day it can't remove stays listed and is named in the error. What the masker holds for each placeholder stays, so a placeholder still in an agent's context is put back in its answers.diff --git a/internal/gateway/redact_test.go b/internal/gateway/redact_test.goindex 799cbded7..cf9fc872f 100644--- a/internal/gateway/redact_test.go+++ b/internal/gateway/redact_test.go@@ -185,6 +185,11 @@ func TestCountTokensRedacted(t *testing.T) {// a word in other capitals only when the user asked for any case{"words_as_written", settings.Settings{RedactWords: []string{"Nightjar"}}, "NIGHTJAR", ""},{"words_any_case", settings.Settings{RedactWords: []string{"Nightjar"}, RedactWordsAnyCase: true}, "NIGHTJAR", "{{TERM_"},+ // a word inside a longer one goes as written only with whole words+ // on (acmilan_ on Discord), and a whole one is masked still+ {"words_inside", settings.Settings{RedactWords: []string{"ABC"}}, "XABCD", "{{TERM_"},+ {"words_whole", settings.Settings{RedactWords: []string{"ABC"}, RedactWordsWhole: true}, "XABCD", ""},+ {"words_whole_alone", settings.Settings{RedactWords: []string{"ABC"}, RedactWordsWhole: true}, "name='ABC'", "{{TERM_"},{"rules", settings.Settings{Redact: true, RedactRules: []redact.Rule{{Kind: "GW_KEY", Prefix: "acme-"}}}, "acme-Zx9ab12cdEF", "{{GW_KEY_"},{"rules_off", settings.Settings{RedactRules: []redact.Rule{{Kind: "GW_KEY", Prefix: "acme-"}}}, "acme-Zx9ab12cdEF", ""},{"off", settings.Settings{}, key, ""},diff --git a/internal/gui/api.go b/internal/gui/api.goindex fa427ddb1..fb786a98b 100644--- a/internal/gui/api.go+++ b/internal/gui/api.go@@ -1240,6 +1240,10 @@ func Handler(w Windows, gw *gateway.Server) http.Handler {if _, sent := fields["redactWordsAnyCase"]; !sent {in.RedactWordsAnyCase = cur.RedactWordsAnyCase}+ // nor whether they match only as whole words+ if _, sent := fields["redactWordsWhole"]; !sent {+ in.RedactWordsWhole = cur.RedactWordsWhole+ }// nor whether a key goes as a made-up one of its formif _, sent := fields["redactShape"]; !sent {in.RedactShape = cur.RedactShapediff --git a/internal/gui/api_settings_test.go b/internal/gui/api_settings_test.goindex 92cac8a3e..accb96d80 100644--- a/internal/gui/api_settings_test.go+++ b/internal/gui/api_settings_test.go@@ -326,3 +326,28 @@ func TestSettingsRedactWordsAnyCaseKept(t *testing.T) {}}}++// Whole words only is the Privacy page's too: a page from before it sends+// no redactWordsWhole, and its save leaves the user's choice as it was; a+// page that sends it sets it (acmilan_ on Discord).+func TestSettingsRedactWordsWholeKept(t *testing.T) {+ t.Setenv("XDG_CONFIG_HOME", t.TempDir())+ h := Handler(nil, nil)+ for _, c := range []struct {+ body string+ want bool+ }{+ {`{"theme":"dark","redactWords":["ABC"],"redactWordsWhole":true}`, true},+ {`{"theme":"light","redactWords":["ABC"],"redactWordsAnyCase":true}`, true},+ {`{"theme":"light","redactWords":["ABC"],"redactWordsWhole":false}`, false},+ } {+ rec := httptest.NewRecorder()+ h.ServeHTTP(rec, httptest.NewRequest("POST", "/api/settings", strings.NewReader(c.body)))+ if rec.Code != http.StatusOK {+ t.Fatalf("saving %s: %d %s", c.body, rec.Code, rec.Body)+ }+ if got := settings.Load(); got.RedactWordsWhole != c.want || !reflect.DeepEqual(got.RedactWords, []string{"ABC"}) {+ t.Fatalf("after %s: whole words %v, words %v; want %v", c.body, got.RedactWordsWhole, got.RedactWords, c.want)+ }+ }+}diff --git a/internal/gui/assets/app.js b/internal/gui/assets/app.jsindex 83db6c7c3..c41574d39 100644--- a/internal/gui/assets/app.js+++ b/internal/gui/assets/app.js@@ -21892,6 +21892,8 @@ function renderRedact(s, keep) {// plain words, matched as written unless the user asks for any caserow(t("Masked words in any case"), t("A word also matches in other capitals: Nightjar masks NIGHTJAR and nightjar too"),onOff(s.redactWordsAnyCase, (redactWordsAnyCase) => savePrefs({ ...keep, redactWordsAnyCase }))).id = "redactWordsAnyCaseRow";+ row(t("Whole words only"), t("A word is not masked inside a longer one: ABC masks ABC and 'ABC' but not XABCD, and 123456 not 0.123456"),+ onOff(s.redactWordsWhole, (redactWordsWhole) => savePrefs({ ...keep, redactWordsWhole }))).id = "redactWordsWholeRow";renderRedactRules(s, row);renderMaskActions(s);renderMaskTry(s);@@ -23037,7 +23039,7 @@ function prefsKeep(s) {uiFont: s.uiFont || null, codeFont: s.codeFont || null,otel: s.otel || {},trayUsages: s.trayUsages || [],- redact: !!s.redact, redactPersonal: !!s.redactPersonal, redactKinds: s.redactKinds || {}, redactWords: s.redactWords || [], redactWordsAnyCase: !!s.redactWordsAnyCase, redactShape: !!s.redactShape, codexWarmup: s.codexWarmup || "",+ redact: !!s.redact, redactPersonal: !!s.redactPersonal, redactKinds: s.redactKinds || {}, redactWords: s.redactWords || [], redactWordsAnyCase: !!s.redactWordsAnyCase, redactWordsWhole: !!s.redactWordsWhole, redactShape: !!s.redactShape, codexWarmup: s.codexWarmup || "",claudeWarmup: s.claudeWarmup || "", codexWarmAt: s.codexWarmAt || "", claudeWarmAt: s.claudeWarmAt || "",codexWarmAts: warmTimes(s.codexWarmAts, s.codexWarmAt), claudeWarmAts: warmTimes(s.claudeWarmAts, s.claudeWarmAt), workbuddyCheckin: !!s.workbuddyCheckin, traeCheckin: !!s.traeCheckin, minimaxCheckin: !!s.minimaxCheckin, qoderCheckin: !!s.qoderCheckin, noStats: !!s.noStats, noUsageStats: !!s.noUsageStats,memberModel: !!s.memberModel, noLoopGuard: !!s.noLoopGuard, claudeLeanPrompt: !!s.claudeLeanPrompt, refusalRetries: s.refusalRetries ?? null,diff --git a/internal/gui/assets/i18n.js b/internal/gui/assets/i18n.jsindex d0092e45a..5ac9b65a8 100644--- a/internal/gui/assets/i18n.js+++ b/internal/gui/assets/i18n.js@@ -1198,6 +1198,8 @@ const I18N = {"Masked words": "脱敏词","Masked words in any case": "脱敏词不区分大小写","A word also matches in other capitals: Nightjar masks NIGHTJAR and nightjar too": "大小写不同也算同一个词:Nightjar 也会遮住 NIGHTJAR 和 nightjar",+ "Whole words only": "仅完整单词匹配",+ "A word is not masked inside a longer one: ABC masks ABC and 'ABC' but not XABCD, and 123456 not 0.123456": "不遮住长词中的一部分:ABC 遮住 ABC 和 'ABC',不遮 XABCD;123456 不遮 0.123456","Find with a local model": "用本地模型查找","Delete model": "删除模型","Delete the model?": "删除模型?",@@ -5208,6 +5210,8 @@ const I18N = {"Masked words": "脫敏詞","Masked words in any case": "脫敏詞不分大小寫","A word also matches in other capitals: Nightjar masks NIGHTJAR and nightjar too": "大小寫不同也算同一個詞:Nightjar 也會遮住 NIGHTJAR 和 nightjar",+ "Whole words only": "僅完整單字匹配",+ "A word is not masked inside a longer one: ABC masks ABC and 'ABC' but not XABCD, and 123456 not 0.123456": "不遮住長詞中的一部分:ABC 遮住 ABC 和 'ABC',不遮 XABCD;123456 不遮 0.123456","Find with a local model": "用本機模型查找","Delete model": "刪除模型","Delete the model?": "刪除模型?",@@ -9188,6 +9192,8 @@ const I18N = {"Masked words": "マスクする語","Masked words in any case": "マスクする語の大文字・小文字を区別しない","A word also matches in other capitals: Nightjar masks NIGHTJAR and nightjar too": "大文字・小文字が違っても同じ語として扱います:Nightjar で NIGHTJAR や nightjar もマスクされます",+ "Whole words only": "単語全体のみ",+ "A word is not masked inside a longer one: ABC masks ABC and 'ABC' but not XABCD, and 123456 not 0.123456": "長い語の一部はマスクしません:ABC は ABC と 'ABC' をマスクし、XABCD はマスクしません。123456 は 0.123456 をマスクしません","Find with a local model": "ローカルモデルで検出","Delete model": "モデルを削除","Delete the model?": "モデルを削除しますか?",@@ -13198,6 +13204,8 @@ const I18N = {"Masked words": "Maskierte Wörter","Masked words in any case": "Maskierte Wörter ohne Groß-/Kleinschreibung","A word also matches in other capitals: Nightjar masks NIGHTJAR and nightjar too": "Ein Wort passt auch in anderer Schreibung: Nightjar maskiert auch NIGHTJAR und nightjar",+ "Whole words only": "Nur ganze Wörter",+ "A word is not masked inside a longer one: ABC masks ABC and 'ABC' but not XABCD, and 123456 not 0.123456": "Ein Wort wird in einem längeren nicht maskiert: ABC maskiert ABC und 'ABC', aber nicht XABCD, und 123456 nicht 0.123456","Find with a local model": "Mit lokalem Modell finden","Delete model": "Modell löschen","Delete the model?": "Modell löschen?",diff --git a/internal/gui/tests/redact-words-whole.test.cjs b/internal/gui/tests/redact-words-whole.test.cjsnew file mode 100644index 000000000..13a0c1279--- /dev/null+++ b/internal/gui/tests/redact-words-whole.test.cjs@@ -0,0 +1,113 @@+// Run with Node's test runner and Playwright on the module path; see README.md.+// Settings › Privacy's "Whole words only" (acmilan_ on Discord): a row under+// Masked words in any case, Off unless the user turned it on, so an+// existing user's words are found inside longer ones as before. A click on+// On saves redactWordsWhole with the words, any case and the rest of the+// settings as they were, and moves nothing; a later save from the page+// keeps it. In every language, Chromium and WebKit, wide and narrow.+const assert = require("node:assert/strict");+const fs = require("node:fs/promises");+const path = require("node:path");+const { test } = require("node:test");+const { chromium, webkit } = require("playwright");+const { reach, wheelTo, watchSettingsReads, settingsDrawn } = require("./reader.cjs");++const assets = path.resolve(__dirname, "../assets");+const words = {+ en: { name: "Whole words only", on: "On", off: "Off" },+ zh: { name: "仅完整单词匹配", on: "开启", off: "关闭" },+ "zh-TW": { name: "僅完整單字匹配", on: "開啟", off: "關閉" },+ ja: { name: "単語全体のみ", on: "オン", off: "オフ" },+ de: { name: "Nur ganze Wörter", on: "An", off: "Aus" },+};++function serve(lang, saved) {+ let settings = { lang, theme: "light", redact: true, redactPersonal: false, redactWords: ["ABC", "123456"], redactWordsAnyCase: true };+ const state = { agents: [], profiles: [], settings };+ return async (r) => {+ const url = new URL(r.request().url());+ const json = (data) => r.fulfill({ json: data });+ if (url.pathname === "/boot.js") return r.fulfill({ contentType: "text/javascript", body: `window.bootPrefs = {lang:"${lang}",theme:"light",web:true};` });+ if (url.pathname === "/wails/runtime.js") return r.fulfill({ contentType: "text/javascript", body: "export const Window = {};" });+ if (url.pathname === "/api/state") return json(state);+ if (url.pathname === "/api/settings") {+ if (r.request().method() === "POST") {+ settings = JSON.parse(r.request().postData());+ saved.push(settings);+ }+ return json(settings);+ }+ if (url.pathname === "/api/groups") return json({ models: [], groups: [], pools: [] });+ if (url.pathname === "/api/providers") return json({ providers: [], presets: [], excluded: [], gateway: { running: true, window: true, url: "http://127.0.0.1:3999" } });+ if (url.pathname.startsWith("/api/")) return json({});+ const file = path.join(assets, url.pathname === "/" ? "index.html" : url.pathname);+ const contentType = { ".html": "text/html", ".js": "text/javascript", ".css": "text/css", ".svg": "image/svg+xml", ".png": "image/png" }[path.extname(file)];+ await r.fulfill({ body: await fs.readFile(file), contentType });+ };+}++for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium", "webkit"])) {+ for (const [lang, w] of Object.entries(words)) {+ for (const width of [1100, 640]) {+ test(`${engine} ${lang} ${width}px: masked words match as whole words only when turned on`, async (t) => {+ const browser = await (engine === "webkit" ? webkit.launch() : chromium.launch({ channel: "chromium" }));+ t.after(() => browser.close());+ const page = await (await browser.newContext({ viewport: { width, height: 1400 } })).newPage();+ page.setDefaultTimeout(5000);+ const errors = [], saved = [];+ page.on("pageerror", (e) => errors.push(e.message));+ await page.route("**/*", serve(lang, saved));+ await watchSettingsReads(page);+ await page.goto("http://magpie.test/?view=settings&tab=privacy");+ const row = page.locator("#redactWordsWholeRow");+ await row.waitFor();+ await settingsDrawn(page);+ // the local model's section is above it: wheel it up, clear of the+ // bar along a phone-width page's foot+ await wheelTo(page, row, 300);++ const m = await row.evaluate((r) => {+ const sub = r.querySelector(".sub"), list = document.querySelector("#redactList").getBoundingClientRect(), b = r.getBoundingClientRect();+ return {+ name: r.querySelector(".name").textContent, sub: sub.textContent,+ on: r.querySelector(".segs .opt.on")?.textContent,+ // right under Masked words in any case+ after: r.previousElementSibling?.id === "redactWordsAnyCaseRow",+ cut: sub.scrollWidth > sub.clientWidth + 1 || sub.scrollHeight > sub.clientHeight + 1,+ inside: b.left >= list.left - 1 && b.right <= list.right + 1,+ wide: document.documentElement.scrollWidth > document.documentElement.clientWidth,+ };+ });+ assert.equal(m.name, w.name);+ assert.equal(m.on, w.off, "off unless the user turned it on");+ assert(m.after, "under Masked words in any case");+ assert(!m.cut && m.inside && !m.wide, "shown in full, inside the page");+ if (lang !== "en") assert(!m.sub.includes("A word is not masked"), "its words translated");++ const before = await page.evaluate(() => [scrollX, scrollY, document.querySelector("#view-settings").scrollTop]);+ const b = await row.locator(".segs .opt", { hasText: w.on }).boundingBox();+ await page.mouse.click(b.x + b.width / 2, b.y + b.height / 2);+ await page.waitForFunction(() => document.querySelector("#redactWordsWholeRow .segs .opt.on")?.textContent === t("On"));+ await new Promise((r) => setTimeout(r, 300));+ assert.equal(saved.length, 1, "one save");+ assert.equal(saved[0].redactWordsWhole, true);+ assert.equal(saved[0].redactWordsAnyCase, true, "any case kept");+ assert.deepEqual(saved[0].redactWords, ["ABC", "123456"]);+ assert.equal(saved[0].redact, true);+ assert.deepEqual(await page.evaluate(() => [scrollX, scrollY, document.querySelector("#view-settings").scrollTop]), before, "the click moved nothing");++ // another choice on the page keeps it: Mask personal data, under+ // the key's form+ const p = page.locator("#redactShapeRow + .row").locator(".segs .opt", { hasText: w.on });+ await reach(page, p); // above it now, maybe under the tabs+ await p.click();+ await page.waitForFunction(() => document.querySelector("#redactShapeRow + .row .segs .opt.on")?.textContent === t("On"));+ await new Promise((r) => setTimeout(r, 300));+ assert.equal(saved.length, 2);+ assert.equal(saved[1].redactPersonal, true);+ assert.equal(saved[1].redactWordsWhole, true, "a later save keeps it");+ assert.deepEqual(errors, []);+ });+ }+ }+}diff --git a/internal/masking/masking.go b/internal/masking/masking.goindex 88bcb4886..c6a1f1dd3 100644--- a/internal/masking/masking.go+++ b/internal/masking/masking.go@@ -28,7 +28,7 @@ func options() redact.Options { return optionsCtx(context.Background()) }// optionsCtx is options with the model's wait ending with ctx.func optionsCtx(ctx context.Context) redact.Options {st := settings.Shared()- o := redact.Options{Secrets: st.Redact, Personal: st.RedactPersonal, Kinds: st.RedactKinds, Words: st.RedactWords, WordsAnyCase: st.RedactWordsAnyCase, Rules: st.RedactRules,+ o := redact.Options{Secrets: st.Redact, Personal: st.RedactPersonal, Kinds: st.RedactKinds, Words: st.RedactWords, WordsAnyCase: st.RedactWordsAnyCase, WordsWhole: st.RedactWordsWhole, Rules: st.RedactRules,Actions: st.RedactActions, Allow: st.RedactAllow, Shape: st.RedactShape}if st.RedactModel {m := picked(st)diff --git a/internal/redact/redact.go b/internal/redact/redact.goindex c7e149dd6..ba89d5a6f 100644--- a/internal/redact/redact.go+++ b/internal/redact/redact.go@@ -29,6 +29,7 @@ import ("sort""strings""sync"+ "unicode""unicode/utf8""golang.org/x/net/publicsuffix"@@ -42,7 +43,12 @@ type Options struct {// WordsAnyCase matches Words in any case: Nightjar masks NIGHTJAR and// nightjar too. Off, a word is matched as written.WordsAnyCase bool- Rules []Rule // the user's own rules, masked with the secrets+ // WordsWhole matches Words only as whole words (acmilan_ on Discord):+ // ABC masks "ABC IS" and 'ABC' but not XABCD, and 123456 masks no part+ // of 91234567, 0.123456 or 123456.78 (wholeWord). Off, a word is found+ // anywhere, inside a longer one too.+ WordsWhole bool+ Rules []Rule // the user's own rules, masked with the secrets// Kinds turns each of the Categories of personal data on or off by its// id, where the user chose; one not in it is as its Category.On says.// None of them is masked while Personal is off.@@ -678,6 +684,53 @@ func anyCase(w string) *regexp.Regexp {return re}+// wordRune says r is a letter, a digit or _ of a script that puts spaces+// between words. Chinese and Japanese don't, so a word of theirs is found+// inside their text, and a Latin word right beside a Chinese letter+// ("用ABC做") is still whole.+func wordRune(r rune) bool {+ if r == '_' {+ return true+ }+ if !unicode.IsLetter(r) && !unicode.IsDigit(r) {+ return false+ }+ return !unicode.In(r, unicode.Han, unicode.Hiragana, unicode.Katakana)+}++// wholeWord says s[a:b], a word the user asked to mask, is a word of its+// own in s: where its first or last letter is a wordRune, the letter+// beside it in s isn't one. A number is one with its decimals, so a digit+// at its edge also has no "." and digit beside it: 123456 is not whole in+// 0.123456 or 123456.78, while "v1." ends a sentence.+func wholeWord(s string, a, b int) bool {+ first, _ := utf8.DecodeRuneInString(s[a:b])+ last, _ := utf8.DecodeLastRuneInString(s[a:b])+ if wordRune(first) && a > 0 {+ r, n := utf8.DecodeLastRuneInString(s[:a])+ if wordRune(r) {+ return false+ }+ if unicode.IsDigit(first) && r == '.' {+ if p, _ := utf8.DecodeLastRuneInString(s[:a-n]); unicode.IsDigit(p) {+ return false+ }+ }+ }+ if wordRune(last) && b < len(s) {+ r, n := utf8.DecodeRuneInString(s[b:])+ if wordRune(r) {+ return false+ }+ if unicode.IsDigit(last) && r == '.' {+ if f, _ := utf8.DecodeRuneInString(s[b+n:]); unicode.IsDigit(f) {+ return false+ }+ }+ }+ return true+}+// Mask swaps what o covers in s for placeholders, and says how many.func Mask(s string, o Options) (string, int) {o.sent = true@@ -730,19 +783,33 @@ func mask(raw string, o Options, put func(kind, v string) string, escaped bool)if w = strings.TrimSpace(w); len(w) < 2 {continue}- if o.WordsAnyCase {- for _, m := range anyCase(w).FindAllStringIndex(s, -1) {- found = append(found, span{m[0], m[1], "TERM", ByWord})+ // next finds w at or after i; one inside a longer word, where the+ // user asked for whole words, is passed over a letter at a time, so+ // "XABC ABC" still finds the second+ next := func(i int) (int, int) {+ if o.WordsAnyCase {+ if m := anyCase(w).FindStringIndex(s[i:]); m != nil {+ return i + m[0], i + m[1]+ }+ return -1, -1}- continue+ if j := strings.Index(s[i:], w); j >= 0 {+ return i + j, i + j + len(w)+ }+ return -1, -1}- for i := 0; ; {- j := strings.Index(s[i:], w)- if j < 0 {+ for i := 0; i <= len(s); {+ a, b := next(i)+ if a < 0 {break}- found = append(found, span{i + j, i + j + len(w), "TERM", ByWord})- i += j + len(w)+ if o.WordsWhole && !wholeWord(s, a, b) {+ _, n := utf8.DecodeRuneInString(s[a:])+ i = a + max(n, 1)+ continue+ }+ found = append(found, span{a, b, "TERM", ByWord})+ i = max(b, a+1)}}all := rulesdiff --git a/internal/redact/wholeword_test.go b/internal/redact/wholeword_test.gonew file mode 100644index 000000000..2f9342a68--- /dev/null+++ b/internal/redact/wholeword_test.go@@ -0,0 +1,90 @@+package redact++import (+ "strings"+ "testing"+)++// Masked words as whole words only, when the user asks for it (acmilan_ on+// Discord), with the reporter's own examples: ABC masks "ABC IS xxx" and+// name='ABC' but no part of XABCD or ABCDEF, and 123456 masks a 123456 of+// its own but no part of 91234567, 0.123456 or 123456.78. Off, the+// default, a word is found inside longer ones as before. Each masked one+// comes back as it was written.+func TestWordsWhole(t *testing.T) {+ for _, c := range []struct {+ in string+ word string+ anyCase bool+ whole int // how many masked with whole words on+ anywhere int // and off+ keptWhole []string+ }{+ {"ABC IS xxx", "ABC", false, 1, 1, nil},+ {"name='ABC'", "ABC", false, 1, 1, nil},+ {"XABCD", "ABC", false, 0, 1, []string{"XABCD"}},+ {"ABCDEF", "ABC", false, 0, 1, []string{"ABCDEF"}},+ {"ABC_DEF and ABC1", "ABC", false, 0, 2, []string{"ABC_DEF", "ABC1"}},+ {"XABCD then ABC.", "ABC", false, 1, 2, []string{"XABCD then "}},+ {"id 123456 here", "123456", false, 1, 1, nil},+ {"91234567", "123456", false, 0, 1, []string{"91234567"}},+ {"0.123456", "123456", false, 0, 1, []string{"0.123456"}},+ {"123456.78", "123456", false, 0, 1, []string{"123456.78"}},+ // a full stop that ends a sentence is no decimal point+ {"the code is 123456.", "123456", false, 1, 1, nil},+ {"v.123456 and 123456,7", "123456", false, 2, 2, nil},+ // Chinese and Japanese put no spaces between words: a word of+ // theirs is found inside their text, and a Latin one beside it+ {"我的同事张三今天来了", "张三", false, 1, 1, nil},+ {"用ABC做的项目", "ABC", false, 1, 1, nil},+ {"これはABCです", "ABC", false, 1, 1, nil},+ {"用XABC做的", "ABC", false, 0, 1, []string{"XABC"}},+ // with any case too, and a word inside another is passed over to+ // the next whole one+ {"xabc then Abc!", "ABC", true, 1, 2, []string{"xabc then "}},+ {"Москва, москвабург", "москва", true, 1, 2, []string{"бург"}},+ } {+ for _, whole := range []bool{true, false} {+ o := Options{Words: []string{c.word}, WordsAnyCase: c.anyCase, WordsWhole: whole}+ out, n := Mask(c.in, o)+ want := c.anywhere+ if whole {+ want = c.whole+ }+ if n != want {+ t.Errorf("%q in %q, whole %v: %d masked, want %d: %q", c.word, c.in, whole, n, want, out)+ }+ if whole {+ for _, k := range c.keptWhole {+ if !strings.Contains(out, k) {+ t.Errorf("%q in %q: %q gone from %q", c.word, c.in, k, out)+ }+ }+ }+ if back := Restore(out, false); back != c.in {+ t.Errorf("%q in %q, whole %v: round trip %q → %q", c.word, c.in, whole, out, back)+ }+ }+ }+}++// In a request body, as the gateway masks it, a tool call's arguments+// included: the escapes around a word are read as the vendor reads them.+func TestWordsWholeJSON(t *testing.T) {+ o := Options{Words: []string{"ABC", "123456"}, WordsWhole: true}+ in := `{"messages":[{"role":"user","content":"name='ABC' XABCD 0.123456 pin 123456"},` ++ `{"role":"assistant","tool_calls":[{"type":"function","function":{"name":"f","arguments":"{\"q\":\"ABC\\nXABCD 123456.78\"}"}}]}]}`+ out, _ := MaskJSON([]byte(in), o)+ s := string(out)+ for _, k := range []string{"XABCD", "0.123456", "123456.78"} {+ if strings.Count(s, k) != map[string]int{"XABCD": 2, "0.123456": 1, "123456.78": 1}[k] {+ t.Errorf("%q not kept as written: %s", k, s)+ }+ }+ if strings.Contains(unmarked(s), "'ABC'") || strings.Contains(unmarked(s), "pin 123456") || strings.Contains(s, `\"ABC\\n`) {+ t.Errorf("a whole word went unmasked: %s", s)+ }+ if back := Restore(s, false); back == s || !strings.Contains(back, "name='ABC'") {+ t.Errorf("not put back: %s", back)+ }+}diff --git a/internal/settings/settings.go b/internal/settings/settings.goindex 80d9be241..3e39e8055 100644--- a/internal/settings/settings.go+++ b/internal/settings/settings.go@@ -121,8 +121,11 @@ type Settings struct {RedactWords []string `json:"redactWords,omitempty"`// RedactWordsAnyCase matches RedactWords in any case (ZoeCats on// Discord); off, the default, a word is matched as written.- RedactWordsAnyCase bool `json:"redactWordsAnyCase,omitempty"`- RedactRules []redact.Rule `json:"redactRules,omitempty"`+ RedactWordsAnyCase bool `json:"redactWordsAnyCase,omitempty"`+ // RedactWordsWhole matches RedactWords only as whole words (acmilan_+ // on Discord); off, the default, a word is found inside longer ones too.+ RedactWordsWhole bool `json:"redactWordsWhole,omitempty"`+ RedactRules []redact.Rule `json:"redactRules,omitempty"`// RedactActions is what is done with a value of a kind (redact.Hit's// Kind: API_KEY, EMAIL, a rule's own…) other than masking it, where// the user chose: redact.Warn sends it as written and records it,