Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
redact: a key can go as a made-up key of its own form, not a placeholder (Charles Cao on X)
yetonebea9f905d095
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.mdindex a75fa1982..735d46dde 100644--- a/docs/subsystems/gateway-routing.md+++ b/docs/subsystems/gateway-routing.md@@ -92,6 +92,7 @@ recorded as a prediction failure (`noPrediction`).- With *Mask personal data* on, each kind in `redact.Categories` ([`internal/redact/more.go`](../../internal/redact/more.go)) is masked unless the user turned it off (`redactKinds` in settings, `Options.Kinds`). On by default: US SSNs, passport numbers, IBANs that pass mod 97, birthdays next to a word that says so, MAC addresses, and serial numbers next to S/N or the like. Off by default: the user name in a home folder, user@host in a prompt or ssh command, public IPv4 and IPv6 addresses (never private, loopback, link-local, CGNAT or documentation ranges) and cloud bucket names, since agents send them in nearly every request and the model must copy the placeholder exactly inside a path or URL. Context rules run only near their words (`rule.matches`). What reads as an address but ends in a file's extension that no domain has (`…@2x.jpg`, `[email protected]`; `fileExt` in `realEmail`), or a scale before an extension that is a domain too (`[email protected]`, `[email protected]`; `scaleExt`), is a file's name, for the rule and the local model alike (#3029). `[email protected]` is still an address. The Settings rows come from `REDACT_KINDS` in app.js, which `TestRedactKindsAreRedacts` holds to the same ids and defaults.- With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken, nor a snake_case word as a name (Codex's `sequential_cutoff`, #3144), nor as a secret what names a thing: a UUID, a digest with its algorithm's name, a file's path (`namesThing`; the model takes all three for secrets). The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with it on a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left. The model is OpenAI's Privacy Filter as a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, from hf-mirror first in China, `source.chinaFirst`) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU is started on the CPU for the rest of the run), joins short texts into 12 KB chunks and maps the spans back, keeps the last 20000 answers by hash, and ends it after 10 minutes with no request ([`internal/pii`](../../internal/pii)). A model that is on and can't answer (not on disk, didn't start, hung) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).- With *Mask secrets* on, the user's own masking rules (`redactRules`, `redact.Rule`: a placeholder name and a prefix or an RE2 pattern, [`internal/redact/rules.go`](../../internal/redact/rules.go)) run before magpie's. Settings → Privacy adds them, removes them, and edits one in place (`POST /api/settings/redact-rule` with the rule's index and its current value, `editRedactRule` in [`internal/gui/api.go`](../../internal/gui/api.go)). Every rule is checked by `redact.CheckRules` when settings are saved; an edit that doesn't compile, matches the empty string, repeats another rule or names a rule no longer at that index is refused with the rules left as they were. The masker reads the rules from settings on each call (`options` in [`internal/masking`](../../internal/masking/masking.go)), so a change applies to the next request.+ - With *Keep a key's form* on (`settings.RedactShape`, `Options.Shape`, under *Mask secrets*; Charles Cao on X), a secret of 16 characters or more, all letters, digits, `-`, `_` or `.`, goes as a made-up key of its form instead of a placeholder: its prefix kept (`sk-proj-`, `ghp_`, `AKIA`, up to four letters and their separator; `keyHead`), the same length and separators, and each other character a random one of its kind (a hex key's any hex digit of its case), drawn from an HMAC of the value under the placeholder key, so one key goes as the same made-up one every turn and a vendor's cache still matches (`shapeOf`, `shaped` in [`internal/redact/shape.go`](../../internal/redact/shape.go)). Personal data, the user's words and anything shorter or with other characters go as placeholders still. A made-up key is kept in the masker's memory with the placeholders, so it is recorded, re-masked in what a vendor sealed, and put back wherever the reply has it, before the placeholders (`restoreShapes`); a stream holds the end of an event that may be the start of one (`shapeTail`, after the last whole one), and a made-up key the vendor sends back unrestored isn't masked again (`outsideShapes`). `TestShapeKeepsForm`, `TestShapeJSON`, `TestShapeStream`.- What is done with a value found is the user's per kind (#1977, `redactActions` in settings, `Options.Actions`, set by `POST /api/settings/redact-actions`; a kind left out is masked). `redact.Warn` sends it as written: the masker tells it, with a placeholder no one remembers, and it isn't counted as masked. `redact.Block` masks it as any other, and the gateway holds the request back with a 400 that names the kind and the value's hint, never the value (`blocked`, `maskBlocked` in [`redact.go`](../../internal/gateway/redact.go)); nothing is sent and the record is kept as `Blocked` (`noteBlocked`). A value on the *Never mask* list (`redactAllow`, `Options.Allow`, `POST /api/settings/redact-allow`) goes as written when it is the whole value found, in any case, and isn't told; a longer value it is part of is masked. Settings → Privacy's *Try it* posts a text to `POST /api/masking/preview` ([`masking.go`](../../internal/gui/masking.go)), which masks it as a request would be (`redact.Preview`) and remembers nothing: its placeholders stand for nothing afterwards, its values aren't masked later as seen before, and it returns only the masked text and each hit's kind, finder, action, placeholder and hint. The page keeps the text in no storage.- Each request that had anything masked is recorded as it goes to the vendor (`noteMasked` in [`masked.go`](../../internal/gateway/masked.go), from chat, the Codex backend, search, images, video, System One, embeddings and audio; count_tokens is masked and not recorded). The masker says what it swapped through `gwmw.Teller` (`JSONHits`, `TextHits`: each placeholder's kind, its finder, `redact.ByRule`, `ByYours`, `ByWord`, `ByModel` or `ByKnown`, and a hint of its first characters and its length), and [`internal/maskrec`](../../internal/maskrec/maskrec.go) keeps one line per request in a file per day under `~/.config/magpie/masking`, 30 days and 100 MB in all: the agent, session, provider, model, and each placeholder with its kind, finder, action and hint, never the value. The text around each placeholder as the vendor saw it is kept in memory only, for the last 400 records, and what a placeholder stood for only in the masker's own memory (`Teller.Value`); after a restart neither is there. A recent call (`Call.Masked`) and the route trace (`Route.Masked`) carry the counts and the record's key, which the Gateway and Routing pages' badge opens in Settings → Privacy (`/api/masking/*`, [`internal/gui/masking.go`](../../internal/gui/masking.go)). A masker without `Teller`, a plugin's, masks as before and records nothing. Under a test binary testenv isolated nothing is written (`maskrec.inTest`).- `DecidesModel` gives a 400.diff --git a/internal/gui/api.go b/internal/gui/api.goindex 70f603df5..94443b630 100644--- a/internal/gui/api.go+++ b/internal/gui/api.go@@ -1226,6 +1226,10 @@ func Handler(w Windows, gw *gateway.Server) http.Handler {if _, sent := fields["redactWordsAnyCase"]; !sent {in.RedactWordsAnyCase = cur.RedactWordsAnyCase}+ // nor whether a key goes as a made-up one of its form+ if _, sent := fields["redactShape"]; !sent {+ in.RedactShape = cur.RedactShape+ }// nor how long an image is given to be drawnif _, sent := fields["imageTimeout"]; !sent {in.ImageTimeout = cur.ImageTimeoutdiff --git a/internal/gui/assets/app.js b/internal/gui/assets/app.jsindex d3092d575..79379b0d6 100644--- a/internal/gui/assets/app.js+++ b/internal/gui/assets/app.js@@ -21933,6 +21933,8 @@ function renderRedact(s, keep) {const onOff = (on, fn) => segs([["off", t("Off")], ["on", t("On")]], on ? "on" : "off", (v) => fn(v === "on"));row(t("Mask secrets"), t("API keys, private keys, tokens and passwords go to vendors as placeholders, and come back as they were"),onOff(s.redact, (redact) => savePrefs({ ...keep, redact })));+ if (s.redact) row(t("Keep a key's form"), t("A key goes as a made-up one of the same form (sk_… stays sk_…), not a placeholder, and comes back as it was"),+ onOff(s.redactShape, (redactShape) => savePrefs({ ...keep, redactShape }))).id = "redactShapeRow";row(t("Mask personal data"), t("Emails, phone numbers, ID and bank card numbers too"),onOff(s.redactPersonal, (redactPersonal) => {// turned on, the accounts on screen are hidden too@@ -23125,7 +23127,7 @@ function prefsKeep(s) {uiFont: s.uiFont || null, codeFont: s.codeFont || null,otel: s.otel || {},trayUsages: s.trayUsages || [],- redact: !!s.redact, redactPersonal: !!s.redactPersonal, redactKinds: s.redactKinds || {}, redactWords: s.redactWords || [], redactWordsAnyCase: !!s.redactWordsAnyCase, codexWarmup: s.codexWarmup || "",+ redact: !!s.redact, redactPersonal: !!s.redactPersonal, redactKinds: s.redactKinds || {}, redactWords: s.redactWords || [], redactWordsAnyCase: !!s.redactWordsAnyCase, redactShape: !!s.redactShape, codexWarmup: s.codexWarmup || "",claudeWarmup: s.claudeWarmup || "", codexWarmAt: s.codexWarmAt || "", claudeWarmAt: s.claudeWarmAt || "",codexWarmAts: warmTimes(s.codexWarmAts, s.codexWarmAt), claudeWarmAts: warmTimes(s.claudeWarmAts, s.claudeWarmAt), workbuddyCheckin: !!s.workbuddyCheckin, traeCheckin: !!s.traeCheckin, minimaxCheckin: !!s.minimaxCheckin, qoderCheckin: !!s.qoderCheckin, noStats: !!s.noStats, noUsageStats: !!s.noUsageStats,memberModel: !!s.memberModel, noLoopGuard: !!s.noLoopGuard, claudeLeanPrompt: !!s.claudeLeanPrompt, refusalRetries: s.refusalRetries ?? null,diff --git a/internal/gui/assets/i18n.js b/internal/gui/assets/i18n.jsindex 76a9904ce..da9eff0db 100644--- a/internal/gui/assets/i18n.js+++ b/internal/gui/assets/i18n.js@@ -943,6 +943,8 @@ const I18N = {"Environment variables override these saved OTLP preferences": "环境变量会覆盖这里保存的 OTLP 设置","Privacy": "隐私","Mask secrets": "脱敏密钥",+ "Keep a key's form": "保留密钥格式",+ "A key goes as a made-up one of the same form (sk_… stays sk_…), not a placeholder, and comes back as it was": "密钥以同样格式的虚构密钥发出(sk_… 仍是 sk_…),而非占位符,回来时还原","Local network": "局域网","Public address": "公网地址","MAGPIE_PUBLIC_URL: where other computers reach the gateway, with a gateway key": "MAGPIE_PUBLIC_URL:其他电脑访问网关的地址,需带网关密钥",@@ -4910,6 +4912,8 @@ const I18N = {"Environment variables override these saved OTLP preferences": "環境變數會覆蓋這裡儲存的 OTLP 設定","Privacy": "隱私","Mask secrets": "脫敏金鑰",+ "Keep a key's form": "保留金鑰格式",+ "A key goes as a made-up one of the same form (sk_… stays sk_…), not a placeholder, and comes back as it was": "金鑰以同樣格式的虛構金鑰送出(sk_… 仍是 sk_…),而非佔位符,回來時還原","Local network": "區域網","Public address": "公開位址","MAGPIE_PUBLIC_URL: where other computers reach the gateway, with a gateway key": "MAGPIE_PUBLIC_URL:其他電腦連到閘道的位址,需帶閘道金鑰",@@ -8849,6 +8853,8 @@ const I18N = {"Environment variables override these saved OTLP preferences": "環境変数は、ここに保存された OTLP の設定より優先されます","Privacy": "プライバシー","Mask secrets": "シークレットをマスク",+ "Keep a key's form": "キーの形式を保つ",+ "A key goes as a made-up one of the same form (sk_… stays sk_…), not a placeholder, and comes back as it was": "キーはプレースホルダーではなく同じ形式の架空のキー(sk_… は sk_… のまま)として送られ、戻るときに元に戻ります","Local network": "ローカルネットワーク","Public address": "公開アドレス","MAGPIE_PUBLIC_URL: where other computers reach the gateway, with a gateway key": "MAGPIE_PUBLIC_URL:他のコンピュータがゲートウェイに接続するアドレス(ゲートウェイキーが必要)",@@ -12802,6 +12808,8 @@ const I18N = {"Environment variables override these saved OTLP preferences": "Umgebungsvariablen überschreiben diese gespeicherten OTLP-Einstellungen","Privacy": "Datenschutz","Mask secrets": "Geheimnisse maskieren",+ "Keep a key's form": "Schlüsselformat beibehalten",+ "A key goes as a made-up one of the same form (sk_… stays sk_…), not a placeholder, and comes back as it was": "Ein Schlüssel geht als erfundener Schlüssel derselben Form (sk_… bleibt sk_…) statt als Platzhalter und kommt unverändert zurück","Local network": "Lokales Netzwerk","Public address": "Öffentliche Adresse","MAGPIE_PUBLIC_URL: where other computers reach the gateway, with a gateway key": "MAGPIE_PUBLIC_URL: wo andere Computer das Gateway erreichen, mit einem Gateway-Schlüssel",diff --git a/internal/gui/core_budget_test.go b/internal/gui/core_budget_test.goindex 0ed6dc238..04fc031e1 100644--- a/internal/gui/core_budget_test.go+++ b/internal/gui/core_budget_test.go@@ -15,8 +15,8 @@ import (// so a page can't quietly grow back into the core; a change that needs more// raises them and says why in its commit.const (- appScriptBudget = 1_321_994 // app.js: 1,198,239 after Usage left (P2); 1,218,098 with the 10-15 fixes to pages still in the core; 1,232,466 with the provider editor's Advanced fold and every dialog holding its place, the core's own (10-16); 1,250,533 with the quota refresh kept through redraws, the core's own since the tray panel draws it too, plus 1.5% (#2397); 1,269,083 with Free first's option in the provider editor's ROUTINGS, which the Routing page reads too, plus 1.5%; 1,272,726 with the Providers page drawing its last list at once (#1519), the core's page, and the Routing trace's account names read without Object.entries (#1481), plus 1.5%; a dialog growing out of the row that opened it and folding back into it, the core's own since every page's dialogs open through it (10-20); Codex's and Claude Code's own settings moved from Settings into the agent's row on the Agents page, the core's own since the tray panel and the window both draw them (10-20); 1,314,432 with Settings › Privacy's record of what was masked (its summary, the records view and the local model's controls), which belongs to Settings, the core's own, plus 1.5%; 1,336,481 with a provider's Names & levels saying Provider in model names is Settings' and opening it there (joe), and Settings' Image recognition saying where a custom provider's model is marked (nexnull), both in the core's own pages, with no room added (10-23); 1,337,633 with Settings › Models › Replies' Intent classifier timeout (#2934, reyoung), Settings' own row, drawn by waitRow, which the Image generation timeout's row now shares, with no room added; 1,337,735 with the provider editor's link naming a partner's page by its own host (siteHost) instead of usemagpie.ai's /go link, the core's own editor, with no room added (10-23); 1,338,084 with Check in now saying what the press came to (claimed +credits, already today, not available yet) where it said "refreshed" (#3326), the check-in row the tray panel draws too, the core's own, with no room added (10-23); 1,338,929 with a new partner's tile keeping a small dot until it is opened, in the add sheet, the core's own, with no room added (10-23); 1,343,324 with the sidebar's open-page row marked by one shape that moves on a spring (navTopGo), the window's own shell, which every page sits beside, with no room added (10-23); 1,343,465 with a dialog kept centred while its content comes in, until it is pressed (holdModal, every page's dialogs), the core's own, with no room added (10-24); 1,317,486 with what masking did moved out to Insights › Masking (plugins/redact), the budget lowered to it (10-24); 1,321,336 with Codex's one subagent model, its card's square saying plainly what each choice does and a row with a name in its sheet, where two squares read alike (the owner), the Agents page's, which the tray panel draws too, the core's own, with no room added (10-24); 1,321,424 with Settings › Models' Image generation MCP server row beside Image recognition's (JonahFu on X), the two drawn by one helper (mcpRow), Settings' own, with no room added (10-25); 1,321,994 with a provider of the user's set never to rest for a limit (zoecats), the provider editor's Advanced row, the core's own editor, with no room added (10-25)- coreScriptBudget = 1_529_542 // every core script: 1,358,034 after Usage left; 1,380,003 with them; 1,405,937 with the fold; 1,426,102 with 10-16's core work (the GitHub list loading, the move to plugins, Settings › Long conversations), plus 1.5% (#2612); 1,447,368 with loadBack, the window's one state read per return, which every page needs and so is the core's own (1,446,845 before it, 155 bytes under), plus 1.5%; OpenCode Zen's card saying OpenCode tells no key its balance adds ~390 bytes inside it; 1,472,589 with the dialog that grows out of its row and the agents' own settings in their rows, on main's 10-20 core work (the Providers page's last list, the Routing trace's names), plus 1.5% (10-20); 1,506,192 with Settings › Privacy's record of what was masked, plus 1.5%; 1,529,393 with the Intent classifier timeout row (#2934), with no room added; 1,529,495 with the editor's partner link host, with no room added; 1,529,844 with Check in now's press summary (#3326), with no room added; 1,530,689 with the new partner tile's dot, with no room added; 1,535,084 with the sidebar's spring, with no room added; 1,549,478 with the context window card's map (tiles by tokens, the default look, the block grid a switch away) and its text view showing an inline image, in context.js, which the Routing page draws as well as Usage, so the core's own, with no room added (10-24); 1,550,768 with a usage card's refresh opening after its text (#840's card, which the tray panel draws too), the Add key spinner, and a dialog kept centred while its content comes in, with no room added (10-24); 1,524,789 with what masking did moved out to Insights › Masking, lowered to it; 1,528,525 with Codex's one subagent model, with no room added (10-24); 1,528,972 with the Image generation MCP server row (JonahFu on X), with no room added (10-25); 1,529,542 with the limit-rest row (zoecats), with no room added (10-25)+ appScriptBudget = 1_322_289 // app.js: 1,198,239 after Usage left (P2); 1,218,098 with the 10-15 fixes to pages still in the core; 1,232,466 with the provider editor's Advanced fold and every dialog holding its place, the core's own (10-16); 1,250,533 with the quota refresh kept through redraws, the core's own since the tray panel draws it too, plus 1.5% (#2397); 1,269,083 with Free first's option in the provider editor's ROUTINGS, which the Routing page reads too, plus 1.5%; 1,272,726 with the Providers page drawing its last list at once (#1519), the core's page, and the Routing trace's account names read without Object.entries (#1481), plus 1.5%; a dialog growing out of the row that opened it and folding back into it, the core's own since every page's dialogs open through it (10-20); Codex's and Claude Code's own settings moved from Settings into the agent's row on the Agents page, the core's own since the tray panel and the window both draw them (10-20); 1,314,432 with Settings › Privacy's record of what was masked (its summary, the records view and the local model's controls), which belongs to Settings, the core's own, plus 1.5%; 1,336,481 with a provider's Names & levels saying Provider in model names is Settings' and opening it there (joe), and Settings' Image recognition saying where a custom provider's model is marked (nexnull), both in the core's own pages, with no room added (10-23); 1,337,633 with Settings › Models › Replies' Intent classifier timeout (#2934, reyoung), Settings' own row, drawn by waitRow, which the Image generation timeout's row now shares, with no room added; 1,337,735 with the provider editor's link naming a partner's page by its own host (siteHost) instead of usemagpie.ai's /go link, the core's own editor, with no room added (10-23); 1,338,084 with Check in now saying what the press came to (claimed +credits, already today, not available yet) where it said "refreshed" (#3326), the check-in row the tray panel draws too, the core's own, with no room added (10-23); 1,338,929 with a new partner's tile keeping a small dot until it is opened, in the add sheet, the core's own, with no room added (10-23); 1,343,324 with the sidebar's open-page row marked by one shape that moves on a spring (navTopGo), the window's own shell, which every page sits beside, with no room added (10-23); 1,343,465 with a dialog kept centred while its content comes in, until it is pressed (holdModal, every page's dialogs), the core's own, with no room added (10-24); 1,317,486 with what masking did moved out to Insights › Masking (plugins/redact), the budget lowered to it (10-24); 1,321,336 with Codex's one subagent model, its card's square saying plainly what each choice does and a row with a name in its sheet, where two squares read alike (the owner), the Agents page's, which the tray panel draws too, the core's own, with no room added (10-24); 1,321,424 with Settings › Models' Image generation MCP server row beside Image recognition's (JonahFu on X), the two drawn by one helper (mcpRow), Settings' own, with no room added (10-25); 1,321,994 with a provider of the user's set never to rest for a limit (zoecats), the provider editor's Advanced row, the core's own editor, with no room added (10-25); 1,322,289 with Settings › Privacy's Keep a key's form under Mask secrets (Charles Cao on X), Settings' own row, with no room added (10-25)+ coreScriptBudget = 1_529_837 // every core script: 1,358,034 after Usage left; 1,380,003 with them; 1,405,937 with the fold; 1,426,102 with 10-16's core work (the GitHub list loading, the move to plugins, Settings › Long conversations), plus 1.5% (#2612); 1,447,368 with loadBack, the window's one state read per return, which every page needs and so is the core's own (1,446,845 before it, 155 bytes under), plus 1.5%; OpenCode Zen's card saying OpenCode tells no key its balance adds ~390 bytes inside it; 1,472,589 with the dialog that grows out of its row and the agents' own settings in their rows, on main's 10-20 core work (the Providers page's last list, the Routing trace's names), plus 1.5% (10-20); 1,506,192 with Settings › Privacy's record of what was masked, plus 1.5%; 1,529,393 with the Intent classifier timeout row (#2934), with no room added; 1,529,495 with the editor's partner link host, with no room added; 1,529,844 with Check in now's press summary (#3326), with no room added; 1,530,689 with the new partner tile's dot, with no room added; 1,535,084 with the sidebar's spring, with no room added; 1,549,478 with the context window card's map (tiles by tokens, the default look, the block grid a switch away) and its text view showing an inline image, in context.js, which the Routing page draws as well as Usage, so the core's own, with no room added (10-24); 1,550,768 with a usage card's refresh opening after its text (#840's card, which the tray panel draws too), the Add key spinner, and a dialog kept centred while its content comes in, with no room added (10-24); 1,524,789 with what masking did moved out to Insights › Masking, lowered to it; 1,528,525 with Codex's one subagent model, with no room added (10-24); 1,528,972 with the Image generation MCP server row (JonahFu on X), with no room added (10-25); 1,529,542 with the limit-rest row (zoecats), with no room added (10-25); 1,529,837 with Keep a key's form (Charles Cao on X), with no room added (10-25))func TestCoreScriptBudget(t *testing.T) {diff --git a/internal/gui/tests/redact-shape.test.cjs b/internal/gui/tests/redact-shape.test.cjsnew file mode 100644index 000000000..8433705a2--- /dev/null+++ b/internal/gui/tests/redact-shape.test.cjs@@ -0,0 +1,119 @@+// Run with Node's test runner and Playwright on the module path; see README.md.+// Settings › Privacy's "Keep a key's form" (Charles Cao on X): a row right+// under Mask secrets while secrets are masked, none while they aren't, Off+// unless the user turned it on, so a key goes as a placeholder as before.+// A click on On saves redactShape with the rest of the settings as they+// were, and moves nothing; a later save from the page keeps it. In every+// language, Chromium and WebKit, in a wide window and a narrow one.+const assert = require("node:assert/strict");+const fs = require("node:fs/promises");+const path = require("node:path");+const { test } = require("node:test");+const { chromium, webkit } = require("playwright");+const { reach, watchSettingsReads, settingsDrawn } = require("./reader.cjs");++const assets = path.resolve(__dirname, "../assets");+const words = {+ en: { name: "Keep a key's form", on: "On", off: "Off" },+ zh: { name: "保留密钥格式", on: "开启", off: "关闭" },+ "zh-TW": { name: "保留金鑰格式", on: "開啟", off: "關閉" },+ ja: { name: "キーの形式を保つ", on: "オン", off: "オフ" },+ de: { name: "Schlüsselformat beibehalten", on: "An", off: "Aus" },+};++function serve(lang, saved, redact = true) {+ let settings = { lang, theme: "light", redact, redactPersonal: false, redactWords: ["Nightjar"] };+ const state = { agents: [], profiles: [], settings };+ return async (r) => {+ const url = new URL(r.request().url());+ const json = (data) => r.fulfill({ json: data });+ if (url.pathname === "/boot.js") return r.fulfill({ contentType: "text/javascript", body: `window.bootPrefs = {lang:"${lang}",theme:"light",web:true};` });+ if (url.pathname === "/wails/runtime.js") return r.fulfill({ contentType: "text/javascript", body: "export const Window = {};" });+ if (url.pathname === "/api/state") return json(state);+ if (url.pathname === "/api/settings") {+ if (r.request().method() === "POST") {+ settings = JSON.parse(r.request().postData());+ saved.push(settings);+ }+ return json(settings);+ }+ if (url.pathname === "/api/groups") return json({ models: [], groups: [], pools: [] });+ if (url.pathname === "/api/providers") return json({ providers: [], presets: [], excluded: [], gateway: { running: true, window: true, url: "http://127.0.0.1:3999" } });+ if (url.pathname.startsWith("/api/")) return json({});+ const file = path.join(assets, url.pathname === "/" ? "index.html" : url.pathname);+ const contentType = { ".html": "text/html", ".js": "text/javascript", ".css": "text/css", ".svg": "image/svg+xml", ".png": "image/png" }[path.extname(file)];+ await r.fulfill({ body: await fs.readFile(file), contentType });+ };+}++async function open(engine, lang, width, saved, redact) {+ const browser = await (engine === "webkit" ? webkit.launch() : chromium.launch({ channel: "chromium" }));+ const page = await (await browser.newContext({ viewport: { width, height: 1400 } })).newPage();+ page.setDefaultTimeout(5000);+ const errors = [];+ page.on("pageerror", (e) => errors.push(e.message));+ await page.route("**/*", serve(lang, saved, redact));+ await watchSettingsReads(page);+ await page.goto("http://magpie.test/?view=settings&tab=privacy");+ await page.locator("#redactList .row").first().waitFor();+ await settingsDrawn(page);+ return { browser, page, errors };+}++for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium", "webkit"])) {+ for (const [lang, w] of Object.entries(words)) {+ for (const width of [1100, 640]) {+ test(`${engine} ${lang} ${width}px: a key keeps its form only when turned on`, async (t) => {+ const saved = [];+ const { browser, page, errors } = await open(engine, lang, width, saved, true);+ t.after(() => browser.close());+ const row = page.locator("#redactShapeRow");+ await reach(page, row);+ const m = await row.evaluate((r) => {+ const sub = r.querySelector(".sub"), list = document.querySelector("#redactList").getBoundingClientRect(), b = r.getBoundingClientRect();+ return {+ name: r.querySelector(".name").textContent, sub: sub.textContent,+ on: r.querySelector(".segs .opt.on")?.textContent,+ first: r.previousElementSibling?.querySelector(".name")?.textContent === t("Mask secrets"),+ cut: sub.scrollWidth > sub.clientWidth + 1 || sub.scrollHeight > sub.clientHeight + 1,+ inside: b.left >= list.left - 1 && b.right <= list.right + 1,+ wide: document.documentElement.scrollWidth > document.documentElement.clientWidth,+ };+ });+ assert.equal(m.name, w.name);+ assert.equal(m.on, w.off, "off unless the user turned it on");+ assert(m.first, "right under Mask secrets");+ assert(!m.cut && m.inside && !m.wide, "shown in full, inside the page");+ if (lang !== "en") assert(!m.sub.includes("made-up"), "its words translated");++ const before = await page.evaluate(() => [scrollX, scrollY, document.querySelector("#view-settings").scrollTop]);+ const b = await row.locator(".segs .opt", { hasText: w.on }).boundingBox();+ await page.mouse.click(b.x + b.width / 2, b.y + b.height / 2);+ await page.waitForFunction(() => document.querySelector("#redactShapeRow .segs .opt.on")?.textContent === t("On"));+ await new Promise((r) => setTimeout(r, 300));+ assert.equal(saved.length, 1, "one save");+ assert.equal(saved[0].redactShape, true);+ assert.equal(saved[0].redact, true);+ assert.deepEqual(saved[0].redactWords, ["Nightjar"]);+ assert.deepEqual(await page.evaluate(() => [scrollX, scrollY, document.querySelector("#view-settings").scrollTop]), before, "the click moved nothing");++ // another choice on the page keeps it: Mask personal data+ const p = page.locator("#redactShapeRow + .row").locator(".segs .opt", { hasText: w.on });+ await reach(page, p);+ await p.click();+ await page.waitForFunction(() => document.querySelector("#redactShapeRow + .row .segs .opt.on")?.textContent === t("On"));+ await new Promise((r) => setTimeout(r, 300));+ assert.equal(saved.length, 2);+ assert.equal(saved[1].redactPersonal, true);+ assert.equal(saved[1].redactShape, true, "a later save keeps it");+ assert.deepEqual(errors, []);+ });+ }+ }+ test(`${engine}: no key's form while secrets aren't masked`, async (t) => {+ const { browser, page, errors } = await open(engine, "en", 1100, [], false);+ t.after(() => browser.close());+ assert.equal(await page.locator("#redactShapeRow").count(), 0);+ assert.deepEqual(errors, []);+ });+}diff --git a/internal/gui/tests/redact-words-any-case.test.cjs b/internal/gui/tests/redact-words-any-case.test.cjsindex 8938d7e72..8d1c7e80c 100644--- a/internal/gui/tests/redact-words-any-case.test.cjs+++ b/internal/gui/tests/redact-words-any-case.test.cjs@@ -95,11 +95,12 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",assert.equal(saved[0].redact, true);assert.deepEqual(await page.evaluate(() => [scrollX, scrollY, document.querySelector("#view-settings").scrollTop]), before, "the click moved nothing");- // another choice on the page keeps it- const p = page.locator("#redactList .row").nth(1).locator(".segs .opt", { hasText: w.on });+ // another choice on the page keeps it: Mask personal data, under+ // the key's form+ const p = page.locator("#redactShapeRow + .row").locator(".segs .opt", { hasText: w.on });await reach(page, p); // above it now, maybe under the tabsawait p.click();- await page.waitForFunction(() => document.querySelector("#redactList .row:nth-child(2) .segs .opt.on")?.textContent === t("On"));+ await page.waitForFunction(() => document.querySelector("#redactShapeRow + .row .segs .opt.on")?.textContent === t("On"));await new Promise((r) => setTimeout(r, 300));assert.equal(saved.length, 2);assert.equal(saved[1].redactPersonal, true);diff --git a/internal/masking/masking.go b/internal/masking/masking.goindex e16af2196..af1d1f0ca 100644--- a/internal/masking/masking.go+++ b/internal/masking/masking.go@@ -24,7 +24,7 @@ func Wanted(st settings.Settings) bool {func options() redact.Options {st := settings.Shared()o := redact.Options{Secrets: st.Redact, Personal: st.RedactPersonal, Kinds: st.RedactKinds, Words: st.RedactWords, WordsAnyCase: st.RedactWordsAnyCase, Rules: st.RedactRules,- Actions: st.RedactActions, Allow: st.RedactAllow}+ Actions: st.RedactActions, Allow: st.RedactAllow, Shape: st.RedactShape}if st.RedactModel {o.Model = model}diff --git a/internal/redact/json.go b/internal/redact/json.goindex 09671f124..e7e2a25f4 100644--- a/internal/redact/json.go+++ b/internal/redact/json.go@@ -401,7 +401,7 @@ func MaskJSON(body []byte, o Options) ([]byte, int) {total := 0var seen []string // value, placeholder, value, placeholder…put := func(kind, v string) string {- p := placeholder(kind, v)+ p := o.put()(kind, v)seen = append(seen, v, p)return p}@@ -440,7 +440,7 @@ func MaskJSON(body []byte, o Options) ([]byte, int) {// number or an address the vendor wrote of its own can be one// masked in another conversation (13800138000 is in many), and// masked, its text would no longer match its signature- t, n := apply(t, sentKnownSpans(t, Options{Secrets: o.Secrets}), placeholder, o)+ t, n := apply(t, sentKnownSpans(t, Options{Secrets: o.Secrets}), o.put(), o)total += nreturn t})@@ -479,7 +479,7 @@ func RestoreJSON(body []byte) []byte { return restoreJSON(body, nil) }// restoreJSON is RestoreJSON, counting in put as restore does.func restoreJSON(body []byte, put map[string]int) []byte {- if !bytes.Contains(body, []byte("{{")) {+ if !bytes.Contains(body, []byte("{{")) && !hasShapes(body) {return body}typ := ""diff --git a/internal/redact/known.go b/internal/redact/known.goindex 1d6b3ddbc..1e50af107 100644--- a/internal/redact/known.go+++ b/internal/redact/known.go@@ -84,8 +84,12 @@ func reindexKnown() {}}-// kindOf is the kind in a placeholder: SECRET of {{SECRET_abcdefgh}}.+// kindOf is the kind in a placeholder: SECRET of {{SECRET_abcdefgh}}, or+// a made-up key's (shape.go); called with mu held.func kindOf(p string) string {+ if k, ok := shapes[p]; ok {+ return k+ }if i := strings.LastIndexByte(p, '_'); i > 2 {return p[2:i]}diff --git a/internal/redact/redact.go b/internal/redact/redact.goindex c0d18010d..43b63388f 100644--- a/internal/redact/redact.go+++ b/internal/redact/redact.go@@ -47,6 +47,9 @@ type Options struct {Kinds map[string]bool// Model, where set, finds more in the texts to mask (model.go)Model Model+ // Shape sends a key as a made-up key of its form, not a placeholder+ // (shape.go)+ Shape bool// Actions is what is done with a value of a kind (a Hit's Kind) other// than masking it, where the user chose: Warn sends it as written,@@ -491,16 +494,7 @@ func placeholder(kind, v string) string {p := tag(kind, v)mu.Lock()if _, ok := values[p]; !ok {- if len(values) >= maxValues {- n := 0- for k := range values {- delete(values, k)- if n++; n >= maxValues/2 {- break- }- }- reindexKnown()- }+ makeRoom()values[p] = vkinds[kind] = trueaddKnown(p, v)@@ -509,6 +503,32 @@ func placeholder(kind, v string) string {return p}+// makeRoom lets the oldest half of values go when they are full; called+// with mu held.+func makeRoom() {+ if len(values) < maxValues {+ return+ }+ n := 0+ for k := range values {+ delete(values, k)+ delete(shapes, k)+ if n++; n >= maxValues/2 {+ break+ }+ }+ reindexKnown()+}++// put is how o sends a value: as its placeholder, or as a made-up key of+// its form (Shape).+func (o Options) put() func(kind, v string) string {+ if o.Shape {+ return shaped+ }+ return placeholder+}+// ---- masking --------------------------------------------------------------------type span struct {@@ -598,7 +618,7 @@ func anyCase(w string) *regexp.Regexp {// Mask swaps what o covers in s for placeholders, and says how many.func Mask(s string, o Options) (string, int) {o.sent = true- return mask(s, withModel(o, []string{s}), placeholder, false)+ return mask(s, withModel(o, []string{s}), o.put(), false)}// Preview is Mask with nothing remembered: each value goes as the@@ -953,7 +973,8 @@ func apply(s string, found []span, put func(kind, v string) string, o Options) (if len(found) == 0 {return s, 0}- // nothing inside a placeholder already there+ // nothing inside a placeholder already there, or a made-up key+ found = outsideShapes(s, found)if strings.Contains(s, "{{") {for _, p := range placeholderRe.FindAllStringIndex(s, -1) {found = slices.DeleteFunc(found, func(f span) bool { return f.start < p[1] && f.end > p[0] })@@ -1045,14 +1066,15 @@ func Restore(s string, escaped bool) string { return restore(s, escaped, nil) }// restore is Restore, counting in put, when it isn't nil, each// placeholder it put a value back for.func restore(s string, escaped bool, put map[string]int) string {- if !strings.Contains(s, "_") {- return s- }mu.RLock()defer mu.RUnlock()if len(values) == 0 {return s}+ s = restoreShapes(s, put)+ if !strings.Contains(s, "_") {+ return s+ }var b strings.Builderlast := 0for _, m := range placeholderBody.FindAllStringIndex(s, -1) {@@ -1139,7 +1161,7 @@ func partialTail(s string) int {if n := braceTail(s); n > 0 {return n}- return wordTail(s)+ return max(wordTail(s), shapeTail(s))}// braceTail is a tail from {, or {{, on: then spaces, a placeholder's bodydiff --git a/internal/redact/shape.go b/internal/redact/shape.gonew file mode 100644index 000000000..5ddc472f9--- /dev/null+++ b/internal/redact/shape.go@@ -0,0 +1,241 @@+package redact++import (+ "crypto/hmac"+ "crypto/sha256"+ "encoding/binary"+ "slices"+ "strings"+)++// A key can go to the vendor as a made-up key of its own form instead of+// a placeholder (Options.Shape): sk_w3axCByTQgK1Px… as sk_hYQWRBFmdHNI…,+// with the same prefix, length and separators and the same kind of+// character in each place. An agent's or a tool's check of a key's form+// passes, and the model has no placeholder to remark on (Charles Cao on X).+// It is made from the placeholder key as a placeholder is, so one key goes+// as the same made-up one every turn and a vendor's cache still matches,+// and it comes back as it was wherever the vendor writes it.+//+// Only a secret's kind, of minShape characters or more, and only letters,+// digits, '-', '_' and '.': a made-up key that long is in no other text by+// chance, and it needs no escaping in JSON. Anything else goes as a+// placeholder still.++const minShape = 16++var (+ // shapes are the made-up keys in values, with their kinds; guarded by+ // mu, as values is+ shapes = map[string]string{}+)++// keyPrefixes name a key's kind across more than one part, and are kept as+// they are; keyHead keeps one part of a few letters.+var keyPrefixes = []string{"sk-ant-api03-", "sk-ant-oat01-", "sk-ant-admin01-", "sk-proj-", "sk-svcacct-", "sk-or-v1-", "github_pat_"}++// keyHead is the start of v that names its kind and stays as it is: one of+// keyPrefixes, or up to four letters and the '-' or '_' after them (sk-,+// ghp_, xoxb-), or AWS's AKIA and ASIA. A head without its separator could+// be the key's own random letters, which stay with the user.+func keyHead(v string) string {+ for _, p := range keyPrefixes {+ if strings.HasPrefix(v, p) {+ return p+ }+ }+ if strings.HasPrefix(v, "AKIA") || strings.HasPrefix(v, "ASIA") {+ return v[:4]+ }+ for i := 0; i < len(v) && i <= 4; i++ {+ c := v[i]+ if (c == '-' || c == '_') && i > 0 {+ return v[:i+1]+ }+ if !(c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z') {+ break+ }+ }+ return ""+}++func shapeByte(c byte) bool {+ return c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || c == '-' || c == '_' || c == '.'+}++// shapeOf is the made-up key v of kind goes as, if it may go as one.+func shapeOf(kind, v string) (string, bool) {+ if _, personal := kindCats[kind]; personal || kind == "TERM" || len(v) < minShape {+ return "", false+ }+ for i := 0; i < len(v); i++ {+ if !shapeByte(v[i]) {+ return "", false+ }+ }+ head := keyHead(v)+ rest := v[len(head):]+ // a hex key's characters are any hex digit, of the case it has+ hexLower, hexUpper := true, true+ for i := 0; i < len(rest); i++ {+ c := rest[i]+ switch {+ case c >= '0' && c <= '9', c == '-' || c == '_' || c == '.':+ case c >= 'a' && c <= 'f':+ hexUpper = false+ case c >= 'A' && c <= 'F':+ hexLower = false+ default:+ hexLower, hexUpper = false, false+ }+ }+ h := hmac.New(sha256.New, key())+ h.Write([]byte("shape\x00" + kind + "\x00" + v))+ seed := h.Sum(nil)+ var stream []byte+ n := 0+ next := func(k int) byte {+ if len(stream) == 0 {+ h := hmac.New(sha256.New, seed)+ var c [8]byte+ binary.BigEndian.PutUint64(c[:], uint64(n))+ n+++ h.Write(c[:])+ stream = h.Sum(nil)+ }+ b := stream[0]+ stream = stream[1:]+ return b % byte(k)+ }+ const (+ upper = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"+ lower = "abcdefghijklmnopqrstuvwxyz"+ digit = "0123456789"+ )+ out := []byte(head)+ for i := 0; i < len(rest); i++ {+ c := rest[i]+ set := ""+ switch {+ case c == '-' || c == '_' || c == '.':+ case hexLower:+ set = digit + "abcdef"+ case hexUpper:+ set = digit + "ABCDEF"+ case c >= 'A' && c <= 'Z':+ set = upper+ case c >= 'a' && c <= 'z':+ set = lower+ default:+ set = digit+ }+ if set == "" {+ out = append(out, c)+ continue+ }+ out = append(out, set[next(len(set))])+ }+ f := string(out)+ return f, f != v+}++// shaped is put for Options.Shape: v as a made-up key of its form where it+// may go as one, else as its placeholder.+func shaped(kind, v string) string {+ f, ok := shapeOf(kind, v)+ if !ok {+ return placeholder(kind, v)+ }+ mu.Lock()+ have, taken := values[f]+ if !taken {+ makeRoom()+ values[f] = v+ kinds[kind] = true+ shapes[f] = kind+ addKnown(f, v)+ }+ mu.Unlock()+ if taken && have != v {+ // another value's made-up key, which nothing ever is by chance+ return placeholder(kind, v)+ }+ return f+}++// restoreShapes puts back each made-up key in s, counting it in put+// where put isn't nil; called with mu held for reading. A made-up key+// needs no escaping, nor does the value it stands for.+func restoreShapes(s string, put map[string]int) string {+ for f := range shapes {+ if n := strings.Count(s, f); n > 0 {+ s = strings.ReplaceAll(s, f, values[f])+ if put != nil {+ put[f] += n+ }+ }+ }+ return s+}++// shapeTail is how much of the end of s may be the start of a made-up+// key, which the next piece of a stream goes on with: the longest end of s+// that one of them starts with, after the last whole one in s. A key that+// ends as one starts (…sqZs, sk_…) has come whole, and is put back whole.+func shapeTail(s string) int {+ mu.RLock()+ defer mu.RUnlock()+ whole := 0+ for f := range shapes {+ if i := strings.LastIndex(s, f); i >= 0 {+ whole = max(whole, i+len(f))+ }+ }+ best := 0+ for f := range shapes {+ for k := min(len(f)-1, len(s)-whole); k > best; k-- {+ if strings.HasSuffix(s, f[:k]) {+ best = k+ break+ }+ }+ }+ return best+}++// hasShapes says some made-up key is in b. With none made, as while+// Options.Shape is off, a whole reply isn't copied to find out.+func hasShapes(b []byte) bool {+ mu.RLock()+ defer mu.RUnlock()+ if len(shapes) == 0 {+ return false+ }+ s := string(b)+ for f := range shapes {+ if strings.Contains(s, f) {+ return true+ }+ }+ return false+}++// outsideShapes is found without what lies in a made-up key already in s,+// as a placeholder's inside is left alone: a key that came back unrestored+// is not masked again as a key of its own.+func outsideShapes(s string, found []span) []span {+ mu.RLock()+ defer mu.RUnlock()+ for f := range shapes {+ for i := 0; ; {+ j := strings.Index(s[i:], f)+ if j < 0 {+ break+ }+ a, z := i+j, i+j+len(f)+ found = slices.DeleteFunc(found, func(x span) bool { return x.start < z && x.end > a })+ i = z+ }+ }+ return found+}diff --git a/internal/redact/shape_test.go b/internal/redact/shape_test.gonew file mode 100644index 000000000..92e24504e--- /dev/null+++ b/internal/redact/shape_test.go@@ -0,0 +1,186 @@+package redact++import (+ "encoding/json"+ "net/http/httptest"+ "strings"+ "testing"+)++// Charles Cao's key on X, sent as a made-up key of its form.+const shapeKey = "sk_w3axCByTQgK1PxqPhfKTrJOLBqz6sbZv"++// sameForm says f has v's head, length and separators, and the same kind+// of character in each other place.+func sameForm(f, v string) bool {+ if len(f) != len(v) || keyHead(f) != keyHead(v) {+ return false+ }+ class := func(c byte) byte {+ switch {+ case c >= 'A' && c <= 'Z':+ return 'A'+ case c >= 'a' && c <= 'z':+ return 'a'+ case c >= '0' && c <= '9':+ return '0'+ }+ return c+ }+ for i := len(keyHead(v)); i < len(v); i++ {+ if class(f[i]) != class(v[i]) {+ return false+ }+ }+ return true+}++func shapeOut(t *testing.T, s string, o Options) string {+ t.Helper()+ out, n := Mask(s, o)+ if n == 0 || strings.Contains(out, shapeKey) {+ t.Fatalf("not masked: %q", out)+ }+ return out+}++func TestShapeKeepsForm(t *testing.T) {+ o := Options{Secrets: true, Shape: true}+ out := shapeOut(t, "OPENAI_API_KEY="+shapeKey, o)+ f := strings.TrimPrefix(out, "OPENAI_API_KEY=")+ if strings.Contains(f, "{{") || !sameForm(f, shapeKey) {+ t.Fatalf("made-up key %q isn't of %q's form", f, shapeKey)+ }+ // the same made-up key every turn, as a vendor's cache needs+ if again, _ := Mask("key: "+shapeKey, o); again != "key: "+f {+ t.Fatalf("again %q, want %q", again, "key: "+f)+ }+ if got := Restore("use "+f+" here", false); got != "use "+shapeKey+" here" {+ t.Fatalf("restored %q", got)+ }+ // a hex key stays hex of its case+ for _, v := range []string{"9f2c4e1a7b3d5c8e9f2c4e1a7b3d5c8e", "ghp_Ab12Cd34Ef56Gh78Ij90Kl12Mn34Op56Qr78", "sk-proj-Xy12_ab-CD34ef56GH78ij90KL12mn34"} {+ f, ok := shapeOf("API_KEY", v)+ hex := strings.Trim(v, "0123456789abcdef") == ""+ if !ok || f == v || hex && (len(f) != len(v) || strings.Trim(f, "0123456789abcdef") != "") || !hex && !sameForm(f, v) {+ t.Fatalf("%q as %q", v, f)+ }+ }+}++// Off, the default, a key goes as a placeholder still; personal data and+// short or odd values do with it on.+func TestShapeOnlyForKeys(t *testing.T) {+ out := shapeOut(t, "OPENAI_API_KEY="+shapeKey, Options{Secrets: true})+ if !strings.Contains(out, "{{") {+ t.Fatalf("off: %q", out)+ }+ for _, c := range []struct{ kind, v string }{+ {"EMAIL", "[email protected]"},+ {"PHONE", "13800138000123456"},+ {"PASSWORD", "short1"},+ {"PASSWORD", "has space in it 12345"},+ {"PASSWORD", `quote"inside1234567`},+ } {+ if f, ok := shapeOf(c.kind, c.v); ok {+ t.Errorf("%s %q as %q", c.kind, c.v, f)+ }+ }+ out, _ = Mask("write to [email protected]", Options{Personal: true, Shape: true})+ if !strings.Contains(out, "{{EMAIL_") {+ t.Fatalf("email: %q", out)+ }+}++// In a request body and back from the vendor's: a tool call's arguments,+// escaped, and a whole reply.+func TestShapeJSON(t *testing.T) {+ o := Options{Secrets: true, Shape: true}+ body := `{"messages":[{"role":"user","content":"OPENAI_API_KEY=` + shapeKey + `"}]}`+ out, n := MaskJSON([]byte(body), o)+ if n == 0 || strings.Contains(string(out), shapeKey) || strings.Contains(string(out), "{{") {+ t.Fatalf("%s", out)+ }+ f := string(out[strings.Index(string(out), "sk_"):][:len(shapeKey)])+ args, _ := json.Marshal(`{"cmd":"export OPENAI_API_KEY=` + f + `"}`)+ reply := `{"content":[{"type":"tool_use","input":` + string(args) + `},{"type":"text","text":"set ` + f + `"}]}`+ got := string(RestoreJSON([]byte(reply)))+ if strings.Contains(got, f) || strings.Count(got, shapeKey) != 2 {+ t.Fatalf("%s", got)+ }+ // what a vendor sealed is masked again as it was sent+ signed := `{"messages":[{"role":"assistant","content":[{"type":"thinking","thinking":"the key ` + shapeKey + `","signature":"Eqk1"}]}]}`+ out, _ = MaskJSON([]byte(signed), o)+ if !strings.Contains(string(out), "the key "+f) {+ t.Fatalf("signed: %s", out)+ }+ // a made-up key that came back unrestored isn't masked again+ if again, _ := Mask("OPENAI_API_KEY="+f, o); again != "OPENAI_API_KEY="+f {+ t.Fatalf("masked again: %q", again)+ }+}++// A stream that splits the made-up key, after one character too, reads as+// the key with the events as they came.+func TestShapeStream(t *testing.T) {+ o := Options{Secrets: true, Shape: true}+ out := shapeOut(t, "OPENAI_API_KEY="+shapeKey, o)+ f := strings.TrimPrefix(out, "OPENAI_API_KEY=")+ chat := func(s string) string {+ b, _ := json.Marshal(s)+ return `data: {"id":"c1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":` + string(b) + `}}]}` + "\n\n"+ }+ pieces := []string{"key s", f[1:9], f[9:20], f[20:], " ok; ", f[:1], f[1:], "", "again " + f[:len(f)-1], f[len(f)-1:], "."}+ var events []string+ for _, p := range pieces {+ events = append(events, chat(p))+ }+ events = append(events, "data: [DONE]\n\n")+ rec := httptest.NewRecorder()+ rec.Header().Set("Content-Type", "text/event-stream")+ w := NewWriter(rec)+ w.WriteHeader(200)+ for _, e := range events {+ w.Write([]byte(e))+ }+ w.Finish()+ var text strings.Builder+ for _, ev := range strings.Split(rec.Body.String(), "\n\n") {+ d, ok := strings.CutPrefix(ev, "data: ")+ if !ok || d == "[DONE]" {+ continue+ }+ var e struct {+ Choices []struct{ Delta struct{ Content string } }+ }+ if err := json.Unmarshal([]byte(d), &e); err != nil {+ t.Fatalf("event %q: %v", d, err)+ }+ text.WriteString(e.Choices[0].Delta.Content)+ }+ // a made-up key ending as one starts, in some of them, isn't held at+ // its end once it has come whole+ mu.Lock()+ sk := "sk_Ab12Cd34Ef56Gh78s"+ values[sk], shapes[sk] = "sk_Zy98Xw76Vu54Ts32r", "SECRET"+ mu.Unlock()+ t.Cleanup(func() {+ mu.Lock()+ delete(values, sk)+ delete(shapes, sk)+ mu.Unlock()+ })+ if n := shapeTail("got " + sk); n != 0 {+ t.Fatalf("held %d of a whole key", n)+ }+ if n := shapeTail("got " + sk + " and s"); n != 1 {+ t.Fatalf("held %d of the next one's start", n)+ }+ want := "key " + shapeKey + " ok; " + shapeKey + "again " + shapeKey + "."+ if text.String() != want {+ t.Fatalf("text:\n got %q\nwant %q\n%s", text.String(), want, rec.Body.String())+ }+ if strings.Count(rec.Body.String(), "\n\n") != len(events) {+ t.Fatalf("events changed:\n%s", rec.Body.String())+ }+}diff --git a/internal/settings/settings.go b/internal/settings/settings.goindex 1ee7c692a..e42fd33ef 100644--- a/internal/settings/settings.go+++ b/internal/settings/settings.go@@ -97,8 +97,11 @@ type Settings struct {// category's On says. RedactModel has a local model (internal/pii) find// what the rules can't as well, a name or an address in free text,// under Redact and RedactPersonal; it is this computer's own, as the- // model is downloaded on it.+ // model is downloaded on it. RedactShape sends a key as a made-up key+ // of its own form (sk_… as another sk_…) instead of a placeholder+ // (redact.Options.Shape; Charles Cao on X).Redact bool `json:"redact,omitempty"`+ RedactShape bool `json:"redactShape,omitempty"`RedactModel bool `json:"redactModel,omitempty"`RedactPersonal bool `json:"redactPersonal,omitempty"`RedactKinds map[string]bool `json:"redactKinds,omitempty"`