Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
privacy: each local model's confidence threshold is the user's to set
yetoneea7c0067e008
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.mdindex fa0f40690..a00f276b0 100644--- a/docs/subsystems/gateway-routing.md+++ b/docs/subsystems/gateway-routing.md@@ -133,7 +133,7 @@ recorded as a prediction failure (`noPrediction`).1. `handle(proto)` reads the body and the model (`requestBody`, `requestModel`), then calls `serveAgent`. Every route that reads an agent's body (each API, count_tokens, Gemini's, embeddings, images, videos, System One, the Codex backend path) reads it through `readBoundedRequestBody` in [`request_bounds.go`](../../internal/gateway/request_bounds.go). It decodes a `Content-Encoding` of `gzip` or `zstd` (`decodedBody`; Codex sends zstd to a custom provider's `/v1/responses` too, #1223) and drops the header, so the provider is sent plain JSON, with one exception: a ChatGPT account's turn (POST `CodexBase`/responses, from the accounts loop in `forwardOnce` or relayed by `codexUpstreamOn`) is compressed again with zstd and a window of at most 2 MiB, as Codex's own client sends it (`provider.CodexCompressed`; openai/codex `responses_request_compression` and `http-client/src/request.rs`). It is compressed once the body is final, and again on each try. `/responses/compact`, an API key's requests and every other provider stay plain. Sent plain, a 1.1 MB text-only turn was answered 503 (ylorn on X). The plain body is what magpie keeps and shows, and `codexDropped` judges a reset by the bytes that went on the wire. `codexDropped` also reads Envoy's 507 "exceeded request buffer limit while retrying upstream" (the backend failed and the front couldn't retry a body that large) as that reset, at any size, told as a 503 so magpie, Codex and pi ask again and nobody rests (#2034). The size limit (`defaultBodyLimit`, 512 MB) holds for the decoded bytes (413), and any other encoding is refused 415. It was 128 MB until a long Codex session that had read many images went past it decoded, its inline screenshots resent each turn, and got the 413 on all five of Codex's retries (#1422, `TestImageHeavyCodexTurnIsRead`). magpie's own 413 says it is magpie's gateway limit, with the size, and that the provider was not asked (`overLimit`); `writeError` adds to any 413 it isn't holding that this is a byte limit, not the model's context, and to `/compact` or start a new conversation (`TestOverLimitSaysWhoseAndWhatToDo`). Every shape but Gemini's types a 413 `request_too_large`, Anthropic's own type, which pi's overflow table knows: pi compacts and tries once more rather than stopping on an OpenAI-shaped `api_error` (@wilsonxuuuu on X, `TestPayloadTooLargeSaysRequestTooLargeToPi`). `keyLimited` reads a limited key's compressed body the same way, so its reservation is the decoded size. `serveAgent` runs the [middleware](gateway-middleware.md) chain, then `serve`.2. `serve` redacts secrets in the request (`redacted`; `redact.MaskJSON` leaves ids alone, and in any text a backend's id (a UUID, or 32 hex digits or more) goes as written: no rule, model label or value known from before masks a piece of one (a phone number at the end of `…-a13800138000`, a card number across a UUID's groups), and a whole one is masked only by the user's own words and rules or as a credential, by the password, API key and token rules or a secret rule whose field is named for one (`api_key=`, `refresh_token`), never one named for an id, a cursor, a request or a digest (`secret_id`, `next_page_token`, `Idempotency-Key`, `secret_hash`), a bare `key`, a serial number, or what the model takes a UUID for (`noIDs`, [`internal/redact/ids.go`](../../internal/redact/ids.go)); and a tool's name and its MCP server's (`namespace`, `server_label`, `server_name`, #3143); the request's tool definitions and output schemas, found by their place in each protocol (top-level `tools`, `tool_choice`, `response_format`, `text`, `generationConfig`, `stream_options`, the client's ids in `client_metadata` and the like, and the tools of a Responses `additional_tools` or `tool_search_output` item), go as written save their descriptions, so a word the user masks can't break a grammar or an enum (#1911); and what the vendor sealed with a signature, a thinking block, Gemini's thoughts or Responses' encrypted reasoning, goes back as the vendor wrote it: only the values masked elsewhere in the request, and secrets masked before, are put back as their placeholders; a value masked once, at least 8 characters long, is masked again wherever a request has it, as an exact match whatever words are around it, so what the vendor wrote about a placeholder doesn't go back with the value in it, `redact/known.go`; a text masked for a vendor before, such as a system prompt or a turn already sent, is masked as it was then, so a value a rule first finds later in a conversation doesn't change the prompt the vendor has cached and rewrite all of it to the cache, `sentKnownSpans`), resolves the model to a provider or a group, and refuses what can't be served. The reply has the values put back on its way to the agent (`redact.Writer`, a stream event by event, a whole body at its end): a placeholder the model wrote as it was given, or as models rewrite one (spaces inside the braces, a brace or both dropped, another case), is the value again when what is left names one magpie made, and one split across events is held until it is whole (`redact.Restore`, `partialTail`, [`internal/redact/redact.go`](../../internal/redact/redact.go)). A request whose provider and fallbacks, or whose group's models, all skip redaction (`unredactedRoute`, `provider.SkipsRedaction`: set *Send requests unmasked*, `Unredacted`, with every address on this machine or the local network) goes to them as the agent wrote it; the log and recent calls keep the masked body. count_tokens and embeddings follow the same rule. What a routing group's Jev is told of the turn (the message, the user's message before it and the reply to it) is masked all the same unless Jev's own provider skips redaction too (`maskedState` in `askJev`, [`decide.go`](../../internal/gateway/decide.go)): a remote Jev is none of the group's members. The masking is the built-in plugin `redact`'s (`gwmw.Masker`, [Plugin runtime](plugin-runtime.md)); when the settings ask for masking and no masker serves, or it fails, the request is a 503 and nothing is sent (`redacted`, `gwmw.Masking`).- With *Mask personal data* on, each kind in `redact.Categories` ([`internal/redact/more.go`](../../internal/redact/more.go)) is masked unless the user turned it off (`redactKinds` in settings, `Options.Kinds`). On by default: US SSNs, passport numbers, IBANs that pass mod 97, birthdays next to a word that says so (a date over a year ago: one in the last year or ahead is the day an agent runs on, `dateOK`), MAC addresses, and serial numbers next to S/N or the like. Off by default: the user name in a home folder, user@host in a prompt or ssh command, public IPv4 and IPv6 addresses (never private, loopback, link-local, CGNAT or documentation ranges) and cloud bucket names, since agents send them in nearly every request and the model must copy the placeholder exactly inside a path or URL. Context rules run only near their words (`rule.matches`). What reads as an address but ends in a file's extension that no domain has (`…@2x.jpg`, `[email protected]`; `fileExt` in `realEmail`), or a scale before an extension that is a domain too (`[email protected]`, `[email protected]`; `scaleExt`), is a file's name, for the rule and the local model alike (#3029). `[email protected]` is still an address. The Settings rows come from `REDACT_KINDS` in app.js, which `TestRedactKindsAreRedacts` holds to the same ids and defaults.- - With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken, nor a snake_case word as a name (Codex's `sequential_cutoff`, #3144), nor as a secret what names a thing: a UUID, a digest with its algorithm's name, a file's path (`namesThing`; the model takes all three for secrets). The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with it on a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left. The model is the user's pick (`settings.RedactModelName`, `pii.Choose`; the app's menu beside the switch, `pmPick`): AstrLink Guard (QuantumNous/astrlink-guard at a pinned revision, Apache-2.0, a 39 MB int8 Multilingual-MiniLM token classifier for Chinese and English), run in magpie's own process on the CPU by [`internal/pii/guard`](../../internal/pii/guard) (an ONNX reader, the XLM-R tokenizer and int8 kernels in Go, NEON/SDOT on arm64 and SSE2 on amd64; its labels named as Privacy Filter's, `guardLabels`), loaded on the first request and let go after 10 minutes with none; or OpenAI's Privacy Filter. With no pick, a user who had Privacy Filter on or downloaded keeps it and everyone else gets AstrLink Guard (#3578): on 234 segments of real code and config it found 2 spans where Privacy Filter found 100 (38 of them go.sum hashes after redact's own checks), and with the rules it found 42 of 42 test values to Privacy Filter's 41. Each model has its own download, cache file and turn (`backend`). Privacy Filter is a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags; on a GPU it holds the model once, in the device's memory, #3590; `pf-serve-vulkan` runs its MoE in 2048-token chunks, `PF_MOE_CHUNK`, 1.77× faster than its default 4096 on an RX 470/580-class GPU and a little slower on Metal, so Vulkan's alone, and a value the user set is kept, `moeChunk`): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, through magpie's proxy (`netproxy.Func`, as its other requests; hh0592821 on Discord, whose download went direct before), from hf-mirror first in China (`source.chinaFirst`) and from hf-mirror after Hugging Face everywhere else, the switch off too (`source.modelMirror`), so a poisoned or blocked huggingface.co doesn't stop it; hf-mirror serves AstrLink Guard's files itself and sends Privacy Filter's back to Hugging Face) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU, doesn't load there in 40s (`gpuLoadTimeout`, well inside the gateway's 2 minutes), or doesn't answer in time there twice in a row, is started on the CPU, and Settings says why; a request it didn't answer in time once is tried on the GPU again, since a slow request on a weak GPU is no lost GPU (#3682). The GPU is tried again 10 minutes later, doubling to 2 hours while it keeps failing, with no restart (`gpuRetryAfter`), and one that answers is the model's again), joins short texts into 12 KB chunks and maps the spans back, cuts a text over 12 KB into pieces of 12 KB at most that overlap by 1 KB (`pieces`: after a line's end, else a space, else at a rune's start), each kept by its own hash, their spans put back at the text's offsets and merged where pieces overlap (`merge`: one found twice is one, one cut by a piece's end is whole), and asks pf-serve a chunk at a time (#3682: a 480 KB text whole took an RX 470/580 90s and kept nothing when cut off), keeps the last 20000 answers by hash, on disk too (`found-<model>.bin` beside the model: hashes, offsets and labels, never text; written through temp-and-rename, a bad read taken as empty), so a restart doesn't ask the model about every history again (#3682), and ends it after 10 minutes with no request. Its requests take turns on its one pipe (`turn`), and a request reads the cache again when its turn comes; a request waits for its turn only while its context lives (`pii.FindContext`, through `gwmw.ContextTeller`), so one whose client went away leaves the queue (the gateway's 2 minutes leave masking's time out, #3682); the chunk pf-serve is on is finished and kept, so a request sent again asks only about what is left, and between chunks the turn is given up and taken again, so a short request goes before the rest of a long one. The status (`pii.Now`), `pii.Stop` and the idle timer never wait for a load or a request: `srv.mu` is held only to read or set the state, and `Stop` ends a load or a request under way at once. A request held before routing while it waits for the model is answered 504 saying so, and how to send without it (`unroutedMsg`); one of an older tag (`oldTags`, magpie updated since the download) runs until the pinned tag's is fetched (`upgrade`, on its start), which is then started on the next request and the old deleted ([`internal/pii`](../../internal/pii)). A failure names what ended `pf-serve`: magpie's own end of a request it didn't answer in time, else its exit status or signal and its last stderr lines past ggml's startup chatter (`died`, `lastSaid`). After one, requests are held back at once with that failure, not after a load each, until the model is tried again 30s later, doubling to 10 minutes while it keeps failing (`heldBack`, `retryAfter`; lazyeo on Discord, an M2 whose every request waited out the 504). A model that is on and can't answer (not on disk, didn't start, hung, held back) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).+ - With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken, nor a snake_case word as a name (Codex's `sequential_cutoff`, #3144), nor as a secret what names a thing: a UUID, a digest with its algorithm's name, a file's path (`namesThing`; the model takes all three for secrets). The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with it on a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left. The model is the user's pick (`settings.RedactModelName`, `pii.Choose`; the app's menu beside the switch, `pmPick`): AstrLink Guard (QuantumNous/astrlink-guard at a pinned revision, Apache-2.0, a 39 MB int8 Multilingual-MiniLM token classifier for Chinese and English), run in magpie's own process on the CPU by [`internal/pii/guard`](../../internal/pii/guard) (an ONNX reader, the XLM-R tokenizer and int8 kernels in Go, NEON/SDOT on arm64 and SSE2 on amd64; its labels named as Privacy Filter's, `guardLabels`), loaded on the first request and let go after 10 minutes with none; or OpenAI's Privacy Filter. With no pick, a user who had Privacy Filter on or downloaded keeps it and everyone else gets AstrLink Guard (#3578): on 234 segments of real code and config it found 2 spans where Privacy Filter found 100 (38 of them go.sum hashes after redact's own checks), and with the rules it found 42 of 42 test values to Privacy Filter's 41. Each model has its own download, cache file and turn (`backend`). Both find at a floor of 0.5 (`pii.Floor`) and give each span its score (AstrLink Guard: the least of its tokens' probabilities, rounded to a float32 so a cached span reads back the same; Privacy Filter: pf-serve's own), and masking keeps those at the model's threshold or over (`pii.Threshold` in `masking.model`): the user's for that model (`settings.RedactModelScore`, this computer's own; Settings → Privacy's Confidence threshold, `pmScore`, 0.5 to 0.95), else the model's own (`MinScore`: Privacy Filter 0.5, AstrLink Guard 0.8, its publisher's). The defaults stay because on the eval a higher one mostly loses values the checks after the model would have kept (Privacy Filter at 0.85 lost a Chinese name; AstrLink Guard at 0.9 a phone in a tool result), while the false alarms it would drop are mostly dropped by those checks already (Yayoi_no_yume on X). Privacy Filter is a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags; on a GPU it holds the model once, in the device's memory, #3590; `pf-serve-vulkan` runs its MoE in 2048-token chunks, `PF_MOE_CHUNK`, 1.77× faster than its default 4096 on an RX 470/580-class GPU and a little slower on Metal, so Vulkan's alone, and a value the user set is kept, `moeChunk`): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, through magpie's proxy (`netproxy.Func`, as its other requests; hh0592821 on Discord, whose download went direct before), from hf-mirror first in China (`source.chinaFirst`) and from hf-mirror after Hugging Face everywhere else, the switch off too (`source.modelMirror`), so a poisoned or blocked huggingface.co doesn't stop it; hf-mirror serves AstrLink Guard's files itself and sends Privacy Filter's back to Hugging Face) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU, doesn't load there in 40s (`gpuLoadTimeout`, well inside the gateway's 2 minutes), or doesn't answer in time there twice in a row, is started on the CPU, and Settings says why; a request it didn't answer in time once is tried on the GPU again, since a slow request on a weak GPU is no lost GPU (#3682). The GPU is tried again 10 minutes later, doubling to 2 hours while it keeps failing, with no restart (`gpuRetryAfter`), and one that answers is the model's again), joins short texts into 12 KB chunks and maps the spans back, cuts a text over 12 KB into pieces of 12 KB at most that overlap by 1 KB (`pieces`: after a line's end, else a space, else at a rune's start), each kept by its own hash, their spans put back at the text's offsets and merged where pieces overlap (`merge`: one found twice is one, one cut by a piece's end is whole), and asks pf-serve a chunk at a time (#3682: a 480 KB text whole took an RX 470/580 90s and kept nothing when cut off), keeps the last 20000 answers by hash, on disk too (`found-<model>.bin` beside the model: hashes, offsets, labels and each span's score, never text; a `pfound01` file from before scores is read with every span sure, so it masks as it did; written through temp-and-rename, a bad read taken as empty), so a restart doesn't ask the model about every history again (#3682), and ends it after 10 minutes with no request. Its requests take turns on its one pipe (`turn`), and a request reads the cache again when its turn comes; a request waits for its turn only while its context lives (`pii.FindContext`, through `gwmw.ContextTeller`), so one whose client went away leaves the queue (the gateway's 2 minutes leave masking's time out, #3682); the chunk pf-serve is on is finished and kept, so a request sent again asks only about what is left, and between chunks the turn is given up and taken again, so a short request goes before the rest of a long one. The status (`pii.Now`), `pii.Stop` and the idle timer never wait for a load or a request: `srv.mu` is held only to read or set the state, and `Stop` ends a load or a request under way at once. A request held before routing while it waits for the model is answered 504 saying so, and how to send without it (`unroutedMsg`); one of an older tag (`oldTags`, magpie updated since the download) runs until the pinned tag's is fetched (`upgrade`, on its start), which is then started on the next request and the old deleted ([`internal/pii`](../../internal/pii)). A failure names what ended `pf-serve`: magpie's own end of a request it didn't answer in time, else its exit status or signal and its last stderr lines past ggml's startup chatter (`died`, `lastSaid`). After one, requests are held back at once with that failure, not after a load each, until the model is tried again 30s later, doubling to 10 minutes while it keeps failing (`heldBack`, `retryAfter`; lazyeo on Discord, an M2 whose every request waited out the 504). A model that is on and can't answer (not on disk, didn't start, hung, held back) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).- With *Mask secrets* on, the user's own masking rules (`redactRules`, `redact.Rule`: a placeholder name and a prefix or an RE2 pattern, [`internal/redact/rules.go`](../../internal/redact/rules.go)) run before magpie's. Settings → Privacy adds them, removes them, and edits one in place (`POST /api/settings/redact-rule` with the rule's index and its current value, `editRedactRule` in [`internal/gui/api.go`](../../internal/gui/api.go)). Every rule is checked by `redact.CheckRules` when settings are saved; an edit that doesn't compile, matches the empty string, repeats another rule or names a rule no longer at that index is refused with the rules left as they were. The masker reads the rules from settings on each call (`options` in [`internal/masking`](../../internal/masking/masking.go)), so a change applies to the next request.- With *Keep a key's form* on (`settings.RedactShape`, `Options.Shape`, under *Mask secrets*; Charles Cao on X), a secret of 16 characters or more, all letters, digits, `-`, `_` or `.`, goes as a made-up key of its form instead of a placeholder: its prefix kept (`sk-proj-`, `ghp_`, `AKIA`, up to four letters and their separator; `keyHead`), the same length and separators, and each other character a random one of its kind (a hex key's any hex digit of its case), drawn from an HMAC of the value under the placeholder key, so one key goes as the same made-up one every turn and a vendor's cache still matches (`shapeOf`, `shaped` in [`internal/redact/shape.go`](../../internal/redact/shape.go)). Personal data, the user's words and anything shorter or with other characters go as placeholders still. A made-up key is kept in the masker's memory with the placeholders, so it is recorded, re-masked in what a vendor sealed, and put back wherever the reply has it, before the placeholders (`restoreShapes`); a stream holds the end of an event that may be the start of one (`shapeTail`, after the last whole one), and a made-up key the vendor sends back unrestored isn't masked again (`outsideShapes`). `TestShapeKeepsForm`, `TestShapeJSON`, `TestShapeStream`. Insights › Masking and Settings' *Try it* mark a made-up key where they mark a placeholder, matching the text against the request's own values only, so a key of the same form the user wrote isn't marked (`maskMatches` in `plugins/redact/page.js`; #3599, masking-records and mask-actions GUI tests).- What is done with a value found is the user's per kind (#1977, `redactActions` in settings, `Options.Actions`, set by `POST /api/settings/redact-actions`; a kind left out is masked). `redact.Warn` sends it as written: the masker tells it, with a placeholder no one remembers, and it isn't counted as masked. `redact.Block` masks it as any other, and the gateway holds the request back with a 400 that names the kind and the value's hint, never the value (`blocked`, `maskBlocked` in [`redact.go`](../../internal/gateway/redact.go)); nothing is sent and the record is kept as `Blocked` (`noteBlocked`). A value on the *Never mask* list (`redactAllow`, `Options.Allow`, `POST /api/settings/redact-allow`) goes as written when it is the whole value found, in any case, and isn't told; a longer value it is part of is masked. Settings → Privacy's *Try it* posts a text to `POST /api/masking/preview` ([`masking.go`](../../internal/gui/masking.go)), which masks it as a request would be (`redact.Preview`) and remembers nothing: its placeholders stand for nothing afterwards, its values aren't masked later as seen before, and it returns only the masked text and each hit's kind, finder, action, placeholder and hint. The page keeps the text in no storage.diff --git a/internal/gui/api.go b/internal/gui/api.goindex 3ad3c3394..c67006274 100644--- a/internal/gui/api.go+++ b/internal/gui/api.go@@ -1287,6 +1287,7 @@ func Handler(w Windows, gw *gateway.Server) http.Handler {in.RedactRules = cur.RedactRules // the masking rules, set on their ownin.RedactModel = cur.RedactModel // set on its own (privacy-model)in.RedactModelName = cur.RedactModelName // set on its own (privacy-model)+ in.RedactModelScore = cur.RedactModelScore // set on its own (privacy-model)in.RedactActions = cur.RedactActions // set on their own (redact-actions below)in.RedactAllow = cur.RedactAllow // set on its own (redact-allow below)in.DisabledModels = cur.DisabledModels // set on their own (disabled-models below, #1867)diff --git a/internal/gui/assets/app.js b/internal/gui/assets/app.jsindex 715abc876..e320d0330 100644--- a/internal/gui/assets/app.js+++ b/internal/gui/assets/app.js@@ -21911,12 +21911,15 @@ function renderRedact(s, keep) {// privacyModelRow: the local masking model (internal/pii), downloaded when// it is turned on and then asked beside the rules. Its state is its own// (privacy-model): read when the row is drawn, and each second while it-// downloads or starts.+// downloads or starts. Under it, once the model can run, its threshold+// (pmScore).let pmState = null, pmTimer = 0;function privacyModelRow(box, why) {const r = el("div", "row pref" + (why ? " pm-wait" : ""));r.id = "privacyModelRow";box.append(r);+ const sr = el("div", "row pref");+ sr.id = "pmScoreRow";const name = () => { const n = el("div", "name", t("Find with a local model")); n.append(el("span", "badge new", t("New"))); return n; };// nothing is masked: the model has nothing to find more ofif (why) {@@ -21947,13 +21950,21 @@ function privacyModelRow(box, why) {val.append(segs([["off", t("Off")], ["on", t("On")]], m.on || m.downloading ? "on" : "off", (v) => act(v)));}r.replaceChildren(who, val);+ if (m && (m.supported || m.installed) && m.defaultScore) {+ const sw = el("div", "who");+ sw.append(el("div", "name", t("Confidence threshold")), el("div", "sub", t("What the model is less sure of goes as written. Higher masks less, with fewer false alarms")));+ const sv = el("div", "val");+ sv.append(pmScore(m, (score) => act("score", m.model, score)));+ sr.replaceChildren(sw, sv);+ if (!sr.isConnected) r.after(sr);+ } else sr.remove();clearTimeout(pmTimer);// a download's progress, and where pf-serve runs once it has startedconst starting = m?.on && m.installed && !m.running && !m.failed && tries++ < 30;if (pmBusy(m) || starting) pmTimer = setTimeout(load, 1000);};const load = () => api("privacy-model").then((m) => { pmState = m; draw(m); }).catch(() => {});- const act = (doing, model) => { tries = 0; api("privacy-model", { do: doing, model }).then((m) => { pmState = m; draw(m); }).catch((e) => { status(t(e.message), "err"); load(); }); };+ const act = (doing, model, score) => { tries = 0; api("privacy-model", { do: doing, model, score }).then((m) => { pmState = m; draw(m); }).catch((e) => { status(t(e.message), "err"); load(); }); };draw(pmState);load();}@@ -21978,6 +21989,27 @@ function pmPick(m, pick) {};return b;}+// pmScore picks the model's threshold (privacy-model's score) by the+// app's menu: its default, or one of PM_SCORES+const PM_SCORES = [0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95];+function pmScore(m, pick) {+ const cur = m.minScore === m.defaultScore ? "0" : String(m.minScore);+ const b = el("button", "sess-pick pm-score");+ b.type = "button";+ b.setAttribute("aria-haspopup", "menu");+ b.setAttribute("aria-expanded", "false");+ b.title = t("Confidence threshold");+ b.append(el("span", "", String(m.minScore)), svg(CHEV, 11, 1.6));+ b.onclick = (e) => {+ e.stopPropagation();+ if (b.classList.contains("open")) return closeProtoMenu();+ const note = (s) => s === PM_SCORES[0] ? "Masks the most" : s === PM_SCORES.at(-1) ? "Masks the least" : "";+ const opts = [{ v: "0", name: t("Default ({score})", { score: m.defaultScore }), literalName: true, note: note(m.defaultScore) },+ ...PM_SCORES.filter((s) => s !== m.defaultScore).map((s) => ({ v: String(s), name: String(s), literalName: true, note: note(s) }))];+ openProtoMenu(b, opts, cur, (v) => v !== cur && pick(Number(v)), "Confidence threshold", "sess-menu", "right");+ };+ return b;+}function pmSub(m) {const about = t("Names, addresses and passwords no rule can find, found by a model that runs on this computer");if (!m) return about;diff --git a/internal/gui/assets/i18n.js b/internal/gui/assets/i18n.jsindex 6109aac58..e350f6ea7 100644--- a/internal/gui/assets/i18n.js+++ b/internal/gui/assets/i18n.js@@ -1201,6 +1201,11 @@ const I18N = {"Delete model": "删除模型","Delete the model?": "删除模型?","The local model": "本地模型",+ "Confidence threshold": "置信度阈值",+ "What the model is less sure of goes as written. Higher masks less, with fewer false alarms": "模型把握低于此值的内容原样发送。调高则遮得更少,误报也更少",+ "Default ({score})": "默认({score})",+ "Masks the most": "遮得最多",+ "Masks the least": "遮得最少","Chinese and English, few false alarms in code": "中英文,代码里很少误判","OpenAI's, masks more ordinary values in code": "OpenAI 出品,代码里会遮盖更多普通值","downloaded": "已下载",@@ -5205,6 +5210,11 @@ const I18N = {"Delete model": "刪除模型","Delete the model?": "刪除模型?","The local model": "本機模型",+ "Confidence threshold": "信賴度門檻",+ "What the model is less sure of goes as written. Higher masks less, with fewer false alarms": "模型把握低於此值的內容原樣送出。調高則遮得更少,誤報也更少",+ "Default ({score})": "預設({score})",+ "Masks the most": "遮得最多",+ "Masks the least": "遮得最少","Chinese and English, few false alarms in code": "中英文,程式碼裡很少誤判","OpenAI's, masks more ordinary values in code": "OpenAI 出品,程式碼裡會遮蓋更多普通值","downloaded": "已下載",@@ -9179,6 +9189,11 @@ const I18N = {"Delete model": "モデルを削除","Delete the model?": "モデルを削除しますか?","The local model": "ローカルモデル",+ "Confidence threshold": "信頼度のしきい値",+ "What the model is less sure of goes as written. Higher masks less, with fewer false alarms": "モデルの確信度がこれより低いものはそのまま送ります。高くするとマスクが減り、誤検出も減ります",+ "Default ({score})": "デフォルト({score})",+ "Masks the most": "最も多くマスク",+ "Masks the least": "最も少なくマスク","Chinese and English, few false alarms in code": "中国語と英語、コードでの誤検出が少ない","OpenAI's, masks more ordinary values in code": "OpenAI 製、コード中の普通の値も多く隠す","downloaded": "ダウンロード済み",@@ -13183,6 +13198,11 @@ const I18N = {"Delete model": "Modell löschen","Delete the model?": "Modell löschen?","The local model": "Lokales Modell",+ "Confidence threshold": "Konfidenzschwelle",+ "What the model is less sure of goes as written. Higher masks less, with fewer false alarms": "Was das Modell für weniger sicher hält, geht unverändert raus. Höher maskiert weniger, mit weniger Fehlalarmen",+ "Default ({score})": "Standard ({score})",+ "Masks the most": "Maskiert am meisten",+ "Masks the least": "Maskiert am wenigsten","Chinese and English, few false alarms in code": "Chinesisch und Englisch, wenig Fehlalarme in Code","OpenAI's, masks more ordinary values in code": "Von OpenAI, maskiert in Code mehr gewöhnliche Werte","downloaded": "geladen",diff --git a/internal/gui/privacymodel.go b/internal/gui/privacymodel.goindex 789c1739b..0ba997846 100644--- a/internal/gui/privacymodel.go+++ b/internal/gui/privacymodel.go@@ -11,12 +11,15 @@ import (// privacyModelJSON is Settings → Privacy's local masking model: whether it// is on, the model masking uses (or would, turned on) with its own status-// (downloaded, downloading, where it runs), and every model there is to-// pick, with theirs.+// (downloaded, downloading, where it runs) and threshold (the one the+// user set, else its default), and every model there is to pick, with+// theirs.type privacyModelJSON struct {On bool `json:"on"`pii.Status- Models []pii.Status `json:"models"`+ MinScore float64 `json:"minScore"`+ DefaultScore float64 `json:"defaultScore"`+ Models []pii.Status `json:"models"`}// modelOf and models are pii's; vars for tests.@@ -24,7 +27,8 @@ var modelOf, models = pii.Of, pii.Modelsfunc privacyModelState() privacyModelJSON {st := settings.Load()- out := privacyModelJSON{On: st.RedactModel, Status: modelOf(pii.Choose(st.RedactModelName, st.RedactModel)).Now()}+ m := modelOf(pii.Choose(st.RedactModelName, st.RedactModel))+ out := privacyModelJSON{On: st.RedactModel, Status: m.Now(), MinScore: pii.Threshold(st.RedactModelScore, m), DefaultScore: m.MinScore()}for _, m := range models() {out.Models = append(out.Models, m.Now())}@@ -32,21 +36,24 @@ func privacyModelState() privacyModelJSON {}// privacyModelRoutes: GET /api/privacy-model is its state; POST with-// {"do": "on" | "off" | "cancel" | "remove" | "pick", "model": id} turns+// {"do": "on" | "off" | "cancel" | "remove" | "pick" | "score", "model": id,+// "score": s} turns// it on (as model, or the one it would use: downloading it first, and// saving it on once that is done), off (a download is stopped, the model// let go and kept), stops a download, turns it off and deletes every// model downloaded, or picks model: while it is on, that is turning it on// as model (the one before masks until model is downloaded); while off,-// the pick is kept for turning it on.+// the pick is kept for turning it on; score keeps s as the threshold of+// model (the one masking uses, with none), 0 for its default.func privacyModelRoutes(mux *http.ServeMux) {mux.HandleFunc("GET /api/privacy-model", func(rw http.ResponseWriter, r *http.Request) {writeJSON(rw, privacyModelState())})mux.HandleFunc("POST /api/privacy-model", func(rw http.ResponseWriter, r *http.Request) {var in struct {- Do string `json:"do"`- Model string `json:"model"`+ Do string `json:"do"`+ Model string `json:"model"`+ Score float64 `json:"score"`}if err := json.NewDecoder(r.Body).Decode(&in); err != nil {fail(rw, err)@@ -69,6 +76,16 @@ func privacyModelRoutes(mux *http.ServeMux) {default:err = settings.SetRedactModelName(in.Model)}+ case "score":+ id := in.Model+ if id == "" {+ id = was+ }+ if in.Score != 0 && !(in.Score >= pii.Floor && in.Score < 1) {+ err = fmt.Errorf("score is 0 (the model's own) or from %g to under 1, not %g", pii.Floor, in.Score)+ } else {+ err = settings.SetRedactModelScore(id, in.Score)+ }case "on":id := in.Modelif id == "" {@@ -96,7 +113,7 @@ func privacyModelRoutes(mux *http.ServeMux) {}}default:- err = fmt.Errorf("do is on, off, cancel, remove or pick, not %q", in.Do)+ err = fmt.Errorf("do is on, off, cancel, remove, pick or score, not %q", in.Do)}if err != nil {fail(rw, err)diff --git a/internal/gui/privacymodel_test.go b/internal/gui/privacymodel_test.goindex 53426421f..041f884a1 100644--- a/internal/gui/privacymodel_test.go+++ b/internal/gui/privacymodel_test.go@@ -78,10 +78,11 @@ type fakeModel struct {did *[]string}-func (f *fakeModel) ID() string { return f.id }-func (f *fakeModel) Supported() bool { return true }-func (f *fakeModel) Installed() bool { return f.installed }-func (f *fakeModel) Size() int64 { return 1 }+func (f *fakeModel) ID() string { return f.id }+func (f *fakeModel) Supported() bool { return true }+func (f *fakeModel) Installed() bool { return f.installed }+func (f *fakeModel) MinScore() float64 { return 0.5 }+func (f *fakeModel) Size() int64 { return 1 }func (f *fakeModel) Find(context.Context, []string) ([][]redact.Found, error) {return nil, nil}@@ -152,6 +153,28 @@ func TestPrivacyModelPick(t *testing.T) {if strings.Join(did, ", ") != "download astrlink-guard, stop privacy-filter, warm astrlink-guard" {t.Fatalf("did %v", did)}+ // the threshold: the model's own until one is set, for the model+ // masking uses with none named; 0 goes back to its own, and one out+ // of range is refused+ if st := do(""); st.MinScore != 0.5 || st.DefaultScore != 0.5 {+ t.Fatalf("threshold before one is set: %+v", st)+ }+ if st := do(`{"do":"score","score":0.9}`); st.MinScore != 0.9 || st.DefaultScore != 0.5 || settings.Load().RedactModelScore[pii.AstrLinkGuard] != 0.9 {+ t.Fatalf("threshold set: %+v, %v", st, settings.Load().RedactModelScore)+ }+ if st := do(`{"do":"score","model":"privacy-filter","score":0.6}`); st.MinScore != 0.9 || settings.Load().RedactModelScore[pii.PrivacyFilter] != 0.6 {+ t.Fatalf("another model's threshold: %+v, %v", st, settings.Load().RedactModelScore)+ }+ for _, bad := range []string{`0.3`, `1`, `-1`} {+ rec := httptest.NewRecorder()+ srv.ServeHTTP(rec, httptest.NewRequest("POST", "/api/privacy-model", strings.NewReader(`{"do":"score","score":`+bad+`}`)))+ if rec.Code == http.StatusOK || settings.Load().RedactModelScore[pii.AstrLinkGuard] != 0.9 {+ t.Fatalf("threshold %s: %d", bad, rec.Code)+ }+ }+ if st := do(`{"do":"score","score":0}`); st.MinScore != 0.5 || len(settings.Load().RedactModelScore) != 1 {+ t.Fatalf("back to the default: %+v, %v", st, settings.Load().RedactModelScore)+ }rec := httptest.NewRecorder()srv.ServeHTTP(rec, httptest.NewRequest("POST", "/api/privacy-model", strings.NewReader(`{"do":"pick","model":"gpt"}`)))if rec.Code == http.StatusOK || settings.Load().RedactModelName != pii.AstrLinkGuard {diff --git a/internal/gui/settings_keep_test.go b/internal/gui/settings_keep_test.goindex 7c91fc104..3caf872e2 100644--- a/internal/gui/settings_keep_test.go+++ b/internal/gui/settings_keep_test.go@@ -182,6 +182,7 @@ func TestSettingsSaveKeepsWhatItDoesNotSend(t *testing.T) {NoAccountDiscovery: true,RedactModel: true,RedactModelName: "astrlink-guard",+ RedactModelScore: map[string]float64{"astrlink-guard": 0.9},}was.DesktopSubagentEffort = "high"was.CodexAsyncCompact, was.CodexAsyncCompactPercent = true, 70diff --git a/internal/gui/tests/privacy-model.test.cjs b/internal/gui/tests/privacy-model.test.cjsindex 742506c32..a1a7d86fd 100644--- a/internal/gui/tests/privacy-model.test.cjs+++ b/internal/gui/tests/privacy-model.test.cjs@@ -4,7 +4,8 @@// shown and a Cancel beside it; once it is on, the row says where it runs.// Off, a downloaded model can be deleted, after asking. Its words are shown// in full, in every language, in a wide window and a narrow one, and a-// click moves nothing.+// click moves nothing. Under it, the model's threshold is picked from the+// app's menu too.const assert = require("node:assert/strict");const fs = require("node:fs/promises");const path = require("node:path");@@ -36,8 +37,9 @@ function serve(lang, masking, model, posts) {if (url.pathname === "/api/settings") return json(settings());if (url.pathname === "/api/privacy-model") {if (r.request().method() === "POST") {- const { do: doing, model: id } = JSON.parse(r.request().postData());- posts.push(id ? doing + " " + id : doing);+ const { do: doing, model: id, score } = JSON.parse(r.request().postData());+ posts.push([doing, id, score].filter((x) => x !== undefined && x !== "").join(" "));+ if (doing === "score") model.minScore = score || model.defaultScore;if (doing === "on") Object.assign(model, { downloading: true, done: 0, total: size, ticks: 0 });if (doing === "cancel") Object.assign(model, { downloading: false, done: 0 });if (doing === "remove") Object.assign(model, { installed: false, on: false });@@ -84,6 +86,14 @@ const pickWords = {de: { head: "Lokales Modell", note: "Chinesisch und Englisch, wenig Fehlalarme in Code · 39 MB" },};+const scoreWords = {+ en: { name: "Confidence threshold", def: "Default (0.8)", most: "Masks the most" },+ zh: { name: "置信度阈值", def: "默认(0.8)", most: "遮得最多" },+ "zh-TW": { name: "信賴度門檻", def: "預設(0.8)", most: "遮得最多" },+ ja: { name: "信頼度のしきい値", def: "デフォルト(0.8)", most: "最も多くマスク" },+ de: { name: "Konfidenzschwelle", def: "Standard (0.8)", most: "Maskiert am meisten" },+};+// what the row shows: its words and its New mark, whether they are cut,// that it is inside the list, and that it is the page's first settingconst look = (page) => page.evaluate(() => {@@ -229,4 +239,63 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",});}}++ // the threshold (@Yayoi_no_yume on X): under the model, its default+ // or one the user picks from the app's menu, in every language; the+ // pick is sent with the model it is for, and shown once magpie has it+ for (const lang of Object.keys(scoreWords)) {+ for (const width of [1100, 640]) {+ test(`${engine} ${lang} ${width}px: the model's threshold is picked from the app's menu`, async (t) => {+ const w = scoreWords[lang];+ const posts = [], model = { ...guard, installed: true, on: true, running: true, device: "CPU", minScore: 0.8, defaultScore: 0.8 };+ const { browser, page, errors } = await open(engine, lang, width, true, model, posts);+ t.after(() => browser.close());+ const row = page.locator("#pmScoreRow");+ await row.waitFor();+ await wheelTo(page, row, 300);+ assert(await page.evaluate(() => document.querySelector("#privacyModelRow").nextElementSibling === document.querySelector("#pmScoreRow")), "right under the model");+ assert.equal(await row.locator("select").count(), 0, "no native select");+ const shown = () => row.evaluate((r) => {+ const sub = r.querySelector(".sub"), b = r.getBoundingClientRect(), list = r.closest(".list").getBoundingClientRect();+ return {+ name: r.querySelector(".name").textContent, sub: sub.textContent, pick: r.querySelector("button.pm-score").textContent.trim(),+ cut: sub.scrollWidth > sub.clientWidth + 1 || sub.scrollHeight > sub.clientHeight + 1,+ inside: b.left >= list.left - 1 && b.right <= list.right + 1, wide: document.documentElement.scrollWidth > document.documentElement.clientWidth,+ };+ });+ let m = await shown();+ assert.equal(m.name, w.name);+ if (lang !== "en") assert(!/less sure/.test(m.sub), `translated: ${m.sub}`);+ assert.equal(m.pick, "0.8");+ assert(!m.cut && m.inside && !m.wide, `shown in full and inside: ${JSON.stringify(m)}`);+ const before = await page.evaluate(() => [scrollX, scrollY, document.querySelector("#view-settings").scrollTop]);+ const menu = page.locator(".proto-menu");+ const items = async () => {+ await row.locator("button.pm-score").click();+ await menu.waitFor();+ return menu.evaluate((b) => ({+ head: b.querySelector(".pm-head").textContent,+ items: [...b.querySelectorAll(".pm-item")].map((i) => [i.querySelector(".pm-name").textContent, i.querySelector(".pm-note").textContent, i.classList.contains("on")]),+ inside: b.getBoundingClientRect().left >= 0 && b.getBoundingClientRect().right <= innerWidth,+ }));+ };+ let got = await items();+ assert.equal(got.head, w.name);+ assert.deepEqual(got.items.map((i) => i[0]), [w.def, "0.5", "0.6", "0.7", "0.85", "0.9", "0.95"]);+ assert.deepEqual(got.items.map((i) => i[2]), [true, false, false, false, false, false, false], "the default is ticked");+ assert.equal(got.items[1][1], w.most);+ assert(got.inside, "the menu is inside the window");+ await menu.locator(".pm-item", { hasText: /^0\.9$/ }).click();+ await page.waitForFunction(() => document.querySelector("#pmScoreRow button.pm-score")?.textContent.trim() === "0.9");+ assert.deepEqual(posts, ["score astrlink-guard 0.9"]);+ got = await items();+ assert.deepEqual(got.items.map((i) => i[2]), [false, false, false, false, false, true, false], "0.9 is ticked");+ await menu.locator(".pm-item").first().click();+ await page.waitForFunction(() => document.querySelector("#pmScoreRow button.pm-score")?.textContent.trim() === "0.8");+ assert.deepEqual(posts, ["score astrlink-guard 0.9", "score astrlink-guard 0"]);+ assert.deepEqual(await page.evaluate(() => [scrollX, scrollY, document.querySelector("#view-settings").scrollTop]), before, "the picks moved nothing");+ assert.deepEqual(errors, []);+ });+ }+ }}diff --git a/internal/masking/masking.go b/internal/masking/masking.goindex c61bd6e96..88bcb4886 100644--- a/internal/masking/masking.go+++ b/internal/masking/masking.go@@ -31,7 +31,8 @@ func optionsCtx(ctx context.Context) redact.Options {o := redact.Options{Secrets: st.Redact, Personal: st.RedactPersonal, Kinds: st.RedactKinds, Words: st.RedactWords, WordsAnyCase: st.RedactWordsAnyCase, Rules: st.RedactRules,Actions: st.RedactActions, Allow: st.RedactAllow, Shape: st.RedactShape}if st.RedactModel {- o.Model = model(ctx, picked(st))+ m := picked(st)+ o.Model = model(ctx, m, pii.Threshold(st.RedactModelScore, m))}return o}@@ -40,16 +41,25 @@ func optionsCtx(ctx context.Context) redact.Options {var picked = func(st settings.Settings) pii.Model { return pii.Of(pii.Choose(st.RedactModelName, true)) }// model is the local model m (internal/pii) as redact's Model, waited for-// until ctx ends. One that can't answer, or isn't waited for any longer,-// panics, and the request is held back (gwmw.Safely) rather than sent-// with only the rules' masking: the user asked for the model's.-func model(ctx context.Context, m pii.Model) redact.Model {+// until ctx ends, keeping what it is at least min sure of. One that can't+// answer, or isn't waited for any longer, panics, and the request is held+// back (gwmw.Safely) rather than sent with only the rules' masking: the+// user asked for the model's.+func model(ctx context.Context, m pii.Model, min float64) redact.Model {return func(texts []string) [][]redact.Found {got, err := m.Find(ctx, texts)if err != nil {panic(err)}- return got+ out := make([][]redact.Found, len(got))+ for i, fs := range got {+ for _, f := range fs {+ if f.Score >= min {+ out[i] = append(out[i], f)+ }+ }+ }+ return out}}diff --git a/internal/masking/masking_test.go b/internal/masking/masking_test.goindex 937f9d53e..b51de33dc 100644--- a/internal/masking/masking_test.go+++ b/internal/masking/masking_test.go@@ -1,9 +1,12 @@package maskingimport (+ "context"+ "slices""testing""github.com/yetone/magpie/internal/pii"+ "github.com/yetone/magpie/internal/redact""github.com/yetone/magpie/internal/settings")@@ -16,3 +19,39 @@ func TestMaskingUsesThePickedModel(t *testing.T) {}}}++// scored is a model that finds three names, as sure as their scores.+type scored struct{ pii.Model }++func (scored) ID() string { return pii.AstrLinkGuard }+func (scored) MinScore() float64 { return 0.8 }+func (scored) Find(context.Context, []string) ([][]redact.Found, error) {+ return [][]redact.Found{{{Start: 0, End: 1, Label: "private_person", Score: 0.6},+ {Start: 2, End: 3, Label: "private_person", Score: 0.85}, {Start: 4, End: 5, Label: "private_person", Score: 0.95}}}, nil+}++// What the model is less sure of than the threshold the user set for it+// goes as written; with none set, or one out of range, its own default+// (@Yayoi_no_yume on X).+func TestModelThreshold(t *testing.T) {+ for _, c := range []struct {+ set map[string]float64+ want []int+ }{+ {nil, []int{2, 4}},+ {map[string]float64{pii.AstrLinkGuard: 0.9}, []int{4}},+ {map[string]float64{pii.AstrLinkGuard: 0.5}, []int{0, 2, 4}},+ {map[string]float64{pii.PrivacyFilter: 0.95}, []int{2, 4}},+ {map[string]float64{pii.AstrLinkGuard: 0.3}, []int{2, 4}},+ {map[string]float64{pii.AstrLinkGuard: 1}, []int{2, 4}},+ } {+ var m pii.Model = scored{}+ var got []int+ for _, f := range model(context.Background(), m, pii.Threshold(c.set, m))([]string{"a b c"})[0] {+ got = append(got, f.Start)+ }+ if !slices.Equal(got, c.want) {+ t.Errorf("threshold set %v: kept %v, want %v", c.set, got, c.want)+ }+ }+}diff --git a/internal/pii/astrlink.go b/internal/pii/astrlink.goindex da4d1364c..142dbdfa7 100644--- a/internal/pii/astrlink.go+++ b/internal/pii/astrlink.go@@ -209,7 +209,10 @@ func guardClassifyTurn(texts []string) ([][]redact.Found, error) {out := make([][]redact.Found, len(texts))for i, t := range texts {for _, s := range m.Find(t) {- out[i] = append(out[i], redact.Found{Start: s.Start, End: s.End, Label: guardLabel(s.Label)})+ // the score as the cache file keeps it, so a restart doesn't+ // move it over or under the threshold+ out[i] = append(out[i], redact.Found{Start: s.Start, End: s.End, Label: guardLabel(s.Label),+ Score: float64(float32(s.Score))})}}gsrv.mu.Lock()@@ -291,6 +294,8 @@ func (guardModel) Installed() bool { return guardInstalled() }func (guardModel) Size() int64 { return guardSize() }+func (guardModel) MinScore() float64 { return guard.MinScore }+func (guardModel) Find(ctx context.Context, texts []string) ([][]redact.Found, error) {return guardBackend.find(ctx, texts)}diff --git a/internal/pii/cache.go b/internal/pii/cache.goindex 867e14eb4..6b0a318fc 100644--- a/internal/pii/cache.go+++ b/internal/pii/cache.go@@ -22,9 +22,15 @@ import (// model it came from, so another model's answers are never read as this// one's, and lives with the model, in magpie's cache (Remove deletes it).-// cacheMagic starts the file; a file that doesn't start with it, or ends-// part way, is read as empty.-const cacheMagic = "pfound01"+// cacheMagic starts the file; a file that doesn't start with it (or+// cacheMagic1), or ends part way, is read as empty.+const cacheMagic = "pfound02"++// cacheMagic1 starts a file written before a span kept its score, and+// only what was over the model's default threshold. Its spans are read+// as sure (score 1), masked at any threshold as they were before; a+// text not in it is asked about with its scores.+const cacheMagic1 = "pfound01"func cachePath() string { return filepath.Join(dir(), "found-"+modelSHA[:12]+".bin") }@@ -123,7 +129,8 @@ func (bk *backend) saveCache() error {}// encodeCache is c, oldest first: the magic, a count, then each entry's-// hash, its spans' count and each span's start, end and label.+// hash, its spans' count and each span's start, end, score (a float32)+// and label.func encodeCache(c *lru) []byte {var b bytes.Bufferb.WriteString(cacheMagic)@@ -140,6 +147,7 @@ func encodeCache(c *lru) []byte {}put(uint32(f.Start))put(uint32(f.End))+ put(float32(f.Score))b.WriteByte(byte(len(label)))b.WriteString(label)}@@ -150,7 +158,8 @@ func encodeCache(c *lru) []byte {var errBadCache = errors.New("not a cache file")func decodeCache(b []byte) ([]entry, error) {- if !bytes.HasPrefix(b, []byte(cacheMagic)) {+ scored := bytes.HasPrefix(b, []byte(cacheMagic))+ if !scored && !bytes.HasPrefix(b, []byte(cacheMagic1)) {return nil, errBadCache}r := bufio.NewReader(bytes.NewReader(b[len(cacheMagic):]))@@ -176,6 +185,15 @@ func decodeCache(b []byte) ([]entry, error) {if err := binary.Read(r, binary.LittleEndian, &se); err != nil {return nil, err}+ score := float32(1)+ if scored {+ if err := binary.Read(r, binary.LittleEndian, &score); err != nil {+ return nil, err+ }+ if !(score >= 0 && score <= 1) {+ return nil, errBadCache+ }+ }l, err := r.ReadByte()if err != nil {return nil, err@@ -187,7 +205,7 @@ func decodeCache(b []byte) ([]entry, error) {if se[0] >= se[1] {return nil, errBadCache}- e.fs = append(e.fs, redact.Found{Start: int(se[0]), End: int(se[1]), Label: string(label)})+ e.fs = append(e.fs, redact.Found{Start: int(se[0]), End: int(se[1]), Label: string(label), Score: float64(score)})}out = append(out, e)}diff --git a/internal/pii/find.go b/internal/pii/find.goindex 56d517f14..e0265d6af 100644--- a/internal/pii/find.go+++ b/internal/pii/find.go@@ -206,7 +206,7 @@ func (b *backend) known(texts []string, got map[[32]byte][]redact.Found) (out []break}for _, f := range fs {- all = append(all, redact.Found{Start: f.Start + p.at, End: f.End + p.at, Label: f.Label})+ all = append(all, redact.Found{Start: f.Start + p.at, End: f.End + p.at, Label: f.Label, Score: f.Score})}}if !ok {@@ -312,6 +312,8 @@ func merge(fs []redact.Found) []redact.Found {last.Label = f.Label}last.End = max(last.End, f.End)+ // as sure as the surer: what either would mask is masked+ last.Score = max(last.Score, f.Score)continue}}@@ -396,7 +398,7 @@ func within(fs []redact.Found, a, b int) []redact.Found {for _, f := range fs {s, e := max(f.Start, a), min(f.End, b)if s < e {- out = append(out, redact.Found{Start: s - a, End: e - a, Label: f.Label})+ out = append(out, redact.Found{Start: s - a, End: e - a, Label: f.Label, Score: f.Score})}}return outdiff --git a/internal/pii/guard/guard.go b/internal/pii/guard/guard.goindex 13a52847e..09fe5e0b4 100644--- a/internal/pii/guard/guard.go+++ b/internal/pii/guard/guard.go@@ -38,10 +38,14 @@ type Span struct {Score float64}-// MinScore is the probability under which a span is dropped: the-// publisher's default.+// MinScore is the publisher's default threshold, under which masking+// drops a span unless the user set another (internal/masking).const MinScore = 0.8+// Floor is the probability under which Find drops a span: the lowest+// threshold a user can set.+const Floor = 0.5+// window is how many tokens the model reads at once, besides <s> and// </s>; a longer text is read in windows overlapping by overlap, each// token labelled by the window it is furthest inside.@@ -66,7 +70,7 @@ func Load(dir string) (*Model, error) {return &Model{tok, nt}, nil}-// Find is what the model finds in text, at MinScore.+// Find is what the model finds in text, at Floor, each span with its score.func (m *Model) Find(text string) []Span {toks := m.tok.encode(text)n := len(toks)@@ -118,7 +122,7 @@ func (m *Model) Find(text string) []Span {// decode reads the tokens' BIO tags (lab, by index into labels) into// spans: a B tag, or an I tag of another kind than the span before it,// starts one; an I tag of the same kind extends it; O ends it. A span's-// score is its least token's, and those under MinScore are dropped.+// score is its least token's, and those under Floor are dropped.// Tokens of no text (a lone "▁") neither start nor end a span.func decode(toks []token, lab []int, prob []float64) []Span {var spans []Span@@ -143,7 +147,7 @@ func decode(toks []token, lab []int, prob []float64) []Span {}out := spans[:0]for _, s := range spans {- if s.Score >= MinScore {+ if s.Score >= Floor {out = append(out, s)}}diff --git a/internal/pii/guard/guard_test.go b/internal/pii/guard/guard_test.goindex f6f51a898..1f5b6eaf3 100644--- a/internal/pii/guard/guard_test.go+++ b/internal/pii/guard/guard_test.go@@ -27,10 +27,16 @@ func TestDecodeBIO(t *testing.T) {tag("I-private_person")}prob := []float64{1, .99, .9, .95, 1, 1, .2, .97, .85, .9, .7}got := decode(toks, lab, prob)- want := []Span{{5, 11, "email", .9}, {20, 33, "phone", .85}}+ want := []Span{{5, 11, "email", .9}, {20, 33, "phone", .85}, {35, 38, "private_person", .7}}if !reflect.DeepEqual(got, want) {t.Fatalf("decode = %v, want %v", got, want)}+ // One under Floor is dropped; masking drops those under the+ // threshold set, MinScore or the user's, from what is left.+ prob[10] = Floor - .01+ if got := decode(toks, lab, prob); !reflect.DeepEqual(got, want[:2]) {+ t.Fatalf("under Floor: %v", got)+ }// A B tag right after a span of its kind starts another.lab[3] = tag("B-email")if got := decode(toks, lab, prob); len(got) != 3 || got[0].End != 7 || got[1].Start != 7 {diff --git a/internal/pii/pieces_test.go b/internal/pii/pieces_test.goindex dfdb16009..97d291339 100644--- a/internal/pii/pieces_test.go+++ b/internal/pii/pieces_test.go@@ -80,17 +80,21 @@ func TestPiecesCut(t *testing.T) {// Spans from overlapping pieces: one found twice is one, one cut by a// piece's end and whole in the next is the whole, labels kept apart where-// spans only touch, and of one label joined.+// spans only touch, and of one label joined, as sure as the surer.func TestMergeAcrossPieces(t *testing.T) {email := func(s, e int) redact.Found { return redact.Found{Start: s, End: e, Label: "private_email"} }+ sure := email(209, 222)+ sure.Score = .99got := merge([]redact.Found{person(100, 105), person(10, 15), person(100, 105), // twice, from two piecesperson(200, 205), person(200, 211), // cut by a piece's end, whole in the next- email(209, 222), // runs into the name: one, the longer's label+ sure, // runs into the name: one, the longer's labelperson(300, 305), email(305, 310), // touching, two labelsperson(400, 405), person(405, 410), // touching, one label})- want := []redact.Found{person(10, 15), person(100, 105), email(200, 222), person(300, 305), email(305, 310), person(400, 410)}+ joined := email(200, 222)+ joined.Score = sure.Score+ want := []redact.Found{person(10, 15), person(100, 105), joined, person(300, 305), email(305, 310), person(400, 410)}if !reflect.DeepEqual(got, want) {t.Fatalf("merge =\n%v\nwant\n%v", got, want)}diff --git a/internal/pii/pii.go b/internal/pii/pii.goindex e87869f7a..846dcf11f 100644--- a/internal/pii/pii.go+++ b/internal/pii/pii.go@@ -35,6 +35,7 @@ import ("time""github.com/yetone/magpie/internal/appdir"+ "github.com/yetone/magpie/internal/pii/guard""github.com/yetone/magpie/internal/redact")@@ -185,6 +186,9 @@ type Model interface {Installed() bool// Size is how much turning it on downloads.Size() int64+ // MinScore is its default threshold: masking drops what it is less+ // sure of, unless the user set another.+ MinScore() float64Find(ctx context.Context, texts []string) ([][]redact.Found, error)Now() Status// Download fetches it in the background and calls done once it is on@@ -201,6 +205,21 @@ type Model interface {// Models are the local models, Privacy Filter first.func Models() []Model { return []Model{pfModel{}, guardModel{}} }+// Floor is the lowest threshold a user can set (settings.RedactModelScore):+// what either model sends is at least this sure. pf-serve's default+// threshold is 0.5, and AstrLink Guard's Find keeps what is (guard.Floor).+const Floor = guard.Floor++// Threshold is model m's threshold of those the user set (by model id,+// settings.RedactModelScore), or its own where none is, or one is out of+// [Floor, 1).+func Threshold(set map[string]float64, m Model) float64 {+ if v, ok := set[m.ID()]; ok && v >= Floor && v < 1 {+ return v+ }+ return m.MinScore()+}+// Of is the model id names; AstrLink Guard's for an id there's none of.func Of(id string) Model {if id == PrivacyFilter {@@ -233,6 +252,7 @@ func (pfModel) ID() string { return PrivacyFilter }func (pfModel) Supported() bool { return Supported() }func (pfModel) Installed() bool { return Installed() }func (pfModel) Size() int64 { return Size() }+func (pfModel) MinScore() float64 { return 0.5 }func (pfModel) Now() Status { return Now() }func (pfModel) Download(done func()) error { return Download(done) }func (pfModel) Cancel() { Cancel() }diff --git a/internal/pii/serve.go b/internal/pii/serve.goindex e42a346e0..0565393c1 100644--- a/internal/pii/serve.go+++ b/internal/pii/serve.go@@ -8,6 +8,7 @@ import ("errors""fmt""io"+ "math""os""os/exec""regexp"@@ -638,9 +639,10 @@ func (h *helper) classify(texts []string) ([][]redact.Found, error) {}for j := 0; j < n && r.err == nil; j++ {start, end := int32(r.u32()), int32(r.u32())- r.u32() // the score: pf-serve sends only those over its threshold+ // pf-serve sends only those over its default threshold, 0.5: Floor+ score := math.Float32frombits(r.u32())label := string(r.bytes(int(r.u8())))- out[i] = append(out[i], redact.Found{Start: int(start), End: int(end), Label: label})+ out[i] = append(out[i], redact.Found{Start: int(start), End: int(end), Label: label, Score: float64(score)})}}if status != 0 && r.err == nil {diff --git a/internal/pii/serve_test.go b/internal/pii/serve_test.goindex 25fc50b59..17910c618 100644--- a/internal/pii/serve_test.go+++ b/internal/pii/serve_test.go@@ -14,7 +14,10 @@ import ("github.com/yetone/magpie/internal/redact")-func person(s, e int) redact.Found { return redact.Found{Start: s, End: e, Label: "private_person"} }+// person is a name at [s, e), as sure as the fake pf-serve says (main_test.go).+func person(s, e int) redact.Found {+ return redact.Found{Start: s, End: e, Label: "private_person", Score: float64(float32(0.98))}+}// Short texts go to pf-serve joined, in one request, and what it finds// comes back to each text at that text's own offsets; a text seen beforediff --git a/internal/pii/turn_test.go b/internal/pii/turn_test.goindex 80d272409..f8aa2e995 100644--- a/internal/pii/turn_test.go+++ b/internal/pii/turn_test.go@@ -1,8 +1,10 @@package piiimport (+ "bytes""context""crypto/sha256"+ "encoding/binary""errors""os""reflect"@@ -216,6 +218,33 @@ func TestBadCacheFileIsReadAsEmpty(t *testing.T) {}}+// A file written before a span kept its score (cacheMagic1) is read, its+// spans as sure: masked at any threshold, as they were before, and not+// asked about again.+func TestOldCacheFileIsReadAsSure(t *testing.T) {+ log := fakeInstall(t, "ok", false)+ var b bytes.Buffer+ b.WriteString(cacheMagic1)+ k := sumOf("hi Alice")+ binary.Write(&b, binary.LittleEndian, uint32(1))+ b.Write(k[:])+ binary.Write(&b, binary.LittleEndian, uint16(1))+ binary.Write(&b, binary.LittleEndian, [2]uint32{3, 8})+ b.WriteByte(byte(len("private_person")))+ b.WriteString("private_person")+ if err := os.WriteFile(cachePath(), b.Bytes(), 0o644); err != nil {+ t.Fatal(err)+ }+ restart(t)+ got, err := Find([]string{"hi Alice"})+ if err != nil || !reflect.DeepEqual(got, [][]redact.Found{{{Start: 3, End: 8, Label: "private_person", Score: 1}}}) {+ t.Fatalf("Find = %v, %v", got, err)+ }+ if reqs := fakeLines(t, log, "request"); len(reqs) != 0 {+ t.Fatalf("asked again: %q", reqs)+ }+}+// A span read back that doesn't fit its text is asked about again.func TestCachedSpanOutsideItsTextIsAskedAgain(t *testing.T) {log := fakeInstall(t, "ok", false)diff --git a/internal/redact/eval_test.go b/internal/redact/eval_test.goindex 5937529ff..712e574c5 100644--- a/internal/redact/eval_test.go+++ b/internal/redact/eval_test.go@@ -57,7 +57,7 @@ func TestMaskEval(t *testing.T) {asked++}}- evalModel(t, rec, em.known)+ evalModel(t, rec, em.known, pii.Of(em.id).MinScore())})}if os.Getenv("MAGPIE_EVAL_CACHE") != "" && asked == 0 {@@ -75,7 +75,9 @@ var evalModels = []struct {{pii.AstrLinkGuard, "testdata/eval-astrlink.json", func(v evalValue) string { return v.GuardKnown }},}-func evalModel(t *testing.T, rec map[string]recorded, known func(evalValue) string) {+// evalModel scores the recording rec, keeping what the model was at+// least min sure of, as masking does at that threshold.+func evalModel(t *testing.T, rec map[string]recorded, known func(evalValue) string, min float64) {var stale []stringfor _, c := range evalCases {if r, ok := rec[c.ID]; !ok || r.Text != textHash(c.Text) {@@ -85,6 +87,17 @@ func evalModel(t *testing.T, rec map[string]recorded, known func(evalValue) striif len(stale) > 0 {t.Fatalf("the model's recording is not of cases %v; record it again with MAGPIE_EVAL_CACHE (see TestMaskEval)", stale)}+ kept := map[string]recorded{}+ for id, r := range rec {+ fs := []redact.Found{}+ for _, f := range r.Found {+ if f.Score >= min {+ fs = append(fs, f)+ }+ }+ kept[id] = recorded{Text: r.Text, Found: fs}+ }+ rec = keptreplay := func(texts []string) [][]redact.Found {out := make([][]redact.Found, len(texts))for i, s := range texts {diff --git a/internal/redact/model.go b/internal/redact/model.goindex c3812ad52..583da4010 100644--- a/internal/redact/model.go+++ b/internal/redact/model.go@@ -13,10 +13,13 @@ import (// rules', under the same switches: a name, an address, a phone number, an// email or an account number is personal data, a secret is a secret.-// Found is a span the model found in a text: byte offsets, and its label.+// Found is a span the model found in a text: byte offsets, its label,+// and how sure the model is of it (0 to 1; masking drops what is under+// the threshold the user set, internal/masking).type Found struct {Start, End intLabel string+ Score float64 `json:"score,omitempty"`}// Model finds spans in each of texts, in order. It never fails: one thatdiff --git a/internal/redact/model_test.go b/internal/redact/model_test.goindex 4da54aaf4..82b065937 100644--- a/internal/redact/model_test.go+++ b/internal/redact/model_test.go@@ -29,7 +29,7 @@ func (f *fakeModel) find(texts []string) [][]Found {if a > 0 && t[a-1] == ' ' {a--}- out[i] = append(out[i], Found{a, at + j + len(v), label})+ out[i] = append(out[i], Found{Start: a, End: at + j + len(v), Label: label})at += j + len(v)}}@@ -104,7 +104,7 @@ func TestModelSpansFiltered(t *testing.T) {}// a span outside the text, or splitting a character, is no spanbad := func([]string) [][]Found {- return [][]Found{{{-1, 3, "secret"}, {0, 99, "secret"}, {1, 4, "private_person"}}}+ return [][]Found{{{Start: -1, End: 3, Label: "secret"}, {Start: 0, End: 99, Label: "secret"}, {Start: 1, End: 4, Label: "private_person"}}}}if out, _ := Mask("é名字abc", Options{Personal: true, Secrets: true, Model: bad}); out != "é名字abc" {t.Fatalf("bad spans: %q", out)diff --git a/internal/redact/testdata/eval-astrlink.json b/internal/redact/testdata/eval-astrlink.jsonindex 856159dde..750ec6215 100644--- a/internal/redact/testdata/eval-astrlink.json+++ b/internal/redact/testdata/eval-astrlink.json@@ -10,10 +10,29 @@"assistant-prose": {"text": "2f09c57d2bcd4a33","found": [+ {+ "Start": 7,+ "End": 13,+ "Label": "private_person",+ "score": 0.6422655582427979+ },{"Start": 57,"End": 64,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.8788238167762756+ },+ {+ "Start": 72,+ "End": 73,+ "Label": "private_person",+ "score": 0.5269465446472168+ },+ {+ "Start": 92,+ "End": 94,+ "Label": "private_person",+ "score": 0.5114759802818298}]},@@ -31,12 +50,14 @@{"Start": 31,"End": 148,- "Label": "secret"+ "Label": "secret",+ "score": 0.9962389469146729},{"Start": 150,"End": 180,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.99592125415802}]},@@ -46,12 +67,14 @@{"Start": 30,"End": 49,- "Label": "account_number"+ "Label": "account_number",+ "score": 0.9930477142333984},{"Start": 58,"End": 63,- "Label": "private_date"+ "Label": "private_date",+ "score": 0.9923847317695618}]},@@ -62,15 +85,23 @@"code-keys": {"text": "d5f835bbf0a332b8","found": [+ {+ "Start": 12,+ "End": 50,+ "Label": "secret",+ "score": 0.7978969812393188+ },{"Start": 67,"End": 108,- "Label": "secret"+ "Label": "secret",+ "score": 0.9611479640007019},{"Start": 135,"End": 175,- "Label": "secret"+ "Label": "secret",+ "score": 0.9937745928764343}]},@@ -84,12 +115,14 @@{"Start": 155,"End": 169,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.995353639125824},{"Start": 240,"End": 253,- "Label": "secret"+ "Label": "secret",+ "score": 0.995952844619751}]},@@ -99,7 +132,8 @@{"Start": 45,"End": 54,- "Label": "ip_address"+ "Label": "ip_address",+ "score": 0.9942322373390198}]},@@ -109,12 +143,14 @@{"Start": 56,"End": 62,- "Label": "account_number"+ "Label": "account_number",+ "score": 0.8200094699859619},{"Start": 64,"End": 95,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.9951349496841431}]},@@ -128,12 +164,14 @@{"Start": 15,"End": 63,- "Label": "secret"+ "Label": "secret",+ "score": 0.9948821663856506},{"Start": 76,"End": 90,- "Label": "secret"+ "Label": "secret",+ "score": 0.9570835828781128}]},@@ -143,7 +181,8 @@{"Start": 112,"End": 127,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.9948705434799194}]},@@ -153,7 +192,8 @@{"Start": 14,"End": 30,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.9941899180412292}]},@@ -167,12 +207,14 @@{"Start": 23,"End": 33,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9949986338615417},{"Start": 35,"End": 55,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.9958543181419373}]},@@ -186,7 +228,8 @@{"Start": 10,"End": 55,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.9963787198066711}]},@@ -216,7 +259,8 @@{"Start": 24,"End": 51,- "Label": "account_number"+ "Label": "account_number",+ "score": 0.9904018640518188}]},@@ -226,17 +270,20 @@{"Start": 21,"End": 30,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9956386089324951},{"Start": 41,"End": 62,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.9967396855354309},{"Start": 73,"End": 90,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.9951627850532532}]},@@ -250,12 +297,14 @@{"Start": 40,"End": 52,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.9952855706214905},{"Start": 68,"End": 76,- "Label": "ip_address"+ "Label": "ip_address",+ "score": 0.9939125776290894}]},@@ -273,11 +322,31 @@},"noreply": {"text": "271c7650b6474a19",- "found": []+ "found": [+ {+ "Start": 50,+ "End": 56,+ "Label": "private_email",+ "score": 0.5142685174942017+ },+ {+ "Start": 60,+ "End": 63,+ "Label": "private_email",+ "score": 0.5299198627471924+ }+ ]},"one-letter": {"text": "0ff1555b12fd8004",- "found": []+ "found": [+ {+ "Start": 0,+ "End": 1,+ "Label": "private_person",+ "score": 0.6900092959403992+ }+ ]},"phone-intl": {"text": "28bf596367da1b2b",@@ -285,12 +354,14 @@{"Start": 36,"End": 39,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9818559288978577},{"Start": 43,"End": 58,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.9949420094490051}]},@@ -304,7 +375,8 @@{"Start": 67,"End": 103,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.996593177318573}]},@@ -314,7 +386,8 @@{"Start": 27,"End": 38,- "Label": "secret"+ "Label": "secret",+ "score": 0.9967206716537476}]},@@ -324,7 +397,14 @@{"Start": 13,"End": 27,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9951192736625671+ },+ {+ "Start": 61,+ "End": 67,+ "Label": "private_date",+ "score": 0.6461724638938904}]},@@ -338,7 +418,14 @@{"Start": 46,"End": 98,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.9966269731521606+ },+ {+ "Start": 111,+ "End": 112,+ "Label": "account_number",+ "score": 0.5739760994911194}]},@@ -352,7 +439,8 @@{"Start": 51,"End": 83,- "Label": "secret"+ "Label": "secret",+ "score": 0.9945904016494751}]},@@ -366,18 +454,27 @@{"Start": 26,"End": 31,- "Label": "secret"+ "Label": "secret",+ "score": 0.985226035118103},{"Start": 85,"End": 110,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.9965609908103943}]},"short-values": {"text": "0133fede621b7801",- "found": []+ "found": [+ {+ "Start": 13,+ "End": 14,+ "Label": "private_address",+ "score": 0.6322974562644958+ }+ ]},"sizes-versions": {"text": "e08a850f552a995b",@@ -405,12 +502,14 @@{"Start": 41,"End": 55,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9950873255729675},{"Start": 118,"End": 123,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9923510551452637}]},@@ -420,7 +519,8 @@{"Start": 31,"End": 39,- "Label": "secret"+ "Label": "secret",+ "score": 0.9052966833114624}]},@@ -434,27 +534,38 @@{"Start": 36,"End": 50,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9953241944313049},{"Start": 51,"End": 70,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.9960401058197021+ },+ {+ "Start": 71,+ "End": 88,+ "Label": "private_phone",+ "score": 0.759522557258606},{"Start": 88,"End": 99,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9947472810745239},{"Start": 100,"End": 121,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.9956356287002563},{"Start": 122,"End": 137,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.9943153858184814}]},@@ -484,7 +595,20 @@{"Start": 10,"End": 58,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.9965047836303711+ },+ {+ "Start": 67,+ "End": 70,+ "Label": "account_number",+ "score": 0.5625114440917969+ },+ {+ "Start": 71,+ "End": 79,+ "Label": "ip_address",+ "score": 0.5526543259620667}]},@@ -494,7 +618,8 @@{"Start": 13,"End": 67,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.9906603097915649}]},@@ -504,12 +629,14 @@{"Start": 24,"End": 68,- "Label": "private_address"+ "Label": "private_address",+ "score": 0.9942578077316284},{"Start": 78,"End": 92,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.9951914548873901}]},@@ -523,17 +650,20 @@{"Start": 12,"End": 18,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9953292012214661},{"Start": 28,"End": 39,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.994989275932312},{"Start": 51,"End": 93,- "Label": "private_address"+ "Label": "private_address",+ "score": 0.9943665266036987}]},@@ -547,12 +677,14 @@{"Start": 18,"End": 27,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9911744594573975},{"Start": 51,"End": 57,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.993349015712738}]},@@ -562,7 +694,8 @@{"Start": 19,"End": 29,- "Label": "secret"+ "Label": "secret",+ "score": 0.9968122839927673}]},@@ -572,17 +705,20 @@{"Start": 0,"End": 6,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9937225580215454},{"Start": 6,"End": 15,- "Label": "private_date"+ "Label": "private_date",+ "score": 0.903574526309967},{"Start": 66,"End": 72,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.994594931602478}]},diff --git a/internal/redact/testdata/eval-model.json b/internal/redact/testdata/eval-model.jsonindex e33f25d02..692a591a6 100644--- a/internal/redact/testdata/eval-model.json+++ b/internal/redact/testdata/eval-model.json@@ -13,22 +13,26 @@{"Start": 7,"End": 13,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9996709823608398},{"Start": 57,"End": 64,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9999828338623047},{"Start": 68,"End": 73,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9997822642326355},{"Start": 115,"End": 121,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.8679556846618652}]},@@ -38,17 +42,20 @@{"Start": 39,"End": 48,- "Label": "secret"+ "Label": "secret",+ "score": 0.7519519925117493},{"Start": 195,"End": 204,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.5430417060852051},{"Start": 236,"End": 245,- "Label": "secret"+ "Label": "secret",+ "score": 0.6623177528381348}]},@@ -62,12 +69,14 @@{"Start": 31,"End": 148,- "Label": "secret"+ "Label": "secret",+ "score": 0.9999741315841675},{"Start": 150,"End": 180,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.8646273612976074}]},@@ -81,7 +90,8 @@{"Start": 26,"End": 36,- "Label": "account_number"+ "Label": "account_number",+ "score": 0.9979659914970398}]},@@ -91,17 +101,20 @@{"Start": 12,"End": 50,- "Label": "secret"+ "Label": "secret",+ "score": 0.7302199602127075},{"Start": 67,"End": 108,- "Label": "secret"+ "Label": "secret",+ "score": 0.9999991059303284},{"Start": 135,"End": 175,- "Label": "secret"+ "Label": "secret",+ "score": 0.9993591904640198}]},@@ -115,17 +128,20 @@{"Start": 83,"End": 119,- "Label": "account_number"+ "Label": "account_number",+ "score": 0.994881808757782},{"Start": 155,"End": 169,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9996635317802429},{"Start": 240,"End": 253,- "Label": "secret"+ "Label": "secret",+ "score": 0.999958872795105}]},@@ -135,7 +151,8 @@{"Start": 14,"End": 21,- "Label": "secret"+ "Label": "secret",+ "score": 0.7929887175559998}]},@@ -145,12 +162,14 @@{"Start": 56,"End": 62,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9999889135360718},{"Start": 64,"End": 95,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.9926241636276245}]},@@ -160,7 +179,8 @@{"Start": 8,"End": 72,- "Label": "secret"+ "Label": "secret",+ "score": 0.9951086640357971}]},@@ -170,12 +190,14 @@{"Start": 15,"End": 63,- "Label": "secret"+ "Label": "secret",+ "score": 0.9999755620956421},{"Start": 76,"End": 90,- "Label": "secret"+ "Label": "secret",+ "score": 0.9994818568229675}]},@@ -185,27 +207,32 @@{"Start": 36,"End": 46,- "Label": "account_number"+ "Label": "account_number",+ "score": 0.9865620136260986},{"Start": 54,"End": 67,- "Label": "account_number"+ "Label": "account_number",+ "score": 0.9779103994369507},{"Start": 76,"End": 100,- "Label": "private_date"+ "Label": "private_date",+ "score": 0.9683437943458557},{"Start": 111,"End": 127,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.9999693632125854},{"Start": 138,"End": 142,- "Label": "private_address"+ "Label": "private_address",+ "score": 0.9999595880508423}]},@@ -215,7 +242,8 @@{"Start": 14,"End": 30,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.9822543263435364}]},@@ -225,22 +253,26 @@{"Start": 35,"End": 82,- "Label": "secret"+ "Label": "secret",+ "score": 0.9390993118286133},{"Start": 85,"End": 104,- "Label": "secret"+ "Label": "secret",+ "score": 0.9990366697311401},{"Start": 109,"End": 132,- "Label": "secret"+ "Label": "secret",+ "score": 0.999983549118042},{"Start": 141,"End": 161,- "Label": "secret"+ "Label": "secret",+ "score": 0.852599561214447}]},@@ -250,12 +282,14 @@{"Start": 23,"End": 33,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.999956488609314},{"Start": 35,"End": 55,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.9999942183494568}]},@@ -269,7 +303,8 @@{"Start": 10,"End": 55,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.999872088432312}]},@@ -291,12 +326,14 @@{"Start": 10,"End": 52,- "Label": "secret"+ "Label": "secret",+ "score": 0.9979620575904846},{"Start": 53,"End": 96,- "Label": "secret"+ "Label": "secret",+ "score": 0.9856805205345154}]},@@ -306,7 +343,8 @@{"Start": 4,"End": 40,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.9622886776924133}]},@@ -316,7 +354,8 @@{"Start": 24,"End": 51,- "Label": "account_number"+ "Label": "account_number",+ "score": 0.999630868434906}]},@@ -326,17 +365,20 @@{"Start": 21,"End": 30,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9999991655349731},{"Start": 41,"End": 62,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.9999995231628418},{"Start": 70,"End": 90,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.9999877214431763}]},@@ -350,17 +392,20 @@{"Start": 0,"End": 19,- "Label": "private_date"+ "Label": "private_date",+ "score": 0.6735039949417114},{"Start": 35,"End": 52,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.7250260710716248},{"Start": 68,"End": 76,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.9995778203010559}]},@@ -370,27 +415,32 @@{"Start": 0,"End": 10,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.755682110786438},{"Start": 15,"End": 18,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.7198829054832458},{"Start": 59,"End": 70,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.8445316553115845},{"Start": 86,"End": 90,- "Label": "private_address"+ "Label": "private_address",+ "score": 0.8700133562088013},{"Start": 91,"End": 97,- "Label": "private_date"+ "Label": "private_date",+ "score": 0.8610206246376038}]},@@ -400,32 +450,38 @@{"Start": 0,"End": 19,- "Label": "private_date"+ "Label": "private_date",+ "score": 0.9984123706817627},{"Start": 21,"End": 38,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.9999645352363586},{"Start": 39,"End": 58,- "Label": "private_date"+ "Label": "private_date",+ "score": 0.9472267627716064},{"Start": 60,"End": 78,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.9999365210533142},{"Start": 79,"End": 98,- "Label": "private_date"+ "Label": "private_date",+ "score": 0.9862194061279297},{"Start": 100,"End": 118,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.9995419383049011}]},@@ -435,17 +491,20 @@{"Start": 67,"End": 99,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.7719004154205322},{"Start": 154,"End": 180,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9730507135391235},{"Start": 198,"End": 225,- "Label": "secret"+ "Label": "secret",+ "score": 0.5167003273963928}]},@@ -459,12 +518,14 @@{"Start": 26,"End": 27,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9247977137565613},{"Start": 32,"End": 33,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9909683465957642}]},@@ -474,12 +535,14 @@{"Start": 36,"End": 39,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9999797940254211},{"Start": 43,"End": 58,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.999995768070221}]},@@ -493,12 +556,14 @@{"Start": 67,"End": 103,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.783750057220459},{"Start": 148,"End": 162,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.6178609132766724}]},@@ -508,7 +573,8 @@{"Start": 27,"End": 38,- "Label": "secret"+ "Label": "secret",+ "score": 0.999982476234436}]},@@ -518,7 +584,8 @@{"Start": 13,"End": 27,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9999990463256836}]},@@ -532,17 +599,20 @@{"Start": 29,"End": 44,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.7199236154556274},{"Start": 46,"End": 98,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.9997158050537109},{"Start": 193,"End": 204,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.7728715538978577}]},@@ -552,7 +622,8 @@{"Start": 67,"End": 85,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.8553599119186401}]},@@ -562,7 +633,8 @@{"Start": 0,"End": 83,- "Label": "secret"+ "Label": "secret",+ "score": 0.9803686738014221}]},@@ -572,7 +644,8 @@{"Start": 39,"End": 81,- "Label": "secret"+ "Label": "secret",+ "score": 0.993749737739563}]},@@ -582,17 +655,20 @@{"Start": 26,"End": 34,- "Label": "secret"+ "Label": "secret",+ "score": 0.6450760364532471},{"Start": 52,"End": 67,- "Label": "secret"+ "Label": "secret",+ "score": 0.9475275874137878},{"Start": 130,"End": 156,- "Label": "secret"+ "Label": "secret",+ "score": 0.997515082359314}]},@@ -602,7 +678,8 @@{"Start": 25,"End": 30,- "Label": "private_address"+ "Label": "private_address",+ "score": 0.9985845685005188}]},@@ -616,7 +693,8 @@{"Start": 116,"End": 125,- "Label": "private_address"+ "Label": "private_address",+ "score": 0.7423255443572998}]},@@ -634,12 +712,14 @@{"Start": 8,"End": 26,- "Label": "account_number"+ "Label": "account_number",+ "score": 0.9553443789482117},{"Start": 43,"End": 85,- "Label": "account_number"+ "Label": "account_number",+ "score": 0.8112580180168152}]},@@ -649,12 +729,14 @@{"Start": 41,"End": 55,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9999991059303284},{"Start": 118,"End": 123,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9996482133865356}]},@@ -664,7 +746,8 @@{"Start": 31,"End": 39,- "Label": "account_number"+ "Label": "account_number",+ "score": 0.9996710419654846}]},@@ -674,12 +757,14 @@{"Start": 37,"End": 67,- "Label": "account_number"+ "Label": "account_number",+ "score": 0.9796360731124878},{"Start": 93,"End": 121,- "Label": "account_number"+ "Label": "account_number",+ "score": 0.9456615447998047}]},@@ -689,32 +774,38 @@{"Start": 36,"End": 50,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9950388669967651},{"Start": 50,"End": 70,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.9999969601631165},{"Start": 70,"End": 87,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.9999991059303284},{"Start": 88,"End": 99,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9999955892562866},{"Start": 100,"End": 121,- "Label": "private_email"+ "Label": "private_email",+ "score": 0.9999992251396179},{"Start": 121,"End": 137,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.9999945163726807}]},@@ -728,17 +819,20 @@{"Start": 37,"End": 60,- "Label": "account_number"+ "Label": "account_number",+ "score": 0.9922814965248108},{"Start": 68,"End": 95,- "Label": "secret"+ "Label": "secret",+ "score": 0.8133253455162048},{"Start": 105,"End": 126,- "Label": "secret"+ "Label": "secret",+ "score": 0.9985485076904297}]},@@ -748,7 +842,8 @@{"Start": 82,"End": 108,- "Label": "secret"+ "Label": "secret",+ "score": 0.6138984560966492}]},@@ -766,12 +861,14 @@{"Start": 10,"End": 58,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.9996631741523743},{"Start": 63,"End": 79,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.9925283789634705}]},@@ -781,7 +878,8 @@{"Start": 27,"End": 55,- "Label": "secret"+ "Label": "secret",+ "score": 0.8250589370727539}]},@@ -791,12 +889,14 @@{"Start": 24,"End": 68,- "Label": "private_address"+ "Label": "private_address",+ "score": 0.9999116659164429},{"Start": 78,"End": 92,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.998772144317627}]},@@ -806,12 +906,14 @@{"Start": 11,"End": 54,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.8804416060447693},{"Start": 84,"End": 116,- "Label": "private_url"+ "Label": "private_url",+ "score": 0.9207233190536499}]},@@ -821,17 +923,20 @@{"Start": 12,"End": 18,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9999974370002747},{"Start": 28,"End": 39,- "Label": "private_phone"+ "Label": "private_phone",+ "score": 0.9999875426292419},{"Start": 51,"End": 93,- "Label": "private_address"+ "Label": "private_address",+ "score": 0.9999922513961792}]},@@ -841,7 +946,8 @@{"Start": 13,"End": 31,- "Label": "account_number"+ "Label": "account_number",+ "score": 0.9986587166786194}]},@@ -851,12 +957,14 @@{"Start": 18,"End": 39,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9349985718727112},{"Start": 51,"End": 63,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.9998366832733154}]},@@ -866,7 +974,8 @@{"Start": 19,"End": 29,- "Label": "secret"+ "Label": "secret",+ "score": 0.9998962879180908}]},@@ -876,7 +985,8 @@{"Start": 0,"End": 15,- "Label": "private_person"+ "Label": "private_person",+ "score": 0.8192970752716064}]},diff --git a/internal/settings/settings.go b/internal/settings/settings.goindex 705077818..f62e6f7d4 100644--- a/internal/settings/settings.go+++ b/internal/settings/settings.go@@ -106,14 +106,19 @@ type Settings struct {// (redact.Options.Shape; Charles Cao on X). RedactModelName is the// local model picked (pii.PrivacyFilter, pii.AstrLinkGuard); "" for// none picked, which pii.Choose reads. It is this computer's, as- // RedactModel is.- Redact bool `json:"redact,omitempty"`- RedactShape bool `json:"redactShape,omitempty"`- RedactModel bool `json:"redactModel,omitempty"`- RedactModelName string `json:"redactModelName,omitempty"`- RedactPersonal bool `json:"redactPersonal,omitempty"`- RedactKinds map[string]bool `json:"redactKinds,omitempty"`- RedactWords []string `json:"redactWords,omitempty"`+ // RedactModel is. RedactModelScore is each model's threshold the user+ // set, by its id: masking drops what the model is less sure of (an+ // AstrLink Guard user on X found 0.85 to 0.95 right for it). A model+ // not in it has its own default (pii.Model.MinScore); this+ // computer's too.+ Redact bool `json:"redact,omitempty"`+ RedactModelScore map[string]float64 `json:"redactModelScore,omitempty"`+ RedactShape bool `json:"redactShape,omitempty"`+ RedactModel bool `json:"redactModel,omitempty"`+ RedactModelName string `json:"redactModelName,omitempty"`+ RedactPersonal bool `json:"redactPersonal,omitempty"`+ RedactKinds map[string]bool `json:"redactKinds,omitempty"`+ RedactWords []string `json:"redactWords,omitempty"`// RedactWordsAnyCase matches RedactWords in any case (ZoeCats on// Discord); off, the default, a word is matched as written.RedactWordsAnyCase bool `json:"redactWordsAnyCase,omitempty"`@@ -1116,7 +1121,7 @@ func (s *Settings) KeepOwn(cur Settings) {s.LAN, s.AllowIPs = cur.LAN, cur.AllowIPss.FetchPrivate = cur.FetchPrivates.SessionTerminal = cur.SessionTerminal- s.RedactModel, s.RedactModelName = cur.RedactModel, cur.RedactModelName+ s.RedactModel, s.RedactModelName, s.RedactModelScore = cur.RedactModel, cur.RedactModelName, cur.RedactModelScores.ClaudeRunsMost = cur.ClaudeRunsMosts.RequestArchiveLocal, s.RequestArchiveLocalMB = cur.RequestArchiveLocal, cur.RequestArchiveLocalMB}@@ -1449,6 +1454,28 @@ func SetRedactModelName(name string) error {return save(s, false)}+// SetRedactModelScore keeps score as the local model id's threshold; 0+// is back to the model's own.+func SetRedactModelScore(id string, score float64) error {+ fileMu.Lock()+ defer fileMu.Unlock()+ s := load()+ m := maps.Clone(s.RedactModelScore)+ if m == nil {+ m = map[string]float64{}+ }+ if score == 0 {+ delete(m, id)+ } else {+ m[id] = score+ }+ if len(m) == 0 {+ m = nil+ }+ s.RedactModelScore = m+ return save(s, false)+}+// UseRedactModel turns the local masking model on as name, the model// the user picked, once that model is downloaded.func UseRedactModel(name string) error {