Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

redact: ids, digests and 毛利 go as written; the slider reaches known values; a user name goes as a made-up name

yetone189db9ba5b49
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index 494f3bdf9..1c7c29855 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -137,9 +137,9 @@ recorded as a prediction failure (`noPrediction`).
## Runtime path
1. `handle(proto)` reads the body and the model (`requestBody`, `requestModel`), then calls `serveAgent`. Every route that reads an agent's body (each API, count_tokens, Gemini's, embeddings, images, videos, System One, the Codex backend path) reads it through `readBoundedRequestBody` in [`request_bounds.go`](../../internal/gateway/request_bounds.go). It decodes a `Content-Encoding` of `gzip` or `zstd` (`decodedBody`; Codex sends zstd to a custom provider's `/v1/responses` too, #1223) and drops the header, so the provider is sent plain JSON, with one exception: a ChatGPT account's turn (POST `CodexBase`/responses, from the accounts loop in `forwardOnce` or relayed by `codexUpstreamOn`) is compressed again with zstd and a window of at most 2 MiB, as Codex's own client sends it (`provider.CodexCompressed`; openai/codex `responses_request_compression` and `http-client/src/request.rs`). It is compressed once the body is final, and again on each try. `/responses/compact`, an API key's requests and every other provider stay plain. Sent plain, a 1.1 MB text-only turn was answered 503 (ylorn on X). The plain body is what magpie keeps and shows, and `codexDropped` judges a reset by the bytes that went on the wire. `codexDropped` also reads Envoy's 507 "exceeded request buffer limit while retrying upstream" (the backend failed and the front couldn't retry a body that large) as that reset, at any size, told as a 503 so magpie, Codex and pi ask again and nobody rests (#2034). The size limit (`defaultBodyLimit`, 512 MB) holds for the decoded bytes (413), and any other encoding is refused 415. It was 128 MB until a long Codex session that had read many images went past it decoded, its inline screenshots resent each turn, and got the 413 on all five of Codex's retries (#1422, `TestImageHeavyCodexTurnIsRead`). magpie's own 413 says it is magpie's gateway limit, with the size, and that the provider was not asked (`overLimit`); `writeError` adds to any 413 it isn't holding that this is a byte limit, not the model's context, and to `/compact` or start a new conversation (`TestOverLimitSaysWhoseAndWhatToDo`). Every shape but Gemini's types a 413 `request_too_large`, Anthropic's own type, which pi's overflow table knows: pi compacts and tries once more rather than stopping on an OpenAI-shaped `api_error` (@wilsonxuuuu on X, `TestPayloadTooLargeSaysRequestTooLargeToPi`). `keyLimited` reads a limited key's compressed body the same way, so its reservation is the decoded size. `serveAgent` runs the [middleware](gateway-middleware.md) chain, then `serve`.
-2. `serve` redacts secrets in the request (`redacted`; `redact.MaskJSON` leaves ids alone, and in any text a backend's id (a UUID, or 32 hex digits or more), or an element token cua-driver hands the model for computer use (`s0000002a:14`, #3782), goes as written: no rule, model label or value known from before masks a piece of one (a phone number at the end of `…-a13800138000`, a card number across a UUID's groups), and a whole one is masked only by the user's own words and rules or as a credential, by the password, API key and token rules or a secret rule whose field is named for one (`api_key=`, `refresh_token`), never one named for an id, a cursor, a request or a digest (`secret_id`, `next_page_token`, `Idempotency-Key`, `secret_hash`), nor cua-driver's own `element_token` (any other name with `element` in it, `element_api_key`, is a credential's as before), a bare `key`, a serial number, or what the model takes a UUID or an element token for, nor an element token known from before (one the model labelled in prose a turn earlier) (`noIDs`, [`internal/redact/ids.go`](../../internal/redact/ids.go)); and a tool's name and its MCP server's (`namespace`, `server_label`, `server_name`, #3143); the request's tool definitions and output schemas, found by their place in each protocol (top-level `tools`, `tool_choice`, `response_format`, `text`, `generationConfig`, `stream_options`, the client's ids in `client_metadata` and the like, and the tools of a Responses `additional_tools` or `tool_search_output` item), go as written save their descriptions, so a word the user masks can't break a grammar or an enum (#1911); and what the vendor sealed with a signature, a thinking block, Gemini's thoughts or Responses' encrypted reasoning, goes back as the vendor wrote it: only the values masked elsewhere in the request, and secrets masked before, are put back as their placeholders; a value masked once, at least 8 characters long, is masked again wherever a request has it, as an exact match whatever words are around it, so what the vendor wrote about a placeholder doesn't go back with the value in it, `redact/known.go`; a text masked for a vendor before, such as a system prompt or a turn already sent, is masked as it was then, so a value a rule first finds later in a conversation doesn't change the prompt the vendor has cached and rewrite all of it to the cache, `sentKnownSpans`), resolves the model to a provider or a group, and refuses what can't be served. The reply has the values put back on its way to the agent (`redact.Writer`, a stream event by event, a whole body at its end): a placeholder the model wrote as it was given, or as models rewrite one (spaces inside the braces, a brace or both dropped, another case), is the value again when what is left names one magpie made, and one split across events is held until it is whole (`redact.Restore`, `partialTail`, [`internal/redact/redact.go`](../../internal/redact/redact.go)). A request whose provider and fallbacks, or whose group's models, all skip redaction (`unredactedRoute`, `provider.SkipsRedaction`: set *Send requests unmasked*, `Unredacted`, with every address on this machine or the local network) goes to them as the agent wrote it, and is masked only for a body an observer wants or the request archive keeps, which may go elsewhere (`bodiesKeptAway`); without those it isn't masked at all, and the log and recent calls keep it as it went, since a request is masked only once its route is known (#3973: masking a body whose masked copy was thrown away took up to minutes of magpie's own time). A request turned away before it is routed is logged masked (`maskedCopy`). count_tokens and embeddings follow the same rule. What a routing group's Jev is told of the turn (the message, the user's message before it and the reply to it) is masked all the same unless Jev's own provider skips redaction too (`maskedState` in `askJev`, [`decide.go`](../../internal/gateway/decide.go)): a remote Jev is none of the group's members. The masking is the built-in plugin `redact`'s (`gwmw.Masker`, [Plugin runtime](plugin-runtime.md)); when the settings ask for masking and no masker serves, or it fails, the request is a 503 and nothing is sent (`redacted`, `gwmw.Masking`).
- - With *Mask personal data* on, each kind in `redact.Categories` ([`internal/redact/more.go`](../../internal/redact/more.go)) is masked unless the user turned it off (`redactKinds` in settings, `Options.Kinds`). On by default: US SSNs, passport numbers, IBANs that pass mod 97, birthdays next to a word that says so (a date over a year ago: one in the last year or ahead is the day an agent runs on, `dateOK`); the word must be right before the date, so a log line's or a file name's date is none, and a time beside the date doesn't make it a log's: a date of birth stored as a time (`1990-05-11T16:00:00.000Z`) is one (`TestBirthdayLogDates`; #3878), MAC addresses, and serial numbers next to S/N or the like. Off by default: the user name in a home folder, user@host in a prompt or ssh command, public IPv4 and IPv6 addresses (never private, loopback, link-local, CGNAT or documentation ranges) and cloud bucket names, since agents send them in nearly every request and the model must copy the placeholder exactly inside a path or URL. Context rules run only near their words (`rule.matches`). What reads as an address but ends in a file's extension that no domain has (`…@2x.jpg`, `[email protected]`; `fileExt` in `realEmail`), or a scale before an extension that is a domain too (`[email protected]`, `[email protected]`; `scaleExt`), is a file's name, for the rule and the local model alike (#3029). `[email protected]` is still an address. The Settings rows come from `REDACT_KINDS` in app.js, which `TestRedactKindsAreRedacts` holds to the same ids and defaults.
- - With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken, nor a snake_case word as a name (Codex's `sequential_cutoff`, #3144), nor as a secret what names a thing: a UUID, a digest with its algorithm's name, a file's path (`namesThing`; the model takes all three for secrets). The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with it on a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name; 和 or 与 says so only after a person, 吴敏和郑浩 but not 判断和边界, 发 or 打 only as the verb, 发了个 but not 关键发现, and a sentence starts after punctuation, not after a space after code, `gh pr view` 易触发; xuanwo on X; 一下 only after a verb that asks a person, 问一下张伟 but not 改一下文案, a word any subject takes after it, 已经, 明天 or 他, only with a person's cue before it, and 说明 is no 说; zeewell on X), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left, and so is a common word that starts with one (`zhWords`: 文案, 文档, 方案), the model's own name too when it is that word whole. The model is the user's pick (`settings.RedactModelName`, `pii.Choose`; the app's menu beside the switch, `pmPick`): AstrLink Guard (QuantumNous/astrlink-guard at a pinned revision, Apache-2.0, a 39 MB int8 Multilingual-MiniLM token classifier for Chinese and English), run in magpie's own process on the CPU by [`internal/pii/guard`](../../internal/pii/guard) (an ONNX reader, the XLM-R tokenizer and int8 kernels in Go, NEON/SDOT on arm64 and SSE2 on amd64; its labels named as Privacy Filter's, `guardLabels`), and on a Mac through CoreML on the Neural Engine or the GPU ([`coreml.go`](../../internal/pii/guard/coreml.go), [`internal/pii/guard/coreml`](../../internal/pii/guard/coreml); yyhhyyyyyy on X): the same network from the same int8 weights in float16, written from the downloaded model into a program made by `coreml_gen.py` and compiled once into the model's folder, 512 tokens in about 12 ms where the CPU takes 400, its probabilities within .07 of the publisher's fp32 model where the CPU's are within .13 (the CPU quantizes activations as the int8 model does, CoreML doesn't), so a span near the threshold can fall the other side (a go.mod module path is a secret at .81 on CoreML, .79 on the CPU, .785 in fp32); a window under 16 tokens (`coremlMin`), a Mac before macOS 13 or any failure goes to the CPU, loaded on the first request and let go after 10 minutes with none; or OpenAI's Privacy Filter. With no pick it is AstrLink Guard, the default; a user who had Privacy Filter on before there was a choice keeps it only while AstrLink Guard downloads, which the first masked request starts and which then saves AstrLink Guard as the pick (`masking.picked`, `toGuard`; an hour between tries after a failure). Measured for that default (#3578): on 234 segments of real code and config it found 2 spans where Privacy Filter found 100 (38 of them go.sum hashes after redact's own checks), and with the rules it found 42 of 42 test values to Privacy Filter's 41. Each model has its own download, cache file and turn (`backend`). Both find at a floor of 0.5 (`pii.Floor`) and give each span its score (AstrLink Guard: the least of its tokens' probabilities, rounded to a float32 so a cached span reads back the same; Privacy Filter: pf-serve's own), and masking keeps those at the model's threshold or over (`pii.Threshold` in `masking.model`): the user's for that model (`settings.RedactModelScore`, this computer's own; Settings → Privacy's Confidence threshold, `pmScore`, 0.5 to 0.95), else the model's own (`MinScore`: Privacy Filter 0.5, AstrLink Guard 0.8, its publisher's). The defaults stay because on the eval a higher one mostly loses values the checks after the model would have kept (Privacy Filter at 0.85 lost a Chinese name; AstrLink Guard at 0.9 a phone in a tool result), while the false alarms it would drop are mostly dropped by those checks already (Yayoi_no_yume on X). Measured again on 2000 segments of real agent traffic (Codex and Claude Code sessions: prompts, tool arguments and results, a quarter with Chinese) and 30 more labelled cases: AstrLink Guard at 0.88 rather than 0.8 drops 3 false alarms (a model id, a sandbox path word, a token counter) and lets a colleague's name and a signed name through; Privacy Filter at 0.88 rather than 0.5 drops 11 (timestamps in paths, log lines, a plugin id) and lets 6 real names and a phone through (4 of the names in the real traffic). Lowering Guard to 0.5 finds 2 more names in the real traffic and 3 more labelled values, for 10 more false alarms (9 of them Claude Code's `<total_tokens>` counter). Those false alarms are shapes a check can turn away; a threshold can't without losing real values. Privacy Filter is a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags; on a GPU it holds the model once, in the device's memory, #3590; every pf-serve keeps the engine's own MoE chunk, `PF_MOE_CHUNK` at the window's 4096, which an RX 470 through Vulkan read 6-9% faster than 2048 in interleaved runs and Metal ~6% faster, and inherits a value the user set): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, through magpie's proxy (`netproxy.Func`, as its other requests; hh0592821 on Discord, whose download went direct before), from hf-mirror first in China (`source.chinaFirst`) and from hf-mirror after Hugging Face everywhere else, the switch off too (`source.modelMirror`), so a poisoned or blocked huggingface.co doesn't stop it; hf-mirror serves AstrLink Guard's files itself and sends Privacy Filter's back to Hugging Face) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU, doesn't load there in 40s (`gpuLoadTimeout`, well inside the gateway's 2 minutes), or doesn't answer in time there twice in a row, is started on the CPU, and Settings says why; a request it didn't answer in time once is tried on the GPU again, since a slow request on a weak GPU is no lost GPU (#3682). The GPU is tried again 10 minutes later, doubling to 2 hours while it keeps failing, with no restart (`gpuRetryAfter`), and one that answers is the model's again), joins short texts into 12 KB chunks and maps the spans back, cuts a text over 12 KB into pieces of 12 KB at most that overlap by 1 KB (`pieces`: after a line's end, else a space, else at a rune's start), each kept by its own hash, their spans put back at the text's offsets and merged where pieces overlap (`merge`: one found twice is one, one cut by a piece's end is whole), and asks pf-serve a chunk at a time (#3682: a 480 KB text whole took an RX 470/580 90s and kept nothing when cut off), keeps the last 20000 answers by hash, on disk too (`found-<model>.bin` beside the model: hashes, offsets, labels and each span's score, never text; a `pfound01` file from before scores is read with every span sure, so it masks as it did; written through temp-and-rename, a bad read taken as empty), so a restart doesn't ask the model about every history again (#3682), and ends it after 10 minutes with no request. Its requests take turns on its one pipe (`turn`), and a request reads the cache again when its turn comes; a request waits for its turn only while its context lives (`pii.FindContext`, through `gwmw.ContextTeller`), so one whose client went away leaves the queue (the gateway's 2 minutes leave masking's time out, #3682); the chunk pf-serve is on is finished and kept, so a request sent again asks only about what is left, and between chunks the turn is given up and taken again, so a short request goes before the rest of a long one. The status (`pii.Now`), `pii.Stop` and the idle timer never wait for a load or a request: `srv.mu` is held only to read or set the state, and `Stop` ends a load or a request under way at once. A request held before routing while it waits for the model is answered 504 saying so, and how to send without it (`unroutedMsg`); one of an older tag (`oldTags`, magpie updated since the download) runs until the pinned tag's is fetched (`upgrade`, on its start), which is then started on the next request and the old deleted ([`internal/pii`](../../internal/pii)). A failure names what ended `pf-serve`: magpie's own end of a request it didn't answer in time, else its exit status or signal and its last stderr lines past ggml's startup chatter (`died`, `lastSaid`). After one, requests are held back at once with that failure, not after a load each, until the model is tried again 30s later, doubling to 10 minutes while it keeps failing (`heldBack`, `retryAfter`; lazyeo on Discord, an M2 whose every request waited out the 504). A model that is on and can't answer (not on disk, didn't start, hung, held back) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).
+2. `serve` redacts secrets in the request (`redacted`; `redact.MaskJSON` leaves ids alone, and in any text a backend's id (a UUID, 32 hex digits or more, or a digest of 16 to 31 hex digits in one case with letters and digits both, a cache key or a short commit, which went as a key before: _kaichen on X, 10-28; `digestShaped`), or an element token cua-driver hands the model for computer use (`s0000002a:14`, #3782), goes as written: no rule, model label or value known from before masks a piece of one (a phone number at the end of `…-a13800138000`, a card number across a UUID's groups), and a whole one is masked only by the user's own words and rules or as a credential, by the password, API key and token rules or a secret rule whose field is named for one (`api_key=`, `refresh_token`), never one named for an id, a cursor, a request or a digest (`secret_id`, `next_page_token`, `Idempotency-Key`, `secret_hash`), nor cua-driver's own `element_token` (any other name with `element` in it, `element_api_key`, is a credential's as before), a bare `key` (not after api, access, secret, private, signing or license: `keyPhrase`), a field named for a cache, commit, object, blob, file, content, revision, partition, shard, bucket, dedup, lookup, row or sort key, a serial number, or what the model takes a UUID or an element token for, nor an element token known from before (one the model labelled in prose a turn earlier) (`noIDs`, [`internal/redact/ids.go`](../../internal/redact/ids.go)); and a tool's name and its MCP server's (`namespace`, `server_label`, `server_name`, #3143); the request's tool definitions and output schemas, found by their place in each protocol (top-level `tools`, `tool_choice`, `response_format`, `text`, `generationConfig`, `stream_options`, the client's ids in `client_metadata` and the like, and the tools of a Responses `additional_tools` or `tool_search_output` item), go as written save their descriptions, so a word the user masks can't break a grammar or an enum (#1911); and what the vendor sealed with a signature, a thinking block, Gemini's thoughts or Responses' encrypted reasoning, goes back as the vendor wrote it: only the values masked elsewhere in the request, and secrets masked before, are put back as their placeholders; a value masked once, at least 8 characters long, is masked again wherever a request has it, as an exact match whatever words are around it, so what the vendor wrote about a placeholder doesn't go back with the value in it, `redact/known.go`; a text masked for a vendor before, such as a system prompt or a turn already sent, is masked as it was then, so a value a rule first finds later in a conversation doesn't change the prompt the vendor has cached and rewrite all of it to the cache, `sentKnownSpans`), resolves the model to a provider or a group, and refuses what can't be served. The reply has the values put back on its way to the agent (`redact.Writer`, a stream event by event, a whole body at its end): a placeholder the model wrote as it was given, or as models rewrite one (spaces inside the braces, a brace or both dropped, another case), is the value again when what is left names one magpie made, and one split across events is held until it is whole (`redact.Restore`, `partialTail`, [`internal/redact/redact.go`](../../internal/redact/redact.go)). A request whose provider and fallbacks, or whose group's models, all skip redaction (`unredactedRoute`, `provider.SkipsRedaction`: set *Send requests unmasked*, `Unredacted`, with every address on this machine or the local network) goes to them as the agent wrote it, and is masked only for a body an observer wants or the request archive keeps, which may go elsewhere (`bodiesKeptAway`); without those it isn't masked at all, and the log and recent calls keep it as it went, since a request is masked only once its route is known (#3973: masking a body whose masked copy was thrown away took up to minutes of magpie's own time). A request turned away before it is routed is logged masked (`maskedCopy`). count_tokens and embeddings follow the same rule. What a routing group's Jev is told of the turn (the message, the user's message before it and the reply to it) is masked all the same unless Jev's own provider skips redaction too (`maskedState` in `askJev`, [`decide.go`](../../internal/gateway/decide.go)): a remote Jev is none of the group's members. The masking is the built-in plugin `redact`'s (`gwmw.Masker`, [Plugin runtime](plugin-runtime.md)); when the settings ask for masking and no masker serves, or it fails, the request is a 503 and nothing is sent (`redacted`, `gwmw.Masking`).
+ - With *Mask personal data* on, each kind in `redact.Categories` ([`internal/redact/more.go`](../../internal/redact/more.go)) is masked unless the user turned it off (`redactKinds` in settings, `Options.Kinds`). On by default: US SSNs, passport numbers, IBANs that pass mod 97, birthdays next to a word that says so (a date over a year ago: one in the last year or ahead is the day an agent runs on, `dateOK`); the word must be right before the date, so a log line's or a file name's date is none, and a time beside the date doesn't make it a log's: a date of birth stored as a time (`1990-05-11T16:00:00.000Z`) is one (`TestBirthdayLogDates`; #3878), MAC addresses, and serial numbers next to S/N or the like. Eleven digits from 13 to 19 are a mainland mobile's form and an id's too, so they are a phone only with +86 or 86 before them, as the value of a field whose name says phone (phone, mobile, tel, cell, contact, 手机, 电话, a field named `number`), or in prose with no id's word right before them (order, id, no., #, 订单号, 编号, 单号, 流水号); under order_id or uid, after 订单号, or with a letter or digit against them they go as written (`phoneAt` in [`internal/redact/phone.go`](../../internal/redact/phone.go); _kaichen on X, 10-28). Off by default: the user name in a home folder, user@host in a prompt or ssh command, public IPv4 and IPv6 addresses (never private, loopback, link-local, CGNAT or documentation ranges) and cloud bucket names, since agents send them in nearly every request and the model must copy the placeholder exactly inside a path or URL. When the user name is on, it goes as a made-up name of 8 lowercase letters, the same for that name every turn and put back in replies like any placeholder (`standIn` in [`shape.go`](../../internal/redact/shape.go)), not as `{{USERNAME_…}}`: with mimo-v2.6-flash a model read `C:\Users\{{…}}` as a temporary folder and set out to delete it (eerttreeeert on X, 10-28). Context rules run only near their words (`rule.matches`). What reads as an address but ends in a file's extension that no domain has (`…@2x.jpg`, `[email protected]`; `fileExt` in `realEmail`), or a scale before an extension that is a domain too (`[email protected]`, `[email protected]`; `scaleExt`), is a file's name, for the rule and the local model alike (#3029). `[email protected]` is still an address. The Settings rows come from `REDACT_KINDS` in app.js, which `TestRedactKindsAreRedacts` holds to the same ids and defaults.
+ - With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken, nor a snake_case word as a name (Codex's `sequential_cutoff`, #3144), nor as a secret what names a thing: a UUID, a digest with its algorithm's name, a file's path (`namesThing`; the model takes all three for secrets). The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with it on a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name; 和 or 与 says so only after a person, 吴敏和郑浩 but not 判断和边界, 发 or 打 only as the verb, 发了个 but not 关键发现, and a sentence starts after punctuation, not after a space after code, `gh pr view` 易触发; xuanwo on X; 一下 only after a verb that asks a person, 问一下张伟 but not 改一下文案, a word any subject takes after it, 已经, 明天 or 他, only with a person's cue before it, and 说明 is no 说; zeewell on X), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left, and so is a common word that starts with one (`zhWords`: 文案, 文档, 方案; 毛利, 毛重, 毛巾, since 毛利 is a margin far more often than the Japanese surname, so 毛利低 goes as written: _kaichen on X, 10-28; the model still finds 毛利兰, and neither model nor rule masks 毛利小五郎), the model's own name too when it is that word whole. The model is the user's pick (`settings.RedactModelName`, `pii.Choose`; the app's menu beside the switch, `pmPick`): AstrLink Guard (QuantumNous/astrlink-guard at a pinned revision, Apache-2.0, a 39 MB int8 Multilingual-MiniLM token classifier for Chinese and English), run in magpie's own process on the CPU by [`internal/pii/guard`](../../internal/pii/guard) (an ONNX reader, the XLM-R tokenizer and int8 kernels in Go, NEON/SDOT on arm64 and SSE2 on amd64; its labels named as Privacy Filter's, `guardLabels`), and on a Mac through CoreML on the Neural Engine or the GPU ([`coreml.go`](../../internal/pii/guard/coreml.go), [`internal/pii/guard/coreml`](../../internal/pii/guard/coreml); yyhhyyyyyy on X): the same network from the same int8 weights in float16, written from the downloaded model into a program made by `coreml_gen.py` and compiled once into the model's folder, 512 tokens in about 12 ms where the CPU takes 400, its probabilities within .07 of the publisher's fp32 model where the CPU's are within .13 (the CPU quantizes activations as the int8 model does, CoreML doesn't), so a span near the threshold can fall the other side (a go.mod module path is a secret at .81 on CoreML, .79 on the CPU, .785 in fp32); a window under 16 tokens (`coremlMin`), a Mac before macOS 13 or any failure goes to the CPU, loaded on the first request and let go after 10 minutes with none; or OpenAI's Privacy Filter. With no pick it is AstrLink Guard, the default; a user who had Privacy Filter on before there was a choice keeps it only while AstrLink Guard downloads, which the first masked request starts and which then saves AstrLink Guard as the pick (`masking.picked`, `toGuard`; an hour between tries after a failure). Measured for that default (#3578): on 234 segments of real code and config it found 2 spans where Privacy Filter found 100 (38 of them go.sum hashes after redact's own checks), and with the rules it found 42 of 42 test values to Privacy Filter's 41. Each model has its own download, cache file and turn (`backend`). Both find at a floor of 0.5 (`pii.Floor`) and give each span its score (AstrLink Guard: the least of its tokens' probabilities, rounded to a float32 so a cached span reads back the same; Privacy Filter: pf-serve's own), and masking keeps those at the model's threshold or over (`pii.Threshold` in `masking.model`): the user's for that model (`settings.RedactModelScore`, this computer's own; Settings → Privacy's Confidence threshold, `pmScore`, 0.5 to 0.95), else the model's own (`MinScore`: Privacy Filter 0.5, AstrLink Guard 0.8, its publisher's). A value known from before follows the same threshold: each value masked remembers the highest score anything found it with (a rule's counts as sure), and one only the model found, under the threshold now, is no longer masked again as known (`noteScores`, `knownSpans` in [`known.go`](../../internal/redact/known.go); before, a raised slider changed nothing for a value masked once: _kaichen on X, 10-28). The defaults stay because on the eval a higher one mostly loses values the checks after the model would have kept (Privacy Filter at 0.85 lost a Chinese name; AstrLink Guard at 0.9 a phone in a tool result), while the false alarms it would drop are mostly dropped by those checks already (Yayoi_no_yume on X). Measured again on 2000 segments of real agent traffic (Codex and Claude Code sessions: prompts, tool arguments and results, a quarter with Chinese) and 30 more labelled cases: AstrLink Guard at 0.88 rather than 0.8 drops 3 false alarms (a model id, a sandbox path word, a token counter) and lets a colleague's name and a signed name through; Privacy Filter at 0.88 rather than 0.5 drops 11 (timestamps in paths, log lines, a plugin id) and lets 6 real names and a phone through (4 of the names in the real traffic). Lowering Guard to 0.5 finds 2 more names in the real traffic and 3 more labelled values, for 10 more false alarms (9 of them Claude Code's `<total_tokens>` counter). Those false alarms are shapes a check can turn away; a threshold can't without losing real values. Privacy Filter is a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags; on a GPU it holds the model once, in the device's memory, #3590; every pf-serve keeps the engine's own MoE chunk, `PF_MOE_CHUNK` at the window's 4096, which an RX 470 through Vulkan read 6-9% faster than 2048 in interleaved runs and Metal ~6% faster, and inherits a value the user set): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, through magpie's proxy (`netproxy.Func`, as its other requests; hh0592821 on Discord, whose download went direct before), from hf-mirror first in China (`source.chinaFirst`) and from hf-mirror after Hugging Face everywhere else, the switch off too (`source.modelMirror`), so a poisoned or blocked huggingface.co doesn't stop it; hf-mirror serves AstrLink Guard's files itself and sends Privacy Filter's back to Hugging Face) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU, doesn't load there in 40s (`gpuLoadTimeout`, well inside the gateway's 2 minutes), or doesn't answer in time there twice in a row, is started on the CPU, and Settings says why; a request it didn't answer in time once is tried on the GPU again, since a slow request on a weak GPU is no lost GPU (#3682). The GPU is tried again 10 minutes later, doubling to 2 hours while it keeps failing, with no restart (`gpuRetryAfter`), and one that answers is the model's again), joins short texts into 12 KB chunks and maps the spans back, cuts a text over 12 KB into pieces of 12 KB at most that overlap by 1 KB (`pieces`: after a line's end, else a space, else at a rune's start), each kept by its own hash, their spans put back at the text's offsets and merged where pieces overlap (`merge`: one found twice is one, one cut by a piece's end is whole), and asks pf-serve a chunk at a time (#3682: a 480 KB text whole took an RX 470/580 90s and kept nothing when cut off), keeps the last 20000 answers by hash, on disk too (`found-<model>.bin` beside the model: hashes, offsets, labels and each span's score, never text; a `pfound01` file from before scores is read with every span sure, so it masks as it did; written through temp-and-rename, a bad read taken as empty), so a restart doesn't ask the model about every history again (#3682), and ends it after 10 minutes with no request. Its requests take turns on its one pipe (`turn`), and a request reads the cache again when its turn comes; a request waits for its turn only while its context lives (`pii.FindContext`, through `gwmw.ContextTeller`), so one whose client went away leaves the queue (the gateway's 2 minutes leave masking's time out, #3682); the chunk pf-serve is on is finished and kept, so a request sent again asks only about what is left, and between chunks the turn is given up and taken again, so a short request goes before the rest of a long one. The status (`pii.Now`), `pii.Stop` and the idle timer never wait for a load or a request: `srv.mu` is held only to read or set the state, and `Stop` ends a load or a request under way at once. A request held before routing while it waits for the model is answered 504 saying so, and how to send without it (`unroutedMsg`); one of an older tag (`oldTags`, magpie updated since the download) runs until the pinned tag's is fetched (`upgrade`, on its start), which is then started on the next request and the old deleted ([`internal/pii`](../../internal/pii)). A failure names what ended `pf-serve`: magpie's own end of a request it didn't answer in time, else its exit status or signal and its last stderr lines past ggml's startup chatter (`died`, `lastSaid`). After one, requests are held back at once with that failure, not after a load each, until the model is tried again 30s later, doubling to 10 minutes while it keeps failing (`heldBack`, `retryAfter`; lazyeo on Discord, an M2 whose every request waited out the 504). A model that is on and can't answer (not on disk, didn't start, hung, held back) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).
- With *Mask secrets* on, the user's own masking rules (`redactRules`, `redact.Rule`: a placeholder name and a prefix or an RE2 pattern, [`internal/redact/rules.go`](../../internal/redact/rules.go)) run before magpie's. Settings → Privacy adds them, removes them, and edits one in place (`POST /api/settings/redact-rule` with the rule's index and its current value, `editRedactRule` in [`internal/gui/api.go`](../../internal/gui/api.go)). Every rule is checked by `redact.CheckRules` when settings are saved; an edit that doesn't compile, matches the empty string, repeats another rule or names a rule no longer at that index is refused with the rules left as they were. The masker reads the rules from settings on each call (`options` in [`internal/masking`](../../internal/masking/masking.go)), so a change applies to the next request.
- The user's own masked words (`redactWords`, `Options.Words`; Settings → Privacy's *Masked words*) go as `{{TERM_…}}` whatever else is on, and come back as written. They match as written, inside longer words too, unless the user asks: *Masked words in any case* (`redactWordsAnyCase`, `WordsAnyCase`) matches in any capitals by Unicode simple folding, and *Whole words only* (`redactWordsWhole`, `WordsWhole`; acmilan_ on Discord) skips a match with a letter, digit or `_` beside it, and for a number a `.` and digit beside it, so `ABC` masks `name='ABC'` but not `XABCD`, and `123456` not `91234567`, `0.123456` or `123456.78` (`wholeWord` in [`redact.go`](../../internal/redact/redact.go)). Chinese and Japanese letters count as no letter there (`wordRune`): a word of theirs is found inside their text, and `ABC` in 用ABC做 is whole. Both are off by default; a page from before them that saves without the field keeps the user's choice (`api.go`). `TestWordsWhole`, `TestWordsWholeJSON`, `TestCountTokensRedacted`. A word that is a date (`1990-05-12`, `1990/5/12`, `1990年5月12日`, `May 12, 1990`; eight digits alone are not one) matches that calendar date in every common form instead (`dateSpans` in [`dateword.go`](../../internal/redact/dateword.go); JackyCcao on X, who masks their own birthday with birthdays off): `-` `/` `.` `、` or a space between, 年月日 or 년월일, with or without leading zeros, `19900512`, the month's English name, and day-first or month-first only where the day is over 12 or equals the month (`13/05/1990`, never `05/12/1990`, which is December 5th too; such a word matches only as written). A match never has a digit, or a dot and a digit, beside it, whatever *Whole words only* says, so `1990.5.12.1` and `11990-05-12` are not that date. `TestDateWordEveryForm`, `TestDateWordNotAnotherDate`, `TestDateWordDayMonthOrder`.
- With *Keep a key's form* on (`settings.RedactShape`, `Options.Shape`, under *Mask secrets*; Charles Cao on X), a secret of 16 characters or more, all letters, digits, `-`, `_` or `.`, goes as a made-up key of its form instead of a placeholder: its prefix kept (`sk-proj-`, `ghp_`, `AKIA`, up to four letters and their separator; `keyHead`), the same length and separators, and each other character a random one of its kind (a hex key's any hex digit of its case), drawn from an HMAC of the value under the placeholder key, so one key goes as the same made-up one every turn and a vendor's cache still matches (`shapeOf`, `shaped` in [`internal/redact/shape.go`](../../internal/redact/shape.go)). Personal data, the user's words and anything shorter or with other characters go as placeholders still. A made-up key is kept in the masker's memory with the placeholders, so it is recorded, re-masked in what a vendor sealed, and put back wherever the reply has it, before the placeholders (`restoreShapes`); a stream holds the end of an event that may be the start of one (`shapeTail`, after the last whole one), and a made-up key the vendor sends back unrestored isn't masked again (`outsideShapes`). `TestShapeKeepsForm`, `TestShapeJSON`, `TestShapeStream`. Insights › Masking and Settings' *Try it* mark a made-up key where they mark a placeholder, matching the text against the request's own values only, so a key of the same form the user wrote isn't marked (`maskMatches` in `plugins/redact/page.js`; #3599, masking-records and mask-actions GUI tests).
diff --git a/internal/masking/masking.go b/internal/masking/masking.go
index 26de84839..92ebc23b9 100644
--- a/internal/masking/masking.go
+++ b/internal/masking/masking.go
@@ -34,7 +34,8 @@ func optionsCtx(ctx context.Context) redact.Options {
Actions: st.RedactActions, Allow: st.RedactAllow, Shape: st.RedactShape}
if st.RedactModel {
m := picked(st)
- o.Model = model(ctx, m, pii.Threshold(st.RedactModelScore, m))
+ o.MinScore = pii.Threshold(st.RedactModelScore, m)
+ o.Model = model(ctx, m, o.MinScore)
}
return o
}
diff --git a/internal/redact/dateword.go b/internal/redact/dateword.go
index 8f4ffbdae..b8c013c4e 100644
--- a/internal/redact/dateword.go
+++ b/internal/redact/dateword.go
@@ -176,7 +176,7 @@ func dateSpans(s, w string) ([]span, bool) {
var out []span
for _, m := range re.FindAllStringIndex(s, -1) {
if dateWhole(s, m[0], m[1]) {
- out = append(out, span{m[0], m[1], "TERM", ByWord})
+ out = append(out, span{m[0], m[1], "TERM", ByWord, 0})
}
}
return out, true
diff --git a/internal/redact/digest_test.go b/internal/redact/digest_test.go
new file mode 100644
index 000000000..fdb8274b0
--- /dev/null
+++ b/internal/redact/digest_test.go
@@ -0,0 +1,71 @@
+package redact
+
+import (
+ "strings"
+ "testing"
+)
+
+// A hash's hex digits short of 32 are an id, as 32 or more are: under a
+// bare "key", a cache's key, a field named for a hash or as a JSON key of
+// data keyed by hashes they go as written, the rules' and the model's
+// both (_kaichen on X, 10-28: "hash类型也会被过滤,有些以hash为key的数据,
+// 被当作key被mask"). Under a credential's name, or after a key's name in
+// a sentence, the same digits are a key still.
+func TestShortDigestsGoAsWritten(t *testing.T) {
+ Forget()
+ defer Forget()
+ o := Options{Secrets: true, Personal: true}
+ kept := []string{
+ `{"key": "5d41402abc4b2a76"}`,
+ "key: 1a2b3c4d5e6f7a8b9c0d",
+ `{"key":"0123456789abcdef0123"}`,
+ `{"token_hash": "0123456789abcdef01234567"}`,
+ `{"cache_key": "a1b2c3d4e5f60718293a"}`,
+ `cacheKey = "a1b2c3d4e5f60718293b"`,
+ `{"object_key": "A1B2C3D4E5F60718293C"}`,
+ `{"hash": "9f13800138000abc"}`,
+ `{"keys": {"5d41402abc4b2a76": {"hits": 3}, "7c4a8d09ca3762af": {"hits": 1}}}`,
+ "git checkout 5d41402abc4b2a76e",
+ }
+ for _, s := range kept {
+ if out := Preview(s, o); out != s {
+ t.Errorf("%q masked: %q", s, out)
+ }
+ }
+ masked := []struct{ s, v string }{
+ {"password=deadbeef12345678", "deadbeef12345678"},
+ {"api_key: 9f2c4e6a8b0d1f3e5a7c", "9f2c4e6a8b0d1f3e5a7c"},
+ {`{"access_token": "0123456789abcdef0123"}`, "0123456789abcdef0123"},
+ {"DB_PASSWORD=0123456789ABCDEF01", "0123456789ABCDEF01"},
+ {"密钥为 5d41402abc4b2a76", "5d41402abc4b2a76"},
+ {"apikey 为 5d41402abc4b2a77", "5d41402abc4b2a77"},
+ // not a digest: mixed case, or letters past f
+ {`{"key": "AbC123dEf456GhI789xyz"}`, "AbC123dEf456GhI789xyz"},
+ }
+ for _, c := range masked {
+ if out := Preview(c.s, o); strings.Contains(out, c.v) {
+ t.Errorf("%q: key not masked: %q", c.s, out)
+ }
+ }
+
+ // the model's SECRET on the same digits: kept under "key", a cache's
+ // key and as a JSON key; masked in prose and under a credential's name
+ m := &fakeModel{values: map[string]string{}}
+ om := Options{Secrets: true, Personal: true, Model: m.find}
+ for _, c := range []struct {
+ s, v string
+ gone bool
+ }{
+ {`{"key": "5d41402abc4b2a78"}`, "5d41402abc4b2a78", false},
+ {`{"cacheKey": "5d41402abc4b2a79"}`, "5d41402abc4b2a79", false},
+ {`{"keys": {"5d41402abc4b2a7a": 1}}`, "5d41402abc4b2a7a", false},
+ {"the api key is 5d41402abc4b2a7b", "5d41402abc4b2a7b", true},
+ {`{"secret": "5d41402abc4b2a7c"}`, "5d41402abc4b2a7c", true},
+ } {
+ m.values = map[string]string{c.v: "secret"}
+ out := Preview(c.s, om)
+ if gone := !strings.Contains(out, c.v); gone != c.gone {
+ t.Errorf("%q, the model's secret: masked %v, want %v: %q", c.s, gone, c.gone, out)
+ }
+ }
+}
diff --git a/internal/redact/escape_test.go b/internal/redact/escape_test.go
index b1a17b875..7696cd50a 100644
--- a/internal/redact/escape_test.go
+++ b/internal/redact/escape_test.go
@@ -64,7 +64,7 @@ func TestMaskJSONKeepsArgumentsJSON(t *testing.T) {
// a Windows path in it goes as before
body2 := `{"cwd":"C:\\Users\\rkato\\proj"}`
out2, n2 := MaskJSON([]byte(body2), Options{Personal: true, Kinds: map[string]bool{"home": true}})
- if n2 != 1 || !strings.Contains(string(out2), `C:\\Users\\{{USERNAME_`) {
+ if n2 != 1 || !strings.Contains(string(out2), `C:\\Users\\`+standInOf("rkato")+`\\proj`) {
t.Fatalf("%d: %s", n2, out2)
}
}
@@ -233,12 +233,12 @@ func TestMaskJSONArgumentsCutShort(t *testing.T) {
// inside it too
s := `x\u0040y\nz`
found := outsideEscapes(s, []span{
- {3, 5, "A", ByRule}, // 00 of \u0040: nothing left
- {1, 7, "B", ByRule}, // the whole \u0040: as it is
- {0, 4, "C", ByRule}, // ends in it: x
- {7, 9, "D", ByRule}, // y and the backslash of \n: y
- {9, 11, "E", ByRule}, // the n of \n and z: z
- {8, 11, "F", ByRule}, // \nz: as it is
+ {3, 5, "A", ByRule, 0}, // 00 of \u0040: nothing left
+ {1, 7, "B", ByRule, 0}, // the whole \u0040: as it is
+ {0, 4, "C", ByRule, 0}, // ends in it: x
+ {7, 9, "D", ByRule, 0}, // y and the backslash of \n: y
+ {9, 11, "E", ByRule, 0}, // the n of \n and z: z
+ {8, 11, "F", ByRule, 0}, // \nz: as it is
})
var kept []string
for _, f := range found {
diff --git a/internal/redact/export_test.go b/internal/redact/export_test.go
index 7db3e8e57..24d768ad2 100644
--- a/internal/redact/export_test.go
+++ b/internal/redact/export_test.go
@@ -11,6 +11,7 @@ func Forget() {
mu.Lock()
values = map[string]string{}
shapes = map[string]string{}
+ scores = map[string]float64{}
reindexKnown()
mu.Unlock()
resetSent()
diff --git a/internal/redact/ids.go b/internal/redact/ids.go
index fec1e506d..7ff495b74 100644
--- a/internal/redact/ids.go
+++ b/internal/redact/ids.go
@@ -52,13 +52,20 @@ func noIDs(s string, found []span) []span {
case ByModel:
// a key the model found in prose: 32 hex digits it calls a
// secret ("the api key is 9f8e…"); never a UUID or an element
- // token
- return f.kind != "SECRET" || uuidRe.MatchString(s[l:r]) || elementToken(s, l, r)
+ // token, nor a field's value whose name is not a credential's
+ // ("key", cacheKey)
+ return f.kind != "SECRET" || uuidRe.MatchString(s[l:r]) || elementToken(s, l, r) ||
+ len(fieldWords(s, l)) > 0 && !credentialField(s, l) || nameOfField(s, l, r)
case ByRule:
switch f.kind {
case "PASSWORD", "API_KEY", "TOKEN", "PRIVATE_KEY":
return false
case "SECRET":
+ // a short digest after a key's name in a sentence (密钥为
+ // 5d41402abc4b2a76) is that key; in a field, as a long id
+ if digestShaped(s[l:r]) && len(fieldWords(s, l)) == 0 {
+ return false
+ }
return !credentialField(s, l)
}
}
@@ -91,9 +98,35 @@ func idAround(s string, a, b int) (l, r int, ok bool) {
!idShaped(s[l:r]) {
return 0, 0, false
}
+ // a short digest is a word of its own, not the hex end of a longer
+ // one (a key's sk_…deadbeef…)
+ if digestShaped(s[l:r]) && (l > 0 && wordByte(s[l-1]) || r < len(s) && wordByte(s[r])) {
+ return 0, 0, false
+ }
return l, r, true
}
+// nameOfField says s[l:r] is a field's name, not its value: a JSON
+// object's key ({"5d41402abc4b2a76": {…}}), as data keyed by a hash has
+// it. A name is never a secret.
+func nameOfField(s string, l, r int) bool {
+ if l == 0 || s[l-1] != '"' {
+ return false
+ }
+ i := r
+ if i < len(s) && s[i] == '\\' {
+ i++
+ }
+ if i >= len(s) || s[i] != '"' {
+ return false
+ }
+ i++
+ for i < len(s) && (s[i] == ' ' || s[i] == '\t') {
+ i++
+ }
+ return i < len(s) && s[i] == ':'
+}
+
// elementToken says s[a:b] is an element token, or its snapshot's part
// (s0000002a, where the secret rule's value stops, at the colon): s and 8
// lowercase hex digits, then a colon and the index, with no letter, digit
@@ -138,9 +171,40 @@ func hexOrHyphen(c byte) bool {
}
// idShaped says v, hex digits and hyphens, is a UUID or 32 hex digits or
-// more without a hyphen.
+// more without a hyphen, or a shorter digest (digestShaped).
func idShaped(v string) bool {
- return uuidRe.MatchString(v) || len(v) >= 32 && !strings.Contains(v, "-")
+ return uuidRe.MatchString(v) || len(v) >= 32 && !strings.Contains(v, "-") || digestShaped(v)
+}
+
+// digestShaped says v is a hash's hex digits short of 32: 16 to 31 of
+// them in one case, letters and digits both, as a short commit, a 64-bit
+// hash (xxhash, a cache's key) or a truncated sha256 is written
+// (5d41402abc4b2a76). Data keyed by one went to the vendor as
+// {{SECRET_…}} under "key", cacheKey or "token_hash", and a phone's form
+// in the middle of one (9f13800138000abc) as half a placeholder (_kaichen
+// on X, 10-28). Digits alone are a number's, a card's or a phone's.
+func digestShaped(v string) bool {
+ if len(v) < 16 || len(v) >= 32 {
+ return false
+ }
+ var lower, upper, digits int
+ for i := 0; i < len(v); i++ {
+ switch c := v[i]; {
+ case c >= '0' && c <= '9':
+ digits++
+ case c >= 'a' && c <= 'f':
+ lower++
+ case c >= 'A' && c <= 'F':
+ upper++
+ default:
+ return false
+ }
+ }
+ // a hash's digits are random: of 16, about 6 are letters and 10
+ // digits. Two letters among digits are an IBAN's (DE89370400…) or a
+ // code's, not a hash's.
+ letters := lower + upper
+ return (lower == 0 || upper == 0) && letters >= max(3, len(v)/8) && digits >= max(3, len(v)/8)
}
// credentialField says the field whose value starts at s[at] is named for
@@ -149,6 +213,9 @@ func idShaped(v string) bool {
// bare "key".
func credentialField(s string, at int) bool {
words := fieldWords(s, at)
+ if len(words) == 1 && words[0] == "key" && keyPhrase(s, at) {
+ return true // api key: …, Secret Key = …
+ }
if len(words) < 2 && (len(words) == 0 || words[0] == "key") || elementField(words) {
return false
}
@@ -159,13 +226,34 @@ func credentialField(s string, at int) bool {
for _, w := range words {
switch w {
case "page", "next", "cursor", "continuation", "csrf", "xsrf", "trace", "request", "idempotency", "sync", "nonce",
- "hash", "digest", "checksum", "sha", "fingerprint", "etag":
+ "hash", "digest", "checksum", "sha", "fingerprint", "etag",
+ // a key for finding data, not for getting in: cache_key,
+ // cacheKey, object_key, dedupe_key
+ "cache", "commit", "object", "blob", "file", "content", "revision", "rev", "partition", "shard", "bucket", "dedup", "dedupe", "lookup", "row", "sort":
return false
}
}
return true
}
+// keyPhrase says the bare key whose value starts at s[at] is the second
+// word of a credential's name in words: api key, access key, secret key,
+// private key.
+func keyPhrase(s string, at int) bool {
+ before := strings.ToLower(s[max(0, at-24):at])
+ i := strings.LastIndex(before, "key")
+ if i < 0 {
+ return false
+ }
+ before = strings.TrimRight(before[:i], " ")
+ for _, w := range []string{"api", "access", "secret", "private", "signing", "license", "licence"} {
+ if strings.HasSuffix(before, w) && (len(before) == len(w) || !wordByte(before[len(before)-len(w)-1])) {
+ return true
+ }
+ }
+ return false
+}
+
// fieldWords are the words of the name of the field whose value starts at
// s[at] (nameWords), none where no field's name is before it.
func fieldWords(s string, at int) []string {
diff --git a/internal/redact/ids_test.go b/internal/redact/ids_test.go
index 06be4880b..5837101b7 100644
--- a/internal/redact/ids_test.go
+++ b/internal/redact/ids_test.go
@@ -76,7 +76,7 @@ func TestIDsGoAsWritten(t *testing.T) {
}
// a phone number and a card number of their own are still masked
- for _, in := range []string{"call 13800138000", "card 4111-1111-1111-1111", "call a13800138000"} {
+ for _, in := range []string{"call 13800138000", "card 4111-1111-1111-1111", "call +8613800138000"} {
forgetIDs()
if out, n := Mask(in, o); n != 1 || out == in {
t.Errorf("%q went as %q", in, out)
diff --git a/internal/redact/known.go b/internal/redact/known.go
index 1e50af107..c5719f9e8 100644
--- a/internal/redact/known.go
+++ b/internal/redact/known.go
@@ -137,6 +137,47 @@ func (o Options) coversKind(kind string) bool {
return false
}
+// A value the model alone found is known at the score it had: it is
+// masked again as known only while the user's slider (Options.MinScore)
+// is at or below that. Moving the slider up did nothing to what the model
+// had found at a lower score, until magpie started again ("即使把匹配拉到
+// 0.95,也是一样行为", _kaichen on X, 10-28). One any other finder found
+// (a rule, the user's words or rules) is known whatever the slider says,
+// as it was.
+
+// ruleFound is a value's score where no model found it.
+const ruleFound = 2
+
+// scores are the values found, with the model's best score for each
+// (ruleFound where anything else found it); guarded by mu.
+var scores = map[string]float64{}
+
+// noteScores keeps what found's values in s were found at.
+func noteScores(s string, found []span) {
+ if len(found) == 0 {
+ return
+ }
+ mu.Lock()
+ defer mu.Unlock()
+ if len(scores) >= maxValues {
+ // forgotten, a value is known whatever the slider says, as all
+ // were before
+ scores = map[string]float64{}
+ }
+ for _, f := range found {
+ if f.by == ByKnown || f.end-f.start < minKnown {
+ continue
+ }
+ v, sc := s[f.start:f.end], float64(ruleFound)
+ if f.by == ByModel && f.score > 0 {
+ sc = f.score
+ }
+ if old, ok := scores[v]; !ok || sc > old {
+ scores[v] = sc
+ }
+ }
+}
+
// knownSpans are where s has a known value o masks (coversKind). Where two
// start at the same place, the longer.
func knownSpans(s string, o Options) []span {
@@ -158,13 +199,16 @@ func knownSpans(s string, o Options) []span {
if !o.coversKind(k.kind) {
continue
}
+ if sc, ok := scores[k.v]; ok && sc < o.MinScore {
+ continue // the model's alone, under the slider
+ }
if best != nil && len(k.v) <= len(best.v) || !strings.HasPrefix(s[i:], k.v) || !standsAlone(k.kind, s, i, i+len(k.v)) {
continue
}
best = &known[s[i:i+minKnown]][j]
}
if best != nil {
- out = append(out, span{i, i + len(best.v), best.kind, ByKnown})
+ out = append(out, span{i, i + len(best.v), best.kind, ByKnown, 0})
i += len(best.v) - 1
}
}
@@ -172,7 +216,8 @@ func knownSpans(s string, o Options) []span {
}
// standsAlone says s[a:b] is the known value itself, not a piece of a
-// longer one: a number with no digit beside it, an email that is not part
+// longer one: a number with no letter or digit beside it (as the rules
+// take one: 9f13800138000abc is a hash's piece), an email that is not part
// of a longer address (though a full stop may end the sentence after it),
// a secret with no letter or digit against it (twelve hex digits a rule of
// the user's took are also in the middle of a hash).
@@ -186,9 +231,7 @@ func standsAlone(kind, s string, a, b int) bool {
}
is := func(c byte, set string) bool { return c != 0 && strings.IndexByte(set, c) >= 0 }
switch kind {
- case "PHONE", "BANK_CARD":
- return !is(before, digits) && !is(after, digits)
- case "ID_CARD", "IBAN", "SSN", "PASSPORT", "SERIAL":
+ case "PHONE", "BANK_CARD", "ID_CARD", "IBAN", "SSN", "PASSPORT", "SERIAL":
return !is(before, alnum) && !is(after, alnum)
case "IP", "MAC":
// not a piece of a longer address or a version: 1.2.3.45, 1.2.3.4.5
diff --git a/internal/redact/model.go b/internal/redact/model.go
index 0bc9fe535..6cde647e7 100644
--- a/internal/redact/model.go
+++ b/internal/redact/model.go
@@ -83,7 +83,7 @@ func modelSpans(s string, o Options) []span {
a, b = zhTrim(s, a, b)
}
if a < b && utf8.ValidString(s[a:b]) && modelTakes(kind, s[a:b]) && modelFits(s, a, b, kind) {
- out = append(out, span{a, b, kind, ByModel})
+ out = append(out, span{a, b, kind, ByModel, f.Score})
}
}
if o.covers(catPersonal) {
diff --git a/internal/redact/more_test.go b/internal/redact/more_test.go
index bb6993a36..91e9bb4d5 100644
--- a/internal/redact/more_test.go
+++ b/internal/redact/more_test.go
@@ -59,7 +59,7 @@ func TestMoreKinds(t *testing.T) {
{in: `C:\Users\Rina Kato\Desktop\a.txt`, gone: []find{{"Rina Kato", "USERNAME"}}, kept: []string{`C:\Users\`, `\Desktop\a.txt`}},
{in: `"path":"C:\\Users\\rkato\\proj"`, gone: []find{{"rkato", "USERNAME"}}},
{in: "file:///Users/rinak/notes.md", gone: []find{{"rinak", "USERNAME"}}},
- {in: "see /Users/rinak.", gone: []find{{"rinak", "USERNAME"}}, kept: []string{"}}."}},
+ {in: "see /Users/rinak.", gone: []find{{"rinak", "USERNAME"}}, kept: []string{standInOf("rinak") + "."}},
// user@host
{in: "rinak@rina-mbp:~/src$ ls", gone: []find{{"rinak", "USERNAME"}, {"rina-mbp", "HOSTNAME"}}, kept: []string{":~/src$ ls"}},
{in: "[deploy@build-07 ~]$ make", gone: []find{{"deploy", "USERNAME"}, {"build-07", "HOSTNAME"}}},
@@ -95,7 +95,11 @@ func TestMoreKinds(t *testing.T) {
if strings.Contains(unmarked(out), g.value) {
t.Errorf("%q: %q still in %q", c.in, g.value, out)
}
- if p := placeholder(g.kind, g.value); !strings.Contains(out, p) {
+ p := placeholder(g.kind, g.value)
+ if g.kind == "USERNAME" {
+ p = standInOf(g.value) // a made-up name (shape.go)
+ }
+ if !strings.Contains(out, p) {
t.Errorf("%q: no %s placeholder for %q in %q", c.in, g.kind, g.value, out)
}
}
@@ -245,9 +249,9 @@ func TestMoreKindsKnown(t *testing.T) {
o := allKinds()
a, _ := Mask("cwd /Users/rkatodev/src", o)
b, _ := Mask("rkatodev@rina-mbp:~$ ls", o)
- p := placeholder("USERNAME", "rkatodev")
+ p := standInOf("rkatodev")
if !strings.Contains(a, p) || !strings.Contains(b, p) {
- t.Fatalf("not one placeholder: %q %q", a, b)
+ t.Fatalf("not one made-up name: %q %q", a, b)
}
reply := "Your account rkatodev has no access to 34.117.59.82."
Mask("connect 34.117.59.82", o)
diff --git a/internal/redact/phone.go b/internal/redact/phone.go
new file mode 100644
index 000000000..466e203a0
--- /dev/null
+++ b/internal/redact/phone.go
@@ -0,0 +1,169 @@
+package redact
+
+import (
+ "strings"
+ "unicode/utf8"
+)
+
+// Eleven digits that start 13 to 19 are a mainland mobile number's form,
+// and an id's: an order's, a user's, a row's (13800138000 as order_id,
+// WHERE id = 18612345678, 订单号 13912345678). Every one went to the
+// vendor as {{PHONE_…}} (_kaichen on X, 10-28: "长int,假如以1为开头的,
+// 刚好匹配到长度,会被当作电话被mask"), and an agent can't look up an
+// order it can't read. So the form alone is a phone only where nothing
+// says it is something else:
+//
+// - with a phone's own marks it is one wherever it is: +86 or 86 before
+// it (+86 138 0013 8000 reads as +86 and the rest);
+// - as a field's value (JSON, k=v, YAML, SQL's =, an XML element) it is
+// one only when the field's name has a phone's word in it: phone,
+// mobile, tel, cell, contact, 手机, 电话 (phone_number, mobileNo,
+// <tel>); under order_id, uid or id it is that id;
+// - in prose it is one, unless an id's word is right before it: order,
+// id, no., #, 订单号, 编号, 单号, 流水号, 工单 (我的手机号是 13800138000
+// still masks, 订单号 13912345678 已发货 doesn't).
+//
+// A letter or a digit against it on either side makes it a piece of a
+// longer word, never a phone (the rule's bound): 9f13800138000abc is a
+// hash.
+
+// phoneWords are the words of a field's name that make its value a phone.
+var phoneWords = []string{"phone", "phones", "mobile", "mobiles", "tel", "telephone", "cell", "cellphone", "contact", "contacts", "whatsapp", "sms", "msisdn", "fax"}
+
+// phoneMarks, in a field's name in Chinese, make its value a phone.
+var phoneMarks = []string{"手机", "电话", "電話", "联系", "聯繫", "联络"}
+
+// idWords, right before a number in prose, make it an id.
+var idWords = []string{"order", "id", "uid", "no", "no.", "#", "number", "ticket", "invoice", "订单", "订单号", "单号", "编号", "流水号", "工单", "工单号", "运单号", "快递单号", "订单编号", "交易号", "序号", "账单号", "id号"}
+
+// phoneAt says the PHONE rule's s[a:b] is a phone, not an id of the same
+// form.
+func phoneAt(s string, a, b int) bool {
+ if b-a > 11 {
+ return true // +86, 86
+ }
+ if name, prev, ok := fieldName(s, a); ok {
+ return phoneName(name) || prev != "" && phoneName(prev) || strings.EqualFold(name, "number")
+ }
+ if a > 0 && s[a-1] == '#' {
+ return false // #13912345678
+ }
+ before := strings.ToLower(strings.TrimRight(s[max(0, a-32):a], " \t(\"'“「"))
+ for _, c := range []string{"是", "为", "為"} {
+ before = strings.TrimRight(strings.TrimSuffix(before, c), " \t")
+ }
+ for _, w := range idWords {
+ if !strings.HasSuffix(before, w) {
+ continue
+ }
+ // a whole word: "order", not the end of "recorder"
+ rest := before[:len(before)-len(w)]
+ r, _ := utf8.DecodeLastRuneInString(rest)
+ if len(rest) > 0 && w[0] < 0x80 && (r >= 'a' && r <= 'z' || r >= '0' && r <= '9' || r == '_') {
+ continue
+ }
+ // but a phone's number: phone no. 13800138000
+ if w == "no" || w == "no." || w == "number" {
+ rest = strings.TrimRight(rest, " ")
+ if phoneName(rest[nameStart(rest, len(rest)):]) {
+ return true
+ }
+ }
+ return false
+ }
+ return true
+}
+
+// phoneName says a field's name is a phone's: one of its words is one of
+// phoneWords (phone_number, mobileNo), or it has one of phoneMarks (手机号).
+func phoneName(name string) bool {
+ for _, w := range nameWords(name) {
+ for _, p := range phoneWords {
+ if w == p {
+ return true
+ }
+ }
+ }
+ for _, m := range phoneMarks {
+ if strings.Contains(name, m) {
+ return true
+ }
+ }
+ return false
+}
+
+// fieldName is the name of the field whose value starts at s[a]: the
+// word before :, =, :or an XML element's >, with quotes and spaces
+// between, as JSON, k=v, YAML, SQL and XML have one, and prev the word
+// before it with one space between (the phone of "phone no: …"). ok is
+// false in prose, where no such separator is right before the value.
+func fieldName(s string, a int) (name, prev string, ok bool) {
+ i := a
+ for i > 0 && strings.IndexByte(" \t\"'", s[i-1]) >= 0 {
+ i--
+ }
+ switch {
+ case i > 0 && s[i-1] == '>':
+ // <tel>13800138000</tel>: the element's name
+ k := strings.LastIndexByte(s[max(0, i-65):i-1], '<')
+ if k < 0 {
+ return "", "", false
+ }
+ tag := s[max(0, i-65)+k+1 : i-1]
+ if sp := strings.IndexAny(tag, " \t"); sp >= 0 {
+ tag = tag[:sp]
+ }
+ if tag == "" || strings.IndexFunc(tag, func(r rune) bool {
+ return !(r < 0x80 && strings.IndexByte(alnum+"_.:-", byte(r)) >= 0)
+ }) >= 0 {
+ return "", "", false
+ }
+ return tag, "", true
+ case i > 0 && (s[i-1] == ':' || s[i-1] == '='):
+ i--
+ case i >= len(":") && s[i-len(":"):i] == ":":
+ i -= len(":")
+ default:
+ return "", "", false
+ }
+ for i > 0 && strings.IndexByte(" \t\"'", s[i-1]) >= 0 {
+ i--
+ }
+ j := nameStart(s, i)
+ if j == i {
+ return "", "", false
+ }
+ name = s[j:i]
+ if n := utf8.RuneCountInString(name); n > 6 && hasCJK(name) {
+ // a long run of CJK before a colon is a sentence's: its last few
+ // characters name the value (我的手机号是:…)
+ name = string([]rune(name)[n-6:])
+ }
+ if j >= 2 && s[j-1] == ' ' && s[j-2] != ' ' {
+ prev = s[nameStart(s, j-1) : j-1]
+ }
+ return name, prev, true
+}
+
+// nameStart is where the name that ends at s[i] starts: letters, digits,
+// _, . and - back from i, or Chinese characters.
+func nameStart(s string, i int) int {
+ j := i
+ for j > 0 {
+ r, n := utf8.DecodeLastRuneInString(s[:j])
+ if !(r < 0x80 && strings.IndexByte(alnum+"_.-", byte(r)) >= 0 || r >= 0x4e00 && r <= 0x9fff) {
+ break
+ }
+ j -= n
+ }
+ return j
+}
+
+func hasCJK(s string) bool {
+ for _, r := range s {
+ if r >= 0x4e00 && r <= 0x9fff {
+ return true
+ }
+ }
+ return false
+}
diff --git a/internal/redact/phone_test.go b/internal/redact/phone_test.go
new file mode 100644
index 000000000..8888550fb
--- /dev/null
+++ b/internal/redact/phone_test.go
@@ -0,0 +1,78 @@
+package redact
+
+import (
+ "strings"
+ "testing"
+)
+
+// Eleven digits that start 13 to 19 are a phone only where something says
+// so: a phone's marks (+86), a field named for one, or prose with no id's
+// word before them. Ids of the same form go as written (_kaichen on X,
+// 10-28: "长int,假如以1为开头的,刚好匹配到长度,会被当作电话被mask").
+func TestPhoneNeedsAPhonesContext(t *testing.T) {
+ Forget()
+ defer Forget()
+ o := Options{Secrets: true, Personal: true}
+ masked := []struct{ s, v string }{
+ {"我的手机号是 13800138000", "13800138000"},
+ {"电话:13800138001", "13800138001"},
+ {"联系电话:13800138012,找张经理", "13800138012"},
+ {"联系 13800138008 张经理", "13800138008"},
+ {"请打 13800138009", "13800138009"},
+ {"+86 13800138002", "13800138002"},
+ {"+8613800138003", "13800138003"},
+ {"8613800138014", "13800138014"},
+ {`{"phone": "13800138004"}`, "13800138004"},
+ {`{"mobile":"13800138005"}`, "13800138005"},
+ {`{"phoneNumber": 13800138015}`, ""}, // a bare JSON number is no phone (noNumbers), as before
+ {`{"contact_phone": "13800138016"}`, "13800138016"},
+ {"phone_number=13800138006", "13800138006"},
+ {"tel: 13800138007", "13800138007"},
+ {"<tel>13800138010</tel>", "13800138010"},
+ {"<mobile type=\"cn\">13800138017</mobile>", "13800138017"},
+ {"phone no. 13800138011", "13800138011"},
+ {"Phone No: 13800138018", "13800138018"},
+ {"mobile number: 13800138019", "13800138019"},
+ {"call 13800138013 now", "13800138013"},
+ {"mail me: [email protected] or 13812345678", "13812345678"},
+ {"13800138020", "13800138020"},
+ {"张三 13800138021 北京", "13800138021"},
+ }
+ for _, c := range masked {
+ out := Preview(c.s, o)
+ if c.v == "" {
+ continue
+ }
+ if strings.Contains(out, c.v) || !strings.Contains(out, "{{PHONE_") {
+ t.Errorf("%q: phone not masked: %q", c.s, out)
+ }
+ }
+ kept := []string{
+ // _kaichen's: an id of a phone's form
+ `{"order_id": "13800138100"}`,
+ `{"id": "13912345678"}`,
+ `{"userId":"15012345679"}`,
+ "ts=2026-10-28T10:00:00Z level=info id=13912345679 msg=done",
+ "订单号 13912345670 已发货",
+ "订单号:13912345671",
+ "订单号是 13912345673",
+ "编号 13912345674",
+ "user_id=15012345678&page=2",
+ "SELECT * FROM orders WHERE id = 18612345678;",
+ "id: 17712345678",
+ "<id>17712345670</id>",
+ "order 13800138101 failed",
+ "Order #13800138103 shipped",
+ "ticket 13800138104",
+ "#13912345672",
+ // a piece of a hash or of a longer word
+ `{"hash": "9f13800138102abc"}`,
+ "build-13800138105x",
+ "v13800138106",
+ }
+ for _, s := range kept {
+ if out := Preview(s, o); out != s {
+ t.Errorf("%q masked: %q", s, out)
+ }
+ }
+}
diff --git a/internal/redact/redact.go b/internal/redact/redact.go
index 647f8d036..710befcaa 100644
--- a/internal/redact/redact.go
+++ b/internal/redact/redact.go
@@ -55,6 +55,11 @@ type Options struct {
Kinds map[string]bool
// Model, where set, finds more in the texts to mask (model.go)
Model Model
+ // MinScore is the score the user has Model's finds need (Settings →
+ // Privacy's slider), which Model's own finds are held to already. A
+ // value only the model found, masked before at a lower score, isn't
+ // masked again as known under it (knownSpans).
+ MinScore float64
// Shape sends a key as a made-up key of its form, not a placeholder
// (shape.go)
Shape bool
@@ -183,8 +188,8 @@ var rules = append([]rule{
{kind: "EMAIL", re: regexp.MustCompile(`[A-Za-z0-9._%+-]+@[A-Za-z0-9-]+(?:\.[A-Za-z0-9-]+)*\.[A-Za-z]{2,}`), markers: []string{"@"}, bound: alnum + "_%+-", ok: realEmail, at: emailAt, cat: catPersonal},
{kind: "ID_CARD", re: regexp.MustCompile(`[1-9]\d{5}(?:18|19|20)\d{2}(?:0[1-9]|1[0-2])(?:0[1-9]|[12]\d|3[01])\d{3}[\dXx]`), bound: alnum, ok: chineseID, cat: catPersonal},
- {kind: "PHONE", re: regexp.MustCompile(`(?:\+?86[- ]?)?1[3-9]\d{9}`), bound: digits, cat: catPersonal},
- {kind: "BANK_CARD", re: regexp.MustCompile(`[3-6]\d{3}(?:[ -]?\d{4}){2,3}(?:[ -]?\d{1,3})?`), bound: digits, ok: luhn, cat: catPersonal},
+ {kind: "PHONE", re: regexp.MustCompile(`(?:\+?86[- ]?)?1[3-9]\d{9}`), bound: alnum, at: phoneAt, cat: catPersonal},
+ {kind: "BANK_CARD", re: regexp.MustCompile(`[3-6]\d{3}(?:[ -]?\d{4}){2,3}(?:[ -]?\d{1,3})?`), bound: alnum, ok: luhn, cat: catPersonal},
}, moreRules...)
// cjkPasswordNames are what a password is called in Chinese, Japanese and
@@ -637,9 +642,9 @@ func makeRoom() {
// its form (Shape).
func (o Options) put() func(kind, v string) string {
if o.Shape {
- return shaped
+ return standIn(shaped)
}
- return placeholder
+ return standIn(placeholder)
}
// ---- masking --------------------------------------------------------------------
@@ -647,7 +652,8 @@ func (o Options) put() func(kind, v string) string {
type span struct {
start, end int
kind string
- by string // which finder found it: one of the By constants
+ by string // which finder found it: one of the By constants
+ score float64 // the model's score for one it found, else 0
}
// Which finder found a value (Hit.By): magpie's rules, the user's own
@@ -782,7 +788,8 @@ func Mask(s string, o Options) (string, int) {
}
// Preview is Mask with nothing remembered: each value goes as the
-// placeholder, or with Options.Shape the made-up key, it would have, which
+// placeholder, or with Options.Shape the made-up key, it would have (a
+// user name its made-up name), which
// isn't kept for Restore or as a value seen before (known.go).
func Preview(s string, o Options) string {
put := tag
@@ -795,6 +802,13 @@ func Preview(s string, o Options) string {
return tag(kind, v)
}
}
+ tagOrShape := put
+ put = func(kind, v string) string {
+ if kind == "USERNAME" {
+ return standInOf(v) // a user name's made-up name (shape.go)
+ }
+ return tagOrShape(kind, v)
+ }
out, _ := mask(s, withModel(o, []string{s}), put, false)
return out
}
@@ -857,7 +871,7 @@ func mask(raw string, o Options, put func(kind, v string) string, escaped bool)
i = a + max(n, 1)
continue
}
- found = append(found, span{a, b, "TERM", ByWord})
+ found = append(found, span{a, b, "TERM", ByWord, 0})
i = max(b, a+1)
}
}
@@ -892,12 +906,12 @@ func mask(raw string, o Options, put func(kind, v string) string, escaped bool)
took := false
for g, kind := range r.groups {
if a, b := m[2*g+2], m[2*g+3]; a >= 0 && r.takes(s, a, b, r.groupOK[g]) {
- found = append(found, span{a, b, kind, r.by()})
+ found = append(found, span{a, b, kind, r.by(), 0})
took = true
}
}
if took {
- split = append(split, span{m[2], m[1], "", ""})
+ split = append(split, span{m[2], m[1], "", "", 0})
}
continue
}
@@ -920,7 +934,7 @@ func mask(raw string, o Options, put func(kind, v string) string, escaped bool)
ok = nil
}
if r.takes(s, a, b, ok) {
- found = append(found, span{a, b, r.kind, r.by()})
+ found = append(found, span{a, b, r.kind, r.by(), 0})
}
}
}
@@ -940,7 +954,11 @@ func mask(raw string, o Options, put func(kind, v string) string, escaped bool)
if escaped {
found = outsideEscapes(raw, found)
}
- return apply(raw, noNumbers(s, noIDs(s, found)), put, o)
+ found = noNumbers(s, noIDs(s, found))
+ if o.sent {
+ noteScores(raw, found) // what is masked for a vendor, not a preview
+ }
+ return apply(raw, found, put, o)
}
// unescapedView is s, JSON text, as long as s and read as the vendor reads
diff --git a/internal/redact/redact_test.go b/internal/redact/redact_test.go
index ca5220061..fb0fb90ec 100644
--- a/internal/redact/redact_test.go
+++ b/internal/redact/redact_test.go
@@ -247,7 +247,7 @@ func TestMaskJSONToolArgsMasked(t *testing.T) {
for _, c := range []struct{ name, body string }{
{"signature key", `{"messages":[{"role":"assistant","content":[{"type":"tool_use","id":"toolu_1","name":"send_email","input":{"signature":"Best, Bob","body":"DB_PASSWORD=hunter2abc1 call 13800138000"}}]}]}`},
{"id keys", `{"messages":[{"role":"assistant","content":[{"type":"tool_use","id":"toolu_1","name":"lookup","input":{"user_id":"DB_PASSWORD=hunter2abc1","phone_id":"13800138000"}}]}]}`},
- {"gemini args", `{"contents":[{"role":"model","parts":[{"functionCall":{"name":"lookup","args":{"thoughtSignature":"x","customer_id":"13800138000","note":"DB_PASSWORD=hunter2abc1"}}}]}]}`},
+ {"gemini args", `{"contents":[{"role":"model","parts":[{"functionCall":{"name":"lookup","args":{"thoughtSignature":"x","customer_phone":"13800138000","note":"DB_PASSWORD=hunter2abc1"}}}]}]}`},
{"text block", `{"messages":[{"role":"user","content":[{"type":"text","text":"DB_PASSWORD=hunter2abc1 call 13800138000","signature":"x"}]}]}`},
} {
out, _ := MaskJSON([]byte(c.body), o)
diff --git a/internal/redact/score_test.go b/internal/redact/score_test.go
new file mode 100644
index 000000000..179e2b434
--- /dev/null
+++ b/internal/redact/score_test.go
@@ -0,0 +1,49 @@
+package redact
+
+import (
+ "strings"
+ "testing"
+)
+
+// A value only the model found is masked again as known only while the
+// user's slider is at or below the score it was found at (_kaichen on X,
+// 10-28: "即使把匹配拉到 0.95,也是一样行为"). One a rule found is known
+// whatever the slider says.
+func TestKnownModelValuesFollowTheSlider(t *testing.T) {
+ Forget()
+ defer Forget()
+ const name = "Quentin Marlowe"
+ scored := func(min float64) Model {
+ return func(texts []string) [][]Found {
+ out := make([][]Found, len(texts))
+ for i, s := range texts {
+ if j := strings.Index(s, "met "+name); j >= 0 && 0.6 >= min {
+ out[i] = []Found{{Start: j + 4, End: j + 4 + len(name), Label: "private_person", Score: 0.6}}
+ }
+ }
+ return out
+ }
+ }
+ o := func(min float64) Options {
+ return Options{Secrets: true, Personal: true, Model: scored(min), MinScore: min}
+ }
+ if out, _ := Mask("I met "+name+" today", o(0.5)); strings.Contains(out, name) {
+ t.Fatalf("not found at 0.5: %q", out)
+ }
+ later := "a note from " + name + " came"
+ if out, _ := Mask(later, o(0.5)); strings.Contains(out, name) {
+ t.Errorf("known at 0.5, under the slider's 0.5: %q", out)
+ }
+ if out, _ := Mask(later+" again", o(0.95)); !strings.Contains(out, name) {
+ t.Errorf("the model's 0.6 masked as known at 0.95: %q", out)
+ }
+ // a rule's value: known at any slider
+ const mail = "[email protected]"
+ Mask("write to "+mail, o(0.95))
+ mu.RLock()
+ sc := scores[mail]
+ mu.RUnlock()
+ if sc != ruleFound {
+ t.Errorf("a rule's value at %v, not known at any slider", sc)
+ }
+}
diff --git a/internal/redact/shape.go b/internal/redact/shape.go
index a8a24eb6d..d255e093d 100644
--- a/internal/redact/shape.go
+++ b/internal/redact/shape.go
@@ -263,3 +263,60 @@ func outsideShapes(s string, found []span) []span {
}
return found
}
+
+// A user name goes to the vendor as a made-up name, not a placeholder:
+// C:\Users\eerttreeeert as C:\Users\qmzvkrta, never C:\Users\{{USERNAME_…}}.
+// A path with braces in it reads as a template's or a temporary folder's,
+// and an agent on mimo-v2.6-flash took the user's home for a temporary
+// folder and set out to delete it (eerttreeeert on X, 10-28). A made-up
+// name keeps the path a home folder's, which is what it is, and still
+// tells the vendor nothing of the user. It is made from the name as a
+// made-up key is (shapeOf), so it is the same every turn and a vendor's
+// cache still matches, and it is put back wherever the vendor writes it
+// (restoreShapes), in a tool call's path too. Only USERNAME's: what the
+// user chose to mask (their words) goes as they chose.
+
+// standInLen is a made-up name's length: eight lowercase letters, which
+// read as a login name and are in no other text by chance.
+const standInLen = 8
+
+// standInOf is the made-up name the user name v goes as.
+func standInOf(v string) string {
+ h := hmac.New(sha256.New, key())
+ h.Write([]byte("user\x00" + v))
+ sum := h.Sum(nil)
+ out := make([]byte, standInLen)
+ for i := range out {
+ out[i] = 'a' + sum[i]%26
+ }
+ return string(out)
+}
+
+// standIn is put with a user name sent as its made-up name, which is
+// kept to be put back as a made-up key is; put's own where that name is
+// another value's already.
+func standIn(put func(kind, v string) string) func(kind, v string) string {
+ return func(kind, v string) string {
+ if kind != "USERNAME" {
+ return put(kind, v)
+ }
+ f := standInOf(v)
+ if f == v {
+ return put(kind, v)
+ }
+ mu.Lock()
+ have, taken := values[f]
+ if !taken {
+ makeRoom()
+ values[f] = v
+ kinds[kind] = true
+ shapes[f] = kind
+ addKnown(f, v)
+ }
+ mu.Unlock()
+ if taken && have != v {
+ return put(kind, v)
+ }
+ return f
+ }
+}
diff --git a/internal/redact/standin_test.go b/internal/redact/standin_test.go
new file mode 100644
index 000000000..938773339
--- /dev/null
+++ b/internal/redact/standin_test.go
@@ -0,0 +1,44 @@
+package redact
+
+import (
+ "strings"
+ "testing"
+)
+
+// A user name in a home folder goes as a made-up name, so the path still
+// reads as a home folder's (eerttreeeert on X, 10-28: C:\Users\{{…}} was
+// taken for a temporary folder, and the agent set out to delete it). It
+// is the same every turn, and comes back as the user's in a reply and in
+// a tool call's path.
+func TestUserNameGoesAsAMadeUpName(t *testing.T) {
+ o := Options{Personal: true, Kinds: map[string]bool{"home": true}}
+ for _, c := range []struct{ s, name, before, after string }{
+ {`C:\Users\eerttreeeert\AppData\Local\Temp\x.txt`, "eerttreeeert", `C:\Users\`, `\AppData\Local\Temp\x.txt`},
+ {`{"cwd":"C:\\Users\\eerttreeeert\\proj"}`, "eerttreeeert", `C:\\Users\\`, `\\proj`},
+ {"cd /Users/rinak/workspace/app", "rinak", "/Users/", "/workspace/app"},
+ {"/home/ops/.bashrc", "ops", "/home/", "/.bashrc"},
+ } {
+ out, n := Mask(c.s, o)
+ f := standInOf(c.name)
+ if n != 1 || strings.Contains(out, "{{") || out != strings.Replace(c.s, c.before+c.name+c.after, c.before+f+c.after, 1) {
+ t.Errorf("%q: sent %q, want the name as %q", c.s, out, f)
+ }
+ if len(f) != standInLen || strings.Trim(f, "abcdefghijklmnopqrstuvwxyz") != "" {
+ t.Errorf("%q: made-up name %q", c.name, f)
+ }
+ if again, _ := Mask(c.s, o); again != out {
+ t.Errorf("%q: not the same name twice: %q, %q", c.s, out, again)
+ }
+ if back := Restore(out, false); back != c.s {
+ t.Errorf("%q: put back as %q", c.s, back)
+ }
+ // the vendor's tool call with the path in it
+ call := `{"command":"Remove-Item C:\\Users\\` + f + `\\AppData\\Local\\Temp\\old -Recurse"}`
+ if back := Restore(call, true); !strings.Contains(back, c.name) || strings.Contains(back, f) {
+ t.Errorf("%q: tool call put back as %q", c.s, back)
+ }
+ if p := Preview(c.s, o); p != out {
+ t.Errorf("%q: preview %q, sent %q", c.s, p, out)
+ }
+ }
+}
diff --git a/internal/redact/zhname.go b/internal/redact/zhname.go
index 0fc63743f..c691d93a8 100644
--- a/internal/redact/zhname.go
+++ b/internal/redact/zhname.go
@@ -30,12 +30,15 @@ const zhNotGiven = "的了在是和与跟同说他她它我你您们这那个不
// zhWords begin with a surname but are words, not names: 文案 in
// "改一下文案" is no one (zeewell on X). Each is two characters, so a
-// candidate's first two are looked up (zhWord).
+// candidate's first two are looked up (zhWord). 毛利 is a gross margin
+// (毛利低 went as a name, _kaichen on X, 10-28) and a Japanese surname
+// too (毛利兰): the rules leave both, and the model, which reads the
+// sentence, finds the name.
var zhWords = runes2(
"张量", "张开", "张贴", "高度", "高效", "高级", "高速", "高亮", "高手", "高兴", "王国", "王者",
"黄河", "黄色", "方法", "方向", "方式", "方案", "方便", "方面", "程序", "程度", "温度", "白色",
"金额", "金融", "江湖", "石头", "马上", "周期", "周末", "周围", "周边", "文件", "文本", "文档",
- "文字", "文章", "余额", "安全", "安装", "安排", "钱包", "叶子", "毛病", "万一", "雷达", "宋体",
+ "文字", "文章", "余额", "安全", "安装", "安排", "钱包", "叶子", "毛病", "毛利", "毛重", "毛坯", "毛巾", "毛衣", "万一", "雷达", "宋体",
"常量", "常用", "常见", "向量", "向上", "全部", "全局", "全面", "史上", "易用", "武器", "时间",
"时候", "明天", "明确", "明白", "任何", "任务", "任意", "华为", "管理", "简单", "简介", "范围",
"许多", "许可", "包括", "包含", "关于", "关系", "关闭", "连接", "成功", "成为", "应用", "应该",
@@ -139,7 +142,7 @@ func zhNameSpans(s string, person []span) []span {
if ok {
kind = "PERSON"
}
- cs = append(cs, span{a, b, kind, ByModel})
+ cs = append(cs, span{a, b, kind, ByModel, 0})
i = b
}
// a name joined by 和, 与, 跟, 及 or 、 to one that is a name is one
diff --git a/internal/redact/zhname_test.go b/internal/redact/zhname_test.go
index 1200e05cd..c59528cbb 100644
--- a/internal/redact/zhname_test.go
+++ b/internal/redact/zhname_test.go
@@ -154,3 +154,20 @@ func TestChineseNamesNeedModel(t *testing.T) {
t.Fatalf("personal data off: %q", out)
}
}
+
+// 毛利 is a gross margin: 毛利低 is no one (_kaichen on X, 10-28), in the
+// sentences that made it a name. The rules leave 毛利兰 and 毛利小五郎 to
+// the model too, which reads the sentence; what it finds is masked.
+func TestMaoliIsAMargin(t *testing.T) {
+ f := &fakeModel{values: map[string]string{}}
+ o := Options{Personal: true, Model: f.find}
+ for _, s := range []string{"毛利低", "毛利低说", "我是毛利低", "这个产品毛利低说明定价有问题", "我是毛利低的那个品类负责人", "毛利低说明成本高", "告诉毛利低的门店先别补货", "毛利率下降了", "毛利额是多少", "看一下毛利润"} {
+ if out, _ := Mask(s, o); out != s {
+ t.Errorf("%q masked: %q", s, out)
+ }
+ }
+ f.values = map[string]string{"毛利小五郎": "private_person"}
+ if out, _ := Mask("毛利小五郎姓什么", o); strings.Contains(out, "毛利小五郎") {
+ t.Errorf("the model's name not masked: %q", out)
+ }
+}
diff --git a/site/public/docs/ja/masking.html b/site/public/docs/ja/masking.html
index 43bcbd3a1..816fe6e81 100644
--- a/site/public/docs/ja/masking.html
+++ b/site/public/docs/ja/masking.html
@@ -111,7 +111,7 @@
<div class="table-wrap"><table>
<tr><th>スイッチ</th><th>見つけるもの</th></tr>
<tr><td><span class="ui">シークレットをマスク</span></td><td>秘密鍵のブロック(<code>-----BEGIN … PRIVATE KEY-----</code>)。形で見分ける API キー:<code>sk-…</code>(OpenAI、Anthropic、OpenRouter)、<code>ghp_</code>/<code>github_pat_</code>、<code>glpat-</code>、<code>AIza…</code>、<code>xox…-</code>、<code>sk_live_</code>、<code>AKIA</code>/<code>ASIA</code>、<code>hf_</code>、<code>gsk_</code>、<code>xai-</code>、<code>npm_</code>、<code>pypi-</code>、<code>dop_v1_</code>、<code>SG.</code>、それに <code>oc_sk_…</code> のような中継サービスのキー。JWT。<code>user:password@host</code> のパスワード。<code>.env</code> や設定ファイルでシークレットの名前が付いたフィールドの値:<code>DB_PASSWORD=…</code>、<code>api_key: …</code>、<code>client_secret</code>、<code>token</code>、<code>credential</code>。それと自分の <span class="ui">マスクルール</span>。</td></tr>
- <tr><td><span class="ui">個人情報をマスク</span></td><td>メールアドレス、中国本土の携帯番号、チェックサムの合う中国の身分証番号と銀行カード番号。既定でオンのもの:米国の社会保障番号、パスポート番号(<em>passport</em>、护照、パスポートの後)、IBAN、誕生日(<em>birthday</em>、<em>DOB</em>、生日の近くにある、一年以上前の日付。今日の日付はそのまま)、MAC アドレス、シリアル番号(<em>S/N</em> の後)。エージェントがほぼ毎回送るので、自分でオンにするまでオフのもの:ホームフォルダのユーザー名、ユーザー名とコンピュータ名(<code>user@host</code>)、パブリック IP アドレス、クラウドストレージのバケット。</td></tr>
+ <tr><td><span class="ui">個人情報をマスク</span></td><td>メールアドレス、中国本土の携帯番号(11 桁の ID は別:<code>order_id</code> の値や 订单号 の後ろのものはそのまま)、チェックサムの合う中国の身分証番号と銀行カード番号。既定でオンのもの:米国の社会保障番号、パスポート番号(<em>passport</em>、护照、パスポートの後)、IBAN、誕生日(<em>birthday</em>、<em>DOB</em>、生日の近くにある、一年以上前の日付。今日の日付はそのまま)、MAC アドレス、シリアル番号(<em>S/N</em> の後)。エージェントがほぼ毎回送るので、自分でオンにするまでオフのもの:ホームフォルダのユーザー名(毎回同じ仮の名前で送るので、モデルにはプレースホルダーではなく普通のフォルダに見えます)、ユーザー名とコンピュータ名(<code>user@host</code>)、パブリック IP アドレス、クラウドストレージのバケット。</td></tr>
<tr><td><span class="ui">マスクする語</span></td><td>入れた語そのままを <code>{{TERM_…}}</code> に。日付(<em>1990-05-12</em> など)を入れると、同じ日のどの書き方もマスクされます:<em>1990/5/12</em>、<em>1990.05.12</em>、<em>1990年5月12日</em>、<em>19900512</em>、<em>May 12, 1990</em>。<em>13/05/1990</em> のような日が先の書き方は、日と月を取り違えようがないときだけです。別の日付や <em>1990.5.12.1</em> のようなバージョン番号はマスクされません。</td></tr>
<tr><td><span class="ui">ローカルモデルで検出</span></td><td><span class="ui">検出方法</span> の中にあります。このコンピュータで動くモデルが、ルールでは見つからない自由文中の名前や住所を見つけます。<span class="ui">シークレットをマスク</span> か <span class="ui">個人情報をマスク</span> がオンのときに使え、見つけたものは同じスイッチに従います。横のメニューでモデルを選べます:AstrLink Guard(中国語と英語、39 MB のダウンロード、コードでの誤検出が少ない、Mac では Neural Engine で動く。既定)か OpenAI の Privacy Filter(1.6 GB、GPU を使える、コード中の普通の値も多く隠す)。その下の <span class="ui">信頼度のしきい値</span> は、値をマスクするのにモデルがどれだけ確かである必要があるかです。既定は Privacy Filter が 0.5、AstrLink Guard が公開元の 0.8。上げると普通の値のマスクは減りますが、本物の名前や住所もそのまま送られやすくなります。誤検出の多くはモデルのあとのルールが既に外すので、既定値は測った中で本物の値を最も逃さない値です。</td></tr>
</table></div>
diff --git a/site/public/docs/masking.html b/site/public/docs/masking.html
index 7e81d038f..200693c2f 100644
--- a/site/public/docs/masking.html
+++ b/site/public/docs/masking.html
@@ -111,7 +111,7 @@
<div class="table-wrap"><table>
<tr><th>Switch</th><th>What it finds</th></tr>
<tr><td><span class="ui">Mask secrets</span></td><td>Private key blocks (<code>-----BEGIN … PRIVATE KEY-----</code>). API keys by their shape: <code>sk-…</code> (OpenAI, Anthropic, OpenRouter), <code>ghp_</code>/<code>github_pat_</code>, <code>glpat-</code>, <code>AIza…</code>, <code>xox…-</code>, <code>sk_live_</code>, <code>AKIA</code>/<code>ASIA</code>, <code>hf_</code>, <code>gsk_</code>, <code>xai-</code>, <code>npm_</code>, <code>pypi-</code>, <code>dop_v1_</code>, <code>SG.</code>, and relay keys like <code>oc_sk_…</code>. JWTs. The password in <code>user:password@host</code>. The value of a field named for a secret, as <code>.env</code> files and configs have them: <code>DB_PASSWORD=…</code>, <code>api_key: …</code>, <code>client_secret</code>, <code>token</code>, <code>credential</code>. Your own <span class="ui">Masking rules</span>.</td></tr>
- <tr><td><span class="ui">Mask personal data</span></td><td>Emails, mainland China mobile numbers, Chinese ID numbers and bank card numbers whose checksum adds up. Also on by default: US Social Security numbers, passport numbers (after <em>passport</em>, 护照 or パスポート), IBANs, birthdays (next to <em>birthday</em>, <em>DOB</em> or 生日, and over a year ago, so today&rsquo;s date stays), MAC addresses and serial numbers (after <em>S/N</em>). Off until you turn them on, since agents send them in nearly every request: the user name in home folders, user and computer names (<code>user@host</code>), public IP addresses and cloud storage buckets.</td></tr>
+ <tr><td><span class="ui">Mask personal data</span></td><td>Emails, mainland China mobile numbers (not an 11-digit id: one under <code>order_id</code> or after 订单号 goes as written), Chinese ID numbers and bank card numbers whose checksum adds up. Also on by default: US Social Security numbers, passport numbers (after <em>passport</em>, 护照 or パスポート), IBANs, birthdays (next to <em>birthday</em>, <em>DOB</em> or 生日, and over a year ago, so today&rsquo;s date stays), MAC addresses and serial numbers (after <em>S/N</em>). Off until you turn them on, since agents send them in nearly every request: the user name in home folders (sent as a made-up name, the same every turn, so the model reads a real folder rather than a placeholder), user and computer names (<code>user@host</code>), public IP addresses and cloud storage buckets.</td></tr>
<tr><td><span class="ui">Masked words</span></td><td>Each word you listed, exactly, as <code>{{TERM_…}}</code>. A date, such as <em>1990-05-12</em>, matches that day however it is written: <em>1990/5/12</em>, <em>1990.05.12</em>, <em>1990年5月12日</em>, <em>19900512</em>, <em>May 12, 1990</em>, and <em>13/05/1990</em> style only where the day can’t be read as a month. Never another date or a version like <em>1990.5.12.1</em>.</td></tr>
<tr><td><span class="ui">Find with a local model</span></td><td>Under <span class="ui">How it’s found</span>. A model on this computer finds what no rule can: a name or an address in free text. It needs <span class="ui">Mask secrets</span> or <span class="ui">Mask personal data</span> on, and finds under the same switches. The menu beside it picks the model: AstrLink Guard (Chinese and English, a 39 MB download, few false alarms in code, on a Mac it runs on the Neural Engine; the default) or OpenAI’s Privacy Filter (1.6 GB, uses the GPU, masks more ordinary values in code). <span class="ui">Confidence threshold</span> under it is how sure the model must be before a value is masked: Privacy Filter’s default is 0.5, AstrLink Guard’s 0.8, its publisher’s. Higher masks fewer ordinary values, and lets more real names and addresses through: the rules after the model drop most false alarms already, so the default is where we measured the fewest real values lost.</td></tr>
</table></div>
diff --git a/site/public/docs/zh/masking.html b/site/public/docs/zh/masking.html
index 68ce0bbf5..5bd522387 100644
--- a/site/public/docs/zh/masking.html
+++ b/site/public/docs/zh/masking.html
@@ -111,7 +111,7 @@
<div class="table-wrap"><table>
<tr><th>开关</th><th>能找到什么</th></tr>
<tr><td><span class="ui">脱敏密钥</span></td><td>私钥块(<code>-----BEGIN … PRIVATE KEY-----</code>)。按格式识别的 API Key:<code>sk-…</code>(OpenAI、Anthropic、OpenRouter)、<code>ghp_</code>/<code>github_pat_</code>、<code>glpat-</code>、<code>AIza…</code>、<code>xox…-</code>、<code>sk_live_</code>、<code>AKIA</code>/<code>ASIA</code>、<code>hf_</code>、<code>gsk_</code>、<code>xai-</code>、<code>npm_</code>、<code>pypi-</code>、<code>dop_v1_</code>、<code>SG.</code>,以及 <code>oc_sk_…</code> 这类中转站的 key。JWT。<code>user:password@host</code> 里的密码。<code>.env</code> 和配置文件里以密钥命名的字段的值:<code>DB_PASSWORD=…</code>、<code>api_key: …</code>、<code>client_secret</code>、<code>token</code>、<code>credential</code>。以及你的 <span class="ui">自定义脱敏规则</span>。</td></tr>
- <tr><td><span class="ui">脱敏个人信息</span></td><td>邮箱、中国大陆手机号、校验位对得上的身份证号和银行卡号。默认也开着的:美国社会安全号、护照号(跟在 <em>passport</em>、护照或パスポート后面)、IBAN、生日(旁边有 <em>birthday</em>、<em>DOB</em> 或「生日」,且是一年以前的日期,所以今天的日期不动)、MAC 地址、序列号(跟在 <em>S/N</em> 后面)。默认关闭、需要自己打开的(Agent 几乎每个请求都会带):主目录里的用户名、用户名和主机名(<code>user@host</code>)、公网 IP 地址、云存储桶。</td></tr>
+ <tr><td><span class="ui">脱敏个人信息</span></td><td>邮箱、中国大陆手机号(11 位的编号不算:在 <code>order_id</code> 下或跟在「订单号」后面的照原样发出)、校验位对得上的身份证号和银行卡号。默认也开着的:美国社会安全号、护照号(跟在 <em>passport</em>、护照或パスポート后面)、IBAN、生日(旁边有 <em>birthday</em>、<em>DOB</em> 或「生日」,且是一年以前的日期,所以今天的日期不动)、MAC 地址、序列号(跟在 <em>S/N</em> 后面)。默认关闭、需要自己打开的(Agent 几乎每个请求都会带):主目录里的用户名(以一个每轮都相同的假名发出,模型看到的是正常的文件夹,而不是占位符)、用户名和主机名(<code>user@host</code>)、公网 IP 地址、云存储桶。</td></tr>
<tr><td><span class="ui">脱敏词</span></td><td>你填的每个词,精确匹配,换成 <code>{{TERM_…}}</code>。填的是日期(比如 <em>1990-05-12</em>)时,同一天的各种写法都会遮掉:<em>1990/5/12</em>、<em>1990.05.12</em>、<em>1990年5月12日</em>、<em>19900512</em>、<em>May 12, 1990</em>;<em>13/05/1990</em> 这种日在前的写法只在日和月不会混淆时才算。别的日期和 <em>1990.5.12.1</em> 这样的版本号不会被遮。</td></tr>
<tr><td><span class="ui">用本地模型查找</span></td><td>在 <span class="ui">怎么找</span> 下面。一个在这台电脑上运行的模型能找出规则找不到的东西:自由文本里的人名、地址。需要先打开 <span class="ui">脱敏密钥</span> 或 <span class="ui">脱敏个人信息</span>,找到的内容按同样的开关处理。旁边的菜单选择模型:AstrLink Guard(中英文,下载 39 MB,代码里很少误判,在 Mac 上用神经网络引擎运行;默认)或 OpenAI 的 Privacy Filter(1.6 GB,可用 GPU,代码里会遮盖更多普通值)。下面的 <span class="ui">置信度阈值</span> 是模型要多确定才遮盖一个值:Privacy Filter 默认 0.5,AstrLink Guard 默认 0.8(发布者的值)。调高会少遮一些普通值,也会让更多真实的人名、地址原样发出去;模型之后的规则已经去掉了大部分误判,所以默认值是我们测下来漏掉真实信息最少的那一档。</td></tr>
</table></div>