Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

redact: Chinese personal names are masked, where the model misses or overreaches them (#1977)

yetone3806014ac859
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index b010b2f99..7b069dbb6 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -77,7 +77,7 @@ recorded as a prediction failure (`noPrediction`).
1. `handle(proto)` reads the body and the model (`requestBody`, `requestModel`), then calls `serveAgent`. Every route that reads an agent's body (each API, count_tokens, Gemini's, embeddings, images, videos, System One, the Codex backend path) reads it through `readBoundedRequestBody` in [`request_bounds.go`](../../internal/gateway/request_bounds.go). It decodes a `Content-Encoding` of `gzip` or `zstd` (`decodedBody`; Codex sends zstd to a custom provider's `/v1/responses` too, #1223) and drops the header, so the provider is sent plain JSON, with one exception: a ChatGPT account's turn (POST `CodexBase`/responses, from the accounts loop in `forwardOnce` or relayed by `codexUpstreamOn`) is compressed again with zstd and a window of at most 2 MiB, as Codex's own client sends it (`provider.CodexCompressed`; openai/codex `responses_request_compression` and `http-client/src/request.rs`). It is compressed once the body is final, and again on each try. `/responses/compact`, an API key's requests and every other provider stay plain. Sent plain, a 1.1 MB text-only turn was answered 503 (ylorn on X). The plain body is what magpie keeps and shows, and `codexDropped` judges a reset by the bytes that went on the wire. `codexDropped` also reads Envoy's 507 "exceeded request buffer limit while retrying upstream" (the backend failed and the front couldn't retry a body that large) as that reset, at any size, told as a 503 so magpie, Codex and pi ask again and nobody rests (#2034). The size limit (`defaultBodyLimit`, 512 MB) holds for the decoded bytes (413), and any other encoding is refused 415. It was 128 MB until a long Codex session that had read many images went past it decoded, its inline screenshots resent each turn, and got the 413 on all five of Codex's retries (#1422, `TestImageHeavyCodexTurnIsRead`). magpie's own 413 says it is magpie's gateway limit, with the size, and that the provider was not asked (`overLimit`); `writeError` adds to any 413 it isn't holding that this is a byte limit, not the model's context, and to `/compact` or start a new conversation (`TestOverLimitSaysWhoseAndWhatToDo`). Every shape but Gemini's types a 413 `request_too_large`, Anthropic's own type, which pi's overflow table knows: pi compacts and tries once more rather than stopping on an OpenAI-shaped `api_error` (@wilsonxuuuu on X, `TestPayloadTooLargeSaysRequestTooLargeToPi`). `keyLimited` reads a limited key's compressed body the same way, so its reservation is the decoded size. `serveAgent` runs the [middleware](gateway-middleware.md) chain, then `serve`.
2. `serve` redacts secrets in the request (`redacted`; `redact.MaskJSON` leaves ids alone; the request's tool definitions and output schemas, found by their place in each protocol (top-level `tools`, `tool_choice`, `response_format`, `text`, `generationConfig` and the like, and the tools of a Responses `additional_tools` or `tool_search_output` item), go as written save their descriptions, so a word the user masks can't break a grammar or an enum (#1911); and what the vendor sealed with a signature, a thinking block, Gemini's thoughts or Responses' encrypted reasoning, goes back as the vendor wrote it: only the values masked elsewhere in the request, and secrets masked before, are put back as their placeholders; a value masked once, at least 8 characters long, is masked again wherever a request has it, as an exact match whatever words are around it, so what the vendor wrote about a placeholder doesn't go back with the value in it, `redact/known.go`), resolves the model to a provider or a group, and refuses what can't be served. The reply has the values put back on its way to the agent (`redact.Writer`, a stream event by event, a whole body at its end): a placeholder the model wrote as it was given, or as models rewrite one (spaces inside the braces, a brace or both dropped, another case), is the value again when what is left names one magpie made, and one split across events is held until it is whole (`redact.Restore`, `partialTail`, [`internal/redact/redact.go`](../../internal/redact/redact.go)). A request whose provider and fallbacks, or whose group's models, all skip redaction (`unredactedRoute`, `provider.SkipsRedaction`: set *Send requests unmasked*, `Unredacted`, with every address on this machine or the local network) goes to them as the agent wrote it; the log and recent calls keep the masked body. count_tokens and embeddings follow the same rule. What a routing group's Jev is told of the turn (the message, the user's message before it and the reply to it) is masked all the same unless Jev's own provider skips redaction too (`maskedState` in `askJev`, [`decide.go`](../../internal/gateway/decide.go)): a remote Jev is none of the group's members. The masking is the built-in plugin `redact`'s (`gwmw.Masker`, [Plugin runtime](plugin-runtime.md)); when the settings ask for masking and no masker serves, or it fails, the request is a 503 and nothing is sent (`redacted`, `gwmw.Masking`).
- With *Mask personal data* on, each kind in `redact.Categories` ([`internal/redact/more.go`](../../internal/redact/more.go)) is masked unless the user turned it off (`redactKinds` in settings, `Options.Kinds`). On by default: US SSNs, passport numbers, IBANs that pass mod 97, birthdays next to a word that says so, MAC addresses, and serial numbers next to S/N or the like. Off by default: the user name in a home folder, user@host in a prompt or ssh command, public IPv4 and IPv6 addresses (never private, loopback, link-local, CGNAT or documentation ranges) and cloud bucket names, since agents send them in nearly every request and the model must copy the placeholder exactly inside a path or URL. Context rules run only near their words (`rule.matches`). The Settings rows come from `REDACT_KINDS` in app.js, which `TestRedactKindsAreRedacts` holds to the same ids and defaults.
- - With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken. The model is OpenAI's Privacy Filter as a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, from hf-mirror first in China, `source.chinaFirst`) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU is started on the CPU for the rest of the run), joins short texts into 12 KB chunks and maps the spans back, keeps the last 20000 answers by hash, and ends it after 10 minutes with no request ([`internal/pii`](../../internal/pii)). A model that is on and can't answer (not on disk, didn't start, hung) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).
+ - With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken. The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with it on a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left. The model is OpenAI's Privacy Filter as a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, from hf-mirror first in China, `source.chinaFirst`) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU is started on the CPU for the rest of the run), joins short texts into 12 KB chunks and maps the spans back, keeps the last 20000 answers by hash, and ends it after 10 minutes with no request ([`internal/pii`](../../internal/pii)). A model that is on and can't answer (not on disk, didn't start, hung) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).
- With *Mask secrets* on, the user's own masking rules (`redactRules`, `redact.Rule`: a placeholder name and a prefix or an RE2 pattern, [`internal/redact/rules.go`](../../internal/redact/rules.go)) run before magpie's. Settings → Privacy adds them, removes them, and edits one in place (`POST /api/settings/redact-rule` with the rule's index and its current value, `editRedactRule` in [`internal/gui/api.go`](../../internal/gui/api.go)). Every rule is checked by `redact.CheckRules` when settings are saved; an edit that doesn't compile, matches the empty string, repeats another rule or names a rule no longer at that index is refused with the rules left as they were. The gateway reads the rules from settings on each request (`redactionOptions`), so a change applies to the next request.
- `DecidesModel` gives a 400.
- A gateway key held to some models (`keyHolds`, #882) gets a 403 for any other model.
diff --git a/internal/redact/model.go b/internal/redact/model.go
index 38d6efd0f..81f5b8396 100644
--- a/internal/redact/model.go
+++ b/internal/redact/model.go
@@ -44,15 +44,15 @@ func modelCat(kind string) string {
}
// modelSpans are o.found's spans for s that o covers, trimmed of the
-// spaces the model takes in with a word, and with what it gets wrong
-// often left out (modelTakes).
+// spaces the model takes in with a word (and a Chinese name of what it
+// ran on into, zhTrim), and with what it gets wrong often left out
+// (modelTakes); and the Chinese names it missed (zhNameSpans).
func modelSpans(s string, o Options) []span {
- fs := o.found[s]
- if len(fs) == 0 {
+ if o.Model == nil || !o.covers(catPersonal) && len(o.found[s]) == 0 {
return nil
}
var out []span
- for _, f := range fs {
+ for _, f := range o.found[s] {
kind := modelKinds[f.Label]
if kind == "" || !o.covers(modelCat(kind)) || f.Start < 0 || f.End > len(s) || f.Start >= f.End {
continue
@@ -64,10 +64,22 @@ func modelSpans(s string, o Options) []span {
for b > a && isSpace(s[b-1]) {
b--
}
+ if kind == "PERSON" {
+ a, b = zhTrim(s, a, b)
+ }
if a < b && utf8.ValidString(s[a:b]) && modelTakes(kind, s[a:b]) {
out = append(out, span{a, b, kind})
}
}
+ if o.covers(catPersonal) {
+ var person []span
+ for _, f := range out {
+ if f.kind == "PERSON" {
+ person = append(person, f)
+ }
+ }
+ out = append(out, zhNameSpans(s, person)...)
+ }
return out
}
diff --git a/internal/redact/zhname.go b/internal/redact/zhname.go
new file mode 100644
index 000000000..b3c7ce0bc
--- /dev/null
+++ b/internal/redact/zhname.go
@@ -0,0 +1,302 @@
+package redact
+
+import (
+ "strings"
+ "unicode"
+ "unicode/utf8"
+)
+
+// The model misses many Chinese names: 张伟 in "写封信给张伟,他住在…",
+// 李娜, 陈建国 after 客户 all come back unlabelled, however low its
+// threshold, and the names it finds often run on (「王芳,她下周一」,
+// 「张伟先生」, 「林黛玉进贾府」). So, with the model on, a name is also
+// found the way a reader finds one: a surname, one or two characters
+// after it, and something around it that says a person is meant (客户,
+// 我叫, 给…发, …先生, …说). A surname with nothing around it is left as
+// it is: 王国的军队 and 张开的括号 are no one's.
+
+// zhSurnames are the common single-character surnames.
+const zhSurnames = "王李张刘陈杨黄赵吴周徐孙马朱胡郭何高林罗郑梁谢宋唐许韩冯邓曹彭曾肖萧田董袁潘于蒋蔡余杜叶程苏魏吕丁任沈姚卢姜崔钟谭陆汪范金石廖贾夏韦付傅方白邹孟熊秦邱江尹薛闫段雷侯龙史陶黎贺顾毛郝龚邵万钱严覃武戴莫孔向汤常温康施文牛樊葛邢安齐易乔伍庞颜倪庄聂章鲁岳翟殷詹申欧耿关兰焦俞左柳甘祝包宁尚符舒阮柯纪梅童凌毕单季裴霍涂成苗谷盛曲翁冉骆蓝路游辛靳管柴蒙鲍华喻祁蒲房滕屈饶解牟艾尤阳时穆农司卓古吉缪简车项连芦麦褚娄窦戚岑景党宫费卜冷晏席卫米柏宗瞿桂全佟应臧闵苟邬边卞姬师仇栾隋商刁沙荣巫寇桑郎甄丛仲虞敖巩佘池查麻苑迟邝"
+
+// zhCompound are the two-character surnames.
+var zhCompound = []string{"欧阳", "司马", "上官", "诸葛", "东方", "皇甫", "尉迟", "公孙", "慕容", "令狐", "司徒", "夏侯", "长孙", "宇文", "轩辕", "端木", "独孤", "南宫", "西门", "钟离", "呼延", "申屠", "闻人", "澹台", "拓跋", "万俟"}
+
+// zhNickSurnames are those 小 and 老 go before for a name (小王, 老李).
+const zhNickSurnames = "王李张刘陈杨黄赵吴周徐孙马朱胡郭何高林罗郑梁谢宋唐许韩冯邓曹彭曾肖田董袁潘蒋蔡杜叶程苏魏吕丁沈姚卢姜崔钟谭陆汪范金廖贾夏方邹孟熊秦江薛段雷侯史陶贺顾毛郝龚邵钱严武戴孔汤"
+
+// zhNotGiven can't be a given name's character: words a sentence goes on
+// with after a name.
+const zhNotGiven = "的了在是和与跟同说他她它我你您们这那个不也都就还把被给到对从向为以等及或并而吗呢吧啊去来有没要会能让请已将再又很更最先后上下里外时年日号发打回写问讲叫找用做看想知道得地着过之其此每各某该所反表认提负"
+
+// zhWords begin with a surname but are words, not names.
+var zhWords = []string{"张量", "张开", "张贴", "高度", "高效", "高级", "高速", "高亮", "高手", "高兴", "王国", "王者", "黄河", "黄色", "方法", "方向", "方式", "方案", "方便", "方面", "程序", "程度", "温度", "白色", "金额", "金融", "江湖", "石头", "马上", "周期", "周末", "周围", "周边", "文件", "文本", "文档", "文字", "文章", "余额", "安全", "安装", "安排", "钱包", "叶子", "毛病", "万一", "雷达", "宋体", "常量", "常用", "常见", "向量", "向上", "全部", "全局", "全面", "史上", "易用", "武器", "时间", "时候", "明天", "明确", "明白", "任何", "任务", "任意", "华为", "管理", "简单", "简介", "范围", "许多", "许可", "包括", "包含", "关于", "关系", "关闭", "连接", "成功", "成为", "应用", "应该", "单位", "单元", "单独", "司机", "车辆", "车子", "路径", "路由", "林业", "龙头", "柏林", "米饭", "宁可", "师傅", "老师", "老板", "老婆", "老公", "老家", "小时", "小心", "小说", "小姐", "小孩", "小组", "周一", "周二", "周三", "周四", "周五", "周六", "周日", "周天"}
+
+// zhRoles before a name say it is one: 客户陈建国, 联系人:赵丽颖,
+// 我叫黄晓明, 提交者是欧阳娜娜.
+var zhRoles = []string{"客户", "联系人", "作者", "提交者", "审核人", "审批人", "负责人", "经办人", "收件人", "发件人", "收货人", "申请人", "报告人", "同事", "姓名", "名字", "署名", "签名", "我叫", "他叫", "她叫", "名叫", "叫做", "尊敬的", "亲爱的", "致", "@"}
+
+// zhBefore are words that take a person after them (给张伟, 问李娜), so
+// a name after them is one where something after it says so too.
+var zhBefore = []string{"给", "找", "问", "让", "告诉", "通知", "联系", "抄送", "转给", "交给", "一下", "叫", "和", "与", "跟", "说", "请", "我是", "他是", "她是"}
+
+// zhAfter are words that follow a person (张伟先生, 马云说, 张三反馈).
+var zhAfter = []string{"先生", "女士", "小姐", "老师", "同学", "同志", "经理", "总监", "主任", "师傅", "医生", "律师", "教授", "博士", "说", "讲", "表示", "认为", "反馈", "提到", "提交", "负责", "告诉", "回复", "发", "打", "明天", "今天", "昨天", "已经", "他", "她"}
+
+// zhTitles are what the model takes in after a name and isn't it.
+var zhTitles = []string{"先生", "女士", "小姐", "老师", "同学", "同志", "经理", "总监", "主任", "师傅", "医生", "律师", "教授", "博士"}
+
+// the lists above as sets, and mayStart the characters a name can start
+// with
+var (
+ isSurname, isNick, notGiven = runes(zhSurnames), runes(zhNickSurnames), runes(zhNotGiven)
+ mayStart = runes(zhSurnames + zhFirsts() + "小老")
+)
+
+func runes(s string) map[rune]bool {
+ m := map[rune]bool{}
+ for _, r := range s {
+ m[r] = true
+ }
+ return m
+}
+
+func zhFirsts() string {
+ var b strings.Builder
+ for _, c := range zhCompound {
+ r, _ := utf8.DecodeRuneInString(c)
+ b.WriteRune(r)
+ }
+ return b.String()
+}
+
+// zhNameSpans are the Chinese names in s the model didn't find, as
+// PERSON spans; person are those it did.
+func zhNameSpans(s string, person []span) []span {
+ if !hasHan(s) {
+ return nil
+ }
+ var cs []span // candidates: kind "PERSON" where one is a name
+ for i := 0; i < len(s); {
+ a, b, ok := zhName(s, i)
+ if b == 0 {
+ _, n := utf8.DecodeRuneInString(s[i:])
+ i += n
+ continue
+ }
+ kind := ""
+ if ok {
+ kind = "PERSON"
+ }
+ cs = append(cs, span{a, b, kind})
+ i = b
+ }
+ // a name joined by 和, 与, 跟, 及 or 、 to one that is a name is one
+ // too: 李娜和刘洋, where only 刘洋 has a word after it that says so
+ starts, ends := map[int]bool{}, map[int]bool{}
+ for _, p := range person {
+ starts[p.start], ends[p.end] = true, true
+ }
+ for _, c := range cs {
+ if c.kind != "" {
+ starts[c.start], ends[c.end] = true, true
+ }
+ }
+ for grew := true; grew; {
+ grew = false
+ for i, c := range cs {
+ if c.kind == "" && joined(s, c, starts, ends) {
+ cs[i].kind, grew = "PERSON", true
+ starts[c.start], ends[c.end] = true, true
+ }
+ }
+ }
+ var out []span
+ for _, c := range cs {
+ if c.kind != "" {
+ out = append(out, c)
+ }
+ }
+ return out
+}
+
+// joined says only 和, 与, 跟, 及 or 、 is between c and a name, one
+// that ends in ends or starts in starts.
+func joined(s string, c span, starts, ends map[int]bool) bool {
+ for _, j := range []string{"和", "与", "跟", "及", "、"} {
+ if strings.HasPrefix(s[c.end:], j) && starts[c.end+len(j)] ||
+ strings.HasSuffix(s[:c.start], j) && ends[c.start-len(j)] {
+ return true
+ }
+ }
+ return false
+}
+
+// zhName is the name that may start at i: its span, and whether what is
+// around it says it is one (b is 0 where none can start there).
+func zhName(s string, i int) (a, b int, ok bool) {
+ rest := s[i:]
+ if r, _ := utf8.DecodeRuneInString(rest); !mayStart[r] {
+ return 0, 0, false
+ }
+ sur := 0
+ for _, c := range zhCompound {
+ if strings.HasPrefix(rest, c) {
+ sur = len(c)
+ break
+ }
+ }
+ nick := false
+ if sur == 0 {
+ r, n := utf8.DecodeRuneInString(rest)
+ switch {
+ case (r == '小' || r == '老') && len(rest) > n:
+ if r2, n2 := utf8.DecodeRuneInString(rest[n:]); isNick[r2] {
+ sur, nick = n+n2, true
+ }
+ case isSurname[r]:
+ sur = n
+ }
+ }
+ if sur == 0 {
+ return 0, 0, false
+ }
+ // the given name: none after 小王, else one or two characters
+ ends := []int{}
+ if nick {
+ ends = append(ends, i+sur)
+ } else {
+ at := i + sur
+ for k := 0; k < 2 && at < len(s); k++ {
+ r, n := utf8.DecodeRuneInString(s[at:])
+ if !unicode.Is(unicode.Han, r) || notGiven[r] {
+ break
+ }
+ at += n
+ ends = append(ends, at)
+ }
+ }
+ if len(ends) == 0 {
+ return 0, 0, false
+ }
+ for _, w := range zhWords {
+ if strings.HasPrefix(rest, w) {
+ return 0, 0, false
+ }
+ }
+ // a Han character before it that is no cue makes it part of a word
+ // (建国路), unless a cue ends there
+ before := s[:i]
+ role, verb := endsAny(before, zhRoles, true), endsAny(before, zhBefore, false)
+ start := sentenceStart(before)
+ if !role && !verb && !start {
+ return 0, 0, false
+ }
+ // a surname and a title is a person (孙经理, 邓总)
+ if !nick {
+ for _, t := range append(zhTitles, "总") {
+ if rest := s[i+sur:]; strings.HasPrefix(rest, t) && !(t == "总" && zhTotal(rest[len(t):])) {
+ return i, i + sur + len(t), true
+ }
+ }
+ }
+ // the longest that ends where something after it says so
+ for k := len(ends) - 1; k >= 0; k-- {
+ e := ends[k]
+ if after := s[e:]; startsAny(after, zhAfter) {
+ return i, e, true
+ } else if (role || verb) && endsWord(after) {
+ return i, e, true
+ }
+ }
+ if role {
+ return i, ends[len(ends)-1], true
+ }
+ return i, ends[len(ends)-1], false
+}
+
+// zhTotal says 总 before s is a word's (总是, 总结, 总共), not a title.
+func zhTotal(s string) bool {
+ r, _ := utf8.DecodeRuneInString(s)
+ return strings.ContainsRune("是结共体之数计线量部归算会得要", r)
+}
+
+// endsAny says s ends with one of ws; for a role, with one followed by
+// 是, 为, a colon or spaces.
+func endsAny(s string, ws []string, role bool) bool {
+ if role {
+ s = strings.TrimRight(s, " \t")
+ for _, sep := range []string{":", ":", "是", "为"} {
+ if t, ok := strings.CutSuffix(s, sep); ok {
+ s = strings.TrimRight(t, " \t")
+ break
+ }
+ }
+ }
+ for _, w := range ws {
+ if strings.HasSuffix(s, w) {
+ return true
+ }
+ }
+ return false
+}
+
+func startsAny(s string, ws []string) bool {
+ for _, w := range ws {
+ if strings.HasPrefix(s, w) {
+ return true
+ }
+ }
+ return false
+}
+
+// endsWord says a word ends where s starts: s is empty, or starts with no
+// Han character.
+func endsWord(s string) bool {
+ r, _ := utf8.DecodeRuneInString(s)
+ return s == "" || !unicode.Is(unicode.Han, r)
+}
+
+// sentenceStart says a name may start after s: s is empty or ends in
+// punctuation or a space.
+func sentenceStart(s string) bool {
+ if s == "" {
+ return true
+ }
+ r, _ := utf8.DecodeLastRuneInString(s)
+ return unicode.IsPunct(r) || unicode.IsSpace(r) || unicode.IsSymbol(r)
+}
+
+// zhTrim is the model's name s[a:b] with what it ran on into cut off: at
+// the first punctuation after a Han character, at a title
+// (张伟先生). One that runs on past its name (林黛玉进贾府) is left so:
+// a name of four characters is as often a whole one (田中太郎).
+func zhTrim(s string, a, b int) (int, int) {
+ v := s[a:b]
+ if !hasHan(v) {
+ return a, b
+ }
+ han := false
+ for k, r := range v {
+ if han && unicode.IsPunct(r) && r != '·' && r != '・' && utf8.RuneCountInString(v[:k]) >= 2 {
+ v = v[:k]
+ break
+ }
+ han = han || unicode.Is(unicode.Han, r)
+ }
+ for k := range v {
+ if utf8.RuneCountInString(v[:k]) >= 2 && startsAny(v[k:], zhTitles) {
+ v = v[:k]
+ break
+ }
+ }
+ return a, a + len(v)
+}
+
+func hasHan(s string) bool {
+ for _, r := range s {
+ if r >= 0x3400 && unicode.Is(unicode.Han, r) {
+ return true
+ }
+ }
+ return false
+}
diff --git a/internal/redact/zhname_test.go b/internal/redact/zhname_test.go
new file mode 100644
index 000000000..011c3d100
--- /dev/null
+++ b/internal/redact/zhname_test.go
@@ -0,0 +1,93 @@
+package redact
+
+import (
+ "strings"
+ "testing"
+)
+
+// Chinese names are masked with the model on, both those it finds and
+// those it misses. modelSaid is what the real model (privacy-filter q8,
+// threshold 0.5, on an M4 Pro) found in each sentence, run on as it was.
+func TestChineseNames(t *testing.T) {
+ modelSaid := map[string]string{
+ "王芳,她下周一": "private_person",
+ "刘洋": "private_person",
+ "赵丽颖": "private_person",
+ "欧阳娜娜": "private_person",
+ "司马光": "private_person",
+ "张伟先生": "private_person",
+ "周杰伦": "private_person",
+ "田中太郎": "private_person",
+ // as it might: a name spaced out, a surname and a title
+ "张 伟": "private_person",
+ "李老师": "private_person",
+ }
+ f := &fakeModel{values: modelSaid}
+ o := Options{Personal: true, Model: f.find}
+ for _, c := range []struct {
+ s string
+ names []string // masked
+ keep []string // left as they are
+ }{
+ {"帮我写封信给张伟,他住在北京市朝阳区建国路88号", []string{"张伟"}, []string{",他住在"}},
+ {"请把合同发给王芳,她下周一到上海。", []string{"王芳"}, []string{",她下周一到上海"}},
+ {"李娜和刘洋昨天开会讨论了项目进度。", []string{"李娜", "刘洋"}, []string{"和", "昨天开会"}},
+ {"客户陈建国反馈导出功能在他电脑上失败了。", []string{"陈建国"}, []string{"反馈导出"}},
+ {"客户张三反馈导出失败", []string{"张三"}, []string{"反馈"}},
+ {"联系人:赵丽颖,手机", []string{"赵丽颖"}, nil},
+ {"提交者是欧阳娜娜,审核人是司马光。", []string{"欧阳娜娜", "司马光"}, []string{"审核人是"}},
+ {"张伟先生,您好!感谢您的来信。", []string{"张伟"}, []string{"先生,您好"}},
+ {"我叫黄晓明,是这个仓库的维护者。", []string{"黄晓明"}, []string{"我叫"}},
+ {"// author: 周杰伦", []string{"周杰伦"}, nil},
+ {"给小王发个消息,说老李明天不来了。", []string{"小王", "老李"}, []string{"发个消息", "明天不来了"}},
+ {"马云说过,今天很残酷。", []string{"马云"}, []string{"说过"}},
+ {"田中太郎さんに連絡してください。", []string{"田中太郎"}, nil},
+ {"转交张 伟处理", []string{"张 伟", "伟"}, nil},
+ {"感谢李老师的帮助", []string{"李老师", "李"}, nil},
+ // none of these is a name
+ {"这个函数的作用是把高度转换成像素。", nil, []string{"高度"}},
+ {"王国的军队在黄河边扎营。", nil, []string{"王国", "黄河"}},
+ {"我们需要一个更高效的排序算法。", nil, []string{"高效"}},
+ {"请帮我修复这个 bug,张开的括号没有闭合。", nil, []string{"张开"}},
+ {"给张量赋值之前先检查形状。", nil, []string{"张量"}},
+ {"在建国路上,路边有一家店。", nil, []string{"建国路", "路边"}},
+ {"把方法和文件名都改一下", nil, []string{"方法", "文件"}},
+ {"给老师发个消息", nil, []string{"老师"}},
+ {"王老板说这个需求下周要上线,邓总明天要来看演示。", []string{"王老板", "邓总"}, []string{"明天要来"}},
+ {"请帮我回复一下孙经理的邮件", []string{"孙经理"}, []string{"的邮件"}},
+ {"抄送给吴敏和郑浩。", []string{"吴敏", "郑浩"}, nil},
+ {"负责人:许文强", []string{"许文强"}, nil},
+ {"我是郭靖,想咨询一下退款的问题。", []string{"郭靖"}, []string{"想咨询"}},
+ {"和平时一样,周五发版。", nil, []string{"周五", "和平"}},
+ {"方总结一下:说明文档里要写清楚安装步骤。", nil, []string{"方总结", "说明"}},
+ {"运行测试的时候,内存一直在涨。签名和工具调用都会保留。", nil, []string{"签名和工具"}},
+ } {
+ out, _ := Mask(c.s, o)
+ for _, n := range c.names {
+ if strings.Contains(out, n) {
+ t.Errorf("%q: %q not masked: %q", c.s, n, out)
+ }
+ }
+ for _, k := range c.keep {
+ if !strings.Contains(out, k) {
+ t.Errorf("%q: %q masked: %q", c.s, k, out)
+ }
+ }
+ if back := Restore(out, false); back != c.s {
+ t.Errorf("round trip: %q", back)
+ }
+ }
+}
+
+// Without the model, or with personal data off, no Chinese name is
+// looked for.
+func TestChineseNamesNeedModel(t *testing.T) {
+ const s = "客户钟离秋反馈导出失败" // a name no other test masks: one masked once stays masked (known.go)
+ if out, _ := Mask(s, Options{Personal: true}); out != s {
+ t.Fatalf("no model: %q", out)
+ }
+ f := &fakeModel{}
+ if out, _ := Mask(s, Options{Secrets: true, Model: f.find}); out != s {
+ t.Fatalf("personal data off: %q", out)
+ }
+}