Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
privacy: Settings names what reads AstrLink Guard, CoreML on its device or the CPU
yetone1b7a08756597
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.mdindex 6e81f7afd..7231c95dd 100644--- a/docs/subsystems/gateway-routing.md+++ b/docs/subsystems/gateway-routing.md@@ -139,7 +139,7 @@ recorded as a prediction failure (`noPrediction`).1. `handle(proto)` reads the body and the model (`requestBody`, `requestModel`), then calls `serveAgent`. Every route that reads an agent's body (each API, count_tokens, Gemini's, embeddings, images, videos, System One, the Codex backend path) reads it through `readBoundedRequestBody` in [`request_bounds.go`](../../internal/gateway/request_bounds.go). It decodes a `Content-Encoding` of `gzip` or `zstd` (`decodedBody`; Codex sends zstd to a custom provider's `/v1/responses` too, #1223) and drops the header, so the provider is sent plain JSON, with one exception: a ChatGPT account's turn (POST `CodexBase`/responses, from the accounts loop in `forwardOnce` or relayed by `codexUpstreamOn`) is compressed again with zstd and a window of at most 2 MiB, as Codex's own client sends it (`provider.CodexCompressed`; openai/codex `responses_request_compression` and `http-client/src/request.rs`). It is compressed once the body is final, and again on each try. `/responses/compact`, an API key's requests and every other provider stay plain. Sent plain, a 1.1 MB text-only turn was answered 503 (ylorn on X). The plain body is what magpie keeps and shows, and `codexDropped` judges a reset by the bytes that went on the wire. `codexDropped` also reads Envoy's 507 "exceeded request buffer limit while retrying upstream" (the backend failed and the front couldn't retry a body that large) as that reset, at any size, told as a 503 so magpie, Codex and pi ask again and nobody rests (#2034). The size limit (`defaultBodyLimit`, 512 MB) holds for the decoded bytes (413), and any other encoding is refused 415. It was 128 MB until a long Codex session that had read many images went past it decoded, its inline screenshots resent each turn, and got the 413 on all five of Codex's retries (#1422, `TestImageHeavyCodexTurnIsRead`). magpie's own 413 says it is magpie's gateway limit, with the size, and that the provider was not asked (`overLimit`); `writeError` adds to any 413 it isn't holding that this is a byte limit, not the model's context, and to `/compact` or start a new conversation (`TestOverLimitSaysWhoseAndWhatToDo`). Every shape but Gemini's types a 413 `request_too_large`, Anthropic's own type, which pi's overflow table knows: pi compacts and tries once more rather than stopping on an OpenAI-shaped `api_error` (@wilsonxuuuu on X, `TestPayloadTooLargeSaysRequestTooLargeToPi`). `keyLimited` reads a limited key's compressed body the same way, so its reservation is the decoded size. `serveAgent` runs the [middleware](gateway-middleware.md) chain, then `serve`.2. `serve` redacts secrets in the request (`redacted`; `redact.MaskJSON` leaves ids alone, and in any text a backend's id (a UUID, 32 hex digits or more, or a digest of 16 to 31 hex digits in one case with letters and digits both, a cache key or a short commit, which went as a key before: _kaichen on X, 10-28; `digestShaped`), or an element token cua-driver hands the model for computer use (`s0000002a:14`, #3782), goes as written: no rule, model label or value known from before masks a piece of one (a phone number at the end of `…-a13800138000`, a card number across a UUID's groups), and a whole one is masked only by the user's own words and rules or as a credential, by the password, API key and token rules or a secret rule whose field is named for one (`api_key=`, `refresh_token`), never one named for an id, a cursor, a request or a digest (`secret_id`, `next_page_token`, `Idempotency-Key`, `secret_hash`), nor cua-driver's own `element_token` (any other name with `element` in it, `element_api_key`, is a credential's as before), a bare `key` (not after api, access, secret, private, signing or license: `keyPhrase`), a field named for a cache, commit, object, blob, file, content, revision, partition, shard, bucket, dedup, lookup, row or sort key, a serial number, or what the model takes a UUID or an element token for, nor an element token known from before (one the model labelled in prose a turn earlier) (`noIDs`, [`internal/redact/ids.go`](../../internal/redact/ids.go)); and a tool's name and its MCP server's (`namespace`, `server_label`, `server_name`, #3143); the request's tool definitions and output schemas, found by their place in each protocol (top-level `tools`, `tool_choice`, `response_format`, `text`, `generationConfig`, `stream_options`, the client's ids in `client_metadata` and the like, and the tools of a Responses `additional_tools` or `tool_search_output` item), go as written save their descriptions, so a word the user masks can't break a grammar or an enum (#1911); and what the vendor sealed with a signature, a thinking block, Gemini's thoughts or Responses' encrypted reasoning, goes back as the vendor wrote it: only the values masked elsewhere in the request, and secrets masked before, are put back as their placeholders; a value masked once, at least 8 characters long, is masked again wherever a request has it, as an exact match whatever words are around it, so what the vendor wrote about a placeholder doesn't go back with the value in it, `redact/known.go`; a text masked for a vendor before, such as a system prompt or a turn already sent, is masked as it was then, so a value a rule first finds later in a conversation doesn't change the prompt the vendor has cached and rewrite all of it to the cache, `sentKnownSpans`), resolves the model to a provider or a group, and refuses what can't be served. The reply has the values put back on its way to the agent (`redact.Writer`, a stream event by event, a whole body at its end, before the writer outside it that names the member, `memberWriter`, writes what it holds: the other way round a whole reply with a value masked in its request went out a 200 with an empty body, #4129, `TestMaskedWholeReplyNamesMember`): a placeholder the model wrote as it was given, or as models rewrite one (spaces inside the braces, a brace or both dropped, another case), is the value again when what is left names one magpie made, and one split across events is held until it is whole (`redact.Restore`, `partialTail`, [`internal/redact/redact.go`](../../internal/redact/redact.go)). A request whose provider and fallbacks, or whose group's models, all skip redaction (`unredactedRoute`, `provider.SkipsRedaction`: set *Send requests unmasked*, `Unredacted`, with every address on this machine or the local network) goes to them as the agent wrote it and isn't masked on its way, since a request is masked only once its route is known (#3973: masking a body whose masked copy was thrown away took up to minutes of magpie's own time). The log, recent calls and the request archive keep it as it went (the archive's bodies have their secrets taken out, `redact.ScrubJSON`, as every call's do), whether the archive is on or off; an observer that wants the bodies (the OTLP export, a JS plugin's) is handed it masked, masked once when it is first handed it after a try (`Call.maskIn`, `laterMasked`), never before the request is sent. A request turned away before it is routed is logged masked (`maskedCopy`). count_tokens and embeddings follow the same rule. What a routing group's Jev is told of the turn (the message, the user's message before it and the reply to it) is masked all the same unless Jev's own provider skips redaction too (`maskedState` in `askJev`, [`decide.go`](../../internal/gateway/decide.go)): a remote Jev is none of the group's members. The masking is the built-in plugin `redact`'s (`gwmw.Masker`, [Plugin runtime](plugin-runtime.md)); when the settings ask for masking and no masker serves, or it fails, the request is a 503 and nothing is sent (`redacted`, `gwmw.Masking`).- With *Mask personal data* on, each kind in `redact.Categories` ([`internal/redact/more.go`](../../internal/redact/more.go)) is masked unless the user turned it off (`redactKinds` in settings, `Options.Kinds`). On by default: US SSNs, passport numbers, IBANs that pass mod 97, birthdays next to a word that says so (a date over a year ago: one in the last year or ahead is the day an agent runs on, `dateOK`); the word must be right before the date, so a log line's or a file name's date is none, and a time beside the date doesn't make it a log's: a date of birth stored as a time (`1990-05-11T16:00:00.000Z`) is one (`TestBirthdayLogDates`; #3878), MAC addresses, and serial numbers next to S/N or the like. Eleven digits from 13 to 19 are a mainland mobile's form and an id's too, so they are a phone only with +86 or 86 before them, as the value of a field whose name says phone (phone, mobile, tel, cell, contact, 手机, 电话, a field named `number`), or in prose with no id's word right before them (order, id, no., #, 订单号, 编号, 单号, 流水号); under order_id or uid, after 订单号, or with a letter or digit against them they go as written (`phoneAt` in [`internal/redact/phone.go`](../../internal/redact/phone.go); _kaichen on X, 10-28). Off by default: the user name in a home folder, user@host in a prompt or ssh command, public IPv4 and IPv6 addresses (never private, loopback, link-local, CGNAT or documentation ranges) and cloud bucket names, since agents send them in nearly every request and the model must copy the placeholder exactly inside a path or URL. When the user name is on, it goes as a made-up name of 8 lowercase letters, the same for that name every turn and put back in replies like any placeholder (`standIn` in [`shape.go`](../../internal/redact/shape.go)), not as `{{USERNAME_…}}`: with mimo-v2.6-flash a model read `C:\Users\{{…}}` as a temporary folder and set out to delete it (eerttreeeert on X, 10-28). A public IP address goes the same way as an address from the documentation ranges (192.0.2.0/24, 198.51.100.0/24 and 203.0.113.0/24, host .101 to .254; 2001:db8::/32 for IPv6), the same one for that address while magpie runs and never another address's, put back only where it stands as a whole address, in a streamed tool call's arguments too (`ipStandIn`, `restoreAddr`; `TestIPGoesAsADocumentationAddress`). Told `{{IP_…}}`, models asked the user for the real address or searched `~/.ssh` for it instead of running ssh (0x0_air on X, 10-28); past the 462 IPv4 stand-ins an address goes as its placeholder. An EVM address (`0x` and 40 hex digits) is public, so the local model's finds leave it as written unless a key's name is right before it; a 64-digit private key is still masked (`evmRe` in `modelFits`; `TestEVMAddressGoesAsWritten`). Context rules run only near their words (`rule.matches`). What reads as an address but ends in a file's extension that no domain has (`…@2x.jpg`, `[email protected]`; `fileExt` in `realEmail`), or a scale before an extension that is a domain too (`[email protected]`, `[email protected]`; `scaleExt`), is a file's name, for the rule and the local model alike (#3029). A domain needs a dot and labels a host can have, so a disassembly's symbols go as written for both too: `system@plt`, `[email protected]`, `memcpy@GLIBC_2.14`, `foo@@GLIBC_2.2.5`, and `system@got` (`.got` is a top-level domain; `mailHost` in `realEmail`, `TestSymbolIsNoEmail`; ch35tnut_ on X, 10-28). `[email protected]` is still an address. The Settings rows come from `REDACT_KINDS` in app.js, which `TestRedactKindsAreRedacts` holds to the same ids and defaults.- - With *Find with a local model* on (`masking.ModelWanted`: on by default since 10-28, `settings.RedactModel` the user turning it on and `RedactModelOff` off, this computer's own, kept by `KeepOwn`; a user with a model downloaded or picked and neither flag turned it off before the default, and `RedactModelAuto` marks the default's own download), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`), but for what the agent defines (its tools' and schemas' descriptions, under `defKeys`), which the rules alone mask, as with the model off, after the rest of the request so a value the model found in a message is masked there too as known: those were 52,577 of Claude Code 2.1.296's 67,532 bytes the model read, and a 4-core server's first request waited out the gateway's 2 minutes (niuniu_8964 on X; 11.5s cold at 4 procs on a Ryzen AI Max+, 2.6s without them, and on three real Claude Code requests the model masked nothing there the rules didn't); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken, nor a snake_case word as a name (Codex's `sequential_cutoff`, #3144), nor as a secret what names a thing: a UUID, a digest with its algorithm's name, a file's path (`namesThing`; the model takes all three for secrets). A package's name goes as written, for the model and the rules alike (dniray on Discord, v0.1.1917): the module a line of code loads (`from ' drizzle-orm '`, `require('jsonwebtoken')`, `from x import`, `use x::`, with an editor's line number or a diff's mark before it; `moduleAt`, `codeHead`), a manifest's dependency pinned at a version (`"drizzle-orm": "^0.36.4"`, TOML's `x = "9"`; `dependencyAt`), a folder under node_modules or site-packages, the module an error could not find (`Cannot find module 'x'`, `No module named 'x'`; `missingModuleAt`), a package an install names (`pnpm add x`; `installedAt`) or pins with @ (`[email protected]`; `pinnedAt`), all in `packageAt`; a value known from an earlier mask (`knownSpans`) goes as written there too, since once the model took the name for a secret anywhere in a request it was masked on the import line as known (dniray, v0.1.1965), a requirement's name before its pinned version (`django-rest-passwordreset==1.4.0`; `pinnedAfter`), a lock file's integrity hash (`sha512-…`; `sriRe` in `namesThing`), a version as the value of a field named for a secret (`"js-tokens": "^4.0.0-beta.1"`; `versionValue` in `fieldAt` and `keyAt`), and a package pinned at its version as user@host has it (`[email protected]:` in pnpm-lock.yaml; `notPackagePin`); `TestPackageNamesGoAsWritten` and `TestModelPackageNamesNotMasked` hold them. The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with Privacy Filter a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name; 和 or 与 says so only after a person, 吴敏和郑浩 but not 判断和边界, 发 or 打 only as the verb, 发了个 but not 关键发现, and a sentence starts after punctuation, not after a space after code, `gh pr view` 易触发; xuanwo on X; 一下 only after a verb that asks a person, 问一下张伟 but not 改一下文案, a word any subject takes after it, 已经, 明天 or 他, only with a person's cue before it, and 说明 is no 说; zeewell on X), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left, and so is a common word that starts with one (`zhWords`: 文案, 文档, 方案; 毛利, 毛重, 毛巾, since 毛利 is a margin far more often than the Japanese surname, so 毛利低 goes as written: _kaichen on X, 10-28; the model still finds 毛利兰, and neither model nor rule masks 毛利小五郎), the model's own name too when it is that word whole. A name the model or the cues take is also no name when it is an ordinary word (`zhWordShaped`, `zhWordNotName`, the one check both go through): a colour (黄绿色, 红色, any that ends in 色 or is colour characters alone: hh05928822 on X after v0.1.1850), a word that starts with a surname from `zhWords` or `zhDict` (about 1,550 two-character words from jieba's dictionary, [`zhdict.txt`](../../internal/redact/zhdict.txt), its header saying how they were chosen; 边界, 应该, 尚可), or such a word with what makes a longer word after it (关键词, 关键字: wangjunxiao128 on X; 应用层, 毛利率), unless a role before it or a title after it says a person is meant (客户方圆, 江山老师), where a name that is also a word is masked. The surnames stay (黄 starts 黄绿色 and 黄晓明), and 毛利兰, 关羽 and 黄晓明 are still masked. With AstrLink Guard only a nickname is (小王, 老李, `zhNick`; `redact.Options.NamesByModel`): Guard at 0.8 finds the full names those cues did in zhname_test's sentences (张伟, 陈建国, 欧阳娜娜…), and misses the nicknames, 孙经理, and all but the surname of 王老板 and 邓总; the cues took words for names (关键发现; xuanwo on X). The model is the user's pick (`settings.RedactModelName`, `pii.Choose`; the app's menu beside the switch, `pmPick`): AstrLink Guard (QuantumNous/astrlink-guard at a pinned revision, Apache-2.0, a 39 MB int8 Multilingual-MiniLM token classifier for Chinese and English), run in magpie's own process on the CPU by [`internal/pii/guard`](../../internal/pii/guard) (an ONNX reader, the XLM-R tokenizer and int8 kernels in Go, NEON/SDOT on arm64 and SSE2 on amd64; its labels named as Privacy Filter's, `guardLabels`); a text over one window, 510 tokens, is read in windows that overlap by 128, and on the CPU as many windows at once as it has cores, half of `GOMAXPROCS` and 4 at most (`guard.Cores`: on every core a long history unread kept a 32-core Linux box at 13 cores' worth for as long as requests came, xinvxueyuan and shensmart on Discord, 10-28): one window's matmuls split over the cores left them idle between, and a 4-core server's first Claude Code request waited minutes (niuniu_8964 on X; 22,488 tokens took 23.9s at 4 procs, 17.9s at once, and 48.8s against 30.5s at 2), and on a Mac through CoreML on the Neural Engine or the GPU ([`coreml.go`](../../internal/pii/guard/coreml.go), [`internal/pii/guard/coreml`](../../internal/pii/guard/coreml); yyhhyyyyyy on X): the same network from the same int8 weights in float16, written from the downloaded model into a program made by `coreml_gen.py` and compiled once into the model's folder, 512 tokens in about 12 ms where the CPU takes 400, its probabilities within .07 of the publisher's fp32 model where the CPU's are within .13 (the CPU quantizes activations as the int8 model does, CoreML doesn't), so a span near the threshold can fall the other side (a go.mod module path is a secret at .81 on CoreML, .79 on the CPU, .785 in fp32); a window under 16 tokens (`coremlMin`), a Mac before macOS 13 or any failure goes to the CPU, loaded on the first request and let go after 10 minutes with none; or OpenAI's Privacy Filter. With no pick it is AstrLink Guard, the default; a user who had Privacy Filter on before there was a choice keeps it only while AstrLink Guard downloads, which the first masked request starts and which then saves AstrLink Guard as the pick (`masking.picked`, `toGuard`; an hour between tries after a failure). Measured for that default (#3578): on 234 segments of real code and config it found 2 spans where Privacy Filter found 100 (38 of them go.sum hashes after redact's own checks), and with the rules it found 42 of 42 test values to Privacy Filter's 41. Each model has its own download, cache file and turn (`backend`). Both find at a floor of 0.5 (`pii.Floor`) and give each span its score (AstrLink Guard: the least of its tokens' probabilities, rounded to a float32 so a cached span reads back the same; Privacy Filter: pf-serve's own), and masking keeps those at the model's threshold or over (`pii.Threshold` in `masking.model`): the user's for that model (`settings.RedactModelScore`, this computer's own; Settings → Privacy's Confidence threshold, `pmScore`, 0.5 to 0.95), else the model's own (`MinScore`: Privacy Filter 0.5, AstrLink Guard 0.8, its publisher's). A value known from before follows the same threshold: each value masked remembers the highest score anything found it with (a rule's counts as sure), and one only the model found, under the threshold now, is no longer masked again as known (`noteScores`, `knownSpans` in [`known.go`](../../internal/redact/known.go); before, a raised slider changed nothing for a value masked once: _kaichen on X, 10-28). The defaults stay because on the eval a higher one mostly loses values the checks after the model would have kept (Privacy Filter at 0.85 lost a Chinese name; AstrLink Guard at 0.9 a phone in a tool result), while the false alarms it would drop are mostly dropped by those checks already (Yayoi_no_yume on X). Measured again on 2000 segments of real agent traffic (Codex and Claude Code sessions: prompts, tool arguments and results, a quarter with Chinese) and 30 more labelled cases: AstrLink Guard at 0.88 rather than 0.8 drops 3 false alarms (a model id, a sandbox path word, a token counter) and lets a colleague's name and a signed name through; Privacy Filter at 0.88 rather than 0.5 drops 11 (timestamps in paths, log lines, a plugin id) and lets 6 real names and a phone through (4 of the names in the real traffic). Lowering Guard to 0.5 finds 2 more names in the real traffic and 3 more labelled values, for 10 more false alarms (9 of them Claude Code's `<total_tokens>` counter). Those false alarms are shapes a check can turn away; a threshold can't without losing real values. Privacy Filter is a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags; on a GPU it holds the model once, in the device's memory, #3590; every pf-serve keeps the engine's own MoE chunk, `PF_MOE_CHUNK` at the window's 4096, which an RX 470 through Vulkan read 6-9% faster than 2048 in interleaved runs and Metal ~6% faster, and inherits a value the user set): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, through magpie's proxy (`netproxy.Func`, as its other requests; hh0592821 on Discord, whose download went direct before), from hf-mirror first in China (`source.chinaFirst`) and from hf-mirror after Hugging Face everywhere else, the switch off too (`source.modelMirror`), so a poisoned or blocked huggingface.co doesn't stop it; hf-mirror serves AstrLink Guard's files itself and sends Privacy Filter's back to Hugging Face) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU, doesn't load there in 40s (`gpuLoadTimeout`, well inside the gateway's 2 minutes), or doesn't answer in time there twice in a row, is started on the CPU, and Settings says why; a request it didn't answer in time once is tried on the GPU again, since a slow request on a weak GPU is no lost GPU (#3682). The GPU is tried again 10 minutes later, doubling to 2 hours while it keeps failing, with no restart (`gpuRetryAfter`), and one that answers is the model's again), joins short texts into 12 KB chunks and maps the spans back, cuts a text over 12 KB into pieces of 12 KB at most that overlap by 1 KB (`pieces`: after a line's end, else a space, of those in the piece's second half the one whose 48 bytes before it hash least (`landmark`), so a line edited early in a text leaves the pieces after it as they were, else at a rune's start), each kept by its own hash, their spans put back at the text's offsets and merged where pieces overlap (`merge`: one found twice is one, one cut by a piece's end is whole), and a value of 8 bytes or more found in a text is found wherever else it is in it, as sure as its surest find (`spread`: the model scored an address on 400 lines alike 0.9 on some and 0.5 on most, wangwu25685333 on X), and asks pf-serve a chunk at a time (#3682: a 480 KB text whole took an RX 470/580 90s and kept nothing when cut off), keeps the last 20000 answers by hash, on disk too (`found-<model>.bin` beside the model: hashes, offsets, labels and each span's score, never text; a `pfound01` file from before scores is read with every span sure, so it masks as it did; written through temp-and-rename, a bad read taken as empty), so a restart doesn't ask the model about every history again (#3682), and ends it after 10 minutes with no request. Its requests take turns on its one pipe (`turn`), and a request reads the cache again when its turn comes; a request waits for its turn only while its context lives (`pii.FindContext`, through `gwmw.ContextTeller`), so one whose client went away leaves the queue (the gateway's 2 minutes leave masking's time out, #3682); the chunk pf-serve is on is finished and kept, so a request sent again asks only about what is left, and between chunks the turn is given up and taken again, by the request waiting with the least left to read (`takeTurn`, `queue`), so a short request goes before the rest of every long one (wangwu25685333 on X: a 5 KB request behind two long ones on 4 cores waited 8.3s in the order they came, 4.5s now). The status (`pii.Now`), `pii.Stop` and the idle timer never wait for a load or a request: `srv.mu` is held only to read or set the state, and `Stop` ends a load or a request under way at once. A request held before routing while it waits for the model is answered 504 saying so, and how to send without it (`unroutedMsg`); one of an older tag (`oldTags`, magpie updated since the download) runs until the pinned tag's is fetched (`upgrade`, on its start), which is then started on the next request and the old deleted ([`internal/pii`](../../internal/pii)). A failure names what ended `pf-serve`: magpie's own end of a request it didn't answer in time, else its exit status or signal and its last stderr lines past ggml's startup chatter (`died`, `lastSaid`). After one, requests are held back at once with that failure, not after a load each, until the model is tried again 30s later, doubling to 10 minutes while it keeps failing (`heldBack`, `retryAfter`; lazyeo on Discord, an M2 whose every request waited out the 504). Until the model is on disk the rules mask alone, and its download starts (`masking.fetch`, an hour between tries; the default's, done, saved by `settings.UseDefaultRedactModel` unless the user picked or turned it off meanwhile; cancelling it is turning it off). A model the user turned on that fails (didn't start, isn't on disk, held back after a failure) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`, strict); the default's finds nothing then, and the rules mask alone. A model that is only slow is waited for at most 2 seconds from the request's masking start when on by default (`masking.defaultWait`) and 15 seconds when the user turned it on (`masking.strictWait`): its turn, its load and its reading. Past that, or when the request's context ends first, `pii.FindContext` answers with what was known of the request's texts by then (its cache: the history read on earlier turns, and the chunks read meanwhile) and a `*pii.Partial` saying how many it hadn't read; `masking.model` masks those with the model's finds and the rest with the rules alone, logging it at most once a minute (`notePartial`), and the request goes. Nothing the rules mask goes as written. Before, the user's model was waited for with no bound and any end of the wait held the request back whole, its known finds thrown away: in a burst of requests with text the model hadn't read, xRayySummers on X (0.1.1877, Linux) saw masking take 1.2s to 290s in one minute and each request it ended was "masking failed, so nothing was sent". Reproduced on 8 cores of a Linux CPU: AstrLink Guard reads about 6 KB a second there (a 125 KB history 20.8s), and six requests at once with 67 to 145 KB of new tool results each took 62 to 110s, all ending near the last, since the turn goes round between chunks; with the bound each went at 15.0s with the history's finds and the model's for 1 to 4 of its 9 texts. `masking.find` gives a model that doesn't stop for its context 100ms to answer with what it knew (`handBack`) before giving up on it. AstrLink Guard reads on the CPU, so a long Codex history it hasn't seen took minutes there, and every request queued behind it showed masking with no bound (shensmart on Discord, 10-28, magpie in Docker). The wait is given up even where `Find` doesn't stop for its context. A request that stops waiting leaves the chunks after the one being read (`backend.find`), so it doesn't hold the turn; that chunk's answers are kept for the next request. Each model reads a request's unread texts newest first, the last text and a long one's last piece before the rest (`backend.missing`), since an agent's history is oldest first and what a request adds is at its end; AstrLink Guard joins them into chunks of 4 KB, not Privacy Filter's 12 KB (`backend.chunk`, `guardChunk`), so the chunk a request leaves the model finishing past its wait is about a second on 4 cores. The default's request is `pii.Bounded`: what one went without the model's reading is remembered by hash (`backend.sent`), and a request after it reads what it adds first and that backlog `backlogCap` (4 KB) at most, nothing found for the rest, so a history resumed or never read is read a little a request rather than on every core for each one. Measured on a 2 MB, 200-turn Codex history never read, 30 requests 3s apart (`TestLiveHistoryCost`, gated on `MAGPIE_GUARD_FILES`): on a 32-core Linux box 33 to 60 CPU seconds a request before, 23 minutes of CPU over the session's 108s (13 cores busy), 6 to 10 after, 4 minutes over 90s (2.7 cores), and on an M5 Pro's CoreML 2.2 CPU seconds a request for the first three before, the first alone after; nothing runs on after the last request either way. A request the user turned the model on for still reads all of it, waited for `strictWait`.+ - With *Find with a local model* on (`masking.ModelWanted`: on by default since 10-28, `settings.RedactModel` the user turning it on and `RedactModelOff` off, this computer's own, kept by `KeepOwn`; a user with a model downloaded or picked and neither flag turned it off before the default, and `RedactModelAuto` marks the default's own download), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`), but for what the agent defines (its tools' and schemas' descriptions, under `defKeys`), which the rules alone mask, as with the model off, after the rest of the request so a value the model found in a message is masked there too as known: those were 52,577 of Claude Code 2.1.296's 67,532 bytes the model read, and a 4-core server's first request waited out the gateway's 2 minutes (niuniu_8964 on X; 11.5s cold at 4 procs on a Ryzen AI Max+, 2.6s without them, and on three real Claude Code requests the model masked nothing there the rules didn't); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken, nor a snake_case word as a name (Codex's `sequential_cutoff`, #3144), nor as a secret what names a thing: a UUID, a digest with its algorithm's name, a file's path (`namesThing`; the model takes all three for secrets). A package's name goes as written, for the model and the rules alike (dniray on Discord, v0.1.1917): the module a line of code loads (`from ' drizzle-orm '`, `require('jsonwebtoken')`, `from x import`, `use x::`, with an editor's line number or a diff's mark before it; `moduleAt`, `codeHead`), a manifest's dependency pinned at a version (`"drizzle-orm": "^0.36.4"`, TOML's `x = "9"`; `dependencyAt`), a folder under node_modules or site-packages, the module an error could not find (`Cannot find module 'x'`, `No module named 'x'`; `missingModuleAt`), a package an install names (`pnpm add x`; `installedAt`) or pins with @ (`[email protected]`; `pinnedAt`), all in `packageAt`; a value known from an earlier mask (`knownSpans`) goes as written there too, since once the model took the name for a secret anywhere in a request it was masked on the import line as known (dniray, v0.1.1965), a requirement's name before its pinned version (`django-rest-passwordreset==1.4.0`; `pinnedAfter`), a lock file's integrity hash (`sha512-…`; `sriRe` in `namesThing`), a version as the value of a field named for a secret (`"js-tokens": "^4.0.0-beta.1"`; `versionValue` in `fieldAt` and `keyAt`), and a package pinned at its version as user@host has it (`[email protected]:` in pnpm-lock.yaml; `notPackagePin`); `TestPackageNamesGoAsWritten` and `TestModelPackageNamesNotMasked` hold them. The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with Privacy Filter a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name; 和 or 与 says so only after a person, 吴敏和郑浩 but not 判断和边界, 发 or 打 only as the verb, 发了个 but not 关键发现, and a sentence starts after punctuation, not after a space after code, `gh pr view` 易触发; xuanwo on X; 一下 only after a verb that asks a person, 问一下张伟 but not 改一下文案, a word any subject takes after it, 已经, 明天 or 他, only with a person's cue before it, and 说明 is no 说; zeewell on X), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left, and so is a common word that starts with one (`zhWords`: 文案, 文档, 方案; 毛利, 毛重, 毛巾, since 毛利 is a margin far more often than the Japanese surname, so 毛利低 goes as written: _kaichen on X, 10-28; the model still finds 毛利兰, and neither model nor rule masks 毛利小五郎), the model's own name too when it is that word whole. A name the model or the cues take is also no name when it is an ordinary word (`zhWordShaped`, `zhWordNotName`, the one check both go through): a colour (黄绿色, 红色, any that ends in 色 or is colour characters alone: hh05928822 on X after v0.1.1850), a word that starts with a surname from `zhWords` or `zhDict` (about 1,550 two-character words from jieba's dictionary, [`zhdict.txt`](../../internal/redact/zhdict.txt), its header saying how they were chosen; 边界, 应该, 尚可), or such a word with what makes a longer word after it (关键词, 关键字: wangjunxiao128 on X; 应用层, 毛利率), unless a role before it or a title after it says a person is meant (客户方圆, 江山老师), where a name that is also a word is masked. The surnames stay (黄 starts 黄绿色 and 黄晓明), and 毛利兰, 关羽 and 黄晓明 are still masked. With AstrLink Guard only a nickname is (小王, 老李, `zhNick`; `redact.Options.NamesByModel`): Guard at 0.8 finds the full names those cues did in zhname_test's sentences (张伟, 陈建国, 欧阳娜娜…), and misses the nicknames, 孙经理, and all but the surname of 王老板 and 邓总; the cues took words for names (关键发现; xuanwo on X). The model is the user's pick (`settings.RedactModelName`, `pii.Choose`; the app's menu beside the switch, `pmPick`): AstrLink Guard (QuantumNous/astrlink-guard at a pinned revision, Apache-2.0, a 39 MB int8 Multilingual-MiniLM token classifier for Chinese and English), run in magpie's own process on the CPU by [`internal/pii/guard`](../../internal/pii/guard) (an ONNX reader, the XLM-R tokenizer and int8 kernels in Go, NEON/SDOT on arm64 and SSE2 on amd64; its labels named as Privacy Filter's, `guardLabels`); a text over one window, 510 tokens, is read in windows that overlap by 128, and on the CPU as many windows at once as it has cores, half of `GOMAXPROCS` and 4 at most (`guard.Cores`: on every core a long history unread kept a 32-core Linux box at 13 cores' worth for as long as requests came, xinvxueyuan and shensmart on Discord, 10-28): one window's matmuls split over the cores left them idle between, and a 4-core server's first Claude Code request waited minutes (niuniu_8964 on X; 22,488 tokens took 23.9s at 4 procs, 17.9s at once, and 48.8s against 30.5s at 2), and on a Mac through CoreML on the Neural Engine or the GPU ([`coreml.go`](../../internal/pii/guard/coreml.go), [`internal/pii/guard/coreml`](../../internal/pii/guard/coreml); yyhhyyyyyy on X; Settings › Privacy names which, from CoreML's own compute plan of the compiled model, `MLComputePlan` on macOS 14.4 and later, the device most of its program's estimated cost is on (`coreml.Plan`, `Model.Backend`, `Status.Via` and `Device`): it said "on the CPU" under CoreML, yyhhyyyyyy on Discord, 10-27; an M5 Pro's plan puts all of it on the GPU; after a CoreML read fails it says the CPU and why until one doesn't): the same network from the same int8 weights in float16, written from the downloaded model into a program made by `coreml_gen.py` and compiled once into the model's folder, 512 tokens in about 12 ms where the CPU takes 400, its probabilities within .07 of the publisher's fp32 model where the CPU's are within .13 (the CPU quantizes activations as the int8 model does, CoreML doesn't), so a span near the threshold can fall the other side (a go.mod module path is a secret at .81 on CoreML, .79 on the CPU, .785 in fp32); a window under 16 tokens (`coremlMin`), a Mac before macOS 13 or any failure goes to the CPU, loaded on the first request and let go after 10 minutes with none; or OpenAI's Privacy Filter. With no pick it is AstrLink Guard, the default; a user who had Privacy Filter on before there was a choice keeps it only while AstrLink Guard downloads, which the first masked request starts and which then saves AstrLink Guard as the pick (`masking.picked`, `toGuard`; an hour between tries after a failure). Measured for that default (#3578): on 234 segments of real code and config it found 2 spans where Privacy Filter found 100 (38 of them go.sum hashes after redact's own checks), and with the rules it found 42 of 42 test values to Privacy Filter's 41. Each model has its own download, cache file and turn (`backend`). Both find at a floor of 0.5 (`pii.Floor`) and give each span its score (AstrLink Guard: the least of its tokens' probabilities, rounded to a float32 so a cached span reads back the same; Privacy Filter: pf-serve's own), and masking keeps those at the model's threshold or over (`pii.Threshold` in `masking.model`): the user's for that model (`settings.RedactModelScore`, this computer's own; Settings → Privacy's Confidence threshold, `pmScore`, 0.5 to 0.95), else the model's own (`MinScore`: Privacy Filter 0.5, AstrLink Guard 0.8, its publisher's). A value known from before follows the same threshold: each value masked remembers the highest score anything found it with (a rule's counts as sure), and one only the model found, under the threshold now, is no longer masked again as known (`noteScores`, `knownSpans` in [`known.go`](../../internal/redact/known.go); before, a raised slider changed nothing for a value masked once: _kaichen on X, 10-28). The defaults stay because on the eval a higher one mostly loses values the checks after the model would have kept (Privacy Filter at 0.85 lost a Chinese name; AstrLink Guard at 0.9 a phone in a tool result), while the false alarms it would drop are mostly dropped by those checks already (Yayoi_no_yume on X). Measured again on 2000 segments of real agent traffic (Codex and Claude Code sessions: prompts, tool arguments and results, a quarter with Chinese) and 30 more labelled cases: AstrLink Guard at 0.88 rather than 0.8 drops 3 false alarms (a model id, a sandbox path word, a token counter) and lets a colleague's name and a signed name through; Privacy Filter at 0.88 rather than 0.5 drops 11 (timestamps in paths, log lines, a plugin id) and lets 6 real names and a phone through (4 of the names in the real traffic). Lowering Guard to 0.5 finds 2 more names in the real traffic and 3 more labelled values, for 10 more false alarms (9 of them Claude Code's `<total_tokens>` counter). Those false alarms are shapes a check can turn away; a threshold can't without losing real values. Privacy Filter is a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags; on a GPU it holds the model once, in the device's memory, #3590; every pf-serve keeps the engine's own MoE chunk, `PF_MOE_CHUNK` at the window's 4096, which an RX 470 through Vulkan read 6-9% faster than 2048 in interleaved runs and Metal ~6% faster, and inherits a value the user set): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, through magpie's proxy (`netproxy.Func`, as its other requests; hh0592821 on Discord, whose download went direct before), from hf-mirror first in China (`source.chinaFirst`) and from hf-mirror after Hugging Face everywhere else, the switch off too (`source.modelMirror`), so a poisoned or blocked huggingface.co doesn't stop it; hf-mirror serves AstrLink Guard's files itself and sends Privacy Filter's back to Hugging Face) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU, doesn't load there in 40s (`gpuLoadTimeout`, well inside the gateway's 2 minutes), or doesn't answer in time there twice in a row, is started on the CPU, and Settings says why; a request it didn't answer in time once is tried on the GPU again, since a slow request on a weak GPU is no lost GPU (#3682). The GPU is tried again 10 minutes later, doubling to 2 hours while it keeps failing, with no restart (`gpuRetryAfter`), and one that answers is the model's again), joins short texts into 12 KB chunks and maps the spans back, cuts a text over 12 KB into pieces of 12 KB at most that overlap by 1 KB (`pieces`: after a line's end, else a space, of those in the piece's second half the one whose 48 bytes before it hash least (`landmark`), so a line edited early in a text leaves the pieces after it as they were, else at a rune's start), each kept by its own hash, their spans put back at the text's offsets and merged where pieces overlap (`merge`: one found twice is one, one cut by a piece's end is whole), and a value of 8 bytes or more found in a text is found wherever else it is in it, as sure as its surest find (`spread`: the model scored an address on 400 lines alike 0.9 on some and 0.5 on most, wangwu25685333 on X), and asks pf-serve a chunk at a time (#3682: a 480 KB text whole took an RX 470/580 90s and kept nothing when cut off), keeps the last 20000 answers by hash, on disk too (`found-<model>.bin` beside the model: hashes, offsets, labels and each span's score, never text; a `pfound01` file from before scores is read with every span sure, so it masks as it did; written through temp-and-rename, a bad read taken as empty), so a restart doesn't ask the model about every history again (#3682), and ends it after 10 minutes with no request. Its requests take turns on its one pipe (`turn`), and a request reads the cache again when its turn comes; a request waits for its turn only while its context lives (`pii.FindContext`, through `gwmw.ContextTeller`), so one whose client went away leaves the queue (the gateway's 2 minutes leave masking's time out, #3682); the chunk pf-serve is on is finished and kept, so a request sent again asks only about what is left, and between chunks the turn is given up and taken again, by the request waiting with the least left to read (`takeTurn`, `queue`), so a short request goes before the rest of every long one (wangwu25685333 on X: a 5 KB request behind two long ones on 4 cores waited 8.3s in the order they came, 4.5s now). The status (`pii.Now`), `pii.Stop` and the idle timer never wait for a load or a request: `srv.mu` is held only to read or set the state, and `Stop` ends a load or a request under way at once. A request held before routing while it waits for the model is answered 504 saying so, and how to send without it (`unroutedMsg`); one of an older tag (`oldTags`, magpie updated since the download) runs until the pinned tag's is fetched (`upgrade`, on its start), which is then started on the next request and the old deleted ([`internal/pii`](../../internal/pii)). A failure names what ended `pf-serve`: magpie's own end of a request it didn't answer in time, else its exit status or signal and its last stderr lines past ggml's startup chatter (`died`, `lastSaid`). After one, requests are held back at once with that failure, not after a load each, until the model is tried again 30s later, doubling to 10 minutes while it keeps failing (`heldBack`, `retryAfter`; lazyeo on Discord, an M2 whose every request waited out the 504). Until the model is on disk the rules mask alone, and its download starts (`masking.fetch`, an hour between tries; the default's, done, saved by `settings.UseDefaultRedactModel` unless the user picked or turned it off meanwhile; cancelling it is turning it off). A model the user turned on that fails (didn't start, isn't on disk, held back after a failure) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`, strict); the default's finds nothing then, and the rules mask alone. A model that is only slow is waited for at most 2 seconds from the request's masking start when on by default (`masking.defaultWait`) and 15 seconds when the user turned it on (`masking.strictWait`): its turn, its load and its reading. Past that, or when the request's context ends first, `pii.FindContext` answers with what was known of the request's texts by then (its cache: the history read on earlier turns, and the chunks read meanwhile) and a `*pii.Partial` saying how many it hadn't read; `masking.model` masks those with the model's finds and the rest with the rules alone, logging it at most once a minute (`notePartial`), and the request goes. Nothing the rules mask goes as written. Before, the user's model was waited for with no bound and any end of the wait held the request back whole, its known finds thrown away: in a burst of requests with text the model hadn't read, xRayySummers on X (0.1.1877, Linux) saw masking take 1.2s to 290s in one minute and each request it ended was "masking failed, so nothing was sent". Reproduced on 8 cores of a Linux CPU: AstrLink Guard reads about 6 KB a second there (a 125 KB history 20.8s), and six requests at once with 67 to 145 KB of new tool results each took 62 to 110s, all ending near the last, since the turn goes round between chunks; with the bound each went at 15.0s with the history's finds and the model's for 1 to 4 of its 9 texts. `masking.find` gives a model that doesn't stop for its context 100ms to answer with what it knew (`handBack`) before giving up on it. AstrLink Guard reads on the CPU, so a long Codex history it hasn't seen took minutes there, and every request queued behind it showed masking with no bound (shensmart on Discord, 10-28, magpie in Docker). The wait is given up even where `Find` doesn't stop for its context. A request that stops waiting leaves the chunks after the one being read (`backend.find`), so it doesn't hold the turn; that chunk's answers are kept for the next request. Each model reads a request's unread texts newest first, the last text and a long one's last piece before the rest (`backend.missing`), since an agent's history is oldest first and what a request adds is at its end; AstrLink Guard joins them into chunks of 4 KB, not Privacy Filter's 12 KB (`backend.chunk`, `guardChunk`), so the chunk a request leaves the model finishing past its wait is about a second on 4 cores. The default's request is `pii.Bounded`: what one went without the model's reading is remembered by hash (`backend.sent`), and a request after it reads what it adds first and that backlog `backlogCap` (4 KB) at most, nothing found for the rest, so a history resumed or never read is read a little a request rather than on every core for each one. Measured on a 2 MB, 200-turn Codex history never read, 30 requests 3s apart (`TestLiveHistoryCost`, gated on `MAGPIE_GUARD_FILES`): on a 32-core Linux box 33 to 60 CPU seconds a request before, 23 minutes of CPU over the session's 108s (13 cores busy), 6 to 10 after, 4 minutes over 90s (2.7 cores), and on an M5 Pro's CoreML 2.2 CPU seconds a request for the first three before, the first alone after; nothing runs on after the last request either way. A request the user turned the model on for still reads all of it, waited for `strictWait`.- With *Mask secrets* on, the user's own masking rules (`redactRules`, `redact.Rule`: a placeholder name and a prefix or an RE2 pattern, [`internal/redact/rules.go`](../../internal/redact/rules.go)) run before magpie's. Settings → Privacy adds them, removes them, and edits one in place (`POST /api/settings/redact-rule` with the rule's index and its current value, `editRedactRule` in [`internal/gui/api.go`](../../internal/gui/api.go)). Every rule is checked by `redact.CheckRules` when settings are saved; an edit that doesn't compile, matches the empty string, repeats another rule or names a rule no longer at that index is refused with the rules left as they were. The masker reads the rules from settings on each call (`options` in [`internal/masking`](../../internal/masking/masking.go)), so a change applies to the next request.- The user's own masked words (`redactWords`, `Options.Words`; Settings → Privacy's *Masked words*) go as `{{TERM_…}}` whatever else is on, and come back as written. They match as written, inside longer words too, unless the user asks: *Masked words in any case* (`redactWordsAnyCase`, `WordsAnyCase`) matches in any capitals by Unicode simple folding, and *Whole words only* (`redactWordsWhole`, `WordsWhole`; acmilan_ on Discord) skips a match with a letter, digit or `_` beside it, and for a number a `.` and digit beside it, so `ABC` masks `name='ABC'` but not `XABCD`, and `123456` not `91234567`, `0.123456` or `123456.78` (`wholeWord` in [`redact.go`](../../internal/redact/redact.go)). Chinese and Japanese letters count as no letter there (`wordRune`): a word of theirs is found inside their text, and `ABC` in 用ABC做 is whole. Both are off by default; a page from before them that saves without the field keeps the user's choice (`api.go`). `TestWordsWhole`, `TestWordsWholeJSON`, `TestCountTokensRedacted`. A word that is a date (`1990-05-12`, `1990/5/12`, `1990年5月12日`, `May 12, 1990`; eight digits alone are not one) matches that calendar date in every common form instead (`dateSpans` in [`dateword.go`](../../internal/redact/dateword.go); JackyCcao on X, who masks their own birthday with birthdays off): `-` `/` `.` `、` or a space between, 年月日 or 년월일, with or without leading zeros, `19900512`, the month's English name, and day-first or month-first only where the day is over 12 or equals the month (`13/05/1990`, never `05/12/1990`, which is December 5th too; such a word matches only as written). A match never has a digit, or a dot and a digit, beside it, whatever *Whole words only* says, so `1990.5.12.1` and `11990-05-12` are not that date. `TestDateWordEveryForm`, `TestDateWordNotAnotherDate`, `TestDateWordDayMonthOrder`.- With *Keep a key's form* on (`settings.RedactShape`, `Options.Shape`, under *Mask secrets*; Charles Cao on X), a secret of 16 characters or more, all letters, digits, `-`, `_` or `.`, goes as a made-up key of its form instead of a placeholder: its prefix kept (`sk-proj-`, `ghp_`, `AKIA`, up to four letters and their separator; `keyHead`), the same length and separators, and each other character a random one of its kind (a hex key's any hex digit of its case), drawn from an HMAC of the value under the placeholder key, so one key goes as the same made-up one every turn and a vendor's cache still matches (`shapeOf`, `shaped` in [`internal/redact/shape.go`](../../internal/redact/shape.go)). Personal data, the user's words and anything shorter or with other characters go as placeholders still. A made-up key is kept in the masker's memory with the placeholders, so it is recorded, re-masked in what a vendor sealed, and put back wherever the reply has it, before the placeholders (`restoreShapes`); a stream holds the end of an event that may be the start of one (`shapeTail`, after the last whole one), and a made-up key the vendor sends back unrestored isn't masked again (`outsideShapes`). `TestShapeKeepsForm`, `TestShapeJSON`, `TestShapeStream`. Insights › Masking and Settings' *Try it* mark a made-up key where they mark a placeholder, matching the text against the request's own values only, so a key of the same form the user wrote isn't marked (`maskMatches` in `plugins/redact/page.js`; #3599, masking-records and mask-actions GUI tests).diff --git a/internal/gui/assets/app.js b/internal/gui/assets/app.jsindex 877af6749..c62c04931 100644--- a/internal/gui/assets/app.js+++ b/internal/gui/assets/app.js@@ -21722,6 +21722,9 @@ function pmSub(m) {if (m.on && m.failed) return m.chosen ? t("The model didn't answer, so requests are held back: {error}", { error: m.failed }) : t("The model didn't answer, so the rules mask requests alone: {error}", { error: m.failed });if (m.on && m.running) {if (m.gpu) return t("On, on the GPU ({device})", { device: m.device });+ // AstrLink Guard on a Mac: CoreML, on the device its plan picked+ if (m.via) return { "Neural Engine": t("On, with CoreML on the Neural Engine"), GPU: t("On, with CoreML on the GPU"), CPU: t("On, with CoreML on the CPU") }[m.device] || t("On, with CoreML");+ if (m.gpuError && m.model === "astrlink-guard") return t("On, on the CPU: CoreML failed ({error})", { error: m.gpuError });return m.gpuError ? t("On, on the CPU: the GPU failed ({error})", { error: m.gpuError }) : t("On, on the CPU");}if (m.on) return t("On: the model starts with the next request");diff --git a/internal/gui/assets/i18n.js b/internal/gui/assets/i18n.jsindex 17588d2c7..74b622952 100644--- a/internal/gui/assets/i18n.js+++ b/internal/gui/assets/i18n.js@@ -1251,6 +1251,11 @@ const I18N = {"On, on the GPU ({device})": "已开启,运行在 GPU({device})","On, on the CPU: the GPU failed ({error})": "已开启,运行在 CPU:GPU 出错({error})","On, on the CPU": "已开启,运行在 CPU",+ "On, with CoreML on the Neural Engine": "已开启,通过 CoreML 运行在神经网络引擎",+ "On, with CoreML on the GPU": "已开启,通过 CoreML 运行在 GPU",+ "On, with CoreML on the CPU": "已开启,通过 CoreML 运行在 CPU",+ "On, with CoreML": "已开启,通过 CoreML 运行",+ "On, on the CPU: CoreML failed ({error})": "已开启,运行在 CPU:CoreML 出错({error})","On: the model starts with the next request": "已开启:模型会在下一个请求时启动","Off. The model is downloaded ({size})": "已关闭。模型已下载({size})","Your own words to keep from vendors, separated by commas": "不想让模型厂商看到的词,用逗号分隔",@@ -5323,6 +5328,11 @@ const I18N = {"On, on the GPU ({device})": "已開啟,執行於 GPU({device})","On, on the CPU: the GPU failed ({error})": "已開啟,執行於 CPU:GPU 出錯({error})","On, on the CPU": "已開啟,執行於 CPU",+ "On, with CoreML on the Neural Engine": "已開啟,透過 CoreML 執行於神經網路引擎",+ "On, with CoreML on the GPU": "已開啟,透過 CoreML 執行於 GPU",+ "On, with CoreML on the CPU": "已開啟,透過 CoreML 執行於 CPU",+ "On, with CoreML": "已開啟,透過 CoreML 執行",+ "On, on the CPU: CoreML failed ({error})": "已開啟,執行於 CPU:CoreML 出錯({error})","On: the model starts with the next request": "已開啟:模型會在下一個請求時啟動","Off. The model is downloaded ({size})": "已關閉。模型已下載({size})","Your own words to keep from vendors, separated by commas": "不想讓模型廠商看到的詞,用逗號分隔",@@ -9365,6 +9375,11 @@ const I18N = {"On, on the GPU ({device})": "オン、GPU で実行中({device})","On, on the CPU: the GPU failed ({error})": "オン、CPU で実行中:GPU が失敗しました({error})","On, on the CPU": "オン、CPU で実行中",+ "On, with CoreML on the Neural Engine": "オン、CoreML により Neural Engine で実行中",+ "On, with CoreML on the GPU": "オン、CoreML により GPU で実行中",+ "On, with CoreML on the CPU": "オン、CoreML により CPU で実行中",+ "On, with CoreML": "オン、CoreML で実行中",+ "On, on the CPU: CoreML failed ({error})": "オン、CPU で実行中:CoreML が失敗しました({error})","On: the model starts with the next request": "オン:次のリクエストでモデルが起動します","Off. The model is downloaded ({size})": "オフ。モデルはダウンロード済み({size})","Your own words to keep from vendors, separated by commas": "ベンダーに見せたくない語(カンマ区切り)",@@ -13437,6 +13452,11 @@ const I18N = {"On, on the GPU ({device})": "An, auf der GPU ({device})","On, on the CPU: the GPU failed ({error})": "An, auf der CPU: die GPU ist ausgefallen ({error})","On, on the CPU": "An, auf der CPU",+ "On, with CoreML on the Neural Engine": "An, mit CoreML auf der Neural Engine",+ "On, with CoreML on the GPU": "An, mit CoreML auf der GPU",+ "On, with CoreML on the CPU": "An, mit CoreML auf der CPU",+ "On, with CoreML": "An, mit CoreML",+ "On, on the CPU: CoreML failed ({error})": "An, auf der CPU: CoreML ist ausgefallen ({error})","On: the model starts with the next request": "An: das Modell startet mit der nächsten Anfrage","Off. The model is downloaded ({size})": "Aus. Das Modell ist geladen ({size})","Your own words to keep from vendors, separated by commas": "Eigene Wörter, die Anbieter nicht sehen sollen, durch Kommas getrennt",diff --git a/internal/gui/tests/privacy-model.test.cjs b/internal/gui/tests/privacy-model.test.cjsindex 9b9600109..238afee8f 100644--- a/internal/gui/tests/privacy-model.test.cjs+++ b/internal/gui/tests/privacy-model.test.cjs@@ -94,6 +94,21 @@ const scoreWords = {de: { name: "Konfidenzschwelle", def: "Standard (0.8)", most: "Maskiert am meisten" },};+// where AstrLink Guard runs (yyhhyyyyyy on Discord: it said "on the CPU"+// while CoreML ran it): CoreML on the device its plan picked, the CPU+// with CoreML's error after a failed read, the CPU where there's no CoreML+const runWords = {+ en: { ne: "On, with CoreML on the Neural Engine", gpu: "On, with CoreML on the GPU", ml: "On, with CoreML", failed: "On, on the CPU: CoreML failed (prediction failed)", cpu: "On, on the CPU" },+ zh: { ne: "已开启,通过 CoreML 运行在神经网络引擎", gpu: "已开启,通过 CoreML 运行在 GPU", ml: "已开启,通过 CoreML 运行", failed: "已开启,运行在 CPU:CoreML 出错(prediction failed)", cpu: "已开启,运行在 CPU" },+ "zh-TW": { ne: "已開啟,透過 CoreML 執行於神經網路引擎", gpu: "已開啟,透過 CoreML 執行於 GPU", ml: "已開啟,透過 CoreML 執行", failed: "已開啟,執行於 CPU:CoreML 出錯(prediction failed)", cpu: "已開啟,執行於 CPU" },+ ja: { ne: "オン、CoreML により Neural Engine で実行中", gpu: "オン、CoreML により GPU で実行中", ml: "オン、CoreML で実行中", failed: "オン、CPU で実行中:CoreML が失敗しました(prediction failed)", cpu: "オン、CPU で実行中" },+ de: { ne: "An, mit CoreML auf der Neural Engine", gpu: "An, mit CoreML auf der GPU", ml: "An, mit CoreML", failed: "An, auf der CPU: CoreML ist ausgefallen (prediction failed)", cpu: "An, auf der CPU" },+};+const runs = {+ ne: { via: "CoreML", device: "Neural Engine" }, gpu: { via: "CoreML", device: "GPU" }, ml: { via: "CoreML" },+ failed: { device: "CPU", gpuError: "prediction failed" }, cpu: { device: "CPU" },+};+// what the row shows: its words and its New mark, whether they are cut,// that it is inside the list, and that it is the page's first settingconst look = (page) => page.evaluate(() => {@@ -154,6 +169,23 @@ for (const engine of (process.env.BROWSER ? [process.env.BROWSER] : ["chromium",}}+ for (const lang of Object.keys(runWords)) {+ for (const width of [1100, 640]) {+ test(`${engine} ${lang} ${width}px: AstrLink Guard says what runs it`, async (t) => {+ for (const [k, run] of Object.entries(runs)) {+ const model = { ...guard, installed: true, on: true, running: true, minScore: 0.8, defaultScore: 0.8, ...run };+ const { browser, page, errors } = await open(engine, lang, width, true, model, []);+ await page.locator("#privacyModelRow .segs").waitFor();+ const m = await look(page);+ await browser.close();+ assert.equal(m.sub, runWords[lang][k], k);+ assert(!m.cut && m.inside && !m.wide, `${k} shown in full and inside: ${JSON.stringify(m)}`);+ assert.deepEqual(errors, []);+ }+ });+ }+ }+test(`${engine}: a download is cancelled, and a downloaded model deleted after asking`, async (t) => {const posts = [], model = { supported: true, installed: false, size };const { browser, page, errors } = await open(engine, "en", 1100, true, model, posts);diff --git a/internal/pii/astrlink.go b/internal/pii/astrlink.goindex 0792420a0..c311c9037 100644--- a/internal/pii/astrlink.go+++ b/internal/pii/astrlink.go@@ -17,7 +17,7 @@ import (// AstrLink Guard (QuantumNous/astrlink-guard, Apache-2.0) is the second// local model: a 37 MB int8 token classifier for Chinese and English,// run in magpie's own process by internal/pii/guard, on the CPU of every-// system, with nothing native to download. Its files are fetched from+// system and through CoreML on a Mac, with nothing native to download. Its files are fetched from// Hugging Face at the revision their sha256 were pinned at (the// publisher's own listing), and are loaded on the first request that// needs them and let go after idleAfter with none.@@ -136,6 +136,11 @@ type spanFinder interface {Find(text string) []guard.Span}+// backender is a model that says what reads it (guard.Model.Backend).+type backender interface {+ Backend() (via, device string, err error)+}+// guardLoad reads the model; a var for tests.var guardLoad = func(dir string) (spanFinder, error) {m, err := guard.Load(dir)@@ -307,6 +312,12 @@ func (guardModel) Now() Status {gsrv.mu.Lock()if gsrv.m != nil {s.Running, s.Device, s.LoadMS = true, "CPU", gsrv.loadMS+ if b, ok := gsrv.m.(backender); ok {+ var err error+ if s.Via, s.Device, err = b.Backend(); err != nil {+ s.GPUError = err.Error()+ }+ }}s.Failed, s.Loading = gsrv.failed, gsrv.loadinggsrv.mu.Unlock()diff --git a/internal/pii/astrlink_test.go b/internal/pii/astrlink_test.goindex c9f89ec5f..c3c846ead 100644--- a/internal/pii/astrlink_test.go+++ b/internal/pii/astrlink_test.go@@ -2,6 +2,7 @@ package piiimport ("context"+ "errors""net/http/httptest""os""path/filepath"@@ -183,6 +184,44 @@ func TestGuardFindsUnderRedactsLabels(t *testing.T) {}}+// backendFinder is a fakeFinder that says what reads it, as guard.Model+// does.+type backendFinder struct {+ fakeFinder+ via, device string+ err error+}++func (f backendFinder) Backend() (string, string, error) { return f.via, f.device, f.err }++// Now names what reads AstrLink Guard: CoreML and the device its plan+// picked while CoreML reads, the CPU with CoreML's error when that failed,+// and the CPU where there is no CoreML (yyhhyyyyyy on Discord: Settings+// said "on the CPU" while CoreML ran it).+func TestGuardNowNamesItsBackend(t *testing.T) {+ for _, c := range []struct {+ via, device, err string+ }{{"CoreML", "Neural Engine", ""}, {"CoreML", "GPU", ""}, {"CoreML", "", ""}, {"", "CPU", "prediction failed"}, {"", "CPU", ""}} {+ t.Run(c.via+c.device+c.err, func(t *testing.T) {+ guardSandbox(t)+ fakeGuardFiles(t)+ var reads atomic.Int32+ f := backendFinder{fakeFinder: fakeFinder{map[string]string{}, &reads}, via: c.via, device: c.device}+ if c.err != "" {+ f.err = errors.New(c.err)+ }+ guardLoad = func(string) (spanFinder, error) { return f, nil }+ if _, err := Of(AstrLinkGuard).Find(context.Background(), []string{"hi"}); err != nil {+ t.Fatal(err)+ }+ st := Of(AstrLinkGuard).Now()+ if !st.Running || st.Via != c.via || st.Device != c.device || st.GPUError != c.err || st.GPU {+ t.Errorf("Now = %+v", st)+ }+ })+ }+}+// One whose files aren't on disk fails, rather than mask by the rules// alone; one that didn't load is held back at once until it is tried// again.diff --git a/internal/pii/guard/coreml.go b/internal/pii/guard/coreml.goindex 3a4047c69..914feee2e 100644--- a/internal/pii/guard/coreml.go+++ b/internal/pii/guard/coreml.go@@ -218,6 +218,9 @@ func half(f float32) uint16 {return sign | uint16(h)}+// coremlRead is coremlLogits; a var for tests.+var coremlRead = coremlLogits+// coremlLogits are nt.logits(ids), from m.func coremlLogits(m *coreml.Model, nt *net, ids []int) ([]float32, error) {n := len(ids)diff --git a/internal/pii/guard/coreml/coreml_darwin.go b/internal/pii/guard/coreml/coreml_darwin.goindex 9ebf02dc9..4e27df877 100644--- a/internal/pii/guard/coreml/coreml_darwin.go+++ b/internal/pii/guard/coreml/coreml_darwin.go@@ -55,6 +55,51 @@ static void *gmlOpen(const char *path, char **errp) {static void gmlClose(void *m) { CFRelease(m); }+// gmlWalk adds each operation's estimated cost in b, and its blocks', to+// cost by the device CoreML's plan prefers for it: the Neural Engine, the+// GPU, the CPU; ops counts the operations placed.+API_AVAILABLE(macos(14.4))+static void gmlWalk(MLComputePlan *plan, MLModelStructureProgramBlock *b, double *cost, long *ops) {+ for (MLModelStructureProgramOperation *op in b.operations) {+ MLComputePlanDeviceUsage *u = [plan computeDeviceUsageForMLProgramOperation:op];+ if (u != nil) {+ MLComputePlanCost *c = [plan estimatedCostOfMLProgramOperation:op];+ double w = c != nil ? c.weight : 0;+ id<MLComputeDeviceProtocol> d = u.preferredComputeDevice;+ int i = [(id)d isKindOfClass:[MLNeuralEngineComputeDevice class]] ? 0 : [(id)d isKindOfClass:[MLGPUComputeDevice class]] ? 1 : 2;+ cost[i] += w;+ ops[i]++;+ }+ for (MLModelStructureProgramBlock *sb in op.blocks) gmlWalk(plan, sb, cost, ops);+ }+}++// gmlPlan fills cost (and ops) by device, Neural Engine, GPU, CPU, as+// CoreML's compute plan for the compiled program at path places it.+static char *gmlPlan(const char *path, double *cost, long *ops) {+ @autoreleasepool {+ if (@available(macOS 14.4, *)) {+ MLModelConfiguration *cfg = [MLModelConfiguration new];+ cfg.computeUnits = MLComputeUnitsAll;+ dispatch_semaphore_t sem = dispatch_semaphore_create(0);+ __block MLComputePlan *plan = nil;+ __block NSError *perr = nil;+ [MLComputePlan loadContentsOfURL:[NSURL fileURLWithPath:@(path)] configuration:cfg completionHandler:^(MLComputePlan *p, NSError *e) {+ plan = p;+ perr = e;+ dispatch_semaphore_signal(sem);+ }];+ if (dispatch_semaphore_wait(sem, dispatch_time(DISPATCH_TIME_NOW, 60 * NSEC_PER_SEC)) != 0) return strdup("the compute plan took over a minute");+ if (plan == nil) return gmlErr(perr, "no compute plan");+ MLModelStructureProgram *prog = plan.modelStructure.program;+ if (prog == nil) return strdup("not an ML program");+ for (MLModelStructureProgramFunction *f in prog.functions.allValues) gmlWalk(plan, f.block, cost, ops);+ return NULL;+ }+ return strdup("the compute plan needs macOS 14.4");+ }+}+static MLMultiArray *gmlArray(const float *src, NSArray<NSNumber *> *shape, NSArray<NSNumber *> *strides, size_t n, NSError **err) {float *b = malloc(n * sizeof(float));memcpy(b, src, n * sizeof(float));@@ -124,6 +169,44 @@ func Compile(pkg, out string) error {type Model struct {mu sync.Mutexm unsafe.Pointer+ // Device is where CoreML's plan runs most of the program, by its+ // estimated cost: "Neural Engine", "GPU" or "CPU"; "" where the plan+ // can't be read (before macOS 14.4), and DeviceErr says why.+ Device string+ DeviceErr error+}++// Plan is the share of the compiled program at path that CoreML's compute+// plan places on each of Devices, by the operations' estimated cost (by+// their count where it estimates none).+func Plan(path string) ([3]float64, error) {+ cp := C.CString(path)+ defer C.free(unsafe.Pointer(cp))+ var cost [3]C.double+ var ops [3]C.long+ if ce := C.gmlPlan(cp, &cost[0], &ops[0]); ce != nil {+ defer C.free(unsafe.Pointer(ce))+ return [3]float64{}, errors.New(C.GoString(ce))+ }+ return shares(cost, ops), nil+}++func shares(cost [3]C.double, ops [3]C.long) [3]float64 {+ var out [3]float64+ var sum, n float64+ for i := range cost {+ sum += float64(cost[i])+ n += float64(ops[i])+ }+ for i := range out {+ switch {+ case sum > 0:+ out[i] = float64(cost[i]) / sum+ case n > 0:+ out[i] = float64(ops[i]) / n+ }+ }+ return out}// Open opens the compiled program at path.@@ -137,6 +220,11 @@ func Open(path string) (*Model, error) {return nil, errors.New(C.GoString(ce))}m := &Model{m: p}+ if sh, err := Plan(path); err != nil {+ m.DeviceErr = err+ } else if m.Device = Most(sh); m.Device == "" {+ m.DeviceErr = errors.New("the compute plan places no operation")+ }runtime.AddCleanup(m, func(p unsafe.Pointer) { C.gmlClose(p) }, p)return m, nil}diff --git a/internal/pii/guard/coreml/coreml_other.go b/internal/pii/guard/coreml/coreml_other.goindex f045df316..ddf000711 100644--- a/internal/pii/guard/coreml/coreml_other.go+++ b/internal/pii/guard/coreml/coreml_other.go@@ -11,7 +11,12 @@ var errNone = errors.New("no CoreML here")func Compile(pkg, out string) error { return errNone }-type Model struct{}+type Model struct {+ Device string+ DeviceErr error+}++func Plan(path string) ([3]float64, error) { return [3]float64{}, errNone }func Open(path string) (*Model, error) { return nil, errNone }diff --git a/internal/pii/guard/coreml/plan.go b/internal/pii/guard/coreml/plan.gonew file mode 100644index 000000000..bc1c40897--- /dev/null+++ b/internal/pii/guard/coreml/plan.go@@ -0,0 +1,18 @@+package coreml++// Devices are the devices a plan places operations on, in gmlPlan's order.+var Devices = [3]string{"Neural Engine", "GPU", "CPU"}++// Most is the device of shares with the largest, "" with none.+func Most(shares [3]float64) string {+ best := -1+ for i, s := range shares {+ if s > 0 && (best < 0 || s > shares[best]) {+ best = i+ }+ }+ if best < 0 {+ return ""+ }+ return Devices[best]+}diff --git a/internal/pii/guard/coreml/plan_test.go b/internal/pii/guard/coreml/plan_test.gonew file mode 100644index 000000000..5e9106903--- /dev/null+++ b/internal/pii/guard/coreml/plan_test.go@@ -0,0 +1,20 @@+package coreml++import "testing"++// Most is the device with the largest share, none where no share is.+func TestMost(t *testing.T) {+ for _, c := range []struct {+ sh [3]float64+ want string+ }{+ {[3]float64{.9, .05, .05}, "Neural Engine"},+ {[3]float64{.1, .7, .2}, "GPU"},+ {[3]float64{0, 0, 1}, "CPU"},+ {[3]float64{}, ""},+ } {+ if got := Most(c.sh); got != c.want {+ t.Errorf("Most(%v) = %q, want %q", c.sh, got, c.want)+ }+ }+}diff --git a/internal/pii/guard/coreml_test.go b/internal/pii/guard/coreml_test.goindex bfec47458..be8e8b205 100644--- a/internal/pii/guard/coreml_test.go+++ b/internal/pii/guard/coreml_test.go@@ -2,6 +2,7 @@ package guardimport ("bytes"+ "errors""math""os""path/filepath"@@ -135,3 +136,102 @@ func argmax(r []float32) int {}return best}++// Backend names what reads the model: CoreML on the device its plan puts+// most of the program on while CoreML reads, and the CPU with why once+// CoreML's last read failed or it couldn't be made (Settings said "on+// the CPU" while CoreML read every window: yyhhyyyyyy on Discord).+func TestBackend(t *testing.T) {+ made := errors.New("no CoreML program")+ via, d, err := (&Model{MLErr: made}).Backend()+ if via != "" || d != "CPU" || (coreml.Available && err != made) || (!coreml.Available && err != nil) {+ t.Fatalf("without CoreML: %q %q %v", via, d, err)+ }+ m := &Model{ml: &coreml.Model{Device: "Neural Engine"}}+ if via, d, err := m.Backend(); via != "CoreML" || d != "Neural Engine" || err != nil {+ t.Fatalf("with CoreML: %q %q %v", via, d, err)+ }+ e := "prediction failed"+ m.mlFail.Store(&e)+ if via, d, err := m.Backend(); via != "" || d != "CPU" || err == nil || err.Error() != e {+ t.Fatalf("after a failed read: %q %q %v", via, d, err)+ }+ m.mlFail.Store(nil)+ if via, _, _ := m.Backend(); via != "CoreML" {+ t.Fatalf("after a good read: %q", via)+ }+}++// A failed CoreML read is what Backend says until one succeeds: logits+// records it as bert.go reads the window instead.+func TestFailedCoreMLReadIsSaid(t *testing.T) {+ d := dir(t)+ if !coreml.Available {+ t.Skip("no CoreML here")+ }+ m, err := Load(d)+ if err != nil {+ t.Fatal(err)+ }+ if m.ml == nil {+ t.Fatalf("CoreML didn't open: %v", m.MLErr)+ }+ ids := make([]int, 40)+ for i := range ids {+ ids[i] = 5+ }+ ids[0], ids[len(ids)-1] = 0, 2+ m.logits(ids, 1)+ if via, _, err := m.Backend(); via != "CoreML" || err != nil {+ t.Fatalf("after a good read: %q %v", via, err)+ }+ t.Cleanup(func() { coremlRead = coremlLogits })+ coremlRead = func(*coreml.Model, *net, []int) ([]float32, error) { return nil, errors.New("prediction failed") }+ if lg := m.logits(ids, 1); len(lg) != len(ids)*len(labels) {+ t.Fatalf("bert.go read %d logits", len(lg))+ }+ if via, d, err := m.Backend(); via != "" || d != "CPU" || err == nil || err.Error() != "prediction failed" {+ t.Fatalf("after a failed read: %q %q %v", via, d, err)+ }+ coremlRead = coremlLogits+ m.logits(ids, 1)+ if via, _, err := m.Backend(); via != "CoreML" || err != nil {+ t.Fatalf("after CoreML read again: %q %v", via, err)+ }+}++// CoreML's compute plan names where it runs the program: on a Mac with+// macOS 14.4 or later, a device for every operation it places, shares+// adding to one, and the loaded model's Device the largest of them.+func TestCoreMLPlanNamesADevice(t *testing.T) {+ d := dir(t)+ if !coreml.Available {+ t.Skip("no CoreML here")+ }+ m, err := Load(d)+ if err != nil {+ t.Fatal(err)+ }+ if m.ml == nil {+ t.Fatalf("CoreML didn't open: %v", m.MLErr)+ }+ es, _ := filepath.Glob(filepath.Join(d, "coreml", "*.mlmodelc"))+ if len(es) != 1 {+ t.Fatalf("compiled programs: %v", es)+ }+ sh, err := coreml.Plan(es[0])+ if err != nil {+ t.Fatal(err)+ }+ var sum float64+ for _, s := range sh {+ sum += s+ }+ t.Logf("plan: Neural Engine %.3f, GPU %.3f, CPU %.3f; Device %q", sh[0], sh[1], sh[2], m.ml.Device)+ if math.Abs(sum-1) > 1e-6 || m.ml.Device != coreml.Most(sh) || m.ml.Device == "" || m.ml.DeviceErr != nil {+ t.Fatalf("shares %v (sum %v), Device %q, %v", sh, sum, m.ml.Device, m.ml.DeviceErr)+ }+ if via, dev, err := m.Backend(); via != "CoreML" || dev != m.ml.Device || err != nil {+ t.Fatalf("Backend: %q %q %v", via, dev, err)+ }+}diff --git a/internal/pii/guard/guard.go b/internal/pii/guard/guard.goindex 7c0cf89f3..a9d2c7f8f 100644--- a/internal/pii/guard/guard.go+++ b/internal/pii/guard/guard.go@@ -6,6 +6,7 @@package guardimport (+ "errors""fmt""math""path/filepath"@@ -34,6 +35,28 @@ type Model struct {ml *coreml.Model // nil where CoreML isn't, or couldn't be made// MLErr is why CoreML isn't used, on a Mac.MLErr error+ // mlFail is why CoreML's last read failed and bert.go read the window+ // instead; nil once one doesn't.+ mlFail atomic.Pointer[string]+}++// Backend is what reads the model now, as Settings › Privacy names it:+// via "CoreML" on the device its compute plan puts most of the program+// on ("Neural Engine", "GPU" or "CPU"; "" where the plan can't be read),+// or bert.go on the CPU (via ""), and then on a Mac why not CoreML: it+// couldn't be made, or its last read failed. Settings said "on the CPU"+// while CoreML read every window (yyhhyyyyyy on Discord, 10-27).+func (m *Model) Backend() (via, device string, err error) {+ if m.ml == nil {+ if !coreml.Available {+ return "", "CPU", nil+ }+ return "", "CPU", m.MLErr+ }+ if e := m.mlFail.Load(); e != nil {+ return "", "CPU", errors.New(*e)+ }+ return "CoreML", m.ml.Device, nil}// Span is what the model found: text[Start:End], a Label of its (email,@@ -84,9 +107,13 @@ func Load(dir string) (*Model, error) {// and the window is long enough to gain by it, else over procs goroutines.func (m *Model) logits(ids []int, procs int) []float32 {if m.ml != nil && len(ids) >= coremlMin {- if lg, err := coremlLogits(m.ml, m.net, ids); err == nil {+ lg, err := coremlRead(m.ml, m.net, ids)+ if err == nil {+ m.mlFail.Store(nil)return lg}+ e := err.Error()+ m.mlFail.Store(&e)}return m.net.logits(ids, procs)}diff --git a/internal/pii/pii.go b/internal/pii/pii.goindex 83435e033..484dee7ad 100644--- a/internal/pii/pii.go+++ b/internal/pii/pii.go@@ -132,8 +132,12 @@ type Status struct {Error string `json:"error,omitempty"`// pf-serve, while it runs: where the model is, and how long it took// to start- Running bool `json:"running,omitempty"`- Device string `json:"device,omitempty"`+ Running bool `json:"running,omitempty"`+ Device string `json:"device,omitempty"`+ // Via is the runtime AstrLink Guard runs on, "CoreML" on a Mac where+ // it does (Device then the one its plan puts most of the model on, or+ // ""); "" for magpie's own Go code on the CPU+ Via string `json:"via,omitempty"`GPU bool `json:"gpu,omitempty"`LoadMS int `json:"loadMs,omitempty"`GPUError string `json:"gpuError,omitempty"`