Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
pii: a hung GPU load moves to the CPU, a failure says what ended pf-serve, and a failing model holds requests back at once
yetonecebd0d4c3529
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.mdindex 28cf475e5..10a624b95 100644--- a/docs/subsystems/gateway-routing.md+++ b/docs/subsystems/gateway-routing.md@@ -97,7 +97,7 @@ recorded as a prediction failure (`noPrediction`).1. `handle(proto)` reads the body and the model (`requestBody`, `requestModel`), then calls `serveAgent`. Every route that reads an agent's body (each API, count_tokens, Gemini's, embeddings, images, videos, System One, the Codex backend path) reads it through `readBoundedRequestBody` in [`request_bounds.go`](../../internal/gateway/request_bounds.go). It decodes a `Content-Encoding` of `gzip` or `zstd` (`decodedBody`; Codex sends zstd to a custom provider's `/v1/responses` too, #1223) and drops the header, so the provider is sent plain JSON, with one exception: a ChatGPT account's turn (POST `CodexBase`/responses, from the accounts loop in `forwardOnce` or relayed by `codexUpstreamOn`) is compressed again with zstd and a window of at most 2 MiB, as Codex's own client sends it (`provider.CodexCompressed`; openai/codex `responses_request_compression` and `http-client/src/request.rs`). It is compressed once the body is final, and again on each try. `/responses/compact`, an API key's requests and every other provider stay plain. Sent plain, a 1.1 MB text-only turn was answered 503 (ylorn on X). The plain body is what magpie keeps and shows, and `codexDropped` judges a reset by the bytes that went on the wire. `codexDropped` also reads Envoy's 507 "exceeded request buffer limit while retrying upstream" (the backend failed and the front couldn't retry a body that large) as that reset, at any size, told as a 503 so magpie, Codex and pi ask again and nobody rests (#2034). The size limit (`defaultBodyLimit`, 512 MB) holds for the decoded bytes (413), and any other encoding is refused 415. It was 128 MB until a long Codex session that had read many images went past it decoded, its inline screenshots resent each turn, and got the 413 on all five of Codex's retries (#1422, `TestImageHeavyCodexTurnIsRead`). magpie's own 413 says it is magpie's gateway limit, with the size, and that the provider was not asked (`overLimit`); `writeError` adds to any 413 it isn't holding that this is a byte limit, not the model's context, and to `/compact` or start a new conversation (`TestOverLimitSaysWhoseAndWhatToDo`). Every shape but Gemini's types a 413 `request_too_large`, Anthropic's own type, which pi's overflow table knows: pi compacts and tries once more rather than stopping on an OpenAI-shaped `api_error` (@wilsonxuuuu on X, `TestPayloadTooLargeSaysRequestTooLargeToPi`). `keyLimited` reads a limited key's compressed body the same way, so its reservation is the decoded size. `serveAgent` runs the [middleware](gateway-middleware.md) chain, then `serve`.2. `serve` redacts secrets in the request (`redacted`; `redact.MaskJSON` leaves ids alone, and in any text a backend's id (a UUID, or 32 hex digits or more) goes as written: no rule, model label or value known from before masks a piece of one (a phone number at the end of `…-a13800138000`, a card number across a UUID's groups), and a whole one is masked only by the user's own words and rules or as a credential, by the password, API key and token rules or a secret rule whose field is named for one (`api_key=`, `refresh_token`), never one named for an id, a cursor, a request or a digest (`secret_id`, `next_page_token`, `Idempotency-Key`, `secret_hash`), a bare `key`, a serial number, or what the model takes a UUID for (`noIDs`, [`internal/redact/ids.go`](../../internal/redact/ids.go)); and a tool's name and its MCP server's (`namespace`, `server_label`, `server_name`, #3143); the request's tool definitions and output schemas, found by their place in each protocol (top-level `tools`, `tool_choice`, `response_format`, `text`, `generationConfig`, `stream_options`, the client's ids in `client_metadata` and the like, and the tools of a Responses `additional_tools` or `tool_search_output` item), go as written save their descriptions, so a word the user masks can't break a grammar or an enum (#1911); and what the vendor sealed with a signature, a thinking block, Gemini's thoughts or Responses' encrypted reasoning, goes back as the vendor wrote it: only the values masked elsewhere in the request, and secrets masked before, are put back as their placeholders; a value masked once, at least 8 characters long, is masked again wherever a request has it, as an exact match whatever words are around it, so what the vendor wrote about a placeholder doesn't go back with the value in it, `redact/known.go`; a text masked for a vendor before, such as a system prompt or a turn already sent, is masked as it was then, so a value a rule first finds later in a conversation doesn't change the prompt the vendor has cached and rewrite all of it to the cache, `sentKnownSpans`), resolves the model to a provider or a group, and refuses what can't be served. The reply has the values put back on its way to the agent (`redact.Writer`, a stream event by event, a whole body at its end): a placeholder the model wrote as it was given, or as models rewrite one (spaces inside the braces, a brace or both dropped, another case), is the value again when what is left names one magpie made, and one split across events is held until it is whole (`redact.Restore`, `partialTail`, [`internal/redact/redact.go`](../../internal/redact/redact.go)). A request whose provider and fallbacks, or whose group's models, all skip redaction (`unredactedRoute`, `provider.SkipsRedaction`: set *Send requests unmasked*, `Unredacted`, with every address on this machine or the local network) goes to them as the agent wrote it; the log and recent calls keep the masked body. count_tokens and embeddings follow the same rule. What a routing group's Jev is told of the turn (the message, the user's message before it and the reply to it) is masked all the same unless Jev's own provider skips redaction too (`maskedState` in `askJev`, [`decide.go`](../../internal/gateway/decide.go)): a remote Jev is none of the group's members. The masking is the built-in plugin `redact`'s (`gwmw.Masker`, [Plugin runtime](plugin-runtime.md)); when the settings ask for masking and no masker serves, or it fails, the request is a 503 and nothing is sent (`redacted`, `gwmw.Masking`).- With *Mask personal data* on, each kind in `redact.Categories` ([`internal/redact/more.go`](../../internal/redact/more.go)) is masked unless the user turned it off (`redactKinds` in settings, `Options.Kinds`). On by default: US SSNs, passport numbers, IBANs that pass mod 97, birthdays next to a word that says so (a date over a year ago: one in the last year or ahead is the day an agent runs on, `dateOK`), MAC addresses, and serial numbers next to S/N or the like. Off by default: the user name in a home folder, user@host in a prompt or ssh command, public IPv4 and IPv6 addresses (never private, loopback, link-local, CGNAT or documentation ranges) and cloud bucket names, since agents send them in nearly every request and the model must copy the placeholder exactly inside a path or URL. Context rules run only near their words (`rule.matches`). What reads as an address but ends in a file's extension that no domain has (`…@2x.jpg`, `[email protected]`; `fileExt` in `realEmail`), or a scale before an extension that is a domain too (`[email protected]`, `[email protected]`; `scaleExt`), is a file's name, for the rule and the local model alike (#3029). `[email protected]` is still an address. The Settings rows come from `REDACT_KINDS` in app.js, which `TestRedactKindsAreRedacts` holds to the same ids and defaults.- - With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken, nor a snake_case word as a name (Codex's `sequential_cutoff`, #3144), nor as a secret what names a thing: a UUID, a digest with its algorithm's name, a file's path (`namesThing`; the model takes all three for secrets). The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with it on a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left. The model is OpenAI's Privacy Filter as a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags; on a GPU it holds the model once, in the device's memory, #3590): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, from hf-mirror first in China, `source.chinaFirst`) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU is started on the CPU for the rest of the run), joins short texts into 12 KB chunks and maps the spans back, keeps the last 20000 answers by hash, on disk too (`found-<model>.bin` beside the model: hashes, offsets and labels, never text; written through temp-and-rename, a bad read taken as empty), so a restart doesn't ask the model about every history again (#3682), and ends it after 10 minutes with no request. Its requests take turns on its one pipe (`turn`), and a request reads the cache again when its turn comes; a request waits for its turn only while its context lives (`pii.FindContext`, through `gwmw.ContextTeller`), so one whose client went away, or that the gateway answered 504 after 2 minutes, leaves the queue; a turn begun is finished and kept. The status (`pii.Now`), `pii.Stop` and the idle timer never wait for a load or a request: `srv.mu` is held only to read or set the state, and `Stop` ends a load or a request under way at once. A request held before routing while it waits for the model is answered 504 saying so, and how to send without it (`unroutedMsg`); one of an older tag (`oldTags`, magpie updated since the download) runs until the pinned tag's is fetched (`upgrade`, on its start), which is then started on the next request and the old deleted ([`internal/pii`](../../internal/pii)). A model that is on and can't answer (not on disk, didn't start, hung) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).+ - With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken, nor a snake_case word as a name (Codex's `sequential_cutoff`, #3144), nor as a secret what names a thing: a UUID, a digest with its algorithm's name, a file's path (`namesThing`; the model takes all three for secrets). The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with it on a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left. The model is OpenAI's Privacy Filter as a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags; on a GPU it holds the model once, in the device's memory, #3590): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, from hf-mirror first in China, `source.chinaFirst`) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU, or doesn't load there in 40s (`gpuLoadTimeout`, well inside the gateway's 2 minutes), is started on the CPU for the rest of the run, and Settings says why), joins short texts into 12 KB chunks and maps the spans back, keeps the last 20000 answers by hash, on disk too (`found-<model>.bin` beside the model: hashes, offsets and labels, never text; written through temp-and-rename, a bad read taken as empty), so a restart doesn't ask the model about every history again (#3682), and ends it after 10 minutes with no request. Its requests take turns on its one pipe (`turn`), and a request reads the cache again when its turn comes; a request waits for its turn only while its context lives (`pii.FindContext`, through `gwmw.ContextTeller`), so one whose client went away, or that the gateway answered 504 after 2 minutes, leaves the queue; a turn begun is finished and kept. The status (`pii.Now`), `pii.Stop` and the idle timer never wait for a load or a request: `srv.mu` is held only to read or set the state, and `Stop` ends a load or a request under way at once. A request held before routing while it waits for the model is answered 504 saying so, and how to send without it (`unroutedMsg`); one of an older tag (`oldTags`, magpie updated since the download) runs until the pinned tag's is fetched (`upgrade`, on its start), which is then started on the next request and the old deleted ([`internal/pii`](../../internal/pii)). A failure names what ended `pf-serve`: magpie's own end of a request it didn't answer in time, else its exit status or signal and its last stderr lines past ggml's startup chatter (`died`, `lastSaid`). After one, requests are held back at once with that failure, not after a load each, until the model is tried again 30s later, doubling to 10 minutes while it keeps failing (`heldBack`, `retryAfter`; lazyeo on Discord, an M2 whose every request waited out the 504). A model that is on and can't answer (not on disk, didn't start, hung, held back) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).- With *Mask secrets* on, the user's own masking rules (`redactRules`, `redact.Rule`: a placeholder name and a prefix or an RE2 pattern, [`internal/redact/rules.go`](../../internal/redact/rules.go)) run before magpie's. Settings → Privacy adds them, removes them, and edits one in place (`POST /api/settings/redact-rule` with the rule's index and its current value, `editRedactRule` in [`internal/gui/api.go`](../../internal/gui/api.go)). Every rule is checked by `redact.CheckRules` when settings are saved; an edit that doesn't compile, matches the empty string, repeats another rule or names a rule no longer at that index is refused with the rules left as they were. The masker reads the rules from settings on each call (`options` in [`internal/masking`](../../internal/masking/masking.go)), so a change applies to the next request.- With *Keep a key's form* on (`settings.RedactShape`, `Options.Shape`, under *Mask secrets*; Charles Cao on X), a secret of 16 characters or more, all letters, digits, `-`, `_` or `.`, goes as a made-up key of its form instead of a placeholder: its prefix kept (`sk-proj-`, `ghp_`, `AKIA`, up to four letters and their separator; `keyHead`), the same length and separators, and each other character a random one of its kind (a hex key's any hex digit of its case), drawn from an HMAC of the value under the placeholder key, so one key goes as the same made-up one every turn and a vendor's cache still matches (`shapeOf`, `shaped` in [`internal/redact/shape.go`](../../internal/redact/shape.go)). Personal data, the user's words and anything shorter or with other characters go as placeholders still. A made-up key is kept in the masker's memory with the placeholders, so it is recorded, re-masked in what a vendor sealed, and put back wherever the reply has it, before the placeholders (`restoreShapes`); a stream holds the end of an event that may be the start of one (`shapeTail`, after the last whole one), and a made-up key the vendor sends back unrestored isn't masked again (`outsideShapes`). `TestShapeKeepsForm`, `TestShapeJSON`, `TestShapeStream`. Insights › Masking and Settings' *Try it* mark a made-up key where they mark a placeholder, matching the text against the request's own values only, so a key of the same form the user wrote isn't marked (`maskMatches` in `plugins/redact/page.js`; #3599, masking-records and mask-actions GUI tests).- What is done with a value found is the user's per kind (#1977, `redactActions` in settings, `Options.Actions`, set by `POST /api/settings/redact-actions`; a kind left out is masked). `redact.Warn` sends it as written: the masker tells it, with a placeholder no one remembers, and it isn't counted as masked. `redact.Block` masks it as any other, and the gateway holds the request back with a 400 that names the kind and the value's hint, never the value (`blocked`, `maskBlocked` in [`redact.go`](../../internal/gateway/redact.go)); nothing is sent and the record is kept as `Blocked` (`noteBlocked`). A value on the *Never mask* list (`redactAllow`, `Options.Allow`, `POST /api/settings/redact-allow`) goes as written when it is the whole value found, in any case, and isn't told; a longer value it is part of is masked. Settings → Privacy's *Try it* posts a text to `POST /api/masking/preview` ([`masking.go`](../../internal/gui/masking.go)), which masks it as a request would be (`redact.Preview`) and remembers nothing: its placeholders stand for nothing afterwards, its values aren't masked later as seen before, and it returns only the masked text and each hit's kind, finder, action, placeholder and hint. The page keeps the text in no storage.diff --git a/internal/pii/cache.go b/internal/pii/cache.goindex fb96284e3..98e1e8142 100644--- a/internal/pii/cache.go+++ b/internal/pii/cache.go@@ -37,6 +37,9 @@ var disk struct {loaded string // the file read into cache, oncetimer *time.Timersaving sync.Mutex+ // saves counts a timer's write from when it is set until it is+ // done, so a test's cleanup can wait for one under way.+ saves sync.WaitGroup}// loadCache reads the file into cache, the first time Find is asked.@@ -70,7 +73,9 @@ func saveSoon() {disk.mu.Lock()defer disk.mu.Unlock()if disk.timer == nil {+ disk.saves.Add(1)disk.timer = time.AfterFunc(saveAfter, func() {+ defer disk.saves.Done()disk.mu.Lock()disk.timer = nildisk.mu.Unlock()diff --git a/internal/pii/find.go b/internal/pii/find.goindex b63ed112e..b618b9136 100644--- a/internal/pii/find.go+++ b/internal/pii/find.go@@ -35,6 +35,9 @@ func FindContext(ctx context.Context, texts []string) ([][]redact.Found, error)if ctx.Err() != nil {return nil, waitErr(ctx)}+ if err := heldBack(); err != nil {+ return nil, err+ }waiting.Add(1)select {case turn <- struct{}{}:@@ -54,6 +57,12 @@ func FindContext(ctx context.Context, texts []string) ([][]redact.Found, error)// a history in common with it, and asked alreadyout, miss := cached(texts)if len(miss) > 0 {+ // the one ahead may have failed: those queued behind it+ // don't each try again, minutes apiece+ if err := heldBack(); err != nil {+ done <- result{nil, err}+ return+ }chunks, parts := join(texts, miss)got, err := classify(chunks)if err != nil {diff --git a/internal/pii/main_test.go b/internal/pii/main_test.goindex de90733e5..919844dbf 100644--- a/internal/pii/main_test.go+++ b/internal/pii/main_test.go@@ -41,6 +41,10 @@ const (// slow answers each request after 300ms// noload says it couldn't load the model// novulkan as pf-serve-vulkan, says Vulkan is missing; else ok+// gpuloadhang on the GPU, never says the model is loaded; else ok+// asserts fails a ggml assert at its first request+//+// Like pf-serve, it first writes ggml's startup lines to stderr.func fakeServe(mode string) {name := strings.TrimSuffix(filepath.Base(os.Args[0]), ".exe")gpu := os.Getenv("PF_DEVICE") != "cpu"@@ -58,13 +62,18 @@ func fakeServe(mode string) {}}logf("start %s %s", name, device)+ // as ggml_metal writes them on an M2, GPU or CPU alike+ fmt.Fprintln(os.Stderr, "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices")+ fmt.Fprintln(os.Stderr, "ggml_metal_library_init: using embedded metal library")+ fmt.Fprintln(os.Stderr, "ggml_metal_device_init: GPU name: MTL0 (Apple M2)")+ fmt.Fprintln(os.Stderr, "ggml_metal_device_init: recommendedMaxWorkingSetSize = 12713.12 MB")logf("path %s", filepath.Base(filepath.Dir(os.Args[0])))logf("residency %s", os.Getenv("GGML_METAL_NO_RESIDENCY"))if mode == "noload" || mode == "novulkan" && name == "pf-serve-vulkan" {json.NewEncoder(os.Stdout).Encode(map[string]string{"error": "can't load " + mode})os.Exit(1)}- if mode == "loadhang" {+ if mode == "loadhang" || mode == "gpuloadhang" && gpu {time.Sleep(time.Hour)}json.NewEncoder(os.Stdout).Encode(map[string]any{"device": device, "gpu": gpu, "ms": 7})@@ -95,6 +104,10 @@ func fakeServe(mode string) {os.Exit(3)case mode == "hang":time.Sleep(time.Hour)+ case mode == "asserts":+ fmt.Fprintln(os.Stderr, "ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_add_id'")+ fmt.Fprintln(os.Stderr, "/ggml/src/ggml-metal/ggml-metal-ops.cpp:1234: GGML_ASSERT(ne00 % 4 == 0) failed")+ os.Exit(3)case mode == "slow":time.Sleep(300 * time.Millisecond)}@@ -138,14 +151,16 @@ func fakeInstall(t *testing.T, mode string, vulkan bool) string {<-turnupgrades.Wait()disk.mu.Lock()- if disk.timer != nil {- disk.timer.Stop()- disk.timer = nil+ if disk.timer != nil && disk.timer.Stop() {+ disk.saves.Done()}+ disk.timer = nildisk.loaded = ""disk.mu.Unlock()+ disk.saves.Wait() // a write under way reads dirsrv.mu.Lock()srv.cpu, srv.gpuLost, srv.failed = false, "", ""+ srv.lastErr, srv.fails, srv.retryAt = nil, 0, time.Time{}srv.mu.Unlock()cache.mu.Lock()cache.ll.Init()diff --git a/internal/pii/pii.go b/internal/pii/pii.goindex 6171a634e..77a7ac6a1 100644--- a/internal/pii/pii.go+++ b/internal/pii/pii.go@@ -13,8 +13,10 @@//// A process of its own, not linked in: 1.7 GB the gateway would otherwise// hold for good, and a GPU driver's crash that would end the gateway with-// it. One that dies is started again; one that dies on the GPU is started-// on the CPU for the rest of magpie's run.+// it. One that dies is started again; one that dies on the GPU, or doesn't+// load there in gpuLoadTimeout, is started on the CPU for the rest of+// magpie's run. One that keeps failing holds requests back at once, not+// after a load each, until it is tried again (retryAfter).package piiimport (diff --git a/internal/pii/serve.go b/internal/pii/serve.goindex 549eeb549..33a0c5b47 100644--- a/internal/pii/serve.go+++ b/internal/pii/serve.go@@ -10,6 +10,8 @@ import ("io""os""os/exec"+ "regexp"+ "runtime""strings""sync""sync/atomic"@@ -36,6 +38,8 @@ type helper struct {stderr *tailhello helloexited chan struct{}+ // slow says magpie ended it, given too long to answer+ slow atomic.Pointer[string]}// srv is the pf-serve magpie runs, one at a time. Its requests take@@ -53,8 +57,15 @@ var srv struct {// the CPU; gpuLost says whycpu boolgpuLost string- // failed is why the last request got no answer, until one does- failed string+ // failed is why the last request got no answer, until one does;+ // lastErr is that error, fails how many requests in a row got none,+ // and retryAt when the model is tried again: until then a request+ // that needs it is held back at once (heldBack) rather than wait+ // minutes for a load that failed the time before+ failed string+ lastErr error+ fails int+ retryAt time.Time// loading is set while pf-serve loads the modelloading bool// stop is closed by Stop: a load under way gives up, and the request@@ -76,10 +87,36 @@ func stopCh() chan struct{} {return srv.stop}-// loadTimeout is how long pf-serve has to load the model: 1.6 GB read-// from a slow disk, and the GPU's kernels built.+// loadTimeout is how long pf-serve has to load the model on the CPU:+// 1.6 GB read from a slow disk, and the warm-up's texts read on a slow+// CPU (20s on an M4 Pro's).var loadTimeout = 3 * time.Minute+// gpuLoadTimeout is how long it has on a GPU, where it takes seconds+// (0.9-10s on an M4 Pro): one that takes longer is taken for hung, and+// the CPU is tried, for the rest of magpie's run. It is well under the 2+// minutes the gateway holds a request for (unroutedMost), so a GPU's+// hung load and the CPU's after it fit in one request's wait: with 3+// minutes for either, every request in a hung load was answered 504.+var gpuLoadTimeout = 40 * time.Second++// giveUp is when a load is given up on; a test ends one when its state+// says so rather than after a duration.+var giveUp = time.After++// retryAfter is how long after the model failed a request that needs it+// is held back at once, its n-th failure in a row: 30s, doubled each+// time, 10 minutes at most. A model that fails on each try (a GPU's+// driver that hangs, a CPU too slow for the requests) held every+// request for its 2 minutes before, and each one tried it again.+func retryAfter(n int) time.Duration {+ d := 30 * time.Second+ for i := 1; i < n && d < 10*time.Minute; i++ {+ d *= 2+ }+ return min(d, 10*time.Minute)+}+// classifyTimeout is how long one request may take before pf-serve is// taken for hung and ended: a second for each 4 KB on the slowest CPU,// and 30 more.@@ -87,17 +124,28 @@ var classifyTimeout = func(bytes int) time.Duration {return 30*time.Second + time.Duration(bytes/4096)*time.Second}-// candidates are the programs to start, in turn: pf-serve-vulkan, where-// there is one, needs the Vulkan loader a GPU's driver installs and-// can't start without it; pf-serve runs anywhere.-func candidates(cpu bool) []string {- var out []string+// candidate is a program to start, and whether on the CPU.+type candidate struct {+ path string+ cpu bool+}++// candidates are the programs to start, in turn: on a GPU,+// pf-serve-vulkan where there is one (it needs the Vulkan loader a GPU's+// driver installs, and can't start without it), and macOS's pf-serve,+// which runs on Metal; then pf-serve on the CPU, which runs anywhere.+// macOS's was tried on Metal alone: one whose load hung there, or ended,+// was tried again on Metal by every request (lazyeo, an M2).+func candidates(cpu bool) []candidate {+ var out []candidateif !cpu {if _, err := os.Stat(exe("pf-serve-vulkan")); err == nil {- out = append(out, exe("pf-serve-vulkan"))+ out = append(out, candidate{exe("pf-serve-vulkan"), false})+ } else if runtime.GOOS == "darwin" {+ out = append(out, candidate{exe("pf-serve"), false})}}- return append(out, exe("pf-serve"))+ return append(out, candidate{exe("pf-serve"), true})}// start runs pf-serve, the first of candidates that loads the model.@@ -112,14 +160,25 @@ func start(cpu bool, stop <-chan struct{}) (*helper, error) {upgrade()}()var errs []error- for _, path := range candidates(cpu) {- h, err := startOne(path, cpu, stop)+ gpuLost := ""+ for _, c := range candidates(cpu) {+ h, err := startOne(c.path, c.cpu, stop)if err == nil {+ if gpuLost != "" {+ // the GPU's failed and the CPU's loaded: the CPU for+ // the rest of the run, as for a GPU that dies answering+ srv.mu.Lock()+ srv.cpu, srv.gpuLost = true, gpuLost+ srv.mu.Unlock()+ }return h, nil}if errors.Is(err, errStopped) {return nil, err}+ if !c.cpu {+ gpuLost = err.Error()+ }errs = append(errs, err)}return nil, fmt.Errorf("the local masking model didn't start: %w", errors.Join(errs...))@@ -143,6 +202,10 @@ func startOne(path string, cpu bool, stop <-chan struct{}) (*helper, error) {if cpu {cmd.Env = append(cmd.Env, "PF_DEVICE=cpu")}+ wait := loadTimeout+ if !cpu {+ wait = gpuLoadTimeout+ }in, err := cmd.StdinPipe()if err != nil {return nil, err@@ -168,9 +231,9 @@ func startOne(path string, cpu bool, stop <-chan struct{}) (*helper, error) {var s stringselect {case s = <-line:- case <-time.After(loadTimeout):+ case <-giveUp(wait):h.kill()- return nil, fmt.Errorf("%s: the model didn't load in %s", baseName(path), loadTimeout)+ return nil, fmt.Errorf("%s: the model didn't load in %s%s", baseName(path), wait, onWhat(cpu))case <-stop:h.kill()return nil, errStopped@@ -191,20 +254,62 @@ func baseName(path string) string {return path[strings.LastIndexByte(path, '/')+1:]}-// died says why pf-serve ended: what it wrote to stderr last, or how it-// exited.+func onWhat(cpu bool) string {+ if cpu {+ return " on the CPU"+ }+ return " on the GPU"+}++// died says why pf-serve ended: that magpie ended it, given too long to+// answer, or how it exited; and what it said last that isn't ggml's+// startup lines. Those were all a user saw of a pf-serve magpie had+// ended for being slow (lazyeo: "pf-serve ended: ggml_metal_device_init:+// tensor API disabled for pre-M5 …", which every Mac before the M5+// prints and means nothing).func (h *helper) died() string {+ exited := trueselect {case <-h.exited:case <-time.After(2 * time.Second):+ exited = false}- if t := strings.TrimSpace(h.stderr.String()); t != "" {- return t+ var why []string+ if s := h.slow.Load(); s != nil {+ why = append(why, *s)+ } else if exited && h.cmd.ProcessState != nil {+ why = append(why, "ended ("+h.cmd.ProcessState.String()+")")+ } else {+ why = append(why, "ended")}- if st := h.cmd.ProcessState; st != nil {- return st.String()+ if t := lastSaid(h.stderr.String(), 3); t != "" {+ why = append(why, t)}- return "it ended"+ return strings.Join(why, ": ")+}++// startup are the lines ggml writes as it starts, every time, on a GPU+// and on the CPU alike: what the device is, the kernels it built.+var startup = regexp.MustCompile(`^(ggml_metal_(device_init|library_init|init|free|library_compile_pipeline)|ggml_vulkan|load_backend|register_backend|register_device):`)++// trouble is a line that says something went wrong, startup's or not.+var trouble = regexp.MustCompile(`(?i)error|fail|assert|abort|unable|cannot|can't|could not|out of memory|exception|lost`)++// lastSaid is the last n lines of stderr that say more than ggml's+// startup, joined.+func lastSaid(stderr string, n int) string {+ var out []string+ for _, l := range strings.Split(stderr, "\n") {+ l = strings.TrimSpace(l)+ if l == "" || startup.MatchString(l) && !trouble.MatchString(l) {+ continue+ }+ out = append(out, l)+ }+ if len(out) > n {+ out = out[len(out)-n:]+ }+ return strings.Join(out, " | ")}func (h *helper) kill() {@@ -241,6 +346,8 @@ func Stop() {}srv.mu.Lock()h := detachLocked()+ // turned on again, it is tried at once+ srv.lastErr, srv.fails, srv.retryAt = nil, 0, time.Time{}if srv.stop != nil {close(srv.stop)srv.stop = nil@@ -295,14 +402,34 @@ func end(h *helper) {func classify(texts []string) ([][]redact.Found, error) {out, err := classifyTurn(texts)srv.mu.Lock()- srv.failed = ""- if err != nil {+ srv.failed, srv.lastErr = "", nil+ switch {+ case err == nil:+ srv.fails, srv.retryAt = 0, time.Time{}+ case errors.Is(err, errStopped):+ // turned off, not failedsrv.failed = err.Error()+ default:+ srv.failed, srv.lastErr = err.Error(), err+ srv.fails+++ srv.retryAt = time.Now().Add(retryAfter(srv.fails))}srv.mu.Unlock()return out, err}+// heldBack is the error of a request that needs the model while it isn't+// tried again yet, after it failed; nil when it may be tried.+func heldBack() error {+ srv.mu.Lock()+ defer srv.mu.Unlock()+ if srv.lastErr == nil || !time.Now().Before(srv.retryAt) {+ return nil+ }+ return fmt.Errorf("%w; it is tried again in %s, and until then a request it hasn't read is held back at once. To send requests masked by the rules alone, turn the model off in Settings → Privacy",+ srv.lastErr, time.Until(srv.retryAt).Round(time.Second))+}+func classifyTurn(texts []string) ([][]redact.Found, error) {var last errorfor try := 0; try < 2; try++ {@@ -409,11 +536,13 @@ func (h *helper) classify(texts []string) ([][]redact.Found, error) {case <-done:case <-time.After(timeout):// a hung pf-serve: killed, its pipe ends the read below+ s := fmt.Sprintf("didn't answer in %s (%d KB%s), so magpie ended it", timeout, (size+1023)/1024, onWhat(!h.hello.GPU))+ h.slow.Store(&s)h.cmd.Process.Kill()}}()if _, err := h.in.Write(b.Bytes()); err != nil {- return nil, fmt.Errorf("%v (%s)", err, h.died())+ return nil, fmt.Errorf("pf-serve %s (%v)", h.died(), err)}r := &reader{r: h.out}status := r.u32()@@ -482,7 +611,7 @@ func (r *reader) bytes(n int) []byte {func (r *reader) fail(h *helper) error {if errors.Is(r.err, io.EOF) || errors.Is(r.err, io.ErrUnexpectedEOF) {- return fmt.Errorf("pf-serve ended: %s", h.died())+ return fmt.Errorf("pf-serve %s", h.died())}return r.err}diff --git a/internal/pii/serve_test.go b/internal/pii/serve_test.goindex 99f76f2d7..6f885361d 100644--- a/internal/pii/serve_test.go+++ b/internal/pii/serve_test.go@@ -1,8 +1,12 @@package piiimport (+ "context""errors"+ "fmt"+ "os""reflect"+ "runtime""strings""testing""time"@@ -109,11 +113,184 @@ func TestVulkanThatWontStartFallsBack(t *testing.T) {if _, err := Find([]string{"hi Alice"}); err != nil {t.Fatal(err)}- if starts := fakeLines(t, log, "start"); !reflect.DeepEqual(starts, []string{"start pf-serve-vulkan MTL0", "start pf-serve MTL0"}) {+ if starts := fakeLines(t, log, "start"); !reflect.DeepEqual(starts, []string{"start pf-serve-vulkan MTL0", "start pf-serve CPU"}) {t.Fatalf("starts = %q", starts)}}+// A GPU whose load hangs (lazyeo's M2: pf-serve kept loading for over a+// minute while every request was answered 504 after 2) is given+// gpuLoadTimeout, then the CPU is tried, and kept for the rest of the+// run. On macOS, whose pf-serve is Metal's, too: there it was the only+// one tried.+func TestHungGPULoadMovesToCPU(t *testing.T) {+ for _, vulkan := range []bool{false, true} {+ if !vulkan && runtime.GOOS != "darwin" {+ continue // pf-serve is the CPU's alone there+ }+ t.Run(fmt.Sprint("vulkan=", vulkan), func(t *testing.T) {+ log := fakeInstall(t, "gpuloadhang", vulkan)+ gpu := "start pf-serve MTL0"+ if vulkan {+ gpu = "start pf-serve-vulkan MTL0"+ }+ // The GPU's load is given up once the stand-in has started+ // on it, not after a duration a busy machine can outlast.+ was, wasGiveUp := gpuLoadTimeout, giveUp+ gpuLoadTimeout = 3 * time.Second+ done := make(chan struct{})+ giveUp = func(d time.Duration) <-chan time.Time {+ if d != gpuLoadTimeout {+ return wasGiveUp(d)+ }+ c := make(chan time.Time, 1)+ go func() {+ for {+ b, _ := os.ReadFile(log)+ if strings.Contains(string(b), gpu+"\n") {+ c <- time.Now()+ return+ }+ select {+ case <-done:+ return+ case <-time.After(10 * time.Millisecond):+ }+ }+ }()+ return c+ }+ t.Cleanup(func() { close(done); gpuLoadTimeout, giveUp = was, wasGiveUp })+ got := within10s(t, "a request in a hung GPU load", func() found {+ fs, err := Find([]string{"hi Alice"})+ return found{fs, err}+ })+ if got.err != nil || !reflect.DeepEqual(got.fs, [][]redact.Found{{person(3, 8)}}) {+ t.Fatalf("Find = %v, %v", got.fs, got.err)+ }+ if starts := fakeLines(t, log, "start"); !reflect.DeepEqual(starts, []string{gpu, "start pf-serve CPU"}) {+ t.Fatalf("starts = %q, want the GPU, then the CPU", starts)+ }+ if st := Now(); st.GPU || !strings.Contains(st.GPUError, "didn't load in 3s on the GPU") {+ t.Fatalf("Now = %+v, want on the CPU saying the GPU didn't load", st)+ }+ Stop()+ if _, err := Find([]string{"Alice again"}); err != nil {+ t.Fatal(err)+ }+ if starts := fakeLines(t, log, "start"); len(starts) != 3 || starts[2] != "start pf-serve CPU" {+ t.Fatalf("starts = %q, want the CPU straight away after", starts)+ }+ })+ }+}++// The GPU's load budget leaves room in a request's 2 minutes (the+// gateway's unroutedMost) for the CPU's load after it.+func TestGPULoadBudgetFitsARequest(t *testing.T) {+ if gpuLoadTimeout > time.Minute {+ t.Fatalf("gpuLoadTimeout = %s: a hung GPU load and the CPU's after it don't fit in a request's 2m", gpuLoadTimeout)+ }+}++// pf-serve that magpie ended for taking too long says so, not ggml's+// startup lines, which were all lazyeo's Settings showed ("pf-serve+// ended: ggml_metal_device_init: tensor API disabled for pre-M5 …").+func TestEndedForSlownessSaysSo(t *testing.T) {+ fakeInstall(t, "hang", false)+ was := classifyTimeout+ classifyTimeout = func(int) time.Duration { return 300 * time.Millisecond }+ t.Cleanup(func() { classifyTimeout = was })+ _, err := Find([]string{"hi Alice"})+ if err == nil || !strings.Contains(err.Error(), "didn't answer in 300ms (1 KB") || !strings.Contains(err.Error(), "magpie ended it") {+ t.Fatalf("err = %v, want it said magpie ended it for not answering", err)+ }+ if strings.Contains(err.Error(), "ggml_metal") || strings.Contains(Now().Failed, "ggml_metal") {+ t.Fatalf("err = %v, want no ggml startup lines", err)+ }+}++// One that dies says how it exited and its last words, past the startup.+func TestDeathSaysExitAndLastWords(t *testing.T) {+ fakeInstall(t, "asserts", false)+ _, err := Find([]string{"hi Alice"})+ if err == nil || !strings.Contains(err.Error(), "exit status 3") || !strings.Contains(err.Error(), "GGML_ASSERT(ne00 % 4 == 0) failed") {+ t.Fatalf("err = %v, want the exit status and the assert", err)+ }+ if strings.Contains(err.Error(), "tensor API") || strings.Contains(err.Error(), "compiling pipeline") {+ t.Fatalf("err = %v, want no ggml startup lines", err)+ }+}++func TestLastSaid(t *testing.T) {+ for _, x := range []struct{ in, want string }{+ {"ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: loaded in 0.323 sec\n", ""},+ {"ggml_metal_init: allocating\nggml_metal_device_init: failed to allocate buffer\n", "ggml_metal_device_init: failed to allocate buffer"},+ {"ggml_vulkan: Found 1 Vulkan devices:\nggml_metal: device lost\n", "ggml_metal: device lost"},+ {"a\nb\nc\nd\n", "b | c | d"},+ } {+ if got := lastSaid(x.in, 3); got != x.want {+ t.Errorf("lastSaid(%q) = %q, want %q", x.in, got, x.want)+ }+ }+}++func TestRetryAfter(t *testing.T) {+ for n, want := range map[int]time.Duration{1: 30 * time.Second, 2: time.Minute, 3: 2 * time.Minute, 5: 8 * time.Minute, 6: 10 * time.Minute, 40: 10 * time.Minute} {+ if got := retryAfter(n); got != want {+ t.Errorf("retryAfter(%d) = %s, want %s", n, got, want)+ }+ }+}++// A model that failed holds the requests that need it back at once until+// it is tried again, the ones queued behind the failing one too: each+// tried it again before, a load and a timeout apiece, and every request+// waited out its 2 minutes for a 504. They are still held back, never+// sent masked by the rules alone. Stop (the model turned off and on)+// tries it at once.+func TestFailedModelHoldsRequestsBackAtOnce(t *testing.T) {+ log := fakeInstall(t, "hang", false)+ was := classifyTimeout+ classifyTimeout = func(int) time.Duration { return 300 * time.Millisecond }+ t.Cleanup(func() { classifyTimeout = was })+ first := findAsync(context.Background(), "hi Alice")+ waitFor(t, "the first request reaching pf-serve", func() bool { return len(fakeLines(t, log, "request")) == 1 })+ queued := findAsync(context.Background(), "another Alice")+ if r := within10s(t, "the first request", func() found { return <-first }); r.err == nil {+ t.Fatal("the first request was answered by a hung pf-serve")+ }+ r := within10s(t, "the queued request", func() found { return <-queued })+ if r.err == nil || !strings.Contains(r.err.Error(), "tried again in") || !strings.Contains(r.err.Error(), "didn't answer") {+ t.Fatalf("the queued request: %v, want held back with the model's failure", r.err)+ }+ if _, err := Find([]string{"a third Alice"}); err == nil || !strings.Contains(err.Error(), "tried again in") {+ t.Fatalf("a later request: %v, want held back", err)+ }+ if n := len(fakeLines(t, log, "start")); n != 2 {+ t.Fatalf("started %d times, want 2: the first request's try and its one again", n)+ }+ srv.mu.Lock()+ srv.retryAt = time.Now().Add(-time.Second)+ srv.mu.Unlock()+ if _, err := Find([]string{"a fourth Alice"}); err == nil || strings.Contains(err.Error(), "tried again in") {+ t.Fatalf("once it's due: %v, want it tried and failing again", err)+ }+ if n := len(fakeLines(t, log, "start")); n != 4 {+ t.Fatalf("started %d times, want 4 once it was due", n)+ }+ srv.mu.Lock()+ again := srv.retryAt.Sub(time.Now())+ srv.mu.Unlock()+ if again < 40*time.Second {+ t.Fatalf("tried again in %s after the second failure, want a minute", again)+ }+ Stop()+ if err := heldBack(); err != nil {+ t.Fatalf("after Stop: %v, want tried at once", err)+ }+}+// Every pf-serve starts with ggml's Metal residency heartbeat off: it// woke an idle one ~146 times a second (#3185). The CPU one after a GPU// death too.