Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

pii: a GPU too slow for one request is tried again, and a lost GPU comes back without a restart (#3682)

yetonedd5b0ab31150
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index eea6dd590..b3f79fee3 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -119,7 +119,7 @@ recorded as a prediction failure (`noPrediction`).
1. `handle(proto)` reads the body and the model (`requestBody`, `requestModel`), then calls `serveAgent`. Every route that reads an agent's body (each API, count_tokens, Gemini's, embeddings, images, videos, System One, the Codex backend path) reads it through `readBoundedRequestBody` in [`request_bounds.go`](../../internal/gateway/request_bounds.go). It decodes a `Content-Encoding` of `gzip` or `zstd` (`decodedBody`; Codex sends zstd to a custom provider's `/v1/responses` too, #1223) and drops the header, so the provider is sent plain JSON, with one exception: a ChatGPT account's turn (POST `CodexBase`/responses, from the accounts loop in `forwardOnce` or relayed by `codexUpstreamOn`) is compressed again with zstd and a window of at most 2 MiB, as Codex's own client sends it (`provider.CodexCompressed`; openai/codex `responses_request_compression` and `http-client/src/request.rs`). It is compressed once the body is final, and again on each try. `/responses/compact`, an API key's requests and every other provider stay plain. Sent plain, a 1.1 MB text-only turn was answered 503 (ylorn on X). The plain body is what magpie keeps and shows, and `codexDropped` judges a reset by the bytes that went on the wire. `codexDropped` also reads Envoy's 507 "exceeded request buffer limit while retrying upstream" (the backend failed and the front couldn't retry a body that large) as that reset, at any size, told as a 503 so magpie, Codex and pi ask again and nobody rests (#2034). The size limit (`defaultBodyLimit`, 512 MB) holds for the decoded bytes (413), and any other encoding is refused 415. It was 128 MB until a long Codex session that had read many images went past it decoded, its inline screenshots resent each turn, and got the 413 on all five of Codex's retries (#1422, `TestImageHeavyCodexTurnIsRead`). magpie's own 413 says it is magpie's gateway limit, with the size, and that the provider was not asked (`overLimit`); `writeError` adds to any 413 it isn't holding that this is a byte limit, not the model's context, and to `/compact` or start a new conversation (`TestOverLimitSaysWhoseAndWhatToDo`). Every shape but Gemini's types a 413 `request_too_large`, Anthropic's own type, which pi's overflow table knows: pi compacts and tries once more rather than stopping on an OpenAI-shaped `api_error` (@wilsonxuuuu on X, `TestPayloadTooLargeSaysRequestTooLargeToPi`). `keyLimited` reads a limited key's compressed body the same way, so its reservation is the decoded size. `serveAgent` runs the [middleware](gateway-middleware.md) chain, then `serve`.
2. `serve` redacts secrets in the request (`redacted`; `redact.MaskJSON` leaves ids alone, and in any text a backend's id (a UUID, or 32 hex digits or more) goes as written: no rule, model label or value known from before masks a piece of one (a phone number at the end of `…-a13800138000`, a card number across a UUID's groups), and a whole one is masked only by the user's own words and rules or as a credential, by the password, API key and token rules or a secret rule whose field is named for one (`api_key=`, `refresh_token`), never one named for an id, a cursor, a request or a digest (`secret_id`, `next_page_token`, `Idempotency-Key`, `secret_hash`), a bare `key`, a serial number, or what the model takes a UUID for (`noIDs`, [`internal/redact/ids.go`](../../internal/redact/ids.go)); and a tool's name and its MCP server's (`namespace`, `server_label`, `server_name`, #3143); the request's tool definitions and output schemas, found by their place in each protocol (top-level `tools`, `tool_choice`, `response_format`, `text`, `generationConfig`, `stream_options`, the client's ids in `client_metadata` and the like, and the tools of a Responses `additional_tools` or `tool_search_output` item), go as written save their descriptions, so a word the user masks can't break a grammar or an enum (#1911); and what the vendor sealed with a signature, a thinking block, Gemini's thoughts or Responses' encrypted reasoning, goes back as the vendor wrote it: only the values masked elsewhere in the request, and secrets masked before, are put back as their placeholders; a value masked once, at least 8 characters long, is masked again wherever a request has it, as an exact match whatever words are around it, so what the vendor wrote about a placeholder doesn't go back with the value in it, `redact/known.go`; a text masked for a vendor before, such as a system prompt or a turn already sent, is masked as it was then, so a value a rule first finds later in a conversation doesn't change the prompt the vendor has cached and rewrite all of it to the cache, `sentKnownSpans`), resolves the model to a provider or a group, and refuses what can't be served. The reply has the values put back on its way to the agent (`redact.Writer`, a stream event by event, a whole body at its end): a placeholder the model wrote as it was given, or as models rewrite one (spaces inside the braces, a brace or both dropped, another case), is the value again when what is left names one magpie made, and one split across events is held until it is whole (`redact.Restore`, `partialTail`, [`internal/redact/redact.go`](../../internal/redact/redact.go)). A request whose provider and fallbacks, or whose group's models, all skip redaction (`unredactedRoute`, `provider.SkipsRedaction`: set *Send requests unmasked*, `Unredacted`, with every address on this machine or the local network) goes to them as the agent wrote it; the log and recent calls keep the masked body. count_tokens and embeddings follow the same rule. What a routing group's Jev is told of the turn (the message, the user's message before it and the reply to it) is masked all the same unless Jev's own provider skips redaction too (`maskedState` in `askJev`, [`decide.go`](../../internal/gateway/decide.go)): a remote Jev is none of the group's members. The masking is the built-in plugin `redact`'s (`gwmw.Masker`, [Plugin runtime](plugin-runtime.md)); when the settings ask for masking and no masker serves, or it fails, the request is a 503 and nothing is sent (`redacted`, `gwmw.Masking`).
- With *Mask personal data* on, each kind in `redact.Categories` ([`internal/redact/more.go`](../../internal/redact/more.go)) is masked unless the user turned it off (`redactKinds` in settings, `Options.Kinds`). On by default: US SSNs, passport numbers, IBANs that pass mod 97, birthdays next to a word that says so (a date over a year ago: one in the last year or ahead is the day an agent runs on, `dateOK`), MAC addresses, and serial numbers next to S/N or the like. Off by default: the user name in a home folder, user@host in a prompt or ssh command, public IPv4 and IPv6 addresses (never private, loopback, link-local, CGNAT or documentation ranges) and cloud bucket names, since agents send them in nearly every request and the model must copy the placeholder exactly inside a path or URL. Context rules run only near their words (`rule.matches`). What reads as an address but ends in a file's extension that no domain has (`…@2x.jpg`, `[email protected]`; `fileExt` in `realEmail`), or a scale before an extension that is a domain too (`[email protected]`, `[email protected]`; `scaleExt`), is a file's name, for the rule and the local model alike (#3029). `[email protected]` is still an address. The Settings rows come from `REDACT_KINDS` in app.js, which `TestRedactKindsAreRedacts` holds to the same ids and defaults.
- - With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken, nor a snake_case word as a name (Codex's `sequential_cutoff`, #3144), nor as a secret what names a thing: a UUID, a digest with its algorithm's name, a file's path (`namesThing`; the model takes all three for secrets). The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with it on a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left. The model is OpenAI's Privacy Filter as a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags; on a GPU it holds the model once, in the device's memory, #3590; `pf-serve-vulkan` runs its MoE in 2048-token chunks, `PF_MOE_CHUNK`, 1.77× faster than its default 4096 on an RX 470/580-class GPU and a little slower on Metal, so Vulkan's alone, and a value the user set is kept, `moeChunk`): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, from hf-mirror first in China, `source.chinaFirst`) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU, or doesn't load there in 40s (`gpuLoadTimeout`, well inside the gateway's 2 minutes), is started on the CPU for the rest of the run, and Settings says why), joins short texts into 12 KB chunks and maps the spans back, keeps the last 20000 answers by hash, on disk too (`found-<model>.bin` beside the model: hashes, offsets and labels, never text; written through temp-and-rename, a bad read taken as empty), so a restart doesn't ask the model about every history again (#3682), and ends it after 10 minutes with no request. Its requests take turns on its one pipe (`turn`), and a request reads the cache again when its turn comes; a request waits for its turn only while its context lives (`pii.FindContext`, through `gwmw.ContextTeller`), so one whose client went away, or that the gateway answered 504 after 2 minutes, leaves the queue; a turn begun is finished and kept. The status (`pii.Now`), `pii.Stop` and the idle timer never wait for a load or a request: `srv.mu` is held only to read or set the state, and `Stop` ends a load or a request under way at once. A request held before routing while it waits for the model is answered 504 saying so, and how to send without it (`unroutedMsg`); one of an older tag (`oldTags`, magpie updated since the download) runs until the pinned tag's is fetched (`upgrade`, on its start), which is then started on the next request and the old deleted ([`internal/pii`](../../internal/pii)). A failure names what ended `pf-serve`: magpie's own end of a request it didn't answer in time, else its exit status or signal and its last stderr lines past ggml's startup chatter (`died`, `lastSaid`). After one, requests are held back at once with that failure, not after a load each, until the model is tried again 30s later, doubling to 10 minutes while it keeps failing (`heldBack`, `retryAfter`; lazyeo on Discord, an M2 whose every request waited out the 504). A model that is on and can't answer (not on disk, didn't start, hung, held back) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).
+ - With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken, nor a snake_case word as a name (Codex's `sequential_cutoff`, #3144), nor as a secret what names a thing: a UUID, a digest with its algorithm's name, a file's path (`namesThing`; the model takes all three for secrets). The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with it on a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left. The model is OpenAI's Privacy Filter as a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags; on a GPU it holds the model once, in the device's memory, #3590; `pf-serve-vulkan` runs its MoE in 2048-token chunks, `PF_MOE_CHUNK`, 1.77× faster than its default 4096 on an RX 470/580-class GPU and a little slower on Metal, so Vulkan's alone, and a value the user set is kept, `moeChunk`): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, from hf-mirror first in China, `source.chinaFirst`) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU, doesn't load there in 40s (`gpuLoadTimeout`, well inside the gateway's 2 minutes), or doesn't answer in time there twice in a row, is started on the CPU, and Settings says why; a request it didn't answer in time once is tried on the GPU again, since a slow request on a weak GPU is no lost GPU (#3682). The GPU is tried again 10 minutes later, doubling to 2 hours while it keeps failing, with no restart (`gpuRetryAfter`), and one that answers is the model's again), joins short texts into 12 KB chunks and maps the spans back, keeps the last 20000 answers by hash, on disk too (`found-<model>.bin` beside the model: hashes, offsets and labels, never text; written through temp-and-rename, a bad read taken as empty), so a restart doesn't ask the model about every history again (#3682), and ends it after 10 minutes with no request. Its requests take turns on its one pipe (`turn`), and a request reads the cache again when its turn comes; a request waits for its turn only while its context lives (`pii.FindContext`, through `gwmw.ContextTeller`), so one whose client went away, or that the gateway answered 504 after 2 minutes, leaves the queue; a turn begun is finished and kept. The status (`pii.Now`), `pii.Stop` and the idle timer never wait for a load or a request: `srv.mu` is held only to read or set the state, and `Stop` ends a load or a request under way at once. A request held before routing while it waits for the model is answered 504 saying so, and how to send without it (`unroutedMsg`); one of an older tag (`oldTags`, magpie updated since the download) runs until the pinned tag's is fetched (`upgrade`, on its start), which is then started on the next request and the old deleted ([`internal/pii`](../../internal/pii)). A failure names what ended `pf-serve`: magpie's own end of a request it didn't answer in time, else its exit status or signal and its last stderr lines past ggml's startup chatter (`died`, `lastSaid`). After one, requests are held back at once with that failure, not after a load each, until the model is tried again 30s later, doubling to 10 minutes while it keeps failing (`heldBack`, `retryAfter`; lazyeo on Discord, an M2 whose every request waited out the 504). A model that is on and can't answer (not on disk, didn't start, hung, held back) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).
- With *Mask secrets* on, the user's own masking rules (`redactRules`, `redact.Rule`: a placeholder name and a prefix or an RE2 pattern, [`internal/redact/rules.go`](../../internal/redact/rules.go)) run before magpie's. Settings → Privacy adds them, removes them, and edits one in place (`POST /api/settings/redact-rule` with the rule's index and its current value, `editRedactRule` in [`internal/gui/api.go`](../../internal/gui/api.go)). Every rule is checked by `redact.CheckRules` when settings are saved; an edit that doesn't compile, matches the empty string, repeats another rule or names a rule no longer at that index is refused with the rules left as they were. The masker reads the rules from settings on each call (`options` in [`internal/masking`](../../internal/masking/masking.go)), so a change applies to the next request.
- With *Keep a key's form* on (`settings.RedactShape`, `Options.Shape`, under *Mask secrets*; Charles Cao on X), a secret of 16 characters or more, all letters, digits, `-`, `_` or `.`, goes as a made-up key of its form instead of a placeholder: its prefix kept (`sk-proj-`, `ghp_`, `AKIA`, up to four letters and their separator; `keyHead`), the same length and separators, and each other character a random one of its kind (a hex key's any hex digit of its case), drawn from an HMAC of the value under the placeholder key, so one key goes as the same made-up one every turn and a vendor's cache still matches (`shapeOf`, `shaped` in [`internal/redact/shape.go`](../../internal/redact/shape.go)). Personal data, the user's words and anything shorter or with other characters go as placeholders still. A made-up key is kept in the masker's memory with the placeholders, so it is recorded, re-masked in what a vendor sealed, and put back wherever the reply has it, before the placeholders (`restoreShapes`); a stream holds the end of an event that may be the start of one (`shapeTail`, after the last whole one), and a made-up key the vendor sends back unrestored isn't masked again (`outsideShapes`). `TestShapeKeepsForm`, `TestShapeJSON`, `TestShapeStream`. Insights › Masking and Settings' *Try it* mark a made-up key where they mark a placeholder, matching the text against the request's own values only, so a key of the same form the user wrote isn't marked (`maskMatches` in `plugins/redact/page.js`; #3599, masking-records and mask-actions GUI tests).
- What is done with a value found is the user's per kind (#1977, `redactActions` in settings, `Options.Actions`, set by `POST /api/settings/redact-actions`; a kind left out is masked). `redact.Warn` sends it as written: the masker tells it, with a placeholder no one remembers, and it isn't counted as masked. `redact.Block` masks it as any other, and the gateway holds the request back with a 400 that names the kind and the value's hint, never the value (`blocked`, `maskBlocked` in [`redact.go`](../../internal/gateway/redact.go)); nothing is sent and the record is kept as `Blocked` (`noteBlocked`). A value on the *Never mask* list (`redactAllow`, `Options.Allow`, `POST /api/settings/redact-allow`) goes as written when it is the whole value found, in any case, and isn't told; a longer value it is part of is masked. Settings → Privacy's *Try it* posts a text to `POST /api/masking/preview` ([`masking.go`](../../internal/gui/masking.go)), which masks it as a request would be (`redact.Preview`) and remembers nothing: its placeholders stand for nothing afterwards, its values aren't masked later as seen before, and it returns only the masked text and each hit's kind, finder, action, placeholder and hint. The page keeps the text in no storage.
diff --git a/internal/pii/main_test.go b/internal/pii/main_test.go
index c766c0a3b..41aaf456d 100644
--- a/internal/pii/main_test.go
+++ b/internal/pii/main_test.go
@@ -44,6 +44,11 @@ const (
// novulkan as pf-serve-vulkan, says Vulkan is missing; else ok
// gpuloadhang on the GPU, never says the model is loaded; else ok
// asserts fails a ggml assert at its first request
+// gpuhang on the GPU, never answers a request; else ok
+// gpuhangonce on the GPU, never answers the first request of the
+// test; else ok
+// gpudiesonce on the GPU, ends at the first request of the test;
+// else ok
//
// Like pf-serve, it first writes ggml's startup lines to stderr.
func fakeServe(mode string) {
@@ -100,11 +105,19 @@ func fakeServe(mode string) {
texts[i] = string(b)
}
logf("request %d %q", n, texts)
+ // once: the first request of the test, on the GPU
+ once := false
+ if p := os.Getenv(fakeLog); p != "" && gpu && strings.HasSuffix(mode, "once") {
+ if f, err := os.OpenFile(p+".once", os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o644); err == nil {
+ f.Close()
+ once = true
+ }
+ }
switch {
- case mode == "gpudies" && gpu:
+ case mode == "gpudies" && gpu, mode == "gpudiesonce" && once:
fmt.Fprintln(os.Stderr, "ggml_metal: device lost")
os.Exit(3)
- case mode == "hang":
+ case mode == "hang", mode == "gpuhang" && gpu, mode == "gpuhangonce" && once:
time.Sleep(time.Hour)
case mode == "asserts":
fmt.Fprintln(os.Stderr, "ggml_metal_library_compile_pipeline: compiling pipeline: base = 'kernel_add_id'")
@@ -162,6 +175,7 @@ func fakeInstall(t *testing.T, mode string, vulkan bool) string {
disk.saves.Wait() // a write under way reads dir
srv.mu.Lock()
srv.cpu, srv.gpuLost, srv.failed = false, "", ""
+ srv.gpuFails, srv.gpuAgain = 0, time.Time{}
srv.lastErr, srv.fails, srv.retryAt = nil, 0, time.Time{}
srv.mu.Unlock()
cache.mu.Lock()
diff --git a/internal/pii/pii.go b/internal/pii/pii.go
index 77a7ac6a1..dd6401548 100644
--- a/internal/pii/pii.go
+++ b/internal/pii/pii.go
@@ -13,9 +13,10 @@
//
// A process of its own, not linked in: 1.7 GB the gateway would otherwise
// hold for good, and a GPU driver's crash that would end the gateway with
-// it. One that dies is started again; one that dies on the GPU, or doesn't
-// load there in gpuLoadTimeout, is started on the CPU for the rest of
-// magpie's run. One that keeps failing holds requests back at once, not
+// it. One that dies is started again; one that dies on the GPU, doesn't
+// load there in gpuLoadTimeout, or hangs there twice in a row, is started
+// on the CPU until the GPU is tried again (gpuRetryAfter), with no restart
+// of magpie. One that keeps failing holds requests back at once, not
// after a load each, until it is tried again (retryAfter).
package pii
diff --git a/internal/pii/serve.go b/internal/pii/serve.go
index 2ec854c90..39c711214 100644
--- a/internal/pii/serve.go
+++ b/internal/pii/serve.go
@@ -53,10 +53,17 @@ var srv struct {
mu sync.Mutex
h *helper
idle *time.Timer
- // cpu is set once pf-serve died on the GPU: from then on it runs on
- // the CPU; gpuLost says why
- cpu bool
- gpuLost string
+ // cpu is set once pf-serve died on the GPU, didn't load there, or
+ // hung there twice in a row: it runs on the CPU until gpuAgain, when
+ // the GPU is tried again (gpuRetryAfter); gpuLost says why, gpuFails
+ // how many times in a row the GPU was lost. A GPU that answers clears
+ // them. Before, nothing did short of a restart of magpie, and one
+ // request too slow for its time on a weak GPU left every request after
+ // it on a CPU 3x slower (#3682).
+ cpu bool
+ gpuLost string
+ gpuFails int
+ gpuAgain time.Time
// failed is why the last request got no answer, until one does;
// lastErr is that error, fails how many requests in a row got none,
// and retryAt when the model is tried again: until then a request
@@ -94,7 +101,7 @@ var loadTimeout = 3 * time.Minute
// gpuLoadTimeout is how long it has on a GPU, where it takes seconds
// (0.9-10s on an M4 Pro): one that takes longer is taken for hung, and
-// the CPU is tried, for the rest of magpie's run. It is well under the 2
+// the CPU is tried, until the GPU is tried again. It is well under the 2
// minutes the gateway holds a request for (unroutedMost), so a GPU's
// hung load and the CPU's after it fit in one request's wait: with 3
// minutes for either, every request in a hung load was answered 504.
@@ -117,6 +124,26 @@ func retryAfter(n int) time.Duration {
return min(d, 10*time.Minute)
}
+// gpuRetryAfter is how long after the GPU was lost, the n-th time in a
+// row, the GPU is tried again: 10 minutes, doubled each time, 2 hours at
+// most. A try costs a load (seconds; gpuLoadTimeout for one that hangs)
+// and the request it is for goes on to the CPU's if it fails.
+func gpuRetryAfter(n int) time.Duration {
+ d := 10 * time.Minute
+ for i := 1; i < n && d < 2*time.Hour; i++ {
+ d *= 2
+ }
+ return min(d, 2*time.Hour)
+}
+
+// lostGPULocked moves the model to the CPU, why being what went wrong on
+// the GPU, until the GPU is tried again; mu is held.
+func lostGPULocked(why string) {
+ srv.gpuFails++
+ srv.cpu, srv.gpuLost = true, why
+ srv.gpuAgain = time.Now().Add(gpuRetryAfter(srv.gpuFails))
+}
+
// classifyTimeout is how long one request may take before pf-serve is
// taken for hung and ended: a second for each 4 KB on the slowest CPU,
// and 30 more.
@@ -165,10 +192,11 @@ func start(cpu bool, stop <-chan struct{}) (*helper, error) {
h, err := startOne(c.path, c.cpu, stop)
if err == nil {
if gpuLost != "" {
- // the GPU's failed and the CPU's loaded: the CPU for
- // the rest of the run, as for a GPU that dies answering
+ // the GPU's failed and the CPU's loaded: the CPU until
+ // the GPU is tried again, as for a GPU that dies
+ // answering
srv.mu.Lock()
- srv.cpu, srv.gpuLost = true, gpuLost
+ lostGPULocked(gpuLost)
srv.mu.Unlock()
}
return h, nil
@@ -422,8 +450,11 @@ func end(h *helper) {
}
// classify asks pf-serve, started if it isn't, what is in texts. One that
-// fails is ended and started again; one that failed on the GPU is started
-// on the CPU, for good. The caller holds turn.
+// fails is ended and started again; one that died on the GPU is started
+// on the CPU, until the GPU is tried again. One that magpie ended on the
+// GPU for taking too long is started again on the GPU: a request too
+// slow for its time is no lost GPU. Twice in a row there is, and the CPU
+// is tried for the next. The caller holds turn.
func classify(texts []string) ([][]redact.Found, error) {
out, err := classifyTurn(texts)
srv.mu.Lock()
@@ -460,10 +491,21 @@ func classifyTurn(texts []string) ([][]redact.Found, error) {
for try := 0; try < 2; try++ {
srv.mu.Lock()
h, cpu, stop := srv.h, srv.cpu, stopCh()
+ var gone *helper
+ if cpu && !time.Now().Before(srv.gpuAgain) {
+ // the GPU is tried again: a pf-serve on the CPU makes way.
+ // One that fails there moves back to the CPU (start, or
+ // below), with longer until the next try.
+ cpu = false
+ if h != nil && !h.hello.GPU {
+ gone, h = detachLocked(), nil
+ }
+ }
if h == nil {
srv.loading = true
}
srv.mu.Unlock()
+ end(gone)
if h == nil {
var err error
h, err = start(cpu, stop)
@@ -488,6 +530,10 @@ func classifyTurn(texts []string) ([][]redact.Found, error) {
if srv.h == h {
idleLocked(h)
}
+ if h.hello.GPU {
+ // the GPU answers: it is the model's again
+ srv.cpu, srv.gpuLost, srv.gpuFails = false, "", 0
+ }
srv.mu.Unlock()
return out, nil
}
@@ -501,7 +547,13 @@ func classifyTurn(texts []string) ([][]redact.Found, error) {
return nil, errStopped
}
if h.hello.GPU {
- srv.cpu, srv.gpuLost = true, err.Error()
+ if h.slow.Load() != nil && try == 0 {
+ // ended for taking too long: a slow request, or a
+ // GPU's hang that a new pf-serve gets past, is no lost
+ // GPU, and the try again is on the GPU
+ } else {
+ lostGPULocked(err.Error())
+ }
}
srv.mu.Unlock()
h.kill()
diff --git a/internal/pii/serve_test.go b/internal/pii/serve_test.go
index e2dcbebc4..f753e2cdb 100644
--- a/internal/pii/serve_test.go
+++ b/internal/pii/serve_test.go
@@ -120,8 +120,8 @@ func TestVulkanThatWontStartFallsBack(t *testing.T) {
// A GPU whose load hangs (lazyeo's M2: pf-serve kept loading for over a
// minute while every request was answered 504 after 2) is given
-// gpuLoadTimeout, then the CPU is tried, and kept for the rest of the
-// run. On macOS, whose pf-serve is Metal's, too: there it was the only
+// gpuLoadTimeout, then the CPU is tried, and kept until the GPU is tried
+// again (TestLostGPUTriedAgain). On macOS, whose pf-serve is Metal's, too: there it was the only
// one tried.
func TestHungGPULoadMovesToCPU(t *testing.T) {
for _, vulkan := range []bool{false, true} {
@@ -442,3 +442,106 @@ func TestVulkanServeGetsMoEChunk(t *testing.T) {
})
}
}
+
+// A request pf-serve takes too long over on the GPU (#3682: 480 KB on an
+// RX 470/580, killed at classifyTimeout) is tried again on the GPU, not
+// moved to a CPU 3x slower for the rest of the run.
+func TestGPUTimeoutStaysOnGPU(t *testing.T) {
+ log, gpu := gpuInstall(t, "gpuhangonce")
+ was := classifyTimeout
+ classifyTimeout = func(int) time.Duration { return 300 * time.Millisecond }
+ t.Cleanup(func() { classifyTimeout = was })
+ got, err := Find([]string{"hi Alice"})
+ if err != nil || !reflect.DeepEqual(got, [][]redact.Found{{person(3, 8)}}) {
+ t.Fatalf("Find = %v, %v", got, err)
+ }
+ if starts := fakeLines(t, log, "start"); !reflect.DeepEqual(starts, []string{gpu, gpu}) {
+ t.Fatalf("starts = %q, want the GPU tried again after its timeout", starts)
+ }
+ if st := Now(); !st.GPU || st.GPUError != "" {
+ t.Fatalf("Now = %+v, want on the GPU with nothing lost", st)
+ }
+ if _, err := Find([]string{"Alice again"}); err != nil {
+ t.Fatal(err)
+ }
+ if starts := fakeLines(t, log, "start"); len(starts) != 2 {
+ t.Fatalf("starts = %q, want the GPU's pf-serve kept", starts)
+ }
+}
+
+// A GPU that hangs twice in a row is a GPU lost: the next request goes
+// to the CPU, as for one that dies. Until the GPU is tried again.
+func TestGPUHungTwiceMovesToCPU(t *testing.T) {
+ log, gpu := gpuInstall(t, "gpuhang")
+ was := classifyTimeout
+ classifyTimeout = func(int) time.Duration { return 300 * time.Millisecond }
+ t.Cleanup(func() { classifyTimeout = was })
+ if _, err := Find([]string{"hi Alice"}); err == nil {
+ t.Fatal("a GPU that hangs twice answered")
+ }
+ if starts := fakeLines(t, log, "start"); !reflect.DeepEqual(starts, []string{gpu, gpu}) {
+ t.Fatalf("starts = %q, want the GPU twice", starts)
+ }
+ srv.mu.Lock()
+ srv.retryAt = time.Time{} // not held back: tried at once
+ srv.mu.Unlock()
+ got, err := Find([]string{"hi Alice"})
+ if err != nil || !reflect.DeepEqual(got, [][]redact.Found{{person(3, 8)}}) {
+ t.Fatalf("Find = %v, %v", got, err)
+ }
+ if starts := fakeLines(t, log, "start"); len(starts) != 3 || starts[2] != "start pf-serve CPU" {
+ t.Fatalf("starts = %q, want the CPU next", starts)
+ }
+ if st := Now(); st.GPU || !strings.Contains(st.GPUError, "didn't answer in") {
+ t.Fatalf("Now = %+v, want on the CPU saying the GPU didn't answer", st)
+ }
+}
+
+// A GPU lost is tried again when its time comes (gpuRetryAfter), with
+// no restart of magpie: the CPU's pf-serve makes way for the GPU's, and a
+// GPU that answers is the model's again.
+func TestLostGPUTriedAgain(t *testing.T) {
+ log, gpu := gpuInstall(t, "gpudiesonce")
+ if _, err := Find([]string{"hi Alice"}); err != nil {
+ t.Fatal(err)
+ }
+ if starts := fakeLines(t, log, "start"); !reflect.DeepEqual(starts, []string{gpu, "start pf-serve CPU"}) {
+ t.Fatalf("starts = %q, want the GPU, then the CPU", starts)
+ }
+ if _, err := Find([]string{"Alice two"}); err != nil {
+ t.Fatal(err)
+ }
+ if starts := fakeLines(t, log, "start"); len(starts) != 2 {
+ t.Fatalf("starts = %q, want the CPU kept until the GPU's time", starts)
+ }
+ srv.mu.Lock()
+ if !srv.cpu || srv.gpuFails != 1 || time.Until(srv.gpuAgain) < 9*time.Minute {
+ t.Fatalf("cpu=%v fails=%d again in %s, want the CPU for 10 minutes", srv.cpu, srv.gpuFails, time.Until(srv.gpuAgain))
+ }
+ srv.gpuAgain = time.Now() // its time has come
+ srv.mu.Unlock()
+ got, err := Find([]string{"Alice three"})
+ if err != nil || !reflect.DeepEqual(got, [][]redact.Found{{person(0, 5)}}) {
+ t.Fatalf("Find = %v, %v", got, err)
+ }
+ if starts := fakeLines(t, log, "start"); len(starts) != 3 || starts[2] != gpu {
+ t.Fatalf("starts = %q, want the GPU tried again", starts)
+ }
+ if st := Now(); !st.GPU || st.GPUError != "" {
+ t.Fatalf("Now = %+v, want the GPU's again", st)
+ }
+ srv.mu.Lock()
+ cpu, fails := srv.cpu, srv.gpuFails
+ srv.mu.Unlock()
+ if cpu || fails != 0 {
+ t.Fatalf("cpu=%v fails=%d after the GPU answered", cpu, fails)
+ }
+}
+
+func TestGPURetryAfter(t *testing.T) {
+ for n, want := range map[int]time.Duration{1: 10 * time.Minute, 2: 20 * time.Minute, 3: 40 * time.Minute, 4: 80 * time.Minute, 5: 2 * time.Hour, 50: 2 * time.Hour} {
+ if got := gpuRetryAfter(n); got != want {
+ t.Errorf("gpuRetryAfter(%d) = %s, want %s", n, got, want)
+ }
+ }
+}