Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

pii: the masking model's requests take turns without holding the status, Stop or a restart (#3682)

yetonee5dd86965b56
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index e6f6f2cfa..18c997a47 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -90,7 +90,7 @@ recorded as a prediction failure (`noPrediction`).
1. `handle(proto)` reads the body and the model (`requestBody`, `requestModel`), then calls `serveAgent`. Every route that reads an agent's body (each API, count_tokens, Gemini's, embeddings, images, videos, System One, the Codex backend path) reads it through `readBoundedRequestBody` in [`request_bounds.go`](../../internal/gateway/request_bounds.go). It decodes a `Content-Encoding` of `gzip` or `zstd` (`decodedBody`; Codex sends zstd to a custom provider's `/v1/responses` too, #1223) and drops the header, so the provider is sent plain JSON, with one exception: a ChatGPT account's turn (POST `CodexBase`/responses, from the accounts loop in `forwardOnce` or relayed by `codexUpstreamOn`) is compressed again with zstd and a window of at most 2 MiB, as Codex's own client sends it (`provider.CodexCompressed`; openai/codex `responses_request_compression` and `http-client/src/request.rs`). It is compressed once the body is final, and again on each try. `/responses/compact`, an API key's requests and every other provider stay plain. Sent plain, a 1.1 MB text-only turn was answered 503 (ylorn on X). The plain body is what magpie keeps and shows, and `codexDropped` judges a reset by the bytes that went on the wire. `codexDropped` also reads Envoy's 507 "exceeded request buffer limit while retrying upstream" (the backend failed and the front couldn't retry a body that large) as that reset, at any size, told as a 503 so magpie, Codex and pi ask again and nobody rests (#2034). The size limit (`defaultBodyLimit`, 512 MB) holds for the decoded bytes (413), and any other encoding is refused 415. It was 128 MB until a long Codex session that had read many images went past it decoded, its inline screenshots resent each turn, and got the 413 on all five of Codex's retries (#1422, `TestImageHeavyCodexTurnIsRead`). magpie's own 413 says it is magpie's gateway limit, with the size, and that the provider was not asked (`overLimit`); `writeError` adds to any 413 it isn't holding that this is a byte limit, not the model's context, and to `/compact` or start a new conversation (`TestOverLimitSaysWhoseAndWhatToDo`). Every shape but Gemini's types a 413 `request_too_large`, Anthropic's own type, which pi's overflow table knows: pi compacts and tries once more rather than stopping on an OpenAI-shaped `api_error` (@wilsonxuuuu on X, `TestPayloadTooLargeSaysRequestTooLargeToPi`). `keyLimited` reads a limited key's compressed body the same way, so its reservation is the decoded size. `serveAgent` runs the [middleware](gateway-middleware.md) chain, then `serve`.
2. `serve` redacts secrets in the request (`redacted`; `redact.MaskJSON` leaves ids alone, and in any text a backend's id (a UUID, or 32 hex digits or more) goes as written: no rule, model label or value known from before masks a piece of one (a phone number at the end of `…-a13800138000`, a card number across a UUID's groups), and a whole one is masked only by the user's own words and rules or as a credential, by the password, API key and token rules or a secret rule whose field is named for one (`api_key=`, `refresh_token`), never one named for an id, a cursor, a request or a digest (`secret_id`, `next_page_token`, `Idempotency-Key`, `secret_hash`), a bare `key`, a serial number, or what the model takes a UUID for (`noIDs`, [`internal/redact/ids.go`](../../internal/redact/ids.go)); and a tool's name and its MCP server's (`namespace`, `server_label`, `server_name`, #3143); the request's tool definitions and output schemas, found by their place in each protocol (top-level `tools`, `tool_choice`, `response_format`, `text`, `generationConfig`, `stream_options`, the client's ids in `client_metadata` and the like, and the tools of a Responses `additional_tools` or `tool_search_output` item), go as written save their descriptions, so a word the user masks can't break a grammar or an enum (#1911); and what the vendor sealed with a signature, a thinking block, Gemini's thoughts or Responses' encrypted reasoning, goes back as the vendor wrote it: only the values masked elsewhere in the request, and secrets masked before, are put back as their placeholders; a value masked once, at least 8 characters long, is masked again wherever a request has it, as an exact match whatever words are around it, so what the vendor wrote about a placeholder doesn't go back with the value in it, `redact/known.go`; a text masked for a vendor before, such as a system prompt or a turn already sent, is masked as it was then, so a value a rule first finds later in a conversation doesn't change the prompt the vendor has cached and rewrite all of it to the cache, `sentKnownSpans`), resolves the model to a provider or a group, and refuses what can't be served. The reply has the values put back on its way to the agent (`redact.Writer`, a stream event by event, a whole body at its end): a placeholder the model wrote as it was given, or as models rewrite one (spaces inside the braces, a brace or both dropped, another case), is the value again when what is left names one magpie made, and one split across events is held until it is whole (`redact.Restore`, `partialTail`, [`internal/redact/redact.go`](../../internal/redact/redact.go)). A request whose provider and fallbacks, or whose group's models, all skip redaction (`unredactedRoute`, `provider.SkipsRedaction`: set *Send requests unmasked*, `Unredacted`, with every address on this machine or the local network) goes to them as the agent wrote it; the log and recent calls keep the masked body. count_tokens and embeddings follow the same rule. What a routing group's Jev is told of the turn (the message, the user's message before it and the reply to it) is masked all the same unless Jev's own provider skips redaction too (`maskedState` in `askJev`, [`decide.go`](../../internal/gateway/decide.go)): a remote Jev is none of the group's members. The masking is the built-in plugin `redact`'s (`gwmw.Masker`, [Plugin runtime](plugin-runtime.md)); when the settings ask for masking and no masker serves, or it fails, the request is a 503 and nothing is sent (`redacted`, `gwmw.Masking`).
- With *Mask personal data* on, each kind in `redact.Categories` ([`internal/redact/more.go`](../../internal/redact/more.go)) is masked unless the user turned it off (`redactKinds` in settings, `Options.Kinds`). On by default: US SSNs, passport numbers, IBANs that pass mod 97, birthdays next to a word that says so (a date over a year ago: one in the last year or ahead is the day an agent runs on, `dateOK`), MAC addresses, and serial numbers next to S/N or the like. Off by default: the user name in a home folder, user@host in a prompt or ssh command, public IPv4 and IPv6 addresses (never private, loopback, link-local, CGNAT or documentation ranges) and cloud bucket names, since agents send them in nearly every request and the model must copy the placeholder exactly inside a path or URL. Context rules run only near their words (`rule.matches`). What reads as an address but ends in a file's extension that no domain has (`…@2x.jpg`, `[email protected]`; `fileExt` in `realEmail`), or a scale before an extension that is a domain too (`[email protected]`, `[email protected]`; `scaleExt`), is a file's name, for the rule and the local model alike (#3029). `[email protected]` is still an address. The Settings rows come from `REDACT_KINDS` in app.js, which `TestRedactKindsAreRedacts` holds to the same ids and defaults.
- - With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken, nor a snake_case word as a name (Codex's `sequential_cutoff`, #3144), nor as a secret what names a thing: a UUID, a digest with its algorithm's name, a file's path (`namesThing`; the model takes all three for secrets). The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with it on a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left. The model is OpenAI's Privacy Filter as a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags; on a GPU it holds the model once, in the device's memory, #3590): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, from hf-mirror first in China, `source.chinaFirst`) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU is started on the CPU for the rest of the run), joins short texts into 12 KB chunks and maps the spans back, keeps the last 20000 answers by hash, and ends it after 10 minutes with no request; one of an older tag (`oldTags`, magpie updated since the download) runs until the pinned tag's is fetched (`upgrade`, on its start), which is then started on the next request and the old deleted ([`internal/pii`](../../internal/pii)). A model that is on and can't answer (not on disk, didn't start, hung) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).
+ - With *Find with a local model* on (`settings.RedactModel`, this computer's own, kept by `KeepOwn`), a local model finds what no rule can, a name or an address in free text, under the same switches: a name, address, phone, email or account number is personal data, a secret a secret (`modelKinds`, `modelSpans` in [`internal/redact/model.go`](../../internal/redact/model.go)). `MaskJSON` asks it once per request, for every text it would mask, each once (`withModel`); its spans join the rules' with a rule's kind kept where both find a value, and ids a vendor or tool made (`req_…`, `toolu_…`) are never taken, nor a snake_case word as a name (Codex's `sequential_cutoff`, #3144), nor as a secret what names a thing: a UUID, a digest with its algorithm's name, a file's path (`namesThing`; the model takes all three for secrets). The model misses many Chinese names (张伟 after 给, 陈建国 after 客户, however low its threshold) and runs a name it finds on into the words after it (「王芳,她下周一」), so with it on a Chinese name is also found by its surname and a word around it that says a person is meant (客户, 我叫, 给…发, …先生, …说, joined by 和 to another name), and the model's are cut at punctuation and titles ([`zhname.go`](../../internal/redact/zhname.go)); a surname with nothing around it (王国的, 张开的) is left. The model is OpenAI's Privacy Filter as a q8 GGUF (1.6 GB), run by `pf-serve` ([`native/privacy-filter`](../../native/privacy-filter), built from privacy-filter.cpp on ggml and released under `pf-v*` tags; on a GPU it holds the model once, in the device's memory, #3590): Settings → Privacy's On downloads both (`pii.Download`, each checked against its pinned sha256, from hf-mirror first in China, `source.chinaFirst`) and saves the setting only once they are on disk (`/api/privacy-model`, [`privacymodel.go`](../../internal/gui/privacymodel.go)). `pii.Find` starts `pf-serve` on the first request (Metal, Vulkan, else the CPU; one that dies on the GPU is started on the CPU for the rest of the run), joins short texts into 12 KB chunks and maps the spans back, keeps the last 20000 answers by hash, on disk too (`found-<model>.bin` beside the model: hashes, offsets and labels, never text; written through temp-and-rename, a bad read taken as empty), so a restart doesn't ask the model about every history again (#3682), and ends it after 10 minutes with no request. Its requests take turns on its one pipe (`turn`), and a request reads the cache again when its turn comes; a request waits for its turn only while its context lives (`pii.FindContext`, through `gwmw.ContextTeller`), so one whose client went away, or that the gateway answered 504 after 2 minutes, leaves the queue; a turn begun is finished and kept. The status (`pii.Now`), `pii.Stop` and the idle timer never wait for a load or a request: `srv.mu` is held only to read or set the state, and `Stop` ends a load or a request under way at once. A request held before routing while it waits for the model is answered 504 saying so, and how to send without it (`unroutedMsg`); one of an older tag (`oldTags`, magpie updated since the download) runs until the pinned tag's is fetched (`upgrade`, on its start), which is then started on the next request and the old deleted ([`internal/pii`](../../internal/pii)). A model that is on and can't answer (not on disk, didn't start, hung) panics, so the request is held back with the 503 above rather than sent with the rules' masking alone (`masking.model`).
- With *Mask secrets* on, the user's own masking rules (`redactRules`, `redact.Rule`: a placeholder name and a prefix or an RE2 pattern, [`internal/redact/rules.go`](../../internal/redact/rules.go)) run before magpie's. Settings → Privacy adds them, removes them, and edits one in place (`POST /api/settings/redact-rule` with the rule's index and its current value, `editRedactRule` in [`internal/gui/api.go`](../../internal/gui/api.go)). Every rule is checked by `redact.CheckRules` when settings are saved; an edit that doesn't compile, matches the empty string, repeats another rule or names a rule no longer at that index is refused with the rules left as they were. The masker reads the rules from settings on each call (`options` in [`internal/masking`](../../internal/masking/masking.go)), so a change applies to the next request.
- With *Keep a key's form* on (`settings.RedactShape`, `Options.Shape`, under *Mask secrets*; Charles Cao on X), a secret of 16 characters or more, all letters, digits, `-`, `_` or `.`, goes as a made-up key of its form instead of a placeholder: its prefix kept (`sk-proj-`, `ghp_`, `AKIA`, up to four letters and their separator; `keyHead`), the same length and separators, and each other character a random one of its kind (a hex key's any hex digit of its case), drawn from an HMAC of the value under the placeholder key, so one key goes as the same made-up one every turn and a vendor's cache still matches (`shapeOf`, `shaped` in [`internal/redact/shape.go`](../../internal/redact/shape.go)). Personal data, the user's words and anything shorter or with other characters go as placeholders still. A made-up key is kept in the masker's memory with the placeholders, so it is recorded, re-masked in what a vendor sealed, and put back wherever the reply has it, before the placeholders (`restoreShapes`); a stream holds the end of an event that may be the start of one (`shapeTail`, after the last whole one), and a made-up key the vendor sends back unrestored isn't masked again (`outsideShapes`). `TestShapeKeepsForm`, `TestShapeJSON`, `TestShapeStream`. Insights › Masking and Settings' *Try it* mark a made-up key where they mark a placeholder, matching the text against the request's own values only, so a key of the same form the user wrote isn't marked (`maskMatches` in `plugins/redact/page.js`; #3599, masking-records and mask-actions GUI tests).
- What is done with a value found is the user's per kind (#1977, `redactActions` in settings, `Options.Actions`, set by `POST /api/settings/redact-actions`; a kind left out is masked). `redact.Warn` sends it as written: the masker tells it, with a placeholder no one remembers, and it isn't counted as masked. `redact.Block` masks it as any other, and the gateway holds the request back with a 400 that names the kind and the value's hint, never the value (`blocked`, `maskBlocked` in [`redact.go`](../../internal/gateway/redact.go)); nothing is sent and the record is kept as `Blocked` (`noteBlocked`). A value on the *Never mask* list (`redactAllow`, `Options.Allow`, `POST /api/settings/redact-allow`) goes as written when it is the whole value found, in any case, and isn't told; a longer value it is part of is masked. Settings → Privacy's *Try it* posts a text to `POST /api/masking/preview` ([`masking.go`](../../internal/gui/masking.go)), which masks it as a request would be (`redact.Preview`) and remembers nothing: its placeholders stand for nothing afterwards, its values aren't masked later as seen before, and it returns only the masked text and each hit's kind, finder, action, placeholder and hint. The page keeps the text in no storage.
diff --git a/internal/gateway/codex_backend.go b/internal/gateway/codex_backend.go
index b12f61288..57b3629c2 100644
--- a/internal/gateway/codex_backend.go
+++ b/internal/gateway/codex_backend.go
@@ -880,7 +880,7 @@ func (s *Server) codexUpstreamOn(w http.ResponseWriter, r *http.Request, rest st
if r.Method == http.MethodPost {
var unmask func()
var err error
- w, body, unmask, maskedHits, err = redactedHits(w, body)
+ w, body, unmask, maskedHits, err = redactedHits(r.Context(), w, body)
defer unmask()
if err != nil {
writeError(w, provider.Responses, maskStatus(err), err.Error())
diff --git a/internal/gateway/codex_search.go b/internal/gateway/codex_search.go
index 555444959..3c894f0f8 100644
--- a/internal/gateway/codex_search.go
+++ b/internal/gateway/codex_search.go
@@ -123,7 +123,7 @@ func chatGPTSearchers(r *http.Request, model string) ([]provider.Provider, strin
// with why returned.
func (s *Server) alphaSearchOn(w http.ResponseWriter, r *http.Request, body []byte, model string, ons []provider.Provider) (bool, string) {
start := time.Now()
- w, body, unmask, hits, err := redactedHits(w, body)
+ w, body, unmask, hits, err := redactedHits(r.Context(), w, body)
defer unmask()
if err != nil {
return false, err.Error()
diff --git a/internal/gateway/decide.go b/internal/gateway/decide.go
index 8ac49e754..cd6814358 100644
--- a/internal/gateway/decide.go
+++ b/internal/gateway/decide.go
@@ -376,7 +376,7 @@ func (s *Server) askJev(p provider.Provider, model string, intents []string, pre
// remote Jev is none of those members
if !p.SkipsRedaction() {
var err error
- if text, prev.User, prev.Reply, err = maskedState(text, prev.User, prev.Reply); err != nil {
+ if text, prev.User, prev.Reply, err = maskedState(ctx, text, prev.User, prev.Reply); err != nil {
return verdict{}, err // fails closed: never sent unmasked
}
}
@@ -395,14 +395,14 @@ func (s *Server) askJev(p provider.Provider, model string, intents []string, pre
}
// maskedState masks what Jev is told of a turn, each as maskedText does.
-func maskedState(text, user, reply string) (string, string, string, error) {
+func maskedState(ctx context.Context, text, user, reply string) (string, string, string, error) {
out := [3]string{text, user, reply}
var all maskHits
for i, v := range out {
if v == "" {
continue
}
- _, m, h, err := maskTextHits(v)
+ _, m, h, err := maskTextHits(ctx, v)
if err != nil {
return "", "", "", err
}
diff --git a/internal/gateway/draw.go b/internal/gateway/draw.go
index 6ea8cf876..fa8de26e5 100644
--- a/internal/gateway/draw.go
+++ b/internal/gateway/draw.go
@@ -469,7 +469,7 @@ func (s *Server) images(edit bool) http.HandlerFunc {
}
var unmask func()
var hits maskHits
- w, d.Prompt, unmask, hits, err = redactedPromptHits(w, d.Prompt)
+ w, d.Prompt, unmask, hits, err = redactedPromptHits(r.Context(), w, d.Prompt)
defer unmask()
if err != nil {
fail(maskStatus(err), err.Error())
diff --git a/internal/gateway/embeddings.go b/internal/gateway/embeddings.go
index 9d4e6881c..17671a097 100644
--- a/internal/gateway/embeddings.go
+++ b/internal/gateway/embeddings.go
@@ -53,7 +53,7 @@ type retrieval interface {
// mask masks what goes to the vendors, and gives the writer that puts
// the masked values back in their answer, what it swapped, and the
// text it sent
- mask(w http.ResponseWriter) (http.ResponseWriter, func(), maskHits, string, error)
+ mask(ctx context.Context, w http.ResponseWriter) (http.ResponseWriter, func(), maskHits, string, error)
// forModel is the body and its Content-Type for the provider's own id
// of the model
forModel(model string) ([]byte, string, error)
@@ -65,11 +65,11 @@ type retrieval interface {
// jsonRetrieval is an embeddings or rerank body: a JSON object.
type jsonRetrieval struct{ req map[string]json.RawMessage }
-func (j *jsonRetrieval) mask(w http.ResponseWriter) (http.ResponseWriter, func(), maskHits, string, error) {
+func (j *jsonRetrieval) mask(ctx context.Context, w http.ResponseWriter) (http.ResponseWriter, func(), maskHits, string, error) {
// masked once, as the settings say: a document a reranker gives back
// has its secrets again
all, _ := json.Marshal(j.req)
- w, all, done, hits, err := redactedHits(w, all)
+ w, all, done, hits, err := redactedHits(ctx, w, all)
json.Unmarshal(all, &j.req)
return w, done, hits, string(all), err
}
@@ -180,7 +180,7 @@ func (s *Server) serveRetrieval(w http.ResponseWriter, r *http.Request, path, op
var err error
var hits maskHits
var sent string
- w, done, hits, sent, err = req.mask(w)
+ w, done, hits, sent, err = req.mask(r.Context(), w)
defer done()
if err != nil { // fails closed
call.Status, call.Error = maskStatus(err), err.Error()
diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.go
index 35e684ffa..0f1e908ab 100644
--- a/internal/gateway/gateway.go
+++ b/internal/gateway/gateway.go
@@ -1337,7 +1337,7 @@ func (s *Server) countTokens(w http.ResponseWriter, r *http.Request) {
}
// Count the same masked prompt that generation sends to the vendor.
plain := body
- w, body, unmask, err := redacted(w, body)
+ w, body, unmask, err := redacted(r.Context(), w, body)
defer unmask()
if err != nil {
writeError(w, provider.Anthropic, maskStatus(err), err.Error())
@@ -1676,7 +1676,7 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, from provider.Pro
// unmasked (unredactedRoute)
plain := body
var masked []byte // the masked body, when plain goes instead
- w, body, unmask, maskedHits, err := redactedHits(w, body)
+ w, body, unmask, maskedHits, err := redactedHits(r.Context(), w, body)
defer unmask()
if err != nil { // fails closed: never sent unmasked
log.Printf("gateway: %v", err)
diff --git a/internal/gateway/masking_wait_test.go b/internal/gateway/masking_wait_test.go
new file mode 100644
index 000000000..4f95dda44
--- /dev/null
+++ b/internal/gateway/masking_wait_test.go
@@ -0,0 +1,120 @@
+package gateway
+
+import (
+ "context"
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/yetone/magpie/internal/gwmw"
+ "github.com/yetone/magpie/internal/provider"
+ "github.com/yetone/magpie/internal/settings"
+)
+
+// waitingMasker is masking that waits for the local model, which never
+// answers: a pf-serve loading or busy with the requests ahead (#3682).
+// let is closed when the test ends; left is closed when a wait let go
+// because its request's context ended.
+type waitingMasker struct{ let, left chan struct{} }
+
+func (m waitingMasker) wait(ctx context.Context) {
+ select {
+ case <-ctx.Done():
+ close(m.left)
+ panic("stopped waiting for the local masking model")
+ case <-m.let:
+ panic("the test ended")
+ }
+}
+
+func (m waitingMasker) JSONHitsContext(ctx context.Context, b []byte) ([]byte, []gwmw.Hit) {
+ m.wait(ctx)
+ return b, nil
+}
+
+func (m waitingMasker) TextHitsContext(ctx context.Context, s string) (string, []gwmw.Hit) {
+ m.wait(ctx)
+ return s, nil
+}
+
+func (m waitingMasker) JSONHits(b []byte) ([]byte, []gwmw.Hit) {
+ return m.JSONHitsContext(context.Background(), b)
+}
+
+func (m waitingMasker) TextHits(s string) (string, []gwmw.Hit) {
+ return m.TextHitsContext(context.Background(), s)
+}
+func (m waitingMasker) JSON(b []byte) ([]byte, int) { b, _ = m.JSONHits(b); return b, 0 }
+func (m waitingMasker) Text(s string) (string, int) { s, _ = m.TextHits(s); return s, 0 }
+func (waitingMasker) Restore(s string, _ bool) string { return s }
+func (waitingMasker) Value(string) (string, bool) { return "", false }
+func (waitingMasker) Writer(w http.ResponseWriter) (http.ResponseWriter, func()) {
+ return w, func() {}
+}
+
+// #3682: with Settings → Privacy's local model on, every request after a
+// restart waited for it, and was answered after 2m that magpie "held this
+// request without routing it to any provider". Its answer says it waited
+// for the local masking model, what the model was doing and how to send
+// without it, and the wait lets go when magpie answers, rather than keep
+// its place in the model's queue.
+func TestHeldForTheMaskingModelSaysSo(t *testing.T) {
+ unroutedLimits(t, 50*time.Millisecond, 500*time.Millisecond)
+ fresh(t)
+ if err := settings.Save(settings.Settings{Redact: true}); err != nil {
+ t.Fatal(err)
+ }
+ if err := provider.Save(provider.Provider{ID: "probe", Name: "Probe", Chat: "http://127.0.0.1:9/v1", Key: "key", Models: []string{"text"}}); err != nil {
+ t.Fatal(err)
+ }
+ on, busy := modelOn, modelBusy
+ modelOn, modelBusy = func() bool { return true }, func() string { return "it is still loading" }
+ t.Cleanup(func() { modelOn, modelBusy = on, busy })
+ withoutMasker(t)
+ m := waitingMasker{let: make(chan struct{}), left: make(chan struct{})}
+ t.Cleanup(func() { close(m.let) })
+ remove := gwmw.Mask(m)
+ t.Cleanup(remove)
+
+ s := New()
+ srv := httptest.NewServer(s.Handler())
+ t.Cleanup(srv.Close)
+ c := &http.Client{Timeout: 20 * time.Second}
+ resp, err := c.Post(srv.URL+"/v1/chat/completions", "application/json", strings.NewReader(`{"model":"probe/text","messages":[{"role":"user","content":"hi"}]}`))
+ if err != nil {
+ t.Fatal(err)
+ }
+ b, _ := io.ReadAll(resp.Body)
+ resp.Body.Close()
+ body := string(b)
+ if resp.StatusCode != http.StatusGatewayTimeout || !strings.Contains(body, "waiting for the local masking model") ||
+ !strings.Contains(body, "it is still loading") || !strings.Contains(body, "turn it off in Settings") || strings.Contains(body, "without routing") {
+ t.Fatalf("held for the model: %d %s", resp.StatusCode, body)
+ }
+ select {
+ case <-m.left:
+ case <-time.After(10 * time.Second):
+ t.Fatal("the request's wait for the model went on after magpie answered it")
+ }
+ rs := routesOf(s, "probe/text")
+ if len(rs) != 1 || !strings.Contains(rs[0].Error, "waiting for the local masking model") {
+ t.Fatalf("its route: %+v", rs)
+ }
+}
+
+// With the model off, a request held while masked says it was held while
+// masking, not that it waited for a model that isn't on.
+func TestUnroutedMsgNamesTheModelOnlyWhenItIsOn(t *testing.T) {
+ on := modelOn
+ modelOn = func() bool { return false }
+ t.Cleanup(func() { modelOn = on })
+ if m := unroutedMsg(2*time.Minute, true); strings.Contains(m, "model") || !strings.Contains(m, "while masking it") {
+ t.Fatalf("model off: %s", m)
+ }
+ if m := unroutedMsg(2*time.Minute, false); !strings.Contains(m, "without routing it to any provider") {
+ t.Fatalf("not masking: %s", m)
+ }
+}
diff --git a/internal/gateway/middleware_off_test.go b/internal/gateway/middleware_off_test.go
index bb6a872ac..0d1ee5c8d 100644
--- a/internal/gateway/middleware_off_test.go
+++ b/internal/gateway/middleware_off_test.go
@@ -1,6 +1,7 @@
package gateway
import (
+ "context"
"encoding/json"
"io"
"net/http"
@@ -55,9 +56,13 @@ func (e *echoing) last() string {
}
// panicky is a masker that fails, on each way the gateway asks it to
-// mask (a Teller is asked through JSONHits and TextHits).
+// mask (a Teller is asked through JSONHits and TextHits, a ContextTeller
+// through JSONHitsContext and TextHitsContext, which masking.Masker has).
type panicky struct{ masking.Masker }
+func (panicky) JSONHitsContext(context.Context, []byte) ([]byte, []gwmw.Hit) { panic("masker broke") }
+func (panicky) TextHitsContext(context.Context, string) (string, []gwmw.Hit) { panic("masker broke") }
+
func (panicky) JSON([]byte) ([]byte, int) { panic("masker broke") }
func (panicky) JSONHits([]byte) ([]byte, []gwmw.Hit) { panic("masker broke") }
func (panicky) Text(string) (string, int) { panic("masker broke") }
@@ -134,17 +139,17 @@ func TestMaskedQueryFailsClosed(t *testing.T) {
t.Fatal(err)
}
const secret = "sk-proj-abcdEFGH1234ijklMNOP5678qrst"
- if q, err := maskedQuery("find "+secret, nil); err != nil || strings.Contains(q, secret) {
+ if q, err := maskedQuery(context.Background(), "find "+secret, nil); err != nil || strings.Contains(q, secret) {
t.Fatalf("on: %q %v", q, err)
}
withoutMasker(t)
gwmw.ForgetRefreshForTest()
- if q, err := maskedQuery("find "+secret, nil); err == nil {
+ if q, err := maskedQuery(context.Background(), "find "+secret, nil); err == nil {
t.Fatalf("off: sent %q", q)
}
// one the user set to go unmasked keeps it, plugin or not
p := &provider.Provider{ID: "local", Unredacted: true, Chat: "http://127.0.0.1:1/v1"}
- if q, err := maskedQuery("find "+secret, p); err != nil || q != "find "+secret {
+ if q, err := maskedQuery(context.Background(), "find "+secret, p); err != nil || q != "find "+secret {
t.Fatalf("off, to a provider set unmasked: %q %v", q, err)
}
}
diff --git a/internal/gateway/redact.go b/internal/gateway/redact.go
index 3052962fc..afb8528f6 100644
--- a/internal/gateway/redact.go
+++ b/internal/gateway/redact.go
@@ -1,6 +1,7 @@
package gateway
import (
+ "context"
"errors"
"fmt"
"net/http"
@@ -18,14 +19,15 @@ import (
// writes what the wrapper still holds. Nothing masked, nothing wrapped. It
// fails closed: masking wanted with no masker serving, or a masker that
// fails, is an error, and the request goes nowhere (gwmw.Masking).
-func redacted(w http.ResponseWriter, body []byte) (http.ResponseWriter, []byte, func(), error) {
- w, body, done, _, err := redactedHits(w, body)
+func redacted(ctx context.Context, w http.ResponseWriter, body []byte) (http.ResponseWriter, []byte, func(), error) {
+ w, body, done, _, err := redactedHits(ctx, w, body)
return w, body, done, err
}
// redactedHits is redacted with what it swapped, for the record of it
-// (noteMasked).
-func redactedHits(w http.ResponseWriter, body []byte) (http.ResponseWriter, []byte, func(), maskHits, error) {
+// (noteMasked). A masker that waits (for the local model's turn) stops
+// waiting when ctx ends, and the request goes nowhere.
+func redactedHits(ctx context.Context, w http.ResponseWriter, body []byte) (http.ResponseWriter, []byte, func(), maskHits, error) {
m, err := gwmw.Masking(redactionWanted())
if m == nil {
return w, body, func() {}, maskHits{}, err
@@ -35,7 +37,12 @@ func redactedHits(w http.ResponseWriter, body []byte) (http.ResponseWriter, []by
h maskHits
}
began := time.Now()
+ defer noteMasking(ctx)()
o, err := gwmw.Safely(func() out {
+ if t, ok := m.(gwmw.ContextTeller); ok {
+ b, hits := t.JSONHitsContext(ctx, body)
+ return out{b, hitsOf(hits)}
+ }
if t, ok := m.(gwmw.Teller); ok {
b, hits := t.JSONHits(body)
return out{b, hitsOf(hits)}
@@ -76,14 +83,14 @@ func toldBack(rw http.ResponseWriter, done func(), h *maskHits) func() {
// redactedPrompt masks a parsed image or video prompt before any vendor
// body is built, including multipart forms, and restores the reply's text.
// It fails closed as redacted does.
-func redactedPrompt(w http.ResponseWriter, prompt string) (http.ResponseWriter, string, func(), error) {
- w, masked, done, _, err := redactedPromptHits(w, prompt)
+func redactedPrompt(ctx context.Context, w http.ResponseWriter, prompt string) (http.ResponseWriter, string, func(), error) {
+ w, masked, done, _, err := redactedPromptHits(ctx, w, prompt)
return w, masked, done, err
}
// redactedPromptHits is redactedPrompt with what it swapped.
-func redactedPromptHits(w http.ResponseWriter, prompt string) (http.ResponseWriter, string, func(), maskHits, error) {
- m, masked, h, err := maskTextHits(prompt)
+func redactedPromptHits(ctx context.Context, w http.ResponseWriter, prompt string) (http.ResponseWriter, string, func(), maskHits, error) {
+ m, masked, h, err := maskTextHits(ctx, prompt)
if err != nil {
return w, prompt, func() {}, maskHits{}, err
}
@@ -97,14 +104,15 @@ func redactedPromptHits(w http.ResponseWriter, prompt string) (http.ResponseWrit
// maskedText is s masked as the settings say, and how many values it
// swapped; an error when it can't be (fail closed).
-func maskedText(s string) (string, int, error) {
- _, masked, h, err := maskTextHits(s)
+func maskedText(ctx context.Context, s string) (string, int, error) {
+ _, masked, h, err := maskTextHits(ctx, s)
return masked, h.n, err
}
// maskTextHits is maskedText with the masker that masked it and what it
-// swapped.
-func maskTextHits(s string) (gwmw.Masker, string, maskHits, error) {
+// swapped. Its wait for the local model ends with ctx, as redactedHits'
+// does.
+func maskTextHits(ctx context.Context, s string) (gwmw.Masker, string, maskHits, error) {
m, err := gwmw.Masking(redactionWanted())
if m == nil {
return nil, s, maskHits{}, err
@@ -114,7 +122,12 @@ func maskTextHits(s string) (gwmw.Masker, string, maskHits, error) {
h maskHits
}
began := time.Now()
+ defer noteMasking(ctx)()
o, err := gwmw.Safely(func() out {
+ if t, ok := m.(gwmw.ContextTeller); ok {
+ got, hits := t.TextHitsContext(ctx, s)
+ return out{got, hitsOf(hits)}
+ }
if t, ok := m.(gwmw.Teller); ok {
got, hits := t.TextHits(s)
return out{got, hitsOf(hits)}
@@ -212,11 +225,11 @@ func redactionWanted() bool {
// its query from a request already masked, but an agent's call to the
// search MCP carries it as the agent's model wrote it, values put back
// (lc on Discord). It fails closed as redacted does.
-func maskedQuery(query string, p *provider.Provider) (string, error) {
+func maskedQuery(ctx context.Context, query string, p *provider.Provider) (string, error) {
if p != nil && p.SkipsRedaction() {
return query, nil
}
- _, masked, h, err := maskTextHits(query)
+ _, masked, h, err := maskTextHits(ctx, query)
if err == nil {
id := ""
if p != nil {
diff --git a/internal/gateway/search_api.go b/internal/gateway/search_api.go
index 1425681c9..e8a0f295b 100644
--- a/internal/gateway/search_api.go
+++ b/internal/gateway/search_api.go
@@ -105,7 +105,7 @@ func SearcherUnused() string {
// apiSearch asks the search APIs, in their order, until one answers.
func (s *Server) apiSearch(ctx context.Context, query string) (string, []Hit, error) {
- query, err := maskedQuery(query, nil)
+ query, err := maskedQuery(ctx, query, nil)
if err != nil {
return "", nil, err
}
diff --git a/internal/gateway/search_kimi.go b/internal/gateway/search_kimi.go
index 0c68f6c28..d2a2f4157 100644
--- a/internal/gateway/search_kimi.go
+++ b/internal/gateway/search_kimi.go
@@ -103,7 +103,7 @@ func (s *Server) kimiSearch(ctx context.Context, p provider.Provider, query stri
// through the plan's own proxy, if it has one (#237, #1522)
ctx, cancel := context.WithTimeout(s.metered(p.Via(ctx), p, ""), searchTimeout/2)
defer cancel()
- q, err := maskedQuery(query, &p)
+ q, err := maskedQuery(ctx, query, &p)
if err != nil {
return "", nil, err
}
diff --git a/internal/gateway/search_workbuddy.go b/internal/gateway/search_workbuddy.go
index 8c5542c95..618eed692 100644
--- a/internal/gateway/search_workbuddy.go
+++ b/internal/gateway/search_workbuddy.go
@@ -55,7 +55,7 @@ func (s *Server) wbSearch(ctx context.Context, p provider.Provider, query string
// its client gives the service 20 s
ctx, cancel := context.WithTimeout(s.metered(p.Via(ctx), p, ""), searchTimeout/2)
defer cancel()
- q, err := maskedQuery(query, &p)
+ q, err := maskedQuery(ctx, query, &p)
if err != nil {
return "", nil, err
}
diff --git a/internal/gateway/transcribe.go b/internal/gateway/transcribe.go
index 1f0db4cf3..49d271d27 100644
--- a/internal/gateway/transcribe.go
+++ b/internal/gateway/transcribe.go
@@ -2,6 +2,7 @@ package gateway
import (
"bytes"
+ "context"
"encoding/json"
"errors"
"io"
@@ -93,11 +94,11 @@ func readAudioForm(ctype string, body []byte) (*audioForm, error) {
// mask masks the prompt, the form's only text a vendor is sent; the
// audio can't be, and the transcript gets the masked values back.
-func (a *audioForm) mask(w http.ResponseWriter) (http.ResponseWriter, func(), maskHits, string, error) {
+func (a *audioForm) mask(ctx context.Context, w http.ResponseWriter) (http.ResponseWriter, func(), maskHits, string, error) {
if a.prompt == "" {
return w, func() {}, maskHits{}, "", nil
}
- w, masked, done, hits, err := redactedPromptHits(w, a.prompt)
+ w, masked, done, hits, err := redactedPromptHits(ctx, w, a.prompt)
if masked != a.prompt {
a.masked = &masked
}
diff --git a/internal/gateway/unrouted.go b/internal/gateway/unrouted.go
index 456b62eb9..8102ed30a 100644
--- a/internal/gateway/unrouted.go
+++ b/internal/gateway/unrouted.go
@@ -16,6 +16,7 @@ import (
"time"
"github.com/yetone/magpie/internal/access"
+ "github.com/yetone/magpie/internal/masking"
"github.com/yetone/magpie/internal/provider"
)
@@ -174,6 +175,9 @@ type unroutedHold struct {
shown *Route // under f.mu
ended bool // under f.mu: answered by magpie
over bool // under f.mu: its handler returned
+ // masking counts the maskings under way for it: one held there waits
+ // for Settings → Privacy's local model, and its error says so
+ masking atomic.Int32
}
type unroutedKey struct{}
@@ -186,6 +190,17 @@ func noteAsked(ctx context.Context, model string) {
}
}
+// noteMasking tells ctx's watch its request is being masked, until the
+// func it answers is called.
+func noteMasking(ctx context.Context) func() {
+ u, _ := ctx.Value(unroutedKey{}).(*unroutedHold)
+ if u == nil {
+ return func() {}
+ }
+ u.masking.Add(1)
+ return func() { u.masking.Add(-1) }
+}
+
// watchUnrouted watches r, counted as f, until the func it answers is
// called as its handler returns.
func (s *Server) watchUnrouted(w http.ResponseWriter, r *http.Request, f *inflight) (http.ResponseWriter, *http.Request, func()) {
@@ -264,7 +279,10 @@ func (u *unroutedHold) giveUp() {
return
}
held := time.Since(f.start).Round(time.Second)
- msg := fmt.Sprintf("magpie held this request for %s without routing it to any provider, so it was stopped rather than left waiting. Send it again; if it keeps happening, restart magpie and report it with magpie's log, which says where it was held.", held)
+ msg, where := unroutedMsg(held, u.masking.Load() > 0), ""
+ if u.masking.Load() > 0 {
+ where = " masking it"
+ }
if !u.w.answer(unroutedProto(u.r.URL.Path), http.StatusGatewayTimeout, msg) {
f.mu.Unlock()
return // the handler is answering
@@ -280,9 +298,31 @@ func (u *unroutedHold) giveUp() {
})
u.cancel(errUnrouted)
asked, _ := u.asked.Load().(string)
- log.Printf("gateway: %s %s for %s from %s held %s before routing; stopped with 504%s", u.r.Method, u.r.URL.Path, cmpOr(asked, "?"), cmpOr(f.agent, "?"), held, unroutedStacks())
+ log.Printf("gateway: %s %s for %s from %s held %s before routing,%s; stopped with 504%s", u.r.Method, u.r.URL.Path, cmpOr(asked, "?"), cmpOr(f.agent, "?"), held, cmpOr(where, " not masking it"), unroutedStacks())
+}
+
+// unroutedMsg is the error of a request held for held before routing:
+// one held while it was masked, with Settings → Privacy's local model on,
+// waited for that model (#3682), and is told so, and how to send without
+// waiting for it.
+func unroutedMsg(held time.Duration, masking bool) string {
+ if masking && modelOn() {
+ why := ""
+ if b := modelBusy(); b != "" {
+ why = " (" + b + ")"
+ }
+ return fmt.Sprintf("magpie held this request for %s waiting for the local masking model (Settings → Privacy) to mask it%s, so it was stopped rather than sent masked by the rules alone. Send it again: what the model has already read is kept, so it goes quicker. To send requests without waiting for the model, turn it off in Settings → Privacy.", held, why)
+ }
+ if masking {
+ return fmt.Sprintf("magpie held this request for %s while masking it (Settings → Privacy), before routing it to any provider, so it was stopped rather than left waiting. Send it again; if it keeps happening, restart magpie and report it with magpie's log, which says where it was held.", held)
+ }
+ return fmt.Sprintf("magpie held this request for %s without routing it to any provider, so it was stopped rather than left waiting. Send it again; if it keeps happening, restart magpie and report it with magpie's log, which says where it was held.", held)
}
+// modelOn and modelBusy are internal/masking's, a test's
+// to stand in for.
+var modelOn, modelBusy = masking.ModelOn, masking.ModelBusy
+
func cmpOr(a, b string) string {
if a != "" {
return a
diff --git a/internal/gateway/video.go b/internal/gateway/video.go
index 6c13012fc..17808b106 100644
--- a/internal/gateway/video.go
+++ b/internal/gateway/video.go
@@ -732,13 +732,13 @@ func (s *Server) videosCreate(w http.ResponseWriter, r *http.Request) {
var unmask func()
prompt := f.Prompt
var hits maskHits
- w, f.Prompt, unmask, hits, err = redactedPromptHits(w, f.Prompt)
+ w, f.Prompt, unmask, hits, err = redactedPromptHits(r.Context(), w, f.Prompt)
defer unmask()
// the fields passed on as they came are masked as the prompt is: a
// negative_prompt is the user's text all the same
var extra maskHits
if err == nil {
- err = maskStrings(f.Extra, &extra)
+ err = maskStrings(r.Context(), f.Extra, &extra)
}
if err != nil { // fails closed
fail(maskStatus(err), err.Error())
@@ -1125,9 +1125,9 @@ func (s *Server) remoteVideoContent(ctx context.Context, w http.ResponseWriter,
// maskStrings masks every string in v (a decoded JSON value) in place as a
// prompt is masked, adding what it swapped to h; an error when it can't
// (fail closed).
-func maskStrings(v any, h *maskHits) error {
+func maskStrings(ctx context.Context, v any, h *maskHits) error {
one := func(str string) (string, error) {
- _, m, got, err := maskTextHits(str)
+ _, m, got, err := maskTextHits(ctx, str)
h.add(got)
return m, err
}
@@ -1138,7 +1138,7 @@ func maskStrings(v any, h *maskHits) error {
if str, ok := e.(string); ok {
x[k], err = one(str)
} else {
- err = maskStrings(e, h)
+ err = maskStrings(ctx, e, h)
}
if err != nil {
return err
@@ -1149,7 +1149,7 @@ func maskStrings(v any, h *maskHits) error {
if str, ok := e.(string); ok {
x[i], err = one(str)
} else {
- err = maskStrings(e, h)
+ err = maskStrings(ctx, e, h)
}
if err != nil {
return err
diff --git a/internal/gwmw/gwmw.go b/internal/gwmw/gwmw.go
index aea7d501e..b2f8f9167 100644
--- a/internal/gwmw/gwmw.go
+++ b/internal/gwmw/gwmw.go
@@ -16,6 +16,7 @@
package gwmw
import (
+ "context"
"fmt"
"net/http"
"slices"
@@ -121,6 +122,15 @@ type Teller interface {
Value(placeholder string) (string, bool)
}
+// ContextTeller is a Teller whose masking can wait (for a local model's
+// turn) and stops waiting when ctx ends: the client went away, or the
+// gateway gave up on the request. It then fails, as a masker that can't
+// mask does, and nothing is sent.
+type ContextTeller interface {
+ JSONHitsContext(ctx context.Context, body []byte) ([]byte, []Hit)
+ TextHitsContext(ctx context.Context, s string) (string, []Hit)
+}
+
var (
mu sync.Mutex
agents []*Agent
diff --git a/internal/masking/masking.go b/internal/masking/masking.go
index af1d1f0ca..439a3c19d 100644
--- a/internal/masking/masking.go
+++ b/internal/masking/masking.go
@@ -5,6 +5,8 @@
package masking
import (
+ "context"
+ "fmt"
"net/http"
"github.com/yetone/magpie/internal/gwmw"
@@ -21,25 +23,53 @@ func Wanted(st settings.Settings) bool {
return st.Redact || st.RedactPersonal || len(st.RedactWords) > 0
}
-func options() redact.Options {
+func options() redact.Options { return optionsCtx(context.Background()) }
+
+// optionsCtx is options with the model's wait ending with ctx.
+func optionsCtx(ctx context.Context) redact.Options {
st := settings.Shared()
o := redact.Options{Secrets: st.Redact, Personal: st.RedactPersonal, Kinds: st.RedactKinds, Words: st.RedactWords, WordsAnyCase: st.RedactWordsAnyCase, Rules: st.RedactRules,
Actions: st.RedactActions, Allow: st.RedactAllow, Shape: st.RedactShape}
if st.RedactModel {
- o.Model = model
+ o.Model = model(ctx)
}
return o
}
-// model is the local model (internal/pii) as redact's Model. One that
-// can't answer panics, and the request is held back (gwmw.Safely) rather
-// than sent with only the rules' masking: the user asked for the model's.
-func model(texts []string) [][]redact.Found {
- got, err := pii.Find(texts)
- if err != nil {
- panic(err)
+// model is the local model (internal/pii) as redact's Model, waited for
+// until ctx ends. One that can't answer, or isn't waited for any longer,
+// panics, and the request is held back (gwmw.Safely) rather than sent
+// with only the rules' masking: the user asked for the model's.
+func model(ctx context.Context) redact.Model {
+ return func(texts []string) [][]redact.Found {
+ got, err := pii.FindContext(ctx, texts)
+ if err != nil {
+ panic(err)
+ }
+ return got
+ }
+}
+
+// ModelOn says Settings → Privacy's local model masks requests.
+func ModelOn() bool { return settings.Shared().RedactModel }
+
+// ModelBusy says what the local model is doing that a request may wait
+// on, for the error of one that waited too long: "" when nothing.
+func ModelBusy() string {
+ st := pii.Now()
+ var what string
+ switch {
+ case st.Loading:
+ what = "it is still loading"
+ case st.Running:
+ what = "it is masking another request"
+ default:
+ return ""
+ }
+ if st.Waiting > 0 {
+ what += fmt.Sprintf(", with %d more waiting for it", st.Waiting)
}
- return got
+ return what
}
func (Masker) JSON(body []byte) ([]byte, int) {
@@ -53,8 +83,14 @@ func (Masker) JSON(body []byte) ([]byte, int) {
func (Masker) Text(s string) (string, int) { return redact.Mask(s, options()) }
// JSONHits is JSON, with what it masked (gwmw.Teller).
-func (Masker) JSONHits(body []byte) ([]byte, []gwmw.Hit) {
- o := options()
+func (m Masker) JSONHits(body []byte) ([]byte, []gwmw.Hit) {
+ return m.JSONHitsContext(context.Background(), body)
+}
+
+// JSONHitsContext is JSONHits, its wait for the model ending with ctx
+// (gwmw.ContextTeller).
+func (Masker) JSONHitsContext(ctx context.Context, body []byte) ([]byte, []gwmw.Hit) {
+ o := optionsCtx(ctx)
if !o.Secrets && !o.Personal && len(o.Words) == 0 {
return body, nil
}
@@ -65,8 +101,13 @@ func (Masker) JSONHits(body []byte) ([]byte, []gwmw.Hit) {
}
// TextHits is Text, with what it masked.
-func (Masker) TextHits(s string) (string, []gwmw.Hit) {
- o := options()
+func (m Masker) TextHits(s string) (string, []gwmw.Hit) {
+ return m.TextHitsContext(context.Background(), s)
+}
+
+// TextHitsContext is TextHits, its wait for the model ending with ctx.
+func (Masker) TextHitsContext(ctx context.Context, s string) (string, []gwmw.Hit) {
+ o := optionsCtx(ctx)
var hits []gwmw.Hit
o.Seen = told(&hits)
out, _ := redact.Mask(s, o)
@@ -105,4 +146,7 @@ func (Masker) Writer(w http.ResponseWriter) (http.ResponseWriter, func()) {
return rw, rw.Finish
}
-var _ gwmw.Teller = Masker{}
+var (
+ _ gwmw.Teller = Masker{}
+ _ gwmw.ContextTeller = Masker{}
+)
diff --git a/internal/pii/cache.go b/internal/pii/cache.go
new file mode 100644
index 000000000..fb96284e3
--- /dev/null
+++ b/internal/pii/cache.go
@@ -0,0 +1,184 @@
+package pii
+
+import (
+ "bufio"
+ "bytes"
+ "encoding/binary"
+ "errors"
+ "io"
+ "os"
+ "path/filepath"
+ "sync"
+ "time"
+
+ "github.com/yetone/magpie/internal/redact"
+)
+
+// What Find learned is kept on disk too, so that a restart doesn't ask the
+// model about every history again: after one, each request's whole
+// history was a miss, and on a CPU the requests queued for minutes behind
+// each other (#3682). The file holds a text's sha256, and where in it the
+// model found what, with the label: never the text. It is named for the
+// model it came from, so another model's answers are never read as this
+// one's, and lives with the model, in magpie's cache (Remove deletes it).
+
+// cacheMagic starts the file; a file that doesn't start with it, or ends
+// part way, is read as empty.
+const cacheMagic = "pfound01"
+
+func cachePath() string { return filepath.Join(dir(), "found-"+modelSHA[:12]+".bin") }
+
+// saveAfter is how long after an answer the file is written: one write
+// for a burst of requests.
+var saveAfter = 2 * time.Second
+
+var disk struct {
+ mu sync.Mutex
+ loaded string // the file read into cache, once
+ timer *time.Timer
+ saving sync.Mutex
+}
+
+// loadCache reads the file into cache, the first time Find is asked.
+func loadCache() {
+ disk.mu.Lock()
+ defer disk.mu.Unlock()
+ p := cachePath()
+ if disk.loaded == p {
+ return
+ }
+ disk.loaded = p
+ b, err := os.ReadFile(p)
+ if err != nil {
+ return
+ }
+ es, err := decodeCache(b)
+ if err != nil {
+ return // a file cut short or not ours: nothing is kept, the model asked again
+ }
+ cache.mu.Lock()
+ defer cache.mu.Unlock()
+ for _, e := range es {
+ if _, ok := cache.m[e.k]; !ok {
+ cache.put(e.k, e.fs)
+ }
+ }
+}
+
+// saveSoon writes the file saveAfter from now, unless a write is due.
+func saveSoon() {
+ disk.mu.Lock()
+ defer disk.mu.Unlock()
+ if disk.timer == nil {
+ disk.timer = time.AfterFunc(saveAfter, func() {
+ disk.mu.Lock()
+ disk.timer = nil
+ disk.mu.Unlock()
+ saveCache()
+ })
+ }
+}
+
+// saveCache writes what cache holds, oldest first, to a file beside the
+// old one and renames it over it: a reader never sees half of it.
+func saveCache() error {
+ disk.saving.Lock()
+ defer disk.saving.Unlock()
+ cache.mu.Lock()
+ b := encodeCache(cache)
+ cache.mu.Unlock()
+ p := cachePath()
+ f, err := os.CreateTemp(filepath.Dir(p), "found-*.tmp")
+ if err != nil {
+ return err
+ }
+ _, err = f.Write(b)
+ if err == nil {
+ err = f.Sync()
+ }
+ if cerr := f.Close(); err == nil {
+ err = cerr
+ }
+ if err == nil {
+ err = os.Rename(f.Name(), p)
+ }
+ if err != nil {
+ os.Remove(f.Name())
+ }
+ return err
+}
+
+// encodeCache is c, oldest first: the magic, a count, then each entry's
+// hash, its spans' count and each span's start, end and label.
+func encodeCache(c *lru) []byte {
+ var b bytes.Buffer
+ b.WriteString(cacheMagic)
+ put := func(v any) { binary.Write(&b, binary.LittleEndian, v) }
+ put(uint32(c.ll.Len()))
+ for e := c.ll.Back(); e != nil; e = e.Prev() {
+ en := e.Value.(*entry)
+ b.Write(en.k[:])
+ put(uint16(len(en.fs)))
+ for _, f := range en.fs {
+ label := f.Label
+ if len(label) > 255 {
+ label = label[:255]
+ }
+ put(uint32(f.Start))
+ put(uint32(f.End))
+ b.WriteByte(byte(len(label)))
+ b.WriteString(label)
+ }
+ }
+ return b.Bytes()
+}
+
+var errBadCache = errors.New("not a cache file")
+
+func decodeCache(b []byte) ([]entry, error) {
+ if !bytes.HasPrefix(b, []byte(cacheMagic)) {
+ return nil, errBadCache
+ }
+ r := bufio.NewReader(bytes.NewReader(b[len(cacheMagic):]))
+ var n uint32
+ if err := binary.Read(r, binary.LittleEndian, &n); err != nil {
+ return nil, err
+ }
+ if n > uint32(len(b)/34) { // each entry is 34 bytes at least
+ return nil, errBadCache
+ }
+ out := make([]entry, 0, n)
+ for i := uint32(0); i < n; i++ {
+ var e entry
+ if _, err := io.ReadFull(r, e.k[:]); err != nil {
+ return nil, err
+ }
+ var spans uint16
+ if err := binary.Read(r, binary.LittleEndian, &spans); err != nil {
+ return nil, err
+ }
+ for j := uint16(0); j < spans; j++ {
+ var se [2]uint32
+ if err := binary.Read(r, binary.LittleEndian, &se); err != nil {
+ return nil, err
+ }
+ l, err := r.ReadByte()
+ if err != nil {
+ return nil, err
+ }
+ label := make([]byte, l)
+ if _, err := io.ReadFull(r, label); err != nil {
+ return nil, err
+ }
+ if se[0] >= se[1] {
+ return nil, errBadCache
+ }
+ e.fs = append(e.fs, redact.Found{Start: int(se[0]), End: int(se[1]), Label: string(label)})
+ }
+ out = append(out, e)
+ }
+ if _, err := r.ReadByte(); err != io.EOF {
+ return nil, errBadCache // more than it said
+ }
+ return out, nil
+}
diff --git a/internal/pii/fetch.go b/internal/pii/fetch.go
index 70fcfd809..66cb2fac5 100644
--- a/internal/pii/fetch.go
+++ b/internal/pii/fetch.go
@@ -139,7 +139,7 @@ func upgrade() {
if err := ensureServe(ctx); err != nil {
return
}
- Stop()
+ stopWhenFree()
removeOld()
}
diff --git a/internal/pii/find.go b/internal/pii/find.go
index 106cb0908..b63ed112e 100644
--- a/internal/pii/find.go
+++ b/internal/pii/find.go
@@ -2,7 +2,9 @@ package pii
import (
"container/list"
+ "context"
"crypto/sha256"
+ "fmt"
"sync"
"github.com/yetone/magpie/internal/redact"
@@ -10,42 +12,105 @@ import (
// Find is redact's Model: what the model finds in each of texts. A
// request's history comes back turn after turn, so what was found in a
-// text is kept (by its hash) and the model is asked only about texts it
-// hasn't seen. Those go to it joined into chunks of up to chunkSize: one
-// pass over a few joined texts takes about what one over a single short
-// text does (100 short messages: 683 ms one by one, 44 ms joined, on an
-// M4 Pro's GPU).
+// text is kept (by its hash, on disk too: cache.go) and the model is
+// asked only about texts it hasn't seen. Those go to it joined into
+// chunks of up to chunkSize: one pass over a few joined texts takes about
+// what one over a single short text does (100 short messages: 683 ms one
+// by one, 44 ms joined, on an M4 Pro's GPU).
func Find(texts []string) ([][]redact.Found, error) {
- out := make([][]redact.Found, len(texts))
- var miss []int
+ return FindContext(context.Background(), texts)
+}
+
+// FindContext is Find for a caller that may stop waiting: when ctx ends
+// (the client went away, the gateway gave up on the request) it returns
+// at once, whether its turn hadn't come or pf-serve is on it. A turn
+// begun is finished all the same and its answers kept, so the request
+// sent again finds them.
+func FindContext(ctx context.Context, texts []string) ([][]redact.Found, error) {
+ loadCache()
+ out, miss := cached(texts)
+ if len(miss) == 0 {
+ return out, nil
+ }
+ if ctx.Err() != nil {
+ return nil, waitErr(ctx)
+ }
+ waiting.Add(1)
+ select {
+ case turn <- struct{}{}:
+ waiting.Add(-1)
+ case <-ctx.Done():
+ waiting.Add(-1)
+ return nil, waitErr(ctx)
+ }
+ type result struct {
+ out [][]redact.Found
+ err error
+ }
+ done := make(chan result, 1)
+ go func() {
+ defer func() { <-turn }()
+ // read again: the requests before this one in turn had most of
+ // a history in common with it, and asked already
+ out, miss := cached(texts)
+ if len(miss) > 0 {
+ chunks, parts := join(texts, miss)
+ got, err := classify(chunks)
+ if err != nil {
+ done <- result{nil, err}
+ return
+ }
+ for c, ps := range parts {
+ for _, p := range ps {
+ out[p.text] = within(got[c], p.at, p.at+len(texts[p.text]))
+ }
+ }
+ cache.mu.Lock()
+ for _, i := range miss {
+ cache.put(sha256.Sum256([]byte(texts[i])), out[i])
+ }
+ cache.mu.Unlock()
+ saveSoon()
+ }
+ done <- result{out, nil}
+ }()
+ select {
+ case r := <-done:
+ return r.out, r.err
+ case <-ctx.Done():
+ return nil, waitErr(ctx)
+ }
+}
+
+// waitErr is a request's that stopped waiting for the model.
+func waitErr(ctx context.Context) error {
+ return fmt.Errorf("stopped waiting for the local masking model: %w", context.Cause(ctx))
+}
+
+// cached is what the cache has of texts, and which it hasn't.
+func cached(texts []string) (out [][]redact.Found, miss []int) {
+ out = make([][]redact.Found, len(texts))
cache.mu.Lock()
+ defer cache.mu.Unlock()
for i, t := range texts {
- if fs, ok := cache.get(sha256.Sum256([]byte(t))); ok {
+ if fs, ok := cache.get(sha256.Sum256([]byte(t))); ok && fits(fs, t) {
out[i] = fs
} else {
miss = append(miss, i)
}
}
- cache.mu.Unlock()
- if len(miss) == 0 {
- return out, nil
- }
- chunks, parts := join(texts, miss)
- got, err := classify(chunks)
- if err != nil {
- return nil, err
- }
- for c, ps := range parts {
- for _, p := range ps {
- out[p.text] = within(got[c], p.at, p.at+len(texts[p.text]))
+ return out, miss
+}
+
+// fits says every span of fs is inside t: one read from a damaged file
+// that isn't is asked about again.
+func fits(fs []redact.Found, t string) bool {
+ for _, f := range fs {
+ if f.Start < 0 || f.End > len(t) || f.Start >= f.End {
+ return false
}
}
- cache.mu.Lock()
- for _, i := range miss {
- cache.put(sha256.Sum256([]byte(texts[i])), out[i])
- }
- cache.mu.Unlock()
- return out, nil
+ return true
}
// chunkSize is how long a chunk of joined texts grows; a longer text
diff --git a/internal/pii/main_test.go b/internal/pii/main_test.go
index d127e8271..de90733e5 100644
--- a/internal/pii/main_test.go
+++ b/internal/pii/main_test.go
@@ -37,6 +37,8 @@ const (
// ok as pf-serve does
// gpudies on the GPU, ends at its first request
// hang never answers a request
+// loadhang never says the model is loaded
+// slow answers each request after 300ms
// noload says it couldn't load the model
// novulkan as pf-serve-vulkan, says Vulkan is missing; else ok
func fakeServe(mode string) {
@@ -62,6 +64,9 @@ func fakeServe(mode string) {
json.NewEncoder(os.Stdout).Encode(map[string]string{"error": "can't load " + mode})
os.Exit(1)
}
+ if mode == "loadhang" {
+ time.Sleep(time.Hour)
+ }
json.NewEncoder(os.Stdout).Encode(map[string]any{"device": device, "gpu": gpu, "ms": 7})
in := bufio.NewReader(os.Stdin)
out := bufio.NewWriter(os.Stdout)
@@ -90,6 +95,8 @@ func fakeServe(mode string) {
os.Exit(3)
case mode == "hang":
time.Sleep(time.Hour)
+ case mode == "slow":
+ time.Sleep(300 * time.Millisecond)
}
put(uint32(0))
for _, t := range texts {
@@ -120,11 +127,23 @@ func fakeServe(mode string) {
// test's own; it gives the file the stand-in logs to.
func fakeInstall(t *testing.T, mode string, vulkan bool) string {
t.Helper()
+ upgrades.Wait() // one started before reads dir
d := t.TempDir()
was, wasSize, wasIdle := dir, modelSize, idleAfter
dir, modelSize = func() string { return d }, 4
t.Cleanup(func() {
Stop()
+ // a request Stop failed lets its turn go
+ turn <- struct{}{}
+ <-turn
+ upgrades.Wait()
+ disk.mu.Lock()
+ if disk.timer != nil {
+ disk.timer.Stop()
+ disk.timer = nil
+ }
+ disk.loaded = ""
+ disk.mu.Unlock()
srv.mu.Lock()
srv.cpu, srv.gpuLost, srv.failed = false, "", ""
srv.mu.Unlock()
diff --git a/internal/pii/pii.go b/internal/pii/pii.go
index d92a473c6..6171a634e 100644
--- a/internal/pii/pii.go
+++ b/internal/pii/pii.go
@@ -126,6 +126,10 @@ type Status struct {
GPUError string `json:"gpuError,omitempty"`
// why the last request got no answer
Failed string `json:"failed,omitempty"`
+ // pf-serve is loading the model, and how many requests wait for it
+ // (or for the one it is on)
+ Loading bool `json:"loading,omitempty"`
+ Waiting int `json:"waiting,omitempty"`
}
// Now is the model's status.
@@ -139,11 +143,12 @@ func Now() Status {
if srv.h != nil {
s.Running, s.Device, s.GPU, s.LoadMS, s.GPUError = true, srv.h.hello.Device, srv.h.hello.GPU, srv.h.hello.MS, srv.h.hello.GPUError
}
- s.Failed = srv.failed
+ s.Failed, s.Loading = srv.failed, srv.loading
if s.GPUError == "" {
s.GPUError = srv.gpuLost
}
srv.mu.Unlock()
+ s.Waiting = int(waiting.Load())
return s
}
diff --git a/internal/pii/serve.go b/internal/pii/serve.go
index a43d6d26f..549eeb549 100644
--- a/internal/pii/serve.go
+++ b/internal/pii/serve.go
@@ -12,6 +12,7 @@ import (
"os/exec"
"strings"
"sync"
+ "sync/atomic"
"time"
"github.com/yetone/magpie/internal/proc"
@@ -37,8 +38,13 @@ type helper struct {
exited chan struct{}
}
-// srv is the pf-serve magpie runs, one at a time: its requests take
-// turns.
+// srv is the pf-serve magpie runs, one at a time. Its requests take
+// turns on its one pipe: turn is held for each, from start to answer, and
+// a request's caller can stop waiting for it (FindContext). mu guards
+// only the fields below, and is never held while pf-serve loads or
+// answers: the status (Now), Stop and the idle timer took it, and on a
+// cold start every request and the Settings page waited minutes behind
+// one load (#3682).
var srv struct {
mu sync.Mutex
h *helper
@@ -49,6 +55,25 @@ var srv struct {
gpuLost string
// failed is why the last request got no answer, until one does
failed string
+ // loading is set while pf-serve loads the model
+ loading bool
+ // stop is closed by Stop: a load under way gives up, and the request
+ // it was for fails rather than start pf-serve again
+ stop chan struct{}
+}
+
+// turn is pf-serve's pipe, held by the one request on it.
+var turn = make(chan struct{}, 1)
+
+// waiting counts the requests waiting for their turn.
+var waiting atomic.Int32
+
+// stopCh is srv.stop, made the first time; mu is held.
+func stopCh() chan struct{} {
+ if srv.stop == nil {
+ srv.stop = make(chan struct{})
+ }
+ return srv.stop
}
// loadTimeout is how long pf-serve has to load the model: 1.6 GB read
@@ -76,23 +101,37 @@ func candidates(cpu bool) []string {
}
// start runs pf-serve, the first of candidates that loads the model.
-func start(cpu bool) (*helper, error) {
+// stop ends a load under way.
+func start(cpu bool, stop <-chan struct{}) (*helper, error) {
if !Installed() {
return nil, ErrNotInstalled
}
- go upgrade()
+ upgrades.Add(1)
+ go func() {
+ defer upgrades.Done()
+ upgrade()
+ }()
var errs []error
for _, path := range candidates(cpu) {
- h, err := startOne(path, cpu)
+ h, err := startOne(path, cpu, stop)
if err == nil {
return h, nil
}
+ if errors.Is(err, errStopped) {
+ return nil, err
+ }
errs = append(errs, err)
}
return nil, fmt.Errorf("the local masking model didn't start: %w", errors.Join(errs...))
}
-func startOne(path string, cpu bool) (*helper, error) {
+// upgrades are the upgrade checks start set going, for a test to wait for.
+var upgrades sync.WaitGroup
+
+// errStopped is a request's when Stop ended pf-serve under it.
+var errStopped = errors.New("the local masking model was stopped")
+
+func startOne(path string, cpu bool, stop <-chan struct{}) (*helper, error) {
cmd := proc.Command(path, modelPath())
// Without GGML_METAL_NO_RESIDENCY, ggml's Metal residency set keeps a
// heartbeat that woke an idle pf-serve ~146 times a second for the 10
@@ -132,6 +171,9 @@ func startOne(path string, cpu bool) (*helper, error) {
case <-time.After(loadTimeout):
h.kill()
return nil, fmt.Errorf("%s: the model didn't load in %s", baseName(path), loadTimeout)
+ case <-stop:
+ h.kill()
+ return nil, errStopped
}
if err := json.Unmarshal([]byte(s), &h.hello); err != nil || h.hello.Error != "" || h.hello.Device == "" {
h.kill()
@@ -178,84 +220,176 @@ func (h *helper) kill() {
// where it runs.
func Warm() {
if Installed() {
- go classify([]string{"warm"})
+ go func() {
+ turn <- struct{}{}
+ defer func() { <-turn }()
+ classify([]string{"warm"})
+ }()
}
}
-// Stop ends pf-serve, if it runs.
+// Stop ends pf-serve, if it runs, and a load under way, at once: it
+// waits for no request's turn. A request on it fails, and pf-serve is
+// killed rather than given the 5s an idle one has to end.
func Stop() {
+ free := false
+ select {
+ case turn <- struct{}{}:
+ free = true
+ defer func() { <-turn }()
+ default:
+ }
srv.mu.Lock()
- defer srv.mu.Unlock()
- stopLocked()
+ h := detachLocked()
+ if srv.stop != nil {
+ close(srv.stop)
+ srv.stop = nil
+ }
+ srv.mu.Unlock()
+ if h != nil && !free {
+ h.kill()
+ return
+ }
+ end(h)
+}
+
+// stopWhenFree ends pf-serve once no request is on it, so that the next
+// one starts the pf-serve on disk now (upgrade).
+func stopWhenFree() {
+ turn <- struct{}{}
+ defer func() { <-turn }()
+ srv.mu.Lock()
+ h := detachLocked()
+ srv.mu.Unlock()
+ end(h)
}
-func stopLocked() {
+// detachLocked takes pf-serve out of srv, for end; mu is held.
+func detachLocked() *helper {
if srv.idle != nil {
srv.idle.Stop()
srv.idle = nil
}
- if h := srv.h; h != nil {
- srv.h = nil
- // stdin ending ends it; one that doesn't is killed
- h.in.Close()
- select {
- case <-h.exited:
- case <-time.After(5 * time.Second):
- h.kill()
- }
+ h := srv.h
+ srv.h = nil
+ return h
+}
+
+// end ends h, with no lock held: stdin ending ends it, and one that
+// doesn't is killed.
+func end(h *helper) {
+ if h == nil {
+ return
+ }
+ h.in.Close()
+ select {
+ case <-h.exited:
+ case <-time.After(5 * time.Second):
+ h.kill()
}
}
// classify asks pf-serve, started if it isn't, what is in texts. One that
// fails is ended and started again; one that failed on the GPU is started
-// on the CPU, for good.
+// on the CPU, for good. The caller holds turn.
func classify(texts []string) ([][]redact.Found, error) {
+ out, err := classifyTurn(texts)
srv.mu.Lock()
- defer srv.mu.Unlock()
- out, err := classifyLocked(texts)
srv.failed = ""
if err != nil {
srv.failed = err.Error()
}
+ srv.mu.Unlock()
return out, err
}
-func classifyLocked(texts []string) ([][]redact.Found, error) {
+func classifyTurn(texts []string) ([][]redact.Found, error) {
var last error
for try := 0; try < 2; try++ {
- if srv.h == nil {
- h, err := start(srv.cpu)
+ srv.mu.Lock()
+ h, cpu, stop := srv.h, srv.cpu, stopCh()
+ if h == nil {
+ srv.loading = true
+ }
+ srv.mu.Unlock()
+ if h == nil {
+ var err error
+ h, err = start(cpu, stop)
+ srv.mu.Lock()
+ srv.loading = false
+ if err == nil && stopped(stop) {
+ err = errStopped
+ }
if err != nil {
+ srv.mu.Unlock()
+ if h != nil {
+ h.kill()
+ }
return nil, err
}
srv.h = h
+ srv.mu.Unlock()
}
- out, err := srv.h.classify(texts)
+ out, err := h.classify(texts)
+ srv.mu.Lock()
if err == nil {
- if srv.idle != nil {
- srv.idle.Stop()
+ if srv.h == h {
+ idleLocked(h)
}
- h := srv.h
- srv.idle = time.AfterFunc(idleAfter, func() {
- srv.mu.Lock()
- defer srv.mu.Unlock()
- if srv.h == h {
- stopLocked()
- }
- })
+ srv.mu.Unlock()
return out, nil
}
last = err
- gpu := srv.h.hello.GPU
- srv.h.kill()
- srv.h = nil
- if gpu {
+ if srv.h == h {
+ srv.h = nil
+ }
+ if stopped(stop) {
+ srv.mu.Unlock()
+ h.kill()
+ return nil, errStopped
+ }
+ if h.hello.GPU {
srv.cpu, srv.gpuLost = true, err.Error()
}
+ srv.mu.Unlock()
+ h.kill()
}
return nil, fmt.Errorf("the local masking model failed: %w", last)
}
+func stopped(stop <-chan struct{}) bool {
+ select {
+ case <-stop:
+ return true
+ default:
+ return false
+ }
+}
+
+// idleLocked ends h after idleAfter with no request; mu is held. The timer
+// takes the turn only when it is free: a request on it sets the timer
+// again when it is answered.
+func idleLocked(h *helper) {
+ if srv.idle != nil {
+ srv.idle.Stop()
+ }
+ srv.idle = time.AfterFunc(idleAfter, func() {
+ select {
+ case turn <- struct{}{}:
+ default:
+ return
+ }
+ defer func() { <-turn }()
+ srv.mu.Lock()
+ var gone *helper
+ if srv.h == h {
+ gone = detachLocked()
+ }
+ srv.mu.Unlock()
+ end(gone)
+ })
+}
+
// classify is one request: n, then each text's length and bytes; the
// reply is a status, then each text's spans (pf-serve.cpp says how).
func (h *helper) classify(texts []string) ([][]redact.Found, error) {
diff --git a/internal/pii/turn_test.go b/internal/pii/turn_test.go
new file mode 100644
index 000000000..80d272409
--- /dev/null
+++ b/internal/pii/turn_test.go
@@ -0,0 +1,237 @@
+package pii
+
+import (
+ "context"
+ "crypto/sha256"
+ "errors"
+ "os"
+ "reflect"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/yetone/magpie/internal/redact"
+)
+
+// waitFor waits for ok, up to 10s.
+func waitFor(t *testing.T, what string, ok func() bool) {
+ t.Helper()
+ for deadline := time.Now().Add(10 * time.Second); !ok(); time.Sleep(10 * time.Millisecond) {
+ if time.Now().After(deadline) {
+ t.Fatalf("%s: never", what)
+ }
+ }
+}
+
+// within10s runs f, and fails the test when it hasn't returned in 10s: what
+// it waits on is a request stuck for an hour.
+func within10s[T any](t *testing.T, what string, f func() T) T {
+ t.Helper()
+ got := make(chan T, 1)
+ go func() { got <- f() }()
+ select {
+ case v := <-got:
+ return v
+ case <-time.After(10 * time.Second):
+ t.Fatalf("%s waited for the request pf-serve is stuck on", what)
+ panic("unreachable")
+ }
+}
+
+type found struct {
+ fs [][]redact.Found
+ err error
+}
+
+func findAsync(ctx context.Context, texts ...string) chan found {
+ c := make(chan found, 1)
+ go func() {
+ fs, err := FindContext(ctx, texts)
+ c <- found{fs, err}
+ }()
+ return c
+}
+
+// #3682: a request pf-serve took long over held the lock the status and
+// Stop take, and the Settings page and turning the model off waited with
+// every request behind it. Now neither waits: the status says it runs,
+// and Stop ends it, the request on it failing.
+func TestStatusAndStopDontWaitForARequest(t *testing.T) {
+ log := fakeInstall(t, "hang", false)
+ was := classifyTimeout
+ classifyTimeout = func(int) time.Duration { return time.Hour }
+ t.Cleanup(func() { classifyTimeout = was })
+ stuck := findAsync(context.Background(), "hi Alice")
+ waitFor(t, "the request reaching pf-serve", func() bool { return len(fakeLines(t, log, "request")) == 1 })
+ if st := within10s(t, "Now", Now); !st.Running || st.Loading {
+ t.Fatalf("Now = %+v, want running", st)
+ }
+ within10s(t, "Stop", func() bool { Stop(); return true })
+ r := within10s(t, "the stopped request", func() found { return <-stuck })
+ if !errors.Is(r.err, errStopped) {
+ t.Fatalf("the request on a stopped pf-serve: %v, want errStopped", r.err)
+ }
+ if n := len(fakeLines(t, log, "start")); n != 1 {
+ t.Fatalf("started %d times, want once: a stopped one isn't started again", n)
+ }
+}
+
+// The same while pf-serve loads the model (up to loadTimeout, 3m): the
+// status says it is loading, and Stop ends the load.
+func TestStatusAndStopDontWaitForALoad(t *testing.T) {
+ log := fakeInstall(t, "loadhang", false)
+ loading := findAsync(context.Background(), "hi Alice")
+ waitFor(t, "pf-serve starting", func() bool { return len(fakeLines(t, log, "start")) == 1 })
+ if st := within10s(t, "Now", Now); !st.Loading || st.Running {
+ t.Fatalf("Now = %+v, want loading", st)
+ }
+ within10s(t, "Stop", func() bool { Stop(); return true })
+ if r := within10s(t, "the request", func() found { return <-loading }); !errors.Is(r.err, errStopped) {
+ t.Fatalf("the request whose load was stopped: %v", r.err)
+ }
+}
+
+// A request waiting for its turn leaves when its context ends (its client
+// went away, or the gateway answered it 504), not when the one ahead of
+// it is done; one on pf-serve too, while its turn is finished all the
+// same.
+func TestWaiterLeavesWhenItsContextEnds(t *testing.T) {
+ log := fakeInstall(t, "hang", false)
+ was := classifyTimeout
+ classifyTimeout = func(int) time.Duration { return time.Hour }
+ t.Cleanup(func() { classifyTimeout = was })
+ ahead, cancelAhead := context.WithCancel(context.Background())
+ first := findAsync(ahead, "hi Alice")
+ waitFor(t, "the first request reaching pf-serve", func() bool { return len(fakeLines(t, log, "request")) == 1 })
+
+ ctx, cancel := context.WithCancelCause(context.Background())
+ second := findAsync(ctx, "another Alice")
+ waitFor(t, "the second request waiting", func() bool { return Now().Waiting == 1 })
+ gaveUp := errors.New("gave up")
+ cancel(gaveUp)
+ r := within10s(t, "the request whose context ended", func() found { return <-second })
+ if !errors.Is(r.err, gaveUp) || !strings.Contains(r.err.Error(), "waiting for the local masking model") {
+ t.Fatalf("the waiter: %v", r.err)
+ }
+ if n := Now().Waiting; n != 0 {
+ t.Fatalf("Waiting = %d after the waiter left", n)
+ }
+ cancelAhead()
+ if r := within10s(t, "the request on pf-serve", func() found { return <-first }); !errors.Is(r.err, context.Canceled) {
+ t.Fatalf("the request on pf-serve: %v", r.err)
+ }
+ if n := len(fakeLines(t, log, "request")); n != 1 {
+ t.Fatalf("pf-serve was asked %d times, want once: the waiter never got its turn", n)
+ }
+}
+
+// Requests queued behind one another after a restart had most of their
+// history in common: each was asked about all over again when its turn
+// came. Now one reads the cache again when its turn comes.
+func TestQueuedRequestReadsWhatTheOneAheadLearned(t *testing.T) {
+ log := fakeInstall(t, "slow", false)
+ history := []string{"hi Alice", "nothing here", "Bob and Alice"}
+ first := findAsync(context.Background(), history...)
+ waitFor(t, "the first request reaching pf-serve", func() bool { return len(fakeLines(t, log, "request")) == 1 })
+ second := findAsync(context.Background(), history...)
+ a, b := <-first, <-second
+ if a.err != nil || b.err != nil || !reflect.DeepEqual(a.fs, b.fs) {
+ t.Fatalf("answers: %v %v / %v %v", a.fs, a.err, b.fs, b.err)
+ }
+ if reqs := fakeLines(t, log, "request"); len(reqs) != 1 {
+ t.Fatalf("pf-serve was asked %d times, want once: %q", len(reqs), reqs)
+ }
+}
+
+// restart is magpie started again: nothing in memory, pf-serve not running.
+func restart(t *testing.T) {
+ t.Helper()
+ Stop()
+ cache.mu.Lock()
+ cache.ll.Init()
+ clear(cache.m)
+ cache.mu.Unlock()
+ disk.mu.Lock()
+ disk.loaded = ""
+ disk.mu.Unlock()
+}
+
+// #3682: after a restart every request's whole history went to the model
+// again, one request after another, minutes of it on a CPU. What was found
+// is kept on disk, by hash, without the text, and read back.
+func TestRestartReadsWhatWasFound(t *testing.T) {
+ log := fakeInstall(t, "ok", false)
+ saveAfter = 10 * time.Millisecond
+ t.Cleanup(func() { saveAfter = 2 * time.Second })
+ history := []string{"hi Alice", "nothing here", "Bob, Alice and Alice"}
+ want, err := Find(history)
+ if err != nil {
+ t.Fatal(err)
+ }
+ waitFor(t, "the cache file", func() bool { _, err := os.Stat(cachePath()); return err == nil })
+ b, _ := os.ReadFile(cachePath())
+ for _, text := range history {
+ if strings.Contains(string(b), text) {
+ t.Fatalf("the cache file holds the text %q", text)
+ }
+ }
+ restart(t)
+ got, err := Find(append(history, "new Alice"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !reflect.DeepEqual(got[:3], want) || !reflect.DeepEqual(got[3], []redact.Found{person(4, 9)}) {
+ t.Fatalf("after a restart: %v, want %v and the new text's", got, want)
+ }
+ reqs := fakeLines(t, log, "request")
+ if len(reqs) != 2 || strings.Contains(reqs[1], "Bob") || !strings.Contains(reqs[1], "new Alice") {
+ t.Fatalf("requests = %q, want the one after the restart to ask about the new text alone", reqs)
+ }
+}
+
+// A cache file cut short, or not one, is read as empty: the model is asked
+// again, and its answers are right.
+func TestBadCacheFileIsReadAsEmpty(t *testing.T) {
+ log := fakeInstall(t, "ok", false)
+ if _, err := Find([]string{"hi Alice", "Bob, Alice"}); err != nil {
+ t.Fatal(err)
+ }
+ if err := saveCache(); err != nil {
+ t.Fatal(err)
+ }
+ whole, _ := os.ReadFile(cachePath())
+ for i, bad := range [][]byte{whole[:len(whole)-3], []byte("not a cache"), append(append([]byte{}, whole...), 0)} {
+ restart(t)
+ if err := os.WriteFile(cachePath(), bad, 0o644); err != nil {
+ t.Fatal(err)
+ }
+ before := len(fakeLines(t, log, "request"))
+ got, err := Find([]string{"hi Alice"})
+ if err != nil || !reflect.DeepEqual(got, [][]redact.Found{{person(3, 8)}}) {
+ t.Fatalf("file %d: %v %v", i, got, err)
+ }
+ if len(fakeLines(t, log, "request")) != before+1 {
+ t.Fatalf("file %d: a bad file's answers were used", i)
+ }
+ }
+}
+
+// A span read back that doesn't fit its text is asked about again.
+func TestCachedSpanOutsideItsTextIsAskedAgain(t *testing.T) {
+ log := fakeInstall(t, "ok", false)
+ cache.mu.Lock()
+ cache.put(sumOf("hi Alice"), []redact.Found{person(3, 80)})
+ cache.mu.Unlock()
+ disk.mu.Lock()
+ disk.loaded = cachePath()
+ disk.mu.Unlock()
+ got, err := Find([]string{"hi Alice"})
+ if err != nil || !reflect.DeepEqual(got, [][]redact.Found{{person(3, 8)}}) {
+ t.Fatalf("Find = %v %v", got, err)
+ }
+ if n := len(fakeLines(t, log, "request")); n != 1 {
+ t.Fatalf("asked %d times, want once", n)
+ }
+}
+
+func sumOf(s string) [32]byte { return sha256.Sum256([]byte(s)) }