~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Credential scrubbing in error/log text
local_deep_research.security.log_sanitizer scrubs credentials out of
exception/error strings before they reach a client (HTTP/SSE/JSON
responses) or the logs. It is a runtime, defense-in-depth sanitizer — not a
git/CI secret scanner (use gitleaks
for that, which this repo already runs in pre-commit/CI).
The two layers
sanitize_error_message(text)— a regex first-pass over_CREDENTIAL_PATTERNSfor common credential shapes (API-key query params,Authorization:/x-api-key:headers,user:pass@host, and well-known token prefixes:sk-/pk-, GoogleAIza/ya29., GitHubghp_/github_pat_, AWSAKIA/ASIA, Slackxox*-, JWTs).sanitize_error_for_client()composes it with control-char stripping + length capping. Log records get more coverage than this helper: the loguru patcher (redact_log_record) also recognizes prefixed and dotted credential assignments such asOPENAI_API_KEY=...andllm.openai.api_key=.... After its ordinary passes, it applies_ADDITIONAL_CREDENTIAL_PATTERNSto messages,extrastring leaves, and rendered exception fields. These patterns cover slash-bearing DSN passwords, unprefixedApi-Key:headers, credential parameters after;or,, and quoted credential keys in JSON or Python reprs. The database sink applies them to the exception prefix before and after its 240-character cap. Values whose boundaries the patterns cannot determine are left whole. Client and agent error helpers,scrub_error, andsanitize_error_messagedo not apply these additional log-only patterns; their output may pass through other redaction stages with different boundaries.sanitize_error_for_agent()is the agent/tool-facing sibling — it delegates tosanitize_error_for_client()with a 500-char cap instead of 200, so a classification signal such as a rate-limit phrase sitting deep in a provider message survives — and carries the same caveat: both remove credential shapes only, so use them on text already known to be safe apart from an embedded secret, never as a filter that makes an arbitrary exception safe to show.redact_secrets(text, *known_literals)— the backstop. When you hold the actual secret value (e.g. the configured API key), pass it here so it is scrubbed regardless of shape. This is the real guarantee for known secrets; the regex layer is best-effort for unknown ones.
Always pair them ("dual-scrub") on any path that surfaces secret-adjacent
text — via scrub_error(error, *known_literals) (security.log_sanitizer),
which composes both passes plus the defensive guards (a raising __str__
or a non-string secret cannot crash the except handler). Engine
subclasses use self._scrub_error(error), which resolves the engine's
_secret_attrs and delegates to the same helper. Don't hand-inline the
two calls: per-site copies are how scrub passes drift apart.
scrub_error keeps ordinary long diagnostics intact. When an error combines a
URL query with a credential assignment (including a name split by invisible
characters), preserving later log redaction requires more expensive tracking.
In each scrub round, those diagnostics keep at most the log redactor's 32 KiB
prefix, cut at its whitespace-safe boundary, before that tracking runs. Shape
and known-literal redaction run before the cut, so a known secret containing
spaces cannot be cut into an unrecognized fragment. The omitted tail is
replaced by one character-count marker after the fixed-point rounds. The
initial shape and literal scans still process the full input; the bound applies
to the added query-composition tracking. As with the other length-capped
helpers, scrubbing above-limit output again may omit additional text.
Design constraints (why it is curated, not exhaustive)
This runs on arbitrary, possibly attacker-influenced strings at runtime, so:
- No ReDoS. Every pattern must scale linearly — prefer prefix-anchored, single-quantifier regexes; avoid nested quantifiers / ambiguous alternations. Spot-check new patterns on a 200k-char adversarial input.
- Over-redaction is the safe failure; under-redaction (a leak) is not — but gratuitous over-redaction harms log readability, so patterns are anchored/length-floored to avoid eating prose.
- Idempotent, and never by redacting less. Scrubbed text gets scrubbed
again (an error is scrubbed, re-raised and scrubbed by an outer handler;
the log patcher re-runs on every record), so each public scrub function
returns its own output unchanged, within the limits below. They get
there by repeating their passes until the text stops changing
(
_scrub_to_fixed_point, at most five rounds), not by narrowing a pattern: the first round includes all existing redaction readings, and later rounds only replace more text. When a new stop rule shortens a match, keep the previous reading in the pipeline union so its masked values stay masked; otherwise text after the stop could be logged.TestScrubIsIdempotentandTestScrubNeverRedactsLessThanOnePassintests/security/test_log_sanitizer.pycheck both directions. Limits: a length cap is not a fixed point, and a known secret can occur inside the marker.redact_log_messageand the log patcher cut text longer than_LOG_REDACTION_MAX_CHARSand append an omitted-length marker, and redaction can push text over that bound (token=abecomestoken=[REDACTED]), so scrubbing such output again cuts it again; that only drops more text, never shows more.sanitize_error_for_clientandsanitize_error_for_agentare only guaranteed fixed points below their length cap. Andscrub_errorgiven a known secret that occurs inside its own***REDACTED***marker (such asREDACTED) lengthens the marker on every call. - Keep the set small and high-signal. Matching gitleaks' full 100+ rules
at runtime is overkill and raises false positives; the
redact_secretsbackstop covers the long tail for known secrets.
Keeping the patterns current
The prefix regexes mirror the canonical, actively-maintained gitleaks
ruleset — gitleaks' own upstream config/gitleaks.toml
(https://github.com/gitleaks/gitleaks), not this repo's root
.gitleaks.toml (which only configures the gitleaks scan run in
pre-commit/CI). To refresh when a provider introduces a new token format:
- Find the rule in gitleaks' upstream
config/gitleaks.toml(or detect-secrets). - Adapt the regex to a prefix-anchored, ReDoS-safe form here; redact to
[REDACTED_KEY]. - Add a positive (redacts) and a negative (prose not over-redacted) case
to
tests/security/test_log_sanitizer.py. - Run
pytest tests/security/and a 200k-char ReDoS spot-check.
A periodic (e.g. quarterly) glance at gitleaks' changelog for new widely-used token prefixes is enough — this layer only needs the formats that plausibly appear in this app's error/log text.