Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

7.0 KB
Raw Blame

Credential scrubbing in error/log text

local_deep_research.security.log_sanitizer scrubs credentials out of exception/error strings before they reach a client (HTTP/SSE/JSON responses) or the logs. It is a runtime, defense-in-depth sanitizer — not a git/CI secret scanner (use gitleaks for that, which this repo already runs in pre-commit/CI).

The two layers

  1. sanitize_error_message(text) — a regex first-pass over _CREDENTIAL_PATTERNS for common credential shapes (API-key query params, Authorization:/ x-api-key: headers, user:pass@host, and well-known token prefixes: sk-/pk-, Google AIza/ya29., GitHub ghp_/github_pat_, AWS AKIA/ASIA, Slack xox*-, JWTs). sanitize_error_for_client() composes it with control-char stripping + length capping. Log records get more coverage than this helper: the loguru patcher (redact_log_record) also recognizes prefixed and dotted credential assignments such as OPENAI_API_KEY=... and llm.openai.api_key=.... After its ordinary passes, it applies _ADDITIONAL_CREDENTIAL_PATTERNS to messages, extra string leaves, and rendered exception fields. These patterns cover slash-bearing DSN passwords, unprefixed Api-Key: headers, credential parameters after ; or ,, and quoted credential keys in JSON or Python reprs. The database sink applies them to the exception prefix before and after its 240-character cap. Values whose boundaries the patterns cannot determine are left whole. Client and agent error helpers, scrub_error, and sanitize_error_message do not apply these additional log-only patterns; their output may pass through other redaction stages with different boundaries. sanitize_error_for_agent() is the agent/tool-facing sibling — it delegates to sanitize_error_for_client() with a 500-char cap instead of 200, so a classification signal such as a rate-limit phrase sitting deep in a provider message survives — and carries the same caveat: both remove credential shapes only, so use them on text already known to be safe apart from an embedded secret, never as a filter that makes an arbitrary exception safe to show.
  2. redact_secrets(text, *known_literals) — the backstop. When you hold the actual secret value (e.g. the configured API key), pass it here so it is scrubbed regardless of shape. This is the real guarantee for known secrets; the regex layer is best-effort for unknown ones.

Always pair them ("dual-scrub") on any path that surfaces secret-adjacent text — via scrub_error(error, *known_literals) (security.log_sanitizer), which composes both passes plus the defensive guards (a raising __str__ or a non-string secret cannot crash the except handler). Engine subclasses use self._scrub_error(error), which resolves the engine's _secret_attrs and delegates to the same helper. Don't hand-inline the two calls: per-site copies are how scrub passes drift apart.

scrub_error keeps ordinary long diagnostics intact. When an error combines a URL query with a credential assignment (including a name split by invisible characters), preserving later log redaction requires more expensive tracking. In each scrub round, those diagnostics keep at most the log redactor's 32 KiB prefix, cut at its whitespace-safe boundary, before that tracking runs. Shape and known-literal redaction run before the cut, so a known secret containing spaces cannot be cut into an unrecognized fragment. The omitted tail is replaced by one character-count marker after the fixed-point rounds. The initial shape and literal scans still process the full input; the bound applies to the added query-composition tracking. As with the other length-capped helpers, scrubbing above-limit output again may omit additional text.

Design constraints (why it is curated, not exhaustive)

This runs on arbitrary, possibly attacker-influenced strings at runtime, so:

  • No ReDoS. Every pattern must scale linearly — prefer prefix-anchored, single-quantifier regexes; avoid nested quantifiers / ambiguous alternations. Spot-check new patterns on a 200k-char adversarial input.
  • Over-redaction is the safe failure; under-redaction (a leak) is not — but gratuitous over-redaction harms log readability, so patterns are anchored/length-floored to avoid eating prose.
  • Idempotent, and never by redacting less. Scrubbed text gets scrubbed again (an error is scrubbed, re-raised and scrubbed by an outer handler; the log patcher re-runs on every record), so each public scrub function returns its own output unchanged, within the limits below. They get there by repeating their passes until the text stops changing (_scrub_to_fixed_point, at most five rounds), not by narrowing a pattern: the first round includes all existing redaction readings, and later rounds only replace more text. When a new stop rule shortens a match, keep the previous reading in the pipeline union so its masked values stay masked; otherwise text after the stop could be logged. TestScrubIsIdempotent and TestScrubNeverRedactsLessThanOnePass in tests/security/test_log_sanitizer.py check both directions. Limits: a length cap is not a fixed point, and a known secret can occur inside the marker. redact_log_message and the log patcher cut text longer than _LOG_REDACTION_MAX_CHARS and append an omitted-length marker, and redaction can push text over that bound (token=a becomes token=[REDACTED]), so scrubbing such output again cuts it again; that only drops more text, never shows more. sanitize_error_for_client and sanitize_error_for_agent are only guaranteed fixed points below their length cap. And scrub_error given a known secret that occurs inside its own ***REDACTED*** marker (such as REDACTED) lengthens the marker on every call.
  • Keep the set small and high-signal. Matching gitleaks' full 100+ rules at runtime is overkill and raises false positives; the redact_secrets backstop covers the long tail for known secrets.

Keeping the patterns current

The prefix regexes mirror the canonical, actively-maintained gitleaks ruleset — gitleaks' own upstream config/gitleaks.toml (https://github.com/gitleaks/gitleaks), not this repo's root .gitleaks.toml (which only configures the gitleaks scan run in pre-commit/CI). To refresh when a provider introduces a new token format:

  1. Find the rule in gitleaks' upstream config/gitleaks.toml (or detect-secrets).
  2. Adapt the regex to a prefix-anchored, ReDoS-safe form here; redact to [REDACTED_KEY].
  3. Add a positive (redacts) and a negative (prose not over-redacted) case to tests/security/test_log_sanitizer.py.
  4. Run pytest tests/security/ and a 200k-char ReDoS spot-check.

A periodic (e.g. quarterly) glance at gitleaks' changelog for new widely-used token prefixes is enough — this layer only needs the formats that plausibly appear in this app's error/log text.